docs(sim): зафиксировать контур AI-улучшения гауссов
This commit is contained in:
@@ -122,3 +122,23 @@ Primary implementation references:
|
|||||||
- a ready project mounts direct PlayCanvas Engine and loads Streamed SOG with preview fallback;
|
- a ready project mounts direct PlayCanvas Engine and loads Streamed SOG with preview fallback;
|
||||||
- visual, collision and combined remain distinct runtime modes, and collision is loaded lazily;
|
- visual, collision and combined remain distinct runtime modes, and collision is loaded lazily;
|
||||||
- viewer quality, layer-axis correction and camera inversion survive navigation and reload.
|
- viewer quality, layer-axis correction and camera inversion survive navigation and reload.
|
||||||
|
|
||||||
|
## AI-assisted visual quality extension
|
||||||
|
|
||||||
|
AI-assisted visual repair is a candidate lifecycle of one ready Simulation World project. It does
|
||||||
|
not replace or modify the accepted ingest/optimization path. The original LCC/LCC2 bundle remains
|
||||||
|
the source of record, while an explicitly promoted enhanced generation may replace only the active
|
||||||
|
visual SOG references.
|
||||||
|
|
||||||
|
The product entry point is `Улучшить качество` in the existing project edit window. It opens a
|
||||||
|
canonical bounded Window for XGRIDS Creator Data admission, region-of-interest selection, actual
|
||||||
|
provider state and baseline/candidate review. It is not placed in the live scene controls because
|
||||||
|
source admission and a long-running candidate build are project operations. It does not receive a
|
||||||
|
separate workspace because the candidate has no identity outside its parent project.
|
||||||
|
|
||||||
|
The control is not shipped as a placeholder. It becomes available only with a ready project and an
|
||||||
|
enhancement provider publishing the accepted capability. The provider consumes full-quality PLY
|
||||||
|
derived from the immutable LCC/LCC2 source plus aligned Creator Data images/COLMAP cameras; SOG is
|
||||||
|
delivery output only. Generated visual regions remain forbidden as collision, navigation or
|
||||||
|
qualification evidence. The detailed boundary and first experiment are defined by
|
||||||
|
[ADR 0044](adr/0044-ai-assisted-gaussian-visual-repair.md).
|
||||||
|
|||||||
@@ -0,0 +1,136 @@
|
|||||||
|
# ADR 0044: AI-assisted Gaussian visual repair remains a candidate pipeline
|
||||||
|
|
||||||
|
## Status
|
||||||
|
|
||||||
|
Proposed for a Worker 006 ROI experiment on 2026-08-29. The existing Gaussian optimization and
|
||||||
|
collision paths remain accepted and unchanged. No repair model is admitted for production use by
|
||||||
|
this decision.
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
The current Simulation Worlds vertical accepts one XGRIDS LCC/LCC2 source, builds preview and
|
||||||
|
streamed SOG artifacts through DC Gaussian Pipeline, and publishes source-mesh collision
|
||||||
|
independently. The two current ready projects prove that path. AutoCap repairs only conservative
|
||||||
|
holes in the source collision mesh; it does not repair the visual Gaussian representation.
|
||||||
|
|
||||||
|
The retained MAROSEYKA archive contains an LCC Quality scene with 86,471,152 splats and a
|
||||||
|
44,787-sample 10 Hz device trajectory. It does not contain the captured camera images, camera
|
||||||
|
intrinsics or camera extrinsics. The pose records do not identify RGB frames and cannot be treated
|
||||||
|
as calibrated camera poses.
|
||||||
|
|
||||||
|
AI repair methods that permanently improve a 3DGS model need rendered novel views, real reference
|
||||||
|
images and known cameras. Plausible generation alone cannot recover the factual appearance or
|
||||||
|
geometry of a surface that was never observed.
|
||||||
|
|
||||||
|
XGRIDS LCC Studio Creator Data provides the missing supported interchange boundary:
|
||||||
|
|
||||||
|
- `perspective/images/` contains undistorted perspective images;
|
||||||
|
- `perspective/masks/` contains invalid-region masks;
|
||||||
|
- `perspective/sparse/` contains COLMAP camera intrinsics and extrinsics aligned with the optimized
|
||||||
|
LiDAR point cloud;
|
||||||
|
- `poses.csv` and `high_frequency_poses.csv` retain device trajectories when they are useful for
|
||||||
|
selecting a repair corridor.
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
1. The immutable original LCC/LCC2 bundle is the visual source of record. SOG is a compressed web
|
||||||
|
delivery artifact and is never the input to AI repair.
|
||||||
|
2. A repair candidate starts from full-quality `LCC/LCC2 -> standard 3DGS PLY` conversion. It may
|
||||||
|
operate only on bounded spatial tiles or route-derived regions of interest; loading or training
|
||||||
|
the complete 86M-splat scene as one model is not an accepted Worker 006 profile.
|
||||||
|
3. A repair request additionally admits XGRIDS Creator Data. The minimum input is undistorted
|
||||||
|
images plus a complete COLMAP sparse model. Masks are strongly preferred. Device poses alone do
|
||||||
|
not satisfy camera admission.
|
||||||
|
4. The first experiment uses FreeFix at an exact source revision with the SDXL refinement path. It
|
||||||
|
was selected because it is fine-tuning-free at the diffusion-model level, uses per-pixel
|
||||||
|
confidence to preserve reliable regions, reports outdoor/Waymo evaluation, and publishes MIT
|
||||||
|
code. An adapter must import standard XGRIDS/PlayCanvas PLY attributes into the gsplat checkpoint
|
||||||
|
layout and export a standard PLY after refinement.
|
||||||
|
5. Difix3D+ is the mandatory comparison baseline for the same ROI. It directly targets artifacts in
|
||||||
|
under-constrained views and supports progressive distillation into gsplat, but its combined
|
||||||
|
NVIDIA/Stability licensing needs a separate commercial-use review.
|
||||||
|
6. The experiment runs in an adjacent `ndc-` prefixed Docker composition on Worker 006. It shares
|
||||||
|
neither Python/CUDA environments nor writable model directories with DC Gaussian Pipeline. The
|
||||||
|
existing pipeline is called only after a candidate PLY is complete and immutable.
|
||||||
|
7. Every result is a candidate generation. Mission Core keeps the active visual generation until
|
||||||
|
an operator compares the baseline and candidate and explicitly promotes the candidate. Failure
|
||||||
|
or cancellation cannot modify the active scene.
|
||||||
|
8. Generated visual content never becomes collision, navigation, traversability or ground-truth
|
||||||
|
evidence. Existing source-mesh collision and AutoCap remain independent. Each candidate retains
|
||||||
|
an uncertainty/hallucination mask and exact model/config provenance.
|
||||||
|
|
||||||
|
## Candidate contract
|
||||||
|
|
||||||
|
A quality-enhancement request must bind all inputs by digest:
|
||||||
|
|
||||||
|
- project ID and original source-bundle SHA-256;
|
||||||
|
- LCC/LCC2 entrypoint and full-quality PLY conversion revision;
|
||||||
|
- Creator Data bundle SHA-256;
|
||||||
|
- COLMAP cameras, images and points model plus their coordinate-alignment report;
|
||||||
|
- selected route interval and/or world-space ROI;
|
||||||
|
- algorithm, source revision, model IDs/digests, prompt policy and numerical parameters.
|
||||||
|
|
||||||
|
The adjacent provider returns:
|
||||||
|
|
||||||
|
- a standard enhanced PLY for the selected tile or composed candidate;
|
||||||
|
- preview and streamed SOG artifacts created by the unchanged optimization pipeline;
|
||||||
|
- baseline/candidate camera-path renders and objective image metrics where held-out real views
|
||||||
|
exist;
|
||||||
|
- changed-region, confidence and generated-content masks;
|
||||||
|
- wall time, peak VRAM, source revision, image digest and terminal job state.
|
||||||
|
|
||||||
|
The provider states are transport-neutral and bounded: `queued`, `validating_source`,
|
||||||
|
`aligning_cameras`, `extracting_roi`, `importing_splats`, `refining`, `building_delivery`,
|
||||||
|
`evaluating`, `ready`, `failed`, and `cancelled`.
|
||||||
|
|
||||||
|
## Product placement
|
||||||
|
|
||||||
|
The action belongs in the existing project edit window as `Улучшить качество`, because it acts on
|
||||||
|
one durable world and does not create a new workspace. The action opens a canonical bounded Window
|
||||||
|
that owns Creator Data admission, ROI choice, actual provider state and baseline/candidate review.
|
||||||
|
It is shown only for a ready outdoor/interior project and becomes an executable action only when
|
||||||
|
the enhancement provider publishes the exact accepted capability.
|
||||||
|
|
||||||
|
Alternatives considered:
|
||||||
|
|
||||||
|
1. Add controls to the live PlayCanvas scene. Rejected because source admission and a long-running
|
||||||
|
candidate lifecycle are project operations, not per-frame runtime controls.
|
||||||
|
2. Add a new top-level workspace. Rejected because repair has no independent identity outside one
|
||||||
|
Simulation World project.
|
||||||
|
3. Add an always-enabled button before provider/source admission exists. Rejected because it would
|
||||||
|
be placeholder product UI and would misrepresent the current system state.
|
||||||
|
|
||||||
|
## Experiment acceptance
|
||||||
|
|
||||||
|
The first ROI experiment is accepted only when:
|
||||||
|
|
||||||
|
- the adjacent composition starts and stops without changing the current Gaussian Pipeline
|
||||||
|
containers or native SplatTransform spool;
|
||||||
|
- one Creator Data bundle passes COLMAP/image/alignment validation;
|
||||||
|
- one bounded road-and-facade ROI fits the RTX 4090 24 GB profile without host OOM or impact on the
|
||||||
|
active optimization queue;
|
||||||
|
- FreeFix and Difix3D+ run on identical admitted cameras and ROI;
|
||||||
|
- candidate output round-trips through standard PLY and the existing SOG build;
|
||||||
|
- held-out views and an operator review show improvement without unacceptable changes in reliable
|
||||||
|
regions;
|
||||||
|
- active SOG and collision artifacts remain byte-identical until explicit promotion.
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
- The first useful input request is XGRIDS Creator Data, not a raw device track and not SOG.
|
||||||
|
- Existing ready RAR archives cannot start factual image-conditioned repair by themselves.
|
||||||
|
- Large scenes require ROI/tile scheduling, overlap blending and candidate composition.
|
||||||
|
- Visually plausible fill may be valuable for rendering while remaining inadmissible as physical
|
||||||
|
truth.
|
||||||
|
- Worker deployment is intentionally blocked until a Creator Data sample and restored Worker 006
|
||||||
|
provider connectivity are available.
|
||||||
|
|
||||||
|
## Primary references
|
||||||
|
|
||||||
|
- [XGRIDS Creator Data](https://docs.xgrids.com/en-us/06-lixel-cybercolor/01-lcc-studio/v2.3.0/06-model-reconstruction.html#creator-data-and-nvidia-ncore-data)
|
||||||
|
- [PlayCanvas SplatTransform](https://github.com/playcanvas/splat-transform)
|
||||||
|
- [FreeFix](https://github.com/hyzhou404/FreeFix)
|
||||||
|
- [Difix3D+](https://github.com/nv-tlabs/Difix3D)
|
||||||
|
- [GSFix3D](https://github.com/GSFix3D/GSFix3D)
|
||||||
|
- [ArtifactWorld](https://github.com/fyting/ArtifactWorld)
|
||||||
|
|
||||||
Reference in New Issue
Block a user