diff --git a/docs/20_SIMULATION_WORLD_SURFACE_BRIEF.md b/docs/20_SIMULATION_WORLD_SURFACE_BRIEF.md index 51be5c8..b97dd50 100644 --- a/docs/20_SIMULATION_WORLD_SURFACE_BRIEF.md +++ b/docs/20_SIMULATION_WORLD_SURFACE_BRIEF.md @@ -122,3 +122,23 @@ Primary implementation references: - a ready project mounts direct PlayCanvas Engine and loads Streamed SOG with preview fallback; - visual, collision and combined remain distinct runtime modes, and collision is loaded lazily; - viewer quality, layer-axis correction and camera inversion survive navigation and reload. + +## AI-assisted visual quality extension + +AI-assisted visual repair is a candidate lifecycle of one ready Simulation World project. It does +not replace or modify the accepted ingest/optimization path. The original LCC/LCC2 bundle remains +the source of record, while an explicitly promoted enhanced generation may replace only the active +visual SOG references. + +The product entry point is `Улучшить качество` in the existing project edit window. It opens a +canonical bounded Window for XGRIDS Creator Data admission, region-of-interest selection, actual +provider state and baseline/candidate review. It is not placed in the live scene controls because +source admission and a long-running candidate build are project operations. It does not receive a +separate workspace because the candidate has no identity outside its parent project. + +The control is not shipped as a placeholder. It becomes available only with a ready project and an +enhancement provider publishing the accepted capability. The provider consumes full-quality PLY +derived from the immutable LCC/LCC2 source plus aligned Creator Data images/COLMAP cameras; SOG is +delivery output only. Generated visual regions remain forbidden as collision, navigation or +qualification evidence. The detailed boundary and first experiment are defined by +[ADR 0044](adr/0044-ai-assisted-gaussian-visual-repair.md). diff --git a/docs/adr/0044-ai-assisted-gaussian-visual-repair.md b/docs/adr/0044-ai-assisted-gaussian-visual-repair.md new file mode 100644 index 0000000..3a366c8 --- /dev/null +++ b/docs/adr/0044-ai-assisted-gaussian-visual-repair.md @@ -0,0 +1,136 @@ +# ADR 0044: AI-assisted Gaussian visual repair remains a candidate pipeline + +## Status + +Proposed for a Worker 006 ROI experiment on 2026-08-29. The existing Gaussian optimization and +collision paths remain accepted and unchanged. No repair model is admitted for production use by +this decision. + +## Context + +The current Simulation Worlds vertical accepts one XGRIDS LCC/LCC2 source, builds preview and +streamed SOG artifacts through DC Gaussian Pipeline, and publishes source-mesh collision +independently. The two current ready projects prove that path. AutoCap repairs only conservative +holes in the source collision mesh; it does not repair the visual Gaussian representation. + +The retained MAROSEYKA archive contains an LCC Quality scene with 86,471,152 splats and a +44,787-sample 10 Hz device trajectory. It does not contain the captured camera images, camera +intrinsics or camera extrinsics. The pose records do not identify RGB frames and cannot be treated +as calibrated camera poses. + +AI repair methods that permanently improve a 3DGS model need rendered novel views, real reference +images and known cameras. Plausible generation alone cannot recover the factual appearance or +geometry of a surface that was never observed. + +XGRIDS LCC Studio Creator Data provides the missing supported interchange boundary: + +- `perspective/images/` contains undistorted perspective images; +- `perspective/masks/` contains invalid-region masks; +- `perspective/sparse/` contains COLMAP camera intrinsics and extrinsics aligned with the optimized + LiDAR point cloud; +- `poses.csv` and `high_frequency_poses.csv` retain device trajectories when they are useful for + selecting a repair corridor. + +## Decision + +1. The immutable original LCC/LCC2 bundle is the visual source of record. SOG is a compressed web + delivery artifact and is never the input to AI repair. +2. A repair candidate starts from full-quality `LCC/LCC2 -> standard 3DGS PLY` conversion. It may + operate only on bounded spatial tiles or route-derived regions of interest; loading or training + the complete 86M-splat scene as one model is not an accepted Worker 006 profile. +3. A repair request additionally admits XGRIDS Creator Data. The minimum input is undistorted + images plus a complete COLMAP sparse model. Masks are strongly preferred. Device poses alone do + not satisfy camera admission. +4. The first experiment uses FreeFix at an exact source revision with the SDXL refinement path. It + was selected because it is fine-tuning-free at the diffusion-model level, uses per-pixel + confidence to preserve reliable regions, reports outdoor/Waymo evaluation, and publishes MIT + code. An adapter must import standard XGRIDS/PlayCanvas PLY attributes into the gsplat checkpoint + layout and export a standard PLY after refinement. +5. Difix3D+ is the mandatory comparison baseline for the same ROI. It directly targets artifacts in + under-constrained views and supports progressive distillation into gsplat, but its combined + NVIDIA/Stability licensing needs a separate commercial-use review. +6. The experiment runs in an adjacent `ndc-` prefixed Docker composition on Worker 006. It shares + neither Python/CUDA environments nor writable model directories with DC Gaussian Pipeline. The + existing pipeline is called only after a candidate PLY is complete and immutable. +7. Every result is a candidate generation. Mission Core keeps the active visual generation until + an operator compares the baseline and candidate and explicitly promotes the candidate. Failure + or cancellation cannot modify the active scene. +8. Generated visual content never becomes collision, navigation, traversability or ground-truth + evidence. Existing source-mesh collision and AutoCap remain independent. Each candidate retains + an uncertainty/hallucination mask and exact model/config provenance. + +## Candidate contract + +A quality-enhancement request must bind all inputs by digest: + +- project ID and original source-bundle SHA-256; +- LCC/LCC2 entrypoint and full-quality PLY conversion revision; +- Creator Data bundle SHA-256; +- COLMAP cameras, images and points model plus their coordinate-alignment report; +- selected route interval and/or world-space ROI; +- algorithm, source revision, model IDs/digests, prompt policy and numerical parameters. + +The adjacent provider returns: + +- a standard enhanced PLY for the selected tile or composed candidate; +- preview and streamed SOG artifacts created by the unchanged optimization pipeline; +- baseline/candidate camera-path renders and objective image metrics where held-out real views + exist; +- changed-region, confidence and generated-content masks; +- wall time, peak VRAM, source revision, image digest and terminal job state. + +The provider states are transport-neutral and bounded: `queued`, `validating_source`, +`aligning_cameras`, `extracting_roi`, `importing_splats`, `refining`, `building_delivery`, +`evaluating`, `ready`, `failed`, and `cancelled`. + +## Product placement + +The action belongs in the existing project edit window as `Улучшить качество`, because it acts on +one durable world and does not create a new workspace. The action opens a canonical bounded Window +that owns Creator Data admission, ROI choice, actual provider state and baseline/candidate review. +It is shown only for a ready outdoor/interior project and becomes an executable action only when +the enhancement provider publishes the exact accepted capability. + +Alternatives considered: + +1. Add controls to the live PlayCanvas scene. Rejected because source admission and a long-running + candidate lifecycle are project operations, not per-frame runtime controls. +2. Add a new top-level workspace. Rejected because repair has no independent identity outside one + Simulation World project. +3. Add an always-enabled button before provider/source admission exists. Rejected because it would + be placeholder product UI and would misrepresent the current system state. + +## Experiment acceptance + +The first ROI experiment is accepted only when: + +- the adjacent composition starts and stops without changing the current Gaussian Pipeline + containers or native SplatTransform spool; +- one Creator Data bundle passes COLMAP/image/alignment validation; +- one bounded road-and-facade ROI fits the RTX 4090 24 GB profile without host OOM or impact on the + active optimization queue; +- FreeFix and Difix3D+ run on identical admitted cameras and ROI; +- candidate output round-trips through standard PLY and the existing SOG build; +- held-out views and an operator review show improvement without unacceptable changes in reliable + regions; +- active SOG and collision artifacts remain byte-identical until explicit promotion. + +## Consequences + +- The first useful input request is XGRIDS Creator Data, not a raw device track and not SOG. +- Existing ready RAR archives cannot start factual image-conditioned repair by themselves. +- Large scenes require ROI/tile scheduling, overlap blending and candidate composition. +- Visually plausible fill may be valuable for rendering while remaining inadmissible as physical + truth. +- Worker deployment is intentionally blocked until a Creator Data sample and restored Worker 006 + provider connectivity are available. + +## Primary references + +- [XGRIDS Creator Data](https://docs.xgrids.com/en-us/06-lixel-cybercolor/01-lcc-studio/v2.3.0/06-model-reconstruction.html#creator-data-and-nvidia-ncore-data) +- [PlayCanvas SplatTransform](https://github.com/playcanvas/splat-transform) +- [FreeFix](https://github.com/hyzhou404/FreeFix) +- [Difix3D+](https://github.com/nv-tlabs/Difix3D) +- [GSFix3D](https://github.com/GSFix3D/GSFix3D) +- [ArtifactWorld](https://github.com/fyting/ArtifactWorld) +