feat: add RELLIS dataset compatibility smoke

2026-07-25 20:45:30 +03:00
parent 13c35ff446
commit 054feec0d2
19 changed files with 1633 additions and 277 deletions
@@ -203,6 +203,14 @@ qualification bootstrap, never the product data plane. A shared
 operator-facing deployment still requires authentication/RBAC, service
 supervision and an accepted routed worker transport.

+The Dataset Gateway is now multi-source. RELLIS S0 pins official repository
+frame `000104`, verifies `131,072` Ouster XYZI points against aligned labels and
+publishes a bounded semantic/ground-target preview beside GOOSE in the same
+`Полигон → Датасеты` workflow. This closes only reader, axes, ontology,
+licensing and versioned ignore-policy compatibility. Full RELLIS Ouster scans,
+labels and poses remain D-only admission work before the cross-dataset
+Current/Patchwork++ decision.
+
 Mission Core remains a distributed web product. React is served by a long-lived
 Gateway/API and renders canonical state through browser-native components.
 Gazebo/PX4/ROS 2/Nav2 run headless on a registered Linux simulation worker; the
@@ -878,6 +878,11 @@ workflow:
 - `Датасеты → Открыть датасет` shows the admitted source. GOOSE validation is
  explicitly grouped into eight independent fragments; Play is bounded to one
  fragment and described as accelerated review of sparse annotated scans.
+- The catalog is multi-source. RELLIS-3D v1.1 appears as an independent Ouster
+  domain with its own license and admission state. Its current `smoke-ready`
+  view shows official frame `000104`, semantic colors, FLU axes and the
+  versioned `ground / non-ground / ignore` mapping; it does not show invented
+  algorithm metrics.
 - The dataset viewer exposes source frame number, nanosecond timestamp gaps,
  ground truth and Current/Patchwork++ overlays.
 - The fixed sensor-centric coordinate frame and operator camera are preserved
@@ -894,6 +899,12 @@ The accepted `3.3 GB` annotated validation ZIP is not a continuous recording.
 Continuous ego-motion requires a separately admitted raw ROS bag and
 localization source.

+RELLIS S0 uses only a bounded derivative of the pinned official example for
+browser inspection. The full `14 GB` Ouster SemanticKITTI scans, labels and
+poses remain a separate D-only admission gate. Its `CC-BY-NC-SA-3.0` license
+means cross-dataset research evidence cannot silently become an unrestricted
+commercial runtime or training dependency.
+
 The backend reads the configured repository from
 `MISSIONCORE_POLYGON_RUNS_ROOT` using
 `QualificationRunStore(read_only=True)`. It exposes only:
@@ -293,7 +293,7 @@ path. K1 manual review or a new real vehicle dataset is reserved for later
 domain adaptation after a public baseline proves that the pipeline and metric
 harness work.

-### L2.5 — Dataset Gateway — first-frame A/B complete
+### L2.5 — Dataset Gateway — GOOSE qualification and RELLIS S0 complete

 - [x] Define separate `native-scan`, `normalized-scan` and
  `rolling-local-map` representations.
@@ -317,12 +317,21 @@ harness work.
  through an immutable R1 Polygon replay-shadow run.
 - [x] Add named range/FOV/density/noise/dropout degradation profiles without
  overwriting the native frame.
+- [x] Upgrade the gateway to two independently identified sources instead of
+  treating GOOSE as a universal dataset contract.
+- [x] Pin and verify the official RELLIS-3D Ouster example, preserve its
+  `131,072` point/label alignment and publish a bounded semantic viewer.
+- [x] Version the RELLIS `ground / non-ground / ignore` mapping and expose
+  licensing as research-only evidence.
+- [ ] Admit the full RELLIS Ouster SemanticKITTI scans, labels and poses on
+  worker D, then run the same Current/Patchwork++ harness.

 The architectural contract and run sequence are fixed in
 `docs/14_LIDAR_DATASET_GATEWAY.md` and ADR 0021. GOOSE is first because its
 published 3D format is one LiDAR revolution with point-wise semantic and
-instance labels in off-road environments. RELLIS-3D remains the second-source
-cross-check after the GOOSE harness is stable.
+instance labels in off-road environments. The RELLIS-3D compatibility smoke is
+now complete against official frame `000104`; the full second-source algorithm
+cross-check remains the next gate.

 ### L3 — LiDAR-native 3D detection

@@ -394,7 +403,10 @@ The near-term value is not a prettier point cloud:
 The GOOSE validation-split and deterministic degradation gates are complete:
 pinned Patchwork++ improves micro Ground IoU from `47.42%` to `66.40%`, raises
 natural-ground recall from `51.76%` to `76.22%` and stays below `23.41 ms` p95
-across 961 frames. The decision remains shadow-only. The highest-value
-immediate work is now a second-source cross-check plus LiDAR-native detection
-on the same gateway. Nvblox and alternative SLAM remain later because their
-timing, pose and scan-geometry gates are not yet satisfied.
+across 961 frames. RELLIS S0 proves that the shared reader, axes, ontology and
+ground-target mapping work on a different Ouster off-road domain; it does not
+yet score an algorithm. The decision remains shadow-only. The highest-value
+immediate work is full RELLIS ground qualification, followed by the real
+sensor input contract and LiDAR-native detection on the same gateway. Nvblox
+and alternative SLAM remain later because their timing, pose and scan-geometry
+gates are not yet satisfied.
@@ -19,12 +19,13 @@ datasets and future real onboard sensors.

 Implemented now:

- `missioncore.dataset-gateway-catalog/v2`, exposed read-only at
+- multi-source `missioncore.dataset-gateway-catalog/v3`, exposed read-only at
  `GET /api/v1/lidar/dataset-gateway`;
 - path-free `missioncore.dataset-admission/v1` worker evidence instead of a
  static React status;
- a bounded `missioncore.dataset-native-scan-preview/v1`, exposed read-only at
-  `GET /api/v1/lidar/dataset-gateway/preview`;
+- bounded `missioncore.dataset-native-scan-preview/v1` (GOOSE) and `/v2`
+  (RELLIS smoke) artifacts, selected by source at
+  `GET /api/v1/lidar/dataset-gateway/preview?source_id=...`;
 - explicit `native-scan`, `normalized-scan` and `rolling-local-map`
  representations;
 - a lossless GOOSE/SemanticKITTI frame reader for little-endian float32 XYZI
@@ -39,6 +40,10 @@ Implemented now:
  `Открыть` action;
 - one admitted GOOSE validation frame with source colors, normalized remission
  and an independent ground-truth view;
+- a pinned RELLIS-3D v1.1 compatibility smoke using the official Ouster example
+  frame `000104`, its native `131,072` XYZI points, point-aligned labels,
+  official ontology colors, explicit FLU sensor axes and a versioned
+  `ground / non-ground / ignore` evaluation policy;
 - a published, dimensioned MuCAR-3/VLS-128 Patchwork++ input profile:
  sensor frame `x-forward / y-left / z-up`, physical height `2.24 m`, native
  one-revolution scan and explicit absence of deskew/full vehicle TF claims;
@@ -54,6 +59,8 @@ Not implemented:
 - no implicit coordinate conversion;
 - no fake ring/timestamp reconstruction for K1 MQTT evidence;
 - no model training or production promotion;
+- no full RELLIS Ouster archive, pose set, algorithm comparison or ROS bag
+  admission yet;
 - no rolling-map implementation yet.

 ## Product surface boundary
@@ -73,10 +80,11 @@ The gateway is not part of device quality diagnostics.
  backend evidence contract. They are not a second operator navigation item
  and are not presented as vehicle motion.

-Before real dataset bytes exist, the catalog explains the blocked storage gate
-and keeps `Открыть датасет` disabled. It becomes available
-only after the worker manifest says `frame-ready` and the bounded preview passes
-its own point-alignment and safety checks.
+Before compatible source evidence exists, the catalog explains the blocked
+storage gate and keeps `Открыть датасет` disabled. GOOSE becomes available only
+after the worker manifest says `frame-ready`; RELLIS becomes available at
+`smoke-ready` only for the pinned official example. Both bounded previews must
+pass their own identity, point-alignment and safety checks.

 ## Why public recordings look different

@@ -191,8 +199,53 @@ time; TTL/dynamic filtering prevents stale ghosts.
 - [x] Patchwork++ one-frame accuracy measured against ground truth.
 - [x] Current/Patchwork++ validation-split gate qualified.
 - [x] Sensor-degradation matrix qualified.
+- [x] Dataset catalog accepts more than one independently identified source.
+- [x] Official RELLIS Ouster example passes the shared SemanticKITTI reader.
+- [x] RELLIS axes, ontology colors and versioned ground-target/ignore mapping
+  are visible in the same Dataset workflow.
+- [ ] Full RELLIS Ouster SemanticKITTI scans, labels and poses admitted on
+  worker D.
+- [ ] Current/Patchwork++ RELLIS comparison qualified without obstacle loss.
 - [ ] Rolling local map with pose/TTL/dynamic policy qualified.

+## RELLIS S0 compatibility evidence
+
+RELLIS is the independent second-source check, not extra decoration in the
+catalog. GOOSE and RELLIS share the ingress format and viewer, but retain
+separate source identity, sensor domain, ontology, licensing and results.
+
+The pinned smoke input is the official repository example at commit
+`c17a118fcaed1559f03cc32cc3a91dedc557f8b8`:
+
+- point file: `utils/example/000104.bin`;
+- label file: `utils/example/000104.label`;
+- source points: `131,072`;
+- point SHA-256:
+  `ed81a9c3636d55b17d78058c72545d5d22419beecf174d50596d23ae178752af`;
+- label SHA-256:
+  `9b8c65b710873e931af4ac6dfc7d3dd2298696514ab721e50300bd55ad5b634e`;
+- official ontology config SHA-256:
+  `573379a232ac561805987466c391a61fd5ad338be7fb9c4c2842f3f28067e0ad`;
+- bounded browser preview: `20,000` deterministic even-index points;
+- coordinate frame: right-handed, `x` forward, `y` left, `z` up; no vehicle
+  transform, pose registration or deskew is claimed;
+- license: `CC-BY-NC-SA-3.0`; this is a research qualification source, not an
+  unrestricted commercial runtime dependency.
+
+Mission Core ground policy v1 treats `dirt`, `grass`, `asphalt`, `concrete` and
+`mud` as ground; clear structures, vegetation, people, vehicles and rubble as
+non-ground; and `void`, `water`, `sky`, generic `object` and `puddle` as
+`ignore`. The ignore set is excluded from future metrics instead of being
+silently counted as either free ground or obstacle. This mapping is a Mission
+Core evaluation decision, not an upstream RELLIS claim.
+
+The smoke closes reader, alignment, axes, palette and mapping compatibility
+only. The next gate admits the `14 GB` Ouster SemanticKITTI scans plus the
+`174 MB` labels and `174 MB` poses to D-only worker storage. Full ROS bags are
+not needed for the ground cross-check. Patchwork++ may be compared only after a
+physical Ouster mounting-height profile is evidenced; no height is inferred
+from the example cloud.
+
 ## First GOOSE admission evidence

 - worker root: canonical D-only root, path never exposed by the HTTP API;
@@ -39,6 +39,12 @@ first admitted source because it publishes off-road point-wise semantic and
 instance labels in SemanticKITTI-compatible `XYZI + uint32 label` files. Its
 annotated point-cloud file represents one LiDAR revolution.

+The gateway contract is multi-source, not GOOSE-shaped. RELLIS-3D v1.1 is the
+second source: it uses the same lossless SemanticKITTI ingress but keeps a
+separate Ouster OS1 domain, ontology, admission state, ground-target mapping,
+license and result set. Shared format never implies shared calibration,
+mounting height or commercial-use rights.
+
 The gateway:

 - preserves the native GOOSE frame before adaptation;
@@ -48,6 +54,10 @@ The gateway:
 - refuses automatic downloads of large archives;
 - admits storage only under `D:\NDC_MISSIONCORE\datasets` or its WSL mirror;
 - never promotes K1 `lio_pcl` to `native-scan`.
+- excludes explicitly ambiguous RELLIS classes through a versioned `ignore`
+  mask instead of silently scoring them as ground or obstacle;
+- exposes the RELLIS `CC-BY-NC-SA-3.0` restriction as research evidence and
+  never treats it as an unrestricted production dependency.

 Large-artifact execution remains worker-local. The browser receives only a
 path-free admission manifest and a bounded visualization preview. The canonical
@@ -82,6 +92,8 @@ unavailable rather than inventing timestamps.
 - Until bytes are installed, the UI exposes an explicit empty state. After a
  `frame-ready` worker admission, the same surface becomes an interactive
  source/remission/ground-truth viewer without changing the representation.
+- A bounded `smoke-ready` source may prove reader, axes, ontology and mapping
+  compatibility without claiming full-dataset or algorithm qualification.
 - Dataset and device inputs can share downstream algorithms only after their
  normalized contracts match.
 - Sensor adaptation may change range, FOV, point density, noise and dropout for
@@ -115,6 +127,7 @@ unavailable rather than inventing timestamps.
 - [GOOSE MuCAR-3 sensor setup](https://goose-dataset.de/docs/mucar3/)
 - [GOOSE paper and dimensioned sensor schematic](https://arxiv.org/pdf/2310.16788)
 - [Patchwork++ v1.4.1](https://github.com/url-kaist/patchwork-plusplus/tree/v1.4.1)
+- [RELLIS-3D official repository](https://github.com/unmannedlab/RELLIS-3D)
 - [Livox ROS Driver 2 point formats](https://github.com/Livox-SDK/livox_ros_driver2)
 - [Livox LIO motion-distortion handling](https://github.com/Livox-SDK/LIO-Livox)
 - [ROS FilterDeskew timestamp requirement](https://docs.ros.org/en/noetic/api/mp2p_icp/html/classmp2p__icp__filters_1_1FilterDeskew.html)