NODEDC_MISSION_CORE/docs/adr/0021-dataset-gateway-representation-boundary.md

# ADR 0021: separate native scans, normalized scans and rolling local maps

Status: accepted
Date: 2026-07-25

## Context

Public autonomous-driving and field-robotics dataset viewers commonly display
one LiDAR revolution. Fixed-channel rotating sensors therefore produce the
familiar sparse rings. The current XGRIDS K1 MQTT `lio_pcl` evidence is a
different product: firmware evidence places it after LIO/modeling, and the
field-review UI accumulates multiple already registered increments in the map
frame.

Point count alone does not make these representations comparable. A cloud can
be sparse per publication and still look dense after several seconds of pose
registration. Conversely, voxel downsampling does not restore timing, scan
lines or raw sensor geometry that the source no longer carries.

Patchwork++ is a ground classifier. It does not decode sensor packets, deskew
motion distortion, estimate pose, stabilize a rolling map or remove ghosts
from stale/dynamic observations. A successful Patchwork++ call against
`lio_pcl` remains diagnostic and does not repair the input domain.

## Decision

Mission Core defines three non-interchangeable LiDAR products:

1. `native-scan`: one losslessly decoded source scan/frame with its native
   point-aligned fields and labels. It is never accumulated.
2. `normalized-scan`: one sensor-frame scan after an explicit transform,
   deskew and bounded cleanup profile. Every transformation retains source
   identity and point alignment.
3. `rolling-local-map`: normalized scans registered by pose into a bounded
   local map with explicit TTL, voxel deduplication and dynamic-point policy.

The Dataset Gateway is the first producer of this contract. GOOSE 3D is the
first admitted source because it publishes off-road point-wise semantic and
instance labels in SemanticKITTI-compatible `XYZI + uint32 label` files. Its
annotated point-cloud file represents one LiDAR revolution.

The gateway:

- preserves the native GOOSE frame before adaptation;
- never transforms labels independently of their points;
- does not assume a coordinate convention, mounting transform or sensor height
  unless source metadata supplies it;
- refuses automatic downloads of large archives;
- admits storage only under `D:\NDC_MISSIONCORE\datasets` or its WSL mirror;
- never promotes K1 `lio_pcl` to `native-scan`.

Large-artifact execution remains worker-local. The browser receives only a
path-free admission manifest and a bounded visualization preview. The canonical
archive, extracted source frame, labels, mapping and LICENSE remain under the
worker D root. SSH/SCP may mirror the small manifest and preview during the
recorded laboratory bootstrap; it is not the product data plane.

The normalized pipeline is:

```text
native packet/source frame
  -> decode + calibration
  -> per-point-time deskew
  -> range/self/outlier/voxel policy
  -> normalized-scan
  -> ground/object inference
  -> pose registration + TTL + voxel deduplication
  -> rolling-local-map
```

Deskew is conditional: it requires per-point time plus synchronized IMU or
odometry. If those fields are missing, the gateway reports the stage as
unavailable rather than inventing timestamps.

## Consequences

- The UI must label accumulated K1 evidence as a map product, not a scan.
- Real-sensor diagnostics live under Fleet and never mix public dataset frames
  into device evidence.
- Dataset catalog, preview and qualification-run entry live under Polygon;
  Data remains the source-of-record and artifact store.
- Until bytes are installed, the UI exposes an explicit empty state. After a
  `frame-ready` worker admission, the same surface becomes an interactive
  source/remission/ground-truth viewer without changing the representation.
- Dataset and device inputs can share downstream algorithms only after their
  normalized contracts match.
- Sensor adaptation may change range, FOV, point density, noise and dropout for
  robustness experiments, but it cannot recreate lost timestamps, occlusions
  or material response.
- Patchwork++ becomes eligible for an algorithm-specific quality gate on a
  sensor-centric scan with declared axes, physical mounting height and
  independent labels. That narrow admission does not itself create a general
  `normalized-scan`.
- Stable operator visualization is owned by rolling-map policy, not by the
  ground classifier.
- Public labels may qualify an algorithm independently of the production
  sensor. The first GOOSE frame exposed a `49.62%` Ground IoU and only `49.45%`
  natural-ground recall for the current local-percentile baseline; these are
  diagnostic results, not production promotion.
- A dataset label contract does not imply a mounting contract. For MuCAR-3,
  GOOSE's published dimensioned schematic admits the Patchwork-specific
  `2.24 m` sensor height (`0.64 + 1.60 m`) and `x-forward / y-left / z-up`
  axes. The complete numeric vehicle transform, deskew and general
  `normalized-scan` remain unavailable.
- On the first independently labeled frame, pinned Patchwork++ `v1.4.1`
  achieved `60.37%` Ground IoU, `71.59%` natural-ground recall and `15.89 ms`
  worker latency versus Current's `49.62%`, `49.45%` and `3966.53 ms`.
  This is a one-frame diagnostic candidate decision, not production promotion.

## Primary references

- [GOOSE dataset structure](https://goose-dataset.de/docs/dataset-structure/)
- [GOOSE setup and archive sizes](https://goose-dataset.de/docs/setup/)
- [GOOSE 3D challenge ontology](https://goose-dataset.de/docs/3d-semantic-segmentation-challenge/)
- [GOOSE MuCAR-3 sensor setup](https://goose-dataset.de/docs/mucar3/)
- [GOOSE paper and dimensioned sensor schematic](https://arxiv.org/pdf/2310.16788)
- [Patchwork++ v1.4.1](https://github.com/url-kaist/patchwork-plusplus/tree/v1.4.1)
- [Livox ROS Driver 2 point formats](https://github.com/Livox-SDK/livox_ros_driver2)
- [Livox LIO motion-distortion handling](https://github.com/Livox-SDK/LIO-Livox)
- [ROS FilterDeskew timestamp requirement](https://docs.ros.org/en/noetic/api/mp2p_icp/html/classmp2p__icp__filters_1_1FilterDeskew.html)