SILVER
/
NODEDC_MISSION_CORE
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
NODEDC_MISSION_CORE/docs/adr/0021-dataset-gateway-repres...

6.0 KiB
Raw Blame History

ADR 0021: separate native scans, normalized scans and rolling local maps

Status: accepted Date: 2026-07-25

Context

Public autonomous-driving and field-robotics dataset viewers commonly display one LiDAR revolution. Fixed-channel rotating sensors therefore produce the familiar sparse rings. The current XGRIDS K1 MQTT lio_pcl evidence is a different product: firmware evidence places it after LIO/modeling, and the field-review UI accumulates multiple already registered increments in the map frame.

Point count alone does not make these representations comparable. A cloud can be sparse per publication and still look dense after several seconds of pose registration. Conversely, voxel downsampling does not restore timing, scan lines or raw sensor geometry that the source no longer carries.

Patchwork++ is a ground classifier. It does not decode sensor packets, deskew motion distortion, estimate pose, stabilize a rolling map or remove ghosts from stale/dynamic observations. A successful Patchwork++ call against lio_pcl remains diagnostic and does not repair the input domain.

Decision

Mission Core defines three non-interchangeable LiDAR products:

  1. native-scan: one losslessly decoded source scan/frame with its native point-aligned fields and labels. It is never accumulated.
  2. normalized-scan: one sensor-frame scan after an explicit transform, deskew and bounded cleanup profile. Every transformation retains source identity and point alignment.
  3. rolling-local-map: normalized scans registered by pose into a bounded local map with explicit TTL, voxel deduplication and dynamic-point policy.

The Dataset Gateway is the first producer of this contract. GOOSE 3D is the first admitted source because it publishes off-road point-wise semantic and instance labels in SemanticKITTI-compatible XYZI + uint32 label files. Its annotated point-cloud file represents one LiDAR revolution.

The gateway:

  • preserves the native GOOSE frame before adaptation;
  • never transforms labels independently of their points;
  • does not assume a coordinate convention, mounting transform or sensor height unless source metadata supplies it;
  • refuses automatic downloads of large archives;
  • admits storage only under D:\NDC_MISSIONCORE\datasets or its WSL mirror;
  • never promotes K1 lio_pcl to native-scan.

Large-artifact execution remains worker-local. The browser receives only a path-free admission manifest and a bounded visualization preview. The canonical archive, extracted source frame, labels, mapping and LICENSE remain under the worker D root. SSH/SCP may mirror the small manifest and preview during the recorded laboratory bootstrap; it is not the product data plane.

The normalized pipeline is:

native packet/source frame
  -> decode + calibration
  -> per-point-time deskew
  -> range/self/outlier/voxel policy
  -> normalized-scan
  -> ground/object inference
  -> pose registration + TTL + voxel deduplication
  -> rolling-local-map

Deskew is conditional: it requires per-point time plus synchronized IMU or odometry. If those fields are missing, the gateway reports the stage as unavailable rather than inventing timestamps.

Consequences

  • The UI must label accumulated K1 evidence as a map product, not a scan.
  • Real-sensor diagnostics live under Fleet and never mix public dataset frames into device evidence.
  • Dataset catalog, preview and qualification-run entry live under Polygon; Data remains the source-of-record and artifact store.
  • Until bytes are installed, the UI exposes an explicit empty state. After a frame-ready worker admission, the same surface becomes an interactive source/remission/ground-truth viewer without changing the representation.
  • Dataset and device inputs can share downstream algorithms only after their normalized contracts match.
  • Sensor adaptation may change range, FOV, point density, noise and dropout for robustness experiments, but it cannot recreate lost timestamps, occlusions or material response.
  • Patchwork++ becomes eligible for an algorithm-specific quality gate on a sensor-centric scan with declared axes, physical mounting height and independent labels. That narrow admission does not itself create a general normalized-scan.
  • Stable operator visualization is owned by rolling-map policy, not by the ground classifier.
  • Public labels may qualify an algorithm independently of the production sensor. The first GOOSE frame exposed a 49.62% Ground IoU and only 49.45% natural-ground recall for the current local-percentile baseline; these are diagnostic results, not production promotion.
  • A dataset label contract does not imply a mounting contract. For MuCAR-3, GOOSE's published dimensioned schematic admits the Patchwork-specific 2.24 m sensor height (0.64 + 1.60 m) and x-forward / y-left / z-up axes. The complete numeric vehicle transform, deskew and general normalized-scan remain unavailable.
  • On the first independently labeled frame, pinned Patchwork++ v1.4.1 achieved 60.37% Ground IoU, 71.59% natural-ground recall and 15.89 ms worker latency versus Current's 49.62%, 49.45% and 3966.53 ms. This is a one-frame diagnostic candidate decision, not production promotion.

Primary references