NODEDC_MISSION_CORE/docs/14_LIDAR_DATASET_GATEWAY.md

LiDAR Dataset Gateway

Product value

The Dataset Gateway gives Mission Core a repeatable perception laboratory before the production vehicle and its final sensor installation exist. It separates four questions that were previously mixed together:

• whether the transport preserved the sensor evidence;
• whether preprocessing produces a valid one-scan perception input;
• whether an algorithm is accurate against independent labels;
• whether several scans form a stable local map for an operator or planner.

This prevents tuning an algorithm until a visually dense vendor map merely looks plausible. It also keeps work reusable across Gazebo, Unreal, public datasets and future real onboard sensors.

Current S0 slice

Implemented now:

• missioncore.dataset-gateway-catalog/v1, exposed read-only at GET /api/v1/lidar/dataset-gateway;
• explicit native-scan, normalized-scan and rolling-local-map representations;
• a lossless GOOSE/SemanticKITTI frame reader for little-endian float32 XYZI and packed uint32 semantic/instance labels;
• count, finite-value and maximum-point safety gates;
• immutable point-aligned arrays;
• a fail-closed K1 lio_pcl boundary;
• worker storage admission for D:\NDC_MISSIONCORE\datasets and /mnt/d/NDC_MISSIONCORE/datasets;
• a visible Dataset Gateway panel in Данные → Качество LiDAR.

Not implemented in S0:

• no automatic 3.3 GB validation archive download;
• no implicit coordinate conversion;
• no fake ring/timestamp reconstruction for K1 MQTT evidence;
• no model training or production promotion;
• no rolling-map implementation yet.

Why public recordings look different

GOOSE stores one VLS-128 revolution per annotated .bin file. A rotating multi-channel sensor produces discrete scan lines, so a single sensor-frame view looks like sparse rings.

The current field review is explicitly an accumulated map-frame window. It combines many source publications after pose registration. This fills surfaces and hides the original scan pattern. The external K1 stream is also already a post-LIO/modeling product and lacks the raw driver fields needed to reconstruct an original scan.

Livox sensors additionally use a scan pattern that differs from classic fixed vertical channels. Time integration therefore changes their visual density in a different way. “Ring-like” is a sensor geometry property, not a universal quality target.

Canonical processing profiles

P0 — native evidence

Required:

• source ID and immutable frame ID;
• XYZ and the original return/remission/intensity field;
• semantic and instance labels when present;
• calibration/mounting/timing evidence as separate metadata;
• no accumulation and no hidden world transform.

Output: native-scan.

P1 — normalized perception scan

Ordered operations:

1. decode and apply only evidenced factory calibration;
2. assign an explicit sensor coordinate frame;
3. deskew when per-point time and synchronized motion are available;
4. apply bounded range and field-of-view policy;
5. remove the vehicle/self mask;
6. apply named outlier and voxel policies;
7. retain a reversible index/provenance map to the native frame.

Output: normalized-scan.

P2 — inference

Ground, semantic and object providers consume P1. Patchwork++ belongs here. It does not own P0/P1 or P3.

Output: point-aligned predictions and reproducible metrics against labels.

P3 — rolling local map

Ordered operations:

1. bind each normalized scan to an evidenced pose;
2. transform to odom or a declared local-map frame;
3. deduplicate with a named voxel policy;
4. expire points by TTL or travelled distance;
5. keep dynamic points short-lived or track them separately;
6. publish bounded map state and its contributing frame identities.

Output: rolling-local-map.

This is the stage that should stop static geometry from “jumping”. Deskew reduces within-scan motion distortion; registration stabilizes scans across time; TTL/dynamic filtering prevents stale ghosts.

First dataset sequence

1. Configure MISSIONCORE_DATASET_ROOT=/mnt/d/NDC_MISSIONCORE/datasets on the Windows/WSL worker.
2. Verify free space and record archive size/hash/license.
3. Download only the GOOSE 3D validation archive first (published size 3.3 GB).
4. Import one labeled frame and expose it in React as native-scan.
5. Show native remission and ground-truth superclass coloring.
6. Add a declared GOOSE frame/mounting profile and produce normalized-scan.
7. Run current ground heuristic and Patchwork++ against independent labels.
8. Add sensor-degradation profiles for range, FOV, density, noise and dropout.
9. Only after the one-frame contract passes, expand to the validation split and add a rolling-map sequence with localization evidence.

Acceptance checklist

• Representations cannot be silently interchanged.
• Large artifacts require operator-admitted D-only storage.
• GOOSE XYZI and labels remain point aligned.
• Invalid length and non-finite frames fail closed.
• K1 mapped increments cannot claim raw-scan fields.
• React exposes the architectural truth before dataset bytes exist.
• Worker D root configured.
• GOOSE validation archive hash recorded.
• First real labeled frame visible in React.
• Coordinate and mounting profile admitted.
• Patchwork++ accuracy measured against ground truth.
• Sensor-degradation matrix qualified.
• Rolling local map with pose/TTL/dynamic policy qualified.