# LiDAR Dataset Gateway

## Product value

The Dataset Gateway gives Mission Core a repeatable perception laboratory
before the production vehicle and its final sensor installation exist. It
separates four questions that were previously mixed together:

- whether the transport preserved the sensor evidence;
- whether preprocessing produces a valid one-scan perception input;
- whether an algorithm is accurate against independent labels;
- whether several scans form a stable local map for an operator or planner.

This prevents tuning an algorithm until a visually dense vendor map merely
looks plausible. It also keeps work reusable across Gazebo, Unreal, public
datasets and future real onboard sensors.

## Current S0 slice

Implemented now:

- `missioncore.dataset-gateway-catalog/v1`, exposed read-only at
  `GET /api/v1/lidar/dataset-gateway`;
- explicit `native-scan`, `normalized-scan` and `rolling-local-map`
  representations;
- a lossless GOOSE/SemanticKITTI frame reader for little-endian float32 XYZI
  and packed uint32 semantic/instance labels;
- count, finite-value and maximum-point safety gates;
- immutable point-aligned arrays;
- a fail-closed K1 `lio_pcl` boundary;
- worker storage admission for `D:\NDC_MISSIONCORE\datasets` and
  `/mnt/d/NDC_MISSIONCORE/datasets`;
- a visible Dataset Gateway panel in **Данные → Качество LiDAR**.

Not implemented in S0:

- no automatic 3.3 GB validation archive download;
- no implicit coordinate conversion;
- no fake ring/timestamp reconstruction for K1 MQTT evidence;
- no model training or production promotion;
- no rolling-map implementation yet.

## Why public recordings look different

GOOSE stores one VLS-128 revolution per annotated `.bin` file. A rotating
multi-channel sensor produces discrete scan lines, so a single sensor-frame
view looks like sparse rings.

The current field review is explicitly an accumulated map-frame window. It
combines many source publications after pose registration. This fills surfaces
and hides the original scan pattern. The external K1 stream is also already a
post-LIO/modeling product and lacks the raw driver fields needed to reconstruct
an original scan.

Livox sensors additionally use a scan pattern that differs from classic fixed
vertical channels. Time integration therefore changes their visual density in
a different way. “Ring-like” is a sensor geometry property, not a universal
quality target.

## Canonical processing profiles

### P0 — native evidence

Required:

- source ID and immutable frame ID;
- XYZ and the original return/remission/intensity field;
- semantic and instance labels when present;
- calibration/mounting/timing evidence as separate metadata;
- no accumulation and no hidden world transform.

Output: `native-scan`.

### P1 — normalized perception scan

Ordered operations:

1. decode and apply only evidenced factory calibration;
2. assign an explicit sensor coordinate frame;
3. deskew when per-point time and synchronized motion are available;
4. apply bounded range and field-of-view policy;
5. remove the vehicle/self mask;
6. apply named outlier and voxel policies;
7. retain a reversible index/provenance map to the native frame.

Output: `normalized-scan`.

### P2 — inference

Ground, semantic and object providers consume P1. Patchwork++ belongs here. It
does not own P0/P1 or P3.

Output: point-aligned predictions and reproducible metrics against labels.

### P3 — rolling local map

Ordered operations:

1. bind each normalized scan to an evidenced pose;
2. transform to `odom` or a declared local-map frame;
3. deduplicate with a named voxel policy;
4. expire points by TTL or travelled distance;
5. keep dynamic points short-lived or track them separately;
6. publish bounded map state and its contributing frame identities.

Output: `rolling-local-map`.

This is the stage that should stop static geometry from “jumping”. Deskew
reduces within-scan motion distortion; registration stabilizes scans across
time; TTL/dynamic filtering prevents stale ghosts.

## First dataset sequence

1. Configure `MISSIONCORE_DATASET_ROOT=/mnt/d/NDC_MISSIONCORE/datasets` on the
   Windows/WSL worker.
2. Verify free space and record archive size/hash/license.
3. Download only the GOOSE 3D validation archive first (published size 3.3 GB).
4. Import one labeled frame and expose it in React as `native-scan`.
5. Show native remission and ground-truth superclass coloring.
6. Add a declared GOOSE frame/mounting profile and produce `normalized-scan`.
7. Run current ground heuristic and Patchwork++ against independent labels.
8. Add sensor-degradation profiles for range, FOV, density, noise and dropout.
9. Only after the one-frame contract passes, expand to the validation split and
   add a rolling-map sequence with localization evidence.

## Acceptance checklist

- [x] Representations cannot be silently interchanged.
- [x] Large artifacts require operator-admitted D-only storage.
- [x] GOOSE XYZI and labels remain point aligned.
- [x] Invalid length and non-finite frames fail closed.
- [x] K1 mapped increments cannot claim raw-scan fields.
- [x] React exposes the architectural truth before dataset bytes exist.
- [ ] Worker D root configured.
- [ ] GOOSE validation archive hash recorded.
- [ ] First real labeled frame visible in React.
- [ ] Coordinate and mounting profile admitted.
- [ ] Patchwork++ accuracy measured against ground truth.
- [ ] Sensor-degradation matrix qualified.
- [ ] Rolling local map with pose/TTL/dynamic policy qualified.