# LiDAR Dataset Gateway

## Product value

The Dataset Gateway gives Mission Core a repeatable perception laboratory
before the production vehicle and its final sensor installation exist. It
separates four questions that were previously mixed together:

- whether the transport preserved the sensor evidence;
- whether preprocessing produces a valid one-scan perception input;
- whether an algorithm is accurate against independent labels;
- whether several scans form a stable local map for an operator or planner.

This prevents tuning an algorithm until a visually dense vendor map merely
looks plausible. It also keeps work reusable across Gazebo, Unreal, public
datasets and future real onboard sensors.

## Current admitted slice

Implemented now:

- `missioncore.dataset-gateway-catalog/v2`, exposed read-only at
  `GET /api/v1/lidar/dataset-gateway`;
- path-free `missioncore.dataset-admission/v1` worker evidence instead of a
  static React status;
- a bounded `missioncore.dataset-native-scan-preview/v1`, exposed read-only at
  `GET /api/v1/lidar/dataset-gateway/preview`;
- explicit `native-scan`, `normalized-scan` and `rolling-local-map`
  representations;
- a lossless GOOSE/SemanticKITTI frame reader for little-endian float32 XYZI
  and packed uint32 semantic/instance labels;
- count, finite-value and maximum-point safety gates;
- immutable point-aligned arrays;
- a fail-closed K1 `lio_pcl` boundary;
- worker storage admission for `D:\NDC_MISSIONCORE\datasets` and
  `/mnt/d/NDC_MISSIONCORE/datasets`;
- a dedicated, honest dataset catalog in **Полигон → Датасеты**;
- live admission states (`downloading`, `verifying`, `frame-ready`) and a real
  `Открыть` action;
- one admitted GOOSE validation frame with source colors, normalized remission
  and an independent ground-truth view;
- a separate **Парк → Диагностика LiDAR** surface containing only real sensor
  recordings and their operational evidence.

Not implemented:

- no browser-triggered large-artifact download;
- no implicit coordinate conversion;
- no fake ring/timestamp reconstruction for K1 MQTT evidence;
- no model training or production promotion;
- no rolling-map implementation yet.

## Product surface boundary

The gateway is not part of device quality diagnostics.

- **Парк → Диагностика LiDAR** answers whether a selected real sensor recording
  is present, reproducible and operationally usable. It never mixes public
  dataset frames into the selected device evidence.
- **Наблюдение** opens a concrete live or recorded spatial scene.
- **Данные** owns source-of-record, retention, replay preparation and export.
- **Полигон → Датасеты** lists admitted evaluation inputs.
- **Полигон → Прогоны** owns the resulting algorithm comparison, metrics,
  provenance and decision.

Before real dataset bytes exist, the catalog explains the blocked storage gate
and keeps `Открыть` / `Создать прогон` disabled. `Открыть` becomes available
only after the worker manifest says `frame-ready` and the bounded preview passes
its own point-alignment and safety checks. `Создать прогон` remains disabled
until an executable comparison profile exists.

## Why public recordings look different

GOOSE stores one VLS-128 revolution per annotated `.bin` file. A rotating
multi-channel sensor produces discrete scan lines, so a single sensor-frame
view looks like sparse rings.

The admitted `goose_3d_val.zip` uses `lidar/val/.../*_vls128.bin` even though
some GOOSE documentation examples call the point-cloud root `velodyne`. Mission
Core accepts either declared root but still requires an exactly aligned
`labels/val/.../*_goose.label`.

The current field review is explicitly an accumulated map-frame window. It
combines many source publications after pose registration. This fills surfaces
and hides the original scan pattern. The external K1 stream is also already a
post-LIO/modeling product and lacks the raw driver fields needed to reconstruct
an original scan.

Livox sensors additionally use a scan pattern that differs from classic fixed
vertical channels. Time integration therefore changes their visual density in
a different way. “Ring-like” is a sensor geometry property, not a universal
quality target.

## Canonical processing profiles

### P0 — native evidence

Required:

- source ID and immutable frame ID;
- XYZ and the original return/remission/intensity field;
- semantic and instance labels when present;
- calibration/mounting/timing evidence as separate metadata;
- no accumulation and no hidden world transform.

Output: `native-scan`.

### P1 — normalized perception scan

Ordered operations:

1. decode and apply only evidenced factory calibration;
2. assign an explicit sensor coordinate frame;
3. deskew when per-point time and synchronized motion are available;
4. apply bounded range and field-of-view policy;
5. remove the vehicle/self mask;
6. apply named outlier and voxel policies;
7. retain a reversible index/provenance map to the native frame.

Output: `normalized-scan`.

### P2 — inference

Ground, semantic and object providers consume P1. Patchwork++ belongs here. It
does not own P0/P1 or P3.

Output: point-aligned predictions and reproducible metrics against labels.

### P3 — rolling local map

Ordered operations:

1. bind each normalized scan to an evidenced pose;
2. transform to `odom` or a declared local-map frame;
3. deduplicate with a named voxel policy;
4. expire points by TTL or travelled distance;
5. keep dynamic points short-lived or track them separately;
6. publish bounded map state and its contributing frame identities.

Output: `rolling-local-map`.

This is the stage that should stop static geometry from “jumping”. Deskew
reduces within-scan motion distortion; registration stabilizes scans across
time; TTL/dynamic filtering prevents stale ghosts.

## First dataset sequence

1. [Done] Admit `/mnt/d/NDC_MISSIONCORE/datasets` on the Windows/WSL worker.
2. [Done] Verify free space and record archive size/hash/license.
3. [Done] Download only the GOOSE 3D validation archive first (published size
   3.3 GB).
4. [Done] Import one labeled frame and expose it in React as `native-scan`.
5. [Done] Show native remission, the original semantic palette and
   ground-truth superclass coloring.
6. Add a declared GOOSE frame/mounting profile and produce `normalized-scan`.
7. [Current baseline done; Patchwork++ blocked on mounting evidence] Run the
   current ground heuristic and Patchwork++ against independent labels.
8. Add sensor-degradation profiles for range, FOV, density, noise and dropout.
9. Only after the one-frame contract passes, expand to the validation split and
   add a rolling-map sequence with localization evidence.

## Acceptance checklist

- [x] Representations cannot be silently interchanged.
- [x] Large artifacts require operator-admitted D-only storage.
- [x] GOOSE XYZI and labels remain point aligned.
- [x] Invalid length and non-finite frames fail closed.
- [x] K1 mapped increments cannot claim raw-scan fields.
- [x] React exposes an honest catalog/empty state before dataset bytes exist.
- [x] Dataset Gateway is isolated from real-sensor diagnostics and placed under
  Polygon qualification.
- [x] Worker D root configured.
- [x] GOOSE validation archive hash recorded.
- [x] First real labeled frame visible in React.
- [ ] Coordinate and mounting profile admitted.
- [x] Current local-percentile baseline measured against ground truth.
- [ ] Patchwork++ accuracy measured against ground truth.
- [ ] Sensor-degradation matrix qualified.
- [ ] Rolling local map with pose/TTL/dynamic policy qualified.

## First GOOSE admission evidence

- worker root: canonical D-only root, path never exposed by the HTTP API;
- source URL:
  `https://goose-dataset.de/storage/goose_3d_val.zip`;
- observed archive bytes: `3,498,402,435`;
- observed SHA-256:
  `0be9e0f8459bafcbc92ff7c3cc366e9b4e2f6e1e9bdf50e6439e1557e864c26f`;
- archive integrity: ZIP structure, bounded expansion, safe paths, one LICENSE,
  one label mapping and aligned point/label frames;
- license artifact: Creative Commons Attribution-ShareAlike 4.0 International;
- first deterministic frame:
  `2022-07-22_flight__0071_1658494234334310308`;
- source points: `169,883`;
- semantic classes present: `17`;
- independently labeled artificial/natural ground: `45,968` points
  (`27.0586%`);
- browser preview: deterministic even-index sample of `50,000` points,
  visualization-only.

GOOSE does not publish a checksum beside this download. The recorded SHA-256 is
therefore Mission Core's observed content pin after a TLS download, not a claim
of vendor-signed authenticity. Any future byte change creates a new admission
decision rather than silently replacing this installation.

## First independent ground result

The current `missioncore-local-percentile-ground/v1` provider was run against
all `169,883` points, excluding `353` GOOSE void points from scoring:

- Ground IoU: `49.6165%`;
- precision: `73.5892%`;
- recall: `60.3659%`;
- F1: `66.3249%`;
- artificial-ground recall: `95.2364%`;
- natural-ground recall: `49.4516%`;
- obstacle non-ground recall: `79.6145%`;
- provider latency on the worker: `3,858.16 ms`.

This is the first concrete product value from the public dataset. The current
heuristic looks plausible on a dense map and is strong on artificial ground,
but it misses about half of the independently labeled natural ground and is far
from a real-time full-scan provider. The UI exposes `Current` and `Ошибки`
views, so the failure geometry is inspectable instead of being hidden behind
one aggregate score.

The provider now uses a bounded grid-neighborhood index while preserving the
previous exact-radius result. This removes the previous all-points scan for
every occupied cell, but the measured latency still classifies it as a
diagnostic baseline rather than an onboard candidate.

Patchwork++ is intentionally not scored yet. The validation ZIP contains XYZI,
labels, mapping, LICENSE and CHANGELOG but no numeric TF/mounting calibration.
GOOSE documents the VLS-128 as a roof LiDAR and publishes a separate MuCAR-3 TF
tree, but the graph image alone is not physical-height evidence. The next gate
is to admit the numeric transform from `base_link_ground` to
`sensor/lidar/vls128_roof`, declare the source axis convention, and only then
run Patchwork++.