# LiDAR Dataset Gateway

## Product value

The Dataset Gateway gives Mission Core a repeatable perception laboratory
before the production vehicle and its final sensor installation exist. It
separates four questions that were previously mixed together:

- whether the transport preserved the sensor evidence;
- whether preprocessing produces a valid one-scan perception input;
- whether an algorithm is accurate against independent labels;
- whether several scans form a stable local map for an operator or planner.

This prevents tuning an algorithm until a visually dense vendor map merely
looks plausible. It also keeps work reusable across Gazebo, Unreal, public
datasets and future real onboard sensors.

## Current admitted slice

Implemented now:

- `missioncore.dataset-gateway-catalog/v2`, exposed read-only at
  `GET /api/v1/lidar/dataset-gateway`;
- path-free `missioncore.dataset-admission/v1` worker evidence instead of a
  static React status;
- a bounded `missioncore.dataset-native-scan-preview/v1`, exposed read-only at
  `GET /api/v1/lidar/dataset-gateway/preview`;
- explicit `native-scan`, `normalized-scan` and `rolling-local-map`
  representations;
- a lossless GOOSE/SemanticKITTI frame reader for little-endian float32 XYZI
  and packed uint32 semantic/instance labels;
- count, finite-value and maximum-point safety gates;
- immutable point-aligned arrays;
- a fail-closed K1 `lio_pcl` boundary;
- worker storage admission for `D:\NDC_MISSIONCORE\datasets` and
  `/mnt/d/NDC_MISSIONCORE/datasets`;
- a dedicated, honest dataset catalog in **Полигон → Датасеты**;
- live admission states (`downloading`, `verifying`, `frame-ready`) and a real
  `Открыть` action;
- one admitted GOOSE validation frame with source colors, normalized remission
  and an independent ground-truth view;
- a published, dimensioned MuCAR-3/VLS-128 Patchwork++ input profile:
  sensor frame `x-forward / y-left / z-up`, physical height `2.24 m`, native
  one-revolution scan and explicit absence of deskew/full vehicle TF claims;
- a reproducible Current/Patchwork++ A/B artifact with point-aligned masks,
  disagreement views, independent-label metrics, latency and exact provider
  identities;
- a separate **Парк → Диагностика LiDAR** surface containing only real sensor
  recordings and their operational evidence.

Not implemented:

- no browser-triggered large-artifact download;
- no implicit coordinate conversion;
- no fake ring/timestamp reconstruction for K1 MQTT evidence;
- no model training or production promotion;
- no rolling-map implementation yet.

## Product surface boundary

The gateway is not part of device quality diagnostics.

- **Парк → Диагностика LiDAR** answers whether a selected real sensor recording
  is present, reproducible and operationally usable. It never mixes public
  dataset frames into the selected device evidence.
- **Наблюдение** opens a concrete live or recorded spatial scene.
- **Данные** owns source-of-record, retention, replay preparation and export.
- **Полигон → Датасеты** lists admitted evaluation inputs.
- **Полигон → Прогоны** owns the resulting algorithm comparison, metrics,
  provenance and decision.

Before real dataset bytes exist, the catalog explains the blocked storage gate
and keeps `Открыть` / `Создать прогон` disabled. `Открыть` becomes available
only after the worker manifest says `frame-ready` and the bounded preview passes
its own point-alignment and safety checks. `Создать прогон` remains disabled
until an executable comparison profile exists.

## Why public recordings look different

GOOSE stores one VLS-128 revolution per annotated `.bin` file. A rotating
multi-channel sensor produces discrete scan lines, so a single sensor-frame
view looks like sparse rings.

The admitted `goose_3d_val.zip` uses `lidar/val/.../*_vls128.bin` even though
some GOOSE documentation examples call the point-cloud root `velodyne`. Mission
Core accepts either declared root but still requires an exactly aligned
`labels/val/.../*_goose.label`.

The current field review is explicitly an accumulated map-frame window. It
combines many source publications after pose registration. This fills surfaces
and hides the original scan pattern. The external K1 stream is also already a
post-LIO/modeling product and lacks the raw driver fields needed to reconstruct
an original scan.

Livox sensors additionally use a scan pattern that differs from classic fixed
vertical channels. Time integration therefore changes their visual density in
a different way. “Ring-like” is a sensor geometry property, not a universal
quality target.

## Canonical processing profiles

### P0 — native evidence

Required:

- source ID and immutable frame ID;
- XYZ and the original return/remission/intensity field;
- semantic and instance labels when present;
- calibration/mounting/timing evidence as separate metadata;
- no accumulation and no hidden world transform.

Output: `native-scan`.

### P1 — normalized perception scan

Ordered operations:

1. decode and apply only evidenced factory calibration;
2. assign an explicit sensor coordinate frame;
3. deskew when per-point time and synchronized motion are available;
4. apply bounded range and field-of-view policy;
5. remove the vehicle/self mask;
6. apply named outlier and voxel policies;
7. retain a reversible index/provenance map to the native frame.

Output: `normalized-scan`.

### P2 — inference

Ground, semantic and object providers consume P1. Patchwork++ belongs here. It
does not own P0/P1 or P3.

Output: point-aligned predictions and reproducible metrics against labels.

### P3 — rolling local map

Ordered operations:

1. bind each normalized scan to an evidenced pose;
2. transform to `odom` or a declared local-map frame;
3. deduplicate with a named voxel policy;
4. expire points by TTL or travelled distance;
5. keep dynamic points short-lived or track them separately;
6. publish bounded map state and its contributing frame identities.

Output: `rolling-local-map`.

This is the stage that should stop static geometry from “jumping”. Deskew
reduces within-scan motion distortion; registration stabilizes scans across
time; TTL/dynamic filtering prevents stale ghosts.

## First dataset sequence

1. [Done] Admit `/mnt/d/NDC_MISSIONCORE/datasets` on the Windows/WSL worker.
2. [Done] Verify free space and record archive size/hash/license.
3. [Done] Download only the GOOSE 3D validation archive first (published size
   3.3 GB).
4. [Done] Import one labeled frame and expose it in React as `native-scan`.
5. [Done] Show native remission, the original semantic palette and
   ground-truth superclass coloring.
6. [Patchwork-specific profile done; general normalization still pending] Admit
   the published VLS-128 axes and physical height without claiming a complete
   numeric vehicle TF or deskewed `normalized-scan`.
7. [One-frame A/B done] Run the current ground heuristic and pinned
   Patchwork++ against the same native scan and independent labels.
8. Add sensor-degradation profiles for range, FOV, density, noise and dropout.
9. Expand the A/B to the validation split; only after that qualification gate,
   add a rolling-map sequence with localization evidence.

## Acceptance checklist

- [x] Representations cannot be silently interchanged.
- [x] Large artifacts require operator-admitted D-only storage.
- [x] GOOSE XYZI and labels remain point aligned.
- [x] Invalid length and non-finite frames fail closed.
- [x] K1 mapped increments cannot claim raw-scan fields.
- [x] React exposes an honest catalog/empty state before dataset bytes exist.
- [x] Dataset Gateway is isolated from real-sensor diagnostics and placed under
  Polygon qualification.
- [x] Worker D root configured.
- [x] GOOSE validation archive hash recorded.
- [x] First real labeled frame visible in React.
- [x] Patchwork-specific axes and physical-height profile admitted.
- [ ] Complete vehicle transform and general `normalized-scan` admitted.
- [x] Current local-percentile baseline measured against ground truth.
- [x] Patchwork++ one-frame accuracy measured against ground truth.
- [ ] Current/Patchwork++ validation-split gate qualified.
- [ ] Sensor-degradation matrix qualified.
- [ ] Rolling local map with pose/TTL/dynamic policy qualified.

## First GOOSE admission evidence

- worker root: canonical D-only root, path never exposed by the HTTP API;
- source URL:
  `https://goose-dataset.de/storage/goose_3d_val.zip`;
- observed archive bytes: `3,498,402,435`;
- observed SHA-256:
  `0be9e0f8459bafcbc92ff7c3cc366e9b4e2f6e1e9bdf50e6439e1557e864c26f`;
- archive integrity: ZIP structure, bounded expansion, safe paths, one LICENSE,
  one label mapping and aligned point/label frames;
- license artifact: Creative Commons Attribution-ShareAlike 4.0 International;
- first deterministic frame:
  `2022-07-22_flight__0071_1658494234334310308`;
- source points: `169,883`;
- semantic classes present: `17`;
- independently labeled artificial/natural ground: `45,968` points
  (`27.0586%`);
- browser preview: deterministic even-index sample of `50,000` points,
  visualization-only.

GOOSE does not publish a checksum beside this download. The recorded SHA-256 is
therefore Mission Core's observed content pin after a TLS download, not a claim
of vendor-signed authenticity. Any future byte change creates a new admission
decision rather than silently replacing this installation.

## First independent ground result

The current `missioncore-local-percentile-ground/v1` provider was run against
all `169,883` points, excluding `353` GOOSE void points from scoring:

- Ground IoU: `49.6165%`;
- precision: `73.5892%`;
- recall: `60.3659%`;
- F1: `66.3249%`;
- artificial-ground recall: `95.2364%`;
- natural-ground recall: `49.4516%`;
- obstacle non-ground recall: `79.6145%`;
- provider latency on the worker: `3,858.16 ms`.

This is the first concrete product value from the public dataset. The current
heuristic looks plausible on a dense map and is strong on artificial ground,
but it misses about half of the independently labeled natural ground and is far
from a real-time full-scan provider. The UI exposes `Current` and `Ошибки`
views, so the failure geometry is inspectable instead of being hidden behind
one aggregate score.

The provider now uses a bounded grid-neighborhood index while preserving the
previous exact-radius result. This removes the previous all-points scan for
every occupied cell, but the measured latency still classifies it as a
diagnostic baseline rather than an onboard candidate.

## First Patchwork++ A/B result

GOOSE's dimensioned MuCAR-3 schematic provides the two vertical dimensions
needed by this algorithm-specific gate: `base_link` is `0.64 m` above ground
and the VLS-128 optical center is `1.60 m` above `base_link`. The admitted
Patchwork++ height is therefore `2.24 m`. The same published schematic declares
the sensor axes as `x` forward, `y` left and `z` up. An independent fit to
GOOSE-labeled near-field ground observed a `-2.18 .. -2.14 m` intercept; this
was a non-calibrating cross-check, not the source of the height.

This narrowly admits the input required by Patchwork++ on the native
sensor-centric scan. It does **not** claim a complete numeric vehicle transform,
per-point timing, deskew or a general `normalized-scan`.

Official Patchwork++ `v1.4.1`, source commit
`3e6903a1d5537a4cc2ace897b0bbb98a92d6014c`, was run against the same first
frame and independent labels as Current:

| Metric | Current | Patchwork++ |
| --- | ---: | ---: |
| Ground IoU | `49.6165%` | `60.3702%` |
| Precision | `73.5892%` | `72.6612%` |
| Recall | `60.3659%` | `78.1130%` |
| F1 | `66.3249%` | `75.2886%` |
| Artificial-ground recall | `95.2364%` | `98.9688%` |
| Natural-ground recall | `49.4516%` | `71.5853%` |
| Obstacle non-ground recall | `79.6145%` | `92.1729%` |
| Worker latency | `3966.53 ms` | `15.89 ms` |

Reproducibility pins:

- benchmark identity:
  `2a6d05f54a9e2ac727d9c850c1133f2fb539bfeddbd97c07cfd198ccb239162c`;
- full point-aligned prediction SHA-256:
  `190f455c6e47911921b7f6454913e1bb2d0b8b5814ee2e34d7b63295afddc7ef`;
- bounded browser preview SHA-256:
  `e23ca573103faf931523415e6c263248737eb6600f482b63de685b725bbc28c3`;
- Patchwork++ binary SHA-256:
  `be8038b2098c83fe53841aa8ae19e362910e9056fe0ee7b9304c1ee5c5941094`.

Patchwork++ wins this frame by `10.75` percentage points of Ground IoU, raises
natural-ground recall by `22.13` points and is roughly `250x` faster in this
run. The result is deliberately `one-frame-diagnostic`: it makes Patchwork++
the candidate for validation-split and degradation qualification, not an
accepted navigation or safety provider.

Primary source evidence:

- [GOOSE MuCAR-3 sensor setup](https://goose-dataset.de/docs/mucar3/);
- [GOOSE paper, Figure 3](https://arxiv.org/pdf/2310.16788);
- [Patchwork++ v1.4.1](https://github.com/url-kaist/patchwork-plusplus/tree/v1.4.1).