# LiDAR Dataset Gateway

## Product value

The Dataset Gateway gives Mission Core a repeatable perception laboratory
before the production vehicle and its final sensor installation exist. It
separates four questions that were previously mixed together:

- whether the transport preserved the sensor evidence;
- whether preprocessing produces a valid one-scan perception input;
- whether an algorithm is accurate against independent labels;
- whether several scans form a stable local map for an operator or planner.

This prevents tuning an algorithm until a visually dense vendor map merely
looks plausible. It also keeps work reusable across Gazebo, Unreal, public
datasets and future real onboard sensors.

## Current admitted slice

Implemented now:

- `missioncore.dataset-gateway-catalog/v2`, exposed read-only at
  `GET /api/v1/lidar/dataset-gateway`;
- path-free `missioncore.dataset-admission/v1` worker evidence instead of a
  static React status;
- a bounded `missioncore.dataset-native-scan-preview/v1`, exposed read-only at
  `GET /api/v1/lidar/dataset-gateway/preview`;
- explicit `native-scan`, `normalized-scan` and `rolling-local-map`
  representations;
- a lossless GOOSE/SemanticKITTI frame reader for little-endian float32 XYZI
  and packed uint32 semantic/instance labels;
- count, finite-value and maximum-point safety gates;
- immutable point-aligned arrays;
- a fail-closed K1 `lio_pcl` boundary;
- worker storage admission for `D:\NDC_MISSIONCORE\datasets` and
  `/mnt/d/NDC_MISSIONCORE/datasets`;
- a dedicated, honest dataset catalog in **Полигон → Датасеты**;
- live admission states (`downloading`, `verifying`, `frame-ready`) and a real
  `Открыть` action;
- one admitted GOOSE validation frame with source colors, normalized remission
  and an independent ground-truth view;
- a published, dimensioned MuCAR-3/VLS-128 Patchwork++ input profile:
  sensor frame `x-forward / y-left / z-up`, physical height `2.24 m`, native
  one-revolution scan and explicit absence of deskew/full vehicle TF claims;
- a reproducible Current/Patchwork++ A/B artifact with point-aligned masks,
  disagreement views, independent-label metrics, latency and exact provider
  identities;
- a separate **Парк → Диагностика LiDAR** surface containing only real sensor
  recordings and their operational evidence.

Not implemented:

- no browser-triggered large-artifact download;
- no implicit coordinate conversion;
- no fake ring/timestamp reconstruction for K1 MQTT evidence;
- no model training or production promotion;
- no rolling-map implementation yet.

## Product surface boundary

The gateway is not part of device quality diagnostics.

- **Парк → Диагностика LiDAR** answers whether a selected real sensor recording
  is present, reproducible and operationally usable. It never mixes public
  dataset frames into the selected device evidence.
- **Наблюдение** opens a concrete live or recorded spatial scene.
- **Данные** owns source-of-record, retention, replay preparation and export.
- **Полигон → Датасеты** lists admitted evaluation inputs and owns source
  inspection: source sequences, sparse annotated scans and optional overlays
  from a selected qualification run.
- **Полигон → Прогоны** owns the resulting algorithm comparison, metrics,
  provenance and decision. It does not present the dataset itself as a
  simulation or continuous vehicle recording.

Before real dataset bytes exist, the catalog explains the blocked storage gate
and keeps `Открыть` / `Создать прогон` disabled. `Открыть` becomes available
only after the worker manifest says `frame-ready` and the bounded preview passes
its own point-alignment and safety checks. `Создать прогон` remains disabled
until an executable comparison profile exists.

## Why public recordings look different

GOOSE stores one VLS-128 revolution per annotated `.bin` file. A rotating
multi-channel sensor produces discrete scan lines, so a single sensor-frame
view looks like sparse rings.

The admitted `goose_3d_val.zip` uses `lidar/val/.../*_vls128.bin` even though
some GOOSE documentation examples call the point-cloud root `velodyne`. Mission
Core accepts either declared root but still requires an exactly aligned
`labels/val/.../*_goose.label`.

The current field review is explicitly an accumulated map-frame window. It
combines many source publications after pose registration. This fills surfaces
and hides the original scan pattern. The external K1 stream is also already a
post-LIO/modeling product and lacks the raw driver fields needed to reconstruct
an original scan.

Livox sensors additionally use a scan pattern that differs from classic fixed
vertical channels. Time integration therefore changes their visual density in
a different way. “Ring-like” is a sensor geometry property, not a universal
quality target.

## Canonical processing profiles

### P0 — native evidence

Required:

- source ID and immutable frame ID;
- XYZ and the original return/remission/intensity field;
- semantic and instance labels when present;
- calibration/mounting/timing evidence as separate metadata;
- no accumulation and no hidden world transform.

Output: `native-scan`.

### P1 — normalized perception scan

Ordered operations:

1. decode and apply only evidenced factory calibration;
2. assign an explicit sensor coordinate frame;
3. deskew when per-point time and synchronized motion are available;
4. apply bounded range and field-of-view policy;
5. remove the vehicle/self mask;
6. apply named outlier and voxel policies;
7. retain a reversible index/provenance map to the native frame.

Output: `normalized-scan`.

### P2 — inference

Ground, semantic and object providers consume P1. Patchwork++ belongs here. It
does not own P0/P1 or P3.

Output: point-aligned predictions and reproducible metrics against labels.

### P3 — rolling local map

Ordered operations:

1. bind each normalized scan to an evidenced pose;
2. transform to `odom` or a declared local-map frame;
3. deduplicate with a named voxel policy;
4. expire points by TTL or travelled distance;
5. keep dynamic points short-lived or track them separately;
6. publish bounded map state and its contributing frame identities.

Output: `rolling-local-map`.

This is the stage that should stop static geometry from “jumping”. Deskew
reduces within-scan motion distortion; registration stabilizes scans across
time; TTL/dynamic filtering prevents stale ghosts.

## First dataset sequence

1. [Done] Admit `/mnt/d/NDC_MISSIONCORE/datasets` on the Windows/WSL worker.
2. [Done] Verify free space and record archive size/hash/license.
3. [Done] Download only the GOOSE 3D validation archive first (published size
   3.3 GB).
4. [Done] Import one labeled frame and expose it in React as `native-scan`.
5. [Done] Show native remission, the original semantic palette and
   ground-truth superclass coloring.
6. [Patchwork-specific profile done; general normalization still pending] Admit
   the published VLS-128 axes and physical height without claiming a complete
   numeric vehicle TF or deskewed `normalized-scan`.
7. [One-frame A/B done] Run the current ground heuristic and pinned
   Patchwork++ against the same native scan and independent labels.
8. [Done] Add deterministic sensor-degradation profiles for range, FOV,
   density, noise and dropout without modifying native source frames.
9. [Done] Expand the A/B to all 961 validation frames and seal the result as a
   Polygon replay-shadow run. Add a rolling-map sequence with localization
   evidence only after this qualification boundary.

## Acceptance checklist

- [x] Representations cannot be silently interchanged.
- [x] Large artifacts require operator-admitted D-only storage.
- [x] GOOSE XYZI and labels remain point aligned.
- [x] Invalid length and non-finite frames fail closed.
- [x] K1 mapped increments cannot claim raw-scan fields.
- [x] React exposes an honest catalog/empty state before dataset bytes exist.
- [x] Dataset Gateway is isolated from real-sensor diagnostics and placed under
  Polygon qualification.
- [x] Worker D root configured.
- [x] GOOSE validation archive hash recorded.
- [x] First real labeled frame visible in React.
- [x] Patchwork-specific axes and physical-height profile admitted.
- [ ] Complete vehicle transform and general `normalized-scan` admitted.
- [x] Current local-percentile baseline measured against ground truth.
- [x] Patchwork++ one-frame accuracy measured against ground truth.
- [x] Current/Patchwork++ validation-split gate qualified.
- [x] Sensor-degradation matrix qualified.
- [ ] Rolling local map with pose/TTL/dynamic policy qualified.

## First GOOSE admission evidence

- worker root: canonical D-only root, path never exposed by the HTTP API;
- source URL:
  `https://goose-dataset.de/storage/goose_3d_val.zip`;
- observed archive bytes: `3,498,402,435`;
- observed SHA-256:
  `0be9e0f8459bafcbc92ff7c3cc366e9b4e2f6e1e9bdf50e6439e1557e864c26f`;
- archive integrity: ZIP structure, bounded expansion, safe paths, one LICENSE,
  one label mapping and aligned point/label frames;
- license artifact: Creative Commons Attribution-ShareAlike 4.0 International;
- first deterministic frame:
  `2022-07-22_flight__0071_1658494234334310308`;
- source points: `169,883`;
- semantic classes present: `17`;
- independently labeled artificial/natural ground: `45,968` points
  (`27.0586%`);
- browser preview: deterministic even-index sample of `50,000` points,
  visualization-only.

GOOSE does not publish a checksum beside this download. The recorded SHA-256 is
therefore Mission Core's observed content pin after a TLS download, not a claim
of vendor-signed authenticity. Any future byte change creates a new admission
decision rather than silently replacing this installation.

## First independent ground result

The current `missioncore-local-percentile-ground/v1` provider was run against
all `169,883` points, excluding `353` GOOSE void points from scoring:

- Ground IoU: `49.6165%`;
- precision: `73.5892%`;
- recall: `60.3659%`;
- F1: `66.3249%`;
- artificial-ground recall: `95.2364%`;
- natural-ground recall: `49.4516%`;
- obstacle non-ground recall: `79.6145%`;
- provider latency on the worker: `3,858.16 ms`.

This is the first concrete product value from the public dataset. The current
heuristic looks plausible on a dense map and is strong on artificial ground,
but it misses about half of the independently labeled natural ground and is far
from a real-time full-scan provider. The UI exposes `Current` and `Ошибки`
views, so the failure geometry is inspectable instead of being hidden behind
one aggregate score.

The provider now uses a bounded grid-neighborhood index while preserving the
previous exact-radius result. This removes the previous all-points scan for
every occupied cell, but the measured latency still classifies it as a
diagnostic baseline rather than an onboard candidate.

## First Patchwork++ A/B result

GOOSE's dimensioned MuCAR-3 schematic provides the two vertical dimensions
needed by this algorithm-specific gate: `base_link` is `0.64 m` above ground
and the VLS-128 optical center is `1.60 m` above `base_link`. The admitted
Patchwork++ height is therefore `2.24 m`. The same published schematic declares
the sensor axes as `x` forward, `y` left and `z` up. An independent fit to
GOOSE-labeled near-field ground observed a `-2.18 .. -2.14 m` intercept; this
was a non-calibrating cross-check, not the source of the height.

This narrowly admits the input required by Patchwork++ on the native
sensor-centric scan. It does **not** claim a complete numeric vehicle transform,
per-point timing, deskew or a general `normalized-scan`.

Official Patchwork++ `v1.4.1`, source commit
`3e6903a1d5537a4cc2ace897b0bbb98a92d6014c`, was run against the same first
frame and independent labels as Current:

| Metric | Current | Patchwork++ |
| --- | ---: | ---: |
| Ground IoU | `49.6165%` | `60.3702%` |
| Precision | `73.5892%` | `72.6612%` |
| Recall | `60.3659%` | `78.1130%` |
| F1 | `66.3249%` | `75.2886%` |
| Artificial-ground recall | `95.2364%` | `98.9688%` |
| Natural-ground recall | `49.4516%` | `71.5853%` |
| Obstacle non-ground recall | `79.6145%` | `92.1729%` |
| Worker latency | `3966.53 ms` | `15.89 ms` |

Reproducibility pins:

- benchmark identity:
  `2a6d05f54a9e2ac727d9c850c1133f2fb539bfeddbd97c07cfd198ccb239162c`;
- full point-aligned prediction SHA-256:
  `190f455c6e47911921b7f6454913e1bb2d0b8b5814ee2e34d7b63295afddc7ef`;
- bounded browser preview SHA-256:
  `e23ca573103faf931523415e6c263248737eb6600f482b63de685b725bbc28c3`;
- Patchwork++ binary SHA-256:
  `be8038b2098c83fe53841aa8ae19e362910e9056fe0ee7b9304c1ee5c5941094`.

Patchwork++ wins this frame by `10.75` percentage points of Ground IoU, raises
natural-ground recall by `22.13` points and is roughly `250x` faster in this
run. The result is deliberately `one-frame-diagnostic`: it makes Patchwork++
the candidate for validation-split and degradation qualification, not an
accepted navigation or safety provider.

## Validation-split qualification

The immutable R1 replay-shadow run
`goose-ground-035de3ae6155844b636d` processed all `961` aligned validation
frames directly from the D-only archive. It is bound to Mission Core commit
`d6decbe05ccecd32d6a745767ee7a795bd1c88b1`, qualification identity
`035de3ae6155844b636d89d8f23b36e561edab105720fa850094a5c6f680c0ef`
and the same pinned Patchwork++ source/binary as the one-frame diagnostic.

| Validation micro metric | Current | Patchwork++ |
| --- | ---: | ---: |
| Ground IoU | `47.4199%` | `66.3974%` |
| Natural-ground recall | `51.7650%` | `76.2215%` |
| Obstacle non-ground recall | `93.5831%` | `94.6666%` |
| Assigned fraction | `100%` | `100%` |
| Latency p50 | `3527.02 ms` | `16.04 ms` |
| Latency p95 | `6115.18 ms` | `23.40 ms` |
| Maximum latency | `11162.62 ms` | `34.12 ms` |

The predeclared nominal gates passed: Ground IoU gain is `18.98` percentage
points (required `>= 7`), natural-ground recall gain is `24.46` points
(required `>= 10`), Patchwork++ obstacle non-ground recall is `94.67%`
(required `>= 90%`), assigned fraction is `100%` (required `>= 99.9%`) and
p95 latency is `23.40 ms` (required `<= 50 ms`). No frame exceeded the
catastrophic regression boundary of `-20` Ground-IoU points; the worst observed
delta was `-13.51` points.

| Deterministic profile | Source coverage | Ground IoU | Natural recall | p95 latency |
| --- | ---: | ---: | ---: | ---: |
| `range-20m` | `52.70%` | `63.1304%` | `72.1331%` | `16.29 ms` |
| `range-40m` | `80.03%` | `68.5958%` | `80.0635%` | `20.74 ms` |
| `density-50` | `50.00%` | `66.3575%` | `76.8925%` | `11.78 ms` |
| `density-25` | `25.00%` | `65.5975%` | `76.8987%` | `5.99 ms` |
| `noise-05m` | `100%` | `63.4378%` | `70.7234%` | `23.28 ms` |
| `dropout-30` | `70.00%` | `66.4658%` | `76.6721%` | `16.24 ms` |
| `front-180` | `51.48%` | `67.6924%` | `77.8012%` | `11.82 ms` |

Every degradation profile stayed inside the predeclared `15`-point Ground-IoU
loss, `20`-point natural-recall loss and `50 ms` p95 limits. The sealed
decision is therefore `shadow-candidate`, explicitly
`promoted_to_navigation_or_safety = false`.

The first aggregate-only Polygon screen was rejected as an operator product:
five selected failure previews could not let an operator verify the decision.
Commit `b037076` adds a separate immutable visual derivative bound to the
qualification report:

- schema `missioncore.goose-ground-review-pack/v1`;
- identity
  `6ddc2de0154196088049b8654d9187fba0c52e28783e5ada6be5266dd16bd641`;
- all `961/961` validation frames in source order;
- deterministic `12,000`-point frame previews with remission, public
  point-aligned ground truth, Current and Patchwork++ masks;
- compact size `67 MB`, versus multiple gigabytes for equivalent JSON;
- direct modes for geometry, ground truth, both providers and both error maps.

The subsequent product review found a second semantic error: the `961` frames
were presented as one playable recording. They are actually `8` independent
GOOSE validation sequences:

- `2022-07-22_flight` — `151` annotated scans;
- `2022-08-30_siegertsbrunn_feldwege` — `103`;
- `2022-09-21_garching_uebungsplatz_2` — `123`;
- `2022-12-07_aying_hills` — `133`;
- `2023-01-20_aying_mangfall_2` — `191`;
- `2023-03-03_garching_2` — `106`;
- `2023-05-15_neubiberg_rain` — `73`;
- `2023-05-17_neubiberg_sunny` — `81`.

Each admitted `.bin` is one annotated VLS-128 revolution. Within those eight
sequences the validation scans are sparse: observed adjacent timestamps range
from about `0.1 s` to `167.5 s`. The annotated ZIP contains neither a
continuous playback stream nor a pose chain that could support visually
continuous ego-motion. Mission Core therefore:

- never plays across a sequence boundary;
- labels Play as an accelerated scan review, not real-time playback;
- exposes the exact source frame number and timestamp gap;
- keeps a fixed sensor-centric origin and preserves the operator camera within
  one sequence instead of refitting every frame;
- does not interpolate missing scans or invent a route.

Continuous vehicle motion, odometry and synchronized sensor playback require
the separately admitted raw GOOSE ROS bags plus localization. Those assets are
not part of the current `3.3 GB` annotated validation archive.

The source archive and full-resolution crash-resume cache remain only on worker
D. A bounded review mirror may be served by a Mission Core backend, but SSH is
not a product data plane. The replay UI does not render the unrelated live
Ackermann rover and does not depend on Simulation Worker online status.

Primary source evidence:

- [GOOSE MuCAR-3 sensor setup](https://goose-dataset.de/docs/mucar3/);
- [GOOSE paper, Figure 3](https://arxiv.org/pdf/2310.16788);
- [Patchwork++ v1.4.1](https://github.com/url-kaist/patchwork-plusplus/tree/v1.4.1).