# LiDAR Dataset Gateway

## Product value

The Dataset Gateway gives Mission Core a repeatable perception laboratory
before the production vehicle and its final sensor installation exist. It
separates four questions that were previously mixed together:

- whether the transport preserved the sensor evidence;
- whether preprocessing produces a valid one-scan perception input;
- whether an algorithm is accurate against independent labels;
- whether several scans form a stable local map for an operator or planner.

This prevents tuning an algorithm until a visually dense vendor map merely
looks plausible. It also keeps work reusable across Gazebo, Unreal, public
datasets and future real onboard sensors.

`Dataset` names an immutable experiment input, not one universal point-cloud
representation. Mission Core keeps two evidence classes:

- public labeled native-scan datasets such as GOOSE and RELLIS qualify
  algorithms against independent answers;
- private real-sensor datasets such as `RAVNOVES00` replay the exact K1
  `lio_pcl`/`lio_pose` representation received during a physical session.

`RAVNOVES00` is therefore a dataset for K1 local-world-model experiments even
though it must never be relabeled as a native scan. Jobs bind its existing
session/replay identity and create replaceable derivatives; they do not copy,
rewrite or mix public points into the K1 source.

## Current admitted slice

Implemented now:

- multi-source `missioncore.dataset-gateway-catalog/v3`, exposed read-only at
  `GET /api/v1/lidar/dataset-gateway`;
- path-free `missioncore.dataset-admission/v1` worker evidence instead of a
  static React status;
- bounded `missioncore.dataset-native-scan-preview/v1` (GOOSE) and `/v2`
  (RELLIS) artifacts, selected by source at
  `GET /api/v1/lidar/dataset-gateway/preview?source_id=...`;
- explicit `native-scan`, `normalized-scan` and `rolling-local-map`
  representations;
- a lossless GOOSE/SemanticKITTI frame reader for little-endian float32 XYZI
  and packed uint32 semantic/instance labels;
- count, finite-value and maximum-point safety gates;
- immutable point-aligned arrays;
- a fail-closed K1 `lio_pcl` boundary;
- worker storage admission for `D:\NDC_MISSIONCORE\datasets` and
  `/mnt/d/NDC_MISSIONCORE/datasets`;
- a dedicated, honest dataset catalog in **Полигон → Датасеты**;
- live admission states (`downloading`, `verifying`, `frame-ready`) and a real
  `Открыть` action;
- one admitted GOOSE validation frame with source colors, normalized remission
  and an independent ground-truth view;
- a fail-closed admission of the complete official RELLIS-3D v1.1 release:
  `13,556` aligned Ouster scans and labels, all five sequences, official
  train/validation/test splits and complete LiDAR pose coverage;
- a full RELLIS validation viewer with its two real source sequences,
  `2,413` source scans, ground truth, Current, Patchwork++ and error modes;
- a train-only, label-backed Ouster height calibration and a sealed R1
  Current/Patchwork++ qualification over every RELLIS validation frame;
- a published, dimensioned MuCAR-3/VLS-128 Patchwork++ input profile:
  sensor frame `x-forward / y-left / z-up`, physical height `2.24 m`, native
  one-revolution scan and explicit absence of deskew/full vehicle TF claims;
- a reproducible Current/Patchwork++ A/B artifact with point-aligned masks,
  disagreement views, independent-label metrics, latency and exact provider
  identities;
- a separate **Парк → Диагностика LiDAR** surface containing only real sensor
  recordings and their operational evidence.

Not implemented:

- no browser-triggered large-artifact download;
- no implicit coordinate conversion;
- no fake ring/timestamp reconstruction for K1 MQTT evidence;
- no model training or production promotion;
- no RELLIS ROS bag admission, continuous synchronized playback or production
  promotion;
- no rolling-map implementation yet.

## Product surface boundary

The gateway is not part of device quality diagnostics.

- **Парк → Диагностика LiDAR** answers whether a selected real sensor recording
  is present, reproducible and operationally usable. It never mixes public
  dataset frames into the selected device evidence.
- **Наблюдение** opens a concrete live or recorded spatial scene.
- **Данные** owns source-of-record, retention, replay preparation and export.
- **Полигон → Датасеты** is the single operator surface for admitted evaluation
  inputs. Opening one dataset exposes its source fragments, sparse annotated
  scans, visual overlays and useful analysis calculated only for the selected
  fragment.
- Internal replay-run identity, provenance and immutable artifacts remain a
  backend evidence contract. They are not a second operator navigation item
  and are not presented as vehicle motion.

Before compatible source evidence exists, the catalog explains the blocked
storage gate and keeps `Открыть датасет` disabled. GOOSE becomes available only
after the worker manifest says `frame-ready`; RELLIS becomes `dataset-ready`
only after all four pinned archives, every split pair and pose coverage pass
admission. Both bounded previews must pass their own identity, point-alignment
and safety checks.

## Why public recordings look different

GOOSE stores one VLS-128 revolution per annotated `.bin` file. A rotating
multi-channel sensor produces discrete scan lines, so a single sensor-frame
view looks like sparse rings.

The admitted `goose_3d_val.zip` uses `lidar/val/.../*_vls128.bin` even though
some GOOSE documentation examples call the point-cloud root `velodyne`. Mission
Core accepts either declared root but still requires an exactly aligned
`labels/val/.../*_goose.label`.

The current field review is explicitly an accumulated map-frame window. It
combines many source publications after pose registration. This fills surfaces
and hides the original scan pattern. The external K1 stream is also already a
post-LIO/modeling product and lacks the raw driver fields needed to reconstruct
an original scan.

Livox sensors additionally use a scan pattern that differs from classic fixed
vertical channels. Time integration therefore changes their visual density in
a different way. “Ring-like” is a sensor geometry property, not a universal
quality target.

## Canonical processing profiles

### P0 — native evidence

Required:

- source ID and immutable frame ID;
- XYZ and the original return/remission/intensity field;
- semantic and instance labels when present;
- calibration/mounting/timing evidence as separate metadata;
- no accumulation and no hidden world transform.

Output: `native-scan`.

### P1 — normalized perception scan

Ordered operations:

1. decode and apply only evidenced factory calibration;
2. assign an explicit sensor coordinate frame;
3. deskew when per-point time and synchronized motion are available;
4. apply bounded range and field-of-view policy;
5. remove the vehicle/self mask;
6. apply named outlier and voxel policies;
7. retain a reversible index/provenance map to the native frame.

Output: `normalized-scan`.

### P2 — inference

Ground, semantic and object providers consume P1. Patchwork++ belongs here. It
does not own P0/P1 or P3.

Output: point-aligned predictions and reproducible metrics against labels.

### P3 — rolling local map

Ordered operations:

1. bind each normalized scan to an evidenced pose;
2. transform to `odom` or a declared local-map frame;
3. deduplicate with a named voxel policy;
4. expire points by TTL or travelled distance;
5. keep dynamic points short-lived or track them separately;
6. publish bounded map state and its contributing frame identities.

Output: `rolling-local-map`.

This is the stage that should stop static geometry from “jumping”. Deskew
reduces within-scan motion distortion; registration stabilizes scans across
time; TTL/dynamic filtering prevents stale ghosts.

## First dataset sequence

1. [Done] Admit `/mnt/d/NDC_MISSIONCORE/datasets` on the Windows/WSL worker.
2. [Done] Verify free space and record archive size/hash/license.
3. [Done] Download only the GOOSE 3D validation archive first (published size
   3.3 GB).
4. [Done] Import one labeled frame and expose it in React as `native-scan`.
5. [Done] Show native remission, the original semantic palette and
   ground-truth superclass coloring.
6. [Patchwork-specific profile done; general normalization still pending] Admit
   the published VLS-128 axes and physical height without claiming a complete
   numeric vehicle TF or deskewed `normalized-scan`.
7. [One-frame A/B done] Run the current ground heuristic and pinned
   Patchwork++ against the same native scan and independent labels.
8. [Done] Add deterministic sensor-degradation profiles for range, FOV,
   density, noise and dropout without modifying native source frames.
9. [Done] Expand the A/B to all 961 validation frames and seal the result as a
   Polygon replay-shadow run. Add a rolling-map sequence with localization
   evidence only after this qualification boundary.

## Acceptance checklist

- [x] Representations cannot be silently interchanged.
- [x] Large artifacts require operator-admitted D-only storage.
- [x] GOOSE XYZI and labels remain point aligned.
- [x] Invalid length and non-finite frames fail closed.
- [x] K1 mapped increments cannot claim raw-scan fields.
- [x] React exposes an honest catalog/empty state before dataset bytes exist.
- [x] Dataset Gateway is isolated from real-sensor diagnostics and placed under
  Polygon qualification.
- [x] Worker D root configured.
- [x] GOOSE validation archive hash recorded.
- [x] First real labeled frame visible in React.
- [x] Patchwork-specific axes and physical-height profile admitted.
- [ ] Complete vehicle transform and general `normalized-scan` admitted.
- [x] Current local-percentile baseline measured against ground truth.
- [x] Patchwork++ one-frame accuracy measured against ground truth.
- [x] Current/Patchwork++ validation-split gate qualified.
- [x] Sensor-degradation matrix qualified.
- [x] Dataset catalog accepts more than one independently identified source.
- [x] Official RELLIS Ouster example passes the shared SemanticKITTI reader.
- [x] RELLIS axes, ontology colors and versioned ground-target/ignore mapping
  are visible in the same Dataset workflow.
- [x] Full RELLIS Ouster SemanticKITTI scans, labels and poses admitted on
  worker D.
- [x] Current/Patchwork++ comparison executed and sealed over all `2,413`
  RELLIS validation frames.
- [ ] Current/Patchwork++ RELLIS comparison qualified without obstacle loss.
- [ ] Rolling local map with pose/TTL/dynamic policy qualified.

## RELLIS S0 compatibility evidence

RELLIS is the independent second-source check, not extra decoration in the
catalog. GOOSE and RELLIS share the ingress format and viewer, but retain
separate source identity, sensor domain, ontology, licensing and results.

The pinned smoke input is the official repository example at commit
`c17a118fcaed1559f03cc32cc3a91dedc557f8b8`:

- point file: `utils/example/000104.bin`;
- label file: `utils/example/000104.label`;
- source points: `131,072`;
- point SHA-256:
  `ed81a9c3636d55b17d78058c72545d5d22419beecf174d50596d23ae178752af`;
- label SHA-256:
  `9b8c65b710873e931af4ac6dfc7d3dd2298696514ab721e50300bd55ad5b634e`;
- official ontology config SHA-256:
  `573379a232ac561805987466c391a61fd5ad338be7fb9c4c2842f3f28067e0ad`;
- bounded browser preview: `20,000` deterministic even-index points;
- coordinate frame: right-handed, `x` forward, `y` left, `z` up; no vehicle
  transform, pose registration or deskew is claimed;
- license: `CC-BY-NC-SA-3.0`; this is a research qualification source, not an
  unrestricted commercial runtime dependency.

Mission Core ground policy v1 treats `dirt`, `grass`, `asphalt`, `concrete` and
`mud` as ground; clear structures, vegetation, people, vehicles and rubble as
non-ground; and `void`, `water`, `sky`, generic `object` and `puddle` as
`ignore`. The ignore set is excluded from future metrics instead of being
silently counted as either free ground or obstacle. This mapping is a Mission
Core evaluation decision, not an upstream RELLIS claim.

The smoke closed reader, alignment, axes, palette and mapping compatibility
only. The complete release was then admitted from four pinned archives kept
exclusively on worker D:

- scans: `15,037,868,425` bytes, SHA-256
  `4e2bb5654bda5b9d08b9ac1ddbd39bd6dd231163a77ff811f2f282f3bbdf8ee7`;
- labels: `182,265,710` bytes, SHA-256
  `03297c30b9f2182c74be93f8564d6ce1349237fcc4f89fc96479fe71bb40d905`;
- poses: `609,344` bytes, SHA-256
  `6deaf38a9cac3a480cfdb70fa9a1d9278f25c36f5efd2487a3c4a6a400250f67`;
- splits: `76,774` bytes, SHA-256
  `de639728e0058d7d477188028d9b811313a601884ef00d267ce13736be7cf54f`.

Admission identity
`a3f3f161a5a7edccdf66cea75ecf9004b8f6a895282faabe661c075c15a82774`
proves `13,556` scan/label pairs, the official `7,800 / 2,413 / 3,343`
train/validation/test split counts, all five sequences and matching pose
coverage. No ROS bag is needed for this single-scan ground cross-check.

## Full RELLIS qualification result

The immutable R1 replay-shadow run
`rellis-ground-5b49d241a42b41d9b427` processed all `2,413` validation frames.
Its qualification identity is
`5b49d241a42b41d9b4279062c7517c99e16880e025cd2b70d3966d49f3ff6b7f`.
The physical-height input was calibrated from `64` deterministic
sequence-stratified **train** frames only: `1.0937 m` median height,
`0.0538 m` median absolute deviation and no validation labels used.

| Validation micro metric | Current | Patchwork++ |
| --- | ---: | ---: |
| Ground IoU | `40.5450%` | `41.9599%` |
| Natural-ground recall | `72.2992%` | `93.2555%` |
| Obstacle non-ground recall | `80.4612%` | `69.7040%` |
| Assigned fraction | `100%` | `100%` |
| Latency p50 | `900.22 ms` | `8.58 ms` |
| Latency p95 | `1284.29 ms` | `13.98 ms` |
| Maximum latency | `1706.78 ms` | `20.50 ms` |

The sealed decision is `qualification-rejected`. Patchwork++ improved Ground
IoU by only `1.41` percentage points against the required `2.00`, preserved
only `69.70%` of obstacle points against the required `90%`, and regressed
obstacle recall by `10.76` points against the allowed `1%`. Coverage, latency
and catastrophic per-frame regression gates passed, but none of those can
override obstacle loss. The provider therefore remains visualization and
research evidence only:
`promoted_to_navigation_or_safety = false`.

The bound review-pack identity is
`1ee454fedd37bd56c3c9d07e17a3b5f59cdb29f7c1ae2086f8bc746149f3eb1b`.
It contains bounded `12,000`-point previews for every validation scan and keeps
the two real validation sequences separate. This lets an operator inspect raw
geometry, public ground truth, both predictions and both error maps before
changing the profile or acceptance thresholds.

## First GOOSE admission evidence

- worker root: canonical D-only root, path never exposed by the HTTP API;
- source URL:
  `https://goose-dataset.de/storage/goose_3d_val.zip`;
- observed archive bytes: `3,498,402,435`;
- observed SHA-256:
  `0be9e0f8459bafcbc92ff7c3cc366e9b4e2f6e1e9bdf50e6439e1557e864c26f`;
- archive integrity: ZIP structure, bounded expansion, safe paths, one LICENSE,
  one label mapping and aligned point/label frames;
- license artifact: Creative Commons Attribution-ShareAlike 4.0 International;
- first deterministic frame:
  `2022-07-22_flight__0071_1658494234334310308`;
- source points: `169,883`;
- semantic classes present: `17`;
- independently labeled artificial/natural ground: `45,968` points
  (`27.0586%`);
- browser preview: deterministic even-index sample of `50,000` points,
  visualization-only.

GOOSE does not publish a checksum beside this download. The recorded SHA-256 is
therefore Mission Core's observed content pin after a TLS download, not a claim
of vendor-signed authenticity. Any future byte change creates a new admission
decision rather than silently replacing this installation.

## First independent ground result

The current `missioncore-local-percentile-ground/v1` provider was run against
all `169,883` points, excluding `353` GOOSE void points from scoring:

- Ground IoU: `49.6165%`;
- precision: `73.5892%`;
- recall: `60.3659%`;
- F1: `66.3249%`;
- artificial-ground recall: `95.2364%`;
- natural-ground recall: `49.4516%`;
- obstacle non-ground recall: `79.6145%`;
- provider latency on the worker: `3,858.16 ms`.

This is the first concrete product value from the public dataset. The current
heuristic looks plausible on a dense map and is strong on artificial ground,
but it misses about half of the independently labeled natural ground and is far
from a real-time full-scan provider. The UI exposes `Current` and `Ошибки`
views, so the failure geometry is inspectable instead of being hidden behind
one aggregate score.

The provider now uses a bounded grid-neighborhood index while preserving the
previous exact-radius result. This removes the previous all-points scan for
every occupied cell, but the measured latency still classifies it as a
diagnostic baseline rather than an onboard candidate.

## First Patchwork++ A/B result

GOOSE's dimensioned MuCAR-3 schematic provides the two vertical dimensions
needed by this algorithm-specific gate: `base_link` is `0.64 m` above ground
and the VLS-128 optical center is `1.60 m` above `base_link`. The admitted
Patchwork++ height is therefore `2.24 m`. The same published schematic declares
the sensor axes as `x` forward, `y` left and `z` up. An independent fit to
GOOSE-labeled near-field ground observed a `-2.18 .. -2.14 m` intercept; this
was a non-calibrating cross-check, not the source of the height.

This narrowly admits the input required by Patchwork++ on the native
sensor-centric scan. It does **not** claim a complete numeric vehicle transform,
per-point timing, deskew or a general `normalized-scan`.

Official Patchwork++ `v1.4.1`, source commit
`3e6903a1d5537a4cc2ace897b0bbb98a92d6014c`, was run against the same first
frame and independent labels as Current:

| Metric | Current | Patchwork++ |
| --- | ---: | ---: |
| Ground IoU | `49.6165%` | `60.3702%` |
| Precision | `73.5892%` | `72.6612%` |
| Recall | `60.3659%` | `78.1130%` |
| F1 | `66.3249%` | `75.2886%` |
| Artificial-ground recall | `95.2364%` | `98.9688%` |
| Natural-ground recall | `49.4516%` | `71.5853%` |
| Obstacle non-ground recall | `79.6145%` | `92.1729%` |
| Worker latency | `3966.53 ms` | `15.89 ms` |

Reproducibility pins:

- benchmark identity:
  `2a6d05f54a9e2ac727d9c850c1133f2fb539bfeddbd97c07cfd198ccb239162c`;
- full point-aligned prediction SHA-256:
  `190f455c6e47911921b7f6454913e1bb2d0b8b5814ee2e34d7b63295afddc7ef`;
- bounded browser preview SHA-256:
  `e23ca573103faf931523415e6c263248737eb6600f482b63de685b725bbc28c3`;
- Patchwork++ binary SHA-256:
  `be8038b2098c83fe53841aa8ae19e362910e9056fe0ee7b9304c1ee5c5941094`.

Patchwork++ wins this frame by `10.75` percentage points of Ground IoU, raises
natural-ground recall by `22.13` points and is roughly `250x` faster in this
run. The result is deliberately `one-frame-diagnostic`: it makes Patchwork++
the candidate for validation-split and degradation qualification, not an
accepted navigation or safety provider.

## Validation-split qualification

The immutable R1 replay-shadow run
`goose-ground-035de3ae6155844b636d` processed all `961` aligned validation
frames directly from the D-only archive. It is bound to Mission Core commit
`d6decbe05ccecd32d6a745767ee7a795bd1c88b1`, qualification identity
`035de3ae6155844b636d89d8f23b36e561edab105720fa850094a5c6f680c0ef`
and the same pinned Patchwork++ source/binary as the one-frame diagnostic.

| Validation micro metric | Current | Patchwork++ |
| --- | ---: | ---: |
| Ground IoU | `47.4199%` | `66.3974%` |
| Natural-ground recall | `51.7650%` | `76.2215%` |
| Obstacle non-ground recall | `93.5831%` | `94.6666%` |
| Assigned fraction | `100%` | `100%` |
| Latency p50 | `3527.02 ms` | `16.04 ms` |
| Latency p95 | `6115.18 ms` | `23.40 ms` |
| Maximum latency | `11162.62 ms` | `34.12 ms` |

The predeclared nominal gates passed: Ground IoU gain is `18.98` percentage
points (required `>= 7`), natural-ground recall gain is `24.46` points
(required `>= 10`), Patchwork++ obstacle non-ground recall is `94.67%`
(required `>= 90%`), assigned fraction is `100%` (required `>= 99.9%`) and
p95 latency is `23.40 ms` (required `<= 50 ms`). No frame exceeded the
catastrophic regression boundary of `-20` Ground-IoU points; the worst observed
delta was `-13.51` points.

| Deterministic profile | Source coverage | Ground IoU | Natural recall | p95 latency |
| --- | ---: | ---: | ---: | ---: |
| `range-20m` | `52.70%` | `63.1304%` | `72.1331%` | `16.29 ms` |
| `range-40m` | `80.03%` | `68.5958%` | `80.0635%` | `20.74 ms` |
| `density-50` | `50.00%` | `66.3575%` | `76.8925%` | `11.78 ms` |
| `density-25` | `25.00%` | `65.5975%` | `76.8987%` | `5.99 ms` |
| `noise-05m` | `100%` | `63.4378%` | `70.7234%` | `23.28 ms` |
| `dropout-30` | `70.00%` | `66.4658%` | `76.6721%` | `16.24 ms` |
| `front-180` | `51.48%` | `67.6924%` | `77.8012%` | `11.82 ms` |

Every degradation profile stayed inside the predeclared `15`-point Ground-IoU
loss, `20`-point natural-recall loss and `50 ms` p95 limits. The sealed
decision is therefore `shadow-candidate`, explicitly
`promoted_to_navigation_or_safety = false`.

The first aggregate-only Polygon screen was rejected as an operator product:
five selected failure previews could not let an operator verify the decision.
Commit `b037076` adds a separate immutable visual derivative bound to the
qualification report:

- schema `missioncore.goose-ground-review-pack/v1`;
- identity
  `6ddc2de0154196088049b8654d9187fba0c52e28783e5ada6be5266dd16bd641`;
- all `961/961` validation frames in source order;
- deterministic `12,000`-point frame previews with remission, public
  point-aligned ground truth, Current and Patchwork++ masks;
- compact size `67 MB`, versus multiple gigabytes for equivalent JSON;
- direct modes for geometry, ground truth, both providers and both error maps.

The subsequent product review found a second semantic error: the `961` frames
were presented as one playable recording. They are actually `8` independent
GOOSE validation sequences:

- `2022-07-22_flight` — `151` annotated scans;
- `2022-08-30_siegertsbrunn_feldwege` — `103`;
- `2022-09-21_garching_uebungsplatz_2` — `123`;
- `2022-12-07_aying_hills` — `133`;
- `2023-01-20_aying_mangfall_2` — `191`;
- `2023-03-03_garching_2` — `106`;
- `2023-05-15_neubiberg_rain` — `73`;
- `2023-05-17_neubiberg_sunny` — `81`.

Each admitted `.bin` is one annotated VLS-128 revolution. Within those eight
sequences the validation scans are sparse: observed adjacent timestamps range
from about `0.1 s` to `167.5 s`. The annotated ZIP contains neither a
continuous playback stream nor a pose chain that could support visually
continuous ego-motion. Mission Core therefore:

- never plays across a sequence boundary;
- labels Play as an accelerated scan review, not real-time playback;
- exposes the exact source frame number and timestamp gap;
- keeps a fixed sensor-centric origin and preserves the operator camera within
  one sequence instead of refitting every frame;
- does not interpolate missing scans or invent a route.

Continuous vehicle motion, odometry and synchronized sensor playback require
the separately admitted raw GOOSE ROS bags plus localization. Those assets are
not part of the current `3.3 GB` annotated validation archive.

The source archive and full-resolution crash-resume cache remain only on worker
D. A bounded review mirror may be served by a Mission Core backend, but SSH is
not a product data plane. The replay UI does not render the unrelated live
Ackermann rover and does not depend on Simulation Worker online status.

Primary source evidence:

- [GOOSE MuCAR-3 sensor setup](https://goose-dataset.de/docs/mucar3/);
- [GOOSE paper, Figure 3](https://arxiv.org/pdf/2310.16788);
- [Patchwork++ v1.4.1](https://github.com/url-kaist/patchwork-plusplus/tree/v1.4.1).