25 KiB

Raw Blame History

LiDAR Dataset Gateway

Product value

The Dataset Gateway gives Mission Core a repeatable perception laboratory before the production vehicle and its final sensor installation exist. It separates four questions that were previously mixed together:

whether the transport preserved the sensor evidence;
whether preprocessing produces a valid one-scan perception input;
whether an algorithm is accurate against independent labels;
whether several scans form a stable local map for an operator or planner.

This prevents tuning an algorithm until a visually dense vendor map merely looks plausible. It also keeps work reusable across Gazebo, Unreal, public datasets and future real onboard sensors.

Dataset names an immutable experiment input, not one universal point-cloud representation. Mission Core keeps two evidence classes:

public labeled native-scan datasets such as GOOSE and RELLIS qualify algorithms against independent answers;
private real-sensor datasets such as RAVNOVES00 replay the exact K1 lio_pcl/lio_pose representation received during a physical session.

RAVNOVES00 is therefore a dataset for K1 local-world-model experiments even though it must never be relabeled as a native scan. Jobs bind its existing session/replay identity and create replaceable derivatives; they do not copy, rewrite or mix public points into the K1 source.

Current admitted slice

Implemented now:

multi-source missioncore.dataset-gateway-catalog/v3, exposed read-only at GET /api/v1/lidar/dataset-gateway;
path-free missioncore.dataset-admission/v1 worker evidence instead of a static React status;
bounded missioncore.dataset-native-scan-preview/v1 (GOOSE) and /v2 (RELLIS) artifacts, selected by source at GET /api/v1/lidar/dataset-gateway/preview?source_id=...;
explicit native-scan, normalized-scan and rolling-local-map representations;
a lossless GOOSE/SemanticKITTI frame reader for little-endian float32 XYZI and packed uint32 semantic/instance labels;
count, finite-value and maximum-point safety gates;
immutable point-aligned arrays;
a fail-closed K1 lio_pcl boundary;
a content-addressed missioncore.k1-local-surface/v1 replay derivative over immutable RAVNOVES00, with dynamic height/slope/roughness/confidence, explicit pose-binding age and conservative observed occupied/unknown policy;
leave-current-frame-out next-frame qualification over 525 samples, temporal jump evidence and unverified local-discontinuity candidates;
deterministic missioncore.k1-local-surface-review/v1 replay triage: 37 attention frames grouped into 21 episodes, with four high-priority frames and direct source-frame navigation;
point/cell-aligned prediction evidence in missioncore.k1-local-surface-frame/v2: the prior-only plane, current lower-cell coordinates, signed residual and derived inlier mask are content-bound and visible as a read-only 3D overlay;
a provider-neutral read-only local-surface view in Парк → Диагностика LiDAR, synchronized to the five existing RAVNOVES00 scene selectors and a clickable complete-recording timeline;
worker storage admission for D:\NDC_MISSIONCORE\datasets and /mnt/d/NDC_MISSIONCORE/datasets;
a dedicated, honest dataset catalog in Полигон → Датасеты;
live admission states (downloading, verifying, frame-ready) and a real Открыть action;
one admitted GOOSE validation frame with source colors, normalized remission and an independent ground-truth view;
a fail-closed admission of the complete official RELLIS-3D v1.1 release: 13,556 aligned Ouster scans and labels, all five sequences, official train/validation/test splits and complete LiDAR pose coverage;
a full RELLIS validation viewer with its two real source sequences, 2,413 source scans, ground truth, Current, Patchwork++ and error modes;
a train-only, label-backed Ouster height calibration and a sealed R1 Current/Patchwork++ qualification over every RELLIS validation frame;
a published, dimensioned MuCAR-3/VLS-128 Patchwork++ input profile: sensor frame x-forward / y-left / z-up, physical height 2.24 m, native one-revolution scan and explicit absence of deskew/full vehicle TF claims;
a reproducible Current/Patchwork++ A/B artifact with point-aligned masks, disagreement views, independent-label metrics, latency and exact provider identities;
a separate Парк → Диагностика LiDAR surface containing only real sensor recordings and their operational evidence.

Not implemented:

no browser-triggered large-artifact download;
no implicit coordinate conversion;
no fake ring/timestamp reconstruction for K1 MQTT evidence;
no model training or production promotion;
no RELLIS ROS bag admission, continuous synchronized playback or production promotion;
no ray-cleared free-space or planner-authoritative rolling occupancy map;
the accepted K1 local-surface profile now passes a 15-second recorded-source-paced bounded shadow gate; authenticated physical lio_pcl + lio_pose binding and its pinned acceptance profile are implemented, but no attached-K1 run has qualified them yet. The residual overlay remains non-authoritative.

Product surface boundary

The gateway is not part of device quality diagnostics.

Парк → Диагностика LiDAR answers whether a selected real sensor recording is present, reproducible and operationally usable. It never mixes public dataset frames into the selected device evidence.
Наблюдение opens a concrete live or recorded spatial scene.
Данные owns source-of-record, retention, replay preparation and export.
Полигон → Датасеты is the single operator surface for admitted evaluation inputs. Opening one dataset exposes its source fragments, sparse annotated scans, visual overlays and useful analysis calculated only for the selected fragment.
Internal replay-run identity, provenance and immutable artifacts remain a backend evidence contract. They are not a second operator navigation item and are not presented as vehicle motion.

Before compatible source evidence exists, the catalog explains the blocked storage gate and keeps Открыть датасет disabled. GOOSE becomes available only after the worker manifest says frame-ready; RELLIS becomes dataset-ready only after all four pinned archives, every split pair and pose coverage pass admission. Both bounded previews must pass their own identity, point-alignment and safety checks.

Why public recordings look different

GOOSE stores one VLS-128 revolution per annotated .bin file. A rotating multi-channel sensor produces discrete scan lines, so a single sensor-frame view looks like sparse rings.

The admitted goose_3d_val.zip uses lidar/val/.../*_vls128.bin even though some GOOSE documentation examples call the point-cloud root velodyne. Mission Core accepts either declared root but still requires an exactly aligned labels/val/.../*_goose.label.

The current field review is explicitly an accumulated map-frame window. It combines many source publications after pose registration. This fills surfaces and hides the original scan pattern. The external K1 stream is also already a post-LIO/modeling product and lacks the raw driver fields needed to reconstruct an original scan.

Livox sensors additionally use a scan pattern that differs from classic fixed vertical channels. Time integration therefore changes their visual density in a different way. “Ring-like” is a sensor geometry property, not a universal quality target.

Canonical processing profiles

P0 — native evidence

Required:

source ID and immutable frame ID;
XYZ and the original return/remission/intensity field;
semantic and instance labels when present;
calibration/mounting/timing evidence as separate metadata;
no accumulation and no hidden world transform.

Output: native-scan.

P1 — normalized perception scan

Ordered operations:

decode and apply only evidenced factory calibration;
assign an explicit sensor coordinate frame;
deskew when per-point time and synchronized motion are available;
apply bounded range and field-of-view policy;
remove the vehicle/self mask;
apply named outlier and voxel policies;
retain a reversible index/provenance map to the native frame.

Output: normalized-scan.

P2 — inference

Ground, semantic and object providers consume P1. Patchwork++ belongs here. It does not own P0/P1 or P3.

Output: point-aligned predictions and reproducible metrics against labels.

P3 — rolling local map

Ordered operations:

bind each normalized scan to an evidenced pose;
transform to odom or a declared local-map frame;
deduplicate with a named voxel policy;
expire points by TTL or travelled distance;
keep dynamic points short-lived or track them separately;
publish bounded map state and its contributing frame identities.

Output: rolling-local-map.

This is the stage that should stop static geometry from “jumping”. Deskew reduces within-scan motion distortion; registration stabilizes scans across time; TTL/dynamic filtering prevents stale ghosts.

First dataset sequence

[Done] Admit /mnt/d/NDC_MISSIONCORE/datasets on the Windows/WSL worker.
[Done] Verify free space and record archive size/hash/license.
[Done] Download only the GOOSE 3D validation archive first (published size 3.3 GB).
[Done] Import one labeled frame and expose it in React as native-scan.
[Done] Show native remission, the original semantic palette and ground-truth superclass coloring.
[Patchwork-specific profile done; general normalization still pending] Admit the published VLS-128 axes and physical height without claiming a complete numeric vehicle TF or deskewed normalized-scan.
[One-frame A/B done] Run the current ground heuristic and pinned Patchwork++ against the same native scan and independent labels.
[Done] Add deterministic sensor-degradation profiles for range, FOV, density, noise and dropout without modifying native source frames.
[Done] Expand the A/B to all 961 validation frames and seal the result as a Polygon replay-shadow run. Add a rolling-map sequence with localization evidence only after this qualification boundary.

Acceptance checklist

Representations cannot be silently interchanged.
Large artifacts require operator-admitted D-only storage.
GOOSE XYZI and labels remain point aligned.
Invalid length and non-finite frames fail closed.
K1 mapped increments cannot claim raw-scan fields.
React exposes an honest catalog/empty state before dataset bytes exist.
Dataset Gateway is isolated from real-sensor diagnostics and placed under Polygon qualification.
Worker D root configured.
GOOSE validation archive hash recorded.
First real labeled frame visible in React.
Patchwork-specific axes and physical-height profile admitted.
Complete vehicle transform and general normalized-scan admitted.
Current local-percentile baseline measured against ground truth.
Patchwork++ one-frame accuracy measured against ground truth.
Current/Patchwork++ validation-split gate qualified.
Sensor-degradation matrix qualified.
Dataset catalog accepts more than one independently identified source.
Official RELLIS Ouster example passes the shared SemanticKITTI reader.
RELLIS axes, ontology colors and versioned ground-target/ignore mapping are visible in the same Dataset workflow.
Full RELLIS Ouster SemanticKITTI scans, labels and poses admitted on worker D.
Current/Patchwork++ comparison executed and sealed over all 2,413 RELLIS validation frames.
Current/Patchwork++ RELLIS comparison qualified without obstacle loss.
Rolling local map with pose/TTL/dynamic policy qualified.

RELLIS S0 compatibility evidence

RELLIS is the independent second-source check, not extra decoration in the catalog. GOOSE and RELLIS share the ingress format and viewer, but retain separate source identity, sensor domain, ontology, licensing and results.

The pinned smoke input is the official repository example at commit c17a118fcaed1559f03cc32cc3a91dedc557f8b8:

point file: utils/example/000104.bin;
label file: utils/example/000104.label;
source points: 131,072;
point SHA-256: ed81a9c3636d55b17d78058c72545d5d22419beecf174d50596d23ae178752af;
label SHA-256: 9b8c65b710873e931af4ac6dfc7d3dd2298696514ab721e50300bd55ad5b634e;
official ontology config SHA-256: 573379a232ac561805987466c391a61fd5ad338be7fb9c4c2842f3f28067e0ad;
bounded browser preview: 20,000 deterministic even-index points;
coordinate frame: right-handed, x forward, y left, z up; no vehicle transform, pose registration or deskew is claimed;
license: CC-BY-NC-SA-3.0; this is a research qualification source, not an unrestricted commercial runtime dependency.

Mission Core ground policy v1 treats dirt, grass, asphalt, concrete and mud as ground; clear structures, vegetation, people, vehicles and rubble as non-ground; and void, water, sky, generic object and puddle as ignore. The ignore set is excluded from future metrics instead of being silently counted as either free ground or obstacle. This mapping is a Mission Core evaluation decision, not an upstream RELLIS claim.

The smoke closed reader, alignment, axes, palette and mapping compatibility only. The complete release was then admitted from four pinned archives kept exclusively on worker D:

scans: 15,037,868,425 bytes, SHA-256 4e2bb5654bda5b9d08b9ac1ddbd39bd6dd231163a77ff811f2f282f3bbdf8ee7;
labels: 182,265,710 bytes, SHA-256 03297c30b9f2182c74be93f8564d6ce1349237fcc4f89fc96479fe71bb40d905;
poses: 609,344 bytes, SHA-256 6deaf38a9cac3a480cfdb70fa9a1d9278f25c36f5efd2487a3c4a6a400250f67;
splits: 76,774 bytes, SHA-256 de639728e0058d7d477188028d9b811313a601884ef00d267ce13736be7cf54f.

Admission identity a3f3f161a5a7edccdf66cea75ecf9004b8f6a895282faabe661c075c15a82774 proves 13,556 scan/label pairs, the official 7,800 / 2,413 / 3,343 train/validation/test split counts, all five sequences and matching pose coverage. No ROS bag is needed for this single-scan ground cross-check.

Full RELLIS qualification result

The immutable R1 replay-shadow run rellis-ground-5b49d241a42b41d9b427 processed all 2,413 validation frames. Its qualification identity is 5b49d241a42b41d9b4279062c7517c99e16880e025cd2b70d3966d49f3ff6b7f. The physical-height input was calibrated from 64 deterministic sequence-stratified train frames only: 1.0937 m median height, 0.0538 m median absolute deviation and no validation labels used.

Validation micro metric	Current	Patchwork++
Ground IoU	`40.5450%`	`41.9599%`
Natural-ground recall	`72.2992%`	`93.2555%`
Obstacle non-ground recall	`80.4612%`	`69.7040%`
Assigned fraction	`100%`	`100%`
Latency p50	`900.22 ms`	`8.58 ms`
Latency p95	`1284.29 ms`	`13.98 ms`
Maximum latency	`1706.78 ms`	`20.50 ms`

The sealed decision is qualification-rejected. Patchwork++ improved Ground IoU by only 1.41 percentage points against the required 2.00, preserved only 69.70% of obstacle points against the required 90%, and regressed obstacle recall by 10.76 points against the allowed 1%. Coverage, latency and catastrophic per-frame regression gates passed, but none of those can override obstacle loss. The provider therefore remains visualization and research evidence only: promoted_to_navigation_or_safety = false.

The bound review-pack identity is 1ee454fedd37bd56c3c9d07e17a3b5f59cdb29f7c1ae2086f8bc746149f3eb1b. It contains bounded 12,000-point previews for every validation scan and keeps the two real validation sequences separate. This lets an operator inspect raw geometry, public ground truth, both predictions and both error maps before changing the profile or acceptance thresholds.

First GOOSE admission evidence

worker root: canonical D-only root, path never exposed by the HTTP API;
source URL: https://goose-dataset.de/storage/goose_3d_val.zip;
observed archive bytes: 3,498,402,435;
observed SHA-256: 0be9e0f8459bafcbc92ff7c3cc366e9b4e2f6e1e9bdf50e6439e1557e864c26f;
archive integrity: ZIP structure, bounded expansion, safe paths, one LICENSE, one label mapping and aligned point/label frames;
license artifact: Creative Commons Attribution-ShareAlike 4.0 International;
first deterministic frame: 2022-07-22_flight__0071_1658494234334310308;
source points: 169,883;
semantic classes present: 17;
independently labeled artificial/natural ground: 45,968 points (27.0586%);
browser preview: deterministic even-index sample of 50,000 points, visualization-only.

GOOSE does not publish a checksum beside this download. The recorded SHA-256 is therefore Mission Core's observed content pin after a TLS download, not a claim of vendor-signed authenticity. Any future byte change creates a new admission decision rather than silently replacing this installation.

First independent ground result

The current missioncore-local-percentile-ground/v1 provider was run against all 169,883 points, excluding 353 GOOSE void points from scoring:

Ground IoU: 49.6165%;
precision: 73.5892%;
recall: 60.3659%;
F1: 66.3249%;
artificial-ground recall: 95.2364%;
natural-ground recall: 49.4516%;
obstacle non-ground recall: 79.6145%;
provider latency on the worker: 3,858.16 ms.

This is the first concrete product value from the public dataset. The current heuristic looks plausible on a dense map and is strong on artificial ground, but it misses about half of the independently labeled natural ground and is far from a real-time full-scan provider. The UI exposes Current and Ошибки views, so the failure geometry is inspectable instead of being hidden behind one aggregate score.

The provider now uses a bounded grid-neighborhood index while preserving the previous exact-radius result. This removes the previous all-points scan for every occupied cell, but the measured latency still classifies it as a diagnostic baseline rather than an onboard candidate.

First Patchwork++ A/B result

GOOSE's dimensioned MuCAR-3 schematic provides the two vertical dimensions needed by this algorithm-specific gate: base_link is 0.64 m above ground and the VLS-128 optical center is 1.60 m above base_link. The admitted Patchwork++ height is therefore 2.24 m. The same published schematic declares the sensor axes as x forward, y left and z up. An independent fit to GOOSE-labeled near-field ground observed a -2.18 .. -2.14 m intercept; this was a non-calibrating cross-check, not the source of the height.

This narrowly admits the input required by Patchwork++ on the native sensor-centric scan. It does not claim a complete numeric vehicle transform, per-point timing, deskew or a general normalized-scan.

Official Patchwork++ v1.4.1, source commit 3e6903a1d5537a4cc2ace897b0bbb98a92d6014c, was run against the same first frame and independent labels as Current:

Metric	Current	Patchwork++
Ground IoU	`49.6165%`	`60.3702%`
Precision	`73.5892%`	`72.6612%`
Recall	`60.3659%`	`78.1130%`
F1	`66.3249%`	`75.2886%`
Artificial-ground recall	`95.2364%`	`98.9688%`
Natural-ground recall	`49.4516%`	`71.5853%`
Obstacle non-ground recall	`79.6145%`	`92.1729%`
Worker latency	`3966.53 ms`	`15.89 ms`

Reproducibility pins:

benchmark identity: 2a6d05f54a9e2ac727d9c850c1133f2fb539bfeddbd97c07cfd198ccb239162c;
full point-aligned prediction SHA-256: 190f455c6e47911921b7f6454913e1bb2d0b8b5814ee2e34d7b63295afddc7ef;
bounded browser preview SHA-256: e23ca573103faf931523415e6c263248737eb6600f482b63de685b725bbc28c3;
Patchwork++ binary SHA-256: be8038b2098c83fe53841aa8ae19e362910e9056fe0ee7b9304c1ee5c5941094.

Patchwork++ wins this frame by 10.75 percentage points of Ground IoU, raises natural-ground recall by 22.13 points and is roughly 250x faster in this run. The result is deliberately one-frame-diagnostic: it makes Patchwork++ the candidate for validation-split and degradation qualification, not an accepted navigation or safety provider.

Validation-split qualification

The immutable R1 replay-shadow run goose-ground-035de3ae6155844b636d processed all 961 aligned validation frames directly from the D-only archive. It is bound to Mission Core commit d6decbe05ccecd32d6a745767ee7a795bd1c88b1, qualification identity 035de3ae6155844b636d89d8f23b36e561edab105720fa850094a5c6f680c0ef and the same pinned Patchwork++ source/binary as the one-frame diagnostic.

Validation micro metric	Current	Patchwork++
Ground IoU	`47.4199%`	`66.3974%`
Natural-ground recall	`51.7650%`	`76.2215%`
Obstacle non-ground recall	`93.5831%`	`94.6666%`
Assigned fraction	`100%`	`100%`
Latency p50	`3527.02 ms`	`16.04 ms`
Latency p95	`6115.18 ms`	`23.40 ms`
Maximum latency	`11162.62 ms`	`34.12 ms`

The predeclared nominal gates passed: Ground IoU gain is 18.98 percentage points (required >= 7), natural-ground recall gain is 24.46 points (required >= 10), Patchwork++ obstacle non-ground recall is 94.67% (required >= 90%), assigned fraction is 100% (required >= 99.9%) and p95 latency is 23.40 ms (required <= 50 ms). No frame exceeded the catastrophic regression boundary of -20 Ground-IoU points; the worst observed delta was -13.51 points.

Deterministic profile	Source coverage	Ground IoU	Natural recall	p95 latency
`range-20m`	`52.70%`	`63.1304%`	`72.1331%`	`16.29 ms`
`range-40m`	`80.03%`	`68.5958%`	`80.0635%`	`20.74 ms`
`density-50`	`50.00%`	`66.3575%`	`76.8925%`	`11.78 ms`
`density-25`	`25.00%`	`65.5975%`	`76.8987%`	`5.99 ms`
`noise-05m`	`100%`	`63.4378%`	`70.7234%`	`23.28 ms`
`dropout-30`	`70.00%`	`66.4658%`	`76.6721%`	`16.24 ms`
`front-180`	`51.48%`	`67.6924%`	`77.8012%`	`11.82 ms`

Every degradation profile stayed inside the predeclared 15-point Ground-IoU loss, 20-point natural-recall loss and 50 ms p95 limits. The sealed decision is therefore shadow-candidate, explicitly promoted_to_navigation_or_safety = false.

The first aggregate-only Polygon screen was rejected as an operator product: five selected failure previews could not let an operator verify the decision. Commit b037076 adds a separate immutable visual derivative bound to the qualification report:

schema missioncore.goose-ground-review-pack/v1;
identity 6ddc2de0154196088049b8654d9187fba0c52e28783e5ada6be5266dd16bd641;
all 961/961 validation frames in source order;
deterministic 12,000-point frame previews with remission, public point-aligned ground truth, Current and Patchwork++ masks;
compact size 67 MB, versus multiple gigabytes for equivalent JSON;
direct modes for geometry, ground truth, both providers and both error maps.

The subsequent product review found a second semantic error: the 961 frames were presented as one playable recording. They are actually 8 independent GOOSE validation sequences:

2022-07-22_flight — 151 annotated scans;
2022-08-30_siegertsbrunn_feldwege — 103;
2022-09-21_garching_uebungsplatz_2 — 123;
2022-12-07_aying_hills — 133;
2023-01-20_aying_mangfall_2 — 191;
2023-03-03_garching_2 — 106;
2023-05-15_neubiberg_rain — 73;
2023-05-17_neubiberg_sunny — 81.

Each admitted .bin is one annotated VLS-128 revolution. Within those eight sequences the validation scans are sparse: observed adjacent timestamps range from about 0.1 s to 167.5 s. The annotated ZIP contains neither a continuous playback stream nor a pose chain that could support visually continuous ego-motion. Mission Core therefore:

never plays across a sequence boundary;
labels Play as an accelerated scan review, not real-time playback;
exposes the exact source frame number and timestamp gap;
keeps a fixed sensor-centric origin and preserves the operator camera within one sequence instead of refitting every frame;
does not interpolate missing scans or invent a route.

Continuous vehicle motion, odometry and synchronized sensor playback require the separately admitted raw GOOSE ROS bags plus localization. Those assets are not part of the current 3.3 GB annotated validation archive.

The source archive and full-resolution crash-resume cache remain only on worker D. A bounded review mirror may be served by a Mission Core backend, but SSH is not a product data plane. The replay UI does not render the unrelated live Ackermann rover and does not depend on Simulation Worker online status.

Primary source evidence:

25 KiB Raw Blame History