From cea63f9a1eab97c217ba0c34b7855aac84e1b8fa Mon Sep 17 00:00:00 2001 From: DCCONSTRUCTIONS Date: Sat, 25 Jul 2026 16:19:07 +0300 Subject: [PATCH] docs(lidar): record GOOSE qualification evidence --- docs/13_LIDAR_WORKER_PRODUCT_AND_ROADMAP.md | 17 ++++--- docs/14_LIDAR_DATASET_GATEWAY.md | 56 +++++++++++++++++++-- 2 files changed, 61 insertions(+), 12 deletions(-) diff --git a/docs/13_LIDAR_WORKER_PRODUCT_AND_ROADMAP.md b/docs/13_LIDAR_WORKER_PRODUCT_AND_ROADMAP.md index 0c078a8..ed0cc84 100644 --- a/docs/13_LIDAR_WORKER_PRODUCT_AND_ROADMAP.md +++ b/docs/13_LIDAR_WORKER_PRODUCT_AND_ROADMAP.md @@ -313,8 +313,9 @@ harness work. claiming full vehicle TF, deskew or a general normalized scan. - [x] Run current ground and Patchwork++ against the same GOOSE native scan and ground truth; keep the result one-frame diagnostic. -- [ ] Expand Current/Patchwork++ qualification to the validation split. -- [ ] Add named range/FOV/density/noise/dropout degradation profiles without +- [x] Expand Current/Patchwork++ qualification to all 961 validation frames + through an immutable R1 Polygon replay-shadow run. +- [x] Add named range/FOV/density/noise/dropout degradation profiles without overwriting the native frame. The architectural contract and run sequence are fixed in @@ -390,8 +391,10 @@ The near-term value is not a prettier point cloud: - hardware selection becomes evidence-driven: a future vehicle LiDAR is accepted by its timing/fields/profile, not by vendor marketing. -The highest-value immediate work is now the validation-split -Current/Patchwork++ gate plus named range/FOV/density/noise/dropout degradation -profiles. LiDAR-native detection follows on the same gateway. Nvblox and -alternative SLAM remain later because their timing, pose and scan-geometry -gates are not yet satisfied. +The GOOSE validation-split and deterministic degradation gates are complete: +pinned Patchwork++ improves micro Ground IoU from `47.42%` to `66.40%`, raises +natural-ground recall from `51.76%` to `76.22%` and stays below `23.41 ms` p95 +across 961 frames. The decision remains shadow-only. The highest-value +immediate work is now a second-source cross-check plus LiDAR-native detection +on the same gateway. Nvblox and alternative SLAM remain later because their +timing, pose and scan-geometry gates are not yet satisfied. diff --git a/docs/14_LIDAR_DATASET_GATEWAY.md b/docs/14_LIDAR_DATASET_GATEWAY.md index 366f53f..9b4b7d6 100644 --- a/docs/14_LIDAR_DATASET_GATEWAY.md +++ b/docs/14_LIDAR_DATASET_GATEWAY.md @@ -163,9 +163,11 @@ time; TTL/dynamic filtering prevents stale ghosts. numeric vehicle TF or deskewed `normalized-scan`. 7. [One-frame A/B done] Run the current ground heuristic and pinned Patchwork++ against the same native scan and independent labels. -8. Add sensor-degradation profiles for range, FOV, density, noise and dropout. -9. Expand the A/B to the validation split; only after that qualification gate, - add a rolling-map sequence with localization evidence. +8. [Done] Add deterministic sensor-degradation profiles for range, FOV, + density, noise and dropout without modifying native source frames. +9. [Done] Expand the A/B to all 961 validation frames and seal the result as a + Polygon replay-shadow run. Add a rolling-map sequence with localization + evidence only after this qualification boundary. ## Acceptance checklist @@ -184,8 +186,8 @@ time; TTL/dynamic filtering prevents stale ghosts. - [ ] Complete vehicle transform and general `normalized-scan` admitted. - [x] Current local-percentile baseline measured against ground truth. - [x] Patchwork++ one-frame accuracy measured against ground truth. -- [ ] Current/Patchwork++ validation-split gate qualified. -- [ ] Sensor-degradation matrix qualified. +- [x] Current/Patchwork++ validation-split gate qualified. +- [x] Sensor-degradation matrix qualified. - [ ] Rolling local map with pose/TTL/dynamic policy qualified. ## First GOOSE admission evidence @@ -285,6 +287,50 @@ run. The result is deliberately `one-frame-diagnostic`: it makes Patchwork++ the candidate for validation-split and degradation qualification, not an accepted navigation or safety provider. +## Validation-split qualification + +The immutable R1 replay-shadow run +`goose-ground-035de3ae6155844b636d` processed all `961` aligned validation +frames directly from the D-only archive. It is bound to Mission Core commit +`d6decbe05ccecd32d6a745767ee7a795bd1c88b1`, qualification identity +`035de3ae6155844b636d89d8f23b36e561edab105720fa850094a5c6f680c0ef` +and the same pinned Patchwork++ source/binary as the one-frame diagnostic. + +| Validation micro metric | Current | Patchwork++ | +| --- | ---: | ---: | +| Ground IoU | `47.4199%` | `66.3974%` | +| Natural-ground recall | `51.7650%` | `76.2215%` | +| Obstacle non-ground recall | `93.5831%` | `94.6666%` | +| Assigned fraction | `100%` | `100%` | +| Latency p50 | `3527.02 ms` | `16.04 ms` | +| Latency p95 | `6115.18 ms` | `23.40 ms` | +| Maximum latency | `11162.62 ms` | `34.12 ms` | + +The predeclared nominal gates passed: Ground IoU gain is `18.98` percentage +points (required `>= 7`), natural-ground recall gain is `24.46` points +(required `>= 10`), Patchwork++ obstacle non-ground recall is `94.67%` +(required `>= 90%`), assigned fraction is `100%` (required `>= 99.9%`) and +p95 latency is `23.40 ms` (required `<= 50 ms`). No frame exceeded the +catastrophic regression boundary of `-20` Ground-IoU points; the worst observed +delta was `-13.51` points and remains visible as a bounded failure preview. + +| Deterministic profile | Source coverage | Ground IoU | Natural recall | p95 latency | +| --- | ---: | ---: | ---: | ---: | +| `range-20m` | `52.70%` | `63.1304%` | `72.1331%` | `16.29 ms` | +| `range-40m` | `80.03%` | `68.5958%` | `80.0635%` | `20.74 ms` | +| `density-50` | `50.00%` | `66.3575%` | `76.8925%` | `11.78 ms` | +| `density-25` | `25.00%` | `65.5975%` | `76.8987%` | `5.99 ms` | +| `noise-05m` | `100%` | `63.4378%` | `70.7234%` | `23.28 ms` | +| `dropout-30` | `70.00%` | `66.4658%` | `76.6721%` | `16.24 ms` | +| `front-180` | `51.48%` | `67.6924%` | `77.8012%` | `11.82 ms` | + +Every degradation profile stayed inside the predeclared `15`-point Ground-IoU +loss, `20`-point natural-recall loss and `50 ms` p95 limits. The sealed +decision is therefore `shadow-candidate`, explicitly +`promoted_to_navigation_or_safety = false`. The compact `12 MB` run journal, +report and five worst-frame previews are available in `Polygon -> Runs`; the +source archive and crash-resume frame cache remain only on worker D. + Primary source evidence: - [GOOSE MuCAR-3 sensor setup](https://goose-dataset.de/docs/mucar3/);