feat(lidar): add ground segmentation diagnostic benchmark

This commit is contained in:
DCCONSTRUCTIONS
2026-07-25 02:09:27 +03:00
parent e4fdd9fa20
commit 75a3e669d9
14 changed files with 2235 additions and 15 deletions
+10
View File
@@ -14,6 +14,16 @@ equivalence report. The React surface reads those reports through the read-only
`/api/v1/lidar/replay-packs` boundary; CUDA/TensorRT and ROS do not move into the
browser.
`missioncore.lidar-ground-benchmark/v1` extends the same boundary for L2. It
keeps point-aligned ground/assigned masks for the current local-percentile
proposal and a candidate provider, exact provider/source/binary identities,
host latency and algorithm disagreement. The first official Patchwork++ v1.4.1
run is diagnostic-only: current K1 evidence is a vendor-map increment, not the
sensor-centric scan and physical-height contract Patchwork++ expects. The
read-only `/api/v1/lidar/ground-benchmarks` surface therefore publishes
`production_promotion=false` until an independent annotation generation or an
admitted raw scan closes the input gate.
## Boundary
```text
+38 -11
View File
@@ -1,7 +1,8 @@
# LiDAR worker: product value, evidence boundary and implementation roadmap
Date: 2026-07-25
Status: accepted architecture plan; L0 and L1 implemented
Status: accepted architecture plan; L0/L1 implemented; L2 diagnostic A/B complete,
independent labels pending
Scope: real scanner records, replay and future live shadow processing
Explicitly out of scope: Unreal U0/U1, Gaussian assets and simulator rendering
@@ -111,7 +112,7 @@ and training-domain fit.
| Component | Correct use | Decision |
| --- | --- | --- |
| [Patchwork++](https://github.com/url-kaist/patchwork-plusplus) | Fast adaptive ground segmentation, including reflection-noise handling | First non-neural geometry baseline |
| [Patchwork++](https://github.com/url-kaist/patchwork-plusplus) | Fast adaptive ground segmentation, including reflection-noise handling | v1.4.1 benchmarked; do not promote on the current vendor-map feed |
| [Autoware CenterPoint](https://github.com/autowarefoundation/autoware_universe/tree/main/perception/autoware_lidar_centerpoint) | Mature ROS 2/TensorRT 3D detection and multi-frame reference | Second detector baseline after PointPillars |
| [MMDetection3D](https://github.com/open-mmlab/mmdetection3d) | Training/evaluation harness and dataset adapters | Laboratory only |
| [OpenPCDet](https://github.com/open-mmlab/OpenPCDet) | Alternative LiDAR detector benchmark/model zoo | Laboratory only; not the production runtime |
@@ -192,17 +193,43 @@ Accepted real slice:
The long interval tail is evidence to investigate in the scanner/transport
quality line. It is not repaired or hidden by replay.
### L2 — geometric baseline
### L2 — geometric baseline — diagnostic A/B complete, accuracy gate open
- [ ] Run Patchwork++ over the frozen XYZI slice.
- [ ] Retain ground and non-ground outputs separately; never delete raw points.
- [ ] Label a small independent ground/obstacle evaluation set in CVAT or an
equivalent accepted annotation workspace.
- [ ] Measure ground IoU, curb/low-obstacle recall, reflection-noise rejection
and CPU/GPU latency.
- [ ] Compare against the current heuristic ground proposal branch.
- [x] Pin and run official Patchwork++ v1.4.1 at source commit
`3e6903a1d5537a4cc2ace897b0bbb98a92d6014c`.
- [x] Compare it against a full-frame, point-aligned extension of the current
E19 local-percentile ground proposal.
- [x] Retain separate current/candidate ground and assigned masks for every
source point; raw replay remains unchanged.
- [x] Seal latency, ground-fraction, algorithm-IoU and disagreement
distributions in `missioncore.lidar-ground-benchmark/v1`.
- [x] Create an immutable eight-frame annotation template in which every point
starts as `ignore-unreviewed`; it is explicitly not ground truth.
- [ ] Complete independent human review for ground, curb, low obstacle,
reflection noise and other non-ground points.
- [ ] Measure accepted ground IoU, curb/low-obstacle recall and
reflection-noise rejection against that reviewed generation.
Exit: an evidence-backed decision to retain or reject Patchwork++.
The real diagnostic run is
`ground-benchmark-68cfd7a8f1dd4c0006183bb4f63a23f9ff1dd7459317ff0886e995f6c320d984`.
It covers all 66 frames and 226,963 points. The current local-percentile
proposal classified 18.31% ground at p50 with 9.74 ms p95 host latency.
Patchwork++ classified only 0.53% ground at p50 with 0.26 ms p95 host latency.
Their point-aligned ground IoU was 2.90% p50 and disagreement was 17.92% p50.
These last two values compare algorithms; they are not accuracy metrics.
The result is an evidence-backed **do-not-promote** decision for the current K1
feed. Patchwork++ is fast, but its input model assumes a sensor-centric scan
and physical sensor height. K1 `lio_pcl` is a vendor-mapped increment, its
physical sensor height is not encoded by the best-effort pose, and scan
geometry remains unknown. Translating the cloud until Patchwork++ looks
plausible would tune against the candidate and invalidate the comparison.
The independent-label exit remains open. It can be closed by reviewing the
content-bound template
`ground-annotation-template-12e12eab0756c14f5e06adcaf13189e93188df9bb6d31224cb9b7d566ec15b18`,
or by acquiring an admitted raw sensor scan with known physical sensor height
and then producing a new benchmark generation.
### L3 — LiDAR-native 3D detection
@@ -0,0 +1,69 @@
# ADR 0020: do not promote Patchwork++ on vendor-mapped LiDAR increments
Date: 2026-07-25
Status: accepted and implemented as a diagnostic gate
## Context
The L2 roadmap selected Patchwork++ as the first non-neural ground segmentation
baseline. The official implementation is designed around a sensor-centric
LiDAR scan, radial zones and a physical sensor height. Current K1 `lio_pcl`
evidence instead contains a vendor-mapped point increment in `map`, an
independent best-effort pose and no admitted raw sweep or scan geometry. The
pose describes vendor odometry; it does not prove the physical height of the
LiDAR above terrain.
Treating a successful Patchwork++ call as a valid baseline would conflate API
compatibility with input-domain compatibility. Choosing a synthetic Z
translation until the output looks plausible would use the candidate itself to
define the normalization.
## Decision
1. Pin the official Patchwork++ v1.4.1 source at commit
`3e6903a1d5537a4cc2ace897b0bbb98a92d6014c`.
2. Run it only behind the provider-neutral `lidar-ground/v1` diagnostic
boundary.
3. Bind the exact source commit and compiled binary SHA-256 into every result.
4. Preserve point-aligned ground and assigned masks separately from raw replay.
5. Compare it with the current local-percentile proposal using
algorithm-to-algorithm IoU, disagreement and latency.
6. Mark input-domain acceptance, labeled accuracy and production promotion
false for current K1 vendor-map evidence.
7. Never present algorithm IoU as ground IoU.
8. Create an all-ignore, content-bound annotation template; only a separate
human-reviewed generation may unlock ground IoU, curb/low-obstacle recall
and reflection-noise rejection.
9. Keep command, navigation and safety authority false.
## Consequences
- Patchwork++ remains a useful candidate for a future raw vehicle LiDAR feed.
- Its sub-millisecond host latency on the current slice does not compensate for
the unaccepted input domain.
- Current K1 work should prioritize independent labels and source evidence, not
threshold tuning around a mis-specified sensor model.
- The same immutable benchmark/API/React surface can compare a future raw scan,
replay or simulation provider without moving C++ processing into React.
## Real diagnostic evidence
Benchmark
`ground-benchmark-68cfd7a8f1dd4c0006183bb4f63a23f9ff1dd7459317ff0886e995f6c320d984`
processed 66 frames and 226,963 points:
| Measurement | Current local percentile | Patchwork++ |
| --- | ---: | ---: |
| Ground fraction p50 | 18.31% | 0.53% |
| Host latency p95 | 9.74 ms | 0.26 ms |
Algorithm ground IoU was 2.90% p50 and point disagreement was 17.92% p50.
Neither is an accuracy metric. Independent labels are still missing.
## References
- `src/k1link/compute/lidar_ground.py`
- `experiments/perception/run_lidar_ground_benchmark.py`
- `src/k1link/web/lidar_api.py`
- `apps/control-station/src/workspaces/LidarQualityWorkspace.tsx`
- `docs/13_LIDAR_WORKER_PRODUCT_AND_ROADMAP.md`