feat(lidar): add ground segmentation diagnostic benchmark
This commit is contained in:
@@ -14,6 +14,16 @@ equivalence report. The React surface reads those reports through the read-only
|
||||
`/api/v1/lidar/replay-packs` boundary; CUDA/TensorRT and ROS do not move into the
|
||||
browser.
|
||||
|
||||
`missioncore.lidar-ground-benchmark/v1` extends the same boundary for L2. It
|
||||
keeps point-aligned ground/assigned masks for the current local-percentile
|
||||
proposal and a candidate provider, exact provider/source/binary identities,
|
||||
host latency and algorithm disagreement. The first official Patchwork++ v1.4.1
|
||||
run is diagnostic-only: current K1 evidence is a vendor-map increment, not the
|
||||
sensor-centric scan and physical-height contract Patchwork++ expects. The
|
||||
read-only `/api/v1/lidar/ground-benchmarks` surface therefore publishes
|
||||
`production_promotion=false` until an independent annotation generation or an
|
||||
admitted raw scan closes the input gate.
|
||||
|
||||
## Boundary
|
||||
|
||||
```text
|
||||
|
||||
@@ -1,7 +1,8 @@
|
||||
# LiDAR worker: product value, evidence boundary and implementation roadmap
|
||||
|
||||
Date: 2026-07-25
|
||||
Status: accepted architecture plan; L0 and L1 implemented
|
||||
Status: accepted architecture plan; L0/L1 implemented; L2 diagnostic A/B complete,
|
||||
independent labels pending
|
||||
Scope: real scanner records, replay and future live shadow processing
|
||||
Explicitly out of scope: Unreal U0/U1, Gaussian assets and simulator rendering
|
||||
|
||||
@@ -111,7 +112,7 @@ and training-domain fit.
|
||||
|
||||
| Component | Correct use | Decision |
|
||||
| --- | --- | --- |
|
||||
| [Patchwork++](https://github.com/url-kaist/patchwork-plusplus) | Fast adaptive ground segmentation, including reflection-noise handling | First non-neural geometry baseline |
|
||||
| [Patchwork++](https://github.com/url-kaist/patchwork-plusplus) | Fast adaptive ground segmentation, including reflection-noise handling | v1.4.1 benchmarked; do not promote on the current vendor-map feed |
|
||||
| [Autoware CenterPoint](https://github.com/autowarefoundation/autoware_universe/tree/main/perception/autoware_lidar_centerpoint) | Mature ROS 2/TensorRT 3D detection and multi-frame reference | Second detector baseline after PointPillars |
|
||||
| [MMDetection3D](https://github.com/open-mmlab/mmdetection3d) | Training/evaluation harness and dataset adapters | Laboratory only |
|
||||
| [OpenPCDet](https://github.com/open-mmlab/OpenPCDet) | Alternative LiDAR detector benchmark/model zoo | Laboratory only; not the production runtime |
|
||||
@@ -192,17 +193,43 @@ Accepted real slice:
|
||||
The long interval tail is evidence to investigate in the scanner/transport
|
||||
quality line. It is not repaired or hidden by replay.
|
||||
|
||||
### L2 — geometric baseline
|
||||
### L2 — geometric baseline — diagnostic A/B complete, accuracy gate open
|
||||
|
||||
- [ ] Run Patchwork++ over the frozen XYZI slice.
|
||||
- [ ] Retain ground and non-ground outputs separately; never delete raw points.
|
||||
- [ ] Label a small independent ground/obstacle evaluation set in CVAT or an
|
||||
equivalent accepted annotation workspace.
|
||||
- [ ] Measure ground IoU, curb/low-obstacle recall, reflection-noise rejection
|
||||
and CPU/GPU latency.
|
||||
- [ ] Compare against the current heuristic ground proposal branch.
|
||||
- [x] Pin and run official Patchwork++ v1.4.1 at source commit
|
||||
`3e6903a1d5537a4cc2ace897b0bbb98a92d6014c`.
|
||||
- [x] Compare it against a full-frame, point-aligned extension of the current
|
||||
E19 local-percentile ground proposal.
|
||||
- [x] Retain separate current/candidate ground and assigned masks for every
|
||||
source point; raw replay remains unchanged.
|
||||
- [x] Seal latency, ground-fraction, algorithm-IoU and disagreement
|
||||
distributions in `missioncore.lidar-ground-benchmark/v1`.
|
||||
- [x] Create an immutable eight-frame annotation template in which every point
|
||||
starts as `ignore-unreviewed`; it is explicitly not ground truth.
|
||||
- [ ] Complete independent human review for ground, curb, low obstacle,
|
||||
reflection noise and other non-ground points.
|
||||
- [ ] Measure accepted ground IoU, curb/low-obstacle recall and
|
||||
reflection-noise rejection against that reviewed generation.
|
||||
|
||||
Exit: an evidence-backed decision to retain or reject Patchwork++.
|
||||
The real diagnostic run is
|
||||
`ground-benchmark-68cfd7a8f1dd4c0006183bb4f63a23f9ff1dd7459317ff0886e995f6c320d984`.
|
||||
It covers all 66 frames and 226,963 points. The current local-percentile
|
||||
proposal classified 18.31% ground at p50 with 9.74 ms p95 host latency.
|
||||
Patchwork++ classified only 0.53% ground at p50 with 0.26 ms p95 host latency.
|
||||
Their point-aligned ground IoU was 2.90% p50 and disagreement was 17.92% p50.
|
||||
These last two values compare algorithms; they are not accuracy metrics.
|
||||
|
||||
The result is an evidence-backed **do-not-promote** decision for the current K1
|
||||
feed. Patchwork++ is fast, but its input model assumes a sensor-centric scan
|
||||
and physical sensor height. K1 `lio_pcl` is a vendor-mapped increment, its
|
||||
physical sensor height is not encoded by the best-effort pose, and scan
|
||||
geometry remains unknown. Translating the cloud until Patchwork++ looks
|
||||
plausible would tune against the candidate and invalidate the comparison.
|
||||
|
||||
The independent-label exit remains open. It can be closed by reviewing the
|
||||
content-bound template
|
||||
`ground-annotation-template-12e12eab0756c14f5e06adcaf13189e93188df9bb6d31224cb9b7d566ec15b18`,
|
||||
or by acquiring an admitted raw sensor scan with known physical sensor height
|
||||
and then producing a new benchmark generation.
|
||||
|
||||
### L3 — LiDAR-native 3D detection
|
||||
|
||||
|
||||
@@ -0,0 +1,69 @@
|
||||
# ADR 0020: do not promote Patchwork++ on vendor-mapped LiDAR increments
|
||||
|
||||
Date: 2026-07-25
|
||||
Status: accepted and implemented as a diagnostic gate
|
||||
|
||||
## Context
|
||||
|
||||
The L2 roadmap selected Patchwork++ as the first non-neural ground segmentation
|
||||
baseline. The official implementation is designed around a sensor-centric
|
||||
LiDAR scan, radial zones and a physical sensor height. Current K1 `lio_pcl`
|
||||
evidence instead contains a vendor-mapped point increment in `map`, an
|
||||
independent best-effort pose and no admitted raw sweep or scan geometry. The
|
||||
pose describes vendor odometry; it does not prove the physical height of the
|
||||
LiDAR above terrain.
|
||||
|
||||
Treating a successful Patchwork++ call as a valid baseline would conflate API
|
||||
compatibility with input-domain compatibility. Choosing a synthetic Z
|
||||
translation until the output looks plausible would use the candidate itself to
|
||||
define the normalization.
|
||||
|
||||
## Decision
|
||||
|
||||
1. Pin the official Patchwork++ v1.4.1 source at commit
|
||||
`3e6903a1d5537a4cc2ace897b0bbb98a92d6014c`.
|
||||
2. Run it only behind the provider-neutral `lidar-ground/v1` diagnostic
|
||||
boundary.
|
||||
3. Bind the exact source commit and compiled binary SHA-256 into every result.
|
||||
4. Preserve point-aligned ground and assigned masks separately from raw replay.
|
||||
5. Compare it with the current local-percentile proposal using
|
||||
algorithm-to-algorithm IoU, disagreement and latency.
|
||||
6. Mark input-domain acceptance, labeled accuracy and production promotion
|
||||
false for current K1 vendor-map evidence.
|
||||
7. Never present algorithm IoU as ground IoU.
|
||||
8. Create an all-ignore, content-bound annotation template; only a separate
|
||||
human-reviewed generation may unlock ground IoU, curb/low-obstacle recall
|
||||
and reflection-noise rejection.
|
||||
9. Keep command, navigation and safety authority false.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Patchwork++ remains a useful candidate for a future raw vehicle LiDAR feed.
|
||||
- Its sub-millisecond host latency on the current slice does not compensate for
|
||||
the unaccepted input domain.
|
||||
- Current K1 work should prioritize independent labels and source evidence, not
|
||||
threshold tuning around a mis-specified sensor model.
|
||||
- The same immutable benchmark/API/React surface can compare a future raw scan,
|
||||
replay or simulation provider without moving C++ processing into React.
|
||||
|
||||
## Real diagnostic evidence
|
||||
|
||||
Benchmark
|
||||
`ground-benchmark-68cfd7a8f1dd4c0006183bb4f63a23f9ff1dd7459317ff0886e995f6c320d984`
|
||||
processed 66 frames and 226,963 points:
|
||||
|
||||
| Measurement | Current local percentile | Patchwork++ |
|
||||
| --- | ---: | ---: |
|
||||
| Ground fraction p50 | 18.31% | 0.53% |
|
||||
| Host latency p95 | 9.74 ms | 0.26 ms |
|
||||
|
||||
Algorithm ground IoU was 2.90% p50 and point disagreement was 17.92% p50.
|
||||
Neither is an accuracy metric. Independent labels are still missing.
|
||||
|
||||
## References
|
||||
|
||||
- `src/k1link/compute/lidar_ground.py`
|
||||
- `experiments/perception/run_lidar_ground_benchmark.py`
|
||||
- `src/k1link/web/lidar_api.py`
|
||||
- `apps/control-station/src/workspaces/LidarQualityWorkspace.tsx`
|
||||
- `docs/13_LIDAR_WORKER_PRODUCT_AND_ROADMAP.md`
|
||||
Reference in New Issue
Block a user