feat(perception): add persistent lidar motion evidence
This commit is contained in:
DCCONSTRUCTIONS
@@ -0,0 +1,273 @@
|
||||
# LAB E25 — persistent LiDAR support and honest motion evidence
|
||||
|
||||
Date: 2026-07-24
|
||||
Status: diagnostic artifact accepted; motion benchmark not accepted
|
||||
Best immutable replay:
|
||||
`LAB E25.3 · Persistent support · full RAVNOVES00`
|
||||
|
||||
## Objective
|
||||
|
||||
Test a different motion measurement after E24 proved that the center of a
|
||||
completed 3D cuboid is not a stable world-space observation. E25 tracks the
|
||||
LiDAR cells that actually support an object in `k1-map`, compares their
|
||||
occupancy through time and emits `static`, `dynamic`, or `unknown`.
|
||||
|
||||
This is not a box beautification experiment. The target is a bounded,
|
||||
streaming-equivalent evidence branch for an unmanned vehicle. 3D boxes remain
|
||||
an operator-facing presentation. Missing sensor evidence must remain
|
||||
`unknown/occupied`; it must not be converted into an invented motion estimate
|
||||
or free space.
|
||||
|
||||
## Source data and immutable inputs
|
||||
|
||||
- Source session: `20260720T065719Z_viewer_live` (`RAVNOVES00`).
|
||||
- Camera source: `sensor.camera.right`, factory slot `camera_1`.
|
||||
- Source frames: 4,489.
|
||||
- Source timeline: 35.421857292–484.044857292 session seconds.
|
||||
- Full map-frame LiDAR pack: 9,207,270 points.
|
||||
- Semantic masks: 898 frames at 600×800.
|
||||
- Factory camera model: KB4.
|
||||
- Calibration SHA-256:
|
||||
`05f3ad9b38b3a4fc95388a8ec83da83c745e217709e51787b3d5aad0969f6fa9`.
|
||||
- Source integrated result:
|
||||
`e10-integrated-perception-34ade557b5636717aa497fc00355b84df7063e483a175f2f5d3f04c03df1c898`.
|
||||
- Source detector: YOLOX-S, COCO-80, 640×640.
|
||||
- Source semantic model:
|
||||
`tue-mps/cityscapes_semantic_eomt_large_1024`, FP16 autocast.
|
||||
|
||||
E25 does not repeat GPU inference. It reuses the accepted detector and semantic
|
||||
outputs and recomputes object support from the immutable full LiDAR pack,
|
||||
factory projection and rig poses. Raw LiDAR, ground returns, camera media,
|
||||
calibration, detector output and semantic masks were not modified.
|
||||
|
||||
## Why E25 was required
|
||||
|
||||
E24 classified motion from the temporal behavior of fitted cuboid centers.
|
||||
That measurement is biased by visibility: while the rig passes a parked car,
|
||||
different portions of the car become visible, and an amodal box fitter can move
|
||||
its completed center even though the physical object is static.
|
||||
|
||||
This creates a structurally false velocity. More smoothing only delays it.
|
||||
Looser association hides missing observations but can join neighboring cars.
|
||||
Stricter association preserves identity but exposes gaps. E25 therefore changes
|
||||
the measurement rather than tuning another center filter.
|
||||
|
||||
## Implemented pipeline
|
||||
|
||||
For every source detector object, E25:
|
||||
|
||||
1. takes the synchronized full LiDAR scan in `k1-map`;
|
||||
2. projects candidate points through the factory KB4 camera calibration;
|
||||
3. applies detector-box inset and compatible semantic labels;
|
||||
4. performs depth and spatial clustering to reject unrelated projected points;
|
||||
5. excludes ground only from the object-proposal branch while preserving the
|
||||
complete ground and raw point cloud for visualization and downstream use;
|
||||
6. quantizes object support into 0.25 m map-frame occupancy cells;
|
||||
7. stores a bounded history of occupied cells and support points;
|
||||
8. compares the current support with a reference 0.3–2.5 s in the past;
|
||||
9. classifies static evidence from persistent overlap;
|
||||
10. classifies dynamic evidence only when overlap is low and displacement,
|
||||
robust speed, direction consistency and repeated current observations agree.
|
||||
|
||||
The E25 branch is deliberately bounded:
|
||||
|
||||
- maximum 192 live tracks;
|
||||
- maximum 32 support snapshots per track;
|
||||
- maximum 256 points and 256 cells per object snapshot;
|
||||
- no lookahead;
|
||||
- no commands, navigation authority or safety authority.
|
||||
|
||||
The final pass also enforces an important evidence invariant: one current LiDAR
|
||||
measurement can belong only to the current `e24-observed` object. An E24
|
||||
`held-hypothesis` may preserve a short-lived track state, but it receives
|
||||
`current=false` and cannot duplicate the same present-time support.
|
||||
|
||||
## Benchmark contract
|
||||
|
||||
Benchmark v2 is bound to source detector IDs and time windows rather than to
|
||||
any nearby object of the expected class. This prevents a parked car from
|
||||
accidentally satisfying a moving-car event.
|
||||
|
||||
Dynamic operator anchors:
|
||||
|
||||
- 58 s: woman with dog, left;
|
||||
- 65 s: person loading a car, right;
|
||||
- 115 s: adult and child, left;
|
||||
- 127 s: pedestrian on road, right;
|
||||
- 155 s: woman with stroller and child, left;
|
||||
- 170 s: oncoming vehicle, right;
|
||||
- 173 s: vulnerable-road-user group, left;
|
||||
- 183 s: vehicle moving with the rig direction, right.
|
||||
|
||||
Static controls:
|
||||
|
||||
- parked vehicles at 70–90 s;
|
||||
- parked vehicles at 132–148 s;
|
||||
- parked vehicles at 161–166 s.
|
||||
|
||||
Frame review corrected the 183 s anchor from source track 391, which is parked,
|
||||
to source track 373, which is the moving vehicle. Static controls report
|
||||
coverage, classified fraction and false-dynamic fraction separately. A lack of
|
||||
classification is therefore not counted as a correct static result.
|
||||
|
||||
## Run series
|
||||
|
||||
### E25.1 — first persistent-support pass
|
||||
|
||||
- Immutable session: `lab-e25-1-persistent-support`.
|
||||
- Result:
|
||||
`e10-integrated-perception-fac9421996ce5d8622c534f66af1a9bfaea25c7ea8a2bfd1ec954330d12ce9ed`.
|
||||
- Benchmark: 7/11.
|
||||
- Processing p95: 3.966 ms/frame.
|
||||
|
||||
An initial CPU-bound attempt was stopped because the NPZ container was
|
||||
decompressing LiDAR members repeatedly for every frame. This was a laboratory
|
||||
loader defect, not tracker cost. Materializing the immutable arrays once in
|
||||
memory reduced the full pass to about 20 seconds.
|
||||
|
||||
### E25.2 — benchmark identity correction
|
||||
|
||||
- Immutable session: `lab-e25-2-persistent-support`.
|
||||
- Result:
|
||||
`e10-integrated-perception-e24715571eb5c40d6f0a853bb0551737b2935e0ccbb65e2b90061c0f422a49e3`.
|
||||
- Benchmark: 7/11.
|
||||
- Processing p95: 3.970 ms/frame.
|
||||
|
||||
This pass binds the 183 s event to moving source track 373 and adds defensive
|
||||
deduplication for source/frame evidence in the evaluator.
|
||||
|
||||
### E25.3 — single-owner current evidence
|
||||
|
||||
- Immutable session: `lab-e25-3-persistent-support`.
|
||||
- Result:
|
||||
`e10-integrated-perception-9034a5cf306b379cc248c1da17639f140fc896bb55e412bef43f89974de892a3`.
|
||||
- Benchmark: 6/11.
|
||||
- Processing mean/p50/p95:
|
||||
2.114/2.078/4.322 ms per frame.
|
||||
- Processing maximum: 84.791 ms/frame.
|
||||
- Peak live tracker states: 16 of 192.
|
||||
- Current support observations: 7,204.
|
||||
- Explicit held states: 3,073.
|
||||
- Full laboratory packaging: approximately 20.5 s.
|
||||
|
||||
The score decreased from 7/11 to 6/11 because held hypotheses no longer reuse
|
||||
current LiDAR evidence. This is the accepted result: it is less flattering and
|
||||
more correct.
|
||||
|
||||
## E25.3 event results
|
||||
|
||||
Passed:
|
||||
|
||||
- woman/dog at 58 s;
|
||||
- adult/child at 115 s;
|
||||
- vulnerable-road-user group at 173 s;
|
||||
- all three parked-vehicle control windows.
|
||||
|
||||
Static control details:
|
||||
|
||||
- 70–90 s: coverage 0.4619, classified fraction 0.77665,
|
||||
306 static, 88 unknown, false-dynamic fraction 0.0;
|
||||
- 132–148 s: coverage 0.5683, classified fraction 0.63948,
|
||||
298 static, 168 unknown, false-dynamic fraction 0.0;
|
||||
- 161–166 s: coverage 0.1722, classified fraction 0.61538,
|
||||
16 static, 10 unknown, false-dynamic fraction 0.0.
|
||||
|
||||
Failed:
|
||||
|
||||
- person loading car at 65 s: two current dynamic hits, minimum is three;
|
||||
- right-side person at 127 s: only 3/21 frames contain usable LiDAR support;
|
||||
- stroller group at 155 s: support exists, but association is ambiguous and
|
||||
forks across source tracks;
|
||||
- oncoming car at 170 s: only 2/15 current LiDAR-support observations, so no
|
||||
reliable temporal history can be formed;
|
||||
- same-direction car at 183 s, source track 373: the available evidence is
|
||||
strong (`overlap=0`, approximately 4.8–4.9 m/s), but only two current dynamic
|
||||
hits remain after held-evidence duplication is removed.
|
||||
|
||||
## Performance and replay
|
||||
|
||||
The tracker timing is CPU diagnostic timing without repeated detector or
|
||||
semantic inference. Its p95 of 4.322 ms/frame is compatible with a future
|
||||
near-real-time evidence branch, but it does not prove end-to-end real-time
|
||||
performance.
|
||||
|
||||
The first cold Rerun publication exposed a fail-soft viewer bug: a provisional
|
||||
cuboid can legitimately have `distance_smoothed_m=null`, while the label
|
||||
renderer called `float(None)`. The renderer now keeps the cuboid and simply
|
||||
omits an unconfirmed range from its label.
|
||||
|
||||
The final E25.3 overlay:
|
||||
|
||||
- Rerun magic: `RRF2`;
|
||||
- payload: 69,809,981 bytes;
|
||||
- saved session: `ready`;
|
||||
- replayable: `true`;
|
||||
- warm endpoint response on the existing localhost server: HTTP 200 in 0.11 s.
|
||||
|
||||
The server on `127.0.0.1:8000` was not stopped or restarted.
|
||||
|
||||
## Main conclusion
|
||||
|
||||
E25 fixes an important E24 failure: persistent map-frame LiDAR support no longer
|
||||
turns cuboid-center drift on parked cars into false motion. All static control
|
||||
windows have a false-dynamic fraction of 0.0.
|
||||
|
||||
E25 also establishes the sensor limit. A LiDAR-only support branch cannot infer
|
||||
motion when the object receives too few current returns. Threshold changes
|
||||
cannot create absent observations. Treating those gaps as a confident velocity
|
||||
would make the output visually smoother but less safe.
|
||||
|
||||
E25.3 is therefore accepted as a diagnostic and evidence-quality result, not as
|
||||
a navigation or safety motion classifier.
|
||||
|
||||
## Change of emphasis for E26
|
||||
|
||||
E26 must add information from an independent source instead of further tuning
|
||||
the LiDAR branch:
|
||||
|
||||
1. maintain a temporally stable camera track in image space;
|
||||
2. remove apparent image motion caused by rig ego-motion;
|
||||
3. turn the factory KB4 pixel ray, rig pose/extrinsic and a bounded
|
||||
ground-contact/range hypothesis into world-motion evidence;
|
||||
4. fuse that evidence conservatively with the E25 LiDAR branch;
|
||||
5. emit explicit `agree`, `single-source/unknown` and `conflict` outcomes;
|
||||
6. keep unknown objects occupied for planning;
|
||||
7. expose occupancy footprint, velocity and uncertainty to planner-facing
|
||||
consumers while retaining boxes for the operator UI.
|
||||
|
||||
Acceptance must be split into independent gates:
|
||||
|
||||
- camera-track association integrity and ID-switch rate;
|
||||
- current LiDAR-support coverage;
|
||||
- false dynamic on parked vehicles;
|
||||
- recall on the known moving people and two moving vehicles;
|
||||
- camera/LiDAR conflict rate;
|
||||
- end-to-end stage latency and bounded memory.
|
||||
|
||||
This follows the established autonomous-driving separation between detection,
|
||||
tracking, occupancy and validation. Foxglove and Rerun visualize those
|
||||
products; they do not supply the motion-estimation algorithm.
|
||||
|
||||
## Validation
|
||||
|
||||
- 30 targeted E10/E24/E25 tests pass after the viewer fail-soft fix.
|
||||
- Ruff passes for the changed modules.
|
||||
- Targeted strict mypy passes for the changed compute modules.
|
||||
- E25.3 passes the integrated-perception validator.
|
||||
- The immutable saved session is ready and replayable on
|
||||
`http://127.0.0.1:8000/`.
|
||||
- Raw evidence was not modified.
|
||||
- Navigation and safety acceptance remain explicitly false.
|
||||
|
||||
## External implementation references
|
||||
|
||||
- Autoware multi-object tracker:
|
||||
<https://autowarefoundation.github.io/autoware_universe/pr-10143/perception/autoware_multi_object_tracker/>
|
||||
- Autoware detection-by-tracker:
|
||||
<https://autowarefoundation.github.io/autoware_universe/main/perception/autoware_detection_by_tracker/>
|
||||
- Autoware probabilistic occupancy grid:
|
||||
<https://autowarefoundation.github.io/autoware_universe/pr-10077/perception/autoware_probabilistic_occupancy_grid_map/>
|
||||
- Autoware obstacle point-cloud validator:
|
||||
<https://autowarefoundation.github.io/autoware_universe/pr-10075/perception/autoware_detected_object_validation/obstacle-pointcloud-based-validator/>
|
||||
- Autoware perception architecture:
|
||||
<https://autowarefoundation.github.io/autoware-documentation/main/design/autoware-architecture-v1/components/perception/>
|
||||
@@ -0,0 +1,131 @@
|
||||
{
|
||||
"schema_version": "missioncore.e25-motion-benchmark/v1",
|
||||
"benchmark_id": "ravnoves00-spatially-bound-operator-anchors-v2",
|
||||
"source_session_id": "20260720T065719Z_viewer_live",
|
||||
"timeline": "session_seconds",
|
||||
"annotation_status": "operator-approximate-with-reviewed-detector-track-binding",
|
||||
"events": [
|
||||
{
|
||||
"id": "dynamic-person-woman-dog-left-58s",
|
||||
"kind": "target-motion",
|
||||
"window_seconds": [54.0, 62.0],
|
||||
"class_group": "person",
|
||||
"target_source_track_ids": [83],
|
||||
"expected_motion": "dynamic",
|
||||
"minimum_hits": 3,
|
||||
"minimum_span_seconds": 0.2,
|
||||
"minimum_coverage_fraction": 0.08
|
||||
},
|
||||
{
|
||||
"id": "dynamic-person-loading-car-right-65s",
|
||||
"kind": "target-motion",
|
||||
"window_seconds": [62.0, 69.0],
|
||||
"class_group": "person",
|
||||
"target_source_track_ids": [112],
|
||||
"expected_motion": "dynamic",
|
||||
"minimum_hits": 3,
|
||||
"minimum_span_seconds": 0.2,
|
||||
"minimum_coverage_fraction": 0.1
|
||||
},
|
||||
{
|
||||
"id": "static-vehicles-before-moving-70-90s",
|
||||
"kind": "static-control",
|
||||
"window_seconds": [70.0, 90.0],
|
||||
"class_group": "vehicle",
|
||||
"expected_motion": "static",
|
||||
"minimum_source_observations": 80,
|
||||
"minimum_coverage_fraction": 0.2,
|
||||
"minimum_classified_fraction": 0.2,
|
||||
"maximum_false_dynamic_fraction": 0.1
|
||||
},
|
||||
{
|
||||
"id": "dynamic-person-adult-child-left-115s",
|
||||
"kind": "target-motion",
|
||||
"window_seconds": [111.0, 120.0],
|
||||
"class_group": "person",
|
||||
"target_source_track_ids": [229],
|
||||
"expected_motion": "dynamic",
|
||||
"minimum_hits": 3,
|
||||
"minimum_span_seconds": 0.2,
|
||||
"minimum_coverage_fraction": 0.08
|
||||
},
|
||||
{
|
||||
"id": "dynamic-person-road-right-127s",
|
||||
"kind": "target-motion",
|
||||
"window_seconds": [124.0, 132.0],
|
||||
"class_group": "person",
|
||||
"target_source_track_ids": [253],
|
||||
"expected_motion": "dynamic",
|
||||
"minimum_hits": 2,
|
||||
"minimum_span_seconds": 0.15,
|
||||
"minimum_coverage_fraction": 0.08
|
||||
},
|
||||
{
|
||||
"id": "static-vehicles-mid-run-132-148s",
|
||||
"kind": "static-control",
|
||||
"window_seconds": [132.0, 148.0],
|
||||
"class_group": "vehicle",
|
||||
"expected_motion": "static",
|
||||
"minimum_source_observations": 80,
|
||||
"minimum_coverage_fraction": 0.2,
|
||||
"minimum_classified_fraction": 0.2,
|
||||
"maximum_false_dynamic_fraction": 0.1
|
||||
},
|
||||
{
|
||||
"id": "dynamic-person-stroller-group-left-155s",
|
||||
"kind": "target-motion",
|
||||
"window_seconds": [151.0, 161.0],
|
||||
"class_group": "person",
|
||||
"target_source_track_ids": [322, 328],
|
||||
"expected_motion": "dynamic",
|
||||
"minimum_hits": 3,
|
||||
"minimum_span_seconds": 0.2,
|
||||
"minimum_coverage_fraction": 0.08
|
||||
},
|
||||
{
|
||||
"id": "static-vehicles-before-oncoming-161-166s",
|
||||
"kind": "static-control",
|
||||
"window_seconds": [161.0, 166.0],
|
||||
"class_group": "vehicle",
|
||||
"expected_motion": "static",
|
||||
"minimum_source_observations": 25,
|
||||
"minimum_coverage_fraction": 0.15,
|
||||
"minimum_classified_fraction": 0.15,
|
||||
"maximum_false_dynamic_fraction": 0.1
|
||||
},
|
||||
{
|
||||
"id": "dynamic-vehicle-oncoming-right-170s",
|
||||
"kind": "target-motion",
|
||||
"window_seconds": [167.0, 173.0],
|
||||
"class_group": "vehicle",
|
||||
"target_source_track_ids": [365],
|
||||
"expected_motion": "dynamic",
|
||||
"minimum_hits": 3,
|
||||
"minimum_span_seconds": 0.15,
|
||||
"minimum_coverage_fraction": 0.1
|
||||
},
|
||||
{
|
||||
"id": "dynamic-person-bike-group-left-173s",
|
||||
"kind": "target-motion",
|
||||
"window_seconds": [170.0, 179.0],
|
||||
"class_group": "vulnerable_road_user",
|
||||
"target_source_track_ids": [379, 386, 388],
|
||||
"expected_motion": "dynamic",
|
||||
"minimum_hits": 3,
|
||||
"minimum_span_seconds": 0.2,
|
||||
"minimum_coverage_fraction": 0.08
|
||||
},
|
||||
{
|
||||
"id": "dynamic-vehicle-same-direction-right-183s",
|
||||
"kind": "target-motion",
|
||||
"window_seconds": [180.0, 188.0],
|
||||
"class_group": "vehicle",
|
||||
"target_source_track_ids": [373],
|
||||
"binding_review": "frame-and-2d-track-reviewed; 391 is a parked vehicle",
|
||||
"expected_motion": "dynamic",
|
||||
"minimum_hits": 3,
|
||||
"minimum_span_seconds": 0.2,
|
||||
"minimum_coverage_fraction": 0.1
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,77 @@
|
||||
{
|
||||
"schema_version": "missioncore.e25-persistent-support-profile/v1",
|
||||
"profile_id": "lab-e25-map-frame-persistent-support-v1",
|
||||
"mode": "recorded-streaming-qualification",
|
||||
"source": {
|
||||
"source_id": "sensor.camera.right",
|
||||
"coordinate_frame": "k1-map",
|
||||
"calibration_slot": "camera_1",
|
||||
"calibration_sha256": "05f3ad9b38b3a4fc95388a8ec83da83c745e217709e51787b3d5aad0969f6fa9"
|
||||
},
|
||||
"support": {
|
||||
"minimum_evidence_points": {
|
||||
"person": 2,
|
||||
"bicycle": 2,
|
||||
"motorcycle": 2,
|
||||
"vehicle": 2,
|
||||
"default": 2
|
||||
},
|
||||
"maximum_points_per_object": 256
|
||||
},
|
||||
"occupancy": {
|
||||
"voxel_size_m": 0.25,
|
||||
"neighbor_radius_cells": 1,
|
||||
"reference_minimum_age_seconds": {
|
||||
"person": 0.3,
|
||||
"bicycle": 0.3,
|
||||
"motorcycle": 0.3,
|
||||
"vehicle": 0.55,
|
||||
"default": 0.55
|
||||
},
|
||||
"reference_maximum_age_seconds": 2.5,
|
||||
"history_seconds": 2.0,
|
||||
"static_overlap_enter": 0.55,
|
||||
"static_overlap_exit": 0.35,
|
||||
"dynamic_overlap_enter": 0.25,
|
||||
"dynamic_overlap_exit": 0.4,
|
||||
"dynamic_minimum_speed_mps": {
|
||||
"person": 0.4,
|
||||
"bicycle": 0.5,
|
||||
"motorcycle": 0.6,
|
||||
"vehicle": 0.7,
|
||||
"default": 0.7
|
||||
},
|
||||
"dynamic_minimum_displacement_m": {
|
||||
"person": 0.4,
|
||||
"bicycle": 0.45,
|
||||
"motorcycle": 0.55,
|
||||
"vehicle": 0.65,
|
||||
"default": 0.65
|
||||
},
|
||||
"minimum_direction_consistency": 0.6,
|
||||
"dynamic_confirmation_frames": {
|
||||
"person": 2,
|
||||
"bicycle": 2,
|
||||
"motorcycle": 2,
|
||||
"vehicle": 5,
|
||||
"default": 5
|
||||
},
|
||||
"static_confirmation_frames": 4,
|
||||
"contradiction_frames": 2,
|
||||
"maximum_support_hold_seconds": 0.45,
|
||||
"maximum_snapshots_per_track": 32,
|
||||
"maximum_cells_per_snapshot": 256
|
||||
},
|
||||
"bounds": {
|
||||
"maximum_tracks": 192,
|
||||
"maximum_track_idle_seconds": 1.5
|
||||
},
|
||||
"acceptance": {
|
||||
"maximum_processing_p95_ms": 12.0,
|
||||
"maximum_tracks_observed": 192
|
||||
},
|
||||
"authority": {
|
||||
"commands_enabled": false,
|
||||
"navigation_or_safety_accepted": false
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user