feat(perception): add camera ego-motion evidence
This commit is contained in:
DCCONSTRUCTIONS
@@ -0,0 +1,261 @@
|
||||
# LAB E26 — camera ego-motion evidence and conservative LiDAR fusion
|
||||
|
||||
Date: 2026-07-24
|
||||
Status: recorded diagnostic accepted; navigation and safety acceptance are false
|
||||
Immutable replay:
|
||||
`LAB E26.1 · Camera + ego-motion · full RAVNOVES00`
|
||||
|
||||
## Objective
|
||||
|
||||
E25 proved that persistent map-frame LiDAR support can stop parked vehicles
|
||||
from looking dynamic, but it cannot recover motion when the K1 supplies too few
|
||||
current LiDAR returns. E26 adds an independent camera measurement without
|
||||
inventing metric velocity:
|
||||
|
||||
1. preserve the original detector track in image space;
|
||||
2. remove the apparent motion caused by the moving rig;
|
||||
3. test whether observations from several calibrated camera poses are
|
||||
consistent with one static point in `k1-map`;
|
||||
4. fuse the resulting relative camera state with E25 LiDAR evidence;
|
||||
5. represent disagreement and missing evidence explicitly as `unknown`;
|
||||
6. keep every unknown object occupied and outside navigation authority.
|
||||
|
||||
The experiment changes the measurement, not the visual smoothing of 3D boxes.
|
||||
Rerun remains the operator viewer. The perception result is produced before
|
||||
Rerun and can be consumed independently of it.
|
||||
|
||||
## Immutable inputs
|
||||
|
||||
- Source observation: `20260720T065719Z_viewer_live` (`RAVNOVES00`).
|
||||
- Published lab session: `lab-e26-1-camera-ego-motion`.
|
||||
- Camera: `sensor.camera.right`, factory calibration slot `camera_1`.
|
||||
- Coordinate frame: `k1-map`.
|
||||
- Frames: 4,489.
|
||||
- Source timeline: 35.421857292–484.044857292 s.
|
||||
- Published replay duration: 448.623 s.
|
||||
- Factory projection model: KB4.
|
||||
- Calibration SHA-256:
|
||||
`05f3ad9b38b3a4fc95388a8ec83da83c745e217709e51787b3d5aad0969f6fa9`.
|
||||
- Camera/detector source result:
|
||||
`e10-integrated-perception-34ade557b5636717aa497fc00355b84df7063e483a175f2f5d3f04c03df1c898`.
|
||||
- E25.3 LiDAR-evidence source result:
|
||||
`e10-integrated-perception-9034a5cf306b379cc248c1da17639f140fc896bb55e412bef43f89974de892a3`.
|
||||
- Detector: YOLOX-S, COCO-80, 640×640.
|
||||
- Semantic source:
|
||||
`tue-mps/cityscapes_semantic_eomt_large_1024`, FP16 autocast.
|
||||
- E26 profile SHA-256:
|
||||
`17668cea5a8bddd621271ed0457ca3a91e6d3957133d53213d6705d9b03d039e`.
|
||||
- E26 benchmark SHA-256:
|
||||
`c3523c8f1ca22631120a129d20c73a8429be8c908279cc1c3286661f33d5fe57`.
|
||||
|
||||
E26 does not repeat detector or segmentation inference. It reuses immutable
|
||||
model outputs, E25 LiDAR evidence, rig poses and factory calibration. It writes
|
||||
a new integrated result and hard-links immutable source payloads where
|
||||
possible. The source results and raw sensor data are not modified.
|
||||
|
||||
## Camera measurement
|
||||
|
||||
For each original 2D detector track, E26 takes a bottom-centre image point with
|
||||
a small configurable inset. The point is unprojected through the factory KB4
|
||||
model into a three-dimensional camera ray. The calibrated camera-to-rig
|
||||
extrinsic and the recorded rig pose transform that ray into `k1-map`.
|
||||
|
||||
Over a bounded two-second history, a least-squares ray intersection estimates
|
||||
the one world point that would explain the observations if the target were
|
||||
static. The tracker evaluates:
|
||||
|
||||
- median angular residual to the static-world hypothesis;
|
||||
- p80 angular residual;
|
||||
- positive-depth fraction;
|
||||
- estimated range;
|
||||
- rig baseline covered by the observations;
|
||||
- observation count, time span, detector score and box size.
|
||||
|
||||
Low residuals support `static`. A well-observed track with residuals beyond the
|
||||
class thresholds supports `dynamic`. A hypothesis behind the camera can support
|
||||
`dynamic` only under the stricter score and repeated-observation gate.
|
||||
Degenerate geometry, insufficient baseline, weak detections, missing pose or an
|
||||
invalid KB4 edge ray yields `unknown`.
|
||||
|
||||
The camera branch measures relative inconsistency with a static world
|
||||
hypothesis. It does not measure metric object velocity. Camera-only
|
||||
`speed_mps` and `velocity_map_mps` are therefore deliberately null.
|
||||
|
||||
## Bounded streaming contract
|
||||
|
||||
The implementation is a single forward pass with no lookahead:
|
||||
|
||||
- history: at most 2.0 s;
|
||||
- maximum 32 observations per camera track;
|
||||
- maximum 256 live tracks;
|
||||
- maximum idle time: 2.5 s;
|
||||
- minimum observations: 5;
|
||||
- minimum observation span: 0.4 s;
|
||||
- minimum rig baseline: 0.25 m;
|
||||
- maximum normal-matrix condition: 1,000,000;
|
||||
- minimum box diagonal: 35 px;
|
||||
- minimum median detector score: 0.60;
|
||||
- dynamic confirmation: 2 frames;
|
||||
- static confirmation: 4 frames;
|
||||
- evidence decay: 1 frame.
|
||||
|
||||
The run reached only 34 simultaneous tracks. Five unprojectable KB4 edge rays
|
||||
were rejected fail-soft and counted; detector and LiDAR objects remained in the
|
||||
result as unavailable camera evidence.
|
||||
|
||||
## Fusion contract
|
||||
|
||||
Camera and LiDAR are treated as independent evidence sources:
|
||||
|
||||
- camera and LiDAR agree: publish the common state with conservative
|
||||
confidence;
|
||||
- camera and LiDAR disagree: publish explicit `unknown`, clear velocity and
|
||||
increment the conflict counter;
|
||||
- camera only: publish relative `static` or `dynamic` with confidence scaled by
|
||||
0.65, but without metric velocity;
|
||||
- LiDAR only: preserve the metric E25 state and velocity;
|
||||
- neither source qualifies: publish `unknown`.
|
||||
|
||||
Every branch carries:
|
||||
|
||||
- `unknown_is_occupied=true`;
|
||||
- `camera_only_metric_velocity_valid=false`;
|
||||
- `navigation_or_safety_accepted=false`.
|
||||
|
||||
There is no command output and no planner authority in E26.
|
||||
|
||||
## Operator-reviewed benchmark
|
||||
|
||||
The benchmark contains eight dynamic target windows and three parked-vehicle
|
||||
control windows. The dynamic windows are bound to reviewed source detector
|
||||
track IDs, so a nearby parked object cannot satisfy a moving-object event.
|
||||
|
||||
Dynamic windows:
|
||||
|
||||
- woman with dog at 54–62 s: 24 dynamic hits, 82.76% evidence coverage;
|
||||
- person loading a car at 62–69 s: best run 4 hits over 0.896 s, 81.82% coverage;
|
||||
- adult and child at 111–120 s: 26 hits over 4.387 s, 91.49% coverage;
|
||||
- person on the road at 124–132 s: 11 hits over 1.0 s, 76.19% coverage;
|
||||
- stroller group at 151–161 s: best run 5 hits over 0.498 s, 84.0% coverage;
|
||||
- oncoming vehicle at 167–173 s: 3 hits over 0.172 s, 73.33% coverage;
|
||||
- vulnerable-road-user group at 170–179 s: best run 5 hits over 0.38 s,
|
||||
72.09% coverage;
|
||||
- same-direction vehicle at 180–188 s: 12 hits over 1.734 s, 100% coverage.
|
||||
|
||||
Parked-vehicle controls:
|
||||
|
||||
- 70–90 s: 853 source observations, 732 evidence observations,
|
||||
57.51% classified, 418 static / 311 unknown / 3 dynamic,
|
||||
false-dynamic fraction 0.00410;
|
||||
- 132–148 s: 820 source observations, 746 evidence observations,
|
||||
78.42% classified, 585 static / 161 unknown / 0 dynamic,
|
||||
false-dynamic fraction 0.0;
|
||||
- 161–166 s: 151 source observations, 120 evidence observations,
|
||||
62.5% classified, 75 static / 45 unknown / 0 dynamic,
|
||||
false-dynamic fraction 0.0.
|
||||
|
||||
Result: 11/11 benchmark windows pass.
|
||||
|
||||
This is a useful regression result, not independent ground truth. The windows
|
||||
come from operator timestamps and reviewed detector-track binding on the same
|
||||
recording used to develop the experiment. The result does not measure global
|
||||
precision, recall, ID-switch rate, distance error or safety performance outside
|
||||
those windows.
|
||||
|
||||
## Runtime and resource result
|
||||
|
||||
Camera ego-motion postprocessor:
|
||||
|
||||
- mean: 1.982 ms/frame;
|
||||
- p50: 2.059 ms/frame;
|
||||
- p95: 3.499 ms/frame;
|
||||
- maximum: 106.171 ms/frame;
|
||||
- camera observations: 20,513;
|
||||
- created tracks: 1,466;
|
||||
- peak live tracks: 34;
|
||||
- pose-unavailable frames: 561;
|
||||
- invalid camera rays: 5;
|
||||
- published camera states: 6,105 static, 1,117 dynamic, 13,291 unknown;
|
||||
- explicit camera/LiDAR conflicts: 374.
|
||||
|
||||
The p95 satisfies the configured 12 ms budget for this postprocessor. It is not
|
||||
an end-to-end real-time measurement: detector inference, semantic inference,
|
||||
sensor transport, decoding, synchronization, Rerun serialization, network
|
||||
transfer and browser rendering are not included.
|
||||
|
||||
## Replay acceptance
|
||||
|
||||
The saved session is `ready`, `replayable=true`, and visible through the single
|
||||
existing server on `http://127.0.0.1:8000/`.
|
||||
|
||||
An acceptance request through the same perception endpoint used by the browser
|
||||
returned:
|
||||
|
||||
- HTTP 200;
|
||||
- media type `application/vnd.rerun.rrd`;
|
||||
- Rerun magic `RRF2`;
|
||||
- payload: 69,735,518 bytes;
|
||||
- SHA-256:
|
||||
`0d7978468f829acb6afe2aa14f6e2f2f96d55ee35e21d8439cdd3ec3bdd107b1`;
|
||||
- cold response time: 47.172 s.
|
||||
|
||||
The 47.172 s cold Rerun materialization is an operator-path cost, not perception
|
||||
compute. It remains too slow for first connection in the field and must be
|
||||
addressed separately through prebuilt/cached, chunked or live incremental
|
||||
delivery. It does not invalidate the 3.499 ms motion-evidence p95, and the
|
||||
3.499 ms number does not excuse the 47.172 s operator wait.
|
||||
|
||||
The server was not stopped, restarted or moved to another port during
|
||||
acceptance.
|
||||
|
||||
## Validation
|
||||
|
||||
- Ruff passes for `src`, `tests` and `experiments/perception`.
|
||||
- Strict mypy passes for the E26 compute, publication, calibration and CLI
|
||||
modules.
|
||||
- 44 targeted E10/E24/E25/E26 and KB4 projection tests pass.
|
||||
- The complete repository `pytest` suite passes.
|
||||
- The on-disk implementation, profile and benchmark SHA-256 values exactly
|
||||
match those embedded in the immutable E26.1 report.
|
||||
- The saved-session API reports `ready`, `replayable=true` and the expected
|
||||
result ID.
|
||||
- The single localhost service remains available on port 8000.
|
||||
|
||||
## Conclusion
|
||||
|
||||
E26 closes the specific E25 evidence gap on the reviewed recording: moving
|
||||
people and the two moving vehicles can be detected from calibrated camera
|
||||
parallax when current LiDAR support is sparse, while parked-vehicle false
|
||||
dynamic remains near zero. The implementation is bounded and has no lookahead,
|
||||
so its state structure is compatible with a later live shadow deployment.
|
||||
|
||||
The result is still not a planner-ready world model:
|
||||
|
||||
- camera-only motion has no metric velocity;
|
||||
- 13,291 published camera observations remain unknown;
|
||||
- 374 camera/LiDAR conflicts require explicit downstream handling;
|
||||
- the benchmark is small and not independent;
|
||||
- the detector and track association still inherit COCO-domain errors;
|
||||
- no end-to-end live latency or dropped-frame envelope has been measured;
|
||||
- the cold operator replay still takes approximately 47 s.
|
||||
|
||||
## E27 gate
|
||||
|
||||
E27 should not be another threshold-tuning pass. It should turn E26 into a
|
||||
measurable world-model qualification:
|
||||
|
||||
1. create frame-level independent labels for the reviewed moving objects,
|
||||
parked controls, ID switches and visibility;
|
||||
2. quantify camera association integrity, dynamic/static precision and recall,
|
||||
time-to-confirm and conflict duration;
|
||||
3. add metric range/velocity only when LiDAR, ground contact or another
|
||||
observable depth source supports it;
|
||||
4. preserve occupancy and uncertainty independently of cuboid presentation;
|
||||
5. replay the bounded stages at sensor cadence and record end-to-end deadline
|
||||
misses, queue depth and dropped-frame policy;
|
||||
6. keep the first live step shadow-only, with no navigation or safety authority;
|
||||
7. prebuild or incrementally stream the operator product so connecting to the
|
||||
vehicle never waits for a full 69 MB Rerun serialization.
|
||||
|
||||
E26.1 is accepted as a diagnostic evidence-fusion laboratory result. It is not
|
||||
accepted for autonomous navigation, collision avoidance or safety decisions.
|
||||
@@ -0,0 +1,77 @@
|
||||
{
|
||||
"schema_version": "missioncore.e26-camera-ego-motion-profile/v1",
|
||||
"profile_id": "lab-e26-kb4-multiview-static-hypothesis-v1",
|
||||
"mode": "recorded-streaming-qualification",
|
||||
"source": {
|
||||
"source_id": "sensor.camera.right",
|
||||
"coordinate_frame": "k1-map",
|
||||
"calibration_slot": "camera_1",
|
||||
"calibration_sha256": "05f3ad9b38b3a4fc95388a8ec83da83c745e217709e51787b3d5aad0969f6fa9"
|
||||
},
|
||||
"camera_evidence": {
|
||||
"history_seconds": 2.0,
|
||||
"maximum_observations_per_track": 32,
|
||||
"minimum_observations": 5,
|
||||
"minimum_span_seconds": 0.4,
|
||||
"minimum_ego_baseline_m": 0.25,
|
||||
"maximum_normal_matrix_condition": 1000000.0,
|
||||
"footpoint_inset_fraction": 0.03,
|
||||
"minimum_bbox_diagonal_px": 35.0,
|
||||
"minimum_median_detector_score": 0.6,
|
||||
"minimum_negative_depth_detector_score": 0.7,
|
||||
"minimum_valid_positive_depth_fraction": 0.8,
|
||||
"negative_depth_dynamic_fraction": 0.6,
|
||||
"minimum_static_hypothesis_range_m": 0.8,
|
||||
"maximum_static_hypothesis_range_m": 80.0,
|
||||
"dynamic_median_angular_residual_degrees": {
|
||||
"person": 1.8,
|
||||
"bicycle": 1.8,
|
||||
"motorcycle": 1.8,
|
||||
"vehicle": 3.5,
|
||||
"default": 3.5
|
||||
},
|
||||
"dynamic_p80_angular_residual_degrees": {
|
||||
"person": 2.8,
|
||||
"bicycle": 2.8,
|
||||
"motorcycle": 2.8,
|
||||
"vehicle": 5.0,
|
||||
"default": 5.0
|
||||
},
|
||||
"static_median_angular_residual_degrees": {
|
||||
"person": 1.0,
|
||||
"bicycle": 1.0,
|
||||
"motorcycle": 1.0,
|
||||
"vehicle": 1.4,
|
||||
"default": 1.4
|
||||
},
|
||||
"static_p80_angular_residual_degrees": {
|
||||
"person": 2.0,
|
||||
"bicycle": 2.0,
|
||||
"motorcycle": 2.0,
|
||||
"vehicle": 2.5,
|
||||
"default": 2.5
|
||||
},
|
||||
"dynamic_confirmation_frames": 2,
|
||||
"static_confirmation_frames": 4,
|
||||
"evidence_decay_frames": 1
|
||||
},
|
||||
"fusion": {
|
||||
"camera_only_confidence_scale": 0.65,
|
||||
"conflict_state": "unknown",
|
||||
"unknown_is_occupied": true,
|
||||
"camera_only_metric_velocity_valid": false
|
||||
},
|
||||
"bounds": {
|
||||
"maximum_tracks": 256,
|
||||
"maximum_track_idle_seconds": 2.5
|
||||
},
|
||||
"acceptance": {
|
||||
"maximum_processing_p95_ms": 12.0,
|
||||
"maximum_tracks_observed": 256,
|
||||
"maximum_static_control_false_dynamic_fraction": 0.1
|
||||
},
|
||||
"authority": {
|
||||
"commands_enabled": false,
|
||||
"navigation_or_safety_accepted": false
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,131 @@
|
||||
{
|
||||
"schema_version": "missioncore.e26-motion-benchmark/v1",
|
||||
"benchmark_id": "ravnoves00-camera-ego-reviewed-anchors-v1",
|
||||
"source_session_id": "20260720T065719Z_viewer_live",
|
||||
"timeline": "session_seconds",
|
||||
"annotation_status": "operator-approximate-with-reviewed-detector-track-binding",
|
||||
"events": [
|
||||
{
|
||||
"id": "dynamic-person-woman-dog-left-58s",
|
||||
"kind": "target-motion",
|
||||
"window_seconds": [54.0, 62.0],
|
||||
"class_group": "person",
|
||||
"target_source_track_ids": [83],
|
||||
"expected_motion": "dynamic",
|
||||
"minimum_hits": 3,
|
||||
"minimum_span_seconds": 0.2,
|
||||
"minimum_coverage_fraction": 0.08
|
||||
},
|
||||
{
|
||||
"id": "dynamic-person-loading-car-right-65s",
|
||||
"kind": "target-motion",
|
||||
"window_seconds": [62.0, 69.0],
|
||||
"class_group": "person",
|
||||
"target_source_track_ids": [112],
|
||||
"expected_motion": "dynamic",
|
||||
"minimum_hits": 3,
|
||||
"minimum_span_seconds": 0.2,
|
||||
"minimum_coverage_fraction": 0.1
|
||||
},
|
||||
{
|
||||
"id": "static-vehicles-before-moving-70-90s",
|
||||
"kind": "static-control",
|
||||
"window_seconds": [70.0, 90.0],
|
||||
"class_group": "vehicle",
|
||||
"expected_motion": "static",
|
||||
"minimum_source_observations": 80,
|
||||
"minimum_coverage_fraction": 0.2,
|
||||
"minimum_classified_fraction": 0.2,
|
||||
"maximum_false_dynamic_fraction": 0.1
|
||||
},
|
||||
{
|
||||
"id": "dynamic-person-adult-child-left-115s",
|
||||
"kind": "target-motion",
|
||||
"window_seconds": [111.0, 120.0],
|
||||
"class_group": "person",
|
||||
"target_source_track_ids": [229],
|
||||
"expected_motion": "dynamic",
|
||||
"minimum_hits": 3,
|
||||
"minimum_span_seconds": 0.2,
|
||||
"minimum_coverage_fraction": 0.08
|
||||
},
|
||||
{
|
||||
"id": "dynamic-person-road-right-127s",
|
||||
"kind": "target-motion",
|
||||
"window_seconds": [124.0, 132.0],
|
||||
"class_group": "person",
|
||||
"target_source_track_ids": [253],
|
||||
"expected_motion": "dynamic",
|
||||
"minimum_hits": 2,
|
||||
"minimum_span_seconds": 0.15,
|
||||
"minimum_coverage_fraction": 0.08
|
||||
},
|
||||
{
|
||||
"id": "static-vehicles-mid-run-132-148s",
|
||||
"kind": "static-control",
|
||||
"window_seconds": [132.0, 148.0],
|
||||
"class_group": "vehicle",
|
||||
"expected_motion": "static",
|
||||
"minimum_source_observations": 80,
|
||||
"minimum_coverage_fraction": 0.2,
|
||||
"minimum_classified_fraction": 0.2,
|
||||
"maximum_false_dynamic_fraction": 0.1
|
||||
},
|
||||
{
|
||||
"id": "dynamic-person-stroller-group-left-155s",
|
||||
"kind": "target-motion",
|
||||
"window_seconds": [151.0, 161.0],
|
||||
"class_group": "person",
|
||||
"target_source_track_ids": [322, 328],
|
||||
"expected_motion": "dynamic",
|
||||
"minimum_hits": 3,
|
||||
"minimum_span_seconds": 0.2,
|
||||
"minimum_coverage_fraction": 0.08
|
||||
},
|
||||
{
|
||||
"id": "static-vehicles-before-oncoming-161-166s",
|
||||
"kind": "static-control",
|
||||
"window_seconds": [161.0, 166.0],
|
||||
"class_group": "vehicle",
|
||||
"expected_motion": "static",
|
||||
"minimum_source_observations": 25,
|
||||
"minimum_coverage_fraction": 0.15,
|
||||
"minimum_classified_fraction": 0.15,
|
||||
"maximum_false_dynamic_fraction": 0.1
|
||||
},
|
||||
{
|
||||
"id": "dynamic-vehicle-oncoming-right-170s",
|
||||
"kind": "target-motion",
|
||||
"window_seconds": [167.0, 173.0],
|
||||
"class_group": "vehicle",
|
||||
"target_source_track_ids": [365],
|
||||
"expected_motion": "dynamic",
|
||||
"minimum_hits": 3,
|
||||
"minimum_span_seconds": 0.15,
|
||||
"minimum_coverage_fraction": 0.1
|
||||
},
|
||||
{
|
||||
"id": "dynamic-person-bike-group-left-173s",
|
||||
"kind": "target-motion",
|
||||
"window_seconds": [170.0, 179.0],
|
||||
"class_group": "vulnerable_road_user",
|
||||
"target_source_track_ids": [379, 386, 388],
|
||||
"expected_motion": "dynamic",
|
||||
"minimum_hits": 3,
|
||||
"minimum_span_seconds": 0.2,
|
||||
"minimum_coverage_fraction": 0.08
|
||||
},
|
||||
{
|
||||
"id": "dynamic-vehicle-same-direction-right-183s",
|
||||
"kind": "target-motion",
|
||||
"window_seconds": [180.0, 188.0],
|
||||
"class_group": "vehicle",
|
||||
"target_source_track_ids": [373],
|
||||
"binding_review": "frame-and-2d-track-reviewed; 391 is a parked vehicle",
|
||||
"expected_motion": "dynamic",
|
||||
"minimum_hits": 3,
|
||||
"minimum_span_seconds": 0.2,
|
||||
"minimum_coverage_fraction": 0.1
|
||||
}
|
||||
]
|
||||
}
|
||||
Reference in New Issue
Block a user