# LAB E26 — camera ego-motion evidence and conservative LiDAR fusion

Date: 2026-07-24
Status: recorded diagnostic accepted; navigation and safety acceptance are false
Immutable replay:
`LAB E26.1 · Camera + ego-motion · full RAVNOVES00`

## Objective

E25 proved that persistent map-frame LiDAR support can stop parked vehicles
from looking dynamic, but it cannot recover motion when the K1 supplies too few
current LiDAR returns. E26 adds an independent camera measurement without
inventing metric velocity:

1. preserve the original detector track in image space;
2. remove the apparent motion caused by the moving rig;
3. test whether observations from several calibrated camera poses are
   consistent with one static point in `k1-map`;
4. fuse the resulting relative camera state with E25 LiDAR evidence;
5. represent disagreement and missing evidence explicitly as `unknown`;
6. keep every unknown object occupied and outside navigation authority.

The experiment changes the measurement, not the visual smoothing of 3D boxes.
Rerun remains the operator viewer. The perception result is produced before
Rerun and can be consumed independently of it.

## Immutable inputs

- Source observation: `20260720T065719Z_viewer_live` (`RAVNOVES00`).
- Published lab session: `lab-e26-1-camera-ego-motion`.
- Camera: `sensor.camera.right`, factory calibration slot `camera_1`.
- Coordinate frame: `k1-map`.
- Frames: 4,489.
- Source timeline: 35.421857292–484.044857292 s.
- Published replay duration: 448.623 s.
- Factory projection model: KB4.
- Calibration SHA-256:
  `05f3ad9b38b3a4fc95388a8ec83da83c745e217709e51787b3d5aad0969f6fa9`.
- Camera/detector source result:
  `e10-integrated-perception-34ade557b5636717aa497fc00355b84df7063e483a175f2f5d3f04c03df1c898`.
- E25.3 LiDAR-evidence source result:
  `e10-integrated-perception-9034a5cf306b379cc248c1da17639f140fc896bb55e412bef43f89974de892a3`.
- Detector: YOLOX-S, COCO-80, 640×640.
- Semantic source:
  `tue-mps/cityscapes_semantic_eomt_large_1024`, FP16 autocast.
- E26 profile SHA-256:
  `17668cea5a8bddd621271ed0457ca3a91e6d3957133d53213d6705d9b03d039e`.
- E26 benchmark SHA-256:
  `c3523c8f1ca22631120a129d20c73a8429be8c908279cc1c3286661f33d5fe57`.

E26 does not repeat detector or segmentation inference. It reuses immutable
model outputs, E25 LiDAR evidence, rig poses and factory calibration. It writes
a new integrated result and hard-links immutable source payloads where
possible. The source results and raw sensor data are not modified.

## Camera measurement

For each original 2D detector track, E26 takes a bottom-centre image point with
a small configurable inset. The point is unprojected through the factory KB4
model into a three-dimensional camera ray. The calibrated camera-to-rig
extrinsic and the recorded rig pose transform that ray into `k1-map`.

Over a bounded two-second history, a least-squares ray intersection estimates
the one world point that would explain the observations if the target were
static. The tracker evaluates:

- median angular residual to the static-world hypothesis;
- p80 angular residual;
- positive-depth fraction;
- estimated range;
- rig baseline covered by the observations;
- observation count, time span, detector score and box size.

Low residuals support `static`. A well-observed track with residuals beyond the
class thresholds supports `dynamic`. A hypothesis behind the camera can support
`dynamic` only under the stricter score and repeated-observation gate.
Degenerate geometry, insufficient baseline, weak detections, missing pose or an
invalid KB4 edge ray yields `unknown`.

The camera branch measures relative inconsistency with a static world
hypothesis. It does not measure metric object velocity. Camera-only
`speed_mps` and `velocity_map_mps` are therefore deliberately null.

## Bounded streaming contract

The implementation is a single forward pass with no lookahead:

- history: at most 2.0 s;
- maximum 32 observations per camera track;
- maximum 256 live tracks;
- maximum idle time: 2.5 s;
- minimum observations: 5;
- minimum observation span: 0.4 s;
- minimum rig baseline: 0.25 m;
- maximum normal-matrix condition: 1,000,000;
- minimum box diagonal: 35 px;
- minimum median detector score: 0.60;
- dynamic confirmation: 2 frames;
- static confirmation: 4 frames;
- evidence decay: 1 frame.

The run reached only 34 simultaneous tracks. Five unprojectable KB4 edge rays
were rejected fail-soft and counted; detector and LiDAR objects remained in the
result as unavailable camera evidence.

## Fusion contract

Camera and LiDAR are treated as independent evidence sources:

- camera and LiDAR agree: publish the common state with conservative
  confidence;
- camera and LiDAR disagree: publish explicit `unknown`, clear velocity and
  increment the conflict counter;
- camera only: publish relative `static` or `dynamic` with confidence scaled by
  0.65, but without metric velocity;
- LiDAR only: preserve the metric E25 state and velocity;
- neither source qualifies: publish `unknown`.

Every branch carries:

- `unknown_is_occupied=true`;
- `camera_only_metric_velocity_valid=false`;
- `navigation_or_safety_accepted=false`.

There is no command output and no planner authority in E26.

## Operator-reviewed benchmark

The benchmark contains eight dynamic target windows and three parked-vehicle
control windows. The dynamic windows are bound to reviewed source detector
track IDs, so a nearby parked object cannot satisfy a moving-object event.

Dynamic windows:

- woman with dog at 54–62 s: 24 dynamic hits, 82.76% evidence coverage;
- person loading a car at 62–69 s: best run 4 hits over 0.896 s, 81.82% coverage;
- adult and child at 111–120 s: 26 hits over 4.387 s, 91.49% coverage;
- person on the road at 124–132 s: 11 hits over 1.0 s, 76.19% coverage;
- stroller group at 151–161 s: best run 5 hits over 0.498 s, 84.0% coverage;
- oncoming vehicle at 167–173 s: 3 hits over 0.172 s, 73.33% coverage;
- vulnerable-road-user group at 170–179 s: best run 5 hits over 0.38 s,
  72.09% coverage;
- same-direction vehicle at 180–188 s: 12 hits over 1.734 s, 100% coverage.

Parked-vehicle controls:

- 70–90 s: 853 source observations, 732 evidence observations,
  57.51% classified, 418 static / 311 unknown / 3 dynamic,
  false-dynamic fraction 0.00410;
- 132–148 s: 820 source observations, 746 evidence observations,
  78.42% classified, 585 static / 161 unknown / 0 dynamic,
  false-dynamic fraction 0.0;
- 161–166 s: 151 source observations, 120 evidence observations,
  62.5% classified, 75 static / 45 unknown / 0 dynamic,
  false-dynamic fraction 0.0.

Result: 11/11 benchmark windows pass.

This is a useful regression result, not independent ground truth. The windows
come from operator timestamps and reviewed detector-track binding on the same
recording used to develop the experiment. The result does not measure global
precision, recall, ID-switch rate, distance error or safety performance outside
those windows.

## Runtime and resource result

Camera ego-motion postprocessor:

- mean: 1.982 ms/frame;
- p50: 2.059 ms/frame;
- p95: 3.499 ms/frame;
- maximum: 106.171 ms/frame;
- camera observations: 20,513;
- created tracks: 1,466;
- peak live tracks: 34;
- pose-unavailable frames: 561;
- invalid camera rays: 5;
- published camera states: 6,105 static, 1,117 dynamic, 13,291 unknown;
- explicit camera/LiDAR conflicts: 374.

The p95 satisfies the configured 12 ms budget for this postprocessor. It is not
an end-to-end real-time measurement: detector inference, semantic inference,
sensor transport, decoding, synchronization, Rerun serialization, network
transfer and browser rendering are not included.

## Replay acceptance

The saved session is `ready`, `replayable=true`, and visible through the single
existing server on `http://127.0.0.1:8000/`.

An acceptance request through the same perception endpoint used by the browser
returned:

- HTTP 200;
- media type `application/vnd.rerun.rrd`;
- Rerun magic `RRF2`;
- payload: 69,735,518 bytes;
- SHA-256:
  `0d7978468f829acb6afe2aa14f6e2f2f96d55ee35e21d8439cdd3ec3bdd107b1`;
- cold response time: 47.172 s.

The 47.172 s cold Rerun materialization is an operator-path cost, not perception
compute. It remains too slow for first connection in the field and must be
addressed separately through prebuilt/cached, chunked or live incremental
delivery. It does not invalidate the 3.499 ms motion-evidence p95, and the
3.499 ms number does not excuse the 47.172 s operator wait.

The server was not stopped, restarted or moved to another port during
acceptance.

## Validation

- Ruff passes for `src`, `tests` and `experiments/perception`.
- Strict mypy passes for the E26 compute, publication, calibration and CLI
  modules.
- 44 targeted E10/E24/E25/E26 and KB4 projection tests pass.
- The complete repository `pytest` suite passes.
- The on-disk implementation, profile and benchmark SHA-256 values exactly
  match those embedded in the immutable E26.1 report.
- The saved-session API reports `ready`, `replayable=true` and the expected
  result ID.
- The single localhost service remains available on port 8000.

## Conclusion

E26 closes the specific E25 evidence gap on the reviewed recording: moving
people and the two moving vehicles can be detected from calibrated camera
parallax when current LiDAR support is sparse, while parked-vehicle false
dynamic remains near zero. The implementation is bounded and has no lookahead,
so its state structure is compatible with a later live shadow deployment.

The result is still not a planner-ready world model:

- camera-only motion has no metric velocity;
- 13,291 published camera observations remain unknown;
- 374 camera/LiDAR conflicts require explicit downstream handling;
- the benchmark is small and not independent;
- the detector and track association still inherit COCO-domain errors;
- no end-to-end live latency or dropped-frame envelope has been measured;
- the cold operator replay still takes approximately 47 s.

## E27 gate

E27 should not be another threshold-tuning pass. It should turn E26 into a
measurable world-model qualification:

1. create frame-level independent labels for the reviewed moving objects,
   parked controls, ID switches and visibility;
2. quantify camera association integrity, dynamic/static precision and recall,
   time-to-confirm and conflict duration;
3. add metric range/velocity only when LiDAR, ground contact or another
   observable depth source supports it;
4. preserve occupancy and uncertainty independently of cuboid presentation;
5. replay the bounded stages at sensor cadence and record end-to-end deadline
   misses, queue depth and dropped-frame policy;
6. keep the first live step shadow-only, with no navigation or safety authority;
7. prebuild or incrementally stream the operator product so connecting to the
   vehicle never waits for a full 69 MB Rerun serialization.

E26.1 is accepted as a diagnostic evidence-fusion laboratory result. It is not
accepted for autonomous navigation, collision avoidance or safety decisions.
