feat: add camera-first lidar geometry fusion
This commit is contained in:
DCCONSTRUCTIONS
@@ -0,0 +1,208 @@
|
||||
# LAB E29 — camera-first semantics with independent local-surface geometry
|
||||
|
||||
Date: 2026-07-26
|
||||
Status: diagnostic replay complete; operator review, navigation and safety
|
||||
acceptance are false
|
||||
Immutable result:
|
||||
`e29-camera-geometry-421a9d930638bef12cd5eb10979a477917fa4a389e655ed95f73ba4bd62e13dc`
|
||||
|
||||
## Decision under test
|
||||
|
||||
E29 implements the product boundary selected after E28:
|
||||
|
||||
1. camera detection/tracking owns semantic class and image-space identity;
|
||||
2. the passive K1 point stream owns metric range and occupied geometry;
|
||||
3. L2.6 supplies the source-aligned local-surface reference;
|
||||
4. object presence publishes `agree`, `single-source-camera`, `conflict` or
|
||||
`unknown`;
|
||||
5. unassociated occupied LiDAR components remain a separate
|
||||
`single-source-geometry` layer with no invented semantic label;
|
||||
6. absent returns never mean free space;
|
||||
7. no branch has command, navigation or safety authority.
|
||||
|
||||
This is not a LiDAR-native object detector and it does not replace camera
|
||||
segmentation. It is the conservative validation layer between the accepted E26
|
||||
camera-first result and a future planner-facing occupancy product.
|
||||
|
||||
## Immutable inputs
|
||||
|
||||
- Source recording: `RAVNOVES00`.
|
||||
- Frames: `4,489`; source-aligned LiDAR/L2.6 frames: `3,928`.
|
||||
- Source E26 result:
|
||||
`e10-integrated-perception-459aac93918d8f6414b342986ccc6968fefcef6c1f3a78a5254df0b565255ad2`.
|
||||
- Source E26 fusion-frame SHA-256:
|
||||
`a1eb6c87880d283888c25a259167078377cdb0a32f9830bdd8cf3bc6e2a69f5f`.
|
||||
- L2.6 model:
|
||||
`k1-local-surface-23762244c8bdb97de26fb721ac957d7a00bc9a63571ac4cfa4be19c4effc7d55`.
|
||||
- Source pack:
|
||||
`e10-lidar-pack-576c994a6c814e2592dd6240ace3902a5db94843312c759a73ba0c9166157d2b`.
|
||||
- Factory projection: calibrated K1 KB4, `camera_1`, `800×600`.
|
||||
- Pinned profile:
|
||||
`experiments/perception/e29_camera_geometry_fusion_profile.json`.
|
||||
|
||||
The source point pack and E26 lab copy have different session/result identities
|
||||
but the same immutable `lidar-pack.npz` payload, timeline, frame indices and
|
||||
factory calibration. E29 binds every fusion row back to the exact source frame
|
||||
and session timestamp. It sends no request to K1 and changes no raw or
|
||||
persistent reconstruction data.
|
||||
|
||||
## Implemented contract
|
||||
|
||||
Each current camera object keeps its E26 class, box, track and motion evidence.
|
||||
For the same source frame E29:
|
||||
|
||||
1. projects the complete K1 map-frame cloud through the factory KB4 model;
|
||||
2. selects visible points inside an inset detector box;
|
||||
3. reads the L2.6 point class relative to the rolling local surface;
|
||||
4. depth-clusters and spatially clusters only `occupied-above-surface` support;
|
||||
5. publishes a robust median range only for connected occupied support;
|
||||
6. preserves camera-only observations when LiDAR support is absent or
|
||||
insufficient;
|
||||
7. emits `conflict` only when an object that previously had observed 3D
|
||||
geometry is currently covered by enough classified points and the L2.6
|
||||
result calls that region surface rather than occupied.
|
||||
|
||||
The support gate is not “20 points means an object”. It combines:
|
||||
|
||||
- calibrated image association;
|
||||
- L2.6 height-over-local-surface classification;
|
||||
- contiguous depth support;
|
||||
- spatial connectedness;
|
||||
- distinct occupied voxels;
|
||||
- explicit source availability.
|
||||
|
||||
All remaining occupied L2.6 points inside the 10 m local model are grouped into
|
||||
bounded connected voxel components. A component touched by accepted semantic
|
||||
support is not repeated as geometry-only. Every retained component is
|
||||
`occupied/unknown-class`, not a car, person, wall or curb.
|
||||
|
||||
## Full replay result
|
||||
|
||||
### Semantic observations
|
||||
|
||||
E29 processed `20,513` E26 observations; `19,625` were current semantic
|
||||
observations.
|
||||
|
||||
| geometry status | observations | meaning |
|
||||
|---|---:|---|
|
||||
| `agree` | 6,341 | camera semantic has connected L2.6 occupied support |
|
||||
| `single-source-camera` | 13,246 | camera semantic remains valid, metric geometry is not qualified |
|
||||
| `conflict` | 38 | prior object geometry and current local-surface class disagree |
|
||||
| `unknown` | 888 | held observation or source/local-surface evidence unavailable |
|
||||
|
||||
Agreement among current semantic observations is `32.31%`. This is coverage,
|
||||
not precision or recall. L2.6 classifies only the local 10 m area, objects are
|
||||
frequently sparse or occluded, and the replay has no independent object ground
|
||||
truth.
|
||||
|
||||
Per camera group:
|
||||
|
||||
- vehicle: `5,937 agree`, `12,518 camera-only`, `38 conflict`, `814 unknown`;
|
||||
- person: `363 agree`, `713 camera-only`, `70 unknown`;
|
||||
- bicycle: `29 agree`, `10 camera-only`;
|
||||
- motorcycle: `12 agree`, `5 camera-only`, `4 unknown`.
|
||||
|
||||
E29 adds useful evidence to cases that E26 could not settle:
|
||||
|
||||
- `1,764` E26 `insufficient-independent-evidence` observations now have
|
||||
connected occupied support;
|
||||
- `1,092` camera-relative-only observations now have occupied metric geometry;
|
||||
- `235` E26 motion-conflict observations have presence geometry. This confirms
|
||||
occupied support but deliberately does not resolve the independent motion
|
||||
conflict.
|
||||
|
||||
Of the `13,246` camera-only observations, `11,259` have no L2.6-classified
|
||||
point inside the box and `11,688` have no occupied point. Another `1,256`
|
||||
contain exactly one occupied point, which remains insufficient rather than
|
||||
being promoted by threshold wishful thinking.
|
||||
|
||||
### Explicit conflicts
|
||||
|
||||
The `38` conflicts belong to nine source tracks and 16 short episodes. All are
|
||||
vehicle-class observations. The largest concentrations are:
|
||||
|
||||
- track `115`, `69.799–71.398 s`;
|
||||
- track `470`, `215.324–218.708 s`;
|
||||
- track `679`, `282.569–283.281 s`;
|
||||
- track `1011`, `354.937–355.242 s`;
|
||||
- track `1276`, `402.606–402.995 s`.
|
||||
|
||||
These are review candidates, not proof that the camera is wrong. In those
|
||||
boxes L2.6 classified 6–38 points, at least 80% as surface and none as
|
||||
occupied, while prior E26 geometry existed. The episode can therefore expose a
|
||||
camera false positive, temporal misbinding, projection edge case or a local
|
||||
surface failure. E29 keeps the disagreement instead of averaging it away.
|
||||
|
||||
### Geometry-only occupied layer
|
||||
|
||||
All `3,928` source-available frames contain at least one unassociated occupied
|
||||
component:
|
||||
|
||||
- `21,321` bounded components;
|
||||
- `2,021,343` supporting points;
|
||||
- `5` components/frame p50, `9` p95, `15` maximum;
|
||||
- `16` points/component p50, `562` p95;
|
||||
- nearest range `6.087 m` p50 and `9.477 m` p95.
|
||||
|
||||
This proves the required parallel path exists. It does not prove that every
|
||||
component is a navigation obstacle. The layer includes structures, vegetation,
|
||||
vehicles, people, terrain discontinuities, possible scanner-carrier/self
|
||||
returns and local-surface mistakes. It needs operator episode review and a
|
||||
vehicle-footprint/self-filter contract before planner qualification.
|
||||
|
||||
## Runtime
|
||||
|
||||
The complete offline build took `8.467 s`:
|
||||
|
||||
- processing mean: `1.826 ms/frame`;
|
||||
- p50: `1.948 ms/frame`;
|
||||
- p95: `2.517 ms/frame`;
|
||||
- maximum: `9.552 ms/frame`.
|
||||
|
||||
This measures only E29 postprocessing on materialized replay arrays. It excludes
|
||||
camera inference, sensor transport, decoding, L2.6 compute, serialization and
|
||||
browser rendering. A second invocation returned the same immutable result ID
|
||||
and validated artifact digests rather than rebuilding the result.
|
||||
|
||||
## Conclusion
|
||||
|
||||
The selected architecture is feasible on the existing K1 recording without a
|
||||
second scanner and without changing K1 firmware:
|
||||
|
||||
```text
|
||||
camera detection / segmentation / tracking
|
||||
│
|
||||
▼
|
||||
semantic observation
|
||||
│
|
||||
factory KB4 + source frame/time
|
||||
│
|
||||
▼
|
||||
K1 points ── L2.6 local surface ── occupied support / range
|
||||
│
|
||||
┌───────────┴───────────┐
|
||||
▼ ▼
|
||||
semantic geometry status geometry-only occupied
|
||||
```
|
||||
|
||||
E29 is accepted as a diagnostic replay implementation. It is not promoted to
|
||||
navigation or safety. The next useful gate is operator review of the 16
|
||||
conflict episodes, representative camera-only gaps and geometry-only
|
||||
components, followed by recorded-source-paced shadow delivery with deadline
|
||||
and staleness health.
|
||||
|
||||
## Reproduction and validation
|
||||
|
||||
```bash
|
||||
PYTHONPATH=src .venv/bin/python \
|
||||
experiments/perception/run_e29_camera_geometry_fusion.py \
|
||||
--fusion-result .runtime/compute-experiments/e10/worker-results/e10-integrated-perception-459aac93918d8f6414b342986ccc6968fefcef6c1f3a78a5254df0b565255ad2 \
|
||||
--source-pack .runtime/compute-experiments/e10/lidar-packs/e10-lidar-pack-576c994a6c814e2592dd6240ace3902a5db94843312c759a73ba0c9166157d2b \
|
||||
--local-surface .runtime/compute-experiments/k1-local-surface-v1/models/k1-local-surface-23762244c8bdb97de26fb721ac957d7a00bc9a63571ac4cfa4be19c4effc7d55
|
||||
```
|
||||
|
||||
- `4` targeted E29 tests pass.
|
||||
- Ruff passes for the E29 module, runner and tests.
|
||||
- Strict mypy passes for the E29 module, runner and tests.
|
||||
- Raw K1 evidence and persistent reconstruction are unchanged.
|
||||
- Commands, free-space inference, navigation and safety authority remain false.
|
||||
@@ -0,0 +1,17 @@
|
||||
{
|
||||
"profile_id": "camera-first-local-surface-validation/v1",
|
||||
"bbox_inset_fraction": 0.03,
|
||||
"depth_cluster_minimum_gap_m": 0.45,
|
||||
"depth_cluster_gap_fraction": 0.08,
|
||||
"spatial_cluster_radius_m": 0.6,
|
||||
"semantic_minimum_occupied_points": 2,
|
||||
"semantic_minimum_occupied_voxels": 1,
|
||||
"semantic_voxel_size_m": 0.35,
|
||||
"conflict_minimum_classified_points": 6,
|
||||
"conflict_surface_fraction": 0.8,
|
||||
"geometry_local_radius_m": 10.0,
|
||||
"geometry_voxel_size_m": 0.45,
|
||||
"geometry_minimum_cluster_points": 4,
|
||||
"geometry_minimum_cluster_voxels": 2,
|
||||
"maximum_geometry_clusters_per_frame": 64
|
||||
}
|
||||
@@ -0,0 +1,85 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Build LAB E29 from immutable E26 and L2.6 replay artifacts."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from k1link.compute.lidar_field_review import E10LidarFieldSource
|
||||
from k1link.compute.lidar_local_surface import K1LocalSurfaceV1
|
||||
from k1link.compute.semantic_geometry_fusion import (
|
||||
CameraGeometryFusionProfile,
|
||||
build_camera_geometry_fusion,
|
||||
)
|
||||
|
||||
|
||||
def _sha256(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as stream:
|
||||
while chunk := stream.read(1024 * 1024):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def _profile(path: Path) -> CameraGeometryFusionProfile:
|
||||
value: Any = json.loads(path.read_text(encoding="utf-8"))
|
||||
if not isinstance(value, dict):
|
||||
raise ValueError("LAB E29 profile must be an object")
|
||||
return CameraGeometryFusionProfile(**value)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--fusion-result", type=Path, required=True)
|
||||
parser.add_argument("--source-pack", type=Path, required=True)
|
||||
parser.add_argument("--local-surface", type=Path, required=True)
|
||||
parser.add_argument(
|
||||
"--profile",
|
||||
type=Path,
|
||||
default=Path("experiments/perception/e29_camera_geometry_fusion_profile.json"),
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output-root",
|
||||
type=Path,
|
||||
default=Path(".runtime/compute-experiments/e29/results"),
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
fusion_result = args.fusion_result.expanduser().resolve(strict=True)
|
||||
result_document = json.loads((fusion_result / "result.json").read_text(encoding="utf-8"))
|
||||
result_id = result_document.get("result_id")
|
||||
if not isinstance(result_id, str) or result_id != fusion_result.name:
|
||||
raise ValueError("LAB E29 source result id is invalid")
|
||||
frames_path = fusion_result / "fusion-frames.jsonl"
|
||||
source = E10LidarFieldSource(args.source_pack)
|
||||
surface = K1LocalSurfaceV1(args.local_surface)
|
||||
try:
|
||||
build = build_camera_geometry_fusion(
|
||||
fusion_frames_path=frames_path,
|
||||
source_result_id=result_id,
|
||||
source_fusion_frames_sha256=_sha256(frames_path),
|
||||
source=source,
|
||||
surface=surface,
|
||||
output_root=args.output_root,
|
||||
profile=_profile(args.profile),
|
||||
)
|
||||
finally:
|
||||
surface.close()
|
||||
source.close()
|
||||
summary = {
|
||||
"result_id": build.result_id,
|
||||
"result_root": str(build.result_root),
|
||||
"status": build.report["status"],
|
||||
"metrics": build.report["metrics"],
|
||||
"decision": build.report["decision"],
|
||||
"authority": build.report["authority"],
|
||||
}
|
||||
print(json.dumps(summary, ensure_ascii=False, indent=2))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user