feat(perception): qualify lossless lidar observations

2026-07-30 11:37:57 +03:00
parent e56fa0c074
commit 7d1a70d8e0
13 changed files with 1874 additions and 119 deletions
@@ -0,0 +1,133 @@
+# LAB E51 · lossless replay parity and diagnostic observation layer
+
+Date: 2026-07-30
+
+Status: accepted diagnostic derivative; no production motion, collision,
+navigation or safety authority
+
+## Goal
+
+Close the source-contract and qualification gaps that remained in L2.6:
+
+1. run the accepted local-surface algorithm over native
+   `lidar-replay-pack/v2` without losing its identity or retained fields;
+2. publish motion, proximity and camera-semantic evidence as a separate
+   derivative rather than mutating the persistent map;
+3. measure per-frame latency, accepted point age, obstacle preservation and
+   process memory growth over the complete accepted RAVNOVES00 source.
+
+No UI, K1 command, firmware behavior or physical acquisition path changed.
+
+## Immutable inputs
+
+- E10 LiDAR:
+  `e10-lidar-pack-576c994a6c814e2592dd6240ace3902a5db94843312c759a73ba0c9166157d2b`;
+- E32 track geometry:
+  `e32-track-geometry-a14ca0e7fb3850ca0dfa3c41634e1b490a2d58ab74d101afc6d6921fbdb0e6fd`;
+- E34 temporal occupied/unknown:
+  `e34-temporal-occupied-8d9abb3f2cc072cfdbb16cc4e55798e05c35a0abe0b8f691096770e091573a73`;
+- raw session:
+  `20260720T065719Z_viewer_live` (`RAVNOVES00`).
+
+All upstream artifact hashes were verified before and after E51.
+
+## Lossless replay-pack-v2
+
+Accepted pack:
+
+```text
+lidar-replay-pack-8fc0fb418578b8ee2ac88d502d2acbc63ae533437a9da14f1a9f9d8916f613ce
+```
+
+| Measurement | Result |
+| --- | ---: |
+| Native LiDAR frames | 4,570 |
+| Pose frames | 4,598 |
+| Retained points | 10,751,258 |
+| Pose coverage | 100% |
+| Exact array mismatches | 0 |
+| Equivalence | passed |
+
+The pack contains more frames and points than E10 because it retains native
+LiDAR cadence rather than only camera-aligned slices. Raw XYZ, scaled map XYZ,
+RGBI, low-byte intensity, device header fields, capture sequence and exact host
+epoch/monotonic time remain distinct retained fields.
+
+The initial full-scale reader attempt exposed repeated NPZ decompression inside
+the XYZ validation loop. That attempt was stopped before acceptance. The
+implemented reader now materializes every compressed member once. The builder
+uses a bounded two-pass source scan and preallocated arrays; it does not retain
+all decoded point objects.
+
+## Local-surface parity
+
+Accepted v2 model:
+
+```text
+k1-local-surface-61a0307497b1e9b32d5521f59aa853bce22d81def7ca4d3fc81273a4162e6d22
+```
+
+| Measurement | E10 camera-aligned | v2 native cadence |
+| --- | ---: | ---: |
+| Source frames | 4,489 | 4,570 |
+| Available/valid | 3,928/3,928 | 4,570/4,570 |
+| Pose stale | 0 | 0 |
+| Fit failed | 0 | 0 |
+| Pose age p95 | 21.8545 ms | 20.6421 ms |
+| Prediction samples | 3,927 | 4,569 |
+| Prediction residual p50 | 0.04065 m | 0.04119 m |
+| Prediction residual p95 | 0.07304 m | 0.07983 m |
+
+Rolling surface, classification, pose-binding and temporal parameters are
+identical. The input representation is not: v2 remains separately identified
+and reports intensity availability and complete field retention.
+
+## E51 method
+
+For every accepted E32/E34 frame, E51:
+
+- carries current/held freshness and E32 evidence conflict;
+- derives a motion candidate only from a bounded E34 centroid history;
+- suppresses motion on a map-frame jump candidate;
+- derives proximity only from a current hit-backed E32 range;
+- carries camera-owned semantic value or held E34 semantic provenance;
+- publishes numeric confidence as explicitly unavailable;
+- preserves E34 current occupied rows without rewriting any cell or point;
+- leaves collision unavailable because vehicle-body and LiDAR-mount geometry
+  are not bound.
+
+The result contains at most 25 signals in any frame against a frozen maximum
+of 256.
+
+## Accepted E51 result
+
+```text
+e51-motion-semantic-1abb7eb9940608fc5af95a1f318cadfbc42ac2412a8662b6622e000e03da1555
+```
+
+| Measurement | Result |
+| --- | ---: |
+| Frames processed | 4,489/4,489 |
+| Map-frame jump candidates | 0 |
+| Current occupied rows | 2,119,302/2,119,302 exact |
+| Frame processing p50 / p95 / max | 0.2358 / 0.4148 / 8.0526 ms |
+| Process peak-RSS growth | 0.2969 MiB |
+| LiDAR-to-camera point age p95 / max | 70.7886 / 99.5738 ms |
+| Pose-to-LiDAR age p95 / max | 21.8545 / 73.7976 ms |
+| Motion candidates | 22,885 |
+| Proximity candidates | 2,430 |
+| Semantic signals | 15,604 |
+| Explicit conflicts | 90 |
+
+All acceptance checks passed.
+
+## Decision
+
+The lossless v2 source path, local-surface parity and bounded diagnostic
+motion/proximity/semantic derivative are accepted for replay evidence.
+
+This result does not admit a dynamic class or collision truth. The next
+geometry gate is a versioned vehicle body plus LiDAR mount/extrinsic contract.
+Only after that gate may near-occupied evidence be evaluated as
+`recent-collision`. LiDAR-native 3D detection remains deferred behind the
+remaining L2.6 physical/geometry boundary.
@@ -0,0 +1,47 @@
+{
+  "schema_version": "missioncore.e51-motion-semantic-profile/v1",
+  "profile_id": "e51-motion-proximity-semantic-qualification/v1",
+  "expected": {
+    "e32_result_id": "e32-track-geometry-a14ca0e7fb3850ca0dfa3c41634e1b490a2d58ab74d101afc6d6921fbdb0e6fd",
+    "e34_result_id": "e34-temporal-occupied-8d9abb3f2cc072cfdbb16cc4e55798e05c35a0abe0b8f691096770e091573a73",
+    "source_pack_id": "e10-lidar-pack-576c994a6c814e2592dd6240ace3902a5db94843312c759a73ba0c9166157d2b"
+  },
+  "motion": {
+    "minimum_observations": 3,
+    "minimum_span_seconds": 0.2,
+    "minimum_displacement_m": 0.25,
+    "minimum_speed_mps": 0.4,
+    "maximum_speed_mps": 20.0,
+    "classification": "diagnostic-motion-candidate",
+    "dynamic_class_available": false,
+    "reject_map_frame_jump_candidates": true
+  },
+  "proximity": {
+    "threshold_m": 2.0,
+    "classification": "diagnostic-near-occupied-candidate",
+    "collision_state_available": false,
+    "reason": "vehicle-body-and-lidar-mount-geometry-not-bound"
+  },
+  "semantic": {
+    "camera_owns_semantics": true,
+    "numeric_confidence_available": false,
+    "require_explicit_freshness": true,
+    "require_explicit_conflict": true
+  },
+  "acceptance": {
+    "maximum_signals_per_frame": 256,
+    "maximum_latency_p95_ms": 20.0,
+    "maximum_rss_growth_mib": 256.0,
+    "maximum_lidar_camera_age_p95_ms": 100.0,
+    "maximum_pose_age_p95_ms": 100.0,
+    "require_exact_current_obstacle_row_preservation": true,
+    "require_complete_frame_accounting": true,
+    "require_no_free_space_publication": true,
+    "require_no_dynamic_or_collision_authority": true,
+    "require_upstream_immutability": true
+  },
+  "authority": {
+    "commands_enabled": false,
+    "navigation_or_safety_accepted": false
+  }
+}
@@ -0,0 +1,44 @@
+#!/usr/bin/env python3
+"""Build the immutable E51 motion/proximity/semantic qualification."""
+
+from __future__ import annotations
+
+import argparse
+import json
+from pathlib import Path
+
+from k1link.compute import build_e51_motion_semantic_qualification
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--e32-result", type=Path, required=True)
+    parser.add_argument("--e34-result", type=Path, required=True)
+    parser.add_argument("--e10-source", type=Path, required=True)
+    parser.add_argument("--profile", type=Path, required=True)
+    parser.add_argument("--output-root", type=Path, required=True)
+    args = parser.parse_args()
+    result = build_e51_motion_semantic_qualification(
+        e32_result_root=args.e32_result,
+        e34_result_root=args.e34_result,
+        e10_source_root=args.e10_source,
+        profile_path=args.profile,
+        output_root=args.output_root,
+    )
+    print(
+        json.dumps(
+            {
+                "result_id": result.result_id,
+                "result_root": str(result.result_root),
+                "accepted": result.accepted,
+                "metrics": result.report["metrics"],
+            },
+            ensure_ascii=False,
+            sort_keys=True,
+        )
+    )
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())
@@ -6,9 +6,12 @@ import json
 from pathlib import Path

 from k1link.compute import (
+    E10_LIDAR_PACK_SCHEMA,
+    LIDAR_REPLAY_PACK_SCHEMA,
    E10LidarFieldSource,
    K1LocalSurfaceProfile,
    K1LocalSurfaceV1,
+    LidarReplayPackV2,
    build_k1_local_surface,
 )

@@ -43,7 +46,18 @@ def main() -> int:
        obstacle_min_height_m=arguments.obstacle_min_height_m,
        obstacle_max_height_m=arguments.obstacle_max_height_m,
    )
-    source = E10LidarFieldSource(arguments.source_pack)
+    manifest = json.loads(
+        (arguments.source_pack / "manifest.json").read_text(encoding="utf-8")
+    )
+    schema_version = manifest.get("schema_version")
+    source: E10LidarFieldSource | LidarReplayPackV2
+    if schema_version == E10_LIDAR_PACK_SCHEMA:
+        source = E10LidarFieldSource(arguments.source_pack)
+    elif schema_version == LIDAR_REPLAY_PACK_SCHEMA:
+        source = LidarReplayPackV2(arguments.source_pack)
+    else:
+        parser_schema = str(schema_version) if schema_version is not None else "missing"
+        raise SystemExit(f"unsupported local-surface source schema: {parser_schema}")
    try:
        output = build_k1_local_surface(
            source,