feat(perception): qualify lossless lidar observations

2026-07-30 11:37:57 +03:00
parent e56fa0c074
commit 7d1a70d8e0
13 changed files with 1874 additions and 119 deletions
@@ -19,6 +19,8 @@ Each gate produces evidence and an explicit GO, PAUSE or BLOCKED result.
 | Detector candidate comparison | FROZEN BEFORE TRUTH — E47 freezes raw-KB4 and fixed-valid-FOV-fill predictions from the same exact Mask R-CNN checkpoint. No accuracy result or winner exists before the E46 truth seal. |
 | Detector truth/evaluation executors | DEFERRED BY OWNER, READY, NOT RUN — E48 fail-closed review/adjudication sealing and separate post-seal E49 scoring are implemented. Neither result exists because real independent reviews are absent. The deferral does not convert E37–E40 into blind truth and does not block R2/R4 operational work. |
 | RAVNOVES00 recorded AI publication | GO — after a real backend restart, the admitted E10 overlay `e10-integrated-perception-36964643f1c0727a434671e3cddc2e76c536b9c28b8c56845f18941ae791c39a` recovered from its sealed cache without model execution or source revalidation. Recording `f83e29dc-5d25-42a5-824e-228cdbb83d21` streamed `69,227,327` bytes with verified SHA-256 `c2bb73fca8bb616a3929df236c6dc63d071b6466a6ca57d06487ef18ed985b57`; a warm localhost request completed in `0.391 s`, and its steady-state server RSS delta was `48 KiB`. |
+| L2.6 lossless replay parity | GO — replay-pack-v2 exact equivalence passed for all 4,570 native LiDAR frames, 4,598 pose frames and 10,751,258 points. The same local-surface algorithm produced 4,570/4,570 valid frames with 20.6421 ms pose-age p95 while retaining separate v2 identity, RGBI/intensity and host-time provenance. |
+| L2.6 motion/semantic qualification | GO (diagnostic only) — E51 processed 4,489/4,489 E32/E34 frames, preserved all 2,119,302 current occupied point rows, measured 0.4148 ms frame p95 and 0.2969 MiB peak-RSS growth, and emitted explicit freshness/conflict/proximity/motion evidence. Dynamic class, collision state, free space, command, navigation and safety authority remain unavailable. |
 | Product interface | DEFERRED — no new windows, page anatomy or design changes are part of this stabilization increment. |

 The governing decision is
@@ -26,6 +28,9 @@ The governing decision is
 Historical E37–E40 artifacts remain immutable; only the claims made from them
 change.

+The L2.6 replay and observation decisions are fixed by
+[`ADR 0034`](adr/0034-lossless-replay-and-diagnostic-observation-boundary.md).
+
 ## Earlier checkpoint — 2026-07-24

 | Stage | Result |
@@ -10,7 +10,8 @@ parallel geometry-only replay implemented; E30–E35 source-scoped qualification
 accepted; RAVNOVES00 reference-source product maturation active; E36 transfer
 preregistered and deferred by ADR 0030/0032; E41 methodology boundary, E42
 metamorphic checks, E44 amplification audit and E50 exact-content reference
-index complete
+index complete; E51 motion/proximity/semantic derivative and full lossless
+replay-pack-v2 local-surface parity accepted
 Scope: passively received real-time K1 point/pose evidence, immutable replay and
 future live shadow processing
 Explicitly out of scope: K1 firmware modification, a new onboard exporter, new
@@ -395,7 +396,7 @@ Dataset expansion is no longer the next gate.

 - [x] Bind the immutable `RAVNOVES00` E10 source by pack identity and artifact
  hash without copying or rewriting the source generation.
- [ ] Mirror the same accepted profile over replay-pack-v2 evidence while
+- [x] Mirror the same accepted profile over replay-pack-v2 evidence while
  preserving its separate identity and field-retention contract.
 - [x] Reject stale pose binding and publish pose-binding age explicitly; keep
  map-native processing honest instead of claiming that pose inversion
@@ -411,9 +412,12 @@ Dataset expansion is no longer the next gate.
  Do not infer `free` merely because a mapped point is absent.
 - [x] Leave the immutable persistent reconstruction untouched by the local
  derivative.
- [ ] Add recent-collision and dynamic-observation layers as separate
-  derivatives with independent decay and provenance.
- [ ] Reuse the accepted camera-to-LiDAR projection as an optional semantic
+- [x] Add a separate diagnostic motion-observation and near-occupied proximity
+  derivative with temporal freshness, source provenance and no persistent-map
+  mutation.
+- [ ] Admit a `recent-collision` state only after vehicle-body and LiDAR-mount
+  geometry are bound; proximity is not collision truth.
+- [x] Reuse the accepted camera-to-LiDAR projection as an optional semantic
  layer with source, confidence, freshness and conflict fields.
 - [x] Qualify next-frame prediction and temporal stability with the evaluated
  frame excluded from prediction input.
@@ -423,7 +427,7 @@ Dataset expansion is no longer the next gate.
 - [x] Preserve the prior prediction plane and point/cell-aligned residual
  evidence so a selected tail can be explained spatially instead of only by an
  aggregate p95.
- [ ] Complete the remaining qualification report with per-frame latency,
+- [x] Complete the remaining qualification report with per-frame latency,
  point age, obstacle preservation and memory growth.
 - [x] Replay the same profile through a bounded latest-wins shadow queue at the
  recorded 1× source rate; no K1 command, navigation or safety authority is
@@ -455,6 +459,30 @@ distributions are: derived sensor-to-surface height `1.2816 m` p50, roughness
 These values describe this recording only; they are not calibration, ground
 truth or a navigation gate.

+The lossless replay-pack-v2 path is now qualified independently of that
+camera-aligned E10 generation. Its immutable pack
+`lidar-replay-pack-8fc0fb418578b8ee2ac88d502d2acbc63ae533437a9da14f1a9f9d8916f613ce`
+retains all `4,570` native LiDAR frames, `4,598` pose frames, `10,751,258`
+points, raw RGBI/intensity and exact host timing. Source-to-pack equivalence
+passed with zero array mismatches and `100%` pose coverage. The higher frame
+and point counts are expected: replay-pack-v2 is native LiDAR cadence, whereas
+E10 contains only camera-aligned available slices.
+
+The same local-surface parameters produced immutable model
+`k1-local-surface-61a0307497b1e9b32d5521f59aa853bce22d81def7ca4d3fc81273a4162e6d22`.
+All `4,570/4,570` frames are valid, with zero stale-pose, insufficient-surface
+or fit-failure results. Pose-binding age is `20.6421 ms` p95. Prediction has
+`4,569` prior-only samples, `0.04119 m` residual p50 and `0.07983 m` p95.
+The source identity remains replay-pack-v2 and intensity remains available;
+the accepted E10 artifact is neither rewritten nor relabelled.
+
+The v2 builder uses a bounded two-pass ingest. It counts and freezes scalar
+frame contracts first, then fills preallocated numeric arrays one decoded
+frame at a time. The strict reader materializes each compressed retained array
+once before repeated validation. On the full capture the old repeated-NPZ
+access was rejected after it demonstrated unbounded CPU amplification; the
+accepted path completed exact equivalence without Docker or parallel workers.
+
 The L2.6b qualification scores each available frame against a local plane built
 only from the preceding TTL window; the frame being scored is excluded from
 the prediction input. It produced `3,927` independent next-frame samples. The
@@ -563,6 +591,24 @@ health. Physical K1 execution is deferred until hardware-clock, field-network
 or acquisition behavior is the test subject. Commands, free-space, navigation
 and safety authority remain disabled.

+LAB E51 consumes accepted E32/E34 evidence without modifying it and publishes
+only bounded diagnostic signals. It processed `4,489/4,489` frames with zero
+map-frame jump candidates. The derivative observed exactly the same
+`2,119,302` current occupied point rows as E34. Frame processing was
+`0.4148 ms` p95; process peak-RSS growth was `0.2969 MiB`. Accepted E10 point
+age was `70.7886 ms` p95 from LiDAR to camera and `21.8545 ms` p95 from pose
+to LiDAR.
+
+E51 emitted `22,885` motion candidates, `2,430` current near-occupied
+proximity candidates, `15,604` camera-owned semantic signals and `90`
+explicit conflicts. A semantic numeric-confidence field is present but marked
+unavailable because the upstream E32 contract has no admitted numeric
+confidence. Held evidence cannot publish current proximity. Map-frame jump
+candidates cannot publish motion. Dynamic class and collision state remain
+unavailable, and every signal carries false command/navigation/safety
+authority. The accepted result is
+`e51-motion-semantic-1abb7eb9940608fc5af95a1f318cadfbc42ac2412a8662b6622e000e03da1555`.
+
 Exit: one immutable K1 session yields both a persistent reconstruction and a
 bounded local world state without hard-coded terrain height or scanner-side
 changes.
@@ -0,0 +1,91 @@
+# ADR 0034: Lossless replay and diagnostic observation boundary
+
+Date: 2026-07-30
+Status: accepted and implemented
+
+## Decision
+
+The accepted K1 local-surface algorithm consumes one normalized, read-only
+source view. Two immutable source schemas are admitted:
+
+- camera-aligned `missioncore.e10-lidar-replay-pack/v1`;
+- native-cadence, field-retaining `missioncore.lidar-replay-pack/v2`.
+
+The normalized view carries exact source pack identity, artifact SHA-256,
+session identity, representation, schema, native point offsets, map-frame XYZ,
+host-monotonic timing and nearest-pose age. A replay-pack-v2 source additionally
+retains RGBI/intensity provenance. The derivative never rewrites or disguises
+one source as the other.
+
+The lossless v2 builder uses two bounded passes over raw capture evidence. The
+first pass freezes frame counts and scalar header/timing contracts. The second
+pass preallocates and fills numeric arrays one decoded frame at a time, failing
+closed if the source differs between passes. The strict reader decompresses
+each NPZ member once before repeated integrity, field and logical-content
+validation. Repeated on-demand decompression inside a frame loop is forbidden.
+
+## Diagnostic observation derivative
+
+E51 is separate from the persistent reconstruction and the E34 short-TTL
+occupied/unknown layer. It may publish:
+
+- temporal motion candidates derived from a bounded centroid history;
+- current near-occupied proximity candidates from admitted E32 range;
+- camera-owned semantics and semantic provenance;
+- explicit current/held freshness and evidence conflict.
+
+It may not publish:
+
+- a dynamic object class;
+- a collision state;
+- free space from missing points;
+- numeric semantic confidence when the upstream contract has none;
+- command, navigation or safety authority.
+
+A map-frame jump candidate rejects motion publication. Held evidence cannot
+publish current proximity. Collision remains unavailable until the physical
+vehicle body and LiDAR mount/extrinsic geometry are separately bound and
+qualified.
+
+## Resource boundary
+
+All full-session Mac execution is sequential. The current 14-inch 2023 MacBook
+Pro has 18 GiB RAM; replay build, strict validation, local-surface build and
+test execution are never intentionally overlapped. Docker is not part of this
+path. Port `8000` remains the canonical running Mission Core service.
+
+## Accepted evidence
+
+Full native replay:
+
+```text
+lidar-replay-pack-8fc0fb418578b8ee2ac88d502d2acbc63ae533437a9da14f1a9f9d8916f613ce
+```
+
+- 4,570 LiDAR frames;
+- 4,598 pose frames;
+- 10,751,258 points;
+- exact live/replay equivalence passed;
+- pose coverage 100%.
+
+Full v2 local surface:
+
+```text
+k1-local-surface-61a0307497b1e9b32d5521f59aa853bce22d81def7ca4d3fc81273a4162e6d22
+```
+
+- 4,570/4,570 valid;
+- zero stale-pose, insufficient-surface or fit-failure frames;
+- pose-binding age 20.6421 ms p95.
+
+E51:
+
+```text
+e51-motion-semantic-1abb7eb9940608fc5af95a1f318cadfbc42ac2412a8662b6622e000e03da1555
+```
+
+- 4,489/4,489 frames;
+- 2,119,302/2,119,302 current occupied point rows preserved;
+- 0.4148 ms frame-processing p95;
+- 0.2969 MiB process peak-RSS growth;
+- all acceptance checks passed.