docs(perception): record full-graph network and recovery evidence
This commit is contained in:
DCCONSTRUCTIONS
@@ -0,0 +1,157 @@
|
||||
# Stage 2, increment 13: complete graph over a bounded network stream
|
||||
|
||||
2026-09-02, 19:00 MSK. Implementation/test commit: `6469505`.
|
||||
This is an increment of the existing four-stage
|
||||
[ExecPlan](../../docs/OBSERVATORY_REALTIME_PROFILES_EXECPLAN.md), not another stage
|
||||
or a standalone profile release. No product LAB/registry cutover or actuation.
|
||||
|
||||
## Result and scope
|
||||
|
||||
The real DDRNet-39/RF-DETR/LiDAR-distance/motion/TRAVEL-TGS/costmap/policy graph
|
||||
now takes input and returns results over the authenticated gRPC/TLS candidate.
|
||||
An independent **CPU-only** container owns the recorded source and result
|
||||
consumer. The one GPU container has **no camera index, camera directory or
|
||||
sensor recording mount**. It retains only the calibration/model/config/code
|
||||
mounts needed by the existing diagnostic graph. Models are resident and GPU
|
||||
inferences remain serialized; no second full profile runs alongside it.
|
||||
|
||||
Both containers share the same Linux kernel boot and zero time-namespace
|
||||
offsets. The pilot explicitly checks this before subtracting monotonic times.
|
||||
This is a real container-network proof, **not Mac↔Worker latency**, not hardware
|
||||
sensor synchronization and not a 300–500 Mbit/s radio-link qualification.
|
||||
|
||||
The source reads the original camera fragments and normalized point/pose
|
||||
archive incrementally at original 1× arrival time. Only a bounded 128-frame
|
||||
prefix was exercised. Existing archive assets were not uploaded again. The
|
||||
source read 545,590 of 10,751,258 point rows, not the whole recording.
|
||||
|
||||
## Implementation
|
||||
|
||||
- `BinaryGraphInput` is the unchanged decoder/causal sensor-window assembly
|
||||
shared by the original local IPC bridge and `NetworkGraphBridge`. There is
|
||||
no second model or geometry implementation behind the network path.
|
||||
- `pilot_grpc_source.py` shares raw event serialization with `RecordingSource`.
|
||||
It neither owns nor renews the Worker lease. Source gaps discard observations,
|
||||
never change their timestamps or accelerate a catch-up replay.
|
||||
- The local controller issues a new one-shot grant only after old ingress and
|
||||
work have drained. Grants are read-only to the source; source terminal status
|
||||
is a separate read-only mount to the graph. These are temporary trusted pilot
|
||||
control files, not a production application grant API.
|
||||
- `streaming_scene_payload.py` frames the real scene JSON plus a **512×512
|
||||
uint8 mask**, not just hashes. The scene includes detections, metric distances,
|
||||
motion, actual grid coordinates/material/state, cell evidence and policy.
|
||||
Grid coordinates participate in the costmap layer digest. The receiver checks
|
||||
binding, sequence, mask and six-layer payload integrity, then derives its own
|
||||
current freshness/policy view without rewriting published evidence.
|
||||
- Output remains capped at 1 MiB, with a 512 KiB JSON bound and 512 KiB plane
|
||||
bound plus total-length validation. Duplicate JSON keys/non-finite constants,
|
||||
bad shape/hash/binding and control-authority escalation are rejected.
|
||||
- Socket EOF on an established stream preserves resident models and lease,
|
||||
clears temporal history, and waits for a fresh decoded keyframe and causal
|
||||
sensor pair. The source adapter is still a bounded diagnostic: production
|
||||
startup/grant delivery retry and cross-host clock mapping remain open.
|
||||
|
||||
## Sequential Worker measurements
|
||||
|
||||
RTX 4090, temporary stock SM/memory clocks **2610/10251 MHz**, unchanged 450 W
|
||||
limit. GPU container 8 CPU / 8 GiB; source container 2 CPU / 1 GiB and explicit
|
||||
`runc` without GPU devices. Separate internal Docker network; no host listener.
|
||||
These are conditional results, not qualification under automatic downclocking.
|
||||
|
||||
| Measurement | Normal `normal128b` | Gap `gap128b` |
|
||||
| --- | ---: | ---: |
|
||||
| Original camera observations | 128 | 128 |
|
||||
| Validated mask + scene receipts | 128 | 102 |
|
||||
| Accounted omissions | 0 | 26, sequences 24–49 |
|
||||
| Result queue drops / unaccounted | 0 / 0 | 0 / 0 |
|
||||
| Input epochs / temporal resets | 1 / 0 | 2 / 1 |
|
||||
| Consumer-ready p95 / p99 | 106.150 / 115.349 ms | 107.098 / 110.817 ms |
|
||||
| Consumer-ready maximum | 126.910 ms | 117.317 ms |
|
||||
| Frames above 125 ms | 1 | 0 |
|
||||
| Complete six-layer freshness at assessment | 76 / 128 | 58 / 102 |
|
||||
| Receipts before source EOF | 127 | 101 |
|
||||
| Source release lag maximum | 9.333 ms | 9.231 ms |
|
||||
| Maximum result bytes | 459,608 | 460,248 |
|
||||
| Peak tracked bytes / residual | 13,217,791 / 0 | 13,217,791 / 0 |
|
||||
| Peak observed VRAM | 2,369 MiB | 2,372 MiB |
|
||||
| Peak GPU-container memory | 3,348.31 MiB | 3,368.86 MiB |
|
||||
| Shutdown/resource retirement | 4.067 s | 4.065 s |
|
||||
|
||||
Three different receipt clocks are retained: complete network payload arrival;
|
||||
completed identity/payload validation; consumer view ready after freshness and
|
||||
per-cell policy derivation. The table uses the **last** boundary. Freshness is
|
||||
assessed at the explicitly recorded check instant, not retimestamped when the
|
||||
view calculation ends. No UI rendering or motor/controller transport is timed.
|
||||
|
||||
The 2.2 s injected disconnect resumed at camera sequence 50. All four resident
|
||||
PIDs (11, 40, 48, 49) remained alive, lease generation stayed 1, and exactly one
|
||||
temporal reset emptied surface/body/component/rolling state. The source skipped
|
||||
23 cameras while disconnected; the decoder skipped 47–49 until a fresh random
|
||||
access frame/pair. Time between consumer-ready scenes 23 and 50 was **2.725 s**.
|
||||
This availability gap is separate from latency of accepted frames.
|
||||
|
||||
## Integrity and interpretation
|
||||
|
||||
All 230 final scene JSON documents matched the corresponding Worker-published
|
||||
bytes exactly and were independently rechecked against mask hashes, six layer
|
||||
digests, freshness and derived policy counts. All 230 decoded BGR hashes match
|
||||
the earlier reference at their source sequence.
|
||||
|
||||
Normal run: all 128 segmentation masks, detections, metric observations, tracks,
|
||||
threats, material maps, sensor lineage/bindings and raw costmap-state hashes are
|
||||
exact to the previous full-graph reference. It still has 52 original unavailable
|
||||
sensor pairs; no fresh timestamps or permissive policies were invented.
|
||||
|
||||
Gap run: segmentation and detections match 102/102 reference frames. After the
|
||||
reset, history-dependent observations/tracks/costmap can and do differ from the
|
||||
uninterrupted reference (observations 94/102 exact, tracks/threats 24/102, material
|
||||
99/102, raw costmap 85/102). This is not whole-scene parity across missing data.
|
||||
It is explicit reset-and-rebuild behavior, with no reuse of pre-gap history.
|
||||
|
||||
## Rejected attempts and corrections
|
||||
|
||||
All attempts and their diagnostics are retained, not overwritten:
|
||||
|
||||
1. `canary32`: missing pinned DDRNet runner in the diagnostic pack; source never
|
||||
started. `canary32b`: attempted nested mount lacked a mountpoint inside a
|
||||
read-only `/probe`; no profile process started. The corrected pack contains
|
||||
the unchanged hash-verified runner. The second attempt's empty lease volume
|
||||
was independently inspected before removal.
|
||||
2. `canary32c` delivered 32/32 real results; the PowerShell launcher incorrectly
|
||||
read a null exited-process `ExitCode`. Caching its process handle before exit
|
||||
corrected the diagnostic, without changing model execution.
|
||||
3. Initial `normal128` and `gap128` exposed an extra startup reconnect: init was
|
||||
sent two seconds before the delayed source-clock start, tripping the 2 s idle
|
||||
deadline. Source connection now opens 100 ms before release; the timeout was
|
||||
**not** enlarged. Final normal/gap repeats have respectively zero/one genuine
|
||||
recovery and no transport errors. The source-start regression has a test.
|
||||
|
||||
## Checks, evidence and remaining gates
|
||||
|
||||
146 focused tests PASS, 2 Worker-only large-load tests skipped locally; Ruff,
|
||||
format, targeted mypy and diff checks PASS. No local build/browser/load test at
|
||||
Mac memory pressure 2. The earlier UI visual acceptance remains pending.
|
||||
|
||||
Session evidence: `.runtime/perception-stage2-network-graph-20260902T1541Z/`.
|
||||
`review.json` reconciles source, graph and receiver; `verify.py` rechecks one
|
||||
scene at a time. `manifest.json` hashes code, logs, reports, accepted and rejected
|
||||
attempts (236 artifacts), SHA-256
|
||||
`d1bc7e26850a2d9050ee1d8f8a5ece2e6b8c53e2e7dba381cbd4445d68a5ec17`.
|
||||
Code-v3 archive SHA-256:
|
||||
`3e9091524f62cdd528846b2d1e97e9107133c9cd267549f324991c0e63b184fc`.
|
||||
Pinned DDRNet runner additionally inventoried:
|
||||
`b18ad60f277eea69a240a28f290611b94627fb9707faf1bb3e6e22102dad67c1`.
|
||||
Final evidence archive SHA-256:
|
||||
`58ca7f1f59a79bce436d4e029a063563da3081c0c1cf47b5fbbdf49931f0fdd2`.
|
||||
|
||||
Temporary containers, networks and isolated lease volumes are removed; four
|
||||
prior services restored, Triton healthy, GPU back to automatic clocks. Ollama
|
||||
and Frigate remain stopped with restart=no. Ephemeral TLS private key and grants
|
||||
were removed, not archived. Canonical Mac8000 remains running with fresh Worker
|
||||
telemetry; no8765 listener. No push or external/Synology deployment.
|
||||
|
||||
Next within **stage 2**: authenticated application-side grant delivery and
|
||||
cross-host source clock mapping, then the full graph Mac↔Worker and a deliberately
|
||||
slow result consumer, followed by the standalone image. Packaging must remove
|
||||
the current developer/model dependency mounts. Native-host GPU inventory,
|
||||
multiple recordings, physical-live/field quality and autonomy remain unproved.
|
||||
Reference in New Issue
Block a user