docs(perception): record full graph recovery and separate availability latency

This commit is contained in:
DCCONSTRUCTIONS
2026-09-02 16:01:36 +03:00
parent 70927ea585
commit e91721fe6a
3 changed files with 162 additions and 9 deletions
@@ -1,9 +1,9 @@
# ADR 0049 — Stream-first full perception profiles
Date: 2026-09-01; updated 2026-09-02 15:30 MSK. Status: stage-1 engineering
baseline complete; stage-2 binary graph, readiness and resumable-input primitives.
Full-graph recovery adapter, production inventory collector and product cutover
are not implemented by the latest CPU-only continuity proof.
Date: 2026-09-01; updated 2026-09-02 15:56 MSK. Status: stage-1 engineering
baseline complete; stage-2 full binary GPU pilot now supports input recovery.
Production collector, external transport, standalone image and product cutover
remain open; local recording/IPC proof is not physical-live acceptance.
Execution order remains the four stages in
[the ExecPlan](../OBSERVATORY_REALTIME_PROFILES_EXECPLAN.md).
@@ -102,6 +102,50 @@ explicit byte bound is required. No duplicate new scanner feed is introduced.
## Time and lifecycle
Stage-2 full-graph recovery increment, 2026-09-02 15:56 (`70927ea`):
`--recover-input` connects lifecycle to PilotController, RecordingSource,
BinaryGraphBridge, GPU/CPU calls and result binding. Source clock mapping is
fixed before ingress; fault windows never restart/slow it. Bounded diagnostic
`--input-gap sequence:milliseconds` is explicit in effective config. Reconnect
polls quiescence without queuing old events; complete accounting distinguishes
skipped from processed input. Historical one-shot defaults remain available.
The receiver waits for an independently decodable fragment and fresh causal
sensor pair. Decoder-local index resets; original source sequence/timestamps
do not. Actual decoder keyframe validation remains. The graph recreates CPU
surface/geometry/temporal/motion/rolling/threat providers using the same configs
and clears cached segmentation. RF-DETR backend, DDRNet/decoder/TGS children
remain. TGS C++ creates an estimator per request; costmap arrays are per-call,
so no native temporal reset command or restart is needed. Rolling input window
resets independently. GPU/CPU/result gates bind the input epoch, not activation.
Obsolete results release their input rather than terminating the profile;
review removes a lingering collector tuple reference to prior input too.
Two normal/faulted full-profile pairs pass on Worker006. Normal raw parity is
128/128; final p95/p99 85.775/89.385ms. Faulted runs preserve four PIDs and clear
nonempty temporal stores, resuming on seq30/110 after gaps at seq16/72. All
post-resume histories and observed costmap timestamps belong to the new epoch;
first scenes have fresh six-layer evidence and new DDRNet inference. No result
was published during pauses. GPU calls were idle at injection; mid-compute
interruption is covered by focused tests, not new real-GPU cancellation proof.
Recovery availability differs from accepted-frame processing latency: 150ms gap
takes1.476s to next complete scene; 2.2s gap takes3.857s, including waiting for
keyframe/sensor alignment. Faulted output is76 processed +52 skipped cameras,
zero unaccounted; none is replayed. Its p95/p99 83.163/89.729ms describes accepted
frames only. Timings use temporary stock clocks2610/10251MHz; whole-route
availability, auto-clock and network timing remain unqualified. Real collector,
authentication, cross-host clock mapping, standalone and onboard safety are open.
383 tests pass. Evidence330 artifacts,236 Worker snapshot hashes/114 current
files verified; `.runtime/perception-stage2-resume-worker-20260902T1545MSK/manifest.json`,
SHA-256 `a9449fa4c012e30ce8a06025790c0e110eeebab7c1859fdebf56d5e785961ea3`.
Initial preflight abort (512MiB threshold below known534MiB idle baseline) is
preserved; no models/clocks ran in it. Auto clocks/four scoped services restored,
empty/retired fixture volumes and containers removed; legacy restart loops not
fixed. The earlier CPU-only increment below is historical, not latest status.
No full-source transfer, new model, UI or actuator path is introduced.
Owner clarification and stage-2 continuity increment, 2026-09-02 15:30
(`c31c46c`): temporary input/network loss must not unload the resident models.
Unknown/stale GPU telemetry is not proof of a competing owner. The desired