docs(perception): record full-graph network and recovery evidence

docs/OBSERVATORY_REALTIME_PROFILES_EXECPLAN.md

@@ -1,10 +1,14 @@
 # Observatory: четыре этапа создания полного real-time Perception-профиля
 
-Дата: 2026-09-01; обновлено 2026-09-02 18:11 МСК. **Этап 1 закрыт; этап 2 в работе. Инкремент 12: внешний gRPC/TLS binary candidate, одноразовый доступ и bounded backpressure.** Код `0310592`. Два CPU-only межмашинных повтора Mac↔Worker через существующий SSH-маршрут дали16/16 точных payload/reply; обрыв2.20s сохранил resident CPU sentinel и локальный lease. Итоговые105 focused tests и20 Worker transport tests PASS. Это не full-graph network qualification: реальный `BinaryGraphBridge`, scene/layer codecs, production выдача grant/clock mapping и standalone ещё не подключены к этому endpoint. Этапы3–4 не начаты.
+Дата: 2026-09-01; обновлено 2026-09-02 19:00 МСК. **Этап 1 закрыт; этап 2 в работе. Инкремент 13: полный graph с реальным сетевым входом и выходом.** CPU source/receiver вынесен в отдельный контейнер без GPU; модельный контейнер не получает запись через mounts. Через gRPC/TLS возвращаются настоящая маска512×512, detections/distance/motion и grid/costmap/policy. Обычный1× прогон:128/128 exact raw к reference, p95/p99 до готового consumer-view106.150/115.349ms, max126.910ms (один кадр выше125ms). 76/128 полных свежих scene;52 исходные sensor gaps не скрыты.
 
-Финальный same-Mac RTT min/median/max14.360/23.919/120.815ms (не one-way age/FPS), peak mailbox6,750,559B, RSS57,244KiB, после закрытия0bytes. Source clock и resume proof синтетические, моделей0. Default64KiB HTTP/2 window не выдержал250ms для1,048,595B; fixed1MiB прошёл без изменения deadline. Реальный slow-reader выявил блокирование отправки error status; его deadline и deferred quarantine cleanup исправлены и проверены. Evidence16 artifacts: `.runtime/perception-stage2-grpc-20260902T1500Z/manifest.json`, SHA-256 `adf5eaf092feaed6721f66e2adaceded0cdbf55754e1f9953f54623bfb52d331`.
+Отдельный обрыв2.2s:102 результатов +26 учтённых пропусков24–49, resume50 с теми же четырьмя PID и lease generation1; одна очистка temporal history, без backlog. p95/p99 принятых результатов107.098/110.817ms; отдельный gap доступности2.725s. Начальный лишний reconnect из-за раннего init исправлен без увеличения idle-timeout и проверен повтором.146 focused tests PASS,2 больших Worker-only теста пропущены локально. Максимальный ответ460248B остаётся внутри1MiB; peak tracked13,217,791B, после закрытия0.
 
-Следующий проверяемый результат этапа2: подать реальную запись с application-side source через этот endpoint в существующий полный graph, вернуть общие scene/layer outputs с явными размерными/возрастными ограничениями, повторить normal/reconnect/slow-consumer canary последовательно. Транспорт не имеет права вызывать batch materializer или заново запускать модели при сетевой паузе. Текущий outbound cap1MiB требует проверки реальных слоёв/фрагментации; расширять его молча нельзя. После этого — самостоятельный image. Продуктовый LAB path пока не переключён.
+Граница доказательства: два контейнера одного Worker с проверенным общим Linux monotonic clock, временные штатные частоты2610/10251MHz; **не Mac↔Worker full-graph, не radio/live и не standalone**. История зависимых слоёв после gap закономерно отличается от uninterrupted reference. Все230 опубликованных scene/масок проверены на стороне приёмника и повторно сверены. [Подробный отчёт](../experiments/perception/PERCEPTION_STREAM_STAGE2_NETWORK_2026-09-02.md). Evidence236 artifacts: `.runtime/perception-stage2-network-graph-20260902T1541Z/manifest.json`, SHA-256 `d1bc7e26850a2d9050ee1d8f8a5ece2e6b8c53e2e7dba381cbd4445d68a5ec17`.
+
+Следующий проверяемый результат этапа2: application-side выдача grant + измеренное сопоставление часов, полный graph Mac↔Worker и slow-consumer canary, затем самостоятельный image без developer/model mounts. Начальный retry/выдача grant текущего bounded pilot не являются production recovery service. Продуктовый LAB path пока не переключён; этапы3–4 не начаты. Worker services и auto clocks восстановлены, Ollama/Frigate остаются exited/restart=no, временные контейнеры/network/lease удалены. Mac8000 и Worker telemetry работают.
+
+Предыдущий инкремент12 (`0310592`, `1916122`): CPU-only gRPC/TLS proof Mac↔Worker,16/16 payload/reply; синтетический source clock и CPU sentinel, моделей0. Same-Mac RTT14.360/23.919/120.815ms min/median/max — не one-way age/FPS.105 local и20 Worker tests PASS, проверены bounded slow-reader/quarantine. Evidence `.runtime/perception-stage2-grpc-20260902T1500Z/manifest.json`, SHA-256 `adf5eaf092feaed6721f66e2adaceded0cdbf55754e1f9953f54623bfb52d331`.
 
 Предыдущий инкремент11 (`c570f7d`, `2ef8c08`): восстановлены System/Worker telemetry plane и независимый runtime exporter. Телеметрия остаётся отдельным наблюдателем, а не выдающим lease/GPU authority контуром.
 
@@ -336,6 +340,6 @@ EoMT — семейство ViT-моделей сегментации изобр
 
 ## Outcomes / retrospective
 
-Этап 1 завершён; этап 2 продолжается. Реальный scoped host collector/control adapter подключён к полному профилю. Normal exact raw128/128, задержка reply2.2s → WAIT/resync с теми же четырьмя PID,26 пропусков без backlog/старой истории,404 focused tests. Сохранены исправленный warmup abort и независимые показатели доступности/accepted-frame latency. Следом внешний data-plane/auth/recovery, затем standalone packaging. Native-host GPU inventory не квалифицирован; Docker scope не выдаётся за полный host audit. Batch/registry/UI и этапы3–4 не начаты, actuation=false. Mac8000=200,8765 закрыт; auto clocks и четыре Worker services восстановлены, Ollama/Frigate exited/restart=no, временных collectors/container/lease нет.
+Этап1 завершён; этап2 продолжается. Помимо scoped host control, полный graph теперь связан gRPC/TLS с отдельным CPU source/receiver: normal128/128 exact raw, gap2.2s → WAIT/resync с теми же четырьмя PID/lease и26 учтёнными пропусками.146 focused tests текущего инкремента PASS; ранее404 checks относились к host-control инкременту. Consumer-ready p95/p99 normal106.150/115.349ms; это same-kernel container-network при stock fixed clocks, не Mac↔Worker. Следом межмашинные clock/grant integration и slow consumer, затем standalone packaging. Native-host GPU inventory не квалифицирован; Docker scope не выдаётся за полный host audit. Batch/registry/UI и этапы3–4 не начаты, actuation=false. Mac8000=200,8765 закрыт; auto clocks и четыре Worker services восстановлены, Ollama/Frigate exited/restart=no, временных collectors/container/network/lease/key/grant нет.
 
 После этапа 4 здесь будут перечислены digest самостоятельного полного образа, измеренные статусы и ошибки по recordings, реально проверенные source/hardware/config комбинации, состояние сохранённого EoMT-варианта и оставшиеся physical-live/quality/vehicle-integration ограничения. Готовый Docker и готовая автономия не отождествляются.

docs/adr/0049-stream-first-perception-profiles.md

@@ -1,9 +1,9 @@
 # ADR 0049 — Stream-first full perception profiles
 
-Date: 2026-09-01; updated 2026-09-02 18:11 MSK. Status: stage-1 engineering
+Date: 2026-09-01; updated 2026-09-02 19:00 MSK. Status: stage-1 engineering
-baseline complete; stage-2 authenticated gRPC candidate has cross-host CPU proof.
+baseline complete; stage-2 full graph has authenticated container-network proof.
-Full-graph network integration, native-host GPU inventory, standalone image and
+Cross-host full-graph clock/grant integration, native-host GPU inventory,
-product cutover remain open; transport proof is not physical-live acceptance.
+standalone image and product cutover remain open. This is not physical-live acceptance.
 Execution order remains the four stages in
 [the ExecPlan](../OBSERVATORY_REALTIME_PROFILES_EXECPLAN.md).
 
@@ -777,3 +777,55 @@ warns that a completed write need not mean network delivery;
 [channel options](https://github.com/grpc/grpc/blob/master/include/grpc/impl/channel_arg_names.h)
 documents the fixed lookahead window. Source and actual behavior were checked;
 these references alone are not a performance qualification.
+
+## Stage 2 increment 13: full graph over container networking (2026-09-02)
+
+The same `BinaryGraphInput` decoder and causal sensor cut now feed the full
+graph from `NetworkGraphBridge`. Source and receiver execute in a separate
+CPU-only container. No camera/index/sensor recording mount exists in the GPU
+container. Resident DDRNet/RF-DETR/decoder/TGS children and the local Worker
+lease remain owned by the existing controller; remote ingress cannot restart
+models or renew the lease.
+
+The return payload carries the existing scene plus the actual uint8 segmentation
+plane. Grid coordinates join the costmap digest. Result1MiB, JSON512KiB and plane
+512KiB caps are checked before parsing/retention; total framing also fits1MiB.
+The receiver verifies raw mask and all layer hashes and independently derives
+freshness/policy. Network arrival, integrity-validation completion and complete
+consumer-view readiness are recorded separately.
+
+The pilot verifies a common Linux boot and zero monotonic namespace offsets.
+Its grant file is trusted local control, read-only to the source; source status
+uses a separate mount. It is **not** the production grant delivery or cross-host
+clock mapping implementation. A foreign clock cannot be silently subtracted.
+
+Final normal128b:128/128 byte-exact received scenes, raw model/geometry/motion/
+costmap parity to the prior reference, no queue drops, one input epoch, no resets.
+Consumer-ready p95/p99/max106.150/115.349/126.910ms. One frame exceeds125ms;76/128
+scenes have complete six-layer freshness, with52 original missing sensor pairs.
+Gap128b:2.2s disconnect,102 completed +26 omissions24–49, fresh keyframe/pair
+resume50, exactly one temporal reset, same four PIDs and lease generation1.
+Consumer-ready p95/p99/max107.098/110.817/117.317ms; availability gap2.725s.
+Post-gap history-dependent outputs are not asserted equal to uninterrupted
+history. All230 mask/scene receipts and their derived views were rechecked.
+
+These runs use temporary stock2610/10251MHz clocks,450W,8CPU/8GiB model and
+2CPU/1GiB source envelopes. They prove container-network behavior on one
+Worker, **not** Mac↔Worker/radio/live latency. Observed result maximum460248B,
+tracked peak13,217,791B, residual0. Full-model slow consumer and cross-host
+clock/grant integration precede standalone packaging; product cutover remains
+closed. Diagnostic startup retry is not a production recovery service.
+
+Source init originally preceded the delayed start by2s and triggered a spurious
+idle timeout. The source now connects100ms before release, without relaxing
+the timeout. Final normal/gap have zero/one intended recovery. Missing runner,
+read-only mountpoint and launcher ExitCode failures are retained as rejected
+attempts.146 focused tests pass;2 larger Worker-only tests skipped on Mac.
+
+Evidence236 artifacts at
+`.runtime/perception-stage2-network-graph-20260902T1541Z/manifest.json`, SHA256
+`d1bc7e26850a2d9050ee1d8f8a5ece2e6b8c53e2e7dba381cbd4445d68a5ec17`.
+See [the detailed report](../../experiments/perception/PERCEPTION_STREAM_STAGE2_NETWORK_2026-09-02.md).
+Four prior services and automatic clocks restored; transient containers,
+networks, lease volumes, keys and grants removed. Ollama/Frigate stay disabled.
+Mac8000 and Worker telemetry remain available; no8765 or external deployment.

experiments/perception/PERCEPTION_STREAM_STAGE2_NETWORK_2026-09-02.md

@@ -0,0 +1,157 @@
+# Stage 2, increment 13: complete graph over a bounded network stream
+
+2026-09-02, 19:00 MSK. Implementation/test commit: `6469505`.
+This is an increment of the existing four-stage
+[ExecPlan](../../docs/OBSERVATORY_REALTIME_PROFILES_EXECPLAN.md), not another stage
+or a standalone profile release. No product LAB/registry cutover or actuation.
+
+## Result and scope
+
+The real DDRNet-39/RF-DETR/LiDAR-distance/motion/TRAVEL-TGS/costmap/policy graph
+now takes input and returns results over the authenticated gRPC/TLS candidate.
+An independent **CPU-only** container owns the recorded source and result
+consumer. The one GPU container has **no camera index, camera directory or
+sensor recording mount**. It retains only the calibration/model/config/code
+mounts needed by the existing diagnostic graph. Models are resident and GPU
+inferences remain serialized; no second full profile runs alongside it.
+
+Both containers share the same Linux kernel boot and zero time-namespace
+offsets. The pilot explicitly checks this before subtracting monotonic times.
+This is a real container-network proof, **not Mac↔Worker latency**, not hardware
+sensor synchronization and not a 300–500 Mbit/s radio-link qualification.
+
+The source reads the original camera fragments and normalized point/pose
+archive incrementally at original 1× arrival time. Only a bounded 128-frame
+prefix was exercised. Existing archive assets were not uploaded again. The
+source read 545,590 of 10,751,258 point rows, not the whole recording.
+
+## Implementation
+
+- `BinaryGraphInput` is the unchanged decoder/causal sensor-window assembly
+  shared by the original local IPC bridge and `NetworkGraphBridge`. There is
+  no second model or geometry implementation behind the network path.
+- `pilot_grpc_source.py` shares raw event serialization with `RecordingSource`.
+  It neither owns nor renews the Worker lease. Source gaps discard observations,
+  never change their timestamps or accelerate a catch-up replay.
+- The local controller issues a new one-shot grant only after old ingress and
+  work have drained. Grants are read-only to the source; source terminal status
+  is a separate read-only mount to the graph. These are temporary trusted pilot
+  control files, not a production application grant API.
+- `streaming_scene_payload.py` frames the real scene JSON plus a **512×512
+  uint8 mask**, not just hashes. The scene includes detections, metric distances,
+  motion, actual grid coordinates/material/state, cell evidence and policy.
+  Grid coordinates participate in the costmap layer digest. The receiver checks
+  binding, sequence, mask and six-layer payload integrity, then derives its own
+  current freshness/policy view without rewriting published evidence.
+- Output remains capped at 1 MiB, with a 512 KiB JSON bound and 512 KiB plane
+  bound plus total-length validation. Duplicate JSON keys/non-finite constants,
+  bad shape/hash/binding and control-authority escalation are rejected.
+- Socket EOF on an established stream preserves resident models and lease,
+  clears temporal history, and waits for a fresh decoded keyframe and causal
+  sensor pair. The source adapter is still a bounded diagnostic: production
+  startup/grant delivery retry and cross-host clock mapping remain open.
+
+## Sequential Worker measurements
+
+RTX 4090, temporary stock SM/memory clocks **2610/10251 MHz**, unchanged 450 W
+limit. GPU container 8 CPU / 8 GiB; source container 2 CPU / 1 GiB and explicit
+`runc` without GPU devices. Separate internal Docker network; no host listener.
+These are conditional results, not qualification under automatic downclocking.
+
+| Measurement | Normal `normal128b` | Gap `gap128b` |
+| --- | ---: | ---: |
+| Original camera observations | 128 | 128 |
+| Validated mask + scene receipts | 128 | 102 |
+| Accounted omissions | 0 | 26, sequences 24–49 |
+| Result queue drops / unaccounted | 0 / 0 | 0 / 0 |
+| Input epochs / temporal resets | 1 / 0 | 2 / 1 |
+| Consumer-ready p95 / p99 | 106.150 / 115.349 ms | 107.098 / 110.817 ms |
+| Consumer-ready maximum | 126.910 ms | 117.317 ms |
+| Frames above 125 ms | 1 | 0 |
+| Complete six-layer freshness at assessment | 76 / 128 | 58 / 102 |
+| Receipts before source EOF | 127 | 101 |
+| Source release lag maximum | 9.333 ms | 9.231 ms |
+| Maximum result bytes | 459,608 | 460,248 |
+| Peak tracked bytes / residual | 13,217,791 / 0 | 13,217,791 / 0 |
+| Peak observed VRAM | 2,369 MiB | 2,372 MiB |
+| Peak GPU-container memory | 3,348.31 MiB | 3,368.86 MiB |
+| Shutdown/resource retirement | 4.067 s | 4.065 s |
+
+Three different receipt clocks are retained: complete network payload arrival;
+completed identity/payload validation; consumer view ready after freshness and
+per-cell policy derivation. The table uses the **last** boundary. Freshness is
+assessed at the explicitly recorded check instant, not retimestamped when the
+view calculation ends. No UI rendering or motor/controller transport is timed.
+
+The 2.2 s injected disconnect resumed at camera sequence 50. All four resident
+PIDs (11, 40, 48, 49) remained alive, lease generation stayed 1, and exactly one
+temporal reset emptied surface/body/component/rolling state. The source skipped
+23 cameras while disconnected; the decoder skipped 47–49 until a fresh random
+access frame/pair. Time between consumer-ready scenes 23 and 50 was **2.725 s**.
+This availability gap is separate from latency of accepted frames.
+
+## Integrity and interpretation
+
+All 230 final scene JSON documents matched the corresponding Worker-published
+bytes exactly and were independently rechecked against mask hashes, six layer
+digests, freshness and derived policy counts. All 230 decoded BGR hashes match
+the earlier reference at their source sequence.
+
+Normal run: all 128 segmentation masks, detections, metric observations, tracks,
+threats, material maps, sensor lineage/bindings and raw costmap-state hashes are
+exact to the previous full-graph reference. It still has 52 original unavailable
+sensor pairs; no fresh timestamps or permissive policies were invented.
+
+Gap run: segmentation and detections match 102/102 reference frames. After the
+reset, history-dependent observations/tracks/costmap can and do differ from the
+uninterrupted reference (observations 94/102 exact, tracks/threats 24/102, material
+99/102, raw costmap 85/102). This is not whole-scene parity across missing data.
+It is explicit reset-and-rebuild behavior, with no reuse of pre-gap history.
+
+## Rejected attempts and corrections
+
+All attempts and their diagnostics are retained, not overwritten:
+
+1. `canary32`: missing pinned DDRNet runner in the diagnostic pack; source never
+   started. `canary32b`: attempted nested mount lacked a mountpoint inside a
+   read-only `/probe`; no profile process started. The corrected pack contains
+   the unchanged hash-verified runner. The second attempt's empty lease volume
+   was independently inspected before removal.
+2. `canary32c` delivered 32/32 real results; the PowerShell launcher incorrectly
+   read a null exited-process `ExitCode`. Caching its process handle before exit
+   corrected the diagnostic, without changing model execution.
+3. Initial `normal128` and `gap128` exposed an extra startup reconnect: init was
+   sent two seconds before the delayed source-clock start, tripping the 2 s idle
+   deadline. Source connection now opens 100 ms before release; the timeout was
+   **not** enlarged. Final normal/gap repeats have respectively zero/one genuine
+   recovery and no transport errors. The source-start regression has a test.
+
+## Checks, evidence and remaining gates
+
+146 focused tests PASS, 2 Worker-only large-load tests skipped locally; Ruff,
+format, targeted mypy and diff checks PASS. No local build/browser/load test at
+Mac memory pressure 2. The earlier UI visual acceptance remains pending.
+
+Session evidence: `.runtime/perception-stage2-network-graph-20260902T1541Z/`.
+`review.json` reconciles source, graph and receiver; `verify.py` rechecks one
+scene at a time. `manifest.json` hashes code, logs, reports, accepted and rejected
+attempts (236 artifacts), SHA-256
+`d1bc7e26850a2d9050ee1d8f8a5ece2e6b8c53e2e7dba381cbd4445d68a5ec17`.
+Code-v3 archive SHA-256:
+`3e9091524f62cdd528846b2d1e97e9107133c9cd267549f324991c0e63b184fc`.
+Pinned DDRNet runner additionally inventoried:
+`b18ad60f277eea69a240a28f290611b94627fb9707faf1bb3e6e22102dad67c1`.
+Final evidence archive SHA-256:
+`58ca7f1f59a79bce436d4e029a063563da3081c0c1cf47b5fbbdf49931f0fdd2`.
+
+Temporary containers, networks and isolated lease volumes are removed; four
+prior services restored, Triton healthy, GPU back to automatic clocks. Ollama
+and Frigate remain stopped with restart=no. Ephemeral TLS private key and grants
+were removed, not archived. Canonical Mac8000 remains running with fresh Worker
+telemetry; no8765 listener. No push or external/Synology deployment.
+
+Next within **stage 2**: authenticated application-side grant delivery and
+cross-host source clock mapping, then the full graph Mac↔Worker and a deliberately
+slow result consumer, followed by the standalone image. Packaging must remove
+the current developer/model dependency mounts. Native-host GPU inventory,
+multiple recordings, physical-live/field quality and autonomy remain unproved.