From 0cc6b963066e79c6868c3bd6b339f58b70815d84 Mon Sep 17 00:00:00 2001 From: DCCONSTRUCTIONS Date: Wed, 2 Sep 2026 19:05:13 +0300 Subject: [PATCH] docs(perception): record full-graph network and recovery evidence --- .../OBSERVATORY_REALTIME_PROFILES_EXECPLAN.md | 12 +- .../0049-stream-first-perception-profiles.md | 60 ++++++- ...EPTION_STREAM_STAGE2_NETWORK_2026-09-02.md | 157 ++++++++++++++++++ 3 files changed, 221 insertions(+), 8 deletions(-) create mode 100644 experiments/perception/PERCEPTION_STREAM_STAGE2_NETWORK_2026-09-02.md diff --git a/docs/OBSERVATORY_REALTIME_PROFILES_EXECPLAN.md b/docs/OBSERVATORY_REALTIME_PROFILES_EXECPLAN.md index fe0c07c..603c9e8 100644 --- a/docs/OBSERVATORY_REALTIME_PROFILES_EXECPLAN.md +++ b/docs/OBSERVATORY_REALTIME_PROFILES_EXECPLAN.md @@ -1,10 +1,14 @@ # Observatory: четыре этапа создания полного real-time Perception-профиля -Дата: 2026-09-01; обновлено 2026-09-02 18:11 МСК. **Этап 1 закрыт; этап 2 в работе. Инкремент 12: внешний gRPC/TLS binary candidate, одноразовый доступ и bounded backpressure.** Код `0310592`. Два CPU-only межмашинных повтора Mac↔Worker через существующий SSH-маршрут дали16/16 точных payload/reply; обрыв2.20s сохранил resident CPU sentinel и локальный lease. Итоговые105 focused tests и20 Worker transport tests PASS. Это не full-graph network qualification: реальный `BinaryGraphBridge`, scene/layer codecs, production выдача grant/clock mapping и standalone ещё не подключены к этому endpoint. Этапы3–4 не начаты. +Дата: 2026-09-01; обновлено 2026-09-02 19:00 МСК. **Этап 1 закрыт; этап 2 в работе. Инкремент 13: полный graph с реальным сетевым входом и выходом.** CPU source/receiver вынесен в отдельный контейнер без GPU; модельный контейнер не получает запись через mounts. Через gRPC/TLS возвращаются настоящая маска512×512, detections/distance/motion и grid/costmap/policy. Обычный1× прогон:128/128 exact raw к reference, p95/p99 до готового consumer-view106.150/115.349ms, max126.910ms (один кадр выше125ms). 76/128 полных свежих scene;52 исходные sensor gaps не скрыты. -Финальный same-Mac RTT min/median/max14.360/23.919/120.815ms (не one-way age/FPS), peak mailbox6,750,559B, RSS57,244KiB, после закрытия0bytes. Source clock и resume proof синтетические, моделей0. Default64KiB HTTP/2 window не выдержал250ms для1,048,595B; fixed1MiB прошёл без изменения deadline. Реальный slow-reader выявил блокирование отправки error status; его deadline и deferred quarantine cleanup исправлены и проверены. Evidence16 artifacts: `.runtime/perception-stage2-grpc-20260902T1500Z/manifest.json`, SHA-256 `adf5eaf092feaed6721f66e2adaceded0cdbf55754e1f9953f54623bfb52d331`. +Отдельный обрыв2.2s:102 результатов +26 учтённых пропусков24–49, resume50 с теми же четырьмя PID и lease generation1; одна очистка temporal history, без backlog. p95/p99 принятых результатов107.098/110.817ms; отдельный gap доступности2.725s. Начальный лишний reconnect из-за раннего init исправлен без увеличения idle-timeout и проверен повтором.146 focused tests PASS,2 больших Worker-only теста пропущены локально. Максимальный ответ460248B остаётся внутри1MiB; peak tracked13,217,791B, после закрытия0. -Следующий проверяемый результат этапа2: подать реальную запись с application-side source через этот endpoint в существующий полный graph, вернуть общие scene/layer outputs с явными размерными/возрастными ограничениями, повторить normal/reconnect/slow-consumer canary последовательно. Транспорт не имеет права вызывать batch materializer или заново запускать модели при сетевой паузе. Текущий outbound cap1MiB требует проверки реальных слоёв/фрагментации; расширять его молча нельзя. После этого — самостоятельный image. Продуктовый LAB path пока не переключён. +Граница доказательства: два контейнера одного Worker с проверенным общим Linux monotonic clock, временные штатные частоты2610/10251MHz; **не Mac↔Worker full-graph, не radio/live и не standalone**. История зависимых слоёв после gap закономерно отличается от uninterrupted reference. Все230 опубликованных scene/масок проверены на стороне приёмника и повторно сверены. [Подробный отчёт](../experiments/perception/PERCEPTION_STREAM_STAGE2_NETWORK_2026-09-02.md). Evidence236 artifacts: `.runtime/perception-stage2-network-graph-20260902T1541Z/manifest.json`, SHA-256 `d1bc7e26850a2d9050ee1d8f8a5ece2e6b8c53e2e7dba381cbd4445d68a5ec17`. + +Следующий проверяемый результат этапа2: application-side выдача grant + измеренное сопоставление часов, полный graph Mac↔Worker и slow-consumer canary, затем самостоятельный image без developer/model mounts. Начальный retry/выдача grant текущего bounded pilot не являются production recovery service. Продуктовый LAB path пока не переключён; этапы3–4 не начаты. Worker services и auto clocks восстановлены, Ollama/Frigate остаются exited/restart=no, временные контейнеры/network/lease удалены. Mac8000 и Worker telemetry работают. + +Предыдущий инкремент12 (`0310592`, `1916122`): CPU-only gRPC/TLS proof Mac↔Worker,16/16 payload/reply; синтетический source clock и CPU sentinel, моделей0. Same-Mac RTT14.360/23.919/120.815ms min/median/max — не one-way age/FPS.105 local и20 Worker tests PASS, проверены bounded slow-reader/quarantine. Evidence `.runtime/perception-stage2-grpc-20260902T1500Z/manifest.json`, SHA-256 `adf5eaf092feaed6721f66e2adaceded0cdbf55754e1f9953f54623bfb52d331`. Предыдущий инкремент11 (`c570f7d`, `2ef8c08`): восстановлены System/Worker telemetry plane и независимый runtime exporter. Телеметрия остаётся отдельным наблюдателем, а не выдающим lease/GPU authority контуром. @@ -336,6 +340,6 @@ EoMT — семейство ViT-моделей сегментации изобр ## Outcomes / retrospective -Этап 1 завершён; этап 2 продолжается. Реальный scoped host collector/control adapter подключён к полному профилю. Normal exact raw128/128, задержка reply2.2s → WAIT/resync с теми же четырьмя PID,26 пропусков без backlog/старой истории,404 focused tests. Сохранены исправленный warmup abort и независимые показатели доступности/accepted-frame latency. Следом внешний data-plane/auth/recovery, затем standalone packaging. Native-host GPU inventory не квалифицирован; Docker scope не выдаётся за полный host audit. Batch/registry/UI и этапы3–4 не начаты, actuation=false. Mac8000=200,8765 закрыт; auto clocks и четыре Worker services восстановлены, Ollama/Frigate exited/restart=no, временных collectors/container/lease нет. +Этап1 завершён; этап2 продолжается. Помимо scoped host control, полный graph теперь связан gRPC/TLS с отдельным CPU source/receiver: normal128/128 exact raw, gap2.2s → WAIT/resync с теми же четырьмя PID/lease и26 учтёнными пропусками.146 focused tests текущего инкремента PASS; ранее404 checks относились к host-control инкременту. Consumer-ready p95/p99 normal106.150/115.349ms; это same-kernel container-network при stock fixed clocks, не Mac↔Worker. Следом межмашинные clock/grant integration и slow consumer, затем standalone packaging. Native-host GPU inventory не квалифицирован; Docker scope не выдаётся за полный host audit. Batch/registry/UI и этапы3–4 не начаты, actuation=false. Mac8000=200,8765 закрыт; auto clocks и четыре Worker services восстановлены, Ollama/Frigate exited/restart=no, временных collectors/container/network/lease/key/grant нет. После этапа 4 здесь будут перечислены digest самостоятельного полного образа, измеренные статусы и ошибки по recordings, реально проверенные source/hardware/config комбинации, состояние сохранённого EoMT-варианта и оставшиеся physical-live/quality/vehicle-integration ограничения. Готовый Docker и готовая автономия не отождествляются. diff --git a/docs/adr/0049-stream-first-perception-profiles.md b/docs/adr/0049-stream-first-perception-profiles.md index b5d0fb2..b4e3dd3 100644 --- a/docs/adr/0049-stream-first-perception-profiles.md +++ b/docs/adr/0049-stream-first-perception-profiles.md @@ -1,9 +1,9 @@ # ADR 0049 — Stream-first full perception profiles -Date: 2026-09-01; updated 2026-09-02 18:11 MSK. Status: stage-1 engineering -baseline complete; stage-2 authenticated gRPC candidate has cross-host CPU proof. -Full-graph network integration, native-host GPU inventory, standalone image and -product cutover remain open; transport proof is not physical-live acceptance. +Date: 2026-09-01; updated 2026-09-02 19:00 MSK. Status: stage-1 engineering +baseline complete; stage-2 full graph has authenticated container-network proof. +Cross-host full-graph clock/grant integration, native-host GPU inventory, +standalone image and product cutover remain open. This is not physical-live acceptance. Execution order remains the four stages in [the ExecPlan](../OBSERVATORY_REALTIME_PROFILES_EXECPLAN.md). @@ -777,3 +777,55 @@ warns that a completed write need not mean network delivery; [channel options](https://github.com/grpc/grpc/blob/master/include/grpc/impl/channel_arg_names.h) documents the fixed lookahead window. Source and actual behavior were checked; these references alone are not a performance qualification. + +## Stage 2 increment 13: full graph over container networking (2026-09-02) + +The same `BinaryGraphInput` decoder and causal sensor cut now feed the full +graph from `NetworkGraphBridge`. Source and receiver execute in a separate +CPU-only container. No camera/index/sensor recording mount exists in the GPU +container. Resident DDRNet/RF-DETR/decoder/TGS children and the local Worker +lease remain owned by the existing controller; remote ingress cannot restart +models or renew the lease. + +The return payload carries the existing scene plus the actual uint8 segmentation +plane. Grid coordinates join the costmap digest. Result1MiB, JSON512KiB and plane +512KiB caps are checked before parsing/retention; total framing also fits1MiB. +The receiver verifies raw mask and all layer hashes and independently derives +freshness/policy. Network arrival, integrity-validation completion and complete +consumer-view readiness are recorded separately. + +The pilot verifies a common Linux boot and zero monotonic namespace offsets. +Its grant file is trusted local control, read-only to the source; source status +uses a separate mount. It is **not** the production grant delivery or cross-host +clock mapping implementation. A foreign clock cannot be silently subtracted. + +Final normal128b:128/128 byte-exact received scenes, raw model/geometry/motion/ +costmap parity to the prior reference, no queue drops, one input epoch, no resets. +Consumer-ready p95/p99/max106.150/115.349/126.910ms. One frame exceeds125ms;76/128 +scenes have complete six-layer freshness, with52 original missing sensor pairs. +Gap128b:2.2s disconnect,102 completed +26 omissions24–49, fresh keyframe/pair +resume50, exactly one temporal reset, same four PIDs and lease generation1. +Consumer-ready p95/p99/max107.098/110.817/117.317ms; availability gap2.725s. +Post-gap history-dependent outputs are not asserted equal to uninterrupted +history. All230 mask/scene receipts and their derived views were rechecked. + +These runs use temporary stock2610/10251MHz clocks,450W,8CPU/8GiB model and +2CPU/1GiB source envelopes. They prove container-network behavior on one +Worker, **not** Mac↔Worker/radio/live latency. Observed result maximum460248B, +tracked peak13,217,791B, residual0. Full-model slow consumer and cross-host +clock/grant integration precede standalone packaging; product cutover remains +closed. Diagnostic startup retry is not a production recovery service. + +Source init originally preceded the delayed start by2s and triggered a spurious +idle timeout. The source now connects100ms before release, without relaxing +the timeout. Final normal/gap have zero/one intended recovery. Missing runner, +read-only mountpoint and launcher ExitCode failures are retained as rejected +attempts.146 focused tests pass;2 larger Worker-only tests skipped on Mac. + +Evidence236 artifacts at +`.runtime/perception-stage2-network-graph-20260902T1541Z/manifest.json`, SHA256 +`d1bc7e26850a2d9050ee1d8f8a5ece2e6b8c53e2e7dba381cbd4445d68a5ec17`. +See [the detailed report](../../experiments/perception/PERCEPTION_STREAM_STAGE2_NETWORK_2026-09-02.md). +Four prior services and automatic clocks restored; transient containers, +networks, lease volumes, keys and grants removed. Ollama/Frigate stay disabled. +Mac8000 and Worker telemetry remain available; no8765 or external deployment. diff --git a/experiments/perception/PERCEPTION_STREAM_STAGE2_NETWORK_2026-09-02.md b/experiments/perception/PERCEPTION_STREAM_STAGE2_NETWORK_2026-09-02.md new file mode 100644 index 0000000..82f858b --- /dev/null +++ b/experiments/perception/PERCEPTION_STREAM_STAGE2_NETWORK_2026-09-02.md @@ -0,0 +1,157 @@ +# Stage 2, increment 13: complete graph over a bounded network stream + +2026-09-02, 19:00 MSK. Implementation/test commit: `6469505`. +This is an increment of the existing four-stage +[ExecPlan](../../docs/OBSERVATORY_REALTIME_PROFILES_EXECPLAN.md), not another stage +or a standalone profile release. No product LAB/registry cutover or actuation. + +## Result and scope + +The real DDRNet-39/RF-DETR/LiDAR-distance/motion/TRAVEL-TGS/costmap/policy graph +now takes input and returns results over the authenticated gRPC/TLS candidate. +An independent **CPU-only** container owns the recorded source and result +consumer. The one GPU container has **no camera index, camera directory or +sensor recording mount**. It retains only the calibration/model/config/code +mounts needed by the existing diagnostic graph. Models are resident and GPU +inferences remain serialized; no second full profile runs alongside it. + +Both containers share the same Linux kernel boot and zero time-namespace +offsets. The pilot explicitly checks this before subtracting monotonic times. +This is a real container-network proof, **not Mac↔Worker latency**, not hardware +sensor synchronization and not a 300–500 Mbit/s radio-link qualification. + +The source reads the original camera fragments and normalized point/pose +archive incrementally at original 1× arrival time. Only a bounded 128-frame +prefix was exercised. Existing archive assets were not uploaded again. The +source read 545,590 of 10,751,258 point rows, not the whole recording. + +## Implementation + +- `BinaryGraphInput` is the unchanged decoder/causal sensor-window assembly + shared by the original local IPC bridge and `NetworkGraphBridge`. There is + no second model or geometry implementation behind the network path. +- `pilot_grpc_source.py` shares raw event serialization with `RecordingSource`. + It neither owns nor renews the Worker lease. Source gaps discard observations, + never change their timestamps or accelerate a catch-up replay. +- The local controller issues a new one-shot grant only after old ingress and + work have drained. Grants are read-only to the source; source terminal status + is a separate read-only mount to the graph. These are temporary trusted pilot + control files, not a production application grant API. +- `streaming_scene_payload.py` frames the real scene JSON plus a **512×512 + uint8 mask**, not just hashes. The scene includes detections, metric distances, + motion, actual grid coordinates/material/state, cell evidence and policy. + Grid coordinates participate in the costmap layer digest. The receiver checks + binding, sequence, mask and six-layer payload integrity, then derives its own + current freshness/policy view without rewriting published evidence. +- Output remains capped at 1 MiB, with a 512 KiB JSON bound and 512 KiB plane + bound plus total-length validation. Duplicate JSON keys/non-finite constants, + bad shape/hash/binding and control-authority escalation are rejected. +- Socket EOF on an established stream preserves resident models and lease, + clears temporal history, and waits for a fresh decoded keyframe and causal + sensor pair. The source adapter is still a bounded diagnostic: production + startup/grant delivery retry and cross-host clock mapping remain open. + +## Sequential Worker measurements + +RTX 4090, temporary stock SM/memory clocks **2610/10251 MHz**, unchanged 450 W +limit. GPU container 8 CPU / 8 GiB; source container 2 CPU / 1 GiB and explicit +`runc` without GPU devices. Separate internal Docker network; no host listener. +These are conditional results, not qualification under automatic downclocking. + +| Measurement | Normal `normal128b` | Gap `gap128b` | +| --- | ---: | ---: | +| Original camera observations | 128 | 128 | +| Validated mask + scene receipts | 128 | 102 | +| Accounted omissions | 0 | 26, sequences 24–49 | +| Result queue drops / unaccounted | 0 / 0 | 0 / 0 | +| Input epochs / temporal resets | 1 / 0 | 2 / 1 | +| Consumer-ready p95 / p99 | 106.150 / 115.349 ms | 107.098 / 110.817 ms | +| Consumer-ready maximum | 126.910 ms | 117.317 ms | +| Frames above 125 ms | 1 | 0 | +| Complete six-layer freshness at assessment | 76 / 128 | 58 / 102 | +| Receipts before source EOF | 127 | 101 | +| Source release lag maximum | 9.333 ms | 9.231 ms | +| Maximum result bytes | 459,608 | 460,248 | +| Peak tracked bytes / residual | 13,217,791 / 0 | 13,217,791 / 0 | +| Peak observed VRAM | 2,369 MiB | 2,372 MiB | +| Peak GPU-container memory | 3,348.31 MiB | 3,368.86 MiB | +| Shutdown/resource retirement | 4.067 s | 4.065 s | + +Three different receipt clocks are retained: complete network payload arrival; +completed identity/payload validation; consumer view ready after freshness and +per-cell policy derivation. The table uses the **last** boundary. Freshness is +assessed at the explicitly recorded check instant, not retimestamped when the +view calculation ends. No UI rendering or motor/controller transport is timed. + +The 2.2 s injected disconnect resumed at camera sequence 50. All four resident +PIDs (11, 40, 48, 49) remained alive, lease generation stayed 1, and exactly one +temporal reset emptied surface/body/component/rolling state. The source skipped +23 cameras while disconnected; the decoder skipped 47–49 until a fresh random +access frame/pair. Time between consumer-ready scenes 23 and 50 was **2.725 s**. +This availability gap is separate from latency of accepted frames. + +## Integrity and interpretation + +All 230 final scene JSON documents matched the corresponding Worker-published +bytes exactly and were independently rechecked against mask hashes, six layer +digests, freshness and derived policy counts. All 230 decoded BGR hashes match +the earlier reference at their source sequence. + +Normal run: all 128 segmentation masks, detections, metric observations, tracks, +threats, material maps, sensor lineage/bindings and raw costmap-state hashes are +exact to the previous full-graph reference. It still has 52 original unavailable +sensor pairs; no fresh timestamps or permissive policies were invented. + +Gap run: segmentation and detections match 102/102 reference frames. After the +reset, history-dependent observations/tracks/costmap can and do differ from the +uninterrupted reference (observations 94/102 exact, tracks/threats 24/102, material +99/102, raw costmap 85/102). This is not whole-scene parity across missing data. +It is explicit reset-and-rebuild behavior, with no reuse of pre-gap history. + +## Rejected attempts and corrections + +All attempts and their diagnostics are retained, not overwritten: + +1. `canary32`: missing pinned DDRNet runner in the diagnostic pack; source never + started. `canary32b`: attempted nested mount lacked a mountpoint inside a + read-only `/probe`; no profile process started. The corrected pack contains + the unchanged hash-verified runner. The second attempt's empty lease volume + was independently inspected before removal. +2. `canary32c` delivered 32/32 real results; the PowerShell launcher incorrectly + read a null exited-process `ExitCode`. Caching its process handle before exit + corrected the diagnostic, without changing model execution. +3. Initial `normal128` and `gap128` exposed an extra startup reconnect: init was + sent two seconds before the delayed source-clock start, tripping the 2 s idle + deadline. Source connection now opens 100 ms before release; the timeout was + **not** enlarged. Final normal/gap repeats have respectively zero/one genuine + recovery and no transport errors. The source-start regression has a test. + +## Checks, evidence and remaining gates + +146 focused tests PASS, 2 Worker-only large-load tests skipped locally; Ruff, +format, targeted mypy and diff checks PASS. No local build/browser/load test at +Mac memory pressure 2. The earlier UI visual acceptance remains pending. + +Session evidence: `.runtime/perception-stage2-network-graph-20260902T1541Z/`. +`review.json` reconciles source, graph and receiver; `verify.py` rechecks one +scene at a time. `manifest.json` hashes code, logs, reports, accepted and rejected +attempts (236 artifacts), SHA-256 +`d1bc7e26850a2d9050ee1d8f8a5ece2e6b8c53e2e7dba381cbd4445d68a5ec17`. +Code-v3 archive SHA-256: +`3e9091524f62cdd528846b2d1e97e9107133c9cd267549f324991c0e63b184fc`. +Pinned DDRNet runner additionally inventoried: +`b18ad60f277eea69a240a28f290611b94627fb9707faf1bb3e6e22102dad67c1`. +Final evidence archive SHA-256: +`58ca7f1f59a79bce436d4e029a063563da3081c0c1cf47b5fbbdf49931f0fdd2`. + +Temporary containers, networks and isolated lease volumes are removed; four +prior services restored, Triton healthy, GPU back to automatic clocks. Ollama +and Frigate remain stopped with restart=no. Ephemeral TLS private key and grants +were removed, not archived. Canonical Mac8000 remains running with fresh Worker +telemetry; no8765 listener. No push or external/Synology deployment. + +Next within **stage 2**: authenticated application-side grant delivery and +cross-host source clock mapping, then the full graph Mac↔Worker and a deliberately +slow result consumer, followed by the standalone image. Packaging must remove +the current developer/model dependency mounts. Native-host GPU inventory, +multiple recordings, physical-live/field quality and autonomy remain unproved.