docs(perception): record cross-host canary failure and joint startup gate

docs/OBSERVATORY_REALTIME_PROFILES_EXECPLAN.md

@@ -1,6 +1,10 @@
 # Observatory: четыре этапа создания полного real-time Perception-профиля
 
-Дата: 2026-09-01; обновлено 2026-09-02 19:42 МСК. **Этап 1 закрыт; этап 2 в работе. Инкремент 14: межмашинная выдача grant и измеренные границы часов.** Новый TLS control endpoint доставляет уже выданный Worker-контроллером одноразовый data grant, но не умеет получать/продлевать GPU lease, выбирать epoch, перезапускать модели или actuate. Четыре timestamp дают диапазон remote-minus-local без предположения симметрии сети; до16 наблюдений, expiry2s, нарушение envelope карантинит clock session.
+Дата: 2026-09-01; обновлено 2026-09-02 20:37 МСК. **Этап 1 закрыт; этап 2 в работе. Инкремент 15: полный Mac↔Worker graph подключён к continuous clock/freshness; canary FAIL.** Добавлены ответное подтверждение часов Worker, immutable source anchor, uncertainty для всех слоёв/ячеек и WAIT без потери локального владельца. Исправлены EOF ordering, повторное чтение часов внутри одного результата и ранний disconnect до application OPEN.297 local/173 Worker tests PASS; ранний reconnect дополнительно12/12 PASS.
+
+Финальный32-кадровый1× canary:13 кадров пропущены при ожидании допуска,17 при keyframe/sensor resync, приняты только30–31; оба результата после конца короткого source window,143.881/155.743ms. Все пропуски учтены, но availability/latency/result-before-EOF gates не пройдены.78 clock samples,28 ready; условная uncertainty median5.312ms при неизменном пределе5ms. Peak VRAM2365MiB — не свидетельство предела24GiB. Более ранняя попытка вернула30 exact результатов, но имела2 compute discards и EOF error; её нельзя выдавать за финальный PASS. [Отчёт](../experiments/perception/PERCEPTION_STREAM_STAGE2_CROSSHOST_GRAPH_2026-09-02.md), код `35b6cd9`, evidence198 artifacts/manifest `7b435cac111b3e0e789aa137ba064651e6d344a5f8fb8fa35b0abbd0e7cb1d74`.
+
+Предыдущий инкремент14: межмашинная выдача grant и измеренные границы часов. TLS control endpoint доставляет уже выданный Worker-контроллером одноразовый data grant, но не умеет получать/продлевать GPU lease, выбирать epoch, перезапускать модели или actuate. Четыре timestamp дают диапазон remote-minus-local без предположения симметрии сети; до16 наблюдений, expiry2s, нарушение envelope карантинит clock session.
 
 CPU-only Mac↔Worker006 proof:72/72 timestamp-наблюдения независимо пересчитаны,60/72 соответствуют условному uncertainty≤5ms; min/median/max3.883/4.469/27.118ms. Восемь synthetic32KiB events exact8/8. Обрыв2.202s сохранил PID/lease1, старое clock evidence истекло, новый epoch/grant восстановили вход. Это доказывает механизм WAIT/recovery, но не full-graph cross-host freshness: readiness переключалась и после warmup. Поэтому5ms не ослаблен, а модельный прогон с неподключённым uncertainty gate не запускался. [Отчёт](../experiments/perception/PERCEPTION_STREAM_STAGE2_CONTROL_CLOCK_2026-09-02.md); evidence manifest SHA-256 `cd479c577fa3da78b1b01a6a90e03e8c742ecb53561e606d939408120a53a622`.
 
@@ -10,7 +14,7 @@ CPU-only Mac↔Worker006 proof:72/72 timestamp-наблюдения незави
 
 Граница доказательства: два контейнера одного Worker с проверенным общим Linux monotonic clock, временные штатные частоты2610/10251MHz; **не Mac↔Worker full-graph, не radio/live и не standalone**. История зависимых слоёв после gap закономерно отличается от uninterrupted reference. Все230 опубликованных scene/масок проверены на стороне приёмника и повторно сверены. [Подробный отчёт](../experiments/perception/PERCEPTION_STREAM_STAGE2_NETWORK_2026-09-02.md). Evidence236 artifacts: `.runtime/perception-stage2-network-graph-20260902T1541Z/manifest.json`, SHA-256 `d1bc7e26850a2d9050ee1d8f8a5ece2e6b8c53e2e7dba381cbd4445d68a5ec17`.
 
-Следующий проверяемый результат этапа2: подключить continuous clock-readiness/uncertainty к полному graph Mac↔Worker, выполнить его и slow-consumer canary, затем самостоятельный image без developer/model mounts. Начальный retry текущих bounded pilots не является production recovery service. Продуктовый LAB path пока не переключён; этапы3–4 не начаты. CPU-проба не меняла GPU/clocks/services; Ollama/Frigate остаются exited/restart=no, временные контейнер/listener/key/bootstrap удалены. Mac8000 и Worker telemetry работают.
+Следующий проверяемый результат этапа2: отделить двусторонний clock warmup от фиксации source anchor, подтвердить готовность и Mac, и Worker до старта1×; текущий фиксированный2s lead этого не доказывает. После старта clock не переносить, backlog не догонять. Снять raw probe/ack timings, получить стабильный короткий canary; затем controlled cross-host gap/slow-consumer и standalone image без developer/model mounts. Начальный retry bounded pilots не является production recovery service. Продуктовый LAB path не переключён; этапы3–4 не начаты. GPU/clocks после последовательных замеров восстановлены, Ollama/Frigate exited/restart=no, временные контейнеры/volumes/listener/key/bootstrap удалены. Mac8000 и Worker telemetry работают.
 
 Предыдущий инкремент12 (`0310592`, `1916122`): CPU-only gRPC/TLS proof Mac↔Worker,16/16 payload/reply; синтетический source clock и CPU sentinel, моделей0. Same-Mac RTT14.360/23.919/120.815ms min/median/max — не one-way age/FPS.105 local и20 Worker tests PASS, проверены bounded slow-reader/quarantine. Evidence `.runtime/perception-stage2-grpc-20260902T1500Z/manifest.json`, SHA-256 `adf5eaf092feaed6721f66e2adaceded0cdbf55754e1f9953f54623bfb52d331`.
 
@@ -241,6 +245,8 @@ EoMT — семейство ViT-моделей сегментации изобр
 
 ## Progress
 
+- 2026-09-02 20:37 МСК: `35b6cd9`, инкремент15. Full Mac↔Worker graph использует acknowledged clock bounds, uncertainty для слоёв/ячеек, WAIT и immutable source anchor. Исправлены EOF ordering, repeated clock read и ранний disconnect до OPEN;297 local/173 Worker/12 repeated reconnect PASS. Финальный canary2/32,13 source WAIT+17 sync skips, оба результата после EOF,143.881/155.743ms; real-time FAIL, пороги не ослаблены. Следующий пункт — совместный pre-start clock readiness обеих сторон, затем короткий повтор. Все попытки сохранены;198 artifacts, manifest `7b435cac111b3e0e789aa137ba064651e6d344a5f8fb8fa35b0abbd0e7cb1d74`. Temporary resources удалены, leases released, stock-clock lock снят, четыре сервиса восстановлены,8000/telemetry работают. Этап2 остаётся открытым, stages3–4 не начаты.
+
 - 2026-09-02 19:42 МСК: `d7b8989`, инкремент14. TLS Poll выдаёт pending grant с activation binding без GPU/model authority; four-timestamp mapping с conditional500ppm/50us envelope,16 samples/expiry2s. CPU-only Mac↔Worker:72 probes,60 внутри5ms,8/8 exact echoes, gap2.202s сохраняет PID9/lease1 и22 локальных renewal. Mapping истёк и восстановился; readiness менялась также после warmup.252 local PASS/2 Worker-only skips,65 Worker PASS; Ruff/mypy PASS.81 source hashes сверены; manifest18 artifacts SHA-256 `cd479c577fa3da78b1b01a6a90e03e8c742ecb53561e606d939408120a53a622`. Full-graph mapping/freshness integration остаётся впереди; модели не запускались. Временные ресурсы/секреты удалены, owner released,8000 и телеметрия работают. Старый perception-worker self-restart26→27 зафиксирован, не объявлен исправленным.
 
 - 2026-09-02 15:56 МСК: `70927ea`, full-graph recovery. CPU normal128/gap76 и два повторения full normal128/gap76; 383 tests PASS. Четыре PID сохранены, reset stores 92/16/6/291 →0 и507/42/6/358 →0; stale history/cells отсутствуют. Final normal p95/p99 85.775/89.385ms, gap83.163/89.729ms при SM2610/mem10251. First fresh scene после fault1476.169/3857.475ms, seq30/110,52 intentional skips учтены. Первая GPU попытка прервана до моделей: ошибочный новый idle threshold512MiB был ниже известного baseline534MiB, факт530MiB; сохранена, проверка исправлена, пустой owned lease удалён. 330 artifacts,236 Worker hashes,114 current files verified; `.runtime/perception-stage2-resume-worker-20260902T1545MSK/manifest.json`, SHA-256 `a9449fa4c012e30ce8a06025790c0e110eeebab7c1859fdebf56d5e785961ea3`. Clocks/services восстановлены; legacy restart loops не исправлялись, product/vehicle scope не расширен.
@@ -346,6 +352,6 @@ EoMT — семейство ViT-моделей сегментации изобр
 
 ## Outcomes / retrospective
 
-Этап1 завершён; этап2 продолжается. Полный graph уже проверен между контейнерами одного Worker: normal128/128 exact raw, gap2.2s с теми же четырьмя PID/lease и26 учтёнными пропусками. Инкремент14 отдельно проверил межмашинный control/clock transport без моделей:72 реальных clock samples,60 условно внутри5ms,8/8 exact echoes и сохранение CPU sentinel/lease при gap.252 local и65 Worker checks PASS. Условные границы часов меняются и истекают; подключение continuous WAIT/resync/uncertainty к полному графу — следующий обязательный шаг, не уже пройденная квалификация. Затем полный network/slow-consumer canary и standalone. Native-host GPU inventory и физический радио/live не квалифицированы; batch/registry/UI и этапы3–4 не начаты, actuation=false. Mac8000/telemetry работают,8765/temporary18561 закрыты, Ollama/Frigate exited/restart=no. Временные контейнеры/секреты удалены, owned lease released; прежние service restart loops не скрыты.
+Этап1 завершён; этап2 продолжается. Инкремент15 подключил continuous WAIT/resync/uncertainty к полному Mac↔Worker graph и дал отрицательный canary:2/32 после ожидания допуска и синхронизации, без неучтённых потерь.297 local/173 Worker tests и12 повторных early-reconnect cases PASS; исправлены конкретные ошибки clock assessment, EOF и early disconnect. Нужен двусторонний pre-start readiness barrier, затем повторный короткий full-graph canary и только после него gap/slow-consumer/standalone. Прежние same-Worker128/128 и gap результаты сохраняют свои границы, не заменяют межмашинную квалификацию. Native-host GPU inventory и физический radio/live не квалифицированы; batch/registry/UI и этапы3–4 не начаты, actuation=false. Mac8000/telemetry работают,8765/temporary18561 закрыты, Ollama/Frigate exited/restart=no. Временные контейнеры/секреты удалены, owned leases released, четыре сервиса восстановлены; прежние service defects не объявлены исправленными.
 
 После этапа 4 здесь будут перечислены digest самостоятельного полного образа, измеренные статусы и ошибки по recordings, реально проверенные source/hardware/config комбинации, состояние сохранённого EoMT-варианта и оставшиеся physical-live/quality/vehicle-integration ограничения. Готовый Docker и готовая автономия не отождествляются.

docs/adr/0049-stream-first-perception-profiles.md

@@ -848,3 +848,24 @@ Of72 measured observations,60 meet conditional≤5ms; min/median/max uncertainty
 does not claim full-graph cross-host freshness and does not cherry-pick a passing sample.
 The next full-graph run must consume continuous mapping state and carry uncertainty to
 the receiving freshness decision. [Detailed report](../../experiments/perception/PERCEPTION_STREAM_STAGE2_CONTROL_CLOCK_2026-09-02.md).
+
+## Stage 2 increment 15: acknowledged clocks in the full cross-host graph (2026-09-02)
+
+The Worker validates a one-use echoed Poll challenge before accepting responder-side
+clock evidence and the immutable source anchor. Clock validity is checked continuously,
+including idle periods, input admission and GPU/CPU/publication boundaries. Unknown or
+excess uncertainty enters WAIT without releasing resident models/local ownership.
+Upper age bounds govern source cutoff; scene and cell freshness include uncertainty.
+Each assessment uses one immutable clock snapshot, and EOF acknowledgement precedes
+data-channel retirement. Early transport loss after grant admission but before OPEN
+must also enter WAIT; malformed unbound data cannot stop the owner.
+
+The final full-graph Mac source/receiver canary is **not qualified**:13 source WAIT skips,
+17 synchronization skips,2/32 receipts (frames30–31), both after source-window EOF.
+Latency143.881/155.743ms; no hidden losses, no model reload, sampled VRAM2365MiB,
+not a24GiB limit.297 local and173 Worker tests PASS;
+early disconnect additionally12/12 across six repeats. Source-first clock readiness
+and a fixed2s lead do not establish Worker readiness. Next gate separates two-sided
+clock warmup from source-anchor activation before1x starts; after start the source
+timeline remains immutable through outages. No gate widening, backlog replay,
+standalone or Stage3 promotion. [Evidence and rejected attempts](../../experiments/perception/PERCEPTION_STREAM_STAGE2_CROSSHOST_GRAPH_2026-09-02.md).

experiments/perception/PERCEPTION_STREAM_STAGE2_CROSSHOST_GRAPH_2026-09-02.md

@@ -0,0 +1,163 @@
+# Stage 2 increment 15 — full graph with acknowledged cross-host clocks
+
+2026-09-02, 20:37 MSK. **Implementation/regressions accepted; real-time canary FAIL.**
+Stage 2 remains open. No stage 3 cutover, standalone release or actuation is claimed.
+Code commit: `35b6cd9`.
+
+## Objective and retained architecture
+
+Connect the existing Mac recording source and real result receiver to the single
+Worker006 full graph: DDRNet-39 GOOSE, RF-DETR, online LiDAR geometry/distance,
+temporal/motion, TRAVEL TGS/costmap and advisory policy. The source supplies original
+camera fragments, normalized point increments and poses incrementally at 1x.
+No whole recording upload, predecode, precomputed perception layers or per-recording
+application is introduced. Original source timestamps never change during a run.
+
+The Worker model container has no recording/index/camera/sensor-archive mount. This
+is still a diagnostic dependency image with explicit code/model mounts, **not** the
+portable standalone profile. Mac only reads/releases raw events and validates replies.
+Masks and full scene/grid payloads return over gRPC/TLS through the existing authenticated
+SSH/Tailscale route. This is not the prospective rover radio or an onboard deployment.
+
+## Implemented contracts
+
+- `StreamControl/ReportClock` echoes the last issued Poll challenge once, including the
+  source receipt timestamp. The Worker checks nonce, both clock identities and its own
+  receive/send timestamps; an acknowledgement must arrive within 500ms. The control
+  capability cannot renew a GPU lease, restart models, select epochs or actuate.
+- `ClockReceipt` derives the responder-side offset interval from the original causal
+  exchange, not a reversed/symmetric-delay shortcut. The source anchor is immutable.
+  The explicit conditional envelope remains 500ppm relative drift, 50us per timestamp,
+  2s expiry and uncertainty <=5ms. These assumptions are not hardware clock certification.
+- The Worker checks current clock validity at ingress, GPU/CPU work, publication and
+  while idle. The source checks it during sending and receipt. Unavailable mapping means
+  WAIT, bounded discard, a new controller-owned input epoch and fresh keyframe/sensors.
+  Lease renewal/model ownership remains local and independent of the network.
+- Source cutoff uses the upper mapped age bound. Scene and per-cell expiry include
+  uncertainty; the Mac receiver measures elapsed source-to-consumer time entirely on its
+  own monotonic clock. Worker-local mapped timing is diagnostic, not the acceptance clock.
+- One clock snapshot is used throughout each publication/receipt assessment. A concurrent
+  clock update cannot change the timestamp midway through cell/policy assessment.
+- Source completion is acknowledged before data EOF can retire the control endpoint.
+  Clock warmup precedes initial source-anchor assignment; once assigned, it is never
+  shifted to recover skipped observations.
+- An admitted transport disconnect before the application OPEN now enters WAIT too.
+  Previously the one-use grant could be consumed while the runtime remained active with
+  no input, preventing a replacement epoch. An unbound malformed packet still cannot
+  stop the owner; obsolete peers cannot pause a replacement epoch.
+
+## Method and sequential results
+
+Each model attempt used one profile and the same serialized GPU schedule. Dedicated
+RTX4090, 8 CPU/8GiB limit, pids256, shm256MiB; temporary stock-clock reference
+2610/10251MHz and unchanged 450W power limit. No EoMT, Frigate or Ollama inference.
+Four pre-existing Mission Core services were quiesced and restored per attempt.
+
+All attempts remain separately identified; none is overwritten or omitted:
+
+| Attempt | Outcome | Accepted interpretation |
+| --- | --- | --- |
+| canary32a | Missing candidate manifest before model startup | Packaging failure, no measurement |
+| canary32b | Missing pinned DDRNet runner | Packaging failure, no source processed |
+| canary32c, code-v1 | 30 results, 2 compute discards; source EOF acknowledgement timed out | Full graph/receipt evidence, not accepted real-time |
+| canary32d, code-v2 | 0 results, clean bounded completion | 3 source WAIT skips +29 synchronization skips |
+| canary32e, code-v3 | Source command incorrectly named the index directory, not index.jsonl | Operator-command failure, no source data; explicitly ended via the pilot watchdog handler |
+| canary32f, code-v4 | 2 results, clean bounded completion | Final canary FAIL:13 source WAIT skips +17 synchronization skips |
+
+The launcher now checks the manifest, pinned runner hash and all bind sources before
+quiescing services. Source command generation checks exact input files before activation.
+The pinned runner is unchanged (`b18ad60f277eea69a240a28f290611b94627fb9707faf1bb3e6e22102dad67c1`).
+The rejected early source command was stopped by signalling the confirmed diagnostic
+main PID7 through its existing SIGALRM handler, allowing ordinary cleanup/lease release.
+
+### First full cross-host observation: canary32c
+
+32 source frames ->32 accepted inputs ->30 byte-exact scene receipts +2 explicit
+`input-gap` compute discards (sequences17,27). No reply drops or unaccounted frames.
+The discarded calculations coincided with the old repeated-clock-read assessment path;
+the run did not retain enough fine-grained tracing to prove that as their sole cause.
+All30 received raw model/geometry/temporal/costmap reference comparisons match;28 replies
+arrived before source EOF. Twelve scenes are fully fresh; missing/aged source geometry
+does not become permissive policy.
+
+Measured source-to-consumer-ready p95/p99:208.197/211.873ms, range90.877–211.873ms.
+The125ms latency gate fails even apart from discards/EOF error. Published clock
+uncertainty:3.487–4.543ms. Peak reply440373B; tracked input14640679B, below16MiB.
+These figures belong to the rejected code-v1 attempt, not the final candidate.
+
+### Final candidate: canary32f
+
+Source released all32 frames at1x. Sequences0–12 were omitted while waiting for
+admission. Sequences13–29 were skipped for fresh sensor/keyframe synchronization.
+The full graph resumed at30; frames30 and31 returned, byte/hash validated. Both arrived
+after the bounded source window ended. Therefore neither complete delivery nor the
+result-before-EOF gate passed. This was startup WAIT followed by synchronization,
+**not** two successful initial frames followed by a late outage.
+
+The two source-to-consumer-ready delays were143.881 and155.743ms. Two observations
+do not establish meaningful p95/p99 or sustainable FPS. Both receipts are fresh under
+the explicit250ms freshness envelope; this does not make2/32 availability acceptable.
+Segmentation/detector output matches the reference for both frames. History-dependent
+geometry/motion/costmap differs after the omitted prefix and reset; full-scene reference
+parity is not claimed.
+
+Clock loop:78 observations,28 marked ready; uncertainty min/median/max
+3.915/5.312/30.808ms. Published uncertainty4.517–4.562ms. The gate was never widened.
+One input epoch, one lease generation and four resident children; no model reload
+while awaiting admission. Peak reply403572B, tracked input10479771B, final buffers0.
+Final run sampled VRAM peak2365MiB and cgroup peak3313.4MiB; Mac source RSS59.9MB.
+This is not evidence of a24GiB capacity limit. Sensor archive rows are read incrementally,
+not fully loaded or transferred.
+
+## Validation and retained limitations
+
+- Final local:297PASS,2 intentionally Worker-only skips. Worker:173PASS including
+  the native large-result/slow-reader tests. Early-open cancellation: six additional
+  sequential repeats, two cases each,12/12PASS. No timeout was increased.
+- The previous intermittent reconnect test was reproduced three times. Diagnostics
+  found `opened=false`, drained transport and still-active continuity; the correction
+  covers cancellation both before and after OPEN. An initial overly broad failure
+  handler was rejected by four existing unbound-peer tests and corrected before GPU use.
+- Ruff, format, six-file strict mypy and diff checks pass.128 staged Python files match
+  measured commit35b6cd9, including the pinned runner mapping and final test-only overlays.
+  Subsequent formatting in two files is independently AST-identical; no new model result
+  is attributed to a changed algorithm.
+- `verify.py` independently reconciles source/compute/synchronization/receipt counts,
+  raw scene bytes, mask/layer hashes, Mac receipt ages and clock-bound widening arithmetic.
+  This campaign retained aggregate clock bounds, not every raw five-timestamp exchange;
+  raw clock reconstruction remains a next diagnostic, not a claimed proof here.
+- No full-session, controlled cross-host full-graph outage or full-graph slow-consumer
+  acceptance was performed after the negative canary. The earlier same-Worker results
+  retain their original narrower scope. Product UI/registry was not changed.
+
+## Decision and next Stage 2 gate
+
+The graph works over this real route, but its current start/readiness protocol does
+not provide stable availability. The source can first pass its clock bound before
+the Worker has accumulated equivalent responder-side evidence; its fixed2s lead does
+not prove joint readiness. Missing the initial keyframe/sensor prefix then magnifies
+a short admission delay. This is a protocol/startup finding, not a GPU sizing result.
+
+Next: separate two-sided clock warmup from immutable source-anchor activation. Confirm
+both peers' current readiness before starting the1x source, then keep the source clock
+running unchanged through real faults. Retain raw exchange/ack timing and state-change
+ledgers to distinguish route latency, scheduling and protocol overhead. Repeat one short
+normal canary; only after that pass, run controlled outage/slow-consumer and longer windows.
+Do not weaken the5ms gate, replay backlog or infer an onboard/radio result from SSH.
+Standalone packaging remains later within Stage2; Stage3–4 remain unopened.
+
+## Cleanup and evidence
+
+All temporary containers, lease volumes, host collectors and port18561/tunnel are gone.
+Owner records released; four service container IDs restored, Triton200, Telegraf Running.
+GPU automatic210/405MHz,980MiB,0%,450W at final sample. Existing legacy service defects
+are not declared fixed. Ollama/Frigate remain exited/restart=no. Canonical Mac8000 PID33360
+and identity-matched agent-MQTT telemetry remain available;8765 absent. No physical K1,
+motor/autopilot, recording, Synology or external deployment mutation.
+
+Private TLS keys/bootstrap capabilities were removed on both hosts; public certificates
+and raw non-secret diagnostics retained. Session:
+`.runtime/perception-stage2-crosshost-20260902T1705Z/`,198 artifacts,
+manifest SHA-256 `7b435cac111b3e0e789aa137ba064651e6d344a5f8fb8fa35b0abbd0e7cb1d74`.
+Final code-v4 archive SHA-256:
+`866d709baa785101d10b32b31a17a1b97b15e6de4de932bfd2e79c00825911ab`.