diff --git a/docs/OBSERVATORY_REALTIME_PROFILES_EXECPLAN.md b/docs/OBSERVATORY_REALTIME_PROFILES_EXECPLAN.md index 603c9e8..9a9c057 100644 --- a/docs/OBSERVATORY_REALTIME_PROFILES_EXECPLAN.md +++ b/docs/OBSERVATORY_REALTIME_PROFILES_EXECPLAN.md @@ -1,12 +1,16 @@ # Observatory: четыре этапа создания полного real-time Perception-профиля -Дата: 2026-09-01; обновлено 2026-09-02 19:00 МСК. **Этап 1 закрыт; этап 2 в работе. Инкремент 13: полный graph с реальным сетевым входом и выходом.** CPU source/receiver вынесен в отдельный контейнер без GPU; модельный контейнер не получает запись через mounts. Через gRPC/TLS возвращаются настоящая маска512×512, detections/distance/motion и grid/costmap/policy. Обычный1× прогон:128/128 exact raw к reference, p95/p99 до готового consumer-view106.150/115.349ms, max126.910ms (один кадр выше125ms). 76/128 полных свежих scene;52 исходные sensor gaps не скрыты. +Дата: 2026-09-01; обновлено 2026-09-02 19:42 МСК. **Этап 1 закрыт; этап 2 в работе. Инкремент 14: межмашинная выдача grant и измеренные границы часов.** Новый TLS control endpoint доставляет уже выданный Worker-контроллером одноразовый data grant, но не умеет получать/продлевать GPU lease, выбирать epoch, перезапускать модели или actuate. Четыре timestamp дают диапазон remote-minus-local без предположения симметрии сети; до16 наблюдений, expiry2s, нарушение envelope карантинит clock session. + +CPU-only Mac↔Worker006 proof:72/72 timestamp-наблюдения независимо пересчитаны,60/72 соответствуют условному uncertainty≤5ms; min/median/max3.883/4.469/27.118ms. Восемь synthetic32KiB events exact8/8. Обрыв2.202s сохранил PID/lease1, старое clock evidence истекло, новый epoch/grant восстановили вход. Это доказывает механизм WAIT/recovery, но не full-graph cross-host freshness: readiness переключалась и после warmup. Поэтому5ms не ослаблен, а модельный прогон с неподключённым uncertainty gate не запускался. [Отчёт](../experiments/perception/PERCEPTION_STREAM_STAGE2_CONTROL_CLOCK_2026-09-02.md); evidence manifest SHA-256 `cd479c577fa3da78b1b01a6a90e03e8c742ecb53561e606d939408120a53a622`. + +Предыдущий инкремент13 — полный graph с реальным сетевым входом и выходом внутри одного Worker. CPU source/receiver вынесен в отдельный контейнер без GPU; модельный контейнер не получает запись через mounts. Через gRPC/TLS возвращаются настоящая маска512×512, detections/distance/motion и grid/costmap/policy. Обычный1× прогон:128/128 exact raw к reference, p95/p99 до готового consumer-view106.150/115.349ms, max126.910ms (один кадр выше125ms).76/128 полных свежих scene;52 исходные sensor gaps не скрыты. Отдельный обрыв2.2s:102 результатов +26 учтённых пропусков24–49, resume50 с теми же четырьмя PID и lease generation1; одна очистка temporal history, без backlog. p95/p99 принятых результатов107.098/110.817ms; отдельный gap доступности2.725s. Начальный лишний reconnect из-за раннего init исправлен без увеличения idle-timeout и проверен повтором.146 focused tests PASS,2 больших Worker-only теста пропущены локально. Максимальный ответ460248B остаётся внутри1MiB; peak tracked13,217,791B, после закрытия0. Граница доказательства: два контейнера одного Worker с проверенным общим Linux monotonic clock, временные штатные частоты2610/10251MHz; **не Mac↔Worker full-graph, не radio/live и не standalone**. История зависимых слоёв после gap закономерно отличается от uninterrupted reference. Все230 опубликованных scene/масок проверены на стороне приёмника и повторно сверены. [Подробный отчёт](../experiments/perception/PERCEPTION_STREAM_STAGE2_NETWORK_2026-09-02.md). Evidence236 artifacts: `.runtime/perception-stage2-network-graph-20260902T1541Z/manifest.json`, SHA-256 `d1bc7e26850a2d9050ee1d8f8a5ece2e6b8c53e2e7dba381cbd4445d68a5ec17`. -Следующий проверяемый результат этапа2: application-side выдача grant + измеренное сопоставление часов, полный graph Mac↔Worker и slow-consumer canary, затем самостоятельный image без developer/model mounts. Начальный retry/выдача grant текущего bounded pilot не являются production recovery service. Продуктовый LAB path пока не переключён; этапы3–4 не начаты. Worker services и auto clocks восстановлены, Ollama/Frigate остаются exited/restart=no, временные контейнеры/network/lease удалены. Mac8000 и Worker telemetry работают. +Следующий проверяемый результат этапа2: подключить continuous clock-readiness/uncertainty к полному graph Mac↔Worker, выполнить его и slow-consumer canary, затем самостоятельный image без developer/model mounts. Начальный retry текущих bounded pilots не является production recovery service. Продуктовый LAB path пока не переключён; этапы3–4 не начаты. CPU-проба не меняла GPU/clocks/services; Ollama/Frigate остаются exited/restart=no, временные контейнер/listener/key/bootstrap удалены. Mac8000 и Worker telemetry работают. Предыдущий инкремент12 (`0310592`, `1916122`): CPU-only gRPC/TLS proof Mac↔Worker,16/16 payload/reply; синтетический source clock и CPU sentinel, моделей0. Same-Mac RTT14.360/23.919/120.815ms min/median/max — не one-way age/FPS.105 local и20 Worker tests PASS, проверены bounded slow-reader/quarantine. Evidence `.runtime/perception-stage2-grpc-20260902T1500Z/manifest.json`, SHA-256 `adf5eaf092feaed6721f66e2adaceded0cdbf55754e1f9953f54623bfb52d331`. @@ -237,6 +241,8 @@ EoMT — семейство ViT-моделей сегментации изобр ## Progress +- 2026-09-02 19:42 МСК: `d7b8989`, инкремент14. TLS Poll выдаёт pending grant с activation binding без GPU/model authority; four-timestamp mapping с conditional500ppm/50us envelope,16 samples/expiry2s. CPU-only Mac↔Worker:72 probes,60 внутри5ms,8/8 exact echoes, gap2.202s сохраняет PID9/lease1 и22 локальных renewal. Mapping истёк и восстановился; readiness менялась также после warmup.252 local PASS/2 Worker-only skips,65 Worker PASS; Ruff/mypy PASS.81 source hashes сверены; manifest18 artifacts SHA-256 `cd479c577fa3da78b1b01a6a90e03e8c742ecb53561e606d939408120a53a622`. Full-graph mapping/freshness integration остаётся впереди; модели не запускались. Временные ресурсы/секреты удалены, owner released,8000 и телеметрия работают. Старый perception-worker self-restart26→27 зафиксирован, не объявлен исправленным. + - 2026-09-02 15:56 МСК: `70927ea`, full-graph recovery. CPU normal128/gap76 и два повторения full normal128/gap76; 383 tests PASS. Четыре PID сохранены, reset stores 92/16/6/291 →0 и507/42/6/358 →0; stale history/cells отсутствуют. Final normal p95/p99 85.775/89.385ms, gap83.163/89.729ms при SM2610/mem10251. First fresh scene после fault1476.169/3857.475ms, seq30/110,52 intentional skips учтены. Первая GPU попытка прервана до моделей: ошибочный новый idle threshold512MiB был ниже известного baseline534MiB, факт530MiB; сохранена, проверка исправлена, пустой owned lease удалён. 330 artifacts,236 Worker hashes,114 current files verified; `.runtime/perception-stage2-resume-worker-20260902T1545MSK/manifest.json`, SHA-256 `a9449fa4c012e30ce8a06025790c0e110eeebab7c1859fdebf56d5e785961ea3`. Clocks/services восстановлены; legacy restart loops не исправлялись, product/vehicle scope не расширен. - 2026-09-02 15:30 МСК: `c31c46c`, resumable lifecycle/input epoch, recoverable readiness, decoder/sensor reset. 375 focused tests PASS, Ruff/format 14 files и mypy 7 modules PASS. Worker CPU-only три input epochs, EOF-паузы 150 ms / 2.2 s, 3/44 synthetic source ticks во время пауз, 54 local renewals; PID decoder/sentinel неизменны, BGR exact 3/3. Peak input 5,275,461 bytes, final input/children=0, released lease; temporary container удалён. Один реальный H.264 fixture + synthetic timestamps/sensors/temporal reset, НЕ полный graph/GPU recovery. 73 code hashes сверены, 85 evidence artifacts; `.runtime/perception-stage2-continuity-worker-20260902T1518MSK/manifest.json`, SHA-256 `fbd327d129c88caa13b7183b1ae2c4767e877a6f5472c0f430875cfd7401ed46`. Сервисы/clocks не менялись; прежние legacy restart loops не исправлялись. Старый terminal-readiness эксперимент ниже сохранён как история; новый TARGET по решению владельца — wait/resync для временной недоступности. - 2026-09-02 14:52 МСК: `92625bf`, readiness/lifecycle increment. 349 focused tests PASS; Worker Linux CPU-only 4/4 scenarios, stop 10.676–63.994 ms, 0 residual bytes, terminal fixture lease generation 4 released. Expired inventory остановил runtime через 1043.960 ms после последнего refresh, несмотря на 13 heartbeat renewals; late refresh/result rejected. Allowed overload сохранил violation после восстановления clocks. 70 measured code files verified, 86 evidence artifacts; `.runtime/perception-stage2-readiness-worker-20260902T1450MSK/manifest.json`, SHA-256 `4588995d763fe82d896b42689e981f1d00feedaf9d6aab29f4579f771d03da3a`. GPU facts synthetic, no model timing or collector qualification. Service IDs/StartedAt/restart counts неизменны до/после; у legacy agents уже были restart counts 21/2/2 — состояние running не объявляется устойчивой готовностью. Clock setters, service mutations, network listeners, registry/UI отсутствуют. @@ -340,6 +346,6 @@ EoMT — семейство ViT-моделей сегментации изобр ## Outcomes / retrospective -Этап1 завершён; этап2 продолжается. Помимо scoped host control, полный graph теперь связан gRPC/TLS с отдельным CPU source/receiver: normal128/128 exact raw, gap2.2s → WAIT/resync с теми же четырьмя PID/lease и26 учтёнными пропусками.146 focused tests текущего инкремента PASS; ранее404 checks относились к host-control инкременту. Consumer-ready p95/p99 normal106.150/115.349ms; это same-kernel container-network при stock fixed clocks, не Mac↔Worker. Следом межмашинные clock/grant integration и slow consumer, затем standalone packaging. Native-host GPU inventory не квалифицирован; Docker scope не выдаётся за полный host audit. Batch/registry/UI и этапы3–4 не начаты, actuation=false. Mac8000=200,8765 закрыт; auto clocks и четыре Worker services восстановлены, Ollama/Frigate exited/restart=no, временных collectors/container/network/lease/key/grant нет. +Этап1 завершён; этап2 продолжается. Полный graph уже проверен между контейнерами одного Worker: normal128/128 exact raw, gap2.2s с теми же четырьмя PID/lease и26 учтёнными пропусками. Инкремент14 отдельно проверил межмашинный control/clock transport без моделей:72 реальных clock samples,60 условно внутри5ms,8/8 exact echoes и сохранение CPU sentinel/lease при gap.252 local и65 Worker checks PASS. Условные границы часов меняются и истекают; подключение continuous WAIT/resync/uncertainty к полному графу — следующий обязательный шаг, не уже пройденная квалификация. Затем полный network/slow-consumer canary и standalone. Native-host GPU inventory и физический радио/live не квалифицированы; batch/registry/UI и этапы3–4 не начаты, actuation=false. Mac8000/telemetry работают,8765/temporary18561 закрыты, Ollama/Frigate exited/restart=no. Временные контейнеры/секреты удалены, owned lease released; прежние service restart loops не скрыты. После этапа 4 здесь будут перечислены digest самостоятельного полного образа, измеренные статусы и ошибки по recordings, реально проверенные source/hardware/config комбинации, состояние сохранённого EoMT-варианта и оставшиеся physical-live/quality/vehicle-integration ограничения. Готовый Docker и готовая автономия не отождествляются. diff --git a/docs/adr/0049-stream-first-perception-profiles.md b/docs/adr/0049-stream-first-perception-profiles.md index b4e3dd3..87dfce8 100644 --- a/docs/adr/0049-stream-first-perception-profiles.md +++ b/docs/adr/0049-stream-first-perception-profiles.md @@ -1,8 +1,10 @@ # ADR 0049 — Stream-first full perception profiles -Date: 2026-09-01; updated 2026-09-02 19:00 MSK. Status: stage-1 engineering -baseline complete; stage-2 full graph has authenticated container-network proof. -Cross-host full-graph clock/grant integration, native-host GPU inventory, +Date: 2026-09-01; updated 2026-09-02 19:42 MSK. Status: stage-1 engineering +baseline complete; stage-2 has full graph container-network proof plus authenticated +cross-host grant delivery and bounded clock intervals. Continuous clock readiness is +not yet connected to full-graph freshness. Cross-host full-graph qualification, +native-host GPU inventory, standalone image and product cutover remain open. This is not physical-live acceptance. Execution order remains the four stages in [the ExecPlan](../OBSERVATORY_REALTIME_PROFILES_EXECPLAN.md). @@ -829,3 +831,20 @@ See [the detailed report](../../experiments/perception/PERCEPTION_STREAM_STAGE2_ Four prior services and automatic clocks restored; transient containers, networks, lease volumes, keys and grants removed. Ollama/Frigate stay disabled. Mac8000 and Worker telemetry remain available; no8765 or external deployment. + +## Stage 2 increment 14: cross-host clock/control foundation (2026-09-02) + +Application-side TLS Poll may deliver only a pending data grant already issued by the +trusted Worker controller. It cannot acquire/renew the local lease, reset/start models, +choose an epoch or cancel. Data admission remains single-use and independently scoped. +Four timestamps bound remote-minus-local offset without equal-path assumptions; recent +intervals are intersected, aged under an explicit rate/error envelope and expire. +Expired/excess/contradictory evidence means unavailable/WAIT, not a wider hidden budget. + +CPU-only Mac↔Worker proof delivers both grants and8/8 exact echoes. A2.202s gap retains +the CPU sentinel and lease1, expires the old mapping and resumes with a new input epoch. +Of72 measured observations,60 meet conditional≤5ms; min/median/max uncertainty are +3.883/4.469/27.118ms and readiness changes after warmup. Consequently this increment +does not claim full-graph cross-host freshness and does not cherry-pick a passing sample. +The next full-graph run must consume continuous mapping state and carry uncertainty to +the receiving freshness decision. [Detailed report](../../experiments/perception/PERCEPTION_STREAM_STAGE2_CONTROL_CLOCK_2026-09-02.md). diff --git a/experiments/perception/PERCEPTION_STREAM_STAGE2_CONTROL_CLOCK_2026-09-02.md b/experiments/perception/PERCEPTION_STREAM_STAGE2_CONTROL_CLOCK_2026-09-02.md new file mode 100644 index 0000000..8564272 --- /dev/null +++ b/experiments/perception/PERCEPTION_STREAM_STAGE2_CONTROL_CLOCK_2026-09-02.md @@ -0,0 +1,109 @@ +# Stage 2 increment 14 — authenticated grant delivery and cross-host clock bounds + +Date:2026-09-02. Result: **the application-side control/clock candidate works; +continuous full-graph cross-host freshness is not yet qualified.** This increment +contains no model/GPU run. It does not change the four-stage plan or open stage3. + +## Objective and architecture boundary + +The source must receive a short-lived data-stream grant without gaining authority +to acquire/renew the Worker GPU lease, restart models, select an epoch or actuate. +Mac and Worker process monotonic clocks also need an explicit error interval before +cross-host scene age is accepted. A low RTT or one successful sample is not clock +synchronization. A temporary network/clock failure means WAIT and fresh input epoch; +the local lease and resident processes remain owned by the Worker controller. + +`StreamControl/Poll` is a small TLS unary method served beside the binary bidirectional +stream. One per-activation bootstrap capability authenticates polling. It returns the +controller's already-issued pending data grant and four clock timestamps. Messages are +8KiB bounded, closed-schema, rate-limited to40/s and limited to one outstanding client +probe. The control endpoint has no lifecycle/model method. Data admission remains +single-use and independently authenticated by `Exchange`. + +Clock evidence bounds remote-minus-local offset without assuming equal forward and +return delay. Up to16 recent intervals are intersected, widened by controller-supplied +relative-rate and timestamp-error envelopes, and expire after2s. Contradictory samples +quarantine the clock session. All integer nanoseconds remain exact beyond2^53. This is +a bounded application mapping, not an OS-clock setter or PTP/NTP implementation. The +four-timestamp basis follows the standard exchange model described by +[RFC5905](https://www.rfc-editor.org/rfc/rfc5905.html); the candidate deliberately does +not use the midpoint/equal-path estimate as proof. + +During regression, gRPC aio with a native maximum of2 marked the next accepted Poll +over-limit while Exchange and a completing Poll overlapped. Its implementation checks +the next request slot before awaiting that request. The combined server therefore has +three bounded native RPC slots: one active Exchange, one Poll, one accept slot. The +application still enforces exactly one data owner; this is not GPU parallelism. The +behavior is covered by the upstream [gRPC aio server limiter source](https://github.com/grpc/grpc/blob/v1.83.0/src/python/grpcio/grpc/_cython/_cygrpc/aio/server.pyx.pxi#L808-L825). + +## Real route method + +- Existing authenticated SSH/Tailscale route from Mac to Worker006; TLS gRPC tunneled + through it. No firewall/listener persisted. +- Worker isolated `runc` container:1CPU,512MiB, no NVIDIA runtime, zero models, + pinned diagnostic image + `sha256:664824aa25de1db178f177d67a81b01541a938b479812b9383ddbf03f6b59dbe`. +- Exactly72 sequential real monotonic probes. Conditional envelope:500ppm relative + rate,50us per timestamp,2s maximum age,5ms admission uncertainty. +- Two data phases, four32KiB synthetic events each. Between them:2.2s disconnect; + local lease heartbeat continues. Source clock and resume proof are synthetic and + never presented as sensor freshness. +- Bootstrap ticket/certificate copied once by SSH; no stream grant file copied. + Both stream grants arrived via authenticated Poll. Secrets are absent from evidence. + +## Results + +| Observation | Result | +| --- | ---: | +| Clock probes independently verified |72/72 | +| Conditional uncertainty min / median / max |3.883 /4.469 /27.118ms | +| Samples inside5ms |60/72 | +| Readiness state transitions |10 | +| Payload/reply SHA-256 exact |8/8 | +| Reply RTT min / median / max |15.796 /22.408 /85.344ms | +| Controlled network gap |2.202s | +| Resident CPU sentinel / lease across gap |same PID9 / generation1 | +| Local lease renewals across gap |136→158 | +| Peak tracked input / final residual |4,718,905B /0B | + +Old clock evidence expired during the2.3s client pause. The new epoch used a new data +grant; the control bootstrap remained scoped to the same activation and could not renew +the lease. Mapping recovered after new probes. False→true and true→false clock-ready +transitions also occurred after warmup, so continuous checking is required. It would be +invalid to pick one of the60 passing samples and call the whole run qualified. + +No full model run followed: the control/clock foundation is proved, but the current +full-graph adapter does not yet propagate uncertainty into every result freshness +decision or pause/resynchronize on mapping expiry. Running it now would produce a +mislabelled cross-host latency claim. The5ms gate was not widened. + +## Validation and retained limits + +Local expanded regressions:252PASS,2 intentional Worker-only skips. Final Worker: +65PASS including1MiB result and real slow-reader flow control. Ruff/format, strict mypy +and diff checks pass. A first expanded local command named two nonexistent test files +and ran zero tests; that XML is retained and excluded from acceptance. A first Worker +invocation omitted the Worker-only flag (63PASS/2skips); the explicit final rerun is the +accepted65PASS evidence. + +The Worker recorded one pre-existing perception-service self-restart26→27 while the +probe ran; same container ID/config, and this harness issued no service-control command. +It is not hidden and not attributed to the CPU probe. Canonical8000 telemetry remained +reachable/identity-matched; Frigate/Ollama remain exited/restart=no. Final owned +containers/listeners/private keys/bootstrap capabilities are zero. Owner record is +released. No K1, recording, UI, LAB registry, motor/autopilot, Synology or external +deployment was touched. + +Evidence:`.runtime/perception-stage2-control-clock-20260902T1635Z/manifest.json`,18 +artifacts, SHA-256 `cd479c577fa3da78b1b01a6a90e03e8c742ecb53561e606d939408120a53a622`. +Candidate archive SHA-256 +`980c98fc280f593e470a187870f2b1b08d013386db87c70f688336dc1389d7e3`. + +## Next stage-2 gate + +Wire refreshed clock bounds into the existing Mac source, Worker controller and result +receiver. On expiry/excess uncertainty: stop admitting freshness, WAIT, discard backlog, +obtain a new input epoch and decoder/sensor proof while retaining local models/lease. +Carry uncertainty into consumer-side scene age. Then run one sequential full-profile +Mac↔Worker canary plus bounded slow-consumer case. Standalone packaging remains after +that; product cutover remains stage3.