test(perception): record cross-host gRPC recovery and backpressure proof
This commit is contained in:
DCCONSTRUCTIONS
@@ -1,6 +1,12 @@
|
||||
# Observatory: четыре этапа создания полного real-time Perception-профиля
|
||||
|
||||
Дата: 2026-09-01; обновлено 2026-09-02 17:37 МСК. **Этап 1 закрыт; этап 2 в работе. Инкремент 11: восстановлен существующий System/Worker telemetry plane и добавлен независимый экспорт состояния streaming runtime.** Код `c570f7d`, `2ef8c08`. Исправлены остановка Telegraf при недоступном MQTT, потерянная публикация порта, превышающие лимит контейнера настройки PostgreSQL и блокирующая ingest очистка истории. CPU-only переходы running → waiting → synchronizing → running прошли через реальный агент/Timescale. Это observability, не новый внешний sensor data-plane; следующий пункт остаётся binary stream/auth/recovery между приложением и Worker, затем standalone packaging. Этапы3–4 не начаты.
|
||||
Дата: 2026-09-01; обновлено 2026-09-02 18:11 МСК. **Этап 1 закрыт; этап 2 в работе. Инкремент 12: внешний gRPC/TLS binary candidate, одноразовый доступ и bounded backpressure.** Код `0310592`. Два CPU-only межмашинных повтора Mac↔Worker через существующий SSH-маршрут дали16/16 точных payload/reply; обрыв2.20s сохранил resident CPU sentinel и локальный lease. Итоговые105 focused tests и20 Worker transport tests PASS. Это не full-graph network qualification: реальный `BinaryGraphBridge`, scene/layer codecs, production выдача grant/clock mapping и standalone ещё не подключены к этому endpoint. Этапы3–4 не начаты.
|
||||
|
||||
Финальный same-Mac RTT min/median/max14.360/23.919/120.815ms (не one-way age/FPS), peak mailbox6,750,559B, RSS57,244KiB, после закрытия0bytes. Source clock и resume proof синтетические, моделей0. Default64KiB HTTP/2 window не выдержал250ms для1,048,595B; fixed1MiB прошёл без изменения deadline. Реальный slow-reader выявил блокирование отправки error status; его deadline и deferred quarantine cleanup исправлены и проверены. Evidence16 artifacts: `.runtime/perception-stage2-grpc-20260902T1500Z/manifest.json`, SHA-256 `adf5eaf092feaed6721f66e2adaceded0cdbf55754e1f9953f54623bfb52d331`.
|
||||
|
||||
Следующий проверяемый результат этапа2: подать реальную запись с application-side source через этот endpoint в существующий полный graph, вернуть общие scene/layer outputs с явными размерными/возрастными ограничениями, повторить normal/reconnect/slow-consumer canary последовательно. Транспорт не имеет права вызывать batch materializer или заново запускать модели при сетевой паузе. Текущий outbound cap1MiB требует проверки реальных слоёв/фрагментации; расширять его молча нельзя. После этого — самостоятельный image. Продуктовый LAB path пока не переключён.
|
||||
|
||||
Предыдущий инкремент11 (`c570f7d`, `2ef8c08`): восстановлены System/Worker telemetry plane и независимый runtime exporter. Телеметрия остаётся отдельным наблюдателем, а не выдающим lease/GPU authority контуром.
|
||||
|
||||
104 Python tests, 703 frontend tests, typecheck и production build PASS. Полный frontend проверен последовательно на CPU Worker из-за Mac memory pressure=2; собранный UI установлен на canonical8000. Визуальная приёмка по `mission-core-product-ui` отложена до освобождения памяти, не объявлена пройденной. Расширен существующий Worker, без новых страниц и без product profile/registry cutover. MQTT outage/startup проверен с сохранением PID9268; CPU canary сохранил lease generation1 и освободил его после завершения. Evidence `.runtime/worker-telemetry-20260902T1422Z/manifest.json`, SHA-256 `825b1d8035dbf907b2b731d35ede4c5742b81e653a6f24670f4775981f853492`.
|
||||
|
||||
|
||||
@@ -1,9 +1,9 @@
|
||||
# ADR 0049 — Stream-first full perception profiles
|
||||
|
||||
Date: 2026-09-01; updated 2026-09-02 16:52 MSK. Status: stage-1 engineering
|
||||
baseline complete; stage-2 full binary GPU pilot consumes real scoped host facts.
|
||||
External transport/authentication, native-host GPU inventory, standalone image
|
||||
and product cutover remain open; local proof is not physical-live acceptance.
|
||||
Date: 2026-09-01; updated 2026-09-02 18:11 MSK. Status: stage-1 engineering
|
||||
baseline complete; stage-2 authenticated gRPC candidate has cross-host CPU proof.
|
||||
Full-graph network integration, native-host GPU inventory, standalone image and
|
||||
product cutover remain open; transport proof is not physical-live acceptance.
|
||||
Execution order remains the four stages in
|
||||
[the ExecPlan](../OBSERVATORY_REALTIME_PROFILES_EXECPLAN.md).
|
||||
|
||||
@@ -719,3 +719,61 @@ integration does not constitute the external camera/points binary data plane.
|
||||
See `experiments/perception/PERCEPTION_STREAM_STAGE1_2026-09-01.md` for measured
|
||||
component results, bounded validation, Worker maintenance and remaining gates.
|
||||
Neither this ADR nor passing synthetic contract tests marks the new runtime ready.
|
||||
|
||||
## Stage 2 increment 12: external gRPC/TLS candidate (2026-09-02)
|
||||
|
||||
`streaming_grpc.py` exposes `/missioncore.perception.v1.BinaryStream/Exchange`.
|
||||
It is an optional `perception-stream` dependency, locked grpcio1.83.1; it is not
|
||||
auto-started by the backend or installed into the product LAB path. Both sides
|
||||
require TLS; the client verifies its supplied trust root and server name. The
|
||||
local controller alone issues a256-bit random single-use capability with30s
|
||||
admission expiry, bound to the complete StreamStart and acquisition session.
|
||||
The endpoint retains only its digest. Invalid/duplicate/stale credentials cannot
|
||||
reserve a stream, stop its owner, renew a lease, launch a model or create an epoch.
|
||||
One active stream is admitted; reconnect needs a controller-approved new epoch
|
||||
and, separately, real decoder/causal-sensor resumption proof. Network disconnect
|
||||
is not explicit End or Cancel. Production grant delivery is still an integration
|
||||
gate; the CPU probe transfers ephemeral files only through authenticated SSH.
|
||||
|
||||
The generic gRPC method uses identity byte serialization, not generated protobuf
|
||||
messages. Request messages contain≤64KiB pieces of the unchanged MCI2 wire;
|
||||
StreamingIngress still validates metadata, fragments, hashes and epoch/source
|
||||
binding. No archive, duration, file path or executable command is sent. Response
|
||||
MCR3 is only an envelope over existing domain bytes: magic4, uint64 sequence8,
|
||||
binding SHA25632, payload SHA25632, then1..1MiB payload. It does not invent a new
|
||||
scene ontology or claim that full-graph scene/layer serialization is connected.
|
||||
The actual profile's larger outputs require reviewed fragmentation, not a silent
|
||||
cap increase. Result age validation and the full result receipt ledger remain
|
||||
with the graph/application adapter, not the transport envelope.
|
||||
|
||||
There are two pending latest replies and one writer/reader per direction; no
|
||||
background source queue/retry or event-loop callback backlog. Python bridge
|
||||
storage is reserved inside the existing16MiB mailbox budget. Fixed1MiB HTTP/2
|
||||
lookahead, disabled BDP growth/retries, bounded metadata/messages/IO are explicit;
|
||||
native gRPC/TLS/socket memory also needs measured RSS, not a claimed byte-perfect
|
||||
mapping into mailbox accounting. Public/multi-client DoS hardening is not proved
|
||||
by this private Worker experiment. A250ms event deadline covers the complete
|
||||
source event. Even final error-status sending is deadline-bound; otherwise an
|
||||
unread response can delay cleanup. Late trusted callbacks are quarantined, with
|
||||
no replacement grant or early byte release until they return. Pending reply
|
||||
drops are counted; they are not equivalent to end-to-end delivery receipts.
|
||||
|
||||
Evidence: initial default64KiB lookahead timed out on1,048,595B. Fixed1MiB passed
|
||||
two16/16 cross-host samples without changing250ms. Final min/median/max RTT
|
||||
14.360/23.919/120.815ms on the Mac includes SSH/echo overhead, not clock-subtracted
|
||||
one-way age.2.202s wait retained the same CPU sentinel PID40 and lease; renewals
|
||||
32→54. Source clock and resume evidence are explicitly synthetic, model count0.
|
||||
Peak mailbox6,750,559B; Linux RSS57,244KiB.20 Worker transport tests include1MiB
|
||||
output, a real slow reader and late-callback cleanup;105 focused Mac tests pass
|
||||
with the two larger probes intentionally skipped. Final cleanup leaves0bytes,
|
||||
no temporary listener/container/tunnel/private key/grant. No GPU setters or
|
||||
durable services changed. Frigate/Ollama remain disabled, Telegraf is Running.
|
||||
Manifest SHA256 `adf5eaf092feaed6721f66e2adaceded0cdbf55754e1f9953f54623bfb52d331`
|
||||
at `.runtime/perception-stage2-grpc-20260902T1500Z/manifest.json` (16 artifacts).
|
||||
|
||||
Design references: [gRPC flow control](https://grpc.io/docs/guides/flow-control/)
|
||||
warns that a completed write need not mean network delivery;
|
||||
[authentication](https://grpc.io/docs/guides/auth/) documents TLS credentials;
|
||||
[channel options](https://github.com/grpc/grpc/blob/master/include/grpc/impl/channel_arg_names.h)
|
||||
documents the fixed lookahead window. Source and actual behavior were checked;
|
||||
these references alone are not a performance qualification.
|
||||
|
||||
Reference in New Issue
Block a user