docs(perception): record cross-host clock and control proof

2026-09-02 19:46:27 +03:00
parent d7b8989e43
commit d80df61a7e
3 changed files with 140 additions and 6 deletions
@@ -0,0 +1,109 @@
+# Stage 2 increment 14 — authenticated grant delivery and cross-host clock bounds
+
+Date:2026-09-02. Result: **the application-side control/clock candidate works;
+continuous full-graph cross-host freshness is not yet qualified.** This increment
+contains no model/GPU run. It does not change the four-stage plan or open stage3.
+
+## Objective and architecture boundary
+
+The source must receive a short-lived data-stream grant without gaining authority
+to acquire/renew the Worker GPU lease, restart models, select an epoch or actuate.
+Mac and Worker process monotonic clocks also need an explicit error interval before
+cross-host scene age is accepted. A low RTT or one successful sample is not clock
+synchronization. A temporary network/clock failure means WAIT and fresh input epoch;
+the local lease and resident processes remain owned by the Worker controller.
+
+`StreamControl/Poll` is a small TLS unary method served beside the binary bidirectional
+stream. One per-activation bootstrap capability authenticates polling. It returns the
+controller's already-issued pending data grant and four clock timestamps. Messages are
+8KiB bounded, closed-schema, rate-limited to40/s and limited to one outstanding client
+probe. The control endpoint has no lifecycle/model method. Data admission remains
+single-use and independently authenticated by `Exchange`.
+
+Clock evidence bounds remote-minus-local offset without assuming equal forward and
+return delay. Up to16 recent intervals are intersected, widened by controller-supplied
+relative-rate and timestamp-error envelopes, and expire after2s. Contradictory samples
+quarantine the clock session. All integer nanoseconds remain exact beyond2^53. This is
+a bounded application mapping, not an OS-clock setter or PTP/NTP implementation. The
+four-timestamp basis follows the standard exchange model described by
+[RFC5905](https://www.rfc-editor.org/rfc/rfc5905.html); the candidate deliberately does
+not use the midpoint/equal-path estimate as proof.
+
+During regression, gRPC aio with a native maximum of2 marked the next accepted Poll
+over-limit while Exchange and a completing Poll overlapped. Its implementation checks
+the next request slot before awaiting that request. The combined server therefore has
+three bounded native RPC slots: one active Exchange, one Poll, one accept slot. The
+application still enforces exactly one data owner; this is not GPU parallelism. The
+behavior is covered by the upstream [gRPC aio server limiter source](https://github.com/grpc/grpc/blob/v1.83.0/src/python/grpcio/grpc/_cython/_cygrpc/aio/server.pyx.pxi#L808-L825).
+
+## Real route method
+
+- Existing authenticated SSH/Tailscale route from Mac to Worker006; TLS gRPC tunneled
+  through it. No firewall/listener persisted.
+- Worker isolated `runc` container:1CPU,512MiB, no NVIDIA runtime, zero models,
+  pinned diagnostic image
+  `sha256:664824aa25de1db178f177d67a81b01541a938b479812b9383ddbf03f6b59dbe`.
+- Exactly72 sequential real monotonic probes. Conditional envelope:500ppm relative
+  rate,50us per timestamp,2s maximum age,5ms admission uncertainty.
+- Two data phases, four32KiB synthetic events each. Between them:2.2s disconnect;
+  local lease heartbeat continues. Source clock and resume proof are synthetic and
+  never presented as sensor freshness.
+- Bootstrap ticket/certificate copied once by SSH; no stream grant file copied.
+  Both stream grants arrived via authenticated Poll. Secrets are absent from evidence.
+
+## Results
+
+| Observation | Result |
+| --- | ---: |
+| Clock probes independently verified |72/72 |
+| Conditional uncertainty min / median / max |3.883 /4.469 /27.118ms |
+| Samples inside5ms |60/72 |
+| Readiness state transitions |10 |
+| Payload/reply SHA-256 exact |8/8 |
+| Reply RTT min / median / max |15.796 /22.408 /85.344ms |
+| Controlled network gap |2.202s |
+| Resident CPU sentinel / lease across gap |same PID9 / generation1 |
+| Local lease renewals across gap |136→158 |
+| Peak tracked input / final residual |4,718,905B /0B |
+
+Old clock evidence expired during the2.3s client pause. The new epoch used a new data
+grant; the control bootstrap remained scoped to the same activation and could not renew
+the lease. Mapping recovered after new probes. False→true and true→false clock-ready
+transitions also occurred after warmup, so continuous checking is required. It would be
+invalid to pick one of the60 passing samples and call the whole run qualified.
+
+No full model run followed: the control/clock foundation is proved, but the current
+full-graph adapter does not yet propagate uncertainty into every result freshness
+decision or pause/resynchronize on mapping expiry. Running it now would produce a
+mislabelled cross-host latency claim. The5ms gate was not widened.
+
+## Validation and retained limits
+
+Local expanded regressions:252PASS,2 intentional Worker-only skips. Final Worker:
+65PASS including1MiB result and real slow-reader flow control. Ruff/format, strict mypy
+and diff checks pass. A first expanded local command named two nonexistent test files
+and ran zero tests; that XML is retained and excluded from acceptance. A first Worker
+invocation omitted the Worker-only flag (63PASS/2skips); the explicit final rerun is the
+accepted65PASS evidence.
+
+The Worker recorded one pre-existing perception-service self-restart26→27 while the
+probe ran; same container ID/config, and this harness issued no service-control command.
+It is not hidden and not attributed to the CPU probe. Canonical8000 telemetry remained
+reachable/identity-matched; Frigate/Ollama remain exited/restart=no. Final owned
+containers/listeners/private keys/bootstrap capabilities are zero. Owner record is
+released. No K1, recording, UI, LAB registry, motor/autopilot, Synology or external
+deployment was touched.
+
+Evidence:`.runtime/perception-stage2-control-clock-20260902T1635Z/manifest.json`,18
+artifacts, SHA-256 `cd479c577fa3da78b1b01a6a90e03e8c742ecb53561e606d939408120a53a622`.
+Candidate archive SHA-256
+`980c98fc280f593e470a187870f2b1b08d013386db87c70f688336dc1389d7e3`.
+
+## Next stage-2 gate
+
+Wire refreshed clock bounds into the existing Mac source, Worker controller and result
+receiver. On expiry/excess uncertainty: stop admitting freshness, WAIT, discard backlog,
+obtain a new input epoch and decoder/sensor proof while retaining local models/lease.
+Carry uncertainty into consumer-side scene age. Then run one sequential full-profile
+Mac↔Worker canary plus bounded slow-consumer case. Standalone packaging remains after
+that; product cutover remains stage3.