docs(perception): record joint startup proof and remaining clock wait

experiments/perception/PERCEPTION_STREAM_STAGE2_JOINT_START_2026-09-02.md

@@ -0,0 +1,123 @@
+# Stage 2 increment 16 — joint source activation and measured clock WAIT
+
+2026-09-02, 21:12 MSK. Code `48835a0`. **Joint startup PASS; full real-time canary FAIL.**
+Stage 2 remains open; no Stage 3/4 cutover, standalone release or actuation.
+
+## Change and invariants
+
+Clock warmup no longer starts the recording. The Mac acknowledges every issued probe,
+including pre-start probes with a null source anchor. Worker accumulates responder
+evidence and returns typed bounds plus its accepted anchor. Both sides must satisfy
+the current clock gate before proposing a start; Worker must explicitly accept it.
+Mac additionally waits for the pending data grant before returning from startup.
+
+ReportClock uses closed `missioncore.stream-clock-report/v2` and
+`missioncore.stream-clock-receipt/v2` documents; legacy/malformed responses fail closed.
+Nanoseconds, including signed clock offsets, remain exact canonical decimal strings.
+Client checks activation, nonce, clock identities, envelope and accepted anchor.
+Peer bounds are aged to the latest possible Worker time at source use, including ACK
+transit; a cached readiness boolean cannot authorize a later observation.
+
+The one-second lead is only for bounded grant delivery/setup AFTER joint readiness.
+Lost ACK retains the exact same proposal because Worker may already have accepted it.
+Explicit nonacceptance permits another pre-start proposal; an accepted anchor never
+changes. Missing the accepted start fails startup rather than retiming the recording.
+Source input paths are checked before clock activation. Receipt diagnostics retain
+bounded raw t1–t6, with one evaluation timestamp per reported readiness snapshot.
+
+Unchanged: 1x original source clock, every-frame candidate, <=16MiB tracked inputs,
+two pending cameras, serialized GPU, 125ms p95/p99 whole-path gate, <=5ms conditional
+clock uncertainty, 500ppm relative-rate budget, 50us timestamp-error budget, 2s expiry
+and 16 recent samples. These are conditional assumptions, not a measurement that the
+physical clocks actually drift by 500ppm. WAIT preserves models/local ownership;
+resync discards backlog and requires a new epoch/keyframe/current sensors.
+
+## One full-graph canary
+
+Session `.runtime/perception-stage2-joint-start-20260902T1805Z`, run `joint32-worker`.
+Actual source start 18:04:32.831227Z; session ID is an identifier, not the run timestamp.
+Mac incrementally reads the existing raw recording and receives real scene/mask payloads;
+Worker006 runs DDRNet-39 GOOSE, RF-DETR, LiDAR/distance, motion, TRAVEL TGS/costmap/policy.
+Model container has no recording mount. One GPU profile, 8 CPUs/8GiB, temporary
+2610/10251MHz stock-clock reference, unchanged 450W. No alternate model run or retry.
+Pinned dependency image plus developer/model mounts remains **not standalone**.
+Transport is gRPC/TLS through SSH/Tailscale, not a measured rover radio/onboard link.
+
+| Original camera sequences | Observed outcome |
+| --- | --- |
+| 0–5 | Six results; initial camera/keyframe and sensor prefix admitted |
+| 6 | Accepted, then explicitly discarded as `input-gap` |
+| 7–15 | Nine source WAIT skips |
+| 16–29 | Fourteen synchronization skips while waiting for a fresh keyframe/sensor pair |
+| 30–31 | Two results after recovery; both arrive after the short source window ends |
+
+Ledger: 32 released =8 results +1 compute discard +9 WAIT skips +14 sync skips.
+No unaccounted frame, reply drop, source error or control/data RPC error. Source remains
+1x; release-lag p95/p99 3.062/17.286ms, max36.725ms. Joint startup admitted frame0,
+unlike increment15's final attempt; this does not establish continuous availability.
+
+Eight Worker/Mac scene payloads are byte-exact; all masks, detections, material,
+lineage and sensor bindings match the uninterrupted reference. Geometry/tracks/threats
+and raw TGS state match on the six pre-gap results, not on the two post-resync results:
+their temporal history was intentionally reset. Do not claim eight uninterrupted
+full-graph reference matches. Five received scenes are fully fresh; six arrive before EOF.
+
+Same-Mac source-due → consumer-ready: min101.345ms, median144.951ms,
+p95/p99/max215.767ms, n=8. This tiny failing sample is not a stable throughput estimate.
+DDRNet GPU-model mean12.895ms, detector mean12.862ms; these are component intervals,
+not end-to-end latency or evidence of spare realtime capacity. Peak sampled VRAM2363MiB,
+cgroup3341.08MiB, Mac source RSS61,767,680B, tracked inputs12,055,035B, reply419,194B.
+The result does not indicate exhaustion of24GiB GPU memory.
+
+## Independently reconstructed clock evidence
+
+All55 exchanges retained t1(source send), t2/t3(Worker receive/send), t4(source receipt),
+t5(Worker receipt of report), t6(source ACK return). Independent integer calculations
+reproduce BOTH published interval windows and all55 readiness decisions;41 are ready.
+The sole accepted source anchor and a ready observation precede initial data connect.
+
+After startup, source samples27–35 are unavailable: uncertainty5.031–5.667ms at the
+unchanged5ms limit. Sample36 returns to4.867ms. Worker continuous WAIT lasts864.139ms;
+keyframe recovery magnifies the resulting output gap. Source uncertainty min/median/max
+4.217/4.620/32.053ms includes initial warmup; the maximum is not running-source drift.
+Probe-cycle min/median/max8.420/13.991/101.253ms; report/ACK-cycle7.392/13.269/104.500ms.
+These intervals include application scheduling and cannot alone attribute delay to
+radio, TCP, SSH, gRPC or event-loop contention.
+
+An offline diagnostic retaining EVERY sample within the same2s horizon also exceeds
+5ms throughout these nine unavailable samples. Increasing the16-sample count alone
+would not remove this WAIT in the trace. No runtime window/threshold was changed.
+Worker host-control observations33/33 accepted, no expiry or envelope violations;
+the observed WAIT was clock admission, not missing GPU telemetry or owner loss.
+Two input epochs retain lease generation1 and resident PIDs11/40/48/49. Recovery clears
+temporal stores with all children alive; final orderly completion releases all resources.
+
+## Validation, evidence and next gate
+
+241 focused local tests PASS,2 Worker-only cases skipped locally;198 Worker CPU tests
+PASS, including real TLS joint startup, malformed/legacy receipts, lost accepted-anchor
+ACK, one-sided/stale bounds, missing grant, unchanged anchor after recovery and existing
+ingress/lifecycle/backpressure cases. Ruff/check-format and typed clock modules mypy PASS.
+129 measured Python source files match the local committed code and code-v2 archive.
+Code-v1 CPU evidence remains retained; v2 only makes diagnostic readiness use its
+recorded timestamp. Only v2 ran the GPU canary.
+
+Archive SHA256 `52e9778cdd54aab33b9e347081ae67ebcc53968773717b9c708174381f2d6eed`.
+53 retained artifacts; manifest SHA256
+`9cb369fe7ae185d1a92f6050bc1018a878483f5eec169267f100843245428691`.
+`verify.py`, `verify_clocks.py`, `clock-reconstruction.json`, `review.json` and
+`acceptance.json` reproduce receipt accounting, freshness, timing and closed resources.
+
+Next: CPU-only attribution/A/B of clock/control scheduling versus concurrent scene
+transfer/receipt work, preserving the existing route and admission budgets. Determine
+what is application scheduling and what is transport; then optimize the proven cause.
+Do not add artificial outages or start a long GPU qualification on this failing baseline.
+After a stable short full-graph canary: controlled gap/slow-consumer, longer runs and
+standalone packaging within Stage2. Product integration stays in later stages.
+
+Temporary model/test containers, collector, lease volume, tunnel18561, key/bootstrap
+are gone; owner released, four exact prior service IDs restored, Triton200, Telegraf
+Running. Clock-lock reset commands succeeded; subsequent automatic210/405MHz observed,
+450W unchanged. Ollama/Frigate remain exited/restart=no. Canonical Mac8000 PID33360 and
+identity-matched Worker telemetry work;8765 absent. Existing legacy-service defects
+are not declared fixed. No K1, motor, autonomous-driving or external/Synology deployment.