docs(perception): record clock route A/B/A and continuous LAN canary

experiments/perception/PERCEPTION_STREAM_STAGE2_LAN_ROUTE_2026-09-02.md

@@ -0,0 +1,126 @@
+# Stage 2 increment 17 — clock route A/B/A and continuous full LAN canary
+
+2026-09-02,22:06 MSK. Code `59a74a3`. **32/32 continuity/reference PASS; latency FAIL.**
+Stage2 remains open. No standalone release, Stage3/4 cutover or physical actuation.
+
+## Diagnostic and architectural scope
+
+Added `pilot_clock_route_probe.py`: bounded clock-only TLS endpoint/client and10ms
+event-loop scheduling witness. It cannot issue a data grant, start a recording, own a
+GPU lease or run models. Both raw monotonic interval windows are returned, and t1–t6
+plus Worker handler entry/finish times are retained. Scheduling observations are not
+subtracted from timestamps or used to manufacture clock readiness.
+
+Three sequential96-exchange CPU probes used the existing pinned image in runc mode,
+1CPU/512MiB/noGPU. The Mac ran sparse control requests only, not a scene/load generator.
+Both model/data processing and recorded-source traffic were absent from this A/B/A.
+No durable service, host clock, SSH config, firewall or router setting changed.
+
+A and A2 used the existing SSH/Tailscale route. B changed only the SSH HostName to the
+confirmed Worker LAN address. The same `mission-gpu` host-key alias, pinned known-hosts
+file and strict checking authenticated the host; returned hostname matched Worker006.
+Tailscale preflight itself reported a direct LAN endpoint, not a proved DERP relay.
+Do not describe this result as removal of an Internet relay.
+
+| Sequential case | Joint clock-ready | First ready index | Ready continuously afterward | Median uncertainty |
+| --- | --- | --- | --- | --- |
+| A: SSH/Tailscale |48/96|3|No|5.001ms|
+| B: SSH/confirmed LAN |94/96|2|Yes|4.107ms|
+| A2: return to SSH/Tailscale |53/96|1|No|4.860ms|
+
+All288 probe/report exchanges succeeded and both interval windows/readiness decisions
+were independently reconstructed with integer arithmetic. A/A2 fail without any scene
+processing, so the shared source/receiver event loop is not a sufficient explanation.
+Mac scheduling-lag maxima A/B/A2:2.535/4.038/4.110ms; Worker maxima2.040/1.934/2.040ms.
+Worker Poll handlers stay below0.217ms; Report handlers below0.257ms in these probes.
+
+The LAN case improved the observed interval/availability, **not every latency tail**:
+Poll p95 A/B/A2 is88.838/92.403/89.950ms; B report/ACK p95 is98.266ms. Approximately
+100ms outliers remain even without models. These short measurements do not identify
+their exact OS/gRPC/SSH/TCP/network source, certify hardware clocks or establish a
+long-term route SLA. In particular, do not infer physical clock drift from the500ppm
+conditional rate budget or attribute every stalled request to Mac Python scheduling.
+
+## One full profile through the LAN route
+
+After the CPU route comparison, one32-frame full canary (`lan32-worker`) used B.
+Worker start18:57:32.273183Z; Mac source start18:57:42.275382Z. Source remains original
+1x, stride1, incremental raw camera/points/pose; no full transfer/predecode or previously
+computed perception inputs. The model container has no recording mount.
+
+The previous129 staged runtime/test source files remain byte-identical. DDRNet-39
+GOOSE, RF-DETR, LiDAR geometry/distance, motion, TRAVEL TGS/costmap/policy, model weights,
+effective config and pinned dependency image are unchanged. DDRNet and detector each
+execute32 times; no mask reuse/subsampling. One profile, serializedGPU,8CPU/8GiB,
+temporary stock-clock2610/10251MHz,450W unchanged. This is still a mounted diagnostic
+image, not a standalone image. Native host-process GPU exclusivity remains unproved;
+the existing observer's admitted scope is `docker-gpu-access`.
+
+32 released ->32 accepted ->32 Worker/Mac byte-exact results; no drop, synchronization
+skip, source/clock RPC error or post-start WAIT. One anchor, one input epoch, lease1,
+resident PIDs11/40/48/49. All32 decoded BGR, segmentation, proposals, observations,
+tracks, threats, material, lineage, sensor binding and raw costmap comparisons match
+the uninterrupted reference. Thirty results arrive before source-window EOF.
+
+All50 full-profile clock exchanges reconstruct independently. First2 are pre-anchor
+warmup; the remaining48 stay ready. Published uncertainty3.645–4.771ms, with the
+unchanged<=5ms gate,2s expiry,16 samples,500ppm/50us conditional envelope.
+
+Same-Mac source-due -> consumer-ready: min86.598ms, median121.850ms,
+p95/p99 205.864/213.751ms. **125ms latency gate FAIL**, not a general realtime pass.
+The prior8-result sample is not a matched latency benchmark for these32 results.
+Seventeen source sensor pairs are available; only12 received scenes are fully fresh.
+Missing/aged source sensors and added processing/delivery age remain fail-closed.
+Receiving every scene does not make every distance/policy result current or usable.
+
+Source release lag max21.552ms; p95/p99 12.253/15.783ms. Peak reply440,369B,
+tracked input14,611,263B (<16MiB), sampled VRAM2357MiB, cgroup3360.91MiB.
+Mac source RSS62,865,408B,5.08s CLI wall. No24GiB exhaustion is demonstrated.
+Worker host-control recorded32 accepted observations of33 requests,0 expired;
+one request remains pending in its last EOF snapshot. Do not call this33/33 accepted.
+No post-warmup envelope violation/input pause; control thread and collector close.
+
+## What remains measurable, not assumed
+
+Worker-local compute-start -> local receiver p95/p99 74.243/76.322ms; detector
+p95/p99 18.301/21.848ms, DDRNet RPC19.633/21.304ms, decoder RPC18.447/23.343ms,
+queue wait26.850/26.891ms. Worker mapped source-due -> local receiver157.266/160.027ms;
+Mac source-due -> network arrival188.601/207.768ms. These intervals have different
+boundaries and cannot be summed/subtracted as independent p95 stage costs.
+
+The Worker performs diagnostic local parse/freshness assessment AFTER enqueuing a
+network publication. Its local-receiver timing is not actual Mac receipt timing.
+Its source accounting/release-lag fields intentionally remain externally owned, so
+local `zero_drops_complete_accounting`/`source_release_lag_25ms` flags are false there;
+the joined verifier proves zero losses and max release lag21.552ms. No report field
+was rewritten to turn unknown worker-only evidence into network qualification.
+
+Next Stage2 increment: instrument per-frame source packing/write, Worker ingress/
+decode/queue/publication and result read/parse boundaries, plus control overlap.
+Use bounded CPU transport checks first; separate queue/serialization costs from the
+remaining ~100ms transport completion outliers. Optimize the measured cause, then one
+short full-profile recheck. Keep budgets unchanged; no long GPU stress or blind model
+replacement. Controlled gap/slow-consumer and standalone remain later Stage2 work.
+Per-run connection routing belongs to Worker configuration, not the model Docker:
+this test did not hard-code a LAN address into the profile or promote a product default.
+
+## Verification, evidence and cleanup
+
+245 focused local tests PASS,2 Worker-only cases skipped locally;202 Worker tests PASS,
+including real TLS, malformed/stale bounds and Worker-only backpressure checks.
+Ruff/check-format PASS;131 measured Python files match current source and code-v1.tgz.
+The129 files from increment16 remain unchanged; new files add only bounded diagnostics.
+CPU before/after snapshots preserve container IDs, restart policies and observed states.
+
+Session `.runtime/perception-stage2-clock-route-20260902T1850Z`,67 artifacts.
+Archive SHA256 `552e12ce24bdd2a246edf1559c1251c35400d455447fe14474bef079a5e572ee`.
+Manifest SHA256 `2645a0faacb89b5af5d3756feea05550a12cc64820be926d32f2103b5f225caf`.
+`verify_route.py` reconstructs288 CPU exchanges; `verify_clocks.py` reconstructs50
+full-graph exchanges. `verify.py` reconciles all32 original frames, raw parity and
+receipt freshness; `finalize.py` verifies source hashes and closed resources.
+
+Temporary CPU/model containers, lease volume, collectors, tunnels, key/bootstrap files
+are gone. GPU clock locks reset; auto210/405MHz observed,450W unchanged. Four exact
+prior Mission Core service IDs restored, Triton200, TelegrafRunning. Ollama/Frigate
+remain exited/restart=no. Mac8000 PID33360 and identity-matched Worker telemetry work,
+8765/18561 absent. No legacy service repair, K1/motor action or external deployment.