docs(perception): record TLS and direct TCP latency attribution

experiments/perception/PERCEPTION_STREAM_STAGE2_TLS_ATTRIBUTION_2026-09-02.md

@@ -0,0 +1,120 @@
+# Stage 2 increment 18 — localization below the perception application
+
+2026-09-02,22:35 MSK. Code `6af4c20`. **One substantial transport tail localized;
+no runtime optimization or new full-profile real-time qualification claimed.**
+Stage2 remains open; Stage3/4, standalone and actuation remain unqualified/disabled.
+
+## Question and bounded method
+
+The preceding full LAN canary returned32/32 reference-exact scenes, but consumer
+p95/p99 was205.864/213.751ms and only12 scenes were fully fresh. Its clocks stayed
+ready while small control requests still sometimes took~100ms. This increment
+isolates that independently reproduced tail before changing models or data queues.
+
+Added optional encrypted TLS-record witnesses around the existing CPU clock probe:
+one relay at the Mac application edge, one inside the Worker diagnostic container.
+They record monotonic receipt/drain times, offsets, lengths and SHA256 of ciphertext
+records; they neither decrypt nor log payloads, TLS keys or capability tokens.
+RPC invocation/completion and Worker handler entry/return bind the observations by
+nonce. At most2 connections,8MiB/direction,4096 records/direction, bounded I/O and
+overall deadlines. This is diagnostic tooling, **not a replacement runtime route**.
+
+The relays can perturb buffering/segmentation/scheduling. Therefore three sequential
+cases used the SAME strict-host-key-pinned LAN SSH route and unchanged dependency
+image: A without relays, B with both, A2 without them again. Each case96 Poll/Report
+pairs, no recording/scene payloads, data grant, source anchor, GPU lease or model work.
+One1CPU/512MiB runc container at a time; only sparse control requests on the Mac.
+No service stop/restart, GPU-clock setter, Docker build or network configuration edit.
+
+## A/B/A retained the observed tail
+
+| Case | RPC count | Median | p95 | Max | RPCs >50ms | Clock ready |
+| --- | ---: | ---: | ---: | ---: | ---: | ---: |
+| A: no relay |192|9.789ms|96.093ms|116.939ms|13|95/96|
+| B: two witnesses |192|9.904ms|93.895ms|98.966ms|14|94/96|
+| A2: no relay |192|10.062ms|95.145ms|137.224ms|13|94/96|
+
+Readiness stays continuous after initial warmup in all three short cases. All288
+raw t1–t6 exchanges and both clock windows/readiness decisions were independently
+reconstructed. No5ms/2s/500ppm/50us gate was widened. The small difference between
+case quantiles is not a demonstrated optimization or an equivalence/non-inferiority test.
+
+B has211 request +213 response TLS records, **424/424 matching offsets/lengths/hashes**
+at both ends, no partial records or relay errors;192/192 RPCs match their handlers.
+
+| B local application boundary | p95 | Max including startup |
+| --- | ---: | ---: |
+| Mac RPC invocation → request TLS record |1.013ms|56.609ms|
+| Worker request TLS record → handler entry |0.473ms|2.692ms|
+| Worker handler execution |0.190ms|0.319ms|
+| Worker handler return → response TLS record |0.121ms|0.713ms|
+| Mac response TLS record → RPC completion |0.649ms|0.995ms|
+
+The first long B call includes connection/TLS startup and lacks admitted clock
+mapping; it is retained as unqualified for directional attribution. Of the other13
+calls >50ms,8 have an inter-edge request delay lower bound >40ms;5 have an inter-edge
+response delay lower bound >40ms. Examples: request84.236–93.023ms, response88.172–
+95.374ms, under the existing conditional clock envelope. Local relay drain maxima
+are0.215ms on Mac and0.113ms on Worker. No scheduler-lag subtraction is applied.
+
+These are application-edge timestamps, not packet departure, ACK or hardware-clock
+measurements. TLS-record/RPC association is for this one serial control exchange,
+not an asserted universal per-frame tracer. Do not sum/subtract unrelated p95s or
+claim that this measures large scene serialization, decoder queues or model speed.
+
+## Independent TCP check narrows it below gRPC/Docker/forwarding
+
+Sixteen sparse direct connections to the already-confirmed Worker SSH port22 used
+no SSH authentication, forwarding, gRPC or Docker. All16 connected and returned a
+valid SSH banner. TCP-connect times: min6.380ms, median25.338ms, max91.142ms;7/16
+exceeded50ms, with67.921–91.142ms peaks. This uses one Mac monotonic clock and does
+not depend on cross-host clock mapping.32 ICMP echo requests had no replies; this
+is not evidence of general packet loss or a dead Worker, and no firewall was changed.
+
+Sixteen native Worker loopback connections to its same SSH port: first8.229ms;
+remaining15:0.240–0.299ms, median0.264ms. The startup value is not hidden. This is
+an unpaired short diagnostic, not a promise of future LAN latency or proof that
+SSH forwarding contributes zero overhead to large data transfers.
+
+Read-only adapter inspection found **both ends on Wi-Fi**: Mac route uses en0;
+Worker192.168.68.50 belongs to Realtek8812BU Wireless LAN802.11ac USB NIC. Reported
+link866.7Mbps is not measured throughput, a low-jitter guarantee or the source's
+future300–500Mbps qualification. No specific radio/AP/driver/power-saving mechanism
+has been identified. In particular, do not infer an AWDL cause or a defective NIC
+from periodicity alone. A comparative Ethernet run should remove both Wi-Fi legs;
+it needs the operator's available physical connection, not a hidden config change.
+
+## Decision within the existing four stages
+
+One major observed control tail occurs below the perception application and also
+appears during direct TCP connect. Do not replace DDRNet/RF-DETR, increase VRAM, widen
+clock/latency budgets or tune JSON blindly to fix that symptom. This does **not**
+exonerate all runtime costs or establish that the entire previous206ms p95 is network.
+
+Continue Stage2 with per-frame source pack/write, ingress/decode/admission, GPU/CPU
+queues, publication and result parse timing. A separate code finding is the10ms
+polling loop for ready replies in `streaming_grpc.py`; its actual contribution and
+an event-driven replacement need their own bounded tests. It was not modified here.
+Ethernet comparison can further split external jitter when available, but is not
+a prerequisite for optimizing/packaging the runtime under the accepted experimental
+scope. No long GPU run, new profile, standalone claim or Stage3 UI cutover was made.
+
+## Verification and retained evidence
+
+82 focused local tests and82 Worker tests PASS; Ruff/check-format and diff check PASS.
+All142 staged files, including132 Python files, match the local code, remote snapshot
+and archive. Production transport/clock implementations, graph, models, weights and
+effective profile config are unchanged; only isolated diagnostic code/tests changed.
+
+Session `.runtime/perception-stage2-tls-attribution-20260902T1923Z`;43 artifacts.
+Archive SHA256 `94affc1d74a6835880f76145cc4245d29e6ab559025a8bcdb8c11762642ca793`.
+Manifest SHA256 `abc3ac677b81e0d78f8d5891517022e33ac7fa962166bc399b4009ee48d87d8e`.
+`analyze.py` independently reconstructs clocks and links every ciphertext record;
+`finalize.py` checks code hashes, tests, exact resources, direct/native TCP evidence.
+
+All temporary containers, relays, tunnel PID52687, bootstrap files and private key
+are gone. No GPU workload/clocks or durable service settings were changed. Container
+IDs/restart policies match preflight; existing autonomous restart counts changed
+for m49 agent2→3 and perception worker18→23, not a claimed repair. Ollama/Frigate
+remain exited/restart=no. Canonical Mac8000 PID33360 and identity-matched Worker
+telemetry work;8765/18561 absent. No physical K1, vehicle/motor or external deployment.