NODEDC_MISSION_CORE/docs/adr/0049-stream-first-perception-profiles.md

ADR 0049 — Stream-first full perception profiles

Product scope/order partially superseded on 2026-09-02 by ADR 0050: recorded calculation and cached review come first. Live-stream contracts and historical measurements below remain intact; remote realtime is no longer the Observatory release gate.

Date: 2026-09-01; updated 2026-09-02 19:42 MSK. Status: stage-1 engineering baseline complete; stage-2 has full graph container-network proof plus authenticated cross-host grant delivery and bounded clock intervals. Continuous clock readiness is not yet connected to full-graph freshness. Cross-host full-graph qualification, native-host GPU inventory, standalone image and product cutover remain open. This is not physical-live acceptance. Execution order remains the four stages in the ExecPlan.

Decision and owner intent

Observatory LABs test the real-time behavior of a complete perception profile, not just the quality of an offline export. A recording replaces the physical source, preserving original release timing at 1×. It must not enable a different batch implementation. The first profile contains DDRNet-39 GOOSE, RF-DETR native, online LiDAR geometry/ranges, temporal/motion, CPU TRAVEL TGS/costmap and advisory policy. One dedicated RTX 4090 owns one active full profile.

The first rural prototype accepts existing coarse hard_surface, including asphalt, sidewalk, bikeway and cobble. Existing static_obstacle/static.unknown is sufficient. No new bollard classifier, extra segmenter, road/sidewalk taxonomy or annotation campaign precedes this prototype. Other material rules remain an explicit effective mission configuration; rural context alone is not permission for all vegetation or soil. Geometry/unknown/freshness overrides material allowance.

EoMT is preserved as a separate future profile. It is not loaded with DDRNet. RF-DETR and DDRNet are two required components of the same first profile: one supervisor and serialized GPU scheduling, not competing profile jobs.

CURRENT → TARGET / ownership

Boundary	Current evidence	Decision / stage
Portable definitions	portable_run_definitions.py owns pins, resources and admission	KEEP identity/sealing; EXTEND stream capabilities and qualification in stage 3
Source requirements	Recorded-only requirements include seekability, one media epoch and init digest	REPLACE as live admission criteria with channel/format/calibration/clock requirements; keep init digest as per-stream integrity
Job queue	recorded_jobs.py has durable claims but global single-active-job scope	KEEP durable queue; EXTEND worker-scoped fencing, not unbounded GPU concurrency
Worker agent	Blocking execute(job) and final publication	EXTEND incremental lifecycle and cancellation; no per-LAB agent
Runtime	portable_worker_runtime.py:735 materializes before execution	NEW stream runtime in stage 2; never call this materializer in the real-time path
Source transport	Whole camera archive download/hash/extraction before use	REPLACE startup barrier with bounded metadata, chunks and incremental integrity
Installed LAB V1	prepare → complete EoMT → complete DDRNet → assemble	Legacy evidence only; not the new profile and not real-time qualified
Camera/source replay	recorded_source.py has useful pacing and bounded prefetch, but source-specific timeline and paths	REUSE mechanisms; generic unknown-duration source adapter, no route-sized prefetch
Existing live ingress	LivePerceptionIngress, LiveIngressEvent.wire_bytes, LiveSensorSynchronizer already provide raw-first modality queues, hashes, epochs and bounded binding	REUSE, not a second acquisition system; extend worker/profile fencing and aggregate byte limits; map existing envelopes to the selected transport
Local surface	RecordedGeometryStore reads prepared NPZ	REUSE K1LocalSurfaceShadowEstimator.process and lidar_local_surface_geometry; feed current points/pose
Geometry/ranges	Current point association, exclusive ownership and geometry-only clusters exist	REUSE algorithms; remove recorded-store dependency and expose estimator/frame identity
Temporal/motion	BoundedSpatialTemporalProvider, ClassIndependentMotionEstimator	REUSE with current-run inputs and reset on epoch/lease change
TGS	Core algorithm reusable; old runners read a full schedule, some require exactly 4489 frames	REUSE TGS parameters/core; REPLACE schedule/file CLI with incremental input
Fusion	M49 TGS can be a neighboring shadow with tgs_modifies_reference_graph_state=false	TGS and semantics must influence the current advisory scene, not only viewer composition
Publication	Verified immutable result/publication/recovery exist	KEEP asynchronous final sealing; live results do not await it
Frontend	Setup/job/result surfaces exist, historical M49 viewer links other results	EXTEND common renderer capabilities with current-run layer references and freshness in stage 3
Recording archive	Source identities and raw evidence must be retained	KEEP; create metadata/index/integrity during acquisition or incrementally, not via preflight full scan

No changes to the active registry, old queue, source endpoint, UI or canonical local server are made by this ADR or the stage-1 validators.

Versioned handshake and observations

Current host observation adapter (2026-09-02 16:52, 61cbdb3)

WorkerControlChannel / WorkerControlPump supply the common readiness monitor with actual GPU name/driver/clocks and Docker image/configured resource limits / GPU-access inventory from collect_worker_host.ps1 on the trusted Windows host. Only the host collector executes fixed read-only Docker/NVIDIA queries. The AI container has no Docker socket, host setters or network access. Request and reply directories are separate; the reply mount must be read-only and cannot also be reachable through the writable output mount. This local filesystem authority is NOT authentication for external sensor/GCS traffic.

One pending random nonce, activation digest, sequence and a16KiB reply bound prevent history/queue growth. The collector reads facts AFTER receiving that challenge. Observation time is the local Linux request-start monotonic value, not the reply receipt or Windows timestamp. Delayed replies stay old; missing, malformed or different-nonce replies cannot refresh readiness. A known matching reply with changed container/image/owner or competitors fences execution. Owner, effective config and warmup remain local-controller facts, never accepted from the sensor stream or inferred from utilization. The pump does not renew the lease; its I/O runs outside lifecycle locks and the heartbeat thread.

Scope is part of the sealed effective configuration: docker-gpu-access means observed Docker GPU-capable containers, NOT all native host/WSL GPU processes. It cannot satisfy an envelope requiring host-compute. Windows reports PID4 / Insufficient Permissions; native host coverage stays explicitly unproved. Incomplete inventory or unavailable GPU telemetry cannot be waived even by labelled-experiment; measured low clocks can be logged as overload conditions.

Bootstrap acquires the local lease and attaches readiness before spawning any model. Warmup telemetry lag blocks further startup without destroying already loaded children or invalidating an input epoch that has not started. Active stream lag pauses input, keeps models and local lease, then requires the existing fresh-keyframe/sensor resynchronization and temporal reset. The bounded pilot's outer watchdog remains; this is not yet a durable production controller.

Measured normal graph:128/128 exact raw outputs,50 real observations, p95/p99 91.157/97.946ms. Delaying a host reply2200ms:102 completed,26 intentional omissions, two expired requests, same four resident PIDs/lease; resume seq50, old temporal stores/cells absent, first scene fresh. Result receipt gap2.719s is NOT latency from the Windows fault start (clock domains are not subtracted). Both trials use the temporary authorized stock-clock envelope, not general qualification. 404 focused tests; evidence SHA-256 aac125bd1b182d164c95d18089793fa54bf9797478bce1afe3de2a1d9093f6ef. The first warmup abort and Windows atomic-replacement correction are retained in the experiment journal. Historical readiness notes below retain their dates.

missioncore.perception-stream-start/v1 is represented by StreamStart in src/k1link/perception/realtime_contract.py. It contains run/source/worker/epoch, lease generation, profile/image/effective-config/calibration identities, clock domain, input mode and channel declarations. The first profile requires camera, point-cloud and pose. Temporary absence of a sample is an explicit modality outcome, not removal of a required capability.

It deliberately has no total source bytes, frame count, total duration, whole archive, command, environment, complete member inventory or required EOF. The initial metadata budget is 64 KiB. Large constant model assets and FOV masks belong inside the image; calibration references resolve to bounded metadata.

The stage-2 wire mapping must carry:

• Run/epoch/lease binding on every message; source/channel sequence, capture timestamp and clock domain; independently verifiable payload length/hash.
• Camera codec initialization and then incremental encoded access units, or bounded fragments thereof. Native decoded representation is BGR8 800×600 KB4. Do not send an entire fMP4 epoch as one observation or turn all frames into PNGs.
• Current vendor-registered map point increments with point IDs, coordinate frame, units, scalar layout and retained raw attributes; map→sensor pose and calibration identity. The adapter owns vendor decoding; the profile is not a new K1 protocol implementation and cannot command the scanner.
• Explicit unavailable/gap/end/cancel messages. End is a termination event, never a prerequisite for first inference. Final recording inventory/hash can be sealed after consumption; per-chunk integrity is verified before use.
• Integer nanoseconds use protobuf integer fields; any JSON projection uses decimal strings for 64-bit times rather than lossy JavaScript numbers.

Maximum payload fragment: 1 MiB; total application inflight bytes: 16 MiB; pending decoded camera frames: 2. Fragment reassembly is included in that budget, with a deadline and incomplete-chunk accounting. Decoded image/tensor, cloud, costmap and archive sinks need their own measured bounds inside the RSS budget. Transport flow control alone is not a memory bound or a freshness policy.

gRPC bidirectional streaming is the first transport candidate, not an already measured implementation. HTTP control-plane operations remain. The browser does not connect directly to model RPCs. One authenticated data-plane endpoint feeds the supervised profile; model processes have no arbitrary external network. Protocol/codec selection must be measured on actual payload sizes in stage 2; changing an archive POST to gRPC without changing execution is not acceptance. Existing missioncore.live-perception-wire/v1 and result wire v2 are migration inputs: reuse their semantics and raw-first producer hooks. Their modality count caps alone permit more than the proposed 16 MiB aggregate inflight budget; an explicit byte bound is required. No duplicate new scanner feed is introduced.

Time and lifecycle

Stage-2 full-graph recovery increment, 2026-09-02 15:56 (70927ea): --recover-input connects lifecycle to PilotController, RecordingSource, BinaryGraphBridge, GPU/CPU calls and result binding. Source clock mapping is fixed before ingress; fault windows never restart/slow it. Bounded diagnostic --input-gap sequence:milliseconds is explicit in effective config. Reconnect polls quiescence without queuing old events; complete accounting distinguishes skipped from processed input. Historical one-shot defaults remain available.

The receiver waits for an independently decodable fragment and fresh causal sensor pair. Decoder-local index resets; original source sequence/timestamps do not. Actual decoder keyframe validation remains. The graph recreates CPU surface/geometry/temporal/motion/rolling/threat providers using the same configs and clears cached segmentation. RF-DETR backend, DDRNet/decoder/TGS children remain. TGS C++ creates an estimator per request; costmap arrays are per-call, so no native temporal reset command or restart is needed. Rolling input window resets independently. GPU/CPU/result gates bind the input epoch, not activation. Obsolete results release their input rather than terminating the profile; review removes a lingering collector tuple reference to prior input too.

Two normal/faulted full-profile pairs pass on Worker006. Normal raw parity is 128/128; final p95/p99 85.775/89.385ms. Faulted runs preserve four PIDs and clear nonempty temporal stores, resuming on seq30/110 after gaps at seq16/72. All post-resume histories and observed costmap timestamps belong to the new epoch; first scenes have fresh six-layer evidence and new DDRNet inference. No result was published during pauses. GPU calls were idle at injection; mid-compute interruption is covered by focused tests, not new real-GPU cancellation proof.

Recovery availability differs from accepted-frame processing latency: 150ms gap takes1.476s to next complete scene; 2.2s gap takes3.857s, including waiting for keyframe/sensor alignment. Faulted output is76 processed +52 skipped cameras, zero unaccounted; none is replayed. Its p95/p99 83.163/89.729ms describes accepted frames only. Timings use temporary stock clocks2610/10251MHz; whole-route availability, auto-clock and network timing remain unqualified. Real collector, authentication, cross-host clock mapping, standalone and onboard safety are open.

383 tests pass. Evidence330 artifacts,236 Worker snapshot hashes/114 current files verified; .runtime/perception-stage2-resume-worker-20260902T1545MSK/manifest.json, SHA-256 a9449fa4c012e30ce8a06025790c0e110eeebab7c1859fdebf56d5e785961ea3. Initial preflight abort (512MiB threshold below known534MiB idle baseline) is preserved; no models/clocks ran in it. Auto clocks/four scoped services restored, empty/retired fixture volumes and containers removed; legacy restart loops not fixed. The earlier CPU-only increment below is historical, not latest status. No full-source transfer, new model, UI or actuator path is introduced.

Owner clarification and stage-2 continuity increment, 2026-09-02 15:30 (c31c46c): temporary input/network loss must not unload the resident models. Unknown/stale GPU telemetry is not proof of a competing owner. The desired response is waiting and resynchronization; verified owner conflict, actual local lease loss, child/model/decoder failure and explicit cancellation remain terminal. This supersedes the blanket terminal-readiness TARGET below; previous terminal fault probes are retained as historical evidence of their opt-in mode.

Two lifetimes are explicit. The resident activation keeps its original StreamStart, local Worker lease and lease generation. Reconnecting input gets a new StreamStart.epoch_id, bound by existing wire v2 on every packet/result; acquisition session/generation remains independently checked. A network/GCS heartbeat is not the local ownership heartbeat. The trusted resident controller continues renewing its own lease while the input waits. No second profile may claim the GPU during that wait, and telemetry alone cannot renew/release a lease.

StreamingLifecycle(recover_input=True, source_clock_ns=...) requires a trusted source-to-local-clock mapping and, if monitoring is configured, a WorkerReadinessMonitor(recoverable=True). Both choices belong in the sealed effective config. Their defaults preserve the historical one-shot diagnostic; the current full PilotController/BinaryGraphBridge has not been switched. This compatibility default is not the final product's desired outage policy.

The input phases are active, waiting and synchronizing. Resident GraphState may remain RUNNING while input is waiting; that means the processes remain loaded, not that results or actuation are ready. EOF without End, an IPC read deadline, connection loss or a declared/source-sequence gap pauses the enabled adapter. Pending input is discarded; active borrowed buffers remain owned until their callback drains, and its old result cannot pass the epoch/phase gate. A new connection waits for the old receiver, active callbacks and mailbox-owned work to drain. No unbounded queue, replay catch-up, future evidence or new model process is introduced. Explicit End terminates the source; Cancel remains a real cancellation. Malformed data and actual decoder failure are not relabelled as a recoverable transport outage. External network error handling remains an unqualified adapter boundary, not proved by the IPC socket tests.

Resume requires a new init and a successfully decoded random-access H.264 frame, fresh pose/points from the new epoch, and a temporal-state reset. The measured decoder resets codec references/DTS inside the same child process; sensor-window reset retains its bounded allocation but clears rolling state. ResumeEvidence must be derived by the trusted profile adapter, not accepted as a client Boolean. Every timestamp is at/after reconnect cutoff and no later than the camera; camera age ≤250 ms, pose/newest-points age ≤100 ms, oldest-points age ≤250 ms and pose/points skew ≤100 ms. The lifecycle rechecks evidence after reset and cannot resume if another pause arrived meanwhile. Full graph tracking/motion/TGS/costmap reset and CPU-consumer old-result disposal are required next, not yet proved by the diagnostic callback. Replay source time must keep advancing at 1× through the outage; the CPU fixture proves this only with a synthetic tick source.

Recoverable readiness maps expired/missing inventory, incomplete warmup and strict-envelope performance unavailability to waiting, not process destruction. Fresh facts permit beginning resynchronization, not automatic reuse of the old epoch. Confirmed wrong owner/generation/worker/image/config, known competing GPU clients and invalid trusted clock/snapshot ordering still fence the activation. Labelled performance experiments keep their recorded violations and remain unqualified. The runtime does not manipulate GPU clocks, other services or a vehicle. Eventual onboard stop/hold, stale-command rejection and motion-resume policy are independent future safety gates; restored inference is not permission to move, and GCS heartbeat is not proof of fresh perception.

375 focused tests pass, including repeated reconnects, stale input/results, pause during reset, telemetry recovery and retained hard-fault fencing. Worker CPU-only proof: three epochs, 150-ms and 2.2-s outages, unchanged decoder/sentinel PIDs, identical BGR 3/3, peak input 5,275,461 bytes and complete cleanup. It uses one real H.264 fragment and synthetic timestamps/pose/points; the sentinel is not a GPU model, and temporal reset is a diagnostic fixture. No full-graph recovery, new GPU latency, real inventory collector, external network or standalone-image qualification follows. Evidence: 85 artifacts, 73 verified code hashes; .runtime/perception-stage2-continuity-worker-20260902T1518MSK/manifest.json, SHA-256 fbd327d129c88caa13b7183b1ae2c4767e877a6f5472c0f430875cfd7401ed46.

Stage-2 controller-readiness increment, 2026-09-02 14:52 (92625bf): WorkerReadinessMonitor consumes existing WorkerOperatingEnvelope and WorkerSnapshot under the same lifecycle lock as admission. No host I/O occurs under this lock. The trusted controller, not a sensor payload, provides the snapshot in a locally comparable Worker monotonic clock domain and seals the mode/envelope into its effective configuration. Snapshot arrival is not a lease renewal, and renewing a lease does not refresh inventory. Identity, exclusive ownership, known inventory and freshness are mandatory in both modes.

strict-envelope rejects post-warmup performance conditions outside the preregistered envelope. labelled-experiment permits these conditions, including unknown performance facts, but records violations and never grants real-time qualification. It cannot waive unknown/competing GPU clients, stale inventory, changed worker/image/config/owner or missing warmup. Low clocks during STARTING do not prohibit model warmup; transition to RUNNING checks warmup explicitly. Freshness is checked before spawn/admission/compute/result/heartbeat and in the 50-ms watchdog. A late telemetry refresh cannot resurrect expired authority; old StreamStart updates are rejected without stopping the current owner.

Readiness loss enters STOPPING and stops only owned child groups. A new owner still requires verified child/thread/callback/input release and a new generation and epoch. The monitor cannot clear an active durable lease, force recovery, start a replacement model, change host clocks/quotas or stop other services. It retains one snapshot and bounded reason sets, not unlimited telemetry history.

The existing diagnostic PilotController has no continuous real collector yet; unconfigured runtimes expose worker_readiness.enabled=false and cannot claim monitored readiness. No authentication or host-clock translation is implemented by this in-process method. External inventory collection, control-channel delivery, production admission/recovery and standalone packaging remain stage 2. 349 focused tests and four sequential Worker Linux CPU-only scenarios pass. The latter use synthetic GPU facts, zero GPU devices/models and real child processes: stop 10.676–63.994 ms, zero residual input, released fixture generation 4. These are not new GPU graph timings. Evidence manifest SHA-256: 4588995d763fe82d896b42689e981f1d00feedaf9d6aab29f4579f771d03da3a.

Stage-2 raw IPC increment, 2026-09-02 (da60fef, a937400, 79ce55d): streaming_wire.py maps existing LiveIngressEvent fields into missioncore.live-perception-wire/v2. The v1 producer hooks, class and legacy serialization remain unchanged. A bounded length-prefixed JSON header carries decimal uint64 projections; payloads remain binary. This is a controller-supplied IPC socket candidate, not a selection/replacement of the intended gRPC network transport. No listener, arbitrary endpoint, scanner feed or command path is added.

Open compares the entire expected StreamStart and acquisition session binding; every fragment/control carries a digest of that full binding. A hash is integrity, not authentication. Fragment size is at most 1 MiB, event caps reuse the existing 1 MiB camera / 2 MiB LiDAR/pose limits. Headers are capped at 64 KiB. One observation assembles at a time with exact offsets, fragment and whole-event SHA-256, per-channel source identity/sequence/clock checks. End, Cancel and gap/unavailable notices are explicit. EOF without End, timeout, malformed, cross-binding or incomplete input fails the bound stream; an unbound wrong Open is rejected without stopping the current owner. Camera frames require init; camera gap invalidates init. Actual codec/keyframe/IDR validation is NOT yet here.

StreamingSender owns no queue and fails on its bounded write deadline rather than stretching replay time. StreamingIngress is a lifecycle-tracked thread; idle reads poll the lease and cannot keep accepting an expired owner. Raw/header and reassembly reservations charge the SAME StreamMailbox 16 MiB input budget as queued/active bundles. They survive cancellation until the borrowing callback returns and buffers are dropped. Reservations are count-bounded too. Trusted decoder adapters must reserve scratch BEFORE allocation and must not retain raw callback bytes outside that ownership contract. Python metadata object overhead, socket/kernel buffers, source adapter memory and model tensors additionally need RSS/OS resource bounds; this application counter alone is not a total-RSS claim.

Worker evidence uses a separate source process, original camera init/fMP4 segments and existing normalized point/pose increments paced at original 1x times. All 103 events (32 camera + init + 34 LiDAR + 36 pose) match exact bytes/metadata, each data modality reaches the hash-only consumer before End, and incomplete EOF delivers no observation. Normal peak input is 553,991 bytes. This validates raw IPC, NOT decoder output, the full graph on this ingress, model performance, authenticated network behavior or a standalone image. The next adapter must feed incremental decode and causal sensors into the same graph without using the old concatenated camera file or precomputed geometry. Product source queues, backend routes and the browser are not switched by this increment.

Stage-2 decoder increment, 2026-09-02 (350366b, f102712): media_fragments.py now owns the existing bounded ISO-BMFF timing parser. Archive inspection delegates to it and preserves SessionIntegrityError at its boundary. perception.streaming_decoder.FragmentDecoder imports neither the session store nor its materializer. It receives only bounded init/fragment bytes; source filenames, recording length and EOF are not decoder inputs.

The measured subset is H.264, one 800×600 frame per fragment, one video track, explicit moof-relative sample size/offset and no composition offset. The first fragment must be random-access and the decoded frame must be keyframe; DTS must remain contiguous. One persistent codec context, one slice thread, no flush or future packet: each call must yield exactly one frame with the same PTS/raster. Reordered/B-frame streams, unknown layouts, discontinuities and invalid output fail closed. Other valid camera formats require explicit qualification, not a hidden decoder fallback. These are current profile bounds, not a new K1 protocol.

PyAV 18.0.0 is pinned to wheel SHA-256 ae56b40b6f8b067a8ad2dac664fbfbabac7f7a55b9a7bb031eb99289252bc017. It is installed only into a private experiment target and mounted read-only; standalone packaging must include it. The public PyAV codec documentation describes persistent packet decoding and thread modes; compatibility claims here come from the actual 18.0.0 Worker probe, not from documentation version alone.

Sequential CPU-only 128-frame 1× comparisons produced exact BGR parity against the previous pinned OpenCV decoder. Direct validated sample extraction replaces opening/demuxing init+fragment on every frame; NumPy loads before readiness. Measured new-decoder p95 improves 19.927→11.156 ms, p99 20.920→18.675 ms. This is one bounded component comparison, not a full-profile speedup claim. The reference full-MP4 path exists only in the separate pixel-comparison process; neither new decoder path mounts/reads that MP4.

Combined IPC/decoder probe accepts 103 raw events and yields 32 exact BGR frames. 31 frames arrive before the source End marker; the final frame drains before the receiver handles End. First decoded callback is 3.071 s before the marker. Input reservations peak at 5,727,751 bytes including conservative decoder scratch and init/extradata retention; normal and truncated-input cleanup return to zero. Native codec/DPB allocations are bounded separately by the CPU-only container; application payload counters alone do not constrain native allocation or execution time. Before full-graph integration, native decoding must run in a supervised CPU child with bounded RPC and explicit decoded-bundle ownership transfer.

Point/pose are preserved raw in this probe, not causally fused into a scene; calibration identity is explicitly non-scene/synthetic. No full graph, GPU model, network, physical live feed, vehicle control or product cutover is qualified. Next: preserve the existing causal sensor cut at camera release, connect decoded bundles to the same full graph, then repeat full-profile parity/timing. The last full-profile p95/p99 remains 124.98/136.12 ms, not this decoder's timing.

Stage-2 full binary graph increment, 2026-09-02 13:55 MSK (221e429): the preceding decoder-only boundary is now connected to the existing full graph. The recording adapter alone reads bounded current fMP4 fragments and normalized map point/pose rows. It emits original timestamps at 1× through the existing binary socket ABI; the consumer and decoder receive neither source paths nor recording length. The diagnostic container still mounts source/code/weights and a private pinned PyAV target: this is not standalone packaging or a network test.

Native decoding runs in a lifecycle-owned CPU child with a 1-GiB address-space limit. POSIX pipe RPC reserves header scratch, accepts ≤1 MiB input and writes exactly 1,440,000 BGR bytes into the caller's pre-reserved buffer. A 250-ms request deadline, malformed reply or lost lease terminates the stream; no implicit retry across predictive codec state. Process/native memory remains under container limits, separately from the 16-MiB application payload budget.

streaming_sensors owns the existing causal selection rules. Its rolling and fresh caches are each bounded by 64,000 points/64 increments; camera arrival freezes the original-time cut before decode, never selecting future pose/points. The legacy pilot delegates to the same selection rules. A separate CPU oracle compares the old producer's complete points/rolling/times/pose/lineage bundles. admit_reserved transfers the decoded allocation atomically to the common mailbox; rejection retains caller ownership until buffers are discarded. The global two-pending limit and active-input protection remain unchanged.

Worker evidence: 384/384 input events, 128/128 full graph outputs, no drops, 126 outputs before source End; first result 583 ms. BGR, masks, proposals, tracks, threats, material and raw TGS costmap match the reference on 128/128. Two metric_geometry.range_m values differ by 1.3877787807814457e-17 m (seq13/observation3 and seq29/observation2): strict raw parity remains false. No rounding/tolerance was added; the numerical cause remains unreproduced. The geometry distance-estimator definition and all thresholds are unchanged.

Full p95/p99 161.974/191.806 ms FAIL the unchanged 125-ms target. Available camera/sensor pairs remain 76/128; 74 scenes are fresh at receipt, with seq106/107 additionally expiring. Auto GPU memory clocks fall from 10,251 to 405–810 MHz; this correlates with longer DDRNet/queue times, but is not a controlled transport A/B. Peak input 8,883,895 bytes; remaining input/children/lease zero after stop. VRAM peak 2,367 MiB is not GPU utilization or a compute-capacity percentage. Normal 32-frame CPU oracles and a synthetic hung-decoder cleanup probe pass.

The full GPU sample preserves its exact measured code snapshot. Subsequent review bounds finite-check scratch, closes sockets/cache after constructor or pre-start failure, and removes CPU-harness heartbeat shutdown noise; those three files pass a separate Worker CPU oracle, not a second GPU performance sample. 271 focused tests, Ruff/format and five-module mypy pass. Four temporarily stopped Mission Core services are restored; Ollama/Frigate remain exited/restart=no. Manifest: 161 artifacts, SHA-256 e224de4e9f2c5fd419fe0fb0b9be6e01ec267628568e104795ae211ec803bce4, .runtime/perception-stage2-binary-graph-worker-20260902T1335MSK/manifest.json. Next: explain numerical reproducibility, measure/tune remaining pipeline tails, then complete controller/network/standalone boundaries within stage 2. No quality, physical-live, vehicle-control or real-time qualification is granted.

Stage-2 numeric/IPC increment, 2026-09-02 14:22 MSK (380b6ea, f39f1ff): the two preceding raw range differences are resolved, not tolerated. A bounded CPU-only probe on Worker NumPy 1.26.4 reproduces both exact deviations using identical quaternion values at addresses 8 mod 16. The wire quaternion starts at byte 24 of a combined pose buffer; its norm changes by one ULP on that layout. Offsets 0/16/32/48 restore reference norm/rotation/depth, while 8/24/40/56 reproduce the deviations. The adapter now owns an immutable 32-byte quaternion copy and checks 16-byte alignment. Input bytes, projection algorithm and tolerances are unchanged. Other CPU/NumPy combinations still require numerical qualification.

Synchronous local IPC now borrows contiguous image buffers. Decoder and parent no longer call BGR tobytes() for these handoffs; the sender writes framing and the borrowed body separately instead of concatenating a full message copy. Partial writes loop with explicit progress validation; noncontiguous buffers fail before output. The owner retains the image until send completes. This removes four application-level BGR copies across decoder→parent→DDRNet, not kernel copies, network bytes, or all intermediate allocations. Framing and lease/deadline checks are unchanged; a hung-decoder probe still fails and releases all owned resources.

Preregistered sequential A–B–B–A at authorized temporary stock clock locks: all samples observed SM2610/memory10251 MHz, unchanged 8 CPU/8 GiB/power limit. Every run completes 128/128 without drops. Control p95/p99 84.702/92.961 and 83.930/91.759 ms; candidate 85.113/91.219 and 82.425/90.530 ms. Both candidates match reference BGR, masks, objects, geometry/ranges, tracks, threats, materials, lineage/binding and raw costmap exactly on 128/128. Baselines still reproduce the two old float deviations. Median decoder RPC-minus-decode falls 1.642–1.825 to 1.100–1.118 ms; DDRNet RPC-minus-component 2.109–2.217 to 1.558–1.613 ms. This is a modest IPC improvement; full p95 does not show an unambiguous win. The previous auto-clock 191.806-ms p99 cannot be compared as a copy-only baseline.

Local 125-ms timing passes at this fixed operating envelope, but only 76/128 scenes are fresh because the same 52 input sensor gaps remain. No overall or auto-clock/network/physical-live qualification is granted. Auto clocks and four Mission Core services are restored; Ollama/Frigate remain disabled. 279 focused tests pass. Manifest: 305 artifacts, SHA-256 9c72a93833a1dda51c4787dca13fb9791a1bf2e31c763b7740e1b21406703707, .runtime/perception-stage2-ipc-worker-20260902T1410MSK/manifest.json. Next implementation boundary is the controller/envelope/network integration and standalone package. Host clock management is not granted implicitly to that controller.

Open → validate bounded metadata/claim → warm models → Ready → start the replay clock → observations and incremental scenes → stop/drain bounded state → close and asynchronously seal the immutable receipt. A live source already producing data does not queue its entire warmup history: readiness begins at a current decodable keyframe, with the skipped interval recorded.

Replay release uses original source intervals, not a convenient fixed 12 Hz and not the processing speed. A slow consumer must not stretch source time to make inference look real-time. Overload may drop bounded work to preserve liveness, but the dropped observations remain in the ledger and fail the strict initial performance target. Initial baseline is every camera frame, stride 1. Choosing multirate later changes the explicit profile contract; a retained mask is not a new inference and cannot receive a fresh capture timestamp.

UTC identifies the run; monotonic clocks measure local durations. Cross-host age requires a clock mapping and its uncertainty. No direct subtraction of unrelated host monotonic clocks. The common source timeline is separate from worker wall-clock time. Causality is bounded by the released-observation watermark at decision time, not by whatever later samples exist in the file. Each layer retains its own channel/sequence/time; camera anchors and pose/cloud association times remain separately visible. Future file look-ahead and offline nearest-pose lookup across unreleased data are forbidden.

Lease expiry, cancellation and epoch changes fence the previous producer. No new profile may take the GPU until old processes and work have stopped. A reconnect creates a new epoch, discards stale backlog, resets temporal/rolling state and reacquires codec initialization/keyframe. It never resends a stale motor action; this profile has no motor actions at all.

Outputs and interpretation

Every scene accounts for segmentation, objects, geometry, motion, costmap and policy exactly once, with current/held/stale/unavailable state and current-run payload identity. Existing source/object/obstacle/temporal/map/threat contracts remain the domain vocabulary. LayerEvidence adds freshness/completeness checks, not a second semantic ontology. In stage 2, layer manifests also carry their full input reference sets; a single anchor does not replace camera/cloud/pose lineage.

Stage-1 implementation update, 2026-09-02 (097e450): missioncore.perception-scene-freshness/v1 in realtime_scene.py extends LayerEvidence with oldest-required-input time and checks the six-layer dependency graph. A derived result cannot refresh its input age. Decimal int64 JSON strings preserve timestamps/sequence exactly; transport-neutral Python objects use integers. The pilot validates payload hashes and assesses freshness again at receipt, and must reassess at later use. Missing/stale output remains inspectable but cannot retain permissive advisory policy. This envelope is NOT a replacement for StreamStart identities, complete input lineage or controller lease fencing.

The stage-1 pilot propagated the oldest last-seen permissive TGS cell into the costmap/policy age and suppressed the entire scene on expiry. Stage-2 increment 1f8101e adds missioncore.costmap-cell-freshness/v1: bounded (8192 cells), index-aligned original support timestamps, decimal int64 strings/null, included in the costmap payload digest together with the guard mode. Unobserved support is never assigned a timestamp. The pilot now defaults to per-cell; the old whole-scene mode remains an explicit comparison control, not a product cutover.

Both publication and receipt remove permission from expired ground cells and mark them rejected/NO_GO. Occupied cells retain prohibition. The remaining permissions propagate their original support age AND the mandatory segmentation, geometry and motion dependencies. A missing/stale mandatory layer still blocks the whole policy. A derived consumer view has new costmap/policy hashes while preserving source identity/timestamps; the original published bytes stay immutable. Reassessment cannot restore a suppressed action or move the observer clock back. The enclosing StreamStart must bind the grid/profile/epoch/clock; this descriptor alone does not establish those identities or authenticate a remote producer.

In the measured 128-frame window, raw model/geometry/motion/TGS/material outputs remain identical to pinned PyTorch. Effective costmap states and permissions change deliberately on expiry. A fresh six-layer envelope can still contain rejected/unknown cells; it is not a claim of full free-space coverage, semantic correctness, vehicle clearance or physical safety. Recheck again at actual use.

bcacb02 promotes the proven mailbox and serial GPU stage to common perception modules, with the pilot importing compatibility aliases rather than owning another scheduler. Pending ingress + completed GPU outputs share two slots; active inputs share 16 MiB and are released by their owning stage. Cancellation discards pending work, a timed-out callback retains ownership, and CPU-owned input is not freed by GPU shutdown. Diagnostic drop history is capped at 256 entries with exact per-reason totals; it is not the durable terminal ledger. These primitives are NOT the controller supervisor, a cross-process GPU lease, StreamStart fencing or the binary data plane. Those stage-2 boundaries remain open.

Stage-2 lifecycle increment, 2026-09-02 (6acf468, ac69e3b): StreamingLifecycle now supplies the subprocess-backed profile supervisor using the existing GraphState and full StreamStart identity. WorkerLease uses a stable POSIX file lock on one controller-selected, private directory per Worker, plus an atomic/fsynced ownership record. All managed profile containers must use that SAME directory/volume. Neither the directory nor child commands come from an incoming source/job. This cooperative local fence supplements, not replaces, the backend claim and the trusted inventory of unmanaged GPU clients.

Open acquires ownership before process spawn/warmup; Ready enables admission. Renewal cannot resurrect expired ownership. Admission, each GPU/CPU lane and result publication/receipt validate run/source/worker/epoch/generation and pinned image/profile/config/calibration/clock identities. A mismatched client is rejected without cancelling the current owner. Each compute lane remains single-owner. The watchdog fences an expired owner even without a new frame and terminates only its dedicated child process groups. Cleanup keeps ownership while callbacks, tracked threads, pending/active payloads or child groups remain. Clean retirement persists released before unlocking; a higher generation and new epoch may then activate. A controller crash leaves active even when the kernel unlocks, so the next owner stays quarantined. No force/unverified recovery API exists.

Cross-container contention/clean succession/crash quarantine and full-profile lease expiry were exercised on Worker 006. The final timed expiry trace had no receipt at/after the lease deadline, stop requested after 45.74 ms and retirement after 4109.82 ms. This is bounded evidence, not a hard-real-time scheduler guarantee. The holder has a 2 s lease and receives local pilot heartbeats every 250 ms; those settings and the 50 ms watchdog poll are NOT network/vehicle safety limits.

Current integration remains explicitly diagnostic: pilot_lifecycle.py acts as a local controller, legacy GPU services are quiesced by the authorized launcher, the named volume is an isolated test volume, and the image still has code/model mounts. Mounted-code hashes remain in the experiment manifest; an image digest alone cannot seal those overrides. Backend claims/production-wide canonical root, trusted post-crash resource-release recovery, continuous GPU inventory/envelope enforcement, network authentication/heartbeat and binary live ingress are NOT installed or qualified by this increment. Standalone packaging must remove the developer mounts and retain the same lifecycle boundaries.

worker_operating_envelope.py checks a trusted post-warmup snapshot against preregistered hardware/driver/resource/clock conditions and StreamStart identities, ownership and client inventory. Unknown or expired facts fail readiness. The candidate records the previously measured 4090 fixed-clock envelope; a compatible auto-clock experiment cannot inherit its latency result. The evaluator performs no I/O, host clock changes, lease acquisition or qualification; collection and continuous enforcement belong to the common controller/runtime in stage 2.

DDRNet currently crops the center 600×600 to 512×512; outside that ROI is undefined, not hard_surface/free. RF-DETR accepts the full 800×600 native raster with its pinned valid-FOV handling. Their coordinates must be related explicitly.

RF-DETR's existing filter emits person/cat/dog, but minimum box area 64 px, maximum box fraction 0.5, FOV fraction 0.5 and required valid center create known small-animal/close-large-object edge cases. They are retained and documented, not silently retuned or presented as proven field quality.

Detected-object range is the median camera-Z of owned current support points; geometry-only range is nearest Euclidean distance from the sensor. Neither is automatically clearance from the physical vehicle body. Missing/ambiguous support produces unavailable range, not zero/infinity. Generic static obstacles survive absence of a detector class name.

TGS keeps ground/occupied/rejected/unobserved separate. Ground support alone is not a traversability or actuation decision. Policy can produce an advisory allowed-candidate/high-cost/blocked/unknown outcome with explanation. No planner, physical motor command, autonomous driving acceptance or mission configurator implementation is introduced.

Package/dependency decision

The candidate manifest is config/perception/k1-perception-ddrnet39-rfdetr-tgs-prototype-v1.json. It is not installed into the active portable registry and has no invented image digest. Model/checkpoint/config identities are pinned; source session IDs and old derived output paths in reference experiments are not inherited requirements.

Select process/environment isolation inside one future image: Python 3.12 supervisor and geometry, RF-DETR TensorRT 11 runtime, existing Python 3.9 / Torch 1.13.1 cu117 / super-gradients 3.2.0 environment, and CPU C++ TGS. Exchange bounded shared-memory/IPC payload references; one supervisor serializes GPU work. This avoids a forced dependency upgrade or checkpoint conversion in stage 1. Image assembly of the final standalone package and the full schedule pilot remain explicit implementation evidence. A temporary common-base image passed separate DDRNet, TensorRT, Python 3.12 geometry and C++ TGS execution probes; DDRNet masks matched on 64 frames. This proves bounded ABI/execution compatibility in that image, not jointly resident models, the supervisor/IPC or a complete profile. The probes still mount pinned assets explicitly; they are not standalone proof.

The temporary Triton base lacks Python grpc/protobuf, cv2 and TensorRT bindings; native trtexec/server are present. The actual supervisor/transport environment must explicitly include and validate its dependencies. Numeric-library thread limits are pinned to one after the bounded synthetic local-surface comparison (mean 198.0 ms default versus 63.2 ms with limits); algorithm thresholds were not changed. Real-cloud timing, output parity and whole-graph scheduling remain gates.

The initial DDRNet image-only probe failed before inference: the installed image does not contain the checkpoint at its logical asset path. The bounded baseline therefore uses an explicit read-only pinned checkpoint and runner mount. This is valid component measurement, but fails the target's standalone packaging claim. Super-gradients also needs a writable log directory; a bounded temporary log mount was required. The new image must declare scratch areas explicitly without requiring a writable host home, code checkout or implicit weight cache.

Preregistered engineering acceptance

Owner clarification, 2026-09-02 MSK: bounded laboratory overload is an admissible experimental outcome. Keep useful profiles that fail the current 4090 target; do not block reusable runtime/packaging work solely on that performance result. Functional execution, experimental availability, quality and real-time qualification are separate dimensions. Qualification belongs to a measured profile/config/hardware/source/transport combination, not permanently to a model name. More powerful Worker/onboard placement is a future retest target, not a claim that existing latency or internal IPC disappears automatically. The original budgets and failed measurements below remain unchanged. No slowed source clock, hidden drops, stale-as-current evidence or actuation is authorized.

The manifest freezes an initial engineering candidate: p95/p99 full output age ≤125 ms, required-layer age ≤250 ms, release lag ≤25 ms, first incremental result ≤1 s after source admission, warmup ≤120 s, stop ≤5 s, VRAM ≤22000 MiB, RSS ≤8192 MiB. First/last-window backlog growth ≤25 ms and no capacity drops, failures, expired selected frames or unaccounted observations. These are prototype targets, not physical braking/safety limits or claims of achieved performance.

ReplayMeasurements/realtime_failures reject full-source preload, EOF-only results, slowed replay, incomplete accounting, hidden missing layers and budget violations. Declared source gaps must be verified from the input ledger; they produce explicit degraded scenes and are not counted as fully fresh perception. At least one fully evaluable scene is required. A component FPS figure cannot populate this whole-path receipt. A quality comparison or successful export is not a substitute for the real-time gate.

Every observation receives a terminal ledger outcome. Track ingress/release, selection, decode, inference, completion, emission, drop/expiry/failure and source gaps separately. Store startup separately from steady state; record payload bytes, queue high-water marks, clock error, RSS/VRAM, image/weights/config/source/equipment identities. Export and UI encoding cannot block inference through an unlimited queue. A receiver-side render/receipt timestamp, not just Worker completion, ends the end-to-end interval.

Evidence and remaining stage-1 work

Operator telemetry is not runtime authority (2026-09-02)

Reuse System/Worker and the existing Telegraf → MQTT → Timescale route for operator observability. A profile may export one bounded 8 KiB current snapshot at 1 Hz to the host collector, independently from controller heartbeat/model work. Missing or stale observations are unavailable, never fabricated idle, loaded-model readiness, per-stage timing or real-time qualification. Agent restart/reconnection and database retention must not block inference. The faster Worker-local readiness/inventory channel remains separate; MQTT/Timescale/UI cannot renew a lease or authorize GPU/vehicle activity. This observability integration does not constitute the external camera/points binary data plane.

See experiments/perception/PERCEPTION_STREAM_STAGE1_2026-09-01.md for measured component results, bounded validation, Worker maintenance and remaining gates. Neither this ADR nor passing synthetic contract tests marks the new runtime ready.

Stage 2 increment 12: external gRPC/TLS candidate (2026-09-02)

streaming_grpc.py exposes /missioncore.perception.v1.BinaryStream/Exchange. It is an optional perception-stream dependency, locked grpcio1.83.1; it is not auto-started by the backend or installed into the product LAB path. Both sides require TLS; the client verifies its supplied trust root and server name. The local controller alone issues a256-bit random single-use capability with30s admission expiry, bound to the complete StreamStart and acquisition session. The endpoint retains only its digest. Invalid/duplicate/stale credentials cannot reserve a stream, stop its owner, renew a lease, launch a model or create an epoch. One active stream is admitted; reconnect needs a controller-approved new epoch and, separately, real decoder/causal-sensor resumption proof. Network disconnect is not explicit End or Cancel. Production grant delivery is still an integration gate; the CPU probe transfers ephemeral files only through authenticated SSH.

The generic gRPC method uses identity byte serialization, not generated protobuf messages. Request messages contain≤64KiB pieces of the unchanged MCI2 wire; StreamingIngress still validates metadata, fragments, hashes and epoch/source binding. No archive, duration, file path or executable command is sent. Response MCR3 is only an envelope over existing domain bytes: magic4, uint64 sequence8, binding SHA25632, payload SHA25632, then1..1MiB payload. It does not invent a new scene ontology or claim that full-graph scene/layer serialization is connected. The actual profile's larger outputs require reviewed fragmentation, not a silent cap increase. Result age validation and the full result receipt ledger remain with the graph/application adapter, not the transport envelope.

There are two pending latest replies and one writer/reader per direction; no background source queue/retry or event-loop callback backlog. Python bridge storage is reserved inside the existing16MiB mailbox budget. Fixed1MiB HTTP/2 lookahead, disabled BDP growth/retries, bounded metadata/messages/IO are explicit; native gRPC/TLS/socket memory also needs measured RSS, not a claimed byte-perfect mapping into mailbox accounting. Public/multi-client DoS hardening is not proved by this private Worker experiment. A250ms event deadline covers the complete source event. Even final error-status sending is deadline-bound; otherwise an unread response can delay cleanup. Late trusted callbacks are quarantined, with no replacement grant or early byte release until they return. Pending reply drops are counted; they are not equivalent to end-to-end delivery receipts.

Evidence: initial default64KiB lookahead timed out on1,048,595B. Fixed1MiB passed two16/16 cross-host samples without changing250ms. Final min/median/max RTT 14.360/23.919/120.815ms on the Mac includes SSH/echo overhead, not clock-subtracted one-way age.2.202s wait retained the same CPU sentinel PID40 and lease; renewals 32→54. Source clock and resume evidence are explicitly synthetic, model count0. Peak mailbox6,750,559B; Linux RSS57,244KiB.20 Worker transport tests include1MiB output, a real slow reader and late-callback cleanup;105 focused Mac tests pass with the two larger probes intentionally skipped. Final cleanup leaves0bytes, no temporary listener/container/tunnel/private key/grant. No GPU setters or durable services changed. Frigate/Ollama remain disabled, Telegraf is Running. Manifest SHA256 adf5eaf092feaed6721f66e2adaceded0cdbf55754e1f9953f54623bfb52d331 at .runtime/perception-stage2-grpc-20260902T1500Z/manifest.json (16 artifacts).

Design references: gRPC flow control warns that a completed write need not mean network delivery; authentication documents TLS credentials; channel options documents the fixed lookahead window. Source and actual behavior were checked; these references alone are not a performance qualification.

Stage 2 increment 13: full graph over container networking (2026-09-02)

The same BinaryGraphInput decoder and causal sensor cut now feed the full graph from NetworkGraphBridge. Source and receiver execute in a separate CPU-only container. No camera/index/sensor recording mount exists in the GPU container. Resident DDRNet/RF-DETR/decoder/TGS children and the local Worker lease remain owned by the existing controller; remote ingress cannot restart models or renew the lease.

The return payload carries the existing scene plus the actual uint8 segmentation plane. Grid coordinates join the costmap digest. Result1MiB, JSON512KiB and plane 512KiB caps are checked before parsing/retention; total framing also fits1MiB. The receiver verifies raw mask and all layer hashes and independently derives freshness/policy. Network arrival, integrity-validation completion and complete consumer-view readiness are recorded separately.

The pilot verifies a common Linux boot and zero monotonic namespace offsets. Its grant file is trusted local control, read-only to the source; source status uses a separate mount. It is not the production grant delivery or cross-host clock mapping implementation. A foreign clock cannot be silently subtracted.

Final normal128b:128/128 byte-exact received scenes, raw model/geometry/motion/ costmap parity to the prior reference, no queue drops, one input epoch, no resets. Consumer-ready p95/p99/max106.150/115.349/126.910ms. One frame exceeds125ms;76/128 scenes have complete six-layer freshness, with52 original missing sensor pairs. Gap128b:2.2s disconnect,102 completed +26 omissions24–49, fresh keyframe/pair resume50, exactly one temporal reset, same four PIDs and lease generation1. Consumer-ready p95/p99/max107.098/110.817/117.317ms; availability gap2.725s. Post-gap history-dependent outputs are not asserted equal to uninterrupted history. All230 mask/scene receipts and their derived views were rechecked.

These runs use temporary stock2610/10251MHz clocks,450W,8CPU/8GiB model and 2CPU/1GiB source envelopes. They prove container-network behavior on one Worker, not Mac↔Worker/radio/live latency. Observed result maximum460248B, tracked peak13,217,791B, residual0. Full-model slow consumer and cross-host clock/grant integration precede standalone packaging; product cutover remains closed. Diagnostic startup retry is not a production recovery service.

Source init originally preceded the delayed start by2s and triggered a spurious idle timeout. The source now connects100ms before release, without relaxing the timeout. Final normal/gap have zero/one intended recovery. Missing runner, read-only mountpoint and launcher ExitCode failures are retained as rejected attempts.146 focused tests pass;2 larger Worker-only tests skipped on Mac.

Evidence236 artifacts at .runtime/perception-stage2-network-graph-20260902T1541Z/manifest.json, SHA256 d1bc7e26850a2d9050ee1d8f8a5ece2e6b8c53e2e7dba381cbd4445d68a5ec17. See the detailed report. Four prior services and automatic clocks restored; transient containers, networks, lease volumes, keys and grants removed. Ollama/Frigate stay disabled. Mac8000 and Worker telemetry remain available; no8765 or external deployment.

Stage 2 increment 14: cross-host clock/control foundation (2026-09-02)

Application-side TLS Poll may deliver only a pending data grant already issued by the trusted Worker controller. It cannot acquire/renew the local lease, reset/start models, choose an epoch or cancel. Data admission remains single-use and independently scoped. Four timestamps bound remote-minus-local offset without equal-path assumptions; recent intervals are intersected, aged under an explicit rate/error envelope and expire. Expired/excess/contradictory evidence means unavailable/WAIT, not a wider hidden budget.

CPU-only Mac↔Worker proof delivers both grants and8/8 exact echoes. A2.202s gap retains the CPU sentinel and lease1, expires the old mapping and resumes with a new input epoch. Of72 measured observations,60 meet conditional≤5ms; min/median/max uncertainty are 3.883/4.469/27.118ms and readiness changes after warmup. Consequently this increment does not claim full-graph cross-host freshness and does not cherry-pick a passing sample. The next full-graph run must consume continuous mapping state and carry uncertainty to the receiving freshness decision. Detailed report.

Stage 2 increment 15: acknowledged clocks in the full cross-host graph (2026-09-02)

The Worker validates a one-use echoed Poll challenge before accepting responder-side clock evidence and the immutable source anchor. Clock validity is checked continuously, including idle periods, input admission and GPU/CPU/publication boundaries. Unknown or excess uncertainty enters WAIT without releasing resident models/local ownership. Upper age bounds govern source cutoff; scene and cell freshness include uncertainty. Each assessment uses one immutable clock snapshot, and EOF acknowledgement precedes data-channel retirement. Early transport loss after grant admission but before OPEN must also enter WAIT; malformed unbound data cannot stop the owner.

The final full-graph Mac source/receiver canary is not qualified:13 source WAIT skips, 17 synchronization skips,2/32 receipts (frames30–31), both after source-window EOF. Latency143.881/155.743ms; no hidden losses, no model reload, sampled VRAM2365MiB, not a24GiB limit.297 local and173 Worker tests PASS; early disconnect additionally12/12 across six repeats. Source-first clock readiness and a fixed2s lead do not establish Worker readiness. Next gate separates two-sided clock warmup from source-anchor activation before1x starts; after start the source timeline remains immutable through outages. No gate widening, backlog replay, standalone or Stage3 promotion. Evidence and rejected attempts.

Stage 2 increment 16: joint startup before immutable 1x activation (2026-09-02)

Pre-start clock reports carry a null anchor and warm both observation windows. Version2 replies bind responder bounds and accepted anchor to the issued challenge. Source checks both current intervals, ages peer evidence through ACK transit, obtains explicit anchor acceptance and a data grant before startup. A lost ACK cannot create a different anchor; missing the agreed start fails rather than silently retiming input. The existing5ms/2s/500ppm/50us budgets and lease/data authority separation remain intact.

The single32-frame GPU canary now admits frame0, but is not realtime-qualified: 8 results,1 input-gap discard,9 WAIT and14 synchronization skips. All55 six-timestamp exchanges independently reconstruct; running uncertainty5.031–5.667ms causes a real clock WAIT without an RPC error, GPU telemetry loss or model restart. More samples inside the same2s horizon would not remove the measured WAIT.241 local/198 Worker tests PASS,129 measured source hashes match code48835a0. Next investigate control-loop scheduling/transport before another full-graph qualification, without budget widening. Detailed evidence.

Stage 2 increment 17: transport-specific clock readiness (2026-09-02)

Clock-only A/B/A separates route effects from scene processing: SSH/Tailscale48/96, strictly host-key-pinned SSH over confirmed LAN94/96 (only initial2 unready), return to Tailscale53/96. All288 exchanges reconstruct; application handler/loop delays do not alone explain ~100ms request tails. LAN is an observed connection option, not a new model dependency, universal speedup or permanent product/default route change.

One full LAN canary returns32/32 byte-exact/reference-exact scenes at1x with no WAIT, drop or reconnect;50 clock exchanges reconstruct,48 after initial warmup remain ready. All129 prior runtime source files and the model/effective config remain unchanged. Latency205.864/213.751ms p95/p99 still fails125ms; only12/32 scenes are fully fresh. Do not equate complete output accounting with usable geometry/policy or autonomous readiness. Next instrument data-plane stage boundaries before optimizing remaining tails; no gate widening.245 local/202 Worker tests PASS,131 source hashes matched. Detailed route and full-profile evidence.

Stage 2 increment 18: application-edge transport attribution (2026-09-02)

Optional bounded TLS-record witnesses belong to isolated diagnostic tooling, not the runtime route or model profile. A/B/A keeps the~100ms tail with and without the witnesses;424 ciphertext records match between edges and192 serial RPCs match their handlers.13 post-start long calls localize to request(8)/response(5) inter-edge transit, not local JSON/handler work. Startup without admitted clock mapping stays unqualified; timestamps are application receipt/drain, not wire ACKs.

Direct TCP connects to the known Worker SSH port reproduce67.9–91.1ms peaks without gRPC, Docker or SSH forwarding. Both active links are Wi-Fi; native Worker loopback after startup is0.240–0.299ms. The specific radio/AP/OS/driver mechanism is unproved. Do not weaken5ms/125ms gates or replace models to address an external transport symptom. Conversely, this does not exonerate all application costs or qualify large scene buffers. Continue per-frame queue/serialization/publication attribution; an available wired comparison should remove both radio legs but is not a prerequisite for experimental runtime/standalone work. No product transport or GPU workload changed. 82 local/82 Worker tests PASS;142 staged files exact; previous full-graph latency FAIL stands. Detailed evidence.