NODEDC_MISSION_CORE/docs/adr/0049-stream-first-perception-profiles.md

ADR 0049 — Stream-first full perception profiles

Date: 2026-09-01. Status: stage-1 contract candidate; not a runtime cutover. Execution order remains the four stages in the ExecPlan.

Decision and owner intent

Observatory LABs test the real-time behavior of a complete perception profile, not just the quality of an offline export. A recording replaces the physical source, preserving original release timing at 1×. It must not enable a different batch implementation. The first profile contains DDRNet-39 GOOSE, RF-DETR native, online LiDAR geometry/ranges, temporal/motion, CPU TRAVEL TGS/costmap and advisory policy. One dedicated RTX 4090 owns one active full profile.

The first rural prototype accepts existing coarse hard_surface, including asphalt, sidewalk, bikeway and cobble. Existing static_obstacle/static.unknown is sufficient. No new bollard classifier, extra segmenter, road/sidewalk taxonomy or annotation campaign precedes this prototype. Other material rules remain an explicit effective mission configuration; rural context alone is not permission for all vegetation or soil. Geometry/unknown/freshness overrides material allowance.

EoMT is preserved as a separate future profile. It is not loaded with DDRNet. RF-DETR and DDRNet are two required components of the same first profile: one supervisor and serialized GPU scheduling, not competing profile jobs.

CURRENT → TARGET / ownership

Boundary	Current evidence	Decision / stage
Portable definitions	portable_run_definitions.py owns pins, resources and admission	KEEP identity/sealing; EXTEND stream capabilities and qualification in stage 3
Source requirements	Recorded-only requirements include seekability, one media epoch and init digest	REPLACE as live admission criteria with channel/format/calibration/clock requirements; keep init digest as per-stream integrity
Job queue	recorded_jobs.py has durable claims but global single-active-job scope	KEEP durable queue; EXTEND worker-scoped fencing, not unbounded GPU concurrency
Worker agent	Blocking execute(job) and final publication	EXTEND incremental lifecycle and cancellation; no per-LAB agent
Runtime	portable_worker_runtime.py:735 materializes before execution	NEW stream runtime in stage 2; never call this materializer in the real-time path
Source transport	Whole camera archive download/hash/extraction before use	REPLACE startup barrier with bounded metadata, chunks and incremental integrity
Installed LAB V1	prepare → complete EoMT → complete DDRNet → assemble	Legacy evidence only; not the new profile and not real-time qualified
Camera/source replay	recorded_source.py has useful pacing and bounded prefetch, but source-specific timeline and paths	REUSE mechanisms; generic unknown-duration source adapter, no route-sized prefetch
Existing live ingress	LivePerceptionIngress, LiveIngressEvent.wire_bytes, LiveSensorSynchronizer already provide raw-first modality queues, hashes, epochs and bounded binding	REUSE, not a second acquisition system; extend worker/profile fencing and aggregate byte limits; map existing envelopes to the selected transport
Local surface	RecordedGeometryStore reads prepared NPZ	REUSE K1LocalSurfaceShadowEstimator.process and lidar_local_surface_geometry; feed current points/pose
Geometry/ranges	Current point association, exclusive ownership and geometry-only clusters exist	REUSE algorithms; remove recorded-store dependency and expose estimator/frame identity
Temporal/motion	BoundedSpatialTemporalProvider, ClassIndependentMotionEstimator	REUSE with current-run inputs and reset on epoch/lease change
TGS	Core algorithm reusable; old runners read a full schedule, some require exactly 4489 frames	REUSE TGS parameters/core; REPLACE schedule/file CLI with incremental input
Fusion	M49 TGS can be a neighboring shadow with tgs_modifies_reference_graph_state=false	TGS and semantics must influence the current advisory scene, not only viewer composition
Publication	Verified immutable result/publication/recovery exist	KEEP asynchronous final sealing; live results do not await it
Frontend	Setup/job/result surfaces exist, historical M49 viewer links other results	EXTEND common renderer capabilities with current-run layer references and freshness in stage 3
Recording archive	Source identities and raw evidence must be retained	KEEP; create metadata/index/integrity during acquisition or incrementally, not via preflight full scan

No changes to the active registry, old queue, source endpoint, UI or canonical local server are made by this ADR or the stage-1 validators.

Versioned handshake and observations

missioncore.perception-stream-start/v1 is represented by StreamStart in src/k1link/perception/realtime_contract.py. It contains run/source/worker/epoch, lease generation, profile/image/effective-config/calibration identities, clock domain, input mode and channel declarations. The first profile requires camera, point-cloud and pose. Temporary absence of a sample is an explicit modality outcome, not removal of a required capability.

It deliberately has no total source bytes, frame count, total duration, whole archive, command, environment, complete member inventory or required EOF. The initial metadata budget is 64 KiB. Large constant model assets and FOV masks belong inside the image; calibration references resolve to bounded metadata.

The stage-2 wire mapping must carry:

• Run/epoch/lease binding on every message; source/channel sequence, capture timestamp and clock domain; independently verifiable payload length/hash.
• Camera codec initialization and then incremental encoded access units, or bounded fragments thereof. Native decoded representation is BGR8 800×600 KB4. Do not send an entire fMP4 epoch as one observation or turn all frames into PNGs.
• Current vendor-registered map point increments with point IDs, coordinate frame, units, scalar layout and retained raw attributes; map→sensor pose and calibration identity. The adapter owns vendor decoding; the profile is not a new K1 protocol implementation and cannot command the scanner.
• Explicit unavailable/gap/end/cancel messages. End is a termination event, never a prerequisite for first inference. Final recording inventory/hash can be sealed after consumption; per-chunk integrity is verified before use.
• Integer nanoseconds use protobuf integer fields; any JSON projection uses decimal strings for 64-bit times rather than lossy JavaScript numbers.

Maximum payload fragment: 1 MiB; total application inflight bytes: 16 MiB; pending decoded camera frames: 2. Fragment reassembly is included in that budget, with a deadline and incomplete-chunk accounting. Decoded image/tensor, cloud, costmap and archive sinks need their own measured bounds inside the RSS budget. Transport flow control alone is not a memory bound or a freshness policy.

gRPC bidirectional streaming is the first transport candidate, not an already measured implementation. HTTP control-plane operations remain. The browser does not connect directly to model RPCs. One authenticated data-plane endpoint feeds the supervised profile; model processes have no arbitrary external network. Protocol/codec selection must be measured on actual payload sizes in stage 2; changing an archive POST to gRPC without changing execution is not acceptance. Existing missioncore.live-perception-wire/v1 and result wire v2 are migration inputs: reuse their semantics and raw-first producer hooks. Their modality count caps alone permit more than the proposed 16 MiB aggregate inflight budget; an explicit byte bound is required. No duplicate new scanner feed is introduced.

Time and lifecycle

Stage-2 raw IPC increment, 2026-09-02 (da60fef, a937400, 79ce55d): streaming_wire.py maps existing LiveIngressEvent fields into missioncore.live-perception-wire/v2. The v1 producer hooks, class and legacy serialization remain unchanged. A bounded length-prefixed JSON header carries decimal uint64 projections; payloads remain binary. This is a controller-supplied IPC socket candidate, not a selection/replacement of the intended gRPC network transport. No listener, arbitrary endpoint, scanner feed or command path is added.

Open compares the entire expected StreamStart and acquisition session binding; every fragment/control carries a digest of that full binding. A hash is integrity, not authentication. Fragment size is at most 1 MiB, event caps reuse the existing 1 MiB camera / 2 MiB LiDAR/pose limits. Headers are capped at 64 KiB. One observation assembles at a time with exact offsets, fragment and whole-event SHA-256, per-channel source identity/sequence/clock checks. End, Cancel and gap/unavailable notices are explicit. EOF without End, timeout, malformed, cross-binding or incomplete input fails the bound stream; an unbound wrong Open is rejected without stopping the current owner. Camera frames require init; camera gap invalidates init. Actual codec/keyframe/IDR validation is NOT yet here.

StreamingSender owns no queue and fails on its bounded write deadline rather than stretching replay time. StreamingIngress is a lifecycle-tracked thread; idle reads poll the lease and cannot keep accepting an expired owner. Raw/header and reassembly reservations charge the SAME StreamMailbox 16 MiB input budget as queued/active bundles. They survive cancellation until the borrowing callback returns and buffers are dropped. Reservations are count-bounded too. Trusted decoder adapters must reserve scratch BEFORE allocation and must not retain raw callback bytes outside that ownership contract. Python metadata object overhead, socket/kernel buffers, source adapter memory and model tensors additionally need RSS/OS resource bounds; this application counter alone is not a total-RSS claim.

Worker evidence uses a separate source process, original camera init/fMP4 segments and existing normalized point/pose increments paced at original 1x times. All 103 events (32 camera + init + 34 LiDAR + 36 pose) match exact bytes/metadata, each data modality reaches the hash-only consumer before End, and incomplete EOF delivers no observation. Normal peak input is 553,991 bytes. This validates raw IPC, NOT decoder output, the full graph on this ingress, model performance, authenticated network behavior or a standalone image. The next adapter must feed incremental decode and causal sensors into the same graph without using the old concatenated camera file or precomputed geometry. Product source queues, backend routes and the browser are not switched by this increment.

Open → validate bounded metadata/claim → warm models → Ready → start the replay clock → observations and incremental scenes → stop/drain bounded state → close and asynchronously seal the immutable receipt. A live source already producing data does not queue its entire warmup history: readiness begins at a current decodable keyframe, with the skipped interval recorded.

Replay release uses original source intervals, not a convenient fixed 12 Hz and not the processing speed. A slow consumer must not stretch source time to make inference look real-time. Overload may drop bounded work to preserve liveness, but the dropped observations remain in the ledger and fail the strict initial performance target. Initial baseline is every camera frame, stride 1. Choosing multirate later changes the explicit profile contract; a retained mask is not a new inference and cannot receive a fresh capture timestamp.

UTC identifies the run; monotonic clocks measure local durations. Cross-host age requires a clock mapping and its uncertainty. No direct subtraction of unrelated host monotonic clocks. The common source timeline is separate from worker wall-clock time. Causality is bounded by the released-observation watermark at decision time, not by whatever later samples exist in the file. Each layer retains its own channel/sequence/time; camera anchors and pose/cloud association times remain separately visible. Future file look-ahead and offline nearest-pose lookup across unreleased data are forbidden.

Lease expiry, cancellation and epoch changes fence the previous producer. No new profile may take the GPU until old processes and work have stopped. A reconnect creates a new epoch, discards stale backlog, resets temporal/rolling state and reacquires codec initialization/keyframe. It never resends a stale motor action; this profile has no motor actions at all.

Outputs and interpretation

Every scene accounts for segmentation, objects, geometry, motion, costmap and policy exactly once, with current/held/stale/unavailable state and current-run payload identity. Existing source/object/obstacle/temporal/map/threat contracts remain the domain vocabulary. LayerEvidence adds freshness/completeness checks, not a second semantic ontology. In stage 2, layer manifests also carry their full input reference sets; a single anchor does not replace camera/cloud/pose lineage.

Stage-1 implementation update, 2026-09-02 (097e450): missioncore.perception-scene-freshness/v1 in realtime_scene.py extends LayerEvidence with oldest-required-input time and checks the six-layer dependency graph. A derived result cannot refresh its input age. Decimal int64 JSON strings preserve timestamps/sequence exactly; transport-neutral Python objects use integers. The pilot validates payload hashes and assesses freshness again at receipt, and must reassess at later use. Missing/stale output remains inspectable but cannot retain permissive advisory policy. This envelope is NOT a replacement for StreamStart identities, complete input lineage or controller lease fencing.

The stage-1 pilot propagated the oldest last-seen permissive TGS cell into the costmap/policy age and suppressed the entire scene on expiry. Stage-2 increment 1f8101e adds missioncore.costmap-cell-freshness/v1: bounded (8192 cells), index-aligned original support timestamps, decimal int64 strings/null, included in the costmap payload digest together with the guard mode. Unobserved support is never assigned a timestamp. The pilot now defaults to per-cell; the old whole-scene mode remains an explicit comparison control, not a product cutover.

Both publication and receipt remove permission from expired ground cells and mark them rejected/NO_GO. Occupied cells retain prohibition. The remaining permissions propagate their original support age AND the mandatory segmentation, geometry and motion dependencies. A missing/stale mandatory layer still blocks the whole policy. A derived consumer view has new costmap/policy hashes while preserving source identity/timestamps; the original published bytes stay immutable. Reassessment cannot restore a suppressed action or move the observer clock back. The enclosing StreamStart must bind the grid/profile/epoch/clock; this descriptor alone does not establish those identities or authenticate a remote producer.

In the measured 128-frame window, raw model/geometry/motion/TGS/material outputs remain identical to pinned PyTorch. Effective costmap states and permissions change deliberately on expiry. A fresh six-layer envelope can still contain rejected/unknown cells; it is not a claim of full free-space coverage, semantic correctness, vehicle clearance or physical safety. Recheck again at actual use.

bcacb02 promotes the proven mailbox and serial GPU stage to common perception modules, with the pilot importing compatibility aliases rather than owning another scheduler. Pending ingress + completed GPU outputs share two slots; active inputs share 16 MiB and are released by their owning stage. Cancellation discards pending work, a timed-out callback retains ownership, and CPU-owned input is not freed by GPU shutdown. Diagnostic drop history is capped at 256 entries with exact per-reason totals; it is not the durable terminal ledger. These primitives are NOT the controller supervisor, a cross-process GPU lease, StreamStart fencing or the binary data plane. Those stage-2 boundaries remain open.

Stage-2 lifecycle increment, 2026-09-02 (6acf468, ac69e3b): StreamingLifecycle now supplies the subprocess-backed profile supervisor using the existing GraphState and full StreamStart identity. WorkerLease uses a stable POSIX file lock on one controller-selected, private directory per Worker, plus an atomic/fsynced ownership record. All managed profile containers must use that SAME directory/volume. Neither the directory nor child commands come from an incoming source/job. This cooperative local fence supplements, not replaces, the backend claim and the trusted inventory of unmanaged GPU clients.

Open acquires ownership before process spawn/warmup; Ready enables admission. Renewal cannot resurrect expired ownership. Admission, each GPU/CPU lane and result publication/receipt validate run/source/worker/epoch/generation and pinned image/profile/config/calibration/clock identities. A mismatched client is rejected without cancelling the current owner. Each compute lane remains single-owner. The watchdog fences an expired owner even without a new frame and terminates only its dedicated child process groups. Cleanup keeps ownership while callbacks, tracked threads, pending/active payloads or child groups remain. Clean retirement persists released before unlocking; a higher generation and new epoch may then activate. A controller crash leaves active even when the kernel unlocks, so the next owner stays quarantined. No force/unverified recovery API exists.

Cross-container contention/clean succession/crash quarantine and full-profile lease expiry were exercised on Worker 006. The final timed expiry trace had no receipt at/after the lease deadline, stop requested after 45.74 ms and retirement after 4109.82 ms. This is bounded evidence, not a hard-real-time scheduler guarantee. The holder has a 2 s lease and receives local pilot heartbeats every 250 ms; those settings and the 50 ms watchdog poll are NOT network/vehicle safety limits.

Current integration remains explicitly diagnostic: pilot_lifecycle.py acts as a local controller, legacy GPU services are quiesced by the authorized launcher, the named volume is an isolated test volume, and the image still has code/model mounts. Mounted-code hashes remain in the experiment manifest; an image digest alone cannot seal those overrides. Backend claims/production-wide canonical root, trusted post-crash resource-release recovery, continuous GPU inventory/envelope enforcement, network authentication/heartbeat and binary live ingress are NOT installed or qualified by this increment. Standalone packaging must remove the developer mounts and retain the same lifecycle boundaries.

worker_operating_envelope.py checks a trusted post-warmup snapshot against preregistered hardware/driver/resource/clock conditions and StreamStart identities, ownership and client inventory. Unknown or expired facts fail readiness. The candidate records the previously measured 4090 fixed-clock envelope; a compatible auto-clock experiment cannot inherit its latency result. The evaluator performs no I/O, host clock changes, lease acquisition or qualification; collection and continuous enforcement belong to the common controller/runtime in stage 2.

DDRNet currently crops the center 600×600 to 512×512; outside that ROI is undefined, not hard_surface/free. RF-DETR accepts the full 800×600 native raster with its pinned valid-FOV handling. Their coordinates must be related explicitly.

RF-DETR's existing filter emits person/cat/dog, but minimum box area 64 px, maximum box fraction 0.5, FOV fraction 0.5 and required valid center create known small-animal/close-large-object edge cases. They are retained and documented, not silently retuned or presented as proven field quality.

Detected-object range is the median camera-Z of owned current support points; geometry-only range is nearest Euclidean distance from the sensor. Neither is automatically clearance from the physical vehicle body. Missing/ambiguous support produces unavailable range, not zero/infinity. Generic static obstacles survive absence of a detector class name.

TGS keeps ground/occupied/rejected/unobserved separate. Ground support alone is not a traversability or actuation decision. Policy can produce an advisory allowed-candidate/high-cost/blocked/unknown outcome with explanation. No planner, physical motor command, autonomous driving acceptance or mission configurator implementation is introduced.

Package/dependency decision

The candidate manifest is config/perception/k1-perception-ddrnet39-rfdetr-tgs-prototype-v1.json. It is not installed into the active portable registry and has no invented image digest. Model/checkpoint/config identities are pinned; source session IDs and old derived output paths in reference experiments are not inherited requirements.

Select process/environment isolation inside one future image: Python 3.12 supervisor and geometry, RF-DETR TensorRT 11 runtime, existing Python 3.9 / Torch 1.13.1 cu117 / super-gradients 3.2.0 environment, and CPU C++ TGS. Exchange bounded shared-memory/IPC payload references; one supervisor serializes GPU work. This avoids a forced dependency upgrade or checkpoint conversion in stage 1. Image assembly of the final standalone package and the full schedule pilot remain explicit implementation evidence. A temporary common-base image passed separate DDRNet, TensorRT, Python 3.12 geometry and C++ TGS execution probes; DDRNet masks matched on 64 frames. This proves bounded ABI/execution compatibility in that image, not jointly resident models, the supervisor/IPC or a complete profile. The probes still mount pinned assets explicitly; they are not standalone proof.

The temporary Triton base lacks Python grpc/protobuf, cv2 and TensorRT bindings; native trtexec/server are present. The actual supervisor/transport environment must explicitly include and validate its dependencies. Numeric-library thread limits are pinned to one after the bounded synthetic local-surface comparison (mean 198.0 ms default versus 63.2 ms with limits); algorithm thresholds were not changed. Real-cloud timing, output parity and whole-graph scheduling remain gates.

The initial DDRNet image-only probe failed before inference: the installed image does not contain the checkpoint at its logical asset path. The bounded baseline therefore uses an explicit read-only pinned checkpoint and runner mount. This is valid component measurement, but fails the target's standalone packaging claim. Super-gradients also needs a writable log directory; a bounded temporary log mount was required. The new image must declare scratch areas explicitly without requiring a writable host home, code checkout or implicit weight cache.

Preregistered engineering acceptance

Owner clarification, 2026-09-02 MSK: bounded laboratory overload is an admissible experimental outcome. Keep useful profiles that fail the current 4090 target; do not block reusable runtime/packaging work solely on that performance result. Functional execution, experimental availability, quality and real-time qualification are separate dimensions. Qualification belongs to a measured profile/config/hardware/source/transport combination, not permanently to a model name. More powerful Worker/onboard placement is a future retest target, not a claim that existing latency or internal IPC disappears automatically. The original budgets and failed measurements below remain unchanged. No slowed source clock, hidden drops, stale-as-current evidence or actuation is authorized.

The manifest freezes an initial engineering candidate: p95/p99 full output age ≤125 ms, required-layer age ≤250 ms, release lag ≤25 ms, first incremental result ≤1 s after source admission, warmup ≤120 s, stop ≤5 s, VRAM ≤22000 MiB, RSS ≤8192 MiB. First/last-window backlog growth ≤25 ms and no capacity drops, failures, expired selected frames or unaccounted observations. These are prototype targets, not physical braking/safety limits or claims of achieved performance.

ReplayMeasurements/realtime_failures reject full-source preload, EOF-only results, slowed replay, incomplete accounting, hidden missing layers and budget violations. Declared source gaps must be verified from the input ledger; they produce explicit degraded scenes and are not counted as fully fresh perception. At least one fully evaluable scene is required. A component FPS figure cannot populate this whole-path receipt. A quality comparison or successful export is not a substitute for the real-time gate.

Every observation receives a terminal ledger outcome. Track ingress/release, selection, decode, inference, completion, emission, drop/expiry/failure and source gaps separately. Store startup separately from steady state; record payload bytes, queue high-water marks, clock error, RSS/VRAM, image/weights/config/source/equipment identities. Export and UI encoding cannot block inference through an unlimited queue. A receiver-side render/receipt timestamp, not just Worker completion, ends the end-to-end interval.

Evidence and remaining stage-1 work

See experiments/perception/PERCEPTION_STREAM_STAGE1_2026-09-01.md for measured component results, bounded validation, Worker maintenance and remaining gates. Neither this ADR nor passing synthetic contract tests marks the new runtime ready.