NODEDC_MISSION_CORE/docs/adr/0049-stream-first-perception-profiles.md

ADR 0049 — Stream-first full perception profiles

Date: 2026-09-01. Status: stage-1 contract candidate; not a runtime cutover. Execution order remains the four stages in the ExecPlan.

Decision and owner intent

Observatory LABs test the real-time behavior of a complete perception profile, not just the quality of an offline export. A recording replaces the physical source, preserving original release timing at 1×. It must not enable a different batch implementation. The first profile contains DDRNet-39 GOOSE, RF-DETR native, online LiDAR geometry/ranges, temporal/motion, CPU TRAVEL TGS/costmap and advisory policy. One dedicated RTX 4090 owns one active full profile.

The first rural prototype accepts existing coarse hard_surface, including asphalt, sidewalk, bikeway and cobble. Existing static_obstacle/static.unknown is sufficient. No new bollard classifier, extra segmenter, road/sidewalk taxonomy or annotation campaign precedes this prototype. Other material rules remain an explicit effective mission configuration; rural context alone is not permission for all vegetation or soil. Geometry/unknown/freshness overrides material allowance.

EoMT is preserved as a separate future profile. It is not loaded with DDRNet. RF-DETR and DDRNet are two required components of the same first profile: one supervisor and serialized GPU scheduling, not competing profile jobs.

CURRENT → TARGET / ownership

Boundary	Current evidence	Decision / stage
Portable definitions	portable_run_definitions.py owns pins, resources and admission	KEEP identity/sealing; EXTEND stream capabilities and qualification in stage 3
Source requirements	Recorded-only requirements include seekability, one media epoch and init digest	REPLACE as live admission criteria with channel/format/calibration/clock requirements; keep init digest as per-stream integrity
Job queue	recorded_jobs.py has durable claims but global single-active-job scope	KEEP durable queue; EXTEND worker-scoped fencing, not unbounded GPU concurrency
Worker agent	Blocking execute(job) and final publication	EXTEND incremental lifecycle and cancellation; no per-LAB agent
Runtime	portable_worker_runtime.py:735 materializes before execution	NEW stream runtime in stage 2; never call this materializer in the real-time path
Source transport	Whole camera archive download/hash/extraction before use	REPLACE startup barrier with bounded metadata, chunks and incremental integrity
Installed LAB V1	prepare → complete EoMT → complete DDRNet → assemble	Legacy evidence only; not the new profile and not real-time qualified
Camera/source replay	recorded_source.py has useful pacing and bounded prefetch, but source-specific timeline and paths	REUSE mechanisms; generic unknown-duration source adapter, no route-sized prefetch
Existing live ingress	LivePerceptionIngress, LiveIngressEvent.wire_bytes, LiveSensorSynchronizer already provide raw-first modality queues, hashes, epochs and bounded binding	REUSE, not a second acquisition system; extend worker/profile fencing and aggregate byte limits; map existing envelopes to the selected transport
Local surface	RecordedGeometryStore reads prepared NPZ	REUSE K1LocalSurfaceShadowEstimator.process and lidar_local_surface_geometry; feed current points/pose
Geometry/ranges	Current point association, exclusive ownership and geometry-only clusters exist	REUSE algorithms; remove recorded-store dependency and expose estimator/frame identity
Temporal/motion	BoundedSpatialTemporalProvider, ClassIndependentMotionEstimator	REUSE with current-run inputs and reset on epoch/lease change
TGS	Core algorithm reusable; old runners read a full schedule, some require exactly 4489 frames	REUSE TGS parameters/core; REPLACE schedule/file CLI with incremental input
Fusion	M49 TGS can be a neighboring shadow with tgs_modifies_reference_graph_state=false	TGS and semantics must influence the current advisory scene, not only viewer composition
Publication	Verified immutable result/publication/recovery exist	KEEP asynchronous final sealing; live results do not await it
Frontend	Setup/job/result surfaces exist, historical M49 viewer links other results	EXTEND common renderer capabilities with current-run layer references and freshness in stage 3
Recording archive	Source identities and raw evidence must be retained	KEEP; create metadata/index/integrity during acquisition or incrementally, not via preflight full scan

No changes to the active registry, old queue, source endpoint, UI or canonical local server are made by this ADR or the stage-1 validators.

Versioned handshake and observations

missioncore.perception-stream-start/v1 is represented by StreamStart in src/k1link/perception/realtime_contract.py. It contains run/source/worker/epoch, lease generation, profile/image/effective-config/calibration identities, clock domain, input mode and channel declarations. The first profile requires camera, point-cloud and pose. Temporary absence of a sample is an explicit modality outcome, not removal of a required capability.

It deliberately has no total source bytes, frame count, total duration, whole archive, command, environment, complete member inventory or required EOF. The initial metadata budget is 64 KiB. Large constant model assets and FOV masks belong inside the image; calibration references resolve to bounded metadata.

The stage-2 wire mapping must carry:

• Run/epoch/lease binding on every message; source/channel sequence, capture timestamp and clock domain; independently verifiable payload length/hash.
• Camera codec initialization and then incremental encoded access units, or bounded fragments thereof. Native decoded representation is BGR8 800×600 KB4. Do not send an entire fMP4 epoch as one observation or turn all frames into PNGs.
• Current vendor-registered map point increments with point IDs, coordinate frame, units, scalar layout and retained raw attributes; map→sensor pose and calibration identity. The adapter owns vendor decoding; the profile is not a new K1 protocol implementation and cannot command the scanner.
• Explicit unavailable/gap/end/cancel messages. End is a termination event, never a prerequisite for first inference. Final recording inventory/hash can be sealed after consumption; per-chunk integrity is verified before use.
• Integer nanoseconds use protobuf integer fields; any JSON projection uses decimal strings for 64-bit times rather than lossy JavaScript numbers.

Maximum payload fragment: 1 MiB; total application inflight bytes: 16 MiB; pending decoded camera frames: 2. Fragment reassembly is included in that budget, with a deadline and incomplete-chunk accounting. Decoded image/tensor, cloud, costmap and archive sinks need their own measured bounds inside the RSS budget. Transport flow control alone is not a memory bound or a freshness policy.

gRPC bidirectional streaming is the first transport candidate, not an already measured implementation. HTTP control-plane operations remain. The browser does not connect directly to model RPCs. One authenticated data-plane endpoint feeds the supervised profile; model processes have no arbitrary external network. Protocol/codec selection must be measured on actual payload sizes in stage 2; changing an archive POST to gRPC without changing execution is not acceptance. Existing missioncore.live-perception-wire/v1 and result wire v2 are migration inputs: reuse their semantics and raw-first producer hooks. Their modality count caps alone permit more than the proposed 16 MiB aggregate inflight budget; an explicit byte bound is required. No duplicate new scanner feed is introduced.

Time and lifecycle

Open → validate bounded metadata/claim → warm models → Ready → start the replay clock → observations and incremental scenes → stop/drain bounded state → close and asynchronously seal the immutable receipt. A live source already producing data does not queue its entire warmup history: readiness begins at a current decodable keyframe, with the skipped interval recorded.

Replay release uses original source intervals, not a convenient fixed 12 Hz and not the processing speed. A slow consumer must not stretch source time to make inference look real-time. Overload may drop bounded work to preserve liveness, but the dropped observations remain in the ledger and fail the strict initial performance target. Initial baseline is every camera frame, stride 1. Choosing multirate later changes the explicit profile contract; a retained mask is not a new inference and cannot receive a fresh capture timestamp.

UTC identifies the run; monotonic clocks measure local durations. Cross-host age requires a clock mapping and its uncertainty. No direct subtraction of unrelated host monotonic clocks. The common source timeline is separate from worker wall-clock time. Causality is bounded by the released-observation watermark at decision time, not by whatever later samples exist in the file. Each layer retains its own channel/sequence/time; camera anchors and pose/cloud association times remain separately visible. Future file look-ahead and offline nearest-pose lookup across unreleased data are forbidden.

Lease expiry, cancellation and epoch changes fence the previous producer. No new profile may take the GPU until old processes and work have stopped. A reconnect creates a new epoch, discards stale backlog, resets temporal/rolling state and reacquires codec initialization/keyframe. It never resends a stale motor action; this profile has no motor actions at all.

Outputs and interpretation

Every scene accounts for segmentation, objects, geometry, motion, costmap and policy exactly once, with current/held/stale/unavailable state and current-run payload identity. Existing source/object/obstacle/temporal/map/threat contracts remain the domain vocabulary. LayerEvidence adds freshness/completeness checks, not a second semantic ontology. In stage 2, layer manifests also carry their full input reference sets; a single anchor does not replace camera/cloud/pose lineage.

Stage-1 implementation update, 2026-09-02 (097e450): missioncore.perception-scene-freshness/v1 in realtime_scene.py extends LayerEvidence with oldest-required-input time and checks the six-layer dependency graph. A derived result cannot refresh its input age. Decimal int64 JSON strings preserve timestamps/sequence exactly; transport-neutral Python objects use integers. The pilot validates payload hashes and assesses freshness again at receipt, and must reassess at later use. Missing/stale output remains inspectable but cannot retain permissive advisory policy. This envelope is NOT a replacement for StreamStart identities, complete input lineage or controller lease fencing.

The current pilot propagates the oldest last-seen permissive TGS cell into the costmap/policy age. Expiry currently suppresses the entire scene's permissions; stage 2 must implement explicit per-cell expiration without granting authority from retained occupied/unknown history. Existing model/geometry/motion/TGS/material outputs remain identical to the pinned reference on the measured 128-frame window. Freshness/completeness is not semantic correctness or physical safety.

worker_operating_envelope.py checks a trusted post-warmup snapshot against preregistered hardware/driver/resource/clock conditions and StreamStart identities, ownership and client inventory. Unknown or expired facts fail readiness. The candidate records the previously measured 4090 fixed-clock envelope; a compatible auto-clock experiment cannot inherit its latency result. The evaluator performs no I/O, host clock changes, lease acquisition or qualification; collection and continuous enforcement belong to the common controller/runtime in stage 2.

DDRNet currently crops the center 600×600 to 512×512; outside that ROI is undefined, not hard_surface/free. RF-DETR accepts the full 800×600 native raster with its pinned valid-FOV handling. Their coordinates must be related explicitly.

RF-DETR's existing filter emits person/cat/dog, but minimum box area 64 px, maximum box fraction 0.5, FOV fraction 0.5 and required valid center create known small-animal/close-large-object edge cases. They are retained and documented, not silently retuned or presented as proven field quality.

Detected-object range is the median camera-Z of owned current support points; geometry-only range is nearest Euclidean distance from the sensor. Neither is automatically clearance from the physical vehicle body. Missing/ambiguous support produces unavailable range, not zero/infinity. Generic static obstacles survive absence of a detector class name.

TGS keeps ground/occupied/rejected/unobserved separate. Ground support alone is not a traversability or actuation decision. Policy can produce an advisory allowed-candidate/high-cost/blocked/unknown outcome with explanation. No planner, physical motor command, autonomous driving acceptance or mission configurator implementation is introduced.

Package/dependency decision

The candidate manifest is config/perception/k1-perception-ddrnet39-rfdetr-tgs-prototype-v1.json. It is not installed into the active portable registry and has no invented image digest. Model/checkpoint/config identities are pinned; source session IDs and old derived output paths in reference experiments are not inherited requirements.

Select process/environment isolation inside one future image: Python 3.12 supervisor and geometry, RF-DETR TensorRT 11 runtime, existing Python 3.9 / Torch 1.13.1 cu117 / super-gradients 3.2.0 environment, and CPU C++ TGS. Exchange bounded shared-memory/IPC payload references; one supervisor serializes GPU work. This avoids a forced dependency upgrade or checkpoint conversion in stage 1. Image assembly of the final standalone package and the full schedule pilot remain explicit implementation evidence. A temporary common-base image passed separate DDRNet, TensorRT, Python 3.12 geometry and C++ TGS execution probes; DDRNet masks matched on 64 frames. This proves bounded ABI/execution compatibility in that image, not jointly resident models, the supervisor/IPC or a complete profile. The probes still mount pinned assets explicitly; they are not standalone proof.

The temporary Triton base lacks Python grpc/protobuf, cv2 and TensorRT bindings; native trtexec/server are present. The actual supervisor/transport environment must explicitly include and validate its dependencies. Numeric-library thread limits are pinned to one after the bounded synthetic local-surface comparison (mean 198.0 ms default versus 63.2 ms with limits); algorithm thresholds were not changed. Real-cloud timing, output parity and whole-graph scheduling remain gates.

The initial DDRNet image-only probe failed before inference: the installed image does not contain the checkpoint at its logical asset path. The bounded baseline therefore uses an explicit read-only pinned checkpoint and runner mount. This is valid component measurement, but fails the target's standalone packaging claim. Super-gradients also needs a writable log directory; a bounded temporary log mount was required. The new image must declare scratch areas explicitly without requiring a writable host home, code checkout or implicit weight cache.

Preregistered engineering acceptance

Owner clarification, 2026-09-02 MSK: bounded laboratory overload is an admissible experimental outcome. Keep useful profiles that fail the current 4090 target; do not block reusable runtime/packaging work solely on that performance result. Functional execution, experimental availability, quality and real-time qualification are separate dimensions. Qualification belongs to a measured profile/config/hardware/source/transport combination, not permanently to a model name. More powerful Worker/onboard placement is a future retest target, not a claim that existing latency or internal IPC disappears automatically. The original budgets and failed measurements below remain unchanged. No slowed source clock, hidden drops, stale-as-current evidence or actuation is authorized.

The manifest freezes an initial engineering candidate: p95/p99 full output age ≤125 ms, required-layer age ≤250 ms, release lag ≤25 ms, first incremental result ≤1 s after source admission, warmup ≤120 s, stop ≤5 s, VRAM ≤22000 MiB, RSS ≤8192 MiB. First/last-window backlog growth ≤25 ms and no capacity drops, failures, expired selected frames or unaccounted observations. These are prototype targets, not physical braking/safety limits or claims of achieved performance.

ReplayMeasurements/realtime_failures reject full-source preload, EOF-only results, slowed replay, incomplete accounting, hidden missing layers and budget violations. Declared source gaps must be verified from the input ledger; they produce explicit degraded scenes and are not counted as fully fresh perception. At least one fully evaluable scene is required. A component FPS figure cannot populate this whole-path receipt. A quality comparison or successful export is not a substitute for the real-time gate.

Every observation receives a terminal ledger outcome. Track ingress/release, selection, decode, inference, completion, emission, drop/expiry/failure and source gaps separately. Store startup separately from steady state; record payload bytes, queue high-water marks, clock error, RSS/VRAM, image/weights/config/source/equipment identities. Export and UI encoding cannot block inference through an unlimited queue. A receiver-side render/receipt timestamp, not just Worker completion, ends the end-to-end interval.

Evidence and remaining stage-1 work

See experiments/perception/PERCEPTION_STREAM_STAGE1_2026-09-01.md for measured component results, bounded validation, Worker maintenance and remaining gates. Neither this ADR nor passing synthetic contract tests marks the new runtime ready.