NODEDC_MISSION_CORE/docs/adr/0049-stream-first-perception-profiles.md

# ADR 0049 — Stream-first full perception profiles

Date: 2026-09-01. Status: stage-1 contract candidate; not a runtime cutover.
Execution order remains the four stages in
[the ExecPlan](../OBSERVATORY_REALTIME_PROFILES_EXECPLAN.md).

## Decision and owner intent

Observatory LABs test the real-time behavior of a complete perception profile,
not just the quality of an offline export. A recording replaces the physical
source, preserving original release timing at 1×. It must not enable a different
batch implementation. The first profile contains DDRNet-39 GOOSE, RF-DETR native,
online LiDAR geometry/ranges, temporal/motion, CPU TRAVEL TGS/costmap and advisory
policy. One dedicated RTX 4090 owns one active full profile.

The first rural prototype accepts existing coarse `hard_surface`, including
asphalt, sidewalk, bikeway and cobble. Existing `static_obstacle`/`static.unknown`
is sufficient. No new bollard classifier, extra segmenter, road/sidewalk taxonomy
or annotation campaign precedes this prototype. Other material rules remain an
explicit effective mission configuration; rural context alone is not permission
for all vegetation or soil. Geometry/unknown/freshness overrides material allowance.

EoMT is preserved as a separate future profile. It is not loaded with DDRNet.
RF-DETR and DDRNet are two required components of the same first profile: one
supervisor and serialized GPU scheduling, not competing profile jobs.

## CURRENT → TARGET / ownership

| Boundary | Current evidence | Decision / stage |
| --- | --- | --- |
| Portable definitions | `portable_run_definitions.py` owns pins, resources and admission | KEEP identity/sealing; EXTEND stream capabilities and qualification in stage 3 |
| Source requirements | Recorded-only requirements include seekability, one media epoch and init digest | REPLACE as live admission criteria with channel/format/calibration/clock requirements; keep init digest as per-stream integrity |
| Job queue | `recorded_jobs.py` has durable claims but global single-active-job scope | KEEP durable queue; EXTEND worker-scoped fencing, not unbounded GPU concurrency |
| Worker agent | Blocking `execute(job)` and final publication | EXTEND incremental lifecycle and cancellation; no per-LAB agent |
| Runtime | `portable_worker_runtime.py:735` materializes before execution | NEW stream runtime in stage 2; never call this materializer in the real-time path |
| Source transport | Whole camera archive download/hash/extraction before use | REPLACE startup barrier with bounded metadata, chunks and incremental integrity |
| Installed LAB V1 | prepare → complete EoMT → complete DDRNet → assemble | Legacy evidence only; not the new profile and not real-time qualified |
| Camera/source replay | `recorded_source.py` has useful pacing and bounded prefetch, but source-specific timeline and paths | REUSE mechanisms; generic unknown-duration source adapter, no route-sized prefetch |
| Existing live ingress | `LivePerceptionIngress`, `LiveIngressEvent.wire_bytes`, `LiveSensorSynchronizer` already provide raw-first modality queues, hashes, epochs and bounded binding | REUSE, not a second acquisition system; extend worker/profile fencing and aggregate byte limits; map existing envelopes to the selected transport |
| Local surface | `RecordedGeometryStore` reads prepared NPZ | REUSE `K1LocalSurfaceShadowEstimator.process` and `lidar_local_surface_geometry`; feed current points/pose |
| Geometry/ranges | Current point association, exclusive ownership and geometry-only clusters exist | REUSE algorithms; remove recorded-store dependency and expose estimator/frame identity |
| Temporal/motion | `BoundedSpatialTemporalProvider`, `ClassIndependentMotionEstimator` | REUSE with current-run inputs and reset on epoch/lease change |
| TGS | Core algorithm reusable; old runners read a full schedule, some require exactly 4489 frames | REUSE TGS parameters/core; REPLACE schedule/file CLI with incremental input |
| Fusion | M49 TGS can be a neighboring shadow with `tgs_modifies_reference_graph_state=false` | TGS and semantics must influence the current advisory scene, not only viewer composition |
| Publication | Verified immutable result/publication/recovery exist | KEEP asynchronous final sealing; live results do not await it |
| Frontend | Setup/job/result surfaces exist, historical M49 viewer links other results | EXTEND common renderer capabilities with current-run layer references and freshness in stage 3 |
| Recording archive | Source identities and raw evidence must be retained | KEEP; create metadata/index/integrity during acquisition or incrementally, not via preflight full scan |

No changes to the active registry, old queue, source endpoint, UI or canonical
local server are made by this ADR or the stage-1 validators.

## Versioned handshake and observations

`missioncore.perception-stream-start/v1` is represented by `StreamStart` in
`src/k1link/perception/realtime_contract.py`. It contains run/source/worker/epoch,
lease generation, profile/image/effective-config/calibration identities, clock
domain, input mode and channel declarations. The first profile requires camera,
point-cloud and pose. Temporary absence of a sample is an explicit modality
outcome, not removal of a required capability.

It deliberately has no total source bytes, frame count, total duration, whole
archive, command, environment, complete member inventory or required EOF. The
initial metadata budget is 64 KiB. Large constant model assets and FOV masks
belong inside the image; calibration references resolve to bounded metadata.

The stage-2 wire mapping must carry:

- Run/epoch/lease binding on every message; source/channel sequence, capture
  timestamp and clock domain; independently verifiable payload length/hash.
- Camera codec initialization and then incremental encoded access units, or
  bounded fragments thereof. Native decoded representation is BGR8 800×600 KB4.
  Do not send an entire fMP4 epoch as one observation or turn all frames into PNGs.
- Current vendor-registered map point increments with point IDs, coordinate
  frame, units, scalar layout and retained raw attributes; map→sensor pose and
  calibration identity. The adapter owns vendor decoding; the profile is not a
  new K1 protocol implementation and cannot command the scanner.
- Explicit unavailable/gap/end/cancel messages. End is a termination event,
  never a prerequisite for first inference. Final recording inventory/hash can
  be sealed after consumption; per-chunk integrity is verified before use.
- Integer nanoseconds use protobuf integer fields; any JSON projection uses
  decimal strings for 64-bit times rather than lossy JavaScript numbers.

Maximum payload fragment: 1 MiB; total application inflight bytes: 16 MiB;
pending decoded camera frames: 2. Fragment reassembly is included in that budget,
with a deadline and incomplete-chunk accounting. Decoded image/tensor, cloud,
costmap and archive sinks need their own measured bounds inside the RSS budget.
Transport flow control alone is not a memory bound or a freshness policy.

gRPC bidirectional streaming is the first transport candidate, not an already
measured implementation. HTTP control-plane operations remain. The browser does
not connect directly to model RPCs. One authenticated data-plane endpoint feeds
the supervised profile; model processes have no arbitrary external network.
Protocol/codec selection must be measured on actual payload sizes in stage 2;
changing an archive POST to gRPC without changing execution is not acceptance.
Existing `missioncore.live-perception-wire/v1` and result wire v2 are migration
inputs: reuse their semantics and raw-first producer hooks. Their modality count
caps alone permit more than the proposed 16 MiB aggregate inflight budget; an
explicit byte bound is required. No duplicate new scanner feed is introduced.

## Time and lifecycle

Open → validate bounded metadata/claim → warm models → Ready → start the replay
clock → observations and incremental scenes → stop/drain bounded state → close
and asynchronously seal the immutable receipt. A live source already producing
data does not queue its entire warmup history: readiness begins at a current
decodable keyframe, with the skipped interval recorded.

Replay release uses original source intervals, not a convenient fixed 12 Hz and
not the processing speed. A slow consumer must not stretch source time to make
inference look real-time. Overload may drop bounded work to preserve liveness,
but the dropped observations remain in the ledger and fail the strict initial
performance target. Initial baseline is every camera frame, stride 1. Choosing
multirate later changes the explicit profile contract; a retained mask is not a
new inference and cannot receive a fresh capture timestamp.

UTC identifies the run; monotonic clocks measure local durations. Cross-host
age requires a clock mapping and its uncertainty. No direct subtraction of
unrelated host monotonic clocks. The common source timeline is separate from
worker wall-clock time. Causality is bounded by the released-observation
watermark at decision time, not by whatever later samples exist in the file.
Each layer retains its own channel/sequence/time; camera anchors and pose/cloud
association times remain separately visible. Future file look-ahead and offline
nearest-pose lookup across unreleased data are forbidden.

Lease expiry, cancellation and epoch changes fence the previous producer. No
new profile may take the GPU until old processes and work have stopped. A
reconnect creates a new epoch, discards stale backlog, resets temporal/rolling
state and reacquires codec initialization/keyframe. It never resends a stale
motor action; this profile has no motor actions at all.

## Outputs and interpretation

Every scene accounts for segmentation, objects, geometry, motion, costmap and
policy exactly once, with current/held/stale/unavailable state and current-run
payload identity. Existing source/object/obstacle/temporal/map/threat contracts
remain the domain vocabulary. `LayerEvidence` adds freshness/completeness checks,
not a second semantic ontology. In stage 2, layer manifests also carry their full
input reference sets; a single anchor does not replace camera/cloud/pose lineage.

Stage-1 implementation update, 2026-09-02 (`097e450`):
`missioncore.perception-scene-freshness/v1` in `realtime_scene.py` extends
`LayerEvidence` with oldest-required-input time and checks the six-layer dependency
graph. A derived result cannot refresh its input age. Decimal int64 JSON strings
preserve timestamps/sequence exactly; transport-neutral Python objects use integers.
The pilot validates payload hashes and assesses freshness again at receipt, and
must reassess at later use. Missing/stale output remains inspectable but cannot
retain permissive advisory policy. This envelope is NOT a replacement for
StreamStart identities, complete input lineage or controller lease fencing.

The stage-1 pilot propagated the oldest last-seen permissive TGS cell into the
costmap/policy age and suppressed the entire scene on expiry. Stage-2 increment
`1f8101e` adds `missioncore.costmap-cell-freshness/v1`: bounded (8192 cells),
index-aligned original support timestamps, decimal int64 strings/null, included
in the costmap payload digest together with the guard mode. Unobserved support
is never assigned a timestamp. The pilot now defaults to `per-cell`; the old
`whole-scene` mode remains an explicit comparison control, not a product cutover.

Both publication and receipt remove permission from expired ground cells and
mark them rejected/NO_GO. Occupied cells retain prohibition. The remaining
permissions propagate their original support age AND the mandatory segmentation,
geometry and motion dependencies. A missing/stale mandatory layer still blocks
the whole policy. A derived consumer view has new costmap/policy hashes while
preserving source identity/timestamps; the original published bytes stay immutable.
Reassessment cannot restore a suppressed action or move the observer clock back.
The enclosing StreamStart must bind the grid/profile/epoch/clock; this descriptor
alone does not establish those identities or authenticate a remote producer.

In the measured 128-frame window, raw model/geometry/motion/TGS/material outputs
remain identical to pinned PyTorch. Effective costmap states and permissions
change deliberately on expiry. A fresh six-layer envelope can still contain
rejected/unknown cells; it is not a claim of full free-space coverage, semantic
correctness, vehicle clearance or physical safety. Recheck again at actual use.

`bcacb02` promotes the proven mailbox and serial GPU stage to common perception
modules, with the pilot importing compatibility aliases rather than owning
another scheduler. Pending ingress + completed GPU outputs share two slots;
active inputs share 16 MiB and are released by their owning stage. Cancellation
discards pending work, a timed-out callback retains ownership, and CPU-owned
input is not freed by GPU shutdown. Diagnostic drop history is capped at 256
entries with exact per-reason totals; it is not the durable terminal ledger.
These primitives are NOT the controller supervisor, a cross-process GPU lease,
StreamStart fencing or the binary data plane. Those stage-2 boundaries remain open.

`worker_operating_envelope.py` checks a trusted post-warmup snapshot against
preregistered hardware/driver/resource/clock conditions and StreamStart identities,
ownership and client inventory. Unknown or expired facts fail readiness. The
candidate records the previously measured 4090 fixed-clock envelope; a compatible
auto-clock experiment cannot inherit its latency result. The evaluator performs
no I/O, host clock changes, lease acquisition or qualification; collection and
continuous enforcement belong to the common controller/runtime in stage 2.

DDRNet currently crops the center 600×600 to 512×512; outside that ROI is undefined,
not hard_surface/free. RF-DETR accepts the full 800×600 native raster with its
pinned valid-FOV handling. Their coordinates must be related explicitly.

RF-DETR's existing filter emits person/cat/dog, but minimum box area 64 px,
maximum box fraction 0.5, FOV fraction 0.5 and required valid center create known
small-animal/close-large-object edge cases. They are retained and documented,
not silently retuned or presented as proven field quality.

Detected-object range is the median camera-Z of owned current support points;
geometry-only range is nearest Euclidean distance from the sensor. Neither is
automatically clearance from the physical vehicle body. Missing/ambiguous
support produces unavailable range, not zero/infinity. Generic static obstacles
survive absence of a detector class name.

TGS keeps ground/occupied/rejected/unobserved separate. Ground support alone is
not a traversability or actuation decision. Policy can produce an advisory
allowed-candidate/high-cost/blocked/unknown outcome with explanation. No planner,
physical motor command, autonomous driving acceptance or mission configurator
implementation is introduced.

## Package/dependency decision

The candidate manifest is
`config/perception/k1-perception-ddrnet39-rfdetr-tgs-prototype-v1.json`.
It is not installed into the active portable registry and has no invented image
digest. Model/checkpoint/config identities are pinned; source session IDs and old
derived output paths in reference experiments are not inherited requirements.

Select process/environment isolation inside one future image: Python 3.12
supervisor and geometry, RF-DETR TensorRT 11 runtime, existing Python 3.9 / Torch
1.13.1 cu117 / super-gradients 3.2.0 environment, and CPU C++ TGS. Exchange bounded
shared-memory/IPC payload references; one supervisor serializes GPU work. This
avoids a forced dependency upgrade or checkpoint conversion in stage 1. Image
assembly of the final standalone package and the full schedule pilot remain
explicit implementation evidence. A temporary common-base image passed separate
DDRNet, TensorRT, Python 3.12 geometry and C++ TGS execution probes; DDRNet masks
matched on 64 frames. This proves bounded ABI/execution compatibility in that
image, not jointly resident models, the supervisor/IPC or a complete profile.
The probes still mount pinned assets explicitly; they are not standalone proof.

The temporary Triton base lacks Python grpc/protobuf, cv2 and TensorRT bindings;
native trtexec/server are present. The actual supervisor/transport environment
must explicitly include and validate its dependencies. Numeric-library thread
limits are pinned to one after the bounded synthetic local-surface comparison
(mean 198.0 ms default versus 63.2 ms with limits); algorithm thresholds were not
changed. Real-cloud timing, output parity and whole-graph scheduling remain gates.

The initial DDRNet image-only probe failed before inference: the installed image
does not contain the checkpoint at its logical asset path. The bounded baseline
therefore uses an explicit read-only pinned checkpoint and runner mount. This is
valid component measurement, but fails the target's standalone packaging claim.
Super-gradients also needs a writable log directory; a bounded temporary log
mount was required. The new image must declare scratch areas explicitly without
requiring a writable host home, code checkout or implicit weight cache.

## Preregistered engineering acceptance

Owner clarification, 2026-09-02 MSK: bounded laboratory overload is an admissible
experimental outcome. Keep useful profiles that fail the current 4090 target;
do not block reusable runtime/packaging work solely on that performance result.
Functional execution, experimental availability, quality and real-time
qualification are separate dimensions. Qualification belongs to a measured
profile/config/hardware/source/transport combination, not permanently to a
model name. More powerful Worker/onboard placement is a future retest target,
not a claim that existing latency or internal IPC disappears automatically.
The original budgets and failed measurements below remain unchanged. No slowed
source clock, hidden drops, stale-as-current evidence or actuation is authorized.

The manifest freezes an initial engineering candidate: p95/p99 full output age
≤125 ms, required-layer age ≤250 ms, release lag ≤25 ms, first incremental result
≤1 s after source admission, warmup ≤120 s, stop ≤5 s, VRAM ≤22000 MiB, RSS ≤8192
MiB. First/last-window backlog growth ≤25 ms and no capacity drops, failures,
expired selected frames or unaccounted observations. These are prototype targets,
not physical braking/safety limits or claims of achieved performance.

`ReplayMeasurements`/`realtime_failures` reject full-source preload, EOF-only
results, slowed replay, incomplete accounting, hidden missing layers and budget
violations. Declared source gaps must be verified from the input ledger; they
produce explicit degraded scenes and are not counted as fully fresh perception.
At least one fully evaluable scene is required. A component FPS figure cannot
populate this whole-path receipt. A quality comparison or successful export is
not a substitute for the real-time gate.

Every observation receives a terminal ledger outcome. Track ingress/release,
selection, decode, inference, completion, emission, drop/expiry/failure and source
gaps separately. Store startup separately from steady state; record payload bytes,
queue high-water marks, clock error, RSS/VRAM, image/weights/config/source/equipment
identities. Export and UI encoding cannot block inference through an unlimited
queue. A receiver-side render/receipt timestamp, not just Worker completion, ends
the end-to-end interval.

## Evidence and remaining stage-1 work

See `experiments/perception/PERCEPTION_STREAM_STAGE1_2026-09-01.md` for measured
component results, bounded validation, Worker maintenance and remaining gates.
Neither this ADR nor passing synthetic contract tests marks the new runtime ready.