237 lines
16 KiB
Markdown
237 lines
16 KiB
Markdown
# ADR 0049 — Stream-first full perception profiles
|
||
|
||
Date: 2026-09-01. Status: stage-1 contract candidate; not a runtime cutover.
|
||
Execution order remains the four stages in
|
||
[the ExecPlan](../OBSERVATORY_REALTIME_PROFILES_EXECPLAN.md).
|
||
|
||
## Decision and owner intent
|
||
|
||
Observatory LABs test the real-time behavior of a complete perception profile,
|
||
not just the quality of an offline export. A recording replaces the physical
|
||
source, preserving original release timing at 1×. It must not enable a different
|
||
batch implementation. The first profile contains DDRNet-39 GOOSE, RF-DETR native,
|
||
online LiDAR geometry/ranges, temporal/motion, CPU TRAVEL TGS/costmap and advisory
|
||
policy. One dedicated RTX 4090 owns one active full profile.
|
||
|
||
The first rural prototype accepts existing coarse `hard_surface`, including
|
||
asphalt, sidewalk, bikeway and cobble. Existing `static_obstacle`/`static.unknown`
|
||
is sufficient. No new bollard classifier, extra segmenter, road/sidewalk taxonomy
|
||
or annotation campaign precedes this prototype. Other material rules remain an
|
||
explicit effective mission configuration; rural context alone is not permission
|
||
for all vegetation or soil. Geometry/unknown/freshness overrides material allowance.
|
||
|
||
EoMT is preserved as a separate future profile. It is not loaded with DDRNet.
|
||
RF-DETR and DDRNet are two required components of the same first profile: one
|
||
supervisor and serialized GPU scheduling, not competing profile jobs.
|
||
|
||
## CURRENT → TARGET / ownership
|
||
|
||
| Boundary | Current evidence | Decision / stage |
|
||
| --- | --- | --- |
|
||
| Portable definitions | `portable_run_definitions.py` owns pins, resources and admission | KEEP identity/sealing; EXTEND stream capabilities and qualification in stage 3 |
|
||
| Source requirements | Recorded-only requirements include seekability, one media epoch and init digest | REPLACE as live admission criteria with channel/format/calibration/clock requirements; keep init digest as per-stream integrity |
|
||
| Job queue | `recorded_jobs.py` has durable claims but global single-active-job scope | KEEP durable queue; EXTEND worker-scoped fencing, not unbounded GPU concurrency |
|
||
| Worker agent | Blocking `execute(job)` and final publication | EXTEND incremental lifecycle and cancellation; no per-LAB agent |
|
||
| Runtime | `portable_worker_runtime.py:735` materializes before execution | NEW stream runtime in stage 2; never call this materializer in the real-time path |
|
||
| Source transport | Whole camera archive download/hash/extraction before use | REPLACE startup barrier with bounded metadata, chunks and incremental integrity |
|
||
| Installed LAB V1 | prepare → complete EoMT → complete DDRNet → assemble | Legacy evidence only; not the new profile and not real-time qualified |
|
||
| Camera/source replay | `recorded_source.py` has useful pacing and bounded prefetch, but source-specific timeline and paths | REUSE mechanisms; generic unknown-duration source adapter, no route-sized prefetch |
|
||
| Existing live ingress | `LivePerceptionIngress`, `LiveIngressEvent.wire_bytes`, `LiveSensorSynchronizer` already provide raw-first modality queues, hashes, epochs and bounded binding | REUSE, not a second acquisition system; extend worker/profile fencing and aggregate byte limits; map existing envelopes to the selected transport |
|
||
| Local surface | `RecordedGeometryStore` reads prepared NPZ | REUSE `K1LocalSurfaceShadowEstimator.process` and `lidar_local_surface_geometry`; feed current points/pose |
|
||
| Geometry/ranges | Current point association, exclusive ownership and geometry-only clusters exist | REUSE algorithms; remove recorded-store dependency and expose estimator/frame identity |
|
||
| Temporal/motion | `BoundedSpatialTemporalProvider`, `ClassIndependentMotionEstimator` | REUSE with current-run inputs and reset on epoch/lease change |
|
||
| TGS | Core algorithm reusable; old runners read a full schedule, some require exactly 4489 frames | REUSE TGS parameters/core; REPLACE schedule/file CLI with incremental input |
|
||
| Fusion | M49 TGS can be a neighboring shadow with `tgs_modifies_reference_graph_state=false` | TGS and semantics must influence the current advisory scene, not only viewer composition |
|
||
| Publication | Verified immutable result/publication/recovery exist | KEEP asynchronous final sealing; live results do not await it |
|
||
| Frontend | Setup/job/result surfaces exist, historical M49 viewer links other results | EXTEND common renderer capabilities with current-run layer references and freshness in stage 3 |
|
||
| Recording archive | Source identities and raw evidence must be retained | KEEP; create metadata/index/integrity during acquisition or incrementally, not via preflight full scan |
|
||
|
||
No changes to the active registry, old queue, source endpoint, UI or canonical
|
||
local server are made by this ADR or the stage-1 validators.
|
||
|
||
## Versioned handshake and observations
|
||
|
||
`missioncore.perception-stream-start/v1` is represented by `StreamStart` in
|
||
`src/k1link/perception/realtime_contract.py`. It contains run/source/worker/epoch,
|
||
lease generation, profile/image/effective-config/calibration identities, clock
|
||
domain, input mode and channel declarations. The first profile requires camera,
|
||
point-cloud and pose. Temporary absence of a sample is an explicit modality
|
||
outcome, not removal of a required capability.
|
||
|
||
It deliberately has no total source bytes, frame count, total duration, whole
|
||
archive, command, environment, complete member inventory or required EOF. The
|
||
initial metadata budget is 64 KiB. Large constant model assets and FOV masks
|
||
belong inside the image; calibration references resolve to bounded metadata.
|
||
|
||
The stage-2 wire mapping must carry:
|
||
|
||
- Run/epoch/lease binding on every message; source/channel sequence, capture
|
||
timestamp and clock domain; independently verifiable payload length/hash.
|
||
- Camera codec initialization and then incremental encoded access units, or
|
||
bounded fragments thereof. Native decoded representation is BGR8 800×600 KB4.
|
||
Do not send an entire fMP4 epoch as one observation or turn all frames into PNGs.
|
||
- Current vendor-registered map point increments with point IDs, coordinate
|
||
frame, units, scalar layout and retained raw attributes; map→sensor pose and
|
||
calibration identity. The adapter owns vendor decoding; the profile is not a
|
||
new K1 protocol implementation and cannot command the scanner.
|
||
- Explicit unavailable/gap/end/cancel messages. End is a termination event,
|
||
never a prerequisite for first inference. Final recording inventory/hash can
|
||
be sealed after consumption; per-chunk integrity is verified before use.
|
||
- Integer nanoseconds use protobuf integer fields; any JSON projection uses
|
||
decimal strings for 64-bit times rather than lossy JavaScript numbers.
|
||
|
||
Maximum payload fragment: 1 MiB; total application inflight bytes: 16 MiB;
|
||
pending decoded camera frames: 2. Fragment reassembly is included in that budget,
|
||
with a deadline and incomplete-chunk accounting. Decoded image/tensor, cloud,
|
||
costmap and archive sinks need their own measured bounds inside the RSS budget.
|
||
Transport flow control alone is not a memory bound or a freshness policy.
|
||
|
||
gRPC bidirectional streaming is the first transport candidate, not an already
|
||
measured implementation. HTTP control-plane operations remain. The browser does
|
||
not connect directly to model RPCs. One authenticated data-plane endpoint feeds
|
||
the supervised profile; model processes have no arbitrary external network.
|
||
Protocol/codec selection must be measured on actual payload sizes in stage 2;
|
||
changing an archive POST to gRPC without changing execution is not acceptance.
|
||
Existing `missioncore.live-perception-wire/v1` and result wire v2 are migration
|
||
inputs: reuse their semantics and raw-first producer hooks. Their modality count
|
||
caps alone permit more than the proposed 16 MiB aggregate inflight budget; an
|
||
explicit byte bound is required. No duplicate new scanner feed is introduced.
|
||
|
||
## Time and lifecycle
|
||
|
||
Open → validate bounded metadata/claim → warm models → Ready → start the replay
|
||
clock → observations and incremental scenes → stop/drain bounded state → close
|
||
and asynchronously seal the immutable receipt. A live source already producing
|
||
data does not queue its entire warmup history: readiness begins at a current
|
||
decodable keyframe, with the skipped interval recorded.
|
||
|
||
Replay release uses original source intervals, not a convenient fixed 12 Hz and
|
||
not the processing speed. A slow consumer must not stretch source time to make
|
||
inference look real-time. Overload may drop bounded work to preserve liveness,
|
||
but the dropped observations remain in the ledger and fail the strict initial
|
||
performance target. Initial baseline is every camera frame, stride 1. Choosing
|
||
multirate later changes the explicit profile contract; a retained mask is not a
|
||
new inference and cannot receive a fresh capture timestamp.
|
||
|
||
UTC identifies the run; monotonic clocks measure local durations. Cross-host
|
||
age requires a clock mapping and its uncertainty. No direct subtraction of
|
||
unrelated host monotonic clocks. The common source timeline is separate from
|
||
worker wall-clock time. Causality is bounded by the released-observation
|
||
watermark at decision time, not by whatever later samples exist in the file.
|
||
Each layer retains its own channel/sequence/time; camera anchors and pose/cloud
|
||
association times remain separately visible. Future file look-ahead and offline
|
||
nearest-pose lookup across unreleased data are forbidden.
|
||
|
||
Lease expiry, cancellation and epoch changes fence the previous producer. No
|
||
new profile may take the GPU until old processes and work have stopped. A
|
||
reconnect creates a new epoch, discards stale backlog, resets temporal/rolling
|
||
state and reacquires codec initialization/keyframe. It never resends a stale
|
||
motor action; this profile has no motor actions at all.
|
||
|
||
## Outputs and interpretation
|
||
|
||
Every scene accounts for segmentation, objects, geometry, motion, costmap and
|
||
policy exactly once, with current/held/stale/unavailable state and current-run
|
||
payload identity. Existing source/object/obstacle/temporal/map/threat contracts
|
||
remain the domain vocabulary. `LayerEvidence` adds freshness/completeness checks,
|
||
not a second semantic ontology. In stage 2, layer manifests also carry their full
|
||
input reference sets; a single anchor does not replace camera/cloud/pose lineage.
|
||
|
||
DDRNet currently crops the center 600×600 to 512×512; outside that ROI is undefined,
|
||
not hard_surface/free. RF-DETR accepts the full 800×600 native raster with its
|
||
pinned valid-FOV handling. Their coordinates must be related explicitly.
|
||
|
||
RF-DETR's existing filter emits person/cat/dog, but minimum box area 64 px,
|
||
maximum box fraction 0.5, FOV fraction 0.5 and required valid center create known
|
||
small-animal/close-large-object edge cases. They are retained and documented,
|
||
not silently retuned or presented as proven field quality.
|
||
|
||
Detected-object range is the median camera-Z of owned current support points;
|
||
geometry-only range is nearest Euclidean distance from the sensor. Neither is
|
||
automatically clearance from the physical vehicle body. Missing/ambiguous
|
||
support produces unavailable range, not zero/infinity. Generic static obstacles
|
||
survive absence of a detector class name.
|
||
|
||
TGS keeps ground/occupied/rejected/unobserved separate. Ground support alone is
|
||
not a traversability or actuation decision. Policy can produce an advisory
|
||
allowed-candidate/high-cost/blocked/unknown outcome with explanation. No planner,
|
||
physical motor command, autonomous driving acceptance or mission configurator
|
||
implementation is introduced.
|
||
|
||
## Package/dependency decision
|
||
|
||
The candidate manifest is
|
||
`config/perception/k1-perception-ddrnet39-rfdetr-tgs-prototype-v1.json`.
|
||
It is not installed into the active portable registry and has no invented image
|
||
digest. Model/checkpoint/config identities are pinned; source session IDs and old
|
||
derived output paths in reference experiments are not inherited requirements.
|
||
|
||
Select process/environment isolation inside one future image: Python 3.12
|
||
supervisor and geometry, RF-DETR TensorRT 11 runtime, existing Python 3.9 / Torch
|
||
1.13.1 cu117 / super-gradients 3.2.0 environment, and CPU C++ TGS. Exchange bounded
|
||
shared-memory/IPC payload references; one supervisor serializes GPU work. This
|
||
avoids a forced dependency upgrade or checkpoint conversion in stage 1. Image
|
||
assembly of the final standalone package and the full schedule pilot remain
|
||
explicit implementation evidence. A temporary common-base image passed separate
|
||
DDRNet, TensorRT, Python 3.12 geometry and C++ TGS execution probes; DDRNet masks
|
||
matched on 64 frames. This proves bounded ABI/execution compatibility in that
|
||
image, not jointly resident models, the supervisor/IPC or a complete profile.
|
||
The probes still mount pinned assets explicitly; they are not standalone proof.
|
||
|
||
The temporary Triton base lacks Python grpc/protobuf, cv2 and TensorRT bindings;
|
||
native trtexec/server are present. The actual supervisor/transport environment
|
||
must explicitly include and validate its dependencies. Numeric-library thread
|
||
limits are pinned to one after the bounded synthetic local-surface comparison
|
||
(mean 198.0 ms default versus 63.2 ms with limits); algorithm thresholds were not
|
||
changed. Real-cloud timing, output parity and whole-graph scheduling remain gates.
|
||
|
||
The initial DDRNet image-only probe failed before inference: the installed image
|
||
does not contain the checkpoint at its logical asset path. The bounded baseline
|
||
therefore uses an explicit read-only pinned checkpoint and runner mount. This is
|
||
valid component measurement, but fails the target's standalone packaging claim.
|
||
Super-gradients also needs a writable log directory; a bounded temporary log
|
||
mount was required. The new image must declare scratch areas explicitly without
|
||
requiring a writable host home, code checkout or implicit weight cache.
|
||
|
||
## Preregistered engineering acceptance
|
||
|
||
Owner clarification, 2026-09-02 MSK: bounded laboratory overload is an admissible
|
||
experimental outcome. Keep useful profiles that fail the current 4090 target;
|
||
do not block reusable runtime/packaging work solely on that performance result.
|
||
Functional execution, experimental availability, quality and real-time
|
||
qualification are separate dimensions. Qualification belongs to a measured
|
||
profile/config/hardware/source/transport combination, not permanently to a
|
||
model name. More powerful Worker/onboard placement is a future retest target,
|
||
not a claim that existing latency or internal IPC disappears automatically.
|
||
The original budgets and failed measurements below remain unchanged. No slowed
|
||
source clock, hidden drops, stale-as-current evidence or actuation is authorized.
|
||
|
||
The manifest freezes an initial engineering candidate: p95/p99 full output age
|
||
≤125 ms, required-layer age ≤250 ms, release lag ≤25 ms, first incremental result
|
||
≤1 s after source admission, warmup ≤120 s, stop ≤5 s, VRAM ≤22000 MiB, RSS ≤8192
|
||
MiB. First/last-window backlog growth ≤25 ms and no capacity drops, failures,
|
||
expired selected frames or unaccounted observations. These are prototype targets,
|
||
not physical braking/safety limits or claims of achieved performance.
|
||
|
||
`ReplayMeasurements`/`realtime_failures` reject full-source preload, EOF-only
|
||
results, slowed replay, incomplete accounting, hidden missing layers and budget
|
||
violations. Declared source gaps must be verified from the input ledger; they
|
||
produce explicit degraded scenes and are not counted as fully fresh perception.
|
||
At least one fully evaluable scene is required. A component FPS figure cannot
|
||
populate this whole-path receipt. A quality comparison or successful export is
|
||
not a substitute for the real-time gate.
|
||
|
||
Every observation receives a terminal ledger outcome. Track ingress/release,
|
||
selection, decode, inference, completion, emission, drop/expiry/failure and source
|
||
gaps separately. Store startup separately from steady state; record payload bytes,
|
||
queue high-water marks, clock error, RSS/VRAM, image/weights/config/source/equipment
|
||
identities. Export and UI encoding cannot block inference through an unlimited
|
||
queue. A receiver-side render/receipt timestamp, not just Worker completion, ends
|
||
the end-to-end interval.
|
||
|
||
## Evidence and remaining stage-1 work
|
||
|
||
See `experiments/perception/PERCEPTION_STREAM_STAGE1_2026-09-01.md` for measured
|
||
component results, bounded validation, Worker maintenance and remaining gates.
|
||
Neither this ADR nor passing synthetic contract tests marks the new runtime ready.
|