feat(k1): add live cameras and reliable spatial following
This commit is contained in:
DCCONSTRUCTIONS
@@ -7,9 +7,9 @@ Status: accepted for the experimental Mission Core runtime, 2026-07-16.
|
||||
The spatial scene and camera workspace must accept one device, a custom rig or a
|
||||
future multi-sensor vehicle without adding vendor-specific branches to the host
|
||||
UI. The currently verified K1 profile exposes one spatial visualization path and
|
||||
two independent H.264 camera preview producers. The browser does not yet have a
|
||||
camera delivery adapter, a shared point-cloud/video clock or confirmed camera
|
||||
intrinsics/extrinsics.
|
||||
two path-labelled H.264 camera preview producers. The direct physical gate showed
|
||||
that only one K1 preview producer can deliver frames at a time. A shared
|
||||
point-cloud/video clock and camera intrinsics/extrinsics remain unavailable.
|
||||
|
||||
Hard-coded camera slots, direct RTSP URLs in React components and an apparently
|
||||
seekable timeline would therefore create three false contracts: a fixed sensor
|
||||
@@ -27,12 +27,15 @@ count, browser ownership of device transport and replay that does not exist.
|
||||
acquisition may change layout context without changing the logical producer.
|
||||
4. The host owns visibility, z-order, position, resize and fullscreen. A plugin
|
||||
cannot inject global layout behavior.
|
||||
5. Direct device RTSP URLs do not cross into the generic UI. A future read-only
|
||||
camera adapter must publish a browser-decodable local delivery URL.
|
||||
6. Until a bounded buffer and clock mapping exist, the shared timeline is
|
||||
5. Direct device RTSP URLs do not cross into the generic UI. The read-only K1
|
||||
adapter publishes an opaque, same-origin fMP4/WebSocket delivery lease.
|
||||
6. A descriptor may declare a generic activation group, capacity and selection
|
||||
state. The K1 camera producers share one group with `maxActive=1`; frontend
|
||||
layout enforces it for UX and the plugin remains the backend authority.
|
||||
7. Until a bounded buffer and clock mapping exist, the shared timeline is
|
||||
`live-only`, `seekable=false`, `sessionRecording=false` and explicitly marked
|
||||
`host-arrival-best-effort`.
|
||||
7. Directional 3D camera frustums are not rendered until calibration and
|
||||
8. Directional 3D camera frustums are not rendered until calibration and
|
||||
extrinsics are available and verified. An unresolved camera may be listed as
|
||||
a source, but its orientation must not be invented.
|
||||
|
||||
@@ -45,8 +48,9 @@ count, browser ownership of device transport and replay that does not exist.
|
||||
| `sensor.camera.right` | `camera.preview.live` | auxiliary media window / camera card |
|
||||
|
||||
Both camera producers share the proven canonical channel and remain distinct by
|
||||
`sourceId`. Their current state is `declared`: the compatibility profile proves
|
||||
the channels, but no browser preview URL is advertised yet.
|
||||
`sourceId`. The plugin publishes both catalog rows dynamically, but delivery is
|
||||
present only on the selected producer. Selecting the peer tears down the old
|
||||
RTSP/FFmpeg generation before a new browser lease is issued.
|
||||
|
||||
## UI consequences
|
||||
|
||||
@@ -62,11 +66,12 @@ the channels, but no browser preview URL is advertised yet.
|
||||
|
||||
## Follow-up gate
|
||||
|
||||
The next implementation step is a plugin-owned, read-only H.264 adapter that
|
||||
terminates RTSP/RTP locally and exposes a browser-supported stream plus measured
|
||||
per-source metrics. Synchronized rewind becomes eligible only after the adapter
|
||||
provides bounded buffering and an evidenced mapping between the RTP 90 kHz clock
|
||||
and the spatial stream clock.
|
||||
The local laboratory adapter now terminates RTSP/RTP with FFmpeg, copy-remuxes
|
||||
H.264 High L4 without transcoding and publishes complete bounded fMP4 segments
|
||||
to a generic MSE player. Portable FFmpeg packaging, multi-consumer fan-out and
|
||||
remote/WAN delivery are not claimed by this milestone. Synchronized rewind
|
||||
becomes eligible only after persisted buffering and an evidenced mapping between
|
||||
the RTP 90 kHz clock and the spatial stream clock. See ADR 0007.
|
||||
|
||||
## Code anchors
|
||||
|
||||
@@ -76,3 +81,4 @@ and the spatial stream clock.
|
||||
- `apps/control-station/src/components/ObservationTimeline.tsx`
|
||||
- `apps/control-station/src/workspaces/Workspaces.tsx`
|
||||
- `apps/control-station/src/device-plugins/xgrids-k1/observationSources.ts`
|
||||
- `src/k1link/web/xgrids_k1_camera.py`
|
||||
|
||||
@@ -0,0 +1,88 @@
|
||||
# ADR 0007: K1 camera preview copy-remux gateway
|
||||
|
||||
Status: accepted for the local laboratory runtime, 2026-07-16.
|
||||
|
||||
## Context
|
||||
|
||||
The exact K1 firmware 3.0.2 profile exposes two RTSP/TCP H.264 preview paths.
|
||||
Traffic evidence and the direct physical gate establish High Profile Level 4.0,
|
||||
800x600, approximately 10 fps, about 6.5 Mbit/s and a one-second GOP. The device
|
||||
delivers frames to only one preview consumer/path at a time. Browsers cannot read
|
||||
RTSP directly and device addresses must not enter generic UI contracts.
|
||||
|
||||
The product target will eventually require portable Edge packaging and remote
|
||||
delivery. This milestone only needs a low-latency, same-Mac laboratory path that
|
||||
does not bake XGRIDS knowledge into the host UI.
|
||||
|
||||
## Decision
|
||||
|
||||
1. The XGRIDS plugin owns the RTSP session and accepts only the two reviewed
|
||||
`source_id` values. The browser never supplies a host, port, path or RTSP URL.
|
||||
2. The target is taken from the active, attested device session and validated as
|
||||
private IPv4. Every selection has a monotonically increasing generation.
|
||||
3. Switching sources terminates the previous FFmpeg process before publishing
|
||||
the new generation. A stale stop cannot close a newer preview.
|
||||
4. FFmpeg uses `-c:v copy`: it depacketizes the observed vendor RTP framing and
|
||||
copy-remuxes H.264 into fragmented MP4. No video decode, image processing or
|
||||
transcode runs in Python.
|
||||
5. A background ISO-BMFF parser publishes one `ftyp+moov` init segment followed
|
||||
by complete `moof+mdat` media segments. Boxes and segments are bounded; the
|
||||
server queue holds at most four segments. A slow consumer causes a clean
|
||||
stream stop instead of unbounded latency.
|
||||
6. The generic browser renderer uses MSE with a 2 MiB pending queue and retains
|
||||
at most about three seconds in `SourceBuffer`. It never drops an arbitrary
|
||||
dependent fragment; overflow resets the lease.
|
||||
7. Plugin state exposes only an opaque same-origin WebSocket URL, media type,
|
||||
generation and generic activation metadata. Device RTSP details remain inside
|
||||
the plugin.
|
||||
|
||||
## Verified gate
|
||||
|
||||
The exact captured K1 codec sequence was passed in memory through the selected
|
||||
copy-remux flags and Chrome MSE. Chrome decoded High L4 at 800x600 with no media
|
||||
error and without transcoding. No camera frame fixture was persisted or added to
|
||||
Git.
|
||||
|
||||
The final physical gate was completed on 2026-07-17 with the exact firmware
|
||||
3.0.2 profile. In one Mission Core acquisition the operator confirmed:
|
||||
|
||||
- continuously updating point-cloud geometry in the embedded spatial scene;
|
||||
- a visible trajectory matching an approximately 3.2 by 3.35 metre physical
|
||||
route (about 1.29 metres of vertical span in the captured poses);
|
||||
- live left and right camera selection through the generic source controls;
|
||||
- graceful operator-confirmed scanner stop, receiver finalization and media
|
||||
process cleanup.
|
||||
|
||||
The retained out-of-Git evidence records 11,189 point-cloud frames, 11,564 pose
|
||||
frames, 33,481,543 published points and zero decode errors. The live preview
|
||||
queue reported 2,273 bounded drops under load; this is preview backpressure,
|
||||
not loss of the raw-first evidence, and remains a performance follow-up.
|
||||
|
||||
During the gate the scene initially appeared frozen even though counters and
|
||||
captures advanced. The cause was the embedded Rerun time cursor leaving the live
|
||||
edge. The accepted implementation now declares the `stream_time` timeline in
|
||||
native `Following` state in the backend blueprint; the frontend selects that
|
||||
timeline once when the recording opens and does not poll or force the cursor.
|
||||
|
||||
## Deliberate limits
|
||||
|
||||
- The current gateway allows one browser consumer because the local UI owns one
|
||||
operator preview. Multi-tab fan-out is not claimed.
|
||||
- The Homebrew FFmpeg found on this Mac is marked `development-system`. Portable
|
||||
releases must supply a pinned repository-local or configured binary.
|
||||
- Same-origin local WebSocket delivery is not the future WAN transport. A later
|
||||
Edge/Control Station split should use an authenticated WebRTC/WHEP or equivalent
|
||||
congestion-aware media plane behind the same descriptor contract.
|
||||
- Camera recording, synchronized rewind, calibration, frustums, panoramic
|
||||
stitching and full-resolution raw imagery remain unavailable.
|
||||
- A longer physical soak test is still required to characterize sustained
|
||||
preview backpressure and rendering cost on target Edge hardware.
|
||||
|
||||
## Code anchors
|
||||
|
||||
- `src/k1link/web/xgrids_k1_camera.py`
|
||||
- `src/k1link/web/xgrids_k1_facade.py`
|
||||
- `apps/control-station/src/components/MseFmp4WebSocketPlayer.tsx`
|
||||
- `apps/control-station/src/core/observation/useObservationLayout.ts`
|
||||
- `apps/control-station/src/device-plugins/xgrids-k1/observationSources.ts`
|
||||
- `tests/test_xgrids_camera_gateway.py`
|
||||
Reference in New Issue
Block a user