feat(k1): add live cameras and reliable spatial following

This commit is contained in:
DCCONSTRUCTIONS
2026-07-17 00:26:03 +03:00
parent a281faf923
commit 2bda1986bd
33 changed files with 2927 additions and 330 deletions
+18 -6
View File
@@ -101,10 +101,20 @@ receiver and operator workflow only. K1 scanning is still started and stopped
by physical double-click; no modeling request is published.
Left/right camera preview transport, endpoint paths and H.264 framing are now
observed under the exact compatibility profile. The read-only runtime adapter is
not implemented yet. Device status and heartbeat remain raw observed channels
without semantic decoders. Device calibration command and sensor-to-vehicle
extrinsics are unavailable.
observed under the exact compatibility profile. The local read-only adapter
copy-remuxes one selected RTSP producer into bounded fMP4/WebSocket delivery for
the generic MSE UI. Portable FFmpeg packaging, fan-out and remote delivery remain
open. Device status and heartbeat remain raw observed channels without semantic
decoders. Device calibration command and sensor-to-vehicle extrinsics are
unavailable.
The 2026-07-17 physical acceptance gate confirmed continuously updating point
clouds, a matching live trajectory, and both camera selections in the same
Mission Core acquisition. The embedded Rerun blueprint owns live-edge following
on `stream_time`; the React shell selects the timeline once and does not drive it
with a timer. Raw captures, camera frames, device identity and network details
remain outside Git. See [`ADR 0007`](adr/0007-k1-camera-preview-copy-remux-gateway.md)
for the measured gate and remaining limits.
## Remaining extraction order
@@ -118,7 +128,8 @@ extrinsics are unavailable.
5. Replace compatibility routes and singleton state with multi-device session
routing.
6. Split Edge execution from the Control Station behind authenticated transport.
7. Implement the now-observed read-only RTSP/H.264 camera adapter; keep the
7. Package the read-only RTSP/H.264 camera adapter for each target OS and evolve
same-host MSE delivery toward an authenticated Edge media plane; keep the
modeling-command publisher disabled until its separate safety gate closes.
## Invariants
@@ -149,7 +160,8 @@ extrinsics are unavailable.
- complete SDK-envelope/EvidenceStore hot-path integration;
- remote Edge split, authenticated WAN relay and fleet orchestration;
- automatic K1 start/stop/calibration commands;
- camera discovery, decoding, synchronization and visualization.
- automatic camera capability discovery, synchronized recording/rewind,
calibration, panoramic stitching and multi-consumer delivery.
## Workspace-repeatable locked architecture gate
@@ -7,9 +7,9 @@ Status: accepted for the experimental Mission Core runtime, 2026-07-16.
The spatial scene and camera workspace must accept one device, a custom rig or a
future multi-sensor vehicle without adding vendor-specific branches to the host
UI. The currently verified K1 profile exposes one spatial visualization path and
two independent H.264 camera preview producers. The browser does not yet have a
camera delivery adapter, a shared point-cloud/video clock or confirmed camera
intrinsics/extrinsics.
two path-labelled H.264 camera preview producers. The direct physical gate showed
that only one K1 preview producer can deliver frames at a time. A shared
point-cloud/video clock and camera intrinsics/extrinsics remain unavailable.
Hard-coded camera slots, direct RTSP URLs in React components and an apparently
seekable timeline would therefore create three false contracts: a fixed sensor
@@ -27,12 +27,15 @@ count, browser ownership of device transport and replay that does not exist.
acquisition may change layout context without changing the logical producer.
4. The host owns visibility, z-order, position, resize and fullscreen. A plugin
cannot inject global layout behavior.
5. Direct device RTSP URLs do not cross into the generic UI. A future read-only
camera adapter must publish a browser-decodable local delivery URL.
6. Until a bounded buffer and clock mapping exist, the shared timeline is
5. Direct device RTSP URLs do not cross into the generic UI. The read-only K1
adapter publishes an opaque, same-origin fMP4/WebSocket delivery lease.
6. A descriptor may declare a generic activation group, capacity and selection
state. The K1 camera producers share one group with `maxActive=1`; frontend
layout enforces it for UX and the plugin remains the backend authority.
7. Until a bounded buffer and clock mapping exist, the shared timeline is
`live-only`, `seekable=false`, `sessionRecording=false` and explicitly marked
`host-arrival-best-effort`.
7. Directional 3D camera frustums are not rendered until calibration and
8. Directional 3D camera frustums are not rendered until calibration and
extrinsics are available and verified. An unresolved camera may be listed as
a source, but its orientation must not be invented.
@@ -45,8 +48,9 @@ count, browser ownership of device transport and replay that does not exist.
| `sensor.camera.right` | `camera.preview.live` | auxiliary media window / camera card |
Both camera producers share the proven canonical channel and remain distinct by
`sourceId`. Their current state is `declared`: the compatibility profile proves
the channels, but no browser preview URL is advertised yet.
`sourceId`. The plugin publishes both catalog rows dynamically, but delivery is
present only on the selected producer. Selecting the peer tears down the old
RTSP/FFmpeg generation before a new browser lease is issued.
## UI consequences
@@ -62,11 +66,12 @@ the channels, but no browser preview URL is advertised yet.
## Follow-up gate
The next implementation step is a plugin-owned, read-only H.264 adapter that
terminates RTSP/RTP locally and exposes a browser-supported stream plus measured
per-source metrics. Synchronized rewind becomes eligible only after the adapter
provides bounded buffering and an evidenced mapping between the RTP 90 kHz clock
and the spatial stream clock.
The local laboratory adapter now terminates RTSP/RTP with FFmpeg, copy-remuxes
H.264 High L4 without transcoding and publishes complete bounded fMP4 segments
to a generic MSE player. Portable FFmpeg packaging, multi-consumer fan-out and
remote/WAN delivery are not claimed by this milestone. Synchronized rewind
becomes eligible only after persisted buffering and an evidenced mapping between
the RTP 90 kHz clock and the spatial stream clock. See ADR 0007.
## Code anchors
@@ -76,3 +81,4 @@ and the spatial stream clock.
- `apps/control-station/src/components/ObservationTimeline.tsx`
- `apps/control-station/src/workspaces/Workspaces.tsx`
- `apps/control-station/src/device-plugins/xgrids-k1/observationSources.ts`
- `src/k1link/web/xgrids_k1_camera.py`
@@ -0,0 +1,88 @@
# ADR 0007: K1 camera preview copy-remux gateway
Status: accepted for the local laboratory runtime, 2026-07-16.
## Context
The exact K1 firmware 3.0.2 profile exposes two RTSP/TCP H.264 preview paths.
Traffic evidence and the direct physical gate establish High Profile Level 4.0,
800x600, approximately 10 fps, about 6.5 Mbit/s and a one-second GOP. The device
delivers frames to only one preview consumer/path at a time. Browsers cannot read
RTSP directly and device addresses must not enter generic UI contracts.
The product target will eventually require portable Edge packaging and remote
delivery. This milestone only needs a low-latency, same-Mac laboratory path that
does not bake XGRIDS knowledge into the host UI.
## Decision
1. The XGRIDS plugin owns the RTSP session and accepts only the two reviewed
`source_id` values. The browser never supplies a host, port, path or RTSP URL.
2. The target is taken from the active, attested device session and validated as
private IPv4. Every selection has a monotonically increasing generation.
3. Switching sources terminates the previous FFmpeg process before publishing
the new generation. A stale stop cannot close a newer preview.
4. FFmpeg uses `-c:v copy`: it depacketizes the observed vendor RTP framing and
copy-remuxes H.264 into fragmented MP4. No video decode, image processing or
transcode runs in Python.
5. A background ISO-BMFF parser publishes one `ftyp+moov` init segment followed
by complete `moof+mdat` media segments. Boxes and segments are bounded; the
server queue holds at most four segments. A slow consumer causes a clean
stream stop instead of unbounded latency.
6. The generic browser renderer uses MSE with a 2 MiB pending queue and retains
at most about three seconds in `SourceBuffer`. It never drops an arbitrary
dependent fragment; overflow resets the lease.
7. Plugin state exposes only an opaque same-origin WebSocket URL, media type,
generation and generic activation metadata. Device RTSP details remain inside
the plugin.
## Verified gate
The exact captured K1 codec sequence was passed in memory through the selected
copy-remux flags and Chrome MSE. Chrome decoded High L4 at 800x600 with no media
error and without transcoding. No camera frame fixture was persisted or added to
Git.
The final physical gate was completed on 2026-07-17 with the exact firmware
3.0.2 profile. In one Mission Core acquisition the operator confirmed:
- continuously updating point-cloud geometry in the embedded spatial scene;
- a visible trajectory matching an approximately 3.2 by 3.35 metre physical
route (about 1.29 metres of vertical span in the captured poses);
- live left and right camera selection through the generic source controls;
- graceful operator-confirmed scanner stop, receiver finalization and media
process cleanup.
The retained out-of-Git evidence records 11,189 point-cloud frames, 11,564 pose
frames, 33,481,543 published points and zero decode errors. The live preview
queue reported 2,273 bounded drops under load; this is preview backpressure,
not loss of the raw-first evidence, and remains a performance follow-up.
During the gate the scene initially appeared frozen even though counters and
captures advanced. The cause was the embedded Rerun time cursor leaving the live
edge. The accepted implementation now declares the `stream_time` timeline in
native `Following` state in the backend blueprint; the frontend selects that
timeline once when the recording opens and does not poll or force the cursor.
## Deliberate limits
- The current gateway allows one browser consumer because the local UI owns one
operator preview. Multi-tab fan-out is not claimed.
- The Homebrew FFmpeg found on this Mac is marked `development-system`. Portable
releases must supply a pinned repository-local or configured binary.
- Same-origin local WebSocket delivery is not the future WAN transport. A later
Edge/Control Station split should use an authenticated WebRTC/WHEP or equivalent
congestion-aware media plane behind the same descriptor contract.
- Camera recording, synchronized rewind, calibration, frustums, panoramic
stitching and full-resolution raw imagery remain unavailable.
- A longer physical soak test is still required to characterize sustained
preview backpressure and rendering cost on target Edge hardware.
## Code anchors
- `src/k1link/web/xgrids_k1_camera.py`
- `src/k1link/web/xgrids_k1_facade.py`
- `apps/control-station/src/components/MseFmp4WebSocketPlayer.tsx`
- `apps/control-station/src/core/observation/useObservationLayout.ts`
- `apps/control-station/src/device-plugins/xgrids-k1/observationSources.ts`
- `tests/test_xgrids_camera_gateway.py`