# K1 calibrated perception roadmap

Status date: 2026-07-20.

## Outcome

Mission Core will use the K1 factory camera/LiDAR calibration instead of
building a new calibration workflow for the assembled rig. The first product
vertical is recorded, deterministic and fail-closed:

```text
canonical camera epoch + canonical LiDAR epoch
  + device-bound factory calibration snapshot
  + versioned segmentation/detection result
  -> pixel masks
  -> labeled LiDAR points and object distances
  -> evidence-backed 3D boxes
  -> optional derived Rerun layer
```

Only after this vertical passes on `RAVNOVES00` does the same contract move to
bounded live processing. Rerun remains the viewer and recording projection;
inference, calibration validation and fusion remain Mission Core concerns.

## Facts already established

### Camera identity is already canonical

The K1 plugin does not need to infer left versus right from image content. It
owns the exact RTSP selection:

| Mission Core source | K1 RTSP endpoint |
|---|---|
| `sensor.camera.left` | `/live/chn_left_main` |
| `sensor.camera.right` | `/live/chn_right_main` |

The selected source ID is retained by the camera gateway, the durable archive
summary and `missioncore.compute-job/v1`. A source switch creates another codec
epoch rather than silently changing the meaning of an existing epoch.

The accepted session `20260720T065719Z_viewer_live`, displayed as
`RAVNOVES00`, contains one recorded video source:

```text
source_id: sensor.camera.right
codec_epoch: 1
decoded video: H.264, 800x600
```

Consequently camera selection for perception must be a lookup from the job's
canonical `source_id`, never a UI default and never a computer-vision guess.

### Factory calibration exists and is used by K1

Read-only firmware analysis of K1 firmware `3.0.2` established:

- factory camera parameters are read from
  `/mnt/system/factory-data/config/camera.yaml`;
- camera/LiDAR extrinsics are read from
  `/mnt/system/factory-data/config/extrinsic_camera_lidar.yaml`;
- the internal frontend consumes camera intrinsics, distortion and
  `T_rgb_lidar`;
- `libxcolor` consumes the calibration and produces the colorized point cloud;
- K1 enables PTP, synchronized left/right main cameras and real-time xcolor;
- the main camera calibration model is KB4 fisheye at `4000x3000`;
- the exposed file protocol has a distinct read action, `CacFileRead = 5`, and
  a distinct write action, `CacFileWrite = 6`.

The OTA image contains model defaults. The exact values for one physical K1
live in that unit's `factory-data`; Mission Core must snapshot those exact
files before it claims calibrated fusion.

### Geometric calibration is now normalized; timing remains separate

The physical A4/FW 3.0.2 unit was read on 2026-07-20. Firmware declarations,
the xcolor loader and the unit's exact native resolutions jointly establish:

| Mission Core source | firmware camera | factory slot | native | admitted |
|---|---|---|---:|---:|
| `sensor.camera.left` | `camera_left_main` | `camera_0` | 4000x3000 | 800x600 |
| `sensor.camera.right` | `camera_right_main` | `camera_1` | 4000x3000 | 800x600 |

`camera_2` and `camera_3` are the 1280x800 left/right secondary pair and are
not silently substituted for a main RTSP source. The main H.264 substream is a
firmware-configured 800x600 linear resize with scale `(0.2, 0.2)` and no crop or
warp stage in the admitted profile. KB4 distortion is unchanged; `fx`, `fy`,
`cx` and `cy` are scaled on their respective axes.

The normalized matrix notation is `T_destination_from_source`, row-major
homogeneous 4x4, with translation in metres. The file-level `transform` is
`T_camera_0_from_lidar`. A serialized `camera_N.camera_pose` is
`T_camera_0_from_camera_N`; the same inverse-and-compose operation used by
xcolor produces:

```text
T_camera_N_from_lidar = inverse(T_camera_0_from_camera_N)
                        * T_camera_0_from_lidar
```

The remaining uncertainty is temporal: exact camera exposure time versus the
host-arrival timestamps currently retained by the external archive. This is a
separate temporal-calibration gate and is not hidden inside accepted geometry.

## Calibration product boundary

### What the UI should expose

The normal device panel should show a small calibration status card:

- `Ready`, `Missing`, `Mismatch` or `Unverified`;
- device model and firmware profile;
- captured-at time and immutable snapshot digest;
- canonical left/right source to calibration-slot mapping;
- native calibrated resolution and admitted stream resolution;
- geometry verification status;
- timing status separately from geometry status.

It may expose one explicit **Read snapshot from connected K1** operation. That
operation is read-only at the product level and must show the exact two admitted
files and the resulting digest.

### What the UI should not expose

Mission Core must not accept or upload a firmware archive merely to obtain
calibration. A general firmware uploader would add OTA, integrity, rollback and
device-bricking responsibilities without helping the perception path.

An advanced offline import is useful only for replay, recovery or a device that
cannot currently be reached. It must accept a small Mission Core calibration
bundle containing only the two admitted YAML documents plus identity metadata.
It must not accept a firmware TAR, arbitrary filesystem path or device write.

### Read-only device snapshot rules

The existing plugin action `calibration.device-snapshot.read` becomes the only
live entry point. Its implementation must:

1. require an already verified, owner-controlled K1 session;
2. publish only the reviewed `CacFileRead` request;
3. use an exact path allowlist for the two factory YAML files;
4. reject redirects, relative paths, extra response paths and write commands;
5. bound each response before YAML parsing;
6. parse with a safe loader and an exact schema/finite-number validation;
7. record original byte length and SHA-256 without rewriting the source bytes;
8. bind the snapshot to device identity, model, firmware and compatibility
   profile;
9. publish the complete snapshot atomically in private storage;
10. leave `CacFileWrite` unavailable.

The snapshot contract must name transform direction and coordinate convention
explicitly. A matrix called only `extrinsic` or `T` is not admissible at the
Mission Core boundary.

### Implementation status — 2026-07-20

The first read-only boundary is now implemented:

- `protocol/calibration_file.py` encodes only command `5` and has no API for
  command `6`;
- its request path is a closed two-value type at runtime: `camera.yaml` or
  `extrinsic_camera_lidar.yaml` at their exact factory-data locations;
- response identity, session, authority, command, path, result code, UTF-8,
  duplicates and byte bounds fail closed;
- `protocol/calibration_mqtt.py` performs one `DeviceInfo` read followed by the
  two sequential file reads on one clean MQTT connection, with no reconnect or
  automatic retry;
- `calibration.device-snapshot.read` now invokes that physical read instead of
  returning the generic plugin state;
- before that read, Bridge/Direct Connect refresh the selected K1's DHCP lease
  from the read-only BLE status characteristic; the IP is session state, while
  the correlated MQTT `DeviceInfo` is the identity barrier;
- each successful read seals a new private `0700` directory with `0600` source
  files, a manifest and SHA-256 identities; no earlier snapshot is replaced;
- `calibration_schema.py` rejects duplicate keys, anchors, aliases, explicit
  tags, unknown fields, non-finite values, non-K1 dimensions and invalid rigid
  transforms before a snapshot can become available;
- the manifest contains explicit left/right bindings, the 800x600 image
  transform, KB4 parameters and `T_camera_N_from_lidar` matrices;
- the physical normalized snapshot captured at `2026-07-20T10:54:32.613Z`
  has content identity
  `05f3ad9b38b3a4fc95388a8ec83da83c745e217709e51787b3d5aad0969f6fa9`;
- the complete Python suite passes with the boundary installed.

The P0 geometric-input subgate is accepted. The recorded Rerun diagnostic is
also generated and opens in the native viewer. P0 as a whole still requires
measured reprojection agreement across the image and a temporal-offset budget;
parsing a valid matrix or visually plausible overlay is not a substitute for
those measurements.

The first reproducible recorded diagnostic was generated on 2026-07-20 by the
new `k1link analyze calibrated-overlay` path. Experiment
`20260720T112515Z_k1_calibrated_overlay_2655c5c7c38e` is bound to the immutable
RAVNOVES00 MQTT, camera and calibration generations. It produced four
depth-coloured PNG overlays plus a 4,016,830-byte Rerun diagnostic with SHA-256
`83bc5c1134563987c3be5734d6c069d07ad8a9e5b6806682169c89f4bd5354d9`.

The experiment established one previously implicit frame operation: current
`lio_pcl` coordinates are in the K1 map frame, so projection uses
`inverse(T_map_from_lidar)` before `T_camera_1_from_lidar` and KB4. The four
selected camera frames projected 1,613/2,617, 983/1,813, 1,543/2,744 and
1,730/1,880 source points respectively. Visual inspection shows coherent
facade, vehicle, vegetation and ground structure across the fisheye image.
This is diagnostic evidence, not a measured pixel-error acceptance: explicit
static landmark correspondences and the temporal-offset sweep remain open.

## Recorded segmentation and calibrated fusion probe

A four-frame P1/P2/P3 probe now exists on the same immutable RAVNOVES00 camera
anchors. The RTX worker received the raw camera pixels, not the LiDAR-coloured
overlay, and the transferred PNG SHA-256 values were verified on both hosts.

The admitted research profiles are:

- TorchVision Mask R-CNN ResNet50-FPN v2 / COCO_V1 for instance masks, with
  weight SHA-256
  `73cbd0190fcbe3ba339921fbce2c3a0b6bb9126c9a133c85e43a2a8e060a109e`;
- Microsoft BEiT base / ADE20K-150 for semantic masks, revision
  `a8b6f5ef4acb2ea55d882989deaa02d39401e2b2`, with weight SHA-256
  `e0747360d190bd7c0f53d2fe3b2ed560c304d3eefef94574af9c7e93aaf8e7a9`.

BEiT was published only after an exact checkpoint load with no missing,
unexpected or mismatched keys. An earlier Intel DPT/ADE probe was rejected:
its checkpoint left twelve fusion-layer parameters newly initialized. The
strict repeat failed before publication, so that output is diagnostic only and
cannot enter fusion.

The accepted segmentation generation is
`1ef80a6df60830d8ac37f09db679696325adb65ef61094b666165f561b992a08`.
On the three warm frames, sequential Mask R-CNN inference took 100–124 ms and
BEiT took 244–312 ms. The combined 365–412 ms is only about 2.4–2.7 FPS before
decode, transfer and fusion; this is not a live real-time acceptance.

Mission Core sampled both masks at factory-calibrated LiDAR projections and
published fusion experiment
`20260720T121000Z_k1_segmentation_fusion_d09020d0cb13`, generation
`3d150012cf88ad477c602aa072d0b6c2675ca472204b61d9377eb8b83c128842`.
Distance is Euclidean range from the current LiDAR origin. A diagnostic box is
published only after the instance mask contains at least four points in one
contiguous depth cluster; its current bounds are map-frame axis-aligned p05–p95
bounds with a 12 m maximum span gate.

| camera session | projected points | diagnostic Boxes3D |
|---:|---:|---:|
| 95.471 s | 1,613 | 2 |
| 215.408 s | 983 | 5 |
| 335.424 s | 1,543 | 3 |
| 455.450 s | 1,730 | 4 |

The span gate rejected a large false `car` mask even though it contained 854
clustered points. Sparse objects remain 2D observations with an explicit
insufficient-support status. The 11,582,442-byte Rerun RRD has SHA-256
`1c190e6dc64b8c1e187c053cd950dabc4167d1d7fe3c8ec5e2499c0d5d3fa471`.
Native headless viewer QA confirmed synchronized raw, semantic and fusion image
views, semantic `Points3D`, and translucent `Boxes3D` with distance/support
labels. Distances and boxes are not ground-truthed, tracked or safety accepted.

## Automatic camera/calibration binding

The binding is performed when a compute job is prepared:

```text
camera archive summary.source_id
  -> compatibility-profile source mapping
  -> exact camera slot in device calibration snapshot
  -> image-transform profile for the admitted codec epoch
  -> calibration generation SHA-256 in the compute job
```

Preparation fails closed when any link is absent or ambiguous. The worker must
not choose a default camera. The result repeats the calibration generation and
camera source; Mission Core revalidates both before accepting a derived layer.

The left/right mapping is now device-snapshot data backed by the exact FW 3.0.2
profile. The compute-job binding is still pending; no worker may infer a slot
from list order or use a default camera.

## Minimal recorded vertical

### Gate P0 — calibration snapshot and camera mapping

Deliverables:

- bounded protobuf codec for `CalibFileRequest` and `CalibFileResponse`;
- whitelisted read-only snapshot action and tests;
- immutable private snapshot with source digests;
- verified mapping for both canonical camera source IDs;
- validated 4000x3000-to-800x600 image transform;
- a Rerun diagnostic layer showing projected LiDAR samples over the image.

Acceptance requires visible and measured agreement on several static regions
across the image, not one hand-picked object at the center.

### Gate P1 — full-timeline 2D perception

Run the complete `RAVNOVES00` camera epoch, not one still image. Preserve every
decoded frame's admitted session timestamp and produce two independent
versioned outputs:

- instance masks for countable objects such as people and vehicles;
- semantic classes for background/stuff such as ground, building and
  vegetation.

Together these form the requested panoptic view. The existing YOLOX-S result is
a detector smoke test and remains useful as a baseline, but it cannot satisfy
this gate. Model choice remains replaceable behind the result contract and is
accepted only after measured accuracy, latency, VRAM and license review.

Operator playback uses a generation-bound H.264 panoptic MP4 beside the native
Rerun canvas. Embedding every 800x600 `SegmentationImage` row in RRD would make
multi-hour raster playback depend on browser memory. Lossless masks remain
content-addressed derived evidence for fusion, while the MP4 is read through
native HTTP Range and follows the same `session_time` controller as the source
camera and point cloud. Small diagnostic RRDs may still use
`SegmentationImage`, `AnnotationContext` and `Boxes2D`; that is not the scalable
full-epoch transport.

### Gate P2 — 2D-to-3D fusion and distance

For each admitted camera frame:

1. select a temporally compatible LiDAR frame/window;
2. transform LiDAR points into the selected camera frame;
3. project with the K1 KB4 model and the validated stream transform;
4. reject points behind the camera and resolve occlusion with a depth buffer;
5. sample the panoptic mask at each visible projected point;
6. retain labeled points in LiDAR coordinates;
7. compute per-instance nearest robust distance, centroid and distance spread;
8. cluster instance points and fit an oriented 3D box only when support is
   sufficient.

A 2D rectangle must never be blindly extruded into a fake 3D box. An object
with insufficient or stale LiDAR support remains a 2D observation with an
explicit `depth unavailable` reason.

Distance v1 should be displayed directly on each object label. Click-to-query
is a later interaction enhancement, not a dependency of correct distance.

### Gate P3 — Rerun 3D presentation

The derived layer logs:

- semantic/instance masks on the camera image;
- labeled points as `Points3D` using the same class ontology;
- supported objects as oriented `Boxes3D`;
- object ID, class, confidence, distance and support count;
- calibration/result generation metadata and degraded-state reasons.

Rerun's native camera projection is pinhole-only. K1 uses KB4 fisheye, so
Mission Core must perform the calibrated KB4 projection itself or log a proven
undistorted image. It must not present an uncorrected pinhole frustum as exact
K1 geometry.

### Gate P4 — recorded qualification

Qualify the entire 8–9 minute session with:

- no frame or segment ceiling;
- deterministic repeat identity;
- end-to-end throughput and wall time;
- CPU, RAM, GPU, VRAM and disk telemetry;
- calibration reprojection statistics;
- temporal-offset sweep and moving-object error report;
- explicit failure/degraded reasons;
- immutable raw inputs and separately replaceable derived results.

## Bounded live vertical

Only after P0–P4 pass does live mode reuse the same model and fusion contracts.

The live path has two different policies:

- archival inputs are never dropped by the viewer or inference queue;
- derived preview is latest-wins with a small bounded queue and explicit
  dropped/stale counters.

The first live target is stable 5–10 Hz derived perception, not an unsupported
claim of matching every camera or LiDAR message. It must expose:

- capture, queue, decode, inference, fusion and render latency;
- queue depth and dropped derived frames;
- calibration generation in use;
- worker disconnect/recovery state;
- `healthy`, `degraded`, `stale` and `unavailable` states;
- raw recording continuity when the worker is slow or absent.

The external RTX worker remains an executor. It receives bounded observations
and calibration references; it gains no K1 command authority.

## Deferred intentionally

The following are not prerequisites for the first correct vertical:

- uploading full firmware through Mission Core;
- calibrating the already factory-calibrated assembled K1 rig again;
- native LiDAR-only neural segmentation;
- multi-camera stitching;
- multi-object tracking across long occlusions;
- HD maps, ROS 2, vehicle control or safety decisions;
- a universal model marketplace or scheduler.

Native point-cloud segmentation and tracking may be added later as independent
pipelines and compared against the camera-fused labels. They must not block the
first evidence-backed 3D boxes and distances.

## Immediate implementation order

1. Keep the accepted Windows reboot path healthy and reproducible.
2. ~~Implement and offline-test the bounded calibration file protobuf codec.~~
3. ~~Implement `calibration.device-snapshot.read` with the two-path allowlist.~~
4. ~~Capture the physical K1 snapshot and prove left/right slot mapping.~~
5. ~~Generate a reproducible calibrated LiDAR-over-video diagnostic on
   `RAVNOVES00`.~~ Record measured static-landmark reprojection error and the
   temporal-offset budget for the accepted 800x600 transform.
6. Extend compute job/result contracts with camera source and calibration
   generation binding.
7. ~~Prove instance and semantic masks on four physical frames.~~ Produce the
   complete recorded epoch with immutable result identity and exact-repeat
   reuse.
8. ~~Prove mask-to-point fusion, diagnostic distances and support-gated 3D
   boxes on four frames.~~ Add ground truth, tracking and accepted oriented-box
   fitting.
9. ~~Open the four-frame fused result in native Rerun.~~ Qualify the full
   session with resource and timing telemetry.
10. ~~Qualify the bounded latest-wins scheduler and machine world-state contract
    with source-paced replay.~~ Move the same contract onto the real worker
    inference path, then connect the live K1.

## LAB E7 replay-real-time checkpoint · 2026-07-21

LAB E7 accepted the first source-paced downstream perception runtime on the
same 601-frame E6 interval. The result
`e7-live-replay-e339aaed75fff05ed893ae48770bd127cc0c764f3124fc4b6be94b5fd9f8389c`
published 601/601 machine world states at 10.021845 Hz with zero drops, a
capacity-two queue and 5.927 ms p95 scheduled-tick-to-publication latency. The
526 fused frames were healthy; all 75 E6 sync-gated frames were explicitly
degraded, with no stale or unavailable states.

The world state carries object identity, class, map and K1-LiDAR position,
visible-surface 3D size, range, support, guarded diagnostic velocity and a
72-sector diagnostic clearance observation. Vehicle-body coordinates remain
unavailable until a real sensor-to-vehicle transform is supplied. Rerun is a
subscriber showing point cloud, `Boxes3D`, latency, queue depth and health; it
is not the perception authority.

An intentional 250 ms consumer-delay control processed 33/80 frames, discarded
47 obsolete derived frames, never exceeded queue depth two and marked 32
published states stale. Raw evidence remained outside the derived queue. This
proves bounded overload behavior, not live inference. E4, E5 and E6 inputs were
precomputed and the AI worker was untouched.

Full provenance, pilot corrections, profile hashes, latency tables, velocity
guards, clearance limits, artifacts and the LAB E8 gate are recorded in
`experiments/perception/LAB_E7_REPORT_2026-07-21.md`.
