fix(k1): restore low-latency live recovery path

docs/adr/0042-k1-run-ledger-and-regression-telemetry.md

@@ -0,0 +1,150 @@
+# ADR 0042 — K1 run ledger and regression telemetry
+
+Status: proposed for acceptance
+Date: 2026-08-22
+
+Named physical references and redacted evidence hashes are retained in
+[`docs/lab/010_K1_MISSION_CORE_INTERNAL_LIVE_BASELINE_20260822.redacted.md`](../lab/010_K1_MISSION_CORE_INTERNAL_LIVE_BASELINE_20260822.redacted.md).
+That report deliberately distinguishes the fast Mission Core internal
+ideal-path reference from the accepted recovery-capable checkpoint and from
+manufacturer/LixelGO evidence.
+
+## Context
+
+The K1 integration has repeatedly returned to the same visible failures after
+otherwise successful recovery work: delayed first point cloud, a camera that
+repeatedly disappears and reopens, browser decoder failures, slow START/STOP,
+and connection-mode transitions that disturb an already accepted Bridge path.
+
+The current evidence is individually strong but operationally fragmented. Raw
+MQTT capture, camera fMP4 epochs, acquisition checkpoints, browser diagnostics,
+redacted manifests and operator timing notes use different identities and must
+be correlated manually. A successful run is described in Ops, but is not yet an
+executable baseline that can reject a later regression.
+
+The repository already owns an accepted telemetry plane under ADR 0031:
+Telegraf and a bounded native JSONL outbox publish through authenticated MQTT to
+the Mission Core normalizer and TimescaleDB. Introducing another broker or
+database solely for K1 would create a second lifecycle, credential boundary and
+recovery problem without improving the local K1-to-Mission-Core evidence path.
+
+## Decision
+
+Every physical or replayed K1 acquisition is one explicit **K1 run**. The run is
+identified by the acquisition id and its evidence-session lineage. All derived
+events carry the same redacted identity, code revision, UI build id, connection
+mode and monotonic clock origin.
+
+The authoritative write path is:
+
+```text
+K1 control, MQTT receiver, camera gateway and browser diagnostics
+    -> local append-only K1 run journal
+    -> bounded durable outbox
+    -> authenticated telemetry MQTT topic
+    -> existing Mission Core telemetry normalizer
+    -> existing TimescaleDB
+    -> run comparison and acceptance report
+```
+
+The local journal remains the source of truth during network loss. Database
+availability never gates START, STOP, raw evidence capture, camera archival or
+local recovery. Delivery resumes from the outbox when the telemetry contour is
+available.
+
+The initial event contract is `missioncore.k1-run-event/v1`. Each event contains:
+
+- `run_id`, `acquisition_id`, redacted `evidence_session_id` lineage;
+- UTC and monotonic timestamps plus a per-run sequence number;
+- Mission Core commit, UI build id and K1 compatibility-profile id;
+- connection mode, host-path epoch and producer generations without addresses,
+  credentials, project names or raw device identifiers;
+- event code, stage, outcome and bounded numeric facts;
+- source component and source schema version.
+
+The required timeline includes at least:
+
+- operator START intent, physical command publication and confirmed SCANNING;
+- calibration start/end when the device exposes those facts;
+- first authoritative PCL received and first PCL published to Rerun;
+- browser Rerun store admitted and first visible point-cloud frame;
+- camera producer init, first media, browser first frame and sustained playing;
+- camera decoder errors, preview-reader retirements, IDR admission, queue age,
+  bytes and fragment counts;
+- connection gap start, recovery attempts, recovered binding and gap end;
+- operator STOP intent, confirmed READY and archive finalization;
+- terminal run result and automatically evaluated acceptance checks.
+
+An acceptance report is generated from the journal after each run. It compares
+the run with a named accepted baseline and reports exact deltas. The following
+conditions are hard failures rather than performance warnings:
+
+- any browser `MEDIA_ERR_DECODE`;
+- any decode error in the archived camera stream;
+- a missing first point-cloud admission or missing terminal archive;
+- loss of acquisition/control lineage;
+- a device state asserted by the UI without authoritative device evidence.
+
+Latency and FPS thresholds are versioned baseline checks. They are never
+implemented by silently reducing point count, image quality, bitrate or source
+FPS. A threshold change requires its own reviewed acceptance run.
+
+The engineering work format changes accordingly:
+
+1. Name the accepted baseline and the single invariant targeted by the change.
+2. Make one bounded implementation increment.
+3. Run focused synthetic tests before a physical test.
+4. Perform one canonical UI run; do not substitute direct device commands.
+5. Generate and retain the run comparison automatically.
+6. Do not merge a change that improves its target while regressing an already
+   accepted invariant.
+
+## Rejected alternatives
+
+### Redis
+
+Redis is useful for disposable cache and coordination. It is not the evidence
+system of record for offline physical runs and does not replace a durable local
+journal or analytical store.
+
+### Kafka
+
+Kafka is not admitted for this local single-producer contour. The existing MQTT
+plane already supplies authenticated fan-out and reconnect delivery, while the
+local outbox supplies offline durability. Kafka would add a second broker and
+operational quorum without repairing the K1, RTSP, browser or local-network
+boundaries. It can be reconsidered only when independent high-volume consumers,
+cross-site retention or replay requirements exceed the accepted MQTT contour.
+
+### ClickHouse
+
+ClickHouse is not admitted as a second telemetry database. Existing TimescaleDB
+already stores run identity, timestamps and JSON payloads and is sufficient for
+the current event volume and comparisons. The event schema remains storage
+neutral, so a later measured analytical-volume limit can justify a reviewed
+ClickHouse migration without changing producers.
+
+## Implementation increments
+
+1. Generate a local K1 run report from the evidence and diagnostic artifacts
+   already produced today; no runtime transport change.
+2. Emit the versioned append-only event contract directly during a run and seal
+   it with the evidence session.
+3. Tail the bounded outbox through the accepted ADR 0031 telemetry plane and
+   extend the normalizer with K1 run series.
+4. Add baseline comparison and a CI/replay gate. Physical acceptance remains a
+   separate operator-controlled UI test.
+
+No product window, status family or K1 control command is introduced by this
+decision.
+
+## Consequences
+
+- A stable K1 run becomes executable regression evidence rather than a prose
+  memory.
+- Camera, point-cloud, control and recovery events can be compared on one clock
+  and one run identity.
+- Wi-Fi or telemetry-plane loss cannot erase the local run or block the device.
+- The existing telemetry stack is extended instead of duplicated.
+- The first implementation work is an evidence/reporting increment; it does not
+  authorize broad recomposition of the fragile K1 communication path.

docs/lab/010_K1_MISSION_CORE_INTERNAL_LIVE_BASELINE_20260822.redacted.md

@@ -0,0 +1,213 @@
+# K1 ↔ Mission Core internal live baseline — 2026-08-22
+
+Status: accepted internal engineering reference
+
+Scope: one owner-controlled XGRIDS/LixelKity K1, exact firmware 3.0.2
+
+Connection: Mission Core Bridge/direct-LAN path
+
+Operator path: canonical Mission Core UI only
+
+## Classification
+
+This is the fast **Mission Core ↔ K1 internal reference**, not a vendor/LixelGO
+reference, not a production SLA and not proof for arbitrary Wi-Fi conditions,
+another K1, another firmware, Quick Connect, Direct Connect, LTE, a tunnel or a
+vehicle installation.
+
+The reference exists because the same visible regressions repeatedly returned:
+slow START/STOP, delayed point-cloud admission, a camera that disappeared and
+reopened, browser decoder errors and a discrete trajectory. It records both the
+fastest clean internal run and the accepted recovery-capable checkpoint that
+followed it. Raw MQTT, video and device evidence remain outside Git; only
+redacted summaries, hashes and engineering conclusions are retained here.
+
+No point count, image quality, bitrate, source FPS or K1 protocol setting was
+reduced to obtain either result.
+
+## Baseline A — fast ideal-path reference
+
+- Code: `eaad9de` (`fix(k1): stabilize live recovery and media admission`).
+- Evidence session: `20260822T105904Z_viewer_live`.
+- Local redacted manifest SHA-256:
+  `990b9433d40965b2b712df155f5f96d38770d53fddde804852c11eca90d96b3e`.
+- Local MQTT summary SHA-256:
+  `9438cc8947c7a868ff8ffcfc163233dfea40a95b7aae34050806bb072b2a7750`.
+- Local camera summary SHA-256:
+  `bc333e1dd1e7800204b242eecd01a83d1c91c388ab2202a238e8d2fc4a43aa14`.
+- Final phase: `idle`; stop reason: `external_stop`; rejected MQTT messages: 0.
+
+Operator-observed UI/physical timing:
+
+| Boundary | Observed time |
+| --- | ---: |
+| START click → K1 calibration onset | < 5 s |
+| K1 calibration | 21 s |
+| calibration end → visible point cloud | 2 s |
+| calibration end → visible right camera | 4 s |
+| STOP click → physical stop onset | 1 s |
+
+Recorded Mission Core facts:
+
+| Metric | Value |
+| --- | ---: |
+| PCL frames / pose frames | 121 / 132 |
+| points published | 412,175 |
+| point decode errors | 0 |
+| preview dropped | 0 |
+| MQTT callback → publish p50 / p95 | 23.839 / 41.541 ms |
+| camera archive | complete, 70 segments, 5,645,411 bytes |
+| camera failure code | none |
+| offline H.264 decode errors | 0 |
+
+This run used a fresh, short physical-command history: the terminal ledger was
+2,989 bytes at revision 15 with no archive segments. That distinction is
+material. The timings are a low-latency reference, not evidence that a mature
+recovery ledger can be cryptographically reread several times per UI poll for
+free.
+
+## Regression that exposed the shared bottleneck
+
+The recovery-capable working line later produced session
+`20260822T124524Z_viewer_live`:
+
+- START click → calibration onset: 26 s;
+- calibration end → cloud: about 2 s;
+- calibration end → camera: 19 s with repeated disappearance and large lag;
+- STOP click → physical stop onset: 12 s;
+- preview dropped: 393;
+- callback → publish p50/p95: 110.331/309.434 ms;
+- the retained fMP4 archive contained real H.264 macroblock/bytestream decode
+  errors, while the ideal reference did not.
+
+The camera UI resets were therefore a symptom, not the primary cause. Increasing
+preview queues or reducing media/point quality would only hide overload.
+
+Profiling found repeated full parsing, validation and archive-hash traversal of
+the physical START/STOP ledger inside passive `state.read`, nested public state
+projections and the first-PCL camera path. The mature ledger was 18,378 bytes at
+revision 1,219 plus five immutable archive segments / 318,027 bytes. One passive
+state request took 240–680 ms while the UI requested state four times per second.
+That contention delayed MQTT publication and FFmpeg stdout consumption together.
+
+## Baseline B — accepted recovery-capable checkpoint
+
+Evidence session: `20260822T130323Z_viewer_live`.
+
+- Local redacted manifest SHA-256:
+  `f870bedd9036a3f03993344d405eab4ef668d1f0fdae55e16adf8ecdcb51f48a`.
+- Local MQTT summary SHA-256:
+  `2d42d31c14d98d77bcefc8a1b9983e102cace2375ada9e5f89f021a33920a55d`.
+- Local camera summary SHA-256:
+  `1386ea16615a9716b699d6ba35733ef26d367b6a5257d572cde11509e546cdda`.
+- Final phase: `idle`; stop reason: `external_stop`; rejected MQTT messages: 0.
+
+Operator-observed UI/physical timing:
+
+| Boundary | Observed time |
+| --- | ---: |
+| START click → K1 calibration onset | 14 s |
+| K1 calibration | 22 s |
+| calibration end → visible point cloud | about 1 s |
+| calibration end → visible right camera | 8 s |
+| STOP click → physical stop onset | 7 s |
+
+Recorded Mission Core facts:
+
+| Metric | Value |
+| --- | ---: |
+| PCL frames / pose frames | 369 / 400 |
+| points published | 1,318,018 |
+| point decode errors | 0 |
+| preview dropped | 70 |
+| callback → publish p50 / p95 | 83.934 / 223.589 ms |
+| first-PCL camera admission | 65 ms |
+| camera authority / FFmpeg / post-spawn commit | 5 / 16 / 2 ms |
+| total backend camera activation | 23 ms |
+| backend activation → browser playing | 5.383 s |
+| browser camera restarts / slow-consumer retirements | 0 / 0 |
+| camera archive | complete, 342 segments, 27,469,046 bytes |
+| camera failure code | none |
+| offline H.264 decode errors | 0 |
+
+The accepted STOP was emitted once. Mission Core received an exactly correlated
+successful K1 application response after about 2.35 s, later observed fresh
+`READY` and resolved the durable physical command as
+`stop-standby-observed`. No automatic physical retry was introduced.
+
+After the bounded read-path change, passive `state.read` measured 23–43 ms on
+the same mature ledger instead of 240–680 ms.
+
+## What restored the usable path
+
+1. A single verified physical-ledger snapshot is shared across one public state
+   projection instead of being recursively reloaded by nested projections.
+2. Process-local control facts can be read without implicitly traversing the
+   durable physical ledger again.
+3. Passive snapshots reuse an already cryptographically verified ledger only
+   while the ledger directory, main file, archive directory and every bounded
+   archive entry retain the exact device/inode/mode/owner/link-count/size/
+   `mtime_ns`/`ctime_ns` fingerprint. Every mutation and proof still performs a
+   complete reload. Cross-process atomic writes and same-size archive tampering
+   invalidate the cache and fail closed.
+4. Camera authority is fully proved before FFmpeg `Popen`; after spawn only a
+   bounded in-memory compare-and-swap may commit the producer. Reader threads
+   therefore start without a second durable-ledger pause that can back up the
+   fMP4 pipe.
+5. First-authoritative-PCL camera admission no longer recursively rereads the
+   same physical ledger on the 10 Hz ingress callback.
+6. Structured timings were added for first-PCL admission, camera activation and
+   STOP dispatch. These are diagnostics only; they do not add a K1 command,
+   status, retry or product window.
+
+## Fragile boundaries that remain open
+
+- Baseline B is usable but does not equal Baseline A: 70 preview drops and
+  83.934/223.589 ms p50/p95 remain above the clean reference of 0 drops and
+  23.839/41.541 ms.
+- START and physical STOP onset remain materially slower than the fast reference.
+  The current evidence separates local validation, correlated application
+  response and later `READY`, but does not yet expose a device-side calibration
+  onset or physical motor/LED timestamp on the same clock.
+- Camera presentation still depends on RTSP startup, the next usable H.264
+  parameter-set/IDR boundary and browser MSE admission. The accepted archive has
+  non-monotonic-DTS warnings but no H.264 decode corruption.
+- A successful application response is necessary but not sufficient to dismiss
+  the interface. Mission Core must keep the active scene until authoritative K1
+  state proves the requested physical transition.
+- Wi-Fi loss, controller-process loss, K1 power/battery loss and return must
+  retain the current no-automatic-START/STOP rule and exact lineage. They need
+  separate UI acceptance runs; this baseline must not be used to infer them.
+- Bridge remains the accepted product path. Quick Connect/Bridge switching must
+  not reset the whole reactive application or reuse the other mode's reconnect
+  affordance, and must not perturb the Bridge protocol to improve Quick Connect.
+- The original LixelGO packet captures remain manufacturer-app evidence. They
+  are not interchangeable with this Mission Core internal baseline.
+
+## Regression rule
+
+Future K1 work starts from one named baseline and changes one bounded invariant.
+The canonical acceptance path is the Mission Core UI. A run is rejected if it
+introduces an archived-camera H.264 decode error, browser `MEDIA_ERR_DECODE`,
+missing point-cloud admission, missing terminal archive, lost control/acquisition
+lineage, duplicate physical command or a UI state not backed by authoritative K1
+evidence. Performance must not be recovered by silently reducing source data.
+
+## Delivery validation
+
+- Canonical UI physical run: accepted as Baseline B above.
+- Focused backend regression suite for the touched viewer diagnostics,
+  acquisition lifecycle, application session, camera gateway and physical
+  ledger: 800/800 passed.
+- Ruff: passed for every changed Python source and test file.
+- mypy: passed for all five changed source modules.
+- `git diff --check`: passed.
+- Offline decode of the accepted 50-second camera window: no H.264 decode
+  corruption; only non-monotonic-DTS warnings were observed.
+- Canonical Mission Core service remained available on `127.0.0.1:8000` after
+  validation.
+
+The next performance increment should be driven by a generated run comparison
+under ADR 0042. Kafka, Redis or ClickHouse are not a substitute for fixing the
+local control/media hot path and are not admitted without a measured requirement
+beyond the existing MQTT/Timescale telemetry contour.