docs(observation): document durable sessions and viewer lifecycle

docs/09_OBSERVATION_SESSIONS.md

@@ -0,0 +1,421 @@
+# Observation sessions, playback and workspace layout
+
+Status: implemented for native K1 point/pose evidence and host-side camera
+archival. Recorded spatial playback is exposed in the Mission Core observation
+workspace. The camera archive/player contract is implemented and covered by
+tests, but all retained real K1 sessions predate canonical camera archival.
+Physical archived-camera playback therefore remains an open acceptance gate;
+historical sessions contain no recoverable video.
+
+## Operator path
+
+1. Start a normal acquisition from **Парк → Локальное устройство**.
+2. Open **Наблюдение → Пространственная сцена**. Live point cloud, trajectory
+   and the selected camera remain live-only while acquisition is running.
+3. Stop acquisition normally, or allow the local service to recover an
+   unexpected interruption on its next start. Session recording is automatic;
+   the disk action is not required.
+4. Open **Сохранённые сессии** in the observation header. The menu shows the
+   three most recent indexed runs, their date, duration, state, modalities and
+   background preparation state.
+5. Choose a replayable run. Opening a replay never performs conversion in the
+   request. A ready recording opens immediately; otherwise the client receives
+   HTTP 202, keeps the current scene mounted and polls the preparation status.
+   Only after the server returns a verified launch descriptor does Mission Core
+   pause and unload the previous viewer.
+6. The client then receives and decodes the complete RRD. The new viewport and
+   timeline stay hidden until the declared recording range is fully buffered
+   (`fullyBuffered`). Playback starts once at that point. Partial frames are
+   never shown. If the run contains canonical camera archives, its recorded
+   camera sources replace all live device overlays.
+7. Use **Воспроизвести / Пауза**, the scrubber, **К началу** and **К концу** on
+   the right half of the bottom timeline. The left half changes point-cloud
+   accumulation. The recorded timeline is zero-based `session_time`.
+8. Use the first disk button to save the current workspace layout. The next
+   opening restores display settings, tool windows and dynamic source-window
+   positions. This action saves no sensor evidence.
+
+Display controls have no separate Apply/Reset transaction. Boolean controls
+commit immediately; sliders, colors and selects commit on release/blur or after
+a short quiet period, and concurrent commits collapse to the newest value. A
+layout save flushes and awaits this queue before serializing the confirmed
+settings. Recorded display updates reuse stable Rerun scene identifiers and do
+not mutate playback state, so changing accumulation, size, color or visibility
+does not pause the active replay or replace its camera view.
+
+Completed and recovered sessions are discovered on startup and by the catalog
+reconciler, then deduplicated into one bounded, single-worker preparation queue.
+The worker validates the native source, exports point/pose data, verifies source
+stability and both SHA-256 digests, finalizes once, and atomically publishes the
+digest-bound derived RRD plus its cache sidecar. Before the same job becomes
+`ready`, it also parses and validates every archived camera index and retains
+the immutable manifest generation in memory. The catalog exposes preparation
+state and progress while this runs; a recording URL and camera descriptors are
+returned only after the complete launch generation has passed validation.
+
+Preparation is a backend lifecycle, not a viewer action: sealing or recovering
+a native session makes it eligible for the reconciler, which builds the derived
+RRD once in the background. Reopening, seeking or changing the workspace reads
+that prepared artifact and never reruns conversion. A cold process may validate
+an existing cache and rebuild its in-memory camera manifest in the worker, but
+no replay, status, RRD or manifest request hashes native evidence or parses a
+camera index synchronously.
+
+The session menu uses three operator indicators:
+
+| Indicator | Catalog/preparation state | Meaning |
+| --- | --- | --- |
+| Solid green + `Готово` | `ready` | Verified RRD is available. |
+| Pulsing green + `Обработка` | `queued`, `validating`, `exporting`, `finalizing` | Background preparation is active. |
+| Dim gray + `Ошибка` | failed, cancelled, non-replayable or invalid session | No launchable recording; inspect the message or retry. |
+
+Saved-session status never uses yellow or red. Switching sessions aborts only
+the obsolete browser poll; it does not cancel the process-owned conversion.
+A queued job may legitimately wait behind another export and therefore uses the
+overall preparation deadline rather than the active-export heartbeat timeout.
+
+The first preparation of a long capture can take time because every decodable
+point/pose message is projected into RRD. Later openings reuse a verified,
+digest-bound cache. Derived RRD cache v6 is intentionally incompatible with v4
+and v5 because only v6 guarantees that the payload itself contains a real
+`session_time = 0` anchor; older generations are rebuilt once in the background.
+The browser may use the strict `source_url` with `If-Match`, while the embedded
+Rerun loader uses the canonical `viewer_source_url` whose lowercase SHA-256
+`generation` query is bound to the same launch descriptor. A missing or stale
+generation fails with `412`; a matching URL is served with exact length, strong
+ETag and private immutable/no-transform caching. Every cache pin is released when the
+HTTP response completes, disconnects or fails, so an interrupted switch cannot
+make a derived recording permanently non-evictable. A bounded 120-second launch
+reservation also protects the artifact between the verified launch response and
+the WebViewer's subsequent RRD request, including a cold WASM startup; it
+expires automatically if that request never arrives.
+
+## Storage roots
+
+By default, both catalog state and new source evidence are private to the
+checkout:
+
+```text
+.runtime/mission-core/
+├── mission-core.sqlite3
+├── mission-core.sqlite3-shm
+├── mission-core.sqlite3-wal
+├── evidence/
+│   └── sessions/
+│       ├── .current_session             # present only while a writer owns the root
+│       └── <UTC>_viewer_live/
+│           ├── captures/               # native MQTT source evidence
+│           └── media/                  # canonical camera archives
+└── recordings/
+    ├── .export.lock                     # cross-process conversion/eviction lock
+    └── <opaque-session-id>/
+        ├── scene.rrd
+        └── scene.rrd.cache.json
+```
+
+`MISSIONCORE_DATA_DIR` relocates the catalog and derived cache. Unless it is
+overridden separately, new evidence is written to
+`$MISSIONCORE_DATA_DIR/evidence/sessions`. `MISSIONCORE_EVIDENCE_DIR` can select
+another absolute MQTT evidence root. In the current K1 integration the camera
+gateway additionally confines its recording directory to the repository
+checkout, so a full point-plus-camera acquisition must keep the evidence root
+inside that checkout. An external evidence volume currently supports MQTT
+capture/cataloging but camera archival will fail closed until the gateway gets
+a separately attested storage root:
+
+```bash
+export MISSIONCORE_DATA_DIR=/absolute/private/path/mission-core
+# Full K1 point-plus-camera evidence must currently remain below the checkout.
+export MISSIONCORE_EVIDENCE_DIR=/absolute/path/to/NODEDC_MISSION_CORE/.runtime/mission-core/evidence/sessions
+export MISSIONCORE_RRD_CACHE_MAX_BYTES=8589934592
+export MISSIONCORE_RRD_FREE_SPACE_RESERVE_BYTES=2147483648
+uv run k1link serve
+```
+
+The directories and derived recording cache use owner-only permissions where
+the host filesystem permits them. A new acquisition writer is assigned only a
+direct child of the private evidence root; it no longer writes a new run to the
+repository-level `sessions/` directory.
+
+Repository-level `sessions/*_viewer_live` runs are a legacy, import-only source
+for the catalog. They are discovered and confined in place: refresh does not
+copy, move or migrate their large payloads, and no current writer is assigned
+that root. Startup may recovery-seal an already existing, incomplete canonical
+camera epoch there by preserving its valid segment prefix and writing an
+`interrupted` summary; this is evidence recovery, not a new acquisition write.
+
+Never add `.runtime/`, `sessions/`, raw captures, RRD files or camera media to
+Git. They can contain mapped interiors, trajectories and identifiable images.
+
+## Session source of record
+
+For the current K1 profile:
+
+```text
+MQTT callback
+  ├─ durable native .k1mqtt + aligned metadata  (source of record)
+  └─ bounded latest-wins Rerun live preview     (disposable)
+
+selected RTSP producer
+  ├─ durable init + fMP4 segments + JSONL index (camera evidence)
+  └─ bounded WebSocket/MSE preview              (disposable)
+```
+
+The live preview is intentionally allowed to drop frames under load. Native
+point/pose evidence and camera archive writes do not traverse that queue.
+
+Native `.k1mqtt` bytes with aligned metadata and canonical camera fMP4 archives
+are the evidence source of truth. The derived RRD contains every decodable point
+and pose frame from that source, but remains a rebuildable view rather than an
+evidence master. A cache entry is rebuilt only when native source
+identity/digests change or an incompatible derived-data export revision is
+introduced. A UI blueprint or workspace-layout revision never invalidates or
+rewrites the data RRD. Cache v4/v5 payloads are not reusable as v6 because they
+do not guarantee the real zero-time anchor.
+
+Expensive cache misses run through the bounded preparation worker and one global
+cross-process export gate to cap peak RAM, CPU and temporary-disk use. Crash
+leftovers from candidates, exporter temporary files and staged replay prefixes
+are scavenged under that lock before quota accounting. Ready cache hits and
+active response leases do not wait behind that gate. The derived cache has an
+8 GiB default quota, preserves a 2 GiB default filesystem reserve and evicts
+least-recently-used RRDs only; it never deletes native evidence.
+
+## Camera archive contract
+
+New acquisitions archive each selected source and codec epoch below the same
+private evidence session:
+
+```text
+<evidence-root>/sessions/<session-id>/media/<stable-source-id>/epoch-1/
+├── init.mp4
+├── segments/
+│   ├── 1.m4s
+│   └── ...
+├── index.jsonl
+└── summary.json
+```
+
+Each index row binds a segment sequence, byte length, SHA-256 and host arrival
+epoch/monotonic timestamps. Camera durability uses a
+`per-segment-fsync` policy: `init.mp4` and every complete media segment are
+individually fsynced and their directory entries synchronized before the
+matching JSONL index row is committed. The index and an atomic `interrupted`
+checkpoint summary are fsynced before the append returns. The old interval and
+byte constructor options are compatibility-only and cannot weaken this
+segment-bound RPO. A power failure may still lose or leave uncommitted the
+fragment currently being produced; it cannot make a committed index row point
+past durable media. Camera windows may close or reconnect without terminating
+archival while the owning acquisition remains active.
+
+Synchronization is `host-arrival-best-effort`: LiDAR/MQTT and camera segments
+share the Mac host clock boundary, but K1 sensor exposure time and LiDAR firing
+time are not proven to use a shared device clock. Do not infer frame-accurate
+calibration from the playback timeline.
+
+Finalized media is prepared once by the same process-owned background job that
+materializes the RRD. The gateway records host time only after a complete
+`moof+mdat` fragment has arrived, so that timestamp is an availability/end
+anchor, never a fragment-start timestamp. Preparation reads and SHA-verifies
+every fragment, parses bounded ISO-BMFF timing tables (`mdhd`, `trex`, `tfhd`,
+`trun`), anchors the epoch at
+`max(0, first_arrival - first_fragment_duration)`, and sets its end to that start
+plus the checked sum of every decoded fragment duration. The declared interval
+therefore has exactly the duration MSE is expected to expose and is not stretched
+by host scheduling jitter.
+
+Epoch arrivals must be strictly monotonic; epoch intervals must be finite,
+monotonic and non-overlapping. Replay v2 also requires every media interval to
+fit inside the spatial RRD interval with a 50 ms numeric tolerance; it fails
+preparation instead of clamping unreachable evidence. A future replay v3 must
+separate `spatial_range` from a session-wide union range before out-of-RRD camera
+coverage can be navigated. A missing, ambiguous, oversized or otherwise
+unparseable timing table fails preparation; the archive remains evidence but is
+never advertised as seekable media.
+
+The prepared path-free v2 descriptor and full source stat identity (native raw
+and timing metadata plus every camera summary, index, init and segment file) are
+written under the private derived cache with a schema, generation and checksum.
+Publication uses a private temporary file, file and directory `fsync`, and atomic
+rename; startup scavenges crash-left temporary files. A restart reuses this
+sidecar after confined O(n) stat validation, without rereading, hashing or parsing
+media. A missing, corrupt or stale sidecar is rebuilt only by the background
+worker. No replay, status, manifest or payload request performs conversion or
+recalculates these intervals.
+
+Every derived RRD contains a real `session_time = 0` row at the internal
+`/__mission_core/session_origin` entity. The anchor is deliberately outside
+`/world`, so it establishes the actual recording time range without creating a
+3D layer or drawable scene object. Preparation verifies the derived recording
+against the declared spatial range rather than relying on summary metadata
+alone.
+
+Sessions made before this archive contract have no recoverable video even if a
+camera preview was visible at the time. In particular, the 2026-07-16 browser
+camera acceptance run retained point/pose evidence only.
+
+## Crash and interruption behavior
+
+- SQLite uses WAL, foreign keys and full synchronous durability.
+- Before creating a session directory, the acquisition takes an exclusive,
+  cross-process lease by atomically creating and locking
+  `<evidence-root>/.current_session`. The marker contains only the direct child
+  session name. While the lock is held, a second writer fails closed and
+  discovery excludes that in-progress session from replay.
+- A process exit releases the operating-system lock. On the next service start,
+  stale-marker recovery removes the marker only after it can take the lock and
+  revalidate the marker inode without following symlinks. It never deletes the
+  interrupted session directory; normal catalog recovery then evaluates the
+  durable prefix. A locked/live or suspicious marker is left untouched.
+- Native MQTT writes use a bounded group commit: at most 0.5 seconds, 4 MiB or
+  32 messages per group. Raw bytes are fsynced before their metadata rows are
+  written and fsynced. This is a bounded RPO, not a zero-loss guarantee: a hard
+  process or power failure may discard the final uncommitted group. Raw bytes
+  from an interrupted commit window may survive beyond the durable metadata;
+  replay uses only the last validated metadata-aligned raw boundary.
+- A native capture with a missing final summary is accepted only when the raw
+  and metadata prefix is bounded, aligned and structurally valid. It is cataloged
+  as `interrupted`, never silently promoted to `ready`.
+- A non-newline metadata crash tail can be ignored. Newline-terminated or
+  mid-file corruption fails closed.
+- Camera recovery retains a contiguous valid segment prefix, quarantines
+  non-contiguous/orphan fragments instead of deleting them, rebuilds the index
+  when necessary and writes an `interrupted` summary. Unindexed bytes are not
+  presented as valid media.
+- If a valid normal summary appears later, repeat discovery updates the same
+  session to `ready`.
+- Materialization detects a native capture changing during export and refuses
+  to publish the derived RRD. It reopens the prepared raw source with no symlink
+  following inside the cataloged session roots, and interrupted recordings are
+  materialized from the last validated raw/metadata boundary only.
+- Internal preparation cancellation is cooperative. A queued job cancels
+  immediately; an active job stops at a safe checkpoint, removes its candidate
+  and never publishes a partial RRD. The operator UI does not cancel automatic
+  preparation when switching sessions or closing a tab: preparation belongs to
+  the application worker, not to an HTTP request.
+- `failed` and `cancelled` are terminal, retryable states. The API exposes a
+  sanitized error, and an explicit retry creates a fresh job for the current
+  source identity. A stalled poll or failed switch leaves the current scene
+  mounted; the latest operator selection wins over obsolete responses.
+- A lifecycle shutdown marks only its own interrupted work for automatic
+  reconciliation after restart. An operator cancellation and a genuine failed
+  export remain terminal and are never retried forever by the reconciler.
+
+These rules provide crash recovery, not replication. A single host disk failure
+can still destroy local data. Vehicle deployment must add independent onboard
+and control-station copies, capacity monitoring and a documented retention
+policy.
+
+## HTTP API
+
+All paths are same-origin and expose opaque identifiers only:
+
+```text
+GET  /api/v1/observation-sessions?limit=3
+GET  /api/v1/observation-sessions/{id}
+POST /api/v1/observation-sessions/{id}/replay
+GET  /api/v1/observation-sessions/{id}/recording-preparation
+DELETE /api/v1/observation-sessions/{id}/recording-preparation
+GET  /api/v1/observation-sessions/{id}/recording.rrd
+POST /api/v1/observation-sessions/{id}/blueprint.rrd
+GET  /api/v1/observation-sessions/{id}/media/{artifact}/manifest
+GET  /api/v1/observation-sessions/{id}/media/{artifact}/epochs/{n}/init.mp4
+GET  /api/v1/observation-sessions/{id}/media/{artifact}/epochs/{n}/segments/{m}.m4s
+
+GET  /api/v1/workspace-layouts/observation.spatial
+PUT  /api/v1/workspace-layouts/observation.spatial
+```
+
+The catalog embeds each replayable session's preparation state and progress.
+`POST .../replay` returns either a verified replay v2 launch document or HTTP
+202 with the preparation v1 document, `Location`, `Retry-After`, an exact quoted
+preparation `ETag` and a same-origin status URL. Polling
+`GET .../recording-preparation` sends that value as `If-Match` and returns HTTP
+202 while the job is active, HTTP 409 for retryable `failed`/`cancelled` states,
+and the launch document only when the same job and artifact are ready; every
+response echoes the same `ETag`. `DELETE` also requires `If-Match`, so a stale
+tab cannot cancel a replacement job. Conversion is never executed synchronously
+by a replay-open request. Playback speed and loop are request-local launch
+policy; they are not stored on or shared through a preparation job.
+
+Production `recording.rrd` access is bound to the launch generation. The client
+can use the strict query-free URL with exact strong
+`If-Match: "sha256:<launch.sha256>"`; a request with neither an `If-Match` nor a
+generation returns 428. The embedded Rerun receiver instead gets only the
+canonical `viewer_source_url = source_url + "?generation=<launch.sha256>"`.
+The server returns 412 for a malformed or replaced generation before opening
+the body. A successful response echoes the strong ETag, exact `Content-Length`,
+`application/vnd.rerun.rrd` and immutable private/no-transform cache policy.
+Rerun's native HTTP receiver owns incremental decoding; `LogChannel.send_rrd`
+is reserved for independently complete RRD payloads such as the small generated
+blueprint and must never receive arbitrary HTTP byte slices. The canvas,
+timeline, controller and autoplay remain closed until the decoded spatial range
+exactly matches the launch descriptor and every declared camera is ready. API
+documents never contain local paths. Layout updates require the quoted current
+revision in `If-Match`; stale writers receive HTTP 412 rather than overwriting
+another saved profile.
+
+Recorded media routes expose only opaque catalog identifiers and ordinal codec
+epochs. The required `missioncore.observation-recorded-media/v2` manifest carries
+a strong `generation_sha256`, exact JS-safe aggregate `byte_length`, and finite
+`timeline_start_seconds` / `timeline_end_seconds` for every epoch. Epoch ends
+participate in the generation digest. Every epoch also declares its init byte
+length/digest and a complete contiguous segment list with sequence, URL, byte
+length and digest. The launch source repeats the same aggregate `byte_length`
+and uses exactly `max(epoch.timeline_end_seconds)` as its end; the spatial RRD
+end must never pad camera coverage. The browser cross-checks launch, manifest and
+component lengths. The current browser laboratory policy accepts at most 16
+camera sources, 128 MiB per source and 512 MiB across the session. This launch
+preflight finishes before the first camera manifest/init/segment GET; admitted
+cameras are then downloaded and decode-probed one at a time. Verified raw
+buffers remain immutable so route changes and player remounts cannot append
+emptied data. This bounded in-memory strategy is deliberate for the laboratory
+milestone; an OPFS-backed sealed-generation cache is the next scaling step for
+larger rigs and must preserve the same launch/manifest/hash admission contract.
+
+The manifest response ETag is the exact generation, and the manifest GET itself
+requires that generation in `If-Match`. Init and media GETs likewise require the
+descriptor's exact SHA `If-Match`, then open through a confined directory file
+descriptor with no symlink following, verify the declared SHA-256 and serve
+exact-length bytes with immutable private `no-transform` caching and byte-range
+support. Physical source ids, RTSP addresses and storage paths never cross the
+API boundary.
+
+## Current limit
+
+The spatial RRD, trajectory and archived fMP4 cameras use the same operator
+scrubber now. Camera epochs are aligned to zero-based `session_time` from the
+shared host-arrival monotonic clock and rendered through MSE. The client selects
+an epoch only inside its declared inclusive interval and verifies that the
+decoded MSE seekable duration covers that interval. This remains best-effort
+correlation: codec PTS and K1 sensor exposure time are not proven to share the
+LiDAR clock. A codec epoch whose init segment does not expose a browser-supported
+codec or whose timing cannot be proven remains retained evidence and fails
+closed in the UI/background preparation.
+Historical sessions with no canonical camera archive honestly show no recorded
+video. Recorded blueprints can change accumulation, grid visibility, point and
+trajectory visibility, point radius and a uniform custom point color without
+rewriting the recording. The client deliberately has no progressive recorded
+mode: the viewport, timeline and autoplay gate remain closed until the complete
+declared RRD has been received and decoded. Height, intensity, distance and RGB
+palettes remain baked into current RRD rows; fully dynamic recoloring requires
+exporting the corresponding scalar components in a future recording schema.
+
+## Verification checkpoint — 2026-07-17
+
+The repository state described above passed the complete local pre-push gate:
+
+- `uv sync --frozen --group dev` completed against the locked Python graph;
+- `uv run pytest`: **284 passed**;
+- `uv run ruff check .`: clean;
+- project mypy and strict plugin-SDK mypy: clean across 51 and 12 source files;
+- the XGRIDS K1 profile loader and emitted v0alpha2 JSON schema validated;
+- clean `npm ci` ran the required Rerun 0.34.1 postinstall patch;
+- frontend unit suite: **111 passed, 0 failed**;
+- TypeScript project build and Vite production build completed;
+- the generated `dist/index.html` and non-empty Rerun WASM artifact were verified.
+
+Vite still reports its expected large-chunk warning for the embedded Rerun
+viewer/WASM payload. That is a packaging optimization item, not a failed gate.
+No retained physical K1 session contains the new canonical camera archive yet,
+so real recorded-camera playback remains an explicit hardware acceptance test.