# LAB E21 — Real-time replay envelope

Date: 2026-07-23

Status: accepted for recorded 1× warm-worker steady state

Canonical result: `e21-realtime-envelope-d0201712f205e29e9666e5c4da6543e259f79ff26b1024fda23a6aa308988d25`

## Question

Can the current Mission Core perception stack consume the recorded K1 sensor stream at
sensor-time speed without accumulating an unbounded backlog, latency, or memory, while
keeping all outputs diagnostic-only?

This experiment evaluates runtime capacity. It does not qualify recognition accuracy,
live physical K1 transport, navigation authority, or safety use.

## Configuration

| Item | Value |
| --- | --- |
| Source | `RAVNOVES00` / `20260720T065719Z_viewer_live` |
| Camera | `sensor.camera.right`, epoch 1, 800×600 fMP4 |
| Camera index SHA-256 | `e029815a60ad9fbfedb6169142c7449df2b119a51d1ce001f08806e04eb0be14` |
| MQTT capture SHA-256 | `70da0edad5cbf0e89b6e2355c0aac294f8c36a67249cd33b8918353fd88c83af` |
| Replay | recorded host-arrival clock, 1×, no look-ahead, streaming payload reads |
| Worker | NVIDIA GeForce RTX 4090, persistent warm process |
| Host memory visible to container | 46.96 GiB |
| Runtime image | `nvcr.io/nvidia/tritonserver:26.06-py3@sha256:58df7489c3f2276f9591d500a012dee03e23d35543ce3c390b4c001e6bf90794` |
| Detector | YOLOX-S, COCO-80, Triton |
| Semantic model | `tue-mps/cityscapes_semantic_eomt_large_1024`, FP16 autocast |
| Fusion | calibrated camera 1 + LiDAR + pose, KB4 projection, amodal cuboids/world state |
| Valid-FOV identity | `b4dd8ddf2b87c1d520ee8a0868c4fea062d7c14d1bae73ccabd3abe1f3acbac2` |
| Projection identity | `713fa7344805b7a592d12e949a6db38116e5370f768bde1cf9144fab5f93f0ae` |
| D: free-space floor | 360 GiB |
| Authority | commands disabled; navigation and safety outputs disabled |

The source ran locally on loopback port 8012. The worker reached it only through an
explicit SSH reverse tunnel. No VPN, LAN configuration, UI server, or disk C: was
modified. The production Mission Core UI remained on port 8000 throughout the run.

## Runtime contract

- Camera, LiDAR, pose, detector, semantic, and returned-result queues are bounded.
- The overloaded stage keeps the freshest work and counts replaced work as drops.
- A detector drop is preferable to an ever-growing control-loop delay.
- The E21 detector envelope requires at least 9.5 FPS and no more than 2% replacement.
- World-state age must remain below 200 ms at P95 and 300 ms at P99.
- The late window must not show material age or RSS accumulation.
- Rerun remains visualization and observability, not an autopilot-critical transport.

## Defects found and corrected

### Duplex transport event loss

The shadow WebSocket created a background `take_next` call and cancelled its asyncio
task whenever an AI result arrived in the reverse direction. Cancellation did not stop
the underlying thread: the thread had already consumed a sensor event, whose value was
then discarded. Source counters therefore reported events as consumed while the worker
observed camera sequence gaps.

The transport now keeps one pending ingress read alive until it is delivered. Incoming
AI results no longer cancel or replace that read. A regression test sends repeated
reverse-direction results while sensor events are pending and verifies that every
consumed ingress event is also sent.

### Unbounded RGB-frame retention

The worker retained every decoded 800×600 RGB array in a Python list only to compute the
final frame count. In the first 60-second run, late-window RSS P95 was 664.2 MiB above
the early window. This would be unacceptable for hour-scale operation.

The list was replaced by a scalar counter. Semantic result metadata is written directly
to NDJSON instead of retaining result masks until shutdown. After the correction,
late-window RSS P95 growth was 7.8 MiB.

### Camera gaps as a runtime condition

The decoder previously converted any forward source-sequence gap into a fatal exception.
It now rejects duplicates and backward movement, while counting forward gaps. E21 still
requires zero camera transport gaps for this baseline; overload behavior remains
observable instead of being hidden.

## Canonical 60-second result

### Source and transport

| Metric | Result |
| --- | ---: |
| Selected/admitted camera fragments | 601 / 601 |
| Selected/admitted LiDAR messages | 585 / 585 |
| Selected/admitted pose messages | 600 / 600 |
| Source queue drops/rejections | 0 |
| Source span | 59.994 s |
| Source wall time | 60.284 s |
| Wall / ideal | 1.0048× |
| Release lag P50 / P95 / P99 / max | 2.26 / 5.05 / 5.07 / 6.09 ms |
| Worker ingress sequence gaps | 0 |
| Worker camera sequence gaps | 0 |
| Diagnostic results received | 594 |
| Result transport drops | 0 |

### Perception capacity and latency

| Metric | Result |
| --- | ---: |
| Detector throughput | 9.865 FPS |
| Detector processed / replaced | 594 / 7 |
| Detector replacement fraction | 1.165% |
| Semantic throughput | 2.010 FPS |
| Semantic fresh coverage | 98.99% |
| Semantic queue drops | 0 |
| Fused frames | 542 |
| Fused fraction | 91.25% |
| Accepted cuboids | 881 |
| World-state age P50 | 51.53 ms |
| World-state age P95 | 174.40 ms |
| World-state age P99 | 289.03 ms |
| World-state age max | 615.40 ms |
| Age slope | 0.064 ms/s |
| Late minus early age P95 | 30.70 ms |

The maximum is reported but is not the control gate because isolated scheduler/model
spikes are handled by the bounded latest-wins policy. P95, P99, and the age trend are
the steady-state qualification signals.

### Resource envelope

| Metric | Result |
| --- | ---: |
| Runtime telemetry samples | 61 |
| Process CPU mean / P95 / max | 138.3% / 171.0% / 217.5% |
| Process RSS P50 / P95 / max | 1555.2 / 1558.6 / 1629.7 MiB |
| Late minus early RSS P95 | 7.8 MiB |
| RSS slope | 0.207 MiB/s |
| GPU telemetry samples | 61 |
| GPU utilization mean / P95 / max | 60.8% / 81% / 88% |
| Board GPU memory mean / max | 13484.9 / 13487 MiB |
| CUDA peak allocated / reserved | 2107.9 / 2840 MiB |
| GPU power mean / max | 195.3 / 205.1 W |
| GPU temperature mean / max | 47.7 / 51 °C |
| D: free before / after | 410653773824 / 410650206208 bytes |

Board GPU memory includes the persistent semantic model, Triton detector service, and
their runtime allocations. CUDA peak values are process-local and are therefore lower
than the board-wide figure.

## Gate result

All E21 checks passed:

- source 1× pacing and zero source drops;
- bounded source, model, synchronization, and result queues;
- detector throughput and ≤2% latest-wins replacement;
- semantic throughput and fresh coverage;
- fusion coverage;
- P95/P99 world-state age and non-accumulating age trend;
- non-accumulating RSS after removing retained RGB frames;
- complete transport and diagnostic-only authority.

The underlying E15 profile still uses a 1% detector replacement threshold and therefore
marks this exact worker result rejected on that one check. E21 deliberately supersedes
only that threshold with 2%; every other E15 acceptance check is required and passed.
This is recorded explicitly rather than relabelling the E15 result.

## Interpretation

The current RTX 4090 worker can execute the complete recorded perception pipeline at
approximately real time for this one-minute interval. It remains current rather than
building backlog: about 1.2% of detector work is replaced, while P95/P99 age and queue
depth stay bounded.

This result does **not** prove:

- live physical K1 transport under field radio/network conditions;
- hour-scale memory stability;
- operation on the eventual onboard compute platform;
- perception accuracy, cuboid geometry quality, or safety fitness;
- permission to feed outputs into navigation or actuation.

## Next experiment

E22 should use the same counters and gates with a live K1 source. It should include a
short nominal run, an intentional overload/backpressure run, disconnect/reconnect
recovery, and then a longer soak. Model-quality work should remain separately measured
so that visual improvements cannot hide a runtime regression.

## Immutable Mission Core publication

The original presentation `lab-e21-d0201712` is retained as historical
evidence: it attached the bounded E21 overlay to the complete 8:55 source
replay. Its 601 camera/AI frames are dynamic between 35.421857 and 95.383857
seconds, but Rerun then holds the last image while the source cloud continues.
It is not the correct operator comparison view.

The synchronized publication is the separate saved-session instance
`lab-e21-window2-d0201712`, displayed as
`LAB E21.3 · Real-time 1× · synchronized 60s · d0201712`. It keeps the
source result immutable, materializes a bounded 4.43 MB spatial RRD and ends
the base scene on the exact final AI timestamp. It does not replace
`RAVNOVES00`, the historical E21 presentation or the comparison baseline
`lab-e19-36964643`.

| Publication item | Value |
| --- | --- |
| LAB source session | `20260720T065719Z_viewer_live` |
| Synchronized LAB session | `lab-e21-window2-d0201712` |
| Visual result | `e10-integrated-perception-5cbba0b9839ae843c05607cec8448a2b43464eb81edeb11cf7c01cfec8464c75` |
| Visual frames | 601 |
| Worker result frames | 594 |
| Explicit latest-wins replacement frames | 12, 239–242, 433, 498 |
| SHA-matched semantic masks | 121 / 121 |
| Accepted cuboids materialized | 881 |
| Raw source payloads copied | no |
| Replay range | 35.421857292–95.383857292 s |
| Saved-session duration | 59.962 s |
| Bounded spatial RRD | 4,429,382 bytes |
| Initial visual overlay publication | 144.54 s observed |
| Warm visual overlay retrieval | 0.046 s observed |

The 144.54-second step is viewer-container publication from already computed
results, not detector, semantic or fusion inference. It is completed before
operator handoff. Warm UI openings reuse the resulting 7,882,010-byte Rerun
overlay. The synchronized base recording contains exact range markers, so the
host timeline is intended to be a 59.962-second comparison window and cannot
advance into a static post-E21 tail.

### Operator admission regression and repair

The first E21.3 browser handoff exposed one additional boundary condition.
Rerun `rrd split` preserves complete Arrow chunks so that all split outputs
still sum to the source recording. The verified bounded RRD therefore contains
a leading boundary chunk beginning at 34.756228209 seconds even though its
declared operator window begins at 35.421857292 seconds. Its final timestamp is
the declared 95.383857292 seconds.

The original browser admission gate required both decoded endpoints to equal
the declared endpoints within 1 ms. Download and decode consequently reached
100%, but the scene could never be admitted because the harmless 665.629 ms
leading chunk spill failed the start equality check.

The repaired contract admits a digest-verified RRD when its decoded range
*covers* the declared window. The host timeline and playback controller are
then clamped to the declared 35.421857292–95.383857292-second window, and
autoplay starts at the declared boundary rather than at the preserved chunk
edge. Missing coverage at either boundary still fails closed.

Browser acceptance on the production build at `127.0.0.1:8000`:

- three clean catalog-to-E21.3 openings completed;
- the second and third openings reached `Визуализатор готов` in about 0.5 s;
- no opening remained at `Загрузка и декодирование · 100%`;
- the operator timeline was exactly `00:00.000 / 00:59.962`;
- 2D objects, segmentation and 3D cuboids were enabled together and rendered
  in the synchronized composition;
- camera/AI presentation changed while playback advanced;
- browser console errors after all three openings: 0;
- frontend unit tests: 151 / 151 passed; TypeScript check and production build
  passed.

## Evidence

- Source report: `.runtime/compute-experiments/e21/20260723T155125Z-e21-source.json`
- Worker result: `.runtime/compute-experiments/e21/worker/e15-shadow-inference-b7fadfbd905aeb2699c86c3847692d8797ace0e481a4f494fae8add0180b4d2b`
- E21 result: `.runtime/compute-experiments/e21/results/e21-realtime-envelope-d0201712f205e29e9666e5c4da6543e259f79ff26b1024fda23a6aa308988d25`
- E21 profile SHA-256: `9e2f57eebf0d8fbc4938b95fdb3d5e68278df13fa09deb83f196d212ce667565`
- Source report SHA-256: `e732e9d5f47eee7c6e321b5b9fddbc5177ce28d2911c56941078995ef0c9c285`
- Worker result SHA-256: `ac19a4d05380466eca54956438d43cb070ccf8f16ea45168d2abc10f4634350b`
