# LAB E4 — Full-session EoMT segmentation playback

Date: 2026-07-21
Completed at: 2026-07-21T07:21:16.791Z
State: completed and published, model output, not ground truth
Ops card: MISSIONCOR-18

## Objective

Turn the plain EoMT valid-FOV configuration selected in LAB E3 into the first
complete, watchable semantic-segmentation result for a saved Mission Core
session. The acceptance target was deliberately narrow and observable:

1. process every recorded camera frame, without sampling;
2. preserve the exact saved-session timeline;
3. publish an H.264 semantic-overlay video and per-frame semantic masks through
   the existing recorded-perception contract;
4. make that video selectable and playable in the RAVNOVES00 saved-session UI;
5. measure full inference, publication, GPU, memory, power, temperature and disk
   behavior on the connected AI worker;
6. keep all task-controlled worker data on `D:\NDC_MISSIONCORE` and enforce a
   360 GiB free-space floor.

LAB E4 is semantic-only. It does not claim instance detection, tracking, 3D
cuboids, LiDAR association, metric distance, point-cloud segmentation or
safety-grade perception.

## Immutable input and provenance

- UI session alias: `RAVNOVES00`
- Session ID: `20260720T065719Z_viewer_live`
- Compute job: `recorded-camera-602ac89026ed12978619801d`
- Source: `sensor.camera.right`
- Factory calibration slot: `camera_1`
- Input SHA-256:
  `602ac89026ed12978619801d4edea0cae24b5cc3afabd9f7af2858de6505a20e`
- Factory calibration content SHA-256:
  `05f3ad9b38b3a4fc95388a8ec83da83c745e217709e51787b3d5aad0969f6fa9`
- Codec epoch: 1
- Resolution: 800x600
- Segments/frames admitted: 4,489/4,489
- Timeline: 35.421857292–484.144857292 session seconds
- Media duration: 448.723 seconds
- Approximate source rate: 10.0037 FPS
- Frame timeline SHA-256:
  `16098d7606e5462f1d6ccbc1b6e99158f68e2b2ca85ea11794799ef709b61c61`

The valid image circle is the immutable LAB E1 artifact
`valid-fov-mask-b4dd8ddf2b87c1d520ee8a0868c4fea062d7c14d1bae73ccabd3abe1f3acbac2`.
Its mask SHA-256 is
`a40cee06b7c6f69b6a09a11563dcfd237f3de833b1ccd31459e66692e528ba63`.
It admits 270,606 of 480,000 pixels. The mask is loaded once and reused; it is
not recomputed from every frame.

## Accepted configuration

LAB E4 deliberately promotes the LAB E3 speed/quality control, not the more
expensive five-view challenger:

- pipeline: `recorded-semantic-eomt-fisheye-mask/v1`;
- model: `tue-mps/cityscapes_semantic_eomt_large_1024`;
- architecture: `EomtForUniversalSegmentation`;
- pinned revision:
  `8d6b6d1a3f7b50d441afd7d247c2ed10db186e8f`;
- model weights: 1,276,175,488 bytes, SHA-256
  `c265da9a74f58f5c3f4826d23ca4ca78beac0b106cca5842beca61580de5b782`;
- config SHA-256:
  `7f4aa94fa4e43c0dbd79a5420edb511120aef62bd82bfbcbcece79948286a650`;
- preprocessor SHA-256:
  `97e2fbf7f0bdba2cfc90251c5133bae9c27ddc9c4410509f40670be2332854e7`;
- precision: FP16 autocast;
- batch size: 1;
- frame policy: all frames, no sampling;
- semantic-overlay alpha: 0.48;
- CLAHE: disabled;
- KB4 cubemap rectification: disabled;
- instance branch: disabled;
- container: existing `nvcr.io/nvidia/tritonserver:26.06-py3`;
- PyTorch: `2.13.0+cu130`;
- CUDA runtime: `13.0`;
- Transformers: `4.57.6`;
- NumPy: `1.26.4`;
- Pillow: `12.3.0`.

Producer identities:

- runner SHA-256:
  `651e8e06c3912dffb036b7fd08f2c0623f7563d8306cc7aee05db562798518f4`;
- orchestrator SHA-256:
  `c1c16fc351e7c8c4806fe2fd17a56e659495d843db5ff3e9ac3699a8d9722640`;
- profile SHA-256:
  `ea583966bc3409f5cf563cbf4fad05e366907e67187082eb692aff53d9f5d875`;
- dependency identity SHA-256:
  `4eb1f8d33236806e74f9e5bb96b7dce2ac37623dc39b2184be2aa8d7d00e983e`.

The Cityscapes outputs are mapped into the existing Mission Core 0–15
robotics taxonomy. Pixels outside the admitted FOV are category 0. Cityscapes
does not provide the required forest-specific `ground_dirt`, `animal` and
general `other_background` classes, so those mapped totals are zero. This is a
known taxonomy limitation, not evidence that those contents are absent.

## Execution topology

The work was sequential at frame level:

```text
recorded fMP4 epoch
  -> exact RGB frame reconstruction
  -> fixed calibration-bound valid-FOV fill
  -> EoMT semantic forward pass on RTX 4090
  -> resize/map semantic labels into Mission Core taxonomy
  -> force excluded FOV pixels to category 0
  -> write indexed semantic PNG and 48%-alpha overlay PNG
  -> append timestamped frame metadata
  -> encode overlays as H.264 MP4
  -> archive semantic masks
  -> hash, validate and publish result
```

The input stream and model loop were not split across parallel jobs. Batch size
was one. GPU work is internally asynchronous where PyTorch/CUDA permits, but
the next frame was not admitted until the current frame's semantic result and
artifacts had been produced. The final video was encoded on NVENC after the
overlay sequence was complete.

No software power, utilization or memory cap was applied. The main deliberate
limits were batch size one, one model, a 360 GiB D: free-space floor and a
conservative preflight working-set reserve.

## Pilot gate

An eight-frame non-publishable pilot ran first under
`pilot-8-4038b699764048579f9fb7c6886a0a10`.

- frames: 8/8;
- inference wall: 2.381609 seconds;
- throughput including frame artifacts: 3.359074 FPS;
- visual check: valid-FOV alignment, road, vegetation, vehicles, person and
  building overlays were usable;
- publication: prohibited by design.

The pilot exposed a report-only PowerShell parsing defect in a free-space
measurement: status text and the numeric value reached the same output stream.
The full-run orchestrator separated them before execution. The pilot data was
retained as evidence; no pilot result entered the saved-session result store.

## Full-run result

Published result:
`result-793785170472c519486ccd666be102fb04d169d92383acda3fcc29eecf045d30`.

| Measurement | Result |
|---|---:|
| Frames expected / processed | 4,489 / 4,489 |
| Frames failed / skipped | 0 / 0 |
| Inference wall | 1,447.565362 s (24:07.565) |
| Inference throughput | 3.101069 FPS |
| Extract/reconstruction phase | 31.837595 s |
| H.264 encoding | 3.374570 s |
| Mask archive | 1.332036 s |
| Full worker wall | 1,668.259063 s (27:48.259) |
| End-to-end throughput | 2.690829 FPS |

The source plays at approximately 10.0037 FPS. This offline E4 path is therefore
about 3.72 times slower than source real time. It is a complete playback
baseline, not yet a live-stream configuration.

The earlier LAB E0 full path achieved 1.678305 inference FPS and 1.593908
end-to-end FPS while executing both Mask R-CNN and BEiT. E4 is approximately
1.85 times faster on the measured inference loop and 1.69 times faster overall,
but the comparison is not like-for-like: E4 is semantic-only and intentionally
disables the instance branch.

### Per-frame latency

| Stage | Mean | p50 | p95 | Max |
|---|---:|---:|---:|---:|
| End to end | 322.458 ms | 320.654 ms | 348.467 ms | 441.481 ms |
| Model forward | 141.142 ms | 140.483 ms | 153.122 ms | 185.848 ms |
| Artifact write | 99.477 ms | 98.231 ms | 113.678 ms | 155.797 ms |
| Image decode | 30.677 ms | 28.915 ms | 42.952 ms | 82.429 ms |
| Processor | 10.656 ms | 9.597 ms | 15.529 ms | 85.791 ms |
| Host to device | 7.737 ms | 7.515 ms | 10.726 ms | 21.986 ms |
| Model postprocess | 10.214 ms | 10.421 ms | 14.078 ms | 25.200 ms |
| Overlay | 18.712 ms | 17.595 ms | 25.332 ms | 42.439 ms |
| Valid-FOV fill | 1.952 ms | 1.834 ms | 2.582 ms | 7.701 ms |

The measured stage means account for the frame loop. Approximately 184.15
seconds of the full worker wall remain outside extract, inference, encode and
archive. This includes full input/model hashing, model load, container/preflight
work, exact stream setup, result finalization, output hashing and validation.

## GPU, process and neighboring services

Worker GPU: NVIDIA GeForce RTX 4090, compute capability 8.9.

- one-second GPU samples: 1,448;
- GPU utilization: 73.05% mean, 73% p50, 89% p95, 95% max;
- board memory used, including pre-existing services: 13,397.65 MiB mean,
  13,409 MiB max;
- GPU memory-controller utilization: 21.41% mean, 32% p95, 35% max;
- power: 234.24 W mean, 232.91 W p50, 247.15 W p95, 253.23 W max;
- temperature: 52.94 C mean, 53 C p50, 57 C p95, 58 C max;
- process CUDA peak allocated: 2,099.8 MiB;
- process CUDA peak reserved: 2,840.0 MiB;
- process peak RSS: 2,009.332 MiB.

The board-memory figure is not the E4 process allocation. It is the total
device usage observed by `nvidia-smi`, including already running services. The
per-process CUDA peaks above are the relevant E4 allocation measurements.
`mission-core-triton`, `sentinel-frigate` and `sentinel-ollama` remained up.

## Disk policy and measured usage

The orchestrator admitted only explicit paths rooted at
`D:\NDC_MISSIONCORE`. It performed a conservative space preflight, sampled free
space every 100 frames and failed closed if the floor would be crossed.

| Disk point | Free bytes | Free GiB |
|---|---:|---:|
| Before run | 410,711,453,696 | 382.505 |
| After input extraction | 407,553,695,744 | 379.564 |
| After inference | 404,899,336,192 | 377.092 |
| After final artifacts | 404,796,903,424 | 376.996 |
| After task temporary cleanup | 407,372,025,856 | 379.395 |
| Enforced floor | 386,547,056,640 | 360.000 |

The conservative working-set reserve was 18,462,883,615 bytes (17.195 GiB).
The floor was never crossed. Final headroom above the floor was approximately
19.395 GiB. No task-controlled C: path was used, created or mounted. This does
not claim to observe incidental internal writes by Windows or the pre-existing
Docker Desktop engine.

Only exact E4 temporary extraction, mask and overlay work directories were
removed after successful publication. Immutable result artifacts, reports,
telemetry, the pilot and runner identities remain recoverable on D:.

## Published artifacts and integrity

| Artifact | Bytes | SHA-256 |
|---|---:|---|
| `perception.mp4` | 75,353,780 | `6e58cf72d9571ebbb003a814132b44907b7c0e31d18b5449cce330afcad29829` |
| `masks.tar.gz` | 39,812,627 | `ef291034e51c5c719dcc1c0902bae1e3daedf919d7b55a21fcb0a31af298071f` |
| `frames.jsonl` | 3,927,616 | `4780339f5ef94e1f58922dfe2df67ecdf983c4b4e030761f4a990f65b511d5cf` |
| `gpu-telemetry.jsonl` | 325,407 | `c6a4db1a8ea01d1ca66aded4bcd990ed91fc015930113869b20f024632a2aacf` |
| `run-report.json` | 7,894 | `8ee04bf2ed8e563641dc1dacd320aee6fc8cd4077a43fab564033a9c29cbaa07` |

The published local result occupies approximately 123 MiB. Independent checks
confirmed:

- all five artifact size/digest pairs match `result.json`;
- `frames.jsonl` contains exactly 4,489 ordered timestamp rows;
- the mask archive contains exactly 4,489 indexed masks;
- FFprobe reports H.264, 800x600, 448.723 seconds and exactly 4,489 frames;
- the complete recorded-perception v2 result validates against its job, input,
  session, source, timestamp bounds and `camera_1` calibration binding;
- the result store selects this result as the latest admitted generation for
  `recorded.perception.right`.

Durable local result root:

```text
.runtime/compute-results/recorded-camera-602ac89026ed12978619801d/
  result-793785170472c519486ccd666be102fb04d169d92383acda3fcc29eecf045d30/
```

Worker evidence root:

```text
D:\NDC_MISSIONCORE\runtime\derived\
  result-793785170472c519486ccd666be102fb04d169d92383acda3fcc29eecf045d30
```

## Mission Core publication and UI proof

The saved-session replay API returned schema
`missioncore.observation-session-replay/v2` with source
`recorded.perception.right`, label `Сегментация · камера right`, the exact result
ID and the MP4 generation SHA above. The media manifest validated as
`missioncore.observation-recorded-media/v3`. A byte-range request returned HTTP
206 with `Content-Range: bytes 0-1023/75353780`.

The final acceptance was performed in the real Mission Core browser UI:

1. open **Сохранённые сессии**;
2. select **RAVNOVES00**;
3. open the scene source picker;
4. select **Сегментация · камера right**;
5. press **Воспроизвести**.

The visible video element loaded the exact new result, reported 800x600,
duration 448.723 seconds, `readyState=4` and no media error. Its current time
advanced from 128.522 to 130.282 seconds during a timed browser check, proving
that the result is not merely discoverable but actually plays.

## Visual findings

Start, middle and end frames were inspected from the final encoded video.

1. The fixed valid-FOV circle remains aligned across the complete recording;
   excluded lens pixels do not generate semantic clutter.
2. Road, sidewalk, sky, buildings and vegetation form spatially coherent
   regions at the inspected start/middle/end points.
3. Cars and people are visibly segmented when present, but this is semantic
   class color, not a persistent object identity.
4. The output is substantially more usable than the original E0 visual result
   for playback, while still showing expected Cityscapes/domain artifacts on
   close structures, thin boundaries and forest/off-road content.
5. No visual inspection can substitute for the pending human-reviewed E2
   accuracy set. The result remains `ground_truth=false`.

Final contact sheet:
`.runtime/compute-experiments/e4/full-preview/start-mid-end.png`, SHA-256
`a16f78732346db4b476b7d66dd140857d92eb1f11add5826c49e4dddf0ac1d5f`.

## Acceptance status and limitations

Accepted:

- complete no-sampling processing of the chosen saved session;
- zero failed/skipped frames;
- immutable semantic masks and frame timestamps;
- exact H.264 playback on the original session timeline;
- source discovery, manifest and HTTP range transport;
- real UI selection and playback;
- measured worker resource and disk behavior;
- D-only task-controlled worker storage with enforced floor.

Not accepted or not attempted:

- live real-time throughput;
- instance masks, object IDs or tracking;
- 2D detector boxes from an E4 model branch;
- 3D cuboids or point-cloud semantic labels;
- LiDAR-camera association and distance-to-object accuracy;
- forest/off-road model fitness;
- mIoU, boundary IoU, AP or safety metrics;
- production or safety qualification.

## Decision and next gate

E4 closes the first full-session semantic playback milestone. Plain EoMT with
the immutable valid-FOV mask is now the operational reference for recorded
video: reproducible, watchable and measured.

The next work should not be another blind full-epoch run. It should proceed in
two bounded tracks:

1. finish human review of the E2 evaluation pack and measure semantic accuracy,
   especially people, vehicles, road/free-space boundaries and near distorted
   structures; use those metrics to decide whether to keep the raw fisheye
   baseline, improve the model/taxonomy or revive seam-aware rectification;
2. add an instance/detection and tracking branch, then project accepted 2D
   observations through the factory camera/LiDAR calibration into the point
   cloud for distance and 3D-cuboid experiments.

Live processing should be admitted only after the chosen branches are profiled
with bounded queues and an explicit drop/sampling policy. At the present 3.10
inference FPS, this exact configuration cannot consume the recorded ~10 FPS
camera stream without backlog.
