# LAB E8 — Source-paced RTX detector/tracker qualification

Date: 2026-07-21
State: completed; qualification and overload-control results accepted
Ground truth: no
Navigation/safety acceptance: no

## Purpose

LAB E8 answers one narrow but essential autonomous-perception question:

> Can the existing RTX worker consume the K1 right-camera stream at its recorded
> approximately 10 Hz rate, produce YOLOX detections and temporal tracks before
> the next source deadlines, and remain bounded if processing becomes slower than
> the source?

The lab deliberately removes synchronous overlay rendering and per-frame output
images from the inference critical path. It does not improve playback video in
isolation. It qualifies the detector/tracker branch that will feed the
near-real-time fused world state.

The answer is **yes for this recorded 60-second qualification interval**. The
worker processed 601/601 source frames with zero drops at 10.009945 effective
FPS. Result-age p95 was 93.158479 ms. A separate forced-overload run proved that
the queue remains bounded and discards superseded derived frames instead of
building an unbounded latency tail.

## Input identity

- Compute job: `recorded-camera-602ac89026ed12978619801d`
- Session: `RAVNOVES00`
- Source: `sensor.camera.right`
- Calibration slot: `camera_1`
- Calibration content SHA-256:
  `05f3ad9b38b3a4fc95388a8ec83da83c745e217709e51787b3d5aad0969f6fa9`
- Codec epoch: 1
- Resolution: 800x600
- Source selection: frames 1,000–1,600 inclusive
- Frames scheduled: 601
- Session timeline: 135.365857292–195.334857292 seconds
- Source span: 59.969 seconds
- Approximate source rate: 10.004 FPS

The interval is the same predeclared E5/E6/E7 qualification slice. It contains
parked and moving vehicles, people at several scales, a close woman/child/stroller
pass, fisheye distortion, occlusion and viewpoint change.

The valid-FOV mask is the immutable E1 result
`valid-fov-mask-b4dd8ddf2b87c1d520ee8a0868c4fea062d7c14d1bae73ccabd3abe1f3acbac2`.
Its mask SHA-256 is
`a40cee06b7c6f69b6a09a11563dcfd237f3de833b1ccd31459e66692e528ba63`.
It admits 270,606 of 480,000 pixels and is loaded once per run.

## Worker and neighboring services

Read-only preflight immediately before implementation confirmed:

- host alias: `mission-gpu`;
- Windows hostname: `<worker-host>`;
- GPU: NVIDIA GeForce RTX 4090, 24,564 MiB;
- driver: 610.47;
- worker free space at preflight: 407,939,772,416 bytes;
- `mission-core-triton`: healthy;
- `sentinel-frigate`: healthy;
- `sentinel-ollama`: running.

The qualification ran inside the existing
`nvcr.io/nvidia/tritonserver:26.06-py3` image and used the existing D-backed
Python environment. No task-controlled C: path was created, mounted or used.
All run inputs, temporary files, model data and immutable evidence remained
under `D:\NDC_MISSIONCORE`.

After all E8 runs and exact temporary cleanup:

- D: free bytes: 407,465,738,240 (approximately 379.37 GiB);
- enforced floor: 386,547,056,640 bytes (360 GiB);
- YOLOX returned to its pre-run unloaded state;
- Triton, Frigate and Ollama remained running.

## Accepted software configuration

Detector and tracker configuration are inherited without threshold changes from
accepted LAB E5:

- detector: official Megvii YOLOX-S ONNX release `0.1.1rc0`;
- model SHA-256:
  `c5c2d13e59ae883e6af3b45daea64af4833a4951c92d116ec270d9ddbe998063`;
- Triton config SHA-256:
  `5795c737a7935a655961b069e8404d336d891f9762fb6dffb93956a076479604`;
- Triton backend: ONNX Runtime GPU;
- input: FP32 `[1,3,640,640]`, BGR, bilinear top-left letterbox, fill 114;
- COCO classes admitted: person, bicycle, car, motorcycle, bus and truck;
- detector minimum score: 0.10;
- NMS IoU: 0.45;
- nested-box containment suppression: 0.80;
- ByteTrack-style, class-consistent, two-stage IoU association;
- high/new-track threshold: 0.25;
- primary/secondary match IoU: 0.20/0.10;
- lost-track buffer: 15 processed frames;
- minimum confirmation: two hits;
- no appearance ReID and no camera-motion compensation.

Real-time contract:

- source pacing: exact recorded session timestamps at 1.0x;
- queue: bounded latest-wins;
- capacity: two decoded frames;
- overflow action: discard the oldest unprocessed derived frame;
- stale threshold: 150 ms from scheduled source time;
- unavailable threshold: 500 ms;
- synchronous overlays and video encoding: disabled;
- output in the critical path: compact frame and telemetry JSONL only.

Accepted producer identities:

- runner SHA-256:
  `3eebf55479151788467af804c551b11660a551364c29b128a78e3ca7de36b2b2`;
- orchestrator SHA-256:
  `a98990024a719b2ac1f0b75c0aea93d9ec18341be653f44594afff391b700f76`;
- qualification profile SHA-256:
  `819bdfb0da521d187ec9ce3d3039a0ff75daa0ab473fb3d8e5fa92db9c1ddca1`;
- overload profile SHA-256:
  `ac2ca648b46857953baa1bd6f07254bcc680c70566ed32d48b7431bf428bf757`;
- independent result validator SHA-256:
  `32eb8f1d56b80ceac1a63b850767f3f7ccb1aea14b530412199f5f48f1c7be20`.

## Execution topology

```text
recorded fMP4 camera epoch
  -> temporary exact PNG reconstruction on D:
  -> producer waits for each recorded source timestamp
  -> decode RGB + fixed valid-FOV fill
  -> bounded latest-wins queue (capacity 2)
  -> YOLOX-S Triton request
  -> YOLOX decode, FOV/class/NMS filtering
  -> ByteTrack-style temporal association
  -> compact timestamped result + freshness telemetry
```

The producer and inference consumer run independently. The producer never waits
for an unbounded queue. If the queue is full, the oldest unprocessed derived
frame is replaced by the new frame. Raw source preservation is outside this
drop policy; only recomputable perception preview work may be dropped.

The measured `result_age_ms` begins at the scheduled source time and ends after
detection, association and result construction. It therefore includes source
release lag, PNG decode, queue wait, preprocessing, Triton, postprocessing and
tracking. It does not include the one-time job validation, model load, fMP4
reconstruction or PNG extraction that occurs before the paced interval.

## Pilot gate

Pilot result:
`e8-realtime-tracking-ba7f73c8f4e0bde1634cce89d7fda5c94c43d0b55af8080a1eeec8af636474ba`.

- selection: source frames 1,230–1,293;
- source span: 6.296 seconds;
- processed: 64/64;
- dropped: 0;
- effective rate: 10.052679 FPS;
- queue maximum depth: 1/2;
- result-age p95: 78.527889 ms;
- health: 64 healthy;
- accepted: yes.

The pilot proved the model, queue, timing basis, disk guard and publication
contract before the full interval was admitted.

## Full qualification result

Published result:
`e8-realtime-tracking-2802f407cb9d2c24576a4ba32d3e35f4d2b2092bfa99b37c438a8ac8120e1b16`.

| Measurement | Result |
|---|---:|
| Frames scheduled / processed | 601 / 601 |
| Frames dropped / failed | 0 / 0 |
| Source span | 59.969 s |
| Paced replay wall | 60.040290 s |
| Effective output rate | 10.009945 FPS |
| Queue capacity / maximum depth | 2 / 1 |
| Healthy results | 601 |
| Stale / unavailable results | 0 / 0 |
| Detections | 4,049 |
| Unique confirmed track IDs | 167 |

All qualification checks passed:

- minimum 9.5 effective FPS;
- maximum 0.5% drop fraction;
- result-age p95 at or below 100 ms;
- zero failures;
- exact producer/consumer/drop accounting;
- queue depth never above its declared capacity.

### Critical-path latency

| Stage | Mean | p50 | p95 | Max |
|---|---:|---:|---:|---:|
| Result age from scheduled source time | 66.719 ms | 62.533 ms | 93.158 ms | 145.973 ms |
| Image decode | 32.693 ms | 30.396 ms | 48.431 ms | 70.110 ms |
| Source release lag | 1.865 ms | 0.088 ms | 21.608 ms | 46.695 ms |
| Queue wait | 0.090 ms | 0.064 ms | 0.121 ms | 8.993 ms |
| Processing after dequeue | 32.070 ms | 30.770 ms | 43.526 ms | 83.312 ms |
| Preprocess | 7.045 ms | 6.926 ms | 9.574 ms | 20.348 ms |
| Triton request | 12.223 ms | 11.684 ms | 18.589 ms | 36.336 ms |
| Detector postprocess | 8.813 ms | 8.729 ms | 15.167 ms | 28.547 ms |
| Tracking | 0.305 ms | 0.253 ms | 0.638 ms | 2.210 ms |

The maximum result age crossed the 100 ms nominal frame period on isolated
frames, but p95 remained below 100 ms and no queue accumulation or drop
occurred. The current performance margin is usable for this 10 Hz source but is
not large enough to add the E4 semantic model synchronously to every frame.

### GPU telemetry

The 61 one-second samples during the paced interval measured total board state,
including neighboring services:

- GPU utilization: 52.08% mean, 57% p95, 63% max;
- board memory used: 10,240.57 MiB mean, 10,250 MiB max;
- power: 144.57 W mean, 147.49 W p95, 150.23 W max;
- temperature: 42.95 C mean, 44 C max.

These are board totals, not per-process CUDA allocation claims.

## Forced-overload negative control

Published result:
`e8-realtime-tracking-93ab963d9e4f25a93b60148a9a512e5ac7e1df3cdacec60ab105435b344fcf61`.

The same 64-frame pilot interval was replayed at 1.0x with an intentional
140 ms consumer delay after every processed frame. This makes the consumer
slower than the approximately 100 ms source period.

| Measurement | Result |
|---|---:|
| Frames published by source | 64 |
| Frames processed | 40 |
| Superseded frames dropped | 24 (37.5%) |
| Effective output rate | 5.895414 FPS |
| Queue capacity / maximum depth | 2 / 2 |
| Result-age p95 | 442.345170 ms |
| Health | 40 stale |

This is the intended failure behavior: derived perception loses temporal
resolution and declares staleness, but memory/latency does not grow without a
bound. The result is accepted only as an overload-control proof, not as an
operational performance result.

## Comparison with LAB E5

LAB E5 processed the same 601 frames sequentially while writing a PNG overlay
for every frame. Its inference/tracking/artifact loop achieved 6.822348 FPS and
had 146.279 ms mean end-to-end time. The synchronous overlay write alone cost
87.363 ms mean.

LAB E8 removes that visualization work from the inference critical path:

- 10.009945 FPS instead of 6.822348 FPS;
- 32.070 ms mean dequeue-to-result processing;
- 12.223 ms mean Triton request;
- 0.305 ms mean tracking;
- no source frame drops in the 60-second qualification.

This proves that LAB E5's offline rate was not the detector/tracker ceiling.
The remaining largest per-frame cost in E8 is temporary PNG decode. Direct
camera/video ingest can reduce it later, but it is not required to meet the
present 10 Hz gate.

## Disk policy and measured usage

The full qualification reserved a conservative 3.273 GiB working set and
failed closed if the 360 GiB D: floor would be crossed.

| Disk point | Free bytes | Free GiB |
|---|---:|---:|
| Before qualification | 407,538,298,880 | 379.550 |
| After temporary frame extraction | 406,955,905,024 | 379.007 |
| After paced replay | 406,868,586,496 | 378.926 |
| After immutable publication, before temp cleanup | 406,868,582,400 | 378.926 |
| After all E8 runs and exact temp cleanup | 407,465,738,240 | 379.370 |
| Enforced floor | 386,547,056,640 | 360.000 |

No unrelated data was removed. The immutable pilot, qualification, overload
control and staged runner bundles remain on D: as evidence.

## Artifact integrity

The full result contains:

| Artifact | Bytes | SHA-256 |
|---|---:|---|
| `frames.jsonl` | 1,156,936 | `df5fe55f52df887c8d04e1ed3937f0faf5b08d379b6f41e45638016f1bd2a479` |
| `telemetry.jsonl` | 145,235 | `8d904465fd8a352d59a62bb7ecdeebdc0c5ea07b1700c95136e14a197c617182` |
| `gpu-telemetry.jsonl` | 13,652 | `6d87dcf9944831ad5bd9bb1c88381ac940ff30d175475068891893de1e7dcdb4` |
| `run-report.json` | 9,478 | `6fb136fa54ebf7d655b559da3218aacab1aa5838308d5ed83211081857f8e2f5` |

Independent host validation confirmed for pilot, full qualification and
overload control:

- result directory equals the canonical identity SHA-256;
- every declared artifact size and SHA-256 matches;
- result identity binds job, input, session, source, selection, calibration,
  model, profile, runner and orchestrator;
- all frame and telemetry rows are strictly ordered and mutually bound;
- full qualification contains all 601 source indices;
- overload-control gaps exactly account for the 24 latest-wins drops;
- all detection and track boxes are finite and confined to 800x600;
- queue producer, consumer and drop totals are exact;
- all three immutable results pass the dedicated E8 validator.

Durable local evidence root:

```text
.runtime/compute-experiments/e8/worker-results/
```

Worker evidence roots:

```text
D:\NDC_MISSIONCORE\runtime\derived\
  e8-realtime-tracking-ba7f73c8f4e0bde1634cce89d7fda5c94c43d0b55af8080a1eeec8af636474ba
  e8-realtime-tracking-2802f407cb9d2c24576a4ba32d3e35f4d2b2092bfa99b37c438a8ac8120e1b16
  e8-realtime-tracking-93ab963d9e4f25a93b60148a9a512e5ac7e1df3cdacec60ab105435b344fcf61
```

## What is accepted

- actual YOLOX-S inference on the RTX worker at the recorded 10 Hz source rate;
- temporal tracking in the same source-paced critical path;
- fixed calibration-bound valid-FOV preprocessing;
- 601/601 processed frames, zero drops and p95 below 100 ms;
- explicit health/freshness state;
- bounded latest-wins overload behavior;
- D-only task-controlled storage and exact temporary cleanup;
- restoration of the pre-run Triton model state;
- immutable, content-addressed evidence and independent validation.

## What is not accepted

- direct live K1 network/camera transport;
- real-time semantic segmentation;
- live LiDAR association, 3D cuboids or metric range in this worker run;
- appearance ReID or camera-motion compensation;
- human-reviewed accuracy, IDF1/HOTA/MOTA or safety metrics;
- navigation or actuation use;
- production transport between the worker and Mission Core;
- evidence that this exact margin holds for arbitrarily long sessions.

## Next gate

The next experiment must preserve this accepted detector/tracker path and add
semantics as a separate lower-rate asynchronous branch:

1. keep detector/tracker at the 10 Hz source cadence;
2. schedule semantic inference at a lower rate with a capacity-one latest-wins
   queue;
3. attach each semantic result to its exact source timestamp;
4. reuse a semantic result only within an explicit TTL and expose its age;
5. never block detector/tracker or raw recording on semantic completion;
6. measure GPU contention against the accepted E8 baseline;
7. merge fresh tracking, semantics and existing E6 LiDAR geometry into the E7
   world-state contract;
8. qualify overload and semantic-worker loss before any live K1 trial.

Ops publication remains pending because this Codex session did not expose the
required direct `nodedc-ops-agent` tools. No legacy Ops path was used.
