feat(perception): integrate calibrated operator pipeline

Add calibrated K1 projection, recorded and near-live perception qualification, unified Rerun operator layers, bounded replay admission, audited viewer controls, worker experiments, and lab evidence.
2026-07-23 00:23:28 +03:00
parent ada2a55ee6
commit b53d6d5a45
221 changed files with 55923 additions and 1357 deletions
@@ -61,6 +61,306 @@ The result manifest binds the detection artifact's length and SHA-256. A repeat
 with the same identity validates the existing directory and returns it without
 calling Triton.

+## Full-epoch panoptic result v2
+
+`missioncore.recorded-perception-result/v2` is the complete recorded-camera
+profile. It is separate from the YOLOX detector proof and processes every
+admitted frame without sampling. Its immutable identity binds the complete
+compute job, factory-calibration generation and camera slot, exact model
+revisions/weight files, thresholds, alpha values, runner SHA-256 and publication
+encoder.
+
+The current research configuration produces:
+
+- TorchVision Mask R-CNN ResNet50-FPN v2 instance masks and COCO labels;
+- Microsoft BEiT ADE20K-150 semantic masks at the original 800x600 frame size;
+- a seekable H.264 MP4 with the two overlays combined for operator playback;
+- lossless instance and semantic mask PNGs for later calibrated fusion;
+- ordered per-frame JSONL and one-second GPU telemetry;
+- a machine-readable run report with input/config/model identities, decode,
+  inference, encode and end-to-end timing, latency percentiles, throughput,
+  CUDA peak allocation/reservation, process peak RSS, system load and GPU
+  utilization/VRAM/temperature/power samples.
+
+The worker runs with `--network none` and cached model weights. A preflight
+revalidates the complete transferred payload, CUDA execution and both cached
+model generations before decoding a long epoch. The run then repeats payload
+validation inside the inference container, requires exactly the declared frame
+count and a strictly increasing session-time row for every frame, and publishes
+only by atomic rename after every output digest is sealed.
+
+There is no recorded-duration, frame-count or aggregate-video-byte admission
+ceiling in this profile. Resource use therefore scales with the real input and
+is reported, not hidden behind an arbitrary eight-minute laboratory limit.
+
+## Native panoptic playback
+
+Full raster masks are not copied into the RRD. That would turn a long video into
+a multi-gigabyte browser-memory object. Instead Mission Core validates the v2
+result and exposes its MP4 through the same generation-bound, seekable HTTP
+Range contract as a recorded camera. Replay advertises an additional opaque
+source such as `recorded.perception.right`; the Control Station opens it in a
+native video window on the shared `session_time` timeline. A one-, three- or
+ten-hour video remains disk/range streamed and does not have to fit in RAM.
+
+The lossless masks remain private derived evidence. A host-side calibrated
+fusion step samples them at K1 KB4 LiDAR projections and publishes compact
+semantic `Points3D`, support-gated `Boxes3D` and diagnostic distances as a
+separate replaceable generation. Missing or rejected v2/fusion results never
+replace or invalidate the base raw point-cloud recording.
+
+## RAVNOVES00 qualification · 2026-07-20
+
+The first full recorded run admitted all 4,489 frames from
+`sensor.camera.right` without sampling, failures or skips. The sealed input was
+363,235,615 bytes over `35.421857292–484.144857292` session seconds. Its
+immutable result is
+`result-f4cebdea8a82698a5b8a65d2c3fbdb0428b88b9dc49fe45f8cb37d740ed83d02`.
+
+Measured RTX 4090 worker results:
+
+- inference: 2,674.722 s and 1.678 frames/s;
+- end to end: 2,816.349 s and 1.594 frames/s;
+- instance latency: 89.826 ms p50, 121.344 ms p95, 915.325 ms max;
+- semantic latency: 249.678 ms p50, 284.826 ms p95, 406.594 ms max;
+- GPU utilization: 67% p50, 82% p95, 90% max over 2,675 one-second samples;
+- process CUDA peak: 2,230.8 MiB allocated and 2,872 MiB reserved;
+- total GPU memory observed, including the worker's shared resident services:
+  13,373 MiB p50 and 13,388 MiB max;
+- GPU power/temperature: 182.46 W p50, 190.10 W p95 and 49 C p50, 54 C max;
+- process peak RSS: 2,375.9 MiB;
+- publication: 31.229 s decode, 3.397 s NVENC, 81,109,627-byte H.264 MP4,
+  60,326,719-byte lossless mask archive.
+
+The factory-calibrated full fusion generation
+`fusion-0b1be23128ebd3d230562cffd96491169e99f0e839e56b812661c974e4fdc00b`
+matched LiDAR and pose within the admitted 250 ms host-arrival window for 4,323
+frames and declared 166 frames `depth-unavailable`. It produced 6,124,145
+semantic points and 13,496 support-gated diagnostic boxes in 77.708 s. The
+compact fusion payload is 43 MiB.
+
+Native browser QA opened `RAVNOVES00`, played the raw and panoptic 800x600
+videos together at ready-state 4, and measured approximately 12 ms between
+their media clocks. The Rerun scene showed the synchronized semantic points and
+distance-labeled diagnostic boxes. The current baseline is deliberately not an
+accuracy or safety acceptance: generic perspective-trained models produce
+large fisheye false positives in 916 frames, and timing/distance have not been
+ground-truthed. The next A/B should compare an admitted undistort/ROI transform
+before inference rather than silently hiding these observations.
+
+## E1 valid-FOV preprocessing qualification · 2026-07-20
+
+The first post-baseline A/B uses two immutable inputs derived from the same
+RAVNOVES00 job and factory calibration:
+
+- valid-FOV generation
+  `valid-fov-mask-b4dd8ddf2b87c1d520ee8a0868c4fea062d7c14d1bae73ccabd3abe1f3acbac2`;
+- qualification-slice generation
+  `qualification-slice-2394070b4f3e38f1b8c483e878fcc11fd3c29f751f3d4cd7e3a2553304c5c142`.
+
+The mask is not estimated from each image. It is bound to the exact calibration
+SHA, `sensor.camera.right`, `camera_1`, the admitted 800x600 linear-resize
+profile and the KB4 principal point `(396.319, 301.496)`. A four-pixel inner
+margin produces a 293.504-pixel radius, 270,606 valid pixels (56.37625%) and an
+exclusive crop rectangle `[103, 8, 690, 595]`. Repeated preparation reuses the
+same content-addressed PNG and manifest.
+
+The slice selects 256 exact frame indices uniformly across all 4,489 frames,
+including both endpoints. The same loaded FP32 Mask R-CNN and BEiT generations
+were interleaved per frame across three variants: unmodified baseline, fixed
+valid-FOV fill, and valid-FOV crop remapped into the original pixel coordinates.
+The sealed result is
+`qualification-result-98a2fee3d22979f3e18847719667cc76bdeacf25904c0fd1da4c5202253b3940`.
+
+The run completed in 318.969 s and emitted 12 preview frames per variant. GPU
+telemetry recorded 319 one-second samples: utilization was 74% p50 and 82% p95,
+power was 199.34 W p50 and 206.96 W p95, and temperature was 50 C p50 and 55 C
+max. The qualification process peaked at 1,849.9 MiB CUDA allocated, 2,256 MiB
+reserved and 2,298.2 MiB RSS.
+
+Measured mean model paths, excluding the frame decode shared by all variants:
+
+| Variant | Mask R-CNN path | BEiT path | Combined |
+|---|---:|---:|---:|
+| baseline | 106.292 ms | 278.064 ms | 384.356 ms |
+| valid-FOV fill | 107.027 ms | 280.762 ms | 387.789 ms |
+| valid-FOV crop | 104.055 ms | 282.840 ms | 386.895 ms |
+
+The crop reduced Mask R-CNN forward time by 6.38%, but BEiT still receives its
+fixed 640x640 tensor and became 1.18% slower. With mask/crop preprocessing
+included, neither variant improved the combined path; fill was 0.89% slower and
+crop was 0.66% slower than baseline. A binary mask improves admission quality,
+but multiplying an unchanged tensor by it does not remove dense neural FLOPs.
+
+The quality proxies are useful but are not ground truth. Baseline produced 58
+instance masks and 58 boxes larger than half the admitted comparison area; both
+masked variants produced zero. The fraction of raw predicted instance-mask
+pixels outside the canonical FOV fell from 41.263% to 0.077% for fill and 0.911%
+for crop before the final output clamp. Inside the valid circle, mean BEiT
+disagreement with baseline was 8.713% for fill and 11.547% for crop. This is a
+measure of change, not accuracy.
+
+E1 therefore accepts the immutable valid-FOV artifact and the 256-frame gate.
+Fixed fill is the conservative next accuracy baseline because it preserves the
+800x600 geometry, removes the exterior lens region and changes the semantic
+result less than crop. Crop remains an experimental model-specific option, not
+a general speed optimization. The next run needs human labels/ground truth and
+must compare native KB4 input, calibrated virtual views and fisheye-trained
+models before promoting any preprocessing profile to a full-epoch result.
+
+## E2 evaluation pack and annotation gate · 2026-07-20
+
+LAB E2 starts from the same immutable RAVNOVES00 compute job, E1 qualification
+slice and factory-calibrated valid-FOV generation. All eight contact sheets,
+covering the 256 uniformly distributed E1 candidates, were reviewed before
+selection. The sealed evaluation generation is
+`evaluation-pack-7a983bba75d46c7c260252cb2d461e1384dcb92cda9e164397e841e6ebb37789`.
+
+The pack contains 64 exact 800x600 images: 48 reviewed full-epoch anchors from
+the E1 slice and four four-frame consecutive clips for temporal measurements.
+The clips cover a person with a stroller, a close moving car, vehicle
+occlusion/relative motion and a near building/terrace scene with a partially
+visible carried laptop. The anchors retain the recording's road, sidewalk,
+ground, grass, woody vegetation, buildings, sky, people, cars, trucks, lens
+boundary and hard-negative diversity.
+
+Every image is stored both as the raw decoded RGB frame and as the accepted E1
+fixed-valid-FOV-fill input. The identity binds the job/input SHA, source, codec
+epoch, selected segment SHA, decoded session timestamp, E1 qualification,
+valid-FOV generation, calibration SHA, camera slot, FFmpeg 7.1.1 generation and
+the raw/fill RGB pixel hashes. It also binds the reviewed selection-document
+SHA and both producer-code hashes. The pack has 130 hashed payload artifacts
+plus its manifest and occupies approximately 67 MiB locally.
+
+One earlier local preparation generation, `evaluation-pack-b6d9215a…`, was not
+promoted because its identity omitted the producer-code and selection-document
+hashes. It remains a superseded diagnostic artifact and is not an accepted E2
+input.
+
+The immutable pack is deliberately `unannotated`. Its annotation contract
+defines 15 robotics-oriented thing/stuff classes, label 0 for the excluded lens
+exterior, label 255 for genuinely unresolved pixels, two-pass human review and
+required semantic, instance, safety-proxy and temporal metrics. The empty
+annotation template must be copied to a review workspace; it must never be
+edited inside the sealed pack. Model-generated prelabels may accelerate review
+but are not accepted as ground truth without a human pass.
+
+No AP, mIoU or model-ranking claim is attached to E2 yet. The next gate is to
+complete and seal the reviewed annotations. Only then may candidate models be
+ranked on this pack; a full 4,489-frame run remains prohibited until one
+configuration passes both the accuracy and throughput gates.
+
+The first model-assisted draft is sealed separately as
+`evaluation-prelabels-4ba26bbf6eb8a49631f5caf984267e0445958540aeda2b5b0d82ca6440835cf1`.
+It reuses the exact E0 Mask R-CNN and BEiT weights and maps their COCO/ADE
+classes into the E2 taxonomy. It is explicitly marked
+`unreviewed-model-draft`; it never mutates the evaluation pack or annotation
+template.
+
+The isolated RTX 4090 run processed 64/64 frames in 31.551 s. Mean forward time
+was 57.539 ms for Mask R-CNN, 215.164 ms for BEiT and 272.703 ms combined. The
+process peaked at 1,842.8 MiB CUDA allocated, 2,768 MiB reserved and 2,239.9 MiB
+RSS. Across 32 one-second samples, GPU utilization was 51% p50 / 71% p95, power
+167.73 W p50 / 186.89 W p95 and temperature 42 C p50 / 47 C max. Shared Triton,
+Frigate and Ollama services remained running and healthy.
+
+The draft emitted 775 mapped instances: 640 car, 58 static obstacle, 45 person,
+25 heavy vehicle, five bicycle and one each motorcycle/animal. Sixteen previews
+were reviewed. They confirm that the fixed-FOV exterior stays clean and that
+the draft is useful for annotation assistance, but also expose the expected E0
+domain errors: duplicated/distant car boxes, unstable small instances, coarse
+fisheye boundaries and excessive static-obstacle proposals on planters. These
+counts are workload indicators for review, not precision or recall.
+
+The first prelabel attempt stopped before model loading because the container
+mountpoint `/evaluation-pack` was incorrectly required to equal the
+content-addressed generation basename. The path-name check was removed while
+all manifest, artifact and identity hashes remained mandatory. No failed result
+was published; the second attempt completed and 147 payload artifacts were
+reverified locally with zero digest/length mismatches.
+
+The review handoff is sealed separately as
+`annotation-workspace-9a950d1c37d56dc12cc285b13c5addd7795285879cbcb1fbb2d5811c3c69821a`.
+It contains a deterministic 64-image upload, a 775-instance COCO RLE draft, a
+dense CVAT Segmentation Mask archive, an exact frame/timestamp map, the fixed
+valid-FOV mask, the 15-class label specification and an unreviewed two-pass
+checklist. The three ZIP archives passed both the workspace validator and
+independent ZIP integrity checks. The 11 payload artifacts occupy 31,861,798
+bytes. This workspace remains `ground_truth=false`; the two synchronized CVAT
+tasks must be reviewed and their accepted exports sealed as a separate
+generation before any AP or mIoU claim is allowed.
+
+The real prelabel generation also exposed one identity-serialization defect in
+its producer: `target_categories` used integer dictionary keys while hashing,
+but JSON reloads them as strings and changes their sorted order. The stored
+artifact files and all 147 recorded payload hashes are unchanged. The workspace
+validator admits only this exact reversible legacy representation and binds the
+serialized `result.json` SHA separately. The worker producer now emits string
+keys before hashing, so subsequent prelabel identities are stable across a JSON
+round trip. The existing result was neither rewritten nor renamed.
+
+Prepare or reproduce the review inputs with:
+
+```console
+.venv/bin/python experiments/perception/prepare_e2_evaluation_pack.py candidates \
+  --job-root .runtime/compute-jobs/<job-id> \
+  --qualification-root .runtime/compute-experiments/e1/qualification-slices/<generation> \
+  --output-root .runtime/compute-experiments/e2/<candidate-review>
+
+.venv/bin/python experiments/perception/prepare_e2_evaluation_pack.py seal \
+  --job-root .runtime/compute-jobs/<job-id> \
+  --qualification-root .runtime/compute-experiments/e1/qualification-slices/<generation> \
+  --valid-fov-root .runtime/compute-experiments/e1/valid-fov/<generation> \
+  --selection .runtime/compute-experiments/e2/selection-e2.json \
+  --output-root .runtime/compute-experiments/e2/evaluation-packs
+
+.venv/bin/python experiments/perception/prepare_e2_annotation_workspace.py prepare \
+  --evaluation-pack .runtime/compute-experiments/e2/evaluation-packs/<generation> \
+  --prelabels .runtime/compute-experiments/e2/prelabels/<generation> \
+  --valid-fov-root .runtime/compute-experiments/e1/valid-fov/<generation> \
+  --output-root .runtime/compute-experiments/e2/annotation-workspaces
+```
+
+## E4 full-session semantic playback · 2026-07-21
+
+LAB E4 promotes the plain EoMT valid-FOV control from LAB E3 into the first
+complete saved-session semantic playback. It consumed all 4,489 frames of
+RAVNOVES00 (`20260720T065719Z_viewer_live`) from
+`sensor.camera.right`, using factory calibration slot `camera_1`, FP16
+autocast, batch size one and no sampling. CLAHE, five-view rectification and the
+instance branch were deliberately disabled.
+
+The immutable published result is
+`result-793785170472c519486ccd666be102fb04d169d92383acda3fcc29eecf045d30`.
+It contains an 800x600 H.264 semantic-overlay video, 4,489 semantic masks,
+4,489 timestamp rows, one-second GPU telemetry and a run report. FFprobe and the
+recorded-perception validator independently confirmed the exact frame count,
+448.723-second timeline, artifact hashes and input/job/calibration binding.
+
+The inference loop ran for 1,447.565 seconds at 3.101 FPS. Full extraction,
+inference, publication, hashing and validation took 1,668.259 seconds at 2.691
+FPS. GPU utilization was 73.05% mean / 89% p95, E4 process CUDA allocation
+peaked at 2,099.8 MiB, process RSS at 2,009.3 MiB, power at 253.23 W and
+temperature at 58 C. The exact configuration is about 3.72 times slower than
+the source recording rate and is therefore an offline baseline, not a live
+configuration.
+
+All task-controlled worker paths remained under `D:\NDC_MISSIONCORE`. The
+orchestrator enforced a 360 GiB free-space floor and a 17.195 GiB conservative
+working-set reserve. Final free space after exact task-temporary cleanup was
+379.395 GiB; C: was not used or mounted by the task.
+
+Mission Core exposes the result in **Сохранённые сессии → RAVNOVES00 →
+Источники данных сцены → Сегментация · камера right**. Browser acceptance
+confirmed the exact result source, 800x600 dimensions, full duration, no media
+error and advancing playback time. The detailed configuration, timing tables,
+artifact hashes, disk checkpoints, limitations and next gates are recorded in
+`experiments/perception/LAB_E4_REPORT_2026-07-21.md` and Ops card MISSIONCOR-18.
+
+E4 remains `ground_truth=false` and semantic-only. It makes no claim about live
+latency, instances, tracking, 3D cuboids, LiDAR association, distance accuracy,
+point-cloud labels or safety fitness.
+
 ## Recorded Rerun projection

 Mission Core discovers only results whose validated job names the opened
@@ -105,3 +405,84 @@ accepted for obstacle avoidance, free-space estimation or safety decisions.
   tests.
 4. Introduce tracking, segmentation/free-space, calibration and point-cloud
   models as separate versioned pipelines.
+
+## E5 recorded instance tracking qualification · 2026-07-21
+
+LAB E5 establishes the first measured temporal object-identity baseline on a
+preselected 601-frame, 60.069-second RAVNOVES00 interval. It uses the official
+Apache-2.0 YOLOX-S ONNX release through the existing Triton service, the
+immutable `camera_1`-bound valid-FOV mask, and a ByteTrack-style two-stage IoU
+tracker. The accepted profile, runner, model and configuration are pinned by
+SHA-256; output remains `ground_truth=false` and qualification-only.
+
+The immutable result is
+`e5-tracking-88aace13ef9963f8dc07f85228e530f9d28c2b49aca9192409f7975512b058f6`.
+It contains an 800x600 H.264 ID-overlay video, exactly 601 timestamped detection
+and track rows, one-second GPU telemetry, a contact sheet and the complete run
+report. The result validator rehashed all artifacts and verified the exact job,
+input, session, source, clip and timeline binding. FFprobe independently
+confirmed 601 declared/read frames and 60.068948 seconds.
+
+The frame loop ran for 88.093 seconds at 6.822 FPS; the complete worker run took
+166.336 seconds at 3.613 FPS. Mean per-frame time was 12.359 ms in Triton,
+0.247 ms in tracking, 30.132 ms decoding and 87.363 ms writing overlays. The
+detector and tracker are therefore not the main live-rate bottleneck; artifact
+I/O must be decoupled before a live path is admitted. GPU utilization was
+51.65% mean / 56% max, power 144.78 W mean / 148.40 W max, temperature 43 C max
+and process RSS 126.199 MiB.
+
+The run admitted 4,049 detections and emitted 3,320 observations across 167
+confirmed IDs. Useful clear-view persistence is proven, including vehicle
+tracks lasting 100–212 frames. The result also exposes real limitations: an ID
+fragments during the close woman/stroller occlusion, the stroller can be called
+`motorcycle`, and parked vehicles can flicker between `car` and `truck`. There
+is no identity ground truth, so IDF1, HOTA, MOTA and true ID-switch counts are
+not claimed.
+
+All task-controlled worker paths remained under `D:\NDC_MISSIONCORE`; the
+orchestrator enforced the 360 GiB floor and a 4.402 GiB working-set reserve.
+Free space after exact temporary cleanup was 380.595 GiB. The YOLOX model was
+loaded only for the run and restored to its prior unloaded state. Detailed
+provenance, pilot history, thresholds, timing tables, artifacts, hashes,
+quality findings and the LAB E6 gate are in
+`experiments/perception/LAB_E5_REPORT_2026-07-21.md` and Ops card
+MISSIONCOR-19.
+
+The next bounded gate is calibrated 2D-track/LiDAR association on the same
+601-frame interval: robust point support, metric range, coarse 3D cuboids and a
+qualification-only Rerun recording. Full-session and live promotion remain
+blocked on measured identity quality and a bounded queue/drop design.
+
+## E6 factory-calibrated tracked LiDAR fusion · 2026-07-21
+
+LAB E6 closes the recorded 2D-track/LiDAR association gate on the exact E5
+601-frame interval. It reuses the immutable E4 semantic masks, E5 track IDs,
+raw K1 point/pose capture and the XGRIDS `camera_1` factory KB4 calibration.
+Fused observations must satisfy 100 ms camera/point and pose/point gates.
+Nearest-depth buffering, semantic support, depth splitting, 3D connected
+components, robust range history and plausible-size gates prevent unsupported
+image rectangles from becoming fabricated 3D boxes.
+
+The immutable result is
+`e6-fusion-b4e4226674a66f6196c033785eb307c255a7bdd8493ef9809dfb1d6e5bd68eaa`.
+It processed 601/601 frames, fused 526, emitted 918 point-supported oriented
+cuboids across 56 track IDs and failed closed for depth on 75 frames outside
+the strict timing gate. LiDAR/camera absolute delta was 29.389 ms mean,
+70.794 ms p95 and 96.107 ms maximum. Accepted range spans 1.239–37.749 m with a
+10.189 m median.
+
+The single-process local geometry/artifact path took 14.774 seconds, peaked at
+240.469 MiB RSS and is faster than the source rate. This is not an end-to-end
+live result because E4 and E5 were precomputed. The external worker and both
+worker disks were untouched. The output includes an 800x600 H.264 overlay,
+timestamped JSONL/NPZ data, a contact sheet and a standalone 60 MiB Rerun
+recording with camera, map-frame cloud, support points and translucent
+`Boxes3D`. Visual QA confirmed that these are real oriented Rerun primitives;
+they remain visible-surface envelopes, not ground-truthed complete object
+volumes.
+
+Detailed provenance, profile freeze, pilot failures, thresholds, timing,
+rejection counts, artifact hashes, limitations and the LAB E7 gate are in
+`experiments/perception/LAB_E6_REPORT_2026-07-21.md`. Ops synchronization is
+pending restoration of the direct `nodedc-ops-agent` tasker tools; the legacy
+Ops API was not used.