26 KiB

Raw Blame History

External perception worker contract

Boundary

K1
  -> XGRIDS device plugin / Mission Core Edge
  -> authoritative raw session on Mac
  -> bounded missioncore.compute-job/v1
  -> replaceable GPU worker / Triton
  -> content-addressed missioncore.compute-result/v1
  -> optional validated Rerun derived layer

The GPU worker cannot discover, provision, start or stop a K1. It receives an observation package and returns derived observations. The Mac archive is never rewritten when a job is prepared or a result is received.

Recorded camera job v1

k1link compute prepare-camera-job accepts a sealed cataloged session, one camera source and one physical codec epoch. It performs the canonical camera checks before and after copying into a private staging directory:

summary schema/source/epoch and per-segment durability contract;
exact ordered JSONL index and its summary SHA-256;
init SHA-256, every segment length/SHA-256 and complete stream SHA-256;
continuous ISO-BMFF decode timeline and finite media duration;
source session-time mapping through the captured host clock origin.

Publication atomically renames this layout below ignored storage:

<jobs>/<job-id>/
├── job.json
└── input/camera/<source-id>/epoch-N/
    ├── summary.json
    ├── index.jsonl
    ├── init.mp4
    └── segments/*.m4s

input_sha256 is SHA-256 over canonical JSON for the complete path-free input descriptor. job_id includes its first 96 bits and validators require the full digest, so a prefix collision fails closed.

Recorded result v1

The accepted worker profile verifies every transferred file, reconstructs the stream outside the job, decodes frames and preserves strictly increasing best-effort timestamps in session seconds. Its result identity is canonical JSON covering:

job and full input generation;
pipeline ID/version;
model ID/version/weight SHA-256;
score/NMS thresholds, tensor shape, color order and letterbox policy.

The identity's full SHA-256 names an immutable result-<sha256> directory. The result manifest binds the detection artifact's length and SHA-256. A repeat with the same identity validates the existing directory and returns it without calling Triton.

Full-epoch panoptic result v2

missioncore.recorded-perception-result/v2 is the complete recorded-camera profile. It is separate from the YOLOX detector proof and processes every admitted frame without sampling. Its immutable identity binds the complete compute job, factory-calibration generation and camera slot, exact model revisions/weight files, thresholds, alpha values, runner SHA-256 and publication encoder.

The current research configuration produces:

TorchVision Mask R-CNN ResNet50-FPN v2 instance masks and COCO labels;
Microsoft BEiT ADE20K-150 semantic masks at the original 800x600 frame size;
a seekable H.264 MP4 with the two overlays combined for operator playback;
lossless instance and semantic mask PNGs for later calibrated fusion;
ordered per-frame JSONL and one-second GPU telemetry;
a machine-readable run report with input/config/model identities, decode, inference, encode and end-to-end timing, latency percentiles, throughput, CUDA peak allocation/reservation, process peak RSS, system load and GPU utilization/VRAM/temperature/power samples.

The worker runs with --network none and cached model weights. A preflight revalidates the complete transferred payload, CUDA execution and both cached model generations before decoding a long epoch. The run then repeats payload validation inside the inference container, requires exactly the declared frame count and a strictly increasing session-time row for every frame, and publishes only by atomic rename after every output digest is sealed.

There is no recorded-duration, frame-count or aggregate-video-byte admission ceiling in this profile. Resource use therefore scales with the real input and is reported, not hidden behind an arbitrary eight-minute laboratory limit.

Native panoptic playback

Full raster masks are not copied into the RRD. That would turn a long video into a multi-gigabyte browser-memory object. Instead Mission Core validates the v2 result and exposes its MP4 through the same generation-bound, seekable HTTP Range contract as a recorded camera. Replay advertises an additional opaque source such as recorded.perception.right; the Control Station opens it in a native video window on the shared session_time timeline. A one-, three- or ten-hour video remains disk/range streamed and does not have to fit in RAM.

The lossless masks remain private derived evidence. A host-side calibrated fusion step samples them at K1 KB4 LiDAR projections and publishes compact semantic Points3D, support-gated Boxes3D and diagnostic distances as a separate replaceable generation. Missing or rejected v2/fusion results never replace or invalidate the base raw point-cloud recording.

RAVNOVES00 qualification · 2026-07-20

The first full recorded run admitted all 4,489 frames from sensor.camera.right without sampling, failures or skips. The sealed input was 363,235,615 bytes over 35.421857292–484.144857292 session seconds. Its immutable result is result-f4cebdea8a82698a5b8a65d2c3fbdb0428b88b9dc49fe45f8cb37d740ed83d02.

Measured RTX 4090 worker results:

inference: 2,674.722 s and 1.678 frames/s;
end to end: 2,816.349 s and 1.594 frames/s;
instance latency: 89.826 ms p50, 121.344 ms p95, 915.325 ms max;
semantic latency: 249.678 ms p50, 284.826 ms p95, 406.594 ms max;
GPU utilization: 67% p50, 82% p95, 90% max over 2,675 one-second samples;
process CUDA peak: 2,230.8 MiB allocated and 2,872 MiB reserved;
total GPU memory observed, including the worker's shared resident services: 13,373 MiB p50 and 13,388 MiB max;
GPU power/temperature: 182.46 W p50, 190.10 W p95 and 49 C p50, 54 C max;
process peak RSS: 2,375.9 MiB;
publication: 31.229 s decode, 3.397 s NVENC, 81,109,627-byte H.264 MP4, 60,326,719-byte lossless mask archive.

The factory-calibrated full fusion generation fusion-0b1be23128ebd3d230562cffd96491169e99f0e839e56b812661c974e4fdc00b matched LiDAR and pose within the admitted 250 ms host-arrival window for 4,323 frames and declared 166 frames depth-unavailable. It produced 6,124,145 semantic points and 13,496 support-gated diagnostic boxes in 77.708 s. The compact fusion payload is 43 MiB.

Native browser QA opened RAVNOVES00, played the raw and panoptic 800x600 videos together at ready-state 4, and measured approximately 12 ms between their media clocks. The Rerun scene showed the synchronized semantic points and distance-labeled diagnostic boxes. The current baseline is deliberately not an accuracy or safety acceptance: generic perspective-trained models produce large fisheye false positives in 916 frames, and timing/distance have not been ground-truthed. The next A/B should compare an admitted undistort/ROI transform before inference rather than silently hiding these observations.

E1 valid-FOV preprocessing qualification · 2026-07-20

The first post-baseline A/B uses two immutable inputs derived from the same RAVNOVES00 job and factory calibration:

valid-FOV generation valid-fov-mask-b4dd8ddf2b87c1d520ee8a0868c4fea062d7c14d1bae73ccabd3abe1f3acbac2;
qualification-slice generation qualification-slice-2394070b4f3e38f1b8c483e878fcc11fd3c29f751f3d4cd7e3a2553304c5c142.

The mask is not estimated from each image. It is bound to the exact calibration SHA, sensor.camera.right, camera_1, the admitted 800x600 linear-resize profile and the KB4 principal point (396.319, 301.496). A four-pixel inner margin produces a 293.504-pixel radius, 270,606 valid pixels (56.37625%) and an exclusive crop rectangle [103, 8, 690, 595]. Repeated preparation reuses the same content-addressed PNG and manifest.

The slice selects 256 exact frame indices uniformly across all 4,489 frames, including both endpoints. The same loaded FP32 Mask R-CNN and BEiT generations were interleaved per frame across three variants: unmodified baseline, fixed valid-FOV fill, and valid-FOV crop remapped into the original pixel coordinates. The sealed result is qualification-result-98a2fee3d22979f3e18847719667cc76bdeacf25904c0fd1da4c5202253b3940.

The run completed in 318.969 s and emitted 12 preview frames per variant. GPU telemetry recorded 319 one-second samples: utilization was 74% p50 and 82% p95, power was 199.34 W p50 and 206.96 W p95, and temperature was 50 C p50 and 55 C max. The qualification process peaked at 1,849.9 MiB CUDA allocated, 2,256 MiB reserved and 2,298.2 MiB RSS.

Measured mean model paths, excluding the frame decode shared by all variants:

Variant	Mask R-CNN path	BEiT path	Combined
baseline	106.292 ms	278.064 ms	384.356 ms
valid-FOV fill	107.027 ms	280.762 ms	387.789 ms
valid-FOV crop	104.055 ms	282.840 ms	386.895 ms

The crop reduced Mask R-CNN forward time by 6.38%, but BEiT still receives its fixed 640x640 tensor and became 1.18% slower. With mask/crop preprocessing included, neither variant improved the combined path; fill was 0.89% slower and crop was 0.66% slower than baseline. A binary mask improves admission quality, but multiplying an unchanged tensor by it does not remove dense neural FLOPs.

The quality proxies are useful but are not ground truth. Baseline produced 58 instance masks and 58 boxes larger than half the admitted comparison area; both masked variants produced zero. The fraction of raw predicted instance-mask pixels outside the canonical FOV fell from 41.263% to 0.077% for fill and 0.911% for crop before the final output clamp. Inside the valid circle, mean BEiT disagreement with baseline was 8.713% for fill and 11.547% for crop. This is a measure of change, not accuracy.

E1 therefore accepts the immutable valid-FOV artifact and the 256-frame gate. Fixed fill is the conservative next accuracy baseline because it preserves the 800x600 geometry, removes the exterior lens region and changes the semantic result less than crop. Crop remains an experimental model-specific option, not a general speed optimization. The next run needs human labels/ground truth and must compare native KB4 input, calibrated virtual views and fisheye-trained models before promoting any preprocessing profile to a full-epoch result.

E2 evaluation pack and annotation gate · 2026-07-20

LAB E2 starts from the same immutable RAVNOVES00 compute job, E1 qualification slice and factory-calibrated valid-FOV generation. All eight contact sheets, covering the 256 uniformly distributed E1 candidates, were reviewed before selection. The sealed evaluation generation is evaluation-pack-7a983bba75d46c7c260252cb2d461e1384dcb92cda9e164397e841e6ebb37789.

The pack contains 64 exact 800x600 images: 48 reviewed full-epoch anchors from the E1 slice and four four-frame consecutive clips for temporal measurements. The clips cover a person with a stroller, a close moving car, vehicle occlusion/relative motion and a near building/terrace scene with a partially visible carried laptop. The anchors retain the recording's road, sidewalk, ground, grass, woody vegetation, buildings, sky, people, cars, trucks, lens boundary and hard-negative diversity.

Every image is stored both as the raw decoded RGB frame and as the accepted E1 fixed-valid-FOV-fill input. The identity binds the job/input SHA, source, codec epoch, selected segment SHA, decoded session timestamp, E1 qualification, valid-FOV generation, calibration SHA, camera slot, FFmpeg 7.1.1 generation and the raw/fill RGB pixel hashes. It also binds the reviewed selection-document SHA and both producer-code hashes. The pack has 130 hashed payload artifacts plus its manifest and occupies approximately 67 MiB locally.

One earlier local preparation generation, evaluation-pack-b6d9215a…, was not promoted because its identity omitted the producer-code and selection-document hashes. It remains a superseded diagnostic artifact and is not an accepted E2 input.

The immutable pack is deliberately unannotated. Its annotation contract defines 15 robotics-oriented thing/stuff classes, label 0 for the excluded lens exterior, label 255 for genuinely unresolved pixels, two-pass human review and required semantic, instance, safety-proxy and temporal metrics. The empty annotation template must be copied to a review workspace; it must never be edited inside the sealed pack. Model-generated prelabels may accelerate review but are not accepted as ground truth without a human pass.

No AP, mIoU or model-ranking claim is attached to E2 yet. The next gate is to complete and seal the reviewed annotations. Only then may candidate models be ranked on this pack; a full 4,489-frame run remains prohibited until one configuration passes both the accuracy and throughput gates.

The first model-assisted draft is sealed separately as evaluation-prelabels-4ba26bbf6eb8a49631f5caf984267e0445958540aeda2b5b0d82ca6440835cf1. It reuses the exact E0 Mask R-CNN and BEiT weights and maps their COCO/ADE classes into the E2 taxonomy. It is explicitly marked unreviewed-model-draft; it never mutates the evaluation pack or annotation template.

The isolated RTX 4090 run processed 64/64 frames in 31.551 s. Mean forward time was 57.539 ms for Mask R-CNN, 215.164 ms for BEiT and 272.703 ms combined. The process peaked at 1,842.8 MiB CUDA allocated, 2,768 MiB reserved and 2,239.9 MiB RSS. Across 32 one-second samples, GPU utilization was 51% p50 / 71% p95, power 167.73 W p50 / 186.89 W p95 and temperature 42 C p50 / 47 C max. Shared Triton, Frigate and Ollama services remained running and healthy.

The draft emitted 775 mapped instances: 640 car, 58 static obstacle, 45 person, 25 heavy vehicle, five bicycle and one each motorcycle/animal. Sixteen previews were reviewed. They confirm that the fixed-FOV exterior stays clean and that the draft is useful for annotation assistance, but also expose the expected E0 domain errors: duplicated/distant car boxes, unstable small instances, coarse fisheye boundaries and excessive static-obstacle proposals on planters. These counts are workload indicators for review, not precision or recall.

The first prelabel attempt stopped before model loading because the container mountpoint /evaluation-pack was incorrectly required to equal the content-addressed generation basename. The path-name check was removed while all manifest, artifact and identity hashes remained mandatory. No failed result was published; the second attempt completed and 147 payload artifacts were reverified locally with zero digest/length mismatches.

The review handoff is sealed separately as annotation-workspace-9a950d1c37d56dc12cc285b13c5addd7795285879cbcb1fbb2d5811c3c69821a. It contains a deterministic 64-image upload, a 775-instance COCO RLE draft, a dense CVAT Segmentation Mask archive, an exact frame/timestamp map, the fixed valid-FOV mask, the 15-class label specification and an unreviewed two-pass checklist. The three ZIP archives passed both the workspace validator and independent ZIP integrity checks. The 11 payload artifacts occupy 31,861,798 bytes. This workspace remains ground_truth=false; the two synchronized CVAT tasks must be reviewed and their accepted exports sealed as a separate generation before any AP or mIoU claim is allowed.

The real prelabel generation also exposed one identity-serialization defect in its producer: target_categories used integer dictionary keys while hashing, but JSON reloads them as strings and changes their sorted order. The stored artifact files and all 147 recorded payload hashes are unchanged. The workspace validator admits only this exact reversible legacy representation and binds the serialized result.json SHA separately. The worker producer now emits string keys before hashing, so subsequent prelabel identities are stable across a JSON round trip. The existing result was neither rewritten nor renamed.

Prepare or reproduce the review inputs with:

.venv/bin/python experiments/perception/prepare_e2_evaluation_pack.py candidates \
  --job-root .runtime/compute-jobs/<job-id> \
  --qualification-root .runtime/compute-experiments/e1/qualification-slices/<generation> \
  --output-root .runtime/compute-experiments/e2/<candidate-review>

.venv/bin/python experiments/perception/prepare_e2_evaluation_pack.py seal \
  --job-root .runtime/compute-jobs/<job-id> \
  --qualification-root .runtime/compute-experiments/e1/qualification-slices/<generation> \
  --valid-fov-root .runtime/compute-experiments/e1/valid-fov/<generation> \
  --selection .runtime/compute-experiments/e2/selection-e2.json \
  --output-root .runtime/compute-experiments/e2/evaluation-packs

.venv/bin/python experiments/perception/prepare_e2_annotation_workspace.py prepare \
  --evaluation-pack .runtime/compute-experiments/e2/evaluation-packs/<generation> \
  --prelabels .runtime/compute-experiments/e2/prelabels/<generation> \
  --valid-fov-root .runtime/compute-experiments/e1/valid-fov/<generation> \
  --output-root .runtime/compute-experiments/e2/annotation-workspaces

E4 full-session semantic playback · 2026-07-21

LAB E4 promotes the plain EoMT valid-FOV control from LAB E3 into the first complete saved-session semantic playback. It consumed all 4,489 frames of RAVNOVES00 (20260720T065719Z_viewer_live) from sensor.camera.right, using factory calibration slot camera_1, FP16 autocast, batch size one and no sampling. CLAHE, five-view rectification and the instance branch were deliberately disabled.

The immutable published result is result-793785170472c519486ccd666be102fb04d169d92383acda3fcc29eecf045d30. It contains an 800x600 H.264 semantic-overlay video, 4,489 semantic masks, 4,489 timestamp rows, one-second GPU telemetry and a run report. FFprobe and the recorded-perception validator independently confirmed the exact frame count, 448.723-second timeline, artifact hashes and input/job/calibration binding.

The inference loop ran for 1,447.565 seconds at 3.101 FPS. Full extraction, inference, publication, hashing and validation took 1,668.259 seconds at 2.691 FPS. GPU utilization was 73.05% mean / 89% p95, E4 process CUDA allocation peaked at 2,099.8 MiB, process RSS at 2,009.3 MiB, power at 253.23 W and temperature at 58 C. The exact configuration is about 3.72 times slower than the source recording rate and is therefore an offline baseline, not a live configuration.

All task-controlled worker paths remained under D:\NDC_MISSIONCORE. The orchestrator enforced a 360 GiB free-space floor and a 17.195 GiB conservative working-set reserve. Final free space after exact task-temporary cleanup was 379.395 GiB; C: was not used or mounted by the task.

Mission Core exposes the result in Сохранённые сессии → RAVNOVES00 → Источники данных сцены → Сегментация · камера right. Browser acceptance confirmed the exact result source, 800x600 dimensions, full duration, no media error and advancing playback time. The detailed configuration, timing tables, artifact hashes, disk checkpoints, limitations and next gates are recorded in experiments/perception/LAB_E4_REPORT_2026-07-21.md and Ops card MISSIONCOR-18.

E4 remains ground_truth=false and semantic-only. It makes no claim about live latency, instances, tracking, 3D cuboids, LiDAR association, distance accuracy, point-cloud labels or safety fitness.

Recorded Rerun projection

Mission Core discovers only results whose validated job names the opened session. Before projection it revalidates the complete job/result binding, artifact digests, ordered timestamps and result metrics. It then reconstructs the exact fMP4 epoch, verifies decoded dimensions/frame count with ffprobe, decodes exactly the admitted number of RGB frames and rejects boxes outside the image.

The optional perception.rrd endpoint returns either HTTP 204 or one complete, bounded RRF2 stream for the opened Rerun recording ID. The browser never splits that stream and never replaces the base recording when the optional layer is absent or rejected. Accepted frames are logged under /perception/camera on the canonical zero-based session_time timeline; Image and Boxes2D rows therefore seek with the point-cloud archive. Generated overlays are a private, content-addressed cache and remain rebuildable from the immutable job/result.

In Наблюдение → Пространственная сцена, a Распознавание button appears only after the layer has been admitted. It switches the active native Rerun view between the camera/detections and the point cloud. TEST007's accepted fragment occupies approximately 01:07.351–01:12.848; Rerun's latest-at query retains the last admitted frame after that short fragment.

Accepted model profile

YOLOX-S official ONNX from upstream release 0.1.1rc0;
Apache-2.0;
SHA-256 c5c2d13e59ae883e6af3b45daea64af4833a4951c92d116ec270d9ddbe998063;
FP32 input [1,3,640,640], BGR, bilinear top-left letterbox, pad 114;
FP32 output [1,8400,85], official grid/stride decode, COCO-80;
Triton 2.70.0 / ONNX Runtime GPU backend in the pinned 26.06 image.

This generic model is useful for proving the contract and timing path. It is not accepted for obstacle avoidance, free-space estimation or safety decisions.

Open gates

Qualify TEST007's 206-second epoch and record throughput/resource telemetry.
Replace SSH/SCP with a reviewed authenticated worker transport.
Add bounded live queues, sampling/drop policy and acquisition-isolation tests.
Introduce tracking, segmentation/free-space, calibration and point-cloud models as separate versioned pipelines.

E5 recorded instance tracking qualification · 2026-07-21

LAB E5 establishes the first measured temporal object-identity baseline on a preselected 601-frame, 60.069-second RAVNOVES00 interval. It uses the official Apache-2.0 YOLOX-S ONNX release through the existing Triton service, the immutable camera_1-bound valid-FOV mask, and a ByteTrack-style two-stage IoU tracker. The accepted profile, runner, model and configuration are pinned by SHA-256; output remains ground_truth=false and qualification-only.

The immutable result is e5-tracking-88aace13ef9963f8dc07f85228e530f9d28c2b49aca9192409f7975512b058f6. It contains an 800x600 H.264 ID-overlay video, exactly 601 timestamped detection and track rows, one-second GPU telemetry, a contact sheet and the complete run report. The result validator rehashed all artifacts and verified the exact job, input, session, source, clip and timeline binding. FFprobe independently confirmed 601 declared/read frames and 60.068948 seconds.

The frame loop ran for 88.093 seconds at 6.822 FPS; the complete worker run took 166.336 seconds at 3.613 FPS. Mean per-frame time was 12.359 ms in Triton, 0.247 ms in tracking, 30.132 ms decoding and 87.363 ms writing overlays. The detector and tracker are therefore not the main live-rate bottleneck; artifact I/O must be decoupled before a live path is admitted. GPU utilization was 51.65% mean / 56% max, power 144.78 W mean / 148.40 W max, temperature 43 C max and process RSS 126.199 MiB.

The run admitted 4,049 detections and emitted 3,320 observations across 167 confirmed IDs. Useful clear-view persistence is proven, including vehicle tracks lasting 100–212 frames. The result also exposes real limitations: an ID fragments during the close woman/stroller occlusion, the stroller can be called motorcycle, and parked vehicles can flicker between car and truck. There is no identity ground truth, so IDF1, HOTA, MOTA and true ID-switch counts are not claimed.

All task-controlled worker paths remained under D:\NDC_MISSIONCORE; the orchestrator enforced the 360 GiB floor and a 4.402 GiB working-set reserve. Free space after exact temporary cleanup was 380.595 GiB. The YOLOX model was loaded only for the run and restored to its prior unloaded state. Detailed provenance, pilot history, thresholds, timing tables, artifacts, hashes, quality findings and the LAB E6 gate are in experiments/perception/LAB_E5_REPORT_2026-07-21.md and Ops card MISSIONCOR-19.

The next bounded gate is calibrated 2D-track/LiDAR association on the same 601-frame interval: robust point support, metric range, coarse 3D cuboids and a qualification-only Rerun recording. Full-session and live promotion remain blocked on measured identity quality and a bounded queue/drop design.

E6 factory-calibrated tracked LiDAR fusion · 2026-07-21

LAB E6 closes the recorded 2D-track/LiDAR association gate on the exact E5 601-frame interval. It reuses the immutable E4 semantic masks, E5 track IDs, raw K1 point/pose capture and the XGRIDS camera_1 factory KB4 calibration. Fused observations must satisfy 100 ms camera/point and pose/point gates. Nearest-depth buffering, semantic support, depth splitting, 3D connected components, robust range history and plausible-size gates prevent unsupported image rectangles from becoming fabricated 3D boxes.

The immutable result is e6-fusion-b4e4226674a66f6196c033785eb307c255a7bdd8493ef9809dfb1d6e5bd68eaa. It processed 601/601 frames, fused 526, emitted 918 point-supported oriented cuboids across 56 track IDs and failed closed for depth on 75 frames outside the strict timing gate. LiDAR/camera absolute delta was 29.389 ms mean, 70.794 ms p95 and 96.107 ms maximum. Accepted range spans 1.239–37.749 m with a 10.189 m median.

The single-process local geometry/artifact path took 14.774 seconds, peaked at 240.469 MiB RSS and is faster than the source rate. This is not an end-to-end live result because E4 and E5 were precomputed. The external worker and both worker disks were untouched. The output includes an 800x600 H.264 overlay, timestamped JSONL/NPZ data, a contact sheet and a standalone 60 MiB Rerun recording with camera, map-frame cloud, support points and translucent Boxes3D. Visual QA confirmed that these are real oriented Rerun primitives; they remain visible-surface envelopes, not ground-truthed complete object volumes.

Detailed provenance, profile freeze, pilot failures, thresholds, timing, rejection counts, artifact hashes, limitations and the LAB E7 gate are in experiments/perception/LAB_E6_REPORT_2026-07-21.md. Ops synchronization is pending restoration of the direct nodedc-ops-agent tasker tools; the legacy Ops API was not used.

26 KiB Raw Blame History Unescape Escape