feat(perception): integrate calibrated operator pipeline
Add calibrated K1 projection, recorded and near-live perception qualification, unified Rerun operator layers, bounded replay admission, audited viewer controls, worker experiments, and lab evidence.
This commit is contained in:
DCCONSTRUCTIONS
@@ -0,0 +1,111 @@
|
||||
# External perception stack review — 2026-07-20
|
||||
|
||||
## Objective
|
||||
|
||||
Select mature, reusable components for Mission Core's K1 camera/LiDAR pipeline
|
||||
instead of growing the LAB implementation into a bespoke stack by trial and
|
||||
error. The target outputs are reviewed 2D/dense semantic labels, classified 3D
|
||||
boxes in the point cloud, distance to objects, and a path to recorded and live
|
||||
execution.
|
||||
|
||||
## Decision
|
||||
|
||||
Keep Rerun as the synchronized viewer and experiment evidence surface. Do not
|
||||
treat it as the perception engine. Use CVAT as the human-review and
|
||||
ground-truth control plane. Reuse NVIDIA's production inference patterns for
|
||||
LiDAR and multimodal execution, adapting only K1 data loading, timestamps, and
|
||||
factory calibration. Use open model frameworks as benchmark sources, not as a
|
||||
second production runtime.
|
||||
|
||||
The shortest credible route is:
|
||||
|
||||
1. Review LAB E2 drafts in CVAT and freeze a versioned ground-truth subset.
|
||||
2. Benchmark corrected/masked K1 images on that same subset.
|
||||
3. Replace heuristic 3D cube construction with a LiDAR-native PointPillars
|
||||
baseline.
|
||||
4. Serve the selected 2D and 3D models through the existing Triton worker and
|
||||
log outputs to Rerun.
|
||||
5. Gate camera/LiDAR BEVFusion on proven synchronization, calibration, and
|
||||
baseline metrics.
|
||||
|
||||
## Evidence from mature implementations
|
||||
|
||||
| Reference | Reusable practice | Mission Core use | Do not copy blindly |
|
||||
| --- | --- | --- | --- |
|
||||
| [Rerun](https://rerun.io/docs/overview/what-is-rerun) and its [examples](https://rerun.io/examples) | Time-aligned logging of images, point clouds, masks, 2D/3D boxes, trajectories, and model outputs | Viewer, debugging, playback, run evidence | It does not infer segmentation, depth, or boxes |
|
||||
| [OctoSense](https://github.com/anthonytec2/OctoSense) | Camera rectification before inference; RGB/LiDAR timestamp alignment; semantic labeling; dynamic-object masks; calibrated LiDAR projection; dataset packaging | Copy the pipeline shape and QA gates | It is primarily a dataset-generation recipe, not a drop-in real-time Mission Core runtime |
|
||||
| [DeepStream LiDAR 3D inference](https://docs.nvidia.com/metropolis/deepstream/7.1/text/DS_3D_Lidar_Inference.html) | PointPillars preprocessing, inference, postprocessing, and 3D box visualization in a production pipeline | First classified 3D-box baseline | Its sample data loader and calibration assumptions must be replaced with K1 adapters |
|
||||
| [DeepStream multimodal fusion](https://docs.nvidia.com/metropolis/deepstream/8.0/text/DS_3D_MultiModal_Lidar_Sensor_Fusion.html) | Camera/LiDAR preprocessing and BEVFusion deployment | Later accuracy/robustness stage after LiDAR-only baseline | Too expensive as the first 3D fix; requires trustworthy synchronization and extrinsics |
|
||||
| [MMDetection3D](https://github.com/open-mmlab/mmdetection3d) and [OpenPCDet](https://github.com/open-mmlab/OpenPCDet) | Reproducible 3D model/config baselines and pretrained checkpoints | Offline comparison and checkpoint selection | Do not introduce a parallel permanent serving stack beside Triton |
|
||||
| [TAO auto-label](https://docs.nvidia.com/tao/tao-toolkit/text/data_services/auto-label.html) | Assisted labeling and model-driven annotation | Candidate accelerator after the first reviewed GT set exists | Auto-label output is still draft data, not accuracy evidence |
|
||||
| [FiftyOne annotation integration](https://docs.voxel51.com/integrations/annotation.html) | Dataset curation, slice analysis, and CVAT round trips | Add when error analysis across many sessions becomes the bottleneck | Not needed to complete the first GT gate |
|
||||
| [Isaac ROS nvblox](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_nvblox) | GPU mapping, reconstruction, and cost-map production | Later navigation/mapping consumer | It does not replace object detection or classified 3D boxes |
|
||||
| [Isaac Sim camera sensors](https://docs.isaacsim.omniverse.nvidia.com/latest/sensors/isaacsim_sensors_camera.html) | Fisheye/OpenCV camera models and calibration-aware simulation | Synthetic regression cases after real-data metrics exist | Simulation is not a substitute for validating K1 factory calibration on recorded data |
|
||||
|
||||
## Selected stack boundaries
|
||||
|
||||
| Layer | Selected role | Current state | Gate before expansion |
|
||||
| --- | --- | --- | --- |
|
||||
| K1 ingestion | Preserve native timestamps, camera identity, point cloud, and vendor calibration | Recording works | Validate camera identity and projection on representative frames |
|
||||
| Image preprocessing | Stable optical-circle mask plus calibration-derived correction profile | LAB E2 mask/draft path exists | Freeze profile ID and compare corrected versus raw on identical frames |
|
||||
| 2D/dense perception | Generate prelabels, then evaluate against reviewed CVAT exports | LAB E2 drafts imported | Human two-pass review and class-level metrics |
|
||||
| 3D detection | LiDAR-native PointPillars baseline | Next model integration | K1 point format adapter, coordinate convention test, reviewed 3D subset |
|
||||
| Multimodal fusion | BEVFusion/DeepStream candidate | Deferred | Baselines, sync tolerance, extrinsic validation, GPU budget |
|
||||
| Serving | Existing Triton worker | Running | Pin model/config artifacts and record per-stage latency/resource metrics |
|
||||
| Visualization | Rerun | Running | Log reviewed labels and model outputs separately |
|
||||
| Annotation/GT | CVAT `v2.70.0` | Deployed on D; LAB E2 tasks imported | Two-pass review; export with immutable manifest and hashes |
|
||||
|
||||
## Configuration practices to adopt
|
||||
|
||||
- Rectify or calibration-map the fisheye image before model inference when the
|
||||
model expects a pinhole-like domain. Keep the raw frame alongside it.
|
||||
- Apply a cached optical-circle mask/profile by camera identity; do not
|
||||
recompute the mask for every frame.
|
||||
- Synchronize camera frames to LiDAR timestamps explicitly and record the
|
||||
chosen tolerance and dropped/unmatched frames.
|
||||
- Keep 2D detections, dense semantics, LiDAR-native 3D detections, and fused
|
||||
results as separate evidence streams. Fusion must not erase provenance.
|
||||
- Project 3D boxes into the camera for validation, but do not construct final
|
||||
3D boxes by simply extruding every 2D detection. That heuristic produces the
|
||||
overlapping-cube failure observed in LAB E1.
|
||||
- Record model identity, checkpoint hash, preprocessing profile, precision,
|
||||
batch size, GPU memory, stage latency, wall time, and output hashes for every
|
||||
material experiment.
|
||||
- Treat every model-produced label as `ground_truth=false` until reviewed and
|
||||
exported through the ground-truth gate.
|
||||
|
||||
## Next experiment gates
|
||||
|
||||
### LAB E2 review gate
|
||||
|
||||
- Review both imported tasks across the same 64 frames.
|
||||
- First pass: correct labels, masks, and missed/false objects.
|
||||
- Second pass: independent consistency review.
|
||||
- Export reviewed instance and semantic annotations with manifest hashes.
|
||||
|
||||
### LAB E3 image baseline
|
||||
|
||||
- Compare raw-circle-mask, rectified, and correction-profile inputs on the
|
||||
frozen reviewed subset.
|
||||
- Select by class metrics and temporal stability, not by visual preference.
|
||||
- Measure preprocessing, inference, postprocessing, and end-to-end time
|
||||
separately.
|
||||
|
||||
### LAB E4 3D baseline
|
||||
|
||||
- Integrate a PointPillars-compatible K1 point loader.
|
||||
- Lock coordinate axes, units, timestamps, and calibration version in the run
|
||||
manifest.
|
||||
- Evaluate classified 3D boxes on a small reviewed subset before any BEVFusion
|
||||
work.
|
||||
|
||||
## Rejected shortcuts
|
||||
|
||||
- Replacing Rerun with OctoSense: wrong boundary; OctoSense supplies useful
|
||||
pipeline patterns while Rerun remains the viewer.
|
||||
- Moving the full stack to Omniverse: useful for simulation and synthetic data,
|
||||
not the shortest path to recorded K1 accuracy.
|
||||
- Treating current semantic/instance drafts as GT: invalidates all subsequent
|
||||
model comparisons.
|
||||
- Starting with BEVFusion: adds synchronization, calibration, conversion, and
|
||||
runtime risks before the simpler LiDAR-native baseline is measured.
|
||||
@@ -0,0 +1,223 @@
|
||||
# LAB E10-N1 — Semantic-worker loss negative control
|
||||
|
||||
Date: 2026-07-22
|
||||
State: completed; negative control accepted
|
||||
Ground truth: no
|
||||
Navigation/safety acceptance: no
|
||||
Publication scope: laboratory negative control only
|
||||
|
||||
## Purpose and answer
|
||||
|
||||
E10-N1 tests a failure property required by the unmanned-system runtime:
|
||||
|
||||
> If the lower-rate semantic worker stops, does the 10 Hz detector/world-state
|
||||
> path continue, does semantic freshness become stale explicitly, and does 3D
|
||||
> fusion refuse to use an expired mask?
|
||||
|
||||
The answer is **yes for the accepted 60-second recorded source-paced control**:
|
||||
|
||||
- semantics was intentionally stopped after exactly 20 completed results;
|
||||
- detector/tracker processed 601/601 frames at 10.010952 FPS with zero drops;
|
||||
- world-state age p95 remained 90.268570 ms;
|
||||
- semantic binding transitioned to 498 `stale` detector states;
|
||||
- 3D fusion stopped after 96 fresh-mask frames and never resumed;
|
||||
- freshness violations: zero;
|
||||
- every negative-control acceptance check passed.
|
||||
|
||||
This proves fail-closed behavior for a controlled semantic-worker stop. It does
|
||||
not yet prove recovery/restart, direct K1 transport or a process/container crash.
|
||||
|
||||
## Input and configuration identity
|
||||
|
||||
The camera, calibration, LiDAR pack, models and 60-second source interval are
|
||||
identical to accepted LAB E10:
|
||||
|
||||
- session: `RAVNOVES00` / `20260720T065719Z_viewer_live`;
|
||||
- source: `sensor.camera.right`, 800x600;
|
||||
- source frames: 1,000–1,600 inclusive;
|
||||
- session time: 135.365857292–195.334857292 seconds;
|
||||
- source span: 59.969 seconds;
|
||||
- calibration slot: `camera_1`;
|
||||
- calibration SHA-256:
|
||||
`05f3ad9b38b3a4fc95388a8ec83da83c745e217709e51787b3d5aad0969f6fa9`;
|
||||
- LiDAR pack:
|
||||
`e10-lidar-pack-5da0396d32a27f9d1ca537cc2e8a371d386078d6f0dc71737b78620992af9625`;
|
||||
- detector: YOLOX-S/Triton every frame;
|
||||
- semantics: EoMT/FP16 scheduled every fifth frame;
|
||||
- semantic TTL: 750 ms;
|
||||
- detector queue capacity: 2;
|
||||
- semantic queue capacity: 1.
|
||||
|
||||
The only intentional change is:
|
||||
|
||||
```text
|
||||
mode = semantic-loss-negative-control
|
||||
stop_after_completed_results = 20
|
||||
```
|
||||
|
||||
Profile SHA-256:
|
||||
`eb2b114048336c273e77f3046d1eeac5187d27ec01e830de89ff201fb326c8c8`.
|
||||
|
||||
## Failure injection and expected behavior
|
||||
|
||||
The semantic worker exits cleanly immediately after publishing its twentieth
|
||||
real mask. The source producer and detector consumer remain unchanged. The last
|
||||
semantic result may be reused only until its 750 ms TTL expires.
|
||||
|
||||
After TTL expiration:
|
||||
|
||||
- semantic status must become `stale`;
|
||||
- world-state delivery health becomes degraded where LiDAR exists;
|
||||
- LiDAR association and cuboid generation must not execute with that mask;
|
||||
- raw source scheduling and detector/tracker must continue to the final frame;
|
||||
- the bounded semantic queue may discard scheduled work because its consumer is
|
||||
intentionally absent, but this must not block the detector queue.
|
||||
|
||||
The result uses publication scope
|
||||
`recorded-integrated-semantic-loss-negative-control-only`. Mission Core's E10
|
||||
overlay store explicitly excludes that scope, so the negative control cannot
|
||||
replace the accepted visual E10 result in `RAVNOVES00`.
|
||||
|
||||
## Result
|
||||
|
||||
Result ID:
|
||||
|
||||
`e10-integrated-perception-cf3b763731515120dc2714b6c3d7030496ed75057ce41ca88a1df62c21c4149c`
|
||||
|
||||
### Detector and world-state continuity
|
||||
|
||||
| Measurement | Result |
|
||||
|---|---:|
|
||||
| Detector scheduled / processed | 601 / 601 |
|
||||
| Detector drops | 0 |
|
||||
| Detector effective rate | 10.010952 FPS |
|
||||
| Detector queue maximum / capacity | 1 / 2 |
|
||||
| Replay wall / source span | 60.034251 / 59.969 s |
|
||||
| World-state age mean | 63.686918 ms |
|
||||
| World-state age p95 | 90.268570 ms |
|
||||
| World-state age max | 140.979884 ms |
|
||||
|
||||
The main path remained source-paced after semantic loss. Its p95 stayed well
|
||||
inside the frozen 175 ms laboratory budget.
|
||||
|
||||
### Semantic loss and freshness
|
||||
|
||||
| Measurement | Result |
|
||||
|---|---:|
|
||||
| Scheduled semantic frames | 121 |
|
||||
| Completed before injected stop | 20 |
|
||||
| Queue-overwritten after stop | 100 |
|
||||
| Final bounded queue depth | 1 |
|
||||
| Unavailable detector states | 5 |
|
||||
| Fresh detector states | 98 |
|
||||
| Stale detector states | 498 |
|
||||
| Fresh coverage | 16.3062% |
|
||||
|
||||
Queue accounting is exact: 20 consumed + 100 overwritten + 1 final bounded
|
||||
entry = 121 scheduled frames. These 100 semantic discards are the expected
|
||||
effect of intentionally removing the consumer; detector drops remained zero.
|
||||
|
||||
### Fail-closed fusion
|
||||
|
||||
| Fusion state | Frames |
|
||||
|---|---:|
|
||||
| `fused` | 96 |
|
||||
| `semantic-stale` | 428 |
|
||||
| `semantic-unavailable` | 2 |
|
||||
| `depth-unavailable-sync-gate` | 75 |
|
||||
|
||||
- accepted cuboid observations before loss became stale: 279;
|
||||
- fused frames after the mask became stale: zero;
|
||||
- fusion freshness violations: zero.
|
||||
|
||||
The 428 `semantic-stale` fusion states are fewer than the 498 stale detector
|
||||
states because the depth synchronization gate takes precedence on 70 of those
|
||||
frames. Neither path performs fusion with an expired semantic mask.
|
||||
|
||||
## Acceptance contract
|
||||
|
||||
| Check | Threshold | Result |
|
||||
|---|---:|---:|
|
||||
| Detector accounting | exact | pass |
|
||||
| Detector rate | >= 9.5 FPS | 10.010952 |
|
||||
| Detector drop fraction | <= 0.5% | 0% |
|
||||
| World-state age p95 | <= 175 ms | 90.268570 ms |
|
||||
| Semantic stop | exactly 20 results | 20 |
|
||||
| Stale detector states | >= 450 | 498 |
|
||||
| Semantic queue accounting | exact | pass |
|
||||
| Fusion with non-fresh mask | zero | zero |
|
||||
| Worker/producer errors | zero | zero |
|
||||
|
||||
All checks passed. `navigation_or_safety_accepted` remains false.
|
||||
|
||||
## GPU, memory and disk
|
||||
|
||||
Across 61 telemetry samples:
|
||||
|
||||
- GPU utilization: 58.72% mean, 90% p95, 93% max;
|
||||
- board memory used: 13,151.64 MiB mean, 13,161 MiB max;
|
||||
- power: 160.05 W mean, 211.02 W p95, 219.41 W max;
|
||||
- temperature: 41.28 C mean, 42 C max;
|
||||
- CUDA peak allocated: 2,107.925 MiB;
|
||||
- CUDA peak reserved: 2,840.0 MiB;
|
||||
- process peak RSS: 2,023.730 MiB.
|
||||
|
||||
The worker orchestrator used only `D:\NDC_MISSIONCORE`, enforced a 360 GiB
|
||||
floor, and completed with approximately 376.88 GiB free after cleanup. Windows
|
||||
C was not used. YOLOX was returned to its previous load state.
|
||||
|
||||
## Producer and artifact identity
|
||||
|
||||
- runner SHA-256:
|
||||
`894e679c91d398b0084fa7894346c827e6a54b01e1841cf505a21be76a6c7188`;
|
||||
- fusion runtime SHA-256:
|
||||
`042136fffbc6e06ef62c6d223d5272ce48e0136651f430ab1d59729b32c4bd90`;
|
||||
- orchestrator SHA-256:
|
||||
`ac0ee42495839773a2de60090ebcb49eee979dc7ee3b8587662a3c59e19a4d03`;
|
||||
- profile SHA-256:
|
||||
`eb2b114048336c273e77f3046d1eeac5187d27ec01e830de89ff201fb326c8c8`;
|
||||
- independent validator/renderer SHA-256:
|
||||
`deb9e9f581b0868c9cf3ecae73b17b3253152c969d46089f11ad364efa6f6804`.
|
||||
|
||||
| Artifact | SHA-256 |
|
||||
|---|---|
|
||||
| `semantic-frames.jsonl` | `ad187a84335cefed40b2d2db7261c16ff053057d49554dc9097e1258a63fa81e` |
|
||||
| `fusion-frames.jsonl` | `dbd8f203c41ed3f9ba0325b9a53227c7d25c66a34733caf0f2d687a18c26b5a6` |
|
||||
| `world-state.jsonl` | `2a4e1e0e1bade8f07b12e880602c24fb4b4f8814b683ab01b407174592bd4230` |
|
||||
| `transient-perception.npz` | `b653d8be9bc2474804268e9aa0a8c7dfdb1b2cacc779b515ef6b112aa9231e89` |
|
||||
| `gpu-telemetry.jsonl` | `07708357da7a5c464db2015f897137deb730762c68e247ed5eeb40cee19ac108` |
|
||||
| `run-report.json` | `06a8170ccdf561c51616050887a0ddd489894e36d22d35f0dd02ebbc700cf9e1` |
|
||||
| `result.json` | `e23821353a3afdd3789b9fee57d58915ca56c002828453662ec30824a878d3f5` |
|
||||
|
||||
Independent validation accepted the result and confirmed its negative-control
|
||||
publication scope. A store-selection regression proved that the UI continues to
|
||||
select the accepted qualification result
|
||||
`e10-integrated-perception-c6e0263...`, not E10-N1. The targeted regression
|
||||
suite now passes 48 tests.
|
||||
|
||||
## Accepted scope
|
||||
|
||||
- controlled semantic-worker stop after real inference results;
|
||||
- detector/tracker continuity at source pace;
|
||||
- explicit unavailable/fresh/stale state transition;
|
||||
- fail-closed LiDAR fusion after TTL expiration;
|
||||
- bounded semantic queue behavior without detector backpressure;
|
||||
- negative-control isolation from the user-facing Rerun result;
|
||||
- D-only storage and disk-floor enforcement.
|
||||
|
||||
## Not accepted
|
||||
|
||||
- automatic semantic-worker restart or state recovery;
|
||||
- hard process/container/GPU failure;
|
||||
- direct K1 live transport;
|
||||
- long-duration operation;
|
||||
- target onboard hardware;
|
||||
- navigation or safety use.
|
||||
|
||||
## Next gate
|
||||
|
||||
The next gate is the full `RAVNOVES00` stability run with the normal accepted E10
|
||||
configuration. It must cover the complete approximately nine-minute camera
|
||||
track, build a matching full-session LiDAR replay pack, preserve bounded queues
|
||||
and memory, and keep the D free-space floor. After that, the recorded source
|
||||
adapter can be replaced with direct K1 transport under the same contracts.
|
||||
@@ -0,0 +1,443 @@
|
||||
# LAB E10 — Integrated source-paced perception and world state
|
||||
|
||||
Date: 2026-07-22 (Europe/Moscow)
|
||||
Run timestamp: 2026-07-21T22:30:03.785Z
|
||||
State: completed; 60-second qualification accepted
|
||||
Ground truth: no
|
||||
Navigation/safety acceptance: no
|
||||
|
||||
## Purpose and answer
|
||||
|
||||
LAB E10 is the first Mission Core experiment that executes the current
|
||||
perception branches as one source-paced pipeline rather than composing separate
|
||||
offline outputs:
|
||||
|
||||
```text
|
||||
recorded K1 camera at original timestamps
|
||||
-> valid fisheye FOV
|
||||
-> YOLOX/Triton detections at ~10 Hz
|
||||
-> ByteTrack-style temporal IDs
|
||||
+ concurrent EoMT semantics at ~2 Hz
|
||||
+ factory KB4 camera/LiDAR projection
|
||||
+ strict recorded LiDAR synchronization gate
|
||||
-> point-supported distance and oriented 3D cuboids
|
||||
-> clearance sectors and transient world state
|
||||
-> Mission Core/Rerun recorded-session overlay
|
||||
```
|
||||
|
||||
The answer is **yes for the accepted recorded 60-second interval on the RTX
|
||||
4090 worker**:
|
||||
|
||||
- detector/tracker: 601/601 frames, 0 drops, 9.996120 FPS;
|
||||
- semantics: 121/121 scheduled frames, 0 drops, 2.012530 FPS;
|
||||
- fresh semantic coverage: 597/601 states, 99.3344%;
|
||||
- LiDAR-fused states: 525;
|
||||
- accepted point-supported cuboids: 760 observations;
|
||||
- world-state age p95: 76.514918 ms;
|
||||
- measured replay wall time: 60.123327 seconds for 59.969 source seconds;
|
||||
- every frozen acceptance check passed.
|
||||
|
||||
This closes the integration gap left by E9. Semantic masks used by LiDAR
|
||||
association were the actual newest in-memory EoMT results from the concurrent
|
||||
worker. E6 fusion and E7-style world-state calculation were executed inside the
|
||||
same critical replay loop. Per-frame masks were not synchronously encoded as
|
||||
PNG during the measurement.
|
||||
|
||||
This result does **not** prove direct live K1 transport, long-duration stability,
|
||||
forest-domain accuracy, target onboard compute performance or safety fitness.
|
||||
|
||||
## Immutable input identity
|
||||
|
||||
- Compute job: `recorded-camera-602ac89026ed12978619801d`
|
||||
- Input SHA-256:
|
||||
`602ac89026ed12978619801d4edea0cae24b5cc3afabd9f7af2858de6505a20e`
|
||||
- Session title: `RAVNOVES00`
|
||||
- Session ID: `20260720T065719Z_viewer_live`
|
||||
- Camera source: `sensor.camera.right`
|
||||
- Resolution: 800x600
|
||||
- Factory calibration slot: `camera_1`
|
||||
- Calibration content SHA-256:
|
||||
`05f3ad9b38b3a4fc95388a8ec83da83c745e217709e51787b3d5aad0969f6fa9`
|
||||
- Source frame selection: 1,000–1,600 inclusive, 601 frames
|
||||
- Session timeline: 135.365857292–195.334857292 seconds
|
||||
- Timeline SHA-256:
|
||||
`06e3514228b488137b92c303cbca171154ede00368783e32107ed2c39c41baa6`
|
||||
- Source span: 59.969 seconds
|
||||
|
||||
Valid-FOV identity:
|
||||
|
||||
- generation:
|
||||
`valid-fov-mask-b4dd8ddf2b87c1d520ee8a0868c4fea062d7c14d1bae73ccabd3abe1f3acbac2`;
|
||||
- mask SHA-256:
|
||||
`a40cee06b7c6f69b6a09a11563dcfd237f3de833b1ccd31459e66692e528ba63`;
|
||||
- valid pixels: 270,606 of 480,000.
|
||||
|
||||
The FOV mask is generated once and content-addressed. Pixels outside the usable
|
||||
fisheye circle are excluded before model processing; no repeated per-frame mask
|
||||
discovery occurs.
|
||||
|
||||
## LiDAR replay pack
|
||||
|
||||
The 60-second LiDAR input is a private immutable replay pack derived from the
|
||||
accepted E6 synchronization evidence:
|
||||
|
||||
- pack ID:
|
||||
`e10-lidar-pack-5da0396d32a27f9d1ca537cc2e8a371d386078d6f0dc71737b78620992af9625`;
|
||||
- 601 camera timeline rows;
|
||||
- 526 LiDAR rows passing the accepted strict synchronization gate;
|
||||
- 1,182,292 total points;
|
||||
- source coordinate system: K1 map frame;
|
||||
- target camera: `sensor.camera.right` / factory `camera_1`;
|
||||
- projection model: KB4, 800x600;
|
||||
- temporal contract: accepted E6 nearest-host-arrival best effort;
|
||||
- `lidar-pack.npz`: 9,629,449 bytes;
|
||||
- pack SHA-256:
|
||||
`72aa73340b20fcfaa21b330ef5b93b975c14a70e2cd16a57752d9952ff05ad9a`;
|
||||
- manifest SHA-256:
|
||||
`189a56660b69b05d39308b8dfa6c5ad8ed8812e870a218188181787cc6ea00ba`.
|
||||
|
||||
The pack is a replay optimization only. It does not change camera/LiDAR
|
||||
calibration, points, timestamps or the E6 synchronization decision.
|
||||
|
||||
## Models and worker runtime
|
||||
|
||||
Detector/tracker branch:
|
||||
|
||||
- official Megvii YOLOX-S ONNX release `0.1.1rc0`;
|
||||
- COCO-80 classes;
|
||||
- model SHA-256:
|
||||
`c5c2d13e59ae883e6af3b45daea64af4833a4951c92d116ec270d9ddbe998063`;
|
||||
- ONNX Runtime GPU through the existing Triton container;
|
||||
- ByteTrack-style two-stage IoU tracking inherited from E5/E8/E9.
|
||||
|
||||
Semantic branch:
|
||||
|
||||
- `tue-mps/cityscapes_semantic_eomt_large_1024`;
|
||||
- revision: `8d6b6d1a3f7b50d441afd7d247c2ed10db186e8f`;
|
||||
- weights: 1,276,175,488 bytes;
|
||||
- weights SHA-256:
|
||||
`c265da9a74f58f5c3f4826d23ca4ca78beac0b106cca5842beca61580de5b782`;
|
||||
- batch size one, FP16 autocast;
|
||||
- Cityscapes output mapped to the existing Mission Core 0–15 taxonomy.
|
||||
|
||||
Worker runtime:
|
||||
|
||||
- GPU: NVIDIA GeForce RTX 4090;
|
||||
- container: `nvcr.io/nvidia/tritonserver:26.06-py3`;
|
||||
- Python 3.12.3;
|
||||
- PyTorch 2.13.0+cu130;
|
||||
- Transformers 4.57.6;
|
||||
- NumPy 1.26.4;
|
||||
- SciPy 1.16.3.
|
||||
|
||||
## Frozen scheduling and fusion configuration
|
||||
|
||||
Detector path:
|
||||
|
||||
- every source frame, nominal 10 Hz;
|
||||
- bounded latest-wins queue capacity: 2;
|
||||
- observed maximum queue depth: 1;
|
||||
- overflow drops: 0.
|
||||
|
||||
Semantic path:
|
||||
|
||||
- every fifth source frame, nominal 2 Hz;
|
||||
- bounded latest-wins queue capacity: 1;
|
||||
- TTL: 750 ms;
|
||||
- overflow drops: 0;
|
||||
- states before the first completed mask are explicitly `unavailable`.
|
||||
|
||||
LiDAR association:
|
||||
|
||||
- nearest-depth KB4 projection buffer;
|
||||
- horizontal box inset 5%; top inset 4%; bottom inset 2%;
|
||||
- semantic support required for person, bicycle, motorcycle and vehicle groups;
|
||||
- class-dependent spatial clustering radius: 0.9 m for person/two-wheelers,
|
||||
1.5 m for vehicles;
|
||||
- minimum support points: 3 for person/two-wheelers, 5 for vehicles;
|
||||
- depth-cluster split: max(0.5 m, 6% of depth);
|
||||
- same-group NMS IoU: 0.55;
|
||||
- distance history: 5 frames;
|
||||
- distance innovation gate: max(1.5 m, 25%);
|
||||
- oriented cuboid dimensions use robust p05–p95 surface extents and explicit
|
||||
class-dependent plausibility limits.
|
||||
|
||||
World state:
|
||||
|
||||
- velocity history: 1.0 second;
|
||||
- clearance: 72 azimuth sectors;
|
||||
- considered range: 0.5–30 m;
|
||||
- obstacle height above estimated ground: 0.2–3.0 m;
|
||||
- forward clearance half-angle: 15 degrees.
|
||||
|
||||
## Frozen acceptance contract
|
||||
|
||||
| Check | Threshold | Result |
|
||||
|---|---:|---:|
|
||||
| Detector effective rate | >= 9.5 FPS | 9.996120 FPS |
|
||||
| Detector drop fraction | <= 0.5% | 0% |
|
||||
| Semantic effective rate | >= 1.8 FPS | 2.012530 FPS |
|
||||
| Semantic drop fraction | <= 5% | 0% |
|
||||
| Semantic completion age p95 | <= 400 ms | 179.682194 ms |
|
||||
| Fresh semantic coverage | >= 90% | 99.3344% |
|
||||
| World-state age p95 | <= 175 ms | 76.514918 ms |
|
||||
| LiDAR-fused frames | >= 450 | 525 |
|
||||
| Accepted cuboid observations | >= 100 | 760 |
|
||||
| Failures | zero | zero |
|
||||
|
||||
All checks were frozen before the qualification output and all passed. The
|
||||
result remains explicitly `navigation_or_safety_accepted=false`.
|
||||
|
||||
## Timing result
|
||||
|
||||
### Critical detector/fusion/world-state path
|
||||
|
||||
| Stage | Mean | p50 | p95 | Max |
|
||||
|---|---:|---:|---:|---:|
|
||||
| World-state age from source deadline | 56.074 ms | 52.017 ms | 76.515 ms | 118.755 ms |
|
||||
| Camera decode | 28.826 ms | 28.332 ms | 33.181 ms | 51.831 ms |
|
||||
| Detector inference path | 23.442 ms | 20.298 ms | 36.127 ms | 48.794 ms |
|
||||
| Detector queue wait | 0.087 ms | 0.066 ms | 0.155 ms | 5.694 ms |
|
||||
| LiDAR projection | 0.411 ms | 0.358 ms | 0.723 ms | 4.258 ms |
|
||||
| Semantic/LiDAR association | 1.649 ms | 1.391 ms | 3.858 ms | 12.513 ms |
|
||||
| Clearance | 0.187 ms | 0.175 ms | 0.394 ms | 0.667 ms |
|
||||
| World-state serialization/calculation | 0.088 ms | 0.076 ms | 0.226 ms | 0.586 ms |
|
||||
|
||||
The critical path stayed below its 175 ms laboratory ceiling without frame
|
||||
drops. Decode plus detection dominates the 10 Hz path; LiDAR projection,
|
||||
association, cuboid generation, clearance and world-state computation consume
|
||||
only a few milliseconds at p95 on this input.
|
||||
|
||||
### Semantic branch
|
||||
|
||||
| Stage | Mean | p50 | p95 | Max |
|
||||
|---|---:|---:|---:|---:|
|
||||
| Completion age from source deadline | 155.067 ms | 150.439 ms | 179.682 ms | 384.761 ms |
|
||||
| Processing after dequeue | 124.748 ms | 122.595 ms | 138.430 ms | 353.011 ms |
|
||||
| EoMT forward | 98.849 ms | 96.101 ms | 112.343 ms | 309.115 ms |
|
||||
| Processor | 9.647 ms | 9.347 ms | 11.958 ms | 31.297 ms |
|
||||
| Host to device | 5.680 ms | 6.063 ms | 8.184 ms | 11.627 ms |
|
||||
| Model postprocess | 7.451 ms | 6.703 ms | 11.166 ms | 12.413 ms |
|
||||
| Valid-FOV fill | 1.927 ms | 1.882 ms | 2.249 ms | 3.075 ms |
|
||||
| Queue wait | 0.145 ms | 0.155 ms | 0.194 ms | 0.308 ms |
|
||||
|
||||
The semantic branch is deliberately asynchronous. A semantic outlier therefore
|
||||
does not block detector admission or raw recording. The newest completed mask
|
||||
is bound only to current/future detector states and only while its TTL remains
|
||||
valid.
|
||||
|
||||
## Fusion and world-state output
|
||||
|
||||
Fusion states across 601 detector rows:
|
||||
|
||||
- `fused`: 525;
|
||||
- `depth-unavailable-sync-gate`: 75;
|
||||
- `semantic-unavailable`: 1.
|
||||
|
||||
The LiDAR pack exposes 526 strict-sync frames. One of those occurred before the
|
||||
first semantic result completed, explaining 525 actual fused states without
|
||||
inventing a mask.
|
||||
|
||||
Cuboid association counts:
|
||||
|
||||
- accepted point-supported cuboid observations: 760;
|
||||
- rejected, fewer than five clustered points: 996;
|
||||
- rejected, no semantic LiDAR support: 574;
|
||||
- rejected, distance innovation: 172;
|
||||
- rejected, fewer than three clustered points: 45;
|
||||
- rejected, implausible cuboid: 14.
|
||||
|
||||
The 760 value is a count of accepted per-frame cuboid observations, not 760
|
||||
unique physical objects. Cuboids bound the visible LiDAR-supported surface. They
|
||||
must not be interpreted as complete object volume or ground truth.
|
||||
|
||||
## GPU, memory and disk
|
||||
|
||||
Across 61 one-second telemetry samples:
|
||||
|
||||
- GPU utilization: 45.57% mean, 61% p95, 65% max;
|
||||
- board memory used: 13,119.30 MiB mean, 13,128 MiB max;
|
||||
- board memory utilization: 19.51% mean, 26% max;
|
||||
- power: 154.70 W mean, 166.75 W p95, 178.45 W max;
|
||||
- temperature: 43.87 C mean, 46 C max;
|
||||
- EoMT process CUDA peak allocated: 2,107.925 MiB;
|
||||
- EoMT process CUDA peak reserved: 2,840.0 MiB;
|
||||
- process peak RSS: 2,022.535 MiB.
|
||||
|
||||
This shows usable compute headroom on the laboratory RTX 4090 for this frozen
|
||||
configuration. It says nothing yet about the final onboard GPU/accelerator.
|
||||
That hardware must receive its own experiment and budgets.
|
||||
|
||||
Task-controlled worker storage remained under `D:\NDC_MISSIONCORE`:
|
||||
|
||||
| Disk point | Free bytes | Approx. free GiB |
|
||||
|---|---:|---:|
|
||||
| Before measured replay | 404,946,915,328 | 377.136 |
|
||||
| After measured replay | 404,933,578,752 | 377.124 |
|
||||
| Enforced floor | 386,547,056,640 | 360.000 |
|
||||
|
||||
Temporary decoded frames were cleaned from the exact run-owned D path. The
|
||||
YOLOX model state was restored after the run. Triton, Frigate and Ollama were
|
||||
left running. Windows C was not used or modified.
|
||||
|
||||
## Result artifacts and integrity
|
||||
|
||||
Published result:
|
||||
|
||||
`e10-integrated-perception-c6e0263b9e18be7e4f447597065b5ff93b8511463e8f453800f98a61f25c37df`
|
||||
|
||||
| Artifact | Bytes | SHA-256 |
|
||||
|---|---:|---|
|
||||
| `semantic-frames.jsonl` | 55,025 | `16cb53c7df8491b6620d0483f5004953d0aa54e83198bb101ae357c06e34be89` |
|
||||
| `fusion-frames.jsonl` | 1,418,427 | `1b28baf9dfc6c8a793f60233f5ffd03a3e40f63a4dfac7a97bff2813f5bd967d` |
|
||||
| `world-state.jsonl` | 1,524,066 | `9bdab9c1749fca98797ab428bd94f3e1b5a80b7977f013f323283ce9559040ee` |
|
||||
| `transient-perception.npz` | 1,106,484 | `7e0d0761584f4700e55caf31965d53de1c13b4f8a17c7cfbd2279af4c5f4cabd` |
|
||||
| `gpu-telemetry.jsonl` | 13,704 | `e036cdd35addc96152c3c534e75e249ac25b18b588cecabb30a8729d36d80f3c` |
|
||||
| `run-report.json` | 13,563 | `a053d629ea1b48e4a039d7277e0753108926e6efa0000971bc89008eac66ec26` |
|
||||
| `result.json` | immutable manifest | `48ac8865ff629b00e56ca90cb4a3a68b161f06275465a86ae7973808f8472c2b` |
|
||||
|
||||
Accepted producer identities:
|
||||
|
||||
- runner SHA-256:
|
||||
`3f667e291f18bcccd58d0ced391e23e0cca05ba3cdeb0da50b0f2fb814c4ff70`;
|
||||
- fusion runtime SHA-256:
|
||||
`4d4d9d7fcf8f3b86ac84228902df1948aafc2f48643c53f77250e77bf9854d5d`;
|
||||
- profile SHA-256:
|
||||
`42e2a49945e65daf86e7891794e30ef2c5182b71d073c76ae764c03ea2501589`;
|
||||
- accepted run orchestrator SHA-256:
|
||||
`8bc939e9a5e94707cd30bf7d948fed893087c03d3345e31c20c853b237ecde76`.
|
||||
|
||||
After acceptance, only the orchestrator's human-readable free-space line was
|
||||
changed from pipeline output to host output so future variable assignment does
|
||||
not capture the label. Current orchestrator SHA-256 is
|
||||
`ac0ee42495839773a2de60090ebcb49eee979dc7ee3b8587662a3c59e19a4d03`.
|
||||
This post-run correction does not alter or relabel the accepted E10 result.
|
||||
|
||||
The local E10 Python sources were subsequently normalized by the repository's
|
||||
Ruff formatter and passed the same 47-test regression suite. The accepted
|
||||
worker staging directory remains the immutable source of the producer hashes
|
||||
above. Current local source hashes for the next run are:
|
||||
|
||||
- fusion runtime:
|
||||
`042136fffbc6e06ef62c6d223d5272ce48e0136651f430ab1d59729b32c4bd90`;
|
||||
- LiDAR-pack builder:
|
||||
`e75875342f11a0573eb5eef106f10ff9545debf6c012e717c1df281d479b746b`;
|
||||
- worker runner:
|
||||
`9e434a82d370d1ecf079d76cec5c7a4cf955ca631651862eacb8572b94e33c8b`;
|
||||
- Mission Core validator/renderer:
|
||||
`260a03c623bcab15c6dafd63caa36514e9e35a40e0e6bd4d6551eb3db832c058`.
|
||||
|
||||
Durable local evidence root:
|
||||
|
||||
```text
|
||||
.runtime/compute-experiments/e10/worker-results/
|
||||
```
|
||||
|
||||
Worker result root:
|
||||
|
||||
```text
|
||||
D:\NDC_MISSIONCORE\runtime\derived\
|
||||
e10-integrated-perception-c6e0263b9e18be7e4f447597065b5ff93b8511463e8f453800f98a61f25c37df
|
||||
```
|
||||
|
||||
## Independent validation and regression checks
|
||||
|
||||
The independent E10 validator confirmed:
|
||||
|
||||
- content-addressed result identity and artifact hashes;
|
||||
- exact job/session/source/calibration/model/profile/runner binding;
|
||||
- detector and semantic queue accounting;
|
||||
- ordered source timestamps and semantic stride;
|
||||
- semantic binding never comes from a future source frame;
|
||||
- unavailable states contain no invented perception payload;
|
||||
- LiDAR pack identity, size and hash;
|
||||
- every frozen acceptance check.
|
||||
|
||||
The standalone E10 fusion runtime was also run against the accepted E6
|
||||
precomputed E4/E5 inputs. It reproduced E6 exactly: 526 fused frames and 918
|
||||
accepted cuboid observations. This is the control that the E10 integration did
|
||||
not silently change the established E6 geometry algorithm.
|
||||
|
||||
Targeted automated tests passed: 47 tests across E10, compute-result discovery,
|
||||
session API, E9, and E6. Direct Rerun rendering produced a valid 58,342,680-byte
|
||||
`RRF2` stream. The Mission Core perception endpoint returned HTTP 200 with a
|
||||
58,342,936-byte `RRF2` response.
|
||||
|
||||
## Mission Core / Rerun publication
|
||||
|
||||
Mission Core now discovers the newest accepted E10 result for the active
|
||||
recording before falling back to E6 or legacy 2D results. The generated Rerun
|
||||
stream contains:
|
||||
|
||||
- camera frames;
|
||||
- semantic masks;
|
||||
- 2D tracked detections;
|
||||
- raw map-frame point cloud;
|
||||
- LiDAR support points;
|
||||
- oriented point-supported 3D cuboids;
|
||||
- range and world-state evidence.
|
||||
|
||||
Visual acceptance was completed in the local Control Station:
|
||||
|
||||
1. `Наблюдение` -> `Пространственная сцена`;
|
||||
2. saved session `RAVNOVES00`;
|
||||
3. three active sources loaded;
|
||||
4. original camera and `Сегментация · камера right` played together on the
|
||||
common session timeline;
|
||||
5. real point cloud and trajectory remained visible behind the camera panels;
|
||||
6. playback time advanced normally, proving the result is not a single image.
|
||||
|
||||
The E10 perception overlay covers only its qualified 60-second source interval.
|
||||
The underlying saved session remains the full 8:55.718 recording.
|
||||
|
||||
## Accepted scope
|
||||
|
||||
- actual concurrent YOLOX/Triton and EoMT/PyTorch execution;
|
||||
- actual current in-memory semantic mask consumed by LiDAR association;
|
||||
- source-paced detector/tracker near 10 Hz and semantics near 2 Hz;
|
||||
- bounded latest-wins queues with zero measured drops;
|
||||
- factory KB4 projection using XGRIDS calibration;
|
||||
- point-supported distance and oriented 3D cuboids;
|
||||
- clearance and transient world-state publication in the critical loop;
|
||||
- 60-second latency, GPU, memory and disk evidence;
|
||||
- playback in Mission Core/Rerun for the recorded session;
|
||||
- D-only worker storage and restored service/model state.
|
||||
|
||||
## Not accepted
|
||||
|
||||
- direct K1 MQTT/camera transport into the perception scheduler;
|
||||
- the full 8:55.718 session or multi-hour continuous operation;
|
||||
- hardware-clock camera/LiDAR synchronization proof;
|
||||
- failure behavior after semantic-worker loss;
|
||||
- forest/off-road model quality or class coverage;
|
||||
- final onboard compute rate, thermals or power;
|
||||
- complete-object 3D geometry;
|
||||
- navigation or safety use.
|
||||
|
||||
## Next gates
|
||||
|
||||
The next work is aimed at the unmanned-system operating contour, not cosmetic
|
||||
offline quality:
|
||||
|
||||
1. **Semantic-loss negative control.** Kill or stall the semantic branch during
|
||||
source-paced replay and prove detector/raw recording continue, freshness
|
||||
becomes stale/unavailable, and no false fresh mask is published.
|
||||
2. **Full RAVNOVES00 stability run.** Build a full-session immutable LiDAR pack,
|
||||
run the same frozen pipeline for approximately nine minutes, verify bounded
|
||||
memory/queues, zero recording interference and D free-space floor.
|
||||
3. **Direct K1 transport.** Replace the recorded camera/LiDAR source adapter
|
||||
with the existing live K1 ingestion while preserving the accepted scheduler,
|
||||
freshness and world-state contracts.
|
||||
4. **Target onboard worker qualification.** Repeat the same gates on the actual
|
||||
deployable compute hardware; optimize models only against explicit rate,
|
||||
latency, thermal and power budgets.
|
||||
5. **Domain quality branch.** In parallel with runtime work, create forest and
|
||||
off-road evaluation data for classes, traversability and geometry. This must
|
||||
not block proving the live transport and scheduling contour.
|
||||
|
||||
Item 1 was subsequently completed and accepted as
|
||||
`LAB_E10_N1_REPORT_2026-07-22.md`: detector continuity remained 10.010952 FPS
|
||||
with zero drops, 498 states became explicitly stale, and fusion recorded zero
|
||||
freshness violations after the controlled semantic stop. The immediate gate is
|
||||
therefore the full-session run. No model upgrade or UI polish is justified
|
||||
before long-duration boundedness is measured.
|
||||
@@ -0,0 +1,369 @@
|
||||
# LAB E11 — Full-session integrated perception stability qualification
|
||||
|
||||
Date: 2026-07-22
|
||||
Accepted result creation time: 2026-07-22T06:27:51.941Z
|
||||
Laboratory status: **accepted**
|
||||
Navigation or safety status: **not accepted**
|
||||
|
||||
## Objective
|
||||
|
||||
LAB E11 tested whether the integrated Mission Core perception contour proven on
|
||||
the 60-second LAB E10 interval remains bounded and source-paced across the
|
||||
complete recorded right-camera epoch from `RAVNOVES00`.
|
||||
|
||||
The qualification retained the intended onboard architecture:
|
||||
|
||||
```text
|
||||
recorded camera at original source timestamps
|
||||
-> bounded latest-wins detector queue
|
||||
-> YOLOX-S + ByteTrack-compatible tracking at camera rate
|
||||
-> bounded sampled semantic queue
|
||||
-> EoMT Cityscapes semantics at 1/5 camera rate
|
||||
-> factory camera_1 KB4 camera/LiDAR projection
|
||||
-> visible LiDAR-supported 3D cuboids and distance
|
||||
-> clearance sectors and timestamped world state
|
||||
```
|
||||
|
||||
This was not an offline maximum-quality render. The replay speed was exactly
|
||||
`1.0`, queues were bounded, and acceptance required the worker to keep up with
|
||||
the recorded source without detector loss.
|
||||
|
||||
## Result summary
|
||||
|
||||
Accepted immutable result:
|
||||
|
||||
`e10-integrated-perception-7613b09021455ddec941144224142f7879e3c6ecac5b2370ea8a7d4cb01918ba`
|
||||
|
||||
Worker result root:
|
||||
|
||||
`D:\NDC_MISSIONCORE\runtime\derived\e10-integrated-perception-7613b09021455ddec941144224142f7879e3c6ecac5b2370ea8a7d4cb01918ba`
|
||||
|
||||
Local verified mirror:
|
||||
|
||||
`.runtime/compute-experiments/e10/worker-results/e10-integrated-perception-7613b09021455ddec941144224142f7879e3c6ecac5b2370ea8a7d4cb01918ba`
|
||||
|
||||
| Measure | Result | Frozen gate | State |
|
||||
|---|---:|---:|---|
|
||||
| Source frames | 4,489/4,489 | exactly 4,489 | pass |
|
||||
| Detector drops | 0 | 0 | pass |
|
||||
| Detector effective rate | 10.004402 FPS | >=9.5 FPS | pass |
|
||||
| Semantic results | 898/898 scheduled | accounting exact | pass |
|
||||
| Semantic drops | 0 | <=5% | pass |
|
||||
| Semantic effective rate | 2.001326 FPS | >=1.8 FPS | pass |
|
||||
| Fresh semantic coverage | 99.754956% | >=90% | pass |
|
||||
| Semantic completion age p95 | 267.211864 ms | <=400 ms | pass |
|
||||
| World-state age p95 | 102.714712 ms | <=175 ms | pass |
|
||||
| LiDAR-fused frames | 3,917 | >=3,500 | pass |
|
||||
| Accepted cuboid observations | 6,157 | >=500 | pass |
|
||||
| Inference/producer failures | 0 | 0 | pass |
|
||||
| Fusion with non-fresh semantics | 0 | 0 | pass |
|
||||
|
||||
The complete 448.623-second camera interval was replayed in 448.702 seconds.
|
||||
This is source-paced operation, not an accelerated batch claim.
|
||||
|
||||
## Frozen inputs
|
||||
|
||||
### Recorded source
|
||||
|
||||
- UI session alias: `RAVNOVES00`;
|
||||
- session ID: `20260720T065719Z_viewer_live`;
|
||||
- compute job: `recorded-camera-602ac89026ed12978619801d`;
|
||||
- job input SHA-256:
|
||||
`602ac89026ed12978619801d4edea0cae24b5cc3afabd9f7af2858de6505a20e`;
|
||||
- source: `sensor.camera.right`;
|
||||
- codec epoch: `1`;
|
||||
- resolution: 800x600;
|
||||
- source frame indices: 0–4,488 inclusive;
|
||||
- camera timeline: 35.421857292–484.044857292 seconds;
|
||||
- timeline SHA-256:
|
||||
`f91dad0a3cc998f4250be794b48f437943b3c60edccbe7e5ceed59030854a807`;
|
||||
- job payload: 363,235,615 bytes;
|
||||
- raw MQTT evidence: 133,572,722 bytes.
|
||||
|
||||
### Factory calibration
|
||||
|
||||
- slot: `camera_1`;
|
||||
- content identity SHA-256:
|
||||
`05f3ad9b38b3a4fc95388a8ec83da83c745e217709e51787b3d5aad0969f6fa9`;
|
||||
- projection: factory KB4 intrinsics plus camera/LiDAR extrinsic;
|
||||
- LiDAR coordinate source: K1 map frame.
|
||||
|
||||
No calibration was fitted or modified during LAB E11.
|
||||
|
||||
### Full LiDAR replay pack
|
||||
|
||||
Immutable pack:
|
||||
|
||||
`e10-lidar-pack-576c994a6c814e2592dd6240ace3902a5db94843312c759a73ba0c9166157d2b`
|
||||
|
||||
- frames: 4,489;
|
||||
- frames with accepted LiDAR and pose: 3,928;
|
||||
- strict-sync coverage: 87.502785%;
|
||||
- point observations: 9,207,270;
|
||||
- compressed payload: 72,996,000 bytes;
|
||||
- payload SHA-256:
|
||||
`0685d24219d8236caf8b7f1685e93f6d6b59e7fd015a768d88a92bbe8b154944`;
|
||||
- builder SHA-256:
|
||||
`bc81a1aa8759a750480745a52af34f4c700261e8bfbf5d615e8c6e5fa96bb69c`;
|
||||
- accepted E6 temporal-profile SHA-256:
|
||||
`b8d3a6f043fde54ffd32ea05bfe580e2be7aace1a23cdbf7e266eb3325d147b2`;
|
||||
- accepted E6 result:
|
||||
`e6-fusion-b4e4226674a66f6196c033785eb307c255a7bdd8493ef9809dfb1d6e5bd68eaa`.
|
||||
|
||||
The pack uses the unchanged E6 temporal policy:
|
||||
|
||||
- clock source: recorded host monotonic arrival;
|
||||
- binding: nearest host-arrival best effort;
|
||||
- maximum camera/LiDAR delta: 100 ms;
|
||||
- maximum point/pose delta: 100 ms.
|
||||
|
||||
The pack is a replay optimization and immutable input. It does not improve or
|
||||
reinterpret synchronization accuracy.
|
||||
|
||||
## Frozen software configuration
|
||||
|
||||
### Integrated scheduling profile
|
||||
|
||||
Profile: `e10_full_session_profile.json`
|
||||
SHA-256: `73dd8472b59c9ad48b781600c89eadb0aab12cb6ef59c4572ce5a8e17dcd4472`
|
||||
|
||||
- mode: `full-session-qualification`;
|
||||
- replay speed: 1.0;
|
||||
- detector queue: latest-wins, capacity 2;
|
||||
- semantic queue: latest-wins, capacity 1;
|
||||
- semantic scheduling: every fifth camera frame;
|
||||
- semantic TTL: 750 ms;
|
||||
- detector loss gate: exactly zero;
|
||||
- required source selection: frames 0–4,488 and at least 448 seconds.
|
||||
|
||||
### Producer identities
|
||||
|
||||
| Component | SHA-256 |
|
||||
|---|---|
|
||||
| Integrated runner | `5548cc607f245cd1c01c85d6b821fdf5b91e512ee9b2f8ad6862175c93b7628c` |
|
||||
| Fusion runtime | `844a3a3ecc56313be2f96a6b861abd8f55100ec7427a72379e52a55fbd4c5749` |
|
||||
| PowerShell orchestrator | `ac0ee42495839773a2de60090ebcb49eee979dc7ee3b8587662a3c59e19a4d03` |
|
||||
| Detector profile | `819bdfb0da521d187ec9ce3d3039a0ff75daa0ab473fb3d8e5fa92db9c1ddca1` |
|
||||
| Semantic profile | `ea583966bc3409f5cf563cbf4fad05e366907e67187082eb692aff53d9f5d875` |
|
||||
|
||||
Worker runner generation:
|
||||
|
||||
`D:\NDC_MISSIONCORE\runtime\derived\e10-runner-5548cc607f24-ac0ee4249583`
|
||||
|
||||
### Models
|
||||
|
||||
Detector:
|
||||
|
||||
- YOLOX-S, COCO-80;
|
||||
- ONNX SHA-256:
|
||||
`c5c2d13e59ae883e6af3b45daea64af4833a4951c92d116ec270d9ddbe998063`;
|
||||
- Triton config SHA-256:
|
||||
`5795c737a7935a655961b069e8404d336d891f9762fb6dffb93956a076479604`.
|
||||
|
||||
Semantic model:
|
||||
|
||||
- `tue-mps/cityscapes_semantic_eomt_large_1024`;
|
||||
- revision `8d6b6d1a3f7b50d441afd7d247c2ed10db186e8f`;
|
||||
- `model.safetensors` SHA-256:
|
||||
`c265da9a74f58f5c3f4826d23ca4ca78beac0b106cca5842beca61580de5b782`;
|
||||
- precision: FP16 autocast;
|
||||
- batch size: 1.
|
||||
|
||||
Valid-FOV mask:
|
||||
|
||||
- generation:
|
||||
`valid-fov-mask-b4dd8ddf2b87c1d520ee8a0868c4fea062d7c14d1bae73ccabd3abe1f3acbac2`;
|
||||
- mask SHA-256:
|
||||
`a40cee06b7c6f69b6a09a11563dcfd237f3de833b1ccd31459e66692e528ba63`;
|
||||
- admitted pixels: 270,606/480,000.
|
||||
|
||||
## Worker and resource envelope
|
||||
|
||||
- GPU: NVIDIA GeForce RTX 4090;
|
||||
- container: `nvcr.io/nvidia/tritonserver:26.06-py3`;
|
||||
- Python: 3.12.3;
|
||||
- PyTorch: 2.13.0+cu130;
|
||||
- Transformers: 4.57.6;
|
||||
- NumPy: 1.26.4;
|
||||
- SciPy: 1.16.3.
|
||||
|
||||
GPU telemetry, 449 one-second samples:
|
||||
|
||||
| Measure | Mean | p95 | Max |
|
||||
|---|---:|---:|---:|
|
||||
| GPU utilization | 67.70% | 92% | 96% |
|
||||
| Board GPU memory used | 13,151.48 MiB | 13,161 MiB | 13,161 MiB |
|
||||
| GPU-memory controller utilization | 16.40% | 33% | 34% |
|
||||
| Power | 207.34 W | 215.51 W | 229.74 W |
|
||||
| Temperature | 48.46 C | 53 C | 54 C |
|
||||
|
||||
Process and CUDA measurements:
|
||||
|
||||
- process peak RSS: 2,944.93 MiB;
|
||||
- CUDA peak allocated by the runner: 2,107.93 MiB;
|
||||
- CUDA peak reserved by the runner: 2,840 MiB.
|
||||
|
||||
The board-memory figure includes the existing Triton/model context and is not
|
||||
equivalent to runner-only allocation.
|
||||
|
||||
## Latency detail
|
||||
|
||||
| Stage | Mean | p95 | Max |
|
||||
|---|---:|---:|---:|
|
||||
| Frame decode | 34.198 ms | 47.488 ms | 84.767 ms |
|
||||
| Detector | 32.099 ms | 47.759 ms | 70.328 ms |
|
||||
| Detector queue wait | 0.230 ms | 0.202 ms | 33.917 ms |
|
||||
| LiDAR projection | 0.491 ms | 0.940 ms | 7.396 ms |
|
||||
| Object association | 2.176 ms | 5.795 ms | 20.093 ms |
|
||||
| Clearance | 0.236 ms | 0.577 ms | 4.740 ms |
|
||||
| World-state construction | 0.117 ms | 0.333 ms | 4.562 ms |
|
||||
| End-to-end world-state age | 71.634 ms | 102.715 ms | 189.403 ms |
|
||||
|
||||
Semantic inference:
|
||||
|
||||
| Stage | Mean | p95 | Max |
|
||||
|---|---:|---:|---:|
|
||||
| Processing | 191.693 ms | 209.808 ms | 448.629 ms |
|
||||
| Forward | 156.680 ms | 170.785 ms | 414.754 ms |
|
||||
| Completion age | 228.427 ms | 267.212 ms | 477.220 ms |
|
||||
| Semantic queue wait | 0.186 ms | 0.279 ms | 4.548 ms |
|
||||
|
||||
The acceptance contract uses p95 rather than the isolated maximum. Five startup
|
||||
frames had no semantic state and six later frames observed a stale state; no 3D
|
||||
fusion was asserted on any non-fresh semantic state.
|
||||
|
||||
## Fusion detail
|
||||
|
||||
- LiDAR available: 3,928 frames;
|
||||
- fused with fresh semantics: 3,917 frames;
|
||||
- fused share of LiDAR-available frames: 99.719959%;
|
||||
- depth unavailable under the frozen sync gate: 561 frames;
|
||||
- semantic unavailable: 5 frames;
|
||||
- semantic stale: 6 frames;
|
||||
- accepted cuboid observations: 6,157;
|
||||
- fusion freshness violations: 0.
|
||||
|
||||
Rejections are expected diagnostic outcomes:
|
||||
|
||||
| Reason | Count |
|
||||
|---|---:|
|
||||
| Insufficient 5-point support | 8,268 |
|
||||
| No semantic LiDAR support | 4,892 |
|
||||
| Distance innovation gate | 1,045 |
|
||||
| Insufficient 3-point support | 142 |
|
||||
| Implausible cuboid extent | 24 |
|
||||
|
||||
These counts describe per-frame object observations, not unique physical
|
||||
objects.
|
||||
|
||||
## Disk safety and storage behavior
|
||||
|
||||
The worker orchestrator was confined to `D:\NDC_MISSIONCORE`. Drive C was not
|
||||
read or written by this task.
|
||||
|
||||
- configured D free-space floor: 360 GiB;
|
||||
- preflight free space: 376.506 GiB;
|
||||
- calculated additional reserve requirement: 11.834 GiB;
|
||||
- after full stream reconstruction: 376.127 GiB;
|
||||
- after 4,489 PNG extraction: 373.578 GiB;
|
||||
- after publication, before temporary cleanup: 373.347 GiB;
|
||||
- after cleanup and model-state restoration: 376.223 GiB.
|
||||
|
||||
Temporary frames and the reconstructed camera stream were deleted by the
|
||||
orchestrator. No raw recording, prior result, model, cache, or accepted runner
|
||||
generation was deleted.
|
||||
|
||||
Known orchestrator timings:
|
||||
|
||||
- frame preparation: 31.425 seconds;
|
||||
- source-paced replay: 448.702 seconds;
|
||||
- total orchestrator wall time, including model validation/loading and cleanup:
|
||||
600.623 seconds.
|
||||
|
||||
## Published artifacts
|
||||
|
||||
| Artifact | Bytes | SHA-256 |
|
||||
|---|---:|---|
|
||||
| `semantic-frames.jsonl` | 404,380 | `49b1a859924b26ab5f550ca3c6d6ec907c32c0ef01870c270c33ef3eb103db53` |
|
||||
| `fusion-frames.jsonl` | 11,322,702 | `67ee78bc1385e5733ff33106fb7d257825ce297f69356af763c2598a531e8f3c` |
|
||||
| `world-state.jsonl` | 11,703,966 | `d4772484dd0740ba009a947114d4c6bde7b5fc2ffbe55ddf7a341cf9aa11e5a5` |
|
||||
| `transient-perception.npz` | 8,538,476 | `e53f0aea9b94704cd95c898b4b6f9c21da4fdad864373889213a9c1d66080ccf` |
|
||||
| `gpu-telemetry.jsonl` | 100,714 | `2fd0d8f9497f5cff06a3035b9f251c780a0f4fcccf14d3a89f49a18584f43e92` |
|
||||
| `run-report.json` | 14,076 | `3f0577b0dc8b55b38b8528c8f081be71af856327485f088f0ab0305e5a731ab1` |
|
||||
|
||||
Publication scope is
|
||||
`recorded-integrated-realtime-qualification-only`. The accepted result is
|
||||
eligible to become the newest recorded integrated overlay for this session.
|
||||
|
||||
## Verification
|
||||
|
||||
The copied result was independently validated on the Mission Core host using
|
||||
`validate_integrated_perception_result`. Validation rechecked:
|
||||
|
||||
- compute-job identity;
|
||||
- result and configuration identities;
|
||||
- full contiguous source selection;
|
||||
- LiDAR-pack identity and calibration binding;
|
||||
- artifact sizes and SHA-256 values;
|
||||
- 4,489 fusion rows and 4,489 world-state rows;
|
||||
- semantic timeline ordering;
|
||||
- NPZ array shapes and offsets;
|
||||
- report/result acceptance consistency.
|
||||
|
||||
The focused automated suite completed with 24 passing tests, including the E10,
|
||||
E8, E9, live-perception and qualification validators.
|
||||
|
||||
## Controlled preflight rejection
|
||||
|
||||
An initial invocation passed the base E5 detector profile where the integrated
|
||||
runner requires the E8 real-time tracking wrapper. The schema guard rejected it
|
||||
during preflight with `LAB E8 profile schema changed`.
|
||||
|
||||
That invocation did not extract frames, start replay, publish a result, or alter
|
||||
model state. The accepted invocation changed only the detector-profile argument
|
||||
to the already frozen E8 profile. This rejected preflight is retained here so
|
||||
the laboratory history is complete.
|
||||
|
||||
## What LAB E11 proves
|
||||
|
||||
LAB E11 proves that the current RTX 4090 worker and bounded scheduling contour
|
||||
can replay the complete recorded `RAVNOVES00` right-camera epoch at source rate
|
||||
while producing:
|
||||
|
||||
- 10 Hz 2D detection/tracking;
|
||||
- approximately 2 Hz semantic state;
|
||||
- factory-calibrated LiDAR-supported distance and visible-surface cuboids;
|
||||
- timestamped world state and clearance;
|
||||
- bounded queues with no detector or semantic loss in this run.
|
||||
|
||||
It also shows that the one-minute LAB E10 result was not a short-interval
|
||||
artifact: source-paced behavior remained stable for the full 448.623 seconds.
|
||||
|
||||
## What LAB E11 does not prove
|
||||
|
||||
- direct live K1 camera/MQTT ingestion into this scheduler;
|
||||
- hardware-clock camera/LiDAR synchronization;
|
||||
- forest-domain accuracy or robustness;
|
||||
- safety or navigation fitness;
|
||||
- complete amodal 3D object dimensions;
|
||||
- multi-camera fusion;
|
||||
- performance on weaker onboard compute hardware.
|
||||
|
||||
YOLOX COCO and EoMT Cityscapes are generic models. Cuboids describe the visible
|
||||
LiDAR-supported surface selected by the current association policy, not a
|
||||
ground-truth full object volume.
|
||||
|
||||
## Next gate
|
||||
|
||||
The next priority is not another offline quality-only pass. It is a transport
|
||||
integration gate:
|
||||
|
||||
1. feed the same bounded scheduler from direct K1 camera and LiDAR/MQTT sources;
|
||||
2. keep the latest-wins queues and freshness rules unchanged;
|
||||
3. expose live world-state age, drop counters and semantic freshness;
|
||||
4. qualify a controlled near-real-time live interval on this same worker;
|
||||
5. only then compare lower-power onboard targets or more domain-specific models.
|
||||
|
||||
The full-session recorded overlay may be rendered on demand for visual QA, but
|
||||
its generation is presentation work and is not required to accept the compute
|
||||
stability result.
|
||||
@@ -0,0 +1,237 @@
|
||||
# LAB E12 — Bounded shadow transport to the external perception worker
|
||||
|
||||
Date: 2026-07-22
|
||||
State: near-live transport qualification accepted; physical K1 live run pending
|
||||
Ground truth: no
|
||||
Navigation/safety acceptance: no
|
||||
Publication scope: diagnostic shadow transport only
|
||||
|
||||
## Purpose and answer
|
||||
|
||||
LAB E12 asks whether Mission Core can carry the already proven camera,
|
||||
LiDAR and pose inputs to the external RTX worker at source rate without giving
|
||||
that worker a second connection to K1, without allowing it to back-pressure the
|
||||
raw recording, and without granting it any command authority.
|
||||
|
||||
The answer is **yes for a 15-second source-paced RAVNOVES00 near-live run**:
|
||||
|
||||
- Mission Core admitted and the Windows worker verified 446/446 ordered events;
|
||||
- the stream contained 151 camera frames, one camera init, 142 LiDAR frames,
|
||||
150 poses and two lifecycle records;
|
||||
- ingress sequence gaps, payload-integrity failures, queue overflow and oversize
|
||||
rejection were all zero in the nominal run;
|
||||
- the worker completed in 15.073 seconds for a 15-second source interval;
|
||||
- the worker connected only to an authenticated Mission Core endpoint through
|
||||
an SSH reverse tunnel and had no K1 address or K1 transport;
|
||||
- both sides declared `commands_enabled=false` and
|
||||
`navigation_or_safety_accepted=false`.
|
||||
|
||||
The physical K1 run was not attempted because the unit was not present in the
|
||||
BLE scan and the only unclassified LAN neighbour rejected both MQTT and RTSP.
|
||||
No subnet scan or guessed device address was used. The exact same ingress is
|
||||
already attached to the physical acquisition path and will receive live events
|
||||
when K1 is next awake.
|
||||
|
||||
## Architecture under test
|
||||
|
||||
```text
|
||||
K1 camera RTSP ──> Mission Core FFmpeg ──> durable fMP4 segment commit
|
||||
│
|
||||
└─> camera latest-wins queue (2)
|
||||
|
||||
K1 MQTT ──> durable raw K1MQTT commit ──> reviewed live callback
|
||||
├─> LiDAR latest-wins queue (2)
|
||||
└─> pose bounded queue (16)
|
||||
|
||||
separate modality queues ──> one exclusive localhost WebSocket consumer
|
||||
──> bearer authentication ──> SSH reverse tunnel
|
||||
──> read-only Docker probe on RTX worker D:
|
||||
```
|
||||
|
||||
The camera observer is invoked only after `CameraArchiveWriter.append()` has
|
||||
atomically published and fsynced the fragment, its index entry and its crash
|
||||
checkpoint. The MQTT observer is invoked only by `capture_mqtt`'s
|
||||
`on_message_recorded` callback after the raw frame has been committed. Derived
|
||||
perception therefore remains expendable: it cannot become the source of record.
|
||||
|
||||
The ingress is process-local and bounded by modality:
|
||||
|
||||
| Queue | Capacity | Maximum payload |
|
||||
|---|---:|---:|
|
||||
| control | 4 | 16 KiB |
|
||||
| camera init | 1 | 1 MiB |
|
||||
| camera frame | 2 | 1 MiB |
|
||||
| LiDAR | 2 | 2 MiB |
|
||||
| pose | 16 | 2 MiB |
|
||||
|
||||
Camera bursts cannot evict pose or LiDAR observations. A full queue drops its
|
||||
oldest derived item and retains the latest work; it never waits on the worker.
|
||||
Only one authenticated worker consumer may hold the stream lease.
|
||||
|
||||
## Nominal qualification input
|
||||
|
||||
- canonical session: `20260720T065719Z_viewer_live` / `RAVNOVES00`;
|
||||
- camera source: `sensor.camera.right`, codec epoch 1, 800x600 H.264 fMP4;
|
||||
- source interval: the first 15 seconds of the canonical camera epoch;
|
||||
- MQTT input: the matching canonical raw K1MQTT arrival interval;
|
||||
- source pacing: 1.0x recorded host-arrival time;
|
||||
- camera index SHA-256:
|
||||
`e029815a60ad9fbfedb6169142c7449df2b119a51d1ce001f08806e04eb0be14`;
|
||||
- raw MQTT SHA-256:
|
||||
`70da0edad5cbf0e89b6e2355c0aac294f8c36a67249cd33b8918353fd88c83af`.
|
||||
|
||||
This replay is a transport qualification, not a synthetic message generator.
|
||||
Every camera segment was rechecked against its durable length and SHA-256 index
|
||||
record. K1MQTT framing and aligned timing metadata were read through the
|
||||
existing reviewed replay parser.
|
||||
|
||||
## Nominal result
|
||||
|
||||
### Event accounting
|
||||
|
||||
| Modality | Selected | Edge admitted | Worker verified | Payload bytes |
|
||||
|---|---:|---:|---:|---:|
|
||||
| camera init | 1 | 1 | 1 | 763 |
|
||||
| camera frame | 151 | 151 | 151 | 11,583,362 |
|
||||
| LiDAR | 142 | 142 | 142 | 3,674,244 |
|
||||
| pose | 150 | 150 | 150 | 25,346 |
|
||||
| control | 2 | 2 | 2 | 48 |
|
||||
| **Total** | **446** | **446** | **446** | **15,283,763** |
|
||||
|
||||
- worker first/last ingress sequence: 1/446;
|
||||
- sequence gaps: 0;
|
||||
- per-event payload digest mismatches: 0;
|
||||
- session-end observed: yes;
|
||||
- worker wall time: 15.072754 s;
|
||||
- edge publication wall time including orderly disconnect: 15.698036 s.
|
||||
|
||||
### Queue behaviour
|
||||
|
||||
| Queue | Maximum depth / capacity | Consumed | Dropped | Oversize rejected | Final depth |
|
||||
|---|---:|---:|---:|---:|---:|
|
||||
| control | 1 / 4 | 2 | 0 | 0 | 0 |
|
||||
| camera init | 1 / 1 | 1 | 0 | 0 | 0 |
|
||||
| camera frame | 2 / 2 | 151 | 0 | 0 | 0 |
|
||||
| LiDAR | 2 / 2 | 142 | 0 | 0 | 0 |
|
||||
| pose | 2 / 16 | 150 | 0 | 0 | 0 |
|
||||
|
||||
The camera and LiDAR queues reached their configured maximum capacity but did
|
||||
not overflow: the tunnel/worker consumer drained them within source rate.
|
||||
|
||||
## Worker-disconnect negative control
|
||||
|
||||
A separate four-second source-paced run intentionally removed the consumer
|
||||
after approximately one second. The source still admitted its complete selected
|
||||
interval:
|
||||
|
||||
- camera frames: 41/41 admitted;
|
||||
- LiDAR: 38/38 admitted;
|
||||
- pose: 40/40 admitted;
|
||||
- camera init: 1/1 admitted;
|
||||
- source run completed in 4.147395 seconds;
|
||||
- the worker saw 23 ordered events and no session-end, as intended.
|
||||
|
||||
After disconnect, the bounded derived queues behaved as designed:
|
||||
|
||||
| Queue | Consumed | Overflow drops | Retained final depth / capacity |
|
||||
|---|---:|---:|---:|
|
||||
| camera frame | 10 | 29 | 2 / 2 |
|
||||
| LiDAR | 6 | 30 | 2 / 2 |
|
||||
| pose | 6 | 18 | 16 / 16 |
|
||||
| control | 1 | 0 | 1 / 4 |
|
||||
|
||||
Exact accounting holds for every modality: consumed + dropped + retained equals
|
||||
published. The consumer failure did not stop, delay or throw into the source
|
||||
publisher. This is the required fail-open property for raw capture and the
|
||||
required fail-closed property for derived perception availability.
|
||||
|
||||
## Transport and security boundary
|
||||
|
||||
- Mission Core remains bound to `127.0.0.1`.
|
||||
- A private 0600 bearer token is stored under
|
||||
`.runtime/live-perception/shadow-worker.token`; it is not returned in plugin
|
||||
state or passed on a remote command line.
|
||||
- The Mac opens an SSH reverse tunnel using the pinned `mission-gpu` host key.
|
||||
- The worker receives the token on stdin and connects through
|
||||
`host.docker.internal` to that tunnel.
|
||||
- The server permits one consumer lease and immediately releases it on client
|
||||
disconnect.
|
||||
- Every binary event contains an explicit schema, bounded header, exact payload
|
||||
length and SHA-256 digest.
|
||||
- The worker container was read-only, capability-dropped, `no-new-privileges`,
|
||||
PID-limited and given only read-only runner plus D-backed report mounts.
|
||||
- No GPU access was required for this transport-only probe.
|
||||
- The worker never received a K1 address and never opened MQTT or RTSP.
|
||||
|
||||
## Storage and runtime
|
||||
|
||||
All task-controlled Windows files remained under `D:\NDC_MISSIONCORE`:
|
||||
|
||||
- runner:
|
||||
`D:\NDC_MISSIONCORE\workspace\mission-core-compute\run_e12_shadow_transport_probe.py`;
|
||||
- report root:
|
||||
`D:\NDC_MISSIONCORE\runtime\derived\e12-shadow-transport`.
|
||||
|
||||
Windows drive C was not used by this task. D had approximately 375.75 GiB free
|
||||
before the nominal run and 376.06 GiB at the final read; the report is only a
|
||||
small JSON document and raw payloads were not retained on the worker.
|
||||
|
||||
## Artifact and producer identity
|
||||
|
||||
- nominal edge report:
|
||||
`.runtime/compute-experiments/e12/20260722T065957Z-e12-source.json`,
|
||||
SHA-256 `36018374bc82b720d6ee50dace9ca1621431a5f90f49bfaf824054ecf5e6dd15`;
|
||||
- nominal D-worker report:
|
||||
`D:\NDC_MISSIONCORE\runtime\derived\e12-shadow-transport\20260722T065955Z-e12-shadow-transport.json`,
|
||||
SHA-256 `4983b6bcb489892170d7dbc8014dc04e4576202a8590394658259dd8d1b42de6`;
|
||||
- negative edge report:
|
||||
`.runtime/compute-experiments/e12/20260722T070212Z-e12-source.json`,
|
||||
SHA-256 `2cc4f45959b4ce58fa2a58e004a6572a26a483b33e458f46a471012bf5abda74`;
|
||||
- negative worker report:
|
||||
`.runtime/compute-experiments/e12/local-negative/20260722T070209Z-e12-shadow-transport.json`,
|
||||
SHA-256 `311e34f3e514f77caf0a36355902b15a2fc22cd861ecbf00a91d00f08a695a63`;
|
||||
- ingress implementation SHA-256:
|
||||
`73e9359d926f7004ebe9519e0de4a05e4230922b714058ded9d27eff55ca06c0`;
|
||||
- authenticated router SHA-256:
|
||||
`4c409299d99304a0330256113f188c0bca97aadab6f742a9a626c059a91c0ce6`;
|
||||
- camera raw-first hook SHA-256:
|
||||
`086adf61d8143587025b85f46a59c9af746eee5656213cc1aa1c80974592d0a9`;
|
||||
- physical acquisition integration SHA-256:
|
||||
`a9aaf129f083cfb219767e13765b8575fc62df8e0ed05e6d50cff00c45ef8186`;
|
||||
- near-live source runner SHA-256:
|
||||
`9492804d8ecb629cb4a43b7b2d3220484c7b545b2e53dbad09eb8069f680d17d`;
|
||||
- D-worker transport probe SHA-256 on both hosts:
|
||||
`0ad67a1eb6935dd111c490128645b0d4345dc86d171dd7c4045ba990287da44d`.
|
||||
|
||||
## Accepted scope
|
||||
|
||||
- raw-first post-commit camera/MQTT fan-out;
|
||||
- separate bounded modality queues;
|
||||
- exclusive authenticated worker lease;
|
||||
- source-rate camera/LiDAR/pose transport through the real Windows/Docker path;
|
||||
- per-event integrity and ordered lifecycle;
|
||||
- worker-disconnect isolation from the source;
|
||||
- D-only worker persistence;
|
||||
- zero K1 command authority.
|
||||
|
||||
## Not accepted
|
||||
|
||||
- a physical live K1 run in this experiment;
|
||||
- live camera decode on the worker;
|
||||
- live detector, semantic model or LiDAR fusion execution over this transport;
|
||||
- cross-host latency based on unsynchronised wall clocks;
|
||||
- long-duration soak, reconnect/recovery or worker-process restart;
|
||||
- forest/off-road domain accuracy;
|
||||
- vehicle-body extrinsics, navigation or safety use.
|
||||
|
||||
## Next gate
|
||||
|
||||
1. Attach the worker's persistent camera decoder to the camera-init/frame branch.
|
||||
2. Feed decoded frames to the already accepted capacity-2 10 Hz detector queue.
|
||||
3. Decode LiDAR/pose into their verified K1 schemas and carry the latest matched
|
||||
depth epoch into the existing E10 fusion runtime.
|
||||
4. Run YOLOX/Triton at 10 Hz and EoMT semantics at 2 Hz without writing per-frame
|
||||
images; publish only transient diagnostic world state and telemetry.
|
||||
5. Repeat the disconnect negative control with inference loaded.
|
||||
6. When K1 is awake, run the same profile first for 15 seconds and then for 60
|
||||
seconds against the physical acquisition path before any longer soak.
|
||||
@@ -0,0 +1,238 @@
|
||||
# LAB E13 — class-aware amodal 3D cuboids
|
||||
|
||||
Дата прогона: 2026-07-22
|
||||
Статус: accepted for recorded near-live qualification; not accepted for navigation or safety
|
||||
Основной результат: `e10-integrated-perception-eaf7427637326e74dbc5fe66aaffcab83d0a6c18c0c45b93fdc2ecff062b94ee`
|
||||
|
||||
## 1. Цель
|
||||
|
||||
LAB E13 заменяет диагностические p05–p95 оболочки LAB E10/E11 на полные 3D cuboids, пригодные для дальнейшей разработки «глаз» беспилотника. Старый fitter описывал только видимую LiDAR-поверхность: например, у автомобиля получалась длинная, но очень тонкая пластина. E13 должен:
|
||||
|
||||
- сохранить реальное LiDAR-свидетельство отдельно от достроенной геометрии;
|
||||
- получить полный class-aware объём car/truck/bus/person/bicycle/motorcycle;
|
||||
- привязать нижнюю грань к локальной оценке земли;
|
||||
- стабилизировать центр, размеры и yaw по `track_id`;
|
||||
- не публиковать старую оболочку как полноценный 3D object box, если достройка не прошла проверки;
|
||||
- остаться в ранее принятом near-live бюджете: detector около 10 FPS, semantic около 2 FPS, p95 возраста world state не более 175 ms.
|
||||
|
||||
E13 не утверждает, что невидимая часть объекта измерена. Полный cuboid явно маркируется как `class-prior-completed-from-visible-lidar-support`, а исходная измеренная оболочка сохраняется в `observed_cuboid_*`.
|
||||
|
||||
## 2. Входные данные
|
||||
|
||||
- Сессия: `20260720T065719Z_viewer_live` (`RAVNOVES00`).
|
||||
- Камера: `sensor.camera.right`, XGRIDS calibration slot `camera_1`.
|
||||
- Калибровка SHA-256: `05f3ad9b38b3a4fc95388a8ec83da83c745e217709e51787b3d5aad0969f6fa9`.
|
||||
- Camera compute job: `recorded-camera-602ac89026ed12978619801d`.
|
||||
- LiDAR pack: `e10-lidar-pack-5da0396d32a27f9d1ca537cc2e8a371d386078d6f0dc71737b78620992af9625`.
|
||||
- Кадры: 1000–1600 включительно, 601 кадр.
|
||||
- Временной интервал общей шкалы: 135.365857292–195.334857292 s.
|
||||
- Source span: 59.969 s.
|
||||
- Timeline SHA-256: `06e3514228b488137b92c303cbca171154ede00368783e32107ed2c39c41baa6`.
|
||||
- Valid-FOV mask: 270,606 из 480,000 пикселей.
|
||||
- Valid-FOV mask SHA-256: `a40cee06b7c6f69b6a09a11563dcfd237f3de833b1ccd31459e66692e528ba63`.
|
||||
|
||||
## 3. Вычислительный контур
|
||||
|
||||
- Worker: `<worker-host>`, выполнение только на диске `D:`.
|
||||
- GPU: NVIDIA GeForce RTX 4090.
|
||||
- Контейнер: `nvcr.io/nvidia/tritonserver:26.06-py3`.
|
||||
- Python 3.12.3; NumPy 1.26.4; SciPy 1.16.3; PyTorch 2.13.0+cu130; Transformers 4.57.6.
|
||||
- Detector: YOLOX-S, COCO-80, Triton, 640×640.
|
||||
- Detector model SHA-256: `c5c2d13e59ae883e6af3b45daea64af4833a4951c92d116ec270d9ddbe998063`.
|
||||
- Semantic model: `tue-mps/cityscapes_semantic_eomt_large_1024`, FP16 autocast.
|
||||
- Semantic model SHA-256: `c265da9a74f58f5c3f4826d23ca4ca78beac0b106cca5842beca61580de5b782`.
|
||||
- Source pace: 1.0×.
|
||||
- Detector queue: latest-wins capacity 2.
|
||||
- Semantic queue: latest-wins capacity 1; sampling every fifth camera frame.
|
||||
- Profile SHA-256: `1890d5039d93abea29bd75cf5609264d64b2c0b0fa5b032233fa5e45872776cd`.
|
||||
- Runner SHA-256: `bf04efa64c3614688742fb28d2da7c428ff70b4ca61a513318cd2440e79ff776`.
|
||||
- Fusion runtime SHA-256: `9724a4b5af719ea902f02a8a1efdb720e64b5af68b5759367bdc756bdd31b5c9`.
|
||||
- PowerShell orchestrator SHA-256: `ac0ee42495839773a2de60090ebcb49eee979dc7ee3b8587662a3c59e19a4d03`.
|
||||
|
||||
Перед финальным прогоном на `D:` было 375.612 GiB свободно при жёстком floor 360 GiB. После трёх сохранённых A/B-прогонов и публикации текущий объём свободного места составлял 375.317 GiB. Диск `C:` не использовался и не проверялся.
|
||||
|
||||
## 4. Алгоритм E13
|
||||
|
||||
### 4.1 Измеренная часть
|
||||
|
||||
Сохраняется прежний доказанный контур:
|
||||
|
||||
1. calibrated KB4 projection LiDAR → camera;
|
||||
2. nearest-depth-per-rounded-pixel occlusion gate;
|
||||
3. пересечение 2D track, semantic class mask и LiDAR projection;
|
||||
4. depth-contiguous cluster;
|
||||
5. spatial connected cluster;
|
||||
6. robust oriented p05–p95 envelope как `observed_cuboid`.
|
||||
|
||||
### 4.2 Достройка полного объёма
|
||||
|
||||
- Для каждого detector label задан nominal/min/max размер и support padding.
|
||||
- Видимая грань достраивается от сенсора, то есть невидимый объём продолжается за измеренной поверхностью, а не симметрично в обе стороны.
|
||||
- Для вытянутых классов fitter различает длинную боковую поверхность и короткую фронтальную/заднюю грань. При короткой грани yaw поворачивается на 90° по нормали поверхности.
|
||||
- При слабой анизотропии ориентация берётся из track history; при отсутствии истории используется sensor-bearing fallback.
|
||||
- Локальная земля оценивается по нижнему процентилю полного облака в радиусе 2.5 m вокруг объекта. Нереалистичная оценка заменяется ограниченным fallback относительно нижней границы support points.
|
||||
- Размеры ограничиваются физическими class-bounds. Выход за bound означает rejection, а не расширение коробки до заведомо ложного размера.
|
||||
- Центр, размер и yaw фильтруются по `track_id`.
|
||||
- Cuboid должен содержать не менее 75% текущих support points.
|
||||
- Если temporal smoothing уводит box от текущих точек, фильтр сбрасывается на текущий несглаженный candidate.
|
||||
- `failure_policy=reject`: если полный cuboid нельзя построить честно, 2D detection и observed support остаются в журнале, но 3D box не публикуется.
|
||||
|
||||
### 4.3 Классовые nominal sizes
|
||||
|
||||
| Detector label | Nominal L×W×H, m | Допустимый диапазон, m |
|
||||
|---|---:|---:|
|
||||
| person | 0.55×0.55×1.72 | 0.35–1.20 × 0.35–1.20 × 1.30–2.30 |
|
||||
| bicycle | 1.80×0.65×1.50 | 1.20–2.50 × 0.40–1.20 × 1.00–2.20 |
|
||||
| motorcycle | 2.10×0.80×1.45 | 1.40–3.00 × 0.50–1.40 × 1.00–2.20 |
|
||||
| car | 4.50×1.85×1.55 | 3.20–5.80 × 1.45–2.40 × 1.20–2.30 |
|
||||
| truck | 7.00×2.50×3.00 | 4.80–12.50 × 1.80–3.20 × 1.80–4.20 |
|
||||
| bus | 10.50×2.55×3.20 | 7.00–13.50 × 2.10–3.20 × 2.50–4.20 |
|
||||
|
||||
## 5. Итерации A/B
|
||||
|
||||
### E13.1 — completion с визуальным fallback
|
||||
|
||||
- Result: `e10-integrated-perception-54e660dc90333f5d38b9ad7054e05e32a808e268080220db20d2d2a1141cc2c3`.
|
||||
- 494 опубликованных 3D boxes: 382 class-prior-completed и 112 fallback visible envelopes.
|
||||
- World-state p95 age: 97.876 ms.
|
||||
- Решение отклонено как финальное: интерфейс смешивал полные cuboids и старые короткие оболочки.
|
||||
|
||||
### E13.2 — fail-closed и yaw axis hysteresis
|
||||
|
||||
- Result: `e10-integrated-perception-b35c3e94e8c8614ad4cf51a5b9a41ebe5a7be856aa1b3cc837b2d9bd6f05e8fe`.
|
||||
- Только 334 class-prior-completed boxes опубликованы.
|
||||
- 81 rejection по support coverage и 79 rejection по class-size bounds.
|
||||
- 90°/innovation resets снижены с 76 до 16.
|
||||
- World-state p95 age: 100.072 ms.
|
||||
- Найден дефект: при coverage reset смешивались новый центр/yaw и ранее сглаженный размер.
|
||||
|
||||
### E13.3 — полный current-candidate reset
|
||||
|
||||
- Result: `e10-integrated-perception-eaf7427637326e74dbc5fe66aaffcab83d0a6c18c0c45b93fdc2ecff062b94ee`.
|
||||
- 375 class-prior-completed boxes опубликованы.
|
||||
- Coverage rejections снижены с 81 до 39.
|
||||
- Class-bound rejections: 81; они сохранены как корректный fail-closed outcome.
|
||||
- Innovation resets: 15.
|
||||
- Финальная итерация принята.
|
||||
|
||||
## 6. Финальные результаты
|
||||
|
||||
### 6.1 Throughput и latency
|
||||
|
||||
| Метрика | LAB E10 baseline, тот же clip | LAB E13.3 | Gate |
|
||||
|---|---:|---:|---:|
|
||||
| Detector effective FPS | 9.996 | 9.986 | ≥9.5 |
|
||||
| Detector dropped frames | 0 | 0 | 0 |
|
||||
| Semantic effective FPS | 2.013 | 2.011 | ≥1.8 |
|
||||
| Fresh semantic coverage | 99.33% | 99.00% | ≥90% |
|
||||
| Fused frames | 525 | 523 | ≥450 |
|
||||
| Published 3D boxes | 760 visible envelopes | 375 completed cuboids | ≥100 |
|
||||
| Association p50 | 1.391 ms | 1.751 ms | diagnostic |
|
||||
| Association p95 | 3.858 ms | 5.184 ms | diagnostic |
|
||||
| World-state age p50 | 52.017 ms | 68.547 ms | diagnostic |
|
||||
| World-state age p95 | 76.515 ms | 101.829 ms | ≤175 ms |
|
||||
| Replay wall time | 60.123 s | 60.184 s | about 1.0× |
|
||||
|
||||
Добавленная геометрия увеличила p95 association примерно на 1.33 ms. Разница полного world-state age между отдельными запусками также содержит вариативность decode, detector и semantic runtimes; её нельзя целиком приписывать cuboid fitter.
|
||||
|
||||
### 6.2 Ресурсы
|
||||
|
||||
- Process peak RSS: 2022.719 MiB.
|
||||
- CUDA peak allocated: 2107.925 MiB.
|
||||
- CUDA peak reserved: 2840 MiB.
|
||||
- GPU telemetry mean/max utilization: 63.77% / 79%.
|
||||
- GPU telemetry mean/max memory used: 13,381 / 13,391 MiB.
|
||||
- GPU power mean/max: 215.40 / 228.72 W.
|
||||
- GPU temperature mean/max: 46 / 49 °C.
|
||||
|
||||
### 6.3 Геометрия и качество фильтра
|
||||
|
||||
- Accepted cuboids: 375.
|
||||
- По классам: car 267; truck 100; person 6; bicycle 2; motorcycle/bus 0 в accepted set.
|
||||
- Mean support coverage: 85.82%.
|
||||
- Mean completion fraction: 92.67%; p50 95.06%; p95 99.46%.
|
||||
- Средний accepted car box: 4.536×1.891×1.569 m.
|
||||
- Средний accepted truck box: 7.080×2.500×3.001 m.
|
||||
- Средний accepted person box: 0.572×0.618×1.720 m.
|
||||
- Temporal statuses: confirmed 185; tentative 115; reset-support-coverage 60; reset-innovation 15.
|
||||
- Orientation sources: support-PCA 196; support-face-normal 129; track-history axis disambiguation 48; track-history 1; sensor-bearing fallback 1.
|
||||
|
||||
На 256 соседних accepted observations одного track с разрывом не более 0.25 s:
|
||||
|
||||
| Temporal metric | Measured visible envelope | Completed E13 cuboid |
|
||||
|---|---:|---:|
|
||||
| Median size step | 0.548 m | 0.000 m |
|
||||
| p95 size step | 2.167 m | 0.152 m |
|
||||
| Median yaw step | 4.67° | 1.31° |
|
||||
| p95 yaw step | 58.61° | 22.10° |
|
||||
|
||||
Это не является сравнением с ground truth, но подтверждает, что размер и yaw перестали повторять шум каждой отдельной видимой поверхности.
|
||||
|
||||
### 6.4 Fail-closed accounting
|
||||
|
||||
- `rejected-fewer-than-8-clustered-points`: 1288 vehicle observations.
|
||||
- `rejected-fewer-than-4-clustered-points`: 66 non-vehicle observations.
|
||||
- `rejected-no-semantic-lidar-support`: 590.
|
||||
- `rejected-distance-innovation`: 98.
|
||||
- `rejected-amodal-completion-required-size-exceeds-class-bound`: 81.
|
||||
- `rejected-amodal-completion-support-coverage-below-threshold`: 39.
|
||||
- `rejected-implausible-cuboid`: 12.
|
||||
|
||||
Высокое число rejected candidates ожидаемо: E13 сознательно уменьшает визуальный мусор и публикует только полный cuboid, прошедший semantic, clustering, distance, class-size и support-coverage gates.
|
||||
|
||||
## 7. Визуальная проверка и публикация
|
||||
|
||||
- Результат прошёл `validate_integrated_perception_result` локально.
|
||||
- Rerun perception overlay: 58,326,068 bytes; API build time 9.778 s; HTTP 200.
|
||||
- E13 опубликован как newest accepted overlay для `RAVNOVES00`.
|
||||
- Проверено в Mission Core на общей шкале около 02:27: cloud, trajectory, recorded camera, semantic overlay и полные полупрозрачные 3D cuboids отображаются совместно.
|
||||
- Принятый вид оставлен открытым в локальном интерфейсе.
|
||||
|
||||
## 8. Артефакты
|
||||
|
||||
Worker root:
|
||||
|
||||
`D:\NDC_MISSIONCORE\runtime\derived\e10-integrated-perception-eaf7427637326e74dbc5fe66aaffcab83d0a6c18c0c45b93fdc2ecff062b94ee`
|
||||
|
||||
Local laboratory copy:
|
||||
|
||||
`.runtime/compute-experiments/e13/worker-results/e10-integrated-perception-eaf7427637326e74dbc5fe66aaffcab83d0a6c18c0c45b93fdc2ecff062b94ee`
|
||||
|
||||
Published overlay source:
|
||||
|
||||
`.runtime/compute-experiments/e10/worker-results/e10-integrated-perception-eaf7427637326e74dbc5fe66aaffcab83d0a6c18c0c45b93fdc2ecff062b94ee`
|
||||
|
||||
| Artifact | Bytes | SHA-256 |
|
||||
|---|---:|---|
|
||||
| `result.json` | — | `512f5fcb72d3c10f4d3d6beb081b28dbb65addcd166e7d4a4bdc3f55a63be4c6` |
|
||||
| `semantic-frames.jsonl` | 55,023 | `43e72dc9f3ec9fc74f4e1c69dc1ba2fd900ac445aa1744f8d5ea2cc2165cd52b` |
|
||||
| `fusion-frames.jsonl` | 2,096,208 | `ed8256b41c8a56b8d1dd561e6c47479932a913c80e64cf73c4dc761722637acf` |
|
||||
| `world-state.jsonl` | 1,336,724 | `a418d1b90fddd810b3ec3fa20c6e5bfa14ecaf2f348500f4a9b56a761a2063d0` |
|
||||
| `transient-perception.npz` | 1,065,071 | `a68e5bd06eee4f2372eacaf0d94326515b94848189e6527d063002398d0c0f56` |
|
||||
| `gpu-telemetry.jsonl` | 13,704 | `a80cced041a87446cfb9e074a0867ab5f980108f826b347ca5175664d2b9c907` |
|
||||
| `run-report.json` | 17,870 | `b6e67b710e6a2f9d3132a0293ac1fae54ca688b6a1ebbc42174893388c0ab88d` |
|
||||
|
||||
## 9. Ограничения
|
||||
|
||||
1. Это recorded source-paced qualification, а не физический live K1 → worker прогон.
|
||||
2. Host-arrival camera/LiDAR binding не доказывает hardware-clock synchronization.
|
||||
3. COCO YOLOX и Cityscapes EoMT не являются forest-domain или safety-validated моделями.
|
||||
4. Средняя доля достроенного объёма 92.67% показывает, что текущие «нормальные» cuboids в основном опираются на class priors. Это инженерный amodal estimate, не измеренная форма объекта.
|
||||
5. 3D ground truth отсутствует. Нельзя утверждать 3D IoU, heading accuracy, center error или recall относительно истины.
|
||||
6. Truck detections наследуют ограниченность COCO label и могут включать крупные автомобили, которые в конкретном кадре классифицированы неточно.
|
||||
7. Current local-ground percentile не заменяет полноценную road/terrain plane model для уклонов, бордюров и лесной поверхности.
|
||||
8. E13 не имеет navigation/control authority и не принят для safety.
|
||||
|
||||
## 10. Решение и следующий gate
|
||||
|
||||
E13.3 принят как новый визуальный и world-state baseline вместо visible-surface boxes. Он доказывает, что class-aware full cuboids и temporal stabilization укладываются в near-live бюджет на существующем worker.
|
||||
|
||||
Следующий обязательный этап — LAB E14:
|
||||
|
||||
1. добавить 3D ground-truth/prelabel contour хотя бы для репрезентативного набора кадров;
|
||||
2. провести внешний benchmark learned LiDAR/BEV 3D detectors (CenterPoint/PointPillars и camera-LiDAR fusion вариант) на том же K1 point pattern;
|
||||
3. сравнить learned box с E13 prior-completed baseline по latency, 3D IoU/center/yaw и стабильности;
|
||||
4. подключить принятый fitter к bounded E12 shadow ingress и проверить disconnect/overflow вместе с inference;
|
||||
5. после этого выполнить физический live K1 gate без control/navigation authority.
|
||||
|
||||
До появления 3D labels E13 следует использовать как честно маркированный near-live amodal baseline, а не как финальное решение 3D detection.
|
||||
@@ -0,0 +1,276 @@
|
||||
# LAB E14 — full-session recorded near-live qualification
|
||||
|
||||
Дата прогона: 2026-07-22
|
||||
Статус вычислительного прогона: accepted for recorded near-live qualification
|
||||
Статус physical live K1: not tested
|
||||
Navigation/safety acceptance: false
|
||||
Основной результат: `e10-integrated-perception-9c10dc36640b409393c161f6cbd59b000581b39851ef7a8931898cf8f46d7e8b`
|
||||
|
||||
## 1. Цель
|
||||
|
||||
LAB E14 переносит принятый в E13 class-aware amodal 3D cuboid contour с минутной выборки на всю доступную запись `RAVNOVES00`. Эксперимент отвечает на практический вопрос: может ли текущий RTX 4090 worker непрерывно обрабатывать полный recorded stream в темпе источника, одновременно выполняя:
|
||||
|
||||
- декодирование fisheye-видео;
|
||||
- YOLOX-S 2D detection на каждом camera frame;
|
||||
- EoMT semantic segmentation на каждом пятом frame;
|
||||
- calibrated KB4 camera–LiDAR projection;
|
||||
- semantic/LiDAR association;
|
||||
- class-prior amodal 3D cuboid completion и temporal stabilization;
|
||||
- clearance и world-state publication;
|
||||
- GPU/runtime telemetry;
|
||||
- fail-closed accounting без скрытых потерь очередей.
|
||||
|
||||
E14 является capacity qualification на записанном источнике. Он не заменяет физический live K1 gate, не подтверждает hardware-clock synchronization и не выдаёт контуру право управлять беспилотником.
|
||||
|
||||
## 2. Входные данные
|
||||
|
||||
- Сессия: `20260720T065719Z_viewer_live`, операторское имя `RAVNOVES00`.
|
||||
- Камера: `sensor.camera.right`.
|
||||
- XGRIDS calibration slot: `camera_1`.
|
||||
- Калибровка SHA-256: `05f3ad9b38b3a4fc95388a8ec83da83c745e217709e51787b3d5aad0969f6fa9`.
|
||||
- Camera compute job: `recorded-camera-602ac89026ed12978619801d`.
|
||||
- Camera input SHA-256: `602ac89026ed12978619801d4edea0cae24b5cc3afabd9f7af2858de6505a20e`.
|
||||
- LiDAR pack: `e10-lidar-pack-576c994a6c814e2592dd6240ace3902a5db94843312c759a73ba0c9166157d2b`.
|
||||
- Кадры: 0–4488 включительно, 4489 camera frames.
|
||||
- Timeline: 35.421857292–484.044857292 s.
|
||||
- Source span: 448.623 s.
|
||||
- Timeline SHA-256: `f91dad0a3cc998f4250be794b48f437943b3c60edccbe7e5ceed59030854a807`.
|
||||
- Разрешение: 800×600.
|
||||
- Valid-FOV mask: 270,606 из 480,000 пикселей, 56.38% кадра.
|
||||
- Valid-FOV mask SHA-256: `a40cee06b7c6f69b6a09a11563dcfd237f3de833b1ccd31459e66692e528ba63`.
|
||||
|
||||
Valid-FOV mask вычислена заранее из стабильной геометрии объектива и переиспользуется. За пределами полезного круга detector/semantic contour не получает содержательного изображения; маска не пересчитывается для каждого кадра.
|
||||
|
||||
## 3. Воспроизводимая конфигурация
|
||||
|
||||
### 3.1 Worker и модели
|
||||
|
||||
- Worker host: `<worker-host>`.
|
||||
- GPU: NVIDIA GeForce RTX 4090.
|
||||
- Worker storage: только `D:\NDC_MISSIONCORE`.
|
||||
- Диск `C:` не использовался и не проверялся.
|
||||
- Container image: `nvcr.io/nvidia/tritonserver:26.06-py3`.
|
||||
- Python 3.12.3; NumPy 1.26.4; SciPy 1.16.3; PyTorch 2.13.0+cu130; Transformers 4.57.6.
|
||||
- Detector: YOLOX-S, COCO-80, Triton, 640×640.
|
||||
- Detector model SHA-256: `c5c2d13e59ae883e6af3b45daea64af4833a4951c92d116ec270d9ddbe998063`.
|
||||
- Semantic: `tue-mps/cityscapes_semantic_eomt_large_1024`, FP16 autocast.
|
||||
- Semantic model revision: `8d6b6d1a3f7b50d441afd7d247c2ed10db186e8f`.
|
||||
- Semantic model SHA-256: `c265da9a74f58f5c3f4826d23ca4ca78beac0b106cca5842beca61580de5b782`.
|
||||
|
||||
### 3.2 Scheduler и профили
|
||||
|
||||
- Source pace: 1.0×.
|
||||
- Detector: каждый frame, latest-wins queue capacity 2.
|
||||
- Semantic: каждый пятый frame, latest-wins queue capacity 1.
|
||||
- Semantic TTL: 750 ms.
|
||||
- Profile: `experiments/perception/worker/e14_full_session_amodal_profile.json`.
|
||||
- Profile SHA-256: `a010e32070459b123423f3d4f57aa1d839f73ed05f3df8580d12693406a1b218`.
|
||||
- Detector profile SHA-256: `819bdfb0da521d187ec9ce3d3039a0ff75daa0ab473fb3d8e5fa92db9c1ddca1`.
|
||||
- Semantic profile SHA-256: `ea583966bc3409f5cf563cbf4fad05e366907e67187082eb692aff53d9f5d875`.
|
||||
- Runner SHA-256: `bf04efa64c3614688742fb28d2da7c428ff70b4ca61a513318cd2440e79ff776`.
|
||||
- Fusion runtime SHA-256: `9724a4b5af719ea902f02a8a1efdb720e64b5af68b5759367bdc756bdd31b5c9`.
|
||||
- PowerShell orchestrator SHA-256: `ac0ee42495839773a2de60090ebcb49eee979dc7ee3b8587662a3c59e19a4d03`.
|
||||
- Runner generation: `D:\NDC_MISSIONCORE\runtime\derived\e14-runner-bf04efa64c36-ac0ee4249583-a010e3207045`.
|
||||
|
||||
### 3.3 Acceptance gates
|
||||
|
||||
| Gate | Порог |
|
||||
|---|---:|
|
||||
| Detector effective FPS | ≥9.5 |
|
||||
| Detector drop fraction | ≤0.5% |
|
||||
| Semantic effective FPS | ≥1.8 |
|
||||
| Semantic drop fraction | ≤5% |
|
||||
| Semantic completion age p95 | ≤400 ms |
|
||||
| Fresh semantic coverage | ≥90% |
|
||||
| World-state age p95 | ≤175 ms |
|
||||
| Fused frames | ≥3500 |
|
||||
| Accepted completed cuboids | ≥1500 |
|
||||
| Detector/semantic failures | 0 |
|
||||
|
||||
## 4. Выполнение и время
|
||||
|
||||
- Подготовка worker input/runtime: 31.249 s.
|
||||
- Source-paced replay внутри runner: 448.699 s.
|
||||
- Source duration: 448.623 s.
|
||||
- Replay/source ratio: 1.00017×, то есть вычислительный цикл удержал темп записи.
|
||||
- Полный PowerShell orchestration wall time: 597.752 s.
|
||||
|
||||
Разница между replay и orchestration включает подготовку, запуск/проверку контейнерного контура, сбор и валидацию артефактов, восстановление model state и финальную уборку. Она не является inference latency кадра.
|
||||
|
||||
Контур выполнялся параллельно: detector и semantic workers имели отдельные bounded queues; fusion/world-state потреблял detector result на camera cadence и последнюю допустимую semantic result. Медленная semantic модель не блокировала detector loop и не накапливала безразмерную очередь.
|
||||
|
||||
## 5. Итоговые near-live метрики
|
||||
|
||||
| Метрика | Результат | Gate | Статус |
|
||||
|---|---:|---:|---|
|
||||
| Detector frames | 4489/4489 | полное accounting | pass |
|
||||
| Detector effective FPS | 10.0045 | ≥9.5 | pass |
|
||||
| Detector drops | 0 | ≤0.5% | pass |
|
||||
| Detector queue max depth | 1 из 2 | bounded | pass |
|
||||
| Semantic frames | 898 | every fifth frame | pass |
|
||||
| Semantic effective FPS | 2.0013 | ≥1.8 | pass |
|
||||
| Semantic drops | 0 | ≤5% | pass |
|
||||
| Fresh semantic coverage | 99.7104% | ≥90% | pass |
|
||||
| Semantic completion age p95 | 261.259 ms | ≤400 ms | pass |
|
||||
| Fused frames | 3915 | ≥3500 | pass |
|
||||
| Accepted 3D cuboids | 3656 | ≥1500 | pass |
|
||||
| World-state age p50 / p95 | 67.439 / 102.883 ms | p95 ≤175 ms | pass |
|
||||
| Failures | 0 | 0 | pass |
|
||||
|
||||
Fusion state accounting:
|
||||
|
||||
- `fused`: 3915;
|
||||
- `depth-unavailable-sync-gate`: 561;
|
||||
- `semantic-stale`: 8;
|
||||
- `semantic-unavailable`: 5;
|
||||
- semantic freshness violations: 0.
|
||||
|
||||
## 6. Latency decomposition
|
||||
|
||||
| Stage | Mean, ms | p50, ms | p95, ms | Max, ms |
|
||||
|---|---:|---:|---:|---:|
|
||||
| Decode | 33.438 | 31.333 | 48.551 | 77.638 |
|
||||
| Detector | 31.502 | 28.524 | 47.158 | 65.501 |
|
||||
| Detector queue wait | 0.202 | 0.074 | 0.200 | 23.505 |
|
||||
| Projection | 0.476 | 0.397 | 0.884 | 12.028 |
|
||||
| Association + cuboid | 2.575 | 1.964 | 6.834 | 26.413 |
|
||||
| Clearance | 0.222 | 0.187 | 0.517 | 5.949 |
|
||||
| World-state serialization | 0.075 | 0.046 | 0.233 | 0.964 |
|
||||
| End-to-end world-state age | 70.425 | 67.439 | 102.883 | 182.733 |
|
||||
|
||||
Semantic processing:
|
||||
|
||||
| Stage | Mean, ms | p95, ms | Max, ms |
|
||||
|---|---:|---:|---:|
|
||||
| Host → device | 6.869 | 10.309 | 18.386 |
|
||||
| Forward | 154.711 | 167.953 | 407.725 |
|
||||
| Model postprocess | 9.825 | 14.110 | 20.840 |
|
||||
| Full semantic processing | 188.479 | 206.391 | 434.399 |
|
||||
| Completion age | 224.099 | 261.259 | 467.008 |
|
||||
|
||||
## 7. 3D geometry
|
||||
|
||||
Accepted class-prior-completed cuboids: 3656.
|
||||
|
||||
| Class | Count | Mean L×W×H, m |
|
||||
|---|---:|---:|
|
||||
| car | 3118 | 4.514×1.881×1.575 |
|
||||
| truck | 493 | 7.016×2.514×3.125 |
|
||||
| person | 31 | 0.622×0.588×1.720 |
|
||||
| bus | 10 | 10.500×2.631×3.200 |
|
||||
| bicycle | 2 | 1.800×1.016×1.500 |
|
||||
| motorcycle | 2 | 2.100×0.874×1.450 |
|
||||
|
||||
- Mean support coverage: 86.19%.
|
||||
- Mean inferred completion fraction: 91.56%.
|
||||
- Temporal statuses: confirmed 2070; tentative 920; support-coverage reset 574; innovation reset 92.
|
||||
- Orientation: support-PCA 1890; support-face-normal 1034; track-history axis disambiguation 702; track-history 23; sensor-bearing fallback 7.
|
||||
|
||||
Высокая completion fraction означает, что значительная часть полного объёма достроена по class priors из видимой LiDAR-поверхности. Это диагностическая геометрия с измеренным support audit, а не 3D ground truth.
|
||||
|
||||
Fail-closed rejections:
|
||||
|
||||
- fewer than 8 clustered vehicle points: 10,563;
|
||||
- no semantic/LiDAR support: 4,884;
|
||||
- distance innovation: 505;
|
||||
- required size exceeds class bound: 395;
|
||||
- support coverage below threshold: 306;
|
||||
- fewer than 4 clustered non-vehicle points: 184;
|
||||
- implausible cuboid: 20.
|
||||
|
||||
Rejection сохраняет 2D/audit evidence, но не публикует ложный полноценный 3D box.
|
||||
|
||||
## 8. Ресурсы и диски
|
||||
|
||||
- Process peak RSS: 2964.414 MiB.
|
||||
- CUDA peak allocated: 2107.925 MiB.
|
||||
- CUDA peak reserved: 2840 MiB.
|
||||
- GPU utilization mean/p50/p95/max: 74.59% / 75% / 91% / 98%.
|
||||
- GPU memory used mean/p50/p95/max: 13,274.69 / 13,273 / 13,282 / 13,284 MiB.
|
||||
- GPU memory utilization mean/p95/max: 22.67% / 32% / 36%.
|
||||
- GPU power mean/p95/max: 200.44 / 208.43 / 225.21 W.
|
||||
- GPU temperature mean/p95/max: 49.09 / 54 / 55 °C.
|
||||
- Telemetry samples: 449 at 1 s interval.
|
||||
|
||||
На `D:` до staging/preparation было приблизительно 389.65 GiB свободно. Runner стартовал с 415,364,739,072 bytes и закончил с 415,598,956,544 bytes при hard floor 386,547,056,640 bytes (360 GiB). После orchestration и уборки оставалось около 387.05 GiB. Временные camera frames удалены оркестратором; model state восстановлен. Диск `C:` не затрагивался.
|
||||
|
||||
На локальном Mac опубликованный Rerun overlay занимает 443,421,386 bytes. На момент отчёта локально оставалось около 32 GiB; лишние копии видео не создавались.
|
||||
|
||||
## 9. Публикация и интерфейсная проверка
|
||||
|
||||
- Результат дважды прошёл строгую локальную `validate_integrated_perception_result`: до и после immutable publication в canonical provider root.
|
||||
- Full-session perception RRD: 443,421,386 bytes.
|
||||
- RRD SHA-256: `1dc29900ddeb09897af70c7334755823b02aeaa95f203d979be4bd75fae5080b`.
|
||||
- Mission Core API вернул HTTP 200 и сохранил immutable result-scoped cache.
|
||||
- Сервис не перезапускался и оставался healthy во время построения и браузерной проверки.
|
||||
- В Mission Core подтверждена полная шкала `00:00 / 08:55.718`.
|
||||
- В режиме `Распознавание` исходная камера и `Сегментация · камера right` синхронно менялись при воспроизведении; это не статичный кадр.
|
||||
|
||||
Во время проверки обнаружен presentation defect режима `Кубы 3D`: blueprint имел origin `/world/perception`, поэтому исключал native cloud `/world/points` и показывал cuboids в отдельной пустой сцене. Исправление выполнено в source:
|
||||
|
||||
- cuboid view перенесён на origin `/world`;
|
||||
- native cloud и `/world/perception` объединены в одном latest-at view;
|
||||
- 12-second accumulation к cuboid view не применяется, поэтому коробки разных кадров не наслаиваются;
|
||||
- дублирующий серый `/world/perception/lidar` скрыт;
|
||||
- добавлены regression assertions для initial и dynamic blueprints.
|
||||
|
||||
Backend не перезапускался из-за операторского требования никогда не оставлять Mission Core недоступным. Поэтому вычислительный 3D result принят, source fix протестирован, но визуальная runtime-проверка исправленного blueprint остаётся отдельным коротким gate после следующего штатного перезапуска сервиса.
|
||||
|
||||
## 10. Артефакты
|
||||
|
||||
Worker result root:
|
||||
|
||||
`D:\NDC_MISSIONCORE\runtime\derived\e10-integrated-perception-9c10dc36640b409393c161f6cbd59b000581b39851ef7a8931898cf8f46d7e8b`
|
||||
|
||||
Local laboratory copy:
|
||||
|
||||
`.runtime/compute-experiments/e14/worker-results/e10-integrated-perception-9c10dc36640b409393c161f6cbd59b000581b39851ef7a8931898cf8f46d7e8b`
|
||||
|
||||
Published provider root:
|
||||
|
||||
`.runtime/compute-experiments/e10/worker-results/e10-integrated-perception-9c10dc36640b409393c161f6cbd59b000581b39851ef7a8931898cf8f46d7e8b`
|
||||
|
||||
| Artifact | Bytes | SHA-256 |
|
||||
|---|---:|---|
|
||||
| `result.json` | 13,799 | `ff129ee1e51e19b514ffb790a145cf17292bd49e9256de2a77a2ca2557e50e68` |
|
||||
| `run-report.json` | 18,144 | `a85a714215358715120efb057991087f8b214fe6ef86ce2f68124f19287f2a4d` |
|
||||
| `semantic-frames.jsonl` | 404,398 | `0156bd7450cf6e89fcc3e66831377e87089c6f322a0db382d7135246241cf6b3` |
|
||||
| `fusion-frames.jsonl` | 17,008,643 | `04c6589c0d5258b47937114c80a1562b6a427456b22e8aedb8cfa86741fe183b` |
|
||||
| `world-state.jsonl` | 10,699,803 | `5b030270c4ae9029690666cbde7e6191df4129f393a8530dc561b76a68b61f89` |
|
||||
| `transient-perception.npz` | 8,291,832 | `36677d1b2f4260f821dcf29929dc0dea90b28a4e74ff5c777d320dda7213f659` |
|
||||
| `gpu-telemetry.jsonl` | 100,917 | `d4fb026b46d25b97b7d35397d598fec23647276af6339c98898cbfea97345601` |
|
||||
| full-session `perception.rrd` | 443,421,386 | `1dc29900ddeb09897af70c7334755823b02aeaa95f203d979be4bd75fae5080b` |
|
||||
|
||||
## 11. Что доказано и что не доказано
|
||||
|
||||
Доказано:
|
||||
|
||||
1. На RTX 4090 текущий YOLOX + EoMT + calibrated LiDAR fusion + E13 cuboid/world-state contour удерживает полный 448.623-second recorded stream в 1.0×.
|
||||
2. Detector и semantic queues bounded; за 4489 frames нет ни overflow drops, ни model failures.
|
||||
3. p95 world-state age 102.883 ms оставляет запас до лабораторного 175 ms gate.
|
||||
4. Full-session 2D/semantic playback опубликован и работает на общей временной шкале Mission Core.
|
||||
5. Все 3656 опубликованных 3D boxes прошли class bounds и support gates; rejected candidates не маскируются.
|
||||
|
||||
Не доказано:
|
||||
|
||||
1. Физический K1 live ingress → decode → inference → fusion на реальном сетевом потоке.
|
||||
2. Hardware-clock camera/LiDAR synchronization.
|
||||
3. Устойчивость к реальному disconnect/reconnect при работающем inference.
|
||||
4. Forest-domain качество COCO/Cityscapes моделей.
|
||||
5. 3D IoU, center/yaw error и recall относительно ground truth.
|
||||
6. Navigation/safety authority.
|
||||
|
||||
## 12. Следующий gate
|
||||
|
||||
Следующий этап не должен улучшать recorded-видео в вакууме. Он переносит доказанный E14 compute contour на уже принятый E12 ingress:
|
||||
|
||||
1. подключить persistent fMP4 camera decoder к bounded shadow transport;
|
||||
2. выполнять detector на 10 Hz и semantic на 2 Hz через latest-wins queues;
|
||||
3. декодировать LiDAR/pose и публиковать тот же calibrated fusion/world-state contract;
|
||||
4. вывести live telemetry: source age, decode/inference/fusion age, queue depth, drops, semantic freshness, reconnect generation;
|
||||
5. выполнить физический K1 shadow gate 15 s;
|
||||
6. при нулевых gaps/failures выполнить 60 s gate;
|
||||
7. только после этого увеличивать длительность и проверять disconnect/reconnect;
|
||||
8. control/navigation output оставить физически отключённым.
|
||||
|
||||
Отдельная короткая presentation-проверка после следующего штатного перезапуска Mission Core должна подтвердить, что исправленный `Кубы 3D` показывает latest-at cuboids непосредственно поверх native point cloud.
|
||||
@@ -0,0 +1,351 @@
|
||||
# LAB E15 — source-paced live shadow inference qualification
|
||||
|
||||
Дата: 22 июля 2026 года
|
||||
Финальный accepted run: 20:23–20:26 MSK / `2026-07-22T17:26:37Z` source completion
|
||||
Статус: **accepted для recorded-source-paced shadow diagnostic**
|
||||
Safety/navigation authority: **не выдана**
|
||||
|
||||
## 1. Что проверялось
|
||||
|
||||
LAB E15 впервые соединяет принятые ранее части Mission Core в один потоковый контур, близкий к будущему onboard perception loop:
|
||||
|
||||
1. bounded Mission Core shadow ingress;
|
||||
2. инкрементальный decode сохранённого fMP4 без сборки видео целиком и без PNG на диске;
|
||||
3. YOLOX-S object detection на каждом camera frame;
|
||||
4. EoMT semantic segmentation на каждом пятом camera frame;
|
||||
5. nearest-time camera → LiDAR → pose binding;
|
||||
6. KB4 calibrated camera–LiDAR projection;
|
||||
7. semantic/LiDAR association;
|
||||
8. class-prior amodal 3D cuboids;
|
||||
9. clearance и timestamped world-state;
|
||||
10. bounded queues, telemetry и fail-closed acceptance.
|
||||
|
||||
Это не offline-просчёт «как можно качественнее». Источник отдавал 15 секунд реальной записи в темпе 1.0×, а worker обязан был удерживать camera cadence, bounded memory и ограничение на возраст результата.
|
||||
|
||||
## 2. Граница доказательства
|
||||
|
||||
Доказано:
|
||||
|
||||
- один RTX 4090 worker принимает camera/LiDAR/pose через live shadow wire contract;
|
||||
- 151 fMP4 fragment декодируется инкрементально без потерь и без временных frame-файлов;
|
||||
- detector работает практически на camera cadence 10 Hz;
|
||||
- semantic contour работает примерно на 2 Hz и не блокирует detector;
|
||||
- LiDAR/pose fusion и 3D cuboid/world-state укладываются в заданный p95 age budget;
|
||||
- transport, model queues и media buffer остаются bounded;
|
||||
- disconnect/replay transport не вмешивается в raw recorder;
|
||||
- весь persistent worker contour и результаты размещены только на `D:`.
|
||||
|
||||
Не доказано:
|
||||
|
||||
- работа от физически подключённого K1 в этом конкретном E15 run;
|
||||
- hardware-clock synchronization камеры и LiDAR;
|
||||
- forest-domain accuracy, recall или safety quality;
|
||||
- пригодность COCO/Cityscapes labels для автономной навигации;
|
||||
- vehicle-body transform;
|
||||
- управление беспилотником или safety authority;
|
||||
- публикация текущего live world-state в операторский Rerun UI.
|
||||
|
||||
Итоговый scope результата: `live-shadow-diagnostic-only`.
|
||||
|
||||
## 3. Входные данные и provenance
|
||||
|
||||
### 3.1 Источник
|
||||
|
||||
- Session: `20260720T065719Z_viewer_live` (`RAVNOVES00`).
|
||||
- Camera source: `sensor.camera.right`.
|
||||
- Camera calibration slot: `camera_1`.
|
||||
- Camera resolution: 800×600.
|
||||
- Source interval: 14.989973 s.
|
||||
- Replay pace: 1.0× recorded host arrival time.
|
||||
- Camera frames: 151.
|
||||
- LiDAR messages: 142.
|
||||
- Pose messages: 150.
|
||||
- Control messages: 2.
|
||||
- Total wire events: 446.
|
||||
- Camera index SHA-256: `e029815a60ad9fbfedb6169142c7449df2b119a51d1ce001f08806e04eb0be14`.
|
||||
- MQTT raw SHA-256: `70da0edad5cbf0e89b6e2355c0aac294f8c36a67249cd33b8918353fd88c83af`.
|
||||
- Accepted source report: `.runtime/compute-experiments/e12/20260722T172637Z-e12-source.json`.
|
||||
- Accepted source report SHA-256: `8d36a9715b19fcd820e585629e0774d97ea21dfd5b763fd48e9672e098814e48`.
|
||||
|
||||
### 3.2 Калибровка
|
||||
|
||||
- Calibration SHA-256: `05f3ad9b38b3a4fc95388a8ec83da83c745e217709e51787b3d5aad0969f6fa9`.
|
||||
- Projection model: KB4.
|
||||
- Projection pack: `e15-live-projection-713fa7344805b7a592d12e949a6db38116e5370f768bde1cf9144fab5f93f0ae`.
|
||||
- Projection arrays SHA-256: `d37bff4299b3fd6232926ba6e04d3cbd2a5aba0e66bf5ed02e0873340597566c`.
|
||||
- Pack classification: `calibration-only-no-recorded-sensor-frames`.
|
||||
- В projection pack нет записанных camera/LiDAR frames: только intrinsics, KB4 distortion и `T_camera_from_lidar`.
|
||||
|
||||
### 3.3 Valid-FOV
|
||||
|
||||
- Generation: `valid-fov-mask-b4dd8ddf2b87c1d520ee8a0868c4fea062d7c14d1bae73ccabd3abe1f3acbac2`.
|
||||
- Mask SHA-256: `a40cee06b7c6f69b6a09a11563dcfd237f3de833b1ccd31459e66692e528ba63`.
|
||||
- Valid pixels: 270,606 из 480,000.
|
||||
- Маска вычислена заранее и переиспользуется; область за пределами полезного круга не подаётся моделям как содержательное изображение.
|
||||
|
||||
## 4. Software и model configuration
|
||||
|
||||
### 4.1 Runtime
|
||||
|
||||
- Host: `<worker-host>`.
|
||||
- GPU: NVIDIA GeForce RTX 4090.
|
||||
- Container image: `nvcr.io/nvidia/tritonserver:26.06-py3@sha256:58df7489c3f2276f9591d500a012dee03e23d35543ce3c390b4c001e6bf90794`.
|
||||
- Python: 3.12.3.
|
||||
- NumPy: 1.26.4.
|
||||
- SciPy: 1.16.3.
|
||||
- PyTorch: 2.13.0+cu130.
|
||||
- Transformers: 4.57.6.
|
||||
- PyAV: 18.0.0.
|
||||
- lz4: 4.4.5.
|
||||
- Media runtime payload SHA-256: `67487da59bba27676e5a5b4e43566e956fb350f1730589d93d55f22bcddc4c6b`.
|
||||
|
||||
### 4.2 Models
|
||||
|
||||
Detector:
|
||||
|
||||
- YOLOX-S, COCO-80, Triton inference;
|
||||
- input 1×3×640×640;
|
||||
- model SHA-256: `c5c2d13e59ae883e6af3b45daea64af4833a4951c92d116ec270d9ddbe998063`;
|
||||
- config SHA-256: `5795c737a7935a655961b069e8404d336d891f9762fb6dffb93956a076479604`.
|
||||
|
||||
Semantic:
|
||||
|
||||
- `tue-mps/cityscapes_semantic_eomt_large_1024`;
|
||||
- revision `8d6b6d1a3f7b50d441afd7d247c2ed10db186e8f`;
|
||||
- FP16 autocast;
|
||||
- model SHA-256: `c265da9a74f58f5c3f4826d23ca4ca78beac0b106cca5842beca61580de5b782`.
|
||||
|
||||
### 4.3 Immutable code/config identities
|
||||
|
||||
- Accepted live profile SHA-256: `8b7f267ee947932bdd52d1ef4b241adc8c9b4c4d5a8a3cd6c1e7f6e066983699`.
|
||||
- E14 profile SHA-256: `a010e32070459b123423f3d4f57aa1d839f73ed05f3df8580d12693406a1b218`.
|
||||
- Detector profile SHA-256: `819bdfb0da521d187ec9ce3d3039a0ff75daa0ab473fb3d8e5fa92db9c1ddca1`.
|
||||
- Semantic profile SHA-256: `ea583966bc3409f5cf563cbf4fad05e366907e67187082eb692aff53d9f5d875`.
|
||||
- Runner SHA-256: `8e487108404d912739b99d0043b8cbdd605ae148d8ad1d32ab30819ecfbb7dbc`.
|
||||
- fMP4 runtime SHA-256: `6785763350f6e2c00796f3580ea6707e0c44339f18fa8eb56687178c6f0403fc`.
|
||||
- PowerShell orchestrator SHA-256: `129bb6874730191cece10667b580fe5b8aa4e0878d442779679977079aa6069e`.
|
||||
- Minimal worker package: `e15-worker-package-dd67306b6116dd6fa091557371eb80b27afb48dae4568fb0173f253a1fb4613a`.
|
||||
- Minimal worker package identity SHA-256: `dd67306b6116dd6fa091557371eb80b27afb48dae4568fb0173f253a1fb4613a`.
|
||||
- Package содержит 11 runtime modules и не тянет MQTT/BLE/web/device lifecycle в GPU-контейнер.
|
||||
|
||||
## 5. Scheduler и bounded-state contract
|
||||
|
||||
- Detector queue: latest-wins, capacity 2.
|
||||
- Semantic queue: latest-wins, capacity 1.
|
||||
- Semantic sample: каждый пятый camera frame.
|
||||
- Semantic TTL: 750 ms.
|
||||
- Sensor buffers: 32 LiDAR + 32 pose records максимум.
|
||||
- Sensor retention: 3 s.
|
||||
- Camera metadata queue: 16 максимум.
|
||||
- Incremental media buffer: 8 MiB максимум.
|
||||
- Camera → LiDAR maximum absolute delta: 100 ms.
|
||||
- LiDAR → pose maximum absolute delta: 100 ms.
|
||||
- Accepted sensor wait budget: 90 ms.
|
||||
- Команды: disabled.
|
||||
- Navigation/safety acceptance: false.
|
||||
|
||||
Камера не блокируется semantic model. Decoder публикует frame в две bounded очереди. Detector идёт на camera cadence; semantic worker независимо обновляет latest semantic mask. Detector result связывается с latest fresh semantic result и ближайшей допустимой LiDAR/pose парой.
|
||||
|
||||
## 6. Экспериментальные попытки
|
||||
|
||||
### 6.1 Attempt A — 20 ms sensor wait, rejected
|
||||
|
||||
Result ID: `e15-shadow-inference-396152240d878551ffe4913e03aee95be0b0ec572b69d76313edd73a7ee0cf7d`.
|
||||
|
||||
Все transport, decoder, FPS, semantic, latency, queue и authority checks прошли. Единственный failed check:
|
||||
|
||||
- fusion coverage 127/151 = 84.106%; требование ≥85%.
|
||||
|
||||
Причина:
|
||||
|
||||
- 19 camera frames получили `lidar-camera-delta-exceeded`;
|
||||
- для девяти из этих frames следующий LiDAR находился всего в 49–88 ms после camera timestamp;
|
||||
- wait budget 20 ms завершал binding раньше, чем этот LiDAR был декодирован и опубликован;
|
||||
- расширять temporal gate более 100 ms не потребовалось.
|
||||
|
||||
### 6.2 Infrastructure interruption — не считается lab attempt
|
||||
|
||||
Один повтор был остановлен idle SSH timeout во время model preflight. Consumer к source не подключился, события не публиковались, immutable result не создавался. После проверки оставались только штатные `mission-core-triton`, `sentinel-frigate`, `sentinel-ollama`. Повтор выполнен с временными SSH `ServerAlive` options без изменения сетевой/VPN конфигурации.
|
||||
|
||||
### 6.3 Attempt B — 90 ms sensor wait, accepted
|
||||
|
||||
Result ID: `e15-shadow-inference-f3f06e7c6b338add445f7ee4e4a81f831cd763a899ac2fbe29ba3476c4bdf0c8`.
|
||||
|
||||
Изменён только `sensor_wait_ms`: 20 → 90. Temporal gate остался 100 ms, models и все acceptance thresholds не менялись.
|
||||
|
||||
Результат:
|
||||
|
||||
- fused frames: 136/151 = 90.066%;
|
||||
- `lidar-camera-delta-exceeded`: 19 → 10;
|
||||
- accepted cuboid states: 329 → 355;
|
||||
- p95 world-state age: 79.28 → 149.43 ms;
|
||||
- лимит p95 world-state age: 200 ms;
|
||||
- итог: accepted.
|
||||
|
||||
Это осознанный обмен дополнительного bounded ожидания на более полное depth coverage без допуска LiDAR старше 100 ms.
|
||||
|
||||
## 7. Acceptance matrix финального run
|
||||
|
||||
| Gate | Требование | Результат | State |
|
||||
|---|---:|---:|---|
|
||||
| Camera frames | ≥140 | 151 | pass |
|
||||
| Camera decode accounting | 1:1 | 151/151 | pass |
|
||||
| Detector effective FPS | ≥9.5 | 9.881 | pass |
|
||||
| Detector drops | ≤1% | 0/151 | pass |
|
||||
| Semantic effective FPS | ≥1.8 | 2.029 | pass |
|
||||
| Semantic drops | ≤5% | 0/31 | pass |
|
||||
| Semantic completion age p95 | ≤400 ms | 162.162 ms | pass |
|
||||
| Fresh semantic coverage | ≥90% | 96.689% | pass |
|
||||
| Fused fraction | ≥85% | 90.066% | pass |
|
||||
| Decode age p95 | ≤80 ms | 15.150 ms | pass |
|
||||
| World-state age p95 | ≤200 ms | 149.427 ms | pass |
|
||||
| Ingress sequence gaps | 0 | 0 | pass |
|
||||
| Camera sequence gaps | 0 | 0 | pass |
|
||||
| Failures | 0 | 0 | pass |
|
||||
| Session-end | required | seen | pass |
|
||||
| Commands/navigation authority | false | false | pass |
|
||||
|
||||
Все 18 acceptance checks прошли.
|
||||
|
||||
## 8. Производительность финального run
|
||||
|
||||
### 8.1 End-to-end
|
||||
|
||||
- Source span: 14.989973 s.
|
||||
- Measured run wall: 15.281886 s.
|
||||
- Detector: 151 frames, 9.881 FPS.
|
||||
- Semantic: 31 frames, 2.029 FPS.
|
||||
- Fused: 136 frames.
|
||||
- Source-paced factor: около 1.019× wall/source, то есть контур удержал темп источника.
|
||||
- Full orchestrator wall including isolated preflight, second model load, warm-up, run and publication: 113.669 s.
|
||||
|
||||
Большой orchestrator wall — cold-start overhead лабораторного запуска, а не latency активного perception loop. Для onboard-контура модели должны жить в постоянном worker process и не загружаться дважды перед каждой сессией.
|
||||
|
||||
### 8.2 Latency, ms
|
||||
|
||||
| Stage | mean | p50 | p95 | max |
|
||||
|---|---:|---:|---:|---:|
|
||||
| fMP4 decode age | 8.003 | 6.428 | 15.150 | 41.297 |
|
||||
| Detector queue wait | 8.577 | 0.086 | 47.431 | 165.393 |
|
||||
| Detector | 25.607 | 23.899 | 36.631 | 43.492 |
|
||||
| Sensor wait | 10.075 | 0.033 | 90.136 | 110.995 |
|
||||
| KB4 projection | 0.538 | 0.372 | 0.838 | 12.403 |
|
||||
| Association/cuboid | 5.297 | 4.100 | 14.197 | 55.061 |
|
||||
| Clearance | 0.204 | 0.177 | 0.413 | 0.877 |
|
||||
| World-state projection | 0.154 | 0.131 | 0.358 | 0.748 |
|
||||
| End-to-end world-state age | 58.154 | 43.815 | 149.427 | 282.994 |
|
||||
|
||||
World-state health:
|
||||
|
||||
- healthy: 136 frames;
|
||||
- degraded: 13 frames;
|
||||
- stale: 2 frames;
|
||||
- unavailable: 0 frames.
|
||||
|
||||
Два max-latency spikes выше 200 ms не нарушили заданный p95 gate, но подтверждают, что результат пока не имеет safety authority.
|
||||
|
||||
### 8.3 Semantic timing, ms
|
||||
|
||||
| Stage | mean | p50 | p95 | max |
|
||||
|---|---:|---:|---:|---:|
|
||||
| Total processing | 135.524 | 126.185 | 147.835 | 396.316 |
|
||||
| Forward | 107.197 | 99.099 | 109.701 | 369.566 |
|
||||
| Completion age | 146.979 | 137.378 | 162.162 | 437.821 |
|
||||
|
||||
Semantic fresh status был доступен для 146/151 detector frames. Первые пять frames корректно помечены `semantic-unavailable`, а не замаскированы фиктивным результатом.
|
||||
|
||||
### 8.4 Sensor decode
|
||||
|
||||
- LiDAR decode mean/p95/max: 12.373 / 12.913 / 97.968 ms.
|
||||
- Pose decode mean/p95/max: 0.077 / 0.103 / 0.186 ms.
|
||||
- Synchronizer maximum stored depth: 32 point frames + 32 pose frames.
|
||||
- Media buffer maximum actual depth: 178,687 bytes из 8,388,608 bytes.
|
||||
- Camera metadata maximum depth: 2 из 16.
|
||||
|
||||
## 9. Functional output
|
||||
|
||||
- 1,188 detector/fusion object records рассмотрено за 151 frames.
|
||||
- 355 accepted amodal cuboid states опубликовано.
|
||||
- Frames с хотя бы одним accepted 3D object: 130/151.
|
||||
- Maximum simultaneous accepted objects: 7.
|
||||
- Accepted class states: 350 vehicle, 5 person.
|
||||
- Cuboids являются temporal object states, а не 355 уникальными физическими объектами.
|
||||
- Ground-truth quality в E15 не измерялась.
|
||||
|
||||
Основные rejection classes:
|
||||
|
||||
- fewer than 8 clustered points: 572;
|
||||
- no semantic/LiDAR support: 173;
|
||||
- amodal support coverage below threshold: 72;
|
||||
- amodal required size exceeds class bound: 7;
|
||||
- fewer than 4 clustered points: 6;
|
||||
- distance innovation: 3.
|
||||
|
||||
Эти отказы fail-closed: сомнительная геометрия не превращается в принятый 3D cuboid.
|
||||
|
||||
## 10. Hardware и storage
|
||||
|
||||
Accepted run:
|
||||
|
||||
- process peak RSS: 2,023.5 MiB;
|
||||
- process CUDA peak allocated: 2,107.9 MiB;
|
||||
- process CUDA peak reserved: 2,840 MiB;
|
||||
- total GPU memory used telemetry: около 13,298 MiB, включая соседний Triton contour;
|
||||
- GPU utilization mean/p95/max: 27.4% / 45% / 58%;
|
||||
- GPU memory-utilization mean/p95/max: 11.8% / 20% / 25%;
|
||||
- GPU power mean/p95/max: 140.5 / 144.65 / 145.41 W;
|
||||
- GPU temperature mean/p95/max: 44.6 / 47 / 48 °C.
|
||||
|
||||
Storage:
|
||||
|
||||
- hard free-space floor: 360 GiB;
|
||||
- observed final free space: около 389.34 GiB;
|
||||
- accepted result payload: около 1.56 MiB;
|
||||
- no per-frame PNG/JPEG output;
|
||||
- no recorded video copy in result;
|
||||
- disk `C:` не проверялся и не использовался;
|
||||
- persistent inputs, runtime и results находятся на `D:`.
|
||||
|
||||
## 11. Accepted artifacts
|
||||
|
||||
Local audit mirror:
|
||||
|
||||
`.runtime/compute-experiments/e15/results/e15-shadow-inference-f3f06e7c6b338add445f7ee4e4a81f831cd763a899ac2fbe29ba3476c4bdf0c8`
|
||||
|
||||
| Artifact | Bytes | SHA-256 |
|
||||
|---|---:|---|
|
||||
| `fusion-frames.jsonl` | 1,016,064 | `ae510afe881d7d6c5bc94520c5f87c385fdeb984bd30bd0cff70bd103f718963` |
|
||||
| `world-state.jsonl` | 568,384 | `579266c99ae1e65b0240e8c5c224beba4073d05c531c5a725f096e9bbf9a3c05` |
|
||||
| `semantic-frames.jsonl` | 13,955 | `21a8a659cc19e2acc9b971b93b746a4e26f8bb51829d1e1d13c4f42c1f79f1a5` |
|
||||
| `gpu-telemetry.jsonl` | 3,591 | `51024d0774dd7e648ca4ae0cba69cca4de70baf8e47c425fa78c6805c2824712` |
|
||||
| `run-report.json` | 13,230 | `f04fce80d9246d3c19f7368d6174787eb2fd73cdc7c51fa8d18555a0bfa0bf65` |
|
||||
| `result.json` | 6,384 | `5b933109eedba02483ad35c65221d344da19f7e2584bd700d9be7c6b2ecef995` |
|
||||
|
||||
Rejected Attempt A также сохранён неизменяемо:
|
||||
|
||||
`.runtime/compute-experiments/e15/results/e15-shadow-inference-396152240d878551ffe4913e03aee95be0b0ec572b69d76313edd73a7ee0cf7d`
|
||||
|
||||
## 12. Вывод
|
||||
|
||||
На текущем RTX 4090 доказан не просто offline pipeline, а bounded source-paced perception loop на 15-секундном recorded replay:
|
||||
|
||||
- camera 10 Hz;
|
||||
- detector около 10 Hz;
|
||||
- semantic около 2 Hz;
|
||||
- calibrated LiDAR/pose fusion;
|
||||
- diagnostic 3D cuboids и world-state;
|
||||
- p95 результата 149.4 ms;
|
||||
- нулевые transport/model queue drops;
|
||||
- 90.1% fused coverage;
|
||||
- никакой command authority.
|
||||
|
||||
Следующий правильный gate — не дальнейшее улучшение recorded-video качества, а тот же профиль на физическом K1 shadow stream. Параллельно нужно убрать лабораторный cold start, превратив model-loaded contour в постоянный worker service. Только после повторяемого physical live gate имеет смысл подключать live world-state/3D cuboids к Rerun UI и отдельно начинать forest-domain evaluation и safety policy.
|
||||
|
||||
## 13. Следующие действия
|
||||
|
||||
1. Поднять persistent model-loaded E15 worker на `D:` без двойной загрузки моделей на каждую сессию.
|
||||
2. Подключить физический K1 через уже существующий shadow ingress без command path.
|
||||
3. Повторить ≥15 s physical shadow gate с теми же thresholds и отдельным LAB E16 report.
|
||||
4. Добавить reconnect/worker-loss negative control: raw recorder обязан продолжить запись.
|
||||
5. Передавать accepted `world-state` и cuboids в live Rerun entities с source/result timestamps и health.
|
||||
6. После live transport qualification собирать forest-domain ground truth и считать 2D segmentation/object, distance и 3D cuboid quality отдельно от throughput.
|
||||
@@ -0,0 +1,148 @@
|
||||
# LAB E16 — Persistent AI worker and unified live perception publication
|
||||
|
||||
Date: 2026-07-22
|
||||
|
||||
Acceptance time: 2026-07-22 21:23 MSK / 2026-07-22T18:23:49Z
|
||||
|
||||
## Purpose
|
||||
|
||||
LAB E16 validates the infrastructure between the previously accepted E15 perception
|
||||
pipeline and one unified Mission Core Rerun scene. This is an infrastructure
|
||||
acceptance, not a physical K1 perception acceptance. No device, acquisition or camera
|
||||
session was active during this lab.
|
||||
|
||||
The target operator contract is one original-video timeline with independently
|
||||
switchable derived layers:
|
||||
|
||||
- tracked 2D detections with confidence and metric distance when fusion provides it;
|
||||
- semantic segmentation on the original camera image;
|
||||
- accepted, calibrated 3D cuboids in the LiDAR/world scene.
|
||||
|
||||
## Safety and authority
|
||||
|
||||
- The production Mission Core process at `127.0.0.1:8000` was not stopped or restarted.
|
||||
- UI and backend acceptance used an isolated shadow process at `127.0.0.1:8001`.
|
||||
- The AI worker ran only on the Windows worker's `D:` drive.
|
||||
- VPN, Hidemyname, Wi-Fi and host network configuration were not modified.
|
||||
- Worker authority remained `commands_enabled=false` and
|
||||
`navigation_or_safety_accepted=false`.
|
||||
- The result direction is diagnostic-only and contains no command endpoint.
|
||||
|
||||
## Immutable worker inputs
|
||||
|
||||
- Runner SHA-256:
|
||||
`f2f2889e07c609e3ba53b4504c3f2417f151fbb9920828f4680ea375a5d3010e`.
|
||||
- Persistent-service orchestrator SHA-256:
|
||||
`ed2dc35de423d0d9f8ac0501db283a10081e78b6535ebeb8d521a58ca6988b11`.
|
||||
- Persistent-run orchestrator SHA-256:
|
||||
`1c6af15736396ee0124d4aabe309b7bc24bb1425d88e24295c5675c686fc27bc`.
|
||||
- Worker package:
|
||||
`e15-worker-package-34fb22ce506aab6e3b6a0fd8058c868e24076a8cfc739b4299b2474f2cb92b99`.
|
||||
- Runner root on the worker:
|
||||
`D:\NDC_MISSIONCORE\runtime\derived\e16-runner-f2f2889e07c6-ed2dc35de423-1c6af1573639-34fb22ce506a`.
|
||||
- Worker package root:
|
||||
`D:\NDC_MISSIONCORE\runtime\inputs\e16\worker-packages\e15-worker-package-34fb22ce506aab6e3b6a0fd8058c868e24076a8cfc739b4299b2474f2cb92b99`.
|
||||
- Container: `mission-core-perception-worker`.
|
||||
- Pinned base image: `nvcr.io/nvidia/tritonserver:26.06-py3` with the E15 digest.
|
||||
|
||||
## Persistent model-loaded worker result
|
||||
|
||||
- GPU: NVIDIA GeForce RTX 4090.
|
||||
- Model load and warm-up: 38.296654 s.
|
||||
- Health: `ready`.
|
||||
- Models loaded: true.
|
||||
- Completed physical runs: 0.
|
||||
- Failed physical runs: 0.
|
||||
- D: free after startup: 418,065,694,720 bytes, approximately 389.354 GiB.
|
||||
- Guarded D: floor: 360 GiB plus a 512 MiB startup reserve.
|
||||
|
||||
The service binds its control HTTP endpoint only to `127.0.0.1` inside the worker
|
||||
container's shared Triton network namespace. It has no exposed LAN control port. A run
|
||||
request is single-flight, authenticated by the existing shadow token, limited to one
|
||||
direct child of the guarded D:-backed publication root and reuses already loaded models.
|
||||
|
||||
## Result transport contract
|
||||
|
||||
The authenticated K1 shadow WebSocket remains the sensor ingress and now has one
|
||||
bounded worker-to-Mission-Core diagnostic result direction.
|
||||
|
||||
- Result schema: `missioncore.live-perception-result-wire/v1`.
|
||||
- Result queue: latest-wins, capacity 2.
|
||||
- Maximum result payload: 2 MiB.
|
||||
- Maximum JSON header: 256 KiB.
|
||||
- Maximum original JPEG: 1 MiB.
|
||||
- Segmentation: fixed 800x600 uint8 mask compressed with zlib.
|
||||
- Maximum objects per frame: 128.
|
||||
- Payload SHA-256, lengths, geometry, timestamps and authority are validated before
|
||||
the frame is admitted to the visualization runtime.
|
||||
- Partial cuboids, non-finite geometry, invalid JPEG/mask payloads and any authority
|
||||
claim are rejected.
|
||||
|
||||
## Unified Rerun publication
|
||||
|
||||
Accepted live results use the same timeline and stable entities:
|
||||
|
||||
- `/perception/camera/image` — original worker-decoded camera frame;
|
||||
- `/perception/camera/detections` — 2D boxes, track id, class, confidence and distance;
|
||||
- `/perception/camera/segmentation` — semantic mask;
|
||||
- `/world/perception/boxes3d` — calibrated accepted cuboids.
|
||||
|
||||
The live blueprint switches between the normal 3D-only scene and a single horizontal
|
||||
original-video plus 3D-world composition. The three AI controls are independent; they
|
||||
are not mutually exclusive views. When dynamic perception is active, Rerun uses
|
||||
latest-at presentation so historical cuboids do not stack into a cloud of boxes.
|
||||
|
||||
## Validation
|
||||
|
||||
- Complete Python test suite: passed.
|
||||
- Focused live result, WebSocket return, visualization runtime and Rerun bridge tests:
|
||||
passed.
|
||||
- Ruff on the changed Python surface: passed.
|
||||
- Frontend TypeScript build: passed.
|
||||
- Frontend unit suite: 144/144 passed.
|
||||
- Production frontend build: passed.
|
||||
- Windows PowerShell parser accepted both service and run orchestrators.
|
||||
- In-container encode/decode probe returned a 600x800 segmentation mask successfully.
|
||||
- Shadow backend health at `127.0.0.1:8001`: healthy.
|
||||
- Production backend health at `127.0.0.1:8000`: healthy and uninterrupted.
|
||||
- Browser QA opened RAVNOVES00, reached Rerun ready state, enabled 2D,
|
||||
segmentation and 3D cuboids simultaneously, and reported zero browser errors.
|
||||
|
||||
## Disk handling
|
||||
|
||||
The earlier 418 MiB temporary recorded-perception RRD used for payload inspection was
|
||||
removed after validation. It was a regenerable temporary duplicate, not source evidence.
|
||||
No source recording or user data was deleted.
|
||||
|
||||
## Accepted conclusions
|
||||
|
||||
1. The models can remain loaded between runs; the previous double-load startup cost is
|
||||
removed from subsequent run requests.
|
||||
2. Mission Core has a bounded, authenticated, non-authoritative return path for live AI
|
||||
presentation.
|
||||
3. Original video, 2D detections, segmentation and 3D cuboids now share one Rerun
|
||||
timeline and one operator composition.
|
||||
4. Layer visibility is independently controlled and no longer modeled as separate
|
||||
perception streams.
|
||||
|
||||
## Explicitly not accepted yet
|
||||
|
||||
- Physical K1 shadow inference: no device session was active.
|
||||
- End-to-end detector, segmentation and fusion FPS through the new persistent service:
|
||||
requires the next physical 20-second K1 gate.
|
||||
- The new backend code is intentionally not active on production port 8000 yet because
|
||||
that process was not restarted. Port 8001 is the accepted shadow build.
|
||||
- 3D semantic point coloring is not yet returned by the live result frame; E16 covers
|
||||
camera segmentation and 3D cuboids.
|
||||
- The recorded full-session perception RRD remains a large monolith and is not the
|
||||
target real-time transport.
|
||||
- None of the outputs are ground truth or safety/navigation accepted.
|
||||
|
||||
## Next gate
|
||||
|
||||
1. Connect K1 and establish a normal Mission Core device/acquisition/camera session.
|
||||
2. Run one 20-second persistent physical shadow request without command authority.
|
||||
3. Record detector FPS, semantic FPS, fusion fraction, result age, result payload rate,
|
||||
GPU telemetry, queue drops and D: free space.
|
||||
4. Verify the same four entities visually in the live Rerun scene.
|
||||
5. Add bounded semantic LiDAR point coloring, then repeat as LAB E17.
|
||||
@@ -0,0 +1,297 @@
|
||||
# LAB E3 — K1 rectified segmentation baseline
|
||||
|
||||
Date: 2026-07-20
|
||||
Completed at: 2026-07-20T18:57:23.215Z
|
||||
State: completed, model draft, not ground truth
|
||||
Ops card: MISSIONCOR-17
|
||||
|
||||
## Objective
|
||||
|
||||
Compare an adult Cityscapes semantic model on the same immutable XGRIDS K1
|
||||
evaluation pack under three input profiles:
|
||||
|
||||
1. valid-FOV fisheye image;
|
||||
2. valid-FOV fisheye image with CLAHE;
|
||||
3. factory-calibrated KB4 five-view perspective rectification with CLAHE and
|
||||
fusion back into the original fisheye coordinate system.
|
||||
|
||||
The experiment answers whether calibration-derived rectification is technically
|
||||
valid, how much it costs, and whether it is a credible next baseline. It does not
|
||||
measure accuracy because no reviewed ground truth exists yet.
|
||||
|
||||
## Immutable inputs and provenance
|
||||
|
||||
- Evaluation pack:
|
||||
`evaluation-pack-7a983bba75d46c7c260252cb2d461e1384dcb92cda9e164397e841e6ebb37789`
|
||||
- Session: `20260720T065719Z_viewer_live`
|
||||
- Frames: 64, image IDs 1–64, resolution 800×600
|
||||
- Source: `sensor.camera.right`
|
||||
- Factory calibration slot: `camera_1`
|
||||
- Factory calibration content SHA-256:
|
||||
`05f3ad9b38b3a4fc95388a8ec83da83c745e217709e51787b3d5aad0969f6fa9`
|
||||
- Valid-FOV generation:
|
||||
`valid-fov-mask-b4dd8ddf2b87c1d520ee8a0868c4fea062d7c14d1bae73ccabd3abe1f3acbac2`
|
||||
- Valid pixels per frame: 270,606
|
||||
- LAB E2 comparison draft:
|
||||
`evaluation-prelabels-4ba26bbf6eb8a49631f5caf984267e0445958540aeda2b5b0d82ca6440835cf1`
|
||||
|
||||
The LAB E2 draft has a legacy provenance inconsistency. Its declared identity
|
||||
SHA-256 is `4ba26b…`, while a fresh canonical hash of the stored identity document
|
||||
is `33292c85eb8281d68c1b444baaf0d7931c7aa84938ac1d329b89e93f3b525099`.
|
||||
LAB E3 did not alter LAB E2. It independently verified the evaluation-pack
|
||||
binding, the declared result/directory relationship, and the size and SHA-256 of
|
||||
every one of the 64 consumed semantic artifacts. Both hashes and
|
||||
`e2_prelabel_identity_consistent=false` are preserved in the LAB E3 identity.
|
||||
|
||||
## Calibration and preprocessing configuration
|
||||
|
||||
Camera-1 intrinsics (`fx, fy, cx, cy`):
|
||||
|
||||
```text
|
||||
194.59817287616025
|
||||
194.57531427932872
|
||||
396.31861150187996
|
||||
301.49644357408005
|
||||
```
|
||||
|
||||
Kannala–Brandt KB4 coefficients:
|
||||
|
||||
```text
|
||||
-0.023164451386679667
|
||||
-0.0014974198594105452
|
||||
-0.001039213149441563
|
||||
-0.000035237331915978814
|
||||
```
|
||||
|
||||
Rectification profile:
|
||||
|
||||
- projection: five perspective gnomonic views;
|
||||
- views: front, left, right, up, down;
|
||||
- yaw/pitch: `0/0`, `-90/0`, `90/0`, `0/90`, `0/-90` degrees;
|
||||
- tile size: 768×768;
|
||||
- horizontal and vertical FOV: 100 degrees;
|
||||
- RGB interpolation: OpenCV linear remap;
|
||||
- fused-label sampling: nearest pixel;
|
||||
- overlap winner: maximum optical-axis cosine;
|
||||
- contrast: CLAHE on LAB luminance, clip limit 2.0, grid 8×8.
|
||||
|
||||
Measured geometry coverage:
|
||||
|
||||
- valid pixels covered: 270,606 of 270,606;
|
||||
- coverage: 100%;
|
||||
- views covering one valid pixel: minimum 1, maximum 3, mean 1.190099259;
|
||||
- uncovered valid pixels: 0.
|
||||
|
||||
## Model and software identity
|
||||
|
||||
- Model: `tue-mps/cityscapes_semantic_eomt_large_1024`
|
||||
- Architecture: `EomtForUniversalSegmentation`
|
||||
- Revision: `8d6b6d1a3f7b50d441afd7d247c2ed10db186e8f`
|
||||
- Precision: FP16 autocast
|
||||
- Batch size: 1
|
||||
- Model weights: 1,276,175,488 bytes, SHA-256
|
||||
`c265da9a74f58f5c3f4826d23ca4ca78beac0b106cca5842beca61580de5b782`
|
||||
- Config SHA-256:
|
||||
`7f4aa94fa4e43c0dbd79a5420edb511120aef62bd82bfbcbcece79948286a650`
|
||||
- Preprocessor SHA-256:
|
||||
`97e2fbf7f0bdba2cfc90251c5133bae9c27ddc9c4410509f40670be2332854e7`
|
||||
- OpenCV distribution: `opencv-python-headless==4.13.0.92`
|
||||
- OpenCV runtime module: `4.13.0`
|
||||
- Transformers: `4.57.6`
|
||||
- PyTorch: `2.13.0+cu130`
|
||||
- CUDA runtime: `13.0`
|
||||
- Container: existing `nvcr.io/nvidia/tritonserver:26.06-py3`; no image pull
|
||||
- Profile SHA-256:
|
||||
`ea583966bc3409f5cf563cbf4fad05e366907e67187082eb692aff53d9f5d875`
|
||||
- Dependency identity SHA-256:
|
||||
`4eb1f8d33236806e74f9e5bb96b7dce2ac37623dc39b2184be2aa8d7d00e983e`
|
||||
- Final runner SHA-256:
|
||||
`01881862d4eaa218955f776a948124bf19c34be2b5ec282115daeacb15c53ae6`
|
||||
|
||||
The Cityscapes labels were mapped into the existing Mission Core 0–15
|
||||
perception taxonomy. Outputs outside the admitted K1 valid FOV were forced to
|
||||
category 0.
|
||||
|
||||
## Execution topology
|
||||
|
||||
The comparison was deliberately sequential so each profile has isolated timing:
|
||||
|
||||
```text
|
||||
frame
|
||||
-> EoMT on fisheye mask
|
||||
-> CLAHE -> EoMT on fisheye mask
|
||||
-> KB4 remap of five views -> CLAHE -> five sequential EoMT calls
|
||||
-> nearest-label fusion into the raw fisheye coordinates
|
||||
-> masks, preview, metrics, and telemetry
|
||||
```
|
||||
|
||||
There were seven model evaluations per frame and 448 evaluations for the full
|
||||
pack. No software GPU-utilization or power cap was applied. The experiment did
|
||||
not batch or parallelize the five rectified views; that is a future optimization,
|
||||
not part of the present baseline.
|
||||
|
||||
## Full-run results
|
||||
|
||||
Immutable result:
|
||||
`e3-segmentation-01bd497c44c2b940add145ec784d3418010327bce0baddd4420b0925317e8a16`
|
||||
|
||||
| Variant | Mean end-to-end/frame | Mean forward | Approx. standalone throughput | Mean change vs E2 draft |
|
||||
|---|---:|---:|---:|---:|
|
||||
| EoMT fisheye mask | 121.02 ms | 98.64 ms | 8.26 FPS | 16.44% pixels |
|
||||
| EoMT fisheye + CLAHE | 127.41 ms | 99.10 ms | 7.85 FPS | 16.51% pixels |
|
||||
| EoMT KB4 cubemap5 + CLAHE | 341.97 ms | 267.39 ms across 5 views | 2.92 FPS | 18.13% pixels |
|
||||
|
||||
Additional timing facts:
|
||||
|
||||
- full comparison wall time: 46.170785 seconds;
|
||||
- average full comparison time: 721.42 ms/frame including all three variants,
|
||||
preview/mask encoding, metrics, and telemetry;
|
||||
- five-view rectification preprocessing: 22.60 ms/frame mean;
|
||||
- five-view label fusion: 3.03 ms/frame mean;
|
||||
- one rectified tile forward: 53.48 ms mean, 65.85 ms p95;
|
||||
- raw-vs-cubemap pixel disagreement: 8.24% mean, 5.80% median, 24.66% p95,
|
||||
36.76% maximum.
|
||||
|
||||
The largest raw/cubemap difference occurs at image 59 / frame 4224. The
|
||||
near-structure temporal clip (images 60–63, frames 4433–4436) contributes four
|
||||
of the next five largest differences. This is the correct area to prioritize in
|
||||
human review: extreme distortion and nearby planar structures are precisely
|
||||
where projection choice changes the model most.
|
||||
|
||||
## GPU, CPU, memory, and disk
|
||||
|
||||
Worker GPU: NVIDIA GeForce RTX 4090, compute capability 8.9.
|
||||
|
||||
- process CUDA peak allocated: 2,099.8 MiB;
|
||||
- process CUDA peak reserved: 2,840.0 MiB;
|
||||
- total board memory used, including pre-existing services: 13,406 MiB max,
|
||||
13,396.9 MiB mean;
|
||||
- GPU utilization: 72.34% mean, 91% max;
|
||||
- GPU memory-controller utilization: 27.38% mean, 41% max;
|
||||
- GPU power: 288.88 W mean, 334.09 W max;
|
||||
- GPU temperature: 55.94°C mean, 61°C max;
|
||||
- process peak RSS: 2,097.14 MiB;
|
||||
- GPU telemetry samples: 47 at one-second intervals.
|
||||
|
||||
All task-controlled persistent artifacts are on `D:\NDC_MISSIONCORE`:
|
||||
|
||||
| Artifact group | Bytes | MiB |
|
||||
|---|---:|---:|
|
||||
| Pinned E3 model cache | 1,276,294,838 | 1,217.17 |
|
||||
| Pinned OpenCV environment | 211,776,082 | 201.97 |
|
||||
| Four-frame pilot | 2,040,795 | 1.95 |
|
||||
| Full 64-frame result | 7,924,484 | 7.56 |
|
||||
| Total | 1,498,036,199 | 1,428.64 |
|
||||
|
||||
Final D: free space was 414,903,472,128 bytes (386.409 GiB), 26.409 GiB above
|
||||
the enforced 360 GiB floor. No task-controlled C: path was created or mounted.
|
||||
The report does not claim to measure incidental internal writes by Windows or an
|
||||
already configured Docker Desktop engine.
|
||||
|
||||
## Visual findings
|
||||
|
||||
1. The factory KB4 geometry is usable now. The five views are upright, have the
|
||||
expected content, and fuse back without missing valid pixels.
|
||||
2. EoMT produces visibly cleaner road, grass, building, person, and vehicle
|
||||
regions than the LAB E2 BEiT draft on the inspected samples.
|
||||
3. CLAHE changes the global result very little while adding about 6.4 ms/frame.
|
||||
It is not justified as the default from this unreviewed proxy alone.
|
||||
4. Cubemap5 often improves separation of people, cars, and rectified planar
|
||||
surfaces, especially where raw fisheye distortion is severe.
|
||||
5. Hard winner-take-all fusion creates local seam/checker artifacts, most visible
|
||||
near the lower rim, the sun/sky, and close structures. Cubemap5 should not be
|
||||
promoted to the default without seam-aware fusion and reviewed accuracy.
|
||||
6. Cityscapes has no explicit dirt, animal, or general other-background output in
|
||||
this mapping. That limits this model as a complete forest/off-road taxonomy.
|
||||
|
||||
## Integrity and acceptance status
|
||||
|
||||
- 211 of 211 listed output artifacts were rehashed successfully after transfer.
|
||||
- 192 of 192 semantic masks were checked.
|
||||
- Every semantic value was within categories 0–15.
|
||||
- Every pixel outside the valid-FOV mask was category 0.
|
||||
- Four-frame pilot and full-run outputs are immutable and separately identified.
|
||||
- Outputs remain `ground_truth=false`.
|
||||
- No mIoU, AP, distance accuracy, 3D geometry, tracking, or safety claim is made.
|
||||
|
||||
## Engineering gates encountered
|
||||
|
||||
The following preflight failures happened before a result was accepted and are
|
||||
preserved as part of the laboratory history:
|
||||
|
||||
- Git revision and SHA-256 were initially treated as the same hash shape;
|
||||
- the OpenCV package version `4.13.0.92` differs from runtime `cv2` version
|
||||
`4.13.0`;
|
||||
- Hugging Face snapshot files are symlinks into a content-addressed blob store;
|
||||
- the initially recorded weight SHA was replaced by the independently measured
|
||||
blob SHA-256 `c265…`;
|
||||
- EoMT processor output includes a non-tensor `task_inputs` field;
|
||||
- LAB E2 contains the identity inconsistency documented above;
|
||||
- a PowerShell full-run expected-frame lookup initially read the wrong JSON
|
||||
level. That technical run was stopped, its exact transient container was
|
||||
removed, its staging was cleaned, and no result was published.
|
||||
|
||||
These were fail-closed contract checks, not model failures.
|
||||
|
||||
## Decision and next gate
|
||||
|
||||
The current evidence supports this decision:
|
||||
|
||||
- keep plain EoMT fisheye-mask as the speed/quality control baseline;
|
||||
- keep cubemap5 as the calibration-derived challenger;
|
||||
- do not make CLAHE the default yet;
|
||||
- do not select a production winner from proxy disagreement or visual inspection;
|
||||
- review the high-disagreement and temporal frames in CVAT, export reviewed masks,
|
||||
and calculate class-wise IoU, boundary IoU, temporal stability, and object-class
|
||||
recall before changing the default pipeline;
|
||||
- if cubemap5 survives reviewed accuracy, replace hard view selection with
|
||||
overlap-aware/logit fusion and then test batched five-view inference.
|
||||
|
||||
3D cuboids, LiDAR association, distance accuracy, and point-cloud segmentation
|
||||
remain outside LAB E3 and should use the reviewed 2D baseline as an input to the
|
||||
next dedicated experiments rather than being mixed into this result.
|
||||
|
||||
## CVAT review handoff
|
||||
|
||||
The control and challenger were converted into deterministic CVAT Segmentation
|
||||
Mask 1.1 packages and imported into the existing D-hosted CVAT v2.70.0 instance.
|
||||
The original 64-frame image archive is reused and was not duplicated.
|
||||
|
||||
- Workspace:
|
||||
`e3-cvat-review-ca599521e345446ca8e9af5a9013062099278e7317f83ff89740c6b092ddc52f`
|
||||
- Workspace manifest SHA-256:
|
||||
`5e1b318d31431c2081c169235ec877693591ef2b05bbebab61c72061bc1621d9`
|
||||
- CVAT import report SHA-256:
|
||||
`fe0933efc3a5bb1bcb7b995bb1241b308e7a860113cd7a97351aedfbd43c9b64`
|
||||
- [Task 3 — EoMT fisheye control](http://localhost:18080/tasks/3): 64 frames,
|
||||
5,284 imported shapes.
|
||||
- [Task 4 — EoMT KB4 cubemap5 challenger](http://localhost:18080/tasks/4): 64
|
||||
frames, 5,762 imported shapes.
|
||||
- Both task pages returned HTTP 200 through the existing SSH tunnel.
|
||||
- Both tasks remain model drafts and `ground_truth=false`.
|
||||
- Review priority by measured raw/cubemap disagreement: image IDs
|
||||
`59, 63, 62, 61, 60, 64, 50, 30, 17, 29`.
|
||||
- Review bundle size on D: 1,390,051 bytes.
|
||||
- Post-import D: free space: 414,762,061,824 bytes (386.277 GiB), 26.277 GiB
|
||||
above the enforced floor.
|
||||
|
||||
The required human sequence is still two passes: correction/completeness first,
|
||||
then independent consistency review. Importing the drafts does not complete that
|
||||
gate and does not convert either task into ground truth.
|
||||
|
||||
## Durable paths
|
||||
|
||||
Worker full result:
|
||||
|
||||
```text
|
||||
D:\NDC_MISSIONCORE\runtime\derived\e3-segmentation\e3-segmentation-01bd497c44c2b940add145ec784d3418010327bce0baddd4420b0925317e8a16
|
||||
```
|
||||
|
||||
Local evidence copy:
|
||||
|
||||
```text
|
||||
.runtime/compute-experiments/e3/full/e3-segmentation-01bd497c44c2b940add145ec784d3418010327bce0baddd4420b0925317e8a16
|
||||
```
|
||||
|
||||
The `result.json`, `run-report.json`, 16 comparison previews, rectification
|
||||
atlas, 192 semantic masks, and GPU telemetry are contained in that result.
|
||||
@@ -0,0 +1,360 @@
|
||||
# LAB E4 — Full-session EoMT segmentation playback
|
||||
|
||||
Date: 2026-07-21
|
||||
Completed at: 2026-07-21T07:21:16.791Z
|
||||
State: completed and published, model output, not ground truth
|
||||
Ops card: MISSIONCOR-18
|
||||
|
||||
## Objective
|
||||
|
||||
Turn the plain EoMT valid-FOV configuration selected in LAB E3 into the first
|
||||
complete, watchable semantic-segmentation result for a saved Mission Core
|
||||
session. The acceptance target was deliberately narrow and observable:
|
||||
|
||||
1. process every recorded camera frame, without sampling;
|
||||
2. preserve the exact saved-session timeline;
|
||||
3. publish an H.264 semantic-overlay video and per-frame semantic masks through
|
||||
the existing recorded-perception contract;
|
||||
4. make that video selectable and playable in the RAVNOVES00 saved-session UI;
|
||||
5. measure full inference, publication, GPU, memory, power, temperature and disk
|
||||
behavior on the connected AI worker;
|
||||
6. keep all task-controlled worker data on `D:\NDC_MISSIONCORE` and enforce a
|
||||
360 GiB free-space floor.
|
||||
|
||||
LAB E4 is semantic-only. It does not claim instance detection, tracking, 3D
|
||||
cuboids, LiDAR association, metric distance, point-cloud segmentation or
|
||||
safety-grade perception.
|
||||
|
||||
## Immutable input and provenance
|
||||
|
||||
- UI session alias: `RAVNOVES00`
|
||||
- Session ID: `20260720T065719Z_viewer_live`
|
||||
- Compute job: `recorded-camera-602ac89026ed12978619801d`
|
||||
- Source: `sensor.camera.right`
|
||||
- Factory calibration slot: `camera_1`
|
||||
- Input SHA-256:
|
||||
`602ac89026ed12978619801d4edea0cae24b5cc3afabd9f7af2858de6505a20e`
|
||||
- Factory calibration content SHA-256:
|
||||
`05f3ad9b38b3a4fc95388a8ec83da83c745e217709e51787b3d5aad0969f6fa9`
|
||||
- Codec epoch: 1
|
||||
- Resolution: 800x600
|
||||
- Segments/frames admitted: 4,489/4,489
|
||||
- Timeline: 35.421857292–484.144857292 session seconds
|
||||
- Media duration: 448.723 seconds
|
||||
- Approximate source rate: 10.0037 FPS
|
||||
- Frame timeline SHA-256:
|
||||
`16098d7606e5462f1d6ccbc1b6e99158f68e2b2ca85ea11794799ef709b61c61`
|
||||
|
||||
The valid image circle is the immutable LAB E1 artifact
|
||||
`valid-fov-mask-b4dd8ddf2b87c1d520ee8a0868c4fea062d7c14d1bae73ccabd3abe1f3acbac2`.
|
||||
Its mask SHA-256 is
|
||||
`a40cee06b7c6f69b6a09a11563dcfd237f3de833b1ccd31459e66692e528ba63`.
|
||||
It admits 270,606 of 480,000 pixels. The mask is loaded once and reused; it is
|
||||
not recomputed from every frame.
|
||||
|
||||
## Accepted configuration
|
||||
|
||||
LAB E4 deliberately promotes the LAB E3 speed/quality control, not the more
|
||||
expensive five-view challenger:
|
||||
|
||||
- pipeline: `recorded-semantic-eomt-fisheye-mask/v1`;
|
||||
- model: `tue-mps/cityscapes_semantic_eomt_large_1024`;
|
||||
- architecture: `EomtForUniversalSegmentation`;
|
||||
- pinned revision:
|
||||
`8d6b6d1a3f7b50d441afd7d247c2ed10db186e8f`;
|
||||
- model weights: 1,276,175,488 bytes, SHA-256
|
||||
`c265da9a74f58f5c3f4826d23ca4ca78beac0b106cca5842beca61580de5b782`;
|
||||
- config SHA-256:
|
||||
`7f4aa94fa4e43c0dbd79a5420edb511120aef62bd82bfbcbcece79948286a650`;
|
||||
- preprocessor SHA-256:
|
||||
`97e2fbf7f0bdba2cfc90251c5133bae9c27ddc9c4410509f40670be2332854e7`;
|
||||
- precision: FP16 autocast;
|
||||
- batch size: 1;
|
||||
- frame policy: all frames, no sampling;
|
||||
- semantic-overlay alpha: 0.48;
|
||||
- CLAHE: disabled;
|
||||
- KB4 cubemap rectification: disabled;
|
||||
- instance branch: disabled;
|
||||
- container: existing `nvcr.io/nvidia/tritonserver:26.06-py3`;
|
||||
- PyTorch: `2.13.0+cu130`;
|
||||
- CUDA runtime: `13.0`;
|
||||
- Transformers: `4.57.6`;
|
||||
- NumPy: `1.26.4`;
|
||||
- Pillow: `12.3.0`.
|
||||
|
||||
Producer identities:
|
||||
|
||||
- runner SHA-256:
|
||||
`651e8e06c3912dffb036b7fd08f2c0623f7563d8306cc7aee05db562798518f4`;
|
||||
- orchestrator SHA-256:
|
||||
`c1c16fc351e7c8c4806fe2fd17a56e659495d843db5ff3e9ac3699a8d9722640`;
|
||||
- profile SHA-256:
|
||||
`ea583966bc3409f5cf563cbf4fad05e366907e67187082eb692aff53d9f5d875`;
|
||||
- dependency identity SHA-256:
|
||||
`4eb1f8d33236806e74f9e5bb96b7dce2ac37623dc39b2184be2aa8d7d00e983e`.
|
||||
|
||||
The Cityscapes outputs are mapped into the existing Mission Core 0–15
|
||||
robotics taxonomy. Pixels outside the admitted FOV are category 0. Cityscapes
|
||||
does not provide the required forest-specific `ground_dirt`, `animal` and
|
||||
general `other_background` classes, so those mapped totals are zero. This is a
|
||||
known taxonomy limitation, not evidence that those contents are absent.
|
||||
|
||||
## Execution topology
|
||||
|
||||
The work was sequential at frame level:
|
||||
|
||||
```text
|
||||
recorded fMP4 epoch
|
||||
-> exact RGB frame reconstruction
|
||||
-> fixed calibration-bound valid-FOV fill
|
||||
-> EoMT semantic forward pass on RTX 4090
|
||||
-> resize/map semantic labels into Mission Core taxonomy
|
||||
-> force excluded FOV pixels to category 0
|
||||
-> write indexed semantic PNG and 48%-alpha overlay PNG
|
||||
-> append timestamped frame metadata
|
||||
-> encode overlays as H.264 MP4
|
||||
-> archive semantic masks
|
||||
-> hash, validate and publish result
|
||||
```
|
||||
|
||||
The input stream and model loop were not split across parallel jobs. Batch size
|
||||
was one. GPU work is internally asynchronous where PyTorch/CUDA permits, but
|
||||
the next frame was not admitted until the current frame's semantic result and
|
||||
artifacts had been produced. The final video was encoded on NVENC after the
|
||||
overlay sequence was complete.
|
||||
|
||||
No software power, utilization or memory cap was applied. The main deliberate
|
||||
limits were batch size one, one model, a 360 GiB D: free-space floor and a
|
||||
conservative preflight working-set reserve.
|
||||
|
||||
## Pilot gate
|
||||
|
||||
An eight-frame non-publishable pilot ran first under
|
||||
`pilot-8-4038b699764048579f9fb7c6886a0a10`.
|
||||
|
||||
- frames: 8/8;
|
||||
- inference wall: 2.381609 seconds;
|
||||
- throughput including frame artifacts: 3.359074 FPS;
|
||||
- visual check: valid-FOV alignment, road, vegetation, vehicles, person and
|
||||
building overlays were usable;
|
||||
- publication: prohibited by design.
|
||||
|
||||
The pilot exposed a report-only PowerShell parsing defect in a free-space
|
||||
measurement: status text and the numeric value reached the same output stream.
|
||||
The full-run orchestrator separated them before execution. The pilot data was
|
||||
retained as evidence; no pilot result entered the saved-session result store.
|
||||
|
||||
## Full-run result
|
||||
|
||||
Published result:
|
||||
`result-793785170472c519486ccd666be102fb04d169d92383acda3fcc29eecf045d30`.
|
||||
|
||||
| Measurement | Result |
|
||||
|---|---:|
|
||||
| Frames expected / processed | 4,489 / 4,489 |
|
||||
| Frames failed / skipped | 0 / 0 |
|
||||
| Inference wall | 1,447.565362 s (24:07.565) |
|
||||
| Inference throughput | 3.101069 FPS |
|
||||
| Extract/reconstruction phase | 31.837595 s |
|
||||
| H.264 encoding | 3.374570 s |
|
||||
| Mask archive | 1.332036 s |
|
||||
| Full worker wall | 1,668.259063 s (27:48.259) |
|
||||
| End-to-end throughput | 2.690829 FPS |
|
||||
|
||||
The source plays at approximately 10.0037 FPS. This offline E4 path is therefore
|
||||
about 3.72 times slower than source real time. It is a complete playback
|
||||
baseline, not yet a live-stream configuration.
|
||||
|
||||
The earlier LAB E0 full path achieved 1.678305 inference FPS and 1.593908
|
||||
end-to-end FPS while executing both Mask R-CNN and BEiT. E4 is approximately
|
||||
1.85 times faster on the measured inference loop and 1.69 times faster overall,
|
||||
but the comparison is not like-for-like: E4 is semantic-only and intentionally
|
||||
disables the instance branch.
|
||||
|
||||
### Per-frame latency
|
||||
|
||||
| Stage | Mean | p50 | p95 | Max |
|
||||
|---|---:|---:|---:|---:|
|
||||
| End to end | 322.458 ms | 320.654 ms | 348.467 ms | 441.481 ms |
|
||||
| Model forward | 141.142 ms | 140.483 ms | 153.122 ms | 185.848 ms |
|
||||
| Artifact write | 99.477 ms | 98.231 ms | 113.678 ms | 155.797 ms |
|
||||
| Image decode | 30.677 ms | 28.915 ms | 42.952 ms | 82.429 ms |
|
||||
| Processor | 10.656 ms | 9.597 ms | 15.529 ms | 85.791 ms |
|
||||
| Host to device | 7.737 ms | 7.515 ms | 10.726 ms | 21.986 ms |
|
||||
| Model postprocess | 10.214 ms | 10.421 ms | 14.078 ms | 25.200 ms |
|
||||
| Overlay | 18.712 ms | 17.595 ms | 25.332 ms | 42.439 ms |
|
||||
| Valid-FOV fill | 1.952 ms | 1.834 ms | 2.582 ms | 7.701 ms |
|
||||
|
||||
The measured stage means account for the frame loop. Approximately 184.15
|
||||
seconds of the full worker wall remain outside extract, inference, encode and
|
||||
archive. This includes full input/model hashing, model load, container/preflight
|
||||
work, exact stream setup, result finalization, output hashing and validation.
|
||||
|
||||
## GPU, process and neighboring services
|
||||
|
||||
Worker GPU: NVIDIA GeForce RTX 4090, compute capability 8.9.
|
||||
|
||||
- one-second GPU samples: 1,448;
|
||||
- GPU utilization: 73.05% mean, 73% p50, 89% p95, 95% max;
|
||||
- board memory used, including pre-existing services: 13,397.65 MiB mean,
|
||||
13,409 MiB max;
|
||||
- GPU memory-controller utilization: 21.41% mean, 32% p95, 35% max;
|
||||
- power: 234.24 W mean, 232.91 W p50, 247.15 W p95, 253.23 W max;
|
||||
- temperature: 52.94 C mean, 53 C p50, 57 C p95, 58 C max;
|
||||
- process CUDA peak allocated: 2,099.8 MiB;
|
||||
- process CUDA peak reserved: 2,840.0 MiB;
|
||||
- process peak RSS: 2,009.332 MiB.
|
||||
|
||||
The board-memory figure is not the E4 process allocation. It is the total
|
||||
device usage observed by `nvidia-smi`, including already running services. The
|
||||
per-process CUDA peaks above are the relevant E4 allocation measurements.
|
||||
`mission-core-triton`, `sentinel-frigate` and `sentinel-ollama` remained up.
|
||||
|
||||
## Disk policy and measured usage
|
||||
|
||||
The orchestrator admitted only explicit paths rooted at
|
||||
`D:\NDC_MISSIONCORE`. It performed a conservative space preflight, sampled free
|
||||
space every 100 frames and failed closed if the floor would be crossed.
|
||||
|
||||
| Disk point | Free bytes | Free GiB |
|
||||
|---|---:|---:|
|
||||
| Before run | 410,711,453,696 | 382.505 |
|
||||
| After input extraction | 407,553,695,744 | 379.564 |
|
||||
| After inference | 404,899,336,192 | 377.092 |
|
||||
| After final artifacts | 404,796,903,424 | 376.996 |
|
||||
| After task temporary cleanup | 407,372,025,856 | 379.395 |
|
||||
| Enforced floor | 386,547,056,640 | 360.000 |
|
||||
|
||||
The conservative working-set reserve was 18,462,883,615 bytes (17.195 GiB).
|
||||
The floor was never crossed. Final headroom above the floor was approximately
|
||||
19.395 GiB. No task-controlled C: path was used, created or mounted. This does
|
||||
not claim to observe incidental internal writes by Windows or the pre-existing
|
||||
Docker Desktop engine.
|
||||
|
||||
Only exact E4 temporary extraction, mask and overlay work directories were
|
||||
removed after successful publication. Immutable result artifacts, reports,
|
||||
telemetry, the pilot and runner identities remain recoverable on D:.
|
||||
|
||||
## Published artifacts and integrity
|
||||
|
||||
| Artifact | Bytes | SHA-256 |
|
||||
|---|---:|---|
|
||||
| `perception.mp4` | 75,353,780 | `6e58cf72d9571ebbb003a814132b44907b7c0e31d18b5449cce330afcad29829` |
|
||||
| `masks.tar.gz` | 39,812,627 | `ef291034e51c5c719dcc1c0902bae1e3daedf919d7b55a21fcb0a31af298071f` |
|
||||
| `frames.jsonl` | 3,927,616 | `4780339f5ef94e1f58922dfe2df67ecdf983c4b4e030761f4a990f65b511d5cf` |
|
||||
| `gpu-telemetry.jsonl` | 325,407 | `c6a4db1a8ea01d1ca66aded4bcd990ed91fc015930113869b20f024632a2aacf` |
|
||||
| `run-report.json` | 7,894 | `8ee04bf2ed8e563641dc1dacd320aee6fc8cd4077a43fab564033a9c29cbaa07` |
|
||||
|
||||
The published local result occupies approximately 123 MiB. Independent checks
|
||||
confirmed:
|
||||
|
||||
- all five artifact size/digest pairs match `result.json`;
|
||||
- `frames.jsonl` contains exactly 4,489 ordered timestamp rows;
|
||||
- the mask archive contains exactly 4,489 indexed masks;
|
||||
- FFprobe reports H.264, 800x600, 448.723 seconds and exactly 4,489 frames;
|
||||
- the complete recorded-perception v2 result validates against its job, input,
|
||||
session, source, timestamp bounds and `camera_1` calibration binding;
|
||||
- the result store selects this result as the latest admitted generation for
|
||||
`recorded.perception.right`.
|
||||
|
||||
Durable local result root:
|
||||
|
||||
```text
|
||||
.runtime/compute-results/recorded-camera-602ac89026ed12978619801d/
|
||||
result-793785170472c519486ccd666be102fb04d169d92383acda3fcc29eecf045d30/
|
||||
```
|
||||
|
||||
Worker evidence root:
|
||||
|
||||
```text
|
||||
D:\NDC_MISSIONCORE\runtime\derived\
|
||||
result-793785170472c519486ccd666be102fb04d169d92383acda3fcc29eecf045d30
|
||||
```
|
||||
|
||||
## Mission Core publication and UI proof
|
||||
|
||||
The saved-session replay API returned schema
|
||||
`missioncore.observation-session-replay/v2` with source
|
||||
`recorded.perception.right`, label `Сегментация · камера right`, the exact result
|
||||
ID and the MP4 generation SHA above. The media manifest validated as
|
||||
`missioncore.observation-recorded-media/v3`. A byte-range request returned HTTP
|
||||
206 with `Content-Range: bytes 0-1023/75353780`.
|
||||
|
||||
The final acceptance was performed in the real Mission Core browser UI:
|
||||
|
||||
1. open **Сохранённые сессии**;
|
||||
2. select **RAVNOVES00**;
|
||||
3. open the scene source picker;
|
||||
4. select **Сегментация · камера right**;
|
||||
5. press **Воспроизвести**.
|
||||
|
||||
The visible video element loaded the exact new result, reported 800x600,
|
||||
duration 448.723 seconds, `readyState=4` and no media error. Its current time
|
||||
advanced from 128.522 to 130.282 seconds during a timed browser check, proving
|
||||
that the result is not merely discoverable but actually plays.
|
||||
|
||||
## Visual findings
|
||||
|
||||
Start, middle and end frames were inspected from the final encoded video.
|
||||
|
||||
1. The fixed valid-FOV circle remains aligned across the complete recording;
|
||||
excluded lens pixels do not generate semantic clutter.
|
||||
2. Road, sidewalk, sky, buildings and vegetation form spatially coherent
|
||||
regions at the inspected start/middle/end points.
|
||||
3. Cars and people are visibly segmented when present, but this is semantic
|
||||
class color, not a persistent object identity.
|
||||
4. The output is substantially more usable than the original E0 visual result
|
||||
for playback, while still showing expected Cityscapes/domain artifacts on
|
||||
close structures, thin boundaries and forest/off-road content.
|
||||
5. No visual inspection can substitute for the pending human-reviewed E2
|
||||
accuracy set. The result remains `ground_truth=false`.
|
||||
|
||||
Final contact sheet:
|
||||
`.runtime/compute-experiments/e4/full-preview/start-mid-end.png`, SHA-256
|
||||
`a16f78732346db4b476b7d66dd140857d92eb1f11add5826c49e4dddf0ac1d5f`.
|
||||
|
||||
## Acceptance status and limitations
|
||||
|
||||
Accepted:
|
||||
|
||||
- complete no-sampling processing of the chosen saved session;
|
||||
- zero failed/skipped frames;
|
||||
- immutable semantic masks and frame timestamps;
|
||||
- exact H.264 playback on the original session timeline;
|
||||
- source discovery, manifest and HTTP range transport;
|
||||
- real UI selection and playback;
|
||||
- measured worker resource and disk behavior;
|
||||
- D-only task-controlled worker storage with enforced floor.
|
||||
|
||||
Not accepted or not attempted:
|
||||
|
||||
- live real-time throughput;
|
||||
- instance masks, object IDs or tracking;
|
||||
- 2D detector boxes from an E4 model branch;
|
||||
- 3D cuboids or point-cloud semantic labels;
|
||||
- LiDAR-camera association and distance-to-object accuracy;
|
||||
- forest/off-road model fitness;
|
||||
- mIoU, boundary IoU, AP or safety metrics;
|
||||
- production or safety qualification.
|
||||
|
||||
## Decision and next gate
|
||||
|
||||
E4 closes the first full-session semantic playback milestone. Plain EoMT with
|
||||
the immutable valid-FOV mask is now the operational reference for recorded
|
||||
video: reproducible, watchable and measured.
|
||||
|
||||
The next work should not be another blind full-epoch run. It should proceed in
|
||||
two bounded tracks:
|
||||
|
||||
1. finish human review of the E2 evaluation pack and measure semantic accuracy,
|
||||
especially people, vehicles, road/free-space boundaries and near distorted
|
||||
structures; use those metrics to decide whether to keep the raw fisheye
|
||||
baseline, improve the model/taxonomy or revive seam-aware rectification;
|
||||
2. add an instance/detection and tracking branch, then project accepted 2D
|
||||
observations through the factory camera/LiDAR calibration into the point
|
||||
cloud for distance and 3D-cuboid experiments.
|
||||
|
||||
Live processing should be admitted only after the chosen branches are profiled
|
||||
with bounded queues and an explicit drop/sampling policy. At the present 3.10
|
||||
inference FPS, this exact configuration cannot consume the recorded ~10 FPS
|
||||
camera stream without backlog.
|
||||
@@ -0,0 +1,405 @@
|
||||
# LAB E5 — Instance detection and temporal tracking qualification
|
||||
|
||||
Date: 2026-07-21
|
||||
Completed at: 2026-07-21T08:45:46.307Z
|
||||
State: completed qualification baseline, model output, not ground truth
|
||||
Ops card: MISSIONCOR-19
|
||||
|
||||
## Objective
|
||||
|
||||
Establish the first measured temporal object-identity baseline on recorded K1
|
||||
camera data before attempting LiDAR association and 3D cuboids. The acceptance
|
||||
target was deliberately bounded:
|
||||
|
||||
1. run an upstream, reproducibly pinned detector through the existing Triton
|
||||
service on the AI worker;
|
||||
2. reuse the immutable calibration-bound valid-FOV mask instead of processing
|
||||
the black fisheye border;
|
||||
3. attach persistent IDs with a ByteTrack-style two-stage association loop;
|
||||
4. produce a watchable 2D tracking video and exact timestamped frame metadata;
|
||||
5. measure detector, tracking, artifact, GPU, memory and disk behavior;
|
||||
6. retain all task-controlled worker data under `D:\NDC_MISSIONCORE`, enforce
|
||||
the 360 GiB free-space floor and restore the prior Triton model state;
|
||||
7. record visible failure modes honestly before using these identities for 3D.
|
||||
|
||||
LAB E5 is a qualification clip, not a full-session or saved-session production
|
||||
source. It does not claim tracking ground truth, ID-switch accuracy, instance
|
||||
masks, 3D cuboids, LiDAR distance, point-cloud labels, real-time operation or
|
||||
safety fitness.
|
||||
|
||||
## Immutable input and clip selection
|
||||
|
||||
- UI session alias: `RAVNOVES00`
|
||||
- Session ID: `20260720T065719Z_viewer_live`
|
||||
- Compute job: `recorded-camera-602ac89026ed12978619801d`
|
||||
- Source: `sensor.camera.right`
|
||||
- Factory calibration slot: `camera_1`
|
||||
- Input SHA-256:
|
||||
`602ac89026ed12978619801d4edea0cae24b5cc3afabd9f7af2858de6505a20e`
|
||||
- Factory calibration content SHA-256:
|
||||
`05f3ad9b38b3a4fc95388a8ec83da83c745e217709e51787b3d5aad0969f6fa9`
|
||||
- Codec epoch: 1
|
||||
- Resolution: 800x600
|
||||
- Full source: 4,489 frames, 35.421857292–484.144857292 session seconds
|
||||
- Qualification selection: source frames 1,000–1,600 inclusive
|
||||
- Frames admitted: 601/601
|
||||
- Qualification timeline: 135.365857292–195.334857292 session seconds
|
||||
- Qualification media duration: 60.068948 seconds
|
||||
- Qualification timeline SHA-256:
|
||||
`06e3514228b488137b92c303cbca171154ede00368783e32107ed2c39c41baa6`
|
||||
|
||||
Frames 1,000–1,600 were selected before the full run. The contiguous interval
|
||||
contains stationary and changing views, multiple parked vehicles, distant
|
||||
pedestrians, and a woman with a child/stroller passing close to the camera.
|
||||
That close pass deliberately stresses scale change, fisheye distortion,
|
||||
occlusion and identity continuity. It was not selected from the final output to
|
||||
make the result look favorable.
|
||||
|
||||
The valid image circle is the immutable LAB E1 artifact
|
||||
`valid-fov-mask-b4dd8ddf2b87c1d520ee8a0868c4fea062d7c14d1bae73ccabd3abe1f3acbac2`.
|
||||
Its mask SHA-256 is
|
||||
`a40cee06b7c6f69b6a09a11563dcfd237f3de833b1ccd31459e66692e528ba63`.
|
||||
It admits 270,606 of 480,000 pixels. The mask is loaded once and reused; it is
|
||||
not recomputed per frame.
|
||||
|
||||
## Accepted software configuration
|
||||
|
||||
Detector:
|
||||
|
||||
- official Megvii YOLOX-S ONNX release `0.1.1rc0`;
|
||||
- license: Apache-2.0;
|
||||
- classes: COCO-80, filtered to person, bicycle, car, motorcycle, bus and
|
||||
truck;
|
||||
- model SHA-256:
|
||||
`c5c2d13e59ae883e6af3b45daea64af4833a4951c92d116ec270d9ddbe998063`;
|
||||
- Triton config SHA-256:
|
||||
`5795c737a7935a655961b069e8404d336d891f9762fb6dffb93956a076479604`;
|
||||
- input: FP32 `[1,3,640,640]`, BGR, bilinear top-left letterbox, fill 114;
|
||||
- output: FP32 `[1,8400,85]`, official grid/stride decode;
|
||||
- container: `nvcr.io/nvidia/tritonserver:26.06-py3`;
|
||||
- Triton backend: ONNX Runtime GPU.
|
||||
|
||||
Detection and FOV filtering:
|
||||
|
||||
- detector minimum score: 0.10;
|
||||
- high/new-track threshold: 0.25;
|
||||
- same-class NMS IoU: 0.45;
|
||||
- same-class containment suppression: 0.80;
|
||||
- minimum box area: 64 pixels;
|
||||
- maximum box area: 50% of image;
|
||||
- minimum valid-FOV box fraction: 0.50;
|
||||
- box center must lie inside the valid FOV.
|
||||
|
||||
Tracking:
|
||||
|
||||
- algorithm: `bytetrack-style-two-stage-iou/v1`;
|
||||
- assignment: SciPy linear-sum assignment;
|
||||
- primary high-score match IoU: 0.20;
|
||||
- secondary low-score match IoU: 0.10;
|
||||
- lost-track buffer: 15 frames;
|
||||
- confirmation: 2 hits;
|
||||
- association is class-consistent;
|
||||
- no appearance ReID and no camera-motion compensation.
|
||||
|
||||
Producer identities:
|
||||
|
||||
- pipeline: `recorded-yolox-bytetrack-style-qualification/v1`;
|
||||
- runner SHA-256:
|
||||
`adc7c924cb8d01278ccf63c9eeb3c9f810c09d2d1d7fe169e136102ee32c247a`;
|
||||
- orchestrator SHA-256:
|
||||
`8fff9c87e6213883ebde257a4226765d96b16cd4b8a2a5bb492aef728e3e2242`;
|
||||
- profile SHA-256:
|
||||
`1de8a83ef0fe630ca317b70bbe3a8ec497ae7cb5267fabda2ea234ee2cf2e562`;
|
||||
- Python 3.12.3, NumPy 2.5.1 and SciPy 1.16.3 inside the bounded run
|
||||
container.
|
||||
|
||||
The profile was frozen after the second visual pilot. No thresholds were tuned
|
||||
against the 601-frame result.
|
||||
|
||||
## Execution topology
|
||||
|
||||
The pipeline was sequential at frame level:
|
||||
|
||||
```text
|
||||
recorded fMP4 epoch
|
||||
-> exact RGB frame reconstruction
|
||||
-> fixed calibration-bound valid-FOV fill
|
||||
-> YOLOX-S Triton request
|
||||
-> YOLOX decode and class/FOV filtering
|
||||
-> same-class NMS plus nested-box suppression
|
||||
-> high-score association
|
||||
-> low-score association to unmatched tracks
|
||||
-> confirm/age/retire tracks
|
||||
-> write ID boxes and trails
|
||||
-> append exact timestamped detections/tracks
|
||||
-> encode overlays as H.264
|
||||
-> hash, validate and publish qualification result
|
||||
```
|
||||
|
||||
Batch size was one. Frames were not parallelized. Triton/CUDA may execute
|
||||
internal work asynchronously, but each source frame completed decode,
|
||||
inference, association and overlay writing before the next frame was admitted.
|
||||
NVENC encoded the completed overlay sequence after inference.
|
||||
|
||||
The model was absent from Triton's ready set before the run. The orchestrator
|
||||
loaded it only for LAB E5 and restored the original unloaded state in `finally`.
|
||||
The pre-existing `mission-core-triton`, `sentinel-frigate` and
|
||||
`sentinel-ollama` services remained running.
|
||||
|
||||
## Pilot history and configuration freeze
|
||||
|
||||
The first launch failed closed during preflight because the base Triton image
|
||||
did not contain Pillow. It performed no inference, loaded no model and
|
||||
published no result. The correction mounted the already existing D-only
|
||||
perception environment read-only at `/opt/env`; no new environment was placed
|
||||
on C:.
|
||||
|
||||
The first successful 32-frame pilot used frames 1,230–1,261 and published the
|
||||
immutable diagnostic result
|
||||
`e5-tracking-005090ae34dc2c09d75f0dcfe68b4f9eed9e011d480e0c4b0d650be9369ed841`.
|
||||
It exposed two implementation effects:
|
||||
|
||||
1. nested same-class body/torso boxes survived conventional IoU NMS;
|
||||
2. the first SciPy assignment import contaminated the first tracking latency
|
||||
with a 422 ms one-time cost.
|
||||
|
||||
The accepted runner added 80% containment suppression and warmed the assignment
|
||||
solver before timed frames. A second 32-frame pilot then published
|
||||
`e5-tracking-8035c0013e36a7cfb68aa3652b05ab79fe36baf136462d5fbf91b348ee85ec08`.
|
||||
It processed 32/32 frames in 4.843450 seconds at 6.606861 FPS, reduced
|
||||
detections from 235 to 227, retained zero same-class pairs at IoU >= 0.8 and
|
||||
reduced tracking p95 to 0.401 ms. Visual inspection confirmed that the remaining
|
||||
nearby person boxes represented the woman and child, not nested duplicates.
|
||||
|
||||
Both successful pilots remain immutable evidence. Only the second profile was
|
||||
admitted to the main qualification run.
|
||||
|
||||
## Qualification result
|
||||
|
||||
Published result:
|
||||
`e5-tracking-88aace13ef9963f8dc07f85228e530f9d28c2b49aca9192409f7975512b058f6`.
|
||||
|
||||
| Measurement | Result |
|
||||
|---|---:|
|
||||
| Frames expected / processed | 601 / 601 |
|
||||
| Frames failed / skipped | 0 / 0 |
|
||||
| Frames with confirmed tracks | 600 |
|
||||
| Inference/tracking/artifact loop | 88.092835 s |
|
||||
| Loop throughput | 6.822348 FPS |
|
||||
| Stream extraction | 10.243353 s |
|
||||
| H.264 encoding | 0.686277 s |
|
||||
| Full worker wall | 166.335664 s |
|
||||
| End-to-end throughput | 3.613176 FPS |
|
||||
| Process peak RSS | 126.199 MiB |
|
||||
|
||||
The source rate is approximately 10.004 FPS. The measured frame loop is about
|
||||
1.47 times slower than source real time, and the complete worker run is about
|
||||
2.77 times slower. This exact artifact-heavy path is therefore offline. It is
|
||||
not admitted as a live pipeline.
|
||||
|
||||
### Per-frame latency
|
||||
|
||||
| Stage | Mean | p50 | p95 | Max |
|
||||
|---|---:|---:|---:|---:|
|
||||
| End to end | 146.279 ms | 144.622 ms | 163.340 ms | 215.759 ms |
|
||||
| Overlay write | 87.363 ms | 86.227 ms | 99.219 ms | 139.534 ms |
|
||||
| Image decode | 30.132 ms | 28.455 ms | 40.723 ms | 63.155 ms |
|
||||
| Triton request | 12.359 ms | 12.148 ms | 17.948 ms | 31.662 ms |
|
||||
| Detector postprocess | 7.307 ms | 7.016 ms | 10.734 ms | 21.003 ms |
|
||||
| Preprocess | 6.202 ms | 5.803 ms | 7.991 ms | 18.645 ms |
|
||||
| Tracking | 0.247 ms | 0.234 ms | 0.390 ms | 0.894 ms |
|
||||
|
||||
The tracker is not the speed problem. The Triton request itself is also below
|
||||
the approximately 100 ms/frame source budget on average. Synchronous PNG
|
||||
overlay writing and image decoding dominate this qualification loop. A live
|
||||
design must decouple visualization/artifact production from the inference
|
||||
critical path and define bounded queues and a drop policy; it must not simply
|
||||
run this script against an unbounded stream.
|
||||
|
||||
## Detection and identity observations
|
||||
|
||||
| Metric | Result |
|
||||
|---|---:|
|
||||
| Detections admitted | 4,049 |
|
||||
| Confirmed track observations | 3,320 |
|
||||
| Confirmed track IDs | 167 |
|
||||
| Tracker tracks created / retired | 177 / 166 |
|
||||
| Confirmed length mean / p50 / p95 / max | 19.88 / 6 / 94 / 212 frames |
|
||||
| Same-class duplicate pairs at IoU >= 0.8 | 0 |
|
||||
| Rejected oversized boxes | 7 |
|
||||
|
||||
Admitted detections by detector label:
|
||||
|
||||
| Label | Detections | Confirmed observations | Confirmed IDs |
|
||||
|---|---:|---:|---:|
|
||||
| car | 2,972 | 2,556 | 123 |
|
||||
| truck | 646 | 483 | 15 |
|
||||
| person | 364 | 249 | 24 |
|
||||
| bicycle | 45 | 29 | 3 |
|
||||
| motorcycle | 16 | 2 | 1 |
|
||||
| bus | 6 | 1 | 1 |
|
||||
|
||||
Track persistence without ground-truth interpretation:
|
||||
|
||||
- 150/167 IDs had at least 2 confirmed observations;
|
||||
- 97/167 had at least 5 observations;
|
||||
- 63/167 had at least 10 observations;
|
||||
- 30/167 had at least 30 observations;
|
||||
- 15/167 had at least 60 observations;
|
||||
- 7/167 had at least 100 observations.
|
||||
|
||||
These counts prove that the temporal state machine is operating; they do not
|
||||
measure accuracy. The high number of short vehicle IDs is a visible baseline
|
||||
limitation, driven by changing viewpoint, intermittent low-confidence
|
||||
detections and class changes between `car` and `truck` while association is
|
||||
class-consistent.
|
||||
|
||||
### Visual findings
|
||||
|
||||
1. Boxes remain inside the calibrated valid image circle; the black lens border
|
||||
no longer produces visual clutter or tracking candidates.
|
||||
2. Parked vehicles and normally moving people retain IDs over useful clear-view
|
||||
intervals. Several vehicle tracks persist for 100–212 frames.
|
||||
3. The close woman/child sequence demonstrates both success and failure. IDs
|
||||
54 and 60 persist through the unobstructed approach. When the woman/stroller
|
||||
fills the lower-left fisheye region and becomes heavily occluded, ID 60 is
|
||||
lost and a new person ID 76 begins. This is a real identity fragmentation,
|
||||
not hidden by the report.
|
||||
4. YOLOX-S sometimes classifies the stroller as `motorcycle`, and the same
|
||||
parked vehicle can alternate between `car` and `truck`. COCO has no stroller
|
||||
class, and the generic road detector is not calibrated for the K1 fisheye.
|
||||
5. Conventional NMS plus containment suppression removed the visible nested
|
||||
body/torso duplicates from the first pilot without suppressing the distinct
|
||||
woman and child.
|
||||
6. There is no human-reviewed identity ground truth. IDF1, HOTA, MOTA and true
|
||||
ID-switch count therefore remain unknown and must not be inferred from the
|
||||
track-length statistics.
|
||||
|
||||
## GPU and neighboring services
|
||||
|
||||
Worker GPU: NVIDIA GeForce RTX 4090.
|
||||
|
||||
- one-second GPU samples: 88;
|
||||
- GPU utilization: 51.65% mean, 52% p50, 55% p95, 56% max;
|
||||
- board memory used, including pre-existing services: 10,239.0 MiB mean,
|
||||
10,249 MiB max;
|
||||
- memory-controller utilization: 6.11% mean, 7% max;
|
||||
- power: 144.78 W mean, 146.71 W p95, 148.40 W max;
|
||||
- temperature: 42.60 C mean, 43 C max;
|
||||
- process peak RSS: 126.199 MiB.
|
||||
|
||||
The board-memory value is total device use observed by `nvidia-smi`, not an E5
|
||||
process allocation. This run did not expose a trustworthy per-process CUDA
|
||||
peak, so none is claimed.
|
||||
|
||||
## Disk policy and measured usage
|
||||
|
||||
The orchestrator admitted only explicit worker paths rooted at
|
||||
`D:\NDC_MISSIONCORE`. It reserved a conservative 4,726,173,047-byte working set,
|
||||
checked D: at each major phase and every 100 frames, and failed closed if the
|
||||
360 GiB floor would be crossed.
|
||||
|
||||
| Disk point | Free bytes | Free GiB |
|
||||
|---|---:|---:|
|
||||
| Before run | 409,176,440,832 | 381.075 |
|
||||
| After frame extraction | 408,682,668,032 | 380.615 |
|
||||
| After inference | 408,303,550,464 | 380.262 |
|
||||
| After final artifacts | 408,300,429,312 | 380.259 |
|
||||
| After exact temporary cleanup | 408,660,688,896 | 380.595 |
|
||||
| Enforced floor | 386,547,056,640 | 360.000 |
|
||||
|
||||
The floor was never approached closer than 20.259 GiB. Only exact E5 temporary
|
||||
stream, frame and overlay roots were removed. The immutable pilots, accepted
|
||||
result, telemetry, runner and profiles remain on D:. No task-controlled C: path
|
||||
was created, mounted or used. This does not claim visibility into incidental
|
||||
internal writes by Windows or the pre-existing Docker Desktop engine.
|
||||
|
||||
## Artifacts and independent validation
|
||||
|
||||
| Artifact | Bytes | SHA-256 |
|
||||
|---|---:|---|
|
||||
| `tracking.mp4` | 12,270,533 | `d911efc46c5c3df38ca961700f82c407c44222ba465af752d2a7de75911740ba` |
|
||||
| `frames.jsonl` | 1,116,700 | `e5f19e5dd8bf76040cb7859ffada3f1bb68a0f5cd93fac243947276899cce014` |
|
||||
| `gpu-telemetry.jsonl` | 19,694 | `84cbaa4d5a7d503129f2fa329ab2b231cff54a6972cd31ba95fe91f48f01adad` |
|
||||
| `contact-sheet.png` | 926,575 | `d9a23cc76e1a9a4aba1156609f85b781bc2502cd80f3b17ac83ca4228f2121a8` |
|
||||
| `run-report.json` | 9,183 | `96d1cb059b35a0aa3fcc7c3458a0f47f57a4bd52a64c34f3c7607302e49206d9` |
|
||||
|
||||
Independent local validation confirmed:
|
||||
|
||||
- the result directory name equals its canonical identity SHA-256;
|
||||
- all five artifact size/digest pairs match `result.json`;
|
||||
- the result is bound to the exact job, input, session, source, clip bounds and
|
||||
timestamp basis;
|
||||
- `frames.jsonl` contains exactly 601 ordered source indices and strictly
|
||||
increasing session timestamps;
|
||||
- every reported detection and track box is finite and inside 800x600;
|
||||
- no frame failed or was skipped;
|
||||
- FFprobe reports H.264, yuv420p, 800x600, 60.068948 seconds, and exactly
|
||||
601 declared/read frames;
|
||||
- the custom result validator accepts the immutable result;
|
||||
- the E5 source and test files pass Ruff, and the focused E5 test suite covers
|
||||
profile pinning, FOV math, YOLOX decode, containment suppression, two-stage
|
||||
low-score identity retention, class isolation and nonzero source timelines.
|
||||
|
||||
Durable local result root:
|
||||
|
||||
```text
|
||||
.runtime/compute-experiments/e5/results/
|
||||
e5-tracking-88aace13ef9963f8dc07f85228e530f9d28c2b49aca9192409f7975512b058f6/
|
||||
```
|
||||
|
||||
Worker evidence root:
|
||||
|
||||
```text
|
||||
D:\NDC_MISSIONCORE\runtime\derived\
|
||||
e5-tracking-88aace13ef9963f8dc07f85228e530f9d28c2b49aca9192409f7975512b058f6
|
||||
```
|
||||
|
||||
## Acceptance status
|
||||
|
||||
Accepted:
|
||||
|
||||
- reproducible official YOLOX-S inference through the existing Triton worker;
|
||||
- fixed calibration-bound valid-FOV preprocessing;
|
||||
- ByteTrack-style high/low-score temporal association;
|
||||
- 601/601 exact recorded frames with no failures or skips;
|
||||
- watchable 2D ID overlay, trails and complete frame metadata;
|
||||
- honest timing, GPU, process and disk telemetry;
|
||||
- content-addressed immutable qualification artifacts;
|
||||
- D-only task-controlled worker storage and restoration of Triton model state.
|
||||
|
||||
Not accepted:
|
||||
|
||||
- production-quality object classes or identity continuity;
|
||||
- IDF1, HOTA, MOTA or measured ID-switch rate;
|
||||
- handling of heavy close occlusion, re-entry or long disappearance;
|
||||
- appearance ReID or camera-motion compensation;
|
||||
- instance masks, 3D cuboids, LiDAR association or metric distance;
|
||||
- a live unbounded stream or safety use;
|
||||
- promotion of the partial clip into the saved-session production source list.
|
||||
|
||||
## Decision and next gate
|
||||
|
||||
LAB E5 closes the first temporal identity baseline. The result is useful enough
|
||||
to become input to a bounded 3D fusion experiment, but not strong enough to be
|
||||
called production tracking.
|
||||
|
||||
The next gate should be LAB E6 on the same frozen 601-frame interval:
|
||||
|
||||
1. use factory `camera_1` calibration to project synchronized LiDAR points into
|
||||
each accepted 2D observation;
|
||||
2. reject points outside the valid FOV and use robust depth/cluster gates rather
|
||||
than a single nearest pixel;
|
||||
3. compute per-object range and coarse point-supported 3D cuboids with explicit
|
||||
support counts and confidence;
|
||||
4. smooth distance over the existing track ID where continuity is available;
|
||||
5. treat COCO `car`/`truck` as one vehicle association group in the fusion
|
||||
layer so detector class flicker does not automatically break geometry;
|
||||
6. publish a bounded Rerun qualification recording only after visual and
|
||||
numeric checks pass.
|
||||
|
||||
In parallel with, but not as a prerequisite for, this first 3D proof, the 2D
|
||||
branch should later compare camera-motion compensation and appearance ReID on a
|
||||
small human-reviewed identity set. Full-session and live runs should wait until
|
||||
the artifact-writing path is removed from the inference critical section and a
|
||||
bounded queue/drop policy has been tested.
|
||||
@@ -0,0 +1,292 @@
|
||||
# LAB E6 — Factory-calibrated tracked LiDAR fusion
|
||||
|
||||
Date: 2026-07-21
|
||||
Completed at: 2026-07-21T09:18:15.532Z
|
||||
State: completed qualification baseline, diagnostic geometry, not ground truth
|
||||
Ops card: pending direct `nodedc-ops-agent` availability
|
||||
|
||||
## Objective
|
||||
|
||||
Connect the already qualified E5 temporal identities to metric observations from
|
||||
the K1 LiDAR without inventing depth from image boxes. The bounded acceptance
|
||||
target was:
|
||||
|
||||
1. reuse the exact 601-frame E5 interval and E4 semantic masks;
|
||||
2. bind the right camera to factory calibration slot `camera_1`;
|
||||
3. project raw K1 points with the factory KB4 intrinsics and camera/LiDAR
|
||||
extrinsics;
|
||||
4. reject camera/point/pose combinations outside a strict 100 ms host-arrival
|
||||
gate;
|
||||
5. associate only semantically compatible, nearest-depth, spatially connected
|
||||
LiDAR support with each tracked image object;
|
||||
6. estimate a robust metric range and a point-supported oriented 3D envelope;
|
||||
7. publish a watchable overlay and a standalone Rerun recording containing the
|
||||
camera view, map-frame point cloud, support points and translucent `Boxes3D`;
|
||||
8. measure the geometry path and preserve every pilot and accepted artifact.
|
||||
|
||||
LAB E6 proves that the existing XGRIDS factory calibration can drive useful
|
||||
recorded camera/LiDAR fusion. It does not claim amodal vehicle dimensions,
|
||||
object-complete 3D boxes, ground truth, live-rate admission or safety fitness.
|
||||
|
||||
## Immutable inputs
|
||||
|
||||
- UI session alias: `RAVNOVES00`
|
||||
- Session ID: `20260720T065719Z_viewer_live`
|
||||
- Camera job: `recorded-camera-602ac89026ed12978619801d`
|
||||
- Camera source: `sensor.camera.right`
|
||||
- Factory calibration slot: `camera_1`
|
||||
- Camera input SHA-256:
|
||||
`602ac89026ed12978619801d4edea0cae24b5cc3afabd9f7af2858de6505a20e`
|
||||
- Factory calibration content SHA-256:
|
||||
`05f3ad9b38b3a4fc95388a8ec83da83c745e217709e51787b3d5aad0969f6fa9`
|
||||
- Raw MQTT capture SHA-256:
|
||||
`70da0edad5cbf0e89b6e2355c0aac294f8c36a67249cd33b8918353fd88c83af`
|
||||
- E4 semantic result:
|
||||
`result-793785170472c519486ccd666be102fb04d169d92383acda3fcc29eecf045d30`
|
||||
- E5 tracking result:
|
||||
`e5-tracking-88aace13ef9963f8dc07f85228e530f9d28c2b49aca9192409f7975512b058f6`
|
||||
- Source frames: 1,000–1,600 inclusive
|
||||
- Frames admitted: 601/601
|
||||
- Session interval: 135.365857292–195.334857292 seconds
|
||||
- Media duration: 60.068948 seconds
|
||||
- Resolution: 800x600
|
||||
|
||||
The 127 MiB MQTT capture contains the raw point and pose observations used by
|
||||
the run. The calibration snapshot files are preserved separately from Git:
|
||||
|
||||
| File | SHA-256 |
|
||||
|---|---|
|
||||
| `camera.yaml` | `d123ebd2207e18a800cf5d07d055bf38231d28390bf3ba8c4e718e3cb3a7775e` |
|
||||
| `extrinsic_camera_lidar.yaml` | `d09b2eda51e1deb7e3a31ff8441cba6006e8da60afc3e33de44a857437dfcfb8` |
|
||||
| `manifest.json` | `75e1d582f661dabafb48fc6e8c7aabe0f217e0d95ed934b7d9dac18bdfc1216c` |
|
||||
|
||||
## Accepted configuration
|
||||
|
||||
Producer identities:
|
||||
|
||||
- pipeline: `factory-kb4-tracked-box-semantic-lidar/v1`;
|
||||
- profile schema: `missioncore.e6-tracking-lidar-profile/v1`;
|
||||
- profile SHA-256:
|
||||
`b8d3a6f043fde54ffd32ea05bfe580e2be7aace1a23cdbf7e266eb3325d147b2`;
|
||||
- runner SHA-256:
|
||||
`9765be5e0338b9978db7a58a739313b228bc7ceee1352a7a76af0ff2a528a953`.
|
||||
|
||||
Temporal binding:
|
||||
|
||||
- nearest host-arrival observation, best effort;
|
||||
- maximum LiDAR/camera delta: 100 ms;
|
||||
- maximum pose/point delta: 100 ms;
|
||||
- frames outside the gate fail closed for depth; image tracking remains visible.
|
||||
|
||||
Projection and occlusion:
|
||||
|
||||
- factory KB4 projection for `camera_1`;
|
||||
- source points transformed from the K1 map frame through the recorded pose;
|
||||
- only camera-front, in-frame points are retained;
|
||||
- one nearest-depth point is retained per rounded image pixel;
|
||||
- no fabricated depth and no 2D-box extrusion.
|
||||
|
||||
Association:
|
||||
|
||||
- image boxes are inset 5% horizontally, 4% at the top and 2% at the bottom;
|
||||
- `car`, `truck` and `bus` share the `vehicle` association group;
|
||||
- same-frame vehicle overlap suppression uses IoU 0.55;
|
||||
- E4 semantic IDs gate LiDAR support: person 1, bicycle 2, motorcycle 3,
|
||||
vehicle 4/5;
|
||||
- depth splitting uses `max(0.5 m, 6% of range)`;
|
||||
- the selected depth component is split again by 3D connected components;
|
||||
- spatial radii are 0.9 m for person/bicycle/motorcycle and 1.5 m for vehicle;
|
||||
- minimum support is 3 points for person/bicycle/motorcycle and 5 for vehicle;
|
||||
- distance uses cluster median, with p10 retained for diagnostics;
|
||||
- a five-frame track history smooths distance;
|
||||
- an observation is rejected when its distance innovation exceeds
|
||||
`max(1.5 m, 25% of track history)`.
|
||||
|
||||
Cuboid construction:
|
||||
|
||||
- yaw comes from PCA over supported map-frame XY points;
|
||||
- dimensions are the oriented p05–p95 support extent;
|
||||
- every axis has a 0.15 m diagnostic minimum for visibility;
|
||||
- maximum accepted extents are 1.5x1.5x2.8 m for person,
|
||||
3.5x2.0x2.5 m for bicycle/motorcycle and 6.5x3.5x4.0 m for vehicle;
|
||||
- rejected or unsupported detections never produce a 3D cuboid.
|
||||
|
||||
## Execution topology and resource boundary
|
||||
|
||||
The E6 geometry stage ran as a single local CPU process on the Mission Core
|
||||
host, an Apple M3 Pro with 18 GiB RAM. It used Python 3.12.13, NumPy 2.5.1,
|
||||
Pillow 12.3.0, Rerun 0.34.1 and FFmpeg 7.1.1. The external AI worker and its
|
||||
Triton model state were not changed. No path on worker disk C: or D: was read,
|
||||
written or cleaned by this stage.
|
||||
|
||||
```text
|
||||
exact E5 frame + track IDs
|
||||
+ exact E4 semantic mask
|
||||
+ nearest gated K1 point/pose sample
|
||||
-> factory KB4 camera projection
|
||||
-> per-pixel nearest-depth buffer
|
||||
-> box inset + semantic support gate
|
||||
-> 1D depth split + 3D connected component
|
||||
-> range innovation gate
|
||||
-> robust range + PCA-yaw p05/p95 envelope
|
||||
-> overlay / NPZ / JSONL / Rerun recording
|
||||
-> content hash, qualification validation and publish
|
||||
```
|
||||
|
||||
Frames were processed sequentially. Artifact writing was performed inside the
|
||||
same measured process. This is a recorded qualification path, not the live
|
||||
queue/worker architecture.
|
||||
|
||||
## Pilot history and profile freeze
|
||||
|
||||
Three immutable 32-frame pilots were retained:
|
||||
|
||||
1. `e6-fusion-a72468...` exposed 7–8 m person envelopes when a depth-only
|
||||
cluster admitted background structure. It was rejected.
|
||||
2. `e6-fusion-693246...` added 3D connectivity and plausible size gates. Box
|
||||
dimensions became plausible, but range could still jump between foreground
|
||||
and background support on the same track. It was rejected.
|
||||
3. `e6-fusion-70fa670ed9005957266ae7a46ac2bfd2d526f1ea170cd052f4494a2397fff76f`
|
||||
added the distance-innovation gate and was accepted. It fused 32/32 frames,
|
||||
produced 47 cuboids across five track IDs, rejected two range jumps, ran in
|
||||
3.681 seconds and peaked at 159.828 MiB RSS.
|
||||
|
||||
The accepted configuration was frozen after the third pilot. The 601-frame
|
||||
result was not used to tune its thresholds.
|
||||
|
||||
## Qualification result
|
||||
|
||||
Published result:
|
||||
`e6-fusion-b4e4226674a66f6196c033785eb307c255a7bdd8493ef9809dfb1d6e5bd68eaa`.
|
||||
|
||||
| Measurement | Result |
|
||||
|---|---:|
|
||||
| Frames expected / processed | 601 / 601 |
|
||||
| Frames fused | 526 |
|
||||
| Frames outside strict sync gate | 75 |
|
||||
| Frames with at least one accepted cuboid | 438 |
|
||||
| Accepted cuboids | 918 |
|
||||
| Unique track IDs with depth | 56 |
|
||||
| Source cloud points consumed | 1,182,292 |
|
||||
| Accepted support points | 14,025 |
|
||||
| Geometry/artifact wall time | 14.774254 s |
|
||||
| Effective throughput | 40.678 FPS |
|
||||
| User / system CPU | 13.571966 / 0.772126 s |
|
||||
| Process peak RSS | 240.469 MiB |
|
||||
|
||||
This recorded geometry/artifact stage is about 4.07 times faster than the
|
||||
approximately 10.004 FPS camera source. That does not establish a live
|
||||
end-to-end rate: E4 segmentation and E5 detection/tracking are precomputed
|
||||
inputs and were not included in the 14.774-second measurement.
|
||||
|
||||
### Synchronization
|
||||
|
||||
| Delta | Mean | p50 | p95 | Max |
|
||||
|---|---:|---:|---:|---:|
|
||||
| LiDAR to camera | 29.389 ms | 23.905 ms | 70.794 ms | 96.107 ms |
|
||||
| Pose to point cloud | 7.940 ms | 6.370 ms | 19.113 ms | 46.139 ms |
|
||||
|
||||
All fused frames satisfy the configured 100 ms gates. The 75 nonconforming
|
||||
camera frames retain their 2D presentation but carry no depth or 3D claim.
|
||||
Host-arrival time is still only a best-effort clock; hardware timestamps have
|
||||
not been ground-truthed.
|
||||
|
||||
### Accepted geometry distribution
|
||||
|
||||
| Group | Cuboids | Unique track IDs |
|
||||
|---|---:|---:|
|
||||
| Vehicle | 809 | 43 |
|
||||
| Person | 96 | 12 |
|
||||
| Bicycle | 13 | 1 |
|
||||
| Total | 918 | 56 |
|
||||
|
||||
- accepted range: 1.239–37.749 m, p50 10.189 m, p95 24.126 m;
|
||||
- support per cuboid: minimum 3, p50 10, p95 45, maximum 89 points;
|
||||
- p95 oriented extent: 5.383x1.197x2.222 m;
|
||||
- maximum admitted extent: 6.490x2.948x2.783 m.
|
||||
|
||||
The extent is an envelope of the visible supported surface. It is intentionally
|
||||
not an amodal estimate of the object's complete physical volume. A parked car
|
||||
seen from one side can therefore receive a thin cuboid that does not wrap the
|
||||
whole vehicle silhouette.
|
||||
|
||||
### Fail-closed decisions
|
||||
|
||||
| Status | Observations |
|
||||
|---|---:|
|
||||
| Accepted point-supported cuboid | 918 |
|
||||
| Fewer than 5 vehicle support points | 1,023 |
|
||||
| No semantically compatible LiDAR support | 365 |
|
||||
| Rejected distance innovation | 209 |
|
||||
| Fewer than 3 non-vehicle support points | 28 |
|
||||
| Implausible cuboid extent | 24 |
|
||||
|
||||
These rejection counts are part of the result, not discarded errors. In
|
||||
particular, the high low-support count prevents distant image detections from
|
||||
being turned into false metric objects.
|
||||
|
||||
## Rerun and visual QA
|
||||
|
||||
The Rerun recording logs seven entities across the `session_time` timeline:
|
||||
|
||||
- encoded camera fusion overlay;
|
||||
- raw map-frame LiDAR points;
|
||||
- selected object support points;
|
||||
- translucent oriented `Boxes3D` with track/class/range labels;
|
||||
- a visible qualification contract stating the timing and safety limits.
|
||||
|
||||
The web viewer was loaded from the local gRPC proxy and inspected at multiple
|
||||
timestamps. Camera overlay, point cloud, support and `Boxes3D` layers load
|
||||
together. Selecting a cuboid exposes its real center, half-sizes, quaternion,
|
||||
fill mode and label in Rerun. Browser logs contained no load/render errors;
|
||||
only Rerun web-backend deprecation/default warnings were present.
|
||||
|
||||
## Artifacts
|
||||
|
||||
| Artifact | Bytes | SHA-256 |
|
||||
|---|---:|---|
|
||||
| `fusion.npz` | 9,745,489 | `3e0e63797c56505eb9755d253f262459d46ec841e80cc5b785a6b495ed2007f9` |
|
||||
| `fusion-frames.jsonl` | 1,607,124 | `ea1621bb337c6c268f91f6e2c7cef997d78672da4025975328c9e2583a0718e0` |
|
||||
| `box-labels.json` | 30,328 | `0c423024a086344fcdbac8665635a45e88fef81270be73d9e1af3e6bc4eb860c` |
|
||||
| `fusion-overlay.mp4` | 24,156,459 | `a863d1910eb973cef1f4cca28432cb16b26030149e197ff7f9ad3d168b69954c` |
|
||||
| `fusion.rrd` | 63,092,299 | `77b0b91171176dd7ea6d2ef726cc946ab1fc24367750101d3f10617360cb4d63` |
|
||||
| `contact-sheet.png` | 3,066,509 | `adcad4db4fd3089175ed947b312129fc86f04abc065314175eceb62368adac49` |
|
||||
| `run-report.json` | 5,405 | `ca01cb5fa0325d80504199b623f2972141f45b680b4b16ad6a5271f9dd1deb2c` |
|
||||
|
||||
FFprobe independently confirmed H.264/yuv420p, 800x600, 601 declared/read
|
||||
frames and 60.068948 seconds. The custom E6 validator rehashed every artifact
|
||||
and verified the exact session, job, camera, calibration, clip, upstream result
|
||||
and configuration identities. The complete E6 workspace including all pilots
|
||||
uses 124 MiB on the Mission Core host.
|
||||
|
||||
## What is proven and what remains open
|
||||
|
||||
Proven:
|
||||
|
||||
- the right-camera/`camera_1` factory calibration is operational for recorded
|
||||
K1 camera/LiDAR projection;
|
||||
- E5 track identities can receive robust metric LiDAR ranges;
|
||||
- unsupported 2D boxes do not become fake 3D geometry;
|
||||
- Rerun can display the camera, map cloud, support and true oriented 3D
|
||||
primitives on one timeline;
|
||||
- the geometry stage itself is comfortably faster than source rate on this
|
||||
host.
|
||||
|
||||
Not proven:
|
||||
|
||||
- object-complete vehicle/person volume or metric box accuracy;
|
||||
- calibration error in pixels or centimetres against a measured target;
|
||||
- hardware-clock synchronization accuracy;
|
||||
- tracking accuracy, instance-mask accuracy or semantic ground truth;
|
||||
- live end-to-end throughput, bounded queues, drop policy or recovery;
|
||||
- safety or navigation fitness.
|
||||
|
||||
## Next gate — LAB E7
|
||||
|
||||
Use this exact E6 result as the control and improve 3D geometry without hiding
|
||||
rejections. The bounded E7 experiment should add ground removal, object-aware
|
||||
support completion and a reviewed upstream 3D detection baseline; compare box
|
||||
coverage, range stability, support/rejection rates and runtime on the same
|
||||
601-frame interval. Only after that comparison should the winning geometry
|
||||
path be connected to a bounded live queue. Multi-camera fusion and product UI
|
||||
work remain outside this gate.
|
||||
@@ -0,0 +1,348 @@
|
||||
# LAB E7 — Bounded replay-real-time perception loop
|
||||
|
||||
Date: 2026-07-21
|
||||
Completed at: 2026-07-21T10:12:16.828Z
|
||||
State: accepted replay scheduler and world-state qualification
|
||||
Navigation/safety state: not accepted
|
||||
Ops card: pending direct `nodedc-ops-agent` availability
|
||||
|
||||
## Objective
|
||||
|
||||
Stop treating recorded perception as a video-enhancement exercise and establish
|
||||
the first measured runtime contract for the K1 as the eyes of an autonomous
|
||||
platform. LAB E7 replays the immutable E6 interval at its original timing and
|
||||
requires the derived perception path to:
|
||||
|
||||
1. publish a machine-readable world state, not only a rendered overlay;
|
||||
2. keep derived work in a small bounded queue;
|
||||
3. absorb short source bursts without discarding processable frames;
|
||||
4. discard old derived frames under real overload instead of building latency;
|
||||
5. expose result age, queue depth, drops and explicit health states;
|
||||
6. preserve object identity, metric range and 3D geometry in both map and
|
||||
LiDAR-relative coordinates;
|
||||
7. reject untrustworthy velocity estimates instead of publishing extreme
|
||||
motion caused by cuboid jitter;
|
||||
8. emit a diagnostic polar clearance observation from the LiDAR;
|
||||
9. prove the scheduler at the approximately 10 Hz source rate before connecting
|
||||
the real inference worker or live K1 stream.
|
||||
|
||||
This experiment qualifies the downstream scheduler, world-state contract and
|
||||
diagnostic publication path. E4 segmentation, E5 detection/tracking and E6
|
||||
fusion are immutable precomputed inputs. LAB E7 therefore does not claim live
|
||||
inference, live sensor transport, collision avoidance or navigation fitness.
|
||||
|
||||
## Immutable input binding
|
||||
|
||||
- Session: `20260720T065719Z_viewer_live`
|
||||
- Source: `sensor.camera.right`
|
||||
- Camera slot: `camera_1`
|
||||
- Camera job: `recorded-camera-602ac89026ed12978619801d`
|
||||
- Input SHA-256:
|
||||
`602ac89026ed12978619801d4edea0cae24b5cc3afabd9f7af2858de6505a20e`
|
||||
- Factory calibration SHA-256:
|
||||
`05f3ad9b38b3a4fc95388a8ec83da83c745e217709e51787b3d5aad0969f6fa9`
|
||||
- E6 result:
|
||||
`e6-fusion-b4e4226674a66f6196c033785eb307c255a7bdd8493ef9809dfb1d6e5bd68eaa`
|
||||
- E7 selection: E6 frames 0–600 inclusive
|
||||
- Source frames: 1,000–1,600 inclusive
|
||||
- Frames attempted: 601
|
||||
- Replay timing span: 59.969 seconds
|
||||
- Replay speed: 1.0x
|
||||
|
||||
The adapter spent 9.490253 seconds before the timed replay revalidating inputs,
|
||||
reading the raw point/pose capture and reconstructing the exact E6 pose binding.
|
||||
That preparation is measured separately and is not counted as live latency. A
|
||||
real source will receive frames incrementally and must not preload a multi-hour
|
||||
session into memory.
|
||||
|
||||
## Accepted configuration
|
||||
|
||||
Producer identities:
|
||||
|
||||
- pipeline: `bounded-latest-wins-e6-world-state-replay/v1`;
|
||||
- profile schema: `missioncore.e7-replay-realtime-profile/v1`;
|
||||
- profile SHA-256:
|
||||
`7222c65ddef20798a242d83edc53973cdfd6f5dec8fb9ffb6fcb5dbcef57f1c5`;
|
||||
- runner SHA-256:
|
||||
`ce36b919fe5860bbb81f04b29c58a8a333915b1ba9bd7f233f256ce77163b15d`;
|
||||
- runtime contract SHA-256:
|
||||
`4c65140550e47c08c528b10d51922d5f73b7d19a1c34ef07896c052f492309f5`;
|
||||
- result validator SHA-256:
|
||||
`719b966ff263f196a97b4d72e946b00fd10705a0158b7df6ecf452c6c9bc4e5b`;
|
||||
- E6 replay adapter SHA-256:
|
||||
`9765be5e0338b9978db7a58a739313b228bc7ceee1352a7a76af0ff2a528a953`.
|
||||
|
||||
Scheduler:
|
||||
|
||||
- one source-paced producer and one derived-state consumer;
|
||||
- replay speed 1.0x using the original per-frame session timestamps;
|
||||
- queue capacity: two derived frames;
|
||||
- overflow policy: evict the oldest queued derived frame;
|
||||
- normal consumer delay injection: zero;
|
||||
- stale threshold: 300 ms;
|
||||
- unavailable threshold: 1,000 ms;
|
||||
- raw evidence is outside this queue and is never dropped by it.
|
||||
|
||||
Acceptance thresholds:
|
||||
|
||||
- delivery rate at least 9.5 Hz;
|
||||
- p95 result age at most 300 ms;
|
||||
- drop fraction at most 1%;
|
||||
- maximum queue depth two;
|
||||
- process RSS at most 512 MiB.
|
||||
|
||||
## World-state contract
|
||||
|
||||
Each published state uses
|
||||
`missioncore.live-perception-world-state/v1` and contains:
|
||||
|
||||
- exact E6/source frame identity and session timestamp;
|
||||
- `healthy`, `degraded`, `stale` or `unavailable` delivery health;
|
||||
- the reason for every non-healthy state;
|
||||
- accepted obstacle track ID, class, detector label and confidence;
|
||||
- map-frame and K1-LiDAR-frame 3D positions;
|
||||
- map-frame orientation and visible-surface cuboid dimensions;
|
||||
- robust LiDAR range and support-point count;
|
||||
- qualified diagnostic map-frame velocity or an explicit rejection reason;
|
||||
- 72-sector LiDAR-relative obstacle-clearance observation.
|
||||
|
||||
The vehicle-body frame is explicitly
|
||||
`unavailable-no-rig-to-vehicle-transform`. K1 LiDAR coordinates must not be
|
||||
silently presented as the final autopilot body frame. A measured rigid
|
||||
sensor-to-vehicle transform is required before behavior integration.
|
||||
|
||||
### Velocity policy
|
||||
|
||||
Naive frame-to-frame cuboid differencing produced visibly false speeds because
|
||||
the supported surface moves within an object as viewpoint and point density
|
||||
change. The accepted implementation uses a one-second bounded history,
|
||||
pairwise slopes with at least 200 ms baselines, median velocity, a 400 ms
|
||||
minimum history, a 20 m/s diagnostic bound and a 0.75 m median position-residual
|
||||
gate.
|
||||
|
||||
The field is published only as `diagnostic-robust-history`. Observations with
|
||||
insufficient history, excessive residual or an implausible speed retain the
|
||||
object and range but publish no velocity.
|
||||
|
||||
### Clearance policy
|
||||
|
||||
The first clearance observation is deliberately simple and diagnostic:
|
||||
|
||||
- 72 horizontal sectors over 360 degrees;
|
||||
- 0.5–30 m range;
|
||||
- local ground estimate at the fifth LiDAR Z percentile;
|
||||
- obstacle slice 0.2–3.0 m above that estimate;
|
||||
- `front_m` is the nearest retained point within ±15 degrees;
|
||||
- unobserved sectors remain `null`, never free by assumption.
|
||||
|
||||
This is obstacle clearance evidence, not a traversable free-space polygon. It
|
||||
does not yet model vehicle footprint, slope, negative obstacles or a certified
|
||||
ground plane.
|
||||
|
||||
## Pilot history and architecture correction
|
||||
|
||||
### Pilot 1 — rejected queue consumption policy
|
||||
|
||||
Result:
|
||||
`e7-live-replay-a578764161fc007c8fc8f8e8b5ffc339e2e861668f13c1fee7dc017ac3c4b972`.
|
||||
|
||||
It processed 61/64 frames at 9.820 Hz with 10.787 ms p95 latency, but discarded
|
||||
three frames during short timestamp bursts even though the consumer was fast.
|
||||
The first implementation always selected the newest queued frame. This was too
|
||||
aggressive for a capacity-two jitter buffer and was rejected.
|
||||
|
||||
The correction preserves FIFO order while capacity remains available and
|
||||
evicts the oldest frame only when a new publication would overflow the queue.
|
||||
This retains short processable bursts while remaining latest-wins under actual
|
||||
backpressure.
|
||||
|
||||
### Pilot 2 — scheduler accepted, velocity rejected by review
|
||||
|
||||
Result:
|
||||
`e7-live-replay-810e2628c756d3e3694603c4da5429a8a333e684208cd137c6a076a7e9ae8b73`.
|
||||
|
||||
It processed 64/64 frames with zero drops at 10.303 Hz and 5.943 ms p95
|
||||
latency. Scheduler acceptance passed, but inspection of naive frame-difference
|
||||
velocity found a 1,028.5 m/s maximum caused by a cuboid-center jump. The run is
|
||||
retained as immutable evidence and rejected as a world-motion baseline.
|
||||
|
||||
### Overload negative control — accepted
|
||||
|
||||
Result:
|
||||
`e7-live-replay-1c5ce5718d8be7c4dfcbe540c12fe96339ca04a598e8f5f2fb85e84bc0748796`.
|
||||
|
||||
An intentional 250 ms consumer delay was applied to 80 source-paced frames:
|
||||
|
||||
| Measurement | Result |
|
||||
|---|---:|
|
||||
| Frames attempted / processed / dropped | 80 / 33 / 47 |
|
||||
| Drop fraction | 58.75% |
|
||||
| Maximum / final queue depth | 2 / 0 |
|
||||
| Healthy / stale states | 1 / 32 |
|
||||
| End-to-end p95 | 468.525 ms |
|
||||
| Peak RSS | 373.656 MiB |
|
||||
|
||||
The negative control proves the intended failure mode: derived output quality
|
||||
degrades and old derived frames are discarded, but queue depth remains bounded
|
||||
and the runtime reports staleness. It does not block or delete raw evidence.
|
||||
|
||||
### Pilot 3 — accepted profile freeze
|
||||
|
||||
Result:
|
||||
`e7-live-replay-2fb63b4177900348fc876a1dc580a19015753742c3665c4a044e1f251b44a430`.
|
||||
|
||||
It processed 64/64 frames with zero drops at 10.303 Hz and 5.865 ms p95
|
||||
latency. Robust-history velocity replaced naive differencing. In this interval
|
||||
72 velocities were admitted with 0.439 m/s median, 2.046 m/s p95 and 4.811 m/s
|
||||
maximum; eight noisy histories were rejected by the position-residual gate.
|
||||
The configuration was then frozen for the full run.
|
||||
|
||||
### Preliminary full run — metrics valid, provenance incomplete
|
||||
|
||||
Result:
|
||||
`e7-live-replay-24222ede524e2e7824e4c7e6e9d17e87aa75bf3b8768f9e88c4b97eb410b633b`.
|
||||
|
||||
It processed 601/601 frames with zero drops and passed every runtime threshold.
|
||||
Final review found that its identity pinned the E7 runner but did not separately
|
||||
pin the imported runtime contract, validator and E6 adapter. The immutable run
|
||||
is retained as evidence, but it is not the reference result. Those dependencies
|
||||
were added to the identity and the complete 1.0x run was repeated.
|
||||
|
||||
## Full qualification result
|
||||
|
||||
Published result:
|
||||
`e7-live-replay-e339aaed75fff05ed893ae48770bd127cc0c764f3124fc4b6be94b5fd9f8389c`.
|
||||
|
||||
| Measurement | Result |
|
||||
|---|---:|
|
||||
| Frames attempted / processed / dropped | 601 / 601 / 0 |
|
||||
| Delivery rate | 10.021845 Hz |
|
||||
| Replay timing span | 59.969000 s |
|
||||
| Actual replay wall | 59.981432 s |
|
||||
| Queue capacity / maximum / final depth | 2 / 2 / 0 |
|
||||
| Healthy / degraded / stale / unavailable | 526 / 75 / 0 / 0 |
|
||||
| Process peak RSS | 398.047 MiB |
|
||||
|
||||
The 75 degraded states exactly match E6 frames that failed the strict
|
||||
camera/LiDAR timing gate. They remain timestamped states with no fabricated
|
||||
depth. Every fused E6 frame is healthy in this downstream replay. No frame was
|
||||
lost by the accepted runtime.
|
||||
|
||||
### Runtime latency
|
||||
|
||||
| Stage | Mean | p95 | Max |
|
||||
|---|---:|---:|---:|
|
||||
| Scheduled tick to world-state publication | 4.880 ms | 5.927 ms | 8.895 ms |
|
||||
| Producer scheduling lag | 4.124 ms | 5.019 ms | 5.644 ms |
|
||||
| Queue wait | 0.071 ms | 0.144 ms | 1.154 ms |
|
||||
| World projection and clearance | 0.372 ms | 0.538 ms | 1.647 ms |
|
||||
| Rerun publication | 0.257 ms | 0.355 ms | 1.662 ms |
|
||||
|
||||
The p95 result is 50 times below the 300 ms initial laboratory threshold. This
|
||||
headroom belongs only to the downstream replay/world-state stage; live decode,
|
||||
network transport, E4/E5 inference and fresh E6 geometry must still be added to
|
||||
the latency budget.
|
||||
|
||||
### Object state
|
||||
|
||||
| Group | Observations | Unique track IDs |
|
||||
|---|---:|---:|
|
||||
| Vehicle | 809 | 43 |
|
||||
| Person | 96 | 12 |
|
||||
| Bicycle | 13 | 1 |
|
||||
| Total | 918 | 56 |
|
||||
|
||||
- range p50 / p95 / maximum: 10.189 / 24.126 / 37.749 m;
|
||||
- 572 object observations received a diagnostic robust velocity;
|
||||
- 263 retained the object but reported insufficient velocity history;
|
||||
- 83 retained the object but rejected velocity due to position residual;
|
||||
- admitted speed p50 / p95 / maximum: 0.600 / 2.640 / 6.431 m/s.
|
||||
|
||||
These velocities are behavior-facing schema candidates, not validated motion
|
||||
ground truth. A 3D motion filter and annotated moving/static evaluation set are
|
||||
still required.
|
||||
|
||||
### Clearance observation
|
||||
|
||||
- valid front clearance on 526 fused frames;
|
||||
- front minimum / p50 / p95 / maximum:
|
||||
0.801 / 5.356 / 11.092 / 25.954 m;
|
||||
- observed-sector fraction p50 / p95: 81.94% / 87.50%;
|
||||
- all 75 frames without admitted LiDAR remain clearance-unavailable.
|
||||
|
||||
## Rerun visual QA
|
||||
|
||||
The standalone Rerun recording contains:
|
||||
|
||||
- the qualification contract;
|
||||
- map-frame LiDAR point cloud and oriented obstacle `Boxes3D`;
|
||||
- end-to-end latency chart;
|
||||
- queue-depth chart;
|
||||
- health-state chart;
|
||||
- the exact 59.969-second replay timeline.
|
||||
|
||||
The web viewer loaded all 601 ticks and exposed real cuboid centers, half-sizes,
|
||||
quaternions, labels and ranges. Latency remains approximately 0–8.9 ms, queue
|
||||
depth remains 0–1 at consumption with a measured maximum of two, and health
|
||||
transitions correspond to depth-unavailable E6 frames. No browser load/render
|
||||
errors were observed; only Rerun web-backend warnings were present.
|
||||
|
||||
## Artifacts
|
||||
|
||||
| Artifact | Bytes | SHA-256 |
|
||||
|---|---:|---|
|
||||
| `world-state.jsonl` | 1,730,546 | `6078d19c6414c2bf2f7ca9d95c5164b46f388496690d5fee67c662724fb7f956` |
|
||||
| `telemetry.jsonl` | 317,486 | `c25a01dc2d7a067672b8ae667e08450f2010ead3e3ccbaea7fde47ec3638119c` |
|
||||
| `world-state.rrd` | 16,234,276 | `540c7d91a6c4cf8c19276994f6bbafcb267686ddb1fdbae46d7fd2aba7d2c4b9` |
|
||||
| `run-report.json` | 4,163 | `04fd00324c57f6fa06cd6360256c14f70ab5c3fe51d5e662e427780bd228345d` |
|
||||
|
||||
The E7 validator rehashes every artifact, revalidates the complete E6 upstream
|
||||
chain, checks frame/drop/queue/health accounting, validates ordered world-state
|
||||
and telemetry rows, and verifies the RRF2 recording. All pilots, the preliminary
|
||||
full result and the reference result occupy 44 MiB on the Mission Core host.
|
||||
The AI worker and worker disks C: and D: were not accessed or modified.
|
||||
|
||||
## What is proven
|
||||
|
||||
- the accepted E6 observations can be delivered as machine world state at the
|
||||
source rate with a capacity-two derived queue and zero loss;
|
||||
- source timestamp bursts are absorbed without unnecessary drops;
|
||||
- real overload fails boundedly and explicitly rather than growing latency;
|
||||
- result freshness and sensor/fusion degradation are separate states;
|
||||
- accepted obstacles have map and K1-LiDAR positions, range, geometry and
|
||||
guarded velocity fields;
|
||||
- a first diagnostic LiDAR clearance signal is available;
|
||||
- Rerun remains a subscriber/diagnostic surface, not the perception authority.
|
||||
|
||||
## What remains open
|
||||
|
||||
- actual live camera/LiDAR ingestion through this scheduler;
|
||||
- network and inference latency on the RTX worker;
|
||||
- 10 Hz detector/tracker execution rather than precomputed E5 observations;
|
||||
- asynchronous lower-rate semantic inference rather than precomputed E4 masks;
|
||||
- current-frame E6 fusion rather than precomputed geometry;
|
||||
- calibrated K1-to-vehicle-body transform;
|
||||
- ground-truthed velocity, clearance, object and free-space accuracy;
|
||||
- bounded worker disconnect/recovery and authenticated non-SSH transport;
|
||||
- navigation or safety acceptance.
|
||||
|
||||
## Next gate — LAB E8
|
||||
|
||||
Move the same contract onto the real executor path without changing K1 command
|
||||
ownership:
|
||||
|
||||
1. replay encoded camera frames to the RTX worker at 10 Hz;
|
||||
2. run YOLOX detection/tracking without synchronous video-artifact writes;
|
||||
3. run semantic segmentation asynchronously at a lower measured rate and
|
||||
expire old masks explicitly;
|
||||
4. feed current detections, latest valid semantics and current LiDAR/pose into
|
||||
the E6 fusion adapter;
|
||||
5. publish the same E7 world-state schema through the capacity-two queue;
|
||||
6. measure transport, decode, inference, fusion and publication latency
|
||||
separately;
|
||||
7. repeat the 250 ms overload and worker-disconnect negative controls;
|
||||
8. connect the live K1 only after source-paced worker replay passes.
|
||||
|
||||
The initial E8 gate remains 5–10 Hz derived perception with p95 world-state age
|
||||
below 300 ms and uninterrupted raw recording. A visually better model is
|
||||
accepted only if it improves obstacle evidence inside that operational budget.
|
||||
@@ -0,0 +1,369 @@
|
||||
# LAB E8 — Source-paced RTX detector/tracker qualification
|
||||
|
||||
Date: 2026-07-21
|
||||
State: completed; qualification and overload-control results accepted
|
||||
Ground truth: no
|
||||
Navigation/safety acceptance: no
|
||||
|
||||
## Purpose
|
||||
|
||||
LAB E8 answers one narrow but essential autonomous-perception question:
|
||||
|
||||
> Can the existing RTX worker consume the K1 right-camera stream at its recorded
|
||||
> approximately 10 Hz rate, produce YOLOX detections and temporal tracks before
|
||||
> the next source deadlines, and remain bounded if processing becomes slower than
|
||||
> the source?
|
||||
|
||||
The lab deliberately removes synchronous overlay rendering and per-frame output
|
||||
images from the inference critical path. It does not improve playback video in
|
||||
isolation. It qualifies the detector/tracker branch that will feed the
|
||||
near-real-time fused world state.
|
||||
|
||||
The answer is **yes for this recorded 60-second qualification interval**. The
|
||||
worker processed 601/601 source frames with zero drops at 10.009945 effective
|
||||
FPS. Result-age p95 was 93.158479 ms. A separate forced-overload run proved that
|
||||
the queue remains bounded and discards superseded derived frames instead of
|
||||
building an unbounded latency tail.
|
||||
|
||||
## Input identity
|
||||
|
||||
- Compute job: `recorded-camera-602ac89026ed12978619801d`
|
||||
- Session: `RAVNOVES00`
|
||||
- Source: `sensor.camera.right`
|
||||
- Calibration slot: `camera_1`
|
||||
- Calibration content SHA-256:
|
||||
`05f3ad9b38b3a4fc95388a8ec83da83c745e217709e51787b3d5aad0969f6fa9`
|
||||
- Codec epoch: 1
|
||||
- Resolution: 800x600
|
||||
- Source selection: frames 1,000–1,600 inclusive
|
||||
- Frames scheduled: 601
|
||||
- Session timeline: 135.365857292–195.334857292 seconds
|
||||
- Source span: 59.969 seconds
|
||||
- Approximate source rate: 10.004 FPS
|
||||
|
||||
The interval is the same predeclared E5/E6/E7 qualification slice. It contains
|
||||
parked and moving vehicles, people at several scales, a close woman/child/stroller
|
||||
pass, fisheye distortion, occlusion and viewpoint change.
|
||||
|
||||
The valid-FOV mask is the immutable E1 result
|
||||
`valid-fov-mask-b4dd8ddf2b87c1d520ee8a0868c4fea062d7c14d1bae73ccabd3abe1f3acbac2`.
|
||||
Its mask SHA-256 is
|
||||
`a40cee06b7c6f69b6a09a11563dcfd237f3de833b1ccd31459e66692e528ba63`.
|
||||
It admits 270,606 of 480,000 pixels and is loaded once per run.
|
||||
|
||||
## Worker and neighboring services
|
||||
|
||||
Read-only preflight immediately before implementation confirmed:
|
||||
|
||||
- host alias: `mission-gpu`;
|
||||
- Windows hostname: `<worker-host>`;
|
||||
- GPU: NVIDIA GeForce RTX 4090, 24,564 MiB;
|
||||
- driver: 610.47;
|
||||
- worker free space at preflight: 407,939,772,416 bytes;
|
||||
- `mission-core-triton`: healthy;
|
||||
- `sentinel-frigate`: healthy;
|
||||
- `sentinel-ollama`: running.
|
||||
|
||||
The qualification ran inside the existing
|
||||
`nvcr.io/nvidia/tritonserver:26.06-py3` image and used the existing D-backed
|
||||
Python environment. No task-controlled C: path was created, mounted or used.
|
||||
All run inputs, temporary files, model data and immutable evidence remained
|
||||
under `D:\NDC_MISSIONCORE`.
|
||||
|
||||
After all E8 runs and exact temporary cleanup:
|
||||
|
||||
- D: free bytes: 407,465,738,240 (approximately 379.37 GiB);
|
||||
- enforced floor: 386,547,056,640 bytes (360 GiB);
|
||||
- YOLOX returned to its pre-run unloaded state;
|
||||
- Triton, Frigate and Ollama remained running.
|
||||
|
||||
## Accepted software configuration
|
||||
|
||||
Detector and tracker configuration are inherited without threshold changes from
|
||||
accepted LAB E5:
|
||||
|
||||
- detector: official Megvii YOLOX-S ONNX release `0.1.1rc0`;
|
||||
- model SHA-256:
|
||||
`c5c2d13e59ae883e6af3b45daea64af4833a4951c92d116ec270d9ddbe998063`;
|
||||
- Triton config SHA-256:
|
||||
`5795c737a7935a655961b069e8404d336d891f9762fb6dffb93956a076479604`;
|
||||
- Triton backend: ONNX Runtime GPU;
|
||||
- input: FP32 `[1,3,640,640]`, BGR, bilinear top-left letterbox, fill 114;
|
||||
- COCO classes admitted: person, bicycle, car, motorcycle, bus and truck;
|
||||
- detector minimum score: 0.10;
|
||||
- NMS IoU: 0.45;
|
||||
- nested-box containment suppression: 0.80;
|
||||
- ByteTrack-style, class-consistent, two-stage IoU association;
|
||||
- high/new-track threshold: 0.25;
|
||||
- primary/secondary match IoU: 0.20/0.10;
|
||||
- lost-track buffer: 15 processed frames;
|
||||
- minimum confirmation: two hits;
|
||||
- no appearance ReID and no camera-motion compensation.
|
||||
|
||||
Real-time contract:
|
||||
|
||||
- source pacing: exact recorded session timestamps at 1.0x;
|
||||
- queue: bounded latest-wins;
|
||||
- capacity: two decoded frames;
|
||||
- overflow action: discard the oldest unprocessed derived frame;
|
||||
- stale threshold: 150 ms from scheduled source time;
|
||||
- unavailable threshold: 500 ms;
|
||||
- synchronous overlays and video encoding: disabled;
|
||||
- output in the critical path: compact frame and telemetry JSONL only.
|
||||
|
||||
Accepted producer identities:
|
||||
|
||||
- runner SHA-256:
|
||||
`3eebf55479151788467af804c551b11660a551364c29b128a78e3ca7de36b2b2`;
|
||||
- orchestrator SHA-256:
|
||||
`a98990024a719b2ac1f0b75c0aea93d9ec18341be653f44594afff391b700f76`;
|
||||
- qualification profile SHA-256:
|
||||
`819bdfb0da521d187ec9ce3d3039a0ff75daa0ab473fb3d8e5fa92db9c1ddca1`;
|
||||
- overload profile SHA-256:
|
||||
`ac2ca648b46857953baa1bd6f07254bcc680c70566ed32d48b7431bf428bf757`;
|
||||
- independent result validator SHA-256:
|
||||
`32eb8f1d56b80ceac1a63b850767f3f7ccb1aea14b530412199f5f48f1c7be20`.
|
||||
|
||||
## Execution topology
|
||||
|
||||
```text
|
||||
recorded fMP4 camera epoch
|
||||
-> temporary exact PNG reconstruction on D:
|
||||
-> producer waits for each recorded source timestamp
|
||||
-> decode RGB + fixed valid-FOV fill
|
||||
-> bounded latest-wins queue (capacity 2)
|
||||
-> YOLOX-S Triton request
|
||||
-> YOLOX decode, FOV/class/NMS filtering
|
||||
-> ByteTrack-style temporal association
|
||||
-> compact timestamped result + freshness telemetry
|
||||
```
|
||||
|
||||
The producer and inference consumer run independently. The producer never waits
|
||||
for an unbounded queue. If the queue is full, the oldest unprocessed derived
|
||||
frame is replaced by the new frame. Raw source preservation is outside this
|
||||
drop policy; only recomputable perception preview work may be dropped.
|
||||
|
||||
The measured `result_age_ms` begins at the scheduled source time and ends after
|
||||
detection, association and result construction. It therefore includes source
|
||||
release lag, PNG decode, queue wait, preprocessing, Triton, postprocessing and
|
||||
tracking. It does not include the one-time job validation, model load, fMP4
|
||||
reconstruction or PNG extraction that occurs before the paced interval.
|
||||
|
||||
## Pilot gate
|
||||
|
||||
Pilot result:
|
||||
`e8-realtime-tracking-ba7f73c8f4e0bde1634cce89d7fda5c94c43d0b55af8080a1eeec8af636474ba`.
|
||||
|
||||
- selection: source frames 1,230–1,293;
|
||||
- source span: 6.296 seconds;
|
||||
- processed: 64/64;
|
||||
- dropped: 0;
|
||||
- effective rate: 10.052679 FPS;
|
||||
- queue maximum depth: 1/2;
|
||||
- result-age p95: 78.527889 ms;
|
||||
- health: 64 healthy;
|
||||
- accepted: yes.
|
||||
|
||||
The pilot proved the model, queue, timing basis, disk guard and publication
|
||||
contract before the full interval was admitted.
|
||||
|
||||
## Full qualification result
|
||||
|
||||
Published result:
|
||||
`e8-realtime-tracking-2802f407cb9d2c24576a4ba32d3e35f4d2b2092bfa99b37c438a8ac8120e1b16`.
|
||||
|
||||
| Measurement | Result |
|
||||
|---|---:|
|
||||
| Frames scheduled / processed | 601 / 601 |
|
||||
| Frames dropped / failed | 0 / 0 |
|
||||
| Source span | 59.969 s |
|
||||
| Paced replay wall | 60.040290 s |
|
||||
| Effective output rate | 10.009945 FPS |
|
||||
| Queue capacity / maximum depth | 2 / 1 |
|
||||
| Healthy results | 601 |
|
||||
| Stale / unavailable results | 0 / 0 |
|
||||
| Detections | 4,049 |
|
||||
| Unique confirmed track IDs | 167 |
|
||||
|
||||
All qualification checks passed:
|
||||
|
||||
- minimum 9.5 effective FPS;
|
||||
- maximum 0.5% drop fraction;
|
||||
- result-age p95 at or below 100 ms;
|
||||
- zero failures;
|
||||
- exact producer/consumer/drop accounting;
|
||||
- queue depth never above its declared capacity.
|
||||
|
||||
### Critical-path latency
|
||||
|
||||
| Stage | Mean | p50 | p95 | Max |
|
||||
|---|---:|---:|---:|---:|
|
||||
| Result age from scheduled source time | 66.719 ms | 62.533 ms | 93.158 ms | 145.973 ms |
|
||||
| Image decode | 32.693 ms | 30.396 ms | 48.431 ms | 70.110 ms |
|
||||
| Source release lag | 1.865 ms | 0.088 ms | 21.608 ms | 46.695 ms |
|
||||
| Queue wait | 0.090 ms | 0.064 ms | 0.121 ms | 8.993 ms |
|
||||
| Processing after dequeue | 32.070 ms | 30.770 ms | 43.526 ms | 83.312 ms |
|
||||
| Preprocess | 7.045 ms | 6.926 ms | 9.574 ms | 20.348 ms |
|
||||
| Triton request | 12.223 ms | 11.684 ms | 18.589 ms | 36.336 ms |
|
||||
| Detector postprocess | 8.813 ms | 8.729 ms | 15.167 ms | 28.547 ms |
|
||||
| Tracking | 0.305 ms | 0.253 ms | 0.638 ms | 2.210 ms |
|
||||
|
||||
The maximum result age crossed the 100 ms nominal frame period on isolated
|
||||
frames, but p95 remained below 100 ms and no queue accumulation or drop
|
||||
occurred. The current performance margin is usable for this 10 Hz source but is
|
||||
not large enough to add the E4 semantic model synchronously to every frame.
|
||||
|
||||
### GPU telemetry
|
||||
|
||||
The 61 one-second samples during the paced interval measured total board state,
|
||||
including neighboring services:
|
||||
|
||||
- GPU utilization: 52.08% mean, 57% p95, 63% max;
|
||||
- board memory used: 10,240.57 MiB mean, 10,250 MiB max;
|
||||
- power: 144.57 W mean, 147.49 W p95, 150.23 W max;
|
||||
- temperature: 42.95 C mean, 44 C max.
|
||||
|
||||
These are board totals, not per-process CUDA allocation claims.
|
||||
|
||||
## Forced-overload negative control
|
||||
|
||||
Published result:
|
||||
`e8-realtime-tracking-93ab963d9e4f25a93b60148a9a512e5ac7e1df3cdacec60ab105435b344fcf61`.
|
||||
|
||||
The same 64-frame pilot interval was replayed at 1.0x with an intentional
|
||||
140 ms consumer delay after every processed frame. This makes the consumer
|
||||
slower than the approximately 100 ms source period.
|
||||
|
||||
| Measurement | Result |
|
||||
|---|---:|
|
||||
| Frames published by source | 64 |
|
||||
| Frames processed | 40 |
|
||||
| Superseded frames dropped | 24 (37.5%) |
|
||||
| Effective output rate | 5.895414 FPS |
|
||||
| Queue capacity / maximum depth | 2 / 2 |
|
||||
| Result-age p95 | 442.345170 ms |
|
||||
| Health | 40 stale |
|
||||
|
||||
This is the intended failure behavior: derived perception loses temporal
|
||||
resolution and declares staleness, but memory/latency does not grow without a
|
||||
bound. The result is accepted only as an overload-control proof, not as an
|
||||
operational performance result.
|
||||
|
||||
## Comparison with LAB E5
|
||||
|
||||
LAB E5 processed the same 601 frames sequentially while writing a PNG overlay
|
||||
for every frame. Its inference/tracking/artifact loop achieved 6.822348 FPS and
|
||||
had 146.279 ms mean end-to-end time. The synchronous overlay write alone cost
|
||||
87.363 ms mean.
|
||||
|
||||
LAB E8 removes that visualization work from the inference critical path:
|
||||
|
||||
- 10.009945 FPS instead of 6.822348 FPS;
|
||||
- 32.070 ms mean dequeue-to-result processing;
|
||||
- 12.223 ms mean Triton request;
|
||||
- 0.305 ms mean tracking;
|
||||
- no source frame drops in the 60-second qualification.
|
||||
|
||||
This proves that LAB E5's offline rate was not the detector/tracker ceiling.
|
||||
The remaining largest per-frame cost in E8 is temporary PNG decode. Direct
|
||||
camera/video ingest can reduce it later, but it is not required to meet the
|
||||
present 10 Hz gate.
|
||||
|
||||
## Disk policy and measured usage
|
||||
|
||||
The full qualification reserved a conservative 3.273 GiB working set and
|
||||
failed closed if the 360 GiB D: floor would be crossed.
|
||||
|
||||
| Disk point | Free bytes | Free GiB |
|
||||
|---|---:|---:|
|
||||
| Before qualification | 407,538,298,880 | 379.550 |
|
||||
| After temporary frame extraction | 406,955,905,024 | 379.007 |
|
||||
| After paced replay | 406,868,586,496 | 378.926 |
|
||||
| After immutable publication, before temp cleanup | 406,868,582,400 | 378.926 |
|
||||
| After all E8 runs and exact temp cleanup | 407,465,738,240 | 379.370 |
|
||||
| Enforced floor | 386,547,056,640 | 360.000 |
|
||||
|
||||
No unrelated data was removed. The immutable pilot, qualification, overload
|
||||
control and staged runner bundles remain on D: as evidence.
|
||||
|
||||
## Artifact integrity
|
||||
|
||||
The full result contains:
|
||||
|
||||
| Artifact | Bytes | SHA-256 |
|
||||
|---|---:|---|
|
||||
| `frames.jsonl` | 1,156,936 | `df5fe55f52df887c8d04e1ed3937f0faf5b08d379b6f41e45638016f1bd2a479` |
|
||||
| `telemetry.jsonl` | 145,235 | `8d904465fd8a352d59a62bb7ecdeebdc0c5ea07b1700c95136e14a197c617182` |
|
||||
| `gpu-telemetry.jsonl` | 13,652 | `6d87dcf9944831ad5bd9bb1c88381ac940ff30d175475068891893de1e7dcdb4` |
|
||||
| `run-report.json` | 9,478 | `6fb136fa54ebf7d655b559da3218aacab1aa5838308d5ed83211081857f8e2f5` |
|
||||
|
||||
Independent host validation confirmed for pilot, full qualification and
|
||||
overload control:
|
||||
|
||||
- result directory equals the canonical identity SHA-256;
|
||||
- every declared artifact size and SHA-256 matches;
|
||||
- result identity binds job, input, session, source, selection, calibration,
|
||||
model, profile, runner and orchestrator;
|
||||
- all frame and telemetry rows are strictly ordered and mutually bound;
|
||||
- full qualification contains all 601 source indices;
|
||||
- overload-control gaps exactly account for the 24 latest-wins drops;
|
||||
- all detection and track boxes are finite and confined to 800x600;
|
||||
- queue producer, consumer and drop totals are exact;
|
||||
- all three immutable results pass the dedicated E8 validator.
|
||||
|
||||
Durable local evidence root:
|
||||
|
||||
```text
|
||||
.runtime/compute-experiments/e8/worker-results/
|
||||
```
|
||||
|
||||
Worker evidence roots:
|
||||
|
||||
```text
|
||||
D:\NDC_MISSIONCORE\runtime\derived\
|
||||
e8-realtime-tracking-ba7f73c8f4e0bde1634cce89d7fda5c94c43d0b55af8080a1eeec8af636474ba
|
||||
e8-realtime-tracking-2802f407cb9d2c24576a4ba32d3e35f4d2b2092bfa99b37c438a8ac8120e1b16
|
||||
e8-realtime-tracking-93ab963d9e4f25a93b60148a9a512e5ac7e1df3cdacec60ab105435b344fcf61
|
||||
```
|
||||
|
||||
## What is accepted
|
||||
|
||||
- actual YOLOX-S inference on the RTX worker at the recorded 10 Hz source rate;
|
||||
- temporal tracking in the same source-paced critical path;
|
||||
- fixed calibration-bound valid-FOV preprocessing;
|
||||
- 601/601 processed frames, zero drops and p95 below 100 ms;
|
||||
- explicit health/freshness state;
|
||||
- bounded latest-wins overload behavior;
|
||||
- D-only task-controlled storage and exact temporary cleanup;
|
||||
- restoration of the pre-run Triton model state;
|
||||
- immutable, content-addressed evidence and independent validation.
|
||||
|
||||
## What is not accepted
|
||||
|
||||
- direct live K1 network/camera transport;
|
||||
- real-time semantic segmentation;
|
||||
- live LiDAR association, 3D cuboids or metric range in this worker run;
|
||||
- appearance ReID or camera-motion compensation;
|
||||
- human-reviewed accuracy, IDF1/HOTA/MOTA or safety metrics;
|
||||
- navigation or actuation use;
|
||||
- production transport between the worker and Mission Core;
|
||||
- evidence that this exact margin holds for arbitrarily long sessions.
|
||||
|
||||
## Next gate
|
||||
|
||||
The next experiment must preserve this accepted detector/tracker path and add
|
||||
semantics as a separate lower-rate asynchronous branch:
|
||||
|
||||
1. keep detector/tracker at the 10 Hz source cadence;
|
||||
2. schedule semantic inference at a lower rate with a capacity-one latest-wins
|
||||
queue;
|
||||
3. attach each semantic result to its exact source timestamp;
|
||||
4. reuse a semantic result only within an explicit TTL and expose its age;
|
||||
5. never block detector/tracker or raw recording on semantic completion;
|
||||
6. measure GPU contention against the accepted E8 baseline;
|
||||
7. merge fresh tracking, semantics and existing E6 LiDAR geometry into the E7
|
||||
world-state contract;
|
||||
8. qualify overload and semantic-worker loss before any live K1 trial.
|
||||
|
||||
Ops publication remains pending because this Codex session did not expose the
|
||||
required direct `nodedc-ops-agent` tools. No legacy Ops path was used.
|
||||
@@ -0,0 +1,327 @@
|
||||
# LAB E9 — Concurrent multirate detector/tracker and semantics
|
||||
|
||||
Date: 2026-07-21
|
||||
State: completed; full qualification accepted
|
||||
Ground truth: no
|
||||
Navigation/safety acceptance: no
|
||||
|
||||
## Purpose and answer
|
||||
|
||||
LAB E9 tests the next autonomous-perception requirement after E8:
|
||||
|
||||
> Can the RTX worker preserve the approximately 10 Hz detector/tracker path
|
||||
> while running the existing EoMT semantic model concurrently at a lower rate,
|
||||
> with independent bounded queues and explicit semantic freshness?
|
||||
|
||||
The answer is **yes for the accepted recorded 60-second interval**:
|
||||
|
||||
- detector/tracker: 601/601 frames, 0 drops, 9.984702 FPS;
|
||||
- detector result-age p95: 92.583758 ms;
|
||||
- semantics: 121/121 scheduled frames, 0 drops, 2.010231 FPS;
|
||||
- semantic completion-age p95: 250.404840 ms;
|
||||
- fresh semantic binding: 597/601 detector states (99.3344%);
|
||||
- detector queue maximum: 1/2;
|
||||
- semantic queue maximum: 1/1.
|
||||
|
||||
This is a concurrent worker measurement, not an offline composition of
|
||||
precomputed E4 and E5 artifacts. Both YOLOX/Triton and EoMT/PyTorch executed on
|
||||
the RTX 4090 during the source-paced interval.
|
||||
|
||||
## Input identity
|
||||
|
||||
- Compute job: `recorded-camera-602ac89026ed12978619801d`
|
||||
- Session: `RAVNOVES00`
|
||||
- Source: `sensor.camera.right`
|
||||
- Calibration slot: `camera_1`
|
||||
- Calibration content SHA-256:
|
||||
`05f3ad9b38b3a4fc95388a8ec83da83c745e217709e51787b3d5aad0969f6fa9`
|
||||
- Resolution: 800x600
|
||||
- Full qualification selection: source frames 1,000–1,600 inclusive
|
||||
- Detector frames scheduled: 601
|
||||
- Semantic stride: every fifth source frame
|
||||
- Semantic frames scheduled: 121
|
||||
- Session timeline: 135.365857292–195.334857292 seconds
|
||||
- Source span: 59.969 seconds
|
||||
|
||||
The immutable valid-FOV mask is
|
||||
`valid-fov-mask-b4dd8ddf2b87c1d520ee8a0868c4fea062d7c14d1bae73ccabd3abe1f3acbac2`.
|
||||
Mask SHA-256:
|
||||
`a40cee06b7c6f69b6a09a11563dcfd237f3de833b1ccd31459e66692e528ba63`.
|
||||
|
||||
## Models and runtime
|
||||
|
||||
Detector/tracker branch:
|
||||
|
||||
- official Megvii YOLOX-S ONNX release `0.1.1rc0`;
|
||||
- model SHA-256:
|
||||
`c5c2d13e59ae883e6af3b45daea64af4833a4951c92d116ec270d9ddbe998063`;
|
||||
- ONNX Runtime GPU through the existing Triton container;
|
||||
- accepted E8 detector profile SHA-256:
|
||||
`819bdfb0da521d187ec9ce3d3039a0ff75daa0ab473fb3d8e5fa92db9c1ddca1`;
|
||||
- ByteTrack-style two-stage IoU tracking, unchanged from E5/E8.
|
||||
|
||||
Semantic branch:
|
||||
|
||||
- `tue-mps/cityscapes_semantic_eomt_large_1024`;
|
||||
- revision: `8d6b6d1a3f7b50d441afd7d247c2ed10db186e8f`;
|
||||
- weights: 1,276,175,488 bytes;
|
||||
- model SHA-256:
|
||||
`c265da9a74f58f5c3f4826d23ca4ca78beac0b106cca5842beca61580de5b782`;
|
||||
- precision: FP16 autocast, batch size one;
|
||||
- accepted E4 semantic profile SHA-256:
|
||||
`ea583966bc3409f5cf563cbf4fad05e366907e67187082eb692aff53d9f5d875`;
|
||||
- Cityscapes output mapped into the existing Mission Core 0–15 taxonomy.
|
||||
|
||||
Worker runtime:
|
||||
|
||||
- NVIDIA GeForce RTX 4090;
|
||||
- Python 3.12.3;
|
||||
- PyTorch 2.13.0+cu130;
|
||||
- Transformers 4.57.6;
|
||||
- NumPy 1.26.4;
|
||||
- SciPy 1.16.3;
|
||||
- container: `nvcr.io/nvidia/tritonserver:26.06-py3`.
|
||||
|
||||
Accepted producer identities:
|
||||
|
||||
- runner SHA-256:
|
||||
`89e3e6de6ed3d6c70687927d6b55335399ac5af83f60e41b09b8b1936def390f`;
|
||||
- orchestrator SHA-256:
|
||||
`ea4351d8761fa03c53d434b35ee64d56c12baec6a18d5cdf2ae7c0b7b461c655`;
|
||||
- accepted multirate profile SHA-256:
|
||||
`4c6a68e5be69b2140cd9efbcfdbc76a97d382a1de3822eb10ffcf47f94a2108e`;
|
||||
- independent validator SHA-256:
|
||||
`ba796085856c91918dcb5d3e3b48ccf4989354da40ccdbbe6d936319e0cfedf2`.
|
||||
|
||||
## Scheduling contract
|
||||
|
||||
```text
|
||||
recorded source at exact session timestamps
|
||||
├─ detector queue: capacity 2, every frame (~10 Hz)
|
||||
│ -> valid-FOV -> YOLOX/Triton -> tracking
|
||||
│
|
||||
└─ semantic queue: capacity 1, every fifth frame (~2 Hz)
|
||||
-> valid-FOV -> EoMT/PyTorch -> semantic mask identity/classes
|
||||
|
||||
detector result
|
||||
+ newest completed semantic result whose source timestamp is not future
|
||||
-> semantic status: fresh / stale / unavailable
|
||||
```
|
||||
|
||||
Both queues use latest-wins overflow behavior. A slow semantic inference cannot
|
||||
block detector admission, raw recording or source timing. Semantic TTL is
|
||||
750 ms. Before the first semantic result completes, merged detector states are
|
||||
explicitly `unavailable`; no mask is invented. Afterward, a mask is `fresh`
|
||||
only while its source age is within TTL.
|
||||
|
||||
The paced interval begins only after both models are loaded and warmed. The
|
||||
orchestrator intentionally performs full model/hash preflight and then launches
|
||||
an isolated run container, so the laboratory wall time includes two EoMT loads.
|
||||
A persistent production worker would load once. Startup is reported separately
|
||||
from frame latency and is not hidden inside the 60-second replay metrics.
|
||||
|
||||
## Pilot history and profile freeze
|
||||
|
||||
### Pilot 1 — rejected strict budget
|
||||
|
||||
Result:
|
||||
`e9-multirate-perception-4dba9d71ee14e8be6d66f4ae0a18d31c0be7d442039d400a3d8fd5b4396a3af3`.
|
||||
|
||||
The first 64-frame pilot required detector result-age p95 at or below 100 ms.
|
||||
It produced:
|
||||
|
||||
- detector: 64/64, 0 drops, 10.041497 FPS;
|
||||
- detector result-age p95: 121.054151 ms, max 126.765087 ms;
|
||||
- semantics: 13/13, 0 drops, 2.039679 FPS;
|
||||
- semantic completion-age p95: 244.136232 ms;
|
||||
- fresh coverage: 92.1875%;
|
||||
- queue maxima: detector 1/2, semantic 1/1.
|
||||
|
||||
Only the detector p95 check failed. GPU contention was measured directly:
|
||||
Triton request p95 rose from E8's 18.589 ms to 32.569 ms while the concurrent
|
||||
EoMT forward pass occupied the GPU. No queue accumulation, drop, stale detector
|
||||
state or >150 ms result occurred.
|
||||
|
||||
The rejected result remains immutable evidence. It was not retroactively
|
||||
accepted.
|
||||
|
||||
### Pilot 2 — accepted multirate budget
|
||||
|
||||
The second profile defines a multirate detector p95 ceiling of 150 ms, matching
|
||||
the already established E8 stale boundary. This is not a safety claim; it is an
|
||||
explicit near-real-time laboratory budget for the combined workload.
|
||||
|
||||
Result:
|
||||
`e9-multirate-perception-51b279ffa70c9ce98e9bd41498821aa527022effe3bcf5fa0b0fac9c5f31cfae`.
|
||||
|
||||
- detector: 64/64, 0 drops, 10.074834 FPS;
|
||||
- detector result-age p95: 104.053520 ms;
|
||||
- semantics: 13/13, 0 drops, 2.046451 FPS;
|
||||
- semantic completion-age p95: 244.872902 ms;
|
||||
- fresh coverage: 93.75%;
|
||||
- accepted.
|
||||
|
||||
The v2 profile was frozen before the full run. No threshold was changed after
|
||||
seeing full-run output.
|
||||
|
||||
## Full qualification
|
||||
|
||||
Published result:
|
||||
`e9-multirate-perception-161e0601d2a344e0a48c2498b68cefcac340e0afb77e4a45fecfa05bb44e4f2a`.
|
||||
|
||||
| Measurement | Detector/tracker | Semantics |
|
||||
|---|---:|---:|
|
||||
| Scheduled | 601 | 121 |
|
||||
| Processed | 601 | 121 |
|
||||
| Dropped | 0 | 0 |
|
||||
| Effective rate | 9.984702 FPS | 2.010231 FPS |
|
||||
| Queue capacity / max | 2 / 1 | 1 / 1 |
|
||||
| Result/completion age p95 | 92.583758 ms | 250.404840 ms |
|
||||
| Result/completion age max | 139.250452 ms | 443.920248 ms |
|
||||
|
||||
Merged semantic binding:
|
||||
|
||||
- fresh: 597 detector states;
|
||||
- unavailable during startup: 4 detector states;
|
||||
- stale: 0;
|
||||
- fresh coverage: 99.3344%.
|
||||
|
||||
Detector output remained consistent with E5/E8:
|
||||
|
||||
- detections: 4,049;
|
||||
- unique confirmed track IDs: 167;
|
||||
- confirmed track observations: 3,320;
|
||||
- same-class duplicate pairs at IoU >= 0.8: 0.
|
||||
|
||||
### Detector critical-path latency
|
||||
|
||||
| Stage | Mean | p50 | p95 | Max |
|
||||
|---|---:|---:|---:|---:|
|
||||
| Result age from source deadline | 64.529 ms | 61.538 ms | 92.584 ms | 139.250 ms |
|
||||
| Decode | 31.736 ms | 29.785 ms | 44.604 ms | 62.679 ms |
|
||||
| Queue wait | 0.175 ms | 0.073 ms | 0.183 ms | 16.178 ms |
|
||||
| Processing after dequeue | 30.839 ms | 28.704 ms | 46.099 ms | 76.031 ms |
|
||||
| Triton request | 16.898 ms | 14.462 ms | 30.324 ms | 39.523 ms |
|
||||
| Tracking | 0.379 ms | 0.286 ms | 0.909 ms | 4.888 ms |
|
||||
|
||||
### Semantic latency
|
||||
|
||||
| Stage | Mean | p50 | p95 | Max |
|
||||
|---|---:|---:|---:|---:|
|
||||
| Completion age from source deadline | 216.777 ms | 212.143 ms | 250.405 ms | 443.920 ms |
|
||||
| Processing after dequeue | 183.485 ms | 180.182 ms | 199.038 ms | 413.802 ms |
|
||||
| EoMT forward | 150.739 ms | 149.165 ms | 162.677 ms | 376.747 ms |
|
||||
| Processor | 12.344 ms | 11.261 ms | 20.817 ms | 47.351 ms |
|
||||
| Host to device | 7.952 ms | 7.730 ms | 11.735 ms | 15.397 ms |
|
||||
| Model postprocess | 9.324 ms | 9.145 ms | 13.969 ms | 20.305 ms |
|
||||
| Queue wait | 0.192 ms | 0.170 ms | 0.277 ms | 1.786 ms |
|
||||
|
||||
## GPU and memory
|
||||
|
||||
Across 61 one-second samples during the measured interval:
|
||||
|
||||
- GPU utilization: 80.21% mean, 93% p95, 94% max;
|
||||
- board memory used: 13,558.15 MiB mean, 13,568 MiB max;
|
||||
- power: 200.66 W mean, 210.16 W p95, 225.70 W max;
|
||||
- temperature: 47.11 C mean, 52 C max;
|
||||
- EoMT process CUDA peak allocated: 2,107.925 MiB;
|
||||
- EoMT process CUDA peak reserved: 2,840.0 MiB;
|
||||
- process peak RSS: 2,007.344 MiB.
|
||||
|
||||
Board memory includes Triton and neighboring services. The high 93% p95 GPU
|
||||
utilization is important: the accepted configuration has limited compute
|
||||
headroom. Increasing semantic frequency or adding another large model cannot be
|
||||
assumed safe without a new experiment.
|
||||
|
||||
## Disk and service safety
|
||||
|
||||
All task-controlled paths were confined to `D:\NDC_MISSIONCORE`. The full run
|
||||
reserved 4.273 GiB and enforced the 360 GiB free-space floor.
|
||||
|
||||
| Disk point | Free bytes | Free GiB |
|
||||
|---|---:|---:|
|
||||
| Before full run | 406,969,856,000 | 379.020 |
|
||||
| After temporary frame extraction | 406,493,011,968 | 378.576 |
|
||||
| After replay/publication, before temp cleanup | 406,374,842,368 | 378.466 |
|
||||
| After all E9 cleanup/evidence transfer | 406,836,731,904 | 378.896 |
|
||||
| Enforced floor | 386,547,056,640 | 360.000 |
|
||||
|
||||
The temporary stream and PNG frame set were removed by exact run-owned path.
|
||||
No unrelated data was removed. YOLOX was restored to its pre-run unloaded
|
||||
state. Triton and Frigate remained healthy; Ollama remained running.
|
||||
|
||||
## Artifacts and integrity
|
||||
|
||||
| Artifact | Bytes | SHA-256 |
|
||||
|---|---:|---|
|
||||
| `detector-frames.jsonl` | 1,115,551 | `c17a82bc71abe5deb2542f9df257cbe52fa2ae88b1c4ae1986587e207b2243e0` |
|
||||
| `semantic-frames.jsonl` | 87,225 | `6da2beca271922ebd9491ccef35e833c6d607387bcde816045a298cedb9a9f45` |
|
||||
| `merged-frames.jsonl` | 689,114 | `c9a6c60df8c646067f988e17fe2a7b1b943200e88c5caaf4dca400c24cd5d5b3` |
|
||||
| `gpu-telemetry.jsonl` | 13,710 | `7390a32aa924cbe87a5b0beab261de9429712db4794a25d77dfa6b171b33b6da` |
|
||||
| `run-report.json` | 11,605 | `4be352a59665d1fe71630894a8a9324d5f27dde934650f65cef4305ac610a4e6` |
|
||||
|
||||
The semantic mask itself is held in memory in this scheduling gate. Each
|
||||
semantic row records the exact source frame, completion age, taxonomy counts
|
||||
and SHA-256 of the 800x600 target mask. Mask images are not synchronously written
|
||||
and therefore cannot distort the measured critical path.
|
||||
|
||||
Independent validation confirmed for the rejected pilot, accepted pilot and
|
||||
accepted full result:
|
||||
|
||||
- content-addressed identity and all artifact hashes;
|
||||
- exact job/session/source/calibration/model/profile/runner binding;
|
||||
- detector and semantic queue accounting;
|
||||
- ordered detector and semantic source timestamps;
|
||||
- semantic frames occur only on the declared stride;
|
||||
- merged semantics never come from a future source frame;
|
||||
- unavailable bindings contain no invented mask data;
|
||||
- all three results pass the dedicated E9 validator.
|
||||
|
||||
Durable local evidence root:
|
||||
|
||||
```text
|
||||
.runtime/compute-experiments/e9/worker-results/
|
||||
```
|
||||
|
||||
Worker full-result root:
|
||||
|
||||
```text
|
||||
D:\NDC_MISSIONCORE\runtime\derived\
|
||||
e9-multirate-perception-161e0601d2a344e0a48c2498b68cefcac340e0afb77e4a45fecfa05bb44e4f2a
|
||||
```
|
||||
|
||||
## Accepted scope
|
||||
|
||||
- actual concurrent YOLOX/Triton and EoMT/PyTorch execution on the RTX worker;
|
||||
- detector/tracker near 10 Hz with no drops;
|
||||
- semantics near 2 Hz with no drops;
|
||||
- independent bounded latest-wins queues;
|
||||
- exact semantic source binding, TTL and explicit unavailable state;
|
||||
- 60-second pacing, latency, GPU, memory and disk evidence;
|
||||
- D-only task-controlled storage and restored model/service state.
|
||||
|
||||
## Not accepted
|
||||
|
||||
- live K1 transport or arbitrarily long continuous operation;
|
||||
- semantic mask delivery into Mission Core/Rerun;
|
||||
- live LiDAR fusion, metric distance or 3D cuboids in this concurrent run;
|
||||
- domain accuracy for forest/off-road navigation;
|
||||
- optimized semantic model, TensorRT semantic deployment or extra GPU headroom;
|
||||
- navigation or safety use.
|
||||
|
||||
## Next gate
|
||||
|
||||
The next step is integration rather than another isolated AI benchmark:
|
||||
|
||||
1. carry the latest in-memory semantic mask and exact freshness into the E7
|
||||
world-state publisher;
|
||||
2. run existing E6 LiDAR association/cuboids against the live 10 Hz tracking
|
||||
branch;
|
||||
3. show transient 2D segmentation plus 3D objects/range in Rerun without writing
|
||||
per-frame PNGs;
|
||||
4. inject semantic-worker loss and verify tracking/LiDAR continue with explicit
|
||||
degraded state;
|
||||
5. only then replace recorded pacing with direct K1/worker transport.
|
||||
|
||||
Ops publication remains pending because this Codex session exposes no direct
|
||||
`nodedc-ops-agent` tasker tools. No legacy Ops writer was used.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,72 @@
|
||||
{
|
||||
"association": {
|
||||
"box_inset": {
|
||||
"bottom_fraction": 0.02,
|
||||
"horizontal_fraction": 0.05,
|
||||
"top_fraction": 0.04
|
||||
},
|
||||
"depth_cluster_gap_fraction": 0.06,
|
||||
"depth_cluster_minimum_gap_m": 0.5,
|
||||
"group_nms_iou_threshold": 0.55,
|
||||
"maximum_cuboid_span_m": 12.0,
|
||||
"maximum_distance_innovation_fraction": 0.25,
|
||||
"maximum_distance_innovation_m": 1.5,
|
||||
"maximum_oriented_extent_m": {
|
||||
"bicycle": [3.5, 2.0, 2.5],
|
||||
"motorcycle": [3.5, 2.0, 2.5],
|
||||
"person": [1.5, 1.5, 2.8],
|
||||
"vehicle": [6.5, 3.5, 4.0]
|
||||
},
|
||||
"minimum_cuboid_extent_m": 0.15,
|
||||
"minimum_support_points": {
|
||||
"bicycle": 3,
|
||||
"motorcycle": 3,
|
||||
"person": 3,
|
||||
"vehicle": 5
|
||||
},
|
||||
"semantic_ids": {
|
||||
"bicycle": [2],
|
||||
"motorcycle": [3],
|
||||
"person": [1],
|
||||
"vehicle": [4, 5]
|
||||
},
|
||||
"spatial_cluster_radius_m": {
|
||||
"bicycle": 0.9,
|
||||
"motorcycle": 0.9,
|
||||
"person": 0.9,
|
||||
"vehicle": 1.5
|
||||
},
|
||||
"vehicle_labels": ["car", "truck", "bus"]
|
||||
},
|
||||
"inputs": {
|
||||
"semantic_result_id": "result-793785170472c519486ccd666be102fb04d169d92383acda3fcc29eecf045d30",
|
||||
"tracking_result_id": "e5-tracking-8035c0013e36a7cfb68aa3652b05ab79fe36baf136462d5fbf91b348ee85ec08"
|
||||
},
|
||||
"presentation": {
|
||||
"contact_sheet_columns": 3,
|
||||
"contact_sheet_frames": 6,
|
||||
"distance_history_frames": 5,
|
||||
"jpeg_quality": 85,
|
||||
"overlay_crf": 20,
|
||||
"point_radius_ui": 1.5,
|
||||
"support_radius_ui": 3.0
|
||||
},
|
||||
"projection": {
|
||||
"occlusion_policy": "nearest-depth-per-rounded-pixel",
|
||||
"pixel_rounding": "nearest",
|
||||
"source_coordinates": "k1-map-frame",
|
||||
"visible_point_policy": "camera-front-in-frame"
|
||||
},
|
||||
"schema_version": "missioncore.e6-tracking-lidar-profile/v1",
|
||||
"source": {
|
||||
"calibration_sha256": "05f3ad9b38b3a4fc95388a8ec83da83c745e217709e51787b3d5aad0969f6fa9",
|
||||
"calibration_slot": "camera_1",
|
||||
"resolution": [800, 600],
|
||||
"source_id": "sensor.camera.right"
|
||||
},
|
||||
"temporal": {
|
||||
"binding": "nearest-host-arrival-best-effort",
|
||||
"maximum_lidar_camera_delta_ms": 100.0,
|
||||
"maximum_pose_point_delta_ms": 100.0
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,72 @@
|
||||
{
|
||||
"association": {
|
||||
"box_inset": {
|
||||
"bottom_fraction": 0.02,
|
||||
"horizontal_fraction": 0.05,
|
||||
"top_fraction": 0.04
|
||||
},
|
||||
"depth_cluster_gap_fraction": 0.06,
|
||||
"depth_cluster_minimum_gap_m": 0.5,
|
||||
"group_nms_iou_threshold": 0.55,
|
||||
"maximum_cuboid_span_m": 12.0,
|
||||
"maximum_distance_innovation_fraction": 0.25,
|
||||
"maximum_distance_innovation_m": 1.5,
|
||||
"maximum_oriented_extent_m": {
|
||||
"bicycle": [3.5, 2.0, 2.5],
|
||||
"motorcycle": [3.5, 2.0, 2.5],
|
||||
"person": [1.5, 1.5, 2.8],
|
||||
"vehicle": [6.5, 3.5, 4.0]
|
||||
},
|
||||
"minimum_cuboid_extent_m": 0.15,
|
||||
"minimum_support_points": {
|
||||
"bicycle": 3,
|
||||
"motorcycle": 3,
|
||||
"person": 3,
|
||||
"vehicle": 5
|
||||
},
|
||||
"semantic_ids": {
|
||||
"bicycle": [2],
|
||||
"motorcycle": [3],
|
||||
"person": [1],
|
||||
"vehicle": [4, 5]
|
||||
},
|
||||
"spatial_cluster_radius_m": {
|
||||
"bicycle": 0.9,
|
||||
"motorcycle": 0.9,
|
||||
"person": 0.9,
|
||||
"vehicle": 1.5
|
||||
},
|
||||
"vehicle_labels": ["car", "truck", "bus"]
|
||||
},
|
||||
"inputs": {
|
||||
"semantic_result_id": "result-793785170472c519486ccd666be102fb04d169d92383acda3fcc29eecf045d30",
|
||||
"tracking_result_id": "e5-tracking-88aace13ef9963f8dc07f85228e530f9d28c2b49aca9192409f7975512b058f6"
|
||||
},
|
||||
"presentation": {
|
||||
"contact_sheet_columns": 3,
|
||||
"contact_sheet_frames": 6,
|
||||
"distance_history_frames": 5,
|
||||
"jpeg_quality": 85,
|
||||
"overlay_crf": 20,
|
||||
"point_radius_ui": 1.5,
|
||||
"support_radius_ui": 3.0
|
||||
},
|
||||
"projection": {
|
||||
"occlusion_policy": "nearest-depth-per-rounded-pixel",
|
||||
"pixel_rounding": "nearest",
|
||||
"source_coordinates": "k1-map-frame",
|
||||
"visible_point_policy": "camera-front-in-frame"
|
||||
},
|
||||
"schema_version": "missioncore.e6-tracking-lidar-profile/v1",
|
||||
"source": {
|
||||
"calibration_sha256": "05f3ad9b38b3a4fc95388a8ec83da83c745e217709e51787b3d5aad0969f6fa9",
|
||||
"calibration_slot": "camera_1",
|
||||
"resolution": [800, 600],
|
||||
"source_id": "sensor.camera.right"
|
||||
},
|
||||
"temporal": {
|
||||
"binding": "nearest-host-arrival-best-effort",
|
||||
"maximum_lidar_camera_delta_ms": 100.0,
|
||||
"maximum_pose_point_delta_ms": 100.0
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,38 @@
|
||||
{
|
||||
"schema_version": "missioncore.e7-replay-realtime-profile/v1",
|
||||
"mode": "overload-negative-control",
|
||||
"selection": {
|
||||
"start_frame_index": 120,
|
||||
"frame_count": 80
|
||||
},
|
||||
"replay": {
|
||||
"speed": 1.0,
|
||||
"queue_capacity": 2,
|
||||
"consumer_delay_ms": 250.0
|
||||
},
|
||||
"health": {
|
||||
"stale_after_ms": 300.0,
|
||||
"unavailable_after_ms": 1000.0
|
||||
},
|
||||
"world_state": {
|
||||
"velocity_history_limit_s": 1.0,
|
||||
"clearance": {
|
||||
"sector_count": 72,
|
||||
"minimum_range_m": 0.5,
|
||||
"maximum_range_m": 30.0,
|
||||
"ground_percentile": 5.0,
|
||||
"minimum_height_above_ground_m": 0.2,
|
||||
"maximum_height_above_ground_m": 3.0,
|
||||
"front_half_angle_degrees": 15.0
|
||||
}
|
||||
},
|
||||
"presentation": {
|
||||
"log_point_cloud": false,
|
||||
"point_radius_ui": 1.25
|
||||
},
|
||||
"acceptance": {
|
||||
"minimum_drop_fraction": 0.25,
|
||||
"maximum_queue_depth": 2,
|
||||
"maximum_process_rss_mib": 512.0
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,40 @@
|
||||
{
|
||||
"schema_version": "missioncore.e7-replay-realtime-profile/v1",
|
||||
"mode": "pilot",
|
||||
"selection": {
|
||||
"start_frame_index": 120,
|
||||
"frame_count": 64
|
||||
},
|
||||
"replay": {
|
||||
"speed": 1.0,
|
||||
"queue_capacity": 2,
|
||||
"consumer_delay_ms": 0.0
|
||||
},
|
||||
"health": {
|
||||
"stale_after_ms": 300.0,
|
||||
"unavailable_after_ms": 1000.0
|
||||
},
|
||||
"world_state": {
|
||||
"velocity_history_limit_s": 1.0,
|
||||
"clearance": {
|
||||
"sector_count": 72,
|
||||
"minimum_range_m": 0.5,
|
||||
"maximum_range_m": 30.0,
|
||||
"ground_percentile": 5.0,
|
||||
"minimum_height_above_ground_m": 0.2,
|
||||
"maximum_height_above_ground_m": 3.0,
|
||||
"front_half_angle_degrees": 15.0
|
||||
}
|
||||
},
|
||||
"presentation": {
|
||||
"log_point_cloud": true,
|
||||
"point_radius_ui": 1.25
|
||||
},
|
||||
"acceptance": {
|
||||
"minimum_delivery_rate_hz": 9.0,
|
||||
"maximum_end_to_end_p95_ms": 300.0,
|
||||
"maximum_drop_fraction": 0.02,
|
||||
"maximum_queue_depth": 2,
|
||||
"maximum_process_rss_mib": 512.0
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,40 @@
|
||||
{
|
||||
"schema_version": "missioncore.e7-replay-realtime-profile/v1",
|
||||
"mode": "qualification",
|
||||
"selection": {
|
||||
"start_frame_index": 0,
|
||||
"frame_count": 601
|
||||
},
|
||||
"replay": {
|
||||
"speed": 1.0,
|
||||
"queue_capacity": 2,
|
||||
"consumer_delay_ms": 0.0
|
||||
},
|
||||
"health": {
|
||||
"stale_after_ms": 300.0,
|
||||
"unavailable_after_ms": 1000.0
|
||||
},
|
||||
"world_state": {
|
||||
"velocity_history_limit_s": 1.0,
|
||||
"clearance": {
|
||||
"sector_count": 72,
|
||||
"minimum_range_m": 0.5,
|
||||
"maximum_range_m": 30.0,
|
||||
"ground_percentile": 5.0,
|
||||
"minimum_height_above_ground_m": 0.2,
|
||||
"maximum_height_above_ground_m": 3.0,
|
||||
"front_half_angle_degrees": 15.0
|
||||
}
|
||||
},
|
||||
"presentation": {
|
||||
"log_point_cloud": true,
|
||||
"point_radius_ui": 1.25
|
||||
},
|
||||
"acceptance": {
|
||||
"minimum_delivery_rate_hz": 9.5,
|
||||
"maximum_end_to_end_p95_ms": 300.0,
|
||||
"maximum_drop_fraction": 0.01,
|
||||
"maximum_queue_depth": 2,
|
||||
"maximum_process_rss_mib": 512.0
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,543 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Fuse accepted recorded image masks into calibrated K1 LiDAR observations."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
from dataclasses import dataclass
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
from uuid import uuid4
|
||||
|
||||
import numpy as np
|
||||
import numpy.typing as npt
|
||||
import rerun as rr
|
||||
from PIL import Image, ImageDraw
|
||||
from rerun.components import FillMode
|
||||
|
||||
from k1link.device_plugins.xgrids_k1.analyze.calibrated_overlay import (
|
||||
_extract_camera_frames,
|
||||
_load_calibration_snapshot,
|
||||
_select_camera_anchors,
|
||||
_select_lidar_samples,
|
||||
)
|
||||
from k1link.device_plugins.xgrids_k1.analyze.calibrated_projection import (
|
||||
Kb4ProjectionProfile,
|
||||
map_points_to_lidar,
|
||||
project_map_points_kb4,
|
||||
)
|
||||
from k1link.device_plugins.xgrids_k1.mqtt.capture import read_capture_clock_origin
|
||||
from k1link.sessions import inspect_recorded_media_epoch
|
||||
|
||||
SCHEMA = "missioncore.calibrated-segmentation-fusion-experiment/v1"
|
||||
ACCEPTED_INSTANCE_MODEL = "torchvision/maskrcnn_resnet50_fpn_v2"
|
||||
ACCEPTED_SEMANTIC_MODEL = "microsoft/beit-base-finetuned-ade-640-640"
|
||||
MIN_BOX_POINTS = 4
|
||||
MAX_BOX_SPAN_M = 12.0
|
||||
|
||||
FloatArray = npt.NDArray[np.float64]
|
||||
RgbImage = npt.NDArray[np.uint8]
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ObjectFusion:
|
||||
instance_id: int
|
||||
label: str
|
||||
score: float
|
||||
candidate_points: int
|
||||
clustered_points: int
|
||||
distance_p10_m: float | None
|
||||
distance_median_m: float | None
|
||||
box_center_map: tuple[float, float, float] | None
|
||||
box_half_size_map: tuple[float, float, float] | None
|
||||
box_status: str
|
||||
source_indices: npt.NDArray[np.int64]
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class FusedFrame:
|
||||
stem: str
|
||||
session_time_seconds: float
|
||||
image_rgb: RgbImage
|
||||
semantic_overlay_rgb: RgbImage
|
||||
fusion_overlay_rgb: RgbImage
|
||||
points_map: FloatArray
|
||||
projected_source_indices: npt.NDArray[np.int64]
|
||||
projected_pixels: FloatArray
|
||||
semantic_colors: RgbImage
|
||||
objects: tuple[ObjectFusion, ...]
|
||||
|
||||
|
||||
def _arguments() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--session", type=Path, required=True)
|
||||
parser.add_argument("--calibration", type=Path, required=True)
|
||||
parser.add_argument("--calibrated-manifest", type=Path, required=True)
|
||||
parser.add_argument("--segmentation", type=Path, required=True)
|
||||
parser.add_argument("--output-root", type=Path, required=True)
|
||||
parser.add_argument("--ffmpeg", type=Path, required=True)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def _read_json(path: Path) -> dict[str, Any]:
|
||||
value = json.loads(path.read_text(encoding="utf-8"))
|
||||
if not isinstance(value, dict):
|
||||
raise RuntimeError(f"{path.name} is not a JSON object")
|
||||
return value
|
||||
|
||||
|
||||
def _sha256(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as stream:
|
||||
for chunk in iter(lambda: stream.read(1024 * 1024), b""):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def _palette(index: int) -> tuple[int, int, int]:
|
||||
digest = hashlib.sha256(f"mission-core-segment-{index}".encode()).digest()
|
||||
return (64 + digest[0] % 176, 64 + digest[1] % 176, 64 + digest[2] % 176)
|
||||
|
||||
|
||||
def _depth_cluster(indices: npt.NDArray[np.int64], depths: FloatArray) -> npt.NDArray[np.int64]:
|
||||
if indices.size < 2:
|
||||
return indices
|
||||
order = np.argsort(depths[indices])
|
||||
ordered = indices[order]
|
||||
ordered_depths = depths[ordered]
|
||||
groups: list[npt.NDArray[np.int64]] = []
|
||||
start = 0
|
||||
for offset, gap in enumerate(np.diff(ordered_depths), start=1):
|
||||
threshold = max(0.6, 0.08 * float(ordered_depths[offset - 1]))
|
||||
if float(gap) > threshold:
|
||||
groups.append(ordered[start:offset])
|
||||
start = offset
|
||||
groups.append(ordered[start:])
|
||||
return min(
|
||||
groups,
|
||||
key=lambda group: (-int(group.size), float(np.median(depths[group]))),
|
||||
)
|
||||
|
||||
|
||||
def _object_fusions(
|
||||
*,
|
||||
instance_map: npt.NDArray[np.uint16],
|
||||
instances: list[dict[str, Any]],
|
||||
pixel_x: npt.NDArray[np.int64],
|
||||
pixel_y: npt.NDArray[np.int64],
|
||||
depths: FloatArray,
|
||||
projected_source_indices: npt.NDArray[np.int64],
|
||||
points_map: FloatArray,
|
||||
points_lidar: FloatArray,
|
||||
) -> tuple[ObjectFusion, ...]:
|
||||
sampled_instances = instance_map[pixel_y, pixel_x]
|
||||
fused: list[ObjectFusion] = []
|
||||
for item in instances:
|
||||
instance_id = int(item["instance_id"])
|
||||
candidates = np.flatnonzero(sampled_instances == instance_id).astype(np.int64)
|
||||
clustered = _depth_cluster(candidates, depths)
|
||||
source_indices = projected_source_indices[clustered]
|
||||
ranges = np.linalg.norm(points_lidar[source_indices], axis=1)
|
||||
distance_p10 = None if ranges.size == 0 else float(np.percentile(ranges, 10.0))
|
||||
distance_median = None if ranges.size == 0 else float(np.median(ranges))
|
||||
center: tuple[float, float, float] | None = None
|
||||
half_size: tuple[float, float, float] | None = None
|
||||
if source_indices.size < MIN_BOX_POINTS:
|
||||
status = f"rejected-fewer-than-{MIN_BOX_POINTS}-clustered-points"
|
||||
else:
|
||||
selected = points_map[source_indices]
|
||||
lower = np.percentile(selected, 5.0, axis=0)
|
||||
upper = np.percentile(selected, 95.0, axis=0)
|
||||
spans = upper - lower
|
||||
if not np.isfinite(spans).all() or np.any(spans > MAX_BOX_SPAN_M):
|
||||
status = "rejected-implausible-span"
|
||||
else:
|
||||
sizes = np.maximum(spans, 0.15)
|
||||
center = tuple(float(value) for value in ((lower + upper) * 0.5))
|
||||
half_size = tuple(float(value) for value in (sizes * 0.5))
|
||||
status = "accepted-diagnostic-axis-aligned"
|
||||
fused.append(
|
||||
ObjectFusion(
|
||||
instance_id=instance_id,
|
||||
label=str(item["label"]),
|
||||
score=float(item["score"]),
|
||||
candidate_points=int(candidates.size),
|
||||
clustered_points=int(clustered.size),
|
||||
distance_p10_m=distance_p10,
|
||||
distance_median_m=distance_median,
|
||||
box_center_map=center,
|
||||
box_half_size_map=half_size,
|
||||
box_status=status,
|
||||
source_indices=source_indices,
|
||||
)
|
||||
)
|
||||
return tuple(fused)
|
||||
|
||||
|
||||
def _fusion_overlay(
|
||||
semantic_overlay: RgbImage,
|
||||
projected_pixels: FloatArray,
|
||||
semantic_colors: RgbImage,
|
||||
objects: tuple[ObjectFusion, ...],
|
||||
instances: list[dict[str, Any]],
|
||||
) -> RgbImage:
|
||||
canvas = Image.fromarray(semantic_overlay).convert("RGBA")
|
||||
draw = ImageDraw.Draw(canvas, "RGBA")
|
||||
for (u, v), color in zip(projected_pixels, semantic_colors, strict=True):
|
||||
red, green, blue = (int(value) for value in color)
|
||||
draw.ellipse((u - 1.5, v - 1.5, u + 1.5, v + 1.5), fill=(red, green, blue, 210))
|
||||
by_id = {item.instance_id: item for item in objects}
|
||||
for item in instances:
|
||||
fused = by_id[int(item["instance_id"])]
|
||||
x1, y1, x2, y2 = (float(value) for value in item["box_xyxy"])
|
||||
color = _palette(500 + fused.instance_id)
|
||||
accepted = fused.box_center_map is not None
|
||||
alpha = 255 if accepted else 135
|
||||
draw.rectangle((x1, y1, x2, y2), outline=(*color, alpha), width=2)
|
||||
distance = (
|
||||
"no LiDAR"
|
||||
if fused.distance_median_m is None
|
||||
else f"{fused.distance_median_m:.1f}m/{fused.clustered_points}pts"
|
||||
)
|
||||
draw.text(
|
||||
(x1 + 2, max(0.0, y1 - 12)),
|
||||
f"{fused.label} {distance}",
|
||||
fill=(255, 255, 255, alpha),
|
||||
stroke_width=2,
|
||||
stroke_fill=(0, 0, 0, alpha),
|
||||
)
|
||||
return np.asarray(canvas.convert("RGB"), dtype=np.uint8)
|
||||
|
||||
|
||||
def _object_document(item: ObjectFusion) -> dict[str, Any]:
|
||||
return {
|
||||
"instance_id": item.instance_id,
|
||||
"label": item.label,
|
||||
"score": round(item.score, 9),
|
||||
"candidate_projected_points": item.candidate_points,
|
||||
"clustered_points": item.clustered_points,
|
||||
"distance_p10_m": item.distance_p10_m,
|
||||
"distance_median_m": item.distance_median_m,
|
||||
"box_status": item.box_status,
|
||||
"box_center_map": item.box_center_map,
|
||||
"box_half_size_map": item.box_half_size_map,
|
||||
}
|
||||
|
||||
|
||||
def _write_rerun(path: Path, experiment_id: str, frames: tuple[FusedFrame, ...]) -> None:
|
||||
recording = rr.RecordingStream("nodedc_k1_calibrated_segmentation", recording_id=experiment_id)
|
||||
recording.set_sinks(rr.FileSink(path, write_footer=True))
|
||||
try:
|
||||
recording.log(
|
||||
"contract",
|
||||
rr.TextDocument(
|
||||
"Recorded-only diagnostic: exact image masks sampled at factory-calibrated "
|
||||
"K1 LiDAR projections; 3D boxes require clustered LiDAR support."
|
||||
),
|
||||
static=True,
|
||||
)
|
||||
for frame in frames:
|
||||
recording.set_time(
|
||||
"session_time",
|
||||
duration=np.timedelta64(round(frame.session_time_seconds * 1_000_000_000), "ns"),
|
||||
)
|
||||
recording.log("camera/raw", rr.Image(frame.image_rgb))
|
||||
recording.log("camera/semantic", rr.Image(frame.semantic_overlay_rgb))
|
||||
recording.log("camera/fusion", rr.Image(frame.fusion_overlay_rgb))
|
||||
recording.log(
|
||||
"camera/projected_lidar_semantic",
|
||||
rr.Points2D(
|
||||
frame.projected_pixels.astype(np.float32),
|
||||
colors=frame.semantic_colors,
|
||||
radii=rr.Radius.ui_points(2.0),
|
||||
),
|
||||
)
|
||||
recording.log(
|
||||
"world/semantic_points",
|
||||
rr.Points3D(
|
||||
frame.points_map[frame.projected_source_indices].astype(np.float32),
|
||||
colors=frame.semantic_colors,
|
||||
radii=rr.Radius.ui_points(2.0),
|
||||
),
|
||||
)
|
||||
boxed = [item for item in frame.objects if item.box_center_map is not None]
|
||||
if boxed:
|
||||
recording.log(
|
||||
"world/boxes3d",
|
||||
rr.Boxes3D(
|
||||
centers=[item.box_center_map for item in boxed],
|
||||
half_sizes=[item.box_half_size_map for item in boxed],
|
||||
colors=[(*_palette(500 + item.instance_id), 96) for item in boxed],
|
||||
fill_mode=FillMode.Solid,
|
||||
labels=[
|
||||
f"{item.label} · {item.distance_median_m:.1f} m · "
|
||||
f"{item.clustered_points} pts"
|
||||
for item in boxed
|
||||
],
|
||||
show_labels=True,
|
||||
),
|
||||
)
|
||||
finally:
|
||||
recording.flush(timeout_sec=30.0)
|
||||
recording.disconnect()
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = _arguments()
|
||||
session = args.session.resolve(strict=True)
|
||||
calibration_root = args.calibration.resolve(strict=True)
|
||||
calibrated_manifest = _read_json(args.calibrated_manifest.resolve(strict=True))
|
||||
segmentation_root = args.segmentation.resolve(strict=True)
|
||||
segmentation_manifest = _read_json(segmentation_root / "manifest.redacted.json")
|
||||
ffmpeg = args.ffmpeg.resolve(strict=True)
|
||||
|
||||
models = segmentation_manifest.get("models")
|
||||
acceptance = segmentation_manifest.get("acceptance")
|
||||
if not isinstance(models, dict) or not isinstance(acceptance, dict):
|
||||
raise RuntimeError("segmentation manifest is incomplete")
|
||||
instance_model = models.get("instance")
|
||||
semantic_model = models.get("semantic")
|
||||
if (
|
||||
not isinstance(instance_model, dict)
|
||||
or instance_model.get("id") != ACCEPTED_INSTANCE_MODEL
|
||||
or not isinstance(semantic_model, dict)
|
||||
or semantic_model.get("id") != ACCEPTED_SEMANTIC_MODEL
|
||||
or semantic_model.get("checkpoint_load") != "exact-no-missing-unexpected-or-mismatched-keys"
|
||||
or acceptance.get("recorded_segmentation_artifact") != "generated"
|
||||
):
|
||||
raise RuntimeError("segmentation result is not an admitted strict-load experiment")
|
||||
|
||||
calibration, calibration_identity = _load_calibration_snapshot(calibration_root)
|
||||
calibrated_input = calibrated_manifest.get("input")
|
||||
if (
|
||||
not isinstance(calibrated_input, dict)
|
||||
or calibrated_input.get("session_id") != session.name
|
||||
or calibrated_input.get("calibration_content_identity_sha256") != calibration_identity
|
||||
):
|
||||
raise RuntimeError("calibrated experiment does not bind this session/calibration")
|
||||
source_id = str(calibrated_input["source_id"])
|
||||
offsets = tuple(float(value) for value in calibrated_input["video_offsets_seconds"])
|
||||
profile = Kb4ProjectionProfile.from_factory_calibration(calibration, source_id)
|
||||
|
||||
capture_root = session / "captures" / "mqtt_live"
|
||||
origin = read_capture_clock_origin(capture_root / "mqtt.timeline.origin.json")
|
||||
epoch_root = session / "media" / source_id / "epoch-1"
|
||||
epoch = inspect_recorded_media_epoch(
|
||||
epoch_root,
|
||||
expected_source_name=source_id,
|
||||
origin_epoch_ns=origin.started_at_epoch_ns,
|
||||
origin_monotonic_ns=origin.started_monotonic_ns,
|
||||
)
|
||||
anchors = _select_camera_anchors(
|
||||
epoch_root / "index.jsonl",
|
||||
expected_count=len(epoch.segments),
|
||||
offsets_seconds=offsets,
|
||||
timeline_start_seconds=epoch.timeline_start_seconds,
|
||||
origin_monotonic_ns=origin.started_monotonic_ns,
|
||||
)
|
||||
images = _extract_camera_frames(
|
||||
ffmpeg,
|
||||
init_path=epoch.init_path,
|
||||
segment_paths=tuple(item.path for item in epoch.segments),
|
||||
frame_sequences=tuple(item.sequence for item in anchors),
|
||||
width=profile.width,
|
||||
height=profile.height,
|
||||
)
|
||||
lidar_samples = _select_lidar_samples(
|
||||
capture_root / "mqtt.raw.k1mqtt",
|
||||
anchors,
|
||||
origin_monotonic_ns=origin.started_monotonic_ns,
|
||||
temporal_offset_seconds=float(calibrated_input["temporal_offset_seconds"]),
|
||||
)
|
||||
|
||||
segmentation_inputs = {
|
||||
str(item["name"]): str(item["sha256"])
|
||||
for item in segmentation_manifest["input"]
|
||||
if isinstance(item, dict)
|
||||
}
|
||||
segmentation_frames = segmentation_manifest["frames"]
|
||||
fused_frames: list[FusedFrame] = []
|
||||
for image, anchor, lidar_sample in zip(images, anchors, lidar_samples, strict=True):
|
||||
stem = (
|
||||
f"camera-{anchor.sequence:06d}-"
|
||||
f"session-{round(anchor.session_time_seconds * 1000):09d}ms"
|
||||
)
|
||||
input_name = f"{stem}.png"
|
||||
input_artifact = args.calibrated_manifest.parent / input_name
|
||||
if segmentation_inputs.get(input_name) != _sha256(input_artifact):
|
||||
raise RuntimeError(f"segmentation input identity changed for {input_name}")
|
||||
if _sha256(input_artifact) != hashlib.sha256(image.tobytes()).hexdigest():
|
||||
decoded = np.asarray(Image.open(input_artifact).convert("RGB"), dtype=np.uint8)
|
||||
if not np.array_equal(decoded, image):
|
||||
raise RuntimeError(f"decoded camera pixels changed for {input_name}")
|
||||
|
||||
semantic_map = np.asarray(Image.open(segmentation_root / f"{stem}.semantic.png"))
|
||||
instance_map = np.asarray(Image.open(segmentation_root / f"{stem}.instances.png"))
|
||||
semantic_overlay = np.asarray(
|
||||
Image.open(segmentation_root / f"{stem}.semantic-overlay.png").convert("RGB")
|
||||
)
|
||||
if semantic_map.shape != image.shape[:2] or instance_map.shape != image.shape[:2]:
|
||||
raise RuntimeError("segmentation maps do not match the camera epoch")
|
||||
|
||||
points_map = np.asarray(
|
||||
[
|
||||
point.scaled_xyz(lidar_sample.point_frame.header.scaler)
|
||||
for point in lidar_sample.point_frame.points
|
||||
],
|
||||
dtype=np.float64,
|
||||
)
|
||||
points_lidar = map_points_to_lidar(
|
||||
points_map,
|
||||
position_map_xyz=lidar_sample.pose_frame.position_xyz,
|
||||
orientation_map_from_lidar_xyzw=lidar_sample.pose_frame.orientation_xyzw,
|
||||
)
|
||||
projection = project_map_points_kb4(
|
||||
points_map,
|
||||
position_map_xyz=lidar_sample.pose_frame.position_xyz,
|
||||
orientation_map_from_lidar_xyzw=lidar_sample.pose_frame.orientation_xyzw,
|
||||
profile=profile,
|
||||
)
|
||||
pixel_x = np.clip(
|
||||
np.rint(projection.pixels_xy[:, 0]).astype(np.int64), 0, profile.width - 1
|
||||
)
|
||||
pixel_y = np.clip(
|
||||
np.rint(projection.pixels_xy[:, 1]).astype(np.int64), 0, profile.height - 1
|
||||
)
|
||||
semantic_ids = semantic_map[pixel_y, pixel_x].astype(np.int64)
|
||||
semantic_colors = np.asarray(
|
||||
[_palette(int(value)) for value in semantic_ids], dtype=np.uint8
|
||||
)
|
||||
frame_document = segmentation_frames[stem]
|
||||
instances = frame_document["instance"]["instances"]
|
||||
objects = _object_fusions(
|
||||
instance_map=instance_map.astype(np.uint16),
|
||||
instances=instances,
|
||||
pixel_x=pixel_x,
|
||||
pixel_y=pixel_y,
|
||||
depths=projection.depths_m,
|
||||
projected_source_indices=projection.source_indices,
|
||||
points_map=points_map,
|
||||
points_lidar=points_lidar,
|
||||
)
|
||||
fusion_overlay = _fusion_overlay(
|
||||
semantic_overlay,
|
||||
projection.pixels_xy,
|
||||
semantic_colors,
|
||||
objects,
|
||||
instances,
|
||||
)
|
||||
fused_frames.append(
|
||||
FusedFrame(
|
||||
stem=stem,
|
||||
session_time_seconds=anchor.session_time_seconds,
|
||||
image_rgb=image,
|
||||
semantic_overlay_rgb=semantic_overlay,
|
||||
fusion_overlay_rgb=fusion_overlay,
|
||||
points_map=points_map,
|
||||
projected_source_indices=projection.source_indices,
|
||||
projected_pixels=projection.pixels_xy,
|
||||
semantic_colors=semantic_colors,
|
||||
objects=objects,
|
||||
)
|
||||
)
|
||||
|
||||
created_at = datetime.now(UTC)
|
||||
experiment_id = (
|
||||
f"{created_at.strftime('%Y%m%dT%H%M%SZ')}_k1_segmentation_fusion_{uuid4().hex[:12]}"
|
||||
)
|
||||
private_root = args.output_root.resolve() / "private" / "perception-experiments"
|
||||
private_root.mkdir(mode=0o700, parents=True, exist_ok=True)
|
||||
output = private_root / experiment_id
|
||||
staging = private_root / f".{experiment_id}.incomplete"
|
||||
staging.mkdir(mode=0o700, exist_ok=False)
|
||||
try:
|
||||
for frame in fused_frames:
|
||||
Image.fromarray(frame.fusion_overlay_rgb).save(staging / f"{frame.stem}.fusion.png")
|
||||
mosaic_rows = [
|
||||
np.concatenate(
|
||||
(frame.image_rgb, frame.semantic_overlay_rgb, frame.fusion_overlay_rgb),
|
||||
axis=1,
|
||||
)
|
||||
for frame in fused_frames
|
||||
]
|
||||
Image.fromarray(np.concatenate(mosaic_rows, axis=0)).save(staging / "fusion-mosaic.png")
|
||||
_write_rerun(staging / "fusion.rrd", experiment_id, tuple(fused_frames))
|
||||
identity = {
|
||||
"calibrated_generation_sha256": calibrated_manifest["generation_sha256"],
|
||||
"segmentation_generation_sha256": segmentation_manifest["generation_sha256"],
|
||||
"calibration_content_identity_sha256": calibration_identity,
|
||||
"session_id": session.name,
|
||||
"source_id": source_id,
|
||||
}
|
||||
outputs = sorted(staging.iterdir())
|
||||
manifest = {
|
||||
"schema_version": SCHEMA,
|
||||
"experiment_id": experiment_id,
|
||||
"created_at_utc": created_at.isoformat(timespec="milliseconds").replace("+00:00", "Z"),
|
||||
"generation_sha256": hashlib.sha256(
|
||||
json.dumps(identity, sort_keys=True, separators=(",", ":")).encode()
|
||||
).hexdigest(),
|
||||
"input": identity,
|
||||
"fusion": {
|
||||
"pixel_sampling": "nearest-calibrated-projection",
|
||||
"occlusion_policy": "largest-depth-contiguous-cluster-nearest-tiebreak",
|
||||
"distance": "LiDAR-origin Euclidean p10 and median",
|
||||
"box": {
|
||||
"frame": "map",
|
||||
"orientation": "axis-aligned-diagnostic",
|
||||
"minimum_clustered_points": MIN_BOX_POINTS,
|
||||
"robust_bounds": "p05-p95",
|
||||
"maximum_span_m": MAX_BOX_SPAN_M,
|
||||
},
|
||||
},
|
||||
"frames": [
|
||||
{
|
||||
"stem": frame.stem,
|
||||
"session_time_seconds": frame.session_time_seconds,
|
||||
"projected_points": int(frame.projected_source_indices.size),
|
||||
"semantic_class_ids": sorted(
|
||||
{
|
||||
int(value)
|
||||
for value in np.asarray(
|
||||
Image.open(segmentation_root / f"{frame.stem}.semantic.png")
|
||||
).reshape(-1)
|
||||
}
|
||||
),
|
||||
"objects": [_object_document(item) for item in frame.objects],
|
||||
"accepted_boxes": sum(
|
||||
item.box_center_map is not None for item in frame.objects
|
||||
),
|
||||
}
|
||||
for frame in fused_frames
|
||||
],
|
||||
"acceptance": {
|
||||
"recorded_mask_to_lidar_fusion": "generated",
|
||||
"distance": "diagnostic-not-ground-truthed",
|
||||
"boxes3d": "diagnostic-not-tracked-or-ground-truthed",
|
||||
"live": "not-tested",
|
||||
"safety": "not-accepted",
|
||||
},
|
||||
"outputs": [
|
||||
{"name": path.name, "bytes": path.stat().st_size, "sha256": _sha256(path)}
|
||||
for path in outputs
|
||||
],
|
||||
}
|
||||
(staging / "manifest.redacted.json").write_text(
|
||||
json.dumps(manifest, ensure_ascii=False, indent=2, sort_keys=True) + "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
os.rename(staging, output)
|
||||
except BaseException:
|
||||
shutil.rmtree(staging, ignore_errors=True)
|
||||
raise
|
||||
print(json.dumps({"experiment_id": experiment_id, "output": str(output)}))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,498 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Fuse one admitted full camera-perception epoch into compact 3D observations."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import shutil
|
||||
import tarfile
|
||||
import time
|
||||
from collections import deque
|
||||
from collections.abc import Iterator
|
||||
from dataclasses import dataclass
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import numpy as np
|
||||
import numpy.typing as npt
|
||||
from fuse_calibrated_segmentation import _object_fusions, _palette
|
||||
from PIL import Image
|
||||
|
||||
from k1link.compute import validate_recorded_perception_epoch_result
|
||||
from k1link.device_plugins.xgrids_k1.analyze.calibrated_overlay import (
|
||||
MAX_SYNC_DELTA_SECONDS,
|
||||
_load_calibration_snapshot,
|
||||
)
|
||||
from k1link.device_plugins.xgrids_k1.analyze.calibrated_projection import (
|
||||
Kb4ProjectionProfile,
|
||||
map_points_to_lidar,
|
||||
project_map_points_kb4,
|
||||
)
|
||||
from k1link.device_plugins.xgrids_k1.mqtt.capture import read_capture_clock_origin
|
||||
from k1link.device_plugins.xgrids_k1.protocol.streams import (
|
||||
LioPointCloudFrame,
|
||||
LioPoseFrame,
|
||||
decode_lio_pcl,
|
||||
decode_lio_pose,
|
||||
)
|
||||
from k1link.device_plugins.xgrids_k1.viewer.replay import iter_replay_messages
|
||||
|
||||
SCHEMA = "missioncore.recorded-calibrated-fusion/v1"
|
||||
IDENTITY_SCHEMA = "missioncore.recorded-calibrated-fusion-identity/v1"
|
||||
MAX_MASK_MEMBER_BYTES = 4 * 1024 * 1024
|
||||
MAX_INDEX_LINE_BYTES = 64 * 1024
|
||||
|
||||
FloatArray = npt.NDArray[np.float64]
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class CameraAnchor:
|
||||
sequence: int
|
||||
host_session_seconds: float
|
||||
video_session_seconds: float
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class LidarSample:
|
||||
point_session_seconds: float
|
||||
pose_session_seconds: float
|
||||
point_frame: LioPointCloudFrame
|
||||
pose_frame: LioPoseFrame
|
||||
|
||||
|
||||
def _arguments() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--job", type=Path, required=True)
|
||||
parser.add_argument("--result", type=Path, required=True)
|
||||
parser.add_argument("--session", type=Path, required=True)
|
||||
parser.add_argument("--calibration", type=Path, required=True)
|
||||
parser.add_argument("--output-root", type=Path, required=True)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def _canonical_json(value: object) -> bytes:
|
||||
return json.dumps(
|
||||
value,
|
||||
ensure_ascii=False,
|
||||
sort_keys=True,
|
||||
separators=(",", ":"),
|
||||
allow_nan=False,
|
||||
).encode()
|
||||
|
||||
|
||||
def _sha256(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as stream:
|
||||
while chunk := stream.read(1024 * 1024):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def _read_frame_metadata(path: Path, expected_count: int) -> list[dict[str, Any]]:
|
||||
frames: list[dict[str, Any]] = []
|
||||
with path.open("r", encoding="utf-8") as stream:
|
||||
for expected_index, line in enumerate(stream):
|
||||
value = json.loads(line)
|
||||
if not isinstance(value, dict) or value.get("frame_index") != expected_index:
|
||||
raise RuntimeError("perception frame metadata changed")
|
||||
frames.append(value)
|
||||
if len(frames) != expected_count:
|
||||
raise RuntimeError("perception frame metadata is incomplete")
|
||||
return frames
|
||||
|
||||
|
||||
def _extract_masks(archive: Path, destination: Path, frame_count: int) -> None:
|
||||
expected = {
|
||||
f"{kind}/frame-{sequence:06d}.png"
|
||||
for kind in ("instance-masks", "semantic-masks")
|
||||
for sequence in range(1, frame_count + 1)
|
||||
}
|
||||
with tarfile.open(archive, mode="r:gz") as source:
|
||||
members = [member for member in source.getmembers() if member.isfile()]
|
||||
found = {member.name.replace("\\", "/").removeprefix("./") for member in members}
|
||||
if found != expected or len(members) != len(expected):
|
||||
raise RuntimeError("panoptic mask archive contents changed")
|
||||
for member in members:
|
||||
name = member.name.replace("\\", "/").removeprefix("./")
|
||||
if (
|
||||
member.issym()
|
||||
or member.islnk()
|
||||
or member.size < 1
|
||||
or member.size > MAX_MASK_MEMBER_BYTES
|
||||
):
|
||||
raise RuntimeError("panoptic mask archive member is unsafe")
|
||||
target = destination.joinpath(*name.split("/"))
|
||||
target.parent.mkdir(mode=0o700, parents=True, exist_ok=True)
|
||||
payload = source.extractfile(member)
|
||||
if payload is None:
|
||||
raise RuntimeError("panoptic mask archive member is unavailable")
|
||||
with target.open("xb") as sink:
|
||||
shutil.copyfileobj(payload, sink, length=1024 * 1024)
|
||||
if target.stat().st_size != member.size:
|
||||
raise RuntimeError("panoptic mask archive member was truncated")
|
||||
|
||||
|
||||
def _camera_anchors(
|
||||
index_path: Path,
|
||||
frame_metadata: list[dict[str, Any]],
|
||||
*,
|
||||
origin_monotonic_ns: int,
|
||||
) -> list[CameraAnchor]:
|
||||
anchors: list[CameraAnchor] = []
|
||||
with index_path.open("rb") as stream:
|
||||
for sequence, frame in enumerate(frame_metadata, start=1):
|
||||
line = stream.readline(MAX_INDEX_LINE_BYTES + 1)
|
||||
if not line or len(line) > MAX_INDEX_LINE_BYTES or not line.endswith(b"\n"):
|
||||
raise RuntimeError("camera index is incomplete")
|
||||
value = json.loads(line)
|
||||
monotonic_ns = value.get("host_monotonic_ns") if isinstance(value, dict) else None
|
||||
video_time = frame.get("session_seconds")
|
||||
if (
|
||||
not isinstance(value, dict)
|
||||
or value.get("sequence") != sequence
|
||||
or not isinstance(monotonic_ns, int)
|
||||
or monotonic_ns < origin_monotonic_ns
|
||||
or not isinstance(video_time, (int, float))
|
||||
or not math.isfinite(float(video_time))
|
||||
):
|
||||
raise RuntimeError("camera anchor is invalid")
|
||||
anchors.append(
|
||||
CameraAnchor(
|
||||
sequence=sequence,
|
||||
host_session_seconds=(monotonic_ns - origin_monotonic_ns) / 1e9,
|
||||
video_session_seconds=float(video_time),
|
||||
)
|
||||
)
|
||||
if stream.read(1):
|
||||
raise RuntimeError("camera index has undeclared rows")
|
||||
return anchors
|
||||
|
||||
|
||||
def _lidar_samples(
|
||||
raw_path: Path,
|
||||
anchors: list[CameraAnchor],
|
||||
*,
|
||||
origin_monotonic_ns: int,
|
||||
) -> Iterator[LidarSample | None]:
|
||||
messages = iter(iter_replay_messages(raw_path))
|
||||
pending = next(messages, None)
|
||||
points: deque[tuple[float, LioPointCloudFrame]] = deque()
|
||||
poses: deque[tuple[float, LioPoseFrame]] = deque()
|
||||
|
||||
def message_time(message: Any) -> float:
|
||||
monotonic_ns = message.received_monotonic_ns
|
||||
if not isinstance(monotonic_ns, int) or monotonic_ns < origin_monotonic_ns:
|
||||
raise RuntimeError("MQTT replay message has no compatible monotonic time")
|
||||
return (monotonic_ns - origin_monotonic_ns) / 1e9
|
||||
|
||||
for anchor in anchors:
|
||||
target = anchor.host_session_seconds
|
||||
while pending is not None and message_time(pending) <= target + 0.5:
|
||||
timestamp = message_time(pending)
|
||||
if timestamp >= target - 0.5:
|
||||
if pending.topic.endswith("/lio_pcl"):
|
||||
points.append((timestamp, decode_lio_pcl(pending.payload)))
|
||||
elif pending.topic.endswith("/lio_pose"):
|
||||
poses.append((timestamp, decode_lio_pose(pending.payload)))
|
||||
pending = next(messages, None)
|
||||
while points and points[0][0] < target - 0.5:
|
||||
points.popleft()
|
||||
while poses and poses[0][0] < target - 0.5:
|
||||
poses.popleft()
|
||||
point_candidates = [
|
||||
item for item in points if abs(item[0] - target) <= MAX_SYNC_DELTA_SECONDS
|
||||
]
|
||||
if not point_candidates:
|
||||
yield None
|
||||
continue
|
||||
point_time, point_frame = min(point_candidates, key=lambda item: abs(item[0] - target))
|
||||
pose_candidates = [
|
||||
item for item in poses if abs(item[0] - point_time) <= MAX_SYNC_DELTA_SECONDS
|
||||
]
|
||||
if not pose_candidates:
|
||||
yield None
|
||||
continue
|
||||
pose_time, pose_frame = min(pose_candidates, key=lambda item: abs(item[0] - point_time))
|
||||
yield LidarSample(point_time, pose_time, point_frame, pose_frame)
|
||||
|
||||
|
||||
def _empty_points() -> npt.NDArray[np.float32]:
|
||||
return np.empty((0, 3), dtype=np.float32)
|
||||
|
||||
|
||||
def _empty_colors(channels: int) -> npt.NDArray[np.uint8]:
|
||||
return np.empty((0, channels), dtype=np.uint8)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = _arguments()
|
||||
started = time.perf_counter()
|
||||
result = validate_recorded_perception_epoch_result(args.job, args.result)
|
||||
session = args.session.resolve(strict=True)
|
||||
if session.name != result.job.session_id:
|
||||
raise RuntimeError("fusion session does not match the perception result")
|
||||
calibration, calibration_sha256 = _load_calibration_snapshot(
|
||||
args.calibration.resolve(strict=True)
|
||||
)
|
||||
if calibration_sha256 != result.calibration_sha256:
|
||||
raise RuntimeError("fusion calibration generation does not match perception")
|
||||
profile = Kb4ProjectionProfile.from_factory_calibration(calibration, result.job.source_id)
|
||||
frame_metadata = _read_frame_metadata(
|
||||
result.artifact("panoptic-frame-metadata").path,
|
||||
result.job.segment_count,
|
||||
)
|
||||
capture_root = session / "captures" / "mqtt_live"
|
||||
origin = read_capture_clock_origin(capture_root / "mqtt.timeline.origin.json")
|
||||
epoch_root = (
|
||||
result.job.job_root
|
||||
/ "input"
|
||||
/ "camera"
|
||||
/ result.job.source_id
|
||||
/ f"epoch-{result.job.codec_epoch}"
|
||||
)
|
||||
anchors = _camera_anchors(
|
||||
epoch_root / "index.jsonl",
|
||||
frame_metadata,
|
||||
origin_monotonic_ns=origin.started_monotonic_ns,
|
||||
)
|
||||
identity = {
|
||||
"schema_version": IDENTITY_SCHEMA,
|
||||
"job_id": result.job.job_id,
|
||||
"input_sha256": result.job.input_sha256,
|
||||
"perception_result_id": result.result_id,
|
||||
"calibration_sha256": calibration_sha256,
|
||||
"camera_slot": result.calibration_slot,
|
||||
"configuration": {
|
||||
"pipeline": "factory-kb4-mask-to-lidar/v1",
|
||||
"temporal_binding": "nearest-host-arrival-best-effort",
|
||||
"maximum_sync_delta_seconds": MAX_SYNC_DELTA_SECONDS,
|
||||
"semantic_sampling": "nearest-projected-pixel",
|
||||
"box_policy": "depth-clustered-map-axis-aligned-p05-p95-diagnostic",
|
||||
},
|
||||
}
|
||||
identity_sha256 = hashlib.sha256(_canonical_json(identity)).hexdigest()
|
||||
fusion_id = f"fusion-{identity_sha256}"
|
||||
parent = args.output_root.resolve() / result.job.job_id
|
||||
parent.mkdir(mode=0o700, parents=True, exist_ok=True)
|
||||
output = parent / fusion_id
|
||||
if output.exists():
|
||||
print(json.dumps({"fusion_id": fusion_id, "output": str(output), "reused": True}))
|
||||
return 0
|
||||
staging = parent / f".{fusion_id}.{os.getpid()}.incomplete"
|
||||
staging.mkdir(mode=0o700, exist_ok=False)
|
||||
masks_root = staging / "masks"
|
||||
masks_root.mkdir(mode=0o700)
|
||||
try:
|
||||
_extract_masks(
|
||||
result.artifact("panoptic-mask-archive").path,
|
||||
masks_root,
|
||||
result.job.segment_count,
|
||||
)
|
||||
frame_times_ns: list[int] = []
|
||||
point_offsets = [0]
|
||||
point_arrays: list[npt.NDArray[np.float32]] = []
|
||||
point_color_arrays: list[npt.NDArray[np.uint8]] = []
|
||||
box_offsets = [0]
|
||||
box_centers: list[tuple[float, float, float]] = []
|
||||
box_half_sizes: list[tuple[float, float, float]] = []
|
||||
box_colors: list[tuple[int, int, int, int]] = []
|
||||
box_labels: list[str] = []
|
||||
frame_reports: list[dict[str, Any]] = []
|
||||
compatible_frames = 0
|
||||
accepted_boxes = 0
|
||||
for index, (anchor, sample) in enumerate(
|
||||
zip(
|
||||
anchors,
|
||||
_lidar_samples(
|
||||
capture_root / "mqtt.raw.k1mqtt",
|
||||
anchors,
|
||||
origin_monotonic_ns=origin.started_monotonic_ns,
|
||||
),
|
||||
strict=True,
|
||||
)
|
||||
):
|
||||
frame_times_ns.append(round(anchor.video_session_seconds * 1e9))
|
||||
if sample is None:
|
||||
point_arrays.append(_empty_points())
|
||||
point_color_arrays.append(_empty_colors(3))
|
||||
point_offsets.append(point_offsets[-1])
|
||||
box_offsets.append(box_offsets[-1])
|
||||
frame_reports.append(
|
||||
{"frame_index": index, "state": "depth-unavailable", "points": 0, "boxes": 0}
|
||||
)
|
||||
continue
|
||||
compatible_frames += 1
|
||||
with Image.open(
|
||||
masks_root / "semantic-masks" / f"frame-{index + 1:06d}.png"
|
||||
) as semantic_image:
|
||||
semantic_map = np.array(semantic_image, dtype=np.uint8, copy=True)
|
||||
with Image.open(
|
||||
masks_root / "instance-masks" / f"frame-{index + 1:06d}.png"
|
||||
) as instance_image:
|
||||
instance_map = np.array(instance_image, dtype=np.uint16, copy=True)
|
||||
if semantic_map.shape != (profile.height, profile.width) or instance_map.shape != (
|
||||
profile.height,
|
||||
profile.width,
|
||||
):
|
||||
raise RuntimeError("panoptic mask dimensions changed")
|
||||
points_map = np.asarray(
|
||||
[
|
||||
point.scaled_xyz(sample.point_frame.header.scaler)
|
||||
for point in sample.point_frame.points
|
||||
],
|
||||
dtype=np.float64,
|
||||
)
|
||||
points_lidar = map_points_to_lidar(
|
||||
points_map,
|
||||
position_map_xyz=sample.pose_frame.position_xyz,
|
||||
orientation_map_from_lidar_xyzw=sample.pose_frame.orientation_xyzw,
|
||||
)
|
||||
projection = project_map_points_kb4(
|
||||
points_map,
|
||||
position_map_xyz=sample.pose_frame.position_xyz,
|
||||
orientation_map_from_lidar_xyzw=sample.pose_frame.orientation_xyzw,
|
||||
profile=profile,
|
||||
)
|
||||
pixel_x = np.clip(
|
||||
np.rint(projection.pixels_xy[:, 0]).astype(np.int64),
|
||||
0,
|
||||
profile.width - 1,
|
||||
)
|
||||
pixel_y = np.clip(
|
||||
np.rint(projection.pixels_xy[:, 1]).astype(np.int64),
|
||||
0,
|
||||
profile.height - 1,
|
||||
)
|
||||
semantic_ids = semantic_map[pixel_y, pixel_x].astype(np.int64)
|
||||
colors = np.asarray([_palette(int(value)) for value in semantic_ids], dtype=np.uint8)
|
||||
projected = points_map[projection.source_indices].astype(np.float32)
|
||||
point_arrays.append(projected)
|
||||
point_color_arrays.append(colors)
|
||||
point_offsets.append(point_offsets[-1] + projected.shape[0])
|
||||
instances = frame_metadata[index]["instances"]
|
||||
objects = _object_fusions(
|
||||
instance_map=instance_map,
|
||||
instances=instances,
|
||||
pixel_x=pixel_x,
|
||||
pixel_y=pixel_y,
|
||||
depths=projection.depths_m,
|
||||
projected_source_indices=projection.source_indices,
|
||||
points_map=points_map,
|
||||
points_lidar=points_lidar,
|
||||
)
|
||||
frame_box_count = 0
|
||||
for item in objects:
|
||||
if item.box_center_map is None or item.box_half_size_map is None:
|
||||
continue
|
||||
frame_box_count += 1
|
||||
accepted_boxes += 1
|
||||
box_centers.append(item.box_center_map)
|
||||
box_half_sizes.append(item.box_half_size_map)
|
||||
box_colors.append((*_palette(500 + item.instance_id), 96))
|
||||
box_labels.append(
|
||||
f"{item.label} · {item.distance_median_m:.1f} m · {item.clustered_points} pts"
|
||||
)
|
||||
box_offsets.append(box_offsets[-1] + frame_box_count)
|
||||
frame_reports.append(
|
||||
{
|
||||
"frame_index": index,
|
||||
"state": "fused",
|
||||
"points": int(projected.shape[0]),
|
||||
"boxes": frame_box_count,
|
||||
"lidar_camera_delta_ms": round(
|
||||
(sample.point_session_seconds - anchor.host_session_seconds) * 1000,
|
||||
6,
|
||||
),
|
||||
"pose_point_delta_ms": round(
|
||||
(sample.pose_session_seconds - sample.point_session_seconds) * 1000,
|
||||
6,
|
||||
),
|
||||
}
|
||||
)
|
||||
if (index + 1) % 250 == 0 or index + 1 == result.job.segment_count:
|
||||
print(
|
||||
json.dumps(
|
||||
{"phase": "fusion", "frames": index + 1, "total": result.job.segment_count}
|
||||
),
|
||||
flush=True,
|
||||
)
|
||||
arrays_path = staging / "fusion.npz"
|
||||
np.savez_compressed(
|
||||
arrays_path,
|
||||
frame_times_ns=np.asarray(frame_times_ns, dtype=np.int64),
|
||||
point_offsets=np.asarray(point_offsets, dtype=np.int64),
|
||||
points=(np.concatenate(point_arrays) if point_arrays else _empty_points()),
|
||||
point_colors=(
|
||||
np.concatenate(point_color_arrays) if point_color_arrays else _empty_colors(3)
|
||||
),
|
||||
box_offsets=np.asarray(box_offsets, dtype=np.int64),
|
||||
box_centers=np.asarray(box_centers, dtype=np.float32).reshape((-1, 3)),
|
||||
box_half_sizes=np.asarray(box_half_sizes, dtype=np.float32).reshape((-1, 3)),
|
||||
box_colors=np.asarray(box_colors, dtype=np.uint8).reshape((-1, 4)),
|
||||
)
|
||||
labels_path = staging / "box-labels.json"
|
||||
labels_path.write_text(
|
||||
json.dumps(box_labels, ensure_ascii=False, separators=(",", ":")),
|
||||
encoding="utf-8",
|
||||
)
|
||||
frames_path = staging / "fusion-frames.jsonl"
|
||||
frames_path.write_text(
|
||||
"".join(json.dumps(item, sort_keys=True) + "\n" for item in frame_reports),
|
||||
encoding="utf-8",
|
||||
)
|
||||
shutil.rmtree(masks_root)
|
||||
created_at = datetime.now(UTC).isoformat(timespec="milliseconds").replace("+00:00", "Z")
|
||||
artifacts = [
|
||||
{
|
||||
"name": path.name,
|
||||
"byte_length": path.stat().st_size,
|
||||
"sha256": _sha256(path),
|
||||
}
|
||||
for path in (arrays_path, labels_path, frames_path)
|
||||
]
|
||||
manifest = {
|
||||
"schema_version": SCHEMA,
|
||||
"fusion_id": fusion_id,
|
||||
"identity_sha256": identity_sha256,
|
||||
"identity": identity,
|
||||
"created_at_utc": created_at,
|
||||
"session_id": result.job.session_id,
|
||||
"source_id": result.job.source_id,
|
||||
"frame_count": result.job.segment_count,
|
||||
"timeline_start_seconds": result.job.timeline_start_seconds,
|
||||
"timeline_end_seconds": result.job.timeline_end_seconds,
|
||||
"metrics": {
|
||||
"frames_fused": compatible_frames,
|
||||
"frames_depth_unavailable": result.job.segment_count - compatible_frames,
|
||||
"semantic_points": point_offsets[-1],
|
||||
"accepted_diagnostic_boxes": accepted_boxes,
|
||||
"wall_seconds": round(time.perf_counter() - started, 6),
|
||||
},
|
||||
"acceptance": {
|
||||
"geometry": "factory-calibrated-kb4",
|
||||
"timing": "host-arrival-best-effort-not-ground-truthed",
|
||||
"distance": "diagnostic-not-ground-truthed",
|
||||
"boxes3d": "support-gated-diagnostic-not-tracked",
|
||||
"safety": "not-accepted",
|
||||
},
|
||||
"artifacts": artifacts,
|
||||
}
|
||||
(staging / "manifest.json").write_text(
|
||||
json.dumps(manifest, ensure_ascii=False, indent=2, sort_keys=True) + "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
os.replace(staging, output)
|
||||
except BaseException:
|
||||
shutil.rmtree(staging, ignore_errors=True)
|
||||
raise
|
||||
print(json.dumps({"fusion_id": fusion_id, "output": str(output), "reused": False}))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,307 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Build a full-camera-epoch LAB E10 LiDAR/pose replay pack from raw evidence."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import numpy as np
|
||||
from e10_fusion_runtime import PACK_SCHEMA, LidarReplayPack, canonical_json, sha256
|
||||
from fuse_e6_tracking_lidar import _camera_anchors, _lidar_samples, _read_profile
|
||||
|
||||
from k1link.compute import validate_tracked_fusion_qualification_result
|
||||
from k1link.device_plugins.xgrids_k1.analyze.calibrated_overlay import (
|
||||
_load_calibration_snapshot,
|
||||
)
|
||||
from k1link.device_plugins.xgrids_k1.analyze.calibrated_projection import (
|
||||
Kb4ProjectionProfile,
|
||||
)
|
||||
from k1link.device_plugins.xgrids_k1.mqtt.capture import read_capture_clock_origin
|
||||
|
||||
|
||||
def arguments() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--job", type=Path, required=True)
|
||||
parser.add_argument("--tracking", type=Path, required=True)
|
||||
parser.add_argument("--semantic", type=Path, required=True)
|
||||
parser.add_argument("--e6-result", type=Path, required=True)
|
||||
parser.add_argument("--e6-profile", type=Path, required=True)
|
||||
parser.add_argument("--session", type=Path, required=True)
|
||||
parser.add_argument("--calibration", type=Path, required=True)
|
||||
parser.add_argument("--output-root", type=Path, required=True)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def _read_e6_identity(result_root: Path) -> dict[str, Any]:
|
||||
document = json.loads(
|
||||
(result_root.resolve(strict=True) / "result.json").read_text(encoding="utf-8")
|
||||
)
|
||||
identity = document.get("identity") if isinstance(document, dict) else None
|
||||
if not isinstance(identity, dict):
|
||||
raise RuntimeError("accepted LAB E6 identity is unavailable")
|
||||
return identity
|
||||
|
||||
|
||||
def _read_full_timeline(path: Path, expected_count: int) -> list[dict[str, Any]]:
|
||||
frames: list[dict[str, Any]] = []
|
||||
previous_seconds = -1.0
|
||||
with path.resolve(strict=True).open(encoding="utf-8") as stream:
|
||||
for frame_index, line in enumerate(stream):
|
||||
value = json.loads(line)
|
||||
session_seconds = value.get("session_seconds") if isinstance(value, dict) else None
|
||||
if (
|
||||
not isinstance(value, dict)
|
||||
or value.get("frame_index") != frame_index
|
||||
or value.get("sequence") != frame_index + 1
|
||||
or not isinstance(session_seconds, (int, float))
|
||||
or isinstance(session_seconds, bool)
|
||||
or not float(session_seconds) > previous_seconds
|
||||
):
|
||||
raise RuntimeError("full camera timeline is invalid")
|
||||
frames.append(
|
||||
{
|
||||
"frame_index": frame_index,
|
||||
"source_frame_index": frame_index,
|
||||
"session_seconds": float(session_seconds),
|
||||
"tracks": [],
|
||||
}
|
||||
)
|
||||
previous_seconds = float(session_seconds)
|
||||
if len(frames) != expected_count:
|
||||
raise RuntimeError("full camera timeline count changed")
|
||||
return frames
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = arguments()
|
||||
upstream = validate_tracked_fusion_qualification_result(
|
||||
args.job,
|
||||
args.tracking,
|
||||
args.semantic,
|
||||
args.e6_result,
|
||||
)
|
||||
session = args.session.resolve(strict=True)
|
||||
if session.name != upstream.job.session_id:
|
||||
raise RuntimeError("LAB E10 full session differs from accepted E6")
|
||||
e6_profile, e6_profile_sha256 = _read_profile(args.e6_profile)
|
||||
e6_identity = _read_e6_identity(upstream.result_root)
|
||||
if (
|
||||
e6_identity.get("profile_sha256") != e6_profile_sha256
|
||||
or e6_identity.get("configuration") != e6_profile
|
||||
):
|
||||
raise RuntimeError("LAB E10 full pack temporal policy differs from accepted E6")
|
||||
calibration, calibration_sha256 = _load_calibration_snapshot(
|
||||
args.calibration.resolve(strict=True)
|
||||
)
|
||||
if calibration_sha256 != upstream.semantic.calibration_sha256:
|
||||
raise RuntimeError("LAB E10 full pack calibration differs from accepted E6")
|
||||
projection = Kb4ProjectionProfile.from_factory_calibration(
|
||||
calibration,
|
||||
upstream.job.source_id,
|
||||
)
|
||||
timeline_frames = _read_full_timeline(
|
||||
upstream.semantic.artifact("panoptic-frame-metadata").path,
|
||||
upstream.job.segment_count,
|
||||
)
|
||||
capture_root = session / "captures" / "mqtt_live"
|
||||
origin = read_capture_clock_origin(capture_root / "mqtt.timeline.origin.json")
|
||||
epoch_root = (
|
||||
upstream.job.job_root
|
||||
/ "input"
|
||||
/ "camera"
|
||||
/ upstream.job.source_id
|
||||
/ f"epoch-{upstream.job.codec_epoch}"
|
||||
)
|
||||
anchors = _camera_anchors(
|
||||
epoch_root / "index.jsonl",
|
||||
timeline_frames,
|
||||
origin_monotonic_ns=origin.started_monotonic_ns,
|
||||
)
|
||||
maximum_lidar_delta_s = float(e6_profile["temporal"]["maximum_lidar_camera_delta_ms"]) / 1000.0
|
||||
maximum_pose_delta_s = float(e6_profile["temporal"]["maximum_pose_point_delta_ms"]) / 1000.0
|
||||
samples = _lidar_samples(
|
||||
capture_root / "mqtt.raw.k1mqtt",
|
||||
anchors,
|
||||
origin_monotonic_ns=origin.started_monotonic_ns,
|
||||
maximum_lidar_camera_delta_s=maximum_lidar_delta_s,
|
||||
maximum_pose_point_delta_s=maximum_pose_delta_s,
|
||||
)
|
||||
|
||||
count = len(anchors)
|
||||
available = np.zeros((count,), dtype=np.bool_)
|
||||
offsets = [0]
|
||||
clouds: list[np.ndarray] = []
|
||||
positions = np.full((count, 3), np.nan, dtype=np.float64)
|
||||
quaternions = np.full((count, 4), np.nan, dtype=np.float64)
|
||||
lidar_delta = np.full((count,), np.nan, dtype=np.float64)
|
||||
pose_delta = np.full((count,), np.nan, dtype=np.float64)
|
||||
for index, (anchor, sample) in enumerate(zip(anchors, samples, strict=True)):
|
||||
if sample is None:
|
||||
offsets.append(offsets[-1])
|
||||
else:
|
||||
cloud = np.asarray(
|
||||
[
|
||||
point.scaled_xyz(sample.point_frame.header.scaler)
|
||||
for point in sample.point_frame.points
|
||||
],
|
||||
dtype=np.float32,
|
||||
).reshape((-1, 3))
|
||||
available[index] = True
|
||||
clouds.append(cloud)
|
||||
offsets.append(offsets[-1] + cloud.shape[0])
|
||||
positions[index] = sample.pose_frame.position_xyz
|
||||
quaternions[index] = sample.pose_frame.orientation_xyzw
|
||||
lidar_delta[index] = (
|
||||
sample.point_session_seconds - anchor.host_session_seconds
|
||||
) * 1000.0
|
||||
pose_delta[index] = (
|
||||
sample.pose_session_seconds - sample.point_session_seconds
|
||||
) * 1000.0
|
||||
if (index + 1) % 500 == 0 or index + 1 == count:
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"phase": "full-lidar-pack",
|
||||
"frames": index + 1,
|
||||
"total": count,
|
||||
"lidar_frames": int(available[: index + 1].sum()),
|
||||
"points": offsets[-1],
|
||||
},
|
||||
sort_keys=True,
|
||||
),
|
||||
flush=True,
|
||||
)
|
||||
|
||||
source_indices = np.arange(count, dtype=np.int64)
|
||||
session_seconds = np.asarray(
|
||||
[anchor.video_session_seconds for anchor in anchors], dtype=np.float64
|
||||
)
|
||||
temporal = {
|
||||
"binding": e6_profile["temporal"]["binding"],
|
||||
"maximum_lidar_camera_delta_ms": float(
|
||||
e6_profile["temporal"]["maximum_lidar_camera_delta_ms"]
|
||||
),
|
||||
"maximum_pose_point_delta_ms": float(e6_profile["temporal"]["maximum_pose_point_delta_ms"]),
|
||||
"clock_source": "recorded-host-monotonic-arrival",
|
||||
}
|
||||
identity = {
|
||||
"schema_version": PACK_SCHEMA,
|
||||
"job_id": upstream.job.job_id,
|
||||
"input_sha256": upstream.job.input_sha256,
|
||||
"session_id": upstream.job.session_id,
|
||||
"source_id": upstream.job.source_id,
|
||||
"camera_slot": upstream.semantic.calibration_slot,
|
||||
"calibration_sha256": calibration_sha256,
|
||||
"e6_result_id": upstream.result_id,
|
||||
"e6_profile_sha256": e6_profile_sha256,
|
||||
"semantic_timeline_result_id": upstream.semantic.result_id,
|
||||
"frame_count": count,
|
||||
"source_start_frame_index": 0,
|
||||
"source_end_frame_index": count - 1,
|
||||
"timeline_start_seconds": float(session_seconds[0]),
|
||||
"timeline_end_seconds": float(session_seconds[-1]),
|
||||
"available_lidar_frames": int(available.sum()),
|
||||
"point_count": int(offsets[-1]),
|
||||
"temporal_binding": "accepted-e6-nearest-host-arrival-best-effort",
|
||||
"temporal_policy": temporal,
|
||||
"projection": {
|
||||
"model": "kb4",
|
||||
"width": projection.width,
|
||||
"height": projection.height,
|
||||
"source_coordinates": "k1-map",
|
||||
"target_camera": upstream.job.source_id,
|
||||
},
|
||||
"producer_sha256": sha256(Path(__file__).resolve(strict=True)),
|
||||
}
|
||||
identity_sha256 = hashlib.sha256(canonical_json(identity)).hexdigest()
|
||||
pack_id = f"e10-lidar-pack-{identity_sha256}"
|
||||
parent = args.output_root.resolve()
|
||||
parent.mkdir(mode=0o700, parents=True, exist_ok=True)
|
||||
output = parent / pack_id
|
||||
if output.exists():
|
||||
print(json.dumps({"pack_id": pack_id, "output": str(output), "reused": True}))
|
||||
return 0
|
||||
staging = parent / f".{pack_id}.{os.getpid()}.incomplete"
|
||||
staging.mkdir(mode=0o700, exist_ok=False)
|
||||
try:
|
||||
arrays_path = staging / "lidar-pack.npz"
|
||||
np.savez_compressed(
|
||||
arrays_path,
|
||||
frame_indices=np.arange(count, dtype=np.int64),
|
||||
source_frame_indices=source_indices,
|
||||
session_seconds=session_seconds,
|
||||
sample_available=available,
|
||||
cloud_offsets=np.asarray(offsets, dtype=np.int64),
|
||||
cloud_points_map=(
|
||||
np.concatenate(clouds) if clouds else np.empty((0, 3), dtype=np.float32)
|
||||
),
|
||||
pose_positions_map=positions,
|
||||
pose_quaternions_map_from_lidar=quaternions,
|
||||
lidar_camera_delta_ms=lidar_delta,
|
||||
pose_point_delta_ms=pose_delta,
|
||||
intrinsic_fx_fy_cx_cy=np.asarray(projection.intrinsic_fx_fy_cx_cy, dtype=np.float64),
|
||||
distortion_kb4=np.asarray(projection.distortion_kb4, dtype=np.float64),
|
||||
t_camera_from_lidar=np.asarray(projection.t_camera_from_lidar, dtype=np.float64),
|
||||
)
|
||||
manifest = {
|
||||
"schema_version": PACK_SCHEMA,
|
||||
"pack_id": pack_id,
|
||||
"identity_sha256": identity_sha256,
|
||||
"identity": identity,
|
||||
"created_at_utc": datetime.now(UTC)
|
||||
.isoformat(timespec="milliseconds")
|
||||
.replace("+00:00", "Z"),
|
||||
"classification": "private-recorded-sensor-replay-input",
|
||||
"ground_truth": False,
|
||||
"artifact": {
|
||||
"path": arrays_path.name,
|
||||
"media_type": "application/x-npz",
|
||||
"byte_length": arrays_path.stat().st_size,
|
||||
"sha256": sha256(arrays_path),
|
||||
},
|
||||
}
|
||||
(staging / "manifest.json").write_bytes(
|
||||
json.dumps(
|
||||
manifest,
|
||||
ensure_ascii=False,
|
||||
sort_keys=True,
|
||||
indent=2,
|
||||
allow_nan=False,
|
||||
).encode()
|
||||
+ b"\n"
|
||||
)
|
||||
os.replace(staging, output)
|
||||
try:
|
||||
validation = LidarReplayPack(output, expected_job_id=upstream.job.job_id)
|
||||
validation.close()
|
||||
except BaseException:
|
||||
shutil.rmtree(output, ignore_errors=True)
|
||||
raise
|
||||
except BaseException:
|
||||
shutil.rmtree(staging, ignore_errors=True)
|
||||
raise
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"pack_id": pack_id,
|
||||
"output": str(output),
|
||||
"frames": count,
|
||||
"lidar_frames": int(available.sum()),
|
||||
"points": int(offsets[-1]),
|
||||
"byte_length": (output / "lidar-pack.npz").stat().st_size,
|
||||
},
|
||||
sort_keys=True,
|
||||
)
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,211 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Build a content-addressed LAB E10 LiDAR/pose replay pack from accepted E6."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import sys
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
from types import ModuleType
|
||||
|
||||
import numpy as np
|
||||
from e10_fusion_runtime import PACK_SCHEMA, canonical_json, sha256
|
||||
|
||||
from k1link.compute import validate_tracked_fusion_qualification_result
|
||||
from k1link.device_plugins.xgrids_k1.analyze.calibrated_overlay import (
|
||||
_load_calibration_snapshot,
|
||||
)
|
||||
from k1link.device_plugins.xgrids_k1.analyze.calibrated_projection import (
|
||||
Kb4ProjectionProfile,
|
||||
)
|
||||
|
||||
|
||||
def arguments() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--job", type=Path, required=True)
|
||||
parser.add_argument("--tracking", type=Path, required=True)
|
||||
parser.add_argument("--semantic", type=Path, required=True)
|
||||
parser.add_argument("--e6-result", type=Path, required=True)
|
||||
parser.add_argument("--session", type=Path, required=True)
|
||||
parser.add_argument("--calibration", type=Path, required=True)
|
||||
parser.add_argument("--output-root", type=Path, required=True)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def load_e7_runner() -> ModuleType:
|
||||
path = Path(__file__).with_name("run_e7_replay_realtime.py").resolve(strict=True)
|
||||
spec = importlib.util.spec_from_file_location("missioncore_e10_e7_adapter", path)
|
||||
if spec is None or spec.loader is None:
|
||||
raise RuntimeError("LAB E7 adapter cannot be loaded")
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
sys.modules[spec.name] = module
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = arguments()
|
||||
upstream = validate_tracked_fusion_qualification_result(
|
||||
args.job,
|
||||
args.tracking,
|
||||
args.semantic,
|
||||
args.e6_result,
|
||||
)
|
||||
session = args.session.resolve(strict=True)
|
||||
if session.name != upstream.job.session_id:
|
||||
raise RuntimeError("LAB E10 session differs from the accepted E6 input")
|
||||
calibration, calibration_sha256 = _load_calibration_snapshot(
|
||||
args.calibration.resolve(strict=True)
|
||||
)
|
||||
if calibration_sha256 != upstream.semantic.calibration_sha256:
|
||||
raise RuntimeError("LAB E10 calibration differs from the accepted E6 input")
|
||||
projection = Kb4ProjectionProfile.from_factory_calibration(
|
||||
calibration,
|
||||
upstream.job.source_id,
|
||||
)
|
||||
adapter = load_e7_runner()
|
||||
frames = adapter._prepare_frames(
|
||||
e6=upstream,
|
||||
session=session,
|
||||
selection_start=0,
|
||||
selection_count=upstream.frame_count,
|
||||
)
|
||||
if len(frames) != upstream.frame_count:
|
||||
raise RuntimeError("LAB E10 E6 replay frame count changed")
|
||||
|
||||
frame_indices = np.arange(len(frames), dtype=np.int64)
|
||||
source_indices = np.asarray(
|
||||
[int(frame.frame["source_frame_index"]) for frame in frames], dtype=np.int64
|
||||
)
|
||||
session_seconds = np.asarray(
|
||||
[float(frame.frame["session_seconds"]) for frame in frames], dtype=np.float64
|
||||
)
|
||||
available = np.asarray([frame.position_map_xyz is not None for frame in frames], dtype=np.bool_)
|
||||
offsets = [0]
|
||||
clouds = []
|
||||
positions = np.full((len(frames), 3), np.nan, dtype=np.float64)
|
||||
quaternions = np.full((len(frames), 4), np.nan, dtype=np.float64)
|
||||
lidar_delta = np.full((len(frames),), np.nan, dtype=np.float64)
|
||||
pose_delta = np.full((len(frames),), np.nan, dtype=np.float64)
|
||||
for index, frame in enumerate(frames):
|
||||
cloud = np.asarray(frame.cloud_map, dtype=np.float32)
|
||||
clouds.append(cloud)
|
||||
offsets.append(offsets[-1] + cloud.shape[0])
|
||||
if frame.position_map_xyz is not None and frame.orientation_map_from_lidar_xyzw is not None:
|
||||
positions[index] = frame.position_map_xyz
|
||||
quaternions[index] = frame.orientation_map_from_lidar_xyzw
|
||||
lidar_delta[index] = float(frame.frame["lidar_camera_delta_ms"])
|
||||
pose_delta[index] = float(frame.frame["pose_point_delta_ms"])
|
||||
|
||||
identity = {
|
||||
"schema_version": PACK_SCHEMA,
|
||||
"job_id": upstream.job.job_id,
|
||||
"input_sha256": upstream.job.input_sha256,
|
||||
"session_id": upstream.job.session_id,
|
||||
"source_id": upstream.job.source_id,
|
||||
"camera_slot": upstream.semantic.calibration_slot,
|
||||
"calibration_sha256": calibration_sha256,
|
||||
"e6_result_id": upstream.result_id,
|
||||
"frame_count": len(frames),
|
||||
"source_start_frame_index": int(source_indices[0]),
|
||||
"source_end_frame_index": int(source_indices[-1]),
|
||||
"timeline_start_seconds": float(session_seconds[0]),
|
||||
"timeline_end_seconds": float(session_seconds[-1]),
|
||||
"available_lidar_frames": int(available.sum()),
|
||||
"point_count": int(offsets[-1]),
|
||||
"temporal_binding": "accepted-e6-nearest-host-arrival-best-effort",
|
||||
"projection": {
|
||||
"model": "kb4",
|
||||
"width": projection.width,
|
||||
"height": projection.height,
|
||||
"source_coordinates": "k1-map",
|
||||
"target_camera": upstream.job.source_id,
|
||||
},
|
||||
"producer_sha256": sha256(Path(__file__).resolve(strict=True)),
|
||||
}
|
||||
identity_sha256 = hashlib.sha256(canonical_json(identity)).hexdigest()
|
||||
pack_id = f"e10-lidar-pack-{identity_sha256}"
|
||||
parent = args.output_root.resolve()
|
||||
parent.mkdir(mode=0o700, parents=True, exist_ok=True)
|
||||
output = parent / pack_id
|
||||
if output.exists():
|
||||
print(json.dumps({"pack_id": pack_id, "output": str(output), "reused": True}))
|
||||
return 0
|
||||
staging = parent / f".{pack_id}.{os.getpid()}.incomplete"
|
||||
staging.mkdir(mode=0o700, exist_ok=False)
|
||||
try:
|
||||
arrays_path = staging / "lidar-pack.npz"
|
||||
np.savez_compressed(
|
||||
arrays_path,
|
||||
frame_indices=frame_indices,
|
||||
source_frame_indices=source_indices,
|
||||
session_seconds=session_seconds,
|
||||
sample_available=available,
|
||||
cloud_offsets=np.asarray(offsets, dtype=np.int64),
|
||||
cloud_points_map=(
|
||||
np.concatenate(clouds) if clouds else np.empty((0, 3), dtype=np.float32)
|
||||
),
|
||||
pose_positions_map=positions,
|
||||
pose_quaternions_map_from_lidar=quaternions,
|
||||
lidar_camera_delta_ms=lidar_delta,
|
||||
pose_point_delta_ms=pose_delta,
|
||||
intrinsic_fx_fy_cx_cy=np.asarray(projection.intrinsic_fx_fy_cx_cy, dtype=np.float64),
|
||||
distortion_kb4=np.asarray(projection.distortion_kb4, dtype=np.float64),
|
||||
t_camera_from_lidar=np.asarray(projection.t_camera_from_lidar, dtype=np.float64),
|
||||
)
|
||||
manifest = {
|
||||
"schema_version": PACK_SCHEMA,
|
||||
"pack_id": pack_id,
|
||||
"identity_sha256": identity_sha256,
|
||||
"identity": identity,
|
||||
"created_at_utc": datetime.now(UTC)
|
||||
.isoformat(timespec="milliseconds")
|
||||
.replace("+00:00", "Z"),
|
||||
"classification": "private-recorded-sensor-replay-input",
|
||||
"ground_truth": False,
|
||||
"artifact": {
|
||||
"path": arrays_path.name,
|
||||
"media_type": "application/x-npz",
|
||||
"byte_length": arrays_path.stat().st_size,
|
||||
"sha256": sha256(arrays_path),
|
||||
},
|
||||
}
|
||||
manifest_path = staging / "manifest.json"
|
||||
manifest_path.write_bytes(
|
||||
json.dumps(
|
||||
manifest,
|
||||
ensure_ascii=False,
|
||||
sort_keys=True,
|
||||
indent=2,
|
||||
allow_nan=False,
|
||||
).encode()
|
||||
+ b"\n"
|
||||
)
|
||||
os.replace(staging, output)
|
||||
except BaseException:
|
||||
shutil.rmtree(staging, ignore_errors=True)
|
||||
raise
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"pack_id": pack_id,
|
||||
"output": str(output),
|
||||
"frames": len(frames),
|
||||
"lidar_frames": int(available.sum()),
|
||||
"points": int(offsets[-1]),
|
||||
"byte_length": (output / "lidar-pack.npz").stat().st_size,
|
||||
},
|
||||
sort_keys=True,
|
||||
)
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,195 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Extract a minimal content-addressed live projection pack from accepted E14 input."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import numpy as np
|
||||
|
||||
SCHEMA = "missioncore.e15-live-projection-pack/v1"
|
||||
PACK_ARRAYS = {
|
||||
"intrinsic_fx_fy_cx_cy",
|
||||
"distortion_kb4",
|
||||
"t_camera_from_lidar",
|
||||
}
|
||||
|
||||
|
||||
def _canonical(value: object) -> bytes:
|
||||
return json.dumps(
|
||||
value,
|
||||
ensure_ascii=False,
|
||||
sort_keys=True,
|
||||
separators=(",", ":"),
|
||||
allow_nan=False,
|
||||
).encode()
|
||||
|
||||
|
||||
def _sha256(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as stream:
|
||||
while block := stream.read(1024 * 1024):
|
||||
digest.update(block)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def _read_object(path: Path) -> dict[str, Any]:
|
||||
value = json.loads(path.read_text(encoding="utf-8"))
|
||||
if not isinstance(value, dict):
|
||||
raise RuntimeError("projection source manifest is invalid")
|
||||
return value
|
||||
|
||||
|
||||
def build_projection_pack(
|
||||
source_root: Path,
|
||||
output_root: Path,
|
||||
*,
|
||||
expected_calibration_sha256: str,
|
||||
) -> Path:
|
||||
source = source_root.expanduser().resolve(strict=True)
|
||||
manifest_path = source / "manifest.json"
|
||||
arrays_path = source / "lidar-pack.npz"
|
||||
manifest = _read_object(manifest_path)
|
||||
identity = manifest.get("identity")
|
||||
if (
|
||||
not isinstance(identity, dict)
|
||||
or identity.get("calibration_sha256") != expected_calibration_sha256
|
||||
or identity.get("source_id") != "sensor.camera.right"
|
||||
or identity.get("camera_slot") != "camera_1"
|
||||
or manifest.get("artifact", {}).get("sha256") != _sha256(arrays_path)
|
||||
):
|
||||
raise RuntimeError("accepted LiDAR pack calibration binding changed")
|
||||
with np.load(arrays_path, allow_pickle=False) as source_arrays:
|
||||
if not PACK_ARRAYS.issubset(source_arrays.files):
|
||||
raise RuntimeError("accepted LiDAR pack projection arrays changed")
|
||||
intrinsic = np.asarray(source_arrays["intrinsic_fx_fy_cx_cy"], dtype=np.float64)
|
||||
distortion = np.asarray(source_arrays["distortion_kb4"], dtype=np.float64)
|
||||
transform = np.asarray(source_arrays["t_camera_from_lidar"], dtype=np.float64)
|
||||
if (
|
||||
intrinsic.shape != (4,)
|
||||
or distortion.shape != (4,)
|
||||
or transform.shape != (4, 4)
|
||||
or not np.isfinite(intrinsic).all()
|
||||
or not np.isfinite(distortion).all()
|
||||
or not np.isfinite(transform).all()
|
||||
):
|
||||
raise RuntimeError("live projection arrays are invalid")
|
||||
|
||||
pack_identity = {
|
||||
"schema_version": SCHEMA,
|
||||
"source_id": "sensor.camera.right",
|
||||
"calibration_slot": "camera_1",
|
||||
"calibration_sha256": expected_calibration_sha256,
|
||||
"resolution": [800, 600],
|
||||
"projection_model": "kb4",
|
||||
"coordinate_contract": "k1-map-to-camera_1-via-inverse-map-pose",
|
||||
"source_lidar_pack_id": manifest.get("pack_id"),
|
||||
"source_lidar_pack_identity_sha256": manifest.get("identity_sha256"),
|
||||
"producer_sha256": _sha256(Path(__file__).resolve(strict=True)),
|
||||
}
|
||||
identity_sha256 = hashlib.sha256(_canonical(pack_identity)).hexdigest()
|
||||
pack_id = f"e15-live-projection-{identity_sha256}"
|
||||
parent = output_root.expanduser().resolve()
|
||||
parent.mkdir(mode=0o700, parents=True, exist_ok=True)
|
||||
destination = parent / pack_id
|
||||
if destination.exists():
|
||||
validate_projection_pack(destination, expected_calibration_sha256)
|
||||
return destination
|
||||
staging = parent / f".{pack_id}.{os.getpid()}.incomplete"
|
||||
staging.mkdir(mode=0o700, exist_ok=False)
|
||||
try:
|
||||
output_arrays = staging / "projection.npz"
|
||||
np.savez(
|
||||
output_arrays,
|
||||
intrinsic_fx_fy_cx_cy=intrinsic,
|
||||
distortion_kb4=distortion,
|
||||
t_camera_from_lidar=transform,
|
||||
)
|
||||
document = {
|
||||
"schema_version": SCHEMA,
|
||||
"pack_id": pack_id,
|
||||
"identity_sha256": identity_sha256,
|
||||
"identity": pack_identity,
|
||||
"created_at_utc": datetime.now(UTC)
|
||||
.isoformat(timespec="milliseconds")
|
||||
.replace("+00:00", "Z"),
|
||||
"classification": "calibration-only-no-recorded-sensor-frames",
|
||||
"artifact": {
|
||||
"path": output_arrays.name,
|
||||
"byte_length": output_arrays.stat().st_size,
|
||||
"sha256": _sha256(output_arrays),
|
||||
},
|
||||
}
|
||||
(staging / "manifest.json").write_bytes(
|
||||
json.dumps(document, indent=2, sort_keys=True).encode() + b"\n"
|
||||
)
|
||||
os.replace(staging, destination)
|
||||
except BaseException:
|
||||
shutil.rmtree(staging, ignore_errors=True)
|
||||
raise
|
||||
validate_projection_pack(destination, expected_calibration_sha256)
|
||||
return destination
|
||||
|
||||
|
||||
def validate_projection_pack(root: Path, expected_calibration_sha256: str) -> dict[str, Any]:
|
||||
resolved = root.expanduser().resolve(strict=True)
|
||||
manifest = _read_object(resolved / "manifest.json")
|
||||
identity = manifest.get("identity")
|
||||
identity_sha256 = manifest.get("identity_sha256")
|
||||
artifact = manifest.get("artifact")
|
||||
arrays_path = resolved / "projection.npz"
|
||||
if (
|
||||
manifest.get("schema_version") != SCHEMA
|
||||
or not isinstance(identity, dict)
|
||||
or identity.get("schema_version") != SCHEMA
|
||||
or identity.get("calibration_sha256") != expected_calibration_sha256
|
||||
or not isinstance(identity_sha256, str)
|
||||
or hashlib.sha256(_canonical(identity)).hexdigest() != identity_sha256
|
||||
or resolved.name != f"e15-live-projection-{identity_sha256}"
|
||||
or manifest.get("pack_id") != resolved.name
|
||||
or not isinstance(artifact, dict)
|
||||
or artifact.get("path") != arrays_path.name
|
||||
or artifact.get("byte_length") != arrays_path.stat().st_size
|
||||
or artifact.get("sha256") != _sha256(arrays_path)
|
||||
):
|
||||
raise RuntimeError("live projection pack identity is invalid")
|
||||
with np.load(arrays_path, allow_pickle=False) as arrays:
|
||||
if set(arrays.files) != PACK_ARRAYS:
|
||||
raise RuntimeError("live projection pack arrays changed")
|
||||
if (
|
||||
arrays["intrinsic_fx_fy_cx_cy"].shape != (4,)
|
||||
or arrays["distortion_kb4"].shape != (4,)
|
||||
or arrays["t_camera_from_lidar"].shape != (4, 4)
|
||||
):
|
||||
raise RuntimeError("live projection pack shapes changed")
|
||||
return manifest
|
||||
|
||||
|
||||
def _arguments() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--source-root", type=Path, required=True)
|
||||
parser.add_argument("--output-root", type=Path, required=True)
|
||||
parser.add_argument("--calibration-sha256", required=True)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = _arguments()
|
||||
output = build_projection_pack(
|
||||
args.source_root,
|
||||
args.output_root,
|
||||
expected_calibration_sha256=args.calibration_sha256,
|
||||
)
|
||||
print(json.dumps({"projection_pack": str(output)}, sort_keys=True))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,199 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Build the minimal hash-addressed K1 runtime imported by the E15 worker."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import uuid
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
SCHEMA = "missioncore.e15-worker-package/v1"
|
||||
COPIED_FILES = (
|
||||
"k1link/__init__.py",
|
||||
"k1link/compute/live_perception.py",
|
||||
"k1link/data_plane/__init__.py",
|
||||
"k1link/data_plane/views.py",
|
||||
"k1link/device_plugins/__init__.py",
|
||||
"k1link/device_plugins/xgrids_k1/protocol/__init__.py",
|
||||
"k1link/device_plugins/xgrids_k1/protocol/normalizer.py",
|
||||
"k1link/device_plugins/xgrids_k1/protocol/protobuf_wire.py",
|
||||
"k1link/device_plugins/xgrids_k1/protocol/streams.py",
|
||||
)
|
||||
GENERATED_FILES = {
|
||||
"k1link/compute/__init__.py": (
|
||||
'"""Minimal E15 worker projection; import live_perception explicitly."""\n'
|
||||
),
|
||||
"k1link/device_plugins/xgrids_k1/__init__.py": (
|
||||
'"""Minimal E15 K1 decoder projection; no device-runtime side effects."""\n'
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def _canonical(value: object) -> bytes:
|
||||
return json.dumps(
|
||||
value,
|
||||
sort_keys=True,
|
||||
separators=(",", ":"),
|
||||
allow_nan=False,
|
||||
).encode()
|
||||
|
||||
|
||||
def _sha256(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as stream:
|
||||
while chunk := stream.read(1024 * 1024):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def _identity(source_root: Path) -> dict[str, Any]:
|
||||
source_files = []
|
||||
for relative in COPIED_FILES:
|
||||
source = source_root / relative
|
||||
if not source.is_file() or source.is_symlink():
|
||||
raise RuntimeError(f"E15 worker source is not a regular file: {relative}")
|
||||
source_files.append({"path": relative, "sha256": _sha256(source)})
|
||||
return {
|
||||
"schema_version": SCHEMA,
|
||||
"classification": "minimal-live-worker-import-projection",
|
||||
"source_files": source_files,
|
||||
"generated_initializers": {
|
||||
path: hashlib.sha256(content.encode()).hexdigest()
|
||||
for path, content in sorted(GENERATED_FILES.items())
|
||||
},
|
||||
"imports": [
|
||||
"k1link.compute.live_perception.LiveSensorSynchronizer",
|
||||
"k1link.data_plane.DecodedPointCloudView",
|
||||
"k1link.data_plane.DecodedPoseView",
|
||||
"k1link.device_plugins.xgrids_k1.protocol.normalizer.normalize_k1_message",
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def validate_worker_package(root: Path, *, allow_staging: bool = False) -> dict[str, Any]:
|
||||
resolved = root.resolve(strict=True)
|
||||
manifest_path = resolved / "manifest.json"
|
||||
manifest = json.loads(manifest_path.read_text(encoding="utf-8-sig"))
|
||||
identity = manifest.get("identity")
|
||||
identity_sha256 = manifest.get("identity_sha256")
|
||||
artifacts = manifest.get("artifacts")
|
||||
expected_paths = {*COPIED_FILES, *GENERATED_FILES, "manifest.json"}
|
||||
actual_paths = {
|
||||
path.relative_to(resolved).as_posix()
|
||||
for path in resolved.rglob("*")
|
||||
if path.is_file()
|
||||
}
|
||||
if (
|
||||
manifest.get("schema_version") != SCHEMA
|
||||
or not isinstance(identity, dict)
|
||||
or identity.get("schema_version") != SCHEMA
|
||||
or not isinstance(identity_sha256, str)
|
||||
or hashlib.sha256(_canonical(identity)).hexdigest() != identity_sha256
|
||||
or manifest.get("package_id") != f"e15-worker-package-{identity_sha256}"
|
||||
or (
|
||||
resolved.name != manifest.get("package_id")
|
||||
and not (
|
||||
allow_staging
|
||||
and resolved.name.startswith(f".{manifest.get('package_id')}.")
|
||||
and resolved.name.endswith(".tmp")
|
||||
)
|
||||
)
|
||||
or not isinstance(artifacts, list)
|
||||
or actual_paths != expected_paths
|
||||
):
|
||||
raise RuntimeError("E15 worker package identity is invalid")
|
||||
artifact_paths = set()
|
||||
for artifact in artifacts:
|
||||
if not isinstance(artifact, dict):
|
||||
raise RuntimeError("E15 worker package artifact is invalid")
|
||||
relative = str(artifact.get("path", ""))
|
||||
path = resolved / relative
|
||||
if (
|
||||
relative not in expected_paths - {"manifest.json"}
|
||||
or relative in artifact_paths
|
||||
or not path.is_file()
|
||||
or path.is_symlink()
|
||||
or artifact.get("byte_length") != path.stat().st_size
|
||||
or artifact.get("sha256") != _sha256(path)
|
||||
):
|
||||
raise RuntimeError("E15 worker package artifact changed")
|
||||
artifact_paths.add(relative)
|
||||
if artifact_paths != expected_paths - {"manifest.json"}:
|
||||
raise RuntimeError("E15 worker package artifact set changed")
|
||||
return manifest
|
||||
|
||||
|
||||
def build_worker_package(source_root: Path, output_root: Path) -> Path:
|
||||
source = source_root.resolve(strict=True)
|
||||
output = output_root.resolve()
|
||||
output.mkdir(mode=0o700, parents=True, exist_ok=True)
|
||||
identity = _identity(source)
|
||||
identity_sha256 = hashlib.sha256(_canonical(identity)).hexdigest()
|
||||
package_id = f"e15-worker-package-{identity_sha256}"
|
||||
destination = output / package_id
|
||||
if destination.exists():
|
||||
validate_worker_package(destination)
|
||||
return destination
|
||||
staging = output / f".{package_id}.{uuid.uuid4().hex}.tmp"
|
||||
staging.mkdir(mode=0o700)
|
||||
try:
|
||||
for relative in COPIED_FILES:
|
||||
target = staging / relative
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
shutil.copyfile(source / relative, target)
|
||||
for relative, content in GENERATED_FILES.items():
|
||||
target = staging / relative
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
target.write_text(content, encoding="utf-8")
|
||||
artifacts = []
|
||||
for relative in sorted((*COPIED_FILES, *GENERATED_FILES)):
|
||||
path = staging / relative
|
||||
artifacts.append(
|
||||
{
|
||||
"path": relative,
|
||||
"byte_length": path.stat().st_size,
|
||||
"sha256": _sha256(path),
|
||||
}
|
||||
)
|
||||
manifest = {
|
||||
"schema_version": SCHEMA,
|
||||
"package_id": package_id,
|
||||
"identity_sha256": identity_sha256,
|
||||
"identity": identity,
|
||||
"created_at_utc": datetime.now(UTC)
|
||||
.isoformat(timespec="milliseconds")
|
||||
.replace("+00:00", "Z"),
|
||||
"artifacts": artifacts,
|
||||
}
|
||||
with (staging / "manifest.json").open("x", encoding="utf-8", newline="\n") as stream:
|
||||
json.dump(manifest, stream, indent=2, sort_keys=True)
|
||||
stream.write("\n")
|
||||
stream.flush()
|
||||
os.fsync(stream.fileno())
|
||||
validate_worker_package(staging, allow_staging=True)
|
||||
staging.replace(destination)
|
||||
finally:
|
||||
if staging.exists():
|
||||
shutil.rmtree(staging)
|
||||
validate_worker_package(destination)
|
||||
return destination
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--source-root", type=Path, required=True)
|
||||
parser.add_argument("--output-root", type=Path, required=True)
|
||||
args = parser.parse_args()
|
||||
destination = build_worker_package(args.source_root, args.output_root)
|
||||
print(destination)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,90 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Prepare immutable E1 preprocessing and frame-selection inputs."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
from k1link.compute.qualification import (
|
||||
DEFAULT_QUALIFICATION_FRAME_COUNT,
|
||||
prepare_recorded_qualification_slice,
|
||||
)
|
||||
from k1link.device_plugins.xgrids_k1.analyze.valid_fov import (
|
||||
DEFAULT_EDGE_MARGIN_PIXELS,
|
||||
prepare_k1_valid_fov_mask,
|
||||
)
|
||||
|
||||
|
||||
def _arguments() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--job-root", type=Path, required=True)
|
||||
parser.add_argument("--calibration-root", type=Path, required=True)
|
||||
parser.add_argument("--source-id", required=True)
|
||||
parser.add_argument("--output-root", type=Path, required=True)
|
||||
parser.add_argument(
|
||||
"--sample-count",
|
||||
type=int,
|
||||
default=DEFAULT_QUALIFICATION_FRAME_COUNT,
|
||||
)
|
||||
parser.add_argument(
|
||||
"--edge-margin-pixels",
|
||||
type=float,
|
||||
default=DEFAULT_EDGE_MARGIN_PIXELS,
|
||||
)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = _arguments()
|
||||
mask = prepare_k1_valid_fov_mask(
|
||||
calibration_snapshot_root=args.calibration_root,
|
||||
source_id=args.source_id,
|
||||
output_root=args.output_root / "valid-fov",
|
||||
edge_margin_pixels=args.edge_margin_pixels,
|
||||
)
|
||||
qualification = prepare_recorded_qualification_slice(
|
||||
job_root=args.job_root,
|
||||
output_root=args.output_root / "qualification-slices",
|
||||
sample_count=args.sample_count,
|
||||
)
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"schema_version": "missioncore.e1-qualification-preparation/v1",
|
||||
"valid_fov": {
|
||||
"generation_id": mask.generation_id,
|
||||
"manifest": str(mask.manifest_path),
|
||||
"mask": str(mask.mask_path),
|
||||
"calibration_sha256": mask.calibration_sha256,
|
||||
"source_id": mask.source_id,
|
||||
"calibration_slot": mask.calibration_slot,
|
||||
"center_xy": list(mask.center_xy),
|
||||
"radius_pixels": mask.radius_pixels,
|
||||
"crop_xyxy_exclusive": list(mask.crop_xyxy),
|
||||
"valid_pixel_count": mask.valid_pixel_count,
|
||||
"valid_fraction": mask.valid_fraction,
|
||||
},
|
||||
"qualification_slice": {
|
||||
"generation_id": qualification.generation_id,
|
||||
"manifest": str(qualification.manifest_path),
|
||||
"job_id": qualification.job_id,
|
||||
"input_sha256": qualification.input_sha256,
|
||||
"policy": qualification.policy,
|
||||
"source_frame_count": qualification.source_frame_count,
|
||||
"selected_frame_count": len(qualification.frames),
|
||||
"first_frame_index": qualification.frames[0].frame_index,
|
||||
"last_frame_index": qualification.frames[-1].frame_index,
|
||||
},
|
||||
},
|
||||
ensure_ascii=False,
|
||||
indent=2,
|
||||
sort_keys=True,
|
||||
)
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,82 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Build or validate the immutable LAB E2 CVAT annotation handoff."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import k1link.compute.annotation_workspace as workspace_module
|
||||
from k1link.compute import prepare_annotation_workspace, validate_annotation_workspace
|
||||
|
||||
|
||||
def _sha256(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as stream:
|
||||
for chunk in iter(lambda: stream.read(1024 * 1024), b""):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def _arguments() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser()
|
||||
commands = parser.add_subparsers(dest="command", required=True)
|
||||
prepare = commands.add_parser("prepare")
|
||||
prepare.add_argument("--evaluation-pack", type=Path, required=True)
|
||||
prepare.add_argument("--prelabels", type=Path, required=True)
|
||||
prepare.add_argument("--valid-fov-root", type=Path, required=True)
|
||||
prepare.add_argument("--output-root", type=Path, required=True)
|
||||
|
||||
validate = commands.add_parser("validate")
|
||||
validate.add_argument("--workspace", type=Path, required=True)
|
||||
validate.add_argument("--evaluation-pack", type=Path, required=True)
|
||||
validate.add_argument("--prelabels", type=Path, required=True)
|
||||
validate.add_argument("--valid-fov-root", type=Path, required=True)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = _arguments()
|
||||
if args.command == "prepare":
|
||||
result = prepare_annotation_workspace(
|
||||
evaluation_pack_root=args.evaluation_pack,
|
||||
prelabels_root=args.prelabels,
|
||||
valid_fov_root=args.valid_fov_root,
|
||||
output_root=args.output_root,
|
||||
producer_files=(
|
||||
(
|
||||
"annotation_workspace.py",
|
||||
_sha256(Path(str(workspace_module.__file__)).resolve(strict=True)),
|
||||
),
|
||||
(
|
||||
"prepare_e2_annotation_workspace.py",
|
||||
_sha256(Path(__file__).resolve(strict=True)),
|
||||
),
|
||||
),
|
||||
)
|
||||
else:
|
||||
result = validate_annotation_workspace(
|
||||
args.workspace,
|
||||
evaluation_pack_root=args.evaluation_pack,
|
||||
prelabels_root=args.prelabels,
|
||||
valid_fov_root=args.valid_fov_root,
|
||||
)
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"generation_id": result.generation_id,
|
||||
"root": str(result.root),
|
||||
"frame_count": result.frame_count,
|
||||
"draft_instance_count": result.draft_instance_count,
|
||||
"ground_truth": False,
|
||||
},
|
||||
sort_keys=True,
|
||||
)
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,364 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Review and seal the LAB E2 recorded-perception evaluation pack."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from PIL import Image, ImageDraw, ImageFont
|
||||
|
||||
import k1link.compute.evaluation_pack as evaluation_pack_module
|
||||
from k1link.artifacts import utc_now_iso, write_json_atomic
|
||||
from k1link.compute import (
|
||||
EvaluationFrameRequest,
|
||||
prepare_recorded_evaluation_pack,
|
||||
validate_camera_compute_job,
|
||||
validate_recorded_qualification_slice,
|
||||
)
|
||||
|
||||
CANDIDATE_SCHEMA = "missioncore.e2-evaluation-candidates/v1"
|
||||
SELECTION_SCHEMA = "missioncore.e2-evaluation-selection/v1"
|
||||
|
||||
|
||||
def _arguments() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser()
|
||||
commands = parser.add_subparsers(dest="command", required=True)
|
||||
|
||||
candidates = commands.add_parser("candidates")
|
||||
candidates.add_argument("--job-root", type=Path, required=True)
|
||||
candidates.add_argument("--qualification-root", type=Path, required=True)
|
||||
candidates.add_argument("--output-root", type=Path, required=True)
|
||||
|
||||
seal = commands.add_parser("seal")
|
||||
seal.add_argument("--job-root", type=Path, required=True)
|
||||
seal.add_argument("--qualification-root", type=Path, required=True)
|
||||
seal.add_argument("--valid-fov-root", type=Path, required=True)
|
||||
seal.add_argument("--selection", type=Path, required=True)
|
||||
seal.add_argument("--output-root", type=Path, required=True)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def _run(command: list[str]) -> subprocess.CompletedProcess[str]:
|
||||
return subprocess.run(command, check=True, text=True, capture_output=True)
|
||||
|
||||
|
||||
def _ffmpeg_version() -> str:
|
||||
completed = _run(["ffmpeg", "-version"])
|
||||
return completed.stdout.splitlines()[0].strip()
|
||||
|
||||
|
||||
def _reconstruct_stream(job_root: Path, destination: Path) -> None:
|
||||
job = validate_camera_compute_job(job_root)
|
||||
epoch = (
|
||||
job.job_root
|
||||
/ "input"
|
||||
/ "camera"
|
||||
/ job.source_id
|
||||
/ f"epoch-{job.codec_epoch}"
|
||||
)
|
||||
with destination.open("xb") as output:
|
||||
with (epoch / "init.mp4").open("rb") as source:
|
||||
shutil.copyfileobj(source, output, 1024 * 1024)
|
||||
for sequence in range(1, job.segment_count + 1):
|
||||
with (epoch / "segments" / f"{sequence}.m4s").open("rb") as source:
|
||||
shutil.copyfileobj(source, output, 1024 * 1024)
|
||||
output.flush()
|
||||
os.fsync(output.fileno())
|
||||
|
||||
|
||||
def _timeline(stream_path: Path, start_seconds: float, expected_count: int) -> list[float]:
|
||||
completed = _run(
|
||||
[
|
||||
"ffprobe",
|
||||
"-v",
|
||||
"error",
|
||||
"-select_streams",
|
||||
"v:0",
|
||||
"-show_entries",
|
||||
"frame=best_effort_timestamp_time",
|
||||
"-of",
|
||||
"json",
|
||||
str(stream_path),
|
||||
]
|
||||
)
|
||||
document = json.loads(completed.stdout)
|
||||
frames = document.get("frames")
|
||||
if not isinstance(frames, list) or len(frames) != expected_count:
|
||||
raise RuntimeError("decoded camera timestamp count differs from the compute job")
|
||||
epoch_values = [float(frame["best_effort_timestamp_time"]) for frame in frames]
|
||||
first = epoch_values[0]
|
||||
values = [start_seconds + value - first for value in epoch_values]
|
||||
if any(right <= left for left, right in zip(values, values[1:], strict=False)):
|
||||
raise RuntimeError("decoded camera timestamps are not strictly monotonic")
|
||||
return values
|
||||
|
||||
|
||||
def _extract_frames(stream_path: Path, indices: list[int], output_root: Path) -> None:
|
||||
if not indices or indices != sorted(set(indices)):
|
||||
raise RuntimeError("frame extraction indices must be sorted and unique")
|
||||
output_root.mkdir(mode=0o700, parents=True, exist_ok=False)
|
||||
expression = "+".join(f"eq(n\\,{index})" for index in indices)
|
||||
pattern = output_root / "selected-%06d.png"
|
||||
_run(
|
||||
[
|
||||
"ffmpeg",
|
||||
"-hide_banner",
|
||||
"-loglevel",
|
||||
"error",
|
||||
"-i",
|
||||
str(stream_path),
|
||||
"-map",
|
||||
"0:v:0",
|
||||
"-vf",
|
||||
f"select={expression}",
|
||||
"-fps_mode",
|
||||
"passthrough",
|
||||
str(pattern),
|
||||
]
|
||||
)
|
||||
extracted = sorted(output_root.glob("selected-*.png"))
|
||||
if len(extracted) != len(indices):
|
||||
raise RuntimeError("ffmpeg did not extract the requested frame set")
|
||||
for source, frame_index in zip(extracted, indices, strict=True):
|
||||
destination = output_root / f"frame-{frame_index:06d}.png"
|
||||
source.replace(destination)
|
||||
os.chmod(destination, 0o600)
|
||||
|
||||
|
||||
def _sha256(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as stream:
|
||||
while chunk := stream.read(1024 * 1024):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def _contact_sheets(
|
||||
frames_root: Path,
|
||||
indices: list[int],
|
||||
timestamps: list[float],
|
||||
output_root: Path,
|
||||
) -> list[dict[str, Any]]:
|
||||
output_root.mkdir(mode=0o700, parents=True, exist_ok=False)
|
||||
columns = 8
|
||||
rows = 4
|
||||
thumb_size = (200, 150)
|
||||
label_height = 25
|
||||
font = ImageFont.load_default(size=15)
|
||||
records: list[dict[str, Any]] = []
|
||||
page_size = columns * rows
|
||||
for page, offset in enumerate(range(0, len(indices), page_size), start=1):
|
||||
page_indices = indices[offset : offset + page_size]
|
||||
canvas = Image.new(
|
||||
"RGB",
|
||||
(columns * thumb_size[0], rows * (thumb_size[1] + label_height)),
|
||||
(18, 18, 18),
|
||||
)
|
||||
draw = ImageDraw.Draw(canvas)
|
||||
for position, frame_index in enumerate(page_indices):
|
||||
row, column = divmod(position, columns)
|
||||
x = column * thumb_size[0]
|
||||
y = row * (thumb_size[1] + label_height)
|
||||
with Image.open(frames_root / f"frame-{frame_index:06d}.png") as opened:
|
||||
image = opened.convert("RGB")
|
||||
image.thumbnail(thumb_size, Image.Resampling.LANCZOS)
|
||||
canvas.paste(image, (x, y))
|
||||
draw.text(
|
||||
(x + 4, y + thumb_size[1] + 3),
|
||||
f"f={frame_index} t={timestamps[frame_index]:.3f}s",
|
||||
fill=(240, 240, 240),
|
||||
font=font,
|
||||
)
|
||||
path = output_root / f"contact-sheet-{page:02d}.png"
|
||||
canvas.save(path, format="PNG", optimize=False)
|
||||
os.chmod(path, 0o600)
|
||||
records.append(
|
||||
{
|
||||
"path": path.name,
|
||||
"first_frame_index": page_indices[0],
|
||||
"last_frame_index": page_indices[-1],
|
||||
"frame_count": len(page_indices),
|
||||
"byte_length": path.stat().st_size,
|
||||
"sha256": _sha256(path),
|
||||
}
|
||||
)
|
||||
return records
|
||||
|
||||
|
||||
def _candidates(args: argparse.Namespace) -> int:
|
||||
job = validate_camera_compute_job(args.job_root)
|
||||
qualification = validate_recorded_qualification_slice(
|
||||
args.qualification_root,
|
||||
job_root=job.job_root,
|
||||
)
|
||||
output = args.output_root.expanduser()
|
||||
output.mkdir(mode=0o700, parents=True, exist_ok=False)
|
||||
published = False
|
||||
try:
|
||||
with tempfile.TemporaryDirectory(prefix="missioncore-e2-") as temporary:
|
||||
stream_path = Path(temporary) / "camera.mp4"
|
||||
_reconstruct_stream(job.job_root, stream_path)
|
||||
timestamps = _timeline(stream_path, job.timeline_start_seconds, job.segment_count)
|
||||
indices = [frame.frame_index for frame in qualification.frames]
|
||||
frames_root = output / "frames"
|
||||
_extract_frames(stream_path, indices, frames_root)
|
||||
sheets = _contact_sheets(frames_root, indices, timestamps, output / "contact-sheets")
|
||||
manifest = {
|
||||
"schema_version": CANDIDATE_SCHEMA,
|
||||
"created_at_utc": utc_now_iso(),
|
||||
"state": "review-candidates",
|
||||
"job_id": job.job_id,
|
||||
"input_sha256": job.input_sha256,
|
||||
"qualification_generation_id": qualification.generation_id,
|
||||
"ffmpeg": _ffmpeg_version(),
|
||||
"frames": [
|
||||
{
|
||||
"qualification_order": order,
|
||||
"frame_index": frame.frame_index,
|
||||
"sequence": frame.sequence,
|
||||
"segment_sha256": frame.segment_sha256,
|
||||
"session_seconds": timestamps[frame.frame_index],
|
||||
"path": f"frames/frame-{frame.frame_index:06d}.png",
|
||||
"byte_length": (
|
||||
frames_root / f"frame-{frame.frame_index:06d}.png"
|
||||
).stat().st_size,
|
||||
"sha256": _sha256(frames_root / f"frame-{frame.frame_index:06d}.png"),
|
||||
}
|
||||
for order, frame in enumerate(qualification.frames, start=1)
|
||||
],
|
||||
"contact_sheets": sheets,
|
||||
"selection_guidance": {
|
||||
"anchor_target": 48,
|
||||
"temporal_clip_target": {"clips": 4, "frames_per_clip": 4},
|
||||
"total_target": 64,
|
||||
"required_coverage": [
|
||||
"person",
|
||||
"car_or_heavy_vehicle",
|
||||
"building_structure",
|
||||
"paved_or_unpaved_ground",
|
||||
"grass_and_woody_vegetation",
|
||||
"lens_boundary_hard_negative",
|
||||
"motion_or_occlusion",
|
||||
],
|
||||
},
|
||||
}
|
||||
write_json_atomic(output / "manifest.json", manifest)
|
||||
os.chmod(output / "manifest.json", 0o600)
|
||||
published = True
|
||||
finally:
|
||||
if not published and output.exists():
|
||||
shutil.rmtree(output)
|
||||
print(json.dumps(manifest, ensure_ascii=False, indent=2, sort_keys=True))
|
||||
return 0
|
||||
|
||||
|
||||
def _load_selection(path: Path) -> tuple[EvaluationFrameRequest, ...]:
|
||||
document = json.loads(path.expanduser().read_text(encoding="utf-8"))
|
||||
rows = document.get("frames") if isinstance(document, dict) else None
|
||||
if document.get("schema_version") != SELECTION_SCHEMA or not isinstance(rows, list):
|
||||
raise RuntimeError("E2 selection document is incompatible")
|
||||
selection: list[EvaluationFrameRequest] = []
|
||||
for row in rows:
|
||||
if not isinstance(row, dict):
|
||||
raise RuntimeError("E2 selection frame is invalid")
|
||||
selection.append(
|
||||
EvaluationFrameRequest(
|
||||
frame_index=int(row["frame_index"]),
|
||||
role=str(row["role"]),
|
||||
group_id=str(row["group_id"]),
|
||||
)
|
||||
)
|
||||
return tuple(selection)
|
||||
|
||||
|
||||
def _write_timeline(path: Path, timestamps: list[float]) -> None:
|
||||
with path.open("x", encoding="utf-8", newline="\n") as stream:
|
||||
for frame_index, session_seconds in enumerate(timestamps):
|
||||
stream.write(
|
||||
json.dumps(
|
||||
{"frame_index": frame_index, "session_seconds": session_seconds},
|
||||
separators=(",", ":"),
|
||||
)
|
||||
+ "\n"
|
||||
)
|
||||
stream.flush()
|
||||
os.fsync(stream.fileno())
|
||||
|
||||
|
||||
def _seal(args: argparse.Namespace) -> int:
|
||||
job = validate_camera_compute_job(args.job_root)
|
||||
selection = _load_selection(args.selection)
|
||||
indices = [frame.frame_index for frame in selection]
|
||||
with tempfile.TemporaryDirectory(prefix="missioncore-e2-") as temporary:
|
||||
temporary_root = Path(temporary)
|
||||
stream_path = temporary_root / "camera.mp4"
|
||||
frames_root = temporary_root / "frames"
|
||||
timeline_path = temporary_root / "timeline.jsonl"
|
||||
_reconstruct_stream(job.job_root, stream_path)
|
||||
timestamps = _timeline(stream_path, job.timeline_start_seconds, job.segment_count)
|
||||
_extract_frames(stream_path, indices, frames_root)
|
||||
_write_timeline(timeline_path, timestamps)
|
||||
pack = prepare_recorded_evaluation_pack(
|
||||
job_root=job.job_root,
|
||||
qualification_root=args.qualification_root,
|
||||
valid_fov_root=args.valid_fov_root,
|
||||
decoded_frames_root=frames_root,
|
||||
timeline_path=timeline_path,
|
||||
output_root=args.output_root,
|
||||
selection=selection,
|
||||
decoder_version=_ffmpeg_version(),
|
||||
selection_document_sha256=_sha256(args.selection.expanduser().resolve(strict=True)),
|
||||
producer_files=(
|
||||
(
|
||||
"evaluation_pack.py",
|
||||
_sha256(Path(str(evaluation_pack_module.__file__)).resolve(strict=True)),
|
||||
),
|
||||
(
|
||||
"prepare_e2_evaluation_pack.py",
|
||||
_sha256(Path(__file__).resolve(strict=True)),
|
||||
),
|
||||
),
|
||||
)
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"schema_version": "missioncore.e2-evaluation-preparation/v1",
|
||||
"generation_id": pack.generation_id,
|
||||
"root": str(pack.root),
|
||||
"job_id": pack.job_id,
|
||||
"input_sha256": pack.input_sha256,
|
||||
"qualification_generation_id": pack.qualification_generation_id,
|
||||
"valid_fov_generation_id": pack.valid_fov_generation_id,
|
||||
"calibration_sha256": pack.calibration_sha256,
|
||||
"frame_count": len(pack.frames),
|
||||
"anchor_count": sum(frame.role == "anchor" for frame in pack.frames),
|
||||
"temporal_frame_count": sum(frame.role == "temporal" for frame in pack.frames),
|
||||
"annotation_state": "unannotated",
|
||||
},
|
||||
ensure_ascii=False,
|
||||
indent=2,
|
||||
sort_keys=True,
|
||||
)
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = _arguments()
|
||||
if args.command == "candidates":
|
||||
return _candidates(args)
|
||||
if args.command == "seal":
|
||||
return _seal(args)
|
||||
raise RuntimeError("unknown E2 command")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,336 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Build an immutable CVAT review handoff for the LAB E3 control and challenger."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import zipfile
|
||||
from pathlib import Path, PurePosixPath
|
||||
from typing import Any
|
||||
|
||||
import numpy as np
|
||||
from PIL import Image
|
||||
|
||||
RESULT_SCHEMA = "missioncore.k1-e3-rectified-segmentation-result/v1"
|
||||
PACK_SCHEMA = "missioncore.perception-evaluation-pack/v1"
|
||||
LABEL_SCHEMA = "missioncore.perception-cvat-labels/v1"
|
||||
MANIFEST_SCHEMA = "missioncore.lab-e3-cvat-review-workspace/v1"
|
||||
IDENTITY_SCHEMA = "missioncore.lab-e3-cvat-review-identity/v1"
|
||||
CONTROL = "eomt-fisheye-mask"
|
||||
CHALLENGER = "eomt-kb4-cubemap5-clahe"
|
||||
EXPECTED_FRAMES = 64
|
||||
PRIORITY_IMAGE_IDS = (59, 63, 62, 61, 60, 64, 50, 30, 17, 29)
|
||||
ZIP_TIMESTAMP = (1980, 1, 1, 0, 0, 0)
|
||||
|
||||
|
||||
def _arguments() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--evaluation-pack", type=Path, required=True)
|
||||
parser.add_argument("--e3-result", type=Path, required=True)
|
||||
parser.add_argument("--labels", type=Path, required=True)
|
||||
parser.add_argument("--images-zip", type=Path, required=True)
|
||||
parser.add_argument("--output-root", type=Path, required=True)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def _canonical_json(value: object) -> bytes:
|
||||
return json.dumps(
|
||||
value,
|
||||
ensure_ascii=False,
|
||||
sort_keys=True,
|
||||
separators=(",", ":"),
|
||||
allow_nan=False,
|
||||
).encode()
|
||||
|
||||
|
||||
def _sha256(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as source:
|
||||
for chunk in iter(lambda: source.read(1024 * 1024), b""):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def _read_object(path: Path) -> dict[str, Any]:
|
||||
value = json.loads(path.read_text(encoding="utf-8"))
|
||||
if not isinstance(value, dict):
|
||||
raise RuntimeError(f"JSON root is not an object: {path}")
|
||||
return value
|
||||
|
||||
|
||||
def _valid_sha256(value: object) -> bool:
|
||||
return (
|
||||
isinstance(value, str)
|
||||
and len(value) == 64
|
||||
and all(character in "0123456789abcdef" for character in value)
|
||||
)
|
||||
|
||||
|
||||
def _safe_artifact(root: Path, encoded: object) -> Path:
|
||||
if not isinstance(encoded, str):
|
||||
raise RuntimeError("artifact path is not a string")
|
||||
relative = PurePosixPath(encoded)
|
||||
if relative.is_absolute() or ".." in relative.parts or not relative.parts:
|
||||
raise RuntimeError("artifact path is unsafe")
|
||||
path = root.joinpath(*relative.parts).resolve(strict=True)
|
||||
if not path.is_file() or not path.is_relative_to(root):
|
||||
raise RuntimeError("artifact path escaped its root")
|
||||
return path
|
||||
|
||||
|
||||
def _artifact(path: Path, root: Path) -> dict[str, Any]:
|
||||
return {
|
||||
"path": path.relative_to(root).as_posix(),
|
||||
"bytes": path.stat().st_size,
|
||||
"sha256": _sha256(path),
|
||||
}
|
||||
|
||||
|
||||
def _validate_pack(root: Path) -> tuple[dict[str, Any], list[dict[str, Any]]]:
|
||||
manifest = _read_object(root / "manifest.json")
|
||||
identity = manifest.get("identity")
|
||||
identity_sha256 = manifest.get("identity_sha256")
|
||||
if (
|
||||
manifest.get("schema_version") != PACK_SCHEMA
|
||||
or not isinstance(identity, dict)
|
||||
or not _valid_sha256(identity_sha256)
|
||||
or hashlib.sha256(_canonical_json(identity)).hexdigest() != identity_sha256
|
||||
or manifest.get("generation_id") != f"evaluation-pack-{identity_sha256}"
|
||||
or identity.get("source_id") != "sensor.camera.right"
|
||||
or identity.get("calibration_slot") != "camera_1"
|
||||
):
|
||||
raise RuntimeError("evaluation pack is incompatible")
|
||||
frames = identity.get("frames")
|
||||
if (
|
||||
not isinstance(frames, list)
|
||||
or len(frames) != EXPECTED_FRAMES
|
||||
or [frame.get("image_id") for frame in frames] != list(range(1, 65))
|
||||
):
|
||||
raise RuntimeError("evaluation pack frame contract changed")
|
||||
return manifest, frames
|
||||
|
||||
|
||||
def _validate_result(root: Path, pack: dict[str, Any]) -> dict[str, Any]:
|
||||
result = _read_object(root / "result.json")
|
||||
identity = result.get("identity")
|
||||
identity_sha256 = result.get("identity_sha256")
|
||||
if (
|
||||
result.get("schema_version") != RESULT_SCHEMA
|
||||
or result.get("ground_truth") is not False
|
||||
or not isinstance(identity, dict)
|
||||
or not _valid_sha256(identity_sha256)
|
||||
or hashlib.sha256(_canonical_json(identity)).hexdigest() != identity_sha256
|
||||
or result.get("result_id") != f"e3-segmentation-{identity_sha256}"
|
||||
or root.name != result.get("result_id")
|
||||
or identity.get("evaluation_pack_id") != pack["generation_id"]
|
||||
or identity.get("evaluation_identity_sha256") != pack["identity_sha256"]
|
||||
or identity.get("frame_count") != EXPECTED_FRAMES
|
||||
or CONTROL not in identity.get("variants", ())
|
||||
or CHALLENGER not in identity.get("variants", ())
|
||||
):
|
||||
raise RuntimeError("LAB E3 result is incompatible")
|
||||
artifacts = result.get("artifacts")
|
||||
if not isinstance(artifacts, list):
|
||||
raise RuntimeError("LAB E3 artifact list is invalid")
|
||||
descriptors = {
|
||||
item.get("path"): item
|
||||
for item in artifacts
|
||||
if isinstance(item, dict) and isinstance(item.get("path"), str)
|
||||
}
|
||||
for variant in (CONTROL, CHALLENGER):
|
||||
for image_id in range(1, 65):
|
||||
encoded = f"semantic-masks/{variant}/image-{image_id:03d}.png"
|
||||
descriptor = descriptors.get(encoded)
|
||||
path = _safe_artifact(root, encoded)
|
||||
if (
|
||||
not isinstance(descriptor, dict)
|
||||
or path.stat().st_size != descriptor.get("bytes")
|
||||
or not _valid_sha256(descriptor.get("sha256"))
|
||||
or _sha256(path) != descriptor["sha256"]
|
||||
):
|
||||
raise RuntimeError(f"LAB E3 semantic artifact changed: {encoded}")
|
||||
return result
|
||||
|
||||
|
||||
def _semantic_labels(document: dict[str, Any]) -> list[dict[str, Any]]:
|
||||
if document.get("schema_version") != LABEL_SCHEMA:
|
||||
raise RuntimeError("CVAT label document is incompatible")
|
||||
raw = document.get("semantic_task")
|
||||
if not isinstance(raw, list):
|
||||
raise RuntimeError("CVAT semantic labels are absent")
|
||||
labels = [
|
||||
{"id": 0, "name": "background", "color": "#000000"},
|
||||
*[
|
||||
{"id": int(item["id"]), "name": str(item["name"]), "color": str(item["color"])}
|
||||
for item in raw
|
||||
],
|
||||
]
|
||||
if [label["id"] for label in labels] != list(range(16)) + [255]:
|
||||
raise RuntimeError("CVAT semantic taxonomy changed")
|
||||
return labels
|
||||
|
||||
|
||||
def _rgb(value: str) -> tuple[int, int, int]:
|
||||
if len(value) != 7 or not value.startswith("#"):
|
||||
raise RuntimeError("CVAT label color is invalid")
|
||||
return tuple(int(value[index : index + 2], 16) for index in (1, 3, 5))
|
||||
|
||||
|
||||
def _zip_info(name: str) -> zipfile.ZipInfo:
|
||||
info = zipfile.ZipInfo(name, date_time=ZIP_TIMESTAMP)
|
||||
info.compress_type = zipfile.ZIP_DEFLATED
|
||||
info.external_attr = 0o100644 << 16
|
||||
return info
|
||||
|
||||
|
||||
def _write_segmentation_zip(
|
||||
destination: Path,
|
||||
*,
|
||||
result_root: Path,
|
||||
variant: str,
|
||||
frames: list[dict[str, Any]],
|
||||
labels: list[dict[str, Any]],
|
||||
) -> None:
|
||||
palette = np.zeros((256, 3), dtype=np.uint8)
|
||||
for label in labels:
|
||||
palette[label["id"]] = _rgb(label["color"])
|
||||
labelmap = "".join(
|
||||
f"{label['name']}:{','.join(str(value) for value in _rgb(label['color']))}::\n"
|
||||
for label in labels
|
||||
).encode()
|
||||
stems = [
|
||||
f"image-{frame['image_id']:03d}-frame-{frame['frame_index']:06d}"
|
||||
for frame in frames
|
||||
]
|
||||
with zipfile.ZipFile(destination, mode="x", compression=zipfile.ZIP_DEFLATED) as archive:
|
||||
archive.writestr(_zip_info("labelmap.txt"), labelmap)
|
||||
archive.writestr(
|
||||
_zip_info("ImageSets/Segmentation/default.txt"),
|
||||
("\n".join(stems) + "\n").encode(),
|
||||
)
|
||||
for image_id, stem in enumerate(stems, start=1):
|
||||
source = result_root / "semantic-masks" / variant / f"image-{image_id:03d}.png"
|
||||
with Image.open(source) as opened:
|
||||
indexed = np.asarray(opened.convert("L"), dtype=np.uint8)
|
||||
if indexed.shape != (600, 800) or np.any(~np.isin(indexed, list(range(16)))):
|
||||
raise RuntimeError(f"semantic mask taxonomy changed: {source}")
|
||||
rgb = palette[indexed]
|
||||
from io import BytesIO
|
||||
|
||||
stream = BytesIO()
|
||||
Image.fromarray(rgb, mode="RGB").save(stream, format="PNG", optimize=True)
|
||||
archive.writestr(
|
||||
_zip_info(f"SegmentationClass/{stem}.png"),
|
||||
stream.getvalue(),
|
||||
)
|
||||
|
||||
|
||||
def _validate_existing(root: Path, identity_sha256: str) -> dict[str, Any]:
|
||||
manifest = _read_object(root / "manifest.json")
|
||||
if (
|
||||
manifest.get("schema_version") != MANIFEST_SCHEMA
|
||||
or manifest.get("workspace_id") != f"e3-cvat-review-{identity_sha256}"
|
||||
or manifest.get("identity_sha256") != identity_sha256
|
||||
or manifest.get("ground_truth") is not False
|
||||
):
|
||||
raise RuntimeError("existing LAB E3 CVAT workspace is incompatible")
|
||||
for descriptor in manifest.get("artifacts", ()):
|
||||
path = _safe_artifact(root, descriptor.get("path"))
|
||||
if path.stat().st_size != descriptor.get("bytes") or _sha256(path) != descriptor.get(
|
||||
"sha256"
|
||||
):
|
||||
raise RuntimeError("existing LAB E3 CVAT artifact changed")
|
||||
return manifest
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = _arguments()
|
||||
pack_root = args.evaluation_pack.resolve(strict=True)
|
||||
result_root = args.e3_result.resolve(strict=True)
|
||||
labels_path = args.labels.resolve(strict=True)
|
||||
images_zip = args.images_zip.resolve(strict=True)
|
||||
output_root = args.output_root.resolve()
|
||||
pack, frames = _validate_pack(pack_root)
|
||||
result = _validate_result(result_root, pack)
|
||||
labels_document = _read_object(labels_path)
|
||||
labels = _semantic_labels(labels_document)
|
||||
with zipfile.ZipFile(images_zip) as archive:
|
||||
if len(archive.namelist()) != EXPECTED_FRAMES:
|
||||
raise RuntimeError("CVAT image archive frame count changed")
|
||||
|
||||
identity = {
|
||||
"schema_version": IDENTITY_SCHEMA,
|
||||
"evaluation_pack_id": pack["generation_id"],
|
||||
"evaluation_identity_sha256": pack["identity_sha256"],
|
||||
"e3_result_id": result["result_id"],
|
||||
"e3_identity_sha256": result["identity_sha256"],
|
||||
"control_variant": CONTROL,
|
||||
"challenger_variant": CHALLENGER,
|
||||
"frame_count": EXPECTED_FRAMES,
|
||||
"labels_sha256": _sha256(labels_path),
|
||||
"images_zip_sha256": _sha256(images_zip),
|
||||
"priority_image_ids": list(PRIORITY_IMAGE_IDS),
|
||||
"producer_sha256": _sha256(Path(__file__).resolve(strict=True)),
|
||||
}
|
||||
identity_sha256 = hashlib.sha256(_canonical_json(identity)).hexdigest()
|
||||
workspace_id = f"e3-cvat-review-{identity_sha256}"
|
||||
final_root = output_root / workspace_id
|
||||
if final_root.exists():
|
||||
manifest = _validate_existing(final_root.resolve(strict=True), identity_sha256)
|
||||
print(json.dumps({"state": "reused", **manifest}, sort_keys=True))
|
||||
return 0
|
||||
|
||||
output_root.mkdir(parents=True, exist_ok=True)
|
||||
staging = output_root / f".{workspace_id}.{os.getpid()}.publish"
|
||||
staging.mkdir(mode=0o700)
|
||||
try:
|
||||
labels_output = staging / "labels.json"
|
||||
labels_output.write_text(json.dumps(labels_document, indent=2) + "\n", encoding="utf-8")
|
||||
for variant, filename in (
|
||||
(CONTROL, "control-fisheye-mask.zip"),
|
||||
(CHALLENGER, "challenger-kb4-cubemap5.zip"),
|
||||
):
|
||||
_write_segmentation_zip(
|
||||
staging / filename,
|
||||
result_root=result_root,
|
||||
variant=variant,
|
||||
frames=frames,
|
||||
labels=labels,
|
||||
)
|
||||
artifacts = [
|
||||
_artifact(path, staging)
|
||||
for path in sorted(staging.iterdir())
|
||||
if path.is_file()
|
||||
]
|
||||
manifest = {
|
||||
"schema_version": MANIFEST_SCHEMA,
|
||||
"workspace_id": workspace_id,
|
||||
"identity_sha256": identity_sha256,
|
||||
"identity": identity,
|
||||
"ground_truth": False,
|
||||
"artifacts": artifacts,
|
||||
"next_gate": "two-pass CVAT review and reviewed export",
|
||||
}
|
||||
(staging / "manifest.json").write_text(
|
||||
json.dumps(manifest, indent=2) + "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
staging.replace(final_root)
|
||||
finally:
|
||||
if staging.exists():
|
||||
shutil.rmtree(staging)
|
||||
print(
|
||||
json.dumps(
|
||||
{"state": "created", "workspace_id": workspace_id, "root": str(final_root)},
|
||||
sort_keys=True,
|
||||
)
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,285 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Source-paced RAVNOVES00 qualification of the live shadow transport."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import threading
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
from typing import Literal
|
||||
|
||||
import uvicorn
|
||||
from fastapi import FastAPI
|
||||
|
||||
from k1link.compute.live_perception import LiveIngressModality, LivePerceptionIngress
|
||||
from k1link.device_plugins.xgrids_k1.live_perception_shadow import (
|
||||
build_live_perception_shadow_router,
|
||||
ensure_live_shadow_token,
|
||||
)
|
||||
from k1link.device_plugins.xgrids_k1.viewer.replay import iter_replay_messages
|
||||
|
||||
REPORT_SCHEMA = "missioncore.e12-shadow-transport-source-report/v1"
|
||||
PLUGIN_ID = "nodedc.device.xgrids-lixelkity-k1"
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ReplayEvent:
|
||||
modality: LiveIngressModality
|
||||
source_id: str
|
||||
source_sequence: int
|
||||
epoch_ns: int
|
||||
monotonic_ns: int
|
||||
payload: bytes
|
||||
|
||||
|
||||
def _sha256(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as stream:
|
||||
for block in iter(lambda: stream.read(1024 * 1024), b""):
|
||||
digest.update(block)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def _camera_events(epoch_root: Path, duration_seconds: float) -> list[ReplayEvent]:
|
||||
index_path = epoch_root / "index.jsonl"
|
||||
if not index_path.is_file():
|
||||
raise RuntimeError("camera index is missing")
|
||||
events: list[ReplayEvent] = []
|
||||
start_epoch_ns: int | None = None
|
||||
end_epoch_ns: int | None = None
|
||||
with index_path.open("r", encoding="utf-8") as stream:
|
||||
for line in stream:
|
||||
record = json.loads(line)
|
||||
epoch_ns = int(record["host_epoch_ns"])
|
||||
if start_epoch_ns is None:
|
||||
start_epoch_ns = epoch_ns
|
||||
end_epoch_ns = epoch_ns + int(duration_seconds * 1_000_000_000)
|
||||
assert end_epoch_ns is not None
|
||||
if epoch_ns > end_epoch_ns:
|
||||
break
|
||||
payload_path = epoch_root / str(record["path"])
|
||||
payload = payload_path.read_bytes()
|
||||
if len(payload) != int(record["length"]):
|
||||
raise RuntimeError("camera replay segment length differs from its index")
|
||||
if hashlib.sha256(payload).hexdigest() != record["sha256"]:
|
||||
raise RuntimeError("camera replay segment digest differs from its index")
|
||||
events.append(
|
||||
ReplayEvent(
|
||||
modality="camera-frame",
|
||||
source_id="sensor.camera.right",
|
||||
source_sequence=int(record["sequence"]),
|
||||
epoch_ns=epoch_ns,
|
||||
monotonic_ns=int(record["host_monotonic_ns"]),
|
||||
payload=payload,
|
||||
)
|
||||
)
|
||||
if not events or start_epoch_ns is None:
|
||||
raise RuntimeError("camera replay interval is empty")
|
||||
init_payload = (epoch_root / "init.mp4").read_bytes()
|
||||
events.insert(
|
||||
0,
|
||||
ReplayEvent(
|
||||
modality="camera-init",
|
||||
source_id="sensor.camera.right",
|
||||
source_sequence=0,
|
||||
epoch_ns=start_epoch_ns - 1,
|
||||
monotonic_ns=events[0].monotonic_ns - 1,
|
||||
payload=init_payload,
|
||||
),
|
||||
)
|
||||
return events
|
||||
|
||||
|
||||
def _mqtt_events(
|
||||
capture_path: Path,
|
||||
*,
|
||||
start_epoch_ns: int,
|
||||
duration_seconds: float,
|
||||
) -> list[ReplayEvent]:
|
||||
end_epoch_ns = start_epoch_ns + int(duration_seconds * 1_000_000_000)
|
||||
events: list[ReplayEvent] = []
|
||||
for message in iter_replay_messages(capture_path):
|
||||
if message.received_at_epoch_ns < start_epoch_ns:
|
||||
continue
|
||||
if message.received_at_epoch_ns > end_epoch_ns:
|
||||
break
|
||||
if message.topic.endswith("/lio_pcl") or message.topic == "RealtimePointcloud":
|
||||
modality: Literal["lidar", "pose"] = "lidar"
|
||||
elif message.topic.endswith("/lio_pose") or message.topic == "RealtimePath":
|
||||
modality = "pose"
|
||||
else:
|
||||
continue
|
||||
events.append(
|
||||
ReplayEvent(
|
||||
modality=modality,
|
||||
source_id=message.topic,
|
||||
source_sequence=message.sequence,
|
||||
epoch_ns=message.received_at_epoch_ns,
|
||||
monotonic_ns=message.received_monotonic_ns or 0,
|
||||
payload=message.payload,
|
||||
)
|
||||
)
|
||||
return events
|
||||
|
||||
|
||||
def _wait_for(predicate: object, *, timeout_seconds: float, label: str) -> None:
|
||||
if not callable(predicate):
|
||||
raise TypeError("wait predicate must be callable")
|
||||
deadline = time.monotonic() + timeout_seconds
|
||||
while time.monotonic() < deadline:
|
||||
if predicate():
|
||||
return
|
||||
time.sleep(0.05)
|
||||
raise RuntimeError(f"timed out waiting for {label}")
|
||||
|
||||
|
||||
def run(args: argparse.Namespace) -> dict[str, object]:
|
||||
repository_root = Path(args.repository_root).expanduser().resolve()
|
||||
session_root = Path(args.session_root).expanduser().resolve()
|
||||
epoch_root = session_root / "media" / "sensor.camera.right" / "epoch-1"
|
||||
capture_path = session_root / "captures" / "mqtt_live" / "mqtt.raw.k1mqtt"
|
||||
camera = _camera_events(epoch_root, args.duration_seconds)
|
||||
mqtt = _mqtt_events(
|
||||
capture_path,
|
||||
start_epoch_ns=camera[1].epoch_ns,
|
||||
duration_seconds=args.duration_seconds,
|
||||
)
|
||||
events = sorted((*camera, *mqtt), key=lambda event: (event.epoch_ns, event.modality))
|
||||
if not mqtt:
|
||||
raise RuntimeError("selected replay interval contains no LiDAR or pose events")
|
||||
|
||||
ingress = LivePerceptionIngress()
|
||||
_, token = ensure_live_shadow_token(repository_root)
|
||||
app = FastAPI()
|
||||
app.include_router(
|
||||
build_live_perception_shadow_router(
|
||||
ingress,
|
||||
PLUGIN_ID,
|
||||
bearer_token=token,
|
||||
)
|
||||
)
|
||||
server = uvicorn.Server(
|
||||
uvicorn.Config(
|
||||
app,
|
||||
host="127.0.0.1",
|
||||
port=args.port,
|
||||
log_level="warning",
|
||||
access_log=False,
|
||||
)
|
||||
)
|
||||
server_thread = threading.Thread(target=server.run, name="e12-shadow-source", daemon=True)
|
||||
server_thread.start()
|
||||
_wait_for(lambda: server.started, timeout_seconds=10.0, label="E12 source server")
|
||||
|
||||
session_id = f"e12-ravnoves00-{int(time.time())}"
|
||||
ingress.begin_session(session_id)
|
||||
_wait_for(
|
||||
lambda: ingress.snapshot()["consumer_connected"],
|
||||
timeout_seconds=args.consumer_timeout_seconds,
|
||||
label="exclusive shadow worker",
|
||||
)
|
||||
started_monotonic = time.monotonic()
|
||||
source_start_ns = events[0].epoch_ns
|
||||
published: dict[str, int] = {}
|
||||
try:
|
||||
for event in events:
|
||||
target = started_monotonic + (event.epoch_ns - source_start_ns) / 1_000_000_000
|
||||
remaining = target - time.monotonic()
|
||||
if remaining > 0:
|
||||
time.sleep(remaining)
|
||||
accepted = ingress.publish(
|
||||
modality=event.modality,
|
||||
source_id=event.source_id,
|
||||
source_sequence=event.source_sequence,
|
||||
captured_at_epoch_ns=event.epoch_ns,
|
||||
received_monotonic_ns=event.monotonic_ns,
|
||||
payload=event.payload,
|
||||
)
|
||||
if accepted:
|
||||
published[event.modality] = published.get(event.modality, 0) + 1
|
||||
ingress.end_session(session_id)
|
||||
_wait_for(
|
||||
lambda: not ingress.snapshot()["consumer_connected"],
|
||||
timeout_seconds=10.0,
|
||||
label="shadow worker report completion",
|
||||
)
|
||||
finally:
|
||||
ingress.end_session(session_id)
|
||||
ingress.close()
|
||||
server.should_exit = True
|
||||
server_thread.join(timeout=10.0)
|
||||
|
||||
completed_monotonic = time.monotonic()
|
||||
source_snapshot = ingress.snapshot()
|
||||
report: dict[str, object] = {
|
||||
"schema_version": REPORT_SCHEMA,
|
||||
"state": "completed",
|
||||
"session_id": session_id,
|
||||
"source": {
|
||||
"kind": "recorded-source-paced-near-live",
|
||||
"session": session_root.name,
|
||||
"camera_source": "sensor.camera.right",
|
||||
"camera_epoch": 1,
|
||||
"duration_seconds": args.duration_seconds,
|
||||
"mqtt_raw_sha256": _sha256(capture_path),
|
||||
"camera_index_sha256": _sha256(epoch_root / "index.jsonl"),
|
||||
},
|
||||
"events_selected": {
|
||||
"camera-init": sum(event.modality == "camera-init" for event in events),
|
||||
"camera-frame": sum(event.modality == "camera-frame" for event in events),
|
||||
"lidar": sum(event.modality == "lidar" for event in events),
|
||||
"pose": sum(event.modality == "pose" for event in events),
|
||||
},
|
||||
"events_admitted": published,
|
||||
"wall_seconds": completed_monotonic - started_monotonic,
|
||||
"ingress": source_snapshot,
|
||||
"authority": {
|
||||
"mode": "shadow-diagnostic-only",
|
||||
"commands_enabled": False,
|
||||
"navigation_or_safety_accepted": False,
|
||||
},
|
||||
}
|
||||
output_root = repository_root / ".runtime" / "compute-experiments" / "e12"
|
||||
output_root.mkdir(parents=True, exist_ok=True)
|
||||
stamp = datetime.now(UTC).strftime("%Y%m%dT%H%M%SZ")
|
||||
temporary = output_root / f".{stamp}-e12-source.json.tmp"
|
||||
destination = output_root / f"{stamp}-e12-source.json"
|
||||
encoded = json.dumps(report, indent=2, sort_keys=True).encode("utf-8") + b"\n"
|
||||
with temporary.open("xb") as stream:
|
||||
stream.write(encoded)
|
||||
stream.flush()
|
||||
os.fsync(stream.fileno())
|
||||
temporary.replace(destination)
|
||||
report["report_path"] = str(destination)
|
||||
return report
|
||||
|
||||
|
||||
def _arguments() -> argparse.Namespace:
|
||||
root = Path(__file__).resolve().parents[2]
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--repository-root", default=str(root))
|
||||
parser.add_argument(
|
||||
"--session-root",
|
||||
default=str(
|
||||
root
|
||||
/ ".runtime"
|
||||
/ "mission-core"
|
||||
/ "evidence"
|
||||
/ "sessions"
|
||||
/ "20260720T065719Z_viewer_live"
|
||||
),
|
||||
)
|
||||
parser.add_argument("--port", type=int, default=8012)
|
||||
parser.add_argument("--duration-seconds", type=float, default=15.0)
|
||||
parser.add_argument("--consumer-timeout-seconds", type=float, default=30.0)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
print(json.dumps(run(_arguments()), indent=2, sort_keys=True))
|
||||
@@ -0,0 +1,809 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Qualify a bounded replay-as-live world-state loop on immutable LAB E6 data."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import importlib.util
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import re
|
||||
import resource
|
||||
import shutil
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
from collections import Counter
|
||||
from collections.abc import Mapping, Sequence
|
||||
from dataclasses import dataclass
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
from types import ModuleType
|
||||
from typing import Any
|
||||
|
||||
import numpy as np
|
||||
import numpy.typing as npt
|
||||
import rerun as rr
|
||||
from rerun.components import FillMode
|
||||
|
||||
import k1link.compute.live_perception as live_perception_module
|
||||
import k1link.compute.live_replay_qualification as live_validator_module
|
||||
from k1link.compute import (
|
||||
TELEMETRY_SCHEMA,
|
||||
LatestWinsQueue,
|
||||
WorldStateProjector,
|
||||
classify_health,
|
||||
validate_live_replay_qualification_result,
|
||||
validate_tracked_fusion_qualification_result,
|
||||
)
|
||||
from k1link.device_plugins.xgrids_k1.analyze.calibrated_projection import (
|
||||
map_points_to_lidar,
|
||||
)
|
||||
from k1link.device_plugins.xgrids_k1.mqtt.capture import read_capture_clock_origin
|
||||
|
||||
PROFILE_SCHEMA = "missioncore.e7-replay-realtime-profile/v1"
|
||||
RESULT_SCHEMA = "missioncore.e7-live-replay-result/v1"
|
||||
IDENTITY_SCHEMA = "missioncore.e7-live-replay-identity/v1"
|
||||
REPORT_SCHEMA = "missioncore.e7-live-replay-run-report/v1"
|
||||
PIPELINE_ID = "bounded-latest-wins-e6-world-state-replay/v1"
|
||||
_EXECUTION_ID = re.compile(r"^[a-z0-9][a-z0-9._-]{2,95}$")
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class ReplayFrame:
|
||||
ordinal: int
|
||||
frame: dict[str, Any]
|
||||
cloud_map: npt.NDArray[np.float32]
|
||||
position_map_xyz: tuple[float, float, float] | None
|
||||
orientation_map_from_lidar_xyzw: tuple[float, float, float, float] | None
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class QueueEnvelope:
|
||||
replay_frame: ReplayFrame
|
||||
scheduled_monotonic: float
|
||||
enqueued_monotonic: float
|
||||
producer_lag_ms: float
|
||||
|
||||
|
||||
def _arguments() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--job", type=Path, required=True)
|
||||
parser.add_argument("--tracking", type=Path, required=True)
|
||||
parser.add_argument("--semantic", type=Path, required=True)
|
||||
parser.add_argument("--e6-result", type=Path, required=True)
|
||||
parser.add_argument("--session", type=Path, required=True)
|
||||
parser.add_argument("--profile", type=Path, required=True)
|
||||
parser.add_argument("--output-root", type=Path, required=True)
|
||||
parser.add_argument("--execution-id", required=True)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def _canonical_json(value: object) -> bytes:
|
||||
return json.dumps(
|
||||
value,
|
||||
ensure_ascii=False,
|
||||
sort_keys=True,
|
||||
separators=(",", ":"),
|
||||
allow_nan=False,
|
||||
).encode()
|
||||
|
||||
|
||||
def _sha256(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as stream:
|
||||
while chunk := stream.read(1024 * 1024):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def _read_profile(path: Path) -> tuple[dict[str, Any], str]:
|
||||
resolved = path.resolve(strict=True)
|
||||
profile = json.loads(resolved.read_text(encoding="utf-8"))
|
||||
if not isinstance(profile, dict) or profile.get("schema_version") != PROFILE_SCHEMA:
|
||||
raise RuntimeError("LAB E7 profile schema changed")
|
||||
selection = profile.get("selection")
|
||||
replay = profile.get("replay")
|
||||
health = profile.get("health")
|
||||
world = profile.get("world_state")
|
||||
presentation = profile.get("presentation")
|
||||
acceptance = profile.get("acceptance")
|
||||
if not all(
|
||||
isinstance(value, dict)
|
||||
for value in (selection, replay, health, world, presentation, acceptance)
|
||||
):
|
||||
raise RuntimeError("LAB E7 profile sections are invalid")
|
||||
mode = profile.get("mode")
|
||||
if mode not in {"pilot", "qualification", "overload-negative-control"}:
|
||||
raise RuntimeError("LAB E7 mode is invalid")
|
||||
start = selection.get("start_frame_index")
|
||||
count = selection.get("frame_count")
|
||||
capacity = replay.get("queue_capacity")
|
||||
speed = replay.get("speed")
|
||||
delay = replay.get("consumer_delay_ms")
|
||||
stale = health.get("stale_after_ms")
|
||||
unavailable = health.get("unavailable_after_ms")
|
||||
if (
|
||||
not isinstance(start, int)
|
||||
or start < 0
|
||||
or not isinstance(count, int)
|
||||
or count < 2
|
||||
or not isinstance(capacity, int)
|
||||
or not 1 <= capacity <= 8
|
||||
or not isinstance(speed, int | float)
|
||||
or not 0.1 <= float(speed) <= 10.0
|
||||
or not isinstance(delay, int | float)
|
||||
or not 0 <= float(delay) <= 5000
|
||||
or not isinstance(stale, int | float)
|
||||
or not isinstance(unavailable, int | float)
|
||||
or not 0 < float(stale) < float(unavailable)
|
||||
):
|
||||
raise RuntimeError("LAB E7 scheduling contract is invalid")
|
||||
clearance = world.get("clearance")
|
||||
if not isinstance(clearance, dict):
|
||||
raise RuntimeError("LAB E7 clearance contract is invalid")
|
||||
sectors = clearance.get("sector_count")
|
||||
if not isinstance(sectors, int) or not 12 <= sectors <= 360:
|
||||
raise RuntimeError("LAB E7 clearance sector count is invalid")
|
||||
return profile, _sha256(resolved)
|
||||
|
||||
|
||||
def _load_e6_runner() -> ModuleType:
|
||||
path = Path(__file__).with_name("fuse_e6_tracking_lidar.py").resolve(strict=True)
|
||||
spec = importlib.util.spec_from_file_location("missioncore_e6_replay_adapter", path)
|
||||
if spec is None or spec.loader is None:
|
||||
raise RuntimeError("LAB E6 replay adapter cannot be loaded")
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
sys.modules[spec.name] = module
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
def _read_jsonl(path: Path) -> list[dict[str, Any]]:
|
||||
rows: list[dict[str, Any]] = []
|
||||
with path.open(encoding="utf-8") as stream:
|
||||
for line in stream:
|
||||
value = json.loads(line)
|
||||
if not isinstance(value, dict):
|
||||
raise RuntimeError("LAB E7 input JSONL row is invalid")
|
||||
rows.append(value)
|
||||
return rows
|
||||
|
||||
|
||||
def _artifact_path(artifacts: Sequence[Mapping[str, Any]], kind: str, root: Path) -> Path:
|
||||
matches = [item for item in artifacts if item.get("kind") == kind]
|
||||
if len(matches) != 1 or not isinstance(matches[0].get("path"), str):
|
||||
raise RuntimeError(f"LAB E7 upstream artifact is missing: {kind}")
|
||||
return root / str(matches[0]["path"])
|
||||
|
||||
|
||||
def _prepare_frames(
|
||||
*,
|
||||
e6: Any,
|
||||
session: Path,
|
||||
selection_start: int,
|
||||
selection_count: int,
|
||||
) -> list[ReplayFrame]:
|
||||
if selection_start + selection_count > e6.frame_count:
|
||||
raise RuntimeError("LAB E7 selection exceeds the E6 frame range")
|
||||
runner = _load_e6_runner()
|
||||
frames_path = _artifact_path(
|
||||
e6.artifacts,
|
||||
"tracked-lidar-frame-metadata",
|
||||
e6.result_root,
|
||||
)
|
||||
frame_rows = _read_jsonl(frames_path)
|
||||
arrays_path = _artifact_path(e6.artifacts, "tracked-lidar-arrays", e6.result_root)
|
||||
tracking_frames = runner._read_tracking_frames(
|
||||
e6.tracking.artifact("tracking-frame-metadata").path,
|
||||
e6.tracking.frame_count,
|
||||
e6.tracking.source_start_frame_index,
|
||||
)
|
||||
origin = read_capture_clock_origin(
|
||||
session / "captures" / "mqtt_live" / "mqtt.timeline.origin.json"
|
||||
)
|
||||
epoch_root = (
|
||||
e6.job.job_root
|
||||
/ "input"
|
||||
/ "camera"
|
||||
/ e6.job.source_id
|
||||
/ f"epoch-{e6.job.codec_epoch}"
|
||||
)
|
||||
anchors = runner._camera_anchors(
|
||||
epoch_root / "index.jsonl",
|
||||
tracking_frames,
|
||||
origin_monotonic_ns=origin.started_monotonic_ns,
|
||||
)
|
||||
temporal = json.loads(
|
||||
(e6.result_root / "run-report.json").read_text(encoding="utf-8")
|
||||
)["identity"]["configuration"]["temporal"]
|
||||
samples = list(
|
||||
runner._lidar_samples(
|
||||
session / "captures" / "mqtt_live" / "mqtt.raw.k1mqtt",
|
||||
anchors,
|
||||
origin_monotonic_ns=origin.started_monotonic_ns,
|
||||
maximum_lidar_camera_delta_s=float(
|
||||
temporal["maximum_lidar_camera_delta_ms"]
|
||||
)
|
||||
/ 1000.0,
|
||||
maximum_pose_point_delta_s=float(temporal["maximum_pose_point_delta_ms"])
|
||||
/ 1000.0,
|
||||
)
|
||||
)
|
||||
prepared: list[ReplayFrame] = []
|
||||
with np.load(arrays_path, allow_pickle=False) as arrays:
|
||||
offsets = arrays["cloud_offsets"]
|
||||
clouds = arrays["cloud_points"]
|
||||
for ordinal in range(selection_start, selection_start + selection_count):
|
||||
frame = frame_rows[ordinal]
|
||||
sample = samples[ordinal]
|
||||
if (frame["state"] == "fused") != (sample is not None):
|
||||
raise RuntimeError("LAB E7 pose binding changed from the accepted E6 result")
|
||||
start = int(offsets[ordinal])
|
||||
end = int(offsets[ordinal + 1])
|
||||
prepared.append(
|
||||
ReplayFrame(
|
||||
ordinal=ordinal,
|
||||
frame=frame,
|
||||
cloud_map=np.asarray(clouds[start:end], dtype=np.float32).copy(),
|
||||
position_map_xyz=(
|
||||
None
|
||||
if sample is None
|
||||
else tuple(float(value) for value in sample.pose_frame.position_xyz)
|
||||
),
|
||||
orientation_map_from_lidar_xyzw=(
|
||||
None
|
||||
if sample is None
|
||||
else tuple(
|
||||
float(value)
|
||||
for value in sample.pose_frame.orientation_xyzw
|
||||
)
|
||||
),
|
||||
)
|
||||
)
|
||||
return prepared
|
||||
|
||||
|
||||
def _transform_to_lidar(
|
||||
points_map: npt.NDArray[np.float32] | npt.NDArray[np.float64],
|
||||
frame: ReplayFrame,
|
||||
) -> npt.NDArray[np.float64]:
|
||||
if frame.position_map_xyz is None or frame.orientation_map_from_lidar_xyzw is None:
|
||||
return np.empty((0, 3), dtype=np.float64)
|
||||
return map_points_to_lidar(
|
||||
np.asarray(points_map, dtype=np.float64),
|
||||
position_map_xyz=frame.position_map_xyz,
|
||||
orientation_map_from_lidar_xyzw=frame.orientation_map_from_lidar_xyzw,
|
||||
)
|
||||
|
||||
|
||||
def _clearance(
|
||||
points_lidar: npt.NDArray[np.float64],
|
||||
profile: Mapping[str, Any],
|
||||
) -> dict[str, Any]:
|
||||
sector_count = int(profile["sector_count"])
|
||||
empty = {
|
||||
"schema": "diagnostic-polar-obstacle-clearance/v1",
|
||||
"state": "unavailable",
|
||||
"frame": "k1-lidar",
|
||||
"front_m": None,
|
||||
"observed_sector_fraction": 0.0,
|
||||
"sector_ranges_m": [None] * sector_count,
|
||||
}
|
||||
if points_lidar.size == 0:
|
||||
return empty
|
||||
finite = np.all(np.isfinite(points_lidar), axis=1)
|
||||
points = points_lidar[finite]
|
||||
if points.size == 0:
|
||||
return empty
|
||||
ground_z = float(np.percentile(points[:, 2], float(profile["ground_percentile"])))
|
||||
ranges = np.linalg.norm(points[:, :2], axis=1)
|
||||
keep = (
|
||||
(ranges >= float(profile["minimum_range_m"]))
|
||||
& (ranges <= float(profile["maximum_range_m"]))
|
||||
& (points[:, 2] >= ground_z + float(profile["minimum_height_above_ground_m"]))
|
||||
& (points[:, 2] <= ground_z + float(profile["maximum_height_above_ground_m"]))
|
||||
)
|
||||
points = points[keep]
|
||||
ranges = ranges[keep]
|
||||
if points.size == 0:
|
||||
return {**empty, "ground_z_estimate_m": ground_z}
|
||||
angles = np.arctan2(points[:, 1], points[:, 0])
|
||||
indices = np.floor((angles + math.pi) / (2.0 * math.pi) * sector_count).astype(int)
|
||||
indices = np.clip(indices, 0, sector_count - 1)
|
||||
sector_ranges = np.full(sector_count, np.inf, dtype=np.float64)
|
||||
np.minimum.at(sector_ranges, indices, ranges)
|
||||
observed = np.isfinite(sector_ranges)
|
||||
front_half = math.radians(float(profile["front_half_angle_degrees"]))
|
||||
front = ranges[np.abs(angles) <= front_half]
|
||||
return {
|
||||
"schema": "diagnostic-polar-obstacle-clearance/v1",
|
||||
"state": "observed" if np.any(observed) else "unavailable",
|
||||
"frame": "k1-lidar",
|
||||
"ground_z_estimate_m": ground_z,
|
||||
"front_m": None if front.size == 0 else float(np.min(front)),
|
||||
"observed_sector_fraction": float(np.mean(observed)),
|
||||
"sector_ranges_m": [
|
||||
None if not math.isfinite(value) else float(value) for value in sector_ranges
|
||||
],
|
||||
"safety_accepted": False,
|
||||
}
|
||||
|
||||
|
||||
def _percentile(values: Sequence[float], quantile: float) -> float:
|
||||
return float(np.percentile(np.asarray(values, dtype=np.float64), quantile))
|
||||
|
||||
|
||||
def _peak_rss_mib() -> float:
|
||||
value = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
|
||||
return float(value / (1024 * 1024) if sys.platform == "darwin" else value / 1024)
|
||||
|
||||
|
||||
def _queue_document(queue: LatestWinsQueue[Any]) -> dict[str, Any]:
|
||||
snapshot = queue.snapshot()
|
||||
return {
|
||||
"capacity": snapshot.capacity,
|
||||
"depth": snapshot.depth,
|
||||
"maximum_depth": snapshot.maximum_depth,
|
||||
"published": snapshot.published,
|
||||
"consumed": snapshot.consumed,
|
||||
"dropped_overflow": snapshot.dropped_overflow,
|
||||
"dropped_superseded": snapshot.dropped_superseded,
|
||||
"closed": snapshot.closed,
|
||||
}
|
||||
|
||||
|
||||
def _artifact(path: Path, kind: str, media_type: str, schema: str | None = None) -> dict[str, Any]:
|
||||
value: dict[str, Any] = {
|
||||
"kind": kind,
|
||||
"path": path.name,
|
||||
"media_type": media_type,
|
||||
"byte_length": path.stat().st_size,
|
||||
"sha256": _sha256(path),
|
||||
}
|
||||
if schema is not None:
|
||||
value["schema_version"] = schema
|
||||
return value
|
||||
|
||||
|
||||
def _accepted(profile: Mapping[str, Any], metrics: Mapping[str, Any]) -> tuple[bool, list[str]]:
|
||||
acceptance = profile["acceptance"]
|
||||
reasons: list[str] = []
|
||||
if metrics["queue"]["maximum_depth"] > int(acceptance["maximum_queue_depth"]):
|
||||
reasons.append("queue-depth")
|
||||
if metrics["process_peak_rss_mib"] > float(acceptance["maximum_process_rss_mib"]):
|
||||
reasons.append("process-rss")
|
||||
if profile["mode"] == "overload-negative-control":
|
||||
if metrics["drop_fraction"] < float(acceptance["minimum_drop_fraction"]):
|
||||
reasons.append("negative-control-did-not-overload")
|
||||
else:
|
||||
if metrics["delivery_rate_hz"] < float(acceptance["minimum_delivery_rate_hz"]):
|
||||
reasons.append("delivery-rate")
|
||||
if metrics["end_to_end_latency_ms"]["p95"] > float(
|
||||
acceptance["maximum_end_to_end_p95_ms"]
|
||||
):
|
||||
reasons.append("latency-p95")
|
||||
if metrics["drop_fraction"] > float(acceptance["maximum_drop_fraction"]):
|
||||
reasons.append("drop-fraction")
|
||||
return not reasons, reasons
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = _arguments()
|
||||
if _EXECUTION_ID.fullmatch(args.execution_id) is None:
|
||||
raise RuntimeError("LAB E7 execution ID is invalid")
|
||||
profile, profile_sha256 = _read_profile(args.profile)
|
||||
session = args.session.resolve(strict=True)
|
||||
preparation_started = time.perf_counter()
|
||||
e6 = validate_tracked_fusion_qualification_result(
|
||||
args.job,
|
||||
args.tracking,
|
||||
args.semantic,
|
||||
args.e6_result,
|
||||
)
|
||||
if session.name != e6.job.session_id:
|
||||
raise RuntimeError("LAB E7 session does not match the E6 result")
|
||||
selection = profile["selection"]
|
||||
replay_frames = _prepare_frames(
|
||||
e6=e6,
|
||||
session=session,
|
||||
selection_start=int(selection["start_frame_index"]),
|
||||
selection_count=int(selection["frame_count"]),
|
||||
)
|
||||
preparation_wall = time.perf_counter() - preparation_started
|
||||
identity = {
|
||||
"schema_version": IDENTITY_SCHEMA,
|
||||
"execution_id": args.execution_id,
|
||||
"pipeline": PIPELINE_ID,
|
||||
"e6_result_id": e6.result_id,
|
||||
"job_id": e6.job.job_id,
|
||||
"input_sha256": e6.job.input_sha256,
|
||||
"session_id": e6.job.session_id,
|
||||
"source_id": e6.job.source_id,
|
||||
"calibration_sha256": e6.semantic.calibration_sha256,
|
||||
"camera_slot": e6.semantic.calibration_slot,
|
||||
"selection": selection,
|
||||
"configuration": profile,
|
||||
"profile_sha256": profile_sha256,
|
||||
"producer_sha256": _sha256(Path(__file__).resolve(strict=True)),
|
||||
"runtime_contract_sha256": _sha256(
|
||||
Path(live_perception_module.__file__).resolve(strict=True)
|
||||
),
|
||||
"result_validator_sha256": _sha256(
|
||||
Path(live_validator_module.__file__).resolve(strict=True)
|
||||
),
|
||||
"e6_adapter_sha256": _sha256(
|
||||
Path(__file__).with_name("fuse_e6_tracking_lidar.py").resolve(strict=True)
|
||||
),
|
||||
}
|
||||
identity_sha256 = hashlib.sha256(_canonical_json(identity)).hexdigest()
|
||||
result_id = f"e7-live-replay-{identity_sha256}"
|
||||
parent = args.output_root.resolve()
|
||||
parent.mkdir(mode=0o700, parents=True, exist_ok=True)
|
||||
output = parent / result_id
|
||||
if output.exists():
|
||||
print(json.dumps({"result_id": result_id, "output": str(output), "reused": True}))
|
||||
return 0
|
||||
staging = parent / f".{result_id}.{os.getpid()}.incomplete"
|
||||
staging.mkdir(mode=0o700, exist_ok=False)
|
||||
world_path = staging / "world-state.jsonl"
|
||||
telemetry_path = staging / "telemetry.jsonl"
|
||||
rrd_path = staging / "world-state.rrd"
|
||||
report_path = staging / "run-report.json"
|
||||
queue = LatestWinsQueue[QueueEnvelope](int(profile["replay"]["queue_capacity"]))
|
||||
producer_error: list[BaseException] = []
|
||||
recording: rr.RecordingStream | None = None
|
||||
try:
|
||||
recording = rr.RecordingStream("nodedc_mission_core_e7", recording_id=result_id)
|
||||
recording.set_sinks(rr.FileSink(rrd_path, write_footer=True))
|
||||
recording.log("/world", rr.ViewCoordinates.RIGHT_HAND_Z_UP, static=True)
|
||||
recording.log(
|
||||
"/contract",
|
||||
rr.TextDocument(
|
||||
"LAB E7 replay-as-live: bounded latest-wins derived queue, explicit age and "
|
||||
"degradation, E6 diagnostic objects, no vehicle-control authority."
|
||||
),
|
||||
static=True,
|
||||
)
|
||||
replay_speed = float(profile["replay"]["speed"])
|
||||
first_session = float(replay_frames[0].frame["session_seconds"])
|
||||
replay_started = time.perf_counter() + 0.25
|
||||
|
||||
def produce() -> None:
|
||||
try:
|
||||
for replay_frame in replay_frames:
|
||||
offset = (
|
||||
float(replay_frame.frame["session_seconds"]) - first_session
|
||||
) / replay_speed
|
||||
deadline = replay_started + offset
|
||||
remaining = deadline - time.perf_counter()
|
||||
if remaining > 0:
|
||||
time.sleep(remaining)
|
||||
enqueued = time.perf_counter()
|
||||
queue.publish(
|
||||
QueueEnvelope(
|
||||
replay_frame=replay_frame,
|
||||
scheduled_monotonic=deadline,
|
||||
enqueued_monotonic=enqueued,
|
||||
producer_lag_ms=max(0.0, (enqueued - deadline) * 1000.0),
|
||||
)
|
||||
)
|
||||
except BaseException as exc: # noqa: BLE001 - forwarded to main thread
|
||||
producer_error.append(exc)
|
||||
finally:
|
||||
queue.close()
|
||||
|
||||
producer = threading.Thread(target=produce, name="e7-replay-producer", daemon=True)
|
||||
producer.start()
|
||||
projector = WorldStateProjector(
|
||||
velocity_history_limit_s=float(
|
||||
profile["world_state"]["velocity_history_limit_s"]
|
||||
)
|
||||
)
|
||||
health_counts: Counter[str] = Counter()
|
||||
end_to_end: list[float] = []
|
||||
queue_wait: list[float] = []
|
||||
processing: list[float] = []
|
||||
rerun_latency: list[float] = []
|
||||
producer_lag: list[float] = []
|
||||
published_indices: list[int] = []
|
||||
consumer_delay = float(profile["replay"]["consumer_delay_ms"]) / 1000.0
|
||||
with world_path.open("w", encoding="utf-8") as world_stream, telemetry_path.open(
|
||||
"w", encoding="utf-8"
|
||||
) as telemetry_stream:
|
||||
while True:
|
||||
envelope = queue.take_next(timeout=0.5)
|
||||
if envelope is None:
|
||||
if queue.snapshot().closed:
|
||||
break
|
||||
continue
|
||||
dequeued = time.perf_counter()
|
||||
if consumer_delay > 0:
|
||||
time.sleep(consumer_delay)
|
||||
stage_started = time.perf_counter()
|
||||
replay_frame = envelope.replay_frame
|
||||
cloud_lidar = _transform_to_lidar(replay_frame.cloud_map, replay_frame)
|
||||
accepted = [
|
||||
item
|
||||
for item in replay_frame.frame.get("objects", [])
|
||||
if str(item.get("cuboid_status", "")).startswith("accepted-")
|
||||
]
|
||||
centers_map = np.asarray(
|
||||
[item["cuboid_center_map"] for item in accepted], dtype=np.float64
|
||||
).reshape((-1, 3))
|
||||
centers_lidar = _transform_to_lidar(centers_map, replay_frame)
|
||||
lidar_positions = {
|
||||
int(item["track_id"]): centers_lidar[index]
|
||||
for index, item in enumerate(accepted)
|
||||
}
|
||||
world = projector.project(
|
||||
frame=replay_frame.frame,
|
||||
lidar_positions=lidar_positions,
|
||||
clearance=_clearance(
|
||||
cloud_lidar,
|
||||
profile["world_state"]["clearance"],
|
||||
),
|
||||
)
|
||||
projection_finished = time.perf_counter()
|
||||
relative_seconds = (
|
||||
float(replay_frame.frame["session_seconds"]) - first_session
|
||||
)
|
||||
recording.set_time(
|
||||
"replay_time",
|
||||
duration=np.timedelta64(round(relative_seconds * 1e9), "ns"),
|
||||
)
|
||||
if bool(profile["presentation"]["log_point_cloud"]) and replay_frame.cloud_map.size:
|
||||
recording.log(
|
||||
"/world/lidar",
|
||||
rr.Points3D(
|
||||
replay_frame.cloud_map,
|
||||
colors=np.tile(
|
||||
np.asarray([[145, 145, 145]], dtype=np.uint8),
|
||||
(replay_frame.cloud_map.shape[0], 1),
|
||||
),
|
||||
radii=rr.Radius.ui_points(
|
||||
float(profile["presentation"]["point_radius_ui"])
|
||||
),
|
||||
),
|
||||
)
|
||||
elif not replay_frame.cloud_map.size:
|
||||
recording.log("/world/lidar", rr.Clear(recursive=False))
|
||||
if accepted:
|
||||
colors = []
|
||||
labels = []
|
||||
for item in accepted:
|
||||
group = str(item["association_group"])
|
||||
colors.append(
|
||||
{
|
||||
"person": [255, 70, 170, 88],
|
||||
"vehicle": [130, 90, 255, 88],
|
||||
"bicycle": [80, 220, 255, 88],
|
||||
"motorcycle": [255, 180, 65, 88],
|
||||
}.get(group, [220, 220, 220, 88])
|
||||
)
|
||||
labels.append(
|
||||
f"{group} #{item['track_id']} · "
|
||||
f"{float(item['distance_smoothed_m']):.1f} m"
|
||||
)
|
||||
recording.log(
|
||||
"/world/perception/boxes3d",
|
||||
rr.Boxes3D(
|
||||
centers=centers_map.astype(np.float32),
|
||||
half_sizes=np.asarray(
|
||||
[item["cuboid_half_size"] for item in accepted],
|
||||
dtype=np.float32,
|
||||
),
|
||||
quaternions=np.asarray(
|
||||
[item["cuboid_quaternion_xyzw"] for item in accepted],
|
||||
dtype=np.float32,
|
||||
),
|
||||
colors=np.asarray(colors, dtype=np.uint8),
|
||||
labels=labels,
|
||||
fill_mode=FillMode.Solid,
|
||||
show_labels=True,
|
||||
),
|
||||
)
|
||||
else:
|
||||
recording.log("/world/perception/boxes3d", rr.Clear(recursive=False))
|
||||
rerun_finished = time.perf_counter()
|
||||
age_ms = (rerun_finished - envelope.scheduled_monotonic) * 1000.0
|
||||
health, reasons = classify_health(
|
||||
source_available=True,
|
||||
fusion_state=str(replay_frame.frame["state"]),
|
||||
result_age_ms=age_ms,
|
||||
stale_after_ms=float(profile["health"]["stale_after_ms"]),
|
||||
unavailable_after_ms=float(profile["health"]["unavailable_after_ms"]),
|
||||
)
|
||||
health_counts[health] += 1
|
||||
world["delivery"] = {
|
||||
"health": health,
|
||||
"reasons": list(reasons),
|
||||
"result_age_ms": age_ms,
|
||||
"queue_policy": "bounded-latest-wins",
|
||||
}
|
||||
world_stream.write(json.dumps(world, ensure_ascii=False, allow_nan=False) + "\n")
|
||||
published = time.perf_counter()
|
||||
latency_ms = (published - envelope.scheduled_monotonic) * 1000.0
|
||||
recording.log("/telemetry/end_to_end_latency_ms", rr.Scalars([latency_ms]))
|
||||
recording.log("/telemetry/queue_depth", rr.Scalars([queue.snapshot().depth]))
|
||||
recording.log(
|
||||
"/telemetry/health_code",
|
||||
rr.Scalars(
|
||||
[{"healthy": 0, "degraded": 1, "stale": 2, "unavailable": 3}[health]]
|
||||
),
|
||||
)
|
||||
telemetry = {
|
||||
"schema_version": TELEMETRY_SCHEMA,
|
||||
"frame_index": int(replay_frame.frame["frame_index"]),
|
||||
"source_frame_index": int(replay_frame.frame["source_frame_index"]),
|
||||
"session_seconds": float(replay_frame.frame["session_seconds"]),
|
||||
"producer_lag_ms": envelope.producer_lag_ms,
|
||||
"queue_wait_ms": (dequeued - envelope.enqueued_monotonic) * 1000.0,
|
||||
"world_projection_ms": (projection_finished - stage_started) * 1000.0,
|
||||
"rerun_publish_ms": (rerun_finished - projection_finished) * 1000.0,
|
||||
"end_to_end_latency_ms": latency_ms,
|
||||
"health": health,
|
||||
"queue": _queue_document(queue),
|
||||
}
|
||||
telemetry_stream.write(
|
||||
json.dumps(telemetry, ensure_ascii=False, allow_nan=False) + "\n"
|
||||
)
|
||||
end_to_end.append(latency_ms)
|
||||
queue_wait.append(telemetry["queue_wait_ms"])
|
||||
processing.append(telemetry["world_projection_ms"])
|
||||
rerun_latency.append(telemetry["rerun_publish_ms"])
|
||||
producer_lag.append(envelope.producer_lag_ms)
|
||||
published_indices.append(int(replay_frame.frame["frame_index"]))
|
||||
producer.join(timeout=5)
|
||||
if producer.is_alive():
|
||||
raise RuntimeError("LAB E7 replay producer did not stop")
|
||||
if producer_error:
|
||||
raise RuntimeError("LAB E7 replay producer failed") from producer_error[0]
|
||||
if recording is not None:
|
||||
recording.flush()
|
||||
recording.disconnect()
|
||||
recording = None
|
||||
finished = time.perf_counter()
|
||||
queue_metrics = queue.snapshot()
|
||||
replay_span = (
|
||||
float(replay_frames[-1].frame["session_seconds"])
|
||||
- float(replay_frames[0].frame["session_seconds"])
|
||||
) / replay_speed
|
||||
replay_wall = finished - replay_started
|
||||
attempted = len(replay_frames)
|
||||
processed = len(published_indices)
|
||||
metrics = {
|
||||
"frames_attempted": attempted,
|
||||
"frames_processed": processed,
|
||||
"frames_dropped": queue_metrics.dropped_total,
|
||||
"drop_fraction": queue_metrics.dropped_total / attempted,
|
||||
"delivery_rate_hz": processed / replay_span,
|
||||
"replay_span_seconds": replay_span,
|
||||
"replay_wall_seconds": replay_wall,
|
||||
"preparation_wall_seconds": preparation_wall,
|
||||
"process_peak_rss_mib": _peak_rss_mib(),
|
||||
"health_counts": dict(health_counts),
|
||||
"queue": {
|
||||
"capacity": queue_metrics.capacity,
|
||||
"maximum_depth": queue_metrics.maximum_depth,
|
||||
"final_depth": queue_metrics.depth,
|
||||
"published": queue_metrics.published,
|
||||
"consumed": queue_metrics.consumed,
|
||||
"dropped_overflow": queue_metrics.dropped_overflow,
|
||||
"dropped_superseded": queue_metrics.dropped_superseded,
|
||||
},
|
||||
"end_to_end_latency_ms": {
|
||||
"mean": float(np.mean(end_to_end)),
|
||||
"p50": _percentile(end_to_end, 50),
|
||||
"p95": _percentile(end_to_end, 95),
|
||||
"max": max(end_to_end),
|
||||
},
|
||||
"queue_wait_ms": {
|
||||
"mean": float(np.mean(queue_wait)),
|
||||
"p95": _percentile(queue_wait, 95),
|
||||
"max": max(queue_wait),
|
||||
},
|
||||
"world_projection_ms": {
|
||||
"mean": float(np.mean(processing)),
|
||||
"p95": _percentile(processing, 95),
|
||||
"max": max(processing),
|
||||
},
|
||||
"rerun_publish_ms": {
|
||||
"mean": float(np.mean(rerun_latency)),
|
||||
"p95": _percentile(rerun_latency, 95),
|
||||
"max": max(rerun_latency),
|
||||
},
|
||||
"producer_lag_ms": {
|
||||
"mean": float(np.mean(producer_lag)),
|
||||
"p95": _percentile(producer_lag, 95),
|
||||
"max": max(producer_lag),
|
||||
},
|
||||
"published_frame_index_start": min(published_indices),
|
||||
"published_frame_index_end": max(published_indices),
|
||||
}
|
||||
accepted, rejection_reasons = _accepted(profile, metrics)
|
||||
report = {
|
||||
"schema_version": REPORT_SCHEMA,
|
||||
"result_id": result_id,
|
||||
"created_at_utc": datetime.now(UTC).isoformat(timespec="milliseconds").replace(
|
||||
"+00:00", "Z"
|
||||
),
|
||||
"identity": identity,
|
||||
"acceptance": {
|
||||
"accepted": accepted,
|
||||
"mode": profile["mode"],
|
||||
"rejection_reasons": rejection_reasons,
|
||||
"navigation_or_safety_accepted": False,
|
||||
"scope": "replay-scheduler-and-world-state-not-live-inference",
|
||||
},
|
||||
"metrics": metrics,
|
||||
}
|
||||
report_path.write_text(
|
||||
json.dumps(report, ensure_ascii=False, indent=2, sort_keys=True) + "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
artifacts = [
|
||||
_artifact(
|
||||
world_path,
|
||||
"live-world-state",
|
||||
"application/x-ndjson",
|
||||
"missioncore.live-perception-world-state/v1",
|
||||
),
|
||||
_artifact(
|
||||
telemetry_path,
|
||||
"live-telemetry",
|
||||
"application/x-ndjson",
|
||||
TELEMETRY_SCHEMA,
|
||||
),
|
||||
_artifact(rrd_path, "live-rerun", "application/vnd.rerun.rrd"),
|
||||
_artifact(report_path, "live-run-report", "application/json", REPORT_SCHEMA),
|
||||
]
|
||||
result = {
|
||||
"schema_version": RESULT_SCHEMA,
|
||||
"result_id": result_id,
|
||||
"identity_sha256": identity_sha256,
|
||||
"identity": identity,
|
||||
"ground_truth": False,
|
||||
"publication_scope": "replay-qualification-only",
|
||||
"acceptance_state": "accepted" if accepted else "rejected",
|
||||
"metrics": metrics,
|
||||
"artifacts": artifacts,
|
||||
}
|
||||
(staging / "result.json").write_text(
|
||||
json.dumps(result, ensure_ascii=False, indent=2, sort_keys=True) + "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
staging.rename(output)
|
||||
validated = validate_live_replay_qualification_result(
|
||||
args.job,
|
||||
args.tracking,
|
||||
args.semantic,
|
||||
args.e6_result,
|
||||
output,
|
||||
)
|
||||
if validated.result_id != result_id or validated.accepted != accepted:
|
||||
raise RuntimeError("LAB E7 published result failed identity validation")
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"result_id": result_id,
|
||||
"output": str(output),
|
||||
"accepted": accepted,
|
||||
"metrics": metrics,
|
||||
},
|
||||
ensure_ascii=False,
|
||||
)
|
||||
)
|
||||
return 0 if accepted else 2
|
||||
except BaseException:
|
||||
if recording is not None:
|
||||
recording.disconnect()
|
||||
queue.close()
|
||||
shutil.rmtree(staging, ignore_errors=True)
|
||||
raise
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,153 @@
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[ValidateRange(1, 1000)] [int]$FreeGiBFloor = 360,
|
||||
[string]$RuntimeRoot = "D:\NDC_MISSIONCORE\runtime",
|
||||
[string]$ContainerImage = "nvcr.io/nvidia/tritonserver:26.06-py3@sha256:58df7489c3f2276f9591d500a012dee03e23d35543ce3c390b4c001e6bf90794",
|
||||
[string]$RuntimeName = "perception-e15-media-pyav180-lz445-v1"
|
||||
)
|
||||
|
||||
$ErrorActionPreference = "Stop"
|
||||
$ProgressPreference = "SilentlyContinue"
|
||||
|
||||
function Assert-LastExitCode([string]$Operation) {
|
||||
if ($LASTEXITCODE -ne 0) { throw "$Operation failed with exit code $LASTEXITCODE" }
|
||||
}
|
||||
|
||||
function Resolve-DDirectory([string]$Path, [string]$Label) {
|
||||
$item = Get-Item -LiteralPath (Resolve-Path -LiteralPath $Path).Path -Force
|
||||
$root = [IO.Path]::GetPathRoot($item.FullName).TrimEnd("\")
|
||||
if (-not $item.PSIsContainer -or ($item.Attributes -band [IO.FileAttributes]::ReparsePoint) -or $root -ine "D:") {
|
||||
throw "$Label must be a real D: directory"
|
||||
}
|
||||
return $item.FullName
|
||||
}
|
||||
|
||||
function Convert-ToDockerPath([string]$Path) { return $Path.Replace("\", "/") }
|
||||
|
||||
function Assert-FreeSpace([string]$Phase) {
|
||||
$free = [int64](Get-PSDrive -Name D).Free
|
||||
$floor = [int64]$FreeGiBFloor * 1GB
|
||||
Write-Host (
|
||||
"DISK_GUARD PHASE={0} DRIVE=D FREE_BYTES={1} FREE_GIB={2} FLOOR_GIB={3}" -f
|
||||
$Phase, $free, [math]::Round($free / 1GB, 3), $FreeGiBFloor
|
||||
)
|
||||
if ($free -lt ($floor + 2GB)) {
|
||||
throw "D: lacks the guarded LAB E15 media-runtime reserve during $Phase"
|
||||
}
|
||||
return $free
|
||||
}
|
||||
|
||||
function Get-PayloadDigest([string]$Root) {
|
||||
$resolved = Resolve-DDirectory $Root "E15 media runtime"
|
||||
$lines = @(
|
||||
Get-ChildItem -LiteralPath $resolved -Recurse -File -Force |
|
||||
Where-Object { $_.Name -ne "manifest.json" } |
|
||||
Sort-Object FullName |
|
||||
ForEach-Object {
|
||||
$relative = $_.FullName.Substring($resolved.Length).TrimStart("\").Replace("\", "/")
|
||||
$hash = (Get-FileHash -LiteralPath $_.FullName -Algorithm SHA256).Hash.ToLowerInvariant()
|
||||
"{0}`t{1}`t{2}" -f $relative, $_.Length, $hash
|
||||
}
|
||||
)
|
||||
if ($lines.Count -lt 4) { throw "E15 media runtime payload is incomplete" }
|
||||
$bytes = [Text.Encoding]::UTF8.GetBytes(($lines -join "`n") + "`n")
|
||||
$hasher = [Security.Cryptography.SHA256]::Create()
|
||||
try {
|
||||
return ([BitConverter]::ToString($hasher.ComputeHash($bytes))).Replace("-", "").ToLowerInvariant()
|
||||
}
|
||||
finally { $hasher.Dispose() }
|
||||
}
|
||||
|
||||
function Assert-Runtime([string]$Path) {
|
||||
$root = Resolve-DDirectory $Path "E15 media runtime"
|
||||
$manifestPath = Join-Path $root "manifest.json"
|
||||
if (-not (Test-Path -LiteralPath $manifestPath -PathType Leaf)) {
|
||||
throw "E15 media runtime manifest is missing"
|
||||
}
|
||||
$manifest = Get-Content -LiteralPath $manifestPath -Raw | ConvertFrom-Json
|
||||
if (
|
||||
$manifest.schema_version -ne "missioncore.e15-media-runtime/v1" -or
|
||||
$manifest.runtime_name -ne $RuntimeName -or
|
||||
$manifest.container_image -ne $ContainerImage -or
|
||||
$manifest.packages.av -ne "18.0.0" -or
|
||||
$manifest.packages.lz4 -ne "4.4.5" -or
|
||||
$manifest.payload_sha256 -ne (Get-PayloadDigest $root)
|
||||
) { throw "E15 media runtime identity changed" }
|
||||
|
||||
$dockerRoot = Convert-ToDockerPath $root
|
||||
$verifyCode = "import av,lz4.version;print(av.__version__);print(lz4.version.version)"
|
||||
$verifyOutput = @(& docker run --rm --network none --read-only `
|
||||
--security-opt "no-new-privileges:true" --cap-drop ALL --pids-limit 64 `
|
||||
--tmpfs "/tmp:rw,noexec,nosuid,size=64m" `
|
||||
-e "PYTHONPATH=/opt/media" -e "PYTHONDONTWRITEBYTECODE=1" `
|
||||
-v ("{0}:/opt/media:ro" -f $dockerRoot) `
|
||||
--entrypoint python3 $ContainerImage -c $verifyCode)
|
||||
Assert-LastExitCode "E15 media runtime verification"
|
||||
if ($verifyOutput.Count -ne 2 -or $verifyOutput[0] -ne "18.0.0" -or $verifyOutput[1] -ne "4.4.5") {
|
||||
throw "E15 media runtime package versions changed"
|
||||
}
|
||||
Write-Host "MEDIA_RUNTIME_OK pyav=18.0.0 lz4=4.4.5"
|
||||
return $root
|
||||
}
|
||||
|
||||
$runtime = Resolve-DDirectory $RuntimeRoot "Runtime root"
|
||||
$derived = Resolve-DDirectory (Join-Path $runtime "derived") "Runtime derived root"
|
||||
$destination = Join-Path $derived $RuntimeName
|
||||
$freeBefore = Assert-FreeSpace "media-runtime-preflight"
|
||||
|
||||
& docker image inspect $ContainerImage *> $null
|
||||
Assert-LastExitCode "Pinned container image inspection"
|
||||
|
||||
if (Test-Path -LiteralPath $destination) {
|
||||
$resolved = Assert-Runtime $destination
|
||||
Write-Output "STATE=existing-verified"
|
||||
Write-Output ("MEDIA_RUNTIME_ROOT={0}" -f $resolved)
|
||||
Write-Output ("DISK_FREE_BYTES_BEFORE={0}" -f $freeBefore)
|
||||
Write-Output ("DISK_FREE_BYTES_AFTER={0}" -f ([int64](Get-PSDrive -Name D).Free))
|
||||
return
|
||||
}
|
||||
|
||||
$token = [Guid]::NewGuid().ToString("N")
|
||||
$staging = Join-Path $derived (".{0}-{1}.tmp" -f $RuntimeName, $token)
|
||||
$null = New-Item -ItemType Directory -Path $staging
|
||||
$completed = $false
|
||||
try {
|
||||
$dockerStaging = Convert-ToDockerPath $staging
|
||||
Write-Output "PHASE=media-runtime-install-start"
|
||||
& docker run --rm --network bridge --read-only `
|
||||
--security-opt "no-new-privileges:true" --cap-drop ALL --pids-limit 128 `
|
||||
--tmpfs "/tmp:rw,nosuid,size=1g" `
|
||||
-e "PIP_DISABLE_PIP_VERSION_CHECK=1" -e "PYTHONDONTWRITEBYTECODE=1" `
|
||||
-v ("{0}:/target:rw" -f $dockerStaging) `
|
||||
--entrypoint python3 $ContainerImage -m pip install `
|
||||
--no-cache-dir --only-binary ":all:" --target /target `
|
||||
"av==18.0.0" "lz4==4.4.5"
|
||||
Assert-LastExitCode "E15 media runtime installation"
|
||||
|
||||
$payloadSha256 = Get-PayloadDigest $staging
|
||||
$manifest = [ordered]@{
|
||||
schema_version = "missioncore.e15-media-runtime/v1"
|
||||
runtime_name = $RuntimeName
|
||||
created_at_utc = [DateTime]::UtcNow.ToString("yyyy-MM-ddTHH:mm:ss.fffZ")
|
||||
container_image = $ContainerImage
|
||||
packages = [ordered]@{ av = "18.0.0"; lz4 = "4.4.5" }
|
||||
payload_sha256 = $payloadSha256
|
||||
storage_scope = "D-only-immutable-runtime"
|
||||
}
|
||||
$manifest | ConvertTo-Json -Depth 8 | Set-Content -LiteralPath (Join-Path $staging "manifest.json") -Encoding utf8
|
||||
$null = Assert-Runtime $staging
|
||||
Move-Item -LiteralPath $staging -Destination $destination
|
||||
$completed = $true
|
||||
$resolved = Assert-Runtime $destination
|
||||
$freeAfter = Assert-FreeSpace "media-runtime-published"
|
||||
Write-Output "STATE=created-verified"
|
||||
Write-Output ("MEDIA_RUNTIME_ROOT={0}" -f $resolved)
|
||||
Write-Output ("PAYLOAD_SHA256={0}" -f $payloadSha256)
|
||||
Write-Output ("DISK_FREE_BYTES_BEFORE={0}" -f $freeBefore)
|
||||
Write-Output ("DISK_FREE_BYTES_AFTER={0}" -f $freeAfter)
|
||||
}
|
||||
finally {
|
||||
if (-not $completed -and (Test-Path -LiteralPath $staging)) {
|
||||
Remove-Item -LiteralPath $staging -Recurse -Force
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,313 @@
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[Parameter(Mandatory = $true)] [string]$JobRoot,
|
||||
[Parameter(Mandatory = $true)] [string]$RunnerPath,
|
||||
[Parameter(Mandatory = $true)] [string]$ProfilePath,
|
||||
[Parameter(Mandatory = $true)] [string]$DetectorProfilePath,
|
||||
[Parameter(Mandatory = $true)] [string]$SemanticProfilePath,
|
||||
[Parameter(Mandatory = $true)] [string]$ValidFovRoot,
|
||||
[Parameter(Mandatory = $true)] [string]$LidarPackRoot,
|
||||
[ValidateRange(0, 1000000)] [int]$StartFrame = 1000,
|
||||
[ValidateRange(0, 1000000)] [int]$EndFrame = 1600,
|
||||
[ValidateRange(1, 1000)] [int]$FreeGiBFloor = 360,
|
||||
[string]$RuntimeRoot = "D:\NDC_MISSIONCORE\runtime",
|
||||
[string]$ModelRoot = "D:\NDC_MISSIONCORE\runtime\models\yolox_s",
|
||||
[string]$TritonContainer = "mission-core-triton",
|
||||
[string]$ContainerImage = "nvcr.io/nvidia/tritonserver:26.06-py3"
|
||||
)
|
||||
|
||||
$ErrorActionPreference = "Stop"
|
||||
$ProgressPreference = "SilentlyContinue"
|
||||
|
||||
function Assert-LastExitCode([string]$Operation) {
|
||||
if ($LASTEXITCODE -ne 0) { throw "$Operation failed with exit code $LASTEXITCODE" }
|
||||
}
|
||||
|
||||
function Resolve-DDirectory([string]$Path, [string]$Label) {
|
||||
$item = Get-Item -LiteralPath (Resolve-Path -LiteralPath $Path).Path -Force
|
||||
$root = [IO.Path]::GetPathRoot($item.FullName).TrimEnd("\")
|
||||
if (-not $item.PSIsContainer -or ($item.Attributes -band [IO.FileAttributes]::ReparsePoint) -or $root -ine "D:") {
|
||||
throw "$Label must be a real D: directory"
|
||||
}
|
||||
return $item.FullName
|
||||
}
|
||||
|
||||
function Resolve-DFile([string]$Path, [string]$Label) {
|
||||
$item = Get-Item -LiteralPath (Resolve-Path -LiteralPath $Path).Path -Force
|
||||
$root = [IO.Path]::GetPathRoot($item.FullName).TrimEnd("\")
|
||||
if ($item.PSIsContainer -or ($item.Attributes -band [IO.FileAttributes]::ReparsePoint) -or $root -ine "D:") {
|
||||
throw "$Label must be a regular D: file"
|
||||
}
|
||||
return $item.FullName
|
||||
}
|
||||
|
||||
function Convert-ToDockerPath([string]$Path) { return $Path.Replace("\", "/") }
|
||||
function Get-DFreeBytes { return [int64](Get-PSDrive -Name D).Free }
|
||||
function Assert-FreeSpace([string]$Phase, [int64]$RequiredAdditionalBytes = 0) {
|
||||
$free = Get-DFreeBytes
|
||||
$floor = [int64]$FreeGiBFloor * 1GB
|
||||
Write-Host (
|
||||
"DISK_GUARD PHASE={0} DRIVE=D FREE_BYTES={1} FREE_GIB={2} FLOOR_GIB={3} REQUIRED_ADDITIONAL_GIB={4}" -f
|
||||
$Phase, $free, [math]::Round($free / 1GB, 3), $FreeGiBFloor,
|
||||
[math]::Round($RequiredAdditionalBytes / 1GB, 3)
|
||||
)
|
||||
if ($free -lt ($floor + $RequiredAdditionalBytes)) {
|
||||
throw "D: lacks the guarded LAB E10 reserve during $Phase"
|
||||
}
|
||||
return $free
|
||||
}
|
||||
|
||||
function Test-TritonModelReady {
|
||||
try {
|
||||
$response = Invoke-WebRequest -UseBasicParsing -TimeoutSec 10 `
|
||||
-Uri "http://127.0.0.1:8000/v2/models/yolox_s/ready"
|
||||
return $response.StatusCode -eq 200
|
||||
}
|
||||
catch { return $false }
|
||||
}
|
||||
|
||||
$jobDirectory = Resolve-DDirectory $JobRoot "Job root"
|
||||
$runner = Resolve-DFile $RunnerPath "LAB E10 runner"
|
||||
$profile = Resolve-DFile $ProfilePath "LAB E10 profile"
|
||||
$detectorProfile = Resolve-DFile $DetectorProfilePath "Detector profile"
|
||||
$semanticProfile = Resolve-DFile $SemanticProfilePath "Semantic profile"
|
||||
$validFov = Resolve-DDirectory $ValidFovRoot "Valid-FOV root"
|
||||
$lidarPack = Resolve-DDirectory $LidarPackRoot "LiDAR replay pack"
|
||||
$runtime = Resolve-DDirectory $RuntimeRoot "Runtime root"
|
||||
$model = Resolve-DDirectory $ModelRoot "YOLOX model root"
|
||||
$runnerRoot = Split-Path $runner -Parent
|
||||
foreach ($path in @($profile, $detectorProfile, $semanticProfile)) {
|
||||
if ((Split-Path $path -Parent) -ne $runnerRoot) { throw "Runner and profiles must share one mount" }
|
||||
}
|
||||
foreach ($dependency in @(
|
||||
"e10_fusion_runtime.py",
|
||||
"run_e9_multirate_perception.py",
|
||||
"run_e8_realtime_tracking.py",
|
||||
"run_e5_instance_tracking.py",
|
||||
"run_e4_full_session_segmentation.py",
|
||||
"run_recorded_perception_epoch.py",
|
||||
"run_e3_rectified_segmentation.py",
|
||||
"run_evaluation_prelabels.py"
|
||||
)) {
|
||||
$null = Resolve-DFile (Join-Path $runnerRoot $dependency) "LAB E10 dependency"
|
||||
}
|
||||
|
||||
$job = Get-Content -LiteralPath (Join-Path $jobDirectory "job.json") -Raw | ConvertFrom-Json
|
||||
if ($job.schema_version -ne "missioncore.compute-job/v1" -or $job.job_id -ne (Split-Path $jobDirectory -Leaf)) {
|
||||
throw "Compute job manifest is incompatible"
|
||||
}
|
||||
$sourceId = [string]$job.input.source_id
|
||||
$epoch = [int]$job.input.codec_epoch
|
||||
$fullFrameCount = [int]$job.input.segment_count
|
||||
$clipFrameCount = $EndFrame - $StartFrame + 1
|
||||
$timelineStart = [double]$job.input.timeline.start_seconds
|
||||
if ($sourceId -ne "sensor.camera.right" -or $StartFrame -lt 0 -or $EndFrame -lt $StartFrame -or $EndFrame -ge $fullFrameCount -or $clipFrameCount -lt 2) {
|
||||
throw "LAB E10 clip escapes the camera job"
|
||||
}
|
||||
$lidarManifest = Get-Content -LiteralPath (Join-Path $lidarPack "manifest.json") -Raw | ConvertFrom-Json
|
||||
if (
|
||||
$lidarManifest.schema_version -ne "missioncore.e10-lidar-replay-pack/v1" -or
|
||||
$lidarManifest.identity.job_id -ne $job.job_id -or
|
||||
[int]$lidarManifest.identity.source_start_frame_index -ne $StartFrame -or
|
||||
[int]$lidarManifest.identity.source_end_frame_index -ne $EndFrame -or
|
||||
[int]$lidarManifest.identity.frame_count -ne $clipFrameCount
|
||||
) { throw "LAB E10 LiDAR pack differs from the selected camera clip" }
|
||||
Write-Output ("PHASE=inputs-validated JOB={0} CLIP={1}-{2} FRAMES={3} LIDAR_PACK={4}" -f $job.job_id, $StartFrame, $EndFrame, $clipFrameCount, $lidarManifest.pack_id)
|
||||
|
||||
$epochRoot = Resolve-DDirectory (Join-Path $jobDirectory ("input\camera\{0}\epoch-{1}" -f $sourceId, $epoch)) "Camera epoch"
|
||||
$initPath = Resolve-DFile (Join-Path $epochRoot "init.mp4") "Camera init"
|
||||
$segmentsRoot = Resolve-DDirectory (Join-Path $epochRoot "segments") "Camera segments"
|
||||
$derivedRoot = Join-Path $runtime "derived"
|
||||
$tmpRoot = Join-Path $runtime "tmp"
|
||||
$cacheRoot = Resolve-DDirectory (Join-Path $runtime "cache\perception-e3-models-v1") "EoMT cache"
|
||||
$e3Environment = Resolve-DDirectory (Join-Path $derivedRoot "perception-e3-opencv413092-v1") "E3 environment"
|
||||
$torchEnvironment = Resolve-DDirectory (Join-Path $derivedRoot "perception-p0-env-v1") "Torch environment"
|
||||
$transformersEnvironment = Resolve-DDirectory (Join-Path $derivedRoot "perception-p0-transformers4576-v1") "Transformers environment"
|
||||
|
||||
$pixelWorkingSet = [int64]$clipFrameCount * 800 * 600 * 3
|
||||
$partialStreamReserve = [int64][math]::Ceiling(([int64]$job.input.byte_length * ($EndFrame + 1) / $fullFrameCount) * 1.2)
|
||||
$workingSetReserve = [int64][math]::Ceiling($pixelWorkingSet * 1.4) + $partialStreamReserve + 3GB
|
||||
$freeBytesBefore = Assert-FreeSpace "preflight" $workingSetReserve
|
||||
|
||||
& docker image inspect $ContainerImage *> $null
|
||||
Assert-LastExitCode "Existing container image inspection"
|
||||
$tritonState = docker inspect --format "{{.State.Running}}" $TritonContainer
|
||||
Assert-LastExitCode "Triton container inspection"
|
||||
if ($tritonState.Trim().ToLowerInvariant() -ne "true") { throw "LAB E10 requires the existing Triton container" }
|
||||
|
||||
$runnerName = Split-Path $runner -Leaf
|
||||
$profileName = Split-Path $profile -Leaf
|
||||
$detectorProfileName = Split-Path $detectorProfile -Leaf
|
||||
$semanticProfileName = Split-Path $semanticProfile -Leaf
|
||||
$lidarMount = "/" + (Split-Path $lidarPack -Leaf)
|
||||
$orchestratorSha256 = (Get-FileHash -LiteralPath $PSCommandPath -Algorithm SHA256).Hash.ToLowerInvariant()
|
||||
$commonMounts = @(
|
||||
"-e", "PYTHONPATH=/runner:/opt/transformers:/opt/env",
|
||||
"-e", "TORCH_HOME=/cache/torch",
|
||||
"-e", "HF_HOME=/cache/huggingface",
|
||||
"-e", "HF_HUB_OFFLINE=1",
|
||||
"-e", "TRANSFORMERS_OFFLINE=1",
|
||||
"-v", ((Convert-ToDockerPath $jobDirectory) + ":/job:ro"),
|
||||
"-v", ((Convert-ToDockerPath $validFov) + ":/valid-fov:ro"),
|
||||
"-v", ((Convert-ToDockerPath $lidarPack) + (":{0}:ro" -f $lidarMount)),
|
||||
"-v", ((Convert-ToDockerPath $model) + ":/model:ro"),
|
||||
"-v", ((Convert-ToDockerPath $cacheRoot) + ":/cache:ro"),
|
||||
"-v", ((Convert-ToDockerPath $e3Environment) + ":/environment:ro"),
|
||||
"-v", ((Convert-ToDockerPath $torchEnvironment) + ":/opt/env:ro"),
|
||||
"-v", ((Convert-ToDockerPath $transformersEnvironment) + ":/opt/transformers:ro"),
|
||||
"-v", ((Convert-ToDockerPath $runnerRoot) + ":/runner:ro")
|
||||
)
|
||||
$commonRunnerArgs = @(
|
||||
"--job", "/job",
|
||||
"--profile", ("/runner/{0}" -f $profileName),
|
||||
"--detector-profile", ("/runner/{0}" -f $detectorProfileName),
|
||||
"--semantic-profile", ("/runner/{0}" -f $semanticProfileName),
|
||||
"--valid-fov-root", "/valid-fov",
|
||||
"--model-root", "/model",
|
||||
"--cache", "/cache",
|
||||
"--environment", "/environment",
|
||||
"--lidar-pack", $lidarMount
|
||||
)
|
||||
$preflightArgs = @(
|
||||
"run", "--rm", "--gpus", "all", "--network", "none", "--read-only",
|
||||
"--security-opt", "no-new-privileges:true", "--cap-drop", "ALL",
|
||||
"--pids-limit", "512", "--shm-size", "4g",
|
||||
"--tmpfs", "/tmp:rw,noexec,nosuid,size=2g"
|
||||
) + $commonMounts + @(
|
||||
"--entrypoint", "python3", $ContainerImage,
|
||||
("/runner/{0}" -f $runnerName), "preflight"
|
||||
) + $commonRunnerArgs
|
||||
Write-Output "PHASE=e10-preflight-start"
|
||||
& docker @preflightArgs
|
||||
Assert-LastExitCode "LAB E10 preflight"
|
||||
Write-Output "PHASE=e10-preflight-complete"
|
||||
|
||||
$modelWasReady = Test-TritonModelReady
|
||||
$loadedByRun = $false
|
||||
if (-not $modelWasReady) {
|
||||
Invoke-WebRequest -UseBasicParsing -TimeoutSec 60 -Uri "http://127.0.0.1:8000/v2/repository/models/yolox_s/load" -Method Post -ContentType "application/json" -Body "{}" *> $null
|
||||
$loadedByRun = $true
|
||||
$deadline = [DateTime]::UtcNow.AddSeconds(60)
|
||||
while (-not (Test-TritonModelReady)) {
|
||||
if ([DateTime]::UtcNow -ge $deadline) { throw "YOLOX-S did not become ready" }
|
||||
Start-Sleep -Milliseconds 500
|
||||
}
|
||||
}
|
||||
Write-Output ("PHASE=e10-model-ready LOADED_BY_RUN={0}" -f $loadedByRun)
|
||||
|
||||
$token = [Guid]::NewGuid().ToString("N")
|
||||
$workRoot = Join-Path $tmpRoot ("{0}-e10-{1}" -f $job.job_id, $token)
|
||||
$framesRoot = Join-Path $workRoot "frames"
|
||||
$streamPath = Join-Path $workRoot "camera.mp4"
|
||||
$ptsPath = Join-Path $workRoot "pts.json"
|
||||
$timelinePath = Join-Path $workRoot "timeline.jsonl"
|
||||
$publishRoot = Join-Path $derivedRoot (".{0}-e10-{1}.publish" -f $job.job_id, $token)
|
||||
$stagingRoot = Join-Path $publishRoot "output"
|
||||
$null = New-Item -ItemType Directory -Path $framesRoot
|
||||
$null = New-Item -ItemType Directory -Path $publishRoot
|
||||
$completed = $false
|
||||
$totalWatch = [Diagnostics.Stopwatch]::StartNew()
|
||||
|
||||
try {
|
||||
$stream = [IO.File]::Open($streamPath, [IO.FileMode]::CreateNew, [IO.FileAccess]::Write, [IO.FileShare]::None)
|
||||
try {
|
||||
$input = [IO.File]::OpenRead($initPath)
|
||||
try { $input.CopyTo($stream) } finally { $input.Dispose() }
|
||||
for ($sequence = 1; $sequence -le ($EndFrame + 1); $sequence++) {
|
||||
$path = Resolve-DFile (Join-Path $segmentsRoot ("{0}.m4s" -f $sequence)) "Camera segment"
|
||||
$input = [IO.File]::OpenRead($path)
|
||||
try { $input.CopyTo($stream) } finally { $input.Dispose() }
|
||||
}
|
||||
$stream.Flush($true)
|
||||
}
|
||||
finally { $stream.Dispose() }
|
||||
$null = Assert-FreeSpace "post-stream-reconstruction"
|
||||
|
||||
$extractWatch = [Diagnostics.Stopwatch]::StartNew()
|
||||
$selectFilter = "select='between(n,{0},{1})'" -f $StartFrame, $EndFrame
|
||||
& ffmpeg -hide_banner -loglevel fatal -i $streamPath -map 0:v:0 -vf $selectFilter -fps_mode passthrough (Join-Path $framesRoot "frame-%06d.png")
|
||||
Assert-LastExitCode "LAB E10 camera extraction"
|
||||
& ffprobe -v error -select_streams v:0 -show_entries frame=best_effort_timestamp_time -of json $streamPath | Set-Content -LiteralPath $ptsPath -Encoding utf8
|
||||
Assert-LastExitCode "LAB E10 timestamp probe"
|
||||
$decoded = @(Get-ChildItem -LiteralPath $framesRoot -File -Filter "frame-*.png")
|
||||
$pts = @((Get-Content -LiteralPath $ptsPath -Raw | ConvertFrom-Json).frames)
|
||||
if ($decoded.Count -ne $clipFrameCount -or $pts.Count -ne ($EndFrame + 1)) { throw "LAB E10 decoded frame count changed" }
|
||||
$firstEpochSeconds = [double]::Parse(([string]$pts[0].best_effort_timestamp_time).Trim(), [Globalization.CultureInfo]::InvariantCulture)
|
||||
$writer = [IO.StreamWriter]::new($timelinePath, $false, [Text.UTF8Encoding]::new($false))
|
||||
try {
|
||||
for ($local = 0; $local -lt $clipFrameCount; $local++) {
|
||||
$source = $StartFrame + $local
|
||||
$epochSeconds = [double]::Parse(([string]$pts[$source].best_effort_timestamp_time).Trim(), [Globalization.CultureInfo]::InvariantCulture) - $firstEpochSeconds
|
||||
$row = [ordered]@{
|
||||
frame_index = $local
|
||||
sequence = $local + 1
|
||||
source_frame_index = $source
|
||||
source_sequence = $source + 1
|
||||
epoch_seconds = $epochSeconds
|
||||
session_seconds = $timelineStart + $epochSeconds
|
||||
}
|
||||
$writer.WriteLine(($row | ConvertTo-Json -Compress))
|
||||
}
|
||||
$writer.Flush()
|
||||
}
|
||||
finally { $writer.Dispose() }
|
||||
$extractWatch.Stop()
|
||||
$freePostExtract = Assert-FreeSpace "post-frame-extraction"
|
||||
|
||||
$runArgs = @(
|
||||
"run", "--rm", "--gpus", "all", "--network", ("container:{0}" -f $TritonContainer),
|
||||
"--read-only", "--security-opt", "no-new-privileges:true", "--cap-drop", "ALL",
|
||||
"--pids-limit", "512", "--shm-size", "4g",
|
||||
"--tmpfs", "/tmp:rw,noexec,nosuid,size=2g"
|
||||
) + $commonMounts + @(
|
||||
"-v", ((Convert-ToDockerPath $framesRoot) + ":/frames:ro"),
|
||||
"-v", ((Convert-ToDockerPath $workRoot) + ":/work:ro"),
|
||||
"-v", ((Convert-ToDockerPath $publishRoot) + ":/publish:rw"),
|
||||
"--entrypoint", "python3", $ContainerImage,
|
||||
("/runner/{0}" -f $runnerName), "run"
|
||||
) + $commonRunnerArgs + @(
|
||||
"--frames", "/frames",
|
||||
"--timeline", "/work/timeline.jsonl",
|
||||
"--output", "/publish/output",
|
||||
"--triton-url", "http://127.0.0.1:8000",
|
||||
"--free-bytes-floor", [string]([int64]$FreeGiBFloor * 1GB),
|
||||
"--orchestrator-sha256", $orchestratorSha256,
|
||||
"--container-image", $ContainerImage
|
||||
)
|
||||
Write-Output ("PHASE=e10-integrated-replay-start FRAMES={0}" -f $clipFrameCount)
|
||||
& docker @runArgs
|
||||
Assert-LastExitCode "LAB E10 integrated replay"
|
||||
$freePostReplay = Assert-FreeSpace "post-replay"
|
||||
|
||||
$result = Get-Content -LiteralPath (Join-Path $stagingRoot "result.json") -Raw | ConvertFrom-Json
|
||||
if ($result.schema_version -ne "missioncore.e10-integrated-perception-result/v1" -or $result.result_id -notmatch "^e10-integrated-perception-[a-f0-9]{64}$") {
|
||||
throw "LAB E10 result manifest is incompatible"
|
||||
}
|
||||
$finalRoot = Join-Path $derivedRoot ([string]$result.result_id)
|
||||
if (Test-Path -LiteralPath $finalRoot) { throw "Immutable LAB E10 result already exists" }
|
||||
Move-Item -LiteralPath $stagingRoot -Destination $finalRoot
|
||||
Remove-Item -LiteralPath $publishRoot -Force
|
||||
$completed = $true
|
||||
$totalWatch.Stop()
|
||||
$freeFinal = Assert-FreeSpace "post-publication"
|
||||
Write-Output ("RESULT_ROOT={0}" -f $finalRoot)
|
||||
Write-Output ("RESULT_ID={0}" -f $result.result_id)
|
||||
Write-Output ("ACCEPTANCE_STATE={0}" -f $result.acceptance_state)
|
||||
Write-Output ("PREPARATION_SECONDS={0}" -f $extractWatch.Elapsed.TotalSeconds)
|
||||
Write-Output ("ORCHESTRATOR_WALL_SECONDS={0}" -f $totalWatch.Elapsed.TotalSeconds)
|
||||
Write-Output ("DISK_FREE_BYTES_BEFORE={0}" -f $freeBytesBefore)
|
||||
Write-Output ("DISK_FREE_BYTES_POST_EXTRACT={0}" -f $freePostExtract)
|
||||
Write-Output ("DISK_FREE_BYTES_POST_REPLAY={0}" -f $freePostReplay)
|
||||
Write-Output ("DISK_FREE_BYTES_FINAL={0}" -f $freeFinal)
|
||||
}
|
||||
finally {
|
||||
if (Test-Path -LiteralPath $workRoot) { Remove-Item -LiteralPath $workRoot -Recurse -Force }
|
||||
if (-not $completed -and (Test-Path -LiteralPath $publishRoot)) { Remove-Item -LiteralPath $publishRoot -Recurse -Force }
|
||||
if ($loadedByRun) {
|
||||
try {
|
||||
Invoke-WebRequest -UseBasicParsing -TimeoutSec 60 -Uri "http://127.0.0.1:8000/v2/repository/models/yolox_s/unload" -Method Post -ContentType "application/json" -Body "{}" *> $null
|
||||
Write-Output "PHASE=e10-model-state-restored"
|
||||
}
|
||||
catch { Write-Warning "LAB E10 could not restore the prior YOLOX-S state" }
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,100 @@
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[switch]$TokenStdin,
|
||||
[ValidatePattern("^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$")]
|
||||
[string]$RequestId = ("physical-k1-shadow-{0}" -f [Guid]::NewGuid().ToString("N")),
|
||||
[string]$PersistentContainer = "mission-core-perception-worker",
|
||||
[string]$PersistentOutputRoot = "D:\NDC_MISSIONCORE\runtime\derived\.perception-persistent-publish",
|
||||
[ValidateRange(1024, 65535)] [int]$PersistentPort = 18020,
|
||||
[ValidateRange(1, 1000)] [int]$FreeGiBFloor = 360
|
||||
)
|
||||
|
||||
$ErrorActionPreference = "Stop"
|
||||
$ProgressPreference = "SilentlyContinue"
|
||||
|
||||
function Assert-LastExitCode([string]$Operation) {
|
||||
if ($LASTEXITCODE -ne 0) { throw "$Operation failed with exit code $LASTEXITCODE" }
|
||||
}
|
||||
|
||||
function Resolve-DDirectory([string]$Path, [string]$Label) {
|
||||
$item = Get-Item -LiteralPath (Resolve-Path -LiteralPath $Path).Path -Force
|
||||
$root = [IO.Path]::GetPathRoot($item.FullName).TrimEnd("\")
|
||||
if (-not $item.PSIsContainer -or ($item.Attributes -band [IO.FileAttributes]::ReparsePoint) -or $root -ine "D:") {
|
||||
throw "$Label must be a real D: directory"
|
||||
}
|
||||
return $item.FullName
|
||||
}
|
||||
|
||||
function Get-DFreeBytes { return [int64](Get-PSDrive -Name D).Free }
|
||||
|
||||
if (-not $TokenStdin) { throw "Persistent shadow run requires the token through stdin" }
|
||||
$token = [Console]::In.ReadLine()
|
||||
if (-not $token -or $token.Length -lt 40 -or $token.Length -gt 512) {
|
||||
throw "Persistent shadow token is missing or malformed"
|
||||
}
|
||||
|
||||
$outputRoot = Resolve-DDirectory $PersistentOutputRoot "Persistent output root"
|
||||
$freeBefore = Get-DFreeBytes
|
||||
if ($freeBefore -lt ([int64]$FreeGiBFloor * 1GB + 512MB)) {
|
||||
throw "D: lacks the guarded persistent-run reserve"
|
||||
}
|
||||
$containerRunning = docker inspect --format "{{.State.Running}}" $PersistentContainer
|
||||
Assert-LastExitCode "Persistent worker inspection"
|
||||
if ($containerRunning.Trim().ToLowerInvariant() -ne "true") {
|
||||
throw "Persistent worker is not running"
|
||||
}
|
||||
$outputName = $RequestId
|
||||
$request = @{
|
||||
request_id = $RequestId
|
||||
output_name = $outputName
|
||||
token = $token
|
||||
} | ConvertTo-Json -Compress
|
||||
$token = $null
|
||||
$client = (
|
||||
"import sys,urllib.request,urllib.error;data=sys.stdin.buffer.read();" +
|
||||
"request=urllib.request.Request('http://127.0.0.1:{0}/run',data=data," -f $PersistentPort
|
||||
) + "headers={'Content-Type':'application/json'},method='POST');" +
|
||||
"`ntry:`n response=urllib.request.urlopen(request,timeout=3600); body=response.read(); code=response.status" +
|
||||
"`nexcept urllib.error.HTTPError as exc:`n body=exc.read(); code=exc.code" +
|
||||
"`nsys.stdout.buffer.write(body);raise SystemExit(0 if code==200 else 22)"
|
||||
try {
|
||||
$responseJson = $request | & docker exec -i $PersistentContainer python3 -c $client
|
||||
$request = $null
|
||||
Assert-LastExitCode "Persistent worker request"
|
||||
}
|
||||
finally {
|
||||
$token = $null
|
||||
$request = $null
|
||||
}
|
||||
$response = $responseJson | ConvertFrom-Json
|
||||
if ($response.request_id -ne $RequestId -or $response.models_reused -ne $true) {
|
||||
throw "Persistent worker response contract changed"
|
||||
}
|
||||
$staging = Join-Path $outputRoot $outputName
|
||||
$resultPath = Join-Path $staging "result.json"
|
||||
if (-not (Test-Path -LiteralPath $resultPath -PathType Leaf)) {
|
||||
throw "Persistent worker result manifest is missing"
|
||||
}
|
||||
$result = Get-Content -LiteralPath $resultPath -Raw | ConvertFrom-Json
|
||||
if (
|
||||
$result.schema_version -ne "missioncore.e15-shadow-inference-result/v1" -or
|
||||
$result.result_id -notmatch "^e15-shadow-inference-[a-f0-9]{64}$" -or
|
||||
$result.publication_scope -ne "live-shadow-diagnostic-only"
|
||||
) { throw "Persistent worker result manifest is incompatible" }
|
||||
$derivedRoot = Resolve-DDirectory (Split-Path $outputRoot -Parent) "Runtime derived root"
|
||||
$finalRoot = Join-Path $derivedRoot ([string]$result.result_id)
|
||||
if (Test-Path -LiteralPath $finalRoot) {
|
||||
throw "Immutable persistent-worker result already exists"
|
||||
}
|
||||
Move-Item -LiteralPath $staging -Destination $finalRoot
|
||||
$freeFinal = Get-DFreeBytes
|
||||
if ($freeFinal -lt ([int64]$FreeGiBFloor * 1GB)) {
|
||||
throw "D: crossed the guarded floor after persistent run"
|
||||
}
|
||||
Write-Output ("RESULT_ROOT={0}" -f $finalRoot)
|
||||
Write-Output ("RESULT_ID={0}" -f $result.result_id)
|
||||
Write-Output ("ACCEPTANCE_STATE={0}" -f $result.acceptance_state)
|
||||
Write-Output ("RUNNER_EXIT_CODE={0}" -f $response.exit_code)
|
||||
Write-Output "MODELS_REUSED=true"
|
||||
Write-Output ("DISK_FREE_BYTES_BEFORE={0}" -f $freeBefore)
|
||||
Write-Output ("DISK_FREE_BYTES_FINAL={0}" -f $freeFinal)
|
||||
@@ -0,0 +1,432 @@
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[Parameter(Mandatory = $true)] [string]$JobRoot,
|
||||
[Parameter(Mandatory = $true)] [string]$RunnerPath,
|
||||
[Parameter(Mandatory = $true)] [string]$LiveProfilePath,
|
||||
[Parameter(Mandatory = $true)] [string]$E14ProfilePath,
|
||||
[Parameter(Mandatory = $true)] [string]$DetectorProfilePath,
|
||||
[Parameter(Mandatory = $true)] [string]$SemanticProfilePath,
|
||||
[Parameter(Mandatory = $true)] [string]$ValidFovRoot,
|
||||
[Parameter(Mandatory = $true)] [string]$ProjectionPackRoot,
|
||||
[Parameter(Mandatory = $true)] [string]$PackageRoot,
|
||||
[switch]$PreflightOnly,
|
||||
[switch]$PersistentService,
|
||||
[switch]$TokenStdin,
|
||||
[ValidateRange(5, 3600)] [int]$MaximumDurationSeconds = 20,
|
||||
[ValidateRange(1, 1000)] [int]$FreeGiBFloor = 360,
|
||||
[ValidateRange(1024, 65535)] [int]$SourcePort = 18012,
|
||||
[string]$SourceHost = "host.docker.internal",
|
||||
[string]$SourcePath = "/api/v1/device-plugins/nodedc.device.xgrids-lixelkity-k1/live-perception-shadow",
|
||||
[string]$RuntimeRoot = "D:\NDC_MISSIONCORE\runtime",
|
||||
[string]$ModelRoot = "D:\NDC_MISSIONCORE\runtime\models\yolox_s",
|
||||
[string]$MediaRuntimeRoot = "D:\NDC_MISSIONCORE\runtime\derived\perception-e15-media-pyav180-lz445-v1",
|
||||
[string]$TritonContainer = "mission-core-triton",
|
||||
[string]$PersistentContainer = "mission-core-perception-worker",
|
||||
[string]$PersistentOutputRoot = "D:\NDC_MISSIONCORE\runtime\derived\.perception-persistent-publish",
|
||||
[ValidateRange(1024, 65535)] [int]$PersistentPort = 18020,
|
||||
[string]$ContainerImage = "nvcr.io/nvidia/tritonserver:26.06-py3@sha256:58df7489c3f2276f9591d500a012dee03e23d35543ce3c390b4c001e6bf90794"
|
||||
)
|
||||
|
||||
$ErrorActionPreference = "Stop"
|
||||
$ProgressPreference = "SilentlyContinue"
|
||||
|
||||
function Assert-LastExitCode([string]$Operation) {
|
||||
if ($LASTEXITCODE -ne 0) { throw "$Operation failed with exit code $LASTEXITCODE" }
|
||||
}
|
||||
|
||||
function Resolve-DDirectory([string]$Path, [string]$Label) {
|
||||
$item = Get-Item -LiteralPath (Resolve-Path -LiteralPath $Path).Path -Force
|
||||
$root = [IO.Path]::GetPathRoot($item.FullName).TrimEnd("\")
|
||||
if (-not $item.PSIsContainer -or ($item.Attributes -band [IO.FileAttributes]::ReparsePoint) -or $root -ine "D:") {
|
||||
throw "$Label must be a real D: directory"
|
||||
}
|
||||
return $item.FullName
|
||||
}
|
||||
|
||||
function Resolve-DFile([string]$Path, [string]$Label) {
|
||||
$item = Get-Item -LiteralPath (Resolve-Path -LiteralPath $Path).Path -Force
|
||||
$root = [IO.Path]::GetPathRoot($item.FullName).TrimEnd("\")
|
||||
if ($item.PSIsContainer -or ($item.Attributes -band [IO.FileAttributes]::ReparsePoint) -or $root -ine "D:") {
|
||||
throw "$Label must be a regular D: file"
|
||||
}
|
||||
return $item.FullName
|
||||
}
|
||||
|
||||
function Convert-ToDockerPath([string]$Path) { return $Path.Replace("\", "/") }
|
||||
function Get-DFreeBytes { return [int64](Get-PSDrive -Name D).Free }
|
||||
|
||||
function Assert-FreeSpace([string]$Phase, [int64]$RequiredAdditionalBytes = 0) {
|
||||
$free = Get-DFreeBytes
|
||||
$floor = [int64]$FreeGiBFloor * 1GB
|
||||
Write-Host (
|
||||
"DISK_GUARD PHASE={0} DRIVE=D FREE_BYTES={1} FREE_GIB={2} FLOOR_GIB={3} REQUIRED_ADDITIONAL_GIB={4}" -f
|
||||
$Phase, $free, [math]::Round($free / 1GB, 3), $FreeGiBFloor,
|
||||
[math]::Round($RequiredAdditionalBytes / 1GB, 3)
|
||||
)
|
||||
if ($free -lt ($floor + $RequiredAdditionalBytes)) {
|
||||
throw "D: lacks the guarded LAB E15 reserve during $Phase"
|
||||
}
|
||||
return $free
|
||||
}
|
||||
|
||||
function Test-TritonModelReady {
|
||||
try {
|
||||
$response = Invoke-WebRequest -UseBasicParsing -TimeoutSec 10 `
|
||||
-Uri "http://127.0.0.1:8000/v2/models/yolox_s/ready"
|
||||
return $response.StatusCode -eq 200
|
||||
}
|
||||
catch { return $false }
|
||||
}
|
||||
|
||||
$jobDirectory = Resolve-DDirectory $JobRoot "Job root"
|
||||
$runner = Resolve-DFile $RunnerPath "LAB E15 runner"
|
||||
$liveProfile = Resolve-DFile $LiveProfilePath "LAB E15 live profile"
|
||||
$e14Profile = Resolve-DFile $E14ProfilePath "Accepted E14 profile"
|
||||
$detectorProfile = Resolve-DFile $DetectorProfilePath "Detector profile"
|
||||
$semanticProfile = Resolve-DFile $SemanticProfilePath "Semantic profile"
|
||||
$validFov = Resolve-DDirectory $ValidFovRoot "Valid-FOV root"
|
||||
$projectionPack = Resolve-DDirectory $ProjectionPackRoot "E15 projection pack"
|
||||
$package = Resolve-DDirectory $PackageRoot "Mission Core package root"
|
||||
$runtime = Resolve-DDirectory $RuntimeRoot "Runtime root"
|
||||
$model = Resolve-DDirectory $ModelRoot "YOLOX model root"
|
||||
$mediaRuntime = Resolve-DDirectory $MediaRuntimeRoot "E15 media runtime"
|
||||
$runnerRoot = Split-Path $runner -Parent
|
||||
|
||||
foreach ($path in @($liveProfile, $e14Profile, $detectorProfile, $semanticProfile)) {
|
||||
if ((Split-Path $path -Parent) -ne $runnerRoot) {
|
||||
throw "Runner and profiles must share one immutable mount"
|
||||
}
|
||||
}
|
||||
foreach ($dependency in @(
|
||||
"e10_fusion_runtime.py",
|
||||
"e15_shadow_runtime.py",
|
||||
"run_e10_integrated_perception.py",
|
||||
"run_e12_shadow_transport_probe.py",
|
||||
"run_e9_multirate_perception.py",
|
||||
"run_e8_realtime_tracking.py",
|
||||
"run_e5_instance_tracking.py",
|
||||
"run_e4_full_session_segmentation.py",
|
||||
"run_recorded_perception_epoch.py",
|
||||
"run_e3_rectified_segmentation.py",
|
||||
"run_evaluation_prelabels.py"
|
||||
)) {
|
||||
$null = Resolve-DFile (Join-Path $runnerRoot $dependency) "LAB E15 dependency"
|
||||
}
|
||||
if (-not (Test-Path -LiteralPath (Join-Path $package "k1link\compute\live_perception.py") -PathType Leaf)) {
|
||||
throw "Mission Core package mount lacks live perception synchronization"
|
||||
}
|
||||
|
||||
$job = Get-Content -LiteralPath (Join-Path $jobDirectory "job.json") -Raw | ConvertFrom-Json
|
||||
if (
|
||||
$job.schema_version -ne "missioncore.compute-job/v1" -or
|
||||
$job.job_id -ne (Split-Path $jobDirectory -Leaf) -or
|
||||
$job.input.source_id -ne "sensor.camera.right"
|
||||
) { throw "Compute bootstrap job manifest is incompatible" }
|
||||
$live = Get-Content -LiteralPath $liveProfile -Raw | ConvertFrom-Json
|
||||
if (
|
||||
$live.schema_version -ne "missioncore.e15-shadow-inference-profile/v1" -or
|
||||
$live.mode -ne "replay-shadow-gate" -or
|
||||
[bool]$live.authority.commands_enabled -or
|
||||
[bool]$live.authority.navigation_or_safety_accepted -or
|
||||
$live.transport.pyav_version -ne "18.0.0"
|
||||
) { throw "LAB E15 replay-shadow authority contract changed" }
|
||||
$projection = Get-Content -LiteralPath (Join-Path $projectionPack "manifest.json") -Raw | ConvertFrom-Json
|
||||
if (
|
||||
$projection.schema_version -ne "missioncore.e15-live-projection-pack/v1" -or
|
||||
$projection.identity.source_id -ne "sensor.camera.right" -or
|
||||
$projection.identity.calibration_slot -ne "camera_1" -or
|
||||
$projection.identity.calibration_sha256 -ne $live.source.calibration_sha256
|
||||
) { throw "LAB E15 projection pack binding changed" }
|
||||
$mediaManifest = Get-Content -LiteralPath (Join-Path $mediaRuntime "manifest.json") -Raw | ConvertFrom-Json
|
||||
if (
|
||||
$mediaManifest.schema_version -ne "missioncore.e15-media-runtime/v1" -or
|
||||
$mediaManifest.container_image -ne $ContainerImage -or
|
||||
$mediaManifest.packages.av -ne "18.0.0" -or
|
||||
$mediaManifest.packages.lz4 -ne "4.4.5"
|
||||
) { throw "LAB E15 media runtime identity changed" }
|
||||
|
||||
$derivedRoot = Resolve-DDirectory (Join-Path $runtime "derived") "Runtime derived root"
|
||||
$cacheRoot = Resolve-DDirectory (Join-Path $runtime "cache\perception-e3-models-v1") "EoMT cache"
|
||||
$e3Environment = Resolve-DDirectory (Join-Path $derivedRoot "perception-e3-opencv413092-v1") "E3 environment"
|
||||
$torchEnvironment = Resolve-DDirectory (Join-Path $derivedRoot "perception-p0-env-v1") "Torch environment"
|
||||
$transformersEnvironment = Resolve-DDirectory (Join-Path $derivedRoot "perception-p0-transformers4576-v1") "Transformers environment"
|
||||
$freeBytesBefore = Assert-FreeSpace "preflight" 512MB
|
||||
|
||||
& docker image inspect $ContainerImage *> $null
|
||||
Assert-LastExitCode "Pinned container image inspection"
|
||||
$tritonState = docker inspect --format "{{.State.Running}}" $TritonContainer
|
||||
Assert-LastExitCode "Triton container inspection"
|
||||
if ($tritonState.Trim().ToLowerInvariant() -ne "true") {
|
||||
throw "LAB E15 requires the existing Triton container"
|
||||
}
|
||||
|
||||
$runnerName = Split-Path $runner -Leaf
|
||||
$liveProfileName = Split-Path $liveProfile -Leaf
|
||||
$e14ProfileName = Split-Path $e14Profile -Leaf
|
||||
$detectorProfileName = Split-Path $detectorProfile -Leaf
|
||||
$semanticProfileName = Split-Path $semanticProfile -Leaf
|
||||
$projectionMount = "/" + (Split-Path $projectionPack -Leaf)
|
||||
$packageMount = "/" + (Split-Path $package -Leaf)
|
||||
$orchestratorSha256 = (Get-FileHash -LiteralPath $PSCommandPath -Algorithm SHA256).Hash.ToLowerInvariant()
|
||||
$commonMounts = @(
|
||||
"-e", ("PYTHONPATH=/runner:{0}:/opt/media:/opt/transformers:/opt/env" -f $packageMount),
|
||||
"-e", "PYTHONDONTWRITEBYTECODE=1",
|
||||
"-e", "TORCH_HOME=/cache/torch",
|
||||
"-e", "HF_HOME=/cache/huggingface",
|
||||
"-e", "HF_HUB_OFFLINE=1",
|
||||
"-e", "TRANSFORMERS_OFFLINE=1",
|
||||
"-v", ((Convert-ToDockerPath $jobDirectory) + ":/job:ro"),
|
||||
"-v", ((Convert-ToDockerPath $validFov) + ":/valid-fov:ro"),
|
||||
"-v", ((Convert-ToDockerPath $projectionPack) + (":{0}:ro" -f $projectionMount)),
|
||||
"-v", ((Convert-ToDockerPath $model) + ":/model:ro"),
|
||||
"-v", ((Convert-ToDockerPath $cacheRoot) + ":/cache:ro"),
|
||||
"-v", ((Convert-ToDockerPath $e3Environment) + ":/environment:ro"),
|
||||
"-v", ((Convert-ToDockerPath $torchEnvironment) + ":/opt/env:ro"),
|
||||
"-v", ((Convert-ToDockerPath $transformersEnvironment) + ":/opt/transformers:ro"),
|
||||
"-v", ((Convert-ToDockerPath $mediaRuntime) + ":/opt/media:ro"),
|
||||
"-v", ((Convert-ToDockerPath $package) + (":{0}:ro" -f $packageMount)),
|
||||
"-v", ((Convert-ToDockerPath $runnerRoot) + ":/runner:ro")
|
||||
)
|
||||
$commonRunnerArgs = @(
|
||||
"--job", "/job",
|
||||
"--live-profile", ("/runner/{0}" -f $liveProfileName),
|
||||
"--e14-profile", ("/runner/{0}" -f $e14ProfileName),
|
||||
"--detector-profile", ("/runner/{0}" -f $detectorProfileName),
|
||||
"--semantic-profile", ("/runner/{0}" -f $semanticProfileName),
|
||||
"--valid-fov-root", "/valid-fov",
|
||||
"--projection-pack", $projectionMount,
|
||||
"--model-root", "/model",
|
||||
"--cache", "/cache",
|
||||
"--environment", "/environment",
|
||||
"--worker-package", $packageMount
|
||||
)
|
||||
|
||||
if ($PersistentService) {
|
||||
if ($PreflightOnly -or $TokenStdin) {
|
||||
throw "Persistent service cannot be combined with one-shot switches"
|
||||
}
|
||||
$persistentParent = Resolve-DDirectory (Split-Path $PersistentOutputRoot -Parent) "Persistent output parent"
|
||||
if (-not (Test-Path -LiteralPath $PersistentOutputRoot)) {
|
||||
$null = New-Item -ItemType Directory -Path $PersistentOutputRoot
|
||||
}
|
||||
$persistentOutput = Resolve-DDirectory $PersistentOutputRoot "Persistent output root"
|
||||
if ((Split-Path $persistentOutput -Parent) -ne $persistentParent) {
|
||||
throw "Persistent output root must be a direct child of its guarded D: parent"
|
||||
}
|
||||
$runnerSha256 = (Get-FileHash -LiteralPath $runner -Algorithm SHA256).Hash.ToLowerInvariant()
|
||||
$existingContainerId = docker ps -a --filter ("name=^{0}$" -f $PersistentContainer) --format "{{.ID}}"
|
||||
Assert-LastExitCode "Persistent worker container lookup"
|
||||
if ($existingContainerId) {
|
||||
$labels = (docker inspect --format "{{json .Config.Labels}}" $PersistentContainer) | ConvertFrom-Json
|
||||
Assert-LastExitCode "Persistent worker label inspection"
|
||||
if (
|
||||
$labels.'missioncore.role' -ne "perception-persistent-worker" -or
|
||||
$labels.'missioncore.runner.sha256' -ne $runnerSha256
|
||||
) { throw "Existing persistent worker has a different immutable identity" }
|
||||
$running = docker inspect --format "{{.State.Running}}" $PersistentContainer
|
||||
Assert-LastExitCode "Persistent worker state inspection"
|
||||
if ($running.Trim().ToLowerInvariant() -ne "true") {
|
||||
throw "Matching persistent worker exists but is not running"
|
||||
}
|
||||
$healthProbe = (
|
||||
"import json,urllib.request;d=json.load(urllib.request.urlopen(" +
|
||||
"'http://127.0.0.1:{0}/health',timeout=5));" -f $PersistentPort
|
||||
) + "raise SystemExit(0 if d.get('ok') and d.get('models_loaded') else 2)"
|
||||
$previousErrorPreference = $ErrorActionPreference
|
||||
$ErrorActionPreference = "Continue"
|
||||
& docker exec $PersistentContainer python3 -c $healthProbe 2>$null
|
||||
$healthExitCode = $LASTEXITCODE
|
||||
$ErrorActionPreference = $previousErrorPreference
|
||||
if ($healthExitCode -ne 0) { throw "Existing persistent worker health probe failed" }
|
||||
Write-Output "STATE=persistent-worker-reused"
|
||||
Write-Output ("CONTAINER={0}" -f $PersistentContainer)
|
||||
return
|
||||
}
|
||||
|
||||
if (-not (Test-TritonModelReady)) {
|
||||
Invoke-WebRequest -UseBasicParsing -TimeoutSec 60 `
|
||||
-Uri "http://127.0.0.1:8000/v2/repository/models/yolox_s/load" `
|
||||
-Method Post -ContentType "application/json" -Body "{}" *> $null
|
||||
$deadline = [DateTime]::UtcNow.AddSeconds(60)
|
||||
while (-not (Test-TritonModelReady)) {
|
||||
if ([DateTime]::UtcNow -ge $deadline) { throw "YOLOX-S did not become ready" }
|
||||
Start-Sleep -Milliseconds 500
|
||||
}
|
||||
}
|
||||
|
||||
$serveArgs = @(
|
||||
"run", "--detach", "--name", $PersistentContainer,
|
||||
"--label", "missioncore.role=perception-persistent-worker",
|
||||
"--label", ("missioncore.runner.sha256={0}" -f $runnerSha256),
|
||||
"--gpus", "all", "--network", ("container:{0}" -f $TritonContainer),
|
||||
"--read-only", "--security-opt", "no-new-privileges:true", "--cap-drop", "ALL",
|
||||
"--pids-limit", "512", "--shm-size", "4g",
|
||||
"--tmpfs", "/tmp:rw,noexec,nosuid,size=2g"
|
||||
) + $commonMounts + @(
|
||||
"-v", ((Convert-ToDockerPath $persistentOutput) + ":/publish:rw"),
|
||||
"--entrypoint", "python3", $ContainerImage,
|
||||
("/runner/{0}" -f $runnerName), "serve"
|
||||
) + $commonRunnerArgs + @(
|
||||
"--host", $SourceHost,
|
||||
"--port", [string]$SourcePort,
|
||||
"--path", $SourcePath,
|
||||
"--triton-url", "http://127.0.0.1:8000",
|
||||
"--output-root", "/publish",
|
||||
"--listen-host", "127.0.0.1",
|
||||
"--listen-port", [string]$PersistentPort,
|
||||
"--max-duration-seconds", [string]$MaximumDurationSeconds,
|
||||
"--free-bytes-floor", [string]([int64]$FreeGiBFloor * 1GB),
|
||||
"--orchestrator-sha256", $orchestratorSha256,
|
||||
"--container-image", $ContainerImage
|
||||
)
|
||||
Write-Output "PHASE=persistent-worker-model-load-start"
|
||||
& docker @serveArgs
|
||||
Assert-LastExitCode "Persistent worker container start"
|
||||
$healthProbe = (
|
||||
"import json,urllib.request;d=json.load(urllib.request.urlopen(" +
|
||||
"'http://127.0.0.1:{0}/health',timeout=5));" -f $PersistentPort
|
||||
) + "print(json.dumps(d,sort_keys=True));raise SystemExit(0 if d.get('ok') and d.get('models_loaded') else 2)"
|
||||
$deadline = [DateTime]::UtcNow.AddSeconds(240)
|
||||
do {
|
||||
Start-Sleep -Seconds 2
|
||||
$running = docker inspect --format "{{.State.Running}}" $PersistentContainer
|
||||
if ($LASTEXITCODE -ne 0 -or $running.Trim().ToLowerInvariant() -ne "true") {
|
||||
& docker logs --tail 80 $PersistentContainer
|
||||
throw "Persistent worker exited during model load"
|
||||
}
|
||||
$previousErrorPreference = $ErrorActionPreference
|
||||
$ErrorActionPreference = "Continue"
|
||||
$health = & docker exec $PersistentContainer python3 -c $healthProbe 2>$null
|
||||
$healthExitCode = $LASTEXITCODE
|
||||
$ErrorActionPreference = $previousErrorPreference
|
||||
$healthy = $healthExitCode -eq 0
|
||||
} while (-not $healthy -and [DateTime]::UtcNow -lt $deadline)
|
||||
if (-not $healthy) {
|
||||
& docker logs --tail 80 $PersistentContainer
|
||||
throw "Persistent worker model load did not become ready"
|
||||
}
|
||||
Write-Output "STATE=persistent-worker-ready"
|
||||
Write-Output ("CONTAINER={0}" -f $PersistentContainer)
|
||||
Write-Output ("HEALTH={0}" -f $health)
|
||||
Write-Output ("DISK_FREE_BYTES={0}" -f (Get-DFreeBytes))
|
||||
return
|
||||
}
|
||||
|
||||
$preflightArgs = @(
|
||||
"run", "--rm", "--gpus", "all", "--network", "none", "--read-only",
|
||||
"--security-opt", "no-new-privileges:true", "--cap-drop", "ALL",
|
||||
"--pids-limit", "512", "--shm-size", "4g",
|
||||
"--tmpfs", "/tmp:rw,noexec,nosuid,size=2g"
|
||||
) + $commonMounts + @(
|
||||
"--entrypoint", "python3", $ContainerImage,
|
||||
("/runner/{0}" -f $runnerName), "preflight"
|
||||
) + $commonRunnerArgs
|
||||
Write-Output "PHASE=e15-preflight-start"
|
||||
& docker @preflightArgs
|
||||
Assert-LastExitCode "LAB E15 preflight"
|
||||
Write-Output "PHASE=e15-preflight-complete"
|
||||
if ($PreflightOnly) {
|
||||
Write-Output "STATE=preflight-ready"
|
||||
Write-Output ("DISK_FREE_BYTES_BEFORE={0}" -f $freeBytesBefore)
|
||||
Write-Output ("DISK_FREE_BYTES_AFTER={0}" -f (Get-DFreeBytes))
|
||||
return
|
||||
}
|
||||
if (-not $TokenStdin) { throw "LAB E15 requires the shadow token through stdin" }
|
||||
$shadowToken = [Console]::In.ReadLine()
|
||||
if (-not $shadowToken -or $shadowToken.Length -lt 40 -or $shadowToken.Length -gt 512) {
|
||||
throw "LAB E15 shadow token is missing or malformed"
|
||||
}
|
||||
|
||||
$modelWasReady = Test-TritonModelReady
|
||||
$loadedByRun = $false
|
||||
if (-not $modelWasReady) {
|
||||
Invoke-WebRequest -UseBasicParsing -TimeoutSec 60 `
|
||||
-Uri "http://127.0.0.1:8000/v2/repository/models/yolox_s/load" `
|
||||
-Method Post -ContentType "application/json" -Body "{}" *> $null
|
||||
$loadedByRun = $true
|
||||
$deadline = [DateTime]::UtcNow.AddSeconds(60)
|
||||
while (-not (Test-TritonModelReady)) {
|
||||
if ([DateTime]::UtcNow -ge $deadline) { throw "YOLOX-S did not become ready" }
|
||||
Start-Sleep -Milliseconds 500
|
||||
}
|
||||
}
|
||||
Write-Output ("PHASE=e15-model-ready LOADED_BY_RUN={0}" -f $loadedByRun)
|
||||
|
||||
$token = [Guid]::NewGuid().ToString("N")
|
||||
$publishRoot = Join-Path $derivedRoot (".{0}-e15-{1}.publish" -f $job.job_id, $token)
|
||||
$stagingRoot = Join-Path $publishRoot "output"
|
||||
$null = New-Item -ItemType Directory -Path $publishRoot
|
||||
$completed = $false
|
||||
$totalWatch = [Diagnostics.Stopwatch]::StartNew()
|
||||
try {
|
||||
$runArgs = @(
|
||||
"run", "--rm", "--interactive", "--gpus", "all",
|
||||
"--network", ("container:{0}" -f $TritonContainer),
|
||||
"--read-only", "--security-opt", "no-new-privileges:true", "--cap-drop", "ALL",
|
||||
"--pids-limit", "512", "--shm-size", "4g",
|
||||
"--tmpfs", "/tmp:rw,noexec,nosuid,size=2g"
|
||||
) + $commonMounts + @(
|
||||
"-v", ((Convert-ToDockerPath $publishRoot) + ":/publish:rw"),
|
||||
"--entrypoint", "python3", $ContainerImage,
|
||||
("/runner/{0}" -f $runnerName), "run"
|
||||
) + $commonRunnerArgs + @(
|
||||
"--host", $SourceHost,
|
||||
"--port", [string]$SourcePort,
|
||||
"--path", $SourcePath,
|
||||
"--token-stdin",
|
||||
"--triton-url", "http://127.0.0.1:8000",
|
||||
"--output", "/publish/output",
|
||||
"--max-duration-seconds", [string]$MaximumDurationSeconds,
|
||||
"--free-bytes-floor", [string]([int64]$FreeGiBFloor * 1GB),
|
||||
"--orchestrator-sha256", $orchestratorSha256,
|
||||
"--container-image", $ContainerImage
|
||||
)
|
||||
Write-Output ("PHASE=e15-shadow-inference-start MAX_DURATION_SECONDS={0}" -f $MaximumDurationSeconds)
|
||||
$shadowToken | & docker @runArgs
|
||||
$runnerExitCode = $LASTEXITCODE
|
||||
$shadowToken = $null
|
||||
if ($runnerExitCode -ne 0 -and $runnerExitCode -ne 2) {
|
||||
throw "LAB E15 shadow inference failed with exit code $runnerExitCode"
|
||||
}
|
||||
$null = Assert-FreeSpace "post-shadow-inference"
|
||||
|
||||
$resultPath = Join-Path $stagingRoot "result.json"
|
||||
if (-not (Test-Path -LiteralPath $resultPath -PathType Leaf)) {
|
||||
throw "LAB E15 result manifest is missing"
|
||||
}
|
||||
$result = Get-Content -LiteralPath $resultPath -Raw | ConvertFrom-Json
|
||||
if (
|
||||
$result.schema_version -ne "missioncore.e15-shadow-inference-result/v1" -or
|
||||
$result.result_id -notmatch "^e15-shadow-inference-[a-f0-9]{64}$" -or
|
||||
$result.publication_scope -ne "live-shadow-diagnostic-only"
|
||||
) { throw "LAB E15 result manifest is incompatible" }
|
||||
$finalRoot = Join-Path $derivedRoot ([string]$result.result_id)
|
||||
if (Test-Path -LiteralPath $finalRoot) { throw "Immutable LAB E15 result already exists" }
|
||||
Move-Item -LiteralPath $stagingRoot -Destination $finalRoot
|
||||
Remove-Item -LiteralPath $publishRoot -Force
|
||||
$completed = $true
|
||||
$totalWatch.Stop()
|
||||
$freeFinal = Assert-FreeSpace "post-publication"
|
||||
Write-Output ("RESULT_ROOT={0}" -f $finalRoot)
|
||||
Write-Output ("RESULT_ID={0}" -f $result.result_id)
|
||||
Write-Output ("ACCEPTANCE_STATE={0}" -f $result.acceptance_state)
|
||||
Write-Output ("RUNNER_EXIT_CODE={0}" -f $runnerExitCode)
|
||||
Write-Output ("ORCHESTRATOR_WALL_SECONDS={0}" -f $totalWatch.Elapsed.TotalSeconds)
|
||||
Write-Output ("DISK_FREE_BYTES_BEFORE={0}" -f $freeBytesBefore)
|
||||
Write-Output ("DISK_FREE_BYTES_FINAL={0}" -f $freeFinal)
|
||||
}
|
||||
finally {
|
||||
$shadowToken = $null
|
||||
if (-not $completed -and (Test-Path -LiteralPath $publishRoot)) {
|
||||
Remove-Item -LiteralPath $publishRoot -Recurse -Force
|
||||
}
|
||||
if ($loadedByRun) {
|
||||
try {
|
||||
Invoke-WebRequest -UseBasicParsing -TimeoutSec 60 `
|
||||
-Uri "http://127.0.0.1:8000/v2/repository/models/yolox_s/unload" `
|
||||
-Method Post -ContentType "application/json" -Body "{}" *> $null
|
||||
Write-Output "PHASE=e15-model-state-restored"
|
||||
}
|
||||
catch { Write-Warning "LAB E15 could not restore the prior YOLOX-S state" }
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,251 @@
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[Parameter(Mandatory = $true)]
|
||||
[string]$EvaluationPack,
|
||||
|
||||
[Parameter(Mandatory = $true)]
|
||||
[string]$E2Prelabels,
|
||||
|
||||
[Parameter(Mandatory = $true)]
|
||||
[string]$ValidFovRoot,
|
||||
|
||||
[Parameter(Mandatory = $true)]
|
||||
[string]$RunnerPath,
|
||||
|
||||
[Parameter(Mandatory = $true)]
|
||||
[string]$BaselineRunnerPath,
|
||||
|
||||
[Parameter(Mandatory = $true)]
|
||||
[string]$ProfilePath,
|
||||
|
||||
[ValidateRange(0, 64)]
|
||||
[int]$MaxFrames = 0,
|
||||
|
||||
[ValidateRange(1, 1000)]
|
||||
[int]$FreeGiBFloor = 360,
|
||||
|
||||
[string]$RuntimeRoot = "D:\NDC_MISSIONCORE\runtime",
|
||||
|
||||
[string]$ContainerImage = "nvcr.io/nvidia/tritonserver:26.06-py3"
|
||||
)
|
||||
|
||||
$ErrorActionPreference = "Stop"
|
||||
$ProgressPreference = "SilentlyContinue"
|
||||
|
||||
function Assert-LastExitCode {
|
||||
param([string]$Operation)
|
||||
if ($LASTEXITCODE -ne 0) {
|
||||
throw "$Operation failed with exit code $LASTEXITCODE"
|
||||
}
|
||||
}
|
||||
|
||||
function Assert-RegularFile {
|
||||
param([string]$Path, [string]$Label)
|
||||
$item = Get-Item -LiteralPath $Path -Force
|
||||
if (-not $item.PSIsContainer -and -not ($item.Attributes -band [IO.FileAttributes]::ReparsePoint)) {
|
||||
return $item.FullName
|
||||
}
|
||||
throw "$Label is not a regular file"
|
||||
}
|
||||
|
||||
function Assert-Directory {
|
||||
param([string]$Path, [string]$Label)
|
||||
$item = Get-Item -LiteralPath $Path -Force
|
||||
if ($item.PSIsContainer -and -not ($item.Attributes -band [IO.FileAttributes]::ReparsePoint)) {
|
||||
return $item.FullName
|
||||
}
|
||||
throw "$Label is not a real directory"
|
||||
}
|
||||
|
||||
function Assert-DDrivePath {
|
||||
param([string]$Path, [string]$Label)
|
||||
$fullPath = [IO.Path]::GetFullPath($Path)
|
||||
if ([IO.Path]::GetPathRoot($fullPath).TrimEnd("\") -ine "D:") {
|
||||
throw "$Label must be stored on D:"
|
||||
}
|
||||
return $fullPath
|
||||
}
|
||||
|
||||
function Assert-FreeSpace {
|
||||
param([string]$Phase)
|
||||
$freeBytes = [int64](Get-PSDrive -Name D).Free
|
||||
$floorBytes = [int64]$FreeGiBFloor * 1GB
|
||||
$freeGiB = [math]::Round($freeBytes / 1GB, 3)
|
||||
Write-Output ("DISK_GUARD PHASE={0} DRIVE=D FREE_GIB={1} FLOOR_GIB={2}" -f $Phase, $freeGiB, $FreeGiBFloor)
|
||||
if ($freeBytes -lt $floorBytes) {
|
||||
throw "D: free-space floor was crossed during $Phase"
|
||||
}
|
||||
}
|
||||
|
||||
function Convert-ToDockerPath {
|
||||
param([string]$Path)
|
||||
return $Path.Replace("\", "/")
|
||||
}
|
||||
|
||||
$packRoot = Assert-DDrivePath (Assert-Directory (Resolve-Path -LiteralPath $EvaluationPack).Path "Evaluation pack") "Evaluation pack"
|
||||
$prelabelsRoot = Assert-DDrivePath (Assert-Directory (Resolve-Path -LiteralPath $E2Prelabels).Path "E2 prelabels") "E2 prelabels"
|
||||
$validFov = Assert-DDrivePath (Assert-Directory (Resolve-Path -LiteralPath $ValidFovRoot).Path "Valid-FOV root") "Valid-FOV root"
|
||||
$runner = Assert-DDrivePath (Assert-RegularFile (Resolve-Path -LiteralPath $RunnerPath).Path "LAB E3 runner") "LAB E3 runner"
|
||||
$baselineRunner = Assert-DDrivePath (Assert-RegularFile (Resolve-Path -LiteralPath $BaselineRunnerPath).Path "Baseline runner") "Baseline runner"
|
||||
$profile = Assert-DDrivePath (Assert-RegularFile (Resolve-Path -LiteralPath $ProfilePath).Path "LAB E3 profile") "LAB E3 profile"
|
||||
$runtime = Assert-DDrivePath (Assert-Directory (Resolve-Path -LiteralPath $RuntimeRoot).Path "Runtime root") "Runtime root"
|
||||
if (
|
||||
(Split-Path $runner -Parent) -ne (Split-Path $baselineRunner -Parent) -or
|
||||
(Split-Path $runner -Parent) -ne (Split-Path $profile -Parent)
|
||||
) {
|
||||
throw "LAB E3 runner, baseline runner, and profile must share one read-only mount"
|
||||
}
|
||||
|
||||
$pack = Get-Content -LiteralPath (Join-Path $packRoot "manifest.json") -Raw | ConvertFrom-Json
|
||||
$prelabels = Get-Content -LiteralPath (Join-Path $prelabelsRoot "result.json") -Raw | ConvertFrom-Json
|
||||
$profileDocument = Get-Content -LiteralPath $profile -Raw | ConvertFrom-Json
|
||||
$packFrameCount = @($pack.identity.frames).Count
|
||||
if (
|
||||
$pack.schema_version -ne "missioncore.perception-evaluation-pack/v1" -or
|
||||
$pack.generation_id -ne (Split-Path $packRoot -Leaf) -or
|
||||
$pack.identity.source_id -ne "sensor.camera.right" -or
|
||||
$pack.identity.calibration_slot -ne "camera_1" -or
|
||||
$packFrameCount -ne 64 -or
|
||||
$prelabels.schema_version -ne "missioncore.perception-evaluation-prelabels/v1" -or
|
||||
$prelabels.identity.evaluation_pack_id -ne $pack.generation_id -or
|
||||
$profileDocument.schema_version -ne "missioncore.k1-e3-rectified-segmentation-profile/v1" -or
|
||||
$profileDocument.source.source_id -ne $pack.identity.source_id -or
|
||||
$profileDocument.source.calibration_slot -ne $pack.identity.calibration_slot -or
|
||||
$profileDocument.source.calibration_sha256 -ne $pack.identity.calibration_sha256
|
||||
) {
|
||||
throw "LAB E3 inputs are incompatible"
|
||||
}
|
||||
$null = Assert-RegularFile (Join-Path $validFov "manifest.json") "Valid-FOV manifest"
|
||||
$null = Assert-RegularFile (Join-Path $validFov "mask.png") "Valid-FOV mask"
|
||||
|
||||
& docker image inspect $ContainerImage *> $null
|
||||
Assert-LastExitCode "Existing container image inspection"
|
||||
Assert-FreeSpace "preflight"
|
||||
|
||||
$cacheRoot = Join-Path $runtime "cache\perception-e3-models-v1"
|
||||
$derivedRoot = Join-Path $runtime "derived\e3-segmentation"
|
||||
$environmentRoot = Join-Path $runtime "derived\perception-e3-opencv413092-v1"
|
||||
$torchEnvironment = Assert-Directory (Join-Path $runtime "derived\perception-p0-env-v1") "Torch environment"
|
||||
$transformersEnvironment = Assert-Directory (Join-Path $runtime "derived\perception-p0-transformers4576-v1") "Transformers environment"
|
||||
$runnerRoot = Split-Path $runner -Parent
|
||||
$runnerName = Split-Path $runner -Leaf
|
||||
$profileName = Split-Path $profile -Leaf
|
||||
$null = New-Item -ItemType Directory -Path $cacheRoot -Force
|
||||
$null = New-Item -ItemType Directory -Path $derivedRoot -Force
|
||||
|
||||
if (-not (Test-Path -LiteralPath $environmentRoot)) {
|
||||
$environmentToken = [Guid]::NewGuid().ToString("N")
|
||||
$environmentStaging = Join-Path (Split-Path $environmentRoot -Parent) (".e3-environment-{0}.publish" -f $environmentToken)
|
||||
$null = New-Item -ItemType Directory -Path $environmentStaging
|
||||
$null = New-Item -ItemType Directory -Path (Join-Path $environmentStaging "tmp")
|
||||
try {
|
||||
$prepareArgs = @(
|
||||
"run", "--rm", "--network", "bridge", "--read-only",
|
||||
"--security-opt", "no-new-privileges:true", "--cap-drop", "ALL",
|
||||
"--pids-limit", "512", "--tmpfs", "/tmp:rw,noexec,nosuid,size=1g",
|
||||
"-e", "PYTHONPATH=/opt/transformers:/opt/env",
|
||||
"-e", "HF_HOME=/cache/huggingface",
|
||||
"-e", "PIP_NO_CACHE_DIR=1",
|
||||
"-e", "HOME=/environment/tmp",
|
||||
"-e", "TMPDIR=/environment/tmp",
|
||||
"-v", ((Convert-ToDockerPath $cacheRoot) + ":/cache:rw"),
|
||||
"-v", ((Convert-ToDockerPath $environmentStaging) + ":/environment:rw"),
|
||||
"-v", ((Convert-ToDockerPath $torchEnvironment) + ":/opt/env:ro"),
|
||||
"-v", ((Convert-ToDockerPath $transformersEnvironment) + ":/opt/transformers:ro"),
|
||||
"-v", ((Convert-ToDockerPath $runnerRoot) + ":/runner:ro"),
|
||||
"--entrypoint", "python3",
|
||||
$ContainerImage,
|
||||
("/runner/{0}" -f $runnerName), "prepare",
|
||||
"--profile", ("/runner/{0}" -f $profileName),
|
||||
"--cache", "/cache",
|
||||
"--environment", "/environment"
|
||||
)
|
||||
Write-Output "PHASE=e3-dependency-prepare-start"
|
||||
& docker @prepareArgs
|
||||
Assert-LastExitCode "LAB E3 dependency preparation"
|
||||
$null = Assert-RegularFile (Join-Path $environmentStaging "manifest.json") "LAB E3 dependency manifest"
|
||||
Remove-Item -LiteralPath (Join-Path $environmentStaging "tmp") -Recurse -Force
|
||||
Move-Item -LiteralPath $environmentStaging -Destination $environmentRoot
|
||||
Write-Output "PHASE=e3-dependency-prepare-complete"
|
||||
}
|
||||
finally {
|
||||
if (Test-Path -LiteralPath $environmentStaging) {
|
||||
Remove-Item -LiteralPath $environmentStaging -Recurse -Force
|
||||
}
|
||||
}
|
||||
}
|
||||
$environment = Assert-Directory $environmentRoot "LAB E3 dependency environment"
|
||||
$null = Assert-RegularFile (Join-Path $environment "manifest.json") "LAB E3 dependency manifest"
|
||||
Assert-FreeSpace "post-dependency-prepare"
|
||||
|
||||
$runToken = [Guid]::NewGuid().ToString("N")
|
||||
$publishRoot = Join-Path $derivedRoot (".{0}-{1}.publish" -f $pack.generation_id, $runToken)
|
||||
$stagingRoot = Join-Path $publishRoot "output"
|
||||
$null = New-Item -ItemType Directory -Path $publishRoot
|
||||
|
||||
try {
|
||||
$runArgs = @(
|
||||
"run", "--rm", "--gpus", "all", "--network", "none", "--read-only",
|
||||
"--security-opt", "no-new-privileges:true", "--cap-drop", "ALL",
|
||||
"--pids-limit", "512", "--shm-size", "4g",
|
||||
"--tmpfs", "/tmp:rw,noexec,nosuid,size=2g",
|
||||
"-e", "PYTHONPATH=/opt/transformers:/opt/env",
|
||||
"-e", "TORCH_HOME=/cache/torch",
|
||||
"-e", "HF_HOME=/cache/huggingface",
|
||||
"-e", "HF_HUB_OFFLINE=1",
|
||||
"-e", "TRANSFORMERS_OFFLINE=1",
|
||||
"-e", "HOME=/tmp",
|
||||
"-v", ((Convert-ToDockerPath $packRoot) + ":/evaluation-pack:ro"),
|
||||
"-v", ((Convert-ToDockerPath $prelabelsRoot) + ":/e2-prelabels:ro"),
|
||||
"-v", ((Convert-ToDockerPath $validFov) + ":/valid-fov:ro"),
|
||||
"-v", ((Convert-ToDockerPath $publishRoot) + ":/publish:rw"),
|
||||
"-v", ((Convert-ToDockerPath $cacheRoot) + ":/cache:ro"),
|
||||
"-v", ((Convert-ToDockerPath $environment) + ":/environment:ro"),
|
||||
"-v", ((Convert-ToDockerPath $torchEnvironment) + ":/opt/env:ro"),
|
||||
"-v", ((Convert-ToDockerPath $transformersEnvironment) + ":/opt/transformers:ro"),
|
||||
"-v", ((Convert-ToDockerPath $runnerRoot) + ":/runner:ro"),
|
||||
"--entrypoint", "python3",
|
||||
$ContainerImage,
|
||||
("/runner/{0}" -f $runnerName), "run",
|
||||
"--profile", ("/runner/{0}" -f $profileName),
|
||||
"--evaluation-pack", "/evaluation-pack",
|
||||
"--e2-prelabels", "/e2-prelabels",
|
||||
"--valid-fov-root", "/valid-fov",
|
||||
"--cache", "/cache",
|
||||
"--environment", "/environment",
|
||||
"--output", "/publish/output",
|
||||
"--telemetry-interval-seconds", "1"
|
||||
)
|
||||
if ($MaxFrames -gt 0) {
|
||||
$runArgs += @("--max-frames", [string]$MaxFrames)
|
||||
}
|
||||
$expectedFrames = if ($MaxFrames -gt 0) { $MaxFrames } else { $packFrameCount }
|
||||
Write-Output ("PHASE=e3-run-start PACK={0} FRAMES={1}" -f $pack.generation_id, $expectedFrames)
|
||||
& docker @runArgs
|
||||
Assert-LastExitCode "LAB E3 segmentation"
|
||||
|
||||
$result = Get-Content -LiteralPath (Join-Path $stagingRoot "result.json") -Raw | ConvertFrom-Json
|
||||
if (
|
||||
$result.schema_version -ne "missioncore.k1-e3-rectified-segmentation-result/v1" -or
|
||||
$result.result_id -notmatch "^e3-segmentation-[a-f0-9]{64}$" -or
|
||||
$result.identity.evaluation_pack_id -ne $pack.generation_id -or
|
||||
[int]$result.identity.frame_count -ne $expectedFrames -or
|
||||
$result.ground_truth -ne $false
|
||||
) {
|
||||
throw "LAB E3 result manifest is incompatible"
|
||||
}
|
||||
$finalRoot = Join-Path $derivedRoot ([string]$result.result_id)
|
||||
if (Test-Path -LiteralPath $finalRoot) {
|
||||
throw "An immutable LAB E3 result with the same identity already exists: $finalRoot"
|
||||
}
|
||||
Move-Item -LiteralPath $stagingRoot -Destination $finalRoot
|
||||
Remove-Item -LiteralPath $publishRoot -Force
|
||||
Assert-FreeSpace "post-run"
|
||||
Write-Output ("RESULT_ROOT={0}" -f $finalRoot)
|
||||
Write-Output ("RESULT_ID={0}" -f $result.result_id)
|
||||
}
|
||||
finally {
|
||||
if (Test-Path -LiteralPath $publishRoot) {
|
||||
Remove-Item -LiteralPath $publishRoot -Recurse -Force
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,421 @@
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[Parameter(Mandatory = $true)]
|
||||
[string]$JobRoot,
|
||||
|
||||
[Parameter(Mandatory = $true)]
|
||||
[string]$RunnerPath,
|
||||
|
||||
[Parameter(Mandatory = $true)]
|
||||
[string]$ProfilePath,
|
||||
|
||||
[Parameter(Mandatory = $true)]
|
||||
[string]$ValidFovRoot,
|
||||
|
||||
[ValidateRange(0, 256)]
|
||||
[int]$PilotFrames = 0,
|
||||
|
||||
[ValidateRange(1, 1000)]
|
||||
[int]$FreeGiBFloor = 360,
|
||||
|
||||
[string]$RuntimeRoot = "D:\NDC_MISSIONCORE\runtime",
|
||||
|
||||
[string]$ContainerImage = "nvcr.io/nvidia/tritonserver:26.06-py3"
|
||||
)
|
||||
|
||||
$ErrorActionPreference = "Stop"
|
||||
$ProgressPreference = "SilentlyContinue"
|
||||
|
||||
function Assert-LastExitCode {
|
||||
param([string]$Operation)
|
||||
if ($LASTEXITCODE -ne 0) {
|
||||
throw "$Operation failed with exit code $LASTEXITCODE"
|
||||
}
|
||||
}
|
||||
|
||||
function Assert-RegularFile {
|
||||
param([string]$Path, [string]$Label)
|
||||
$item = Get-Item -LiteralPath $Path -Force
|
||||
if (-not $item.PSIsContainer -and -not ($item.Attributes -band [IO.FileAttributes]::ReparsePoint)) {
|
||||
return $item.FullName
|
||||
}
|
||||
throw "$Label is not a regular file"
|
||||
}
|
||||
|
||||
function Assert-Directory {
|
||||
param([string]$Path, [string]$Label)
|
||||
$item = Get-Item -LiteralPath $Path -Force
|
||||
if ($item.PSIsContainer -and -not ($item.Attributes -band [IO.FileAttributes]::ReparsePoint)) {
|
||||
return $item.FullName
|
||||
}
|
||||
throw "$Label is not a real directory"
|
||||
}
|
||||
|
||||
function Assert-DDrivePath {
|
||||
param([string]$Path, [string]$Label)
|
||||
$fullPath = [IO.Path]::GetFullPath($Path)
|
||||
if ([IO.Path]::GetPathRoot($fullPath).TrimEnd("\") -ine "D:") {
|
||||
throw "$Label must be stored on D:"
|
||||
}
|
||||
return $fullPath
|
||||
}
|
||||
|
||||
function Get-DFreeBytes {
|
||||
return [int64](Get-PSDrive -Name D).Free
|
||||
}
|
||||
|
||||
function Assert-FreeSpace {
|
||||
param(
|
||||
[string]$Phase,
|
||||
[int64]$RequiredAdditionalBytes = 0
|
||||
)
|
||||
$freeBytes = Get-DFreeBytes
|
||||
$floorBytes = [int64]$FreeGiBFloor * 1GB
|
||||
$requiredBytes = $floorBytes + $RequiredAdditionalBytes
|
||||
Write-Host (
|
||||
"DISK_GUARD PHASE={0} DRIVE=D FREE_BYTES={1} FREE_GIB={2} FLOOR_GIB={3} REQUIRED_ADDITIONAL_GIB={4}" -f
|
||||
$Phase,
|
||||
$freeBytes,
|
||||
[math]::Round($freeBytes / 1GB, 3),
|
||||
$FreeGiBFloor,
|
||||
[math]::Round($RequiredAdditionalBytes / 1GB, 3)
|
||||
)
|
||||
if ($freeBytes -lt $requiredBytes) {
|
||||
throw "D: does not have the guarded LAB E4 working-set reserve during $Phase"
|
||||
}
|
||||
return $freeBytes
|
||||
}
|
||||
|
||||
function Convert-ToDockerPath {
|
||||
param([string]$Path)
|
||||
return $Path.Replace("\", "/")
|
||||
}
|
||||
|
||||
$jobDirectory = Assert-DDrivePath (Assert-Directory (Resolve-Path -LiteralPath $JobRoot).Path "Job root") "Job root"
|
||||
$runner = Assert-DDrivePath (Assert-RegularFile (Resolve-Path -LiteralPath $RunnerPath).Path "LAB E4 runner") "LAB E4 runner"
|
||||
$profile = Assert-DDrivePath (Assert-RegularFile (Resolve-Path -LiteralPath $ProfilePath).Path "LAB E4 profile") "LAB E4 profile"
|
||||
$validFov = Assert-DDrivePath (Assert-Directory (Resolve-Path -LiteralPath $ValidFovRoot).Path "Valid-FOV root") "Valid-FOV root"
|
||||
$runtime = Assert-DDrivePath (Assert-Directory (Resolve-Path -LiteralPath $RuntimeRoot).Path "Runtime root") "Runtime root"
|
||||
$runnerRoot = Split-Path $runner -Parent
|
||||
if ((Split-Path $profile -Parent) -ne $runnerRoot) {
|
||||
throw "LAB E4 runner and profile must share one read-only mount"
|
||||
}
|
||||
foreach ($dependency in @(
|
||||
"run_recorded_perception_epoch.py",
|
||||
"run_e3_rectified_segmentation.py",
|
||||
"run_evaluation_prelabels.py"
|
||||
)) {
|
||||
$null = Assert-RegularFile (Join-Path $runnerRoot $dependency) "LAB E4 runner dependency"
|
||||
}
|
||||
|
||||
$job = Get-Content -LiteralPath (Join-Path $jobDirectory "job.json") -Raw | ConvertFrom-Json
|
||||
if ($job.schema_version -ne "missioncore.compute-job/v1" -or $job.job_id -ne (Split-Path $jobDirectory -Leaf)) {
|
||||
throw "Compute job manifest is incompatible"
|
||||
}
|
||||
$sourceId = [string]$job.input.source_id
|
||||
$epoch = [int]$job.input.codec_epoch
|
||||
$fullFrameCount = [int]$job.input.segment_count
|
||||
$activeFrameCount = if ($PilotFrames -gt 0) { $PilotFrames } else { $fullFrameCount }
|
||||
$timelineStart = [double]$job.input.timeline.start_seconds
|
||||
$timelineEnd = [double]$job.input.timeline.end_seconds
|
||||
$timelineDuration = $timelineEnd - $timelineStart
|
||||
if (
|
||||
$sourceId -ne "sensor.camera.right" -or
|
||||
$fullFrameCount -lt 1 -or
|
||||
$activeFrameCount -gt $fullFrameCount -or
|
||||
$timelineDuration -le 0
|
||||
) {
|
||||
throw "LAB E4 camera job contract is invalid"
|
||||
}
|
||||
Write-Output ("PHASE=job-manifest-validated JOB={0} FRAMES={1}/{2}" -f $job.job_id, $activeFrameCount, $fullFrameCount)
|
||||
|
||||
$epochRoot = Join-Path $jobDirectory ("input\camera\{0}\epoch-{1}" -f $sourceId, $epoch)
|
||||
$epochRoot = Assert-DDrivePath (Assert-Directory $epochRoot "Camera epoch") "Camera epoch"
|
||||
$initPath = Assert-RegularFile (Join-Path $epochRoot "init.mp4") "Camera init"
|
||||
$segmentsRoot = Assert-Directory (Join-Path $epochRoot "segments") "Camera segments"
|
||||
$derivedRoot = Join-Path $runtime "derived"
|
||||
$tmpRoot = Join-Path $runtime "tmp"
|
||||
$cacheRoot = Assert-DDrivePath (Assert-Directory (Join-Path $runtime "cache\perception-e3-models-v1") "EoMT cache") "EoMT cache"
|
||||
$e3Environment = Assert-DDrivePath (Assert-Directory (Join-Path $derivedRoot "perception-e3-opencv413092-v1") "E3 dependency environment") "E3 dependency environment"
|
||||
$torchEnvironment = Assert-DDrivePath (Assert-Directory (Join-Path $derivedRoot "perception-p0-env-v1") "Torch environment") "Torch environment"
|
||||
$transformersEnvironment = Assert-DDrivePath (Assert-Directory (Join-Path $derivedRoot "perception-p0-transformers4576-v1") "Transformers environment") "Transformers environment"
|
||||
$null = New-Item -ItemType Directory -Path $derivedRoot -Force
|
||||
$null = New-Item -ItemType Directory -Path $tmpRoot -Force
|
||||
|
||||
# Conservative upper bound: decoded RGB + RGB overlay + one-byte semantic mask,
|
||||
# plus 20% filesystem/encoding overhead and the reconstructed source stream.
|
||||
$pixelWorkingSet = [int64]$activeFrameCount * 800 * 600 * 7
|
||||
$workingSetReserve = [int64][math]::Ceiling(($pixelWorkingSet * 1.2) + [int64]$job.input.byte_length)
|
||||
$freeBytesBefore = Assert-FreeSpace "preflight" $workingSetReserve
|
||||
|
||||
& docker image inspect $ContainerImage *> $null
|
||||
Assert-LastExitCode "Existing container image inspection"
|
||||
$runnerName = Split-Path $runner -Leaf
|
||||
$profileName = Split-Path $profile -Leaf
|
||||
$orchestratorSha256 = (Get-FileHash -LiteralPath $PSCommandPath -Algorithm SHA256).Hash.ToLowerInvariant()
|
||||
$preflightArgs = @(
|
||||
"run", "--rm", "--gpus", "all", "--network", "none", "--read-only",
|
||||
"--security-opt", "no-new-privileges:true", "--cap-drop", "ALL",
|
||||
"--pids-limit", "512", "--shm-size", "4g",
|
||||
"--tmpfs", "/tmp:rw,noexec,nosuid,size=2g",
|
||||
"-e", "PYTHONPATH=/opt/transformers:/opt/env",
|
||||
"-e", "TORCH_HOME=/cache/torch",
|
||||
"-e", "HF_HOME=/cache/huggingface",
|
||||
"-e", "HF_HUB_OFFLINE=1",
|
||||
"-e", "TRANSFORMERS_OFFLINE=1",
|
||||
"-v", ((Convert-ToDockerPath $jobDirectory) + ":/job:ro"),
|
||||
"-v", ((Convert-ToDockerPath $validFov) + ":/valid-fov:ro"),
|
||||
"-v", ((Convert-ToDockerPath $cacheRoot) + ":/cache:ro"),
|
||||
"-v", ((Convert-ToDockerPath $e3Environment) + ":/environment:ro"),
|
||||
"-v", ((Convert-ToDockerPath $torchEnvironment) + ":/opt/env:ro"),
|
||||
"-v", ((Convert-ToDockerPath $transformersEnvironment) + ":/opt/transformers:ro"),
|
||||
"-v", ((Convert-ToDockerPath $runnerRoot) + ":/runner:ro"),
|
||||
"--entrypoint", "python3",
|
||||
$ContainerImage,
|
||||
("/runner/{0}" -f $runnerName), "preflight",
|
||||
"--job", "/job",
|
||||
"--profile", ("/runner/{0}" -f $profileName),
|
||||
"--valid-fov-root", "/valid-fov",
|
||||
"--cache", "/cache",
|
||||
"--environment", "/environment"
|
||||
)
|
||||
Write-Output "PHASE=e4-preflight-start"
|
||||
& docker @preflightArgs
|
||||
Assert-LastExitCode "LAB E4 preflight"
|
||||
Write-Output "PHASE=e4-preflight-complete"
|
||||
|
||||
$runToken = [Guid]::NewGuid().ToString("N")
|
||||
$workRoot = Join-Path $tmpRoot ("{0}-e4-{1}" -f $job.job_id, $runToken)
|
||||
$framesRoot = Join-Path $workRoot "frames"
|
||||
$streamPath = Join-Path $workRoot "camera.mp4"
|
||||
$ptsPath = Join-Path $workRoot "pts.json"
|
||||
$timelinePath = Join-Path $workRoot "timeline.jsonl"
|
||||
$publishRoot = Join-Path $derivedRoot (".{0}-e4-{1}.publish" -f $job.job_id, $runToken)
|
||||
$stagingRoot = Join-Path $publishRoot "output"
|
||||
$null = New-Item -ItemType Directory -Path $framesRoot
|
||||
$null = New-Item -ItemType Directory -Path $publishRoot
|
||||
$totalWatch = [Diagnostics.Stopwatch]::StartNew()
|
||||
$completed = $false
|
||||
Write-Output "PHASE=e4-private-staging-created"
|
||||
|
||||
try {
|
||||
$stream = [IO.File]::Open($streamPath, [IO.FileMode]::CreateNew, [IO.FileAccess]::Write, [IO.FileShare]::None)
|
||||
try {
|
||||
$input = [IO.File]::OpenRead($initPath)
|
||||
try {
|
||||
$input.CopyTo($stream)
|
||||
}
|
||||
finally {
|
||||
$input.Dispose()
|
||||
}
|
||||
for ($sequence = 1; $sequence -le $activeFrameCount; $sequence++) {
|
||||
$path = Assert-RegularFile (Join-Path $segmentsRoot ("{0}.m4s" -f $sequence)) "Camera segment"
|
||||
$input = [IO.File]::OpenRead($path)
|
||||
try {
|
||||
$input.CopyTo($stream)
|
||||
}
|
||||
finally {
|
||||
$input.Dispose()
|
||||
}
|
||||
if ($sequence % 500 -eq 0 -or $sequence -eq $activeFrameCount) {
|
||||
Write-Output ("PHASE=e4-stream-reconstruction SEGMENTS={0}/{1}" -f $sequence, $activeFrameCount)
|
||||
}
|
||||
}
|
||||
$stream.Flush($true)
|
||||
}
|
||||
finally {
|
||||
$stream.Dispose()
|
||||
}
|
||||
$null = Assert-FreeSpace "post-stream-reconstruction"
|
||||
|
||||
$extractWatch = [Diagnostics.Stopwatch]::StartNew()
|
||||
Write-Output "PHASE=e4-frame-extraction-start"
|
||||
& ffmpeg -hide_banner -loglevel fatal -i $streamPath -map 0:v:0 -fps_mode passthrough -frames:v $activeFrameCount (Join-Path $framesRoot "frame-%06d.png")
|
||||
Assert-LastExitCode "LAB E4 camera extraction"
|
||||
& ffprobe -v error -select_streams v:0 -show_entries frame=best_effort_timestamp_time -of json $streamPath | Set-Content -LiteralPath $ptsPath -Encoding utf8
|
||||
Assert-LastExitCode "LAB E4 camera timestamp probe"
|
||||
|
||||
$decodedFrames = @(Get-ChildItem -LiteralPath $framesRoot -File -Filter "frame-*.png")
|
||||
$ptsDocument = Get-Content -LiteralPath $ptsPath -Raw | ConvertFrom-Json
|
||||
$pts = @($ptsDocument.frames)
|
||||
if ($decodedFrames.Count -ne $activeFrameCount -or $pts.Count -lt $activeFrameCount) {
|
||||
throw "Decoded LAB E4 frame count differs from the requested camera epoch"
|
||||
}
|
||||
$firstEpochSeconds = [double]::Parse(
|
||||
([string]$pts[0].best_effort_timestamp_time).Trim(),
|
||||
[Globalization.CultureInfo]::InvariantCulture
|
||||
)
|
||||
$previousEpochSeconds = -1.0
|
||||
$timelineWriter = [IO.StreamWriter]::new($timelinePath, $false, [Text.UTF8Encoding]::new($false))
|
||||
try {
|
||||
for ($index = 0; $index -lt $activeFrameCount; $index++) {
|
||||
$epochSeconds = [double]::Parse(
|
||||
([string]$pts[$index].best_effort_timestamp_time).Trim(),
|
||||
[Globalization.CultureInfo]::InvariantCulture
|
||||
) - $firstEpochSeconds
|
||||
if ($epochSeconds -le $previousEpochSeconds -or $epochSeconds -gt ($timelineDuration + 0.001)) {
|
||||
throw "Decoded LAB E4 timestamps are not strictly monotonic inside the camera timeline"
|
||||
}
|
||||
$row = [ordered]@{
|
||||
frame_index = $index
|
||||
epoch_seconds = $epochSeconds
|
||||
session_seconds = $timelineStart + $epochSeconds
|
||||
}
|
||||
$timelineWriter.WriteLine(($row | ConvertTo-Json -Compress))
|
||||
$previousEpochSeconds = $epochSeconds
|
||||
}
|
||||
$timelineWriter.Flush()
|
||||
}
|
||||
finally {
|
||||
$timelineWriter.Dispose()
|
||||
}
|
||||
$extractWatch.Stop()
|
||||
$freeBytesPostExtract = Assert-FreeSpace "post-frame-extraction"
|
||||
Write-Output ("PHASE=e4-frame-extraction-complete FRAMES={0}" -f $activeFrameCount)
|
||||
|
||||
$runArgs = @(
|
||||
"run", "--rm", "--gpus", "all", "--network", "none", "--read-only",
|
||||
"--security-opt", "no-new-privileges:true", "--cap-drop", "ALL",
|
||||
"--pids-limit", "512", "--shm-size", "4g",
|
||||
"--tmpfs", "/tmp:rw,noexec,nosuid,size=2g",
|
||||
"-e", "PYTHONPATH=/opt/transformers:/opt/env",
|
||||
"-e", "TORCH_HOME=/cache/torch",
|
||||
"-e", "HF_HOME=/cache/huggingface",
|
||||
"-e", "HF_HUB_OFFLINE=1",
|
||||
"-e", "TRANSFORMERS_OFFLINE=1",
|
||||
"-v", ((Convert-ToDockerPath $jobDirectory) + ":/job:ro"),
|
||||
"-v", ((Convert-ToDockerPath $validFov) + ":/valid-fov:ro"),
|
||||
"-v", ((Convert-ToDockerPath $framesRoot) + ":/frames:ro"),
|
||||
"-v", ((Convert-ToDockerPath $workRoot) + ":/work:ro"),
|
||||
"-v", ((Convert-ToDockerPath $publishRoot) + ":/publish:rw"),
|
||||
"-v", ((Convert-ToDockerPath $cacheRoot) + ":/cache:ro"),
|
||||
"-v", ((Convert-ToDockerPath $e3Environment) + ":/environment:ro"),
|
||||
"-v", ((Convert-ToDockerPath $torchEnvironment) + ":/opt/env:ro"),
|
||||
"-v", ((Convert-ToDockerPath $transformersEnvironment) + ":/opt/transformers:ro"),
|
||||
"-v", ((Convert-ToDockerPath $runnerRoot) + ":/runner:ro"),
|
||||
"--entrypoint", "python3",
|
||||
$ContainerImage,
|
||||
("/runner/{0}" -f $runnerName), "run",
|
||||
"--job", "/job",
|
||||
"--profile", ("/runner/{0}" -f $profileName),
|
||||
"--valid-fov-root", "/valid-fov",
|
||||
"--frames", "/frames",
|
||||
"--timeline", "/work/timeline.jsonl",
|
||||
"--cache", "/cache",
|
||||
"--environment", "/environment",
|
||||
"--output", "/publish/output",
|
||||
"--frame-limit", [string]$activeFrameCount,
|
||||
"--free-bytes-floor", [string]([int64]$FreeGiBFloor * 1GB),
|
||||
"--orchestrator-sha256", $orchestratorSha256,
|
||||
"--container-image", $ContainerImage,
|
||||
"--telemetry-interval-seconds", "1"
|
||||
)
|
||||
Write-Output ("PHASE=e4-inference-start FRAMES={0}" -f $activeFrameCount)
|
||||
& docker @runArgs
|
||||
Assert-LastExitCode "LAB E4 semantic inference"
|
||||
$freeBytesPostInference = Assert-FreeSpace "post-inference"
|
||||
Write-Output "PHASE=e4-inference-complete"
|
||||
|
||||
if ($PilotFrames -gt 0) {
|
||||
$pilotParent = Join-Path $derivedRoot "e4-pilots"
|
||||
$null = New-Item -ItemType Directory -Path $pilotParent -Force
|
||||
$pilotRoot = Join-Path $pilotParent ("pilot-{0}-{1}" -f $activeFrameCount, $runToken)
|
||||
Move-Item -LiteralPath $stagingRoot -Destination $pilotRoot
|
||||
Remove-Item -LiteralPath $publishRoot -Force
|
||||
$completed = $true
|
||||
Write-Output ("PILOT_ROOT={0}" -f $pilotRoot)
|
||||
Write-Output ("PILOT_FRAMES={0}" -f $activeFrameCount)
|
||||
Write-Output ("DISK_FREE_BYTES_BEFORE={0}" -f $freeBytesBefore)
|
||||
Write-Output ("DISK_FREE_BYTES_POST_EXTRACT={0}" -f $freeBytesPostExtract)
|
||||
Write-Output ("DISK_FREE_BYTES_POST_INFERENCE={0}" -f $freeBytesPostInference)
|
||||
return
|
||||
}
|
||||
|
||||
$videoPath = Join-Path $stagingRoot "perception.mp4"
|
||||
$encodeWatch = [Diagnostics.Stopwatch]::StartNew()
|
||||
$averageFps = $fullFrameCount / $timelineDuration
|
||||
$fpsText = $averageFps.ToString("0.#########", [Globalization.CultureInfo]::InvariantCulture)
|
||||
$durationText = $timelineDuration.ToString("0.#########", [Globalization.CultureInfo]::InvariantCulture)
|
||||
& ffmpeg -hide_banner -loglevel error -framerate $fpsText -i (Join-Path $stagingRoot "overlay-frames\frame-%06d.png") -t $durationText -c:v h264_nvenc -preset p4 -tune hq -rc vbr -cq 21 -b:v 0 -pix_fmt yuv420p -movflags +faststart $videoPath
|
||||
Assert-LastExitCode "LAB E4 video encoding"
|
||||
$encodeWatch.Stop()
|
||||
$null = Assert-FreeSpace "post-video-encoding"
|
||||
Write-Output "PHASE=e4-video-encoding-complete"
|
||||
|
||||
$masksPath = Join-Path $stagingRoot "masks.tar.gz"
|
||||
$archiveWatch = [Diagnostics.Stopwatch]::StartNew()
|
||||
& tar.exe -czf $masksPath -C $stagingRoot semantic-masks
|
||||
Assert-LastExitCode "LAB E4 mask archive publication"
|
||||
$archiveWatch.Stop()
|
||||
$freeBytesPostArtifacts = Assert-FreeSpace "post-mask-archive"
|
||||
Write-Output "PHASE=e4-mask-archive-complete"
|
||||
|
||||
$totalWatch.Stop()
|
||||
$finalizeArgs = @(
|
||||
"run", "--rm", "--network", "none", "--read-only",
|
||||
"--security-opt", "no-new-privileges:true", "--cap-drop", "ALL",
|
||||
"--pids-limit", "128", "--tmpfs", "/tmp:rw,noexec,nosuid,size=256m",
|
||||
"-e", "PYTHONPATH=/opt/transformers:/opt/env",
|
||||
"-v", ((Convert-ToDockerPath $stagingRoot) + ":/output:rw"),
|
||||
"-v", ((Convert-ToDockerPath $runnerRoot) + ":/runner:ro"),
|
||||
"--entrypoint", "python3",
|
||||
$ContainerImage,
|
||||
("/runner/{0}" -f $runnerName), "finalize",
|
||||
"--output", "/output",
|
||||
"--video", "/output/perception.mp4",
|
||||
"--masks", "/output/masks.tar.gz",
|
||||
"--extract-seconds", $extractWatch.Elapsed.TotalSeconds.ToString("0.######", [Globalization.CultureInfo]::InvariantCulture),
|
||||
"--encode-seconds", $encodeWatch.Elapsed.TotalSeconds.ToString("0.######", [Globalization.CultureInfo]::InvariantCulture),
|
||||
"--archive-seconds", $archiveWatch.Elapsed.TotalSeconds.ToString("0.######", [Globalization.CultureInfo]::InvariantCulture),
|
||||
"--wall-seconds", $totalWatch.Elapsed.TotalSeconds.ToString("0.######", [Globalization.CultureInfo]::InvariantCulture),
|
||||
"--encoder", "ffmpeg-h264_nvenc-p4-cq21-yuv420p-faststart",
|
||||
"--disk-free-before-bytes", [string]$freeBytesBefore,
|
||||
"--disk-free-post-extract-bytes", [string]$freeBytesPostExtract,
|
||||
"--disk-free-post-inference-bytes", [string]$freeBytesPostInference,
|
||||
"--disk-free-post-artifacts-bytes", [string]$freeBytesPostArtifacts,
|
||||
"--disk-floor-bytes", [string]([int64]$FreeGiBFloor * 1GB),
|
||||
"--working-set-reserve-bytes", [string]$workingSetReserve
|
||||
)
|
||||
& docker @finalizeArgs
|
||||
Assert-LastExitCode "LAB E4 result finalization"
|
||||
|
||||
$result = Get-Content -LiteralPath (Join-Path $stagingRoot "result.json") -Raw | ConvertFrom-Json
|
||||
if (
|
||||
$result.schema_version -ne "missioncore.recorded-perception-result/v2" -or
|
||||
$result.result_id -notmatch "^result-[a-f0-9]{64}$" -or
|
||||
[int]$result.frames_processed -ne $fullFrameCount -or
|
||||
$result.ground_truth -ne $false
|
||||
) {
|
||||
throw "Final LAB E4 result manifest is incompatible"
|
||||
}
|
||||
$finalRoot = Join-Path $derivedRoot ([string]$result.result_id)
|
||||
if (Test-Path -LiteralPath $finalRoot) {
|
||||
throw "An immutable result with the same LAB E4 identity already exists: $finalRoot"
|
||||
}
|
||||
|
||||
foreach ($temporaryChild in @("overlay-frames", "semantic-masks")) {
|
||||
$temporaryPath = Join-Path $stagingRoot $temporaryChild
|
||||
if (Test-Path -LiteralPath $temporaryPath) {
|
||||
Remove-Item -LiteralPath $temporaryPath -Recurse -Force
|
||||
}
|
||||
}
|
||||
Move-Item -LiteralPath $stagingRoot -Destination $finalRoot
|
||||
Remove-Item -LiteralPath $publishRoot -Force
|
||||
$completed = $true
|
||||
$freeBytesFinal = Assert-FreeSpace "post-publication"
|
||||
Write-Output ("RESULT_ROOT={0}" -f $finalRoot)
|
||||
Write-Output ("RESULT_ID={0}" -f $result.result_id)
|
||||
Write-Output ("DISK_FREE_BYTES_BEFORE={0}" -f $freeBytesBefore)
|
||||
Write-Output ("DISK_FREE_BYTES_POST_EXTRACT={0}" -f $freeBytesPostExtract)
|
||||
Write-Output ("DISK_FREE_BYTES_POST_INFERENCE={0}" -f $freeBytesPostInference)
|
||||
Write-Output ("DISK_FREE_BYTES_POST_ARTIFACTS={0}" -f $freeBytesPostArtifacts)
|
||||
Write-Output ("DISK_FREE_BYTES_FINAL={0}" -f $freeBytesFinal)
|
||||
}
|
||||
finally {
|
||||
if (Test-Path -LiteralPath $workRoot) {
|
||||
Remove-Item -LiteralPath $workRoot -Recurse -Force
|
||||
}
|
||||
if (-not $completed -and (Test-Path -LiteralPath $publishRoot)) {
|
||||
Remove-Item -LiteralPath $publishRoot -Recurse -Force
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,544 @@
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[Parameter(Mandatory = $true)]
|
||||
[string]$JobRoot,
|
||||
|
||||
[Parameter(Mandatory = $true)]
|
||||
[string]$RunnerPath,
|
||||
|
||||
[Parameter(Mandatory = $true)]
|
||||
[string]$ProfilePath,
|
||||
|
||||
[Parameter(Mandatory = $true)]
|
||||
[string]$ValidFovRoot,
|
||||
|
||||
[ValidateRange(0, 1000000)]
|
||||
[int]$StartFrame = 1000,
|
||||
|
||||
[ValidateRange(0, 1000000)]
|
||||
[int]$EndFrame = 1600,
|
||||
|
||||
[ValidateRange(1, 1000)]
|
||||
[int]$FreeGiBFloor = 360,
|
||||
|
||||
[string]$RuntimeRoot = "D:\NDC_MISSIONCORE\runtime",
|
||||
|
||||
[string]$ModelRoot = "D:\NDC_MISSIONCORE\runtime\models\yolox_s",
|
||||
|
||||
[string]$TritonContainer = "mission-core-triton",
|
||||
|
||||
[string]$ContainerImage = "nvcr.io/nvidia/tritonserver:26.06-py3"
|
||||
)
|
||||
|
||||
$ErrorActionPreference = "Stop"
|
||||
$ProgressPreference = "SilentlyContinue"
|
||||
|
||||
function Assert-LastExitCode {
|
||||
param([string]$Operation)
|
||||
if ($LASTEXITCODE -ne 0) {
|
||||
throw "$Operation failed with exit code $LASTEXITCODE"
|
||||
}
|
||||
}
|
||||
|
||||
function Assert-RegularFile {
|
||||
param([string]$Path, [string]$Label)
|
||||
$item = Get-Item -LiteralPath $Path -Force
|
||||
if (-not $item.PSIsContainer -and -not ($item.Attributes -band [IO.FileAttributes]::ReparsePoint)) {
|
||||
return $item.FullName
|
||||
}
|
||||
throw "$Label is not a regular file"
|
||||
}
|
||||
|
||||
function Assert-Directory {
|
||||
param([string]$Path, [string]$Label)
|
||||
$item = Get-Item -LiteralPath $Path -Force
|
||||
if ($item.PSIsContainer -and -not ($item.Attributes -band [IO.FileAttributes]::ReparsePoint)) {
|
||||
return $item.FullName
|
||||
}
|
||||
throw "$Label is not a real directory"
|
||||
}
|
||||
|
||||
function Assert-DDrivePath {
|
||||
param([string]$Path, [string]$Label)
|
||||
$fullPath = [IO.Path]::GetFullPath($Path)
|
||||
if ([IO.Path]::GetPathRoot($fullPath).TrimEnd("\") -ine "D:") {
|
||||
throw "$Label must be stored on D:"
|
||||
}
|
||||
return $fullPath
|
||||
}
|
||||
|
||||
function Get-DFreeBytes {
|
||||
return [int64](Get-PSDrive -Name D).Free
|
||||
}
|
||||
|
||||
function Assert-FreeSpace {
|
||||
param(
|
||||
[string]$Phase,
|
||||
[int64]$RequiredAdditionalBytes = 0
|
||||
)
|
||||
$freeBytes = Get-DFreeBytes
|
||||
$floorBytes = [int64]$FreeGiBFloor * 1GB
|
||||
Write-Host (
|
||||
"DISK_GUARD PHASE={0} DRIVE=D FREE_BYTES={1} FREE_GIB={2} FLOOR_GIB={3} REQUIRED_ADDITIONAL_GIB={4}" -f
|
||||
$Phase,
|
||||
$freeBytes,
|
||||
[math]::Round($freeBytes / 1GB, 3),
|
||||
$FreeGiBFloor,
|
||||
[math]::Round($RequiredAdditionalBytes / 1GB, 3)
|
||||
)
|
||||
if ($freeBytes -lt ($floorBytes + $RequiredAdditionalBytes)) {
|
||||
throw "D: does not have the guarded LAB E5 working-set reserve during $Phase"
|
||||
}
|
||||
return $freeBytes
|
||||
}
|
||||
|
||||
function Convert-ToDockerPath {
|
||||
param([string]$Path)
|
||||
return $Path.Replace("\", "/")
|
||||
}
|
||||
|
||||
function Test-TritonModelReady {
|
||||
param([string]$ModelName)
|
||||
try {
|
||||
$response = Invoke-WebRequest `
|
||||
-Uri ("http://127.0.0.1:8000/v2/models/{0}/ready" -f $ModelName) `
|
||||
-Method Get `
|
||||
-UseBasicParsing `
|
||||
-TimeoutSec 10
|
||||
return $response.StatusCode -eq 200
|
||||
}
|
||||
catch {
|
||||
return $false
|
||||
}
|
||||
}
|
||||
|
||||
$jobDirectory = Assert-DDrivePath (
|
||||
Assert-Directory (Resolve-Path -LiteralPath $JobRoot).Path "Job root"
|
||||
) "Job root"
|
||||
$runner = Assert-DDrivePath (
|
||||
Assert-RegularFile (Resolve-Path -LiteralPath $RunnerPath).Path "LAB E5 runner"
|
||||
) "LAB E5 runner"
|
||||
$profile = Assert-DDrivePath (
|
||||
Assert-RegularFile (Resolve-Path -LiteralPath $ProfilePath).Path "LAB E5 profile"
|
||||
) "LAB E5 profile"
|
||||
$validFov = Assert-DDrivePath (
|
||||
Assert-Directory (Resolve-Path -LiteralPath $ValidFovRoot).Path "Valid-FOV root"
|
||||
) "Valid-FOV root"
|
||||
$runtime = Assert-DDrivePath (
|
||||
Assert-Directory (Resolve-Path -LiteralPath $RuntimeRoot).Path "Runtime root"
|
||||
) "Runtime root"
|
||||
$model = Assert-DDrivePath (
|
||||
Assert-Directory (Resolve-Path -LiteralPath $ModelRoot).Path "YOLOX model root"
|
||||
) "YOLOX model root"
|
||||
$runnerRoot = Split-Path $runner -Parent
|
||||
if ((Split-Path $profile -Parent) -ne $runnerRoot) {
|
||||
throw "LAB E5 runner and profile must share one read-only mount"
|
||||
}
|
||||
foreach ($dependency in @(
|
||||
"run_recorded_perception_epoch.py",
|
||||
"run_e3_rectified_segmentation.py",
|
||||
"run_evaluation_prelabels.py"
|
||||
)) {
|
||||
$null = Assert-RegularFile (Join-Path $runnerRoot $dependency) "LAB E5 runner dependency"
|
||||
}
|
||||
|
||||
$job = Get-Content -LiteralPath (Join-Path $jobDirectory "job.json") -Raw | ConvertFrom-Json
|
||||
if ($job.schema_version -ne "missioncore.compute-job/v1" -or $job.job_id -ne (Split-Path $jobDirectory -Leaf)) {
|
||||
throw "Compute job manifest is incompatible"
|
||||
}
|
||||
$sourceId = [string]$job.input.source_id
|
||||
$epoch = [int]$job.input.codec_epoch
|
||||
$fullFrameCount = [int]$job.input.segment_count
|
||||
$clipFrameCount = $EndFrame - $StartFrame + 1
|
||||
$timelineStart = [double]$job.input.timeline.start_seconds
|
||||
if (
|
||||
$sourceId -ne "sensor.camera.right" -or
|
||||
$StartFrame -lt 0 -or
|
||||
$EndFrame -lt $StartFrame -or
|
||||
$EndFrame -ge $fullFrameCount -or
|
||||
$clipFrameCount -lt 2
|
||||
) {
|
||||
throw "LAB E5 clip escapes the camera job"
|
||||
}
|
||||
Write-Output (
|
||||
"PHASE=job-manifest-validated JOB={0} CLIP={1}-{2} FRAMES={3}" -f
|
||||
$job.job_id, $StartFrame, $EndFrame, $clipFrameCount
|
||||
)
|
||||
|
||||
$epochRoot = Join-Path $jobDirectory ("input\camera\{0}\epoch-{1}" -f $sourceId, $epoch)
|
||||
$epochRoot = Assert-DDrivePath (Assert-Directory $epochRoot "Camera epoch") "Camera epoch"
|
||||
$initPath = Assert-RegularFile (Join-Path $epochRoot "init.mp4") "Camera init"
|
||||
$segmentsRoot = Assert-Directory (Join-Path $epochRoot "segments") "Camera segments"
|
||||
$derivedRoot = Join-Path $runtime "derived"
|
||||
$tmpRoot = Join-Path $runtime "tmp"
|
||||
$pythonEnvironment = Assert-DDrivePath (
|
||||
Assert-Directory (Join-Path $derivedRoot "perception-p0-env-v1") "Python environment"
|
||||
) "Python environment"
|
||||
$null = New-Item -ItemType Directory -Path $derivedRoot -Force
|
||||
$null = New-Item -ItemType Directory -Path $tmpRoot -Force
|
||||
|
||||
# Conservative bound: partial reconstructed stream, decoded RGB PNG payload,
|
||||
# RGB overlays, encoded result and 2 GiB of filesystem/codec overhead.
|
||||
$pixelWorkingSet = [int64]$clipFrameCount * 800 * 600 * 6
|
||||
$partialStreamReserve = [int64][math]::Ceiling(
|
||||
([int64]$job.input.byte_length * ($EndFrame + 1) / $fullFrameCount) * 1.2
|
||||
)
|
||||
$workingSetReserve = [int64][math]::Ceiling($pixelWorkingSet * 1.4) + $partialStreamReserve + 2GB
|
||||
$freeBytesBefore = Assert-FreeSpace "preflight" $workingSetReserve
|
||||
|
||||
& docker image inspect $ContainerImage *> $null
|
||||
Assert-LastExitCode "Existing container image inspection"
|
||||
$tritonState = docker inspect --format "{{.State.Running}}" $TritonContainer
|
||||
Assert-LastExitCode "Triton container inspection"
|
||||
if ($tritonState.Trim().ToLowerInvariant() -ne "true") {
|
||||
throw "LAB E5 requires the existing Triton container"
|
||||
}
|
||||
|
||||
$runnerName = Split-Path $runner -Leaf
|
||||
$profileName = Split-Path $profile -Leaf
|
||||
$orchestratorSha256 = (Get-FileHash -LiteralPath $PSCommandPath -Algorithm SHA256).Hash.ToLowerInvariant()
|
||||
$preflightArgs = @(
|
||||
"run", "--rm", "--network", "none", "--read-only",
|
||||
"--security-opt", "no-new-privileges:true", "--cap-drop", "ALL",
|
||||
"--pids-limit", "128", "--tmpfs", "/tmp:rw,noexec,nosuid,size=256m",
|
||||
"-e", "PYTHONPATH=/runner:/opt/env",
|
||||
"-v", ((Convert-ToDockerPath $jobDirectory) + ":/job:ro"),
|
||||
"-v", ((Convert-ToDockerPath $validFov) + ":/valid-fov:ro"),
|
||||
"-v", ((Convert-ToDockerPath $model) + ":/model:ro"),
|
||||
"-v", ((Convert-ToDockerPath $pythonEnvironment) + ":/opt/env:ro"),
|
||||
"-v", ((Convert-ToDockerPath $runnerRoot) + ":/runner:ro"),
|
||||
"--entrypoint", "python3",
|
||||
$ContainerImage,
|
||||
("/runner/{0}" -f $runnerName), "preflight",
|
||||
"--job", "/job",
|
||||
"--profile", ("/runner/{0}" -f $profileName),
|
||||
"--valid-fov-root", "/valid-fov",
|
||||
"--model-root", "/model"
|
||||
)
|
||||
Write-Output "PHASE=e5-preflight-start"
|
||||
& docker @preflightArgs
|
||||
Assert-LastExitCode "LAB E5 preflight"
|
||||
Write-Output "PHASE=e5-preflight-complete"
|
||||
|
||||
$modelWasReady = Test-TritonModelReady "yolox_s"
|
||||
$loadedByRun = $false
|
||||
if (-not $modelWasReady) {
|
||||
Invoke-WebRequest `
|
||||
-Uri "http://127.0.0.1:8000/v2/repository/models/yolox_s/load" `
|
||||
-Method Post `
|
||||
-ContentType "application/json" `
|
||||
-Body "{}" `
|
||||
-UseBasicParsing `
|
||||
-TimeoutSec 60 *> $null
|
||||
$loadedByRun = $true
|
||||
$deadline = [DateTime]::UtcNow.AddSeconds(60)
|
||||
while (-not (Test-TritonModelReady "yolox_s")) {
|
||||
if ([DateTime]::UtcNow -ge $deadline) {
|
||||
throw "YOLOX-S did not become ready in Triton"
|
||||
}
|
||||
Start-Sleep -Milliseconds 500
|
||||
}
|
||||
}
|
||||
Write-Output ("PHASE=e5-model-ready LOADED_BY_RUN={0}" -f $loadedByRun)
|
||||
|
||||
$runToken = [Guid]::NewGuid().ToString("N")
|
||||
$workRoot = Join-Path $tmpRoot ("{0}-e5-{1}" -f $job.job_id, $runToken)
|
||||
$framesRoot = Join-Path $workRoot "frames"
|
||||
$streamPath = Join-Path $workRoot "camera.mp4"
|
||||
$ptsPath = Join-Path $workRoot "pts.json"
|
||||
$timelinePath = Join-Path $workRoot "timeline.jsonl"
|
||||
$publishRoot = Join-Path $derivedRoot (".{0}-e5-{1}.publish" -f $job.job_id, $runToken)
|
||||
$stagingRoot = Join-Path $publishRoot "output"
|
||||
$null = New-Item -ItemType Directory -Path $framesRoot
|
||||
$null = New-Item -ItemType Directory -Path $publishRoot
|
||||
$totalWatch = [Diagnostics.Stopwatch]::StartNew()
|
||||
$completed = $false
|
||||
Write-Output "PHASE=e5-private-staging-created"
|
||||
|
||||
try {
|
||||
$stream = [IO.File]::Open(
|
||||
$streamPath,
|
||||
[IO.FileMode]::CreateNew,
|
||||
[IO.FileAccess]::Write,
|
||||
[IO.FileShare]::None
|
||||
)
|
||||
try {
|
||||
$input = [IO.File]::OpenRead($initPath)
|
||||
try {
|
||||
$input.CopyTo($stream)
|
||||
}
|
||||
finally {
|
||||
$input.Dispose()
|
||||
}
|
||||
for ($sequence = 1; $sequence -le ($EndFrame + 1); $sequence++) {
|
||||
$path = Assert-RegularFile (
|
||||
Join-Path $segmentsRoot ("{0}.m4s" -f $sequence)
|
||||
) "Camera segment"
|
||||
$input = [IO.File]::OpenRead($path)
|
||||
try {
|
||||
$input.CopyTo($stream)
|
||||
}
|
||||
finally {
|
||||
$input.Dispose()
|
||||
}
|
||||
if ($sequence % 500 -eq 0 -or $sequence -eq ($EndFrame + 1)) {
|
||||
Write-Output (
|
||||
"PHASE=e5-stream-reconstruction SEGMENTS={0}/{1}" -f
|
||||
$sequence, ($EndFrame + 1)
|
||||
)
|
||||
}
|
||||
}
|
||||
$stream.Flush($true)
|
||||
}
|
||||
finally {
|
||||
$stream.Dispose()
|
||||
}
|
||||
$null = Assert-FreeSpace "post-stream-reconstruction"
|
||||
|
||||
$extractWatch = [Diagnostics.Stopwatch]::StartNew()
|
||||
Write-Output "PHASE=e5-frame-extraction-start"
|
||||
$selectFilter = "select='between(n,{0},{1})'" -f $StartFrame, $EndFrame
|
||||
& ffmpeg `
|
||||
-hide_banner -loglevel fatal `
|
||||
-i $streamPath `
|
||||
-map 0:v:0 `
|
||||
-vf $selectFilter `
|
||||
-fps_mode passthrough `
|
||||
(Join-Path $framesRoot "frame-%06d.png")
|
||||
Assert-LastExitCode "LAB E5 camera clip extraction"
|
||||
& ffprobe `
|
||||
-v error `
|
||||
-select_streams v:0 `
|
||||
-show_entries frame=best_effort_timestamp_time `
|
||||
-of json `
|
||||
$streamPath | Set-Content -LiteralPath $ptsPath -Encoding utf8
|
||||
Assert-LastExitCode "LAB E5 camera timestamp probe"
|
||||
|
||||
$decodedFrames = @(Get-ChildItem -LiteralPath $framesRoot -File -Filter "frame-*.png")
|
||||
$ptsDocument = Get-Content -LiteralPath $ptsPath -Raw | ConvertFrom-Json
|
||||
$pts = @($ptsDocument.frames)
|
||||
if ($decodedFrames.Count -ne $clipFrameCount -or $pts.Count -ne ($EndFrame + 1)) {
|
||||
throw "Decoded LAB E5 frame/timestamp count differs from the selected clip"
|
||||
}
|
||||
$firstEpochSeconds = [double]::Parse(
|
||||
([string]$pts[0].best_effort_timestamp_time).Trim(),
|
||||
[Globalization.CultureInfo]::InvariantCulture
|
||||
)
|
||||
$previousSessionSeconds = -1.0
|
||||
$timelineWriter = [IO.StreamWriter]::new(
|
||||
$timelinePath,
|
||||
$false,
|
||||
[Text.UTF8Encoding]::new($false)
|
||||
)
|
||||
try {
|
||||
for ($localIndex = 0; $localIndex -lt $clipFrameCount; $localIndex++) {
|
||||
$sourceIndex = $StartFrame + $localIndex
|
||||
$epochSeconds = [double]::Parse(
|
||||
([string]$pts[$sourceIndex].best_effort_timestamp_time).Trim(),
|
||||
[Globalization.CultureInfo]::InvariantCulture
|
||||
) - $firstEpochSeconds
|
||||
$sessionSeconds = $timelineStart + $epochSeconds
|
||||
if ($sessionSeconds -le $previousSessionSeconds) {
|
||||
throw "Decoded LAB E5 clip timestamps are not strictly monotonic"
|
||||
}
|
||||
$row = [ordered]@{
|
||||
frame_index = $localIndex
|
||||
sequence = $localIndex + 1
|
||||
source_frame_index = $sourceIndex
|
||||
source_sequence = $sourceIndex + 1
|
||||
epoch_seconds = $epochSeconds
|
||||
session_seconds = $sessionSeconds
|
||||
}
|
||||
$timelineWriter.WriteLine(($row | ConvertTo-Json -Compress))
|
||||
$previousSessionSeconds = $sessionSeconds
|
||||
}
|
||||
$timelineWriter.Flush()
|
||||
}
|
||||
finally {
|
||||
$timelineWriter.Dispose()
|
||||
}
|
||||
$extractWatch.Stop()
|
||||
$freeBytesPostExtract = Assert-FreeSpace "post-frame-extraction"
|
||||
Write-Output ("PHASE=e5-frame-extraction-complete FRAMES={0}" -f $clipFrameCount)
|
||||
|
||||
$runArgs = @(
|
||||
"run", "--rm", "--gpus", "all",
|
||||
"--network", ("container:{0}" -f $TritonContainer),
|
||||
"--read-only",
|
||||
"--security-opt", "no-new-privileges:true", "--cap-drop", "ALL",
|
||||
"--pids-limit", "256", "--shm-size", "1g",
|
||||
"--tmpfs", "/tmp:rw,noexec,nosuid,size=512m",
|
||||
"-e", "PYTHONPATH=/runner:/opt/env",
|
||||
"-v", ((Convert-ToDockerPath $jobDirectory) + ":/job:ro"),
|
||||
"-v", ((Convert-ToDockerPath $validFov) + ":/valid-fov:ro"),
|
||||
"-v", ((Convert-ToDockerPath $model) + ":/model:ro"),
|
||||
"-v", ((Convert-ToDockerPath $pythonEnvironment) + ":/opt/env:ro"),
|
||||
"-v", ((Convert-ToDockerPath $framesRoot) + ":/frames:ro"),
|
||||
"-v", ((Convert-ToDockerPath $workRoot) + ":/work:ro"),
|
||||
"-v", ((Convert-ToDockerPath $publishRoot) + ":/publish:rw"),
|
||||
"-v", ((Convert-ToDockerPath $runnerRoot) + ":/runner:ro"),
|
||||
"--entrypoint", "python3",
|
||||
$ContainerImage,
|
||||
("/runner/{0}" -f $runnerName), "run",
|
||||
"--job", "/job",
|
||||
"--profile", ("/runner/{0}" -f $profileName),
|
||||
"--valid-fov-root", "/valid-fov",
|
||||
"--model-root", "/model",
|
||||
"--frames", "/frames",
|
||||
"--timeline", "/work/timeline.jsonl",
|
||||
"--output", "/publish/output",
|
||||
"--triton-url", "http://127.0.0.1:8000",
|
||||
"--free-bytes-floor", [string]([int64]$FreeGiBFloor * 1GB),
|
||||
"--orchestrator-sha256", $orchestratorSha256,
|
||||
"--container-image", $ContainerImage,
|
||||
"--telemetry-interval-seconds", "1"
|
||||
)
|
||||
Write-Output ("PHASE=e5-inference-start FRAMES={0}" -f $clipFrameCount)
|
||||
& docker @runArgs
|
||||
Assert-LastExitCode "LAB E5 detector/tracker inference"
|
||||
$freeBytesPostInference = Assert-FreeSpace "post-inference"
|
||||
Write-Output "PHASE=e5-inference-complete"
|
||||
|
||||
$timelineRows = @(Get-Content -LiteralPath $timelinePath | ForEach-Object {
|
||||
$_ | ConvertFrom-Json
|
||||
})
|
||||
$clipSpanSeconds = [double]$timelineRows[-1].session_seconds - [double]$timelineRows[0].session_seconds
|
||||
if ($clipSpanSeconds -le 0) {
|
||||
throw "LAB E5 clip duration is invalid"
|
||||
}
|
||||
$averageFps = ($clipFrameCount - 1) / $clipSpanSeconds
|
||||
$fpsText = $averageFps.ToString("0.#########", [Globalization.CultureInfo]::InvariantCulture)
|
||||
$videoPath = Join-Path $stagingRoot "tracking.mp4"
|
||||
$encodeWatch = [Diagnostics.Stopwatch]::StartNew()
|
||||
& ffmpeg `
|
||||
-hide_banner -loglevel error `
|
||||
-framerate $fpsText `
|
||||
-i (Join-Path $stagingRoot "overlay-frames\frame-%06d.png") `
|
||||
-c:v h264_nvenc `
|
||||
-preset p4 `
|
||||
-tune hq `
|
||||
-rc vbr `
|
||||
-cq 21 `
|
||||
-b:v 0 `
|
||||
-pix_fmt yuv420p `
|
||||
-movflags +faststart `
|
||||
$videoPath
|
||||
Assert-LastExitCode "LAB E5 video encoding"
|
||||
$encodeWatch.Stop()
|
||||
Write-Output "PHASE=e5-video-encoding-complete"
|
||||
|
||||
$contactSheetPath = Join-Path $stagingRoot "contact-sheet.png"
|
||||
$tileColumns = if ($clipFrameCount -le 60) { 4 } else { 3 }
|
||||
$tileRows = 2
|
||||
$sampleInterval = [math]::Max(0.25, $clipSpanSeconds / ($tileColumns * $tileRows))
|
||||
$sampleText = $sampleInterval.ToString("0.######", [Globalization.CultureInfo]::InvariantCulture)
|
||||
$contactFilter = "fps=1/{0},scale=400:-1,tile={1}x{2}:padding=8:margin=8:color=black" -f `
|
||||
$sampleText, $tileColumns, $tileRows
|
||||
& ffmpeg `
|
||||
-hide_banner -loglevel error `
|
||||
-i $videoPath `
|
||||
-vf $contactFilter `
|
||||
-frames:v 1 `
|
||||
$contactSheetPath
|
||||
Assert-LastExitCode "LAB E5 contact-sheet generation"
|
||||
|
||||
$probe = & ffprobe `
|
||||
-v error `
|
||||
-select_streams v:0 `
|
||||
-count_frames `
|
||||
-show_entries stream=codec_name,width,height,nb_read_frames `
|
||||
-of json `
|
||||
$videoPath | ConvertFrom-Json
|
||||
Assert-LastExitCode "LAB E5 result video probe"
|
||||
$videoStream = @($probe.streams)[0]
|
||||
if (
|
||||
$videoStream.codec_name -ne "h264" -or
|
||||
[int]$videoStream.width -ne 800 -or
|
||||
[int]$videoStream.height -ne 600 -or
|
||||
[int]$videoStream.nb_read_frames -ne $clipFrameCount
|
||||
) {
|
||||
throw "LAB E5 result video contract changed"
|
||||
}
|
||||
$freeBytesPostArtifacts = Assert-FreeSpace "post-artifacts"
|
||||
|
||||
$totalWatch.Stop()
|
||||
$finalizeArgs = @(
|
||||
"run", "--rm", "--network", "none", "--read-only",
|
||||
"--security-opt", "no-new-privileges:true", "--cap-drop", "ALL",
|
||||
"--pids-limit", "128", "--tmpfs", "/tmp:rw,noexec,nosuid,size=256m",
|
||||
"-e", "PYTHONPATH=/runner",
|
||||
"-v", ((Convert-ToDockerPath $stagingRoot) + ":/output:rw"),
|
||||
"-v", ((Convert-ToDockerPath $runnerRoot) + ":/runner:ro"),
|
||||
"--entrypoint", "python3",
|
||||
$ContainerImage,
|
||||
("/runner/{0}" -f $runnerName), "finalize",
|
||||
"--output", "/output",
|
||||
"--video", "/output/tracking.mp4",
|
||||
"--contact-sheet", "/output/contact-sheet.png",
|
||||
"--extract-seconds", $extractWatch.Elapsed.TotalSeconds.ToString("0.######", [Globalization.CultureInfo]::InvariantCulture),
|
||||
"--encode-seconds", $encodeWatch.Elapsed.TotalSeconds.ToString("0.######", [Globalization.CultureInfo]::InvariantCulture),
|
||||
"--wall-seconds", $totalWatch.Elapsed.TotalSeconds.ToString("0.######", [Globalization.CultureInfo]::InvariantCulture),
|
||||
"--encoder", "ffmpeg-h264_nvenc-p4-cq21-yuv420p-faststart",
|
||||
"--disk-free-before-bytes", [string]$freeBytesBefore,
|
||||
"--disk-free-post-extract-bytes", [string]$freeBytesPostExtract,
|
||||
"--disk-free-post-inference-bytes", [string]$freeBytesPostInference,
|
||||
"--disk-free-post-artifacts-bytes", [string]$freeBytesPostArtifacts,
|
||||
"--disk-floor-bytes", [string]([int64]$FreeGiBFloor * 1GB),
|
||||
"--working-set-reserve-bytes", [string]$workingSetReserve
|
||||
)
|
||||
& docker @finalizeArgs
|
||||
Assert-LastExitCode "LAB E5 result finalization"
|
||||
|
||||
$result = Get-Content -LiteralPath (Join-Path $stagingRoot "result.json") -Raw | ConvertFrom-Json
|
||||
if (
|
||||
$result.schema_version -ne "missioncore.e5-tracking-result/v1" -or
|
||||
$result.result_id -notmatch "^e5-tracking-[a-f0-9]{64}$" -or
|
||||
[int]$result.frames_processed -ne $clipFrameCount -or
|
||||
$result.ground_truth -ne $false -or
|
||||
$result.publication_scope -ne "qualification-clip-only"
|
||||
) {
|
||||
throw "Final LAB E5 result manifest is incompatible"
|
||||
}
|
||||
$finalRoot = Join-Path $derivedRoot ([string]$result.result_id)
|
||||
if (Test-Path -LiteralPath $finalRoot) {
|
||||
throw "An immutable result with the same LAB E5 identity already exists: $finalRoot"
|
||||
}
|
||||
$overlayFrames = Join-Path $stagingRoot "overlay-frames"
|
||||
if (Test-Path -LiteralPath $overlayFrames) {
|
||||
Remove-Item -LiteralPath $overlayFrames -Recurse -Force
|
||||
}
|
||||
Move-Item -LiteralPath $stagingRoot -Destination $finalRoot
|
||||
Remove-Item -LiteralPath $publishRoot -Force
|
||||
$completed = $true
|
||||
$freeBytesFinal = Assert-FreeSpace "post-publication"
|
||||
Write-Output ("RESULT_ROOT={0}" -f $finalRoot)
|
||||
Write-Output ("RESULT_ID={0}" -f $result.result_id)
|
||||
Write-Output ("DISK_FREE_BYTES_BEFORE={0}" -f $freeBytesBefore)
|
||||
Write-Output ("DISK_FREE_BYTES_POST_EXTRACT={0}" -f $freeBytesPostExtract)
|
||||
Write-Output ("DISK_FREE_BYTES_POST_INFERENCE={0}" -f $freeBytesPostInference)
|
||||
Write-Output ("DISK_FREE_BYTES_POST_ARTIFACTS={0}" -f $freeBytesPostArtifacts)
|
||||
Write-Output ("DISK_FREE_BYTES_FINAL={0}" -f $freeBytesFinal)
|
||||
}
|
||||
finally {
|
||||
if (Test-Path -LiteralPath $workRoot) {
|
||||
Remove-Item -LiteralPath $workRoot -Recurse -Force
|
||||
}
|
||||
if (-not $completed -and (Test-Path -LiteralPath $publishRoot)) {
|
||||
Remove-Item -LiteralPath $publishRoot -Recurse -Force
|
||||
}
|
||||
if ($loadedByRun) {
|
||||
try {
|
||||
Invoke-WebRequest `
|
||||
-Uri "http://127.0.0.1:8000/v2/repository/models/yolox_s/unload" `
|
||||
-Method Post `
|
||||
-ContentType "application/json" `
|
||||
-Body "{}" `
|
||||
-UseBasicParsing `
|
||||
-TimeoutSec 60 *> $null
|
||||
Write-Output "PHASE=e5-model-state-restored"
|
||||
}
|
||||
catch {
|
||||
Write-Warning "LAB E5 could not restore the prior unloaded YOLOX-S state"
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,411 @@
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[Parameter(Mandatory = $true)]
|
||||
[string]$JobRoot,
|
||||
|
||||
[Parameter(Mandatory = $true)]
|
||||
[string]$RunnerPath,
|
||||
|
||||
[Parameter(Mandatory = $true)]
|
||||
[string]$ProfilePath,
|
||||
|
||||
[Parameter(Mandatory = $true)]
|
||||
[string]$ValidFovRoot,
|
||||
|
||||
[ValidateRange(0, 1000000)]
|
||||
[int]$StartFrame = 1000,
|
||||
|
||||
[ValidateRange(0, 1000000)]
|
||||
[int]$EndFrame = 1600,
|
||||
|
||||
[ValidateRange(1, 1000)]
|
||||
[int]$FreeGiBFloor = 360,
|
||||
|
||||
[string]$RuntimeRoot = "D:\NDC_MISSIONCORE\runtime",
|
||||
|
||||
[string]$ModelRoot = "D:\NDC_MISSIONCORE\runtime\models\yolox_s",
|
||||
|
||||
[string]$TritonContainer = "mission-core-triton",
|
||||
|
||||
[string]$ContainerImage = "nvcr.io/nvidia/tritonserver:26.06-py3"
|
||||
)
|
||||
|
||||
$ErrorActionPreference = "Stop"
|
||||
$ProgressPreference = "SilentlyContinue"
|
||||
|
||||
function Assert-LastExitCode {
|
||||
param([string]$Operation)
|
||||
if ($LASTEXITCODE -ne 0) {
|
||||
throw "$Operation failed with exit code $LASTEXITCODE"
|
||||
}
|
||||
}
|
||||
|
||||
function Assert-RegularFile {
|
||||
param([string]$Path, [string]$Label)
|
||||
$item = Get-Item -LiteralPath $Path -Force
|
||||
if (-not $item.PSIsContainer -and -not ($item.Attributes -band [IO.FileAttributes]::ReparsePoint)) {
|
||||
return $item.FullName
|
||||
}
|
||||
throw "$Label is not a regular file"
|
||||
}
|
||||
|
||||
function Assert-Directory {
|
||||
param([string]$Path, [string]$Label)
|
||||
$item = Get-Item -LiteralPath $Path -Force
|
||||
if ($item.PSIsContainer -and -not ($item.Attributes -band [IO.FileAttributes]::ReparsePoint)) {
|
||||
return $item.FullName
|
||||
}
|
||||
throw "$Label is not a real directory"
|
||||
}
|
||||
|
||||
function Assert-DDrivePath {
|
||||
param([string]$Path, [string]$Label)
|
||||
$fullPath = [IO.Path]::GetFullPath($Path)
|
||||
if ([IO.Path]::GetPathRoot($fullPath).TrimEnd("\") -ine "D:") {
|
||||
throw "$Label must be stored on D:"
|
||||
}
|
||||
return $fullPath
|
||||
}
|
||||
|
||||
function Convert-ToDockerPath {
|
||||
param([string]$Path)
|
||||
return $Path.Replace("\", "/")
|
||||
}
|
||||
|
||||
function Get-DFreeBytes {
|
||||
return [int64](Get-PSDrive -Name D).Free
|
||||
}
|
||||
|
||||
function Assert-FreeSpace {
|
||||
param([string]$Phase, [int64]$RequiredAdditionalBytes = 0)
|
||||
$freeBytes = Get-DFreeBytes
|
||||
$floorBytes = [int64]$FreeGiBFloor * 1GB
|
||||
Write-Host (
|
||||
"DISK_GUARD PHASE={0} DRIVE=D FREE_BYTES={1} FREE_GIB={2} FLOOR_GIB={3} REQUIRED_ADDITIONAL_GIB={4}" -f
|
||||
$Phase,
|
||||
$freeBytes,
|
||||
[math]::Round($freeBytes / 1GB, 3),
|
||||
$FreeGiBFloor,
|
||||
[math]::Round($RequiredAdditionalBytes / 1GB, 3)
|
||||
)
|
||||
if ($freeBytes -lt ($floorBytes + $RequiredAdditionalBytes)) {
|
||||
throw "D: does not have the guarded LAB E8 working-set reserve during $Phase"
|
||||
}
|
||||
return $freeBytes
|
||||
}
|
||||
|
||||
function Test-TritonModelReady {
|
||||
try {
|
||||
$response = Invoke-WebRequest `
|
||||
-Uri "http://127.0.0.1:8000/v2/models/yolox_s/ready" `
|
||||
-Method Get -UseBasicParsing -TimeoutSec 10
|
||||
return $response.StatusCode -eq 200
|
||||
}
|
||||
catch {
|
||||
return $false
|
||||
}
|
||||
}
|
||||
|
||||
$jobDirectory = Assert-DDrivePath (
|
||||
Assert-Directory (Resolve-Path -LiteralPath $JobRoot).Path "Job root"
|
||||
) "Job root"
|
||||
$runner = Assert-DDrivePath (
|
||||
Assert-RegularFile (Resolve-Path -LiteralPath $RunnerPath).Path "LAB E8 runner"
|
||||
) "LAB E8 runner"
|
||||
$profile = Assert-DDrivePath (
|
||||
Assert-RegularFile (Resolve-Path -LiteralPath $ProfilePath).Path "LAB E8 profile"
|
||||
) "LAB E8 profile"
|
||||
$validFov = Assert-DDrivePath (
|
||||
Assert-Directory (Resolve-Path -LiteralPath $ValidFovRoot).Path "Valid-FOV root"
|
||||
) "Valid-FOV root"
|
||||
$runtime = Assert-DDrivePath (
|
||||
Assert-Directory (Resolve-Path -LiteralPath $RuntimeRoot).Path "Runtime root"
|
||||
) "Runtime root"
|
||||
$model = Assert-DDrivePath (
|
||||
Assert-Directory (Resolve-Path -LiteralPath $ModelRoot).Path "YOLOX model root"
|
||||
) "YOLOX model root"
|
||||
$runnerRoot = Split-Path $runner -Parent
|
||||
if ((Split-Path $profile -Parent) -ne $runnerRoot) {
|
||||
throw "LAB E8 runner and profile must share one read-only mount"
|
||||
}
|
||||
foreach ($dependency in @(
|
||||
"run_e5_instance_tracking.py",
|
||||
"run_recorded_perception_epoch.py",
|
||||
"run_e3_rectified_segmentation.py",
|
||||
"run_evaluation_prelabels.py"
|
||||
)) {
|
||||
$null = Assert-RegularFile (Join-Path $runnerRoot $dependency) "LAB E8 runner dependency"
|
||||
}
|
||||
|
||||
$job = Get-Content -LiteralPath (Join-Path $jobDirectory "job.json") -Raw | ConvertFrom-Json
|
||||
if ($job.schema_version -ne "missioncore.compute-job/v1" -or $job.job_id -ne (Split-Path $jobDirectory -Leaf)) {
|
||||
throw "Compute job manifest is incompatible"
|
||||
}
|
||||
$sourceId = [string]$job.input.source_id
|
||||
$epoch = [int]$job.input.codec_epoch
|
||||
$fullFrameCount = [int]$job.input.segment_count
|
||||
$clipFrameCount = $EndFrame - $StartFrame + 1
|
||||
$timelineStart = [double]$job.input.timeline.start_seconds
|
||||
if (
|
||||
$sourceId -ne "sensor.camera.right" -or
|
||||
$StartFrame -lt 0 -or
|
||||
$EndFrame -lt $StartFrame -or
|
||||
$EndFrame -ge $fullFrameCount -or
|
||||
$clipFrameCount -lt 2
|
||||
) {
|
||||
throw "LAB E8 clip escapes the camera job"
|
||||
}
|
||||
Write-Output (
|
||||
"PHASE=job-manifest-validated JOB={0} CLIP={1}-{2} FRAMES={3}" -f
|
||||
$job.job_id, $StartFrame, $EndFrame, $clipFrameCount
|
||||
)
|
||||
|
||||
$epochRoot = Assert-DDrivePath (
|
||||
Assert-Directory (
|
||||
Join-Path $jobDirectory ("input\camera\{0}\epoch-{1}" -f $sourceId, $epoch)
|
||||
) "Camera epoch"
|
||||
) "Camera epoch"
|
||||
$initPath = Assert-RegularFile (Join-Path $epochRoot "init.mp4") "Camera init"
|
||||
$segmentsRoot = Assert-Directory (Join-Path $epochRoot "segments") "Camera segments"
|
||||
$derivedRoot = Join-Path $runtime "derived"
|
||||
$tmpRoot = Join-Path $runtime "tmp"
|
||||
$pythonEnvironment = Assert-DDrivePath (
|
||||
Assert-Directory (Join-Path $derivedRoot "perception-p0-env-v1") "Python environment"
|
||||
) "Python environment"
|
||||
$null = New-Item -ItemType Directory -Path $derivedRoot -Force
|
||||
$null = New-Item -ItemType Directory -Path $tmpRoot -Force
|
||||
|
||||
# LAB E8 has temporary decoded input but no per-frame output images. Reserve
|
||||
# decoded RGB payload, partial stream and two GiB for filesystem/codec overhead.
|
||||
$pixelWorkingSet = [int64]$clipFrameCount * 800 * 600 * 3
|
||||
$partialStreamReserve = [int64][math]::Ceiling(
|
||||
([int64]$job.input.byte_length * ($EndFrame + 1) / $fullFrameCount) * 1.2
|
||||
)
|
||||
$workingSetReserve = [int64][math]::Ceiling($pixelWorkingSet * 1.4) + $partialStreamReserve + 2GB
|
||||
$freeBytesBefore = Assert-FreeSpace "preflight" $workingSetReserve
|
||||
|
||||
& docker image inspect $ContainerImage *> $null
|
||||
Assert-LastExitCode "Existing container image inspection"
|
||||
$tritonState = docker inspect --format "{{.State.Running}}" $TritonContainer
|
||||
Assert-LastExitCode "Triton container inspection"
|
||||
if ($tritonState.Trim().ToLowerInvariant() -ne "true") {
|
||||
throw "LAB E8 requires the existing Triton container"
|
||||
}
|
||||
|
||||
$runnerName = Split-Path $runner -Leaf
|
||||
$profileName = Split-Path $profile -Leaf
|
||||
$orchestratorSha256 = (Get-FileHash -LiteralPath $PSCommandPath -Algorithm SHA256).Hash.ToLowerInvariant()
|
||||
$preflightArgs = @(
|
||||
"run", "--rm", "--network", "none", "--read-only",
|
||||
"--security-opt", "no-new-privileges:true", "--cap-drop", "ALL",
|
||||
"--pids-limit", "128", "--tmpfs", "/tmp:rw,noexec,nosuid,size=256m",
|
||||
"-e", "PYTHONPATH=/runner:/opt/env",
|
||||
"-v", ((Convert-ToDockerPath $jobDirectory) + ":/job:ro"),
|
||||
"-v", ((Convert-ToDockerPath $validFov) + ":/valid-fov:ro"),
|
||||
"-v", ((Convert-ToDockerPath $model) + ":/model:ro"),
|
||||
"-v", ((Convert-ToDockerPath $pythonEnvironment) + ":/opt/env:ro"),
|
||||
"-v", ((Convert-ToDockerPath $runnerRoot) + ":/runner:ro"),
|
||||
"--entrypoint", "python3", $ContainerImage,
|
||||
("/runner/{0}" -f $runnerName), "preflight",
|
||||
"--job", "/job",
|
||||
"--profile", ("/runner/{0}" -f $profileName),
|
||||
"--valid-fov-root", "/valid-fov",
|
||||
"--model-root", "/model"
|
||||
)
|
||||
Write-Output "PHASE=e8-preflight-start"
|
||||
& docker @preflightArgs
|
||||
Assert-LastExitCode "LAB E8 preflight"
|
||||
Write-Output "PHASE=e8-preflight-complete"
|
||||
|
||||
$modelWasReady = Test-TritonModelReady
|
||||
$loadedByRun = $false
|
||||
if (-not $modelWasReady) {
|
||||
Invoke-WebRequest `
|
||||
-Uri "http://127.0.0.1:8000/v2/repository/models/yolox_s/load" `
|
||||
-Method Post -ContentType "application/json" -Body "{}" `
|
||||
-UseBasicParsing -TimeoutSec 60 *> $null
|
||||
$loadedByRun = $true
|
||||
$deadline = [DateTime]::UtcNow.AddSeconds(60)
|
||||
while (-not (Test-TritonModelReady)) {
|
||||
if ([DateTime]::UtcNow -ge $deadline) {
|
||||
throw "YOLOX-S did not become ready in Triton"
|
||||
}
|
||||
Start-Sleep -Milliseconds 500
|
||||
}
|
||||
}
|
||||
Write-Output ("PHASE=e8-model-ready LOADED_BY_RUN={0}" -f $loadedByRun)
|
||||
|
||||
$runToken = [Guid]::NewGuid().ToString("N")
|
||||
$workRoot = Join-Path $tmpRoot ("{0}-e8-{1}" -f $job.job_id, $runToken)
|
||||
$framesRoot = Join-Path $workRoot "frames"
|
||||
$streamPath = Join-Path $workRoot "camera.mp4"
|
||||
$ptsPath = Join-Path $workRoot "pts.json"
|
||||
$timelinePath = Join-Path $workRoot "timeline.jsonl"
|
||||
$publishRoot = Join-Path $derivedRoot (".{0}-e8-{1}.publish" -f $job.job_id, $runToken)
|
||||
$stagingRoot = Join-Path $publishRoot "output"
|
||||
$null = New-Item -ItemType Directory -Path $framesRoot
|
||||
$null = New-Item -ItemType Directory -Path $publishRoot
|
||||
$totalWatch = [Diagnostics.Stopwatch]::StartNew()
|
||||
$completed = $false
|
||||
Write-Output "PHASE=e8-private-staging-created"
|
||||
|
||||
try {
|
||||
$stream = [IO.File]::Open(
|
||||
$streamPath, [IO.FileMode]::CreateNew, [IO.FileAccess]::Write, [IO.FileShare]::None
|
||||
)
|
||||
try {
|
||||
$input = [IO.File]::OpenRead($initPath)
|
||||
try { $input.CopyTo($stream) } finally { $input.Dispose() }
|
||||
for ($sequence = 1; $sequence -le ($EndFrame + 1); $sequence++) {
|
||||
$path = Assert-RegularFile (
|
||||
Join-Path $segmentsRoot ("{0}.m4s" -f $sequence)
|
||||
) "Camera segment"
|
||||
$input = [IO.File]::OpenRead($path)
|
||||
try { $input.CopyTo($stream) } finally { $input.Dispose() }
|
||||
}
|
||||
$stream.Flush($true)
|
||||
}
|
||||
finally {
|
||||
$stream.Dispose()
|
||||
}
|
||||
$null = Assert-FreeSpace "post-stream-reconstruction"
|
||||
|
||||
$extractWatch = [Diagnostics.Stopwatch]::StartNew()
|
||||
Write-Output "PHASE=e8-frame-extraction-start"
|
||||
$selectFilter = "select='between(n,{0},{1})'" -f $StartFrame, $EndFrame
|
||||
& ffmpeg -hide_banner -loglevel fatal -i $streamPath -map 0:v:0 `
|
||||
-vf $selectFilter -fps_mode passthrough (Join-Path $framesRoot "frame-%06d.png")
|
||||
Assert-LastExitCode "LAB E8 camera clip extraction"
|
||||
& ffprobe -v error -select_streams v:0 `
|
||||
-show_entries frame=best_effort_timestamp_time -of json $streamPath |
|
||||
Set-Content -LiteralPath $ptsPath -Encoding utf8
|
||||
Assert-LastExitCode "LAB E8 camera timestamp probe"
|
||||
|
||||
$decodedFrames = @(Get-ChildItem -LiteralPath $framesRoot -File -Filter "frame-*.png")
|
||||
$ptsDocument = Get-Content -LiteralPath $ptsPath -Raw | ConvertFrom-Json
|
||||
$pts = @($ptsDocument.frames)
|
||||
if ($decodedFrames.Count -ne $clipFrameCount -or $pts.Count -ne ($EndFrame + 1)) {
|
||||
throw "Decoded LAB E8 frame/timestamp count differs from the selected clip"
|
||||
}
|
||||
$firstEpochSeconds = [double]::Parse(
|
||||
([string]$pts[0].best_effort_timestamp_time).Trim(),
|
||||
[Globalization.CultureInfo]::InvariantCulture
|
||||
)
|
||||
$previousSessionSeconds = -1.0
|
||||
$timelineWriter = [IO.StreamWriter]::new(
|
||||
$timelinePath, $false, [Text.UTF8Encoding]::new($false)
|
||||
)
|
||||
try {
|
||||
for ($localIndex = 0; $localIndex -lt $clipFrameCount; $localIndex++) {
|
||||
$sourceIndex = $StartFrame + $localIndex
|
||||
$epochSeconds = [double]::Parse(
|
||||
([string]$pts[$sourceIndex].best_effort_timestamp_time).Trim(),
|
||||
[Globalization.CultureInfo]::InvariantCulture
|
||||
) - $firstEpochSeconds
|
||||
$sessionSeconds = $timelineStart + $epochSeconds
|
||||
if ($sessionSeconds -le $previousSessionSeconds) {
|
||||
throw "Decoded LAB E8 clip timestamps are not strictly monotonic"
|
||||
}
|
||||
$row = [ordered]@{
|
||||
frame_index = $localIndex
|
||||
sequence = $localIndex + 1
|
||||
source_frame_index = $sourceIndex
|
||||
source_sequence = $sourceIndex + 1
|
||||
epoch_seconds = $epochSeconds
|
||||
session_seconds = $sessionSeconds
|
||||
}
|
||||
$timelineWriter.WriteLine(($row | ConvertTo-Json -Compress))
|
||||
$previousSessionSeconds = $sessionSeconds
|
||||
}
|
||||
$timelineWriter.Flush()
|
||||
}
|
||||
finally {
|
||||
$timelineWriter.Dispose()
|
||||
}
|
||||
$extractWatch.Stop()
|
||||
$freeBytesPostExtract = Assert-FreeSpace "post-frame-extraction"
|
||||
Write-Output ("PHASE=e8-frame-extraction-complete FRAMES={0}" -f $clipFrameCount)
|
||||
|
||||
$runArgs = @(
|
||||
"run", "--rm", "--gpus", "all",
|
||||
"--network", ("container:{0}" -f $TritonContainer),
|
||||
"--read-only", "--security-opt", "no-new-privileges:true", "--cap-drop", "ALL",
|
||||
"--pids-limit", "256", "--shm-size", "1g",
|
||||
"--tmpfs", "/tmp:rw,noexec,nosuid,size=512m",
|
||||
"-e", "PYTHONPATH=/runner:/opt/env",
|
||||
"-v", ((Convert-ToDockerPath $jobDirectory) + ":/job:ro"),
|
||||
"-v", ((Convert-ToDockerPath $validFov) + ":/valid-fov:ro"),
|
||||
"-v", ((Convert-ToDockerPath $model) + ":/model:ro"),
|
||||
"-v", ((Convert-ToDockerPath $pythonEnvironment) + ":/opt/env:ro"),
|
||||
"-v", ((Convert-ToDockerPath $framesRoot) + ":/frames:ro"),
|
||||
"-v", ((Convert-ToDockerPath $workRoot) + ":/work:ro"),
|
||||
"-v", ((Convert-ToDockerPath $publishRoot) + ":/publish:rw"),
|
||||
"-v", ((Convert-ToDockerPath $runnerRoot) + ":/runner:ro"),
|
||||
"--entrypoint", "python3", $ContainerImage,
|
||||
("/runner/{0}" -f $runnerName), "run",
|
||||
"--job", "/job",
|
||||
"--profile", ("/runner/{0}" -f $profileName),
|
||||
"--valid-fov-root", "/valid-fov",
|
||||
"--model-root", "/model",
|
||||
"--frames", "/frames",
|
||||
"--timeline", "/work/timeline.jsonl",
|
||||
"--output", "/publish/output",
|
||||
"--triton-url", "http://127.0.0.1:8000",
|
||||
"--free-bytes-floor", [string]([int64]$FreeGiBFloor * 1GB),
|
||||
"--orchestrator-sha256", $orchestratorSha256,
|
||||
"--container-image", $ContainerImage,
|
||||
"--telemetry-interval-seconds", "1"
|
||||
)
|
||||
Write-Output ("PHASE=e8-source-paced-replay-start FRAMES={0}" -f $clipFrameCount)
|
||||
& docker @runArgs
|
||||
Assert-LastExitCode "LAB E8 source-paced detector/tracker"
|
||||
$freeBytesPostReplay = Assert-FreeSpace "post-replay"
|
||||
Write-Output "PHASE=e8-source-paced-replay-complete"
|
||||
|
||||
$result = Get-Content -LiteralPath (Join-Path $stagingRoot "result.json") -Raw | ConvertFrom-Json
|
||||
if (
|
||||
$result.schema_version -ne "missioncore.e8-realtime-tracking-result/v1" -or
|
||||
$result.result_id -notmatch "^e8-realtime-tracking-[a-f0-9]{64}$" -or
|
||||
$result.acceptance_state -ne "accepted" -or
|
||||
$result.ground_truth -ne $false
|
||||
) {
|
||||
throw "Final LAB E8 result manifest is incompatible or rejected"
|
||||
}
|
||||
$finalRoot = Join-Path $derivedRoot ([string]$result.result_id)
|
||||
if (Test-Path -LiteralPath $finalRoot) {
|
||||
throw "An immutable result with the same LAB E8 identity already exists: $finalRoot"
|
||||
}
|
||||
Move-Item -LiteralPath $stagingRoot -Destination $finalRoot
|
||||
Remove-Item -LiteralPath $publishRoot -Force
|
||||
$totalWatch.Stop()
|
||||
$completed = $true
|
||||
$freeBytesFinal = Assert-FreeSpace "post-publication"
|
||||
Write-Output ("RESULT_ROOT={0}" -f $finalRoot)
|
||||
Write-Output ("RESULT_ID={0}" -f $result.result_id)
|
||||
Write-Output ("PREPARATION_SECONDS={0}" -f $extractWatch.Elapsed.TotalSeconds)
|
||||
Write-Output ("ORCHESTRATOR_WALL_SECONDS={0}" -f $totalWatch.Elapsed.TotalSeconds)
|
||||
Write-Output ("DISK_FREE_BYTES_BEFORE={0}" -f $freeBytesBefore)
|
||||
Write-Output ("DISK_FREE_BYTES_POST_EXTRACT={0}" -f $freeBytesPostExtract)
|
||||
Write-Output ("DISK_FREE_BYTES_POST_REPLAY={0}" -f $freeBytesPostReplay)
|
||||
Write-Output ("DISK_FREE_BYTES_FINAL={0}" -f $freeBytesFinal)
|
||||
}
|
||||
finally {
|
||||
if (Test-Path -LiteralPath $workRoot) {
|
||||
Remove-Item -LiteralPath $workRoot -Recurse -Force
|
||||
}
|
||||
if (-not $completed -and (Test-Path -LiteralPath $publishRoot)) {
|
||||
Remove-Item -LiteralPath $publishRoot -Recurse -Force
|
||||
}
|
||||
if ($loadedByRun) {
|
||||
try {
|
||||
Invoke-WebRequest `
|
||||
-Uri "http://127.0.0.1:8000/v2/repository/models/yolox_s/unload" `
|
||||
-Method Post -ContentType "application/json" -Body "{}" `
|
||||
-UseBasicParsing -TimeoutSec 60 *> $null
|
||||
Write-Output "PHASE=e8-model-state-restored"
|
||||
}
|
||||
catch {
|
||||
Write-Warning "LAB E8 could not restore the prior unloaded YOLOX-S state"
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,349 @@
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[Parameter(Mandatory = $true)] [string]$JobRoot,
|
||||
[Parameter(Mandatory = $true)] [string]$RunnerPath,
|
||||
[Parameter(Mandatory = $true)] [string]$ProfilePath,
|
||||
[Parameter(Mandatory = $true)] [string]$DetectorProfilePath,
|
||||
[Parameter(Mandatory = $true)] [string]$SemanticProfilePath,
|
||||
[Parameter(Mandatory = $true)] [string]$ValidFovRoot,
|
||||
[ValidateRange(0, 1000000)] [int]$StartFrame = 1000,
|
||||
[ValidateRange(0, 1000000)] [int]$EndFrame = 1600,
|
||||
[ValidateRange(1, 1000)] [int]$FreeGiBFloor = 360,
|
||||
[string]$RuntimeRoot = "D:\NDC_MISSIONCORE\runtime",
|
||||
[string]$ModelRoot = "D:\NDC_MISSIONCORE\runtime\models\yolox_s",
|
||||
[string]$TritonContainer = "mission-core-triton",
|
||||
[string]$ContainerImage = "nvcr.io/nvidia/tritonserver:26.06-py3"
|
||||
)
|
||||
|
||||
$ErrorActionPreference = "Stop"
|
||||
$ProgressPreference = "SilentlyContinue"
|
||||
|
||||
function Assert-LastExitCode {
|
||||
param([string]$Operation)
|
||||
if ($LASTEXITCODE -ne 0) { throw "$Operation failed with exit code $LASTEXITCODE" }
|
||||
}
|
||||
|
||||
function Resolve-DDirectory {
|
||||
param([string]$Path, [string]$Label)
|
||||
$item = Get-Item -LiteralPath (Resolve-Path -LiteralPath $Path).Path -Force
|
||||
$root = [IO.Path]::GetPathRoot($item.FullName).TrimEnd("\")
|
||||
if (-not $item.PSIsContainer -or ($item.Attributes -band [IO.FileAttributes]::ReparsePoint) -or $root -ine "D:") {
|
||||
throw "$Label must be a real D: directory"
|
||||
}
|
||||
return $item.FullName
|
||||
}
|
||||
|
||||
function Resolve-DFile {
|
||||
param([string]$Path, [string]$Label)
|
||||
$item = Get-Item -LiteralPath (Resolve-Path -LiteralPath $Path).Path -Force
|
||||
$root = [IO.Path]::GetPathRoot($item.FullName).TrimEnd("\")
|
||||
if ($item.PSIsContainer -or ($item.Attributes -band [IO.FileAttributes]::ReparsePoint) -or $root -ine "D:") {
|
||||
throw "$Label must be a regular D: file"
|
||||
}
|
||||
return $item.FullName
|
||||
}
|
||||
|
||||
function Convert-ToDockerPath {
|
||||
param([string]$Path)
|
||||
return $Path.Replace("\", "/")
|
||||
}
|
||||
|
||||
function Get-DFreeBytes { return [int64](Get-PSDrive -Name D).Free }
|
||||
|
||||
function Assert-FreeSpace {
|
||||
param([string]$Phase, [int64]$RequiredAdditionalBytes = 0)
|
||||
$freeBytes = Get-DFreeBytes
|
||||
$floorBytes = [int64]$FreeGiBFloor * 1GB
|
||||
Write-Host (
|
||||
"DISK_GUARD PHASE={0} DRIVE=D FREE_BYTES={1} FREE_GIB={2} FLOOR_GIB={3} REQUIRED_ADDITIONAL_GIB={4}" -f
|
||||
$Phase, $freeBytes, [math]::Round($freeBytes / 1GB, 3), $FreeGiBFloor,
|
||||
[math]::Round($RequiredAdditionalBytes / 1GB, 3)
|
||||
)
|
||||
if ($freeBytes -lt ($floorBytes + $RequiredAdditionalBytes)) {
|
||||
throw "D: does not have the guarded LAB E9 working-set reserve during $Phase"
|
||||
}
|
||||
return $freeBytes
|
||||
}
|
||||
|
||||
function Test-TritonModelReady {
|
||||
try {
|
||||
$response = Invoke-WebRequest -UseBasicParsing -TimeoutSec 10 `
|
||||
-Uri "http://127.0.0.1:8000/v2/models/yolox_s/ready"
|
||||
return $response.StatusCode -eq 200
|
||||
}
|
||||
catch { return $false }
|
||||
}
|
||||
|
||||
$jobDirectory = Resolve-DDirectory $JobRoot "Job root"
|
||||
$runner = Resolve-DFile $RunnerPath "LAB E9 runner"
|
||||
$profile = Resolve-DFile $ProfilePath "LAB E9 profile"
|
||||
$detectorProfile = Resolve-DFile $DetectorProfilePath "LAB E9 detector profile"
|
||||
$semanticProfile = Resolve-DFile $SemanticProfilePath "LAB E9 semantic profile"
|
||||
$validFov = Resolve-DDirectory $ValidFovRoot "Valid-FOV root"
|
||||
$runtime = Resolve-DDirectory $RuntimeRoot "Runtime root"
|
||||
$model = Resolve-DDirectory $ModelRoot "YOLOX model root"
|
||||
$runnerRoot = Split-Path $runner -Parent
|
||||
foreach ($path in @($profile, $detectorProfile, $semanticProfile)) {
|
||||
if ((Split-Path $path -Parent) -ne $runnerRoot) {
|
||||
throw "LAB E9 runner and profiles must share one read-only mount"
|
||||
}
|
||||
}
|
||||
foreach ($dependency in @(
|
||||
"run_e8_realtime_tracking.py",
|
||||
"run_e5_instance_tracking.py",
|
||||
"run_e4_full_session_segmentation.py",
|
||||
"run_recorded_perception_epoch.py",
|
||||
"run_e3_rectified_segmentation.py",
|
||||
"run_evaluation_prelabels.py"
|
||||
)) {
|
||||
$null = Resolve-DFile (Join-Path $runnerRoot $dependency) "LAB E9 runner dependency"
|
||||
}
|
||||
|
||||
$job = Get-Content -LiteralPath (Join-Path $jobDirectory "job.json") -Raw | ConvertFrom-Json
|
||||
if ($job.schema_version -ne "missioncore.compute-job/v1" -or $job.job_id -ne (Split-Path $jobDirectory -Leaf)) {
|
||||
throw "Compute job manifest is incompatible"
|
||||
}
|
||||
$sourceId = [string]$job.input.source_id
|
||||
$epoch = [int]$job.input.codec_epoch
|
||||
$fullFrameCount = [int]$job.input.segment_count
|
||||
$clipFrameCount = $EndFrame - $StartFrame + 1
|
||||
$timelineStart = [double]$job.input.timeline.start_seconds
|
||||
if (
|
||||
$sourceId -ne "sensor.camera.right" -or $StartFrame -lt 0 -or
|
||||
$EndFrame -lt $StartFrame -or $EndFrame -ge $fullFrameCount -or $clipFrameCount -lt 2
|
||||
) { throw "LAB E9 clip escapes the camera job" }
|
||||
Write-Output (
|
||||
"PHASE=job-manifest-validated JOB={0} CLIP={1}-{2} FRAMES={3}" -f
|
||||
$job.job_id, $StartFrame, $EndFrame, $clipFrameCount
|
||||
)
|
||||
|
||||
$epochRoot = Resolve-DDirectory (
|
||||
Join-Path $jobDirectory ("input\camera\{0}\epoch-{1}" -f $sourceId, $epoch)
|
||||
) "Camera epoch"
|
||||
$initPath = Resolve-DFile (Join-Path $epochRoot "init.mp4") "Camera init"
|
||||
$segmentsRoot = Resolve-DDirectory (Join-Path $epochRoot "segments") "Camera segments"
|
||||
$derivedRoot = Join-Path $runtime "derived"
|
||||
$tmpRoot = Join-Path $runtime "tmp"
|
||||
$cacheRoot = Resolve-DDirectory (Join-Path $runtime "cache\perception-e3-models-v1") "EoMT cache"
|
||||
$e3Environment = Resolve-DDirectory (Join-Path $derivedRoot "perception-e3-opencv413092-v1") "E3 environment"
|
||||
$torchEnvironment = Resolve-DDirectory (Join-Path $derivedRoot "perception-p0-env-v1") "Torch environment"
|
||||
$transformersEnvironment = Resolve-DDirectory (Join-Path $derivedRoot "perception-p0-transformers4576-v1") "Transformers environment"
|
||||
$null = New-Item -ItemType Directory -Path $derivedRoot -Force
|
||||
$null = New-Item -ItemType Directory -Path $tmpRoot -Force
|
||||
|
||||
$pixelWorkingSet = [int64]$clipFrameCount * 800 * 600 * 3
|
||||
$partialStreamReserve = [int64][math]::Ceiling(
|
||||
([int64]$job.input.byte_length * ($EndFrame + 1) / $fullFrameCount) * 1.2
|
||||
)
|
||||
$workingSetReserve = [int64][math]::Ceiling($pixelWorkingSet * 1.4) + $partialStreamReserve + 3GB
|
||||
$freeBytesBefore = Assert-FreeSpace "preflight" $workingSetReserve
|
||||
|
||||
& docker image inspect $ContainerImage *> $null
|
||||
Assert-LastExitCode "Existing container image inspection"
|
||||
$tritonState = docker inspect --format "{{.State.Running}}" $TritonContainer
|
||||
Assert-LastExitCode "Triton container inspection"
|
||||
if ($tritonState.Trim().ToLowerInvariant() -ne "true") {
|
||||
throw "LAB E9 requires the existing Triton container"
|
||||
}
|
||||
|
||||
$runnerName = Split-Path $runner -Leaf
|
||||
$profileName = Split-Path $profile -Leaf
|
||||
$detectorProfileName = Split-Path $detectorProfile -Leaf
|
||||
$semanticProfileName = Split-Path $semanticProfile -Leaf
|
||||
$orchestratorSha256 = (Get-FileHash -LiteralPath $PSCommandPath -Algorithm SHA256).Hash.ToLowerInvariant()
|
||||
$commonMounts = @(
|
||||
"-e", "PYTHONPATH=/runner:/opt/transformers:/opt/env",
|
||||
"-e", "TORCH_HOME=/cache/torch",
|
||||
"-e", "HF_HOME=/cache/huggingface",
|
||||
"-e", "HF_HUB_OFFLINE=1",
|
||||
"-e", "TRANSFORMERS_OFFLINE=1",
|
||||
"-v", ((Convert-ToDockerPath $jobDirectory) + ":/job:ro"),
|
||||
"-v", ((Convert-ToDockerPath $validFov) + ":/valid-fov:ro"),
|
||||
"-v", ((Convert-ToDockerPath $model) + ":/model:ro"),
|
||||
"-v", ((Convert-ToDockerPath $cacheRoot) + ":/cache:ro"),
|
||||
"-v", ((Convert-ToDockerPath $e3Environment) + ":/environment:ro"),
|
||||
"-v", ((Convert-ToDockerPath $torchEnvironment) + ":/opt/env:ro"),
|
||||
"-v", ((Convert-ToDockerPath $transformersEnvironment) + ":/opt/transformers:ro"),
|
||||
"-v", ((Convert-ToDockerPath $runnerRoot) + ":/runner:ro")
|
||||
)
|
||||
$commonRunnerArgs = @(
|
||||
"--job", "/job",
|
||||
"--profile", ("/runner/{0}" -f $profileName),
|
||||
"--detector-profile", ("/runner/{0}" -f $detectorProfileName),
|
||||
"--semantic-profile", ("/runner/{0}" -f $semanticProfileName),
|
||||
"--valid-fov-root", "/valid-fov",
|
||||
"--model-root", "/model",
|
||||
"--cache", "/cache",
|
||||
"--environment", "/environment"
|
||||
)
|
||||
$preflightArgs = @(
|
||||
"run", "--rm", "--gpus", "all", "--network", "none", "--read-only",
|
||||
"--security-opt", "no-new-privileges:true", "--cap-drop", "ALL",
|
||||
"--pids-limit", "512", "--shm-size", "4g",
|
||||
"--tmpfs", "/tmp:rw,noexec,nosuid,size=2g"
|
||||
) + $commonMounts + @(
|
||||
"--entrypoint", "python3", $ContainerImage,
|
||||
("/runner/{0}" -f $runnerName), "preflight"
|
||||
) + $commonRunnerArgs
|
||||
Write-Output "PHASE=e9-preflight-start"
|
||||
& docker @preflightArgs
|
||||
Assert-LastExitCode "LAB E9 preflight"
|
||||
Write-Output "PHASE=e9-preflight-complete"
|
||||
|
||||
$modelWasReady = Test-TritonModelReady
|
||||
$loadedByRun = $false
|
||||
if (-not $modelWasReady) {
|
||||
Invoke-WebRequest -UseBasicParsing -TimeoutSec 60 `
|
||||
-Uri "http://127.0.0.1:8000/v2/repository/models/yolox_s/load" `
|
||||
-Method Post -ContentType "application/json" -Body "{}" *> $null
|
||||
$loadedByRun = $true
|
||||
$deadline = [DateTime]::UtcNow.AddSeconds(60)
|
||||
while (-not (Test-TritonModelReady)) {
|
||||
if ([DateTime]::UtcNow -ge $deadline) { throw "YOLOX-S did not become ready" }
|
||||
Start-Sleep -Milliseconds 500
|
||||
}
|
||||
}
|
||||
Write-Output ("PHASE=e9-model-ready LOADED_BY_RUN={0}" -f $loadedByRun)
|
||||
|
||||
$runToken = [Guid]::NewGuid().ToString("N")
|
||||
$workRoot = Join-Path $tmpRoot ("{0}-e9-{1}" -f $job.job_id, $runToken)
|
||||
$framesRoot = Join-Path $workRoot "frames"
|
||||
$streamPath = Join-Path $workRoot "camera.mp4"
|
||||
$ptsPath = Join-Path $workRoot "pts.json"
|
||||
$timelinePath = Join-Path $workRoot "timeline.jsonl"
|
||||
$publishRoot = Join-Path $derivedRoot (".{0}-e9-{1}.publish" -f $job.job_id, $runToken)
|
||||
$stagingRoot = Join-Path $publishRoot "output"
|
||||
$null = New-Item -ItemType Directory -Path $framesRoot
|
||||
$null = New-Item -ItemType Directory -Path $publishRoot
|
||||
$completed = $false
|
||||
$totalWatch = [Diagnostics.Stopwatch]::StartNew()
|
||||
|
||||
try {
|
||||
$stream = [IO.File]::Open($streamPath, [IO.FileMode]::CreateNew, [IO.FileAccess]::Write, [IO.FileShare]::None)
|
||||
try {
|
||||
$input = [IO.File]::OpenRead($initPath)
|
||||
try { $input.CopyTo($stream) } finally { $input.Dispose() }
|
||||
for ($sequence = 1; $sequence -le ($EndFrame + 1); $sequence++) {
|
||||
$path = Resolve-DFile (Join-Path $segmentsRoot ("{0}.m4s" -f $sequence)) "Camera segment"
|
||||
$input = [IO.File]::OpenRead($path)
|
||||
try { $input.CopyTo($stream) } finally { $input.Dispose() }
|
||||
}
|
||||
$stream.Flush($true)
|
||||
}
|
||||
finally { $stream.Dispose() }
|
||||
$null = Assert-FreeSpace "post-stream-reconstruction"
|
||||
|
||||
$extractWatch = [Diagnostics.Stopwatch]::StartNew()
|
||||
$selectFilter = "select='between(n,{0},{1})'" -f $StartFrame, $EndFrame
|
||||
& ffmpeg -hide_banner -loglevel fatal -i $streamPath -map 0:v:0 `
|
||||
-vf $selectFilter -fps_mode passthrough (Join-Path $framesRoot "frame-%06d.png")
|
||||
Assert-LastExitCode "LAB E9 camera clip extraction"
|
||||
& ffprobe -v error -select_streams v:0 -show_entries frame=best_effort_timestamp_time `
|
||||
-of json $streamPath | Set-Content -LiteralPath $ptsPath -Encoding utf8
|
||||
Assert-LastExitCode "LAB E9 camera timestamp probe"
|
||||
|
||||
$decodedFrames = @(Get-ChildItem -LiteralPath $framesRoot -File -Filter "frame-*.png")
|
||||
$pts = @((Get-Content -LiteralPath $ptsPath -Raw | ConvertFrom-Json).frames)
|
||||
if ($decodedFrames.Count -ne $clipFrameCount -or $pts.Count -ne ($EndFrame + 1)) {
|
||||
throw "Decoded LAB E9 frame/timestamp count changed"
|
||||
}
|
||||
$firstEpochSeconds = [double]::Parse(
|
||||
([string]$pts[0].best_effort_timestamp_time).Trim(),
|
||||
[Globalization.CultureInfo]::InvariantCulture
|
||||
)
|
||||
$previousSessionSeconds = -1.0
|
||||
$timelineWriter = [IO.StreamWriter]::new(
|
||||
$timelinePath, $false, [Text.UTF8Encoding]::new($false)
|
||||
)
|
||||
try {
|
||||
for ($localIndex = 0; $localIndex -lt $clipFrameCount; $localIndex++) {
|
||||
$sourceIndex = $StartFrame + $localIndex
|
||||
$epochSeconds = [double]::Parse(
|
||||
([string]$pts[$sourceIndex].best_effort_timestamp_time).Trim(),
|
||||
[Globalization.CultureInfo]::InvariantCulture
|
||||
) - $firstEpochSeconds
|
||||
$sessionSeconds = $timelineStart + $epochSeconds
|
||||
if ($sessionSeconds -le $previousSessionSeconds) { throw "LAB E9 timeline is not monotonic" }
|
||||
$row = [ordered]@{
|
||||
frame_index = $localIndex
|
||||
sequence = $localIndex + 1
|
||||
source_frame_index = $sourceIndex
|
||||
source_sequence = $sourceIndex + 1
|
||||
epoch_seconds = $epochSeconds
|
||||
session_seconds = $sessionSeconds
|
||||
}
|
||||
$timelineWriter.WriteLine(($row | ConvertTo-Json -Compress))
|
||||
$previousSessionSeconds = $sessionSeconds
|
||||
}
|
||||
$timelineWriter.Flush()
|
||||
}
|
||||
finally { $timelineWriter.Dispose() }
|
||||
$extractWatch.Stop()
|
||||
$freeBytesPostExtract = Assert-FreeSpace "post-frame-extraction"
|
||||
Write-Output ("PHASE=e9-frame-extraction-complete FRAMES={0}" -f $clipFrameCount)
|
||||
|
||||
$runArgs = @(
|
||||
"run", "--rm", "--gpus", "all", "--network", ("container:{0}" -f $TritonContainer),
|
||||
"--read-only", "--security-opt", "no-new-privileges:true", "--cap-drop", "ALL",
|
||||
"--pids-limit", "512", "--shm-size", "4g",
|
||||
"--tmpfs", "/tmp:rw,noexec,nosuid,size=2g"
|
||||
) + $commonMounts + @(
|
||||
"-v", ((Convert-ToDockerPath $framesRoot) + ":/frames:ro"),
|
||||
"-v", ((Convert-ToDockerPath $workRoot) + ":/work:ro"),
|
||||
"-v", ((Convert-ToDockerPath $publishRoot) + ":/publish:rw"),
|
||||
"--entrypoint", "python3", $ContainerImage,
|
||||
("/runner/{0}" -f $runnerName), "run"
|
||||
) + $commonRunnerArgs + @(
|
||||
"--frames", "/frames",
|
||||
"--timeline", "/work/timeline.jsonl",
|
||||
"--output", "/publish/output",
|
||||
"--triton-url", "http://127.0.0.1:8000",
|
||||
"--free-bytes-floor", [string]([int64]$FreeGiBFloor * 1GB),
|
||||
"--orchestrator-sha256", $orchestratorSha256,
|
||||
"--container-image", $ContainerImage,
|
||||
"--telemetry-interval-seconds", "1"
|
||||
)
|
||||
Write-Output ("PHASE=e9-multirate-replay-start FRAMES={0}" -f $clipFrameCount)
|
||||
& docker @runArgs
|
||||
Assert-LastExitCode "LAB E9 multirate replay"
|
||||
$freeBytesPostReplay = Assert-FreeSpace "post-replay"
|
||||
|
||||
$result = Get-Content -LiteralPath (Join-Path $stagingRoot "result.json") -Raw | ConvertFrom-Json
|
||||
if (
|
||||
$result.schema_version -ne "missioncore.e9-multirate-perception-result/v1" -or
|
||||
$result.result_id -notmatch "^e9-multirate-perception-[a-f0-9]{64}$" -or
|
||||
$result.acceptance_state -notin @("accepted", "rejected") -or
|
||||
$result.ground_truth -ne $false
|
||||
) { throw "LAB E9 result manifest is incompatible" }
|
||||
$finalRoot = Join-Path $derivedRoot ([string]$result.result_id)
|
||||
if (Test-Path -LiteralPath $finalRoot) { throw "Immutable LAB E9 result already exists" }
|
||||
Move-Item -LiteralPath $stagingRoot -Destination $finalRoot
|
||||
Remove-Item -LiteralPath $publishRoot -Force
|
||||
$completed = $true
|
||||
$totalWatch.Stop()
|
||||
$freeBytesFinal = Assert-FreeSpace "post-publication"
|
||||
Write-Output ("RESULT_ROOT={0}" -f $finalRoot)
|
||||
Write-Output ("RESULT_ID={0}" -f $result.result_id)
|
||||
Write-Output ("ACCEPTANCE_STATE={0}" -f $result.acceptance_state)
|
||||
Write-Output ("PREPARATION_SECONDS={0}" -f $extractWatch.Elapsed.TotalSeconds)
|
||||
Write-Output ("ORCHESTRATOR_WALL_SECONDS={0}" -f $totalWatch.Elapsed.TotalSeconds)
|
||||
Write-Output ("DISK_FREE_BYTES_BEFORE={0}" -f $freeBytesBefore)
|
||||
Write-Output ("DISK_FREE_BYTES_POST_EXTRACT={0}" -f $freeBytesPostExtract)
|
||||
Write-Output ("DISK_FREE_BYTES_POST_REPLAY={0}" -f $freeBytesPostReplay)
|
||||
Write-Output ("DISK_FREE_BYTES_FINAL={0}" -f $freeBytesFinal)
|
||||
}
|
||||
finally {
|
||||
if (Test-Path -LiteralPath $workRoot) { Remove-Item -LiteralPath $workRoot -Recurse -Force }
|
||||
if (-not $completed -and (Test-Path -LiteralPath $publishRoot)) {
|
||||
Remove-Item -LiteralPath $publishRoot -Recurse -Force
|
||||
}
|
||||
if ($loadedByRun) {
|
||||
try {
|
||||
Invoke-WebRequest -UseBasicParsing -TimeoutSec 60 `
|
||||
-Uri "http://127.0.0.1:8000/v2/repository/models/yolox_s/unload" `
|
||||
-Method Post -ContentType "application/json" -Body "{}" *> $null
|
||||
Write-Output "PHASE=e9-model-state-restored"
|
||||
}
|
||||
catch { Write-Warning "LAB E9 could not restore the prior unloaded YOLOX-S state" }
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,133 @@
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[Parameter(Mandatory = $true)]
|
||||
[string]$EvaluationPack,
|
||||
|
||||
[Parameter(Mandatory = $true)]
|
||||
[string]$ValidFovRoot,
|
||||
|
||||
[Parameter(Mandatory = $true)]
|
||||
[string]$RunnerPath,
|
||||
|
||||
[Parameter(Mandatory = $true)]
|
||||
[string]$BaselineRunnerPath,
|
||||
|
||||
[string]$RuntimeRoot = "D:\NDC_MISSIONCORE\runtime"
|
||||
)
|
||||
|
||||
$ErrorActionPreference = "Stop"
|
||||
$ProgressPreference = "SilentlyContinue"
|
||||
|
||||
function Assert-LastExitCode {
|
||||
param([string]$Operation)
|
||||
if ($LASTEXITCODE -ne 0) {
|
||||
throw "$Operation failed with exit code $LASTEXITCODE"
|
||||
}
|
||||
}
|
||||
|
||||
function Assert-RegularFile {
|
||||
param([string]$Path, [string]$Label)
|
||||
$item = Get-Item -LiteralPath $Path -Force
|
||||
if (-not $item.PSIsContainer -and -not ($item.Attributes -band [IO.FileAttributes]::ReparsePoint)) {
|
||||
return $item.FullName
|
||||
}
|
||||
throw "$Label is not a regular file"
|
||||
}
|
||||
|
||||
function Assert-Directory {
|
||||
param([string]$Path, [string]$Label)
|
||||
$item = Get-Item -LiteralPath $Path -Force
|
||||
if ($item.PSIsContainer -and -not ($item.Attributes -band [IO.FileAttributes]::ReparsePoint)) {
|
||||
return $item.FullName
|
||||
}
|
||||
throw "$Label is not a real directory"
|
||||
}
|
||||
|
||||
function Convert-ToDockerPath {
|
||||
param([string]$Path)
|
||||
return $Path.Replace("\", "/")
|
||||
}
|
||||
|
||||
$packRoot = Assert-Directory (Resolve-Path -LiteralPath $EvaluationPack).Path "Evaluation pack"
|
||||
$validFov = Assert-Directory (Resolve-Path -LiteralPath $ValidFovRoot).Path "Valid-FOV root"
|
||||
$runner = Assert-RegularFile (Resolve-Path -LiteralPath $RunnerPath).Path "Prelabel runner"
|
||||
$baselineRunner = Assert-RegularFile (Resolve-Path -LiteralPath $BaselineRunnerPath).Path "Baseline runner"
|
||||
if ((Split-Path $runner -Parent) -ne (Split-Path $baselineRunner -Parent)) {
|
||||
throw "Prelabel and baseline runners must share one read-only mount"
|
||||
}
|
||||
$runtime = Assert-Directory (Resolve-Path -LiteralPath $RuntimeRoot).Path "Runtime root"
|
||||
$pack = Get-Content -LiteralPath (Join-Path $packRoot "manifest.json") -Raw | ConvertFrom-Json
|
||||
if (
|
||||
$pack.schema_version -ne "missioncore.perception-evaluation-pack/v1" -or
|
||||
$pack.generation_id -ne (Split-Path $packRoot -Leaf) -or
|
||||
$pack.identity.preprocessing_profile -ne "fixed-valid-fov-fill/v1"
|
||||
) {
|
||||
throw "Evaluation pack manifest is incompatible"
|
||||
}
|
||||
$null = Assert-RegularFile (Join-Path $validFov "manifest.json") "Valid-FOV manifest"
|
||||
$null = Assert-RegularFile (Join-Path $validFov "mask.png") "Valid-FOV mask"
|
||||
|
||||
$cacheRoot = Assert-Directory (Join-Path $runtime "cache\perception-p0-models-v1") "Model cache"
|
||||
$torchEnvironment = Assert-Directory (Join-Path $runtime "derived\perception-p0-env-v1") "Torch environment"
|
||||
$transformersEnvironment = Assert-Directory (Join-Path $runtime "derived\perception-p0-transformers4576-v1") "Transformers environment"
|
||||
$runnerRoot = Split-Path $runner -Parent
|
||||
$runnerName = Split-Path $runner -Leaf
|
||||
$derivedRoot = Join-Path $runtime "derived\evaluation-prelabels"
|
||||
$null = New-Item -ItemType Directory -Path $derivedRoot -Force
|
||||
$runToken = [Guid]::NewGuid().ToString("N")
|
||||
$publishRoot = Join-Path $derivedRoot (".{0}-{1}.publish" -f $pack.generation_id, $runToken)
|
||||
$stagingRoot = Join-Path $publishRoot "output"
|
||||
$null = New-Item -ItemType Directory -Path $publishRoot
|
||||
|
||||
try {
|
||||
$dockerArgs = @(
|
||||
"run", "--rm", "--gpus", "all", "--network", "none",
|
||||
"--security-opt", "no-new-privileges:true", "--cap-drop", "ALL",
|
||||
"--shm-size", "2g",
|
||||
"-e", "PYTHONPATH=/opt/transformers:/opt/env",
|
||||
"-e", "TORCH_HOME=/cache/torch",
|
||||
"-e", "HF_HOME=/cache/huggingface",
|
||||
"-e", "HF_HUB_OFFLINE=1",
|
||||
"-e", "TRANSFORMERS_OFFLINE=1",
|
||||
"-v", ((Convert-ToDockerPath $packRoot) + ":/evaluation-pack:ro"),
|
||||
"-v", ((Convert-ToDockerPath $validFov) + ":/valid-fov:ro"),
|
||||
"-v", ((Convert-ToDockerPath $publishRoot) + ":/publish:rw"),
|
||||
"-v", ((Convert-ToDockerPath $cacheRoot) + ":/cache:rw"),
|
||||
"-v", ((Convert-ToDockerPath $torchEnvironment) + ":/opt/env:ro"),
|
||||
"-v", ((Convert-ToDockerPath $transformersEnvironment) + ":/opt/transformers:ro"),
|
||||
"-v", ((Convert-ToDockerPath $runnerRoot) + ":/runner:ro"),
|
||||
"--entrypoint", "python3",
|
||||
"nvcr.io/nvidia/tritonserver:26.06-py3",
|
||||
("/runner/{0}" -f $runnerName),
|
||||
"--evaluation-pack", "/evaluation-pack",
|
||||
"--valid-fov-root", "/valid-fov",
|
||||
"--output", "/publish/output",
|
||||
"--cache", "/cache",
|
||||
"--telemetry-interval-seconds", "1"
|
||||
)
|
||||
Write-Output ("PHASE=e2-prelabels-start PACK={0}" -f $pack.generation_id)
|
||||
& docker @dockerArgs
|
||||
Assert-LastExitCode "E2 prelabel generation"
|
||||
|
||||
$result = Get-Content -LiteralPath (Join-Path $stagingRoot "result.json") -Raw | ConvertFrom-Json
|
||||
if (
|
||||
$result.schema_version -ne "missioncore.perception-evaluation-prelabels/v1" -or
|
||||
$result.result_id -notmatch "^evaluation-prelabels-[a-f0-9]{64}$" -or
|
||||
$result.identity.evaluation_pack_id -ne $pack.generation_id
|
||||
) {
|
||||
throw "E2 prelabel result manifest is incompatible"
|
||||
}
|
||||
$finalRoot = Join-Path $derivedRoot ([string]$result.result_id)
|
||||
if (Test-Path -LiteralPath $finalRoot) {
|
||||
throw "An immutable E2 prelabel result with the same identity already exists: $finalRoot"
|
||||
}
|
||||
Move-Item -LiteralPath $stagingRoot -Destination $finalRoot
|
||||
Remove-Item -LiteralPath $publishRoot -Force
|
||||
Write-Output ("RESULT_ROOT={0}" -f $finalRoot)
|
||||
Write-Output ("RESULT_ID={0}" -f $result.result_id)
|
||||
}
|
||||
finally {
|
||||
if (Test-Path -LiteralPath $publishRoot) {
|
||||
Remove-Item -LiteralPath $publishRoot -Recurse -Force
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,276 @@
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[Parameter(Mandatory = $true)]
|
||||
[string]$JobRoot,
|
||||
|
||||
[Parameter(Mandatory = $true)]
|
||||
[string]$RunnerPath,
|
||||
|
||||
[Parameter(Mandatory = $true)]
|
||||
[string]$BaselineRunnerPath,
|
||||
|
||||
[Parameter(Mandatory = $true)]
|
||||
[string]$QualificationManifest,
|
||||
|
||||
[Parameter(Mandatory = $true)]
|
||||
[string]$ValidFovRoot,
|
||||
|
||||
[string]$RuntimeRoot = "D:\NDC_MISSIONCORE\runtime",
|
||||
|
||||
[string]$CalibrationSha256 = "05f3ad9b38b3a4fc95388a8ec83da83c745e217709e51787b3d5aad0969f6fa9",
|
||||
|
||||
[string]$CalibrationSlot = "camera_1"
|
||||
)
|
||||
|
||||
$ErrorActionPreference = "Stop"
|
||||
$ProgressPreference = "SilentlyContinue"
|
||||
|
||||
function Assert-LastExitCode {
|
||||
param([string]$Operation)
|
||||
if ($LASTEXITCODE -ne 0) {
|
||||
throw "$Operation failed with exit code $LASTEXITCODE"
|
||||
}
|
||||
}
|
||||
|
||||
function Assert-RegularFile {
|
||||
param([string]$Path, [string]$Label)
|
||||
$item = Get-Item -LiteralPath $Path -Force
|
||||
if (-not $item.PSIsContainer -and -not ($item.Attributes -band [IO.FileAttributes]::ReparsePoint)) {
|
||||
return $item.FullName
|
||||
}
|
||||
throw "$Label is not a regular file"
|
||||
}
|
||||
|
||||
function Assert-Directory {
|
||||
param([string]$Path, [string]$Label)
|
||||
$item = Get-Item -LiteralPath $Path -Force
|
||||
if ($item.PSIsContainer -and -not ($item.Attributes -band [IO.FileAttributes]::ReparsePoint)) {
|
||||
return $item.FullName
|
||||
}
|
||||
throw "$Label is not a real directory"
|
||||
}
|
||||
|
||||
function Convert-ToDockerPath {
|
||||
param([string]$Path)
|
||||
return $Path.Replace("\", "/")
|
||||
}
|
||||
|
||||
$jobDirectory = Assert-Directory (Resolve-Path -LiteralPath $JobRoot).Path "Job root"
|
||||
$runner = Assert-RegularFile (Resolve-Path -LiteralPath $RunnerPath).Path "Qualification runner"
|
||||
$baselineRunner = Assert-RegularFile (Resolve-Path -LiteralPath $BaselineRunnerPath).Path "Baseline runner"
|
||||
$qualification = Assert-RegularFile (Resolve-Path -LiteralPath $QualificationManifest).Path "Qualification manifest"
|
||||
$validFov = Assert-Directory (Resolve-Path -LiteralPath $ValidFovRoot).Path "Valid-FOV root"
|
||||
$null = Assert-RegularFile (Join-Path $validFov "manifest.json") "Valid-FOV manifest"
|
||||
$null = Assert-RegularFile (Join-Path $validFov "mask.png") "Valid-FOV mask"
|
||||
if ((Split-Path $runner -Parent) -ne (Split-Path $baselineRunner -Parent)) {
|
||||
throw "Qualification and baseline runners must share one read-only mount"
|
||||
}
|
||||
$runtime = Assert-Directory (Resolve-Path -LiteralPath $RuntimeRoot).Path "Runtime root"
|
||||
$job = Get-Content -LiteralPath (Join-Path $jobDirectory "job.json") -Raw | ConvertFrom-Json
|
||||
if ($job.schema_version -ne "missioncore.compute-job/v1" -or $job.job_id -ne (Split-Path $jobDirectory -Leaf)) {
|
||||
throw "Compute job manifest is incompatible"
|
||||
}
|
||||
if ($CalibrationSha256 -notmatch "^[a-f0-9]{64}$" -or $CalibrationSlot -notmatch "^[A-Za-z0-9._-]+$") {
|
||||
throw "Calibration binding is invalid"
|
||||
}
|
||||
|
||||
$sourceId = [string]$job.input.source_id
|
||||
$epoch = [int]$job.input.codec_epoch
|
||||
$frameCount = [int]$job.input.segment_count
|
||||
$timelineStart = [double]$job.input.timeline.start_seconds
|
||||
$timelineEnd = [double]$job.input.timeline.end_seconds
|
||||
$timelineDuration = $timelineEnd - $timelineStart
|
||||
if ($frameCount -lt 1 -or $timelineDuration -le 0) {
|
||||
throw "Compute job frame/timeline contract is invalid"
|
||||
}
|
||||
$qualificationDocument = Get-Content -LiteralPath $qualification -Raw | ConvertFrom-Json
|
||||
if (
|
||||
$qualificationDocument.schema_version -ne "missioncore.recorded-qualification-slice/v1" -or
|
||||
$qualificationDocument.identity.job_id -ne $job.job_id -or
|
||||
$qualificationDocument.identity.input_sha256 -ne $job.input_sha256
|
||||
) {
|
||||
throw "Qualification manifest is not bound to this job"
|
||||
}
|
||||
Write-Output ("PHASE=inputs-validated QUALIFICATION_FRAMES={0}" -f @($qualificationDocument.frames).Count)
|
||||
|
||||
$epochRoot = Join-Path $jobDirectory ("input\camera\{0}\epoch-{1}" -f $sourceId, $epoch)
|
||||
$epochRoot = Assert-Directory $epochRoot "Camera epoch"
|
||||
$initPath = Assert-RegularFile (Join-Path $epochRoot "init.mp4") "Camera init"
|
||||
$segmentsRoot = Assert-Directory (Join-Path $epochRoot "segments") "Camera segments"
|
||||
$derivedRoot = Join-Path $runtime "derived\qualification"
|
||||
$tmpRoot = Join-Path $runtime "tmp"
|
||||
$cacheRoot = Assert-Directory (Join-Path $runtime "cache\perception-p0-models-v1") "Model cache"
|
||||
$torchEnvironment = Assert-Directory (Join-Path $runtime "derived\perception-p0-env-v1") "Torch environment"
|
||||
$transformersEnvironment = Assert-Directory (Join-Path $runtime "derived\perception-p0-transformers4576-v1") "Transformers environment"
|
||||
$runnerRoot = Split-Path $runner -Parent
|
||||
$runnerName = Split-Path $runner -Leaf
|
||||
$baselineRunnerName = Split-Path $baselineRunner -Leaf
|
||||
$qualificationRoot = Split-Path $qualification -Parent
|
||||
$qualificationName = Split-Path $qualification -Leaf
|
||||
$null = New-Item -ItemType Directory -Path $derivedRoot -Force
|
||||
$null = New-Item -ItemType Directory -Path $tmpRoot -Force
|
||||
|
||||
$preflightArgs = @(
|
||||
"run", "--rm", "--gpus", "all", "--network", "none",
|
||||
"--security-opt", "no-new-privileges:true", "--cap-drop", "ALL",
|
||||
"-e", "PYTHONPATH=/opt/transformers:/opt/env",
|
||||
"-e", "TORCH_HOME=/cache/torch",
|
||||
"-e", "HF_HOME=/cache/huggingface",
|
||||
"-e", "HF_HUB_OFFLINE=1",
|
||||
"-e", "TRANSFORMERS_OFFLINE=1",
|
||||
"-v", ((Convert-ToDockerPath $jobDirectory) + ":/job:ro"),
|
||||
"-v", ((Convert-ToDockerPath $cacheRoot) + ":/cache:rw"),
|
||||
"-v", ((Convert-ToDockerPath $torchEnvironment) + ":/opt/env:ro"),
|
||||
"-v", ((Convert-ToDockerPath $transformersEnvironment) + ":/opt/transformers:ro"),
|
||||
"-v", ((Convert-ToDockerPath $runnerRoot) + ":/runner:ro"),
|
||||
"--entrypoint", "python3",
|
||||
"nvcr.io/nvidia/tritonserver:26.06-py3",
|
||||
("/runner/{0}" -f $baselineRunnerName), "preflight",
|
||||
"--job", "/job",
|
||||
"--cache", "/cache"
|
||||
)
|
||||
Write-Output "PHASE=worker-preflight-started"
|
||||
& docker @preflightArgs
|
||||
Assert-LastExitCode "Worker preflight"
|
||||
Write-Output "PHASE=worker-preflight-complete"
|
||||
|
||||
$runToken = [Guid]::NewGuid().ToString("N")
|
||||
$workRoot = Join-Path $tmpRoot ("{0}-e1-{1}" -f $job.job_id, $runToken)
|
||||
$framesRoot = Join-Path $workRoot "frames"
|
||||
$streamPath = Join-Path $workRoot "camera.mp4"
|
||||
$ptsPath = Join-Path $workRoot "pts.json"
|
||||
$timelinePath = Join-Path $workRoot "timeline.jsonl"
|
||||
$publishRoot = Join-Path $derivedRoot (".{0}-e1-{1}.publish" -f $job.job_id, $runToken)
|
||||
$stagingRoot = Join-Path $publishRoot "output"
|
||||
$null = New-Item -ItemType Directory -Path $framesRoot
|
||||
$null = New-Item -ItemType Directory -Path $publishRoot
|
||||
Write-Output "PHASE=private-staging-created"
|
||||
|
||||
try {
|
||||
$stream = [IO.File]::Open($streamPath, [IO.FileMode]::CreateNew, [IO.FileAccess]::Write, [IO.FileShare]::None)
|
||||
try {
|
||||
$input = [IO.File]::OpenRead($initPath)
|
||||
try {
|
||||
$input.CopyTo($stream)
|
||||
}
|
||||
finally {
|
||||
$input.Dispose()
|
||||
}
|
||||
for ($sequence = 1; $sequence -le $frameCount; $sequence++) {
|
||||
$path = Assert-RegularFile (Join-Path $segmentsRoot ("{0}.m4s" -f $sequence)) "Camera segment"
|
||||
$input = [IO.File]::OpenRead($path)
|
||||
try {
|
||||
$input.CopyTo($stream)
|
||||
}
|
||||
finally {
|
||||
$input.Dispose()
|
||||
}
|
||||
if ($sequence % 500 -eq 0 -or $sequence -eq $frameCount) {
|
||||
Write-Output ("PHASE=stream-reconstruction SEGMENTS={0}/{1}" -f $sequence, $frameCount)
|
||||
}
|
||||
}
|
||||
$stream.Flush($true)
|
||||
}
|
||||
finally {
|
||||
$stream.Dispose()
|
||||
}
|
||||
|
||||
Write-Output "PHASE=frame-extraction-started"
|
||||
& ffmpeg -hide_banner -loglevel fatal -i $streamPath -map 0:v:0 -fps_mode passthrough (Join-Path $framesRoot "frame-%06d.png")
|
||||
Assert-LastExitCode "Camera extraction"
|
||||
& ffprobe -v error -select_streams v:0 -show_entries frame=best_effort_timestamp_time -of json $streamPath | Set-Content -LiteralPath $ptsPath -Encoding utf8
|
||||
Assert-LastExitCode "Camera timestamp probe"
|
||||
|
||||
$decodedFrames = @(Get-ChildItem -LiteralPath $framesRoot -File -Filter "frame-*.png")
|
||||
$ptsDocument = Get-Content -LiteralPath $ptsPath -Raw | ConvertFrom-Json
|
||||
$pts = @($ptsDocument.frames)
|
||||
if ($decodedFrames.Count -ne $frameCount -or $pts.Count -ne $frameCount) {
|
||||
throw "Decoded frame count differs from the compute job"
|
||||
}
|
||||
$firstEpochSeconds = [double]::Parse(
|
||||
([string]$pts[0].best_effort_timestamp_time).Trim(),
|
||||
[Globalization.CultureInfo]::InvariantCulture
|
||||
)
|
||||
$previousEpochSeconds = -1.0
|
||||
$timelineWriter = [IO.StreamWriter]::new($timelinePath, $false, [Text.UTF8Encoding]::new($false))
|
||||
try {
|
||||
for ($index = 0; $index -lt $frameCount; $index++) {
|
||||
$epochSeconds = [double]::Parse(
|
||||
([string]$pts[$index].best_effort_timestamp_time).Trim(),
|
||||
[Globalization.CultureInfo]::InvariantCulture
|
||||
) - $firstEpochSeconds
|
||||
if ($epochSeconds -le $previousEpochSeconds -or $epochSeconds -gt ($timelineDuration + 0.001)) {
|
||||
throw "Decoded camera timestamps are not strictly monotonic inside the compute job timeline"
|
||||
}
|
||||
$row = [ordered]@{
|
||||
frame_index = $index
|
||||
epoch_seconds = $epochSeconds
|
||||
session_seconds = $timelineStart + $epochSeconds
|
||||
}
|
||||
$timelineWriter.WriteLine(($row | ConvertTo-Json -Compress))
|
||||
$previousEpochSeconds = $epochSeconds
|
||||
}
|
||||
$timelineWriter.Flush()
|
||||
}
|
||||
finally {
|
||||
$timelineWriter.Dispose()
|
||||
}
|
||||
Write-Output ("PHASE=frame-extraction-complete FRAMES={0}" -f $frameCount)
|
||||
|
||||
$dockerArgs = @(
|
||||
"run", "--rm", "--gpus", "all", "--network", "none",
|
||||
"--security-opt", "no-new-privileges:true", "--cap-drop", "ALL",
|
||||
"--shm-size", "2g",
|
||||
"-e", "PYTHONPATH=/opt/transformers:/opt/env",
|
||||
"-e", "TORCH_HOME=/cache/torch",
|
||||
"-e", "HF_HOME=/cache/huggingface",
|
||||
"-e", "HF_HUB_OFFLINE=1",
|
||||
"-e", "TRANSFORMERS_OFFLINE=1",
|
||||
"-v", ((Convert-ToDockerPath $jobDirectory) + ":/job:ro"),
|
||||
"-v", ((Convert-ToDockerPath $framesRoot) + ":/frames:ro"),
|
||||
"-v", ((Convert-ToDockerPath $workRoot) + ":/work:ro"),
|
||||
"-v", ((Convert-ToDockerPath $publishRoot) + ":/publish:rw"),
|
||||
"-v", ((Convert-ToDockerPath $cacheRoot) + ":/cache:rw"),
|
||||
"-v", ((Convert-ToDockerPath $torchEnvironment) + ":/opt/env:ro"),
|
||||
"-v", ((Convert-ToDockerPath $transformersEnvironment) + ":/opt/transformers:ro"),
|
||||
"-v", ((Convert-ToDockerPath $runnerRoot) + ":/runner:ro"),
|
||||
"-v", ((Convert-ToDockerPath $qualificationRoot) + ":/qualification:ro"),
|
||||
"-v", ((Convert-ToDockerPath $validFov) + ":/valid-fov:ro"),
|
||||
"--entrypoint", "python3",
|
||||
"nvcr.io/nvidia/tritonserver:26.06-py3",
|
||||
("/runner/{0}" -f $runnerName),
|
||||
"--job", "/job",
|
||||
"--frames", "/frames",
|
||||
"--timeline", "/work/timeline.jsonl",
|
||||
"--qualification", ("/qualification/{0}" -f $qualificationName),
|
||||
"--valid-fov-root", "/valid-fov",
|
||||
"--output", "/publish/output",
|
||||
"--cache", "/cache",
|
||||
"--calibration-sha256", $CalibrationSha256,
|
||||
"--calibration-slot", $CalibrationSlot,
|
||||
"--telemetry-interval-seconds", "1"
|
||||
)
|
||||
& docker @dockerArgs
|
||||
Assert-LastExitCode "Preprocessing qualification"
|
||||
|
||||
$result = Get-Content -LiteralPath (Join-Path $stagingRoot "result.json") -Raw | ConvertFrom-Json
|
||||
if (
|
||||
$result.schema_version -ne "missioncore.perception-preprocessing-qualification-result/v1" -or
|
||||
$result.result_id -notmatch "^qualification-result-[a-f0-9]{64}$"
|
||||
) {
|
||||
throw "Qualification result manifest is incompatible"
|
||||
}
|
||||
$finalRoot = Join-Path $derivedRoot ([string]$result.result_id)
|
||||
if (Test-Path -LiteralPath $finalRoot) {
|
||||
throw "An immutable qualification result with the same identity already exists: $finalRoot"
|
||||
}
|
||||
Move-Item -LiteralPath $stagingRoot -Destination $finalRoot
|
||||
Remove-Item -LiteralPath $publishRoot -Force
|
||||
Write-Output ("RESULT_ROOT={0}" -f $finalRoot)
|
||||
Write-Output ("RESULT_ID={0}" -f $result.result_id)
|
||||
}
|
||||
finally {
|
||||
if (Test-Path -LiteralPath $workRoot) {
|
||||
Remove-Item -LiteralPath $workRoot -Recurse -Force
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,292 @@
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[Parameter(Mandatory = $true)]
|
||||
[string]$JobRoot,
|
||||
|
||||
[Parameter(Mandatory = $true)]
|
||||
[string]$RunnerPath,
|
||||
|
||||
[string]$RuntimeRoot = "D:\NDC_MISSIONCORE\runtime",
|
||||
|
||||
[string]$CalibrationSha256 = "05f3ad9b38b3a4fc95388a8ec83da83c745e217709e51787b3d5aad0969f6fa9",
|
||||
|
||||
[string]$CalibrationSlot = "camera_1"
|
||||
)
|
||||
|
||||
$ErrorActionPreference = "Stop"
|
||||
$ProgressPreference = "SilentlyContinue"
|
||||
|
||||
function Assert-LastExitCode {
|
||||
param([string]$Operation)
|
||||
if ($LASTEXITCODE -ne 0) {
|
||||
throw "$Operation failed with exit code $LASTEXITCODE"
|
||||
}
|
||||
}
|
||||
|
||||
function Assert-RegularFile {
|
||||
param([string]$Path, [string]$Label)
|
||||
$item = Get-Item -LiteralPath $Path -Force
|
||||
if (-not $item.PSIsContainer -and -not ($item.Attributes -band [IO.FileAttributes]::ReparsePoint)) {
|
||||
return $item.FullName
|
||||
}
|
||||
throw "$Label is not a regular file"
|
||||
}
|
||||
|
||||
function Assert-Directory {
|
||||
param([string]$Path, [string]$Label)
|
||||
$item = Get-Item -LiteralPath $Path -Force
|
||||
if ($item.PSIsContainer -and -not ($item.Attributes -band [IO.FileAttributes]::ReparsePoint)) {
|
||||
return $item.FullName
|
||||
}
|
||||
throw "$Label is not a real directory"
|
||||
}
|
||||
|
||||
function Convert-ToDockerPath {
|
||||
param([string]$Path)
|
||||
return $Path.Replace("\", "/")
|
||||
}
|
||||
|
||||
$jobDirectory = Assert-Directory (Resolve-Path -LiteralPath $JobRoot).Path "Job root"
|
||||
$runner = Assert-RegularFile (Resolve-Path -LiteralPath $RunnerPath).Path "Runner"
|
||||
$runtime = Assert-Directory (Resolve-Path -LiteralPath $RuntimeRoot).Path "Runtime root"
|
||||
$job = Get-Content -LiteralPath (Join-Path $jobDirectory "job.json") -Raw | ConvertFrom-Json
|
||||
if ($job.schema_version -ne "missioncore.compute-job/v1" -or $job.job_id -ne (Split-Path $jobDirectory -Leaf)) {
|
||||
throw "Compute job manifest is incompatible"
|
||||
}
|
||||
if ($CalibrationSha256 -notmatch "^[a-f0-9]{64}$" -or $CalibrationSlot -notmatch "^[A-Za-z0-9._-]+$") {
|
||||
throw "Calibration binding is invalid"
|
||||
}
|
||||
Write-Output "PHASE=job-manifest-validated"
|
||||
|
||||
$sourceId = [string]$job.input.source_id
|
||||
$epoch = [int]$job.input.codec_epoch
|
||||
$frameCount = [int]$job.input.segment_count
|
||||
$timelineStart = [double]$job.input.timeline.start_seconds
|
||||
$timelineEnd = [double]$job.input.timeline.end_seconds
|
||||
$timelineDuration = $timelineEnd - $timelineStart
|
||||
if ($frameCount -lt 1 -or $timelineDuration -le 0) {
|
||||
throw "Compute job frame/timeline contract is invalid"
|
||||
}
|
||||
|
||||
$epochRoot = Join-Path $jobDirectory ("input\camera\{0}\epoch-{1}" -f $sourceId, $epoch)
|
||||
$epochRoot = Assert-Directory $epochRoot "Camera epoch"
|
||||
$initPath = Assert-RegularFile (Join-Path $epochRoot "init.mp4") "Camera init"
|
||||
$segmentsRoot = Assert-Directory (Join-Path $epochRoot "segments") "Camera segments"
|
||||
$derivedRoot = Join-Path $runtime "derived"
|
||||
$tmpRoot = Join-Path $runtime "tmp"
|
||||
$cacheRoot = Assert-Directory (Join-Path $runtime "cache\perception-p0-models-v1") "Model cache"
|
||||
$torchEnvironment = Assert-Directory (Join-Path $derivedRoot "perception-p0-env-v1") "Torch environment"
|
||||
$transformersEnvironment = Assert-Directory (Join-Path $derivedRoot "perception-p0-transformers4576-v1") "Transformers environment"
|
||||
$runnerRoot = Split-Path $runner -Parent
|
||||
$runnerName = Split-Path $runner -Leaf
|
||||
$null = New-Item -ItemType Directory -Path $derivedRoot -Force
|
||||
$null = New-Item -ItemType Directory -Path $tmpRoot -Force
|
||||
|
||||
$preflightArgs = @(
|
||||
"run", "--rm", "--gpus", "all", "--network", "none",
|
||||
"--security-opt", "no-new-privileges:true", "--cap-drop", "ALL",
|
||||
"-e", "PYTHONPATH=/opt/transformers:/opt/env",
|
||||
"-e", "TORCH_HOME=/cache/torch",
|
||||
"-e", "HF_HOME=/cache/huggingface",
|
||||
"-e", "HF_HUB_OFFLINE=1",
|
||||
"-e", "TRANSFORMERS_OFFLINE=1",
|
||||
"-v", ((Convert-ToDockerPath $jobDirectory) + ":/job:ro"),
|
||||
"-v", ((Convert-ToDockerPath $cacheRoot) + ":/cache:rw"),
|
||||
"-v", ((Convert-ToDockerPath $torchEnvironment) + ":/opt/env:ro"),
|
||||
"-v", ((Convert-ToDockerPath $transformersEnvironment) + ":/opt/transformers:ro"),
|
||||
"-v", ((Convert-ToDockerPath $runnerRoot) + ":/runner:ro"),
|
||||
"--entrypoint", "python3",
|
||||
"nvcr.io/nvidia/tritonserver:26.06-py3",
|
||||
("/runner/{0}" -f $runnerName), "preflight",
|
||||
"--job", "/job",
|
||||
"--cache", "/cache"
|
||||
)
|
||||
Write-Output "PHASE=worker-preflight-started"
|
||||
& docker @preflightArgs
|
||||
Assert-LastExitCode "Worker preflight"
|
||||
Write-Output "PHASE=worker-preflight-complete"
|
||||
|
||||
$runToken = [Guid]::NewGuid().ToString("N")
|
||||
$workRoot = Join-Path $tmpRoot ("{0}-panoptic-{1}" -f $job.job_id, $runToken)
|
||||
$framesRoot = Join-Path $workRoot "frames"
|
||||
$streamPath = Join-Path $workRoot "camera.mp4"
|
||||
$ptsPath = Join-Path $workRoot "pts.json"
|
||||
$timelinePath = Join-Path $workRoot "timeline.jsonl"
|
||||
$publishRoot = Join-Path $derivedRoot (".{0}-panoptic-{1}.publish" -f $job.job_id, $runToken)
|
||||
$stagingRoot = Join-Path $publishRoot "output"
|
||||
$null = New-Item -ItemType Directory -Path $framesRoot
|
||||
$null = New-Item -ItemType Directory -Path $publishRoot
|
||||
$totalWatch = [Diagnostics.Stopwatch]::StartNew()
|
||||
Write-Output "PHASE=private-staging-created"
|
||||
|
||||
try {
|
||||
$stream = [IO.File]::Open($streamPath, [IO.FileMode]::CreateNew, [IO.FileAccess]::Write, [IO.FileShare]::None)
|
||||
try {
|
||||
foreach ($path in @($initPath)) {
|
||||
$input = [IO.File]::OpenRead($path)
|
||||
try {
|
||||
$input.CopyTo($stream)
|
||||
}
|
||||
finally {
|
||||
$input.Dispose()
|
||||
}
|
||||
}
|
||||
for ($sequence = 1; $sequence -le $frameCount; $sequence++) {
|
||||
$path = Assert-RegularFile (Join-Path $segmentsRoot ("{0}.m4s" -f $sequence)) "Camera segment"
|
||||
$input = [IO.File]::OpenRead($path)
|
||||
try {
|
||||
$input.CopyTo($stream)
|
||||
}
|
||||
finally {
|
||||
$input.Dispose()
|
||||
}
|
||||
if ($sequence % 500 -eq 0 -or $sequence -eq $frameCount) {
|
||||
Write-Output ("PHASE=stream-reconstruction SEGMENTS={0}/{1}" -f $sequence, $frameCount)
|
||||
}
|
||||
}
|
||||
$stream.Flush($true)
|
||||
}
|
||||
finally {
|
||||
$stream.Dispose()
|
||||
}
|
||||
|
||||
$extractWatch = [Diagnostics.Stopwatch]::StartNew()
|
||||
Write-Output "PHASE=frame-extraction-started"
|
||||
# This archive contains valid strictly-increasing PTS but image2 rounds DTS
|
||||
# to its own time base and reports harmless duplicate-DTS diagnostics. Exact
|
||||
# frame-count and JSON timestamp checks below are the admission boundary.
|
||||
& ffmpeg -hide_banner -loglevel fatal -i $streamPath -map 0:v:0 -fps_mode passthrough (Join-Path $framesRoot "frame-%06d.png")
|
||||
Assert-LastExitCode "Camera extraction"
|
||||
& ffprobe -v error -select_streams v:0 -show_entries frame=best_effort_timestamp_time -of json $streamPath | Set-Content -LiteralPath $ptsPath -Encoding utf8
|
||||
Assert-LastExitCode "Camera timestamp probe"
|
||||
|
||||
$decodedFrames = @(Get-ChildItem -LiteralPath $framesRoot -File -Filter "frame-*.png")
|
||||
$ptsDocument = Get-Content -LiteralPath $ptsPath -Raw | ConvertFrom-Json
|
||||
$pts = @($ptsDocument.frames)
|
||||
if ($decodedFrames.Count -ne $frameCount -or $pts.Count -ne $frameCount) {
|
||||
throw "Decoded frame count differs from the compute job"
|
||||
}
|
||||
$firstEpochSeconds = [double]::Parse(
|
||||
([string]$pts[0].best_effort_timestamp_time).Trim(),
|
||||
[Globalization.CultureInfo]::InvariantCulture
|
||||
)
|
||||
$previousEpochSeconds = -1.0
|
||||
$timelineWriter = [IO.StreamWriter]::new($timelinePath, $false, [Text.UTF8Encoding]::new($false))
|
||||
try {
|
||||
for ($index = 0; $index -lt $frameCount; $index++) {
|
||||
$epochSeconds = [double]::Parse(
|
||||
([string]$pts[$index].best_effort_timestamp_time).Trim(),
|
||||
[Globalization.CultureInfo]::InvariantCulture
|
||||
) - $firstEpochSeconds
|
||||
if ($epochSeconds -le $previousEpochSeconds -or $epochSeconds -gt ($timelineDuration + 0.001)) {
|
||||
throw "Decoded camera timestamps are not strictly monotonic inside the compute job timeline"
|
||||
}
|
||||
$sessionSeconds = $timelineStart + $epochSeconds
|
||||
$row = [ordered]@{
|
||||
frame_index = $index
|
||||
epoch_seconds = $epochSeconds
|
||||
session_seconds = $sessionSeconds
|
||||
}
|
||||
$timelineWriter.WriteLine(($row | ConvertTo-Json -Compress))
|
||||
$previousEpochSeconds = $epochSeconds
|
||||
}
|
||||
$timelineWriter.Flush()
|
||||
}
|
||||
finally {
|
||||
$timelineWriter.Dispose()
|
||||
}
|
||||
$extractWatch.Stop()
|
||||
Write-Output ("PHASE=frame-extraction-complete FRAMES={0}" -f $frameCount)
|
||||
|
||||
$dockerArgs = @(
|
||||
"run", "--rm", "--gpus", "all", "--network", "none",
|
||||
"--security-opt", "no-new-privileges:true", "--cap-drop", "ALL",
|
||||
"--shm-size", "2g",
|
||||
"-e", "PYTHONPATH=/opt/transformers:/opt/env",
|
||||
"-e", "TORCH_HOME=/cache/torch",
|
||||
"-e", "HF_HOME=/cache/huggingface",
|
||||
"-e", "HF_HUB_OFFLINE=1",
|
||||
"-e", "TRANSFORMERS_OFFLINE=1",
|
||||
"-v", ((Convert-ToDockerPath $jobDirectory) + ":/job:ro"),
|
||||
"-v", ((Convert-ToDockerPath $framesRoot) + ":/frames:ro"),
|
||||
"-v", ((Convert-ToDockerPath $workRoot) + ":/work:ro"),
|
||||
"-v", ((Convert-ToDockerPath $publishRoot) + ":/publish:rw"),
|
||||
"-v", ((Convert-ToDockerPath $cacheRoot) + ":/cache:rw"),
|
||||
"-v", ((Convert-ToDockerPath $torchEnvironment) + ":/opt/env:ro"),
|
||||
"-v", ((Convert-ToDockerPath $transformersEnvironment) + ":/opt/transformers:ro"),
|
||||
"-v", ((Convert-ToDockerPath $runnerRoot) + ":/runner:ro"),
|
||||
"--entrypoint", "python3",
|
||||
"nvcr.io/nvidia/tritonserver:26.06-py3",
|
||||
("/runner/{0}" -f $runnerName), "run",
|
||||
"--job", "/job",
|
||||
"--frames", "/frames",
|
||||
"--timeline", "/work/timeline.jsonl",
|
||||
"--output", "/publish/output",
|
||||
"--cache", "/cache",
|
||||
"--calibration-sha256", $CalibrationSha256,
|
||||
"--calibration-slot", $CalibrationSlot,
|
||||
"--telemetry-interval-seconds", "1"
|
||||
)
|
||||
& docker @dockerArgs
|
||||
Assert-LastExitCode "Panoptic inference"
|
||||
Write-Output "PHASE=panoptic-inference-complete"
|
||||
|
||||
$videoPath = Join-Path $stagingRoot "perception.mp4"
|
||||
$encodeWatch = [Diagnostics.Stopwatch]::StartNew()
|
||||
$averageFps = $frameCount / $timelineDuration
|
||||
$fpsText = $averageFps.ToString("0.#########", [Globalization.CultureInfo]::InvariantCulture)
|
||||
$durationText = $timelineDuration.ToString("0.#########", [Globalization.CultureInfo]::InvariantCulture)
|
||||
& ffmpeg -hide_banner -loglevel error -framerate $fpsText -i (Join-Path $stagingRoot "overlay-frames\frame-%06d.png") -t $durationText -c:v h264_nvenc -preset p4 -tune hq -rc vbr -cq 21 -b:v 0 -pix_fmt yuv420p -movflags +faststart $videoPath
|
||||
Assert-LastExitCode "Panoptic video encoding"
|
||||
$encodeWatch.Stop()
|
||||
Write-Output "PHASE=video-encoding-complete"
|
||||
|
||||
$masksPath = Join-Path $stagingRoot "masks.tar.gz"
|
||||
& tar.exe -czf $masksPath -C $stagingRoot instance-masks semantic-masks
|
||||
Assert-LastExitCode "Mask archive publication"
|
||||
|
||||
$totalWatch.Stop()
|
||||
$finalizeArgs = @(
|
||||
"run", "--rm", "--network", "none",
|
||||
"--security-opt", "no-new-privileges:true", "--cap-drop", "ALL",
|
||||
"-v", ((Convert-ToDockerPath $stagingRoot) + ":/output:rw"),
|
||||
"-v", ((Convert-ToDockerPath $runnerRoot) + ":/runner:ro"),
|
||||
"--entrypoint", "python3",
|
||||
"nvcr.io/nvidia/tritonserver:26.06-py3",
|
||||
("/runner/{0}" -f $runnerName), "finalize",
|
||||
"--output", "/output",
|
||||
"--video", "/output/perception.mp4",
|
||||
"--masks", "/output/masks.tar.gz",
|
||||
"--extract-seconds", $extractWatch.Elapsed.TotalSeconds.ToString("0.######", [Globalization.CultureInfo]::InvariantCulture),
|
||||
"--encode-seconds", $encodeWatch.Elapsed.TotalSeconds.ToString("0.######", [Globalization.CultureInfo]::InvariantCulture),
|
||||
"--wall-seconds", $totalWatch.Elapsed.TotalSeconds.ToString("0.######", [Globalization.CultureInfo]::InvariantCulture),
|
||||
"--encoder", "ffmpeg-h264_nvenc-p4-cq21-yuv420p-faststart"
|
||||
)
|
||||
& docker @finalizeArgs
|
||||
Assert-LastExitCode "Result finalization"
|
||||
|
||||
$result = Get-Content -LiteralPath (Join-Path $stagingRoot "result.json") -Raw | ConvertFrom-Json
|
||||
if ($result.schema_version -ne "missioncore.recorded-perception-result/v2" -or $result.result_id -notmatch "^result-[a-f0-9]{64}$") {
|
||||
throw "Final result manifest is incompatible"
|
||||
}
|
||||
$finalRoot = Join-Path $derivedRoot ([string]$result.result_id)
|
||||
if (Test-Path -LiteralPath $finalRoot) {
|
||||
throw "An immutable result with the same identity already exists: $finalRoot"
|
||||
}
|
||||
|
||||
foreach ($temporaryChild in @("overlay-frames", "instance-masks", "semantic-masks")) {
|
||||
$temporaryPath = Join-Path $stagingRoot $temporaryChild
|
||||
if (Test-Path -LiteralPath $temporaryPath) {
|
||||
Remove-Item -LiteralPath $temporaryPath -Recurse -Force
|
||||
}
|
||||
}
|
||||
Move-Item -LiteralPath $stagingRoot -Destination $finalRoot
|
||||
Remove-Item -LiteralPath $publishRoot -Force
|
||||
Write-Output ("RESULT_ROOT={0}" -f $finalRoot)
|
||||
Write-Output ("RESULT_ID={0}" -f $result.result_id)
|
||||
}
|
||||
finally {
|
||||
if (Test-Path -LiteralPath $workRoot) {
|
||||
Remove-Item -LiteralPath $workRoot -Recurse -Force
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,62 @@
|
||||
# Mission Core CVAT D-only profile
|
||||
|
||||
This profile installs the annotation control plane in a dedicated WSL2
|
||||
distribution named `MissionCore-CVAT`. Its VHDX, Docker image store, CVAT
|
||||
source, persistent volumes, reports, and imported datasets live below
|
||||
`D:\NDC_MISSIONCORE`. It does not use the Docker Desktop image store that backs
|
||||
the Triton, Frigate, and Ollama containers.
|
||||
|
||||
Pinned inputs:
|
||||
|
||||
- Ubuntu 24.04.4 WSL AMD64 image, SHA-256
|
||||
`9b2f7730dc68227dd04a9f3e5eab86ad85caf556b8606ad94f1f29ff5c4fd3f5`
|
||||
- CVAT `v2.70.0`
|
||||
- D free-space floor: `360 GiB`
|
||||
- WSL VHD logical ceiling: `32 GB` (sparse allocation)
|
||||
|
||||
The official CVAT Compose topology is retained. The override only replaces its
|
||||
named volumes with explicit bind-backed directories inside the D-hosted WSL
|
||||
VHD. Images are pulled serially and the D free-space floor is checked before and
|
||||
after every image.
|
||||
|
||||
Run from the Windows host:
|
||||
|
||||
```powershell
|
||||
wsl -d MissionCore-CVAT -u root -- bash /mnt/d/NDC_MISSIONCORE/workspace/mission-core-compute/cvat/provision_cvat_wsl.sh
|
||||
wsl -d MissionCore-CVAT -u root -- bash /mnt/d/NDC_MISSIONCORE/workspace/mission-core-compute/cvat/ensure_cvat_admin.sh
|
||||
wsl -d MissionCore-CVAT -u root -- bash /mnt/d/NDC_MISSIONCORE/workspace/mission-core-compute/cvat/import_e2_workspace.sh
|
||||
```
|
||||
|
||||
After a Windows reboot, start the existing deployment without pulling images or
|
||||
re-provisioning it:
|
||||
|
||||
```powershell
|
||||
& D:\NDC_MISSIONCORE\workspace\mission-core-compute\cvat\Start-Cvat.ps1
|
||||
```
|
||||
|
||||
The launcher checks the `360 GiB` D-drive floor, starts the pinned Compose
|
||||
topology in the D-hosted WSL distribution, waits for the API, and keeps the WSL
|
||||
runtime alive. It writes startup logs only below
|
||||
`D:\NDC_MISSIONCORE\runtime\annotation\cvat\logs`.
|
||||
|
||||
From the Mission Core repository on the Mac, start CVAT if needed, discover the
|
||||
current WSL address, and open the SSH tunnel without a hard-coded IP:
|
||||
|
||||
```bash
|
||||
bash experiments/perception/worker/cvat/open_cvat_tunnel.sh
|
||||
```
|
||||
|
||||
Keep that terminal open and use `http://localhost:18080`. The same SSH session
|
||||
keeps the WSL runtime alive. Pass `--background` when detached runtime and
|
||||
tunnel sessions are preferred; rerun the helper after a Mac or Windows reboot.
|
||||
|
||||
The generated administrator password is stored only in
|
||||
`D:\NDC_MISSIONCORE\secrets\cvat\admin.env`; it is not printed by the script or
|
||||
committed to Git.
|
||||
|
||||
Traefik binds inside the dedicated WSL environment to `127.0.0.1:8080` and
|
||||
`127.0.0.1:8090`. It also publishes container port `8080` as WSL-internal port
|
||||
`18080` so the existing Windows SSH service can forward it without relying on
|
||||
Windows-to-WSL localhost forwarding. Remote review still uses the exact SSH
|
||||
host alias and a local forward; this profile does not add routes, Windows port
|
||||
proxies, DNS changes, or firewall rules.
|
||||
@@ -0,0 +1,58 @@
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[int]$TimeoutSeconds = 180
|
||||
)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
$Distro = 'MissionCore-CVAT'
|
||||
$Root = 'D:\NDC_MISSIONCORE'
|
||||
$RuntimeRoot = Join-Path $Root 'runtime\annotation\cvat'
|
||||
$LogRoot = Join-Path $RuntimeRoot 'logs'
|
||||
$LinuxStartScript = '/mnt/d/NDC_MISSIONCORE/workspace/mission-core-compute/cvat/start_cvat_runtime.sh'
|
||||
$FreeGiBFloor = 360
|
||||
|
||||
$drive = Get-PSDrive -Name D
|
||||
$freeGiB = [math]::Floor($drive.Free / 1GB)
|
||||
if ($freeGiB -lt $FreeGiBFloor) {
|
||||
throw "D: free-space floor crossed ($freeGiB GiB free; $FreeGiBFloor GiB required)"
|
||||
}
|
||||
|
||||
try {
|
||||
$about = Invoke-RestMethod -Uri 'http://localhost:8080/api/server/about' -TimeoutSec 3
|
||||
if ($about.version -eq '2.70.0') {
|
||||
Write-Output "CVAT already ready version=$($about.version) free_gib=$freeGiB"
|
||||
exit 0
|
||||
}
|
||||
} catch {
|
||||
# A stopped WSL distribution is the normal condition after a Windows reboot.
|
||||
}
|
||||
|
||||
New-Item -ItemType Directory -Path $LogRoot -Force | Out-Null
|
||||
$timestamp = (Get-Date).ToUniversalTime().ToString('yyyyMMddTHHmmssZ')
|
||||
$stdoutPath = Join-Path $LogRoot "start-$timestamp.stdout.log"
|
||||
$stderrPath = Join-Path $LogRoot "start-$timestamp.stderr.log"
|
||||
$arguments = @(
|
||||
'-d', $Distro,
|
||||
'-u', 'root',
|
||||
'--', 'bash', $LinuxStartScript, '--keepalive'
|
||||
)
|
||||
|
||||
Start-Process -FilePath 'wsl.exe' -ArgumentList $arguments -WindowStyle Hidden `
|
||||
-RedirectStandardOutput $stdoutPath -RedirectStandardError $stderrPath
|
||||
|
||||
$deadline = (Get-Date).AddSeconds($TimeoutSeconds)
|
||||
do {
|
||||
Start-Sleep -Seconds 2
|
||||
try {
|
||||
$about = Invoke-RestMethod -Uri 'http://localhost:8080/api/server/about' -TimeoutSec 3
|
||||
if ($about.version -eq '2.70.0') {
|
||||
Write-Output "CVAT ready version=$($about.version) free_gib=$freeGiB"
|
||||
Write-Output "startup_log=$stdoutPath"
|
||||
exit 0
|
||||
}
|
||||
} catch {
|
||||
# Continue until the complete CVAT Compose topology is ready.
|
||||
}
|
||||
} while ((Get-Date) -lt $deadline)
|
||||
|
||||
throw "CVAT did not become ready within $TimeoutSeconds seconds; inspect $stderrPath"
|
||||
@@ -0,0 +1,50 @@
|
||||
services:
|
||||
traefik:
|
||||
ports: !override
|
||||
- 127.0.0.1:8080:8080
|
||||
- 127.0.0.1:8090:8090
|
||||
- 0.0.0.0:18080:8080
|
||||
|
||||
volumes:
|
||||
cvat_db:
|
||||
driver: local
|
||||
driver_opts:
|
||||
type: none
|
||||
o: bind
|
||||
device: /srv/mission-core-cvat/volumes/cvat_db
|
||||
cvat_data:
|
||||
driver: local
|
||||
driver_opts:
|
||||
type: none
|
||||
o: bind
|
||||
device: /srv/mission-core-cvat/volumes/cvat_data
|
||||
cvat_keys:
|
||||
driver: local
|
||||
driver_opts:
|
||||
type: none
|
||||
o: bind
|
||||
device: /srv/mission-core-cvat/volumes/cvat_keys
|
||||
cvat_logs:
|
||||
driver: local
|
||||
driver_opts:
|
||||
type: none
|
||||
o: bind
|
||||
device: /srv/mission-core-cvat/volumes/cvat_logs
|
||||
cvat_inmem_db:
|
||||
driver: local
|
||||
driver_opts:
|
||||
type: none
|
||||
o: bind
|
||||
device: /srv/mission-core-cvat/volumes/cvat_inmem_db
|
||||
cvat_events_db:
|
||||
driver: local
|
||||
driver_opts:
|
||||
type: none
|
||||
o: bind
|
||||
device: /srv/mission-core-cvat/volumes/cvat_events_db
|
||||
cvat_cache_db:
|
||||
driver: local
|
||||
driver_opts:
|
||||
type: none
|
||||
o: bind
|
||||
device: /srv/mission-core-cvat/volumes/cvat_cache_db
|
||||
@@ -0,0 +1,39 @@
|
||||
#!/usr/bin/env bash
|
||||
set -Eeuo pipefail
|
||||
|
||||
export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
|
||||
|
||||
readonly SECRET_ROOT="/mnt/d/NDC_MISSIONCORE/secrets/cvat"
|
||||
readonly SECRET_FILE="${SECRET_ROOT}/admin.env"
|
||||
|
||||
if [[ "${EUID}" -ne 0 ]]; then
|
||||
echo "ensure_cvat_admin.sh must run as root" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
install -d -m 0700 "${SECRET_ROOT}"
|
||||
if [[ ! -f "${SECRET_FILE}" ]]; then
|
||||
umask 077
|
||||
password="$(openssl rand -hex 24)"
|
||||
{
|
||||
printf 'CVAT_ADMIN_USERNAME=missioncore\n'
|
||||
printf 'CVAT_ADMIN_EMAIL=missioncore@local.invalid\n'
|
||||
printf 'CVAT_ADMIN_PASSWORD=%s\n' "${password}"
|
||||
} >"${SECRET_FILE}"
|
||||
fi
|
||||
|
||||
set -a
|
||||
# shellcheck disable=SC1090
|
||||
. "${SECRET_FILE}"
|
||||
set +a
|
||||
|
||||
docker exec \
|
||||
-e "CVAT_ADMIN_USERNAME=${CVAT_ADMIN_USERNAME}" \
|
||||
-e "CVAT_ADMIN_EMAIL=${CVAT_ADMIN_EMAIL}" \
|
||||
-e "CVAT_ADMIN_PASSWORD=${CVAT_ADMIN_PASSWORD}" \
|
||||
cvat_server \
|
||||
python3 /home/django/manage.py shell -c \
|
||||
'import os; from django.contrib.auth import get_user_model; User = get_user_model(); user, _ = User.objects.get_or_create(username=os.environ["CVAT_ADMIN_USERNAME"]); user.email = os.environ["CVAT_ADMIN_EMAIL"]; user.is_staff = True; user.is_superuser = True; user.set_password(os.environ["CVAT_ADMIN_PASSWORD"]); user.save()'
|
||||
|
||||
printf 'admin_ready=true\nusername=%s\nsecret_file=%s\n' \
|
||||
"${CVAT_ADMIN_USERNAME}" "${SECRET_FILE}"
|
||||
@@ -0,0 +1,284 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import time
|
||||
from collections import Counter
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from cvat_sdk import make_client, models
|
||||
from cvat_sdk.api_client.exceptions import ServiceException
|
||||
from cvat_sdk.core.proxies.tasks import ResourceType
|
||||
|
||||
WORKSPACE_ID = (
|
||||
"annotation-workspace-"
|
||||
"9a950d1c37d56dc12cc285b13c5addd7795285879cbcb1fbb2d5811c3c69821a"
|
||||
)
|
||||
EVALUATION_PACK_ID = (
|
||||
"evaluation-pack-"
|
||||
"7a983bba75d46c7c260252cb2d461e1384dcb92cda9e164397e841e6ebb37789"
|
||||
)
|
||||
EXPECTED_FRAME_COUNT = 64
|
||||
EXPECTED_INSTANCE_COUNT = 775
|
||||
BACKGROUND_LABEL = {
|
||||
"name": "background",
|
||||
"color": "#000000",
|
||||
"attributes": [],
|
||||
}
|
||||
|
||||
|
||||
def _sha256(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as source:
|
||||
for chunk in iter(lambda: source.read(1024 * 1024), b""):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def _label_specs(raw_labels: list[dict[str, Any]]) -> list[dict[str, Any]]:
|
||||
labels = [
|
||||
{
|
||||
"name": item["name"],
|
||||
"color": item["color"],
|
||||
"attributes": [],
|
||||
}
|
||||
for item in raw_labels
|
||||
]
|
||||
if not any(label["name"] == BACKGROUND_LABEL["name"] for label in labels):
|
||||
labels.insert(0, dict(BACKGROUND_LABEL))
|
||||
return labels
|
||||
|
||||
|
||||
def _annotation_counts(task: Any) -> dict[str, int]:
|
||||
annotations = task.get_annotations()
|
||||
return {
|
||||
"shape_count": len(annotations.shapes),
|
||||
"tag_count": len(annotations.tags),
|
||||
"track_count": len(annotations.tracks),
|
||||
}
|
||||
|
||||
|
||||
def _list_tasks_when_ready(client: Any, *, attempts: int = 60) -> list[Any]:
|
||||
for attempt in range(1, attempts + 1):
|
||||
try:
|
||||
return list(client.tasks.list())
|
||||
except ServiceException as error:
|
||||
if error.status not in {500, 502, 503} or attempt == attempts:
|
||||
raise
|
||||
time.sleep(2)
|
||||
raise AssertionError("unreachable")
|
||||
|
||||
|
||||
def _task_record(task: Any, *, disposition: str, expected_labels: list[str]) -> dict[str, Any]:
|
||||
task.fetch()
|
||||
task_labels = list(task.get_labels())
|
||||
actual_labels = sorted(label.name for label in task_labels)
|
||||
if task.size != EXPECTED_FRAME_COUNT:
|
||||
raise RuntimeError(
|
||||
f"Task {task.id} has {task.size} frames, expected {EXPECTED_FRAME_COUNT}"
|
||||
)
|
||||
if actual_labels != sorted(expected_labels):
|
||||
raise RuntimeError(
|
||||
f"Task {task.id} labels differ: actual={actual_labels!r} "
|
||||
f"expected={sorted(expected_labels)!r}"
|
||||
)
|
||||
annotation_counts = _annotation_counts(task)
|
||||
label_names_by_id = {label.id: label.name for label in task_labels}
|
||||
annotations = task.get_annotations()
|
||||
shape_counts_by_label = dict(
|
||||
sorted(
|
||||
Counter(
|
||||
label_names_by_id.get(shape.label_id, f"unknown:{shape.label_id}")
|
||||
for shape in annotations.shapes
|
||||
).items()
|
||||
)
|
||||
)
|
||||
return {
|
||||
"id": task.id,
|
||||
"name": task.name,
|
||||
"disposition": disposition,
|
||||
"frame_count": task.size,
|
||||
"label_names": actual_labels,
|
||||
**annotation_counts,
|
||||
"shape_counts_by_label": shape_counts_by_label,
|
||||
"url_path": f"/tasks/{task.id}",
|
||||
}
|
||||
|
||||
|
||||
def _ensure_task(
|
||||
client: Any,
|
||||
*,
|
||||
name: str,
|
||||
labels: list[dict[str, Any]],
|
||||
images_path: Path,
|
||||
annotation_path: Path,
|
||||
annotation_format: str,
|
||||
expected_shape_count: int | None,
|
||||
) -> dict[str, Any]:
|
||||
matches = [task for task in _list_tasks_when_ready(client) if task.name == name]
|
||||
if len(matches) > 1:
|
||||
raise RuntimeError(f"Multiple CVAT tasks have the reserved name {name!r}")
|
||||
|
||||
expected_labels = [label["name"] for label in labels]
|
||||
if matches:
|
||||
task = matches[0]
|
||||
task.fetch()
|
||||
actual_labels = sorted(label.name for label in task.get_labels())
|
||||
expected_without_background = sorted(
|
||||
label for label in expected_labels if label != BACKGROUND_LABEL["name"]
|
||||
)
|
||||
disposition = "reused"
|
||||
if (
|
||||
actual_labels == expected_without_background
|
||||
and BACKGROUND_LABEL["name"] in expected_labels
|
||||
):
|
||||
task.update(
|
||||
models.PatchedTaskWriteRequest(
|
||||
labels=[models.PatchedLabelRequest(**BACKGROUND_LABEL)]
|
||||
)
|
||||
)
|
||||
disposition = "reused-and-background-label-added"
|
||||
|
||||
record = _task_record(
|
||||
task,
|
||||
disposition=disposition,
|
||||
expected_labels=expected_labels,
|
||||
)
|
||||
if record["shape_count"] == 0:
|
||||
task.import_annotations(annotation_format, annotation_path)
|
||||
record = _task_record(
|
||||
task,
|
||||
disposition=f"{disposition}-and-annotations-imported",
|
||||
expected_labels=expected_labels,
|
||||
)
|
||||
if expected_shape_count is not None and record["shape_count"] != expected_shape_count:
|
||||
raise RuntimeError(
|
||||
f"Task {task.id} has {record['shape_count']} shapes, "
|
||||
f"expected {expected_shape_count}"
|
||||
)
|
||||
return record
|
||||
|
||||
task = client.tasks.create_from_data(
|
||||
spec=models.TaskWriteRequest(
|
||||
name=name,
|
||||
labels=labels,
|
||||
segment_size=EXPECTED_FRAME_COUNT,
|
||||
overlap=0,
|
||||
),
|
||||
resources=[images_path],
|
||||
resource_type=ResourceType.LOCAL,
|
||||
data_params={
|
||||
"image_quality": 100,
|
||||
"sorting_method": "lexicographical",
|
||||
"use_cache": False,
|
||||
},
|
||||
annotation_path=annotation_path,
|
||||
annotation_format=annotation_format,
|
||||
status_check_period=2,
|
||||
)
|
||||
record = _task_record(task, disposition="created", expected_labels=expected_labels)
|
||||
if expected_shape_count is not None and record["shape_count"] != expected_shape_count:
|
||||
raise RuntimeError(
|
||||
f"Task {task.id} has {record['shape_count']} shapes, "
|
||||
f"expected {expected_shape_count}"
|
||||
)
|
||||
return record
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--server", required=True)
|
||||
parser.add_argument("--username", required=True)
|
||||
parser.add_argument("--password-env", required=True)
|
||||
parser.add_argument("--workspace", type=Path, required=True)
|
||||
parser.add_argument("--report", type=Path, required=True)
|
||||
args = parser.parse_args()
|
||||
|
||||
password = os.environ.get(args.password_env)
|
||||
if not password:
|
||||
raise RuntimeError(f"Password environment variable {args.password_env!r} is empty")
|
||||
|
||||
workspace = args.workspace.resolve()
|
||||
if workspace.name != WORKSPACE_ID:
|
||||
raise RuntimeError(f"Unexpected annotation workspace: {workspace}")
|
||||
|
||||
manifest_path = workspace / "manifest.json"
|
||||
labels_path = workspace / "cvat" / "labels.json"
|
||||
images_path = workspace / "cvat" / "images.zip"
|
||||
instance_path = workspace / "cvat" / "instance-coco.zip"
|
||||
semantic_path = workspace / "cvat" / "semantic-mask.zip"
|
||||
manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
|
||||
labels = json.loads(labels_path.read_text(encoding="utf-8"))
|
||||
|
||||
if manifest["ground_truth"] is not False:
|
||||
raise RuntimeError("LAB E2 import must remain an unreviewed model draft")
|
||||
if manifest["identity"]["frame_count"] != EXPECTED_FRAME_COUNT:
|
||||
raise RuntimeError("Unexpected LAB E2 frame count")
|
||||
if manifest["identity"]["draft_instance_count"] != EXPECTED_INSTANCE_COUNT:
|
||||
raise RuntimeError("Unexpected LAB E2 instance count")
|
||||
|
||||
task_specs = [
|
||||
{
|
||||
"name": "LAB E2 | K1 | instance prelabels | pack 7a983bba",
|
||||
"labels": _label_specs(labels["instance_task"]),
|
||||
"annotation_path": instance_path,
|
||||
"annotation_format": "COCO 1.0",
|
||||
"expected_shape_count": EXPECTED_INSTANCE_COUNT,
|
||||
},
|
||||
{
|
||||
"name": "LAB E2 | K1 | dense semantic prelabels | pack 7a983bba",
|
||||
"labels": _label_specs(labels["semantic_task"]),
|
||||
"annotation_path": semantic_path,
|
||||
"annotation_format": "Segmentation mask 1.1",
|
||||
"expected_shape_count": None,
|
||||
},
|
||||
]
|
||||
|
||||
with make_client(args.server, credentials=(args.username, password)) as client:
|
||||
client.check_server_version(fail_if_unsupported=True)
|
||||
task_records = [
|
||||
_ensure_task(
|
||||
client,
|
||||
name=task_spec["name"],
|
||||
labels=task_spec["labels"],
|
||||
images_path=images_path,
|
||||
annotation_path=task_spec["annotation_path"],
|
||||
annotation_format=task_spec["annotation_format"],
|
||||
expected_shape_count=task_spec["expected_shape_count"],
|
||||
)
|
||||
for task_spec in task_specs
|
||||
]
|
||||
|
||||
report = {
|
||||
"schema_version": "missioncore.lab-e2-cvat-import/v1",
|
||||
"created_at_utc": datetime.now(UTC).isoformat(timespec="milliseconds").replace(
|
||||
"+00:00", "Z"
|
||||
),
|
||||
"server": args.server,
|
||||
"cvat_version": "v2.70.0",
|
||||
"workspace_id": WORKSPACE_ID,
|
||||
"evaluation_pack_id": EVALUATION_PACK_ID,
|
||||
"workspace_manifest_sha256": _sha256(manifest_path),
|
||||
"ground_truth": False,
|
||||
"inputs": {
|
||||
"images_zip_sha256": _sha256(images_path),
|
||||
"instance_coco_zip_sha256": _sha256(instance_path),
|
||||
"semantic_mask_zip_sha256": _sha256(semantic_path),
|
||||
},
|
||||
"tasks": task_records,
|
||||
"next_gate": (
|
||||
"two-pass human review and reviewed export; "
|
||||
"do not treat drafts as accuracy evidence"
|
||||
),
|
||||
}
|
||||
args.report.parent.mkdir(parents=True, exist_ok=True)
|
||||
args.report.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8")
|
||||
print(json.dumps(report, indent=2))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,89 @@
|
||||
#!/usr/bin/env bash
|
||||
set -Eeuo pipefail
|
||||
|
||||
export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
|
||||
|
||||
readonly FREE_GIB_FLOOR="${FREE_GIB_FLOOR:-360}"
|
||||
readonly WINDOWS_ROOT="/mnt/d/NDC_MISSIONCORE"
|
||||
readonly COMPUTE_ROOT="${WINDOWS_ROOT}/workspace/mission-core-compute"
|
||||
readonly TOOL_ROOT="/srv/mission-core-cvat/tools/e2-import"
|
||||
readonly SECRET_FILE="${WINDOWS_ROOT}/secrets/cvat/admin.env"
|
||||
readonly WORKSPACE_ROOT="${WINDOWS_ROOT}/runtime/annotation/imports/LAB-E2/annotation-workspace-9a950d1c37d56dc12cc285b13c5addd7795285879cbcb1fbb2d5811c3c69821a"
|
||||
readonly REPORT_ROOT="${WINDOWS_ROOT}/runtime/annotation/cvat/reports"
|
||||
|
||||
if [[ "${EUID}" -ne 0 ]]; then
|
||||
echo "import_e2_workspace.sh must run as root" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
free_gib() {
|
||||
df -BG --output=avail "${WINDOWS_ROOT}" | tail -n 1 | tr -dc '0-9'
|
||||
}
|
||||
|
||||
guard_disk() {
|
||||
local stage="$1"
|
||||
local free
|
||||
free="$(free_gib)"
|
||||
printf 'disk_guard stage=%s free_gib=%s floor_gib=%s\n' \
|
||||
"${stage}" "${free}" "${FREE_GIB_FLOOR}"
|
||||
if (( free < FREE_GIB_FLOOR )); then
|
||||
echo "D: free-space floor crossed; refusing further writes" >&2
|
||||
exit 3
|
||||
fi
|
||||
}
|
||||
|
||||
wait_for_cvat() {
|
||||
local attempt http_code
|
||||
for attempt in $(seq 1 60); do
|
||||
http_code="$(
|
||||
curl --silent --output /dev/null --write-out '%{http_code}' \
|
||||
"http://localhost:8080/api/server/about" || true
|
||||
)"
|
||||
if [[ "${http_code}" == "200" ]]; then
|
||||
http_code="$(
|
||||
curl --silent --output /dev/null --write-out '%{http_code}' \
|
||||
"http://localhost:8080/api/tasks" || true
|
||||
)"
|
||||
if [[ "${http_code}" == "200" || "${http_code}" == "401" || "${http_code}" == "403" ]]; then
|
||||
printf 'cvat_ready attempt=%s tasks_http=%s\n' "${attempt}" "${http_code}"
|
||||
return 0
|
||||
fi
|
||||
fi
|
||||
sleep 2
|
||||
done
|
||||
echo "CVAT API did not become ready within 120 seconds" >&2
|
||||
return 1
|
||||
}
|
||||
|
||||
test -f "${SECRET_FILE}"
|
||||
test -f "${WORKSPACE_ROOT}/manifest.json"
|
||||
test -f "${COMPUTE_ROOT}/cvat/import_e2_workspace.py"
|
||||
guard_disk preflight
|
||||
|
||||
export DEBIAN_FRONTEND=noninteractive
|
||||
apt-get update
|
||||
apt-get install -y python3-venv
|
||||
install -d -m 0750 "${TOOL_ROOT}" "${REPORT_ROOT}"
|
||||
|
||||
if [[ ! -x "${TOOL_ROOT}/venv/bin/python" ]]; then
|
||||
python3 -m venv "${TOOL_ROOT}/venv"
|
||||
"${TOOL_ROOT}/venv/bin/pip" install --disable-pip-version-check \
|
||||
"cvat-sdk==2.70.0"
|
||||
fi
|
||||
|
||||
set -a
|
||||
# shellcheck disable=SC1090
|
||||
. "${SECRET_FILE}"
|
||||
set +a
|
||||
|
||||
wait_for_cvat
|
||||
report="${REPORT_ROOT}/lab-e2-cvat-import-$(date -u +%Y%m%dT%H%M%SZ).json"
|
||||
"${TOOL_ROOT}/venv/bin/python" "${COMPUTE_ROOT}/cvat/import_e2_workspace.py" \
|
||||
--server "http://localhost:8080" \
|
||||
--username "${CVAT_ADMIN_USERNAME}" \
|
||||
--password-env CVAT_ADMIN_PASSWORD \
|
||||
--workspace "${WORKSPACE_ROOT}" \
|
||||
--report "${report}"
|
||||
|
||||
guard_disk imported
|
||||
printf 'import_report=%s\n' "${report}"
|
||||
@@ -0,0 +1,179 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path, PurePosixPath
|
||||
from typing import Any
|
||||
|
||||
from cvat_sdk import make_client
|
||||
from import_e2_workspace import _ensure_task, _label_specs, _sha256
|
||||
|
||||
WORKSPACE_SCHEMA = "missioncore.lab-e3-cvat-review-workspace/v1"
|
||||
IDENTITY_SCHEMA = "missioncore.lab-e3-cvat-review-identity/v1"
|
||||
EXPECTED_PACK_ID = (
|
||||
"evaluation-pack-"
|
||||
"7a983bba75d46c7c260252cb2d461e1384dcb92cda9e164397e841e6ebb37789"
|
||||
)
|
||||
EXPECTED_RESULT_ID = (
|
||||
"e3-segmentation-"
|
||||
"01bd497c44c2b940add145ec784d3418010327bce0baddd4420b0925317e8a16"
|
||||
)
|
||||
EXPECTED_FRAMES = 64
|
||||
|
||||
|
||||
def _canonical_json(value: object) -> bytes:
|
||||
return json.dumps(
|
||||
value,
|
||||
ensure_ascii=False,
|
||||
sort_keys=True,
|
||||
separators=(",", ":"),
|
||||
allow_nan=False,
|
||||
).encode()
|
||||
|
||||
|
||||
def _valid_sha256(value: object) -> bool:
|
||||
return (
|
||||
isinstance(value, str)
|
||||
and len(value) == 64
|
||||
and all(character in "0123456789abcdef" for character in value)
|
||||
)
|
||||
|
||||
|
||||
def _read_object(path: Path) -> dict[str, Any]:
|
||||
value = json.loads(path.read_text(encoding="utf-8"))
|
||||
if not isinstance(value, dict):
|
||||
raise RuntimeError(f"JSON root is not an object: {path}")
|
||||
return value
|
||||
|
||||
|
||||
def _safe_artifact(root: Path, encoded: object) -> Path:
|
||||
if not isinstance(encoded, str):
|
||||
raise RuntimeError("artifact path is not a string")
|
||||
relative = PurePosixPath(encoded)
|
||||
if relative.is_absolute() or ".." in relative.parts or not relative.parts:
|
||||
raise RuntimeError("artifact path is unsafe")
|
||||
path = root.joinpath(*relative.parts).resolve(strict=True)
|
||||
if not path.is_file() or not path.is_relative_to(root):
|
||||
raise RuntimeError("artifact path escaped its root")
|
||||
return path
|
||||
|
||||
|
||||
def _workspace(root: Path, images_zip: Path) -> tuple[dict[str, Any], dict[str, Path]]:
|
||||
manifest_path = root / "manifest.json"
|
||||
manifest = _read_object(manifest_path)
|
||||
identity = manifest.get("identity")
|
||||
identity_sha256 = manifest.get("identity_sha256")
|
||||
if (
|
||||
manifest.get("schema_version") != WORKSPACE_SCHEMA
|
||||
or manifest.get("ground_truth") is not False
|
||||
or not isinstance(identity, dict)
|
||||
or identity.get("schema_version") != IDENTITY_SCHEMA
|
||||
or not _valid_sha256(identity_sha256)
|
||||
or hashlib.sha256(_canonical_json(identity)).hexdigest() != identity_sha256
|
||||
or manifest.get("workspace_id") != f"e3-cvat-review-{identity_sha256}"
|
||||
or root.name != manifest.get("workspace_id")
|
||||
or identity.get("evaluation_pack_id") != EXPECTED_PACK_ID
|
||||
or identity.get("e3_result_id") != EXPECTED_RESULT_ID
|
||||
or identity.get("frame_count") != EXPECTED_FRAMES
|
||||
or _sha256(images_zip) != identity.get("images_zip_sha256")
|
||||
):
|
||||
raise RuntimeError("LAB E3 CVAT review workspace is incompatible")
|
||||
artifacts: dict[str, Path] = {}
|
||||
descriptors = manifest.get("artifacts")
|
||||
if not isinstance(descriptors, list):
|
||||
raise RuntimeError("LAB E3 CVAT artifact list is invalid")
|
||||
for descriptor in descriptors:
|
||||
if not isinstance(descriptor, dict):
|
||||
raise RuntimeError("LAB E3 CVAT artifact descriptor is invalid")
|
||||
path = _safe_artifact(root, descriptor.get("path"))
|
||||
if (
|
||||
path.stat().st_size != descriptor.get("bytes")
|
||||
or not _valid_sha256(descriptor.get("sha256"))
|
||||
or _sha256(path) != descriptor["sha256"]
|
||||
):
|
||||
raise RuntimeError(f"LAB E3 CVAT artifact changed: {path.name}")
|
||||
artifacts[path.name] = path
|
||||
if set(artifacts) != {
|
||||
"labels.json",
|
||||
"control-fisheye-mask.zip",
|
||||
"challenger-kb4-cubemap5.zip",
|
||||
}:
|
||||
raise RuntimeError("LAB E3 CVAT artifact set changed")
|
||||
return manifest, artifacts
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--server", required=True)
|
||||
parser.add_argument("--username", required=True)
|
||||
parser.add_argument("--password-env", required=True)
|
||||
parser.add_argument("--workspace", type=Path, required=True)
|
||||
parser.add_argument("--images-zip", type=Path, required=True)
|
||||
parser.add_argument("--report", type=Path, required=True)
|
||||
args = parser.parse_args()
|
||||
|
||||
password = os.environ.get(args.password_env)
|
||||
if not password:
|
||||
raise RuntimeError(f"Password environment variable {args.password_env!r} is empty")
|
||||
workspace = args.workspace.resolve(strict=True)
|
||||
images_zip = args.images_zip.resolve(strict=True)
|
||||
manifest, artifacts = _workspace(workspace, images_zip)
|
||||
labels_document = _read_object(artifacts["labels.json"])
|
||||
semantic_labels = labels_document.get("semantic_task")
|
||||
if not isinstance(semantic_labels, list):
|
||||
raise RuntimeError("semantic labels are absent")
|
||||
labels = _label_specs(semantic_labels)
|
||||
task_specs = (
|
||||
{
|
||||
"name": "LAB E3 | K1 | EoMT fisheye control | pack 7a983bba",
|
||||
"annotation": artifacts["control-fisheye-mask.zip"],
|
||||
"role": "control",
|
||||
},
|
||||
{
|
||||
"name": "LAB E3 | K1 | EoMT KB4 cubemap5 challenger | pack 7a983bba",
|
||||
"annotation": artifacts["challenger-kb4-cubemap5.zip"],
|
||||
"role": "challenger",
|
||||
},
|
||||
)
|
||||
with make_client(args.server, credentials=(args.username, password)) as client:
|
||||
client.check_server_version(fail_if_unsupported=True)
|
||||
tasks = []
|
||||
for spec in task_specs:
|
||||
record = _ensure_task(
|
||||
client,
|
||||
name=spec["name"],
|
||||
labels=labels,
|
||||
images_path=images_zip,
|
||||
annotation_path=spec["annotation"],
|
||||
annotation_format="Segmentation mask 1.1",
|
||||
expected_shape_count=None,
|
||||
)
|
||||
record["role"] = spec["role"]
|
||||
tasks.append(record)
|
||||
|
||||
report = {
|
||||
"schema_version": "missioncore.lab-e3-cvat-import/v1",
|
||||
"created_at_utc": datetime.now(UTC)
|
||||
.isoformat(timespec="milliseconds")
|
||||
.replace("+00:00", "Z"),
|
||||
"server": args.server,
|
||||
"cvat_version": "v2.70.0",
|
||||
"workspace_id": manifest["workspace_id"],
|
||||
"workspace_manifest_sha256": _sha256(workspace / "manifest.json"),
|
||||
"evaluation_pack_id": EXPECTED_PACK_ID,
|
||||
"e3_result_id": EXPECTED_RESULT_ID,
|
||||
"ground_truth": False,
|
||||
"priority_image_ids": manifest["identity"]["priority_image_ids"],
|
||||
"tasks": tasks,
|
||||
"next_gate": "two-pass human review and reviewed export",
|
||||
}
|
||||
args.report.parent.mkdir(parents=True, exist_ok=True)
|
||||
args.report.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8")
|
||||
print(json.dumps(report, indent=2))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,79 @@
|
||||
#!/usr/bin/env bash
|
||||
set -Eeuo pipefail
|
||||
|
||||
export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
|
||||
|
||||
readonly FREE_GIB_FLOOR="${FREE_GIB_FLOOR:-360}"
|
||||
readonly WINDOWS_ROOT="/mnt/d/NDC_MISSIONCORE"
|
||||
readonly COMPUTE_ROOT="${WINDOWS_ROOT}/workspace/mission-core-compute"
|
||||
readonly TOOL_ROOT="/srv/mission-core-cvat/tools/e2-import"
|
||||
readonly SECRET_FILE="${WINDOWS_ROOT}/secrets/cvat/admin.env"
|
||||
readonly WORKSPACE_ID="e3-cvat-review-ca599521e345446ca8e9af5a9013062099278e7317f83ff89740c6b092ddc52f"
|
||||
readonly WORKSPACE_ROOT="${WINDOWS_ROOT}/runtime/annotation/imports/LAB-E3/${WORKSPACE_ID}"
|
||||
readonly E2_WORKSPACE_ID="annotation-workspace-9a950d1c37d56dc12cc285b13c5addd7795285879cbcb1fbb2d5811c3c69821a"
|
||||
readonly IMAGES_ZIP="${WINDOWS_ROOT}/runtime/annotation/imports/LAB-E2/${E2_WORKSPACE_ID}/cvat/images.zip"
|
||||
readonly REPORT_ROOT="${WINDOWS_ROOT}/runtime/annotation/cvat/reports"
|
||||
|
||||
if [[ "${EUID}" -ne 0 ]]; then
|
||||
echo "import_e3_review.sh must run as root" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
free_gib() {
|
||||
df -BG --output=avail "${WINDOWS_ROOT}" | tail -n 1 | tr -dc '0-9'
|
||||
}
|
||||
|
||||
guard_disk() {
|
||||
local stage="$1"
|
||||
local free
|
||||
free="$(free_gib)"
|
||||
printf 'disk_guard stage=%s free_gib=%s floor_gib=%s\n' \
|
||||
"${stage}" "${free}" "${FREE_GIB_FLOOR}"
|
||||
if (( free < FREE_GIB_FLOOR )); then
|
||||
echo "D: free-space floor crossed; refusing further writes" >&2
|
||||
exit 3
|
||||
fi
|
||||
}
|
||||
|
||||
wait_for_cvat() {
|
||||
local attempt http_code
|
||||
for attempt in $(seq 1 60); do
|
||||
http_code="$(
|
||||
curl --silent --output /dev/null --write-out '%{http_code}' \
|
||||
"http://localhost:8080/api/server/about" || true
|
||||
)"
|
||||
if [[ "${http_code}" == "200" ]]; then
|
||||
printf 'cvat_ready attempt=%s\n' "${attempt}"
|
||||
return 0
|
||||
fi
|
||||
sleep 2
|
||||
done
|
||||
echo "CVAT API did not become ready within 120 seconds" >&2
|
||||
return 1
|
||||
}
|
||||
|
||||
test -x "${TOOL_ROOT}/venv/bin/python"
|
||||
test -f "${SECRET_FILE}"
|
||||
test -f "${WORKSPACE_ROOT}/manifest.json"
|
||||
test -f "${IMAGES_ZIP}"
|
||||
test -f "${COMPUTE_ROOT}/cvat/import_e2_workspace.py"
|
||||
test -f "${COMPUTE_ROOT}/cvat/import_e3_review.py"
|
||||
guard_disk preflight
|
||||
|
||||
set -a
|
||||
# shellcheck disable=SC1090
|
||||
. "${SECRET_FILE}"
|
||||
set +a
|
||||
|
||||
wait_for_cvat
|
||||
report="${REPORT_ROOT}/lab-e3-cvat-import-$(date -u +%Y%m%dT%H%M%SZ).json"
|
||||
"${TOOL_ROOT}/venv/bin/python" "${COMPUTE_ROOT}/cvat/import_e3_review.py" \
|
||||
--server "http://localhost:8080" \
|
||||
--username "${CVAT_ADMIN_USERNAME}" \
|
||||
--password-env CVAT_ADMIN_PASSWORD \
|
||||
--workspace "${WORKSPACE_ROOT}" \
|
||||
--images-zip "${IMAGES_ZIP}" \
|
||||
--report "${report}"
|
||||
|
||||
guard_disk imported
|
||||
printf 'import_report=%s\n' "${report}"
|
||||
@@ -0,0 +1,91 @@
|
||||
#!/usr/bin/env bash
|
||||
set -Eeuo pipefail
|
||||
|
||||
readonly SSH_ALIAS="mission-gpu"
|
||||
readonly LOCAL_PORT="${CVAT_LOCAL_PORT:-18080}"
|
||||
readonly REMOTE_WSL_PORT="18080"
|
||||
readonly LINUX_START_SCRIPT='/mnt/d/NDC_MISSIONCORE/workspace/mission-core-compute/cvat/start_cvat_runtime.sh'
|
||||
|
||||
if curl --max-time 2 --fail --silent \
|
||||
"http://localhost:${LOCAL_PORT}/api/server/about" >/dev/null 2>&1; then
|
||||
printf 'cvat_tunnel_ready url=http://localhost:%s\n' "${LOCAL_PORT}"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if command -v lsof >/dev/null \
|
||||
&& lsof -nP -iTCP:"${LOCAL_PORT}" -sTCP:LISTEN >/dev/null 2>&1; then
|
||||
echo "Local port ${LOCAL_PORT} is already occupied by a non-responsive process" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
runtime_command="wsl.exe -d MissionCore-CVAT -u root -- bash ${LINUX_START_SCRIPT} --keepalive"
|
||||
background=false
|
||||
if [[ "${1:-}" == "--background" ]]; then
|
||||
background=true
|
||||
ssh -f "${SSH_ALIAS}" "${runtime_command}"
|
||||
else
|
||||
ssh "${SSH_ALIAS}" "${runtime_command}" &
|
||||
runtime_ssh_pid=$!
|
||||
cleanup() {
|
||||
if [[ -n "${tunnel_ssh_pid:-}" ]]; then
|
||||
kill "${tunnel_ssh_pid}" 2>/dev/null || true
|
||||
fi
|
||||
kill "${runtime_ssh_pid}" 2>/dev/null || true
|
||||
}
|
||||
trap cleanup EXIT INT TERM
|
||||
fi
|
||||
|
||||
for attempt in $(seq 1 30); do
|
||||
cvat_ip="$(
|
||||
ssh "${SSH_ALIAS}" \
|
||||
"wsl.exe -d MissionCore-CVAT -u root -- hostname -I" \
|
||||
| tr -d '\r' | awk '{print $1}'
|
||||
)"
|
||||
if [[ "${cvat_ip}" =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
|
||||
break
|
||||
fi
|
||||
sleep 1
|
||||
done
|
||||
if [[ ! "${cvat_ip:-}" =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
|
||||
echo "MissionCore-CVAT did not return a valid WSL address" >&2
|
||||
if [[ "${background}" == false ]]; then
|
||||
kill "${runtime_ssh_pid}" 2>/dev/null || true
|
||||
fi
|
||||
exit 3
|
||||
fi
|
||||
|
||||
forward=(
|
||||
-N
|
||||
-L "${LOCAL_PORT}:${cvat_ip}:${REMOTE_WSL_PORT}"
|
||||
-o ExitOnForwardFailure=yes
|
||||
"${SSH_ALIAS}"
|
||||
)
|
||||
|
||||
if [[ "${background}" == true ]]; then
|
||||
ssh -f "${forward[@]}"
|
||||
else
|
||||
ssh "${forward[@]}" &
|
||||
tunnel_ssh_pid=$!
|
||||
fi
|
||||
|
||||
for attempt in $(seq 1 60); do
|
||||
if curl --max-time 2 --fail --silent \
|
||||
"http://localhost:${LOCAL_PORT}/api/server/about" >/dev/null 2>&1; then
|
||||
printf 'cvat_tunnel_ready url=http://localhost:%s wsl_ip=%s\n' \
|
||||
"${LOCAL_PORT}" "${cvat_ip}"
|
||||
if [[ "${background}" == true ]]; then
|
||||
exit 0
|
||||
fi
|
||||
wait "${tunnel_ssh_pid}"
|
||||
tunnel_status=$?
|
||||
kill "${runtime_ssh_pid}" 2>/dev/null || true
|
||||
exit "${tunnel_status}"
|
||||
fi
|
||||
sleep 2
|
||||
done
|
||||
|
||||
echo "CVAT tunnel did not become ready within 120 seconds" >&2
|
||||
if [[ "${background}" == false ]]; then
|
||||
kill "${tunnel_ssh_pid}" "${runtime_ssh_pid}" 2>/dev/null || true
|
||||
fi
|
||||
exit 4
|
||||
@@ -0,0 +1,132 @@
|
||||
#!/usr/bin/env bash
|
||||
set -Eeuo pipefail
|
||||
|
||||
# Do not inherit Windows executables into this D-only runtime. In particular,
|
||||
# Docker Desktop's docker.exe belongs to a different image store.
|
||||
export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
|
||||
|
||||
readonly CVAT_TAG="${CVAT_TAG:-v2.70.0}"
|
||||
readonly FREE_GIB_FLOOR="${FREE_GIB_FLOOR:-360}"
|
||||
readonly WINDOWS_ROOT="/mnt/d/NDC_MISSIONCORE"
|
||||
readonly COMPUTE_ROOT="${WINDOWS_ROOT}/workspace/mission-core-compute"
|
||||
readonly RUNTIME_ROOT="${WINDOWS_ROOT}/runtime/annotation/cvat"
|
||||
readonly CVAT_ROOT="${RUNTIME_ROOT}/source/cvat-${CVAT_TAG}"
|
||||
readonly STATE_ROOT="/srv/mission-core-cvat"
|
||||
readonly OVERRIDE_FILE="${COMPUTE_ROOT}/cvat/docker-compose.override.yml"
|
||||
|
||||
if [[ "${EUID}" -ne 0 ]]; then
|
||||
echo "provision_cvat_wsl.sh must run as root" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
free_gib() {
|
||||
df -BG --output=avail "${WINDOWS_ROOT}" | tail -n 1 | tr -dc '0-9'
|
||||
}
|
||||
|
||||
guard_disk() {
|
||||
local stage="$1"
|
||||
local free
|
||||
free="$(free_gib)"
|
||||
printf 'disk_guard stage=%s free_gib=%s floor_gib=%s\n' \
|
||||
"${stage}" "${free}" "${FREE_GIB_FLOOR}"
|
||||
if (( free < FREE_GIB_FLOOR )); then
|
||||
echo "D: free-space floor crossed; refusing further writes" >&2
|
||||
exit 3
|
||||
fi
|
||||
}
|
||||
|
||||
guard_disk preflight
|
||||
|
||||
export DEBIAN_FRONTEND=noninteractive
|
||||
apt-get update
|
||||
apt-get install -y ca-certificates curl git
|
||||
|
||||
if ! dpkg-query -W -f='${Status}' docker-ce 2>/dev/null \
|
||||
| grep -qx 'install ok installed'; then
|
||||
install -m 0755 -d /etc/apt/keyrings
|
||||
curl -fsSL https://download.docker.com/linux/ubuntu/gpg \
|
||||
-o /etc/apt/keyrings/docker.asc
|
||||
chmod a+r /etc/apt/keyrings/docker.asc
|
||||
# shellcheck disable=SC1091
|
||||
. /etc/os-release
|
||||
arch="$(dpkg --print-architecture)"
|
||||
printf 'deb [arch=%s signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu %s stable\n' \
|
||||
"${arch}" "${VERSION_CODENAME}" >/etc/apt/sources.list.d/docker.list
|
||||
apt-get update
|
||||
apt-get install -y \
|
||||
docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
|
||||
fi
|
||||
|
||||
systemctl enable --now docker
|
||||
docker version
|
||||
docker compose version
|
||||
guard_disk docker-engine
|
||||
|
||||
install -d -m 0750 \
|
||||
"${STATE_ROOT}/volumes/cvat_db" \
|
||||
"${STATE_ROOT}/volumes/cvat_data" \
|
||||
"${STATE_ROOT}/volumes/cvat_keys" \
|
||||
"${STATE_ROOT}/volumes/cvat_logs" \
|
||||
"${STATE_ROOT}/volumes/cvat_inmem_db" \
|
||||
"${STATE_ROOT}/volumes/cvat_events_db" \
|
||||
"${STATE_ROOT}/volumes/cvat_cache_db" \
|
||||
"${RUNTIME_ROOT}/source" \
|
||||
"${RUNTIME_ROOT}/reports"
|
||||
|
||||
if [[ ! -d "${CVAT_ROOT}/.git" ]]; then
|
||||
git clone --depth 1 --branch "${CVAT_TAG}" \
|
||||
https://github.com/cvat-ai/cvat.git "${CVAT_ROOT}"
|
||||
fi
|
||||
|
||||
test "$(git -C "${CVAT_ROOT}" describe --tags --exact-match)" = "${CVAT_TAG}"
|
||||
test -f "${OVERRIDE_FILE}"
|
||||
guard_disk cvat-source
|
||||
|
||||
export CVAT_VERSION="${CVAT_TAG}"
|
||||
export CVAT_HOST="localhost"
|
||||
export CVAT_HTTP_PORT="8080"
|
||||
export COMPOSE_PROJECT_NAME="missioncore-cvat"
|
||||
|
||||
compose=(
|
||||
docker compose
|
||||
--project-directory "${CVAT_ROOT}"
|
||||
-f "${CVAT_ROOT}/docker-compose.yml"
|
||||
-f "${OVERRIDE_FILE}"
|
||||
)
|
||||
|
||||
"${compose[@]}" config --quiet
|
||||
mapfile -t images < <("${compose[@]}" config --images | sort -u)
|
||||
for image in "${images[@]}"; do
|
||||
guard_disk "before-pull:${image}"
|
||||
docker pull "${image}"
|
||||
guard_disk "after-pull:${image}"
|
||||
done
|
||||
|
||||
"${compose[@]}" up -d --no-build
|
||||
guard_disk cvat-started
|
||||
|
||||
deadline=$((SECONDS + 600))
|
||||
until curl --fail --silent --show-error http://localhost:8080/api/server/about \
|
||||
>/dev/null; do
|
||||
if (( SECONDS >= deadline )); then
|
||||
"${compose[@]}" ps
|
||||
echo "CVAT did not become ready within 600 seconds" >&2
|
||||
exit 4
|
||||
fi
|
||||
sleep 5
|
||||
done
|
||||
|
||||
report="${RUNTIME_ROOT}/reports/deployment-$(date -u +%Y%m%dT%H%M%SZ).txt"
|
||||
{
|
||||
printf 'cvat_tag=%s\n' "${CVAT_TAG}"
|
||||
printf 'cvat_commit=%s\n' "$(git -C "${CVAT_ROOT}" rev-parse HEAD)"
|
||||
printf 'docker_version=%s\n' "$(docker version --format '{{.Server.Version}}')"
|
||||
printf 'compose_version=%s\n' "$(docker compose version --short)"
|
||||
printf 'free_gib=%s\n' "$(free_gib)"
|
||||
printf 'generated_at_utc=%s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)"
|
||||
docker image inspect "${images[@]}" \
|
||||
--format 'image={{index .RepoTags 0}} id={{.Id}} size={{.Size}}'
|
||||
"${compose[@]}" ps
|
||||
} | tee "${report}"
|
||||
|
||||
printf 'CVAT ready at http://localhost:8080\nreport=%s\n' "${report}"
|
||||
@@ -0,0 +1,69 @@
|
||||
#!/usr/bin/env bash
|
||||
set -Eeuo pipefail
|
||||
|
||||
export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
|
||||
|
||||
readonly FREE_GIB_FLOOR="${FREE_GIB_FLOOR:-360}"
|
||||
readonly WINDOWS_ROOT="/mnt/d/NDC_MISSIONCORE"
|
||||
readonly COMPUTE_ROOT="${WINDOWS_ROOT}/workspace/mission-core-compute"
|
||||
readonly CVAT_ROOT="${WINDOWS_ROOT}/runtime/annotation/cvat/source/cvat-v2.70.0"
|
||||
readonly OVERRIDE_FILE="${COMPUTE_ROOT}/cvat/docker-compose.override.yml"
|
||||
|
||||
if [[ "${EUID}" -ne 0 ]]; then
|
||||
echo "start_cvat_runtime.sh must run as root" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
free_gib="$(df -BG --output=avail "${WINDOWS_ROOT}" | tail -n 1 | tr -dc '0-9')"
|
||||
printf 'disk_guard stage=start free_gib=%s floor_gib=%s\n' \
|
||||
"${free_gib}" "${FREE_GIB_FLOOR}"
|
||||
if (( free_gib < FREE_GIB_FLOOR )); then
|
||||
echo "D: free-space floor crossed; refusing to start CVAT" >&2
|
||||
exit 3
|
||||
fi
|
||||
|
||||
test -f "${CVAT_ROOT}/docker-compose.yml"
|
||||
test -f "${OVERRIDE_FILE}"
|
||||
systemctl start docker
|
||||
|
||||
export CVAT_VERSION="v2.70.0"
|
||||
export CVAT_HOST="localhost"
|
||||
export CVAT_HTTP_PORT="8080"
|
||||
export COMPOSE_PROJECT_NAME="missioncore-cvat"
|
||||
|
||||
compose=(
|
||||
docker compose
|
||||
--project-directory "${CVAT_ROOT}"
|
||||
-f "${CVAT_ROOT}/docker-compose.yml"
|
||||
-f "${OVERRIDE_FILE}"
|
||||
)
|
||||
"${compose[@]}" up -d --no-build
|
||||
|
||||
deadline=$((SECONDS + 180))
|
||||
while true; do
|
||||
about_http="$(
|
||||
curl --silent --output /dev/null --write-out '%{http_code}' \
|
||||
http://localhost:8080/api/server/about || true
|
||||
)"
|
||||
tasks_http="$(
|
||||
curl --silent --output /dev/null --write-out '%{http_code}' \
|
||||
http://localhost:8080/api/tasks || true
|
||||
)"
|
||||
if [[ "${about_http}" == "200" ]] \
|
||||
&& [[ "${tasks_http}" == "200" || "${tasks_http}" == "401" || "${tasks_http}" == "403" ]]; then
|
||||
break
|
||||
fi
|
||||
if (( SECONDS >= deadline )); then
|
||||
"${compose[@]}" ps
|
||||
echo "CVAT did not become ready within 180 seconds" >&2
|
||||
exit 4
|
||||
fi
|
||||
sleep 2
|
||||
done
|
||||
|
||||
printf 'cvat_ready about_http=%s tasks_http=%s free_gib=%s\n' \
|
||||
"${about_http}" "${tasks_http}" "${free_gib}"
|
||||
|
||||
if [[ "${1:-}" == "--keepalive" ]]; then
|
||||
exec sleep infinity
|
||||
fi
|
||||
@@ -0,0 +1,87 @@
|
||||
{
|
||||
"schema_version": "missioncore.e10-integrated-perception-profile/v1",
|
||||
"mode": "full-session-qualification",
|
||||
"source": {
|
||||
"source_id": "sensor.camera.right",
|
||||
"resolution": [800, 600],
|
||||
"calibration_slot": "camera_1",
|
||||
"calibration_sha256": "05f3ad9b38b3a4fc95388a8ec83da83c745e217709e51787b3d5aad0969f6fa9"
|
||||
},
|
||||
"selection": {
|
||||
"required_frame_count": 4489,
|
||||
"required_source_start_frame_index": 0,
|
||||
"required_source_end_frame_index": 4488,
|
||||
"minimum_source_span_seconds": 448.0
|
||||
},
|
||||
"replay": {
|
||||
"speed": 1.0,
|
||||
"detector_queue_capacity": 2,
|
||||
"semantic_queue_capacity": 1,
|
||||
"semantic_sample_every_frames": 5,
|
||||
"semantic_ttl_ms": 750.0
|
||||
},
|
||||
"association": {
|
||||
"box_inset": {
|
||||
"bottom_fraction": 0.02,
|
||||
"horizontal_fraction": 0.05,
|
||||
"top_fraction": 0.04
|
||||
},
|
||||
"depth_cluster_gap_fraction": 0.06,
|
||||
"depth_cluster_minimum_gap_m": 0.5,
|
||||
"group_nms_iou_threshold": 0.55,
|
||||
"maximum_cuboid_span_m": 12.0,
|
||||
"maximum_distance_innovation_fraction": 0.25,
|
||||
"maximum_distance_innovation_m": 1.5,
|
||||
"maximum_oriented_extent_m": {
|
||||
"bicycle": [3.5, 2.0, 2.5],
|
||||
"motorcycle": [3.5, 2.0, 2.5],
|
||||
"person": [1.5, 1.5, 2.8],
|
||||
"vehicle": [6.5, 3.5, 4.0]
|
||||
},
|
||||
"minimum_cuboid_extent_m": 0.15,
|
||||
"minimum_support_points": {
|
||||
"bicycle": 3,
|
||||
"motorcycle": 3,
|
||||
"person": 3,
|
||||
"vehicle": 5
|
||||
},
|
||||
"semantic_ids": {
|
||||
"bicycle": [2],
|
||||
"motorcycle": [3],
|
||||
"person": [1],
|
||||
"vehicle": [4, 5]
|
||||
},
|
||||
"spatial_cluster_radius_m": {
|
||||
"bicycle": 0.9,
|
||||
"motorcycle": 0.9,
|
||||
"person": 0.9,
|
||||
"vehicle": 1.5
|
||||
},
|
||||
"vehicle_labels": ["car", "truck", "bus"],
|
||||
"distance_history_frames": 5
|
||||
},
|
||||
"world_state": {
|
||||
"velocity_history_limit_s": 1.0,
|
||||
"clearance": {
|
||||
"sector_count": 72,
|
||||
"minimum_range_m": 0.5,
|
||||
"maximum_range_m": 30.0,
|
||||
"ground_percentile": 5.0,
|
||||
"minimum_height_above_ground_m": 0.2,
|
||||
"maximum_height_above_ground_m": 3.0,
|
||||
"front_half_angle_degrees": 15.0
|
||||
}
|
||||
},
|
||||
"acceptance": {
|
||||
"detector_minimum_effective_fps": 9.5,
|
||||
"detector_maximum_drop_fraction": 0.0,
|
||||
"maximum_p95_world_state_age_ms": 175.0,
|
||||
"semantic_minimum_effective_fps": 1.8,
|
||||
"semantic_maximum_drop_fraction": 0.05,
|
||||
"semantic_maximum_p95_completion_age_ms": 400.0,
|
||||
"minimum_fresh_semantic_coverage": 0.9,
|
||||
"minimum_lidar_fused_frames": 3500,
|
||||
"minimum_accepted_cuboids": 500,
|
||||
"require_zero_failures": true
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,81 @@
|
||||
{
|
||||
"schema_version": "missioncore.e10-integrated-perception-profile/v1",
|
||||
"mode": "qualification",
|
||||
"source": {
|
||||
"source_id": "sensor.camera.right",
|
||||
"resolution": [800, 600],
|
||||
"calibration_slot": "camera_1",
|
||||
"calibration_sha256": "05f3ad9b38b3a4fc95388a8ec83da83c745e217709e51787b3d5aad0969f6fa9"
|
||||
},
|
||||
"replay": {
|
||||
"speed": 1.0,
|
||||
"detector_queue_capacity": 2,
|
||||
"semantic_queue_capacity": 1,
|
||||
"semantic_sample_every_frames": 5,
|
||||
"semantic_ttl_ms": 750.0
|
||||
},
|
||||
"association": {
|
||||
"box_inset": {
|
||||
"bottom_fraction": 0.02,
|
||||
"horizontal_fraction": 0.05,
|
||||
"top_fraction": 0.04
|
||||
},
|
||||
"depth_cluster_gap_fraction": 0.06,
|
||||
"depth_cluster_minimum_gap_m": 0.5,
|
||||
"group_nms_iou_threshold": 0.55,
|
||||
"maximum_cuboid_span_m": 12.0,
|
||||
"maximum_distance_innovation_fraction": 0.25,
|
||||
"maximum_distance_innovation_m": 1.5,
|
||||
"maximum_oriented_extent_m": {
|
||||
"bicycle": [3.5, 2.0, 2.5],
|
||||
"motorcycle": [3.5, 2.0, 2.5],
|
||||
"person": [1.5, 1.5, 2.8],
|
||||
"vehicle": [6.5, 3.5, 4.0]
|
||||
},
|
||||
"minimum_cuboid_extent_m": 0.15,
|
||||
"minimum_support_points": {
|
||||
"bicycle": 3,
|
||||
"motorcycle": 3,
|
||||
"person": 3,
|
||||
"vehicle": 5
|
||||
},
|
||||
"semantic_ids": {
|
||||
"bicycle": [2],
|
||||
"motorcycle": [3],
|
||||
"person": [1],
|
||||
"vehicle": [4, 5]
|
||||
},
|
||||
"spatial_cluster_radius_m": {
|
||||
"bicycle": 0.9,
|
||||
"motorcycle": 0.9,
|
||||
"person": 0.9,
|
||||
"vehicle": 1.5
|
||||
},
|
||||
"vehicle_labels": ["car", "truck", "bus"],
|
||||
"distance_history_frames": 5
|
||||
},
|
||||
"world_state": {
|
||||
"velocity_history_limit_s": 1.0,
|
||||
"clearance": {
|
||||
"sector_count": 72,
|
||||
"minimum_range_m": 0.5,
|
||||
"maximum_range_m": 30.0,
|
||||
"ground_percentile": 5.0,
|
||||
"minimum_height_above_ground_m": 0.2,
|
||||
"maximum_height_above_ground_m": 3.0,
|
||||
"front_half_angle_degrees": 15.0
|
||||
}
|
||||
},
|
||||
"acceptance": {
|
||||
"detector_minimum_effective_fps": 9.5,
|
||||
"detector_maximum_drop_fraction": 0.005,
|
||||
"maximum_p95_world_state_age_ms": 175.0,
|
||||
"semantic_minimum_effective_fps": 1.8,
|
||||
"semantic_maximum_drop_fraction": 0.05,
|
||||
"semantic_maximum_p95_completion_age_ms": 400.0,
|
||||
"minimum_fresh_semantic_coverage": 0.9,
|
||||
"minimum_lidar_fused_frames": 450,
|
||||
"minimum_accepted_cuboids": 100,
|
||||
"require_zero_failures": true
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,78 @@
|
||||
{
|
||||
"schema_version": "missioncore.e10-integrated-perception-profile/v1",
|
||||
"mode": "semantic-loss-negative-control",
|
||||
"source": {
|
||||
"source_id": "sensor.camera.right",
|
||||
"resolution": [800, 600],
|
||||
"calibration_slot": "camera_1",
|
||||
"calibration_sha256": "05f3ad9b38b3a4fc95388a8ec83da83c745e217709e51787b3d5aad0969f6fa9"
|
||||
},
|
||||
"replay": {
|
||||
"speed": 1.0,
|
||||
"detector_queue_capacity": 2,
|
||||
"semantic_queue_capacity": 1,
|
||||
"semantic_sample_every_frames": 5,
|
||||
"semantic_ttl_ms": 750.0
|
||||
},
|
||||
"semantic_loss": {
|
||||
"stop_after_completed_results": 20
|
||||
},
|
||||
"association": {
|
||||
"box_inset": {
|
||||
"bottom_fraction": 0.02,
|
||||
"horizontal_fraction": 0.05,
|
||||
"top_fraction": 0.04
|
||||
},
|
||||
"depth_cluster_gap_fraction": 0.06,
|
||||
"depth_cluster_minimum_gap_m": 0.5,
|
||||
"group_nms_iou_threshold": 0.55,
|
||||
"maximum_cuboid_span_m": 12.0,
|
||||
"maximum_distance_innovation_fraction": 0.25,
|
||||
"maximum_distance_innovation_m": 1.5,
|
||||
"maximum_oriented_extent_m": {
|
||||
"bicycle": [3.5, 2.0, 2.5],
|
||||
"motorcycle": [3.5, 2.0, 2.5],
|
||||
"person": [1.5, 1.5, 2.8],
|
||||
"vehicle": [6.5, 3.5, 4.0]
|
||||
},
|
||||
"minimum_cuboid_extent_m": 0.15,
|
||||
"minimum_support_points": {
|
||||
"bicycle": 3,
|
||||
"motorcycle": 3,
|
||||
"person": 3,
|
||||
"vehicle": 5
|
||||
},
|
||||
"semantic_ids": {
|
||||
"bicycle": [2],
|
||||
"motorcycle": [3],
|
||||
"person": [1],
|
||||
"vehicle": [4, 5]
|
||||
},
|
||||
"spatial_cluster_radius_m": {
|
||||
"bicycle": 0.9,
|
||||
"motorcycle": 0.9,
|
||||
"person": 0.9,
|
||||
"vehicle": 1.5
|
||||
},
|
||||
"vehicle_labels": ["car", "truck", "bus"],
|
||||
"distance_history_frames": 5
|
||||
},
|
||||
"world_state": {
|
||||
"velocity_history_limit_s": 1.0,
|
||||
"clearance": {
|
||||
"sector_count": 72,
|
||||
"minimum_range_m": 0.5,
|
||||
"maximum_range_m": 30.0,
|
||||
"ground_percentile": 5.0,
|
||||
"minimum_height_above_ground_m": 0.2,
|
||||
"maximum_height_above_ground_m": 3.0,
|
||||
"front_half_angle_degrees": 15.0
|
||||
}
|
||||
},
|
||||
"acceptance": {
|
||||
"detector_minimum_effective_fps": 9.5,
|
||||
"detector_maximum_drop_fraction": 0.005,
|
||||
"maximum_p95_world_state_age_ms": 175.0,
|
||||
"minimum_stale_detector_frames": 450
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,144 @@
|
||||
{
|
||||
"schema_version": "missioncore.e10-integrated-perception-profile/v1",
|
||||
"mode": "pilot",
|
||||
"source": {
|
||||
"source_id": "sensor.camera.right",
|
||||
"resolution": [800, 600],
|
||||
"calibration_slot": "camera_1",
|
||||
"calibration_sha256": "05f3ad9b38b3a4fc95388a8ec83da83c745e217709e51787b3d5aad0969f6fa9"
|
||||
},
|
||||
"replay": {
|
||||
"speed": 1.0,
|
||||
"detector_queue_capacity": 2,
|
||||
"semantic_queue_capacity": 1,
|
||||
"semantic_sample_every_frames": 5,
|
||||
"semantic_ttl_ms": 750.0
|
||||
},
|
||||
"association": {
|
||||
"box_inset": {
|
||||
"bottom_fraction": 0.02,
|
||||
"horizontal_fraction": 0.05,
|
||||
"top_fraction": 0.04
|
||||
},
|
||||
"depth_cluster_gap_fraction": 0.06,
|
||||
"depth_cluster_minimum_gap_m": 0.5,
|
||||
"group_nms_iou_threshold": 0.5,
|
||||
"maximum_cuboid_span_m": 15.0,
|
||||
"maximum_distance_innovation_fraction": 0.25,
|
||||
"maximum_distance_innovation_m": 1.5,
|
||||
"maximum_oriented_extent_m": {
|
||||
"bicycle": [3.5, 2.0, 2.5],
|
||||
"motorcycle": [3.5, 2.0, 2.5],
|
||||
"person": [1.5, 1.5, 2.8],
|
||||
"vehicle": [12.5, 4.0, 4.5]
|
||||
},
|
||||
"minimum_cuboid_extent_m": 0.15,
|
||||
"minimum_support_points": {
|
||||
"bicycle": 4,
|
||||
"motorcycle": 4,
|
||||
"person": 4,
|
||||
"vehicle": 8
|
||||
},
|
||||
"semantic_ids": {
|
||||
"bicycle": [2],
|
||||
"motorcycle": [3],
|
||||
"person": [1],
|
||||
"vehicle": [4, 5]
|
||||
},
|
||||
"spatial_cluster_radius_m": {
|
||||
"bicycle": 0.9,
|
||||
"motorcycle": 0.9,
|
||||
"person": 0.9,
|
||||
"vehicle": 1.5
|
||||
},
|
||||
"vehicle_labels": ["car", "truck", "bus"],
|
||||
"distance_history_frames": 5
|
||||
},
|
||||
"cuboid_completion": {
|
||||
"mode": "class-prior-amodal-v1",
|
||||
"failure_policy": "reject",
|
||||
"classes": {
|
||||
"person": {
|
||||
"nominal_size_m": [0.55, 0.55, 1.72],
|
||||
"minimum_size_m": [0.35, 0.35, 1.3],
|
||||
"maximum_size_m": [1.2, 1.2, 2.3],
|
||||
"support_padding_m": [0.12, 0.12, 0.12]
|
||||
},
|
||||
"bicycle": {
|
||||
"nominal_size_m": [1.8, 0.65, 1.5],
|
||||
"minimum_size_m": [1.2, 0.4, 1.0],
|
||||
"maximum_size_m": [2.5, 1.2, 2.2],
|
||||
"support_padding_m": [0.18, 0.12, 0.12]
|
||||
},
|
||||
"motorcycle": {
|
||||
"nominal_size_m": [2.1, 0.8, 1.45],
|
||||
"minimum_size_m": [1.4, 0.5, 1.0],
|
||||
"maximum_size_m": [3.0, 1.4, 2.2],
|
||||
"support_padding_m": [0.2, 0.14, 0.14]
|
||||
},
|
||||
"car": {
|
||||
"nominal_size_m": [4.5, 1.85, 1.55],
|
||||
"minimum_size_m": [3.2, 1.45, 1.2],
|
||||
"maximum_size_m": [5.8, 2.4, 2.3],
|
||||
"support_padding_m": [0.25, 0.18, 0.15]
|
||||
},
|
||||
"truck": {
|
||||
"nominal_size_m": [7.0, 2.5, 3.0],
|
||||
"minimum_size_m": [4.8, 1.8, 1.8],
|
||||
"maximum_size_m": [12.5, 3.2, 4.2],
|
||||
"support_padding_m": [0.35, 0.22, 0.2]
|
||||
},
|
||||
"bus": {
|
||||
"nominal_size_m": [10.5, 2.55, 3.2],
|
||||
"minimum_size_m": [7.0, 2.1, 2.5],
|
||||
"maximum_size_m": [13.5, 3.2, 4.2],
|
||||
"support_padding_m": [0.4, 0.24, 0.2]
|
||||
}
|
||||
},
|
||||
"ground": {
|
||||
"local_radius_m": 2.5,
|
||||
"lower_percentile": 8.0,
|
||||
"maximum_below_support_m": 1.2,
|
||||
"maximum_above_support_m": 0.15,
|
||||
"fallback_below_support_m": 0.25
|
||||
},
|
||||
"orientation": {
|
||||
"minimum_anisotropy_ratio": 1.35,
|
||||
"face_width_switch_fraction": 1.05
|
||||
},
|
||||
"temporal": {
|
||||
"center_alpha": 0.4,
|
||||
"size_alpha": 0.2,
|
||||
"yaw_alpha": 0.25,
|
||||
"maximum_center_innovation_m": 2.0,
|
||||
"maximum_yaw_innovation_degrees": 55.0,
|
||||
"maximum_idle_s": 1.0,
|
||||
"confirmation_hits": 3
|
||||
},
|
||||
"minimum_support_coverage_fraction": 0.75
|
||||
},
|
||||
"world_state": {
|
||||
"velocity_history_limit_s": 1.0,
|
||||
"clearance": {
|
||||
"sector_count": 72,
|
||||
"minimum_range_m": 0.5,
|
||||
"maximum_range_m": 30.0,
|
||||
"ground_percentile": 5.0,
|
||||
"minimum_height_above_ground_m": 0.2,
|
||||
"maximum_height_above_ground_m": 3.0,
|
||||
"front_half_angle_degrees": 15.0
|
||||
}
|
||||
},
|
||||
"acceptance": {
|
||||
"detector_minimum_effective_fps": 9.5,
|
||||
"detector_maximum_drop_fraction": 0.005,
|
||||
"maximum_p95_world_state_age_ms": 175.0,
|
||||
"semantic_minimum_effective_fps": 1.8,
|
||||
"semantic_maximum_drop_fraction": 0.05,
|
||||
"semantic_maximum_p95_completion_age_ms": 400.0,
|
||||
"minimum_fresh_semantic_coverage": 0.9,
|
||||
"minimum_lidar_fused_frames": 450,
|
||||
"minimum_accepted_cuboids": 100,
|
||||
"require_zero_failures": true
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,150 @@
|
||||
{
|
||||
"schema_version": "missioncore.e10-integrated-perception-profile/v1",
|
||||
"mode": "full-session-qualification",
|
||||
"source": {
|
||||
"source_id": "sensor.camera.right",
|
||||
"resolution": [800, 600],
|
||||
"calibration_slot": "camera_1",
|
||||
"calibration_sha256": "05f3ad9b38b3a4fc95388a8ec83da83c745e217709e51787b3d5aad0969f6fa9"
|
||||
},
|
||||
"selection": {
|
||||
"required_frame_count": 4489,
|
||||
"required_source_start_frame_index": 0,
|
||||
"required_source_end_frame_index": 4488,
|
||||
"minimum_source_span_seconds": 448.0
|
||||
},
|
||||
"replay": {
|
||||
"speed": 1.0,
|
||||
"detector_queue_capacity": 2,
|
||||
"semantic_queue_capacity": 1,
|
||||
"semantic_sample_every_frames": 5,
|
||||
"semantic_ttl_ms": 750.0
|
||||
},
|
||||
"association": {
|
||||
"box_inset": {
|
||||
"bottom_fraction": 0.02,
|
||||
"horizontal_fraction": 0.05,
|
||||
"top_fraction": 0.04
|
||||
},
|
||||
"depth_cluster_gap_fraction": 0.06,
|
||||
"depth_cluster_minimum_gap_m": 0.5,
|
||||
"group_nms_iou_threshold": 0.5,
|
||||
"maximum_cuboid_span_m": 15.0,
|
||||
"maximum_distance_innovation_fraction": 0.25,
|
||||
"maximum_distance_innovation_m": 1.5,
|
||||
"maximum_oriented_extent_m": {
|
||||
"bicycle": [3.5, 2.0, 2.5],
|
||||
"motorcycle": [3.5, 2.0, 2.5],
|
||||
"person": [1.5, 1.5, 2.8],
|
||||
"vehicle": [12.5, 4.0, 4.5]
|
||||
},
|
||||
"minimum_cuboid_extent_m": 0.15,
|
||||
"minimum_support_points": {
|
||||
"bicycle": 4,
|
||||
"motorcycle": 4,
|
||||
"person": 4,
|
||||
"vehicle": 8
|
||||
},
|
||||
"semantic_ids": {
|
||||
"bicycle": [2],
|
||||
"motorcycle": [3],
|
||||
"person": [1],
|
||||
"vehicle": [4, 5]
|
||||
},
|
||||
"spatial_cluster_radius_m": {
|
||||
"bicycle": 0.9,
|
||||
"motorcycle": 0.9,
|
||||
"person": 0.9,
|
||||
"vehicle": 1.5
|
||||
},
|
||||
"vehicle_labels": ["car", "truck", "bus"],
|
||||
"distance_history_frames": 5
|
||||
},
|
||||
"cuboid_completion": {
|
||||
"mode": "class-prior-amodal-v1",
|
||||
"failure_policy": "reject",
|
||||
"classes": {
|
||||
"person": {
|
||||
"nominal_size_m": [0.55, 0.55, 1.72],
|
||||
"minimum_size_m": [0.35, 0.35, 1.3],
|
||||
"maximum_size_m": [1.2, 1.2, 2.3],
|
||||
"support_padding_m": [0.12, 0.12, 0.12]
|
||||
},
|
||||
"bicycle": {
|
||||
"nominal_size_m": [1.8, 0.65, 1.5],
|
||||
"minimum_size_m": [1.2, 0.4, 1.0],
|
||||
"maximum_size_m": [2.5, 1.2, 2.2],
|
||||
"support_padding_m": [0.18, 0.12, 0.12]
|
||||
},
|
||||
"motorcycle": {
|
||||
"nominal_size_m": [2.1, 0.8, 1.45],
|
||||
"minimum_size_m": [1.4, 0.5, 1.0],
|
||||
"maximum_size_m": [3.0, 1.4, 2.2],
|
||||
"support_padding_m": [0.2, 0.14, 0.14]
|
||||
},
|
||||
"car": {
|
||||
"nominal_size_m": [4.5, 1.85, 1.55],
|
||||
"minimum_size_m": [3.2, 1.45, 1.2],
|
||||
"maximum_size_m": [5.8, 2.4, 2.3],
|
||||
"support_padding_m": [0.25, 0.18, 0.15]
|
||||
},
|
||||
"truck": {
|
||||
"nominal_size_m": [7.0, 2.5, 3.0],
|
||||
"minimum_size_m": [4.8, 1.8, 1.8],
|
||||
"maximum_size_m": [12.5, 3.2, 4.2],
|
||||
"support_padding_m": [0.35, 0.22, 0.2]
|
||||
},
|
||||
"bus": {
|
||||
"nominal_size_m": [10.5, 2.55, 3.2],
|
||||
"minimum_size_m": [7.0, 2.1, 2.5],
|
||||
"maximum_size_m": [13.5, 3.2, 4.2],
|
||||
"support_padding_m": [0.4, 0.24, 0.2]
|
||||
}
|
||||
},
|
||||
"ground": {
|
||||
"local_radius_m": 2.5,
|
||||
"lower_percentile": 8.0,
|
||||
"maximum_below_support_m": 1.2,
|
||||
"maximum_above_support_m": 0.15,
|
||||
"fallback_below_support_m": 0.25
|
||||
},
|
||||
"orientation": {
|
||||
"minimum_anisotropy_ratio": 1.35,
|
||||
"face_width_switch_fraction": 1.05
|
||||
},
|
||||
"temporal": {
|
||||
"center_alpha": 0.4,
|
||||
"size_alpha": 0.2,
|
||||
"yaw_alpha": 0.25,
|
||||
"maximum_center_innovation_m": 2.0,
|
||||
"maximum_yaw_innovation_degrees": 55.0,
|
||||
"maximum_idle_s": 1.0,
|
||||
"confirmation_hits": 3
|
||||
},
|
||||
"minimum_support_coverage_fraction": 0.75
|
||||
},
|
||||
"world_state": {
|
||||
"velocity_history_limit_s": 1.0,
|
||||
"clearance": {
|
||||
"sector_count": 72,
|
||||
"minimum_range_m": 0.5,
|
||||
"maximum_range_m": 30.0,
|
||||
"ground_percentile": 5.0,
|
||||
"minimum_height_above_ground_m": 0.2,
|
||||
"maximum_height_above_ground_m": 3.0,
|
||||
"front_half_angle_degrees": 15.0
|
||||
}
|
||||
},
|
||||
"acceptance": {
|
||||
"detector_minimum_effective_fps": 9.5,
|
||||
"detector_maximum_drop_fraction": 0.005,
|
||||
"maximum_p95_world_state_age_ms": 175.0,
|
||||
"semantic_minimum_effective_fps": 1.8,
|
||||
"semantic_maximum_drop_fraction": 0.05,
|
||||
"semantic_maximum_p95_completion_age_ms": 400.0,
|
||||
"minimum_fresh_semantic_coverage": 0.9,
|
||||
"minimum_lidar_fused_frames": 3500,
|
||||
"minimum_accepted_cuboids": 1500,
|
||||
"require_zero_failures": true
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,51 @@
|
||||
{
|
||||
"schema_version": "missioncore.e15-shadow-inference-profile/v1",
|
||||
"mode": "replay-shadow-gate",
|
||||
"authority": {
|
||||
"commands_enabled": false,
|
||||
"navigation_or_safety_accepted": false
|
||||
},
|
||||
"source": {
|
||||
"source_id": "sensor.camera.right",
|
||||
"resolution": [800, 600],
|
||||
"calibration_slot": "camera_1",
|
||||
"calibration_sha256": "05f3ad9b38b3a4fc95388a8ec83da83c745e217709e51787b3d5aad0969f6fa9"
|
||||
},
|
||||
"transport": {
|
||||
"wire_schema": "missioncore.live-perception-wire/v1",
|
||||
"camera_media": "persistent-fmp4-pyav",
|
||||
"pyav_version": "18.0.0",
|
||||
"maximum_media_buffer_bytes": 8388608,
|
||||
"camera_metadata_capacity": 16
|
||||
},
|
||||
"scheduling": {
|
||||
"detector_queue_capacity": 2,
|
||||
"semantic_queue_capacity": 1,
|
||||
"semantic_sample_every_frames": 5,
|
||||
"semantic_ttl_ms": 750.0,
|
||||
"sensor_wait_ms": 90.0
|
||||
},
|
||||
"temporal": {
|
||||
"binding": "nearest-recorded-host-arrival-best-effort",
|
||||
"maximum_lidar_camera_delta_ms": 100.0,
|
||||
"maximum_pose_point_delta_ms": 100.0,
|
||||
"buffer_capacity_per_modality": 32,
|
||||
"retention_seconds": 3.0,
|
||||
"clock_qualification": "not-hardware-synchronized"
|
||||
},
|
||||
"acceptance": {
|
||||
"minimum_camera_frames": 140,
|
||||
"detector_minimum_effective_fps": 9.5,
|
||||
"detector_maximum_drop_fraction": 0.01,
|
||||
"semantic_minimum_effective_fps": 1.8,
|
||||
"semantic_maximum_drop_fraction": 0.05,
|
||||
"semantic_maximum_p95_completion_age_ms": 400.0,
|
||||
"minimum_fresh_semantic_coverage": 0.9,
|
||||
"minimum_fused_fraction": 0.85,
|
||||
"maximum_p95_decode_age_ms": 80.0,
|
||||
"maximum_p95_world_state_age_ms": 200.0,
|
||||
"require_zero_transport_gaps": true,
|
||||
"require_zero_camera_sequence_gaps": true,
|
||||
"require_zero_failures": true
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,283 @@
|
||||
"""Bounded media/runtime primitives for the LAB E15 shadow inference worker."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import threading
|
||||
import time
|
||||
from collections import deque
|
||||
from collections.abc import Callable
|
||||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
|
||||
import numpy as np
|
||||
|
||||
|
||||
class ShadowRuntimeError(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class CameraFragmentMetadata:
|
||||
ingress_sequence: int
|
||||
source_sequence: int
|
||||
captured_at_epoch_ns: int
|
||||
worker_received_monotonic: float
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class DecodedCameraFrame:
|
||||
frame_index: int
|
||||
metadata: CameraFragmentMetadata
|
||||
image: np.ndarray
|
||||
decoded_monotonic: float
|
||||
decode_age_ms: float
|
||||
|
||||
|
||||
class IncrementalMediaBuffer:
|
||||
"""A bounded non-seekable blocking byte source accepted by PyAV."""
|
||||
|
||||
def __init__(self, maximum_buffer_bytes: int) -> None:
|
||||
if maximum_buffer_bytes < 1024:
|
||||
raise ValueError("media buffer bound is too small")
|
||||
self._maximum_buffer_bytes = maximum_buffer_bytes
|
||||
self._buffer = bytearray()
|
||||
self._condition = threading.Condition()
|
||||
self._finished = False
|
||||
self._failure: BaseException | None = None
|
||||
self._bytes_published = 0
|
||||
self._bytes_read = 0
|
||||
self._maximum_depth = 0
|
||||
|
||||
def readable(self) -> bool:
|
||||
return True
|
||||
|
||||
def read(self, size: int = -1) -> bytes:
|
||||
with self._condition:
|
||||
self._condition.wait_for(
|
||||
lambda: bool(self._buffer) or self._finished or self._failure is not None
|
||||
)
|
||||
if self._failure is not None:
|
||||
raise ShadowRuntimeError("incremental media source failed") from self._failure
|
||||
if not self._buffer:
|
||||
return b""
|
||||
count = len(self._buffer) if size < 0 else min(size, len(self._buffer))
|
||||
result = bytes(self._buffer[:count])
|
||||
del self._buffer[:count]
|
||||
self._bytes_read += count
|
||||
self._condition.notify_all()
|
||||
return result
|
||||
|
||||
def append(self, payload: bytes) -> None:
|
||||
if not payload:
|
||||
raise ShadowRuntimeError("empty fMP4 payload is not admitted")
|
||||
with self._condition:
|
||||
if self._finished or self._failure is not None:
|
||||
raise ShadowRuntimeError("incremental media source is closed")
|
||||
if len(self._buffer) + len(payload) > self._maximum_buffer_bytes:
|
||||
failure = ShadowRuntimeError("incremental media buffer exceeded its hard bound")
|
||||
self._failure = failure
|
||||
self._condition.notify_all()
|
||||
raise failure
|
||||
self._buffer.extend(payload)
|
||||
self._bytes_published += len(payload)
|
||||
self._maximum_depth = max(self._maximum_depth, len(self._buffer))
|
||||
self._condition.notify_all()
|
||||
|
||||
def finish(self) -> None:
|
||||
with self._condition:
|
||||
self._finished = True
|
||||
self._condition.notify_all()
|
||||
|
||||
def fail(self, failure: BaseException) -> None:
|
||||
with self._condition:
|
||||
if self._failure is None:
|
||||
self._failure = failure
|
||||
self._condition.notify_all()
|
||||
|
||||
def snapshot(self) -> dict[str, int | bool]:
|
||||
with self._condition:
|
||||
return {
|
||||
"maximum_buffer_bytes": self._maximum_buffer_bytes,
|
||||
"depth_bytes": len(self._buffer),
|
||||
"maximum_depth_bytes": self._maximum_depth,
|
||||
"bytes_published": self._bytes_published,
|
||||
"bytes_read": self._bytes_read,
|
||||
"finished": self._finished,
|
||||
"failed": self._failure is not None,
|
||||
}
|
||||
|
||||
|
||||
class CameraMetadataQueue:
|
||||
"""Fail-closed segment metadata queue; dropping a fragment would corrupt H.264."""
|
||||
|
||||
def __init__(self, capacity: int) -> None:
|
||||
if capacity < 2:
|
||||
raise ValueError("camera metadata capacity is too small")
|
||||
self._capacity = capacity
|
||||
self._items: deque[CameraFragmentMetadata] = deque()
|
||||
self._condition = threading.Condition()
|
||||
self._finished = False
|
||||
self._published = 0
|
||||
self._consumed = 0
|
||||
self._maximum_depth = 0
|
||||
|
||||
def publish(self, value: CameraFragmentMetadata) -> None:
|
||||
with self._condition:
|
||||
if self._finished:
|
||||
raise ShadowRuntimeError("camera metadata queue is closed")
|
||||
if len(self._items) >= self._capacity:
|
||||
raise ShadowRuntimeError("camera metadata queue exceeded its hard bound")
|
||||
self._items.append(value)
|
||||
self._published += 1
|
||||
self._maximum_depth = max(self._maximum_depth, len(self._items))
|
||||
self._condition.notify_all()
|
||||
|
||||
def take(self) -> CameraFragmentMetadata:
|
||||
with self._condition:
|
||||
self._condition.wait_for(lambda: bool(self._items) or self._finished)
|
||||
if not self._items:
|
||||
raise ShadowRuntimeError("decoder emitted a frame without segment metadata")
|
||||
self._consumed += 1
|
||||
return self._items.popleft()
|
||||
|
||||
def finish(self) -> None:
|
||||
with self._condition:
|
||||
self._finished = True
|
||||
self._condition.notify_all()
|
||||
|
||||
def snapshot(self) -> dict[str, int | bool]:
|
||||
with self._condition:
|
||||
return {
|
||||
"capacity": self._capacity,
|
||||
"depth": len(self._items),
|
||||
"maximum_depth": self._maximum_depth,
|
||||
"published": self._published,
|
||||
"consumed": self._consumed,
|
||||
"finished": self._finished,
|
||||
}
|
||||
|
||||
|
||||
class PersistentFmp4Decoder:
|
||||
"""Decode one committed fMP4 epoch incrementally without writing frames."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
on_frame: Callable[[DecodedCameraFrame], None],
|
||||
width: int = 800,
|
||||
height: int = 600,
|
||||
maximum_buffer_bytes: int = 8 * 1024 * 1024,
|
||||
metadata_capacity: int = 16,
|
||||
) -> None:
|
||||
if width < 1 or height < 1:
|
||||
raise ValueError("decoder resolution is invalid")
|
||||
self._on_frame = on_frame
|
||||
self._width = width
|
||||
self._height = height
|
||||
self._media = IncrementalMediaBuffer(maximum_buffer_bytes)
|
||||
self._metadata = CameraMetadataQueue(metadata_capacity)
|
||||
self._thread = threading.Thread(
|
||||
target=self._decode,
|
||||
name="lab-e15-fmp4-decoder",
|
||||
daemon=True,
|
||||
)
|
||||
self._started = False
|
||||
self._init_seen = False
|
||||
self._last_source_sequence: int | None = None
|
||||
self._decoded_frames = 0
|
||||
self._failure: BaseException | None = None
|
||||
|
||||
def start(self) -> None:
|
||||
if self._started:
|
||||
raise ShadowRuntimeError("fMP4 decoder was already started")
|
||||
self._started = True
|
||||
self._thread.start()
|
||||
|
||||
def feed_init(self, payload: bytes) -> None:
|
||||
if not self._started or self._init_seen or self._last_source_sequence is not None:
|
||||
raise ShadowRuntimeError("camera init ordering is invalid")
|
||||
self._init_seen = True
|
||||
self._media.append(payload)
|
||||
|
||||
def feed_segment(self, metadata: CameraFragmentMetadata, payload: bytes) -> None:
|
||||
if not self._init_seen:
|
||||
raise ShadowRuntimeError("camera media arrived before init")
|
||||
expected = 1 if self._last_source_sequence is None else self._last_source_sequence + 1
|
||||
if metadata.source_sequence != expected:
|
||||
raise ShadowRuntimeError(
|
||||
f"camera source sequence gap: expected {expected}, got {metadata.source_sequence}"
|
||||
)
|
||||
self._metadata.publish(metadata)
|
||||
try:
|
||||
self._media.append(payload)
|
||||
except BaseException as exc:
|
||||
self._media.fail(exc)
|
||||
self._metadata.finish()
|
||||
raise
|
||||
self._last_source_sequence = metadata.source_sequence
|
||||
|
||||
def finish_input(self) -> None:
|
||||
self._media.finish()
|
||||
self._metadata.finish()
|
||||
|
||||
def join(self, timeout_seconds: float = 30.0) -> None:
|
||||
self._thread.join(timeout=timeout_seconds)
|
||||
if self._thread.is_alive():
|
||||
raise ShadowRuntimeError("persistent fMP4 decoder did not stop")
|
||||
if self._failure is not None:
|
||||
raise ShadowRuntimeError("persistent fMP4 decoder failed") from self._failure
|
||||
metadata = self._metadata.snapshot()
|
||||
if metadata["published"] != metadata["consumed"] or metadata["depth"] != 0:
|
||||
raise ShadowRuntimeError("camera segment/frame accounting differs")
|
||||
|
||||
def snapshot(self) -> dict[str, Any]:
|
||||
return {
|
||||
"init_seen": self._init_seen,
|
||||
"last_source_sequence": self._last_source_sequence,
|
||||
"decoded_frames": self._decoded_frames,
|
||||
"failed": self._failure is not None,
|
||||
"media": self._media.snapshot(),
|
||||
"metadata": self._metadata.snapshot(),
|
||||
}
|
||||
|
||||
def _decode(self) -> None:
|
||||
try:
|
||||
import av
|
||||
|
||||
container = av.open(
|
||||
self._media,
|
||||
mode="r",
|
||||
format="mp4",
|
||||
options={"probesize": "32", "analyzeduration": "0"},
|
||||
)
|
||||
try:
|
||||
for frame in container.decode(video=0):
|
||||
metadata = self._metadata.take()
|
||||
image = frame.to_ndarray(format="rgb24")
|
||||
if image.shape != (self._height, self._width, 3):
|
||||
raise ShadowRuntimeError("decoded camera resolution changed")
|
||||
image.setflags(write=False)
|
||||
decoded_monotonic = time.perf_counter()
|
||||
self._on_frame(
|
||||
DecodedCameraFrame(
|
||||
frame_index=self._decoded_frames,
|
||||
metadata=metadata,
|
||||
image=image,
|
||||
decoded_monotonic=decoded_monotonic,
|
||||
decode_age_ms=max(
|
||||
0.0,
|
||||
(
|
||||
decoded_monotonic
|
||||
- metadata.worker_received_monotonic
|
||||
)
|
||||
* 1000,
|
||||
),
|
||||
)
|
||||
)
|
||||
self._decoded_frames += 1
|
||||
finally:
|
||||
container.close()
|
||||
except BaseException as exc:
|
||||
self._failure = exc
|
||||
self._media.fail(exc)
|
||||
self._metadata.finish()
|
||||
@@ -0,0 +1,145 @@
|
||||
{
|
||||
"schema_version": "missioncore.k1-e3-rectified-segmentation-profile/v1",
|
||||
"profile_id": "k1-camera1-kb4-cubemap5-eomt-cityscapes/v1",
|
||||
"source": {
|
||||
"source_id": "sensor.camera.right",
|
||||
"calibration_slot": "camera_1",
|
||||
"calibration_sha256": "05f3ad9b38b3a4fc95388a8ec83da83c745e217709e51787b3d5aad0969f6fa9",
|
||||
"resolution": [
|
||||
800,
|
||||
600
|
||||
],
|
||||
"intrinsic_fx_fy_cx_cy": [
|
||||
194.59817287616025,
|
||||
194.57531427932872,
|
||||
396.31861150187996,
|
||||
301.49644357408005
|
||||
],
|
||||
"distortion_kb4": [
|
||||
-0.023164451386679667,
|
||||
-0.0014974198594105452,
|
||||
-0.001039213149441563,
|
||||
-0.000035237331915978814
|
||||
]
|
||||
},
|
||||
"variants": [
|
||||
"eomt-fisheye-mask",
|
||||
"eomt-fisheye-mask-clahe",
|
||||
"eomt-kb4-cubemap5-clahe"
|
||||
],
|
||||
"rectification": {
|
||||
"projection": "five-perspective-gnomonic/v1",
|
||||
"tile_size": 768,
|
||||
"horizontal_fov_degrees": 100.0,
|
||||
"vertical_fov_degrees": 100.0,
|
||||
"tile_selection": "maximum-optical-axis-cosine/v1",
|
||||
"rgb_sampling": "opencv-remap-linear",
|
||||
"label_sampling": "nearest",
|
||||
"tiles": [
|
||||
{
|
||||
"name": "front",
|
||||
"yaw_degrees": 0.0,
|
||||
"pitch_degrees": 0.0
|
||||
},
|
||||
{
|
||||
"name": "left",
|
||||
"yaw_degrees": -90.0,
|
||||
"pitch_degrees": 0.0
|
||||
},
|
||||
{
|
||||
"name": "right",
|
||||
"yaw_degrees": 90.0,
|
||||
"pitch_degrees": 0.0
|
||||
},
|
||||
{
|
||||
"name": "up",
|
||||
"yaw_degrees": 0.0,
|
||||
"pitch_degrees": 90.0
|
||||
},
|
||||
{
|
||||
"name": "down",
|
||||
"yaw_degrees": 0.0,
|
||||
"pitch_degrees": -90.0
|
||||
}
|
||||
]
|
||||
},
|
||||
"contrast": {
|
||||
"method": "clahe-lab-luminance",
|
||||
"clip_limit": 2.0,
|
||||
"tile_grid_size": [
|
||||
8,
|
||||
8
|
||||
]
|
||||
},
|
||||
"model": {
|
||||
"id": "tue-mps/cityscapes_semantic_eomt_large_1024",
|
||||
"revision": "8d6b6d1a3f7b50d441afd7d247c2ed10db186e8f",
|
||||
"architecture": "EomtForUniversalSegmentation",
|
||||
"precision": "fp16-autocast",
|
||||
"batch_size": 1,
|
||||
"files": {
|
||||
"config.json": {
|
||||
"bytes": 1575,
|
||||
"sha256": "7f4aa94fa4e43c0dbd79a5420edb511120aef62bd82bfbcbcece79948286a650"
|
||||
},
|
||||
"preprocessor_config.json": {
|
||||
"bytes": 666,
|
||||
"sha256": "97e2fbf7f0bdba2cfc90251c5133bae9c27ddc9c4410509f40670be2332854e7"
|
||||
},
|
||||
"model.safetensors": {
|
||||
"bytes": 1276175488,
|
||||
"sha256": "c265da9a74f58f5c3f4826d23ca4ca78beac0b106cca5842beca61580de5b782"
|
||||
}
|
||||
}
|
||||
},
|
||||
"opencv": {
|
||||
"distribution": "opencv-python-headless",
|
||||
"version": "4.13.0.92",
|
||||
"runtime_version": "4.13.0",
|
||||
"wheel": {
|
||||
"filename": "opencv_python_headless-4.13.0.92-cp37-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl",
|
||||
"bytes": 56016764,
|
||||
"sha256": "0525a3d2c0b46c611e2130b5fdebc94cf404845d8fa64d2f3a3b679572a5bd22",
|
||||
"url": "https://files.pythonhosted.org/packages/92/ce/bd17ff5772938267fd49716e94ca24f616ff4cb1ff4c6be13085108037be/opencv_python_headless-4.13.0.92-cp37-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl"
|
||||
}
|
||||
},
|
||||
"target_taxonomy": {
|
||||
"0": "outside_valid_fov",
|
||||
"1": "person",
|
||||
"2": "bicycle",
|
||||
"3": "motorcycle",
|
||||
"4": "car",
|
||||
"5": "heavy_vehicle",
|
||||
"6": "building_structure",
|
||||
"7": "paved_road",
|
||||
"8": "sidewalk_curb",
|
||||
"9": "ground_dirt",
|
||||
"10": "grass_low_vegetation",
|
||||
"11": "tree_woody_vegetation",
|
||||
"12": "sky",
|
||||
"13": "static_obstacle",
|
||||
"14": "animal",
|
||||
"15": "other_background"
|
||||
},
|
||||
"cityscapes_to_target": {
|
||||
"road": 7,
|
||||
"sidewalk": 8,
|
||||
"building": 6,
|
||||
"wall": 6,
|
||||
"fence": 13,
|
||||
"pole": 13,
|
||||
"traffic light": 13,
|
||||
"traffic sign": 13,
|
||||
"vegetation": 11,
|
||||
"terrain": 10,
|
||||
"sky": 12,
|
||||
"person": 1,
|
||||
"rider": 1,
|
||||
"car": 4,
|
||||
"truck": 5,
|
||||
"bus": 5,
|
||||
"train": 5,
|
||||
"motorcycle": 3,
|
||||
"bicycle": 2
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,53 @@
|
||||
{
|
||||
"schema_version": "missioncore.e5-tracking-profile/v1",
|
||||
"source": {
|
||||
"source_id": "sensor.camera.right",
|
||||
"resolution": [800, 600],
|
||||
"calibration_slot": "camera_1",
|
||||
"calibration_sha256": "05f3ad9b38b3a4fc95388a8ec83da83c745e217709e51787b3d5aad0969f6fa9"
|
||||
},
|
||||
"model": {
|
||||
"id": "yolox_s",
|
||||
"version": 1,
|
||||
"architecture": "YOLOX-S",
|
||||
"source": "Megvii-BaseDetection/YOLOX release 0.1.1rc0",
|
||||
"license": "Apache-2.0",
|
||||
"model_sha256": "c5c2d13e59ae883e6af3b45daea64af4833a4951c92d116ec270d9ddbe998063",
|
||||
"config_sha256": "5795c737a7935a655961b069e8404d336d891f9762fb6dffb93956a076479604",
|
||||
"input_name": "images",
|
||||
"output_name": "output",
|
||||
"input_shape": [1, 3, 640, 640],
|
||||
"classes": "COCO-80"
|
||||
},
|
||||
"preprocessing": {
|
||||
"color_order": "BGR",
|
||||
"resize": "bilinear-letterbox-top-left",
|
||||
"pad_value": 114,
|
||||
"valid_fov_fill_value": 114
|
||||
},
|
||||
"detection": {
|
||||
"minimum_score": 0.1,
|
||||
"nms_iou_threshold": 0.45,
|
||||
"nms_containment_threshold": 0.8,
|
||||
"target_class_ids": [0, 1, 2, 3, 5, 7],
|
||||
"minimum_box_area_pixels": 64.0,
|
||||
"maximum_box_area_fraction": 0.5,
|
||||
"minimum_valid_fov_fraction": 0.5,
|
||||
"require_center_inside_valid_fov": true
|
||||
},
|
||||
"tracking": {
|
||||
"algorithm": "bytetrack-style-two-stage-iou/v1",
|
||||
"high_score_threshold": 0.25,
|
||||
"new_track_threshold": 0.25,
|
||||
"primary_match_iou": 0.2,
|
||||
"secondary_match_iou": 0.1,
|
||||
"lost_track_buffer_frames": 15,
|
||||
"minimum_confirmed_hits": 2,
|
||||
"assignment_solver": "scipy-linear-sum-assignment"
|
||||
},
|
||||
"overlay": {
|
||||
"line_width": 3,
|
||||
"trail_length": 24,
|
||||
"show_unconfirmed": false
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,64 @@
|
||||
{
|
||||
"schema_version": "missioncore.e8-realtime-tracking-profile/v1",
|
||||
"mode": "overload-negative-control",
|
||||
"source": {
|
||||
"source_id": "sensor.camera.right",
|
||||
"resolution": [800, 600],
|
||||
"calibration_slot": "camera_1",
|
||||
"calibration_sha256": "05f3ad9b38b3a4fc95388a8ec83da83c745e217709e51787b3d5aad0969f6fa9"
|
||||
},
|
||||
"model": {
|
||||
"id": "yolox_s",
|
||||
"version": 1,
|
||||
"architecture": "YOLOX-S",
|
||||
"source": "Megvii-BaseDetection/YOLOX release 0.1.1rc0",
|
||||
"license": "Apache-2.0",
|
||||
"model_sha256": "c5c2d13e59ae883e6af3b45daea64af4833a4951c92d116ec270d9ddbe998063",
|
||||
"config_sha256": "5795c737a7935a655961b069e8404d336d891f9762fb6dffb93956a076479604",
|
||||
"input_name": "images",
|
||||
"output_name": "output",
|
||||
"input_shape": [1, 3, 640, 640],
|
||||
"classes": "COCO-80"
|
||||
},
|
||||
"preprocessing": {
|
||||
"color_order": "BGR",
|
||||
"resize": "bilinear-letterbox-top-left",
|
||||
"pad_value": 114,
|
||||
"valid_fov_fill_value": 114
|
||||
},
|
||||
"detection": {
|
||||
"minimum_score": 0.1,
|
||||
"nms_iou_threshold": 0.45,
|
||||
"nms_containment_threshold": 0.8,
|
||||
"target_class_ids": [0, 1, 2, 3, 5, 7],
|
||||
"minimum_box_area_pixels": 64.0,
|
||||
"maximum_box_area_fraction": 0.5,
|
||||
"minimum_valid_fov_fraction": 0.5,
|
||||
"require_center_inside_valid_fov": true
|
||||
},
|
||||
"tracking": {
|
||||
"algorithm": "bytetrack-style-two-stage-iou/v1",
|
||||
"high_score_threshold": 0.25,
|
||||
"new_track_threshold": 0.25,
|
||||
"primary_match_iou": 0.2,
|
||||
"secondary_match_iou": 0.1,
|
||||
"lost_track_buffer_frames": 15,
|
||||
"minimum_confirmed_hits": 2,
|
||||
"assignment_solver": "scipy-linear-sum-assignment"
|
||||
},
|
||||
"realtime": {
|
||||
"queue_policy": "bounded-latest-wins",
|
||||
"queue_capacity": 2,
|
||||
"speed": 1.0,
|
||||
"consumer_delay_ms": 140.0,
|
||||
"stale_after_ms": 150.0,
|
||||
"unavailable_after_ms": 500.0
|
||||
},
|
||||
"acceptance": {
|
||||
"minimum_effective_fps": 0.0,
|
||||
"maximum_drop_fraction": 1.0,
|
||||
"maximum_p95_result_age_ms": 5000.0,
|
||||
"require_zero_failures": true,
|
||||
"expect_overload": true
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,64 @@
|
||||
{
|
||||
"schema_version": "missioncore.e8-realtime-tracking-profile/v1",
|
||||
"mode": "qualification",
|
||||
"source": {
|
||||
"source_id": "sensor.camera.right",
|
||||
"resolution": [800, 600],
|
||||
"calibration_slot": "camera_1",
|
||||
"calibration_sha256": "05f3ad9b38b3a4fc95388a8ec83da83c745e217709e51787b3d5aad0969f6fa9"
|
||||
},
|
||||
"model": {
|
||||
"id": "yolox_s",
|
||||
"version": 1,
|
||||
"architecture": "YOLOX-S",
|
||||
"source": "Megvii-BaseDetection/YOLOX release 0.1.1rc0",
|
||||
"license": "Apache-2.0",
|
||||
"model_sha256": "c5c2d13e59ae883e6af3b45daea64af4833a4951c92d116ec270d9ddbe998063",
|
||||
"config_sha256": "5795c737a7935a655961b069e8404d336d891f9762fb6dffb93956a076479604",
|
||||
"input_name": "images",
|
||||
"output_name": "output",
|
||||
"input_shape": [1, 3, 640, 640],
|
||||
"classes": "COCO-80"
|
||||
},
|
||||
"preprocessing": {
|
||||
"color_order": "BGR",
|
||||
"resize": "bilinear-letterbox-top-left",
|
||||
"pad_value": 114,
|
||||
"valid_fov_fill_value": 114
|
||||
},
|
||||
"detection": {
|
||||
"minimum_score": 0.1,
|
||||
"nms_iou_threshold": 0.45,
|
||||
"nms_containment_threshold": 0.8,
|
||||
"target_class_ids": [0, 1, 2, 3, 5, 7],
|
||||
"minimum_box_area_pixels": 64.0,
|
||||
"maximum_box_area_fraction": 0.5,
|
||||
"minimum_valid_fov_fraction": 0.5,
|
||||
"require_center_inside_valid_fov": true
|
||||
},
|
||||
"tracking": {
|
||||
"algorithm": "bytetrack-style-two-stage-iou/v1",
|
||||
"high_score_threshold": 0.25,
|
||||
"new_track_threshold": 0.25,
|
||||
"primary_match_iou": 0.2,
|
||||
"secondary_match_iou": 0.1,
|
||||
"lost_track_buffer_frames": 15,
|
||||
"minimum_confirmed_hits": 2,
|
||||
"assignment_solver": "scipy-linear-sum-assignment"
|
||||
},
|
||||
"realtime": {
|
||||
"queue_policy": "bounded-latest-wins",
|
||||
"queue_capacity": 2,
|
||||
"speed": 1.0,
|
||||
"consumer_delay_ms": 0.0,
|
||||
"stale_after_ms": 150.0,
|
||||
"unavailable_after_ms": 500.0
|
||||
},
|
||||
"acceptance": {
|
||||
"minimum_effective_fps": 9.5,
|
||||
"maximum_drop_fraction": 0.005,
|
||||
"maximum_p95_result_age_ms": 100.0,
|
||||
"require_zero_failures": true,
|
||||
"expect_overload": false
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,27 @@
|
||||
{
|
||||
"schema_version": "missioncore.e9-multirate-perception-profile/v1",
|
||||
"mode": "qualification",
|
||||
"source": {
|
||||
"source_id": "sensor.camera.right",
|
||||
"resolution": [800, 600],
|
||||
"calibration_slot": "camera_1",
|
||||
"calibration_sha256": "05f3ad9b38b3a4fc95388a8ec83da83c745e217709e51787b3d5aad0969f6fa9"
|
||||
},
|
||||
"replay": {
|
||||
"speed": 1.0,
|
||||
"detector_queue_capacity": 2,
|
||||
"semantic_queue_capacity": 1,
|
||||
"semantic_sample_every_frames": 5,
|
||||
"semantic_ttl_ms": 750.0
|
||||
},
|
||||
"acceptance": {
|
||||
"detector_minimum_effective_fps": 9.5,
|
||||
"detector_maximum_drop_fraction": 0.005,
|
||||
"detector_maximum_p95_result_age_ms": 100.0,
|
||||
"semantic_minimum_effective_fps": 1.8,
|
||||
"semantic_maximum_drop_fraction": 0.05,
|
||||
"semantic_maximum_p95_completion_age_ms": 350.0,
|
||||
"minimum_fresh_semantic_coverage": 0.9,
|
||||
"require_zero_failures": true
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,27 @@
|
||||
{
|
||||
"schema_version": "missioncore.e9-multirate-perception-profile/v1",
|
||||
"mode": "qualification",
|
||||
"source": {
|
||||
"source_id": "sensor.camera.right",
|
||||
"resolution": [800, 600],
|
||||
"calibration_slot": "camera_1",
|
||||
"calibration_sha256": "05f3ad9b38b3a4fc95388a8ec83da83c745e217709e51787b3d5aad0969f6fa9"
|
||||
},
|
||||
"replay": {
|
||||
"speed": 1.0,
|
||||
"detector_queue_capacity": 2,
|
||||
"semantic_queue_capacity": 1,
|
||||
"semantic_sample_every_frames": 5,
|
||||
"semantic_ttl_ms": 750.0
|
||||
},
|
||||
"acceptance": {
|
||||
"detector_minimum_effective_fps": 9.5,
|
||||
"detector_maximum_drop_fraction": 0.005,
|
||||
"detector_maximum_p95_result_age_ms": 150.0,
|
||||
"semantic_minimum_effective_fps": 1.8,
|
||||
"semantic_maximum_drop_fraction": 0.05,
|
||||
"semantic_maximum_p95_completion_age_ms": 350.0,
|
||||
"minimum_fresh_semantic_coverage": 0.9,
|
||||
"require_zero_failures": true
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,976 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Run camera inference, LiDAR fusion and world-state publication in one paced loop."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import importlib.metadata
|
||||
import json
|
||||
import os
|
||||
import platform
|
||||
import resource
|
||||
import shutil
|
||||
import threading
|
||||
import time
|
||||
from collections import Counter
|
||||
from dataclasses import dataclass
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import numpy as np
|
||||
import run_e9_multirate_perception as e9
|
||||
from e10_fusion_runtime import (
|
||||
CuboidCompletionTracker,
|
||||
LidarReplayPack,
|
||||
WorldStateProjector,
|
||||
_completion_profile,
|
||||
canonical_json,
|
||||
clearance,
|
||||
distance_history,
|
||||
fuse_tracks,
|
||||
fusion_document,
|
||||
project_points,
|
||||
sha256,
|
||||
)
|
||||
from run_e4_full_session_segmentation import TARGET_CLASS_COUNT, _dependency_manifest, _load_model
|
||||
from run_e4_full_session_segmentation import _profile as read_semantic_profile
|
||||
from run_e4_full_session_segmentation import _validate_source as validate_semantic_source
|
||||
from run_e5_instance_tracking import (
|
||||
TwoStageTracker,
|
||||
_detections,
|
||||
_infer,
|
||||
_load_valid_fov,
|
||||
_preprocess,
|
||||
_read_timeline,
|
||||
_track_document,
|
||||
_validate_source,
|
||||
_verify_model,
|
||||
)
|
||||
from run_e8_realtime_tracking import LatestWinsQueue
|
||||
from run_e8_realtime_tracking import _read_profile as read_detector_profile
|
||||
from run_recorded_perception_epoch import (
|
||||
_artifact,
|
||||
_GpuTelemetry,
|
||||
_percentiles,
|
||||
_valid_sha256,
|
||||
_validate_job,
|
||||
_write_json,
|
||||
)
|
||||
|
||||
PROFILE_SCHEMA = "missioncore.e10-integrated-perception-profile/v1"
|
||||
RESULT_SCHEMA = "missioncore.e10-integrated-perception-result/v1"
|
||||
REPORT_SCHEMA = "missioncore.e10-integrated-perception-report/v1"
|
||||
IDENTITY_SCHEMA = "missioncore.e10-integrated-perception-identity/v1"
|
||||
SEMANTIC_SCHEMA = "missioncore.e10-semantic-frame/v1"
|
||||
FUSION_SCHEMA = "missioncore.e10-fusion-frame/v1"
|
||||
WORLD_SCHEMA = "missioncore.live-perception-world-state/v1"
|
||||
PIPELINE_ID = "source-paced-yolox-eomt-kb4-lidar-world-state/v1"
|
||||
|
||||
|
||||
def arguments() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser()
|
||||
commands = parser.add_subparsers(dest="command", required=True)
|
||||
for name in ("preflight", "run"):
|
||||
command = commands.add_parser(name)
|
||||
command.add_argument("--job", type=Path, required=True)
|
||||
command.add_argument("--profile", type=Path, required=True)
|
||||
command.add_argument("--detector-profile", type=Path, required=True)
|
||||
command.add_argument("--semantic-profile", type=Path, required=True)
|
||||
command.add_argument("--valid-fov-root", type=Path, required=True)
|
||||
command.add_argument("--model-root", type=Path, required=True)
|
||||
command.add_argument("--cache", type=Path, required=True)
|
||||
command.add_argument("--environment", type=Path, required=True)
|
||||
command.add_argument("--lidar-pack", type=Path, required=True)
|
||||
if name == "run":
|
||||
command.add_argument("--frames", type=Path, required=True)
|
||||
command.add_argument("--timeline", type=Path, required=True)
|
||||
command.add_argument("--output", type=Path, required=True)
|
||||
command.add_argument("--triton-url", required=True)
|
||||
command.add_argument("--free-bytes-floor", type=int, default=0)
|
||||
command.add_argument("--orchestrator-sha256", required=True)
|
||||
command.add_argument("--container-image", required=True)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def read_object(path: Path) -> dict[str, Any]:
|
||||
value = json.loads(path.read_text(encoding="utf-8-sig"))
|
||||
if not isinstance(value, dict):
|
||||
raise RuntimeError(f"JSON root is not an object: {path}")
|
||||
return value
|
||||
|
||||
|
||||
def read_profile(path: Path) -> tuple[dict[str, Any], str]:
|
||||
resolved = path.resolve(strict=True)
|
||||
profile = read_object(resolved)
|
||||
replay = profile.get("replay")
|
||||
source = profile.get("source")
|
||||
association = profile.get("association")
|
||||
world = profile.get("world_state")
|
||||
acceptance = profile.get("acceptance")
|
||||
if (
|
||||
profile.get("schema_version") != PROFILE_SCHEMA
|
||||
or profile.get("mode")
|
||||
not in {
|
||||
"pilot",
|
||||
"qualification",
|
||||
"full-session-qualification",
|
||||
"semantic-loss-negative-control",
|
||||
}
|
||||
or not all(
|
||||
isinstance(value, dict) for value in (replay, source, association, world, acceptance)
|
||||
)
|
||||
or source.get("source_id") != "sensor.camera.right"
|
||||
or source.get("resolution") != [800, 600]
|
||||
or set(association.get("semantic_ids", {}))
|
||||
!= {"person", "bicycle", "motorcycle", "vehicle"}
|
||||
or not isinstance(world.get("clearance"), dict)
|
||||
):
|
||||
raise RuntimeError("LAB E10 profile contract is invalid")
|
||||
if (
|
||||
not 0.1 <= float(replay.get("speed", 0)) <= 10
|
||||
or not 1 <= int(replay.get("detector_queue_capacity", 0)) <= 8
|
||||
or not 1 <= int(replay.get("semantic_queue_capacity", 0)) <= 4
|
||||
or not 2 <= int(replay.get("semantic_sample_every_frames", 0)) <= 30
|
||||
or not 100 <= float(replay.get("semantic_ttl_ms", 0)) <= 5000
|
||||
):
|
||||
raise RuntimeError("LAB E10 scheduling contract is invalid")
|
||||
if profile["mode"] == "semantic-loss-negative-control":
|
||||
semantic_loss = profile.get("semantic_loss")
|
||||
if (
|
||||
not isinstance(semantic_loss, dict)
|
||||
or not 1 <= int(semantic_loss.get("stop_after_completed_results", 0)) <= 100
|
||||
or not 1 <= int(acceptance.get("minimum_stale_detector_frames", 0)) <= 1_000_000
|
||||
):
|
||||
raise RuntimeError("LAB E10 semantic-loss contract is invalid")
|
||||
if profile["mode"] == "full-session-qualification":
|
||||
selection = profile.get("selection")
|
||||
if (
|
||||
not isinstance(selection, dict)
|
||||
or int(selection.get("required_frame_count", 0)) < 2
|
||||
or int(selection.get("required_source_start_frame_index", -1)) != 0
|
||||
or int(selection.get("required_source_end_frame_index", -1))
|
||||
!= int(selection["required_frame_count"]) - 1
|
||||
or float(selection.get("minimum_source_span_seconds", 0)) <= 0
|
||||
):
|
||||
raise RuntimeError("LAB E10 full-session selection contract is invalid")
|
||||
completion = profile.get("cuboid_completion")
|
||||
if completion is not None:
|
||||
if not isinstance(completion, dict):
|
||||
raise RuntimeError("LAB E13 cuboid completion contract is invalid")
|
||||
_completion_profile(completion)
|
||||
return profile, sha256(resolved)
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class SemanticResult:
|
||||
frame_index: int
|
||||
source_frame_index: int
|
||||
session_seconds: float
|
||||
completion_age_ms: float
|
||||
completed_monotonic: float
|
||||
mask: np.ndarray
|
||||
mask_sha256: str
|
||||
class_pixels: dict[str, int]
|
||||
|
||||
|
||||
class LatestSemantic:
|
||||
def __init__(self) -> None:
|
||||
self.lock = threading.Lock()
|
||||
self.value: SemanticResult | None = None
|
||||
|
||||
def publish(self, value: SemanticResult) -> None:
|
||||
with self.lock:
|
||||
if self.value is not None and value.frame_index <= self.value.frame_index:
|
||||
raise RuntimeError("LAB E10 semantic results are not monotonic")
|
||||
self.value = value
|
||||
|
||||
def snapshot(self) -> SemanticResult | None:
|
||||
with self.lock:
|
||||
return self.value
|
||||
|
||||
|
||||
def semantic_worker(
|
||||
*,
|
||||
queue: LatestWinsQueue,
|
||||
latest: LatestSemantic,
|
||||
valid_mask: np.ndarray,
|
||||
target_lut: np.ndarray,
|
||||
target_names: dict[int, str],
|
||||
infer: Any,
|
||||
latency: dict[str, list[float]],
|
||||
completed: list[SemanticResult],
|
||||
failures: list[BaseException],
|
||||
stop_after_results: int | None,
|
||||
) -> None:
|
||||
try:
|
||||
while (envelope := queue.take()) is not None:
|
||||
started = time.perf_counter()
|
||||
latency["queue_wait_ms"].append(max(0.0, (started - envelope.decoded_monotonic) * 1000))
|
||||
fill_started = time.perf_counter()
|
||||
model_input = np.where(valid_mask[..., None], envelope.image, 0).astype(np.uint8)
|
||||
latency["valid_fov_fill_ms"].append((time.perf_counter() - fill_started) * 1000)
|
||||
semantic, measured = infer(model_input)
|
||||
if int(semantic.max()) >= len(target_lut):
|
||||
raise RuntimeError("LAB E10 EoMT emitted an unknown category")
|
||||
target = target_lut[semantic].copy()
|
||||
target[~valid_mask] = 0
|
||||
for name, value in measured.items():
|
||||
latency[name].append(float(value))
|
||||
finished = time.perf_counter()
|
||||
age = max(0.0, (finished - envelope.scheduled_monotonic) * 1000)
|
||||
latency["completion_age_ms"].append(age)
|
||||
latency["processing_ms"].append((finished - started) * 1000)
|
||||
counts = np.bincount(target[valid_mask], minlength=TARGET_CLASS_COUNT)
|
||||
result = SemanticResult(
|
||||
frame_index=envelope.frame_index,
|
||||
source_frame_index=int(envelope.timeline["source_frame_index"]),
|
||||
session_seconds=float(envelope.timeline["session_seconds"]),
|
||||
completion_age_ms=age,
|
||||
completed_monotonic=finished,
|
||||
mask=target,
|
||||
mask_sha256=hashlib.sha256(target.tobytes()).hexdigest(),
|
||||
class_pixels={
|
||||
target_names[index]: int(counts[index])
|
||||
for index in range(1, TARGET_CLASS_COUNT)
|
||||
if int(counts[index]) > 0
|
||||
},
|
||||
)
|
||||
completed.append(result)
|
||||
latest.publish(result)
|
||||
if stop_after_results is not None and len(completed) >= stop_after_results:
|
||||
return
|
||||
except BaseException as exc:
|
||||
failures.append(exc)
|
||||
|
||||
|
||||
def semantic_binding(
|
||||
semantic: SemanticResult | None, frame_seconds: float, ttl_ms: float
|
||||
) -> tuple[str, float | None]:
|
||||
if semantic is None:
|
||||
return "unavailable", None
|
||||
age = max(0.0, (frame_seconds - semantic.session_seconds) * 1000)
|
||||
return ("fresh" if age <= ttl_ms else "stale"), age
|
||||
|
||||
|
||||
def preflight(args: argparse.Namespace) -> int:
|
||||
import torch
|
||||
|
||||
job = _validate_job(args.job.resolve(strict=True))
|
||||
profile, profile_sha256 = read_profile(args.profile)
|
||||
detector, detector_sha256 = read_detector_profile(args.detector_profile)
|
||||
semantic, semantic_sha256 = read_semantic_profile(args.semantic_profile)
|
||||
_validate_source(job, detector)
|
||||
validate_semantic_source(job, semantic)
|
||||
if profile["source"] != detector["source"]:
|
||||
raise RuntimeError("LAB E10 detector source binding changed")
|
||||
if profile["source"] != {
|
||||
key: semantic["source"][key]
|
||||
for key in ("source_id", "resolution", "calibration_slot", "calibration_sha256")
|
||||
}:
|
||||
raise RuntimeError("LAB E10 semantic source binding changed")
|
||||
lidar = LidarReplayPack(args.lidar_pack, expected_job_id=job["job_id"])
|
||||
try:
|
||||
if lidar.identity["calibration_sha256"] != profile["source"]["calibration_sha256"]:
|
||||
raise RuntimeError("LAB E10 LiDAR calibration binding changed")
|
||||
if profile["mode"] == "full-session-qualification":
|
||||
selection = profile["selection"]
|
||||
if (
|
||||
lidar.frame_count != int(selection["required_frame_count"])
|
||||
or int(lidar.source_frame_indices[0])
|
||||
!= int(selection["required_source_start_frame_index"])
|
||||
or int(lidar.source_frame_indices[-1])
|
||||
!= int(selection["required_source_end_frame_index"])
|
||||
or float(lidar.session_seconds[-1] - lidar.session_seconds[0])
|
||||
< float(selection["minimum_source_span_seconds"])
|
||||
):
|
||||
raise RuntimeError("LAB E10 full-session LiDAR selection changed")
|
||||
_load_valid_fov(args.valid_fov_root, job, detector)
|
||||
detector_files = _verify_model(detector, args.model_root)
|
||||
dependency = _dependency_manifest(args.environment.resolve(strict=True))
|
||||
if dependency["identity"]["profile_sha256"] != semantic_sha256:
|
||||
raise RuntimeError("LAB E10 semantic dependency identity changed")
|
||||
if not torch.cuda.is_available():
|
||||
raise RuntimeError("CUDA is required for LAB E10")
|
||||
device = torch.device("cuda:0")
|
||||
processor, model, _labels, _lut, semantic_files = _load_model(
|
||||
semantic, args.cache.resolve(strict=True), device
|
||||
)
|
||||
del processor, model, _labels, _lut
|
||||
torch.cuda.empty_cache()
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"state": "preflight-ready",
|
||||
"job_id": job["job_id"],
|
||||
"profile_sha256": profile_sha256,
|
||||
"detector_profile_sha256": detector_sha256,
|
||||
"semantic_profile_sha256": semantic_sha256,
|
||||
"lidar_pack_id": lidar.pack_id,
|
||||
"detector_files": detector_files,
|
||||
"semantic_files": semantic_files,
|
||||
"cuda_device": torch.cuda.get_device_name(),
|
||||
},
|
||||
sort_keys=True,
|
||||
),
|
||||
flush=True,
|
||||
)
|
||||
finally:
|
||||
lidar.close()
|
||||
return 0
|
||||
|
||||
|
||||
def run(args: argparse.Namespace) -> int:
|
||||
import torch
|
||||
import transformers
|
||||
from PIL import Image
|
||||
from scipy.optimize import linear_sum_assignment
|
||||
|
||||
if not _valid_sha256(args.orchestrator_sha256):
|
||||
raise RuntimeError("LAB E10 orchestrator SHA-256 is invalid")
|
||||
job = _validate_job(args.job.resolve(strict=True))
|
||||
profile, profile_sha256 = read_profile(args.profile)
|
||||
detector, detector_sha256 = read_detector_profile(args.detector_profile)
|
||||
semantic, semantic_sha256 = read_semantic_profile(args.semantic_profile)
|
||||
_validate_source(job, detector)
|
||||
validate_semantic_source(job, semantic)
|
||||
valid_mask, valid_fov = _load_valid_fov(args.valid_fov_root, job, detector)
|
||||
detector_files = _verify_model(detector, args.model_root)
|
||||
dependency = _dependency_manifest(args.environment.resolve(strict=True))
|
||||
if dependency["identity"]["profile_sha256"] != semantic_sha256:
|
||||
raise RuntimeError("LAB E10 semantic dependency identity changed")
|
||||
lidar = LidarReplayPack(args.lidar_pack, expected_job_id=job["job_id"])
|
||||
frames = sorted(args.frames.resolve(strict=True).glob("frame-*.png"))
|
||||
timeline_path = args.timeline.resolve(strict=True)
|
||||
timeline = _read_timeline(timeline_path, len(frames))
|
||||
if (
|
||||
len(frames) != lidar.frame_count
|
||||
or [int(row["source_frame_index"]) for row in timeline]
|
||||
!= lidar.source_frame_indices.tolist()
|
||||
or not np.allclose(
|
||||
np.asarray([float(row["session_seconds"]) for row in timeline]),
|
||||
lidar.session_seconds,
|
||||
rtol=0,
|
||||
atol=1e-6,
|
||||
)
|
||||
):
|
||||
lidar.close()
|
||||
raise RuntimeError("LAB E10 camera and LiDAR replay selections differ")
|
||||
if profile["mode"] == "full-session-qualification":
|
||||
selection = profile["selection"]
|
||||
source_span_seconds = float(timeline[-1]["session_seconds"]) - float(
|
||||
timeline[0]["session_seconds"]
|
||||
)
|
||||
if (
|
||||
len(frames) != int(selection["required_frame_count"])
|
||||
or int(timeline[0]["source_frame_index"])
|
||||
!= int(selection["required_source_start_frame_index"])
|
||||
or int(timeline[-1]["source_frame_index"])
|
||||
!= int(selection["required_source_end_frame_index"])
|
||||
or source_span_seconds < float(selection["minimum_source_span_seconds"])
|
||||
):
|
||||
lidar.close()
|
||||
raise RuntimeError("LAB E10 full-session camera selection changed")
|
||||
output = args.output.resolve()
|
||||
if output.exists():
|
||||
raise RuntimeError("LAB E10 output must be absent")
|
||||
output.mkdir(mode=0o700, parents=True, exist_ok=False)
|
||||
assert_disk(output, args.free_bytes_floor, 0)
|
||||
|
||||
device = torch.device("cuda:0")
|
||||
processor, semantic_model, _semantic_labels, target_lut, semantic_files = _load_model(
|
||||
semantic, args.cache.resolve(strict=True), device
|
||||
)
|
||||
target_names = {int(key): str(value) for key, value in semantic["target_taxonomy"].items()}
|
||||
infer_semantic = e9._semantic_infer_factory(processor, semantic_model, device)
|
||||
tracker = TwoStageTracker(detector["tracking"])
|
||||
linear_sum_assignment(np.zeros((1, 1), dtype=np.float64))
|
||||
with Image.open(frames[0]) as opened:
|
||||
warm = np.asarray(opened.convert("RGB"), dtype=np.uint8)
|
||||
_infer(args.triton_url, detector["model"], _preprocess(warm, valid_mask, detector))
|
||||
infer_semantic(np.where(valid_mask[..., None], warm, 0).astype(np.uint8))
|
||||
del warm
|
||||
|
||||
replay = profile["replay"]
|
||||
detector_queue = LatestWinsQueue(int(replay["detector_queue_capacity"]))
|
||||
semantic_queue = LatestWinsQueue(int(replay["semantic_queue_capacity"]))
|
||||
latest = LatestSemantic()
|
||||
completed_semantics: list[SemanticResult] = []
|
||||
producer_errors: list[BaseException] = []
|
||||
semantic_errors: list[BaseException] = []
|
||||
semantic_latency = {
|
||||
name: []
|
||||
for name in (
|
||||
"queue_wait_ms",
|
||||
"valid_fov_fill_ms",
|
||||
"processor_ms",
|
||||
"host_to_device_ms",
|
||||
"forward_ms",
|
||||
"model_postprocess_ms",
|
||||
"processing_ms",
|
||||
"completion_age_ms",
|
||||
)
|
||||
}
|
||||
latency = {
|
||||
name: []
|
||||
for name in (
|
||||
"decode_ms",
|
||||
"queue_wait_ms",
|
||||
"detector_ms",
|
||||
"projection_ms",
|
||||
"association_ms",
|
||||
"clearance_ms",
|
||||
"world_state_ms",
|
||||
"world_state_age_ms",
|
||||
)
|
||||
}
|
||||
status_counts: Counter[str] = Counter()
|
||||
fusion_state_counts: Counter[str] = Counter()
|
||||
rejection_counts: Counter[str] = Counter()
|
||||
accepted_cuboids = 0
|
||||
fused_frames = 0
|
||||
detector_failures = 0
|
||||
fusion_freshness_violations = 0
|
||||
history = distance_history(int(profile["association"]["distance_history_frames"]))
|
||||
projector = WorldStateProjector(float(profile["world_state"]["velocity_history_limit_s"]))
|
||||
completion_tracker = (
|
||||
CuboidCompletionTracker(profile["cuboid_completion"])
|
||||
if "cuboid_completion" in profile
|
||||
else None
|
||||
)
|
||||
semantic_path = output / "semantic-frames.jsonl"
|
||||
fusion_path = output / "fusion-frames.jsonl"
|
||||
world_path = output / "world-state.jsonl"
|
||||
gpu_path = output / "gpu-telemetry.jsonl"
|
||||
support_offsets = [0]
|
||||
support_points: list[np.ndarray] = []
|
||||
support_colors: list[np.ndarray] = []
|
||||
box_offsets = [0]
|
||||
box_centers: list[tuple[float, float, float]] = []
|
||||
box_half_sizes: list[tuple[float, float, float]] = []
|
||||
box_quaternions: list[tuple[float, float, float, float]] = []
|
||||
box_colors: list[tuple[int, int, int, int]] = []
|
||||
frame_times_ns: list[int] = []
|
||||
disk_before = shutil.disk_usage(output).free
|
||||
|
||||
with (
|
||||
semantic_path.open("x", encoding="utf-8", newline="\n") as semantic_stream,
|
||||
fusion_path.open("x", encoding="utf-8", newline="\n") as fusion_stream,
|
||||
world_path.open("x", encoding="utf-8", newline="\n") as world_stream,
|
||||
gpu_path.open("x", encoding="utf-8", newline="\n") as gpu_stream,
|
||||
_GpuTelemetry(gpu_stream, 1.0) as gpu,
|
||||
):
|
||||
semantic_thread = threading.Thread(
|
||||
target=semantic_worker,
|
||||
kwargs={
|
||||
"queue": semantic_queue,
|
||||
"latest": latest,
|
||||
"valid_mask": valid_mask,
|
||||
"target_lut": target_lut,
|
||||
"target_names": target_names,
|
||||
"infer": infer_semantic,
|
||||
"latency": semantic_latency,
|
||||
"completed": completed_semantics,
|
||||
"failures": semantic_errors,
|
||||
"stop_after_results": (
|
||||
int(profile["semantic_loss"]["stop_after_completed_results"])
|
||||
if profile["mode"] == "semantic-loss-negative-control"
|
||||
else None
|
||||
),
|
||||
},
|
||||
name="lab-e10-semantic",
|
||||
daemon=True,
|
||||
)
|
||||
semantic_thread.start()
|
||||
replay_started = time.perf_counter() + 0.25
|
||||
producer = threading.Thread(
|
||||
target=e9._producer,
|
||||
kwargs={
|
||||
"detector_queue": detector_queue,
|
||||
"semantic_queue": semantic_queue,
|
||||
"semantic_stride": int(replay["semantic_sample_every_frames"]),
|
||||
"frame_paths": frames,
|
||||
"timeline_rows": timeline,
|
||||
"replay_started": replay_started,
|
||||
"speed": float(replay["speed"]),
|
||||
"error": producer_errors,
|
||||
},
|
||||
name="lab-e10-source",
|
||||
daemon=True,
|
||||
)
|
||||
producer.start()
|
||||
|
||||
while (envelope := detector_queue.take()) is not None:
|
||||
started = time.perf_counter()
|
||||
latency["decode_ms"].append(envelope.decode_ms)
|
||||
latency["queue_wait_ms"].append(max(0.0, (started - envelope.decoded_monotonic) * 1000))
|
||||
try:
|
||||
detector_started = time.perf_counter()
|
||||
tensor = _preprocess(envelope.image, valid_mask, detector)
|
||||
output_tensor, _request_ms = _infer(args.triton_url, detector["model"], tensor)
|
||||
detections, _rejected = _detections(output_tensor, detector, valid_mask)
|
||||
tracks = tracker.update(detections, envelope.frame_index)
|
||||
latency["detector_ms"].append((time.perf_counter() - detector_started) * 1000)
|
||||
|
||||
frame_seconds = float(envelope.timeline["session_seconds"])
|
||||
current_semantic = latest.snapshot()
|
||||
semantic_status, semantic_age = semantic_binding(
|
||||
current_semantic,
|
||||
frame_seconds,
|
||||
float(replay["semantic_ttl_ms"]),
|
||||
)
|
||||
status_counts[semantic_status] += 1
|
||||
lidar_frame = lidar.frame(envelope.frame_index)
|
||||
fusions = ()
|
||||
points_lidar = np.empty((0, 3), dtype=np.float64)
|
||||
if lidar_frame is None:
|
||||
fusion_state = "depth-unavailable-sync-gate"
|
||||
elif semantic_status != "fresh" or current_semantic is None:
|
||||
fusion_state = f"semantic-{semantic_status}"
|
||||
else:
|
||||
points_map, position, quaternion = lidar_frame
|
||||
projection_started = time.perf_counter()
|
||||
pixels, depths, source_indices, points_lidar = project_points(
|
||||
points_map, position, quaternion, lidar.profile
|
||||
)
|
||||
latency["projection_ms"].append(
|
||||
(time.perf_counter() - projection_started) * 1000
|
||||
)
|
||||
association_started = time.perf_counter()
|
||||
fusions = fuse_tracks(
|
||||
tracks=[_track_document(track) for track in tracks],
|
||||
semantic_map=current_semantic.mask,
|
||||
pixels=pixels,
|
||||
depths=depths,
|
||||
source_indices=source_indices,
|
||||
points_map=points_map,
|
||||
points_lidar=points_lidar,
|
||||
association=profile["association"],
|
||||
distance_history=history,
|
||||
completion_tracker=completion_tracker,
|
||||
sensor_position_map=position,
|
||||
session_seconds=frame_seconds,
|
||||
)
|
||||
latency["association_ms"].append(
|
||||
(time.perf_counter() - association_started) * 1000
|
||||
)
|
||||
fusion_state = "fused"
|
||||
fused_frames += 1
|
||||
if fusion_state == "fused" and semantic_status != "fresh":
|
||||
fusion_freshness_violations += 1
|
||||
fusion_state_counts[fusion_state] += 1
|
||||
accepted = [item for item in fusions if item.cuboid is not None]
|
||||
accepted_cuboids += len(accepted)
|
||||
for item in fusions:
|
||||
rejection_counts[item.status] += 1
|
||||
frame_support = []
|
||||
frame_support_colors = []
|
||||
for item in accepted:
|
||||
color_seed = hashlib.sha256(
|
||||
f"e10:{item.association_group}:{item.track_id}".encode()
|
||||
).digest()
|
||||
color = tuple(64 + value % 176 for value in color_seed[:3])
|
||||
if lidar_frame is None:
|
||||
continue
|
||||
points_map = lidar_frame[0]
|
||||
values = points_map[item.source_indices].astype(np.float32)
|
||||
frame_support.append(values)
|
||||
frame_support_colors.append(
|
||||
np.tile(np.asarray([color], dtype=np.uint8), (values.shape[0], 1))
|
||||
)
|
||||
box_centers.append(item.cuboid.center_map)
|
||||
box_half_sizes.append(item.cuboid.half_size)
|
||||
box_quaternions.append(item.cuboid.quaternion_xyzw)
|
||||
box_colors.append((*color, 88))
|
||||
if frame_support:
|
||||
support = np.concatenate(frame_support)
|
||||
colors = np.concatenate(frame_support_colors)
|
||||
support_points.append(support)
|
||||
support_colors.append(colors)
|
||||
support_offsets.append(support_offsets[-1] + support.shape[0])
|
||||
else:
|
||||
support_offsets.append(support_offsets[-1])
|
||||
box_offsets.append(box_offsets[-1] + len(accepted))
|
||||
clearance_started = time.perf_counter()
|
||||
clearance_state = clearance(points_lidar, profile["world_state"]["clearance"])
|
||||
latency["clearance_ms"].append((time.perf_counter() - clearance_started) * 1000)
|
||||
world_started = time.perf_counter()
|
||||
result_age = max(0.0, (world_started - envelope.scheduled_monotonic) * 1000)
|
||||
if result_age >= 1000:
|
||||
health = "unavailable"
|
||||
elif result_age >= float(profile["acceptance"]["maximum_p95_world_state_age_ms"]):
|
||||
health = "stale"
|
||||
elif fusion_state != "fused":
|
||||
health = "degraded"
|
||||
else:
|
||||
health = "healthy"
|
||||
delivery = {
|
||||
"health": health,
|
||||
"result_age_ms": result_age,
|
||||
"semantic_status": semantic_status,
|
||||
"semantic_source_age_ms": semantic_age,
|
||||
}
|
||||
world = projector.project(
|
||||
frame_index=envelope.frame_index,
|
||||
source_frame_index=int(envelope.timeline["source_frame_index"]),
|
||||
session_seconds=frame_seconds,
|
||||
fusion_state=fusion_state,
|
||||
fusions=fusions,
|
||||
points_lidar=points_lidar,
|
||||
clearance_state=clearance_state,
|
||||
delivery=delivery,
|
||||
)
|
||||
latency["world_state_ms"].append((time.perf_counter() - world_started) * 1000)
|
||||
result_age = max(0.0, (time.perf_counter() - envelope.scheduled_monotonic) * 1000)
|
||||
world["delivery"]["result_age_ms"] = result_age
|
||||
latency["world_state_age_ms"].append(result_age)
|
||||
frame_times_ns.append(round(frame_seconds * 1e9))
|
||||
fusion_stream.write(
|
||||
json.dumps(
|
||||
{
|
||||
"schema_version": FUSION_SCHEMA,
|
||||
"frame_index": envelope.frame_index,
|
||||
"source_frame_index": int(envelope.timeline["source_frame_index"]),
|
||||
"session_seconds": frame_seconds,
|
||||
"fusion_state": fusion_state,
|
||||
"semantic_status": semantic_status,
|
||||
"semantic_source_frame_index": None
|
||||
if current_semantic is None
|
||||
else current_semantic.source_frame_index,
|
||||
"objects": [fusion_document(item) for item in fusions],
|
||||
},
|
||||
sort_keys=True,
|
||||
separators=(",", ":"),
|
||||
allow_nan=False,
|
||||
)
|
||||
+ "\n"
|
||||
)
|
||||
world_stream.write(
|
||||
json.dumps(world, sort_keys=True, separators=(",", ":"), allow_nan=False) + "\n"
|
||||
)
|
||||
except Exception:
|
||||
detector_failures += 1
|
||||
raise
|
||||
consumed = int(detector_queue.snapshot()["consumed"])
|
||||
if consumed % 100 == 0:
|
||||
fusion_stream.flush()
|
||||
world_stream.flush()
|
||||
assert_disk(output, args.free_bytes_floor, consumed)
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"phase": "e10-integrated",
|
||||
"processed": consumed,
|
||||
"detector_dropped": detector_queue.snapshot()["dropped_overflow"],
|
||||
"semantic_processed": semantic_queue.snapshot()["consumed"],
|
||||
"fused_frames": fused_frames,
|
||||
"accepted_cuboids": accepted_cuboids,
|
||||
},
|
||||
sort_keys=True,
|
||||
),
|
||||
flush=True,
|
||||
)
|
||||
producer.join(timeout=5)
|
||||
semantic_thread.join(timeout=30)
|
||||
if producer.is_alive() or producer_errors:
|
||||
raise RuntimeError("LAB E10 producer failed")
|
||||
if semantic_thread.is_alive() or semantic_errors:
|
||||
raise RuntimeError("LAB E10 semantic worker failed")
|
||||
for result in completed_semantics:
|
||||
semantic_stream.write(
|
||||
json.dumps(
|
||||
{
|
||||
"schema_version": SEMANTIC_SCHEMA,
|
||||
"frame_index": result.frame_index,
|
||||
"source_frame_index": result.source_frame_index,
|
||||
"session_seconds": result.session_seconds,
|
||||
"completion_age_ms": result.completion_age_ms,
|
||||
"mask_sha256": result.mask_sha256,
|
||||
"class_pixels": result.class_pixels,
|
||||
},
|
||||
sort_keys=True,
|
||||
separators=(",", ":"),
|
||||
allow_nan=False,
|
||||
)
|
||||
+ "\n"
|
||||
)
|
||||
for stream in (semantic_stream, fusion_stream, world_stream):
|
||||
stream.flush()
|
||||
os.fsync(stream.fileno())
|
||||
|
||||
replay_finished = time.perf_counter()
|
||||
detector_state = detector_queue.snapshot()
|
||||
semantic_state = semantic_queue.snapshot()
|
||||
source_span = (
|
||||
float(timeline[-1]["session_seconds"]) - float(timeline[0]["session_seconds"])
|
||||
) / float(replay["speed"])
|
||||
replay_wall = replay_finished - replay_started
|
||||
detector_fps = int(detector_state["consumed"]) / max(source_span, replay_wall, 1e-9)
|
||||
semantic_fps = int(semantic_state["consumed"]) / max(source_span, replay_wall, 1e-9)
|
||||
scheduled_semantic = ((len(frames) - 1) // int(replay["semantic_sample_every_frames"])) + 1
|
||||
fresh_coverage = status_counts["fresh"] / max(1, int(detector_state["consumed"]))
|
||||
latency_summary = {name: _percentiles(values) for name, values in latency.items()}
|
||||
semantic_summary = {name: _percentiles(values) for name, values in semantic_latency.items()}
|
||||
|
||||
arrays_path = output / "transient-perception.npz"
|
||||
np.savez_compressed(
|
||||
arrays_path,
|
||||
frame_times_ns=np.asarray(frame_times_ns, dtype=np.int64),
|
||||
semantic_frame_indices=np.asarray(
|
||||
[value.frame_index for value in completed_semantics], dtype=np.int64
|
||||
),
|
||||
semantic_masks=np.stack([value.mask for value in completed_semantics]).astype(np.uint8),
|
||||
support_offsets=np.asarray(support_offsets, dtype=np.int64),
|
||||
support_points=np.concatenate(support_points)
|
||||
if support_points
|
||||
else np.empty((0, 3), dtype=np.float32),
|
||||
support_colors=np.concatenate(support_colors)
|
||||
if support_colors
|
||||
else np.empty((0, 3), dtype=np.uint8),
|
||||
box_offsets=np.asarray(box_offsets, dtype=np.int64),
|
||||
box_centers=np.asarray(box_centers, dtype=np.float32).reshape((-1, 3)),
|
||||
box_half_sizes=np.asarray(box_half_sizes, dtype=np.float32).reshape((-1, 3)),
|
||||
box_quaternions=np.asarray(box_quaternions, dtype=np.float32).reshape((-1, 4)),
|
||||
box_colors=np.asarray(box_colors, dtype=np.uint8).reshape((-1, 4)),
|
||||
)
|
||||
acceptance = profile["acceptance"]
|
||||
detector_checks = {
|
||||
"detector_accounting": int(detector_state["consumed"])
|
||||
+ int(detector_state["dropped_overflow"])
|
||||
== len(frames),
|
||||
"detector_minimum_effective_fps": detector_fps
|
||||
>= float(acceptance["detector_minimum_effective_fps"]),
|
||||
"detector_maximum_drop_fraction": int(detector_state["dropped_overflow"]) / len(frames)
|
||||
<= float(acceptance["detector_maximum_drop_fraction"]),
|
||||
"maximum_p95_world_state_age_ms": float(latency_summary["world_state_age_ms"]["p95"])
|
||||
<= float(acceptance["maximum_p95_world_state_age_ms"]),
|
||||
"zero_detector_failures": detector_failures == 0 and not producer_errors,
|
||||
}
|
||||
if profile["mode"] == "semantic-loss-negative-control":
|
||||
stop_after = int(profile["semantic_loss"]["stop_after_completed_results"])
|
||||
checks = {
|
||||
**detector_checks,
|
||||
"semantic_loss_triggered": len(completed_semantics) == stop_after,
|
||||
"semantic_queue_accounting": int(semantic_state["consumed"])
|
||||
+ int(semantic_state["dropped_overflow"])
|
||||
+ int(semantic_state["final_depth"])
|
||||
== scheduled_semantic,
|
||||
"minimum_stale_detector_frames": status_counts["stale"]
|
||||
>= int(acceptance["minimum_stale_detector_frames"]),
|
||||
"no_fusion_with_nonfresh_semantics": fusion_freshness_violations == 0,
|
||||
"semantic_worker_stopped_without_error": not semantic_errors,
|
||||
}
|
||||
else:
|
||||
checks = {
|
||||
**detector_checks,
|
||||
"semantic_accounting": int(semantic_state["consumed"])
|
||||
+ int(semantic_state["dropped_overflow"])
|
||||
== scheduled_semantic,
|
||||
"semantic_minimum_effective_fps": semantic_fps
|
||||
>= float(acceptance["semantic_minimum_effective_fps"]),
|
||||
"semantic_maximum_drop_fraction": int(semantic_state["dropped_overflow"])
|
||||
/ scheduled_semantic
|
||||
<= float(acceptance["semantic_maximum_drop_fraction"]),
|
||||
"semantic_maximum_p95_completion_age_ms": float(
|
||||
semantic_summary["completion_age_ms"]["p95"]
|
||||
)
|
||||
<= float(acceptance["semantic_maximum_p95_completion_age_ms"]),
|
||||
"minimum_fresh_semantic_coverage": fresh_coverage
|
||||
>= float(acceptance["minimum_fresh_semantic_coverage"]),
|
||||
"minimum_lidar_fused_frames": fused_frames
|
||||
>= int(acceptance["minimum_lidar_fused_frames"]),
|
||||
"minimum_accepted_cuboids": accepted_cuboids
|
||||
>= int(acceptance["minimum_accepted_cuboids"]),
|
||||
"zero_semantic_failures": not semantic_errors,
|
||||
}
|
||||
accepted = all(checks.values())
|
||||
identity = {
|
||||
"schema_version": IDENTITY_SCHEMA,
|
||||
"job_id": job["job_id"],
|
||||
"input_sha256": job["input_sha256"],
|
||||
"session_id": job["input"]["session_id"],
|
||||
"source_id": job["input"]["source_id"],
|
||||
"lidar_pack_id": lidar.pack_id,
|
||||
"selection": {
|
||||
"frame_count": len(frames),
|
||||
"source_start_frame_index": timeline[0]["source_frame_index"],
|
||||
"source_end_frame_index": timeline[-1]["source_frame_index"],
|
||||
"timeline_start_seconds": timeline[0]["session_seconds"],
|
||||
"timeline_end_seconds": timeline[-1]["session_seconds"],
|
||||
"timeline_sha256": sha256(timeline_path),
|
||||
},
|
||||
"configuration": {
|
||||
"pipeline": PIPELINE_ID,
|
||||
"profile": profile,
|
||||
"profile_sha256": profile_sha256,
|
||||
"detector_profile_sha256": detector_sha256,
|
||||
"semantic_profile_sha256": semantic_sha256,
|
||||
"runner_sha256": sha256(Path(__file__).resolve(strict=True)),
|
||||
"fusion_runtime_sha256": sha256(
|
||||
Path(__file__).with_name("e10_fusion_runtime.py").resolve(strict=True)
|
||||
),
|
||||
"orchestrator_sha256": args.orchestrator_sha256,
|
||||
"container_image": args.container_image,
|
||||
"valid_fov": valid_fov,
|
||||
},
|
||||
"models": {
|
||||
"detector": detector["model"],
|
||||
"detector_files": detector_files,
|
||||
"semantic": semantic["model"],
|
||||
"semantic_files": semantic_files,
|
||||
},
|
||||
}
|
||||
identity_sha256 = hashlib.sha256(canonical_json(identity)).hexdigest()
|
||||
result_id = f"e10-integrated-perception-{identity_sha256}"
|
||||
metrics = {
|
||||
"source_span_seconds": source_span,
|
||||
"replay_wall_seconds": replay_wall,
|
||||
"detector": {
|
||||
"frames_processed": detector_state["consumed"],
|
||||
"frames_dropped": detector_state["dropped_overflow"],
|
||||
"effective_fps": detector_fps,
|
||||
"queue": detector_state,
|
||||
},
|
||||
"semantic": {
|
||||
"frames_processed": semantic_state["consumed"],
|
||||
"frames_dropped": semantic_state["dropped_overflow"],
|
||||
"effective_fps": semantic_fps,
|
||||
"fresh_coverage": fresh_coverage,
|
||||
"status_counts": dict(status_counts),
|
||||
"latency_ms": semantic_summary,
|
||||
},
|
||||
"fusion": {
|
||||
"fused_frames": fused_frames,
|
||||
"fusion_state_counts": dict(fusion_state_counts),
|
||||
"accepted_cuboids": accepted_cuboids,
|
||||
"rejection_counts": dict(rejection_counts),
|
||||
"freshness_violations": fusion_freshness_violations,
|
||||
},
|
||||
"semantic_loss": {
|
||||
"configured": profile["mode"] == "semantic-loss-negative-control",
|
||||
"stop_after_completed_results": (
|
||||
int(profile["semantic_loss"]["stop_after_completed_results"])
|
||||
if profile["mode"] == "semantic-loss-negative-control"
|
||||
else None
|
||||
),
|
||||
"triggered": (
|
||||
len(completed_semantics)
|
||||
== int(profile["semantic_loss"]["stop_after_completed_results"])
|
||||
if profile["mode"] == "semantic-loss-negative-control"
|
||||
else False
|
||||
),
|
||||
},
|
||||
"latency_ms": latency_summary,
|
||||
"gpu_telemetry": gpu.summary(),
|
||||
"process_peak_rss_mib": resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024,
|
||||
"cuda_peak_memory_allocated_mib": torch.cuda.max_memory_allocated() / 2**20,
|
||||
"cuda_peak_memory_reserved_mib": torch.cuda.max_memory_reserved() / 2**20,
|
||||
"disk": {
|
||||
"free_bytes_before": disk_before,
|
||||
"free_bytes_after": shutil.disk_usage(output).free,
|
||||
"free_bytes_floor": args.free_bytes_floor,
|
||||
},
|
||||
}
|
||||
report = {
|
||||
"schema_version": REPORT_SCHEMA,
|
||||
"result_id": result_id,
|
||||
"created_at_utc": datetime.now(UTC)
|
||||
.isoformat(timespec="milliseconds")
|
||||
.replace("+00:00", "Z"),
|
||||
"state": "accepted" if accepted else "rejected",
|
||||
"ground_truth": False,
|
||||
"identity": identity,
|
||||
"runtime": {
|
||||
"hostname": platform.node(),
|
||||
"python": platform.python_version(),
|
||||
"numpy": np.__version__,
|
||||
"torch": torch.__version__,
|
||||
"transformers": transformers.__version__,
|
||||
"scipy": importlib.metadata.version("scipy"),
|
||||
"gpu": torch.cuda.get_device_name(),
|
||||
},
|
||||
"metrics": metrics,
|
||||
"acceptance": {
|
||||
"accepted": accepted,
|
||||
"checks": checks,
|
||||
"navigation_or_safety_accepted": False,
|
||||
},
|
||||
"limitations": [
|
||||
"Recorded source-paced replay, not direct live K1 transport.",
|
||||
"Host-arrival camera/LiDAR synchronization is not a hardware-clock proof.",
|
||||
"COCO and Cityscapes models are not forest-domain or safety validated.",
|
||||
(
|
||||
"Completed 3D cuboids mix visible LiDAR support with class-size priors; "
|
||||
"unobserved volume is inferred, not measured or ground truth."
|
||||
if "cuboid_completion" in profile
|
||||
else (
|
||||
"3D cuboids describe visible LiDAR-supported surfaces, "
|
||||
"not complete object volume."
|
||||
)
|
||||
),
|
||||
*(
|
||||
["Semantic inference was intentionally stopped for a negative-control run."]
|
||||
if profile["mode"] == "semantic-loss-negative-control"
|
||||
else []
|
||||
),
|
||||
],
|
||||
}
|
||||
report_path = output / "run-report.json"
|
||||
_write_json(report_path, report)
|
||||
artifacts = [
|
||||
_artifact(semantic_path, "e10-semantic-frames", "application/x-ndjson", SEMANTIC_SCHEMA),
|
||||
_artifact(fusion_path, "e10-fusion-frames", "application/x-ndjson", FUSION_SCHEMA),
|
||||
_artifact(world_path, "e10-world-state", "application/x-ndjson", WORLD_SCHEMA),
|
||||
_artifact(arrays_path, "e10-transient-perception", "application/x-npz"),
|
||||
_artifact(gpu_path, "worker-gpu-telemetry", "application/x-ndjson"),
|
||||
_artifact(report_path, "e10-run-report", "application/json", REPORT_SCHEMA),
|
||||
]
|
||||
result = {
|
||||
"schema_version": RESULT_SCHEMA,
|
||||
"result_id": result_id,
|
||||
"identity_sha256": identity_sha256,
|
||||
"identity": identity,
|
||||
"created_at_utc": report["created_at_utc"],
|
||||
"acceptance_state": report["state"],
|
||||
"ground_truth": False,
|
||||
"publication_scope": (
|
||||
"recorded-integrated-semantic-loss-negative-control-only"
|
||||
if profile["mode"] == "semantic-loss-negative-control"
|
||||
else "recorded-integrated-realtime-qualification-only"
|
||||
),
|
||||
"frames_processed": detector_state["consumed"],
|
||||
"artifacts": artifacts,
|
||||
}
|
||||
_write_json(output / "result.json", result)
|
||||
lidar.close()
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"result_id": result_id,
|
||||
"accepted": accepted,
|
||||
"detector_fps": detector_fps,
|
||||
"semantic_fps": semantic_fps,
|
||||
"world_state_age_p95_ms": latency_summary["world_state_age_ms"]["p95"],
|
||||
"fused_frames": fused_frames,
|
||||
"accepted_cuboids": accepted_cuboids,
|
||||
},
|
||||
sort_keys=True,
|
||||
),
|
||||
flush=True,
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
def assert_disk(path: Path, floor: int, frame: int) -> None:
|
||||
if floor < 0 or (floor and shutil.disk_usage(path).free < floor):
|
||||
raise RuntimeError(f"LAB E10 crossed the D-backed free-space floor at frame {frame}")
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = arguments()
|
||||
return preflight(args) if args.command == "preflight" else run(args)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,304 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Receive and qualify the Mission Core live shadow stream without K1 access."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import base64
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import select
|
||||
import socket
|
||||
import struct
|
||||
import sys
|
||||
import time
|
||||
from collections import Counter
|
||||
from contextlib import suppress
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
from typing import Any, BinaryIO
|
||||
|
||||
WIRE_SCHEMA = "missioncore.live-perception-wire/v1"
|
||||
REPORT_SCHEMA = "missioncore.e12-shadow-transport-report/v1"
|
||||
WEBSOCKET_GUID = "258EAFA5-E914-47DA-95CA-C5AB0DC85B11"
|
||||
MAX_HEADER_BYTES = 64 * 1024
|
||||
MAX_FRAME_BYTES = 3 * 1024 * 1024
|
||||
|
||||
|
||||
class ProbeError(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
def _read_exact(stream: BinaryIO, size: int) -> bytes:
|
||||
parts: list[bytes] = []
|
||||
remaining = size
|
||||
while remaining:
|
||||
chunk = stream.read(remaining)
|
||||
if not chunk:
|
||||
raise ProbeError("shadow transport ended unexpectedly")
|
||||
parts.append(chunk)
|
||||
remaining -= len(chunk)
|
||||
return b"".join(parts)
|
||||
|
||||
|
||||
def _read_http_headers(stream: BinaryIO) -> tuple[str, dict[str, str]]:
|
||||
encoded = bytearray()
|
||||
while not encoded.endswith(b"\r\n\r\n"):
|
||||
encoded.extend(_read_exact(stream, 1))
|
||||
if len(encoded) > MAX_HEADER_BYTES:
|
||||
raise ProbeError("websocket response headers exceed the bound")
|
||||
try:
|
||||
lines = encoded.decode("ascii").split("\r\n")
|
||||
except UnicodeDecodeError as exc:
|
||||
raise ProbeError("websocket response headers are not ASCII") from exc
|
||||
headers: dict[str, str] = {}
|
||||
for line in lines[1:]:
|
||||
if not line:
|
||||
continue
|
||||
name, separator, value = line.partition(":")
|
||||
if not separator:
|
||||
raise ProbeError("malformed websocket response header")
|
||||
headers[name.strip().lower()] = value.strip()
|
||||
return lines[0], headers
|
||||
|
||||
|
||||
def _send_client_frame(stream: BinaryIO, opcode: int, payload: bytes) -> None:
|
||||
if len(payload) > 125:
|
||||
raise ProbeError("client control frame exceeds websocket bound")
|
||||
mask = os.urandom(4)
|
||||
masked = bytes(value ^ mask[index % 4] for index, value in enumerate(payload))
|
||||
stream.write(bytes((0x80 | opcode, 0x80 | len(payload))) + mask + masked)
|
||||
stream.flush()
|
||||
|
||||
|
||||
def _read_server_frame(stream: BinaryIO) -> tuple[int, bytes]:
|
||||
first, second = _read_exact(stream, 2)
|
||||
if not first & 0x80:
|
||||
raise ProbeError("fragmented websocket frames are not admitted")
|
||||
opcode = first & 0x0F
|
||||
masked = bool(second & 0x80)
|
||||
length = second & 0x7F
|
||||
if length == 126:
|
||||
length = struct.unpack("!H", _read_exact(stream, 2))[0]
|
||||
elif length == 127:
|
||||
length = struct.unpack("!Q", _read_exact(stream, 8))[0]
|
||||
if length > MAX_FRAME_BYTES:
|
||||
raise ProbeError("websocket frame exceeds the shadow transport bound")
|
||||
mask = _read_exact(stream, 4) if masked else None
|
||||
payload = _read_exact(stream, length)
|
||||
if mask is not None:
|
||||
payload = bytes(value ^ mask[index % 4] for index, value in enumerate(payload))
|
||||
return opcode, payload
|
||||
|
||||
|
||||
def _connect(
|
||||
host: str,
|
||||
port: int,
|
||||
path: str,
|
||||
token: str,
|
||||
timeout_seconds: float,
|
||||
) -> tuple[socket.socket, BinaryIO]:
|
||||
connection = socket.create_connection((host, port), timeout=timeout_seconds)
|
||||
connection.settimeout(timeout_seconds)
|
||||
stream = connection.makefile("rwb", buffering=0)
|
||||
key = base64.b64encode(os.urandom(16)).decode("ascii")
|
||||
request = (
|
||||
f"GET {path} HTTP/1.1\r\n"
|
||||
f"Host: {host}:{port}\r\n"
|
||||
"Upgrade: websocket\r\n"
|
||||
"Connection: Upgrade\r\n"
|
||||
f"Sec-WebSocket-Key: {key}\r\n"
|
||||
"Sec-WebSocket-Version: 13\r\n"
|
||||
f"Authorization: Bearer {token}\r\n"
|
||||
"\r\n"
|
||||
).encode("ascii")
|
||||
stream.write(request)
|
||||
stream.flush()
|
||||
status, headers = _read_http_headers(stream)
|
||||
expected_accept = base64.b64encode(
|
||||
hashlib.sha1(f"{key}{WEBSOCKET_GUID}".encode("ascii")).digest()
|
||||
).decode("ascii")
|
||||
if not status.startswith("HTTP/1.1 101 "):
|
||||
raise ProbeError(f"websocket upgrade failed: {status}")
|
||||
if headers.get("sec-websocket-accept") != expected_accept:
|
||||
raise ProbeError("websocket accept identity mismatch")
|
||||
connection.settimeout(None)
|
||||
return connection, stream
|
||||
|
||||
|
||||
def _decode_event(frame: bytes) -> tuple[dict[str, Any], bytes]:
|
||||
if len(frame) < 4:
|
||||
raise ProbeError("shadow event is truncated")
|
||||
header_bytes = struct.unpack("!I", frame[:4])[0]
|
||||
if header_bytes < 2 or header_bytes > MAX_HEADER_BYTES:
|
||||
raise ProbeError("shadow event header length is invalid")
|
||||
boundary = 4 + header_bytes
|
||||
if boundary > len(frame):
|
||||
raise ProbeError("shadow event header is truncated")
|
||||
try:
|
||||
header = json.loads(frame[4:boundary])
|
||||
except (UnicodeDecodeError, json.JSONDecodeError) as exc:
|
||||
raise ProbeError("shadow event header is invalid") from exc
|
||||
if not isinstance(header, dict) or header.get("schema_version") != WIRE_SCHEMA:
|
||||
raise ProbeError("shadow event schema is incompatible")
|
||||
payload = frame[boundary:]
|
||||
if int(header.get("payload_bytes", -1)) != len(payload):
|
||||
raise ProbeError("shadow event payload length mismatch")
|
||||
if hashlib.sha256(payload).hexdigest() != header.get("payload_sha256"):
|
||||
raise ProbeError("shadow event payload digest mismatch")
|
||||
if header.get("commands_enabled") is not False:
|
||||
raise ProbeError("shadow transport unexpectedly enables commands")
|
||||
if header.get("navigation_or_safety_accepted") is not False:
|
||||
raise ProbeError("shadow transport unexpectedly claims safety acceptance")
|
||||
return header, payload
|
||||
|
||||
|
||||
def _write_report(output_root: Path, report: dict[str, Any]) -> Path:
|
||||
root = output_root.expanduser().resolve()
|
||||
if os.name == "nt" and root.drive.upper() != "D:":
|
||||
raise ProbeError("Windows probe output must stay on drive D")
|
||||
root.mkdir(parents=True, exist_ok=True)
|
||||
stamp = datetime.now(UTC).strftime("%Y%m%dT%H%M%SZ")
|
||||
temporary = root / f".{stamp}-e12-shadow-transport.json.tmp"
|
||||
destination = root / f"{stamp}-e12-shadow-transport.json"
|
||||
encoded = json.dumps(report, indent=2, sort_keys=True).encode("utf-8") + b"\n"
|
||||
with temporary.open("xb") as stream:
|
||||
stream.write(encoded)
|
||||
stream.flush()
|
||||
os.fsync(stream.fileno())
|
||||
temporary.replace(destination)
|
||||
return destination
|
||||
|
||||
|
||||
def run(args: argparse.Namespace) -> dict[str, Any]:
|
||||
token = sys.stdin.readline().strip() if args.token_stdin else args.token
|
||||
if not token or len(token) < 40:
|
||||
raise ProbeError("shadow bearer token is missing or invalid")
|
||||
started_epoch_ns = time.time_ns()
|
||||
started_monotonic_ns = time.monotonic_ns()
|
||||
counts: Counter[str] = Counter()
|
||||
payload_bytes: Counter[str] = Counter()
|
||||
first_ingress_sequence: int | None = None
|
||||
last_ingress_sequence: int | None = None
|
||||
ingress_gaps = 0
|
||||
session_ids: set[str] = set()
|
||||
source_ids: set[str] = set()
|
||||
session_end_seen = False
|
||||
|
||||
connection, stream = _connect(
|
||||
args.host,
|
||||
args.port,
|
||||
args.path,
|
||||
token,
|
||||
args.socket_timeout_seconds,
|
||||
)
|
||||
try:
|
||||
deadline = time.monotonic() + args.max_duration_seconds
|
||||
while time.monotonic() < deadline:
|
||||
readable, _, _ = select.select(
|
||||
[connection],
|
||||
[],
|
||||
[],
|
||||
min(0.5, max(0.0, deadline - time.monotonic())),
|
||||
)
|
||||
if not readable:
|
||||
continue
|
||||
opcode, frame = _read_server_frame(stream)
|
||||
if opcode == 0x8:
|
||||
break
|
||||
if opcode == 0x9:
|
||||
_send_client_frame(stream, 0xA, frame)
|
||||
continue
|
||||
if opcode != 0x2:
|
||||
raise ProbeError(f"unexpected websocket opcode: {opcode}")
|
||||
header, payload = _decode_event(frame)
|
||||
sequence = int(header["ingress_sequence"])
|
||||
if last_ingress_sequence is not None and sequence > last_ingress_sequence + 1:
|
||||
ingress_gaps += sequence - last_ingress_sequence - 1
|
||||
if last_ingress_sequence is not None and sequence <= last_ingress_sequence:
|
||||
raise ProbeError("shadow ingress sequence is not strictly increasing")
|
||||
first_ingress_sequence = first_ingress_sequence or sequence
|
||||
last_ingress_sequence = sequence
|
||||
modality = str(header["modality"])
|
||||
counts[modality] += 1
|
||||
payload_bytes[modality] += len(payload)
|
||||
session_ids.add(str(header["session_id"]))
|
||||
source_ids.add(str(header["source_id"]))
|
||||
if modality == "control":
|
||||
try:
|
||||
control = json.loads(payload)
|
||||
except json.JSONDecodeError as exc:
|
||||
raise ProbeError("control event payload is invalid") from exc
|
||||
if control.get("event") == "session-end":
|
||||
session_end_seen = True
|
||||
break
|
||||
finally:
|
||||
with suppress(Exception):
|
||||
_send_client_frame(stream, 0x8, b"")
|
||||
stream.close()
|
||||
connection.close()
|
||||
|
||||
completed_epoch_ns = time.time_ns()
|
||||
report = {
|
||||
"schema_version": REPORT_SCHEMA,
|
||||
"state": "completed" if session_end_seen else "timed-out",
|
||||
"started_at_epoch_ns": started_epoch_ns,
|
||||
"completed_at_epoch_ns": completed_epoch_ns,
|
||||
"wall_seconds": (time.monotonic_ns() - started_monotonic_ns) / 1_000_000_000,
|
||||
"transport": {
|
||||
"kind": "ssh-reverse-tunnel-websocket",
|
||||
"endpoint": f"{args.host}:{args.port}",
|
||||
"worker_has_k1_connection": False,
|
||||
"payload_integrity": "sha256-verified-per-event",
|
||||
"clock_qualification": "not-qualified-cross-host",
|
||||
},
|
||||
"authority": {
|
||||
"mode": "shadow-diagnostic-only",
|
||||
"commands_enabled": False,
|
||||
"navigation_or_safety_accepted": False,
|
||||
},
|
||||
"events": {
|
||||
"counts": dict(sorted(counts.items())),
|
||||
"payload_bytes": dict(sorted(payload_bytes.items())),
|
||||
"first_ingress_sequence": first_ingress_sequence,
|
||||
"last_ingress_sequence": last_ingress_sequence,
|
||||
"ingress_sequence_gaps": ingress_gaps,
|
||||
"session_end_seen": session_end_seen,
|
||||
"session_ids": sorted(session_ids),
|
||||
"source_ids": sorted(source_ids),
|
||||
},
|
||||
}
|
||||
destination = _write_report(Path(args.output_root), report)
|
||||
report["report_path"] = str(destination)
|
||||
return report
|
||||
|
||||
|
||||
def _arguments() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--host", default="host.docker.internal")
|
||||
parser.add_argument("--port", type=int, default=18012)
|
||||
parser.add_argument(
|
||||
"--path",
|
||||
default=(
|
||||
"/api/v1/device-plugins/"
|
||||
"nodedc.device.xgrids-lixelkity-k1/live-perception-shadow"
|
||||
),
|
||||
)
|
||||
parser.add_argument("--token", default="")
|
||||
parser.add_argument("--token-stdin", action="store_true")
|
||||
parser.add_argument("--max-duration-seconds", type=float, default=900.0)
|
||||
parser.add_argument("--socket-timeout-seconds", type=float, default=30.0)
|
||||
parser.add_argument(
|
||||
"--output-root",
|
||||
default="D:/NDC_MISSIONCORE/runtime/derived/e12-shadow-transport",
|
||||
)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
print(json.dumps(run(_arguments()), indent=2, sort_keys=True))
|
||||
except Exception as exc:
|
||||
print(f"E12_SHADOW_TRANSPORT_ERROR={type(exc).__name__}:{exc}", file=sys.stderr)
|
||||
raise SystemExit(1) from exc
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,768 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Run and seal LAB E4 full-session EoMT semantic segmentation."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import importlib.metadata
|
||||
import json
|
||||
import os
|
||||
import platform
|
||||
import resource
|
||||
import shutil
|
||||
import time
|
||||
from collections import Counter
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from run_e3_rectified_segmentation import (
|
||||
_dependency_manifest,
|
||||
_model_snapshot,
|
||||
_profile,
|
||||
_resource_delta,
|
||||
_resource_snapshot,
|
||||
_verify_model_files,
|
||||
)
|
||||
from run_recorded_perception_epoch import (
|
||||
FRAME_SCHEMA,
|
||||
REPORT_SCHEMA,
|
||||
RESULT_SCHEMA,
|
||||
_artifact,
|
||||
_canonical_json,
|
||||
_GpuTelemetry,
|
||||
_palette,
|
||||
_percentiles,
|
||||
_read_timeline,
|
||||
_sha256,
|
||||
_valid_sha256,
|
||||
_validate_job,
|
||||
_write_json,
|
||||
)
|
||||
|
||||
IDENTITY_SCHEMA = "missioncore.recorded-perception-identity/v2"
|
||||
VALID_FOV_SCHEMA = "missioncore.k1-valid-fov-mask/v1"
|
||||
PIPELINE_ID = "recorded-semantic-eomt-fisheye-mask/v1"
|
||||
SEMANTIC_ALPHA = 0.48
|
||||
TARGET_CLASS_COUNT = 16
|
||||
|
||||
|
||||
def _arguments() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser()
|
||||
commands = parser.add_subparsers(dest="command", required=True)
|
||||
|
||||
preflight = commands.add_parser("preflight")
|
||||
preflight.add_argument("--job", type=Path, required=True)
|
||||
preflight.add_argument("--profile", type=Path, required=True)
|
||||
preflight.add_argument("--valid-fov-root", type=Path, required=True)
|
||||
preflight.add_argument("--cache", type=Path, required=True)
|
||||
preflight.add_argument("--environment", type=Path, required=True)
|
||||
|
||||
run = commands.add_parser("run")
|
||||
run.add_argument("--job", type=Path, required=True)
|
||||
run.add_argument("--profile", type=Path, required=True)
|
||||
run.add_argument("--valid-fov-root", type=Path, required=True)
|
||||
run.add_argument("--frames", type=Path, required=True)
|
||||
run.add_argument("--timeline", type=Path, required=True)
|
||||
run.add_argument("--cache", type=Path, required=True)
|
||||
run.add_argument("--environment", type=Path, required=True)
|
||||
run.add_argument("--output", type=Path, required=True)
|
||||
run.add_argument("--frame-limit", type=int, default=0)
|
||||
run.add_argument("--free-bytes-floor", type=int, default=0)
|
||||
run.add_argument("--orchestrator-sha256", required=True)
|
||||
run.add_argument("--container-image", required=True)
|
||||
run.add_argument("--telemetry-interval-seconds", type=float, default=1.0)
|
||||
|
||||
finalize = commands.add_parser("finalize")
|
||||
finalize.add_argument("--output", type=Path, required=True)
|
||||
finalize.add_argument("--video", type=Path, required=True)
|
||||
finalize.add_argument("--masks", type=Path, required=True)
|
||||
finalize.add_argument("--extract-seconds", type=float, required=True)
|
||||
finalize.add_argument("--encode-seconds", type=float, required=True)
|
||||
finalize.add_argument("--archive-seconds", type=float, required=True)
|
||||
finalize.add_argument("--wall-seconds", type=float, required=True)
|
||||
finalize.add_argument("--encoder", required=True)
|
||||
finalize.add_argument("--disk-free-before-bytes", type=int, required=True)
|
||||
finalize.add_argument("--disk-free-post-extract-bytes", type=int, required=True)
|
||||
finalize.add_argument("--disk-free-post-inference-bytes", type=int, required=True)
|
||||
finalize.add_argument("--disk-free-post-artifacts-bytes", type=int, required=True)
|
||||
finalize.add_argument("--disk-floor-bytes", type=int, required=True)
|
||||
finalize.add_argument("--working-set-reserve-bytes", type=int, required=True)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def _read_object(path: Path) -> dict[str, Any]:
|
||||
value = json.loads(path.read_text(encoding="utf-8-sig"))
|
||||
if not isinstance(value, dict):
|
||||
raise RuntimeError(f"JSON root is not an object: {path}")
|
||||
return value
|
||||
|
||||
|
||||
def _validate_source(job: dict[str, Any], profile: dict[str, Any]) -> None:
|
||||
source = profile["source"]
|
||||
input_document = job["input"]
|
||||
if (
|
||||
input_document.get("kind") != "canonical-camera-epoch"
|
||||
or input_document.get("source_id") != source["source_id"]
|
||||
or source.get("resolution") != [800, 600]
|
||||
):
|
||||
raise RuntimeError("LAB E4 profile does not match the camera job")
|
||||
|
||||
|
||||
def _load_valid_fov(
|
||||
root: Path,
|
||||
job: dict[str, Any],
|
||||
profile: dict[str, Any],
|
||||
) -> tuple[Any, dict[str, Any]]:
|
||||
import numpy as np
|
||||
from PIL import Image
|
||||
|
||||
resolved = root.resolve(strict=True)
|
||||
manifest = _read_object(resolved / "manifest.json")
|
||||
identity = manifest.get("identity")
|
||||
identity_sha256 = manifest.get("identity_sha256")
|
||||
artifact = manifest.get("artifact")
|
||||
source = profile["source"]
|
||||
if (
|
||||
manifest.get("schema_version") != VALID_FOV_SCHEMA
|
||||
or not isinstance(identity, dict)
|
||||
or not _valid_sha256(identity_sha256)
|
||||
or hashlib.sha256(_canonical_json(identity)).hexdigest() != identity_sha256
|
||||
or manifest.get("generation_id") != f"valid-fov-mask-{identity_sha256}"
|
||||
or identity.get("calibration_sha256") != source["calibration_sha256"]
|
||||
or identity.get("calibration_slot") != source["calibration_slot"]
|
||||
or identity.get("source_id") != job["input"]["source_id"]
|
||||
or identity.get("admitted_resolution") != source["resolution"]
|
||||
or not isinstance(artifact, dict)
|
||||
or artifact.get("path") != "mask.png"
|
||||
or artifact.get("mode") != "L"
|
||||
or artifact.get("inside_value") != 255
|
||||
or artifact.get("outside_value") != 0
|
||||
or not _valid_sha256(artifact.get("sha256"))
|
||||
):
|
||||
raise RuntimeError("LAB E4 valid-FOV binding is invalid")
|
||||
mask_path = (resolved / "mask.png").resolve(strict=True)
|
||||
if (
|
||||
mask_path.parent != resolved
|
||||
or mask_path.stat().st_size != artifact.get("byte_length")
|
||||
or _sha256(mask_path) != artifact["sha256"]
|
||||
):
|
||||
raise RuntimeError("LAB E4 valid-FOV artifact changed")
|
||||
with Image.open(mask_path) as opened:
|
||||
mask = np.asarray(opened, dtype=np.uint8)
|
||||
if mask.shape != (600, 800) or not np.isin(mask, (0, 255)).all():
|
||||
raise RuntimeError("LAB E4 valid-FOV mask pixels are invalid")
|
||||
valid = mask > 0
|
||||
geometry = manifest.get("geometry")
|
||||
if (
|
||||
not isinstance(geometry, dict)
|
||||
or geometry.get("valid_pixel_count") != int(valid.sum())
|
||||
or int(valid.sum()) < 1
|
||||
):
|
||||
raise RuntimeError("LAB E4 valid-FOV geometry changed")
|
||||
return valid, {
|
||||
"generation_id": manifest["generation_id"],
|
||||
"identity_sha256": identity_sha256,
|
||||
"mask_sha256": artifact["sha256"],
|
||||
"valid_pixel_count": int(valid.sum()),
|
||||
"total_pixel_count": int(valid.size),
|
||||
}
|
||||
|
||||
|
||||
def _load_model(
|
||||
profile: dict[str, Any],
|
||||
cache: Path,
|
||||
device: Any,
|
||||
) -> tuple[Any, Any, dict[int, str], Any, list[dict[str, Any]]]:
|
||||
import numpy as np
|
||||
import torch
|
||||
from transformers import AutoImageProcessor, EomtForUniversalSegmentation
|
||||
|
||||
snapshot = _model_snapshot(profile, cache, offline=True)
|
||||
model_files = _verify_model_files(profile, snapshot)
|
||||
processor = AutoImageProcessor.from_pretrained(
|
||||
snapshot,
|
||||
local_files_only=True,
|
||||
use_fast=True,
|
||||
)
|
||||
model, loading = EomtForUniversalSegmentation.from_pretrained(
|
||||
snapshot,
|
||||
local_files_only=True,
|
||||
output_loading_info=True,
|
||||
)
|
||||
problems = {
|
||||
name: loading.get(name, [])
|
||||
for name in ("missing_keys", "unexpected_keys", "mismatched_keys", "error_msgs")
|
||||
if loading.get(name)
|
||||
}
|
||||
if problems:
|
||||
raise RuntimeError("EoMT checkpoint did not load exactly: " + json.dumps(problems))
|
||||
model = model.to(device=device, dtype=torch.float16).eval()
|
||||
labels = {int(key): str(value) for key, value in model.config.id2label.items()}
|
||||
mapping = profile["cityscapes_to_target"]
|
||||
if set(labels.values()) != set(mapping):
|
||||
raise RuntimeError("EoMT runtime taxonomy changed")
|
||||
target_lut = np.asarray([mapping[labels[index]] for index in range(19)], dtype=np.uint8)
|
||||
return processor, model, labels, target_lut, model_files
|
||||
|
||||
|
||||
def _preflight(args: argparse.Namespace) -> int:
|
||||
import torch
|
||||
|
||||
job = _validate_job(args.job.resolve(strict=True))
|
||||
profile, profile_sha256 = _profile(args.profile)
|
||||
_validate_source(job, profile)
|
||||
dependency = _dependency_manifest(args.environment.resolve(strict=True))
|
||||
if dependency["identity"]["profile_sha256"] != profile_sha256:
|
||||
raise RuntimeError("LAB E4 dependencies belong to another profile")
|
||||
_load_valid_fov(args.valid_fov_root, job, profile)
|
||||
if not torch.cuda.is_available() or torch.cuda.device_count() < 1:
|
||||
raise RuntimeError("CUDA device 0 is unavailable")
|
||||
device = torch.device("cuda:0")
|
||||
processor, model, _labels, _target_lut, model_files = _load_model(
|
||||
profile,
|
||||
args.cache.resolve(strict=True),
|
||||
device,
|
||||
)
|
||||
del processor, model, _labels, _target_lut
|
||||
torch.cuda.empty_cache()
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"state": "preflight-ready",
|
||||
"job_id": job["job_id"],
|
||||
"source_id": job["input"]["source_id"],
|
||||
"frames": job["input"]["segment_count"],
|
||||
"pipeline": PIPELINE_ID,
|
||||
"model_files": model_files,
|
||||
"cuda_device": torch.cuda.get_device_name(),
|
||||
},
|
||||
sort_keys=True,
|
||||
),
|
||||
flush=True,
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
def _class_document(
|
||||
semantic: Any,
|
||||
valid_mask: Any,
|
||||
target_names: dict[int, str],
|
||||
) -> list[dict[str, Any]]:
|
||||
import numpy as np
|
||||
|
||||
counts = np.bincount(semantic[valid_mask].reshape(-1), minlength=TARGET_CLASS_COUNT)
|
||||
total = int(valid_mask.sum())
|
||||
return [
|
||||
{
|
||||
"id": index,
|
||||
"label": target_names[index],
|
||||
"pixels": int(counts[index]),
|
||||
"fraction_of_valid_fov": round(float(counts[index]) / total, 9),
|
||||
}
|
||||
for index in range(1, TARGET_CLASS_COUNT)
|
||||
if counts[index] > 0
|
||||
]
|
||||
|
||||
|
||||
def _overlay(image: Any, semantic: Any) -> Any:
|
||||
import numpy as np
|
||||
|
||||
colors = np.zeros_like(image)
|
||||
for category_id in range(1, TARGET_CLASS_COUNT):
|
||||
colors[semantic == category_id] = _palette(category_id)
|
||||
blended = np.rint(
|
||||
image.astype(np.float32) * (1.0 - SEMANTIC_ALPHA)
|
||||
+ colors.astype(np.float32) * SEMANTIC_ALPHA
|
||||
)
|
||||
return (
|
||||
np.where(
|
||||
(semantic > 0)[..., None],
|
||||
blended,
|
||||
image,
|
||||
)
|
||||
.clip(0, 255)
|
||||
.astype(np.uint8)
|
||||
)
|
||||
|
||||
|
||||
def _write_png(path: Path, array: Any) -> None:
|
||||
from PIL import Image
|
||||
|
||||
Image.fromarray(array).save(path, format="PNG", optimize=False)
|
||||
|
||||
|
||||
def _assert_disk_floor(path: Path, floor: int, frame: int) -> None:
|
||||
if floor < 0:
|
||||
raise RuntimeError("disk free-space floor is invalid")
|
||||
free = shutil.disk_usage(path).free
|
||||
if floor and free < floor:
|
||||
raise RuntimeError(f"D-backed output crossed its free-space floor at frame {frame}")
|
||||
|
||||
|
||||
def _run(args: argparse.Namespace) -> int:
|
||||
import numpy as np
|
||||
import torch
|
||||
import transformers
|
||||
from PIL import Image
|
||||
|
||||
job_root = args.job.resolve(strict=True)
|
||||
if not _valid_sha256(args.orchestrator_sha256):
|
||||
raise RuntimeError("LAB E4 orchestrator SHA-256 is invalid")
|
||||
if not 1 <= len(args.container_image) <= 256:
|
||||
raise RuntimeError("LAB E4 container image identity is invalid")
|
||||
job = _validate_job(job_root)
|
||||
profile, profile_sha256 = _profile(args.profile)
|
||||
_validate_source(job, profile)
|
||||
dependency = _dependency_manifest(args.environment.resolve(strict=True))
|
||||
if dependency["identity"]["profile_sha256"] != profile_sha256:
|
||||
raise RuntimeError("LAB E4 dependencies belong to another profile")
|
||||
valid_mask, valid_fov = _load_valid_fov(args.valid_fov_root, job, profile)
|
||||
|
||||
input_document = job["input"]
|
||||
full_frame_count = int(input_document["segment_count"])
|
||||
if args.frame_limit < 0 or args.frame_limit > full_frame_count:
|
||||
raise RuntimeError("LAB E4 frame limit escapes the camera job")
|
||||
frame_count = args.frame_limit or full_frame_count
|
||||
frames_root = args.frames.resolve(strict=True)
|
||||
frame_paths = [frames_root / f"frame-{index:06d}.png" for index in range(1, frame_count + 1)]
|
||||
if not all(path.is_file() for path in frame_paths):
|
||||
raise RuntimeError("decoded LAB E4 frame set is incomplete")
|
||||
if len(list(frames_root.glob("frame-*.png"))) != frame_count:
|
||||
raise RuntimeError("decoded LAB E4 frame set contains unexpected files")
|
||||
timeline = input_document["timeline"]
|
||||
timestamps = _read_timeline(
|
||||
args.timeline.resolve(strict=True),
|
||||
frame_count,
|
||||
float(timeline["start_seconds"]),
|
||||
float(timeline["end_seconds"]),
|
||||
)
|
||||
|
||||
output = args.output.resolve()
|
||||
if output.exists():
|
||||
raise RuntimeError("LAB E4 output must be absent")
|
||||
output.mkdir(mode=0o700, parents=True, exist_ok=False)
|
||||
masks_root = output / "semantic-masks"
|
||||
overlays_root = output / "overlay-frames"
|
||||
masks_root.mkdir(mode=0o700)
|
||||
overlays_root.mkdir(mode=0o700)
|
||||
_assert_disk_floor(output, args.free_bytes_floor, 0)
|
||||
|
||||
if not torch.cuda.is_available():
|
||||
raise RuntimeError("CUDA is required for LAB E4")
|
||||
device = torch.device("cuda:0")
|
||||
processor, model, model_labels, target_lut, model_files = _load_model(
|
||||
profile,
|
||||
args.cache.resolve(strict=True),
|
||||
device,
|
||||
)
|
||||
target_names = {int(key): str(value) for key, value in profile["target_taxonomy"].items()}
|
||||
if set(target_names) != set(range(TARGET_CLASS_COUNT)):
|
||||
raise RuntimeError("LAB E4 target taxonomy changed")
|
||||
|
||||
def infer(image: Any) -> tuple[Any, dict[str, float]]:
|
||||
processor_started = time.perf_counter()
|
||||
inputs = processor(images=Image.fromarray(image), return_tensors="pt")
|
||||
processor_ms = (time.perf_counter() - processor_started) * 1000.0
|
||||
torch.cuda.synchronize()
|
||||
transfer_started = time.perf_counter()
|
||||
inputs = {
|
||||
name: value.to(device) if isinstance(value, torch.Tensor) else value
|
||||
for name, value in inputs.items()
|
||||
}
|
||||
torch.cuda.synchronize()
|
||||
transfer_ms = (time.perf_counter() - transfer_started) * 1000.0
|
||||
forward_started = time.perf_counter()
|
||||
with torch.inference_mode(), torch.amp.autocast("cuda", dtype=torch.float16):
|
||||
outputs = model(**inputs)
|
||||
torch.cuda.synchronize()
|
||||
forward_ms = (time.perf_counter() - forward_started) * 1000.0
|
||||
post_started = time.perf_counter()
|
||||
semantic = processor.post_process_semantic_segmentation(
|
||||
outputs,
|
||||
target_sizes=[image.shape[:2]],
|
||||
)[0]
|
||||
semantic = semantic.detach().cpu().numpy().astype(np.uint8)
|
||||
post_ms = (time.perf_counter() - post_started) * 1000.0
|
||||
del inputs, outputs
|
||||
return semantic, {
|
||||
"processor_ms": processor_ms,
|
||||
"host_to_device_ms": transfer_ms,
|
||||
"forward_ms": forward_ms,
|
||||
"model_postprocess_ms": post_ms,
|
||||
}
|
||||
|
||||
with Image.open(frame_paths[0]) as opened:
|
||||
warm_image = np.asarray(opened.convert("RGB"), dtype=np.uint8)
|
||||
if warm_image.shape != (600, 800, 3):
|
||||
raise RuntimeError("LAB E4 input resolution changed")
|
||||
warm_image = np.where(valid_mask[..., None], warm_image, 0).astype(np.uint8)
|
||||
warm_semantic, _warm_latency = infer(warm_image)
|
||||
del warm_semantic, _warm_latency
|
||||
|
||||
latency = {
|
||||
name: []
|
||||
for name in (
|
||||
"image_decode_ms",
|
||||
"valid_fov_fill_ms",
|
||||
"processor_ms",
|
||||
"host_to_device_ms",
|
||||
"forward_ms",
|
||||
"model_postprocess_ms",
|
||||
"overlay_ms",
|
||||
"artifact_write_ms",
|
||||
"end_to_end_ms",
|
||||
)
|
||||
}
|
||||
total_pixels = Counter()
|
||||
metadata_path = output / "frames.jsonl"
|
||||
telemetry_path = output / "gpu-telemetry.jsonl"
|
||||
resource_before = _resource_snapshot()
|
||||
disk_before = shutil.disk_usage(output).free
|
||||
started = time.perf_counter()
|
||||
|
||||
with (
|
||||
telemetry_path.open("x", encoding="utf-8", newline="\n") as telemetry_stream,
|
||||
metadata_path.open("x", encoding="utf-8", newline="\n") as metadata_stream,
|
||||
_GpuTelemetry(telemetry_stream, args.telemetry_interval_seconds) as telemetry,
|
||||
):
|
||||
for frame_index, (path, session_seconds) in enumerate(
|
||||
zip(frame_paths, timestamps, strict=True)
|
||||
):
|
||||
frame_started = time.perf_counter()
|
||||
decode_started = time.perf_counter()
|
||||
with Image.open(path) as opened:
|
||||
image = np.asarray(opened.convert("RGB"), dtype=np.uint8)
|
||||
latency["image_decode_ms"].append((time.perf_counter() - decode_started) * 1000.0)
|
||||
if image.shape != (600, 800, 3):
|
||||
raise RuntimeError("LAB E4 input resolution changed")
|
||||
|
||||
fill_started = time.perf_counter()
|
||||
model_input = np.where(valid_mask[..., None], image, 0).astype(np.uint8)
|
||||
latency["valid_fov_fill_ms"].append((time.perf_counter() - fill_started) * 1000.0)
|
||||
semantic, measured = infer(model_input)
|
||||
if semantic.max() >= len(target_lut):
|
||||
raise RuntimeError("EoMT emitted an unknown category")
|
||||
target = target_lut[semantic]
|
||||
target = target.copy()
|
||||
target[~valid_mask] = 0
|
||||
if target.shape != valid_mask.shape or np.any(target[~valid_mask] != 0):
|
||||
raise RuntimeError("LAB E4 semantic mask escaped valid FOV")
|
||||
for name, value in measured.items():
|
||||
latency[name].append(value)
|
||||
|
||||
overlay_started = time.perf_counter()
|
||||
overlay = _overlay(image, target)
|
||||
classes = _class_document(target, valid_mask, target_names)
|
||||
latency["overlay_ms"].append((time.perf_counter() - overlay_started) * 1000.0)
|
||||
counts = np.bincount(target[valid_mask], minlength=TARGET_CLASS_COUNT)
|
||||
for category_id in range(1, TARGET_CLASS_COUNT):
|
||||
total_pixels[target_names[category_id]] += int(counts[category_id])
|
||||
|
||||
write_started = time.perf_counter()
|
||||
_write_png(masks_root / f"frame-{frame_index + 1:06d}.png", target)
|
||||
_write_png(overlays_root / f"frame-{frame_index + 1:06d}.png", overlay)
|
||||
metadata_stream.write(
|
||||
json.dumps(
|
||||
{
|
||||
"schema_version": FRAME_SCHEMA,
|
||||
"frame_index": frame_index,
|
||||
"sequence": frame_index + 1,
|
||||
"session_seconds": round(session_seconds, 9),
|
||||
"instances": [],
|
||||
"semantic_classes": classes,
|
||||
},
|
||||
ensure_ascii=False,
|
||||
sort_keys=True,
|
||||
separators=(",", ":"),
|
||||
allow_nan=False,
|
||||
)
|
||||
+ "\n"
|
||||
)
|
||||
latency["artifact_write_ms"].append((time.perf_counter() - write_started) * 1000.0)
|
||||
latency["end_to_end_ms"].append((time.perf_counter() - frame_started) * 1000.0)
|
||||
|
||||
completed = frame_index + 1
|
||||
if completed % 100 == 0 or completed == frame_count:
|
||||
metadata_stream.flush()
|
||||
telemetry_stream.flush()
|
||||
_assert_disk_floor(output, args.free_bytes_floor, completed)
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"phase": "semantic",
|
||||
"frames_processed": completed,
|
||||
"frames_total": frame_count,
|
||||
"free_bytes": shutil.disk_usage(output).free,
|
||||
},
|
||||
sort_keys=True,
|
||||
),
|
||||
flush=True,
|
||||
)
|
||||
metadata_stream.flush()
|
||||
os.fsync(metadata_stream.fileno())
|
||||
|
||||
inference_elapsed = time.perf_counter() - started
|
||||
resource_after = _resource_snapshot()
|
||||
disk_after = shutil.disk_usage(output).free
|
||||
report = {
|
||||
"schema_version": REPORT_SCHEMA,
|
||||
"run_id": f"{job['job_id']}-e4-eomt-full-session-v1",
|
||||
"created_at_utc": datetime.now(UTC)
|
||||
.isoformat(timespec="milliseconds")
|
||||
.replace("+00:00", "Z"),
|
||||
"state": (
|
||||
"inference-complete-awaiting-publication"
|
||||
if frame_count == full_frame_count
|
||||
else "technical-pilot-complete-not-publishable"
|
||||
),
|
||||
"ground_truth": False,
|
||||
"input": {
|
||||
"job_id": job["job_id"],
|
||||
"input_sha256": job["input_sha256"],
|
||||
"session_id": input_document["session_id"],
|
||||
"source_id": input_document["source_id"],
|
||||
"codec_epoch": input_document["codec_epoch"],
|
||||
"segment_count": full_frame_count,
|
||||
"frames_admitted": frame_count,
|
||||
"byte_length": input_document["byte_length"],
|
||||
"timeline_start_seconds": timeline["start_seconds"],
|
||||
"timeline_end_seconds": timeline["end_seconds"],
|
||||
"frame_timeline_sha256": _sha256(args.timeline.resolve(strict=True)),
|
||||
},
|
||||
"calibration": {
|
||||
"content_identity_sha256": profile["source"]["calibration_sha256"],
|
||||
"camera_slot": profile["source"]["calibration_slot"],
|
||||
},
|
||||
"configuration": {
|
||||
"pipeline": PIPELINE_ID,
|
||||
"semantic_alpha": SEMANTIC_ALPHA,
|
||||
"batch_size": 1,
|
||||
"precision": "fp16-autocast",
|
||||
"frame_policy": (
|
||||
"all-frames-no-sampling"
|
||||
if frame_count == full_frame_count
|
||||
else f"technical-pilot-first-{frame_count}-frames"
|
||||
),
|
||||
"instance_branch": "disabled",
|
||||
"profile_sha256": profile_sha256,
|
||||
"dependency_identity_sha256": dependency["identity_sha256"],
|
||||
"valid_fov": valid_fov,
|
||||
"runner_sha256": _sha256(Path(__file__).resolve(strict=True)),
|
||||
"orchestrator_sha256": args.orchestrator_sha256,
|
||||
"container_image": args.container_image,
|
||||
},
|
||||
"models": {
|
||||
"semantic": {
|
||||
"id": profile["model"]["id"],
|
||||
"revision": profile["model"]["revision"],
|
||||
"architecture": profile["model"]["architecture"],
|
||||
"checkpoint_load": "exact-no-missing-unexpected-or-mismatched-keys",
|
||||
"runtime_labels": {
|
||||
str(index): label for index, label in sorted(model_labels.items())
|
||||
},
|
||||
},
|
||||
"files": model_files,
|
||||
},
|
||||
"runtime": {
|
||||
"hostname": platform.node(),
|
||||
"platform": platform.platform(),
|
||||
"python": platform.python_version(),
|
||||
"torch": torch.__version__,
|
||||
"transformers": transformers.__version__,
|
||||
"cuda_runtime": torch.version.cuda,
|
||||
"gpu": torch.cuda.get_device_name(),
|
||||
"gpu_compute_capability": list(torch.cuda.get_device_capability()),
|
||||
},
|
||||
"metrics": {
|
||||
"frames_expected": frame_count,
|
||||
"frames_processed": frame_count,
|
||||
"frames_failed": 0,
|
||||
"frames_skipped": 0,
|
||||
"instances": 0,
|
||||
"inference_wall_seconds": round(inference_elapsed, 6),
|
||||
"inference_frames_per_second": round(frame_count / inference_elapsed, 6),
|
||||
"latency_ms": {name: _percentiles(values) for name, values in latency.items()},
|
||||
"semantic_pixel_totals_inside_valid_fov": dict(sorted(total_pixels.items())),
|
||||
"cuda_peak_memory_allocated_mib": round(
|
||||
torch.cuda.max_memory_allocated() / 2**20,
|
||||
3,
|
||||
),
|
||||
"cuda_peak_memory_reserved_mib": round(
|
||||
torch.cuda.max_memory_reserved() / 2**20,
|
||||
3,
|
||||
),
|
||||
"process_peak_rss_mib": round(
|
||||
resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024,
|
||||
3,
|
||||
),
|
||||
"resource_delta": _resource_delta(resource_before, resource_after),
|
||||
"gpu_telemetry": telemetry.summary(),
|
||||
"disk": {
|
||||
"free_bytes_before_inference": disk_before,
|
||||
"free_bytes_after_inference": disk_after,
|
||||
"free_bytes_floor": args.free_bytes_floor,
|
||||
},
|
||||
},
|
||||
"versions": {
|
||||
name: importlib.metadata.version(name)
|
||||
for name in ("numpy", "pillow", "torch", "transformers")
|
||||
},
|
||||
}
|
||||
_write_json(output / "run-report.partial.json", report)
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"state": report["state"],
|
||||
"frames": frame_count,
|
||||
"inference_seconds": round(inference_elapsed, 6),
|
||||
},
|
||||
sort_keys=True,
|
||||
),
|
||||
flush=True,
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
def _finalize(args: argparse.Namespace) -> int:
|
||||
output = args.output.resolve(strict=True)
|
||||
video = args.video.resolve(strict=True)
|
||||
masks = args.masks.resolve(strict=True)
|
||||
if video.parent != output or masks.parent != output:
|
||||
raise RuntimeError("published LAB E4 artifacts must be direct result children")
|
||||
partial_path = output / "run-report.partial.json"
|
||||
frames_path = output / "frames.jsonl"
|
||||
telemetry_path = output / "gpu-telemetry.jsonl"
|
||||
partial = _read_object(partial_path)
|
||||
input_document = partial.get("input")
|
||||
metrics = partial.get("metrics")
|
||||
if (
|
||||
partial.get("schema_version") != REPORT_SCHEMA
|
||||
or partial.get("state") != "inference-complete-awaiting-publication"
|
||||
or not isinstance(input_document, dict)
|
||||
or not isinstance(metrics, dict)
|
||||
or input_document.get("frames_admitted") != input_document.get("segment_count")
|
||||
or metrics.get("frames_expected") != input_document.get("segment_count")
|
||||
or metrics.get("frames_processed") != input_document.get("segment_count")
|
||||
):
|
||||
raise RuntimeError("only a complete LAB E4 camera epoch can be finalized")
|
||||
for value in (
|
||||
args.extract_seconds,
|
||||
args.encode_seconds,
|
||||
args.archive_seconds,
|
||||
args.wall_seconds,
|
||||
):
|
||||
if value <= 0:
|
||||
raise RuntimeError("LAB E4 publication timing is invalid")
|
||||
disk_values = (
|
||||
args.disk_free_before_bytes,
|
||||
args.disk_free_post_extract_bytes,
|
||||
args.disk_free_post_inference_bytes,
|
||||
args.disk_free_post_artifacts_bytes,
|
||||
args.disk_floor_bytes,
|
||||
args.working_set_reserve_bytes,
|
||||
)
|
||||
if any(value < 0 for value in disk_values):
|
||||
raise RuntimeError("LAB E4 publication disk telemetry is invalid")
|
||||
if any(
|
||||
value < args.disk_floor_bytes
|
||||
for value in (
|
||||
args.disk_free_before_bytes,
|
||||
args.disk_free_post_extract_bytes,
|
||||
args.disk_free_post_inference_bytes,
|
||||
args.disk_free_post_artifacts_bytes,
|
||||
)
|
||||
):
|
||||
raise RuntimeError("LAB E4 publication crossed its D: free-space floor")
|
||||
|
||||
identity = {
|
||||
"schema_version": IDENTITY_SCHEMA,
|
||||
"job_id": input_document["job_id"],
|
||||
"input_sha256": input_document["input_sha256"],
|
||||
"calibration": partial["calibration"],
|
||||
"configuration": partial["configuration"],
|
||||
"models": partial["models"],
|
||||
"publication": {
|
||||
"video_encoder": args.encoder,
|
||||
"video_media_type": "video/mp4",
|
||||
"mask_archive_media_type": "application/gzip",
|
||||
"content": "semantic-overlay-and-semantic-masks",
|
||||
},
|
||||
}
|
||||
identity_sha256 = hashlib.sha256(_canonical_json(identity)).hexdigest()
|
||||
result_id = f"result-{identity_sha256}"
|
||||
final_report = {
|
||||
**partial,
|
||||
"state": "published",
|
||||
"result_id": result_id,
|
||||
"publication": {
|
||||
"extract_seconds": round(args.extract_seconds, 6),
|
||||
"encode_seconds": round(args.encode_seconds, 6),
|
||||
"archive_seconds": round(args.archive_seconds, 6),
|
||||
"wall_seconds": round(args.wall_seconds, 6),
|
||||
"end_to_end_frames_per_second": round(
|
||||
int(metrics["frames_processed"]) / args.wall_seconds,
|
||||
6,
|
||||
),
|
||||
"encoder": args.encoder,
|
||||
"disk": {
|
||||
"free_bytes_before": args.disk_free_before_bytes,
|
||||
"free_bytes_post_extract": args.disk_free_post_extract_bytes,
|
||||
"free_bytes_post_inference": args.disk_free_post_inference_bytes,
|
||||
"free_bytes_post_artifacts": args.disk_free_post_artifacts_bytes,
|
||||
"free_bytes_floor": args.disk_floor_bytes,
|
||||
"working_set_reserve_bytes": args.working_set_reserve_bytes,
|
||||
},
|
||||
},
|
||||
}
|
||||
report_path = output / "run-report.json"
|
||||
_write_json(report_path, final_report)
|
||||
artifacts = [
|
||||
_artifact(video, "panoptic-overlay-video", "video/mp4"),
|
||||
_artifact(masks, "panoptic-mask-archive", "application/gzip"),
|
||||
_artifact(frames_path, "panoptic-frame-metadata", "application/x-ndjson", FRAME_SCHEMA),
|
||||
_artifact(telemetry_path, "worker-gpu-telemetry", "application/x-ndjson"),
|
||||
_artifact(report_path, "perception-run-report", "application/json", REPORT_SCHEMA),
|
||||
]
|
||||
result = {
|
||||
"schema_version": RESULT_SCHEMA,
|
||||
"result_id": result_id,
|
||||
"identity_sha256": identity_sha256,
|
||||
"identity": identity,
|
||||
"created_at_utc": final_report["created_at_utc"],
|
||||
"job_id": input_document["job_id"],
|
||||
"input_sha256": input_document["input_sha256"],
|
||||
"session_id": input_document["session_id"],
|
||||
"source_id": input_document["source_id"],
|
||||
"codec_epoch": input_document["codec_epoch"],
|
||||
"timestamp_basis": "session-time-seconds",
|
||||
"timeline_start_seconds": input_document["timeline_start_seconds"],
|
||||
"timeline_end_seconds": input_document["timeline_end_seconds"],
|
||||
"frames_processed": metrics["frames_processed"],
|
||||
"ground_truth": False,
|
||||
"artifacts": artifacts,
|
||||
}
|
||||
_write_json(output / "result.json", result)
|
||||
partial_path.unlink()
|
||||
print(
|
||||
json.dumps(
|
||||
{"result_id": result_id, "identity_sha256": identity_sha256},
|
||||
sort_keys=True,
|
||||
),
|
||||
flush=True,
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = _arguments()
|
||||
if args.command == "preflight":
|
||||
return _preflight(args)
|
||||
if args.command == "run":
|
||||
return _run(args)
|
||||
return _finalize(args)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,687 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Run source-paced LAB E8 YOLOX plus tracking without synchronous overlays."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import platform
|
||||
import resource
|
||||
import shutil
|
||||
import tempfile
|
||||
import threading
|
||||
import time
|
||||
from collections import Counter, deque
|
||||
from dataclasses import dataclass
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import numpy as np
|
||||
from run_e5_instance_tracking import (
|
||||
TwoStageTracker,
|
||||
_detections,
|
||||
_duplicate_pairs,
|
||||
_infer,
|
||||
_load_valid_fov,
|
||||
_preprocess,
|
||||
_read_timeline,
|
||||
_track_document,
|
||||
_validate_source,
|
||||
_verify_model,
|
||||
)
|
||||
from run_e5_instance_tracking import (
|
||||
_read_profile as _read_e5_profile,
|
||||
)
|
||||
from run_recorded_perception_epoch import (
|
||||
_artifact,
|
||||
_canonical_json,
|
||||
_GpuTelemetry,
|
||||
_percentiles,
|
||||
_sha256,
|
||||
_valid_sha256,
|
||||
_validate_job,
|
||||
_write_json,
|
||||
)
|
||||
|
||||
PROFILE_SCHEMA = "missioncore.e8-realtime-tracking-profile/v1"
|
||||
FRAME_SCHEMA = "missioncore.e8-realtime-tracking-frame/v1"
|
||||
TELEMETRY_SCHEMA = "missioncore.e8-realtime-tracking-telemetry/v1"
|
||||
REPORT_SCHEMA = "missioncore.e8-realtime-tracking-report/v1"
|
||||
RESULT_SCHEMA = "missioncore.e8-realtime-tracking-result/v1"
|
||||
IDENTITY_SCHEMA = "missioncore.e8-realtime-tracking-identity/v1"
|
||||
PIPELINE_ID = "source-paced-yolox-bytetrack-latest-wins/v1"
|
||||
|
||||
|
||||
def _arguments() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser()
|
||||
commands = parser.add_subparsers(dest="command", required=True)
|
||||
for name in ("preflight", "run"):
|
||||
command = commands.add_parser(name)
|
||||
command.add_argument("--job", type=Path, required=True)
|
||||
command.add_argument("--profile", type=Path, required=True)
|
||||
command.add_argument("--valid-fov-root", type=Path, required=True)
|
||||
command.add_argument("--model-root", type=Path, required=True)
|
||||
if name == "run":
|
||||
command.add_argument("--frames", type=Path, required=True)
|
||||
command.add_argument("--timeline", type=Path, required=True)
|
||||
command.add_argument("--output", type=Path, required=True)
|
||||
command.add_argument("--triton-url", required=True)
|
||||
command.add_argument("--free-bytes-floor", type=int, default=0)
|
||||
command.add_argument("--orchestrator-sha256", required=True)
|
||||
command.add_argument("--container-image", required=True)
|
||||
command.add_argument("--telemetry-interval-seconds", type=float, default=1.0)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def _read_object(path: Path) -> dict[str, Any]:
|
||||
value = json.loads(path.read_text(encoding="utf-8-sig"))
|
||||
if not isinstance(value, dict):
|
||||
raise RuntimeError(f"JSON root is not an object: {path}")
|
||||
return value
|
||||
|
||||
|
||||
def _read_profile(path: Path) -> tuple[dict[str, Any], str]:
|
||||
resolved = path.resolve(strict=True)
|
||||
profile = _read_object(resolved)
|
||||
if profile.get("schema_version") != PROFILE_SCHEMA:
|
||||
raise RuntimeError("LAB E8 profile schema changed")
|
||||
|
||||
# Reuse the exhaustively tested E5 detector/tracker validation by changing
|
||||
# only the schema in memory. No E5 defaults are inferred.
|
||||
e5_shape = dict(profile)
|
||||
e5_shape["schema_version"] = "missioncore.e5-tracking-profile/v1"
|
||||
temporary: Path | None = None
|
||||
try:
|
||||
with tempfile.NamedTemporaryFile(
|
||||
mode="w",
|
||||
encoding="utf-8",
|
||||
prefix="lab-e8-e5-validation-",
|
||||
suffix=".json",
|
||||
delete=False,
|
||||
) as stream:
|
||||
json.dump(e5_shape, stream, sort_keys=True, separators=(",", ":"))
|
||||
temporary = Path(stream.name)
|
||||
_read_e5_profile(temporary)
|
||||
finally:
|
||||
if temporary is not None:
|
||||
temporary.unlink(missing_ok=True)
|
||||
|
||||
realtime = profile.get("realtime")
|
||||
acceptance = profile.get("acceptance")
|
||||
if not isinstance(realtime, dict) or not isinstance(acceptance, dict):
|
||||
raise RuntimeError("LAB E8 realtime/acceptance configuration is missing")
|
||||
mode = profile.get("mode")
|
||||
capacity = realtime.get("queue_capacity")
|
||||
speed = realtime.get("speed")
|
||||
delay = realtime.get("consumer_delay_ms")
|
||||
stale = realtime.get("stale_after_ms")
|
||||
unavailable = realtime.get("unavailable_after_ms")
|
||||
if (
|
||||
mode not in {"pilot", "qualification", "overload-negative-control"}
|
||||
or realtime.get("queue_policy") != "bounded-latest-wins"
|
||||
or not isinstance(capacity, int)
|
||||
or isinstance(capacity, bool)
|
||||
or not 1 <= capacity <= 8
|
||||
or not isinstance(speed, int | float)
|
||||
or isinstance(speed, bool)
|
||||
or not 0.1 <= float(speed) <= 10.0
|
||||
or not isinstance(delay, int | float)
|
||||
or isinstance(delay, bool)
|
||||
or not 0 <= float(delay) <= 5000
|
||||
or not isinstance(stale, int | float)
|
||||
or not isinstance(unavailable, int | float)
|
||||
or not 0 < float(stale) < float(unavailable)
|
||||
):
|
||||
raise RuntimeError("LAB E8 scheduling configuration is invalid")
|
||||
numeric_acceptance = (
|
||||
acceptance.get("minimum_effective_fps"),
|
||||
acceptance.get("maximum_drop_fraction"),
|
||||
acceptance.get("maximum_p95_result_age_ms"),
|
||||
)
|
||||
if (
|
||||
any(
|
||||
not isinstance(value, int | float)
|
||||
or isinstance(value, bool)
|
||||
or not math.isfinite(float(value))
|
||||
or float(value) < 0
|
||||
for value in numeric_acceptance
|
||||
)
|
||||
or float(acceptance["maximum_drop_fraction"]) > 1
|
||||
or not isinstance(acceptance.get("require_zero_failures"), bool)
|
||||
or not isinstance(acceptance.get("expect_overload"), bool)
|
||||
):
|
||||
raise RuntimeError("LAB E8 acceptance configuration is invalid")
|
||||
return profile, _sha256(resolved)
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class FrameEnvelope:
|
||||
frame_index: int
|
||||
path: Path
|
||||
timeline: dict[str, Any]
|
||||
scheduled_monotonic: float
|
||||
decoded_monotonic: float
|
||||
image: np.ndarray
|
||||
decode_ms: float
|
||||
source_release_lag_ms: float
|
||||
|
||||
|
||||
class LatestWinsQueue:
|
||||
def __init__(self, capacity: int) -> None:
|
||||
self.capacity = capacity
|
||||
self.items: deque[FrameEnvelope] = deque()
|
||||
self.condition = threading.Condition()
|
||||
self.maximum_depth = 0
|
||||
self.published = 0
|
||||
self.consumed = 0
|
||||
self.dropped_overflow = 0
|
||||
self.closed = False
|
||||
|
||||
def publish(self, item: FrameEnvelope) -> None:
|
||||
with self.condition:
|
||||
if self.closed:
|
||||
raise RuntimeError("cannot publish to a closed LAB E8 queue")
|
||||
if len(self.items) == self.capacity:
|
||||
self.items.popleft()
|
||||
self.dropped_overflow += 1
|
||||
self.items.append(item)
|
||||
self.published += 1
|
||||
self.maximum_depth = max(self.maximum_depth, len(self.items))
|
||||
self.condition.notify()
|
||||
|
||||
def take(self) -> FrameEnvelope | None:
|
||||
with self.condition:
|
||||
self.condition.wait_for(lambda: bool(self.items) or self.closed)
|
||||
if not self.items:
|
||||
return None
|
||||
value = self.items.popleft()
|
||||
self.consumed += 1
|
||||
return value
|
||||
|
||||
def close(self) -> None:
|
||||
with self.condition:
|
||||
self.closed = True
|
||||
self.condition.notify_all()
|
||||
|
||||
def snapshot(self) -> dict[str, int | bool]:
|
||||
with self.condition:
|
||||
return {
|
||||
"capacity": self.capacity,
|
||||
"final_depth": len(self.items),
|
||||
"maximum_depth": self.maximum_depth,
|
||||
"published": self.published,
|
||||
"consumed": self.consumed,
|
||||
"dropped_overflow": self.dropped_overflow,
|
||||
"closed": self.closed,
|
||||
}
|
||||
|
||||
|
||||
def _wait_until(deadline: float) -> None:
|
||||
while True:
|
||||
remaining = deadline - time.perf_counter()
|
||||
if remaining <= 0:
|
||||
return
|
||||
time.sleep(min(remaining, 0.02))
|
||||
|
||||
|
||||
def _health(age_ms: float, realtime: dict[str, Any]) -> str:
|
||||
if age_ms >= float(realtime["unavailable_after_ms"]):
|
||||
return "unavailable"
|
||||
if age_ms >= float(realtime["stale_after_ms"]):
|
||||
return "stale"
|
||||
return "healthy"
|
||||
|
||||
|
||||
def _producer(
|
||||
*,
|
||||
queue: LatestWinsQueue,
|
||||
frame_paths: list[Path],
|
||||
timeline_rows: list[dict[str, Any]],
|
||||
replay_started: float,
|
||||
speed: float,
|
||||
error: list[BaseException],
|
||||
) -> None:
|
||||
from PIL import Image
|
||||
|
||||
first_session = float(timeline_rows[0]["session_seconds"])
|
||||
try:
|
||||
for frame_index, (path, row) in enumerate(zip(frame_paths, timeline_rows, strict=True)):
|
||||
scheduled = replay_started + (float(row["session_seconds"]) - first_session) / speed
|
||||
_wait_until(scheduled)
|
||||
decode_started = time.perf_counter()
|
||||
with Image.open(path) as opened:
|
||||
image = np.asarray(opened.convert("RGB"), dtype=np.uint8)
|
||||
decoded = time.perf_counter()
|
||||
if image.shape != (600, 800, 3):
|
||||
raise RuntimeError("LAB E8 input resolution changed")
|
||||
queue.publish(
|
||||
FrameEnvelope(
|
||||
frame_index=frame_index,
|
||||
path=path,
|
||||
timeline=row,
|
||||
scheduled_monotonic=scheduled,
|
||||
decoded_monotonic=decoded,
|
||||
image=image,
|
||||
decode_ms=(decoded - decode_started) * 1000.0,
|
||||
source_release_lag_ms=max(0.0, (decode_started - scheduled) * 1000.0),
|
||||
)
|
||||
)
|
||||
except BaseException as exc:
|
||||
error.append(exc)
|
||||
finally:
|
||||
queue.close()
|
||||
|
||||
|
||||
def _assert_disk_floor(path: Path, floor: int, frame: int) -> None:
|
||||
if floor < 0:
|
||||
raise RuntimeError("LAB E8 disk floor is invalid")
|
||||
free = shutil.disk_usage(path).free
|
||||
if floor and free < floor:
|
||||
raise RuntimeError(f"LAB E8 crossed the D-backed free-space floor at frame {frame}")
|
||||
|
||||
|
||||
def _preflight(args: argparse.Namespace) -> int:
|
||||
job = _validate_job(args.job.resolve(strict=True))
|
||||
profile, profile_sha256 = _read_profile(args.profile)
|
||||
_validate_source(job, profile)
|
||||
valid_mask, valid_fov = _load_valid_fov(args.valid_fov_root, job, profile)
|
||||
model_files = _verify_model(profile, args.model_root)
|
||||
from scipy.optimize import linear_sum_assignment # noqa: F401
|
||||
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"state": "preflight-ready",
|
||||
"job_id": job["job_id"],
|
||||
"profile_sha256": profile_sha256,
|
||||
"model_files": model_files,
|
||||
"valid_fov": valid_fov,
|
||||
"valid_pixels": int(valid_mask.sum()),
|
||||
},
|
||||
sort_keys=True,
|
||||
),
|
||||
flush=True,
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
def _run(args: argparse.Namespace) -> int:
|
||||
from PIL import Image
|
||||
from scipy.optimize import linear_sum_assignment
|
||||
|
||||
if not _valid_sha256(args.orchestrator_sha256):
|
||||
raise RuntimeError("LAB E8 orchestrator SHA-256 is invalid")
|
||||
if not args.triton_url.startswith("http://") or len(args.triton_url) > 256:
|
||||
raise RuntimeError("LAB E8 Triton URL is invalid")
|
||||
job = _validate_job(args.job.resolve(strict=True))
|
||||
profile, profile_sha256 = _read_profile(args.profile)
|
||||
_validate_source(job, profile)
|
||||
valid_mask, valid_fov = _load_valid_fov(args.valid_fov_root, job, profile)
|
||||
model_files = _verify_model(profile, args.model_root)
|
||||
frames_root = args.frames.resolve(strict=True)
|
||||
frame_paths = sorted(frames_root.glob("frame-*.png"))
|
||||
if not frame_paths or [path.name for path in frame_paths] != [
|
||||
f"frame-{index:06d}.png" for index in range(1, len(frame_paths) + 1)
|
||||
]:
|
||||
raise RuntimeError("LAB E8 decoded frame sequence changed")
|
||||
timeline_path = args.timeline.resolve(strict=True)
|
||||
timeline_rows = _read_timeline(timeline_path, len(frame_paths))
|
||||
input_timeline = job["input"]["timeline"]
|
||||
if float(timeline_rows[0]["session_seconds"]) < float(input_timeline["start_seconds"]) or float(
|
||||
timeline_rows[-1]["session_seconds"]
|
||||
) > float(input_timeline["end_seconds"]):
|
||||
raise RuntimeError("LAB E8 clip escaped the source timeline")
|
||||
|
||||
output = args.output.resolve()
|
||||
if output.exists():
|
||||
raise RuntimeError("LAB E8 output must be absent")
|
||||
output.mkdir(mode=0o700, parents=True, exist_ok=False)
|
||||
_assert_disk_floor(output, args.free_bytes_floor, 0)
|
||||
|
||||
tracker = TwoStageTracker(profile["tracking"])
|
||||
linear_sum_assignment(np.zeros((1, 1), dtype=np.float64))
|
||||
with Image.open(frame_paths[0]) as opened:
|
||||
warm_image = np.asarray(opened.convert("RGB"), dtype=np.uint8)
|
||||
warm_tensor = _preprocess(warm_image, valid_mask, profile)
|
||||
_infer(args.triton_url, profile["model"], warm_tensor)
|
||||
del warm_tensor, warm_image
|
||||
|
||||
realtime = profile["realtime"]
|
||||
queue = LatestWinsQueue(int(realtime["queue_capacity"]))
|
||||
producer_errors: list[BaseException] = []
|
||||
latency = {
|
||||
name: []
|
||||
for name in (
|
||||
"image_decode_ms",
|
||||
"source_release_lag_ms",
|
||||
"queue_wait_ms",
|
||||
"preprocess_ms",
|
||||
"triton_request_ms",
|
||||
"postprocess_ms",
|
||||
"tracking_ms",
|
||||
"processing_ms",
|
||||
"result_age_ms",
|
||||
)
|
||||
}
|
||||
detection_labels: Counter[str] = Counter()
|
||||
track_labels: Counter[str] = Counter()
|
||||
rejection_reasons: Counter[str] = Counter()
|
||||
health_counts: Counter[str] = Counter()
|
||||
unique_track_ids: set[int] = set()
|
||||
duplicate_pairs = 0
|
||||
failures = 0
|
||||
metadata_path = output / "frames.jsonl"
|
||||
telemetry_path = output / "telemetry.jsonl"
|
||||
gpu_path = output / "gpu-telemetry.jsonl"
|
||||
disk_before = shutil.disk_usage(output).free
|
||||
replay_started = time.perf_counter() + 0.25
|
||||
producer = threading.Thread(
|
||||
target=_producer,
|
||||
kwargs={
|
||||
"queue": queue,
|
||||
"frame_paths": frame_paths,
|
||||
"timeline_rows": timeline_rows,
|
||||
"replay_started": replay_started,
|
||||
"speed": float(realtime["speed"]),
|
||||
"error": producer_errors,
|
||||
},
|
||||
name="lab-e8-source-producer",
|
||||
daemon=True,
|
||||
)
|
||||
producer.start()
|
||||
|
||||
with (
|
||||
metadata_path.open("x", encoding="utf-8", newline="\n") as metadata_stream,
|
||||
telemetry_path.open("x", encoding="utf-8", newline="\n") as telemetry_stream,
|
||||
gpu_path.open("x", encoding="utf-8", newline="\n") as gpu_stream,
|
||||
_GpuTelemetry(gpu_stream, args.telemetry_interval_seconds) as gpu_telemetry,
|
||||
):
|
||||
while (envelope := queue.take()) is not None:
|
||||
processing_started = time.perf_counter()
|
||||
latency["image_decode_ms"].append(envelope.decode_ms)
|
||||
latency["source_release_lag_ms"].append(envelope.source_release_lag_ms)
|
||||
latency["queue_wait_ms"].append(
|
||||
max(0.0, (processing_started - envelope.decoded_monotonic) * 1000.0)
|
||||
)
|
||||
try:
|
||||
preprocess_started = time.perf_counter()
|
||||
tensor = _preprocess(envelope.image, valid_mask, profile)
|
||||
latency["preprocess_ms"].append((time.perf_counter() - preprocess_started) * 1000.0)
|
||||
output_tensor, request_ms = _infer(args.triton_url, profile["model"], tensor)
|
||||
latency["triton_request_ms"].append(request_ms)
|
||||
|
||||
postprocess_started = time.perf_counter()
|
||||
detections, rejected = _detections(output_tensor, profile, valid_mask)
|
||||
latency["postprocess_ms"].append(
|
||||
(time.perf_counter() - postprocess_started) * 1000.0
|
||||
)
|
||||
rejection_reasons.update(rejected)
|
||||
detection_labels.update(str(item["label"]) for item in detections)
|
||||
|
||||
tracking_started = time.perf_counter()
|
||||
tracks = tracker.update(detections, envelope.frame_index)
|
||||
latency["tracking_ms"].append((time.perf_counter() - tracking_started) * 1000.0)
|
||||
duplicate_pairs += _duplicate_pairs(tracks)
|
||||
for track in tracks:
|
||||
unique_track_ids.add(track.track_id)
|
||||
track_labels[track.label] += 1
|
||||
delay_seconds = float(realtime["consumer_delay_ms"]) / 1000.0
|
||||
if delay_seconds:
|
||||
time.sleep(delay_seconds)
|
||||
except Exception:
|
||||
failures += 1
|
||||
raise
|
||||
|
||||
published = time.perf_counter()
|
||||
processing_ms = (published - processing_started) * 1000.0
|
||||
result_age_ms = max(0.0, (published - envelope.scheduled_monotonic) * 1000.0)
|
||||
latency["processing_ms"].append(processing_ms)
|
||||
latency["result_age_ms"].append(result_age_ms)
|
||||
health = _health(result_age_ms, realtime)
|
||||
health_counts[health] += 1
|
||||
frame_document = {
|
||||
"schema_version": FRAME_SCHEMA,
|
||||
"frame_index": envelope.frame_index,
|
||||
"sequence": envelope.frame_index + 1,
|
||||
"source_frame_index": envelope.timeline["source_frame_index"],
|
||||
"source_sequence": envelope.timeline["source_sequence"],
|
||||
"session_seconds": round(float(envelope.timeline["session_seconds"]), 9),
|
||||
"detections": detections,
|
||||
"tracks": [_track_document(track) for track in tracks],
|
||||
"delivery": {"health": health, "result_age_ms": round(result_age_ms, 6)},
|
||||
}
|
||||
metadata_stream.write(
|
||||
json.dumps(
|
||||
frame_document,
|
||||
ensure_ascii=False,
|
||||
sort_keys=True,
|
||||
separators=(",", ":"),
|
||||
allow_nan=False,
|
||||
)
|
||||
+ "\n"
|
||||
)
|
||||
queue_state = queue.snapshot()
|
||||
telemetry_stream.write(
|
||||
json.dumps(
|
||||
{
|
||||
"schema_version": TELEMETRY_SCHEMA,
|
||||
"frame_index": envelope.frame_index,
|
||||
"session_seconds": frame_document["session_seconds"],
|
||||
"health": health,
|
||||
"result_age_ms": round(result_age_ms, 6),
|
||||
"processing_ms": round(processing_ms, 6),
|
||||
"queue_depth_after_take": queue_state["final_depth"],
|
||||
"queue_dropped_overflow": queue_state["dropped_overflow"],
|
||||
},
|
||||
sort_keys=True,
|
||||
separators=(",", ":"),
|
||||
)
|
||||
+ "\n"
|
||||
)
|
||||
completed = int(queue_state["consumed"])
|
||||
if completed % 100 == 0:
|
||||
metadata_stream.flush()
|
||||
telemetry_stream.flush()
|
||||
gpu_stream.flush()
|
||||
_assert_disk_floor(output, args.free_bytes_floor, completed)
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"phase": "realtime-tracking",
|
||||
"frames_consumed": completed,
|
||||
"frames_published": queue_state["published"],
|
||||
"frames_dropped": queue_state["dropped_overflow"],
|
||||
"maximum_queue_depth": queue_state["maximum_depth"],
|
||||
},
|
||||
sort_keys=True,
|
||||
),
|
||||
flush=True,
|
||||
)
|
||||
producer.join(timeout=5)
|
||||
if producer.is_alive():
|
||||
raise RuntimeError("LAB E8 producer did not terminate")
|
||||
if producer_errors:
|
||||
raise RuntimeError("LAB E8 producer failed") from producer_errors[0]
|
||||
metadata_stream.flush()
|
||||
telemetry_stream.flush()
|
||||
os.fsync(metadata_stream.fileno())
|
||||
os.fsync(telemetry_stream.fileno())
|
||||
|
||||
finished = time.perf_counter()
|
||||
queue_state = queue.snapshot()
|
||||
source_span = (
|
||||
float(timeline_rows[-1]["session_seconds"]) - float(timeline_rows[0]["session_seconds"])
|
||||
) / float(realtime["speed"])
|
||||
wall_seconds = finished - replay_started
|
||||
effective_fps = int(queue_state["consumed"]) / max(source_span, wall_seconds, 1e-9)
|
||||
drop_fraction = int(queue_state["dropped_overflow"]) / len(frame_paths)
|
||||
result_age = _percentiles(latency["result_age_ms"])
|
||||
acceptance = profile["acceptance"]
|
||||
expect_overload = bool(acceptance["expect_overload"])
|
||||
checks = {
|
||||
"queue_bounded": int(queue_state["maximum_depth"]) <= int(queue_state["capacity"]),
|
||||
"producer_accounting": int(queue_state["published"]) == len(frame_paths),
|
||||
"consumer_accounting": (
|
||||
int(queue_state["consumed"]) + int(queue_state["dropped_overflow"]) == len(frame_paths)
|
||||
),
|
||||
"zero_failures": failures == 0,
|
||||
}
|
||||
if expect_overload:
|
||||
checks["overload_drop_observed"] = int(queue_state["dropped_overflow"]) > 0
|
||||
checks["overload_degraded_or_stale_observed"] = (
|
||||
health_counts["stale"] + health_counts["unavailable"] > 0
|
||||
)
|
||||
else:
|
||||
checks.update(
|
||||
{
|
||||
"minimum_effective_fps": effective_fps
|
||||
>= float(acceptance["minimum_effective_fps"]),
|
||||
"maximum_drop_fraction": drop_fraction
|
||||
<= float(acceptance["maximum_drop_fraction"]),
|
||||
"maximum_p95_result_age_ms": float(result_age["p95"])
|
||||
<= float(acceptance["maximum_p95_result_age_ms"]),
|
||||
}
|
||||
)
|
||||
accepted = all(checks.values())
|
||||
|
||||
identity = {
|
||||
"schema_version": IDENTITY_SCHEMA,
|
||||
"job_id": job["job_id"],
|
||||
"input_sha256": job["input_sha256"],
|
||||
"session_id": job["input"]["session_id"],
|
||||
"source_id": job["input"]["source_id"],
|
||||
"selection": {
|
||||
"frame_count": len(frame_paths),
|
||||
"source_start_frame_index": timeline_rows[0]["source_frame_index"],
|
||||
"source_end_frame_index": timeline_rows[-1]["source_frame_index"],
|
||||
"timeline_start_seconds": timeline_rows[0]["session_seconds"],
|
||||
"timeline_end_seconds": timeline_rows[-1]["session_seconds"],
|
||||
"timeline_sha256": _sha256(timeline_path),
|
||||
},
|
||||
"configuration": {
|
||||
"pipeline": PIPELINE_ID,
|
||||
"profile_sha256": profile_sha256,
|
||||
"profile": profile,
|
||||
"runner_sha256": _sha256(Path(__file__).resolve(strict=True)),
|
||||
"orchestrator_sha256": args.orchestrator_sha256,
|
||||
"container_image": args.container_image,
|
||||
"valid_fov": valid_fov,
|
||||
},
|
||||
"models": {"detector": profile["model"], "files": model_files},
|
||||
}
|
||||
identity_sha256 = hashlib.sha256(_canonical_json(identity)).hexdigest()
|
||||
result_id = f"e8-realtime-tracking-{identity_sha256}"
|
||||
metrics = {
|
||||
"frames_expected": len(frame_paths),
|
||||
"frames_published": queue_state["published"],
|
||||
"frames_processed": queue_state["consumed"],
|
||||
"frames_dropped": queue_state["dropped_overflow"],
|
||||
"frames_failed": failures,
|
||||
"drop_fraction": round(drop_fraction, 9),
|
||||
"source_span_seconds": round(source_span, 6),
|
||||
"replay_wall_seconds": round(wall_seconds, 6),
|
||||
"effective_frames_per_second": round(effective_fps, 6),
|
||||
"queue": queue_state,
|
||||
"latency_ms": {name: _percentiles(values) for name, values in latency.items()},
|
||||
"health_counts": dict(sorted(health_counts.items())),
|
||||
"detections": int(sum(detection_labels.values())),
|
||||
"detections_by_label": dict(sorted(detection_labels.items())),
|
||||
"detection_rejections": dict(sorted(rejection_reasons.items())),
|
||||
"unique_confirmed_tracks": len(unique_track_ids),
|
||||
"track_observations": int(sum(track_labels.values())),
|
||||
"track_observations_by_label": dict(sorted(track_labels.items())),
|
||||
"same_class_duplicate_pairs_iou_ge_0_8": duplicate_pairs,
|
||||
"tracker_tracks_created": tracker.created,
|
||||
"tracker_tracks_retired": tracker.retired,
|
||||
"process_peak_rss_mib": round(resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024, 3),
|
||||
"gpu_telemetry": gpu_telemetry.summary(),
|
||||
"disk": {
|
||||
"free_bytes_before_replay": disk_before,
|
||||
"free_bytes_after_replay": shutil.disk_usage(output).free,
|
||||
"free_bytes_floor": args.free_bytes_floor,
|
||||
},
|
||||
}
|
||||
report = {
|
||||
"schema_version": REPORT_SCHEMA,
|
||||
"result_id": result_id,
|
||||
"created_at_utc": datetime.now(UTC)
|
||||
.isoformat(timespec="milliseconds")
|
||||
.replace("+00:00", "Z"),
|
||||
"state": "accepted" if accepted else "rejected",
|
||||
"ground_truth": False,
|
||||
"identity": identity,
|
||||
"runtime": {
|
||||
"hostname": platform.node(),
|
||||
"platform": platform.platform(),
|
||||
"python": platform.python_version(),
|
||||
"numpy": np.__version__,
|
||||
},
|
||||
"metrics": metrics,
|
||||
"acceptance": {
|
||||
"accepted": accepted,
|
||||
"checks": checks,
|
||||
"navigation_or_safety_accepted": False,
|
||||
},
|
||||
"limitations": [
|
||||
"Recorded source-paced replay, not a live K1 transport.",
|
||||
"Generic COCO detector and IoU-only tracking are not safety validated.",
|
||||
"No semantic model, LiDAR fusion or navigation actuation is executed in this gate.",
|
||||
(
|
||||
"Decoded PNG input is temporary lab transport; camera ingest decode "
|
||||
"remains a separate gate."
|
||||
),
|
||||
],
|
||||
}
|
||||
report_path = output / "run-report.json"
|
||||
_write_json(report_path, report)
|
||||
artifacts = [
|
||||
_artifact(metadata_path, "realtime-tracking-frames", "application/x-ndjson", FRAME_SCHEMA),
|
||||
_artifact(
|
||||
telemetry_path, "realtime-tracking-telemetry", "application/x-ndjson", TELEMETRY_SCHEMA
|
||||
),
|
||||
_artifact(gpu_path, "worker-gpu-telemetry", "application/x-ndjson"),
|
||||
_artifact(report_path, "realtime-tracking-report", "application/json", REPORT_SCHEMA),
|
||||
]
|
||||
result = {
|
||||
"schema_version": RESULT_SCHEMA,
|
||||
"result_id": result_id,
|
||||
"identity_sha256": identity_sha256,
|
||||
"identity": identity,
|
||||
"created_at_utc": report["created_at_utc"],
|
||||
"acceptance_state": report["state"],
|
||||
"ground_truth": False,
|
||||
"publication_scope": "recorded-realtime-qualification-only",
|
||||
"frames_processed": queue_state["consumed"],
|
||||
"artifacts": artifacts,
|
||||
}
|
||||
_write_json(output / "result.json", result)
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"result_id": result_id,
|
||||
"accepted": accepted,
|
||||
"frames_processed": queue_state["consumed"],
|
||||
"frames_dropped": queue_state["dropped_overflow"],
|
||||
"effective_fps": round(effective_fps, 6),
|
||||
"result_age_p95_ms": result_age["p95"],
|
||||
},
|
||||
sort_keys=True,
|
||||
),
|
||||
flush=True,
|
||||
)
|
||||
return 0 if accepted else 2
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = _arguments()
|
||||
if args.command == "preflight":
|
||||
return _preflight(args)
|
||||
return _run(args)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,933 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Qualify concurrent 10 Hz tracking and lower-rate semantic perception."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import importlib.metadata
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import platform
|
||||
import resource
|
||||
import shutil
|
||||
import threading
|
||||
import time
|
||||
from collections import Counter
|
||||
from dataclasses import dataclass
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import numpy as np
|
||||
from run_e4_full_session_segmentation import (
|
||||
TARGET_CLASS_COUNT,
|
||||
_dependency_manifest,
|
||||
_load_model,
|
||||
)
|
||||
from run_e4_full_session_segmentation import (
|
||||
_profile as _read_semantic_profile,
|
||||
)
|
||||
from run_e4_full_session_segmentation import (
|
||||
_validate_source as _validate_semantic_source,
|
||||
)
|
||||
from run_e5_instance_tracking import (
|
||||
TwoStageTracker,
|
||||
_detections,
|
||||
_duplicate_pairs,
|
||||
_infer,
|
||||
_load_valid_fov,
|
||||
_preprocess,
|
||||
_read_timeline,
|
||||
_track_document,
|
||||
_validate_source,
|
||||
_verify_model,
|
||||
)
|
||||
from run_e8_realtime_tracking import (
|
||||
FrameEnvelope,
|
||||
LatestWinsQueue,
|
||||
_wait_until,
|
||||
)
|
||||
from run_e8_realtime_tracking import (
|
||||
_read_profile as _read_detector_profile,
|
||||
)
|
||||
from run_recorded_perception_epoch import (
|
||||
_artifact,
|
||||
_canonical_json,
|
||||
_GpuTelemetry,
|
||||
_percentiles,
|
||||
_sha256,
|
||||
_valid_sha256,
|
||||
_validate_job,
|
||||
_write_json,
|
||||
)
|
||||
|
||||
PROFILE_SCHEMA = "missioncore.e9-multirate-perception-profile/v1"
|
||||
DETECTOR_FRAME_SCHEMA = "missioncore.e9-multirate-detector-frame/v1"
|
||||
SEMANTIC_FRAME_SCHEMA = "missioncore.e9-multirate-semantic-frame/v1"
|
||||
MERGED_FRAME_SCHEMA = "missioncore.e9-multirate-merged-frame/v1"
|
||||
REPORT_SCHEMA = "missioncore.e9-multirate-perception-report/v1"
|
||||
RESULT_SCHEMA = "missioncore.e9-multirate-perception-result/v1"
|
||||
IDENTITY_SCHEMA = "missioncore.e9-multirate-perception-identity/v1"
|
||||
PIPELINE_ID = "concurrent-yolox-tracking-eomt-semantic-latest-wins/v1"
|
||||
|
||||
|
||||
def _arguments() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser()
|
||||
commands = parser.add_subparsers(dest="command", required=True)
|
||||
for name in ("preflight", "run"):
|
||||
command = commands.add_parser(name)
|
||||
command.add_argument("--job", type=Path, required=True)
|
||||
command.add_argument("--profile", type=Path, required=True)
|
||||
command.add_argument("--detector-profile", type=Path, required=True)
|
||||
command.add_argument("--semantic-profile", type=Path, required=True)
|
||||
command.add_argument("--valid-fov-root", type=Path, required=True)
|
||||
command.add_argument("--model-root", type=Path, required=True)
|
||||
command.add_argument("--cache", type=Path, required=True)
|
||||
command.add_argument("--environment", type=Path, required=True)
|
||||
if name == "run":
|
||||
command.add_argument("--frames", type=Path, required=True)
|
||||
command.add_argument("--timeline", type=Path, required=True)
|
||||
command.add_argument("--output", type=Path, required=True)
|
||||
command.add_argument("--triton-url", required=True)
|
||||
command.add_argument("--free-bytes-floor", type=int, default=0)
|
||||
command.add_argument("--orchestrator-sha256", required=True)
|
||||
command.add_argument("--container-image", required=True)
|
||||
command.add_argument("--telemetry-interval-seconds", type=float, default=1.0)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def _read_object(path: Path) -> dict[str, Any]:
|
||||
value = json.loads(path.read_text(encoding="utf-8-sig"))
|
||||
if not isinstance(value, dict):
|
||||
raise RuntimeError(f"JSON root is not an object: {path}")
|
||||
return value
|
||||
|
||||
|
||||
def _read_profile(path: Path) -> tuple[dict[str, Any], str]:
|
||||
resolved = path.resolve(strict=True)
|
||||
profile = _read_object(resolved)
|
||||
replay = profile.get("replay")
|
||||
acceptance = profile.get("acceptance")
|
||||
source = profile.get("source")
|
||||
if (
|
||||
profile.get("schema_version") != PROFILE_SCHEMA
|
||||
or profile.get("mode") not in {"pilot", "qualification"}
|
||||
or not isinstance(replay, dict)
|
||||
or not isinstance(acceptance, dict)
|
||||
or not isinstance(source, dict)
|
||||
or source.get("source_id") != "sensor.camera.right"
|
||||
or source.get("resolution") != [800, 600]
|
||||
):
|
||||
raise RuntimeError("LAB E9 profile contract changed")
|
||||
speed = replay.get("speed")
|
||||
detector_capacity = replay.get("detector_queue_capacity")
|
||||
semantic_capacity = replay.get("semantic_queue_capacity")
|
||||
semantic_stride = replay.get("semantic_sample_every_frames")
|
||||
semantic_ttl = replay.get("semantic_ttl_ms")
|
||||
if (
|
||||
not _positive_number(speed)
|
||||
or not 0.1 <= float(speed) <= 10
|
||||
or not isinstance(detector_capacity, int)
|
||||
or isinstance(detector_capacity, bool)
|
||||
or not 1 <= detector_capacity <= 8
|
||||
or not isinstance(semantic_capacity, int)
|
||||
or isinstance(semantic_capacity, bool)
|
||||
or not 1 <= semantic_capacity <= 4
|
||||
or not isinstance(semantic_stride, int)
|
||||
or isinstance(semantic_stride, bool)
|
||||
or not 2 <= semantic_stride <= 30
|
||||
or not _positive_number(semantic_ttl)
|
||||
or not 100 <= float(semantic_ttl) <= 5000
|
||||
):
|
||||
raise RuntimeError("LAB E9 replay configuration is invalid")
|
||||
required = (
|
||||
"detector_minimum_effective_fps",
|
||||
"detector_maximum_drop_fraction",
|
||||
"detector_maximum_p95_result_age_ms",
|
||||
"semantic_minimum_effective_fps",
|
||||
"semantic_maximum_drop_fraction",
|
||||
"semantic_maximum_p95_completion_age_ms",
|
||||
"minimum_fresh_semantic_coverage",
|
||||
)
|
||||
if (
|
||||
any(not _nonnegative_number(acceptance.get(name)) for name in required)
|
||||
or float(acceptance["detector_maximum_drop_fraction"]) > 1
|
||||
or float(acceptance["semantic_maximum_drop_fraction"]) > 1
|
||||
or float(acceptance["minimum_fresh_semantic_coverage"]) > 1
|
||||
or not isinstance(acceptance.get("require_zero_failures"), bool)
|
||||
):
|
||||
raise RuntimeError("LAB E9 acceptance configuration is invalid")
|
||||
return profile, _sha256(resolved)
|
||||
|
||||
|
||||
def _positive_number(value: object) -> bool:
|
||||
return _nonnegative_number(value) and float(value) > 0
|
||||
|
||||
|
||||
def _nonnegative_number(value: object) -> bool:
|
||||
return (
|
||||
isinstance(value, int | float)
|
||||
and not isinstance(value, bool)
|
||||
and math.isfinite(float(value))
|
||||
and float(value) >= 0
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class SemanticResult:
|
||||
frame_index: int
|
||||
source_frame_index: int
|
||||
session_seconds: float
|
||||
completed_monotonic: float
|
||||
completion_age_ms: float
|
||||
mask_sha256: str
|
||||
class_pixels: dict[str, int]
|
||||
class_fractions: dict[str, float]
|
||||
|
||||
|
||||
class LatestSemantic:
|
||||
def __init__(self) -> None:
|
||||
self._lock = threading.Lock()
|
||||
self._value: SemanticResult | None = None
|
||||
|
||||
def publish(self, value: SemanticResult) -> None:
|
||||
with self._lock:
|
||||
if self._value is not None and value.frame_index <= self._value.frame_index:
|
||||
raise RuntimeError("LAB E9 semantic results are not monotonic")
|
||||
self._value = value
|
||||
|
||||
def snapshot(self) -> SemanticResult | None:
|
||||
with self._lock:
|
||||
return self._value
|
||||
|
||||
|
||||
def _producer(
|
||||
*,
|
||||
detector_queue: LatestWinsQueue,
|
||||
semantic_queue: LatestWinsQueue,
|
||||
semantic_stride: int,
|
||||
frame_paths: list[Path],
|
||||
timeline_rows: list[dict[str, Any]],
|
||||
replay_started: float,
|
||||
speed: float,
|
||||
error: list[BaseException],
|
||||
) -> None:
|
||||
from PIL import Image
|
||||
|
||||
first_session = float(timeline_rows[0]["session_seconds"])
|
||||
try:
|
||||
for frame_index, (path, row) in enumerate(zip(frame_paths, timeline_rows, strict=True)):
|
||||
scheduled = replay_started + (float(row["session_seconds"]) - first_session) / speed
|
||||
_wait_until(scheduled)
|
||||
decode_started = time.perf_counter()
|
||||
with Image.open(path) as opened:
|
||||
image = np.asarray(opened.convert("RGB"), dtype=np.uint8)
|
||||
decoded = time.perf_counter()
|
||||
if image.shape != (600, 800, 3):
|
||||
raise RuntimeError("LAB E9 input resolution changed")
|
||||
envelope = FrameEnvelope(
|
||||
frame_index=frame_index,
|
||||
path=path,
|
||||
timeline=row,
|
||||
scheduled_monotonic=scheduled,
|
||||
decoded_monotonic=decoded,
|
||||
image=image,
|
||||
decode_ms=(decoded - decode_started) * 1000.0,
|
||||
source_release_lag_ms=max(0.0, (decode_started - scheduled) * 1000.0),
|
||||
)
|
||||
detector_queue.publish(envelope)
|
||||
if frame_index % semantic_stride == 0:
|
||||
semantic_queue.publish(envelope)
|
||||
except BaseException as exc:
|
||||
error.append(exc)
|
||||
finally:
|
||||
detector_queue.close()
|
||||
semantic_queue.close()
|
||||
|
||||
|
||||
def _semantic_worker(
|
||||
*,
|
||||
queue: LatestWinsQueue,
|
||||
latest: LatestSemantic,
|
||||
stream: Any,
|
||||
valid_mask: np.ndarray,
|
||||
target_lut: np.ndarray,
|
||||
target_names: dict[int, str],
|
||||
infer: Any,
|
||||
latency: dict[str, list[float]],
|
||||
failures: list[BaseException],
|
||||
) -> None:
|
||||
try:
|
||||
while (envelope := queue.take()) is not None:
|
||||
started = time.perf_counter()
|
||||
latency["image_decode_ms"].append(envelope.decode_ms)
|
||||
latency["queue_wait_ms"].append(
|
||||
max(0.0, (started - envelope.decoded_monotonic) * 1000.0)
|
||||
)
|
||||
fill_started = time.perf_counter()
|
||||
model_input = np.where(valid_mask[..., None], envelope.image, 0).astype(np.uint8)
|
||||
latency["valid_fov_fill_ms"].append((time.perf_counter() - fill_started) * 1000.0)
|
||||
semantic, measured = infer(model_input)
|
||||
if semantic.max() >= len(target_lut):
|
||||
raise RuntimeError("LAB E9 EoMT emitted an unknown category")
|
||||
target = target_lut[semantic]
|
||||
target = target.copy()
|
||||
target[~valid_mask] = 0
|
||||
for name, value in measured.items():
|
||||
latency[name].append(float(value))
|
||||
completed = time.perf_counter()
|
||||
completion_age_ms = max(0.0, (completed - envelope.scheduled_monotonic) * 1000.0)
|
||||
latency["completion_age_ms"].append(completion_age_ms)
|
||||
latency["processing_ms"].append((completed - started) * 1000.0)
|
||||
counts = np.bincount(target[valid_mask], minlength=TARGET_CLASS_COUNT)
|
||||
class_pixels = {
|
||||
target_names[index]: int(counts[index])
|
||||
for index in range(1, TARGET_CLASS_COUNT)
|
||||
if int(counts[index]) > 0
|
||||
}
|
||||
valid_pixels = int(valid_mask.sum())
|
||||
result = SemanticResult(
|
||||
frame_index=envelope.frame_index,
|
||||
source_frame_index=int(envelope.timeline["source_frame_index"]),
|
||||
session_seconds=float(envelope.timeline["session_seconds"]),
|
||||
completed_monotonic=completed,
|
||||
completion_age_ms=completion_age_ms,
|
||||
mask_sha256=hashlib.sha256(target.tobytes()).hexdigest(),
|
||||
class_pixels=class_pixels,
|
||||
class_fractions={
|
||||
name: round(count / valid_pixels, 9) for name, count in class_pixels.items()
|
||||
},
|
||||
)
|
||||
latest.publish(result)
|
||||
stream.write(
|
||||
json.dumps(
|
||||
{
|
||||
"schema_version": SEMANTIC_FRAME_SCHEMA,
|
||||
"frame_index": result.frame_index,
|
||||
"source_frame_index": result.source_frame_index,
|
||||
"session_seconds": round(result.session_seconds, 9),
|
||||
"completion_age_ms": round(result.completion_age_ms, 6),
|
||||
"mask_sha256": result.mask_sha256,
|
||||
"class_pixels": result.class_pixels,
|
||||
"class_fractions": result.class_fractions,
|
||||
},
|
||||
sort_keys=True,
|
||||
separators=(",", ":"),
|
||||
allow_nan=False,
|
||||
)
|
||||
+ "\n"
|
||||
)
|
||||
except BaseException as exc:
|
||||
failures.append(exc)
|
||||
|
||||
|
||||
def _semantic_binding(
|
||||
semantic: SemanticResult | None,
|
||||
*,
|
||||
frame_session_seconds: float,
|
||||
ttl_ms: float,
|
||||
) -> dict[str, Any]:
|
||||
if semantic is None:
|
||||
return {
|
||||
"status": "unavailable",
|
||||
"source_frame_index": None,
|
||||
"source_age_ms": None,
|
||||
"completion_age_ms": None,
|
||||
"mask_sha256": None,
|
||||
}
|
||||
source_age_ms = max(0.0, (frame_session_seconds - semantic.session_seconds) * 1000.0)
|
||||
return {
|
||||
"status": "fresh" if source_age_ms <= ttl_ms else "stale",
|
||||
"source_frame_index": semantic.source_frame_index,
|
||||
"source_age_ms": round(source_age_ms, 6),
|
||||
"completion_age_ms": round(semantic.completion_age_ms, 6),
|
||||
"mask_sha256": semantic.mask_sha256,
|
||||
}
|
||||
|
||||
|
||||
def _semantic_infer_factory(
|
||||
processor: Any,
|
||||
model: Any,
|
||||
device: Any,
|
||||
) -> Any:
|
||||
import torch
|
||||
from PIL import Image
|
||||
|
||||
def infer(image: np.ndarray) -> tuple[np.ndarray, dict[str, float]]:
|
||||
processor_started = time.perf_counter()
|
||||
inputs = processor(images=Image.fromarray(image), return_tensors="pt")
|
||||
processor_ms = (time.perf_counter() - processor_started) * 1000.0
|
||||
torch.cuda.synchronize()
|
||||
transfer_started = time.perf_counter()
|
||||
inputs = {
|
||||
name: value.to(device) if isinstance(value, torch.Tensor) else value
|
||||
for name, value in inputs.items()
|
||||
}
|
||||
torch.cuda.synchronize()
|
||||
transfer_ms = (time.perf_counter() - transfer_started) * 1000.0
|
||||
forward_started = time.perf_counter()
|
||||
with torch.inference_mode(), torch.amp.autocast("cuda", dtype=torch.float16):
|
||||
outputs = model(**inputs)
|
||||
torch.cuda.synchronize()
|
||||
forward_ms = (time.perf_counter() - forward_started) * 1000.0
|
||||
post_started = time.perf_counter()
|
||||
semantic = processor.post_process_semantic_segmentation(
|
||||
outputs,
|
||||
target_sizes=[image.shape[:2]],
|
||||
)[0]
|
||||
semantic = semantic.detach().cpu().numpy().astype(np.uint8)
|
||||
post_ms = (time.perf_counter() - post_started) * 1000.0
|
||||
del inputs, outputs
|
||||
return semantic, {
|
||||
"processor_ms": processor_ms,
|
||||
"host_to_device_ms": transfer_ms,
|
||||
"forward_ms": forward_ms,
|
||||
"model_postprocess_ms": post_ms,
|
||||
}
|
||||
|
||||
return infer
|
||||
|
||||
|
||||
def _preflight(args: argparse.Namespace) -> int:
|
||||
import torch
|
||||
|
||||
job = _validate_job(args.job.resolve(strict=True))
|
||||
profile, profile_sha256 = _read_profile(args.profile)
|
||||
detector_profile, detector_profile_sha256 = _read_detector_profile(args.detector_profile)
|
||||
semantic_profile, semantic_profile_sha256 = _read_semantic_profile(args.semantic_profile)
|
||||
_validate_source(job, detector_profile)
|
||||
_validate_semantic_source(job, semantic_profile)
|
||||
if profile["source"] != detector_profile["source"]:
|
||||
raise RuntimeError("LAB E9 source and detector bindings differ")
|
||||
if profile["source"] != {
|
||||
key: semantic_profile["source"][key]
|
||||
for key in ("source_id", "resolution", "calibration_slot", "calibration_sha256")
|
||||
}:
|
||||
raise RuntimeError("LAB E9 source and semantic bindings differ")
|
||||
valid_mask, valid_fov = _load_valid_fov(args.valid_fov_root, job, detector_profile)
|
||||
detector_files = _verify_model(detector_profile, args.model_root)
|
||||
dependency = _dependency_manifest(args.environment.resolve(strict=True))
|
||||
if dependency["identity"]["profile_sha256"] != semantic_profile_sha256:
|
||||
raise RuntimeError("LAB E9 semantic dependencies belong to another profile")
|
||||
if not torch.cuda.is_available():
|
||||
raise RuntimeError("CUDA is required for LAB E9")
|
||||
device = torch.device("cuda:0")
|
||||
processor, model, _labels, _lut, semantic_files = _load_model(
|
||||
semantic_profile,
|
||||
args.cache.resolve(strict=True),
|
||||
device,
|
||||
)
|
||||
del processor, model, _labels, _lut
|
||||
torch.cuda.empty_cache()
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"state": "preflight-ready",
|
||||
"job_id": job["job_id"],
|
||||
"profile_sha256": profile_sha256,
|
||||
"detector_profile_sha256": detector_profile_sha256,
|
||||
"semantic_profile_sha256": semantic_profile_sha256,
|
||||
"detector_files": detector_files,
|
||||
"semantic_files": semantic_files,
|
||||
"valid_fov": valid_fov,
|
||||
"cuda_device": torch.cuda.get_device_name(),
|
||||
},
|
||||
sort_keys=True,
|
||||
),
|
||||
flush=True,
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
def _run(args: argparse.Namespace) -> int:
|
||||
import torch
|
||||
import transformers
|
||||
from PIL import Image
|
||||
from scipy.optimize import linear_sum_assignment
|
||||
|
||||
if not _valid_sha256(args.orchestrator_sha256):
|
||||
raise RuntimeError("LAB E9 orchestrator SHA-256 is invalid")
|
||||
job = _validate_job(args.job.resolve(strict=True))
|
||||
profile, profile_sha256 = _read_profile(args.profile)
|
||||
detector_profile, detector_profile_sha256 = _read_detector_profile(args.detector_profile)
|
||||
semantic_profile, semantic_profile_sha256 = _read_semantic_profile(args.semantic_profile)
|
||||
_validate_source(job, detector_profile)
|
||||
_validate_semantic_source(job, semantic_profile)
|
||||
valid_mask, valid_fov = _load_valid_fov(args.valid_fov_root, job, detector_profile)
|
||||
detector_files = _verify_model(detector_profile, args.model_root)
|
||||
dependency = _dependency_manifest(args.environment.resolve(strict=True))
|
||||
if dependency["identity"]["profile_sha256"] != semantic_profile_sha256:
|
||||
raise RuntimeError("LAB E9 semantic dependency identity changed")
|
||||
frames_root = args.frames.resolve(strict=True)
|
||||
frame_paths = sorted(frames_root.glob("frame-*.png"))
|
||||
if not frame_paths or [path.name for path in frame_paths] != [
|
||||
f"frame-{index:06d}.png" for index in range(1, len(frame_paths) + 1)
|
||||
]:
|
||||
raise RuntimeError("LAB E9 decoded frame sequence changed")
|
||||
timeline_path = args.timeline.resolve(strict=True)
|
||||
timeline_rows = _read_timeline(timeline_path, len(frame_paths))
|
||||
|
||||
output = args.output.resolve()
|
||||
if output.exists():
|
||||
raise RuntimeError("LAB E9 output must be absent")
|
||||
output.mkdir(mode=0o700, parents=True, exist_ok=False)
|
||||
_assert_disk_floor(output, args.free_bytes_floor, 0)
|
||||
|
||||
if not torch.cuda.is_available():
|
||||
raise RuntimeError("CUDA is required for LAB E9")
|
||||
device = torch.device("cuda:0")
|
||||
processor, semantic_model, semantic_labels, target_lut, semantic_files = _load_model(
|
||||
semantic_profile,
|
||||
args.cache.resolve(strict=True),
|
||||
device,
|
||||
)
|
||||
target_names = {
|
||||
int(key): str(value) for key, value in semantic_profile["target_taxonomy"].items()
|
||||
}
|
||||
if set(target_names) != set(range(TARGET_CLASS_COUNT)):
|
||||
raise RuntimeError("LAB E9 target taxonomy changed")
|
||||
infer_semantic = _semantic_infer_factory(processor, semantic_model, device)
|
||||
|
||||
tracker = TwoStageTracker(detector_profile["tracking"])
|
||||
linear_sum_assignment(np.zeros((1, 1), dtype=np.float64))
|
||||
with Image.open(frame_paths[0]) as opened:
|
||||
warm_image = np.asarray(opened.convert("RGB"), dtype=np.uint8)
|
||||
detector_tensor = _preprocess(warm_image, valid_mask, detector_profile)
|
||||
_infer(args.triton_url, detector_profile["model"], detector_tensor)
|
||||
semantic_input = np.where(valid_mask[..., None], warm_image, 0).astype(np.uint8)
|
||||
warm_semantic, _warm_latency = infer_semantic(semantic_input)
|
||||
del detector_tensor, semantic_input, warm_semantic, _warm_latency, warm_image
|
||||
|
||||
replay = profile["replay"]
|
||||
detector_queue = LatestWinsQueue(int(replay["detector_queue_capacity"]))
|
||||
semantic_queue = LatestWinsQueue(int(replay["semantic_queue_capacity"]))
|
||||
latest_semantic = LatestSemantic()
|
||||
producer_errors: list[BaseException] = []
|
||||
semantic_errors: list[BaseException] = []
|
||||
detector_failures = 0
|
||||
detector_latency = {
|
||||
name: []
|
||||
for name in (
|
||||
"image_decode_ms",
|
||||
"source_release_lag_ms",
|
||||
"queue_wait_ms",
|
||||
"preprocess_ms",
|
||||
"triton_request_ms",
|
||||
"postprocess_ms",
|
||||
"tracking_ms",
|
||||
"processing_ms",
|
||||
"result_age_ms",
|
||||
)
|
||||
}
|
||||
semantic_latency = {
|
||||
name: []
|
||||
for name in (
|
||||
"image_decode_ms",
|
||||
"queue_wait_ms",
|
||||
"valid_fov_fill_ms",
|
||||
"processor_ms",
|
||||
"host_to_device_ms",
|
||||
"forward_ms",
|
||||
"model_postprocess_ms",
|
||||
"processing_ms",
|
||||
"completion_age_ms",
|
||||
)
|
||||
}
|
||||
detector_labels: Counter[str] = Counter()
|
||||
track_labels: Counter[str] = Counter()
|
||||
semantic_status: Counter[str] = Counter()
|
||||
unique_tracks: set[int] = set()
|
||||
duplicate_pairs = 0
|
||||
detector_path = output / "detector-frames.jsonl"
|
||||
semantic_path = output / "semantic-frames.jsonl"
|
||||
merged_path = output / "merged-frames.jsonl"
|
||||
gpu_path = output / "gpu-telemetry.jsonl"
|
||||
disk_before = shutil.disk_usage(output).free
|
||||
|
||||
with (
|
||||
detector_path.open("x", encoding="utf-8", newline="\n") as detector_stream,
|
||||
semantic_path.open("x", encoding="utf-8", newline="\n") as semantic_stream,
|
||||
merged_path.open("x", encoding="utf-8", newline="\n") as merged_stream,
|
||||
gpu_path.open("x", encoding="utf-8", newline="\n") as gpu_stream,
|
||||
_GpuTelemetry(gpu_stream, args.telemetry_interval_seconds) as gpu_telemetry,
|
||||
):
|
||||
semantic_thread = threading.Thread(
|
||||
target=_semantic_worker,
|
||||
kwargs={
|
||||
"queue": semantic_queue,
|
||||
"latest": latest_semantic,
|
||||
"stream": semantic_stream,
|
||||
"valid_mask": valid_mask,
|
||||
"target_lut": target_lut,
|
||||
"target_names": target_names,
|
||||
"infer": infer_semantic,
|
||||
"latency": semantic_latency,
|
||||
"failures": semantic_errors,
|
||||
},
|
||||
name="lab-e9-semantic-consumer",
|
||||
daemon=True,
|
||||
)
|
||||
semantic_thread.start()
|
||||
replay_started = time.perf_counter() + 0.25
|
||||
producer = threading.Thread(
|
||||
target=_producer,
|
||||
kwargs={
|
||||
"detector_queue": detector_queue,
|
||||
"semantic_queue": semantic_queue,
|
||||
"semantic_stride": int(replay["semantic_sample_every_frames"]),
|
||||
"frame_paths": frame_paths,
|
||||
"timeline_rows": timeline_rows,
|
||||
"replay_started": replay_started,
|
||||
"speed": float(replay["speed"]),
|
||||
"error": producer_errors,
|
||||
},
|
||||
name="lab-e9-source-producer",
|
||||
daemon=True,
|
||||
)
|
||||
producer.start()
|
||||
|
||||
while (envelope := detector_queue.take()) is not None:
|
||||
started = time.perf_counter()
|
||||
detector_latency["image_decode_ms"].append(envelope.decode_ms)
|
||||
detector_latency["source_release_lag_ms"].append(envelope.source_release_lag_ms)
|
||||
detector_latency["queue_wait_ms"].append(
|
||||
max(0.0, (started - envelope.decoded_monotonic) * 1000.0)
|
||||
)
|
||||
try:
|
||||
preprocess_started = time.perf_counter()
|
||||
tensor = _preprocess(envelope.image, valid_mask, detector_profile)
|
||||
detector_latency["preprocess_ms"].append(
|
||||
(time.perf_counter() - preprocess_started) * 1000.0
|
||||
)
|
||||
output_tensor, request_ms = _infer(
|
||||
args.triton_url, detector_profile["model"], tensor
|
||||
)
|
||||
detector_latency["triton_request_ms"].append(request_ms)
|
||||
post_started = time.perf_counter()
|
||||
detections, _rejected = _detections(output_tensor, detector_profile, valid_mask)
|
||||
detector_latency["postprocess_ms"].append(
|
||||
(time.perf_counter() - post_started) * 1000.0
|
||||
)
|
||||
tracking_started = time.perf_counter()
|
||||
tracks = tracker.update(detections, envelope.frame_index)
|
||||
detector_latency["tracking_ms"].append(
|
||||
(time.perf_counter() - tracking_started) * 1000.0
|
||||
)
|
||||
except Exception:
|
||||
detector_failures += 1
|
||||
raise
|
||||
completed = time.perf_counter()
|
||||
processing_ms = (completed - started) * 1000.0
|
||||
result_age_ms = max(0.0, (completed - envelope.scheduled_monotonic) * 1000.0)
|
||||
detector_latency["processing_ms"].append(processing_ms)
|
||||
detector_latency["result_age_ms"].append(result_age_ms)
|
||||
detector_labels.update(str(item["label"]) for item in detections)
|
||||
duplicate_pairs += _duplicate_pairs(tracks)
|
||||
for track in tracks:
|
||||
unique_tracks.add(track.track_id)
|
||||
track_labels[track.label] += 1
|
||||
semantic_binding = _semantic_binding(
|
||||
latest_semantic.snapshot(),
|
||||
frame_session_seconds=float(envelope.timeline["session_seconds"]),
|
||||
ttl_ms=float(replay["semantic_ttl_ms"]),
|
||||
)
|
||||
semantic_status[str(semantic_binding["status"])] += 1
|
||||
detector_document = {
|
||||
"schema_version": DETECTOR_FRAME_SCHEMA,
|
||||
"frame_index": envelope.frame_index,
|
||||
"source_frame_index": envelope.timeline["source_frame_index"],
|
||||
"session_seconds": round(float(envelope.timeline["session_seconds"]), 9),
|
||||
"result_age_ms": round(result_age_ms, 6),
|
||||
"detections": detections,
|
||||
"tracks": [_track_document(track) for track in tracks],
|
||||
}
|
||||
detector_stream.write(
|
||||
json.dumps(
|
||||
detector_document,
|
||||
sort_keys=True,
|
||||
separators=(",", ":"),
|
||||
allow_nan=False,
|
||||
)
|
||||
+ "\n"
|
||||
)
|
||||
merged_stream.write(
|
||||
json.dumps(
|
||||
{
|
||||
"schema_version": MERGED_FRAME_SCHEMA,
|
||||
"frame_index": envelope.frame_index,
|
||||
"source_frame_index": envelope.timeline["source_frame_index"],
|
||||
"session_seconds": detector_document["session_seconds"],
|
||||
"detector_result_age_ms": round(result_age_ms, 6),
|
||||
"tracks": detector_document["tracks"],
|
||||
"semantic": semantic_binding,
|
||||
},
|
||||
sort_keys=True,
|
||||
separators=(",", ":"),
|
||||
allow_nan=False,
|
||||
)
|
||||
+ "\n"
|
||||
)
|
||||
consumed = int(detector_queue.snapshot()["consumed"])
|
||||
if consumed % 100 == 0:
|
||||
detector_stream.flush()
|
||||
merged_stream.flush()
|
||||
semantic_stream.flush()
|
||||
gpu_stream.flush()
|
||||
_assert_disk_floor(output, args.free_bytes_floor, consumed)
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"phase": "multirate",
|
||||
"detector_consumed": consumed,
|
||||
"detector_dropped": detector_queue.snapshot()["dropped_overflow"],
|
||||
"semantic_consumed": semantic_queue.snapshot()["consumed"],
|
||||
"semantic_dropped": semantic_queue.snapshot()["dropped_overflow"],
|
||||
},
|
||||
sort_keys=True,
|
||||
),
|
||||
flush=True,
|
||||
)
|
||||
|
||||
producer.join(timeout=5)
|
||||
if producer.is_alive() or producer_errors:
|
||||
raise RuntimeError("LAB E9 producer failed")
|
||||
semantic_thread.join(timeout=30)
|
||||
if semantic_thread.is_alive():
|
||||
raise RuntimeError("LAB E9 semantic consumer did not terminate")
|
||||
if semantic_errors:
|
||||
raise RuntimeError("LAB E9 semantic consumer failed") from semantic_errors[0]
|
||||
detector_stream.flush()
|
||||
semantic_stream.flush()
|
||||
merged_stream.flush()
|
||||
os.fsync(detector_stream.fileno())
|
||||
os.fsync(semantic_stream.fileno())
|
||||
os.fsync(merged_stream.fileno())
|
||||
|
||||
replay_finished = time.perf_counter()
|
||||
detector_state = detector_queue.snapshot()
|
||||
semantic_state = semantic_queue.snapshot()
|
||||
source_span = (
|
||||
float(timeline_rows[-1]["session_seconds"]) - float(timeline_rows[0]["session_seconds"])
|
||||
) / float(replay["speed"])
|
||||
replay_wall = replay_finished - replay_started
|
||||
scheduled_semantic = ((len(frame_paths) - 1) // int(replay["semantic_sample_every_frames"])) + 1
|
||||
detector_fps = int(detector_state["consumed"]) / max(source_span, replay_wall, 1e-9)
|
||||
semantic_fps = int(semantic_state["consumed"]) / max(source_span, replay_wall, 1e-9)
|
||||
detector_drop_fraction = int(detector_state["dropped_overflow"]) / len(frame_paths)
|
||||
semantic_drop_fraction = int(semantic_state["dropped_overflow"]) / scheduled_semantic
|
||||
fresh_coverage = semantic_status["fresh"] / max(1, int(detector_state["consumed"]))
|
||||
detector_percentiles = {name: _percentiles(values) for name, values in detector_latency.items()}
|
||||
semantic_percentiles = {name: _percentiles(values) for name, values in semantic_latency.items()}
|
||||
acceptance = profile["acceptance"]
|
||||
checks = {
|
||||
"detector_queue_bounded": int(detector_state["maximum_depth"])
|
||||
<= int(detector_state["capacity"]),
|
||||
"detector_accounting": int(detector_state["consumed"])
|
||||
+ int(detector_state["dropped_overflow"])
|
||||
== len(frame_paths),
|
||||
"detector_minimum_effective_fps": detector_fps
|
||||
>= float(acceptance["detector_minimum_effective_fps"]),
|
||||
"detector_maximum_drop_fraction": detector_drop_fraction
|
||||
<= float(acceptance["detector_maximum_drop_fraction"]),
|
||||
"detector_maximum_p95_result_age_ms": float(detector_percentiles["result_age_ms"]["p95"])
|
||||
<= float(acceptance["detector_maximum_p95_result_age_ms"]),
|
||||
"semantic_queue_bounded": int(semantic_state["maximum_depth"])
|
||||
<= int(semantic_state["capacity"]),
|
||||
"semantic_accounting": int(semantic_state["consumed"])
|
||||
+ int(semantic_state["dropped_overflow"])
|
||||
== scheduled_semantic,
|
||||
"semantic_minimum_effective_fps": semantic_fps
|
||||
>= float(acceptance["semantic_minimum_effective_fps"]),
|
||||
"semantic_maximum_drop_fraction": semantic_drop_fraction
|
||||
<= float(acceptance["semantic_maximum_drop_fraction"]),
|
||||
"semantic_maximum_p95_completion_age_ms": float(
|
||||
semantic_percentiles["completion_age_ms"]["p95"]
|
||||
)
|
||||
<= float(acceptance["semantic_maximum_p95_completion_age_ms"]),
|
||||
"minimum_fresh_semantic_coverage": fresh_coverage
|
||||
>= float(acceptance["minimum_fresh_semantic_coverage"]),
|
||||
"zero_failures": detector_failures == 0 and not semantic_errors,
|
||||
}
|
||||
accepted = all(checks.values())
|
||||
|
||||
identity = {
|
||||
"schema_version": IDENTITY_SCHEMA,
|
||||
"job_id": job["job_id"],
|
||||
"input_sha256": job["input_sha256"],
|
||||
"session_id": job["input"]["session_id"],
|
||||
"source_id": job["input"]["source_id"],
|
||||
"selection": {
|
||||
"frame_count": len(frame_paths),
|
||||
"source_start_frame_index": timeline_rows[0]["source_frame_index"],
|
||||
"source_end_frame_index": timeline_rows[-1]["source_frame_index"],
|
||||
"timeline_start_seconds": timeline_rows[0]["session_seconds"],
|
||||
"timeline_end_seconds": timeline_rows[-1]["session_seconds"],
|
||||
"timeline_sha256": _sha256(timeline_path),
|
||||
},
|
||||
"configuration": {
|
||||
"pipeline": PIPELINE_ID,
|
||||
"profile": profile,
|
||||
"profile_sha256": profile_sha256,
|
||||
"detector_profile_sha256": detector_profile_sha256,
|
||||
"semantic_profile_sha256": semantic_profile_sha256,
|
||||
"runner_sha256": _sha256(Path(__file__).resolve(strict=True)),
|
||||
"orchestrator_sha256": args.orchestrator_sha256,
|
||||
"container_image": args.container_image,
|
||||
"valid_fov": valid_fov,
|
||||
},
|
||||
"models": {
|
||||
"detector": detector_profile["model"],
|
||||
"detector_files": detector_files,
|
||||
"semantic": semantic_profile["model"],
|
||||
"semantic_files": semantic_files,
|
||||
},
|
||||
}
|
||||
identity_sha256 = hashlib.sha256(_canonical_json(identity)).hexdigest()
|
||||
result_id = f"e9-multirate-perception-{identity_sha256}"
|
||||
metrics = {
|
||||
"source_span_seconds": round(source_span, 6),
|
||||
"replay_wall_seconds": round(replay_wall, 6),
|
||||
"detector": {
|
||||
"frames_expected": len(frame_paths),
|
||||
"frames_processed": detector_state["consumed"],
|
||||
"frames_dropped": detector_state["dropped_overflow"],
|
||||
"drop_fraction": round(detector_drop_fraction, 9),
|
||||
"effective_frames_per_second": round(detector_fps, 6),
|
||||
"queue": detector_state,
|
||||
"latency_ms": detector_percentiles,
|
||||
"detections": int(sum(detector_labels.values())),
|
||||
"unique_confirmed_tracks": len(unique_tracks),
|
||||
"track_observations": int(sum(track_labels.values())),
|
||||
"same_class_duplicate_pairs_iou_ge_0_8": duplicate_pairs,
|
||||
},
|
||||
"semantic": {
|
||||
"frames_scheduled": scheduled_semantic,
|
||||
"frames_processed": semantic_state["consumed"],
|
||||
"frames_dropped": semantic_state["dropped_overflow"],
|
||||
"drop_fraction": round(semantic_drop_fraction, 9),
|
||||
"effective_frames_per_second": round(semantic_fps, 6),
|
||||
"queue": semantic_state,
|
||||
"latency_ms": semantic_percentiles,
|
||||
"binding_status_counts": dict(sorted(semantic_status.items())),
|
||||
"fresh_coverage": round(fresh_coverage, 9),
|
||||
},
|
||||
"gpu_telemetry": gpu_telemetry.summary(),
|
||||
"process_peak_rss_mib": round(resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024, 3),
|
||||
"cuda_peak_memory_allocated_mib": round(torch.cuda.max_memory_allocated() / 2**20, 3),
|
||||
"cuda_peak_memory_reserved_mib": round(torch.cuda.max_memory_reserved() / 2**20, 3),
|
||||
"disk": {
|
||||
"free_bytes_before_replay": disk_before,
|
||||
"free_bytes_after_replay": shutil.disk_usage(output).free,
|
||||
"free_bytes_floor": args.free_bytes_floor,
|
||||
},
|
||||
}
|
||||
report = {
|
||||
"schema_version": REPORT_SCHEMA,
|
||||
"result_id": result_id,
|
||||
"created_at_utc": datetime.now(UTC)
|
||||
.isoformat(timespec="milliseconds")
|
||||
.replace("+00:00", "Z"),
|
||||
"state": "accepted" if accepted else "rejected",
|
||||
"ground_truth": False,
|
||||
"identity": identity,
|
||||
"runtime": {
|
||||
"hostname": platform.node(),
|
||||
"platform": platform.platform(),
|
||||
"python": platform.python_version(),
|
||||
"numpy": np.__version__,
|
||||
"torch": torch.__version__,
|
||||
"transformers": transformers.__version__,
|
||||
"scipy": importlib.metadata.version("scipy"),
|
||||
"gpu": torch.cuda.get_device_name(),
|
||||
},
|
||||
"metrics": metrics,
|
||||
"acceptance": {
|
||||
"accepted": accepted,
|
||||
"checks": checks,
|
||||
"navigation_or_safety_accepted": False,
|
||||
},
|
||||
"limitations": [
|
||||
"Recorded source-paced replay, not live K1 transport.",
|
||||
(
|
||||
"Semantic masks are measured in memory; this gate stores hashes and "
|
||||
"class totals, not mask images."
|
||||
),
|
||||
"Generic Cityscapes and COCO models are not forest-domain or safety validated.",
|
||||
"LiDAR fusion and 3D world-state publication remain downstream gates.",
|
||||
],
|
||||
}
|
||||
report_path = output / "run-report.json"
|
||||
_write_json(report_path, report)
|
||||
artifacts = [
|
||||
_artifact(
|
||||
detector_path,
|
||||
"multirate-detector-frames",
|
||||
"application/x-ndjson",
|
||||
DETECTOR_FRAME_SCHEMA,
|
||||
),
|
||||
_artifact(
|
||||
semantic_path,
|
||||
"multirate-semantic-frames",
|
||||
"application/x-ndjson",
|
||||
SEMANTIC_FRAME_SCHEMA,
|
||||
),
|
||||
_artifact(
|
||||
merged_path, "multirate-merged-frames", "application/x-ndjson", MERGED_FRAME_SCHEMA
|
||||
),
|
||||
_artifact(gpu_path, "worker-gpu-telemetry", "application/x-ndjson"),
|
||||
_artifact(report_path, "multirate-run-report", "application/json", REPORT_SCHEMA),
|
||||
]
|
||||
result = {
|
||||
"schema_version": RESULT_SCHEMA,
|
||||
"result_id": result_id,
|
||||
"identity_sha256": identity_sha256,
|
||||
"identity": identity,
|
||||
"created_at_utc": report["created_at_utc"],
|
||||
"acceptance_state": report["state"],
|
||||
"ground_truth": False,
|
||||
"publication_scope": "recorded-multirate-qualification-only",
|
||||
"frames_processed": detector_state["consumed"],
|
||||
"artifacts": artifacts,
|
||||
}
|
||||
_write_json(output / "result.json", result)
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"result_id": result_id,
|
||||
"accepted": accepted,
|
||||
"detector_fps": round(detector_fps, 6),
|
||||
"detector_dropped": detector_state["dropped_overflow"],
|
||||
"detector_result_age_p95_ms": detector_percentiles["result_age_ms"]["p95"],
|
||||
"semantic_fps": round(semantic_fps, 6),
|
||||
"semantic_dropped": semantic_state["dropped_overflow"],
|
||||
"semantic_completion_age_p95_ms": semantic_percentiles["completion_age_ms"]["p95"],
|
||||
"fresh_semantic_coverage": round(fresh_coverage, 6),
|
||||
},
|
||||
sort_keys=True,
|
||||
),
|
||||
flush=True,
|
||||
)
|
||||
# A completed rejected qualification is still immutable diagnostic evidence.
|
||||
# The manifest state, not the process exit code, carries acceptance.
|
||||
return 0
|
||||
|
||||
|
||||
def _assert_disk_floor(path: Path, floor: int, frame: int) -> None:
|
||||
if floor < 0:
|
||||
raise RuntimeError("LAB E9 disk floor is invalid")
|
||||
free = shutil.disk_usage(path).free
|
||||
if floor and free < floor:
|
||||
raise RuntimeError(f"LAB E9 crossed the D-backed free-space floor at frame {frame}")
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = _arguments()
|
||||
if args.command == "preflight":
|
||||
return _preflight(args)
|
||||
return _run(args)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,698 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Generate review-only E2 prelabels from the exact E0 model generations."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import platform
|
||||
import resource
|
||||
import time
|
||||
from collections import Counter
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path, PurePosixPath
|
||||
from typing import Any
|
||||
|
||||
from run_recorded_perception_epoch import (
|
||||
INSTANCE_MASK_THRESHOLD,
|
||||
INSTANCE_SCORE_THRESHOLD,
|
||||
SEMANTIC_MODEL_ID,
|
||||
SEMANTIC_REVISION,
|
||||
_canonical_json,
|
||||
_GpuTelemetry,
|
||||
_model_files,
|
||||
_palette,
|
||||
_percentiles,
|
||||
_read_object,
|
||||
_sha256,
|
||||
_valid_sha256,
|
||||
_write_json,
|
||||
_write_png,
|
||||
)
|
||||
|
||||
PACK_SCHEMA = "missioncore.perception-evaluation-pack/v1"
|
||||
PACK_IDENTITY_SCHEMA = "missioncore.perception-evaluation-pack-identity/v1"
|
||||
CONTRACT_SCHEMA = "missioncore.perception-annotation-contract/v1"
|
||||
VALID_FOV_SCHEMA = "missioncore.k1-valid-fov-mask/v1"
|
||||
RESULT_SCHEMA = "missioncore.perception-evaluation-prelabels/v1"
|
||||
RESULT_IDENTITY_SCHEMA = "missioncore.perception-evaluation-prelabels-identity/v1"
|
||||
REPORT_SCHEMA = "missioncore.perception-evaluation-prelabels-report/v1"
|
||||
FRAME_SCHEMA = "missioncore.perception-evaluation-prelabel-frame/v1"
|
||||
MAPPING_PROFILE = "e0-coco-ade-to-e2-taxonomy/v1"
|
||||
PREVIEW_COUNT = 16
|
||||
|
||||
TARGET_NAMES = {
|
||||
0: "outside_valid_fov",
|
||||
1: "person",
|
||||
2: "bicycle",
|
||||
3: "motorcycle",
|
||||
4: "car",
|
||||
5: "heavy_vehicle",
|
||||
6: "building_structure",
|
||||
7: "paved_road",
|
||||
8: "sidewalk_curb",
|
||||
9: "ground_dirt",
|
||||
10: "grass_low_vegetation",
|
||||
11: "tree_woody_vegetation",
|
||||
12: "sky",
|
||||
13: "static_obstacle",
|
||||
14: "animal",
|
||||
15: "other_background",
|
||||
}
|
||||
|
||||
INSTANCE_MAPPING = {
|
||||
"person": 1,
|
||||
"bicycle": 2,
|
||||
"motorcycle": 3,
|
||||
"car": 4,
|
||||
"bus": 5,
|
||||
"truck": 5,
|
||||
"cat": 14,
|
||||
"dog": 14,
|
||||
"horse": 14,
|
||||
"sheep": 14,
|
||||
"cow": 14,
|
||||
"elephant": 14,
|
||||
"bear": 14,
|
||||
"zebra": 14,
|
||||
"giraffe": 14,
|
||||
"bench": 13,
|
||||
"fire hydrant": 13,
|
||||
"stop sign": 13,
|
||||
"parking meter": 13,
|
||||
"chair": 13,
|
||||
"potted plant": 13,
|
||||
}
|
||||
|
||||
SEMANTIC_MAPPING = {
|
||||
"person": 1,
|
||||
"bicycle": 2,
|
||||
"minibike": 3,
|
||||
"car": 4,
|
||||
"bus": 5,
|
||||
"truck": 5,
|
||||
"van": 5,
|
||||
"building": 6,
|
||||
"house": 6,
|
||||
"skyscraper": 6,
|
||||
"wall": 6,
|
||||
"door": 6,
|
||||
"windowpane": 6,
|
||||
"hovel": 6,
|
||||
"awning": 6,
|
||||
"road": 7,
|
||||
"runway": 7,
|
||||
"path": 7,
|
||||
"sidewalk": 8,
|
||||
"stairs": 8,
|
||||
"stairway": 8,
|
||||
"step": 8,
|
||||
"earth": 9,
|
||||
"sand": 9,
|
||||
"field": 9,
|
||||
"land": 9,
|
||||
"dirt track": 9,
|
||||
"grass": 10,
|
||||
"plant": 10,
|
||||
"flower": 10,
|
||||
"tree": 11,
|
||||
"palm": 11,
|
||||
"sky": 12,
|
||||
"animal": 14,
|
||||
}
|
||||
|
||||
|
||||
def _arguments() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--evaluation-pack", type=Path, required=True)
|
||||
parser.add_argument("--valid-fov-root", type=Path, required=True)
|
||||
parser.add_argument("--output", type=Path, required=True)
|
||||
parser.add_argument("--cache", type=Path, required=True)
|
||||
parser.add_argument("--telemetry-interval-seconds", type=float, default=1.0)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def _safe_artifact(root: Path, encoded: object) -> Path:
|
||||
if not isinstance(encoded, str):
|
||||
raise RuntimeError("evaluation artifact path is not a string")
|
||||
relative = PurePosixPath(encoded)
|
||||
if relative.is_absolute() or ".." in relative.parts or not relative.parts:
|
||||
raise RuntimeError("evaluation artifact path is unsafe")
|
||||
path = root.joinpath(*relative.parts).resolve(strict=True)
|
||||
if not path.is_file() or not path.is_relative_to(root):
|
||||
raise RuntimeError("evaluation artifact escapes the pack")
|
||||
return path
|
||||
|
||||
|
||||
def _validate_pack(root: Path) -> tuple[dict[str, Any], list[dict[str, Any]]]:
|
||||
resolved = root.resolve(strict=True)
|
||||
manifest = _read_object(resolved / "manifest.json")
|
||||
identity = manifest.get("identity")
|
||||
identity_sha256 = manifest.get("identity_sha256")
|
||||
if (
|
||||
manifest.get("schema_version") != PACK_SCHEMA
|
||||
or not isinstance(identity, dict)
|
||||
or identity.get("schema_version") != PACK_IDENTITY_SCHEMA
|
||||
or not _valid_sha256(identity_sha256)
|
||||
or hashlib.sha256(_canonical_json(identity)).hexdigest() != identity_sha256
|
||||
or manifest.get("generation_id") != f"evaluation-pack-{identity_sha256}"
|
||||
or identity.get("preprocessing_profile") != "fixed-valid-fov-fill/v1"
|
||||
):
|
||||
raise RuntimeError("evaluation pack identity is invalid")
|
||||
artifacts = manifest.get("artifacts")
|
||||
if not isinstance(artifacts, list) or not artifacts:
|
||||
raise RuntimeError("evaluation pack artifacts are unavailable")
|
||||
seen: set[str] = set()
|
||||
for artifact in artifacts:
|
||||
if not isinstance(artifact, dict) or not isinstance(artifact.get("path"), str):
|
||||
raise RuntimeError("evaluation artifact descriptor is invalid")
|
||||
encoded = str(artifact["path"])
|
||||
if encoded in seen:
|
||||
raise RuntimeError("evaluation artifact descriptor is duplicated")
|
||||
path = _safe_artifact(resolved, encoded)
|
||||
if (
|
||||
artifact.get("byte_length") != path.stat().st_size
|
||||
or not _valid_sha256(artifact.get("sha256"))
|
||||
or _sha256(path) != artifact["sha256"]
|
||||
):
|
||||
raise RuntimeError(f"evaluation artifact changed: {encoded}")
|
||||
seen.add(encoded)
|
||||
contract = _read_object(resolved / "annotation-contract.json")
|
||||
if (
|
||||
contract.get("schema_version") != CONTRACT_SCHEMA
|
||||
or contract.get("evaluation_identity_sha256") != identity_sha256
|
||||
or [row.get("id") for row in contract.get("categories", [])]
|
||||
!= list(range(1, 16))
|
||||
):
|
||||
raise RuntimeError("evaluation annotation contract is invalid")
|
||||
frames = identity.get("frames")
|
||||
if not isinstance(frames, list) or len(frames) != 64:
|
||||
raise RuntimeError("evaluation frame set is invalid")
|
||||
for expected_image_id, frame in enumerate(frames, start=1):
|
||||
if not isinstance(frame, dict):
|
||||
raise RuntimeError("evaluation frame descriptor is invalid")
|
||||
image_id = frame.get("image_id")
|
||||
frame_index = frame.get("frame_index")
|
||||
if (
|
||||
image_id != expected_image_id
|
||||
or not isinstance(frame_index, int)
|
||||
or isinstance(frame_index, bool)
|
||||
):
|
||||
raise RuntimeError("evaluation frame order changed")
|
||||
name = f"image-{image_id:03d}-frame-{frame_index:06d}.png"
|
||||
expected_paths = {
|
||||
f"images/raw/{name}",
|
||||
f"images/valid-fov-fill/{name}",
|
||||
}
|
||||
if not expected_paths <= seen:
|
||||
raise RuntimeError("evaluation frame artifacts are incomplete")
|
||||
return manifest, frames
|
||||
|
||||
|
||||
def _load_valid_fov(root: Path, pack: dict[str, Any]) -> Any:
|
||||
import numpy as np
|
||||
from PIL import Image
|
||||
|
||||
resolved = root.resolve(strict=True)
|
||||
manifest = _read_object(resolved / "manifest.json")
|
||||
identity = manifest.get("identity")
|
||||
identity_sha256 = manifest.get("identity_sha256")
|
||||
artifact = manifest.get("artifact")
|
||||
pack_identity = pack["identity"]
|
||||
if (
|
||||
manifest.get("schema_version") != VALID_FOV_SCHEMA
|
||||
or not isinstance(identity, dict)
|
||||
or not _valid_sha256(identity_sha256)
|
||||
or hashlib.sha256(_canonical_json(identity)).hexdigest() != identity_sha256
|
||||
or manifest.get("generation_id") != f"valid-fov-mask-{identity_sha256}"
|
||||
or manifest.get("generation_id") != pack_identity["valid_fov_generation_id"]
|
||||
or identity.get("calibration_sha256") != pack_identity["calibration_sha256"]
|
||||
or identity.get("calibration_slot") != pack_identity["calibration_slot"]
|
||||
or identity.get("source_id") != pack_identity["source_id"]
|
||||
or identity.get("admitted_resolution") != [800, 600]
|
||||
or not isinstance(artifact, dict)
|
||||
or artifact.get("path") != "mask.png"
|
||||
or not _valid_sha256(artifact.get("sha256"))
|
||||
):
|
||||
raise RuntimeError("valid-FOV binding is invalid")
|
||||
mask_path = (resolved / "mask.png").resolve(strict=True)
|
||||
if mask_path.parent != resolved or _sha256(mask_path) != artifact["sha256"]:
|
||||
raise RuntimeError("valid-FOV artifact changed")
|
||||
with Image.open(mask_path) as opened:
|
||||
mask = np.asarray(opened, dtype=np.uint8)
|
||||
if mask.shape != (600, 800) or not np.isin(mask, (0, 255)).all():
|
||||
raise RuntimeError("valid-FOV mask pixels are invalid")
|
||||
return mask > 0
|
||||
|
||||
|
||||
def _resource_snapshot() -> dict[str, float]:
|
||||
usage = resource.getrusage(resource.RUSAGE_SELF)
|
||||
result = {
|
||||
"user_cpu_seconds": float(usage.ru_utime),
|
||||
"system_cpu_seconds": float(usage.ru_stime),
|
||||
"minor_page_faults": float(usage.ru_minflt),
|
||||
"major_page_faults": float(usage.ru_majflt),
|
||||
"voluntary_context_switches": float(usage.ru_nvcsw),
|
||||
"involuntary_context_switches": float(usage.ru_nivcsw),
|
||||
}
|
||||
try:
|
||||
for line in Path("/proc/self/io").read_text(encoding="ascii").splitlines():
|
||||
name, value = line.split(":", 1)
|
||||
if name in {"read_bytes", "write_bytes", "rchar", "wchar"}:
|
||||
result[f"proc_io_{name}"] = float(value.strip())
|
||||
except (OSError, UnicodeDecodeError, ValueError):
|
||||
pass
|
||||
return result
|
||||
|
||||
|
||||
def _resource_delta(before: dict[str, float], after: dict[str, float]) -> dict[str, float]:
|
||||
return {
|
||||
name: round(value - before[name], 6)
|
||||
for name, value in after.items()
|
||||
if name in before
|
||||
}
|
||||
|
||||
|
||||
def _semantic_target_map(labels: dict[int, str]) -> Any:
|
||||
import numpy as np
|
||||
|
||||
mapping = np.full(256, 15, dtype=np.uint8)
|
||||
for source_id, name in labels.items():
|
||||
mapping[source_id] = SEMANTIC_MAPPING.get(name.strip().lower(), 15)
|
||||
return mapping
|
||||
|
||||
|
||||
def _preview_indices(count: int) -> set[int]:
|
||||
admitted = min(PREVIEW_COUNT, count)
|
||||
if admitted == 1:
|
||||
return {0}
|
||||
return {
|
||||
(position * (count - 1) + (admitted - 1) // 2) // (admitted - 1)
|
||||
for position in range(admitted)
|
||||
}
|
||||
|
||||
|
||||
def _preview(image: Any, semantic: Any, instance_map: Any, instances: list[dict[str, Any]]) -> Any:
|
||||
import numpy as np
|
||||
from PIL import Image, ImageDraw
|
||||
|
||||
colors = np.zeros_like(image)
|
||||
for category_id in range(1, 16):
|
||||
colors[semantic == category_id] = _palette(category_id)
|
||||
blended = np.clip(
|
||||
image.astype(np.float32) * 0.52 + colors.astype(np.float32) * 0.48,
|
||||
0,
|
||||
255,
|
||||
).astype(np.uint8)
|
||||
canvas = Image.fromarray(blended)
|
||||
draw = ImageDraw.Draw(canvas)
|
||||
for instance in instances:
|
||||
instance_id = int(instance["instance_id"])
|
||||
color = _palette(500 + instance_id)
|
||||
x1, y1, x2, y2 = (int(round(value)) for value in instance["box_xyxy"])
|
||||
draw.rectangle((x1, y1, x2, y2), outline=color, width=2)
|
||||
label = f"{instance['draft_category']} {float(instance['score']):.0%}"
|
||||
draw.text((x1, max(0, y1 - 13)), label, fill=color)
|
||||
if not np.any(instance_map == instance_id):
|
||||
raise RuntimeError("preview instance lost its mask")
|
||||
return np.asarray(canvas, dtype=np.uint8)
|
||||
|
||||
|
||||
def _artifact(path: Path, root: Path) -> dict[str, Any]:
|
||||
return {
|
||||
"path": path.relative_to(root).as_posix(),
|
||||
"byte_length": path.stat().st_size,
|
||||
"sha256": _sha256(path),
|
||||
}
|
||||
|
||||
|
||||
def _run(args: argparse.Namespace) -> int:
|
||||
pack_root = args.evaluation_pack.resolve(strict=True)
|
||||
output_root = args.output.resolve()
|
||||
cache_root = args.cache.resolve(strict=True)
|
||||
if output_root.exists():
|
||||
raise RuntimeError("prelabel output must be absent")
|
||||
pack, frames = _validate_pack(pack_root)
|
||||
valid_mask = _load_valid_fov(args.valid_fov_root, pack)
|
||||
|
||||
import numpy as np
|
||||
import torch
|
||||
import torch.nn.functional as functional
|
||||
import torchvision
|
||||
import transformers
|
||||
from huggingface_hub import snapshot_download
|
||||
from PIL import Image
|
||||
from torchvision.models.detection import (
|
||||
MaskRCNN_ResNet50_FPN_V2_Weights,
|
||||
maskrcnn_resnet50_fpn_v2,
|
||||
)
|
||||
from transformers import AutoImageProcessor, BeitForSemanticSegmentation
|
||||
|
||||
if not torch.cuda.is_available():
|
||||
raise RuntimeError("CUDA is required for E2 prelabels")
|
||||
output_root.mkdir(mode=0o700, parents=True, exist_ok=False)
|
||||
instance_root = output_root / "instance-prelabels"
|
||||
semantic_root = output_root / "semantic-prelabels"
|
||||
preview_root = output_root / "previews"
|
||||
instance_root.mkdir(mode=0o700)
|
||||
semantic_root.mkdir(mode=0o700)
|
||||
preview_root.mkdir(mode=0o700)
|
||||
telemetry_path = output_root / "gpu-telemetry.jsonl"
|
||||
metadata_path = output_root / "frames.jsonl"
|
||||
device = torch.device("cuda:0")
|
||||
frame_paths = []
|
||||
for frame in frames:
|
||||
name = f"image-{int(frame['image_id']):03d}-frame-{int(frame['frame_index']):06d}.png"
|
||||
frame_paths.append(pack_root / "images" / "valid-fov-fill" / name)
|
||||
|
||||
resource_before = _resource_snapshot()
|
||||
wall_started = time.perf_counter()
|
||||
instance_latencies: list[float] = []
|
||||
semantic_latencies: list[float] = []
|
||||
instances_by_frame: list[list[dict[str, Any]]] = []
|
||||
instance_counts: Counter[str] = Counter()
|
||||
ignored_instance_counts: Counter[str] = Counter()
|
||||
preview_indices = _preview_indices(len(frames))
|
||||
torch.cuda.reset_peak_memory_stats()
|
||||
|
||||
with telemetry_path.open("x", encoding="utf-8", newline="\n") as telemetry_stream:
|
||||
with _GpuTelemetry(telemetry_stream, args.telemetry_interval_seconds) as telemetry:
|
||||
instance_weights = MaskRCNN_ResNet50_FPN_V2_Weights.DEFAULT
|
||||
instance_model = maskrcnn_resnet50_fpn_v2(weights=instance_weights).to(device).eval()
|
||||
instance_labels = list(instance_weights.meta["categories"])
|
||||
with Image.open(frame_paths[0]) as opened:
|
||||
warm_image = opened.convert("RGB")
|
||||
with torch.inference_mode():
|
||||
warm_tensor = instance_weights.transforms()(warm_image).to(device)
|
||||
_ = instance_model([warm_tensor])[0]
|
||||
torch.cuda.synchronize()
|
||||
del warm_tensor
|
||||
|
||||
with torch.inference_mode():
|
||||
for order, path in enumerate(frame_paths, start=1):
|
||||
with Image.open(path) as opened:
|
||||
image = np.asarray(opened.convert("RGB"), dtype=np.uint8)
|
||||
tensor = instance_weights.transforms()(Image.fromarray(image)).to(device)
|
||||
torch.cuda.synchronize()
|
||||
started = time.perf_counter()
|
||||
prediction = instance_model([tensor])[0]
|
||||
torch.cuda.synchronize()
|
||||
instance_latencies.append((time.perf_counter() - started) * 1000.0)
|
||||
scores = prediction["scores"].detach().cpu().numpy()
|
||||
keep = np.flatnonzero(scores >= INSTANCE_SCORE_THRESHOLD)
|
||||
index_map = np.zeros(valid_mask.shape, dtype=np.uint16)
|
||||
instances: list[dict[str, Any]] = []
|
||||
for output_index in keep:
|
||||
source_id = int(prediction["labels"][output_index].item())
|
||||
source_name = instance_labels[source_id].strip().lower()
|
||||
target_id = INSTANCE_MAPPING.get(source_name)
|
||||
if target_id is None:
|
||||
ignored_instance_counts[source_name] += 1
|
||||
continue
|
||||
mask = prediction["masks"][output_index, 0].detach().cpu().numpy()
|
||||
mask = np.logical_and(mask >= INSTANCE_MASK_THRESHOLD, valid_mask)
|
||||
admitted_mask = np.logical_and(mask, index_map == 0)
|
||||
mask_pixels = int(np.count_nonzero(admitted_mask))
|
||||
if mask_pixels < 8:
|
||||
continue
|
||||
instance_id = len(instances) + 1
|
||||
index_map[admitted_mask] = instance_id
|
||||
box = [
|
||||
round(float(value), 6)
|
||||
for value in prediction["boxes"][output_index]
|
||||
.detach()
|
||||
.cpu()
|
||||
.tolist()
|
||||
]
|
||||
instance = {
|
||||
"instance_id": instance_id,
|
||||
"draft_category_id": target_id,
|
||||
"draft_category": TARGET_NAMES[target_id],
|
||||
"source_model_category_id": source_id,
|
||||
"source_model_category": source_name,
|
||||
"score": round(float(scores[output_index]), 9),
|
||||
"box_xyxy": box,
|
||||
"mask_pixels": mask_pixels,
|
||||
"review_state": "unreviewed-model-draft",
|
||||
}
|
||||
instances.append(instance)
|
||||
instance_counts[TARGET_NAMES[target_id]] += 1
|
||||
_write_png(instance_root / f"image-{order:03d}.png", index_map)
|
||||
instances_by_frame.append(instances)
|
||||
del tensor, prediction
|
||||
if order % 16 == 0 or order == len(frames):
|
||||
print(
|
||||
json.dumps(
|
||||
{"phase": "instance", "processed": order, "total": len(frames)},
|
||||
sort_keys=True,
|
||||
),
|
||||
flush=True,
|
||||
)
|
||||
checkpoint = Path(instance_weights.url).name
|
||||
checkpoint_path = Path(torch.hub.get_dir()) / "checkpoints" / checkpoint
|
||||
del instance_model
|
||||
torch.cuda.empty_cache()
|
||||
|
||||
semantic_snapshot = Path(
|
||||
snapshot_download(
|
||||
repo_id=SEMANTIC_MODEL_ID,
|
||||
revision=SEMANTIC_REVISION,
|
||||
cache_dir=cache_root / "huggingface",
|
||||
allow_patterns=("config.json", "preprocessor_config.json", "pytorch_model.bin"),
|
||||
)
|
||||
)
|
||||
processor = AutoImageProcessor.from_pretrained(
|
||||
semantic_snapshot,
|
||||
local_files_only=True,
|
||||
use_fast=False,
|
||||
)
|
||||
semantic_model, loading = BeitForSemanticSegmentation.from_pretrained(
|
||||
semantic_snapshot,
|
||||
local_files_only=True,
|
||||
output_loading_info=True,
|
||||
)
|
||||
load_problems = {
|
||||
name: loading.get(name, [])
|
||||
for name in ("missing_keys", "unexpected_keys", "mismatched_keys", "error_msgs")
|
||||
if loading.get(name)
|
||||
}
|
||||
if load_problems:
|
||||
raise RuntimeError(
|
||||
"semantic checkpoint did not load exactly: " + json.dumps(load_problems)
|
||||
)
|
||||
semantic_model = semantic_model.to(device).eval()
|
||||
semantic_labels = {
|
||||
int(key): str(value) for key, value in semantic_model.config.id2label.items()
|
||||
}
|
||||
semantic_mapping = _semantic_target_map(semantic_labels)
|
||||
with Image.open(frame_paths[0]) as opened:
|
||||
warm_image = opened.convert("RGB")
|
||||
warm_inputs = processor(images=warm_image, return_tensors="pt")
|
||||
warm_inputs = {name: value.to(device) for name, value in warm_inputs.items()}
|
||||
with torch.inference_mode():
|
||||
_ = semantic_model(**warm_inputs).logits
|
||||
torch.cuda.synchronize()
|
||||
del warm_inputs
|
||||
|
||||
target_pixels: Counter[str] = Counter()
|
||||
with metadata_path.open("x", encoding="utf-8", newline="\n") as metadata_stream:
|
||||
with torch.inference_mode():
|
||||
for order, (frame, path, instances) in enumerate(
|
||||
zip(frames, frame_paths, instances_by_frame, strict=True),
|
||||
start=1,
|
||||
):
|
||||
with Image.open(path) as opened:
|
||||
image = np.asarray(opened.convert("RGB"), dtype=np.uint8)
|
||||
inputs = processor(images=Image.fromarray(image), return_tensors="pt")
|
||||
inputs = {name: value.to(device) for name, value in inputs.items()}
|
||||
torch.cuda.synchronize()
|
||||
started = time.perf_counter()
|
||||
logits = semantic_model(**inputs).logits
|
||||
resized = functional.interpolate(
|
||||
logits,
|
||||
size=image.shape[:2],
|
||||
mode="bilinear",
|
||||
align_corners=False,
|
||||
)
|
||||
source_semantic = (
|
||||
resized.argmax(dim=1)[0].detach().cpu().numpy().astype(np.uint8)
|
||||
)
|
||||
torch.cuda.synchronize()
|
||||
semantic_latencies.append((time.perf_counter() - started) * 1000.0)
|
||||
semantic = semantic_mapping[source_semantic]
|
||||
semantic = semantic.copy()
|
||||
semantic[~valid_mask] = 0
|
||||
instance_map = np.asarray(
|
||||
Image.open(instance_root / f"image-{order:03d}.png"),
|
||||
dtype=np.uint16,
|
||||
)
|
||||
for instance in instances:
|
||||
semantic[instance_map == int(instance["instance_id"])] = int(
|
||||
instance["draft_category_id"]
|
||||
)
|
||||
labels, counts = np.unique(semantic[valid_mask], return_counts=True)
|
||||
for label_id, count in zip(labels, counts, strict=True):
|
||||
target_pixels[TARGET_NAMES[int(label_id)]] += int(count)
|
||||
_write_png(semantic_root / f"image-{order:03d}.png", semantic)
|
||||
if order - 1 in preview_indices:
|
||||
_write_png(
|
||||
preview_root / f"image-{order:03d}.png",
|
||||
_preview(image, semantic, instance_map, instances),
|
||||
)
|
||||
metadata_stream.write(
|
||||
json.dumps(
|
||||
{
|
||||
"schema_version": FRAME_SCHEMA,
|
||||
"image_id": frame["image_id"],
|
||||
"frame_index": frame["frame_index"],
|
||||
"session_seconds": frame["session_seconds"],
|
||||
"role": frame["role"],
|
||||
"group_id": frame["group_id"],
|
||||
"instances": instances,
|
||||
"review_state": "unreviewed-model-draft",
|
||||
},
|
||||
ensure_ascii=False,
|
||||
sort_keys=True,
|
||||
separators=(",", ":"),
|
||||
allow_nan=False,
|
||||
)
|
||||
+ "\n"
|
||||
)
|
||||
del inputs, logits, resized
|
||||
if order % 16 == 0 or order == len(frames):
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"phase": "semantic",
|
||||
"processed": order,
|
||||
"total": len(frames),
|
||||
},
|
||||
sort_keys=True,
|
||||
),
|
||||
flush=True,
|
||||
)
|
||||
metadata_stream.flush()
|
||||
os.fsync(metadata_stream.fileno())
|
||||
model_files = _model_files(semantic_snapshot, checkpoint_path)
|
||||
del semantic_model
|
||||
torch.cuda.empty_cache()
|
||||
telemetry_summary = telemetry.summary()
|
||||
|
||||
wall_seconds = time.perf_counter() - wall_started
|
||||
resource_after = _resource_snapshot()
|
||||
identity = {
|
||||
"schema_version": RESULT_IDENTITY_SCHEMA,
|
||||
"evaluation_pack_id": pack["generation_id"],
|
||||
"evaluation_identity_sha256": pack["identity_sha256"],
|
||||
"valid_fov_generation_id": pack["identity"]["valid_fov_generation_id"],
|
||||
"pipeline": "maskrcnn-beit-e2-prelabels/v1",
|
||||
"mapping_profile": MAPPING_PROFILE,
|
||||
# String keys make the content identity stable across a JSON round trip.
|
||||
"target_categories": {str(category_id): name for category_id, name in TARGET_NAMES.items()},
|
||||
"instance_mapping": INSTANCE_MAPPING,
|
||||
"semantic_mapping": SEMANTIC_MAPPING,
|
||||
"instance_score_threshold": INSTANCE_SCORE_THRESHOLD,
|
||||
"instance_mask_threshold": INSTANCE_MASK_THRESHOLD,
|
||||
"runner_sha256": _sha256(Path(__file__).resolve(strict=True)),
|
||||
"models": {
|
||||
"instance": {
|
||||
"id": "torchvision/maskrcnn_resnet50_fpn_v2",
|
||||
"weights": str(instance_weights),
|
||||
},
|
||||
"semantic": {"id": SEMANTIC_MODEL_ID, "revision": SEMANTIC_REVISION},
|
||||
"files": model_files,
|
||||
},
|
||||
}
|
||||
identity_sha256 = hashlib.sha256(_canonical_json(identity)).hexdigest()
|
||||
result_id = f"evaluation-prelabels-{identity_sha256}"
|
||||
report = {
|
||||
"schema_version": REPORT_SCHEMA,
|
||||
"result_id": result_id,
|
||||
"created_at_utc": datetime.now(UTC)
|
||||
.isoformat(timespec="milliseconds")
|
||||
.replace("+00:00", "Z"),
|
||||
"state": "completed-unreviewed-model-draft",
|
||||
"identity": identity,
|
||||
"warning": (
|
||||
"These outputs are model-assisted prelabels, not ground truth and not an accuracy "
|
||||
"measurement. Every accepted label requires human review."
|
||||
),
|
||||
"metrics": {
|
||||
"frames": len(frames),
|
||||
"instances": sum(instance_counts.values()),
|
||||
"instances_by_target": dict(instance_counts.most_common()),
|
||||
"ignored_instances_by_source": dict(ignored_instance_counts.most_common()),
|
||||
"semantic_pixels_by_target_inside_valid_fov": dict(target_pixels.most_common()),
|
||||
"instance_forward_ms": _percentiles(instance_latencies),
|
||||
"semantic_forward_ms": _percentiles(semantic_latencies),
|
||||
"combined_forward_mean_ms": round(
|
||||
_percentiles(instance_latencies)["mean"]
|
||||
+ _percentiles(semantic_latencies)["mean"],
|
||||
6,
|
||||
),
|
||||
"wall_seconds": round(wall_seconds, 6),
|
||||
},
|
||||
"runtime": {
|
||||
"hostname": platform.node(),
|
||||
"platform": platform.platform(),
|
||||
"python": platform.python_version(),
|
||||
"torch": torch.__version__,
|
||||
"torchvision": torchvision.__version__,
|
||||
"transformers": transformers.__version__,
|
||||
"cuda_runtime": torch.version.cuda,
|
||||
"gpu": torch.cuda.get_device_name(),
|
||||
"gpu_compute_capability": list(torch.cuda.get_device_capability()),
|
||||
"cuda_peak_memory_allocated_mib": round(
|
||||
torch.cuda.max_memory_allocated() / 2**20,
|
||||
3,
|
||||
),
|
||||
"cuda_peak_memory_reserved_mib": round(
|
||||
torch.cuda.max_memory_reserved() / 2**20,
|
||||
3,
|
||||
),
|
||||
"process_peak_rss_mib": round(
|
||||
resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024,
|
||||
3,
|
||||
),
|
||||
"resource_delta": _resource_delta(resource_before, resource_after),
|
||||
"gpu_telemetry": telemetry_summary,
|
||||
},
|
||||
}
|
||||
_write_json(output_root / "run-report.json", report)
|
||||
artifacts = [
|
||||
_artifact(path, output_root)
|
||||
for path in sorted(output_root.rglob("*"))
|
||||
if path.is_file()
|
||||
]
|
||||
result = {
|
||||
"schema_version": RESULT_SCHEMA,
|
||||
"result_id": result_id,
|
||||
"identity_sha256": identity_sha256,
|
||||
"identity": identity,
|
||||
"review_state": "unreviewed-model-draft",
|
||||
"artifacts": artifacts,
|
||||
}
|
||||
_write_json(output_root / "result.json", result)
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"state": "completed-unreviewed-model-draft",
|
||||
"result_id": result_id,
|
||||
"frames": len(frames),
|
||||
"wall_seconds": round(wall_seconds, 6),
|
||||
},
|
||||
sort_keys=True,
|
||||
),
|
||||
flush=True,
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
def main() -> int:
|
||||
return _run(_arguments())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,875 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Benchmark E1 valid-FOV preprocessing variants on one sealed frame slice."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import resource
|
||||
import time
|
||||
from collections import Counter
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from run_recorded_perception_epoch import (
|
||||
INSTANCE_MASK_THRESHOLD,
|
||||
INSTANCE_SCORE_THRESHOLD,
|
||||
SEMANTIC_MODEL_ID,
|
||||
SEMANTIC_REVISION,
|
||||
_canonical_json,
|
||||
_GpuTelemetry,
|
||||
_instance_overlay,
|
||||
_model_files,
|
||||
_percentiles,
|
||||
_read_object,
|
||||
_read_timeline,
|
||||
_semantic_overlay,
|
||||
_sha256,
|
||||
_valid_sha256,
|
||||
_validate_job,
|
||||
_write_json,
|
||||
_write_png,
|
||||
)
|
||||
|
||||
QUALIFICATION_SCHEMA = "missioncore.recorded-qualification-slice/v1"
|
||||
QUALIFICATION_IDENTITY_SCHEMA = "missioncore.recorded-qualification-slice-identity/v1"
|
||||
VALID_FOV_SCHEMA = "missioncore.k1-valid-fov-mask/v1"
|
||||
VALID_FOV_IDENTITY_SCHEMA = "missioncore.k1-valid-fov-mask-identity/v1"
|
||||
REPORT_SCHEMA = "missioncore.perception-preprocessing-qualification-report/v1"
|
||||
RESULT_SCHEMA = "missioncore.perception-preprocessing-qualification-result/v1"
|
||||
RESULT_IDENTITY_SCHEMA = "missioncore.perception-preprocessing-qualification-identity/v1"
|
||||
VARIANTS = ("baseline", "valid-fov-fill", "valid-fov-crop")
|
||||
PREVIEW_FRAME_COUNT = 12
|
||||
|
||||
|
||||
def _arguments() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--job", type=Path, required=True)
|
||||
parser.add_argument("--frames", type=Path, required=True)
|
||||
parser.add_argument("--timeline", type=Path, required=True)
|
||||
parser.add_argument("--qualification", type=Path, required=True)
|
||||
parser.add_argument("--valid-fov-root", type=Path, required=True)
|
||||
parser.add_argument("--output", type=Path, required=True)
|
||||
parser.add_argument("--cache", type=Path, required=True)
|
||||
parser.add_argument("--calibration-sha256", required=True)
|
||||
parser.add_argument("--calibration-slot", required=True)
|
||||
parser.add_argument("--telemetry-interval-seconds", type=float, default=1.0)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def _load_qualification(path: Path, job: dict[str, Any]) -> tuple[dict[str, Any], list[int]]:
|
||||
manifest = _read_object(path.resolve(strict=True))
|
||||
identity = manifest.get("identity")
|
||||
identity_sha256 = manifest.get("identity_sha256")
|
||||
input_document = job["input"]
|
||||
if (
|
||||
manifest.get("schema_version") != QUALIFICATION_SCHEMA
|
||||
or not isinstance(identity, dict)
|
||||
or identity.get("schema_version") != QUALIFICATION_IDENTITY_SCHEMA
|
||||
or not _valid_sha256(identity_sha256)
|
||||
or hashlib.sha256(_canonical_json(identity)).hexdigest() != identity_sha256
|
||||
or manifest.get("generation_id") != f"qualification-slice-{identity_sha256}"
|
||||
or identity.get("job_id") != job["job_id"]
|
||||
or identity.get("input_sha256") != job["input_sha256"]
|
||||
or identity.get("source_id") != input_document["source_id"]
|
||||
or identity.get("codec_epoch") != input_document["codec_epoch"]
|
||||
or identity.get("source_frame_count") != input_document["segment_count"]
|
||||
or identity.get("policy") != "uniform-frame-index-full-epoch/v1"
|
||||
):
|
||||
raise RuntimeError("qualification slice identity is invalid")
|
||||
rows = manifest.get("frames")
|
||||
selected = identity.get("selected_frames")
|
||||
if not isinstance(rows, list) or not isinstance(selected, list) or len(rows) != len(selected):
|
||||
raise RuntimeError("qualification slice frame set is invalid")
|
||||
indices: list[int] = []
|
||||
previous = -1
|
||||
for row, selected_row in zip(rows, selected, strict=True):
|
||||
if not isinstance(row, dict) or not isinstance(selected_row, dict):
|
||||
raise RuntimeError("qualification frame descriptor is invalid")
|
||||
frame_index = row.get("frame_index")
|
||||
sequence = row.get("sequence")
|
||||
digest = row.get("segment_sha256")
|
||||
if (
|
||||
not isinstance(frame_index, int)
|
||||
or isinstance(frame_index, bool)
|
||||
or not previous < frame_index < int(input_document["segment_count"])
|
||||
or sequence != frame_index + 1
|
||||
or not _valid_sha256(digest)
|
||||
or selected_row
|
||||
!= {
|
||||
"frame_index": frame_index,
|
||||
"sequence": sequence,
|
||||
"segment_sha256": digest,
|
||||
}
|
||||
):
|
||||
raise RuntimeError("qualification frame binding is invalid")
|
||||
indices.append(frame_index)
|
||||
previous = frame_index
|
||||
if not indices:
|
||||
raise RuntimeError("qualification slice is empty")
|
||||
return manifest, indices
|
||||
|
||||
|
||||
def _load_valid_fov(
|
||||
root: Path,
|
||||
*,
|
||||
job: dict[str, Any],
|
||||
calibration_sha256: str,
|
||||
calibration_slot: str,
|
||||
) -> tuple[dict[str, Any], Any, tuple[int, int, int, int]]:
|
||||
import numpy as np
|
||||
from PIL import Image
|
||||
|
||||
resolved = root.resolve(strict=True)
|
||||
manifest = _read_object(resolved / "manifest.json")
|
||||
identity = manifest.get("identity")
|
||||
identity_sha256 = manifest.get("identity_sha256")
|
||||
artifact = manifest.get("artifact")
|
||||
geometry = manifest.get("geometry")
|
||||
if (
|
||||
manifest.get("schema_version") != VALID_FOV_SCHEMA
|
||||
or not isinstance(identity, dict)
|
||||
or identity.get("schema_version") != VALID_FOV_IDENTITY_SCHEMA
|
||||
or not _valid_sha256(identity_sha256)
|
||||
or hashlib.sha256(_canonical_json(identity)).hexdigest() != identity_sha256
|
||||
or manifest.get("generation_id") != f"valid-fov-mask-{identity_sha256}"
|
||||
or identity.get("calibration_sha256") != calibration_sha256
|
||||
or identity.get("calibration_slot") != calibration_slot
|
||||
or identity.get("source_id") != job["input"]["source_id"]
|
||||
or identity.get("admitted_resolution") != [800, 600]
|
||||
or not isinstance(artifact, dict)
|
||||
or artifact.get("path") != "mask.png"
|
||||
or artifact.get("media_type") != "image/png"
|
||||
or not _valid_sha256(artifact.get("sha256"))
|
||||
or not isinstance(geometry, dict)
|
||||
):
|
||||
raise RuntimeError("valid-FOV identity is invalid")
|
||||
mask_path = (resolved / "mask.png").resolve(strict=True)
|
||||
if mask_path.parent != resolved or _sha256(mask_path) != artifact["sha256"]:
|
||||
raise RuntimeError("valid-FOV mask artifact changed")
|
||||
with Image.open(mask_path) as opened:
|
||||
if opened.mode != "L" or opened.size != (800, 600):
|
||||
raise RuntimeError("valid-FOV mask format changed")
|
||||
mask = np.asarray(opened, dtype=np.uint8)
|
||||
if not np.isin(mask, (0, 255)).all():
|
||||
raise RuntimeError("valid-FOV mask is not binary")
|
||||
crop_value = geometry.get("crop_xyxy_exclusive")
|
||||
if (
|
||||
not isinstance(crop_value, list)
|
||||
or len(crop_value) != 4
|
||||
or not all(isinstance(value, int) and not isinstance(value, bool) for value in crop_value)
|
||||
):
|
||||
raise RuntimeError("valid-FOV crop is invalid")
|
||||
crop = tuple(int(value) for value in crop_value)
|
||||
left, top, right, bottom = crop
|
||||
if not 0 <= left < right <= 800 or not 0 <= top < bottom <= 600:
|
||||
raise RuntimeError("valid-FOV crop escapes the image")
|
||||
if int(np.count_nonzero(mask)) != geometry.get("valid_pixel_count"):
|
||||
raise RuntimeError("valid-FOV pixel count changed")
|
||||
return manifest, mask > 0, crop
|
||||
|
||||
|
||||
def _variant_image(
|
||||
image: Any,
|
||||
valid_mask: Any,
|
||||
crop: tuple[int, int, int, int],
|
||||
variant: str,
|
||||
) -> Any:
|
||||
import numpy as np
|
||||
|
||||
if variant == "baseline":
|
||||
return image
|
||||
masked = np.where(valid_mask[..., None], image, 0).astype(np.uint8, copy=False)
|
||||
if variant == "valid-fov-fill":
|
||||
return masked
|
||||
if variant == "valid-fov-crop":
|
||||
left, top, right, bottom = crop
|
||||
return masked[top:bottom, left:right]
|
||||
raise RuntimeError(f"unknown preprocessing variant: {variant}")
|
||||
|
||||
|
||||
def _full_mask(
|
||||
local_mask: Any,
|
||||
*,
|
||||
variant: str,
|
||||
valid_mask: Any,
|
||||
crop: tuple[int, int, int, int],
|
||||
) -> Any:
|
||||
import numpy as np
|
||||
|
||||
if variant != "valid-fov-crop":
|
||||
result = np.asarray(local_mask, dtype=bool)
|
||||
else:
|
||||
result = np.zeros(valid_mask.shape, dtype=bool)
|
||||
left, top, right, bottom = crop
|
||||
if local_mask.shape != (bottom - top, right - left):
|
||||
raise RuntimeError("cropped instance mask shape changed")
|
||||
result[top:bottom, left:right] = local_mask
|
||||
if variant != "baseline":
|
||||
result = np.logical_and(result, valid_mask)
|
||||
return result
|
||||
|
||||
|
||||
def _full_box(
|
||||
box_xyxy: list[float],
|
||||
*,
|
||||
variant: str,
|
||||
crop: tuple[int, int, int, int],
|
||||
) -> list[float]:
|
||||
if variant == "valid-fov-crop":
|
||||
left, top, _right, _bottom = crop
|
||||
return [
|
||||
box_xyxy[0] + left,
|
||||
box_xyxy[1] + top,
|
||||
box_xyxy[2] + left,
|
||||
box_xyxy[3] + top,
|
||||
]
|
||||
return box_xyxy
|
||||
|
||||
|
||||
def _full_semantic(
|
||||
local: Any,
|
||||
*,
|
||||
variant: str,
|
||||
valid_mask: Any,
|
||||
crop: tuple[int, int, int, int],
|
||||
) -> Any:
|
||||
import numpy as np
|
||||
|
||||
if variant != "valid-fov-crop":
|
||||
result = np.asarray(local, dtype=np.uint8)
|
||||
else:
|
||||
result = np.full(valid_mask.shape, 255, dtype=np.uint8)
|
||||
left, top, right, bottom = crop
|
||||
if local.shape != (bottom - top, right - left):
|
||||
raise RuntimeError("cropped semantic mask shape changed")
|
||||
result[top:bottom, left:right] = local
|
||||
if variant != "baseline":
|
||||
result = result.copy()
|
||||
result[~valid_mask] = 255
|
||||
return result
|
||||
|
||||
|
||||
def _resource_snapshot() -> dict[str, float]:
|
||||
usage = resource.getrusage(resource.RUSAGE_SELF)
|
||||
result = {
|
||||
"user_cpu_seconds": float(usage.ru_utime),
|
||||
"system_cpu_seconds": float(usage.ru_stime),
|
||||
"minor_page_faults": float(usage.ru_minflt),
|
||||
"major_page_faults": float(usage.ru_majflt),
|
||||
"voluntary_context_switches": float(usage.ru_nvcsw),
|
||||
"involuntary_context_switches": float(usage.ru_nivcsw),
|
||||
}
|
||||
try:
|
||||
for line in Path("/proc/self/io").read_text(encoding="ascii").splitlines():
|
||||
name, value = line.split(":", 1)
|
||||
if name in {"read_bytes", "write_bytes", "rchar", "wchar"}:
|
||||
result[f"proc_io_{name}"] = float(value.strip())
|
||||
except (OSError, UnicodeDecodeError, ValueError):
|
||||
pass
|
||||
return result
|
||||
|
||||
|
||||
def _resource_delta(before: dict[str, float], after: dict[str, float]) -> dict[str, float]:
|
||||
return {
|
||||
name: round(value - before.get(name, value), 6)
|
||||
for name, value in after.items()
|
||||
if name in before
|
||||
}
|
||||
|
||||
|
||||
def _metric_state() -> dict[str, Any]:
|
||||
return {
|
||||
"instance_source_decode_ms": [],
|
||||
"instance_preprocess_ms": [],
|
||||
"instance_host_to_device_ms": [],
|
||||
"instance_forward_ms": [],
|
||||
"instance_postprocess_ms": [],
|
||||
"semantic_source_decode_ms": [],
|
||||
"semantic_preprocess_ms": [],
|
||||
"semantic_host_to_device_ms": [],
|
||||
"semantic_forward_ms": [],
|
||||
"semantic_postprocess_ms": [],
|
||||
"instances": 0,
|
||||
"instance_scores": [],
|
||||
"instance_labels": Counter(),
|
||||
"huge_masks_over_half_valid_fov": 0,
|
||||
"huge_boxes_over_half_full_frame": 0,
|
||||
"predicted_mask_pixels": 0,
|
||||
"predicted_mask_pixels_outside_valid_fov": 0,
|
||||
"frames_with_outside_valid_fov_instance_pixels": 0,
|
||||
"semantic_labels": Counter(),
|
||||
"semantic_disagreement_with_baseline": [],
|
||||
}
|
||||
|
||||
|
||||
def _latency_document(state: dict[str, Any]) -> dict[str, Any]:
|
||||
return {
|
||||
branch: {
|
||||
name: _percentiles(
|
||||
[float(value) for value in state[f"{branch}_{name}"]]
|
||||
)
|
||||
for name in (
|
||||
"source_decode_ms",
|
||||
"preprocess_ms",
|
||||
"host_to_device_ms",
|
||||
"forward_ms",
|
||||
"postprocess_ms",
|
||||
)
|
||||
}
|
||||
for branch in ("instance", "semantic")
|
||||
}
|
||||
|
||||
|
||||
def _metric_document(state: dict[str, Any], frame_count: int) -> dict[str, Any]:
|
||||
scores = [float(value) for value in state["instance_scores"]]
|
||||
outside = int(state["predicted_mask_pixels_outside_valid_fov"])
|
||||
total = int(state["predicted_mask_pixels"])
|
||||
disagreements = [float(value) for value in state["semantic_disagreement_with_baseline"]]
|
||||
return {
|
||||
"frames": frame_count,
|
||||
"latency_ms": _latency_document(state),
|
||||
"instances": int(state["instances"]),
|
||||
"instances_per_frame": round(int(state["instances"]) / frame_count, 6),
|
||||
"instance_score": _percentiles(scores),
|
||||
"instance_labels": dict(state["instance_labels"].most_common()),
|
||||
"huge_masks_over_half_valid_fov": int(state["huge_masks_over_half_valid_fov"]),
|
||||
"huge_boxes_over_half_full_frame": int(state["huge_boxes_over_half_full_frame"]),
|
||||
"predicted_mask_pixels": total,
|
||||
"predicted_mask_pixels_outside_valid_fov": outside,
|
||||
"predicted_mask_outside_fraction": round(outside / total, 9) if total else 0.0,
|
||||
"frames_with_outside_valid_fov_instance_pixels": int(
|
||||
state["frames_with_outside_valid_fov_instance_pixels"]
|
||||
),
|
||||
"semantic_labels_inside_valid_fov": dict(state["semantic_labels"].most_common()),
|
||||
"semantic_disagreement_with_baseline_inside_valid_fov": _percentiles(disagreements),
|
||||
}
|
||||
|
||||
|
||||
def _preview_indices(indices: list[int]) -> set[int]:
|
||||
admitted = min(PREVIEW_FRAME_COUNT, len(indices))
|
||||
if admitted == 1:
|
||||
return {indices[len(indices) // 2]}
|
||||
positions = {
|
||||
(position * (len(indices) - 1) + (admitted - 1) // 2) // (admitted - 1)
|
||||
for position in range(admitted)
|
||||
}
|
||||
return {indices[position] for position in positions}
|
||||
|
||||
|
||||
def _run(args: argparse.Namespace) -> int:
|
||||
if not _valid_sha256(args.calibration_sha256):
|
||||
raise RuntimeError("calibration SHA-256 is invalid")
|
||||
job_root = args.job.resolve(strict=True)
|
||||
frames_root = args.frames.resolve(strict=True)
|
||||
timeline_path = args.timeline.resolve(strict=True)
|
||||
output_root = args.output.resolve()
|
||||
cache_root = args.cache.resolve(strict=True)
|
||||
if output_root.exists() or not frames_root.is_dir():
|
||||
raise RuntimeError("output must be absent and frames must be a directory")
|
||||
job = _validate_job(job_root)
|
||||
input_document = job["input"]
|
||||
frame_count = int(input_document["segment_count"])
|
||||
timeline = input_document["timeline"]
|
||||
timestamps = _read_timeline(
|
||||
timeline_path,
|
||||
frame_count,
|
||||
float(timeline["start_seconds"]),
|
||||
float(timeline["end_seconds"]),
|
||||
)
|
||||
qualification, selected_indices = _load_qualification(args.qualification, job)
|
||||
valid_fov, valid_mask, crop = _load_valid_fov(
|
||||
args.valid_fov_root,
|
||||
job=job,
|
||||
calibration_sha256=args.calibration_sha256,
|
||||
calibration_slot=args.calibration_slot,
|
||||
)
|
||||
frame_paths = [frames_root / f"frame-{index + 1:06d}.png" for index in selected_indices]
|
||||
if not all(path.is_file() for path in frame_paths):
|
||||
raise RuntimeError("qualification decoded frame set is incomplete")
|
||||
|
||||
import numpy as np
|
||||
import torch
|
||||
import torch.nn.functional as functional
|
||||
import torchvision
|
||||
import transformers
|
||||
from huggingface_hub import snapshot_download
|
||||
from PIL import Image
|
||||
from torchvision.models.detection import (
|
||||
MaskRCNN_ResNet50_FPN_V2_Weights,
|
||||
maskrcnn_resnet50_fpn_v2,
|
||||
)
|
||||
from transformers import AutoImageProcessor, BeitForSemanticSegmentation
|
||||
|
||||
if not torch.cuda.is_available():
|
||||
raise RuntimeError("CUDA is required for preprocessing qualification")
|
||||
device = torch.device("cuda:0")
|
||||
output_root.mkdir(mode=0o700, parents=True, exist_ok=False)
|
||||
previews_root = output_root / "previews"
|
||||
previews_root.mkdir(mode=0o700)
|
||||
for variant in VARIANTS:
|
||||
(previews_root / variant).mkdir(mode=0o700)
|
||||
telemetry_path = output_root / "gpu-telemetry.jsonl"
|
||||
metrics = {variant: _metric_state() for variant in VARIANTS}
|
||||
preview_frame_indices = _preview_indices(selected_indices)
|
||||
preview_instances: dict[tuple[str, int], tuple[Any, list[dict[str, Any]]]] = {}
|
||||
valid_pixel_count = int(np.count_nonzero(valid_mask))
|
||||
total_pixel_count = int(valid_mask.size)
|
||||
resource_before = _resource_snapshot()
|
||||
wall_started = time.perf_counter()
|
||||
|
||||
with telemetry_path.open("x", encoding="utf-8", newline="\n") as telemetry_stream:
|
||||
with _GpuTelemetry(telemetry_stream, args.telemetry_interval_seconds) as telemetry:
|
||||
instance_weights = MaskRCNN_ResNet50_FPN_V2_Weights.DEFAULT
|
||||
instance_model = maskrcnn_resnet50_fpn_v2(weights=instance_weights).to(device).eval()
|
||||
instance_labels = list(instance_weights.meta["categories"])
|
||||
with Image.open(frame_paths[0]) as opened:
|
||||
warm_image = opened.convert("RGB")
|
||||
with torch.inference_mode():
|
||||
warm_tensor = instance_weights.transforms()(warm_image).to(device)
|
||||
_ = instance_model([warm_tensor])[0]
|
||||
torch.cuda.synchronize()
|
||||
del warm_tensor
|
||||
torch.cuda.reset_peak_memory_stats()
|
||||
|
||||
with torch.inference_mode():
|
||||
for order, (frame_index, path) in enumerate(
|
||||
zip(selected_indices, frame_paths, strict=True),
|
||||
start=1,
|
||||
):
|
||||
decode_started = time.perf_counter()
|
||||
with Image.open(path) as opened:
|
||||
image = np.asarray(opened.convert("RGB"), dtype=np.uint8)
|
||||
decode_ms = (time.perf_counter() - decode_started) * 1000.0
|
||||
for variant in VARIANTS:
|
||||
state = metrics[variant]
|
||||
state["instance_source_decode_ms"].append(decode_ms)
|
||||
preprocess_started = time.perf_counter()
|
||||
variant_image = _variant_image(image, valid_mask, crop, variant)
|
||||
tensor = instance_weights.transforms()(Image.fromarray(variant_image))
|
||||
state["instance_preprocess_ms"].append(
|
||||
(time.perf_counter() - preprocess_started) * 1000.0
|
||||
)
|
||||
torch.cuda.synchronize()
|
||||
transfer_started = time.perf_counter()
|
||||
tensor = tensor.to(device)
|
||||
torch.cuda.synchronize()
|
||||
state["instance_host_to_device_ms"].append(
|
||||
(time.perf_counter() - transfer_started) * 1000.0
|
||||
)
|
||||
torch.cuda.synchronize()
|
||||
forward_started = time.perf_counter()
|
||||
prediction = instance_model([tensor])[0]
|
||||
torch.cuda.synchronize()
|
||||
state["instance_forward_ms"].append(
|
||||
(time.perf_counter() - forward_started) * 1000.0
|
||||
)
|
||||
post_started = time.perf_counter()
|
||||
scores = prediction["scores"].detach().cpu().numpy()
|
||||
keep = np.flatnonzero(scores >= INSTANCE_SCORE_THRESHOLD)
|
||||
index_map = np.zeros(valid_mask.shape, dtype=np.uint16)
|
||||
instances: list[dict[str, Any]] = []
|
||||
frame_outside = 0
|
||||
for output_index in keep:
|
||||
local_mask = (
|
||||
prediction["masks"][output_index, 0].detach().cpu().numpy()
|
||||
>= INSTANCE_MASK_THRESHOLD
|
||||
)
|
||||
full_mask = _full_mask(
|
||||
local_mask,
|
||||
variant=variant,
|
||||
valid_mask=valid_mask,
|
||||
crop=crop,
|
||||
)
|
||||
mask_pixels = int(np.count_nonzero(full_mask))
|
||||
if mask_pixels < 8:
|
||||
continue
|
||||
original_full_mask = _full_mask(
|
||||
local_mask,
|
||||
variant="baseline" if variant != "valid-fov-crop" else variant,
|
||||
valid_mask=np.ones_like(valid_mask),
|
||||
crop=crop,
|
||||
)
|
||||
outside_pixels = int(
|
||||
np.count_nonzero(np.logical_and(original_full_mask, ~valid_mask))
|
||||
)
|
||||
frame_outside += outside_pixels
|
||||
label_id = int(prediction["labels"][output_index].item())
|
||||
score = float(scores[output_index])
|
||||
box_values = (
|
||||
prediction["boxes"][output_index].detach().cpu().tolist()
|
||||
)
|
||||
box = _full_box(
|
||||
[float(value) for value in box_values],
|
||||
variant=variant,
|
||||
crop=crop,
|
||||
)
|
||||
instance_id = len(instances) + 1
|
||||
index_map[np.logical_and(full_mask, index_map == 0)] = instance_id
|
||||
instances.append(
|
||||
{
|
||||
"instance_id": instance_id,
|
||||
"class_id": label_id,
|
||||
"label": instance_labels[label_id],
|
||||
"score": round(score, 9),
|
||||
"box_xyxy": [round(value, 6) for value in box],
|
||||
"mask_pixels": mask_pixels,
|
||||
"outside_valid_fov_pixels": outside_pixels,
|
||||
}
|
||||
)
|
||||
state["instances"] += 1
|
||||
state["instance_scores"].append(score)
|
||||
state["instance_labels"][instance_labels[label_id]] += 1
|
||||
state["predicted_mask_pixels"] += int(
|
||||
np.count_nonzero(original_full_mask)
|
||||
)
|
||||
state["predicted_mask_pixels_outside_valid_fov"] += outside_pixels
|
||||
if mask_pixels / valid_pixel_count > 0.5:
|
||||
state["huge_masks_over_half_valid_fov"] += 1
|
||||
box_area = max(0.0, box[2] - box[0]) * max(0.0, box[3] - box[1])
|
||||
if box_area / total_pixel_count > 0.5:
|
||||
state["huge_boxes_over_half_full_frame"] += 1
|
||||
if frame_outside:
|
||||
state["frames_with_outside_valid_fov_instance_pixels"] += 1
|
||||
if frame_index in preview_frame_indices:
|
||||
preview_instances[(variant, frame_index)] = (index_map, instances)
|
||||
state["instance_postprocess_ms"].append(
|
||||
(time.perf_counter() - post_started) * 1000.0
|
||||
)
|
||||
del tensor, prediction
|
||||
if order % 32 == 0 or order == len(selected_indices):
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"phase": "instance",
|
||||
"qualification_frames_processed": order,
|
||||
"qualification_frames_total": len(selected_indices),
|
||||
},
|
||||
sort_keys=True,
|
||||
),
|
||||
flush=True,
|
||||
)
|
||||
|
||||
checkpoint = Path(instance_weights.url).name
|
||||
checkpoint_path = Path(torch.hub.get_dir()) / "checkpoints" / checkpoint
|
||||
del instance_model
|
||||
torch.cuda.empty_cache()
|
||||
|
||||
semantic_snapshot = Path(
|
||||
snapshot_download(
|
||||
repo_id=SEMANTIC_MODEL_ID,
|
||||
revision=SEMANTIC_REVISION,
|
||||
cache_dir=cache_root / "huggingface",
|
||||
allow_patterns=(
|
||||
"config.json",
|
||||
"preprocessor_config.json",
|
||||
"pytorch_model.bin",
|
||||
),
|
||||
)
|
||||
)
|
||||
processor = AutoImageProcessor.from_pretrained(
|
||||
semantic_snapshot,
|
||||
local_files_only=True,
|
||||
use_fast=False,
|
||||
)
|
||||
semantic_model, loading = BeitForSemanticSegmentation.from_pretrained(
|
||||
semantic_snapshot,
|
||||
local_files_only=True,
|
||||
output_loading_info=True,
|
||||
)
|
||||
load_problems = {
|
||||
name: loading.get(name, [])
|
||||
for name in ("missing_keys", "unexpected_keys", "mismatched_keys", "error_msgs")
|
||||
if loading.get(name)
|
||||
}
|
||||
if load_problems:
|
||||
raise RuntimeError(
|
||||
"semantic checkpoint did not load exactly: " + json.dumps(load_problems)
|
||||
)
|
||||
semantic_model = semantic_model.to(device).eval()
|
||||
semantic_labels = {
|
||||
int(key): str(value) for key, value in semantic_model.config.id2label.items()
|
||||
}
|
||||
with Image.open(frame_paths[0]) as opened:
|
||||
warm_image = opened.convert("RGB")
|
||||
warm_inputs = processor(images=warm_image, return_tensors="pt")
|
||||
warm_inputs = {name: value.to(device) for name, value in warm_inputs.items()}
|
||||
with torch.inference_mode():
|
||||
_ = semantic_model(**warm_inputs).logits
|
||||
torch.cuda.synchronize()
|
||||
del warm_inputs
|
||||
|
||||
preview_sequence = {
|
||||
frame_index: sequence
|
||||
for sequence, frame_index in enumerate(sorted(preview_frame_indices), start=1)
|
||||
}
|
||||
with torch.inference_mode():
|
||||
for order, (frame_index, path) in enumerate(
|
||||
zip(selected_indices, frame_paths, strict=True),
|
||||
start=1,
|
||||
):
|
||||
decode_started = time.perf_counter()
|
||||
with Image.open(path) as opened:
|
||||
image = np.asarray(opened.convert("RGB"), dtype=np.uint8)
|
||||
decode_ms = (time.perf_counter() - decode_started) * 1000.0
|
||||
semantic_by_variant: dict[str, Any] = {}
|
||||
for variant in VARIANTS:
|
||||
state = metrics[variant]
|
||||
state["semantic_source_decode_ms"].append(decode_ms)
|
||||
preprocess_started = time.perf_counter()
|
||||
variant_image = _variant_image(image, valid_mask, crop, variant)
|
||||
inputs = processor(
|
||||
images=Image.fromarray(variant_image),
|
||||
return_tensors="pt",
|
||||
)
|
||||
state["semantic_preprocess_ms"].append(
|
||||
(time.perf_counter() - preprocess_started) * 1000.0
|
||||
)
|
||||
torch.cuda.synchronize()
|
||||
transfer_started = time.perf_counter()
|
||||
inputs = {name: value.to(device) for name, value in inputs.items()}
|
||||
torch.cuda.synchronize()
|
||||
state["semantic_host_to_device_ms"].append(
|
||||
(time.perf_counter() - transfer_started) * 1000.0
|
||||
)
|
||||
torch.cuda.synchronize()
|
||||
forward_started = time.perf_counter()
|
||||
logits = semantic_model(**inputs).logits
|
||||
resized = functional.interpolate(
|
||||
logits,
|
||||
size=variant_image.shape[:2],
|
||||
mode="bilinear",
|
||||
align_corners=False,
|
||||
)
|
||||
local_semantic = (
|
||||
resized.argmax(dim=1)[0].detach().cpu().numpy().astype(np.uint8)
|
||||
)
|
||||
torch.cuda.synchronize()
|
||||
state["semantic_forward_ms"].append(
|
||||
(time.perf_counter() - forward_started) * 1000.0
|
||||
)
|
||||
post_started = time.perf_counter()
|
||||
semantic = _full_semantic(
|
||||
local_semantic,
|
||||
variant=variant,
|
||||
valid_mask=valid_mask,
|
||||
crop=crop,
|
||||
)
|
||||
semantic_by_variant[variant] = semantic
|
||||
labels, counts = np.unique(semantic[valid_mask], return_counts=True)
|
||||
for label_id, count in zip(labels, counts, strict=True):
|
||||
state["semantic_labels"][
|
||||
semantic_labels.get(int(label_id), f"class-{int(label_id)}")
|
||||
] += int(count)
|
||||
if frame_index in preview_frame_indices:
|
||||
display_image = _variant_image(image, valid_mask, crop, variant)
|
||||
if variant == "valid-fov-crop":
|
||||
full_display = np.zeros_like(image)
|
||||
left, top, right, bottom = crop
|
||||
full_display[top:bottom, left:right] = display_image
|
||||
display_image = full_display
|
||||
display_semantic = semantic.copy()
|
||||
display_semantic[display_semantic == 255] = 0
|
||||
semantic_overlay, _classes = _semantic_overlay(
|
||||
display_image,
|
||||
display_semantic,
|
||||
semantic_labels,
|
||||
)
|
||||
instance_map, instances = preview_instances[(variant, frame_index)]
|
||||
overlay = _instance_overlay(
|
||||
semantic_overlay,
|
||||
instance_map,
|
||||
instances,
|
||||
)
|
||||
sequence = preview_sequence[frame_index]
|
||||
_write_png(
|
||||
previews_root / variant / f"frame-{sequence:03d}.png",
|
||||
overlay,
|
||||
)
|
||||
state["semantic_postprocess_ms"].append(
|
||||
(time.perf_counter() - post_started) * 1000.0
|
||||
)
|
||||
del inputs, logits, resized
|
||||
baseline_semantic = semantic_by_variant["baseline"]
|
||||
for variant in VARIANTS:
|
||||
selected_semantic = semantic_by_variant[variant][valid_mask]
|
||||
disagreement = float(
|
||||
np.mean(selected_semantic != baseline_semantic[valid_mask])
|
||||
)
|
||||
metrics[variant]["semantic_disagreement_with_baseline"].append(
|
||||
disagreement
|
||||
)
|
||||
if order % 32 == 0 or order == len(selected_indices):
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"phase": "semantic",
|
||||
"qualification_frames_processed": order,
|
||||
"qualification_frames_total": len(selected_indices),
|
||||
},
|
||||
sort_keys=True,
|
||||
),
|
||||
flush=True,
|
||||
)
|
||||
del semantic_model
|
||||
torch.cuda.empty_cache()
|
||||
|
||||
wall_seconds = time.perf_counter() - wall_started
|
||||
resource_after = _resource_snapshot()
|
||||
model_files = _model_files(semantic_snapshot, checkpoint_path)
|
||||
metric_documents = {
|
||||
variant: _metric_document(metrics[variant], len(selected_indices))
|
||||
for variant in VARIANTS
|
||||
}
|
||||
identity = {
|
||||
"schema_version": RESULT_IDENTITY_SCHEMA,
|
||||
"job_id": job["job_id"],
|
||||
"input_sha256": job["input_sha256"],
|
||||
"qualification_generation_id": qualification["generation_id"],
|
||||
"qualification_identity_sha256": qualification["identity_sha256"],
|
||||
"valid_fov_generation_id": valid_fov["generation_id"],
|
||||
"valid_fov_identity_sha256": valid_fov["identity_sha256"],
|
||||
"calibration_sha256": args.calibration_sha256,
|
||||
"calibration_slot": args.calibration_slot,
|
||||
"configuration": {
|
||||
"pipeline": "recorded-preprocessing-ab-mask-crop/v1",
|
||||
"variants": list(VARIANTS),
|
||||
"precision": "fp32",
|
||||
"batch_size": 1,
|
||||
"execution": "same-model-load-interleaved-variants-per-frame",
|
||||
"instance_score_threshold": INSTANCE_SCORE_THRESHOLD,
|
||||
"instance_mask_threshold": INSTANCE_MASK_THRESHOLD,
|
||||
"runner_sha256": _sha256(Path(__file__).resolve(strict=True)),
|
||||
"preview_frame_count": len(preview_frame_indices),
|
||||
},
|
||||
"models": {
|
||||
"instance": {
|
||||
"id": "torchvision/maskrcnn_resnet50_fpn_v2",
|
||||
"weights": str(instance_weights),
|
||||
},
|
||||
"semantic": {
|
||||
"id": SEMANTIC_MODEL_ID,
|
||||
"revision": SEMANTIC_REVISION,
|
||||
"checkpoint_load": "exact-no-missing-unexpected-or-mismatched-keys",
|
||||
},
|
||||
"files": model_files,
|
||||
},
|
||||
}
|
||||
identity_sha256 = hashlib.sha256(_canonical_json(identity)).hexdigest()
|
||||
result_id = f"qualification-result-{identity_sha256}"
|
||||
report = {
|
||||
"schema_version": REPORT_SCHEMA,
|
||||
"result_id": result_id,
|
||||
"created_at_utc": datetime.now(UTC)
|
||||
.isoformat(timespec="milliseconds")
|
||||
.replace("+00:00", "Z"),
|
||||
"state": "completed",
|
||||
"identity": identity,
|
||||
"input": {
|
||||
"job_id": job["job_id"],
|
||||
"input_sha256": job["input_sha256"],
|
||||
"session_id": input_document["session_id"],
|
||||
"source_id": input_document["source_id"],
|
||||
"codec_epoch": input_document["codec_epoch"],
|
||||
"source_frame_count": frame_count,
|
||||
"qualification_frame_count": len(selected_indices),
|
||||
"selected_frame_indices": selected_indices,
|
||||
"selected_session_seconds": [timestamps[index] for index in selected_indices],
|
||||
},
|
||||
"valid_fov": {
|
||||
"generation_id": valid_fov["generation_id"],
|
||||
"calibration_sha256": args.calibration_sha256,
|
||||
"calibration_slot": args.calibration_slot,
|
||||
"geometry": valid_fov["geometry"],
|
||||
},
|
||||
"metrics": metric_documents,
|
||||
"runtime": {
|
||||
"wall_seconds": round(wall_seconds, 6),
|
||||
"variant_model_evaluations": len(selected_indices) * len(VARIANTS) * 2,
|
||||
"gpu": torch.cuda.get_device_name(),
|
||||
"torch": torch.__version__,
|
||||
"torchvision": torchvision.__version__,
|
||||
"transformers": transformers.__version__,
|
||||
"cuda_runtime": torch.version.cuda,
|
||||
"cuda_peak_memory_allocated_mib": round(
|
||||
torch.cuda.max_memory_allocated() / 2**20,
|
||||
3,
|
||||
),
|
||||
"cuda_peak_memory_reserved_mib": round(
|
||||
torch.cuda.max_memory_reserved() / 2**20,
|
||||
3,
|
||||
),
|
||||
"process_peak_rss_mib": round(
|
||||
resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024,
|
||||
3,
|
||||
),
|
||||
"resource_delta": _resource_delta(resource_before, resource_after),
|
||||
"system_load_average": [round(value, 6) for value in os.getloadavg()],
|
||||
"gpu_telemetry": telemetry.summary(),
|
||||
},
|
||||
"quality_status": {
|
||||
"ground_truth": "absent",
|
||||
"decision_scope": "preprocessing proxy comparison only",
|
||||
"not_accepted": [
|
||||
"2D mIoU/AP",
|
||||
"3D geometry",
|
||||
"distance accuracy",
|
||||
"tracking",
|
||||
"safety",
|
||||
],
|
||||
},
|
||||
}
|
||||
_write_json(output_root / "run-report.json", report)
|
||||
preview_artifacts = []
|
||||
for path in sorted(previews_root.glob("*/*.png")):
|
||||
preview_artifacts.append(
|
||||
{
|
||||
"path": path.relative_to(output_root).as_posix(),
|
||||
"byte_length": path.stat().st_size,
|
||||
"sha256": _sha256(path),
|
||||
}
|
||||
)
|
||||
result = {
|
||||
"schema_version": RESULT_SCHEMA,
|
||||
"result_id": result_id,
|
||||
"identity_sha256": identity_sha256,
|
||||
"identity": identity,
|
||||
"artifacts": {
|
||||
"run_report": {
|
||||
"path": "run-report.json",
|
||||
"byte_length": (output_root / "run-report.json").stat().st_size,
|
||||
"sha256": _sha256(output_root / "run-report.json"),
|
||||
},
|
||||
"gpu_telemetry": {
|
||||
"path": "gpu-telemetry.jsonl",
|
||||
"byte_length": telemetry_path.stat().st_size,
|
||||
"sha256": _sha256(telemetry_path),
|
||||
},
|
||||
"previews": preview_artifacts,
|
||||
},
|
||||
}
|
||||
_write_json(output_root / "result.json", result)
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"state": "completed",
|
||||
"result_id": result_id,
|
||||
"qualification_frames": len(selected_indices),
|
||||
"wall_seconds": round(wall_seconds, 6),
|
||||
},
|
||||
sort_keys=True,
|
||||
),
|
||||
flush=True,
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
def main() -> int:
|
||||
return _run(_arguments())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,824 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Run and seal a complete recorded panoptic-perception camera epoch."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import importlib.metadata
|
||||
import json
|
||||
import os
|
||||
import platform
|
||||
import resource
|
||||
import statistics
|
||||
import subprocess
|
||||
import threading
|
||||
import time
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path, PurePosixPath
|
||||
from typing import Any, TextIO
|
||||
|
||||
INSTANCE_SCORE_THRESHOLD = 0.5
|
||||
INSTANCE_MASK_THRESHOLD = 0.5
|
||||
SEMANTIC_ALPHA = 0.46
|
||||
INSTANCE_ALPHA = 0.58
|
||||
SEMANTIC_MODEL_ID = "microsoft/beit-base-finetuned-ade-640-640"
|
||||
SEMANTIC_REVISION = "a8b6f5ef4acb2ea55d882989deaa02d39401e2b2"
|
||||
RESULT_SCHEMA = "missioncore.recorded-perception-result/v2"
|
||||
REPORT_SCHEMA = "missioncore.perception-run-report/v1"
|
||||
FRAME_SCHEMA = "missioncore.panoptic-frame/v1"
|
||||
SAFE_SHA256 = set("0123456789abcdef")
|
||||
|
||||
|
||||
def _arguments() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser()
|
||||
subparsers = parser.add_subparsers(dest="command", required=True)
|
||||
preflight = subparsers.add_parser("preflight")
|
||||
preflight.add_argument("--job", type=Path, required=True)
|
||||
preflight.add_argument("--cache", type=Path, required=True)
|
||||
|
||||
run = subparsers.add_parser("run")
|
||||
run.add_argument("--job", type=Path, required=True)
|
||||
run.add_argument("--frames", type=Path, required=True)
|
||||
run.add_argument("--timeline", type=Path, required=True)
|
||||
run.add_argument("--output", type=Path, required=True)
|
||||
run.add_argument("--cache", type=Path, required=True)
|
||||
run.add_argument("--calibration-sha256", required=True)
|
||||
run.add_argument("--calibration-slot", required=True)
|
||||
run.add_argument("--telemetry-interval-seconds", type=float, default=1.0)
|
||||
|
||||
finalize = subparsers.add_parser("finalize")
|
||||
finalize.add_argument("--output", type=Path, required=True)
|
||||
finalize.add_argument("--video", type=Path, required=True)
|
||||
finalize.add_argument("--masks", type=Path, required=True)
|
||||
finalize.add_argument("--extract-seconds", type=float, required=True)
|
||||
finalize.add_argument("--encode-seconds", type=float, required=True)
|
||||
finalize.add_argument("--wall-seconds", type=float, required=True)
|
||||
finalize.add_argument("--encoder", required=True)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def _sha256(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as stream:
|
||||
for chunk in iter(lambda: stream.read(1024 * 1024), b""):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def _canonical_json(value: object) -> bytes:
|
||||
return json.dumps(
|
||||
value,
|
||||
ensure_ascii=False,
|
||||
sort_keys=True,
|
||||
separators=(",", ":"),
|
||||
allow_nan=False,
|
||||
).encode("utf-8")
|
||||
|
||||
|
||||
def _read_object(path: Path) -> dict[str, Any]:
|
||||
value = json.loads(path.read_text(encoding="utf-8-sig"))
|
||||
if not isinstance(value, dict):
|
||||
raise RuntimeError(f"{path.name} is not a JSON object")
|
||||
return value
|
||||
|
||||
|
||||
def _write_json(path: Path, value: object) -> None:
|
||||
temporary = path.with_name(f".{path.name}.{os.getpid()}.tmp")
|
||||
with temporary.open("x", encoding="utf-8", newline="\n") as stream:
|
||||
json.dump(value, stream, ensure_ascii=False, sort_keys=True, indent=2, allow_nan=False)
|
||||
stream.write("\n")
|
||||
stream.flush()
|
||||
os.fsync(stream.fileno())
|
||||
os.replace(temporary, path)
|
||||
|
||||
|
||||
def _valid_sha256(value: object) -> bool:
|
||||
return isinstance(value, str) and len(value) == 64 and set(value) <= SAFE_SHA256
|
||||
|
||||
|
||||
def _safe_job_path(root: Path, encoded: object) -> Path:
|
||||
if not isinstance(encoded, str):
|
||||
raise RuntimeError("job artifact path is not a string")
|
||||
relative = PurePosixPath(encoded)
|
||||
if relative.is_absolute() or ".." in relative.parts or not relative.parts:
|
||||
raise RuntimeError("job artifact path is unsafe")
|
||||
path = root.joinpath(*relative.parts).resolve(strict=True)
|
||||
if not path.is_file() or not path.is_relative_to(root):
|
||||
raise RuntimeError("job artifact escapes the job root")
|
||||
return path
|
||||
|
||||
|
||||
def _validate_job(job_root: Path) -> dict[str, Any]:
|
||||
root = job_root.resolve(strict=True)
|
||||
job = _read_object(root / "job.json")
|
||||
input_document = job.get("input")
|
||||
if (
|
||||
job.get("schema_version") != "missioncore.compute-job/v1"
|
||||
or not isinstance(input_document, dict)
|
||||
or hashlib.sha256(_canonical_json(input_document)).hexdigest() != job.get("input_sha256")
|
||||
or job.get("job_id") != f"recorded-camera-{str(job.get('input_sha256'))[:24]}"
|
||||
):
|
||||
raise RuntimeError("compute job identity is invalid")
|
||||
files = input_document.get("files")
|
||||
if not isinstance(files, list) or not files:
|
||||
raise RuntimeError("compute job has no files")
|
||||
seen: set[str] = set()
|
||||
total_bytes = 0
|
||||
for artifact in files:
|
||||
if not isinstance(artifact, dict):
|
||||
raise RuntimeError("compute job artifact descriptor is invalid")
|
||||
encoded = artifact.get("path")
|
||||
if not isinstance(encoded, str) or encoded in seen:
|
||||
raise RuntimeError("compute job artifact descriptor is duplicated")
|
||||
seen.add(encoded)
|
||||
path = _safe_job_path(root, encoded)
|
||||
byte_length = artifact.get("byte_length")
|
||||
digest = artifact.get("sha256")
|
||||
if (
|
||||
not isinstance(byte_length, int)
|
||||
or byte_length < 1
|
||||
or path.stat().st_size != byte_length
|
||||
or not _valid_sha256(digest)
|
||||
or _sha256(path) != digest
|
||||
):
|
||||
raise RuntimeError(f"compute job artifact changed: {encoded}")
|
||||
total_bytes += byte_length
|
||||
if total_bytes != input_document.get("byte_length"):
|
||||
raise RuntimeError("compute job byte length changed")
|
||||
return job
|
||||
|
||||
|
||||
def _read_timeline(path: Path, expected_count: int, start: float, end: float) -> list[float]:
|
||||
timestamps: list[float] = []
|
||||
with path.open("r", encoding="utf-8") as stream:
|
||||
for expected_index, line in enumerate(stream):
|
||||
value = json.loads(line)
|
||||
if not isinstance(value, dict) or value.get("frame_index") != expected_index:
|
||||
raise RuntimeError("decoded frame timeline index changed")
|
||||
session_seconds = value.get("session_seconds")
|
||||
if not isinstance(session_seconds, (int, float)):
|
||||
raise RuntimeError("decoded frame timestamp is invalid")
|
||||
timestamp = float(session_seconds)
|
||||
if timestamps and timestamp <= timestamps[-1]:
|
||||
raise RuntimeError("decoded frame timestamps are not increasing")
|
||||
timestamps.append(timestamp)
|
||||
if len(timestamps) != expected_count:
|
||||
raise RuntimeError("decoded frame count differs from the camera job")
|
||||
if timestamps[0] < start - 0.001 or timestamps[-1] > end + 0.001:
|
||||
raise RuntimeError("decoded frame timeline escapes the camera epoch")
|
||||
return timestamps
|
||||
|
||||
|
||||
def _palette(index: int) -> tuple[int, int, int]:
|
||||
digest = hashlib.sha256(f"mission-core-segment-{index}".encode()).digest()
|
||||
return (64 + digest[0] % 176, 64 + digest[1] % 176, 64 + digest[2] % 176)
|
||||
|
||||
|
||||
def _blend(image: Any, colors: Any, alpha: float) -> Any:
|
||||
import numpy as np
|
||||
|
||||
return np.clip(
|
||||
image.astype(np.float32) * (1.0 - alpha) + colors.astype(np.float32) * alpha,
|
||||
0,
|
||||
255,
|
||||
).astype(np.uint8)
|
||||
|
||||
|
||||
def _semantic_overlay(
|
||||
image: Any,
|
||||
index_map: Any,
|
||||
labels: dict[int, str],
|
||||
) -> tuple[Any, list[dict[str, Any]]]:
|
||||
import numpy as np
|
||||
|
||||
colors = np.zeros_like(image)
|
||||
counts = np.bincount(index_map.reshape(-1), minlength=max(labels) + 1)
|
||||
present = np.flatnonzero(counts)
|
||||
for index in present:
|
||||
colors[index_map == index] = _palette(int(index))
|
||||
total = int(index_map.size)
|
||||
classes = [
|
||||
{
|
||||
"id": int(index),
|
||||
"label": labels.get(int(index), f"class-{int(index)}"),
|
||||
"pixels": int(counts[index]),
|
||||
"fraction": round(float(counts[index]) / total, 9),
|
||||
}
|
||||
for index in sorted(present, key=lambda value: int(counts[value]), reverse=True)
|
||||
]
|
||||
return _blend(image, colors, SEMANTIC_ALPHA), classes
|
||||
|
||||
|
||||
def _instance_overlay(image: Any, index_map: Any, instances: list[dict[str, Any]]) -> Any:
|
||||
import numpy as np
|
||||
from PIL import Image, ImageDraw
|
||||
|
||||
result = image.copy()
|
||||
for item in instances:
|
||||
instance_id = int(item["instance_id"])
|
||||
mask = index_map == instance_id
|
||||
color = np.asarray(_palette(500 + instance_id), dtype=np.uint8)
|
||||
if bool(mask.any()):
|
||||
result[mask] = _blend(
|
||||
result[mask],
|
||||
np.broadcast_to(color, result[mask].shape),
|
||||
INSTANCE_ALPHA,
|
||||
)
|
||||
canvas = Image.fromarray(result)
|
||||
draw = ImageDraw.Draw(canvas)
|
||||
for item in instances:
|
||||
color = _palette(500 + int(item["instance_id"]))
|
||||
x1, y1, x2, y2 = (int(round(value)) for value in item["box_xyxy"])
|
||||
draw.rectangle((x1, y1, x2, y2), outline=color, width=2)
|
||||
label = f"{item['label']} {float(item['score']):.0%}"
|
||||
text_box = draw.textbbox((x1, max(0, y1 - 13)), label)
|
||||
draw.rectangle(text_box, fill=(7, 8, 10))
|
||||
draw.text((x1, max(0, y1 - 13)), label, fill=color)
|
||||
return np.asarray(canvas, dtype=np.uint8)
|
||||
|
||||
|
||||
def _write_png(path: Path, array: Any) -> None:
|
||||
from PIL import Image
|
||||
|
||||
Image.fromarray(array).save(path, format="PNG", optimize=False)
|
||||
|
||||
|
||||
def _percentiles(values: list[float]) -> dict[str, float]:
|
||||
if not values:
|
||||
return {"mean": 0.0, "p50": 0.0, "p95": 0.0, "max": 0.0}
|
||||
ordered = sorted(values)
|
||||
|
||||
def percentile(fraction: float) -> float:
|
||||
index = min(len(ordered) - 1, max(0, round((len(ordered) - 1) * fraction)))
|
||||
return round(ordered[index], 6)
|
||||
|
||||
return {
|
||||
"mean": round(statistics.fmean(values), 6),
|
||||
"p50": percentile(0.5),
|
||||
"p95": percentile(0.95),
|
||||
"max": round(max(values), 6),
|
||||
}
|
||||
|
||||
|
||||
class _GpuTelemetry:
|
||||
def __init__(self, stream: TextIO, interval_seconds: float) -> None:
|
||||
if not 0.25 <= interval_seconds <= 60:
|
||||
raise RuntimeError("telemetry interval is outside bounds")
|
||||
self._stream = stream
|
||||
self._interval = interval_seconds
|
||||
self._stop = threading.Event()
|
||||
self._thread = threading.Thread(target=self._run, name="gpu-telemetry", daemon=True)
|
||||
self.samples: list[dict[str, float]] = []
|
||||
|
||||
def __enter__(self) -> _GpuTelemetry:
|
||||
self._thread.start()
|
||||
return self
|
||||
|
||||
def __exit__(self, *_: object) -> None:
|
||||
self._stop.set()
|
||||
self._thread.join(timeout=self._interval + 5)
|
||||
|
||||
def _run(self) -> None:
|
||||
query = (
|
||||
"timestamp,utilization.gpu,utilization.memory,memory.used,memory.total,"
|
||||
"temperature.gpu,power.draw"
|
||||
)
|
||||
while not self._stop.is_set():
|
||||
before = time.time()
|
||||
try:
|
||||
completed = subprocess.run(
|
||||
[
|
||||
"nvidia-smi",
|
||||
f"--query-gpu={query}",
|
||||
"--format=csv,noheader,nounits",
|
||||
],
|
||||
capture_output=True,
|
||||
check=True,
|
||||
text=True,
|
||||
timeout=5,
|
||||
)
|
||||
values = [item.strip() for item in completed.stdout.splitlines()[0].split(",")]
|
||||
sample = {
|
||||
"epoch_seconds": round(before, 6),
|
||||
"gpu_utilization_percent": float(values[1]),
|
||||
"gpu_memory_utilization_percent": float(values[2]),
|
||||
"gpu_memory_used_mib": float(values[3]),
|
||||
"gpu_memory_total_mib": float(values[4]),
|
||||
"gpu_temperature_c": float(values[5]),
|
||||
"gpu_power_w": float(values[6]),
|
||||
}
|
||||
self.samples.append(sample)
|
||||
self._stream.write(json.dumps(sample, sort_keys=True) + "\n")
|
||||
self._stream.flush()
|
||||
except (OSError, subprocess.SubprocessError, ValueError, IndexError):
|
||||
pass
|
||||
self._stop.wait(max(0.0, self._interval - (time.time() - before)))
|
||||
|
||||
def summary(self) -> dict[str, object]:
|
||||
fields = (
|
||||
"gpu_utilization_percent",
|
||||
"gpu_memory_utilization_percent",
|
||||
"gpu_memory_used_mib",
|
||||
"gpu_temperature_c",
|
||||
"gpu_power_w",
|
||||
)
|
||||
return {
|
||||
"sample_count": len(self.samples),
|
||||
"interval_seconds": self._interval,
|
||||
**{
|
||||
field: _percentiles([float(sample[field]) for sample in self.samples])
|
||||
for field in fields
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def _model_files(semantic_snapshot: Path, checkpoint_path: Path) -> list[dict[str, object]]:
|
||||
files = [
|
||||
{"name": f"beit/{path.name}", "bytes": path.stat().st_size, "sha256": _sha256(path)}
|
||||
for path in sorted(semantic_snapshot.iterdir())
|
||||
if path.is_file()
|
||||
]
|
||||
files.append(
|
||||
{
|
||||
"name": f"torchvision/{checkpoint_path.name}",
|
||||
"bytes": checkpoint_path.stat().st_size,
|
||||
"sha256": _sha256(checkpoint_path),
|
||||
}
|
||||
)
|
||||
return files
|
||||
|
||||
|
||||
def _preflight(args: argparse.Namespace) -> int:
|
||||
job = _validate_job(args.job.resolve(strict=True))
|
||||
cache_root = args.cache.resolve(strict=True)
|
||||
|
||||
import torch
|
||||
import torchvision # noqa: F401
|
||||
import transformers # noqa: F401
|
||||
from huggingface_hub import snapshot_download
|
||||
from torchvision.models.detection import MaskRCNN_ResNet50_FPN_V2_Weights
|
||||
|
||||
if not torch.cuda.is_available() or torch.cuda.device_count() < 1:
|
||||
raise RuntimeError("CUDA device 0 is unavailable")
|
||||
cuda_probe = torch.zeros(1, device="cuda:0")
|
||||
torch.cuda.synchronize()
|
||||
del cuda_probe
|
||||
checkpoint = Path(MaskRCNN_ResNet50_FPN_V2_Weights.DEFAULT.url).name
|
||||
checkpoint_path = Path(torch.hub.get_dir()) / "checkpoints" / checkpoint
|
||||
if not checkpoint_path.is_file() or checkpoint_path.stat().st_size < 1:
|
||||
raise RuntimeError("cached instance checkpoint is unavailable")
|
||||
semantic_snapshot = Path(
|
||||
snapshot_download(
|
||||
repo_id=SEMANTIC_MODEL_ID,
|
||||
revision=SEMANTIC_REVISION,
|
||||
cache_dir=cache_root / "huggingface",
|
||||
allow_patterns=("config.json", "preprocessor_config.json", "pytorch_model.bin"),
|
||||
)
|
||||
)
|
||||
required = ("config.json", "preprocessor_config.json", "pytorch_model.bin")
|
||||
if any(not (semantic_snapshot / name).is_file() for name in required):
|
||||
raise RuntimeError("cached semantic checkpoint is incomplete")
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"state": "preflight-ready",
|
||||
"job_id": job["job_id"],
|
||||
"cuda_device": torch.cuda.get_device_name(),
|
||||
},
|
||||
sort_keys=True,
|
||||
),
|
||||
flush=True,
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
def _run(args: argparse.Namespace) -> int:
|
||||
if not _valid_sha256(args.calibration_sha256):
|
||||
raise RuntimeError("calibration SHA-256 is invalid")
|
||||
job_root = args.job.resolve(strict=True)
|
||||
frames_root = args.frames.resolve(strict=True)
|
||||
timeline_path = args.timeline.resolve(strict=True)
|
||||
output_root = args.output.resolve()
|
||||
cache_root = args.cache.resolve()
|
||||
if output_root.exists() or not frames_root.is_dir():
|
||||
raise RuntimeError("output must be absent and frames must be a directory")
|
||||
job = _validate_job(job_root)
|
||||
input_document = job["input"]
|
||||
timeline = input_document["timeline"]
|
||||
segment_count = int(input_document["segment_count"])
|
||||
frame_paths = [frames_root / f"frame-{index:06d}.png" for index in range(1, segment_count + 1)]
|
||||
if not all(path.is_file() for path in frame_paths):
|
||||
raise RuntimeError("decoded frame set is incomplete")
|
||||
if len(list(frames_root.glob("frame-*.png"))) != segment_count:
|
||||
raise RuntimeError("decoded frame set contains unexpected files")
|
||||
timestamps = _read_timeline(
|
||||
timeline_path,
|
||||
segment_count,
|
||||
float(timeline["start_seconds"]),
|
||||
float(timeline["end_seconds"]),
|
||||
)
|
||||
|
||||
import numpy as np
|
||||
import torch
|
||||
import torch.nn.functional as functional
|
||||
import torchvision
|
||||
import transformers
|
||||
from huggingface_hub import snapshot_download
|
||||
from PIL import Image
|
||||
from torchvision.models.detection import (
|
||||
MaskRCNN_ResNet50_FPN_V2_Weights,
|
||||
maskrcnn_resnet50_fpn_v2,
|
||||
)
|
||||
from transformers import AutoImageProcessor, BeitForSemanticSegmentation
|
||||
|
||||
if not torch.cuda.is_available():
|
||||
raise RuntimeError("CUDA is required for a perception epoch run")
|
||||
cuda_index = 0
|
||||
device = torch.device(f"cuda:{cuda_index}")
|
||||
cache_root.mkdir(mode=0o700, parents=True, exist_ok=True)
|
||||
output_root.mkdir(mode=0o700, parents=True, exist_ok=False)
|
||||
instance_root = output_root / "instance-masks"
|
||||
semantic_root = output_root / "semantic-masks"
|
||||
overlay_root = output_root / "overlay-frames"
|
||||
instance_root.mkdir(mode=0o700)
|
||||
semantic_root.mkdir(mode=0o700)
|
||||
overlay_root.mkdir(mode=0o700)
|
||||
telemetry_path = output_root / "gpu-telemetry.jsonl"
|
||||
started = time.perf_counter()
|
||||
instance_latencies: list[float] = []
|
||||
semantic_latencies: list[float] = []
|
||||
instances_by_frame: list[list[dict[str, Any]]] = []
|
||||
total_instances = 0
|
||||
|
||||
with telemetry_path.open("x", encoding="utf-8", newline="\n") as telemetry_stream:
|
||||
with _GpuTelemetry(telemetry_stream, args.telemetry_interval_seconds) as telemetry:
|
||||
instance_weights = MaskRCNN_ResNet50_FPN_V2_Weights.DEFAULT
|
||||
instance_model = maskrcnn_resnet50_fpn_v2(weights=instance_weights).to(device).eval()
|
||||
instance_labels = list(instance_weights.meta["categories"])
|
||||
with torch.inference_mode():
|
||||
for frame_index, path in enumerate(frame_paths):
|
||||
with Image.open(path) as opened:
|
||||
image = np.asarray(opened.convert("RGB"), dtype=np.uint8)
|
||||
tensor = instance_weights.transforms()(Image.fromarray(image)).to(device)
|
||||
torch.cuda.synchronize()
|
||||
before = time.perf_counter()
|
||||
prediction = instance_model([tensor])[0]
|
||||
torch.cuda.synchronize()
|
||||
instance_latencies.append((time.perf_counter() - before) * 1000.0)
|
||||
scores = prediction["scores"].detach().cpu().numpy()
|
||||
keep = np.flatnonzero(scores >= INSTANCE_SCORE_THRESHOLD)
|
||||
index_map = np.zeros(image.shape[:2], dtype=np.uint16)
|
||||
instances: list[dict[str, Any]] = []
|
||||
for output_index in keep:
|
||||
mask = (
|
||||
prediction["masks"][output_index, 0].detach().cpu().numpy()
|
||||
>= INSTANCE_MASK_THRESHOLD
|
||||
)
|
||||
mask_pixels = int(mask.sum())
|
||||
if mask_pixels < 8:
|
||||
continue
|
||||
label_id = int(prediction["labels"][output_index].item())
|
||||
instance_id = len(instances) + 1
|
||||
index_map[np.logical_and(mask, index_map == 0)] = instance_id
|
||||
instances.append(
|
||||
{
|
||||
"instance_id": instance_id,
|
||||
"class_id": label_id,
|
||||
"label": instance_labels[label_id],
|
||||
"score": round(float(scores[output_index]), 9),
|
||||
"box_xyxy": [
|
||||
round(float(value), 6)
|
||||
for value in prediction["boxes"][output_index]
|
||||
.detach()
|
||||
.cpu()
|
||||
.tolist()
|
||||
],
|
||||
"mask_pixels": mask_pixels,
|
||||
}
|
||||
)
|
||||
_write_png(instance_root / f"frame-{frame_index + 1:06d}.png", index_map)
|
||||
instances_by_frame.append(instances)
|
||||
total_instances += len(instances)
|
||||
if (frame_index + 1) % 100 == 0 or frame_index + 1 == segment_count:
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"phase": "instance",
|
||||
"frames_processed": frame_index + 1,
|
||||
"frames_total": segment_count,
|
||||
},
|
||||
sort_keys=True,
|
||||
),
|
||||
flush=True,
|
||||
)
|
||||
checkpoint = Path(instance_weights.url).name
|
||||
checkpoint_path = Path(torch.hub.get_dir()) / "checkpoints" / checkpoint
|
||||
del instance_model
|
||||
torch.cuda.empty_cache()
|
||||
|
||||
semantic_snapshot = Path(
|
||||
snapshot_download(
|
||||
repo_id=SEMANTIC_MODEL_ID,
|
||||
revision=SEMANTIC_REVISION,
|
||||
cache_dir=cache_root / "huggingface",
|
||||
allow_patterns=("config.json", "preprocessor_config.json", "pytorch_model.bin"),
|
||||
)
|
||||
)
|
||||
processor = AutoImageProcessor.from_pretrained(
|
||||
semantic_snapshot,
|
||||
local_files_only=True,
|
||||
use_fast=False,
|
||||
)
|
||||
semantic_model, loading = BeitForSemanticSegmentation.from_pretrained(
|
||||
semantic_snapshot,
|
||||
local_files_only=True,
|
||||
output_loading_info=True,
|
||||
)
|
||||
load_problems = {
|
||||
name: loading.get(name, [])
|
||||
for name in ("missing_keys", "unexpected_keys", "mismatched_keys", "error_msgs")
|
||||
if loading.get(name)
|
||||
}
|
||||
if load_problems:
|
||||
raise RuntimeError(
|
||||
"semantic checkpoint did not load exactly: "
|
||||
+ json.dumps(load_problems)
|
||||
)
|
||||
semantic_model = semantic_model.to(device).eval()
|
||||
semantic_labels = {
|
||||
int(key): str(value)
|
||||
for key, value in semantic_model.config.id2label.items()
|
||||
}
|
||||
frame_metadata = output_root / "frames.jsonl"
|
||||
with frame_metadata.open("x", encoding="utf-8", newline="\n") as metadata_stream:
|
||||
with torch.inference_mode():
|
||||
for frame_index, (path, session_seconds) in enumerate(
|
||||
zip(frame_paths, timestamps, strict=True)
|
||||
):
|
||||
with Image.open(path) as opened:
|
||||
image = np.asarray(opened.convert("RGB"), dtype=np.uint8)
|
||||
inputs = processor(images=Image.fromarray(image), return_tensors="pt")
|
||||
inputs = {name: value.to(device) for name, value in inputs.items()}
|
||||
torch.cuda.synchronize()
|
||||
before = time.perf_counter()
|
||||
logits = semantic_model(**inputs).logits
|
||||
resized = functional.interpolate(
|
||||
logits,
|
||||
size=image.shape[:2],
|
||||
mode="bilinear",
|
||||
align_corners=False,
|
||||
)
|
||||
semantic = resized.argmax(dim=1)[0].detach().cpu().numpy().astype(np.uint8)
|
||||
torch.cuda.synchronize()
|
||||
semantic_latencies.append((time.perf_counter() - before) * 1000.0)
|
||||
semantic_overlay, classes = _semantic_overlay(
|
||||
image,
|
||||
semantic,
|
||||
semantic_labels,
|
||||
)
|
||||
instance_map = np.asarray(
|
||||
Image.open(instance_root / f"frame-{frame_index + 1:06d}.png"),
|
||||
dtype=np.uint16,
|
||||
)
|
||||
overlay = _instance_overlay(
|
||||
semantic_overlay,
|
||||
instance_map,
|
||||
instances_by_frame[frame_index],
|
||||
)
|
||||
_write_png(semantic_root / f"frame-{frame_index + 1:06d}.png", semantic)
|
||||
_write_png(overlay_root / f"frame-{frame_index + 1:06d}.png", overlay)
|
||||
metadata_stream.write(
|
||||
json.dumps(
|
||||
{
|
||||
"schema_version": FRAME_SCHEMA,
|
||||
"frame_index": frame_index,
|
||||
"sequence": frame_index + 1,
|
||||
"session_seconds": round(session_seconds, 9),
|
||||
"instances": instances_by_frame[frame_index],
|
||||
"semantic_classes": classes,
|
||||
},
|
||||
ensure_ascii=False,
|
||||
sort_keys=True,
|
||||
separators=(",", ":"),
|
||||
allow_nan=False,
|
||||
)
|
||||
+ "\n"
|
||||
)
|
||||
if (frame_index + 1) % 100 == 0 or frame_index + 1 == segment_count:
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"phase": "semantic",
|
||||
"frames_processed": frame_index + 1,
|
||||
"frames_total": segment_count,
|
||||
},
|
||||
sort_keys=True,
|
||||
),
|
||||
flush=True,
|
||||
)
|
||||
metadata_stream.flush()
|
||||
os.fsync(metadata_stream.fileno())
|
||||
del semantic_model
|
||||
torch.cuda.empty_cache()
|
||||
|
||||
inference_elapsed = time.perf_counter() - started
|
||||
model_files = _model_files(semantic_snapshot, checkpoint_path)
|
||||
report = {
|
||||
"schema_version": REPORT_SCHEMA,
|
||||
"run_id": f"{job['job_id']}-panoptic-v1",
|
||||
"created_at_utc": datetime.now(UTC)
|
||||
.isoformat(timespec="milliseconds")
|
||||
.replace("+00:00", "Z"),
|
||||
"state": "inference-complete-awaiting-publication",
|
||||
"input": {
|
||||
"job_id": job["job_id"],
|
||||
"input_sha256": job["input_sha256"],
|
||||
"session_id": input_document["session_id"],
|
||||
"source_id": input_document["source_id"],
|
||||
"codec_epoch": input_document["codec_epoch"],
|
||||
"segment_count": segment_count,
|
||||
"byte_length": input_document["byte_length"],
|
||||
"timeline_start_seconds": timeline["start_seconds"],
|
||||
"timeline_end_seconds": timeline["end_seconds"],
|
||||
"frame_timeline_sha256": _sha256(timeline_path),
|
||||
},
|
||||
"calibration": {
|
||||
"content_identity_sha256": args.calibration_sha256,
|
||||
"camera_slot": args.calibration_slot,
|
||||
},
|
||||
"configuration": {
|
||||
"pipeline": "recorded-panoptic-maskrcnn-beit/v1",
|
||||
"instance_score_threshold": INSTANCE_SCORE_THRESHOLD,
|
||||
"instance_mask_threshold": INSTANCE_MASK_THRESHOLD,
|
||||
"semantic_alpha": SEMANTIC_ALPHA,
|
||||
"instance_alpha": INSTANCE_ALPHA,
|
||||
"batch_size": 1,
|
||||
"frame_policy": "all-frames-no-sampling",
|
||||
"runner_sha256": _sha256(Path(__file__).resolve(strict=True)),
|
||||
},
|
||||
"models": {
|
||||
"instance": {
|
||||
"id": "torchvision/maskrcnn_resnet50_fpn_v2",
|
||||
"weights": str(instance_weights),
|
||||
"score_threshold": INSTANCE_SCORE_THRESHOLD,
|
||||
},
|
||||
"semantic": {
|
||||
"id": SEMANTIC_MODEL_ID,
|
||||
"revision": SEMANTIC_REVISION,
|
||||
"checkpoint_load": "exact-no-missing-unexpected-or-mismatched-keys",
|
||||
},
|
||||
"files": model_files,
|
||||
},
|
||||
"runtime": {
|
||||
"hostname": platform.node(),
|
||||
"platform": platform.platform(),
|
||||
"python": platform.python_version(),
|
||||
"torch": torch.__version__,
|
||||
"torchvision": torchvision.__version__,
|
||||
"transformers": transformers.__version__,
|
||||
"cuda_runtime": torch.version.cuda,
|
||||
"gpu": torch.cuda.get_device_name(),
|
||||
"gpu_compute_capability": list(torch.cuda.get_device_capability()),
|
||||
},
|
||||
"metrics": {
|
||||
"frames_expected": segment_count,
|
||||
"frames_processed": segment_count,
|
||||
"frames_failed": 0,
|
||||
"frames_skipped": 0,
|
||||
"instances": total_instances,
|
||||
"inference_wall_seconds": round(inference_elapsed, 6),
|
||||
"inference_frames_per_second": round(segment_count / inference_elapsed, 6),
|
||||
"instance_latency_ms": _percentiles(instance_latencies),
|
||||
"semantic_latency_ms": _percentiles(semantic_latencies),
|
||||
"cuda_peak_memory_allocated_mib": round(
|
||||
torch.cuda.max_memory_allocated() / 2**20,
|
||||
3,
|
||||
),
|
||||
"cuda_peak_memory_reserved_mib": round(
|
||||
torch.cuda.max_memory_reserved() / 2**20,
|
||||
3,
|
||||
),
|
||||
"process_peak_rss_mib": round(
|
||||
resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024,
|
||||
3,
|
||||
),
|
||||
"system_load_average": [round(value, 6) for value in os.getloadavg()],
|
||||
"gpu_telemetry": telemetry.summary(),
|
||||
},
|
||||
"versions": {
|
||||
name: importlib.metadata.version(name)
|
||||
for name in ("numpy", "pillow", "torch", "torchvision", "transformers")
|
||||
},
|
||||
}
|
||||
_write_json(output_root / "run-report.partial.json", report)
|
||||
print(json.dumps({"state": "inference-complete", "frames": segment_count}, sort_keys=True))
|
||||
return 0
|
||||
|
||||
|
||||
def _artifact(
|
||||
path: Path,
|
||||
kind: str,
|
||||
media_type: str,
|
||||
schema_version: str | None = None,
|
||||
) -> dict[str, object]:
|
||||
result: dict[str, object] = {
|
||||
"kind": kind,
|
||||
"path": path.name,
|
||||
"media_type": media_type,
|
||||
"byte_length": path.stat().st_size,
|
||||
"sha256": _sha256(path),
|
||||
}
|
||||
if schema_version is not None:
|
||||
result["schema_version"] = schema_version
|
||||
return result
|
||||
|
||||
|
||||
def _finalize(args: argparse.Namespace) -> int:
|
||||
output_root = args.output.resolve(strict=True)
|
||||
video = args.video.resolve(strict=True)
|
||||
masks = args.masks.resolve(strict=True)
|
||||
if video.parent != output_root or masks.parent != output_root:
|
||||
raise RuntimeError("published artifacts must be direct result children")
|
||||
partial_path = output_root / "run-report.partial.json"
|
||||
frames_path = output_root / "frames.jsonl"
|
||||
telemetry_path = output_root / "gpu-telemetry.jsonl"
|
||||
partial = _read_object(partial_path)
|
||||
if partial.get("schema_version") != REPORT_SCHEMA:
|
||||
raise RuntimeError("partial run report is incompatible")
|
||||
input_document = partial["input"]
|
||||
configuration = partial["configuration"]
|
||||
models = partial["models"]
|
||||
identity = {
|
||||
"schema_version": "missioncore.recorded-perception-identity/v2",
|
||||
"job_id": input_document["job_id"],
|
||||
"input_sha256": input_document["input_sha256"],
|
||||
"calibration": partial["calibration"],
|
||||
"configuration": configuration,
|
||||
"models": models,
|
||||
"publication": {
|
||||
"video_encoder": args.encoder,
|
||||
"video_media_type": "video/mp4",
|
||||
"mask_archive_media_type": "application/gzip",
|
||||
},
|
||||
}
|
||||
identity_sha256 = hashlib.sha256(_canonical_json(identity)).hexdigest()
|
||||
result_id = f"result-{identity_sha256}"
|
||||
final_report = {
|
||||
**partial,
|
||||
"state": "published",
|
||||
"result_id": result_id,
|
||||
"publication": {
|
||||
"extract_seconds": round(args.extract_seconds, 6),
|
||||
"encode_seconds": round(args.encode_seconds, 6),
|
||||
"wall_seconds": round(args.wall_seconds, 6),
|
||||
"end_to_end_frames_per_second": round(
|
||||
int(partial["metrics"]["frames_processed"]) / args.wall_seconds,
|
||||
6,
|
||||
),
|
||||
"encoder": args.encoder,
|
||||
},
|
||||
}
|
||||
report_path = output_root / "run-report.json"
|
||||
_write_json(report_path, final_report)
|
||||
artifacts = [
|
||||
_artifact(video, "panoptic-overlay-video", "video/mp4"),
|
||||
_artifact(masks, "panoptic-mask-archive", "application/gzip"),
|
||||
_artifact(frames_path, "panoptic-frame-metadata", "application/x-ndjson", FRAME_SCHEMA),
|
||||
_artifact(telemetry_path, "worker-gpu-telemetry", "application/x-ndjson"),
|
||||
_artifact(report_path, "perception-run-report", "application/json", REPORT_SCHEMA),
|
||||
]
|
||||
result = {
|
||||
"schema_version": RESULT_SCHEMA,
|
||||
"result_id": result_id,
|
||||
"identity_sha256": identity_sha256,
|
||||
"identity": identity,
|
||||
"created_at_utc": final_report["created_at_utc"],
|
||||
"job_id": input_document["job_id"],
|
||||
"input_sha256": input_document["input_sha256"],
|
||||
"session_id": input_document["session_id"],
|
||||
"source_id": input_document["source_id"],
|
||||
"codec_epoch": input_document["codec_epoch"],
|
||||
"timestamp_basis": "session-time-seconds",
|
||||
"timeline_start_seconds": input_document["timeline_start_seconds"],
|
||||
"timeline_end_seconds": input_document["timeline_end_seconds"],
|
||||
"frames_processed": final_report["metrics"]["frames_processed"],
|
||||
"artifacts": artifacts,
|
||||
}
|
||||
_write_json(output_root / "result.json", result)
|
||||
partial_path.unlink()
|
||||
print(json.dumps({"result_id": result_id, "identity_sha256": identity_sha256}, sort_keys=True))
|
||||
return 0
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = _arguments()
|
||||
if args.command == "preflight":
|
||||
return _preflight(args)
|
||||
if args.command == "run":
|
||||
return _run(args)
|
||||
return _finalize(args)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,425 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Run a bounded, recorded-only segmentation probe on an external GPU worker."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import importlib.metadata
|
||||
import json
|
||||
import os
|
||||
import platform
|
||||
import shutil
|
||||
import statistics
|
||||
import time
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
MAX_INPUT_IMAGES = 16
|
||||
MAX_INPUT_BYTES = 32 * 1024 * 1024
|
||||
MAX_DIMENSION = 4096
|
||||
INSTANCE_SCORE_THRESHOLD = 0.5
|
||||
SEMANTIC_MODEL_ID = "microsoft/beit-base-finetuned-ade-640-640"
|
||||
SEMANTIC_REVISION = "a8b6f5ef4acb2ea55d882989deaa02d39401e2b2"
|
||||
RESULT_SCHEMA = "missioncore.recorded-segmentation-experiment/v1"
|
||||
|
||||
|
||||
def _arguments() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--input", type=Path, required=True)
|
||||
parser.add_argument("--output", type=Path, required=True)
|
||||
parser.add_argument("--cache", type=Path, required=True)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def _sha256(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as stream:
|
||||
for chunk in iter(lambda: stream.read(1024 * 1024), b""):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def _canonical_json(value: object) -> bytes:
|
||||
return json.dumps(
|
||||
value,
|
||||
ensure_ascii=False,
|
||||
sort_keys=True,
|
||||
separators=(",", ":"),
|
||||
allow_nan=False,
|
||||
).encode("utf-8")
|
||||
|
||||
|
||||
def _palette(index: int) -> tuple[int, int, int]:
|
||||
digest = hashlib.sha256(f"mission-core-segment-{index}".encode()).digest()
|
||||
return (64 + digest[0] % 176, 64 + digest[1] % 176, 64 + digest[2] % 176)
|
||||
|
||||
|
||||
def _blend(image: Any, colors: Any, alpha: float) -> Any:
|
||||
import numpy as np
|
||||
|
||||
return np.clip(
|
||||
image.astype(np.float32) * (1.0 - alpha) + colors.astype(np.float32) * alpha,
|
||||
0,
|
||||
255,
|
||||
).astype(np.uint8)
|
||||
|
||||
|
||||
def _semantic_overlay(
|
||||
image: Any, index_map: Any, labels: dict[int, str]
|
||||
) -> tuple[Any, list[dict[str, Any]]]:
|
||||
import numpy as np
|
||||
|
||||
colors = np.zeros_like(image)
|
||||
counts = np.bincount(index_map.reshape(-1), minlength=max(labels) + 1)
|
||||
present = np.flatnonzero(counts)
|
||||
for index in present:
|
||||
colors[index_map == index] = _palette(int(index))
|
||||
total = int(index_map.size)
|
||||
classes = [
|
||||
{
|
||||
"id": int(index),
|
||||
"label": labels.get(int(index), f"class-{int(index)}"),
|
||||
"pixels": int(counts[index]),
|
||||
"fraction": round(float(counts[index]) / total, 9),
|
||||
}
|
||||
for index in sorted(present, key=lambda value: int(counts[value]), reverse=True)
|
||||
]
|
||||
return _blend(image, colors, 0.46), classes
|
||||
|
||||
|
||||
def _instance_overlay(image: Any, instances: list[dict[str, Any]], masks: list[Any]) -> Any:
|
||||
import numpy as np
|
||||
|
||||
result = image.copy()
|
||||
for index, (instance, mask) in enumerate(zip(instances, masks, strict=True), start=1):
|
||||
color = np.asarray(_palette(500 + index), dtype=np.uint8)
|
||||
result[mask] = _blend(result[mask], np.broadcast_to(color, result[mask].shape), 0.56)
|
||||
x1, y1, x2, y2 = (int(round(value)) for value in instance["box_xyxy"])
|
||||
result[max(0, y1) : min(result.shape[0], y1 + 2), max(0, x1) : min(result.shape[1], x2)] = (
|
||||
color
|
||||
)
|
||||
result[max(0, y2 - 2) : min(result.shape[0], y2), max(0, x1) : min(result.shape[1], x2)] = (
|
||||
color
|
||||
)
|
||||
result[max(0, y1) : min(result.shape[0], y2), max(0, x1) : min(result.shape[1], x1 + 2)] = (
|
||||
color
|
||||
)
|
||||
result[max(0, y1) : min(result.shape[0], y2), max(0, x2 - 2) : min(result.shape[1], x2)] = (
|
||||
color
|
||||
)
|
||||
return result
|
||||
|
||||
|
||||
def _write_png(path: Path, array: Any) -> None:
|
||||
from PIL import Image
|
||||
|
||||
Image.fromarray(array).save(path, format="PNG", optimize=False)
|
||||
|
||||
|
||||
def _artifact(path: Path) -> dict[str, object]:
|
||||
return {"name": path.name, "bytes": path.stat().st_size, "sha256": _sha256(path)}
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = _arguments()
|
||||
input_root = args.input.expanduser().resolve(strict=True)
|
||||
output_root = args.output.expanduser().resolve()
|
||||
cache_root = args.cache.expanduser().resolve()
|
||||
if not input_root.is_dir() or output_root.exists():
|
||||
raise RuntimeError("input must be a directory and output must not exist")
|
||||
images = sorted(input_root.glob("camera-*.png"))
|
||||
if not images or len(images) > MAX_INPUT_IMAGES:
|
||||
raise RuntimeError("input image count is outside the recorded probe bound")
|
||||
for path in images:
|
||||
if not path.is_file() or path.stat().st_size > MAX_INPUT_BYTES:
|
||||
raise RuntimeError("input image is missing or too large")
|
||||
|
||||
import numpy as np
|
||||
import torch
|
||||
import torch.nn.functional as functional
|
||||
import torchvision
|
||||
import transformers
|
||||
from huggingface_hub import snapshot_download
|
||||
from PIL import Image
|
||||
from torchvision.models.detection import (
|
||||
MaskRCNN_ResNet50_FPN_V2_Weights,
|
||||
maskrcnn_resnet50_fpn_v2,
|
||||
)
|
||||
from transformers import AutoImageProcessor, BeitForSemanticSegmentation
|
||||
|
||||
if not torch.cuda.is_available():
|
||||
raise RuntimeError("CUDA is required for this worker experiment")
|
||||
device = torch.device("cuda:0")
|
||||
cache_root.mkdir(mode=0o700, parents=True, exist_ok=True)
|
||||
staging = output_root.with_name(f".{output_root.name}.incomplete")
|
||||
staging.mkdir(mode=0o700, parents=True, exist_ok=False)
|
||||
started = time.perf_counter()
|
||||
try:
|
||||
decoded: list[tuple[Path, Any]] = []
|
||||
width: int | None = None
|
||||
height: int | None = None
|
||||
for path in images:
|
||||
with Image.open(path) as opened:
|
||||
opened.verify()
|
||||
with Image.open(path) as opened:
|
||||
rgb = opened.convert("RGB")
|
||||
if max(rgb.size) > MAX_DIMENSION:
|
||||
raise RuntimeError("input image dimension is outside the probe bound")
|
||||
if width is None:
|
||||
width, height = rgb.size
|
||||
elif rgb.size != (width, height):
|
||||
raise RuntimeError("input images do not share one physical epoch resolution")
|
||||
decoded.append((path, np.asarray(rgb, dtype=np.uint8)))
|
||||
|
||||
instance_weights = MaskRCNN_ResNet50_FPN_V2_Weights.DEFAULT
|
||||
instance_model = maskrcnn_resnet50_fpn_v2(weights=instance_weights).to(device).eval()
|
||||
instance_labels = list(instance_weights.meta["categories"])
|
||||
instance_results: dict[str, dict[str, Any]] = {}
|
||||
instance_latencies: list[float] = []
|
||||
with torch.inference_mode():
|
||||
for path, image in decoded:
|
||||
tensor = instance_weights.transforms()(Image.fromarray(image)).to(device)
|
||||
torch.cuda.synchronize()
|
||||
before = time.perf_counter()
|
||||
prediction = instance_model([tensor])[0]
|
||||
torch.cuda.synchronize()
|
||||
latency_ms = (time.perf_counter() - before) * 1000.0
|
||||
instance_latencies.append(latency_ms)
|
||||
scores = prediction["scores"].detach().cpu().numpy()
|
||||
keep = np.flatnonzero(scores >= INSTANCE_SCORE_THRESHOLD)
|
||||
masks: list[Any] = []
|
||||
instances: list[dict[str, Any]] = []
|
||||
index_map = np.zeros(image.shape[:2], dtype=np.uint16)
|
||||
for output_index in keep:
|
||||
mask = prediction["masks"][output_index, 0].detach().cpu().numpy() >= 0.5
|
||||
if int(mask.sum()) < 8:
|
||||
continue
|
||||
label_id = int(prediction["labels"][output_index].item())
|
||||
instance_id = len(instances) + 1
|
||||
index_map[np.logical_and(mask, index_map == 0)] = instance_id
|
||||
masks.append(mask)
|
||||
instances.append(
|
||||
{
|
||||
"instance_id": instance_id,
|
||||
"class_id": label_id,
|
||||
"label": instance_labels[label_id],
|
||||
"score": round(float(scores[output_index]), 9),
|
||||
"box_xyxy": [
|
||||
round(float(value), 6)
|
||||
for value in prediction["boxes"][output_index]
|
||||
.detach()
|
||||
.cpu()
|
||||
.tolist()
|
||||
],
|
||||
"mask_pixels": int(mask.sum()),
|
||||
}
|
||||
)
|
||||
stem = path.stem
|
||||
_write_png(staging / f"{stem}.instances.png", index_map)
|
||||
_write_png(
|
||||
staging / f"{stem}.instance-overlay.png",
|
||||
_instance_overlay(image, instances, masks),
|
||||
)
|
||||
instance_results[stem] = {
|
||||
"latency_ms": round(latency_ms, 6),
|
||||
"instances": instances,
|
||||
}
|
||||
del instance_model
|
||||
torch.cuda.empty_cache()
|
||||
|
||||
semantic_snapshot = Path(
|
||||
snapshot_download(
|
||||
repo_id=SEMANTIC_MODEL_ID,
|
||||
revision=SEMANTIC_REVISION,
|
||||
cache_dir=cache_root / "huggingface",
|
||||
allow_patterns=(
|
||||
"config.json",
|
||||
"preprocessor_config.json",
|
||||
"pytorch_model.bin",
|
||||
),
|
||||
)
|
||||
)
|
||||
processor = AutoImageProcessor.from_pretrained(
|
||||
semantic_snapshot,
|
||||
local_files_only=True,
|
||||
use_fast=False,
|
||||
)
|
||||
semantic_model, semantic_loading = BeitForSemanticSegmentation.from_pretrained(
|
||||
semantic_snapshot,
|
||||
local_files_only=True,
|
||||
output_loading_info=True,
|
||||
)
|
||||
semantic_load_problems = {
|
||||
name: semantic_loading.get(name, [])
|
||||
for name in ("missing_keys", "unexpected_keys", "mismatched_keys", "error_msgs")
|
||||
if semantic_loading.get(name)
|
||||
}
|
||||
if semantic_load_problems:
|
||||
raise RuntimeError(
|
||||
"semantic checkpoint did not load exactly: "
|
||||
+ json.dumps(semantic_load_problems, sort_keys=True)
|
||||
)
|
||||
semantic_model = semantic_model.to(device).eval()
|
||||
semantic_labels = {
|
||||
int(key): str(value) for key, value in semantic_model.config.id2label.items()
|
||||
}
|
||||
semantic_results: dict[str, dict[str, Any]] = {}
|
||||
semantic_latencies: list[float] = []
|
||||
with torch.inference_mode():
|
||||
for path, image in decoded:
|
||||
inputs = processor(images=Image.fromarray(image), return_tensors="pt")
|
||||
inputs = {name: value.to(device) for name, value in inputs.items()}
|
||||
torch.cuda.synchronize()
|
||||
before = time.perf_counter()
|
||||
logits = semantic_model(**inputs).logits
|
||||
resized = functional.interpolate(
|
||||
logits,
|
||||
size=image.shape[:2],
|
||||
mode="bilinear",
|
||||
align_corners=False,
|
||||
)
|
||||
semantic = resized.argmax(dim=1)[0].detach().cpu().numpy().astype(np.uint8)
|
||||
torch.cuda.synchronize()
|
||||
latency_ms = (time.perf_counter() - before) * 1000.0
|
||||
semantic_latencies.append(latency_ms)
|
||||
overlay, classes = _semantic_overlay(image, semantic, semantic_labels)
|
||||
stem = path.stem
|
||||
_write_png(staging / f"{stem}.semantic.png", semantic)
|
||||
_write_png(staging / f"{stem}.semantic-overlay.png", overlay)
|
||||
semantic_results[stem] = {
|
||||
"latency_ms": round(latency_ms, 6),
|
||||
"classes": classes,
|
||||
}
|
||||
del semantic_model
|
||||
torch.cuda.empty_cache()
|
||||
|
||||
rows = []
|
||||
for path, image in decoded:
|
||||
stem = path.stem
|
||||
instance_overlay = np.asarray(Image.open(staging / f"{stem}.instance-overlay.png"))
|
||||
semantic_overlay = np.asarray(Image.open(staging / f"{stem}.semantic-overlay.png"))
|
||||
rows.append(np.concatenate((image, instance_overlay, semantic_overlay), axis=1))
|
||||
_write_png(staging / "segmentation-mosaic.png", np.concatenate(rows, axis=0))
|
||||
|
||||
checkpoint = Path(instance_weights.url).name
|
||||
checkpoint_path = Path(torch.hub.get_dir()) / "checkpoints" / checkpoint
|
||||
model_files = [
|
||||
{
|
||||
"name": f"beit/{path.name}",
|
||||
"bytes": path.stat().st_size,
|
||||
"sha256": _sha256(path),
|
||||
}
|
||||
for path in sorted(semantic_snapshot.iterdir())
|
||||
if path.is_file()
|
||||
]
|
||||
model_files.append(
|
||||
{
|
||||
"name": f"torchvision/{checkpoint}",
|
||||
"bytes": checkpoint_path.stat().st_size,
|
||||
"sha256": _sha256(checkpoint_path),
|
||||
}
|
||||
)
|
||||
inputs_manifest = [
|
||||
{"name": path.name, "bytes": path.stat().st_size, "sha256": _sha256(path)}
|
||||
for path in images
|
||||
]
|
||||
identity = {
|
||||
"inputs": inputs_manifest,
|
||||
"instance_model": "torchvision/maskrcnn_resnet50_fpn_v2/default",
|
||||
"instance_threshold": INSTANCE_SCORE_THRESHOLD,
|
||||
"semantic_model": SEMANTIC_MODEL_ID,
|
||||
"semantic_revision": SEMANTIC_REVISION,
|
||||
"model_files": model_files,
|
||||
}
|
||||
manifest: dict[str, Any] = {
|
||||
"schema_version": RESULT_SCHEMA,
|
||||
"created_at_utc": datetime.now(UTC)
|
||||
.isoformat(timespec="milliseconds")
|
||||
.replace("+00:00", "Z"),
|
||||
"classification": "private-derived-recorded-perception-experiment",
|
||||
"generation_sha256": hashlib.sha256(_canonical_json(identity)).hexdigest(),
|
||||
"input": inputs_manifest,
|
||||
"models": {
|
||||
"instance": {
|
||||
"id": "torchvision/maskrcnn_resnet50_fpn_v2",
|
||||
"weights": str(instance_weights),
|
||||
"weights_url": instance_weights.url,
|
||||
"score_threshold": INSTANCE_SCORE_THRESHOLD,
|
||||
"license": "BSD-3-Clause (TorchVision code/weight distribution)",
|
||||
},
|
||||
"semantic": {
|
||||
"id": SEMANTIC_MODEL_ID,
|
||||
"revision": SEMANTIC_REVISION,
|
||||
"dataset": "ADE20K scene_parse_150",
|
||||
"license": "Apache-2.0 (model-card metadata)",
|
||||
"checkpoint_load": "exact-no-missing-unexpected-or-mismatched-keys",
|
||||
},
|
||||
"files": model_files,
|
||||
},
|
||||
"runtime": {
|
||||
"base_image": os.environ.get("MISSION_CORE_BASE_IMAGE", "unknown"),
|
||||
"python": platform.python_version(),
|
||||
"torch": torch.__version__,
|
||||
"torchvision": torchvision.__version__,
|
||||
"transformers": transformers.__version__,
|
||||
"cuda_runtime": torch.version.cuda,
|
||||
"cudnn": torch.backends.cudnn.version(),
|
||||
"gpu": torch.cuda.get_device_name(0),
|
||||
"packages": sorted(
|
||||
(
|
||||
{
|
||||
"name": distribution.metadata["Name"],
|
||||
"version": distribution.version,
|
||||
}
|
||||
for distribution in importlib.metadata.distributions()
|
||||
if distribution.metadata["Name"]
|
||||
),
|
||||
key=lambda item: str(item["name"]).lower(),
|
||||
),
|
||||
},
|
||||
"metrics": {
|
||||
"frame_count": len(decoded),
|
||||
"instance_latency_ms": {
|
||||
"mean": round(statistics.fmean(instance_latencies), 6),
|
||||
"max": round(max(instance_latencies), 6),
|
||||
},
|
||||
"semantic_latency_ms": {
|
||||
"mean": round(statistics.fmean(semantic_latencies), 6),
|
||||
"max": round(max(semantic_latencies), 6),
|
||||
},
|
||||
"elapsed_seconds": round(time.perf_counter() - started, 6),
|
||||
},
|
||||
"frames": {
|
||||
stem: {
|
||||
"instance": instance_results[stem],
|
||||
"semantic": semantic_results[stem],
|
||||
}
|
||||
for stem in sorted(instance_results)
|
||||
},
|
||||
"acceptance": {
|
||||
"recorded_segmentation_artifact": "generated",
|
||||
"quality": "operator-review-required",
|
||||
"live": "not-tested",
|
||||
"safety": "not-accepted",
|
||||
},
|
||||
}
|
||||
artifacts = sorted(staging.glob("*.png"))
|
||||
manifest["outputs"] = [_artifact(path) for path in artifacts]
|
||||
(staging / "manifest.redacted.json").write_bytes(
|
||||
json.dumps(manifest, ensure_ascii=False, indent=2, sort_keys=True).encode("utf-8")
|
||||
+ b"\n"
|
||||
)
|
||||
os.rename(staging, output_root)
|
||||
except BaseException:
|
||||
shutil.rmtree(staging, ignore_errors=True)
|
||||
raise
|
||||
print(
|
||||
json.dumps({"output": output_root.name, "generation_sha256": manifest["generation_sha256"]})
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
Reference in New Issue
Block a user