# External perception stack review — 2026-07-20

## Objective

Select mature, reusable components for Mission Core's K1 camera/LiDAR pipeline
instead of growing the LAB implementation into a bespoke stack by trial and
error. The target outputs are reviewed 2D/dense semantic labels, classified 3D
boxes in the point cloud, distance to objects, and a path to recorded and live
execution.

## Decision

Keep Rerun as the synchronized viewer and experiment evidence surface. Do not
treat it as the perception engine. Use CVAT as the human-review and
ground-truth control plane. Reuse NVIDIA's production inference patterns for
LiDAR and multimodal execution, adapting only K1 data loading, timestamps, and
factory calibration. Use open model frameworks as benchmark sources, not as a
second production runtime.

The shortest credible route is:

1. Review LAB E2 drafts in CVAT and freeze a versioned ground-truth subset.
2. Benchmark corrected/masked K1 images on that same subset.
3. Replace heuristic 3D cube construction with a LiDAR-native PointPillars
   baseline.
4. Serve the selected 2D and 3D models through the existing Triton worker and
   log outputs to Rerun.
5. Gate camera/LiDAR BEVFusion on proven synchronization, calibration, and
   baseline metrics.

## Evidence from mature implementations

| Reference | Reusable practice | Mission Core use | Do not copy blindly |
| --- | --- | --- | --- |
| [Rerun](https://rerun.io/docs/overview/what-is-rerun) and its [examples](https://rerun.io/examples) | Time-aligned logging of images, point clouds, masks, 2D/3D boxes, trajectories, and model outputs | Viewer, debugging, playback, run evidence | It does not infer segmentation, depth, or boxes |
| [OctoSense](https://github.com/anthonytec2/OctoSense) | Camera rectification before inference; RGB/LiDAR timestamp alignment; semantic labeling; dynamic-object masks; calibrated LiDAR projection; dataset packaging | Copy the pipeline shape and QA gates | It is primarily a dataset-generation recipe, not a drop-in real-time Mission Core runtime |
| [DeepStream LiDAR 3D inference](https://docs.nvidia.com/metropolis/deepstream/7.1/text/DS_3D_Lidar_Inference.html) | PointPillars preprocessing, inference, postprocessing, and 3D box visualization in a production pipeline | First classified 3D-box baseline | Its sample data loader and calibration assumptions must be replaced with K1 adapters |
| [DeepStream multimodal fusion](https://docs.nvidia.com/metropolis/deepstream/8.0/text/DS_3D_MultiModal_Lidar_Sensor_Fusion.html) | Camera/LiDAR preprocessing and BEVFusion deployment | Later accuracy/robustness stage after LiDAR-only baseline | Too expensive as the first 3D fix; requires trustworthy synchronization and extrinsics |
| [MMDetection3D](https://github.com/open-mmlab/mmdetection3d) and [OpenPCDet](https://github.com/open-mmlab/OpenPCDet) | Reproducible 3D model/config baselines and pretrained checkpoints | Offline comparison and checkpoint selection | Do not introduce a parallel permanent serving stack beside Triton |
| [TAO auto-label](https://docs.nvidia.com/tao/tao-toolkit/text/data_services/auto-label.html) | Assisted labeling and model-driven annotation | Candidate accelerator after the first reviewed GT set exists | Auto-label output is still draft data, not accuracy evidence |
| [FiftyOne annotation integration](https://docs.voxel51.com/integrations/annotation.html) | Dataset curation, slice analysis, and CVAT round trips | Add when error analysis across many sessions becomes the bottleneck | Not needed to complete the first GT gate |
| [Isaac ROS nvblox](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_nvblox) | GPU mapping, reconstruction, and cost-map production | Later navigation/mapping consumer | It does not replace object detection or classified 3D boxes |
| [Isaac Sim camera sensors](https://docs.isaacsim.omniverse.nvidia.com/latest/sensors/isaacsim_sensors_camera.html) | Fisheye/OpenCV camera models and calibration-aware simulation | Synthetic regression cases after real-data metrics exist | Simulation is not a substitute for validating K1 factory calibration on recorded data |

## Selected stack boundaries

| Layer | Selected role | Current state | Gate before expansion |
| --- | --- | --- | --- |
| K1 ingestion | Preserve native timestamps, camera identity, point cloud, and vendor calibration | Recording works | Validate camera identity and projection on representative frames |
| Image preprocessing | Stable optical-circle mask plus calibration-derived correction profile | LAB E2 mask/draft path exists | Freeze profile ID and compare corrected versus raw on identical frames |
| 2D/dense perception | Generate prelabels, then evaluate against reviewed CVAT exports | LAB E2 drafts imported | Human two-pass review and class-level metrics |
| 3D detection | LiDAR-native PointPillars baseline | Next model integration | K1 point format adapter, coordinate convention test, reviewed 3D subset |
| Multimodal fusion | BEVFusion/DeepStream candidate | Deferred | Baselines, sync tolerance, extrinsic validation, GPU budget |
| Serving | Existing Triton worker | Running | Pin model/config artifacts and record per-stage latency/resource metrics |
| Visualization | Rerun | Running | Log reviewed labels and model outputs separately |
| Annotation/GT | CVAT `v2.70.0` | Deployed on D; LAB E2 tasks imported | Two-pass review; export with immutable manifest and hashes |

## Configuration practices to adopt

- Rectify or calibration-map the fisheye image before model inference when the
  model expects a pinhole-like domain. Keep the raw frame alongside it.
- Apply a cached optical-circle mask/profile by camera identity; do not
  recompute the mask for every frame.
- Synchronize camera frames to LiDAR timestamps explicitly and record the
  chosen tolerance and dropped/unmatched frames.
- Keep 2D detections, dense semantics, LiDAR-native 3D detections, and fused
  results as separate evidence streams. Fusion must not erase provenance.
- Project 3D boxes into the camera for validation, but do not construct final
  3D boxes by simply extruding every 2D detection. That heuristic produces the
  overlapping-cube failure observed in LAB E1.
- Record model identity, checkpoint hash, preprocessing profile, precision,
  batch size, GPU memory, stage latency, wall time, and output hashes for every
  material experiment.
- Treat every model-produced label as `ground_truth=false` until reviewed and
  exported through the ground-truth gate.

## Next experiment gates

### LAB E2 review gate

- Review both imported tasks across the same 64 frames.
- First pass: correct labels, masks, and missed/false objects.
- Second pass: independent consistency review.
- Export reviewed instance and semantic annotations with manifest hashes.

### LAB E3 image baseline

- Compare raw-circle-mask, rectified, and correction-profile inputs on the
  frozen reviewed subset.
- Select by class metrics and temporal stability, not by visual preference.
- Measure preprocessing, inference, postprocessing, and end-to-end time
  separately.

### LAB E4 3D baseline

- Integrate a PointPillars-compatible K1 point loader.
- Lock coordinate axes, units, timestamps, and calibration version in the run
  manifest.
- Evaluate classified 3D boxes on a small reviewed subset before any BEVFusion
  work.

## Rejected shortcuts

- Replacing Rerun with OctoSense: wrong boundary; OctoSense supplies useful
  pipeline patterns while Rerun remains the viewer.
- Moving the full stack to Omniverse: useful for simulation and synthetic data,
  not the shortest path to recorded K1 accuracy.
- Treating current semantic/instance drafts as GT: invalidates all subsequent
  model comparisons.
- Starting with BEVFusion: adds synchronization, calibration, conversion, and
  runtime risks before the simpler LiDAR-native baseline is measured.
