# LiDAR worker: product value, evidence boundary and implementation roadmap

Date: 2026-07-25
Status: accepted architecture plan; L0/L1 implemented; L2 diagnostic A/B complete;
Dataset Gateway first-frame Current/Patchwork++ A/B complete
Scope: real scanner records, replay and future live shadow processing
Explicitly out of scope: Unreal U0/U1, Gaussian assets and simulator rendering

## 1. Decision

Mission Core does not treat “LiDAR processing” as one model. It owns four
separate products:

1. scanner evidence and quality;
2. geometric preprocessing and map quality;
3. object/occupancy perception;
4. planner-facing local world state.

Simulation can exercise their contracts and generate synthetic regressions, but
it cannot prove real scanner timing, returns, reflectance, calibration, motion
distortion or environmental failure modes.

The AI worker remains external, Linux/NVIDIA-capable and replaceable. React is
the control and review surface. Mission Core owns immutable inputs, exact
profiles, result identities, acceptance gates and diagnostic-only authority.
Models do not gain scanner, navigation, command or safety authority.

## 2. What the current K1 source actually contains

The firmware-3 `lio_pcl` stream currently exposes:

- finite XYZ points already expressed in the canonical `map` frame;
- one verified uint8 intensity value in the low byte of `rgbi`;
- a frame header with sequence, stamp and scaler;
- a separate `T_map_from_lidar` pose stream.

The recorded XYZ values are metric. With the independent pose they can be
expressed relative to the reported LiDAR pose and produce useful local
distance diagnostics. What is not proven by that inverse transform is that the
result is the original unregistered sweep with preserved beam origin,
acquisition order and motion timing.

Static, read-only analysis of the K1 3.0.2 deployment artifacts additionally
shows that the appliance internally:

- selects a Livox MID-360 adapter;
- consumes `/livox/lidar` and `/imu`;
- carries XYZ, intensity, timestamp and ring in its internal point type;
- supports raw-packet and point/IMU callbacks in its MID-360 library;
- publishes the external cloud after the LIO/modeling path;
- configures the external MQTT point-cloud sample factor to four.

This proves a richer onboard data path exists. It does not prove that the
current external MQTT contract preserves those fields, and private firmware
artifacts remain outside Git. The redacted evidence is recorded in
`docs/lab/005_K1_FW302_LIDAR_PIPELINE_20260725.redacted.md`.

The external contract does not currently expose an admitted:

- raw sensor-frame sweep;
- per-point firing time;
- ring/channel number;
- IMU sample stream;
- LiDAR/IMU extrinsic;
- proven common hardware clock for LiDAR and cameras;
- LiDAR scan model required by projective integrations.

The accepted `missioncore.e10-lidar-replay-pack/v1` narrows this further: it
keeps XYZ, pose and best-effort camera binding, but drops intensity. It must not
be silently mutated because E10–E26 results are content-bound to that schema.

These facts have architectural consequences:

- current K1 points can support display, calibrated projection, persistent
  support and bounded geometric analysis;
- the live stream can be converted back to a pose-relative sensor XYZI tensor;
- the v1 replay pack cannot feed the admitted NVIDIA PointPillars baseline;
- KISS-ICP, KISS-SLAM, FAST-LIO2, LIO-SAM or GLIM cannot honestly rebuild K1
  odometry from points that are already vendor-mapped;
- deskew and LiDAR-inertial SLAM are blocked until the scanner or a future
  vehicle LiDAR driver supplies raw scans, timing and IMU evidence.

The executable truth is
`missioncore.lidar-evidence-profile/v1` in
`src/k1link/compute/lidar_contract.py`. It assesses each stage as `ready`,
`degraded` or `blocked` and refuses to infer absent sensor fields.

`missioncore.lidar-replay-pack/v2` is now the admitted field-retaining replay
input. It preserves raw integer XYZ, derived map XYZ, raw uint32 `rgbi`, its
verified intensity low byte, source/payload identity, header seq/stamp/scaler,
exact host epoch/monotonic receive times and the independent pose stream. Pack
v1 remains immutable for E10–E26.

## 3. Current system result

The existing camera-heavy E10–E26 line is valuable and remains in place:

- raw recording and content-addressed worker handoff;
- camera detection and semantic segmentation;
- factory KB4 camera/LiDAR projection;
- map-frame LiDAR support and ground-aware cuboids;
- bounded temporal stabilization;
- persistent support motion evidence;
- camera ego-motion evidence and conservative fusion.

E25 proved the LiDAR limit: missing current returns cannot be recovered by
threshold tuning. E26 passed its reviewed 11/11 diagnostic windows by adding
camera parallax, but it is not planner-ready: most camera observations remain
unknown, camera-only velocity is not metric, 374 conflicts remain, and the
benchmark is not independent ground truth.

The next work must therefore improve the LiDAR-native input and evaluation
surface, not add another visual smoothing pass. Mission Core will use
independently labeled public datasets before requesting any new manual K1
annotation. The current K1 evidence stays an unlabeled out-of-domain smoke test.

## 4. Market and stack assessment

### 4.1 NVIDIA components worth retaining

| Component | Correct use in Mission Core | Decision |
| --- | --- | --- |
| [TAO PointPillars](https://docs.nvidia.com/tao/tao-toolkit/latest/text/cv_finetuning/pytorch/point_cloud/pointpillars.html) | Train/evaluate a LiDAR-native 3D detector over sensor-frame XYZI | First neural 3D baseline after replay v2 and labels |
| [DeepStream LiDAR 3D inference](https://docs.nvidia.com/metropolis/deepstream/7.1/text/DS_3D_Lidar_Inference.html) | Reference production pipeline for XYZI → Triton/TensorRT → 3D boxes | Reuse the inference pattern, not its file loader or UI |
| [TAO Deploy PointPillars](https://docs.nvidia.com/tao/tao-toolkit/latest/text/tao_deploy/pointpillars.html) | Build a pinned FP16/FP32 TensorRT engine and evaluate it | Worker optimization only after an accuracy baseline |
| [Isaac ROS nvblox](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_nvblox/isaac_ros_nvblox/index.html) | GPU local TSDF/occupancy/ESDF and Nav2 cost-map producer | Later; blocked on admitted scan geometry and better timing |
| DeepStream multimodal 3D fusion | Later camera/LiDAR BEV baseline | Deferred until LiDAR-only, sync and calibration gates pass |

NVIDIA does not provide a magic “clean the K1 map” stage. PointPillars produces
classified 3D objects. Nvblox produces reconstruction and distance fields. The
quality of both remains bounded by source timing, calibration, scan geometry
and training-domain fit.

### 4.2 Independent components worth benchmarking

| Component | Correct use | Decision |
| --- | --- | --- |
| [Patchwork++](https://github.com/url-kaist/patchwork-plusplus) | Fast adaptive ground segmentation, including reflection-noise handling | v1.4.1 benchmarked; do not promote on the current vendor-map feed |
| [Autoware CenterPoint](https://github.com/autowarefoundation/autoware_universe/tree/main/perception/autoware_lidar_centerpoint) | Mature ROS 2/TensorRT 3D detection and multi-frame reference | Second detector baseline after PointPillars |
| [MMDetection3D](https://github.com/open-mmlab/mmdetection3d) | Training/evaluation harness and dataset adapters | Laboratory only |
| [OpenPCDet](https://github.com/open-mmlab/OpenPCDet) | Alternative LiDAR detector benchmark/model zoo | Laboratory only; not the production runtime |
| [KISS-ICP](https://github.com/PRBonn/kiss-icp) | Simple LiDAR-only odometry baseline | Future raw sensor scans only |
| [KISS-SLAM](https://github.com/PRBonn/kiss-slam) | Global LiDAR SLAM/loop-closure baseline | Future raw sensor scans only |
| [FAST-LIO2](https://github.com/hku-mars/FAST_LIO) | Raw LiDAR + IMU odometry/mapping | Future vehicle sensor profile only |
| [LIO-SAM](https://github.com/TixiaoShan/LIO-SAM) | Deskewed LiDAR-inertial factor-graph reference | Future profile with ring/time/IMU only |
| [GLIM](https://github.com/koide3/glim) | GPU-accelerated range-inertial mapping, loop correction and map cleanup | Strong future mapping candidate; blocked for current K1 evidence |

Autoware is a useful architecture and component reference; importing the whole
autonomous-driving distribution into the worker would add a large operational
surface that Mission Core does not currently need.

## 5. Target worker shape

The worker exposes provider-neutral jobs rather than one growing camera script:

| Job profile | Input | Output | Authority |
| --- | --- | --- | --- |
| `lidar-quality/v1` | immutable LiDAR evidence | field/timing/density/intensity report | diagnostic |
| `lidar-ground/v1` | sensor-frame XYZI + pose | ground/non-ground points and metrics | diagnostic |
| `lidar-3d-detection/v1` | sensor-frame XYZI | classified 3D observations + uncertainty | shadow |
| `lidar-local-map/v1` | scans + synchronized pose | occupancy/TSDF/ESDF artifacts | shadow |
| `lidar-mapping-benchmark/v1` | raw scans + optional IMU | trajectory/map comparison | offline diagnostic |

Every result is content-addressed and binds:

- source evidence identity;
- exact evidence profile/readiness document;
- model, weights, runtime and preprocessing identities;
- coordinate transforms and timestamp basis;
- resource/queue/drop telemetry;
- acceptance policy and explicit authority.

Offline replay may batch work for throughput. Live shadow uses bounded
latest-wins queues. Accuracy is frozen before FP16/INT8, CUDA graph, pinned
memory, zero-copy or batching optimizations are accepted.

## 6. Implementation sequence

### L0 — evidence truth and detector adapter — complete

- [x] Add `missioncore.lidar-evidence-profile/v1`.
- [x] Encode current live K1 and replay-pack-v1 facts.
- [x] Produce deterministic stage readiness and blockers.
- [x] Add tested `map + T_map_from_lidar + intensity → sensor XYZI`.
- [x] Add bounded live point-count, cadence and intensity telemetry to the
  existing external worker report.
- [x] Keep authority diagnostic-only.

### L1 — lossless replay v2 and scanner quality — complete

- [x] Create a new replay schema; do not rewrite pack v1.
- [x] Preserve XYZ, intensity, source sequence/header stamp/scaler and exact
  host capture/receive times.
- [x] Record explicitly absent ring, per-point time and IMU fields.
- [x] Add immutable reports for sequence gaps, arrival jitter, points/frame,
  intensity distribution and pose coverage. The sealed source report keeps
  native sensor-frame range explicitly unavailable because the source is a
  vendor map increment; later diagnostics may separately report pose-derived
  local distances without relabeling them as raw beam ranges.
- [x] Compare source native capture vs persisted replay fields byte-for-byte on
  a frozen real slice.
- [x] Publish the report to the React observation view without bundling a
  desktop renderer.

Exit: a recording cannot be called detector-ready when required fields were
dropped, and an operator can distinguish scanner dropout from model failure.

Accepted real slice:

- session `20260719T220917Z_viewer_live`;
- 66 point frames, 226,963 points and 70 pose frames;
- 24 arrays compared, zero mismatches, identical logical content SHA-256;
- pose coverage 100% at a 100 ms host-arrival gate;
- 3,439 mean points/frame;
- point-frame interval p50 80.4 ms, p95 186.6 ms and maximum 1,153.4 ms;
- intensity p50 248 and p95 255 on the verified `[0, 255]` low byte.

The long interval tail is evidence to investigate in the scanner/transport
quality line. It is not repaired or hidden by replay.

### L2 — geometric baseline — diagnostic A/B complete, accuracy gate open

- [x] Pin and run official Patchwork++ v1.4.1 at source commit
  `3e6903a1d5537a4cc2ace897b0bbb98a92d6014c`.
- [x] Compare it against a full-frame, point-aligned extension of the current
  E19 local-percentile ground proposal.
- [x] Retain separate current/candidate ground and assigned masks for every
  source point; raw replay remains unchanged.
- [x] Add a bounded point-aligned frame API and browser 3D review for
  intensity, current ground, candidate ground and disagreement.
- [x] Separate the 66-frame indoor contract slice from a named RAVNOVES00
  field review with five camera-bound urban windows and accumulated map clouds.
- [x] Record the K1 3.0.2 internal MID-360/LIO field path from static,
  redacted firmware evidence without changing device state.
- [x] Bind physical height, applied vertical-origin offset and evidence class
  into every new benchmark identity.
- [x] Seal latency, ground-fraction, algorithm-IoU and disagreement
  distributions in `missioncore.lidar-ground-benchmark/v1`.
- [x] Create an immutable eight-frame annotation template in which every point
  starts as `ignore-unreviewed`; it is explicitly not ground truth.
- [ ] Keep the K1 annotation template frozen as an optional later
  domain-adaptation asset; do not make manual review the current critical path.
- [ ] Measure accepted ground IoU and obstacle recall first against an admitted
  GOOSE native scan and its published point-wise labels.

The real diagnostic run is
`ground-benchmark-68cfd7a8f1dd4c0006183bb4f63a23f9ff1dd7459317ff0886e995f6c320d984`.
It covers all 66 frames and 226,963 points. The current local-percentile
proposal classified 18.31% ground at p50 with 9.74 ms p95 host latency.
Patchwork++ classified only 0.53% ground at p50 with 0.26 ms p95 host latency.
Their point-aligned ground IoU was 2.90% p50 and disagreement was 17.92% p50.
These last two values compare algorithms; they are not accuracy metrics.

An additional operator-height diagnostic,
`ground-benchmark-87cb3150701f7e21756f46ef5b6ce110df1e07720ef6dc2303c95520932cf43f`,
binds the measured 1.27 m handheld height and applies the same explicit map-Z
offset. Patchwork++ ground fraction rises from 0.53% to 2.47% p50; its p95
latency remains 0.42 ms. Algorithm IoU is 4.07% p50 and disagreement is 19.13%
p50. The run is useful for visual review, but its evidence class is
`operator-estimated`, so input acceptance and production promotion stay false.

The operator-facing field generation is
`lidar-field-review-57f359dae336f06962e3a29e69e2da1bb8365f9af46d5d3173609f92d43db1ef`.
It uses the immutable RAVNOVES00 E10 derivative
`e10-lidar-pack-5da0396d32a27f9d1ca537cc2e8a371d386078d6f0dc71737b78620992af9625`
over `135.365857292–195.334857292` session seconds. The source has 526
available LiDAR samples and 1,182,292 points. Five central urban windows cover
roads, facades, sidewalks, parked vehicles and vegetation. Each window:

- retains the exact source interval and camera preview identity;
- computes both ground masks independently on every original LiDAR sample;
- accumulates those point-aligned results only afterwards in the map frame;
- reports 53–68 source samples and 98,538–172,027 source points;
- publishes a deterministic uniform display sample of at most 80,000 points.

This field review fixes the operator-context problem of the indoor technical
slice; it does not upgrade evidence. The E10 derivative remains intensity-free,
vendor-mapped and without independent ground labels. Its contract is therefore
`missioncore.lidar-field-review/v1`, `status=diagnostic-only`,
`decision.status=visual-review-only` and `production_promotion=false`.

The result is an evidence-backed **do-not-promote** decision for the current K1
feed. Patchwork++ is fast, but its input model assumes a sensor-centric scan
and physical sensor height. K1 `lio_pcl` is a vendor-mapped increment, its
physical sensor height is not encoded by the best-effort pose, and raw scan
geometry is not exported. The measured 1.27 m estimate can be reproduced as a
named diagnostic profile; tuning that offset until Patchwork++ merely looks
plausible would still invalidate the comparison.

The K1-specific independent-label exit remains open, but it is no longer the
next step. The Dataset Gateway provides a labeled public-domain evaluation
path. K1 manual review or a new real vehicle dataset is reserved for later
domain adaptation after a public baseline proves that the pipeline and metric
harness work.

### L2.5 — Dataset Gateway — GOOSE qualification and RELLIS S0 complete

- [x] Define separate `native-scan`, `normalized-scan` and
  `rolling-local-map` representations.
- [x] Add a lossless GOOSE/SemanticKITTI XYZI + packed semantic/instance label
  reader with point-alignment and finite-value gates.
- [x] Publish the read-only Dataset Gateway contract in React.
- [x] Refuse to relabel current `lio_pcl` as a native scan or to invent missing
  per-point time and line/ring fields.
- [x] Require operator-admitted storage under
  `D:\NDC_MISSIONCORE\datasets`.
- [x] Configure the worker dataset root and record disk/resource baseline.
- [x] Download only the 3.3 GB GOOSE validation archive first and record its
  hash/license/provenance.
- [x] Show one real labeled revolution in React with native remission and
  ground-truth superclass coloring.
- [x] Admit the published Patchwork-specific axes/height profile without
  claiming full vehicle TF, deskew or a general normalized scan.
- [x] Run current ground and Patchwork++ against the same GOOSE native scan and
  ground truth; keep the result one-frame diagnostic.
- [x] Expand Current/Patchwork++ qualification to all 961 validation frames
  through an immutable R1 Polygon replay-shadow run.
- [x] Add named range/FOV/density/noise/dropout degradation profiles without
  overwriting the native frame.
- [x] Upgrade the gateway to two independently identified sources instead of
  treating GOOSE as a universal dataset contract.
- [x] Pin and verify the official RELLIS-3D Ouster example, preserve its
  `131,072` point/label alignment and publish a bounded semantic viewer.
- [x] Version the RELLIS `ground / non-ground / ignore` mapping and expose
  licensing as research-only evidence.
- [ ] Admit the full RELLIS Ouster SemanticKITTI scans, labels and poses on
  worker D, then run the same Current/Patchwork++ harness.

The architectural contract and run sequence are fixed in
`docs/14_LIDAR_DATASET_GATEWAY.md` and ADR 0021. GOOSE is first because its
published 3D format is one LiDAR revolution with point-wise semantic and
instance labels in off-road environments. The RELLIS-3D compatibility smoke is
now complete against official frame `000104`; the full second-source algorithm
cross-check remains the next gate.

### L3 — LiDAR-native 3D detection

- [ ] Establish the public-dataset baseline first; freeze K1-specific 3D
  annotations only when a measured domain gap justifies them.
- [ ] Run NVIDIA PointPillars through the existing external worker/Triton seam.
- [ ] Treat pretrained output as a baseline, not an accepted product model.
- [ ] Measure class precision/recall, center/range/yaw error, distance-bucket
  recall, false occupied objects and end-to-end latency.
- [ ] Compare Autoware CenterPoint only after the PointPillars harness is stable.
- [ ] Fine-tune only if the baseline demonstrates useful transfer and the
  annotation budget is justified.

Exit: the selected detector beats the camera-derived cuboid baseline on the
independent gate without increasing unsafe false-free or false-dynamic output.

### L4 — live shadow integration

- [ ] Add a bounded LiDAR queue independent of camera cadence.
- [ ] Run the accepted detector profile on the NVIDIA worker.
- [ ] Fuse LiDAR-native objects with E26 camera evidence as independent sources.
- [ ] Publish `agree`, `single-source`, `conflict` and `unknown`; unknown remains
  occupied.
- [ ] Measure sensor-to-result latency, deadline misses, drops, memory and GPU
  headroom on a physical run.

Exit: repeatable shadow telemetry only. Navigation and safety acceptance remain
false.

### L5 — local occupancy and Nav2

- [ ] Obtain and validate the K1/future vehicle LiDAR scan model, or use a
  different admitted source that supplies it.
- [ ] Prove pose and time behavior required by nvblox.
- [ ] Benchmark static occupancy/TSDF and ESDF output on real replay.
- [ ] Keep dynamic observations in a separate decaying layer.
- [ ] Connect the accepted 2D slice to Nav2 through the existing world-state
  boundary.

Exit: local collision-space quality and deadline gates pass in replay and
shadow. This still does not authorize control.

### L6 — alternative odometry/mapping

This stage starts only when a source supplies unregistered sensor scans. Add
per-point time and IMU/extrinsic requirements before FAST-LIO2, LIO-SAM or GLIM.
Benchmark KISS-ICP first, then a global SLAM candidate using ATE/RPE, loop
closure residual, wall/surface thickness, map entropy, repeat-pass alignment
and compute cost.

The current K1 `lio_pcl` stream cannot satisfy this gate.

## 7. Product value

The near-term value is not a prettier point cloud:

- a trustworthy observation tells the operator whether the scanner, transport,
  pose, calibration or model failed;
- lossless replay makes model and worker upgrades repeatable;
- LiDAR-native objects reduce dependence on camera visibility and provide
  metric geometry;
- ground/non-ground and local occupancy become the bridge from archive review
  to route validation and later collision checking;
- the same job/result contracts accept real, replayed or simulated sources
  without moving heavy compute into React;
- hardware selection becomes evidence-driven: a future vehicle LiDAR is
  accepted by its timing/fields/profile, not by vendor marketing.

The GOOSE validation-split and deterministic degradation gates are complete:
pinned Patchwork++ improves micro Ground IoU from `47.42%` to `66.40%`, raises
natural-ground recall from `51.76%` to `76.22%` and stays below `23.41 ms` p95
across 961 frames. RELLIS S0 proves that the shared reader, axes, ontology and
ground-target mapping work on a different Ouster off-road domain; it does not
yet score an algorithm. The decision remains shadow-only. The highest-value
immediate work is full RELLIS ground qualification, followed by the real
sensor input contract and LiDAR-native detection on the same gateway. Nvblox
and alternative SLAM remain later because their timing, pose and scan-geometry
gates are not yet satisfied.
