docs(perception): record PointPillars transfer verdict
This commit is contained in:
DCCONSTRUCTIONS
@@ -620,21 +620,96 @@ Exit: one immutable K1 session yields both a persistent reconstruction and a
|
||||
bounded local world state without hard-coded terrain height or scanner-side
|
||||
changes.
|
||||
|
||||
### L3 — LiDAR-native 3D detection — deferred behind L2.6
|
||||
### L3 — LiDAR-native 3D detection — active after L2.6
|
||||
|
||||
- [ ] Establish the public-dataset baseline first; freeze K1-specific 3D
|
||||
annotations only when a measured domain gap justifies them.
|
||||
- [ ] Run NVIDIA PointPillars through the existing external worker/Triton seam.
|
||||
- [ ] Treat pretrained output as a baseline, not an accepted product model.
|
||||
- [ ] Measure class precision/recall, center/range/yaw error, distance-bucket
|
||||
recall, false occupied objects and end-to-end latency.
|
||||
- [x] Freeze a fail-closed benchmark admission that distinguishes independent
|
||||
oriented 3D box truth from point-wise semantic/instance labels.
|
||||
- [x] Extract and seal the contract from the exact NVIDIA ONNX instead of
|
||||
copying a generic KITTI example range into the product profile.
|
||||
- [x] Build the target TensorRT engine on Worker 006, install it into the
|
||||
existing canonical Triton repository and execute both GOOSE and KITTI schema
|
||||
smokes without creating or restarting a second serving stack.
|
||||
- [x] Admit all KITTI 3D Object Detection 2017 archives and the pinned complete
|
||||
`3,712 / 3,769` train/validation split on Worker 006.
|
||||
- [x] Measure the complete public cross-domain transfer probe through the
|
||||
canonical worker seam: `3,769/3,769` frames, fixed postprocess and no
|
||||
validation retuning.
|
||||
- [x] Reject the exact `deployable_v1.1` checkpoint as a K1 product candidate:
|
||||
runtime is healthy, while BEV mAP40 is `2.46322%`, 3D mAP40 is
|
||||
`0.000216745%` and false occupied rate is `99.9981%`.
|
||||
- [x] Preserve the claim boundary: the NGC model card's proprietary
|
||||
solid-state-LiDAR result is not independently reproducible, KITTI is
|
||||
cross-domain, and `RAVNOVES01` without independent cuboids cannot establish
|
||||
detector accuracy.
|
||||
- [ ] Freeze a separate training/adaptation admission before downloading a
|
||||
trainable checkpoint or starting a training runtime.
|
||||
- [ ] Train only on the frozen `3,712` KITTI train frames; keep all `3,769`
|
||||
validation frames untouched until the candidate ONNX is sealed.
|
||||
- [ ] Rebuild and evaluate the adapted engine through the already proven
|
||||
Worker 006/Triton/postprocess seam.
|
||||
- [ ] After a useful public box-truth result, run `RAVNOVES01` as a K1
|
||||
deterministic-replay, schema, latency, queue/drop and visual-plausibility
|
||||
transfer. Do not call it a K1 accuracy benchmark.
|
||||
- [ ] Compare Autoware CenterPoint only after the PointPillars harness is stable.
|
||||
- [ ] Fine-tune only if the baseline demonstrates useful transfer and the
|
||||
annotation budget is justified.
|
||||
|
||||
Exit: the selected detector beats the camera-derived cuboid baseline on the
|
||||
independent gate without increasing unsafe false-free or false-dynamic output.
|
||||
|
||||
The current immutable L3 admission result is
|
||||
`l3-pointpillars-admission-857a8c6a958850f82db7a58e894e13e2a54add09379ca58fb8e47db1e6aba65c`.
|
||||
It freezes NGC `nvidia/tao/pointpillarnet:deployable_v1.1` as ONNX digest
|
||||
`2dcabddc3a365e9608a112d7bbbb7db769a6dddeeaa59aa03611a83113326da1`.
|
||||
The signed companion label file has digest
|
||||
`0adaeb5a374421b61bf83b8fa4522e11abd68461f239a4c72cf5627de913b3da`
|
||||
and names `Vehicle`, `Pedestrian`, `Cyclist`. The evaluation profile explicitly
|
||||
maps only `Vehicle → Car`; the other two class names are identity mappings.
|
||||
The decoded row contract and class-agnostic BEV NMS are pinned to NVIDIA's
|
||||
`tao_toolkit_recipes` commit
|
||||
`a540badc47812a17a94e924b537d49ad3969b5a8`: `4,096` pre-NMS candidates and
|
||||
IoU threshold `0.01`. The ONNX itself embeds score threshold `0.1` and the
|
||||
voxel range `[-51.2, -51.2, -1.4]` to `[51.2, 51.2, 4.4]`; the sealed graph
|
||||
contract has digest
|
||||
`2fd29cd054ab058c2cfec3dfba305c71e123ef3f04b457d0c64de0c8dac2e1be`.
|
||||
Postprocessing executes on Worker 006; the raw
|
||||
`[1, 393216, 9]` tensor is never a Mac/browser transport contract.
|
||||
The public metric is explicitly a cross-domain transfer probe, not native
|
||||
checkpoint accuracy. It is a frozen local 40-point AP gate without KITTI
|
||||
difficulty filtering or test-server submission: IoU is `0.7` for `Car` and
|
||||
`0.5` for `Pedestrian/Cyclist`, with distance recall buckets
|
||||
`0–20 / 20–40 / 40–70 m`.
|
||||
Worker 006 built a TensorRT 11.0 engine for its RTX 4090 with digest
|
||||
`12005d972a4632d56342a5da44442b632c1dcc5144fa3c70b162dec334532481`.
|
||||
The build completed in `9.139 s`, used approximately `192 MiB` peak builder
|
||||
GPU memory and stayed inside the existing `ndc-mission-core-triton` container.
|
||||
The model was installed into the existing repository and loaded through
|
||||
explicit model control without restarting Triton.
|
||||
|
||||
One GOOSE native XYZI frame with `169,883` points then passed a single-query
|
||||
representation smoke at `55.2069 ms` GPU compute time. This proves only that
|
||||
the admitted sensor-frame XYZI shape can execute through the staged engine.
|
||||
GOOSE and RELLIS provide independent point-wise semantic/instance truth, not
|
||||
oriented 3D cuboids, so they cannot produce PointPillars 3D mAP, center or yaw
|
||||
accuracy. KITTI 3D Object Detection 2017 is the independent public
|
||||
cross-domain box-truth source under `CC-BY-NC-SA-3.0`, with the OpenPCDet split
|
||||
at commit
|
||||
`233f849829b6ac19afb8af8837a0246890908755`. Its admission is archive-only and
|
||||
fail-closed: three official archive sizes and digests, XYZI packing, all
|
||||
training/test calibrations, all training label rows, and the disjoint complete
|
||||
`3,712 / 3,769` train/validation partition are verified before a path-free
|
||||
state may be published. The source archives stay on Worker 006 and are not
|
||||
silently copied to an operator host.
|
||||
|
||||
The complete immutable transfer probe is
|
||||
`l3-pointpillars-kitti-1a6b499e194a363644854dc324bd1b565c100b809f145c1324a25328e7ae0910`.
|
||||
It processed `3,769/3,769` frames sequentially. Mean inference was
|
||||
`31.1366 ms`, p95 was `45.9418 ms` and maximum was `68.8731 ms`. Of `315,589`
|
||||
post-NMS boxes, `159,263` were inside the shared model/KITTI evaluation volume;
|
||||
the remaining `156,326` were explicitly excluded rather than scored against
|
||||
an uncovered volume.
|
||||
|
||||
The exact contract, metrics, rejection decision and reproduction notes are in
|
||||
`experiments/perception/L3_POINTPILLARS_TRANSFER_REPORT_2026-07-31.md`.
|
||||
|
||||
### L4 — live shadow integration
|
||||
|
||||
- [x] Add a provider-neutral bounded LiDAR local-surface queue independent of
|
||||
|
||||
@@ -0,0 +1,251 @@
|
||||
# L3 PointPillars transfer report — 2026-07-31
|
||||
|
||||
## Decision
|
||||
|
||||
The LiDAR-native detector seam is operational on Worker 006, but the frozen
|
||||
NVIDIA deployable checkpoint is **rejected as a K1 product detector
|
||||
candidate**.
|
||||
|
||||
The result is not a runtime failure. The exact ONNX model was built on the
|
||||
target GPU, loaded into the existing canonical Triton service and evaluated
|
||||
over all `3,769` frozen KITTI validation frames. Runtime latency is acceptable,
|
||||
but the public cross-domain accuracy is not.
|
||||
|
||||
Immutable admission:
|
||||
|
||||
`l3-pointpillars-admission-857a8c6a958850f82db7a58e894e13e2a54add09379ca58fb8e47db1e6aba65c`
|
||||
|
||||
Immutable transfer probe:
|
||||
|
||||
`l3-pointpillars-kitti-1a6b499e194a363644854dc324bd1b565c100b809f145c1324a25328e7ae0910`
|
||||
|
||||
Worker package:
|
||||
|
||||
`l3-pointpillars-worker-package-c219d3b35be63b03d68be318a0709af2422aa67ae12561cec6c1de6252abad12`
|
||||
|
||||
The next gate is a separately admitted training/adaptation candidate. The
|
||||
current deployable checkpoint must not be promoted into a detector claim,
|
||||
navigation, safety or camera-replacement path.
|
||||
|
||||
## Corrected model contract
|
||||
|
||||
The generic KITTI example configuration is not the contract of the downloaded
|
||||
NGC checkpoint. The source of truth is the exact ONNX graph with SHA-256:
|
||||
|
||||
`2dcabddc3a365e9608a112d7bbbb7db769a6dddeeaa59aa03611a83113326da1`.
|
||||
|
||||
Read-only graph inspection produced the sealed evidence file
|
||||
`l3_pointpillars_onnx_contract_2026-07-31.json`, SHA-256:
|
||||
|
||||
`2fd29cd054ab058c2cfec3dfba305c71e123ef3f04b457d0c64de0c8dac2e1be`.
|
||||
|
||||
The embedded contract is:
|
||||
|
||||
- input: `points [batch, 204800, 4] FP32`;
|
||||
- count: `num_points [batch] INT32`;
|
||||
- fields: LiDAR-frame `x, y, z, intensity`;
|
||||
- voxel range:
|
||||
`[-51.2000008, -51.2000008, -1.39999998]` to
|
||||
`[51.2000008, 51.2000008, 4.4000001]`;
|
||||
- voxel size: `[0.2, 0.2, 5.8]`;
|
||||
- maximum voxels: `10,000`;
|
||||
- maximum points per voxel: `32`;
|
||||
- embedded score threshold: `0.1`;
|
||||
- native labels: `Vehicle`, `Pedestrian`, `Cyclist`;
|
||||
- decoded output:
|
||||
`output_boxes [batch, 393216, 9] FP32` plus
|
||||
`num_boxes [batch] INT32`;
|
||||
- decoded row:
|
||||
`x, y, z, length, width, height, yaw, class_id, score`.
|
||||
|
||||
External postprocessing is pinned to NVIDIA
|
||||
`tao_toolkit_recipes@a540badc47812a17a94e924b537d49ad3969b5a8`:
|
||||
stable descending score order, `4,096` pre-NMS candidates and class-agnostic
|
||||
oriented-BEV NMS at IoU `0.01`.
|
||||
|
||||
The NGC model card states that this checkpoint was trained and evaluated on a
|
||||
proprietary solid-state LiDAR dataset. Its native accuracy cannot be
|
||||
independently reproduced from the public model package. KITTI is therefore a
|
||||
public **cross-domain transfer probe**, not a native model benchmark.
|
||||
|
||||
## Runtime evidence
|
||||
|
||||
No second container or serving stack was created.
|
||||
|
||||
- canonical service: `ndc-mission-core-triton`;
|
||||
- image:
|
||||
`nvcr.io/nvidia/tritonserver:26.06-py3@sha256:58df7489c3f2276f9591d500a012dee03e23d35543ce3c390b4c001e6bf90794`;
|
||||
- model control: explicit;
|
||||
- strict readiness: enabled;
|
||||
- model repository mount: read-only;
|
||||
- GPU: NVIDIA GeForce RTX 4090, compute capability `8.9`;
|
||||
- TensorRT: `11.0.0`;
|
||||
- precision policy: strongly typed;
|
||||
- engine SHA-256:
|
||||
`12005d972a4632d56342a5da44442b632c1dcc5144fa3c70b162dec334532481`;
|
||||
- engine size: `8,785,436` bytes;
|
||||
- target build time: `9.13932 s`;
|
||||
- observed peak builder allocation: approximately `192 MiB`.
|
||||
|
||||
The model was added to the existing repository and loaded through Triton's
|
||||
explicit load endpoint. The container was not restarted.
|
||||
|
||||
One GOOSE native XYZI frame with `169,883` points passed a representation
|
||||
smoke at `55.2069 ms` GPU compute time. One KITTI frame with `120,268` points
|
||||
also passed the live schema smoke. These smokes prove input/output
|
||||
compatibility only.
|
||||
|
||||
## Dataset admission
|
||||
|
||||
KITTI 3D Object Detection 2017 was admitted archive-only on Worker 006:
|
||||
|
||||
- release identity:
|
||||
`2c9615bedca56b492b204b614d4419db6431626e2a6867e207a95999beefcf47`;
|
||||
- labeled training frames: `7,481`;
|
||||
- official test frames: `7,518`;
|
||||
- split source: OpenPCDet commit
|
||||
`233f849829b6ac19afb8af8837a0246890908755`;
|
||||
- train split: `3,712`, SHA-256
|
||||
`b6417a1d9b18c8fdb085128e633d28ff321b7674a6d1b3841b8f43d865b281cb`;
|
||||
- validation split: `3,769`, SHA-256
|
||||
`657ac4bcc1e156e5b106a4ca18e1f88e012787ea1d2b5d0adeea97fee903fa86`.
|
||||
|
||||
The archives remain on Worker 006 and were not copied to the Mac. Admission
|
||||
verifies official archive byte lengths and digests, safe ZIP structure, XYZI
|
||||
packing, label/calibration alignment, finite positive target cuboids and the
|
||||
complete disjoint split.
|
||||
|
||||
GOOSE and RELLIS remain valid for representation, semantic and instance
|
||||
experiments. Their point-wise labels are not silently converted into oriented
|
||||
3D-box truth.
|
||||
|
||||
## Evaluation boundary
|
||||
|
||||
KITTI's common detector volume and the embedded NVIDIA checkpoint volume are
|
||||
different. The measured intersection is:
|
||||
|
||||
`[0, -39.68, -1.39999998]` to `[51.2000008, 39.68, 1]`.
|
||||
|
||||
Predictions outside this shared volume are excluded rather than counted as
|
||||
false positives against a region the probe does not jointly cover.
|
||||
|
||||
The metric is deliberately not the official KITTI server metric:
|
||||
|
||||
- 40-point interpolated BEV and 3D AP;
|
||||
- no KITTI easy/moderate/hard filtering;
|
||||
- IoU `0.7` for `Car`;
|
||||
- IoU `0.5` for `Pedestrian` and `Cyclist`;
|
||||
- fixed mapping `Vehicle → Car`;
|
||||
- no validation retuning;
|
||||
- one sequential worker process.
|
||||
|
||||
## Complete transfer result
|
||||
|
||||
All `3,769/3,769` validation frames completed.
|
||||
|
||||
| Metric | Result |
|
||||
| --- | ---: |
|
||||
| BEV mAP40 | `0.0246322` (`2.46322%`) |
|
||||
| 3D mAP40 | `0.00000216745` (`0.000216745%`) |
|
||||
| False occupied rate | `0.999981` (`99.9981%`) |
|
||||
| Mean inference latency | `31.1366 ms` |
|
||||
| P50 inference latency | `29.0994 ms` |
|
||||
| P95 inference latency | `45.9418 ms` |
|
||||
| Maximum inference latency | `68.8731 ms` |
|
||||
|
||||
Per class:
|
||||
|
||||
| Class | BEV AP40 | 3D AP40 | TP | FP | GT |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: |
|
||||
| Car | `0.0734788` | `0.0000034125` | `2` | `71,888` | `12,912` |
|
||||
| Pedestrian | `0.0000400679` | `0.00000308985` | `1` | `12,612` | `2,239` |
|
||||
| Cyclist | `0.000377644` | `0` | `0` | `74,760` | `801` |
|
||||
|
||||
Prediction accounting:
|
||||
|
||||
- post-NMS model boxes: `315,589`;
|
||||
- boxes inside the shared evaluation volume: `159,263`;
|
||||
- boxes outside the shared volume and explicitly ignored: `156,326`.
|
||||
|
||||
The frame-result set identity is:
|
||||
|
||||
`30b1933d508a09025a7d3c3c460fc2d06128e4bbe96a753bec7ba8545fda3e9c`.
|
||||
|
||||
The compact report SHA-256 is:
|
||||
|
||||
`f53d8d03a7c1e092d87b732872b26fea16447f373cf567bfe57737cb3ccbebba`.
|
||||
|
||||
## Interpretation
|
||||
|
||||
The worker seam, TensorRT engine, Triton serving path, bounded postprocess and
|
||||
full-dataset runner are usable. The pretrained checkpoint is not.
|
||||
|
||||
The result demonstrates severe domain mismatch; it does not demonstrate that
|
||||
PointPillars as an architecture is unsuitable for K1. It demonstrates that
|
||||
this exact proprietary-domain checkpoint cannot be used as the product
|
||||
detector without an admitted adaptation/training step.
|
||||
|
||||
No threshold was tuned to improve the result. Changing the score threshold on
|
||||
the frozen validation output would be validation leakage and would not repair
|
||||
the sensor-domain mismatch.
|
||||
|
||||
## Next gate
|
||||
|
||||
1. Freeze a new training/adaptation admission. It must name the trainable
|
||||
checkpoint, training container/toolchain, dataset identities, resource
|
||||
limits and output model identity before execution.
|
||||
2. Train only on the frozen `3,712` KITTI train frames.
|
||||
3. Keep all `3,769` validation frames untouched until the candidate is sealed.
|
||||
4. Export a new ONNX, build its TensorRT engine on Worker 006 and evaluate
|
||||
through the already proven canonical Triton seam.
|
||||
5. Require a useful public box-truth result before K1 accuracy language.
|
||||
6. Run `RAVNOVES01` as a K1 representation/stability transfer after the public
|
||||
candidate is useful. Because that session has no camera and no independent
|
||||
3D cuboids, it can prove deterministic replay, output stability, latency,
|
||||
queue/drop behavior and visual plausibility, but not detector accuracy.
|
||||
|
||||
The adaptation task is a new admission. This report does not authorize a
|
||||
training container, a second serving stack, K1 quality claims or navigation
|
||||
authority.
|
||||
|
||||
## Claim boundary
|
||||
|
||||
Not proved:
|
||||
|
||||
- native accuracy of the proprietary NVIDIA checkpoint;
|
||||
- K1 detector precision, recall, mAP or range/yaw accuracy;
|
||||
- that absence of a detection means free space;
|
||||
- superiority over the camera-first object candidate;
|
||||
- navigation or safety fitness.
|
||||
|
||||
Explicitly false:
|
||||
|
||||
- command authority;
|
||||
- navigation authority;
|
||||
- safety acceptance;
|
||||
- camera-first replacement;
|
||||
- LAB product publication.
|
||||
|
||||
## Reproduction
|
||||
|
||||
Admission:
|
||||
|
||||
```bash
|
||||
uv run python experiments/perception/run_l3_pointpillars_admission.py \
|
||||
--profile experiments/perception/l3_pointpillars_benchmark_profile.json \
|
||||
--dataset-inventory experiments/perception/l3_dataset_inventory_2026-07-30.json \
|
||||
--worker-inventory experiments/perception/l3_worker_inventory_2026-07-30.json
|
||||
```
|
||||
|
||||
The Worker package is built by:
|
||||
|
||||
```bash
|
||||
uv run python experiments/perception/prepare_l3_pointpillars_worker_package.py \
|
||||
--repository-root . \
|
||||
--output-root <worker-package-output> \
|
||||
--admission-result <accepted-admission-result>
|
||||
```
|
||||
|
||||
The complete probe runs only from the exact package on Worker 006 with
|
||||
`PYTHONPATH=<package>/runtime`, Python `-B`, the admitted KITTI root, the
|
||||
canonical model provenance and the existing local Triton endpoint.
|
||||
Reference in New Issue
Block a user