docs(perception): record PointPillars transfer verdict
This commit is contained in:
@@ -620,21 +620,96 @@ Exit: one immutable K1 session yields both a persistent reconstruction and a
|
|||||||
bounded local world state without hard-coded terrain height or scanner-side
|
bounded local world state without hard-coded terrain height or scanner-side
|
||||||
changes.
|
changes.
|
||||||
|
|
||||||
### L3 — LiDAR-native 3D detection — deferred behind L2.6
|
### L3 — LiDAR-native 3D detection — active after L2.6
|
||||||
|
|
||||||
- [ ] Establish the public-dataset baseline first; freeze K1-specific 3D
|
- [x] Freeze a fail-closed benchmark admission that distinguishes independent
|
||||||
annotations only when a measured domain gap justifies them.
|
oriented 3D box truth from point-wise semantic/instance labels.
|
||||||
- [ ] Run NVIDIA PointPillars through the existing external worker/Triton seam.
|
- [x] Extract and seal the contract from the exact NVIDIA ONNX instead of
|
||||||
- [ ] Treat pretrained output as a baseline, not an accepted product model.
|
copying a generic KITTI example range into the product profile.
|
||||||
- [ ] Measure class precision/recall, center/range/yaw error, distance-bucket
|
- [x] Build the target TensorRT engine on Worker 006, install it into the
|
||||||
recall, false occupied objects and end-to-end latency.
|
existing canonical Triton repository and execute both GOOSE and KITTI schema
|
||||||
|
smokes without creating or restarting a second serving stack.
|
||||||
|
- [x] Admit all KITTI 3D Object Detection 2017 archives and the pinned complete
|
||||||
|
`3,712 / 3,769` train/validation split on Worker 006.
|
||||||
|
- [x] Measure the complete public cross-domain transfer probe through the
|
||||||
|
canonical worker seam: `3,769/3,769` frames, fixed postprocess and no
|
||||||
|
validation retuning.
|
||||||
|
- [x] Reject the exact `deployable_v1.1` checkpoint as a K1 product candidate:
|
||||||
|
runtime is healthy, while BEV mAP40 is `2.46322%`, 3D mAP40 is
|
||||||
|
`0.000216745%` and false occupied rate is `99.9981%`.
|
||||||
|
- [x] Preserve the claim boundary: the NGC model card's proprietary
|
||||||
|
solid-state-LiDAR result is not independently reproducible, KITTI is
|
||||||
|
cross-domain, and `RAVNOVES01` without independent cuboids cannot establish
|
||||||
|
detector accuracy.
|
||||||
|
- [ ] Freeze a separate training/adaptation admission before downloading a
|
||||||
|
trainable checkpoint or starting a training runtime.
|
||||||
|
- [ ] Train only on the frozen `3,712` KITTI train frames; keep all `3,769`
|
||||||
|
validation frames untouched until the candidate ONNX is sealed.
|
||||||
|
- [ ] Rebuild and evaluate the adapted engine through the already proven
|
||||||
|
Worker 006/Triton/postprocess seam.
|
||||||
|
- [ ] After a useful public box-truth result, run `RAVNOVES01` as a K1
|
||||||
|
deterministic-replay, schema, latency, queue/drop and visual-plausibility
|
||||||
|
transfer. Do not call it a K1 accuracy benchmark.
|
||||||
- [ ] Compare Autoware CenterPoint only after the PointPillars harness is stable.
|
- [ ] Compare Autoware CenterPoint only after the PointPillars harness is stable.
|
||||||
- [ ] Fine-tune only if the baseline demonstrates useful transfer and the
|
|
||||||
annotation budget is justified.
|
|
||||||
|
|
||||||
Exit: the selected detector beats the camera-derived cuboid baseline on the
|
Exit: the selected detector beats the camera-derived cuboid baseline on the
|
||||||
independent gate without increasing unsafe false-free or false-dynamic output.
|
independent gate without increasing unsafe false-free or false-dynamic output.
|
||||||
|
|
||||||
|
The current immutable L3 admission result is
|
||||||
|
`l3-pointpillars-admission-857a8c6a958850f82db7a58e894e13e2a54add09379ca58fb8e47db1e6aba65c`.
|
||||||
|
It freezes NGC `nvidia/tao/pointpillarnet:deployable_v1.1` as ONNX digest
|
||||||
|
`2dcabddc3a365e9608a112d7bbbb7db769a6dddeeaa59aa03611a83113326da1`.
|
||||||
|
The signed companion label file has digest
|
||||||
|
`0adaeb5a374421b61bf83b8fa4522e11abd68461f239a4c72cf5627de913b3da`
|
||||||
|
and names `Vehicle`, `Pedestrian`, `Cyclist`. The evaluation profile explicitly
|
||||||
|
maps only `Vehicle → Car`; the other two class names are identity mappings.
|
||||||
|
The decoded row contract and class-agnostic BEV NMS are pinned to NVIDIA's
|
||||||
|
`tao_toolkit_recipes` commit
|
||||||
|
`a540badc47812a17a94e924b537d49ad3969b5a8`: `4,096` pre-NMS candidates and
|
||||||
|
IoU threshold `0.01`. The ONNX itself embeds score threshold `0.1` and the
|
||||||
|
voxel range `[-51.2, -51.2, -1.4]` to `[51.2, 51.2, 4.4]`; the sealed graph
|
||||||
|
contract has digest
|
||||||
|
`2fd29cd054ab058c2cfec3dfba305c71e123ef3f04b457d0c64de0c8dac2e1be`.
|
||||||
|
Postprocessing executes on Worker 006; the raw
|
||||||
|
`[1, 393216, 9]` tensor is never a Mac/browser transport contract.
|
||||||
|
The public metric is explicitly a cross-domain transfer probe, not native
|
||||||
|
checkpoint accuracy. It is a frozen local 40-point AP gate without KITTI
|
||||||
|
difficulty filtering or test-server submission: IoU is `0.7` for `Car` and
|
||||||
|
`0.5` for `Pedestrian/Cyclist`, with distance recall buckets
|
||||||
|
`0–20 / 20–40 / 40–70 m`.
|
||||||
|
Worker 006 built a TensorRT 11.0 engine for its RTX 4090 with digest
|
||||||
|
`12005d972a4632d56342a5da44442b632c1dcc5144fa3c70b162dec334532481`.
|
||||||
|
The build completed in `9.139 s`, used approximately `192 MiB` peak builder
|
||||||
|
GPU memory and stayed inside the existing `ndc-mission-core-triton` container.
|
||||||
|
The model was installed into the existing repository and loaded through
|
||||||
|
explicit model control without restarting Triton.
|
||||||
|
|
||||||
|
One GOOSE native XYZI frame with `169,883` points then passed a single-query
|
||||||
|
representation smoke at `55.2069 ms` GPU compute time. This proves only that
|
||||||
|
the admitted sensor-frame XYZI shape can execute through the staged engine.
|
||||||
|
GOOSE and RELLIS provide independent point-wise semantic/instance truth, not
|
||||||
|
oriented 3D cuboids, so they cannot produce PointPillars 3D mAP, center or yaw
|
||||||
|
accuracy. KITTI 3D Object Detection 2017 is the independent public
|
||||||
|
cross-domain box-truth source under `CC-BY-NC-SA-3.0`, with the OpenPCDet split
|
||||||
|
at commit
|
||||||
|
`233f849829b6ac19afb8af8837a0246890908755`. Its admission is archive-only and
|
||||||
|
fail-closed: three official archive sizes and digests, XYZI packing, all
|
||||||
|
training/test calibrations, all training label rows, and the disjoint complete
|
||||||
|
`3,712 / 3,769` train/validation partition are verified before a path-free
|
||||||
|
state may be published. The source archives stay on Worker 006 and are not
|
||||||
|
silently copied to an operator host.
|
||||||
|
|
||||||
|
The complete immutable transfer probe is
|
||||||
|
`l3-pointpillars-kitti-1a6b499e194a363644854dc324bd1b565c100b809f145c1324a25328e7ae0910`.
|
||||||
|
It processed `3,769/3,769` frames sequentially. Mean inference was
|
||||||
|
`31.1366 ms`, p95 was `45.9418 ms` and maximum was `68.8731 ms`. Of `315,589`
|
||||||
|
post-NMS boxes, `159,263` were inside the shared model/KITTI evaluation volume;
|
||||||
|
the remaining `156,326` were explicitly excluded rather than scored against
|
||||||
|
an uncovered volume.
|
||||||
|
|
||||||
|
The exact contract, metrics, rejection decision and reproduction notes are in
|
||||||
|
`experiments/perception/L3_POINTPILLARS_TRANSFER_REPORT_2026-07-31.md`.
|
||||||
|
|
||||||
### L4 — live shadow integration
|
### L4 — live shadow integration
|
||||||
|
|
||||||
- [x] Add a provider-neutral bounded LiDAR local-surface queue independent of
|
- [x] Add a provider-neutral bounded LiDAR local-surface queue independent of
|
||||||
|
|||||||
@@ -0,0 +1,251 @@
|
|||||||
|
# L3 PointPillars transfer report — 2026-07-31
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
The LiDAR-native detector seam is operational on Worker 006, but the frozen
|
||||||
|
NVIDIA deployable checkpoint is **rejected as a K1 product detector
|
||||||
|
candidate**.
|
||||||
|
|
||||||
|
The result is not a runtime failure. The exact ONNX model was built on the
|
||||||
|
target GPU, loaded into the existing canonical Triton service and evaluated
|
||||||
|
over all `3,769` frozen KITTI validation frames. Runtime latency is acceptable,
|
||||||
|
but the public cross-domain accuracy is not.
|
||||||
|
|
||||||
|
Immutable admission:
|
||||||
|
|
||||||
|
`l3-pointpillars-admission-857a8c6a958850f82db7a58e894e13e2a54add09379ca58fb8e47db1e6aba65c`
|
||||||
|
|
||||||
|
Immutable transfer probe:
|
||||||
|
|
||||||
|
`l3-pointpillars-kitti-1a6b499e194a363644854dc324bd1b565c100b809f145c1324a25328e7ae0910`
|
||||||
|
|
||||||
|
Worker package:
|
||||||
|
|
||||||
|
`l3-pointpillars-worker-package-c219d3b35be63b03d68be318a0709af2422aa67ae12561cec6c1de6252abad12`
|
||||||
|
|
||||||
|
The next gate is a separately admitted training/adaptation candidate. The
|
||||||
|
current deployable checkpoint must not be promoted into a detector claim,
|
||||||
|
navigation, safety or camera-replacement path.
|
||||||
|
|
||||||
|
## Corrected model contract
|
||||||
|
|
||||||
|
The generic KITTI example configuration is not the contract of the downloaded
|
||||||
|
NGC checkpoint. The source of truth is the exact ONNX graph with SHA-256:
|
||||||
|
|
||||||
|
`2dcabddc3a365e9608a112d7bbbb7db769a6dddeeaa59aa03611a83113326da1`.
|
||||||
|
|
||||||
|
Read-only graph inspection produced the sealed evidence file
|
||||||
|
`l3_pointpillars_onnx_contract_2026-07-31.json`, SHA-256:
|
||||||
|
|
||||||
|
`2fd29cd054ab058c2cfec3dfba305c71e123ef3f04b457d0c64de0c8dac2e1be`.
|
||||||
|
|
||||||
|
The embedded contract is:
|
||||||
|
|
||||||
|
- input: `points [batch, 204800, 4] FP32`;
|
||||||
|
- count: `num_points [batch] INT32`;
|
||||||
|
- fields: LiDAR-frame `x, y, z, intensity`;
|
||||||
|
- voxel range:
|
||||||
|
`[-51.2000008, -51.2000008, -1.39999998]` to
|
||||||
|
`[51.2000008, 51.2000008, 4.4000001]`;
|
||||||
|
- voxel size: `[0.2, 0.2, 5.8]`;
|
||||||
|
- maximum voxels: `10,000`;
|
||||||
|
- maximum points per voxel: `32`;
|
||||||
|
- embedded score threshold: `0.1`;
|
||||||
|
- native labels: `Vehicle`, `Pedestrian`, `Cyclist`;
|
||||||
|
- decoded output:
|
||||||
|
`output_boxes [batch, 393216, 9] FP32` plus
|
||||||
|
`num_boxes [batch] INT32`;
|
||||||
|
- decoded row:
|
||||||
|
`x, y, z, length, width, height, yaw, class_id, score`.
|
||||||
|
|
||||||
|
External postprocessing is pinned to NVIDIA
|
||||||
|
`tao_toolkit_recipes@a540badc47812a17a94e924b537d49ad3969b5a8`:
|
||||||
|
stable descending score order, `4,096` pre-NMS candidates and class-agnostic
|
||||||
|
oriented-BEV NMS at IoU `0.01`.
|
||||||
|
|
||||||
|
The NGC model card states that this checkpoint was trained and evaluated on a
|
||||||
|
proprietary solid-state LiDAR dataset. Its native accuracy cannot be
|
||||||
|
independently reproduced from the public model package. KITTI is therefore a
|
||||||
|
public **cross-domain transfer probe**, not a native model benchmark.
|
||||||
|
|
||||||
|
## Runtime evidence
|
||||||
|
|
||||||
|
No second container or serving stack was created.
|
||||||
|
|
||||||
|
- canonical service: `ndc-mission-core-triton`;
|
||||||
|
- image:
|
||||||
|
`nvcr.io/nvidia/tritonserver:26.06-py3@sha256:58df7489c3f2276f9591d500a012dee03e23d35543ce3c390b4c001e6bf90794`;
|
||||||
|
- model control: explicit;
|
||||||
|
- strict readiness: enabled;
|
||||||
|
- model repository mount: read-only;
|
||||||
|
- GPU: NVIDIA GeForce RTX 4090, compute capability `8.9`;
|
||||||
|
- TensorRT: `11.0.0`;
|
||||||
|
- precision policy: strongly typed;
|
||||||
|
- engine SHA-256:
|
||||||
|
`12005d972a4632d56342a5da44442b632c1dcc5144fa3c70b162dec334532481`;
|
||||||
|
- engine size: `8,785,436` bytes;
|
||||||
|
- target build time: `9.13932 s`;
|
||||||
|
- observed peak builder allocation: approximately `192 MiB`.
|
||||||
|
|
||||||
|
The model was added to the existing repository and loaded through Triton's
|
||||||
|
explicit load endpoint. The container was not restarted.
|
||||||
|
|
||||||
|
One GOOSE native XYZI frame with `169,883` points passed a representation
|
||||||
|
smoke at `55.2069 ms` GPU compute time. One KITTI frame with `120,268` points
|
||||||
|
also passed the live schema smoke. These smokes prove input/output
|
||||||
|
compatibility only.
|
||||||
|
|
||||||
|
## Dataset admission
|
||||||
|
|
||||||
|
KITTI 3D Object Detection 2017 was admitted archive-only on Worker 006:
|
||||||
|
|
||||||
|
- release identity:
|
||||||
|
`2c9615bedca56b492b204b614d4419db6431626e2a6867e207a95999beefcf47`;
|
||||||
|
- labeled training frames: `7,481`;
|
||||||
|
- official test frames: `7,518`;
|
||||||
|
- split source: OpenPCDet commit
|
||||||
|
`233f849829b6ac19afb8af8837a0246890908755`;
|
||||||
|
- train split: `3,712`, SHA-256
|
||||||
|
`b6417a1d9b18c8fdb085128e633d28ff321b7674a6d1b3841b8f43d865b281cb`;
|
||||||
|
- validation split: `3,769`, SHA-256
|
||||||
|
`657ac4bcc1e156e5b106a4ca18e1f88e012787ea1d2b5d0adeea97fee903fa86`.
|
||||||
|
|
||||||
|
The archives remain on Worker 006 and were not copied to the Mac. Admission
|
||||||
|
verifies official archive byte lengths and digests, safe ZIP structure, XYZI
|
||||||
|
packing, label/calibration alignment, finite positive target cuboids and the
|
||||||
|
complete disjoint split.
|
||||||
|
|
||||||
|
GOOSE and RELLIS remain valid for representation, semantic and instance
|
||||||
|
experiments. Their point-wise labels are not silently converted into oriented
|
||||||
|
3D-box truth.
|
||||||
|
|
||||||
|
## Evaluation boundary
|
||||||
|
|
||||||
|
KITTI's common detector volume and the embedded NVIDIA checkpoint volume are
|
||||||
|
different. The measured intersection is:
|
||||||
|
|
||||||
|
`[0, -39.68, -1.39999998]` to `[51.2000008, 39.68, 1]`.
|
||||||
|
|
||||||
|
Predictions outside this shared volume are excluded rather than counted as
|
||||||
|
false positives against a region the probe does not jointly cover.
|
||||||
|
|
||||||
|
The metric is deliberately not the official KITTI server metric:
|
||||||
|
|
||||||
|
- 40-point interpolated BEV and 3D AP;
|
||||||
|
- no KITTI easy/moderate/hard filtering;
|
||||||
|
- IoU `0.7` for `Car`;
|
||||||
|
- IoU `0.5` for `Pedestrian` and `Cyclist`;
|
||||||
|
- fixed mapping `Vehicle → Car`;
|
||||||
|
- no validation retuning;
|
||||||
|
- one sequential worker process.
|
||||||
|
|
||||||
|
## Complete transfer result
|
||||||
|
|
||||||
|
All `3,769/3,769` validation frames completed.
|
||||||
|
|
||||||
|
| Metric | Result |
|
||||||
|
| --- | ---: |
|
||||||
|
| BEV mAP40 | `0.0246322` (`2.46322%`) |
|
||||||
|
| 3D mAP40 | `0.00000216745` (`0.000216745%`) |
|
||||||
|
| False occupied rate | `0.999981` (`99.9981%`) |
|
||||||
|
| Mean inference latency | `31.1366 ms` |
|
||||||
|
| P50 inference latency | `29.0994 ms` |
|
||||||
|
| P95 inference latency | `45.9418 ms` |
|
||||||
|
| Maximum inference latency | `68.8731 ms` |
|
||||||
|
|
||||||
|
Per class:
|
||||||
|
|
||||||
|
| Class | BEV AP40 | 3D AP40 | TP | FP | GT |
|
||||||
|
| --- | ---: | ---: | ---: | ---: | ---: |
|
||||||
|
| Car | `0.0734788` | `0.0000034125` | `2` | `71,888` | `12,912` |
|
||||||
|
| Pedestrian | `0.0000400679` | `0.00000308985` | `1` | `12,612` | `2,239` |
|
||||||
|
| Cyclist | `0.000377644` | `0` | `0` | `74,760` | `801` |
|
||||||
|
|
||||||
|
Prediction accounting:
|
||||||
|
|
||||||
|
- post-NMS model boxes: `315,589`;
|
||||||
|
- boxes inside the shared evaluation volume: `159,263`;
|
||||||
|
- boxes outside the shared volume and explicitly ignored: `156,326`.
|
||||||
|
|
||||||
|
The frame-result set identity is:
|
||||||
|
|
||||||
|
`30b1933d508a09025a7d3c3c460fc2d06128e4bbe96a753bec7ba8545fda3e9c`.
|
||||||
|
|
||||||
|
The compact report SHA-256 is:
|
||||||
|
|
||||||
|
`f53d8d03a7c1e092d87b732872b26fea16447f373cf567bfe57737cb3ccbebba`.
|
||||||
|
|
||||||
|
## Interpretation
|
||||||
|
|
||||||
|
The worker seam, TensorRT engine, Triton serving path, bounded postprocess and
|
||||||
|
full-dataset runner are usable. The pretrained checkpoint is not.
|
||||||
|
|
||||||
|
The result demonstrates severe domain mismatch; it does not demonstrate that
|
||||||
|
PointPillars as an architecture is unsuitable for K1. It demonstrates that
|
||||||
|
this exact proprietary-domain checkpoint cannot be used as the product
|
||||||
|
detector without an admitted adaptation/training step.
|
||||||
|
|
||||||
|
No threshold was tuned to improve the result. Changing the score threshold on
|
||||||
|
the frozen validation output would be validation leakage and would not repair
|
||||||
|
the sensor-domain mismatch.
|
||||||
|
|
||||||
|
## Next gate
|
||||||
|
|
||||||
|
1. Freeze a new training/adaptation admission. It must name the trainable
|
||||||
|
checkpoint, training container/toolchain, dataset identities, resource
|
||||||
|
limits and output model identity before execution.
|
||||||
|
2. Train only on the frozen `3,712` KITTI train frames.
|
||||||
|
3. Keep all `3,769` validation frames untouched until the candidate is sealed.
|
||||||
|
4. Export a new ONNX, build its TensorRT engine on Worker 006 and evaluate
|
||||||
|
through the already proven canonical Triton seam.
|
||||||
|
5. Require a useful public box-truth result before K1 accuracy language.
|
||||||
|
6. Run `RAVNOVES01` as a K1 representation/stability transfer after the public
|
||||||
|
candidate is useful. Because that session has no camera and no independent
|
||||||
|
3D cuboids, it can prove deterministic replay, output stability, latency,
|
||||||
|
queue/drop behavior and visual plausibility, but not detector accuracy.
|
||||||
|
|
||||||
|
The adaptation task is a new admission. This report does not authorize a
|
||||||
|
training container, a second serving stack, K1 quality claims or navigation
|
||||||
|
authority.
|
||||||
|
|
||||||
|
## Claim boundary
|
||||||
|
|
||||||
|
Not proved:
|
||||||
|
|
||||||
|
- native accuracy of the proprietary NVIDIA checkpoint;
|
||||||
|
- K1 detector precision, recall, mAP or range/yaw accuracy;
|
||||||
|
- that absence of a detection means free space;
|
||||||
|
- superiority over the camera-first object candidate;
|
||||||
|
- navigation or safety fitness.
|
||||||
|
|
||||||
|
Explicitly false:
|
||||||
|
|
||||||
|
- command authority;
|
||||||
|
- navigation authority;
|
||||||
|
- safety acceptance;
|
||||||
|
- camera-first replacement;
|
||||||
|
- LAB product publication.
|
||||||
|
|
||||||
|
## Reproduction
|
||||||
|
|
||||||
|
Admission:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv run python experiments/perception/run_l3_pointpillars_admission.py \
|
||||||
|
--profile experiments/perception/l3_pointpillars_benchmark_profile.json \
|
||||||
|
--dataset-inventory experiments/perception/l3_dataset_inventory_2026-07-30.json \
|
||||||
|
--worker-inventory experiments/perception/l3_worker_inventory_2026-07-30.json
|
||||||
|
```
|
||||||
|
|
||||||
|
The Worker package is built by:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv run python experiments/perception/prepare_l3_pointpillars_worker_package.py \
|
||||||
|
--repository-root . \
|
||||||
|
--output-root <worker-package-output> \
|
||||||
|
--admission-result <accepted-admission-result>
|
||||||
|
```
|
||||||
|
|
||||||
|
The complete probe runs only from the exact package on Worker 006 with
|
||||||
|
`PYTHONPATH=<package>/runtime`, Python `-B`, the admitted KITTI root, the
|
||||||
|
canonical model provenance and the existing local Triton endpoint.
|
||||||
Reference in New Issue
Block a user