diff --git a/docs/13_LIDAR_WORKER_PRODUCT_AND_ROADMAP.md b/docs/13_LIDAR_WORKER_PRODUCT_AND_ROADMAP.md index 0dc7967..46b356e 100644 --- a/docs/13_LIDAR_WORKER_PRODUCT_AND_ROADMAP.md +++ b/docs/13_LIDAR_WORKER_PRODUCT_AND_ROADMAP.md @@ -620,21 +620,96 @@ Exit: one immutable K1 session yields both a persistent reconstruction and a bounded local world state without hard-coded terrain height or scanner-side changes. -### L3 — LiDAR-native 3D detection — deferred behind L2.6 +### L3 — LiDAR-native 3D detection — active after L2.6 -- [ ] Establish the public-dataset baseline first; freeze K1-specific 3D - annotations only when a measured domain gap justifies them. -- [ ] Run NVIDIA PointPillars through the existing external worker/Triton seam. -- [ ] Treat pretrained output as a baseline, not an accepted product model. -- [ ] Measure class precision/recall, center/range/yaw error, distance-bucket - recall, false occupied objects and end-to-end latency. +- [x] Freeze a fail-closed benchmark admission that distinguishes independent + oriented 3D box truth from point-wise semantic/instance labels. +- [x] Extract and seal the contract from the exact NVIDIA ONNX instead of + copying a generic KITTI example range into the product profile. +- [x] Build the target TensorRT engine on Worker 006, install it into the + existing canonical Triton repository and execute both GOOSE and KITTI schema + smokes without creating or restarting a second serving stack. +- [x] Admit all KITTI 3D Object Detection 2017 archives and the pinned complete + `3,712 / 3,769` train/validation split on Worker 006. +- [x] Measure the complete public cross-domain transfer probe through the + canonical worker seam: `3,769/3,769` frames, fixed postprocess and no + validation retuning. +- [x] Reject the exact `deployable_v1.1` checkpoint as a K1 product candidate: + runtime is healthy, while BEV mAP40 is `2.46322%`, 3D mAP40 is + `0.000216745%` and false occupied rate is `99.9981%`. +- [x] Preserve the claim boundary: the NGC model card's proprietary + solid-state-LiDAR result is not independently reproducible, KITTI is + cross-domain, and `RAVNOVES01` without independent cuboids cannot establish + detector accuracy. +- [ ] Freeze a separate training/adaptation admission before downloading a + trainable checkpoint or starting a training runtime. +- [ ] Train only on the frozen `3,712` KITTI train frames; keep all `3,769` + validation frames untouched until the candidate ONNX is sealed. +- [ ] Rebuild and evaluate the adapted engine through the already proven + Worker 006/Triton/postprocess seam. +- [ ] After a useful public box-truth result, run `RAVNOVES01` as a K1 + deterministic-replay, schema, latency, queue/drop and visual-plausibility + transfer. Do not call it a K1 accuracy benchmark. - [ ] Compare Autoware CenterPoint only after the PointPillars harness is stable. -- [ ] Fine-tune only if the baseline demonstrates useful transfer and the - annotation budget is justified. Exit: the selected detector beats the camera-derived cuboid baseline on the independent gate without increasing unsafe false-free or false-dynamic output. +The current immutable L3 admission result is +`l3-pointpillars-admission-857a8c6a958850f82db7a58e894e13e2a54add09379ca58fb8e47db1e6aba65c`. +It freezes NGC `nvidia/tao/pointpillarnet:deployable_v1.1` as ONNX digest +`2dcabddc3a365e9608a112d7bbbb7db769a6dddeeaa59aa03611a83113326da1`. +The signed companion label file has digest +`0adaeb5a374421b61bf83b8fa4522e11abd68461f239a4c72cf5627de913b3da` +and names `Vehicle`, `Pedestrian`, `Cyclist`. The evaluation profile explicitly +maps only `Vehicle → Car`; the other two class names are identity mappings. +The decoded row contract and class-agnostic BEV NMS are pinned to NVIDIA's +`tao_toolkit_recipes` commit +`a540badc47812a17a94e924b537d49ad3969b5a8`: `4,096` pre-NMS candidates and +IoU threshold `0.01`. The ONNX itself embeds score threshold `0.1` and the +voxel range `[-51.2, -51.2, -1.4]` to `[51.2, 51.2, 4.4]`; the sealed graph +contract has digest +`2fd29cd054ab058c2cfec3dfba305c71e123ef3f04b457d0c64de0c8dac2e1be`. +Postprocessing executes on Worker 006; the raw +`[1, 393216, 9]` tensor is never a Mac/browser transport contract. +The public metric is explicitly a cross-domain transfer probe, not native +checkpoint accuracy. It is a frozen local 40-point AP gate without KITTI +difficulty filtering or test-server submission: IoU is `0.7` for `Car` and +`0.5` for `Pedestrian/Cyclist`, with distance recall buckets +`0–20 / 20–40 / 40–70 m`. +Worker 006 built a TensorRT 11.0 engine for its RTX 4090 with digest +`12005d972a4632d56342a5da44442b632c1dcc5144fa3c70b162dec334532481`. +The build completed in `9.139 s`, used approximately `192 MiB` peak builder +GPU memory and stayed inside the existing `ndc-mission-core-triton` container. +The model was installed into the existing repository and loaded through +explicit model control without restarting Triton. + +One GOOSE native XYZI frame with `169,883` points then passed a single-query +representation smoke at `55.2069 ms` GPU compute time. This proves only that +the admitted sensor-frame XYZI shape can execute through the staged engine. +GOOSE and RELLIS provide independent point-wise semantic/instance truth, not +oriented 3D cuboids, so they cannot produce PointPillars 3D mAP, center or yaw +accuracy. KITTI 3D Object Detection 2017 is the independent public +cross-domain box-truth source under `CC-BY-NC-SA-3.0`, with the OpenPCDet split +at commit +`233f849829b6ac19afb8af8837a0246890908755`. Its admission is archive-only and +fail-closed: three official archive sizes and digests, XYZI packing, all +training/test calibrations, all training label rows, and the disjoint complete +`3,712 / 3,769` train/validation partition are verified before a path-free +state may be published. The source archives stay on Worker 006 and are not +silently copied to an operator host. + +The complete immutable transfer probe is +`l3-pointpillars-kitti-1a6b499e194a363644854dc324bd1b565c100b809f145c1324a25328e7ae0910`. +It processed `3,769/3,769` frames sequentially. Mean inference was +`31.1366 ms`, p95 was `45.9418 ms` and maximum was `68.8731 ms`. Of `315,589` +post-NMS boxes, `159,263` were inside the shared model/KITTI evaluation volume; +the remaining `156,326` were explicitly excluded rather than scored against +an uncovered volume. + +The exact contract, metrics, rejection decision and reproduction notes are in +`experiments/perception/L3_POINTPILLARS_TRANSFER_REPORT_2026-07-31.md`. + ### L4 — live shadow integration - [x] Add a provider-neutral bounded LiDAR local-surface queue independent of diff --git a/experiments/perception/L3_POINTPILLARS_TRANSFER_REPORT_2026-07-31.md b/experiments/perception/L3_POINTPILLARS_TRANSFER_REPORT_2026-07-31.md new file mode 100644 index 0000000..fdd5b77 --- /dev/null +++ b/experiments/perception/L3_POINTPILLARS_TRANSFER_REPORT_2026-07-31.md @@ -0,0 +1,251 @@ +# L3 PointPillars transfer report — 2026-07-31 + +## Decision + +The LiDAR-native detector seam is operational on Worker 006, but the frozen +NVIDIA deployable checkpoint is **rejected as a K1 product detector +candidate**. + +The result is not a runtime failure. The exact ONNX model was built on the +target GPU, loaded into the existing canonical Triton service and evaluated +over all `3,769` frozen KITTI validation frames. Runtime latency is acceptable, +but the public cross-domain accuracy is not. + +Immutable admission: + +`l3-pointpillars-admission-857a8c6a958850f82db7a58e894e13e2a54add09379ca58fb8e47db1e6aba65c` + +Immutable transfer probe: + +`l3-pointpillars-kitti-1a6b499e194a363644854dc324bd1b565c100b809f145c1324a25328e7ae0910` + +Worker package: + +`l3-pointpillars-worker-package-c219d3b35be63b03d68be318a0709af2422aa67ae12561cec6c1de6252abad12` + +The next gate is a separately admitted training/adaptation candidate. The +current deployable checkpoint must not be promoted into a detector claim, +navigation, safety or camera-replacement path. + +## Corrected model contract + +The generic KITTI example configuration is not the contract of the downloaded +NGC checkpoint. The source of truth is the exact ONNX graph with SHA-256: + +`2dcabddc3a365e9608a112d7bbbb7db769a6dddeeaa59aa03611a83113326da1`. + +Read-only graph inspection produced the sealed evidence file +`l3_pointpillars_onnx_contract_2026-07-31.json`, SHA-256: + +`2fd29cd054ab058c2cfec3dfba305c71e123ef3f04b457d0c64de0c8dac2e1be`. + +The embedded contract is: + +- input: `points [batch, 204800, 4] FP32`; +- count: `num_points [batch] INT32`; +- fields: LiDAR-frame `x, y, z, intensity`; +- voxel range: + `[-51.2000008, -51.2000008, -1.39999998]` to + `[51.2000008, 51.2000008, 4.4000001]`; +- voxel size: `[0.2, 0.2, 5.8]`; +- maximum voxels: `10,000`; +- maximum points per voxel: `32`; +- embedded score threshold: `0.1`; +- native labels: `Vehicle`, `Pedestrian`, `Cyclist`; +- decoded output: + `output_boxes [batch, 393216, 9] FP32` plus + `num_boxes [batch] INT32`; +- decoded row: + `x, y, z, length, width, height, yaw, class_id, score`. + +External postprocessing is pinned to NVIDIA +`tao_toolkit_recipes@a540badc47812a17a94e924b537d49ad3969b5a8`: +stable descending score order, `4,096` pre-NMS candidates and class-agnostic +oriented-BEV NMS at IoU `0.01`. + +The NGC model card states that this checkpoint was trained and evaluated on a +proprietary solid-state LiDAR dataset. Its native accuracy cannot be +independently reproduced from the public model package. KITTI is therefore a +public **cross-domain transfer probe**, not a native model benchmark. + +## Runtime evidence + +No second container or serving stack was created. + +- canonical service: `ndc-mission-core-triton`; +- image: + `nvcr.io/nvidia/tritonserver:26.06-py3@sha256:58df7489c3f2276f9591d500a012dee03e23d35543ce3c390b4c001e6bf90794`; +- model control: explicit; +- strict readiness: enabled; +- model repository mount: read-only; +- GPU: NVIDIA GeForce RTX 4090, compute capability `8.9`; +- TensorRT: `11.0.0`; +- precision policy: strongly typed; +- engine SHA-256: + `12005d972a4632d56342a5da44442b632c1dcc5144fa3c70b162dec334532481`; +- engine size: `8,785,436` bytes; +- target build time: `9.13932 s`; +- observed peak builder allocation: approximately `192 MiB`. + +The model was added to the existing repository and loaded through Triton's +explicit load endpoint. The container was not restarted. + +One GOOSE native XYZI frame with `169,883` points passed a representation +smoke at `55.2069 ms` GPU compute time. One KITTI frame with `120,268` points +also passed the live schema smoke. These smokes prove input/output +compatibility only. + +## Dataset admission + +KITTI 3D Object Detection 2017 was admitted archive-only on Worker 006: + +- release identity: + `2c9615bedca56b492b204b614d4419db6431626e2a6867e207a95999beefcf47`; +- labeled training frames: `7,481`; +- official test frames: `7,518`; +- split source: OpenPCDet commit + `233f849829b6ac19afb8af8837a0246890908755`; +- train split: `3,712`, SHA-256 + `b6417a1d9b18c8fdb085128e633d28ff321b7674a6d1b3841b8f43d865b281cb`; +- validation split: `3,769`, SHA-256 + `657ac4bcc1e156e5b106a4ca18e1f88e012787ea1d2b5d0adeea97fee903fa86`. + +The archives remain on Worker 006 and were not copied to the Mac. Admission +verifies official archive byte lengths and digests, safe ZIP structure, XYZI +packing, label/calibration alignment, finite positive target cuboids and the +complete disjoint split. + +GOOSE and RELLIS remain valid for representation, semantic and instance +experiments. Their point-wise labels are not silently converted into oriented +3D-box truth. + +## Evaluation boundary + +KITTI's common detector volume and the embedded NVIDIA checkpoint volume are +different. The measured intersection is: + +`[0, -39.68, -1.39999998]` to `[51.2000008, 39.68, 1]`. + +Predictions outside this shared volume are excluded rather than counted as +false positives against a region the probe does not jointly cover. + +The metric is deliberately not the official KITTI server metric: + +- 40-point interpolated BEV and 3D AP; +- no KITTI easy/moderate/hard filtering; +- IoU `0.7` for `Car`; +- IoU `0.5` for `Pedestrian` and `Cyclist`; +- fixed mapping `Vehicle → Car`; +- no validation retuning; +- one sequential worker process. + +## Complete transfer result + +All `3,769/3,769` validation frames completed. + +| Metric | Result | +| --- | ---: | +| BEV mAP40 | `0.0246322` (`2.46322%`) | +| 3D mAP40 | `0.00000216745` (`0.000216745%`) | +| False occupied rate | `0.999981` (`99.9981%`) | +| Mean inference latency | `31.1366 ms` | +| P50 inference latency | `29.0994 ms` | +| P95 inference latency | `45.9418 ms` | +| Maximum inference latency | `68.8731 ms` | + +Per class: + +| Class | BEV AP40 | 3D AP40 | TP | FP | GT | +| --- | ---: | ---: | ---: | ---: | ---: | +| Car | `0.0734788` | `0.0000034125` | `2` | `71,888` | `12,912` | +| Pedestrian | `0.0000400679` | `0.00000308985` | `1` | `12,612` | `2,239` | +| Cyclist | `0.000377644` | `0` | `0` | `74,760` | `801` | + +Prediction accounting: + +- post-NMS model boxes: `315,589`; +- boxes inside the shared evaluation volume: `159,263`; +- boxes outside the shared volume and explicitly ignored: `156,326`. + +The frame-result set identity is: + +`30b1933d508a09025a7d3c3c460fc2d06128e4bbe96a753bec7ba8545fda3e9c`. + +The compact report SHA-256 is: + +`f53d8d03a7c1e092d87b732872b26fea16447f373cf567bfe57737cb3ccbebba`. + +## Interpretation + +The worker seam, TensorRT engine, Triton serving path, bounded postprocess and +full-dataset runner are usable. The pretrained checkpoint is not. + +The result demonstrates severe domain mismatch; it does not demonstrate that +PointPillars as an architecture is unsuitable for K1. It demonstrates that +this exact proprietary-domain checkpoint cannot be used as the product +detector without an admitted adaptation/training step. + +No threshold was tuned to improve the result. Changing the score threshold on +the frozen validation output would be validation leakage and would not repair +the sensor-domain mismatch. + +## Next gate + +1. Freeze a new training/adaptation admission. It must name the trainable + checkpoint, training container/toolchain, dataset identities, resource + limits and output model identity before execution. +2. Train only on the frozen `3,712` KITTI train frames. +3. Keep all `3,769` validation frames untouched until the candidate is sealed. +4. Export a new ONNX, build its TensorRT engine on Worker 006 and evaluate + through the already proven canonical Triton seam. +5. Require a useful public box-truth result before K1 accuracy language. +6. Run `RAVNOVES01` as a K1 representation/stability transfer after the public + candidate is useful. Because that session has no camera and no independent + 3D cuboids, it can prove deterministic replay, output stability, latency, + queue/drop behavior and visual plausibility, but not detector accuracy. + +The adaptation task is a new admission. This report does not authorize a +training container, a second serving stack, K1 quality claims or navigation +authority. + +## Claim boundary + +Not proved: + +- native accuracy of the proprietary NVIDIA checkpoint; +- K1 detector precision, recall, mAP or range/yaw accuracy; +- that absence of a detection means free space; +- superiority over the camera-first object candidate; +- navigation or safety fitness. + +Explicitly false: + +- command authority; +- navigation authority; +- safety acceptance; +- camera-first replacement; +- LAB product publication. + +## Reproduction + +Admission: + +```bash +uv run python experiments/perception/run_l3_pointpillars_admission.py \ + --profile experiments/perception/l3_pointpillars_benchmark_profile.json \ + --dataset-inventory experiments/perception/l3_dataset_inventory_2026-07-30.json \ + --worker-inventory experiments/perception/l3_worker_inventory_2026-07-30.json +``` + +The Worker package is built by: + +```bash +uv run python experiments/perception/prepare_l3_pointpillars_worker_package.py \ + --repository-root . \ + --output-root \ + --admission-result +``` + +The complete probe runs only from the exact package on Worker 006 with +`PYTHONPATH=/runtime`, Python `-B`, the admitted KITTI root, the +canonical model provenance and the existing local Triton endpoint.