diff --git a/docs/22_MILESTONE_4_OBJECT_CENTRIC_RECORDED_REALTIME_CV.md b/docs/22_MILESTONE_4_OBJECT_CENTRIC_RECORDED_REALTIME_CV.md index 47104cb..c682d7a 100644 --- a/docs/22_MILESTONE_4_OBJECT_CENTRIC_RECORDED_REALTIME_CV.md +++ b/docs/22_MILESTONE_4_OBJECT_CENTRIC_RECORDED_REALTIME_CV.md @@ -572,6 +572,16 @@ Exit: ### M4.9 — recorded-realtime release candidate and cutover +The first complete integrated source-paced load gate passed on `2026-08-27`: +the frozen RF-DETR world-state graph and unchanged CPU TGS candidate accounted +for all `4,489` frames at `11.860865 FPS`, with joined output p95/p99 of +`46.825886/63.94485 ms` and zero capacity drops. The immutable identities and +resource envelope are recorded in +[`experiments/perception/M49_TGS_INTEGRATED_GRAPH_ACCEPTANCE_2026-08-27.md`](../experiments/perception/M49_TGS_INTEGRATED_GRAPH_ACCEPTANCE_2026-08-27.md). +This closes only the one-loop integrated load gate. Visual traversability, +uncapped parity, repeated-loop soak, restart/recovery, navigation, actuation +and production remain open. + Execute on Worker 006 with the frozen RAVNOVES00 source: 1. one full `1.0×` 448.723-second replay; diff --git a/docs/23_TRAVERSABILITY_AND_STATIC_OBSTACLE_STACK_DECISION.md b/docs/23_TRAVERSABILITY_AND_STATIC_OBSTACLE_STACK_DECISION.md index ee2026e..1f19599 100644 --- a/docs/23_TRAVERSABILITY_AND_STATIC_OBSTACLE_STACK_DECISION.md +++ b/docs/23_TRAVERSABILITY_AND_STATIC_OBSTACLE_STACK_DECISION.md @@ -344,6 +344,17 @@ This accepts the CPU source-paced shadow and retains the candidate. It does not accept visual traversability quality, integrated graph performance, navigation or actuation. +The product-integration load gate then passed on `2026-08-27`. The unchanged +TGS candidate ran concurrently with the frozen RF-DETR world-state graph for +all `4,489` frames at a requested `12 Hz`: every graph frame was delivered, +every TGS frame was accounted and capacity drops remained zero. The joined +output sustained `11.860865 FPS`, measured `46.825886/63.94485 ms` at p95/p99, +and the CPU-only TGS stage measured `1.735/2.14019 ms` at p95/p99 with a +`22.6 MiB` memory maximum. See +[`experiments/perception/M49_TGS_INTEGRATED_GRAPH_ACCEPTANCE_2026-08-27.md`](../experiments/perception/M49_TGS_INTEGRATED_GRAPH_ACCEPTANCE_2026-08-27.md). +Integrated runtime capacity is accepted; visual traversability, physical free +space, navigation, actuation and production remain unaccepted. + ### T4 — Candidate C occupancy/ESDF probe - Run nvblox separately with RF-DETR disabled for the isolated probe. @@ -361,15 +372,16 @@ or actuation. ## Immediate next action -Review the published full gravity-aligned TGS timeline in camera, metric 3D and -costmap space. Preserve +Keep the accepted integrated graph frozen and review the published full +gravity-aligned TGS timeline in camera, metric 3D and costmap space. Preserve `GROUND_SUPPORT`, `NONGROUND_OCCUPIED`, `UNKNOWN_REJECTED` and `UNOBSERVED` as separate products; do not use AOS or infer free cells from absent -republication. Attach the unchanged candidate to the realtime world-state graph -in non-authoritative shadow mode and measure complete-graph FPS, latency, -resource use and queue drops. If occupied evidence carpets the route, soft -traversable vegetation or usable gaps, reject TGS without tuning it against the -review evidence. Candidate A T2 replay remains blocked by the failed unmodified -T1 gate. No LOW-STEP tuning, new object model, camera resize, fisheye +republication. Integrated FPS, latency, resources and queue behavior are now +accepted for the one-loop source-paced gate. If occupied evidence carpets the +route, soft traversable vegetation or usable gaps, reject TGS without tuning it +against the review evidence. Close and induce-test the Worker inference-runtime +restart guard, then proceed to repeated-loop soak only if visual review also passes. +Candidate A T2 replay remains blocked by the failed unmodified T1 gate. No +LOW-STEP tuning, new object model, camera resize, fisheye rectification, manual dataset or parallel heavy Worker job is authorized by this decision. diff --git a/experiments/perception/M49_TGS_INTEGRATED_GRAPH_ACCEPTANCE_2026-08-27.md b/experiments/perception/M49_TGS_INTEGRATED_GRAPH_ACCEPTANCE_2026-08-27.md new file mode 100644 index 0000000..36da934 --- /dev/null +++ b/experiments/perception/M49_TGS_INTEGRATED_GRAPH_ACCEPTANCE_2026-08-27.md @@ -0,0 +1,123 @@ +# M4.9T5 integrated TGS + reference graph shadow — 2026-08-27 + +Status: **complete source-paced integrated shadow accepted**. Integrated +runtime capacity is accepted for the frozen RAVNOVES00 graph. Visual +traversability quality, physical free space, navigation, actuation and +production remain unaccepted. + +## Decision + +The unchanged gravity-aligned TGS candidate and the frozen RF-DETR +world-state graph completed the same `4,489`-frame RAVNOVES00 timeline from a +shared `12 Hz` start barrier on Worker 006. Every graph frame was delivered, +every TGS frame was accounted, all queues remained bounded and no capacity +drop or supersession occurred. + +The integrated graph sustained `11.860865 FPS`, compared with the frozen +reference result at `11.79902 FPS`. TGS added `1.735 ms` at p95 and used at +most `22.6 MiB` of container memory; it did not add a GPU model. The exact +per-frame join of graph and TGS outputs measured `46.825886 ms` at p95 and +`63.94485 ms` at p99. This accepts the runtime integration gate without +changing any navigation or safety authority. + +## Frozen identity + +| Item | Identity | +| --- | --- | +| Integrated result | `m49-tgs-integrated-graph-shadow-75e48fd8acf0246be16613e087fcf26fe909f7834718d217c74785a57f494b24` | +| Worker run | `Worker 006 / ravnoves00-integrated-12fps-003` | +| Mission Core revision used by Worker | `f153d671d316ea7b2200c0edcb80206d896ffcd5` | +| Deterministic Worker artifact | `1818ea57887db819e9d986713b050bd15eae1a43ee4af70c5b8b5d39f4cd8522` | +| Worker wheel | `b9414a1f2f1998543aeb63824d3cdfe55bbc6e4eecc3146d65962ebc7da86b8b` | +| TGS native binary | `2c3b4c7e6b66d19d2e32ee3ef867fa8c14073be29441bcd19fe565d9632c6538` | +| Integrated profile | `b61e018b2d04eec58802e2d4186ce7a3dd3a15b254db106b57b609e903eeef80` | +| Frozen graph config | `1db6f6fa0561d819505a5e4f62fe256dab6fe4d9da6b4f1a9a5073bfcf772c90` | +| Frozen TGS profile | `c2e07010aaee78259d36c057962d6bfb885349251ff7356d867e5813e632881c` | +| Linked accepted TGS result | `m49-tgs-full-shadow-ef98de7db7596d48e8c8c0549ce68e6704ee03e87c8c4bcf1e3e748b7ccb032e` | + +The release was built from `git archive` of the declared revision. Uncommitted +GAUSS and PlayCanvas files were neither packaged nor changed by the run. + +## Full-run result + +| Measurement | Result | +| --- | ---: | +| Graph admitted / delivered | `4,489 / 4,489` | +| TGS timeline accounted | `4,489 / 4,489` | +| LiDAR available | `3,928` | +| Delivery ratio | `1.0` | +| Capacity drops / supersessions / failures | `0 / 0 / 0` | +| Effective world-state rate | `11.860865 FPS` | +| Frozen reference rate | `11.79902 FPS` | +| World-state completion p50 / p95 / p99 / max | `28.437492 / 46.696395 / 63.94485 / 170.202343 ms` | +| Joined graph+TGS output p50 / p95 / p99 / max | `28.57367 / 46.825886 / 63.94485 / 170.202343 ms` | +| TGS stage p50 / p95 / p99 / max | `1.186 / 1.735 / 2.14019 / 6.151 ms` | +| TGS completion p50 / p95 / p99 / max | `17.605461 / 31.771832 / 42.957350 / 70.851977 ms` | +| Queue high-water marks | `2` for every bounded stage | +| Worker wrapper wall time | `544.887447 s` | + +All nineteen formal identity, accounting, throughput, latency, telemetry, +capacity and false-authority checks passed. + +## Resource envelope + +| Resource | Result | +| --- | ---: | +| GPU utilization mean / p95 / max | `43.114973 / 53 / 57 %` | +| GPU memory mean / p95 / max | `9,737.144 / 9,744 / 9,751 MiB` | +| GPU power p95 / max | `149.342 / 153.81 W` | +| TGS CPU mean / p95 / max | `3.474793 / 5.27 / 6.04 %` | +| TGS memory mean / p95 / max | `20.571818 / 21.84 / 22.6 MiB` | +| Graph container memory max | `1,769.472 MiB` | +| Triton container memory max | `582.9 MiB` | + +The measured GPU envelope belongs to the already accepted reference graph. +TGS is a CPU-only stage and does not request CUDA, Vulkan or a second model. + +## LAB publication boundary + +No third visual viewer is created for this gate. The TGS cells, source points, +camera frames and semantic archive are byte-identical to the already published +`m49-tgs-full-shadow` LAB. This run proves concurrent runtime behavior, so its +new evidence is the immutable timing, queue and resource report above. Visual +quality review continues in the existing full-shadow LAB. + +## Runtime recovery finding + +Preflight found the canonical `ndc-mission-core-triton` container stopped after +the Docker runtime restarted. Docker reported exit code `255`, no OOM event and +Compose configured `restart: "no"`. The accepted run began only after the exact +same container was started and healthy; its container ID remained +`4232fb040062a8384809e73612baa92b343ddcb60f00c23e7091eb4909959223` +throughout the run. + +This is a deployment-lifecycle defect, not a perception overload result. The +container must use a durable restart policy and an unhealthy-state watchdog; +changing the TGS or detector graph cannot fix it. + +The live container was subsequently changed in place to +`restart: unless-stopped`. Docker confirmed the service remained `running` and +`healthy` with the same container ID, so applying the mitigation caused no +inference restart. This protects the current container from process and Docker +daemon restarts. The versioned Worker Compose source still declares +`restart: "no"`; closing that source-level drift and inducing a controlled +recovery remain a separate deployment gate. + +## Sealed compact evidence + +| File | SHA-256 | +| --- | --- | +| `result.json` | `bfc881aa943d46582a248b6024a413c3c4b3a6959fa4a674f6ec33d658fdcbdf` | +| `graph-result.json` | `784d27b908137843f6adf08d267576a74b0676f5f98226923399ae59f538b78a` | +| `tgs-result.json` | `c0245744892b66c7f24af3cd7ea9c8cd8d543114fffa70e32d8d5fef81054716` | +| `worker-summary.json` | `c449246d14fb927b6490bcf657950b354f101fd3eb4b296bc0f6dd8543aafe10` | + +## Next gate + +Keep this integrated graph frozen. Complete the independent visual +traversability review in the existing TGS LAB, concentrating on vegetation, +compact obstacles, false occupied carpets, usable gaps and missing-LiDAR +behavior. In parallel, close the Worker inference-runtime restart policy and +verify an induced stop/recovery without changing result identity. Only after +both gates pass may the candidate advance to repeated-loop soak and recovery +acceptance; navigation and actuation remain off.