feat(perception): evaluate fixed-class detector candidates
This commit is contained in:
DCCONSTRUCTIONS
@@ -0,0 +1,133 @@
|
||||
# M48S fixed-class detector tournament и Worker deployment gate
|
||||
|
||||
Дата: 2026-08-25
|
||||
Режим: experimental shadow
|
||||
Worker: `worker-006`, NVIDIA GeForce RTX 4090
|
||||
Production acceptance: **нет**
|
||||
|
||||
## Решение
|
||||
|
||||
RF-DETR-L выиграл bounded-турнир, был экспортирован в ONNX, преобразован в
|
||||
strongly typed FP16 graph, собран TensorRT 11 и проверен через изолированный
|
||||
Triton. Detector-only нагрузочный gate длиной 30 минут принят. Кандидат готов к
|
||||
следующему shadow-gate внутри полного reference graph, но не получил
|
||||
navigation, safety, command или actuation authority.
|
||||
|
||||
Production critical path не включает Mask Grounding DINO, SAM 2 или OpenCLIP.
|
||||
Статические неизвестные объекты остаются ответственностью geometry/occupancy и
|
||||
объезжаются без расхода detector inference на их название. Один fixed-class
|
||||
проход камеры используется только для поведенчески значимых классов.
|
||||
|
||||
## Поведенческая граница
|
||||
|
||||
RF-DETR shadow provider выпускает только:
|
||||
|
||||
- `person`;
|
||||
- `bicycle`, `motorcycle`, `skateboard`;
|
||||
- `car`, `bus`, `truck`;
|
||||
- COCO animal classes, включая `dog`.
|
||||
|
||||
Урны, столбы, полусферы, бордюры и прочие статические препятствия не обязаны
|
||||
получать семантическое имя: их наличие и геометрия принадлежат class-free
|
||||
occupancy. Неизвестный движущийся объект остаётся conservative. Отдельного
|
||||
COCO-класса `scooter` нет, поэтому самокат пока нельзя считать надёжно
|
||||
классифицированным: до отдельного admission gate он остаётся geometry/motion
|
||||
hazard, а не безопасным отрицанием.
|
||||
|
||||
## Турнир на immutable 11-frame slice
|
||||
|
||||
Все профили выполняли один inference pass на кадр; ручная проверка не объявлена
|
||||
ground truth.
|
||||
|
||||
| Профиль | Core capacity | p95 | Собака на frame 253 | Решение |
|
||||
|---|---:|---:|---|---|
|
||||
| YOLOX-S all-COCO/v2 | 39,136 FPS | 38,267 мс | нет | regression baseline |
|
||||
| D-FINE-S COCO FP16 | 31,100 FPS | 42,945 мс | нет при 0,25 и 0,5 | отклонён |
|
||||
| RF-DETR-L COCO FP16 | 44,786 FPS | 32,588 мс | да, score 0,740723 | finalist |
|
||||
|
||||
D-FINE также давал заметные semantic confusions: собака как `skateboard`, корпус
|
||||
сканера как `surfboard`, дублирующиеся risk-labels на одном объекте. RF-DETR на
|
||||
этом slice дал более чистые person/vehicle labels и корректную собаку.
|
||||
|
||||
Immutable tournament result:
|
||||
`m48s-fixed-detector-tournament-0e61d75e6dc575d53e4bb98772a41d240fe627ad642de5178beb1154636e1299`.
|
||||
|
||||
## TensorRT/Triton квалификация
|
||||
|
||||
Закреплены следующие identities:
|
||||
|
||||
- upstream RF-DETR revision: `9b009fa928d6218320439803d1da01869a85c072`;
|
||||
- checkpoint SHA-256: `0f4e20e19a99c0f8a62b5685f57f6c8b5c371c59081feda6752a0561a79ccf38`;
|
||||
- exported ONNX SHA-256: `9c1948e56bbb6ff03349012b8bb334cacaf8ae480f22caa0704ee70de9a72300`;
|
||||
- strongly typed FP16 ONNX SHA-256: `9015fcc1317f268ce866bed6b5a33132c24963e1502b02f145fa184e11de5ecb`;
|
||||
- Worker 006 TensorRT engine SHA-256: `986399ce706b7380472cf5e473232249fed6e628971d8007f6609e83128d46b8`.
|
||||
|
||||
TensorRT parity на frame 253:
|
||||
|
||||
- PyTorch dog score: 0,740723;
|
||||
- TensorRT dog score: 0,741674;
|
||||
- абсолютная разница score: 0,000951;
|
||||
- box IoU: 0,990117;
|
||||
- class counts при threshold 0,5 совпадают точно: 57 `car`, 8 `truck`,
|
||||
6 `person`, 1 `dog`, 1 `fire hydrant`.
|
||||
|
||||
100-iteration Triton benchmark: 42,496 FPS end-to-end; mean 23,531 мс; p95
|
||||
34,156 мс. Production Triton во время проверки не изменялся: использовался
|
||||
отдельный безпортовый Triton-контейнер в namespace эксперимента.
|
||||
|
||||
## 30-минутный source-paced gate
|
||||
|
||||
Источник: RAVNOVES00, SHA-256
|
||||
`cadd1696ff000904eb78633a0a8418104b8024f178b91f3421789021ccb160e8`,
|
||||
10,0039 FPS. Durable background services на Worker оставались включёнными.
|
||||
|
||||
| Метрика | Результат | Gate |
|
||||
|---|---:|---:|
|
||||
| Время | 1800,020 с | ≥ 1800 с |
|
||||
| Кадры | 18 008 produced / 18 008 consumed | без потерь |
|
||||
| Effective FPS | 10,004 | ≥ 9,5 |
|
||||
| Detector end-to-end p95 | 32,415 мс | наблюдение |
|
||||
| Detector completion age p95 | 40,621 мс | ≤ 175 мс |
|
||||
| Queue | max depth 1/2, replacements 0 | bounded latest-wins |
|
||||
| GPU utilization | mean 51,408%, p95 55%, max 65% | без sustained 100% |
|
||||
| Worker VRAM | mean 9542,7 MiB, max 9556 MiB | ≤ 20 GiB |
|
||||
| Ошибки | 0 | 0 |
|
||||
|
||||
Все восемь автоматических load checks приняты. Это detector-only gate, поэтому
|
||||
он не доказывает p95 полного world state, качество tracker association или
|
||||
корректность risk-policy.
|
||||
|
||||
Immutable deployment result:
|
||||
`m48s-rf-detr-deployment-gate-2feb9e1b12a5588951ad35d63bf23cf6bdd579d54b5329d46d7696f88c444547`.
|
||||
|
||||
## Реализация
|
||||
|
||||
- `src/k1link/perception/rf_detr_object_detector.py` — pinned preprocessing,
|
||||
Triton V2 binary HTTP backend, FP16 output validation, fixed risk-class
|
||||
qualification и fail-closed FOV/area gates.
|
||||
- `src/k1link/perception/detector.py` — `RfDetrShadowDetectorProvider`, один
|
||||
inference pass и semantic hints без authority.
|
||||
- `config/perception/rf-detr-large-risk-shadow-v0.json` — неизменяемый профиль,
|
||||
threshold 0,25, bounded latest-wins queue capacity 2, geometry-owned static
|
||||
occupancy.
|
||||
- `experiments/perception/worker/` — воспроизводимые export/build/Triton
|
||||
declarations.
|
||||
- `experiments/perception/run_m48s_rf_detr_load_worker.py` — source-paced
|
||||
concurrent-load gate с GPU, queue и Triton accounting.
|
||||
- `experiments/perception/seal_m48s_rf_detr_deployment.py` — content-addressed
|
||||
immutable seal с false authority.
|
||||
|
||||
## Следующий gate
|
||||
|
||||
Подключить этот provider в полный reference graph вместе с существующими
|
||||
geometry observations, tracker и advisory risk-policy. На том же записанном
|
||||
источнике и при сохранённых Worker services требуется:
|
||||
|
||||
1. world-state p95 не более 175 мс;
|
||||
2. не менее 9,5 source FPS, bounded latest-wins без неучтённых потерь;
|
||||
3. стабильные track identities и conservative unknown-moving handling;
|
||||
4. раздельные реакции на person/animal/light-road-user/vehicle;
|
||||
5. отсутствие navigation/safety/command authority до отдельного acceptance.
|
||||
|
||||
Только после этого можно решать вопрос о замене текущего production detector.
|
||||
Текущий результат разрешает reference-graph shadow, а не production switch.
|
||||
@@ -0,0 +1,62 @@
|
||||
# M48S YOLOX-S all-COCO shadow report
|
||||
|
||||
Date: 2026-08-25
|
||||
Status: executable shadow completed; full-load promotion gate open
|
||||
|
||||
## Why six classes were previously emitted
|
||||
|
||||
The accepted `triton-yolox-s-raw-kb4/v1` provider was deliberately frozen on
|
||||
COCO ids `0, 1, 2, 3, 5, 7`: person, bicycle, car, motorcycle, bus and truck.
|
||||
That was a bounded detector qualification and reproducibility boundary, not an
|
||||
inference optimization. YOLOX-S already returns an `[1, 8400, 85]` tensor with
|
||||
all 80 COCO class scores. The six-class filter ran after the single inference.
|
||||
|
||||
The old provider, hashes and M4 replay results remain unchanged. The new
|
||||
`triton-yolox-s-raw-kb4-all-coco/v2` provider uses the same model, tensor,
|
||||
preprocess, thresholds and valid-FOV gates, but emits every qualified COCO class.
|
||||
|
||||
## Worker comparison
|
||||
|
||||
Both profiles were applied to the exact same tensor response for each of the 11
|
||||
M48S frames. The client shared the Worker's `mission-core-compute_default`
|
||||
network with Triton, avoiding host-NAT tensor transport.
|
||||
|
||||
| Measure | Frozen six classes | All COCO-80 |
|
||||
|---|---:|---:|
|
||||
| Inference passes per frame | 1 | 1, shared |
|
||||
| Detections | 44 | 45 |
|
||||
| Postprocess mean, 220 balanced iterations | 6.430 ms | 6.725 ms |
|
||||
| Postprocess p50 | 5.671 ms | 5.679 ms |
|
||||
| Postprocess p95 | 11.013 ms | 11.592 ms |
|
||||
|
||||
The measured mean postprocess difference was `0.295 ms`; the p50 difference was
|
||||
`0.008 ms`. The combined preprocess + inference + all-COCO postprocess capacity
|
||||
was `39.136 FPS` on this bounded slice. This is a capacity diagnostic, not a
|
||||
full-route load acceptance.
|
||||
|
||||
The only newly emitted detection was `handbag` on frame 253, correctly covering
|
||||
the bag carried by the visible person. The class counts were 36 car, five truck,
|
||||
three person and one handbag. The visible dog on frame 253 was not detected.
|
||||
Removing the filter therefore exposes all model answers at negligible compute
|
||||
cost, but does not repair classes the model fails to recognize.
|
||||
|
||||
Immutable result:
|
||||
`m48s-yolox-all-coco-shadow-7dbe6043b3fc12c7ddb162f609f883d86b34a4f2dd3785a632795f257e192d06`.
|
||||
|
||||
## Policy boundary
|
||||
|
||||
All qualified COCO labels are now available to downstream consumers. Emission
|
||||
does not grant every class behavioral authority:
|
||||
|
||||
- person, animal, light road user and vehicle labels may enter a separately
|
||||
versioned risk policy after qualification;
|
||||
- other labels remain advisory diagnostics;
|
||||
- geometry owns occupancy for every object;
|
||||
- unknown moving objects retain conservative risk;
|
||||
- unknown stationary objects remain route-around;
|
||||
- commands, actuation, navigation and safety authority remain false.
|
||||
|
||||
The next gate is the full recorded source under representative concurrent Worker
|
||||
load. It must compare source delivery, detector FPS, p95 latency, queue depth,
|
||||
drops, GPU utilization and VRAM against the frozen six-class baseline before v2
|
||||
can replace v1 in the production assembly.
|
||||
@@ -0,0 +1,156 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Convert the pinned RF-DETR ONNX graph to a strongly typed FP16 graph."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any, Final
|
||||
|
||||
import onnx # type: ignore[import-not-found]
|
||||
from onnx import TensorProto
|
||||
from onnxconverter_common import float16 # type: ignore[import-not-found]
|
||||
|
||||
SCHEMA_VERSION: Final = "missioncore.m48s-rf-detr-onnx-fp16-conversion/v3"
|
||||
FALSE_AUTHORITY: Final = {
|
||||
"ground_truth": False,
|
||||
"candidate_accepted": False,
|
||||
"commands_enabled": False,
|
||||
"actuation_allowed": False,
|
||||
"navigation_or_safety_accepted": False,
|
||||
}
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--input", type=Path, required=True)
|
||||
parser.add_argument("--expected-input-sha256", required=True)
|
||||
parser.add_argument("--output", type=Path, required=True)
|
||||
parser.add_argument("--manifest", type=Path, required=True)
|
||||
arguments = parser.parse_args()
|
||||
|
||||
source = arguments.input.resolve(strict=True)
|
||||
source_sha256 = sha256_path(source)
|
||||
if source_sha256 != arguments.expected_input_sha256:
|
||||
raise RuntimeError("source ONNX SHA-256 does not match the export manifest")
|
||||
output = arguments.output.absolute()
|
||||
manifest = arguments.manifest.absolute()
|
||||
if output.exists() or manifest.exists():
|
||||
raise RuntimeError("FP16 ONNX output or manifest already exists")
|
||||
|
||||
started_utc_ns = time.time_ns()
|
||||
graph = onnx.load(str(source))
|
||||
converted = float16.convert_float_to_float16(
|
||||
graph,
|
||||
keep_io_types=False,
|
||||
disable_shape_infer=False,
|
||||
)
|
||||
retargeted_casts = _retarget_float_casts_to_fp16(converted)
|
||||
_insert_fp32_input_cast(converted)
|
||||
onnx.checker.check_model(converted)
|
||||
output.parent.mkdir(mode=0o700, parents=True, exist_ok=True)
|
||||
onnx.save(converted, str(output))
|
||||
verified = onnx.load(str(output), load_external_data=False)
|
||||
onnx.checker.check_model(verified)
|
||||
input_types = {value.name: value.type.tensor_type.elem_type for value in verified.graph.input}
|
||||
output_types = {
|
||||
value.name: value.type.tensor_type.elem_type for value in verified.graph.output
|
||||
}
|
||||
if input_types != {"input": TensorProto.FLOAT}:
|
||||
raise RuntimeError(f"FP16 ONNX input boundary is not FLOAT: {input_types}")
|
||||
if output_types != {"dets": TensorProto.FLOAT16, "labels": TensorProto.FLOAT16}:
|
||||
raise RuntimeError(f"FP16 ONNX outputs are not FLOAT16: {output_types}")
|
||||
initializer_counts = _initializer_type_counts(verified)
|
||||
if initializer_counts.get("FLOAT16", 0) == 0:
|
||||
raise RuntimeError("FP16 ONNX has no FLOAT16 initializers")
|
||||
document = {
|
||||
"schema_version": SCHEMA_VERSION,
|
||||
"profile_id": "rf-detr-large-coco-704-trt11-fp16/v0",
|
||||
"source_onnx_sha256": source_sha256,
|
||||
"output_onnx_sha256": sha256_path(output),
|
||||
"output_size_bytes": output.stat().st_size,
|
||||
"boundary_types": {"inputs": input_types, "outputs": output_types},
|
||||
"initializer_type_counts": initializer_counts,
|
||||
"float_casts_retargeted_to_fp16": retargeted_casts,
|
||||
"started_utc_ns": started_utc_ns,
|
||||
"completed_utc_ns": time.time_ns(),
|
||||
"completed": True,
|
||||
"authority": FALSE_AUTHORITY,
|
||||
}
|
||||
manifest.parent.mkdir(mode=0o700, parents=True, exist_ok=True)
|
||||
manifest.write_bytes(canonical_json(document) + b"\n")
|
||||
print(output)
|
||||
print(json.dumps(document, indent=2, sort_keys=True))
|
||||
return 0
|
||||
|
||||
|
||||
def _initializer_type_counts(graph: Any) -> dict[str, int]:
|
||||
counts: dict[str, int] = {}
|
||||
for initializer in graph.graph.initializer:
|
||||
name = TensorProto.DataType.Name(initializer.data_type)
|
||||
counts[name] = counts.get(name, 0) + 1
|
||||
return dict(sorted(counts.items()))
|
||||
|
||||
|
||||
def _insert_fp32_input_cast(graph: Any) -> None:
|
||||
"""Keep a conventional FP32 client boundary before the strongly typed FP16 graph."""
|
||||
|
||||
input_value = next((item for item in graph.graph.input if item.name == "input"), None)
|
||||
if input_value is None:
|
||||
raise RuntimeError("RF-DETR graph has no input tensor named 'input'")
|
||||
if input_value.type.tensor_type.elem_type != TensorProto.FLOAT16:
|
||||
raise RuntimeError("RF-DETR converted input is not FLOAT16 before boundary adaptation")
|
||||
cast_output = "missioncore_input_fp16"
|
||||
for node in graph.graph.node:
|
||||
for index, name in enumerate(node.input):
|
||||
if name == "input":
|
||||
node.input[index] = cast_output
|
||||
cast = onnx.helper.make_node(
|
||||
"Cast",
|
||||
inputs=["input"],
|
||||
outputs=[cast_output],
|
||||
name="missioncore_input_fp32_to_fp16",
|
||||
to=TensorProto.FLOAT16,
|
||||
)
|
||||
graph.graph.node.insert(0, cast)
|
||||
input_value.type.tensor_type.elem_type = TensorProto.FLOAT
|
||||
|
||||
|
||||
def _retarget_float_casts_to_fp16(graph: Any) -> int:
|
||||
"""Retarget explicit PyTorch FLOAT casts that would re-expand an FP16 data path."""
|
||||
|
||||
count = 0
|
||||
for node in graph.graph.node:
|
||||
if node.op_type != "Cast":
|
||||
continue
|
||||
for attribute in node.attribute:
|
||||
if attribute.name == "to" and attribute.i == TensorProto.FLOAT:
|
||||
attribute.i = TensorProto.FLOAT16
|
||||
count += 1
|
||||
if count == 0:
|
||||
raise RuntimeError("RF-DETR graph has no FLOAT casts to retarget")
|
||||
return count
|
||||
|
||||
|
||||
def sha256_path(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as stream:
|
||||
for chunk in iter(lambda: stream.read(1024 * 1024), b""):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def canonical_json(value: object) -> bytes:
|
||||
return json.dumps(
|
||||
value,
|
||||
sort_keys=True,
|
||||
separators=(",", ":"),
|
||||
allow_nan=False,
|
||||
).encode("utf-8")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,147 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Export the pinned M48S RF-DETR finalist to a static ONNX artifact."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import time
|
||||
from collections.abc import Iterable
|
||||
from pathlib import Path
|
||||
from typing import Any, Final
|
||||
|
||||
import onnx # type: ignore[import-not-found]
|
||||
|
||||
SCHEMA_VERSION: Final = "missioncore.m48s-rf-detr-onnx-export/v0"
|
||||
FALSE_AUTHORITY: Final = {
|
||||
"ground_truth": False,
|
||||
"candidate_accepted": False,
|
||||
"commands_enabled": False,
|
||||
"actuation_allowed": False,
|
||||
"navigation_or_safety_accepted": False,
|
||||
}
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--checkpoint", type=Path, required=True)
|
||||
parser.add_argument("--expected-checkpoint-sha256", required=True)
|
||||
parser.add_argument("--output-root", type=Path, required=True)
|
||||
parser.add_argument("--manifest", type=Path, required=True)
|
||||
parser.add_argument("--upstream-revision", required=True)
|
||||
arguments = parser.parse_args()
|
||||
|
||||
checkpoint = arguments.checkpoint.resolve(strict=True)
|
||||
checkpoint_sha256 = sha256_path(checkpoint)
|
||||
if checkpoint_sha256 != arguments.expected_checkpoint_sha256:
|
||||
raise RuntimeError("checkpoint SHA-256 does not match the pinned finalist")
|
||||
output_root = arguments.output_root.absolute()
|
||||
manifest_path = arguments.manifest.absolute()
|
||||
if output_root.exists():
|
||||
raise RuntimeError("ONNX output root already exists")
|
||||
if manifest_path.exists():
|
||||
raise RuntimeError("ONNX export manifest already exists")
|
||||
|
||||
from rfdetr import RFDETRLarge # type: ignore[import-not-found]
|
||||
|
||||
started_utc_ns = time.time_ns()
|
||||
model = RFDETRLarge(pretrain_weights=str(checkpoint))
|
||||
exported_path = Path(
|
||||
model.export(
|
||||
output_dir=str(output_root),
|
||||
format="onnx",
|
||||
shape=(704, 704),
|
||||
batch_size=1,
|
||||
dynamic_batch=False,
|
||||
opset_version=17,
|
||||
verbose=False,
|
||||
notes={
|
||||
"missioncore_profile_id": "rf-detr-large-coco-704-fp16/v0",
|
||||
"upstream_revision": arguments.upstream_revision,
|
||||
"checkpoint_sha256": checkpoint_sha256,
|
||||
"authority": FALSE_AUTHORITY,
|
||||
},
|
||||
)
|
||||
).resolve(strict=True)
|
||||
graph = onnx.load(str(exported_path), load_external_data=False)
|
||||
onnx.checker.check_model(graph)
|
||||
inputs = [_tensor_description(value) for value in graph.graph.input]
|
||||
outputs = [_tensor_description(value) for value in graph.graph.output]
|
||||
expected_input = [{"name": "input", "element_type": 1, "shape": [1, 3, 704, 704]}]
|
||||
if inputs != expected_input:
|
||||
raise RuntimeError(f"unexpected RF-DETR ONNX input contract: {inputs}")
|
||||
if [item["name"] for item in outputs] != ["dets", "labels"]:
|
||||
raise RuntimeError(f"unexpected RF-DETR ONNX outputs: {outputs}")
|
||||
|
||||
document = {
|
||||
"schema_version": SCHEMA_VERSION,
|
||||
"profile_id": "rf-detr-large-coco-704-fp16/v0",
|
||||
"provider_id": "shadow-rf-detr-large-coco-onnx/v0",
|
||||
"upstream_revision": arguments.upstream_revision,
|
||||
"checkpoint_sha256": checkpoint_sha256,
|
||||
"onnx": {
|
||||
"path": str(exported_path),
|
||||
"sha256": sha256_path(exported_path),
|
||||
"size_bytes": exported_path.stat().st_size,
|
||||
"opset_imports": [
|
||||
{"domain": item.domain, "version": item.version}
|
||||
for item in graph.opset_import
|
||||
],
|
||||
"inputs": inputs,
|
||||
"outputs": outputs,
|
||||
},
|
||||
"started_utc_ns": started_utc_ns,
|
||||
"completed_utc_ns": time.time_ns(),
|
||||
"completed": True,
|
||||
"authority": FALSE_AUTHORITY,
|
||||
}
|
||||
manifest_path.parent.mkdir(mode=0o700, parents=True, exist_ok=True)
|
||||
manifest_path.write_bytes(canonical_json(document) + b"\n")
|
||||
print(exported_path)
|
||||
print(json.dumps(document["onnx"], indent=2, sort_keys=True))
|
||||
return 0
|
||||
|
||||
|
||||
def _tensor_description(value: Any) -> dict[str, object]:
|
||||
tensor = value.type.tensor_type
|
||||
return {
|
||||
"name": value.name,
|
||||
"element_type": tensor.elem_type,
|
||||
"shape": [_dimension_value(item) for item in tensor.shape.dim],
|
||||
}
|
||||
|
||||
|
||||
def _dimension_value(value: Any) -> int | str | None:
|
||||
if value.HasField("dim_value"):
|
||||
return int(value.dim_value)
|
||||
if value.HasField("dim_param"):
|
||||
return str(value.dim_param)
|
||||
return None
|
||||
|
||||
|
||||
def sha256_path(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as stream:
|
||||
for chunk in iter(lambda: stream.read(1024 * 1024), b""):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def canonical_json(value: object) -> bytes:
|
||||
return json.dumps(
|
||||
value,
|
||||
sort_keys=True,
|
||||
separators=(",", ":"),
|
||||
allow_nan=False,
|
||||
).encode("utf-8")
|
||||
|
||||
|
||||
def _names(values: Iterable[dict[str, object]]) -> tuple[object, ...]:
|
||||
"""Keep static analyzers honest when ONNX collections are inspected in tests."""
|
||||
|
||||
return tuple(value.get("name") for value in values)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,288 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Compare frozen road classes with all COCO classes on one YOLOX tensor pass."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import time
|
||||
from collections import Counter
|
||||
from pathlib import Path
|
||||
from typing import Any, Final
|
||||
|
||||
import numpy as np
|
||||
from PIL import Image
|
||||
|
||||
from k1link.perception.yolox_object_detector import (
|
||||
ALL_COCO_YOLOX_CONFIG,
|
||||
COCO_CLASSES,
|
||||
FROZEN_YOLOX_CONFIG,
|
||||
YOLOX_MODEL_SHA256,
|
||||
TritonHttpInferenceBackend,
|
||||
YoloxPostprocessConfig,
|
||||
load_valid_fov_mask,
|
||||
postprocess_yolox,
|
||||
preprocess_raw_kb4,
|
||||
)
|
||||
|
||||
SCHEMA: Final = "missioncore.m48s-yolox-all-coco-shadow/v0"
|
||||
POSTPROCESS_BENCHMARK_ITERATIONS: Final = 20
|
||||
AUTHORITY: Final = {
|
||||
"ground_truth": False,
|
||||
"candidate_accepted": False,
|
||||
"commands_enabled": False,
|
||||
"actuation_allowed": False,
|
||||
"navigation_or_safety_accepted": False,
|
||||
}
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--input-root", type=Path, required=True)
|
||||
parser.add_argument("--valid-fov-mask", type=Path, required=True)
|
||||
parser.add_argument("--output-root", type=Path, required=True)
|
||||
parser.add_argument(
|
||||
"--triton-endpoint",
|
||||
default="http://host.docker.internal:8000",
|
||||
)
|
||||
arguments = parser.parse_args()
|
||||
input_root = arguments.input_root.resolve(strict=True)
|
||||
images = tuple(sorted(input_root.glob("frame-*.jpg")))
|
||||
if len(images) != 11:
|
||||
raise RuntimeError("M48S all-COCO shadow requires the exact 11-frame slice")
|
||||
output_parent = arguments.output_root.expanduser().absolute()
|
||||
output_parent.mkdir(mode=0o700, parents=True, exist_ok=True)
|
||||
mask = load_valid_fov_mask(arguments.valid_fov_mask)
|
||||
backend = TritonHttpInferenceBackend(arguments.triton_endpoint)
|
||||
rows = []
|
||||
totals: Counter[str] = Counter()
|
||||
timings: dict[str, list[float]] = {
|
||||
"decode_ms": [],
|
||||
"preprocess_ms": [],
|
||||
"inference_ms": [],
|
||||
"frozen_postprocess_ms": [],
|
||||
"all_coco_postprocess_ms": [],
|
||||
"all_coco_core_ms": [],
|
||||
}
|
||||
postprocess_benchmark: dict[str, list[float]] = {
|
||||
"frozen_ms": [],
|
||||
"all_coco_ms": [],
|
||||
}
|
||||
try:
|
||||
for image_path in images:
|
||||
started = time.perf_counter_ns()
|
||||
with Image.open(image_path) as opened:
|
||||
rgb = np.asarray(opened.convert("RGB"), dtype=np.uint8)
|
||||
bgr = np.ascontiguousarray(rgb[:, :, ::-1])
|
||||
decoded = time.perf_counter_ns()
|
||||
tensor = preprocess_raw_kb4(bgr, mask, config=ALL_COCO_YOLOX_CONFIG)
|
||||
preprocessed = time.perf_counter_ns()
|
||||
output = backend.infer(tensor)
|
||||
inferred = time.perf_counter_ns()
|
||||
frozen = postprocess_yolox(output, mask, config=FROZEN_YOLOX_CONFIG)
|
||||
frozen_postprocessed = time.perf_counter_ns()
|
||||
all_coco = postprocess_yolox(output, mask, config=ALL_COCO_YOLOX_CONFIG)
|
||||
all_postprocessed = time.perf_counter_ns()
|
||||
_benchmark_postprocess(
|
||||
output,
|
||||
mask,
|
||||
destination=postprocess_benchmark,
|
||||
)
|
||||
frozen_ids = {
|
||||
_detection_identity(item.class_id, item.score, item.bbox_xyxy)
|
||||
for item in frozen.detections
|
||||
}
|
||||
added = tuple(
|
||||
item
|
||||
for item in all_coco.detections
|
||||
if _detection_identity(item.class_id, item.score, item.bbox_xyxy)
|
||||
not in frozen_ids
|
||||
)
|
||||
totals["frame_count"] += 1
|
||||
totals["frozen_detection_count"] += len(frozen.detections)
|
||||
totals["all_coco_detection_count"] += len(all_coco.detections)
|
||||
totals["added_detection_count"] += len(added)
|
||||
for item in all_coco.detections:
|
||||
totals[f"class:{item.label}"] += 1
|
||||
timings["decode_ms"].append(_milliseconds(started, decoded))
|
||||
timings["preprocess_ms"].append(_milliseconds(decoded, preprocessed))
|
||||
timings["inference_ms"].append(_milliseconds(preprocessed, inferred))
|
||||
timings["frozen_postprocess_ms"].append(
|
||||
_milliseconds(inferred, frozen_postprocessed)
|
||||
)
|
||||
timings["all_coco_postprocess_ms"].append(
|
||||
_milliseconds(frozen_postprocessed, all_postprocessed)
|
||||
)
|
||||
timings["all_coco_core_ms"].append(
|
||||
_milliseconds(decoded, preprocessed)
|
||||
+ _milliseconds(preprocessed, inferred)
|
||||
+ _milliseconds(frozen_postprocessed, all_postprocessed)
|
||||
)
|
||||
rows.append(
|
||||
{
|
||||
"frame_name": image_path.name,
|
||||
"source_sha256": _sha256(image_path),
|
||||
"frozen_detections": [
|
||||
_detection_document(item) for item in frozen.detections
|
||||
],
|
||||
"all_coco_detections": [
|
||||
_detection_document(item) for item in all_coco.detections
|
||||
],
|
||||
"added_detections": [_detection_document(item) for item in added],
|
||||
"timing_ms": {name: values[-1] for name, values in timings.items()},
|
||||
"authority": AUTHORITY,
|
||||
}
|
||||
)
|
||||
finally:
|
||||
backend.close()
|
||||
frame_bytes = b"".join(_canonical_json(item) + b"\n" for item in rows)
|
||||
timing_metrics = {name: _timing_summary(values) for name, values in timings.items()}
|
||||
mean_core_ms = timing_metrics["all_coco_core_ms"]["mean"]
|
||||
metrics = {
|
||||
"frames": {"requested": 11, "completed": totals["frame_count"]},
|
||||
"inference_passes_per_frame": 1,
|
||||
"frozen_detection_count": totals["frozen_detection_count"],
|
||||
"all_coco_detection_count": totals["all_coco_detection_count"],
|
||||
"added_detection_count": totals["added_detection_count"],
|
||||
"all_coco_class_counts": {
|
||||
key.removeprefix("class:"): value
|
||||
for key, value in sorted(totals.items())
|
||||
if key.startswith("class:")
|
||||
},
|
||||
"timing_ms": timing_metrics,
|
||||
"postprocess_benchmark": {
|
||||
"iterations_per_profile_per_frame": POSTPROCESS_BENCHMARK_ITERATIONS,
|
||||
"timing_ms": {
|
||||
name: _timing_summary(values)
|
||||
for name, values in postprocess_benchmark.items()
|
||||
},
|
||||
},
|
||||
"all_coco_core_capacity_fps": round(1000.0 / mean_core_ms, 6),
|
||||
"authority": AUTHORITY,
|
||||
}
|
||||
identity = {
|
||||
"schema_version": SCHEMA,
|
||||
"model_sha256": YOLOX_MODEL_SHA256,
|
||||
"class_count": len(COCO_CLASSES),
|
||||
"frozen_target_class_ids": list(FROZEN_YOLOX_CONFIG.target_class_ids),
|
||||
"all_coco_target_class_ids": list(ALL_COCO_YOLOX_CONFIG.target_class_ids),
|
||||
"valid_fov_mask_sha256": _sha256(arguments.valid_fov_mask),
|
||||
"producer_sha256": _sha256(Path(__file__)),
|
||||
"frames_sha256": hashlib.sha256(frame_bytes).hexdigest(),
|
||||
"metrics": metrics,
|
||||
"completed": totals["frame_count"] == 11,
|
||||
"accepted": False,
|
||||
"authority": AUTHORITY,
|
||||
}
|
||||
result_id = "m48s-yolox-all-coco-shadow-" + hashlib.sha256(
|
||||
_canonical_json(identity)
|
||||
).hexdigest()
|
||||
destination = output_parent / result_id
|
||||
if destination.exists():
|
||||
raise RuntimeError("immutable M48S all-COCO result already exists")
|
||||
destination.mkdir(mode=0o700)
|
||||
(destination / "frames.jsonl").write_bytes(frame_bytes)
|
||||
(destination / "manifest.json").write_bytes(
|
||||
_canonical_json({"result_id": result_id, **identity}) + b"\n"
|
||||
)
|
||||
(destination / "report.json").write_bytes(
|
||||
_canonical_json(
|
||||
{
|
||||
"schema_version": SCHEMA,
|
||||
"result_id": result_id,
|
||||
"completed": identity["completed"],
|
||||
"accepted": False,
|
||||
"metrics": metrics,
|
||||
"decision": {
|
||||
"all_coco_emission_completed": True,
|
||||
"additional_inference_passes": 0,
|
||||
"navigation_or_safety_accepted": False,
|
||||
"next_gate": "full-load all-COCO detector replay",
|
||||
},
|
||||
"authority": AUTHORITY,
|
||||
}
|
||||
)
|
||||
+ b"\n"
|
||||
)
|
||||
print(result_id)
|
||||
print(json.dumps(metrics, indent=2, sort_keys=True))
|
||||
return 0
|
||||
|
||||
|
||||
def _detection_identity(
|
||||
class_id: int,
|
||||
score: float,
|
||||
box: tuple[float, float, float, float],
|
||||
) -> tuple[int, float, tuple[float, float, float, float]]:
|
||||
return class_id, score, box
|
||||
|
||||
|
||||
def _benchmark_postprocess(
|
||||
output: np.ndarray[Any, Any],
|
||||
mask: np.ndarray[Any, Any],
|
||||
*,
|
||||
destination: dict[str, list[float]],
|
||||
) -> None:
|
||||
for iteration in range(POSTPROCESS_BENCHMARK_ITERATIONS):
|
||||
profiles: tuple[tuple[str, YoloxPostprocessConfig], ...]
|
||||
if iteration % 2:
|
||||
profiles = (
|
||||
("frozen_ms", FROZEN_YOLOX_CONFIG),
|
||||
("all_coco_ms", ALL_COCO_YOLOX_CONFIG),
|
||||
)
|
||||
else:
|
||||
profiles = (
|
||||
("all_coco_ms", ALL_COCO_YOLOX_CONFIG),
|
||||
("frozen_ms", FROZEN_YOLOX_CONFIG),
|
||||
)
|
||||
for name, profile in profiles:
|
||||
started = time.perf_counter_ns()
|
||||
postprocess_yolox(output, mask, config=profile)
|
||||
completed = time.perf_counter_ns()
|
||||
destination[name].append(_milliseconds(started, completed))
|
||||
|
||||
|
||||
def _detection_document(item: Any) -> dict[str, object]:
|
||||
return {
|
||||
"class_id": item.class_id,
|
||||
"label": item.label,
|
||||
"score": item.score,
|
||||
"bbox_xyxy": list(item.bbox_xyxy),
|
||||
"valid_fov_fraction": item.valid_fov_fraction,
|
||||
}
|
||||
|
||||
|
||||
def _milliseconds(started: int, completed: int) -> float:
|
||||
return round(max(0, completed - started) / 1_000_000.0, 6)
|
||||
|
||||
|
||||
def _timing_summary(values: list[float]) -> dict[str, float]:
|
||||
array = np.asarray(values, dtype=np.float64)
|
||||
return {
|
||||
"mean": round(float(array.mean()), 6),
|
||||
"p50": round(float(np.percentile(array, 50)), 6),
|
||||
"p95": round(float(np.percentile(array, 95)), 6),
|
||||
"max": round(float(array.max()), 6),
|
||||
}
|
||||
|
||||
|
||||
def _sha256(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as stream:
|
||||
for chunk in iter(lambda: stream.read(1024 * 1024), b""):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def _canonical_json(value: object) -> bytes:
|
||||
return json.dumps(
|
||||
value,
|
||||
sort_keys=True,
|
||||
separators=(",", ":"),
|
||||
allow_nan=False,
|
||||
).encode("utf-8")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,546 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Run one fixed-class detector candidate on the exact M48S risk slice."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from collections import Counter
|
||||
from pathlib import Path
|
||||
from typing import Any, Final, Protocol, cast
|
||||
|
||||
import numpy as np
|
||||
import torch # type: ignore[import-not-found]
|
||||
from PIL import Image, ImageDraw
|
||||
|
||||
WORKER_RUN_SCHEMA: Final = "missioncore.m48s-fixed-detector-candidate-worker/v0"
|
||||
EXACT_FRAME_NAMES: Final = (
|
||||
"frame-000121.png",
|
||||
"frame-000131.png",
|
||||
"frame-000253.png",
|
||||
"frame-000275.png",
|
||||
"frame-000443.png",
|
||||
"frame-000463.png",
|
||||
"frame-001094.png",
|
||||
"frame-001228.png",
|
||||
"frame-001454.png",
|
||||
"frame-001856.png",
|
||||
"frame-002386.png",
|
||||
)
|
||||
COCO_CLASSES: Final = (
|
||||
"person",
|
||||
"bicycle",
|
||||
"car",
|
||||
"motorcycle",
|
||||
"airplane",
|
||||
"bus",
|
||||
"train",
|
||||
"truck",
|
||||
"boat",
|
||||
"traffic light",
|
||||
"fire hydrant",
|
||||
"stop sign",
|
||||
"parking meter",
|
||||
"bench",
|
||||
"bird",
|
||||
"cat",
|
||||
"dog",
|
||||
"horse",
|
||||
"sheep",
|
||||
"cow",
|
||||
"elephant",
|
||||
"bear",
|
||||
"zebra",
|
||||
"giraffe",
|
||||
"backpack",
|
||||
"umbrella",
|
||||
"handbag",
|
||||
"tie",
|
||||
"suitcase",
|
||||
"frisbee",
|
||||
"skis",
|
||||
"snowboard",
|
||||
"sports ball",
|
||||
"kite",
|
||||
"baseball bat",
|
||||
"baseball glove",
|
||||
"skateboard",
|
||||
"surfboard",
|
||||
"tennis racket",
|
||||
"bottle",
|
||||
"wine glass",
|
||||
"cup",
|
||||
"fork",
|
||||
"knife",
|
||||
"spoon",
|
||||
"bowl",
|
||||
"banana",
|
||||
"apple",
|
||||
"sandwich",
|
||||
"orange",
|
||||
"broccoli",
|
||||
"carrot",
|
||||
"hot dog",
|
||||
"pizza",
|
||||
"donut",
|
||||
"cake",
|
||||
"chair",
|
||||
"couch",
|
||||
"potted plant",
|
||||
"bed",
|
||||
"dining table",
|
||||
"toilet",
|
||||
"tv",
|
||||
"laptop",
|
||||
"mouse",
|
||||
"remote",
|
||||
"keyboard",
|
||||
"cell phone",
|
||||
"microwave",
|
||||
"oven",
|
||||
"toaster",
|
||||
"sink",
|
||||
"refrigerator",
|
||||
"book",
|
||||
"clock",
|
||||
"vase",
|
||||
"scissors",
|
||||
"teddy bear",
|
||||
"hair drier",
|
||||
"toothbrush",
|
||||
)
|
||||
AUTHORITY: Final = {
|
||||
"ground_truth": False,
|
||||
"candidate_accepted": False,
|
||||
"commands_enabled": False,
|
||||
"actuation_allowed": False,
|
||||
"navigation_or_safety_accepted": False,
|
||||
}
|
||||
MINIMUM_BOX_AREA_PIXELS: Final = 64.0
|
||||
MAXIMUM_BOX_AREA_FRACTION: Final = 0.5
|
||||
MINIMUM_VALID_FOV_FRACTION: Final = 0.5
|
||||
OVERLAY_THRESHOLD: Final = 0.25
|
||||
|
||||
|
||||
class Detector(Protocol):
|
||||
def infer(self, image: Image.Image) -> tuple[RawDetection, ...]: ...
|
||||
|
||||
|
||||
class RawDetection(tuple[int, str, float, tuple[float, float, float, float]]):
|
||||
"""Normalized detector output: class id, label, score and source-pixel box."""
|
||||
|
||||
__slots__ = ()
|
||||
|
||||
def __new__(
|
||||
cls,
|
||||
class_id: int,
|
||||
label: str,
|
||||
score: float,
|
||||
box: tuple[float, float, float, float],
|
||||
) -> RawDetection:
|
||||
return tuple.__new__(cls, (class_id, label, score, box))
|
||||
|
||||
@property
|
||||
def class_id(self) -> int:
|
||||
return self[0]
|
||||
|
||||
@property
|
||||
def label(self) -> str:
|
||||
return self[1]
|
||||
|
||||
@property
|
||||
def score(self) -> float:
|
||||
return self[2]
|
||||
|
||||
@property
|
||||
def box(self) -> tuple[float, float, float, float]:
|
||||
return self[3]
|
||||
|
||||
|
||||
class DfineDetector:
|
||||
"""Pinned D-FINE-S COCO PyTorch qualification adapter."""
|
||||
|
||||
def __init__(self, source_root: Path, config_path: Path, checkpoint: Path) -> None:
|
||||
sys.path.insert(0, str(source_root))
|
||||
from src.core import YAMLConfig # type: ignore[import-not-found]
|
||||
|
||||
config = YAMLConfig(str(config_path), resume=str(checkpoint))
|
||||
if "HGNetv2" in config.yaml_cfg:
|
||||
config.yaml_cfg["HGNetv2"]["pretrained"] = False
|
||||
state = torch.load(checkpoint, map_location="cpu", weights_only=True)
|
||||
weights = state["ema"]["module"] if "ema" in state else state["model"]
|
||||
config.model.load_state_dict(weights)
|
||||
self._model = config.model.deploy().to("cuda").eval()
|
||||
self._postprocessor = config.postprocessor.deploy()
|
||||
|
||||
def infer(self, image: Image.Image) -> tuple[RawDetection, ...]:
|
||||
tensor, ratio, padding = _dfine_preprocess(image)
|
||||
with torch.inference_mode(), torch.autocast("cuda", dtype=torch.float16):
|
||||
output = self._model(tensor)
|
||||
size = torch.tensor([[640, 640]], device="cuda")
|
||||
labels, boxes, scores = self._postprocessor(output, size)
|
||||
labels_array = labels[0].detach().to("cpu").numpy()
|
||||
boxes_array = boxes[0].detach().to("cpu").numpy()
|
||||
scores_array = scores[0].detach().to("cpu").numpy()
|
||||
pad_x, pad_y = padding
|
||||
detections = []
|
||||
for raw_label, raw_score, raw_box in zip(
|
||||
labels_array,
|
||||
scores_array,
|
||||
boxes_array,
|
||||
strict=True,
|
||||
):
|
||||
class_id = int(raw_label)
|
||||
if not 0 <= class_id < len(COCO_CLASSES):
|
||||
continue
|
||||
box = (
|
||||
(float(raw_box[0]) - pad_x) / ratio,
|
||||
(float(raw_box[1]) - pad_y) / ratio,
|
||||
(float(raw_box[2]) - pad_x) / ratio,
|
||||
(float(raw_box[3]) - pad_y) / ratio,
|
||||
)
|
||||
detections.append(
|
||||
RawDetection(class_id, COCO_CLASSES[class_id], float(raw_score), box)
|
||||
)
|
||||
return tuple(detections)
|
||||
|
||||
|
||||
class RfDetrDetector:
|
||||
"""Pinned RF-DETR-L COCO PyTorch qualification adapter."""
|
||||
|
||||
def __init__(self, checkpoint: Path) -> None:
|
||||
from rfdetr import RFDETRLarge # type: ignore[import-not-found]
|
||||
|
||||
self._model = RFDETRLarge(pretrain_weights=str(checkpoint))
|
||||
self._model.inference(compile=False, dtype=torch.float16, inplace=True)
|
||||
|
||||
def infer(self, image: Image.Image) -> tuple[RawDetection, ...]:
|
||||
prediction = self._model.predict(
|
||||
image,
|
||||
threshold=0.1,
|
||||
include_source_image=False,
|
||||
)
|
||||
boxes = np.asarray(prediction.xyxy)
|
||||
scores = np.asarray(prediction.confidence)
|
||||
names = np.asarray(prediction.data["class_name"])
|
||||
detections = []
|
||||
for raw_name, raw_score, raw_box in zip(names, scores, boxes, strict=True):
|
||||
label = str(raw_name)
|
||||
try:
|
||||
class_id = COCO_CLASSES.index(label)
|
||||
except ValueError:
|
||||
continue
|
||||
detections.append(
|
||||
RawDetection(
|
||||
class_id,
|
||||
label,
|
||||
float(raw_score),
|
||||
cast(
|
||||
tuple[float, float, float, float],
|
||||
tuple(float(value) for value in raw_box),
|
||||
),
|
||||
)
|
||||
)
|
||||
return tuple(detections)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument(
|
||||
"--candidate",
|
||||
choices=("dfine-s-coco", "rf-detr-large-coco"),
|
||||
required=True,
|
||||
)
|
||||
parser.add_argument("--profile-id", required=True)
|
||||
parser.add_argument("--provider-id", required=True)
|
||||
parser.add_argument("--upstream-revision", required=True)
|
||||
parser.add_argument("--input-root", type=Path, required=True)
|
||||
parser.add_argument("--valid-fov-mask", type=Path, required=True)
|
||||
parser.add_argument("--checkpoint", type=Path, required=True)
|
||||
parser.add_argument("--expected-checkpoint-sha256", required=True)
|
||||
parser.add_argument("--output", type=Path, required=True)
|
||||
parser.add_argument("--dfine-root", type=Path, default=Path("/opt/dfine"))
|
||||
parser.add_argument(
|
||||
"--dfine-config",
|
||||
type=Path,
|
||||
default=Path("/opt/dfine/configs/dfine/dfine_hgnetv2_s_coco.yml"),
|
||||
)
|
||||
parser.add_argument("--warmup-iterations", type=int, default=5)
|
||||
parser.add_argument("--benchmark-iterations", type=int, default=30)
|
||||
arguments = parser.parse_args()
|
||||
if arguments.warmup_iterations < 1 or arguments.benchmark_iterations < 1:
|
||||
raise RuntimeError("warmup and benchmark iterations must be positive")
|
||||
input_root = arguments.input_root.resolve(strict=True)
|
||||
images = tuple(sorted(input_root.glob("frame-*.png")))
|
||||
if tuple(path.name for path in images) != EXACT_FRAME_NAMES:
|
||||
raise RuntimeError("candidate Worker requires the exact M48S risk slice")
|
||||
mask = _load_mask(arguments.valid_fov_mask.resolve(strict=True))
|
||||
checkpoint = arguments.checkpoint.resolve(strict=True)
|
||||
checkpoint_sha256 = _sha256(checkpoint)
|
||||
if checkpoint_sha256 != arguments.expected_checkpoint_sha256:
|
||||
raise RuntimeError("checkpoint SHA-256 does not match the pinned profile")
|
||||
output = arguments.output.absolute()
|
||||
if output.exists():
|
||||
raise RuntimeError("candidate Worker output already exists")
|
||||
output.parent.mkdir(mode=0o700, parents=True, exist_ok=True)
|
||||
overlay_root = output.parent / f"{output.stem}-overlays"
|
||||
overlay_root.mkdir(mode=0o700)
|
||||
gpu_before = _gpu_sample()
|
||||
started_utc_ns = time.time_ns()
|
||||
torch.cuda.reset_peak_memory_stats()
|
||||
detector: Detector
|
||||
if arguments.candidate == "dfine-s-coco":
|
||||
detector = DfineDetector(
|
||||
arguments.dfine_root.resolve(strict=True),
|
||||
arguments.dfine_config.resolve(strict=True),
|
||||
checkpoint,
|
||||
)
|
||||
else:
|
||||
detector = RfDetrDetector(checkpoint)
|
||||
with Image.open(images[0]) as opened:
|
||||
warmup_image = opened.convert("RGB")
|
||||
for _ in range(arguments.warmup_iterations):
|
||||
detector.infer(warmup_image)
|
||||
_synchronize()
|
||||
frames = []
|
||||
totals: Counter[str] = Counter()
|
||||
evidence_timings = []
|
||||
for image_path in images:
|
||||
with Image.open(image_path) as opened:
|
||||
image = opened.convert("RGB")
|
||||
_synchronize()
|
||||
started = time.perf_counter_ns()
|
||||
raw = detector.infer(image)
|
||||
_synchronize()
|
||||
completed = time.perf_counter_ns()
|
||||
elapsed_ms = _milliseconds(started, completed)
|
||||
evidence_timings.append(elapsed_ms)
|
||||
detections = _qualify(raw, mask, image.size)
|
||||
totals["frame_count"] += 1
|
||||
totals["detection_count"] += len(detections)
|
||||
for detection in detections:
|
||||
totals[f"class:{detection['label']}"] += 1
|
||||
frames.append(
|
||||
{
|
||||
"frame_name": image_path.name,
|
||||
"source_sha256": _sha256(image_path),
|
||||
"detections": detections,
|
||||
"timing_ms": {"end_to_end": elapsed_ms},
|
||||
"authority": AUTHORITY,
|
||||
}
|
||||
)
|
||||
_write_overlay(image, detections, overlay_root / image_path.name)
|
||||
with Image.open(input_root / "frame-000253.png") as opened:
|
||||
benchmark_image = opened.convert("RGB")
|
||||
benchmark_timings = []
|
||||
for _ in range(arguments.benchmark_iterations):
|
||||
_synchronize()
|
||||
started = time.perf_counter_ns()
|
||||
detector.infer(benchmark_image)
|
||||
_synchronize()
|
||||
benchmark_timings.append(_milliseconds(started, time.perf_counter_ns()))
|
||||
gpu_after = _gpu_sample()
|
||||
timing = _timing_summary(evidence_timings)
|
||||
benchmark_timing = _timing_summary(benchmark_timings)
|
||||
metrics = {
|
||||
"frames": {"requested": 11, "completed": totals["frame_count"]},
|
||||
"detection_count_at_minimum_score_0_1": totals["detection_count"],
|
||||
"class_counts_at_minimum_score_0_1": {
|
||||
key.removeprefix("class:"): value
|
||||
for key, value in sorted(totals.items())
|
||||
if key.startswith("class:")
|
||||
},
|
||||
"evidence_timing_ms": timing,
|
||||
"benchmark": {
|
||||
"frame_name": "frame-000253.png",
|
||||
"iterations": arguments.benchmark_iterations,
|
||||
"timing_ms": benchmark_timing,
|
||||
"core_capacity_fps": round(1000.0 / benchmark_timing["mean"], 6),
|
||||
},
|
||||
"torch_peak_memory": {
|
||||
"allocated_bytes": torch.cuda.max_memory_allocated(),
|
||||
"reserved_bytes": torch.cuda.max_memory_reserved(),
|
||||
},
|
||||
"gpu_before": gpu_before,
|
||||
"gpu_after": gpu_after,
|
||||
}
|
||||
document = {
|
||||
"schema_version": WORKER_RUN_SCHEMA,
|
||||
"profile_id": arguments.profile_id,
|
||||
"provider_id": arguments.provider_id,
|
||||
"candidate": arguments.candidate,
|
||||
"upstream_revision": arguments.upstream_revision,
|
||||
"checkpoint_sha256": checkpoint_sha256,
|
||||
"started_utc_ns": started_utc_ns,
|
||||
"completed_utc_ns": time.time_ns(),
|
||||
"completed": totals["frame_count"] == 11,
|
||||
"execution": {
|
||||
"worker_id": "worker-006",
|
||||
"device": torch.cuda.get_device_name(0),
|
||||
"precision": "fp16",
|
||||
"inference_passes_per_evidence_frame": 1,
|
||||
"warmup_iterations": arguments.warmup_iterations,
|
||||
"benchmark_iterations": arguments.benchmark_iterations,
|
||||
"concurrent_services_retained": True,
|
||||
},
|
||||
"frames": frames,
|
||||
"metrics": metrics,
|
||||
"authority": AUTHORITY,
|
||||
}
|
||||
output.write_bytes(_canonical_json(document) + b"\n")
|
||||
print(output)
|
||||
print(json.dumps(metrics, indent=2, sort_keys=True))
|
||||
return 0
|
||||
|
||||
|
||||
def _dfine_preprocess(
|
||||
image: Image.Image,
|
||||
) -> tuple[torch.Tensor, float, tuple[int, int]]:
|
||||
width, height = image.size
|
||||
ratio = min(640.0 / width, 640.0 / height)
|
||||
resized_width = int(width * ratio)
|
||||
resized_height = int(height * ratio)
|
||||
resized = image.resize((resized_width, resized_height), Image.Resampling.BILINEAR)
|
||||
padded = Image.new("RGB", (640, 640))
|
||||
pad_x = (640 - resized_width) // 2
|
||||
pad_y = (640 - resized_height) // 2
|
||||
padded.paste(resized, (pad_x, pad_y))
|
||||
array = np.asarray(padded, dtype=np.float32) / 255.0
|
||||
tensor = torch.from_numpy(np.ascontiguousarray(array.transpose(2, 0, 1)))
|
||||
return tensor.unsqueeze(0).to("cuda", non_blocking=True), ratio, (pad_x, pad_y)
|
||||
|
||||
|
||||
def _qualify(
|
||||
detections: tuple[RawDetection, ...],
|
||||
mask: np.ndarray[Any, Any],
|
||||
image_size: tuple[int, int],
|
||||
) -> list[dict[str, object]]:
|
||||
width, height = image_size
|
||||
image_area = float(width * height)
|
||||
qualified = []
|
||||
for detection in detections:
|
||||
if detection.score < 0.1:
|
||||
continue
|
||||
x1, y1, x2, y2 = detection.box
|
||||
x1 = max(0.0, min(float(width), x1))
|
||||
y1 = max(0.0, min(float(height), y1))
|
||||
x2 = max(0.0, min(float(width), x2))
|
||||
y2 = max(0.0, min(float(height), y2))
|
||||
area = max(0.0, x2 - x1) * max(0.0, y2 - y1)
|
||||
if area < MINIMUM_BOX_AREA_PIXELS or area > MAXIMUM_BOX_AREA_FRACTION * image_area:
|
||||
continue
|
||||
center_x = min(width - 1, max(0, int((x1 + x2) / 2.0)))
|
||||
center_y = min(height - 1, max(0, int((y1 + y2) / 2.0)))
|
||||
if not bool(mask[center_y, center_x]):
|
||||
continue
|
||||
ix1 = min(width - 1, max(0, int(np.floor(x1))))
|
||||
iy1 = min(height - 1, max(0, int(np.floor(y1))))
|
||||
ix2 = min(width, max(ix1 + 1, int(np.ceil(x2))))
|
||||
iy2 = min(height, max(iy1 + 1, int(np.ceil(y2))))
|
||||
valid_fraction = float(mask[iy1:iy2, ix1:ix2].mean())
|
||||
if valid_fraction < MINIMUM_VALID_FOV_FRACTION:
|
||||
continue
|
||||
qualified.append(
|
||||
{
|
||||
"class_id": detection.class_id,
|
||||
"label": detection.label,
|
||||
"score": round(detection.score, 6),
|
||||
"bbox_xyxy": [round(value, 3) for value in (x1, y1, x2, y2)],
|
||||
"valid_fov_fraction": round(valid_fraction, 6),
|
||||
}
|
||||
)
|
||||
qualified.sort(key=lambda item: (-cast(float, item["score"]), cast(int, item["class_id"])))
|
||||
return qualified
|
||||
|
||||
|
||||
def _write_overlay(
|
||||
image: Image.Image,
|
||||
detections: list[dict[str, object]],
|
||||
destination: Path,
|
||||
) -> None:
|
||||
annotated = image.copy()
|
||||
draw = ImageDraw.Draw(annotated)
|
||||
for detection in detections:
|
||||
score = cast(float, detection["score"])
|
||||
if score < OVERLAY_THRESHOLD:
|
||||
continue
|
||||
box = cast(list[float], detection["bbox_xyxy"])
|
||||
label = cast(str, detection["label"])
|
||||
draw.rectangle(box, outline=(255, 84, 0), width=3)
|
||||
draw.text((box[0] + 3, box[1] + 3), f"{label} {score:.2f}", fill=(255, 255, 255))
|
||||
annotated.save(destination)
|
||||
|
||||
|
||||
def _load_mask(path: Path) -> np.ndarray[Any, Any]:
|
||||
with Image.open(path) as image:
|
||||
array = np.asarray(image.convert("L"), dtype=np.uint8)
|
||||
if array.shape != (600, 800):
|
||||
raise RuntimeError("valid-FOV mask must be 800x600")
|
||||
return array > 0
|
||||
|
||||
|
||||
def _gpu_sample() -> dict[str, object]:
|
||||
completed = subprocess.run(
|
||||
[
|
||||
"nvidia-smi",
|
||||
"--query-gpu=name,memory.total,memory.used,utilization.gpu,temperature.gpu,power.draw",
|
||||
"--format=csv,noheader,nounits",
|
||||
],
|
||||
check=True,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
values = [value.strip() for value in completed.stdout.strip().split(",")]
|
||||
if len(values) != 6:
|
||||
raise RuntimeError("unexpected nvidia-smi response")
|
||||
return {
|
||||
"name": values[0],
|
||||
"memory_total_mib": float(values[1]),
|
||||
"memory_used_mib": float(values[2]),
|
||||
"utilization_gpu_percent": float(values[3]),
|
||||
"temperature_c": float(values[4]),
|
||||
"power_w": float(values[5]),
|
||||
}
|
||||
|
||||
|
||||
def _synchronize() -> None:
|
||||
torch.cuda.synchronize()
|
||||
|
||||
|
||||
def _timing_summary(values: list[float]) -> dict[str, float]:
|
||||
array = np.asarray(values, dtype=np.float64)
|
||||
return {
|
||||
"mean": round(float(array.mean()), 6),
|
||||
"p50": round(float(np.percentile(array, 50)), 6),
|
||||
"p95": round(float(np.percentile(array, 95)), 6),
|
||||
"max": round(float(array.max()), 6),
|
||||
}
|
||||
|
||||
|
||||
def _milliseconds(started: int, completed: int) -> float:
|
||||
return round(max(0, completed - started) / 1_000_000.0, 6)
|
||||
|
||||
|
||||
def _sha256(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as stream:
|
||||
for chunk in iter(lambda: stream.read(1024 * 1024), b""):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def _canonical_json(value: object) -> bytes:
|
||||
return json.dumps(
|
||||
value,
|
||||
sort_keys=True,
|
||||
separators=(",", ":"),
|
||||
allow_nan=False,
|
||||
).encode("utf-8")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,417 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Run a source-paced bounded-queue RF-DETR/Triton stability qualification."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import resource
|
||||
import subprocess
|
||||
import threading
|
||||
import time
|
||||
from collections import Counter, deque
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any, Final
|
||||
|
||||
import av # type: ignore[import-not-found]
|
||||
import numpy as np
|
||||
import run_rf_detr_triton as qualifier # type: ignore[import-not-found]
|
||||
import tritonclient.http as httpclient # type: ignore[import-not-found]
|
||||
from PIL import Image
|
||||
|
||||
SCHEMA_VERSION: Final = "missioncore.m48s-rf-detr-source-paced-load/v0"
|
||||
SOURCE_SHA256: Final = "cadd1696ff000904eb78633a0a8418104b8024f178b91f3421789021ccb160e8"
|
||||
SOURCE_FPS: Final = 10.003944527024467
|
||||
AUTHORITY: Final = {
|
||||
"ground_truth": False,
|
||||
"candidate_accepted": False,
|
||||
"commands_enabled": False,
|
||||
"actuation_allowed": False,
|
||||
"navigation_or_safety_accepted": False,
|
||||
}
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class SourceFrame:
|
||||
sequence: int
|
||||
scheduled_ns: int
|
||||
image: Image.Image
|
||||
|
||||
|
||||
class LatestWinsQueue:
|
||||
"""Bounded source queue that reports every replacement explicitly."""
|
||||
|
||||
def __init__(self, capacity: int) -> None:
|
||||
if capacity < 1:
|
||||
raise ValueError("queue capacity must be positive")
|
||||
self.capacity = capacity
|
||||
self._items: deque[SourceFrame] = deque()
|
||||
self._condition = threading.Condition()
|
||||
self._closed = False
|
||||
self.replacements = 0
|
||||
self.maximum_depth = 0
|
||||
|
||||
def put(self, item: SourceFrame) -> None:
|
||||
with self._condition:
|
||||
if self._closed:
|
||||
return
|
||||
if len(self._items) == self.capacity:
|
||||
self._items.popleft()
|
||||
self.replacements += 1
|
||||
self._items.append(item)
|
||||
self.maximum_depth = max(self.maximum_depth, len(self._items))
|
||||
self._condition.notify()
|
||||
|
||||
def get(self) -> SourceFrame | None:
|
||||
with self._condition:
|
||||
while not self._items and not self._closed:
|
||||
self._condition.wait(timeout=1.0)
|
||||
if self._items:
|
||||
return self._items.popleft()
|
||||
return None
|
||||
|
||||
def close(self) -> None:
|
||||
with self._condition:
|
||||
self._closed = True
|
||||
self._condition.notify_all()
|
||||
|
||||
|
||||
class GpuTelemetry:
|
||||
def __init__(self, interval_seconds: float) -> None:
|
||||
self.interval_seconds = interval_seconds
|
||||
self.samples: list[dict[str, float]] = []
|
||||
self._stop = threading.Event()
|
||||
self._thread = threading.Thread(target=self._run, daemon=True)
|
||||
|
||||
def __enter__(self) -> GpuTelemetry:
|
||||
self._thread.start()
|
||||
return self
|
||||
|
||||
def __exit__(self, *_args: object) -> None:
|
||||
self._stop.set()
|
||||
self._thread.join(timeout=10.0)
|
||||
|
||||
def _run(self) -> None:
|
||||
while not self._stop.is_set():
|
||||
try:
|
||||
completed = subprocess.run(
|
||||
[
|
||||
"nvidia-smi",
|
||||
"--query-gpu=utilization.gpu,memory.used,power.draw,temperature.gpu",
|
||||
"--format=csv,noheader,nounits",
|
||||
],
|
||||
check=True,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=10.0,
|
||||
)
|
||||
values = [float(value.strip()) for value in completed.stdout.split(",")]
|
||||
if len(values) == 4:
|
||||
self.samples.append(
|
||||
{
|
||||
"gpu_utilization_percent": values[0],
|
||||
"gpu_memory_used_mib": values[1],
|
||||
"gpu_power_w": values[2],
|
||||
"gpu_temperature_c": values[3],
|
||||
}
|
||||
)
|
||||
except (OSError, ValueError, subprocess.SubprocessError):
|
||||
pass
|
||||
self._stop.wait(self.interval_seconds)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--source-video", type=Path, required=True)
|
||||
parser.add_argument("--valid-fov-mask", type=Path, required=True)
|
||||
parser.add_argument("--endpoint", default="localhost:8100")
|
||||
parser.add_argument("--model-name", default="rf_detr_large")
|
||||
parser.add_argument("--duration-seconds", type=float, default=1800.0)
|
||||
parser.add_argument("--queue-capacity", type=int, default=2)
|
||||
parser.add_argument("--telemetry-interval-seconds", type=float, default=1.0)
|
||||
parser.add_argument("--output", type=Path, required=True)
|
||||
parser.add_argument("--progress", type=Path, required=True)
|
||||
arguments = parser.parse_args()
|
||||
if arguments.duration_seconds <= 0:
|
||||
raise RuntimeError("duration must be positive")
|
||||
if arguments.telemetry_interval_seconds <= 0:
|
||||
raise RuntimeError("telemetry interval must be positive")
|
||||
source = arguments.source_video.resolve(strict=True)
|
||||
if _sha256(source) != SOURCE_SHA256:
|
||||
raise RuntimeError("RAVNOVES00 camera stream identity changed")
|
||||
mask = qualifier._load_mask(arguments.valid_fov_mask.resolve(strict=True))
|
||||
output = arguments.output.absolute()
|
||||
progress = arguments.progress.absolute()
|
||||
if output.exists() or progress.exists():
|
||||
raise RuntimeError("load result or progress artifact already exists")
|
||||
output.parent.mkdir(mode=0o700, parents=True, exist_ok=True)
|
||||
|
||||
client = httpclient.InferenceServerClient(arguments.endpoint, concurrency=1)
|
||||
if not client.is_server_ready() or not client.is_model_ready(arguments.model_name):
|
||||
raise RuntimeError("isolated Triton or RF-DETR model is not ready")
|
||||
metadata = client.get_model_metadata(arguments.model_name)
|
||||
qualifier._validate_metadata(metadata)
|
||||
triton_before = client.get_inference_statistics(arguments.model_name)
|
||||
gpu_before = qualifier._gpu_sample()
|
||||
rss_before_kib = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
|
||||
queue = LatestWinsQueue(arguments.queue_capacity)
|
||||
producer_stop = threading.Event()
|
||||
producer_errors: list[str] = []
|
||||
produced_count = [0]
|
||||
source_loops = [0]
|
||||
started_ns = time.monotonic_ns()
|
||||
started_utc_ns = time.time_ns()
|
||||
producer = threading.Thread(
|
||||
target=_produce,
|
||||
args=(
|
||||
source,
|
||||
queue,
|
||||
producer_stop,
|
||||
producer_errors,
|
||||
produced_count,
|
||||
source_loops,
|
||||
started_ns,
|
||||
arguments.duration_seconds,
|
||||
),
|
||||
daemon=True,
|
||||
)
|
||||
consumed = 0
|
||||
failures = 0
|
||||
class_counts: Counter[str] = Counter()
|
||||
end_to_end_ms: list[float] = []
|
||||
triton_round_trip_ms: list[float] = []
|
||||
completion_age_ms: list[float] = []
|
||||
last_progress_ns = started_ns
|
||||
producer.start()
|
||||
with progress.open("x", encoding="utf-8") as progress_stream, GpuTelemetry(
|
||||
arguments.telemetry_interval_seconds
|
||||
) as telemetry:
|
||||
try:
|
||||
while True:
|
||||
item = queue.get()
|
||||
if item is None:
|
||||
break
|
||||
raw, timing = qualifier._infer(client, arguments.model_name, item.image)
|
||||
detections = qualifier._qualify(raw, mask, item.image.size)
|
||||
for detection in detections:
|
||||
if float(detection["score"]) >= 0.5:
|
||||
class_counts[str(detection["label"])] += 1
|
||||
consumed += 1
|
||||
end_to_end_ms.append(timing["end_to_end"])
|
||||
triton_round_trip_ms.append(timing["triton_round_trip"])
|
||||
completion_age_ms.append((time.monotonic_ns() - item.scheduled_ns) / 1_000_000.0)
|
||||
now_ns = time.monotonic_ns()
|
||||
if now_ns - last_progress_ns >= 60_000_000_000:
|
||||
row = {
|
||||
"elapsed_seconds": round((now_ns - started_ns) / 1_000_000_000.0, 3),
|
||||
"produced": produced_count[0],
|
||||
"consumed": consumed,
|
||||
"replacements": queue.replacements,
|
||||
"completion_age_p95_ms": _distribution(completion_age_ms)["p95"],
|
||||
}
|
||||
progress_stream.write(json.dumps(row, separators=(",", ":")) + "\n")
|
||||
progress_stream.flush()
|
||||
print(json.dumps(row, sort_keys=True), flush=True)
|
||||
last_progress_ns = now_ns
|
||||
except BaseException:
|
||||
failures += 1
|
||||
raise
|
||||
finally:
|
||||
producer_stop.set()
|
||||
queue.close()
|
||||
producer.join(timeout=15.0)
|
||||
|
||||
completed_ns = time.monotonic_ns()
|
||||
wall_seconds = (completed_ns - started_ns) / 1_000_000_000.0
|
||||
if producer.is_alive():
|
||||
raise RuntimeError("source producer did not stop")
|
||||
if producer_errors:
|
||||
raise RuntimeError(f"source producer failed: {producer_errors}")
|
||||
triton_after = client.get_inference_statistics(arguments.model_name)
|
||||
gpu_after = qualifier._gpu_sample()
|
||||
rss_after_kib = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
|
||||
core = _distribution(end_to_end_ms)
|
||||
completion_age = _distribution(completion_age_ms)
|
||||
telemetry_summary = _telemetry_summary(telemetry.samples)
|
||||
checks = {
|
||||
"minimum_duration": wall_seconds >= arguments.duration_seconds,
|
||||
"zero_failures": failures == 0,
|
||||
"minimum_source_fps": consumed / arguments.duration_seconds >= 9.5,
|
||||
"maximum_detector_completion_age_p95_ms": completion_age["p95"] <= 175.0,
|
||||
"maximum_worker_vram_gib": (
|
||||
float(telemetry_summary["gpu_memory_used_mib"]["maximum"]) <= 20 * 1024
|
||||
),
|
||||
"no_sustained_100_percent_gpu": _longest_full_gpu_run(telemetry.samples)
|
||||
< max(5, round(30.0 / arguments.telemetry_interval_seconds)),
|
||||
"bounded_latest_wins_queue": queue.maximum_depth <= arguments.queue_capacity,
|
||||
"triton_request_accounting": _triton_inference_count(triton_after)
|
||||
- _triton_inference_count(triton_before)
|
||||
== consumed,
|
||||
}
|
||||
detector_load_gate_passed = all(checks.values())
|
||||
document = {
|
||||
"schema_version": SCHEMA_VERSION,
|
||||
"profile_id": "rf-detr-large-coco-704-trt11-fp16-source-paced/v0",
|
||||
"source": {
|
||||
"source_id": "RAVNOVES00",
|
||||
"sha256": SOURCE_SHA256,
|
||||
"frame_rate": SOURCE_FPS,
|
||||
"duration_seconds": arguments.duration_seconds,
|
||||
"source_loops": source_loops[0],
|
||||
},
|
||||
"model_metadata": metadata,
|
||||
"execution": {
|
||||
"worker_id": "worker-006",
|
||||
"queue_policy": "bounded-latest-wins",
|
||||
"queue_capacity": arguments.queue_capacity,
|
||||
"queue_maximum_depth": queue.maximum_depth,
|
||||
"source_frames_produced": produced_count[0],
|
||||
"source_frames_consumed": consumed,
|
||||
"source_frame_replacements": queue.replacements,
|
||||
"failures": failures,
|
||||
"wall_seconds": round(wall_seconds, 6),
|
||||
"effective_consumed_fps": round(consumed / arguments.duration_seconds, 6),
|
||||
"background_services_retained": True,
|
||||
},
|
||||
"metrics": {
|
||||
"end_to_end_ms": core,
|
||||
"triton_round_trip_ms": _distribution(triton_round_trip_ms),
|
||||
"detector_completion_age_ms": completion_age,
|
||||
"class_counts_at_score_0_5": dict(sorted(class_counts.items())),
|
||||
"gpu": telemetry_summary,
|
||||
"gpu_before": gpu_before,
|
||||
"gpu_after": gpu_after,
|
||||
"process_peak_rss_before_mib": round(rss_before_kib / 1024.0, 6),
|
||||
"process_peak_rss_after_mib": round(rss_after_kib / 1024.0, 6),
|
||||
"triton_statistics_before": triton_before,
|
||||
"triton_statistics_after": triton_after,
|
||||
},
|
||||
"checks": checks,
|
||||
"detector_load_gate_passed": detector_load_gate_passed,
|
||||
"integrated_world_state_gate_evaluated": False,
|
||||
"candidate_accepted": False,
|
||||
"started_utc_ns": started_utc_ns,
|
||||
"completed_utc_ns": time.time_ns(),
|
||||
"completed": True,
|
||||
"authority": AUTHORITY,
|
||||
}
|
||||
output.write_bytes(_canonical_json(document) + b"\n")
|
||||
print(output)
|
||||
print(json.dumps(document["execution"], indent=2, sort_keys=True))
|
||||
print(json.dumps(checks, indent=2, sort_keys=True))
|
||||
return 0 if detector_load_gate_passed else 2
|
||||
|
||||
|
||||
def _produce(
|
||||
source: Path,
|
||||
queue: LatestWinsQueue,
|
||||
stop: threading.Event,
|
||||
errors: list[str],
|
||||
produced_count: list[int],
|
||||
source_loops: list[int],
|
||||
started_ns: int,
|
||||
duration_seconds: float,
|
||||
) -> None:
|
||||
try:
|
||||
period_ns = round(1_000_000_000.0 / SOURCE_FPS)
|
||||
while not stop.is_set():
|
||||
container = av.open(str(source))
|
||||
try:
|
||||
streams = container.streams.video
|
||||
if len(streams) != 1:
|
||||
raise RuntimeError("RAVNOVES00 video stream count changed")
|
||||
for decoded in container.decode(streams[0]):
|
||||
sequence = produced_count[0]
|
||||
scheduled_ns = started_ns + sequence * period_ns
|
||||
if scheduled_ns - started_ns >= round(duration_seconds * 1_000_000_000):
|
||||
queue.close()
|
||||
return
|
||||
remaining_seconds = (scheduled_ns - time.monotonic_ns()) / 1_000_000_000.0
|
||||
if remaining_seconds > 0 and stop.wait(remaining_seconds):
|
||||
queue.close()
|
||||
return
|
||||
bgr = decoded.to_ndarray(format="bgr24")
|
||||
if bgr.shape != (600, 800, 3):
|
||||
raise RuntimeError("RAVNOVES00 source raster changed")
|
||||
rgb = np.ascontiguousarray(bgr[:, :, ::-1])
|
||||
queue.put(SourceFrame(sequence, scheduled_ns, Image.fromarray(rgb, "RGB")))
|
||||
produced_count[0] += 1
|
||||
if stop.is_set():
|
||||
queue.close()
|
||||
return
|
||||
source_loops[0] += 1
|
||||
finally:
|
||||
container.close()
|
||||
except BaseException as error:
|
||||
errors.append(f"{type(error).__name__}: {error}")
|
||||
queue.close()
|
||||
|
||||
|
||||
def _distribution(values: list[float]) -> dict[str, float]:
|
||||
if not values:
|
||||
return {"mean": 0.0, "p50": 0.0, "p95": 0.0, "maximum": 0.0}
|
||||
array = np.asarray(values, dtype=np.float64)
|
||||
return {
|
||||
"mean": round(float(array.mean()), 6),
|
||||
"p50": round(float(np.percentile(array, 50)), 6),
|
||||
"p95": round(float(np.percentile(array, 95)), 6),
|
||||
"maximum": round(float(array.max()), 6),
|
||||
}
|
||||
|
||||
|
||||
def _telemetry_summary(samples: list[dict[str, float]]) -> dict[str, Any]:
|
||||
summary: dict[str, Any] = {"sample_count": len(samples)}
|
||||
for key in (
|
||||
"gpu_utilization_percent",
|
||||
"gpu_memory_used_mib",
|
||||
"gpu_power_w",
|
||||
"gpu_temperature_c",
|
||||
):
|
||||
summary[key] = _distribution([sample[key] for sample in samples])
|
||||
summary["longest_100_percent_gpu_sample_run"] = _longest_full_gpu_run(samples)
|
||||
return summary
|
||||
|
||||
|
||||
def _longest_full_gpu_run(samples: list[dict[str, float]]) -> int:
|
||||
longest = 0
|
||||
current = 0
|
||||
for sample in samples:
|
||||
if sample["gpu_utilization_percent"] >= 100.0:
|
||||
current += 1
|
||||
longest = max(longest, current)
|
||||
else:
|
||||
current = 0
|
||||
return longest
|
||||
|
||||
|
||||
def _triton_inference_count(statistics: dict[str, Any]) -> int:
|
||||
model_stats = statistics.get("model_stats")
|
||||
if not isinstance(model_stats, list) or len(model_stats) != 1:
|
||||
raise RuntimeError("unexpected Triton model statistics")
|
||||
count = model_stats[0].get("inference_count")
|
||||
if not isinstance(count, int):
|
||||
raise RuntimeError("Triton inference count is unavailable")
|
||||
return count
|
||||
|
||||
|
||||
def _sha256(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as stream:
|
||||
for chunk in iter(lambda: stream.read(1024 * 1024), b""):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def _canonical_json(value: object) -> bytes:
|
||||
return json.dumps(
|
||||
value,
|
||||
sort_keys=True,
|
||||
separators=(",", ":"),
|
||||
allow_nan=False,
|
||||
).encode("utf-8")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,508 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Qualify the pinned RF-DETR TensorRT finalist through isolated Triton."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import subprocess
|
||||
import time
|
||||
from collections import Counter
|
||||
from pathlib import Path
|
||||
from typing import Any, Final, cast
|
||||
|
||||
import numpy as np
|
||||
import tritonclient.http as httpclient # type: ignore[import-not-found]
|
||||
from PIL import Image, ImageDraw
|
||||
from torchvision.transforms import functional as vision_functional # type: ignore[import-not-found]
|
||||
|
||||
WORKER_RUN_SCHEMA: Final = "missioncore.m48s-fixed-detector-candidate-worker/v0"
|
||||
EXACT_FRAME_NAMES: Final = (
|
||||
"frame-000121.png",
|
||||
"frame-000131.png",
|
||||
"frame-000253.png",
|
||||
"frame-000275.png",
|
||||
"frame-000443.png",
|
||||
"frame-000463.png",
|
||||
"frame-001094.png",
|
||||
"frame-001228.png",
|
||||
"frame-001454.png",
|
||||
"frame-001856.png",
|
||||
"frame-002386.png",
|
||||
)
|
||||
COCO_CLASSES: Final = (
|
||||
"person", "bicycle", "car", "motorcycle", "airplane", "bus", "train", "truck",
|
||||
"boat", "traffic light", "fire hydrant", "stop sign", "parking meter", "bench",
|
||||
"bird", "cat", "dog", "horse", "sheep", "cow", "elephant", "bear", "zebra",
|
||||
"giraffe", "backpack", "umbrella", "handbag", "tie", "suitcase", "frisbee",
|
||||
"skis", "snowboard", "sports ball", "kite", "baseball bat", "baseball glove",
|
||||
"skateboard", "surfboard", "tennis racket", "bottle", "wine glass", "cup", "fork",
|
||||
"knife", "spoon", "bowl", "banana", "apple", "sandwich", "orange", "broccoli",
|
||||
"carrot", "hot dog", "pizza", "donut", "cake", "chair", "couch", "potted plant",
|
||||
"bed", "dining table", "toilet", "tv", "laptop", "mouse", "remote", "keyboard",
|
||||
"cell phone", "microwave", "oven", "toaster", "sink", "refrigerator", "book",
|
||||
"clock", "vase", "scissors", "teddy bear", "hair drier", "toothbrush",
|
||||
)
|
||||
COCO_SPARSE_IDS: Final = (
|
||||
1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 16, 17, 18, 19, 20, 21,
|
||||
22, 23, 24, 25, 27, 28, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42,
|
||||
43, 44, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60,
|
||||
61, 62, 63, 64, 65, 67, 70, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81,
|
||||
82, 84, 85, 86, 87, 88, 89, 90,
|
||||
)
|
||||
COCO_SPARSE_NAMES: Final = dict(zip(COCO_SPARSE_IDS, COCO_CLASSES, strict=True))
|
||||
AUTHORITY: Final = {
|
||||
"ground_truth": False,
|
||||
"candidate_accepted": False,
|
||||
"commands_enabled": False,
|
||||
"actuation_allowed": False,
|
||||
"navigation_or_safety_accepted": False,
|
||||
}
|
||||
MEANS: Final = (0.485, 0.456, 0.406)
|
||||
STDS: Final = (0.229, 0.224, 0.225)
|
||||
MINIMUM_BOX_AREA_PIXELS: Final = 64.0
|
||||
MAXIMUM_BOX_AREA_FRACTION: Final = 0.5
|
||||
MINIMUM_VALID_FOV_FRACTION: Final = 0.5
|
||||
OVERLAY_THRESHOLD: Final = 0.25
|
||||
|
||||
|
||||
class RawDetection(tuple[int, str, float, tuple[float, float, float, float]]):
|
||||
__slots__ = ()
|
||||
|
||||
def __new__(
|
||||
cls,
|
||||
class_id: int,
|
||||
label: str,
|
||||
score: float,
|
||||
box: tuple[float, float, float, float],
|
||||
) -> RawDetection:
|
||||
return tuple.__new__(cls, (class_id, label, score, box))
|
||||
|
||||
@property
|
||||
def class_id(self) -> int:
|
||||
return self[0]
|
||||
|
||||
@property
|
||||
def label(self) -> str:
|
||||
return self[1]
|
||||
|
||||
@property
|
||||
def score(self) -> float:
|
||||
return self[2]
|
||||
|
||||
@property
|
||||
def box(self) -> tuple[float, float, float, float]:
|
||||
return self[3]
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--endpoint", default="localhost:8100")
|
||||
parser.add_argument("--model-name", default="rf_detr_large")
|
||||
parser.add_argument("--profile-id", required=True)
|
||||
parser.add_argument("--provider-id", required=True)
|
||||
parser.add_argument("--upstream-revision", required=True)
|
||||
parser.add_argument("--input-root", type=Path, required=True)
|
||||
parser.add_argument("--valid-fov-mask", type=Path, required=True)
|
||||
parser.add_argument("--engine", type=Path, required=True)
|
||||
parser.add_argument("--expected-engine-sha256", required=True)
|
||||
parser.add_argument("--checkpoint-sha256", required=True)
|
||||
parser.add_argument("--pytorch-reference", type=Path, required=True)
|
||||
parser.add_argument("--output", type=Path, required=True)
|
||||
parser.add_argument("--warmup-iterations", type=int, default=10)
|
||||
parser.add_argument("--benchmark-iterations", type=int, default=100)
|
||||
arguments = parser.parse_args()
|
||||
if arguments.warmup_iterations < 1 or arguments.benchmark_iterations < 1:
|
||||
raise RuntimeError("warmup and benchmark iterations must be positive")
|
||||
input_root = arguments.input_root.resolve(strict=True)
|
||||
images = tuple(sorted(input_root.glob("frame-*.png")))
|
||||
if tuple(path.name for path in images) != EXACT_FRAME_NAMES:
|
||||
raise RuntimeError("Triton qualifier requires the exact M48S risk slice")
|
||||
mask = _load_mask(arguments.valid_fov_mask.resolve(strict=True))
|
||||
engine = arguments.engine.resolve(strict=True)
|
||||
engine_sha256 = _sha256(engine)
|
||||
if engine_sha256 != arguments.expected_engine_sha256:
|
||||
raise RuntimeError("TensorRT engine SHA-256 does not match the pinned finalist")
|
||||
reference_path = arguments.pytorch_reference.resolve(strict=True)
|
||||
output = arguments.output.absolute()
|
||||
if output.exists():
|
||||
raise RuntimeError("Triton Worker output already exists")
|
||||
output.parent.mkdir(mode=0o700, parents=True, exist_ok=True)
|
||||
overlay_root = output.parent / f"{output.stem}-overlays"
|
||||
overlay_root.mkdir(mode=0o700)
|
||||
|
||||
client = httpclient.InferenceServerClient(arguments.endpoint, concurrency=1)
|
||||
if not client.is_server_ready() or not client.is_model_ready(arguments.model_name):
|
||||
raise RuntimeError("isolated Triton or RF-DETR model is not ready")
|
||||
metadata = client.get_model_metadata(arguments.model_name)
|
||||
_validate_metadata(metadata)
|
||||
statistics_before = client.get_inference_statistics(arguments.model_name)
|
||||
gpu_before = _gpu_sample()
|
||||
started_utc_ns = time.time_ns()
|
||||
with Image.open(images[0]) as opened:
|
||||
warmup_image = opened.convert("RGB")
|
||||
for _ in range(arguments.warmup_iterations):
|
||||
_infer(client, arguments.model_name, warmup_image)
|
||||
|
||||
frames = []
|
||||
totals: Counter[str] = Counter()
|
||||
evidence_timings = []
|
||||
network_timings = []
|
||||
for image_path in images:
|
||||
with Image.open(image_path) as opened:
|
||||
image = opened.convert("RGB")
|
||||
raw, timing = _infer(client, arguments.model_name, image)
|
||||
evidence_timings.append(timing["end_to_end"])
|
||||
network_timings.append(timing["triton_round_trip"])
|
||||
detections = _qualify(raw, mask, image.size)
|
||||
totals["frame_count"] += 1
|
||||
totals["detection_count"] += len(detections)
|
||||
for detection in detections:
|
||||
totals[f"class:{detection['label']}"] += 1
|
||||
frames.append(
|
||||
{
|
||||
"frame_name": image_path.name,
|
||||
"source_sha256": _sha256(image_path),
|
||||
"detections": detections,
|
||||
"timing_ms": timing,
|
||||
"authority": AUTHORITY,
|
||||
}
|
||||
)
|
||||
_write_overlay(image, detections, overlay_root / image_path.name)
|
||||
|
||||
with Image.open(input_root / "frame-000253.png") as opened:
|
||||
benchmark_image = opened.convert("RGB")
|
||||
benchmark_timings = []
|
||||
benchmark_network_timings = []
|
||||
for _ in range(arguments.benchmark_iterations):
|
||||
_, timing = _infer(client, arguments.model_name, benchmark_image)
|
||||
benchmark_timings.append(timing["end_to_end"])
|
||||
benchmark_network_timings.append(timing["triton_round_trip"])
|
||||
gpu_after = _gpu_sample()
|
||||
statistics_after = client.get_inference_statistics(arguments.model_name)
|
||||
benchmark = _timing_summary(benchmark_timings)
|
||||
parity = _reference_parity(frames, reference_path)
|
||||
metrics = {
|
||||
"frames": {"requested": 11, "completed": totals["frame_count"]},
|
||||
"detection_count_at_minimum_score_0_1": totals["detection_count"],
|
||||
"class_counts_at_minimum_score_0_1": {
|
||||
key.removeprefix("class:"): value
|
||||
for key, value in sorted(totals.items())
|
||||
if key.startswith("class:")
|
||||
},
|
||||
"evidence_timing_ms": _timing_summary(evidence_timings),
|
||||
"evidence_triton_round_trip_ms": _timing_summary(network_timings),
|
||||
"benchmark": {
|
||||
"frame_name": "frame-000253.png",
|
||||
"iterations": arguments.benchmark_iterations,
|
||||
"timing_ms": benchmark,
|
||||
"triton_round_trip_ms": _timing_summary(benchmark_network_timings),
|
||||
"end_to_end_capacity_fps": round(1000.0 / benchmark["mean"], 6),
|
||||
},
|
||||
"pytorch_reference_parity": parity,
|
||||
"triton_statistics_before": statistics_before,
|
||||
"triton_statistics_after": statistics_after,
|
||||
"gpu_before": gpu_before,
|
||||
"gpu_after": gpu_after,
|
||||
}
|
||||
document = {
|
||||
"schema_version": WORKER_RUN_SCHEMA,
|
||||
"profile_id": arguments.profile_id,
|
||||
"provider_id": arguments.provider_id,
|
||||
"candidate": "rf-detr-large-coco-tensorrt",
|
||||
"upstream_revision": arguments.upstream_revision,
|
||||
"checkpoint_sha256": arguments.checkpoint_sha256,
|
||||
"engine_sha256": engine_sha256,
|
||||
"started_utc_ns": started_utc_ns,
|
||||
"completed_utc_ns": time.time_ns(),
|
||||
"completed": totals["frame_count"] == 11,
|
||||
"execution": {
|
||||
"worker_id": "worker-006",
|
||||
"device": gpu_after["name"],
|
||||
"precision": "strongly-typed-fp16",
|
||||
"inference_passes_per_evidence_frame": 1,
|
||||
"warmup_iterations": arguments.warmup_iterations,
|
||||
"benchmark_iterations": arguments.benchmark_iterations,
|
||||
"isolated_triton": True,
|
||||
"concurrent_services_retained": True,
|
||||
},
|
||||
"model_metadata": metadata,
|
||||
"frames": frames,
|
||||
"metrics": metrics,
|
||||
"authority": AUTHORITY,
|
||||
}
|
||||
output.write_bytes(_canonical_json(document) + b"\n")
|
||||
print(output)
|
||||
print(json.dumps(metrics, indent=2, sort_keys=True))
|
||||
return 0
|
||||
|
||||
|
||||
def _infer(
|
||||
client: httpclient.InferenceServerClient,
|
||||
model_name: str,
|
||||
image: Image.Image,
|
||||
) -> tuple[tuple[RawDetection, ...], dict[str, float]]:
|
||||
started = time.perf_counter_ns()
|
||||
tensor = vision_functional.to_tensor(image)
|
||||
tensor = vision_functional.resize(tensor, [704, 704], antialias=False)
|
||||
tensor = vision_functional.normalize(tensor, MEANS, STDS)
|
||||
batch = np.ascontiguousarray(tensor.unsqueeze(0).numpy(), dtype=np.float32)
|
||||
preprocessed = time.perf_counter_ns()
|
||||
infer_input = httpclient.InferInput("input", batch.shape, "FP32")
|
||||
infer_input.set_data_from_numpy(batch, binary_data=True)
|
||||
response = client.infer(
|
||||
model_name,
|
||||
[infer_input],
|
||||
outputs=[
|
||||
httpclient.InferRequestedOutput("dets", binary_data=True),
|
||||
httpclient.InferRequestedOutput("labels", binary_data=True),
|
||||
],
|
||||
)
|
||||
inferred = time.perf_counter_ns()
|
||||
boxes = response.as_numpy("dets")
|
||||
logits = response.as_numpy("labels")
|
||||
if boxes is None or logits is None:
|
||||
raise RuntimeError("Triton RF-DETR response is missing outputs")
|
||||
detections = decode_outputs(boxes, logits, image.size)
|
||||
completed = time.perf_counter_ns()
|
||||
return detections, {
|
||||
"preprocess": _milliseconds(started, preprocessed),
|
||||
"triton_round_trip": _milliseconds(preprocessed, inferred),
|
||||
"postprocess": _milliseconds(inferred, completed),
|
||||
"end_to_end": _milliseconds(started, completed),
|
||||
}
|
||||
|
||||
|
||||
def decode_outputs(
|
||||
boxes: np.ndarray[Any, Any],
|
||||
logits: np.ndarray[Any, Any],
|
||||
image_size: tuple[int, int],
|
||||
) -> tuple[RawDetection, ...]:
|
||||
if boxes.shape != (1, 300, 4) or logits.shape != (1, 300, 91):
|
||||
raise RuntimeError(f"unexpected RF-DETR output shapes: {boxes.shape}, {logits.shape}")
|
||||
probabilities = 1.0 / (1.0 + np.exp(-np.clip(logits[0].astype(np.float32), -80.0, 80.0)))
|
||||
flattened = probabilities.reshape(-1)
|
||||
topk = np.argsort(-flattened, kind="stable")[:300]
|
||||
width, height = image_size
|
||||
decoded = []
|
||||
for flat_index in topk:
|
||||
score = float(flattened[flat_index])
|
||||
if score <= 0.1:
|
||||
continue
|
||||
query_index = int(flat_index // logits.shape[2])
|
||||
sparse_class_id = int(flat_index % logits.shape[2])
|
||||
label = COCO_SPARSE_NAMES.get(sparse_class_id)
|
||||
if label is None:
|
||||
continue
|
||||
center_x, center_y, box_width, box_height = (
|
||||
float(value) for value in boxes[0, query_index].astype(np.float32)
|
||||
)
|
||||
box = (
|
||||
(center_x - box_width / 2.0) * width,
|
||||
(center_y - box_height / 2.0) * height,
|
||||
(center_x + box_width / 2.0) * width,
|
||||
(center_y + box_height / 2.0) * height,
|
||||
)
|
||||
decoded.append(RawDetection(COCO_CLASSES.index(label), label, score, box))
|
||||
return tuple(decoded)
|
||||
|
||||
|
||||
def _qualify(
|
||||
detections: tuple[RawDetection, ...],
|
||||
mask: np.ndarray[Any, Any],
|
||||
image_size: tuple[int, int],
|
||||
) -> list[dict[str, object]]:
|
||||
width, height = image_size
|
||||
image_area = float(width * height)
|
||||
qualified = []
|
||||
for detection in detections:
|
||||
x1, y1, x2, y2 = detection.box
|
||||
x1 = max(0.0, min(float(width), x1))
|
||||
y1 = max(0.0, min(float(height), y1))
|
||||
x2 = max(0.0, min(float(width), x2))
|
||||
y2 = max(0.0, min(float(height), y2))
|
||||
area = max(0.0, x2 - x1) * max(0.0, y2 - y1)
|
||||
if area < MINIMUM_BOX_AREA_PIXELS or area > MAXIMUM_BOX_AREA_FRACTION * image_area:
|
||||
continue
|
||||
center_x = min(width - 1, max(0, int((x1 + x2) / 2.0)))
|
||||
center_y = min(height - 1, max(0, int((y1 + y2) / 2.0)))
|
||||
if not bool(mask[center_y, center_x]):
|
||||
continue
|
||||
ix1 = min(width - 1, max(0, int(np.floor(x1))))
|
||||
iy1 = min(height - 1, max(0, int(np.floor(y1))))
|
||||
ix2 = min(width, max(ix1 + 1, int(np.ceil(x2))))
|
||||
iy2 = min(height, max(iy1 + 1, int(np.ceil(y2))))
|
||||
valid_fraction = float(mask[iy1:iy2, ix1:ix2].mean())
|
||||
if valid_fraction < MINIMUM_VALID_FOV_FRACTION:
|
||||
continue
|
||||
qualified.append(
|
||||
{
|
||||
"class_id": detection.class_id,
|
||||
"label": detection.label,
|
||||
"score": round(detection.score, 6),
|
||||
"bbox_xyxy": [round(value, 3) for value in (x1, y1, x2, y2)],
|
||||
"valid_fov_fraction": round(valid_fraction, 6),
|
||||
}
|
||||
)
|
||||
qualified.sort(key=lambda item: (-cast(float, item["score"]), cast(int, item["class_id"])))
|
||||
return qualified
|
||||
|
||||
|
||||
def _reference_parity(frames: list[dict[str, object]], reference_path: Path) -> dict[str, object]:
|
||||
reference = json.loads(reference_path.read_text(encoding="utf-8"))
|
||||
if not isinstance(reference, dict):
|
||||
raise RuntimeError("PyTorch reference must be a JSON object")
|
||||
reference_frames = reference.get("frames")
|
||||
if not isinstance(reference_frames, list):
|
||||
raise RuntimeError("PyTorch reference lacks frames")
|
||||
actual = _best_detection(frames, "frame-000253.png", "dog")
|
||||
expected = _best_detection(reference_frames, "frame-000253.png", "dog")
|
||||
if actual is None or expected is None:
|
||||
return {
|
||||
"frame_000253_dog_present_in_pytorch": expected is not None,
|
||||
"frame_000253_dog_present_in_tensorrt": actual is not None,
|
||||
"score_absolute_delta": None,
|
||||
"box_iou": None,
|
||||
"passed": False,
|
||||
}
|
||||
score_delta = abs(
|
||||
float(cast(float, actual["score"])) - float(cast(float, expected["score"]))
|
||||
)
|
||||
iou = _box_iou(
|
||||
cast(list[float], actual["bbox_xyxy"]),
|
||||
cast(list[float], expected["bbox_xyxy"]),
|
||||
)
|
||||
return {
|
||||
"frame_000253_dog_present_in_pytorch": True,
|
||||
"frame_000253_dog_present_in_tensorrt": True,
|
||||
"pytorch_score": expected["score"],
|
||||
"tensorrt_score": actual["score"],
|
||||
"score_absolute_delta": round(score_delta, 6),
|
||||
"box_iou": round(iou, 6),
|
||||
"passed": score_delta <= 0.05 and iou >= 0.9,
|
||||
}
|
||||
|
||||
|
||||
def _best_detection(
|
||||
frames: list[Any], frame_name: str, label: str
|
||||
) -> dict[str, object] | None:
|
||||
for frame in frames:
|
||||
if not isinstance(frame, dict) or frame.get("frame_name") != frame_name:
|
||||
continue
|
||||
detections = frame.get("detections")
|
||||
if not isinstance(detections, list):
|
||||
raise RuntimeError("reference frame detections must be a list")
|
||||
selected = [
|
||||
item
|
||||
for item in detections
|
||||
if isinstance(item, dict) and item.get("label") == label
|
||||
]
|
||||
return max(selected, key=lambda item: float(item["score"])) if selected else None
|
||||
raise RuntimeError(f"reference frame not found: {frame_name}")
|
||||
|
||||
|
||||
def _box_iou(left: list[float], right: list[float]) -> float:
|
||||
intersection_width = max(0.0, min(left[2], right[2]) - max(left[0], right[0]))
|
||||
intersection_height = max(0.0, min(left[3], right[3]) - max(left[1], right[1]))
|
||||
intersection = intersection_width * intersection_height
|
||||
left_area = max(0.0, left[2] - left[0]) * max(0.0, left[3] - left[1])
|
||||
right_area = max(0.0, right[2] - right[0]) * max(0.0, right[3] - right[1])
|
||||
union = left_area + right_area - intersection
|
||||
return intersection / union if union > 0 else 0.0
|
||||
|
||||
|
||||
def _validate_metadata(metadata: dict[str, Any]) -> None:
|
||||
expected_inputs = [{"name": "input", "datatype": "FP32", "shape": [1, 3, 704, 704]}]
|
||||
expected_outputs = [
|
||||
{"name": "dets", "datatype": "FP16", "shape": [1, 300, 4]},
|
||||
{"name": "labels", "datatype": "FP16", "shape": [1, 300, 91]},
|
||||
]
|
||||
if metadata.get("inputs") != expected_inputs or metadata.get("outputs") != expected_outputs:
|
||||
raise RuntimeError(f"unexpected isolated Triton model metadata: {metadata}")
|
||||
|
||||
|
||||
def _write_overlay(
|
||||
image: Image.Image, detections: list[dict[str, object]], destination: Path
|
||||
) -> None:
|
||||
annotated = image.copy()
|
||||
draw = ImageDraw.Draw(annotated)
|
||||
for detection in detections:
|
||||
score = cast(float, detection["score"])
|
||||
if score < OVERLAY_THRESHOLD:
|
||||
continue
|
||||
box = cast(list[float], detection["bbox_xyxy"])
|
||||
draw.rectangle(box, outline=(0, 220, 112), width=3)
|
||||
draw.text(
|
||||
(box[0] + 3, box[1] + 3),
|
||||
f"{detection['label']} {score:.2f}",
|
||||
fill=(255, 255, 255),
|
||||
)
|
||||
annotated.save(destination)
|
||||
|
||||
|
||||
def _load_mask(path: Path) -> np.ndarray[Any, Any]:
|
||||
with Image.open(path) as image:
|
||||
array = np.asarray(image.convert("L"), dtype=np.uint8)
|
||||
if array.shape != (600, 800):
|
||||
raise RuntimeError("valid-FOV mask must be 800x600")
|
||||
return array > 0
|
||||
|
||||
|
||||
def _gpu_sample() -> dict[str, object]:
|
||||
completed = subprocess.run(
|
||||
[
|
||||
"nvidia-smi",
|
||||
"--query-gpu=name,memory.total,memory.used,utilization.gpu,temperature.gpu,power.draw",
|
||||
"--format=csv,noheader,nounits",
|
||||
],
|
||||
check=True,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
values = [value.strip() for value in completed.stdout.strip().split(",")]
|
||||
if len(values) != 6:
|
||||
raise RuntimeError("unexpected nvidia-smi response")
|
||||
return {
|
||||
"name": values[0],
|
||||
"memory_total_mib": float(values[1]),
|
||||
"memory_used_mib": float(values[2]),
|
||||
"utilization_gpu_percent": float(values[3]),
|
||||
"temperature_c": float(values[4]),
|
||||
"power_w": float(values[5]),
|
||||
}
|
||||
|
||||
|
||||
def _timing_summary(values: list[float]) -> dict[str, float]:
|
||||
array = np.asarray(values, dtype=np.float64)
|
||||
return {
|
||||
"mean": round(float(array.mean()), 6),
|
||||
"p50": round(float(np.percentile(array, 50)), 6),
|
||||
"p95": round(float(np.percentile(array, 95)), 6),
|
||||
"max": round(float(array.max()), 6),
|
||||
}
|
||||
|
||||
|
||||
def _milliseconds(started: int, completed: int) -> float:
|
||||
return round(max(0, completed - started) / 1_000_000.0, 6)
|
||||
|
||||
|
||||
def _sha256(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as stream:
|
||||
for chunk in iter(lambda: stream.read(1024 * 1024), b""):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def _canonical_json(value: object) -> bytes:
|
||||
return json.dumps(
|
||||
value,
|
||||
sort_keys=True,
|
||||
separators=(",", ":"),
|
||||
allow_nan=False,
|
||||
).encode("utf-8")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,242 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Seal the bounded M48S fixed-class detector tournament."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
from collections import Counter
|
||||
from collections.abc import Mapping
|
||||
from pathlib import Path
|
||||
from typing import Any, Final
|
||||
|
||||
from k1link.perception.fixed_class_detector_tournament import (
|
||||
TOURNAMENT_SCHEMA,
|
||||
CandidateWorkerRun,
|
||||
canonical_json,
|
||||
false_authority,
|
||||
sha256_path,
|
||||
)
|
||||
|
||||
RESULT_PREFIX: Final = "m48s-fixed-detector-tournament-"
|
||||
THRESHOLDS: Final = (0.25, 0.5)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
repository = Path(__file__).resolve().parents[2]
|
||||
runtime = repository / ".runtime/compute-experiments/m48s-semantic-shadow"
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument(
|
||||
"--profile",
|
||||
type=Path,
|
||||
default=repository / "config/perception/fixed-class-detector-tournament-v0.json",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--dfine-result",
|
||||
type=Path,
|
||||
default=runtime / "fixed-detector-tournament-worker/dfine-s-worker.json",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--rf-detr-result",
|
||||
type=Path,
|
||||
default=runtime / "fixed-detector-tournament-worker/rf-detr-large-worker.json",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--yolox-result",
|
||||
type=Path,
|
||||
default=(
|
||||
runtime
|
||||
/ "yolox-all-coco-results"
|
||||
/ (
|
||||
"m48s-yolox-all-coco-shadow-"
|
||||
"7dbe6043b3fc12c7ddb162f609f883d86b34a4f2dd3785a632795f257e192d06"
|
||||
)
|
||||
),
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output-root",
|
||||
type=Path,
|
||||
default=runtime / "fixed-detector-tournament-results",
|
||||
)
|
||||
arguments = parser.parse_args()
|
||||
profile_path = arguments.profile.resolve(strict=True)
|
||||
profile = _load_object(profile_path)
|
||||
if profile.get("schema_version") != "missioncore.fixed-class-detector-tournament-profile/v0":
|
||||
raise RuntimeError("unexpected fixed-class tournament profile schema")
|
||||
dfine_path = arguments.dfine_result.resolve(strict=True)
|
||||
rf_detr_path = arguments.rf_detr_result.resolve(strict=True)
|
||||
dfine = CandidateWorkerRun.from_path(dfine_path)
|
||||
rf_detr = CandidateWorkerRun.from_path(rf_detr_path)
|
||||
yolox_root = arguments.yolox_result.resolve(strict=True)
|
||||
yolox_manifest_path = yolox_root / "manifest.json"
|
||||
yolox_frames_path = yolox_root / "frames.jsonl"
|
||||
yolox_manifest = _load_object(yolox_manifest_path)
|
||||
yolox_frames = _load_jsonl(yolox_frames_path)
|
||||
if yolox_manifest.get("result_id") != profile["source"]["baseline_result_id"]:
|
||||
raise RuntimeError("YOLOX baseline result does not match the tournament profile")
|
||||
worker_paths = {
|
||||
dfine.profile_id: dfine_path,
|
||||
rf_detr.profile_id: rf_detr_path,
|
||||
}
|
||||
candidate_summaries = {
|
||||
dfine.profile_id: _candidate_summary(dfine),
|
||||
rf_detr.profile_id: _candidate_summary(rf_detr),
|
||||
}
|
||||
baseline_summary = {
|
||||
"profile_id": "yolox-s-raw-kb4-all-coco-shadow/v2",
|
||||
"provider_id": "triton-yolox-s-raw-kb4-all-coco/v2",
|
||||
"quality": {
|
||||
str(threshold): _yolox_quality(yolox_frames, threshold)
|
||||
for threshold in THRESHOLDS
|
||||
},
|
||||
"worker_metrics": yolox_manifest["metrics"],
|
||||
}
|
||||
evidence = {
|
||||
"profile_sha256": sha256_path(profile_path),
|
||||
"worker_result_sha256": {
|
||||
profile_id: sha256_path(path) for profile_id, path in sorted(worker_paths.items())
|
||||
},
|
||||
"yolox_manifest_sha256": sha256_path(yolox_manifest_path),
|
||||
"yolox_frames_sha256": sha256_path(yolox_frames_path),
|
||||
"manual_visual_review": {
|
||||
"reviewed_frames": [253, 275, 443, 1228],
|
||||
"rf_detr_frame_253_dog_box_correct": True,
|
||||
"dfine_frame_253_dog_box_present_at_0_25": False,
|
||||
"dfine_observed_confusions": [
|
||||
"dog-as-skateboard",
|
||||
"scanner-body-as-surfboard",
|
||||
"duplicate-risk-labels-on-one-object",
|
||||
],
|
||||
"rf_detr_observed_advantage": "correct dog and cleaner person/vehicle labeling",
|
||||
"ground_truth": False,
|
||||
},
|
||||
}
|
||||
decision = {
|
||||
"finalist_profile_id": rf_detr.profile_id,
|
||||
"finalist_provider_id": rf_detr.provider_id,
|
||||
"eliminated_profile_ids": [
|
||||
"yolox-s-raw-kb4-all-coco-shadow/v2",
|
||||
dfine.profile_id,
|
||||
],
|
||||
"reasons": {
|
||||
"yolox-s-raw-kb4-all-coco-shadow/v2": (
|
||||
"visible frame-253 dog missed; retained only as regression baseline"
|
||||
),
|
||||
dfine.profile_id: (
|
||||
"frame-253 dog missed at 0.25 and 0.5; more risk-class confusions; slower qualifier"
|
||||
),
|
||||
rf_detr.profile_id: (
|
||||
"correct frame-253 dog at 0.741; cleaner risk labels; 44.786 FPS PyTorch qualifier"
|
||||
),
|
||||
},
|
||||
"candidate_accepted": False,
|
||||
"next_gate": (
|
||||
"RF-DETR-L TensorRT FP16 through isolated Triton, then full recorded "
|
||||
"concurrent-load replay"
|
||||
),
|
||||
}
|
||||
identity = {
|
||||
"schema_version": TOURNAMENT_SCHEMA,
|
||||
"profile_id": profile["profile_id"],
|
||||
"evidence": evidence,
|
||||
"baseline": baseline_summary,
|
||||
"candidates": candidate_summaries,
|
||||
"decision": decision,
|
||||
"completed": True,
|
||||
"accepted": False,
|
||||
"authority": false_authority(),
|
||||
}
|
||||
result_id = RESULT_PREFIX + hashlib.sha256(canonical_json(identity)).hexdigest()
|
||||
output_root = arguments.output_root.absolute()
|
||||
output_root.mkdir(mode=0o700, parents=True, exist_ok=True)
|
||||
destination = output_root / result_id
|
||||
if destination.exists():
|
||||
raise RuntimeError("immutable fixed-class tournament result already exists")
|
||||
destination.mkdir(mode=0o700)
|
||||
for source, name in (
|
||||
(dfine_path, "dfine-s-worker.json"),
|
||||
(rf_detr_path, "rf-detr-large-worker.json"),
|
||||
):
|
||||
(destination / name).write_bytes(source.read_bytes())
|
||||
manifest = {"result_id": result_id, **identity}
|
||||
(destination / "manifest.json").write_bytes(canonical_json(manifest) + b"\n")
|
||||
(destination / "report.json").write_bytes(
|
||||
canonical_json(
|
||||
{
|
||||
"schema_version": TOURNAMENT_SCHEMA,
|
||||
"result_id": result_id,
|
||||
"completed": True,
|
||||
"accepted": False,
|
||||
"baseline": baseline_summary,
|
||||
"candidates": candidate_summaries,
|
||||
"decision": decision,
|
||||
"authority": false_authority(),
|
||||
}
|
||||
)
|
||||
+ b"\n"
|
||||
)
|
||||
print(result_id)
|
||||
print(json.dumps(decision, indent=2, sort_keys=True))
|
||||
return 0
|
||||
|
||||
|
||||
def _candidate_summary(run: CandidateWorkerRun) -> dict[str, object]:
|
||||
return {
|
||||
"profile_id": run.profile_id,
|
||||
"provider_id": run.provider_id,
|
||||
"upstream_revision": run.upstream_revision,
|
||||
"checkpoint_sha256": run.checkpoint_sha256,
|
||||
"quality": {
|
||||
str(threshold): run.quality_summary(threshold=threshold)
|
||||
for threshold in THRESHOLDS
|
||||
},
|
||||
"worker_metrics": run.metrics,
|
||||
}
|
||||
|
||||
|
||||
def _yolox_quality(frames: list[Mapping[str, Any]], threshold: float) -> dict[str, object]:
|
||||
selected = []
|
||||
dog_selected = []
|
||||
for frame in frames:
|
||||
detections = frame.get("all_coco_detections")
|
||||
if not isinstance(detections, list):
|
||||
raise RuntimeError("YOLOX frame lacks all-COCO detections")
|
||||
for detection in detections:
|
||||
if not isinstance(detection, dict):
|
||||
raise RuntimeError("YOLOX detection must be an object")
|
||||
score = detection.get("score")
|
||||
label = detection.get("label")
|
||||
if isinstance(score, int | float) and score >= threshold and isinstance(label, str):
|
||||
selected.append(label)
|
||||
if frame.get("frame_name") == "frame-000253.jpg" and label == "dog":
|
||||
dog_selected.append(float(score))
|
||||
counts = Counter(selected)
|
||||
return {
|
||||
"threshold": threshold,
|
||||
"detection_count": len(selected),
|
||||
"class_counts": dict(sorted(counts.items())),
|
||||
"frame_000253_dog_detected": bool(dog_selected),
|
||||
"frame_000253_dog_max_score": max(dog_selected) if dog_selected else None,
|
||||
}
|
||||
|
||||
|
||||
def _load_object(path: Path) -> Mapping[str, Any]:
|
||||
document = json.loads(path.read_text(encoding="utf-8"))
|
||||
if not isinstance(document, dict):
|
||||
raise RuntimeError(f"JSON document must be an object: {path}")
|
||||
return document
|
||||
|
||||
|
||||
def _load_jsonl(path: Path) -> list[Mapping[str, Any]]:
|
||||
rows: list[Mapping[str, Any]] = []
|
||||
for line in path.read_text(encoding="utf-8").splitlines():
|
||||
document = json.loads(line)
|
||||
if not isinstance(document, dict):
|
||||
raise RuntimeError(f"JSONL row must be an object: {path}")
|
||||
rows.append(document)
|
||||
return rows
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,264 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Seal the RF-DETR TensorRT/Triton detector deployment gate."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
from typing import Any, Final
|
||||
|
||||
from k1link.perception.fixed_class_detector_tournament import (
|
||||
CandidateWorkerRun,
|
||||
canonical_json,
|
||||
false_authority,
|
||||
sha256_path,
|
||||
)
|
||||
|
||||
SCHEMA_VERSION: Final = "missioncore.m48s-rf-detr-deployment-gate/v0"
|
||||
RESULT_PREFIX: Final = "m48s-rf-detr-deployment-gate-"
|
||||
ENGINE_SHA256: Final = "986399ce706b7380472cf5e473232249fed6e628971d8007f6609e83128d46b8"
|
||||
EXPORTED_ONNX_SHA256: Final = (
|
||||
"9c1948e56bbb6ff03349012b8bb334cacaf8ae480f22caa0704ee70de9a72300"
|
||||
)
|
||||
FP16_ONNX_SHA256: Final = "9015fcc1317f268ce866bed6b5a33132c24963e1502b02f145fa184e11de5ecb"
|
||||
|
||||
|
||||
def main() -> int:
|
||||
repository = Path(__file__).resolve().parents[2]
|
||||
runtime = repository / ".runtime/compute-experiments/m48s-semantic-shadow"
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument(
|
||||
"--profile",
|
||||
type=Path,
|
||||
default=repository / "config/perception/rf-detr-large-risk-shadow-v0.json",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--tournament",
|
||||
type=Path,
|
||||
default=(
|
||||
runtime
|
||||
/ "fixed-detector-tournament-results"
|
||||
/ (
|
||||
"m48s-fixed-detector-tournament-"
|
||||
"0e61d75e6dc575d53e4bb98772a41d240fe627ad642de5178beb1154636e1299"
|
||||
)
|
||||
/ "manifest.json"
|
||||
),
|
||||
)
|
||||
parser.add_argument(
|
||||
"--pytorch-result",
|
||||
type=Path,
|
||||
default=runtime / "fixed-detector-tournament-worker/rf-detr-large-worker.json",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--triton-result",
|
||||
type=Path,
|
||||
default=runtime / "fixed-detector-tournament-worker/rf-detr-large-triton-worker.json",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--onnx-export",
|
||||
type=Path,
|
||||
default=runtime / "rf-detr-deployment-worker/rf-detr-large-onnx-export.json",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--fp16-conversion",
|
||||
type=Path,
|
||||
default=(
|
||||
runtime
|
||||
/ "rf-detr-deployment-worker/rf-detr-large-onnx-fp16-conversion-v4.json"
|
||||
),
|
||||
)
|
||||
parser.add_argument(
|
||||
"--trtexec-log",
|
||||
type=Path,
|
||||
default=runtime / "rf-detr-deployment-worker/rf-detr-large-trtexec-build-v6.log",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--load-result",
|
||||
type=Path,
|
||||
default=runtime / "rf-detr-deployment-worker/rf-detr-load-30m.json",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output-root",
|
||||
type=Path,
|
||||
default=runtime / "rf-detr-deployment-results",
|
||||
)
|
||||
arguments = parser.parse_args()
|
||||
|
||||
paths = {
|
||||
"profile": arguments.profile.resolve(strict=True),
|
||||
"tournament": arguments.tournament.resolve(strict=True),
|
||||
"pytorch_result": arguments.pytorch_result.resolve(strict=True),
|
||||
"triton_result": arguments.triton_result.resolve(strict=True),
|
||||
"onnx_export": arguments.onnx_export.resolve(strict=True),
|
||||
"fp16_conversion": arguments.fp16_conversion.resolve(strict=True),
|
||||
"trtexec_log": arguments.trtexec_log.resolve(strict=True),
|
||||
"load_result": arguments.load_result.resolve(strict=True),
|
||||
}
|
||||
profile = _load_object(paths["profile"])
|
||||
tournament = _load_object(paths["tournament"])
|
||||
pytorch = CandidateWorkerRun.from_path(paths["pytorch_result"])
|
||||
triton = CandidateWorkerRun.from_path(paths["triton_result"])
|
||||
triton_document = _load_object(paths["triton_result"])
|
||||
onnx_export = _load_object(paths["onnx_export"])
|
||||
fp16_conversion = _load_object(paths["fp16_conversion"])
|
||||
load_result = _load_object(paths["load_result"])
|
||||
build_log = paths["trtexec_log"].read_text("utf-8")
|
||||
|
||||
_validate(
|
||||
profile=profile,
|
||||
tournament=tournament,
|
||||
pytorch=pytorch,
|
||||
triton=triton,
|
||||
triton_document=triton_document,
|
||||
onnx_export=onnx_export,
|
||||
fp16_conversion=fp16_conversion,
|
||||
load_result=load_result,
|
||||
build_log=build_log,
|
||||
)
|
||||
pytorch_quality = pytorch.quality_summary(threshold=0.5)
|
||||
triton_quality = triton.quality_summary(threshold=0.5)
|
||||
evidence = {
|
||||
"files": {
|
||||
name: {"sha256": sha256_path(path), "size_bytes": path.stat().st_size}
|
||||
for name, path in sorted(paths.items())
|
||||
},
|
||||
"engine_sha256": ENGINE_SHA256,
|
||||
"exported_onnx_sha256": EXPORTED_ONNX_SHA256,
|
||||
"strongly_typed_fp16_onnx_sha256": FP16_ONNX_SHA256,
|
||||
"pytorch_quality_at_0_5": pytorch_quality,
|
||||
"triton_quality_at_0_5": triton_quality,
|
||||
"tensorrt_parity": triton_document["metrics"]["pytorch_reference_parity"],
|
||||
"triton_benchmark": triton_document["metrics"]["benchmark"],
|
||||
"source_paced_load": {
|
||||
"execution": load_result["execution"],
|
||||
"checks": load_result["checks"],
|
||||
"end_to_end_ms": load_result["metrics"]["end_to_end_ms"],
|
||||
"detector_completion_age_ms": load_result["metrics"][
|
||||
"detector_completion_age_ms"
|
||||
],
|
||||
"gpu": load_result["metrics"]["gpu"],
|
||||
},
|
||||
}
|
||||
decision = {
|
||||
"tournament_finalist": True,
|
||||
"tensorrt_numeric_parity_passed": True,
|
||||
"detector_source_paced_load_gate_passed": True,
|
||||
"ready_for_reference_graph_shadow": True,
|
||||
"integrated_world_state_gate_evaluated": False,
|
||||
"production_accepted": False,
|
||||
"next_gate": (
|
||||
"run the RF-DETR shadow provider inside the complete reference graph and require "
|
||||
"world-state p95 <= 175 ms without changing false authority"
|
||||
),
|
||||
}
|
||||
identity = {
|
||||
"schema_version": SCHEMA_VERSION,
|
||||
"profile_id": profile["profile_id"],
|
||||
"evidence": evidence,
|
||||
"decision": decision,
|
||||
"completed": True,
|
||||
"accepted": False,
|
||||
"authority": false_authority(),
|
||||
}
|
||||
result_id = RESULT_PREFIX + hashlib.sha256(canonical_json(identity)).hexdigest()
|
||||
destination = arguments.output_root.absolute() / result_id
|
||||
if destination.exists():
|
||||
raise RuntimeError("immutable RF-DETR deployment result already exists")
|
||||
destination.mkdir(mode=0o700, parents=True)
|
||||
for name, path in paths.items():
|
||||
suffix = path.suffix or ".evidence"
|
||||
(destination / f"{name}{suffix}").write_bytes(path.read_bytes())
|
||||
manifest = {"result_id": result_id, **identity}
|
||||
(destination / "manifest.json").write_bytes(canonical_json(manifest) + b"\n")
|
||||
(destination / "report.json").write_bytes(
|
||||
canonical_json(
|
||||
{
|
||||
"schema_version": SCHEMA_VERSION,
|
||||
"result_id": result_id,
|
||||
"completed": True,
|
||||
"accepted": False,
|
||||
"evidence": evidence,
|
||||
"decision": decision,
|
||||
"authority": false_authority(),
|
||||
}
|
||||
)
|
||||
+ b"\n"
|
||||
)
|
||||
print(result_id)
|
||||
print(json.dumps(decision, indent=2, sort_keys=True))
|
||||
return 0
|
||||
|
||||
|
||||
def _validate(
|
||||
*,
|
||||
profile: dict[str, Any],
|
||||
tournament: dict[str, Any],
|
||||
pytorch: CandidateWorkerRun,
|
||||
triton: CandidateWorkerRun,
|
||||
triton_document: dict[str, Any],
|
||||
onnx_export: dict[str, Any],
|
||||
fp16_conversion: dict[str, Any],
|
||||
load_result: dict[str, Any],
|
||||
build_log: str,
|
||||
) -> None:
|
||||
if profile.get("schema_version") != "missioncore.rf-detr-risk-shadow-profile/v0":
|
||||
raise RuntimeError("unexpected RF-DETR shadow profile schema")
|
||||
decision = tournament.get("decision")
|
||||
if not isinstance(decision, dict) or decision.get("finalist_profile_id") != pytorch.profile_id:
|
||||
raise RuntimeError("tournament does not select the RF-DETR PyTorch reference")
|
||||
if triton.profile_id != "rf-detr-large-coco-704-trt11-fp16/v0":
|
||||
raise RuntimeError("unexpected RF-DETR Triton profile")
|
||||
if triton_document.get("engine_sha256") != ENGINE_SHA256:
|
||||
raise RuntimeError("RF-DETR Triton engine identity changed")
|
||||
parity = triton_document.get("metrics", {}).get("pytorch_reference_parity", {})
|
||||
if not isinstance(parity, dict) or parity.get("passed") is not True:
|
||||
raise RuntimeError("RF-DETR TensorRT numeric parity failed")
|
||||
if pytorch.quality_summary(threshold=0.5)["class_counts"] != triton.quality_summary(
|
||||
threshold=0.5
|
||||
)["class_counts"]:
|
||||
raise RuntimeError("RF-DETR TensorRT 0.5 class counts diverged from PyTorch")
|
||||
if onnx_export.get("onnx", {}).get("sha256") != EXPORTED_ONNX_SHA256:
|
||||
raise RuntimeError("RF-DETR exported ONNX identity changed")
|
||||
if fp16_conversion.get("output_onnx_sha256") != FP16_ONNX_SHA256:
|
||||
raise RuntimeError("RF-DETR strongly typed FP16 ONNX identity changed")
|
||||
required_build_markers = (
|
||||
"Precision: Strongly Typed",
|
||||
"Input binding for input with dimensions 1x3x704x704 and type fp32",
|
||||
"Output binding for dets with dimensions 1x300x4 and type fp16",
|
||||
"Output binding for labels with dimensions 1x300x91 and type fp16",
|
||||
"&&&& PASSED TensorRT.trtexec",
|
||||
)
|
||||
if any(marker not in build_log for marker in required_build_markers):
|
||||
raise RuntimeError("TensorRT build log is incomplete")
|
||||
if (
|
||||
load_result.get("schema_version")
|
||||
!= "missioncore.m48s-rf-detr-source-paced-load/v0"
|
||||
or load_result.get("completed") is not True
|
||||
or load_result.get("detector_load_gate_passed") is not True
|
||||
or load_result.get("candidate_accepted") is not False
|
||||
or load_result.get("integrated_world_state_gate_evaluated") is not False
|
||||
):
|
||||
raise RuntimeError("RF-DETR source-paced load result is incompatible")
|
||||
checks = load_result.get("checks")
|
||||
if (
|
||||
not isinstance(checks, dict)
|
||||
or not checks
|
||||
or not all(value is True for value in checks.values())
|
||||
):
|
||||
raise RuntimeError("RF-DETR source-paced load checks did not all pass")
|
||||
if load_result.get("authority") != false_authority():
|
||||
raise RuntimeError("RF-DETR source-paced load gained authority")
|
||||
|
||||
|
||||
def _load_object(path: Path) -> dict[str, Any]:
|
||||
document = json.loads(path.read_text("utf-8"))
|
||||
if not isinstance(document, dict):
|
||||
raise RuntimeError(f"JSON document must be an object: {path}")
|
||||
return document
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,30 @@
|
||||
FROM nvcr.io/nvidia/tritonserver:26.06-py3
|
||||
|
||||
ARG DFINE_REVISION=956d1709314c2c6a4df6f34de232054578a7449f
|
||||
|
||||
RUN python3 -m pip install --no-cache-dir \
|
||||
--index-url https://download.pytorch.org/whl/cu130 \
|
||||
"torch==2.9.1+cu130" \
|
||||
"torchvision==0.24.1+cu130"
|
||||
|
||||
RUN git clone https://github.com/Peterande/D-FINE.git /opt/dfine \
|
||||
&& git -C /opt/dfine checkout --detach "${DFINE_REVISION}" \
|
||||
&& test "$(git -C /opt/dfine rev-parse HEAD)" = "${DFINE_REVISION}"
|
||||
|
||||
RUN python3 -m pip install --no-cache-dir \
|
||||
"rfdetr[onnx]==1.9.4" \
|
||||
"numpy==1.26.4" \
|
||||
"ml_dtypes==0.5.4" \
|
||||
"onnxconverter-common==1.16.0" \
|
||||
"tritonclient[http]==2.71.0" \
|
||||
"faster-coco-eval>=1.6.6" \
|
||||
"PyYAML>=6.0" \
|
||||
"scipy>=1.10" \
|
||||
"calflops>=0.3" \
|
||||
"loguru>=0.7" \
|
||||
"tensorboard>=2.17"
|
||||
|
||||
LABEL com.nodedc.product="mission-core" \
|
||||
com.nodedc.stack="ndc-mission-core-compute" \
|
||||
com.nodedc.role="bounded-detector-qualification" \
|
||||
com.nodedc.managed-by="codex-bounded-experiment"
|
||||
@@ -0,0 +1,28 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
readonly DFINE_REVISION="956d1709314c2c6a4df6f34de232054578a7449f"
|
||||
|
||||
python3 -m pip install --no-cache-dir \
|
||||
--index-url https://download.pytorch.org/whl/cu130 \
|
||||
"torch==2.9.1+cu130" \
|
||||
"torchvision==0.24.1+cu130"
|
||||
|
||||
git clone https://github.com/Peterande/D-FINE.git /opt/dfine
|
||||
git -C /opt/dfine checkout --detach "${DFINE_REVISION}"
|
||||
test "$(git -C /opt/dfine rev-parse HEAD)" = "${DFINE_REVISION}"
|
||||
|
||||
python3 -m pip install --no-cache-dir \
|
||||
"rfdetr[onnx]==1.9.4" \
|
||||
"numpy==1.26.4" \
|
||||
"ml_dtypes==0.5.4" \
|
||||
"onnxconverter-common==1.16.0" \
|
||||
"tritonclient[http]==2.71.0" \
|
||||
"faster-coco-eval>=1.6.6" \
|
||||
"PyYAML>=6.0" \
|
||||
"scipy>=1.10" \
|
||||
"calflops>=0.3" \
|
||||
"loguru>=0.7" \
|
||||
"tensorboard>=2.17"
|
||||
|
||||
python3 -c "import importlib.metadata, rfdetr, torch, torchvision; print(torch.__version__, torchvision.__version__, importlib.metadata.version('rfdetr'))"
|
||||
@@ -0,0 +1,30 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
if [[ "$#" -ne 3 ]]; then
|
||||
echo "usage: $0 ONNX_PATH ENGINE_PATH LOG_PATH" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
readonly onnx_path="$1"
|
||||
readonly engine_path="$2"
|
||||
readonly log_path="$3"
|
||||
readonly timing_cache="${engine_path}.timing-cache"
|
||||
|
||||
test -f "${onnx_path}"
|
||||
test ! -e "${engine_path}"
|
||||
test ! -e "${log_path}"
|
||||
mkdir -p "$(dirname "${engine_path}")" "$(dirname "${log_path}")"
|
||||
|
||||
/usr/bin/trtexec \
|
||||
--onnx="${onnx_path}" \
|
||||
--saveEngine="${engine_path}" \
|
||||
--timingCacheFile="${timing_cache}" \
|
||||
--memPoolSize=workspace:4096 \
|
||||
--warmUp=1000 \
|
||||
--duration=5 \
|
||||
--avgRuns=100 \
|
||||
2>&1 | tee "${log_path}"
|
||||
|
||||
test -s "${engine_path}"
|
||||
sha256sum "${onnx_path}" "${engine_path}" "${log_path}"
|
||||
@@ -0,0 +1,47 @@
|
||||
name: "rf_detr_large"
|
||||
platform: "tensorrt_plan"
|
||||
max_batch_size: 0
|
||||
|
||||
input [
|
||||
{
|
||||
name: "input"
|
||||
data_type: TYPE_FP32
|
||||
dims: [ 1, 3, 704, 704 ]
|
||||
}
|
||||
]
|
||||
|
||||
output [
|
||||
{
|
||||
name: "dets"
|
||||
data_type: TYPE_FP16
|
||||
dims: [ 1, 300, 4 ]
|
||||
},
|
||||
{
|
||||
name: "labels"
|
||||
data_type: TYPE_FP16
|
||||
dims: [ 1, 300, 91 ]
|
||||
}
|
||||
]
|
||||
|
||||
instance_group [
|
||||
{
|
||||
count: 1
|
||||
kind: KIND_GPU
|
||||
gpus: [ 0 ]
|
||||
}
|
||||
]
|
||||
|
||||
model_warmup [
|
||||
{
|
||||
name: "rf_detr_large_zero"
|
||||
batch_size: 0
|
||||
inputs: {
|
||||
key: "input"
|
||||
value: {
|
||||
data_type: TYPE_FP32
|
||||
dims: [ 1, 3, 704, 704 ]
|
||||
zero_data: true
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
Reference in New Issue
Block a user