feat(perception): define semantic object understanding
This commit is contained in:
DCCONSTRUCTIONS
@@ -0,0 +1,84 @@
|
||||
# M48S Mask Grounding DINO shadow report
|
||||
|
||||
Date: 2026-08-25
|
||||
Status: completed shadow; rejected for semantic and navigation authority
|
||||
|
||||
## Question
|
||||
|
||||
Can a ready open-vocabulary instance-segmentation model name the urban obstacles
|
||||
already discovered by Mission Core geometry, without a manual annotation program?
|
||||
|
||||
## Frozen setup
|
||||
|
||||
- Source: the same 11 raw 800 × 600 KB4 frames and 73 occupied-geometry
|
||||
observations admitted by the M4.8 semantic slice.
|
||||
- Model: NVIDIA TAO Mask Grounding DINO Swin-Tiny research deployable v2.0,
|
||||
ONNX SHA-256
|
||||
`4e8b066cf7782ae6b4269255b861c205f4b07b33ff752e5eaf297c42ecfb2f5e`.
|
||||
- Runtime: TAO 7.0.1 deploy container on the Worker RTX 4090; generated FP16
|
||||
TensorRT engine SHA-256
|
||||
`1d127107d15f18dcddafd7345a0a766acfee204d63c5ab697eaabada2c7afcd2`.
|
||||
- Prompts: separate static, agent, and vehicle groups. Adult, child, and dog were
|
||||
also rerun as three independent single-class probes on frames 253 and 1228.
|
||||
- Binding: masks were clustered at IoU ≥ 0.90 and bound only through exact
|
||||
projected LiDAR point identities. A mask had to cover ≥ 50% of one obstacle's
|
||||
projected points and ≤ 25% of every secondary obstacle.
|
||||
- Authority: false for ground truth, acceptance, commands, actuation, navigation,
|
||||
and safety throughout.
|
||||
|
||||
The model choice and TensorRT path follow NVIDIA's official
|
||||
[Mask Grounding DINO documentation](https://docs.nvidia.com/tao/tao-toolkit/latest/text/cv_finetuning/pytorch/instance_segmentation/mask_grounding_dino.html)
|
||||
and
|
||||
[TAO Deploy documentation](https://docs.nvidia.com/tao/tao-toolkit/latest/text/tao_deploy/mask_grounding_dino.html).
|
||||
The checkpoint is the official
|
||||
[NGC research deployable v2.0](https://catalog.ngc.nvidia.com/orgs/nvidia/tao/models/pretrained_mask_grounding_dino_v2/mask_grounding_dino_swin_tiny_research_deployable_v2.0/version-history).
|
||||
No commercial-model license was accepted on the user's behalf.
|
||||
|
||||
## Result
|
||||
|
||||
| Measure | Result |
|
||||
|---|---:|
|
||||
| Frames completed | 11 / 11 |
|
||||
| Geometry obstacles retained | 73 |
|
||||
| Raw mask detections | 25 |
|
||||
| Pixel-instance clusters after caption deduplication | 7 |
|
||||
| Instances exclusively bound to one LiDAR obstacle | 4 |
|
||||
| Instances covering multiple LiDAR obstacles | 2 |
|
||||
| Instances without occupied-geometry support | 1 |
|
||||
| Final named instances | 0 |
|
||||
| Adult single-class detections | 0 |
|
||||
| Child single-class detections | 0 |
|
||||
| Dog single-class detections | 0 |
|
||||
|
||||
The mask head was useful: four instances had strong exclusive LiDAR support.
|
||||
The language classification was not. The same near-identical mask was repeatedly
|
||||
named `trash bin`, `shopping cart`, `road sign`, and `truck`. After the required
|
||||
confidence margin, six of seven instances were ambiguous and one unresolved.
|
||||
|
||||
The full evidence result is
|
||||
`m48s-mask-grounding-dino-shadow-b0a37f265223b4138754f76d5d7b8d17e7c4f5395481990f169fccc1645baba8`.
|
||||
|
||||
## Decision
|
||||
|
||||
This research checkpoint is rejected as an object-name provider and is not a
|
||||
candidate for navigation or safety authority. The zero-result single-class probes
|
||||
show that prompt competition was not the reason adult, child, and dog were missed.
|
||||
|
||||
The next candidate separates the two jobs:
|
||||
|
||||
1. LiDAR geometry supplies point/box prompts to a promptable mask segmenter.
|
||||
2. An independent zero-shot image encoder ranks canonical Mission Core classes on
|
||||
the geometry-owned crop and masked crop.
|
||||
3. The name stays unresolved unless the two views agree and the class margin passes.
|
||||
4. Unknown objects remain occupied and route-around; no physical traversability
|
||||
model is introduced.
|
||||
|
||||
SAM 2 officially supports point and box prompts through its
|
||||
[image predictor](https://github.com/facebookresearch/sam2/blob/main/sam2/sam2_image_predictor.py).
|
||||
OpenCLIP is the current open implementation candidate for independent zero-shot
|
||||
crop classification; its official repository is
|
||||
[mlfoundations/open_clip](https://github.com/mlfoundations/open_clip).
|
||||
|
||||
No manual annotation program is authorized by this result. A small two-reviewer
|
||||
truth slice remains necessary only to score candidates; it is evidence QA, not a
|
||||
training-dataset commitment.
|
||||
@@ -0,0 +1,208 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Materialize geometry-owned raw-KB4 crops for the M48S Worker shadow."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import shutil
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
|
||||
from k1link.perception.baseline import BASELINE_RECORDED_JOB_ID
|
||||
from k1link.perception.contracts import SourceEnvelope
|
||||
from k1link.perception.detector_replay_result import (
|
||||
read_detector_replay_result,
|
||||
require_m4_detector_replay_acceptance,
|
||||
)
|
||||
from k1link.perception.geometry import (
|
||||
DEFAULT_GEOMETRY_PROFILE_PATH,
|
||||
Ravnoves00GeometryAssociationProvider,
|
||||
RecordedGeometryStore,
|
||||
load_geometry_profile,
|
||||
)
|
||||
from k1link.perception.geometry_semantic_roi import (
|
||||
GeometrySemanticRoiProfile,
|
||||
build_geometry_semantic_rois,
|
||||
materialize_geometry_semantic_crop,
|
||||
)
|
||||
from k1link.perception.graph_validation import validate_observations
|
||||
from k1link.perception.providers import SourcePacket
|
||||
from k1link.perception.recorded_source import RECORDED_SOURCE_PACK_ID, RecordedFrameReference
|
||||
|
||||
FRAME_INDICES = (121, 131, 253, 275, 443, 463, 1094, 1228, 1454, 1856, 2386)
|
||||
DETECTOR_RESULT_ID = (
|
||||
"m4-detector-replay-11f83f2e0b81758ac2a5a5fc54e9d293b501678df5f6ef97b5c6069ba08605c5"
|
||||
)
|
||||
SCHEMA_VERSION = "missioncore.m48s-geometry-semantic-roi-package/v0"
|
||||
|
||||
|
||||
def main() -> int:
|
||||
repository = Path(__file__).resolve().parents[2]
|
||||
runtime = repository / ".runtime/compute-experiments/m48s-semantic-shadow"
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--repository-root", type=Path, default=repository)
|
||||
parser.add_argument(
|
||||
"--input-root",
|
||||
type=Path,
|
||||
default=runtime / "raw-11-valid-fov-fill-v1",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output-root",
|
||||
type=Path,
|
||||
default=runtime / "geometry-rois-v1",
|
||||
)
|
||||
arguments = parser.parse_args()
|
||||
root = arguments.repository_root.resolve(strict=True)
|
||||
inputs = arguments.input_root.resolve(strict=True)
|
||||
output = arguments.output_root.expanduser().absolute()
|
||||
if output.exists():
|
||||
raise RuntimeError("geometry semantic ROI package already exists")
|
||||
|
||||
detector = read_detector_replay_result(root / ".runtime/worker-results" / DETECTOR_RESULT_ID)
|
||||
require_m4_detector_replay_acceptance(detector)
|
||||
by_sequence = {item.sequence: item for item in detector.frames}
|
||||
if any(index not in by_sequence for index in FRAME_INDICES):
|
||||
raise RuntimeError("geometry semantic frame escaped the accepted detector timeline")
|
||||
geometry_profile_path = root / DEFAULT_GEOMETRY_PROFILE_PATH
|
||||
geometry_profile = load_geometry_profile(geometry_profile_path)
|
||||
store = RecordedGeometryStore.from_repository(root, profile=geometry_profile)
|
||||
provider = Ravnoves00GeometryAssociationProvider(store=store)
|
||||
roi_profile = GeometrySemanticRoiProfile()
|
||||
|
||||
staging = output.parent / f".{output.name}.{uuid.uuid4().hex}.tmp"
|
||||
staging.mkdir(mode=0o700, parents=True, exist_ok=False)
|
||||
try:
|
||||
input_destination = staging / "input"
|
||||
input_destination.mkdir(mode=0o700)
|
||||
frame_documents: list[dict[str, object]] = []
|
||||
artifacts: list[dict[str, object]] = []
|
||||
total_observations = 0
|
||||
total_rois = 0
|
||||
for frame_index in FRAME_INDICES:
|
||||
detector_frame = by_sequence[frame_index]
|
||||
if detector_frame.outcome != "completed":
|
||||
raise RuntimeError("accepted detector frame is incomplete")
|
||||
packet = _packet(detector_frame.envelope)
|
||||
observations = provider.associate(packet, ())
|
||||
validate_observations(packet, (), observations)
|
||||
frame = store.frame(packet)
|
||||
if frame is None:
|
||||
raise RuntimeError("selected geometry frame is unavailable")
|
||||
roi_frame = build_geometry_semantic_rois(
|
||||
frame=frame,
|
||||
observations=observations,
|
||||
profile=roi_profile,
|
||||
)
|
||||
source = inputs / f"frame-{frame_index:06d}.png"
|
||||
frame_rois: list[dict[str, object]] = []
|
||||
for roi in roi_frame.rois:
|
||||
destination = input_destination / roi.crop_name
|
||||
materialize_geometry_semantic_crop(
|
||||
image_path=source,
|
||||
roi=roi,
|
||||
destination=destination,
|
||||
)
|
||||
document = roi.to_dict()
|
||||
document["crop_sha256"] = _sha256(destination)
|
||||
document["crop_width"] = int(roi.crop_region.x_max - roi.crop_region.x_min)
|
||||
document["crop_height"] = int(roi.crop_region.y_max - roi.crop_region.y_min)
|
||||
frame_rois.append(document)
|
||||
artifacts.append(
|
||||
{
|
||||
"path": f"input/{roi.crop_name}",
|
||||
"sha256": document["crop_sha256"],
|
||||
}
|
||||
)
|
||||
frame_documents.append(
|
||||
{
|
||||
"frame_index": frame_index,
|
||||
"frame_id": detector_frame.envelope.frame_id,
|
||||
"source_image": source.name,
|
||||
"source_image_sha256": _sha256(source),
|
||||
"geometry_observation_count": len(observations),
|
||||
"roi_count": len(roi_frame.rois),
|
||||
"not_projected_observations": [
|
||||
item.to_dict() for item in roi_frame.not_projected_observations
|
||||
],
|
||||
"rois": frame_rois,
|
||||
}
|
||||
)
|
||||
total_observations += len(observations)
|
||||
total_rois += len(roi_frame.rois)
|
||||
identity = {
|
||||
"schema_version": SCHEMA_VERSION,
|
||||
"detector_result_id": detector.result_id,
|
||||
"geometry_profile_id": geometry_profile.profile_id,
|
||||
"geometry_profile_sha256": _sha256(geometry_profile_path),
|
||||
"roi_producer_sha256": _sha256(
|
||||
root / "src/k1link/perception/geometry_semantic_roi.py"
|
||||
),
|
||||
"frame_indices": list(FRAME_INDICES),
|
||||
"roi_profile": {
|
||||
"minimum_projected_points": roi_profile.minimum_projected_points,
|
||||
"minimum_crop_width": roi_profile.minimum_crop_width,
|
||||
"minimum_crop_height": roi_profile.minimum_crop_height,
|
||||
"padding_fraction": roi_profile.padding_fraction,
|
||||
"minimum_padding_pixels": roi_profile.minimum_padding_pixels,
|
||||
},
|
||||
"geometry_observation_count": total_observations,
|
||||
"roi_count": total_rois,
|
||||
"not_projected_observation_count": total_observations - total_rois,
|
||||
"frames": frame_documents,
|
||||
"artifacts": artifacts,
|
||||
"authority": _false_authority(),
|
||||
}
|
||||
digest = hashlib.sha256(_canonical_json(identity)).hexdigest()
|
||||
manifest = {
|
||||
"schema_version": SCHEMA_VERSION,
|
||||
"package_id": f"m48s-geometry-semantic-rois-{digest}",
|
||||
"identity_sha256": digest,
|
||||
"identity": identity,
|
||||
}
|
||||
(staging / "manifest.json").write_bytes(_canonical_json(manifest) + b"\n")
|
||||
staging.rename(output)
|
||||
except Exception:
|
||||
shutil.rmtree(staging, ignore_errors=True)
|
||||
raise
|
||||
print(json.dumps(manifest, ensure_ascii=False, indent=2, sort_keys=True))
|
||||
return 0
|
||||
|
||||
|
||||
def _packet(envelope: SourceEnvelope) -> SourcePacket:
|
||||
image = RecordedFrameReference(BASELINE_RECORDED_JOB_ID, envelope.sequence)
|
||||
geometry = RecordedFrameReference(RECORDED_SOURCE_PACK_ID, envelope.sequence)
|
||||
return SourcePacket(
|
||||
envelope=envelope,
|
||||
image_payload=image,
|
||||
registered_point_increment_payload=geometry,
|
||||
pose_payload=geometry,
|
||||
)
|
||||
|
||||
|
||||
def _canonical_json(value: object) -> bytes:
|
||||
return json.dumps(value, sort_keys=True, separators=(",", ":"), allow_nan=False).encode()
|
||||
|
||||
|
||||
def _sha256(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.resolve(strict=True).open("rb") as source:
|
||||
for chunk in iter(lambda: source.read(1024 * 1024), b""):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def _false_authority() -> dict[str, bool]:
|
||||
return {
|
||||
"ground_truth": False,
|
||||
"independent_truth": False,
|
||||
"candidate_accepted": False,
|
||||
"commands_enabled": False,
|
||||
"actuation_allowed": False,
|
||||
"navigation_or_safety_accepted": False,
|
||||
}
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,91 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Generate bounded prompt-group specs for Mask Grounding DINO Worker replay."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import copy
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
|
||||
PROMPT_SUFFIXES = {
|
||||
"urban-static/v0": "static",
|
||||
"urban-agents/v0": "agents",
|
||||
"urban-vehicles/v0": "vehicles",
|
||||
}
|
||||
AGENT_PROBES = {
|
||||
"adult": "adult person",
|
||||
"child": "child",
|
||||
"dog": "dog",
|
||||
}
|
||||
|
||||
|
||||
def main() -> int:
|
||||
repository = Path(__file__).resolve().parents[2]
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--repository-root", type=Path, default=repository)
|
||||
parser.add_argument(
|
||||
"--output-root",
|
||||
type=Path,
|
||||
default=(
|
||||
repository
|
||||
/ ".runtime/compute-experiments/m48s-semantic-shadow"
|
||||
/ "mask-grounding-dino-worker-specs-v0"
|
||||
),
|
||||
)
|
||||
arguments = parser.parse_args()
|
||||
root = arguments.repository_root.resolve(strict=True)
|
||||
output = arguments.output_root.expanduser().absolute()
|
||||
if output.exists():
|
||||
raise RuntimeError("Mask Grounding DINO Worker specs already exist")
|
||||
base = yaml.safe_load(
|
||||
(root / "config/perception/m48s-mask-grounding-dino-shadow-v0.yaml").read_text(
|
||||
"utf-8"
|
||||
)
|
||||
)
|
||||
profile = json.loads(
|
||||
(root / "config/perception/open-vocabulary-semantic-shadow-v0.json").read_text(
|
||||
"utf-8"
|
||||
)
|
||||
)
|
||||
if not isinstance(base, dict) or not isinstance(profile, dict):
|
||||
raise RuntimeError("M48S mask semantic source configs are incompatible")
|
||||
groups = profile.get("prompt_groups")
|
||||
if not isinstance(groups, list):
|
||||
raise RuntimeError("M48S semantic prompt groups are unavailable")
|
||||
output.mkdir(mode=0o700, parents=True)
|
||||
for raw_group in groups:
|
||||
if not isinstance(raw_group, dict):
|
||||
raise RuntimeError("M48S semantic prompt group is incompatible")
|
||||
prompt_id = raw_group.get("prompt_set_id")
|
||||
captions = raw_group.get("captions")
|
||||
if prompt_id not in PROMPT_SUFFIXES or not isinstance(captions, list):
|
||||
raise RuntimeError("M48S semantic prompt group escaped the bounded profile")
|
||||
suffix = PROMPT_SUFFIXES[prompt_id]
|
||||
spec = copy.deepcopy(base)
|
||||
spec["model_name"] = f"missioncore-m48s-mask-grounding-dino-{suffix}"
|
||||
spec["results_dir"] = f"/workspace/probe/results-{suffix}"
|
||||
spec["dataset"]["infer_data_sources"]["captions"] = captions
|
||||
spec["inference"]["results_dir"] = f"/workspace/probe/results-{suffix}"
|
||||
(output / f"mask-{suffix}.yaml").write_text(
|
||||
yaml.safe_dump(spec, sort_keys=False),
|
||||
"utf-8",
|
||||
)
|
||||
for suffix, caption in AGENT_PROBES.items():
|
||||
spec = copy.deepcopy(base)
|
||||
spec["model_name"] = f"missioncore-m48s-mask-grounding-dino-probe-{suffix}"
|
||||
spec["results_dir"] = f"/workspace/probe/results-{suffix}"
|
||||
spec["dataset"]["infer_data_sources"]["captions"] = [caption]
|
||||
spec["inference"]["results_dir"] = f"/workspace/probe/results-{suffix}"
|
||||
(output / f"mask-{suffix}.yaml").write_text(
|
||||
yaml.safe_dump(spec, sort_keys=False),
|
||||
"utf-8",
|
||||
)
|
||||
print(output)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,72 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Build the bounded geometry-first M48S semantic shadow result."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
from k1link.perception.geometry_semantic_shadow_replay import (
|
||||
build_geometry_semantic_shadow_replay,
|
||||
)
|
||||
|
||||
FRAME_INDICES = (121, 131, 253, 275, 443, 463, 1094, 1228, 1454, 1856, 2386)
|
||||
DETECTOR_RESULT_ID = (
|
||||
"m4-detector-replay-11f83f2e0b81758ac2a5a5fc54e9d293b501678df5f6ef97b5c6069ba08605c5"
|
||||
)
|
||||
VALID_FOV_RESULT_ID = (
|
||||
"valid-fov-mask-b4dd8ddf2b87c1d520ee8a0868c4fea062d7c14d1bae73ccabd3abe1f3acbac2"
|
||||
)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
root = Path(__file__).resolve().parents[2]
|
||||
runtime = root / ".runtime/compute-experiments/m48s-semantic-shadow"
|
||||
worker = runtime / "worker-results/geometry-roi58-20260825T0804Z"
|
||||
result = build_geometry_semantic_shadow_replay(
|
||||
repository_root=root,
|
||||
profile_path=root / "config/perception/open-vocabulary-semantic-shadow-v0.json",
|
||||
vocabulary_path=root / "config/perception/object-semantic-vocabulary-v0.json",
|
||||
detector_result_root=root / ".runtime/worker-results" / DETECTOR_RESULT_ID,
|
||||
roi_package_root=runtime / "geometry-rois-v1",
|
||||
valid_fov_mask_path=(
|
||||
root / ".runtime/compute-experiments/e1/valid-fov" / VALID_FOV_RESULT_ID / "mask.png"
|
||||
),
|
||||
worker_result_roots={
|
||||
"urban-static/v0": worker / "results-static/trt_inference",
|
||||
"urban-agents/v0": worker / "results-agents/trt_inference",
|
||||
"urban-vehicles/v0": worker / "results-vehicles/trt_inference",
|
||||
},
|
||||
frame_indices=FRAME_INDICES,
|
||||
worker_execution={
|
||||
"worker_node": "DESKTOP-OPJ8J04",
|
||||
"gpu_name": "NVIDIA GeForce RTX 4090",
|
||||
"container_reference": "nvcr.io/nvidia/tao/tao-toolkit:7.0.1-deploy",
|
||||
"container_image_id": (
|
||||
"sha256:2a3095330dd83e4314aada21fc7d184fcdb442a9131d591e5593ca5497394e7a"
|
||||
),
|
||||
"remote_root": (
|
||||
"D:/NDC_MISSIONCORE/runtime/experiments/"
|
||||
"m48s-geometry-roi58-20260825T0804Z"
|
||||
),
|
||||
"network_observation": (
|
||||
"TAO performed Hugging Face metadata requests despite a populated cache"
|
||||
),
|
||||
"authority": {
|
||||
"ground_truth": False,
|
||||
"independent_truth": False,
|
||||
"candidate_accepted": False,
|
||||
"commands_enabled": False,
|
||||
"actuation_allowed": False,
|
||||
"navigation_or_safety_accepted": False,
|
||||
},
|
||||
},
|
||||
output_root=runtime / "geometry-first-results",
|
||||
)
|
||||
print(result.result_id)
|
||||
print(json.dumps(result.metrics, ensure_ascii=False, indent=2, sort_keys=True))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,323 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Build immutable Mask Grounding DINO × LiDAR shadow evidence for M48S."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
from collections import Counter
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
from typing import Final
|
||||
|
||||
from k1link.perception.detector_replay_result import (
|
||||
read_detector_replay_result,
|
||||
require_m4_detector_replay_acceptance,
|
||||
)
|
||||
from k1link.perception.geometry import Ravnoves00GeometryAssociationProvider, RecordedGeometryStore
|
||||
from k1link.perception.geometry_math import project_map_points_kb4
|
||||
from k1link.perception.mask_grounding_semantics import (
|
||||
MaskBindingResolution,
|
||||
MaskGeometryBindingProfile,
|
||||
MaskGroundingDetection,
|
||||
MaskLabelResolution,
|
||||
bind_mask_instances_to_geometry,
|
||||
cluster_mask_instances,
|
||||
load_mask_grounding_evidence,
|
||||
resolve_mask_instance_label,
|
||||
)
|
||||
from k1link.perception.semantic_shadow_replay import semantic_replay_packet
|
||||
|
||||
SCHEMA: Final = "missioncore.m48s-mask-grounding-dino-shadow-analysis/v0"
|
||||
FRAME_INDICES: Final = (121, 131, 253, 275, 443, 463, 1094, 1228, 1454, 1856, 2386)
|
||||
DETECTOR_RESULT_ID: Final = (
|
||||
"m4-detector-replay-11f83f2e0b81758ac2a5a5fc54e9d293b501678df5f6ef97b5c6069ba08605c5"
|
||||
)
|
||||
GROUPS: Final = {
|
||||
"static": "urban-static/v0",
|
||||
"agents": "urban-agents/v0",
|
||||
"vehicles": "urban-vehicles/v0",
|
||||
}
|
||||
AUTHORITY: Final = {
|
||||
"ground_truth": False,
|
||||
"independent_truth": False,
|
||||
"candidate_accepted": False,
|
||||
"commands_enabled": False,
|
||||
"actuation_allowed": False,
|
||||
"navigation_or_safety_accepted": False,
|
||||
}
|
||||
|
||||
|
||||
def main() -> int:
|
||||
root = Path(__file__).resolve().parents[2]
|
||||
runtime = root / ".runtime/compute-experiments/m48s-semantic-shadow"
|
||||
split_root = (
|
||||
runtime
|
||||
/ "worker-results/mask-grounding-dino-split11-20260825T0833Z"
|
||||
).resolve(strict=True)
|
||||
agent_root = (
|
||||
runtime
|
||||
/ "worker-results/mask-grounding-dino-agent2-20260825T0836Z"
|
||||
).resolve(strict=True)
|
||||
profile_path = root / "config/perception/m48s-mask-grounding-dino-evidence-v0.json"
|
||||
profile_document = json.loads(profile_path.read_text("utf-8"))
|
||||
binding_profile = MaskGeometryBindingProfile(**profile_document["binding"])
|
||||
detector = read_detector_replay_result(
|
||||
root / ".runtime/worker-results" / DETECTOR_RESULT_ID
|
||||
)
|
||||
require_m4_detector_replay_acceptance(detector)
|
||||
detector_by_sequence = {item.sequence: item for item in detector.frames}
|
||||
store = RecordedGeometryStore.from_repository(root)
|
||||
geometry = Ravnoves00GeometryAssociationProvider(store=store)
|
||||
totals: Counter[str] = Counter()
|
||||
geometry_resolutions: Counter[str] = Counter()
|
||||
label_resolutions: Counter[str] = Counter()
|
||||
combined_resolutions: Counter[str] = Counter()
|
||||
frame_documents = []
|
||||
input_artifacts: dict[str, str] = {}
|
||||
evidence_writer_sha256 = _matching_evidence_writer_sha256(split_root, agent_root)
|
||||
for frame_index in FRAME_INDICES:
|
||||
detector_frame = detector_by_sequence[frame_index]
|
||||
packet = semantic_replay_packet(detector_frame.envelope)
|
||||
observations = geometry.associate(packet, ())
|
||||
geometry_frame = store.frame(packet)
|
||||
if geometry_frame is None:
|
||||
raise RuntimeError("selected M48S geometry frame is unavailable")
|
||||
projected = project_map_points_kb4(
|
||||
geometry_frame.points_map,
|
||||
position_map_xyz=geometry_frame.sensor_position_map,
|
||||
orientation_map_from_lidar_xyzw=geometry_frame.sensor_orientation_xyzw,
|
||||
profile=geometry_frame.projection,
|
||||
)
|
||||
detections: list[MaskGroundingDetection] = []
|
||||
source_file_sha256: str | None = None
|
||||
group_counts = {}
|
||||
for suffix, prompt_set_id in GROUPS.items():
|
||||
evidence_path = (
|
||||
split_root / f"results-{suffix}/masks/frame-{frame_index:06d}.npz"
|
||||
)
|
||||
evidence = load_mask_grounding_evidence(
|
||||
evidence_path,
|
||||
prompt_set_id=prompt_set_id,
|
||||
)
|
||||
if source_file_sha256 not in (None, evidence.source_file_sha256):
|
||||
raise RuntimeError("Mask Grounding DINO groups used different source rasters")
|
||||
source_file_sha256 = evidence.source_file_sha256
|
||||
detections.extend(evidence.detections)
|
||||
group_counts[prompt_set_id] = len(evidence.detections)
|
||||
input_artifacts[str(evidence_path.relative_to(root))] = _sha256(evidence_path)
|
||||
for name in ("experiment.yaml", "status.json"):
|
||||
artifact = split_root / f"results-{suffix}" / name
|
||||
input_artifacts[str(artifact.relative_to(root))] = _sha256(artifact)
|
||||
instances = cluster_mask_instances(tuple(detections), profile=binding_profile)
|
||||
bindings = bind_mask_instances_to_geometry(
|
||||
instances,
|
||||
observations=observations,
|
||||
projected=projected,
|
||||
profile=binding_profile,
|
||||
)
|
||||
decisions = tuple(
|
||||
resolve_mask_instance_label(item, profile=binding_profile) for item in instances
|
||||
)
|
||||
instance_documents = []
|
||||
for instance, binding, decision in zip(
|
||||
instances, bindings, decisions, strict=True
|
||||
):
|
||||
geometry_resolutions[binding.resolution.value] += 1
|
||||
label_resolutions[decision.resolution.value] += 1
|
||||
combined = _combined_resolution(binding.resolution, decision.resolution)
|
||||
combined_resolutions[combined] += 1
|
||||
instance_documents.append(
|
||||
{
|
||||
"instance_id": instance.instance_id,
|
||||
"member_detection_ids": [
|
||||
item.detection_id for item in instance.detections
|
||||
],
|
||||
"member_mask_sha256": sorted(
|
||||
{item.mask_sha256 for item in instance.detections}
|
||||
),
|
||||
"ranked_labels": [
|
||||
{"label": label, "confidence": confidence}
|
||||
for label, confidence in decision.ranked_labels
|
||||
],
|
||||
"geometry_binding": {
|
||||
"resolution": binding.resolution.value,
|
||||
"selected_observation_id": binding.selected_observation_id,
|
||||
"reason_code": binding.reason_code,
|
||||
"supports": [
|
||||
{
|
||||
"observation_id": item.observation_id,
|
||||
"projected_point_count": item.projected_point_count,
|
||||
"support_fraction": item.support_fraction,
|
||||
}
|
||||
for item in binding.supports
|
||||
],
|
||||
},
|
||||
"label_decision": {
|
||||
"resolution": decision.resolution.value,
|
||||
"selected_label": decision.selected_label,
|
||||
"reason_code": decision.reason_code,
|
||||
},
|
||||
"combined_resolution": combined,
|
||||
"authority": AUTHORITY,
|
||||
}
|
||||
)
|
||||
totals["frame_count"] += 1
|
||||
totals["geometry_observation_count"] += len(observations)
|
||||
totals["raw_detection_count"] += len(detections)
|
||||
totals["mask_instance_count"] += len(instances)
|
||||
frame_documents.append(
|
||||
{
|
||||
"frame_index": frame_index,
|
||||
"frame_id": detector_frame.envelope.frame_id,
|
||||
"source_file_sha256": source_file_sha256,
|
||||
"geometry_observation_count": len(observations),
|
||||
"group_detection_counts": group_counts,
|
||||
"instances": instance_documents,
|
||||
"authority": AUTHORITY,
|
||||
}
|
||||
)
|
||||
agent_probe = _agent_probe(agent_root, root=root, input_artifacts=input_artifacts)
|
||||
metrics = {
|
||||
"frames": {
|
||||
"requested": len(FRAME_INDICES),
|
||||
"completed": totals["frame_count"],
|
||||
},
|
||||
"geometry_observation_count": totals["geometry_observation_count"],
|
||||
"raw_detection_count": totals["raw_detection_count"],
|
||||
"mask_instance_count": totals["mask_instance_count"],
|
||||
"geometry_binding_resolution_counts": dict(sorted(geometry_resolutions.items())),
|
||||
"label_resolution_counts": dict(sorted(label_resolutions.items())),
|
||||
"combined_resolution_counts": dict(sorted(combined_resolutions.items())),
|
||||
"single_class_agent_probe": agent_probe,
|
||||
"authority": AUTHORITY,
|
||||
}
|
||||
frames_bytes = b"".join(
|
||||
_canonical_json(item) + b"\n" for item in frame_documents
|
||||
)
|
||||
identity = {
|
||||
"schema_version": SCHEMA,
|
||||
"profile_sha256": _sha256(profile_path),
|
||||
"detector_result_id": detector.result_id,
|
||||
"frame_indices": list(FRAME_INDICES),
|
||||
"worker_input_artifacts": dict(sorted(input_artifacts.items())),
|
||||
"evidence_writer_sha256": evidence_writer_sha256,
|
||||
"producer_sha256": {
|
||||
"mask_grounding_semantics.py": _sha256(
|
||||
root / "src/k1link/perception/mask_grounding_semantics.py"
|
||||
),
|
||||
"run_m48s_mask_grounding_dino_analysis.py": _sha256(Path(__file__))
|
||||
},
|
||||
"frames_sha256": hashlib.sha256(frames_bytes).hexdigest(),
|
||||
"metrics": metrics,
|
||||
"completed": totals["frame_count"] == len(FRAME_INDICES),
|
||||
"accepted": False,
|
||||
"authority": AUTHORITY,
|
||||
}
|
||||
result_id = "m48s-mask-grounding-dino-shadow-" + hashlib.sha256(
|
||||
_canonical_json(identity)
|
||||
).hexdigest()
|
||||
output = runtime / "mask-grounding-dino-results" / result_id
|
||||
if output.exists():
|
||||
raise RuntimeError("immutable Mask Grounding DINO result already exists")
|
||||
output.mkdir(mode=0o700, parents=True)
|
||||
(output / "frames.jsonl").write_bytes(frames_bytes)
|
||||
manifest = {"result_id": result_id, **identity}
|
||||
(output / "manifest.json").write_bytes(_canonical_json(manifest) + b"\n")
|
||||
report = {
|
||||
"schema_version": SCHEMA,
|
||||
"result_id": result_id,
|
||||
"created_at_utc": datetime.now(UTC).isoformat(timespec="milliseconds").replace(
|
||||
"+00:00", "Z"
|
||||
),
|
||||
"completed": identity["completed"],
|
||||
"accepted": False,
|
||||
"metrics": metrics,
|
||||
"decision": {
|
||||
"mask_instance_geometry_binding_completed": True,
|
||||
"semantic_quality_accepted": False,
|
||||
"agent_semantics_accepted": False,
|
||||
"reason_codes": [
|
||||
"single-class-adult-child-dog-probes-returned-zero-masks",
|
||||
"geometry-supported-instances-retained-conflicting-class-labels",
|
||||
"two-reviewer-independent-truth-unavailable",
|
||||
"research-checkpoint-not-for-commercial-deployment",
|
||||
],
|
||||
"next_candidate": (
|
||||
"geometry-prompted-mask-segmentation-plus-independent-zero-shot-"
|
||||
"crop-classifier"
|
||||
),
|
||||
},
|
||||
"authority": AUTHORITY,
|
||||
}
|
||||
(output / "report.json").write_bytes(_canonical_json(report) + b"\n")
|
||||
print(result_id)
|
||||
print(json.dumps(metrics, ensure_ascii=False, indent=2, sort_keys=True))
|
||||
return 0
|
||||
|
||||
|
||||
def _combined_resolution(
|
||||
geometry: MaskBindingResolution,
|
||||
label: MaskLabelResolution,
|
||||
) -> str:
|
||||
if geometry is MaskBindingResolution.UNRESOLVED:
|
||||
return "unresolved"
|
||||
if geometry is MaskBindingResolution.AMBIGUOUS:
|
||||
return "ambiguous"
|
||||
return label.value
|
||||
|
||||
|
||||
def _agent_probe(
|
||||
agent_root: Path,
|
||||
*,
|
||||
root: Path,
|
||||
input_artifacts: dict[str, str],
|
||||
) -> dict[str, object]:
|
||||
counts: Counter[str] = Counter()
|
||||
for label in ("adult", "child", "dog"):
|
||||
for frame_index in (253, 1228):
|
||||
path = agent_root / f"results-{label}/masks/frame-{frame_index:06d}.npz"
|
||||
evidence = load_mask_grounding_evidence(
|
||||
path,
|
||||
prompt_set_id=f"agent-probe-{label}/v0",
|
||||
)
|
||||
counts[label] += len(evidence.detections)
|
||||
input_artifacts[str(path.relative_to(root))] = _sha256(path)
|
||||
for name in ("experiment.yaml", "status.json"):
|
||||
artifact = agent_root / f"results-{label}" / name
|
||||
input_artifacts[str(artifact.relative_to(root))] = _sha256(artifact)
|
||||
return {
|
||||
"frame_indices": [253, 1228],
|
||||
"detection_counts": dict(sorted(counts.items())),
|
||||
"passed": False,
|
||||
}
|
||||
|
||||
|
||||
def _matching_evidence_writer_sha256(split_root: Path, agent_root: Path) -> str:
|
||||
split = _sha256(split_root / "sitecustomize.py")
|
||||
agent = _sha256(agent_root / "sitecustomize.py")
|
||||
if split != agent:
|
||||
raise RuntimeError("Mask Grounding DINO evidence writers changed between runs")
|
||||
return split
|
||||
|
||||
|
||||
def _sha256(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as stream:
|
||||
for chunk in iter(lambda: stream.read(1024 * 1024), b""):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def _canonical_json(value: object) -> bytes:
|
||||
return json.dumps(
|
||||
value,
|
||||
ensure_ascii=False,
|
||||
sort_keys=True,
|
||||
separators=(",", ":"),
|
||||
allow_nan=False,
|
||||
).encode("utf-8")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,114 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Persist raw Mask Grounding DINO tensors before TAO visualization.
|
||||
|
||||
This Worker-only wrapper leaves NVIDIA's inference and postprocessing intact. It
|
||||
intercepts the documented OD drawing seam solely to persist the already-filtered
|
||||
class, score, box and binary-mask evidence in a lossless NPZ ledger.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import numpy as np
|
||||
from nvidia_tao_deploy.cv.mask_grounding_dino.entrypoint.mask_grounding_dino import main
|
||||
from nvidia_tao_deploy.cv.mask_grounding_dino.inferencer import MaskGDINOInferencer
|
||||
from PIL import Image
|
||||
|
||||
INPUT_ROOT = Path(os.environ.get("M48S_INPUT_ROOT", "/workspace/probe/input"))
|
||||
MASK_ROOT = Path(os.environ.get("M48S_MASK_ROOT", "/workspace/probe/results/masks"))
|
||||
|
||||
|
||||
def _pixel_sha256(image: Image.Image) -> str:
|
||||
digest = hashlib.sha256()
|
||||
normalized = image.convert("RGB")
|
||||
digest.update(normalized.width.to_bytes(4, "big"))
|
||||
digest.update(normalized.height.to_bytes(4, "big"))
|
||||
digest.update(normalized.tobytes())
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def _source_index() -> dict[str, tuple[str, str]]:
|
||||
result: dict[str, tuple[str, str]] = {}
|
||||
for path in sorted(INPUT_ROOT.glob("*.png")):
|
||||
with Image.open(path) as image:
|
||||
pixel_sha256 = _pixel_sha256(image)
|
||||
file_sha256 = hashlib.sha256(path.read_bytes()).hexdigest()
|
||||
if pixel_sha256 in result:
|
||||
raise RuntimeError("M48S mask input rasters are duplicated")
|
||||
result[pixel_sha256] = (path.stem, file_sha256)
|
||||
if not result:
|
||||
raise RuntimeError("M48S mask input root is empty")
|
||||
return result
|
||||
|
||||
|
||||
SOURCE_INDEX = _source_index()
|
||||
ORIGINAL_DRAW_BBOX = MaskGDINOInferencer.draw_bbox
|
||||
|
||||
|
||||
def _evidence_draw_bbox(
|
||||
self: MaskGDINOInferencer,
|
||||
image: Image.Image,
|
||||
prediction: np.ndarray[Any, Any],
|
||||
masks: np.ndarray[Any, Any],
|
||||
class_mapping: dict[int, str],
|
||||
threshold: float = 0.3,
|
||||
color_map: dict[str, object] | None = None,
|
||||
) -> tuple[Image.Image, list[str]]:
|
||||
source = SOURCE_INDEX.get(_pixel_sha256(image))
|
||||
if source is None:
|
||||
raise RuntimeError("M48S mask inference image escaped the admitted input set")
|
||||
stem, source_file_sha256 = source
|
||||
if masks.ndim != 3 or masks.shape[-1] != prediction.shape[0]:
|
||||
raise RuntimeError("M48S mask output shape is incompatible")
|
||||
selected = tuple(
|
||||
index
|
||||
for index, item in enumerate(prediction)
|
||||
if int(item[0]) in class_mapping and float(item[1]) >= threshold
|
||||
)
|
||||
class_ids = np.asarray([int(prediction[index, 0]) for index in selected], dtype=np.int16)
|
||||
class_names = np.asarray([class_mapping[int(item)] for item in class_ids], dtype="U128")
|
||||
scores = np.asarray([prediction[index, 1] for index in selected], dtype=np.float32)
|
||||
boxes = np.asarray(
|
||||
[prediction[index, 2:6] for index in selected], dtype=np.float32
|
||||
).reshape((-1, 4))
|
||||
if selected:
|
||||
binary_masks = np.transpose(masks[..., selected] > 0.5, (2, 0, 1)).astype(np.uint8)
|
||||
else:
|
||||
binary_masks = np.empty((0, masks.shape[0], masks.shape[1]), dtype=np.uint8)
|
||||
MASK_ROOT.mkdir(mode=0o700, parents=True, exist_ok=True)
|
||||
destination = MASK_ROOT / f"{stem}.npz"
|
||||
if destination.exists():
|
||||
raise RuntimeError("M48S mask evidence destination already exists")
|
||||
np.savez_compressed(
|
||||
destination,
|
||||
schema_version=np.asarray("missioncore.m48s-mask-grounding-dino-evidence/v0"),
|
||||
source_file_sha256=np.asarray(source_file_sha256),
|
||||
source_pixel_sha256=np.asarray(_pixel_sha256(image)),
|
||||
class_ids=class_ids,
|
||||
class_names=class_names,
|
||||
scores=scores,
|
||||
boxes_xyxy=boxes,
|
||||
masks=binary_masks,
|
||||
)
|
||||
return ORIGINAL_DRAW_BBOX(
|
||||
self,
|
||||
image,
|
||||
prediction,
|
||||
masks,
|
||||
class_mapping,
|
||||
threshold,
|
||||
color_map,
|
||||
)
|
||||
|
||||
|
||||
MaskGDINOInferencer.draw_bbox = _evidence_draw_bbox
|
||||
print("M48S raw-mask evidence hook installed", file=sys.stderr)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,78 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Build the bounded raw-KB4 M48S semantic shadow result."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
from k1link.perception.semantic_shadow_replay import build_semantic_shadow_replay
|
||||
|
||||
FRAME_INDICES = (121, 131, 253, 275, 443, 463, 1094, 1228, 1454, 1856, 2386)
|
||||
DETECTOR_RESULT_ID = (
|
||||
"m4-detector-replay-11f83f2e0b81758ac2a5a5fc54e9d293b501678df5f6ef97b5c6069ba08605c5"
|
||||
)
|
||||
VALID_FOV_RESULT_ID = (
|
||||
"valid-fov-mask-b4dd8ddf2b87c1d520ee8a0868c4fea062d7c14d1bae73ccabd3abe1f3acbac2"
|
||||
)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
repository = Path(__file__).resolve().parents[2]
|
||||
default_runtime = (
|
||||
repository
|
||||
/ ".runtime/compute-experiments/m48s-semantic-shadow/worker-results"
|
||||
/ "fovfill11-20260825T0755Z"
|
||||
)
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--repository-root", type=Path, default=repository)
|
||||
parser.add_argument("--worker-results", type=Path, default=default_runtime)
|
||||
parser.add_argument(
|
||||
"--source-frames",
|
||||
type=Path,
|
||||
default=(repository / ".runtime/compute-experiments/m48s-semantic-shadow/raw-11-frames-v1"),
|
||||
)
|
||||
parser.add_argument(
|
||||
"--inference-frames",
|
||||
type=Path,
|
||||
default=(
|
||||
repository
|
||||
/ ".runtime/compute-experiments/m48s-semantic-shadow"
|
||||
/ "raw-11-valid-fov-fill-v1"
|
||||
),
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output-root",
|
||||
type=Path,
|
||||
default=(repository / ".runtime/compute-experiments/m48s-semantic-shadow/results"),
|
||||
)
|
||||
arguments = parser.parse_args()
|
||||
root = arguments.repository_root.resolve(strict=True)
|
||||
worker = arguments.worker_results.resolve(strict=True)
|
||||
result = build_semantic_shadow_replay(
|
||||
repository_root=root,
|
||||
profile_path=root / "config/perception/open-vocabulary-semantic-shadow-v0.json",
|
||||
vocabulary_path=root / "config/perception/object-semantic-vocabulary-v0.json",
|
||||
detector_result_root=root / ".runtime/worker-results" / DETECTOR_RESULT_ID,
|
||||
source_frames_root=arguments.source_frames,
|
||||
inference_frames_root=arguments.inference_frames,
|
||||
valid_fov_mask_path=(
|
||||
root / ".runtime/compute-experiments/e1/valid-fov" / VALID_FOV_RESULT_ID / "mask.png"
|
||||
),
|
||||
worker_result_roots={
|
||||
"urban-static/v0": worker / "results-static/trt_inference",
|
||||
"urban-agents/v0": worker / "results-agents/trt_inference",
|
||||
"urban-vehicles/v0": worker / "results-vehicles/trt_inference",
|
||||
},
|
||||
worker_identity_path=worker / "worker-identity.json",
|
||||
frame_indices=FRAME_INDICES,
|
||||
output_root=arguments.output_root,
|
||||
)
|
||||
print(result.result_id)
|
||||
print(json.dumps(result.metrics, ensure_ascii=False, indent=2, sort_keys=True))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
Reference in New Issue
Block a user