feat(perception): audit E36 second-source eligibility

This commit is contained in:
DCCONSTRUCTIONS
2026-07-27 18:27:40 +03:00
parent b894945344
commit fa20079c9e
9 changed files with 1213 additions and 1 deletions
@@ -643,6 +643,14 @@ with a complete `missioncore.laboratory-method/v1` manifest.
| **E35 — deterministic degradation and recovery** | Does the accepted pipeline fail safely when one source degrades? | Derive immutable replay variants for camera loss, LiDAR loss, pose staleness, delayed frames, bounded drops and timing offset. Preserve the original source and record every injected transformation. Measure status transitions, recovery time, queue accounting and false confidence. | Pass only if missing evidence becomes camera-only, geometry-only, stale or unknown; never guessed class, false free space or hidden success. |
| **E36 — second-source transfer gate** | Does the result survive another real capture rather than only RAVNOVES00? | First audit the source catalog for a second recording with the required camera, LiDAR, pose, time and calibration identities. Run the accepted E32E35 profile unchanged when such evidence exists. Public GOOSE/RELLIS data may qualify an algorithm but cannot substitute for this K1 transfer gate. | Compare the frozen profile without retuning. If no eligible source exists, keep the gate blocked only in the roadmap/Ops plan: do not publish an empty LAB page, fabricate a result or collect new data by default. |
The E36 catalog-admission audit is complete in immutable result
`e36-source-catalog-audit-c5a7b259f043c0a4f3be38f869af1e675dfaa43743e45fd4f43fac166d07f849`.
TEST007 passes the camera, LiDAR, pose, host-time, device-identity and minimum
camera-window checks, but lacks calibration and physical-mount identities bound
to its source session. The transfer replay remains blocked; no LAB E36 page,
retuning or automatic collection was created. ADR 0029 records the admission
contract and exact unblocking evidence.
The current E30 AI-assisted engineering generation is
`e30-engineering-generation-62a4fea10dea9b77f69ceac1af5bf0e4928d9c7716083c22258a03670fe5bd4f`.
It covers all `486` selected items: `403` confirmed, `81` corrected and `2`
@@ -209,7 +209,13 @@ Six deterministic variants each replay all 4,489 frames; the result records
against the `0.25 s` gate. Hidden success, false free-space, unsupported
semantic/metric claims, timing-mismatch `agree`, late-result return and
upstream changes are all zero. A0A8 are closed. A9/E36 is now the critical
path and requires an eligible second mounted real source.
path. Its immutable catalog-admission audit
`e36-source-catalog-audit-c5a7b259f043c0a4f3be38f869af1e675dfaa43743e45fd4f43fac166d07f849`
checked all 11 physical source rows. TEST007 is a real 206.073-second candidate
with camera, LiDAR, pose, host time and device identity, but no calibration or
mount identity is bound to that source session. A9 transfer therefore remains
blocked without publishing an empty LAB. ADR 0029 defines the exact unblocking
contract.
- [x] Reproduce all 4,489 immutable E29 frames with the exact frozen profile
before applying E31/E30 changes.
@@ -223,6 +229,12 @@ path and requires an eligible second mounted real source.
TrackGeometryFrame reconstruction/validation.
- [x] Package the exact E32 result, frozen runtime and profile as immutable
E33 worker input with complete artifact digests.
- [x] Audit all 11 physical source rows through the frozen E36 eligibility
contract without creating an empty LAB.
- [ ] Bind TEST007 calibration and physical mount identities to its exact
source session using primary evidence.
- [ ] Run and compare the frozen E32E35 profile on an eligible second source
without retuning.
- [x] Replay all 4,489 frames at recorded `1.0×` pace through independent
bounded latest-wins work/result queues.
- [x] Close every frame as delivered, input-superseded or result-superseded
@@ -0,0 +1,67 @@
# ADR 0029 — E36 second-source catalog admission
Date: 2026-07-27
Status: accepted; transfer replay blocked by missing source provenance
## Context
E31E35 are accepted only for the immutable RAVNOVES00 source session. E36 is
the first transfer gate: the exact accepted profile must run without retuning
on another real K1 capture. A second recording is not eligible merely because
it contains a point cloud or can be opened in the viewer. Camera, LiDAR, pose,
host time, device, calibration and physical mount identities must all be
bindable to that source.
The local source catalog contains historical recordings, including TEST007.
Publishing an empty LAB while no source is eligible would confuse a planning
blocker with an experimental result.
## Decision
1. Source admission is a separate content-addressed audit before any E36 replay.
2. The admission profile freezes the exact E31E35 result identities and
forbids retuning.
3. A candidate must be a ready physical K1 source with point cloud, trajectory,
video, exact host-time identity and at least 60 seconds of contiguous camera
evidence.
4. Device identity is retained only as irreversible hashes in the redacted
inventory.
5. Calibration and mount evidence must each bind explicitly to the candidate
`source_session_id`. A later snapshot from the same scanner and an assumed
handheld setup are not substitutes.
6. Public datasets cannot satisfy this mounted K1 transfer gate.
7. A blocked audit authorizes neither a new capture nor an E36 LAB
publication. Collection remains a separate operator decision.
8. The audit result has no command, navigation or safety authority.
## Catalog outcome
Immutable audit:
`e36-source-catalog-audit-c5a7b259f043c0a4f3be38f869af1e675dfaa43743e45fd4f43fac166d07f849`.
Eleven physical source records were audited. TEST007 is the closest candidate:
it is a ready distinct capture from the same accepted K1, has camera, LiDAR,
pose and exact host-time evidence, and contains a sealed 2,061-frame camera
window of approximately 206.073 seconds.
TEST007 is not yet eligible because:
- no calibration identity is bound to the TEST007 source session;
- no physical mount profile or mount identity is bound to that session.
CAM_REC_TEST also lacks those bindings and its longest contiguous camera window
is only 22.884 seconds, below the frozen minimum. The remaining sources do not
contain the required camera modality or accepted physical-source provenance.
## Consequences
- A9 has completed its catalog-admission sub-gate, but E36 transfer remains
blocked.
- The next admissible action is provenance recovery or a new explicit binding
artifact for TEST007, based only on primary evidence. It is not threshold
tuning and does not rerun E32E35.
- If both bindings become available, the same audit can be rerun. Only an
eligible result authorizes packaging the frozen E32E35 transfer replay.
- Until then, Mission Core publishes no LAB E36 page and claims no
generalization beyond RAVNOVES00.
@@ -0,0 +1,108 @@
# E36 source catalog admission audit
Date: 2026-07-27
Status: blocked — no eligible second mounted real source
Immutable audit:
`e36-source-catalog-audit-c5a7b259f043c0a4f3be38f869af1e675dfaa43743e45fd4f43fac166d07f849`
This is an engineering admission result, not a LAB E36 run. No transfer replay,
LAB catalog entry or interface was published.
## Question
Does the current Mission Core source catalog contain a second real K1 capture
on which the accepted E32E35 pipeline can run unchanged?
The gate freezes:
- E31 source qualification
`e31-source-qualification-b2460a5eb143688c7eea6821b2277e13aea79868abe81d83f7e78548c119159a`;
- E32 TrackGeometry
`e32-track-geometry-a14ca0e7fb3850ca0dfa3c41634e1b490a2d58ab74d101afc6d6921fbdb0e6fd`;
- E33 worker shadow
`e33-worker-shadow-05cc0bb264410fd49536df90e94067ac39731aff0322a8873700d40008a8bb3a`;
- E34 temporal occupied layer
`e34-temporal-occupied-8d9abb3f2cc072cfdbb16cc4e55798e05c35a0abe0b8f691096770e091573a73`;
- E35 degradation recovery
`e35-degradation-recovery-82bdbd5c5bfde6d932737f077153c3a8472c993c343fcbe8a207c39bfa2a6288`.
Retuning is forbidden.
## Admission rules
A candidate must have all of the following:
1. a distinct ready K1 source session;
2. camera, point-cloud and trajectory modalities;
3. at least 60 seconds of contiguous camera evidence;
4. an exact host-time identity;
5. a hashed physical device identity;
6. a calibration identity bound to that `source_session_id`;
7. a physical mount identity bound to that `source_session_id`.
Missing evidence fails closed. A later calibration snapshot or assumed mount
cannot be silently transferred to an earlier capture.
## Catalog result
Eleven physical source rows were audited.
| Source | Camera/LiDAR/pose/time | Longest camera window | Calibration bound | Mount bound | Decision |
| --- | --- | ---: | --- | --- | --- |
| RAVNOVES00 | yes | 448.8 s | qualified through E31 | unavailable | baseline, not a candidate |
| TEST007 | yes | 206.073 s | no | no | closest candidate; blocked |
| CAM_REC_TEST | yes | 22.884 s | no | no | blocked |
| TEST009 | no camera | — | no | no | blocked |
| Seven legacy sources | no accepted camera package | — | no | no | blocked |
TEST007 is a substantive candidate, not an empty recording. Its accepted source
package contains a sealed 2,061-frame camera epoch plus synchronous point-cloud,
pose and host-time evidence. Device-status evidence hashes to the same physical
K1 identity as RAVNOVES00. That proves the device identity, but it does not
prove which calibration and physical mount applied when TEST007 was recorded.
The two later factory-calibration snapshots have the same device identity and
calibration content, but neither manifest names TEST007 as its source session.
No source-bound mount profile, body transform or mount identity was found.
## Decision
`eligible_source_count = 0`.
E36 replay is not authorized. The exact remaining blockers for TEST007 are:
- `calibration-bound-to-source-session`;
- `mount-bound-to-source-session`.
The audit explicitly keeps these actions false:
- empty LAB publication;
- profile retuning;
- public-dataset substitution;
- automatic new capture;
- command, navigation or safety authority.
## Reproduction
```bash
PYTHONPATH=src .venv/bin/python \
experiments/perception/run_e36_source_catalog_audit.py \
--profile experiments/perception/e36_second_source_eligibility_profile.json \
--inventory experiments/perception/e36_source_catalog_inventory_2026-07-27.json \
--output-root .runtime/compute-experiments/e36/source-audits
```
Exit code `3` means the audited catalog is valid but the transfer gate is
blocked. Invalid profile, inventory or artifact integrity fails separately.
## Next admissible step
Search only primary evidence for a source-scoped TEST007 calibration and mount
binding. If both can be proven, publish those bindings as immutable provenance,
rerun this admission audit and then package E32E35 unchanged for transfer.
If they cannot be proven, A9 remains blocked until an operator separately
authorizes a new capture with calibration and mount identities recorded at
acquisition time.
@@ -0,0 +1,34 @@
{
"authority": {
"commands_enabled": false,
"navigation_or_safety_accepted": false,
"transfer_replay_authorized": false
},
"baseline": {
"frozen_upstream": {
"e31": "e31-source-qualification-b2460a5eb143688c7eea6821b2277e13aea79868abe81d83f7e78548c119159a",
"e32": "e32-track-geometry-a14ca0e7fb3850ca0dfa3c41634e1b490a2d58ab74d101afc6d6921fbdb0e6fd",
"e33": "e33-worker-shadow-05cc0bb264410fd49536df90e94067ac39731aff0322a8873700d40008a8bb3a",
"e34": "e34-temporal-occupied-8d9abb3f2cc072cfdbb16cc4e55798e05c35a0abe0b8f691096770e091573a73",
"e35": "e35-degradation-recovery-82bdbd5c5bfde6d932737f077153c3a8472c993c343fcbe8a207c39bfa2a6288"
},
"source_session_id": "20260720T065719Z_viewer_live"
},
"eligibility": {
"minimum_contiguous_camera_seconds": 60.0,
"require_device_identity": true,
"require_host_time_identity": true,
"require_session_bound_calibration": true,
"require_session_bound_mount": true,
"required_modalities": [
"point-cloud",
"trajectory",
"video"
],
"required_origin": "xgrids-k1.viewer-live.evidence",
"required_status": "ready",
"retuning_allowed": false
},
"profile_id": "e36-second-mounted-real-source/v1",
"schema_version": "missioncore.e36-source-eligibility-profile/v1"
}
@@ -0,0 +1,217 @@
{
"catalog_audited_at_utc": "2026-07-27T15:20:47Z",
"schema_version": "missioncore.e36-source-catalog-inventory/v1",
"sources": [
{
"calibration_binding": null,
"camera": null,
"device_identity": null,
"display_name": "20260715T192051Z_viewer_live",
"duration_seconds": 1915.012314,
"host_time_identity_sha256": null,
"modalities": [
"point-cloud",
"trajectory"
],
"mount_binding": null,
"origin": "legacy-viewer-live",
"raw_transport_sha256": "ace976ddfc6dd5059531167fa7b31705d62641d21dd7cc5ce145ceb49a8c51a4",
"session_id": "20260715T192051Z_viewer_live",
"status": "interrupted"
},
{
"calibration_binding": null,
"camera": null,
"device_identity": null,
"display_name": "20260715T222341Z_viewer_live",
"duration_seconds": null,
"host_time_identity_sha256": null,
"modalities": [],
"mount_binding": null,
"origin": "legacy-viewer-live",
"raw_transport_sha256": null,
"session_id": "20260715T222341Z_viewer_live",
"status": "failed"
},
{
"calibration_binding": null,
"camera": null,
"device_identity": null,
"display_name": "20260716T075132Z_viewer_live",
"duration_seconds": 860.643795,
"host_time_identity_sha256": null,
"modalities": [
"point-cloud",
"trajectory"
],
"mount_binding": null,
"origin": "legacy-viewer-live",
"raw_transport_sha256": "467cbfaf815e7fc73ac737cf102fa4a1a6e0be189ed12be847b98a8e0ed0b35f",
"session_id": "20260716T075132Z_viewer_live",
"status": "interrupted"
},
{
"calibration_binding": null,
"camera": null,
"device_identity": null,
"display_name": "20260716T144355Z_viewer_live",
"duration_seconds": 218.737673,
"host_time_identity_sha256": null,
"modalities": [
"point-cloud",
"trajectory"
],
"mount_binding": null,
"origin": "legacy-viewer-live",
"raw_transport_sha256": "abb2513fb0307e67815ce2a58d23d4327961844b1c811154a4eba54cc6939602",
"session_id": "20260716T144355Z_viewer_live",
"status": "ready"
},
{
"calibration_binding": null,
"camera": null,
"device_identity": null,
"display_name": "20260716T182745Z_viewer_live",
"duration_seconds": 2404.214195,
"host_time_identity_sha256": null,
"modalities": [
"point-cloud",
"trajectory"
],
"mount_binding": null,
"origin": "legacy-viewer-live",
"raw_transport_sha256": "654586831029228398fc9c082a88e3f2bdc49096a40049d3ebde3e3527d30329",
"session_id": "20260716T182745Z_viewer_live",
"status": "ready"
},
{
"calibration_binding": null,
"camera": null,
"device_identity": null,
"display_name": "20260716T191025Z_viewer_live",
"duration_seconds": 699.248385,
"host_time_identity_sha256": null,
"modalities": [
"point-cloud",
"trajectory"
],
"mount_binding": null,
"origin": "legacy-viewer-live",
"raw_transport_sha256": "85e0d1ce0820a1c74b8771aea7fcedd6cce39301d7d17bf404c848552133ef2d",
"session_id": "20260716T191025Z_viewer_live",
"status": "ready"
},
{
"calibration_binding": null,
"camera": null,
"device_identity": null,
"display_name": "20260716T205632Z_viewer_live",
"duration_seconds": 1450.282194,
"host_time_identity_sha256": null,
"modalities": [
"point-cloud",
"trajectory"
],
"mount_binding": null,
"origin": "legacy-viewer-live",
"raw_transport_sha256": "8f82e722ac252018e30139200f4122a6b2621f73594260ebfd4da22f96751d2d",
"session_id": "20260716T205632Z_viewer_live",
"status": "ready"
},
{
"calibration_binding": null,
"camera": {
"frame_count": 230,
"longest_contiguous_seconds": 22.884,
"source_id": "sensor.camera.right",
"status": "complete"
},
"device_identity": {
"device_serial_sha256": "74de39837b34d6db90c3dc583e4c58391fb4cb20a8b77ef8ca8f095471ab1c3a",
"vendor_device_id_sha256": "d9044906a5e5a0501d3db64869397a0307037cce163da72ad55c9b86d7f83fcd"
},
"display_name": "CAM_REC_TEST",
"duration_seconds": 96.918679,
"host_time_identity_sha256": "8722563749941e39cfc999bbc7d5faa85caddab6c381edf5ac76cc1963ea2e83",
"modalities": [
"point-cloud",
"trajectory",
"video"
],
"mount_binding": null,
"origin": "xgrids-k1.viewer-live.evidence",
"raw_transport_sha256": "97c10258c34abeec71d8214a051fb50a46e871ef361e52bb070eeeaf75fab583",
"session_id": "20260718T071709Z_viewer_live",
"status": "ready"
},
{
"calibration_binding": null,
"camera": {
"frame_count": 2061,
"longest_contiguous_seconds": 206.073,
"source_id": "sensor.camera.left",
"status": "sealed-interrupted-source-end"
},
"device_identity": {
"device_serial_sha256": "74de39837b34d6db90c3dc583e4c58391fb4cb20a8b77ef8ca8f095471ab1c3a",
"vendor_device_id_sha256": "d9044906a5e5a0501d3db64869397a0307037cce163da72ad55c9b86d7f83fcd"
},
"display_name": "TEST007",
"duration_seconds": 302.677438,
"host_time_identity_sha256": "0903dad5c7c5901722c91c014042e9aa71a2bd5a2d54a88f947415408449bb2f",
"modalities": [
"point-cloud",
"trajectory",
"video"
],
"mount_binding": null,
"origin": "xgrids-k1.viewer-live.evidence",
"raw_transport_sha256": "e7f60bf589b2885c68ae18b4bf78d5f79a0a7e95c18bca5f9c52d1d960636ed0",
"session_id": "20260718T201659Z_viewer_live",
"status": "ready"
},
{
"calibration_binding": null,
"camera": null,
"device_identity": null,
"display_name": "TEST009",
"duration_seconds": 90.63152,
"host_time_identity_sha256": "8c5c84ddaa742ee358df58d196bfc2bccb8878f5e56070711a6b60e8fc5f9bb0",
"modalities": [
"point-cloud",
"trajectory"
],
"mount_binding": null,
"origin": "xgrids-k1.viewer-live.evidence",
"raw_transport_sha256": "29f68cecfda38339146277914ab8a4db94f9c0a5f43a6018aa4871fafc2f9062",
"session_id": "20260719T220917Z_viewer_live",
"status": "interrupted"
},
{
"calibration_binding": null,
"camera": {
"frame_count": 4489,
"longest_contiguous_seconds": 448.8,
"source_id": "sensor.camera.right",
"status": "complete"
},
"device_identity": {
"device_serial_sha256": "74de39837b34d6db90c3dc583e4c58391fb4cb20a8b77ef8ca8f095471ab1c3a",
"vendor_device_id_sha256": "d9044906a5e5a0501d3db64869397a0307037cce163da72ad55c9b86d7f83fcd"
},
"display_name": "RAVNOVES00",
"duration_seconds": 535.71762,
"host_time_identity_sha256": "582d52b144fc8f72d11787692e44349cf8cfc36c14d0cca37e9c1344b75319c4",
"modalities": [
"point-cloud",
"trajectory",
"video"
],
"mount_binding": null,
"origin": "xgrids-k1.viewer-live.evidence",
"raw_transport_sha256": "70da0edad5cbf0e89b6e2355c0aac294f8c36a67249cd33b8918353fd88c83af",
"session_id": "20260720T065719Z_viewer_live",
"status": "ready"
}
]
}
@@ -0,0 +1,47 @@
#!/usr/bin/env python3
"""Build the immutable E36 second-source catalog audit."""
from __future__ import annotations
import argparse
import json
from pathlib import Path
from k1link.compute.e36_source_catalog_audit import (
build_e36_source_catalog_audit,
)
def main() -> int:
parser = argparse.ArgumentParser()
parser.add_argument("--profile", type=Path, required=True)
parser.add_argument("--inventory", type=Path, required=True)
parser.add_argument(
"--output-root",
type=Path,
default=Path(".runtime/compute-experiments/e36/source-audits"),
)
args = parser.parse_args()
result = build_e36_source_catalog_audit(
profile_path=args.profile,
inventory_path=args.inventory,
output_root=args.output_root,
)
print(
json.dumps(
{
"result_id": result.result_id,
"result_root": str(result.result_root),
"status": result.report["status"],
"eligible_source_ids": list(result.eligible_source_ids),
"next_gate": result.report["next_gate"],
},
ensure_ascii=False,
sort_keys=True,
)
)
return 0 if result.eligible_source_ids else 3
if __name__ == "__main__":
raise SystemExit(main())
@@ -0,0 +1,488 @@
"""Fail-closed E36 catalog admission for a second mounted real source.
The audit is deliberately separate from an E36 replay. It decides whether the
catalog contains a source on which the frozen E32--E35 profile may be run
without inventing calibration or mount provenance. A blocked audit is a real,
content-addressed engineering result, but it is not a LAB publication.
"""
from __future__ import annotations
import hashlib
import json
import math
import os
import re
import shutil
from dataclasses import dataclass
from pathlib import Path
from typing import Any, Final
from k1link.artifacts import utc_now_iso
E36_PROFILE_SCHEMA: Final = "missioncore.e36-source-eligibility-profile/v1"
E36_INVENTORY_SCHEMA: Final = "missioncore.e36-source-catalog-inventory/v1"
E36_AUDIT_SCHEMA: Final = "missioncore.e36-source-catalog-audit/v1"
E36_REPORT_SCHEMA: Final = "missioncore.e36-source-catalog-audit-report/v1"
E36_REPORT_NAME: Final = "audit-report.json"
E36_MANIFEST_NAME: Final = "manifest.json"
_RESULT_ID = re.compile(r"^e36-source-catalog-audit-[a-f0-9]{64}$")
_SHA256 = re.compile(r"^[a-f0-9]{64}$")
_SESSION_ID = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$")
class E36SourceCatalogAuditError(RuntimeError):
"""The E36 profile, inventory or immutable result is invalid."""
@dataclass(frozen=True, slots=True)
class E36SourceCatalogAudit:
result_root: Path
result_id: str
manifest: dict[str, Any]
report: dict[str, Any]
@property
def eligible_source_ids(self) -> tuple[str, ...]:
candidates = _array(self.report.get("candidates"), "E36 candidates")
return tuple(
_required_string(item, "session_id")
for item in candidates
if isinstance(item, dict) and item.get("eligible") is True
)
def build_e36_source_catalog_audit(
*,
profile_path: Path,
inventory_path: Path,
output_root: Path,
) -> E36SourceCatalogAudit:
"""Build or reopen one immutable source-catalog admission result."""
profile_path = profile_path.resolve(strict=True)
inventory_path = inventory_path.resolve(strict=True)
profile = _read_json(profile_path)
inventory = _read_json(inventory_path)
_validate_profile(profile)
_validate_inventory(inventory)
baseline = _object(profile.get("baseline"), "E36 baseline")
baseline_session_id = _required_string(baseline, "source_session_id")
candidates = _array(inventory.get("sources"), "E36 inventory sources")
source_ids = {
_required_string(_object(item, "E36 source"), "session_id")
for item in candidates
}
if baseline_session_id not in source_ids:
raise E36SourceCatalogAuditError("E36 inventory does not contain its baseline")
identity = {
"schema_version": E36_AUDIT_SCHEMA,
"profile": profile,
"profile_sha256": _sha256(profile_path),
"inventory_sha256": _sha256(inventory_path),
"producer_sha256": _sha256(Path(__file__)),
"authority": _authority(),
}
identity_sha256 = hashlib.sha256(_canonical_json(identity)).hexdigest()
result_id = f"e36-source-catalog-audit-{identity_sha256}"
destination = output_root.expanduser().absolute()
destination.mkdir(mode=0o700, parents=True, exist_ok=True)
result_root = destination / result_id
if result_root.exists():
return read_e36_source_catalog_audit(result_root)
report = _audit_sources(
result_id=result_id,
profile=profile,
inventory=inventory,
)
staging = destination / f".{result_id}.{os.getpid()}.incomplete"
staging.mkdir(mode=0o700, exist_ok=False)
try:
_write_json(staging / E36_REPORT_NAME, report)
report_artifact = _artifact(staging / E36_REPORT_NAME, "source-catalog-audit-report")
manifest = {
"schema_version": E36_AUDIT_SCHEMA,
"result_id": result_id,
"identity_sha256": identity_sha256,
"identity": identity,
"status": report["status"],
"eligible_source_ids": report["eligible_source_ids"],
"artifacts": [report_artifact],
"created_at_utc": utc_now_iso(),
"authority": _authority(),
}
_write_json(staging / E36_MANIFEST_NAME, manifest)
os.replace(staging, result_root)
except BaseException:
shutil.rmtree(staging, ignore_errors=True)
raise
return read_e36_source_catalog_audit(result_root)
def read_e36_source_catalog_audit(root: Path) -> E36SourceCatalogAudit:
"""Read and fully validate an immutable E36 catalog result."""
resolved = root.expanduser().resolve(strict=True)
if not resolved.is_dir() or _RESULT_ID.fullmatch(resolved.name) is None:
raise E36SourceCatalogAuditError("E36 result root is invalid")
manifest = _read_json(resolved / E36_MANIFEST_NAME)
report = _read_json(resolved / E36_REPORT_NAME)
if (
manifest.get("schema_version") != E36_AUDIT_SCHEMA
or manifest.get("result_id") != resolved.name
or report.get("schema_version") != E36_REPORT_SCHEMA
or report.get("result_id") != resolved.name
or manifest.get("status") != report.get("status")
or manifest.get("eligible_source_ids") != report.get("eligible_source_ids")
or manifest.get("authority") != _authority()
or report.get("authority") != _authority()
):
raise E36SourceCatalogAuditError("E36 manifest and report are inconsistent")
identity = _object(manifest.get("identity"), "E36 identity")
expected_identity_sha256 = hashlib.sha256(_canonical_json(identity)).hexdigest()
if (
manifest.get("identity_sha256") != expected_identity_sha256
or resolved.name != f"e36-source-catalog-audit-{expected_identity_sha256}"
):
raise E36SourceCatalogAuditError("E36 result identity is invalid")
artifacts = _array(manifest.get("artifacts"), "E36 artifacts")
if len(artifacts) != 1:
raise E36SourceCatalogAuditError("E36 result has an invalid artifact set")
artifact = _object(artifacts[0], "E36 report artifact")
if (
artifact.get("role") != "source-catalog-audit-report"
or artifact.get("path") != E36_REPORT_NAME
or artifact.get("sha256") != _sha256(resolved / E36_REPORT_NAME)
or artifact.get("byte_length") != (resolved / E36_REPORT_NAME).stat().st_size
):
raise E36SourceCatalogAuditError("E36 report artifact is invalid")
return E36SourceCatalogAudit(
result_root=resolved,
result_id=resolved.name,
manifest=manifest,
report=report,
)
def _audit_sources(
*,
result_id: str,
profile: dict[str, Any],
inventory: dict[str, Any],
) -> dict[str, Any]:
baseline = _object(profile.get("baseline"), "E36 baseline")
baseline_session_id = _required_string(baseline, "source_session_id")
eligibility = _object(profile.get("eligibility"), "E36 eligibility")
required_modalities = set(
_string_array(eligibility.get("required_modalities"), "required modalities")
)
required_status = _required_string(eligibility, "required_status")
minimum_camera_seconds = _number(
eligibility.get("minimum_contiguous_camera_seconds"),
"minimum camera seconds",
)
required_origin = _required_string(eligibility, "required_origin")
audited: list[dict[str, Any]] = []
for raw_source in _array(inventory.get("sources"), "E36 inventory sources"):
source = _object(raw_source, "E36 source")
session_id = _required_string(source, "session_id")
if session_id == baseline_session_id:
audited.append(
{
"session_id": session_id,
"display_name": _required_string(source, "display_name"),
"role": "baseline",
"eligible": False,
"requirements": {"distinct_from_baseline": False},
"blocker_codes": ["baseline-source"],
}
)
continue
modalities = set(_string_array(source.get("modalities"), "source modalities"))
camera = source.get("camera")
camera_window_seconds = (
0.0
if camera is None
else _number(
_object(camera, "camera evidence").get("longest_contiguous_seconds"),
"camera window seconds",
)
)
device = source.get("device_identity")
calibration = source.get("calibration_binding")
mount = source.get("mount_binding")
requirements = {
"distinct_from_baseline": True,
"accepted_real_source_origin": source.get("origin") == required_origin,
"catalog_status_ready": source.get("status") == required_status,
"required_modalities_present": required_modalities.issubset(modalities),
"contiguous_camera_window_sufficient": (
camera_window_seconds >= minimum_camera_seconds
),
"host_time_identity_present": _valid_sha256(
source.get("host_time_identity_sha256")
),
"device_identity_present": _device_identity_complete(device),
"calibration_bound_to_source_session": _binding_complete(
calibration,
session_id=session_id,
identity_key="calibration_identity_sha256",
),
"mount_bound_to_source_session": _binding_complete(
mount,
session_id=session_id,
identity_key="mount_identity_sha256",
),
}
blocker_codes = [
name.replace("_", "-")
for name, passed in requirements.items()
if not passed
]
audited.append(
{
"session_id": session_id,
"display_name": _required_string(source, "display_name"),
"role": "candidate",
"catalog_status": source.get("status"),
"modalities": sorted(modalities),
"duration_seconds": source.get("duration_seconds"),
"longest_contiguous_camera_seconds": camera_window_seconds,
"eligible": not blocker_codes,
"requirements": requirements,
"blocker_codes": blocker_codes,
}
)
eligible_ids = [
item["session_id"] for item in audited if item.get("eligible") is True
]
candidates = [item for item in audited if item["role"] == "candidate"]
nearest = min(
candidates,
key=lambda item: (len(item["blocker_codes"]), item["session_id"]),
default=None,
)
status = (
"eligible-second-mounted-source-found"
if eligible_ids
else "blocked-no-eligible-second-mounted-source"
)
decision: dict[str, Any] = {
"e36_replay_authorized": bool(eligible_ids),
"eligible_source_count": len(eligible_ids),
"retuning_allowed": False,
"empty_lab_publication_allowed": False,
"new_capture_authorized": False,
"public_dataset_substitution_allowed": False,
}
if nearest is not None:
decision["nearest_candidate"] = {
"session_id": nearest["session_id"],
"display_name": nearest["display_name"],
"remaining_blocker_codes": nearest["blocker_codes"],
}
return {
"schema_version": E36_REPORT_SCHEMA,
"result_id": result_id,
"status": status,
"catalog_audited_at_utc": inventory["catalog_audited_at_utc"],
"source_count": len(audited),
"candidate_count": len(candidates),
"eligible_source_ids": eligible_ids,
"frozen_upstream": baseline["frozen_upstream"],
"candidates": audited,
"decision": decision,
"next_gate": (
"run-frozen-e32-e35-transfer"
if eligible_ids
else "bind-candidate-calibration-and-mount-provenance"
),
"authority": _authority(),
}
def _validate_profile(profile: dict[str, Any]) -> None:
if profile.get("schema_version") != E36_PROFILE_SCHEMA:
raise E36SourceCatalogAuditError("E36 profile schema is invalid")
baseline = _object(profile.get("baseline"), "E36 baseline")
_validate_session_id(_required_string(baseline, "source_session_id"))
upstream = _object(baseline.get("frozen_upstream"), "E36 frozen upstream")
if set(upstream) != {"e31", "e32", "e33", "e34", "e35"}:
raise E36SourceCatalogAuditError("E36 frozen upstream set is incomplete")
if any(not _required_string(upstream, key) for key in upstream):
raise E36SourceCatalogAuditError("E36 frozen upstream identity is invalid")
eligibility = _object(profile.get("eligibility"), "E36 eligibility")
if (
_required_string(eligibility, "required_status") != "ready"
or _required_string(eligibility, "required_origin")
!= "xgrids-k1.viewer-live.evidence"
or set(
_string_array(
eligibility.get("required_modalities"),
"required modalities",
)
)
!= {"point-cloud", "trajectory", "video"}
or _number(
eligibility.get("minimum_contiguous_camera_seconds"),
"minimum camera seconds",
)
< 60.0
or eligibility.get("require_host_time_identity") is not True
or eligibility.get("require_device_identity") is not True
or eligibility.get("require_session_bound_calibration") is not True
or eligibility.get("require_session_bound_mount") is not True
or eligibility.get("retuning_allowed") is not False
):
raise E36SourceCatalogAuditError("E36 eligibility policy is invalid")
if profile.get("authority") != _authority():
raise E36SourceCatalogAuditError("E36 profile authority is invalid")
def _validate_inventory(inventory: dict[str, Any]) -> None:
if inventory.get("schema_version") != E36_INVENTORY_SCHEMA:
raise E36SourceCatalogAuditError("E36 inventory schema is invalid")
if not _required_string(inventory, "catalog_audited_at_utc").endswith("Z"):
raise E36SourceCatalogAuditError("E36 inventory audit time is invalid")
sources = _array(inventory.get("sources"), "E36 inventory sources")
if not sources:
raise E36SourceCatalogAuditError("E36 inventory is empty")
seen: set[str] = set()
for raw_source in sources:
source = _object(raw_source, "E36 source")
session_id = _required_string(source, "session_id")
_validate_session_id(session_id)
if session_id in seen:
raise E36SourceCatalogAuditError("E36 inventory has duplicate sessions")
seen.add(session_id)
_required_string(source, "display_name")
_required_string(source, "status")
_required_string(source, "origin")
_string_array(source.get("modalities"), "source modalities")
raw_sha256 = source.get("raw_transport_sha256")
if raw_sha256 is not None and not _valid_sha256(raw_sha256):
raise E36SourceCatalogAuditError("E36 source transport identity is invalid")
def _binding_complete(
value: object,
*,
session_id: str,
identity_key: str,
) -> bool:
if not isinstance(value, dict):
return False
return (
value.get("source_session_id") == session_id
and _valid_sha256(value.get(identity_key))
)
def _device_identity_complete(value: object) -> bool:
if not isinstance(value, dict):
return False
return _valid_sha256(value.get("device_serial_sha256")) and _valid_sha256(
value.get("vendor_device_id_sha256")
)
def _authority() -> dict[str, bool]:
return {
"commands_enabled": False,
"navigation_or_safety_accepted": False,
"transfer_replay_authorized": False,
}
def _artifact(path: Path, role: str) -> dict[str, object]:
return {
"role": role,
"path": path.name,
"media_type": "application/json",
"byte_length": path.stat().st_size,
"sha256": _sha256(path),
}
def _read_json(path: Path) -> dict[str, Any]:
try:
value = json.loads(path.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError) as exc:
raise E36SourceCatalogAuditError(f"cannot read E36 JSON: {path.name}") from exc
return _object(value, f"E36 JSON {path.name}")
def _write_json(path: Path, value: object) -> None:
path.write_text(
json.dumps(value, ensure_ascii=False, indent=2, sort_keys=True) + "\n",
encoding="utf-8",
)
def _canonical_json(value: object) -> bytes:
return json.dumps(
value,
ensure_ascii=False,
sort_keys=True,
separators=(",", ":"),
).encode("utf-8")
def _sha256(path: Path) -> str:
digest = hashlib.sha256()
with path.open("rb") as stream:
for chunk in iter(lambda: stream.read(1024 * 1024), b""):
digest.update(chunk)
return digest.hexdigest()
def _valid_sha256(value: object) -> bool:
return isinstance(value, str) and _SHA256.fullmatch(value) is not None
def _object(value: object, label: str) -> dict[str, Any]:
if not isinstance(value, dict):
raise E36SourceCatalogAuditError(f"{label} must be an object")
return value
def _array(value: object, label: str) -> list[Any]:
if not isinstance(value, list):
raise E36SourceCatalogAuditError(f"{label} must be an array")
return value
def _string_array(value: object, label: str) -> list[str]:
values = _array(value, label)
if any(not isinstance(item, str) or not item for item in values):
raise E36SourceCatalogAuditError(f"{label} must contain strings")
return values
def _required_string(value: dict[str, Any], key: str) -> str:
item = value.get(key)
if not isinstance(item, str) or not item.strip():
raise E36SourceCatalogAuditError(f"E36 {key} is invalid")
return item
def _number(value: object, label: str) -> float:
if isinstance(value, bool) or not isinstance(value, (int, float)):
raise E36SourceCatalogAuditError(f"{label} must be numeric")
number = float(value)
if not math.isfinite(number) or number < 0:
raise E36SourceCatalogAuditError(f"{label} must be finite and non-negative")
return number
def _validate_session_id(value: str) -> None:
if _SESSION_ID.fullmatch(value) is None:
raise E36SourceCatalogAuditError("E36 session id is invalid")
+231
View File
@@ -0,0 +1,231 @@
from __future__ import annotations
import copy
import json
from pathlib import Path
from typing import Any
import pytest
from k1link.compute.e36_source_catalog_audit import (
E36SourceCatalogAuditError,
build_e36_source_catalog_audit,
read_e36_source_catalog_audit,
)
SHA = "a" * 64
BASELINE = "20260720T065719Z_viewer_live"
CANDIDATE = "20260718T201659Z_viewer_live"
def _profile() -> dict[str, Any]:
return {
"schema_version": "missioncore.e36-source-eligibility-profile/v1",
"profile_id": "test/v1",
"baseline": {
"source_session_id": BASELINE,
"frozen_upstream": {
"e31": "e31-" + SHA,
"e32": "e32-" + SHA,
"e33": "e33-" + SHA,
"e34": "e34-" + SHA,
"e35": "e35-" + SHA,
},
},
"eligibility": {
"required_status": "ready",
"required_origin": "xgrids-k1.viewer-live.evidence",
"required_modalities": ["point-cloud", "trajectory", "video"],
"minimum_contiguous_camera_seconds": 60.0,
"require_host_time_identity": True,
"require_device_identity": True,
"require_session_bound_calibration": True,
"require_session_bound_mount": True,
"retuning_allowed": False,
},
"authority": {
"commands_enabled": False,
"navigation_or_safety_accepted": False,
"transfer_replay_authorized": False,
},
}
def _source(session_id: str, display_name: str) -> dict[str, Any]:
return {
"session_id": session_id,
"display_name": display_name,
"status": "ready",
"origin": "xgrids-k1.viewer-live.evidence",
"duration_seconds": 200.0,
"modalities": ["point-cloud", "trajectory", "video"],
"raw_transport_sha256": SHA,
"host_time_identity_sha256": SHA,
"device_identity": {
"device_serial_sha256": SHA,
"vendor_device_id_sha256": SHA,
},
"camera": {
"source_id": "sensor.camera.left",
"frame_count": 1000,
"longest_contiguous_seconds": 100.0,
"status": "complete",
},
"calibration_binding": {
"source_session_id": session_id,
"calibration_identity_sha256": SHA,
},
"mount_binding": {
"source_session_id": session_id,
"mount_identity_sha256": SHA,
},
}
def _write(path: Path, value: object) -> None:
path.write_text(json.dumps(value), encoding="utf-8")
def _build(
tmp_path: Path,
*,
candidate: dict[str, Any] | None = None,
):
profile_path = tmp_path / "profile.json"
inventory_path = tmp_path / "inventory.json"
_write(profile_path, _profile())
_write(
inventory_path,
{
"schema_version": "missioncore.e36-source-catalog-inventory/v1",
"catalog_audited_at_utc": "2026-07-27T13:00:00Z",
"sources": [
_source(BASELINE, "RAVNOVES00"),
candidate or _source(CANDIDATE, "TEST007"),
],
},
)
return build_e36_source_catalog_audit(
profile_path=profile_path,
inventory_path=inventory_path,
output_root=tmp_path / "results",
)
def test_eligible_second_source_authorizes_only_the_next_replay_gate(
tmp_path: Path,
) -> None:
result = _build(tmp_path)
assert result.report["status"] == "eligible-second-mounted-source-found"
assert result.eligible_source_ids == (CANDIDATE,)
assert result.report["decision"] == {
"e36_replay_authorized": True,
"eligible_source_count": 1,
"retuning_allowed": False,
"empty_lab_publication_allowed": False,
"new_capture_authorized": False,
"public_dataset_substitution_allowed": False,
"nearest_candidate": {
"session_id": CANDIDATE,
"display_name": "TEST007",
"remaining_blocker_codes": [],
},
}
assert result.report["authority"]["commands_enabled"] is False
def test_missing_session_bound_provenance_blocks_without_empty_lab(
tmp_path: Path,
) -> None:
candidate = _source(CANDIDATE, "TEST007")
candidate["calibration_binding"] = None
candidate["mount_binding"] = None
result = _build(tmp_path, candidate=candidate)
assert result.eligible_source_ids == ()
assert result.report["status"] == "blocked-no-eligible-second-mounted-source"
assert result.report["next_gate"] == (
"bind-candidate-calibration-and-mount-provenance"
)
nearest = result.report["decision"]["nearest_candidate"]
assert nearest["remaining_blocker_codes"] == [
"calibration-bound-to-source-session",
"mount-bound-to-source-session",
]
assert result.report["decision"]["empty_lab_publication_allowed"] is False
def test_calibration_from_another_session_does_not_transfer(
tmp_path: Path,
) -> None:
candidate = _source(CANDIDATE, "TEST007")
candidate["calibration_binding"]["source_session_id"] = BASELINE
result = _build(tmp_path, candidate=candidate)
nearest = result.report["decision"]["nearest_candidate"]
assert nearest["remaining_blocker_codes"] == [
"calibration-bound-to-source-session"
]
def test_result_is_content_addressed_and_detects_report_tampering(
tmp_path: Path,
) -> None:
first = _build(tmp_path)
second = _build(tmp_path)
assert first.result_id == second.result_id
report_path = first.result_root / "audit-report.json"
report = json.loads(report_path.read_text(encoding="utf-8"))
report["status"] = "changed"
_write(report_path, report)
with pytest.raises(E36SourceCatalogAuditError):
read_e36_source_catalog_audit(first.result_root)
def test_profile_cannot_relax_the_mount_gate(tmp_path: Path) -> None:
profile = _profile()
relaxed = copy.deepcopy(profile)
relaxed["eligibility"]["require_session_bound_mount"] = False
profile_path = tmp_path / "profile.json"
inventory_path = tmp_path / "inventory.json"
_write(profile_path, relaxed)
_write(
inventory_path,
{
"schema_version": "missioncore.e36-source-catalog-inventory/v1",
"catalog_audited_at_utc": "2026-07-27T13:00:00Z",
"sources": [_source(BASELINE, "RAVNOVES00")],
},
)
with pytest.raises(E36SourceCatalogAuditError):
build_e36_source_catalog_audit(
profile_path=profile_path,
inventory_path=inventory_path,
output_root=tmp_path / "results",
)
def test_profile_rejects_non_finite_camera_gate(tmp_path: Path) -> None:
profile = _profile()
profile["eligibility"]["minimum_contiguous_camera_seconds"] = float("inf")
profile_path = tmp_path / "profile.json"
inventory_path = tmp_path / "inventory.json"
_write(profile_path, profile)
_write(
inventory_path,
{
"schema_version": "missioncore.e36-source-catalog-inventory/v1",
"catalog_audited_at_utc": "2026-07-27T13:00:00Z",
"sources": [_source(BASELINE, "RAVNOVES00")],
},
)
with pytest.raises(E36SourceCatalogAuditError):
build_e36_source_catalog_audit(
profile_path=profile_path,
inventory_path=inventory_path,
output_root=tmp_path / "results",
)