feat(perception): gate sealed truth evaluation
This commit is contained in:
@@ -0,0 +1,128 @@
|
||||
# E48–E49 detector truth gate implementation
|
||||
|
||||
Date: 2026-07-29
|
||||
|
||||
Status: executable gate ready; no E48 or E49 result exists
|
||||
|
||||
## Purpose
|
||||
|
||||
E46 prepared a references-only detector Truth Island and E47 froze two
|
||||
prediction candidates before truth reveal. E48–E49 make the remaining boundary
|
||||
executable before either reviewer submission exists:
|
||||
|
||||
1. E48 accepts exactly two completed independent blind reviews and one explicit
|
||||
adjudication.
|
||||
2. E48 seals only adjudicated truth and never reads E47 prediction content.
|
||||
3. E49 refuses to run without an accepted E48 seal.
|
||||
4. E49 then joins the exact E47 prediction generation and computes the frozen
|
||||
detector metrics without selecting a winner automatically.
|
||||
|
||||
The implementation does not fabricate reviewer submissions, adjudication,
|
||||
truth, metrics or a candidate decision.
|
||||
|
||||
## E48 review and seal contract
|
||||
|
||||
Each reviewer submission must use
|
||||
`missioncore.e48-independent-detector-review/v1` and preserve all 32 E46 image
|
||||
identities in their original order. Reviewer IDs must be different opaque
|
||||
identifiers. Every submission explicitly attests that candidate identity,
|
||||
prelabels, predictions and scores were not seen.
|
||||
|
||||
Every image must end in `reviewed` with:
|
||||
|
||||
- a required hard-negative decision;
|
||||
- every task-relevant object represented by `object_id`, target class and
|
||||
`xyxy` box in the original 800×600 coordinate system;
|
||||
- required `occluded` and `truncated` flags;
|
||||
- no confidence score, prediction reference or unknown field.
|
||||
|
||||
The adjudication document uses
|
||||
`missioncore.e48-detector-review-adjudication/v1`, binds the canonical SHA-256
|
||||
of both reviewer submissions, covers every image, and explicitly accepts that
|
||||
all disagreements are resolved. Its sealed time may not predate the immutable
|
||||
E47 prediction freeze.
|
||||
|
||||
The E48 sealer rejects:
|
||||
|
||||
- duplicate reviewer identities;
|
||||
- missing, reordered or changed source-image identities;
|
||||
- incomplete frame coverage;
|
||||
- hidden model/prediction fields;
|
||||
- unknown classes, duplicate object IDs or invalid boxes;
|
||||
- a hard-negative flag that conflicts with object presence;
|
||||
- incomplete review/adjudication acceptance;
|
||||
- review hashes that do not match the adjudication;
|
||||
- an E47 manifest bound to another Truth Island.
|
||||
|
||||
E48 reads the E47 manifest identity only. It does not open
|
||||
`candidate-predictions.jsonl`. A successful generation records this boundary
|
||||
as `prediction_content_read_by_sealer=false`.
|
||||
|
||||
Human independence and blindness remain explicit signed-process attestations
|
||||
bound to distinct opaque identities; the code cannot prove a person's identity
|
||||
or what they saw outside the controlled package.
|
||||
|
||||
## E49 frozen evaluation
|
||||
|
||||
E49 accepts only:
|
||||
|
||||
- the exact E46 Truth Island;
|
||||
- an accepted `sealed-adjudicated-independent-truth` E48 generation;
|
||||
- the exact E47 prediction freeze referenced by E48;
|
||||
- a content-addressed valid-FOV mask with the same calibration SHA, camera slot
|
||||
and 800×600 resolution.
|
||||
|
||||
It computes:
|
||||
|
||||
- COCO-style 101-point interpolated AP averaged over IoU 0.50:0.05:0.95;
|
||||
- AP50 and AP75;
|
||||
- AR100;
|
||||
- per-class AP and recall at IoU 0.50;
|
||||
- combined person/vehicle miss rate at IoU 0.50;
|
||||
- false-large-box rate for boxes occupying at least 25% of the image;
|
||||
- valid-FOV leakage as prediction-box centres outside the calibrated mask;
|
||||
- temporal class-count flicker over adjacent frames in each frozen E46 clip.
|
||||
|
||||
The valid-FOV metric is deliberately named and bounded: it is box-centre
|
||||
admission, not full box-area or mask leakage. Temporal flicker is class-count
|
||||
stability because the frozen candidates do not publish track identities.
|
||||
|
||||
E49 always leaves `candidate_winner_selected=false` and
|
||||
`model_retraining_authorized=false`. Candidate acceptance remains a separate
|
||||
explicit product decision after reviewing all metrics and limitations.
|
||||
|
||||
## Execution
|
||||
|
||||
After two real reviewer files and an adjudication exist:
|
||||
|
||||
```text
|
||||
python experiments/perception/run_e48_detector_truth_seal.py \
|
||||
--truth-island-root <E46> \
|
||||
--prediction-freeze-root <E47> \
|
||||
--reviewer-a <review-a.json> \
|
||||
--reviewer-b <review-b.json> \
|
||||
--adjudication <adjudication.json> \
|
||||
--output-root .runtime/compute-experiments/e48/results
|
||||
```
|
||||
|
||||
Only after E48 succeeds:
|
||||
|
||||
```text
|
||||
python experiments/perception/run_e49_detector_truth_evaluation.py \
|
||||
--truth-island-root <E46> \
|
||||
--truth-seal-root <E48> \
|
||||
--prediction-freeze-root <E47> \
|
||||
--valid-fov-root <valid-fov-generation> \
|
||||
--output-root .runtime/compute-experiments/e49/results
|
||||
```
|
||||
|
||||
No real command is run now because the two reviewer submissions and
|
||||
adjudication do not exist.
|
||||
|
||||
## Resource and authority boundary
|
||||
|
||||
The tooling is pure local validation and metric code. It starts no Docker
|
||||
container, model, Worker 006 job, video decoder, browser session, network
|
||||
mutation or second Mission Core service.
|
||||
|
||||
Navigation, safety and command authority remain false.
|
||||
@@ -0,0 +1,49 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Seal completed E46 independent reviews into immutable E48 truth."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
from k1link.compute.e48_detector_truth_seal import (
|
||||
build_e48_detector_truth_seal,
|
||||
)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--truth-island-root", type=Path, required=True)
|
||||
parser.add_argument("--prediction-freeze-root", type=Path, required=True)
|
||||
parser.add_argument("--reviewer-a", type=Path, required=True)
|
||||
parser.add_argument("--reviewer-b", type=Path, required=True)
|
||||
parser.add_argument("--adjudication", type=Path, required=True)
|
||||
parser.add_argument("--output-root", type=Path, required=True)
|
||||
args = parser.parse_args()
|
||||
result = build_e48_detector_truth_seal(
|
||||
truth_island_root=args.truth_island_root,
|
||||
prediction_freeze_root=args.prediction_freeze_root,
|
||||
reviewer_a_path=args.reviewer_a,
|
||||
reviewer_b_path=args.reviewer_b,
|
||||
adjudication_path=args.adjudication,
|
||||
output_root=args.output_root,
|
||||
)
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"result_id": result["result_id"],
|
||||
"result_root": str(result["result_root"]),
|
||||
"status": result["report"]["status"],
|
||||
"frame_count": result["report"]["frame_count"],
|
||||
"object_count": result["report"]["object_count"],
|
||||
},
|
||||
ensure_ascii=False,
|
||||
sort_keys=True,
|
||||
)
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,46 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Evaluate exact E47 predictions after the E48 truth seal."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
from k1link.compute.e49_detector_truth_evaluation import (
|
||||
build_e49_detector_truth_evaluation,
|
||||
)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--truth-island-root", type=Path, required=True)
|
||||
parser.add_argument("--truth-seal-root", type=Path, required=True)
|
||||
parser.add_argument("--prediction-freeze-root", type=Path, required=True)
|
||||
parser.add_argument("--valid-fov-root", type=Path, required=True)
|
||||
parser.add_argument("--output-root", type=Path, required=True)
|
||||
args = parser.parse_args()
|
||||
result = build_e49_detector_truth_evaluation(
|
||||
truth_island_root=args.truth_island_root,
|
||||
truth_seal_root=args.truth_seal_root,
|
||||
prediction_freeze_root=args.prediction_freeze_root,
|
||||
valid_fov_root=args.valid_fov_root,
|
||||
output_root=args.output_root,
|
||||
)
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"result_id": result["result_id"],
|
||||
"result_root": str(result["result_root"]),
|
||||
"status": result["report"]["status"],
|
||||
"candidates": result["report"]["candidates"],
|
||||
},
|
||||
ensure_ascii=False,
|
||||
sort_keys=True,
|
||||
)
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
Reference in New Issue
Block a user