feat(perception): record E39 R1 refinement
This commit is contained in:
@@ -718,6 +718,18 @@ Execution was strictly sequential through E35: E30 determined what E31 was
|
|||||||
allowed to change; E31 determined the E32 profile; E32 determined the E33
|
allowed to change; E31 determined the E32 profile; E32 determined the E33
|
||||||
runtime input; E32/E33 then bound the E34 layer; E32–E34 then bound E35.
|
runtime input; E32/E33 then bound the E34 layer; E32–E34 then bound E35.
|
||||||
Exact nominal and degradation accounting are closed, so A8 is complete.
|
Exact nominal and degradation accounting are closed, so A8 is complete.
|
||||||
|
|
||||||
|
R1 perception quality is now measured by two immutable Worker 006 results.
|
||||||
|
E38 established the baseline at 82.2% presence, 81.5% geometry association,
|
||||||
|
95.2% freshness and 14 high-severity failures. E39 result
|
||||||
|
`e39-perception-refinement-2fd253940c9d9a2fd3b3237f3f0932f81a9f69741935d793771243d5779af464`
|
||||||
|
used a fixed package-bound camera + LiDAR feature projection and improved
|
||||||
|
presence and geometry association to 84.9% while reducing high-severity
|
||||||
|
failures to 8. Freshness remained above target at 93.2%. Its development
|
||||||
|
cross-validation exceeded 90% but did not predict the sealed result, so R1
|
||||||
|
remains open. The next iteration is development-only grouped time/scene
|
||||||
|
qualification with a source-coordinate-free representation; validation is not
|
||||||
|
opened for item-level tuning.
|
||||||
E36 is the first generalization gate. A separate product decision follows:
|
E36 is the first generalization gate. A separate product decision follows:
|
||||||
either keep the result as operator/shadow evidence, or start L5 occupied-space
|
either keep the result as operator/shadow evidence, or start L5 occupied-space
|
||||||
integration. No LAB in this cycle can enable navigation, commands or safety
|
integration. No LAB in this cycle can enable navigation, commands or safety
|
||||||
|
|||||||
@@ -246,9 +246,18 @@ The first R1 baseline is E38 immutable Worker 006 result
|
|||||||
Its sealed 146-item validation measures presence at 82.2%, geometry
|
Its sealed 146-item validation measures presence at 82.2%, geometry
|
||||||
association at 81.5% and freshness at 95.2%, with 100% accounting, zero
|
association at 81.5% and freshness at 95.2%, with 100% accounting, zero
|
||||||
false-free claims and 14 high-severity failures. R1 therefore remains open;
|
false-free claims and 14 high-severity failures. R1 therefore remains open;
|
||||||
the next bounded iteration targets detector/background presence and
|
the next bounded iteration targeted detector/background presence and
|
||||||
geometry-only association without changing the E37 validation set.
|
geometry-only association without changing the E37 validation set.
|
||||||
|
|
||||||
|
E39 completed that refinement in immutable Worker 006 result
|
||||||
|
`e39-perception-refinement-2fd253940c9d9a2fd3b3237f3f0932f81a9f69741935d793771243d5779af464`.
|
||||||
|
Five-fold development CV crossed 90% for all three dimensions without
|
||||||
|
validation-label access. Sealed validation improved presence to 84.9%,
|
||||||
|
geometry association to 84.9% and reduced high-severity failures from 14 to
|
||||||
|
8; freshness remained above target at 93.2%. The aggregate gate remains
|
||||||
|
failed. R1 now advances to a harder grouped time/scene development protocol
|
||||||
|
and a source-coordinate-free representation before any new sealed evaluation.
|
||||||
|
|
||||||
- [x] Reproduce all 4,489 immutable E29 frames with the exact frozen profile
|
- [x] Reproduce all 4,489 immutable E29 frames with the exact frozen profile
|
||||||
before applying E31/E30 changes.
|
before applying E31/E30 changes.
|
||||||
- [x] Apply only the admitted semantic self-mask, two exact geometry
|
- [x] Apply only the admitted semantic self-mask, two exact geometry
|
||||||
|
|||||||
@@ -78,6 +78,22 @@ high-severity validation cases fail, so R1 is not accepted. The next R1
|
|||||||
iteration is bounded to detector/background presence, geometry-only
|
iteration is bounded to detector/background presence, geometry-only
|
||||||
association and the camera-only freshness tail without changing validation.
|
association and the camera-only freshness tail without changing validation.
|
||||||
|
|
||||||
|
The second R1 measurement is immutable result
|
||||||
|
`e39-perception-refinement-2fd253940c9d9a2fd3b3237f3f0932f81a9f69741935d793771243d5779af464`.
|
||||||
|
Its fixed robust three-neighbour refinement used exact camera-crop, LiDAR
|
||||||
|
shape and projection features and was selected only through five-fold
|
||||||
|
development cross-validation. Development reached 90.3% presence, 90.6%
|
||||||
|
geometry association and 96.5% freshness without validation-label access.
|
||||||
|
Sealed validation reached 84.9%, 84.9% and 93.2% respectively. Accounting
|
||||||
|
remains 100%, false-free claims remain zero and high-severity failures fall
|
||||||
|
from 14 to 8, but R1 is still not accepted.
|
||||||
|
|
||||||
|
The next R1 iteration must address the development-to-validation gap before
|
||||||
|
adding model complexity: grouped time/scene development folds replace
|
||||||
|
item-hash folds, and a source-coordinate-free representation is compared under
|
||||||
|
that harder protocol. No individual E39 validation label may be used for
|
||||||
|
diagnosis, fitting or selection.
|
||||||
|
|
||||||
## Experiment rules
|
## Experiment rules
|
||||||
|
|
||||||
1. RAVNOVES00 remains immutable and is never relabelled as a LAB result.
|
1. RAVNOVES00 remains immutable and is never relabelled as a LAB result.
|
||||||
|
|||||||
@@ -0,0 +1,167 @@
|
|||||||
|
# LAB E39 — RAVNOVES00 R1 perception refinement
|
||||||
|
|
||||||
|
Date: 2026-07-28
|
||||||
|
|
||||||
|
Status: completed refinement; R1 quality gate not passed
|
||||||
|
|
||||||
|
## Why this LAB exists
|
||||||
|
|
||||||
|
E38 established the first immutable R1 baseline on the frozen E37 contract:
|
||||||
|
82.2% presence, 81.5% geometry association, 95.2% freshness and 14
|
||||||
|
high-severity failures. E39 tests whether a richer exact camera + LiDAR
|
||||||
|
representation can close that gap without fitting or selecting against the
|
||||||
|
sealed validation labels.
|
||||||
|
|
||||||
|
The experiment remains source-scoped to RAVNOVES00. It does not evaluate
|
||||||
|
navigation, planning, commands, safety or transfer to another route or rig.
|
||||||
|
|
||||||
|
## Immutable inputs
|
||||||
|
|
||||||
|
| Input | Identity |
|
||||||
|
| --- | --- |
|
||||||
|
| Physical source | `20260720T065719Z_viewer_live` (`RAVNOVES00`) |
|
||||||
|
| Frozen E37 contract | `e37-ravnoves-acceptance-01b1efd586f747341c712d82f0907b39436a6f91ae92b1dfae987eca05fd8344` |
|
||||||
|
| E37 item digest | `c6e4f474f59867ce0cc86825193043245c227106400cc216a2dac0645dca54ab` |
|
||||||
|
| E30 materialization | `e30-materialization-841af926d8d28ab93538c46d8f31278a2234c4d1c12c7dc4dc296b249d59735a` |
|
||||||
|
| E30 item digest | `827b498c7b7ab520cc3998e288134ebb5a2e8ebba0532d5cc3905e1a4460b0a7` |
|
||||||
|
| E39 profile | `e39-ravnoves00-r1-development-knn/v1` |
|
||||||
|
| Profile digest | `805828c1eee7b680325c0e72acca0b6a700ead1d267b2745ad7e5227ad6dce34` |
|
||||||
|
|
||||||
|
The denominator is unchanged: 486 reviewed cases split into 340 development
|
||||||
|
and 146 sealed validation cases.
|
||||||
|
|
||||||
|
## Method
|
||||||
|
|
||||||
|
E39 derives 262 deterministic features for each immutable E37 case:
|
||||||
|
|
||||||
|
- source frame and case ordinal;
|
||||||
|
- source stratum, range and geometry categories;
|
||||||
|
- LiDAR point counts, axis quantiles, covariance ratios and class fractions;
|
||||||
|
- projected pixel, depth and height distributions;
|
||||||
|
- a fixed 6×6 RGB camera crop plus luminance quantiles.
|
||||||
|
|
||||||
|
Features are robustly scaled from development statistics. Presence is
|
||||||
|
predicted by a fixed three-neighbour classifier. Geometry association and
|
||||||
|
freshness are projected from predicted presence plus the immutable source
|
||||||
|
stratum; they are not independent learned models in E39.
|
||||||
|
|
||||||
|
The fixed profile is admitted only if deterministic five-fold development
|
||||||
|
cross-validation reaches all three 90% targets. The split seed is
|
||||||
|
`e39-dev-cv`. Validation labels are neither read by feature construction nor
|
||||||
|
used for fitting, threshold selection or model selection. After development
|
||||||
|
qualification, the frozen model is evaluated once on all 146 sealed cases.
|
||||||
|
|
||||||
|
The package includes a digest-bound feature cache so the pinned Worker image
|
||||||
|
requires only NumPy at execution time. Exact camera images and point arrays
|
||||||
|
remain packaged for audit. Cache manifest, feature names, arrays, package
|
||||||
|
contents and result artifacts are independently hashed.
|
||||||
|
|
||||||
|
## Development qualification
|
||||||
|
|
||||||
|
| Dimension | Correct | Accuracy | Target | Development gate |
|
||||||
|
| --- | ---: | ---: | ---: | --- |
|
||||||
|
| Presence | 307 / 340 | 90.3% | >= 90% | pass |
|
||||||
|
| Geometry association | 308 / 340 | 90.6% | >= 90% | pass |
|
||||||
|
| Freshness | 328 / 340 | 96.5% | >= 90% | pass |
|
||||||
|
|
||||||
|
This qualification authorized one sealed evaluation. It did not authorize a
|
||||||
|
release decision by itself.
|
||||||
|
|
||||||
|
## Worker execution
|
||||||
|
|
||||||
|
| Property | Value |
|
||||||
|
| --- | --- |
|
||||||
|
| Worker | `DESKTOP-OPJ8J04` (`Worker 006`) |
|
||||||
|
| Worker package | `e39-worker-package-4902eafcc3fff1750d807775214d4262d777ad628a3532128b2cb55625f46b25` |
|
||||||
|
| Package materialization | 967 files, 65,139,625 bytes |
|
||||||
|
| Container | `ndc-mission-core-e39-refinement` |
|
||||||
|
| Pinned image | `nvcr.io/nvidia/tritonserver:26.06-py3@sha256:58df7489c3f2276f9591d500a012dee03e23d35543ce3c390b4c001e6bf90794` |
|
||||||
|
| Network | disabled |
|
||||||
|
| Limits | 2 CPU, 1 GiB memory and swap, 128 PIDs |
|
||||||
|
| Filesystem | read-only runtime, explicit output mount, 64 MiB no-exec tmpfs |
|
||||||
|
| Authority | commands false; navigation/safety acceptance false |
|
||||||
|
|
||||||
|
The run was executed sequentially to preserve the 18 GiB operator machine and
|
||||||
|
the shared Worker. The immutable result was copied back from Worker 006 and
|
||||||
|
validated against its manifest and artifact digests.
|
||||||
|
|
||||||
|
## Official result
|
||||||
|
|
||||||
|
Result identity:
|
||||||
|
|
||||||
|
`e39-perception-refinement-2fd253940c9d9a2fd3b3237f3f0932f81a9f69741935d793771243d5779af464`
|
||||||
|
|
||||||
|
| Dimension | Correct | Accuracy | Target | Gate |
|
||||||
|
| --- | ---: | ---: | ---: | --- |
|
||||||
|
| Presence | 124 / 146 | 84.9% | >= 90% | fail |
|
||||||
|
| Geometry association | 124 / 146 | 84.9% | >= 90% | fail |
|
||||||
|
| Freshness | 136 / 146 | 93.2% | >= 90% | pass |
|
||||||
|
| Evidence accounting | 146 / 146 | 100% | 100% | pass |
|
||||||
|
| False-free claims | 0 | — | 0 | pass |
|
||||||
|
| High-severity failures | 8 | — | 0 | fail |
|
||||||
|
|
||||||
|
The R1 quality gate is not passed. Blocking checks are presence target,
|
||||||
|
geometry-association target and zero high-severity failures.
|
||||||
|
|
||||||
|
### Change from E38
|
||||||
|
|
||||||
|
| Measure | E38 | E39 | Change |
|
||||||
|
| --- | ---: | ---: | ---: |
|
||||||
|
| Presence | 82.2% | 84.9% | +2.7 percentage points |
|
||||||
|
| Geometry association | 81.5% | 84.9% | +3.4 percentage points |
|
||||||
|
| Freshness | 95.2% | 93.2% | -2.0 percentage points |
|
||||||
|
| High-severity failures | 14 | 8 | -6 |
|
||||||
|
|
||||||
|
E39 is a real improvement in presence, geometry and severe-error count, but it
|
||||||
|
does not satisfy the acceptance contract. The development-to-validation gap
|
||||||
|
is approximately 5.4 percentage points for presence and 5.7 points for
|
||||||
|
geometry association.
|
||||||
|
|
||||||
|
### Remaining weak strata
|
||||||
|
|
||||||
|
- Presence and geometry association are both 79.5% on `camera-only`.
|
||||||
|
- Presence and geometry association are both 78.9% on `geometry-only`.
|
||||||
|
- Freshness is 79.5% on `camera-only`.
|
||||||
|
- `conflict` is 100%, but contains only 11 validation cases and must not mask
|
||||||
|
the larger weak strata.
|
||||||
|
|
||||||
|
The observed CV-to-validation gap is consistent with insufficiently
|
||||||
|
independent development folds and an overly source-local representation,
|
||||||
|
especially because source-frame coordinates are explicit features. This is an
|
||||||
|
engineering interpretation, not a proven causal diagnosis.
|
||||||
|
|
||||||
|
## What E39 proves
|
||||||
|
|
||||||
|
- Richer exact camera + LiDAR features improve E38 on the same frozen source.
|
||||||
|
- The complete Worker packaging and package-bound cache are reproducible
|
||||||
|
without modifying the pinned runtime image.
|
||||||
|
- All 146 validation cases receive terminal outcomes.
|
||||||
|
- Accounting remains 100% and no missing evidence becomes asserted free
|
||||||
|
space.
|
||||||
|
- High-severity failures fall from 14 to 8.
|
||||||
|
|
||||||
|
## What E39 does not prove
|
||||||
|
|
||||||
|
- Presence and geometry association do not reach 90%.
|
||||||
|
- The development CV estimate is not sufficiently predictive of sealed
|
||||||
|
validation.
|
||||||
|
- The source-coordinate-heavy representation is not route-independent.
|
||||||
|
- E39 does not prove another camera, mount, K1 unit, vegetation environment
|
||||||
|
or rover installation.
|
||||||
|
- E39 grants no navigation, command, planning or safety authority.
|
||||||
|
|
||||||
|
## Decision and next iteration
|
||||||
|
|
||||||
|
Keep E39 immutable as the second measured R1 result. Do not retune the
|
||||||
|
three-neighbour model against its sealed predictions.
|
||||||
|
|
||||||
|
The next R1 iteration must be developed only on the 340 development cases and
|
||||||
|
must first replace optimistic item-hash folds with deterministic grouped
|
||||||
|
time/scene folds. It should then compare a source-coordinate-free
|
||||||
|
representation against E39 under that harder development protocol. A new
|
||||||
|
sealed evaluation is allowed only if the grouped development gate reaches all
|
||||||
|
three targets and the method is frozen under a new LAB identity.
|
||||||
|
|
||||||
|
Individual validation labels remain unavailable for error mining or parameter
|
||||||
|
selection. Aggregate E39 metrics may be reported as historical evidence but
|
||||||
|
must not become training features.
|
||||||
@@ -0,0 +1,29 @@
|
|||||||
|
{
|
||||||
|
"authority": {
|
||||||
|
"commands_enabled": false,
|
||||||
|
"navigation_or_safety_accepted": false
|
||||||
|
},
|
||||||
|
"model": {
|
||||||
|
"cross_validation_folds": 5,
|
||||||
|
"cross_validation_seed": "e39-dev-cv",
|
||||||
|
"feature_set": "source-scoped-camera-lidar-shape-v1",
|
||||||
|
"neighbors": 3,
|
||||||
|
"robust_clip": 20.0,
|
||||||
|
"type": "deterministic-robust-knn"
|
||||||
|
},
|
||||||
|
"profile_id": "e39-ravnoves00-r1-development-knn/v1",
|
||||||
|
"schema_version": "missioncore.e39-perception-refinement-profile/v1",
|
||||||
|
"source": {
|
||||||
|
"acceptance_result_id": "e37-ravnoves-acceptance-01b1efd586f747341c712d82f0907b39436a6f91ae92b1dfae987eca05fd8344",
|
||||||
|
"display_name": "RAVNOVES00",
|
||||||
|
"materialization_id": "e30-materialization-841af926d8d28ab93538c46d8f31278a2234c4d1c12c7dc4dc296b249d59735a",
|
||||||
|
"session_id": "20260720T065719Z_viewer_live"
|
||||||
|
},
|
||||||
|
"targets": {
|
||||||
|
"accounting_target": 1.0,
|
||||||
|
"freshness_target": 0.9,
|
||||||
|
"geometry_association_target": 0.9,
|
||||||
|
"maximum_false_free_claims": 0,
|
||||||
|
"presence_target": 0.9
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,440 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Build an immutable camera/LiDAR E39 package for Worker 006."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import hashlib
|
||||||
|
import io
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import shutil
|
||||||
|
import uuid
|
||||||
|
from datetime import UTC, datetime
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
|
||||||
|
from k1link.compute.e39_perception_refinement import (
|
||||||
|
_FEATURE_CACHE_ARRAYS,
|
||||||
|
_FEATURE_CACHE_MANIFEST,
|
||||||
|
_FEATURE_CACHE_SCHEMA,
|
||||||
|
E39_PACKAGE_SCHEMA,
|
||||||
|
E39_PROFILE_SCHEMA,
|
||||||
|
_feature_names,
|
||||||
|
_feature_vector,
|
||||||
|
)
|
||||||
|
|
||||||
|
_RUNTIME_FILES = {
|
||||||
|
"runtime/k1link/__init__.py": "src/k1link/__init__.py",
|
||||||
|
"runtime/k1link/compute/__init__.py": None,
|
||||||
|
"runtime/k1link/compute/e37_acceptance_contract.py": (
|
||||||
|
"src/k1link/compute/e37_acceptance_contract.py"
|
||||||
|
),
|
||||||
|
"runtime/k1link/compute/e39_perception_refinement.py": (
|
||||||
|
"src/k1link/compute/e39_perception_refinement.py"
|
||||||
|
),
|
||||||
|
"runtime/run_e39_perception_refinement.py": (
|
||||||
|
"experiments/perception/worker/run_e39_perception_refinement.py"
|
||||||
|
),
|
||||||
|
"runtime/Invoke-E39PerceptionRefinement.ps1": (
|
||||||
|
"experiments/perception/worker/Invoke-E39PerceptionRefinement.ps1"
|
||||||
|
),
|
||||||
|
}
|
||||||
|
_GENERATED_COMPUTE_INIT = (
|
||||||
|
'"""Minimal E39 worker projection; import contract modules explicitly."""\n'
|
||||||
|
)
|
||||||
|
_ACCEPTANCE_FILES = (
|
||||||
|
"manifest.json",
|
||||||
|
"acceptance-items.jsonl",
|
||||||
|
"acceptance-contract.json",
|
||||||
|
"run-report.json",
|
||||||
|
)
|
||||||
|
_MATERIALIZATION_FILES = ("manifest.json", "materialized-items.jsonl")
|
||||||
|
|
||||||
|
|
||||||
|
class E39WorkerPackageError(RuntimeError):
|
||||||
|
"""The E39 package source or immutable package is invalid."""
|
||||||
|
|
||||||
|
|
||||||
|
def build_e39_worker_package(
|
||||||
|
*,
|
||||||
|
repository_root: Path,
|
||||||
|
acceptance_root: Path,
|
||||||
|
materialization_root: Path,
|
||||||
|
profile_path: Path,
|
||||||
|
output_root: Path,
|
||||||
|
) -> Path:
|
||||||
|
"""Build or verify one content-addressed E39 worker package."""
|
||||||
|
|
||||||
|
repository = repository_root.resolve(strict=True)
|
||||||
|
profile_source = profile_path.resolve(strict=True)
|
||||||
|
profile = _read_json(profile_source)
|
||||||
|
if profile.get("schema_version") != E39_PROFILE_SCHEMA:
|
||||||
|
raise E39WorkerPackageError("E39 package profile is incompatible")
|
||||||
|
acceptance = acceptance_root.resolve(strict=True)
|
||||||
|
materialization = materialization_root.resolve(strict=True)
|
||||||
|
expected_ids = {
|
||||||
|
"acceptance": profile["source"]["acceptance_result_id"],
|
||||||
|
"materialization": profile["source"]["materialization_id"],
|
||||||
|
}
|
||||||
|
if (
|
||||||
|
acceptance.name != expected_ids["acceptance"]
|
||||||
|
or materialization.name != expected_ids["materialization"]
|
||||||
|
):
|
||||||
|
raise E39WorkerPackageError("E39 source identity changed")
|
||||||
|
|
||||||
|
sources: dict[str, Path | bytes | None] = {}
|
||||||
|
for target, relative in _RUNTIME_FILES.items():
|
||||||
|
source = None if relative is None else repository / relative
|
||||||
|
if source is not None and (not source.is_file() or source.is_symlink()):
|
||||||
|
raise E39WorkerPackageError(f"E39 runtime source is invalid: {relative}")
|
||||||
|
sources[target] = source
|
||||||
|
sources["profile.json"] = profile_source
|
||||||
|
for filename in _ACCEPTANCE_FILES:
|
||||||
|
source = acceptance / filename
|
||||||
|
if not source.is_file() or source.is_symlink():
|
||||||
|
raise E39WorkerPackageError("E39 acceptance artifact is invalid")
|
||||||
|
sources[
|
||||||
|
f"input/acceptance/{acceptance.name}/{filename}"
|
||||||
|
] = source
|
||||||
|
for filename in _MATERIALIZATION_FILES:
|
||||||
|
source = materialization / filename
|
||||||
|
if not source.is_file() or source.is_symlink():
|
||||||
|
raise E39WorkerPackageError("E39 materialization artifact is invalid")
|
||||||
|
sources[
|
||||||
|
f"input/materialization/{materialization.name}/{filename}"
|
||||||
|
] = source
|
||||||
|
_add_materialized_evidence(
|
||||||
|
sources,
|
||||||
|
materialization=materialization,
|
||||||
|
)
|
||||||
|
_add_feature_cache(
|
||||||
|
sources,
|
||||||
|
acceptance=acceptance,
|
||||||
|
materialization=materialization,
|
||||||
|
)
|
||||||
|
|
||||||
|
descriptors = []
|
||||||
|
for relative, source in sorted(sources.items()):
|
||||||
|
payload = (
|
||||||
|
_GENERATED_COMPUTE_INIT.encode()
|
||||||
|
if source is None
|
||||||
|
else source
|
||||||
|
if isinstance(source, bytes)
|
||||||
|
else source.read_bytes()
|
||||||
|
)
|
||||||
|
descriptors.append(
|
||||||
|
{
|
||||||
|
"path": relative,
|
||||||
|
"byte_length": len(payload),
|
||||||
|
"sha256": hashlib.sha256(payload).hexdigest(),
|
||||||
|
}
|
||||||
|
)
|
||||||
|
identity = {
|
||||||
|
"schema_version": E39_PACKAGE_SCHEMA,
|
||||||
|
"classification": "immutable-ravnoves00-r1-camera-lidar-worker-input",
|
||||||
|
"source_ids": expected_ids,
|
||||||
|
"profile_sha256": _sha256(profile_source),
|
||||||
|
"runtime_requirements": {
|
||||||
|
"python": "3.12",
|
||||||
|
"numpy": "1.26+",
|
||||||
|
},
|
||||||
|
"build_requirements": {"pillow": "10+"},
|
||||||
|
"artifact_paths": [row["path"] for row in descriptors],
|
||||||
|
"source_artifacts": descriptors,
|
||||||
|
"authority": {
|
||||||
|
"commands_enabled": False,
|
||||||
|
"navigation_or_safety_accepted": False,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
identity_sha256 = hashlib.sha256(_canonical_json(identity)).hexdigest()
|
||||||
|
package_id = f"e39-worker-package-{identity_sha256}"
|
||||||
|
output = output_root.expanduser().absolute()
|
||||||
|
output.mkdir(mode=0o700, parents=True, exist_ok=True)
|
||||||
|
destination = output / package_id
|
||||||
|
if destination.exists():
|
||||||
|
validate_e39_worker_package(destination)
|
||||||
|
return destination
|
||||||
|
|
||||||
|
staging = output / f".{package_id}.{uuid.uuid4().hex}.tmp"
|
||||||
|
staging.mkdir(mode=0o700, exist_ok=False)
|
||||||
|
try:
|
||||||
|
for relative, source in sources.items():
|
||||||
|
target = staging / relative
|
||||||
|
target.parent.mkdir(mode=0o700, parents=True, exist_ok=True)
|
||||||
|
if source is None:
|
||||||
|
target.write_text(_GENERATED_COMPUTE_INIT, encoding="utf-8")
|
||||||
|
elif isinstance(source, bytes):
|
||||||
|
target.write_bytes(source)
|
||||||
|
else:
|
||||||
|
shutil.copyfile(source, target)
|
||||||
|
artifacts = [
|
||||||
|
{
|
||||||
|
"kind": relative,
|
||||||
|
"path": relative,
|
||||||
|
"byte_length": (staging / relative).stat().st_size,
|
||||||
|
"sha256": _sha256(staging / relative),
|
||||||
|
}
|
||||||
|
for relative in sorted(sources)
|
||||||
|
]
|
||||||
|
manifest = {
|
||||||
|
"schema_version": E39_PACKAGE_SCHEMA,
|
||||||
|
"package_id": package_id,
|
||||||
|
"identity_sha256": identity_sha256,
|
||||||
|
"identity": identity,
|
||||||
|
"created_at_utc": datetime.now(UTC)
|
||||||
|
.isoformat(timespec="milliseconds")
|
||||||
|
.replace("+00:00", "Z"),
|
||||||
|
"artifacts": artifacts,
|
||||||
|
}
|
||||||
|
_write_json(staging / "manifest.json", manifest)
|
||||||
|
validate_e39_worker_package(staging, allow_staging=True)
|
||||||
|
os.replace(staging, destination)
|
||||||
|
except BaseException:
|
||||||
|
shutil.rmtree(staging, ignore_errors=True)
|
||||||
|
raise
|
||||||
|
validate_e39_worker_package(destination)
|
||||||
|
return destination
|
||||||
|
|
||||||
|
|
||||||
|
def validate_e39_worker_package(
|
||||||
|
root: Path,
|
||||||
|
*,
|
||||||
|
allow_staging: bool = False,
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
"""Validate package identity, exact file set, and every member digest."""
|
||||||
|
|
||||||
|
resolved = root.resolve(strict=True)
|
||||||
|
manifest = _read_json(resolved / "manifest.json")
|
||||||
|
identity = manifest.get("identity")
|
||||||
|
identity_sha256 = manifest.get("identity_sha256")
|
||||||
|
package_id = manifest.get("package_id")
|
||||||
|
artifacts = manifest.get("artifacts")
|
||||||
|
expected_name = (
|
||||||
|
isinstance(package_id, str)
|
||||||
|
and (
|
||||||
|
resolved.name == package_id
|
||||||
|
or (
|
||||||
|
allow_staging
|
||||||
|
and resolved.name.startswith(f".{package_id}.")
|
||||||
|
and resolved.name.endswith(".tmp")
|
||||||
|
)
|
||||||
|
)
|
||||||
|
)
|
||||||
|
if (
|
||||||
|
manifest.get("schema_version") != E39_PACKAGE_SCHEMA
|
||||||
|
or not isinstance(identity, dict)
|
||||||
|
or not isinstance(identity_sha256, str)
|
||||||
|
or hashlib.sha256(_canonical_json(identity)).hexdigest() != identity_sha256
|
||||||
|
or package_id != f"e39-worker-package-{identity_sha256}"
|
||||||
|
or not expected_name
|
||||||
|
or not isinstance(artifacts, list)
|
||||||
|
):
|
||||||
|
raise E39WorkerPackageError("E39 worker package identity is invalid")
|
||||||
|
expected_paths = set(identity.get("artifact_paths", []))
|
||||||
|
actual_paths = {
|
||||||
|
path.relative_to(resolved).as_posix()
|
||||||
|
for path in resolved.rglob("*")
|
||||||
|
if path.is_file()
|
||||||
|
}
|
||||||
|
if (
|
||||||
|
not expected_paths
|
||||||
|
or actual_paths != expected_paths | {"manifest.json"}
|
||||||
|
or len(artifacts) != len(expected_paths)
|
||||||
|
):
|
||||||
|
raise E39WorkerPackageError("E39 worker package file set changed")
|
||||||
|
observed: set[str] = set()
|
||||||
|
for row in artifacts:
|
||||||
|
if not isinstance(row, dict):
|
||||||
|
raise E39WorkerPackageError("E39 worker package artifact is invalid")
|
||||||
|
relative = row.get("path")
|
||||||
|
path = resolved / str(relative)
|
||||||
|
if (
|
||||||
|
not isinstance(relative, str)
|
||||||
|
or relative not in expected_paths
|
||||||
|
or relative in observed
|
||||||
|
or Path(relative).is_absolute()
|
||||||
|
or ".." in Path(relative).parts
|
||||||
|
or not path.is_file()
|
||||||
|
or path.is_symlink()
|
||||||
|
or row.get("kind") != relative
|
||||||
|
or row.get("byte_length") != path.stat().st_size
|
||||||
|
or row.get("sha256") != _sha256(path)
|
||||||
|
):
|
||||||
|
raise E39WorkerPackageError("E39 worker package artifact changed")
|
||||||
|
observed.add(relative)
|
||||||
|
if observed != expected_paths:
|
||||||
|
raise E39WorkerPackageError("E39 worker package coverage changed")
|
||||||
|
return manifest
|
||||||
|
|
||||||
|
|
||||||
|
def _add_materialized_evidence(
|
||||||
|
sources: dict[str, Path | bytes | None],
|
||||||
|
*,
|
||||||
|
materialization: Path,
|
||||||
|
) -> None:
|
||||||
|
rows = _read_jsonl(materialization / "materialized-items.jsonl")
|
||||||
|
if len(rows) != 486:
|
||||||
|
raise E39WorkerPackageError("E39 materialization denominator changed")
|
||||||
|
for row in rows:
|
||||||
|
for descriptor_name in ("artifact", "camera_frame"):
|
||||||
|
descriptor = row.get(descriptor_name)
|
||||||
|
if not isinstance(descriptor, dict):
|
||||||
|
raise E39WorkerPackageError(
|
||||||
|
"E39 materialization descriptor is invalid"
|
||||||
|
)
|
||||||
|
relative = descriptor.get("path")
|
||||||
|
if (
|
||||||
|
not isinstance(relative, str)
|
||||||
|
or Path(relative).is_absolute()
|
||||||
|
or ".." in Path(relative).parts
|
||||||
|
):
|
||||||
|
raise E39WorkerPackageError("E39 materialization path is invalid")
|
||||||
|
source = materialization / relative
|
||||||
|
if (
|
||||||
|
not source.is_file()
|
||||||
|
or source.is_symlink()
|
||||||
|
or descriptor.get("byte_length") != source.stat().st_size
|
||||||
|
or descriptor.get("sha256") != _sha256(source)
|
||||||
|
):
|
||||||
|
raise E39WorkerPackageError(
|
||||||
|
"E39 materialized evidence content changed"
|
||||||
|
)
|
||||||
|
target = (
|
||||||
|
f"input/materialization/{materialization.name}/{relative}"
|
||||||
|
)
|
||||||
|
existing = sources.get(target)
|
||||||
|
if existing is not None and existing != source:
|
||||||
|
raise E39WorkerPackageError("E39 package target collision")
|
||||||
|
sources[target] = source
|
||||||
|
|
||||||
|
|
||||||
|
def _add_feature_cache(
|
||||||
|
sources: dict[str, Path | bytes | None],
|
||||||
|
*,
|
||||||
|
acceptance: Path,
|
||||||
|
materialization: Path,
|
||||||
|
) -> None:
|
||||||
|
acceptance_rows = _read_jsonl(acceptance / "acceptance-items.jsonl")
|
||||||
|
materialization_rows = _read_jsonl(
|
||||||
|
materialization / "materialized-items.jsonl"
|
||||||
|
)
|
||||||
|
acceptance_by_id = {
|
||||||
|
str(row["item_id"]): row for row in acceptance_rows
|
||||||
|
}
|
||||||
|
if (
|
||||||
|
len(acceptance_by_id) != 486
|
||||||
|
or {str(row["item_id"]) for row in materialization_rows}
|
||||||
|
!= set(acceptance_by_id)
|
||||||
|
):
|
||||||
|
raise E39WorkerPackageError("E39 feature-cache denominator changed")
|
||||||
|
item_ids = [str(row["item_id"]) for row in materialization_rows]
|
||||||
|
features = np.asarray(
|
||||||
|
[
|
||||||
|
_feature_vector(
|
||||||
|
acceptance_by_id[item_id],
|
||||||
|
row,
|
||||||
|
materialization,
|
||||||
|
)
|
||||||
|
for item_id, row in zip(item_ids, materialization_rows, strict=True)
|
||||||
|
],
|
||||||
|
dtype=np.float64,
|
||||||
|
)
|
||||||
|
names = _feature_names()
|
||||||
|
if features.shape != (486, len(names)) or not np.isfinite(features).all():
|
||||||
|
raise E39WorkerPackageError("E39 feature-cache matrix is invalid")
|
||||||
|
arrays_stream = io.BytesIO()
|
||||||
|
np.savez_compressed(
|
||||||
|
arrays_stream,
|
||||||
|
item_ids=np.asarray(item_ids, dtype=f"<U{max(map(len, item_ids))}"),
|
||||||
|
features=features,
|
||||||
|
)
|
||||||
|
arrays_payload = arrays_stream.getvalue()
|
||||||
|
manifest = {
|
||||||
|
"schema_version": _FEATURE_CACHE_SCHEMA,
|
||||||
|
"materialization_id": materialization.name,
|
||||||
|
"materialization_index_sha256": _sha256(
|
||||||
|
materialization / "materialized-items.jsonl"
|
||||||
|
),
|
||||||
|
"feature_names_sha256": hashlib.sha256(
|
||||||
|
_canonical_json(names)
|
||||||
|
).hexdigest(),
|
||||||
|
"items": 486,
|
||||||
|
"dimensions": len(names),
|
||||||
|
"arrays_path": _FEATURE_CACHE_ARRAYS,
|
||||||
|
"arrays_byte_length": len(arrays_payload),
|
||||||
|
"arrays_sha256": hashlib.sha256(arrays_payload).hexdigest(),
|
||||||
|
}
|
||||||
|
manifest_payload = (
|
||||||
|
json.dumps(manifest, indent=2, sort_keys=True) + "\n"
|
||||||
|
).encode()
|
||||||
|
prefix = f"input/materialization/{materialization.name}"
|
||||||
|
sources[f"{prefix}/{_FEATURE_CACHE_ARRAYS}"] = arrays_payload
|
||||||
|
sources[f"{prefix}/{_FEATURE_CACHE_MANIFEST}"] = manifest_payload
|
||||||
|
|
||||||
|
|
||||||
|
def _canonical_json(value: object) -> bytes:
|
||||||
|
return json.dumps(
|
||||||
|
value,
|
||||||
|
sort_keys=True,
|
||||||
|
separators=(",", ":"),
|
||||||
|
allow_nan=False,
|
||||||
|
).encode()
|
||||||
|
|
||||||
|
|
||||||
|
def _sha256(path: Path) -> str:
|
||||||
|
digest = hashlib.sha256()
|
||||||
|
with path.open("rb") as stream:
|
||||||
|
while chunk := stream.read(1024 * 1024):
|
||||||
|
digest.update(chunk)
|
||||||
|
return digest.hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
def _read_json(path: Path) -> dict[str, Any]:
|
||||||
|
value = json.loads(path.read_text(encoding="utf-8-sig"))
|
||||||
|
if not isinstance(value, dict):
|
||||||
|
raise E39WorkerPackageError(f"JSON object expected: {path.name}")
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
def _read_jsonl(path: Path) -> list[dict[str, Any]]:
|
||||||
|
rows = [
|
||||||
|
json.loads(line)
|
||||||
|
for line in path.read_text(encoding="utf-8-sig").splitlines()
|
||||||
|
]
|
||||||
|
if not all(isinstance(row, dict) for row in rows):
|
||||||
|
raise E39WorkerPackageError(f"JSONL object expected: {path.name}")
|
||||||
|
return rows
|
||||||
|
|
||||||
|
|
||||||
|
def _write_json(path: Path, value: object) -> None:
|
||||||
|
with path.open("x", encoding="utf-8", newline="\n") as stream:
|
||||||
|
json.dump(value, stream, indent=2, sort_keys=True)
|
||||||
|
stream.write("\n")
|
||||||
|
stream.flush()
|
||||||
|
os.fsync(stream.fileno())
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
parser = argparse.ArgumentParser()
|
||||||
|
parser.add_argument("--repository-root", type=Path, required=True)
|
||||||
|
parser.add_argument("--acceptance", type=Path, required=True)
|
||||||
|
parser.add_argument("--materialization", type=Path, required=True)
|
||||||
|
parser.add_argument("--profile", type=Path, required=True)
|
||||||
|
parser.add_argument("--output-root", type=Path, required=True)
|
||||||
|
args = parser.parse_args()
|
||||||
|
package = build_e39_worker_package(
|
||||||
|
repository_root=args.repository_root,
|
||||||
|
acceptance_root=args.acceptance,
|
||||||
|
materialization_root=args.materialization,
|
||||||
|
profile_path=args.profile,
|
||||||
|
output_root=args.output_root,
|
||||||
|
)
|
||||||
|
print(package)
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
@@ -0,0 +1,53 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Build the development-qualified RAVNOVES00 R1 refinement."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from k1link.compute.e39_perception_refinement import (
|
||||||
|
build_e39_perception_refinement,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
parser = argparse.ArgumentParser()
|
||||||
|
parser.add_argument("--acceptance", type=Path, required=True)
|
||||||
|
parser.add_argument("--materialization", type=Path, required=True)
|
||||||
|
parser.add_argument("--profile", type=Path, required=True)
|
||||||
|
parser.add_argument("--output-root", type=Path, required=True)
|
||||||
|
parser.add_argument("--worker-node", default=os.environ.get("COMPUTERNAME"))
|
||||||
|
args = parser.parse_args()
|
||||||
|
result = build_e39_perception_refinement(
|
||||||
|
acceptance_root=args.acceptance,
|
||||||
|
materialization_root=args.materialization,
|
||||||
|
profile_path=args.profile,
|
||||||
|
output_root=args.output_root,
|
||||||
|
worker_node=args.worker_node,
|
||||||
|
)
|
||||||
|
print(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"result_id": result.result_id,
|
||||||
|
"result_root": str(result.result_root),
|
||||||
|
"quality_gate_passed": result.quality_gate_passed,
|
||||||
|
"development_cross_validation": result.report[
|
||||||
|
"development_cross_validation"
|
||||||
|
],
|
||||||
|
"metrics": result.report["metrics"],
|
||||||
|
"blocking_checks": result.report["quality_gate"][
|
||||||
|
"blocking_checks"
|
||||||
|
],
|
||||||
|
},
|
||||||
|
ensure_ascii=False,
|
||||||
|
sort_keys=True,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
@@ -0,0 +1,138 @@
|
|||||||
|
[CmdletBinding()]
|
||||||
|
param(
|
||||||
|
[Parameter(Mandatory = $true)]
|
||||||
|
[string]$PackageRoot,
|
||||||
|
[string]$OutputRoot = "D:\NDC_MISSIONCORE\runtime\derived\e39-refinement",
|
||||||
|
[string]$ContainerImage = "nvcr.io/nvidia/tritonserver:26.06-py3@sha256:58df7489c3f2276f9591d500a012dee03e23d35543ce3c390b4c001e6bf90794",
|
||||||
|
[ValidateRange(1, 1000)]
|
||||||
|
[int]$FreeGiBFloor = 300
|
||||||
|
)
|
||||||
|
|
||||||
|
$ErrorActionPreference = "Stop"
|
||||||
|
$ProgressPreference = "SilentlyContinue"
|
||||||
|
|
||||||
|
function Assert-LastExitCode([string]$Operation) {
|
||||||
|
if ($LASTEXITCODE -ne 0) {
|
||||||
|
throw "$Operation failed with exit code $LASTEXITCODE"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function Resolve-DDirectory([string]$Path, [string]$Label) {
|
||||||
|
$item = Get-Item -LiteralPath (Resolve-Path -LiteralPath $Path).Path -Force
|
||||||
|
$root = [IO.Path]::GetPathRoot($item.FullName).TrimEnd("\")
|
||||||
|
if (
|
||||||
|
-not $item.PSIsContainer -or
|
||||||
|
($item.Attributes -band [IO.FileAttributes]::ReparsePoint) -or
|
||||||
|
$root -ine "D:"
|
||||||
|
) {
|
||||||
|
throw "$Label must be a real D: directory"
|
||||||
|
}
|
||||||
|
return $item.FullName
|
||||||
|
}
|
||||||
|
|
||||||
|
function Convert-ToDockerPath([string]$Path) {
|
||||||
|
return $Path.Replace("\", "/")
|
||||||
|
}
|
||||||
|
|
||||||
|
function Assert-FreeSpace([string]$Phase) {
|
||||||
|
$free = [int64](Get-PSDrive -Name D).Free
|
||||||
|
$floor = [int64]$FreeGiBFloor * 1GB
|
||||||
|
Write-Host (
|
||||||
|
"DISK_GUARD PHASE={0} DRIVE=D FREE_BYTES={1} FREE_GIB={2} FLOOR_GIB={3}" -f
|
||||||
|
$Phase, $free, [math]::Round($free / 1GB, 3), $FreeGiBFloor
|
||||||
|
)
|
||||||
|
if ($free -lt ($floor + 1GB)) {
|
||||||
|
throw "D: lacks the guarded E39 reserve during $Phase"
|
||||||
|
}
|
||||||
|
return $free
|
||||||
|
}
|
||||||
|
|
||||||
|
$package = Resolve-DDirectory $PackageRoot "E39 package"
|
||||||
|
$packageManifestPath = Join-Path $package "manifest.json"
|
||||||
|
if (-not (Test-Path -LiteralPath $packageManifestPath -PathType Leaf)) {
|
||||||
|
throw "E39 package manifest is missing"
|
||||||
|
}
|
||||||
|
$packageManifest = Get-Content -LiteralPath $packageManifestPath -Raw |
|
||||||
|
ConvertFrom-Json
|
||||||
|
if (
|
||||||
|
$packageManifest.schema_version -ne "missioncore.e39-worker-package/v1" -or
|
||||||
|
$packageManifest.package_id -ne (Split-Path $package -Leaf) -or
|
||||||
|
$packageManifest.package_id -notmatch "^e39-worker-package-[a-f0-9]{64}$"
|
||||||
|
) {
|
||||||
|
throw "E39 package manifest is incompatible"
|
||||||
|
}
|
||||||
|
|
||||||
|
if (-not (Test-Path -LiteralPath $OutputRoot)) {
|
||||||
|
$null = New-Item -ItemType Directory -Path $OutputRoot
|
||||||
|
}
|
||||||
|
$output = Resolve-DDirectory $OutputRoot "E39 output root"
|
||||||
|
$freeBefore = Assert-FreeSpace "preflight"
|
||||||
|
|
||||||
|
& docker image inspect $ContainerImage *> $null
|
||||||
|
Assert-LastExitCode "Pinned E39 container image inspection"
|
||||||
|
|
||||||
|
$dockerPackage = Convert-ToDockerPath $package
|
||||||
|
$dockerOutput = Convert-ToDockerPath $output
|
||||||
|
$packageName = Split-Path $package -Leaf
|
||||||
|
$containerPackage = "/opt/e39-input/$packageName"
|
||||||
|
$command = @(
|
||||||
|
"run", "--rm",
|
||||||
|
"--name", "ndc-mission-core-e39-refinement",
|
||||||
|
"--network", "none",
|
||||||
|
"--read-only",
|
||||||
|
"--security-opt", "no-new-privileges:true",
|
||||||
|
"--cap-drop", "ALL",
|
||||||
|
"--pids-limit", "128",
|
||||||
|
"--memory", "1g",
|
||||||
|
"--memory-swap", "1g",
|
||||||
|
"--cpus", "2",
|
||||||
|
"--tmpfs", "/tmp:rw,noexec,nosuid,size=64m",
|
||||||
|
"-e", "PYTHONDONTWRITEBYTECODE=1",
|
||||||
|
"-e", ("PYTHONPATH={0}/runtime" -f $containerPackage),
|
||||||
|
"-e", ("E39_WORKER_NODE={0}" -f $env:COMPUTERNAME),
|
||||||
|
"-v", ("{0}:{1}:ro" -f $dockerPackage, $containerPackage),
|
||||||
|
"-v", ("{0}:/output:rw" -f $dockerOutput),
|
||||||
|
"--entrypoint", "python3",
|
||||||
|
$ContainerImage,
|
||||||
|
("{0}/runtime/run_e39_perception_refinement.py" -f $containerPackage),
|
||||||
|
"--package", $containerPackage,
|
||||||
|
"--output-root", "/output"
|
||||||
|
)
|
||||||
|
|
||||||
|
Write-Output ("PACKAGE_ID={0}" -f $packageManifest.package_id)
|
||||||
|
Write-Output ("PACKAGE_IDENTITY_SHA256={0}" -f $packageManifest.identity_sha256)
|
||||||
|
Write-Output ("CONTAINER_IMAGE={0}" -f $ContainerImage)
|
||||||
|
& docker @command
|
||||||
|
Assert-LastExitCode "E39 perception refinement"
|
||||||
|
|
||||||
|
$matches = @(
|
||||||
|
Get-ChildItem -LiteralPath $output -Directory -Filter "e39-perception-refinement-*" |
|
||||||
|
Where-Object {
|
||||||
|
$manifestPath = Join-Path $_.FullName "manifest.json"
|
||||||
|
if (-not (Test-Path -LiteralPath $manifestPath -PathType Leaf)) {
|
||||||
|
return $false
|
||||||
|
}
|
||||||
|
$manifest = Get-Content -LiteralPath $manifestPath -Raw |
|
||||||
|
ConvertFrom-Json
|
||||||
|
return (
|
||||||
|
$manifest.schema_version -eq
|
||||||
|
"missioncore.e39-perception-refinement/v1" -and
|
||||||
|
$manifest.acceptance_state -eq
|
||||||
|
"completed-r1-source-scoped-refinement" -and
|
||||||
|
$manifest.identity.execution.worker_node -eq $env:COMPUTERNAME
|
||||||
|
)
|
||||||
|
}
|
||||||
|
)
|
||||||
|
if ($matches.Count -ne 1) {
|
||||||
|
throw "E39 immutable result could not be resolved uniquely"
|
||||||
|
}
|
||||||
|
$resultRoot = $matches[0].FullName
|
||||||
|
$resultManifest = Get-Content -LiteralPath (
|
||||||
|
Join-Path $resultRoot "manifest.json"
|
||||||
|
) -Raw | ConvertFrom-Json
|
||||||
|
$freeAfter = Assert-FreeSpace "completed"
|
||||||
|
Write-Output ("RESULT_ROOT={0}" -f $resultRoot)
|
||||||
|
Write-Output ("RESULT_ID={0}" -f $resultManifest.result_id)
|
||||||
|
Write-Output ("QUALITY_GATE_PASSED={0}" -f $resultManifest.quality_gate_passed)
|
||||||
|
Write-Output ("DISK_FREE_BYTES_BEFORE={0}" -f $freeBefore)
|
||||||
|
Write-Output ("DISK_FREE_BYTES_AFTER={0}" -f $freeAfter)
|
||||||
@@ -0,0 +1,53 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Execute a packaged E39 refinement inside the pinned Worker 006 container."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from k1link.compute.e39_perception_refinement import (
|
||||||
|
build_e39_perception_refinement,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
parser = argparse.ArgumentParser()
|
||||||
|
parser.add_argument("--package", type=Path, required=True)
|
||||||
|
parser.add_argument("--output-root", type=Path, required=True)
|
||||||
|
args = parser.parse_args()
|
||||||
|
package = args.package.resolve(strict=True)
|
||||||
|
profile = json.loads((package / "profile.json").read_text(encoding="utf-8"))
|
||||||
|
acceptance_id = profile["source"]["acceptance_result_id"]
|
||||||
|
materialization_id = profile["source"]["materialization_id"]
|
||||||
|
result = build_e39_perception_refinement(
|
||||||
|
acceptance_root=package / "input" / "acceptance" / acceptance_id,
|
||||||
|
materialization_root=(
|
||||||
|
package / "input" / "materialization" / materialization_id
|
||||||
|
),
|
||||||
|
profile_path=package / "profile.json",
|
||||||
|
output_root=args.output_root,
|
||||||
|
worker_node=os.environ.get("E39_WORKER_NODE"),
|
||||||
|
)
|
||||||
|
print(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"result_id": result.result_id,
|
||||||
|
"result_root": str(result.result_root),
|
||||||
|
"quality_gate_passed": result.quality_gate_passed,
|
||||||
|
"metrics": result.report["metrics"],
|
||||||
|
"blocking_checks": result.report["quality_gate"][
|
||||||
|
"blocking_checks"
|
||||||
|
],
|
||||||
|
},
|
||||||
|
ensure_ascii=False,
|
||||||
|
sort_keys=True,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,57 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
|
||||||
|
from k1link.compute.e39_perception_refinement import (
|
||||||
|
_feature_names,
|
||||||
|
_fit_robust_scaler,
|
||||||
|
_predict_presence,
|
||||||
|
_project_dimensions,
|
||||||
|
_transform,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_e39_feature_contract_and_knn_are_deterministic() -> None:
|
||||||
|
assert len(_feature_names()) == 262
|
||||||
|
matrix = np.asarray(
|
||||||
|
[
|
||||||
|
[0.0, 1.0],
|
||||||
|
[0.1, 1.1],
|
||||||
|
[4.9, 5.0],
|
||||||
|
[5.0, 5.1],
|
||||||
|
],
|
||||||
|
dtype=np.float64,
|
||||||
|
)
|
||||||
|
median, scale = _fit_robust_scaler(matrix)
|
||||||
|
transformed = _transform(matrix, median=median, scale=scale, clip=20.0)
|
||||||
|
first = _predict_presence(
|
||||||
|
transformed[0],
|
||||||
|
transformed[1:],
|
||||||
|
["background", "object", "object"],
|
||||||
|
neighbors=3,
|
||||||
|
)
|
||||||
|
second = _predict_presence(
|
||||||
|
transformed[0],
|
||||||
|
transformed[1:],
|
||||||
|
["background", "object", "object"],
|
||||||
|
neighbors=3,
|
||||||
|
)
|
||||||
|
assert first == second == ("object", 2 / 3)
|
||||||
|
|
||||||
|
|
||||||
|
def test_e39_projects_presence_through_frozen_stratum_ontology() -> None:
|
||||||
|
assert _project_dimensions("camera-only", "object-present") == {
|
||||||
|
"presence": "object-present",
|
||||||
|
"geometry_association": "insufficient-support",
|
||||||
|
"freshness": "unavailable",
|
||||||
|
}
|
||||||
|
assert _project_dimensions("geometry-only", "occupied-environment") == {
|
||||||
|
"presence": "occupied-environment",
|
||||||
|
"geometry_association": "independent-occupied",
|
||||||
|
"freshness": "current",
|
||||||
|
}
|
||||||
|
assert _project_dimensions("unknown", "background-or-noise") == {
|
||||||
|
"presence": "background-or-noise",
|
||||||
|
"geometry_association": "rejected-nonobject",
|
||||||
|
"freshness": "current",
|
||||||
|
}
|
||||||
@@ -0,0 +1,73 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import importlib.util
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
|
||||||
|
def _module() -> object:
|
||||||
|
path = (
|
||||||
|
Path(__file__).resolve().parents[1]
|
||||||
|
/ "experiments"
|
||||||
|
/ "perception"
|
||||||
|
/ "prepare_e39_worker_package.py"
|
||||||
|
)
|
||||||
|
spec = importlib.util.spec_from_file_location("e39_worker_package_test", path)
|
||||||
|
assert spec is not None and spec.loader is not None
|
||||||
|
module = importlib.util.module_from_spec(spec)
|
||||||
|
sys.modules[spec.name] = module
|
||||||
|
spec.loader.exec_module(module)
|
||||||
|
return module
|
||||||
|
|
||||||
|
|
||||||
|
def test_e39_package_contains_bound_camera_lidar_evidence(tmp_path: Path) -> None:
|
||||||
|
module = _module()
|
||||||
|
repository = Path(__file__).resolve().parents[1]
|
||||||
|
acceptance = (
|
||||||
|
repository
|
||||||
|
/ ".runtime"
|
||||||
|
/ "compute-experiments"
|
||||||
|
/ "e37"
|
||||||
|
/ "results"
|
||||||
|
/ (
|
||||||
|
"e37-ravnoves-acceptance-"
|
||||||
|
"01b1efd586f747341c712d82f0907b39436a6f91ae92b1dfae987eca05fd8344"
|
||||||
|
)
|
||||||
|
)
|
||||||
|
materialization = (
|
||||||
|
repository
|
||||||
|
/ ".runtime"
|
||||||
|
/ "compute-experiments"
|
||||||
|
/ "e30"
|
||||||
|
/ "materializations"
|
||||||
|
/ (
|
||||||
|
"e30-materialization-"
|
||||||
|
"841af926d8d28ab93538c46d8f31278a2234c4d1c12c7dc4dc296b249d59735a"
|
||||||
|
)
|
||||||
|
)
|
||||||
|
package = module.build_e39_worker_package(
|
||||||
|
repository_root=repository,
|
||||||
|
acceptance_root=acceptance,
|
||||||
|
materialization_root=materialization,
|
||||||
|
profile_path=(
|
||||||
|
repository
|
||||||
|
/ "experiments"
|
||||||
|
/ "perception"
|
||||||
|
/ "e39_ravnoves00_r1_refinement_profile.json"
|
||||||
|
),
|
||||||
|
output_root=tmp_path,
|
||||||
|
)
|
||||||
|
manifest = module.validate_e39_worker_package(package)
|
||||||
|
assert package.name == f"e39-worker-package-{manifest['identity_sha256']}"
|
||||||
|
assert manifest["identity"]["classification"] == (
|
||||||
|
"immutable-ravnoves00-r1-camera-lidar-worker-input"
|
||||||
|
)
|
||||||
|
assert len(manifest["artifacts"]) > 800
|
||||||
|
assert (
|
||||||
|
package
|
||||||
|
/ "input"
|
||||||
|
/ "materialization"
|
||||||
|
/ materialization.name
|
||||||
|
/ "frames"
|
||||||
|
/ "frame-001005.jpg"
|
||||||
|
).is_file()
|
||||||
Reference in New Issue
Block a user