feat(perception): qualify E33 worker shadow
This commit is contained in:
DCCONSTRUCTIONS
@@ -666,9 +666,25 @@ and exclusive PointSlab ownership is enforced. Conflict count remains `38`;
|
|||||||
the result is accepted only as the source-scoped diagnostic/shadow input for
|
the result is accepted only as the source-scoped diagnostic/shadow input for
|
||||||
E33 and is not promoted as a detector-accuracy or staleness improvement.
|
E33 and is not promoted as a detector-accuracy or staleness improvement.
|
||||||
|
|
||||||
Execution is strictly sequential through E33: E30 determines what E31 is
|
E33 accepted immutable worker result
|
||||||
allowed to change; E31 determines the E32 profile; E32 determines the E33
|
`e33-worker-shadow-05cc0bb264410fd49536df90e94067ac39731aff0322a8873700d40008a8bb3a`
|
||||||
runtime input. E34 and E35 may proceed only after E33 closes exact accounting.
|
from package
|
||||||
|
`e33-worker-package-c8609151e71a35b3be3857aef168c0fe98380ff9055ce37560233bb332508823`.
|
||||||
|
The final E32 TrackGeometry stage delivered all `4,489` frames at recorded
|
||||||
|
`1.0×` pace on physical worker `DESKTOP-OPJ8J04`: effective rate was
|
||||||
|
`10.0062 FPS`, wall/source ratio `1.000142`, result-age p95 `2.668 ms` and
|
||||||
|
process RSS maximum `92.34 MiB`. Work and result queues remained bounded at
|
||||||
|
depth two with zero replacement, drop or deadline miss. The pinned container,
|
||||||
|
worker node, complete per-frame outcomes and resource samples are retained in
|
||||||
|
the immutable result. GPU board counters include co-tenant persistent
|
||||||
|
workloads and are evidence of node visibility, not E33 process consumption.
|
||||||
|
E33 qualifies downstream publication against the immutable E10→E32 chain; it
|
||||||
|
does not rerun or independently requalify upstream inference.
|
||||||
|
|
||||||
|
Execution was strictly sequential through E33: E30 determined what E31 was
|
||||||
|
allowed to change; E31 determined the E32 profile; E32 determined the E33
|
||||||
|
runtime input. Exact accounting is now closed, so E34 and E35 are the active
|
||||||
|
critical path.
|
||||||
E36 is the first generalization gate. A separate product decision follows:
|
E36 is the first generalization gate. A separate product decision follows:
|
||||||
either keep the result as operator/shadow evidence, or start L5 occupied-space
|
either keep the result as operator/shadow evidence, or start L5 occupied-space
|
||||||
integration. No LAB in this cycle can enable navigation, commands or safety
|
integration. No LAB in this cycle can enable navigation, commands or safety
|
||||||
|
|||||||
@@ -184,7 +184,15 @@ unqualified ranges and resolves 4,461 overlapping point claims. Of 709
|
|||||||
arbitrated semantic observations, 563 retain `agree` and 146 conservatively
|
arbitrated semantic observations, 563 retain `agree` and 146 conservatively
|
||||||
become `unknown`. The 38 E29 conflicts remain 38; A6 is accepted as a
|
become `unknown`. The 38 E29 conflicts remain 38; A6 is accepted as a
|
||||||
diagnostic/shadow contract, not as detector-accuracy improvement. A7/E33
|
diagnostic/shadow contract, not as detector-accuracy improvement. A7/E33
|
||||||
recorded-source-paced worker execution is now the critical path.
|
is complete in immutable result
|
||||||
|
`e33-worker-shadow-05cc0bb264410fd49536df90e94067ac39731aff0322a8873700d40008a8bb3a`.
|
||||||
|
All 4,489 frames were delivered at `10.0062 FPS` over a `448.623 s` source
|
||||||
|
span with zero queue replacement, result drop or deadline miss. Result age
|
||||||
|
p95 was `2.668 ms`, process RSS peaked at `92.34 MiB`, both queues remained
|
||||||
|
bounded at depth two and the exact physical worker/container identity is
|
||||||
|
retained. This qualifies the final TrackGeometry publication stage against the
|
||||||
|
immutable upstream E10→E32 chain; it is not a new model-inference benchmark.
|
||||||
|
A8/E34+E35 is now the critical path.
|
||||||
|
|
||||||
- [x] Reproduce all 4,489 immutable E29 frames with the exact frozen profile
|
- [x] Reproduce all 4,489 immutable E29 frames with the exact frozen profile
|
||||||
before applying E31/E30 changes.
|
before applying E31/E30 changes.
|
||||||
@@ -196,6 +204,18 @@ recorded-source-paced worker execution is now the critical path.
|
|||||||
cause while preserving source availability and unknown/free-space policy.
|
cause while preserving source availability and unknown/free-space policy.
|
||||||
- [x] Add a digest-bound compact binary PointSlab encoding and strict
|
- [x] Add a digest-bound compact binary PointSlab encoding and strict
|
||||||
TrackGeometryFrame reconstruction/validation.
|
TrackGeometryFrame reconstruction/validation.
|
||||||
|
- [x] Package the exact E32 result, frozen runtime and profile as immutable
|
||||||
|
E33 worker input with complete artifact digests.
|
||||||
|
- [x] Replay all 4,489 frames at recorded `1.0×` pace through independent
|
||||||
|
bounded latest-wins work/result queues.
|
||||||
|
- [x] Close every frame as delivered, input-superseded or result-superseded
|
||||||
|
and publish per-frame health/timing plus resource telemetry.
|
||||||
|
- [x] Bind the accepted result to physical worker `DESKTOP-OPJ8J04` and the
|
||||||
|
pinned container image; independently verify all result artifacts.
|
||||||
|
- [ ] Build E34 as a separate short-TTL occupied/unknown temporal layer over
|
||||||
|
accepted E32/E33 evidence.
|
||||||
|
- [ ] Run E35 deterministic degradation/recovery variants without changing
|
||||||
|
the immutable source or accepted E32/E33 results.
|
||||||
|
|
||||||
### A3 residual and human-exception policy
|
### A3 residual and human-exception policy
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,97 @@
|
|||||||
|
# ADR 0026 — E33 recorded-source-paced TrackGeometry worker shadow
|
||||||
|
|
||||||
|
Date: 2026-07-27
|
||||||
|
|
||||||
|
Status: accepted for source-scoped diagnostic/shadow use
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
E32 closed source, observation and exclusive point-ownership accounting for all
|
||||||
|
4,489 immutable RAVNOVES00 frames. That result proved the data contract, but it
|
||||||
|
did not prove that the final TrackGeometry publication stage could sustain the
|
||||||
|
recorded source timeline on the designated worker with bounded channels and
|
||||||
|
explicit terminal outcomes.
|
||||||
|
|
||||||
|
The upstream E10 result already contains the full source-paced detector and
|
||||||
|
semantic worker execution. E33 therefore qualifies the final E32
|
||||||
|
TrackGeometry transport/publication stage. It does not rerun, replace or tune
|
||||||
|
the detector, segmentation, tracking, calibration or geometry algorithms.
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
1. E33 receives one content-addressed package containing the complete immutable
|
||||||
|
E32 result, the frozen E33 profile and the minimum runtime required to
|
||||||
|
validate and replay it.
|
||||||
|
2. The package and result bind the exact E10, E29, E30, E31 and E32 identities.
|
||||||
|
Every package artifact and every published result artifact has a byte length
|
||||||
|
and SHA-256 digest.
|
||||||
|
3. The producer releases all frames at `1.0×` recorded host pace. Work and
|
||||||
|
result channels are independent latest-wins queues with capacity two.
|
||||||
|
4. Queue replacement is never silent. Every source frame has exactly one
|
||||||
|
terminal outcome: delivered, input-superseded or result-superseded.
|
||||||
|
5. Delivered frames include release lag, work wait, processing time, result
|
||||||
|
wait, result age, health and a digest over the referenced E32 point slab.
|
||||||
|
A late result becomes explicit health state.
|
||||||
|
6. The runtime samples process CPU and RSS plus GPU board visibility,
|
||||||
|
utilization and memory. GPU board values are node-level observations and
|
||||||
|
may include co-tenant workloads; they are not process-level attribution to
|
||||||
|
the E33 publication stage.
|
||||||
|
7. The immutable identity binds both the expected physical worker node and the
|
||||||
|
container hostname. A correct container on an unexpected node fails closed.
|
||||||
|
8. The Windows wrapper admits only a real `D:` package/output directory,
|
||||||
|
enforces a 300 GiB free-space floor, verifies the pinned container image,
|
||||||
|
disables networking, uses a read-only filesystem, drops capabilities and
|
||||||
|
publishes only one atomically completed result directory.
|
||||||
|
9. E33 is diagnostic/shadow only. Commands, navigation and safety acceptance
|
||||||
|
remain false. No persistent reconstruction or source artifact is modified.
|
||||||
|
|
||||||
|
## Accepted result
|
||||||
|
|
||||||
|
Package:
|
||||||
|
`e33-worker-package-c8609151e71a35b3be3857aef168c0fe98380ff9055ce37560233bb332508823`.
|
||||||
|
|
||||||
|
Result:
|
||||||
|
`e33-worker-shadow-05cc0bb264410fd49536df90e94067ac39731aff0322a8873700d40008a8bb3a`.
|
||||||
|
|
||||||
|
Runtime:
|
||||||
|
|
||||||
|
- physical worker: `DESKTOP-OPJ8J04`;
|
||||||
|
- container hostname: `ebca59deeafd`;
|
||||||
|
- image:
|
||||||
|
`nvcr.io/nvidia/tritonserver:26.06-py3@sha256:58df7489c3f2276f9591d500a012dee03e23d35543ce3c390b4c001e6bf90794`;
|
||||||
|
- source span: `448.623 s`;
|
||||||
|
- replay wall time: `448.687 s`;
|
||||||
|
- wall/source ratio: `1.000142`;
|
||||||
|
- effective delivery rate: `10.0062 FPS`.
|
||||||
|
|
||||||
|
Accounting and timing:
|
||||||
|
|
||||||
|
- 4,489 / 4,489 frames delivered;
|
||||||
|
- zero input replacements, result replacements and deadline misses;
|
||||||
|
- work/result queue maximum depth `2/2`;
|
||||||
|
- release lag p50/p95/max:
|
||||||
|
`0.075 / 0.462 / 8.594 ms`;
|
||||||
|
- processing p50/p95/max:
|
||||||
|
`0.174 / 0.750 / 7.698 ms`;
|
||||||
|
- result age p50/p95/max:
|
||||||
|
`0.468 / 2.668 / 13.638 ms`;
|
||||||
|
- process RSS p50/p95/max:
|
||||||
|
`79.28 / 90.33 / 92.34 MiB`.
|
||||||
|
|
||||||
|
All 13 predeclared acceptance requirements pass. The independently copied
|
||||||
|
result archive has SHA-256
|
||||||
|
`9336f8df72ed43c2b46c68c544400db2fda52cf22f7ed8520d23fa0a9fc21081`.
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
- A7 is complete for the final TrackGeometry worker publication stage.
|
||||||
|
- E33 proves source-paced bounded execution and closed accounting for this
|
||||||
|
exact source, profile, package, worker and container identity. It does not
|
||||||
|
independently requalify the already accepted upstream model-inference
|
||||||
|
stages.
|
||||||
|
- An earlier accepted runtime result without a physical worker-node binding is
|
||||||
|
retained as historical evidence and is not the accepted A7 result.
|
||||||
|
- E34 may consume the accepted E32 evidence under the E33 timing/accounting
|
||||||
|
envelope to build a separate short-TTL occupied/unknown layer.
|
||||||
|
- E35 must still prove deterministic degradation and recovery. E33 does not
|
||||||
|
establish second-source transfer, free space, traversability or control.
|
||||||
@@ -0,0 +1,161 @@
|
|||||||
|
# LAB E33 — recorded-source-paced TrackGeometry worker shadow
|
||||||
|
|
||||||
|
Date: 2026-07-27
|
||||||
|
Status: accepted for source-scoped diagnostic/shadow use; navigation, safety
|
||||||
|
and command authority are false
|
||||||
|
Immutable result:
|
||||||
|
`e33-worker-shadow-05cc0bb264410fd49536df90e94067ac39731aff0322a8873700d40008a8bb3a`
|
||||||
|
|
||||||
|
## Objective and architecture stage
|
||||||
|
|
||||||
|
E33 closes A7 of the architecture-audit roadmap. It asks whether the final E32
|
||||||
|
TrackGeometry publication stage can replay every immutable RAVNOVES00 frame at
|
||||||
|
the original recorded host pace on the designated worker while keeping queues,
|
||||||
|
deadlines, health, resource use and every terminal frame outcome explicit.
|
||||||
|
|
||||||
|
The upstream E10 result already ran the complete detector and semantic worker
|
||||||
|
pipeline over the same source at recorded pace. E33 does not rerun or tune
|
||||||
|
those models. It qualifies the downstream E32 TrackGeometry
|
||||||
|
validation/publication stage and preserves the complete upstream identity
|
||||||
|
chain.
|
||||||
|
|
||||||
|
## Source evidence
|
||||||
|
|
||||||
|
- E32 input:
|
||||||
|
`e32-track-geometry-a14ca0e7fb3850ca0dfa3c41634e1b490a2d58ab74d101afc6d6921fbdb0e6fd`.
|
||||||
|
- Source frames: `4,489`.
|
||||||
|
- Recorded timeline: `35.421857292–484.044857292 s`.
|
||||||
|
- Source span: `448.623 s`.
|
||||||
|
- Upstream camera result:
|
||||||
|
`e10-integrated-perception-459aac93918d8f6414b342986ccc6968fefcef6c1f3a78a5254df0b565255ad2`.
|
||||||
|
- E29 result:
|
||||||
|
`e29-camera-geometry-421a9d930638bef12cd5eb10979a477917fa4a389e655ed95f73ba4bd62e13dc`.
|
||||||
|
- E31 source profile:
|
||||||
|
`e31-source-qualification-b2460a5eb143688c7eea6821b2277e13aea79868abe81d83f7e78548c119159a`.
|
||||||
|
|
||||||
|
The E33 package includes the complete E32 manifest, frame JSONL, PointSlab
|
||||||
|
arrays, change journal and report. The package builder verifies every input
|
||||||
|
digest before copying it and hashes every packaged runtime/source artifact.
|
||||||
|
|
||||||
|
## Method, runtime and worker
|
||||||
|
|
||||||
|
Package:
|
||||||
|
`e33-worker-package-c8609151e71a35b3be3857aef168c0fe98380ff9055ce37560233bb332508823`.
|
||||||
|
|
||||||
|
Archive SHA-256:
|
||||||
|
`1495fd04a383f169098c249778bf9234cdf853e53e97d2bd4cdf500e9a921b2c`.
|
||||||
|
|
||||||
|
Execution:
|
||||||
|
|
||||||
|
- ID: `e33-full-1x-20260727-a14ca0e7-v4`;
|
||||||
|
- physical worker: `DESKTOP-OPJ8J04`;
|
||||||
|
- container hostname: `ebca59deeafd`;
|
||||||
|
- Python `3.12.3`, NumPy `1.26.4`, Linux `x86_64`;
|
||||||
|
- pinned image:
|
||||||
|
`nvcr.io/nvidia/tritonserver:26.06-py3@sha256:58df7489c3f2276f9591d500a012dee03e23d35543ce3c390b4c001e6bf90794`;
|
||||||
|
- `1.0×` recorded pacing;
|
||||||
|
- work queue capacity two;
|
||||||
|
- result queue capacity two;
|
||||||
|
- result deadline `100 ms`, stale boundary `150 ms`, unavailable boundary
|
||||||
|
`500 ms`;
|
||||||
|
- worker free-space floor `300 GiB`.
|
||||||
|
|
||||||
|
The container runs without network access, with a read-only filesystem,
|
||||||
|
capabilities dropped, `no-new-privileges`, a bounded PID limit and only
|
||||||
|
read-only package plus writable result mounts. The result appears only through
|
||||||
|
an atomic final directory rename.
|
||||||
|
|
||||||
|
## Implementation and validation
|
||||||
|
|
||||||
|
The runtime:
|
||||||
|
|
||||||
|
1. verifies package and E32 content identities;
|
||||||
|
2. binds the expected physical worker and pinned container image;
|
||||||
|
3. memory-maps the E32 PointSlab arrays;
|
||||||
|
4. releases each frame on its recorded timeline;
|
||||||
|
5. validates the TrackGeometry record and hashes its point-slab byte ranges;
|
||||||
|
6. transports work and results through independent bounded latest-wins queues;
|
||||||
|
7. records exactly one terminal outcome per source frame;
|
||||||
|
8. samples process and GPU-board telemetry once per second;
|
||||||
|
9. evaluates predeclared gates;
|
||||||
|
10. hashes and atomically publishes the immutable result.
|
||||||
|
|
||||||
|
Automated coverage includes the complete positive lifecycle, deterministic
|
||||||
|
overload replacement, artifact tamper rejection, container identity rejection
|
||||||
|
and physical worker-node rejection. The shared queue test now also verifies
|
||||||
|
the exact evicted item so the runtime can close per-frame replacement
|
||||||
|
accounting.
|
||||||
|
|
||||||
|
## Result
|
||||||
|
|
||||||
|
Result:
|
||||||
|
`e33-worker-shadow-05cc0bb264410fd49536df90e94067ac39731aff0322a8873700d40008a8bb3a`.
|
||||||
|
|
||||||
|
Result archive SHA-256:
|
||||||
|
`9336f8df72ed43c2b46c68c544400db2fda52cf22f7ed8520d23fa0a9fc21081`.
|
||||||
|
|
||||||
|
| Measure | Result | Gate |
|
||||||
|
| --- | ---: | ---: |
|
||||||
|
| delivered frames | 4,489 / 4,489 | closed accounting |
|
||||||
|
| input superseded | 0 | ≤ 0.1% |
|
||||||
|
| result superseded | 0 | 0 |
|
||||||
|
| deadline miss fraction | 0 | ≤ 1% |
|
||||||
|
| effective delivery | 10.0062 FPS | ≥ 9.5 FPS |
|
||||||
|
| wall/source ratio | 1.000142 | recorded speed `1.0×` |
|
||||||
|
| release lag p95 | 0.462 ms | ≤ 25 ms |
|
||||||
|
| result age p95 | 2.668 ms | ≤ 100 ms |
|
||||||
|
| process RSS p95 / max | 90.33 / 92.34 MiB | max ≤ 1,024 MiB |
|
||||||
|
| work/result max queue depth | 2 / 2 | bounded by 2 |
|
||||||
|
|
||||||
|
All 4,489 outcomes are `healthy`. Processing p50/p95/max is
|
||||||
|
`0.174 / 0.750 / 7.698 ms`; result age p50/p95/max is
|
||||||
|
`0.468 / 2.668 / 13.638 ms`.
|
||||||
|
|
||||||
|
The GPU is visible. Across 436 node samples, board utilization is
|
||||||
|
`48%` p50, `52%` p95 and `55%` maximum; used board memory is approximately
|
||||||
|
`13,090 MiB`. These board counters include persistent co-tenant inference
|
||||||
|
workloads. They prove node visibility only and must not be described as E33
|
||||||
|
process GPU consumption. The E33 final publication stage is predominantly CPU
|
||||||
|
validation and hashing.
|
||||||
|
|
||||||
|
All 13 predeclared requirements pass. The result is
|
||||||
|
`accepted-recorded-source-paced-shadow`.
|
||||||
|
|
||||||
|
## Regressions, failed attempts and limitations
|
||||||
|
|
||||||
|
Two package attempts failed before result publication:
|
||||||
|
|
||||||
|
- one because the container mount basename did not preserve the
|
||||||
|
content-addressed package ID;
|
||||||
|
- one because the package reader assumed a non-contractual `kind` field in the
|
||||||
|
E32 artifact descriptors.
|
||||||
|
|
||||||
|
Both defects were fixed in new package identities; neither produced a result.
|
||||||
|
An intermediate accepted replay proved the timing path but recorded only the
|
||||||
|
container hostname. It remains immutable history but is not the accepted A7
|
||||||
|
evidence. The final profile and runtime now require
|
||||||
|
`worker_node=DESKTOP-OPJ8J04` and were replayed in full.
|
||||||
|
|
||||||
|
E33 does not:
|
||||||
|
|
||||||
|
- rerun or independently time the upstream detector/semantic model stages;
|
||||||
|
- improve detector accuracy, calibration or E32 geometry quality;
|
||||||
|
- attribute shared GPU-board use to this process;
|
||||||
|
- prove a second source or changed mount;
|
||||||
|
- infer free space, traversability or safety;
|
||||||
|
- enable commands.
|
||||||
|
|
||||||
|
## Decision and next stage
|
||||||
|
|
||||||
|
A7 is complete. E33 proves that the final immutable TrackGeometry stage keeps
|
||||||
|
pace with this exact recorded source on the designated worker with no hidden
|
||||||
|
frame loss and bounded timing/resources.
|
||||||
|
|
||||||
|
The next critical-path stage is A8:
|
||||||
|
|
||||||
|
- E34 builds a separate short-TTL occupied/unknown temporal layer over the
|
||||||
|
accepted E32/E33 evidence without modifying persistent reconstruction;
|
||||||
|
- E35 injects deterministic source loss, staleness, delay, bounded drops and
|
||||||
|
timing offsets and requires explicit safe degradation/recovery.
|
||||||
|
|
||||||
|
Planner, navigation, command and safety authority remain false.
|
||||||
@@ -0,0 +1,276 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Build a minimal immutable E33 worker package around one accepted E32 result."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import shutil
|
||||||
|
import uuid
|
||||||
|
from datetime import UTC, datetime
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from k1link.compute.e32_track_geometry_replay import read_e32_track_geometry_replay
|
||||||
|
from k1link.compute.e33_worker_shadow import E33_PACKAGE_SCHEMA, E33_PROFILE_SCHEMA
|
||||||
|
|
||||||
|
_RUNTIME_FILES = {
|
||||||
|
"runtime/k1link/__init__.py": "src/k1link/__init__.py",
|
||||||
|
"runtime/k1link/compute/__init__.py": None,
|
||||||
|
"runtime/k1link/compute/e33_worker_shadow.py": (
|
||||||
|
"src/k1link/compute/e33_worker_shadow.py"
|
||||||
|
),
|
||||||
|
"runtime/k1link/compute/live_perception.py": "src/k1link/compute/live_perception.py",
|
||||||
|
"runtime/k1link/data_plane/__init__.py": "src/k1link/data_plane/__init__.py",
|
||||||
|
"runtime/k1link/data_plane/views.py": "src/k1link/data_plane/views.py",
|
||||||
|
"runtime/run_e33_worker_shadow.py": (
|
||||||
|
"experiments/perception/worker/run_e33_worker_shadow.py"
|
||||||
|
),
|
||||||
|
"runtime/Invoke-E33WorkerShadow.ps1": (
|
||||||
|
"experiments/perception/worker/Invoke-E33WorkerShadow.ps1"
|
||||||
|
),
|
||||||
|
}
|
||||||
|
_GENERATED_COMPUTE_INIT = (
|
||||||
|
'"""Minimal E33 worker projection; import e33_worker_shadow explicitly."""\n'
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class E33WorkerPackageError(RuntimeError):
|
||||||
|
"""The E33 package source or immutable package is invalid."""
|
||||||
|
|
||||||
|
|
||||||
|
def build_e33_worker_package(
|
||||||
|
*,
|
||||||
|
repository_root: Path,
|
||||||
|
e32_result_root: Path,
|
||||||
|
profile_path: Path,
|
||||||
|
output_root: Path,
|
||||||
|
) -> Path:
|
||||||
|
"""Build or verify one content-addressed E33 worker package."""
|
||||||
|
|
||||||
|
repository = repository_root.resolve(strict=True)
|
||||||
|
e32 = read_e32_track_geometry_replay(e32_result_root)
|
||||||
|
profile_source = profile_path.resolve(strict=True)
|
||||||
|
profile = _read_json(profile_source)
|
||||||
|
if (
|
||||||
|
profile.get("schema_version") != E33_PROFILE_SCHEMA
|
||||||
|
or profile.get("expected_e32_result_id") != e32.result_id
|
||||||
|
):
|
||||||
|
raise E33WorkerPackageError("E33 package profile does not bind the E32 result")
|
||||||
|
runtime_sources: dict[str, Path | None] = {}
|
||||||
|
for target, relative in _RUNTIME_FILES.items():
|
||||||
|
source = None if relative is None else repository / relative
|
||||||
|
if source is not None and (not source.is_file() or source.is_symlink()):
|
||||||
|
raise E33WorkerPackageError(f"E33 runtime source is invalid: {relative}")
|
||||||
|
runtime_sources[target] = source
|
||||||
|
e32_files = sorted(
|
||||||
|
path
|
||||||
|
for path in e32.result_root.iterdir()
|
||||||
|
if path.is_file() and not path.is_symlink()
|
||||||
|
)
|
||||||
|
if not e32_files or any(path.is_symlink() for path in e32.result_root.iterdir()):
|
||||||
|
raise E33WorkerPackageError("E32 result contains an invalid package member")
|
||||||
|
e32_prefix = f"input/e32/{e32.result_id}"
|
||||||
|
sources: dict[str, Path | None] = {
|
||||||
|
**runtime_sources,
|
||||||
|
"profile.json": profile_source,
|
||||||
|
**{f"{e32_prefix}/{path.name}": path for path in e32_files},
|
||||||
|
}
|
||||||
|
source_descriptors = []
|
||||||
|
for relative, source in sorted(sources.items()):
|
||||||
|
payload = (
|
||||||
|
_GENERATED_COMPUTE_INIT.encode()
|
||||||
|
if source is None
|
||||||
|
else source.read_bytes()
|
||||||
|
)
|
||||||
|
source_descriptors.append(
|
||||||
|
{
|
||||||
|
"path": relative,
|
||||||
|
"byte_length": len(payload),
|
||||||
|
"sha256": hashlib.sha256(payload).hexdigest(),
|
||||||
|
}
|
||||||
|
)
|
||||||
|
identity = {
|
||||||
|
"schema_version": E33_PACKAGE_SCHEMA,
|
||||||
|
"classification": "immutable-recorded-shadow-worker-input",
|
||||||
|
"e32_result_id": e32.result_id,
|
||||||
|
"e32_identity_sha256": e32.manifest["identity_sha256"],
|
||||||
|
"profile_sha256": _sha256(profile_source),
|
||||||
|
"artifact_paths": [item["path"] for item in source_descriptors],
|
||||||
|
"source_artifacts": source_descriptors,
|
||||||
|
"authority": {
|
||||||
|
"commands_enabled": False,
|
||||||
|
"navigation_or_safety_accepted": False,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
identity_sha256 = hashlib.sha256(_canonical_json(identity)).hexdigest()
|
||||||
|
package_id = f"e33-worker-package-{identity_sha256}"
|
||||||
|
output = output_root.expanduser().absolute()
|
||||||
|
output.mkdir(mode=0o700, parents=True, exist_ok=True)
|
||||||
|
destination = output / package_id
|
||||||
|
if destination.exists():
|
||||||
|
validate_e33_worker_package(destination)
|
||||||
|
return destination
|
||||||
|
staging = output / f".{package_id}.{uuid.uuid4().hex}.tmp"
|
||||||
|
staging.mkdir(mode=0o700, exist_ok=False)
|
||||||
|
try:
|
||||||
|
for relative, source in sources.items():
|
||||||
|
target_path = staging / relative
|
||||||
|
target_path.parent.mkdir(mode=0o700, parents=True, exist_ok=True)
|
||||||
|
if source is None:
|
||||||
|
target_path.write_text(_GENERATED_COMPUTE_INIT, encoding="utf-8")
|
||||||
|
else:
|
||||||
|
shutil.copyfile(source, target_path)
|
||||||
|
artifacts = [
|
||||||
|
{
|
||||||
|
"kind": relative,
|
||||||
|
"path": relative,
|
||||||
|
"byte_length": (staging / relative).stat().st_size,
|
||||||
|
"sha256": _sha256(staging / relative),
|
||||||
|
}
|
||||||
|
for relative in sorted(sources)
|
||||||
|
]
|
||||||
|
manifest = {
|
||||||
|
"schema_version": E33_PACKAGE_SCHEMA,
|
||||||
|
"package_id": package_id,
|
||||||
|
"identity_sha256": identity_sha256,
|
||||||
|
"identity": identity,
|
||||||
|
"created_at_utc": datetime.now(UTC)
|
||||||
|
.isoformat(timespec="milliseconds")
|
||||||
|
.replace("+00:00", "Z"),
|
||||||
|
"artifacts": artifacts,
|
||||||
|
}
|
||||||
|
_write_json(staging / "manifest.json", manifest)
|
||||||
|
validate_e33_worker_package(staging, allow_staging=True)
|
||||||
|
os.replace(staging, destination)
|
||||||
|
except BaseException:
|
||||||
|
shutil.rmtree(staging, ignore_errors=True)
|
||||||
|
raise
|
||||||
|
validate_e33_worker_package(destination)
|
||||||
|
return destination
|
||||||
|
|
||||||
|
|
||||||
|
def validate_e33_worker_package(
|
||||||
|
root: Path,
|
||||||
|
*,
|
||||||
|
allow_staging: bool = False,
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
"""Validate package identity, exact file set, and every member digest."""
|
||||||
|
|
||||||
|
resolved = root.resolve(strict=True)
|
||||||
|
manifest = _read_json(resolved / "manifest.json")
|
||||||
|
identity = manifest.get("identity")
|
||||||
|
identity_sha256 = manifest.get("identity_sha256")
|
||||||
|
package_id = manifest.get("package_id")
|
||||||
|
artifacts = manifest.get("artifacts")
|
||||||
|
expected_name = (
|
||||||
|
isinstance(package_id, str)
|
||||||
|
and (
|
||||||
|
resolved.name == package_id
|
||||||
|
or (
|
||||||
|
allow_staging
|
||||||
|
and resolved.name.startswith(f".{package_id}.")
|
||||||
|
and resolved.name.endswith(".tmp")
|
||||||
|
)
|
||||||
|
)
|
||||||
|
)
|
||||||
|
if (
|
||||||
|
manifest.get("schema_version") != E33_PACKAGE_SCHEMA
|
||||||
|
or not isinstance(identity, dict)
|
||||||
|
or not isinstance(identity_sha256, str)
|
||||||
|
or hashlib.sha256(_canonical_json(identity)).hexdigest() != identity_sha256
|
||||||
|
or package_id != f"e33-worker-package-{identity_sha256}"
|
||||||
|
or not expected_name
|
||||||
|
or not isinstance(artifacts, list)
|
||||||
|
):
|
||||||
|
raise E33WorkerPackageError("E33 worker package identity is invalid")
|
||||||
|
expected_paths = set(identity.get("artifact_paths", []))
|
||||||
|
actual_paths = {
|
||||||
|
path.relative_to(resolved).as_posix()
|
||||||
|
for path in resolved.rglob("*")
|
||||||
|
if path.is_file()
|
||||||
|
}
|
||||||
|
if (
|
||||||
|
not expected_paths
|
||||||
|
or actual_paths != expected_paths | {"manifest.json"}
|
||||||
|
or len(artifacts) != len(expected_paths)
|
||||||
|
):
|
||||||
|
raise E33WorkerPackageError("E33 worker package file set changed")
|
||||||
|
observed: set[str] = set()
|
||||||
|
for item in artifacts:
|
||||||
|
if not isinstance(item, dict):
|
||||||
|
raise E33WorkerPackageError("E33 worker package artifact is invalid")
|
||||||
|
relative = item.get("path")
|
||||||
|
path = resolved / str(relative)
|
||||||
|
if (
|
||||||
|
not isinstance(relative, str)
|
||||||
|
or relative not in expected_paths
|
||||||
|
or relative in observed
|
||||||
|
or Path(relative).is_absolute()
|
||||||
|
or ".." in Path(relative).parts
|
||||||
|
or not path.is_file()
|
||||||
|
or path.is_symlink()
|
||||||
|
or item.get("kind") != relative
|
||||||
|
or item.get("byte_length") != path.stat().st_size
|
||||||
|
or item.get("sha256") != _sha256(path)
|
||||||
|
):
|
||||||
|
raise E33WorkerPackageError("E33 worker package artifact changed")
|
||||||
|
observed.add(relative)
|
||||||
|
if observed != expected_paths:
|
||||||
|
raise E33WorkerPackageError("E33 worker package artifact coverage changed")
|
||||||
|
return manifest
|
||||||
|
|
||||||
|
|
||||||
|
def _canonical_json(value: object) -> bytes:
|
||||||
|
return json.dumps(
|
||||||
|
value,
|
||||||
|
sort_keys=True,
|
||||||
|
separators=(",", ":"),
|
||||||
|
allow_nan=False,
|
||||||
|
).encode()
|
||||||
|
|
||||||
|
|
||||||
|
def _sha256(path: Path) -> str:
|
||||||
|
digest = hashlib.sha256()
|
||||||
|
with path.open("rb") as stream:
|
||||||
|
while chunk := stream.read(1024 * 1024):
|
||||||
|
digest.update(chunk)
|
||||||
|
return digest.hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
def _read_json(path: Path) -> dict[str, Any]:
|
||||||
|
value = json.loads(path.read_text(encoding="utf-8-sig"))
|
||||||
|
if not isinstance(value, dict):
|
||||||
|
raise E33WorkerPackageError(f"JSON object expected: {path.name}")
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
def _write_json(path: Path, value: object) -> None:
|
||||||
|
with path.open("x", encoding="utf-8", newline="\n") as stream:
|
||||||
|
json.dump(value, stream, indent=2, sort_keys=True)
|
||||||
|
stream.write("\n")
|
||||||
|
stream.flush()
|
||||||
|
os.fsync(stream.fileno())
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
parser = argparse.ArgumentParser()
|
||||||
|
parser.add_argument("--repository-root", type=Path, required=True)
|
||||||
|
parser.add_argument("--e32-result", type=Path, required=True)
|
||||||
|
parser.add_argument("--profile", type=Path, required=True)
|
||||||
|
parser.add_argument("--output-root", type=Path, required=True)
|
||||||
|
args = parser.parse_args()
|
||||||
|
package = build_e33_worker_package(
|
||||||
|
repository_root=args.repository_root,
|
||||||
|
e32_result_root=args.e32_result,
|
||||||
|
profile_path=args.profile,
|
||||||
|
output_root=args.output_root,
|
||||||
|
)
|
||||||
|
print(package)
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
@@ -0,0 +1,141 @@
|
|||||||
|
[CmdletBinding()]
|
||||||
|
param(
|
||||||
|
[Parameter(Mandatory = $true)]
|
||||||
|
[string]$PackageRoot,
|
||||||
|
[Parameter(Mandatory = $true)]
|
||||||
|
[ValidatePattern("^[a-z0-9][a-z0-9._-]{2,95}$")]
|
||||||
|
[string]$ExecutionId,
|
||||||
|
[string]$OutputRoot = "D:\NDC_MISSIONCORE\runtime\derived\e33-worker-shadow",
|
||||||
|
[string]$ContainerImage = "nvcr.io/nvidia/tritonserver:26.06-py3@sha256:58df7489c3f2276f9591d500a012dee03e23d35543ce3c390b4c001e6bf90794",
|
||||||
|
[ValidateRange(1, 1000)]
|
||||||
|
[int]$FreeGiBFloor = 300
|
||||||
|
)
|
||||||
|
|
||||||
|
$ErrorActionPreference = "Stop"
|
||||||
|
$ProgressPreference = "SilentlyContinue"
|
||||||
|
|
||||||
|
function Assert-LastExitCode([string]$Operation) {
|
||||||
|
if ($LASTEXITCODE -ne 0) {
|
||||||
|
throw "$Operation failed with exit code $LASTEXITCODE"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function Resolve-DDirectory([string]$Path, [string]$Label) {
|
||||||
|
$item = Get-Item -LiteralPath (Resolve-Path -LiteralPath $Path).Path -Force
|
||||||
|
$root = [IO.Path]::GetPathRoot($item.FullName).TrimEnd("\")
|
||||||
|
if (
|
||||||
|
-not $item.PSIsContainer -or
|
||||||
|
($item.Attributes -band [IO.FileAttributes]::ReparsePoint) -or
|
||||||
|
$root -ine "D:"
|
||||||
|
) {
|
||||||
|
throw "$Label must be a real D: directory"
|
||||||
|
}
|
||||||
|
return $item.FullName
|
||||||
|
}
|
||||||
|
|
||||||
|
function Convert-ToDockerPath([string]$Path) {
|
||||||
|
return $Path.Replace("\", "/")
|
||||||
|
}
|
||||||
|
|
||||||
|
function Assert-FreeSpace([string]$Phase) {
|
||||||
|
$free = [int64](Get-PSDrive -Name D).Free
|
||||||
|
$floor = [int64]$FreeGiBFloor * 1GB
|
||||||
|
Write-Host (
|
||||||
|
"DISK_GUARD PHASE={0} DRIVE=D FREE_BYTES={1} FREE_GIB={2} FLOOR_GIB={3}" -f
|
||||||
|
$Phase, $free, [math]::Round($free / 1GB, 3), $FreeGiBFloor
|
||||||
|
)
|
||||||
|
if ($free -lt ($floor + 1GB)) {
|
||||||
|
throw "D: lacks the guarded E33 reserve during $Phase"
|
||||||
|
}
|
||||||
|
return $free
|
||||||
|
}
|
||||||
|
|
||||||
|
$package = Resolve-DDirectory $PackageRoot "E33 package"
|
||||||
|
$packageManifestPath = Join-Path $package "manifest.json"
|
||||||
|
if (-not (Test-Path -LiteralPath $packageManifestPath -PathType Leaf)) {
|
||||||
|
throw "E33 package manifest is missing"
|
||||||
|
}
|
||||||
|
$packageManifest = Get-Content -LiteralPath $packageManifestPath -Raw | ConvertFrom-Json
|
||||||
|
if (
|
||||||
|
$packageManifest.schema_version -ne "missioncore.e33-worker-package/v1" -or
|
||||||
|
$packageManifest.package_id -ne (Split-Path $package -Leaf) -or
|
||||||
|
$packageManifest.package_id -notmatch "^e33-worker-package-[a-f0-9]{64}$"
|
||||||
|
) {
|
||||||
|
throw "E33 package manifest is incompatible"
|
||||||
|
}
|
||||||
|
|
||||||
|
if (-not (Test-Path -LiteralPath $OutputRoot)) {
|
||||||
|
$null = New-Item -ItemType Directory -Path $OutputRoot
|
||||||
|
}
|
||||||
|
$output = Resolve-DDirectory $OutputRoot "E33 output root"
|
||||||
|
$freeBefore = Assert-FreeSpace "preflight"
|
||||||
|
|
||||||
|
& docker image inspect $ContainerImage *> $null
|
||||||
|
Assert-LastExitCode "Pinned E33 container image inspection"
|
||||||
|
|
||||||
|
$dockerPackage = Convert-ToDockerPath $package
|
||||||
|
$dockerOutput = Convert-ToDockerPath $output
|
||||||
|
$packageName = Split-Path $package -Leaf
|
||||||
|
$containerPackage = "/opt/e33-input/$packageName"
|
||||||
|
$command = @(
|
||||||
|
"run", "--rm",
|
||||||
|
"--network", "none",
|
||||||
|
"--read-only",
|
||||||
|
"--security-opt", "no-new-privileges:true",
|
||||||
|
"--cap-drop", "ALL",
|
||||||
|
"--pids-limit", "128",
|
||||||
|
"--gpus", "all",
|
||||||
|
"--tmpfs", "/tmp:rw,noexec,nosuid,size=256m",
|
||||||
|
"-e", "PYTHONDONTWRITEBYTECODE=1",
|
||||||
|
"-e", ("PYTHONPATH={0}/runtime" -f $containerPackage),
|
||||||
|
"-e", ("E33_CONTAINER_IMAGE={0}" -f $ContainerImage),
|
||||||
|
"-e", ("E33_WORKER_NODE={0}" -f $env:COMPUTERNAME),
|
||||||
|
"-v", ("{0}:{1}:ro" -f $dockerPackage, $containerPackage),
|
||||||
|
"-v", ("{0}:/output:rw" -f $dockerOutput),
|
||||||
|
"--entrypoint", "python3",
|
||||||
|
$ContainerImage,
|
||||||
|
("{0}/runtime/run_e33_worker_shadow.py" -f $containerPackage),
|
||||||
|
"--package", $containerPackage,
|
||||||
|
"--output-root", "/output",
|
||||||
|
"--execution-id", $ExecutionId
|
||||||
|
)
|
||||||
|
|
||||||
|
Write-Output ("PACKAGE_ID={0}" -f $packageManifest.package_id)
|
||||||
|
Write-Output ("PACKAGE_IDENTITY_SHA256={0}" -f $packageManifest.identity_sha256)
|
||||||
|
Write-Output ("CONTAINER_IMAGE={0}" -f $ContainerImage)
|
||||||
|
Write-Output ("EXECUTION_ID={0}" -f $ExecutionId)
|
||||||
|
& docker @command
|
||||||
|
Assert-LastExitCode "E33 worker shadow"
|
||||||
|
|
||||||
|
$matches = @(
|
||||||
|
Get-ChildItem -LiteralPath $output -Directory -Filter "e33-worker-shadow-*" |
|
||||||
|
Where-Object {
|
||||||
|
$resultPath = Join-Path $_.FullName "result.json"
|
||||||
|
if (-not (Test-Path -LiteralPath $resultPath -PathType Leaf)) {
|
||||||
|
return $false
|
||||||
|
}
|
||||||
|
$result = Get-Content -LiteralPath $resultPath -Raw | ConvertFrom-Json
|
||||||
|
return (
|
||||||
|
$result.identity.execution_id -eq $ExecutionId -and
|
||||||
|
$result.identity.package_id -eq $packageManifest.package_id
|
||||||
|
)
|
||||||
|
}
|
||||||
|
)
|
||||||
|
if ($matches.Count -ne 1) {
|
||||||
|
throw "E33 immutable result could not be resolved uniquely"
|
||||||
|
}
|
||||||
|
$resultRoot = $matches[0].FullName
|
||||||
|
$resultManifest = Get-Content -LiteralPath (Join-Path $resultRoot "result.json") -Raw |
|
||||||
|
ConvertFrom-Json
|
||||||
|
if (
|
||||||
|
$resultManifest.schema_version -ne "missioncore.e33-worker-shadow-result/v1" -or
|
||||||
|
$resultManifest.publication_scope -ne "worker-shadow-diagnostic-only"
|
||||||
|
) {
|
||||||
|
throw "E33 result manifest is incompatible"
|
||||||
|
}
|
||||||
|
$freeAfter = Assert-FreeSpace "completed"
|
||||||
|
Write-Output ("RESULT_ROOT={0}" -f $resultRoot)
|
||||||
|
Write-Output ("RESULT_ID={0}" -f $resultManifest.result_id)
|
||||||
|
Write-Output ("ACCEPTANCE_STATE={0}" -f $resultManifest.acceptance_state)
|
||||||
|
Write-Output ("DISK_FREE_BYTES_BEFORE={0}" -f $freeBefore)
|
||||||
|
Write-Output ("DISK_FREE_BYTES_AFTER={0}" -f $freeAfter)
|
||||||
@@ -0,0 +1,37 @@
|
|||||||
|
{
|
||||||
|
"schema_version": "missioncore.e33-worker-shadow-profile/v1",
|
||||||
|
"mode": "full-session-qualification",
|
||||||
|
"expected_e32_result_id": "e32-track-geometry-a14ca0e7fb3850ca0dfa3c41634e1b490a2d58ab74d101afc6d6921fbdb0e6fd",
|
||||||
|
"expected_worker_node": "DESKTOP-OPJ8J04",
|
||||||
|
"container_image": "nvcr.io/nvidia/tritonserver:26.06-py3@sha256:58df7489c3f2276f9591d500a012dee03e23d35543ce3c390b4c001e6bf90794",
|
||||||
|
"pacing": {
|
||||||
|
"speed": 1.0,
|
||||||
|
"start_delay_ms": 250.0
|
||||||
|
},
|
||||||
|
"queues": {
|
||||||
|
"work_capacity": 2,
|
||||||
|
"result_capacity": 2
|
||||||
|
},
|
||||||
|
"deadlines": {
|
||||||
|
"result_ms": 100.0,
|
||||||
|
"stale_ms": 150.0,
|
||||||
|
"unavailable_ms": 500.0
|
||||||
|
},
|
||||||
|
"resources": {
|
||||||
|
"sample_interval_seconds": 1.0
|
||||||
|
},
|
||||||
|
"acceptance": {
|
||||||
|
"minimum_effective_fps": 9.5,
|
||||||
|
"maximum_input_drop_fraction": 0.001,
|
||||||
|
"maximum_result_drop_fraction": 0.0,
|
||||||
|
"maximum_result_age_p95_ms": 100.0,
|
||||||
|
"maximum_deadline_miss_fraction": 0.01,
|
||||||
|
"maximum_release_lag_p95_ms": 25.0,
|
||||||
|
"maximum_process_rss_mib": 1024.0,
|
||||||
|
"require_gpu_visible": true
|
||||||
|
},
|
||||||
|
"authority": {
|
||||||
|
"commands_enabled": false,
|
||||||
|
"navigation_or_safety_accepted": false
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,40 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Execute one immutable E33 package inside the bounded worker container."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from k1link.compute.e33_worker_shadow import run_e33_worker_shadow
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
parser = argparse.ArgumentParser()
|
||||||
|
parser.add_argument("--package", type=Path, required=True)
|
||||||
|
parser.add_argument("--output-root", type=Path, required=True)
|
||||||
|
parser.add_argument("--execution-id", required=True)
|
||||||
|
args = parser.parse_args()
|
||||||
|
result = run_e33_worker_shadow(
|
||||||
|
args.package,
|
||||||
|
args.output_root,
|
||||||
|
execution_id=args.execution_id,
|
||||||
|
)
|
||||||
|
print(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"result_id": result.result_id,
|
||||||
|
"result_root": str(result.result_root),
|
||||||
|
"accepted": result.accepted,
|
||||||
|
"metrics": result.report["metrics"],
|
||||||
|
},
|
||||||
|
ensure_ascii=False,
|
||||||
|
sort_keys=True,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return 0 if result.accepted else 2
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
@@ -27,6 +27,19 @@ from .e32_track_geometry_replay import (
|
|||||||
e32_track_geometry_frame,
|
e32_track_geometry_frame,
|
||||||
read_e32_track_geometry_replay,
|
read_e32_track_geometry_replay,
|
||||||
)
|
)
|
||||||
|
from .e33_worker_shadow import (
|
||||||
|
E33_FRAME_SCHEMA,
|
||||||
|
E33_OUTCOME_SCHEMA,
|
||||||
|
E33_PACKAGE_SCHEMA,
|
||||||
|
E33_PROFILE_SCHEMA,
|
||||||
|
E33_REPORT_SCHEMA,
|
||||||
|
E33_RESOURCE_SCHEMA,
|
||||||
|
E33_RESULT_SCHEMA,
|
||||||
|
E33WorkerShadowError,
|
||||||
|
E33WorkerShadowResult,
|
||||||
|
read_e33_worker_shadow_result,
|
||||||
|
run_e33_worker_shadow,
|
||||||
|
)
|
||||||
from .evaluation_pack import (
|
from .evaluation_pack import (
|
||||||
ANNOTATION_CONTRACT_SCHEMA,
|
ANNOTATION_CONTRACT_SCHEMA,
|
||||||
EVALUATION_PACK_SCHEMA,
|
EVALUATION_PACK_SCHEMA,
|
||||||
@@ -372,6 +385,17 @@ __all__ = [
|
|||||||
"E32_TRACK_GEOMETRY_RECORD_SCHEMA",
|
"E32_TRACK_GEOMETRY_RECORD_SCHEMA",
|
||||||
"E32TrackGeometryReplay",
|
"E32TrackGeometryReplay",
|
||||||
"E32TrackGeometryReplayError",
|
"E32TrackGeometryReplayError",
|
||||||
|
"E33_FRAME_SCHEMA",
|
||||||
|
"E33_OUTCOME_SCHEMA",
|
||||||
|
"E33_PACKAGE_SCHEMA",
|
||||||
|
"E33_PROFILE_SCHEMA",
|
||||||
|
"E33_REPORT_SCHEMA",
|
||||||
|
"E33_RESOURCE_SCHEMA",
|
||||||
|
"E33_RESULT_SCHEMA",
|
||||||
|
"E33WorkerShadowError",
|
||||||
|
"E33WorkerShadowResult",
|
||||||
|
"read_e33_worker_shadow_result",
|
||||||
|
"run_e33_worker_shadow",
|
||||||
"build_lidar_ground_annotation_template",
|
"build_lidar_ground_annotation_template",
|
||||||
"build_lidar_ground_benchmark",
|
"build_lidar_ground_benchmark",
|
||||||
"build_k1_local_surface",
|
"build_k1_local_surface",
|
||||||
|
|||||||
File diff suppressed because it is too large
Load Diff
@@ -592,17 +592,21 @@ class LatestWinsQueue[T]:
|
|||||||
self._dropped_superseded = 0
|
self._dropped_superseded = 0
|
||||||
self._closed = False
|
self._closed = False
|
||||||
|
|
||||||
def publish(self, item: T) -> None:
|
def publish(self, item: T) -> T | None:
|
||||||
|
"""Publish fresh work and return the evicted oldest item, if any."""
|
||||||
|
|
||||||
with self._condition:
|
with self._condition:
|
||||||
if self._closed:
|
if self._closed:
|
||||||
raise RuntimeError("cannot publish to a closed latest-wins queue")
|
raise RuntimeError("cannot publish to a closed latest-wins queue")
|
||||||
|
dropped: T | None = None
|
||||||
if len(self._items) == self._capacity:
|
if len(self._items) == self._capacity:
|
||||||
self._items.popleft()
|
dropped = self._items.popleft()
|
||||||
self._dropped_overflow += 1
|
self._dropped_overflow += 1
|
||||||
self._items.append(item)
|
self._items.append(item)
|
||||||
self._published += 1
|
self._published += 1
|
||||||
self._maximum_depth = max(self._maximum_depth, len(self._items))
|
self._maximum_depth = max(self._maximum_depth, len(self._items))
|
||||||
self._condition.notify()
|
self._condition.notify()
|
||||||
|
return dropped
|
||||||
|
|
||||||
def take_next(self, timeout: float | None = None) -> T | None:
|
def take_next(self, timeout: float | None = None) -> T | None:
|
||||||
with self._condition:
|
with self._condition:
|
||||||
|
|||||||
@@ -0,0 +1,333 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from k1link.compute.e33_worker_shadow import (
|
||||||
|
E33_PACKAGE_SCHEMA,
|
||||||
|
E33_PROFILE_SCHEMA,
|
||||||
|
E33WorkerShadowError,
|
||||||
|
read_e33_worker_shadow_result,
|
||||||
|
run_e33_worker_shadow,
|
||||||
|
)
|
||||||
|
|
||||||
|
_IMAGE = "test/e33-worker@sha256:" + "a" * 64
|
||||||
|
_AUTHORITY = {
|
||||||
|
"commands_enabled": False,
|
||||||
|
"navigation_or_safety_accepted": False,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def test_e33_worker_shadow_closes_every_frame_and_reuses_existing_result(
|
||||||
|
tmp_path: Path,
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
) -> None:
|
||||||
|
package = _package(tmp_path / "package-source", frame_count=8)
|
||||||
|
monkeypatch.setenv("E33_CONTAINER_IMAGE", _IMAGE)
|
||||||
|
monkeypatch.setenv("E33_WORKER_NODE", "test-worker")
|
||||||
|
|
||||||
|
result = run_e33_worker_shadow(
|
||||||
|
package,
|
||||||
|
tmp_path / "results",
|
||||||
|
execution_id="contract-pass",
|
||||||
|
)
|
||||||
|
reused = run_e33_worker_shadow(
|
||||||
|
package,
|
||||||
|
tmp_path / "results",
|
||||||
|
execution_id="contract-pass",
|
||||||
|
)
|
||||||
|
|
||||||
|
assert result.accepted is True
|
||||||
|
assert reused.result_id == result.result_id
|
||||||
|
accounting = result.report["metrics"]["accounting"]
|
||||||
|
assert accounting == {
|
||||||
|
"source_frames": 8,
|
||||||
|
"delivered": 8,
|
||||||
|
"input_superseded": 0,
|
||||||
|
"result_superseded": 0,
|
||||||
|
"closed": True,
|
||||||
|
}
|
||||||
|
assert result.report["acceptance"]["requirements"][
|
||||||
|
"navigation_or_safety_authority_false"
|
||||||
|
]
|
||||||
|
assert result.result["authority"] == _AUTHORITY
|
||||||
|
|
||||||
|
|
||||||
|
def test_e33_worker_shadow_records_overload_instead_of_hiding_loss(
|
||||||
|
tmp_path: Path,
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
) -> None:
|
||||||
|
package = _package(
|
||||||
|
tmp_path / "package-source",
|
||||||
|
frame_count=24,
|
||||||
|
interval_seconds=0.005,
|
||||||
|
processing_delay_ms=30.0,
|
||||||
|
work_capacity=1,
|
||||||
|
)
|
||||||
|
monkeypatch.setenv("E33_CONTAINER_IMAGE", _IMAGE)
|
||||||
|
monkeypatch.setenv("E33_WORKER_NODE", "test-worker")
|
||||||
|
|
||||||
|
result = run_e33_worker_shadow(
|
||||||
|
package,
|
||||||
|
tmp_path / "results",
|
||||||
|
execution_id="contract-overload",
|
||||||
|
)
|
||||||
|
|
||||||
|
accounting = result.report["metrics"]["accounting"]
|
||||||
|
assert result.accepted is False
|
||||||
|
assert accounting["source_frames"] == 24
|
||||||
|
assert accounting["delivered"] + accounting["input_superseded"] == 24
|
||||||
|
assert accounting["input_superseded"] > 0
|
||||||
|
assert "input_drop_fraction_within_gate" in result.report["acceptance"][
|
||||||
|
"rejection_reasons"
|
||||||
|
]
|
||||||
|
outcomes = [
|
||||||
|
json.loads(line)
|
||||||
|
for line in (result.result_root / "frame-outcomes.jsonl").read_text().splitlines()
|
||||||
|
]
|
||||||
|
assert [row["frame_index"] for row in outcomes] == list(range(24))
|
||||||
|
assert any(row["status"] == "input-superseded" for row in outcomes)
|
||||||
|
|
||||||
|
|
||||||
|
def test_e33_reader_rejects_tampered_artifact(
|
||||||
|
tmp_path: Path,
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
) -> None:
|
||||||
|
package = _package(tmp_path / "package-source", frame_count=4)
|
||||||
|
monkeypatch.setenv("E33_CONTAINER_IMAGE", _IMAGE)
|
||||||
|
monkeypatch.setenv("E33_WORKER_NODE", "test-worker")
|
||||||
|
result = run_e33_worker_shadow(
|
||||||
|
package,
|
||||||
|
tmp_path / "results",
|
||||||
|
execution_id="tamper-check",
|
||||||
|
)
|
||||||
|
with (result.result_root / "frame-outcomes.jsonl").open("a", encoding="utf-8") as stream:
|
||||||
|
stream.write("{}\n")
|
||||||
|
|
||||||
|
with pytest.raises(E33WorkerShadowError, match="artifact identity changed"):
|
||||||
|
read_e33_worker_shadow_result(result.result_root)
|
||||||
|
|
||||||
|
|
||||||
|
def test_e33_rejects_wrong_container_identity(
|
||||||
|
tmp_path: Path,
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
) -> None:
|
||||||
|
package = _package(tmp_path / "package-source", frame_count=4)
|
||||||
|
monkeypatch.setenv("E33_CONTAINER_IMAGE", "different")
|
||||||
|
monkeypatch.setenv("E33_WORKER_NODE", "test-worker")
|
||||||
|
|
||||||
|
with pytest.raises(E33WorkerShadowError, match="container image identity changed"):
|
||||||
|
run_e33_worker_shadow(
|
||||||
|
package,
|
||||||
|
tmp_path / "results",
|
||||||
|
execution_id="wrong-container",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_e33_rejects_wrong_worker_node_identity(
|
||||||
|
tmp_path: Path,
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
) -> None:
|
||||||
|
package = _package(tmp_path / "package-source", frame_count=4)
|
||||||
|
monkeypatch.setenv("E33_CONTAINER_IMAGE", _IMAGE)
|
||||||
|
monkeypatch.setenv("E33_WORKER_NODE", "unexpected-worker")
|
||||||
|
|
||||||
|
with pytest.raises(E33WorkerShadowError, match="worker node identity changed"):
|
||||||
|
run_e33_worker_shadow(
|
||||||
|
package,
|
||||||
|
tmp_path / "results",
|
||||||
|
execution_id="wrong-worker-node",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _package(
|
||||||
|
root: Path,
|
||||||
|
*,
|
||||||
|
frame_count: int,
|
||||||
|
interval_seconds: float = 0.02,
|
||||||
|
processing_delay_ms: float = 0.0,
|
||||||
|
work_capacity: int = 2,
|
||||||
|
) -> Path:
|
||||||
|
root.mkdir(parents=True)
|
||||||
|
e32_identity = {
|
||||||
|
"frame_count": frame_count,
|
||||||
|
"timeline_start_seconds": 0.0,
|
||||||
|
"timeline_end_seconds": (frame_count - 1) * interval_seconds,
|
||||||
|
"source": {
|
||||||
|
"camera_result_id": "e10-integrated-perception-" + "1" * 64,
|
||||||
|
"source_pack_id": "e10-lidar-pack-" + "2" * 64,
|
||||||
|
},
|
||||||
|
"authority": _AUTHORITY,
|
||||||
|
}
|
||||||
|
e32_identity_sha = hashlib.sha256(_canonical(e32_identity)).hexdigest()
|
||||||
|
e32_id = f"e32-track-geometry-{e32_identity_sha}"
|
||||||
|
e32_root = root / "input" / "e32" / e32_id
|
||||||
|
e32_root.mkdir(parents=True)
|
||||||
|
offsets = np.arange(frame_count + 1, dtype="<i8")
|
||||||
|
source_indices = np.arange(frame_count, dtype="<i8")
|
||||||
|
points = np.column_stack(
|
||||||
|
(
|
||||||
|
np.arange(frame_count, dtype=np.float32),
|
||||||
|
np.zeros(frame_count, dtype=np.float32),
|
||||||
|
np.ones(frame_count, dtype=np.float32),
|
||||||
|
)
|
||||||
|
).astype("<f4")
|
||||||
|
owners = np.zeros(frame_count, dtype="<u4")
|
||||||
|
np.save(e32_root / "frame-point-offsets.npy", offsets, allow_pickle=False)
|
||||||
|
np.save(e32_root / "point-source-indices.npy", source_indices, allow_pickle=False)
|
||||||
|
np.save(e32_root / "point-coordinates-map-f32.npy", points, allow_pickle=False)
|
||||||
|
np.save(e32_root / "point-owner-indices.npy", owners, allow_pickle=False)
|
||||||
|
with (e32_root / "track-geometry-frames.jsonl").open(
|
||||||
|
"x",
|
||||||
|
encoding="utf-8",
|
||||||
|
) as stream:
|
||||||
|
for frame_index in range(frame_count):
|
||||||
|
stream.write(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"schema_version": "missioncore.e32-track-geometry-record/v1",
|
||||||
|
"track_geometry_frame_schema": (
|
||||||
|
"missioncore.track-geometry-frame/v1"
|
||||||
|
),
|
||||||
|
"frame_index": frame_index,
|
||||||
|
"source_frame_index": 100 + frame_index,
|
||||||
|
"session_seconds": frame_index * interval_seconds,
|
||||||
|
"source_available": frame_index % 3 != 0,
|
||||||
|
"point_slab": {
|
||||||
|
"schema_version": (
|
||||||
|
"missioncore.e32-point-slab-reference/v1"
|
||||||
|
),
|
||||||
|
"contract_schema": "missioncore.point-slab/v1",
|
||||||
|
"source_point_count": 1,
|
||||||
|
"coordinate_frame": "map",
|
||||||
|
"owner_keys": ["semantic:1"],
|
||||||
|
"row_count": 1,
|
||||||
|
},
|
||||||
|
"geometries": [
|
||||||
|
{
|
||||||
|
"key": "semantic:1",
|
||||||
|
"evidence_state": "agree",
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"policy": {},
|
||||||
|
"authority": _AUTHORITY,
|
||||||
|
},
|
||||||
|
sort_keys=True,
|
||||||
|
)
|
||||||
|
+ "\n"
|
||||||
|
)
|
||||||
|
e32_artifacts = [
|
||||||
|
_artifact(e32_root / name)
|
||||||
|
for name in (
|
||||||
|
"track-geometry-frames.jsonl",
|
||||||
|
"frame-point-offsets.npy",
|
||||||
|
"point-source-indices.npy",
|
||||||
|
"point-coordinates-map-f32.npy",
|
||||||
|
"point-owner-indices.npy",
|
||||||
|
)
|
||||||
|
]
|
||||||
|
_write_json(
|
||||||
|
e32_root / "manifest.json",
|
||||||
|
{
|
||||||
|
"schema_version": "missioncore.e32-track-geometry-replay/v1",
|
||||||
|
"result_id": e32_id,
|
||||||
|
"identity_sha256": e32_identity_sha,
|
||||||
|
"identity": e32_identity,
|
||||||
|
"artifacts": e32_artifacts,
|
||||||
|
"authority": _AUTHORITY,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
profile = {
|
||||||
|
"schema_version": E33_PROFILE_SCHEMA,
|
||||||
|
"mode": "contract-test",
|
||||||
|
"expected_e32_result_id": e32_id,
|
||||||
|
"expected_worker_node": "test-worker",
|
||||||
|
"container_image": _IMAGE,
|
||||||
|
"pacing": {"speed": 1.0, "start_delay_ms": 10.0},
|
||||||
|
"queues": {"work_capacity": work_capacity, "result_capacity": 2},
|
||||||
|
"deadlines": {
|
||||||
|
"result_ms": 100.0,
|
||||||
|
"stale_ms": 150.0,
|
||||||
|
"unavailable_ms": 500.0,
|
||||||
|
},
|
||||||
|
"resources": {"sample_interval_seconds": 0.1},
|
||||||
|
"acceptance": {
|
||||||
|
"minimum_effective_fps": 1.0,
|
||||||
|
"maximum_input_drop_fraction": 0.0,
|
||||||
|
"maximum_result_drop_fraction": 0.0,
|
||||||
|
"maximum_result_age_p95_ms": 100.0,
|
||||||
|
"maximum_deadline_miss_fraction": 0.0,
|
||||||
|
"maximum_release_lag_p95_ms": 25.0,
|
||||||
|
"maximum_process_rss_mib": 4096.0,
|
||||||
|
"require_gpu_visible": False,
|
||||||
|
},
|
||||||
|
"test_controls": {"processing_delay_ms": processing_delay_ms},
|
||||||
|
"authority": _AUTHORITY,
|
||||||
|
}
|
||||||
|
_write_json(root / "profile.json", profile)
|
||||||
|
package_members = [
|
||||||
|
path
|
||||||
|
for path in root.rglob("*")
|
||||||
|
if path.is_file() and path.name != "manifest.json"
|
||||||
|
]
|
||||||
|
relative_paths = sorted(path.relative_to(root).as_posix() for path in package_members)
|
||||||
|
package_identity = {
|
||||||
|
"schema_version": E33_PACKAGE_SCHEMA,
|
||||||
|
"classification": "contract-test",
|
||||||
|
"e32_result_id": e32_id,
|
||||||
|
"e32_identity_sha256": e32_identity_sha,
|
||||||
|
"profile_sha256": _sha256(root / "profile.json"),
|
||||||
|
"artifact_paths": relative_paths,
|
||||||
|
"authority": _AUTHORITY,
|
||||||
|
}
|
||||||
|
package_identity_sha = hashlib.sha256(_canonical(package_identity)).hexdigest()
|
||||||
|
package_id = f"e33-worker-package-{package_identity_sha}"
|
||||||
|
manifest = {
|
||||||
|
"schema_version": E33_PACKAGE_SCHEMA,
|
||||||
|
"package_id": package_id,
|
||||||
|
"identity_sha256": package_identity_sha,
|
||||||
|
"identity": package_identity,
|
||||||
|
"artifacts": [
|
||||||
|
{
|
||||||
|
"kind": relative,
|
||||||
|
"path": relative,
|
||||||
|
"byte_length": (root / relative).stat().st_size,
|
||||||
|
"sha256": _sha256(root / relative),
|
||||||
|
}
|
||||||
|
for relative in relative_paths
|
||||||
|
],
|
||||||
|
}
|
||||||
|
_write_json(root / "manifest.json", manifest)
|
||||||
|
destination = root.parent / package_id
|
||||||
|
root.rename(destination)
|
||||||
|
return destination
|
||||||
|
|
||||||
|
|
||||||
|
def _artifact(path: Path) -> dict[str, Any]:
|
||||||
|
return {
|
||||||
|
"path": path.name,
|
||||||
|
"byte_length": path.stat().st_size,
|
||||||
|
"sha256": _sha256(path),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _write_json(path: Path, value: object) -> None:
|
||||||
|
path.write_text(json.dumps(value, sort_keys=True), encoding="utf-8")
|
||||||
|
|
||||||
|
|
||||||
|
def _canonical(value: object) -> bytes:
|
||||||
|
return json.dumps(
|
||||||
|
value,
|
||||||
|
sort_keys=True,
|
||||||
|
separators=(",", ":"),
|
||||||
|
allow_nan=False,
|
||||||
|
).encode()
|
||||||
|
|
||||||
|
|
||||||
|
def _sha256(path: Path) -> str:
|
||||||
|
return hashlib.sha256(path.read_bytes()).hexdigest()
|
||||||
@@ -210,9 +210,9 @@ def test_live_result_rejects_tampering_and_partial_cuboid() -> None:
|
|||||||
|
|
||||||
def test_latest_wins_queue_never_exceeds_capacity() -> None:
|
def test_latest_wins_queue_never_exceeds_capacity() -> None:
|
||||||
queue = LatestWinsQueue[int](capacity=2)
|
queue = LatestWinsQueue[int](capacity=2)
|
||||||
queue.publish(1)
|
assert queue.publish(1) is None
|
||||||
queue.publish(2)
|
assert queue.publish(2) is None
|
||||||
queue.publish(3)
|
assert queue.publish(3) == 1
|
||||||
|
|
||||||
assert queue.take_next(timeout=0) == 2
|
assert queue.take_next(timeout=0) == 2
|
||||||
assert queue.take_next(timeout=0) == 3
|
assert queue.take_next(timeout=0) == 3
|
||||||
|
|||||||
Reference in New Issue
Block a user