SILVER/NODEDC_MISSION_CORE
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

feat: qualify complete K1 replay on worker

Browse Source
This commit is contained in:
DCCONSTRUCTIONS
2026-07-26 10:11:56 +03:00
parent a1e2cb523f
commit 36ea7ea3d8
17 changed files with 1212 additions and 118 deletions
Download Patch File Download Diff File Expand all files Collapse all files
experiments/perception/LAB_E28_REPORT_2026-07-26.md
+171
View File
@@ -0,0 +1,171 @@
# LAB E28 — complete K1 replay through the external worker
Date: 2026-07-26
State: **accepted for replay shadow diagnostics**
## Scope
E28 qualifies the existing K1 local-surface estimator on the external
`mission-gpu` worker without reconnecting the physical scanner. The complete
immutable `RAVNOVES00` recording is reproduced at recorded 1× host-arrival
pace through the normal authenticated binary WebSocket, K1 normalizer,
point/pose binder and bounded latest-wins estimator.
This is not a physical acquisition, free-space, navigation or safety
acceptance. No K1 firmware change, device command or scanner write was used.
## Source identity
- session: `20260720T065719Z_viewer_live`;
- selection: complete recording, no look-ahead;
- combined source span: `469.908994 s`;
- camera frames: `4,489`;
- raw K1 LiDAR events: `4,570`;
- raw K1 pose events: `4,598`;
- MQTT capture SHA-256:
`70da0edad5cbf0e89b6e2355c0aac294f8c36a67249cd33b8918353fd88c83af`;
- camera index SHA-256:
`e029815a60ad9fbfedb6169142c7449df2b119a51d1ce001f08806e04eb0be14`.
The source process admitted every selected event. Its LiDAR ingress reached a
maximum depth of `4/8` with zero replacement. Release lag was `5.051 ms` p95
and source wall time was `470.169 s`, or `1.00055×` ideal.
## Full passive derivative
Before the wire qualification, the complete content-addressed E10 pack was
processed offline with the same pinned surface profile:
- model:
`k1-local-surface-23762244c8bdb97de26fb721ac957d7a00bc9a63571ac4cfa4be19c4effc7d55`;
- timeline frames: `4,489`;
- available/valid LiDAR frames: `3,928`;
- unavailable frames: `561`;
- admitted points: `9,207,270`;
- stale-pose, insufficient-surface and fit-failed frames: `0`;
- sensor height: `1.2816 m` p50;
- pose-binding age: `21.8545 ms` p95;
- slope: `4.1895°` p95;
- roughness: `0.06736 m` p95;
- prior-only next-frame median residual: `0.04065 m` p50 and
`0.07304 m` p95;
- distribution of per-frame p95 residual: `0.66193 m` p95;
- strict temporal jumps: `205/3,927`.
The immutable NPZ SHA-256 is
`f57eb2485b6cef47f2a97a2d9ff1aa9fd9265fe1eb69cd5852d12f39e13b8bc6`.
The current report SHA-256 is
`e342ecd50a1302bf1fd2250f2dfc8d4ba2909e311ced1d2651696dd200bbf0ca`.
This derivative is the first item returned to
**Парк → Диагностика LiDAR**.
## Measured queue decision
The recording contains real host-arrival bursts: among `4,569` LiDAR
inter-arrival intervals, `457` are below `25 ms` and `2,245` are below
`80 ms`. Requiring a sequential stateful estimator to process every burst
increment increased result age without increasing the product update rate.
Rejected attempts were retained:
| Execution | Queue | Result | Finding |
| --- | ---: | --- | --- |
| Mac, complete 10× stress | 2 | rejected | `851` results; `3,711` runtime replacements; `169.94 ms` bounded p95 age |
| Mac, complete 1× | 2 | rejected | old ingress depth lost `6` LiDAR events; `296/4,564` runtime replacements |
| Mac, complete 1× | 8 | rejected | transport became lossless and runtime replacement fell to `38/4,570`, but full p95 age rose to `128.00 ms` |
| worker, 60 s 1× | 8 | rejected diagnostic slice | `581/585` results; p95 age `163.06 ms` |
| worker, 60 s 1× | 2 | rejected under the old gate | `557/585` results, `9.28 Hz`; p95 age `87.73 ms` |
The accepted policy therefore separates two bounds:
- ingress depth `8` preserves the complete authenticated transport;
- estimator depth `2` keeps a fresh local-world result and explicitly counts
superseded work.
Acceptance is expressed in product quantities: at least `9 Hz`, no more than
`100 ms` p95 result age and no more than `8%` superseded estimator inputs.
Transport and binder accounting remain exact; no threshold hides unknown
events.
## Accepted worker run
Result:
`e28-local-surface-wire-e1b0902d502267aff5f0bf78365b24967f98c83d7c1e6bb0f9dc2934c0684ceb`.
| Measurement | Result |
| --- | ---: |
| Camera / LiDAR / pose events received | `4,489 / 4,570 / 4,598` |
| Ingress sequence gaps | `0` |
| Point frames bound / missed / binder-dropped | `4,570 / 0 / 0` |
| Estimator results persisted | `4,206` |
| Explicitly superseded estimator inputs | `364` (`7.965%`) |
| Effective local-surface rate | `9.1945 Hz` |
| Sensor decode p50 / p95 | `1.672 / 16.776 ms` |
| Surface processing p50 / p95 | `23.689 / 45.470 ms` |
| Full result age p50 / p95 | `37.849 / 89.530 ms` |
| Runtime or evidence-writer failures | `0 / 0` |
| Evidence queue maximum depth | `1/32` |
| Worker wall time | `468.042 s` |
All 21 acceptance checks passed. Every processed result was written to
`local-surface-frames.jsonl` with hashes of its point-class, point-height and
step-candidate arrays. The report copied back from D has SHA-256
`8116d88f903a8c743a3b1c59984a7b4688aaa5fb67e6d2f849a03d3479ce1b97`.
The maximum result-age tail reached `1,275.662 ms`. E28 does not hide that
tail: p95 passes the shadow-diagnostic gate, while any future planner-facing
stage must add deadline-miss/staleness health and a stricter independent
acceptance.
## Worker and storage boundary
- worker: `DESKTOP-OPJ8J04` / `mission-gpu`;
- task-controlled root: `D:\NDC_MISSIONCORE`;
- result:
`D:\NDC_MISSIONCORE\runtime\derived\e28-local-surface\worker-full-1x-q2-gate-v2`;
- worker package:
`e15-worker-package-77e6537cf347c7e03f286a6235c33fc3072403b3ddf64f24749c9ed0e066be73`;
- package manifest SHA-256:
`a530fea8c703caa3ddefc49580a5311206e833b6c8b9bebf48d3e5242babeaf7`;
- runner SHA-256:
`be055a2c75686844b6d9de9d787ac922d2094643d39d9db29502321805fff5d9`;
- profile SHA-256:
`86814d28544138f905f73d14ef3ad58e4815a0dad60022945076eb88c68f0d30`;
- container image:
`sha256:58df7489c3f2276f9591d500a012dee03e23d35543ce3c390b4c001e6bf90794`.
Package, runner, profile, secret repository and result remained on D. Package,
runner, profile, media runtime and token repository were mounted read-only;
only the E28 result mount was writable. The container was read-only,
capability-free and `no-new-privileges`.
The source stayed on Mac loopback. A temporary reverse SSH tunnel exposed it
only as worker loopback `127.0.0.1:18031`; no LAN listener was created. The
bearer token was read from a private file and never passed on the command line.
The tunnel and one-shot container were gone after the run.
## Decision
The complete saved K1 evidence is sufficient for current algorithm and worker
qualification. A physically attached K1 is not required for each software
iteration.
Accepted now:
- complete passive K1 replay on the actual external worker;
- lossless authenticated transport and point/pose binding;
- bounded freshness-first local-surface output at greater than `9 Hz`;
- current operator review of the full offline derivative.
Still unavailable:
- raw MID-360 scan semantics, ray-cleared free space and traversability;
- dynamic-object classification;
- hardware-clock, field-network and physical acquisition commissioning;
- command, navigation or safety authority.
The next implementation gate is a separate dynamic-observation layer over the
accepted local surface, followed by explicit staleness/deadline health.
Physical K1 execution remains a later commissioning test for transport and
hardware behavior, not a prerequisite for this development loop.
experiments/perception/run_e12_shadow_transport_replay.py
+45 -14
View File
@@ -86,7 +86,10 @@ def _percentiles(values: list[float]) -> dict[str, float | int]:
}
def _camera_events(epoch_root: Path, duration_seconds: float) -> Iterator[ReplayEvent]:
def _camera_events(
epoch_root: Path,
duration_seconds: float | None,
) -> Iterator[ReplayEvent]:
index_path = epoch_root / "index.jsonl"
if not index_path.is_file():
raise RuntimeError("camera index is missing")
@@ -96,7 +99,11 @@ def _camera_events(epoch_root: Path, duration_seconds: float) -> Iterator[Replay
raise RuntimeError("camera replay interval is empty")
first_record = json.loads(first_line)
start_epoch_ns = int(first_record["host_epoch_ns"])
end_epoch_ns = start_epoch_ns + int(duration_seconds * 1_000_000_000)
end_epoch_ns = (
None
if duration_seconds is None
else start_epoch_ns + int(duration_seconds * 1_000_000_000)
)
init_payload = (epoch_root / "init.mp4").read_bytes()
yield ReplayEvent(
modality="camera-init",
@@ -109,7 +116,7 @@ def _camera_events(epoch_root: Path, duration_seconds: float) -> Iterator[Replay
for line in itertools.chain((first_line,), stream):
record = json.loads(line)
epoch_ns = int(record["host_epoch_ns"])
if epoch_ns > end_epoch_ns:
if end_epoch_ns is not None and epoch_ns > end_epoch_ns:
break
payload_path = epoch_root / str(record["path"])
payload = payload_path.read_bytes()
@@ -130,14 +137,21 @@ def _camera_events(epoch_root: Path, duration_seconds: float) -> Iterator[Replay
def _mqtt_events(
capture_path: Path,
*,
start_epoch_ns: int,
duration_seconds: float,
start_epoch_ns: int | None,
duration_seconds: float | None,
) -> Iterator[ReplayEvent]:
end_epoch_ns = start_epoch_ns + int(duration_seconds * 1_000_000_000)
end_epoch_ns = (
None
if start_epoch_ns is None or duration_seconds is None
else start_epoch_ns + int(duration_seconds * 1_000_000_000)
)
for message in iter_replay_messages(capture_path):
if message.received_at_epoch_ns < start_epoch_ns:
if (
start_epoch_ns is not None
and message.received_at_epoch_ns < start_epoch_ns
):
continue
if message.received_at_epoch_ns > end_epoch_ns:
if end_epoch_ns is not None and message.received_at_epoch_ns > end_epoch_ns:
break
if message.topic.endswith("/lio_pcl") or message.topic == "RealtimePointcloud":
modality: Literal["lidar", "pose"] = "lidar"
@@ -169,17 +183,22 @@ def _wait_for(predicate: object, *, timeout_seconds: float, label: str) -> None:
def run(args: argparse.Namespace) -> dict[str, object]:
experiment_id = str(getattr(args, "experiment_id", _EXPERIMENT_ID))
speed = float(getattr(args, "speed", 1.0))
duration_seconds = float(args.duration_seconds)
if (
experiment_id not in {"e12", "e21", "e23"}
experiment_id not in {"e12", "e21", "e23", "e28"}
or not math.isfinite(speed)
or not 0.1 <= speed <= 10
or not math.isfinite(duration_seconds)
or duration_seconds < 0
):
raise RuntimeError("replay experiment identity or speed is invalid")
complete_recording = duration_seconds == 0
selected_duration = None if complete_recording else duration_seconds
repository_root = Path(args.repository_root).expanduser().resolve()
session_root = Path(args.session_root).expanduser().resolve()
epoch_root = session_root / "media" / "sensor.camera.right" / "epoch-1"
capture_path = session_root / "captures" / "mqtt_live" / "mqtt.raw.k1mqtt"
camera = iter(_camera_events(epoch_root, args.duration_seconds))
camera = iter(_camera_events(epoch_root, selected_duration))
first_camera = next(camera, None)
first_frame = next(camera, None)
if first_camera is None or first_frame is None or first_frame.modality != "camera-frame":
@@ -188,8 +207,8 @@ def run(args: argparse.Namespace) -> dict[str, object]:
mqtt = iter(
_mqtt_events(
capture_path,
start_epoch_ns=first_frame.epoch_ns,
duration_seconds=args.duration_seconds,
start_epoch_ns=(None if complete_recording else first_frame.epoch_ns),
duration_seconds=selected_duration,
)
)
first_mqtt = next(mqtt, None)
@@ -300,6 +319,7 @@ def run(args: argparse.Namespace) -> dict[str, object]:
"e12": REPORT_SCHEMA,
"e21": "missioncore.e21-replay-source-report/v1",
"e23": "missioncore.e23-replay-source-report/v1",
"e28": "missioncore.e28-worker-replay-source-report/v1",
}[experiment_id]
peak_rss = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
peak_rss_mib = peak_rss / (1024 * 1024 if sys.platform == "darwin" else 1024)
@@ -312,7 +332,14 @@ def run(args: argparse.Namespace) -> dict[str, object]:
"session": session_root.name,
"camera_source": "sensor.camera.right",
"camera_epoch": 1,
"duration_seconds": args.duration_seconds,
"selection": (
"complete-recording"
if complete_recording
else "bounded-duration"
),
"duration_seconds": (
None if complete_recording else duration_seconds
),
"selected_span_seconds": source_span_seconds,
"speed": speed,
"pacing_clock": "recorded-host-arrival",
@@ -379,7 +406,11 @@ def _arguments() -> argparse.Namespace:
parser.add_argument("--port", type=int, default=8012)
parser.add_argument("--duration-seconds", type=float, default=15.0)
parser.add_argument("--speed", type=float, default=1.0)
parser.add_argument("--experiment-id", choices=("e12", "e21", "e23"), default="e12")
parser.add_argument(
"--experiment-id",
choices=("e12", "e21", "e23", "e28"),
default="e12",
)
parser.add_argument("--consumer-timeout-seconds", type=float, default=30.0)
parser.add_argument("--completion-timeout-seconds", type=float, default=30.0)
return parser.parse_args()
experiments/perception/worker/Invoke-E15ShadowInference.ps1
+18 -2
View File
@@ -136,12 +136,16 @@ if (
$live = Get-Content -LiteralPath $liveProfile -Raw | ConvertFrom-Json
if (
$live.schema_version -ne "missioncore.e15-shadow-inference-profile/v1" -or
$live.mode -notin @("replay-shadow-gate", "physical-shadow-gate") -or
$live.mode -notin @(
"replay-shadow-gate",
"worker-replay-gate",
"physical-shadow-gate"
) -or
[bool]$live.authority.commands_enabled -or
[bool]$live.authority.navigation_or_safety_accepted -or
$live.transport.pyav_version -ne "18.0.0"
) { throw "LAB E15/E28 shadow authority contract changed" }
if ($live.mode -eq "physical-shadow-gate") {
if ($live.mode -in @("worker-replay-gate", "physical-shadow-gate")) {
foreach ($relative in @(
"k1link\compute\lidar_local_surface_geometry.py",
"k1link\compute\lidar_local_surface_shadow.py",
@@ -156,6 +160,18 @@ if ($live.mode -eq "physical-shadow-gate") {
$live.local_surface.profile_id -ne "k1-vendor-map-dynamic-local-surface/v1"
) { throw "LAB E28 local-surface profile contract changed" }
}
if (
$live.mode -eq "worker-replay-gate" -and (
$live.replay_source.session_id -ne "20260720T065719Z_viewer_live" -or
$live.replay_source.selection -ne "complete-recording" -or
[double]$live.replay_source.speed -ne 1.0 -or
[bool]$live.replay_source.look_ahead -or
[int]$live.local_surface.point_queue_capacity -ne 2 -or
[double]$live.local_surface.acceptance.minimum_effective_fps -ne 9.0 -or
[double]$live.local_surface.acceptance.maximum_runtime_drop_fraction -ne 0.08 -or
[double]$live.local_surface.acceptance.maximum_p95_result_age_ms -ne 100.0
)
) { throw "LAB E28 complete-recording replay contract changed" }
if ($stabilityProfile) {
$stability = Get-Content -LiteralPath $stabilityProfile -Raw | ConvertFrom-Json
if (
experiments/perception/worker/e28_physical_k1_local_surface_profile.json → experiments/perception/worker/e28_worker_replay_local_surface_profile.json
+17 -5
View File
@@ -1,6 +1,6 @@
{
"schema_version": "missioncore.e15-shadow-inference-profile/v1",
"mode": "physical-shadow-gate",
"mode": "worker-replay-gate",
"authority": {
"commands_enabled": false,
"navigation_or_safety_accepted": false
@@ -14,6 +14,17 @@
"calibration_slot": "camera_1",
"calibration_sha256": "05f3ad9b38b3a4fc95388a8ec83da83c745e217709e51787b3d5aad0969f6fa9"
},
"replay_source": {
"session_id": "20260720T065719Z_viewer_live",
"display_name": "RAVNOVES00",
"selection": "complete-recording",
"speed": 1.0,
"minimum_source_span_seconds": 450.0,
"expected_camera_frames": 4489,
"expected_lidar_events": 4570,
"expected_pose_events": 4598,
"look_ahead": false
},
"transport": {
"wire_schema": "missioncore.live-perception-wire/v1",
"camera_media": "persistent-fmp4-pyav",
@@ -46,15 +57,16 @@
"retention_seconds": 3.0,
"result_capacity": 8,
"acceptance": {
"minimum_bound_frames": 100,
"minimum_bound_frames": 4500,
"minimum_effective_fps": 9.0,
"maximum_pose_miss_fraction": 0.05,
"maximum_point_drop_fraction": 0.01,
"maximum_runtime_drop_fraction": 0.01,
"maximum_p95_result_age_ms": 80.0
"maximum_runtime_drop_fraction": 0.08,
"maximum_p95_result_age_ms": 100.0
}
},
"acceptance": {
"minimum_camera_frames": 140,
"minimum_camera_frames": 4400,
"detector_minimum_effective_fps": 9.5,
"detector_maximum_drop_fraction": 0.01,
"semantic_minimum_effective_fps": 1.8,
experiments/perception/worker/run_e15_shadow_inference.py
+47 -3
View File
@@ -177,10 +177,16 @@ def read_live_profile(path: Path) -> tuple[dict[str, Any], str]:
scheduling = profile.get("scheduling")
temporal = profile.get("temporal")
local_surface = profile.get("local_surface")
replay_source = profile.get("replay_source")
acceptance = profile.get("acceptance")
if (
profile.get("schema_version") != PROFILE_SCHEMA
or profile.get("mode") not in {"replay-shadow-gate", "physical-shadow-gate"}
or profile.get("mode")
not in {
"replay-shadow-gate",
"worker-replay-gate",
"physical-shadow-gate",
}
or not all(
isinstance(value, dict)
for value in (source, authority, transport, scheduling, temporal, acceptance)
@@ -213,6 +219,34 @@ def read_live_profile(path: Path) -> tuple[dict[str, Any], str]:
or not 1 <= float(temporal.get("maximum_pose_point_delta_ms", 0)) <= 1000
):
raise RuntimeError("LAB E15 bounded runtime contract is invalid")
if profile.get("mode") == "worker-replay-gate":
replay_integer_contract = {
"expected_camera_frames": 4489,
"expected_lidar_events": 4570,
"expected_pose_events": 4598,
}
if (
not isinstance(replay_source, dict)
or replay_source.get("session_id")
!= "20260720T065719Z_viewer_live"
or replay_source.get("display_name") != "RAVNOVES00"
or replay_source.get("selection") != "complete-recording"
or float(replay_source.get("speed", 0)) != 1.0
or float(replay_source.get("minimum_source_span_seconds", 0))
< 450
or replay_source.get("look_ahead") is not False
or any(
not isinstance(replay_source.get(key), int)
or isinstance(replay_source.get(key), bool)
or replay_source.get(key) != expected
for key, expected in replay_integer_contract.items()
)
):
raise RuntimeError(
"LAB E28 complete-recording worker replay contract is invalid"
)
elif replay_source is not None:
raise RuntimeError("LAB E15 non-replay profile carries replay source state")
if local_surface is not None:
from k1link.compute.lidar_local_surface_geometry import (
DEFAULT_K1_LOCAL_SURFACE_PROFILE,
@@ -247,7 +281,8 @@ def read_live_profile(path: Path) -> tuple[dict[str, Any], str]:
else None
)
if (
profile.get("mode") != "physical-shadow-gate"
profile.get("mode")
not in {"worker-replay-gate", "physical-shadow-gate"}
or not isinstance(local_surface, dict)
or local_surface.get("enabled") is not True
or local_surface.get("profile_id")
@@ -281,9 +316,11 @@ def read_live_profile(path: Path) -> tuple[dict[str, Any], str]:
local_acceptance.get("maximum_p95_result_age_ms", 0)
)
<= 0
or float(local_acceptance.get("minimum_effective_fps", 0))
<= 0
):
raise RuntimeError(
"LAB E28 physical local-surface profile contract is invalid"
"LAB E28 worker local-surface profile contract is invalid"
)
fractions = (
"detector_maximum_drop_fraction",
@@ -334,6 +371,7 @@ def _local_surface_acceptance_checks(
results = runtime.get("results")
runtime_profile = runtime.get("profile")
result_age = results.get("result_age_ms") if isinstance(results, dict) else None
delivery = runtime.get("delivery")
if not isinstance(points, dict):
points = {}
if not isinstance(poses, dict):
@@ -346,6 +384,8 @@ def _local_surface_acceptance_checks(
runtime_profile = {}
if not isinstance(result_age, dict):
result_age = {}
if not isinstance(delivery, dict):
delivery = {}
point_published = int(points.get("published", 0))
point_bound = int(points.get("bound", 0))
@@ -406,6 +446,10 @@ def _local_surface_acceptance_checks(
"local_surface_maximum_runtime_drop_fraction": runtime_dropped
/ max(1, runtime_published)
<= float(acceptance["maximum_runtime_drop_fraction"]),
"local_surface_minimum_effective_fps": float(
delivery.get("effective_fps", 0)
)
>= float(acceptance["minimum_effective_fps"]),
"local_surface_zero_runtime_failures": result_failed == 0,
"local_surface_maximum_p95_result_age_ms": (
isinstance(p95_result_age, (int, float))
experiments/perception/worker/run_e28_local_surface_wire.py
+615
View File
@@ -0,0 +1,615 @@
#!/usr/bin/env python3
"""Qualify the K1 local-surface runtime through the authenticated replay wire."""
from __future__ import annotations
import argparse
import hashlib
import json
import math
import os
import queue
import select
import threading
import time
from collections import Counter
from contextlib import suppress
from dataclasses import dataclass
from datetime import UTC, datetime
from pathlib import Path
from typing import Any
from run_e12_shadow_transport_probe import (
_connect,
_decode_event,
_read_server_frame,
_send_client_frame,
)
from k1link.compute.lidar_local_surface_geometry import (
DEFAULT_K1_LOCAL_SURFACE_PROFILE,
)
from k1link.compute.lidar_local_surface_shadow import (
K1LocalSurfaceShadowCoordinator,
K1LocalSurfaceShadowResult,
)
from k1link.data_plane import DecodedPointCloudView, DecodedPoseView
from k1link.device_plugins.xgrids_k1.protocol.normalizer import (
normalize_k1_message,
)
REPORT_SCHEMA = "missioncore.e28-local-surface-wire-report/v1"
PROFILE_SCHEMA = "missioncore.e15-shadow-inference-profile/v1"
@dataclass(frozen=True, slots=True)
class _WireMessage:
sequence: int
topic: str
payload: bytes
received_at_epoch_ns: int
received_monotonic_ns: int | None
source: str = "replay"
def _sha256(path: Path) -> str:
digest = hashlib.sha256()
with path.open("rb") as stream:
while block := stream.read(1024 * 1024):
digest.update(block)
return digest.hexdigest()
def _canonical(value: object) -> bytes:
return json.dumps(
value,
ensure_ascii=True,
sort_keys=True,
separators=(",", ":"),
allow_nan=False,
).encode()
def _read_profile(path: Path) -> tuple[dict[str, Any], str]:
resolved = path.resolve(strict=True)
profile = json.loads(resolved.read_text(encoding="utf-8-sig"))
if not isinstance(profile, dict):
raise RuntimeError("LAB E28 worker replay profile is not an object")
replay = profile.get("replay_source")
local = profile.get("local_surface")
authority = profile.get("authority")
expected_profile_sha256 = hashlib.sha256(
_canonical(DEFAULT_K1_LOCAL_SURFACE_PROFILE.to_dict())
).hexdigest()
acceptance = local.get("acceptance") if isinstance(local, dict) else None
if (
profile.get("schema_version") != PROFILE_SCHEMA
or profile.get("mode") != "worker-replay-gate"
or authority
!= {
"commands_enabled": False,
"navigation_or_safety_accepted": False,
}
or not isinstance(replay, dict)
or replay.get("session_id") != "20260720T065719Z_viewer_live"
or replay.get("selection") != "complete-recording"
or float(replay.get("speed", 0)) != 1.0
or replay.get("look_ahead") is not False
or not isinstance(local, dict)
or local.get("enabled") is not True
or local.get("profile_id") != DEFAULT_K1_LOCAL_SURFACE_PROFILE.profile_id
or local.get("profile_sha256") != expected_profile_sha256
or not isinstance(acceptance, dict)
or float(acceptance.get("minimum_effective_fps", 0)) <= 0
or float(acceptance.get("maximum_p95_result_age_ms", 0)) <= 0
or any(
not 0 <= float(acceptance.get(key, -1)) <= 1
for key in (
"maximum_pose_miss_fraction",
"maximum_point_drop_fraction",
"maximum_runtime_drop_fraction",
)
)
):
raise RuntimeError("LAB E28 worker replay profile contract is invalid")
return profile, _sha256(resolved)
def _fraction(numerator: int, denominator: int) -> float:
return numerator / max(1, denominator)
def _read_shadow_token(repository_root: Path) -> str:
path = (
repository_root.resolve(strict=True)
/ ".runtime"
/ "live-perception"
/ "shadow-worker.token"
)
if not path.is_file() or path.is_symlink():
raise RuntimeError("LAB E28 shadow token file is unavailable")
token = path.read_text(encoding="ascii").strip()
if not 40 <= len(token) <= 128 or not token.isascii():
raise RuntimeError("LAB E28 shadow token is invalid")
return token
def _distribution(values: list[float]) -> dict[str, float | int | None]:
if not values:
return {
"sample_count": 0,
"p50": None,
"p95": None,
"maximum": None,
}
ordered = sorted(values)
def percentile(fraction: float) -> float:
position = (len(ordered) - 1) * fraction
lower = math.floor(position)
upper = math.ceil(position)
if lower == upper:
return ordered[lower]
return ordered[lower] + (ordered[upper] - ordered[lower]) * (position - lower)
return {
"sample_count": len(ordered),
"p50": percentile(0.5),
"p95": percentile(0.95),
"maximum": ordered[-1],
}
def run(args: argparse.Namespace) -> dict[str, Any]:
if (
not math.isfinite(args.source_speed)
or not 0.1 <= args.source_speed <= 10
or not math.isfinite(args.max_duration_seconds)
or args.max_duration_seconds <= 0
or not math.isfinite(args.socket_timeout_seconds)
or args.socket_timeout_seconds <= 0
):
raise RuntimeError("LAB E28 worker runtime bounds are invalid")
profile, profile_sha256 = _read_profile(args.profile)
replay = profile["replay_source"]
local = profile["local_surface"]
local_acceptance = local["acceptance"]
output = args.output.resolve()
output.mkdir(mode=0o700, parents=True, exist_ok=False)
frames_path = output / "local-surface-frames.jsonl"
frame_count = 0
all_processing_ms: list[float] = []
all_result_age_ms: list[float] = []
evidence_failures: list[str] = []
evidence_queue_capacity = 32
evidence_queue_maximum_depth = 0
evidence_queue: queue.Queue[object] = queue.Queue(
maxsize=evidence_queue_capacity
)
evidence_stop = object()
with frames_path.open("x", encoding="utf-8", newline="\n") as frame_stream:
def write_evidence() -> None:
nonlocal frame_count
failed = False
while True:
item = evidence_queue.get()
if item is evidence_stop:
return
if failed:
continue
try:
if not isinstance(item, K1LocalSurfaceShadowResult):
raise RuntimeError(
"LAB E28 evidence queue item is invalid"
)
document = item.document()
document["point_evidence"] = {
"point_count": int(item.point_class.shape[0]),
"point_class_sha256": hashlib.sha256(
item.point_class.tobytes(order="C")
).hexdigest(),
"point_height_sha256": hashlib.sha256(
item.point_height_m.tobytes(order="C")
).hexdigest(),
"point_step_candidate_sha256": hashlib.sha256(
item.point_step_candidate.tobytes(order="C")
).hexdigest(),
"source_alignment": "raw-k1-message-sequence",
}
frame_stream.write(
json.dumps(
document,
sort_keys=True,
separators=(",", ":"),
allow_nan=False,
)
+ "\n"
)
frame_count += 1
except Exception as exc:
evidence_failures.append(
f"{type(exc).__name__}: {str(exc)[:240]}"
)
failed = True
evidence_thread = threading.Thread(
target=write_evidence,
name="e28-local-surface-evidence",
daemon=True,
)
evidence_thread.start()
def observe(result: K1LocalSurfaceShadowResult) -> None:
nonlocal evidence_queue_maximum_depth
if evidence_failures:
raise RuntimeError("LAB E28 evidence writer failed")
try:
evidence_queue.put_nowait(result)
except queue.Full as exc:
raise RuntimeError(
"LAB E28 bounded evidence queue overflowed"
) from exc
evidence_queue_maximum_depth = max(
evidence_queue_maximum_depth,
evidence_queue.qsize(),
)
all_processing_ms.append(result.processing_ms)
all_result_age_ms.append(result.result_age_ms)
coordinator = K1LocalSurfaceShadowCoordinator(
point_capacity=int(local["point_queue_capacity"]),
pose_capacity=int(local["pose_buffer_capacity"]),
future_pose_wait_ms=float(local["future_pose_wait_ms"]),
retention_seconds=float(local["retention_seconds"]),
result_capacity=int(local["result_capacity"]),
result_observer=observe,
)
token = _read_shadow_token(args.repository_root)
connection = None
stream = None
counts: Counter[str] = Counter()
payload_bytes: Counter[str] = Counter()
sensor_decode_ms: list[float] = []
first_ingress_sequence: int | None = None
last_ingress_sequence: int | None = None
ingress_sequence_gaps = 0
session_id: str | None = None
session_end_seen = False
first_source_epoch_ns: int | None = None
last_source_epoch_ns: int | None = None
timed_out = False
failures: list[str] = []
started = time.perf_counter()
try:
connection, stream = _connect(
args.host,
args.port,
args.path,
token,
args.socket_timeout_seconds,
)
deadline = time.monotonic() + args.max_duration_seconds
while time.monotonic() < deadline:
readable, _, _ = select.select(
[connection],
[],
[],
min(0.25, max(0.0, deadline - time.monotonic())),
)
if not readable:
continue
opcode, frame = _read_server_frame(stream)
if opcode == 0x8:
break
if opcode == 0x9:
_send_client_frame(stream, 0xA, frame)
continue
if opcode != 0x2:
raise RuntimeError(f"LAB E28 received unexpected websocket opcode {opcode}")
header, payload = _decode_event(frame)
sequence = int(header["ingress_sequence"])
if last_ingress_sequence is not None:
if sequence <= last_ingress_sequence:
raise RuntimeError("LAB E28 ingress sequence is not increasing")
ingress_sequence_gaps += max(
0,
sequence - last_ingress_sequence - 1,
)
if first_ingress_sequence is None:
first_ingress_sequence = sequence
last_ingress_sequence = sequence
event_session_id = str(header["session_id"])
if session_id is None:
session_id = event_session_id
coordinator.begin_session(session_id)
elif session_id != event_session_id:
raise RuntimeError("LAB E28 replay session identity changed")
modality = str(header["modality"])
counts[modality] += 1
payload_bytes[modality] += len(payload)
if modality != "control":
captured_at_epoch_ns = int(header["captured_at_epoch_ns"])
first_source_epoch_ns = (
captured_at_epoch_ns
if first_source_epoch_ns is None
else min(first_source_epoch_ns, captured_at_epoch_ns)
)
last_source_epoch_ns = (
captured_at_epoch_ns
if last_source_epoch_ns is None
else max(last_source_epoch_ns, captured_at_epoch_ns)
)
if modality == "control":
control = json.loads(payload)
if control.get("event") == "session-end":
session_end_seen = True
break
continue
if modality not in {"lidar", "pose"}:
continue
decode_started = time.perf_counter()
normalized = normalize_k1_message(
_WireMessage(
sequence=int(header["source_sequence"]),
topic=str(header["source_id"]),
payload=payload,
received_at_epoch_ns=int(header["captured_at_epoch_ns"]),
received_monotonic_ns=int(header["received_monotonic_ns"]),
),
processing_started_monotonic_ns=time.monotonic_ns(),
)
sensor_decode_ms.append((time.perf_counter() - decode_started) * 1000)
if modality == "lidar" and isinstance(
normalized,
DecodedPointCloudView,
):
coordinator.publish_point_cloud(normalized)
elif modality == "pose" and isinstance(
normalized,
DecodedPoseView,
):
coordinator.publish_pose(normalized)
else:
raise RuntimeError("LAB E28 known sensor modality did not normalize")
else:
timed_out = True
except Exception as exc:
failures.append(f"{type(exc).__name__}: {str(exc)[:240]}")
finally:
try:
coordinator.close(timeout_seconds=30)
except Exception as exc:
failures.append(f"{type(exc).__name__}: {str(exc)[:240]}")
if stream is not None:
with suppress(Exception):
_send_client_frame(stream, 0x8, b"")
with suppress(Exception):
stream.close()
if connection is not None:
with suppress(Exception):
connection.close()
evidence_queue.put(evidence_stop)
evidence_thread.join(timeout=30)
if evidence_thread.is_alive():
failures.append("LAB E28 evidence writer did not stop")
failures.extend(evidence_failures)
frame_stream.flush()
os.fsync(frame_stream.fileno())
snapshot = coordinator.snapshot()
binder = snapshot["binder"]
runtime = snapshot["runtime"]
if not isinstance(binder, dict) or not isinstance(runtime, dict):
raise RuntimeError("LAB E28 local-surface runtime did not initialize")
points = binder["points"]
queue_state = runtime["queue"]
results = runtime["results"]
delivery = runtime["delivery"]
if (
not isinstance(points, dict)
or not isinstance(queue_state, dict)
or not isinstance(results, dict)
or not isinstance(delivery, dict)
):
raise RuntimeError("LAB E28 local-surface telemetry is invalid")
point_published = int(points["published"])
point_bound = int(points["bound"])
point_missed = int(points["missed"])
point_dropped = int(points["dropped_overflow"])
runtime_published = int(queue_state["published"])
runtime_consumed = int(queue_state["consumed"])
runtime_dropped = int(queue_state["dropped_overflow"])
full_processing_distribution = _distribution(all_processing_ms)
full_result_age_distribution = _distribution(all_result_age_ms)
p95_result_age = full_result_age_distribution["p95"]
source_span_seconds = (
0.0
if first_source_epoch_ns is None or last_source_epoch_ns is None
else (last_source_epoch_ns - first_source_epoch_ns) / 1_000_000_000
)
checks = {
"exact_source_speed": math.isclose(
args.source_speed,
float(replay["speed"]),
rel_tol=0,
abs_tol=1e-9,
),
"complete_recording_span": source_span_seconds
>= float(replay["minimum_source_span_seconds"]),
"exact_camera_event_count": counts["camera-frame"] == int(replay["expected_camera_frames"]),
"exact_lidar_event_count": counts["lidar"] == int(replay["expected_lidar_events"]),
"exact_pose_event_count": counts["pose"] == int(replay["expected_pose_events"]),
"session_end_seen": session_end_seen,
"zero_ingress_sequence_gaps": ingress_sequence_gaps == 0,
"minimum_bound_frames": point_bound >= int(local_acceptance["minimum_bound_frames"]),
"binder_accounting": point_bound + point_missed + point_dropped + int(points["depth"])
== point_published,
"binder_to_runtime_accounting": point_bound == runtime_published,
"maximum_pose_miss_fraction": _fraction(
point_missed,
point_published,
)
<= float(local_acceptance["maximum_pose_miss_fraction"]),
"maximum_point_drop_fraction": _fraction(
point_dropped,
point_published,
)
<= float(local_acceptance["maximum_point_drop_fraction"]),
"runtime_accounting": runtime_consumed + runtime_dropped + int(queue_state["depth"])
== runtime_published,
"maximum_runtime_drop_fraction": _fraction(
runtime_dropped,
runtime_published,
)
<= float(local_acceptance["maximum_runtime_drop_fraction"]),
"minimum_effective_fps": float(delivery["effective_fps"])
>= float(local_acceptance["minimum_effective_fps"]),
"zero_runtime_failures": int(results["failed"]) == 0,
"persisted_frame_accounting": frame_count == int(results["published"]),
"maximum_p95_result_age_ms": (
isinstance(p95_result_age, int | float)
and not isinstance(p95_result_age, bool)
and math.isfinite(float(p95_result_age))
and float(p95_result_age) <= float(local_acceptance["maximum_p95_result_age_ms"])
),
"shadow_authority_only": snapshot["authority"]
== {
"commands_enabled": False,
"navigation_or_safety_accepted": False,
},
"zero_failures": not failures,
"zero_timeout": not timed_out,
}
accepted = all(checks.values())
report = {
"schema_version": REPORT_SCHEMA,
"created_at_utc": datetime.now(UTC)
.isoformat(timespec="milliseconds")
.replace("+00:00", "Z"),
"state": "accepted" if accepted else "rejected",
"ground_truth": False,
"identity": {
"profile_sha256": profile_sha256,
"session_id": session_id,
"source_session_id": replay["session_id"],
"runner_sha256": _sha256(Path(__file__).resolve(strict=True)),
},
"source": {
"selection": "complete-recording",
"speed": args.source_speed,
"source_span_seconds": source_span_seconds,
"look_ahead": False,
},
"transport": {
"counts": dict(sorted(counts.items())),
"payload_bytes": dict(sorted(payload_bytes.items())),
"first_ingress_sequence": first_ingress_sequence,
"last_ingress_sequence": last_ingress_sequence,
"ingress_sequence_gaps": ingress_sequence_gaps,
"session_end_seen": session_end_seen,
"timed_out": timed_out,
},
"local_surface": snapshot,
"worker": {
"wall_seconds": time.perf_counter() - started,
"sensor_decode_ms": {
**_distribution(sensor_decode_ms),
},
"local_surface_processing_ms": full_processing_distribution,
"local_surface_result_age_ms": full_result_age_distribution,
"persisted_frames": frame_count,
"evidence_queue": {
"capacity": evidence_queue_capacity,
"maximum_depth": evidence_queue_maximum_depth,
"depth": evidence_queue.qsize(),
"failures": len(evidence_failures),
},
},
"artifacts": {
"local_surface_frames": {
"path": frames_path.name,
"byte_length": frames_path.stat().st_size,
"sha256": _sha256(frames_path),
}
},
"acceptance": {
"accepted": accepted,
"checks": checks,
"commands_enabled": False,
"navigation_or_safety_accepted": False,
},
"failures": failures,
"limitations": [
"Recorded K1 evidence is replayed through the authenticated wire; "
"this is not a physical acquisition test.",
"Recorded host-arrival timing is preserved; there is no hardware clock qualification.",
"Observed points do not prove free or traversable unknown space.",
],
}
report_sha256 = hashlib.sha256(_canonical(report)).hexdigest()
report["result_id"] = f"e28-local-surface-wire-{report_sha256}"
report_path = output / "report.json"
report_path.write_bytes(
json.dumps(
report,
ensure_ascii=False,
indent=2,
sort_keys=True,
allow_nan=False,
).encode()
+ b"\n"
)
return report
def _arguments() -> argparse.Namespace:
script = Path(__file__).resolve()
root = script.parents[3] if len(script.parents) > 3 else Path.cwd()
parser = argparse.ArgumentParser()
parser.add_argument(
"--repository-root",
type=Path,
default=root,
)
parser.add_argument(
"--profile",
type=Path,
default=Path(__file__).with_name("e28_worker_replay_local_surface_profile.json"),
)
parser.add_argument("--host", default="127.0.0.1")
parser.add_argument("--port", type=int, default=8028)
parser.add_argument(
"--path",
default=("/api/v1/device-plugins/nodedc.device.xgrids-lixelkity-k1/live-perception-shadow"),
)
parser.add_argument("--socket-timeout-seconds", type=float, default=30.0)
parser.add_argument("--max-duration-seconds", type=float, default=600.0)
parser.add_argument("--source-speed", type=float, default=1.0)
parser.add_argument("--output", type=Path, required=True)
return parser.parse_args()
def main() -> int:
report = run(_arguments())
print(
json.dumps(
{
"result_id": report["result_id"],
"state": report["state"],
"source": report["source"],
"transport": report["transport"],
"worker": report["worker"],
"acceptance": report["acceptance"],
},
ensure_ascii=False,
indent=2,
sort_keys=True,
)
)
return 0 if report["acceptance"]["accepted"] else 1
if __name__ == "__main__":
raise SystemExit(main())