feat(observatory): add portable calculation profiles
This commit is contained in:
@@ -278,11 +278,13 @@ an integrity or product need justifies a targeted migration.
|
||||
## Observatory durable recorded queue and Worker dispatch
|
||||
|
||||
Selecting a source and a laboratory setup in Observatory is not itself a run.
|
||||
The public submission contains only `source_session_id`, `setup_id` and an
|
||||
idempotency key. It cannot supply commands, executable text, filesystem paths,
|
||||
container images, model identities, resource limits or priority. The server
|
||||
resolves the allowlisted pair and seals all executable identity into one durable
|
||||
record:
|
||||
The exact legacy submission contains only `source_session_id`, `setup_id` and an
|
||||
idempotency key. A portable submission additionally returns the server-owned
|
||||
`definition_sha256` and one content-bound `check_sha256` obtained from preflight;
|
||||
both must be echoed unchanged during submit. Neither request can supply commands,
|
||||
executable text, filesystem paths, container images, model identities, resource
|
||||
limits or priority. The server resolves the allowlisted identities and seals all
|
||||
executable identity into one durable record:
|
||||
|
||||
- the current source-catalog snapshot captured at admission, plus immutable source
|
||||
bundle and source-capability-manifest SHA-256 identities;
|
||||
@@ -307,22 +309,33 @@ not an unknown model dependency. The binding pins the exact source pack rather
|
||||
than a volatile whole-catalog digest; the server captures and seals the current
|
||||
catalog snapshot into each admitted job.
|
||||
|
||||
`LAB V1 · EoMT Cityscapes Large 1024 + DDRNet-39` now has a source-independent
|
||||
portable RunDefinition and lightweight recorded-source capability probe. The
|
||||
definition seals the exact model/component/resource identities, observation-only
|
||||
authority and generic `missioncore.recorded-eomt-ddrnet-review/v2` result
|
||||
contract; it contains no source Session id or label. Compatibility is derived
|
||||
from the selected Session's real K1 capabilities rather than its name. A
|
||||
compatible source may therefore report capability `pass` independently from
|
||||
executor readiness.
|
||||
Two source-independent definitions are projected by the portable catalog:
|
||||
|
||||
That portable foundation is not an executable product path yet. Its executor is
|
||||
`not-installed`, and there is no accepted server-side definition-SHA/check-SHA
|
||||
fenced check/submit API, generic v2 result assembler/publisher or deployed Worker
|
||||
executor. Preflight consequently remains blocked and the UI must not promise or
|
||||
expose enqueue. The old `missioncore.lab-v1-vegetation-shadow/v1` result is not an
|
||||
exact/existing result of the generic portable definition, even for its original
|
||||
source; it remains available only in the immutable legacy LAB catalog.
|
||||
- `LAB V1 · EoMT Cityscapes Large 1024 + DDRNet-39` seals the exact
|
||||
model/component/resource identities and the
|
||||
`missioncore.recorded-eomt-ddrnet-review/v2` result contract;
|
||||
- `M4.9T5 · TRAVEL TGS · CPU-only, без ML` v2 seals an explicitly empty model
|
||||
manifest, dynamic source-derived frame counts, causal TGS invariants and the
|
||||
`missioncore.recorded-tgs-costmap-review/v2` result contract.
|
||||
|
||||
Neither portable definition contains a source Session id, label, fixed frame
|
||||
count or filesystem path. Compatibility is derived independently for every
|
||||
setup from the selected Session's real K1 capabilities. A compatible source may
|
||||
therefore report capability `pass` while that setup's executor remains blocked.
|
||||
Blocked definitions stay projectable and do not prevent an unrelated ready
|
||||
definition from entering the durable queue allowlist.
|
||||
|
||||
The server now owns a definition-SHA/check-SHA fenced portable check/submit
|
||||
boundary. Submission still fails closed unless that exact definition has a
|
||||
sealed `ready` executor release and image and is present in the durable queue.
|
||||
Both repository portable definitions currently declare `not-installed`, so
|
||||
their preflight remains blocked; no release or image hash is fabricated. The
|
||||
generic v2 assemblers, verified publisher and Worker transport exist as a
|
||||
dormant fail-closed implementation, but exact executor installation, central
|
||||
storage, authentication, tunnel acceptance and end-to-end canaries remain
|
||||
pending. The old `missioncore.lab-v1-vegetation-shadow/v1` result and the exact
|
||||
RAVNOVES00 M4.9 result remain only in their existing immutable catalogs;
|
||||
neither is reclassified as a result of a portable definition.
|
||||
|
||||
Recorded work has priority rank `100`. A future live K1 lease has rank `0` and
|
||||
closes new recorded claims while it is pending or active. Cooperative executors
|
||||
@@ -342,8 +355,10 @@ foundation. The production app hard-disables the Worker router even when a valid
|
||||
credential is present until an expiring claim lease and verified result publisher
|
||||
are accepted. Installation of the exact executors, Worker deployment and wiring
|
||||
from the real K1 lifecycle to live-lease triggers remain pending. Therefore a
|
||||
submitted M4.9T5 job may honestly wait in `queued` without implying that Worker
|
||||
006 can execute it yet; portable LAB V1 cannot currently be submitted at all.
|
||||
submitted exact legacy M4.9T5 job may honestly wait in `queued` without implying
|
||||
that Worker 006 can execute it yet. Portable LAB V1 and portable M4.9T5 cannot
|
||||
currently be submitted because their executor states are `not-installed` and
|
||||
the authenticated Worker dispatch gate is closed.
|
||||
|
||||
Worker telemetry remains secondary observation evidence. It does not replace the
|
||||
authoritative queue ledger, result validation or common laboratory receipt. K1
|
||||
|
||||
@@ -0,0 +1,121 @@
|
||||
# ADR 0047: Verified portable Observatory result publication
|
||||
|
||||
Date: 2026-08-31
|
||||
Status: accepted as a backend foundation; production validators, transport and
|
||||
executor wiring remain blocked
|
||||
|
||||
## Context
|
||||
|
||||
The durable Observatory queue deliberately treats a Worker `succeed` call as a
|
||||
transport acknowledgement. Its `result_id` and SHA-256 do not prove that an
|
||||
artifact exists, that it was computed from the admitted source, that it obeys
|
||||
the selected RunDefinition, or that it has observation-only authority. Publishing
|
||||
that acknowledgement directly as a LAB session would let incomplete, corrupt or
|
||||
mislabelled output enter the same catalog as immutable evidence.
|
||||
|
||||
The current canonical LAB V1 result is a separate preserved legacy result. It
|
||||
must not be reinterpreted as a portable v2 result, rewritten with new provenance,
|
||||
or used as evidence that the portable executor exists.
|
||||
|
||||
## Decision
|
||||
|
||||
Portable results cross a new backend-only verification boundary implemented in
|
||||
`k1link.observatory.portable_result_contract` and
|
||||
`k1link.observatory.portable_result_publisher`. The boundary does not change the
|
||||
queue, the source-admission service, Session API, setup projection, UI, K1,
|
||||
Simulation, or the legacy canonical publisher.
|
||||
|
||||
A Worker-side assembler must produce one directory whose basename is the
|
||||
SHA-256 of its canonical `manifest.json`. The manifest schema is
|
||||
`missioncore.observatory-portable-result-package/v1`; JSON bytes are canonical
|
||||
UTF-8 with sorted keys and no insignificant whitespace. Its separately hashed
|
||||
identity binds:
|
||||
|
||||
1. queue `job_id`, request identity, execution identity, submission receipt and
|
||||
claim generation;
|
||||
2. source Session, catalog snapshot, source bundle, capability manifest and
|
||||
source-adapter identities;
|
||||
3. the complete portable RunDefinition identity, including source requirements,
|
||||
components, models, resource profile, result contract, executor and authority;
|
||||
4. result id, result schema, result kind and result-contract SHA-256;
|
||||
5. a canonical, role-sorted artifact list with confined relative paths, media
|
||||
types, byte lengths and SHA-256 identities;
|
||||
6. observation-only authority.
|
||||
|
||||
Exactly one non-empty `result-document` JSON artifact is required. Package roots,
|
||||
the manifest, every path component and every artifact are checked without
|
||||
following symlinks outside the package. The queue's terminal result SHA must equal
|
||||
the canonical manifest SHA and the package directory name.
|
||||
|
||||
Before catalog publication, the server also:
|
||||
|
||||
- resolves the exact `(setup_id, definition_sha256)` in the portable registry and
|
||||
proves every recorded queue field equals the resulting RunDefinition;
|
||||
- rejects not-installed executors and the reserved legacy
|
||||
`lab-v1-vegetation-shadow-<sha256>` namespace;
|
||||
- rechecks the current SessionStore catalog snapshot;
|
||||
- reads both source-admission documents from
|
||||
`observatory-portable-source-contracts/<sha256>.json`, verifies their bytes,
|
||||
schemas, cross-links, adapter, source and authority;
|
||||
- invokes a validator registered by exact result-contract SHA-256;
|
||||
- copies the package manifest and all output artifacts into the central immutable
|
||||
content-addressed artifact store and records its manifest id;
|
||||
- publishes one idempotent `LabSessionBinding` through `SessionStore`.
|
||||
|
||||
There is intentionally no generic “JSON looks plausible” validator. An unknown
|
||||
result contract fails before artifact-store or SessionStore mutation. A validator
|
||||
must understand the exact result schema and determine that the result document
|
||||
and supporting artifacts are accepted evidence.
|
||||
|
||||
## Calculation-profile provenance
|
||||
|
||||
The publisher does not read a browser selection or infer a profile from a result
|
||||
name. It requires a server-owned policy bound to the exact definition id, version
|
||||
and SHA-256. The resulting immutable provenance contains
|
||||
`missioncore.observatory-calculation-profile/v1` with:
|
||||
|
||||
- `setup_id`;
|
||||
- full display name;
|
||||
- origin `archived-definition`;
|
||||
- definition id, version and SHA-256.
|
||||
|
||||
It also stores a SHA-256 of that calculation-profile document. A later Session API
|
||||
projection can therefore read `calculation_profile` from the result's provenance
|
||||
instead of reporting whichever setup happens to be selected now.
|
||||
|
||||
No replay capability is invented. The portable result package is preserved in
|
||||
the central artifact store, while a result-schema-specific viewer/replay adapter
|
||||
must be accepted separately before the catalog binding can claim visual replay.
|
||||
|
||||
## Current fail-closed blockers
|
||||
|
||||
The publisher and focused contract tests are implemented, but the end-to-end
|
||||
production loop remains unavailable for concrete reasons:
|
||||
|
||||
1. neither `recorded-eomt-ddrnet-review-v2` nor
|
||||
`m49-tgs-portable-review-v2` has an installed exact result-contract validator;
|
||||
2. the current Worker protocol returns only `result_id` and manifest SHA-256; it
|
||||
has no accepted package upload/CAS handoff that gives Mission Core the matching
|
||||
content-addressed directory;
|
||||
3. portable executor releases that emit this package contract are not installed
|
||||
and physically accepted on Worker 006;
|
||||
4. the application has not registered definition-bound calculation-profile
|
||||
publication policies or wired the publisher into the terminal Worker flow;
|
||||
5. no result-schema-specific replay-capability adapter has been accepted.
|
||||
|
||||
These are explicit blockers. The backend must not fabricate a package, reuse a
|
||||
legacy result, trust a Worker success receipt, guess model/profile provenance, or
|
||||
expose an enqueue/result promise to bypass them.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Queue success and Observatory publication remain distinct evidence states.
|
||||
- Exact retry is safe: content-addressed artifact publication and the immutable
|
||||
SessionStore binding are idempotent; a conflicting result id fails closed.
|
||||
- Source, definition, model, resource-profile and calculation-profile identities
|
||||
remain available in one portable provenance document.
|
||||
- Artifact-store writes may leave harmless immutable unreferenced objects if the
|
||||
final SessionStore transaction detects a conflict; they cannot overwrite an
|
||||
existing identity.
|
||||
- The legacy canonical LAB V1 result and its admission logic remain byte-for-byte
|
||||
outside this publisher.
|
||||
@@ -0,0 +1,196 @@
|
||||
# Portable Observatory Worker 006 operational boundary
|
||||
|
||||
## Scope and current state
|
||||
|
||||
This runbook covers only recorded, observation-only Observatory jobs. It does
|
||||
not change K1 acquisition/control, Simulation/Gaussian, legacy LAB execution,
|
||||
navigation or safety authority.
|
||||
|
||||
The server queue, renewable claim lease, verified source/result transport and
|
||||
result publisher exist, but the production Worker API remains deliberately
|
||||
disabled. All three application gates stay `False` until exact executors are
|
||||
installed and a complete transport smoke has passed:
|
||||
|
||||
- `OBSERVATORY_WORKER_CLAIM_LEASE_READY`;
|
||||
- `OBSERVATORY_WORKER_VERIFIED_RESULT_PUBLISHER_READY`;
|
||||
- `OBSERVATORY_WORKER_PRODUCTION_API_ENABLED`.
|
||||
|
||||
The canonical central artifact store is already declared in the installed
|
||||
Mission Core LaunchAgent as:
|
||||
|
||||
```text
|
||||
MISSIONCORE_ARTIFACT_STORE_ROOT=/Volumes/docker/nodedc-mission-core/artifact-store
|
||||
```
|
||||
|
||||
On 2026-08-31 `/Volumes/docker` was not mounted. This is a fail-closed
|
||||
preflight failure, not permission to create a checkout-local substitute. The
|
||||
portable Worker server composition now requires `central_status=ready` before
|
||||
it can be constructed.
|
||||
|
||||
The installed `com.nodedc.mission-core.local` LaunchAgent was running from the
|
||||
M5 Observatory feature worktree while retaining the established Mission Core
|
||||
data directory in the main checkout. It declares the central artifact-store
|
||||
root and cache limits, but not the two portable storage roots below. Do not
|
||||
silently edit or restart this hybrid local service; carry the environment and
|
||||
working-directory change through one separately reviewed, hash-gated service
|
||||
transition after the code release is sealed.
|
||||
|
||||
Large portable inputs and in-flight results have no Mission Core data-directory
|
||||
fallback. The server requires both roots through environment-only
|
||||
configuration:
|
||||
|
||||
```text
|
||||
MISSIONCORE_OBSERVATORY_WORKER_SOURCE_CAS_ROOT=/Volumes/docker/nodedc-mission-core/observatory-worker/source-cas
|
||||
MISSIONCORE_OBSERVATORY_WORKER_RESULT_STAGING_ROOT=/Volumes/docker/nodedc-mission-core/observatory-worker/result-staging
|
||||
```
|
||||
|
||||
Both directories must already exist, be canonical non-symlink directories,
|
||||
remain inside `/Volumes/docker/nodedc-mission-core`, and be disjoint from each
|
||||
other and from `artifact-store`. Mission Core never creates these configured
|
||||
roots. If `/Volumes/docker` is absent or is only a local directory rather than
|
||||
a mounted volume, composition fails closed before queue or Worker API exposure.
|
||||
Provision the directories only through the reviewed server deployment after
|
||||
the SMB mount preflight succeeds.
|
||||
|
||||
Read-only Worker evidence on the same date:
|
||||
|
||||
- strict-pinned `ssh -o BatchMode=yes mission-gpu` succeeds;
|
||||
- Worker host identity remains `DESKTOP-OPJ8J04`;
|
||||
- Windows OpenSSH `sshd` is running;
|
||||
- `AllowTcpForwarding` and `GatewayPorts` use OpenSSH defaults: forwarding is
|
||||
allowed and remote listeners are not exposed beyond loopback;
|
||||
- Worker loopback port `18080` had no listener.
|
||||
|
||||
## Network shape
|
||||
|
||||
Mission Core remains the only backend on Mac loopback port `8000`. The Mac
|
||||
owns one reverse SSH tunnel through the existing strict-pinned `mission-gpu`
|
||||
alias:
|
||||
|
||||
```text
|
||||
Worker 006 process
|
||||
-> http://127.0.0.1:18080
|
||||
-> encrypted SSH reverse forwarding
|
||||
-> Mac http://127.0.0.1:8000
|
||||
```
|
||||
|
||||
The exact forwarding declaration is:
|
||||
|
||||
```text
|
||||
-R 127.0.0.1:18080:127.0.0.1:8000
|
||||
```
|
||||
|
||||
It neither opens a LAN listener nor sends a credential in process arguments.
|
||||
`ObservatoryWorkerHttpGateway` independently rejects plaintext HTTP to any
|
||||
non-loopback host. `worker_tunnel_launchd.py` builds a pure, hashable launchd
|
||||
plan; `scripts/plan_observatory_worker_tunnel.py` prints that plan and performs
|
||||
no installation.
|
||||
|
||||
## Bearer credential
|
||||
|
||||
The server reads one credential from the fixed private data path:
|
||||
|
||||
```text
|
||||
<MISSIONCORE_DATA_DIR>/worker-auth/observatory-worker.token
|
||||
```
|
||||
|
||||
The installed Worker release receives the same secret through its own private,
|
||||
runner-managed file and passes only its path as
|
||||
`MISSIONCORE_OBSERVATORY_WORKER_TOKEN_FILE`. The token is ASCII, 32–512 bytes,
|
||||
has no newline, is never stored in Git, an artifact, a plist, an environment
|
||||
value, a command line or Ops plaintext, and is read with no-follow semantics.
|
||||
The admitted service runtime is POSIX and requires mode `0600` or narrower;
|
||||
native Windows ACL handling is intentionally not guessed.
|
||||
|
||||
The Worker service accepts only these non-executable settings:
|
||||
|
||||
```text
|
||||
MISSIONCORE_OBSERVATORY_WORKER_BASE_URL=http://127.0.0.1:18080
|
||||
MISSIONCORE_OBSERVATORY_WORKER_TOKEN_FILE=<absolute-private-path>
|
||||
MISSIONCORE_OBSERVATORY_WORKER_WORK_ROOT=<absolute-private-D-backed-path>
|
||||
MISSIONCORE_OBSERVATORY_WORKER_IDLE_POLL_SECONDS=1
|
||||
MISSIONCORE_OBSERVATORY_WORKER_TRANSPORT_BACKOFF_SECONDS=5
|
||||
MISSIONCORE_OBSERVATORY_WORKER_MAX_TRANSPORT_FAILURES=12
|
||||
```
|
||||
|
||||
There is no configurable module, command, image or executable entrypoint.
|
||||
|
||||
## Install-time executor seam
|
||||
|
||||
`compose_installed_observatory_worker_service` is called only by a reviewed
|
||||
Worker release. That release injects a constructed
|
||||
`ObservatoryWorkerExecutorRegistry` directly in memory. Before opening the
|
||||
HTTP gateway, service composition resolves the four-digest identity of every
|
||||
portable RunDefinition whose executor state is `ready`:
|
||||
|
||||
- executor release SHA-256;
|
||||
- executor image SHA-256;
|
||||
- model manifest SHA-256;
|
||||
- resource profile SHA-256.
|
||||
|
||||
No ready definitions, an empty registry, or one missing identity stops the
|
||||
service before its first claim. Blocked candidates cannot be selected through
|
||||
configuration and are not silently registered.
|
||||
|
||||
## Staged deployment and smoke sequence
|
||||
|
||||
No step below was applied by this implementation increment.
|
||||
|
||||
1. Seal each executor release and installation receipt. Change a portable
|
||||
RunDefinition to `ready` only when its exact release/image identities and
|
||||
local adapter admission agree.
|
||||
2. Mount the existing canonical SMB artifact store. Require Mission Core
|
||||
artifact status `central_status=ready`; do not initialize a local surrogate.
|
||||
Through the reviewed server deployment, provision the two disjoint portable
|
||||
roots above, set both environment values, and verify they resolve inside the
|
||||
same mounted `/Volumes/docker/nodedc-mission-core` boundary.
|
||||
3. Provision one bearer credential through the deployment-owned secret path
|
||||
on both Mac and the admitted POSIX Worker service runtime. Verify file type,
|
||||
no-link handling and private permissions without printing the value.
|
||||
4. Generate the reverse-tunnel launchd plan, review its SHA-256 and exact
|
||||
arguments, then install it through a separate hash-gated local-service
|
||||
change. Accept only Worker-side `127.0.0.1:18080/api/health` reaching the
|
||||
canonical Mac service; no second backend is started.
|
||||
5. Install the exact Worker release. Its fixed entrypoint loads the sealed
|
||||
RunDefinition registry and its in-memory executor registry, then calls
|
||||
`compose_installed_observatory_worker_service`. The process must refuse a
|
||||
missing token, non-loopback plaintext URL, unsafe work root, absent ready
|
||||
definition or executor coverage gap.
|
||||
6. Restart the canonical Mission Core process once with CAS and server token
|
||||
available. Keep the Worker route disabled and verify that K1,
|
||||
Simulation/Gaussian and legacy LAB surfaces are unchanged.
|
||||
7. In a separate reviewed source gate, flip the claim-lease, verified-publisher
|
||||
and production-API flags together. Restart only the canonical port `8000`
|
||||
service. An authenticated empty claim must return `204`; missing/wrong bearer
|
||||
and wrong contour identity must remain `401/403`.
|
||||
8. Submit one short recorded K1 canary for each profile. Require exact source
|
||||
materialization, lease heartbeat, result-package digest validation, central
|
||||
artifact publication, immutable calculation-profile provenance and a
|
||||
reopenable Observatory result.
|
||||
9. During a bounded recorded canary, start the existing live K1 priority
|
||||
transition. Require the recorded job to pause/defer and resume only after
|
||||
live K1 releases the single Worker resource. This test never grants control
|
||||
or navigation authority.
|
||||
|
||||
## Remaining blockers
|
||||
|
||||
- canonical SMB artifact store is currently unmounted on the Mac;
|
||||
- portable source CAS and result-staging directories/environment values are not
|
||||
provisioned;
|
||||
- the installed Mission Core LaunchAgent has not received a sealed
|
||||
working-directory/environment transition for this release;
|
||||
- the shared bearer credential has not been provisioned;
|
||||
- the reverse tunnel plan has not been installed or smoked;
|
||||
- the Worker polling entrypoint has not been packaged into an admitted POSIX
|
||||
Worker release;
|
||||
- both exact executor releases/install receipts still need their own seals;
|
||||
- production flags and the server route remain off by design;
|
||||
- no end-to-end result has yet crossed Worker 006 -> central store ->
|
||||
Observatory under this new path.
|
||||
|
||||
The Synology root-owned `nodedc-deploy` registry has no
|
||||
`mission-core-worker` component. Older Mission Core Worker shadow artifacts
|
||||
explicitly declare that they are outside that registry. Do not route a Windows
|
||||
Worker install through an unrelated NAS component or weaken the deploy canon;
|
||||
the durable Worker release needs its own exact reviewed installation transition
|
||||
and rollback evidence.
|
||||
Reference in New Issue
Block a user