feat(observatory): add durable recorded compute queue

This commit is contained in:
DCCONSTRUCTIONS
2026-08-31 01:04:25 +03:00
parent e72609120f
commit d67e86176e
23 changed files with 6300 additions and 369 deletions
+53 -22
View File
@@ -275,33 +275,64 @@ execution canaries. Once they prove one automatic run-to-evidence path, the LAB
canonicalization slice closes and remaining historical work stays `legacy` unless
an integrity or product need justifies a targeted migration.
## Observatory preparation and Worker dispatch
## Observatory durable recorded queue and Worker dispatch
Selecting a source and a laboratory setup in Observatory is not itself a run.
Before a remote executor is installed, Mission Core may persist only an immutable
`missioncore.observatory-run-preparation/v1` receipt. That receipt binds the exact
source-catalog snapshot, setup id, RunDefinition version and configuration digest
to one idempotency key. Its only admitted terminal state is `blocked`; it has no
`run_id`, dispatch receipt or execution authority.
The public submission contains only `source_session_id`, `setup_id` and an
idempotency key. It cannot supply commands, executable text, filesystem paths,
container images, model identities, resource limits or priority. The server
resolves the allowlisted pair and seals all executable identity into one durable
record:
A real Worker run begins only after a separate durable dispatcher has atomically
published an immutable dispatch receipt. The executor must be an additive,
allowlisted service owned outside the K1 acquisition path. It may accept bounded
versioned data, but never an arbitrary command, filesystem path or PowerShell
fragment from the UI. Exact retries return the original receipt; a reused key with
different identity fails closed. Transport uncertainty produces an explicit
reconciliation state and never an automatic duplicate run.
- the current source-catalog snapshot captured at admission, plus immutable source
bundle and source-capability-manifest SHA-256 identities;
- source-adapter id, version and SHA-256;
- setup RunDefinition id, version and SHA-256;
- executor release and image SHA-256 identities;
- learned-model manifest and resource-profile identities;
- checkpoint policy and the server-owned priority class.
The first executable Observatory setup remains the exact recorded
RAVNOVES00/M4.9T5 definition. Its full and integrated successors are different
definitions and results. The current RAVNOVES004TREE LAB V1 projection remains
replay-only until a new, independently versioned RunDefinition is reconstructed;
Mission Core does not invent a configuration for an already published result.
An exact retry returns the existing job. Reusing an idempotency key for another
identity fails closed. The ordinary lifecycle advances through `accepted`,
`queued`, `claimed` and `running`, then terminates as `succeeded` or `failed`.
`paused`, `preemption-pending` and `reconciliation-required` are explicit safety
branches rather than hidden retries. The durable queue serializes one recorded
compute owner and does not equate a queued receipt with a Worker execution
receipt.
Worker telemetry is secondary observation evidence. It does not replace the
authoritative run ledger, dispatch receipt, result validation or common laboratory
receipt. K1 control, Simulation/Gaussian runtimes and legacy LAB projections are
outside this executor boundary and are not restarted or migrated by it.
The only currently queueable pair is the exact `RAVNOVES00` source and
`M4.9T5 · TRAVEL TGS · CPU-only, без ML` RunDefinition. It seals an explicitly
empty learned-model list because TGS is an algorithmic CPU pipeline, not an
unknown model dependency. The binding pins the exact source pack rather than a
volatile whole-catalog digest; the server captures and seals the current catalog
snapshot into each admitted job. `LAB V1 · EoMT Cityscapes Large 1024 + DDRNet-39`
remains a result-only replay: no independently versioned durable executor exists
for that published result, so Mission Core does not invent one. Every other
session/setup pair fails closed until its own source adapter, capability manifest,
RunDefinition and executor identities are admitted.
Recorded work has priority rank `100`. A future live K1 lease has rank `0` and
closes new recorded claims while it is pending or active. Cooperative executors
yield only at an allowlisted checkpoint. A non-checkpointable monolith must be
cancelled by the scheduler, discard all staging output and later restart from
zero. Cancellation uses a crash-safe two-phase protocol: Mission Core first
persists one stable cancellation intent and activates live work only after an
identity-bound receipt proves resource release and staging discard. A retry uses
the same cancellation identity. A missing callback or receipt remains durably
pending for retry; a conflicting or indeterminate receipt enters reconciliation.
Every unresolved form blocks live activation and concurrent ownership.
An explicit live terminal trigger requeues paused recorded work.
ADR 0046 records this decision. The durable queue and identity contracts are
implemented; Worker claim transport, installation of the exact M4.9T5 executor
and wiring from the real K1 lifecycle to live-lease triggers remain pending.
Therefore a submitted M4.9T5 job may honestly wait in `queued` without implying
that Worker 006 can execute it yet.
Worker telemetry remains secondary observation evidence. It does not replace the
authoritative queue ledger, result validation or common laboratory receipt. K1
acquisition and control, the legacy LAB archive, and Simulation/Gaussian runtimes
are outside this boundary and are not restarted, migrated or modified by it.
## Planning discipline