feat(observatory): add durable recorded compute queue
This commit is contained in:
@@ -275,33 +275,64 @@ execution canaries. Once they prove one automatic run-to-evidence path, the LAB
|
||||
canonicalization slice closes and remaining historical work stays `legacy` unless
|
||||
an integrity or product need justifies a targeted migration.
|
||||
|
||||
## Observatory preparation and Worker dispatch
|
||||
## Observatory durable recorded queue and Worker dispatch
|
||||
|
||||
Selecting a source and a laboratory setup in Observatory is not itself a run.
|
||||
Before a remote executor is installed, Mission Core may persist only an immutable
|
||||
`missioncore.observatory-run-preparation/v1` receipt. That receipt binds the exact
|
||||
source-catalog snapshot, setup id, RunDefinition version and configuration digest
|
||||
to one idempotency key. Its only admitted terminal state is `blocked`; it has no
|
||||
`run_id`, dispatch receipt or execution authority.
|
||||
The public submission contains only `source_session_id`, `setup_id` and an
|
||||
idempotency key. It cannot supply commands, executable text, filesystem paths,
|
||||
container images, model identities, resource limits or priority. The server
|
||||
resolves the allowlisted pair and seals all executable identity into one durable
|
||||
record:
|
||||
|
||||
A real Worker run begins only after a separate durable dispatcher has atomically
|
||||
published an immutable dispatch receipt. The executor must be an additive,
|
||||
allowlisted service owned outside the K1 acquisition path. It may accept bounded
|
||||
versioned data, but never an arbitrary command, filesystem path or PowerShell
|
||||
fragment from the UI. Exact retries return the original receipt; a reused key with
|
||||
different identity fails closed. Transport uncertainty produces an explicit
|
||||
reconciliation state and never an automatic duplicate run.
|
||||
- the current source-catalog snapshot captured at admission, plus immutable source
|
||||
bundle and source-capability-manifest SHA-256 identities;
|
||||
- source-adapter id, version and SHA-256;
|
||||
- setup RunDefinition id, version and SHA-256;
|
||||
- executor release and image SHA-256 identities;
|
||||
- learned-model manifest and resource-profile identities;
|
||||
- checkpoint policy and the server-owned priority class.
|
||||
|
||||
The first executable Observatory setup remains the exact recorded
|
||||
RAVNOVES00/M4.9T5 definition. Its full and integrated successors are different
|
||||
definitions and results. The current RAVNOVES004TREE LAB V1 projection remains
|
||||
replay-only until a new, independently versioned RunDefinition is reconstructed;
|
||||
Mission Core does not invent a configuration for an already published result.
|
||||
An exact retry returns the existing job. Reusing an idempotency key for another
|
||||
identity fails closed. The ordinary lifecycle advances through `accepted`,
|
||||
`queued`, `claimed` and `running`, then terminates as `succeeded` or `failed`.
|
||||
`paused`, `preemption-pending` and `reconciliation-required` are explicit safety
|
||||
branches rather than hidden retries. The durable queue serializes one recorded
|
||||
compute owner and does not equate a queued receipt with a Worker execution
|
||||
receipt.
|
||||
|
||||
Worker telemetry is secondary observation evidence. It does not replace the
|
||||
authoritative run ledger, dispatch receipt, result validation or common laboratory
|
||||
receipt. K1 control, Simulation/Gaussian runtimes and legacy LAB projections are
|
||||
outside this executor boundary and are not restarted or migrated by it.
|
||||
The only currently queueable pair is the exact `RAVNOVES00` source and
|
||||
`M4.9T5 · TRAVEL TGS · CPU-only, без ML` RunDefinition. It seals an explicitly
|
||||
empty learned-model list because TGS is an algorithmic CPU pipeline, not an
|
||||
unknown model dependency. The binding pins the exact source pack rather than a
|
||||
volatile whole-catalog digest; the server captures and seals the current catalog
|
||||
snapshot into each admitted job. `LAB V1 · EoMT Cityscapes Large 1024 + DDRNet-39`
|
||||
remains a result-only replay: no independently versioned durable executor exists
|
||||
for that published result, so Mission Core does not invent one. Every other
|
||||
session/setup pair fails closed until its own source adapter, capability manifest,
|
||||
RunDefinition and executor identities are admitted.
|
||||
|
||||
Recorded work has priority rank `100`. A future live K1 lease has rank `0` and
|
||||
closes new recorded claims while it is pending or active. Cooperative executors
|
||||
yield only at an allowlisted checkpoint. A non-checkpointable monolith must be
|
||||
cancelled by the scheduler, discard all staging output and later restart from
|
||||
zero. Cancellation uses a crash-safe two-phase protocol: Mission Core first
|
||||
persists one stable cancellation intent and activates live work only after an
|
||||
identity-bound receipt proves resource release and staging discard. A retry uses
|
||||
the same cancellation identity. A missing callback or receipt remains durably
|
||||
pending for retry; a conflicting or indeterminate receipt enters reconciliation.
|
||||
Every unresolved form blocks live activation and concurrent ownership.
|
||||
An explicit live terminal trigger requeues paused recorded work.
|
||||
|
||||
ADR 0046 records this decision. The durable queue and identity contracts are
|
||||
implemented; Worker claim transport, installation of the exact M4.9T5 executor
|
||||
and wiring from the real K1 lifecycle to live-lease triggers remain pending.
|
||||
Therefore a submitted M4.9T5 job may honestly wait in `queued` without implying
|
||||
that Worker 006 can execute it yet.
|
||||
|
||||
Worker telemetry remains secondary observation evidence. It does not replace the
|
||||
authoritative queue ledger, result validation or common laboratory receipt. K1
|
||||
acquisition and control, the legacy LAB archive, and Simulation/Gaussian runtimes
|
||||
are outside this boundary and are not restarted, migrated or modified by it.
|
||||
|
||||
## Planning discipline
|
||||
|
||||
|
||||
Reference in New Issue
Block a user