feat(observatory): add durable recorded compute queue
This commit is contained in:
DCCONSTRUCTIONS
@@ -275,33 +275,64 @@ execution canaries. Once they prove one automatic run-to-evidence path, the LAB
|
||||
canonicalization slice closes and remaining historical work stays `legacy` unless
|
||||
an integrity or product need justifies a targeted migration.
|
||||
|
||||
## Observatory preparation and Worker dispatch
|
||||
## Observatory durable recorded queue and Worker dispatch
|
||||
|
||||
Selecting a source and a laboratory setup in Observatory is not itself a run.
|
||||
Before a remote executor is installed, Mission Core may persist only an immutable
|
||||
`missioncore.observatory-run-preparation/v1` receipt. That receipt binds the exact
|
||||
source-catalog snapshot, setup id, RunDefinition version and configuration digest
|
||||
to one idempotency key. Its only admitted terminal state is `blocked`; it has no
|
||||
`run_id`, dispatch receipt or execution authority.
|
||||
The public submission contains only `source_session_id`, `setup_id` and an
|
||||
idempotency key. It cannot supply commands, executable text, filesystem paths,
|
||||
container images, model identities, resource limits or priority. The server
|
||||
resolves the allowlisted pair and seals all executable identity into one durable
|
||||
record:
|
||||
|
||||
A real Worker run begins only after a separate durable dispatcher has atomically
|
||||
published an immutable dispatch receipt. The executor must be an additive,
|
||||
allowlisted service owned outside the K1 acquisition path. It may accept bounded
|
||||
versioned data, but never an arbitrary command, filesystem path or PowerShell
|
||||
fragment from the UI. Exact retries return the original receipt; a reused key with
|
||||
different identity fails closed. Transport uncertainty produces an explicit
|
||||
reconciliation state and never an automatic duplicate run.
|
||||
- the current source-catalog snapshot captured at admission, plus immutable source
|
||||
bundle and source-capability-manifest SHA-256 identities;
|
||||
- source-adapter id, version and SHA-256;
|
||||
- setup RunDefinition id, version and SHA-256;
|
||||
- executor release and image SHA-256 identities;
|
||||
- learned-model manifest and resource-profile identities;
|
||||
- checkpoint policy and the server-owned priority class.
|
||||
|
||||
The first executable Observatory setup remains the exact recorded
|
||||
RAVNOVES00/M4.9T5 definition. Its full and integrated successors are different
|
||||
definitions and results. The current RAVNOVES004TREE LAB V1 projection remains
|
||||
replay-only until a new, independently versioned RunDefinition is reconstructed;
|
||||
Mission Core does not invent a configuration for an already published result.
|
||||
An exact retry returns the existing job. Reusing an idempotency key for another
|
||||
identity fails closed. The ordinary lifecycle advances through `accepted`,
|
||||
`queued`, `claimed` and `running`, then terminates as `succeeded` or `failed`.
|
||||
`paused`, `preemption-pending` and `reconciliation-required` are explicit safety
|
||||
branches rather than hidden retries. The durable queue serializes one recorded
|
||||
compute owner and does not equate a queued receipt with a Worker execution
|
||||
receipt.
|
||||
|
||||
Worker telemetry is secondary observation evidence. It does not replace the
|
||||
authoritative run ledger, dispatch receipt, result validation or common laboratory
|
||||
receipt. K1 control, Simulation/Gaussian runtimes and legacy LAB projections are
|
||||
outside this executor boundary and are not restarted or migrated by it.
|
||||
The only currently queueable pair is the exact `RAVNOVES00` source and
|
||||
`M4.9T5 · TRAVEL TGS · CPU-only, без ML` RunDefinition. It seals an explicitly
|
||||
empty learned-model list because TGS is an algorithmic CPU pipeline, not an
|
||||
unknown model dependency. The binding pins the exact source pack rather than a
|
||||
volatile whole-catalog digest; the server captures and seals the current catalog
|
||||
snapshot into each admitted job. `LAB V1 · EoMT Cityscapes Large 1024 + DDRNet-39`
|
||||
remains a result-only replay: no independently versioned durable executor exists
|
||||
for that published result, so Mission Core does not invent one. Every other
|
||||
session/setup pair fails closed until its own source adapter, capability manifest,
|
||||
RunDefinition and executor identities are admitted.
|
||||
|
||||
Recorded work has priority rank `100`. A future live K1 lease has rank `0` and
|
||||
closes new recorded claims while it is pending or active. Cooperative executors
|
||||
yield only at an allowlisted checkpoint. A non-checkpointable monolith must be
|
||||
cancelled by the scheduler, discard all staging output and later restart from
|
||||
zero. Cancellation uses a crash-safe two-phase protocol: Mission Core first
|
||||
persists one stable cancellation intent and activates live work only after an
|
||||
identity-bound receipt proves resource release and staging discard. A retry uses
|
||||
the same cancellation identity. A missing callback or receipt remains durably
|
||||
pending for retry; a conflicting or indeterminate receipt enters reconciliation.
|
||||
Every unresolved form blocks live activation and concurrent ownership.
|
||||
An explicit live terminal trigger requeues paused recorded work.
|
||||
|
||||
ADR 0046 records this decision. The durable queue and identity contracts are
|
||||
implemented; Worker claim transport, installation of the exact M4.9T5 executor
|
||||
and wiring from the real K1 lifecycle to live-lease triggers remain pending.
|
||||
Therefore a submitted M4.9T5 job may honestly wait in `queued` without implying
|
||||
that Worker 006 can execute it yet.
|
||||
|
||||
Worker telemetry remains secondary observation evidence. It does not replace the
|
||||
authoritative queue ledger, result validation or common laboratory receipt. K1
|
||||
acquisition and control, the legacy LAB archive, and Simulation/Gaussian runtimes
|
||||
are outside this boundary and are not restarted, migrated or modified by it.
|
||||
|
||||
## Planning discipline
|
||||
|
||||
|
||||
@@ -113,11 +113,11 @@ The next bounded slice adds an explicit laboratory Setup selection beside the so
|
||||
Setup is a versioned, immutable compatibility contract, not a mutable bag of UI parameters. The
|
||||
catalog keeps two independently named historical facts:
|
||||
|
||||
- `M4.9T5 · TRAVEL TGS · полный source-paced shadow` is an archived RunDefinition bound exactly to
|
||||
- `M4.9T5 · TRAVEL TGS · CPU-only, без ML` is an archived RunDefinition bound exactly to
|
||||
`20260720T065719Z_viewer_live`. Its configuration references are content-addressed, its primary
|
||||
visual result remains in legacy LAB, and the integrated-graph result is retained as a compute
|
||||
successor rather than presented as a second viewer.
|
||||
- `LAB V1 · финальный записанный контур восприятия` is the current exact RAV004 result bound to
|
||||
- `LAB V1 · EoMT Cityscapes Large 1024 + DDRNet-39` is the current exact RAV004 result bound to
|
||||
`20260828T130511Z_viewer_live`. It predates the product RunDefinition contract and therefore keeps
|
||||
`run_definition = null`; the UI identifies it as an existing result and never fabricates a config
|
||||
digest for it.
|
||||
|
||||
@@ -0,0 +1,138 @@
|
||||
# ADR 0046: Durable Observatory recorded queue and live K1 priority
|
||||
|
||||
Date: 2026-08-31
|
||||
Status: accepted; durable queue core implemented; Worker and live-trigger wiring pending
|
||||
|
||||
## Context
|
||||
|
||||
Observatory must let an operator apply an admitted laboratory setup to a saved
|
||||
session without turning the browser into an execution client. The current compute
|
||||
host is a shared, finite Worker rather than an elastic pool. Recorded experiments
|
||||
may wait, but a future live K1 perception process must acquire that compute
|
||||
immediately and exclusively when its real acquisition lifecycle begins.
|
||||
|
||||
Historical LAB results are not automatically executable definitions. In
|
||||
particular, the existing LAB V1 result proves a recorded EoMT and DDRNet analysis,
|
||||
but it does not retain one independently versioned, durable executor contract that
|
||||
can safely be reconstructed from its display metadata. Conversely, M4.9T5 has an
|
||||
exact accepted RAVNOVES00 source pack and CPU-only TRAVEL TGS release which can be
|
||||
sealed without inventing learned-model dependencies.
|
||||
|
||||
The queue must also survive process restarts and ambiguity around preemption. A
|
||||
database flag alone cannot prove that a non-checkpointable Worker process released
|
||||
CPU, memory and staging storage. Starting live K1 work before physical cancellation
|
||||
is confirmed would permit two owners of the same constrained resource.
|
||||
|
||||
## Decision
|
||||
|
||||
Mission Core owns a bounded SQLite-backed recorded-job queue. The browser submits
|
||||
only:
|
||||
|
||||
- `source_session_id`;
|
||||
- `setup_id`;
|
||||
- one idempotency key.
|
||||
|
||||
The request contains no command, script, path, environment variable, image,
|
||||
model, resource limit or priority. The server resolves an allowlisted source/setup
|
||||
pair and seals the following identities into the accepted job:
|
||||
|
||||
1. the current source-catalog snapshot SHA-256 captured at admission;
|
||||
2. immutable source-bundle SHA-256;
|
||||
3. source-capability-manifest SHA-256;
|
||||
4. source-adapter id, version and SHA-256;
|
||||
5. setup RunDefinition id, version and SHA-256;
|
||||
6. executor release and container-image SHA-256;
|
||||
7. learned-model release list and model-manifest SHA-256;
|
||||
8. resource-profile SHA-256 and checkpoint policy.
|
||||
|
||||
The queue owns idempotency. An exact retry returns the original job and receipt.
|
||||
The same key with changed identity is a conflict. Jobs advance through `accepted`,
|
||||
`queued`, `claimed` and `running`, then terminate as `succeeded` or `failed`.
|
||||
`paused`, `preemption-pending` and `reconciliation-required` expose safety-relevant
|
||||
branches. Mission Core admits only one active recorded compute owner.
|
||||
|
||||
The queue is an orchestration and evidence boundary, not a Worker transport. A
|
||||
queued job proves durable admission, not remote execution or eventual success.
|
||||
Worker claims, terminal result publication and telemetry must later use a separate
|
||||
authenticated, versioned service boundary; no arbitrary PowerShell or filesystem
|
||||
payload is added to this contract.
|
||||
|
||||
## Current setup admission
|
||||
|
||||
The current executable matrix contains one exact pair:
|
||||
|
||||
- source: `RAVNOVES00`;
|
||||
- setup: `M4.9T5 · TRAVEL TGS · CPU-only, без ML`;
|
||||
- learned-model releases: `[]`;
|
||||
- checkpoint policy: `non-checkpointable`.
|
||||
|
||||
The empty model list is affirmative provenance: this TGS configuration is a
|
||||
CPU-only algorithmic pipeline. It is not a placeholder for unidentified weights.
|
||||
Admission still reads and seals the current source-catalog snapshot and verifies
|
||||
the pinned source bundle and capability manifest before the job is accepted. The
|
||||
binding deliberately does not pin a digest of the whole mutable catalog: the
|
||||
exact compute input is the immutable source pack, while the catalog snapshot is
|
||||
job-specific provenance.
|
||||
|
||||
`LAB V1 · EoMT Cityscapes Large 1024 + DDRNet-39` remains result-only. The
|
||||
published evidence stays viewable, but the setup cannot be queued until a new
|
||||
independently versioned source adapter, RunDefinition and executable release are
|
||||
created and accepted. The same fail-closed rule applies to every other
|
||||
session/setup combination. The product may grow toward any admitted setup over
|
||||
any compatible recorded source, but compatibility is evidence, not an assumption.
|
||||
|
||||
## Priority and preemption protocol
|
||||
|
||||
Priority is server-owned:
|
||||
|
||||
- live K1 lease: rank `0`;
|
||||
- recorded replay job: rank `100`.
|
||||
|
||||
A pending or active live lease blocks new recorded claims. A running cooperative
|
||||
job yields at an allowlisted checkpoint and moves to `paused` before live
|
||||
activation. A non-checkpointable job cannot pretend to pause. Its transition is:
|
||||
|
||||
1. persist a stable cancellation intent bound to the job claim, executor release,
|
||||
image, resource profile and live trigger;
|
||||
2. invoke the scheduler-owned physical cancellation boundary;
|
||||
3. persist an identity-bound cancellation receipt proving resource release,
|
||||
staging discard and restart-from-zero;
|
||||
4. only then activate the live K1 lease.
|
||||
|
||||
The cancellation identity is reused after a crash or retry. A missing callback or
|
||||
receipt leaves the durable intent pending for retry; a conflicting or
|
||||
indeterminate receipt moves the record to reconciliation. Both forms block live
|
||||
activation instead of automatically launching duplicate work. When an
|
||||
explicit live terminal trigger completes, fails or cancels the lease, paused
|
||||
recorded jobs return to the queue. A non-checkpointable job starts again from
|
||||
zero; partial staging never becomes evidence.
|
||||
|
||||
## Acceptance boundary
|
||||
|
||||
This decision accepts the durable queue core and its identity, lifecycle,
|
||||
idempotency and preemption contracts. It does not claim that the complete compute
|
||||
loop is deployed. The following remain implementation gates:
|
||||
|
||||
- authenticated Worker claim and result transport;
|
||||
- installation of the exact M4.9T5 executor on the Worker;
|
||||
- binding real K1 acquisition start/terminal events to live-lease triggers;
|
||||
- physical acceptance that cancellation releases the required Worker resources
|
||||
and that recorded work resumes only after the live lease terminates.
|
||||
|
||||
Until those gates close, an M4.9T5 submission may remain honestly `queued`. LAB V1
|
||||
cannot be submitted, and no UI state may present it as executable.
|
||||
|
||||
## Consequences
|
||||
|
||||
- The UI stays a bounded selector and submitter; executable authority remains on
|
||||
the server.
|
||||
- Every accepted job is reproducible from sealed source, setup, executor, model
|
||||
and resource identities rather than mutable host paths.
|
||||
- Live K1 can receive hard priority without silently losing or concurrently
|
||||
running recorded work.
|
||||
- Non-checkpointable replay pays the deliberate cost of discard and
|
||||
restart-from-zero after preemption.
|
||||
- K1 acquisition/control principles, the legacy LAB archive and
|
||||
Simulation/Gaussian remain unchanged.
|
||||
- Adding a setup or source requires an admitted adapter and RunDefinition; a
|
||||
display name or historical result is insufficient.
|
||||
Reference in New Issue
Block a user