feat(observatory): add durable recorded compute queue
This commit is contained in:
DCCONSTRUCTIONS
@@ -0,0 +1,138 @@
|
||||
# ADR 0046: Durable Observatory recorded queue and live K1 priority
|
||||
|
||||
Date: 2026-08-31
|
||||
Status: accepted; durable queue core implemented; Worker and live-trigger wiring pending
|
||||
|
||||
## Context
|
||||
|
||||
Observatory must let an operator apply an admitted laboratory setup to a saved
|
||||
session without turning the browser into an execution client. The current compute
|
||||
host is a shared, finite Worker rather than an elastic pool. Recorded experiments
|
||||
may wait, but a future live K1 perception process must acquire that compute
|
||||
immediately and exclusively when its real acquisition lifecycle begins.
|
||||
|
||||
Historical LAB results are not automatically executable definitions. In
|
||||
particular, the existing LAB V1 result proves a recorded EoMT and DDRNet analysis,
|
||||
but it does not retain one independently versioned, durable executor contract that
|
||||
can safely be reconstructed from its display metadata. Conversely, M4.9T5 has an
|
||||
exact accepted RAVNOVES00 source pack and CPU-only TRAVEL TGS release which can be
|
||||
sealed without inventing learned-model dependencies.
|
||||
|
||||
The queue must also survive process restarts and ambiguity around preemption. A
|
||||
database flag alone cannot prove that a non-checkpointable Worker process released
|
||||
CPU, memory and staging storage. Starting live K1 work before physical cancellation
|
||||
is confirmed would permit two owners of the same constrained resource.
|
||||
|
||||
## Decision
|
||||
|
||||
Mission Core owns a bounded SQLite-backed recorded-job queue. The browser submits
|
||||
only:
|
||||
|
||||
- `source_session_id`;
|
||||
- `setup_id`;
|
||||
- one idempotency key.
|
||||
|
||||
The request contains no command, script, path, environment variable, image,
|
||||
model, resource limit or priority. The server resolves an allowlisted source/setup
|
||||
pair and seals the following identities into the accepted job:
|
||||
|
||||
1. the current source-catalog snapshot SHA-256 captured at admission;
|
||||
2. immutable source-bundle SHA-256;
|
||||
3. source-capability-manifest SHA-256;
|
||||
4. source-adapter id, version and SHA-256;
|
||||
5. setup RunDefinition id, version and SHA-256;
|
||||
6. executor release and container-image SHA-256;
|
||||
7. learned-model release list and model-manifest SHA-256;
|
||||
8. resource-profile SHA-256 and checkpoint policy.
|
||||
|
||||
The queue owns idempotency. An exact retry returns the original job and receipt.
|
||||
The same key with changed identity is a conflict. Jobs advance through `accepted`,
|
||||
`queued`, `claimed` and `running`, then terminate as `succeeded` or `failed`.
|
||||
`paused`, `preemption-pending` and `reconciliation-required` expose safety-relevant
|
||||
branches. Mission Core admits only one active recorded compute owner.
|
||||
|
||||
The queue is an orchestration and evidence boundary, not a Worker transport. A
|
||||
queued job proves durable admission, not remote execution or eventual success.
|
||||
Worker claims, terminal result publication and telemetry must later use a separate
|
||||
authenticated, versioned service boundary; no arbitrary PowerShell or filesystem
|
||||
payload is added to this contract.
|
||||
|
||||
## Current setup admission
|
||||
|
||||
The current executable matrix contains one exact pair:
|
||||
|
||||
- source: `RAVNOVES00`;
|
||||
- setup: `M4.9T5 · TRAVEL TGS · CPU-only, без ML`;
|
||||
- learned-model releases: `[]`;
|
||||
- checkpoint policy: `non-checkpointable`.
|
||||
|
||||
The empty model list is affirmative provenance: this TGS configuration is a
|
||||
CPU-only algorithmic pipeline. It is not a placeholder for unidentified weights.
|
||||
Admission still reads and seals the current source-catalog snapshot and verifies
|
||||
the pinned source bundle and capability manifest before the job is accepted. The
|
||||
binding deliberately does not pin a digest of the whole mutable catalog: the
|
||||
exact compute input is the immutable source pack, while the catalog snapshot is
|
||||
job-specific provenance.
|
||||
|
||||
`LAB V1 · EoMT Cityscapes Large 1024 + DDRNet-39` remains result-only. The
|
||||
published evidence stays viewable, but the setup cannot be queued until a new
|
||||
independently versioned source adapter, RunDefinition and executable release are
|
||||
created and accepted. The same fail-closed rule applies to every other
|
||||
session/setup combination. The product may grow toward any admitted setup over
|
||||
any compatible recorded source, but compatibility is evidence, not an assumption.
|
||||
|
||||
## Priority and preemption protocol
|
||||
|
||||
Priority is server-owned:
|
||||
|
||||
- live K1 lease: rank `0`;
|
||||
- recorded replay job: rank `100`.
|
||||
|
||||
A pending or active live lease blocks new recorded claims. A running cooperative
|
||||
job yields at an allowlisted checkpoint and moves to `paused` before live
|
||||
activation. A non-checkpointable job cannot pretend to pause. Its transition is:
|
||||
|
||||
1. persist a stable cancellation intent bound to the job claim, executor release,
|
||||
image, resource profile and live trigger;
|
||||
2. invoke the scheduler-owned physical cancellation boundary;
|
||||
3. persist an identity-bound cancellation receipt proving resource release,
|
||||
staging discard and restart-from-zero;
|
||||
4. only then activate the live K1 lease.
|
||||
|
||||
The cancellation identity is reused after a crash or retry. A missing callback or
|
||||
receipt leaves the durable intent pending for retry; a conflicting or
|
||||
indeterminate receipt moves the record to reconciliation. Both forms block live
|
||||
activation instead of automatically launching duplicate work. When an
|
||||
explicit live terminal trigger completes, fails or cancels the lease, paused
|
||||
recorded jobs return to the queue. A non-checkpointable job starts again from
|
||||
zero; partial staging never becomes evidence.
|
||||
|
||||
## Acceptance boundary
|
||||
|
||||
This decision accepts the durable queue core and its identity, lifecycle,
|
||||
idempotency and preemption contracts. It does not claim that the complete compute
|
||||
loop is deployed. The following remain implementation gates:
|
||||
|
||||
- authenticated Worker claim and result transport;
|
||||
- installation of the exact M4.9T5 executor on the Worker;
|
||||
- binding real K1 acquisition start/terminal events to live-lease triggers;
|
||||
- physical acceptance that cancellation releases the required Worker resources
|
||||
and that recorded work resumes only after the live lease terminates.
|
||||
|
||||
Until those gates close, an M4.9T5 submission may remain honestly `queued`. LAB V1
|
||||
cannot be submitted, and no UI state may present it as executable.
|
||||
|
||||
## Consequences
|
||||
|
||||
- The UI stays a bounded selector and submitter; executable authority remains on
|
||||
the server.
|
||||
- Every accepted job is reproducible from sealed source, setup, executor, model
|
||||
and resource identities rather than mutable host paths.
|
||||
- Live K1 can receive hard priority without silently losing or concurrently
|
||||
running recorded work.
|
||||
- Non-checkpointable replay pays the deliberate cost of discard and
|
||||
restart-from-zero after preemption.
|
||||
- K1 acquisition/control principles, the legacy LAB archive and
|
||||
Simulation/Gaussian remain unchanged.
|
||||
- Adding a setup or source requires an admitted adapter and RunDefinition; a
|
||||
display name or historical result is insufficient.
|
||||
Reference in New Issue
Block a user