feat(observatory): ship modular AI inference labs
This commit is contained in:
DCCONSTRUCTIONS
@@ -110,3 +110,48 @@ not a realtime inference or navigation claim.
|
||||
- The migration does not improve DDRNet quality, prove terrain traversability
|
||||
or grant navigation/actuation authority. Those remain separate model and
|
||||
safety acceptance questions.
|
||||
|
||||
## Resource ownership addendum — 2026-09-03
|
||||
|
||||
An active calculation/view may allocate what its admitted workload needs. On
|
||||
termination its ephemeral resources must become releasable; lowering evidence
|
||||
quality or deleting durable results is not a substitute for lifecycle cleanup.
|
||||
|
||||
Recorded viewers run the same unmodified upstream SDK in a disposable,
|
||||
same-origin iframe inside the existing viewer surface. A small synchronous
|
||||
facade exposes only the public methods already used by the product adapter.
|
||||
The revision-2 boundary returns primitive JSON strings, including events and
|
||||
errors; the parent parses them in its own realm. Native handles, DOM nodes,
|
||||
channels, unsubscribe closures and SDK startup Promises stay inside the iframe.
|
||||
Parent-owned RRD bytes enter a synchronous channel call, without transfer or
|
||||
changing the payload/quality. Parent startup Promises are settled by primitive
|
||||
notifications and rejected on disposal. Every channel is closed independently;
|
||||
a late SDK startup completion is stopped again. Teardown severs references and
|
||||
removes the iframe. There is still exactly one native renderer and clock,
|
||||
no new playback UI, SDK patch or second backend. The live source path stays
|
||||
direct upstream. Page-cache restoration starts a fresh viewer lifecycle.
|
||||
|
||||
The iframe relays only Escape and primitive pointer fields to parent-owned
|
||||
events for the existing outer controls. Coordinates are translated by the
|
||||
iframe bounds, including the native-chrome crop. Native Rerun still owns its
|
||||
canvas gestures and divider; nothing is cancelled or sent back to the canvas.
|
||||
The parent tracks the same divider for control alignment, not a second layout
|
||||
or renderer. Teardown removes every relay listener and foreign frame reference.
|
||||
|
||||
The backend's ephemeral blueprint store has an exact viewport owner, explicit
|
||||
release, idle expiry (300 seconds; reaped every 30 seconds), and shutdown cleanup.
|
||||
An open owner renews every 30 seconds. Release is serialized with rendering;
|
||||
late updates for that released identity are rejected. Expiry never stops an
|
||||
active render, a recording, a Worker profile or a vehicle. Durable files are not
|
||||
removed. The pre-existing 32-entry cache bound is unchanged; no new active data
|
||||
or image-quality cap is introduced.
|
||||
|
||||
DOM disappearance is not memory acceptance. Compare process footprints before,
|
||||
during and after ordinary sequential open/close cycles without forced GC or
|
||||
page reload. See the dated memory-lifecycle report for measurements and limits.
|
||||
Revision 2 and the input relay passed full typecheck, 740 frontend tests,
|
||||
64 focused backend tests and a production build, served on canonical 8000.
|
||||
Real browser acceptance of this revision remains pending: auto-review rejected
|
||||
opening the heavy saved viewer under elevated pressure; explicit owner approval
|
||||
was requested. This does not gate unrelated input/catalog contract work, nor
|
||||
does a successful build prove that the recorded memory retention is fixed.
|
||||
|
||||
@@ -1,5 +1,10 @@
|
||||
# ADR 0050 — Recorded Observatory first; portable profiles then onboard
|
||||
|
||||
> 2026-09-03: [ADR 0051](0051-modular-observatory-profiles.md) now defines
|
||||
> modular Docker composition and grouped single-provider settings. Its UI and
|
||||
> packaging rules supersede the finite whole-profile selector/one-image target
|
||||
> below. Recorded-first, exact cache, preserved evidence and safety remain.
|
||||
|
||||
Date: 2026-09-02. Decision accepted by the owner; implementation in progress.
|
||||
This supersedes ADR 0049's realtime-first product gate and execution order,
|
||||
not its live-stream integrity, ownership or safety contracts.
|
||||
@@ -182,6 +187,29 @@ See [real-run and restart evidence](../../experiments/perception/OBSERVATORY_REC
|
||||
|
||||
## Boundaries
|
||||
|
||||
Owner clarification, 2026-09-03: Observatory's result list includes only admitted
|
||||
portable-profile publications, not every historical LAB linked to a source.
|
||||
Legacy contours, canonical archive projectors and stored evidence remain
|
||||
unchanged. Filter by executable replay/publication binding, never by LAB label,
|
||||
date, model name or equality to the currently installed definition. Older valid
|
||||
portable versions remain reviewable. Versioned opt-in cursor pages feed the
|
||||
existing source search; legacy catalog consumers retain their previous response.
|
||||
See [catalog boundary evidence](../../experiments/perception/OBSERVATORY_CATALOG_BOUNDARY_2026-09-03.md).
|
||||
|
||||
Source-reuse increment (2026-09-03): both current control agents now share a
|
||||
Worker-local source CAS. Every generation still obtains its own authenticated,
|
||||
claim-bound manifest and checks exact digest/length before reusing bytes. Missing
|
||||
camera members are fetched individually instead of retransferring a partly cached
|
||||
epoch. Job/output/profile identities and model packages are unchanged. Cached
|
||||
visual results remain served by Core, not this Worker input cache.
|
||||
|
||||
M49 now selects an exact current-producer LiDAR v2 pack before raw decoding;
|
||||
the legacy producer file and its identity hash remain unchanged. Full source
|
||||
hash and strict NPZ integrity validation still occur on reuse. Cold whole-source
|
||||
materialization, versioned recorded-analysis and full workflow acceptance remain
|
||||
open. Shared-cache synthetic Worker proof is not a model run or a throughput
|
||||
measurement. See [source-reuse evidence](../../experiments/perception/OBSERVATORY_SOURCE_REUSE_2026-09-03.md).
|
||||
|
||||
No Synology deployment, hardware actuation, motor integration, new capture,
|
||||
silent model substitutions or deletion of recordings/results. A Docker image's
|
||||
portability does not promise that a CUDA build runs unchanged on Apple Silicon.
|
||||
|
||||
@@ -0,0 +1,181 @@
|
||||
# ADR 0051 — Modular AI containers and composed Observatory profiles
|
||||
|
||||
Date: 2026-09-03. Owner-approved target; the first local modular vertical slice is installed.
|
||||
|
||||
## Decision and supersession
|
||||
|
||||
A reusable module is an independently packaged Docker image for a model or a
|
||||
cohesive algorithmic function. A profile is an immutable, validated composition
|
||||
of module versions, parameters, input/output connections and execution policy.
|
||||
A LAB run applies that composition to an exact recording snapshot. A new
|
||||
combination does not require building another image containing all models.
|
||||
|
||||
This replaces the earlier **one full profile = one image/container** packaging
|
||||
requirement in the historical plan. It keeps ADR 0050's recorded-first purpose,
|
||||
ADR 0048's generic installed-package boundary and the stream contracts from
|
||||
ADR 0049. Several containers are not intrinsically a defect. Per-LAB application
|
||||
forks, hidden host dependencies and duplicated data preparation remain defects.
|
||||
|
||||
The future onboard host is CUDA-capable, architecture/runtime compatible
|
||||
hardware. It is explicitly **not Mac Mini**. Exact CPU architecture, GPU,
|
||||
driver and inference-runtime compatibility still need target qualification;
|
||||
CUDA availability alone is not a promise that every existing image will run.
|
||||
|
||||
New owned module images/containers use the `ndc-` namespace, readable model or
|
||||
function names, explicit versions and ownership labels. A composition has its
|
||||
own readable LAB label; module/composition/job identities are linked in runtime
|
||||
metadata. A reused module need not be renamed to match every LAB that uses it.
|
||||
|
||||
## Operator composition
|
||||
|
||||
Each recording exposes AI-layer settings inside the existing Observatory.
|
||||
Use functional groups with a single-choice Select, not a checklist of arbitrary
|
||||
containers. Segmentation admits **at most one** provider: DDRNet or EoMT, never
|
||||
both in the same new LAB. Optional groups may offer None if the resulting graph
|
||||
still satisfies its declared outputs. A geometry-only or segmentation-only LAB
|
||||
is valid and must not claim the omitted capabilities.
|
||||
|
||||
Candidate groups are segmentation, object detection, LiDAR/TGS geometry and
|
||||
costmap, object/range association, motion and policy-shadow. Their final physical
|
||||
module boundaries follow existing code ownership and measured data exchange,
|
||||
not a mandatory container per mathematical operation. Multiple capabilities
|
||||
provided by one module do not instantiate that module multiple times.
|
||||
|
||||
The server validates provider cardinality, dependencies, source capabilities,
|
||||
calibration/time frames, output contracts and installed Worker versions. A client
|
||||
cannot supply executable argv, image names, host paths or arbitrary resource
|
||||
settings. Invalid dependencies are explained, never silently supplemented by
|
||||
old LAB overlays or an unselected second segmenter.
|
||||
|
||||
An exact existing published composition is opened, not recomputed. The settings
|
||||
remain editable so the operator can create another composition. There is no
|
||||
Calculate action for a complete exact cache hit, no “calculation complete”
|
||||
button/badge, and no duplicate LAB. The grouped selectors keep already-used
|
||||
providers: completion belongs to the whole composition, not to an individual
|
||||
dropdown option. This supersedes the old finite-profile selector grammar, not
|
||||
its exact-cache and idempotency guarantees.
|
||||
|
||||
## Execution, identity and caches
|
||||
|
||||
- Keep one recorded owner on Worker006 and serialize heavy GPU execution.
|
||||
Selecting several modules does not authorize concurrent GPU model jobs.
|
||||
- Share source delivery/preparation locally on Worker. Intermediate image/point
|
||||
buffers do not round-trip through Core between modules. Do not reintroduce
|
||||
whole-recording upload/decode as a permanent streaming-start prerequisite.
|
||||
- Keep a common timestamped, bounded I/O contract with explicit sensor gaps,
|
||||
coordinate frames, unavailable outputs and causal temporal state. Recorded
|
||||
analysis may be slower than acquisition; live has separate freshness gates.
|
||||
- Freeze graph topology, module image/weights/code/config identities, input
|
||||
identities, preprocessing, calibration, cadence, precision and state policy.
|
||||
Canonical ordering of the selection UI is not part of the semantic identity.
|
||||
- Reuse a node only for exact validated inputs and producer identity. A changed
|
||||
detector invalidates its dependent range/motion results, not an independent
|
||||
segmentation result. Stateful reuse also binds initialization and history.
|
||||
- Final LAB/cache publication remains on Core. Worker source/preparation caches
|
||||
are not the final viewer store. Failed publication retries the sealed result,
|
||||
not inference. A partially cached graph is not a complete ready LAB.
|
||||
- Record reused versus computed nodes and provenance. Cache playback or cached
|
||||
composition assembly is never reported as measured onboard processing FPS.
|
||||
|
||||
Onboard modules are long-lived within a mission; do not start a container or
|
||||
reload weights per frame. Resident memory, local transport, bounded queues and
|
||||
end-to-end latency require a joint benchmark. An optional later co-location of
|
||||
hot nodes is a measured deployment optimization, not a reason to remove logical
|
||||
modularity. No automatic actuation or navigation acceptance is introduced.
|
||||
|
||||
## Current implementation and migration
|
||||
|
||||
Current M49 runs CPU TGS in its specialized agent. Current LAB V1 is an installed
|
||||
fixed stack: prepare → EoMT → DDRNet → assemble, with host-bound model/runtime
|
||||
assets. Neither is the new modular composition system. Keep the successful M49
|
||||
publication and exact existing releases; retain old dual-segmentation results
|
||||
as historical comparisons, without admitting that combination in new profiles.
|
||||
|
||||
Reuse `InstalledLabPackage`, its generic launcher, queue/claim/recovery, source
|
||||
CAS, verified publication and common viewer. Extend the versioned contracts for
|
||||
compositions and per-node reuse. Do not redesign the queue or create another
|
||||
agent per model. Source CAS alone does not implement intermediate-result reuse.
|
||||
|
||||
The owner authorized Worker cleanup. Classify every exact container/image/cache
|
||||
against running services, pinned packages, model assets, rollback and retained
|
||||
evidence before deleting. Stopped instances can be retired independently of
|
||||
images and volumes. Preserve raw recordings, published LABs, Legacy views,
|
||||
required source/model caches and one usable rollback; no global Docker prune.
|
||||
|
||||
The accepted EoMT disk floor is now **250 GiB plus its working-set estimate**.
|
||||
Source and narrow tests have changed; installed EoMT still has 350 GiB until a
|
||||
new sealed image/release is activated. Never patch a historical digest in place.
|
||||
|
||||
## Acceptance and route
|
||||
|
||||
Done means: select one provider per group; compute two distinct compositions on
|
||||
compatible recordings with exact cache reuse; view their complete bound outputs
|
||||
from Core; recover publication without compute; prove exclusive GPU scheduling
|
||||
and cleanup; cold-start the selected module distribution without developer
|
||||
checkout or unlisted caches. Onboard qualification remains future work.
|
||||
|
||||
The historical combined route remains documented in
|
||||
[the four-stage ExecPlan](../OBSERVATORY_REALTIME_PROFILES_EXECPLAN.md).
|
||||
|
||||
## Local implementation record — 2026-09-03
|
||||
|
||||
The first implemented slice replaces the Observatory's fixed profile selector
|
||||
and separate refresh/calculate controls with one always-available
|
||||
`Сконфигурировать AI-слой` action for the selected source. Its canonical Window
|
||||
contains functional groups, one provider Select per group, the close action and
|
||||
one `Рассчитать` action. The Window uses background, spacing and typography for
|
||||
grouping; it has no group outlines, header divider lines or modal rim.
|
||||
|
||||
DDRNet and EoMT are independent alternatives in the Segmentation group. They
|
||||
share only the model-neutral `camera-source` preparation step and neither model
|
||||
consumes the other model's result. The installed module repositories and exact
|
||||
local image identities are:
|
||||
|
||||
- `ndc/mission-core-ai-module-camera-source` —
|
||||
`da926459aee0a841bbdfaf80a0eb5fbead354c56794d1f3384eeba66d0a49e00`;
|
||||
- `ndc/mission-core-ai-module-ddrnet` —
|
||||
`489fc7d1157fd0f1cd1d82e06a15737b7b2aaaf72b2ddb2aca2992b91a97488e`;
|
||||
- `ndc/mission-core-ai-module-eomt` —
|
||||
`5b770178e4a5c8fbe8f8ddab3b83a598973dbe103b669851110b11a0918ab846`;
|
||||
- `ndc/mission-core-ai-module-rf-detr` —
|
||||
`2b8b44be8e9ee4060aa6997fc4c025ad7120f37ecd720a9e59b5e02ac6c90f66`;
|
||||
- `ndc/mission-core-ai-module-object-distance` —
|
||||
`69d68f64981b41e5bcce1e642433e3d466b88c51942a5ba3aec9180d8ed04263`;
|
||||
- `ndc/mission-core-ai-module-tgs` —
|
||||
`f9278ab21aa65045be993dd19bffc25f49955e19598893ac78cc4761ca63ecf3`.
|
||||
|
||||
The executable graph also contains `ai-detection-rf-detr-v1` for image-space
|
||||
boxes and `ai-range-object-distance-v1` for box-to-LiDAR association. The range
|
||||
package executes RF-DETR as its explicit dependency, then uses synchronized
|
||||
LiDAR, pose and calibration to estimate range. TGS consumes LiDAR, pose and its
|
||||
own history independently. Selecting TGS and range therefore produces two
|
||||
independent durable jobs; segmentation does not feed either of them. The common
|
||||
Worker agent launches package steps with job-specific container names under
|
||||
`ndc-mission-core-ai-module-*`, serializes GPU work, publishes a
|
||||
`missioncore.recorded-ai-layer-review/v1` result and leaves the historical
|
||||
combined LAB V1 and Legacy contour intact.
|
||||
|
||||
Submission creates a durable Observatory job immediately. The source-bound
|
||||
evidence area shows its selected configuration, current phase, measured
|
||||
progress and an estimated remaining time. A published result moves into the
|
||||
existing evidence list and uses the same open, rename and delete actions as
|
||||
the other portable LAB results.
|
||||
|
||||
The active Worker 006 directory release is `20260903-v5`, release SHA-256
|
||||
`a87c82adfa1058f14f850f488ace38e5c7c9d9e2145d4678a81b52f2a1aad573`.
|
||||
Its common agent image SHA-256 is
|
||||
`d6f2ef1a3f38ebc8503b62296ca9def10472a60a41ffc5edd123df565520dec8`.
|
||||
The DDRNet, EoMT, RF-DETR and object-distance definition SHA-256 values are
|
||||
respectively
|
||||
`b04206a8472588fee22e0282228e51b7817c80f9739180e834b704733d7aaf76`,
|
||||
`6d13402883e8fa79a8ea97b15e81e704d3c0f9dac5d94445e5e53c18dcebe98f`,
|
||||
`a89f51a66a070deb50f4904f596545cdcd7289df7e42b8511aec9547dedd52e8`
|
||||
and `cc56e72fdfb38565a0b402c90a91917821463d980f5855e4cdf4d8b9f262a94c`.
|
||||
The standalone detector and composite range package have distinct resource
|
||||
profile identities, so their otherwise shared RF-DETR model manifest cannot
|
||||
collapse into one executor registration. All four package executor identities
|
||||
are unique. DDRNet, EoMT, RF-DETR and object-distance pass live source,
|
||||
executor, sealed-definition and durable-queue preflight for
|
||||
`20260828T130511Z_viewer_live`; the published M49 TGS result is an exact cache
|
||||
hit. No new full-recording RF-DETR or object-distance inference was run as part
|
||||
of this installation acceptance.
|
||||
Reference in New Issue
Block a user