NODEDC_MISSION_CORE/docs/adr/0031-local-compute-contour-telemetry-plane.md

ADR 0031 — Local compute contour telemetry plane

Status: accepted for implementation Date: 2026-07-27

Implementation status

The first local contour is active:

• Worker 006 publishes through the native Windows service NDC Mission Core Telemetry Agent (Telegraf);
• MQTT, normalization and storage run as the single Compose project ndc-mission-core-telemetry;
• the normalizer exposes the read-only product adapter only on 127.0.0.1:18030;
• Mission Core on canonical port 8000 uses this adapter for live System telemetry and reports the source as agent-mqtt;
• SSH is retained only for explicit profile diagnostics and bootstrap checks.
• the selected contour owns a managed network profile: Worker 006 keeps its node identity separate from its SSH address and from the broker endpoint; every saved endpoint and private Docker bind address is probed before an explicit transition;
• the existing contour settings can probe DNS/TCP from both Mac and Worker, apply the listener address to the local broker, and apply the endpoint to the Telegraf Windows service without exposing or replacing its scoped credential.

The existing Worker 006 containers were renamed in place to ndc-mission-core-triton and ndc-mission-core-perception-worker. Their container identities were preserved, so this namespace migration did not restart the inference or perception runtimes.

The compute-side native pipeline contract is implemented and deployed on Worker 006. It emits missioncore.agent-pipeline-telemetry/v1 lifecycle documents through an injected transport, and the telemetry normalizer preserves source, method, stage and stage-state identities in the normalized series key. The E41 runner first exercised the JSONL evidence sink against the immutable predictor package.

The durable path was then accepted on 2026-07-29 with real persistent-worker request durable-native-acceptance-20260729-115418. The request reached the actual live-source boundary and terminated because no K1 shadow source was open. Its native run started and failed documents were appended by the worker, tailed by host Telegraf, published on the contour pipeline topic and returned by the normalized Timescale query API with the same run/request identity. This is transport and lifecycle acceptance, not a successful perception-result or stage-performance claim.

The acceptance also closed two deployment-unit defects found by the real request: the token-stdin launcher now installs its required MissionCoreDockerNames.ps1 companion, and the perception runner updater atomically digest-gates run_e15_shadow_inference.py, e15_shadow_runtime.py and pipeline_telemetry.py. The existing perception container is still the only inference worker; broker credentials remain on the host Telegraf service.

Decision

Mission Core treats a compute worker as a configurable local compute contour, not as one hard-coded workstation.

The portable telemetry path is:

Telegraf agent
    -> authenticated MQTT
    -> Eclipse Mosquitto
    -> Mission Core telemetry normalizer
    -> TimescaleDB OSS
    -> Mission Core System API
    -> product workspaces

The broker, normalizer and telemetry database form one Docker Compose contour named ndc-mission-core-telemetry. They remain separate containers so each process has an independent healthcheck, least-privilege boundary, lifecycle and rollback path. This is one operator-managed stack, not one multi-process container.

The Telegraf agent runs as a native host service. On Worker 006 this is required for portable access to Windows performance counters, Docker Desktop and NVIDIA telemetry. Putting it into the Linux Docker Desktop VM would make the agent less universal and would hide part of the host it is meant to observe.

The direct SSH/PowerShell probe remains a bounded bootstrap and diagnostic fallback for the existing Worker 006 profile. It is not the target live telemetry transport.

Mesh access points belonging to one LAN are not modeled as separate broker addresses. The broker endpoint may be a hostname only when the Worker can resolve it; Mission Core does not assume that macOS .local discovery is portable to Windows. A move to another LAN changes the reviewed endpoint and private bind address, not the agent identity or MQTT topic identity. Applying a saved profile is explicit, verifies DNS and TCP reachability, and rolls back the affected configuration when verification fails. MQTT without TLS cannot be configured for a public endpoint or wildcard listener.

2026-08-05 recovery correction

Worker 006 stopped publishing because its Telegraf MQTT output still referenced the operator Mac through a .local name that Windows no longer resolved. Telegraf exited on the output connection error, and the Windows service had no configured recovery actions. The reviewed compute-contour transition rebound the broker and worker profile to the currently verified private LAN endpoint without replacing the scoped MQTT credential; fresh agent, hardware and pipeline telemetry then returned.

Install, update and existing-agent network transitions now configure the same bounded Windows recovery policy: restart after 5 seconds, 30 seconds and 60 seconds, resetting the failure counter after one day and applying recovery to non-crash failures. This does not make an invalid endpoint valid; it keeps transient broker or network loss from silently leaving the telemetry agent stopped after connectivity returns.

Why these components

• Telegraf is the universal host agent. It already supports Windows and Linux host counters, Docker, NVIDIA SMI, network counters and custom HTTP/Prometheus inputs. Hardware discovery does not belong in a custom Mission Core agent.
• Eclipse Mosquitto is the local, fully open-source broker. It separates worker lifecycle and network location from the Mission Core UI.
• Mission Core telemetry normalizer owns only product semantics: contour identity, runtime ownership, LAB/run identity and processing-stage names. These concepts cannot be delegated to a generic hardware agent.
• TimescaleDB OSS stores normalized samples and permits comparison between laboratory runs without coupling agents to database credentials or schema.

Only the broker and normalizer know the MQTT credentials. Agents receive a scoped publisher identity. The UI never receives broker or database credentials.

Product surface brief

User job

An operator must be able to carry Mission Core and a worker to another local network, register one or more compute contours, install the standard telemetry agent, select the active contour and inspect its hardware, processing and network state.

Placement

• The Система left panel is a contour selector.
• A canonical plus action in its header creates another contour configuration.
• System workspace modes (Вычислительные модули, Интеграции, Сеть, Журнал и аудит) live in the content header because they describe the selected contour.
• A canonical settings utility action opens settings for the selected contour.
• Global Mission Core settings remain in the profile menu and do not mix with contour settings.

Rejected placements:

• keeping the single Worker 006 as a permanent page identity;
• placing contour settings in a global Система / Настройки workspace;
• creating one bespoke page layout per worker;
• placing installation instructions directly in the telemetry dashboard.

State grammar

• unconfigured: contour exists but no agent transport is configured;
• provisioning: bootstrap material was issued but the agent has not published a sample;
• online: a fresh normalized sample matches the expected node identity;
• stale: the last valid sample is older than the freshness window;
• identity-mismatch: an agent published with a different node identity;
• offline: the contour was configured but no current sample is available;
• legacy-diagnostic: the UI is temporarily backed by the direct SSH probe.

The UI must name the evidence source. It must not call an SSH snapshot “live agent telemetry”.

Evidence contract

Each normalized sample carries:

• contour_id, node_id, agent_id, observed_at_utc;
• source and normalizer schema versions;
• hardware, runtime, network and pipeline measurements;
• optional lab_id, run_id, request_id and frame index;
• ingestion timestamp and source topic.

Unknown measurements remain null; they are never synthesized for presentation.

Several Telegraf measurements can have the same name and timestamp but belong to different disks, interfaces, containers or GPU processes. Storage therefore keys a sample by a stable series_key derived from its distinguishing source tags. When a legacy Telegraf input emits one logical series as several field fragments, the normalizer merges those fields on conflict instead of replacing an earlier fragment. This preserves series identity without leaking source-specific tag grammar into the product API.

Deployment boundary

The first stack runs inside one trusted local network and binds only to explicitly configured LAN interfaces. Mosquitto uses password authentication and ACLs; anonymous access is forbidden. TimescaleDB is not exposed outside the compose network.

All NODE.DC-owned Docker containers, networks and named volumes use a lowercase ndc- prefix. The concrete telemetry objects are:

• ndc-mission-core-mqtt-broker;
• ndc-mission-core-telemetry-normalizer;
• ndc-mission-core-telemetry-timescaledb;
• network ndc-mission-core-telemetry;
• volumes ndc-mission-core-mqtt-data and ndc-mission-core-telemetry-timescale-data.

Vendor image names remain upstream-pinned; the NODE.DC ownership boundary is expressed by container names and com.nodedc.* labels rather than by retagging third-party images.

The durable Worker 006 compute processes follow the same namespace:

• ndc-mission-core-triton;
• ndc-mission-core-perception-worker.

Their existing mission-core-compute Compose project and network are a documented legacy predecessor. Renaming that project would recreate the network shared by the persistent perception worker, so it is intentionally deferred to a separate availability-reviewed migration rather than hidden inside this container-name change.

The telemetry probe and laboratory PowerShell launchers accept the previous mission-core-* names only as a bounded migration fallback. They prefer and create the ndc-* names. Historical report text and explicitly named debug/backup containers are evidence, not durable product runtime, and are not rewritten.

The same compose bundle can later move from the operator machine to a NODE.DC server without changing agent topics or UI contracts. Internet relay, multi-tenant access and remote command authority are out of scope for this decision.

MQTT topic contract

mission-core/v1/contours/<contour-id>/agents/<agent-id>/host
mission-core/v1/contours/<contour-id>/agents/<agent-id>/runtime
mission-core/v1/contours/<contour-id>/agents/<agent-id>/pipeline
mission-core/v1/contours/<contour-id>/agents/<agent-id>/heartbeat

Agents may publish only below their own contour and agent prefix. The normalizer may subscribe to mission-core/v1/contours/+/agents/+/+.

Consequences

• Worker 006 becomes the first contour configuration, not an architectural singleton.
• Live System telemetry no longer depends on SSH polling. The existing SSH probe is a bounded diagnostic path and can be removed after contour provisioning no longer needs it.
• Host portability is delegated to Telegraf plugins and configuration templates.
• Mission Core code stays responsible for product semantics and stable API contracts, not OS-specific metric collection.
• Host, container, network and inference telemetry is stored in the normalized path.
• Per-stage processing telemetry derives from the actual durable-worker spans. It emits explicit run start/terminal outcomes and one bounded start/aggregate-terminal pair per stage and run, with exact run, source, package, method, activation count and first/last frame identity. The inference container appends canonical documents to its existing D:-backed publish mount. The host Telegraf service tails that journal with a saved offset and publishes it through its existing authenticated QoS 1 output.
• Broker credentials remain outside the perception container. Journal write failures are observable but fail open for inference; consumers must not infer stages from aggregate hardware load or from the periodic snapshot alone.