NODEDC_MISSION_CORE/docs/adr/0031-local-compute-contour-telemetry-plane.md

# ADR 0031 — Local compute contour telemetry plane

Status: accepted for implementation
Date: 2026-07-27

## Implementation status

The first local contour is active:

- Worker 006 publishes through the native Windows service
  `NDC Mission Core Telemetry Agent` (Telegraf);
- MQTT, normalization and storage run as the single Compose project
  `ndc-mission-core-telemetry`;
- the normalizer exposes the read-only product adapter only on
  `127.0.0.1:18030`;
- Mission Core on canonical port `8000` uses this adapter for live System
  telemetry and reports the source as `agent-mqtt`;
- SSH is retained only for explicit profile diagnostics and bootstrap checks.
- the selected contour owns a managed network profile: Worker 006 keeps its node
  identity separate from its SSH address and from the broker endpoint; every saved
  endpoint and private Docker bind address is probed before an explicit transition;
- the existing contour settings can probe DNS/TCP from both Mac and Worker, apply the
  listener address to the local broker, and apply the endpoint to the Telegraf Windows
  service without exposing or replacing its scoped credential.

The existing Worker 006 containers were renamed in place to
`ndc-mission-core-triton` and `ndc-mission-core-perception-worker`. Their
container identities were preserved, so this namespace migration did not restart the
inference or perception runtimes.

The compute-side native pipeline contract is implemented and deployed on Worker 006.
It emits `missioncore.agent-pipeline-telemetry/v1` lifecycle documents through an
injected transport, and the telemetry normalizer preserves source, method, stage and
stage-state identities in the normalized series key. The E41 runner first exercised
the JSONL evidence sink against the immutable predictor package.

The durable path was then accepted on 2026-07-29 with real persistent-worker request
`durable-native-acceptance-20260729-115418`. The request reached the actual live-source
boundary and terminated because no K1 shadow source was open. Its native run
`started` and `failed` documents were appended by the worker, tailed by host Telegraf,
published on the contour pipeline topic and returned by the normalized Timescale query
API with the same run/request identity. This is transport and lifecycle acceptance,
not a successful perception-result or stage-performance claim.

The acceptance also closed two deployment-unit defects found by the real request:
the token-stdin launcher now installs its required `MissionCoreDockerNames.ps1`
companion, and the perception runner updater atomically digest-gates
`run_e15_shadow_inference.py`, `e15_shadow_runtime.py` and
`pipeline_telemetry.py`. The existing perception container is still the only
inference worker; broker credentials remain on the host Telegraf service.

## Decision

Mission Core treats a compute worker as a configurable **local compute contour**, not as
one hard-coded workstation.

The portable telemetry path is:

```text
Telegraf agent
    -> authenticated MQTT
    -> Eclipse Mosquitto
    -> Mission Core telemetry normalizer
    -> TimescaleDB OSS
    -> Mission Core System API
    -> product workspaces
```

The broker, normalizer and telemetry database form one Docker Compose contour named
`ndc-mission-core-telemetry`. They remain separate containers so each process has an
independent healthcheck, least-privilege boundary, lifecycle and rollback path. This is
one operator-managed stack, not one multi-process container.

The Telegraf agent runs as a native host service. On Worker 006 this is required for
portable access to Windows performance counters, Docker Desktop and NVIDIA telemetry.
Putting it into the Linux Docker Desktop VM would make the agent less universal and
would hide part of the host it is meant to observe.

The direct SSH/PowerShell probe remains a bounded bootstrap and diagnostic fallback for
the existing Worker 006 profile. It is not the target live telemetry transport.

Mesh access points belonging to one LAN are not modeled as separate broker addresses.
The broker endpoint may be a hostname only when the Worker can resolve it; Mission Core
does not assume that macOS `.local` discovery is portable to Windows. A move to another
LAN changes the reviewed endpoint and private bind address, not the agent identity or
MQTT topic identity. Applying a saved profile is explicit, verifies DNS and TCP
reachability, and rolls back the affected configuration when verification fails. MQTT
without TLS cannot be configured for a public endpoint or wildcard listener.

## 2026-08-05 recovery correction

Worker 006 stopped publishing because its Telegraf MQTT output still referenced the
operator Mac through a `.local` name that Windows no longer resolved. Telegraf exited
on the output connection error, and the Windows service had no configured recovery
actions. The reviewed compute-contour transition rebound the broker and worker profile
to the currently verified private LAN endpoint without replacing the scoped MQTT
credential; fresh agent, hardware and pipeline telemetry then returned.

Install, update and existing-agent network transitions now configure the same bounded
Windows recovery policy: restart after 5 seconds, 30 seconds and 60 seconds, resetting
the failure counter after one day and applying recovery to non-crash failures. This
does not make an invalid endpoint valid; it keeps transient broker or network loss from
silently leaving the telemetry agent stopped after connectivity returns.

## 2026-08-05 generic enrollment and bounded outbox

Worker 006 is no longer encoded in the broker bootstrap contract. The private
`missioncore.telemetry-agent-credential-registry/v1` registry admits any globally
unique agent identity, binds it to one contour and rebuilds the broker password file
plus exact writer ACLs from the registry. The old Worker 006 environment variables
are accepted only by a one-time private migration and are not part of the current
example configuration.

The reviewed Windows host-agent package is generated as a deterministic,
content-addressed and secret-free ZIP. A selected agent's credential is exported in a
separate non-overwriting `0600` stdin payload. This closes repeatable agent
installation without pretending to automate GPU drivers, Docker/WSL, model stores or
the perception runtime.

The native run journal is a bounded outbox. It rotates active JSONL at 64 MiB into a
content-addressed segment and retains at most eight segments. Telegraf tails the
active path and segment glob using tracking metrics and a saved offset. Reaching the
bound raises an observable telemetry transport error rather than deleting potentially
unacknowledged evidence; inference remains fail-open. Segment deletion is not
automatic and requires a separate proof that normalized storage contains the rows.
QoS 1 duplicates remain safe because the normalizer upserts the complete normalized
sample identity.

## Why these components

- **Telegraf** is the universal host agent. It already supports Windows and Linux host
  counters, Docker, NVIDIA SMI, network counters and custom HTTP/Prometheus inputs.
  Hardware discovery does not belong in a custom Mission Core agent.
- **Eclipse Mosquitto** is the local, fully open-source broker. It separates worker
  lifecycle and network location from the Mission Core UI.
- **Mission Core telemetry normalizer** owns only product semantics: contour identity,
  runtime ownership, LAB/run identity and processing-stage names. These concepts cannot
  be delegated to a generic hardware agent.
- **TimescaleDB OSS** stores normalized samples and permits comparison between laboratory
  runs without coupling agents to database credentials or schema.

Only the broker and normalizer know the MQTT credentials. Agents receive a scoped
publisher identity. The UI never receives broker or database credentials.

## Product surface brief

### User job

An operator must be able to carry Mission Core and a worker to another local network,
register one or more compute contours, install the standard telemetry agent, select the
active contour and inspect its hardware, processing and network state.

### Placement

- The `Система` left panel is a contour selector.
- A canonical plus action in its header creates another contour configuration.
- System workspace modes (`Вычислительные модули`, `Интеграции`, `Сеть`,
  `Журнал и аудит`) live in the content header because they describe the selected
  contour.
- A canonical settings utility action opens settings for the selected contour.
- Global Mission Core settings remain in the profile menu and do not mix with contour
  settings.

Rejected placements:

- keeping the single `Worker 006` as a permanent page identity;
- placing contour settings in a global `Система / Настройки` workspace;
- creating one bespoke page layout per worker;
- placing installation instructions directly in the telemetry dashboard.

### State grammar

- `unconfigured`: contour exists but no agent transport is configured;
- `provisioning`: bootstrap material was issued but the agent has not published a sample;
- `online`: a fresh normalized sample matches the expected node identity;
- `stale`: the last valid sample is older than the freshness window;
- `identity-mismatch`: an agent published with a different node identity;
- `offline`: the contour was configured but no current sample is available;
- `legacy-diagnostic`: the UI is temporarily backed by the direct SSH probe.

The UI must name the evidence source. It must not call an SSH snapshot “live agent
telemetry”.

### Evidence contract

Each normalized sample carries:

- `contour_id`, `node_id`, `agent_id`, `observed_at_utc`;
- source and normalizer schema versions;
- hardware, runtime, network and pipeline measurements;
- optional `lab_id`, `run_id`, `request_id` and frame index;
- ingestion timestamp and source topic.

Unknown measurements remain `null`; they are never synthesized for presentation.

Several Telegraf measurements can have the same name and timestamp but belong to
different disks, interfaces, containers or GPU processes. Storage therefore keys a
sample by a stable `series_key` derived from its distinguishing source tags. When a
legacy Telegraf input emits one logical series as several field fragments, the
normalizer merges those fields on conflict instead of replacing an earlier fragment.
This preserves series identity without leaking source-specific tag grammar into the
product API.

## Deployment boundary

The first stack runs inside one trusted local network and binds only to explicitly
configured LAN interfaces. Mosquitto uses password authentication and ACLs; anonymous
access is forbidden. TimescaleDB is not exposed outside the compose network.

All NODE.DC-owned Docker containers, networks and named volumes use a lowercase `ndc-`
prefix. The concrete telemetry objects are:

- `ndc-mission-core-mqtt-broker`;
- `ndc-mission-core-telemetry-normalizer`;
- `ndc-mission-core-telemetry-timescaledb`;
- network `ndc-mission-core-telemetry`;
- volumes `ndc-mission-core-mqtt-data` and
  `ndc-mission-core-telemetry-timescale-data`.

Vendor image names remain upstream-pinned; the NODE.DC ownership boundary is expressed
by container names and `com.nodedc.*` labels rather than by retagging third-party
images.

The durable Worker 006 compute processes follow the same namespace:

- `ndc-mission-core-triton`;
- `ndc-mission-core-perception-worker`.

Their existing `mission-core-compute` Compose project and network are a documented
legacy predecessor. Renaming that project would recreate the network shared by the
persistent perception worker, so it is intentionally deferred to a separate
availability-reviewed migration rather than hidden inside this container-name change.

The telemetry probe and laboratory PowerShell launchers accept the previous
`mission-core-*` names only as a bounded migration fallback. They prefer and create the
`ndc-*` names. Historical report text and explicitly named debug/backup containers are
evidence, not durable product runtime, and are not rewritten.

The same compose bundle can later move from the operator machine to a NODE.DC server
without changing agent topics or UI contracts. Internet relay, multi-tenant access and
remote command authority are out of scope for this decision.

## MQTT topic contract

```text
mission-core/v1/contours/<contour-id>/agents/<agent-id>/host
mission-core/v1/contours/<contour-id>/agents/<agent-id>/runtime
mission-core/v1/contours/<contour-id>/agents/<agent-id>/pipeline
mission-core/v1/contours/<contour-id>/agents/<agent-id>/heartbeat
```

Agents may publish only below their own contour and agent prefix. The normalizer may
subscribe to `mission-core/v1/contours/+/agents/+/+`.

## Consequences

- Worker 006 becomes the first contour configuration, not an architectural singleton.
- Live System telemetry no longer depends on SSH polling. The existing SSH probe is a
  bounded diagnostic path and can be removed after contour provisioning no longer
  needs it.
- Host portability is delegated to Telegraf plugins and configuration templates.
- Mission Core code stays responsible for product semantics and stable API contracts,
  not OS-specific metric collection.
- Host, container, network and inference telemetry is stored in the normalized path.
- Per-stage processing telemetry derives from the actual durable-worker spans. It
  emits explicit run start/terminal outcomes and one bounded start/aggregate-terminal
  pair per stage and run, with exact run, source, package, method, activation count and
  first/last frame identity. The inference container appends canonical documents to
  its existing D:-backed publish mount. The host Telegraf service tails that journal
  with a saved offset and publishes it through its existing authenticated QoS 1 output.
- Broker credentials remain outside the perception container. Journal write failures
  are observable but fail open for inference; consumers must not infer stages from
  aggregate hardware load or from the periodic snapshot alone.