feat(telemetry): add portable local telemetry plane
This commit is contained in:
DCCONSTRUCTIONS
@@ -0,0 +1,193 @@
|
||||
# ADR 0031 — Local compute contour telemetry plane
|
||||
|
||||
Status: accepted for implementation
|
||||
Date: 2026-07-27
|
||||
|
||||
## Implementation status
|
||||
|
||||
The first local contour is active:
|
||||
|
||||
- Worker 006 publishes through the native Windows service
|
||||
`NDC Mission Core Telemetry Agent` (Telegraf);
|
||||
- MQTT, normalization and storage run as the single Compose project
|
||||
`ndc-mission-core-telemetry`;
|
||||
- the normalizer exposes the read-only product adapter only on
|
||||
`127.0.0.1:18030`;
|
||||
- Mission Core on canonical port `8000` uses this adapter for live System
|
||||
telemetry and reports the source as `agent-mqtt`;
|
||||
- SSH is retained only for explicit profile diagnostics and bootstrap checks.
|
||||
|
||||
The existing Worker 006 containers were renamed in place to
|
||||
`ndc-mission-core-triton` and `ndc-mission-core-perception-worker`. Their
|
||||
container identities were preserved, so this namespace migration did not restart the
|
||||
inference or perception runtimes.
|
||||
|
||||
## Decision
|
||||
|
||||
Mission Core treats a compute worker as a configurable **local compute contour**, not as
|
||||
one hard-coded workstation.
|
||||
|
||||
The portable telemetry path is:
|
||||
|
||||
```text
|
||||
Telegraf agent
|
||||
-> authenticated MQTT
|
||||
-> Eclipse Mosquitto
|
||||
-> Mission Core telemetry normalizer
|
||||
-> TimescaleDB OSS
|
||||
-> Mission Core System API
|
||||
-> product workspaces
|
||||
```
|
||||
|
||||
The broker, normalizer and telemetry database form one Docker Compose contour named
|
||||
`ndc-mission-core-telemetry`. They remain separate containers so each process has an
|
||||
independent healthcheck, least-privilege boundary, lifecycle and rollback path. This is
|
||||
one operator-managed stack, not one multi-process container.
|
||||
|
||||
The Telegraf agent runs as a native host service. On Worker 006 this is required for
|
||||
portable access to Windows performance counters, Docker Desktop and NVIDIA telemetry.
|
||||
Putting it into the Linux Docker Desktop VM would make the agent less universal and
|
||||
would hide part of the host it is meant to observe.
|
||||
|
||||
The direct SSH/PowerShell probe remains a bounded bootstrap and diagnostic fallback for
|
||||
the existing Worker 006 profile. It is not the target live telemetry transport.
|
||||
|
||||
## Why these components
|
||||
|
||||
- **Telegraf** is the universal host agent. It already supports Windows and Linux host
|
||||
counters, Docker, NVIDIA SMI, network counters and custom HTTP/Prometheus inputs.
|
||||
Hardware discovery does not belong in a custom Mission Core agent.
|
||||
- **Eclipse Mosquitto** is the local, fully open-source broker. It separates worker
|
||||
lifecycle and network location from the Mission Core UI.
|
||||
- **Mission Core telemetry normalizer** owns only product semantics: contour identity,
|
||||
runtime ownership, LAB/run identity and processing-stage names. These concepts cannot
|
||||
be delegated to a generic hardware agent.
|
||||
- **TimescaleDB OSS** stores normalized samples and permits comparison between laboratory
|
||||
runs without coupling agents to database credentials or schema.
|
||||
|
||||
Only the broker and normalizer know the MQTT credentials. Agents receive a scoped
|
||||
publisher identity. The UI never receives broker or database credentials.
|
||||
|
||||
## Product surface brief
|
||||
|
||||
### User job
|
||||
|
||||
An operator must be able to carry Mission Core and a worker to another local network,
|
||||
register one or more compute contours, install the standard telemetry agent, select the
|
||||
active contour and inspect its hardware, processing and network state.
|
||||
|
||||
### Placement
|
||||
|
||||
- The `Система` left panel is a contour selector.
|
||||
- A canonical plus action in its header creates another contour configuration.
|
||||
- System workspace modes (`Вычислительные модули`, `Интеграции`, `Сеть`,
|
||||
`Журнал и аудит`) live in the content header because they describe the selected
|
||||
contour.
|
||||
- A canonical settings utility action opens settings for the selected contour.
|
||||
- Global Mission Core settings remain in the profile menu and do not mix with contour
|
||||
settings.
|
||||
|
||||
Rejected placements:
|
||||
|
||||
- keeping the single `Worker 006` as a permanent page identity;
|
||||
- placing contour settings in a global `Система / Настройки` workspace;
|
||||
- creating one bespoke page layout per worker;
|
||||
- placing installation instructions directly in the telemetry dashboard.
|
||||
|
||||
### State grammar
|
||||
|
||||
- `unconfigured`: contour exists but no agent transport is configured;
|
||||
- `provisioning`: bootstrap material was issued but the agent has not published a sample;
|
||||
- `online`: a fresh normalized sample matches the expected node identity;
|
||||
- `stale`: the last valid sample is older than the freshness window;
|
||||
- `identity-mismatch`: an agent published with a different node identity;
|
||||
- `offline`: the contour was configured but no current sample is available;
|
||||
- `legacy-diagnostic`: the UI is temporarily backed by the direct SSH probe.
|
||||
|
||||
The UI must name the evidence source. It must not call an SSH snapshot “live agent
|
||||
telemetry”.
|
||||
|
||||
### Evidence contract
|
||||
|
||||
Each normalized sample carries:
|
||||
|
||||
- `contour_id`, `node_id`, `agent_id`, `observed_at_utc`;
|
||||
- source and normalizer schema versions;
|
||||
- hardware, runtime, network and pipeline measurements;
|
||||
- optional `lab_id`, `run_id`, `request_id` and frame index;
|
||||
- ingestion timestamp and source topic.
|
||||
|
||||
Unknown measurements remain `null`; they are never synthesized for presentation.
|
||||
|
||||
Several Telegraf measurements can have the same name and timestamp but belong to
|
||||
different disks, interfaces, containers or GPU processes. Storage therefore keys a
|
||||
sample by a stable `series_key` derived from its distinguishing source tags. When a
|
||||
legacy Telegraf input emits one logical series as several field fragments, the
|
||||
normalizer merges those fields on conflict instead of replacing an earlier fragment.
|
||||
This preserves series identity without leaking source-specific tag grammar into the
|
||||
product API.
|
||||
|
||||
## Deployment boundary
|
||||
|
||||
The first stack runs inside one trusted local network and binds only to explicitly
|
||||
configured LAN interfaces. Mosquitto uses password authentication and ACLs; anonymous
|
||||
access is forbidden. TimescaleDB is not exposed outside the compose network.
|
||||
|
||||
All NODE.DC-owned Docker containers, networks and named volumes use a lowercase `ndc-`
|
||||
prefix. The concrete telemetry objects are:
|
||||
|
||||
- `ndc-mission-core-mqtt-broker`;
|
||||
- `ndc-mission-core-telemetry-normalizer`;
|
||||
- `ndc-mission-core-telemetry-timescaledb`;
|
||||
- network `ndc-mission-core-telemetry`;
|
||||
- volumes `ndc-mission-core-mqtt-data` and
|
||||
`ndc-mission-core-telemetry-timescale-data`.
|
||||
|
||||
Vendor image names remain upstream-pinned; the NODE.DC ownership boundary is expressed
|
||||
by container names and `com.nodedc.*` labels rather than by retagging third-party
|
||||
images.
|
||||
|
||||
The durable Worker 006 compute processes follow the same namespace:
|
||||
|
||||
- `ndc-mission-core-triton`;
|
||||
- `ndc-mission-core-perception-worker`.
|
||||
|
||||
Their existing `mission-core-compute` Compose project and network are a documented
|
||||
legacy predecessor. Renaming that project would recreate the network shared by the
|
||||
persistent perception worker, so it is intentionally deferred to a separate
|
||||
availability-reviewed migration rather than hidden inside this container-name change.
|
||||
|
||||
The telemetry probe and laboratory PowerShell launchers accept the previous
|
||||
`mission-core-*` names only as a bounded migration fallback. They prefer and create the
|
||||
`ndc-*` names. Historical report text and explicitly named debug/backup containers are
|
||||
evidence, not durable product runtime, and are not rewritten.
|
||||
|
||||
The same compose bundle can later move from the operator machine to a NODE.DC server
|
||||
without changing agent topics or UI contracts. Internet relay, multi-tenant access and
|
||||
remote command authority are out of scope for this decision.
|
||||
|
||||
## MQTT topic contract
|
||||
|
||||
```text
|
||||
mission-core/v1/contours/<contour-id>/agents/<agent-id>/host
|
||||
mission-core/v1/contours/<contour-id>/agents/<agent-id>/runtime
|
||||
mission-core/v1/contours/<contour-id>/agents/<agent-id>/pipeline
|
||||
mission-core/v1/contours/<contour-id>/agents/<agent-id>/heartbeat
|
||||
```
|
||||
|
||||
Agents may publish only below their own contour and agent prefix. The normalizer may
|
||||
subscribe to `mission-core/v1/contours/+/agents/+/+`.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Worker 006 becomes the first contour configuration, not an architectural singleton.
|
||||
- Live System telemetry no longer depends on SSH polling. The existing SSH probe is a
|
||||
bounded diagnostic path and can be removed after contour provisioning no longer
|
||||
needs it.
|
||||
- Host portability is delegated to Telegraf plugins and configuration templates.
|
||||
- Mission Core code stays responsible for product semantics and stable API contracts,
|
||||
not OS-specific metric collection.
|
||||
- Host, container, network and inference telemetry is stored in the normalized path.
|
||||
Per-stage LAB processing telemetry remains unavailable until a laboratory worker
|
||||
publishes the native `runtime` and `pipeline` topic contracts; the UI must keep those
|
||||
stages explicitly unavailable rather than infer them from aggregate hardware load.
|
||||
Reference in New Issue
Block a user