200 lines
10 KiB
Markdown
200 lines
10 KiB
Markdown
# NDC Mission Core local telemetry plane
|
|
|
|
Portable local-only telemetry infrastructure for compute contours.
|
|
|
|
One Docker Compose project, `ndc-mission-core-telemetry`, owns three long-running
|
|
containers and one bounded bootstrap job:
|
|
|
|
- `ndc-mission-core-mqtt-broker` — Eclipse Mosquitto `2.1.2-alpine`;
|
|
- `ndc-mission-core-telemetry-timescaledb` — TimescaleDB HA OSS
|
|
`pg16.14-ts2.28.2-all-oss`;
|
|
- `ndc-mission-core-telemetry-bootstrap` — one-shot database role/bootstrap job;
|
|
- `ndc-mission-core-telemetry-normalizer` — Mission Core telemetry normalizer.
|
|
|
|
The Telegraf configuration templates cover Windows and Linux, but the agent runs as a
|
|
host service rather than inside this Compose project. This preserves access to native
|
|
Windows performance counters, Docker Desktop and NVIDIA telemetry.
|
|
|
|
The normalizer exposes a read-only normalized telemetry adapter on
|
|
`127.0.0.1:18030`. Mission Core reads this adapter; the browser never receives MQTT or
|
|
Timescale credentials, and Timescale is not published on a host port.
|
|
The normalizer uses a separate `missioncore_normalizer` database role with only
|
|
`SELECT`, `INSERT`, `UPDATE`, and bounded retention `DELETE` on the telemetry
|
|
hypertable. Raw telemetry older than 30 days is removed at most once per day by the
|
|
normalizer. This stays compatible with the Apache-licensed Timescale image without
|
|
depending on the Timescale License retention scheduler.
|
|
|
|
The stack is one deployment contour, not one multi-process container. Keeping broker,
|
|
normalizer and database in separate containers preserves independent health checks,
|
|
least-privilege boundaries and rollback while Compose provides one operator lifecycle:
|
|
|
|
```bash
|
|
docker compose ps
|
|
docker compose up -d
|
|
docker compose down
|
|
```
|
|
|
|
All NODE.DC-owned Docker objects use the lowercase `ndc-` namespace. Docker names are
|
|
case-sensitive identifiers, so the product prefix is normalized to lowercase while the
|
|
product name remains NODE.DC in operator-facing copy.
|
|
|
|
This directory does not contain credentials. Copy `.env.example` to `.env`, generate
|
|
unique passwords and build the ACL/password files before starting the stack.
|
|
|
|
```bash
|
|
uv run python prepare.py --initialize --mqtt-bind-address <MISSION_CORE_HOST_LAN_IP>
|
|
uv run python prepare.py --enroll-agent worker-006 --contour-id worker-006
|
|
uv run python prepare.py
|
|
docker compose up -d --build
|
|
```
|
|
|
|
`--enroll-agent` is generic: repeat it with a new globally unique `agent-id` and
|
|
the contour it owns. Agent credentials live only in the private
|
|
`runtime/agents.json` registry with mode `0600`. A normal `prepare.py` run rebuilds
|
|
the Mosquitto password file from that exact registry and generates one non-wildcard
|
|
writer ACL per agent, so removed or renamed identities cannot survive in the broker
|
|
password file by accident. Existing installations can use `--migrate` to import the
|
|
old Worker 006 variables once without replacing their credential.
|
|
|
|
Build the reviewed Windows installer bundle and export the selected agent's private
|
|
stdin payload as separate files:
|
|
|
|
```bash
|
|
uv run python prepare.py --build-agent-bundle windows \
|
|
--output runtime/missioncore-telemetry-agent-windows.zip
|
|
uv run python prepare.py --export-agent-payload worker-006 \
|
|
--node-id DESKTOP-OPJ8J04 \
|
|
--output runtime/worker-006.private.json
|
|
```
|
|
|
|
The ZIP is deterministic, content-addressed and contains only the pinned installer,
|
|
updater, collector and configuration template. It contains no credential. Transfer
|
|
the ZIP and private payload separately; on the Worker, unpack the ZIP and pass the
|
|
payload through stdin:
|
|
|
|
```powershell
|
|
Get-Content .\worker-006.private.json -Raw |
|
|
.\Install-NdcMissionCoreTelegraf.ps1
|
|
```
|
|
|
|
Delete the transferred payload after the service has been accepted. The installer
|
|
persists the scoped credential only in the ACL-restricted Windows service environment.
|
|
|
|
Expected Docker object names:
|
|
|
|
```text
|
|
project: ndc-mission-core-telemetry
|
|
network: ndc-mission-core-telemetry
|
|
containers:
|
|
ndc-mission-core-mqtt-broker
|
|
ndc-mission-core-telemetry-normalizer
|
|
ndc-mission-core-telemetry-timescaledb
|
|
volumes:
|
|
ndc-mission-core-mqtt-data
|
|
ndc-mission-core-telemetry-timescale-data
|
|
```
|
|
|
|
`prepare.py` passes passwords to `mosquitto_passwd` through stdin. Secrets are not placed
|
|
in process arguments or committed files. `--initialize` generates `.env` with mode
|
|
`0600` and refuses to replace existing credentials. The generated `.env` and `runtime/`
|
|
directory are ignored by Git.
|
|
|
|
After initial provisioning, the selected compute contour owns the operational network
|
|
profile in `Система → Вычислительные модули → Настройки контура`. The profile keeps
|
|
these concerns separate:
|
|
|
|
- the stable broker hostname resolved by the worker, preferably the operator Mac's
|
|
`.local` name;
|
|
- the exact private LAN address on which Docker publishes the authenticated listener;
|
|
- the stable worker hostname used by the trusted SSH bootstrap profile.
|
|
|
|
Mesh access points inside one LAN are not registered as separate broker endpoints.
|
|
Roaming between them preserves the same hostname-based profile. Moving the kit to a
|
|
different LAN requires selecting the Mac's new private bind address, applying it to the
|
|
broker, and then applying the saved endpoint to the worker. Both actions are explicit.
|
|
The broker update changes only `MISSIONCORE_MQTT_BIND_ADDRESS` in the private `.env`;
|
|
the worker update changes only the Telegraf service endpoint and interval, preserves its
|
|
scoped credential, verifies DNS plus TCP, and rolls back on failure. Wildcard
|
|
`0.0.0.0` and public MQTT endpoints are rejected.
|
|
|
|
The product architecture and topic contract are defined in
|
|
`docs/adr/0031-local-compute-contour-telemetry-plane.md`.
|
|
|
|
## Security boundary
|
|
|
|
The current MQTT listener is authenticated and contour-scoped, but intentionally
|
|
uses plaintext MQTT inside one owner-controlled laboratory LAN. Do not expose port
|
|
1883 through a router, cellular WAN, public Wi-Fi, or an Internet-facing host. A
|
|
remote or shared-network deployment requires a reviewed TLS listener, a private CA
|
|
distributed to every agent, and credential rotation. Wi-Fi link encryption is not a
|
|
replacement for MQTT TLS.
|
|
|
|
Each agent credential is bound to one exact
|
|
`contours/<contour-id>/agents/<agent-id>/+` prefix. Adding another contour requires
|
|
`--enroll-agent`; the wildcard contour writer is not permitted. An `agent-id` is
|
|
globally unique because Mosquitto ACL ownership is username-based.
|
|
|
|
## Worker agent
|
|
|
|
Worker 006 uses the official Windows Telegraf distribution as the host service
|
|
`NDC Mission Core Telemetry Agent`. Install or update it with:
|
|
|
|
```powershell
|
|
.\telegraf\Install-NdcMissionCoreTelegraf.ps1
|
|
.\telegraf\Update-NdcMissionCoreTelegraf.ps1
|
|
```
|
|
|
|
The update path validates the candidate configuration, backs up the active
|
|
configuration and rolls back if the service does not return to `Running`. MQTT
|
|
credentials are scoped to the service environment and must not be passed on a command
|
|
line or stored in the repository.
|
|
|
|
Both install and update configure Windows Service Control Manager recovery for
|
|
Telegraf: restart after 5 seconds, then 30 seconds, then 60 seconds, with the failure
|
|
counter reset after one day. The compute-contour worker network transition applies
|
|
the same policy to already-provisioned agents and fails closed unless the worker
|
|
confirms it.
|
|
|
|
The same host service reads the existing perception worker's loopback `/health`
|
|
contract with `Get-NdcMissionCorePipelineTelemetry.ps1` and publishes nine
|
|
stage-keyed snapshots to the contour's `pipeline` topic. The perception container
|
|
does not receive broker credentials and no second agent container is introduced.
|
|
These snapshots expose current durable-worker state and cumulative stage timing.
|
|
|
|
The runner also appends canonical run lifecycle documents plus one bounded
|
|
start/aggregate-terminal pair per stage to `pipeline-telemetry.jsonl` in the
|
|
already-mounted persistent publish directory. Stage records retain the first/last
|
|
frame boundary and aggregate activation count without producing one MQTT row per
|
|
frame activation.
|
|
Telegraf's `inputs.tail` owns the saved file offset, keeps at most 1000 undelivered
|
|
lines in flight and publishes the records through the same QoS 1 pipeline output.
|
|
The source outbox rotates at 64 MiB into content-addressed
|
|
`pipeline-telemetry.<sha256>.jsonl` segments. Telegraf tails both the active file and
|
|
segments. At eight retained segments the writer fails telemetry publication
|
|
observably instead of deleting evidence which may not yet have been acknowledged.
|
|
Segment reclamation therefore remains an explicit operator action after normalized
|
|
storage is verified; inference control flow remains fail-open.
|
|
The normalizer verifies the topic-bound record and restores the original native
|
|
document before storage. A broker outage therefore stays inside the existing
|
|
Telegraf buffer; the perception container receives neither MQTT credentials nor a
|
|
second network client. Telemetry write failure is reported in `/health` but does not
|
|
change inference control flow.
|
|
|
|
When the mounted perception runner itself changes, use
|
|
`Update-NdcMissionCorePerceptionRunner.ps1` with exact predecessor and candidate
|
|
digests for the runner, its `e15_shadow_runtime.py` companion and its standalone
|
|
telemetry module. The three files form one deployment unit: updating the runner
|
|
without the matching runtime is rejected by the digest-gated transaction. It backs
|
|
up the mounted files, restarts the same container, accepts only a ready health
|
|
document with stage metrics and a ready native journal transport, and restores the
|
|
predecessor set on failure. Update the token-stdin launcher separately with
|
|
`Update-NdcMissionCorePersistentLauncher.ps1`. The launcher update always installs
|
|
and digest-checks `MissionCoreDockerNames.ps1` in the same transaction because the
|
|
launcher dot-sources that companion contract before any worker request. Both files
|
|
are restored (or a previously absent companion is removed) if validation fails. This
|
|
keeps the operational contour/agent/node identity explicit and reviewable without
|
|
changing the K1 command sequence.
|
|
|
|
The stack and agent are intentionally not started by repository tests. Provisioning a
|
|
machine is a separate, explicit operation.
|