Files
NODEDC_MISSION_CORE/deploy/telemetry-plane

NDC Mission Core local telemetry plane

Portable local-only telemetry infrastructure for compute contours.

One Docker Compose project, ndc-mission-core-telemetry, owns three long-running containers and one bounded bootstrap job:

  • ndc-mission-core-mqtt-broker — Eclipse Mosquitto 2.1.2-alpine;
  • ndc-mission-core-telemetry-timescaledb — TimescaleDB HA OSS pg16.14-ts2.28.2-all-oss;
  • ndc-mission-core-telemetry-bootstrap — one-shot database role/bootstrap job;
  • ndc-mission-core-telemetry-normalizer — Mission Core telemetry normalizer.

The Telegraf configuration templates cover Windows and Linux, but the agent runs as a host service rather than inside this Compose project. This preserves access to native Windows performance counters, Docker Desktop and NVIDIA telemetry.

The normalizer exposes a read-only normalized telemetry adapter on 127.0.0.1:18030. Mission Core reads this adapter; the browser never receives MQTT or Timescale credentials, and Timescale is not published on a host port. The normalizer uses a separate missioncore_normalizer database role with only SELECT, INSERT, UPDATE, and bounded retention DELETE on the telemetry hypertable. Raw telemetry older than 30 days is removed in batches of at most 1000 rows per minute, on a separate connection/thread (2s statement / 250ms lock deadline). Retention never runs inside the MQTT callback. This stays compatible with the Apache-licensed Timescale image without depending on the Timescale License retention scheduler.

The 1 GiB database container explicitly sets 128 MiB shared buffers and 64 MiB maintenance memory. The HA image's automatic tuning sees the Docker VM instead of this container limit; do not inherit its multi-GiB defaults.

For a memory-constrained local repair, compose.source.yaml mounts the reviewed normalizer source read-only into the existing image. Set MISSIONCORE_NORMALIZER_SOURCE to the absolute normalizer.py path, pass both Compose files and the existing private --env-file / --project-directory, and use up -d --no-deps --no-build normalizer. This is an explicit local source override, not a rebuilt portable image. Normal restart/reboot retains the mount.

The stack is one deployment contour, not one multi-process container. Keeping broker, normalizer and database in separate containers preserves independent health checks, least-privilege boundaries and rollback while Compose provides one operator lifecycle:

docker compose ps
docker compose up -d
docker compose down

All NODE.DC-owned Docker objects use the lowercase ndc- namespace. Docker names are case-sensitive identifiers, so the product prefix is normalized to lowercase while the product name remains NODE.DC in operator-facing copy.

This directory does not contain credentials. Copy .env.example to .env, generate unique passwords and build the ACL/password files before starting the stack.

uv run python prepare.py --initialize --mqtt-bind-address <MISSION_CORE_HOST_LAN_IP>
uv run python prepare.py --enroll-agent worker-006 --contour-id worker-006
uv run python prepare.py
docker compose up -d --build

--enroll-agent is generic: repeat it with a new globally unique agent-id and the contour it owns. Agent credentials live only in the private runtime/agents.json registry with mode 0600. A normal prepare.py run rebuilds the Mosquitto password file from that exact registry and generates one non-wildcard writer ACL per agent, so removed or renamed identities cannot survive in the broker password file by accident. Existing installations can use --migrate to import the old Worker 006 variables once without replacing their credential.

Build the reviewed Windows installer bundle and export the selected agent's private stdin payload as separate files:

uv run python prepare.py --build-agent-bundle windows \
  --output runtime/missioncore-telemetry-agent-windows.zip
uv run python prepare.py --export-agent-payload worker-006 \
  --node-id DESKTOP-OPJ8J04 \
  --output runtime/worker-006.private.json

The ZIP is deterministic, content-addressed and contains only the pinned installer, updater, collector and configuration template. It contains no credential. Transfer the ZIP and private payload separately; on the Worker, unpack the ZIP and pass the payload through stdin:

Get-Content .\worker-006.private.json -Raw |
  .\Install-NdcMissionCoreTelegraf.ps1

Delete the transferred payload after the service has been accepted. The installer persists the scoped credential only in the ACL-restricted Windows service environment.

Expected Docker object names:

project:   ndc-mission-core-telemetry
network:   ndc-mission-core-telemetry
containers:
  ndc-mission-core-mqtt-broker
  ndc-mission-core-telemetry-normalizer
  ndc-mission-core-telemetry-timescaledb
volumes:
  ndc-mission-core-mqtt-data
  ndc-mission-core-telemetry-timescale-data

prepare.py passes passwords to mosquitto_passwd through stdin. Secrets are not placed in process arguments or committed files. --initialize generates .env with mode 0600 and refuses to replace existing credentials. The generated .env and runtime/ directory are ignored by Git.

After initial provisioning, the selected compute contour owns the operational network profile in Система → Вычислительные модули → Настройки контура. The profile keeps these concerns separate:

  • the stable broker hostname resolved by the worker, preferably the operator Mac's .local name;
  • the exact private LAN address on which Docker publishes the authenticated listener;
  • the stable worker hostname used by the trusted SSH bootstrap profile.

Mesh access points inside one LAN are not registered as separate broker endpoints. Roaming between them preserves the same hostname-based profile. Moving the kit to a different LAN requires selecting the Mac's new private bind address, applying it to the broker, and then applying the saved endpoint to the worker. Both actions are explicit. The broker update changes only MISSIONCORE_MQTT_BIND_ADDRESS in the private .env; the worker update changes only the Telegraf service endpoint and interval, preserves its scoped credential, verifies DNS plus TCP, and rolls back on failure. Wildcard 0.0.0.0 and public MQTT endpoints are rejected.

The product architecture and topic contract are defined in docs/adr/0031-local-compute-contour-telemetry-plane.md.

Security boundary

The current MQTT listener is authenticated and contour-scoped, but intentionally uses plaintext MQTT inside one owner-controlled laboratory LAN. Do not expose port 1883 through a router, cellular WAN, public Wi-Fi, or an Internet-facing host. A remote or shared-network deployment requires a reviewed TLS listener, a private CA distributed to every agent, and credential rotation. Wi-Fi link encryption is not a replacement for MQTT TLS.

Each agent credential is bound to one exact contours/<contour-id>/agents/<agent-id>/+ prefix. Adding another contour requires --enroll-agent; the wildcard contour writer is not permitted. An agent-id is globally unique because Mosquitto ACL ownership is username-based.

Worker agent

MQTT outputs use startup_error_behavior="retry", with a 2000-metric buffer and the configured flush interval. A missing broker at agent startup must not end the service. See Telegraf's startup policy. If Docker lost a published listener while the saved LAN address is unchanged, the explicit broker Apply action reconciles that listener; a telemetry GET never restarts infrastructure. Node connectivity does not prove a profile is ready.

The streaming profile can export one bounded (8 KiB) current observation using --telemetry-snapshot /telemetry/current.json, with /telemetry mounted to C:\ProgramData\NDC\MissionCore\telemetry-agent\perception on Worker. The existing collector reads it with a 5s freshness limit and forwards it over the same MQTT pipeline topic. It overrides the legacy service only when explicitly present; an expired/broken file reports unavailable rather than relabeling a legacy model. The publisher runs independently from heartbeat/inference and never renews a lease, grants GPU ownership, or qualifies real-time. Queue memory includes active input and decoder scratch. Stage timings are unavailable until actually measured. No camera, point cloud, model result, MQTT secret, or motor command goes here.

Worker 006 uses the official Windows Telegraf distribution as the host service NDC Mission Core Telemetry Agent. Install or update it with:

.\telegraf\Install-NdcMissionCoreTelegraf.ps1
.\telegraf\Update-NdcMissionCoreTelegraf.ps1

The update path validates the candidate configuration, backs up the active configuration and rolls back if the service does not return to Running. MQTT credentials are scoped to the service environment and must not be passed on a command line or stored in the repository.

Both install and update configure Windows Service Control Manager recovery for Telegraf: restart after 5 seconds, then 30 seconds, then 60 seconds, with the failure counter reset after one day. The compute-contour worker network transition applies the same policy to already-provisioned agents and fails closed unless the worker confirms it.

The same host service reads the existing perception worker's loopback /health contract with Get-NdcMissionCorePipelineTelemetry.ps1 and publishes nine stage-keyed snapshots to the contour's pipeline topic. The perception container does not receive broker credentials and no second agent container is introduced. These snapshots expose current durable-worker state and cumulative stage timing.

The runner also appends canonical run lifecycle documents plus one bounded start/aggregate-terminal pair per stage to pipeline-telemetry.jsonl in the already-mounted persistent publish directory. Stage records retain the first/last frame boundary and aggregate activation count without producing one MQTT row per frame activation. Telegraf's inputs.tail owns the saved file offset, keeps at most 1000 undelivered lines in flight and publishes the records through the same QoS 1 pipeline output. The source outbox rotates at 64 MiB into content-addressed pipeline-telemetry.<sha256>.jsonl segments. Telegraf tails both the active file and segments. At eight retained segments the writer fails telemetry publication observably instead of deleting evidence which may not yet have been acknowledged. Segment reclamation therefore remains an explicit operator action after normalized storage is verified; inference control flow remains fail-open. The normalizer verifies the topic-bound record and restores the original native document before storage. A broker outage therefore stays inside the existing Telegraf buffer; the perception container receives neither MQTT credentials nor a second network client. Telemetry write failure is reported in /health but does not change inference control flow.

When the mounted perception runner itself changes, use Update-NdcMissionCorePerceptionRunner.ps1 with exact predecessor and candidate digests for the runner, its e15_shadow_runtime.py companion and its standalone telemetry module. The three files form one deployment unit: updating the runner without the matching runtime is rejected by the digest-gated transaction. It backs up the mounted files, restarts the same container, accepts only a ready health document with stage metrics and a ready native journal transport, and restores the predecessor set on failure. Update the token-stdin launcher separately with Update-NdcMissionCorePersistentLauncher.ps1. The launcher update always installs and digest-checks MissionCoreDockerNames.ps1 in the same transaction because the launcher dot-sources that companion contract before any worker request. Both files are restored (or a previously absent companion is removed) if validation fails. This keeps the operational contour/agent/node identity explicit and reviewable without changing the K1 command sequence.

The stack and agent are intentionally not started by repository tests. Provisioning a machine is a separate, explicit operation.