fix(telemetry): recover worker agent and Tailscale transport automatically

This commit is contained in:
DCCONSTRUCTIONS
2026-09-25 23:24:27 +03:00
parent bd5873ef34
commit f3640026cf
11 changed files with 755 additions and 4 deletions
+28 -3
View File
@@ -149,9 +149,11 @@ globally unique because Mosquitto ACL ownership is username-based.
## Worker agent
MQTT outputs use `startup_error_behavior="retry"`, with a 2000-metric buffer and
the configured flush interval. A missing broker at agent startup must not end
the service. See [Telegraf's startup policy](https://docs.influxdata.com/telegraf/v1/configuration/plugin-options/).
MQTT outputs request `startup_error_behavior="retry"`, with a 2000-metric buffer
and the configured flush interval. This setting alone does not guarantee
recovery: Telegraf 1.38.4 can exit cleanly when MQTT is unavailable at startup.
The Tailscale profile below therefore also installs a system recovery task.
See [Telegraf's startup policy](https://docs.influxdata.com/telegraf/v1/configuration/plugin-options/).
If Docker lost a published listener while the saved LAN address is unchanged,
the explicit broker Apply action reconciles that listener; a telemetry GET never
restarts infrastructure. Node connectivity does not prove a profile is ready.
@@ -228,3 +230,26 @@ changing the K1 command sequence.
The stack and agent are intentionally not started by repository tests. Provisioning a
machine is a separate, explicit operation.
## Tailscale operator profile and startup recovery
The accepted 2026-09-25 operator profile no longer depends on a shared LAN or
DHCP address. A prepared Windows agent uses `127.0.0.1:1883`; the existing strict
Tailscale SSH identity carries a loopback reverse forward to the operator
broker. Operator deployment is owned by `scripts/manage_telemetry_startup.py`
(plan, hash-bound apply, rollback). This is a macOS **login** profile, not a
pre-login Docker daemon. Preserve the explicit prepared stack root and its
private credentials when moving the Core source checkout.
For this profile the Windows install/update bundle includes and invokes
`Install-NdcMissionCoreTelemetryRecovery.ps1`: boot + once-per-minute SYSTEM
reconciliation for a stopped Telegraf service, independent of login. Its
maintenance/rollback boundary is described in the audit. Do not rely solely on
SCM failure actions or the MQTT `startup_error_behavior` setting: the pinned
1.38.4 release was observed to terminate with a clean service exit when the
broker was unavailable during boot.
See [the measured recovery audit](../../docs/audits/2026-09-25-worker-telemetry-recovery.md)
for exact acceptance, remaining cold-boot/clean-host gates and bounded test
entrypoints. The earlier LAN workflow remains a legacy explicit configuration;
it is not the current Worker 006 transport.