fix(telemetry): recover worker agent and Tailscale transport automatically
This commit is contained in:
@@ -149,9 +149,11 @@ globally unique because Mosquitto ACL ownership is username-based.
|
||||
|
||||
## Worker agent
|
||||
|
||||
MQTT outputs use `startup_error_behavior="retry"`, with a 2000-metric buffer and
|
||||
the configured flush interval. A missing broker at agent startup must not end
|
||||
the service. See [Telegraf's startup policy](https://docs.influxdata.com/telegraf/v1/configuration/plugin-options/).
|
||||
MQTT outputs request `startup_error_behavior="retry"`, with a 2000-metric buffer
|
||||
and the configured flush interval. This setting alone does not guarantee
|
||||
recovery: Telegraf 1.38.4 can exit cleanly when MQTT is unavailable at startup.
|
||||
The Tailscale profile below therefore also installs a system recovery task.
|
||||
See [Telegraf's startup policy](https://docs.influxdata.com/telegraf/v1/configuration/plugin-options/).
|
||||
If Docker lost a published listener while the saved LAN address is unchanged,
|
||||
the explicit broker Apply action reconciles that listener; a telemetry GET never
|
||||
restarts infrastructure. Node connectivity does not prove a profile is ready.
|
||||
@@ -228,3 +230,26 @@ changing the K1 command sequence.
|
||||
|
||||
The stack and agent are intentionally not started by repository tests. Provisioning a
|
||||
machine is a separate, explicit operation.
|
||||
|
||||
## Tailscale operator profile and startup recovery
|
||||
|
||||
The accepted 2026-09-25 operator profile no longer depends on a shared LAN or
|
||||
DHCP address. A prepared Windows agent uses `127.0.0.1:1883`; the existing strict
|
||||
Tailscale SSH identity carries a loopback reverse forward to the operator
|
||||
broker. Operator deployment is owned by `scripts/manage_telemetry_startup.py`
|
||||
(plan, hash-bound apply, rollback). This is a macOS **login** profile, not a
|
||||
pre-login Docker daemon. Preserve the explicit prepared stack root and its
|
||||
private credentials when moving the Core source checkout.
|
||||
|
||||
For this profile the Windows install/update bundle includes and invokes
|
||||
`Install-NdcMissionCoreTelemetryRecovery.ps1`: boot + once-per-minute SYSTEM
|
||||
reconciliation for a stopped Telegraf service, independent of login. Its
|
||||
maintenance/rollback boundary is described in the audit. Do not rely solely on
|
||||
SCM failure actions or the MQTT `startup_error_behavior` setting: the pinned
|
||||
1.38.4 release was observed to terminate with a clean service exit when the
|
||||
broker was unavailable during boot.
|
||||
|
||||
See [the measured recovery audit](../../docs/audits/2026-09-25-worker-telemetry-recovery.md)
|
||||
for exact acceptance, remaining cold-boot/clean-host gates and bounded test
|
||||
entrypoints. The earlier LAN workflow remains a legacy explicit configuration;
|
||||
it is not the current Worker 006 transport.
|
||||
|
||||
Reference in New Issue
Block a user