fix(service): supervise canonical Mission Core lifecycle

2026-08-25 21:58:22 +03:00
parent fb5bf943c9
commit f50e0077c5
10 changed files with 1095 additions and 6 deletions
@@ -0,0 +1,110 @@
+# Mission Core local service recovery
+
+## Scope
+
+This runbook owns the single local Mission Core backend at
+`http://127.0.0.1:8000`. It covers startup, health supervision, bounded
+shutdown and recovery on the current macOS operator station. It does not grant
+physical acquisition, actuation, navigation or safety authority.
+
+The canonical LaunchAgent label is:
+
+```text
+com.nodedc.mission-core.local
+```
+
+Do not start a second backend on another port. Do not use
+`launchctl kickstart -k` for this service: it combines termination and restart
+without proving that the old process group has released the singleton lease.
+
+## Recovery contract
+
+The installed LaunchAgent and application form one bounded recovery ladder:
+
+1. `launchd` starts one `uv run --no-sync k1link serve` process group with
+   `RunAtLoad=true`, `KeepAlive=true`, `AbandonProcessGroup=false` and a
+   five-second throttle.
+2. Mission Core starts a private self-health thread after acquiring the
+   singleton backend lease.
+3. After the 45-second cold-start grace, three consecutive failed exact
+   `/api/health` probes request `SIGTERM` for the complete process group.
+4. Uvicorn stops accepting work and has ten seconds to drain active requests
+   and ASGI lifespan work.
+5. If the service remains alive, the watchdog escalates to `SIGKILL` after
+   twelve seconds. `launchd` also owns a 20-second exit deadline.
+6. `launchd` starts a fresh process group. Startup reconstructs plugin
+   runtimes and the recording-preparation reconciler from durable state.
+
+The watchdog journal is private, bounded and rotated:
+
+```text
+.runtime/mission-core/service-watchdog.jsonl
+.runtime/mission-core/service-watchdog.jsonl.1
+```
+
+Physical operations are deliberately not resumed from an assumed state.
+Interrupted preparation and catalog work is reconciled from durable evidence;
+physical acquisition, commands and actuation remain fail-closed and require a
+new confirmed authority transition.
+
+## Plan and apply
+
+Always plan from the repository root before changing the installed agent:
+
+```bash
+uv run python scripts/manage_mission_core_launch_agent.py plan \
+  --repository-root "$PWD"
+```
+
+Copy the exact `current_sha256` and `desired_sha256` from that output into the
+apply command:
+
+```bash
+uv run python scripts/manage_mission_core_launch_agent.py apply \
+  --repository-root "$PWD" \
+  --expected-current-sha256 <current-sha256> \
+  --expected-desired-sha256 <desired-sha256>
+```
+
+Apply writes a mode-0600 backup below
+`.runtime/mission-core/launch-agent-backups`, atomically replaces the plist,
+waits until `bootout` has fully removed the old label, bootstraps the new
+declaration and accepts only the exact Mission Core health document. Any
+failure restores the previous plist and repeats the same health acceptance.
+
+Read-only status:
+
+```bash
+uv run python scripts/manage_mission_core_launch_agent.py status \
+  --repository-root "$PWD"
+```
+
+## Acceptance after recovery
+
+The service is recovered only when all of the following are true:
+
+- `launchctl print gui/$(id -u)/com.nodedc.mission-core.local` reports
+  `state = running`;
+- the arguments include `uv run --no-sync k1link serve`;
+- the environment contains `MISSIONCORE_SERVICE_WATCHDOG => 1`;
+- the launchd exit timeout is 20 seconds;
+- `GET http://127.0.0.1:8000/api/health` returns HTTP 200 with
+  `ok=true`, `status=ok` and
+  `service=mission-core-control-plane`;
+- the watchdog journal contains `watchdog-started` for the current child PID;
+- there is only one `uv` parent and one Mission Core Python child in their
+  exact process group.
+
+## 2026-08-25 recovery qualification
+
+The reviewed declaration SHA-256 was
+`80fca5ec6bdab21f11a544d8dbee65a35f73a6c34752c6a48e9a1181e5da256a`.
+A controlled `SIGTERM` of the exact Mission Core process group changed the
+LaunchAgent PID from `40866` to `40957`; exact health recovered automatically
+in `26.588` seconds without a manual start. After the explicit
+`AbandonProcessGroup=false` fence was installed, a controlled `SIGKILL` of PID
+and PGID `42895` produced a new LaunchAgent PID `42942` and exact health in
+`22.996` seconds, with no old process-group residue. This is local
+operator-station evidence only. An onboard Linux deployment must express the
+same contract in its init system and pass its own power-loss, crash, hang and
+durable-state reconciliation qualification.