fix(service): supervise canonical Mission Core lifecycle
This commit is contained in:
DCCONSTRUCTIONS
@@ -0,0 +1,116 @@
|
||||
# Mission Core backend lifecycle audit — 2026-08-25
|
||||
|
||||
## Outcome
|
||||
|
||||
The observed outage was operator-tool induced, not an unexplained Python crash,
|
||||
GPU out-of-memory event or host memory leak. A Codex session repeatedly used
|
||||
forced LaunchAgent restarts while the LAB camera/timeline endpoints still had
|
||||
active work. The final forced termination left the old Uvicorn child draining
|
||||
while a replacement tried to acquire the same singleton service lease.
|
||||
|
||||
The failure class is a **process-lifecycle termination leak**: the old process
|
||||
remained alive beyond the restart command's assumption. It is not evidence of
|
||||
unbounded heap growth.
|
||||
|
||||
## Evidence and causal chain
|
||||
|
||||
The private Codex rollout journal
|
||||
`~/.codex/sessions/2026/08/25/rollout-2026-08-25T01-08-27-01a035d1-587d-7112-9506-7cc801c2863c.jsonl`
|
||||
for the active 2026-08-25 task records
|
||||
`launchctl kickstart -k` against `com.nodedc.mission-core.local` at 14:40,
|
||||
15:38, 15:44, 16:22, 16:32, 17:14, 17:29, 17:31 and 17:42 Moscow time. At
|
||||
17:15 it additionally records `SIGTERM`, a ten-second wait, then `SIGKILL`
|
||||
against the exact old process before another kickstart.
|
||||
|
||||
The user-visible 17:09 Moscow-time outage occurred after the 16:32 forced
|
||||
restart and before the later 17:15 TERM/KILL recovery attempt. This ordering
|
||||
rules out the later kill as the start of that outage while still attributes the
|
||||
failure window to the same repeated forced-restart sequence.
|
||||
|
||||
The LaunchAgent evidence showed 139 historical runs and last exit code 143
|
||||
(`SIGTERM`), with no jetsam/OOM record. The application log showed Uvicorn
|
||||
entering graceful shutdown and waiting for connections/background tasks while
|
||||
M4.8S LAB camera/timeline requests were open. Replacement processes reported
|
||||
that Mission Core was already starting or stopping because the old child still
|
||||
held `.runtime/mission-core/.serve.lock`.
|
||||
|
||||
The complete causal chain was:
|
||||
|
||||
```text
|
||||
Codex forced launchctl restart
|
||||
-> SIGTERM reached the uv/Uvicorn generation
|
||||
-> Uvicorn waited without a configured graceful-shutdown deadline
|
||||
-> old child retained the singleton flock during active LAB work
|
||||
-> launchd observed its wrapper transition and attempted a replacement
|
||||
-> replacement failed closed on the singleton lease
|
||||
-> port 8000 remained unavailable until the old generation was killed
|
||||
```
|
||||
|
||||
The Codex agent caused the outage. The backend did not spontaneously fall over.
|
||||
|
||||
## Code and runtime audit
|
||||
|
||||
The audit covered Python service startup/shutdown, ASGI lifespan cleanup,
|
||||
thread joins, subprocess calls, the LaunchAgent declaration and operator
|
||||
restart paths under `src/k1link` and `scripts`.
|
||||
|
||||
| Finding | Severity | State | Resolution |
|
||||
| --- | --- | --- | --- |
|
||||
| MC-LIFE-001: Uvicorn graceful drain had no deadline | P0 | fixed | `timeout_graceful_shutdown=10` |
|
||||
| MC-LIFE-002: PID-only launchd supervision could not detect a live unhealthy service | P0 | fixed | exact-health self-watchdog, three-failure gate, TERM then KILL |
|
||||
| MC-LIFE-003: forced restart did not prove old label/process release | P0 | fixed | SHA-bound plan/apply, full `bootout` disappearance wait, health acceptance and rollback |
|
||||
| MC-LIFE-004: dependency resolution could mutate or delay a recovery launch | P1 | fixed | canonical launcher uses `uv run --no-sync` |
|
||||
| MC-LIFE-004A: launcher parent exit could leave its Python child generation | P1 | fixed | `AbandonProcessGroup=false` makes launchd own the complete group |
|
||||
| MC-LIFE-005: ASGI/plugin close functions can individually block | P1 | bounded externally | Uvicorn 10 s, watchdog 12 s escalation and launchd 20 s deadline bound the whole generation |
|
||||
| MC-LIFE-006: self-health state previously had no separate durable evidence | P1 | fixed | private rotating JSONL watchdog journal |
|
||||
| MC-LIFE-007: three unbounded joins exist in the offline E33 qualification runner | P2 | isolated | daemon-only offline worker path; not imported or executed by the backend lifecycle |
|
||||
| MC-LIFE-008: artifact-build/guardrail scripts contain subprocess calls without local deadlines | P3 | isolated | developer/CI paths only; not service-reachable and cannot hold port 8000 |
|
||||
|
||||
All subprocess calls reachable through the backend probes and compute-network
|
||||
control paths already carry explicit deadlines. Service-owned joins found in
|
||||
the active backend, camera, viewer, preparation, LiDAR-shadow, simulation and
|
||||
protocol lifecycles are bounded. The remaining no-timeout calls identified by
|
||||
the syntax scan are offline artifact/qualification tooling, not request or
|
||||
lifespan paths.
|
||||
|
||||
The direct `.venv/bin/k1link` LaunchAgent entrypoint was also tested and
|
||||
rejected by macOS with `EPERM` while reading `.venv/pyvenv.cfg` below the
|
||||
`Downloads` privacy boundary. The apply tool restored the previous plist and
|
||||
health. The accepted declaration therefore retains the already-authorized
|
||||
Homebrew `uv` boundary and disables syncing; it does not weaken macOS privacy
|
||||
controls.
|
||||
|
||||
## Recovery and resume semantics
|
||||
|
||||
`KeepAlive` now restores a dead process. The self-watchdog converts a
|
||||
live-but-unhealthy event-loop/application generation into a bounded process
|
||||
exit so `KeepAlive` can act. Startup reconstructs read-only plugin runtimes,
|
||||
catalogs and background reconciliation from durable artifacts.
|
||||
|
||||
Auto-resume is intentionally selective:
|
||||
|
||||
- idempotent read/catalog/preparation reconciliation restarts automatically;
|
||||
- browser WebSockets reconnect to the new generation;
|
||||
- interrupted physical acquisition is reconciled or marked interrupted from
|
||||
durable ledgers;
|
||||
- physical commands, acquisition continuation, navigation and actuation are
|
||||
never silently resumed.
|
||||
|
||||
This distinction prevents availability recovery from becoming an authority
|
||||
escalation.
|
||||
|
||||
## Qualification evidence
|
||||
|
||||
Focused lifecycle/perception tests passed before deployment. The installed
|
||||
LaunchAgent accepted exact health with watchdog enabled and a 20-second exit
|
||||
deadline. A controlled termination of the complete Mission Core process group
|
||||
produced a new LaunchAgent PID and exact health in 26.588 seconds without
|
||||
manual intervention. A separate `SIGKILL` crash injection changed PID/PGID
|
||||
`42895` to PID `42942`, restored exact health in 22.996 seconds and left no old
|
||||
group residue.
|
||||
|
||||
The local qualification does not make the current Mac LaunchAgent an onboard
|
||||
deployment artifact. The eventual onboard init declaration must independently
|
||||
prove boot start, crash restart, health-hang restart, power-loss recovery,
|
||||
bounded shutdown, single-generation fencing and fail-closed physical-state
|
||||
reconciliation.
|
||||
@@ -0,0 +1,110 @@
|
||||
# Mission Core local service recovery
|
||||
|
||||
## Scope
|
||||
|
||||
This runbook owns the single local Mission Core backend at
|
||||
`http://127.0.0.1:8000`. It covers startup, health supervision, bounded
|
||||
shutdown and recovery on the current macOS operator station. It does not grant
|
||||
physical acquisition, actuation, navigation or safety authority.
|
||||
|
||||
The canonical LaunchAgent label is:
|
||||
|
||||
```text
|
||||
com.nodedc.mission-core.local
|
||||
```
|
||||
|
||||
Do not start a second backend on another port. Do not use
|
||||
`launchctl kickstart -k` for this service: it combines termination and restart
|
||||
without proving that the old process group has released the singleton lease.
|
||||
|
||||
## Recovery contract
|
||||
|
||||
The installed LaunchAgent and application form one bounded recovery ladder:
|
||||
|
||||
1. `launchd` starts one `uv run --no-sync k1link serve` process group with
|
||||
`RunAtLoad=true`, `KeepAlive=true`, `AbandonProcessGroup=false` and a
|
||||
five-second throttle.
|
||||
2. Mission Core starts a private self-health thread after acquiring the
|
||||
singleton backend lease.
|
||||
3. After the 45-second cold-start grace, three consecutive failed exact
|
||||
`/api/health` probes request `SIGTERM` for the complete process group.
|
||||
4. Uvicorn stops accepting work and has ten seconds to drain active requests
|
||||
and ASGI lifespan work.
|
||||
5. If the service remains alive, the watchdog escalates to `SIGKILL` after
|
||||
twelve seconds. `launchd` also owns a 20-second exit deadline.
|
||||
6. `launchd` starts a fresh process group. Startup reconstructs plugin
|
||||
runtimes and the recording-preparation reconciler from durable state.
|
||||
|
||||
The watchdog journal is private, bounded and rotated:
|
||||
|
||||
```text
|
||||
.runtime/mission-core/service-watchdog.jsonl
|
||||
.runtime/mission-core/service-watchdog.jsonl.1
|
||||
```
|
||||
|
||||
Physical operations are deliberately not resumed from an assumed state.
|
||||
Interrupted preparation and catalog work is reconciled from durable evidence;
|
||||
physical acquisition, commands and actuation remain fail-closed and require a
|
||||
new confirmed authority transition.
|
||||
|
||||
## Plan and apply
|
||||
|
||||
Always plan from the repository root before changing the installed agent:
|
||||
|
||||
```bash
|
||||
uv run python scripts/manage_mission_core_launch_agent.py plan \
|
||||
--repository-root "$PWD"
|
||||
```
|
||||
|
||||
Copy the exact `current_sha256` and `desired_sha256` from that output into the
|
||||
apply command:
|
||||
|
||||
```bash
|
||||
uv run python scripts/manage_mission_core_launch_agent.py apply \
|
||||
--repository-root "$PWD" \
|
||||
--expected-current-sha256 <current-sha256> \
|
||||
--expected-desired-sha256 <desired-sha256>
|
||||
```
|
||||
|
||||
Apply writes a mode-0600 backup below
|
||||
`.runtime/mission-core/launch-agent-backups`, atomically replaces the plist,
|
||||
waits until `bootout` has fully removed the old label, bootstraps the new
|
||||
declaration and accepts only the exact Mission Core health document. Any
|
||||
failure restores the previous plist and repeats the same health acceptance.
|
||||
|
||||
Read-only status:
|
||||
|
||||
```bash
|
||||
uv run python scripts/manage_mission_core_launch_agent.py status \
|
||||
--repository-root "$PWD"
|
||||
```
|
||||
|
||||
## Acceptance after recovery
|
||||
|
||||
The service is recovered only when all of the following are true:
|
||||
|
||||
- `launchctl print gui/$(id -u)/com.nodedc.mission-core.local` reports
|
||||
`state = running`;
|
||||
- the arguments include `uv run --no-sync k1link serve`;
|
||||
- the environment contains `MISSIONCORE_SERVICE_WATCHDOG => 1`;
|
||||
- the launchd exit timeout is 20 seconds;
|
||||
- `GET http://127.0.0.1:8000/api/health` returns HTTP 200 with
|
||||
`ok=true`, `status=ok` and
|
||||
`service=mission-core-control-plane`;
|
||||
- the watchdog journal contains `watchdog-started` for the current child PID;
|
||||
- there is only one `uv` parent and one Mission Core Python child in their
|
||||
exact process group.
|
||||
|
||||
## 2026-08-25 recovery qualification
|
||||
|
||||
The reviewed declaration SHA-256 was
|
||||
`80fca5ec6bdab21f11a544d8dbee65a35f73a6c34752c6a48e9a1181e5da256a`.
|
||||
A controlled `SIGTERM` of the exact Mission Core process group changed the
|
||||
LaunchAgent PID from `40866` to `40957`; exact health recovered automatically
|
||||
in `26.588` seconds without a manual start. After the explicit
|
||||
`AbandonProcessGroup=false` fence was installed, a controlled `SIGKILL` of PID
|
||||
and PGID `42895` produced a new LaunchAgent PID `42942` and exact health in
|
||||
`22.996` seconds, with no old process-group residue. This is local
|
||||
operator-station evidence only. An onboard Linux deployment must express the
|
||||
same contract in its init system and pass its own power-loss, crash, hang and
|
||||
durable-state reconciliation qualification.
|
||||
Reference in New Issue
Block a user