feat(map): add cache-first Cesium gateway and AMD egress

2026-07-16 02:23:56 +03:00
parent 59e9c92415
commit 7b55d887bc
31 changed files with 4956 additions and 167 deletions
@@ -0,0 +1,770 @@
+# DC AMD Proxy — production operations and recovery canon
+
+This document is the reproducible operations contract for the restricted
+Cesium/Bing egress path through the neighbouring Windows AMD workstation. It
+describes the implementation that is in source now. It is not a proposal for
+turning the NAS or the workstation into a general-purpose proxy.
+
+The two non-negotiable boundaries are:
+
+1. the NAS keeps its own routes, DNS, VPN, Tailscale and Internet connectivity
+   unchanged;
+2. only approved Cesium/Bing HTTPS traffic may leave through the AMD host.
+
+Do not solve an incident by adding a system proxy, default route, transparent
+NAT, browser-visible token, permissive CONNECT endpoint, `0.0.0.0` Windows
+bind, or direct NAS provider fallback.
+
+## 1. Canonical architecture
+
+```text
+Browser
+  -> Foundry BFF
+  -> Platform Map Gateway
+       |-- warm object: NAS live/offline cache -> browser
+       `-- cold/refresh object:
+             HTTP + x-proxy-token
+             -> NAS  <NAS_LAN_IP>:<NAS_PROXY_PORT>  dc-amd-proxy
+                  -> authenticated HTTP CONNECT
+                  -> AMD <AMD_LAN_IP>:<CONNECTOR_PORT> dc-amd-connector
+                       -> AMD/Docker DNS
+                       -> Windows VPN/default egress route
+                       -> approved Cesium/Bing host:443
+                  <- end-to-end provider TLS is terminated by dc-amd-proxy
+             <- streamed provider response
+       -> browser and, when enabled, atomic NAS cache fill
+```
+
+The AMD connector is a byte-forwarding CONNECT boundary. It does not parse an
+Ion bearer and cannot read the provider TLS stream. `dc-amd-proxy` establishes
+TLS through that tunnel, validates the provider certificate and SNI, follows
+only approved redirects, and returns the response to Map Gateway.
+
+This distinction is operationally important:
+
+- a **warm cache hit does not traverse AMD, VPN or Cesium**;
+- a **cold miss or refresh does traverse AMD and depends on the workstation's
+  Docker Desktop, DNS, routing and VPN**;
+- objects already cached on NAS must remain usable when AMD or its VPN is
+  unavailable;
+- `dc-amd-proxy` is not in the browser-to-NAS cache read path.
+
+### Current components and ownership
+
+| Component | Runtime | Canonical state | Responsibility |
+| --- | --- | --- | --- |
+| Map Gateway | Synology Docker | `/volume1/docker/nodedc-platform/platform/services/map-gateway` | Provider credential scope, cache-first reads, streaming cache fill and private egress calls |
+| `dc-amd-proxy` | Synology Docker | `/volume1/docker/dc-amd-proxy` | Request authorization, target allowlist, CONNECT/TLS pool, redirect/retry policy and telemetry |
+| Proxy pair state | Synology bind mount | `/volume1/docker/dc-amd-proxy/runtime/connector-access` | Paired connector credential; persistent across container recreation |
+| Map egress credential | Synology root-owned file | `/volume1/docker/nodedc-platform/secrets/map-egress-proxy-token` | Authenticates Map Gateway to `dc-amd-proxy` |
+| `dc-amd-connector` | AMD Windows Docker Desktop | `C:\NODEDC\dc-amd-connector` by default | Restricted authenticated CONNECT to approved hosts on port 443 |
+| Connector credential | AMD local file | `C:\NODEDC\dc-amd-connector\runtime\connector-access` | Generated locally once; never printed or stored in source |
+| Docker Desktop launcher | Current Windows user's Startup folder | `NODE.DC DC AMD Connector - Docker Desktop.cmd` | Starts Docker Desktop after that user signs in |
+
+Current source defaults are NAS listener port `8790`, AMD connector port
+`8791`, eight active sockets and four idle sockets per origin. IP addresses
+are deployment values and must be treated as configuration, not universal
+constants.
+
+## 2. Trust, secrets and traffic boundaries
+
+There are three different credentials. They are not interchangeable.
+
+1. The Cesium Ion credential belongs to Map Gateway provider configuration.
+   It must not be placed in Foundry browser code, an artifact or this proxy's
+   Compose file.
+2. The Map egress credential authenticates Map Gateway to `dc-amd-proxy` using
+   `x-proxy-token`. The canonical runner synchronizes it into the root-owned
+   file mounted read-only into the two services.
+3. The connector credential authenticates `dc-amd-proxy` to the Windows
+   connector using `Proxy-Authorization: Bearer ...`. The Windows installer
+   generates it locally; the one-time pair request stores the same value in
+   the NAS runtime bind mount.
+
+No command in this runbook prints, copies to chat, or accepts a secret as a
+command-line argument.
+
+### Current runner dependency that must not be mistaken for a traffic route
+
+The deploy runner currently seeds the shared Map egress credential from the
+root-owned `PROXY_TOKEN` in `/volume1/docker/proxy-contur/.env`. Therefore a
+valid, safely permissioned historical `proxy-contur` environment file is a
+**deploy-time precondition** for `dc-amd-proxy` and Map Gateway applies.
+
+This does **not** mean Cesium traffic passes through `proxy-contur`. Runtime
+traffic goes directly from Map Gateway to `dc-amd-proxy`, then to the AMD
+connector. Removing this legacy secret-seeding dependency is an explicit
+evolution item later in this document.
+
+### NAS network boundary
+
+`dc-amd-proxy` uses `network_mode: host` so the one-time pair endpoint can see
+the real AMD source address. Synology's published-port bridge path would hide
+that source behind Docker NAT. Host mode does not make the service broad:
+
+- the process binds only `<NAS_LAN_IP>:<NAS_PROXY_PORT>`;
+- `/api/pair` accepts only `<AMD_LAN_IP>` and only while unpaired;
+- `/proxy/cesium/fetch` accepts only `GET`/`HEAD`, requires the private Map
+  egress credential and validates the complete target URL;
+- `/healthz` and `/status` expose safe state/counters but no credentials or
+  target URLs;
+- the container is read-only, drops all capabilities, enables
+  `no-new-privileges`, and has only a small no-exec tmpfs plus the two explicit
+  bind mounts.
+
+The current Map Gateway URL is a NAS-LAN address, not a public hostname and
+not a Docker service discovery name.
+
+### Windows network boundary
+
+The connector publishes only `<AMD_LAN_IP>:<CONNECTOR_PORT>`. The installer
+creates one inbound Windows Firewall rule with all of these restrictions:
+
+- TCP only;
+- local address `<AMD_LAN_IP>`;
+- local port `<CONNECTOR_PORT>`;
+- remote address `<NAS_LAN_IP>`;
+- no bind to the VPN adapter and no bind to all interfaces.
+
+The connector accepts only authenticated `CONNECT <allowed-host>:443`.
+Ordinary forward-proxy HTTP methods, other ports, IP literals and arbitrary
+hosts are rejected. Its container is read-only, drops capabilities and is
+bounded to one CPU, 256 MB memory, 64 processes and 4096 file descriptors.
+These are circuit breakers for a shared production workstation, not bandwidth
+throttles.
+
+The installer does not change the Windows default route, DNS, system proxy,
+VPN configuration or unrelated firewall rules. The connector's DNS lookup and
+TCP connection occur inside Docker Desktop on the AMD host and therefore use
+the egress path exposed to Docker by Windows/VPN.
+
+### Exact provider allowlist
+
+Both NAS proxy and Windows connector currently allow only:
+
+- `api.cesium.com`
+- `assets.ion.cesium.com`
+- `dev.virtualearth.net`
+- `ecn.t0.tiles.virtualearth.net`
+- `ecn.t1.tiles.virtualearth.net`
+- `ecn.t2.tiles.virtualearth.net`
+- `ecn.t3.tiles.virtualearth.net`
+
+Adding a provider hostname requires one reviewed change to both allowlists,
+updated smoke coverage, a new Windows connector package, a new NAS app-overlay
+artifact and the normal plan/apply acceptance. Never add a wildcard such as
+`*.cesium.com` or `*.virtualearth.net`.
+
+## 3. Request lifecycle
+
+For a cache miss, the sequence is:
+
+1. Map Gateway checks the NAS cache before acquiring provider credentials or
+   contacting egress.
+2. Map Gateway calls
+   `http://<NAS_LAN_IP>:<NAS_PROXY_PORT>/proxy/cesium/fetch?url=<encoded-url>`
+   and authenticates with the private Map egress credential.
+3. `dc-amd-proxy` validates method, URL scheme, host, port and credentials.
+4. A per-origin HTTP/1.1 keep-alive agent either reuses an existing TLS tunnel,
+   opens a new one, or queues behind the bounded active-socket limit.
+5. For a new connection, the NAS opens an authenticated CONNECT request to
+   `<AMD_LAN_IP>:<CONNECTOR_PORT>`.
+6. The connector validates its credential, exact hostname and port 443, then
+   resolves/connects through the AMD network/VPN.
+7. The NAS completes TLS over the byte tunnel and validates the provider
+   certificate.
+8. Response headers and body stream back to Map Gateway. Client cancellation
+   propagates through queue, CONNECT, TLS, header wait, redirect drain, retry
+   and response streaming.
+9. Map Gateway can stream the first bytes to the caller while its independent
+   branch completes an atomic cache fill. A failed/partial fill must not become
+   a valid cache object.
+
+Provider headers forwarded upstream are deliberately narrow: `Accept`,
+`Range`, `If-None-Match`, `If-Modified-Since`, scoped provider authorization
+and the Map referer. `Accept-Encoding` is set to `identity`. Hop-by-hop headers
+are not forwarded back downstream.
+
+### Credential scope across redirects
+
+- same-origin redirects retain the provider `Authorization` header;
+- cross-origin redirects remove `Authorization` before the next request;
+- every redirect target is parsed again through the exact HTTPS/443/host
+  allowlist;
+- redirect bodies are drained with abort and idle-timeout protection;
+- the chain is bounded and terminates with
+  `amd_upstream_redirect_limit` rather than looping indefinitely.
+
+This is why the Ion API bearer cannot leak when the API redirects to the
+assets or Bing origin.
+
+### Pool, retry and timeout policy
+
+| Control | Current default | Meaning |
+| --- | ---: | --- |
+| Active sockets | 8 per origin | Maximum concurrent provider tunnels for each approved origin |
+| Idle sockets | 4 per origin | Reusable warm TLS tunnels retained for each origin |
+| TCP keep-alive initial delay | 30 seconds | Socket keep-alive probe setting; actual idle lifetime is also controlled by provider/server socket closure |
+| Connector/TLS timeout | 20 seconds | Bounds NAS-to-AMD CONNECT and provider TLS setup |
+| Upstream header timeout | 30 seconds | Bounds one attempt until provider response headers |
+| Response/redirect body idle timeout | 30 seconds | Resets on body progress; does not cap a healthy long stream |
+| Slow request threshold | 2 seconds | Emits safe slow-request telemetry; does not cancel the request |
+| Windows connector upstream connect timeout | 20 seconds | Bounds AMD-to-provider TCP connect |
+| Windows connector tunnel idle timeout | 90 seconds | Closes a tunnel with no traffic |
+
+The NAS duration values accept 1–120 seconds and pool sizes accept 1–32. Do
+not increase them to hide a route/VPN problem: first interpret queue,
+connection, TTFB and body timings separately.
+
+There is at most one transport retry. It is allowed only when all conditions
+are true:
+
+- method is idempotent `GET` or `HEAD`;
+- the failed attempt used a previously reused keep-alive socket;
+- failure is `ECONNRESET`, `EPIPE`, `ETIMEDOUT` or `ECONNABORTED`;
+- the request has not already retried and the client has not aborted.
+
+A failure on a newly opened socket is not blindly replayed. A retry may open a
+replacement socket after the stale reused socket is discarded. Both attempts
+are included in telemetry and terminal accounting.
+
+### Runtime configuration reference
+
+All changes to these values require a reviewed Compose/source artifact; do not
+inject an ad-hoc live container environment.
+
+| NAS proxy variable | Default | Contract |
+| --- | --- | --- |
+| `PORT` | `8790` | Internal/listener port |
+| `DC_AMD_PROXY_BIND_ADDRESS` | deployment NAS LAN IP | Exact listener address; IPv4 only |
+| `DC_AMD_CONNECTOR_HOST` | deployment AMD LAN IP | Stable connector LAN address; IPv4 only |
+| `DC_AMD_CONNECTOR_PORT` | `8791` | Windows published connector port |
+| `DC_AMD_PAIR_ALLOWED_SOURCE` | deployment AMD LAN IP | Only source allowed to perform first pair |
+| `DC_AMD_CONNECTOR_TOKEN_FILE` | `/var/lib/dc-amd-proxy/connector-access` | Persistent paired connector state |
+| `DC_AMD_MAP_EGRESS_TOKEN_FILE` | `/run/nodedc-secrets/map-egress-proxy-token` | Read-only Map Gateway authentication file |
+| `DC_AMD_CONNECT_TIMEOUT_SECONDS` | `20` | CONNECT and TLS setup timeout |
+| `DC_AMD_UPSTREAM_TIMEOUT_SECONDS` | `30` | Per-attempt response-header timeout |
+| `DC_AMD_BODY_IDLE_TIMEOUT_SECONDS` | `30` | Progress-based response/redirect body timeout |
+| `DC_AMD_POOL_MAX_SOCKETS` | `8` | Active sockets per origin |
+| `DC_AMD_POOL_MAX_FREE_SOCKETS` | `4` | Idle sockets per origin |
+| `DC_AMD_SLOW_REQUEST_SECONDS` | `2` | Safe slow-event threshold |
+
+The Windows Compose `.env` contains only `AMD_CONNECTOR_BIND_IP` and
+`AMD_CONNECTOR_PORT`. The connector server also supports bounded
+`AMD_CONNECTOR_CONNECT_TIMEOUT_SECONDS` (default 20) and
+`AMD_CONNECTOR_IDLE_TIMEOUT_SECONDS` (default 90); changing them should be a
+versioned connector-package change.
+
+## 4. Reproducible initial installation
+
+Replace every angle-bracket placeholder before executing a command. Do not
+paste a credential into any placeholder.
+
+### 4.1 Preconditions
+
+- AMD workstation has a stable LAN address reserved in DHCP or statically
+  configured.
+- NAS and AMD can reach each other on the trusted LAN.
+- Docker Desktop is installed and its Linux engine is running.
+- The intended Windows user can sign in after a reboot.
+- The VPN is connected and exposes its route to Docker Desktop workloads.
+- The reviewed Windows connector package contains a valid `SHA256SUMS`.
+- The canonical NAS runner is installed at
+  `/usr/local/sbin/nodedc-deploy`.
+- No other host is using the selected connector LAN address/port.
+
+### 4.2 Install the Windows connector
+
+Extract the reviewed connector package to a temporary local folder. Open
+PowerShell **as Administrator**, enter that folder and run:
+
+```powershell
+powershell.exe -NoProfile -ExecutionPolicy Bypass -File ".\install.ps1" `
+  -Destination "C:\NODEDC\dc-amd-connector" `
+  -NasAddress "<NAS_LAN_IP>" `
+  -BindAddress "<AMD_LAN_IP>" `
+  -Port <CONNECTOR_PORT> `
+  -EnableAutoStart $true
+```
+
+The installer is fail-closed. It verifies `SHA256SUMS`, refuses to overwrite an
+existing destination or firewall rule, generates a random connector credential
+without printing it, builds/starts the container, waits for `healthy`, creates
+the NAS-only firewall rule and writes the current-user Docker Desktop Startup
+launcher. On failure it removes only the state it created.
+
+Verify locally without displaying the credential:
+
+```powershell
+Set-Location "C:\NODEDC\dc-amd-connector"
+docker compose ps
+docker inspect dc-amd-connector --format '{{.State.Status}} health={{if .State.Health}}{{.State.Health.Status}}{{end}} restart={{.HostConfig.RestartPolicy.Name}}'
+docker compose logs --tail 100
+```
+
+Expected state is `running health=healthy` and restart policy
+`unless-stopped`.
+
+This safe CONNECT probe reads the credential only inside the container and
+prints only the HTTP status line:
+
+```powershell
+docker exec dc-amd-connector node -e "const fs=require('fs'),net=require('net');const t=fs.readFileSync('/run/dc-amd-secrets/connector-access','utf8').trim();const s=net.connect(8791,'127.0.0.1',()=>s.write('CONNECT api.cesium.com:443 HTTP/1.1\r\nHost: api.cesium.com:443\r\nProxy-Authorization: Bearer '+t+'\r\n\r\n'));s.once('data',d=>{const v=d.toString('ascii');console.log(v.split('\r\n')[0]);s.destroy();process.exit(v.startsWith('HTTP/1.1 200')?0:1)});s.on('error',e=>{console.error(e.message);process.exit(1)})"
+```
+
+Expected output is `HTTP/1.1 200 Connection Established`. This proves that the
+connector can currently reach the API host; it does not validate an Ion token
+or prove the VPN exit country.
+
+### 4.3 Build and transfer the NAS artifact
+
+From a clean `platform` repository checkout on the trusted build Mac:
+
+```bash
+node infra/deploy-runner/build-dc-amd-proxy-artifact.mjs \
+  dc-amd-proxy-<CHANGE_SLUG>-<YYYYMMDD>-<NNN>
+
+cd infra/deploy-artifacts
+shasum -a 256 nodedc-dc-amd-proxy-<CHANGE_SLUG>-<YYYYMMDD>-<NNN>.tgz \
+  > nodedc-dc-amd-proxy-<CHANGE_SLUG>-<YYYYMMDD>-<NNN>.tgz.sha256
+```
+
+Copy exactly the `.tgz` and `.tgz.sha256` through SMB into:
+
+```text
+/volume1/docker/nodedc-deploy/inbox
+```
+
+The builder includes only `Dockerfile`, `README.md`, `docker-compose.yml`,
+`package.json` and `server.mjs`. Runtime, `.env`, credentials, logs, Windows
+packages and this operations document are not deploy payload members.
+
+### 4.4 Canonical NAS plan/apply
+
+On Synology, from the canonical inbox:
+
+```bash
+cd /volume1/docker/nodedc-deploy/inbox
+
+sha256sum -c \
+  nodedc-dc-amd-proxy-<CHANGE_SLUG>-<YYYYMMDD>-<NNN>.tgz.sha256
+
+sudo /usr/local/sbin/nodedc-deploy verify-install
+
+sudo /usr/local/sbin/nodedc-deploy plan \
+  /volume1/docker/nodedc-deploy/inbox/nodedc-dc-amd-proxy-<CHANGE_SLUG>-<YYYYMMDD>-<NNN>.tgz
+```
+
+Accept the plan only when all are true:
+
+- `component=dc-amd-proxy`
+- `type=app-overlay`
+- `payload_root=/volume1/docker/dc-amd-proxy`
+- `services=dc-amd-proxy`
+- `runtime_secret=runner-synced:.../map-egress-proxy-token`
+- `runtime_state=runner-prepared:/volume1/docker/dc-amd-proxy/runtime`
+- `state=new`
+- the file list is exactly the five allowlisted source files.
+
+Then apply the exact planned filename; never use “latest” discovery:
+
+```bash
+sudo /usr/local/sbin/nodedc-deploy apply \
+  /volume1/docker/nodedc-deploy/inbox/nodedc-dc-amd-proxy-<CHANGE_SLUG>-<YYYYMMDD>-<NNN>.tgz
+```
+
+The runner preserves/prepares the private pair-state directory, synchronizes
+the Map egress credential, creates a source backup, builds with Compose,
+force-recreates only `dc-amd-proxy`, waits for container health, moves the
+artifact to `applied`, and records the result. A new service should report
+`state=awaiting_pair`.
+
+### 4.5 Pair Windows to NAS
+
+Pair only after NAS status says `awaiting_pair`. Place the reviewed pairing
+tool on the AMD machine and run it from an elevated PowerShell window:
+
+```powershell
+powershell.exe -NoProfile -ExecutionPolicy Bypass `
+  -File ".\pair-dc-amd-proxy.ps1" `
+  -NasAddress "<NAS_LAN_IP>" `
+  -Port <NAS_PROXY_PORT>
+```
+
+Expected output:
+
+```text
+AMD connector paired with NAS. No secret value was displayed.
+```
+
+The pair call is a one-time HTTP LAN handoff with the Windows proxy explicitly
+disabled. Its security boundary is the trusted LAN plus exact AMD source IP.
+After the first successful pair, later attempts return `409 already_paired`.
+
+Verify on NAS:
+
+```bash
+curl -fsS http://<NAS_LAN_IP>:<NAS_PROXY_PORT>/status
+
+sudo /usr/local/bin/docker inspect dc-amd-proxy \
+  --format 'status={{.State.Status}} health={{if .State.Health}}{{.State.Health.Status}}{{end}} network={{.HostConfig.NetworkMode}} read_only={{.HostConfig.ReadonlyRootfs}} restart={{.HostConfig.RestartPolicy.Name}}'
+```
+
+Expected properties are `state=paired`, `forwarding=amd_connector_only`,
+`directEgress=false`, container `healthy`, network `host`, read-only `true`,
+and restart `unless-stopped`.
+
+Finally verify one real Map Gateway request using the product health workflow.
+Do not construct a shell command that reads or prints either private runtime
+credential.
+
+For a first installation, switch Map Gateway to
+`http://<NAS_LAN_IP>:<NAS_PROXY_PORT>` only through a separate reviewed
+Platform artifact and only after the proxy is paired and its tunnel test
+passes. This separation keeps a failed connector bootstrap from breaking an
+already-working provider route.
+
+## 5. Restart and reboot behavior
+
+### Synology
+
+The NAS container has `restart: unless-stopped`. A Docker daemon or NAS reboot
+restarts it. Pair state survives because `./runtime` is a bind mount outside
+the container. The Map egress credential is a separate root-owned bind mount.
+
+If an operator intentionally runs `docker compose down`, the container is
+removed and restart policy cannot recreate it. Restore it from the canonical
+source directory:
+
+```bash
+cd /volume1/docker/dc-amd-proxy
+sudo /usr/local/bin/docker compose -p dc-amd-proxy up -d --build --no-deps dc-amd-proxy
+```
+
+Use that command only for recovery of the already-deployed version. Source
+changes still require a runner artifact.
+
+### Windows AMD host
+
+Docker Desktop on the WSL 2 backend is not guaranteed to be an unattended
+Windows boot service. The installed launcher starts it when the configured
+Windows user signs in. After Docker Engine starts, `restart: unless-stopped`
+restores the existing connector container.
+
+Therefore the actual recovery sequence is:
+
+1. Windows boots;
+2. the configured user signs in;
+3. Startup launches Docker Desktop;
+4. Docker Engine becomes ready;
+5. the existing connector container restarts;
+6. the VPN must be connected and usable from Docker;
+7. cold Cesium requests recover.
+
+Before user sign-in, or while Docker/VPN is unavailable, live misses fail and
+Map Gateway must rely on existing NAS cache. If the connector container was
+removed with `docker compose down`, restart policy alone cannot recreate it;
+run from elevated PowerShell:
+
+```powershell
+Set-Location "C:\NODEDC\dc-amd-connector"
+docker compose up -d --build
+```
+
+## 6. Routine operations and observability
+
+### NAS commands
+
+```bash
+curl -fsS http://<NAS_LAN_IP>:<NAS_PROXY_PORT>/status
+
+sudo /usr/local/bin/docker inspect dc-amd-proxy \
+  --format 'status={{.State.Status}} health={{if .State.Health}}{{.State.Health.Status}}{{end}} restart={{.HostConfig.RestartPolicy.Name}}'
+
+sudo /usr/local/bin/docker logs --tail 200 dc-amd-proxy
+
+cd /volume1/docker/dc-amd-proxy
+sudo /usr/local/bin/docker compose -p dc-amd-proxy ps
+```
+
+### Windows commands
+
+```powershell
+Set-Location "C:\NODEDC\dc-amd-connector"
+docker compose ps
+docker compose logs --tail 200
+docker inspect dc-amd-connector --format '{{.State.Status}} health={{if .State.Health}}{{.State.Health.Status}}{{end}} restart={{.HostConfig.RestartPolicy.Name}}'
+```
+
+### What `/status` means
+
+`/healthz` and `/status` currently return the same safe operational body. They
+prove process/local state, not end-to-end provider reachability.
+
+Important fields:
+
+- `state`: `awaiting_pair` or `paired`;
+- `forwarding`: disabled until paired, then `amd_connector_only`;
+- `directEgress`: must always be `false`;
+- `pool.activeSockets`, `idleSockets`, `queuedRequests`, `maxQueuedRequests`;
+- `pool.byOrigin`: safe active/idle/queued counts per allowlisted origin;
+- `metrics.requests`, `terminalRequests`, `inFlightRequests`;
+- `completedResponses`, `failedRequests`, `clientAborts`, `streamFailures`;
+- `openedTunnels`, `tunnelFailures`, `reusedSocketRequests`, `retries`;
+- complete and partial byte counts;
+- average/max queue, connection, TTFB, attempt, retry and total duration;
+- per-host counters and the last short transport error.
+
+At all times:
+
+```text
+requests = terminalRequests + inFlightRequests
+```
+
+At quiescence, `inFlightRequests` should return to zero. A violation indicates
+a lifecycle/accounting defect. Counters are process-local and reset after a
+container restart; pair state does not reset.
+
+Interpret latency by stage:
+
+| Symptom | Likely stage |
+| --- | --- |
+| High `queuedRequests` / `queueMs` | Per-origin eight-socket pool saturated or requests not terminating |
+| High `connectionMs` / tunnel failures | NAS-to-AMD LAN, Windows connector, DNS, CONNECT or TLS setup |
+| Low connection but high `ttfbMs` | VPN/provider latency or provider throttling |
+| Low TTFB but high total duration | Slow/stalled response body or downstream backpressure |
+| Rising `partialBytes` / `streamFailures` | Provider/VPN body died after headers |
+| Rising `clientAborts` | Browser/navigation cancellation or caller timeout; not automatically an upstream failure |
+| Low `reusedSocketRequests` with high `openedTunnels` | Keep-alive churn, route instability or origin fan-out |
+| Retry spikes | Reused sockets are being reset; one bounded recovery is working but transport is unstable |
+
+JSON logs intentionally contain event names, approved hostname, short error
+code and numeric timing only. They must not include a full URL, query string,
+Ion bearer, connector credential or Map egress credential. Normal fast tile
+requests are not logged individually. Useful events include:
+
+- `cesium_egress_tunnel_opened`
+- `cesium_egress_reused_socket_retry`
+- `cesium_egress_retry_completed`
+- `cesium_egress_retry_failed`
+- `cesium_egress_slow_or_failed`
+- `cesium_egress_stream_failed`
+- `cesium_egress_request_failed`
+- `amd_connector_paired`
+
+A warm Map Gateway cache hit should not increase `dc-amd-proxy` request
+counters. If it does, audit the cache-first ordering before tuning the proxy.
+
+## 7. Failure matrix
+
+| Observable result | Probable cause | Safe action |
+| --- | --- | --- |
+| NAS status unavailable | Container stopped/unhealthy or wrong NAS bind address | Inspect container/Compose state and logs; do not change NAS routes |
+| `state=awaiting_pair` | No durable connector pair state | Verify intended AMD host, then run the one-time pairing tool |
+| Pair HTTP 403 `pair_source_not_allowed` | Request did not originate from configured AMD LAN IP | Verify stable AMD IP, NAS host-mode listener and no proxy/NAT in the pair call |
+| Pair HTTP 409 `already_paired` | Pair state already exists | Do not overwrite it; use the controlled migration/rotation procedure |
+| `map_egress_unauthorized` | Map Gateway and NAS proxy do not share the runner-synced Map egress credential | Re-run canonical preflight/deploy investigation; never copy a token into `.env` by hand |
+| `amd_connector_not_paired` | NAS runtime pair file absent/invalid | Pair from the allowed AMD source; inspect ownership only through canonical runner checks |
+| Connector refused/timeout | Windows off, no user sign-in, Docker stopped, firewall/IP mismatch or connector unhealthy | Restore Windows/Docker/connector; warm cache should continue |
+| `amd_connector_rejected` | Connector credential mismatch or CONNECT rejected | Stop; do not retry pairing blindly. Treat as pair-state/migration incident |
+| `cesium_target_not_allowed` | Provider redirected/requested an unreviewed host or wrong scheme/port | Capture hostname safely, review necessity, update both allowlists through source |
+| TLS timeout/certificate error | VPN path, DNS, time, CA or provider interception problem | Verify AMD route/VPN and NAS container time/CA; never disable TLS verification in production |
+| `amd_upstream_timeout` | No provider headers within attempt timeout | Compare connection vs TTFB metrics; verify VPN/provider rather than increasing timeout first |
+| Body/redirect idle timeout | Stream stopped making progress | Check VPN packet loss/provider; partial object must not enter Map cache |
+| Provider HTTP 401/403 | Ion token, asset permission, referer restriction or provider account issue | Diagnose Map Gateway/provider configuration; transport is functioning if status is passed through |
+| Provider HTTP 429 | Provider rate/throughput policy | Reduce cold fan-out and use cache; do not add unbounded sockets/retries |
+| Rendering survives but cold areas fail | Expected cache-first degradation while AMD/VPN is down | Restore egress; preserve existing cache and avoid destructive cache resets |
+| Queue grows and never drains | Saturation, hung lifecycle or insufficient abort propagation | Inspect in-flight/accounting invariant, body idle errors and client aborts before changing pool size |
+| AMD host rebooted and does not recover | No Windows sign-in, Startup launcher missing, Docker not ready, container removed or VPN disconnected | Follow the explicit Windows reboot sequence |
+
+### Important VPN caveat
+
+The connector trusts the route Docker Desktop receives from Windows. It does
+not cryptographically prove that a request used a particular VPN or country.
+If Windows allows direct Internet egress when the VPN is off, the connector
+may use that direct AMD-host route. It will still never fall back to direct NAS
+egress, but that is a different guarantee.
+
+If VPN-only provider egress is mandatory, enforce it on the AMD host/VPN with
+an outbound kill switch scoped to the connector workload/approved destinations
+and test it with VPN on and off. The current installer creates only an inbound
+NAS-only firewall rule; it does not install this outbound policy.
+
+## 8. Migration to another AMD machine
+
+Migration has two independent identities: stable LAN endpoint and connector
+credential. A clean install on another machine creates a new credential, while
+the current NAS pair endpoint is intentionally write-once. Changing only the
+IP in Compose is therefore insufficient.
+
+### Required zero-guesswork migration sequence
+
+1. Keep the old connector active while preparing the replacement.
+2. Assign the replacement a reserved `<NEW_AMD_LAN_IP>` and verify VPN/Docker
+   routing.
+3. Install the reviewed connector package on the replacement using
+   `<NEW_AMD_LAN_IP>` and the same NAS address/connector port.
+4. Verify local connector health and the safe CONNECT probe.
+5. Create a reviewed source change that updates NAS connector host and allowed
+   pair source to `<NEW_AMD_LAN_IP>`.
+6. Add a runner-supported, audited pair-rotation transition that can retire the
+   old NAS pair state and admit exactly one new pair. The transition must back
+   up state, never print either credential, fail closed and be idempotent.
+7. Build, checksum, SMB-transfer, `plan`, and `apply` the exact versioned
+   artifact/runner change through the deploy canon.
+8. Pair from the replacement machine, verify an end-to-end cold request and
+   confirm `directEgress=false` plus clean accounting.
+9. Disable the old connector, then uninstall it only after acceptance.
+
+The current implementation does **not yet provide step 6 as a first-class
+runner operation**. Until it exists, a fresh-secret migration is not fully
+canonical. Do not work around this by deleting
+`/volume1/docker/dc-amd-proxy/runtime/connector-access`, copying the credential
+through chat/SMB, weakening `/api/pair`, or running two active pair sources.
+
+If an urgent migration is required before controlled rotation exists, open an
+Ops change and implement/review that transition first. This limitation is
+preferable to an undocumented secret reset.
+
+To remove a retired connector from Windows after the new path is accepted:
+
+```powershell
+powershell.exe -NoProfile -ExecutionPolicy Bypass `
+  -File "C:\NODEDC\dc-amd-connector\uninstall.ps1" `
+  -Destination "C:\NODEDC\dc-amd-connector"
+```
+
+The uninstaller removes only this container, its NAS-only firewall rule,
+Startup launcher and installation directory. It does not remove Docker Desktop
+or unrelated containers.
+
+## 9. Deploy and rollback canon
+
+### Source/change acceptance before packaging
+
+Run from the service directory in a clean checkout:
+
+```bash
+node --check server.mjs
+npm run test:connection-pool
+docker build --no-cache -t nodedc/dc-amd-proxy:<VERIFY_TAG> .
+git status --short
+```
+
+The smoke suite must cover pool reuse/saturation, aborts in each lifecycle
+stage, exact terminal accounting, one safe retry, body idle timeout, redirect
+credential boundaries and absence of secret markers in logs/status.
+
+Commit source and documentation together. Build an artifact from the commit,
+not from an unknown dirty tree. The normal deployment sequence is:
+
+```text
+source review -> tests -> commit -> artifact -> checksum -> SMB inbox
+-> verify-install -> exact plan -> human plan review -> exact apply
+-> container health -> proxy status -> end-to-end Map acceptance
+```
+
+Never put `.env`, `runtime`, tokens, logs, backup archives, Windows package
+state or shell hooks inside a `dc-amd-proxy` app-overlay.
+
+### What the runner records
+
+Before applying, the runner creates:
+
+```text
+/volume1/docker/nodedc-deploy/backups/<BACKUP_ID>/
+```
+
+with manifest, file lists and `source-before.tgz`. Successful artifacts move
+to `/volume1/docker/nodedc-deploy/applied`; failed artifacts move to
+`/volume1/docker/nodedc-deploy/failed`. Applied and failed JSONL state lives
+under `/volume1/docker/nodedc-deploy/state`.
+
+For this component the current runner does **not** perform generic automatic
+source rollback after a failed post-copy build/health check. The backup is
+evidence/recovery input, not permission for an improvised live extraction.
+
+### Canonical forward rollback
+
+Use a new patch ID and package the known-good service source from a separate
+clean worktree. Do not reset or overwrite the active developer worktree:
+
+```bash
+git worktree add "../NODEDC-rollback" <KNOWN_GOOD_COMMIT>
+cd "../NODEDC-rollback"
+
+node infra/deploy-runner/build-dc-amd-proxy-artifact.mjs \
+  dc-amd-proxy-rollback-<YYYYMMDD>-<NNN>
+
+cd infra/deploy-artifacts
+shasum -a 256 nodedc-dc-amd-proxy-rollback-<YYYYMMDD>-<NNN>.tgz \
+  > nodedc-dc-amd-proxy-rollback-<YYYYMMDD>-<NNN>.tgz.sha256
+```
+
+Transfer the exact pair to the inbox and run the same checksum,
+`verify-install`, exact `plan`, review and exact `apply` sequence. The runtime
+pair-state directory remains outside the source overlay and must be preserved.
+
+If the failed version changed a runtime contract, Compose contract or secret
+format, stop and design an explicit rollback transition instead of assuming a
+source-only forward rollback is safe.
+
+## 10. Capacity and product behavior
+
+The default eight active sockets are **per approved origin**, not global. A
+Cesium session can use more than one origin, but every origin remains bounded.
+Twenty users do not overwrite one another's transport state; they share the
+pool and may queue. Cache objects are owned by Map Gateway, not by this proxy.
+
+The proxy has no tile index, no cache eviction and no “do not overwrite cache”
+switch. Those belong to Map Gateway. It streams bytes and records transport
+telemetry. Consequently:
+
+- proxy/VPN speed affects only cold/refresh acquisition;
+- NAS cache read speed should be independent from AMD/VPN speed;
+- increasing pool size cannot fix slow warm cache reads;
+- cache fill concurrency and capacity policy must be tuned in Map Gateway;
+- a browser reload should reuse NAS cache even if local browser cache is empty.
+
+## 11. Known limitations and evolution path
+
+The current path is production-usable but not yet a general infrastructure
+egress product. Track these changes explicitly rather than accumulating
+one-off live fixes:
+
+1. **Controlled pair rotation/migration.** Add runner-owned one-time rotation,
+   revocation and rollback state; optionally support overlapping old/new
+   credentials for a bounded cutover window.
+2. **Remove the legacy `proxy-contur` secret source.** Generate/manage a
+   dedicated Map egress credential as first-class runner state.
+3. **VPN-route enforcement.** Add a tested Windows outbound kill switch or a
+   dedicated egress appliance so VPN-off cannot become direct AMD egress.
+4. **Unattended host recovery.** Replace user-sign-in-dependent Docker Desktop
+   with a managed Windows service, signed installer or dedicated Linux egress
+   node if 24/7 cold-cache availability is required.
+5. **Pair-channel hardening.** Replace source-IP-only HTTP pairing with a
+   short-lived nonce plus authenticated/encrypted handoff or mutual TLS.
+6. **Two-level health.** Keep local `/healthz`, add a rate-limited synthetic
+   egress readiness check that proves DNS/CONNECT/TLS without consuming or
+   exposing the master Ion credential.
+7. **Durable metrics.** Export the safe counters to the platform metrics stack;
+   process-local `/status` currently resets on restart.
+8. **Central allowlist contract.** Generate identical proxy/connector host
+   policies from one reviewed provider contract and test that they cannot
+   drift.
+9. **Circuit breaking and backoff.** Add per-origin fail-fast state for a dead
+   AMD/VPN path so many cold misses do not occupy all queues until timeout.
+10. **High availability.** Define an explicit active/standby connector model,
+    health-based selection and credential isolation before adding a second
+    workstation.
+11. **Signed Windows distribution/update.** Provide a versioned package builder,
+    signature verification and in-place upgrade/rollback rather than manual
+    archive handling.
+12. **SLOs and load tests.** Establish cache-hit, cold-TTFB, queue-depth,
+    abort-rate and recovery-time objectives using realistic concurrent Map
+    Page sessions.
+
+Until those changes land, preserve the narrow boundary: official allowlisted
+HTTPS only, no direct NAS fallback, no global workstation proxy, no secret in
+source, and every NAS mutation through a reviewed data-only artifact plus the
+canonical runner.