SILVER
/
NODEDC_PLATFORM
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
NODEDC_PLATFORM/infra/deploy-runner/README.md

15 KiB
Raw Blame History

NODE.DC deploy runner

This directory stores the versioned source for the Synology canonical deploy runner.

Live runner:

/usr/local/sbin/nodedc-deploy

Synology staging candidate:

/volume1/docker/nodedc-deploy/runner-install/nodedc-deploy

The runner accepts data-only app-overlay artifacts from:

/volume1/docker/nodedc-deploy/inbox

Supported components in this source:

  • engine
  • launcher
  • platform
  • tasker
  • ops-agents
  • bim-viewer
  • n8n-private-extension
  • module-foundry
  • proxy-contur
  • dc-amd-proxy

n8n-private-extension is a staging-only trust boundary for reviewed offline n8n private-node releases. Its artifact may contain exactly one digest-bound n8n-nodes-ndc release with package.tgz, release.json and rollback.json. The runner validates the inner npm tarball, rejects lifecycle scripts and runtime dependencies, refuses to overwrite an existing release, and seals the installed release root-owned/read-only under:

/volume1/docker/nodedc-platform/n8n-private-extensions/releases/n8n-nodes-ndc/<version>-<sha256-prefix>

This component has no Compose file, service, container mutation or activation side effect. In particular, staging does not make the node visible to n8n. Activation remains an Engine-owned change: mount the reviewed immutable release at /home/node/.n8n/nodes/node_modules/n8n-nodes-ndc, atomically switch between verified releases, restart every n8n process, and accept only after MCP exposes the package-qualified n8n-nodes-ndc.* schemas. The Platform runner cannot cross that boundary and never runs npm install in a live container.

Build a verified offline release artifact:

node infra/deploy-runner/build-n8n-private-extension-artifact.mjs \
  n8n-nodes-ndc-release-YYYYMMDD-NNN

The builder is byte-reproducible and accepts exactly the three reviewed NDC runtime types:

  • n8n-nodes-ndc.ndcDataProductPublish
  • n8n-nodes-ndc.ndcDataProductRead
  • n8n-nodes-ndc.ndcFoundryBinding

Their three opaque capability credential schemas are ndcDataProductWriterApi, ndcDataProductReaderApi and ndcFoundryBindingApi. A node description containing usableAsTool is rejected because n8n 2.3.2 would synthesize an additional *Tool runtime type and violate the exact-three activation contract. Run the positive and negative release-policy suite before publishing an artifact:

PYTHONDONTWRITEBYTECODE=1 \
  python3 infra/deploy-runner/test_n8n_private_extension.py

Release/rollback manifests use schema v2. Before a first activation, the Engine-owned activator must verify and record the current inactive state. That verified_inactive state is an allowed rollback baseline when no previous verified immutable release exists; later upgrades prefer the previous verified release. Rollback never deletes or mutates a staged release.

The historical 0.1.0 release remains immutable and must not be overwritten. Release 0.1.1-994756958861518e is retained as rejected/inactive: its three node descriptions used usableAsTool, so n8n 2.3.2 exposed six NDC runtime types instead of the required three. It must not be activated, overwritten or deleted.

The corrected candidate is package version 0.1.2, built with patch id n8n-nodes-ndc-release-20260716-003. It receives a new digest-bound release directory and remains inert after staging; only a separately reviewed Engine-owned activator may select it after exact MCP schema acceptance.

The paired Engine activation is built by build-engine-n8n-private-extension-artifact.mjs. It deliberately does not copy from or write generated files into the dirty Engine worktree, and it does not build an image. A fresh transition id is mandatory and previously issued ids are rejected:

node infra/deploy-runner/build-engine-n8n-private-extension-artifact.mjs 20260717-004

The builder emits a narrowly scoped Compose override plus a strict transition descriptor. On apply, the runner validates the staged release again, verifies that the running n8n container and the NAS-local 2.3.2 tag resolve to the same immutable image ID, and extracts the package into the root-owned, read-only Engine release tree:

/volume2/nodedc-demo/n8n-private-extensions/releases/n8n-nodes-ndc/0.1.2-05e4b38b14b4a019/package

The override sets N8N_USER_FOLDER=/home/node, which is required because the actual Engine service runs as root while the canonical community package path is below /home/node/.n8n. The live runtime contract remains the successfully deployed generation-003 contract and does not set NODE_PATH. Only the runner's isolated node -e package-loader probe temporarily initializes the dependency tree bundled inside the exact n8n base image; this reproduces n8n's own loader without redefining the live Compose state. The override enables loading but disables reinstall, mounts only the exact release read-only, and uses both Compose pull_policy: never and docker compose up --pull never. No registry access, lifecycle script, database installed_packages row or custom-extension loader is involved.

The runner pins the exact Engine service topology observed in source and rejects an added worker/webhook generation. Only the single actual n8n service is force-recreated with --no-deps; the Postgres service, .n8n data, encryption key and credentials remain intact. The apply gate verifies readiness, the running image/version, sealed mount, loader environment, package-loader node/credential sets, scoped loader logs, restart stability and content-exact pinned Engine MCP catalogs. The runner pins both the complete 434/385 inactive baseline and the reviewed 437/388 activation catalogs, so a same-count substitution of any built-in schema is rejected. Any gate failure after mutation automatically restores the pre-apply catalogs/descriptor and force-recreates the previous verified runtime. Staged, sealed and failed releases are retained. The separate rollback artifact returns the first activation to the verified inactive 434-node/385-credential catalog baseline.

Run both policy suites before publishing the Engine pair:

PYTHONDONTWRITEBYTECODE=1 \
  python3 infra/deploy-runner/test_n8n_private_extension.py
PYTHONDONTWRITEBYTECODE=1 \
  python3 infra/deploy-runner/test_engine_n8n_private_extension.py

For platform artifacts, the allowlist includes the versioned Ontology Core, the frozen legacy Gelios compatibility service, and the provider-neutral External Data Plane sources. The Gelios service remains reproducible only to protect its existing database/workflow; it is not a template for a provider integration. An External Data Plane artifact builds only its image and force-recreates only external-data-plane; the already healthy external-data-plane-postgres container and its Timescale volume are an independent deploy prerequisite and are never selected by an EDP application artifact. The reviewed Compose source pins the Timescale image, named volume and target, internal database-only network, absence of database host ports, healthy dependency, localhost-only EDP bind and the two read-only provisioner/trust mounts in the reviewed Compose source. The runner does not reinterpret version-dependent docker compose config JSON as a second deploy schema. Its canonical enforcement remains the artifact/path allowlist plus hard-coded build command, selected service set, runtime-secret preparation and health acceptance. Post-apply acceptance also requires database=ready; a first-rollout failure removes only the candidate EDP container without volumes and restores the source overlay. It never contains a provider credential, provider endpoint, collection schedule or command transport. Its contract payload is an exact provider-neutral runtime subset; providers/*, mappings, fixtures and tests are excluded and do not trigger an EDP rebuild. Database credentials remain root-owned live .env.synology configuration and must not reuse NODEDC_INTERNAL_ACCESS_TOKEN.

On the first relevant Platform apply, the root-owned runner creates /volume1/docker/nodedc-platform/secrets/external-data-plane-provisioner/token atomically in a dedicated UID/GID 11006 directory (directory 0500, token 0400). It is never an .env value or an artifact member and is mounted read-only only into External Data Plane.

Manual one-time binding issuance and digest-only managed writer ensure are independently disabled by default. The target control-plane operation generates and stores the capability inside native NDC L2 Credentials and sends only its digest to EDP; users and MCP consumers receive only an opaque compatible reference/status. The runner-owned bearer above authenticates only legacy plaintext issuance and is intentionally rejected by managed ensure/revoke. Managed requests use a deployment Ed25519 Engine service key; EDP mounts only the public-key trust directory read-only. On every reviewed Engine apply and on an EDP runtime apply, this runner creates or validates one matching Ed25519 pair: the Engine-only private key is root:root 0400, while the EDP trust copy is root:11006 0440. Public-only crash state, key mismatch, a non-Ed25519 key, symlinks and permissive modes fail closed. plan discloses both paths without printing key material. The private key must not be broadened into an L2 graph, MCP surface, artifact or shared-token boundary.

The reviewed Engine source candidate has dedicated server-derived MCP plan/apply handling for the exact ndcDataProductWriterApi + Data Product Publish tuple. It is separate from the generic HTTP safe-ref path and accepts no caller-provided provider/scope/credential identity, capability, generation or service URL. The production managed flag remains false during staging. In the explicitly confirmed activation window it is set to true immediately before the Platform EDP artifact; runner acceptance then requires EDP /healthz to report managed provisioning enabled. At that point the old Engine still has no signer mount, so the endpoint remains usable only after the separately accepted Engine artifact. Root/UI transfer is emergency self-hosted diagnostics only, not the user journey or acceptance path. A provider-specific daemon remains forbidden.

Build the narrow Engine source artifact with:

node infra/deploy-runner/build-engine-data-product-publish-grant-artifact.mjs \
  engine-data-product-publish-grant-YYYYMMDD-NNN

The builder includes exactly the pinned provider-security catalog, the Publish-grant service, Engine Agent scope/gateway wiring, the existing n8n adapter and the reviewed additive backend runtime overlay. It deliberately excludes base Compose, frontend/dist, runtime data, tests, native credentials, the NDC L2 process and the legacy generic credential-sink route/core. The runner fixes the runtime action to nodedc-backend with --no-deps and --pull never; n8n, nginx app, database services and volumes are not selected. Acceptance requires backend health, the active immutable backend identity and the exact additive mount inventory. Any failure restores the touched source and recreates only the previous verified backend runtime.

Existing Engine Agents created before Publish-grant support store the original nine scopes as the complete developer profile. They must not require a new agent, setup command or device credential when the profile gains a server-owned capability. Build the compatibility artifact that introduces the durable named full-developer profile with:

node infra/deploy-runner/build-engine-agent-full-grant-migration-artifact.mjs \
  engine-agent-full-grant-migration-YYYYMMDD-NNN

This follow-up slice contains exactly nodedc-source/server/engineAgents/store.js. The runner pins its predecessor to the successfully applied Publish generation, requires the installed Publish overlay and active immutable backend, recreates only nodedc-backend, and proves the exact candidate SHA, named profile and current expanded runtime scope view. Store schema v1 is migrated atomically to v2 only when a grant contains the complete legacy nine-scope developer bundle. From then on the profile name is the authorization authority and its scope list is derived on every read, so capabilities deliberately added to full-developer immediately apply to existing full grants without token or store migrations. Partial grants become custom and remain exact; they are never elevated. The artifact does not touch UI, n8n, credentials, agent tokens, workflow graphs, databases or runtime payloads.

module-foundry is an independent, authenticated application component. Its artifact contains source and compose infrastructure only; its live /volume1/docker/nodedc-platform/module-foundry/source/.env is root-owned and never enters an artifact. The component reuses the existing internal platform credential for Launcher handoff validation and requires that runtime configuration before its first apply.

The Foundry ↔ Map Gateway signing key is not an application or .env setting. On the first relevant platform or module-foundry apply, the root-owned runner creates /volume1/docker/nodedc-platform/secrets/map-gateway-admin-secret atomically (root:gid 1000, mode 0640). Both containers receive that file only as a read-only mount. The value is never printed, backed up with source, included in an artifact, or administered through Foundry.

proxy-contur is the canonical VPN egress for selected Map Gateway provider hosts. Its existing root-owned PROXY_TOKEN is copied by the runner into /volume1/docker/nodedc-platform/secrets/map-egress-proxy-token with root:gid 1000, mode 0640, then mounted read-only only into Map Gateway. The value is neither printed nor contained in an artifact, Foundry setting, or browser response. Apply the proxy-contur artifact before the Platform Map Gateway artifact: it creates the private nodedc-map-egress Docker network.

dc-amd-proxy is the separate, staged connector for the neighbouring AMD VPN machine. Its active artifact attaches only to the private nodedc-map-egress network, exposes a narrow NAS-LAN pairing port, and has no direct provider egress. The runner preserves a 0700, service-user-owned runtime directory for the one-time paired connector credential and synchronizes the existing private Map Gateway egress credential as a read-only file. Neither value is ever placed in an artifact, .env, browser response, or runner output. The separate Platform switch is applied only after the connector and pairing are verified; it does not alter NAS routes, VPN, DNS, or Tailscale.

Install or update the root-owned live runner on Synology:

# First verify that no deploy process is active and state/deploy.lock is absent.
sudo install -o root -g root -m 0755 \
  /volume1/docker/nodedc-deploy/runner-install/nodedc-deploy \
  /usr/local/sbin/nodedc-deploy
sudo /usr/local/sbin/nodedc-deploy verify-install

Runner promotion is a standalone admin step, never an app-overlay artifact and never part of a running apply. A Python process already executing the old file keeps the old code in memory; always invoke a fresh verified process afterward.

Normal service deploys must still use explicit artifacts:

sudo /usr/local/sbin/nodedc-deploy plan /volume1/docker/nodedc-deploy/inbox/<artifact>.tgz
sudo /usr/local/sbin/nodedc-deploy apply /volume1/docker/nodedc-deploy/inbox/<artifact>.tgz