SILVER
/
NODEDC_PLATFORM
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
NODEDC_PLATFORM/infra/deploy-runner/README.md

22 KiB
Raw Blame History

NODE.DC deploy runner

This directory stores the versioned source for the Synology canonical deploy runner.

Live runner:

/usr/local/sbin/nodedc-deploy

Synology staging candidate:

/volume1/docker/nodedc-deploy/runner-install/nodedc-deploy

The runner accepts data-only app-overlay artifacts from:

/volume1/docker/nodedc-deploy/inbox

Supported components in this source:

  • engine
  • launcher
  • platform
  • tasker
  • ops-agents
  • bim-viewer
  • n8n-private-extension
  • module-foundry
  • proxy-contur
  • dc-amd-proxy

n8n-private-extension is a staging-only trust boundary for reviewed offline n8n private-node releases. Its artifact may contain exactly one digest-bound n8n-nodes-ndc release with package.tgz, release.json and rollback.json. The runner validates the inner npm tarball, rejects lifecycle scripts and runtime dependencies, refuses to overwrite an existing release, and seals the installed release root-owned/read-only under:

/volume1/docker/nodedc-platform/n8n-private-extensions/releases/n8n-nodes-ndc/<version>-<sha256-prefix>

This component has no Compose file, service, container mutation or activation side effect. In particular, staging does not make the node visible to n8n. Activation remains an Engine-owned change: mount the reviewed immutable release at /home/node/.n8n/nodes/node_modules/n8n-nodes-ndc, atomically switch between verified releases, restart every n8n process, and accept only after MCP exposes the package-qualified n8n-nodes-ndc.* schemas. The Platform runner cannot cross that boundary and never runs npm install in a live container.

Build a verified offline release artifact:

node infra/deploy-runner/build-n8n-private-extension-artifact.mjs \
  n8n-nodes-ndc-release-YYYYMMDD-NNN

The builder is byte-reproducible and accepts exactly the three reviewed NDC runtime types:

  • n8n-nodes-ndc.ndcDataProductPublish
  • n8n-nodes-ndc.ndcDataProductRead
  • n8n-nodes-ndc.ndcFoundryBinding

Their three opaque capability credential schemas are ndcDataProductWriterApi, ndcDataProductReaderApi and ndcFoundryBindingApi. A node description containing usableAsTool is rejected because n8n 2.3.2 would synthesize an additional *Tool runtime type and violate the exact-three activation contract. Run the positive and negative release-policy suite before publishing an artifact:

PYTHONDONTWRITEBYTECODE=1 \
  python3 infra/deploy-runner/test_n8n_private_extension.py

Release/rollback manifests use schema v2. Before a first activation, the Engine-owned activator must verify and record the current inactive state. That verified_inactive state is an allowed rollback baseline when no previous verified immutable release exists; later upgrades prefer the previous verified release. Rollback never deletes or mutates a staged release.

The historical 0.1.0 release remains immutable and must not be overwritten. Release 0.1.1-994756958861518e is retained as rejected/inactive: its three node descriptions used usableAsTool, so n8n 2.3.2 exposed six NDC runtime types instead of the required three. It must not be activated, overwritten or deleted.

The active predecessor is package version 0.1.4 at immutable release 0.1.4-59dc9f7882721d6a. Package 0.1.5 adds the provider-neutral complete snapshot replace mode used by map.zones.current.v2; its Platform staging remains inert until a separately reviewed Engine-owned transition selects the exact digest and accepts the updated MCP node schema.

The paired Engine activation is built by build-engine-n8n-private-extension-artifact.mjs. It deliberately does not copy from or write generated files into the dirty Engine worktree, and it does not build an image. A fresh transition id is mandatory and previously issued ids are rejected:

node infra/deploy-runner/build-engine-n8n-private-extension-artifact.mjs 20260717-005

The builder emits a narrowly scoped Compose override plus a strict transition descriptor. On apply, the runner validates the staged release again, verifies that the running n8n container and the NAS-local 2.3.2 tag resolve to the same immutable image ID, and extracts the package into the root-owned, read-only Engine release tree:

/volume2/nodedc-demo/n8n-private-extensions/releases/n8n-nodes-ndc/0.1.3-3354149b5245e39a/package

The override sets N8N_USER_FOLDER=/home/node, which is required because the actual Engine service runs as root while the canonical community package path is below /home/node/.n8n. The live runtime contract remains the successfully deployed generation-003 contract and does not set NODE_PATH. Only the runner's isolated node -e package-loader probe temporarily initializes the dependency tree bundled inside the exact n8n base image; this reproduces n8n's own loader without redefining the live Compose state. The override enables loading but disables reinstall, mounts only the exact release read-only, and uses both Compose pull_policy: never and docker compose up --pull never. No registry access, lifecycle script, database installed_packages row or custom-extension loader is involved.

The runner pins the exact Engine service topology observed in source and rejects an added worker/webhook generation. Only the single actual n8n service is force-recreated with --no-deps; the Postgres service, .n8n data, encryption key and credentials remain intact. The apply gate verifies readiness, the running image/version, sealed mount, loader environment, package-loader node/credential sets, scoped loader logs, restart stability and content-exact pinned Engine MCP catalogs. The runner pins the complete 434/385 inactive baseline and a digest registry for every reviewed 437/388 active release, so a same-count substitution of any built-in or private schema is rejected. Upgrade 0.1.2 -> 0.1.3 is accepted only when the verified live predecessor, descriptor, package mount and catalog all agree. Any gate failure after mutation automatically restores the pre-apply catalogs/descriptor and force-recreates the previous verified runtime. Staged, sealed and failed releases are retained. The paired rollback artifact returns 0.1.3 to the verified immutable 0.1.2 release and its exact catalogs; it does not invent an inactive baseline for an already-active upgrade.

Run both policy suites before publishing the Engine pair:

PYTHONDONTWRITEBYTECODE=1 \
  python3 infra/deploy-runner/test_n8n_private_extension.py
PYTHONDONTWRITEBYTECODE=1 \
  python3 infra/deploy-runner/test_engine_n8n_private_extension.py

The source-less Engine MCP control-plane update is a separate exact slice:

node infra/deploy-runner/build-engine-mcp-control-plane-artifact.mjs 20260717-001

Its six-entry registry contains only the Engine Agent gateway, verified graph patch route, Codex installer source/package and the active node-intelligence descriptor with the new gateway digest. It force-recreates only the existing nodedc-backend. The node-intelligence image/service, n8n, L1, credentials, databases and volumes are outside the slice. The runner proves the installed predecessor digests, exact installer archive/source equality, bounded change session policy, no-effect/post-write graph barriers, active immutable backend runtime and descriptor equality. The ordinary overlay backup is the automatic rollback source; rollback restores the predecessor gateway and descriptor together before recreating the same backend service.

The successor autonomy/provider-v5 slice keeps MCP authority explicit without turning every write into a second permission dialogue. MCP tool availability is the capability boundary, while the user's current objective is the intent boundary; Engine L2 may act autonomously only inside their intersection. A graph plan remains a mandatory machine barrier whose target, diff, revision and blockers are inspected by the agent. It is not a repeated approval prompt after an implementation objective is already authorized. Retries must add new evidence or change the attempted variant, and three identical failures without new evidence or state change are a critical stop. The slice also advances the installer to 0.1.6 and adds gelios.provider.v5 -> fleet.positions.current.v4 beside the immutable v4/v3 rollback line:

node infra/deploy-runner/build-engine-mcp-autonomy-provider-v5-artifact.mjs \
  20260720-004

Its seven-entry allowlist recreates only nodedc-backend; n8n, L1, databases, credential values and the node-intelligence image remain outside the change. The runner accepts both the exact target and the exact predecessor after an automatic rollback, and rejects every mixed state.

For platform artifacts, the allowlist includes the versioned Ontology Core, the frozen legacy Gelios compatibility service, and the provider-neutral External Data Plane sources. The Gelios service remains reproducible only to protect its existing database/workflow; it is not a template for a provider integration. An External Data Plane artifact builds only its image and force-recreates only external-data-plane; the already healthy external-data-plane-postgres container and its Timescale volume are an independent deploy prerequisite and are never selected by an EDP application artifact. The reviewed Compose source pins the Timescale image, named volume and target, internal database-only network, absence of database host ports, healthy dependency, localhost-only EDP bind and the three read-only provisioner/trust mounts in the reviewed Compose source. The runner does not reinterpret version-dependent docker compose config JSON as a second deploy schema. Its canonical enforcement remains the artifact/path allowlist plus hard-coded build command, selected service set, runtime-secret preparation and health acceptance. Post-apply acceptance also requires database=ready; the managed Foundry slice additionally requires foundryReaderBindingProvisioning=digest+server-resolved-source. A first-rollout failure removes only the candidate EDP container without volumes and restores the source overlay. It never contains a provider credential, provider endpoint, collection schedule or command transport. Its contract payload is an exact provider-neutral runtime subset; providers/*, mappings, fixtures and tests are excluded and do not trigger an EDP rebuild. Database credentials remain root-owned live .env.synology configuration and must not reuse NODEDC_INTERNAL_ACCESS_TOKEN.

On the first relevant Platform apply, the root-owned runner creates /volume1/docker/nodedc-platform/secrets/external-data-plane-provisioner/token atomically in a dedicated UID/GID 11006 directory (directory 0500, token 0400). It is never an .env value or an artifact member and is mounted read-only only into External Data Plane.

Manual one-time binding issuance and digest-only managed writer ensure are independently disabled by default. The target control-plane operation generates and stores the capability inside native NDC L2 Credentials and sends only its digest to EDP; users and MCP consumers receive only an opaque compatible reference/status. The runner-owned bearer above authenticates only legacy plaintext issuance and is intentionally rejected by managed ensure/revoke. Managed requests use a deployment Ed25519 Engine service key; EDP mounts only the public-key trust directory read-only. On every reviewed Engine apply and on an EDP runtime apply, this runner creates or validates one matching Ed25519 pair: the Engine-only private key is root:root 0400, while the EDP trust copy is root:11006 0440. Public-only crash state, key mismatch, a non-Ed25519 key, symlinks and permissive modes fail closed. plan discloses both paths without printing key material. The private key must not be broadened into an L2 graph, MCP surface, artifact or shared-token boundary.

Module Foundry has a separate Ed25519 managed-provisioner identity for target-scoped Data Product consumer grants. On a relevant platform or module-foundry apply, the runner creates or validates the Foundry-only private key at /volume1/docker/nodedc-platform/secrets/foundry-edp-managed-provisioner/private-key.pem as root:root 0400 and the matching EDP trust copy at /volume1/docker/nodedc-platform/trust/foundry-managed-provisioner/public-key.pem as root:11006 0440. Foundry generates the opaque reader token only inside its persistent private runtime; its signed EDP request contains only a SHA-256 digest and Data Product id. EDP resolves the unique active writer source scope server-side and fails closed on missing or ambiguous coverage. No provider, tenant, connection, token, private key or endpoint is admitted to the Foundry MCP plan, application state or browser response. The Engine signing identity, native n8n credentials and legacy issuance bearer cannot call this endpoint.

The reviewed Engine source candidate has dedicated server-derived MCP plan/apply handling for the exact ndcDataProductWriterApi + Data Product Publish tuple. It is separate from the generic HTTP safe-ref path and accepts no caller-provided provider/scope/credential identity, capability, generation or service URL. The production managed flag remains false during staging. In the explicitly confirmed activation window it is set to true immediately before the Platform EDP artifact; runner acceptance then requires EDP /healthz to report managed provisioning enabled. At that point the old Engine still has no signer mount, so the endpoint remains usable only after the separately accepted Engine artifact. Root/UI transfer is emergency self-hosted diagnostics only, not the user journey or acceptance path. A provider-specific daemon remains forbidden.

Build the narrow Engine source artifact with:

node infra/deploy-runner/build-engine-data-product-publish-grant-artifact.mjs \
  engine-data-product-publish-grant-YYYYMMDD-NNN

The builder includes exactly the pinned provider-security catalog, the Publish-grant service, Engine Agent scope/gateway wiring, the existing n8n adapter and the reviewed additive backend runtime overlay. It deliberately excludes base Compose, frontend/dist, runtime data, tests, native credentials, the NDC L2 process and the legacy generic credential-sink route/core. The runner fixes the runtime action to nodedc-backend with --no-deps and --pull never; n8n, nginx app, database services and volumes are not selected. Acceptance requires backend health, the active immutable backend identity and the exact additive mount inventory. Any failure restores the touched source and recreates only the previous verified backend runtime.

Existing Engine Agents created before Publish-grant support store the original nine scopes as the complete developer profile. They must not require a new agent, setup command or device credential when the profile gains a server-owned capability. Build the compatibility artifact that introduces the durable named full-developer profile with:

node infra/deploy-runner/build-engine-agent-full-grant-migration-artifact.mjs \
  engine-agent-full-grant-migration-YYYYMMDD-NNN

This follow-up slice contains exactly nodedc-source/server/engineAgents/store.js. The runner pins its predecessor to the successfully applied Publish generation, requires the installed Publish overlay and active immutable backend, recreates only nodedc-backend, and proves the exact candidate SHA, named profile and current expanded runtime scope view. Store schema v1 is migrated atomically to v2 only when a grant contains the complete legacy nine-scope developer bundle. From then on the profile name is the authorization authority and its scope list is derived on every read, so capabilities deliberately added to full-developer immediately apply to existing full grants without token or store migrations. Partial grants become custom and remain exact; they are never elevated. The artifact does not touch UI, n8n, credentials, agent tokens, workflow graphs, databases or runtime payloads.

Engine L2 node-intelligence transition

The node-intelligence transition is an additive Engine-owned deployment domain. It does not merge Ontology, provider APIs or Ops into the Engine MCP. It pins the reviewed upstream n8n-mcp implementation at package 2.33.2, commit 974a9fb3492fe2c4984ee0549085d531cdc6242a, and exposes only the safe NDC L2 projection through the existing Engine Agent gateway. Upstream management and write tools are not forwarded.

Production never clones, pulls, installs or builds this dependency. The builder saves the reviewed linux/amd64 image once, embeds that exact archive in a data-only activation artifact and records both the archive and image-config digests. The runner validates every inner blob, revision label, entrypoint, command and platform before an offline docker image load; Compose then uses the fixed tag with pull_policy: never and --pull never.

Build a fresh activation/rollback pair:

node infra/deploy-runner/build-engine-node-intelligence-artifacts.mjs \
  YYYYMMDD-NNN

The activation exact set is:

  • nodedc-source/server/nodeIntelligence
  • nodedc-source/server/routes/engineAgentGateway.js
  • nodedc-source/services/node-intelligence

The runtime adds only nodedc-node-intelligence and recreates the existing nodedc-backend; n8n, UI, databases, L1, provider services and volumes are not selected. The sidecar has no host port, runs as 11007:11007, has a read-only root filesystem, drops all capabilities and receives no Engine or provider credential. Its independent MCP bearer is created by the root-owned runner as a read-only file and mounted only into the sidecar and backend. It never appears in an artifact, shared environment file, log or MCP response.

Acceptance requires the exact image/container/mount/security inventory, immutable backend identity and live authenticated get_node, validate_node and validate_workflow calls. Any apply failure restores source and runtime automatically. The separate rollback artifact first removes only the sidecar without volumes, restores the pinned inactive gateway, and recreates only the verified backend. Loaded images and failed candidate source are retained for audit rather than destructively deleted.

Stage the reviewed runner under a unique candidate name first:

/volume1/docker/nodedc-deploy/runner-install/candidates/
  nodedc-deploy.engine-node-intelligence-YYYYMMDD-NNN

Do not overwrite the canonical staging candidate while another deploy may be in flight. After the no-lock/no-process gate, promote the exact candidate in a standalone root step, run verify-install, then invoke a fresh process for activation plan and only then apply. The generated plan/apply runbook carries the runner, activation and rollback SHA-256 values and is staged beside the unique runner candidate.

module-foundry is an independent, authenticated application component. Its artifact contains source and compose infrastructure only; its live /volume1/docker/nodedc-platform/module-foundry/source/.env is root-owned and never enters an artifact. The component reuses the existing internal platform credential for Launcher handoff validation and requires that runtime configuration before its first apply.

When the managed reader-grant source is present, Module Foundry acceptance also requires /healthz to report the dedicated Ed25519 provisioner as configured. The check is source-aware: if an apply rolls back to the prior source, rollback acceptance uses that prior health contract instead of falsely requiring a feature which the restored generation does not contain.

The Foundry ↔ Map Gateway signing key is not an application or .env setting. On the first relevant platform or module-foundry apply, the root-owned runner creates /volume1/docker/nodedc-platform/secrets/map-gateway-admin-secret atomically (root:gid 1000, mode 0640). Both containers receive that file only as a read-only mount. The value is never printed, backed up with source, included in an artifact, or administered through Foundry.

proxy-contur is the canonical VPN egress for selected Map Gateway provider hosts. Its existing root-owned PROXY_TOKEN is copied by the runner into /volume1/docker/nodedc-platform/secrets/map-egress-proxy-token with root:gid 1000, mode 0640, then mounted read-only only into Map Gateway. The value is neither printed nor contained in an artifact, Foundry setting, or browser response. Apply the proxy-contur artifact before the Platform Map Gateway artifact: it creates the private nodedc-map-egress Docker network.

dc-amd-proxy is the separate, staged connector for the neighbouring AMD VPN machine. Its active artifact attaches only to the private nodedc-map-egress network, exposes a narrow NAS-LAN pairing port, and has no direct provider egress. The runner preserves a 0700, service-user-owned runtime directory for the one-time paired connector credential and synchronizes the existing private Map Gateway egress credential as a read-only file. Neither value is ever placed in an artifact, .env, browser response, or runner output. The separate Platform switch is applied only after the connector and pairing are verified; it does not alter NAS routes, VPN, DNS, or Tailscale.

Install or update the root-owned live runner on Synology:

# First verify that no deploy process is active and state/deploy.lock is absent.
sudo install -o root -g root -m 0755 \
  /volume1/docker/nodedc-deploy/runner-install/nodedc-deploy \
  /usr/local/sbin/nodedc-deploy
sudo /usr/local/sbin/nodedc-deploy verify-install

Runner promotion is a standalone admin step, never an app-overlay artifact and never part of a running apply. A Python process already executing the old file keeps the old code in memory; always invoke a fresh verified process afterward.

Normal service deploys must still use explicit artifacts:

sudo /usr/local/sbin/nodedc-deploy plan /volume1/docker/nodedc-deploy/inbox/<artifact>.tgz
sudo /usr/local/sbin/nodedc-deploy apply /volume1/docker/nodedc-deploy/inbox/<artifact>.tgz