432 lines
22 KiB
Markdown
432 lines
22 KiB
Markdown
# NODE.DC deploy runner
|
|
|
|
This directory stores the versioned source for the Synology canonical deploy runner.
|
|
|
|
Live runner:
|
|
|
|
```text
|
|
/usr/local/sbin/nodedc-deploy
|
|
```
|
|
|
|
Synology staging candidate:
|
|
|
|
```text
|
|
/volume1/docker/nodedc-deploy/runner-install/nodedc-deploy
|
|
```
|
|
|
|
The runner accepts data-only app-overlay artifacts from:
|
|
|
|
```text
|
|
/volume1/docker/nodedc-deploy/inbox
|
|
```
|
|
|
|
Supported components in this source:
|
|
|
|
- `engine`
|
|
- `launcher`
|
|
- `platform`
|
|
- `tasker`
|
|
- `ops-agents`
|
|
- `bim-viewer`
|
|
- `n8n-private-extension`
|
|
- `module-foundry`
|
|
- `proxy-contur`
|
|
- `dc-amd-proxy`
|
|
|
|
`n8n-private-extension` is a staging-only trust boundary for reviewed offline
|
|
n8n private-node releases. Its artifact may contain exactly one digest-bound
|
|
`n8n-nodes-ndc` release with `package.tgz`, `release.json` and
|
|
`rollback.json`. The runner validates the inner npm tarball, rejects lifecycle
|
|
scripts and runtime dependencies, refuses to overwrite an existing release,
|
|
and seals the installed release root-owned/read-only under:
|
|
|
|
```text
|
|
/volume1/docker/nodedc-platform/n8n-private-extensions/releases/n8n-nodes-ndc/<version>-<sha256-prefix>
|
|
```
|
|
|
|
This component has no Compose file, service, container mutation or activation
|
|
side effect. In particular, staging does **not** make the node visible to n8n.
|
|
Activation remains an Engine-owned change: mount the reviewed immutable release
|
|
at `/home/node/.n8n/nodes/node_modules/n8n-nodes-ndc`, atomically switch between
|
|
verified releases, restart every n8n process, and accept only after MCP exposes
|
|
the package-qualified `n8n-nodes-ndc.*` schemas. The Platform runner cannot
|
|
cross that boundary and never runs `npm install` in a live container.
|
|
|
|
Build a verified offline release artifact:
|
|
|
|
```bash
|
|
node infra/deploy-runner/build-n8n-private-extension-artifact.mjs \
|
|
n8n-nodes-ndc-release-YYYYMMDD-NNN
|
|
```
|
|
|
|
The builder is byte-reproducible and accepts exactly the three reviewed NDC
|
|
runtime types:
|
|
|
|
- `n8n-nodes-ndc.ndcDataProductPublish`
|
|
- `n8n-nodes-ndc.ndcDataProductRead`
|
|
- `n8n-nodes-ndc.ndcFoundryBinding`
|
|
|
|
Their three opaque capability credential schemas are
|
|
`ndcDataProductWriterApi`, `ndcDataProductReaderApi` and
|
|
`ndcFoundryBindingApi`. A node description containing `usableAsTool` is
|
|
rejected because n8n 2.3.2 would synthesize an additional `*Tool` runtime type
|
|
and violate the exact-three activation contract. Run the positive and negative
|
|
release-policy suite before publishing an artifact:
|
|
|
|
```bash
|
|
PYTHONDONTWRITEBYTECODE=1 \
|
|
python3 infra/deploy-runner/test_n8n_private_extension.py
|
|
```
|
|
|
|
Release/rollback manifests use schema v2. Before a first activation, the
|
|
Engine-owned activator must verify and record the current inactive state. That
|
|
`verified_inactive` state is an allowed rollback baseline when no previous
|
|
verified immutable release exists; later upgrades prefer the previous verified
|
|
release. Rollback never deletes or mutates a staged release.
|
|
|
|
The historical `0.1.0` release remains immutable and must not be overwritten.
|
|
Release `0.1.1-994756958861518e` is retained as rejected/inactive: its three
|
|
node descriptions used `usableAsTool`, so n8n 2.3.2 exposed six NDC runtime
|
|
types instead of the required three. It must not be activated, overwritten or
|
|
deleted.
|
|
|
|
The active predecessor is package version `0.1.4` at immutable release
|
|
`0.1.4-59dc9f7882721d6a`. Package `0.1.5` adds the provider-neutral complete
|
|
snapshot replace mode used by `map.zones.current.v2`; its Platform staging
|
|
remains inert until a separately reviewed Engine-owned transition selects the
|
|
exact digest and accepts the updated MCP node schema.
|
|
|
|
The paired Engine activation is built by
|
|
`build-engine-n8n-private-extension-artifact.mjs`. It deliberately does not
|
|
copy from or write generated files into the dirty Engine worktree, and it does
|
|
not build an image. A fresh transition id is mandatory and previously issued
|
|
ids are rejected:
|
|
|
|
```bash
|
|
node infra/deploy-runner/build-engine-n8n-private-extension-artifact.mjs 20260717-005
|
|
```
|
|
|
|
The builder emits a narrowly scoped Compose override plus a strict transition
|
|
descriptor. On apply, the runner validates the staged release again, verifies
|
|
that the running n8n container and the NAS-local `2.3.2` tag resolve to the
|
|
same immutable image ID, and extracts the package into the root-owned,
|
|
read-only Engine release tree:
|
|
|
|
```text
|
|
/volume2/nodedc-demo/n8n-private-extensions/releases/n8n-nodes-ndc/0.1.3-3354149b5245e39a/package
|
|
```
|
|
|
|
The override sets `N8N_USER_FOLDER=/home/node`, which is required because the
|
|
actual Engine service runs as root while the canonical community package path
|
|
is below `/home/node/.n8n`. The live runtime contract remains the successfully
|
|
deployed generation-003 contract and does not set `NODE_PATH`. Only the
|
|
runner's isolated `node -e` package-loader probe temporarily initializes the
|
|
dependency tree bundled inside the exact n8n base image; this reproduces n8n's
|
|
own loader without redefining the live Compose state. The override enables
|
|
loading but disables reinstall, mounts only the exact release read-only, and
|
|
uses both Compose `pull_policy: never` and `docker compose up --pull never`.
|
|
No registry access, lifecycle script, database `installed_packages` row or
|
|
custom-extension loader is involved.
|
|
|
|
The runner pins the exact Engine service topology observed in source and
|
|
rejects an added worker/webhook generation. Only the single actual `n8n`
|
|
service is force-recreated with `--no-deps`; the
|
|
Postgres service, `.n8n` data, encryption key and credentials remain intact.
|
|
The apply gate verifies readiness, the running image/version, sealed mount,
|
|
loader environment, package-loader node/credential sets, scoped loader logs,
|
|
restart stability and content-exact pinned Engine MCP catalogs. The runner pins
|
|
the complete 434/385 inactive baseline and a digest registry for every
|
|
reviewed 437/388 active release, so a same-count substitution of any built-in
|
|
or private schema is rejected. Upgrade `0.1.2 -> 0.1.3` is accepted only when
|
|
the verified live predecessor, descriptor, package mount and catalog all agree.
|
|
Any gate failure after
|
|
mutation automatically restores the pre-apply catalogs/descriptor and
|
|
force-recreates the previous verified runtime. Staged, sealed and failed
|
|
releases are retained. The paired rollback artifact returns `0.1.3` to the
|
|
verified immutable `0.1.2` release and its exact catalogs; it does not invent
|
|
an inactive baseline for an already-active upgrade.
|
|
|
|
Run both policy suites before publishing the Engine pair:
|
|
|
|
```bash
|
|
PYTHONDONTWRITEBYTECODE=1 \
|
|
python3 infra/deploy-runner/test_n8n_private_extension.py
|
|
PYTHONDONTWRITEBYTECODE=1 \
|
|
python3 infra/deploy-runner/test_engine_n8n_private_extension.py
|
|
```
|
|
|
|
The source-less Engine MCP control-plane update is a separate exact slice:
|
|
|
|
```bash
|
|
node infra/deploy-runner/build-engine-mcp-control-plane-artifact.mjs 20260717-001
|
|
```
|
|
|
|
Its six-entry registry contains only the Engine Agent gateway, verified graph
|
|
patch route, Codex installer source/package and the active node-intelligence
|
|
descriptor with the new gateway digest. It force-recreates only the existing
|
|
`nodedc-backend`. The node-intelligence image/service, n8n, L1, credentials,
|
|
databases and volumes are outside the slice. The runner proves the installed
|
|
predecessor digests, exact installer archive/source equality, bounded change
|
|
session policy, no-effect/post-write graph barriers, active immutable backend
|
|
runtime and descriptor equality. The ordinary overlay backup is the automatic
|
|
rollback source; rollback restores the predecessor gateway and descriptor
|
|
together before recreating the same backend service.
|
|
|
|
The successor autonomy/provider-v5 slice keeps MCP authority explicit without
|
|
turning every write into a second permission dialogue. MCP tool availability is
|
|
the capability boundary, while the user's current objective is the intent
|
|
boundary; Engine L2 may act autonomously only inside their intersection. A
|
|
graph `plan` remains a mandatory machine barrier whose target, diff, revision
|
|
and blockers are inspected by the agent. It is not a repeated approval prompt
|
|
after an implementation objective is already authorized. Retries must add new
|
|
evidence or change the attempted variant, and three identical failures without
|
|
new evidence or state change are a critical stop. The slice also advances the
|
|
installer to `0.1.6` and adds `gelios.provider.v5 ->
|
|
fleet.positions.current.v4` beside the immutable v4/v3 rollback line:
|
|
|
|
```bash
|
|
node infra/deploy-runner/build-engine-mcp-autonomy-provider-v5-artifact.mjs \
|
|
20260720-004
|
|
```
|
|
|
|
Its seven-entry allowlist recreates only `nodedc-backend`; n8n, L1, databases,
|
|
credential values and the node-intelligence image remain outside the change.
|
|
The runner accepts both the exact target and the exact predecessor after an
|
|
automatic rollback, and rejects every mixed state.
|
|
|
|
For `platform` artifacts, the allowlist includes the versioned Ontology Core,
|
|
the frozen legacy Gelios compatibility service, and the provider-neutral
|
|
External Data Plane sources. The Gelios service remains reproducible only to
|
|
protect its existing database/workflow; it is not a template for a provider
|
|
integration. An External Data Plane artifact builds only its image and
|
|
force-recreates only `external-data-plane`; the already healthy
|
|
`external-data-plane-postgres` container and its Timescale volume are an
|
|
independent deploy prerequisite and are never selected by an EDP application
|
|
artifact. The reviewed Compose source pins the Timescale image, named volume
|
|
and target, internal database-only network, absence of database host ports,
|
|
healthy dependency, localhost-only EDP bind and the three read-only
|
|
provisioner/trust mounts in the reviewed Compose source. The runner does not
|
|
reinterpret version-dependent `docker compose config` JSON as a second deploy
|
|
schema. Its canonical enforcement remains the artifact/path allowlist plus
|
|
hard-coded build command, selected service set, runtime-secret preparation and
|
|
health acceptance. Post-apply acceptance also requires
|
|
`database=ready`; the managed Foundry slice additionally requires
|
|
`foundryReaderBindingProvisioning=digest+server-resolved-source`. A first-rollout failure removes only the candidate EDP
|
|
container without volumes and restores the source overlay. It never contains a
|
|
provider credential, provider endpoint,
|
|
collection schedule or command
|
|
transport. Its contract payload is an exact provider-neutral runtime subset;
|
|
`providers/*`, mappings, fixtures and tests are excluded and do not trigger an
|
|
EDP rebuild. Database credentials remain root-owned live `.env.synology`
|
|
configuration and must not reuse `NODEDC_INTERNAL_ACCESS_TOKEN`.
|
|
|
|
On the first relevant Platform apply, the root-owned runner creates
|
|
`/volume1/docker/nodedc-platform/secrets/external-data-plane-provisioner/token`
|
|
atomically in a dedicated UID/GID `11006` directory (directory `0500`, token
|
|
`0400`). It is never an `.env` value or an artifact member and is mounted
|
|
read-only only into External Data Plane.
|
|
|
|
Manual one-time binding issuance and digest-only managed writer ensure are
|
|
independently disabled by default. The target control-plane operation generates
|
|
and stores the capability inside native NDC L2 Credentials and sends only its
|
|
digest to EDP; users and MCP consumers receive only an opaque compatible
|
|
reference/status. The runner-owned bearer above authenticates only legacy
|
|
plaintext issuance and is intentionally rejected by managed ensure/revoke.
|
|
Managed requests use a deployment Ed25519 Engine service key; EDP mounts only
|
|
the public-key trust directory read-only. On every reviewed Engine apply and on
|
|
an EDP runtime apply, this runner creates or validates one matching Ed25519 pair:
|
|
the Engine-only private key is `root:root 0400`, while the EDP trust copy is
|
|
`root:11006 0440`. Public-only crash state, key mismatch, a non-Ed25519 key,
|
|
symlinks and permissive modes fail closed. `plan` discloses both paths without
|
|
printing key material. The private key must not be broadened into an L2 graph,
|
|
MCP surface, artifact or shared-token boundary.
|
|
|
|
Module Foundry has a separate Ed25519 managed-provisioner identity for
|
|
target-scoped Data Product consumer grants. On a relevant `platform` or
|
|
`module-foundry` apply, the runner creates or validates the Foundry-only private
|
|
key at
|
|
`/volume1/docker/nodedc-platform/secrets/foundry-edp-managed-provisioner/private-key.pem`
|
|
as `root:root 0400` and the matching EDP trust copy at
|
|
`/volume1/docker/nodedc-platform/trust/foundry-managed-provisioner/public-key.pem`
|
|
as `root:11006 0440`. Foundry generates the opaque reader token only inside its
|
|
persistent private runtime; its signed EDP request contains only a SHA-256
|
|
digest and Data Product id. EDP resolves the unique active writer source scope
|
|
server-side and fails closed on missing or ambiguous coverage. No provider,
|
|
tenant, connection, token, private key or endpoint is admitted to the Foundry
|
|
MCP plan, application state or browser response. The Engine signing identity,
|
|
native n8n credentials and legacy issuance bearer cannot call this endpoint.
|
|
|
|
The reviewed Engine source candidate has dedicated server-derived MCP
|
|
plan/apply handling for the exact `ndcDataProductWriterApi` + Data Product
|
|
Publish tuple. It is separate from the generic HTTP safe-ref path and accepts no
|
|
caller-provided provider/scope/credential identity, capability, generation or
|
|
service URL. The production managed flag remains false during staging. In the
|
|
explicitly confirmed activation window it is set to true immediately before
|
|
the Platform EDP artifact; runner acceptance then requires EDP `/healthz` to
|
|
report managed provisioning `enabled`. At that point the old Engine still has
|
|
no signer mount, so the endpoint remains usable only after the separately
|
|
accepted Engine artifact. Root/UI transfer is emergency
|
|
self-hosted diagnostics only, not the user journey or acceptance
|
|
path. A provider-specific daemon remains forbidden.
|
|
|
|
Build the narrow Engine source artifact with:
|
|
|
|
```bash
|
|
node infra/deploy-runner/build-engine-data-product-publish-grant-artifact.mjs \
|
|
engine-data-product-publish-grant-YYYYMMDD-NNN
|
|
```
|
|
|
|
The builder includes exactly the pinned provider-security catalog, the
|
|
Publish-grant service, Engine Agent scope/gateway wiring, the existing n8n
|
|
adapter and the reviewed additive backend runtime overlay. It deliberately
|
|
excludes base Compose, frontend/dist, runtime data, tests, native credentials,
|
|
the NDC L2 process and the legacy generic credential-sink route/core. The
|
|
runner fixes the runtime action to `nodedc-backend` with `--no-deps` and
|
|
`--pull never`; n8n, nginx app, database services and volumes are not selected.
|
|
Acceptance requires backend health, the active immutable backend identity and
|
|
the exact additive mount inventory. Any failure restores the touched source and
|
|
recreates only the previous verified backend runtime.
|
|
|
|
Existing Engine Agents created before Publish-grant support store the original
|
|
nine scopes as the complete developer profile. They must not require a new
|
|
agent, setup command or device credential when the profile gains a server-owned
|
|
capability. Build the compatibility artifact that introduces the durable named
|
|
`full-developer` profile with:
|
|
|
|
```bash
|
|
node infra/deploy-runner/build-engine-agent-full-grant-migration-artifact.mjs \
|
|
engine-agent-full-grant-migration-YYYYMMDD-NNN
|
|
```
|
|
|
|
This follow-up slice contains exactly
|
|
`nodedc-source/server/engineAgents/store.js`. The runner pins its predecessor to
|
|
the successfully applied Publish generation, requires the installed Publish
|
|
overlay and active immutable backend, recreates only `nodedc-backend`, and
|
|
proves the exact candidate SHA, named profile and current expanded runtime
|
|
scope view. Store schema v1 is migrated atomically to v2 only when a grant
|
|
contains the complete legacy nine-scope developer bundle. From then on the
|
|
profile name is the authorization authority and its scope list is derived on
|
|
every read, so capabilities deliberately added to `full-developer` immediately
|
|
apply to existing full grants without token or store migrations. Partial grants
|
|
become `custom` and remain exact; they are never elevated. The artifact does not
|
|
touch UI, n8n, credentials, agent tokens, workflow graphs, databases or runtime
|
|
payloads.
|
|
|
|
## Engine L2 node-intelligence transition
|
|
|
|
The node-intelligence transition is an additive Engine-owned deployment domain.
|
|
It does not merge Ontology, provider APIs or Ops into the Engine MCP. It pins the
|
|
reviewed upstream `n8n-mcp` implementation at package `2.33.2`, commit
|
|
`974a9fb3492fe2c4984ee0549085d531cdc6242a`, and exposes only the safe NDC L2
|
|
projection through the existing Engine Agent gateway. Upstream management and
|
|
write tools are not forwarded.
|
|
|
|
Production never clones, pulls, installs or builds this dependency. The builder
|
|
saves the reviewed `linux/amd64` image once, embeds that exact archive in a
|
|
data-only activation artifact and records both the archive and image-config
|
|
digests. The runner validates every inner blob, revision label, entrypoint,
|
|
command and platform before an offline `docker image load`; Compose then uses
|
|
the fixed tag with `pull_policy: never` and `--pull never`.
|
|
|
|
Build a fresh activation/rollback pair:
|
|
|
|
```bash
|
|
node infra/deploy-runner/build-engine-node-intelligence-artifacts.mjs \
|
|
YYYYMMDD-NNN
|
|
```
|
|
|
|
The activation exact set is:
|
|
|
|
- `nodedc-source/server/nodeIntelligence`
|
|
- `nodedc-source/server/routes/engineAgentGateway.js`
|
|
- `nodedc-source/services/node-intelligence`
|
|
|
|
The runtime adds only `nodedc-node-intelligence` and recreates the existing
|
|
`nodedc-backend`; n8n, UI, databases, L1, provider services and volumes are not
|
|
selected. The sidecar has no host port, runs as `11007:11007`, has a read-only
|
|
root filesystem, drops all capabilities and receives no Engine or provider
|
|
credential. Its independent MCP bearer is created by the root-owned runner as a
|
|
read-only file and mounted only into the sidecar and backend. It never appears
|
|
in an artifact, shared environment file, log or MCP response.
|
|
|
|
Acceptance requires the exact image/container/mount/security inventory,
|
|
immutable backend identity and live authenticated `get_node`, `validate_node`
|
|
and `validate_workflow` calls. Any apply failure restores source and runtime
|
|
automatically. The separate rollback artifact first removes only the sidecar
|
|
without volumes, restores the pinned inactive gateway, and recreates only the
|
|
verified backend. Loaded images and failed candidate source are retained for
|
|
audit rather than destructively deleted.
|
|
|
|
Stage the reviewed runner under a unique candidate name first:
|
|
|
|
```text
|
|
/volume1/docker/nodedc-deploy/runner-install/candidates/
|
|
nodedc-deploy.engine-node-intelligence-YYYYMMDD-NNN
|
|
```
|
|
|
|
Do not overwrite the canonical staging candidate while another deploy may be in
|
|
flight. After the no-lock/no-process gate, promote the exact candidate in a
|
|
standalone root step, run `verify-install`, then invoke a fresh process for
|
|
activation `plan` and only then `apply`. The generated plan/apply runbook carries
|
|
the runner, activation and rollback SHA-256 values and is staged beside the
|
|
unique runner candidate.
|
|
|
|
`module-foundry` is an independent, authenticated application component. Its
|
|
artifact contains source and compose infrastructure only; its live
|
|
`/volume1/docker/nodedc-platform/module-foundry/source/.env` is root-owned and
|
|
never enters an artifact. The component reuses the existing internal platform
|
|
credential for Launcher handoff validation and requires that runtime
|
|
configuration before its first `apply`.
|
|
|
|
When the managed reader-grant source is present, Module Foundry acceptance also
|
|
requires `/healthz` to report the dedicated Ed25519 provisioner as configured.
|
|
The check is source-aware: if an apply rolls back to the prior source, rollback
|
|
acceptance uses that prior health contract instead of falsely requiring a
|
|
feature which the restored generation does not contain.
|
|
|
|
The Foundry ↔ Map Gateway signing key is not an application or `.env` setting.
|
|
On the first relevant `platform` or `module-foundry` apply, the root-owned
|
|
runner creates `/volume1/docker/nodedc-platform/secrets/map-gateway-admin-secret`
|
|
atomically (root:gid 1000, mode `0640`). Both containers receive that file only
|
|
as a read-only mount. The value is never printed, backed up with source,
|
|
included in an artifact, or administered through Foundry.
|
|
|
|
`proxy-contur` is the canonical VPN egress for selected Map Gateway provider
|
|
hosts. Its existing root-owned `PROXY_TOKEN` is copied by the runner into
|
|
`/volume1/docker/nodedc-platform/secrets/map-egress-proxy-token` with
|
|
`root:gid 1000`, mode `0640`, then mounted read-only only into Map Gateway.
|
|
The value is neither printed nor contained in an artifact, Foundry setting, or
|
|
browser response. Apply the `proxy-contur` artifact before the Platform Map
|
|
Gateway artifact: it creates the private `nodedc-map-egress` Docker network.
|
|
|
|
`dc-amd-proxy` is the separate, staged connector for the neighbouring AMD VPN
|
|
machine. Its active artifact attaches only to the private `nodedc-map-egress`
|
|
network, exposes a narrow NAS-LAN pairing port, and has no direct provider
|
|
egress. The runner preserves a `0700`, service-user-owned runtime directory
|
|
for the one-time paired connector credential and synchronizes the existing
|
|
private Map Gateway egress credential as a read-only file. Neither value is
|
|
ever placed in an artifact, `.env`, browser response, or runner output. The
|
|
separate Platform switch is applied only after the connector and pairing are
|
|
verified; it does not alter NAS routes, VPN, DNS, or Tailscale.
|
|
|
|
Install or update the root-owned live runner on Synology:
|
|
|
|
```bash
|
|
# First verify that no deploy process is active and state/deploy.lock is absent.
|
|
sudo install -o root -g root -m 0755 \
|
|
/volume1/docker/nodedc-deploy/runner-install/nodedc-deploy \
|
|
/usr/local/sbin/nodedc-deploy
|
|
sudo /usr/local/sbin/nodedc-deploy verify-install
|
|
```
|
|
|
|
Runner promotion is a standalone admin step, never an app-overlay artifact and
|
|
never part of a running apply. A Python process already executing the old file
|
|
keeps the old code in memory; always invoke a fresh verified process afterward.
|
|
|
|
Normal service deploys must still use explicit artifacts:
|
|
|
|
```bash
|
|
sudo /usr/local/sbin/nodedc-deploy plan /volume1/docker/nodedc-deploy/inbox/<artifact>.tgz
|
|
sudo /usr/local/sbin/nodedc-deploy apply /volume1/docker/nodedc-deploy/inbox/<artifact>.tgz
|
|
```
|