308 lines
15 KiB
Markdown
308 lines
15 KiB
Markdown
# NODE.DC deploy runner
|
|
|
|
This directory stores the versioned source for the Synology canonical deploy runner.
|
|
|
|
Live runner:
|
|
|
|
```text
|
|
/usr/local/sbin/nodedc-deploy
|
|
```
|
|
|
|
Synology staging candidate:
|
|
|
|
```text
|
|
/volume1/docker/nodedc-deploy/runner-install/nodedc-deploy
|
|
```
|
|
|
|
The runner accepts data-only app-overlay artifacts from:
|
|
|
|
```text
|
|
/volume1/docker/nodedc-deploy/inbox
|
|
```
|
|
|
|
Supported components in this source:
|
|
|
|
- `engine`
|
|
- `launcher`
|
|
- `platform`
|
|
- `tasker`
|
|
- `ops-agents`
|
|
- `bim-viewer`
|
|
- `n8n-private-extension`
|
|
- `module-foundry`
|
|
- `proxy-contur`
|
|
- `dc-amd-proxy`
|
|
|
|
`n8n-private-extension` is a staging-only trust boundary for reviewed offline
|
|
n8n private-node releases. Its artifact may contain exactly one digest-bound
|
|
`n8n-nodes-ndc` release with `package.tgz`, `release.json` and
|
|
`rollback.json`. The runner validates the inner npm tarball, rejects lifecycle
|
|
scripts and runtime dependencies, refuses to overwrite an existing release,
|
|
and seals the installed release root-owned/read-only under:
|
|
|
|
```text
|
|
/volume1/docker/nodedc-platform/n8n-private-extensions/releases/n8n-nodes-ndc/<version>-<sha256-prefix>
|
|
```
|
|
|
|
This component has no Compose file, service, container mutation or activation
|
|
side effect. In particular, staging does **not** make the node visible to n8n.
|
|
Activation remains an Engine-owned change: mount the reviewed immutable release
|
|
at `/home/node/.n8n/nodes/node_modules/n8n-nodes-ndc`, atomically switch between
|
|
verified releases, restart every n8n process, and accept only after MCP exposes
|
|
the package-qualified `n8n-nodes-ndc.*` schemas. The Platform runner cannot
|
|
cross that boundary and never runs `npm install` in a live container.
|
|
|
|
Build a verified offline release artifact:
|
|
|
|
```bash
|
|
node infra/deploy-runner/build-n8n-private-extension-artifact.mjs \
|
|
n8n-nodes-ndc-release-YYYYMMDD-NNN
|
|
```
|
|
|
|
The builder is byte-reproducible and accepts exactly the three reviewed NDC
|
|
runtime types:
|
|
|
|
- `n8n-nodes-ndc.ndcDataProductPublish`
|
|
- `n8n-nodes-ndc.ndcDataProductRead`
|
|
- `n8n-nodes-ndc.ndcFoundryBinding`
|
|
|
|
Their three opaque capability credential schemas are
|
|
`ndcDataProductWriterApi`, `ndcDataProductReaderApi` and
|
|
`ndcFoundryBindingApi`. A node description containing `usableAsTool` is
|
|
rejected because n8n 2.3.2 would synthesize an additional `*Tool` runtime type
|
|
and violate the exact-three activation contract. Run the positive and negative
|
|
release-policy suite before publishing an artifact:
|
|
|
|
```bash
|
|
PYTHONDONTWRITEBYTECODE=1 \
|
|
python3 infra/deploy-runner/test_n8n_private_extension.py
|
|
```
|
|
|
|
Release/rollback manifests use schema v2. Before a first activation, the
|
|
Engine-owned activator must verify and record the current inactive state. That
|
|
`verified_inactive` state is an allowed rollback baseline when no previous
|
|
verified immutable release exists; later upgrades prefer the previous verified
|
|
release. Rollback never deletes or mutates a staged release.
|
|
|
|
The historical `0.1.0` release remains immutable and must not be overwritten.
|
|
Release `0.1.1-994756958861518e` is retained as rejected/inactive: its three
|
|
node descriptions used `usableAsTool`, so n8n 2.3.2 exposed six NDC runtime
|
|
types instead of the required three. It must not be activated, overwritten or
|
|
deleted.
|
|
|
|
The corrected candidate is package version `0.1.2`, built with patch id
|
|
`n8n-nodes-ndc-release-20260716-003`. It receives a new digest-bound release
|
|
directory and remains inert after staging; only a separately reviewed
|
|
Engine-owned activator may select it after exact MCP schema acceptance.
|
|
|
|
The paired Engine activation is built by
|
|
`build-engine-n8n-private-extension-artifact.mjs`. It deliberately does not
|
|
copy from or write generated files into the dirty Engine worktree, and it does
|
|
not build an image. A fresh transition id is mandatory and previously issued
|
|
ids are rejected:
|
|
|
|
```bash
|
|
node infra/deploy-runner/build-engine-n8n-private-extension-artifact.mjs 20260717-004
|
|
```
|
|
|
|
The builder emits a narrowly scoped Compose override plus a strict transition
|
|
descriptor. On apply, the runner validates the staged release again, verifies
|
|
that the running n8n container and the NAS-local `2.3.2` tag resolve to the
|
|
same immutable image ID, and extracts the package into the root-owned,
|
|
read-only Engine release tree:
|
|
|
|
```text
|
|
/volume2/nodedc-demo/n8n-private-extensions/releases/n8n-nodes-ndc/0.1.2-05e4b38b14b4a019/package
|
|
```
|
|
|
|
The override sets `N8N_USER_FOLDER=/home/node`, which is required because the
|
|
actual Engine service runs as root while the canonical community package path
|
|
is below `/home/node/.n8n`. The live runtime contract remains the successfully
|
|
deployed generation-003 contract and does not set `NODE_PATH`. Only the
|
|
runner's isolated `node -e` package-loader probe temporarily initializes the
|
|
dependency tree bundled inside the exact n8n base image; this reproduces n8n's
|
|
own loader without redefining the live Compose state. The override enables
|
|
loading but disables reinstall, mounts only the exact release read-only, and
|
|
uses both Compose `pull_policy: never` and `docker compose up --pull never`.
|
|
No registry access, lifecycle script, database `installed_packages` row or
|
|
custom-extension loader is involved.
|
|
|
|
The runner pins the exact Engine service topology observed in source and
|
|
rejects an added worker/webhook generation. Only the single actual `n8n`
|
|
service is force-recreated with `--no-deps`; the
|
|
Postgres service, `.n8n` data, encryption key and credentials remain intact.
|
|
The apply gate verifies readiness, the running image/version, sealed mount,
|
|
loader environment, package-loader node/credential sets, scoped loader logs,
|
|
restart stability and content-exact pinned Engine MCP catalogs. The runner pins
|
|
both the complete 434/385 inactive baseline and the reviewed 437/388
|
|
activation catalogs, so a same-count substitution of any built-in schema is
|
|
rejected. Any gate failure after
|
|
mutation automatically restores the pre-apply catalogs/descriptor and
|
|
force-recreates the previous verified runtime. Staged, sealed and failed
|
|
releases are retained. The separate rollback artifact returns the first
|
|
activation to the verified inactive 434-node/385-credential catalog baseline.
|
|
|
|
Run both policy suites before publishing the Engine pair:
|
|
|
|
```bash
|
|
PYTHONDONTWRITEBYTECODE=1 \
|
|
python3 infra/deploy-runner/test_n8n_private_extension.py
|
|
PYTHONDONTWRITEBYTECODE=1 \
|
|
python3 infra/deploy-runner/test_engine_n8n_private_extension.py
|
|
```
|
|
|
|
For `platform` artifacts, the allowlist includes the versioned Ontology Core,
|
|
the frozen legacy Gelios compatibility service, and the provider-neutral
|
|
External Data Plane sources. The Gelios service remains reproducible only to
|
|
protect its existing database/workflow; it is not a template for a provider
|
|
integration. An External Data Plane artifact builds only its image and
|
|
force-recreates only `external-data-plane`; the already healthy
|
|
`external-data-plane-postgres` container and its Timescale volume are an
|
|
independent deploy prerequisite and are never selected by an EDP application
|
|
artifact. The reviewed Compose source pins the Timescale image, named volume
|
|
and target, internal database-only network, absence of database host ports,
|
|
healthy dependency, localhost-only EDP bind and the two read-only
|
|
provisioner/trust mounts in the reviewed Compose source. The runner does not
|
|
reinterpret version-dependent `docker compose config` JSON as a second deploy
|
|
schema. Its canonical enforcement remains the artifact/path allowlist plus
|
|
hard-coded build command, selected service set, runtime-secret preparation and
|
|
health acceptance. Post-apply acceptance also requires
|
|
`database=ready`; a first-rollout failure removes only the candidate EDP
|
|
container without volumes and restores the source overlay. It never contains a
|
|
provider credential, provider endpoint,
|
|
collection schedule or command
|
|
transport. Its contract payload is an exact provider-neutral runtime subset;
|
|
`providers/*`, mappings, fixtures and tests are excluded and do not trigger an
|
|
EDP rebuild. Database credentials remain root-owned live `.env.synology`
|
|
configuration and must not reuse `NODEDC_INTERNAL_ACCESS_TOKEN`.
|
|
|
|
On the first relevant Platform apply, the root-owned runner creates
|
|
`/volume1/docker/nodedc-platform/secrets/external-data-plane-provisioner/token`
|
|
atomically in a dedicated UID/GID `11006` directory (directory `0500`, token
|
|
`0400`). It is never an `.env` value or an artifact member and is mounted
|
|
read-only only into External Data Plane.
|
|
|
|
Manual one-time binding issuance and digest-only managed writer ensure are
|
|
independently disabled by default. The target control-plane operation generates
|
|
and stores the capability inside native NDC L2 Credentials and sends only its
|
|
digest to EDP; users and MCP consumers receive only an opaque compatible
|
|
reference/status. The runner-owned bearer above authenticates only legacy
|
|
plaintext issuance and is intentionally rejected by managed ensure/revoke.
|
|
Managed requests use a deployment Ed25519 Engine service key; EDP mounts only
|
|
the public-key trust directory read-only. On every reviewed Engine apply and on
|
|
an EDP runtime apply, this runner creates or validates one matching Ed25519 pair:
|
|
the Engine-only private key is `root:root 0400`, while the EDP trust copy is
|
|
`root:11006 0440`. Public-only crash state, key mismatch, a non-Ed25519 key,
|
|
symlinks and permissive modes fail closed. `plan` discloses both paths without
|
|
printing key material. The private key must not be broadened into an L2 graph,
|
|
MCP surface, artifact or shared-token boundary.
|
|
|
|
The reviewed Engine source candidate has dedicated server-derived MCP
|
|
plan/apply handling for the exact `ndcDataProductWriterApi` + Data Product
|
|
Publish tuple. It is separate from the generic HTTP safe-ref path and accepts no
|
|
caller-provided provider/scope/credential identity, capability, generation or
|
|
service URL. The production managed flag remains false during staging. In the
|
|
explicitly confirmed activation window it is set to true immediately before
|
|
the Platform EDP artifact; runner acceptance then requires EDP `/healthz` to
|
|
report managed provisioning `enabled`. At that point the old Engine still has
|
|
no signer mount, so the endpoint remains usable only after the separately
|
|
accepted Engine artifact. Root/UI transfer is emergency
|
|
self-hosted diagnostics only, not the user journey or acceptance
|
|
path. A provider-specific daemon remains forbidden.
|
|
|
|
Build the narrow Engine source artifact with:
|
|
|
|
```bash
|
|
node infra/deploy-runner/build-engine-data-product-publish-grant-artifact.mjs \
|
|
engine-data-product-publish-grant-YYYYMMDD-NNN
|
|
```
|
|
|
|
The builder includes exactly the pinned provider-security catalog, the
|
|
Publish-grant service, Engine Agent scope/gateway wiring, the existing n8n
|
|
adapter and the reviewed additive backend runtime overlay. It deliberately
|
|
excludes base Compose, frontend/dist, runtime data, tests, native credentials,
|
|
the NDC L2 process and the legacy generic credential-sink route/core. The
|
|
runner fixes the runtime action to `nodedc-backend` with `--no-deps` and
|
|
`--pull never`; n8n, nginx app, database services and volumes are not selected.
|
|
Acceptance requires backend health, the active immutable backend identity and
|
|
the exact additive mount inventory. Any failure restores the touched source and
|
|
recreates only the previous verified backend runtime.
|
|
|
|
Existing Engine Agents created before Publish-grant support store the original
|
|
nine scopes as the complete developer profile. They must not require a new
|
|
agent, setup command or device credential when the profile gains a server-owned
|
|
capability. Build the compatibility artifact that introduces the durable named
|
|
`full-developer` profile with:
|
|
|
|
```bash
|
|
node infra/deploy-runner/build-engine-agent-full-grant-migration-artifact.mjs \
|
|
engine-agent-full-grant-migration-YYYYMMDD-NNN
|
|
```
|
|
|
|
This follow-up slice contains exactly
|
|
`nodedc-source/server/engineAgents/store.js`. The runner pins its predecessor to
|
|
the successfully applied Publish generation, requires the installed Publish
|
|
overlay and active immutable backend, recreates only `nodedc-backend`, and
|
|
proves the exact candidate SHA, named profile and current expanded runtime
|
|
scope view. Store schema v1 is migrated atomically to v2 only when a grant
|
|
contains the complete legacy nine-scope developer bundle. From then on the
|
|
profile name is the authorization authority and its scope list is derived on
|
|
every read, so capabilities deliberately added to `full-developer` immediately
|
|
apply to existing full grants without token or store migrations. Partial grants
|
|
become `custom` and remain exact; they are never elevated. The artifact does not
|
|
touch UI, n8n, credentials, agent tokens, workflow graphs, databases or runtime
|
|
payloads.
|
|
|
|
`module-foundry` is an independent, authenticated application component. Its
|
|
artifact contains source and compose infrastructure only; its live
|
|
`/volume1/docker/nodedc-platform/module-foundry/source/.env` is root-owned and
|
|
never enters an artifact. The component reuses the existing internal platform
|
|
credential for Launcher handoff validation and requires that runtime
|
|
configuration before its first `apply`.
|
|
|
|
The Foundry ↔ Map Gateway signing key is not an application or `.env` setting.
|
|
On the first relevant `platform` or `module-foundry` apply, the root-owned
|
|
runner creates `/volume1/docker/nodedc-platform/secrets/map-gateway-admin-secret`
|
|
atomically (root:gid 1000, mode `0640`). Both containers receive that file only
|
|
as a read-only mount. The value is never printed, backed up with source,
|
|
included in an artifact, or administered through Foundry.
|
|
|
|
`proxy-contur` is the canonical VPN egress for selected Map Gateway provider
|
|
hosts. Its existing root-owned `PROXY_TOKEN` is copied by the runner into
|
|
`/volume1/docker/nodedc-platform/secrets/map-egress-proxy-token` with
|
|
`root:gid 1000`, mode `0640`, then mounted read-only only into Map Gateway.
|
|
The value is neither printed nor contained in an artifact, Foundry setting, or
|
|
browser response. Apply the `proxy-contur` artifact before the Platform Map
|
|
Gateway artifact: it creates the private `nodedc-map-egress` Docker network.
|
|
|
|
`dc-amd-proxy` is the separate, staged connector for the neighbouring AMD VPN
|
|
machine. Its active artifact attaches only to the private `nodedc-map-egress`
|
|
network, exposes a narrow NAS-LAN pairing port, and has no direct provider
|
|
egress. The runner preserves a `0700`, service-user-owned runtime directory
|
|
for the one-time paired connector credential and synchronizes the existing
|
|
private Map Gateway egress credential as a read-only file. Neither value is
|
|
ever placed in an artifact, `.env`, browser response, or runner output. The
|
|
separate Platform switch is applied only after the connector and pairing are
|
|
verified; it does not alter NAS routes, VPN, DNS, or Tailscale.
|
|
|
|
Install or update the root-owned live runner on Synology:
|
|
|
|
```bash
|
|
# First verify that no deploy process is active and state/deploy.lock is absent.
|
|
sudo install -o root -g root -m 0755 \
|
|
/volume1/docker/nodedc-deploy/runner-install/nodedc-deploy \
|
|
/usr/local/sbin/nodedc-deploy
|
|
sudo /usr/local/sbin/nodedc-deploy verify-install
|
|
```
|
|
|
|
Runner promotion is a standalone admin step, never an app-overlay artifact and
|
|
never part of a running apply. A Python process already executing the old file
|
|
keeps the old code in memory; always invoke a fresh verified process afterward.
|
|
|
|
Normal service deploys must still use explicit artifacts:
|
|
|
|
```bash
|
|
sudo /usr/local/sbin/nodedc-deploy plan /volume1/docker/nodedc-deploy/inbox/<artifact>.tgz
|
|
sudo /usr/local/sbin/nodedc-deploy apply /volume1/docker/nodedc-deploy/inbox/<artifact>.tgz
|
|
```
|