diff --git a/infra/deploy-runner/README.md b/infra/deploy-runner/README.md index a323764..da3e8af 100644 --- a/infra/deploy-runner/README.md +++ b/infra/deploy-runner/README.md @@ -90,10 +90,11 @@ node descriptions used `usableAsTool`, so n8n 2.3.2 exposed six NDC runtime types instead of the required three. It must not be activated, overwritten or deleted. -The corrected candidate is package version `0.1.2`, built with patch id -`n8n-nodes-ndc-release-20260716-003`. It receives a new digest-bound release -directory and remains inert after staging; only a separately reviewed -Engine-owned activator may select it after exact MCP schema acceptance. +The active predecessor is package version `0.1.2` at immutable release +`0.1.2-05e4b38b14b4a019`. History Read is package version `0.1.3`, built with +patch id `n8n-nodes-ndc-history-read-20260717-004` and immutable release +`0.1.3-3354149b5245e39a`. Staging remains inert; only the separately reviewed +Engine-owned transition may select it after exact MCP schema acceptance. The paired Engine activation is built by `build-engine-n8n-private-extension-artifact.mjs`. It deliberately does not @@ -102,7 +103,7 @@ not build an image. A fresh transition id is mandatory and previously issued ids are rejected: ```bash -node infra/deploy-runner/build-engine-n8n-private-extension-artifact.mjs 20260717-004 +node infra/deploy-runner/build-engine-n8n-private-extension-artifact.mjs 20260717-005 ``` The builder emits a narrowly scoped Compose override plus a strict transition @@ -112,7 +113,7 @@ same immutable image ID, and extracts the package into the root-owned, read-only Engine release tree: ```text -/volume2/nodedc-demo/n8n-private-extensions/releases/n8n-nodes-ndc/0.1.2-05e4b38b14b4a019/package +/volume2/nodedc-demo/n8n-private-extensions/releases/n8n-nodes-ndc/0.1.3-3354149b5245e39a/package ``` The override sets `N8N_USER_FOLDER=/home/node`, which is required because the @@ -134,13 +135,16 @@ Postgres service, `.n8n` data, encryption key and credentials remain intact. The apply gate verifies readiness, the running image/version, sealed mount, loader environment, package-loader node/credential sets, scoped loader logs, restart stability and content-exact pinned Engine MCP catalogs. The runner pins -both the complete 434/385 inactive baseline and the reviewed 437/388 -activation catalogs, so a same-count substitution of any built-in schema is -rejected. Any gate failure after +the complete 434/385 inactive baseline and a digest registry for every +reviewed 437/388 active release, so a same-count substitution of any built-in +or private schema is rejected. Upgrade `0.1.2 -> 0.1.3` is accepted only when +the verified live predecessor, descriptor, package mount and catalog all agree. +Any gate failure after mutation automatically restores the pre-apply catalogs/descriptor and force-recreates the previous verified runtime. Staged, sealed and failed -releases are retained. The separate rollback artifact returns the first -activation to the verified inactive 434-node/385-credential catalog baseline. +releases are retained. The paired rollback artifact returns `0.1.3` to the +verified immutable `0.1.2` release and its exact catalogs; it does not invent +an inactive baseline for an already-active upgrade. Run both policy suites before publishing the Engine pair: @@ -151,6 +155,23 @@ PYTHONDONTWRITEBYTECODE=1 \ python3 infra/deploy-runner/test_engine_n8n_private_extension.py ``` +The source-less Engine MCP control-plane update is a separate exact slice: + +```bash +node infra/deploy-runner/build-engine-mcp-control-plane-artifact.mjs 20260717-001 +``` + +Its six-entry registry contains only the Engine Agent gateway, verified graph +patch route, Codex installer source/package and the active node-intelligence +descriptor with the new gateway digest. It force-recreates only the existing +`nodedc-backend`. The node-intelligence image/service, n8n, L1, credentials, +databases and volumes are outside the slice. The runner proves the installed +predecessor digests, exact installer archive/source equality, bounded change +session policy, no-effect/post-write graph barriers, active immutable backend +runtime and descriptor equality. The ordinary overlay backup is the automatic +rollback source; rollback restores the predecessor gateway and descriptor +together before recreating the same backend service. + For `platform` artifacts, the allowlist includes the versioned Ontology Core, the frozen legacy Gelios compatibility service, and the provider-neutral External Data Plane sources. The Gelios service remains reproducible only to diff --git a/infra/deploy-runner/build-engine-agent-full-grant-migration-artifact.mjs b/infra/deploy-runner/build-engine-agent-full-grant-migration-artifact.mjs index c9b19a2..f11fde7 100644 --- a/infra/deploy-runner/build-engine-agent-full-grant-migration-artifact.mjs +++ b/infra/deploy-runner/build-engine-agent-full-grant-migration-artifact.mjs @@ -1,7 +1,7 @@ #!/usr/bin/env node import { createHash } from 'node:crypto' import { spawnSync } from 'node:child_process' -import { cp, lstat, mkdir, mkdtemp, readFile, rm, writeFile } from 'node:fs/promises' +import { lstat, mkdir, mkdtemp, readFile, rm, writeFile } from 'node:fs/promises' import { tmpdir } from 'node:os' import { dirname, join, resolve } from 'node:path' import { fileURLToPath } from 'node:url' @@ -15,6 +15,7 @@ const [patchId = '', ...extra] = process.argv.slice(2) const storeRelativePath = 'nodedc-source/server/engineAgents/store.js' const predecessorSha256 = '52daa43499d6d9a97fe7ffa891edb9212b7791e733f91dd3ca686d42739b7e9a' const targetSha256 = '2e62654c2dc12905efcc83a9dff45a818dd7b47924a10600160835c4416540e9' +const readerExtendedSourceSha256 = 'debd351fe0b8c72b33b8c79909b8f339cbaf061b970576f3ae72df52ebaa211f' const previouslyIssuedPatchIds = new Set([ 'engine-agent-full-grant-migration-20260717-001', ]) @@ -29,7 +30,7 @@ if (previouslyIssuedPatchIds.has(patchId)) { const source = join(engineRoot, storeRelativePath) const sourceInfo = await lstat(source) if (sourceInfo.isSymbolicLink() || !sourceInfo.isFile()) throw new Error('engine_agent_store_source_unsafe') -const sourceBytes = await readFile(source) +const sourceBytes = materializeFrozenTarget(await readFile(source)) if (digest(sourceBytes) !== targetSha256) throw new Error('engine_agent_store_target_sha256_mismatch') const sourceText = sourceBytes.toString('utf8') for (const required of [ @@ -56,7 +57,7 @@ const stage = await mkdtemp(join(tmpdir(), 'nodedc-engine-agent-grant-migration- try { const destination = join(stage, 'payload', storeRelativePath) await mkdir(dirname(destination), { recursive: true }) - await cp(source, destination, { force: false, verbatimSymlinks: true }) + await writeFile(destination, sourceBytes) await writeFile( join(stage, 'manifest.env'), `id=${patchId}\ncomponent=engine\ntype=app-overlay\n`, @@ -119,3 +120,22 @@ function run(command, args) { function digest(bytes) { return createHash('sha256').update(bytes).digest('hex') } + +function materializeFrozenTarget(bytes) { + const sourceSha256 = digest(bytes) + if (sourceSha256 === targetSha256) return bytes + if (sourceSha256 !== readerExtendedSourceSha256) { + throw new Error('engine_agent_store_target_sha256_mismatch') + } + const text = bytes.toString('utf8') + const readerScopes = [ + " 'engine:l2:data-product-read-grant:plan',\n", + " 'engine:l2:data-product-read-grant:write',\n", + ] + let frozen = text + for (const scope of readerScopes) { + if (!frozen.includes(scope)) throw new Error('engine_agent_store_reader_scope_boundary_missing') + frozen = frozen.replace(scope, '') + } + return Buffer.from(frozen, 'utf8') +} diff --git a/infra/deploy-runner/build-engine-data-product-publish-grant-artifact.mjs b/infra/deploy-runner/build-engine-data-product-publish-grant-artifact.mjs index 18b7c73..72c72cd 100644 --- a/infra/deploy-runner/build-engine-data-product-publish-grant-artifact.mjs +++ b/infra/deploy-runner/build-engine-data-product-publish-grant-artifact.mjs @@ -45,7 +45,7 @@ const credentialSinkPredecessorSha256 = Object.freeze({ 'nodedc-source/server/credentialSink/vendor/engine-credential-sink.mjs': 'b4800eead9bf94793ff1280d06e34aad6b37ef055a9d7f8d83d7fb5f9bd66d8b', 'nodedc-source/server/index.js': 'b2b790b02839570d967a2ca68b00e2724485a99389ac9b3a589a1b22302a36b8', 'nodedc-source/server/routes/engineCredentialSink.js': '9cbb69dbc8cbe6181cd5b0170fe9c4d717b0a173866ca98cba3e3766c0bab94e', - 'nodedc-source/server/routes/ndcAgentMcp.js': 'fbb3342b1a617b956d5b3a6a40d5111aa107c1176b4ff89c37b104995bada081', + 'nodedc-source/server/routes/ndcAgentMcp.js': '534e2c85e1faecc72a00da7ad32d0584ee09b6bd89eeec2221c3b23d80e1e962', 'nodedc-source/services/backend/credential-sink/docker-compose.immutable-runtime.yml': '944fa64b08255eb8207b93fd327aebb98ecd9400d37d25fcfa8e3a040ee44afe', }) const publishGrantRuntimeOverride = [ diff --git a/infra/deploy-runner/build-engine-mcp-control-plane-artifact.mjs b/infra/deploy-runner/build-engine-mcp-control-plane-artifact.mjs new file mode 100644 index 0000000..662a527 --- /dev/null +++ b/infra/deploy-runner/build-engine-mcp-control-plane-artifact.mjs @@ -0,0 +1,177 @@ +#!/usr/bin/env node +import { createHash } from "node:crypto"; +import { spawnSync } from "node:child_process"; +import { cp, lstat, mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { dirname, join, resolve } from "node:path"; +import { fileURLToPath } from "node:url"; + +const here = dirname(fileURLToPath(import.meta.url)); +const platformRoot = resolve(here, "../.."); +const engineRoot = resolve( + process.env.NODEDC_ENGINE_SOURCE_ROOT || resolve(platformRoot, "../NODEDC_ENGINE_INFRA"), +); +const canonicalArtifactRoot = resolve(here, "../deploy-artifacts"); +const artifactRoot = resolve(process.env.NODEDC_DEPLOY_ARTIFACT_DIR || canonicalArtifactRoot); +const [transitionId = "20260718-003", ...extra] = process.argv.slice(2); +if (extra.length || !/^\d{8}-[0-9]{3}$/.test(transitionId)) { + throw new Error("usage: build-engine-mcp-control-plane-artifact.mjs [YYYYMMDD-NNN]"); +} + +const id = `engine-mcp-control-plane-${transitionId}`; +const target = join(artifactRoot, `nodedc-${id}.tgz`); +const nodeIntelligenceArtifact = join( + canonicalArtifactRoot, + "nodedc-engine-node-intelligence-20260717-001.tgz", +); +const nodeIntelligenceArtifactSha256 = "d126afaa0c714fef26362aad4749e3f4647f551d6eb975f0133cdf9ff2e6fc4f"; +const descriptorRel = "nodedc-source/services/node-intelligence/activation.json"; +const gatewayRel = "nodedc-source/server/routes/engineAgentGateway.js"; +const upstreamProjectionRel = "nodedc-source/server/nodeIntelligence/upstreamProjection.js"; +const files = [ + "nodedc-source/server/assets/engine-agent-npm/bin/nodedc-engine-codex-agent.mjs", + "nodedc-source/server/assets/engine-agent-npm/package.json", + "nodedc-source/server/assets/nodedc-engine-codex-agent-0.1.4.tgz", + "nodedc-source/server/dataProductPublishGrant/providerCatalog.js", + "nodedc-source/server/dataProductPublishGrant/signedDataPlaneClient.js", + "nodedc-source/server/dataProductReadGrant/acceptance.js", + "nodedc-source/server/dataProductReadGrant/service.js", + "nodedc-source/server/dataProductReadGrant/store.js", + "nodedc-source/server/engineAgents/store.js", + upstreamProjectionRel, + gatewayRel, + "nodedc-source/server/routes/n8n.js", + "nodedc-source/server/routes/ndcAgentMcp.js", + "nodedc-source/services/backend/data-product-read-grant/docker-compose.immutable-runtime.yml", + "nodedc-source/server/dataProductPublishGrant/service.js", + "nodedc-source/server/dataProductPublishGrant/store.js", + descriptorRel, +]; +const expectedSha256 = new Map([ + [files[0], "521098de69fe288a56bd4158c849c1055ba24be08e19f7a4111618d0e8138445"], + [files[1], "cbe113e9b10bb84b9ccbffa3e261908e58a8430c11abe2cf4fd5304741b04597"], + [files[2], "e74c0136d346f904b42589d75cb11a952b4d2f21153f1b057fba28e9acf4f95f"], + [files[3], "901b8fad80018ce177b34ced804b39cb140a47e831414057f484296b373c651d"], + [files[4], "c2a13d5eb49937fe70ec6451d0465cac54c19c7fae44448db099079473ec02fc"], + [files[5], "202dbe575b2f2e1c584aa7e6e99e38fa84f12496feb454e3dbd3f98e2d0396dd"], + [files[6], "65b8cdd6b603333bac1feb303e9791feb1e9da39dc9b5bfc3df3961273b0052e"], + [files[7], "c3fcdc59a794f57d92e3d2ac713ee28341347cebd25364c4060beaa3dc2151de"], + [files[8], "debd351fe0b8c72b33b8c79909b8f339cbaf061b970576f3ae72df52ebaa211f"], + [files[9], "761a874b102a938bc6018159ddacdaac71ad6ae08e9f0f8d7f3b58a0165a5131"], + [files[10], "96c726dab5cf1341f74e6e1095d518058ca320e0dd5738e25bdbe75db1f4fc15"], + [files[11], "f293a7794405badbabd2bf7ef088f96fe1167d9e249f05cbfc8af280a0d3e8f8"], + [files[12], "534e2c85e1faecc72a00da7ad32d0584ee09b6bd89eeec2221c3b23d80e1e962"], + [files[13], "952df0cb2ac477e644f5f3a9d64b4872ce1da33356eeab0bec17dcb2931cf653"], + [files[14], "ded3832c9677345a988448ae8e69cef254fd9bf39d2de20cffa295c1feedcd7c"], + [files[15], "a92303b2732e21f68c1ac732fa26e4983cbadf3515741cee983b916a283d754c"], +]); + +await assertFresh(target); +assertSha(await readFile(nodeIntelligenceArtifact), nodeIntelligenceArtifactSha256, "node intelligence artifact"); +const stage = await mkdtemp(join(tmpdir(), "nodedc-engine-mcp-control-plane-")); +const payload = join(stage, "payload"); +try { + await mkdir(payload, { recursive: true }); + for (const rel of files.slice(0, -1)) { + const source = join(engineRoot, rel); + const stat = await lstat(source); + if (!stat.isFile() || stat.isSymbolicLink()) throw new Error(`source_boundary_invalid:${rel}`); + assertSha(await readFile(source), expectedSha256.get(rel), rel); + await mkdir(dirname(join(payload, rel)), { recursive: true }); + await cp(source, join(payload, rel), { force: false }); + } + + const descriptorText = extractMember( + nodeIntelligenceArtifact, + `payload/${descriptorRel}`, + ); + const descriptor = JSON.parse(descriptorText); + if ( + descriptor?.action !== "activate" + || descriptor?.releaseId !== "2.33.2-974a9fb3492f" + || descriptor?.source?.gatewaySha256 !== "9642c76fd5765a8a20629c8d09e16579a1c3b2cfb7bdbab38cf55af469d8908f" + ) throw new Error("node_intelligence_predecessor_descriptor_mismatch"); + descriptor.source.gatewaySha256 = expectedSha256.get(gatewayRel); + descriptor.source.upstreamProjectionSha256 = expectedSha256.get(upstreamProjectionRel); + await mkdir(dirname(join(payload, descriptorRel)), { recursive: true }); + await writeFile(join(payload, descriptorRel), `${JSON.stringify(descriptor, null, 2)}\n`, "utf8"); + + await writeFile(join(stage, "manifest.env"), `id=${id}\ncomponent=engine\ntype=app-overlay\n`, "utf8"); + await writeFile(join(stage, "files.txt"), `${files.join("\n")}\n`, "utf8"); + await mkdir(artifactRoot, { recursive: true }); + run("python3", ["-c", canonicalTarScript(), target, stage]); + const artifactSha256 = sha(await readFile(target)); + console.log(JSON.stringify({ + ok: true, + id, + artifact: target, + artifactSha256, + services: ["nodedc-backend"], + predecessorGatewaySha256: "9642c76fd5765a8a20629c8d09e16579a1c3b2cfb7bdbab38cf55af469d8908f", + targetGatewaySha256: expectedSha256.get(gatewayRel), + mcpVersion: "0.5.0", + installerVersion: "0.1.4", + files, + }, null, 2)); +} finally { + await rm(stage, { recursive: true, force: true }); +} + +async function assertFresh(path) { + try { + await lstat(path); + } catch (error) { + if (error?.code === "ENOENT") return; + throw error; + } + throw new Error("artifact_already_exists"); +} + +function extractMember(archive, member) { + const script = [ + "import pathlib,sys,tarfile", + "p=pathlib.Path(sys.argv[1]); name=sys.argv[2]", + "with tarfile.open(p,'r:gz') as t:", + " m=t.getmember(name)", + " if not m.isfile(): raise SystemExit('member-not-file')", + " f=t.extractfile(m)", + " if f is None: raise SystemExit('member-unreadable')", + " sys.stdout.buffer.write(f.read())", + ].join("\n"); + return run("python3", ["-c", script, archive, member]).stdout; +} + +function canonicalTarScript() { + return [ + "import gzip,io,pathlib,sys,tarfile", + "root=pathlib.Path(sys.argv[2])", + "with open(sys.argv[1],'wb') as out:", + " with gzip.GzipFile(filename='',mode='wb',fileobj=out,compresslevel=9,mtime=0) as gz:", + " with tarfile.open(fileobj=gz,mode='w',format=tarfile.PAX_FORMAT) as tar:", + " for top in ('manifest.env','files.txt','payload'):", + " p=root/top; paths=[p]+(sorted(p.rglob('*')) if p.is_dir() else [])", + " for x in paths:", + " info=tar.gettarinfo(str(x),arcname=x.relative_to(root).as_posix())", + " info.uid=info.gid=0; info.uname=info.gname='root'; info.mtime=0; info.mode=0o755 if info.isdir() else 0o644", + " with (open(x,'rb') if info.isfile() else io.BytesIO()) as src: tar.addfile(info,src if info.isfile() else None)", + ].join("\n"); +} + +function assertSha(bytes, expected, label) { + const actual = sha(bytes); + if (!expected || actual !== expected) throw new Error(`${label.replaceAll(" ", "_")}_sha256_mismatch:${actual}`); +} + +function sha(bytes) { + return createHash("sha256").update(bytes).digest("hex"); +} + +function run(command, args) { + const result = spawnSync(command, args, { + encoding: "utf8", + maxBuffer: 128 * 1024 * 1024, + stdio: ["ignore", "pipe", "pipe"], + }); + if (result.status !== 0) throw new Error(`${command}_failed:${result.stderr || result.stdout}`); + return result; +} diff --git a/infra/deploy-runner/build-engine-n8n-private-extension-artifact.mjs b/infra/deploy-runner/build-engine-n8n-private-extension-artifact.mjs index ae2d5f2..bb9c436 100644 --- a/infra/deploy-runner/build-engine-n8n-private-extension-artifact.mjs +++ b/infra/deploy-runner/build-engine-n8n-private-extension-artifact.mjs @@ -13,18 +13,19 @@ const engineRoot = resolve(platformRoot, "../NODEDC_ENGINE_INFRA"); const artifactRoot = resolve(process.env.NODEDC_DEPLOY_ARTIFACT_DIR || join(here, "../deploy-artifacts")); const stageArtifact = resolve( process.env.NODEDC_N8N_EXTENSION_STAGE_ARTIFACT - || join(artifactRoot, "nodedc-n8n-private-extension-n8n-nodes-ndc-release-20260716-003.tgz"), + || join(artifactRoot, "nodedc-n8n-private-extension-n8n-nodes-ndc-history-read-20260717-004.tgz"), ); -const stageArtifactSha256 = "4601c16c57e5182996adf18d0163837511b27ab3f7cfdf97eac679418ee2d078"; -const releaseId = "0.1.2-05e4b38b14b4a019"; -const packageVersion = "0.1.2"; -const packageSha256 = "05e4b38b14b4a019ce1f6eee27b9e320094cb3560903bd68074966b3a1267af5"; +const stageArtifactSha256 = "a78f8250704e4893eab727cdb862b69ce45a84310fbb73ecd3dfd968b72d12ee"; +const releaseId = "0.1.3-3354149b5245e39a"; +const packageVersion = "0.1.3"; +const packageSha256 = "3354149b5245e39a10f6f03a4b43796602d7e57ea1f91dcc6cb6774ead97501d"; const n8nVersion = "2.3.2"; const baseImage = "docker.n8n.io/n8nio/n8n:2.3.2"; const architecture = "amd64"; -const generatedAt = "2026-07-15T21:51:41.000Z"; -const previouslyIssuedTransitionIds = new Set(["20260715-002", "20260716-003"]); +const generatedAt = "2026-07-17T21:00:00.000Z"; +const predecessorGitRevision = "96ad46e3c62943f818233481957c62e5088ae35c"; +const previouslyIssuedTransitionIds = new Set(["20260715-002", "20260716-003", "20260717-004"]); const transitionId = readTransitionId(process.argv.slice(2), process.env.NODEDC_N8N_TRANSITION_ID); const activationId = `engine-n8n-private-extension-${transitionId}`; const rollbackId = `engine-n8n-private-extension-rollback-${transitionId}`; @@ -39,7 +40,6 @@ const metaRel = `${schemaRoot}/meta.json`; const iconRoot = "nodedc-source/server/assets/n8n/icons"; const iconRel = `${iconRoot}/ndc.svg`; const darkIconRel = `${iconRoot}/ndc.dark.svg`; -const sealedReleaseRelativePath = `n8n-private-extensions/releases/n8n-nodes-ndc/${releaseId}/package`; const runtimePackagePath = "/home/node/.n8n/nodes/node_modules/n8n-nodes-ndc"; const expectedNodeTypes = [ @@ -111,10 +111,23 @@ try { assertExact(privateNodes.map((item) => item.name), expectedNodeTypes, "node types"); assertExact(privateCredentials.map((item) => item.name), expectedCredentialTypes, "credential types"); - const baselineNodes = JSON.parse(gitFile(nodesCatalogRel)); - const baselineCredentials = JSON.parse(gitFile(credentialsCatalogRel)); - const baselineMeta = JSON.parse(gitFile(metaRel)); - assertBaselineCatalogs(baselineNodes, baselineCredentials, baselineMeta); + const predecessorDescriptor = JSON.parse(gitFile(descriptorRel)); + const predecessorNodes = JSON.parse(gitFile(nodesCatalogRel)); + const predecessorCredentials = JSON.parse(gitFile(credentialsCatalogRel)); + const predecessorMeta = JSON.parse(gitFile(metaRel)); + assertPredecessorCatalogs( + predecessorDescriptor, + predecessorNodes, + predecessorCredentials, + predecessorMeta, + ); + const baselineNodes = predecessorNodes.filter( + (item) => !String(item?.name || "").startsWith("n8n-nodes-ndc."), + ); + const baselineCredentials = predecessorCredentials.filter( + (item) => !expectedCredentialTypes.includes(String(item?.name || "")), + ); + assertBaselineCatalogs(baselineNodes, baselineCredentials); const activeNodes = [...baselineNodes, ...privateNodes]; const activeCredentials = [...baselineCredentials, ...privateCredentials]; const activeMeta = { @@ -125,9 +138,20 @@ try { credentialCount: activeCredentials.length, }; - const activationDescriptor = descriptor("activate", expectedNodeTypes, expectedCredentialTypes, "verified_inactive"); - const rollbackDescriptor = descriptor("rollback-inactive", [], [], releaseId); - const override = composeOverride(); + const target = { + releaseId, + packageVersion, + packageSha256, + }; + const predecessor = { + releaseId: predecessorDescriptor.releaseId, + packageVersion: predecessorDescriptor.packageVersion, + packageSha256: predecessorDescriptor.packageSha256, + }; + const activationDescriptor = descriptor(target, predecessor.releaseId, predecessor.releaseId); + const rollbackDescriptor = descriptor(predecessor, releaseId, releaseId); + const override = composeOverride(target); + const rollbackOverride = composeOverride(predecessor); const generatedRoot = join(work, "generated-engine-payload"); await writeJson(join(generatedRoot, nodesCatalogRel), activeNodes); @@ -154,13 +178,16 @@ try { } }); - const rollbackEntries = [descriptorRel, nodesCatalogRel, credentialsCatalogRel, metaRel]; + const rollbackEntries = [...activationEntries]; const rollbackArtifact = await buildArtifact(work, rollbackId, rollbackEntries, async (payload) => { await writeJson(join(payload, descriptorRel), rollbackDescriptor); - await mkdir(join(payload, schemaRoot), { recursive: true }); - await writeFile(join(payload, nodesCatalogRel), `${JSON.stringify(baselineNodes, null, 2)}\n`, "utf8"); - await writeFile(join(payload, credentialsCatalogRel), `${JSON.stringify(baselineCredentials, null, 2)}\n`, "utf8"); - await writeFile(join(payload, metaRel), `${JSON.stringify(baselineMeta, null, 2)}\n`, "utf8"); + await writeFile(join(payload, overrideRel), rollbackOverride, "utf8"); + await writeJson(join(payload, nodesCatalogRel), predecessorNodes); + await writeJson(join(payload, credentialsCatalogRel), predecessorCredentials); + await writeJson(join(payload, metaRel), predecessorMeta); + await mkdir(join(payload, iconRoot), { recursive: true }); + await cp(join(engineRoot, iconRel), join(payload, iconRel), { force: false }); + await cp(join(engineRoot, darkIconRel), join(payload, darkIconRel), { force: false }); }); console.log(JSON.stringify({ @@ -177,25 +204,25 @@ try { await rm(work, { recursive: true, force: true }); } -function descriptor(action, nodeTypes, credentialTypes, expectedCurrent) { +function descriptor(target, expectedCurrent, rollbackBaseline) { return { schemaVersion: "nodedc.engine-n8n-private-extension-transition/v1", - action, - releaseId, - packageVersion, - packageSha256, + action: "activate", + releaseId: target.releaseId, + packageVersion: target.packageVersion, + packageSha256: target.packageSha256, n8nVersion, baseImage, baseImageArchitecture: architecture, baseImageIdentityPolicy: "running-container-and-local-tag-must-match", - sealedReleaseRelativePath, + sealedReleaseRelativePath: `n8n-private-extensions/releases/n8n-nodes-ndc/${target.releaseId}/package`, composeOverride: overrideRel, runtimePackagePath, topologyServices: ["n8n"], expectedCurrent, - expectedNodeTypes: nodeTypes, - expectedCredentialTypes: credentialTypes, - rollbackBaseline: "verified_inactive", + expectedNodeTypes, + expectedCredentialTypes, + rollbackBaseline, }; } @@ -235,8 +262,9 @@ async function assertArtifactTargetFresh(path) { throw new Error("transition_artifact_already_exists"); } -function composeOverride() { +function composeOverride(target) { const health = "const http=require('http');const req=http.get('http://127.0.0.1:5678/healthz/readiness',r=>{r.resume();process.exit(r.statusCode===200?0:1)});req.on('error',()=>process.exit(1));req.setTimeout(4000,()=>{req.destroy();process.exit(1)});"; + const sealedReleaseRelativePath = `n8n-private-extensions/releases/n8n-nodes-ndc/${target.releaseId}/package`; return [ "services:", " n8n:", @@ -257,8 +285,8 @@ function composeOverride() { " retries: 30", " start_period: 30s", " labels:", - ` nodedc.n8n-private-extension.release: ${releaseId}`, - ` nodedc.n8n-private-extension.package-sha256: ${packageSha256}`, + ` nodedc.n8n-private-extension.release: ${target.releaseId}`, + ` nodedc.n8n-private-extension.package-sha256: ${target.packageSha256}`, "", ].join("\n"); } @@ -297,7 +325,34 @@ function assertPackage(value) { assertExact(value.n8n?.credentials, credentialModules.map(([path]) => path), "package credentials"); } -function assertBaselineCatalogs(nodes, credentials, meta) { +function assertPredecessorCatalogs(descriptorValue, nodes, credentials, meta) { + if (descriptorValue?.action !== "activate" + || descriptorValue?.releaseId !== "0.1.2-05e4b38b14b4a019" + || descriptorValue?.packageVersion !== "0.1.2" + || descriptorValue?.packageSha256 !== "05e4b38b14b4a019ce1f6eee27b9e320094cb3560903bd68074966b3a1267af5") { + throw new Error("predecessor_descriptor_mismatch"); + } + if (!Array.isArray(nodes) || nodes.length !== 437 + || !Array.isArray(credentials) || credentials.length !== 388 + || meta?.n8nVersion !== n8nVersion + || meta?.source !== "n8n-core+n8n-nodes-ndc@0.1.2" + || meta?.nodeCount !== 437 + || meta?.credentialCount !== 388) { + throw new Error("predecessor_catalog_mismatch"); + } + assertExact( + nodes.filter((item) => String(item?.name || "").startsWith("n8n-nodes-ndc.")).map((item) => item.name), + expectedNodeTypes, + "predecessor node types", + ); + assertExact( + credentials.filter((item) => expectedCredentialTypes.includes(String(item?.name || ""))).map((item) => item.name), + expectedCredentialTypes, + "predecessor credential types", + ); +} + +function assertBaselineCatalogs(nodes, credentials) { if (!Array.isArray(nodes) || nodes.length !== 434 || nodes.some((item) => String(item?.name || "").startsWith("n8n-nodes-ndc."))) { throw new Error("baseline_node_catalog_mismatch"); } @@ -305,9 +360,6 @@ function assertBaselineCatalogs(nodes, credentials, meta) { || credentials.some((item) => expectedCredentialTypes.includes(String(item?.name || "")))) { throw new Error("baseline_credential_catalog_mismatch"); } - if (meta?.n8nVersion !== n8nVersion || meta?.nodeCount !== 434 || meta?.credentialCount !== 385) { - throw new Error("baseline_meta_mismatch"); - } } function assertEngineBaseline(compose) { @@ -330,7 +382,7 @@ function assertSha(bytes, expected, label) { } function gitFile(rel) { - return run("git", ["show", `HEAD:${rel}`], engineRoot).stdout; + return run("git", ["show", `${predecessorGitRevision}:${rel}`], engineRoot).stdout; } async function writeJson(path, value) { diff --git a/infra/deploy-runner/build-external-data-plane-artifact.mjs b/infra/deploy-runner/build-external-data-plane-artifact.mjs index 5c2595c..bd13452 100644 --- a/infra/deploy-runner/build-external-data-plane-artifact.mjs +++ b/infra/deploy-runner/build-external-data-plane-artifact.mjs @@ -23,6 +23,7 @@ const files = [ ["packages/external-provider-contract/src/data-plane.mjs", "platform/packages/external-provider-contract/src/data-plane.mjs"], ["packages/external-provider-contract/src/data-product.mjs", "platform/packages/external-provider-contract/src/data-product.mjs"], ["packages/external-provider-contract/src/intake-batch.mjs", "platform/packages/external-provider-contract/src/intake-batch.mjs"], + ["packages/external-provider-contract/src/index.mjs", "platform/packages/external-provider-contract/src/index.mjs"], ["packages/external-provider-contract/src/sensitive-field-policy.mjs", "platform/packages/external-provider-contract/src/sensitive-field-policy.mjs"], ]; const ignoredBasenames = new Set([".DS_Store", ".git", "node_modules"]); @@ -84,6 +85,21 @@ async function assertSourceBoundary() { compose.includes("private-key.pem") || compose.includes("ENGINE_EDP_MANAGED_PROVISIONER_PRIVATE_KEY_FILE") ) throw new Error("platform_artifact_private_key_boundary_violation"); + const server = await readFile( + resolve(platformRoot, "services/external-data-plane/src/server.mjs"), + "utf8", + ); + for (const marker of [ + 'managedWriterBindingLifetime: config.managedProvisionerApiEnabled ? "explicit-revoke" : "disabled"', + 'managedWriterBindings: "digest+idempotent-generation+explicit-revoke"', + 'app.put("/internal/data-plane/v1/reader-bindings/by-key/:bindingKey"', + 'managedReaderBindingProvisioning: config.managedProvisionerApiEnabled', + 'managedReaderBindingLifetime: config.managedProvisionerApiEnabled ? "explicit-revoke" : "disabled"', + 'managedReaderBindings: "digest+idempotent-generation+explicit-revoke"', + 'where id = $1 and binding_key is null and active = true', + ]) { + if (!server.includes(marker)) throw new Error(`managed_reader_boundary_missing:${marker}`); + } } function canonicalTarScript() { diff --git a/infra/deploy-runner/build-module-foundry-artifact.mjs b/infra/deploy-runner/build-module-foundry-artifact.mjs index cf95dc1..1ad0804 100644 --- a/infra/deploy-runner/build-module-foundry-artifact.mjs +++ b/infra/deploy-runner/build-module-foundry-artifact.mjs @@ -10,10 +10,10 @@ const scriptDir = dirname(fileURLToPath(import.meta.url)); const platformRoot = resolve(scriptDir, "../.."); const workspaceRoot = resolve(platformRoot, ".."); const foundryRoot = resolve(workspaceRoot, "NODEDC_DESIGN_GUIDELINE"); -const artifactDir = resolve(scriptDir, "../deploy-artifacts"); -const patchId = process.argv[2] || "module-foundry-bootstrap-20260714-001"; +const artifactDir = resolve(process.env.NODEDC_DEPLOY_ARTIFACT_DIR || resolve(scriptDir, "../deploy-artifacts")); +const [patchId = "module-foundry-bootstrap-20260714-001", ...extra] = process.argv.slice(2); -if (!/^[A-Za-z0-9._-]{1,96}$/.test(patchId)) { +if (extra.length || !/^[A-Za-z0-9._-]{1,96}$/.test(patchId)) { throw new Error("patch_id_must_contain_only_letters_digits_dot_underscore_hyphen"); } @@ -48,13 +48,9 @@ try { await writeFile(join(stage, "files.txt"), `${files.join("\n")}\n`, "utf8"); await mkdir(artifactDir, { recursive: true }); - const tar = spawnSync("python3", ["-c", [ - "import sys, tarfile", - "with tarfile.open(sys.argv[1], 'w:gz', format=tarfile.PAX_FORMAT) as archive:", - " [archive.add(name, arcname=name, recursive=True) for name in ('manifest.env', 'files.txt', 'payload')]", - ].join("\n"), target], { - cwd: stage, + const tar = spawnSync("python3", ["-c", canonicalTarScript(), target, stage], { encoding: "utf8", + maxBuffer: 128 * 1024 * 1024, }); if (tar.status !== 0) throw new Error(`tar_failed:${tar.stderr || tar.stdout}`); @@ -64,6 +60,22 @@ try { await rm(stage, { recursive: true, force: true }); } +function canonicalTarScript() { + return [ + "import gzip,io,pathlib,sys,tarfile", + "root=pathlib.Path(sys.argv[2])", + "with open(sys.argv[1],'wb') as out:", + " with gzip.GzipFile(filename='',mode='wb',fileobj=out,compresslevel=9,mtime=0) as gz:", + " with tarfile.open(fileobj=gz,mode='w',format=tarfile.PAX_FORMAT) as tar:", + " for top in ('manifest.env','files.txt','payload'):", + " p=root/top; paths=[p]+(sorted(p.rglob('*')) if p.is_dir() else [])", + " for x in paths:", + " info=tar.gettarinfo(str(x),arcname=x.relative_to(root).as_posix())", + " info.uid=info.gid=0; info.uname=info.gname='root'; info.mtime=0; info.mode=0o755 if info.isdir() else 0o644", + " with (open(x,'rb') if info.isfile() else io.BytesIO()) as src: tar.addfile(info,src if info.isfile() else None)", + ].join("\n"); +} + async function copySafe(source, destination) { const sourceStat = await lstat(source); if (sourceStat.isSymbolicLink()) throw new Error(`source_symlink_rejected:${relative(foundryRoot, source)}`); diff --git a/infra/deploy-runner/build-n8n-private-extension-artifact.mjs b/infra/deploy-runner/build-n8n-private-extension-artifact.mjs index 48f2ee2..55c978f 100644 --- a/infra/deploy-runner/build-n8n-private-extension-artifact.mjs +++ b/infra/deploy-runner/build-n8n-private-extension-artifact.mjs @@ -12,8 +12,8 @@ const platformRoot = resolve(scriptDir, "../.."); const packageRoot = resolve(platformRoot, "packages/n8n-nodes-ndc"); const artifactDir = resolve(process.env.NODEDC_DEPLOY_ARTIFACT_DIR || resolve(scriptDir, "../deploy-artifacts")); const requireModule = createRequire(import.meta.url); -const expectedPackageVersion = "0.1.2"; -const [patchId = "n8n-nodes-ndc-release-20260716-003", ...extra] = process.argv.slice(2); +const expectedPackageVersion = "0.1.3"; +const [patchId = "n8n-nodes-ndc-history-read-20260717-004", ...extra] = process.argv.slice(2); const expectedRuntimeNodes = [ { file: "dist/nodes/NdcDataProductPublish/NdcDataProductPublish.node.js", diff --git a/infra/deploy-runner/nodedc-deploy b/infra/deploy-runner/nodedc-deploy index f4ea532..bb8bdfd 100755 --- a/infra/deploy-runner/nodedc-deploy +++ b/infra/deploy-runner/nodedc-deploy @@ -54,6 +54,7 @@ ENGINE_CREDENTIAL_BACKEND_ROOTFS_FILE = ENGINE_CREDENTIAL_BACKEND_RUNTIME_DIR / ENGINE_CREDENTIAL_BACKEND_METADATA_FILE = ENGINE_CREDENTIAL_BACKEND_RUNTIME_DIR / "runtime-metadata.json" ENGINE_CREDENTIAL_BACKEND_ACTIVATION_FILE = ENGINE_CREDENTIAL_BACKEND_RUNTIME_DIR / "activation.ready" ENGINE_DATA_PRODUCT_PUBLISH_GRANT_OVERRIDE_REL = "nodedc-source/services/backend/data-product-publish-grant/docker-compose.immutable-runtime.yml" +ENGINE_DATA_PRODUCT_READ_GRANT_OVERRIDE_REL = "nodedc-source/services/backend/data-product-read-grant/docker-compose.immutable-runtime.yml" ENGINE_AGENT_FULL_GRANT_MIGRATION_STORE_REL = "nodedc-source/server/engineAgents/store.js" ENGINE_AGENT_FULL_GRANT_MIGRATION_PREDECESSOR_SHA256 = "52daa43499d6d9a97fe7ffa891edb9212b7791e733f91dd3ca686d42739b7e9a" ENGINE_AGENT_FULL_GRANT_MIGRATION_TARGET_SHA256 = "2e62654c2dc12905efcc83a9dff45a818dd7b47924a10600160835c4416540e9" @@ -115,13 +116,74 @@ ENGINE_NODE_INTELLIGENCE_ROLLBACK_ENTRIES = ( ENGINE_NODE_INTELLIGENCE_GATEWAY_REL, ENGINE_NODE_INTELLIGENCE_SERVICE_ROOT_REL, ) +ENGINE_MCP_CONTROL_PLANE_DESCRIPTOR_REL = ENGINE_NODE_INTELLIGENCE_DESCRIPTOR_REL +ENGINE_MCP_CONTROL_PLANE_ARTIFACT_ENTRIES = ( + "nodedc-source/server/assets/engine-agent-npm/bin/nodedc-engine-codex-agent.mjs", + "nodedc-source/server/assets/engine-agent-npm/package.json", + "nodedc-source/server/assets/nodedc-engine-codex-agent-0.1.4.tgz", + "nodedc-source/server/dataProductPublishGrant/providerCatalog.js", + "nodedc-source/server/dataProductPublishGrant/signedDataPlaneClient.js", + "nodedc-source/server/dataProductReadGrant/acceptance.js", + "nodedc-source/server/dataProductReadGrant/service.js", + "nodedc-source/server/dataProductReadGrant/store.js", + "nodedc-source/server/engineAgents/store.js", + "nodedc-source/server/nodeIntelligence/upstreamProjection.js", + ENGINE_NODE_INTELLIGENCE_GATEWAY_REL, + "nodedc-source/server/routes/n8n.js", + "nodedc-source/server/routes/ndcAgentMcp.js", + ENGINE_DATA_PRODUCT_READ_GRANT_OVERRIDE_REL, + "nodedc-source/server/dataProductPublishGrant/service.js", + "nodedc-source/server/dataProductPublishGrant/store.js", + ENGINE_MCP_CONTROL_PLANE_DESCRIPTOR_REL, +) +ENGINE_MCP_CONTROL_PLANE_PREDECESSOR_SHA256 = { + ENGINE_NODE_INTELLIGENCE_GATEWAY_REL: "604a75ad3b9b463cea5e578760c8a23592e2e00b8acfc0099730a347e27bf4e8", + "nodedc-source/server/routes/ndcAgentMcp.js": "534e2c85e1faecc72a00da7ad32d0584ee09b6bd89eeec2221c3b23d80e1e962", + "nodedc-source/server/routes/n8n.js": "de6e3c76740af86e2eaeedede140b5ee54eb526f03db270324d7a4fe513f6817", + "nodedc-source/server/dataProductPublishGrant/providerCatalog.js": "a96d3695fdb3ed7012b9041182f37b09e32d1dfa0dbb391d8db2d7d25a39d64d", + "nodedc-source/server/dataProductPublishGrant/signedDataPlaneClient.js": "1a26728ac69c5ef3fdce1ce1154a6ce330f72286b6e07825796a246070d15bbf", + "nodedc-source/server/engineAgents/store.js": "2e62654c2dc12905efcc83a9dff45a818dd7b47924a10600160835c4416540e9", + "nodedc-source/server/nodeIntelligence/upstreamProjection.js": "946353ed8582c5d96e08361603425cc45eaf231f730ae46a75028d3bef352ee9", + "nodedc-source/server/assets/engine-agent-npm/bin/nodedc-engine-codex-agent.mjs": "7d3d7a9bc48614e6cd238199caa384073096b7ec8273703833681151b464ef17", + "nodedc-source/server/assets/engine-agent-npm/package.json": "9f299b20db4855b1bb4a9bd2e1095dfd059cd90ec82e936884a380aa41249e28", + "nodedc-source/server/dataProductPublishGrant/service.js": "85ded82d707e0ef5b1ad739b5a9be24c1432e2dd9e5051108739f09daf20f47c", + "nodedc-source/server/dataProductPublishGrant/store.js": "ef2a5e6bfad9f0b1710db006a0d30d01cd9bcaf06524b0e9f975dd771efe3952", +} +ENGINE_MCP_CONTROL_PLANE_TARGET_SHA256 = { + ENGINE_NODE_INTELLIGENCE_GATEWAY_REL: "96c726dab5cf1341f74e6e1095d518058ca320e0dd5738e25bdbe75db1f4fc15", + "nodedc-source/server/routes/ndcAgentMcp.js": "534e2c85e1faecc72a00da7ad32d0584ee09b6bd89eeec2221c3b23d80e1e962", + "nodedc-source/server/routes/n8n.js": "f293a7794405badbabd2bf7ef088f96fe1167d9e249f05cbfc8af280a0d3e8f8", + "nodedc-source/server/dataProductPublishGrant/providerCatalog.js": "901b8fad80018ce177b34ced804b39cb140a47e831414057f484296b373c651d", + "nodedc-source/server/dataProductPublishGrant/signedDataPlaneClient.js": "c2a13d5eb49937fe70ec6451d0465cac54c19c7fae44448db099079473ec02fc", + "nodedc-source/server/dataProductReadGrant/acceptance.js": "202dbe575b2f2e1c584aa7e6e99e38fa84f12496feb454e3dbd3f98e2d0396dd", + "nodedc-source/server/dataProductReadGrant/service.js": "65b8cdd6b603333bac1feb303e9791feb1e9da39dc9b5bfc3df3961273b0052e", + "nodedc-source/server/dataProductReadGrant/store.js": "c3fcdc59a794f57d92e3d2ac713ee28341347cebd25364c4060beaa3dc2151de", + "nodedc-source/server/engineAgents/store.js": "debd351fe0b8c72b33b8c79909b8f339cbaf061b970576f3ae72df52ebaa211f", + "nodedc-source/server/nodeIntelligence/upstreamProjection.js": "761a874b102a938bc6018159ddacdaac71ad6ae08e9f0f8d7f3b58a0165a5131", + "nodedc-source/server/assets/engine-agent-npm/bin/nodedc-engine-codex-agent.mjs": "521098de69fe288a56bd4158c849c1055ba24be08e19f7a4111618d0e8138445", + "nodedc-source/server/assets/engine-agent-npm/package.json": "cbe113e9b10bb84b9ccbffa3e261908e58a8430c11abe2cf4fd5304741b04597", + "nodedc-source/server/assets/nodedc-engine-codex-agent-0.1.4.tgz": "e74c0136d346f904b42589d75cb11a952b4d2f21153f1b057fba28e9acf4f95f", + ENGINE_DATA_PRODUCT_READ_GRANT_OVERRIDE_REL: "952df0cb2ac477e644f5f3a9d64b4872ce1da33356eeab0bec17dcb2931cf653", + "nodedc-source/server/dataProductPublishGrant/service.js": "ded3832c9677345a988448ae8e69cef254fd9bf39d2de20cffa295c1feedcd7c", + "nodedc-source/server/dataProductPublishGrant/store.js": "a92303b2732e21f68c1ac732fa26e4983cbadf3515741cee983b916a283d754c", +} +ENGINE_MCP_CONTROL_PLANE_NEW_PATHS = ( + "nodedc-source/server/assets/nodedc-engine-codex-agent-0.1.4.tgz", + "nodedc-source/server/dataProductReadGrant/acceptance.js", + "nodedc-source/server/dataProductReadGrant/service.js", + "nodedc-source/server/dataProductReadGrant/store.js", + ENGINE_DATA_PRODUCT_READ_GRANT_OVERRIDE_REL, +) ENGINE_CONTROL_PLANE_STATE_REL = "nodedc-control-plane" ENGINE_PUBLISH_GRANT_STATE_REL = f"{ENGINE_CONTROL_PLANE_STATE_REL}/publish-grants" +ENGINE_READ_GRANT_STATE_REL = f"{ENGINE_CONTROL_PLANE_STATE_REL}/read-grants" ENGINE_CONTROL_PLANE_STATE_PATH = Path("/volume2/nodedc-demo/nodedc-control-plane") ENGINE_PUBLISH_GRANT_STATE_PATH = ENGINE_CONTROL_PLANE_STATE_PATH / "publish-grants" +ENGINE_READ_GRANT_STATE_PATH = ENGINE_CONTROL_PLANE_STATE_PATH / "read-grants" ENGINE_EDP_PRIVATE_KEY_CONTAINER_PATH = "/run/nodedc-secrets/engine-edp-managed-provisioner-private.pem" ENGINE_CONTROL_PLANE_CONTAINER_PATH = "/var/lib/nodedc-control-plane" ENGINE_PUBLISH_GRANT_CONTAINER_PATH = "/var/lib/nodedc-control-plane/publish-grants" +ENGINE_READ_GRANT_CONTAINER_PATH = "/var/lib/nodedc-control-plane/read-grants" EXTERNAL_DATA_PLANE_MANAGED_TRUST_CONTAINER_PATH = "/run/nodedc-trust/engine-managed-provisioner" EXTERNAL_DATA_PLANE_INTERNAL_URL = "http://external-data-plane:18106" PLATFORM_EXTERNAL_DATA_PLANE_COMPOSE_REL = "platform/docker-compose.external-data-plane.yml" @@ -135,9 +197,18 @@ ENGINE_N8N_RUNTIME_PACKAGE_PATH = "/home/node/.n8n/nodes/node_modules/n8n-nodes- ENGINE_N8N_NODE_MODULES_PATH = "/usr/local/lib/node_modules/n8n/node_modules" ENGINE_N8N_ICON_SHA256 = "1c928fc996d8b82121e8f8e327d74b1dd21556c106de99c5e8d317d926c0ba17" ENGINE_N8N_DARK_ICON_SHA256 = "ef3b8551da4ce04527736405afa9a220a2c76d047bd18f6f7124b9adb4216b0c" -ENGINE_N8N_ACTIVATION_NODES_CATALOG_JSON_SHA256 = "230f712230f7ee8debaa4b44a4358cc19c205bc43305d8ed89d5e3daac466506" -ENGINE_N8N_ACTIVATION_CREDENTIALS_CATALOG_JSON_SHA256 = "a26824ffc0e15db857c792153de3417febc0dbba4880380d6c8cd544b3c80f4d" -ENGINE_N8N_ACTIVATION_META_JSON_SHA256 = "caf7635420c61333e65f68350f5567217aed96fb51354b38764bcac2c24e7966" +ENGINE_N8N_RELEASE_CATALOG_JSON_SHA256 = { + "0.1.2-05e4b38b14b4a019": { + "nodes": "230f712230f7ee8debaa4b44a4358cc19c205bc43305d8ed89d5e3daac466506", + "credentials": "a26824ffc0e15db857c792153de3417febc0dbba4880380d6c8cd544b3c80f4d", + "meta": "caf7635420c61333e65f68350f5567217aed96fb51354b38764bcac2c24e7966", + }, + "0.1.3-3354149b5245e39a": { + "nodes": "b0a5215699a6e691b8cfc505ab457d5632ef6d83adb3537520d0b60760ba16b2", + "credentials": "a26824ffc0e15db857c792153de3417febc0dbba4880380d6c8cd544b3c80f4d", + "meta": "b8aa60c0d919573626fa0694a1e4ae9ede015325a5d312e3e05d5ac293084aa7", + }, +} ENGINE_N8N_INACTIVE_NODES_CATALOG_JSON_SHA256 = "b70d9d8130d498c55de46a5d0758c844f1242b70803457b2291ab3a9de8056f2" ENGINE_N8N_INACTIVE_CREDENTIALS_CATALOG_JSON_SHA256 = "680e9f52aac791efbd38e3bd99bd51ef5cded9756d867897c6c2755850e87b50" ENGINE_N8N_INACTIVE_META_JSON_SHA256 = "0ac555d7ab31cb0ca042db4f474e83cfefbf20b5bbb2e1509fead27ed21e8acb" @@ -215,10 +286,10 @@ ENGINE_DATA_PRODUCT_PUBLISH_GRANT_PREDECESSOR_SHA256 = { "nodedc-source/server/credentialSink/store.js": "c78dc285a973b6acd8a2330f0310935ad08720b2905454492f292d235abf12c0", "nodedc-source/server/credentialSink/vendor/engine-credential-sink.mjs": "b4800eead9bf94793ff1280d06e34aad6b37ef055a9d7f8d83d7fb5f9bd66d8b", "nodedc-source/server/index.js": "b2b790b02839570d967a2ca68b00e2724485a99389ac9b3a589a1b22302a36b8", - "nodedc-source/server/routes/engineAgentGateway.js": "e3450a4e1d5318dbac37627b67804d62804e27e2032c9e90478a1e739cea6d3d", + "nodedc-source/server/routes/engineAgentGateway.js": "604a75ad3b9b463cea5e578760c8a23592e2e00b8acfc0099730a347e27bf4e8", "nodedc-source/server/routes/engineCredentialSink.js": "9cbb69dbc8cbe6181cd5b0170fe9c4d717b0a173866ca98cba3e3766c0bab94e", "nodedc-source/server/routes/n8n.js": "783d822e2457d82e890f43bc00c7e33822077dc2841f0511a89ecb210fd36d48", - "nodedc-source/server/routes/ndcAgentMcp.js": "fbb3342b1a617b956d5b3a6a40d5111aa107c1176b4ff89c37b104995bada081", + "nodedc-source/server/routes/ndcAgentMcp.js": "534e2c85e1faecc72a00da7ad32d0584ee09b6bd89eeec2221c3b23d80e1e962", ENGINE_CREDENTIAL_BACKEND_OVERRIDE_TEMPLATE_REL: "944fa64b08255eb8207b93fd327aebb98ecd9400d37d25fcfa8e3a040ee44afe", } ENGINE_CREDENTIAL_SINK_CONTRACT_SHA256 = "b4800eead9bf94793ff1280d06e34aad6b37ef055a9d7f8d83d7fb5f9bd66d8b" @@ -1048,7 +1119,7 @@ def ensure_engine_edp_managed_provisioner_keypair(): return "created" -def ensure_engine_publish_grant_private_state(): +def ensure_engine_data_product_grant_private_state(include_reader=False): # Compose must never create this bind source on our behalf: Docker's normal # 0755 directory would either expose control-plane metadata or be rejected # by the Engine's fail-closed PublishGrantStore. @@ -1060,10 +1131,16 @@ def ensure_engine_publish_grant_private_state(): if stat.S_ISLNK(engine_root_stat.st_mode) or not stat.S_ISDIR(engine_root_stat.st_mode): die("Engine root is unsafe before private state preparation") - for state_path, label in ( + state_paths = [ (engine_root / ENGINE_CONTROL_PLANE_STATE_REL, "Engine control-plane state"), (engine_root / ENGINE_PUBLISH_GRANT_STATE_REL, "Engine Publish grant state"), - ): + ] + if include_reader: + state_paths.append(( + engine_root / ENGINE_READ_GRANT_STATE_REL, + "Engine Read grant state", + )) + for state_path, label in state_paths: try: state_stat = state_path.lstat() except FileNotFoundError: @@ -1079,6 +1156,11 @@ def ensure_engine_publish_grant_private_state(): die(f"{label} ownership or mode is unsafe: {state_path}") +def ensure_engine_publish_grant_private_state(): + # Frozen compatibility seam for the established Publish transition. + return ensure_engine_data_product_grant_private_state(include_reader=False) + + def fsync_directory(path): descriptor = os.open(str(path), os.O_RDONLY | os.O_DIRECTORY) try: @@ -1565,6 +1647,8 @@ def allowed_payload_path(component, rel): return True if rel == ENGINE_DATA_PRODUCT_PUBLISH_GRANT_OVERRIDE_REL: return True + if rel == ENGINE_DATA_PRODUCT_READ_GRANT_OVERRIDE_REL: + return True if rel == ENGINE_NODE_INTELLIGENCE_SERVICE_ROOT_REL or rel.startswith( ENGINE_NODE_INTELLIGENCE_SERVICE_ROOT_REL + "/" ): @@ -2158,14 +2242,21 @@ def read_engine_n8n_transition_descriptor(path, label="Engine n8n transition des "composeOverride": ENGINE_N8N_COMPOSE_OVERRIDE_REL, "runtimePackagePath": ENGINE_N8N_RUNTIME_PACKAGE_PATH, "topologyServices": ["n8n"], - "rollbackBaseline": "verified_inactive", } for key, expected in exact_common.items(): if descriptor.get(key) != expected: die(f"Engine n8n transition {key} mismatch") if action == "activate": - if descriptor.get("expectedCurrent") != "verified_inactive": - die("Engine n8n activation must start from the verified inactive baseline") + expected_current = descriptor.get("expectedCurrent") + if ( + expected_current != "verified_inactive" + and expected_current not in ENGINE_N8N_RELEASE_CATALOG_JSON_SHA256 + ): + die("Engine n8n activation expected-current release is not registered") + if expected_current == release_id: + die("Engine n8n activation target already equals expected current") + if descriptor.get("rollbackBaseline") != expected_current: + die("Engine n8n activation rollback baseline mismatch") if descriptor.get("expectedNodeTypes") != list(ENGINE_N8N_NODE_TYPES): die("Engine n8n activation node type set mismatch") if descriptor.get("expectedCredentialTypes") != list(ENGINE_N8N_CREDENTIAL_TYPES): @@ -2173,6 +2264,8 @@ def read_engine_n8n_transition_descriptor(path, label="Engine n8n transition des else: if descriptor.get("expectedCurrent") != release_id: die("Engine n8n rollback expected-current release mismatch") + if descriptor.get("rollbackBaseline") != "verified_inactive": + die("Engine n8n inactive rollback baseline mismatch") if descriptor.get("expectedNodeTypes") != [] or descriptor.get("expectedCredentialTypes") != []: die("Engine n8n rollback must select the inactive catalog") return descriptor @@ -2275,10 +2368,13 @@ def validate_engine_n8n_catalog_payload(payload_dir, descriptor): die("Engine n8n light icon sha256 mismatch") if sha256_file(payload_dir / ENGINE_N8N_DARK_ICON_REL) != ENGINE_N8N_DARK_ICON_SHA256: die("Engine n8n dark icon sha256 mismatch") + release_hashes = ENGINE_N8N_RELEASE_CATALOG_JSON_SHA256.get(descriptor["releaseId"]) + if release_hashes is None: + die("Engine n8n activation release catalog is not registered") expected_catalog_hashes = ( - (nodes, ENGINE_N8N_ACTIVATION_NODES_CATALOG_JSON_SHA256, "node"), - (credentials, ENGINE_N8N_ACTIVATION_CREDENTIALS_CATALOG_JSON_SHA256, "credential"), - (meta, ENGINE_N8N_ACTIVATION_META_JSON_SHA256, "metadata"), + (nodes, release_hashes["nodes"], "node"), + (credentials, release_hashes["credentials"], "credential"), + (meta, release_hashes["meta"], "metadata"), ) else: expected_catalog_hashes = ( @@ -2683,10 +2779,18 @@ def validate_engine_node_intelligence_transition(payload_dir, entries): "engine_get_node_guidance", "engine_validate_node_configuration", "engine_validate_l2_deep", - "const ENGINE_AGENT_MCP_VERSION = '0.3.0'", ): if required not in gateway_text: die(f"Engine node-intelligence gateway contract missing: {required}") + if not any( + version in gateway_text + for version in ( + "const ENGINE_AGENT_MCP_VERSION = '0.3.0'", + "const ENGINE_AGENT_MCP_VERSION = '0.4.0'", + "const ENGINE_AGENT_MCP_VERSION = '0.5.0'", + ) + ): + die("Engine node-intelligence gateway MCP version is not registered") validate_engine_node_intelligence_image_archive( payload_dir / ENGINE_NODE_INTELLIGENCE_IMAGE_ARCHIVE_REL, descriptor, @@ -2705,6 +2809,245 @@ def validate_engine_node_intelligence_transition(payload_dir, entries): return descriptor +def is_engine_mcp_control_plane_slice(component, entries): + return ( + component == "engine" + and entries is not None + and tuple(entries) == ENGINE_MCP_CONTROL_PLANE_ARTIFACT_ENTRIES + ) + + +def validate_engine_mcp_control_plane_payload(payload_dir, entries): + if tuple(entries) != ENGINE_MCP_CONTROL_PLANE_ARTIFACT_ENTRIES: + die("Engine MCP control-plane files.txt exact set/order mismatch") + for rel, expected_sha256 in ENGINE_MCP_CONTROL_PLANE_TARGET_SHA256.items(): + path = payload_dir / rel + if path.is_symlink() or not path.is_file() or sha256_file(path) != expected_sha256: + die(f"Engine MCP control-plane target sha256 mismatch: {rel}") + + gateway = (payload_dir / ENGINE_NODE_INTELLIGENCE_GATEWAY_REL).read_text(encoding="utf-8") + graph_route = (payload_dir / "nodedc-source/server/routes/ndcAgentMcp.js").read_text( + encoding="utf-8" + ) + installer = ( + payload_dir + / "nodedc-source/server/assets/engine-agent-npm/bin/nodedc-engine-codex-agent.mjs" + ).read_text(encoding="utf-8") + package = read_strict_json( + payload_dir / "nodedc-source/server/assets/engine-agent-npm/package.json", + "Engine MCP Codex installer package", + ) + if package.get("name") != "@nodedc/engine-codex-agent" or package.get("version") != "0.1.4": + die("Engine MCP Codex installer package identity mismatch") + if any( + marker not in gateway + for marker in ( + "const ENGINE_AGENT_MCP_VERSION = '0.5.0'", + "engine_open_l2_change_session", + "engine_get_l2_change_session", + "engine_close_l2_change_session", + "never_infer_expiry", + "three_identical_failures_without_new_evidence", + "engine_plan_data_product_read_grant", + "engine_apply_data_product_read_grant", + "engine_accept_data_product_read_grant", + "engine_rollback_data_product_read_grant", + ) + ): + die("Engine MCP bounded-change-session contract is incomplete") + if any( + marker not in graph_route + for marker in ( + "update_node_no_effect", + "subworkflow_post_write_equality_failed", + "verifiedWrite: true", + "graphDigest: expectedDigest", + ) + ): + die("Engine MCP graph post-write equality contract is incomplete") + if any( + marker not in installer + for marker in ( + "Do not interrupt the user after ordinary recoverable errors", + "Never infer that a provider or managed capability token expired", + "Three identical failures with no new evidence", + "engine_plan_data_product_*_grant", + "durable until an explicit rollback or revoke", + ) + ): + die("Engine MCP Codex bounded-autonomy policy is incomplete") + combined = "\n".join((gateway, graph_route, installer)).lower() + if "gelios" in combined or "robot2b" in combined: + die("provider identity is forbidden in Engine MCP control-plane source") + + archive_path = payload_dir / "nodedc-source/server/assets/nodedc-engine-codex-agent-0.1.4.tgz" + expected_members = { + "package/bin/nodedc-engine-codex-agent.mjs": ( + payload_dir + / "nodedc-source/server/assets/engine-agent-npm/bin/nodedc-engine-codex-agent.mjs" + ).read_bytes(), + "package/package.json": ( + payload_dir / "nodedc-source/server/assets/engine-agent-npm/package.json" + ).read_bytes(), + } + observed_members = {} + try: + with tarfile.open(archive_path, "r:gz") as archive: + for member in archive: + if not (member.isfile() or member.isdir()) or member.name.startswith("/"): + die("Engine MCP Codex installer archive member is unsafe") + if member.isfile(): + source = archive.extractfile(member) + if source is None: + die("Engine MCP Codex installer archive member is unreadable") + observed_members[member.name] = source.read(MAX_FILE_BYTES + 1) + except (OSError, tarfile.TarError): + die("Engine MCP Codex installer archive is invalid") + if observed_members != expected_members: + die("Engine MCP Codex installer archive/source equality mismatch") + + n8n_route = (payload_dir / "nodedc-source/server/routes/n8n.js").read_text( + encoding="utf-8" + ) + if "engineDataProductReadGrantN8nAdapter" not in n8n_route: + die("Engine MCP managed reader native adapter is missing") + if "ndcDataProductReaderApi" not in n8n_route or "ndc_edprb_" not in n8n_route: + die("Engine MCP managed reader credential contract is incomplete") + if "read_grant_must_be_durable" not in n8n_route: + die("Engine MCP managed reader durable lifecycle is missing") + reader_service = ( + payload_dir / "nodedc-source/server/dataProductReadGrant/service.js" + ).read_text(encoding="utf-8") + if ( + "readerGrantLifetime: 'explicit-revoke'" not in reader_service + or "expiresAt: null" not in reader_service + or "ENGINE_READ_GRANT_TTL_DAYS" in reader_service + ): + die("Engine MCP managed reader explicit-revoke lifecycle is incomplete") + publish_service = ( + payload_dir / "nodedc-source/server/dataProductPublishGrant/service.js" + ).read_text(encoding="utf-8") + if ( + "writerGrantLifetime: 'explicit-revoke'" not in publish_service + or "migrateCurrentGrantToDurable" not in publish_service + or "expiresAt: null" not in publish_service + or "ENGINE_PUBLISH_GRANT_TTL_DAYS" in publish_service + ): + die("Engine MCP managed writer explicit-revoke lifecycle is incomplete") + reader_override = payload_dir / ENGINE_DATA_PRODUCT_READ_GRANT_OVERRIDE_REL + try: + reader_override_text = reader_override.read_text(encoding="utf-8") + except (OSError, UnicodeDecodeError): + die("Engine MCP managed reader runtime override is unreadable") + if reader_override_text != expected_engine_data_product_read_grant_override(): + die("Engine MCP managed reader runtime override mismatch") + agent_store = (payload_dir / "nodedc-source/server/engineAgents/store.js").read_text( + encoding="utf-8" + ) + for scope in ( + "'engine:l2:data-product-read-grant:plan'", + "'engine:l2:data-product-read-grant:write'", + ): + if scope not in agent_store: + die(f"Engine MCP managed reader scope is missing: {scope}") + + descriptor = read_engine_node_intelligence_descriptor( + payload_dir / ENGINE_MCP_CONTROL_PLANE_DESCRIPTOR_REL, + "Engine MCP control-plane node-intelligence descriptor", + ) + if ( + descriptor.get("action") != "activate" + or descriptor["source"].get("gatewaySha256") + != ENGINE_MCP_CONTROL_PLANE_TARGET_SHA256[ENGINE_NODE_INTELLIGENCE_GATEWAY_REL] + or descriptor["source"].get("upstreamProjectionSha256") + != ENGINE_MCP_CONTROL_PLANE_TARGET_SHA256[ + "nodedc-source/server/nodeIntelligence/upstreamProjection.js" + ] + ): + die("Engine MCP control-plane descriptor target mismatch") + return descriptor + + +def preflight_engine_mcp_control_plane_predecessor(payload_dir): + candidate = validate_engine_mcp_control_plane_payload( + payload_dir, + ENGINE_MCP_CONTROL_PLANE_ARTIFACT_ENTRIES, + ) + installed = current_engine_node_intelligence_descriptor() + validate_installed_engine_node_intelligence_source(installed) + if installed is None or installed.get("action") != "activate": + die("Engine MCP control-plane requires active node intelligence") + expected_candidate = json.loads(json.dumps(installed)) + expected_candidate["source"]["gatewaySha256"] = ( + ENGINE_MCP_CONTROL_PLANE_TARGET_SHA256[ENGINE_NODE_INTELLIGENCE_GATEWAY_REL] + ) + expected_candidate["source"]["upstreamProjectionSha256"] = ( + ENGINE_MCP_CONTROL_PLANE_TARGET_SHA256[ + "nodedc-source/server/nodeIntelligence/upstreamProjection.js" + ] + ) + if candidate != expected_candidate: + die("Engine MCP control-plane descriptor crosses node-intelligence source boundary") + root = component_root("engine") + for rel, expected_sha256 in ENGINE_MCP_CONTROL_PLANE_PREDECESSOR_SHA256.items(): + path = root / rel + if path.is_symlink() or not path.is_file() or sha256_file(path) != expected_sha256: + die(f"Engine MCP control-plane predecessor drift detected: {rel}") + for rel in ENGINE_MCP_CONTROL_PLANE_NEW_PATHS: + path = root / rel + if path.exists() or path.is_symlink(): + die(f"Engine MCP control-plane new path already exists: {rel}") + backend = preflight_engine_credential_backend_runtime() + if backend["mode"] != "verified-derived-retry": + die("Engine MCP control-plane requires the active immutable backend runtime") + return { + "descriptor": candidate, + "predecessor_gateway_sha256": installed["source"]["gatewaySha256"], + "target_gateway_sha256": candidate["source"]["gatewaySha256"], + "backend_mode": backend["mode"], + } + + +def accept_engine_mcp_control_plane_runtime(): + root = component_root("engine") + descriptor = validate_engine_mcp_control_plane_payload( + root, + ENGINE_MCP_CONTROL_PLANE_ARTIFACT_ENTRIES, + ) + installed = current_engine_node_intelligence_descriptor() + if installed != descriptor: + die("Engine MCP control-plane installed descriptor equality failed") + validate_installed_engine_node_intelligence_source(installed) + backend = preflight_engine_credential_backend_runtime() + if backend["mode"] != "verified-derived-retry": + die("Engine MCP control-plane backend runtime acceptance failed") + live = run_engine_backend_probe( + ( + "node", + "--input-type=module", + "-e", + """ +const module=await import('file:///app/server/routes/engineAgentGateway.js'); +const store=await import('file:///app/server/engineAgents/store.js'); +const names=new Set(module.engineAgentTools.map((tool)=>tool.name)); +const required=['engine_plan_data_product_read_grant','engine_apply_data_product_read_grant','engine_accept_data_product_read_grant','engine_rollback_data_product_read_grant']; +const scopes=['engine:l2:data-product-read-grant:plan','engine:l2:data-product-read-grant:write']; +if(!required.every((name)=>names.has(name))||!scopes.every((scope)=>store.ENGINE_AGENT_SCOPES.includes(scope)))process.exit(2); +process.stdout.write('engine-mcp-reader-grant:0.5.0:'+required.length+':'+scopes.length); +""".strip(), + ), + "Engine MCP managed reader capability", + container_id=engine_backend_container_id(), + ) + if live != "engine-mcp-reader-grant:0.5.0:4:2": + die("Engine MCP managed reader live capability acceptance mismatch") + return { + "gateway_sha256": descriptor["source"]["gatewaySha256"], + "backend_mode": backend["mode"], + "reader_grant": live, + } + + def validate_no_lifecycle_scripts(payload_dir, label): forbidden = ("preinstall", "install", "postinstall", "prepare", "prepack", "postpack") for package_path in payload_dir.rglob("package.json"): @@ -2800,6 +3143,22 @@ def expected_engine_data_product_publish_grant_override(): )) +def expected_engine_data_product_read_grant_override(): + return "\n".join(( + "services:", + " nodedc-backend:", + " environment:", + f" ENGINE_CONTROL_PLANE_READ_GRANT_ROOT: {ENGINE_READ_GRANT_CONTAINER_PATH}", + " volumes:", + " - type: bind", + f" source: {ENGINE_READ_GRANT_STATE_PATH}", + f" target: {ENGINE_READ_GRANT_CONTAINER_PATH}", + " bind:", + " create_host_path: false", + "", + )) + + def validate_engine_credential_sink_slice(payload_dir, entries): if tuple(entries) != ENGINE_CREDENTIAL_SINK_ARTIFACT_ENTRIES: die("Engine credential sink files.txt exact set/order mismatch") @@ -3103,20 +3462,30 @@ def load_artifact(artifact, work_dir): or rel.startswith(ENGINE_NODE_INTELLIGENCE_SERVICE_ROOT_REL + "/") for rel in entries ) - if touches_node_intelligence and not is_engine_node_intelligence_transition( - manifest["component"], entries + if ( + touches_node_intelligence + and not is_engine_node_intelligence_transition(manifest["component"], entries) + and not is_engine_mcp_control_plane_slice(manifest["component"], entries) ): die("Engine node-intelligence payload requires the canonical transition") if is_engine_node_intelligence_transition(manifest["component"], entries): validate_engine_node_intelligence_transition(payload_dir, entries) + if is_engine_mcp_control_plane_slice(manifest["component"], entries): + validate_engine_mcp_control_plane_payload(payload_dir, entries) if touches_engine_credential_sink(manifest["component"], entries): validate_engine_credential_sink_slice(payload_dir, entries) if ( - touches_engine_data_product_publish_grant(entries) - or ENGINE_DATA_PRODUCT_PUBLISH_GRANT_OVERRIDE_REL in entries + not is_engine_mcp_control_plane_slice(manifest["component"], entries) + and ( + touches_engine_data_product_publish_grant(entries) + or ENGINE_DATA_PRODUCT_PUBLISH_GRANT_OVERRIDE_REL in entries + ) ): validate_engine_data_product_publish_grant_slice(payload_dir, entries) - elif ENGINE_AGENT_FULL_GRANT_MIGRATION_STORE_REL in entries: + elif ( + not is_engine_mcp_control_plane_slice(manifest["component"], entries) + and ENGINE_AGENT_FULL_GRANT_MIGRATION_STORE_REL in entries + ): validate_engine_agent_full_grant_migration_slice(payload_dir, entries) return manifest, entries, payload_dir @@ -3202,6 +3571,7 @@ def touches_external_data_plane_files(entries): "platform/packages/external-provider-contract/src/data-plane.mjs", "platform/packages/external-provider-contract/src/data-product.mjs", "platform/packages/external-provider-contract/src/intake-batch.mjs", + "platform/packages/external-provider-contract/src/index.mjs", "platform/packages/external-provider-contract/src/sensitive-field-policy.mjs", } return any( @@ -3462,6 +3832,12 @@ def is_platform_provider_catalog_only(entries): def component_services(component, entries=None): + if is_engine_mcp_control_plane_slice(component, entries): + # This slice updates only the existing Engine backend control plane. + # Node intelligence keeps the same immutable sidecar image and n8n/L1 + # retain their current generations. + return ("nodedc-backend",) + if is_engine_node_intelligence_transition(component, entries): if tuple(entries) == ENGINE_NODE_INTELLIGENCE_ARTIFACT_ENTRIES: return (ENGINE_NODE_INTELLIGENCE_SERVICE, "nodedc-backend") @@ -3653,6 +4029,18 @@ def component_compose_files(component, allow_prepared_engine_backend=False): # Canonical order is immutable: base, active L2 extension, credential # sink, then the additive Publish grant overlay. files.append(publish_grant_override) + read_grant_override = root / ENGINE_DATA_PRODUCT_READ_GRANT_OVERRIDE_REL + if read_grant_override.exists() or read_grant_override.is_symlink(): + if read_grant_override.is_symlink() or not read_grant_override.is_file(): + die("installed Engine data product read grant override is unsafe") + try: + read_grant_override_text = read_grant_override.read_text(encoding="utf-8") + except (OSError, UnicodeDecodeError): + die("installed Engine data product read grant override cannot be read") + if read_grant_override_text != expected_engine_data_product_read_grant_override(): + die("installed Engine data product read grant override drift detected") + # Read authority is additive and ordered after the writer/private-key overlay. + files.append(read_grant_override) node_intelligence_descriptor = current_engine_node_intelligence_descriptor() if ( node_intelligence_descriptor is not None @@ -4164,6 +4552,22 @@ def validate_engine_backend_mounts_for_installed_runtime( str(ENGINE_EDP_MANAGED_PROVISIONER_PRIVATE_KEY_FILE), ), } + read_override = component_root("engine") / ENGINE_DATA_PRODUCT_READ_GRANT_OVERRIDE_REL + read_installed = read_override.exists() or read_override.is_symlink() + if read_installed: + if read_override.is_symlink() or not read_override.is_file(): + die("installed Engine data product read grant override is unsafe") + try: + read_override_text = read_override.read_text(encoding="utf-8") + except (OSError, UnicodeDecodeError): + die("installed Engine data product read grant override cannot be read") + if read_override_text != expected_engine_data_product_read_grant_override(): + die("installed Engine data product read grant override drift detected") + expected_publish_mounts[ENGINE_READ_GRANT_CONTAINER_PATH] = ( + "bind", + True, + str(ENGINE_READ_GRANT_STATE_PATH), + ) mounts = container.get("Mounts") if not isinstance(mounts, list): die("Engine backend mount inventory is missing") @@ -5306,10 +5710,6 @@ def preflight_engine_n8n_transition(descriptor, enforce_expected_current): "Engine n8n transition stale current state: " f"expected={descriptor['expectedCurrent']} actual={current_state}" ) - if current_descriptor and current_state != "verified_inactive": - if current_descriptor.get("packageSha256") != descriptor["packageSha256"]: - if enforce_expected_current: - die("Engine n8n current release package sha256 mismatch") return { "base_image_id": base_image["id"], "base_image_architecture": base_image["architecture"], @@ -5558,6 +5958,7 @@ def plan_artifact(artifact): sha = sha256_file(artifact) publish_grant_preflight = None node_intelligence_preflight = None + mcp_control_plane_preflight = None with tempfile.TemporaryDirectory(prefix="plan-", dir=TMP_DIR) as tmp: manifest, entries, payload_dir = load_artifact(artifact, Path(tmp)) transition_descriptor = None @@ -5578,6 +5979,10 @@ def plan_artifact(artifact): node_intelligence_preflight = preflight_engine_node_intelligence_predecessor( payload_dir ) + if is_engine_mcp_control_plane_slice(manifest["component"], entries): + mcp_control_plane_preflight = preflight_engine_mcp_control_plane_predecessor( + payload_dir + ) component = manifest["component"] root = component_root(component) @@ -5644,6 +6049,22 @@ def plan_artifact(artifact): print("node_intelligence_host_ports=none") print(f"predecessor_gateway_sha256={node_intelligence_preflight['gateway_sha256']}") print(f"backend_current_barrier={node_intelligence_preflight['backend_mode']}") + if mcp_control_plane_preflight: + print("engine_mcp_transition=managed-reader-grant+validator-reconciliation") + print("engine_mcp_version=0.5.0") + print("engine_mcp_installer=0.1.4") + print("engine_mcp_reader_grant=plan+apply+accept+rollback") + print( + "predecessor_gateway_sha256=" + f"{mcp_control_plane_preflight['predecessor_gateway_sha256']}" + ) + print( + "target_gateway_sha256=" + f"{mcp_control_plane_preflight['target_gateway_sha256']}" + ) + print(f"backend_current_barrier={mcp_control_plane_preflight['backend_mode']}") + print("node_intelligence_image=preserved") + print("n8n_l1=untouched") if transition_descriptor: print(f"n8n_transition={transition_descriptor['action']}") print(f"n8n_version={transition_descriptor['n8nVersion']}") @@ -5665,7 +6086,7 @@ def plan_artifact(artifact): print("n8n_database=preserved:n8n-postgres") print(f"n8n_expected_node_types={','.join(transition_descriptor['expectedNodeTypes'])}") print(f"n8n_expected_credential_types={','.join(transition_descriptor['expectedCredentialTypes'])}") - print("n8n_rollback_baseline=verified_inactive") + print(f"n8n_rollback_baseline={transition_descriptor['rollbackBaseline']}") touches_map_gateway = component == "platform" and any(rel == "platform/services/map-gateway" or rel.startswith("platform/services/map-gateway/") for rel in entries) touches_external_data_plane = component == "platform" and touches_external_data_plane_files(entries) if component == "module-foundry" or touches_map_gateway: @@ -6387,6 +6808,10 @@ def prepare_component_runtime(component, entries=None): ensure_engine_edp_managed_provisioner_keypair() ensure_engine_publish_grant_private_state() + if is_engine_mcp_control_plane_slice(component, entries): + ensure_engine_edp_managed_provisioner_keypair() + ensure_engine_data_product_grant_private_state(include_reader=True) + if is_engine_n8n_transition(component, entries): descriptor = current_engine_n8n_transition_descriptor() if descriptor is None: @@ -6553,6 +6978,20 @@ def external_data_plane_healthcheck(require_managed=True): } if require_managed: expected_json["managedWriterBindingProvisioning"] = "enabled" + server_source = ( + component_root("platform") + / "platform/services/external-data-plane/src/server.mjs" + ) + try: + server_text = server_source.read_text(encoding="utf-8") + except (OSError, UnicodeDecodeError): + server_text = "" + if 'managedReaderBindingProvisioning: config.managedProvisionerApiEnabled' in server_text: + expected_json["managedReaderBindingProvisioning"] = "enabled" + if 'managedWriterBindingLifetime: config.managedProvisionerApiEnabled ? "explicit-revoke" : "disabled"' in server_text: + expected_json["managedWriterBindingLifetime"] = "explicit-revoke" + if 'managedReaderBindingLifetime: config.managedProvisionerApiEnabled ? "explicit-revoke" : "disabled"' in server_text: + expected_json["managedReaderBindingLifetime"] = "explicit-revoke" return { "url": "http://127.0.0.1:18106/healthz", "expected_json": expected_json, @@ -6576,6 +7015,7 @@ def component_healthchecks(component, entries=None, services=None): ( is_engine_data_product_publish_grant_slice(component, entries) or is_engine_agent_full_grant_migration_slice(component, entries) + or is_engine_mcp_control_plane_slice(component, entries) ) and services is not None ): @@ -6888,10 +7328,12 @@ def run_healthchecks(component, entries=None, services=None): component, entries, ) + touches_mcp_control_plane = is_engine_mcp_control_plane_slice(component, entries) if ( touches_engine_credential_sink(component, entries) or touches_publish_grant or touches_agent_grant_migration + or touches_mcp_control_plane ): # The HTTP endpoint can become ready before Docker publishes the first # successful health probe. Wait for the Compose health barrier before @@ -6903,12 +7345,15 @@ def run_healthchecks(component, entries=None, services=None): touches_engine_credential_sink(component, entries) or touches_publish_grant or touches_agent_grant_migration + or touches_mcp_control_plane ): runtime = preflight_engine_credential_backend_runtime() if runtime["mode"] != "verified-derived-retry": die("Engine backend immutable runtime activation acceptance failed") if touches_agent_grant_migration: accept_engine_agent_full_grant_migration_runtime() + if touches_mcp_control_plane: + accept_engine_mcp_control_plane_runtime() container_name = COMPONENTS[component].get("health_container") if container_name: healthcheck_container(container_name) @@ -6983,6 +7428,8 @@ def apply_artifact(artifact): payload_dir ) node_intelligence_descriptor = node_intelligence_preflight["descriptor"] + if is_engine_mcp_control_plane_slice(component, entries): + preflight_engine_mcp_control_plane_predecessor(payload_dir) if not artifact_only and not compose_root.is_dir(): if bootstrap_root and is_relative_to(compose_root.resolve(strict=False), root.resolve(strict=False)): compose_root.mkdir(parents=True, exist_ok=True) @@ -7159,6 +7606,7 @@ def apply_artifact(artifact): ( is_engine_data_product_publish_grant_slice(component, entries) or is_engine_agent_full_grant_migration_slice(component, entries) + or is_engine_mcp_control_plane_slice(component, entries) ) and services is not None ): diff --git a/infra/deploy-runner/test_engine_mcp_control_plane.py b/infra/deploy-runner/test_engine_mcp_control_plane.py new file mode 100644 index 0000000..4cd06db --- /dev/null +++ b/infra/deploy-runner/test_engine_mcp_control_plane.py @@ -0,0 +1,198 @@ +#!/usr/bin/env python3 +import hashlib +import importlib.machinery +import importlib.util +import json +import os +import subprocess +import tarfile +import tempfile +import unittest +from unittest import mock +from pathlib import Path + + +SCRIPT_DIR = Path(__file__).resolve().parent +PLATFORM_ROOT = SCRIPT_DIR.parent.parent +RUNNER_PATH = SCRIPT_DIR / "nodedc-deploy" +BUILDER_PATH = SCRIPT_DIR / "build-engine-mcp-control-plane-artifact.mjs" +TRANSITION_ID = "20260718-003" +STAGE_ARTIFACT = ( + PLATFORM_ROOT + / "infra/deploy-artifacts" + / "nodedc-engine-mcp-control-plane-20260718-003.tgz" +) +STAGE_ARTIFACT_SHA256 = "249aef9527666c562e9648b15737e66cc5c1dc7c0788b58ea714da270b5eb4ba" + + +def load_runner(): + loader = importlib.machinery.SourceFileLoader("nodedc_engine_mcp_runner", str(RUNNER_PATH)) + spec = importlib.util.spec_from_loader(loader.name, loader) + module = importlib.util.module_from_spec(spec) + loader.exec_module(module) + return module + + +RUNNER = load_runner() + + +class EngineMcpControlPlaneTest(unittest.TestCase): + @classmethod + def setUpClass(cls): + cls.temporary = tempfile.TemporaryDirectory(prefix="nodedc-engine-mcp-control-plane-") + cls.root = Path(cls.temporary.name) + stage_bytes = STAGE_ARTIFACT.read_bytes() + if hashlib.sha256(stage_bytes).hexdigest() != STAGE_ARTIFACT_SHA256: + raise RuntimeError("engine_mcp_control_plane_stage_artifact_sha256_mismatch") + source_stage = cls.root / "immutable-source" + source_stage.mkdir() + RUNNER.safe_extract(STAGE_ARTIFACT, source_stage) + cls.engine_source_root = source_stage / "payload" + cls.results = [] + for index in range(2): + output = cls.root / f"build-{index}" + output.mkdir() + env = os.environ.copy() + env["NODEDC_DEPLOY_ARTIFACT_DIR"] = str(output) + env["NODEDC_ENGINE_SOURCE_ROOT"] = str(cls.engine_source_root) + result = subprocess.run( + ["node", str(BUILDER_PATH), TRANSITION_ID], + cwd=PLATFORM_ROOT, + env=env, + check=True, + capture_output=True, + text=True, + ) + cls.results.append(json.loads(result.stdout)) + + @classmethod + def tearDownClass(cls): + cls.temporary.cleanup() + + def artifact(self, index=0): + return Path(self.results[index]["artifact"]) + + def extract(self, artifact, directory): + RUNNER.safe_extract(artifact, directory) + return directory / "payload", RUNNER.parse_files_list(directory / "files.txt") + + def test_artifact_is_byte_reproducible_and_canonical(self): + first = self.artifact(0).read_bytes() + second = self.artifact(1).read_bytes() + self.assertEqual(first, second) + self.assertEqual(first, STAGE_ARTIFACT.read_bytes()) + self.assertEqual(first[4:8], b"\0\0\0\0") + self.assertEqual(self.results[0]["artifactSha256"], hashlib.sha256(first).hexdigest()) + with tarfile.open(self.artifact(0), "r:gz") as archive: + for member in archive: + self.assertTrue(member.isfile() or member.isdir()) + self.assertFalse(Path(member.name).name.startswith("._")) + + def test_runner_selects_only_existing_backend(self): + with tempfile.TemporaryDirectory() as directory: + manifest, entries, payload = RUNNER.load_artifact( + self.artifact(0), + Path(directory), + ) + descriptor = RUNNER.validate_engine_mcp_control_plane_payload(payload, entries) + override_text = ( + payload + / RUNNER.ENGINE_DATA_PRODUCT_READ_GRANT_OVERRIDE_REL + ).read_text(encoding="utf-8") + self.assertEqual(manifest["component"], "engine") + self.assertEqual(tuple(entries), RUNNER.ENGINE_MCP_CONTROL_PLANE_ARTIFACT_ENTRIES) + self.assertEqual(RUNNER.component_services("engine", entries), ("nodedc-backend",)) + self.assertEqual(descriptor["action"], "activate") + self.assertEqual( + descriptor["source"]["gatewaySha256"], + RUNNER.ENGINE_MCP_CONTROL_PLANE_TARGET_SHA256[ + RUNNER.ENGINE_NODE_INTELLIGENCE_GATEWAY_REL + ], + ) + self.assertEqual(override_text, RUNNER.expected_engine_data_product_read_grant_override()) + + def test_runtime_preparation_adds_only_private_reader_state(self): + with ( + mock.patch.object( + RUNNER, + "ensure_engine_edp_managed_provisioner_keypair", + ) as keypair, + mock.patch.object( + RUNNER, + "ensure_engine_data_product_grant_private_state", + ) as private_state, + ): + RUNNER.prepare_component_runtime( + "engine", + RUNNER.ENGINE_MCP_CONTROL_PLANE_ARTIFACT_ENTRIES, + ) + keypair.assert_called_once_with() + private_state.assert_called_once_with(include_reader=True) + + def test_candidate_preserves_node_intelligence_identity_except_exact_control_plane_digests(self): + with tempfile.TemporaryDirectory() as directory: + work = Path(directory) + payload, entries = self.extract(self.artifact(0), work) + candidate = RUNNER.validate_engine_mcp_control_plane_payload(payload, entries) + self.assertEqual(candidate["releaseId"], RUNNER.ENGINE_NODE_INTELLIGENCE_RELEASE_ID) + self.assertEqual(candidate["upstream"]["commit"], RUNNER.ENGINE_NODE_INTELLIGENCE_UPSTREAM_COMMIT) + self.assertEqual(candidate["image"]["tag"], RUNNER.ENGINE_NODE_INTELLIGENCE_IMAGE) + self.assertEqual(candidate["expectedCurrent"], "inactive") + self.assertEqual( + candidate["source"]["upstreamProjectionSha256"], + RUNNER.ENGINE_MCP_CONTROL_PLANE_TARGET_SHA256[ + "nodedc-source/server/nodeIntelligence/upstreamProjection.js" + ], + ) + + def test_descriptor_boundary_and_source_digest_are_enforced(self): + with tempfile.TemporaryDirectory() as directory: + work = Path(directory) + payload, entries = self.extract(self.artifact(0), work) + descriptor_path = payload / RUNNER.ENGINE_MCP_CONTROL_PLANE_DESCRIPTOR_REL + descriptor = json.loads(descriptor_path.read_text(encoding="utf-8")) + descriptor["source"]["catalogSha256"] = "0" * 64 + descriptor_path.write_text(json.dumps(descriptor), encoding="utf-8") + # Payload validation permits only a structurally valid descriptor; + # predecessor equality is the state-aware barrier for untouched + # node-intelligence source identities. + with self.assertRaisesRegex( + RUNNER.DeployError, + "descriptor crosses node-intelligence source boundary", + ): + installed = json.loads(json.dumps(descriptor)) + installed["source"]["catalogSha256"] = "1" * 64 + installed["source"]["gatewaySha256"] = ( + RUNNER.ENGINE_MCP_CONTROL_PLANE_PREDECESSOR_SHA256[ + RUNNER.ENGINE_NODE_INTELLIGENCE_GATEWAY_REL + ] + ) + from unittest import mock + with ( + mock.patch.object( + RUNNER, + "current_engine_node_intelligence_descriptor", + return_value=installed, + ), + mock.patch.object(RUNNER, "validate_installed_engine_node_intelligence_source"), + ): + RUNNER.preflight_engine_mcp_control_plane_predecessor(payload) + + def test_existing_artifact_is_not_overwritten(self): + env = os.environ.copy() + env["NODEDC_DEPLOY_ARTIFACT_DIR"] = str(self.artifact(0).parent) + env["NODEDC_ENGINE_SOURCE_ROOT"] = str(self.engine_source_root) + result = subprocess.run( + ["node", str(BUILDER_PATH), TRANSITION_ID], + cwd=PLATFORM_ROOT, + env=env, + check=False, + capture_output=True, + text=True, + ) + self.assertNotEqual(result.returncode, 0) + self.assertIn("artifact_already_exists", result.stderr) + + +if __name__ == "__main__": + unittest.main(verbosity=2) diff --git a/infra/deploy-runner/test_engine_n8n_private_extension.py b/infra/deploy-runner/test_engine_n8n_private_extension.py index 922d148..52f3871 100644 --- a/infra/deploy-runner/test_engine_n8n_private_extension.py +++ b/infra/deploy-runner/test_engine_n8n_private_extension.py @@ -21,11 +21,12 @@ BUILDER_PATH = SCRIPT_DIR / "build-engine-n8n-private-extension-artifact.mjs" STAGE_ARTIFACT = ( PLATFORM_ROOT / "infra/deploy-artifacts" - / "nodedc-n8n-private-extension-n8n-nodes-ndc-release-20260716-003.tgz" + / "nodedc-n8n-private-extension-n8n-nodes-ndc-history-read-20260717-004.tgz" ) -TRANSITION_ID = "20260717-004" -SEALED_RELEASE_ID = "0.1.2-05e4b38b14b4a019" -SEALED_PACKAGE_SHA256 = "05e4b38b14b4a019ce1f6eee27b9e320094cb3560903bd68074966b3a1267af5" +TRANSITION_ID = "20260717-005" +SEALED_RELEASE_ID = "0.1.3-3354149b5245e39a" +SEALED_PACKAGE_SHA256 = "3354149b5245e39a10f6f03a4b43796602d7e57ea1f91dcc6cb6774ead97501d" +PREDECESSOR_RELEASE_ID = "0.1.2-05e4b38b14b4a019" BUILDER_ENGINE_PATHS = ( "nodedc-source/server/assets/n8n/schema/v2.3.2/nodes.catalog.json", "nodedc-source/server/assets/n8n/schema/v2.3.2/credentials.catalog.json", @@ -50,7 +51,7 @@ PRIVATE_EXTENSION_CANON_FUNCTION_SHA256 = { "engine_n8n_package_loader_probe_script": "2b3b3c96fad6e5e48ebea74426b21bc7eea6b6fe0e786b537da3366d18aedd9b", "engine_n8n_private_loader_catalog": "a2ae389b4ef215900cf967b863d01056bd2a8eec55554566d0f6005e4e0bcbac", "validate_engine_n8n_base_compose_source": "adacc5b20e52684cbc427362ddba5a6d2e13ef2091a89859e2b8f7bbf2e20458", - "preflight_engine_n8n_transition": "da3cec853e3e3e20df4d1e301aab98b77815f93ec3d68b0b61c84c6e6a3fb335", + "preflight_engine_n8n_transition": "3d574acb081fb752e45cc0c3286d3ae4596dcee83a85e2c3fee81ac65ec796e0", "accept_engine_n8n_runtime": "1a2614a465161e3833e84aca402817682da1a538bdb2aecc4af66b665497338d", } @@ -138,7 +139,7 @@ class EngineN8nPrivateExtensionTest(unittest.TestCase): self.assertEqual(first[4:8], b"\0\0\0\0") self.assertEqual(first[3] & 0x08, 0) - def test_successful_generation_003_runner_functions_are_frozen(self): + def test_state_aware_upgrade_runner_functions_are_frozen(self): actual = { name: hashlib.sha256(inspect.getsource(getattr(RUNNER, name)).encode("utf-8")).hexdigest() for name in PRIVATE_EXTENSION_CANON_FUNCTION_SHA256 @@ -164,8 +165,8 @@ class EngineN8nPrivateExtensionTest(unittest.TestCase): def test_transition_id_rejects_missing_invalid_and_previously_issued_values(self): cases = ( ([], "transition_id_required"), - (["20260716-003"], "transition_id_already_issued"), - (["engine-n8n-private-extension-20260717-004"], "transition_id_invalid"), + (["20260717-004"], "transition_id_already_issued"), + (["engine-n8n-private-extension-20260717-005"], "transition_id_invalid"), (["20260230-004"], "transition_id_invalid"), (["20260717-000"], "transition_id_invalid"), ([TRANSITION_ID, "unexpected-second-id"], "transition_id_argument_count_invalid"), @@ -202,7 +203,7 @@ class EngineN8nPrivateExtensionTest(unittest.TestCase): self.assertIn("transition_artifact_already_exists", result.stderr) def test_activation_and_rollback_pass_strict_runner_policy(self): - expected = {"activation": ("activate", 7), "rollback": ("rollback-inactive", 4)} + expected = {"activation": ("activate", 7), "rollback": ("activate", 7)} for kind, (action, entry_count) in expected.items(): with self.subTest(kind=kind), tempfile.TemporaryDirectory() as directory: manifest, entries, payload = RUNNER.load_artifact( @@ -234,7 +235,7 @@ class EngineN8nPrivateExtensionTest(unittest.TestCase): self.assertTrue(all("usableAsTool" not in item for item in private_nodes)) self.assertEqual((len(nodes), len(credentials)), (437, 388)) - def test_rollback_catalog_restores_verified_inactive_baseline(self): + def test_rollback_catalog_restores_verified_predecessor_release(self): with tempfile.TemporaryDirectory() as directory: _manifest, _entries, payload = RUNNER.load_artifact( self.artifact(0, "rollback"), @@ -242,9 +243,20 @@ class EngineN8nPrivateExtensionTest(unittest.TestCase): ) nodes = json.loads((payload / RUNNER.ENGINE_N8N_NODES_CATALOG_REL).read_text()) credentials = json.loads((payload / RUNNER.ENGINE_N8N_CREDENTIALS_CATALOG_REL).read_text()) - self.assertEqual([item for item in nodes if item.get("name", "").startswith("n8n-nodes-ndc.")], []) - self.assertEqual([item for item in credentials if item.get("name") in EXPECTED_CREDENTIALS], []) - self.assertEqual((len(nodes), len(credentials)), (434, 385)) + descriptor = RUNNER.read_engine_n8n_transition_descriptor( + payload / RUNNER.ENGINE_N8N_TRANSITION_DESCRIPTOR_REL + ) + self.assertEqual(descriptor["releaseId"], PREDECESSOR_RELEASE_ID) + self.assertEqual(descriptor["expectedCurrent"], SEALED_RELEASE_ID) + self.assertEqual( + [item.get("name") for item in nodes if item.get("name", "").startswith("n8n-nodes-ndc.")], + EXPECTED_NODES, + ) + self.assertEqual( + [item.get("name") for item in credentials if item.get("name") in EXPECTED_CREDENTIALS], + EXPECTED_CREDENTIALS, + ) + self.assertEqual((len(nodes), len(credentials)), (437, 388)) def test_compose_override_is_runner_derived_and_offline(self): with tempfile.TemporaryDirectory() as directory: @@ -259,7 +271,7 @@ class EngineN8nPrivateExtensionTest(unittest.TestCase): self.assertEqual(override, RUNNER.expected_engine_n8n_compose_override(descriptor)) self.assertEqual( hashlib.sha256(override.encode("utf-8")).hexdigest(), - "a29e2f92b87dda59da2b4e3ce250bc9705ed9fa551c0fbe4e95464fca5caa3e1", + "256142c14ae295bf9be4d74fdc8dcd6e134e4ded8a264f15e66b38e85503b589", ) self.assertIn("pull_policy: never", override) self.assertIn("N8N_USER_FOLDER: /home/node", override) @@ -268,7 +280,7 @@ class EngineN8nPrivateExtensionTest(unittest.TestCase): self.assertNotIn("N8N_CUSTOM_EXTENSIONS", override) self.assertNotIn("build:", override) - def test_generation_003_override_is_accepted_as_installed_canon(self): + def test_upgrade_override_is_accepted_as_installed_canon(self): original_root = RUNNER.COMPONENTS["engine"]["payload_root"] original_compose = RUNNER.COMPONENTS["engine"]["compose_root"] try: @@ -551,7 +563,7 @@ class EngineN8nPrivateExtensionTest(unittest.TestCase): def test_sealed_release_exact_set_includes_implicit_directories(self): release_relative = Path( - "payload/releases/n8n-nodes-ndc/0.1.2-05e4b38b14b4a019" + "payload/releases/n8n-nodes-ndc/0.1.3-3354149b5245e39a" ) with tempfile.TemporaryDirectory() as directory: work = Path(directory) diff --git a/infra/deploy-runner/test_external_data_plane_artifact.py b/infra/deploy-runner/test_external_data_plane_artifact.py index 1082d46..009bc74 100644 --- a/infra/deploy-runner/test_external_data_plane_artifact.py +++ b/infra/deploy-runner/test_external_data_plane_artifact.py @@ -19,6 +19,7 @@ EXPECTED_ENTRIES = [ "platform/packages/external-provider-contract/src/data-plane.mjs", "platform/packages/external-provider-contract/src/data-product.mjs", "platform/packages/external-provider-contract/src/intake-batch.mjs", + "platform/packages/external-provider-contract/src/index.mjs", "platform/packages/external-provider-contract/src/sensitive-field-policy.mjs", ] diff --git a/infra/deploy-runner/test_n8n_private_extension.py b/infra/deploy-runner/test_n8n_private_extension.py index 1ceb194..b75c7fc 100644 --- a/infra/deploy-runner/test_n8n_private_extension.py +++ b/infra/deploy-runner/test_n8n_private_extension.py @@ -17,7 +17,7 @@ SCRIPT_DIR = Path(__file__).resolve().parent PLATFORM_ROOT = SCRIPT_DIR.parent.parent RUNNER_PATH = SCRIPT_DIR / "nodedc-deploy" BUILDER_PATH = SCRIPT_DIR / "build-n8n-private-extension-artifact.mjs" -PATCH_ID = "n8n-nodes-ndc-release-20260716-003" +PATCH_ID = "n8n-nodes-ndc-history-read-20260717-004" EXPECTED_NODES = [ "dist/nodes/NdcDataProductPublish/NdcDataProductPublish.node.js", "dist/nodes/NdcDataProductRead/NdcDataProductRead.node.js", @@ -106,7 +106,7 @@ class N8nPrivateExtensionPolicyTest(unittest.TestCase): manifest, entries, _payload = RUNNER.load_artifact(self.artifacts[0], Path(directory)) self.assertEqual(manifest["component"], "n8n-private-extension") self.assertEqual(len(entries), 1) - self.assertRegex(entries[0], r"^releases/n8n-nodes-ndc/0\.1\.2-[a-f0-9]{16}$") + self.assertRegex(entries[0], r"^releases/n8n-nodes-ndc/0\.1\.3-[a-f0-9]{16}$") with tarfile.open(self.artifacts[0], "r:gz") as archive: names = archive.getnames() @@ -118,7 +118,7 @@ class N8nPrivateExtensionPolicyTest(unittest.TestCase): with tarfile.open(fileobj=io.BytesIO(package), mode="r:gz") as archive: package_json_member = archive.getmember("package/package.json") package_json = json.loads(archive.extractfile(package_json_member).read()) - self.assertEqual(package_json["version"], "0.1.2") + self.assertEqual(package_json["version"], "0.1.3") self.assertEqual(package_json["n8n"]["nodes"], EXPECTED_NODES) self.assertEqual(package_json["n8n"]["credentials"], EXPECTED_CREDENTIALS) for node_path in EXPECTED_NODES: @@ -154,7 +154,7 @@ class N8nPrivateExtensionPolicyTest(unittest.TestCase): "requiresPreActivationVerification": True, } self.assertEqual(release["schemaVersion"], "nodedc.n8n-private-extension-release/v2") - self.assertEqual(release["package"]["version"], "0.1.2") + self.assertEqual(release["package"]["version"], "0.1.3") self.assertEqual(release["activation"]["rollbackBaselinePolicy"], expected_policy) self.assertEqual(rollback["schemaVersion"], "nodedc.n8n-private-extension-rollback/v2") self.assertEqual(rollback["baselinePolicy"], expected_policy) diff --git a/packages/external-provider-contract/README.md b/packages/external-provider-contract/README.md index da91af8..f49122b 100644 --- a/packages/external-provider-contract/README.md +++ b/packages/external-provider-contract/README.md @@ -91,6 +91,8 @@ Provider-neutral runtime использует отдельные wire schemas: - `nodedc.data-product.publish/v1` — unscoped publish request от `NDC Data Product Publish`; содержит только batch identity и canonical facts; - `nodedc.data-product.snapshot/v1` — согласованный current snapshot с cursor; +- `nodedc.data-product.history/v1` — bounded Timescale history window с + `from/to`, provider-neutral `sourceIds`, resolution и opaque keyset cursor; - `nodedc.data-product.patch/v1` — committed upsert operations из durable outbox с `previousCursor`/`cursor`. @@ -109,9 +111,11 @@ binding; shared internal bearer и caller-provided scope headers являютс stream с усечённой базой. `nextPageCursor` зарезервирован для будущего отдельно версионируемого query contract и текущим bounded runtime не выдаётся. -Для большей cardinality definition заранее раскладывается по стабильным -partition Data Products с независимыми snapshot/patch cursors либо использует -будущий query contract с единым snapshot barrier и continuation semantics. +Для большей current cardinality definition заранее раскладывается по стабильным +partition Data Products с независимыми snapshot/patch cursors. History является +отдельным immutable-window query: cursor привязан digest-ом к exact +`from/to/resolution/sourceIds`, поэтому его нельзя переиспользовать с другим +запросом; offset pagination запрещена. Offset/source-ID pagination поверх меняющегося current snapshot запрещена: между страницами она способна потерять или задублировать изменения относительно patch cursor. diff --git a/packages/external-provider-contract/src/data-plane.mjs b/packages/external-provider-contract/src/data-plane.mjs index ce4d6ee..b48ba78 100644 --- a/packages/external-provider-contract/src/data-plane.mjs +++ b/packages/external-provider-contract/src/data-plane.mjs @@ -1,9 +1,11 @@ export { EXTERNAL_PROVIDER_CONTRACT_VERSION } from "./contract-version.mjs"; export { validateIntakeBatch } from "./intake-batch.mjs"; export { + DATA_PRODUCT_HISTORY_SCHEMA_VERSION, DATA_PRODUCT_PATCH_SCHEMA_VERSION, DATA_PRODUCT_PUBLISH_SCHEMA_VERSION, DATA_PRODUCT_SNAPSHOT_SCHEMA_VERSION, + validateDataProductHistory, validateDataProductPatch, validateDataProductPublish, validateDataProductSnapshot, diff --git a/packages/external-provider-contract/src/data-product.mjs b/packages/external-provider-contract/src/data-product.mjs index 1a44ffa..32b0fdc 100644 --- a/packages/external-provider-contract/src/data-product.mjs +++ b/packages/external-provider-contract/src/data-product.mjs @@ -1,6 +1,7 @@ export const DATA_PRODUCT_PUBLISH_SCHEMA_VERSION = "nodedc.data-product.publish/v1"; export const DATA_PRODUCT_SNAPSHOT_SCHEMA_VERSION = "nodedc.data-product.snapshot/v1"; export const DATA_PRODUCT_PATCH_SCHEMA_VERSION = "nodedc.data-product.patch/v1"; +export const DATA_PRODUCT_HISTORY_SCHEMA_VERSION = "nodedc.data-product.history/v1"; import { SECRET_LIKE_KEY, SECRET_LIKE_VALUE } from "./sensitive-field-policy.mjs"; @@ -71,6 +72,67 @@ export function validateDataProductSnapshot(value) { return result(errors); } +export function validateDataProductHistory(value) { + const errors = envelopeErrors(value, DATA_PRODUCT_HISTORY_SCHEMA_VERSION, "history"); + rejectUnknownKeys(value, new Set(["schemaVersion", "dataProduct", "generatedAt", "query", "facts", "nextCursor"]), "history", errors); + requiredIsoTimestamp(value?.generatedAt, "generatedAt", errors); + if (!isPlainObject(value?.query)) { + errors.push("query_must_be_object"); + } else { + rejectUnknownKeys(value.query, new Set(["from", "to", "resolutionMs", "sourceIds", "order"]), "query", errors); + requiredIsoTimestamp(value.query.from, "query.from", errors); + requiredIsoTimestamp(value.query.to, "query.to", errors); + if ( + !Number.isNaN(Date.parse(value.query.from)) + && !Number.isNaN(Date.parse(value.query.to)) + && Date.parse(value.query.from) >= Date.parse(value.query.to) + ) errors.push("query.range_invalid"); + if (!Number.isInteger(value.query.resolutionMs) || value.query.resolutionMs < 1000) { + errors.push("query.resolutionMs_invalid"); + } + if (!Array.isArray(value.query.sourceIds)) { + errors.push("query.sourceIds_must_be_array"); + } else { + value.query.sourceIds.forEach((sourceId, index) => requiredIdentifier(sourceId, `query.sourceIds[${index}]`, errors)); + if (JSON.stringify(value.query.sourceIds) !== JSON.stringify([...new Set(value.query.sourceIds)].sort())) { + errors.push("query.sourceIds_must_be_unique_and_sorted"); + } + } + if (value.query.order !== "asc") errors.push("query.order_must_be_asc"); + } + if (!Array.isArray(value?.facts)) { + errors.push("facts_must_be_array"); + } else { + let previousOrderKey = null; + value.facts.forEach((fact, index) => { + validateCanonicalFact(fact, `facts[${index}]`, errors, { allowBucketStart: true }); + if (isPlainObject(fact)) { + requiredIsoTimestamp(fact.bucketStart, `facts[${index}].bucketStart`, errors); + const bucketTime = Date.parse(fact.bucketStart); + if ( + !Number.isNaN(bucketTime) + && isPlainObject(value?.query) + && ( + bucketTime < Date.parse(value.query.from) + || bucketTime >= Date.parse(value.query.to) + ) + ) errors.push(`facts[${index}].bucketStart_outside_query_range`); + const orderKey = `${fact.bucketStart}\u0000${fact.sourceId}\u0000${fact.semanticType}`; + if (previousOrderKey !== null && orderKey <= previousOrderKey) { + errors.push(`facts[${index}]_not_strictly_ordered`); + } + previousOrderKey = orderKey; + } + }); + } + if (value?.nextCursor !== undefined) { + requiredString(value.nextCursor, "nextCursor", errors); + if (!/^[A-Za-z0-9_-]{1,1024}$/.test(String(value.nextCursor || ""))) errors.push("nextCursor_invalid"); + } + if (containsSecretLikeMaterial(value)) errors.push("history_must_not_contain_secret_material"); + return result(errors); +} + export function validateDataProductPatch(value) { const errors = envelopeErrors(value, DATA_PRODUCT_PATCH_SCHEMA_VERSION, "patch"); rejectUnknownKeys(value, new Set(["schemaVersion", "dataProduct", "cursor", "previousCursor", "emittedAt", "operations"]), "patch", errors); @@ -108,13 +170,14 @@ function envelopeErrors(value, schemaVersion, label) { return errors; } -function validateFact(value, path, errors, { maxAttributesBytes, canonical = false }) { +function validateFact(value, path, errors, { maxAttributesBytes, canonical = false, allowBucketStart = false }) { if (!isPlainObject(value)) { errors.push(`${path}_must_be_object`); return; } const allowedKeys = new Set(["sourceId", "semanticType", "observedAt", "attributes", "geometry"]); if (canonical) allowedKeys.add("receivedAt"); + if (allowBucketStart) allowedKeys.add("bucketStart"); rejectUnknownKeys(value, allowedKeys, path, errors); requiredIdentifier(value.sourceId, `${path}.sourceId`, errors); requiredIdentifier(value.semanticType, `${path}.semanticType`, errors); @@ -129,8 +192,8 @@ function validateFact(value, path, errors, { maxAttributesBytes, canonical = fal if (value.geometry !== undefined) validatePointGeometry(value.geometry, `${path}.geometry`, errors); } -function validateCanonicalFact(value, path, errors) { - validateFact(value, path, errors, { maxAttributesBytes: 64 * 1024, canonical: true }); +function validateCanonicalFact(value, path, errors, { allowBucketStart = false } = {}) { + validateFact(value, path, errors, { maxAttributesBytes: 64 * 1024, canonical: true, allowBucketStart }); if (!isPlainObject(value)) return; requiredIsoTimestamp(value.receivedAt, `${path}.receivedAt`, errors); } diff --git a/packages/external-provider-contract/src/index.mjs b/packages/external-provider-contract/src/index.mjs index 2fa8c3e..e0dd018 100644 --- a/packages/external-provider-contract/src/index.mjs +++ b/packages/external-provider-contract/src/index.mjs @@ -19,9 +19,11 @@ export { } from "./provider-package.mjs"; export { + DATA_PRODUCT_HISTORY_SCHEMA_VERSION, DATA_PRODUCT_PATCH_SCHEMA_VERSION, DATA_PRODUCT_PUBLISH_SCHEMA_VERSION, DATA_PRODUCT_SNAPSHOT_SCHEMA_VERSION, + validateDataProductHistory, validateDataProductPatch, validateDataProductPublish, validateDataProductSnapshot, diff --git a/packages/external-provider-contract/test/data-product.test.mjs b/packages/external-provider-contract/test/data-product.test.mjs index 693cb93..2ff5ddd 100644 --- a/packages/external-provider-contract/test/data-product.test.mjs +++ b/packages/external-provider-contract/test/data-product.test.mjs @@ -1,8 +1,10 @@ import assert from "node:assert/strict"; import { + DATA_PRODUCT_HISTORY_SCHEMA_VERSION, DATA_PRODUCT_PATCH_SCHEMA_VERSION, DATA_PRODUCT_PUBLISH_SCHEMA_VERSION, DATA_PRODUCT_SNAPSHOT_SCHEMA_VERSION, + validateDataProductHistory, validateDataProductPatch, validateDataProductPublish, validateDataProductSnapshot, @@ -89,6 +91,45 @@ assert.equal(validateDataProductSnapshot({ ...snapshot, facts: [{ ...canonicalFact, attributes: { status: "ndc_edprb_forbidden-reader-token" } }], }).errors.includes("snapshot_must_not_contain_secret_material"), true); + +const history = { + schemaVersion: DATA_PRODUCT_HISTORY_SCHEMA_VERSION, + dataProduct: { id: "fleet.positions.current.v1", version: "1.0.0" }, + generatedAt: "2026-07-15T10:05:00.000Z", + query: { + from: "2026-07-15T10:00:00.000Z", + to: "2026-07-15T10:05:00.000Z", + resolutionMs: 60_000, + sourceIds: ["unit-42"], + order: "asc", + }, + facts: [{ ...canonicalFact, bucketStart: "2026-07-15T10:00:00.000Z" }], + nextCursor: "opaque_cursor", +}; +assert.equal(validateDataProductHistory(history).ok, true); +assert.equal(validateDataProductHistory({ + ...history, + query: { ...history.query, order: "desc" }, +}).errors.includes("query.order_must_be_asc"), true); +assert.equal(validateDataProductHistory({ + ...history, + query: { ...history.query, from: history.query.to, to: history.query.from }, +}).errors.includes("query.range_invalid"), true); +assert.equal(validateDataProductHistory({ + ...history, + query: { ...history.query, sourceIds: ["unit-42", "unit-42"] }, +}).errors.includes("query.sourceIds_must_be_unique_and_sorted"), true); +assert.equal(validateDataProductHistory({ + ...history, + facts: [ + { ...history.facts[0], sourceId: "unit-43" }, + history.facts[0], + ], +}).errors.includes("facts[1]_not_strictly_ordered"), true); +assert.equal(validateDataProductHistory({ + ...history, + facts: [{ ...history.facts[0], attributes: { token: "forbidden" } }], +}).errors.includes("history_must_not_contain_secret_material"), true); assert.equal(validateDataProductSnapshot({ ...snapshot, facts: [{ ...canonicalFact, attributes: { status: `ndc_edppr_${"P".repeat(43)}` } }], diff --git a/packages/n8n-nodes-ndc/README.md b/packages/n8n-nodes-ndc/README.md index bc8e1ec..5915897 100644 --- a/packages/n8n-nodes-ndc/README.md +++ b/packages/n8n-nodes-ndc/README.md @@ -9,8 +9,8 @@ qualified: - `n8n-nodes-ndc.ndcDataProductPublish` — publishes canonical facts into an approved Data Product. -- `n8n-nodes-ndc.ndcDataProductRead` — reads the current snapshot of an - approved Data Product. +- `n8n-nodes-ndc.ndcDataProductRead` — reads the current snapshot or a bounded, + cursor-paged history window of an approved Data Product. - `n8n-nodes-ndc.ndcFoundryBinding` — creates or updates a declarative Foundry page-slot binding. This is a control-plane operation, not a runtime data transport. @@ -42,6 +42,7 @@ never accepts them from a workflow. - `GET /internal/data-plane/v1/reader/data-products` - `POST /internal/data-plane/v1/data-products/:dataProductId/publish` - `GET /internal/data-plane/v1/data-products/:dataProductId/snapshot` +- `GET /internal/data-plane/v1/data-products/:dataProductId/history` - `GET /internal/foundry/v1/data-products` - `POST /internal/foundry/v1/data-product-bindings` @@ -58,14 +59,20 @@ materializes actor, owner and exact application/page/binding/product scope from that grant; none of those authorization claims are accepted from headers or workflow data. -`NDC Data Product Read` supports only the bounded -`nodedc.data-product.snapshot/v1` contract. Its `Page Size` parameter is a +`NDC Data Product Read` supports the bounded +`nodedc.data-product.snapshot/v1` contract. Its snapshot `Page Size` parameter is a safety ceiling, not pagination: the complete scoped current projection must fit within 5000 entity keys. A larger product returns `data_product_snapshot_limit_exceeded`; the node must not assemble independent pages or start a patch stream from an incomplete snapshot. Large products need -stable partition Data Products or a separately versioned query contract with a -single snapshot barrier. +stable partition Data Products. + +History mode uses `nodedc.data-product.history/v1` and requires a closed +`from`/`to` interval. It supports provider-neutral `sourceIds`, a resolution +that is a multiple of the Data Product native sampling interval, a limit up to +5000 points and an opaque keyset continuation cursor. The response stays as one +envelope item so L2 logic can preserve `query` and `nextCursor` while iterating. +The workflow never selects a database, provider, tenant or connection. ## Publish input diff --git a/packages/n8n-nodes-ndc/nodes/NdcDataProductRead/NdcDataProductRead.node.ts b/packages/n8n-nodes-ndc/nodes/NdcDataProductRead/NdcDataProductRead.node.ts index 01e489d..6d88ee1 100644 --- a/packages/n8n-nodes-ndc/nodes/NdcDataProductRead/NdcDataProductRead.node.ts +++ b/packages/n8n-nodes-ndc/nodes/NdcDataProductRead/NdcDataProductRead.node.ts @@ -9,6 +9,7 @@ import { NodeConnectionTypes, NodeOperationError } from 'n8n-workflow'; import { DATA_PRODUCT_BASE_PATH, + DATA_PRODUCT_HISTORY_SUFFIX, DATA_PRODUCT_READER_CATALOG_PATH, DATA_PRODUCT_SNAPSHOT_SUFFIX, DATA_PLANE_DEFAULT_BASE_URL, @@ -16,7 +17,12 @@ import { NDC_DATA_PRODUCT_READER_CREDENTIAL, NDC_NODE_ICON, } from '../shared/constants'; -import { normalizeSnapshotFacts, snapshotReadParameters } from '../shared/contracts'; +import { + historyReadParameters, + normalizeHistoryResponse, + normalizeSnapshotFacts, + snapshotReadParameters, +} from '../shared/contracts'; import { safeHttpError, safeInputError } from '../shared/errors'; import { loadDataProductOptions, ndcRequest, serviceBaseUrl } from '../shared/http'; @@ -26,14 +32,25 @@ export class NdcDataProductRead implements INodeType { name: 'ndcDataProductRead', icon: NDC_NODE_ICON, group: ['input'], - version: 1, - subtitle: '={{$parameter["dataProductId"]}}', - description: 'Read a scoped NDC Data Product snapshot', + version: [1, 2], + subtitle: '={{$parameter["mode"] || "snapshot"}} · {{$parameter["dataProductId"]}}', + description: 'Read a scoped NDC Data Product snapshot or bounded history', defaults: { name: 'NDC Data Product Read' }, inputs: [NodeConnectionTypes.Main], outputs: [NodeConnectionTypes.Main], credentials: [{ name: NDC_DATA_PRODUCT_READER_CREDENTIAL, required: true }], properties: [ + { + displayName: 'Mode', + name: 'mode', + type: 'options', + options: [ + { name: 'Current Snapshot', value: 'snapshot' }, + { name: 'History', value: 'history' }, + ], + default: 'snapshot', + required: true, + }, { displayName: 'Data Product Name or ID', name: 'dataProductId', @@ -50,6 +67,59 @@ export class NdcDataProductRead implements INodeType { default: 1000, required: true, description: 'Maximum number of snapshot facts to return', + displayOptions: { show: { mode: ['snapshot'] } }, + }, + { + displayName: 'From', + name: 'from', + type: 'dateTime', + default: '', + required: true, + displayOptions: { show: { mode: ['history'] } }, + }, + { + displayName: 'To', + name: 'to', + type: 'dateTime', + default: '', + required: true, + displayOptions: { show: { mode: ['history'] } }, + }, + { + displayName: 'Resolution (MS)', + name: 'resolutionMs', + type: 'number', + typeOptions: { minValue: 1000, maxValue: 86400000, numberStepSize: 1000 }, + default: 60000, + required: true, + description: 'Provider-neutral history bucket size; must be a multiple of the Data Product native history interval', + displayOptions: { show: { mode: ['history'] } }, + }, + { + displayName: 'Source IDs', + name: 'sourceIds', + type: 'string', + default: '', + description: 'Optional comma-separated entity IDs; empty reads all entities in the scoped Data Product', + displayOptions: { show: { mode: ['history'] } }, + }, + { + displayName: 'Limit', + name: 'limit', + type: 'number', + typeOptions: { minValue: 1, maxValue: 5000, numberStepSize: 1 }, + default: 50, + required: true, + description: 'Max number of results to return', + displayOptions: { show: { mode: ['history'] } }, + }, + { + displayName: 'Cursor', + name: 'cursor', + type: 'string', + default: '', + description: 'Opaque continuation cursor returned by the previous history page', + displayOptions: { show: { mode: ['history'] } }, }, ], }; @@ -69,7 +139,48 @@ export class NdcDataProductRead implements INodeType { }; async execute(this: IExecuteFunctions): Promise { + const mode = this.getNodeParameter('mode', 0, 'snapshot') as string; const dataProductId = this.getNodeParameter('dataProductId', 0) as string; + if (mode === 'history') { + let parameters; + try { + parameters = historyReadParameters({ + dataProductId, + from: this.getNodeParameter('from', 0), + to: this.getNodeParameter('to', 0), + resolutionMs: this.getNodeParameter('resolutionMs', 0, 60000), + sourceIds: this.getNodeParameter('sourceIds', 0, ''), + limit: this.getNodeParameter('limit', 0, 1000), + cursor: this.getNodeParameter('cursor', 0, ''), + }); + } catch (error) { + throw new NodeOperationError(this.getNode(), safeInputError(error, 'ndc_data_product_history_input_invalid')); + } + try { + const baseUrl = serviceBaseUrl(DATA_PLANE_BASE_URL_ENV, DATA_PLANE_DEFAULT_BASE_URL); + const response = await ndcRequest(this, NDC_DATA_PRODUCT_READER_CREDENTIAL, { + method: 'GET', + url: `${baseUrl}${DATA_PRODUCT_BASE_PATH}/${parameters.encodedProductId}${DATA_PRODUCT_HISTORY_SUFFIX}`, + qs: { + from: parameters.from, + to: parameters.to, + resolutionMs: parameters.resolutionMs, + ...(parameters.sourceIds.length ? { sourceIds: parameters.sourceIds.join(',') } : {}), + limit: parameters.limit, + ...(parameters.cursor ? { cursor: parameters.cursor } : {}), + }, + }); + return [[{ json: normalizeHistoryResponse(response), pairedItem: { item: 0 } }]]; + } catch (error) { + const safe = error instanceof Error && error.message === 'data_product_history_invalid' + ? safeInputError(error, 'ndc_data_product_history_invalid') + : safeHttpError(error, 'ndc_data_product_history_read_failed'); + throw new NodeOperationError(this.getNode(), safe); + } + } + if (mode !== 'snapshot') { + throw new NodeOperationError(this.getNode(), safeInputError(new Error('read_mode_invalid'), 'ndc_data_product_read_input_invalid')); + } const pageSize = this.getNodeParameter('pageSize', 0, 1000) as number; let parameters; try { diff --git a/packages/n8n-nodes-ndc/nodes/shared/constants.ts b/packages/n8n-nodes-ndc/nodes/shared/constants.ts index 9211afa..482fff5 100644 --- a/packages/n8n-nodes-ndc/nodes/shared/constants.ts +++ b/packages/n8n-nodes-ndc/nodes/shared/constants.ts @@ -5,6 +5,7 @@ export const DATA_PRODUCT_WRITER_CATALOG_PATH = '/internal/data-plane/v1/writer/ export const DATA_PRODUCT_READER_CATALOG_PATH = '/internal/data-plane/v1/reader/data-products'; export const DATA_PRODUCT_BASE_PATH = '/internal/data-plane/v1/data-products'; export const DATA_PRODUCT_SNAPSHOT_SUFFIX = '/snapshot'; +export const DATA_PRODUCT_HISTORY_SUFFIX = '/history'; export const FOUNDRY_BINDING_PATH = '/internal/foundry/v1/data-product-bindings'; export const FOUNDRY_CATALOG_PATH = '/internal/foundry/v1/data-products'; diff --git a/packages/n8n-nodes-ndc/nodes/shared/contracts.ts b/packages/n8n-nodes-ndc/nodes/shared/contracts.ts index 4b035c9..dc88a7e 100644 --- a/packages/n8n-nodes-ndc/nodes/shared/contracts.ts +++ b/packages/n8n-nodes-ndc/nodes/shared/contracts.ts @@ -118,6 +118,19 @@ export function normalizeSnapshotFacts(value: unknown): IDataObject[] { }); } +export function normalizeHistoryResponse(value: unknown): IDataObject { + if (!isObject(value) || value.schemaVersion !== 'nodedc.data-product.history/v1' + || !isObject(value.dataProduct) || !isObject(value.query) || !Array.isArray(value.facts)) { + throw new Error('data_product_history_invalid'); + } + value.facts.forEach((fact, index) => { + if (!isObject(fact) || Number.isNaN(Date.parse(String(fact.bucketStart ?? '')))) { + throw new Error(`history_fact_${index}_invalid`); + } + }); + return value; +} + export function snapshotReadParameters( dataProductId: unknown, pageSize: unknown, @@ -132,6 +145,52 @@ export function snapshotReadParameters( }; } +export function historyReadParameters(value: { + dataProductId: unknown; + from: unknown; + to: unknown; + resolutionMs: unknown; + sourceIds: unknown; + limit: unknown; + cursor: unknown; +}): { + encodedProductId: string; + from: string; + to: string; + resolutionMs: number; + sourceIds: string[]; + limit: number; + cursor: string; +} { + const from = requireIsoTimestamp(value.from, 'from'); + const to = requireIsoTimestamp(value.to, 'to'); + if (Date.parse(from) >= Date.parse(to)) throw new Error('history_range_invalid'); + const resolutionMs = Number(value.resolutionMs); + if (!Number.isInteger(resolutionMs) || resolutionMs < 1000 || resolutionMs > 24 * 60 * 60 * 1000) { + throw new Error('history_resolution_invalid'); + } + const limit = Number(value.limit); + if (!Number.isInteger(limit) || limit < 1 || limit > 5000) throw new Error('history_limit_invalid'); + const sourceIds = [...new Set((Array.isArray(value.sourceIds) + ? value.sourceIds + : String(value.sourceIds ?? '').split(',')) + .map((item) => String(item ?? '').trim()) + .filter(Boolean) + .map((item) => requireIdentifier(item, 'sourceIds')))]; + if (sourceIds.length > 1000) throw new Error('history_source_ids_limit_exceeded'); + const cursor = String(value.cursor ?? '').trim(); + if (cursor && (cursor.length > 1024 || !/^[A-Za-z0-9_-]+$/.test(cursor))) throw new Error('history_cursor_invalid'); + return { + encodedProductId: encodeIdentifierPath(value.dataProductId, 'dataProductId'), + from, + to, + resolutionMs, + sourceIds, + limit, + cursor, + }; +} + export function dataProductOptions(value: unknown): INodeListSearchItems[] { if (!isObject(value)) throw new Error('data_product_catalog_invalid'); const rows = Array.isArray(value.dataProducts) diff --git a/packages/n8n-nodes-ndc/package-lock.json b/packages/n8n-nodes-ndc/package-lock.json index 7342e85..d974659 100644 --- a/packages/n8n-nodes-ndc/package-lock.json +++ b/packages/n8n-nodes-ndc/package-lock.json @@ -1,12 +1,12 @@ { "name": "n8n-nodes-ndc", - "version": "0.1.2", + "version": "0.1.3", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "n8n-nodes-ndc", - "version": "0.1.2", + "version": "0.1.3", "license": "UNLICENSED", "devDependencies": { "@n8n/node-cli": "0.39.3", diff --git a/packages/n8n-nodes-ndc/package.json b/packages/n8n-nodes-ndc/package.json index 9785ce8..ac4c852 100644 --- a/packages/n8n-nodes-ndc/package.json +++ b/packages/n8n-nodes-ndc/package.json @@ -1,6 +1,6 @@ { "name": "n8n-nodes-ndc", - "version": "0.1.2", + "version": "0.1.3", "description": "Private NODE.DC nodes for scoped data products and Foundry bindings.", "private": true, "license": "UNLICENSED", diff --git a/packages/n8n-nodes-ndc/test/package-policy.test.cjs b/packages/n8n-nodes-ndc/test/package-policy.test.cjs index ff8ac94..dd7614d 100644 --- a/packages/n8n-nodes-ndc/test/package-policy.test.cjs +++ b/packages/n8n-nodes-ndc/test/package-policy.test.cjs @@ -172,8 +172,9 @@ async function main() { restoreEnvironment('NDC_DATA_PLANE_BASE_URL', previousDefaultProbe); const readProperties = nodes.get('NdcDataProductRead').description.properties.map((property) => property.name); - assert.equal(readProperties.includes('cursor'), false, 'snapshot endpoint has no cursor input'); - assert.equal(readProperties.includes('history'), false, 'history must stay hidden until its endpoint exists'); + assert.equal(readProperties.includes('mode'), true, 'read mode must expose snapshot/history explicitly'); + assert.equal(readProperties.includes('cursor'), true, 'history endpoint must expose its opaque continuation cursor'); + assert.equal(readProperties.includes('sourceIds'), true, 'history filtering remains provider-neutral'); const pageSizeProperty = nodes.get('NdcDataProductRead').description.properties.find((property) => property.name === 'pageSize'); assert.equal(pageSizeProperty.typeOptions.minValue, 1); assert.equal(pageSizeProperty.typeOptions.maxValue, 5000, 'bounded snapshot v1 must not expose more than 5000 facts'); @@ -255,6 +256,51 @@ async function assertNodeHttpContracts(nodes, externalContract, constants) { assert.deepEqual(readCalls[0].options.qs, { limit: 1000 }); assert.equal(readResult[0][0].json.sourceId, 'fleet.unit.42'); + const historyCalls = []; + const historyResponse = { + schemaVersion: 'nodedc.data-product.history/v1', + dataProduct: { id: 'fleet.positions.current.v1', version: '1.0.0' }, + generatedAt: '2026-07-15T12:05:00.000Z', + query: { + from: '2026-07-15T12:00:00.000Z', + to: '2026-07-15T12:05:00.000Z', + resolutionMs: 60000, + sourceIds: ['fleet.unit.42'], + order: 'asc', + }, + facts: [{ ...canonicalFact(), receivedAt: '2026-07-15T12:00:00.100Z', bucketStart: '2026-07-15T12:00:00.000Z' }], + nextCursor: 'opaque_cursor', + }; + const historyContext = executionContext({ + parameters: { + mode: 'history', + dataProductId: 'fleet.positions.current.v1', + from: '2026-07-15T12:00:00.000Z', + to: '2026-07-15T12:05:00.000Z', + resolutionMs: 60000, + sourceIds: 'fleet.unit.42', + limit: 1000, + cursor: '', + }, + response: historyResponse, + calls: historyCalls, + }); + const historyResult = await nodes.get('NdcDataProductRead').execute.call(historyContext); + assert.equal(historyCalls[0].credentialName, 'ndcDataProductReaderApi'); + assert.equal(historyCalls[0].options.method, 'GET'); + assert.equal( + historyCalls[0].options.url, + 'http://data-plane.test/internal/data-plane/v1/data-products/fleet.positions.current.v1/history', + ); + assert.deepEqual(historyCalls[0].options.qs, { + from: '2026-07-15T12:00:00.000Z', + to: '2026-07-15T12:05:00.000Z', + resolutionMs: 60000, + sourceIds: 'fleet.unit.42', + limit: 1000, + }); + assert.deepEqual(historyResult[0][0].json, historyResponse); + const foundryCalls = []; const foundryContext = executionContext({ parameters: { diff --git a/services/external-data-plane/src/data-product-delivery.mjs b/services/external-data-plane/src/data-product-delivery.mjs index 646d31e..8758953 100644 --- a/services/external-data-plane/src/data-product-delivery.mjs +++ b/services/external-data-plane/src/data-product-delivery.mjs @@ -1,9 +1,14 @@ import { createHash, randomUUID } from "node:crypto"; import { + DATA_PRODUCT_HISTORY_SCHEMA_VERSION, DATA_PRODUCT_PATCH_SCHEMA_VERSION, DATA_PRODUCT_SNAPSHOT_SCHEMA_VERSION, } from "@nodedc/external-provider-contract/data-plane"; +const IDENTIFIER = /^[a-z][a-z0-9._:-]{2,127}$/; +const MAX_HISTORY_RESOLUTION_MS = 24 * 60 * 60 * 1000; +const MAX_HISTORY_SOURCE_IDS = 1000; + export async function loadDataProductDefinition(db, dataProductId, { activeOnly = true } = {}) { const result = await db.query( `select id, version, ontology_revision as "ontologyRevision", @@ -216,6 +221,121 @@ export async function readDataProductSnapshot(pool, binding, definition, { limit } } +export function normalizeHistoryReadOptions(value, definition) { + const policy = definition?.historyPolicy || {}; + if (policy.mode === "none") throw deliveryError("data_product_history_not_supported", 409); + const from = new Date(String(value?.from || "")); + const to = new Date(String(value?.to || "")); + if (Number.isNaN(from.getTime()) || Number.isNaN(to.getTime()) || from >= to) { + throw deliveryError("data_product_history_range_invalid", 400); + } + const retentionDays = Math.max(1, Number(policy.retentionDays) || 1); + if (to.getTime() - from.getTime() > retentionDays * 24 * 60 * 60 * 1000) { + throw deliveryError("data_product_history_range_exceeds_retention", 400); + } + const nativeResolutionMs = policy.mode === "sampled" ? Number(policy.intervalMs) : 1000; + const resolutionMs = value?.resolutionMs === undefined || value?.resolutionMs === "" + ? nativeResolutionMs + : Number(value.resolutionMs); + if ( + !Number.isInteger(resolutionMs) + || resolutionMs < nativeResolutionMs + || resolutionMs > MAX_HISTORY_RESOLUTION_MS + || resolutionMs % nativeResolutionMs !== 0 + ) throw deliveryError("data_product_history_resolution_invalid", 400); + const sourceIds = [...new Set((Array.isArray(value?.sourceIds) ? value.sourceIds : []) + .map((item) => String(item || "").trim()) + .filter(Boolean))]; + if (sourceIds.length > MAX_HISTORY_SOURCE_IDS || sourceIds.some((item) => !IDENTIFIER.test(item))) { + throw deliveryError("data_product_history_source_ids_invalid", 400); + } + const limit = Number(value?.limit ?? 1000); + if (!Number.isInteger(limit) || limit < 1 || limit > 5000) { + throw deliveryError("data_product_history_limit_invalid", 400); + } + const query = { + from: from.toISOString(), + to: to.toISOString(), + resolutionMs, + sourceIds: sourceIds.sort(), + order: "asc", + }; + const queryHash = createHash("sha256").update(JSON.stringify(query)).digest("hex"); + const cursor = decodeHistoryCursor(value?.cursor, queryHash); + return Object.freeze({ ...query, limit, queryHash, cursor }); +} + +export async function readDataProductHistory(pool, binding, definition, options) { + const query = normalizeHistoryReadOptions(options, definition); + const cursor = query.cursor || {}; + const rows = await pool.query( + `with sampled as ( + select + to_timestamp( + floor(extract(epoch from bucket_start) * 1000 / $8::double precision) + * $8::double precision / 1000 + ) as query_bucket, + source_id, semantic_type, observed_at, received_at, attributes, geometry, fingerprint + from external_data_plane_history + where tenant_id = $1 and connection_id = $2 and provider_id = $3 and data_product_id = $4 + and bucket_start >= $5 and bucket_start < $6 + and (cardinality($7::text[]) = 0 or source_id = any($7::text[])) + ), ranked as ( + select *, row_number() over ( + partition by query_bucket, source_id, semantic_type + order by observed_at desc, received_at desc, fingerprint desc + ) as sample_rank + from sampled + ) + select query_bucket as "bucketStart", source_id as "sourceId", semantic_type as "semanticType", + observed_at as "observedAt", received_at as "receivedAt", attributes, + case when geometry is null then null + else jsonb_build_object('type', 'Point', 'coordinates', jsonb_build_array( + ST_X(geometry::geometry), ST_Y(geometry::geometry) + )) end as geometry + from ranked + where sample_rank = 1 + and ( + $9::boolean = false + or (query_bucket, source_id, semantic_type) > ($10::timestamptz, $11::text, $12::text) + ) + order by query_bucket asc, source_id asc, semantic_type asc + limit $13`, + [ + binding.tenantId, binding.connectionId, binding.providerId, definition.id, + query.from, query.to, query.sourceIds, query.resolutionMs, + Boolean(query.cursor), cursor.bucketStart || query.from, cursor.sourceId || "", cursor.semanticType || "", + query.limit + 1, + ], + ); + const page = rows.rows.slice(0, query.limit); + const last = page.at(-1); + return { + schemaVersion: DATA_PRODUCT_HISTORY_SCHEMA_VERSION, + dataProduct: { id: definition.id, version: definition.version }, + generatedAt: new Date().toISOString(), + query: { + from: query.from, + to: query.to, + resolutionMs: query.resolutionMs, + sourceIds: query.sourceIds, + order: query.order, + }, + facts: page.map((fact) => ({ + ...publicFact(fact), + bucketStart: new Date(fact.bucketStart).toISOString(), + })), + ...(rows.rowCount > query.limit && last ? { + nextCursor: encodeHistoryCursor({ + queryHash: query.queryHash, + bucketStart: new Date(last.bucketStart).toISOString(), + sourceId: last.sourceId, + semanticType: last.semanticType, + }), + } : {}), + }; +} + export async function readPatchEvents(pool, binding, definition, after, { limit = 100 } = {}) { const client = await pool.connect(); try { @@ -450,6 +570,36 @@ function publicFact(fact) { return value; } +function encodeHistoryCursor(value) { + return Buffer.from(JSON.stringify(value), "utf8").toString("base64url"); +} + +function decodeHistoryCursor(value, queryHash) { + const raw = String(value || "").trim(); + if (!raw) return null; + if (raw.length > 1024 || !/^[A-Za-z0-9_-]+$/.test(raw)) { + throw deliveryError("data_product_history_cursor_invalid", 400); + } + let cursor; + try { + cursor = JSON.parse(Buffer.from(raw, "base64url").toString("utf8")); + } catch { + throw deliveryError("data_product_history_cursor_invalid", 400); + } + if ( + !cursor || typeof cursor !== "object" || Array.isArray(cursor) + || cursor.queryHash !== queryHash + || Number.isNaN(Date.parse(cursor.bucketStart)) + || !IDENTIFIER.test(String(cursor.sourceId || "")) + || !IDENTIFIER.test(String(cursor.semanticType || "")) + ) throw deliveryError("data_product_history_cursor_invalid", 400); + return Object.freeze({ + bucketStart: new Date(cursor.bucketStart).toISOString(), + sourceId: cursor.sourceId, + semanticType: cursor.semanticType, + }); +} + function chunkOperations(values, maxOperations, maxBytes) { const result = []; let current = []; diff --git a/services/external-data-plane/src/reader-binding.mjs b/services/external-data-plane/src/reader-binding.mjs index 7b614b8..b91e861 100644 --- a/services/external-data-plane/src/reader-binding.mjs +++ b/services/external-data-plane/src/reader-binding.mjs @@ -4,7 +4,15 @@ const IDENTIFIER = /^[a-z][a-z0-9._:-]{2,127}$/; const TOKEN_PREFIX = "ndc_edprb_"; const SECRET_LIKE_KEY = /(token|secret|password|authorization|access[_-]?token|refresh[_-]?token|api[_-]?key)/i; const REQUEST_KEYS = new Set(["source", "allowedDataProductIds", "expiresAt"]); +const MANAGED_REQUEST_KEYS = new Set([ + "source", + "allowedDataProductIds", + "expiresAt", + "generation", + "capabilityDigest", +]); const SOURCE_KEYS = new Set(["tenantId", "connectionId", "providerId"]); +const SHA256_DIGEST = /^[a-f0-9]{64}$/; export function createReaderToken() { return `${TOKEN_PREFIX}${randomBytes(32).toString("base64url")}`; @@ -35,8 +43,61 @@ export function normalizeReaderBindingRequest(value, { now = new Date(), maxTtlD return Object.freeze({ tenantId, connectionId, providerId, allowedDataProductIds, expiresAt: expiresAt.toISOString() }); } +export function normalizeManagedReaderBindingRequest(value) { + if (!isPlainObject(value) || !isPlainObject(value.source)) { + throw readerError("managed_reader_binding_request_invalid"); + } + if (containsSecretLikeKey(value)) { + throw readerError("managed_reader_binding_secret_material_forbidden"); + } + if (!hasOnlyKeys(value, MANAGED_REQUEST_KEYS) || !hasOnlyKeys(value.source, SOURCE_KEYS)) { + throw readerError("managed_reader_binding_request_fields_invalid"); + } + const tenantId = identifier(value.source.tenantId); + const connectionId = identifier(value.source.connectionId); + const providerId = identifier(value.source.providerId); + const allowedDataProductIds = uniqueIdentifiers(value.allowedDataProductIds); + if (!tenantId || !connectionId || !providerId || !allowedDataProductIds.length) { + throw readerError("managed_reader_binding_scope_invalid"); + } + if (value.expiresAt !== null) { + throw readerError("managed_reader_binding_must_be_durable"); + } + const generation = Number(value.generation); + if (!Number.isSafeInteger(generation) || generation < 1 || generation > 2_147_483_647) { + throw readerError("managed_reader_binding_generation_invalid"); + } + const capabilityDigest = String(value.capabilityDigest || "").toLowerCase(); + if (!SHA256_DIGEST.test(capabilityDigest)) { + throw readerError("managed_reader_binding_capability_digest_invalid"); + } + return Object.freeze({ + tenantId, + connectionId, + providerId, + allowedDataProductIds: Object.freeze([...allowedDataProductIds].sort()), + expiresAt: null, + generation, + capabilityDigest, + }); +} + +export function readerBindingRequestHash(policy) { + const canonical = JSON.stringify({ + tenantId: policy.tenantId, + connectionId: policy.connectionId, + providerId: policy.providerId, + allowedDataProductIds: [...policy.allowedDataProductIds].sort(), + expiresAt: null, + generation: policy.generation, + capabilityDigest: policy.capabilityDigest, + }); + return createHash("sha256").update(canonical, "utf8").digest("hex"); +} + export function assertReaderProduct(binding, dataProductId, now = new Date()) { - if (!isPlainObject(binding) || binding.active !== true || new Date(binding.expiresAt) <= now) { + const expired = binding?.expiresAt !== null && new Date(binding?.expiresAt) <= now; + if (!isPlainObject(binding) || binding.active !== true || expired) { throw readerError("reader_binding_inactive", 401); } const normalized = identifier(dataProductId); @@ -49,12 +110,14 @@ export function assertReaderProduct(binding, dataProductId, now = new Date()) { export function safeReaderBinding(binding) { return { id: binding.id, + ...(binding.bindingKey ? { bindingKey: binding.bindingKey } : {}), + ...(Number.isInteger(Number(binding.generation)) ? { generation: Number(binding.generation) } : {}), tenantId: binding.tenantId, connectionId: binding.connectionId, providerId: binding.providerId, allowedDataProductIds: uniqueIdentifiers(binding.allowedDataProductIds), active: binding.active === true, - expiresAt: new Date(binding.expiresAt).toISOString(), + expiresAt: binding.expiresAt === null ? null : new Date(binding.expiresAt).toISOString(), createdAt: binding.createdAt ? new Date(binding.createdAt).toISOString() : undefined, rotatedAt: binding.rotatedAt ? new Date(binding.rotatedAt).toISOString() : undefined, revokedAt: binding.revokedAt ? new Date(binding.revokedAt).toISOString() : undefined, diff --git a/services/external-data-plane/src/schema.mjs b/services/external-data-plane/src/schema.mjs index 17d715d..18f1af8 100644 --- a/services/external-data-plane/src/schema.mjs +++ b/services/external-data-plane/src/schema.mjs @@ -55,7 +55,7 @@ export async function migrate(pool) { payload_ref text, payload_bytes integer, received_at timestamptz not null, - expires_at timestamptz not null, + expires_at timestamptz, check (payload is not null or payload_ref is not null) ) `); @@ -72,7 +72,7 @@ export async function migrate(pool) { connection_id text not null, provider_id text not null, allowed_data_product_ids jsonb not null, - expires_at timestamptz not null, + expires_at timestamptz, active boolean not null default true, created_at timestamptz not null default now(), rotated_at timestamptz, @@ -89,6 +89,7 @@ export async function migrate(pool) { await pool.query("alter table external_data_plane_writer_bindings add column if not exists binding_key text"); await pool.query("alter table external_data_plane_writer_bindings add column if not exists request_hash text"); await pool.query("alter table external_data_plane_writer_bindings add column if not exists generation integer not null default 1"); + await pool.query("alter table external_data_plane_writer_bindings alter column expires_at drop not null"); await pool.query(` do $$ begin @@ -113,6 +114,18 @@ export async function migrate(pool) { or (binding_key is not null and request_hash ~ '^[a-f0-9]{64}$') ); end if; + if not exists ( + select 1 from pg_constraint + where conrelid = 'external_data_plane_writer_bindings'::regclass + and conname = 'external_data_plane_writer_bindings_lifecycle_ck' + ) then + alter table external_data_plane_writer_bindings + add constraint external_data_plane_writer_bindings_lifecycle_ck + check ( + (binding_key is null and expires_at is not null) + or (binding_key is not null and expires_at is null) + ) not valid; + end if; end $$ `); @@ -123,15 +136,19 @@ export async function migrate(pool) { create table if not exists external_data_plane_reader_bindings ( id uuid primary key, token_hash text not null unique, + binding_key text, + request_hash text, + generation integer not null default 1, tenant_id text not null, connection_id text not null, provider_id text not null, allowed_data_product_ids jsonb not null, - expires_at timestamptz not null, + expires_at timestamptz, active boolean not null default true, created_at timestamptz not null default now(), rotated_at timestamptz, revoked_at timestamptz, + check (generation > 0), check ( case when jsonb_typeof(allowed_data_product_ids) = 'array' then jsonb_array_length(allowed_data_product_ids) > 0 @@ -140,6 +157,50 @@ export async function migrate(pool) { ) ) `); + await pool.query("alter table external_data_plane_reader_bindings add column if not exists binding_key text"); + await pool.query("alter table external_data_plane_reader_bindings add column if not exists request_hash text"); + await pool.query("alter table external_data_plane_reader_bindings add column if not exists generation integer not null default 1"); + await pool.query("alter table external_data_plane_reader_bindings alter column expires_at drop not null"); + await pool.query(` + do $$ + begin + if not exists ( + select 1 from pg_constraint + where conrelid = 'external_data_plane_reader_bindings'::regclass + and conname = 'external_data_plane_reader_bindings_generation_positive_ck' + ) then + alter table external_data_plane_reader_bindings + add constraint external_data_plane_reader_bindings_generation_positive_ck + check (generation > 0); + end if; + if not exists ( + select 1 from pg_constraint + where conrelid = 'external_data_plane_reader_bindings'::regclass + and conname = 'external_data_plane_reader_bindings_managed_metadata_ck' + ) then + alter table external_data_plane_reader_bindings + add constraint external_data_plane_reader_bindings_managed_metadata_ck + check ( + (binding_key is null and request_hash is null) + or (binding_key is not null and request_hash ~ '^[a-f0-9]{64}$') + ); + end if; + if not exists ( + select 1 from pg_constraint + where conrelid = 'external_data_plane_reader_bindings'::regclass + and conname = 'external_data_plane_reader_bindings_lifecycle_ck' + ) then + alter table external_data_plane_reader_bindings + add constraint external_data_plane_reader_bindings_lifecycle_ck + check ( + (binding_key is null and expires_at is not null) + or (binding_key is not null and expires_at is null) + ); + end if; + end + $$ + `); + await pool.query("create unique index if not exists external_data_plane_reader_bindings_managed_key_idx on external_data_plane_reader_bindings (binding_key, generation) where binding_key is not null"); await pool.query("create index if not exists external_data_plane_reader_bindings_active_idx on external_data_plane_reader_bindings (active, expires_at)"); await pool.query(` diff --git a/services/external-data-plane/src/server.mjs b/services/external-data-plane/src/server.mjs index d0fa260..0c803c1 100644 --- a/services/external-data-plane/src/server.mjs +++ b/services/external-data-plane/src/server.mjs @@ -12,6 +12,7 @@ import { pruneBatchReceipts, pruneDataProductHistory, prunePatchOutbox, + readDataProductHistory, readDataProductSnapshot, readPatchEvents, } from "./data-product-delivery.mjs"; @@ -26,12 +27,15 @@ import { assertReaderProduct, createReaderToken, hashReaderToken, + normalizeManagedReaderBindingRequest, normalizeReaderBindingRequest, + readerBindingRequestHash, safeReaderBinding, } from "./reader-binding.mjs"; import { migrate } from "./schema.mjs"; import { createWriterToken, + canMigrateManagedWriterBindingToDurable, hashWriterToken, materializeDataProductPublish, materializeWriterBoundBatch, @@ -90,12 +94,16 @@ app.get("/healthz", asyncRoute(async (_req, res) => { providerLogic: "absent", commandTransport: "absent", writerBindings: "supported", - managedWriterBindings: "digest+idempotent-generation", + managedWriterBindings: "digest+idempotent-generation+explicit-revoke", readerBindings: "supported", - dataProductDelivery: "snapshot+durable-patch", + managedReaderBindings: "digest+idempotent-generation+explicit-revoke", + dataProductDelivery: "snapshot+history+durable-patch", legacyIntake: config.legacyIntakeEnabled ? "migration-only" : "disabled", legacyCapabilityProvisioning: config.provisionerApiEnabled ? "enabled" : "disabled", managedWriterBindingProvisioning: config.managedProvisionerApiEnabled ? "enabled" : "disabled", + managedWriterBindingLifetime: config.managedProvisionerApiEnabled ? "explicit-revoke" : "disabled", + managedReaderBindingProvisioning: config.managedProvisionerApiEnabled ? "enabled" : "disabled", + managedReaderBindingLifetime: config.managedProvisionerApiEnabled ? "explicit-revoke" : "disabled", rawRetentionSweep: { mode: "server-scheduled", lastSweepAt: lastRetentionSweepAt, @@ -169,9 +177,7 @@ app.post("/internal/data-plane/v1/intake/writer-bound", requireLegacyIntake, req app.put("/internal/data-plane/v1/writer-bindings/by-key/:bindingKey", requireManagedProvisionerApi, asyncRoute(async (req, res) => { const bindingKey = requireIdentifier(req.params.bindingKey, "managed_writer_binding_key_invalid"); - const policy = normalizeManagedWriterBindingRequest(req.body, { - maxTtlDays: config.writerBindingMaxTtlDays, - }); + const policy = normalizeManagedWriterBindingRequest(req.body); await assertRegisteredProductIds(policy.allowedDataProductIds); const requestHash = writerBindingRequestHash(policy); const client = await pool.connect(); @@ -180,7 +186,7 @@ app.put("/internal/data-plane/v1/writer-bindings/by-key/:bindingKey", requireMan await client.query("begin"); await client.query("select pg_advisory_xact_lock(hashtextextended($1, 0))", [bindingKey]); const existing = await client.query( - `select id, binding_key as "bindingKey", request_hash as "requestHash", generation, + `select id, token_hash as "capabilityDigest", binding_key as "bindingKey", request_hash as "requestHash", generation, tenant_id as "tenantId", connection_id as "connectionId", provider_id as "providerId", allowed_data_product_ids as "allowedDataProductIds", expires_at as "expiresAt", active, created_at as "createdAt", rotated_at as "rotatedAt", revoked_at as "revokedAt" @@ -192,12 +198,27 @@ app.put("/internal/data-plane/v1/writer-bindings/by-key/:bindingKey", requireMan if (existing.rowCount) { const binding = existing.rows[0]; if (binding.requestHash !== requestHash) { - throw httpError(409, "managed_writer_binding_request_conflict"); + const legacyCanMigrate = canMigrateManagedWriterBindingToDurable(binding, policy); + if (!legacyCanMigrate) throw httpError(409, "managed_writer_binding_request_conflict"); + const migrated = await client.query( + `update external_data_plane_writer_bindings + set request_hash = $3, expires_at = null, rotated_at = now() + where binding_key = $1 and generation = $2 and token_hash = $4 + and active = true and expires_at > now() + returning id, binding_key as "bindingKey", generation, + tenant_id as "tenantId", connection_id as "connectionId", provider_id as "providerId", + allowed_data_product_ids as "allowedDataProductIds", expires_at as "expiresAt", + active, created_at as "createdAt", rotated_at as "rotatedAt", revoked_at as "revokedAt"`, + [bindingKey, policy.generation, requestHash, policy.capabilityDigest], + ); + if (!migrated.rowCount) throw httpError(409, "managed_writer_binding_generation_inactive"); + response = { status: 200, idempotent: false, migrated: true, binding: migrated.rows[0] }; + } else { + if (binding.active !== true || binding.expiresAt !== null) { + throw httpError(409, "managed_writer_binding_generation_inactive"); + } + response = { status: 200, idempotent: true, migrated: false, binding }; } - if (binding.active !== true || new Date(binding.expiresAt) <= new Date()) { - throw httpError(409, "managed_writer_binding_generation_inactive"); - } - response = { status: 200, idempotent: true, binding }; } else { const inserted = await client.query( `insert into external_data_plane_writer_bindings ( @@ -237,6 +258,7 @@ app.put("/internal/data-plane/v1/writer-bindings/by-key/:bindingKey", requireMan res.status(response.status).json({ ok: true, idempotent: response.idempotent, + migrated: response.migrated === true, writerBinding: safeWriterBinding(response.binding), }); })); @@ -354,6 +376,126 @@ app.post("/internal/data-plane/v1/writer-bindings/:bindingId/revoke", requirePro res.json({ ok: true, writerBinding: safeWriterBinding(result.rows[0]) }); })); +app.put("/internal/data-plane/v1/reader-bindings/by-key/:bindingKey", requireManagedProvisionerApi, asyncRoute(async (req, res) => { + const bindingKey = requireIdentifier(req.params.bindingKey, "managed_reader_binding_key_invalid"); + const policy = normalizeManagedReaderBindingRequest(req.body); + await assertRegisteredProductIds(policy.allowedDataProductIds); + const requestHash = readerBindingRequestHash(policy); + const client = await pool.connect(); + let response; + try { + await client.query("begin"); + await client.query("select pg_advisory_xact_lock(hashtextextended($1, 0))", [bindingKey]); + const existing = await client.query( + `select id, binding_key as "bindingKey", request_hash as "requestHash", generation, + tenant_id as "tenantId", connection_id as "connectionId", provider_id as "providerId", + allowed_data_product_ids as "allowedDataProductIds", expires_at as "expiresAt", + active, created_at as "createdAt", rotated_at as "rotatedAt", revoked_at as "revokedAt" + from external_data_plane_reader_bindings + where binding_key = $1 and generation = $2 + for update`, + [bindingKey, policy.generation], + ); + if (existing.rowCount) { + const binding = existing.rows[0]; + if (binding.requestHash !== requestHash) { + throw httpError(409, "managed_reader_binding_request_conflict"); + } + if (binding.active !== true || binding.expiresAt !== null) { + throw httpError(409, "managed_reader_binding_generation_inactive"); + } + response = { status: 200, idempotent: true, binding }; + } else { + const inserted = await client.query( + `insert into external_data_plane_reader_bindings ( + id, token_hash, binding_key, request_hash, generation, + tenant_id, connection_id, provider_id, allowed_data_product_ids, expires_at + ) values ($1, $2, $3, $4, $5, $6, $7, $8, $9::jsonb, $10) + returning id, binding_key as "bindingKey", generation, + tenant_id as "tenantId", connection_id as "connectionId", provider_id as "providerId", + allowed_data_product_ids as "allowedDataProductIds", expires_at as "expiresAt", + active, created_at as "createdAt", rotated_at as "rotatedAt", revoked_at as "revokedAt"`, + [ + randomUUID(), + policy.capabilityDigest, + bindingKey, + requestHash, + policy.generation, + policy.tenantId, + policy.connectionId, + policy.providerId, + JSON.stringify(policy.allowedDataProductIds), + policy.expiresAt, + ], + ); + response = { status: 201, idempotent: false, binding: inserted.rows[0] }; + } + await client.query("commit"); + } catch (error) { + await client.query("rollback"); + if (error?.code === "23505") { + throw httpError(409, "managed_reader_binding_capability_digest_conflict"); + } + throw error; + } finally { + client.release(); + } + res.set("Cache-Control", "no-store, max-age=0"); + res.status(response.status).json({ + ok: true, + idempotent: response.idempotent, + readerBinding: safeReaderBinding(response.binding), + }); +})); + +app.post("/internal/data-plane/v1/reader-bindings/by-key/:bindingKey/generations/:generation/revoke", requireManagedProvisionerApi, asyncRoute(async (req, res) => { + const bindingKey = requireIdentifier(req.params.bindingKey, "managed_reader_binding_key_invalid"); + const generation = requirePositiveInteger(req.params.generation, "managed_reader_binding_generation_invalid"); + const client = await pool.connect(); + let binding; + let idempotent; + try { + await client.query("begin"); + await client.query("select pg_advisory_xact_lock(hashtextextended($1, 0))", [bindingKey]); + const existing = await client.query( + `select id, binding_key as "bindingKey", generation, + tenant_id as "tenantId", connection_id as "connectionId", provider_id as "providerId", + allowed_data_product_ids as "allowedDataProductIds", expires_at as "expiresAt", + active, created_at as "createdAt", rotated_at as "rotatedAt", revoked_at as "revokedAt" + from external_data_plane_reader_bindings + where binding_key = $1 and generation = $2 + for update`, + [bindingKey, generation], + ); + if (!existing.rowCount) throw httpError(404, "managed_reader_binding_not_found"); + if (existing.rows[0].active === true) { + const revoked = await client.query( + `update external_data_plane_reader_bindings + set active = false, revoked_at = now() + where binding_key = $1 and generation = $2 and active = true + returning id, binding_key as "bindingKey", generation, + tenant_id as "tenantId", connection_id as "connectionId", provider_id as "providerId", + allowed_data_product_ids as "allowedDataProductIds", expires_at as "expiresAt", + active, created_at as "createdAt", rotated_at as "rotatedAt", revoked_at as "revokedAt"`, + [bindingKey, generation], + ); + binding = revoked.rows[0]; + idempotent = false; + } else { + binding = existing.rows[0]; + idempotent = true; + } + await client.query("commit"); + } catch (error) { + await client.query("rollback"); + throw error; + } finally { + client.release(); + } + res.set("Cache-Control", "no-store, max-age=0"); + res.json({ ok: true, idempotent, readerBinding: safeReaderBinding(binding) }); +})); + app.post("/internal/data-plane/v1/reader-bindings", requireProvisionerApi, asyncRoute(async (req, res) => { const policy = normalizeReaderBindingRequest(req.body, { maxTtlDays: config.writerBindingMaxTtlDays, @@ -389,7 +531,7 @@ app.post("/internal/data-plane/v1/reader-bindings/:bindingId/rotate", requirePro const result = await pool.query( `update external_data_plane_reader_bindings set token_hash = $2, rotated_at = now() - where id = $1 and active = true and expires_at > now() + where id = $1 and binding_key is null and active = true and expires_at > now() returning id, tenant_id as "tenantId", connection_id as "connectionId", provider_id as "providerId", allowed_data_product_ids as "allowedDataProductIds", expires_at as "expiresAt", active, created_at as "createdAt", @@ -405,7 +547,7 @@ app.post("/internal/data-plane/v1/reader-bindings/:bindingId/revoke", requirePro const result = await pool.query( `update external_data_plane_reader_bindings set active = false, revoked_at = now() - where id = $1 and active = true + where id = $1 and binding_key is null and active = true returning id, tenant_id as "tenantId", connection_id as "connectionId", provider_id as "providerId", allowed_data_product_ids as "allowedDataProductIds", expires_at as "expiresAt", active, created_at as "createdAt", @@ -426,6 +568,22 @@ app.get("/internal/data-plane/v1/data-products/:dataProductId/snapshot", require res.json(snapshot); })); +app.get("/internal/data-plane/v1/data-products/:dataProductId/history", requireReaderBinding, asyncRoute(async (req, res) => { + if (hasScopeHeaders(req)) throw httpError(400, "data_product_read_scope_headers_forbidden"); + const dataProductId = assertReaderProduct(req.readerBinding, req.params.dataProductId); + const definition = await loadDataProductDefinition(pool, dataProductId); + if (!definition) throw httpError(404, "data_product_not_found"); + const history = await readDataProductHistory(pool, req.readerBinding, definition, { + from: req.query.from, + to: req.query.to, + resolutionMs: req.query.resolutionMs, + sourceIds: parseSourceIds(req.query.sourceIds), + limit: req.query.limit, + cursor: req.query.cursor, + }); + res.json(history); +})); + app.get("/internal/data-plane/v1/data-products/:dataProductId/stream", requireReaderBinding, asyncRoute(async (req, res) => { if (shuttingDown) throw httpError(503, "service_shutting_down"); if (hasScopeHeaders(req)) throw httpError(400, "data_product_read_scope_headers_forbidden"); @@ -767,7 +925,8 @@ async function resolveWriterBinding(req) { expires_at as "expiresAt", active, created_at as "createdAt", rotated_at as "rotatedAt", revoked_at as "revokedAt" from external_data_plane_writer_bindings - where token_hash = $1 and active = true and expires_at > now()`, + where token_hash = $1 and active = true + and (expires_at is null or expires_at > now())`, [hashWriterToken(token)], ); if (!result.rowCount) throw httpError(401, "writer_binding_unauthorized"); @@ -783,7 +942,8 @@ async function resolveReaderBinding(req) { expires_at as "expiresAt", active, created_at as "createdAt", rotated_at as "rotatedAt", revoked_at as "revokedAt" from external_data_plane_reader_bindings - where token_hash = $1 and active = true and expires_at > now()`, + where token_hash = $1 and active = true + and (expires_at is null or expires_at > now())`, [hashReaderToken(token)], ); if (!result.rowCount) throw httpError(401, "reader_binding_unauthorized"); @@ -797,7 +957,8 @@ async function assertReaderStreamAccess(binding, dataProductId) { join external_data_plane_products as product on product.id = $3 and product.active = true where binding.id = $1 and binding.token_hash = $2 - and binding.active = true and binding.expires_at > now() + and binding.active = true + and (binding.expires_at is null or binding.expires_at > now()) and binding.allowed_data_product_ids ? $3`, [binding.id, binding.tokenHash, dataProductId], ); @@ -883,6 +1044,12 @@ function parseCursor(value) { return cursor; } +function parseSourceIds(value) { + if (value === undefined || value === null || value === "") return []; + const values = Array.isArray(value) ? value : String(value).split(","); + return values.map((item) => String(item || "").trim()).filter(Boolean); +} + function writePatchEvent(res, event) { return writeSseFrame(res, `id: ${event.cursor}\nevent: nodedc.data-product.patch.v1\ndata: ${JSON.stringify(event)}\n\n`); } diff --git a/services/external-data-plane/src/writer-binding.mjs b/services/external-data-plane/src/writer-binding.mjs index bf5322f..55fa97d 100644 --- a/services/external-data-plane/src/writer-binding.mjs +++ b/services/external-data-plane/src/writer-binding.mjs @@ -72,7 +72,7 @@ export function normalizeWriterBindingRequest(value, { now = new Date(), maxTtlD * generated and stored inside Engine; EDP receives only its SHA-256 digest. * `generation` makes a retry address the same immutable binding generation. */ -export function normalizeManagedWriterBindingRequest(value, { now = new Date(), maxTtlDays = 90 } = {}) { +export function normalizeManagedWriterBindingRequest(value) { if (!isPlainObject(value) || !isPlainObject(value.source)) { throw writerBindingError("managed_writer_binding_request_invalid"); } @@ -83,11 +83,16 @@ export function normalizeManagedWriterBindingRequest(value, { now = new Date(), throw writerBindingError("managed_writer_binding_request_fields_invalid"); } - const scope = normalizeWriterBindingRequest({ - source: value.source, - allowedDataProductIds: value.allowedDataProductIds, - expiresAt: value.expiresAt, - }, { now, maxTtlDays }); + const tenantId = normalizeIdentifier(value.source.tenantId); + const connectionId = normalizeIdentifier(value.source.connectionId); + const providerId = normalizeIdentifier(value.source.providerId); + const allowedDataProductIds = uniqueIdentifiers(value.allowedDataProductIds); + if (!tenantId || !connectionId || !providerId || !allowedDataProductIds.length) { + throw writerBindingError("managed_writer_binding_scope_invalid"); + } + if (value.expiresAt !== null) { + throw writerBindingError("managed_writer_binding_must_be_durable"); + } const generation = Number(value.generation); if (!Number.isSafeInteger(generation) || generation < 1 || generation > 2_147_483_647) { throw writerBindingError("managed_writer_binding_generation_invalid"); @@ -98,8 +103,11 @@ export function normalizeManagedWriterBindingRequest(value, { now = new Date(), } return Object.freeze({ - ...scope, - allowedDataProductIds: Object.freeze([...scope.allowedDataProductIds].sort()), + tenantId, + connectionId, + providerId, + allowedDataProductIds: Object.freeze([...allowedDataProductIds].sort()), + expiresAt: null, generation, capabilityDigest, }); @@ -111,13 +119,26 @@ export function writerBindingRequestHash(policy) { connectionId: policy.connectionId, providerId: policy.providerId, allowedDataProductIds: [...policy.allowedDataProductIds].sort(), - expiresAt: new Date(policy.expiresAt).toISOString(), + expiresAt: null, generation: policy.generation, capabilityDigest: policy.capabilityDigest, }); return createHash("sha256").update(canonical, "utf8").digest("hex"); } +export function canMigrateManagedWriterBindingToDurable(binding, policy, now = new Date()) { + if (!isPlainObject(binding) || !isPlainObject(policy) || binding.active !== true) return false; + const expiresAt = new Date(binding.expiresAt); + if (binding.expiresAt === null || Number.isNaN(expiresAt.getTime()) || expiresAt <= now) return false; + return binding.tenantId === policy.tenantId + && binding.connectionId === policy.connectionId + && binding.providerId === policy.providerId + && binding.capabilityDigest === policy.capabilityDigest + && JSON.stringify(uniqueIdentifiers(binding.allowedDataProductIds).sort()) + === JSON.stringify(uniqueIdentifiers(policy.allowedDataProductIds).sort()) + && policy.expiresAt === null; +} + /** * Converts a caller-provided, deliberately unscoped intake envelope into the * canonical scoped form. Caller scope is rejected, never trusted or merged. @@ -202,7 +223,7 @@ export function safeWriterBinding(binding) { providerId: binding.providerId, allowedDataProductIds: uniqueIdentifiers(binding.allowedDataProductIds), active: binding.active === true, - expiresAt: new Date(binding.expiresAt).toISOString(), + expiresAt: binding.expiresAt === null ? null : new Date(binding.expiresAt).toISOString(), createdAt: binding.createdAt ? new Date(binding.createdAt).toISOString() : undefined, rotatedAt: binding.rotatedAt ? new Date(binding.rotatedAt).toISOString() : undefined, revokedAt: binding.revokedAt ? new Date(binding.revokedAt).toISOString() : undefined, @@ -214,6 +235,7 @@ export function writerBindingError(code) { } function bindingIsCurrent(binding, now) { + if (binding.expiresAt === null) return true; const expiresAt = new Date(binding.expiresAt); return !Number.isNaN(expiresAt.getTime()) && expiresAt > now; } diff --git a/services/external-data-plane/test/api.integration.test.mjs b/services/external-data-plane/test/api.integration.test.mjs index c95f37b..7eb9fa9 100644 --- a/services/external-data-plane/test/api.integration.test.mjs +++ b/services/external-data-plane/test/api.integration.test.mjs @@ -5,7 +5,7 @@ import { mkdtemp, rm, writeFile } from "node:fs/promises"; import net from "node:net"; import { tmpdir } from "node:os"; import { join } from "node:path"; -import { validateDataProductPatch, validateDataProductSnapshot } from "@nodedc/external-provider-contract/data-plane"; +import { validateDataProductHistory, validateDataProductPatch, validateDataProductSnapshot } from "@nodedc/external-provider-contract/data-plane"; import { MANAGED_PROVISIONER_HEADERS, managedProvisionerSigningPayload, @@ -77,7 +77,7 @@ try { const managedBody = { source: scope, allowedDataProductIds: [productId], - expiresAt: expiry, + expiresAt: null, generation: 1, capabilityDigest: createHash("sha256").update(managedToken, "utf8").digest("hex"), }; @@ -194,6 +194,48 @@ try { }); assert.equal(rejectedManagedGeneration1.status, 401); + const managedReaderToken = "ndc_edprb_managed_api_test_0123456789abcdefghijklmnopqrstuvwxyz"; + const managedReaderBody = { + source: scope, + allowedDataProductIds: [productId], + expiresAt: null, + generation: 1, + capabilityDigest: createHash("sha256").update(managedReaderToken, "utf8").digest("hex"), + }; + const managedReaderPath = "/internal/data-plane/v1/reader-bindings/by-key/engine.api-connection.positions-reader"; + const managedReaderResults = await Promise.all([ + rawJsonRequest(managedReaderPath, { method: "PUT", managedSignature: true, body: managedReaderBody }), + rawJsonRequest(managedReaderPath, { method: "PUT", managedSignature: true, body: managedReaderBody }), + ]); + assert.deepEqual(managedReaderResults.map(({ value }) => value.idempotent).sort(), [false, true]); + assert.equal(managedReaderResults[0].value.readerBinding.id, managedReaderResults[1].value.readerBinding.id); + assert.equal("token" in managedReaderResults[0].value, false); + assert.equal(JSON.stringify(managedReaderResults[0].value).includes(managedReaderBody.capabilityDigest), false); + assert.deepEqual( + (await jsonRequest("/internal/data-plane/v1/reader/data-products", { token: managedReaderToken })).dataProducts.map((value) => value.id), + [productId], + ); + const legacyManagedReaderRevoke = await fetch( + `${baseUrl}/internal/data-plane/v1/reader-bindings/${managedReaderResults[0].value.readerBinding.id}/revoke`, + { method: "POST", headers: { Authorization: `Bearer ${provisionerSecret}` } }, + ); + assert.equal(legacyManagedReaderRevoke.status, 404); + const managedReaderRevoke = await jsonRequest( + `${managedReaderPath}/generations/1/revoke`, + { method: "POST", managedSignature: true }, + ); + assert.equal(managedReaderRevoke.idempotent, false); + assert.equal(managedReaderRevoke.readerBinding.active, false); + const managedReaderRevokeRetry = await jsonRequest( + `${managedReaderPath}/generations/1/revoke`, + { method: "POST", managedSignature: true }, + ); + assert.equal(managedReaderRevokeRetry.idempotent, true); + const rejectedManagedReader = await fetch(`${baseUrl}/internal/data-plane/v1/reader/data-products`, { + headers: { Authorization: `Bearer ${managedReaderToken}` }, + }); + assert.equal(rejectedManagedReader.status, 401); + const manualWriterBody = { source: scope, allowedDataProductIds: [productId], expiresAt: expiry }; const signedOnlyManualProvisioning = await signedFetch("/internal/data-plane/v1/writer-bindings", { method: "POST", @@ -245,6 +287,13 @@ try { const snapshot = await jsonRequest(`/internal/data-plane/v1/data-products/${productId}/snapshot`, { token: reader.token }); assert.equal(validateDataProductSnapshot(snapshot).ok, true); assert.equal(snapshot.facts.length, 1); + const history = await jsonRequest( + `/internal/data-plane/v1/data-products/${productId}/history?from=2026-07-15T12%3A00%3A00.000Z&to=2026-07-15T12%3A02%3A00.000Z&resolutionMs=60000&sourceIds=unit-01&limit=100`, + { token: reader.token }, + ); + assert.equal(validateDataProductHistory(history).ok, true); + assert.equal(history.facts.length, 1); + assert.equal(history.facts[0].bucketStart, "2026-07-15T12:00:00.000Z"); const controller = new AbortController(); const response = await fetch(`${baseUrl}/internal/data-plane/v1/data-products/${productId}/stream?after=${snapshot.cursor}`, { diff --git a/services/external-data-plane/test/data-product-delivery.integration.test.mjs b/services/external-data-plane/test/data-product-delivery.integration.test.mjs index 4c5ed5c..efd1f6e 100644 --- a/services/external-data-plane/test/data-product-delivery.integration.test.mjs +++ b/services/external-data-plane/test/data-product-delivery.integration.test.mjs @@ -1,10 +1,11 @@ import assert from "node:assert/strict"; import { Pool } from "pg"; -import { validateDataProductPatch, validateDataProductSnapshot } from "@nodedc/external-provider-contract/data-plane"; +import { validateDataProductHistory, validateDataProductPatch, validateDataProductSnapshot } from "@nodedc/external-provider-contract/data-plane"; import { loadDataProductDefinition, persistDataProductDefinition, persistDataProductPublish, + readDataProductHistory, readDataProductSnapshot, readPatchEvents, } from "../src/data-product-delivery.mjs"; @@ -154,6 +155,47 @@ try { assert.equal(unitOneHistory.some((row) => new Date(row.observed_at).toISOString() === "2026-07-15T10:01:30.000Z"), true); assert.equal(unitOneHistory.some((row) => new Date(row.observed_at).toISOString() === "2026-07-15T10:02:00.000Z"), true); + const historyPageOne = await readDataProductHistory(pool, binding, storedDefinition, { + from: "2026-07-15T10:00:00.000Z", + to: "2026-07-15T10:04:00.000Z", + resolutionMs: 60_000, + sourceIds: ["unit-01"], + limit: 2, + }); + assert.equal(validateDataProductHistory(historyPageOne).ok, true); + assert.equal(historyPageOne.facts.length, 2); + assert.equal(typeof historyPageOne.nextCursor, "string"); + const historyPageTwo = await readDataProductHistory(pool, binding, storedDefinition, { + from: "2026-07-15T10:00:00.000Z", + to: "2026-07-15T10:04:00.000Z", + resolutionMs: 60_000, + sourceIds: ["unit-01"], + limit: 2, + cursor: historyPageOne.nextCursor, + }); + assert.equal(validateDataProductHistory(historyPageTwo).ok, true); + assert.equal(historyPageTwo.facts.length, 1); + assert.equal(historyPageTwo.nextCursor, undefined); + assert.deepEqual( + [...historyPageOne.facts, ...historyPageTwo.facts].map((value) => value.bucketStart), + [ + "2026-07-15T10:00:00.000Z", + "2026-07-15T10:01:00.000Z", + "2026-07-15T10:02:00.000Z", + ], + ); + await assert.rejects( + readDataProductHistory(pool, binding, storedDefinition, { + from: "2026-07-15T10:00:00.000Z", + to: "2026-07-15T10:04:00.000Z", + resolutionMs: 60_000, + sourceIds: ["unit-02"], + limit: 2, + cursor: historyPageOne.nextCursor, + }), + (error) => error?.status === 400 && error?.code === "data_product_history_cursor_invalid", + ); + console.log("external-data-plane delivery integration: ok"); } finally { await pool.end(); diff --git a/services/external-data-plane/test/reader-binding.test.mjs b/services/external-data-plane/test/reader-binding.test.mjs index 973d4b0..675ac49 100644 --- a/services/external-data-plane/test/reader-binding.test.mjs +++ b/services/external-data-plane/test/reader-binding.test.mjs @@ -3,7 +3,9 @@ import { assertReaderProduct, createReaderToken, hashReaderToken, + normalizeManagedReaderBindingRequest, normalizeReaderBindingRequest, + readerBindingRequestHash, safeReaderBinding, } from "../src/reader-binding.mjs"; @@ -22,4 +24,44 @@ const safe = safeReaderBinding({ id: "reader-01", ...policy, active: true, token assert.equal("token" in safe, false); assert.equal("tokenHash" in safe, false); +const managed = normalizeManagedReaderBindingRequest({ + source: { tenantId: policy.tenantId, connectionId: policy.connectionId, providerId: policy.providerId }, + allowedDataProductIds: ["fleet.positions.current.v1"], + expiresAt: null, + generation: 2, + capabilityDigest: hashReaderToken(token), +}, { now }); +assert.equal(managed.generation, 2); +assert.match(readerBindingRequestHash(managed), /^[a-f0-9]{64}$/); +const safeManaged = safeReaderBinding({ + id: "reader-managed-01", + bindingKey: "dprg-reader-managed", + generation: 2, + ...managed, + active: true, +}); +assert.equal(safeManaged.bindingKey, "dprg-reader-managed"); +assert.equal(safeManaged.generation, 2); +assert.equal(safeManaged.expiresAt, null); +assert.equal( + assertReaderProduct(safeManaged, "fleet.positions.current.v1", new Date("2036-07-15T12:00:00.000Z")), + "fleet.positions.current.v1", +); +assert.equal(JSON.stringify(safeManaged).includes(managed.capabilityDigest), false); +assert.throws(() => normalizeManagedReaderBindingRequest({ + source: { tenantId: policy.tenantId, connectionId: policy.connectionId, providerId: policy.providerId }, + allowedDataProductIds: managed.allowedDataProductIds, + expiresAt: managed.expiresAt, + generation: managed.generation, + capabilityDigest: managed.capabilityDigest, + token: "forbidden", +}, { now }), /managed_reader_binding_secret_material_forbidden/); +assert.throws(() => normalizeManagedReaderBindingRequest({ + source: { tenantId: policy.tenantId, connectionId: policy.connectionId, providerId: policy.providerId }, + allowedDataProductIds: managed.allowedDataProductIds, + expiresAt: policy.expiresAt, + generation: managed.generation, + capabilityDigest: managed.capabilityDigest, +}), /managed_reader_binding_must_be_durable/); + console.log("external-data-plane reader bindings: ok"); diff --git a/services/external-data-plane/test/writer-binding.test.mjs b/services/external-data-plane/test/writer-binding.test.mjs index b49b940..25ae144 100644 --- a/services/external-data-plane/test/writer-binding.test.mjs +++ b/services/external-data-plane/test/writer-binding.test.mjs @@ -1,6 +1,7 @@ import assert from "node:assert/strict"; import { validateIntakeBatch } from "../../../packages/external-provider-contract/src/data-plane.mjs"; import { + canMigrateManagedWriterBindingToDurable, createWriterToken, hashWriterToken, materializeDataProductPublish, @@ -38,6 +39,7 @@ assert.notEqual(hashWriterToken(token), hashWriterToken(`${token}x`)); const managedPolicy = normalizeManagedWriterBindingRequest({ ...request, + expiresAt: null, allowedDataProductIds: ["fleet.positions.current.v1", "asset.status.current.v1"], generation: 1, capabilityDigest: hashWriterToken(token), @@ -48,26 +50,47 @@ assert.deepEqual(managedPolicy.allowedDataProductIds, [ ]); assert.equal(managedPolicy.generation, 1); assert.equal(managedPolicy.capabilityDigest, hashWriterToken(token)); +assert.equal(managedPolicy.expiresAt, null); +assert.equal(canMigrateManagedWriterBindingToDurable({ + ...managedPolicy, + expiresAt: "2026-08-01T12:00:00.000Z", + active: true, + capabilityDigest: managedPolicy.capabilityDigest, +}, managedPolicy, now), true); +assert.equal(canMigrateManagedWriterBindingToDurable({ + ...managedPolicy, + expiresAt: "2026-07-01T12:00:00.000Z", + active: true, + capabilityDigest: managedPolicy.capabilityDigest, +}, managedPolicy, now), false); assert.equal(writerBindingRequestHash(managedPolicy), writerBindingRequestHash({ ...managedPolicy, allowedDataProductIds: [...managedPolicy.allowedDataProductIds].reverse(), })); assert.throws(() => normalizeManagedWriterBindingRequest({ ...request, + expiresAt: null, generation: 0, capabilityDigest: hashWriterToken(token), }, { now, maxTtlDays: 90 }), /managed_writer_binding_generation_invalid/); assert.throws(() => normalizeManagedWriterBindingRequest({ ...request, + expiresAt: null, generation: 1, capabilityDigest: "not-a-digest", }, { now, maxTtlDays: 90 }), /managed_writer_binding_capability_digest_invalid/); assert.throws(() => normalizeManagedWriterBindingRequest({ ...request, + expiresAt: null, generation: 1, capabilityDigest: hashWriterToken(token), token, }, { now, maxTtlDays: 90 }), /managed_writer_binding_secret_material_forbidden/); +assert.throws(() => normalizeManagedWriterBindingRequest({ + ...request, + generation: 1, + capabilityDigest: hashWriterToken(token), +}), /managed_writer_binding_must_be_durable/); const unscopedBatch = { schemaVersion: "nodedc.external-provider-contract/v1", @@ -148,5 +171,21 @@ const safeBinding = safeWriterBinding({ }); assert.equal("token" in safeBinding, false); assert.equal("tokenHash" in safeBinding, false); +const safeManagedBinding = safeWriterBinding({ + id: "binding-managed-01", + ...managedPolicy, + active: true, + createdAt: now, +}); +assert.equal(safeManagedBinding.expiresAt, null); +assert.equal(materializeDataProductPublish({ + schemaVersion: "nodedc.data-product.publish/v1", + batch: { runId: "run-durable", sequence: 0, idempotencyKey: "run-durable.batch-0" }, + facts: unscopedBatch.facts, +}, safeManagedBinding, { + id: "fleet.positions.current.v1", + version: "1.0.0", + ontologyRevision: "ontology.example-fleet.v1", +}, "fleet.positions.current.v1", { now: new Date("2036-07-15T12:00:00.000Z") }).batch.runId, "run-durable"); console.log("external-data-plane writer bindings: ok");