SILVER
/
NODEDC_PLATFORM
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Compare commits

base: SILVER:31e078d6e52ce01b415ad3d41b5d29959020d295
Branches Tags
SILVER:main
SILVER:ai-workspaces-npm
SILVER:CROSSASSISTANT
..
compare: SILVER:567f1550abc2117db0636b0166ff9a25ed0c1088
Branches Tags
SILVER:main
SILVER:ai-workspaces-npm
SILVER:CROSSASSISTANT

No commits in common. "31e078d6e52ce01b415ad3d41b5d29959020d295" and "567f1550abc2117db0636b0166ff9a25ed0c1088" have entirely different histories.
31e078d6e5 ... 567f1550ab

71 changed files with 1690 additions and 9596 deletions
Show Stats Expand all files Collapse all files

2
README.md
View File

@ -49,4 +49,4 @@ Ontology Core живёт в `services/ontology-core` как docs-first сема
Map Gateway живёт в `services/map-gateway` как общий platform boundary для provider credentials, безопасного asset endpoint exchange, tile/3D Tiles proxy и persistent offline TileCache. Runtime cache не является source artifact и хранится в named volume/object storage, а не в Git. Map Gateway живёт в `services/map-gateway` как общий platform boundary для provider credentials, безопасного asset endpoint exchange, tile/3D Tiles proxy и persistent offline TileCache. Runtime cache не является source artifact и хранится в named volume/object storage, а не в Git.
Внешние бизнес-поставщики подключаются по `packages/external-provider-contract`: adapter конкретного API принадлежит изолированному NDC L2 workflow, а Platform даёт один provider-neutral External Data Plane для raw retention, canonical facts, current/history projections и scoped read products. Connection instance несёт tenant scope, credential reference и collection profile, но не secret value. Provider/domain mapping, entity filtering и renderer logic не попадают в Platform Data Plane. Provider-issued credentials сохраняются в native NDC L2 Credentials; внутренний scoped publish grant генерируется и сохраняется generic control plane внутри native credential boundary, EDP получает только digest, а MCP видит только opaque reference/status. Legacy plaintext provisioning и managed digest-only ensure включаются независимо и по умолчанию закрыты. Подробный канон — `docs/ADR_L2_OWNED_EXTERNAL_CONNECTORS.md`. Внешние бизнес-поставщики подключаются по `packages/external-provider-contract`: adapter конкретного API принадлежит изолированному NDC Agent L2 workflow, а Platform даёт один provider-neutral External Data Plane для raw retention, canonical facts, current/history projections и scoped read products. Connection instance несёт tenant scope, credential reference и collection profile, но не secret value. Provider/domain mapping, entity filtering и renderer logic не попадают в Platform Data Plane; writer capability привязана к immutable EDP binding и выдаётся только отдельному Engine provisioner, не shared internal bearer. Подробный канон — `docs/ADR_L2_OWNED_EXTERNAL_CONNECTORS.md`.

404
docs/ADR_L2_OWNED_EXTERNAL_CONNECTORS.md
View File

@ -1,238 +1,274 @@
# ADR: внешние коннекторы принадлежат NDC L2 workflow # ADR: L2-owned external connectors and provider-neutral Data Plane
Статус: **accepted**. Статус: **accepted**.
Дата: 2026-07-16. Дата: 2026-07-14.
Владелец решения: NODE.DC Platform. Владелец решения: NODE.DC Platform.
## Контекст ## Контекст
NODE.DC состоит из двух разных слоёв: NODE.DC — платформа разработки. Подключение внешнего API не должно создавать
ещё один provider-specific сервис, в котором навсегда зашиты логика клиента,
нормализация и правила отображения. Иначе каждый новый account или новый тип
объекта превращается в изменение Platform runtime.
- платформенный слой даёт NDC L1, NDC L2, Ontology, credentials, Data Products Нужна одна повторяемая форма: Platform даёт безопасные границы, контракт
и Foundry-boundaries; хранения и выдачи данных; изолированный workflow NDC Agent L2 является
- пользовательские автоматизации собирают из этих возможностей конкретную адаптером конкретного API и остаётся редактируемым через предоставленный
бизнес-логику. Codex/Engine MCP.
Новый account, provider credential, tenant, расписание или интерфейс не должны создавать Это решение заменяет provider-adapter часть
новый platform service и не должны требовать изменения NODE.DC source. Команда `ADR_EXTERNAL_PROVIDER_DATA_PLANE.md` и все provider-specific runtime
NODE.DC подключается только для нового поставщика либо для расширения предписания `ADR_GELIOS_DATA_PLANE.md`. Они сохраняются как исторические
подтверждённых capabilities и ontology уже поддержанного поставщика. черновики, но не являются каноном для новой реализации.
Gelios — первый живой источник и acceptance-кейс этого канона, а не отдельная
архитектурная ветка. Документ заменяет provider-runtime решения из
`ADR_EXTERNAL_PROVIDER_DATA_PLANE.md` и `ADR_GELIOS_DATA_PLANE.md`; те
документы остаются только историей решения.
## Решение ## Решение
### 1. Пакет поставщика создаётся один раз ### 1. Адаптер внешнего API принадлежит L2
Каждый поддержанный поставщик получает один версионируемый **provider package**. Один изолированный L2 workflow представляет одно connection instance и
Он является знанием платформы о внешнем API и содержит: является единственным местом для:
- стабильный `providerId` и версии подтверждённого API; - обращения к API поставщика через ссылку на credential из Engine;
- credential contract без значения секрета; - выбора read-capability, cursor/pagination, rate limit, retry и расписания;
- каталог safe-read capabilities, pagination/rate-limit metadata и response - получения raw envelope, разбиения большого ответа на native batch-операции;
shapes; - семантического mapping к версии ontology и формирования canonical facts;
- mappings из provider fields в версии Ontology; - idempotency key, watermark и lifecycle источника;
- совместимые Data Products и scrubbed contract fixtures; - deploy/run/execution/trace через Engine MCP.
- явно отделённый каталог команд без активного command transport.
Пакет не является runtime service, scheduler, базой данных или customer L2 не хранит значение provider secret или writer token: он использует только
configuration. Он расширяется только когда фактически подтверждён новый API, Engine-bound credential references. Plaintext writer token выдаётся trusted
новый тип данных или новая ontology revision. provisioner ровно один раз при create/rotate binding и затем хранится только в
Engine credential store, привязанном к этому L2; External Data Plane хранит
лишь его hash. Provisioner читает отдельный runner-owned secret file, который
read-only монтируется только в EDP и позднее в dedicated Engine provisioner,
работающий под выделенным UID/GID `11006`; это не shared `.env`, не
`NODEDC_INTERNAL_ACCESS_TOKEN` и не provider credential. До deployment этого
provisioner `EXTERNAL_DATA_PLANE_PROVISIONING_ENABLED=false`, поэтому issuance
routes fail closed. Token не попадает в L2 graph/profile, contract artifact,
execution log/trace, raw envelope, Foundry или UI. Command/write-capabilities
не получают transport route в read workflow.
Одна учётная запись поставщика задаётся данными, а не кодом: Для новой учётной записи создаётся новый instance/profile этого workflow с
другой credential reference и connection configuration. Это не требует
изменения Platform source. Если поставщик раскрывает новый объект или поле,
сначала расширяется ontology/capability contract, затем изменяется именно
соответствующий L2 workflow.
`provider package + credential reference + connection profile + NDC L2 workflow instance` ### 2. Platform Data Plane нейтрален к provider и домену
Поэтому второй account или другая provider credential pair создаёт ещё один credential/profile и Platform предоставляет один versioned **External Data Plane**. Он принимает
workflow instance. Платформенный source, Data Plane и Foundry при этом не batch по внутреннему аутентифицированному контракту, хранит raw envelope с
меняются. retention/provenance, canonical facts, current/history projections и отдаёт
scoped data products/realtime updates. В нём запрещены:
### 2. NDC L1 проектирует, NDC L2 workflow исполняет - ветвления по названию provider, customer, account, vehicle или renderer;
- provider field mapping, unit allowlist, business filtering и visual rules;
- provider token, прямой browser-to-provider доступ и command transport.
NDC L1 получает пользовательское намерение, проверяет доступные capabilities и Его canonical write-форма, которую Data Plane валидирует и сохраняет после
Ontology, проектирует или изменяет разрешённый NDC L2 workflow через NDC MCP и авторизации, содержит только нейтральные элементы:
анализирует execution evidence.
Один изолированный NDC L2 workflow представляет один connection instance и ```text
владеет исполняемой механикой: source: providerId + tenantId + connectionId
contract: contractVersion + ontologyRevision + dataProductId
batch: runId + sequence + idempotencyKey + receivedAt
raw: optional restricted ref + contentType + hash + retention policy
facts: stable sourceId + semanticType + observedAt + attributes + geometry?
```
- safe-read вызовами provider API через credential reference; Для нового L2 writer-а это не wire-форма. Он вызывает
- pagination, cursor, retry, rate limit, batch size и collection cadence; `POST /internal/data-plane/v1/data-products/:dataProductId/publish` с bearer
- проверкой response shape и разбиением ответа на items; writer token и формой `nodedc.data-product.publish/v1`, в которой вообще нет
- mapping к точной ontology revision; `source`, contract metadata, transport или persistence policy. Любой caller
- idempotency, watermark и публикацией canonical facts; scope и scope headers запрещены.
- private workflow state, когда он действительно нужен.
Provider-specific mapping сначала реализуется штатными workflow nodes и малым External Data Plane разрешает token в собственный immutable writer binding
Code-преобразователем. Это позволяет проверить реальный API без преждевременного `(tenantId, connectionId, providerId, allowedDataProductIds, expiresAt)`,
создания custom nodes. В custom NDC nodes переносится только повторившаяся и проверяет active/non-expired binding, совпадение `providerId` и разрешение
доказанная **платформенная boundary-механика**, а не уникальная логика `dataProductId`, затем сам materializes canonical `source` с tenant/connection
поставщика. и только после этого валидирует и сохраняет batch. Присланный caller scope
отклоняется, а не merge/override-ится. Изменение tenant, connection, provider,
allowed data products или срока требует нового binding; для существующего
binding допустимы только token rotation и revoke.
### 3. Credentials являются данными connection instance Выпуск, rotation и revoke binding принимаются только от dedicated provisioning
principal с runner-owned secret file; он не монтируется в shared `.env` или L2.
Shared legacy bearer не может создавать writer capability.
Gelios выдаёт ровно два provider secrets: access token и refresh token. Текущий Идентификатор источника стабилен в пределах `(providerId, connectionId,
HTTP request binding `httpBearerAuth` использует access token. Автоматический sourceId)`. Отсутствие объекта в очередном ответе помечается lifecycle-state
refresh в принятом runtime не доказан, поэтому provider package фиксирует его или временем последнего наблюдения; запись и её история не удаляются.
как `operator_managed`; ни один из token values не попадает в graph, package,
Ontology, Data Plane, Foundry, execution logs, MCP или Ops. Новый account или
provider-issued token pair означает новые native credential records и
connection instance, но не новую Platform-сущность.
Локальное имя credential (например, с суффиксом `read access`) не является ### 3. Граница данных и интерфейса
provider scope. В target-контракте Read-классификацию задаёт allowlisted
method/endpoint в capability catalog и Engine workflow policy; тот же access
token нельзя называть отдельным «read token» или «write token» только из-за
label. Сейчас deployed Engine safe-ref policy ограничивает generic HTTP
credential только по host, но ещё не связывает его с package/version и exact
method/path. Поэтому Gelios `GET /api/v1/units` пока является декларативной
capability, а не завершённой runtime-security гарантией; canonical acceptance
требует capability-bound Engine policy и её MCP proof.
Writer capability для публикации Data Product — отдельный внутренний native NDC ```text
L2 credential, не третий Gelios token. Generic Engine/Platform control plane Provider API (safe read)
генерирует capability внутри native credential boundary, сохраняет её там же и -> Engine provider-credential reference
передаёт EDP только SHA-256 digest вместе с точным provider, connection и -> L2 connector instance (adapter + mapping + batches)
набором Data Products. Пользователь и MCP получают только opaque -> Engine writer-credential reference
reference/status. Capability не записывается в graph, provider package, env, -> External Data Plane Data Product publish (scope materialized server-side)
file, Ops или trace и не копируется через пользовательский UI. -> External Data Plane (generic persistence + projections)
-> scoped data product / realtime stream
-> Foundry binding / user interface
```
Широкий credential sink, resolver/daemon с произвольной записью secret и любой Ontology Core описывает semantic types, связи, capability catalog и mapping
provider-specific credential service запрещены. Нужна одна узкая операция versions, но не хранит runtime payloads или секреты. Foundry получает только
`ensure data-product publish grant`: scope выводится из granted L1→L2 target, approved data product и решает presentation: pins, visibility, layers and
зарегистрированного connection profile и разрешённого Data Product; issuance, filters. Оно не получает provider endpoint или credential.
rotation и native binding должны быть idempotent, CAS/crash-safe и auditable.
Engine генерирует capability внутри credential boundary и передаёт EDP только
SHA-256 digest через idempotent binding key + generation; EDP не возвращает
plaintext. Новая generation создаётся до переключения node reference, а старая
отзывается только после успешного bind/acceptance.
Старые ручные EDP `POST/rotate` endpoints, которые возвращают capability один
раз, включаются отдельным legacy-флагом и не входят в canonical acceptance.
Digest-only managed endpoint имеет независимый флаг и принимает только
Ed25519-signed Engine service requests: exact audience/method/request target/raw
body hash входят в подпись, timestamp ограничен по skew, nonce защищён bounded
fail-closed replay cache. Legacy provisioner bearer на managed routes не
действует. EDP получает только deployment public key; matching private key
остаётся только внутри Engine server boundary. Endpoint остаётся выключенным,
пока key provisioning, Engine signer и exact native credential binding policy
не пройдут runtime acceptance.
В deployed Engine MCP этой операции пока нет — это текущий platform gap перед
canonical publish proof, а не действие пользователя. Это решение не заявляет
автоматический refresh Gelios: до отдельного runtime proof он остаётся
`operator_managed`.
### 4. External Data Plane нейтрален к поставщику ### 4. Полнота без entity allowlist
Platform предоставляет один versioned External Data Plane. Он принимает Collection profile выбирает разрешённые **read-capabilities**, а не список
canonical facts через scoped writer binding, хранит current/history конкретных объектов. Если выбранный read endpoint возвращает все доступные
projections и публикует snapshot/patch contracts. В нём запрещены: источнику сущности, L2 передаёт все валидные элементы. Новые сущности
добавляются автоматически; скрытие/отображение — задача data product/UI, а не
сбора. Технические лимиты существуют только как размер batch, backpressure,
quota и защита от повреждённого ответа, но не как бизнес-фильтр по ID.
- ветвления по provider, customer, account, entity или renderer; ### 5. Масштабирование и native NDC Agent nodes
- provider field mapping и бизнес-фильтрация;
- provider credential, endpoint, schedule или command transport;
- caller-supplied tenant/connection scope.
`NDC Data Product Publish` отправляет только L2 не запускает отдельный execution на каждую сущность. Он использует native
`nodedc.data-product.publish/v1`. External Data Plane materializes immutable nodes NDC Agent: HTTP Request, Split Out, Split In Batches/Loop Over Items,
scope из writer binding, проверяет разрешённый Data Product и его ontology Aggregate, Postgres (когда нужен private workflow state), Set/If/Merge,
revision, затем сохраняет batch. Schedule Trigger and Respond to Webhook. Code node допустим только как малый
преобразователь boundary-shape, когда эквивалентной native node нет; он не
становится скрытым сервисом или provider database.
EDP runtime импортирует только package subpath Shared telemetry/history не записывается L2 напрямую в физические таблицы
`@nodedc/external-provider-contract/data-plane`. Его deploy artifact и image Platform Postgres: это создало бы coupling к schema. Direct Postgres допустим
содержат только provider-neutral wire validators; `providers/gelios`, mappings, для private state конкретного workflow или отдельной project-owned базы.
fixtures и tests туда не входят. Изменение или добавление provider package не Общий контур использует только External Data Plane contract и bulk batches.
пересобирает и не перезапускает EDP: каталог поставщиков разворачивается через
Engine/Ontology/control-plane путь отдельно.
Collection cadence, history cadence и presentation cadence независимы: ### 5.1. Каталог custom nodes NODE.DC
- NDC L2 забирает источник с частотой, нужной бизнес-задаче; Встроенные узлы runtime сохраняют свои штатные названия (`HTTP Request`,
- current projection принимает каждое валидное изменение; `Schedule Trigger`, `Loop Over Items` и т. п.). Любой узел, код которого
- declarative history policy может хранить все точки или sampling; принадлежит NODE.DC, обязан одновременно выполнять три условия:
- Foundry читает snapshot+patch и отдельно ограничивает частоту render.
Для разной частоты БД и интерфейса не создаются второй provider sink, отдельный - видимое имя начинается с `NDC `;
gateway или параллельный прямой push в renderer. - runtime type принадлежит package namespace `n8n-nodes-ndc.*`;
- package-level test отклоняет публикацию узла без этого префикса и namespace.
### 5. Полнота означает все сущности разрешённой capability Первый канонический набор состоит из:
Connection profile выбирает safe-read capabilities, а не зашитый в Platform - `NDC Data Product Publish` — runtime write boundary L2 → Data Plane;
список entity IDs. Если разрешённый endpoint возвращает все доступные credential - `NDC Data Product Read` — scoped current snapshot read по reader grant;
сущности, NDC L2 обрабатывает каждый валидный item. Новый объект появляется - `NDC Foundry Binding` — deploy/control-plane связь Data Product с
автоматически. `Application → Page → typed slot`, но не транспорт каждого realtime tick.
Пользовательская фильтрация, слои и видимость находятся после сбора — в `NDC Foundry Output` не используется как техническое имя: оно ошибочно
automation/Data Product/Foundry. Технические ограничения допустимы только как подразумевает прямой transport L2 → renderer. Provider-specific adapter,
pagination, quota, batch size, backpressure и защита от повреждённого ответа. mapping и collection profile остаются логикой конкретного L2 workflow на
штатных nodes; они не оформляются как custom NODE.DC nodes и не добавляют
provider branch в Data Plane или Foundry.
### 6. Custom NDC nodes ограничены платформенными границами `NDC Foundry Binding` отправляет только
`nodedc.foundry.binding-upsert/v1`. Её постоянный L2 credential — отдельный
revocable `ndc_fndbg_*` workload grant с allowlist точных
`Application → Page → Binding → Slot → Data Product` targets. Короткоживущая
`fnd1.*` capability интерактивного Foundry MCP, browser session, EDP
writer/reader grant и `NODEDC_INTERNAL_ACCESS_TOKEN` для этой ноды запрещены.
Первый канонический набор: Runtime node не содержит URL, endpoint, provider ID, tenant ID, connection ID,
token, contract version или history cadence. Узел выбирает только разрешённый
Data Product; всё остальное materializes из opaque writer/reader binding и
immutable Data Product definition.
- `NDC Data Product Publish` — NDC L2 → External Data Plane; Package `n8n-nodes-ndc` устанавливается как штатный private community package в
- `NDC Data Product Read` — scoped snapshot/patch read; `/home/node/.n8n/nodes/node_modules`, чтобы n8n `PackageDirectoryLoader`
- `NDC Foundry Binding` — control-plane связь Data Product с сохранил namespace `n8n-nodes-ndc.*`. `~/.n8n/custom` и
`Application → Page → typed slot`. `N8N_CUSTOM_EXTENSIONS` запрещены для этого пакета: они создают namespace
`CUSTOM.*`. Ручное копирование или `npm install` в живом container, patch
Engine core и подмена built-in node запрещены. Production использует
проверенный offline tarball, immutable release, atomic current/previous switch,
read-only mount и restart всех n8n execution processes; acceptance завершается
только когда Engine schema и MCP catalog возвращают точные package-qualified
types.
Эти nodes не содержат provider ID, tenant ID, connection ID, endpoint, token, ### 5.2. Realtime delivery contract
mapping или cadence. Runtime package сохраняет технический namespace
`n8n-nodes-ndc.*`, но пользовательские документы и интерфейсы используют
только терминологию NDC.
`NDC Foundry Binding` не транспортирует каждый realtime tick. Он создаёт Новый writer использует:
постоянную связь интерфейса с Data Product; Foundry затем читает его
snapshot+patch contract.
### 7. Реальный Gelios acceptance-кейс ```text
POST /internal/data-plane/v1/data-products/:dataProductId/publish
Authorization: Bearer <opaque writer binding>
```
Первая версия provider package должна доказать путь: Body имеет schema `nodedc.data-product.publish/v1` и содержит только batch
identity и canonical facts. Data Plane одной транзакцией:
`Gelios safe read → все доступные units → map.moving_object facts → 1. проверяет idempotency;
fleet.positions.current.v1 → Foundry map` 2. обновляет current projection только более новым или изменившимся фактом;
3. применяет declarative history policy (`none`, `all`, `sampled`);
4. добавляет changed facts в durable patch outbox;
5. фиксирует монотонный cursor.
Canonical product использует ontology revision Reader grant получает snapshot `nodedc.data-product.snapshot/v1`, затем
`ontology.map.moving_object.v1` и только объявленные snake_case поля. NDC L2 подключается к SSE stream с `after=<snapshot.cursor>` или `Last-Event-ID`.
не передаёт caller scope и не собирает собственный batch envelope вокруг Каждый outbox event имеет schema `nodedc.data-product.patch/v1`. Если cursor
`NDC Data Product Publish`. уже удалён retention policy, stream отвечает `409 resync_required`, и consumer
повторяет snapshot. Browser не получает Data Plane token или scope: Foundry BFF
разрешает persisted binding и читает server-only reader grant.
Acceptance выполняется по порядку: `snapshot+patch` в v1 является **bounded product contract**. Current projection
одного scoped Data Product не может содержать больше 5000 entity keys
`(sourceId, semanticType)`. Snapshot возвращает целую согласованную projection
и один barrier cursor из одной `REPEATABLE READ` transaction. Query-параметр
`limit` — только защитный потолок, а не размер страницы: если полная projection
не помещается, Data Plane отвечает `413 data_product_snapshot_limit_exceeded`
и consumer не начинает patch stream с неполной базой.
1. Platform выполняет generic `ensure data-product publish grant`, выпускает Наивная pagination current snapshot по `sourceId`, offset или независимо
scoped EDP binding и атомарно сохраняет capability в native NDC L2 полученным page cursors запрещена: изменения между страницами могут быть
Credentials; пропущены или продублированы относительно patch cursor. Product с ожидаемой
2. NDC MCP видит только совместимый opaque writer credential reference/status; cardinality выше 5000 обязан **до включения realtime** выбрать один из двух
3. применить подтверждённый graph patch без legacy intake; отдельно версионируемых контрактов:
4. validate/preflight;
5. один успешный manual run и проверка execution/trace/Data Product;
6. только затем отдельным изменением добавить Schedule Trigger;
7. после доказанного snapshot+patch подключить Foundry binding.
### 8. Capacity и topology - stable partitioning в несколько Data Products, где partition key является
частью definition, а каждый partition имеет собственный полный snapshot и
независимый patch cursor;
- новый query delivery contract с явно определёнными snapshot barrier,
continuation cursor, query scope и правилами перехода к patch stream.
Количество одновременно активных NDC L2 проектов ограничивается фактическими Такой query contract не входит в `nodedc.data-product.snapshot/v1` и не может
ресурсами железа и профилем нагрузки. Пока capacity проверяется оператором и не быть имитирован полем `nextPageCursor`. До его отдельного утверждения runtime и
автоматизируется. Канон не объявляет отдельные worker/webhook generations или `NDC Data Product Read` работают только с bounded products до 5000 сущностей.
high-load topology, которых ещё нет в принятом runtime.
### 6. Порядок первой реализации
1. Версионировать нейтральный intake/read contract и поднять External Data
Plane без provider-specific logic.
2. Пересобрать существующий L2 proof в native-node pipeline: safe read →
split/batch → semantic mapping → generic append → safe summary. Никаких
provider ID filters и provider gateway endpoint.
3. Выполнить manual run, проверить execution/trace, idempotency and current
projection; только затем включить Schedule Trigger.
4. Provision immutable writer binding, one-time place its token into an opaque
Engine credential, then switch the L2 to
`/data-products/:dataProductId/publish` and remove caller scope/headers and
the shared legacy credential.
5. Подключить Foundry к data product current positions. История, геозоны и
новые сущности добавляются отдельными L2 collection profiles and ontology
revisions.
## Последствия ## Последствия
- `services/<provider>-gateway` не является частью канона и не создаётся для - `services/<provider>-gateway` не является шаблоном для новых интеграций.
новых provider integrations. Существующий self-hosted Gelios Gateway и его Существующий experimental code не расширяется и не деплоится как часть этого
Timescale volume остаются frozen compatibility contour, воспроизводятся из решения.
source/deploy и не изменяются новым L2 pilot без отдельного решения. - Data Plane может иметь собственную БД/Timescale/PostGIS implementation, но
- Новый account или provider-issued access/refresh pair — configuration change, физическая схема остаётся внутренней деталью neutral service, а не API для
а не platform release. L2 или Foundry.
- Новый provider или неподдержанная capability — versioned provider/Ontology - Inline raw запрещён в L2 → Data Plane v1: до отдельного raw-vault допускается
change с contract tests. только restricted ref/hash либо отсутствие raw envelope. Retention reference
- Существующий legacy L1/SDK workflow является frozen compatibility boundary: вычисляется по серверному acceptance time, а не по timestamp из batch; Data
новый L2 pilot его не редактирует, не отзывает его credential и не ставит его Plane выполняет bounded retention sweeps после готовности сервиса и по таймеру.
вывод условием acceptance. - Обычный Codex Desktop получает ровно те L2 grants, которые выданы владельцем
- Legacy `/internal/data-plane/v1/intake` остаётся выключенным migration-only connection/workflow, и может безопасно развивать adapter только в этой
route и не используется новыми NDC L2 workflows. границе.
- Команды устройствам остаются отдельным red-domain контуром с явным - `POST /internal/data-plane/v1/intake` остаётся временным migration-only route:
подтверждением, scope, idempotency и аудитом. он выключен по умолчанию (`EXTERNAL_DATA_PLANE_LEGACY_INTAKE_ENABLED=false`),
- Foundry развивается после доказанного Data Product path; provider API и требует `NODEDC_INTERNAL_ACCESS_TOKEN`, canonical scoped batch и совпадающие
credentials в Foundry не попадают. scope headers. Новый или migrated L2 не получает этот shared token и не
использует fallback в legacy route. После миграции всех writers он удаляется.

16
docs/AI_WORKSPACE_ONTOLOGY_MCP.md
View File

@ -28,7 +28,7 @@ The worker receives only a pairing-bound Hub URL. The Hub verifies that the pair
It intentionally does not expose filesystem paths, evidence ledgers, credentials, raw payloads, live telemetry, databases, command dispatch, workflow mutation or Studio controls. It intentionally does not expose filesystem paths, evidence ledgers, credentials, raw payloads, live telemetry, databases, command dispatch, workflow mutation or Studio controls.
## Boundary for external providers and runtime data ## Boundary for Gelios and future data services
Ontology Core describes the canonical meanings and constraints: Ontology Core describes the canonical meanings and constraints:
@ -36,21 +36,15 @@ Ontology Core describes the canonical meanings and constraints:
gelios.unit -> gelios.telemetry_snapshot -> gelios.position_fix -> map.moving_object gelios.unit -> gelios.telemetry_snapshot -> gelios.position_fix -> map.moving_object
``` ```
It does **not** serve provider runtime data or call a provider API. Provider It does **not** serve the Gelios database or provider API. The future Gelios Gateway/Data API is a distinct capability with its own scope checks, data contract and transport. That capability may later be supplied to an AI Workspace run as another dynamic MCP server. Ontology then gives the assistant the names, relations, guardrails and allowed data-contract route; the data capability enforces access and returns live data.
access and mapping belong to the granted NDC L2 workflow; shared current and
history products belong to the provider-neutral External Data Plane. A scoped
Data Product read capability may later be supplied to an AI Workspace run as a
separate dynamic MCP server. Ontology supplies meanings, relations, guardrails
and the allowed contract route; the data capability enforces access and returns
live data.
This preserves one-way responsibility: This preserves one-way responsibility:
- Ontology Core: semantics, contracts, aliases, guardrails and context advice. - Ontology Core: semantics, contracts, aliases, guardrails and context advice.
- External Data Plane: access-scoped Data Product snapshots and updates. - Gelios Gateway/Data API: access-scoped telemetry and history reads.
- Command Gateway: separate red-domain command route, explicit confirmation and audit. - Command Gateway: separate red-domain command route, explicit confirmation and audit.
- NDC L2 workflow: provider access, collection and mapping using granted capabilities. - Engine L2: workflow orchestration using granted capabilities.
- Foundry: presentation consumer, outside this implementation. - Studio: later presentation consumer, outside this implementation.
## Live catalog updates without AI Workspace rule redeploy ## Live catalog updates without AI Workspace rule redeploy

26
infra/.env.example
View File

@ -75,20 +75,10 @@ EXTERNAL_DATA_PLANE_WRITER_BINDING_MAX_TTL_DAYS=90
EXTERNAL_DATA_PLANE_MAX_FUTURE_SKEW_SECONDS=300 EXTERNAL_DATA_PLANE_MAX_FUTURE_SKEW_SECONDS=300
EXTERNAL_DATA_PLANE_RETENTION_SWEEP_MS=3600000 EXTERNAL_DATA_PLANE_RETENTION_SWEEP_MS=3600000
EXTERNAL_DATA_PLANE_LEGACY_INTAKE_ENABLED=false EXTERNAL_DATA_PLANE_LEGACY_INTAKE_ENABLED=false
# Internal control-plane writer/reader binding issuance; keep false until the # The writer-provisioner secret is not an env value. The root-owned deploy
# atomic NDC L2 ensure-grant operation is deployed. Legacy path returns a # runner keeps it in /volume1/docker/nodedc-platform/secrets/external-data-plane-provisioner/
# plaintext capability and must never be exposed to users. # and mounts it
EXTERNAL_DATA_PLANE_PROVISIONING_ENABLED=false # only into External Data Plane and the future dedicated Engine provisioner.
# Digest-only managed writer-binding ensure. It accepts only signed Engine
# service requests; the legacy provisioner bearer is deliberately invalid here.
# Keep false until the matching Engine private key is provisioned. The trust
# directory is mounted read-only into EDP and contains only `public-key.pem`.
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONING_ENABLED=false
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_SERVICE_ID=nodedc-engine
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_KEY_ID=engine-edp-managed-provisioner-v1
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_AUDIENCE=nodedc-external-data-plane.managed-provisioning.v1
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_MAX_SKEW_SECONDS=60
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_REPLAY_CACHE_MAX_ENTRIES=10000
# notification core # notification core
NOTIFICATION_PG_DB=nodedc_notifications NOTIFICATION_PG_DB=nodedc_notifications
@ -129,10 +119,8 @@ AI_WORKSPACE_HUB_FALLBACK_URLS=
AI_WORKSPACE_ONTOLOGY_MCP_PUBLIC_URL= AI_WORKSPACE_ONTOLOGY_MCP_PUBLIC_URL=
ONTOLOGY_CORE_HOST_BIND=127.0.0.1:18104 ONTOLOGY_CORE_HOST_BIND=127.0.0.1:18104
# Gelios Gateway — frozen legacy storage/read compatibility service. Provider # Gelios Gateway — storage/read service. Provider credentials belong to the
# credentials belong to the protected Engine Collector and are never configured # protected Engine Collector and are never configured in this service.
# in this service. Keep this contour reproducible; do not use it as a template
# for new providers.
GELIOS_TIMESCALE_IMAGE=timescale/timescaledb-ha:pg16.14-ts2.28.2-all GELIOS_TIMESCALE_IMAGE=timescale/timescaledb-ha:pg16.14-ts2.28.2-all
GELIOS_PG_DB=nodedc_gelios GELIOS_PG_DB=nodedc_gelios
GELIOS_PG_USER=nodedc_gelios GELIOS_PG_USER=nodedc_gelios
@ -140,6 +128,8 @@ GELIOS_PG_PASS=change-me-generate-with-infra-scripts-init-dev-env
# URL-encode reserved characters in GELIOS_PG_PASS when forming this URL. # URL-encode reserved characters in GELIOS_PG_PASS when forming this URL.
GELIOS_DATABASE_URL=postgresql://nodedc_gelios:change-me-generate-with-infra-scripts-init-dev-env@gelios-postgres:5432/nodedc_gelios GELIOS_DATABASE_URL=postgresql://nodedc_gelios:change-me-generate-with-infra-scripts-init-dev-env@gelios-postgres:5432/nodedc_gelios
GELIOS_GATEWAY_HOST_BIND=127.0.0.1:18105 GELIOS_GATEWAY_HOST_BIND=127.0.0.1:18105
# Explicit tenant and connection identifiers are deployment configuration;
# do not encode a customer or pilot name in source defaults.
GELIOS_TENANT_ID=replace-with-tenant-id GELIOS_TENANT_ID=replace-with-tenant-id
GELIOS_CONNECTION_ID=gelios-connection-id GELIOS_CONNECTION_ID=gelios-connection-id
# `allowlist` accepts only GELIOS_ALLOWED_UNIT_IDS. `all` accepts every unit # `allowlist` accepts only GELIOS_ALLOWED_UNIT_IDS. `all` accepts every unit

135
infra/deploy-runner/README.md
View File

@ -97,15 +97,8 @@ Engine-owned activator may select it after exact MCP schema acceptance.
The paired Engine activation is built by The paired Engine activation is built by
`build-engine-n8n-private-extension-artifact.mjs`. It deliberately does not `build-engine-n8n-private-extension-artifact.mjs`. It deliberately does not
copy from or write generated files into the dirty Engine worktree, and it does copy the dirty Engine `docker-compose.yml` and does not build an image. Instead
not build an image. A fresh transition id is mandatory and previously issued it installs a narrowly scoped Compose override plus a strict transition
ids are rejected:
```bash
node infra/deploy-runner/build-engine-n8n-private-extension-artifact.mjs 20260717-004
```
The builder emits a narrowly scoped Compose override plus a strict transition
descriptor. On apply, the runner validates the staged release again, verifies descriptor. On apply, the runner validates the staged release again, verifies
that the running n8n container and the NAS-local `2.3.2` tag resolve to the that the running n8n container and the NAS-local `2.3.2` tag resolve to the
same immutable image ID, and extracts the package into the root-owned, same immutable image ID, and extracts the package into the root-owned,
@ -117,15 +110,10 @@ read-only Engine release tree:
The override sets `N8N_USER_FOLDER=/home/node`, which is required because the The override sets `N8N_USER_FOLDER=/home/node`, which is required because the
actual Engine service runs as root while the canonical community package path actual Engine service runs as root while the canonical community package path
is below `/home/node/.n8n`. The live runtime contract remains the successfully is below `/home/node/.n8n`. It enables loading but disables reinstall, mounts
deployed generation-003 contract and does not set `NODE_PATH`. Only the only the exact release read-only, and uses both Compose `pull_policy: never`
runner's isolated `node -e` package-loader probe temporarily initializes the and `docker compose up --pull never`. No registry access, lifecycle script,
dependency tree bundled inside the exact n8n base image; this reproduces n8n's database `installed_packages` row or custom-extension loader is involved.
own loader without redefining the live Compose state. The override enables
loading but disables reinstall, mounts only the exact release read-only, and
uses both Compose `pull_policy: never` and `docker compose up --pull never`.
No registry access, lifecycle script, database `installed_packages` row or
custom-extension loader is involved.
The runner pins the exact Engine service topology observed in source and The runner pins the exact Engine service topology observed in source and
rejects an added worker/webhook generation. Only the single actual `n8n` rejects an added worker/webhook generation. Only the single actual `n8n`
@ -152,106 +140,20 @@ PYTHONDONTWRITEBYTECODE=1 \
``` ```
For `platform` artifacts, the allowlist includes the versioned Ontology Core, For `platform` artifacts, the allowlist includes the versioned Ontology Core,
the frozen legacy Gelios compatibility service, and the provider-neutral the legacy Gelios experiment and the provider-neutral External Data Plane
External Data Plane sources. The Gelios service remains reproducible only to sources. An External Data Plane artifact builds only its image and starts
protect its existing database/workflow; it is not a template for a provider `external-data-plane-postgres` plus `external-data-plane`; it never contains a
integration. An External Data Plane artifact builds only its image and provider credential, provider endpoint, collection schedule or command
force-recreates only `external-data-plane`; the already healthy transport. Database credentials remain root-owned live `.env.synology`
`external-data-plane-postgres` container and its Timescale volume are an
independent deploy prerequisite and are never selected by an EDP application
artifact. The reviewed Compose source pins the Timescale image, named volume
and target, internal database-only network, absence of database host ports,
healthy dependency, localhost-only EDP bind and the two read-only
provisioner/trust mounts in the reviewed Compose source. The runner does not
reinterpret version-dependent `docker compose config` JSON as a second deploy
schema. Its canonical enforcement remains the artifact/path allowlist plus
hard-coded build command, selected service set, runtime-secret preparation and
health acceptance. Post-apply acceptance also requires
`database=ready`; a first-rollout failure removes only the candidate EDP
container without volumes and restores the source overlay. It never contains a
provider credential, provider endpoint,
collection schedule or command
transport. Its contract payload is an exact provider-neutral runtime subset;
`providers/*`, mappings, fixtures and tests are excluded and do not trigger an
EDP rebuild. Database credentials remain root-owned live `.env.synology`
configuration and must not reuse `NODEDC_INTERNAL_ACCESS_TOKEN`. configuration and must not reuse `NODEDC_INTERNAL_ACCESS_TOKEN`.
On the first relevant Platform apply, the root-owned runner creates The External Data Plane writer-provisioner credential is different: on the
first relevant Platform apply, the root-owned runner creates
`/volume1/docker/nodedc-platform/secrets/external-data-plane-provisioner/token` `/volume1/docker/nodedc-platform/secrets/external-data-plane-provisioner/token`
atomically in a dedicated UID/GID `11006` directory (directory `0500`, token atomically in a dedicated UID/GID `11006` directory (directory `0500`, token
`0400`). It is never an `.env` value or an artifact member and is mounted `0400`). It is never an `.env` value or an artifact member; Compose mounts it
read-only only into External Data Plane. read-only only into External Data Plane and, when implemented, its dedicated
Engine provisioner running under the same restricted identity.
Manual one-time binding issuance and digest-only managed writer ensure are
independently disabled by default. The target control-plane operation generates
and stores the capability inside native NDC L2 Credentials and sends only its
digest to EDP; users and MCP consumers receive only an opaque compatible
reference/status. The runner-owned bearer above authenticates only legacy
plaintext issuance and is intentionally rejected by managed ensure/revoke.
Managed requests use a deployment Ed25519 Engine service key; EDP mounts only
the public-key trust directory read-only. On every reviewed Engine apply and on
an EDP runtime apply, this runner creates or validates one matching Ed25519 pair:
the Engine-only private key is `root:root 0400`, while the EDP trust copy is
`root:11006 0440`. Public-only crash state, key mismatch, a non-Ed25519 key,
symlinks and permissive modes fail closed. `plan` discloses both paths without
printing key material. The private key must not be broadened into an L2 graph,
MCP surface, artifact or shared-token boundary.
The reviewed Engine source candidate has dedicated server-derived MCP
plan/apply handling for the exact `ndcDataProductWriterApi` + Data Product
Publish tuple. It is separate from the generic HTTP safe-ref path and accepts no
caller-provided provider/scope/credential identity, capability, generation or
service URL. The production managed flag remains false during staging. In the
explicitly confirmed activation window it is set to true immediately before
the Platform EDP artifact; runner acceptance then requires EDP `/healthz` to
report managed provisioning `enabled`. At that point the old Engine still has
no signer mount, so the endpoint remains usable only after the separately
accepted Engine artifact. Root/UI transfer is emergency
self-hosted diagnostics only, not the user journey or acceptance
path. A provider-specific daemon remains forbidden.
Build the narrow Engine source artifact with:
```bash
node infra/deploy-runner/build-engine-data-product-publish-grant-artifact.mjs \
engine-data-product-publish-grant-YYYYMMDD-NNN
```
The builder includes exactly the pinned provider-security catalog, the
Publish-grant service, Engine Agent scope/gateway wiring, the existing n8n
adapter and the reviewed additive backend runtime overlay. It deliberately
excludes base Compose, frontend/dist, runtime data, tests, native credentials,
the NDC L2 process and the legacy generic credential-sink route/core. The
runner fixes the runtime action to `nodedc-backend` with `--no-deps` and
`--pull never`; n8n, nginx app, database services and volumes are not selected.
Acceptance requires backend health, the active immutable backend identity and
the exact additive mount inventory. Any failure restores the touched source and
recreates only the previous verified backend runtime.
Existing Engine Agents created before Publish-grant support store the original
nine scopes as the complete developer profile. They must not require a new
agent, setup command or device credential when the profile gains a server-owned
capability. Build the compatibility artifact that introduces the durable named
`full-developer` profile with:
```bash
node infra/deploy-runner/build-engine-agent-full-grant-migration-artifact.mjs \
engine-agent-full-grant-migration-YYYYMMDD-NNN
```
This follow-up slice contains exactly
`nodedc-source/server/engineAgents/store.js`. The runner pins its predecessor to
the successfully applied Publish generation, requires the installed Publish
overlay and active immutable backend, recreates only `nodedc-backend`, and
proves the exact candidate SHA, named profile and current expanded runtime
scope view. Store schema v1 is migrated atomically to v2 only when a grant
contains the complete legacy nine-scope developer bundle. From then on the
profile name is the authorization authority and its scope list is derived on
every read, so capabilities deliberately added to `full-developer` immediately
apply to existing full grants without token or store migrations. Partial grants
become `custom` and remain exact; they are never elevated. The artifact does not
touch UI, n8n, credentials, agent tokens, workflow graphs, databases or runtime
payloads.
`module-foundry` is an independent, authenticated application component. Its `module-foundry` is an independent, authenticated application component. Its
artifact contains source and compose infrastructure only; its live artifact contains source and compose infrastructure only; its live
@ -288,17 +190,12 @@ verified; it does not alter NAS routes, VPN, DNS, or Tailscale.
Install or update the root-owned live runner on Synology: Install or update the root-owned live runner on Synology:
```bash ```bash
# First verify that no deploy process is active and state/deploy.lock is absent.
sudo install -o root -g root -m 0755 \ sudo install -o root -g root -m 0755 \
/volume1/docker/nodedc-deploy/runner-install/nodedc-deploy \ /volume1/docker/nodedc-deploy/runner-install/nodedc-deploy \
/usr/local/sbin/nodedc-deploy /usr/local/sbin/nodedc-deploy
sudo /usr/local/sbin/nodedc-deploy verify-install sudo /usr/local/sbin/nodedc-deploy verify-install
``` ```
Runner promotion is a standalone admin step, never an app-overlay artifact and
never part of a running apply. A Python process already executing the old file
keeps the old code in memory; always invoke a fresh verified process afterward.
Normal service deploys must still use explicit artifacts: Normal service deploys must still use explicit artifacts:
```bash ```bash

121
infra/deploy-runner/build-engine-agent-full-grant-migration-artifact.mjs
View File

@ -1,121 +0,0 @@
#!/usr/bin/env node
import { createHash } from 'node:crypto'
import { spawnSync } from 'node:child_process'
import { cp, lstat, mkdir, mkdtemp, readFile, rm, writeFile } from 'node:fs/promises'
import { tmpdir } from 'node:os'
import { dirname, join, resolve } from 'node:path'
import { fileURLToPath } from 'node:url'
const here = dirname(fileURLToPath(import.meta.url))
const platformRoot = resolve(here, '../..')
const workspaceRoot = resolve(platformRoot, '..')
const engineRoot = resolve(workspaceRoot, 'NODEDC_ENGINE_INFRA')
const artifactRoot = resolve(process.env.NODEDC_DEPLOY_ARTIFACT_DIR || join(workspaceRoot, 'deploy-artifacts'))
const [patchId = '', ...extra] = process.argv.slice(2)
const storeRelativePath = 'nodedc-source/server/engineAgents/store.js'
const predecessorSha256 = '52daa43499d6d9a97fe7ffa891edb9212b7791e733f91dd3ca686d42739b7e9a'
const targetSha256 = '2e62654c2dc12905efcc83a9dff45a818dd7b47924a10600160835c4416540e9'
const previouslyIssuedPatchIds = new Set([
'engine-agent-full-grant-migration-20260717-001',
])
if (extra.length || !/^engine-agent-full-grant-migration-\d{8}-\d{3}$/.test(patchId)) {
throw new Error('usage: build-engine-agent-full-grant-migration-artifact.mjs <fresh-patch-id>')
}
if (previouslyIssuedPatchIds.has(patchId)) {
throw new Error('engine_agent_full_grant_migration_patch_id_already_issued')
}
const source = join(engineRoot, storeRelativePath)
const sourceInfo = await lstat(source)
if (sourceInfo.isSymbolicLink() || !sourceInfo.isFile()) throw new Error('engine_agent_store_source_unsafe')
const sourceBytes = await readFile(source)
if (digest(sourceBytes) !== targetSha256) throw new Error('engine_agent_store_target_sha256_mismatch')
const sourceText = sourceBytes.toString('utf8')
for (const required of [
'const STORE_VERSION = 2',
"export const ENGINE_AGENT_FULL_DEVELOPER_PROFILE = 'full-developer'",
"export const ENGINE_AGENT_CUSTOM_PROFILE = 'custom'",
'const LEGACY_FULL_DEVELOPER_SCOPES = Object.freeze([',
"'engine:l2:data-product-publish-grant:plan'",
"'engine:l2:data-product-publish-grant:write'",
'const migrateLegacyFullDeveloper = sourceVersion === 1',
'LEGACY_FULL_DEVELOPER_SCOPES.every((scope) => scopes.includes(scope))',
'profile === ENGINE_AGENT_FULL_DEVELOPER_PROFILE ? [...ENGINE_AGENT_SCOPES] : scopes',
"throw new Error('engine_agent_store_version_unsupported')",
]) {
if (!sourceText.includes(required)) throw new Error(`engine_agent_store_contract_missing:${required}`)
}
if (/gelios|robot2b/i.test(sourceText)) throw new Error('engine_agent_store_provider_logic_forbidden')
run('node', ['--check', source])
await mkdir(artifactRoot, { recursive: true })
const artifact = join(artifactRoot, `nodedc-${patchId}.tgz`)
await assertFresh(artifact)
const stage = await mkdtemp(join(tmpdir(), 'nodedc-engine-agent-grant-migration-'))
try {
const destination = join(stage, 'payload', storeRelativePath)
await mkdir(dirname(destination), { recursive: true })
await cp(source, destination, { force: false, verbatimSymlinks: true })
await writeFile(
join(stage, 'manifest.env'),
`id=${patchId}\ncomponent=engine\ntype=app-overlay\n`,
'utf8',
)
await writeFile(join(stage, 'files.txt'), `${storeRelativePath}\n`, 'utf8')
run('python3', ['-c', canonicalTarScript(), artifact, stage])
const artifactBytes = await readFile(artifact)
console.log(JSON.stringify({
ok: true,
patchId,
artifact,
sha256: digest(artifactBytes),
entries: [storeRelativePath],
predecessorSha256,
targetSha256,
services: ['nodedc-backend'],
excluded: ['n8n', 'app', 'databases', 'credentials', 'runtime-data'],
}, null, 2))
} finally {
await rm(stage, { recursive: true, force: true })
}
async function assertFresh(target) {
try {
await lstat(target)
} catch (error) {
if (error?.code === 'ENOENT') return
throw error
}
throw new Error('engine_agent_full_grant_migration_artifact_already_exists')
}
function canonicalTarScript() {
return [
'import gzip, io, pathlib, sys, tarfile',
'root=pathlib.Path(sys.argv[2])',
"with open(sys.argv[1], 'wb') as out:",
" with gzip.GzipFile(filename='', mode='wb', fileobj=out, compresslevel=9, mtime=0) as gz:",
" with tarfile.open(fileobj=gz, mode='w', format=tarfile.PAX_FORMAT) as tar:",
" for top in ('manifest.env','files.txt','payload'):",
" p=root/top; paths=[p] + (sorted(p.rglob('*')) if p.is_dir() else [])",
' for x in paths:',
" info=tar.gettarinfo(str(x), arcname=x.relative_to(root).as_posix())",
" info.uid=info.gid=0; info.uname=info.gname='root'; info.mtime=0; info.mode=0o755 if info.isdir() else 0o644",
" with (open(x,'rb') if info.isfile() else io.BytesIO()) as src: tar.addfile(info, src if info.isfile() else None)",
].join('\n')
}
function run(command, args) {
const result = spawnSync(command, args, {
encoding: 'utf8',
stdio: ['ignore', 'pipe', 'pipe'],
maxBuffer: 16 * 1024 * 1024,
})
if (result.status !== 0) throw new Error(`${command}_failed:${result.stderr || result.stdout}`)
return result
}
function digest(bytes) {
return createHash('sha256').update(bytes).digest('hex')
}

156
infra/deploy-runner/build-engine-credential-sink-recovery-artifact.mjs
View File

@ -1,156 +0,0 @@
#!/usr/bin/env node
import { createHash } from 'node:crypto'
import { spawnSync } from 'node:child_process'
import { lstat, mkdir, mkdtemp, readFile, rm, writeFile } from 'node:fs/promises'
import { tmpdir } from 'node:os'
import { dirname, join, resolve } from 'node:path'
import { fileURLToPath } from 'node:url'
const here = dirname(fileURLToPath(import.meta.url))
const platformRoot = resolve(here, '../..')
const workspaceRoot = resolve(platformRoot, '..')
const artifactRoot = resolve(process.env.NODEDC_DEPLOY_ARTIFACT_DIR || join(workspaceRoot, 'deploy-artifacts'))
const sourceArtifact = resolve(
process.env.NODEDC_ENGINE_CREDENTIAL_SINK_RECOVERY_SOURCE
|| '/Volumes/docker/nodedc-deploy/applied/nodedc-engine-credential-sink-20260716-001.tgz',
)
const [patchId = '', ...extra] = process.argv.slice(2)
if (extra.length || !/^engine-credential-sink-recovery-\d{8}-\d{3}$/.test(patchId)) {
throw new Error('usage: build-engine-credential-sink-recovery-artifact.mjs <fresh-recovery-patch-id>')
}
const sourceArtifactSha256 = '0a96add05fe59db8f490927f66e07a84490474a7afb3ef7de51e1d6fd96f86a2'
const sourceManifest = 'id=engine-credential-sink-20260716-001\ncomponent=engine\ntype=app-overlay\n'
const entries = Object.freeze([
'nodedc-source/server/credentialPolicies/ndcPrivateNode.js',
'nodedc-source/server/credentialSink',
'nodedc-source/server/index.js',
'nodedc-source/server/routes/engineAgentGateway.js',
'nodedc-source/server/routes/engineCredentialSink.js',
'nodedc-source/server/routes/n8n.js',
'nodedc-source/server/routes/ndcAgentMcp.js',
'nodedc-source/services/backend/credential-sink/docker-compose.immutable-runtime.yml',
])
const payloadSha256 = Object.freeze({
'nodedc-source/server/credentialPolicies/ndcPrivateNode.js': '723874a02dc7b8a68b22ff2304431cfe64f28933cedf5f1e6cda79b2e1cf704a',
'nodedc-source/server/credentialSink/core.js': '9f0facc41fd398fcd955cffdd486abb126cdcd756c87ffde667fcbe00e2c41d3',
'nodedc-source/server/credentialSink/requestAuth.js': 'f8c9237c3e6f4219dee0d4f956d6e97f76f5bd7ba8a6fd0b4fb21aceffa38138',
'nodedc-source/server/credentialSink/store.js': 'c78dc285a973b6acd8a2330f0310935ad08720b2905454492f292d235abf12c0',
'nodedc-source/server/credentialSink/vendor/engine-credential-sink.mjs': 'b4800eead9bf94793ff1280d06e34aad6b37ef055a9d7f8d83d7fb5f9bd66d8b',
'nodedc-source/server/index.js': 'b2b790b02839570d967a2ca68b00e2724485a99389ac9b3a589a1b22302a36b8',
'nodedc-source/server/routes/engineAgentGateway.js': 'e3450a4e1d5318dbac37627b67804d62804e27e2032c9e90478a1e739cea6d3d',
'nodedc-source/server/routes/engineCredentialSink.js': '9cbb69dbc8cbe6181cd5b0170fe9c4d717b0a173866ca98cba3e3766c0bab94e',
'nodedc-source/server/routes/n8n.js': '783d822e2457d82e890f43bc00c7e33822077dc2841f0511a89ecb210fd36d48',
'nodedc-source/server/routes/ndcAgentMcp.js': 'fbb3342b1a617b956d5b3a6a40d5111aa107c1176b4ff89c37b104995bada081',
'nodedc-source/services/backend/credential-sink/docker-compose.immutable-runtime.yml': '944fa64b08255eb8207b93fd327aebb98ecd9400d37d25fcfa8e3a040ee44afe',
})
const sourceInfo = await lstat(sourceArtifact)
if (sourceInfo.isSymbolicLink() || !sourceInfo.isFile()) {
throw new Error('engine_credential_sink_recovery_source_unsafe')
}
if (digest(await readFile(sourceArtifact)) !== sourceArtifactSha256) {
throw new Error('engine_credential_sink_recovery_source_sha256_mismatch')
}
await mkdir(artifactRoot, { recursive: true })
const artifact = join(artifactRoot, `nodedc-${patchId}.tgz`)
await assertArtifactTargetFresh(artifact)
const stage = await mkdtemp(join(tmpdir(), 'nodedc-engine-credential-sink-recovery-'))
try {
run('python3', [
'-c', verifiedExtractScript(), sourceArtifact, stage,
sourceArtifactSha256, sourceManifest, JSON.stringify(entries), JSON.stringify(payloadSha256),
])
await writeFile(
join(stage, 'manifest.env'),
`id=${patchId}\ncomponent=engine\ntype=app-overlay\n`,
'utf8',
)
await writeFile(join(stage, 'files.txt'), `${entries.join('\n')}\n`, 'utf8')
run('python3', ['-c', canonicalTarScript(), artifact, stage])
console.log(JSON.stringify({
ok: true,
patchId,
artifact,
sha256: digest(await readFile(artifact)),
sourceArtifact,
sourceArtifactSha256,
entries,
payloadSha256,
changed: ['manifest.env:id'],
}, null, 2))
} finally {
await rm(stage, { recursive: true, force: true })
}
async function assertArtifactTargetFresh(target) {
try {
await lstat(target)
} catch (error) {
if (error?.code === 'ENOENT') return
throw error
}
throw new Error('engine_credential_sink_recovery_artifact_already_exists')
}
function verifiedExtractScript() {
return [
'import hashlib,json,pathlib,sys,tarfile',
'source=pathlib.Path(sys.argv[1]); stage=pathlib.Path(sys.argv[2])',
'expected_archive=sys.argv[3]; expected_manifest=sys.argv[4].encode()',
'entries=json.loads(sys.argv[5]); hashes=json.loads(sys.argv[6])',
"raw=source.read_bytes()",
"assert hashlib.sha256(raw).hexdigest()==expected_archive, 'source-sha256'",
"expected_files={'manifest.env','files.txt'}|{'payload/'+name for name in hashes}",
"expected_dirs={'payload'}",
"for name in hashes:",
" p=pathlib.PurePosixPath('payload/'+name)",
" expected_dirs.update(str(parent) for parent in p.parents if str(parent)!='.')",
"with tarfile.open(source,'r:gz') as tar:",
" members=tar.getmembers(); names={member.name for member in members}",
" assert names==expected_files|expected_dirs, 'source-member-set'",
" for member in members:",
" p=pathlib.PurePosixPath(member.name)",
" assert not p.is_absolute() and '..' not in p.parts and '\\\\' not in member.name, 'unsafe-member'",
" assert member.isdir() if member.name in expected_dirs else member.isfile(), 'unsafe-member-type'",
" assert tar.extractfile('manifest.env').read()==expected_manifest, 'source-manifest'",
" expected_list=('\\n'.join(entries)+'\\n').encode()",
" assert tar.extractfile('files.txt').read()==expected_list, 'source-files-list'",
" for name,wanted in hashes.items():",
" data=tar.extractfile('payload/'+name).read()",
" assert hashlib.sha256(data).hexdigest()==wanted, 'payload-sha256:'+name",
" target=stage/'payload'/name; target.parent.mkdir(parents=True,exist_ok=True); target.write_bytes(data)",
].join('\n')
}
function canonicalTarScript() {
return [
'import gzip,io,pathlib,sys,tarfile',
'root=pathlib.Path(sys.argv[2])',
"with open(sys.argv[1],'wb') as out:",
" with gzip.GzipFile(filename='',mode='wb',fileobj=out,compresslevel=9,mtime=0) as gz:",
" with tarfile.open(fileobj=gz,mode='w',format=tarfile.PAX_FORMAT) as tar:",
" for top in ('manifest.env','files.txt','payload'):",
" p=root/top; paths=[p]+(sorted(p.rglob('*')) if p.is_dir() else [])",
" for x in paths:",
" info=tar.gettarinfo(str(x),arcname=x.relative_to(root).as_posix())",
" info.uid=info.gid=0; info.uname=info.gname='root'; info.mtime=0; info.mode=0o755 if info.isdir() else 0o644",
" with (open(x,'rb') if info.isfile() else io.BytesIO()) as src: tar.addfile(info,src if info.isfile() else None)",
].join('\n')
}
function run(command, args) {
const result = spawnSync(command, args, {
encoding: 'utf8',
stdio: ['ignore', 'pipe', 'pipe'],
maxBuffer: 128 * 1024 * 1024,
})
if (result.status !== 0) throw new Error(`${command}_failed:${result.stderr || result.stdout}`)
}
function digest(bytes) {
return createHash('sha256').update(bytes).digest('hex')
}

252
infra/deploy-runner/build-engine-data-product-publish-grant-artifact.mjs
View File

@ -1,252 +0,0 @@
#!/usr/bin/env node
import { createHash } from 'node:crypto'
import { spawnSync } from 'node:child_process'
import { cp, lstat, mkdir, mkdtemp, readFile, readdir, rm, writeFile } from 'node:fs/promises'
import { tmpdir } from 'node:os'
import { dirname, join, relative, resolve } from 'node:path'
import { fileURLToPath } from 'node:url'
const here = dirname(fileURLToPath(import.meta.url))
const platformRoot = resolve(here, '../..')
const workspaceRoot = resolve(platformRoot, '..')
const engineRoot = resolve(workspaceRoot, 'NODEDC_ENGINE_INFRA')
const artifactRoot = resolve(process.env.NODEDC_DEPLOY_ARTIFACT_DIR || join(workspaceRoot, 'deploy-artifacts'))
const [patchId = '', ...extra] = process.argv.slice(2)
const previouslyIssuedPatchIds = new Set([
'engine-data-product-publish-grant-20260716-001',
'engine-data-product-publish-grant-20260717-001',
'engine-data-product-publish-grant-20260717-002',
])
if (extra.length || !/^engine-data-product-publish-grant-\d{8}-\d{3}$/.test(patchId)) {
throw new Error('usage: build-engine-data-product-publish-grant-artifact.mjs <fresh-patch-id>')
}
if (previouslyIssuedPatchIds.has(patchId)) throw new Error('engine_publish_grant_patch_id_already_issued')
// Deliberately exclude frontend/dist, runtime data, tests, every credential
// value, and the already-installed credential-sink implementation. The sink
// remains mounted and byte-frozen as the predecessor domain; this artifact
// adds the exact Data Product Publish grant flow beside it.
const entries = Object.freeze([
'nodedc-source/server/assets/provider-packages/v1/catalog.json',
'nodedc-source/server/dataProductPublishGrant',
'nodedc-source/server/engineAgents/store.js',
'nodedc-source/server/routes/engineAgentGateway.js',
'nodedc-source/server/routes/n8n.js',
'nodedc-source/services/backend/data-product-publish-grant/docker-compose.immutable-runtime.yml',
])
const ignoredBasenames = new Set(['.DS_Store', '.git', 'node_modules'])
const credentialSinkPredecessorSha256 = Object.freeze({
'docker-compose.yml': '258cebb64ff1943c939655cc55bdce00fc5c4dced67ec291d84d6df066ace50e',
'nodedc-source/server/credentialPolicies/ndcPrivateNode.js': '723874a02dc7b8a68b22ff2304431cfe64f28933cedf5f1e6cda79b2e1cf704a',
'nodedc-source/server/credentialSink/core.js': '9f0facc41fd398fcd955cffdd486abb126cdcd756c87ffde667fcbe00e2c41d3',
'nodedc-source/server/credentialSink/requestAuth.js': 'f8c9237c3e6f4219dee0d4f956d6e97f76f5bd7ba8a6fd0b4fb21aceffa38138',
'nodedc-source/server/credentialSink/store.js': 'c78dc285a973b6acd8a2330f0310935ad08720b2905454492f292d235abf12c0',
'nodedc-source/server/credentialSink/vendor/engine-credential-sink.mjs': 'b4800eead9bf94793ff1280d06e34aad6b37ef055a9d7f8d83d7fb5f9bd66d8b',
'nodedc-source/server/index.js': 'b2b790b02839570d967a2ca68b00e2724485a99389ac9b3a589a1b22302a36b8',
'nodedc-source/server/routes/engineCredentialSink.js': '9cbb69dbc8cbe6181cd5b0170fe9c4d717b0a173866ca98cba3e3766c0bab94e',
'nodedc-source/server/routes/ndcAgentMcp.js': 'fbb3342b1a617b956d5b3a6a40d5111aa107c1176b4ff89c37b104995bada081',
'nodedc-source/services/backend/credential-sink/docker-compose.immutable-runtime.yml': '944fa64b08255eb8207b93fd327aebb98ecd9400d37d25fcfa8e3a040ee44afe',
})
const publishGrantRuntimeOverride = [
'services:',
' nodedc-backend:',
' user: "0:0"',
' environment:',
' ENGINE_DATA_PLANE_BASE_URL: http://external-data-plane:18106',
' ENGINE_EDP_MANAGED_PROVISIONER_PRIVATE_KEY_FILE: /run/nodedc-secrets/engine-edp-managed-provisioner-private.pem',
' ENGINE_CONTROL_PLANE_PUBLISH_GRANT_ROOT: /var/lib/nodedc-control-plane/publish-grants',
' volumes:',
' - type: bind',
' source: /volume2/nodedc-demo/nodedc-control-plane/publish-grants',
' target: /var/lib/nodedc-control-plane/publish-grants',
' bind:',
' create_host_path: false',
' - type: bind',
' source: /volume1/docker/nodedc-platform/secrets/engine-edp-managed-provisioner/private-key.pem',
' target: /run/nodedc-secrets/engine-edp-managed-provisioner-private.pem',
' read_only: true',
' bind:',
' create_host_path: false',
'',
].join('\n')
await assertSourceBoundary()
await mkdir(artifactRoot, { recursive: true })
const artifact = join(artifactRoot, `nodedc-${patchId}.tgz`)
await assertArtifactTargetFresh(artifact)
const stage = await mkdtemp(join(tmpdir(), 'nodedc-engine-publish-grant-artifact-'))
const payload = join(stage, 'payload')
try {
await mkdir(payload, { recursive: true })
for (const entry of entries) {
await copySafe(resolve(engineRoot, entry), join(payload, entry))
}
await writeFile(
join(stage, 'manifest.env'),
`id=${patchId}\ncomponent=engine\ntype=app-overlay\n`,
'utf8',
)
await writeFile(join(stage, 'files.txt'), `${entries.join('\n')}\n`, 'utf8')
run('python3', ['-c', canonicalTarScript(), artifact, stage])
const sha256 = digest(await readFile(artifact))
console.log(JSON.stringify({
ok: true,
patchId,
artifact,
sha256,
entries,
excluded: [
'docker-compose.yml',
'nodedc-source/server/index.js',
'nodedc-source/server/credentialPolicies/ndcPrivateNode.js',
'nodedc-source/server/credentialSink',
'nodedc-source/server/routes/engineCredentialSink.js',
'nodedc-source/server/middleware/demoAccess.js',
'nodedc-source/server/routes/ndcAgentMcp.js',
'nodedc-source/server/data',
'nodedc-source/dist',
'nodedc-source/server/tests',
],
preservedPredecessor: Object.keys(credentialSinkPredecessorSha256),
}, null, 2))
} finally {
await rm(stage, { recursive: true, force: true })
}
async function assertSourceBoundary() {
for (const [relativePath, expectedSha256] of Object.entries(credentialSinkPredecessorSha256)) {
const bytes = await readFile(join(engineRoot, relativePath))
if (digest(bytes) !== expectedSha256) {
throw new Error(`engine_credential_sink_predecessor_drift:${relativePath}`)
}
}
const runtimeOverridePath = join(
engineRoot,
'nodedc-source/services/backend/data-product-publish-grant/docker-compose.immutable-runtime.yml',
)
if (await readFile(runtimeOverridePath, 'utf8') !== publishGrantRuntimeOverride) {
throw new Error('engine_publish_grant_runtime_override_mismatch')
}
const indexSource = await readFile(join(engineRoot, 'nodedc-source/server/index.js'), 'utf8')
if (!indexSource.includes("app.use('/api/engine-agent-mcp', engineAgentMcpRouter)")) {
throw new Error('engine_agent_mcp_mount_missing')
}
if (
!indexSource.includes("import engineCredentialSinkRouter from './routes/engineCredentialSink.js'")
|| !indexSource.includes("app.use('/internal/engine-credential-sink', express.json({")
) throw new Error('engine_credential_sink_predecessor_mount_missing')
const gateway = await readFile(join(engineRoot, 'nodedc-source/server/routes/engineAgentGateway.js'), 'utf8')
for (const tool of [
'engine_plan_data_product_publish_grant',
'engine_apply_data_product_publish_grant',
'engine_accept_data_product_publish_grant',
'engine_rollback_data_product_publish_grant',
]) {
if (!gateway.includes(tool)) throw new Error(`engine_publish_grant_tool_missing:${tool}`)
}
const n8nRoute = await readFile(join(engineRoot, 'nodedc-source/server/routes/n8n.js'), 'utf8')
if (n8nRoute.includes("from '../credentialSink/")) {
throw new Error('engine_publish_grant_depends_on_legacy_sink')
}
if (!n8nRoute.includes('engineDataProductPublishGrantN8nAdapter')) {
throw new Error('engine_publish_grant_native_adapter_missing')
}
if (!n8nRoute.includes('engineCredentialSinkN8nAdapter')) {
throw new Error('engine_credential_sink_native_adapter_missing')
}
const grantDirectory = join(engineRoot, 'nodedc-source/server/dataProductPublishGrant')
const grantFiles = (await readdir(grantDirectory, { withFileTypes: true }))
.filter((entry) => entry.isFile())
.map((entry) => entry.name)
.sort()
const expectedGrantFiles = [
'acceptance.js',
'providerCatalog.js',
'service.js',
'signedDataPlaneClient.js',
'store.js',
]
if (JSON.stringify(grantFiles) !== JSON.stringify(expectedGrantFiles)) {
throw new Error('engine_publish_grant_source_set_mismatch')
}
for (const entry of entries) {
const source = resolve(engineRoot, entry)
const info = await lstat(source)
if (info.isSymbolicLink() || (!info.isFile() && !info.isDirectory())) {
throw new Error(`engine_artifact_source_unsafe:${entry}`)
}
}
for (const source of [
...expectedGrantFiles.map((name) => join(grantDirectory, name)),
join(engineRoot, 'nodedc-source/server/routes/engineAgentGateway.js'),
join(engineRoot, 'nodedc-source/server/routes/n8n.js'),
join(engineRoot, 'nodedc-source/server/routes/ndcAgentMcp.js'),
join(engineRoot, 'nodedc-source/server/engineAgents/store.js'),
]) run('node', ['--check', source])
}
async function assertArtifactTargetFresh(target) {
try {
await lstat(target)
} catch (error) {
if (error?.code === 'ENOENT') return
throw error
}
throw new Error('engine_publish_grant_artifact_already_exists')
}
async function copySafe(source, destination) {
const info = await lstat(source)
if (info.isSymbolicLink()) throw new Error(`source_symlink_rejected:${source}`)
if (info.isFile()) {
await mkdir(dirname(destination), { recursive: true })
await cp(source, destination, { force: false, verbatimSymlinks: true })
return
}
if (!info.isDirectory()) throw new Error(`source_type_rejected:${source}`)
await mkdir(destination, { recursive: true })
for (const entry of await readdir(source, { withFileTypes: true })) {
if (ignoredBasenames.has(entry.name) || entry.name.startsWith('.env')) continue
const childSource = join(source, entry.name)
if (entry.isSymbolicLink()) {
throw new Error(`source_symlink_rejected:${relative(engineRoot, childSource)}`)
}
await copySafe(childSource, join(destination, entry.name))
}
}
function canonicalTarScript() {
return [
'import gzip, io, pathlib, sys, tarfile',
'root=pathlib.Path(sys.argv[2])',
"with open(sys.argv[1], 'wb') as out:",
" with gzip.GzipFile(filename='', mode='wb', fileobj=out, compresslevel=9, mtime=0) as gz:",
" with tarfile.open(fileobj=gz, mode='w', format=tarfile.PAX_FORMAT) as tar:",
" for top in ('manifest.env','files.txt','payload'):",
" p=root/top; paths=[p] + (sorted(p.rglob('*')) if p.is_dir() else [])",
' for x in paths:',
" info=tar.gettarinfo(str(x), arcname=x.relative_to(root).as_posix())",
" info.uid=info.gid=0; info.uname=info.gname='root'; info.mtime=0; info.mode=0o755 if info.isdir() else 0o644",
" with (open(x,'rb') if info.isfile() else io.BytesIO()) as src: tar.addfile(info, src if info.isfile() else None)",
].join('\n')
}
function run(command, args) {
const result = spawnSync(command, args, {
encoding: 'utf8',
stdio: ['ignore', 'pipe', 'pipe'],
maxBuffer: 128 * 1024 * 1024,
})
if (result.status !== 0) throw new Error(`${command}_failed:${result.stderr || result.stdout}`)
return result
}
function digest(bytes) {
return createHash('sha256').update(bytes).digest('hex')
}

65
infra/deploy-runner/build-engine-n8n-private-extension-artifact.mjs
View File

@ -1,7 +1,7 @@
#!/usr/bin/env node #!/usr/bin/env node
import { createHash } from "node:crypto"; import { createHash } from "node:crypto";
import { spawnSync } from "node:child_process"; import { spawnSync } from "node:child_process";
import { cp, lstat, mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises"; import { cp, mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
import { createRequire, Module } from "node:module"; import { createRequire, Module } from "node:module";
import { tmpdir } from "node:os"; import { tmpdir } from "node:os";
import { dirname, join, resolve } from "node:path"; import { dirname, join, resolve } from "node:path";
@ -24,10 +24,8 @@ const n8nVersion = "2.3.2";
const baseImage = "docker.n8n.io/n8nio/n8n:2.3.2"; const baseImage = "docker.n8n.io/n8nio/n8n:2.3.2";
const architecture = "amd64"; const architecture = "amd64";
const generatedAt = "2026-07-15T21:51:41.000Z"; const generatedAt = "2026-07-15T21:51:41.000Z";
const previouslyIssuedTransitionIds = new Set(["20260715-002", "20260716-003"]); const activationId = "engine-n8n-private-extension-20260716-003";
const transitionId = readTransitionId(process.argv.slice(2), process.env.NODEDC_N8N_TRANSITION_ID); const rollbackId = "engine-n8n-private-extension-rollback-20260716-003";
const activationId = `engine-n8n-private-extension-${transitionId}`;
const rollbackId = `engine-n8n-private-extension-rollback-${transitionId}`;
const transitionRoot = "nodedc-source/services/n8n/private-extensions"; const transitionRoot = "nodedc-source/services/n8n/private-extensions";
const descriptorRel = `${transitionRoot}/ndc-activation.json`; const descriptorRel = `${transitionRoot}/ndc-activation.json`;
@ -64,8 +62,6 @@ const credentialModules = [
]; ];
await mkdir(artifactRoot, { recursive: true }); await mkdir(artifactRoot, { recursive: true });
await assertArtifactTargetFresh(join(artifactRoot, `nodedc-${activationId}.tgz`));
await assertArtifactTargetFresh(join(artifactRoot, `nodedc-${rollbackId}.tgz`));
assertSha(await readFile(stageArtifact), stageArtifactSha256, "staging artifact"); assertSha(await readFile(stageArtifact), stageArtifactSha256, "staging artifact");
assertEngineBaseline(await readFile(join(engineRoot, "docker-compose.yml"), "utf8")); assertEngineBaseline(await readFile(join(engineRoot, "docker-compose.yml"), "utf8"));
@ -129,15 +125,13 @@ try {
const rollbackDescriptor = descriptor("rollback-inactive", [], [], releaseId); const rollbackDescriptor = descriptor("rollback-inactive", [], [], releaseId);
const override = composeOverride(); const override = composeOverride();
const generatedRoot = join(work, "generated-engine-payload"); await writeJson(join(engineRoot, nodesCatalogRel), activeNodes);
await writeJson(join(generatedRoot, nodesCatalogRel), activeNodes); await writeJson(join(engineRoot, credentialsCatalogRel), activeCredentials);
await writeJson(join(generatedRoot, credentialsCatalogRel), activeCredentials); await writeJson(join(engineRoot, metaRel), activeMeta);
await writeJson(join(generatedRoot, metaRel), activeMeta); await writeJson(join(engineRoot, descriptorRel), activationDescriptor);
await writeJson(join(generatedRoot, descriptorRel), activationDescriptor); await writeFile(join(engineRoot, overrideRel), override, "utf8");
await writeFile(join(generatedRoot, overrideRel), override, "utf8"); await cp(join(packageRoot, "dist/icons/ndc.svg"), join(engineRoot, iconRel), { force: true });
await mkdir(join(generatedRoot, iconRoot), { recursive: true }); await cp(join(packageRoot, "dist/icons/ndc.dark.svg"), join(engineRoot, darkIconRel), { force: true });
await cp(join(packageRoot, "dist/icons/ndc.svg"), join(generatedRoot, iconRel), { force: false });
await cp(join(packageRoot, "dist/icons/ndc.dark.svg"), join(generatedRoot, darkIconRel), { force: false });
const activationEntries = [ const activationEntries = [
descriptorRel, descriptorRel,
@ -150,7 +144,7 @@ try {
]; ];
const activationArtifact = await buildArtifact(work, activationId, activationEntries, async (payload) => { const activationArtifact = await buildArtifact(work, activationId, activationEntries, async (payload) => {
for (const rel of activationEntries) { for (const rel of activationEntries) {
await cp(join(generatedRoot, rel), join(payload, rel), { recursive: true, force: false }); await cp(join(engineRoot, rel), join(payload, rel), { recursive: true, force: false });
} }
}); });
@ -165,7 +159,6 @@ try {
console.log(JSON.stringify({ console.log(JSON.stringify({
ok: true, ok: true,
transitionId,
releaseId, releaseId,
packageSha256, packageSha256,
nodeTypes: expectedNodeTypes, nodeTypes: expectedNodeTypes,
@ -199,42 +192,6 @@ function descriptor(action, nodeTypes, credentialTypes, expectedCurrent) {
}; };
} }
function readTransitionId(args, environmentValue) {
if (args.length > 1) throw new Error("transition_id_argument_count_invalid");
const argumentValue = args[0] || "";
const envValue = String(environmentValue || "").trim();
if (argumentValue && envValue && argumentValue !== envValue) {
throw new Error("transition_id_sources_conflict");
}
const value = argumentValue || envValue;
if (!value) throw new Error("transition_id_required");
const match = /^(\d{4})(\d{2})(\d{2})-([0-9]{3})$/.exec(value);
if (!match || match[4] === "000") throw new Error("transition_id_invalid");
const year = Number(match[1]);
const month = Number(match[2]);
const day = Number(match[3]);
const parsed = new Date(Date.UTC(year, month - 1, day));
if (parsed.getUTCFullYear() !== year
|| parsed.getUTCMonth() !== month - 1
|| parsed.getUTCDate() !== day) {
throw new Error("transition_id_invalid");
}
if (previouslyIssuedTransitionIds.has(value)) {
throw new Error("transition_id_already_issued");
}
return value;
}
async function assertArtifactTargetFresh(path) {
try {
await lstat(path);
} catch (error) {
if (error?.code === "ENOENT") return;
throw error;
}
throw new Error("transition_artifact_already_exists");
}
function composeOverride() { function composeOverride() {
const health = "const http=require('http');const req=http.get('http://127.0.0.1:5678/healthz/readiness',r=>{r.resume();process.exit(r.statusCode===200?0:1)});req.on('error',()=>process.exit(1));req.setTimeout(4000,()=>{req.destroy();process.exit(1)});"; const health = "const http=require('http');const req=http.get('http://127.0.0.1:5678/healthz/readiness',r=>{r.resume();process.exit(r.statusCode===200?0:1)});req.on('error',()=>process.exit(1));req.setTimeout(4000,()=>{req.destroy();process.exit(1)});";
return [ return [

76
infra/deploy-runner/build-external-data-plane-artifact.mjs
View File

@ -1,14 +1,14 @@
#!/usr/bin/env node #!/usr/bin/env node
import { createHash } from "node:crypto"; import { createHash } from "node:crypto";
import { spawnSync } from "node:child_process"; import { spawnSync } from "node:child_process";
import { cp, lstat, mkdir, mkdtemp, readFile, readdir, rm, writeFile } from "node:fs/promises"; import { cp, lstat, mkdir, mkdtemp, readdir, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os"; import { tmpdir } from "node:os";
import { dirname, join, relative, resolve } from "node:path"; import { dirname, join, relative, resolve } from "node:path";
import { fileURLToPath } from "node:url"; import { fileURLToPath } from "node:url";
const scriptDir = dirname(fileURLToPath(import.meta.url)); const scriptDir = dirname(fileURLToPath(import.meta.url));
const platformRoot = resolve(scriptDir, "../.."); const platformRoot = resolve(scriptDir, "../..");
const artifactDir = resolve(process.env.NODEDC_DEPLOY_ARTIFACT_DIR || resolve(scriptDir, "../deploy-artifacts")); const artifactDir = resolve(scriptDir, "../deploy-artifacts");
const [patchId = "external-data-plane-20260714-001", ...extra] = process.argv.slice(2); const [patchId = "external-data-plane-20260714-001", ...extra] = process.argv.slice(2);
if (extra.length || !/^[A-Za-z0-9._-]{1,96}$/.test(patchId)) { if (extra.length || !/^[A-Za-z0-9._-]{1,96}$/.test(patchId)) {
@ -18,21 +18,13 @@ if (extra.length || !/^[A-Za-z0-9._-]{1,96}$/.test(patchId)) {
const files = [ const files = [
["infra/synology/docker-compose.external-data-plane.yml", "platform/docker-compose.external-data-plane.yml"], ["infra/synology/docker-compose.external-data-plane.yml", "platform/docker-compose.external-data-plane.yml"],
["services/external-data-plane", "platform/services/external-data-plane"], ["services/external-data-plane", "platform/services/external-data-plane"],
["packages/external-provider-contract/package.json", "platform/packages/external-provider-contract/package.json"], ["packages/external-provider-contract", "platform/packages/external-provider-contract"],
["packages/external-provider-contract/src/contract-version.mjs", "platform/packages/external-provider-contract/src/contract-version.mjs"],
["packages/external-provider-contract/src/data-plane.mjs", "platform/packages/external-provider-contract/src/data-plane.mjs"],
["packages/external-provider-contract/src/data-product.mjs", "platform/packages/external-provider-contract/src/data-product.mjs"],
["packages/external-provider-contract/src/intake-batch.mjs", "platform/packages/external-provider-contract/src/intake-batch.mjs"],
["packages/external-provider-contract/src/sensitive-field-policy.mjs", "platform/packages/external-provider-contract/src/sensitive-field-policy.mjs"],
]; ];
const ignoredBasenames = new Set([".DS_Store", ".git", "node_modules"]); const ignoredBasenames = new Set([".DS_Store", ".git", "node_modules"]);
const ignoredDirectoryNames = new Set(["test"]);
const stage = await mkdtemp(join(tmpdir(), "nodedc-external-data-plane-artifact-")); const stage = await mkdtemp(join(tmpdir(), "nodedc-external-data-plane-artifact-"));
const payload = join(stage, "payload"); const payload = join(stage, "payload");
const target = join(artifactDir, `nodedc-platform-${patchId}.tgz`); const target = join(artifactDir, `nodedc-platform-${patchId}.tgz`);
await assertSourceBoundary();
try { try {
await mkdir(payload, { recursive: true }); await mkdir(payload, { recursive: true });
for (const [sourceRelative, destinationRelative] of files) { for (const [sourceRelative, destinationRelative] of files) {
@ -42,66 +34,19 @@ try {
await writeFile(join(stage, "files.txt"), `${files.map(([, destination]) => destination).join("\n")}\n`, "utf8"); await writeFile(join(stage, "files.txt"), `${files.map(([, destination]) => destination).join("\n")}\n`, "utf8");
await mkdir(artifactDir, { recursive: true }); await mkdir(artifactDir, { recursive: true });
const tar = spawnSync("python3", ["-c", canonicalTarScript(), target, stage], { const tar = spawnSync("python3", ["-c", [
encoding: "utf8", "import sys, tarfile",
maxBuffer: 128 * 1024 * 1024, "with tarfile.open(sys.argv[1], 'w:gz', format=tarfile.PAX_FORMAT) as archive:",
}); " [archive.add(name, arcname=name, recursive=True) for name in ('manifest.env', 'files.txt', 'payload')]",
].join("\n"), target], { cwd: stage, encoding: "utf8" });
if (tar.status !== 0) throw new Error(`tar_failed:${tar.stderr || tar.stdout}`); if (tar.status !== 0) throw new Error(`tar_failed:${tar.stderr || tar.stdout}`);
const digest = createHash("sha256").update(await readFile(target)).digest("hex"); const digest = createHash("sha256").update(await (await import("node:fs/promises")).readFile(target)).digest("hex");
console.log(JSON.stringify({ console.log(JSON.stringify({ ok: true, patchId, artifact: target, sha256: digest }, null, 2));
ok: true,
patchId,
artifact: target,
sha256: digest,
entries: files.map(([, destination]) => destination),
excluded: [
".env*",
"node_modules",
"services/external-data-plane/test",
"private-key.pem",
"secrets",
],
}, null, 2));
} finally { } finally {
await rm(stage, { recursive: true, force: true }); await rm(stage, { recursive: true, force: true });
} }
async function assertSourceBoundary() {
const compose = await readFile(
resolve(platformRoot, "infra/synology/docker-compose.external-data-plane.yml"),
"utf8",
);
for (const fragment of [
"source: /volume1/docker/nodedc-platform/trust/engine-managed-provisioner",
"target: /run/nodedc-trust/engine-managed-provisioner",
"EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_PUBLIC_KEY_FILE: /run/nodedc-trust/engine-managed-provisioner/public-key.pem",
"create_host_path: false",
]) {
if (!compose.includes(fragment)) throw new Error(`platform_compose_boundary_missing:${fragment}`);
}
if (
compose.includes("private-key.pem")
|| compose.includes("ENGINE_EDP_MANAGED_PROVISIONER_PRIVATE_KEY_FILE")
) throw new Error("platform_artifact_private_key_boundary_violation");
}
function canonicalTarScript() {
return [
"import gzip, io, pathlib, sys, tarfile",
"root=pathlib.Path(sys.argv[2])",
"with open(sys.argv[1], 'wb') as out:",
" with gzip.GzipFile(filename='', mode='wb', fileobj=out, compresslevel=9, mtime=0) as gz:",
" with tarfile.open(fileobj=gz, mode='w', format=tarfile.PAX_FORMAT) as tar:",
" for top in ('manifest.env','files.txt','payload'):",
" p=root/top; paths=[p] + (sorted(p.rglob('*')) if p.is_dir() else [])",
" for x in paths:",
" info=tar.gettarinfo(str(x), arcname=x.relative_to(root).as_posix())",
" info.uid=info.gid=0; info.uname=info.gname='root'; info.mtime=0; info.mode=0o755 if info.isdir() else 0o644",
" with (open(x,'rb') if info.isfile() else io.BytesIO()) as src: tar.addfile(info, src if info.isfile() else None)",
].join("\n");
}
async function copySafe(source, destination) { async function copySafe(source, destination) {
const sourceStat = await lstat(source); const sourceStat = await lstat(source);
if (sourceStat.isSymbolicLink()) throw new Error(`source_symlink_rejected:${source}`); if (sourceStat.isSymbolicLink()) throw new Error(`source_symlink_rejected:${source}`);
@ -115,7 +60,6 @@ async function copySafe(source, destination) {
await mkdir(destination, { recursive: true }); await mkdir(destination, { recursive: true });
for (const entry of await readdir(source, { withFileTypes: true })) { for (const entry of await readdir(source, { withFileTypes: true })) {
if (ignoredBasenames.has(entry.name) || entry.name.startsWith(".env")) continue; if (ignoredBasenames.has(entry.name) || entry.name.startsWith(".env")) continue;
if (entry.isDirectory() && ignoredDirectoryNames.has(entry.name)) continue;
const childSource = join(source, entry.name); const childSource = join(source, entry.name);
const childDestination = join(destination, entry.name); const childDestination = join(destination, entry.name);
if (entry.isSymbolicLink()) throw new Error(`source_symlink_rejected:${relative(platformRoot, childSource)}`); if (entry.isSymbolicLink()) throw new Error(`source_symlink_rejected:${relative(platformRoot, childSource)}`);

2851
infra/deploy-runner/nodedc-deploy
View File

File diff suppressed because it is too large Load Diff

189
infra/deploy-runner/test_engine_agent_full_grant_migration_artifact.py
View File

@ -1,189 +0,0 @@
#!/usr/bin/env python3
import hashlib
import importlib.machinery
import importlib.util
import json
import os
import subprocess
import tarfile
import tempfile
import unittest
from pathlib import Path
from unittest import mock
SCRIPT_DIR = Path(__file__).resolve().parent
BUILDER = SCRIPT_DIR / "build-engine-agent-full-grant-migration-artifact.mjs"
RUNNER_PATH = SCRIPT_DIR / "nodedc-deploy"
PATCH_ID = "engine-agent-full-grant-migration-20990101-001"
STORE_REL = "nodedc-source/server/engineAgents/store.js"
PREDECESSOR_SHA256 = "52daa43499d6d9a97fe7ffa891edb9212b7791e733f91dd3ca686d42739b7e9a"
TARGET_SHA256 = "2e62654c2dc12905efcc83a9dff45a818dd7b47924a10600160835c4416540e9"
def load_runner():
loader = importlib.machinery.SourceFileLoader(
"nodedc_agent_grant_migration_runner_under_test",
str(RUNNER_PATH),
)
spec = importlib.util.spec_from_loader(loader.name, loader)
module = importlib.util.module_from_spec(spec)
loader.exec_module(module)
return module
RUNNER = load_runner()
class EngineAgentFullGrantMigrationArtifactTest(unittest.TestCase):
def build(self, artifact_dir, patch_id=PATCH_ID):
environment = os.environ.copy()
environment["NODEDC_DEPLOY_ARTIFACT_DIR"] = str(artifact_dir)
result = subprocess.run(
["node", str(BUILDER), patch_id],
check=True,
capture_output=True,
text=True,
env=environment,
)
return json.loads(result.stdout)
def test_artifact_is_exact_reproducible_and_runner_accepted(self):
with tempfile.TemporaryDirectory(prefix="nodedc-agent-grant-migration-") as directory:
root = Path(directory)
first_dir = root / "first"
second_dir = root / "second"
first_dir.mkdir()
second_dir.mkdir()
first = self.build(first_dir)
second = self.build(second_dir)
first_artifact = Path(first["artifact"])
second_artifact = Path(second["artifact"])
self.assertEqual(first["entries"], [STORE_REL])
self.assertEqual(first["predecessorSha256"], PREDECESSOR_SHA256)
self.assertEqual(first["targetSha256"], TARGET_SHA256)
self.assertEqual(first["sha256"], hashlib.sha256(first_artifact.read_bytes()).hexdigest())
self.assertEqual(first["sha256"], second["sha256"])
self.assertEqual(first_artifact.read_bytes(), second_artifact.read_bytes())
with tarfile.open(first_artifact, "r:gz") as archive:
members = archive.getmembers()
names = {member.name for member in members}
manifest = archive.extractfile("manifest.env").read().decode("utf-8")
files = archive.extractfile("files.txt").read().decode("utf-8")
store = archive.extractfile(f"payload/{STORE_REL}").read()
self.assertEqual(
manifest,
f"id={PATCH_ID}\ncomponent=engine\ntype=app-overlay\n",
)
self.assertEqual(files, f"{STORE_REL}\n")
self.assertEqual(hashlib.sha256(store).hexdigest(), TARGET_SHA256)
self.assertFalse(any(member.issym() or member.islnk() for member in members))
self.assertFalse(any(
name.startswith("payload/nodedc-source/server/data/")
or name.startswith("payload/nodedc-source/dist/")
or name.startswith("payload/nodedc-source/server/credentialSink/")
for name in names
))
with tempfile.TemporaryDirectory(prefix="nodedc-agent-grant-runner-") as work:
manifest_value, entries, _payload = RUNNER.load_artifact(
first_artifact,
Path(work),
)
self.assertEqual(manifest_value["id"], PATCH_ID)
self.assertEqual(tuple(entries), RUNNER.ENGINE_AGENT_FULL_GRANT_MIGRATION_ARTIFACT_ENTRIES)
self.assertTrue(RUNNER.is_engine_agent_full_grant_migration_slice("engine", entries))
self.assertEqual(RUNNER.component_services("engine", entries), ("nodedc-backend",))
self.assertEqual(RUNNER.component_builds("engine", entries), ())
self.assertEqual(
RUNNER.component_healthchecks("engine", entries, ("nodedc-backend",)),
(
"http://127.0.0.1:3001/health",
"http://127.0.0.1:3001/internal/engine-credential-sink/v1/health",
),
)
def test_predecessor_and_runtime_acceptance_are_exact(self):
with tempfile.TemporaryDirectory(prefix="nodedc-agent-grant-predecessor-") as directory:
root = Path(directory)
store = root / STORE_REL
store.parent.mkdir(parents=True)
store.write_text("predecessor", encoding="utf-8")
override = root / RUNNER.ENGINE_DATA_PRODUCT_PUBLISH_GRANT_OVERRIDE_REL
override.parent.mkdir(parents=True)
override.write_text(
RUNNER.expected_engine_data_product_publish_grant_override(),
encoding="utf-8",
)
original_root = RUNNER.COMPONENTS["engine"]["payload_root"]
RUNNER.COMPONENTS["engine"]["payload_root"] = root
try:
with mock.patch.object(RUNNER, "sha256_file", return_value=PREDECESSOR_SHA256):
self.assertEqual(
RUNNER.preflight_engine_agent_full_grant_migration_predecessor(),
PREDECESSOR_SHA256,
)
with mock.patch.object(RUNNER, "sha256_file", return_value="0" * 64):
with self.assertRaisesRegex(RUNNER.DeployError, "predecessor drift"):
RUNNER.preflight_engine_agent_full_grant_migration_predecessor()
finally:
RUNNER.COMPONENTS["engine"]["payload_root"] = original_root
expected = f"store-v2-full-developer:{TARGET_SHA256}"
with mock.patch.object(
RUNNER,
"engine_backend_container_id",
return_value="container-id",
):
with mock.patch.object(
RUNNER,
"run_engine_backend_probe",
return_value=expected,
) as probe:
self.assertEqual(RUNNER.accept_engine_agent_full_grant_migration_runtime(), expected)
arguments, label = probe.call_args.args[:2]
self.assertEqual(label, "agent full grant migration")
self.assertEqual(arguments[0:2], ("node", "--input-type=module"))
self.assertIn(TARGET_SHA256, arguments)
self.assertEqual(probe.call_args.kwargs["container_id"], "container-id")
def test_builder_rejects_invalid_id_and_overwrite(self):
with tempfile.TemporaryDirectory(prefix="nodedc-agent-grant-negative-") as directory:
root = Path(directory)
environment = {**os.environ, "NODEDC_DEPLOY_ARTIFACT_DIR": str(root)}
invalid = subprocess.run(
["node", str(BUILDER), "engine-data-product-publish-grant-20990101-001"],
check=False,
capture_output=True,
text=True,
env=environment,
)
self.assertNotEqual(invalid.returncode, 0)
self.assertIn("fresh-patch-id", invalid.stderr)
issued = subprocess.run(
["node", str(BUILDER), "engine-agent-full-grant-migration-20260717-001"],
check=False,
capture_output=True,
text=True,
env=environment,
)
self.assertNotEqual(issued.returncode, 0)
self.assertIn("patch_id_already_issued", issued.stderr)
self.build(root)
duplicate = subprocess.run(
["node", str(BUILDER), PATCH_ID],
check=False,
capture_output=True,
text=True,
env=environment,
)
self.assertNotEqual(duplicate.returncode, 0)
self.assertIn("artifact_already_exists", duplicate.stderr)
if __name__ == "__main__":
unittest.main(verbosity=2)

147
infra/deploy-runner/test_engine_credential_sink_recovery_artifact.py
View File

@ -1,147 +0,0 @@
#!/usr/bin/env python3
import hashlib
import importlib.machinery
import importlib.util
import json
import os
import shutil
import subprocess
import tarfile
import tempfile
import unittest
from pathlib import Path
SCRIPT_DIR = Path(__file__).resolve().parent
BUILDER = SCRIPT_DIR / "build-engine-credential-sink-recovery-artifact.mjs"
RUNNER_PATH = SCRIPT_DIR / "nodedc-deploy"
SOURCE = Path("/Volumes/docker/nodedc-deploy/applied/nodedc-engine-credential-sink-20260716-001.tgz")
SOURCE_SHA256 = "0a96add05fe59db8f490927f66e07a84490474a7afb3ef7de51e1d6fd96f86a2"
PATCH_ID = "engine-credential-sink-recovery-20990101-001"
def load_runner():
loader = importlib.machinery.SourceFileLoader(
"nodedc_recovery_runner_under_test",
str(RUNNER_PATH),
)
spec = importlib.util.spec_from_loader(loader.name, loader)
module = importlib.util.module_from_spec(spec)
loader.exec_module(module)
return module
RUNNER = load_runner()
class EngineCredentialSinkRecoveryArtifactTest(unittest.TestCase):
def build(self, artifact_dir, patch_id=PATCH_ID, source=SOURCE):
environment = os.environ.copy()
environment["NODEDC_DEPLOY_ARTIFACT_DIR"] = str(artifact_dir)
environment["NODEDC_ENGINE_CREDENTIAL_SINK_RECOVERY_SOURCE"] = str(source)
result = subprocess.run(
["node", str(BUILDER), patch_id],
check=True,
capture_output=True,
text=True,
env=environment,
)
return json.loads(result.stdout)
def test_recovery_changes_only_manifest_id_and_passes_runner_policy(self):
self.assertTrue(SOURCE.is_file())
self.assertEqual(hashlib.sha256(SOURCE.read_bytes()).hexdigest(), SOURCE_SHA256)
with tempfile.TemporaryDirectory(prefix="nodedc-engine-sink-recovery-") as directory:
root = Path(directory)
first_dir = root / "first"
second_dir = root / "second"
first_dir.mkdir()
second_dir.mkdir()
first = self.build(first_dir)
second = self.build(second_dir)
first_artifact = Path(first["artifact"])
second_artifact = Path(second["artifact"])
self.assertEqual(first["sourceArtifactSha256"], SOURCE_SHA256)
self.assertEqual(first["changed"], ["manifest.env:id"])
self.assertEqual(first["sha256"], second["sha256"])
self.assertEqual(first_artifact.read_bytes(), second_artifact.read_bytes())
with tarfile.open(SOURCE, "r:gz") as old, tarfile.open(first_artifact, "r:gz") as new:
old_files = old.extractfile("files.txt").read()
new_files = new.extractfile("files.txt").read()
self.assertEqual(old_files, new_files)
self.assertEqual(
new.extractfile("manifest.env").read().decode("utf-8"),
f"id={PATCH_ID}\ncomponent=engine\ntype=app-overlay\n",
)
for relative_path, expected_sha256 in first["payloadSha256"].items():
old_bytes = old.extractfile(f"payload/{relative_path}").read()
new_bytes = new.extractfile(f"payload/{relative_path}").read()
self.assertEqual(old_bytes, new_bytes)
self.assertEqual(hashlib.sha256(new_bytes).hexdigest(), expected_sha256)
with tempfile.TemporaryDirectory(prefix="nodedc-recovery-runner-") as work:
manifest, entries, _payload = RUNNER.load_artifact(
first_artifact,
Path(work),
)
self.assertEqual(manifest["id"], PATCH_ID)
self.assertEqual(tuple(entries), RUNNER.ENGINE_CREDENTIAL_SINK_ARTIFACT_ENTRIES)
self.assertEqual(RUNNER.component_services("engine", entries), ("nodedc-backend",))
def test_recovery_rejects_tampered_source_invalid_id_and_overwrite(self):
with tempfile.TemporaryDirectory(prefix="nodedc-engine-sink-recovery-negative-") as directory:
root = Path(directory)
tampered = root / "tampered.tgz"
shutil.copyfile(SOURCE, tampered)
with tampered.open("ab") as stream:
stream.write(b"tampered")
environment = os.environ.copy()
environment["NODEDC_DEPLOY_ARTIFACT_DIR"] = str(root / "artifacts")
environment["NODEDC_ENGINE_CREDENTIAL_SINK_RECOVERY_SOURCE"] = str(tampered)
result = subprocess.run(
["node", str(BUILDER), PATCH_ID],
check=False,
capture_output=True,
text=True,
env=environment,
)
self.assertNotEqual(result.returncode, 0)
self.assertIn("source_sha256_mismatch", result.stderr)
invalid = subprocess.run(
["node", str(BUILDER), "engine-credential-sink-20260716-001"],
check=False,
capture_output=True,
text=True,
env={
**os.environ,
"NODEDC_DEPLOY_ARTIFACT_DIR": str(root / "invalid"),
"NODEDC_ENGINE_CREDENTIAL_SINK_RECOVERY_SOURCE": str(SOURCE),
},
)
self.assertNotEqual(invalid.returncode, 0)
self.assertIn("fresh-recovery-patch-id", invalid.stderr)
artifact_dir = root / "duplicate"
artifact_dir.mkdir()
self.build(artifact_dir)
duplicate = subprocess.run(
["node", str(BUILDER), PATCH_ID],
check=False,
capture_output=True,
text=True,
env={
**os.environ,
"NODEDC_DEPLOY_ARTIFACT_DIR": str(artifact_dir),
"NODEDC_ENGINE_CREDENTIAL_SINK_RECOVERY_SOURCE": str(SOURCE),
},
)
self.assertNotEqual(duplicate.returncode, 0)
self.assertIn("artifact_already_exists", duplicate.stderr)
if __name__ == "__main__":
unittest.main(verbosity=2)

323
infra/deploy-runner/test_engine_n8n_private_extension.py
View File

@ -1,8 +1,6 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
import importlib.machinery import importlib.machinery
import importlib.util import importlib.util
import hashlib
import inspect
import json import json
import os import os
import subprocess import subprocess
@ -10,7 +8,6 @@ import tarfile
import tempfile import tempfile
import unittest import unittest
from pathlib import Path from pathlib import Path
from unittest import mock
SCRIPT_DIR = Path(__file__).resolve().parent SCRIPT_DIR = Path(__file__).resolve().parent
@ -23,18 +20,6 @@ STAGE_ARTIFACT = (
/ "infra/deploy-artifacts" / "infra/deploy-artifacts"
/ "nodedc-n8n-private-extension-n8n-nodes-ndc-release-20260716-003.tgz" / "nodedc-n8n-private-extension-n8n-nodes-ndc-release-20260716-003.tgz"
) )
TRANSITION_ID = "20260717-004"
SEALED_RELEASE_ID = "0.1.2-05e4b38b14b4a019"
SEALED_PACKAGE_SHA256 = "05e4b38b14b4a019ce1f6eee27b9e320094cb3560903bd68074966b3a1267af5"
BUILDER_ENGINE_PATHS = (
"nodedc-source/server/assets/n8n/schema/v2.3.2/nodes.catalog.json",
"nodedc-source/server/assets/n8n/schema/v2.3.2/credentials.catalog.json",
"nodedc-source/server/assets/n8n/schema/v2.3.2/meta.json",
"nodedc-source/server/assets/n8n/icons/ndc.svg",
"nodedc-source/server/assets/n8n/icons/ndc.dark.svg",
"nodedc-source/services/n8n/private-extensions/ndc-activation.json",
"nodedc-source/services/n8n/private-extensions/docker-compose.ndc-private-extension.yml",
)
EXPECTED_NODES = [ EXPECTED_NODES = [
"n8n-nodes-ndc.ndcDataProductPublish", "n8n-nodes-ndc.ndcDataProductPublish",
"n8n-nodes-ndc.ndcDataProductRead", "n8n-nodes-ndc.ndcDataProductRead",
@ -45,14 +30,6 @@ EXPECTED_CREDENTIALS = [
"ndcDataProductReaderApi", "ndcDataProductReaderApi",
"ndcFoundryBindingApi", "ndcFoundryBindingApi",
] ]
PRIVATE_EXTENSION_CANON_FUNCTION_SHA256 = {
"expected_engine_n8n_compose_override": "8ee93f3e7c407f5557c711119236449a440d73a7fcac1bbfa50a09e6a13c1cff",
"engine_n8n_package_loader_probe_script": "2b3b3c96fad6e5e48ebea74426b21bc7eea6b6fe0e786b537da3366d18aedd9b",
"engine_n8n_private_loader_catalog": "a2ae389b4ef215900cf967b863d01056bd2a8eec55554566d0f6005e4e0bcbac",
"validate_engine_n8n_base_compose_source": "adacc5b20e52684cbc427362ddba5a6d2e13ef2091a89859e2b8f7bbf2e20458",
"preflight_engine_n8n_transition": "da3cec853e3e3e20df4d1e301aab98b77815f93ec3d68b0b61c84c6e6a3fb335",
"accept_engine_n8n_runtime": "1a2614a465161e3833e84aca402817682da1a538bdb2aecc4af66b665497338d",
}
def load_runner(): def load_runner():
@ -63,32 +40,6 @@ def load_runner():
return module return module
def snapshot_engine_builder_paths():
snapshot = {}
for relative in BUILDER_ENGINE_PATHS:
path = ENGINE_ROOT / relative
try:
path_stat = path.lstat()
except FileNotFoundError:
snapshot[relative] = None
continue
if path.is_symlink():
content = ("symlink", os.readlink(path))
elif path.is_file():
content = ("file", path.read_bytes())
else:
content = ("other", None)
snapshot[relative] = (
path_stat.st_mode,
path_stat.st_uid,
path_stat.st_gid,
path_stat.st_ino,
path_stat.st_mtime_ns,
content,
)
return snapshot
RUNNER = load_runner() RUNNER = load_runner()
@ -98,21 +49,14 @@ class EngineN8nPrivateExtensionTest(unittest.TestCase):
cls.temporary = tempfile.TemporaryDirectory(prefix="nodedc-engine-n8n-policy-") cls.temporary = tempfile.TemporaryDirectory(prefix="nodedc-engine-n8n-policy-")
cls.root = Path(cls.temporary.name) cls.root = Path(cls.temporary.name)
cls.results = [] cls.results = []
cls.engine_snapshot_before = snapshot_engine_builder_paths()
for index in range(2): for index in range(2):
output = cls.root / f"build-{index}" output = cls.root / f"build-{index}"
output.mkdir() output.mkdir()
env = os.environ.copy() env = os.environ.copy()
env.pop("NODEDC_N8N_TRANSITION_ID", None)
env["NODEDC_DEPLOY_ARTIFACT_DIR"] = str(output) env["NODEDC_DEPLOY_ARTIFACT_DIR"] = str(output)
env["NODEDC_N8N_EXTENSION_STAGE_ARTIFACT"] = str(STAGE_ARTIFACT) env["NODEDC_N8N_EXTENSION_STAGE_ARTIFACT"] = str(STAGE_ARTIFACT)
command = ["node", str(BUILDER_PATH)]
if index == 0:
command.append(TRANSITION_ID)
else:
env["NODEDC_N8N_TRANSITION_ID"] = TRANSITION_ID
result = subprocess.run( result = subprocess.run(
command, ["node", str(BUILDER_PATH)],
cwd=PLATFORM_ROOT, cwd=PLATFORM_ROOT,
env=env, env=env,
check=True, check=True,
@ -120,7 +64,6 @@ class EngineN8nPrivateExtensionTest(unittest.TestCase):
text=True, text=True,
) )
cls.results.append(json.loads(result.stdout)) cls.results.append(json.loads(result.stdout))
cls.engine_snapshot_after = snapshot_engine_builder_paths()
@classmethod @classmethod
def tearDownClass(cls): def tearDownClass(cls):
@ -138,69 +81,6 @@ class EngineN8nPrivateExtensionTest(unittest.TestCase):
self.assertEqual(first[4:8], b"\0\0\0\0") self.assertEqual(first[4:8], b"\0\0\0\0")
self.assertEqual(first[3] & 0x08, 0) self.assertEqual(first[3] & 0x08, 0)
def test_successful_generation_003_runner_functions_are_frozen(self):
actual = {
name: hashlib.sha256(inspect.getsource(getattr(RUNNER, name)).encode("utf-8")).hexdigest()
for name in PRIVATE_EXTENSION_CANON_FUNCTION_SHA256
}
self.assertEqual(actual, PRIVATE_EXTENSION_CANON_FUNCTION_SHA256)
def test_explicit_transition_id_does_not_mutate_sealed_release_identity(self):
expected_ids = {
"activation": f"engine-n8n-private-extension-{TRANSITION_ID}",
"rollback": f"engine-n8n-private-extension-rollback-{TRANSITION_ID}",
}
for result in self.results:
self.assertEqual(result["transitionId"], TRANSITION_ID)
self.assertEqual(result["releaseId"], SEALED_RELEASE_ID)
self.assertEqual(result["packageSha256"], SEALED_PACKAGE_SHA256)
for kind, expected_id in expected_ids.items():
self.assertEqual(result[kind]["id"], expected_id)
self.assertEqual(Path(result[kind]["artifact"]).name, f"nodedc-{expected_id}.tgz")
def test_builder_does_not_mutate_engine_source(self):
self.assertEqual(self.engine_snapshot_after, self.engine_snapshot_before)
def test_transition_id_rejects_missing_invalid_and_previously_issued_values(self):
cases = (
([], "transition_id_required"),
(["20260716-003"], "transition_id_already_issued"),
(["engine-n8n-private-extension-20260717-004"], "transition_id_invalid"),
(["20260230-004"], "transition_id_invalid"),
(["20260717-000"], "transition_id_invalid"),
([TRANSITION_ID, "unexpected-second-id"], "transition_id_argument_count_invalid"),
)
for arguments, expected_error in cases:
with self.subTest(arguments=arguments):
env = os.environ.copy()
env.pop("NODEDC_N8N_TRANSITION_ID", None)
result = subprocess.run(
["node", str(BUILDER_PATH), *arguments],
cwd=PLATFORM_ROOT,
env=env,
check=False,
capture_output=True,
text=True,
)
self.assertNotEqual(result.returncode, 0)
self.assertIn(expected_error, result.stderr)
def test_existing_transition_artifact_is_never_overwritten(self):
env = os.environ.copy()
env.pop("NODEDC_N8N_TRANSITION_ID", None)
env["NODEDC_DEPLOY_ARTIFACT_DIR"] = str(self.artifact(0, "activation").parent)
env["NODEDC_N8N_EXTENSION_STAGE_ARTIFACT"] = str(STAGE_ARTIFACT)
result = subprocess.run(
["node", str(BUILDER_PATH), TRANSITION_ID],
cwd=PLATFORM_ROOT,
env=env,
check=False,
capture_output=True,
text=True,
)
self.assertNotEqual(result.returncode, 0)
self.assertIn("transition_artifact_already_exists", result.stderr)
def test_activation_and_rollback_pass_strict_runner_policy(self): def test_activation_and_rollback_pass_strict_runner_policy(self):
expected = {"activation": ("activate", 7), "rollback": ("rollback-inactive", 4)} expected = {"activation": ("activate", 7), "rollback": ("rollback-inactive", 4)}
for kind, (action, entry_count) in expected.items(): for kind, (action, entry_count) in expected.items():
@ -257,213 +137,12 @@ class EngineN8nPrivateExtensionTest(unittest.TestCase):
) )
override = (payload / RUNNER.ENGINE_N8N_COMPOSE_OVERRIDE_REL).read_text() override = (payload / RUNNER.ENGINE_N8N_COMPOSE_OVERRIDE_REL).read_text()
self.assertEqual(override, RUNNER.expected_engine_n8n_compose_override(descriptor)) self.assertEqual(override, RUNNER.expected_engine_n8n_compose_override(descriptor))
self.assertEqual(
hashlib.sha256(override.encode("utf-8")).hexdigest(),
"a29e2f92b87dda59da2b4e3ce250bc9705ed9fa551c0fbe4e95464fca5caa3e1",
)
self.assertIn("pull_policy: never", override) self.assertIn("pull_policy: never", override)
self.assertIn("N8N_USER_FOLDER: /home/node", override) self.assertIn("N8N_USER_FOLDER: /home/node", override)
self.assertNotIn("NODE_PATH", override)
self.assertIn(":/home/node/.n8n/nodes/node_modules/n8n-nodes-ndc:ro", override) self.assertIn(":/home/node/.n8n/nodes/node_modules/n8n-nodes-ndc:ro", override)
self.assertNotIn("N8N_CUSTOM_EXTENSIONS", override) self.assertNotIn("N8N_CUSTOM_EXTENSIONS", override)
self.assertNotIn("build:", override) self.assertNotIn("build:", override)
def test_generation_003_override_is_accepted_as_installed_canon(self):
original_root = RUNNER.COMPONENTS["engine"]["payload_root"]
original_compose = RUNNER.COMPONENTS["engine"]["compose_root"]
try:
with tempfile.TemporaryDirectory() as directory:
work = Path(directory)
extracted = work / "artifact"
extracted.mkdir()
_manifest, _entries, payload = RUNNER.load_artifact(
self.artifact(0, "activation"),
extracted,
)
engine_root = work / "engine"
engine_root.mkdir()
(engine_root / "docker-compose.yml").write_text("services:\n", encoding="utf-8")
for relative in (
RUNNER.ENGINE_N8N_TRANSITION_DESCRIPTOR_REL,
RUNNER.ENGINE_N8N_COMPOSE_OVERRIDE_REL,
):
target = engine_root / relative
target.parent.mkdir(parents=True, exist_ok=True)
target.write_bytes((payload / relative).read_bytes())
RUNNER.COMPONENTS["engine"]["payload_root"] = engine_root
RUNNER.COMPONENTS["engine"]["compose_root"] = engine_root
compose_files = RUNNER.component_compose_files("engine")
self.assertEqual(compose_files, (
engine_root / "docker-compose.yml",
engine_root / RUNNER.ENGINE_N8N_COMPOSE_OVERRIDE_REL,
))
finally:
RUNNER.COMPONENTS["engine"]["payload_root"] = original_root
RUNNER.COMPONENTS["engine"]["compose_root"] = original_compose
def test_loader_probe_pins_n8n_node_path(self):
container = {
"Mounts": [{
"Destination": RUNNER.ENGINE_N8N_RUNTIME_PACKAGE_PATH,
"RW": False,
}],
}
completed = subprocess.CompletedProcess(
args=[],
returncode=0,
stdout=json.dumps({
"packageName": "n8n-nodes-ndc",
"nodes": ["ndcDataProductPublish", "ndcDataProductRead", "ndcFoundryBinding"],
"credentials": EXPECTED_CREDENTIALS,
}),
stderr="",
)
with (
mock.patch.object(RUNNER, "inspect_container", return_value=container),
mock.patch.object(RUNNER.subprocess, "run", return_value=completed) as run,
):
catalog = RUNNER.engine_n8n_private_loader_catalog("n8n-container")
command = run.call_args.args[0]
self.assertEqual(command[:-1], [
str(RUNNER.DOCKER),
"exec",
"n8n-container",
"node",
"-e",
])
probe_script = command[-1]
self.assertIn(
f"process.env.NODE_PATH='{RUNNER.ENGINE_N8N_NODE_MODULES_PATH}'",
probe_script,
)
self.assertIn("Module._initPaths()", probe_script)
self.assertIn("PackageDirectoryLoader", probe_script)
self.assertEqual(catalog, {
"node_types": EXPECTED_NODES,
"credential_types": EXPECTED_CREDENTIALS,
})
def test_loader_probe_failure_emits_only_allowlisted_diagnostic(self):
container = {
"Mounts": [{
"Destination": RUNNER.ENGINE_N8N_RUNTIME_PACKAGE_PATH,
"RW": False,
}],
}
cases = (
("MODULE_NOT_FOUND", "MODULE_NOT_FOUND"),
("Bearer secret-that-must-not-be-logged", "PROBE_ERROR"),
)
for probe_stderr, expected_category in cases:
with (
self.subTest(stderr=probe_stderr),
mock.patch.object(RUNNER, "inspect_container", return_value=container),
mock.patch.object(
RUNNER.subprocess,
"run",
return_value=subprocess.CompletedProcess(
args=[],
returncode=1,
stdout="",
stderr=probe_stderr,
),
),
):
with self.assertRaises(RUNNER.DeployError) as raised:
RUNNER.engine_n8n_private_loader_catalog("n8n-container")
message = str(raised.exception)
self.assertIn(f"category={expected_category}", message)
self.assertNotIn("secret-that-must-not-be-logged", message)
def test_active_runtime_acceptance_does_not_require_node_path(self):
with tempfile.TemporaryDirectory() as directory:
_manifest, _entries, payload = RUNNER.load_artifact(
self.artifact(0, "activation"),
Path(directory),
)
descriptor = RUNNER.read_engine_n8n_transition_descriptor(
payload / RUNNER.ENGINE_N8N_TRANSITION_DESCRIPTOR_REL
)
container = {
"State": {"Status": "running", "StartedAt": "2026-07-16T00:00:00Z"},
"Image": "sha256:base-image",
"Config": {
"Env": [
"N8N_USER_FOLDER=/home/node",
"N8N_COMMUNITY_PACKAGES_ENABLED=true",
"N8N_COMMUNITY_PACKAGES_PREVENT_LOADING=false",
"N8N_REINSTALL_MISSING_PACKAGES=false",
],
"Labels": {
"nodedc.n8n-private-extension.release": descriptor["releaseId"],
"nodedc.n8n-private-extension.package-sha256": descriptor["packageSha256"],
},
},
"Mounts": [{
"Destination": RUNNER.ENGINE_N8N_RUNTIME_PACKAGE_PATH,
"Source": f"/volume2/nodedc-demo/{descriptor['sealedReleaseRelativePath']}",
"RW": False,
}],
"RestartCount": 0,
}
log_result = subprocess.CompletedProcess(args=[], returncode=0, stdout="", stderr="")
with (
mock.patch.object(
RUNNER,
"inspect_engine_n8n_base_image",
return_value={"id": "sha256:base-image"},
),
mock.patch.object(RUNNER, "engine_n8n_container_ids", return_value=("n8n-container",)),
mock.patch.object(RUNNER, "inspect_container", return_value=container),
mock.patch.object(RUNNER, "wait_engine_n8n_readiness"),
mock.patch.object(RUNNER, "engine_n8n_version", return_value=RUNNER.ENGINE_N8N_VERSION),
mock.patch.object(RUNNER.subprocess, "run", return_value=log_result),
mock.patch.object(
RUNNER,
"engine_n8n_private_loader_catalog",
return_value={
"node_types": EXPECTED_NODES,
"credential_types": EXPECTED_CREDENTIALS,
},
),
mock.patch.object(RUNNER, "validate_engine_n8n_catalog_payload"),
mock.patch.object(RUNNER.time, "sleep"),
):
accepted = RUNNER.accept_engine_n8n_runtime(descriptor)
self.assertEqual(accepted["node_types"], EXPECTED_NODES)
def test_private_extension_transition_does_not_enter_publish_grant_domain(self):
entries = (RUNNER.ENGINE_N8N_TRANSITION_DESCRIPTOR_REL,)
descriptor = {"action": "rollback-inactive"}
self.assertFalse(RUNNER.touches_engine_data_product_publish_grant(entries))
self.assertEqual(RUNNER.component_services("engine", entries), ("n8n",))
with (
mock.patch.object(RUNNER, "current_engine_n8n_transition_descriptor", return_value=descriptor),
mock.patch.object(RUNNER, "ensure_engine_edp_managed_provisioner_keypair") as keypair,
mock.patch.object(RUNNER, "ensure_engine_publish_grant_private_state") as private_state,
):
RUNNER.prepare_component_runtime("engine", entries)
keypair.assert_not_called()
private_state.assert_not_called()
def test_generic_engine_compose_artifact_keeps_legacy_service_selection(self):
self.assertEqual(
RUNNER.component_services("engine", ("docker-compose.yml",)),
RUNNER.COMPONENTS["engine"]["services"],
)
self.assertEqual(
RUNNER.component_services(
"engine",
("docker-compose.yml", "nodedc-source/server/dataProductPublishGrant/store.js"),
),
RUNNER.COMPONENTS["engine"]["services"],
)
self.assertFalse(RUNNER.is_engine_data_product_publish_grant_slice(
"engine",
("docker-compose.yml", "nodedc-source/server/dataProductPublishGrant/store.js"),
))
def test_unknown_descriptor_key_is_rejected(self): def test_unknown_descriptor_key_is_rejected(self):
with tempfile.TemporaryDirectory() as directory: with tempfile.TemporaryDirectory() as directory:
work = Path(directory) work = Path(directory)

156
infra/deploy-runner/test_engine_publish_grant_artifact.py
View File

@ -1,156 +0,0 @@
#!/usr/bin/env python3
import hashlib
import json
import os
import subprocess
import tarfile
import tempfile
import unittest
from pathlib import Path
SCRIPT_DIR = Path(__file__).resolve().parent
BUILDER = SCRIPT_DIR / "build-engine-data-product-publish-grant-artifact.mjs"
EXPECTED_ENTRIES = [
"nodedc-source/server/assets/provider-packages/v1/catalog.json",
"nodedc-source/server/dataProductPublishGrant",
"nodedc-source/server/engineAgents/store.js",
"nodedc-source/server/routes/engineAgentGateway.js",
"nodedc-source/server/routes/n8n.js",
"nodedc-source/services/backend/data-product-publish-grant/docker-compose.immutable-runtime.yml",
]
PATCH_ID = "engine-data-product-publish-grant-20990101-001"
SUCCESSFUL_CREDENTIAL_SINK_INDEX_SHA256 = (
"b2b790b02839570d967a2ca68b00e2724485a99389ac9b3a589a1b22302a36b8"
)
class EnginePublishGrantArtifactTest(unittest.TestCase):
def build(self, artifact_dir, patch_id):
environment = os.environ.copy()
environment["NODEDC_DEPLOY_ARTIFACT_DIR"] = str(artifact_dir)
result = subprocess.run(
["node", str(BUILDER), patch_id],
check=True,
capture_output=True,
text=True,
env=environment,
)
return json.loads(result.stdout)
def test_artifact_is_narrow_secret_free_and_deterministic(self):
with tempfile.TemporaryDirectory(prefix="nodedc-engine-publish-artifact-") as directory:
root = Path(directory)
first_dir = root / "first"
second_dir = root / "second"
first_dir.mkdir()
second_dir.mkdir()
first = self.build(first_dir, PATCH_ID)
artifact = Path(first["artifact"])
first_bytes = artifact.read_bytes()
second = self.build(second_dir, PATCH_ID)
second_bytes = Path(second["artifact"]).read_bytes()
self.assertEqual(first["entries"], EXPECTED_ENTRIES)
self.assertEqual(first["sha256"], hashlib.sha256(first_bytes).hexdigest())
self.assertEqual(first["sha256"], second["sha256"])
self.assertEqual(first_bytes, second_bytes)
with tarfile.open(artifact, "r:gz") as archive:
names = {member.name for member in archive.getmembers()}
files = archive.extractfile("files.txt").read().decode("utf-8").splitlines()
manifest = archive.extractfile("manifest.env").read().decode("utf-8")
n8n_source = archive.extractfile(
"payload/nodedc-source/server/routes/n8n.js"
).read().decode("utf-8")
gateway_source = archive.extractfile(
"payload/nodedc-source/server/routes/engineAgentGateway.js"
).read().decode("utf-8")
runtime_override = archive.extractfile(
"payload/nodedc-source/services/backend/data-product-publish-grant/"
"docker-compose.immutable-runtime.yml"
).read().decode("utf-8")
self.assertEqual(files, EXPECTED_ENTRIES)
self.assertEqual(
manifest,
f"id={PATCH_ID}\ncomponent=engine\ntype=app-overlay\n",
)
forbidden_roots = (
"payload/docker-compose.yml",
"payload/nodedc-source/server/index.js",
"payload/nodedc-source/server/credentialPolicies/ndcPrivateNode.js",
"payload/nodedc-source/server/credentialSink/",
"payload/nodedc-source/server/routes/engineCredentialSink.js",
"payload/nodedc-source/server/middleware/demoAccess.js",
"payload/nodedc-source/server/routes/ndcAgentMcp.js",
"payload/nodedc-source/server/tests/",
"payload/nodedc-source/dist/",
"payload/nodedc-source/server/data/",
)
self.assertFalse(any(
name == root.rstrip("/") or name.startswith(root)
for name in names
for root in forbidden_roots
))
self.assertNotIn("from '../credentialSink/", n8n_source)
self.assertIn("engineCredentialSinkN8nAdapter", n8n_source)
self.assertIn("engineDataProductPublishGrantN8nAdapter", n8n_source)
for tool in (
"engine_plan_data_product_publish_grant",
"engine_apply_data_product_publish_grant",
"engine_accept_data_product_publish_grant",
"engine_rollback_data_product_publish_grant",
):
self.assertIn(tool, gateway_source)
self.assertIn(" nodedc-backend:", runtime_override)
self.assertNotIn(" n8n:", runtime_override)
self.assertIn(
"source: /volume2/nodedc-demo/nodedc-control-plane/publish-grants",
runtime_override,
)
self.assertNotIn(
"source: /volume2/nodedc-demo/nodedc-control-plane\n",
runtime_override,
)
self.assertIn("read_only: true", runtime_override)
self.assertEqual(runtime_override.count("create_host_path: false"), 2)
def test_builder_requires_a_fresh_never_issued_patch_id(self):
with tempfile.TemporaryDirectory(prefix="nodedc-engine-publish-id-") as directory:
artifact_dir = Path(directory)
environment = os.environ.copy()
environment["NODEDC_DEPLOY_ARTIFACT_DIR"] = str(artifact_dir)
cases = (
([], "fresh-patch-id"),
(["engine-data-product-publish-grant-20260716-001"], "patch_id_already_issued"),
(["engine-data-product-publish-grant-20260717-001"], "patch_id_already_issued"),
(["engine-data-product-publish-grant-20260717-002"], "patch_id_already_issued"),
(["engine-publish-grant-unit-001"], "fresh-patch-id"),
)
for arguments, expected in cases:
with self.subTest(arguments=arguments):
result = subprocess.run(
["node", str(BUILDER), *arguments],
check=False,
capture_output=True,
text=True,
env=environment,
)
self.assertNotEqual(result.returncode, 0)
self.assertIn(expected, result.stderr)
self.build(artifact_dir, PATCH_ID)
duplicate = subprocess.run(
["node", str(BUILDER), PATCH_ID],
check=False,
capture_output=True,
text=True,
env=environment,
)
self.assertNotEqual(duplicate.returncode, 0)
self.assertIn("artifact_already_exists", duplicate.stderr)
if __name__ == "__main__":
unittest.main(verbosity=2)

88
infra/deploy-runner/test_external_data_plane_artifact.py
View File

@ -1,88 +0,0 @@
#!/usr/bin/env python3
import hashlib
import json
import os
import subprocess
import tarfile
import tempfile
import unittest
from pathlib import Path
SCRIPT_DIR = Path(__file__).resolve().parent
BUILDER = SCRIPT_DIR / "build-external-data-plane-artifact.mjs"
EXPECTED_ENTRIES = [
"platform/docker-compose.external-data-plane.yml",
"platform/services/external-data-plane",
"platform/packages/external-provider-contract/package.json",
"platform/packages/external-provider-contract/src/contract-version.mjs",
"platform/packages/external-provider-contract/src/data-plane.mjs",
"platform/packages/external-provider-contract/src/data-product.mjs",
"platform/packages/external-provider-contract/src/intake-batch.mjs",
"platform/packages/external-provider-contract/src/sensitive-field-policy.mjs",
]
class ExternalDataPlaneArtifactTest(unittest.TestCase):
def build(self, artifact_dir, patch_id):
environment = os.environ.copy()
environment["NODEDC_DEPLOY_ARTIFACT_DIR"] = str(artifact_dir)
result = subprocess.run(
["node", str(BUILDER), patch_id],
check=True,
capture_output=True,
text=True,
env=environment,
)
return json.loads(result.stdout)
def test_artifact_is_runtime_only_public_trust_and_deterministic(self):
with tempfile.TemporaryDirectory(prefix="nodedc-edp-artifact-") as directory:
artifact_dir = Path(directory)
first = self.build(artifact_dir, "edp-managed-publish-unit-001")
artifact = Path(first["artifact"])
first_bytes = artifact.read_bytes()
second = self.build(artifact_dir, "edp-managed-publish-unit-001")
second_bytes = Path(second["artifact"]).read_bytes()
self.assertEqual(first["entries"], EXPECTED_ENTRIES)
self.assertEqual(first["sha256"], hashlib.sha256(first_bytes).hexdigest())
self.assertEqual(first["sha256"], second["sha256"])
self.assertEqual(first_bytes, second_bytes)
with tarfile.open(artifact, "r:gz") as archive:
members = archive.getmembers()
names = {member.name for member in members}
files = archive.extractfile("files.txt").read().decode("utf-8").splitlines()
manifest = archive.extractfile("manifest.env").read().decode("utf-8")
compose = archive.extractfile(
"payload/platform/docker-compose.external-data-plane.yml"
).read().decode("utf-8")
regular_payloads = [
archive.extractfile(member).read()
for member in members
if member.isfile()
]
self.assertEqual(files, EXPECTED_ENTRIES)
self.assertEqual(
manifest,
"id=edp-managed-publish-unit-001\ncomponent=platform\ntype=app-overlay\n",
)
self.assertFalse(any(
name.startswith("payload/platform/services/external-data-plane/test/")
or "/node_modules/" in name
or Path(name).name.startswith(".env")
or Path(name).name == "private-key.pem"
for name in names
))
self.assertIn(
"source: /volume1/docker/nodedc-platform/trust/engine-managed-provisioner",
compose,
)
self.assertNotIn("private-key.pem", compose)
self.assertNotIn(b"-----BEGIN PRIVATE KEY-----", b"\n".join(regular_payloads))
if __name__ == "__main__":
unittest.main(verbosity=2)

926
infra/deploy-runner/test_platform_registry.py
View File

@ -1,926 +0,0 @@
#!/usr/bin/env python3
import importlib.machinery
import importlib.util
import hashlib
import inspect
import json
import stat
import tempfile
import unittest
from pathlib import Path
from unittest import mock
SCRIPT_DIR = Path(__file__).resolve().parent
RUNNER_PATH = SCRIPT_DIR / "nodedc-deploy"
def load_runner():
loader = importlib.machinery.SourceFileLoader(
"nodedc_platform_deploy_under_test",
str(RUNNER_PATH),
)
spec = importlib.util.spec_from_loader(loader.name, loader)
module = importlib.util.module_from_spec(spec)
loader.exec_module(module)
return module
RUNNER = load_runner()
LEGACY_CREDENTIAL_RUNTIME_CANON_FUNCTION_SHA256 = {
"build_canonical_engine_backend_rootfs": "7f8fca6380632d717ee38a3d58b5bbc58d34b0e6c81e2764a759ddd63bfd07d0",
"engine_backend_container_id": "08dc25b6fee37e8c0d942df91fc575d6766b99c306ad06c2c14afded8de3b375",
"engine_backend_immutable_runtime_is_current": "9c6b9a73ac5004830b1f06f56bfc1f25336d11001a97a5de7a04388e10a20990",
"engine_backend_node_modules_tree_sha256": "01749704e1624d247bdf41242aa71961f5e5bc092ffa906b69cdb911907b70c4",
"engine_backend_rootfs_member_allowed": "62342e5891a1ea885f1ff02169a100f9ca94464eb7dfb62a7cbe67750ebcbb91",
"engine_backend_rootfs_toolchain_link_is_omitted": "83772bc1a8aa654e5c1d0757d43b650ee978dc73aef7ae8fac567dbacc8c7c49",
"engine_backend_tool_versions": "c21d7c11ee8c40ac5e3e5c6c24c441d1ce52c9dd2c22fc1d8f26e121138fa78a",
"ensure_engine_backend_release_image": "f65a5f8573b911980f252dc3beed88cd6ee14e0f0c519c333ea90b78c8910953",
"expected_engine_credential_backend_override": "b7c9db0059a4d262b8fe61dc0745b8ee1a4b75d56d227c39ba08115c7e414486",
"inspect_engine_backend_derived_image": "2336ce8068bd321b6c8ac6e62b430dbcbf991201cb9b4178589294d1baf7e818",
"preflight_engine_credential_sink_ready": "3f084ade6346644c86aa9540fbcf70b3a44e32c7add0876935f45177569aad6d",
"prepare_engine_credential_backend_runtime": "0def464639767dcf85acd7978459b2f5c1ebf963424dd401d4c29aae7bb1aaa5",
"quarantine_engine_backend_partial_runtime": "57e455f874c681815cf42bf529d0b8620ae8317e82551d784cbd6dae913603d2",
"read_engine_backend_runtime_metadata": "7ee0ec2c72eb699db4b6a7d3d8d912f1b0c0c724c2a054eee223fba3ec0a1cb2",
"run_compose": "8425528d4934f82e79ebf3551c4246505e4a3cdf3f450949d7618a4312fee169",
"run_engine_backend_probe": "f0db7210b64cf3c80c1f0a00165f27dcab6962781a69dc2050d84a5373cedaa9",
"touches_engine_credential_sink": "5d0e0d680d1a8508fb1ac21d8c8b3aa92466fe8b219d2ddc1fe5411f94fca912",
"validate_engine_backend_activation_marker": "66c4d5d9b0b608c030c2c9c97e2e46e1f1634b7ba024dc107c15478091765483",
"validate_engine_backend_dependencies": "ca1383dffd5a6397ecfeedcc8f88ab59e90a33d5089590f0249f789b05ed29e2",
"validate_engine_backend_mounts": "5614267d1d0f734dd68ed7565292a53f34777b6dc3640137034f6db093ef7431",
"validate_engine_backend_runtime_override_file": "eae1d32ab75cc59916dbd1d4cd73a072c156abf9014d8ef234fcd2333748139d",
"validate_engine_credential_sink_slice": "87e6571baf47c30af120da32a327a88164415bf65b198e7190c28df7365bcfd5",
}
class CanonicalPlatformRegistryTest(unittest.TestCase):
def test_legacy_credential_runtime_functions_match_last_successful_canon(self):
actual = {
name: hashlib.sha256(
inspect.getsource(getattr(RUNNER, name)).encode("utf-8"),
).hexdigest()
for name in LEGACY_CREDENTIAL_RUNTIME_CANON_FUNCTION_SHA256
}
self.assertEqual(actual, LEGACY_CREDENTIAL_RUNTIME_CANON_FUNCTION_SHA256)
def test_publish_mount_extension_delegates_baseline_inventory_to_legacy_guard(self):
baseline_mounts = [
{
"Type": "bind",
"Source": "/baseline/app",
"Destination": "/app",
"RW": True,
},
]
publish_mounts = [
{
"Type": "bind",
"Source": str(RUNNER.ENGINE_PUBLISH_GRANT_STATE_PATH),
"Destination": RUNNER.ENGINE_PUBLISH_GRANT_CONTAINER_PATH,
"RW": True,
},
{
"Type": "bind",
"Source": str(RUNNER.ENGINE_EDP_MANAGED_PROVISIONER_PRIVATE_KEY_FILE),
"Destination": RUNNER.ENGINE_EDP_PRIVATE_KEY_CONTAINER_PATH,
"RW": False,
},
]
container = {"Mounts": [*baseline_mounts, *publish_mounts]}
with tempfile.TemporaryDirectory(prefix="nodedc-publish-mounts-") as directory:
root = Path(directory)
override = root / RUNNER.ENGINE_DATA_PRODUCT_PUBLISH_GRANT_OVERRIDE_REL
override.parent.mkdir(parents=True)
override.write_text(
RUNNER.expected_engine_data_product_publish_grant_override(),
encoding="utf-8",
)
original_root = RUNNER.COMPONENTS["engine"]["payload_root"]
RUNNER.COMPONENTS["engine"]["payload_root"] = root
try:
with mock.patch.object(
RUNNER,
"validate_engine_backend_mounts",
) as legacy_guard:
RUNNER.validate_engine_backend_mounts_for_installed_runtime(
container,
True,
)
legacy_guard.assert_called_once_with(
{"Mounts": baseline_mounts},
True,
)
wrong_source = {
"Mounts": [
*baseline_mounts,
{**publish_mounts[0], "Source": "/wrong"},
publish_mounts[1],
],
}
with self.assertRaisesRegex(
RUNNER.DeployError,
"Publish mount barrier mismatch",
):
RUNNER.validate_engine_backend_mounts_for_installed_runtime(
wrong_source,
True,
)
finally:
RUNNER.COMPONENTS["engine"]["payload_root"] = original_root
def test_publish_mount_extension_accepts_exact_complete_runtime_inventory(self):
baseline = {
"/app": (True, "/engine/source"),
"/app/deploy/docker-compose.yml": (False, "/engine/docker-compose.yml"),
"/seed-api": (False, "/engine/seed-api"),
"/seed-data": (False, "/engine/seed-data"),
"/app/node_modules": (False, str(RUNNER.ENGINE_CREDENTIAL_BACKEND_NODE_MODULES_DIR)),
"/app/server/data": (True, "/engine/data"),
"/app/server/storage": (True, "/engine/storage"),
"/app/server/logs": (True, "/engine/logs"),
"/var/run/docker.sock": (True, "/var/run/docker.sock"),
}
mounts = [
{
"Type": "bind",
"Source": source,
"Destination": destination,
"RW": writable,
}
for destination, (writable, source) in baseline.items()
]
mounts.extend((
{
"Type": "bind",
"Source": str(RUNNER.ENGINE_PUBLISH_GRANT_STATE_PATH),
"Destination": RUNNER.ENGINE_PUBLISH_GRANT_CONTAINER_PATH,
"RW": True,
},
{
"Type": "bind",
"Source": str(RUNNER.ENGINE_EDP_MANAGED_PROVISIONER_PRIVATE_KEY_FILE),
"Destination": RUNNER.ENGINE_EDP_PRIVATE_KEY_CONTAINER_PATH,
"RW": False,
},
))
with tempfile.TemporaryDirectory(prefix="nodedc-publish-full-mounts-") as directory:
root = Path(directory)
override = root / RUNNER.ENGINE_DATA_PRODUCT_PUBLISH_GRANT_OVERRIDE_REL
override.parent.mkdir(parents=True)
override.write_text(
RUNNER.expected_engine_data_product_publish_grant_override(),
encoding="utf-8",
)
original_root = RUNNER.COMPONENTS["engine"]["payload_root"]
RUNNER.COMPONENTS["engine"]["payload_root"] = root
try:
RUNNER.validate_engine_backend_mounts_for_installed_runtime(
{"Mounts": mounts},
True,
)
finally:
RUNNER.COMPONENTS["engine"]["payload_root"] = original_root
def test_absent_publish_overlay_uses_legacy_mount_guard_unchanged(self):
container = {"Mounts": [{"Destination": "/app"}]}
with tempfile.TemporaryDirectory(prefix="nodedc-no-publish-mounts-") as directory:
original_root = RUNNER.COMPONENTS["engine"]["payload_root"]
RUNNER.COMPONENTS["engine"]["payload_root"] = Path(directory)
try:
with mock.patch.object(
RUNNER,
"validate_engine_backend_mounts",
) as legacy_guard:
RUNNER.validate_engine_backend_mounts_for_installed_runtime(
container,
False,
)
legacy_guard.assert_called_once_with(container, False)
finally:
RUNNER.COMPONENTS["engine"]["payload_root"] = original_root
def test_runner_does_not_reinterpret_compose_schema(self):
self.assertFalse(hasattr(RUNNER, "render_compose_model"))
self.assertFalse(hasattr(RUNNER, "validate_platform_compose_model"))
self.assertFalse(hasattr(RUNNER, "validate_engine_compose_model"))
self.assertFalse(hasattr(RUNNER, "preflight_platform_compose_candidate"))
self.assertFalse(hasattr(RUNNER, "preflight_engine_compose_candidate"))
def test_platform_edp_change_recreates_only_application(self):
entries = (
"platform/docker-compose.external-data-plane.yml",
"platform/services/external-data-plane",
)
services = RUNNER.component_services("platform", entries)
self.assertEqual(services, ("external-data-plane",))
self.assertNotIn("external-data-plane-postgres", services)
builds = RUNNER.component_builds("platform", entries)
self.assertEqual(len(builds), 1)
build_root, build_args = builds[0]
self.assertEqual(
build_root,
Path("/volume1/docker/nodedc-platform/platform"),
)
self.assertEqual(
build_args,
(
"build",
"--no-cache",
"-f",
"services/external-data-plane/Dockerfile",
"-t",
RUNNER.EXTERNAL_DATA_PLANE_IMAGE,
".",
),
)
def test_provider_catalog_is_metadata_only(self):
entries = (
"platform/packages/external-provider-contract/providers/gelios/v1/provider.json",
)
self.assertFalse(RUNNER.touches_external_data_plane_files(entries))
self.assertEqual(RUNNER.component_services("platform", entries), ())
self.assertEqual(RUNNER.component_builds("platform", entries), ())
self.assertEqual(RUNNER.component_healthchecks("platform", entries), ())
with (
mock.patch.object(RUNNER, "run_build") as build,
mock.patch.object(RUNNER, "prepare_component_runtime") as prepare,
mock.patch.object(RUNNER, "run_compose") as compose,
mock.patch.object(RUNNER, "healthcheck_url") as healthcheck,
):
RUNNER.run_component_runtime("platform", entries, ())
RUNNER.run_healthchecks("platform", entries, ())
build.assert_not_called()
prepare.assert_not_called()
compose.assert_not_called()
healthcheck.assert_not_called()
def test_generic_engine_compose_change_keeps_legacy_service_selection(self):
services = RUNNER.component_services(
"engine",
(
"docker-compose.yml",
"nodedc-source/server/index.js",
"nodedc-source/server/dataProductPublishGrant/store.js",
),
)
self.assertEqual(services, ("nodedc-backend", "app"))
self.assertNotIn("n8n-postgres", services)
self.assertNotIn("postgresql", services)
self.assertNotIn("n8n", services)
def test_engine_runtime_prepares_managed_private_state(self):
with (
mock.patch.object(
RUNNER,
"ensure_engine_edp_managed_provisioner_keypair",
) as keypair,
mock.patch.object(
RUNNER,
"ensure_engine_publish_grant_private_state",
) as private_state,
):
RUNNER.prepare_component_runtime(
"engine",
RUNNER.ENGINE_DATA_PRODUCT_PUBLISH_GRANT_ARTIFACT_ENTRIES,
)
keypair.assert_called_once_with()
private_state.assert_called_once_with()
def test_partial_publish_path_has_no_publish_runtime_side_effects(self):
entries = ("nodedc-source/server/dataProductPublishGrant/store.js",)
self.assertTrue(RUNNER.touches_engine_data_product_publish_grant(entries))
self.assertFalse(RUNNER.is_engine_data_product_publish_grant_slice(
"engine",
entries,
))
with (
mock.patch.object(
RUNNER,
"ensure_engine_edp_managed_provisioner_keypair",
) as keypair,
mock.patch.object(
RUNNER,
"ensure_engine_publish_grant_private_state",
) as private_state,
):
RUNNER.prepare_component_runtime("engine", entries)
keypair.assert_not_called()
private_state.assert_not_called()
def test_publish_predecessor_guard_checks_every_pinned_file_before_mutation(self):
with tempfile.TemporaryDirectory(prefix="nodedc-publish-predecessor-") as directory:
root = Path(directory)
files = {
"docker-compose.yml": b"services: {}\n",
"nodedc-source/server/routes/n8n.js": b"export const baseline = true\n",
}
expected = {}
for relative_path, value in files.items():
path = root / relative_path
path.parent.mkdir(parents=True, exist_ok=True)
path.write_bytes(value)
expected[relative_path] = hashlib.sha256(value).hexdigest()
original_root = RUNNER.COMPONENTS["engine"]["payload_root"]
original_hashes = RUNNER.ENGINE_DATA_PRODUCT_PUBLISH_GRANT_PREDECESSOR_SHA256
RUNNER.COMPONENTS["engine"]["payload_root"] = root
RUNNER.ENGINE_DATA_PRODUCT_PUBLISH_GRANT_PREDECESSOR_SHA256 = expected
try:
self.assertEqual(
RUNNER.preflight_engine_data_product_publish_grant_predecessor(),
{
"mode": "credential-sink-initial-transition",
"compose_sha256": expected["docker-compose.yml"],
"changed_paths": (),
},
)
(root / "nodedc-source/server/routes/n8n.js").write_bytes(b"drift\n")
with self.assertRaisesRegex(
RUNNER.DeployError,
"path=nodedc-source/server/routes/n8n.js",
):
RUNNER.preflight_engine_data_product_publish_grant_predecessor()
finally:
RUNNER.ENGINE_DATA_PRODUCT_PUBLISH_GRANT_PREDECESSOR_SHA256 = original_hashes
RUNNER.COMPONENTS["engine"]["payload_root"] = original_root
def test_installed_publish_update_preserves_foundation_and_allows_only_grant_source_changes(self):
with tempfile.TemporaryDirectory(prefix="nodedc-publish-update-") as directory:
base = Path(directory)
root = base / "installed"
payload = base / "payload"
root.mkdir()
payload.mkdir()
compose_bytes = b"services: {}\n"
compose = root / "docker-compose.yml"
compose.write_bytes(compose_bytes)
expected = {
"docker-compose.yml": hashlib.sha256(compose_bytes).hexdigest(),
}
shared_files = {
"nodedc-source/server/assets/provider-packages/v1/catalog.json": "{}\n",
"nodedc-source/server/engineAgents/store.js": "export const profile = 'full-developer'\n",
"nodedc-source/server/routes/engineAgentGateway.js": "\n".join((
"engine_plan_data_product_publish_grant",
"engine_apply_data_product_publish_grant",
"engine_accept_data_product_publish_grant",
"engine_rollback_data_product_publish_grant",
)),
"nodedc-source/server/routes/n8n.js": (
"engineCredentialSinkN8nAdapter\n"
"engineDataProductPublishGrantN8nAdapter\n"
),
RUNNER.ENGINE_DATA_PRODUCT_PUBLISH_GRANT_OVERRIDE_REL: (
RUNNER.expected_engine_data_product_publish_grant_override()
),
}
for name in RUNNER.ENGINE_DATA_PRODUCT_PUBLISH_GRANT_SOURCE_FILES:
shared_files[f"nodedc-source/server/dataProductPublishGrant/{name}"] = (
f"export const source = '{name}'\n"
)
for relative_path, value in shared_files.items():
for destination in (root, payload):
path = destination / relative_path
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(value, encoding="utf-8")
acceptance_rel = "nodedc-source/server/dataProductPublishGrant/acceptance.js"
(payload / acceptance_rel).write_text(
"export const source = 'acceptance-fixed'\n",
encoding="utf-8",
)
original_root = RUNNER.COMPONENTS["engine"]["payload_root"]
original_hashes = RUNNER.ENGINE_DATA_PRODUCT_PUBLISH_GRANT_PREDECESSOR_SHA256
RUNNER.COMPONENTS["engine"]["payload_root"] = root
RUNNER.ENGINE_DATA_PRODUCT_PUBLISH_GRANT_PREDECESSOR_SHA256 = expected
try:
self.assertEqual(
RUNNER.preflight_engine_data_product_publish_grant_predecessor(payload),
{
"mode": "installed-source-update",
"compose_sha256": expected["docker-compose.yml"],
"changed_paths": (acceptance_rel,),
},
)
gateway_rel = "nodedc-source/server/routes/engineAgentGateway.js"
(payload / gateway_rel).write_text("cross-boundary change\n", encoding="utf-8")
with self.assertRaisesRegex(
RUNNER.DeployError,
"crosses its source boundary",
):
RUNNER.preflight_engine_data_product_publish_grant_predecessor(payload)
(payload / gateway_rel).write_text(shared_files[gateway_rel], encoding="utf-8")
compose.write_text("drift\n", encoding="utf-8")
with self.assertRaisesRegex(
RUNNER.DeployError,
"foundation drift",
):
RUNNER.preflight_engine_data_product_publish_grant_predecessor(payload)
finally:
RUNNER.ENGINE_DATA_PRODUCT_PUBLISH_GRANT_PREDECESSOR_SHA256 = original_hashes
RUNNER.COMPONENTS["engine"]["payload_root"] = original_root
def test_edp_healthcheck_requires_database_and_managed_provisioning(self):
checks = RUNNER.component_healthchecks(
"platform",
("platform/services/external-data-plane/src/server.mjs",),
("external-data-plane",),
)
self.assertEqual(
checks,
(RUNNER.external_data_plane_healthcheck(),),
)
def test_edp_runtime_never_probes_unselected_platform_services(self):
entries = ("platform/services/external-data-plane/src/server.mjs",)
services = ("external-data-plane",)
with mock.patch.object(RUNNER, "healthcheck_url") as healthcheck:
RUNNER.run_healthchecks("platform", entries, services)
healthcheck.assert_called_once_with(
RUNNER.external_data_plane_healthcheck(),
)
def test_json_healthcheck_accepts_expected_contract(self):
class Response:
status = 200
def __enter__(self):
return self
def __exit__(self, *_args):
return False
def read(self, _limit):
return json.dumps({
"ok": True,
"database": "ready",
}).encode("utf-8")
with mock.patch.object(
RUNNER.NO_REDIRECT_OPENER,
"open",
return_value=Response(),
):
RUNNER.healthcheck_url({
"url": "http://127.0.0.1:18106/healthz",
"expected_json": {
"ok": True,
"database": "ready",
},
})
def test_json_healthcheck_accepts_restored_edp_baseline_contract(self):
class Response:
status = 200
def __enter__(self):
return self
def __exit__(self, *_args):
return False
def read(self, _limit):
return json.dumps({
"ok": True,
"service": "nodedc-external-data-plane",
"database": "ready",
}).encode("utf-8")
with mock.patch.object(
RUNNER.NO_REDIRECT_OPENER,
"open",
return_value=Response(),
):
RUNNER.healthcheck_url(
RUNNER.external_data_plane_healthcheck(require_managed=False),
)
def test_json_healthcheck_does_not_accept_redirect(self):
class Response:
status = 200
def __enter__(self):
return self
def __exit__(self, *_args):
return False
def read(self, _limit):
return json.dumps({
"ok": True,
"database": "ready",
}).encode("utf-8")
url = "http://127.0.0.1:18106/healthz"
redirect = RUNNER.urllib.error.HTTPError(
url,
302,
"Found",
{},
None,
)
with (
mock.patch.object(
RUNNER.NO_REDIRECT_OPENER,
"open",
side_effect=(redirect, Response()),
) as opener,
mock.patch.object(RUNNER.time, "sleep"),
):
RUNNER.healthcheck_url({
"url": url,
"expected_json": {
"ok": True,
"database": "ready",
},
})
self.assertEqual(opener.call_count, 2)
def test_publish_grant_healthcheck_does_not_enter_n8n_transition(self):
services = ("nodedc-backend",)
with (
mock.patch.object(RUNNER, "accept_engine_n8n_runtime") as accept,
mock.patch.object(RUNNER, "healthcheck_compose_service") as compose_health,
mock.patch.object(RUNNER, "healthcheck_url") as healthcheck,
mock.patch.object(
RUNNER,
"preflight_engine_credential_backend_runtime",
return_value={"mode": "verified-derived-retry"},
) as backend_preflight,
):
RUNNER.run_healthchecks(
"engine",
RUNNER.ENGINE_DATA_PRODUCT_PUBLISH_GRANT_ARTIFACT_ENTRIES,
services,
)
accept.assert_not_called()
compose_health.assert_called_once_with("engine", "nodedc-backend")
healthcheck.assert_called_once_with(
"http://127.0.0.1:3001/health",
)
backend_preflight.assert_called_once_with()
def test_engine_ui_healthcheck_requires_selected_app(self):
grant_entries = RUNNER.ENGINE_DATA_PRODUCT_PUBLISH_GRANT_ARTIFACT_ENTRIES
self.assertEqual(
RUNNER.component_healthchecks(
"engine",
grant_entries,
("nodedc-backend",),
),
("http://127.0.0.1:3001/health",),
)
self.assertEqual(
RUNNER.component_healthchecks(
"engine",
("docker-compose.yml",),
("nodedc-backend", "app"),
),
(
"http://127.0.0.1:8080/",
"http://127.0.0.1:3001/health",
),
)
def test_optional_edp_overlay_is_omitted_until_installed(self):
with tempfile.TemporaryDirectory(
prefix="nodedc-compose-files-",
) as directory:
temporary = Path(directory)
base = temporary / "docker-compose.platform-http.yml"
edp = temporary / "docker-compose.external-data-plane.yml"
base.write_text(
"name: nodedc-platform\n",
encoding="utf-8",
)
original = RUNNER.COMPONENTS["platform"]["compose_files"]
RUNNER.COMPONENTS["platform"]["compose_files"] = (base, edp)
try:
self.assertEqual(
RUNNER.component_compose_files("platform"),
(base,),
)
finally:
RUNNER.COMPONENTS["platform"]["compose_files"] = original
class PlatformOverlayRollbackTest(unittest.TestCase):
def test_engine_runtime_failure_restores_source_and_recreates_exact_services(self):
with tempfile.TemporaryDirectory(prefix="nodedc-engine-runtime-rollback-") as directory:
temporary = Path(directory)
root = temporary / "engine"
backup = temporary / "backup"
runner_tmp = temporary / "tmp"
compose = root / "docker-compose.yml"
backend = root / "nodedc-source/server/index.js"
compose.parent.mkdir(parents=True)
backend.parent.mkdir(parents=True)
backup.mkdir()
runner_tmp.mkdir()
compose.write_text("name: baseline\n", encoding="utf-8")
backend.write_text("export const version = 1;\n", encoding="utf-8")
entries = ["docker-compose.yml", "nodedc-source/server/index.js"]
services = ("n8n", "nodedc-backend", "app")
RUNNER.create_backup(root, backup, entries, include_nginx_html=False)
compose.write_text("name: candidate\n", encoding="utf-8")
backend.write_text("export const version = 2;\n", encoding="utf-8")
def assert_restored(component, rollback_entries, rollback_services):
self.assertEqual(component, "engine")
self.assertEqual(rollback_entries, entries)
self.assertEqual(rollback_services, services)
self.assertEqual(compose.read_text(encoding="utf-8"), "name: baseline\n")
self.assertEqual(
backend.read_text(encoding="utf-8"),
"export const version = 1;\n",
)
with (
mock.patch.object(RUNNER, "TMP_DIR", runner_tmp),
mock.patch.object(
RUNNER,
"run_component_runtime",
side_effect=assert_restored,
) as runtime,
mock.patch.object(RUNNER, "run_healthchecks") as healthchecks,
):
status = RUNNER.rollback_engine_apply(
root,
backup,
entries,
"test-stamp",
runtime_started=True,
applied_services=services,
)
self.assertEqual(status, "source+runtime-restored:2")
runtime.assert_called_once()
healthchecks.assert_called_once_with("engine", entries, services)
def test_candidate_only_service_cleanup_never_requests_volume_removal(self):
with (
mock.patch.object(RUNNER, "component_compose_root", return_value=Path("/live/platform")),
mock.patch.object(RUNNER, "compose_base_cmd", return_value=["docker", "compose"]),
mock.patch.object(RUNNER.subprocess, "run") as run,
):
RUNNER.stop_and_remove_compose_services("platform", ("external-data-plane",))
command = run.call_args.args[0]
self.assertEqual(
command,
["docker", "compose", "rm", "--stop", "--force", "external-data-plane"],
)
self.assertNotIn("--volumes", command)
def test_failed_first_edp_activation_removes_containers_without_deleting_volume(self):
with tempfile.TemporaryDirectory(prefix="nodedc-platform-edp-rollback-") as directory:
temporary = Path(directory)
root = temporary / "root"
backup = temporary / "backup"
runner_tmp = temporary / "tmp"
overlay = root / RUNNER.PLATFORM_EXTERNAL_DATA_PLANE_COMPOSE_REL
overlay.parent.mkdir(parents=True)
backup.mkdir()
runner_tmp.mkdir()
entries = [RUNNER.PLATFORM_EXTERNAL_DATA_PLANE_COMPOSE_REL]
RUNNER.create_backup(root, backup, entries, include_nginx_html=False)
overlay.write_text("name: nodedc-platform\n", encoding="utf-8")
with (
mock.patch.object(RUNNER, "TMP_DIR", runner_tmp),
mock.patch.object(RUNNER, "stop_and_remove_compose_services") as remove,
mock.patch.object(RUNNER, "run_component_runtime") as runtime,
mock.patch.object(RUNNER, "run_healthchecks") as healthchecks,
):
status = RUNNER.rollback_platform_apply(
root,
backup,
entries,
"test-stamp",
runtime_started=True,
applied_services=("external-data-plane",),
)
remove.assert_called_once_with(
"platform",
("external-data-plane",),
)
runtime.assert_not_called()
healthchecks.assert_not_called()
self.assertFalse(overlay.exists())
self.assertEqual(status, "source+runtime-restored:1")
def test_candidate_cleanup_failure_still_restores_source_before_reporting_failure(self):
with tempfile.TemporaryDirectory(prefix="nodedc-platform-edp-rollback-") as directory:
temporary = Path(directory)
root = temporary / "root"
backup = temporary / "backup"
runner_tmp = temporary / "tmp"
overlay = root / RUNNER.PLATFORM_EXTERNAL_DATA_PLANE_COMPOSE_REL
overlay.parent.mkdir(parents=True)
backup.mkdir()
runner_tmp.mkdir()
entries = [RUNNER.PLATFORM_EXTERNAL_DATA_PLANE_COMPOSE_REL]
RUNNER.create_backup(root, backup, entries, include_nginx_html=False)
overlay.write_text("name: rejected\n", encoding="utf-8")
with (
mock.patch.object(RUNNER, "TMP_DIR", runner_tmp),
mock.patch.object(
RUNNER,
"stop_and_remove_compose_services",
side_effect=RuntimeError("simulated cleanup failure"),
),
):
with self.assertRaisesRegex(RUNNER.DeployError, "cleanup failed after source restore"):
RUNNER.rollback_platform_apply(
root,
backup,
entries,
"test-stamp",
runtime_started=True,
applied_services=("external-data-plane",),
)
self.assertFalse(overlay.exists())
def test_runtime_failure_rebuilds_recreates_and_accepts_restored_overlay(self):
with tempfile.TemporaryDirectory(prefix="nodedc-platform-runtime-rollback-") as directory:
temporary = Path(directory)
root = temporary / "root"
backup = temporary / "backup"
runner_tmp = temporary / "tmp"
gateway = root / "platform/gelios-gateway"
gateway.mkdir(parents=True)
backup.mkdir()
runner_tmp.mkdir()
source = gateway / "server.mjs"
source.write_text("export const version = 1;\n", encoding="utf-8")
entries = ["platform/gelios-gateway"]
RUNNER.create_backup(root, backup, entries, include_nginx_html=False)
source.write_text("export const version = 2;\n", encoding="utf-8")
def assert_restored_runtime(component, rollback_entries, services):
self.assertEqual(component, "platform")
self.assertEqual(rollback_entries, entries)
self.assertEqual(services, ("gelios-postgres", "gelios-gateway"))
self.assertEqual(source.read_text(encoding="utf-8"), "export const version = 1;\n")
with (
mock.patch.object(RUNNER, "TMP_DIR", runner_tmp),
mock.patch.object(
RUNNER,
"run_component_runtime",
side_effect=assert_restored_runtime,
) as runtime,
mock.patch.object(RUNNER, "run_healthchecks") as healthchecks,
):
status = RUNNER.rollback_platform_apply(
root,
backup,
entries,
"test-stamp",
runtime_started=True,
applied_services=("gelios-postgres", "gelios-gateway"),
)
self.assertEqual(status, "source+runtime-restored:1")
runtime.assert_called_once()
healthchecks.assert_called_once_with(
"platform",
entries,
("gelios-postgres", "gelios-gateway"),
)
def test_existing_edp_rollback_accepts_only_restored_edp(self):
with tempfile.TemporaryDirectory(prefix="nodedc-platform-edp-runtime-rollback-") as directory:
temporary = Path(directory)
root = temporary / "root"
backup = temporary / "backup"
runner_tmp = temporary / "tmp"
service = root / "platform/services/external-data-plane"
service.mkdir(parents=True)
backup.mkdir()
runner_tmp.mkdir()
source = service / "server.mjs"
source.write_text("export const version = 1;\n", encoding="utf-8")
entries = ["platform/services/external-data-plane"]
RUNNER.create_backup(root, backup, entries, include_nginx_html=False)
source.write_text("export const version = 2;\n", encoding="utf-8")
with (
mock.patch.object(RUNNER, "TMP_DIR", runner_tmp),
mock.patch.object(RUNNER, "run_component_runtime") as runtime,
mock.patch.object(RUNNER, "healthcheck_url") as healthcheck,
):
status = RUNNER.rollback_platform_apply(
root,
backup,
entries,
"test-stamp",
runtime_started=True,
applied_services=("external-data-plane",),
)
self.assertEqual(status, "source+runtime-restored:1")
self.assertEqual(
source.read_text(encoding="utf-8"),
"export const version = 1;\n",
)
runtime.assert_called_once_with(
"platform",
entries,
("external-data-plane",),
)
healthcheck.assert_called_once_with(
RUNNER.external_data_plane_healthcheck(require_managed=False),
)
def test_restores_existing_overlay_and_removes_new_paths(self):
with tempfile.TemporaryDirectory(prefix="nodedc-platform-rollback-test-") as directory:
temporary = Path(directory)
root = temporary / "root"
backup = temporary / "backup"
runner_tmp = temporary / "tmp"
root.mkdir()
backup.mkdir()
runner_tmp.mkdir()
compose = root / "platform/docker-compose.platform-http.yml"
gateway = root / "platform/gelios-gateway"
new_service = root / "platform/services/new-service"
unrelated = root / "platform/unrelated.txt"
compose.parent.mkdir(parents=True)
gateway.mkdir()
compose.write_text("name: nodedc-platform\n", encoding="utf-8")
executable = gateway / "run.sh"
executable.write_text("#!/bin/sh\nexit 0\n", encoding="utf-8")
executable.chmod(0o755)
(gateway / "server.mjs").write_text("export const version = 1;\n", encoding="utf-8")
unrelated.write_text("untouched\n", encoding="utf-8")
entries = [
"platform/docker-compose.platform-http.yml",
"platform/gelios-gateway",
"platform/services/new-service",
]
RUNNER.create_backup(root, backup, entries, include_nginx_html=False)
compose.write_text("name: broken\n", encoding="utf-8")
(gateway / "server.mjs").write_text("export const version = 2;\n", encoding="utf-8")
executable.chmod(0o644)
new_service.mkdir(parents=True)
(new_service / "bad.mjs").write_text("broken\n", encoding="utf-8")
with mock.patch.object(RUNNER, "TMP_DIR", runner_tmp):
restored = RUNNER.restore_platform_overlay(root, backup, entries, "test-stamp")
self.assertEqual(restored, len(entries))
self.assertEqual(compose.read_text(encoding="utf-8"), "name: nodedc-platform\n")
self.assertEqual(
(gateway / "server.mjs").read_text(encoding="utf-8"),
"export const version = 1;\n",
)
self.assertTrue(stat.S_IMODE((gateway / "run.sh").stat().st_mode) & stat.S_IXUSR)
self.assertFalse(new_service.exists())
self.assertEqual(unrelated.read_text(encoding="utf-8"), "untouched\n")
def test_rejects_inconsistent_backup_partition_before_restore(self):
with tempfile.TemporaryDirectory(prefix="nodedc-platform-rollback-test-") as directory:
temporary = Path(directory)
root = temporary / "root"
backup = temporary / "backup"
runner_tmp = temporary / "tmp"
target = root / "platform/docker-compose.platform-http.yml"
target.parent.mkdir(parents=True)
target.write_text("original\n", encoding="utf-8")
backup.mkdir()
runner_tmp.mkdir()
entries = ["platform/docker-compose.platform-http.yml"]
RUNNER.create_backup(root, backup, entries, include_nginx_html=False)
(backup / "missing-files.txt").write_text(
"platform/docker-compose.platform-http.yml\n",
encoding="utf-8",
)
target.write_text("candidate\n", encoding="utf-8")
with mock.patch.object(RUNNER, "TMP_DIR", runner_tmp):
with self.assertRaisesRegex(RUNNER.DeployError, "backup entry set mismatch"):
RUNNER.restore_platform_overlay(root, backup, entries, "test-stamp")
self.assertEqual(target.read_text(encoding="utf-8"), "candidate\n")
if __name__ == "__main__":
unittest.main(verbosity=2)

5
infra/docker-compose.dev.yml
View File

@ -192,8 +192,9 @@ services:
retries: 3 retries: 3
start_period: 10s start_period: 10s
# Frozen legacy Gelios compatibility contour. Keep it reproducible and # External-provider telemetry is isolated from Platform identity state. The
# isolated; new provider integrations use the provider-neutral data plane. # database is not published; only the Gateway is exposed on the local host
# for operator checks and Engine development.
gelios-postgres: gelios-postgres:
image: ${GELIOS_TIMESCALE_IMAGE:-timescale/timescaledb-ha:pg16.14-ts2.28.2-all} image: ${GELIOS_TIMESCALE_IMAGE:-timescale/timescaledb-ha:pg16.14-ts2.28.2-all}
restart: unless-stopped restart: unless-stopped

24
infra/synology/.env.synology.example
View File

@ -97,20 +97,9 @@ EXTERNAL_DATA_PLANE_WRITER_BINDING_MAX_TTL_DAYS=90
EXTERNAL_DATA_PLANE_MAX_FUTURE_SKEW_SECONDS=300 EXTERNAL_DATA_PLANE_MAX_FUTURE_SKEW_SECONDS=300
EXTERNAL_DATA_PLANE_RETENTION_SWEEP_MS=3600000 EXTERNAL_DATA_PLANE_RETENTION_SWEEP_MS=3600000
EXTERNAL_DATA_PLANE_LEGACY_INTAKE_ENABLED=false EXTERNAL_DATA_PLANE_LEGACY_INTAKE_ENABLED=false
# Internal control-plane writer/reader binding issuance; keep false until the # The writer-provisioner secret is a root-owned file under
# atomic NDC L2 ensure-grant operation is deployed. Legacy path returns a # /volume1/docker/nodedc-platform/secrets/external-data-plane-provisioner/,
# plaintext capability and must never be exposed to users. # never a shared .env value.
EXTERNAL_DATA_PLANE_PROVISIONING_ENABLED=false
# Digest-only managed writer-binding ensure. It accepts only signed Engine
# service requests; the legacy provisioner bearer is deliberately invalid here.
# Keep false until the matching Engine private key is provisioned. The trust
# directory is mounted read-only into EDP and contains only `public-key.pem`.
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONING_ENABLED=false
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_SERVICE_ID=nodedc-engine
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_KEY_ID=engine-edp-managed-provisioner-v1
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_AUDIENCE=nodedc-external-data-plane.managed-provisioning.v1
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_MAX_SKEW_SECONDS=60
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_REPLAY_CACHE_MAX_ENTRIES=10000
NOTIFICATION_PG_DB=nodedc_notifications NOTIFICATION_PG_DB=nodedc_notifications
NOTIFICATION_PG_USER=nodedc_notifications NOTIFICATION_PG_USER=nodedc_notifications
@ -143,9 +132,8 @@ AI_WORKSPACE_ONTOLOGY_MCP_ENABLED=true
AI_WORKSPACE_ONTOLOGY_MCP_PUBLIC_URL= AI_WORKSPACE_ONTOLOGY_MCP_PUBLIC_URL=
ONTOLOGY_CORE_HOST_BIND=127.0.0.1:18104 ONTOLOGY_CORE_HOST_BIND=127.0.0.1:18104
# Gelios Gateway — frozen legacy storage/read compatibility service. Provider # Gelios Gateway — storage/read service. Provider credentials stay in the
# credentials stay in the protected Engine Collector. Do not extend this # protected Engine Collector. Intake remains disabled until scope is approved.
# provider-specific contour for new providers.
GELIOS_TIMESCALE_IMAGE=timescale/timescaledb-ha:pg16.14-ts2.28.2-all GELIOS_TIMESCALE_IMAGE=timescale/timescaledb-ha:pg16.14-ts2.28.2-all
GELIOS_PG_DB=nodedc_gelios GELIOS_PG_DB=nodedc_gelios
GELIOS_PG_USER=nodedc_gelios GELIOS_PG_USER=nodedc_gelios
@ -153,6 +141,8 @@ GELIOS_PG_PASS=replace-with-random-synology-secret
# URL-encode reserved characters in GELIOS_PG_PASS when forming this URL. # URL-encode reserved characters in GELIOS_PG_PASS when forming this URL.
GELIOS_DATABASE_URL=postgresql://nodedc_gelios:replace-with-url-encoded-synology-secret@gelios-postgres:5432/nodedc_gelios GELIOS_DATABASE_URL=postgresql://nodedc_gelios:replace-with-url-encoded-synology-secret@gelios-postgres:5432/nodedc_gelios
GELIOS_GATEWAY_HOST_BIND=127.0.0.1:18105 GELIOS_GATEWAY_HOST_BIND=127.0.0.1:18105
# Explicit tenant and connection identifiers are deployment configuration;
# do not encode a customer or pilot name in source defaults.
GELIOS_TENANT_ID=replace-with-tenant-id GELIOS_TENANT_ID=replace-with-tenant-id
GELIOS_CONNECTION_ID=gelios-connection-id GELIOS_CONNECTION_ID=gelios-connection-id
# `all` accepts every unit returned by this approved tenant + connection. # `all` accepts every unit returned by this approved tenant + connection.

74
infra/synology/README.md
View File

@ -202,13 +202,7 @@ TASKER_SYNC_SOURCE=1 ./infra/synology/deploy-current.sh
Если emergency-fix был сделан прямо на Synology в этих env-файлах, перенести sanitized-значение в `.env.synology.example`/docs, а секрет оставить только в live env. Если emergency-fix был сделан прямо на Synology в этих env-файлах, перенести sanitized-значение в `.env.synology.example`/docs, а секрет оставить только в live env.
### Gelios: frozen legacy compatibility contour ### Gelios: сбор всех units доверенного подключения
`gelios-postgres` и `gelios-gateway` — уже существующий self-hosted контур с
собственным persistent Timescale volume. Он остаётся полностью воспроизводимым
в source/deploy, но не развивается и не используется как шаблон для новых
провайдеров. Его удаление или миграция требуют отдельного подтверждённого плана;
обычный Platform/EDP deploy не удаляет volume и не изменяет старый workflow.
`GELIOS_UNIT_SCOPE=all` означает **все текущие и будущие units только одной уже `GELIOS_UNIT_SCOPE=all` означает **все текущие и будущие units только одной уже
настроенной пары `GELIOS_TENANT_ID` + `GELIOS_CONNECTION_ID`**. Это не wildcard настроенной пары `GELIOS_TENANT_ID` + `GELIOS_CONNECTION_ID`**. Это не wildcard
@ -216,71 +210,17 @@ TASKER_SYNC_SOURCE=1 ./infra/synology/deploy-current.sh
Пропавший из очередного ответа unit не удаляется: его stable provider ID и Пропавший из очередного ответа unit не удаляется: его stable provider ID и
`last_seen_at` сохраняются; видимость на карте — отдельная логика витрины. `last_seen_at` сохраняются; видимость на карте — отдельная логика витрины.
Перед узким legacy apply включить политику на Synology (скрипт создаёт backup и Перед каноническим `nodedc-deploy apply` включить политику на Synology (скрипт
не выводит секреты): создаёт backup и не выводит секреты):
```bash ```bash
sudo bash /volume1/docker/nodedc-deploy/inbox/prepare-gelios-all-units-env.sh sudo bash /volume1/docker/nodedc-deploy/inbox/prepare-gelios-all-units-env.sh
``` ```
В plan такого Platform-артефакта должны быть только `gelios-postgres` и Затем применить узкий Platform-артефакт, собранный с `--gateway-only`, через
`gelios-gateway`; общий compose в узкий архив не входит. `nodedc-deploy`. В его plan должны быть только `gelios-postgres` и
`gelios-gateway`; общий `docker-compose` в такой архив не входит. После apply
## EDP publish grant: platform-owned credential Gateway начинает принимать весь состав units этого подключения.
`NDC Data Product Writer API` — историческое имя native credential type для
внутреннего scoped EDP publish grant. Это не Gelios access/refresh token и не
право записи в Gelios.
Целевой путь полностью служебный: control plane NDC по разрешённому connection
profile генерирует capability внутри native NDC L2 Credentials, передаёт EDP
только digest для scoped binding и возвращает MCP только opaque compatible
reference/status.
Пользователь не читает, не копирует и не вводит этот внутренний секрет.
EDP уже имеет localhost-only digest-only managed endpoint. Managed endpoint и
legacy plaintext provisioning имеют раздельные флаги и по умолчанию выключены.
Runner-owned provisioner bearer относится только к legacy plaintext routes и
на managed ensure/revoke намеренно не действует. Managed caller contract —
Ed25519-signed Engine service request с exact audience, method, request target,
raw-body SHA-256, timestamp и одноразовым nonce; EDP хранит только public key и
fail-closed replay cache. Ни bearer, ни signing private key нельзя открывать
workflow, MCP или переиспользовать как shared credential. В source-кандидате
Engine добавлен отдельный server-derived plan/apply для exact пары
`ndcDataProductWriterApi` + custom Publish node; он не расширяет generic HTTP
safe-ref policy и не принимает provider/scope/credential identity, capability,
generation или service URL от MCP. До отдельного Engine deploy и runtime proof
текущий deployed MCP этого контракта не имеет. Поэтому отсутствие
совместимого reference — платформенный runtime gap, а не действие пользователя.
Ручной перенос capability из отдельно включаемого legacy endpoint через root/UI допустим только как аварийная
диагностика self-hosted контура; он не является acceptance-путём и не должен
закрепляться в пользовательской автоматизации.
Compose монтирует read-only trust directory
`/volume1/docker/nodedc-platform/trust/engine-managed-provisioner` в EDP; в нём
ожидается один regular, non-symlink файл `public-key.pem` с Ed25519 SPKI public
key. Файл не должен быть group/world-writable и должен читаться runtime UID
`11006`. Private key в этот каталог, Platform artifact, `.env` или EDP
контейнер не попадает. Canonical deploy runner теперь атомарно создаёт или
проверяет matching Ed25519 pair перед relevant Engine/EDP Compose apply: private
key остаётся `root:root 0400`, public trust — `root:11006 0440`; public-only
crash state, mismatch, symlink и неверный тип ключа отклоняются до запуска.
`EXTERNAL_DATA_PLANE_MANAGED_PROVISIONING_ENABLED` остаётся `false` на staging.
В отдельно подтверждённом activation window его переводят в `true`
непосредственно перед Platform EDP artifact; runner принимает deploy только
если `/healthz` явно вернул managed status `enabled`. Старый Engine в этот
момент ещё не имеет signer mount, поэтому usable caller появляется лишь после
следующего отдельно принятого Engine artifact.
EDP application artifact пересобирает и force-recreate только
`external-data-plane`. Уже работающий `external-data-plane-postgres` и его
Timescale volume runner не выбирает и не перезапускает; healthy database —
обязательная deploy prerequisite.
Canonical service/key/audience заданы в `.env.synology.example`; wire headers и
canonical signing payload описаны в `packages/external-provider-contract/README.md`.
До появления opaque compatible reference и успешного manual publish proof
Schedule Trigger не включать. Frozen legacy workflow и его credentials этот
процесс не изменяет.
## AI Hub relay-only deploy ## AI Hub relay-only deploy

4
infra/synology/apply-current-runtime.sh
View File

@ -38,7 +38,7 @@ echo "== ontology core image build =="
cd "${PLATFORM_DIR}/ontology-core" cd "${PLATFORM_DIR}/ontology-core"
"${DOCKER_BIN}" build --no-cache -t nodedc/ontology-core:local . "${DOCKER_BIN}" build --no-cache -t nodedc/ontology-core:local .
echo "== frozen legacy gelios gateway image build ==" echo "== gelios gateway image build =="
cd "${PLATFORM_DIR}/gelios-gateway" cd "${PLATFORM_DIR}/gelios-gateway"
"${DOCKER_BIN}" build --no-cache -t nodedc/gelios-gateway:local . "${DOCKER_BIN}" build --no-cache -t nodedc/gelios-gateway:local .
@ -53,7 +53,7 @@ echo "== ontology core health check =="
"${DOCKER_BIN}" exec nodedc-platform-ontology-core-1 sh -lc \ "${DOCKER_BIN}" exec nodedc-platform-ontology-core-1 sh -lc \
'node -e '"'"'fetch("http://127.0.0.1:18104/healthz").then(async (response) => { console.log(await response.text()); process.exit(response.ok ? 0 : 1); }).catch((error) => { console.error(error); process.exit(1); })'"'"'' 'node -e '"'"'fetch("http://127.0.0.1:18104/healthz").then(async (response) => { console.log(await response.text()); process.exit(response.ok ? 0 : 1); }).catch((error) => { console.error(error); process.exit(1); })'"'"''
echo "== frozen legacy gelios gateway health check ==" echo "== gelios gateway health check =="
"${DOCKER_BIN}" exec nodedc-platform-gelios-gateway-1 sh -lc \ "${DOCKER_BIN}" exec nodedc-platform-gelios-gateway-1 sh -lc \
'node -e '"'"'fetch("http://127.0.0.1:18105/healthz").then(async (response) => { console.log(await response.text()); process.exit(response.ok ? 0 : 1); }).catch((error) => { console.error(error); process.exit(1); })'"'"'' 'node -e '"'"'fetch("http://127.0.0.1:18105/healthz").then(async (response) => { console.log(await response.text()); process.exit(response.ok ? 0 : 1); }).catch((error) => { console.error(error); process.exit(1); })'"'"''

27
infra/synology/docker-compose.external-data-plane.yml
View File

@ -51,23 +51,12 @@ services:
# Migration-only shared-token routes stay closed in a canonical install. # Migration-only shared-token routes stay closed in a canonical install.
EXTERNAL_DATA_PLANE_LEGACY_INTAKE_ENABLED: ${EXTERNAL_DATA_PLANE_LEGACY_INTAKE_ENABLED:-false} EXTERNAL_DATA_PLANE_LEGACY_INTAKE_ENABLED: ${EXTERNAL_DATA_PLANE_LEGACY_INTAKE_ENABLED:-false}
NODEDC_INTERNAL_ACCESS_TOKEN: ${NODEDC_INTERNAL_ACCESS_TOKEN:-} NODEDC_INTERNAL_ACCESS_TOKEN: ${NODEDC_INTERNAL_ACCESS_TOKEN:-}
# Legacy manual one-time issuance uses this EDP-only bearer. Managed # Runner-owned secret file. This credential must never live in the
# Engine ensure/revoke never accepts it and uses the Ed25519 boundary # shared .env.synology that is inherited by unrelated services.
# below. Neither credential is exposed to a graph, MCP client or provider.
EXTERNAL_DATA_PLANE_PROVISIONER_TOKEN_FILE: /run/nodedc-secrets/external-data-plane-provisioner-token EXTERNAL_DATA_PLANE_PROVISIONER_TOKEN_FILE: /run/nodedc-secrets/external-data-plane-provisioner-token
# Disabled until the atomic native NDC L2 ensure-grant control-plane # This stays false until the dedicated Engine provisioner is deployed and
# operation is present. This legacy API returns plaintext capabilities; # receives the same read-only secret mount.
# a root/UI transfer is emergency diagnostics only. EXTERNAL_DATA_PLANE_PROVISIONING_ENABLED: "false"
EXTERNAL_DATA_PLANE_PROVISIONING_ENABLED: ${EXTERNAL_DATA_PLANE_PROVISIONING_ENABLED:-false}
# Digest-only, idempotent writer-binding ensure path. Enable only after
# Engine owns the matching private key and the public trust file exists.
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONING_ENABLED: ${EXTERNAL_DATA_PLANE_MANAGED_PROVISIONING_ENABLED:-false}
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_PUBLIC_KEY_FILE: /run/nodedc-trust/engine-managed-provisioner/public-key.pem
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_SERVICE_ID: ${EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_SERVICE_ID:-nodedc-engine}
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_KEY_ID: ${EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_KEY_ID:-engine-edp-managed-provisioner-v1}
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_AUDIENCE: ${EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_AUDIENCE:-nodedc-external-data-plane.managed-provisioning.v1}
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_MAX_SKEW_SECONDS: ${EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_MAX_SKEW_SECONDS:-60}
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_REPLAY_CACHE_MAX_ENTRIES: ${EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_REPLAY_CACHE_MAX_ENTRIES:-10000}
volumes: volumes:
- type: bind - type: bind
source: /volume1/docker/nodedc-platform/secrets/external-data-plane-provisioner/token source: /volume1/docker/nodedc-platform/secrets/external-data-plane-provisioner/token
@ -75,12 +64,6 @@ services:
read_only: true read_only: true
bind: bind:
create_host_path: false create_host_path: false
- type: bind
source: /volume1/docker/nodedc-platform/trust/engine-managed-provisioner
target: /run/nodedc-trust/engine-managed-provisioner
read_only: true
bind:
create_host_path: false
expose: expose:
- "18106" - "18106"
ports: ports:

4
infra/synology/docker-compose.platform-http.yml
View File

@ -184,8 +184,8 @@ services:
networks: networks:
- engine - engine
# Frozen legacy Gelios compatibility contour. Keep it reproducible and # The provider database is private. Gateway is the only component that can
# isolated; new provider integrations use the provider-neutral data plane. # reach it; the host bind is loopback-only for audited operator diagnostics.
gelios-postgres: gelios-postgres:
image: ${GELIOS_TIMESCALE_IMAGE:-timescale/timescaledb-ha:pg16.14-ts2.28.2-all} image: ${GELIOS_TIMESCALE_IMAGE:-timescale/timescaledb-ha:pg16.14-ts2.28.2-all}
restart: unless-stopped restart: unless-stopped

2
infra/synology/verify-current-runtime.sh
View File

@ -67,7 +67,7 @@ echo "== ontology core health check =="
"${DOCKER_BIN}" exec nodedc-platform-ontology-core-1 sh -lc \ "${DOCKER_BIN}" exec nodedc-platform-ontology-core-1 sh -lc \
'node -e '"'"'fetch("http://127.0.0.1:18104/healthz").then(async (response) => { console.log(await response.text()); process.exit(response.ok ? 0 : 1); }).catch((error) => { console.error(error); process.exit(1); })'"'"'' 'node -e '"'"'fetch("http://127.0.0.1:18104/healthz").then(async (response) => { console.log(await response.text()); process.exit(response.ok ? 0 : 1); }).catch((error) => { console.error(error); process.exit(1); })'"'"''
echo "== frozen legacy gelios gateway health check ==" echo "== gelios gateway health check =="
"${DOCKER_BIN}" exec nodedc-platform-gelios-gateway-1 sh -lc \ "${DOCKER_BIN}" exec nodedc-platform-gelios-gateway-1 sh -lc \
'node -e '"'"'fetch("http://127.0.0.1:18105/healthz").then(async (response) => { console.log(await response.text()); process.exit(response.ok ? 0 : 1); }).catch((error) => { console.error(error); process.exit(1); })'"'"'' 'node -e '"'"'fetch("http://127.0.0.1:18105/healthz").then(async (response) => { console.log(await response.text()); process.exit(response.ok ? 0 : 1); }).catch((error) => { console.error(error); process.exit(1); })'"'"''

289
packages/external-provider-contract/README.md
View File

@ -5,7 +5,7 @@
исполняемый connector code. `src/index.mjs` даёт dependency-free проверку исполняемый connector code. `src/index.mjs` даёт dependency-free проверку
минимальных v1 contracts; она запускается через `npm run check`. минимальных v1 contracts; она запускается через `npm run check`.
Каждый NDC L2 connector instance должен поставлять совместимые versioned Каждый L2 connector instance должен поставлять совместимые versioned
артефакты: артефакты:
```text ```text
@ -21,47 +21,16 @@ command-catalog (metadata only until a red command gateway is approved)
``` ```
`provider-manifest` — декларативный template-артефакт: provider ID, ontology `provider-manifest` — декларативный template-артефакт: provider ID, ontology
revision, NDC L2 template version, capability catalog и data-product IDs. Он не revision, L2 template version, capability catalog и data-product IDs. Он не
является tenant connection, не содержит endpoint/URL, credential reference, является tenant connection, не содержит endpoint/URL, credential reference,
secret, ручной scope или executable provider code. Concrete non-secret secret, ручной scope или executable provider code. Concrete non-secret
connection profile принадлежит и версионируется вместе с конкретным NDC L2 connection profile принадлежит и версионируется вместе с конкретным L2
workflow; caller привязывает только opaque provider credential reference, а workflow; Engine только привязывает к нему opaque credential reference и grant.
внутренний workload grant разрешает control plane.
## Versioned provider packages
`providers/<provider>/<major>` — устанавливаемая contract/data единица, а не
runtime service и не custom node. Она объединяет manifest, auth-mode metadata,
capability catalog, field policies, collection profiles, Data Product
definitions, semantic mapping contracts, provider-neutral NDC L2 template
descriptor и synthetic fixtures. Package не содержит tenant/account state,
credential values или исполняемый provider daemon.
`instantiateL2Connection` создаёт произвольное число connection instances из
одного immutable package. Каждый instance получает собственные tenant,
connection и opaque provider credential reference, но не требует изменения
Platform source. Внутренние workload bindings объявляются отдельно как
`control_plane_managed` и не являются caller input. Канонический collection
scope — `all_visible_to_credential`: выбранная
read-capability передаёт все entities, которые provider возвращает для
привязанного credential. Unit allowlist и provider-group filter на collection
boundary запрещены; сортировка и видимость принадлежат Data Product consumer и
Foundry.
Первый production-shaped package — `providers/gelios/v1`. Он фиксирует реальный
Gelios REST `GET /api/v1/units` transport contract и точный output
`fleet.positions.current.v1@1.0.0` с revision
`ontology.map.moving_object.v1`. Все Data Product fields используют snake_case.
Его optional `tokenLifecycle` точно описывает два provider-issued artifacts —
`access` и `refresh`: request использует `access`, а refresh пока имеет режим
`operator_managed`. Это metadata без secret values и без заявления о
реализованном автоматическом refresh.
## Проверяемые v1 contracts ## Проверяемые v1 contracts
- `Connection` — provider instance, tenant scope и *ссылка* на provider - `Connection` — provider instance, tenant scope и *ссылка* на credential в
credential в NDC L2 Credentials. Любые token/secret/password-like поля Engine. Любые token/secret/password-like поля запрещены.
запрещены; publisher представлен system-managed declaration/status без ref.
- `Collection Profile` — явная policy сбора. `manual` не может скрыто содержать - `Collection Profile` — явная policy сбора. `manual` не может скрыто содержать
polling interval; `realtime` требует interval не чаще одного раза в секунду. polling interval; `realtime` требует interval не чаще одного раза в секунду.
- `Data Product` — нормализованный versioned output с semantic types, полями и - `Data Product` — нормализованный versioned output с semantic types, полями и
@ -69,18 +38,18 @@ Gelios REST `GET /api/v1/units` transport contract и точный output
- `Intake Batch` — canonical **scoped** record, который External Data Plane - `Intake Batch` — canonical **scoped** record, который External Data Plane
валидирует и сохраняет: source, contract revision, idempotency, restricted валидирует и сохраняет: source, contract revision, idempotency, restricted
raw envelope и canonical facts. Его `source` содержит `providerId`, raw envelope и canonical facts. Его `source` содержит `providerId`,
`tenantId` и `connectionId`. Writer-bound NDC L2 request намеренно не является `tenantId` и `connectionId`. Writer-bound L2 request намеренно не является
готовым `Intake Batch`: Data Plane сначала materializes scope и лишь затем готовым `Intake Batch`: Data Plane сначала materializes scope и лишь затем
применяет этот contract. В canonical record нет provider field mapping, применяет этот contract. В canonical record нет provider field mapping,
entity allowlist, token или renderer data. Inline `raw.payload` в v1 entity allowlist, token или renderer data. Inline `raw.payload` в v1
запрещён: если нужна provenance-ссылка, connector передаёт restricted запрещён: если нужна provenance-ссылка, connector передаёт restricted
`raw.ref` вместе с hash. Отдельный raw-vault может быть добавлен только `raw.ref` вместе с hash. Отдельный raw-vault может быть добавлен только
отдельным ADR и не становится частью NDC L2 → Data Plane wire boundary. отдельным ADR и не становится частью L2 → Data Plane wire boundary.
- `NDC Foundry Binding` — адресует data product только в конкретную цепочку - `NDC Foundry Binding` — адресует data product только в конкретную цепочку
`Foundry Application → Page → approved slot`; `templateId` можно сохранить `Foundry Application → Page → approved slot`; `templateId` можно сохранить
как дополнительную типизацию, но он не заменяет `applicationId` и `pageId`. как дополнительную типизацию, но он не заменяет `applicationId` и `pageId`.
В binding запрещены provider transport, endpoint и credential reference. В binding запрещены provider transport, endpoint и credential reference.
- `Provider Manifest` — статическое описание NDC L2 connector template, capability - `Provider Manifest` — статическое описание L2 connector template, capability
catalog и ontology/data-product contracts. Оно не может содержать tenant, catalog и ontology/data-product contracts. Оно не может содержать tenant,
connection, credential, secret или provider transport. connection, credential, secret или provider transport.
@ -125,14 +94,13 @@ PostgreSQL `integer` `0..2147483647`. Эти ограничения нельзя
Manifest, connection/collection profiles, data-product definition и Foundry Manifest, connection/collection profiles, data-product definition и Foundry
binding используют fail-closed allowed-key schemas. Неизвестные поля, а также binding используют fail-closed allowed-key schemas. Неизвестные поля, а также
secret-like имена или значения (включая `ndc_edpwb_`, `ndc_edprb_` и secret-like имена или значения (включая `ndc_edpwb_`/`ndc_edprb_`) отклоняются
`ndc_edppr_`) отклоняются
на общей границе. на общей границе.
Все private custom nodes NODE.DC поставляются package Все private custom nodes NODE.DC поставляются package
`platform/packages/n8n-nodes-ndc`. Их display name обязан начинаться с `NDC `, `platform/packages/n8n-nodes-ndc`. Их display name обязан начинаться с `NDC `,
а runtime type — с `n8n-nodes-ndc.`; package называется строго а runtime type — с `n8n-nodes-ndc.`; package называется строго
`n8n-nodes-ndc`. Provider-specific adapters остаются NDC L2 workflow logic и не `n8n-nodes-ndc`. Provider-specific adapters остаются L2 workflow logic и не
становятся custom nodes или ветками Data Plane. Эти инварианты проверяются становятся custom nodes или ветками Data Plane. Эти инварианты проверяются
package test. package test.
@ -142,72 +110,76 @@ declarative provider artifact и не runtime transport: команда соде
application/page/binding/data-product projection и idempotency key, а право на application/page/binding/data-product projection и idempotency key, а право на
операцию извлекается Foundry из отдельного opaque workload grant. операцию извлекается Foundry из отдельного opaque workload grant.
## Native NDC L2 Credentials ## Engine opaque credential sink
Provider auth и внутренние workload capabilities используют один общий `src/engine-credential-sink.mjs` задаёт dependency-free server-to-server v1
секретный boundary — native Credentials NDC L2, — но являются разными границу для доставки трёх workload capabilities в Engine Credentials:
credential domains. После сохранения ядро владеет secret, а graph и MCP
используют только opaque reference.
Gelios выдаёт ровно access token и refresh token. Текущий credential type - `external-data-plane.writer` → точная нода
`httpBearerAuth` использует access token для HTTP request. Автоматический обмен `n8n-nodes-ndc.ndcDataProductPublish` / `ndcDataProductWriterApi`;
refresh → access в текущем runtime не доказан и поэтому не заявлен: package - `external-data-plane.reader` → точная нода
фиксирует `refreshMode: operator_managed`. Название credential с текстом вроде `n8n-nodes-ndc.ndcDataProductRead` / `ndcDataProductReaderApi`;
`read access` является лишь локальной меткой; read-классификацию задают - `foundry.binding` → точная нода
разрешённые endpoint/method в capability catalog и workflow policy, а не scope `n8n-nodes-ndc.ndcFoundryBinding` / `ndcFoundryBindingApi`.
самого access token. В deployed Engine exact method/path policy ещё не
подключена: текущая HTTP safe-ref граница проверяет host. Поэтому catalog
classification остаётся декларативной до capability-bound Engine/MCP proof.
Для общего Data Product transport используются `ndcDataProductWriterApi`, Provision request фиксирует `workflowId`, `workflowRevision`, `nodeId`, runtime
`ndcDataProductReaderApi` и `ndcFoundryBindingApi`. Это внутренние NDC node type, credential type, grant ID, expiry и issuer policy hash. Aggregate
capabilities, не Gelios tokens и не часть Gelios token lifecycle. Их значения `transaction.policyHash` детерминированно считается по всему secret-free
не являются частью provider package и не передаются между workflow nodes как descriptor; подмена любой цели или policy ломает валидацию. Единственное поле,
данные. которое переносит plaintext capability, — `bindings[].material.value`; request
нельзя писать в логи, traces, очередь или audit.
Connection caller передаёт только provider credential reference. Publisher Provision и rollback envelopes действуют не более 15 минут: sink отклоняет
role остаётся credential binding для `NDC Data Product Publish`, но template истёкшие запросы и допускает максимум 60 секунд положительного clock skew.
маркирует его `management: control_plane_managed`: ни writer secret, ни writer Receipt повторно проверяет freshness относительно собственного `processedAt`,
reference не входят в connection parameters. Instantiation возвращает чтобы старый запрос нельзя было применить или подтвердить через replay.
декларативный `systemBindings.publisher` с desired state и unresolved status;
дальше его разрешает trusted control plane.
Canonical managed EDP writer capability генерируется и сохраняется внутри Каждый binding содержит `capabilityDigest = sha256(material.value)`: Engine
Engine credential boundary, а EDP получает только digest. Старые manual самостоятельно хеширует полученный plaintext и сравнивает digest. Aggregate
writer/reader и Foundry issuance paths могут возвращать capability один раз `policyHash` включает этот digest и issuer identity, а provision transaction
trusted control-plane caller и остаются отдельно закрытыми compatibility несёт обязательную Ed25519 attestation. Engine принимает её только по
границами; MCP и пользователь получают только opaque reference/status. allowlisted `serviceId:keyId`; заменить capability и пересчитать обычный hash
Запрещены password input без приватного issuer key невозможно.
как пользовательский acceptance-путь, graph parameters, connection/profile
files, env, Ops, logs и traces.
Broad credential sink/resolver/daemon и provider-specific provisioning Sink обязан выполнять `rollback-all`: сначала проверить весь request и точное
запрещены. Требуется узкий generic `ensure data-product publish grant` contract состояние graph, затем создать credentials в staging, атомарно привязать весь
с server-derived scope, idempotency, CAS/crash recovery и audit. Текущий набор и только после commit вернуть opaque `credentialRef`. При любой ошибке
deployed Engine MCP этой операции ещё не имеет; root/UI transfer допустим новые credentials удаляются, а прежние bindings остаются без изменений.
только как emergency self-hosted diagnostics, не как user journey. `rollback-failed` означает карантин и ручное восстановление, но никогда не
возвращает частичные credential refs.
## NDC L2 private-extension management Receipt, rollback request/receipt и audit имеют отдельные strict schemas. Они
не способны вернуть capability material; audit хранит только hash opaque
credential reference. Explicit rollback адресует committed transaction через
`transactionId`, `policyHash` и hash committed receipt, поэтому не может
случайно откатить другой набор. Реализация sink принадлежит Engine и не даёт
Platform/Codex доступа к Engine core, runtime files или plaintext credential
storage.
## Engine private-extension management
`src/engine-private-extension.mjs` задаёт строгую Platform-side границу для `src/engine-private-extension.mjs` задаёт строгую Platform-side границу для
NDC L2-owned активации проверенного `n8n-nodes-ndc` release. Контракт не Engine-owned активации проверенного `n8n-nodes-ndc` release. Контракт не
устанавливает package и не меняет NDC L2: он фиксирует async устанавливает package и не меняет Engine: он фиксирует async
`plan -> apply receipt -> operation/status` protocol, где `apply` обязан `plan -> apply receipt -> operation/status` protocol, где `apply` обязан
быстро вернуть `queued` и `operationId`, а долгий recreate/acceptance быстро вернуть `queued` и `operationId`, а долгий recreate/acceptance
отслеживается отдельно. отслеживается отдельно.
Активация и rollback требуют отдельной глобальной capability Активация и rollback требуют отдельной глобальной capability
`engine.private-extension.manage`. Обычные NDC L1/NDC L2 grants её не дают. Чтение `engine.private-extension.manage`. Обычные L1/L2 grants её не дают. Чтение
состояния допускает `engine.private-extension.read` или manage-capability. состояния допускает `engine.private-extension.read` или manage-capability.
Запрос выбирает только allowlisted package, digest-bound `releaseId` и Запрос выбирает только allowlisted package, digest-bound `releaseId` и
`packageSha256`; caller не передаёт host path, package bytes, Compose service, `packageSha256`; caller не передаёт host path, package bytes, Compose service,
shell command или credential material. Plan живёт не более 15 минут, является shell command или credential material. Plan живёт не более 15 минут, является
single-use и применяется только с тем же idempotency key и plan hash. single-use и применяется только с тем же idempotency key и plan hash.
Runtime transition использует platform-managed community-package loader для Runtime transition для n8n 2.3.2 зафиксирован как community-package loader по
immutable `n8n-nodes-ndc` release; private runtime paths и environment settings `/home/node/.n8n/nodes/node_modules/n8n-nodes-ndc`, а не как
не являются частью публичного provider contract: `N8N_CUSTOM_EXTENSIONS` или `CUSTOM.*` loader:
- `N8N_COMMUNITY_PACKAGES_ENABLED=true`,
`N8N_COMMUNITY_PACKAGES_PREVENT_LOADING=false`,
`N8N_REINSTALL_MISSING_PACKAGES=false`;
- Deploy/Run quiesced and execution queue drained before the version switch; - Deploy/Run quiesced and execution queue drained before the version switch;
- read-only mount and atomic current/recovery state; - read-only mount and atomic current/recovery state;
- force-recreate main, every worker and every webhook instance as one version - force-recreate main, every worker and every webhook instance as one version
@ -217,25 +189,24 @@ immutable `n8n-nodes-ndc` release; private runtime paths и environment settings
Любая ошибка после switch запускает automatic rollback и повторный Любая ошибка после switch запускает automatic rollback и повторный
force-recreate/acceptance. Ошибка самого rollback переводит runtime в force-recreate/acceptance. Ошибка самого rollback переводит runtime в
`quarantined`. Immutable release и существующие NDC L2 Credentials `quarantined`. Immutable release и существующие Engine Credentials
сохраняются. Для первой активации предыдущим проверенным состоянием является сохраняются. Для первой активации предыдущим проверенным состоянием является
`n8n-nodes-ndc.inactive/v1`: rollback в этот baseline удаляет package из `n8n-nodes-ndc.inactive/v1`: rollback в этот baseline удаляет package из
loader surface, но не удаляет credentials. loader surface, но не удаляет credentials.
Public Ops gateway уже умеет прозрачно передавать эти операции через Public Ops gateway уже умеет прозрачно передавать эти операции через
`/engine/mcp`, если NDC L2 реализует соответствующие MCP tools. Так как `/engine/mcp`, если Engine реализует соответствующие MCP tools. Так как
gateway имеет 30-second upstream timeout, side effect остаётся асинхронным; gateway имеет 30-second upstream timeout, side effect остаётся асинхронным;
отдельный public REST proxy для management boundary не требуется. отдельный public REST proxy для management boundary не требуется.
`providers/gelios/v1/fixtures` содержит synthetic provider response и ожидаемый Пример `examples/gelios-positions-current.v1.mjs` — provider-specific fixture
publish contract без customer, tenant identity или credential material. без customer, tenant identity или credential material.
## Ownership ## Ownership
- `platform/packages/external-provider-contract` — общий контракт и schemas. - `platform/packages/external-provider-contract` — общий контракт и schemas.
- NDC L2 — provider API adapter: fetch, pagination, batching, - NDC Agent L2 — provider API adapter: fetch, pagination, batching,
semantic mapping, collection profile и ссылка на credential в NDC L2 semantic mapping, collection profile и ссылка на credential в Engine.
Credentials.
- Platform External Data Plane — provider-neutral intake, raw retention, - Platform External Data Plane — provider-neutral intake, raw retention,
canonical facts, current/history projections и scoped read products. Он не canonical facts, current/history projections и scoped read products. Он не
знает provider fields, customer/business filters или renderer rules. знает provider fields, customer/business filters или renderer rules.
@ -251,100 +222,46 @@ publish contract без customer, tenant identity или credential material.
## Mandatory connection boundary ## Mandatory connection boundary
`connection` принадлежит одному tenant/client context и содержит только `connection` принадлежит одному tenant/client context и содержит только
ссылку на provider secret, утверждённый capability scope, collection profile, ссылку на секрет, утверждённый capability scope, collection profile, field
field policy и retention policy. Во всех runtime records Data Plane сохраняет минимум policy и retention policy. Во всех runtime records Data Plane сохраняет минимум
`tenant_id`, `connection_id`, `provider_id`, `observed_at`, `received_at` и `tenant_id`, `connection_id`, `provider_id`, `observed_at`, `received_at` и
provenance/version там, где это применимо. provenance/version там, где это применимо.
Provider credential reference текущего Gelios request адресует access token; Writer token — отдельный EDP runtime credential, а не поле `Connection`,
наличие provider-issued refresh token описывается `tokenLifecycle`, но его value profile, provider manifest или L2 graph.
не входит в `Connection`. Writer capability — отдельный внутренний EDP runtime
credential, а не Gelios token и не caller-supplied поле `Connection`, profile,
provider manifest или NDC L2 graph. Connection artifact содержит только
system-managed publisher declaration/status без credential reference.
## Scoped writer binding ## Scoped writer binding
Writer binding — EDP-owned runtime security state, а не versioned artifact Writer binding — EDP-owned runtime security state, а не versioned artifact
provider manifest или connection profile. Он фиксирует `tenantId`, provider manifest или connection profile. Он фиксирует `tenantId`,
`connectionId`, `providerId`, `allowedDataProductIds`, active/revoked state и `connectionId`, `providerId`, `allowedDataProductIds`, active/revoked state и
`expiresAt`. NDC L2 не может редактировать binding или задавать его scope; `expiresAt`. L2 не может редактировать binding или задавать его scope;
изменение любого scope-поля либо TTL создаёт новый binding, а прежний binding изменение любого scope-поля либо TTL создаёт новый binding, а прежний binding
можно только rotate/revoke. можно только rotate/revoke.
Binding создаёт, сохраняет и привязывает только trusted control plane. Caller Новый L2 вызывает `POST /internal/data-plane/v1/intake/writer-bound` с
connection instance не принимает `writerCredentialRef`; publisher role в L2 `Authorization: Bearer <writer-token>` и unscoped envelope. В `source`
template является обязательным `control_plane_managed` binding requirement. разрешён только `providerId`; `tenantId`, `connectionId` и
`x-nodedc-*-id` headers запрещены. EDP проверяет token, binding, provider и
`contract.dataProductId`, затем materializes immutable canonical scope и
сохраняет обычный `Intake Batch`. Caller-provided scope отклоняется, а не
доверяется и не объединяется с binding.
Новый NDC L2 вызывает Plaintext writer token возвращается только trusted provisioner при создании
`POST /internal/data-plane/v1/data-products/:dataProductId/publish` с opaque или rotation. Provisioner помещает его непосредственно в opaque Engine
writer capability и envelope `nodedc.data-product.publish/v1`. В body есть credential, доступный назначенному L2. EDP хранит только hash token и binding
только batch identity и canonical facts; `source`, provider, tenant, metadata; token запрещён в connection/profile/manifest, L2 graph, logs/traces,
connection, ontology revision, product version и persistence policy запрещены. raw payload, Foundry и UI.
EDP проверяет binding и path `dataProductId`, materializes immutable scope и
contract из server-owned registries, затем сохраняет canonical batch.
Caller-provided scope отклоняется, а не доверяется и не объединяется с
binding.
Plaintext writer capability запрещена в provider package, connection/profile, Provisioning routes принимают только отдельный secret file, созданный
NDC L2 graph, files, environment, logs/traces, raw payload, Foundry и MCP. root-owned deploy runner:
В canonical пути она генерируется и сохраняется внутри native Engine credential `/volume1/docker/nodedc-platform/secrets/external-data-plane-provisioner/token`.
boundary; EDP получает только SHA-256 digest, а workflow — только opaque Он read-only монтируется только в EDP и позднее — в dedicated Engine
reference/status. Старые ручные EDP create/rotate endpoints с one-time provisioner, работающий под выделенным UID/GID `11006`; это не `.env` value,
plaintext response остаются отдельно выключенным compatibility-механизмом и не не `NODEDC_INTERNAL_ACCESS_TOKEN` и не provider credential. Пока generic Engine
являются пользовательским или acceptance-путём. provisioner не развёрнут, `EXTERNAL_DATA_PLANE_PROVISIONING_ENABLED=false`, а
create/rotate/revoke routes отвечают `503` и не делают fallback к shared
Legacy manual one-time issuance использует отдельный EDP-only runner-managed internal bearer.
bearer. Digest-only managed ensure этот bearer не принимает: Engine подписывает
каждый control-plane запрос отдельным Ed25519 service key, EDP хранит только
публичный ключ. Оба пути имеют независимые флаги и по умолчанию выключены.
Generic `ensure data-product publish grant` генерирует и сохраняет capability в
native Engine credential boundary, передаёт EDP её digest и возвращает только
opaque ref/status. Shared internal bearer и Gelios access/refresh token для
этого запрещены. Deployed Engine MCP умеет читать
opaque refs и bind только HTTP Request credentials; native
`ndcDataProductWriterApi` к custom Publish node этим policy не привязывается.
Поэтому ensure + exact private-node bind остаются реальным platform gap.
Root/UI перенос — только аварийная диагностика.
Control-plane EDP boundary — idempotent
`PUT /internal/data-plane/v1/writer-bindings/by-key/:bindingKey`. Engine создаёт
capability внутри native credential boundary и отправляет только её SHA-256
`capabilityDigest`, immutable scope, `generation` и expiry. Одинаковый
`bindingKey + generation + request hash` возвращает тот же binding без нового
secret; несовпадающий retry получает `409`. Rotation использует новую
generation/credential, поэтому старый binding можно оставить активным до
успешного node bind и затем явно revoke.
Managed revoke выполняется идемпотентно по точным `bindingKey + generation`
через managed-only route; для него не открывается legacy plaintext API.
Managed route включается только через
`EXTERNAL_DATA_PLANE_MANAGED_PROVISIONING_ENABLED`; legacy plaintext routes —
через отдельный `EXTERNAL_DATA_PLANE_PROVISIONING_ENABLED`.
Managed auth wire contract —
`nodedc.external-data-plane.managed-provisioner-request/v1`. Engine отправляет
ровно по одному header:
`x-nodedc-engine-service-id`, `x-nodedc-engine-key-id`,
`x-nodedc-request-audience`, `x-nodedc-request-timestamp`,
`x-nodedc-request-nonce`, `x-nodedc-content-sha256` и
`x-nodedc-request-signature`. Timestamp — canonical UTC ISO с миллисекундами,
nonce и signature — unpadded canonical base64url, body hash — lowercase hex
SHA-256 от точных переданных bytes. Подписываются UTF-8 bytes результата
`JSON.stringify` объекта с полями строго в порядке `schemaVersion`, `audience`,
`serviceId`, `keyId`, `method`, `path`, `timestamp`, `nonce`, `bodySha256`.
`method` и request target `path` (включая query) должны совпадать побайтно.
Canonical identity: service `nodedc-engine`, key
`engine-edp-managed-provisioner-v1`, audience
`nodedc-external-data-plane.managed-provisioning.v1`. EDP проверяет bounded
clock skew и одноразовый nonce в bounded fail-closed replay cache; bearer
fallback для managed routes отсутствует, любой `Authorization` header на них
отклоняется.
EDP импортирует только `@nodedc/external-provider-contract/data-plane`.
Artifact/image allowlist для этого subpath содержит contract version, intake,
publish/snapshot/patch validators и scrub policy; provider packages, mappings,
fixtures и tests физически не входят в EDP runtime и не являются причиной его
restart.
## Legacy intake migration ## Legacy intake migration
@ -354,22 +271,22 @@ existing writers. Он принимает только canonical scoped `Intake
`x-nodedc-tenant-id` и `x-nodedc-connection-id`. Writer token этот route не `x-nodedc-tenant-id` и `x-nodedc-connection-id`. Writer token этот route не
заменяет. заменяет.
Для миграции: Platform обеспечивает scoped writer binding и native credential, Для миграции: создать writer binding, один раз записать выданный token в Engine
затем через Engine MCP новый workflow branch переключается на Data Product credential, переключить L2 на `/intake/writer-bound`, удалить tenant/connection
publish route, удаляет tenant/connection из body и headers и проверяет из body и headers, проверить успешный intake, затем удалить у L2 legacy
успешный publish. Существующий legacy workflow/credential остаётся неизменным, credential. Новый или migrated writer не должен fallback-иться на legacy route
пока владелец отдельно не решит его вывести; новый writer не должен после ошибки writer-bound intake. Когда migrated все writers, legacy route и
fallback-иться на legacy route после ошибки publish. его shared-token access удаляются.
## Intake boundary ## Intake boundary
NDC L2 отправляет в общий Data Plane publish request, а не SQL-запрос в общие L2 отправляет в общий Data Plane `Intake Batch`, а не SQL-запрос в общие
таблицы. Platform проверяет boundary и сохраняет raw/history/current таблицы. Platform проверяет boundary и сохраняет raw/history/current
projections; semantic mapping остаётся в NDC L2. Список конкретных source IDs не projections; semantic mapping остаётся в L2. Список конкретных source IDs не
является частью connection scope: scope выбирает read-capability API, а является частью connection scope: scope выбирает read-capability API, а
visibility решает data-product consumer. visibility решает data-product consumer.
Inline `raw.payload` запрещён: NDC L2 не может записать в Data Plane полный ответ Inline `raw.payload` запрещён: L2 не может записать в Data Plane полный ответ
provider-а или произвольную строку. Пока отдельный raw-vault не утверждён, provider-а или произвольную строку. Пока отдельный raw-vault не утверждён,
batch либо не содержит `raw`, либо содержит только restricted `raw.ref` и hash. batch либо не содержит `raw`, либо содержит только restricted `raw.ref` и hash.
Secret-like keys и распознаваемые bearer/JWT/writer-token values в любом участке Secret-like keys и распознаваемые bearer/JWT/writer-token values в любом участке
@ -382,6 +299,4 @@ service делает sweep expired envelopes при старте и по рас
Capabilities классифицируются как `read`, `metadata`, `write`, `destructive` Capabilities классифицируются как `read`, `metadata`, `write`, `destructive`
или `unknown`. Только `read` и согласованные `metadata` могут попасть в или `unknown`. Только `read` и согласованные `metadata` могут попасть в
collector. `write` и `destructive` остаются каталогизированными, но не имеют collector. `write` и `destructive` остаются каталогизированными, но не имеют
transport route в read adapter. Эта классификация относится к разрешённому transport route в read adapter.
method/endpoint workflow, а не утверждает, что provider access token имеет
read-only scope.

11
packages/external-provider-contract/examples/gelios-positions-current.v1.mjs
View File

@ -8,14 +8,15 @@ export const geliosPositionsCurrentExample = Object.freeze({
version: "1.0.0", version: "1.0.0",
ontology: { ontology: {
packageId: "gelios", packageId: "gelios",
revision: "ontology.gelios.v1", revision: "gelios.positions.v1",
}, },
l2Template: { l2Template: {
id: "gelios.positions.current.l2.v1", id: "gelios.positions.current",
version: "1.0.0", version: "1.0.0",
}, },
capabilities: [ capabilities: [
{ id: "gelios.units.current.read", classification: "read" }, { id: "gelios.units.current.read", classification: "read" },
{ id: "gelios.units.command.write", classification: "write" },
], ],
dataProductIds: ["fleet.positions.current.v1"], dataProductIds: ["fleet.positions.current.v1"],
}, },
@ -24,10 +25,10 @@ export const geliosPositionsCurrentExample = Object.freeze({
id: "gelios.sample-fleet", id: "gelios.sample-fleet",
providerId: "gelios", providerId: "gelios",
tenantId: "sample-tenant", tenantId: "sample-tenant",
credentialRef: { owner: "ndc_l2_credentials", reference: "ndc-credref:provider-example-0001" }, credentialRef: { owner: "engine", reference: "engine-credential-reference" },
scope: { scope: {
capabilityIds: ["gelios.units.current.read"], capabilityIds: ["gelios.units.current.read"],
fieldPolicyId: "gelios.positions.current.fields.v1", fieldPolicyId: "fleet.position.display.v1",
}, },
}, },
collectionProfile: { collectionProfile: {
@ -35,7 +36,7 @@ export const geliosPositionsCurrentExample = Object.freeze({
id: "gelios.positions.realtime", id: "gelios.positions.realtime",
connectionId: "gelios.sample-fleet", connectionId: "gelios.sample-fleet",
mode: "realtime", mode: "realtime",
schedule: { intervalMs: 10000 }, schedule: { intervalMs: 15000 },
capabilityIds: ["gelios.units.current.read"], capabilityIds: ["gelios.units.current.read"],
dataProductId: "fleet.positions.current.v1", dataProductId: "fleet.positions.current.v1",
}, },

8
packages/external-provider-contract/package.json
View File

@ -3,12 +3,8 @@
"version": "0.1.0", "version": "0.1.0",
"private": true, "private": true,
"type": "module", "type": "module",
"exports": { "exports": "./src/index.mjs",
".": "./src/index.mjs",
"./data-plane": "./src/data-plane.mjs",
"./providers/*": "./providers/*/index.mjs"
},
"scripts": { "scripts": {
"check": "node test/contract.test.mjs && node test/data-product.test.mjs && node test/provider-package.test.mjs && node test/engine-private-extension.test.mjs" "check": "node test/contract.test.mjs && node test/data-product.test.mjs && node test/engine-credential-sink.test.mjs && node test/engine-private-extension.test.mjs"
} }
} }

105
packages/external-provider-contract/providers/gelios/v1/README.md
View File

@ -1,105 +0,0 @@
# Gelios provider package v1
Это versioned contract/data package для первого живого provider connection в
NODE.DC. Он не является runtime service, custom node, account configuration или
хранилищем telemetry. Provider-specific request и mapping исполняются внутри
изолированного NDC L2 workflow; Platform получает только provider-neutral Data
Product.
Package фиксирует:
- manifest `gelios.provider.v1@1.0.0`;
- точный provider token lifecycle: Gelios выдаёт `access` и `refresh`, а
HTTP request использует `access`;
- safe-read capability `gelios.units.current.read`
(`GET https://api.geliospro.com/api/v1/units` with an opaque bearer
credential);
- dynamic scope `all_visible_to_credential` без unit/group allowlist;
- realtime profile с интервалом 10 секунд и manual profile;
- fail-closed field policy: неизвестные и dynamic fields отбрасываются до
классификации, hardware IDs, phone/address, raw params и sensor payloads не
публикуются;
- mapping в `map.moving_object`;
- стабильный source ID `gelios-unit-<sourceUnitId>` с точным сохранением
строкового provider ID;
- explicit derivation rules для `position_valid`, `operational_status` и
`quality_flags`, включая `no_position` и stale threshold 60 секунд;
- точный Data Product `fleet.positions.current.v1@1.0.0`, revision
`ontology.map.moving_object.v1`, с 13 разрешёнными snake_case fields;
- provider-neutral NDC L2 template descriptor из пяти boundary steps:
trigger → request → extract → map → `NDC Data Product Publish`;
- publish credential binding с `management: control_plane_managed`, без
caller-supplied writer reference;
- synthetic source/publish fixture, включая видимый объект без координат.
## Authentication boundary
Gelios выдаёт ровно два provider secret artifact: access token и refresh
token. Это не отдельные read/write tokens. Текущий NDC L2 HTTP binding типа
`httpBearerAuth` подставляет access token в `Authorization`; автоматический
refresh этим runtime пока не доказан, поэтому package явно фиксирует
`refreshMode: operator_managed`. Ни access, ни refresh value не попадает в
package, graph, fixture, MCP, Ops или trace.
Название native credential, например `Gelios — Robot2B — read access`, является
только операторской меткой. `read` в `gelios.units.current.read` —
классификация разрешённого workflow endpoint `GET /api/v1/units`, а не scope
access token. Метка credential не создаёт отдельный «read token» и не сужает
права, которые фактически выдал Gelios.
Unit без last message не исчезает: mapping использует collection receive time
как `observedAt` fallback и публикует subject без geometry с
`position_valid=false`/`operational_status=no_position`.
Префикс source ID для v1 — строго `gelios-unit-`. Сокращённый `unit-` не
является каноническим: он создал бы второй entity key рядом с уже сохранённым
Gelios subject. Значение provider ID после префикса преобразуется только в
строку; ведущие нули и остальные значимые символы не нормализуются.
`fleet.positions.current.v1` ограничен 5000 current entities. Если credential
видит больше, NDC L2 не должен отбрасывать «лишние» units: package/profile
требует остановить Deploy до новой partitioned Data Product revision. Полное
автоматическое enforcement этого правила остаётся gate будущего
package-to-graph compiler/runtime validator.
Один package обслуживает любое число accounts. Для каждого account вызывается
`instantiateL2Connection` с новым `tenantId`, `connectionId`, provider
credential reference. Это единственная credential reference, которую передаёт
caller: она адресует Gelios access binding в native NDC L2 Credentials.
Внутреннюю capability для публикации в External Data Plane выдаёт и сохраняет
control plane; caller не передаёт writer secret или `writerCredentialRef`.
Результат instantiation содержит только provider ref и декларативный
`systemBindings.publisher` со статусом `unresolved`, который Platform должна
разрешить в native credential binding. Эта capability не выдаётся Gelios, не
является третьим Gelios token и не описывает права provider account. Добавление
account не меняет Platform source.
Расширение Gelios происходит новой версией package: сначала подтверждаются
provider capability/fields, затем обновляются ontology, field policy, mapping,
fixtures и tests. Пользовательские workflow, которые уже покрываются
существующим catalog, не требуют участия разработчика Platform.
Технический runtime type publish boundary остаётся
`n8n-nodes-ndc.ndcDataProductPublish`; это идентификатор общего NDC package, а
не Gelios-specific node.
## Текущая готовность
Package, ontology links и synthetic fixtures проверены в source. Production
Engine L2 target использует зарегистрированный Gelios Bearer credential на
точном safe-read endpoint и managed `ndcDataProductWriterApi` generation g3;
оба binding записаны без раскрытия secret values. Canonical graph содержит
provider read, extraction, mapping и native `NDC Data Product Publish`.
Fresh production execution `881722` завершился успешно и опубликовал
`fleet.positions.current.v1`; managed writer g3 принят и имеет state `current`.
Это доказывает manual provider read → canonical Data Product path. Внутренняя
writer capability остаётся NODE.DC capability и не может быть заменена access
или refresh token Gelios.
Не доказаны: автоматический refresh, Schedule Trigger 10 секунд, несколько
последовательных scheduled cycles, retry/backoff, sampled history/retention на
continuous path и Foundry snapshot/patch → moving pins E2E. Provider package
остаётся declarative: MCP package discovery и package/profile → L2 graph
compiler пока отсутствуют, поэтому первый graph собран через общие Engine MCP
graph tools.

74
packages/external-provider-contract/providers/gelios/v1/fixtures/units-current.fixture.mjs
View File

@ -1,74 +0,0 @@
import { DATA_PRODUCT_PUBLISH_SCHEMA_VERSION } from "../../../../src/data-product.mjs";
export const geliosUnitsCurrentFixtureV1 = Object.freeze({
sourceResponse: {
units: [{
id: 1001,
name: "Synthetic unit 1001",
hw_id: "restricted-fixture-value",
phone: "restricted-fixture-value",
lastMsg: {
time: 1784203200,
lat: "55.7558",
lon: "37.6173",
speed: 24.5,
course: 91,
height: 164,
gps_valid: true,
sats: 11,
hdop: 0.8,
accuracy: 4.2,
params: "restricted-fixture-value",
},
}, {
id: "001002",
name: "Synthetic unit 1002",
}],
},
collectionContext: {
receivedAt: "2026-07-16T12:00:00.000Z",
},
expectedPublish: {
schemaVersion: DATA_PRODUCT_PUBLISH_SCHEMA_VERSION,
batch: {
runId: "gelios-fixture-run-20260716",
sequence: 0,
idempotencyKey: "gelios-fixture-run-20260716.batch-0",
},
facts: [{
sourceId: "gelios-unit-1001",
semanticType: "map.moving_object",
observedAt: "2026-07-16T12:00:00.000Z",
geometry: {
type: "Point",
coordinates: [37.6173, 55.7558],
},
attributes: {
course_degrees: 91,
display_name: "Synthetic unit 1001",
elevation_meters: 164,
hdop: 0.8,
horizontal_accuracy_meters: 4.2,
object_kind: "tracked_unit",
operational_status: "active",
position_source: "gelios",
position_valid: true,
quality_flags: [],
satellite_count: 11,
speed_kph: 24.5,
},
}, {
sourceId: "gelios-unit-001002",
semanticType: "map.moving_object",
observedAt: "2026-07-16T12:00:00.000Z",
attributes: {
display_name: "Synthetic unit 1002",
object_kind: "tracked_unit",
operational_status: "no_position",
position_source: "gelios",
position_valid: false,
quality_flags: ["missing_geometry"],
},
}],
},
});

11
packages/external-provider-contract/providers/gelios/v1/index.mjs
View File

@ -1,11 +0,0 @@
export {
GELIOS_POSITIONS_DATA_PRODUCT_ID,
GELIOS_POSITIONS_DATA_PRODUCT_VERSION,
GELIOS_POSITIONS_ONTOLOGY_REVISION,
GELIOS_PROVIDER_PACKAGE_ID,
GELIOS_PROVIDER_PACKAGE_VERSION,
GELIOS_UNIT_SOURCE_ID_PREFIX,
geliosProviderPackageV1,
} from "./package.mjs";
export { geliosUnitsCurrentFixtureV1 } from "./fixtures/units-current.fixture.mjs";

374
packages/external-provider-contract/providers/gelios/v1/package.mjs
View File

@ -1,374 +0,0 @@
import {
L2_TEMPLATE_SCHEMA_VERSION,
PROVIDER_PACKAGE_SCHEMA_VERSION,
SEMANTIC_MAPPING_SCHEMA_VERSION,
} from "../../../src/provider-package.mjs";
export const GELIOS_PROVIDER_PACKAGE_ID = "gelios.provider.v1";
export const GELIOS_PROVIDER_PACKAGE_VERSION = "1.0.0";
export const GELIOS_POSITIONS_DATA_PRODUCT_ID = "fleet.positions.current.v1";
export const GELIOS_POSITIONS_DATA_PRODUCT_VERSION = "1.0.0";
export const GELIOS_POSITIONS_ONTOLOGY_REVISION = "ontology.map.moving_object.v1";
export const GELIOS_UNIT_SOURCE_ID_PREFIX = "gelios-unit-";
const targetFields = Object.freeze([
"course_degrees",
"display_name",
"elevation_meters",
"geometry",
"hdop",
"horizontal_accuracy_meters",
"object_kind",
"operational_status",
"position_source",
"position_valid",
"quality_flags",
"satellite_count",
"speed_kph",
]);
export const geliosProviderPackageV1 = deepFreeze({
schemaVersion: PROVIDER_PACKAGE_SCHEMA_VERSION,
id: GELIOS_PROVIDER_PACKAGE_ID,
providerId: "gelios",
version: GELIOS_PROVIDER_PACKAGE_VERSION,
manifest: {
id: "gelios.provider.manifest.v1",
providerId: "gelios",
version: GELIOS_PROVIDER_PACKAGE_VERSION,
ontology: {
packageId: "gelios",
revision: "ontology.gelios.v1",
},
authModeIds: ["gelios.rest-bearer.v1"],
capabilityIds: ["gelios.units.current.read"],
fieldPolicyIds: ["gelios.positions.current.fields.v1"],
collectionProfileIds: [
"gelios.positions.current.realtime.v1",
"gelios.positions.current.manual.v1",
],
dataProductIds: [GELIOS_POSITIONS_DATA_PRODUCT_ID],
mappingContractIds: ["gelios.units.to.fleet.positions.current.v1"],
l2TemplateIds: ["gelios.positions.current.l2.v1"],
},
authModes: [{
id: "gelios.rest-bearer.v1",
kind: "api_token",
credentialOwner: "ndc_l2_credentials",
bindingCardinality: "one_per_connection",
transport: {
placement: "header",
name: "Authorization",
},
tokenLifecycle: {
artifacts: ["access", "refresh"],
requestArtifact: "access",
refreshMode: "operator_managed",
},
secretHandling: {
store: "ndc_l2_credentials",
exposeToGraph: false,
persistInPackage: false,
},
}],
capabilities: [{
id: "gelios.units.current.read",
classification: "read",
status: "implemented",
authModeId: "gelios.rest-bearer.v1",
request: {
method: "GET",
baseUrl: "https://api.geliospro.com",
path: "/api/v1/units",
query: {},
response: {
collectionPaths: ["units", "data", "data.units", "result", "$"],
pagination: "single_bounded_response",
},
},
entityScope: {
mode: "all_visible_to_credential",
refresh: "each_collection_run",
businessEntityFilter: "forbidden",
},
}],
fieldPolicies: [{
id: "gelios.positions.current.fields.v1",
version: "1.0.0",
dataProductId: GELIOS_POSITIONS_DATA_PRODUCT_ID,
targetFields: [...targetFields],
unknownSourceFields: "drop",
dynamicSourceFields: "drop_until_classified",
restrictedSourcePaths: [
"decrypt_key",
"hw_id",
"imei",
"info",
"lmsg.address",
"lmsg.params",
"lmsg.sensors",
"lastMsg.address",
"lastMsg.params",
"lastMsg.sensors",
"lastMessage.address",
"lastMessage.params",
"lastMessage.sensors",
"phone",
"phone2",
"sensors",
],
}],
collectionProfiles: [{
id: "gelios.positions.current.realtime.v1",
version: "1.0.0",
mode: "realtime",
schedule: { intervalMs: 10000 },
capabilityIds: ["gelios.units.current.read"],
dataProductId: GELIOS_POSITIONS_DATA_PRODUCT_ID,
mappingContractId: "gelios.units.to.fleet.positions.current.v1",
fieldPolicyId: "gelios.positions.current.fields.v1",
l2TemplateId: "gelios.positions.current.l2.v1",
entityScope: {
mode: "all_visible_to_credential",
refresh: "each_collection_run",
businessEntityFilter: "forbidden",
},
batching: { maxFacts: 5000 },
cardinality: {
maxCurrentEntities: 5000,
onExceed: "require_partitioned_data_product",
},
retry: {
maxAttempts: 4,
backoff: "exponential_with_jitter",
},
}, {
id: "gelios.positions.current.manual.v1",
version: "1.0.0",
mode: "manual",
capabilityIds: ["gelios.units.current.read"],
dataProductId: GELIOS_POSITIONS_DATA_PRODUCT_ID,
mappingContractId: "gelios.units.to.fleet.positions.current.v1",
fieldPolicyId: "gelios.positions.current.fields.v1",
l2TemplateId: "gelios.positions.current.l2.v1",
entityScope: {
mode: "all_visible_to_credential",
refresh: "each_collection_run",
businessEntityFilter: "forbidden",
},
batching: { maxFacts: 5000 },
cardinality: {
maxCurrentEntities: 5000,
onExceed: "require_partitioned_data_product",
},
retry: {
maxAttempts: 4,
backoff: "exponential_with_jitter",
},
}],
dataProducts: [{
id: GELIOS_POSITIONS_DATA_PRODUCT_ID,
version: GELIOS_POSITIONS_DATA_PRODUCT_VERSION,
ontologyRevision: GELIOS_POSITIONS_ONTOLOGY_REVISION,
deliveryMode: "snapshot+patch",
semanticTypes: ["map.moving_object"],
fields: [...targetFields],
history: {
mode: "sampled",
intervalMs: 60000,
strategy: "latest-per-entity-per-bucket",
retentionDays: 90,
},
}],
mappingContracts: [{
schemaVersion: SEMANTIC_MAPPING_SCHEMA_VERSION,
id: "gelios.units.to.fleet.positions.current.v1",
version: "1.0.0",
sourceCapabilityId: "gelios.units.current.read",
fieldPolicyId: "gelios.positions.current.fields.v1",
target: {
dataProductId: GELIOS_POSITIONS_DATA_PRODUCT_ID,
version: GELIOS_POSITIONS_DATA_PRODUCT_VERSION,
ontologyRevision: GELIOS_POSITIONS_ONTOLOGY_REVISION,
semanticType: "map.moving_object",
},
derivations: {
position_valid: {
kind: "boolean_rule",
rules: [
"geometry.coordinates_finite_wgs84",
"provider.gps_valid_not_false",
],
default: false,
},
operational_status: {
kind: "ordered_rules",
rules: [
"position_invalid.no_position",
"observed_age_gt_limit.stale",
"otherwise.active",
],
default: "active",
parameters: { staleAfterMs: 60000 },
},
quality_flags: {
kind: "flag_set",
rules: [
"geometry_missing.missing_geometry",
"provider_gps_false.gps_invalid",
"satellite_count_zero.no_satellites",
"observed_age_gt_limit.stale",
],
default: [],
parameters: { staleAfterMs: 60000 },
},
},
fact: {
sourceId: {
strategy: "first_non_empty",
paths: ["id", "unit_id", "unitId"],
coerce: "string",
prefix: GELIOS_UNIT_SOURCE_ID_PREFIX,
},
semanticType: { constant: "map.moving_object" },
observedAt: {
strategy: "first_non_empty",
paths: [
"lmsg.time",
"lmsg.ts",
"lmsg.timestamp",
"last_msg.time",
"last_msg.timestamp",
"lastMsg.time",
"lastMessage.time",
"last_msg_time",
"lmsg_time",
],
coerce: "unix_or_iso_timestamp",
fallback: "collection_received_at",
},
geometry: {
type: "Point",
longitude: {
strategy: "first_non_empty",
paths: ["lmsg.lon", "lmsg.lng", "lmsg.longitude", "last_msg.lon", "lastMsg.lon", "lastMsg.lng", "lastMsg.longitude", "lastMessage.lon", "lastMessage.lng", "lastMessage.longitude", "lon", "lng", "longitude"],
coerce: "number",
},
latitude: {
strategy: "first_non_empty",
paths: ["lmsg.lat", "lmsg.latitude", "last_msg.lat", "lastMsg.lat", "lastMsg.latitude", "lastMessage.lat", "lastMessage.latitude", "lat", "latitude"],
coerce: "number",
},
omitIfInvalid: true,
},
attributes: {
course_degrees: {
strategy: "first_non_empty",
paths: ["lmsg.course", "lmsg.angle", "last_msg.course", "lastMsg.course", "lastMsg.angle", "lastMessage.course", "lastMessage.angle", "course"],
coerce: "number",
omitIfMissing: true,
},
display_name: {
strategy: "first_non_empty",
paths: ["name", "unit_name", "title", "label"],
coerce: "string",
},
elevation_meters: {
strategy: "first_non_empty",
paths: ["lmsg.height", "lmsg.altitude", "last_msg.height", "lastMsg.height", "lastMsg.altitude", "lastMessage.height", "lastMessage.altitude", "height", "altitude"],
coerce: "number",
omitIfMissing: true,
},
hdop: {
strategy: "first_non_empty",
paths: ["lmsg.hdop", "last_msg.hdop", "lastMsg.hdop", "lastMessage.hdop", "gps_hdop", "hdop"],
coerce: "number",
omitIfMissing: true,
},
horizontal_accuracy_meters: {
strategy: "first_non_empty",
paths: ["lmsg.accuracy", "last_msg.accuracy", "lastMsg.accuracy", "lastMessage.accuracy", "gps_accuracy_m", "accuracy_m", "accuracy"],
coerce: "number",
omitIfMissing: true,
},
object_kind: { constant: "tracked_unit" },
operational_status: { derive: "operational_status" },
position_source: { constant: "gelios" },
position_valid: { derive: "position_valid" },
quality_flags: { derive: "quality_flags" },
satellite_count: {
strategy: "first_non_empty",
paths: ["lmsg.sats", "lmsg.satellites", "last_msg.sats", "lastMsg.sats", "lastMsg.satellites", "lastMessage.sats", "lastMessage.satellites", "gps_sats", "sats", "satellites"],
coerce: "number",
omitIfMissing: true,
},
speed_kph: {
strategy: "first_non_empty",
paths: ["lmsg.speed", "last_msg.speed", "lastMsg.speed", "lastMessage.speed", "speed"],
coerce: "number",
omitIfMissing: true,
},
},
},
}],
l2Templates: [{
schemaVersion: L2_TEMPLATE_SCHEMA_VERSION,
id: "gelios.positions.current.l2.v1",
version: "1.0.0",
runtime: "ndc_l2",
instanceMode: "one_connection_per_workflow",
connectionParameters: [
"tenant_id",
"connection_id",
"collection_profile_id",
"provider_credential_ref",
],
credentialBindings: [{
role: "provider",
owner: "ndc_l2_credentials",
management: "connection_input",
authModeId: "gelios.rest-bearer.v1",
}, {
role: "publisher",
owner: "ndc_l2_credentials",
management: "control_plane_managed",
nodeType: "n8n-nodes-ndc.ndcDataProductPublish",
}],
steps: [{
id: "collection.trigger",
kind: "collection_trigger",
collectionProfileDriven: true,
}, {
id: "provider.fetch",
kind: "provider_request",
capabilityId: "gelios.units.current.read",
}, {
id: "provider.extract-items",
kind: "extract_items",
capabilityId: "gelios.units.current.read",
}, {
id: "ontology.map",
kind: "semantic_mapping",
mappingContractId: "gelios.units.to.fleet.positions.current.v1",
}, {
id: "data-product.publish",
kind: "data_product_publish",
dataProductId: GELIOS_POSITIONS_DATA_PRODUCT_ID,
nodeType: "n8n-nodes-ndc.ndcDataProductPublish",
}],
invariants: {
oneConnectionPerInstance: true,
providerSecretByReferenceOnly: true,
publisherBindingControlPlaneManaged: true,
callerScopeForbidden: true,
providerSpecificServiceForbidden: true,
},
}],
});
function deepFreeze(value) {
if (!value || typeof value !== "object" || Object.isFrozen(value)) return value;
Object.freeze(value);
for (const child of Object.values(value)) deepFreeze(child);
return value;
}

1
packages/external-provider-contract/src/contract-version.mjs
View File

@ -1 +0,0 @@
export const EXTERNAL_PROVIDER_CONTRACT_VERSION = "nodedc.external-provider-contract/v1";

10
packages/external-provider-contract/src/data-plane.mjs
View File

@ -1,10 +0,0 @@
export { EXTERNAL_PROVIDER_CONTRACT_VERSION } from "./contract-version.mjs";
export { validateIntakeBatch } from "./intake-batch.mjs";
export {
DATA_PRODUCT_PATCH_SCHEMA_VERSION,
DATA_PRODUCT_PUBLISH_SCHEMA_VERSION,
DATA_PRODUCT_SNAPSHOT_SCHEMA_VERSION,
validateDataProductPatch,
validateDataProductPublish,
validateDataProductSnapshot,
} from "./data-product.mjs";

4
packages/external-provider-contract/src/data-product.mjs
View File

@ -2,11 +2,11 @@ export const DATA_PRODUCT_PUBLISH_SCHEMA_VERSION = "nodedc.data-product.publish/
export const DATA_PRODUCT_SNAPSHOT_SCHEMA_VERSION = "nodedc.data-product.snapshot/v1"; export const DATA_PRODUCT_SNAPSHOT_SCHEMA_VERSION = "nodedc.data-product.snapshot/v1";
export const DATA_PRODUCT_PATCH_SCHEMA_VERSION = "nodedc.data-product.patch/v1"; export const DATA_PRODUCT_PATCH_SCHEMA_VERSION = "nodedc.data-product.patch/v1";
import { SECRET_LIKE_KEY, SECRET_LIKE_VALUE } from "./sensitive-field-policy.mjs";
const IDENTIFIER = /^[a-z][a-z0-9._:-]{2,127}$/; const IDENTIFIER = /^[a-z][a-z0-9._:-]{2,127}$/;
const SEMVER = /^\d+\.\d+\.\d+(?:[-+][a-z0-9.-]+)?$/i; const SEMVER = /^\d+\.\d+\.\d+(?:[-+][a-z0-9.-]+)?$/i;
const CURSOR = /^(?:0|[1-9]\d*)$/; const CURSOR = /^(?:0|[1-9]\d*)$/;
const SECRET_LIKE_KEY = /(token|secret|password|authorization|access[_-]?token|refresh[_-]?token|api[_-]?key)/i;
const SECRET_LIKE_VALUE = /(?:ndc_edp(?:wb|rb)_[A-Za-z0-9_-]*|[?&](?:token|secret|password|authorization|access[_-]?token|refresh[_-]?token|api[_-]?key)=|(?:bearer|basic)\s+\S+|eyJ[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+)/i;
const MAX_BATCH_SEQUENCE = 2_147_483_647; const MAX_BATCH_SEQUENCE = 2_147_483_647;
const MAX_FACT_ATTRIBUTES_BYTES = 64 * 1024; const MAX_FACT_ATTRIBUTES_BYTES = 64 * 1024;

662
packages/external-provider-contract/src/engine-credential-sink.mjs Normal file
View File

@ -0,0 +1,662 @@
import { createHash, verify as verifySignature } from "node:crypto";
export const ENGINE_CREDENTIAL_SINK_PROVISION_SCHEMA_VERSION = "nodedc.engine.credential-sink.provision/v1";
export const ENGINE_CREDENTIAL_SINK_RECEIPT_SCHEMA_VERSION = "nodedc.engine.credential-sink.receipt/v1";
export const ENGINE_CREDENTIAL_SINK_ROLLBACK_SCHEMA_VERSION = "nodedc.engine.credential-sink.rollback/v1";
export const ENGINE_CREDENTIAL_SINK_ROLLBACK_RECEIPT_SCHEMA_VERSION = "nodedc.engine.credential-sink.rollback-receipt/v1";
export const ENGINE_CREDENTIAL_SINK_AUDIT_SCHEMA_VERSION = "nodedc.engine.credential-sink.audit/v1";
const HASH = /^sha256:[a-f0-9]{64}$/;
const IDENTIFIER = /^[a-z][a-z0-9._:-]{2,127}$/;
const OPAQUE_ID = /^[A-Za-z0-9][A-Za-z0-9._:-]{2,159}$/;
const NODE_TYPE = /^[a-z][A-Za-z0-9.-]{2,159}$/;
const CREDENTIAL_TYPE = /^[a-z][A-Za-z0-9]{2,127}$/;
const CREDENTIAL_REFERENCE = /^[A-Za-z0-9][A-Za-z0-9_-]{5,159}$/;
const REASON_CODE = /^[a-z][a-z0-9_.:-]{2,127}$/;
const ED25519_SIGNATURE = /^[A-Za-z0-9_-]{86}$/;
const SECRET_LIKE_KEY = /(token|secret|password|authorization|access[_-]?token|refresh[_-]?token|api[_-]?key|material|value)/i;
const SECRET_LIKE_VALUE = /(?:ndc_(?:edp(?:wb|rb)|fndbg)_[A-Za-z0-9_-]+|(?:bearer|basic)\s+\S+|eyJ[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+)/i;
const MAX_BINDINGS = 32;
const MAX_REQUEST_LIFETIME_MS = 15 * 60 * 1000;
const MAX_REQUEST_CLOCK_SKEW_MS = 60 * 1000;
const CAPABILITY_SPECS = Object.freeze({
"external-data-plane.writer": Object.freeze({
nodeType: "n8n-nodes-ndc.ndcDataProductPublish",
credentialType: "ndcDataProductWriterApi",
materialPattern: /^ndc_edpwb_[A-Za-z0-9_-]{43}$/,
}),
"external-data-plane.reader": Object.freeze({
nodeType: "n8n-nodes-ndc.ndcDataProductRead",
credentialType: "ndcDataProductReaderApi",
materialPattern: /^ndc_edprb_[A-Za-z0-9_-]{43}$/,
}),
"foundry.binding": Object.freeze({
nodeType: "n8n-nodes-ndc.ndcFoundryBinding",
credentialType: "ndcFoundryBindingApi",
materialPattern: /^ndc_fndbg_[A-Za-z0-9_-]{43}$/,
}),
});
export const ENGINE_CREDENTIAL_SINK_CAPABILITY_TYPES = Object.freeze(Object.keys(CAPABILITY_SPECS));
const PROVISION_KEYS = new Set(["schemaVersion", "transaction", "bindings"]);
const TRANSACTION_KEYS = new Set([
"id",
"idempotencyKey",
"requestedAt",
"requestExpiresAt",
"policyHash",
"failureMode",
"issuer",
"attestation",
]);
const ISSUER_KEYS = new Set(["serviceId", "keyId"]);
const ATTESTATION_KEYS = new Set(["algorithm", "signature"]);
const BINDING_KEYS = new Set([
"bindingId",
"capabilityType",
"grantId",
"target",
"expiresAt",
"policyHash",
"capabilityDigest",
"material",
]);
const TARGET_KEYS = new Set([
"workflowId",
"workflowRevision",
"nodeId",
"nodeType",
"credentialType",
]);
const MATERIAL_KEYS = new Set(["format", "value"]);
const RECEIPT_KEYS = new Set([
"schemaVersion",
"transactionId",
"idempotencyKey",
"outcome",
"policyHash",
"processedAt",
"credentials",
"rollback",
"errorCode",
]);
const RECEIPT_CREDENTIAL_KEYS = new Set([
"bindingId",
"capabilityType",
"grantId",
"target",
"credentialRef",
"expiresAt",
"policyHash",
"capabilityDigest",
"disposition",
]);
const RECEIPT_ROLLBACK_KEYS = new Set(["status", "completedAt"]);
const ROLLBACK_REQUEST_KEYS = new Set(["schemaVersion", "rollback"]);
const ROLLBACK_KEYS = new Set([
"id",
"idempotencyKey",
"transactionId",
"requestedAt",
"requestExpiresAt",
"policyHash",
"committedReceiptHash",
"reasonCode",
]);
const ROLLBACK_RECEIPT_KEYS = new Set([
"schemaVersion",
"rollbackId",
"transactionId",
"outcome",
"policyHash",
"committedReceiptHash",
"processedAt",
"errorCode",
]);
const AUDIT_KEYS = new Set([
"schemaVersion",
"eventId",
"transactionId",
"operationId",
"operation",
"outcome",
"occurredAt",
"policyHash",
"principal",
"targets",
"reasonCode",
]);
const PRINCIPAL_KEYS = new Set(["serviceId", "fingerprint"]);
const AUDIT_TARGET_KEYS = new Set([
"bindingId",
"capabilityType",
"grantId",
"target",
"expiresAt",
"policyHash",
"capabilityDigest",
"credentialRefHash",
]);
/**
* Validates the only request allowed to carry plaintext workload capability
* material across the trusted Platform -> Engine server boundary. Callers and
* receivers must never log, trace, persist or return this request body.
*/
export function validateEngineCredentialSinkProvision(value, { now = Date.now(), issuerPublicKeys } = {}) {
const errors = [];
const nowMs = normalizeNow(now, errors);
if (!isPlainObject(value)) return result(["credentialSinkProvision_must_be_object"]);
if (value.schemaVersion !== ENGINE_CREDENTIAL_SINK_PROVISION_SCHEMA_VERSION) errors.push("schemaVersion_mismatch");
rejectUnknownKeys(value, PROVISION_KEYS, "credentialSinkProvision", errors);
if (!isPlainObject(value.transaction)) {
errors.push("transaction_must_be_object");
} else {
rejectUnknownKeys(value.transaction, TRANSACTION_KEYS, "transaction", errors);
requiredOpaqueId(value.transaction.id, "transaction.id", errors);
requiredOpaqueId(value.transaction.idempotencyKey, "transaction.idempotencyKey", errors);
requiredTimestamp(value.transaction.requestedAt, "transaction.requestedAt", errors);
requiredTimestamp(value.transaction.requestExpiresAt, "transaction.requestExpiresAt", errors);
requiredHash(value.transaction.policyHash, "transaction.policyHash", errors);
if (value.transaction.failureMode !== "rollback-all") errors.push("transaction.failureMode_must_be_rollback-all");
validateIssuer(value.transaction.issuer, "transaction.issuer", errors);
validateAttestation(value.transaction.attestation, "transaction.attestation", errors);
validateRequestWindow(value.transaction.requestedAt, value.transaction.requestExpiresAt, nowMs, errors);
}
if (!Array.isArray(value.bindings) || value.bindings.length === 0) {
errors.push("bindings_must_be_nonempty_array");
} else if (value.bindings.length > MAX_BINDINGS) {
errors.push("bindings_limit_exceeded");
} else {
const bindingIds = new Set();
const targets = new Set();
value.bindings.forEach((binding, index) => {
validateProvisionBinding(binding, index, value.transaction?.requestedAt, errors);
if (!isPlainObject(binding)) return;
if (bindingIds.has(binding.bindingId)) errors.push("bindings_bindingId_must_be_unique");
bindingIds.add(binding.bindingId);
const targetKey = targetIdentity(binding.target);
if (targetKey && targets.has(targetKey)) errors.push("bindings_target_credential_must_be_unique");
if (targetKey) targets.add(targetKey);
});
}
if (isPlainObject(value.transaction) && HASH.test(String(value.transaction.policyHash || ""))) {
const expectedHash = computeEngineCredentialSinkPolicyHash(value);
if (value.transaction.policyHash !== expectedHash) errors.push("transaction.policyHash_mismatch");
verifyProvisionAttestation(value.transaction, issuerPublicKeys, errors);
}
return result(errors);
}
/**
* Computes the aggregate policy digest over the complete secret-free request
* descriptor. Plaintext `material` is excluded by construction, while its
* high-entropy capability digest is included; changing a target, grant,
* capability, expiry, individual policy hash or transaction envelope changes
* the aggregate digest and invalidates the issuer attestation.
*/
export function computeEngineCredentialSinkPolicyHash(value) {
const descriptor = {
schemaVersion: value?.schemaVersion,
transaction: isPlainObject(value?.transaction) ? {
id: value.transaction.id,
idempotencyKey: value.transaction.idempotencyKey,
requestedAt: value.transaction.requestedAt,
requestExpiresAt: value.transaction.requestExpiresAt,
failureMode: value.transaction.failureMode,
issuer: value.transaction.issuer,
} : value?.transaction,
bindings: Array.isArray(value?.bindings)
? value.bindings.map(secretFreeBindingDescriptor)
: value?.bindings,
};
return `sha256:${createHash("sha256").update(stableJson(descriptor), "utf8").digest("hex")}`;
}
/**
* Receipt is intentionally incapable of carrying credential material. When a
* request is supplied, the validator also proves the sink committed the exact
* requested workflow/node/type set without target substitution.
*/
export function validateEngineCredentialSinkReceipt(value, { request, issuerPublicKeys } = {}) {
const errors = [];
if (!isPlainObject(value)) return result(["credentialSinkReceipt_must_be_object"]);
if (value.schemaVersion !== ENGINE_CREDENTIAL_SINK_RECEIPT_SCHEMA_VERSION) errors.push("schemaVersion_mismatch");
rejectUnknownKeys(value, RECEIPT_KEYS, "credentialSinkReceipt", errors);
requiredOpaqueId(value.transactionId, "transactionId", errors);
requiredOpaqueId(value.idempotencyKey, "idempotencyKey", errors);
requiredHash(value.policyHash, "policyHash", errors);
requiredTimestamp(value.processedAt, "processedAt", errors);
const outcomes = new Set(["committed", "rolled-back", "rejected", "rollback-failed"]);
if (!outcomes.has(value.outcome)) errors.push("outcome_invalid");
if (!Array.isArray(value.credentials)) {
errors.push("credentials_must_be_array");
} else {
const bindingIds = new Set();
const refs = new Set();
value.credentials.forEach((credential, index) => {
validateReceiptCredential(credential, index, errors);
if (!isPlainObject(credential)) return;
if (bindingIds.has(credential.bindingId)) errors.push("credentials_bindingId_must_be_unique");
bindingIds.add(credential.bindingId);
if (refs.has(credential.credentialRef)) errors.push("credentials_credentialRef_must_be_unique");
refs.add(credential.credentialRef);
});
}
validateReceiptOutcome(value, errors);
if (containsSecretLikeMaterial(value)) errors.push("receipt_must_not_contain_secret_material");
if (request !== undefined) compareReceiptToProvisionRequest(value, request, issuerPublicKeys, errors);
return result(errors);
}
export function computeEngineCredentialSinkReceiptHash(value) {
return `sha256:${createHash("sha256").update(stableJson(value), "utf8").digest("hex")}`;
}
export function validateEngineCredentialSinkRollback(value, { now = Date.now() } = {}) {
const errors = [];
const nowMs = normalizeNow(now, errors);
if (!isPlainObject(value)) return result(["credentialSinkRollback_must_be_object"]);
if (value.schemaVersion !== ENGINE_CREDENTIAL_SINK_ROLLBACK_SCHEMA_VERSION) errors.push("schemaVersion_mismatch");
rejectUnknownKeys(value, ROLLBACK_REQUEST_KEYS, "credentialSinkRollback", errors);
if (!isPlainObject(value.rollback)) {
errors.push("rollback_must_be_object");
return result(errors);
}
rejectUnknownKeys(value.rollback, ROLLBACK_KEYS, "rollback", errors);
requiredOpaqueId(value.rollback.id, "rollback.id", errors);
requiredOpaqueId(value.rollback.idempotencyKey, "rollback.idempotencyKey", errors);
requiredOpaqueId(value.rollback.transactionId, "rollback.transactionId", errors);
requiredTimestamp(value.rollback.requestedAt, "rollback.requestedAt", errors);
requiredTimestamp(value.rollback.requestExpiresAt, "rollback.requestExpiresAt", errors);
requiredHash(value.rollback.policyHash, "rollback.policyHash", errors);
requiredHash(value.rollback.committedReceiptHash, "rollback.committedReceiptHash", errors);
requiredReason(value.rollback.reasonCode, "rollback.reasonCode", errors);
validateRequestWindow(value.rollback.requestedAt, value.rollback.requestExpiresAt, nowMs, errors);
if (containsSecretLikeMaterial(value)) errors.push("rollback_must_not_contain_secret_material");
return result(errors);
}
export function validateEngineCredentialSinkRollbackReceipt(value, { request } = {}) {
const errors = [];
if (!isPlainObject(value)) return result(["credentialSinkRollbackReceipt_must_be_object"]);
if (value.schemaVersion !== ENGINE_CREDENTIAL_SINK_ROLLBACK_RECEIPT_SCHEMA_VERSION) errors.push("schemaVersion_mismatch");
rejectUnknownKeys(value, ROLLBACK_RECEIPT_KEYS, "credentialSinkRollbackReceipt", errors);
requiredOpaqueId(value.rollbackId, "rollbackId", errors);
requiredOpaqueId(value.transactionId, "transactionId", errors);
requiredHash(value.policyHash, "policyHash", errors);
requiredHash(value.committedReceiptHash, "committedReceiptHash", errors);
requiredTimestamp(value.processedAt, "processedAt", errors);
if (!new Set(["rolled-back", "rejected", "rollback-failed"]).has(value.outcome)) errors.push("outcome_invalid");
if (value.outcome === "rolled-back") {
if (value.errorCode !== undefined) errors.push("errorCode_forbidden_for_success");
} else {
requiredReason(value.errorCode, "errorCode", errors);
}
if (containsSecretLikeMaterial(value)) errors.push("rollbackReceipt_must_not_contain_secret_material");
if (request !== undefined && isPlainObject(request?.rollback)) {
if (!validateEngineCredentialSinkRollback(request, { now: value.processedAt }).ok) {
errors.push("request_invalid_for_rollback_receipt_comparison");
}
if (value.rollbackId !== request.rollback.id) errors.push("rollbackId_request_mismatch");
if (value.transactionId !== request.rollback.transactionId) errors.push("transactionId_request_mismatch");
if (value.policyHash !== request.rollback.policyHash) errors.push("policyHash_request_mismatch");
if (value.committedReceiptHash !== request.rollback.committedReceiptHash) {
errors.push("committedReceiptHash_request_mismatch");
}
}
return result(errors);
}
export function validateEngineCredentialSinkAudit(value) {
const errors = [];
if (!isPlainObject(value)) return result(["credentialSinkAudit_must_be_object"]);
if (value.schemaVersion !== ENGINE_CREDENTIAL_SINK_AUDIT_SCHEMA_VERSION) errors.push("schemaVersion_mismatch");
rejectUnknownKeys(value, AUDIT_KEYS, "credentialSinkAudit", errors);
requiredOpaqueId(value.eventId, "eventId", errors);
requiredOpaqueId(value.transactionId, "transactionId", errors);
requiredOpaqueId(value.operationId, "operationId", errors);
if (!new Set(["provision", "rollback"]).has(value.operation)) errors.push("operation_invalid");
if (!new Set(["committed", "rolled-back", "rejected", "rollback-failed"]).has(value.outcome)) errors.push("outcome_invalid");
if (value.operation === "provision" && value.outcome === "rolled-back" && !value.reasonCode) {
errors.push("reasonCode_required");
}
if (value.operation === "rollback" && value.outcome === "committed") errors.push("rollback_outcome_invalid");
requiredTimestamp(value.occurredAt, "occurredAt", errors);
requiredHash(value.policyHash, "policyHash", errors);
if (!isPlainObject(value.principal)) {
errors.push("principal_must_be_object");
} else {
rejectUnknownKeys(value.principal, PRINCIPAL_KEYS, "principal", errors);
requiredIdentifier(value.principal.serviceId, "principal.serviceId", errors);
requiredHash(value.principal.fingerprint, "principal.fingerprint", errors);
}
if (!Array.isArray(value.targets)) {
errors.push("targets_must_be_array");
} else {
value.targets.forEach((target, index) => validateAuditTarget(target, index, errors));
}
if (value.reasonCode !== undefined) requiredReason(value.reasonCode, "reasonCode", errors);
if (new Set(["rejected", "rollback-failed"]).has(value.outcome) && value.reasonCode === undefined) {
errors.push("reasonCode_required");
}
if (containsSecretLikeMaterial(value)) errors.push("audit_must_not_contain_secret_material");
return result(errors);
}
/** Returns audit-safe target descriptors; material and credential refs cannot escape. */
export function engineCredentialSinkAuditTargets(request, receipt) {
const refByBinding = new Map(
Array.isArray(receipt?.credentials)
? receipt.credentials.map((item) => [item.bindingId, item.credentialRef])
: [],
);
return Array.isArray(request?.bindings) ? request.bindings.map((binding) => {
const descriptor = secretFreeBindingDescriptor(binding);
const credentialRef = refByBinding.get(binding.bindingId);
return credentialRef
? { ...descriptor, credentialRefHash: sha256Value(credentialRef) }
: descriptor;
}) : [];
}
function validateProvisionBinding(value, index, requestedAt, errors) {
const path = `bindings[${index}]`;
if (!isPlainObject(value)) {
errors.push(`${path}_must_be_object`);
return;
}
rejectUnknownKeys(value, BINDING_KEYS, path, errors);
requiredIdentifier(value.bindingId, `${path}.bindingId`, errors);
const spec = CAPABILITY_SPECS[value.capabilityType];
if (!spec) errors.push(`${path}.capabilityType_invalid`);
requiredOpaqueId(value.grantId, `${path}.grantId`, errors);
validateTarget(value.target, path, errors);
requiredTimestamp(value.expiresAt, `${path}.expiresAt`, errors);
requiredHash(value.policyHash, `${path}.policyHash`, errors);
requiredHash(value.capabilityDigest, `${path}.capabilityDigest`, errors);
if (isTimestamp(requestedAt) && isTimestamp(value.expiresAt) && Date.parse(value.expiresAt) <= Date.parse(requestedAt)) {
errors.push(`${path}.expiresAt_must_be_after_requestedAt`);
}
if (spec && isPlainObject(value.target)) {
if (value.target.nodeType !== spec.nodeType) errors.push(`${path}.target.nodeType_capability_mismatch`);
if (value.target.credentialType !== spec.credentialType) errors.push(`${path}.target.credentialType_capability_mismatch`);
}
if (!isPlainObject(value.material)) {
errors.push(`${path}.material_must_be_object`);
} else {
rejectUnknownKeys(value.material, MATERIAL_KEYS, `${path}.material`, errors);
if (value.material.format !== "opaque-bearer") errors.push(`${path}.material.format_must_be_opaque-bearer`);
if (!spec || typeof value.material.value !== "string" || !spec.materialPattern.test(value.material.value)) {
errors.push(`${path}.material.value_invalid_for_capability`);
} else if (value.capabilityDigest !== computeEngineCredentialCapabilityDigest(value.material.value)) {
errors.push(`${path}.capabilityDigest_material_mismatch`);
}
}
}
function validateTarget(value, path, errors) {
if (!isPlainObject(value)) {
errors.push(`${path}.target_must_be_object`);
return;
}
rejectUnknownKeys(value, TARGET_KEYS, `${path}.target`, errors);
requiredOpaqueId(value.workflowId, `${path}.target.workflowId`, errors);
requiredOpaqueId(value.workflowRevision, `${path}.target.workflowRevision`, errors);
requiredOpaqueId(value.nodeId, `${path}.target.nodeId`, errors);
if (typeof value.nodeType !== "string" || !NODE_TYPE.test(value.nodeType)) errors.push(`${path}.target.nodeType_invalid`);
if (typeof value.credentialType !== "string" || !CREDENTIAL_TYPE.test(value.credentialType)) {
errors.push(`${path}.target.credentialType_invalid`);
}
}
function validateReceiptCredential(value, index, errors) {
const path = `credentials[${index}]`;
if (!isPlainObject(value)) {
errors.push(`${path}_must_be_object`);
return;
}
rejectUnknownKeys(value, RECEIPT_CREDENTIAL_KEYS, path, errors);
requiredIdentifier(value.bindingId, `${path}.bindingId`, errors);
if (!CAPABILITY_SPECS[value.capabilityType]) errors.push(`${path}.capabilityType_invalid`);
requiredOpaqueId(value.grantId, `${path}.grantId`, errors);
validateTarget(value.target, path, errors);
if (typeof value.credentialRef !== "string" || !CREDENTIAL_REFERENCE.test(value.credentialRef)) {
errors.push(`${path}.credentialRef_invalid`);
}
requiredTimestamp(value.expiresAt, `${path}.expiresAt`, errors);
requiredHash(value.policyHash, `${path}.policyHash`, errors);
requiredHash(value.capabilityDigest, `${path}.capabilityDigest`, errors);
if (!new Set(["created", "reused", "rotated"]).has(value.disposition)) errors.push(`${path}.disposition_invalid`);
}
function validateReceiptOutcome(value, errors) {
if (!isPlainObject(value.rollback)) {
errors.push("rollback_must_be_object");
return;
}
rejectUnknownKeys(value.rollback, RECEIPT_ROLLBACK_KEYS, "rollback", errors);
const expectedRollback = {
committed: "not-required",
"rolled-back": "complete",
rejected: "not-started",
"rollback-failed": "incomplete",
}[value.outcome];
if (expectedRollback && value.rollback.status !== expectedRollback) errors.push("rollback.status_outcome_mismatch");
if (new Set(["complete", "incomplete"]).has(value.rollback.status)) {
requiredTimestamp(value.rollback.completedAt, "rollback.completedAt", errors);
} else if (value.rollback.completedAt !== undefined) {
errors.push("rollback.completedAt_not_allowed");
}
if (value.outcome === "committed") {
if (!Array.isArray(value.credentials) || value.credentials.length === 0) errors.push("committed_credentials_required");
if (value.errorCode !== undefined) errors.push("errorCode_forbidden_for_success");
} else {
if (Array.isArray(value.credentials) && value.credentials.length !== 0) errors.push("noncommitted_credentials_must_be_empty");
requiredReason(value.errorCode, "errorCode", errors);
}
}
function compareReceiptToProvisionRequest(receipt, request, issuerPublicKeys, errors) {
const requestValidation = validateEngineCredentialSinkProvision(request, {
now: receipt.processedAt,
issuerPublicKeys,
});
if (!requestValidation.ok) {
errors.push("request_invalid_for_receipt_comparison");
return;
}
if (receipt.transactionId !== request.transaction.id) errors.push("transactionId_request_mismatch");
if (receipt.idempotencyKey !== request.transaction.idempotencyKey) errors.push("idempotencyKey_request_mismatch");
if (receipt.policyHash !== request.transaction.policyHash) errors.push("policyHash_request_mismatch");
if (receipt.outcome !== "committed") return;
if (receipt.credentials.length !== request.bindings.length) errors.push("credentials_request_count_mismatch");
const requested = new Map(request.bindings.map((binding) => [binding.bindingId, secretFreeBindingDescriptor(binding)]));
for (const credential of receipt.credentials) {
const expected = requested.get(credential.bindingId);
if (!expected || stableJson({
bindingId: credential.bindingId,
capabilityType: credential.capabilityType,
grantId: credential.grantId,
target: credential.target,
expiresAt: credential.expiresAt,
policyHash: credential.policyHash,
capabilityDigest: credential.capabilityDigest,
}) !== stableJson(expected)) {
errors.push("credentials_request_target_mismatch");
}
}
}
function validateAuditTarget(value, index, errors) {
const path = `targets[${index}]`;
if (!isPlainObject(value)) {
errors.push(`${path}_must_be_object`);
return;
}
rejectUnknownKeys(value, AUDIT_TARGET_KEYS, path, errors);
requiredIdentifier(value.bindingId, `${path}.bindingId`, errors);
if (!CAPABILITY_SPECS[value.capabilityType]) errors.push(`${path}.capabilityType_invalid`);
requiredOpaqueId(value.grantId, `${path}.grantId`, errors);
validateTarget(value.target, path, errors);
requiredTimestamp(value.expiresAt, `${path}.expiresAt`, errors);
requiredHash(value.policyHash, `${path}.policyHash`, errors);
requiredHash(value.capabilityDigest, `${path}.capabilityDigest`, errors);
if (value.credentialRefHash !== undefined) requiredHash(value.credentialRefHash, `${path}.credentialRefHash`, errors);
}
function secretFreeBindingDescriptor(value) {
return {
bindingId: value?.bindingId,
capabilityType: value?.capabilityType,
grantId: value?.grantId,
target: value?.target,
expiresAt: value?.expiresAt,
policyHash: value?.policyHash,
capabilityDigest: value?.capabilityDigest,
};
}
export function computeEngineCredentialCapabilityDigest(value) {
return sha256Value(value);
}
function validateIssuer(value, path, errors) {
if (!isPlainObject(value)) {
errors.push(`${path}_must_be_object`);
return;
}
rejectUnknownKeys(value, ISSUER_KEYS, path, errors);
requiredIdentifier(value.serviceId, `${path}.serviceId`, errors);
requiredOpaqueId(value.keyId, `${path}.keyId`, errors);
}
function validateAttestation(value, path, errors) {
if (!isPlainObject(value)) {
errors.push(`${path}_must_be_object`);
return;
}
rejectUnknownKeys(value, ATTESTATION_KEYS, path, errors);
if (value.algorithm !== "Ed25519") errors.push(`${path}.algorithm_must_be_Ed25519`);
if (typeof value.signature !== "string" || !ED25519_SIGNATURE.test(value.signature)) {
errors.push(`${path}.signature_invalid`);
}
}
function verifyProvisionAttestation(transaction, issuerPublicKeys, errors) {
if (!isPlainObject(transaction?.issuer) || !isPlainObject(transaction?.attestation)) return;
if (transaction.attestation.algorithm !== "Ed25519" || !ED25519_SIGNATURE.test(String(transaction.attestation.signature || ""))) return;
const keyIdentity = `${transaction.issuer.serviceId}:${transaction.issuer.keyId}`;
const publicKey = isPlainObject(issuerPublicKeys) && Object.hasOwn(issuerPublicKeys, keyIdentity)
? issuerPublicKeys[keyIdentity]
: undefined;
if (!publicKey) {
errors.push("transaction.issuer_public_key_required");
return;
}
let valid = false;
try {
valid = verifySignature(
null,
Buffer.from(String(transaction.policyHash), "utf8"),
publicKey,
Buffer.from(transaction.attestation.signature, "base64url"),
);
} catch {
valid = false;
}
if (!valid) errors.push("transaction.attestation_invalid");
}
function validateRequestWindow(requestedAt, requestExpiresAt, nowMs, errors) {
if (!isTimestamp(requestedAt) || !isTimestamp(requestExpiresAt)) return;
const requested = Date.parse(requestedAt);
const expires = Date.parse(requestExpiresAt);
if (expires <= requested) errors.push("requestExpiresAt_must_be_after_requestedAt");
if (expires - requested > MAX_REQUEST_LIFETIME_MS) errors.push("request_lifetime_exceeds_15_minutes");
if (Number.isFinite(nowMs)) {
if (requested > nowMs + MAX_REQUEST_CLOCK_SKEW_MS) errors.push("requestedAt_exceeds_clock_skew");
if (expires <= nowMs) errors.push("request_expired");
}
}
function normalizeNow(value, errors) {
const normalized = value instanceof Date ? value.getTime() : typeof value === "string" ? Date.parse(value) : Number(value);
if (!Number.isFinite(normalized)) {
errors.push("validation_now_invalid");
return Number.NaN;
}
return normalized;
}
function targetIdentity(value) {
if (!isPlainObject(value)) return "";
return [value.workflowId, value.workflowRevision, value.nodeId, value.nodeType, value.credentialType].join("\u0000");
}
function requiredIdentifier(value, path, errors) {
if (typeof value !== "string" || !IDENTIFIER.test(value)) errors.push(`${path}_invalid`);
}
function requiredOpaqueId(value, path, errors) {
if (typeof value !== "string" || !OPAQUE_ID.test(value)) errors.push(`${path}_invalid`);
}
function requiredHash(value, path, errors) {
if (typeof value !== "string" || !HASH.test(value)) errors.push(`${path}_invalid`);
}
function requiredReason(value, path, errors) {
if (typeof value !== "string" || !REASON_CODE.test(value)) errors.push(`${path}_invalid`);
}
function requiredTimestamp(value, path, errors) {
if (!isTimestamp(value)) errors.push(`${path}_invalid_timestamp`);
}
function isTimestamp(value) {
return typeof value === "string" && !Number.isNaN(Date.parse(value)) && new Date(value).toISOString() === value;
}
function sha256Value(value) {
return `sha256:${createHash("sha256").update(String(value), "utf8").digest("hex")}`;
}
function stableJson(value) {
return JSON.stringify(sortValue(value));
}
function sortValue(value) {
if (Array.isArray(value)) return value.map(sortValue);
if (!isPlainObject(value)) return value;
return Object.fromEntries(Object.keys(value).sort().map((key) => [key, sortValue(value[key])]));
}
function containsSecretLikeMaterial(value) {
if (typeof value === "string") return SECRET_LIKE_VALUE.test(value);
if (Array.isArray(value)) return value.some(containsSecretLikeMaterial);
if (!isPlainObject(value)) return false;
return Object.entries(value).some(([key, child]) => SECRET_LIKE_KEY.test(key) || containsSecretLikeMaterial(child));
}
function isPlainObject(value) {
return Boolean(value) && typeof value === "object" && !Array.isArray(value);
}
function rejectUnknownKeys(value, allowed, path, errors) {
if (!isPlainObject(value)) return;
for (const key of Object.keys(value)) {
if (!allowed.has(key)) errors.push(`${path}.${key}_not_allowed`);
}
}
function result(errors) {
const uniqueErrors = [...new Set(errors)];
return Object.freeze({ ok: uniqueErrors.length === 0, errors: Object.freeze(uniqueErrors) });
}

153
packages/external-provider-contract/src/index.mjs
View File

@ -1,23 +1,6 @@
import { EXTERNAL_PROVIDER_CONTRACT_VERSION } from "./contract-version.mjs"; export const EXTERNAL_PROVIDER_CONTRACT_VERSION = "nodedc.external-provider-contract/v1";
export { EXTERNAL_PROVIDER_CONTRACT_VERSION } from "./contract-version.mjs";
export { validateIntakeBatch } from "./intake-batch.mjs";
export const FOUNDRY_BINDING_UPSERT_SCHEMA_VERSION = "nodedc.foundry.binding-upsert/v1"; export const FOUNDRY_BINDING_UPSERT_SCHEMA_VERSION = "nodedc.foundry.binding-upsert/v1";
import {
SECRET_LIKE_KEY,
SECRET_LIKE_VALUE,
} from "./sensitive-field-policy.mjs";
export {
L2_CONNECTION_INSTANCE_SCHEMA_VERSION,
L2_TEMPLATE_SCHEMA_VERSION,
PROVIDER_PACKAGE_SCHEMA_VERSION,
SEMANTIC_MAPPING_SCHEMA_VERSION,
instantiateL2Connection,
validateProviderPackage,
} from "./provider-package.mjs";
export { export {
DATA_PRODUCT_PATCH_SCHEMA_VERSION, DATA_PRODUCT_PATCH_SCHEMA_VERSION,
DATA_PRODUCT_PUBLISH_SCHEMA_VERSION, DATA_PRODUCT_PUBLISH_SCHEMA_VERSION,
@ -27,6 +10,24 @@ export {
validateDataProductSnapshot, validateDataProductSnapshot,
} from "./data-product.mjs"; } from "./data-product.mjs";
export {
ENGINE_CREDENTIAL_SINK_AUDIT_SCHEMA_VERSION,
ENGINE_CREDENTIAL_SINK_CAPABILITY_TYPES,
ENGINE_CREDENTIAL_SINK_PROVISION_SCHEMA_VERSION,
ENGINE_CREDENTIAL_SINK_RECEIPT_SCHEMA_VERSION,
ENGINE_CREDENTIAL_SINK_ROLLBACK_RECEIPT_SCHEMA_VERSION,
ENGINE_CREDENTIAL_SINK_ROLLBACK_SCHEMA_VERSION,
computeEngineCredentialCapabilityDigest,
computeEngineCredentialSinkPolicyHash,
computeEngineCredentialSinkReceiptHash,
engineCredentialSinkAuditTargets,
validateEngineCredentialSinkAudit,
validateEngineCredentialSinkProvision,
validateEngineCredentialSinkReceipt,
validateEngineCredentialSinkRollback,
validateEngineCredentialSinkRollbackReceipt,
} from "./engine-credential-sink.mjs";
export { export {
ENGINE_PRIVATE_EXTENSION_APPLY_RECEIPT_SCHEMA_VERSION, ENGINE_PRIVATE_EXTENSION_APPLY_RECEIPT_SCHEMA_VERSION,
ENGINE_PRIVATE_EXTENSION_APPLY_REQUEST_SCHEMA_VERSION, ENGINE_PRIVATE_EXTENSION_APPLY_REQUEST_SCHEMA_VERSION,
@ -54,11 +55,16 @@ export {
const COLLECTION_MODES = new Set(["realtime", "manual", "weekly", "history"]); const COLLECTION_MODES = new Set(["realtime", "manual", "weekly", "history"]);
const DELIVERY_MODES = new Set(["snapshot", "snapshot+patch", "query"]); const DELIVERY_MODES = new Set(["snapshot", "snapshot+patch", "query"]);
const CAPABILITY_CLASSIFICATIONS = new Set(["read", "metadata", "write", "destructive", "unknown"]); const CAPABILITY_CLASSIFICATIONS = new Set(["read", "metadata", "write", "destructive", "unknown"]);
const SECRET_LIKE_KEY = /(token|secret|password|authorization|access[_-]?token|refresh[_-]?token|api[_-]?key)/i;
const SECRET_LIKE_REFERENCE = /(?:[?&](?:token|secret|password|authorization|access[_-]?token|refresh[_-]?token|api[_-]?key)=|(?:bearer|basic)\s+)/i;
const SECRET_LIKE_VALUE = /(?:ndc_edp(?:wb|rb)_[A-Za-z0-9_-]*|[?&](?:token|secret|password|authorization|access[_-]?token|refresh[_-]?token|api[_-]?key)=|(?:bearer|basic)\s+\S+|eyJ[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+)/i;
const IDENTIFIER = /^[a-z][a-z0-9._:-]{2,127}$/; const IDENTIFIER = /^[a-z][a-z0-9._:-]{2,127}$/;
const FOUNDRY_APPLICATION_ID = /^[0-9a-f]{8}-[0-9a-f]{4}-[1-5][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/i; const FOUNDRY_APPLICATION_ID = /^[0-9a-f]{8}-[0-9a-f]{4}-[1-5][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/i;
const FOUNDRY_PAGE_ID = /^[a-z0-9][a-z0-9-]{0,79}$/; const FOUNDRY_PAGE_ID = /^[a-z0-9][a-z0-9-]{0,79}$/;
const FOUNDRY_SLOT_ID = /^[A-Za-z0-9][A-Za-z0-9-]{0,79}$/; const FOUNDRY_SLOT_ID = /^[A-Za-z0-9][A-Za-z0-9-]{0,79}$/;
const CONTRACT_VERSION = /^\d+\.\d+\.\d+(?:[-+][a-z0-9.-]+)?$/i; const CONTRACT_VERSION = /^\d+\.\d+\.\d+(?:[-+][a-z0-9.-]+)?$/i;
const MAX_BATCH_SEQUENCE = 2_147_483_647;
const MAX_FACT_ATTRIBUTES_BYTES = 64 * 1024;
const PROVIDER_MANIFEST_KEYS = new Set(["schemaVersion", "id", "providerId", "version", "ontology", "l2Template", "capabilities", "dataProductIds"]); const PROVIDER_MANIFEST_KEYS = new Set(["schemaVersion", "id", "providerId", "version", "ontology", "l2Template", "capabilities", "dataProductIds"]);
const CONNECTION_PROFILE_KEYS = new Set(["schemaVersion", "id", "providerId", "tenantId", "credentialRef", "scope"]); const CONNECTION_PROFILE_KEYS = new Set(["schemaVersion", "id", "providerId", "tenantId", "credentialRef", "scope"]);
@ -153,8 +159,8 @@ export function validateConnectionProfile(value) {
requiredString(value?.credentialRef?.owner, "credentialRef.owner", errors); requiredString(value?.credentialRef?.owner, "credentialRef.owner", errors);
requiredString(value?.credentialRef?.reference, "credentialRef.reference", errors); requiredString(value?.credentialRef?.reference, "credentialRef.reference", errors);
if (value?.credentialRef?.owner && value.credentialRef.owner !== "ndc_l2_credentials") { if (value?.credentialRef?.owner && value.credentialRef.owner !== "engine") {
errors.push("credentialRef.owner_must_be_ndc_l2_credentials"); errors.push("credentialRef.owner_must_be_engine");
} }
if (containsSecretLikeMaterial(value)) errors.push("profile_must_not_contain_secret_material"); if (containsSecretLikeMaterial(value)) errors.push("profile_must_not_contain_secret_material");
if (value?.scope !== undefined) { if (value?.scope !== undefined) {
@ -295,6 +301,45 @@ export function validateFoundryBindingUpsert(value) {
return result(errors); return result(errors);
} }
/**
* Provider-neutral batch written by an L2 connector to External Data Plane.
* The payload deliberately describes source facts rather than any provider
* field names, customer entities or renderer representation.
*/
export function validateIntakeBatch(value) {
const errors = baseErrors(value, "intakeBatch");
rejectUnknownKeys(value, new Set(["schemaVersion", "source", "contract", "batch", "raw", "facts"]), "intakeBatch", errors);
rejectUnknownKeys(value?.source, new Set(["providerId", "tenantId", "connectionId"]), "source", errors);
rejectUnknownKeys(value?.contract, new Set(["dataProductId", "ontologyRevision", "version"]), "contract", errors);
rejectUnknownKeys(value?.batch, new Set(["runId", "sequence", "idempotencyKey", "receivedAt"]), "batch", errors);
requiredIdentifier(value?.source?.providerId, "source.providerId", errors);
requiredIdentifier(value?.source?.tenantId, "source.tenantId", errors);
requiredIdentifier(value?.source?.connectionId, "source.connectionId", errors);
requiredIdentifier(value?.contract?.dataProductId, "contract.dataProductId", errors);
requiredIdentifier(value?.contract?.ontologyRevision, "contract.ontologyRevision", errors);
requiredString(value?.contract?.version, "contract.version", errors);
if (value?.contract?.version && !CONTRACT_VERSION.test(value.contract.version)) {
errors.push("contract.version_must_be_semver");
}
requiredIdentifier(value?.batch?.runId, "batch.runId", errors);
requiredIdentifier(value?.batch?.idempotencyKey, "batch.idempotencyKey", errors);
if (!Number.isInteger(value?.batch?.sequence) || value.batch.sequence < 0 || value.batch.sequence > MAX_BATCH_SEQUENCE) {
errors.push("batch.sequence_must_be_integer_0_to_2147483647");
}
requiredIsoTimestamp(value?.batch?.receivedAt, "batch.receivedAt", errors);
if (!Array.isArray(value?.facts) || value.facts.length === 0) {
errors.push("facts_must_be_nonempty_array");
} else {
value.facts.forEach((fact, index) => validateFact(fact, `facts[${index}]`, errors));
}
if (value?.raw !== undefined) validateRawEnvelope(value.raw, errors);
if (containsSecretLikeMaterial(value)) errors.push("intake_must_not_contain_secret_material");
return result(errors);
}
export function assertValid(validator, value) { export function assertValid(validator, value) {
const validation = validator(value); const validation = validator(value);
if (!validation.ok) throw new Error(`external_provider_contract_invalid:${validation.errors.join(",")}`); if (!validation.ok) throw new Error(`external_provider_contract_invalid:${validation.errors.join(",")}`);
@ -349,6 +394,66 @@ function validateUniqueIdentifierArray(value, path, errors) {
if (new Set(value).size !== value.length) errors.push(`${path}_must_not_contain_duplicates`); if (new Set(value).size !== value.length) errors.push(`${path}_must_not_contain_duplicates`);
} }
function requiredIsoTimestamp(value, path, errors) {
if (typeof value !== "string" || Number.isNaN(Date.parse(value))) errors.push(`${path}_invalid_timestamp`);
}
function validateFact(value, path, errors) {
if (!isPlainObject(value)) {
errors.push(`${path}_must_be_object`);
return;
}
rejectUnknownKeys(value, new Set(["sourceId", "semanticType", "observedAt", "attributes", "geometry"]), path, errors);
requiredIdentifier(value.sourceId, `${path}.sourceId`, errors);
requiredIdentifier(value.semanticType, `${path}.semanticType`, errors);
requiredIsoTimestamp(value.observedAt, `${path}.observedAt`, errors);
if (value.attributes !== undefined) {
if (!isPlainObject(value.attributes)) {
errors.push(`${path}.attributes_must_be_object`);
} else if (serializedByteLength(value.attributes) > MAX_FACT_ATTRIBUTES_BYTES) {
errors.push(`${path}.attributes_size_exceeded`);
}
}
if (value.geometry !== undefined) validatePointGeometry(value.geometry, `${path}.geometry`, errors);
}
function validatePointGeometry(value, path, errors) {
if (!isPlainObject(value) || value.type !== "Point" || !Array.isArray(value.coordinates) || value.coordinates.length !== 2) {
errors.push(`${path}_must_be_geojson_point`);
return;
}
rejectUnknownKeys(value, new Set(["type", "coordinates"]), path, errors);
if (!value.coordinates.every((coordinate) => typeof coordinate === "number" && Number.isFinite(coordinate))) {
errors.push(`${path}_coordinates_must_be_finite_numbers`);
return;
}
const [longitude, latitude] = value.coordinates;
if (longitude < -180 || longitude > 180) errors.push(`${path}.longitude_out_of_range`);
if (latitude < -90 || latitude > 90) errors.push(`${path}.latitude_out_of_range`);
}
function validateRawEnvelope(value, errors) {
if (!isPlainObject(value)) {
errors.push("raw_must_be_object");
return;
}
rejectUnknownKeys(value, new Set(["contentType", "payload", "hash", "ref", "retentionDays"]), "raw", errors);
requiredString(value.contentType, "raw.contentType", errors);
if (value.payload === undefined && value.ref === undefined) errors.push("raw_requires_payload_or_ref");
if (value.payload !== undefined) errors.push("raw.inline_payload_not_supported");
if (value.payload === undefined) requiredString(value.hash, "raw.hash", errors);
if (value.hash !== undefined) requiredString(value.hash, "raw.hash", errors);
if (value.ref !== undefined) {
requiredString(value.ref, "raw.ref", errors);
if (typeof value.ref === "string" && SECRET_LIKE_REFERENCE.test(value.ref)) {
errors.push("raw.ref_must_not_contain_secret_material");
}
}
if (value.retentionDays !== undefined && (!Number.isInteger(value.retentionDays) || value.retentionDays < 1)) {
errors.push("raw.retentionDays_must_be_positive_integer");
}
}
function isPlainObject(value) { function isPlainObject(value) {
return Boolean(value) && typeof value === "object" && !Array.isArray(value); return Boolean(value) && typeof value === "object" && !Array.isArray(value);
} }
@ -360,6 +465,14 @@ function rejectUnknownKeys(value, allowedKeys, path, errors) {
} }
} }
function serializedByteLength(value) {
try {
return Buffer.byteLength(JSON.stringify(value));
} catch {
return Number.POSITIVE_INFINITY;
}
}
function containsSecretLikeMaterial(value) { function containsSecretLikeMaterial(value) {
if (typeof value === "string") return SECRET_LIKE_VALUE.test(value); if (typeof value === "string") return SECRET_LIKE_VALUE.test(value);
if (Array.isArray(value)) return value.some(containsSecretLikeMaterial); if (Array.isArray(value)) return value.some(containsSecretLikeMaterial);

152
packages/external-provider-contract/src/intake-batch.mjs
View File

@ -1,152 +0,0 @@
import { EXTERNAL_PROVIDER_CONTRACT_VERSION } from "./contract-version.mjs";
import {
SECRET_LIKE_KEY,
SECRET_LIKE_REFERENCE,
SECRET_LIKE_VALUE,
} from "./sensitive-field-policy.mjs";
const IDENTIFIER = /^[a-z][a-z0-9._:-]{2,127}$/;
const CONTRACT_VERSION = /^\d+\.\d+\.\d+(?:[-+][a-z0-9.-]+)?$/i;
const MAX_BATCH_SEQUENCE = 2_147_483_647;
const MAX_FACT_ATTRIBUTES_BYTES = 64 * 1024;
/** Provider-neutral batch written by an L2 connector to External Data Plane. */
export function validateIntakeBatch(value) {
const errors = baseErrors(value, "intakeBatch");
rejectUnknownKeys(value, new Set(["schemaVersion", "source", "contract", "batch", "raw", "facts"]), "intakeBatch", errors);
rejectUnknownKeys(value?.source, new Set(["providerId", "tenantId", "connectionId"]), "source", errors);
rejectUnknownKeys(value?.contract, new Set(["dataProductId", "ontologyRevision", "version"]), "contract", errors);
rejectUnknownKeys(value?.batch, new Set(["runId", "sequence", "idempotencyKey", "receivedAt"]), "batch", errors);
requiredIdentifier(value?.source?.providerId, "source.providerId", errors);
requiredIdentifier(value?.source?.tenantId, "source.tenantId", errors);
requiredIdentifier(value?.source?.connectionId, "source.connectionId", errors);
requiredIdentifier(value?.contract?.dataProductId, "contract.dataProductId", errors);
requiredIdentifier(value?.contract?.ontologyRevision, "contract.ontologyRevision", errors);
requiredString(value?.contract?.version, "contract.version", errors);
if (value?.contract?.version && !CONTRACT_VERSION.test(value.contract.version)) {
errors.push("contract.version_must_be_semver");
}
requiredIdentifier(value?.batch?.runId, "batch.runId", errors);
requiredIdentifier(value?.batch?.idempotencyKey, "batch.idempotencyKey", errors);
if (!Number.isInteger(value?.batch?.sequence) || value.batch.sequence < 0 || value.batch.sequence > MAX_BATCH_SEQUENCE) {
errors.push("batch.sequence_must_be_integer_0_to_2147483647");
}
requiredIsoTimestamp(value?.batch?.receivedAt, "batch.receivedAt", errors);
if (!Array.isArray(value?.facts) || value.facts.length === 0) {
errors.push("facts_must_be_nonempty_array");
} else {
value.facts.forEach((fact, index) => validateFact(fact, `facts[${index}]`, errors));
}
if (value?.raw !== undefined) validateRawEnvelope(value.raw, errors);
if (containsSecretLikeMaterial(value)) errors.push("intake_must_not_contain_secret_material");
return result(errors);
}
function baseErrors(value, label) {
const errors = [];
if (!isPlainObject(value)) return [`${label}_must_be_object`];
if (value.schemaVersion !== EXTERNAL_PROVIDER_CONTRACT_VERSION) errors.push("schemaVersion_mismatch");
return errors;
}
function requiredString(value, path, errors) {
if (typeof value !== "string" || !value.trim()) errors.push(`${path}_required`);
}
function requiredIdentifier(value, path, errors) {
if (typeof value !== "string" || !IDENTIFIER.test(value)) errors.push(`${path}_invalid`);
}
function requiredIsoTimestamp(value, path, errors) {
if (typeof value !== "string" || Number.isNaN(Date.parse(value))) errors.push(`${path}_invalid_timestamp`);
}
function validateFact(value, path, errors) {
if (!isPlainObject(value)) {
errors.push(`${path}_must_be_object`);
return;
}
rejectUnknownKeys(value, new Set(["sourceId", "semanticType", "observedAt", "attributes", "geometry"]), path, errors);
requiredIdentifier(value.sourceId, `${path}.sourceId`, errors);
requiredIdentifier(value.semanticType, `${path}.semanticType`, errors);
requiredIsoTimestamp(value.observedAt, `${path}.observedAt`, errors);
if (value.attributes !== undefined) {
if (!isPlainObject(value.attributes)) {
errors.push(`${path}.attributes_must_be_object`);
} else if (serializedByteLength(value.attributes) > MAX_FACT_ATTRIBUTES_BYTES) {
errors.push(`${path}.attributes_size_exceeded`);
}
}
if (value.geometry !== undefined) validatePointGeometry(value.geometry, `${path}.geometry`, errors);
}
function validatePointGeometry(value, path, errors) {
if (!isPlainObject(value) || value.type !== "Point" || !Array.isArray(value.coordinates) || value.coordinates.length !== 2) {
errors.push(`${path}_must_be_geojson_point`);
return;
}
rejectUnknownKeys(value, new Set(["type", "coordinates"]), path, errors);
if (!value.coordinates.every((coordinate) => typeof coordinate === "number" && Number.isFinite(coordinate))) {
errors.push(`${path}_coordinates_must_be_finite_numbers`);
return;
}
const [longitude, latitude] = value.coordinates;
if (longitude < -180 || longitude > 180) errors.push(`${path}.longitude_out_of_range`);
if (latitude < -90 || latitude > 90) errors.push(`${path}.latitude_out_of_range`);
}
function validateRawEnvelope(value, errors) {
if (!isPlainObject(value)) {
errors.push("raw_must_be_object");
return;
}
rejectUnknownKeys(value, new Set(["contentType", "payload", "hash", "ref", "retentionDays"]), "raw", errors);
requiredString(value.contentType, "raw.contentType", errors);
if (value.payload === undefined && value.ref === undefined) errors.push("raw_requires_payload_or_ref");
if (value.payload !== undefined) errors.push("raw.inline_payload_not_supported");
if (value.payload === undefined) requiredString(value.hash, "raw.hash", errors);
if (value.hash !== undefined) requiredString(value.hash, "raw.hash", errors);
if (value.ref !== undefined) {
requiredString(value.ref, "raw.ref", errors);
if (typeof value.ref === "string" && SECRET_LIKE_REFERENCE.test(value.ref)) {
errors.push("raw.ref_must_not_contain_secret_material");
}
}
if (value.retentionDays !== undefined && (!Number.isInteger(value.retentionDays) || value.retentionDays < 1)) {
errors.push("raw.retentionDays_must_be_positive_integer");
}
}
function isPlainObject(value) {
return Boolean(value) && typeof value === "object" && !Array.isArray(value);
}
function rejectUnknownKeys(value, allowedKeys, path, errors) {
if (!isPlainObject(value)) return;
for (const key of Object.keys(value)) {
if (!allowedKeys.has(key)) errors.push(`${path}.${key}_not_allowed`);
}
}
function serializedByteLength(value) {
try {
return Buffer.byteLength(JSON.stringify(value));
} catch {
return Number.POSITIVE_INFINITY;
}
}
function containsSecretLikeMaterial(value) {
if (typeof value === "string") return SECRET_LIKE_VALUE.test(value);
if (Array.isArray(value)) return value.some(containsSecretLikeMaterial);
if (!isPlainObject(value)) return false;
return Object.entries(value).some(([key, child]) => SECRET_LIKE_KEY.test(key) || containsSecretLikeMaterial(child));
}
function result(errors) {
const uniqueErrors = [...new Set(errors)];
return Object.freeze({ ok: uniqueErrors.length === 0, errors: Object.freeze(uniqueErrors) });
}

1006
packages/external-provider-contract/src/provider-package.mjs
View File

File diff suppressed because it is too large Load Diff

16
packages/external-provider-contract/src/sensitive-field-policy.mjs
View File

@ -1,16 +0,0 @@
export const NDC_SECRET_CAPABILITY_SOURCE =
String.raw`ndc_(?:edp(?:wb|rb|pr)|fndbg)_[A-Za-z0-9_-]+`;
export const SECRET_LIKE_KEY =
/(token|secret|password|authorization|access[_-]?token|refresh[_-]?token|api[_-]?key)/i;
export const SECRET_LIKE_MATERIAL_KEY =
/(token|secret|password|authorization|access[_-]?token|refresh[_-]?token|api[_-]?key|material|value)/i;
export const SECRET_LIKE_REFERENCE =
/(?:[?&](?:token|secret|password|authorization|access[_-]?token|refresh[_-]?token|api[_-]?key)=|(?:bearer|basic)\s+)/i;
export const SECRET_LIKE_VALUE = new RegExp(
`(?:${NDC_SECRET_CAPABILITY_SOURCE}|[?&](?:token|secret|password|authorization|access[_-]?token|refresh[_-]?token|api[_-]?key)=|(?:bearer|basic)\\s+\\S+|eyJ[A-Za-z0-9_-]+\\.[A-Za-z0-9_-]+\\.[A-Za-z0-9_-]+)`,
"i",
);

8
packages/external-provider-contract/test/contract.test.mjs
View File

@ -15,10 +15,6 @@ import { geliosPositionsCurrentExample } from "../examples/gelios-positions-curr
assert.equal(validateProviderManifest(geliosPositionsCurrentExample.providerManifest).ok, true); assert.equal(validateProviderManifest(geliosPositionsCurrentExample.providerManifest).ok, true);
assert.equal(validateConnectionProfile(geliosPositionsCurrentExample.connection).ok, true); assert.equal(validateConnectionProfile(geliosPositionsCurrentExample.connection).ok, true);
assert.equal(validateConnectionProfile({
...geliosPositionsCurrentExample.connection,
credentialRef: { ...geliosPositionsCurrentExample.connection.credentialRef, owner: "engine" },
}).errors.includes("credentialRef.owner_must_be_ndc_l2_credentials"), true);
assert.equal(validateCollectionProfile(geliosPositionsCurrentExample.collectionProfile).ok, true); assert.equal(validateCollectionProfile(geliosPositionsCurrentExample.collectionProfile).ok, true);
assert.equal(validateDataProduct(geliosPositionsCurrentExample.dataProduct).ok, true); assert.equal(validateDataProduct(geliosPositionsCurrentExample.dataProduct).ok, true);
assert.equal(validateFoundryBinding(geliosPositionsCurrentExample.foundryBinding).ok, true); assert.equal(validateFoundryBinding(geliosPositionsCurrentExample.foundryBinding).ok, true);
@ -112,10 +108,6 @@ assert.equal(validateIntakeBatch({
...neutralIntakeBatch, ...neutralIntakeBatch,
facts: [{ ...neutralIntakeBatch.facts[0], attributes: { metadata: "ndc_edprb_abcdefghijklmnopqrstuvwxyz0123456789ABCDE" } }], facts: [{ ...neutralIntakeBatch.facts[0], attributes: { metadata: "ndc_edprb_abcdefghijklmnopqrstuvwxyz0123456789ABCDE" } }],
}).errors.includes("intake_must_not_contain_secret_material"), true); }).errors.includes("intake_must_not_contain_secret_material"), true);
assert.equal(validateIntakeBatch({
...neutralIntakeBatch,
facts: [{ ...neutralIntakeBatch.facts[0], attributes: { metadata: `ndc_edppr_${"P".repeat(43)}` } }],
}).errors.includes("intake_must_not_contain_secret_material"), true);
assert.equal(validateIntakeBatch({ assert.equal(validateIntakeBatch({
...neutralIntakeBatch, ...neutralIntakeBatch,
raw: { contentType: "application/json", hash: "prefix ndc_edpwb_abcdefghijklmnopqrstuvwxyz0123456789ABCDE suffix", ref: "restricted://raw/fixture" }, raw: { contentType: "application/json", hash: "prefix ndc_edpwb_abcdefghijklmnopqrstuvwxyz0123456789ABCDE suffix", ref: "restricted://raw/fixture" },

8
packages/external-provider-contract/test/data-product.test.mjs
View File

@ -34,10 +34,6 @@ assert.equal(validateDataProductPublish({
...publish, ...publish,
facts: [{ ...fact, attributes: { accessToken: "must-never-cross-the-boundary" } }], facts: [{ ...fact, attributes: { accessToken: "must-never-cross-the-boundary" } }],
}).errors.includes("publish_must_not_contain_secret_material"), true); }).errors.includes("publish_must_not_contain_secret_material"), true);
assert.equal(validateDataProductPublish({
...publish,
facts: [{ ...fact, attributes: { status: `ndc_edppr_${"P".repeat(43)}` } }],
}).errors.includes("publish_must_not_contain_secret_material"), true);
assert.equal(validateDataProductPublish({ assert.equal(validateDataProductPublish({
...publish, ...publish,
facts: [fact, { ...fact, observedAt: "2026-07-15T10:00:01.000Z" }], facts: [fact, { ...fact, observedAt: "2026-07-15T10:00:01.000Z" }],
@ -89,10 +85,6 @@ assert.equal(validateDataProductSnapshot({
...snapshot, ...snapshot,
facts: [{ ...canonicalFact, attributes: { status: "ndc_edprb_forbidden-reader-token" } }], facts: [{ ...canonicalFact, attributes: { status: "ndc_edprb_forbidden-reader-token" } }],
}).errors.includes("snapshot_must_not_contain_secret_material"), true); }).errors.includes("snapshot_must_not_contain_secret_material"), true);
assert.equal(validateDataProductSnapshot({
...snapshot,
facts: [{ ...canonicalFact, attributes: { status: `ndc_edppr_${"P".repeat(43)}` } }],
}).errors.includes("snapshot_must_not_contain_secret_material"), true);
const patch = { const patch = {
schemaVersion: DATA_PRODUCT_PATCH_SCHEMA_VERSION, schemaVersion: DATA_PRODUCT_PATCH_SCHEMA_VERSION,

242
packages/external-provider-contract/test/engine-credential-sink.test.mjs Normal file
View File

@ -0,0 +1,242 @@
import assert from "node:assert/strict";
import { generateKeyPairSync, sign } from "node:crypto";
import {
ENGINE_CREDENTIAL_SINK_AUDIT_SCHEMA_VERSION,
ENGINE_CREDENTIAL_SINK_PROVISION_SCHEMA_VERSION,
ENGINE_CREDENTIAL_SINK_RECEIPT_SCHEMA_VERSION,
ENGINE_CREDENTIAL_SINK_ROLLBACK_RECEIPT_SCHEMA_VERSION,
ENGINE_CREDENTIAL_SINK_ROLLBACK_SCHEMA_VERSION,
computeEngineCredentialCapabilityDigest,
computeEngineCredentialSinkPolicyHash,
computeEngineCredentialSinkReceiptHash,
engineCredentialSinkAuditTargets,
validateEngineCredentialSinkAudit,
validateEngineCredentialSinkProvision,
validateEngineCredentialSinkReceipt,
validateEngineCredentialSinkRollback,
validateEngineCredentialSinkRollbackReceipt,
} from "../src/index.mjs";
const hash = (character) => `sha256:${character.repeat(64)}`;
const issuer = { serviceId: "platform.external-data-plane", keyId: "edp-issuer-20260715-001" };
const { publicKey: issuerPublicKey, privateKey: issuerPrivateKey } = generateKeyPairSync("ed25519");
const issuerPublicKeys = { [`${issuer.serviceId}:${issuer.keyId}`]: issuerPublicKey };
const target = (nodeId, nodeType, credentialType) => ({
workflowId: "WCb62yGL8v",
workflowRevision: "revision-20260715-001",
nodeId,
nodeType,
credentialType,
});
const request = {
schemaVersion: ENGINE_CREDENTIAL_SINK_PROVISION_SCHEMA_VERSION,
transaction: {
id: "credential-transaction-20260715-001",
idempotencyKey: "credential-transaction-20260715-001",
requestedAt: "2026-07-15T18:00:00.000Z",
requestExpiresAt: "2026-07-15T18:10:00.000Z",
policyHash: hash("0"),
failureMode: "rollback-all",
issuer,
attestation: { algorithm: "Ed25519", signature: "A".repeat(86) },
},
bindings: [
{
bindingId: "positions-writer",
capabilityType: "external-data-plane.writer",
grantId: "writer-grant-001",
target: target(
"publish-node-001",
"n8n-nodes-ndc.ndcDataProductPublish",
"ndcDataProductWriterApi",
),
expiresAt: "2026-08-15T18:00:00.000Z",
policyHash: hash("1"),
material: { format: "opaque-bearer", value: `ndc_edpwb_${"A".repeat(43)}` },
},
{
bindingId: "positions-reader",
capabilityType: "external-data-plane.reader",
grantId: "reader-grant-001",
target: target(
"read-node-001",
"n8n-nodes-ndc.ndcDataProductRead",
"ndcDataProductReaderApi",
),
expiresAt: "2026-08-15T18:00:00.000Z",
policyHash: hash("2"),
material: { format: "opaque-bearer", value: `ndc_edprb_${"B".repeat(43)}` },
},
{
bindingId: "positions-foundry",
capabilityType: "foundry.binding",
grantId: "foundry-grant-001",
target: target(
"foundry-node-001",
"n8n-nodes-ndc.ndcFoundryBinding",
"ndcFoundryBindingApi",
),
expiresAt: "2026-08-15T18:00:00.000Z",
policyHash: hash("3"),
material: { format: "opaque-bearer", value: `ndc_fndbg_${"C".repeat(43)}` },
},
],
};
for (const binding of request.bindings) {
binding.capabilityDigest = computeEngineCredentialCapabilityDigest(binding.material.value);
}
attest(request);
const provisionValidationNow = "2026-07-15T18:00:01.000Z";
assert.equal(validateEngineCredentialSinkProvision(request, { now: provisionValidationNow, issuerPublicKeys }).ok, true);
assert.equal(validateEngineCredentialSinkProvision(request, {
now: provisionValidationNow,
}).errors.includes("transaction.issuer_public_key_required"), true);
assert.equal(validateEngineCredentialSinkProvision({
...request,
transaction: { ...request.transaction, policyHash: hash("f") },
}, { now: provisionValidationNow, issuerPublicKeys }).errors.includes("transaction.policyHash_mismatch"), true);
assert.equal(validateEngineCredentialSinkProvision({
...request,
bindings: [{
...request.bindings[0],
target: { ...request.bindings[0].target, nodeType: "n8n-nodes-ndc.ndcDataProductRead" },
}],
}, { now: provisionValidationNow, issuerPublicKeys }).errors.includes("bindings[0].target.nodeType_capability_mismatch"), true);
assert.equal(validateEngineCredentialSinkProvision({
...request,
bindings: [{
...request.bindings[0],
material: { format: "opaque-bearer", value: request.bindings[1].material.value },
}],
}, { now: provisionValidationNow, issuerPublicKeys }).errors.includes("bindings[0].material.value_invalid_for_capability"), true);
const swappedCapabilityRequest = structuredClone(request);
swappedCapabilityRequest.bindings[0].material.value = `ndc_edpwb_${"D".repeat(43)}`;
assert.equal(validateEngineCredentialSinkProvision(swappedCapabilityRequest, {
now: provisionValidationNow,
issuerPublicKeys,
}).errors.includes("bindings[0].capabilityDigest_material_mismatch"), true);
const resignedByAttackerRequest = structuredClone(swappedCapabilityRequest);
resignedByAttackerRequest.bindings[0].capabilityDigest = computeEngineCredentialCapabilityDigest(
resignedByAttackerRequest.bindings[0].material.value,
);
resignedByAttackerRequest.transaction.policyHash = computeEngineCredentialSinkPolicyHash(resignedByAttackerRequest);
assert.equal(validateEngineCredentialSinkProvision(resignedByAttackerRequest, {
now: provisionValidationNow,
issuerPublicKeys,
}).errors.includes("transaction.attestation_invalid"), true);
const staleRequest = structuredClone(request);
staleRequest.transaction.requestedAt = "2020-01-01T00:00:00.000Z";
staleRequest.transaction.requestExpiresAt = "2020-01-01T00:10:00.000Z";
attest(staleRequest);
assert.equal(validateEngineCredentialSinkProvision(staleRequest, { now: provisionValidationNow, issuerPublicKeys }).errors.includes("request_expired"), true);
const futureRequest = structuredClone(request);
futureRequest.transaction.requestedAt = "2026-07-15T18:02:00.000Z";
futureRequest.transaction.requestExpiresAt = "2026-07-15T18:12:00.000Z";
attest(futureRequest);
assert.equal(validateEngineCredentialSinkProvision(futureRequest, { now: provisionValidationNow, issuerPublicKeys }).errors.includes("requestedAt_exceeds_clock_skew"), true);
const credentials = request.bindings.map((binding, index) => ({
bindingId: binding.bindingId,
capabilityType: binding.capabilityType,
grantId: binding.grantId,
target: binding.target,
credentialRef: `engcred_${index + 1}_opaque_reference`,
expiresAt: binding.expiresAt,
policyHash: binding.policyHash,
capabilityDigest: binding.capabilityDigest,
disposition: "created",
}));
const receipt = {
schemaVersion: ENGINE_CREDENTIAL_SINK_RECEIPT_SCHEMA_VERSION,
transactionId: request.transaction.id,
idempotencyKey: request.transaction.idempotencyKey,
outcome: "committed",
policyHash: request.transaction.policyHash,
processedAt: "2026-07-15T18:00:02.000Z",
credentials,
rollback: { status: "not-required" },
};
assert.equal(validateEngineCredentialSinkReceipt(receipt, { request, issuerPublicKeys }).ok, true);
assert.equal(validateEngineCredentialSinkReceipt({
...receipt,
credentials: [{ ...credentials[0], target: { ...credentials[0].target, nodeId: "wrong-node" } }, ...credentials.slice(1)],
}, { request, issuerPublicKeys }).errors.includes("credentials_request_target_mismatch"), true);
assert.equal(validateEngineCredentialSinkReceipt({
...receipt,
capability: request.bindings[0].material.value,
}).errors.includes("credentialSinkReceipt.capability_not_allowed"), true);
assert.equal(validateEngineCredentialSinkReceipt({
...receipt,
outcome: "rolled-back",
credentials: credentials.slice(0, 1),
rollback: { status: "complete", completedAt: "2026-07-15T18:00:02.000Z" },
errorCode: "engine_binding_failed",
}).errors.includes("noncommitted_credentials_must_be_empty"), true);
const committedReceiptHash = computeEngineCredentialSinkReceiptHash(receipt);
const rollbackRequest = {
schemaVersion: ENGINE_CREDENTIAL_SINK_ROLLBACK_SCHEMA_VERSION,
rollback: {
id: "credential-rollback-20260715-001",
idempotencyKey: "credential-rollback-20260715-001",
transactionId: request.transaction.id,
requestedAt: "2026-07-15T18:05:00.000Z",
requestExpiresAt: "2026-07-15T18:10:00.000Z",
policyHash: request.transaction.policyHash,
committedReceiptHash,
reasonCode: "operator_requested",
},
};
assert.equal(validateEngineCredentialSinkRollback(rollbackRequest, { now: "2026-07-15T18:05:01.000Z" }).ok, true);
const staleRollbackRequest = structuredClone(rollbackRequest);
staleRollbackRequest.rollback.requestedAt = "2020-01-01T00:00:00.000Z";
staleRollbackRequest.rollback.requestExpiresAt = "2020-01-01T00:10:00.000Z";
assert.equal(validateEngineCredentialSinkRollback(staleRollbackRequest, { now: "2026-07-15T18:05:01.000Z" }).errors.includes("request_expired"), true);
const rollbackReceipt = {
schemaVersion: ENGINE_CREDENTIAL_SINK_ROLLBACK_RECEIPT_SCHEMA_VERSION,
rollbackId: rollbackRequest.rollback.id,
transactionId: rollbackRequest.rollback.transactionId,
outcome: "rolled-back",
policyHash: rollbackRequest.rollback.policyHash,
committedReceiptHash,
processedAt: "2026-07-15T18:05:01.000Z",
};
assert.equal(validateEngineCredentialSinkRollbackReceipt(rollbackReceipt, { request: rollbackRequest }).ok, true);
const audit = {
schemaVersion: ENGINE_CREDENTIAL_SINK_AUDIT_SCHEMA_VERSION,
eventId: "credential-audit-20260715-001",
transactionId: request.transaction.id,
operationId: request.transaction.id,
operation: "provision",
outcome: "committed",
occurredAt: receipt.processedAt,
policyHash: request.transaction.policyHash,
principal: { serviceId: "platform.credential-provisioner", fingerprint: hash("a") },
targets: engineCredentialSinkAuditTargets(request, receipt),
};
assert.equal(validateEngineCredentialSinkAudit(audit).ok, true);
assert.equal(JSON.stringify(audit).includes("ndc_edpwb_"), false);
assert.equal(JSON.stringify(audit).includes("engcred_1_opaque_reference"), false);
assert.equal(validateEngineCredentialSinkAudit({
...audit,
authorization: `Bearer ${request.bindings[0].material.value}`,
}).errors.includes("audit_must_not_contain_secret_material"), true);
function attest(value) {
value.transaction.policyHash = computeEngineCredentialSinkPolicyHash(value);
value.transaction.attestation.signature = sign(
null,
Buffer.from(value.transaction.policyHash, "utf8"),
issuerPrivateKey,
).toString("base64url");
}
console.log("external-provider credential sink contract: ok");

10
packages/external-provider-contract/test/engine-private-extension.test.mjs
View File

@ -132,16 +132,6 @@ assert.equal(
}).errors.includes("transition_policy_mismatch"), }).errors.includes("transition_policy_mismatch"),
true, true,
); );
assert.equal(
validateEnginePrivateExtensionPlan({
...activatePlan,
transition: {
...transition(),
loaderEnvironment: { ...transition().loaderEnvironment, NODE_PATH: "/tmp/untrusted-node-modules" },
},
}).errors.includes("transition.loaderEnvironment.NODE_PATH_not_allowed"),
true,
);
const applyRequest = { const applyRequest = {
schemaVersion: ENGINE_PRIVATE_EXTENSION_APPLY_REQUEST_SCHEMA_VERSION, schemaVersion: ENGINE_PRIVATE_EXTENSION_APPLY_REQUEST_SCHEMA_VERSION,

464
packages/external-provider-contract/test/provider-package.test.mjs
View File

@ -1,464 +0,0 @@
import assert from "node:assert/strict";
import { readFile } from "node:fs/promises";
import {
L2_CONNECTION_INSTANCE_SCHEMA_VERSION,
instantiateL2Connection,
validateDataProductPublish,
validateProviderPackage,
} from "../src/index.mjs";
import {
GELIOS_POSITIONS_DATA_PRODUCT_ID,
GELIOS_POSITIONS_DATA_PRODUCT_VERSION,
GELIOS_POSITIONS_ONTOLOGY_REVISION,
GELIOS_UNIT_SOURCE_ID_PREFIX,
geliosProviderPackageV1,
geliosUnitsCurrentFixtureV1,
} from "../providers/gelios/v1/index.mjs";
import { normalizeDataProductDefinition } from "../../../services/external-data-plane/src/data-product-policy.mjs";
const expectedFields = [
"course_degrees",
"display_name",
"elevation_meters",
"geometry",
"hdop",
"horizontal_accuracy_meters",
"object_kind",
"operational_status",
"position_source",
"position_valid",
"quality_flags",
"satellite_count",
"speed_kph",
];
assert.deepEqual(validateProviderPackage(geliosProviderPackageV1), { ok: true, errors: [] });
const geliosAuth = geliosProviderPackageV1.authModes[0];
const geliosUnitsRead = geliosProviderPackageV1.capabilities[0];
assert.equal(geliosAuth.id, "gelios.rest-bearer.v1");
assert.deepEqual(geliosAuth.transport, { placement: "header", name: "Authorization" });
assert.deepEqual(geliosAuth.tokenLifecycle, {
artifacts: ["access", "refresh"],
requestArtifact: "access",
refreshMode: "operator_managed",
});
assert.equal(geliosAuth.tokenLifecycle.artifacts.includes("writer"), false);
assert.equal(geliosUnitsRead.request.baseUrl, "https://api.geliospro.com");
assert.equal(geliosUnitsRead.request.path, "/api/v1/units");
assert.deepEqual(geliosUnitsRead.request.query, {});
const product = geliosProviderPackageV1.dataProducts[0];
const registeredProduct = JSON.parse(await readFile(new URL(
"../../../services/external-data-plane/definitions/fleet.positions.current.v1.json",
import.meta.url,
), "utf8"));
assert.equal(product.id, GELIOS_POSITIONS_DATA_PRODUCT_ID);
assert.equal(product.version, GELIOS_POSITIONS_DATA_PRODUCT_VERSION);
assert.equal(product.ontologyRevision, GELIOS_POSITIONS_ONTOLOGY_REVISION);
assert.equal(product.id, "fleet.positions.current.v1");
assert.equal(product.version, "1.0.0");
assert.equal(product.ontologyRevision, "ontology.map.moving_object.v1");
assert.deepEqual(product.fields, expectedFields);
assert.deepEqual(product, registeredProduct);
assert.doesNotThrow(() => normalizeDataProductDefinition(product));
assert.equal(product.fields.every((field) => /^[a-z][a-z0-9_]*$/.test(field)), true);
const mapping = geliosProviderPackageV1.mappingContracts[0];
assert.equal(mapping.target.dataProductId, product.id);
assert.equal(mapping.target.version, product.version);
assert.equal(mapping.target.ontologyRevision, product.ontologyRevision);
assert.deepEqual(
[...Object.keys(mapping.fact.attributes), "geometry"].sort(),
[...product.fields].sort(),
);
assert.equal(mapping.derivations.operational_status.parameters.staleAfterMs, 60000);
assert.deepEqual(mapping.derivations.quality_flags.default, []);
assert.equal(mapping.fact.observedAt.fallback, "collection_received_at");
assert.equal(GELIOS_UNIT_SOURCE_ID_PREFIX, "gelios-unit-");
assert.equal(mapping.fact.sourceId.prefix, GELIOS_UNIT_SOURCE_ID_PREFIX);
assert.equal(mapping.fact.sourceId.coerce, "string");
assert.equal(mapping.fact.geometry.longitude.paths.includes("lastMsg.lon"), true);
assert.equal(mapping.fact.geometry.latitude.paths.includes("lastMsg.lat"), true);
assert.equal(mapping.fact.attributes.speed_kph.paths.includes("lastMsg.speed"), true);
assert.equal(validateDataProductPublish(geliosUnitsCurrentFixtureV1.expectedPublish).ok, true);
assert.equal(
geliosUnitsCurrentFixtureV1.sourceResponse.units.length,
geliosUnitsCurrentFixtureV1.expectedPublish.facts.length,
"every entity returned for the bound credential remains in the publish fixture",
);
assert.deepEqual(
geliosUnitsCurrentFixtureV1.expectedPublish.facts.map((fact) => fact.sourceId),
geliosUnitsCurrentFixtureV1.sourceResponse.units.map(
(unit) => `${GELIOS_UNIT_SOURCE_ID_PREFIX}${String(unit.id)}`,
),
);
assert.equal(
geliosUnitsCurrentFixtureV1.expectedPublish.facts[1].sourceId,
"gelios-unit-001002",
"provider ID strings must remain byte-for-byte stable after the canonical prefix",
);
assert.equal(geliosUnitsCurrentFixtureV1.expectedPublish.facts[1].geometry, undefined);
assert.equal(geliosUnitsCurrentFixtureV1.expectedPublish.facts[1].attributes.operational_status, "no_position");
assert.equal(geliosUnitsCurrentFixtureV1.sourceResponse.units[1].lastMsg, undefined);
assert.equal(
geliosUnitsCurrentFixtureV1.expectedPublish.facts[1].observedAt,
geliosUnitsCurrentFixtureV1.collectionContext.receivedAt,
);
const realtimeProfileId = "gelios.positions.current.realtime.v1";
assert.equal(
geliosProviderPackageV1.collectionProfiles[0].cardinality.onExceed,
"require_partitioned_data_product",
);
const accountA = instantiateL2Connection(geliosProviderPackageV1, {
tenantId: "tenant-alpha",
connectionId: "gelios-account-alpha",
collectionProfileId: realtimeProfileId,
providerCredentialRef: "ndc-credref:provider-alpha-0001",
});
const accountB = instantiateL2Connection(geliosProviderPackageV1, {
tenantId: "tenant-beta",
connectionId: "gelios-account-beta",
collectionProfileId: realtimeProfileId,
providerCredentialRef: "ndc-credref:provider-beta-0001",
});
assert.equal(accountA.schemaVersion, L2_CONNECTION_INSTANCE_SCHEMA_VERSION);
assert.equal(accountA.package.id, accountB.package.id);
assert.equal(accountA.package.version, accountB.package.version);
assert.equal(accountA.l2TemplateId, accountB.l2TemplateId);
assert.notEqual(accountA.connectionId, accountB.connectionId);
assert.notEqual(accountA.credentialRefs.provider.reference, accountB.credentialRefs.provider.reference);
assert.equal(accountA.credentialRefs.provider.owner, "ndc_l2_credentials");
assert.deepEqual(Object.keys(accountA.credentialRefs), ["provider"]);
assert.deepEqual(accountA.systemBindings.publisher, {
role: "publisher",
owner: "ndc_l2_credentials",
management: "control_plane_managed",
nodeType: "n8n-nodes-ndc.ndcDataProductPublish",
desiredState: "bound",
status: "unresolved",
});
assert.equal(accountA.scope.mode, "all_visible_to_credential");
assert.equal(accountA.scope.refresh, "each_collection_run");
assert.equal(Object.isFrozen(accountA), true);
assert.equal(Object.isFrozen(accountA.credentialRefs), true);
assert.equal(Object.isFrozen(accountA.systemBindings), true);
const withEntityAllowlist = structuredClone(geliosProviderPackageV1);
withEntityAllowlist.capabilities[0].entityScope.unitIds = ["gelios-unit-1001"];
assert.equal(
validateProviderPackage(withEntityAllowlist).errors.includes("capabilities[0].entityScope.unitIds_not_allowed"),
true,
);
const withGroupFilter = structuredClone(geliosProviderPackageV1);
withGroupFilter.collectionProfiles[0].entityScope.businessEntityFilter = "provider_group";
assert.equal(
validateProviderPackage(withGroupFilter).errors.includes("collectionProfiles[0].entityScope.businessEntityFilter_must_be_forbidden"),
true,
);
const withWrongRevision = structuredClone(geliosProviderPackageV1);
withWrongRevision.mappingContracts[0].target.ontologyRevision = "gelios.positions.v1";
assert.equal(
validateProviderPackage(withWrongRevision).errors.includes("mappingContract.gelios.units.to.fleet.positions.current.v1.target.ontologyRevision_must_match_data_product"),
true,
);
const withMissingDerivation = structuredClone(geliosProviderPackageV1);
delete withMissingDerivation.mappingContracts[0].derivations.position_valid;
assert.equal(
validateProviderPackage(withMissingDerivation).errors.includes("mappingContracts[0].derive_position_valid_missing_definition"),
true,
);
const withWrongFactSemanticType = structuredClone(geliosProviderPackageV1);
withWrongFactSemanticType.mappingContracts[0].fact.semanticType.constant = "map.other";
assert.equal(
validateProviderPackage(withWrongFactSemanticType).errors.includes(
"mappingContract.gelios.units.to.fleet.positions.current.v1.fact.semanticType_must_equal_target_constant",
),
true,
);
const nonGeospatialPackage = structuredClone(geliosProviderPackageV1);
nonGeospatialPackage.dataProducts[0].fields = ["display_name"];
nonGeospatialPackage.fieldPolicies[0].targetFields = ["display_name"];
delete nonGeospatialPackage.mappingContracts[0].fact.geometry;
nonGeospatialPackage.mappingContracts[0].fact.attributes = {
display_name: { strategy: "first_non_empty", paths: ["name"], coerce: "string" },
};
delete nonGeospatialPackage.mappingContracts[0].derivations;
assert.deepEqual(validateProviderPackage(nonGeospatialPackage), { ok: true, errors: [] });
for (const restrictedPath of ["phone", "phone.number", "lmsg"]) {
const withRestrictedMappingPath = structuredClone(geliosProviderPackageV1);
withRestrictedMappingPath.mappingContracts[0].fact.attributes.display_name.paths = [restrictedPath];
assert.equal(
validateProviderPackage(withRestrictedMappingPath).errors.includes(
"mappingContract.gelios.units.to.fleet.positions.current.v1.fact_source_path_restricted",
),
true,
);
}
const withFieldPolicyProductMismatch = structuredClone(geliosProviderPackageV1);
const secondProduct = structuredClone(withFieldPolicyProductMismatch.dataProducts[0]);
secondProduct.id = "fleet.positions.alternate.v1";
withFieldPolicyProductMismatch.dataProducts.push(secondProduct);
withFieldPolicyProductMismatch.manifest.dataProductIds.push(secondProduct.id);
withFieldPolicyProductMismatch.fieldPolicies[0].dataProductId = secondProduct.id;
assert.equal(
validateProviderPackage(withFieldPolicyProductMismatch).errors.includes(
"mappingContract.gelios.units.to.fleet.positions.current.v1.field_policy_data_product_mismatch",
),
true,
);
const withCamelCaseField = structuredClone(geliosProviderPackageV1);
withCamelCaseField.dataProducts[0].fields[12] = "speedKph";
assert.equal(validateProviderPackage(withCamelCaseField).ok, false);
const withSecretField = structuredClone(geliosProviderPackageV1);
withSecretField.dataProducts[0].fields[12] = "api_token";
withSecretField.fieldPolicies[0].targetFields[12] = "api_token";
withSecretField.mappingContracts[0].fact.attributes.api_token = { constant: "metadata-only" };
delete withSecretField.mappingContracts[0].fact.attributes.speed_kph;
const secretFieldValidation = validateProviderPackage(withSecretField);
assert.equal(secretFieldValidation.errors.includes("dataProducts[0].fields_must_not_contain_secret_fields"), true);
assert.equal(secretFieldValidation.errors.includes("fieldPolicies[0].targetFields_must_not_contain_secret_fields"), true);
assert.equal(secretFieldValidation.errors.includes("mappingContracts[0].fact.attributes.api_token_secret_field_not_allowed"), true);
const withRuntimeAccountState = structuredClone(geliosProviderPackageV1);
withRuntimeAccountState.tenantId = "must-not-live-in-package";
assert.equal(
validateProviderPackage(withRuntimeAccountState).errors.includes("providerPackage_must_not_contain_connection_instance_state"),
true,
);
const withPrimitiveCapability = structuredClone(geliosProviderPackageV1);
withPrimitiveCapability.capabilities.push(null);
assert.doesNotThrow(() => validateProviderPackage(withPrimitiveCapability));
assert.equal(validateProviderPackage(withPrimitiveCapability).ok, false);
const withMalformedNestedArrays = structuredClone(geliosProviderPackageV1);
withMalformedNestedArrays.collectionProfiles[0].capabilityIds = {};
withMalformedNestedArrays.l2Templates[0].credentialBindings = {};
withMalformedNestedArrays.l2Templates[0].steps = {};
assert.doesNotThrow(() => validateProviderPackage(withMalformedNestedArrays));
assert.equal(validateProviderPackage(withMalformedNestedArrays).ok, false);
const withCredentialInStaticQuery = structuredClone(geliosProviderPackageV1);
withCredentialInStaticQuery.authModes[0].transport = { placement: "query", name: "token" };
withCredentialInStaticQuery.capabilities[0].request.query.token = "plain-provider-key";
assert.equal(
validateProviderPackage(withCredentialInStaticQuery).errors.includes("capabilities[0].request.query.token_credential_parameter_not_allowed"),
true,
);
assert.equal(
validateProviderPackage(withCredentialInStaticQuery).errors.includes("capability.gelios.units.current.read.request.query_must_not_embed_credential_parameter"),
true,
);
const withCredentialInBaseUrl = structuredClone(geliosProviderPackageV1);
withCredentialInBaseUrl.capabilities[0].request.baseUrl = "https://user:pass@api.geliospro.com/?token=forbidden";
assert.equal(validateProviderPackage(withCredentialInBaseUrl).ok, false);
for (const unsafePath of ["/ok\nX-Evil: yes", "/back\\slash", "/query?x=1"]) {
const withUnsafeRequestPath = structuredClone(geliosProviderPackageV1);
withUnsafeRequestPath.capabilities[0].request.path = unsafePath;
assert.equal(validateProviderPackage(withUnsafeRequestPath).ok, false);
}
for (const unsafeCollectionPath of ["__proto__.polluted", "data.items[0]", "constructor.prototype"]) {
const withUnsafeCollectionPath = structuredClone(geliosProviderPackageV1);
withUnsafeCollectionPath.capabilities[0].request.response.collectionPaths = [unsafeCollectionPath];
assert.equal(validateProviderPackage(withUnsafeCollectionPath).ok, false);
}
const withUnsafeMappingPath = structuredClone(geliosProviderPackageV1);
withUnsafeMappingPath.mappingContracts[0].fact.attributes.display_name.paths = ["__proto__.polluted"];
assert.equal(validateProviderPackage(withUnsafeMappingPath).ok, false);
const withUnsafeCollectionCapability = structuredClone(geliosProviderPackageV1);
withUnsafeCollectionCapability.capabilities[0].classification = "write";
assert.equal(
validateProviderPackage(withUnsafeCollectionCapability).errors.includes("collectionProfile.gelios.positions.current.realtime.v1.capabilityIds_must_reference_implemented_safe_read"),
true,
);
const withUnexecutedSecondCapability = structuredClone(geliosProviderPackageV1);
const secondCapability = structuredClone(withUnexecutedSecondCapability.capabilities[0]);
secondCapability.id = "gelios.units.metadata.read";
withUnexecutedSecondCapability.capabilities.push(secondCapability);
withUnexecutedSecondCapability.manifest.capabilityIds.push(secondCapability.id);
for (const profile of withUnexecutedSecondCapability.collectionProfiles) {
profile.capabilityIds.push(secondCapability.id);
}
assert.equal(
validateProviderPackage(withUnexecutedSecondCapability).errors.includes(
"collectionProfile.gelios.positions.current.realtime.v1.capabilityIds_must_select_one_v1_read",
),
true,
);
const withReorderedSteps = structuredClone(geliosProviderPackageV1);
[withReorderedSteps.l2Templates[0].steps[0], withReorderedSteps.l2Templates[0].steps[1]] = [
withReorderedSteps.l2Templates[0].steps[1],
withReorderedSteps.l2Templates[0].steps[0],
];
assert.equal(
validateProviderPackage(withReorderedSteps).errors.includes("l2Templates[0].steps_sequence_invalid"),
true,
);
const withUnknownConnectionParameters = structuredClone(geliosProviderPackageV1);
withUnknownConnectionParameters.l2Templates[0].connectionParameters = ["foo"];
assert.equal(
validateProviderPackage(withUnknownConnectionParameters).errors.includes(
"l2Templates[0].connectionParameters_must_match_v1_contract",
),
true,
);
const withCredentialAuthMismatch = structuredClone(geliosProviderPackageV1);
const secondAuthMode = structuredClone(withCredentialAuthMismatch.authModes[0]);
secondAuthMode.id = "gelios.rest-bearer.alternate.v1";
withCredentialAuthMismatch.authModes.push(secondAuthMode);
withCredentialAuthMismatch.manifest.authModeIds.push(secondAuthMode.id);
withCredentialAuthMismatch.l2Templates[0].credentialBindings[0].authModeId = secondAuthMode.id;
assert.equal(
validateProviderPackage(withCredentialAuthMismatch).errors.includes(
"l2Template.gelios.positions.current.l2.v1.provider_credential_auth_mode_mismatch",
),
true,
);
const withCallerManagedPublisher = structuredClone(geliosProviderPackageV1);
withCallerManagedPublisher.l2Templates[0].credentialBindings[1].management = "connection_input";
assert.equal(
validateProviderPackage(withCallerManagedPublisher).errors.includes(
"l2Templates[0].credentialBindings[1].publisher_binding_shape_invalid",
),
true,
);
const withUnknownTokenArtifact = structuredClone(geliosProviderPackageV1);
withUnknownTokenArtifact.authModes[0].tokenLifecycle.artifacts.push("writer");
assert.equal(
validateProviderPackage(withUnknownTokenArtifact).errors.includes(
"authModes[0].tokenLifecycle.artifacts_invalid",
),
true,
);
const withRefreshUsedForRequest = structuredClone(geliosProviderPackageV1);
withRefreshUsedForRequest.authModes[0].tokenLifecycle.requestArtifact = "refresh";
assert.equal(
validateProviderPackage(withRefreshUsedForRequest).errors.includes(
"authModes[0].tokenLifecycle.requestArtifact_must_be_access",
),
true,
);
const withUnmanagedRefreshArtifact = structuredClone(geliosProviderPackageV1);
withUnmanagedRefreshArtifact.authModes[0].tokenLifecycle.refreshMode = "not_applicable";
assert.equal(
validateProviderPackage(withUnmanagedRefreshArtifact).errors.includes(
"authModes[0].tokenLifecycle.refreshMode_required_for_refresh_artifact",
),
true,
);
for (const history of [{
mode: "none",
intervalMs: 60000,
strategy: "latest-per-entity-per-bucket",
retentionDays: 90,
}, {
mode: "sampled",
intervalMs: 24 * 60 * 60 * 1000 + 1,
strategy: "latest-per-entity-per-bucket",
retentionDays: 90,
}, {
mode: "all",
retentionDays: 3651,
}]) {
const withInvalidHistory = structuredClone(geliosProviderPackageV1);
withInvalidHistory.dataProducts[0].history = history;
assert.equal(validateProviderPackage(withInvalidHistory).ok, false);
assert.throws(() => normalizeDataProductDefinition(withInvalidHistory.dataProducts[0]));
}
const withBrokenArtifactChain = structuredClone(geliosProviderPackageV1);
withBrokenArtifactChain.l2Templates[0].steps[4].dataProductId = "unrelated.product.v1";
assert.equal(
validateProviderPackage(withBrokenArtifactChain).errors.includes("collectionProfile.gelios.positions.current.realtime.v1.template_data_product_mismatch"),
true,
);
const withSecretNamedDerivationParameter = structuredClone(geliosProviderPackageV1);
withSecretNamedDerivationParameter.mappingContracts[0].derivations.position_valid.parameters = {
accessToken: "opaque-provider-value",
};
assert.equal(
validateProviderPackage(withSecretNamedDerivationParameter).errors.includes(
"mappingContracts[0].derivations.position_valid.parameters.accessToken_secret_field_not_allowed",
),
true,
);
const withObjectConstant = structuredClone(geliosProviderPackageV1);
withObjectConstant.mappingContracts[0].fact.attributes.object_kind.constant = {
accessToken: "opaque-provider-value",
};
assert.equal(
validateProviderPackage(withObjectConstant).errors.includes(
"mappingContracts[0].fact.attributes.object_kind.constant_must_be_scalar_or_scalar_array",
),
true,
);
assert.throws(() => instantiateL2Connection(geliosProviderPackageV1, {
tenantId: "tenant-alpha",
connectionId: "gelios-account-alpha",
collectionProfileId: realtimeProfileId,
providerCredentialRef: "Bearer plaintext-must-not-cross-boundary",
}), /l2_connection_invalid/);
assert.throws(() => instantiateL2Connection(geliosProviderPackageV1, {
tenantId: "tenant-alpha",
connectionId: "gelios-account-alpha",
collectionProfileId: realtimeProfileId,
providerCredentialRef: "ndc-credref:provider-alpha-0001",
writerCredentialRef: "ndc-credref:caller-writer-forbidden",
}), /options\.writerCredentialRef_not_allowed/);
const serializedPackage = JSON.stringify(geliosProviderPackageV1);
assert.equal(serializedPackage.includes('"prefix":"unit-"'), false);
assert.equal(serializedPackage.includes("unitIds"), false);
assert.equal(serializedPackage.includes("tenant-alpha"), false);
assert.equal(serializedPackage.includes("gelios-gateway"), false);
assert.equal(serializedPackage.includes("writer_credential_ref"), false);
assert.equal(serializedPackage.includes("writerCapabilityByReferenceOnly"), false);
assert.equal(serializedPackage.includes("publisherBindingControlPlaneManaged\":true"), true);
assert.equal(serializedPackage.includes("providerSpecificServiceForbidden\":true"), true);
assert.deepEqual(
geliosProviderPackageV1.l2Templates[0].steps.map((step) => step.kind),
[
"collection_trigger",
"provider_request",
"extract_items",
"semantic_mapping",
"data_product_publish",
],
);
assert.equal(
geliosProviderPackageV1.l2Templates[0].steps.at(-1).nodeType,
"n8n-nodes-ndc.ndcDataProductPublish",
);
console.log("external-provider-contract provider-package: ok");

11
services/external-data-plane/Dockerfile
View File

@ -2,12 +2,7 @@ FROM node:20-alpine AS deps
WORKDIR /workspace WORKDIR /workspace
COPY packages/external-provider-contract/package.json ./packages/external-provider-contract/package.json COPY packages/external-provider-contract ./packages/external-provider-contract
COPY packages/external-provider-contract/src/contract-version.mjs ./packages/external-provider-contract/src/contract-version.mjs
COPY packages/external-provider-contract/src/data-plane.mjs ./packages/external-provider-contract/src/data-plane.mjs
COPY packages/external-provider-contract/src/data-product.mjs ./packages/external-provider-contract/src/data-product.mjs
COPY packages/external-provider-contract/src/intake-batch.mjs ./packages/external-provider-contract/src/intake-batch.mjs
COPY packages/external-provider-contract/src/sensitive-field-policy.mjs ./packages/external-provider-contract/src/sensitive-field-policy.mjs
COPY services/external-data-plane/package.json services/external-data-plane/package-lock.json ./services/external-data-plane/ COPY services/external-data-plane/package.json services/external-data-plane/package-lock.json ./services/external-data-plane/
WORKDIR /workspace/services/external-data-plane WORKDIR /workspace/services/external-data-plane
@ -26,8 +21,8 @@ COPY --from=deps /workspace/packages/external-provider-contract /packages/extern
COPY services/external-data-plane/src ./src COPY services/external-data-plane/src ./src
COPY services/external-data-plane/definitions ./definitions COPY services/external-data-plane/definitions ./definitions
# A dedicated numeric identity owns the legacy EDP bearer boundary and reads the # A dedicated numeric identity can read only the EDP provisioning secret mount;
# separate public Engine trust mount; it is not the shared service uid/gid. # it is intentionally not the shared Node/Map Gateway uid/gid (1000).
RUN addgroup -S -g 11006 nodedc-edp && adduser -S -D -H -u 11006 -G nodedc-edp nodedc-edp RUN addgroup -S -g 11006 nodedc-edp && adduser -S -D -H -u 11006 -G nodedc-edp nodedc-edp
USER 11006:11006 USER 11006:11006

7
services/external-data-plane/package.json
View File

@ -6,10 +6,9 @@
"scripts": { "scripts": {
"start": "node src/server.mjs", "start": "node src/server.mjs",
"dev": "node --watch src/server.mjs", "dev": "node --watch src/server.mjs",
"check": "node --check src/server.mjs && node --check src/schema.mjs && node --check src/config.mjs && node --check src/intake-policy.mjs && node --check src/writer-binding.mjs && node --check src/reader-binding.mjs && node --check src/data-product-policy.mjs && node --check src/data-product-delivery.mjs && node --check src/definitions.mjs && node --check src/managed-provisioner-auth.mjs", "check": "node --check src/server.mjs && node --check src/schema.mjs && node --check src/config.mjs && node --check src/intake-policy.mjs && node --check src/writer-binding.mjs && node --check src/reader-binding.mjs && node --check src/data-product-policy.mjs && node --check src/data-product-delivery.mjs && node --check src/definitions.mjs",
"test": "node test/config.test.mjs && node test/managed-provisioner-auth.test.mjs && node test/intake-policy.test.mjs && node test/writer-binding.test.mjs && node test/reader-binding.test.mjs && node test/data-product-policy.test.mjs && node test/definitions.test.mjs", "test": "node test/config.test.mjs && node test/intake-policy.test.mjs && node test/writer-binding.test.mjs && node test/reader-binding.test.mjs && node test/data-product-policy.test.mjs && node test/definitions.test.mjs",
"test:integration": "node test/data-product-delivery.integration.test.mjs && node test/api.integration.test.mjs", "test:integration": "node test/data-product-delivery.integration.test.mjs && node test/api.integration.test.mjs"
"test:all": "npm test && npm run test:integration"
}, },
"dependencies": { "dependencies": {
"@nodedc/external-provider-contract": "file:../../packages/external-provider-contract", "@nodedc/external-provider-contract": "file:../../packages/external-provider-contract",

40
services/external-data-plane/src/config.mjs
View File

@ -1,54 +1,14 @@
import { readFileSync } from "node:fs"; import { readFileSync } from "node:fs";
import {
loadManagedProvisionerPublicKeyFile,
validateManagedProvisionerAudience,
validateManagedProvisionerIdentity,
} from "./managed-provisioner-auth.mjs";
export function readConfig(env = process.env) { export function readConfig(env = process.env) {
const provisionerApiEnabled = boolean(env.EXTERNAL_DATA_PLANE_PROVISIONING_ENABLED, false); const provisionerApiEnabled = boolean(env.EXTERNAL_DATA_PLANE_PROVISIONING_ENABLED, false);
const managedProvisionerApiEnabled = boolean(env.EXTERNAL_DATA_PLANE_MANAGED_PROVISIONING_ENABLED, false);
const managedProvisionerMaxSkewSeconds = integer(
env.EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_MAX_SKEW_SECONDS,
60,
5,
300,
);
const config = { const config = {
port: integer(env.PORT, 18106, 1, 65535), port: integer(env.PORT, 18106, 1, 65535),
databaseUrl: required(env.EXTERNAL_DATA_PLANE_DATABASE_URL, "EXTERNAL_DATA_PLANE_DATABASE_URL"), databaseUrl: required(env.EXTERNAL_DATA_PLANE_DATABASE_URL, "EXTERNAL_DATA_PLANE_DATABASE_URL"),
databasePoolSize: integer(env.EXTERNAL_DATA_PLANE_DATABASE_POOL_SIZE, 10, 1, 50), databasePoolSize: integer(env.EXTERNAL_DATA_PLANE_DATABASE_POOL_SIZE, 10, 1, 50),
internalAccessToken: optional(env.NODEDC_INTERNAL_ACCESS_TOKEN), internalAccessToken: optional(env.NODEDC_INTERNAL_ACCESS_TOKEN),
provisionerApiEnabled, provisionerApiEnabled,
managedProvisionerApiEnabled,
provisionerAccessToken: provisionerApiEnabled ? secretFile(env.EXTERNAL_DATA_PLANE_PROVISIONER_TOKEN_FILE) : "", provisionerAccessToken: provisionerApiEnabled ? secretFile(env.EXTERNAL_DATA_PLANE_PROVISIONER_TOKEN_FILE) : "",
managedProvisionerPublicKey: managedProvisionerApiEnabled
? loadManagedProvisionerPublicKeyFile(env.EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_PUBLIC_KEY_FILE)
: null,
managedProvisionerServiceId: managedProvisionerApiEnabled
? validateManagedProvisionerIdentity(
required(env.EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_SERVICE_ID, "EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_SERVICE_ID"),
"service_id",
)
: "",
managedProvisionerKeyId: managedProvisionerApiEnabled
? validateManagedProvisionerIdentity(
required(env.EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_KEY_ID, "EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_KEY_ID"),
"key_id",
)
: "",
managedProvisionerAudience: managedProvisionerApiEnabled
? validateManagedProvisionerAudience(
required(env.EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_AUDIENCE, "EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_AUDIENCE"),
)
: "",
managedProvisionerMaxSkewMs: managedProvisionerMaxSkewSeconds * 1000,
managedProvisionerReplayCacheMaxEntries: integer(
env.EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_REPLAY_CACHE_MAX_ENTRIES,
10_000,
100,
100_000,
),
rawRetentionDays: integer(env.EXTERNAL_DATA_PLANE_RAW_RETENTION_DAYS, 14, 1, 3650), rawRetentionDays: integer(env.EXTERNAL_DATA_PLANE_RAW_RETENTION_DAYS, 14, 1, 3650),
maxBatchBytes: integer(env.EXTERNAL_DATA_PLANE_MAX_BATCH_BYTES, 5 * 1024 * 1024, 1024, 50 * 1024 * 1024), maxBatchBytes: integer(env.EXTERNAL_DATA_PLANE_MAX_BATCH_BYTES, 5 * 1024 * 1024, 1024, 50 * 1024 * 1024),
maxFactsPerPublish: integer(env.EXTERNAL_DATA_PLANE_MAX_FACTS_PER_PUBLISH, 5000, 1, 100_000), maxFactsPerPublish: integer(env.EXTERNAL_DATA_PLANE_MAX_FACTS_PER_PUBLISH, 5000, 1, 100_000),

2
services/external-data-plane/src/data-product-delivery.mjs
View File

@ -2,7 +2,7 @@ import { createHash, randomUUID } from "node:crypto";
import { import {
DATA_PRODUCT_PATCH_SCHEMA_VERSION, DATA_PRODUCT_PATCH_SCHEMA_VERSION,
DATA_PRODUCT_SNAPSHOT_SCHEMA_VERSION, DATA_PRODUCT_SNAPSHOT_SCHEMA_VERSION,
} from "@nodedc/external-provider-contract/data-plane"; } from "@nodedc/external-provider-contract";
export async function loadDataProductDefinition(db, dataProductId, { activeOnly = true } = {}) { export async function loadDataProductDefinition(db, dataProductId, { activeOnly = true } = {}) {
const result = await db.query( const result = await db.query(

318
services/external-data-plane/src/managed-provisioner-auth.mjs
View File

@ -1,318 +0,0 @@
import { createHash, createPublicKey, timingSafeEqual, verify as verifySignature } from "node:crypto";
import { closeSync, constants as fsConstants, fstatSync, lstatSync, openSync, readFileSync } from "node:fs";
import { isAbsolute } from "node:path";
export const MANAGED_PROVISIONER_SIGNATURE_SCHEMA = "nodedc.external-data-plane.managed-provisioner-request/v1";
export const MANAGED_PROVISIONER_HEADERS = Object.freeze({
serviceId: "x-nodedc-engine-service-id",
keyId: "x-nodedc-engine-key-id",
audience: "x-nodedc-request-audience",
timestamp: "x-nodedc-request-timestamp",
nonce: "x-nodedc-request-nonce",
bodySha256: "x-nodedc-content-sha256",
signature: "x-nodedc-request-signature",
});
const ISO_TIMESTAMP = /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z$/;
const BASE64URL = /^[A-Za-z0-9_-]+$/;
const IDENTITY = /^[a-z][a-z0-9._:-]{2,127}$/i;
const AUDIENCE = /^[a-z][a-z0-9._:/-]{2,255}$/i;
export function loadManagedProvisionerPublicKeyFile(pathValue) {
const path = String(pathValue ?? "").trim();
if (!path) throw configError("external_data_plane_managed_provisioner_public_key_file_required");
if (!isAbsolute(path)) throw configError("external_data_plane_managed_provisioner_public_key_file_must_be_absolute");
let stat;
try {
stat = lstatSync(path);
} catch {
throw configError("external_data_plane_managed_provisioner_public_key_file_unreadable");
}
if (!stat.isFile() || stat.isSymbolicLink()) {
throw configError("external_data_plane_managed_provisioner_public_key_file_must_be_regular");
}
if ((stat.mode & 0o022) !== 0) {
throw configError("external_data_plane_managed_provisioner_public_key_file_permissions_invalid");
}
if (stat.size < 1 || stat.size > 8192) {
throw configError("external_data_plane_managed_provisioner_public_key_file_size_invalid");
}
let descriptor;
try {
descriptor = openSync(path, fsConstants.O_RDONLY | fsConstants.O_NOFOLLOW);
} catch {
throw configError("external_data_plane_managed_provisioner_public_key_file_unreadable");
}
let pem;
try {
const openedStat = fstatSync(descriptor);
if (!openedStat.isFile()) {
throw configError("external_data_plane_managed_provisioner_public_key_file_must_be_regular");
}
if ((openedStat.mode & 0o022) !== 0) {
throw configError("external_data_plane_managed_provisioner_public_key_file_permissions_invalid");
}
if (openedStat.size < 1 || openedStat.size > 8192) {
throw configError("external_data_plane_managed_provisioner_public_key_file_size_invalid");
}
pem = readFileSync(descriptor, "utf8");
} catch (error) {
if (String(error?.code || "").startsWith("external_data_plane_managed_provisioner_")) throw error;
throw configError("external_data_plane_managed_provisioner_public_key_file_unreadable");
} finally {
closeSync(descriptor);
}
const normalizedPem = pem.replace(/\r\n/g, "\n").trim();
if (!/^-----BEGIN PUBLIC KEY-----\n[A-Za-z0-9+/=\n]+\n-----END PUBLIC KEY-----$/.test(normalizedPem)) {
throw configError("external_data_plane_managed_provisioner_public_key_file_format_invalid");
}
let publicKey;
try {
publicKey = createPublicKey(normalizedPem);
} catch {
throw configError("external_data_plane_managed_provisioner_public_key_file_format_invalid");
}
if (publicKey.type !== "public" || publicKey.asymmetricKeyType !== "ed25519") {
throw configError("external_data_plane_managed_provisioner_public_key_algorithm_invalid");
}
return publicKey;
}
export function validateManagedProvisionerIdentity(value, name) {
const normalized = String(value ?? "").trim();
if (!IDENTITY.test(normalized)) throw configError(`external_data_plane_managed_provisioner_${name}_invalid`);
return normalized;
}
export function validateManagedProvisionerAudience(value) {
const normalized = String(value ?? "").trim();
if (!AUDIENCE.test(normalized)) throw configError("external_data_plane_managed_provisioner_audience_invalid");
return normalized;
}
export function managedProvisionerSigningPayload({
audience,
serviceId,
keyId,
method,
path,
timestamp,
nonce,
bodySha256,
}) {
return JSON.stringify({
schemaVersion: MANAGED_PROVISIONER_SIGNATURE_SCHEMA,
audience,
serviceId,
keyId,
method,
path,
timestamp,
nonce,
bodySha256,
});
}
export function sha256RawBody(rawBody) {
if (!Buffer.isBuffer(rawBody)) throw new TypeError("raw_body_buffer_required");
return createHash("sha256").update(rawBody).digest("hex");
}
export class ManagedProvisionerReplayCache {
constructor({ maxEntries }) {
if (!Number.isInteger(maxEntries) || maxEntries < 1) throw new TypeError("replay_cache_max_entries_invalid");
this.maxEntries = maxEntries;
this.entries = new Map();
}
consume(key, expiresAt, now = Date.now()) {
this.prune(now);
const existing = this.entries.get(key);
if (existing !== undefined && existing > now) {
throw authError(409, "managed_provisioner_request_replayed");
}
if (this.entries.size >= this.maxEntries) {
throw authError(503, "managed_provisioner_replay_cache_exhausted");
}
this.entries.set(key, expiresAt);
}
prune(now = Date.now()) {
for (const [key, expiresAt] of this.entries) {
if (expiresAt <= now) this.entries.delete(key);
}
}
}
export function createManagedProvisionerRequestVerifier({
publicKey,
serviceId,
keyId,
audience,
maxSkewMs,
replayCache,
now = Date.now,
}) {
if (publicKey?.type !== "public" || publicKey?.asymmetricKeyType !== "ed25519") {
throw configError("external_data_plane_managed_provisioner_public_key_algorithm_invalid");
}
if (!IDENTITY.test(serviceId)) throw configError("external_data_plane_managed_provisioner_service_id_invalid");
if (!IDENTITY.test(keyId)) throw configError("external_data_plane_managed_provisioner_key_id_invalid");
if (!AUDIENCE.test(audience)) throw configError("external_data_plane_managed_provisioner_audience_invalid");
if (!Number.isInteger(maxSkewMs) || maxSkewMs < 1) {
throw configError("external_data_plane_managed_provisioner_max_skew_invalid");
}
if (!(replayCache instanceof ManagedProvisionerReplayCache)) {
throw configError("external_data_plane_managed_provisioner_replay_cache_invalid");
}
return function verifyManagedProvisionerRequest(req) {
const requestServiceId = exactHeader(req, MANAGED_PROVISIONER_HEADERS.serviceId);
const requestKeyId = exactHeader(req, MANAGED_PROVISIONER_HEADERS.keyId);
const requestAudience = exactHeader(req, MANAGED_PROVISIONER_HEADERS.audience);
const timestamp = exactHeader(req, MANAGED_PROVISIONER_HEADERS.timestamp);
const nonce = exactHeader(req, MANAGED_PROVISIONER_HEADERS.nonce);
const declaredBodySha256 = exactHeader(req, MANAGED_PROVISIONER_HEADERS.bodySha256);
const encodedSignature = exactHeader(req, MANAGED_PROVISIONER_HEADERS.signature);
if (hasHeader(req, "authorization")) {
throw authError(401, "managed_provisioner_authorization_header_forbidden");
}
if (!safeEqualText(requestServiceId, serviceId)
|| !safeEqualText(requestKeyId, keyId)
|| !safeEqualText(requestAudience, audience)) {
throw authError(401, "managed_provisioner_identity_mismatch");
}
const timestampMs = parseTimestamp(timestamp);
const verificationTime = Number(now());
if (!Number.isFinite(verificationTime) || Math.abs(verificationTime - timestampMs) >= maxSkewMs) {
throw authError(401, "managed_provisioner_timestamp_outside_window");
}
const nonceBytes = decodeCanonicalBase64Url(nonce, null, "managed_provisioner_nonce_invalid");
if (nonceBytes.length < 16 || nonceBytes.length > 96) throw authError(401, "managed_provisioner_nonce_invalid");
if (!/^[a-f0-9]{64}$/.test(declaredBodySha256)) {
throw authError(401, "managed_provisioner_body_hash_invalid");
}
const method = exactMethod(req);
const path = exactPath(req);
const rawBody = exactRawBody(req);
const actualBodySha256 = sha256RawBody(rawBody);
if (!safeEqualText(declaredBodySha256, actualBodySha256)) {
throw authError(401, "managed_provisioner_body_hash_mismatch");
}
const signature = decodeCanonicalBase64Url(encodedSignature, 64, "managed_provisioner_signature_invalid");
const payload = managedProvisionerSigningPayload({
audience: requestAudience,
serviceId: requestServiceId,
keyId: requestKeyId,
method,
path,
timestamp,
nonce,
bodySha256: declaredBodySha256,
});
if (!verifySignature(null, Buffer.from(payload, "utf8"), publicKey, signature)) {
throw authError(401, "managed_provisioner_signature_invalid");
}
const replayKey = `${requestServiceId}\0${requestKeyId}\0${nonce}`;
replayCache.consume(replayKey, timestampMs + maxSkewMs, verificationTime);
return Object.freeze({ serviceId: requestServiceId, keyId: requestKeyId });
};
}
function exactHeader(req, name) {
const rawHeaders = req?.rawHeaders;
if (Array.isArray(rawHeaders)) {
const values = [];
for (let index = 0; index < rawHeaders.length; index += 2) {
if (String(rawHeaders[index]).toLowerCase() === name) values.push(rawHeaders[index + 1]);
}
if (values.length !== 1 || typeof values[0] !== "string" || !values[0]) {
throw authError(401, "managed_provisioner_header_invalid");
}
return values[0];
}
const value = req?.headers?.[name];
if (typeof value !== "string" || !value) throw authError(401, "managed_provisioner_header_invalid");
return value;
}
function hasHeader(req, name) {
if (Array.isArray(req?.rawHeaders)) {
for (let index = 0; index < req.rawHeaders.length; index += 2) {
if (String(req.rawHeaders[index]).toLowerCase() === name) return true;
}
return false;
}
return Object.hasOwn(req?.headers || {}, name);
}
function exactMethod(req) {
const method = String(req?.method ?? "");
if (!/^[A-Z]{3,10}$/.test(method)) throw authError(401, "managed_provisioner_method_invalid");
return method;
}
function exactPath(req) {
const path = String(req?.originalUrl ?? "");
if (!path.startsWith("/") || path.length > 2048 || /[\u0000-\u0020#]/.test(path)) {
throw authError(401, "managed_provisioner_path_invalid");
}
return path;
}
function exactRawBody(req) {
if (req?.rawBodyCaptured === true && Buffer.isBuffer(req.rawBody)) return req.rawBody;
const contentLength = req?.headers?.["content-length"];
const transferEncoding = req?.headers?.["transfer-encoding"];
if ((contentLength === undefined || contentLength === "0") && transferEncoding === undefined) {
return Buffer.alloc(0);
}
throw authError(401, "managed_provisioner_raw_body_unavailable");
}
function parseTimestamp(value) {
if (!ISO_TIMESTAMP.test(value)) throw authError(401, "managed_provisioner_timestamp_invalid");
const timestampMs = Date.parse(value);
if (!Number.isFinite(timestampMs) || new Date(timestampMs).toISOString() !== value) {
throw authError(401, "managed_provisioner_timestamp_invalid");
}
return timestampMs;
}
function decodeCanonicalBase64Url(value, expectedBytes, errorCode) {
if (!BASE64URL.test(value)) throw authError(401, errorCode);
let decoded;
try {
decoded = Buffer.from(value, "base64url");
} catch {
throw authError(401, errorCode);
}
if ((expectedBytes !== null && decoded.length !== expectedBytes) || decoded.toString("base64url") !== value) {
throw authError(401, errorCode);
}
return decoded;
}
function safeEqualText(left, right) {
const leftBuffer = Buffer.from(String(left), "utf8");
const rightBuffer = Buffer.from(String(right), "utf8");
return leftBuffer.length === rightBuffer.length && timingSafeEqual(leftBuffer, rightBuffer);
}
function authError(status, code) {
return Object.assign(new Error(code), { status, code });
}
function configError(code) {
return Object.assign(new Error(code), { code });
}

35
services/external-data-plane/src/schema.mjs
View File

@ -65,9 +65,6 @@ export async function migrate(pool) {
create table if not exists external_data_plane_writer_bindings ( create table if not exists external_data_plane_writer_bindings (
id uuid primary key, id uuid primary key,
token_hash text not null unique, token_hash text not null unique,
binding_key text,
request_hash text,
generation integer not null default 1,
tenant_id text not null, tenant_id text not null,
connection_id text not null, connection_id text not null,
provider_id text not null, provider_id text not null,
@ -77,7 +74,6 @@ export async function migrate(pool) {
created_at timestamptz not null default now(), created_at timestamptz not null default now(),
rotated_at timestamptz, rotated_at timestamptz,
revoked_at timestamptz, revoked_at timestamptz,
check (generation > 0),
check ( check (
case when jsonb_typeof(allowed_data_product_ids) = 'array' case when jsonb_typeof(allowed_data_product_ids) = 'array'
then jsonb_array_length(allowed_data_product_ids) > 0 then jsonb_array_length(allowed_data_product_ids) > 0
@ -86,37 +82,6 @@ export async function migrate(pool) {
) )
) )
`); `);
await pool.query("alter table external_data_plane_writer_bindings add column if not exists binding_key text");
await pool.query("alter table external_data_plane_writer_bindings add column if not exists request_hash text");
await pool.query("alter table external_data_plane_writer_bindings add column if not exists generation integer not null default 1");
await pool.query(`
do $$
begin
if not exists (
select 1 from pg_constraint
where conrelid = 'external_data_plane_writer_bindings'::regclass
and conname = 'external_data_plane_writer_bindings_generation_positive_ck'
) then
alter table external_data_plane_writer_bindings
add constraint external_data_plane_writer_bindings_generation_positive_ck
check (generation > 0);
end if;
if not exists (
select 1 from pg_constraint
where conrelid = 'external_data_plane_writer_bindings'::regclass
and conname = 'external_data_plane_writer_bindings_managed_metadata_ck'
) then
alter table external_data_plane_writer_bindings
add constraint external_data_plane_writer_bindings_managed_metadata_ck
check (
(binding_key is null and request_hash is null)
or (binding_key is not null and request_hash ~ '^[a-f0-9]{64}$')
);
end if;
end
$$
`);
await pool.query("create unique index if not exists external_data_plane_writer_bindings_managed_key_idx on external_data_plane_writer_bindings (binding_key, generation) where binding_key is not null");
await pool.query("create index if not exists external_data_plane_writer_bindings_active_idx on external_data_plane_writer_bindings (active, expires_at)"); await pool.query("create index if not exists external_data_plane_writer_bindings_active_idx on external_data_plane_writer_bindings (active, expires_at)");
await pool.query(` await pool.query(`

188
services/external-data-plane/src/server.mjs
View File

@ -2,7 +2,7 @@ import express from "express";
import { createHash, randomUUID, timingSafeEqual } from "node:crypto"; import { createHash, randomUUID, timingSafeEqual } from "node:crypto";
import { createServer } from "node:http"; import { createServer } from "node:http";
import { Pool } from "pg"; import { Pool } from "pg";
import { validateDataProductPublish, validateIntakeBatch } from "@nodedc/external-provider-contract/data-plane"; import { validateDataProductPublish, validateIntakeBatch } from "@nodedc/external-provider-contract";
import { readConfig } from "./config.mjs"; import { readConfig } from "./config.mjs";
import { import {
loadDataProductDefinition, loadDataProductDefinition,
@ -18,10 +18,6 @@ import {
import { normalizeDataProductDefinition, safeDataProductDefinition } from "./data-product-policy.mjs"; import { normalizeDataProductDefinition, safeDataProductDefinition } from "./data-product-policy.mjs";
import { reconcileDataProductDefinitions } from "./definitions.mjs"; import { reconcileDataProductDefinitions } from "./definitions.mjs";
import { assertBatchTimeBounds, rawRetentionExpiry } from "./intake-policy.mjs"; import { assertBatchTimeBounds, rawRetentionExpiry } from "./intake-policy.mjs";
import {
createManagedProvisionerRequestVerifier,
ManagedProvisionerReplayCache,
} from "./managed-provisioner-auth.mjs";
import { import {
assertReaderProduct, assertReaderProduct,
createReaderToken, createReaderToken,
@ -35,26 +31,11 @@ import {
hashWriterToken, hashWriterToken,
materializeDataProductPublish, materializeDataProductPublish,
materializeWriterBoundBatch, materializeWriterBoundBatch,
normalizeManagedWriterBindingRequest,
normalizeWriterBindingRequest, normalizeWriterBindingRequest,
safeWriterBinding, safeWriterBinding,
writerBindingRequestHash,
} from "./writer-binding.mjs"; } from "./writer-binding.mjs";
const config = readConfig(); const config = readConfig();
const managedProvisionerReplayCache = config.managedProvisionerApiEnabled
? new ManagedProvisionerReplayCache({ maxEntries: config.managedProvisionerReplayCacheMaxEntries })
: null;
const verifyManagedProvisionerRequest = config.managedProvisionerApiEnabled
? createManagedProvisionerRequestVerifier({
publicKey: config.managedProvisionerPublicKey,
serviceId: config.managedProvisionerServiceId,
keyId: config.managedProvisionerKeyId,
audience: config.managedProvisionerAudience,
maxSkewMs: config.managedProvisionerMaxSkewMs,
replayCache: managedProvisionerReplayCache,
})
: null;
const pool = new Pool({ connectionString: config.databaseUrl, max: config.databasePoolSize }); const pool = new Pool({ connectionString: config.databaseUrl, max: config.databasePoolSize });
const app = express(); const app = express();
const httpServer = createServer(app); const httpServer = createServer(app);
@ -66,19 +47,7 @@ let shuttingDown = false;
let shutdownPromise = null; let shutdownPromise = null;
app.disable("x-powered-by"); app.disable("x-powered-by");
app.use((req, _res, next) => { app.use(express.json({ limit: config.maxBatchBytes }));
req.rawBody = Buffer.alloc(0);
req.rawBodyCaptured = false;
next();
});
app.use(express.json({
limit: config.maxBatchBytes,
inflate: false,
verify(req, _res, buffer) {
req.rawBody = Buffer.from(buffer);
req.rawBodyCaptured = true;
},
}));
app.get("/healthz", asyncRoute(async (_req, res) => { app.get("/healthz", asyncRoute(async (_req, res) => {
await pool.query("select 1"); await pool.query("select 1");
@ -90,12 +59,10 @@ app.get("/healthz", asyncRoute(async (_req, res) => {
providerLogic: "absent", providerLogic: "absent",
commandTransport: "absent", commandTransport: "absent",
writerBindings: "supported", writerBindings: "supported",
managedWriterBindings: "digest+idempotent-generation",
readerBindings: "supported", readerBindings: "supported",
dataProductDelivery: "snapshot+durable-patch", dataProductDelivery: "snapshot+durable-patch",
legacyIntake: config.legacyIntakeEnabled ? "migration-only" : "disabled", legacyIntake: config.legacyIntakeEnabled ? "migration-only" : "disabled",
legacyCapabilityProvisioning: config.provisionerApiEnabled ? "enabled" : "disabled", writerBindingProvisioning: config.provisionerApiEnabled ? "enabled" : "disabled",
managedWriterBindingProvisioning: config.managedProvisionerApiEnabled ? "enabled" : "disabled",
rawRetentionSweep: { rawRetentionSweep: {
mode: "server-scheduled", mode: "server-scheduled",
lastSweepAt: lastRetentionSweepAt, lastSweepAt: lastRetentionSweepAt,
@ -167,128 +134,6 @@ app.post("/internal/data-plane/v1/intake/writer-bound", requireLegacyIntake, req
res.status(result.idempotent ? 200 : 201).json({ ok: true, ...result }); res.status(result.idempotent ? 200 : 201).json({ ok: true, ...result });
})); }));
app.put("/internal/data-plane/v1/writer-bindings/by-key/:bindingKey", requireManagedProvisionerApi, asyncRoute(async (req, res) => {
const bindingKey = requireIdentifier(req.params.bindingKey, "managed_writer_binding_key_invalid");
const policy = normalizeManagedWriterBindingRequest(req.body, {
maxTtlDays: config.writerBindingMaxTtlDays,
});
await assertRegisteredProductIds(policy.allowedDataProductIds);
const requestHash = writerBindingRequestHash(policy);
const client = await pool.connect();
let response;
try {
await client.query("begin");
await client.query("select pg_advisory_xact_lock(hashtextextended($1, 0))", [bindingKey]);
const existing = await client.query(
`select id, binding_key as "bindingKey", request_hash as "requestHash", generation,
tenant_id as "tenantId", connection_id as "connectionId", provider_id as "providerId",
allowed_data_product_ids as "allowedDataProductIds", expires_at as "expiresAt",
active, created_at as "createdAt", rotated_at as "rotatedAt", revoked_at as "revokedAt"
from external_data_plane_writer_bindings
where binding_key = $1 and generation = $2
for update`,
[bindingKey, policy.generation],
);
if (existing.rowCount) {
const binding = existing.rows[0];
if (binding.requestHash !== requestHash) {
throw httpError(409, "managed_writer_binding_request_conflict");
}
if (binding.active !== true || new Date(binding.expiresAt) <= new Date()) {
throw httpError(409, "managed_writer_binding_generation_inactive");
}
response = { status: 200, idempotent: true, binding };
} else {
const inserted = await client.query(
`insert into external_data_plane_writer_bindings (
id, token_hash, binding_key, request_hash, generation,
tenant_id, connection_id, provider_id, allowed_data_product_ids, expires_at
) values ($1, $2, $3, $4, $5, $6, $7, $8, $9::jsonb, $10)
returning id, binding_key as "bindingKey", generation,
tenant_id as "tenantId", connection_id as "connectionId", provider_id as "providerId",
allowed_data_product_ids as "allowedDataProductIds", expires_at as "expiresAt",
active, created_at as "createdAt", rotated_at as "rotatedAt", revoked_at as "revokedAt"`,
[
randomUUID(),
policy.capabilityDigest,
bindingKey,
requestHash,
policy.generation,
policy.tenantId,
policy.connectionId,
policy.providerId,
JSON.stringify(policy.allowedDataProductIds),
policy.expiresAt,
],
);
response = { status: 201, idempotent: false, binding: inserted.rows[0] };
}
await client.query("commit");
} catch (error) {
await client.query("rollback");
if (error?.code === "23505") {
throw httpError(409, "managed_writer_binding_capability_digest_conflict");
}
throw error;
} finally {
client.release();
}
res.set("Cache-Control", "no-store, max-age=0");
res.status(response.status).json({
ok: true,
idempotent: response.idempotent,
writerBinding: safeWriterBinding(response.binding),
});
}));
app.post("/internal/data-plane/v1/writer-bindings/by-key/:bindingKey/generations/:generation/revoke", requireManagedProvisionerApi, asyncRoute(async (req, res) => {
const bindingKey = requireIdentifier(req.params.bindingKey, "managed_writer_binding_key_invalid");
const generation = requirePositiveInteger(req.params.generation, "managed_writer_binding_generation_invalid");
const client = await pool.connect();
let binding;
let idempotent;
try {
await client.query("begin");
await client.query("select pg_advisory_xact_lock(hashtextextended($1, 0))", [bindingKey]);
const existing = await client.query(
`select id, binding_key as "bindingKey", generation,
tenant_id as "tenantId", connection_id as "connectionId", provider_id as "providerId",
allowed_data_product_ids as "allowedDataProductIds", expires_at as "expiresAt",
active, created_at as "createdAt", rotated_at as "rotatedAt", revoked_at as "revokedAt"
from external_data_plane_writer_bindings
where binding_key = $1 and generation = $2
for update`,
[bindingKey, generation],
);
if (!existing.rowCount) throw httpError(404, "managed_writer_binding_not_found");
if (existing.rows[0].active === true) {
const revoked = await client.query(
`update external_data_plane_writer_bindings
set active = false, revoked_at = now()
where binding_key = $1 and generation = $2 and active = true
returning id, binding_key as "bindingKey", generation,
tenant_id as "tenantId", connection_id as "connectionId", provider_id as "providerId",
allowed_data_product_ids as "allowedDataProductIds", expires_at as "expiresAt",
active, created_at as "createdAt", rotated_at as "rotatedAt", revoked_at as "revokedAt"`,
[bindingKey, generation],
);
binding = revoked.rows[0];
idempotent = false;
} else {
binding = existing.rows[0];
idempotent = true;
}
await client.query("commit");
} catch (error) {
await client.query("rollback");
throw error;
} finally {
client.release();
}
res.set("Cache-Control", "no-store, max-age=0");
res.json({ ok: true, idempotent, writerBinding: safeWriterBinding(binding) });
}));
app.post("/internal/data-plane/v1/writer-bindings", requireProvisionerApi, asyncRoute(async (req, res) => { app.post("/internal/data-plane/v1/writer-bindings", requireProvisionerApi, asyncRoute(async (req, res) => {
const policy = normalizeWriterBindingRequest(req.body, { const policy = normalizeWriterBindingRequest(req.body, {
maxTtlDays: config.writerBindingMaxTtlDays, maxTtlDays: config.writerBindingMaxTtlDays,
@ -327,7 +172,7 @@ app.post("/internal/data-plane/v1/writer-bindings/:bindingId/rotate", requirePro
const result = await pool.query( const result = await pool.query(
`update external_data_plane_writer_bindings `update external_data_plane_writer_bindings
set token_hash = $2, rotated_at = now() set token_hash = $2, rotated_at = now()
where id = $1 and binding_key is null and active = true and expires_at > now() where id = $1 and active = true and expires_at > now()
returning id, tenant_id as "tenantId", connection_id as "connectionId", returning id, tenant_id as "tenantId", connection_id as "connectionId",
provider_id as "providerId", allowed_data_product_ids as "allowedDataProductIds", provider_id as "providerId", allowed_data_product_ids as "allowedDataProductIds",
expires_at as "expiresAt", active, created_at as "createdAt", expires_at as "expiresAt", active, created_at as "createdAt",
@ -343,7 +188,7 @@ app.post("/internal/data-plane/v1/writer-bindings/:bindingId/revoke", requirePro
const result = await pool.query( const result = await pool.query(
`update external_data_plane_writer_bindings `update external_data_plane_writer_bindings
set active = false, revoked_at = now() set active = false, revoked_at = now()
where id = $1 and binding_key is null and active = true where id = $1 and active = true
returning id, tenant_id as "tenantId", connection_id as "connectionId", returning id, tenant_id as "tenantId", connection_id as "connectionId",
provider_id as "providerId", allowed_data_product_ids as "allowedDataProductIds", provider_id as "providerId", allowed_data_product_ids as "allowedDataProductIds",
expires_at as "expiresAt", active, created_at as "createdAt", expires_at as "expiresAt", active, created_at as "createdAt",
@ -719,21 +564,6 @@ function requireLegacyIntake(_req, _res, next) {
function requireProvisionerApi(req, _res, next) { function requireProvisionerApi(req, _res, next) {
if (!config.provisionerApiEnabled) return next(httpError(503, "provisioner_api_disabled")); if (!config.provisionerApiEnabled) return next(httpError(503, "provisioner_api_disabled"));
return requireProvisionerCredential(req, next);
}
function requireManagedProvisionerApi(req, _res, next) {
if (!config.managedProvisionerApiEnabled) return next(httpError(503, "managed_provisioner_api_disabled"));
if (!verifyManagedProvisionerRequest) return next(httpError(503, "managed_provisioner_api_not_configured"));
try {
req.managedProvisionerIdentity = verifyManagedProvisionerRequest(req);
return next();
} catch (error) {
return next(error);
}
}
function requireProvisionerCredential(req, next) {
if (!config.provisionerAccessToken) return next(httpError(503, "provisioner_api_not_configured")); if (!config.provisionerAccessToken) return next(httpError(503, "provisioner_api_not_configured"));
const value = bearerToken(req); const value = bearerToken(req);
if (!value || !safeEqual(value, config.provisionerAccessToken)) return next(httpError(401, "provisioner_unauthorized")); if (!value || !safeEqual(value, config.provisionerAccessToken)) return next(httpError(401, "provisioner_unauthorized"));
@ -867,14 +697,6 @@ function requireIdentifier(value, code) {
return normalized; return normalized;
} }
function requirePositiveInteger(value, code) {
const normalized = String(value ?? "");
if (!/^[1-9]\d*$/.test(normalized)) throw httpError(400, code);
const number = Number(normalized);
if (!Number.isSafeInteger(number) || number > 2_147_483_647) throw httpError(400, code);
return number;
}
function parseCursor(value) { function parseCursor(value) {
const normalized = String(value ?? ""); const normalized = String(value ?? "");
if (!/^(?:0|[1-9]\d*)$/.test(normalized)) throw httpError(400, "data_product_cursor_invalid"); if (!/^(?:0|[1-9]\d*)$/.test(normalized)) throw httpError(400, "data_product_cursor_invalid");

61
services/external-data-plane/src/writer-binding.mjs
View File

@ -4,15 +4,7 @@ const IDENTIFIER = /^[a-z][a-z0-9._:-]{2,127}$/;
const TOKEN_PREFIX = "ndc_edpwb_"; const TOKEN_PREFIX = "ndc_edpwb_";
const SECRET_LIKE_KEY = /(token|secret|password|access[_-]?token|refresh[_-]?token|api[_-]?key)/i; const SECRET_LIKE_KEY = /(token|secret|password|access[_-]?token|refresh[_-]?token|api[_-]?key)/i;
const BINDING_REQUEST_KEYS = new Set(["source", "allowedDataProductIds", "expiresAt"]); const BINDING_REQUEST_KEYS = new Set(["source", "allowedDataProductIds", "expiresAt"]);
const MANAGED_BINDING_REQUEST_KEYS = new Set([
"source",
"allowedDataProductIds",
"expiresAt",
"generation",
"capabilityDigest",
]);
const BINDING_SOURCE_KEYS = new Set(["tenantId", "connectionId", "providerId"]); const BINDING_SOURCE_KEYS = new Set(["tenantId", "connectionId", "providerId"]);
const SHA256_DIGEST = /^[a-f0-9]{64}$/;
/** /**
* Produces an opaque, high-entropy capability. The plaintext is returned only * Produces an opaque, high-entropy capability. The plaintext is returned only
@ -67,57 +59,6 @@ export function normalizeWriterBindingRequest(value, { now = new Date(), maxTtlD
}); });
} }
/**
* Validates the zero-touch control-plane form. The native credential secret is
* generated and stored inside Engine; EDP receives only its SHA-256 digest.
* `generation` makes a retry address the same immutable binding generation.
*/
export function normalizeManagedWriterBindingRequest(value, { now = new Date(), maxTtlDays = 90 } = {}) {
if (!isPlainObject(value) || !isPlainObject(value.source)) {
throw writerBindingError("managed_writer_binding_request_invalid");
}
if (containsSecretLikeKey(value)) {
throw writerBindingError("managed_writer_binding_secret_material_forbidden");
}
if (!hasOnlyKeys(value, MANAGED_BINDING_REQUEST_KEYS) || !hasOnlyKeys(value.source, BINDING_SOURCE_KEYS)) {
throw writerBindingError("managed_writer_binding_request_fields_invalid");
}
const scope = normalizeWriterBindingRequest({
source: value.source,
allowedDataProductIds: value.allowedDataProductIds,
expiresAt: value.expiresAt,
}, { now, maxTtlDays });
const generation = Number(value.generation);
if (!Number.isSafeInteger(generation) || generation < 1 || generation > 2_147_483_647) {
throw writerBindingError("managed_writer_binding_generation_invalid");
}
const capabilityDigest = String(value.capabilityDigest || "").toLowerCase();
if (!SHA256_DIGEST.test(capabilityDigest)) {
throw writerBindingError("managed_writer_binding_capability_digest_invalid");
}
return Object.freeze({
...scope,
allowedDataProductIds: Object.freeze([...scope.allowedDataProductIds].sort()),
generation,
capabilityDigest,
});
}
export function writerBindingRequestHash(policy) {
const canonical = JSON.stringify({
tenantId: policy.tenantId,
connectionId: policy.connectionId,
providerId: policy.providerId,
allowedDataProductIds: [...policy.allowedDataProductIds].sort(),
expiresAt: new Date(policy.expiresAt).toISOString(),
generation: policy.generation,
capabilityDigest: policy.capabilityDigest,
});
return createHash("sha256").update(canonical, "utf8").digest("hex");
}
/** /**
* Converts a caller-provided, deliberately unscoped intake envelope into the * Converts a caller-provided, deliberately unscoped intake envelope into the
* canonical scoped form. Caller scope is rejected, never trusted or merged. * canonical scoped form. Caller scope is rejected, never trusted or merged.
@ -195,8 +136,6 @@ export function materializeDataProductPublish(value, binding, definition, dataPr
export function safeWriterBinding(binding) { export function safeWriterBinding(binding) {
return { return {
id: binding.id, id: binding.id,
...(binding.bindingKey ? { bindingKey: binding.bindingKey } : {}),
...(Number.isInteger(Number(binding.generation)) ? { generation: Number(binding.generation) } : {}),
tenantId: binding.tenantId, tenantId: binding.tenantId,
connectionId: binding.connectionId, connectionId: binding.connectionId,
providerId: binding.providerId, providerId: binding.providerId,

204
services/external-data-plane/test/api.integration.test.mjs
View File

@ -1,16 +1,10 @@
import assert from "node:assert/strict"; import assert from "node:assert/strict";
import { spawn } from "node:child_process"; import { spawn } from "node:child_process";
import { createHash, generateKeyPairSync, randomBytes, sign } from "node:crypto";
import { mkdtemp, rm, writeFile } from "node:fs/promises"; import { mkdtemp, rm, writeFile } from "node:fs/promises";
import net from "node:net"; import net from "node:net";
import { tmpdir } from "node:os"; import { tmpdir } from "node:os";
import { join } from "node:path"; import { join } from "node:path";
import { validateDataProductPatch, validateDataProductSnapshot } from "@nodedc/external-provider-contract/data-plane"; import { validateDataProductPatch, validateDataProductSnapshot } from "@nodedc/external-provider-contract";
import {
MANAGED_PROVISIONER_HEADERS,
managedProvisionerSigningPayload,
sha256RawBody,
} from "../src/managed-provisioner-auth.mjs";
const databaseUrl = process.env.EXTERNAL_DATA_PLANE_TEST_DATABASE_URL; const databaseUrl = process.env.EXTERNAL_DATA_PLANE_TEST_DATABASE_URL;
if (!databaseUrl) throw new Error("EXTERNAL_DATA_PLANE_TEST_DATABASE_URL_required"); if (!databaseUrl) throw new Error("EXTERNAL_DATA_PLANE_TEST_DATABASE_URL_required");
@ -19,17 +13,7 @@ const baseUrl = `http://127.0.0.1:${port}`;
const directory = await mkdtemp(join(tmpdir(), "nodedc-edp-api-test-")); const directory = await mkdtemp(join(tmpdir(), "nodedc-edp-api-test-"));
const provisionerSecret = "nodedc_edp_integration_provisioner_secret_0123456789abcdef"; const provisionerSecret = "nodedc_edp_integration_provisioner_secret_0123456789abcdef";
const secretPath = join(directory, "provisioner-token"); const secretPath = join(directory, "provisioner-token");
const managedProvisionerServiceId = "nodedc-engine";
const managedProvisionerKeyId = "engine-edp-managed-provisioner-v1";
const managedProvisionerAudience = "nodedc-external-data-plane.managed-provisioning.v1";
const managedProvisionerPublicKeyPath = join(directory, "engine-public-key.pem");
const { privateKey: managedProvisionerPrivateKey, publicKey: managedProvisionerPublicKey } = generateKeyPairSync("ed25519");
await writeFile(secretPath, `${provisionerSecret}\n`, { mode: 0o600 }); await writeFile(secretPath, `${provisionerSecret}\n`, { mode: 0o600 });
await writeFile(
managedProvisionerPublicKeyPath,
managedProvisionerPublicKey.export({ type: "spki", format: "pem" }),
{ mode: 0o600 },
);
const child = spawn(process.execPath, ["src/server.mjs"], { const child = spawn(process.execPath, ["src/server.mjs"], {
cwd: new URL("..", import.meta.url), cwd: new URL("..", import.meta.url),
@ -38,12 +22,7 @@ const child = spawn(process.execPath, ["src/server.mjs"], {
PORT: String(port), PORT: String(port),
EXTERNAL_DATA_PLANE_DATABASE_URL: databaseUrl, EXTERNAL_DATA_PLANE_DATABASE_URL: databaseUrl,
EXTERNAL_DATA_PLANE_PROVISIONING_ENABLED: "true", EXTERNAL_DATA_PLANE_PROVISIONING_ENABLED: "true",
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONING_ENABLED: "true",
EXTERNAL_DATA_PLANE_PROVISIONER_TOKEN_FILE: secretPath, EXTERNAL_DATA_PLANE_PROVISIONER_TOKEN_FILE: secretPath,
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_PUBLIC_KEY_FILE: managedProvisionerPublicKeyPath,
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_SERVICE_ID: managedProvisionerServiceId,
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_KEY_ID: managedProvisionerKeyId,
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_AUDIENCE: managedProvisionerAudience,
EXTERNAL_DATA_PLANE_RETENTION_SWEEP_MS: "60000", EXTERNAL_DATA_PLANE_RETENTION_SWEEP_MS: "60000",
EXTERNAL_DATA_PLANE_STREAM_POLL_MS: "250", EXTERNAL_DATA_PLANE_STREAM_POLL_MS: "250",
EXTERNAL_DATA_PLANE_STREAM_HEARTBEAT_MS: "5000", EXTERNAL_DATA_PLANE_STREAM_HEARTBEAT_MS: "5000",
@ -73,139 +52,10 @@ try {
const expiry = new Date(Date.now() + 24 * 60 * 60 * 1000).toISOString(); const expiry = new Date(Date.now() + 24 * 60 * 60 * 1000).toISOString();
const scope = { tenantId: "api-tenant", connectionId: "api-connection", providerId: "api-provider" }; const scope = { tenantId: "api-tenant", connectionId: "api-connection", providerId: "api-provider" };
const managedToken = "ndc_edpwb_managed_api_test_0123456789abcdefghijklmnopqrstuvwxyz";
const managedBody = {
source: scope,
allowedDataProductIds: [productId],
expiresAt: expiry,
generation: 1,
capabilityDigest: createHash("sha256").update(managedToken, "utf8").digest("hex"),
};
const managedPath = "/internal/data-plane/v1/writer-bindings/by-key/engine.api-connection.positions";
const bearerOnlyManaged = await fetch(`${baseUrl}${managedPath}`, {
method: "PUT",
headers: { Authorization: `Bearer ${provisionerSecret}`, "Content-Type": "application/json" },
body: JSON.stringify(managedBody),
});
assert.equal(bearerOnlyManaged.status, 401);
assert.equal((await bearerOnlyManaged.json()).error, "managed_provisioner_header_invalid");
const managedResults = await Promise.all([
rawJsonRequest(managedPath, {
method: "PUT",
managedSignature: true,
body: managedBody,
}),
rawJsonRequest(managedPath, {
method: "PUT",
managedSignature: true,
body: managedBody,
}),
]);
assert.deepEqual(managedResults.map(({ value }) => value.idempotent).sort(), [false, true]);
assert.equal(managedResults[0].value.writerBinding.id, managedResults[1].value.writerBinding.id);
assert.equal("token" in managedResults[0].value, false);
assert.equal(JSON.stringify(managedResults[0].value).includes(managedBody.capabilityDigest), false);
const managedCatalog = await jsonRequest("/internal/data-plane/v1/writer/data-products", { token: managedToken });
assert.deepEqual(managedCatalog.dataProducts.map((value) => value.id), [productId]);
const replayBody = JSON.stringify(managedBody);
const replayHeaders = signedManagedHeaders(managedPath, "PUT", Buffer.from(replayBody, "utf8"));
const firstReplayAttempt = await fetch(`${baseUrl}${managedPath}`, {
method: "PUT", headers: { ...replayHeaders, "Content-Type": "application/json" }, body: replayBody,
});
assert.equal(firstReplayAttempt.status, 200);
const rejectedReplay = await fetch(`${baseUrl}${managedPath}`, {
method: "PUT", headers: { ...replayHeaders, "Content-Type": "application/json" }, body: replayBody,
});
assert.equal(rejectedReplay.status, 409);
assert.equal((await rejectedReplay.json()).error, "managed_provisioner_request_replayed");
const tamperHeaders = signedManagedHeaders(managedPath, "PUT", Buffer.from(replayBody, "utf8"));
const tamperedBody = await fetch(`${baseUrl}${managedPath}`, {
method: "PUT",
headers: { ...tamperHeaders, "Content-Type": "application/json" },
body: `${replayBody} `,
});
assert.equal(tamperedBody.status, 401);
assert.equal((await tamperedBody.json()).error, "managed_provisioner_body_hash_mismatch");
const pathHeaders = signedManagedHeaders(managedPath, "PUT", Buffer.from(replayBody, "utf8"));
const tamperedPath = await fetch(`${baseUrl}${managedPath}?unexpected=true`, {
method: "PUT", headers: { ...pathHeaders, "Content-Type": "application/json" }, body: replayBody,
});
assert.equal(tamperedPath.status, 401);
assert.equal((await tamperedPath.json()).error, "managed_provisioner_signature_invalid");
const audienceHeaders = signedManagedHeaders(managedPath, "PUT", Buffer.from(replayBody, "utf8"));
audienceHeaders[MANAGED_PROVISIONER_HEADERS.audience] = "other-audience";
const wrongAudience = await fetch(`${baseUrl}${managedPath}`, {
method: "PUT", headers: { ...audienceHeaders, "Content-Type": "application/json" }, body: replayBody,
});
assert.equal(wrongAudience.status, 401);
assert.equal((await wrongAudience.json()).error, "managed_provisioner_identity_mismatch");
const managedConflict = await signedFetch(managedPath, {
method: "PUT",
body: {
...managedBody,
capabilityDigest: createHash("sha256").update(`${managedToken}-different`, "utf8").digest("hex"),
},
});
assert.equal(managedConflict.status, 409);
assert.equal((await managedConflict.json()).error, "managed_writer_binding_request_conflict");
const legacyManagedRevoke = await fetch(
`${baseUrl}/internal/data-plane/v1/writer-bindings/${managedResults[0].value.writerBinding.id}/revoke`,
{ method: "POST", headers: { Authorization: `Bearer ${provisionerSecret}` } },
);
assert.equal(legacyManagedRevoke.status, 404);
const managedTokenGeneration2 = `${managedToken}_generation_2`;
const managedGeneration2 = await jsonRequest(
"/internal/data-plane/v1/writer-bindings/by-key/engine.api-connection.positions",
{
method: "PUT",
managedSignature: true,
body: {
...managedBody,
generation: 2,
capabilityDigest: createHash("sha256").update(managedTokenGeneration2, "utf8").digest("hex"),
},
},
);
assert.equal(managedGeneration2.writerBinding.generation, 2);
assert.deepEqual(
(await jsonRequest("/internal/data-plane/v1/writer/data-products", { token: managedTokenGeneration2 })).dataProducts.map((value) => value.id),
[productId],
);
const managedRevoke = await jsonRequest(
"/internal/data-plane/v1/writer-bindings/by-key/engine.api-connection.positions/generations/1/revoke",
{ method: "POST", managedSignature: true },
);
assert.equal(managedRevoke.idempotent, false);
assert.equal(managedRevoke.writerBinding.active, false);
const managedRevokeRetry = await jsonRequest(
"/internal/data-plane/v1/writer-bindings/by-key/engine.api-connection.positions/generations/1/revoke",
{ method: "POST", managedSignature: true },
);
assert.equal(managedRevokeRetry.idempotent, true);
const rejectedManagedGeneration1 = await fetch(`${baseUrl}/internal/data-plane/v1/writer/data-products`, {
headers: { Authorization: `Bearer ${managedToken}` },
});
assert.equal(rejectedManagedGeneration1.status, 401);
const manualWriterBody = { source: scope, allowedDataProductIds: [productId], expiresAt: expiry };
const signedOnlyManualProvisioning = await signedFetch("/internal/data-plane/v1/writer-bindings", {
method: "POST",
body: manualWriterBody,
});
assert.equal(signedOnlyManualProvisioning.status, 401);
assert.equal((await signedOnlyManualProvisioning.json()).error, "provisioner_unauthorized");
const writerIssuance = await rawJsonRequest("/internal/data-plane/v1/writer-bindings", { const writerIssuance = await rawJsonRequest("/internal/data-plane/v1/writer-bindings", {
method: "POST", method: "POST",
token: provisionerSecret, token: provisionerSecret,
body: manualWriterBody, body: { source: scope, allowedDataProductIds: [productId], expiresAt: expiry },
}); });
const readerIssuance = await rawJsonRequest("/internal/data-plane/v1/reader-bindings", { const readerIssuance = await rawJsonRequest("/internal/data-plane/v1/reader-bindings", {
method: "POST", method: "POST",
@ -326,66 +176,26 @@ function publish(runId, observedAt, longitude) {
}; };
} }
async function jsonRequest(path, { method = "GET", token, managedSignature = false, body } = {}) { async function jsonRequest(path, { method = "GET", token, body } = {}) {
const { response, value } = await rawJsonRequest(path, { method, token, managedSignature, body }); const { response, value } = await rawJsonRequest(path, { method, token, body });
if (!response.ok) throw new Error(`request_failed:${response.status}:${value.error}`); if (!response.ok) throw new Error(`request_failed:${response.status}:${value.error}`);
return value; return value;
} }
async function rawJsonRequest(path, { method = "GET", token, managedSignature = false, body } = {}) { async function rawJsonRequest(path, { method = "GET", token, body } = {}) {
const rawBody = body === undefined ? Buffer.alloc(0) : Buffer.from(JSON.stringify(body), "utf8");
const response = await fetch(`${baseUrl}${path}`, { const response = await fetch(`${baseUrl}${path}`, {
method, method,
headers: { headers: {
...(token ? { Authorization: `Bearer ${token}` } : {}), ...(token ? { Authorization: `Bearer ${token}` } : {}),
...(managedSignature ? signedManagedHeaders(path, method, rawBody) : {}), ...(body ? { "Content-Type": "application/json" } : {}),
...(body === undefined ? {} : { "Content-Type": "application/json" }),
}, },
body: body === undefined ? undefined : rawBody, body: body ? JSON.stringify(body) : undefined,
}); });
const value = await response.json(); const value = await response.json();
if (!response.ok) throw new Error(`request_failed:${response.status}:${value.error}`); if (!response.ok) throw new Error(`request_failed:${response.status}:${value.error}`);
return { response, value }; return { response, value };
} }
async function signedFetch(path, { method, body } = {}) {
const rawBody = body === undefined ? Buffer.alloc(0) : Buffer.from(JSON.stringify(body), "utf8");
return fetch(`${baseUrl}${path}`, {
method,
headers: {
...signedManagedHeaders(path, method, rawBody),
...(body === undefined ? {} : { "Content-Type": "application/json" }),
},
body: body === undefined ? undefined : rawBody,
});
}
function signedManagedHeaders(path, method, rawBody) {
const timestamp = new Date().toISOString();
const nonce = randomBytes(24).toString("base64url");
const bodySha256 = sha256RawBody(rawBody);
const payload = managedProvisionerSigningPayload({
audience: managedProvisionerAudience,
serviceId: managedProvisionerServiceId,
keyId: managedProvisionerKeyId,
method,
path,
timestamp,
nonce,
bodySha256,
});
const signature = sign(null, Buffer.from(payload, "utf8"), managedProvisionerPrivateKey).toString("base64url");
return {
[MANAGED_PROVISIONER_HEADERS.serviceId]: managedProvisionerServiceId,
[MANAGED_PROVISIONER_HEADERS.keyId]: managedProvisionerKeyId,
[MANAGED_PROVISIONER_HEADERS.audience]: managedProvisionerAudience,
[MANAGED_PROVISIONER_HEADERS.timestamp]: timestamp,
[MANAGED_PROVISIONER_HEADERS.nonce]: nonce,
[MANAGED_PROVISIONER_HEADERS.bodySha256]: bodySha256,
[MANAGED_PROVISIONER_HEADERS.signature]: signature,
};
}
function assertOneTimeCapabilityResponse(response) { function assertOneTimeCapabilityResponse(response) {
assert.equal(response.headers.get("cache-control"), "no-store, max-age=0"); assert.equal(response.headers.get("cache-control"), "no-store, max-age=0");
assert.equal(response.headers.get("pragma"), "no-cache"); assert.equal(response.headers.get("pragma"), "no-cache");

45
services/external-data-plane/test/config.test.mjs
View File

@ -1,5 +1,4 @@
import assert from "node:assert/strict"; import assert from "node:assert/strict";
import { generateKeyPairSync } from "node:crypto";
import { mkdtemp, rm, writeFile } from "node:fs/promises"; import { mkdtemp, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os"; import { tmpdir } from "node:os";
import { join } from "node:path"; import { join } from "node:path";
@ -12,12 +11,9 @@ const base = {
const directory = await mkdtemp(join(tmpdir(), "nodedc-edp-config-")); const directory = await mkdtemp(join(tmpdir(), "nodedc-edp-config-"));
const secretPath = join(directory, "provisioner-token"); const secretPath = join(directory, "provisioner-token");
const publicKeyPath = join(directory, "engine-public-key.pem");
const secret = "provisioner_secret_is_separate_from_legacy_internal_token_123456"; const secret = "provisioner_secret_is_separate_from_legacy_internal_token_123456";
try { try {
await writeFile(secretPath, `${secret}\n`, { encoding: "utf8", mode: 0o600 }); await writeFile(secretPath, `${secret}\n`, { encoding: "utf8", mode: 0o600 });
const { publicKey } = generateKeyPairSync("ed25519");
await writeFile(publicKeyPath, publicKey.export({ type: "spki", format: "pem" }), { mode: 0o600 });
const config = readConfig({ const config = readConfig({
...base, ...base,
EXTERNAL_DATA_PLANE_PROVISIONING_ENABLED: "true", EXTERNAL_DATA_PLANE_PROVISIONING_ENABLED: "true",
@ -26,8 +22,6 @@ try {
EXTERNAL_DATA_PLANE_RETENTION_SWEEP_MS: "60000", EXTERNAL_DATA_PLANE_RETENTION_SWEEP_MS: "60000",
}); });
assert.equal(config.provisionerAccessToken, secret); assert.equal(config.provisionerAccessToken, secret);
assert.equal(config.provisionerApiEnabled, true);
assert.equal(config.managedProvisionerApiEnabled, false);
assert.equal(config.maxFutureSkewSeconds, 120); assert.equal(config.maxFutureSkewSeconds, 120);
assert.equal(config.retentionSweepMs, 60000); assert.equal(config.retentionSweepMs, 60000);
assert.equal(config.maxPatchBytes, 262144); assert.equal(config.maxPatchBytes, 262144);
@ -45,49 +39,10 @@ try {
EXTERNAL_DATA_PLANE_PROVISIONING_ENABLED: "true", EXTERNAL_DATA_PLANE_PROVISIONING_ENABLED: "true",
EXTERNAL_DATA_PLANE_PROVISIONER_TOKEN_FILE: join(directory, "missing"), EXTERNAL_DATA_PLANE_PROVISIONER_TOKEN_FILE: join(directory, "missing"),
}), /external_data_plane_provisioner_token_file_unreadable/); }), /external_data_plane_provisioner_token_file_unreadable/);
const managedConfig = readConfig({
...base,
EXTERNAL_DATA_PLANE_PROVISIONING_ENABLED: "false",
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONING_ENABLED: "true",
EXTERNAL_DATA_PLANE_PROVISIONER_TOKEN_FILE: join(directory, "missing-token-is-ignored"),
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_PUBLIC_KEY_FILE: publicKeyPath,
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_SERVICE_ID: "nodedc-engine",
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_KEY_ID: "engine-edp-managed-provisioner-v1",
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_AUDIENCE: "nodedc-external-data-plane.managed-provisioning.v1",
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_MAX_SKEW_SECONDS: "45",
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_REPLAY_CACHE_MAX_ENTRIES: "500",
});
assert.equal(managedConfig.provisionerApiEnabled, false);
assert.equal(managedConfig.managedProvisionerApiEnabled, true);
assert.equal(managedConfig.provisionerAccessToken, "");
assert.equal(managedConfig.managedProvisionerPublicKey.asymmetricKeyType, "ed25519");
assert.equal(managedConfig.managedProvisionerServiceId, "nodedc-engine");
assert.equal(managedConfig.managedProvisionerKeyId, "engine-edp-managed-provisioner-v1");
assert.equal(managedConfig.managedProvisionerAudience, "nodedc-external-data-plane.managed-provisioning.v1");
assert.equal(managedConfig.managedProvisionerMaxSkewMs, 45_000);
assert.equal(managedConfig.managedProvisionerReplayCacheMaxEntries, 500);
assert.throws(() => readConfig({
...base,
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONING_ENABLED: "true",
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_PUBLIC_KEY_FILE: join(directory, "missing"),
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_SERVICE_ID: "nodedc-engine",
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_KEY_ID: "engine-edp-managed-provisioner-v1",
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_AUDIENCE: "nodedc-external-data-plane.managed-provisioning.v1",
}), /external_data_plane_managed_provisioner_public_key_file_unreadable/);
assert.throws(() => readConfig({
...base,
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONING_ENABLED: "true",
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_PUBLIC_KEY_FILE: publicKeyPath,
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_SERVICE_ID: "invalid service id",
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_KEY_ID: "engine-edp-managed-provisioner-v1",
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_AUDIENCE: "nodedc-external-data-plane.managed-provisioning.v1",
}), /external_data_plane_managed_provisioner_service_id_invalid/);
assert.equal(readConfig({ assert.equal(readConfig({
...base, ...base,
EXTERNAL_DATA_PLANE_PROVISIONING_ENABLED: "false", EXTERNAL_DATA_PLANE_PROVISIONING_ENABLED: "false",
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONING_ENABLED: "false",
EXTERNAL_DATA_PLANE_PROVISIONER_TOKEN_FILE: join(directory, "missing"), EXTERNAL_DATA_PLANE_PROVISIONER_TOKEN_FILE: join(directory, "missing"),
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_PUBLIC_KEY_FILE: join(directory, "missing"),
}).provisionerAccessToken, ""); }).provisionerAccessToken, "");
} finally { } finally {
await rm(directory, { recursive: true, force: true }); await rm(directory, { recursive: true, force: true });

2
services/external-data-plane/test/data-product-delivery.integration.test.mjs
View File

@ -1,6 +1,6 @@
import assert from "node:assert/strict"; import assert from "node:assert/strict";
import { Pool } from "pg"; import { Pool } from "pg";
import { validateDataProductPatch, validateDataProductSnapshot } from "@nodedc/external-provider-contract/data-plane"; import { validateDataProductPatch, validateDataProductSnapshot } from "@nodedc/external-provider-contract";
import { import {
loadDataProductDefinition, loadDataProductDefinition,
persistDataProductDefinition, persistDataProductDefinition,

217
services/external-data-plane/test/managed-provisioner-auth.test.mjs
View File

@ -1,217 +0,0 @@
import assert from "node:assert/strict";
import {
generateKeyPairSync,
randomBytes,
sign,
} from "node:crypto";
import { chmod, mkdtemp, rm, symlink, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import {
createManagedProvisionerRequestVerifier,
loadManagedProvisionerPublicKeyFile,
MANAGED_PROVISIONER_HEADERS,
managedProvisionerSigningPayload,
ManagedProvisionerReplayCache,
sha256RawBody,
} from "../src/managed-provisioner-auth.mjs";
const serviceId = "nodedc-engine";
const keyId = "engine-edp-managed-provisioner-v1";
const audience = "nodedc-external-data-plane.managed-provisioning.v1";
const nowMs = Date.parse("2026-07-16T12:00:00.000Z");
const { privateKey, publicKey } = generateKeyPairSync("ed25519");
let nonceSequence = 0;
const replayCache = new ManagedProvisionerReplayCache({ maxEntries: 100 });
const verify = createManagedProvisionerRequestVerifier({
publicKey,
serviceId,
keyId,
audience,
maxSkewMs: 60_000,
replayCache,
now: () => nowMs,
});
const valid = signedRequest();
assert.deepEqual(verify(valid), { serviceId, keyId });
assert.throws(() => verify(valid), hasCode("managed_provisioner_request_replayed", 409));
const whitespaceTamper = signedRequest();
whitespaceTamper.rawBody = Buffer.from(`${whitespaceTamper.rawBody.toString("utf8")} `, "utf8");
assert.throws(() => verify(whitespaceTamper), hasCode("managed_provisioner_body_hash_mismatch", 401));
const methodTamper = signedRequest();
methodTamper.method = "POST";
assert.throws(() => verify(methodTamper), hasCode("managed_provisioner_signature_invalid", 401));
const pathTamper = signedRequest();
pathTamper.originalUrl = `${pathTamper.originalUrl}?unexpected=true`;
assert.throws(() => verify(pathTamper), hasCode("managed_provisioner_signature_invalid", 401));
const audienceTamper = signedRequest();
setHeader(audienceTamper, MANAGED_PROVISIONER_HEADERS.audience, "other-audience");
assert.throws(() => verify(audienceTamper), hasCode("managed_provisioner_identity_mismatch", 401));
const serviceTamper = signedRequest();
setHeader(serviceTamper, MANAGED_PROVISIONER_HEADERS.serviceId, "other-engine");
assert.throws(() => verify(serviceTamper), hasCode("managed_provisioner_identity_mismatch", 401));
const keyTamper = signedRequest();
setHeader(keyTamper, MANAGED_PROVISIONER_HEADERS.keyId, "other-key-v1");
assert.throws(() => verify(keyTamper), hasCode("managed_provisioner_identity_mismatch", 401));
const forbiddenBearer = signedRequest();
forbiddenBearer.headers.authorization = "Bearer value-that-must-not-appear-in-errors";
forbiddenBearer.rawHeaders.push("Authorization", forbiddenBearer.headers.authorization);
assert.throws(
() => verify(forbiddenBearer),
(error) => hasCode("managed_provisioner_authorization_header_forbidden", 401)(error)
&& !error.message.includes("value-that-must-not-appear-in-errors"),
);
const stale = signedRequest({ timestamp: new Date(nowMs - 60_000).toISOString() });
assert.throws(() => verify(stale), hasCode("managed_provisioner_timestamp_outside_window", 401));
const future = signedRequest({ timestamp: new Date(nowMs + 60_000).toISOString() });
assert.throws(() => verify(future), hasCode("managed_provisioner_timestamp_outside_window", 401));
const invalidTimestamp = signedRequest();
setHeader(invalidTimestamp, MANAGED_PROVISIONER_HEADERS.timestamp, "2026-07-16T12:00:00Z");
assert.throws(() => verify(invalidTimestamp), hasCode("managed_provisioner_timestamp_invalid", 401));
const duplicateHeader = signedRequest();
duplicateHeader.rawHeaders.push(MANAGED_PROVISIONER_HEADERS.nonce, header(duplicateHeader, MANAGED_PROVISIONER_HEADERS.nonce));
assert.throws(() => verify(duplicateHeader), hasCode("managed_provisioner_header_invalid", 401));
const uncaptured = signedRequest();
uncaptured.rawBodyCaptured = false;
uncaptured.headers["content-length"] = String(uncaptured.rawBody.length);
assert.throws(() => verify(uncaptured), hasCode("managed_provisioner_raw_body_unavailable", 401));
const badSignature = signedRequest();
setHeader(badSignature, MANAGED_PROVISIONER_HEADERS.signature, randomBytes(64).toString("base64url"));
assert.throws(() => verify(badSignature), hasCode("managed_provisioner_signature_invalid", 401));
const paddedSignature = signedRequest();
setHeader(
paddedSignature,
MANAGED_PROVISIONER_HEADERS.signature,
`${header(paddedSignature, MANAGED_PROVISIONER_HEADERS.signature)}=`,
);
assert.throws(() => verify(paddedSignature), hasCode("managed_provisioner_signature_invalid", 401));
const uppercaseHash = signedRequest();
setHeader(
uppercaseHash,
MANAGED_PROVISIONER_HEADERS.bodySha256,
header(uppercaseHash, MANAGED_PROVISIONER_HEADERS.bodySha256).toUpperCase(),
);
assert.throws(() => verify(uppercaseHash), hasCode("managed_provisioner_body_hash_invalid", 401));
const cache = new ManagedProvisionerReplayCache({ maxEntries: 1 });
cache.consume("first", nowMs + 1000, nowMs);
assert.throws(() => cache.consume("second", nowMs + 1000, nowMs), hasCode("managed_provisioner_replay_cache_exhausted", 503));
assert.throws(() => cache.consume("first", nowMs + 1000, nowMs), hasCode("managed_provisioner_request_replayed", 409));
cache.consume("second", nowMs + 2000, nowMs + 1000);
const directory = await mkdtemp(join(tmpdir(), "nodedc-edp-managed-auth-"));
try {
const publicKeyPath = join(directory, "engine-public-key.pem");
const rsaPublicKeyPath = join(directory, "rsa-public-key.pem");
const privateKeyPath = join(directory, "engine-private-key.pem");
const symlinkPath = join(directory, "engine-public-key-link.pem");
await writeFile(publicKeyPath, publicKey.export({ type: "spki", format: "pem" }), { mode: 0o600 });
assert.equal(loadManagedProvisionerPublicKeyFile(publicKeyPath).asymmetricKeyType, "ed25519");
assert.throws(
() => loadManagedProvisionerPublicKeyFile("relative-public-key.pem"),
/external_data_plane_managed_provisioner_public_key_file_must_be_absolute/,
);
assert.throws(
() => loadManagedProvisionerPublicKeyFile(join(directory, "missing.pem")),
/external_data_plane_managed_provisioner_public_key_file_unreadable/,
);
await symlink(publicKeyPath, symlinkPath);
assert.throws(
() => loadManagedProvisionerPublicKeyFile(symlinkPath),
/external_data_plane_managed_provisioner_public_key_file_must_be_regular/,
);
await chmod(publicKeyPath, 0o622);
assert.throws(
() => loadManagedProvisionerPublicKeyFile(publicKeyPath),
/external_data_plane_managed_provisioner_public_key_file_permissions_invalid/,
);
await chmod(publicKeyPath, 0o600);
const { publicKey: rsaPublicKey } = generateKeyPairSync("rsa", { modulusLength: 2048 });
await writeFile(rsaPublicKeyPath, rsaPublicKey.export({ type: "spki", format: "pem" }), { mode: 0o600 });
assert.throws(
() => loadManagedProvisionerPublicKeyFile(rsaPublicKeyPath),
/external_data_plane_managed_provisioner_public_key_algorithm_invalid/,
);
await writeFile(privateKeyPath, privateKey.export({ type: "pkcs8", format: "pem" }), { mode: 0o600 });
assert.throws(
() => loadManagedProvisionerPublicKeyFile(privateKeyPath),
/external_data_plane_managed_provisioner_public_key_file_format_invalid/,
);
} finally {
await rm(directory, { recursive: true, force: true });
}
console.log("external-data-plane managed provisioner auth: ok");
function signedRequest({
method = "PUT",
path = "/internal/data-plane/v1/writer-bindings/by-key/engine.test.positions",
rawBody = Buffer.from('{"generation":1}', "utf8"),
timestamp = new Date(nowMs).toISOString(),
nonce = nextNonce(),
} = {}) {
const bodySha256 = sha256RawBody(rawBody);
const signature = sign(null, Buffer.from(managedProvisionerSigningPayload({
audience,
serviceId,
keyId,
method,
path,
timestamp,
nonce,
bodySha256,
}), "utf8"), privateKey).toString("base64url");
const headers = {
[MANAGED_PROVISIONER_HEADERS.serviceId]: serviceId,
[MANAGED_PROVISIONER_HEADERS.keyId]: keyId,
[MANAGED_PROVISIONER_HEADERS.audience]: audience,
[MANAGED_PROVISIONER_HEADERS.timestamp]: timestamp,
[MANAGED_PROVISIONER_HEADERS.nonce]: nonce,
[MANAGED_PROVISIONER_HEADERS.bodySha256]: bodySha256,
[MANAGED_PROVISIONER_HEADERS.signature]: signature,
"content-length": String(rawBody.length),
};
return {
method,
originalUrl: path,
headers,
rawHeaders: Object.entries(headers).flat(),
rawBody,
rawBodyCaptured: true,
};
}
function nextNonce() {
nonceSequence += 1;
return Buffer.from(`managed-auth-test-${String(nonceSequence).padStart(6, "0")}`).toString("base64url");
}
function header(req, name) {
return req.headers[name];
}
function setHeader(req, name, value) {
req.headers[name] = value;
const index = req.rawHeaders.findIndex((candidate) => String(candidate).toLowerCase() === name);
req.rawHeaders[index + 1] = value;
}
function hasCode(code, status) {
return (error) => error?.code === code && error?.status === status;
}

37
services/external-data-plane/test/writer-binding.test.mjs
View File

@ -1,14 +1,12 @@
import assert from "node:assert/strict"; import assert from "node:assert/strict";
import { validateIntakeBatch } from "../../../packages/external-provider-contract/src/data-plane.mjs"; import { validateIntakeBatch } from "../../../packages/external-provider-contract/src/index.mjs";
import { import {
createWriterToken, createWriterToken,
hashWriterToken, hashWriterToken,
materializeDataProductPublish, materializeDataProductPublish,
materializeWriterBoundBatch, materializeWriterBoundBatch,
normalizeManagedWriterBindingRequest,
normalizeWriterBindingRequest, normalizeWriterBindingRequest,
safeWriterBinding, safeWriterBinding,
writerBindingRequestHash,
} from "../src/writer-binding.mjs"; } from "../src/writer-binding.mjs";
const now = new Date("2026-07-15T12:00:00.000Z"); const now = new Date("2026-07-15T12:00:00.000Z");
@ -36,39 +34,6 @@ assert.match(token, /^ndc_edpwb_[A-Za-z0-9_-]{40,}$/);
assert.equal(hashWriterToken(token), hashWriterToken(token)); assert.equal(hashWriterToken(token), hashWriterToken(token));
assert.notEqual(hashWriterToken(token), hashWriterToken(`${token}x`)); assert.notEqual(hashWriterToken(token), hashWriterToken(`${token}x`));
const managedPolicy = normalizeManagedWriterBindingRequest({
...request,
allowedDataProductIds: ["fleet.positions.current.v1", "asset.status.current.v1"],
generation: 1,
capabilityDigest: hashWriterToken(token),
}, { now, maxTtlDays: 90 });
assert.deepEqual(managedPolicy.allowedDataProductIds, [
"asset.status.current.v1",
"fleet.positions.current.v1",
]);
assert.equal(managedPolicy.generation, 1);
assert.equal(managedPolicy.capabilityDigest, hashWriterToken(token));
assert.equal(writerBindingRequestHash(managedPolicy), writerBindingRequestHash({
...managedPolicy,
allowedDataProductIds: [...managedPolicy.allowedDataProductIds].reverse(),
}));
assert.throws(() => normalizeManagedWriterBindingRequest({
...request,
generation: 0,
capabilityDigest: hashWriterToken(token),
}, { now, maxTtlDays: 90 }), /managed_writer_binding_generation_invalid/);
assert.throws(() => normalizeManagedWriterBindingRequest({
...request,
generation: 1,
capabilityDigest: "not-a-digest",
}, { now, maxTtlDays: 90 }), /managed_writer_binding_capability_digest_invalid/);
assert.throws(() => normalizeManagedWriterBindingRequest({
...request,
generation: 1,
capabilityDigest: hashWriterToken(token),
token,
}, { now, maxTtlDays: 90 }), /managed_writer_binding_secret_material_forbidden/);
const unscopedBatch = { const unscopedBatch = {
schemaVersion: "nodedc.external-provider-contract/v1", schemaVersion: "nodedc.external-provider-contract/v1",
source: { providerId: "example-provider" }, source: { providerId: "example-provider" },

12
services/ontology-core/catalog/domain-packages/gelios/aliases.json
View File

@ -1,19 +1,19 @@
{ {
"version": "1.0.0", "version": "0.1.0",
"updatedAt": "2026-07-16", "updatedAt": "2026-07-13",
"aliases": [ "aliases": [
{ "alias": "gelios", "canonicalId": "gelios.integration" }, { "alias": "gelios", "canonicalId": "gelios.integration" },
{ "alias": "gelius", "canonicalId": "gelios.integration" }, { "alias": "gelius", "canonicalId": "gelios.integration" },
{ "alias": "helius", "canonicalId": "gelios.integration" }, { "alias": "helius", "canonicalId": "gelios.integration" },
{ "alias": "hel i os", "canonicalId": "gelios.integration" }, { "alias": "hel i os", "canonicalId": "gelios.integration" },
{ "alias": "гелиос", "canonicalId": "gelios.integration" }, { "alias": "гелиос", "canonicalId": "gelios.integration" },
{ "alias": "отслеживаемый объект гелиос", "canonicalId": "gelios.unit" }, { "alias": "трайк robot2b", "canonicalId": "gelios.unit" },
{ "alias": "юнит гелиос", "canonicalId": "gelios.unit" }, { "alias": "юнит гелиос", "canonicalId": "gelios.unit" },
{ "alias": "группа юнитов гелиос", "canonicalId": "gelios.unit_group" }, { "alias": "группа юнитов гелиос", "canonicalId": "gelios.unit_group" },
{ "alias": "последняя телеметрия гелиос", "canonicalId": "gelios.telemetry_snapshot" }, { "alias": "последняя телеметрия гелиос", "canonicalId": "gelios.telemetry_snapshot" },
{ "alias": "сырая телеметрия гелиос", "canonicalId": "gelios.raw_telemetry_message" }, { "alias": "сырая телеметрия гелиос", "canonicalId": "gelios.raw_telemetry_message" },
{ "alias": "позиция объекта гелиос", "canonicalId": "gelios.position_fix" }, { "alias": "позиция трайка", "canonicalId": "gelios.position_fix" },
{ "alias": "статус объекта гелиос", "canonicalId": "gelios.operational_status" }, { "alias": "статус трайка", "canonicalId": "gelios.operational_status" },
{ "alias": "датчик гелиос", "canonicalId": "gelios.sensor_definition" }, { "alias": "датчик гелиос", "canonicalId": "gelios.sensor_definition" },
{ "alias": "показание датчика", "canonicalId": "gelios.sensor_reading" }, { "alias": "показание датчика", "canonicalId": "gelios.sensor_reading" },
{ "alias": "калибровка датчика", "canonicalId": "gelios.sensor_conversion" }, { "alias": "калибровка датчика", "canonicalId": "gelios.sensor_conversion" },
@ -23,7 +23,7 @@
{ "alias": "шаблон команды гелиос", "canonicalId": "gelios.command_template" }, { "alias": "шаблон команды гелиос", "canonicalId": "gelios.command_template" },
{ "alias": "диспетчеризация команды гелиос", "canonicalId": "gelios.command_dispatch" }, { "alias": "диспетчеризация команды гелиос", "canonicalId": "gelios.command_dispatch" },
{ "alias": "аудит команды гелиос", "canonicalId": "gelios.command_audit" }, { "alias": "аудит команды гелиос", "canonicalId": "gelios.command_audit" },
{ "alias": "область доступа credential гелиос", "canonicalId": "gelios.access_scope" }, { "alias": "контур robot2b", "canonicalId": "gelios.access_scope" },
{ "alias": "курсор загрузки гелиос", "canonicalId": "gelios.ingestion_cursor" } { "alias": "курсор загрузки гелиос", "canonicalId": "gelios.ingestion_cursor" }
] ]
} }

32
services/ontology-core/catalog/domain-packages/gelios/entities.json
View File

@ -1,19 +1,19 @@
{ {
"version": "1.0.0", "version": "0.1.0",
"updatedAt": "2026-07-16", "updatedAt": "2026-07-13",
"entities": [ "entities": [
{ "id": "gelios.integration", "name": "Gelios Integration", "surface": "platform", "status": ["product-required", "source-evidenced"], "authority": "NDC L2 connection", "summary": "One tenant-scoped Gelios provider connection backed by one opaque NDC L2 credential reference. It owns no UI, renderer, secret value or provider-specific Platform service." }, { "id": "gelios.integration", "name": "Gelios Integration", "surface": "platform", "status": ["product-required", "source-evidenced"], "authority": "Gelios Gateway", "summary": "Server-side provider integration boundary for Gelios REST OAuth and legacy compatibility. It owns no UI, map renderer or workflow secret." },
{ "id": "gelios.access_scope", "name": "Gelios Access Scope", "surface": "platform", "status": ["product-required", "source-evidenced"], "authority": "Gelios credential visibility + NDC L2 connection policy", "summary": "Dynamic collection boundary: selected safe-read capabilities include every entity visible to the bound credential on each run. Entity allowlists and provider-group business filters are not part of collection scope." }, { "id": "gelios.access_scope", "name": "Gelios Access Scope", "surface": "platform", "status": ["product-required", "source-evidenced"], "authority": "Robot2B owner + Gelios Gateway", "summary": "Approved collection boundary expressed as an allowlist or owner rule for units, groups and allowed capabilities; distinct from broad provider-account visibility." },
{ "id": "gelios.unit", "name": "Gelios Unit", "surface": "domain", "status": ["source-evidenced"], "authority": "Gelios Pro", "summary": "Tracked operational object with a stable provider source ID. Every unit returned for the bound credential is eligible for canonical mapping." }, { "id": "gelios.unit", "name": "Gelios Unit", "surface": "domain", "status": ["source-evidenced"], "authority": "Gelios Pro", "summary": "Tracked operational object, including a Robot2B trike when its stable source unit ID is inside the approved scope." },
{ "id": "gelios.unit_group", "name": "Gelios Unit Group", "surface": "domain", "status": ["source-evidenced"], "authority": "Gelios Pro", "summary": "Provider-managed grouping of units used as domain data and optional consumer metadata, never as an implicit collection filter." }, { "id": "gelios.unit_group", "name": "Gelios Unit Group", "surface": "domain", "status": ["source-evidenced"], "authority": "Gelios Pro", "summary": "Provider-managed grouping of units. It is evidence for navigation and access, not an automatic definition of the Robot2B business scope." },
{ "id": "gelios.tracker_device", "name": "Tracker Device", "surface": "domain", "status": ["source-evidenced"], "authority": "Gelios Pro", "summary": "Hardware tracker associated with a unit, carrying device type/manufacturer and restricted hardware identity fields." }, { "id": "gelios.tracker_device", "name": "Tracker Device", "surface": "domain", "status": ["source-evidenced"], "authority": "Gelios Pro", "summary": "Hardware tracker associated with a unit, carrying device type/manufacturer and restricted hardware identity fields." },
{ "id": "gelios.telemetry_snapshot", "name": "Telemetry Snapshot", "surface": "telemetry", "status": ["source-evidenced", "product-required"], "authority": "NDC L2 semantic mapping", "summary": "Normalized current telemetry state for one unit at observed and received time; derived from the versioned Gelios last-message mapping contract." }, { "id": "gelios.telemetry_snapshot", "name": "Telemetry Snapshot", "surface": "telemetry", "status": ["source-evidenced", "product-required"], "authority": "Gelios Gateway", "summary": "Normalized current telemetry state for one unit at observed and received time; derived from the Gelios pure last message contract." },
{ "id": "gelios.raw_telemetry_message", "name": "Raw Telemetry Message", "surface": "telemetry", "status": ["source-evidenced"], "authority": "Gelios Pro", "summary": "Provider hardware message preserved only behind restricted raw-data policy; it is not the default Studio or analytics contract." }, { "id": "gelios.raw_telemetry_message", "name": "Raw Telemetry Message", "surface": "telemetry", "status": ["source-evidenced"], "authority": "Gelios Pro", "summary": "Provider hardware message preserved only behind restricted raw-data policy; it is not the default Studio or analytics contract." },
{ "id": "gelios.position_fix", "name": "Position Fix", "surface": "telemetry", "status": ["source-evidenced", "product-required"], "authority": "NDC L2 semantic mapping", "summary": "Time-qualified geographic position with latitude, longitude, height, course, speed, satellite and quality attributes for one unit." }, { "id": "gelios.position_fix", "name": "Position Fix", "surface": "telemetry", "status": ["source-evidenced", "product-required"], "authority": "Gelios Gateway", "summary": "Time-qualified geographic position with latitude, longitude, height, course, speed, satellite and quality attributes for one unit." },
{ "id": "gelios.telemetry_parameter", "name": "Telemetry Parameter", "surface": "telemetry", "status": ["source-evidenced"], "authority": "Gelios Pro", "summary": "Name/value parameter received with a message. Parameters are dynamic and require a namespaced registry and whitelist before product exposure." }, { "id": "gelios.telemetry_parameter", "name": "Telemetry Parameter", "surface": "telemetry", "status": ["source-evidenced"], "authority": "Gelios Pro", "summary": "Name/value parameter received with a message. Parameters are dynamic and require a namespaced registry and whitelist before product exposure." },
{ "id": "gelios.operational_status", "name": "Operational Status", "surface": "telemetry", "status": ["source-evidenced", "product-required"], "authority": "NDC L2 semantic mapping", "summary": "Derived unit state such as active, stale, moving, parked, no-position or low-GPS; distinct from raw provider message fields." }, { "id": "gelios.operational_status", "name": "Operational Status", "surface": "telemetry", "status": ["source-evidenced", "product-required"], "authority": "Gelios Gateway", "summary": "Derived unit state such as online, offline, moving, parked, no-position or low-GPS; distinct from raw provider message fields." },
{ "id": "gelios.sensor_definition", "name": "Sensor Definition", "surface": "domain", "status": ["source-evidenced"], "authority": "Gelios Pro", "summary": "Declared sensor on a unit with message parameter, type, unit of measure, visibility and optional fuel semantics." }, { "id": "gelios.sensor_definition", "name": "Sensor Definition", "surface": "domain", "status": ["source-evidenced"], "authority": "Gelios Pro", "summary": "Declared sensor on a unit with message parameter, type, unit of measure, visibility and optional fuel semantics." },
{ "id": "gelios.sensor_reading", "name": "Sensor Reading", "surface": "telemetry", "status": ["source-evidenced", "product-required"], "authority": "NDC L2 semantic mapping", "summary": "Time-qualified normalized reading of a defined sensor, retaining both numeric value and human-readable representation only when permitted by a versioned field policy." }, { "id": "gelios.sensor_reading", "name": "Sensor Reading", "surface": "telemetry", "status": ["source-evidenced", "product-required"], "authority": "Gelios Gateway", "summary": "Time-qualified normalized reading of a defined sensor, retaining both numeric value and human-readable representation when permitted." },
{ "id": "gelios.sensor_conversion", "name": "Sensor Conversion", "surface": "domain", "status": ["source-evidenced"], "authority": "Gelios Pro", "summary": "Calibration/conversion rule or row for translating raw sensor values into operational measures." }, { "id": "gelios.sensor_conversion", "name": "Sensor Conversion", "surface": "domain", "status": ["source-evidenced"], "authority": "Gelios Pro", "summary": "Calibration/conversion rule or row for translating raw sensor values into operational measures." },
{ "id": "gelios.fuel_profile", "name": "Fuel Profile", "surface": "domain", "status": ["source-evidenced"], "authority": "Gelios Pro", "summary": "Fuel and consumption configuration associated with a unit or fuel sensor; use only after sensor semantics are approved." }, { "id": "gelios.fuel_profile", "name": "Fuel Profile", "surface": "domain", "status": ["source-evidenced"], "authority": "Gelios Pro", "summary": "Fuel and consumption configuration associated with a unit or fuel sensor; use only after sensor semantics are approved." },
{ "id": "gelios.maintenance_plan", "name": "Maintenance Plan", "surface": "domain", "status": ["source-evidenced"], "authority": "Gelios Pro", "summary": "Planned maintenance configuration for a unit, separate from historical maintenance events." }, { "id": "gelios.maintenance_plan", "name": "Maintenance Plan", "surface": "domain", "status": ["source-evidenced"], "authority": "Gelios Pro", "summary": "Planned maintenance configuration for a unit, separate from historical maintenance events." },
@ -22,13 +22,13 @@
{ "id": "gelios.geozone_group", "name": "Gelios Geozone Group", "surface": "spatial", "status": ["source-evidenced"], "authority": "Gelios Pro", "summary": "Provider grouping of geozones." }, { "id": "gelios.geozone_group", "name": "Gelios Geozone Group", "surface": "spatial", "status": ["source-evidenced"], "authority": "Gelios Pro", "summary": "Provider grouping of geozones." },
{ "id": "gelios.geopoint", "name": "Gelios Geopoint", "surface": "spatial", "status": ["source-evidenced"], "authority": "Gelios Pro", "summary": "Named provider spatial point; it must remain distinct from a live unit position." }, { "id": "gelios.geopoint", "name": "Gelios Geopoint", "surface": "spatial", "status": ["source-evidenced"], "authority": "Gelios Pro", "summary": "Named provider spatial point; it must remain distinct from a live unit position." },
{ "id": "gelios.report_template", "name": "Gelios Report Template", "surface": "analytics", "status": ["source-evidenced"], "authority": "Gelios Pro", "summary": "Reusable provider report definition available to the account or scope." }, { "id": "gelios.report_template", "name": "Gelios Report Template", "surface": "analytics", "status": ["source-evidenced"], "authority": "Gelios Pro", "summary": "Reusable provider report definition available to the account or scope." },
{ "id": "gelios.report_result", "name": "Gelios Report Result", "surface": "analytics", "status": ["source-evidenced"], "authority": "Gelios Pro + NDC L2 semantic mapping", "summary": "Bounded result, table, graphic or map output produced from an approved report request; not a default realtime stream." }, { "id": "gelios.report_result", "name": "Gelios Report Result", "surface": "analytics", "status": ["source-evidenced"], "authority": "Gelios Pro + Gelios Gateway", "summary": "Bounded result, table, graphic or map output produced from an approved report request; not a default realtime stream." },
{ "id": "gelios.collection_run", "name": "Gelios Collection Run", "surface": "platform", "status": ["product-required"], "authority": "NDC L2 workflow", "summary": "Auditable safe-read collection attempt with credential-visible entity scope, endpoint family, volume controls and outcome; it contains no credential value or raw payload." }, { "id": "gelios.collection_run", "name": "Gelios Collection Run", "surface": "platform", "status": ["product-required"], "authority": "Gelios Gateway", "summary": "Auditable read-only collection attempt with scope, endpoint family, volume controls and outcome; it contains no token or raw payload." },
{ "id": "gelios.ingestion_cursor", "name": "Gelios Ingestion Cursor", "surface": "platform", "status": ["product-required"], "authority": "NDC L2 workflow state + External Data Plane", "summary": "Checkpoint and watermark controlling resumable historical or incremental ingestion for a connection and data family." }, { "id": "gelios.ingestion_cursor", "name": "Gelios Ingestion Cursor", "surface": "platform", "status": ["product-required"], "authority": "Gelios Gateway + telemetry storage", "summary": "Checkpoint and watermark controlling resumable historical or incremental ingestion for an approved unit and data family." },
{ "id": "gelios.command_template", "name": "Gelios Command Template", "surface": "command", "status": ["source-evidenced", "product-required"], "authority": "Gelios Pro", "summary": "Read-only catalogued command template assigned to a unit. Template content is red-domain configuration, not a telemetry field." }, { "id": "gelios.command_template", "name": "Gelios Command Template", "surface": "command", "status": ["source-evidenced", "product-required"], "authority": "Gelios Pro", "summary": "Read-only catalogued command template assigned to a unit. Template content is red-domain configuration, not a telemetry field." },
{ "id": "gelios.command_group", "name": "Gelios Command Group", "surface": "command", "status": ["source-evidenced", "product-required"], "authority": "Gelios Pro", "summary": "Provider grouping of command templates, managed separately from operational unit groups." }, { "id": "gelios.command_group", "name": "Gelios Command Group", "surface": "command", "status": ["source-evidenced", "product-required"], "authority": "Gelios Pro", "summary": "Provider grouping of command templates, managed separately from operational unit groups." },
{ "id": "gelios.command_dispatch", "name": "Gelios Command Dispatch", "surface": "command", "status": ["future-concept"], "authority": "Unimplemented NDC red-domain command boundary", "summary": "Future explicit, user-confirmed request to send a command to one approved unit or approved group. It must never be created by background ingestion or map rendering." }, { "id": "gelios.command_dispatch", "name": "Gelios Command Dispatch", "surface": "command", "status": ["product-required"], "authority": "Future Command Gateway", "summary": "Explicit, user-confirmed request to send a command to one approved unit or approved group. It must never be created by background ingestion or map rendering." },
{ "id": "gelios.command_delivery", "name": "Gelios Command Delivery", "surface": "command", "status": ["future-concept", "source-evidenced"], "authority": "Gelios Pro; future NDC red-domain reconciliation", "summary": "Provider delivery/task status that may be reconciled with a future governed command dispatch; no command transport exists in the current package." }, { "id": "gelios.command_delivery", "name": "Gelios Command Delivery", "surface": "command", "status": ["product-required", "source-evidenced"], "authority": "Gelios Pro + Future Command Gateway", "summary": "Provider delivery/task status associated with a command dispatch." },
{ "id": "gelios.command_audit", "name": "Gelios Command Audit", "surface": "command", "status": ["future-concept"], "authority": "Unimplemented NDC red-domain command boundary", "summary": "Future immutable governance record of an explicit command decision, confirmation, target, outcome and actor; separate from provider command history." } { "id": "gelios.command_audit", "name": "Gelios Command Audit", "surface": "command", "status": ["product-required"], "authority": "Future Command Gateway", "summary": "Immutable governance record of an explicit command decision, confirmation, target, outcome and actor; separate from provider command history." }
] ]
} }

12
services/ontology-core/catalog/domain-packages/gelios/evidence.json
View File

@ -1,6 +1,6 @@
{ {
"version": "1.0.0", "version": "0.1.0",
"updatedAt": "2026-07-16", "updatedAt": "2026-07-13",
"sourceRoots": [ "sourceRoots": [
{ {
"surface": "gelios-rest", "surface": "gelios-rest",
@ -8,9 +8,9 @@
"mode": "OpenAPI inspection and safe read-only runtime audit; no write routes executed" "mode": "OpenAPI inspection and safe read-only runtime audit; no write routes executed"
}, },
{ {
"surface": "ndc-l2-legacy-evidence", "surface": "engine",
"path": "/Users/dcconstructions/Downloads/mnt/NODEDC/NODEDC_ENGINE_INFRA/nodedc-source", "path": "/Users/dcconstructions/Downloads/mnt/NODEDC/NODEDC_ENGINE_INFRA/nodedc-source",
"mode": "Read-only historical NDC L2 donor, Gelios mapping and synthetic-safe field-shape inspection" "mode": "Read-only MMAP workflow, Gelios node and historical snapshot donor inspection"
}, },
{ {
"surface": "module-studio", "surface": "module-studio",
@ -41,9 +41,9 @@
"baselineDocs": ["docs/GELIOS_DOMAIN_ONTOLOGY.md"], "baselineDocs": ["docs/GELIOS_DOMAIN_ONTOLOGY.md"],
"restrictions": [ "restrictions": [
"Do not copy access/refresh tokens, passwords, hardware decrypt keys or runtime snapshots into Ontology Core.", "Do not copy access/refresh tokens, passwords, hardware decrypt keys or runtime snapshots into Ontology Core.",
"Do not make a current NDC L2 workflow node, renderer entities or provider group names canonical domain identities.", "Do not make the current Engine Gelios node, Cesium entities or provider group names canonical domain identities.",
"Do not invoke command send, create, update, delete or purge routes as ontology evidence.", "Do not invoke command send, create, update, delete or purge routes as ontology evidence.",
"Robot2B pilot counts (107 credential-visible units and 95 legacy snapshot units) are historical evidence only; canonical collection scope is dynamically all entities visible to the bound credential.", "Do not treat the 107 currently visible provider-account units or the 95 legacy snapshot units as the final Robot2B scope without owner approval.",
"Do not bulk-download history, media or full geozone geometry before collection policy and storage architecture are approved." "Do not bulk-download history, media or full geozone geometry before collection policy and storage architecture are approved."
] ]
} }

14
services/ontology-core/catalog/domain-packages/gelios/guardrails.json
View File

@ -1,6 +1,6 @@
{ {
"version": "1.0.0", "version": "0.1.0",
"updatedAt": "2026-07-16", "updatedAt": "2026-07-13",
"rules": [ "rules": [
{ {
"id": "guardrail.gelios.ontology_not_runtime_store", "id": "guardrail.gelios.ontology_not_runtime_store",
@ -8,16 +8,10 @@
"summary": "Ontology Core describes Gelios meanings and contracts only. It must not contain access/refresh tokens, raw messages, live positions, provider command text, hardware decrypt keys or bulk geometry.", "summary": "Ontology Core describes Gelios meanings and contracts only. It must not contain access/refresh tokens, raw messages, live positions, provider command text, hardware decrypt keys or bulk geometry.",
"entityIds": ["gelios.integration", "gelios.telemetry_snapshot", "gelios.raw_telemetry_message", "gelios.position_fix", "gelios.command_template", "gelios.geozone"] "entityIds": ["gelios.integration", "gelios.telemetry_snapshot", "gelios.raw_telemetry_message", "gelios.position_fix", "gelios.command_template", "gelios.geozone"]
}, },
{
"id": "guardrail.gelios.provider_auth_pair_not_capability_scope",
"severity": "error",
"summary": "Gelios provider authentication consists of access and refresh tokens only. Credential labels and safe-read classifications are workflow policy, not provider token scope; internal External Data Plane capabilities are not Gelios tokens.",
"entityIds": ["gelios.integration", "gelios.access_scope", "gelios.collection_run"]
},
{ {
"id": "guardrail.gelios.scope_precedes_collection", "id": "guardrail.gelios.scope_precedes_collection",
"severity": "error", "severity": "error",
"summary": "A Gelios safe-read collection run must process every valid entity returned for the bound credential and selected capability. Static unit allowlists and provider-group business filters are forbidden at collection boundary; consumer visibility is handled downstream.", "summary": "A Gelios collection run must resolve an approved Robot2B access scope before addressing units. Provider-account visibility and provider group membership are not sufficient product scope on their own.",
"entityIds": ["gelios.integration", "gelios.access_scope", "gelios.collection_run", "gelios.unit", "gelios.unit_group"] "entityIds": ["gelios.integration", "gelios.access_scope", "gelios.collection_run", "gelios.unit", "gelios.unit_group"]
}, },
{ {
@ -47,7 +41,7 @@
{ {
"id": "guardrail.gelios.history_requires_cursor", "id": "guardrail.gelios.history_requires_cursor",
"severity": "warning", "severity": "warning",
"summary": "Any future history ingestion must be bounded by connection, credential-visible entity scope, time window, field policy and durable cursor. It must not turn a historical API family into an uncontrolled account dump.", "summary": "Any future history ingestion must be bounded by unit scope, time window, field whitelist and durable cursor. It must not turn a historical API family into an uncontrolled account dump.",
"entityIds": ["gelios.collection_run", "gelios.ingestion_cursor", "gelios.raw_telemetry_message", "gelios.report_result"] "entityIds": ["gelios.collection_run", "gelios.ingestion_cursor", "gelios.raw_telemetry_message", "gelios.report_result"]
} }
], ],

6
services/ontology-core/catalog/domain-packages/gelios/package.json
View File

@ -1,7 +1,7 @@
{ {
"id": "gelios", "id": "gelios",
"version": "1.0.0", "version": "0.1.0",
"updatedAt": "2026-07-16", "updatedAt": "2026-07-13",
"status": "product-required/source-evidenced", "status": "product-required/source-evidenced",
"summary": "Provider-neutral Gelios Pro telemetry, fleet, spatial, reporting and command-domain ontology. It describes canonical meanings and contracts; it does not store credentials, account scope or runtime telemetry." "summary": "Gelios Pro telemetry, fleet, spatial, reporting and command-domain ontology for the Robot2B client context. The package describes canonical meanings and contracts; it does not store credentials or runtime telemetry."
} }

26
services/ontology-core/catalog/domain-packages/gelios/relations.json
View File

@ -1,23 +1,23 @@
{ {
"version": "1.0.0", "version": "0.1.0",
"updatedAt": "2026-07-16", "updatedAt": "2026-07-13",
"relations": [ "relations": [
{ "id": "gelios.integration.is_provider_connection", "from": ["gelios.integration"], "to": ["integration.connection"], "status": "product-required", "summary": "A Gelios integration is a tenant-scoped instance of the common external provider connection contract." }, { "id": "gelios.integration.is_provider_connection", "from": ["gelios.integration"], "to": ["integration.connection"], "status": "product-required", "summary": "A Gelios integration is a tenant-scoped instance of the common external provider connection contract." },
{ "id": "gelios.access_scope.specializes_integration_scope", "from": ["gelios.access_scope"], "to": ["integration.access_scope"], "status": "product-required", "summary": "Gelios credential-visible capability scope specializes the common provider access-scope contract." }, { "id": "gelios.access_scope.specializes_integration_scope", "from": ["gelios.access_scope"], "to": ["integration.access_scope"], "status": "product-required", "summary": "Robot2B/Gelios object approval specializes the common provider access-scope contract." },
{ "id": "gelios.collection_run.is_integration_collection_run", "from": ["gelios.collection_run"], "to": ["integration.collection_run"], "status": "product-required", "summary": "Gelios collection runs inherit the bounded collection and audit lifecycle of external integrations." }, { "id": "gelios.collection_run.is_integration_collection_run", "from": ["gelios.collection_run"], "to": ["integration.collection_run"], "status": "product-required", "summary": "Gelios collection runs inherit the bounded collection and audit lifecycle of external integrations." },
{ "id": "gelios.raw_message.is_integration_raw_envelope", "from": ["gelios.raw_telemetry_message"], "to": ["integration.raw_envelope"], "status": "product-required", "summary": "A Gelios raw message is a restricted provider envelope, not a UI read model." }, { "id": "gelios.raw_message.is_integration_raw_envelope", "from": ["gelios.raw_telemetry_message"], "to": ["integration.raw_envelope"], "status": "product-required", "summary": "A Gelios raw message is a restricted provider envelope, not a UI read model." },
{ "id": "gelios.unit.is_integration_canonical_subject", "from": ["gelios.unit"], "to": ["integration.canonical_subject"], "status": "product-required", "summary": "A Gelios unit is a stable canonical subject produced by the provider adapter." }, { "id": "gelios.unit.is_integration_canonical_subject", "from": ["gelios.unit"], "to": ["integration.canonical_subject"], "status": "product-required", "summary": "A Gelios unit is a stable canonical subject produced by the provider adapter." },
{ "id": "gelios.command_template.is_integration_command_capability", "from": ["gelios.command_template"], "to": ["integration.command_capability"], "status": "product-required", "summary": "Gelios command templates are catalogued provider command capabilities only." }, { "id": "gelios.command_template.is_integration_command_capability", "from": ["gelios.command_template"], "to": ["integration.command_capability"], "status": "product-required", "summary": "Gelios command templates are catalogued provider command capabilities only." },
{ "id": "gelios.command_dispatch.is_integration_command_intent", "from": ["gelios.command_dispatch"], "to": ["integration.command_intent"], "status": "future-concept", "summary": "Any future Gelios command dispatch must become an explicitly governed generic command intent." }, { "id": "gelios.command_dispatch.is_integration_command_intent", "from": ["gelios.command_dispatch"], "to": ["integration.command_intent"], "status": "future-concept", "summary": "Any future Gelios command dispatch must become an explicitly governed generic command intent." },
{ "id": "gelios.command_audit.is_integration_command_audit", "from": ["gelios.command_audit"], "to": ["integration.command_audit"], "status": "future-concept", "summary": "Gelios command audit specializes the generic red-domain audit contract." }, { "id": "gelios.command_audit.is_integration_command_audit", "from": ["gelios.command_audit"], "to": ["integration.command_audit"], "status": "future-concept", "summary": "Gelios command audit specializes the generic red-domain audit contract." },
{ "id": "gelios.integration.enforces_scope", "from": ["gelios.integration"], "to": ["gelios.access_scope"], "status": "product-required", "summary": "NDC L2 resolves selected safe-read capabilities and current provider visibility from the bound credential on every collection run." }, { "id": "gelios.integration.enforces_scope", "from": ["gelios.integration"], "to": ["gelios.access_scope"], "status": "product-required", "summary": "The Gateway applies an approved Robot2B scope before reading or publishing data." },
{ "id": "gelios.access_scope.includes_unit", "from": ["gelios.access_scope"], "to": ["gelios.unit", "gelios.unit_group"], "status": "product-required", "summary": "Scope dynamically includes every unit and group record returned by the selected capability; it contains no static entity allowlist." }, { "id": "gelios.access_scope.includes_unit", "from": ["gelios.access_scope"], "to": ["gelios.unit", "gelios.unit_group"], "status": "product-required", "summary": "An approved scope can include explicit units and approved group criteria, but group membership alone is not authoritative." },
{ "id": "gelios.unit.belongs_to_group", "from": ["gelios.unit"], "to": ["gelios.unit_group"], "status": "source-evidenced", "summary": "A provider unit can belong to one or more provider unit groups." }, { "id": "gelios.unit.belongs_to_group", "from": ["gelios.unit"], "to": ["gelios.unit_group"], "status": "source-evidenced", "summary": "A provider unit can belong to one or more provider unit groups." },
{ "id": "gelios.unit.uses_tracker_device", "from": ["gelios.unit"], "to": ["gelios.tracker_device"], "status": "source-evidenced", "summary": "A tracked unit is associated with a hardware tracker device." }, { "id": "gelios.unit.uses_tracker_device", "from": ["gelios.unit"], "to": ["gelios.tracker_device"], "status": "source-evidenced", "summary": "A tracked unit is associated with a hardware tracker device." },
{ "id": "gelios.unit.has_telemetry_snapshot", "from": ["gelios.unit"], "to": ["gelios.telemetry_snapshot"], "status": "product-required", "summary": "The versioned NDC L2 mapping produces normalized current telemetry for every valid returned unit." }, { "id": "gelios.unit.has_telemetry_snapshot", "from": ["gelios.unit"], "to": ["gelios.telemetry_snapshot"], "status": "product-required", "summary": "The Gateway maintains normalized current telemetry for an approved unit." },
{ "id": "gelios.raw_message.normalizes_to_snapshot", "from": ["gelios.raw_telemetry_message"], "to": ["gelios.telemetry_snapshot"], "status": "product-required", "summary": "Restricted raw provider telemetry can be normalized into the stable snapshot contract." }, { "id": "gelios.raw_message.normalizes_to_snapshot", "from": ["gelios.raw_telemetry_message"], "to": ["gelios.telemetry_snapshot"], "status": "product-required", "summary": "Restricted raw provider telemetry can be normalized into the stable snapshot contract." },
{ "id": "gelios.telemetry_snapshot.has_position_fix", "from": ["gelios.telemetry_snapshot"], "to": ["gelios.position_fix"], "status": "product-required", "summary": "A current telemetry snapshot can carry one time-qualified position fix." }, { "id": "gelios.telemetry_snapshot.has_position_fix", "from": ["gelios.telemetry_snapshot"], "to": ["gelios.position_fix"], "status": "product-required", "summary": "A current telemetry snapshot can carry one time-qualified position fix." },
{ "id": "gelios.telemetry_snapshot.has_operational_status", "from": ["gelios.telemetry_snapshot"], "to": ["gelios.operational_status"], "status": "product-required", "summary": "The semantic mapping derives normalized operational state from telemetry and data-quality policy." }, { "id": "gelios.telemetry_snapshot.has_operational_status", "from": ["gelios.telemetry_snapshot"], "to": ["gelios.operational_status"], "status": "product-required", "summary": "Gateway derives normalized operational state from telemetry and data-quality policy." },
{ "id": "gelios.telemetry_snapshot.has_parameter", "from": ["gelios.telemetry_snapshot"], "to": ["gelios.telemetry_parameter"], "status": "source-evidenced", "summary": "A provider snapshot can contain dynamic telemetry parameters." }, { "id": "gelios.telemetry_snapshot.has_parameter", "from": ["gelios.telemetry_snapshot"], "to": ["gelios.telemetry_parameter"], "status": "source-evidenced", "summary": "A provider snapshot can contain dynamic telemetry parameters." },
{ "id": "gelios.unit.has_sensor", "from": ["gelios.unit"], "to": ["gelios.sensor_definition"], "status": "source-evidenced", "summary": "A unit exposes zero or more sensor definitions." }, { "id": "gelios.unit.has_sensor", "from": ["gelios.unit"], "to": ["gelios.sensor_definition"], "status": "source-evidenced", "summary": "A unit exposes zero or more sensor definitions." },
{ "id": "gelios.sensor_definition.has_conversion", "from": ["gelios.sensor_definition"], "to": ["gelios.sensor_conversion"], "status": "source-evidenced", "summary": "A sensor can define conversion/calibration rows." }, { "id": "gelios.sensor_definition.has_conversion", "from": ["gelios.sensor_definition"], "to": ["gelios.sensor_conversion"], "status": "source-evidenced", "summary": "A sensor can define conversion/calibration rows." },
@ -29,14 +29,14 @@
{ "id": "gelios.geozone.belongs_to_group", "from": ["gelios.geozone"], "to": ["gelios.geozone_group"], "status": "source-evidenced", "summary": "A provider geozone can belong to a provider geozone group." }, { "id": "gelios.geozone.belongs_to_group", "from": ["gelios.geozone"], "to": ["gelios.geozone_group"], "status": "source-evidenced", "summary": "A provider geozone can belong to a provider geozone group." },
{ "id": "gelios.report_result.uses_template", "from": ["gelios.report_result"], "to": ["gelios.report_template"], "status": "source-evidenced", "summary": "A report result is produced from an approved report template and bounded request." }, { "id": "gelios.report_result.uses_template", "from": ["gelios.report_result"], "to": ["gelios.report_template"], "status": "source-evidenced", "summary": "A report result is produced from an approved report template and bounded request." },
{ "id": "gelios.collection_run.checkpoints_with_cursor", "from": ["gelios.collection_run"], "to": ["gelios.ingestion_cursor"], "status": "product-required", "summary": "Collection runs advance resumable ingestion only through a durable cursor." }, { "id": "gelios.collection_run.checkpoints_with_cursor", "from": ["gelios.collection_run"], "to": ["gelios.ingestion_cursor"], "status": "product-required", "summary": "Collection runs advance resumable ingestion only through a durable cursor." },
{ "id": "gelios.collection_run.collects_unit", "from": ["gelios.collection_run"], "to": ["gelios.unit"], "status": "product-required", "summary": "A safe-read collection run records every valid credential-visible unit returned by the selected capability." }, { "id": "gelios.collection_run.collects_unit", "from": ["gelios.collection_run"], "to": ["gelios.unit"], "status": "product-required", "summary": "A read-only collection run records the approved unit scope it addressed." },
{ "id": "gelios.unit.has_command_template", "from": ["gelios.unit"], "to": ["gelios.command_template"], "status": "source-evidenced", "summary": "Command templates are catalogued per unit through read-only routes." }, { "id": "gelios.unit.has_command_template", "from": ["gelios.unit"], "to": ["gelios.command_template"], "status": "source-evidenced", "summary": "Command templates are catalogued per unit through read-only routes." },
{ "id": "gelios.command_template.belongs_to_group", "from": ["gelios.command_template"], "to": ["gelios.command_group"], "status": "source-evidenced", "summary": "A command template can belong to a provider command group." }, { "id": "gelios.command_template.belongs_to_group", "from": ["gelios.command_template"], "to": ["gelios.command_group"], "status": "source-evidenced", "summary": "A command template can belong to a provider command group." },
{ "id": "gelios.command_dispatch.uses_template", "from": ["gelios.command_dispatch"], "to": ["gelios.command_template"], "status": "future-concept", "summary": "A future dispatch may reference a reviewed command template." }, { "id": "gelios.command_dispatch.uses_template", "from": ["gelios.command_dispatch"], "to": ["gelios.command_template"], "status": "product-required", "summary": "A dispatch may reference a reviewed command template." },
{ "id": "gelios.command_dispatch.targets_scope", "from": ["gelios.command_dispatch"], "to": ["gelios.unit", "gelios.unit_group", "gelios.access_scope"], "status": "future-concept", "summary": "A future dispatch target must resolve to an approved unit or explicitly approved group in scope." }, { "id": "gelios.command_dispatch.targets_scope", "from": ["gelios.command_dispatch"], "to": ["gelios.unit", "gelios.unit_group", "gelios.access_scope"], "status": "product-required", "summary": "A dispatch target must resolve to an approved unit or explicitly approved group in scope." },
{ "id": "gelios.command_dispatch.has_delivery", "from": ["gelios.command_dispatch"], "to": ["gelios.command_delivery"], "status": "future-concept", "summary": "A future dispatch is associated with delivery/task state rather than assumed successful on request creation." }, { "id": "gelios.command_dispatch.has_delivery", "from": ["gelios.command_dispatch"], "to": ["gelios.command_delivery"], "status": "product-required", "summary": "A dispatch is associated with delivery/task state rather than assumed successful on request creation." },
{ "id": "gelios.command_dispatch.has_audit", "from": ["gelios.command_dispatch"], "to": ["gelios.command_audit"], "status": "future-concept", "summary": "Every future dispatch requires immutable user-confirmation and outcome audit." }, { "id": "gelios.command_dispatch.has_audit", "from": ["gelios.command_dispatch"], "to": ["gelios.command_audit"], "status": "product-required", "summary": "Every dispatch requires immutable user-confirmation and outcome audit." },
{ "id": "gelios.unit.is_map_moving_object", "from": ["gelios.unit"], "to": ["map.moving_object"], "status": "product-required", "summary": "A valid credential-visible Gelios unit is exposed through the provider-neutral moving-object contract, never as a renderer entity identity." }, { "id": "gelios.unit.is_map_moving_object", "from": ["gelios.unit"], "to": ["map.moving_object"], "status": "product-required", "summary": "An approved Gelios unit is rendered through the provider-neutral moving-object contract, never as a Cesium entity identity." },
{ "id": "gelios.position_fix.positions_map_subject", "from": ["gelios.position_fix"], "to": ["map.moving_object"], "status": "product-required", "summary": "A normalized position fix updates the stable map subject bound to the Gelios unit." }, { "id": "gelios.position_fix.positions_map_subject", "from": ["gelios.position_fix"], "to": ["map.moving_object"], "status": "product-required", "summary": "A normalized position fix updates the stable map subject bound to the Gelios unit." },
{ "id": "gelios.geozone.is_map_zone", "from": ["gelios.geozone"], "to": ["map.zone"], "status": "product-required", "summary": "An approved Gelios geozone is exposed to the map through the generic zone contract." }, { "id": "gelios.geozone.is_map_zone", "from": ["gelios.geozone"], "to": ["map.zone"], "status": "product-required", "summary": "An approved Gelios geozone is exposed to the map through the generic zone contract." },
{ "id": "gelios.geopoint.is_map_place_target", "from": ["gelios.geopoint"], "to": ["map.place_target"], "status": "product-required", "summary": "An approved Gelios geopoint is exposed through a provider-neutral place target." } { "id": "gelios.geopoint.is_map_place_target", "from": ["gelios.geopoint"], "to": ["map.place_target"], "status": "product-required", "summary": "An approved Gelios geopoint is exposed through a provider-neutral place target." }

4
services/ontology-core/catalog/domain-packages/integration/aliases.json
View File

@ -1,6 +1,6 @@
{ {
"version": "1.0.0", "version": "0.1.0",
"updatedAt": "2026-07-16", "updatedAt": "2026-07-13",
"aliases": [ "aliases": [
{"alias":"коннектор","canonicalId":"integration.connection","language":"ru","status":"product-required"}, {"alias":"коннектор","canonicalId":"integration.connection","language":"ru","status":"product-required"},
{"alias":"connector","canonicalId":"integration.connection","language":"en","status":"product-required"}, {"alias":"connector","canonicalId":"integration.connection","language":"en","status":"product-required"},

32
services/ontology-core/catalog/domain-packages/integration/entities.json
View File

@ -1,21 +1,21 @@
{ {
"version": "1.0.0", "version": "0.1.0",
"updatedAt": "2026-07-16", "updatedAt": "2026-07-13",
"entities": [ "entities": [
{"id":"integration.provider","name":"External Provider","surface":"integration","status":["product-required"],"authority":"Versioned provider package + Ontology Core","summary":"A third-party product or API family integrated through NDC L2, not a tenant, runtime service or renderer."}, {"id":"integration.provider","name":"External Provider","surface":"integration","status":["product-required"],"authority":"Platform External Provider Data Plane","summary":"A third-party product or API family integrated through an app-owned adapter, not a tenant or a renderer."},
{"id":"integration.connection","name":"Provider Connection","surface":"integration","status":["product-required"],"authority":"NDC L2 connection","summary":"A tenant-scoped instance of one versioned provider package for one provider account; it contains policy and opaque credential references, never secret values."}, {"id":"integration.connection","name":"Provider Connection","surface":"integration","status":["product-required"],"authority":"Provider Gateway","summary":"A tenant-scoped configured connection to one provider account or endpoint; it contains policy and secret references, never secret values."},
{"id":"integration.credential_reference","name":"Provider Credential Reference","surface":"integration","status":["product-required"],"authority":"NDC L2 Credentials","summary":"Opaque reference to an NDC L2-managed provider credential lifecycle, not a credential value, login or password."}, {"id":"integration.credential_reference","name":"Provider Credential Reference","surface":"integration","status":["product-required"],"authority":"Provider Gateway / secret store","summary":"Opaque reference to a server-side secret or token lifecycle, not a token, login or password."},
{"id":"integration.access_scope","name":"Integration Access Scope","surface":"integration","status":["product-required"],"authority":"Provider credential visibility + NDC L2 connection profile","summary":"Selected safe-read capabilities and fields applied to every valid entity visible to the bound credential. Static entity allowlists and provider-group business filters are not collection scope."}, {"id":"integration.access_scope","name":"Integration Access Scope","surface":"integration","status":["product-required"],"authority":"Client owner + Provider Gateway","summary":"Approved product boundary for source accounts, objects, fields and permitted operations; provider visibility alone is insufficient."},
{"id":"integration.capability","name":"Provider Capability","surface":"integration","status":["product-required"],"authority":"Versioned provider package","summary":"Versioned provider operation or feature classified as read, metadata, write, destructive or unknown."}, {"id":"integration.capability","name":"Provider Capability","surface":"integration","status":["product-required"],"authority":"Provider Adapter","summary":"Versioned provider operation or feature classified as read, metadata, write, destructive or unknown."},
{"id":"integration.field_definition","name":"Provider Field Definition","surface":"integration","status":["product-required"],"authority":"Versioned provider package + Ontology Core","summary":"A documented source field, its semantics, sensitivity, units and normalisation mapping; it is not a live value."}, {"id":"integration.field_definition","name":"Provider Field Definition","surface":"integration","status":["product-required"],"authority":"Provider Adapter + Ontology Core","summary":"A documented source field, its semantics, sensitivity, units and normalisation mapping; it is not a live value."},
{"id":"integration.collection_profile","name":"Collection Profile","surface":"integration","status":["product-required"],"authority":"NDC L2 connection profile","summary":"Rate, paging, cursor, safe capability, field policy and batching selection for one connection; every valid entity returned for the credential is processed."}, {"id":"integration.collection_profile","name":"Collection Profile","surface":"integration","status":["product-required"],"authority":"Provider Gateway","summary":"Approved rate, paging, cursor, capability and field selection for a connection; prevents uncontrolled account-wide polling."},
{"id":"integration.retention_policy","name":"Integration Retention Policy","surface":"integration","status":["product-required"],"authority":"External Data Plane Data Product definition","summary":"Lifecycle rule for current state, normalised history, aggregates and any separately approved raw reference."}, {"id":"integration.retention_policy","name":"Integration Retention Policy","surface":"integration","status":["product-required"],"authority":"Client owner + Provider Gateway","summary":"Lifecycle rule for current state, raw envelope, normalised history, aggregates and backups."},
{"id":"integration.collection_run","name":"Integration Collection Run","surface":"integration","status":["product-required"],"authority":"NDC L2 workflow","summary":"Auditable bounded execution of a safe-read collection profile with credential-visible entity scope, cursor, response metrics and outcome."}, {"id":"integration.collection_run","name":"Integration Collection Run","surface":"integration","status":["product-required"],"authority":"Provider Gateway","summary":"Auditable bounded execution of a safe-read collection profile with cursor, response metrics and outcome."},
{"id":"integration.raw_envelope","name":"Raw Provider Envelope","surface":"integration","status":["product-required"],"authority":"Restricted raw-data policy","summary":"Optional provenance reference to a separately approved bounded raw layer; inline provider payload is not a Data Product publish field."}, {"id":"integration.raw_envelope","name":"Raw Provider Envelope","surface":"integration","status":["product-required"],"authority":"Provider Gateway","summary":"Restricted provenance-preserving representation of a safe-read provider response, stored in a bounded cold layer rather than exposed to UI."},
{"id":"integration.canonical_subject","name":"Canonical Integrated Subject","surface":"integration","status":["product-required"],"authority":"NDC L2 semantic mapping + Ontology Core","summary":"Stable domain subject normalised from a provider object and suitable for relations, analytics and interfaces."}, {"id":"integration.canonical_subject","name":"Canonical Integrated Subject","surface":"integration","status":["product-required"],"authority":"Provider Domain Adapter","summary":"Stable domain subject normalised from a provider object and suitable for relations, analytics and interfaces."},
{"id":"integration.read_model","name":"Integration Read Model","surface":"integration","status":["product-required"],"authority":"External Data Plane","summary":"Scoped, versioned Data Product current/history projection delivered to approved consumers without provider transport or raw bypass."}, {"id":"integration.read_model","name":"Integration Read Model","surface":"integration","status":["product-required"],"authority":"Provider Gateway","summary":"Scoped, versioned current, history or aggregate projection delivered to approved consumers without raw provider bypass."},
{"id":"integration.realtime_channel","name":"Integration Realtime Channel","surface":"integration","status":["product-required"],"authority":"External Data Plane","summary":"Bounded snapshot-and-patch contract for Data Product changes; its delivery and history cadence are independent from provider collection rate."}, {"id":"integration.realtime_channel","name":"Integration Realtime Channel","surface":"integration","status":["product-required"],"authority":"Provider Gateway","summary":"Internal bounded event or subscription contract for projection changes; its emission rate is independent from provider collection rate."},
{"id":"integration.command_capability","name":"Provider Command Capability","surface":"integration","status":["product-required"],"authority":"Versioned provider package","summary":"Catalogued command metadata, parameters and risk classification. It is not permission or a send route."}, {"id":"integration.command_capability","name":"Provider Command Capability","surface":"integration","status":["product-required"],"authority":"Provider Adapter","summary":"Catalogued command metadata, parameters and risk classification. It is not permission or a send route."},
{"id":"integration.command_intent","name":"Provider Command Intent","surface":"integration","status":["future-concept"],"authority":"Future Provider Command Gateway","summary":"A future explicitly confirmed, scoped and idempotent request to execute a catalogued command; no read collector may create it."}, {"id":"integration.command_intent","name":"Provider Command Intent","surface":"integration","status":["future-concept"],"authority":"Future Provider Command Gateway","summary":"A future explicitly confirmed, scoped and idempotent request to execute a catalogued command; no read collector may create it."},
{"id":"integration.command_audit","name":"Provider Command Audit","surface":"integration","status":["future-concept"],"authority":"Future Provider Command Gateway","summary":"Immutable approval, execution and delivery audit for a red-domain command lifecycle."} {"id":"integration.command_audit","name":"Provider Command Audit","surface":"integration","status":["future-concept"],"authority":"Future Provider Command Gateway","summary":"Immutable approval, execution and delivery audit for a red-domain command lifecycle."}
] ]

6
services/ontology-core/catalog/domain-packages/integration/evidence.json
View File

@ -2,11 +2,11 @@
"sourceRoots": ["platform/docs", "platform/packages", "platform/services"], "sourceRoots": ["platform/docs", "platform/packages", "platform/services"],
"ledgers": [ "ledgers": [
{ {
"id": "l2_owned_external_connectors_adr", "id": "external_provider_data_plane_adr",
"path": "../../docs/ADR_L2_OWNED_EXTERNAL_CONNECTORS.md", "path": "../../docs/ADR_EXTERNAL_PROVIDER_DATA_PLANE.md",
"entityIds": ["integration.provider", "integration.connection", "integration.access_scope", "integration.capability", "integration.collection_profile", "integration.read_model", "integration.command_capability"] "entityIds": ["integration.provider", "integration.connection", "integration.access_scope", "integration.capability", "integration.collection_profile", "integration.read_model", "integration.command_capability"]
} }
], ],
"baselineDocs": ["../../docs/ADR_L2_OWNED_EXTERNAL_CONNECTORS.md", "../../packages/external-provider-contract/README.md"], "baselineDocs": ["../../docs/ADR_EXTERNAL_PROVIDER_DATA_PLANE.md", "../../packages/external-provider-contract/README.md"],
"restrictions": ["The package is a semantic contract. Provider secrets, runtime data and command transport remain outside Ontology Core."] "restrictions": ["The package is a semantic contract. Provider secrets, runtime data and command transport remain outside Ontology Core."]
} }

10
services/ontology-core/catalog/domain-packages/integration/guardrails.json
View File

@ -1,11 +1,11 @@
{ {
"version": "1.0.0", "version": "0.1.0",
"updatedAt": "2026-07-16", "updatedAt": "2026-07-13",
"rules": [ "rules": [
{"id":"guardrail.integration.ontology_is_not_data_plane","severity":"error","summary":"Ontology Core may describe providers and contracts but must not store secret values, live payloads, tenant telemetry, raw envelopes or provider execution state.","entityIds":["integration.provider","integration.connection","integration.credential_reference","integration.raw_envelope","integration.read_model"]}, {"id":"guardrail.integration.ontology_is_not_data_plane","severity":"error","summary":"Ontology Core may describe providers and contracts but must not store secret values, live payloads, tenant telemetry, raw envelopes or provider execution state.","entityIds":["integration.provider","integration.connection","integration.credential_reference","integration.raw_envelope","integration.read_model"]},
{"id":"guardrail.integration.scope_precedes_collection","severity":"error","summary":"A collection run must resolve tenant/connection, safe capabilities, field policy and bounded profile before calling a provider, then process every valid entity returned for the bound credential. Static entity allowlists and provider-group business filters are forbidden at collection boundary.","entityIds":["integration.connection","integration.access_scope","integration.collection_profile","integration.collection_run"]}, {"id":"guardrail.integration.scope_precedes_collection","severity":"error","summary":"A collection run must resolve tenant scope, field policy and bounded collection profile before it addresses a provider. Account visibility does not define product scope.","entityIds":["integration.connection","integration.access_scope","integration.collection_profile","integration.collection_run"]},
{"id":"guardrail.integration.read_model_not_raw_bypass","severity":"error","summary":"NDC L2 workflows, interfaces and renderer adapters consume scoped Data Products or realtime channels, never provider credentials, raw envelopes or direct adapter database access.","entityIds":["integration.credential_reference","integration.raw_envelope","integration.read_model","integration.realtime_channel"]}, {"id":"guardrail.integration.read_model_not_raw_bypass","severity":"error","summary":"L2 workflows, interfaces and renderer adapters consume scoped read models or realtime channels, never provider credentials, raw envelopes or direct adapter database access.","entityIds":["integration.credential_reference","integration.raw_envelope","integration.read_model","integration.realtime_channel"]},
{"id":"guardrail.integration.commands_are_separate_red_domain","severity":"error","summary":"Cataloguing a command does not activate it. Read collectors, NDC L2 workflows, maps and AI Workspace may not create command intent or delivery. A later command boundary requires explicit human confirmation, scope, idempotency and audit.","entityIds":["integration.command_capability","integration.command_intent","integration.command_audit","integration.collection_run"]}, {"id":"guardrail.integration.commands_are_separate_red_domain","severity":"error","summary":"Cataloguing a command does not activate it. Read collectors, L2 workflows, maps and AI Workspace may not create command intent or delivery. A later command gateway requires explicit human confirmation, scope, idempotency and audit.","entityIds":["integration.command_capability","integration.command_intent","integration.command_audit","integration.collection_run"]},
{"id":"guardrail.integration.full_catalog_not_full_rate","severity":"warning","summary":"Every provider field may be catalogued, but enabling continuous collection requires an explicit collection, retention and rate policy. Do not turn a capability catalog into unbounded account polling.","entityIds":["integration.capability","integration.field_definition","integration.collection_profile","integration.retention_policy"]} {"id":"guardrail.integration.full_catalog_not_full_rate","severity":"warning","summary":"Every provider field may be catalogued, but enabling continuous collection requires an explicit collection, retention and rate policy. Do not turn a capability catalog into unbounded account polling.","entityIds":["integration.capability","integration.field_definition","integration.collection_profile","integration.retention_policy"]}
], ],
"blockedConflations": [ "blockedConflations": [

4
services/ontology-core/catalog/domain-packages/integration/package.json
View File

@ -1,7 +1,7 @@
{ {
"id": "integration", "id": "integration",
"version": "1.0.0", "version": "0.1.0",
"updatedAt": "2026-07-16", "updatedAt": "2026-07-13",
"status": "product-required/source-evidenced", "status": "product-required/source-evidenced",
"summary": "Provider-neutral external integration ontology: tenant-scoped connections, capabilities, collection, controlled storage projections and a separated red command domain." "summary": "Provider-neutral external integration ontology: tenant-scoped connections, capabilities, collection, controlled storage projections and a separated red command domain."
} }

8
services/ontology-core/catalog/domain-packages/integration/relations.json
View File

@ -1,13 +1,13 @@
{ {
"version": "1.0.0", "version": "0.1.0",
"updatedAt": "2026-07-16", "updatedAt": "2026-07-13",
"relations": [ "relations": [
{"id":"integration.provider.offers_connection","from":["integration.provider"],"to":["integration.connection"],"status":"product-required","summary":"A provider is connected through one or more tenant-scoped connection instances."}, {"id":"integration.provider.offers_connection","from":["integration.provider"],"to":["integration.connection"],"status":"product-required","summary":"A provider is connected through one or more tenant-scoped connection instances."},
{"id":"integration.connection.references_credential","from":["integration.connection"],"to":["integration.credential_reference"],"status":"product-required","summary":"A connection uses an opaque server-side credential reference rather than exposing a secret."}, {"id":"integration.connection.references_credential","from":["integration.connection"],"to":["integration.credential_reference"],"status":"product-required","summary":"A connection uses an opaque server-side credential reference rather than exposing a secret."},
{"id":"integration.connection.enforces_scope","from":["integration.connection"],"to":["integration.access_scope"],"status":"product-required","summary":"Every source request is bounded by selected safe capabilities and field policy while entity membership is dynamically all valid objects visible to the bound credential."}, {"id":"integration.connection.enforces_scope","from":["integration.connection"],"to":["integration.access_scope"],"status":"product-required","summary":"Every source request is bounded by an approved product scope."},
{"id":"integration.provider.declares_capability","from":["integration.provider"],"to":["integration.capability","integration.command_capability"],"status":"product-required","summary":"Adapter manifests catalogue source read and command capabilities independently from their activation."}, {"id":"integration.provider.declares_capability","from":["integration.provider"],"to":["integration.capability","integration.command_capability"],"status":"product-required","summary":"Adapter manifests catalogue source read and command capabilities independently from their activation."},
{"id":"integration.capability.exposes_field","from":["integration.capability"],"to":["integration.field_definition"],"status":"product-required","summary":"A safe-read capability documents the fields it may return before a collection profile enables them."}, {"id":"integration.capability.exposes_field","from":["integration.capability"],"to":["integration.field_definition"],"status":"product-required","summary":"A safe-read capability documents the fields it may return before a collection profile enables them."},
{"id":"integration.collection_profile.selects_capability","from":["integration.collection_profile"],"to":["integration.capability","integration.field_definition"],"status":"product-required","summary":"A profile selects bounded safe capabilities and fields rather than polling every API feature; it does not select business entity IDs."}, {"id":"integration.collection_profile.selects_capability","from":["integration.collection_profile"],"to":["integration.capability","integration.field_definition"],"status":"product-required","summary":"A profile selects bounded capabilities and fields rather than polling every API feature continuously."},
{"id":"integration.collection_profile.applies_retention","from":["integration.collection_profile"],"to":["integration.retention_policy"],"status":"product-required","summary":"Collection and storage lifecycle are explicit per connection profile."}, {"id":"integration.collection_profile.applies_retention","from":["integration.collection_profile"],"to":["integration.retention_policy"],"status":"product-required","summary":"Collection and storage lifecycle are explicit per connection profile."},
{"id":"integration.collection_run.uses_connection","from":["integration.collection_run"],"to":["integration.connection","integration.collection_profile"],"status":"product-required","summary":"A collection run records which connection and bounded policy it used."}, {"id":"integration.collection_run.uses_connection","from":["integration.collection_run"],"to":["integration.connection","integration.collection_profile"],"status":"product-required","summary":"A collection run records which connection and bounded policy it used."},
{"id":"integration.collection_run.records_raw_envelope","from":["integration.collection_run"],"to":["integration.raw_envelope"],"status":"product-required","summary":"Safe-read response provenance is retained only according to the active policy."}, {"id":"integration.collection_run.records_raw_envelope","from":["integration.collection_run"],"to":["integration.raw_envelope"],"status":"product-required","summary":"Safe-read response provenance is retained only according to the active policy."},

220
services/ontology-core/docs/GELIOS_DOMAIN_ONTOLOGY.md
View File

@ -2,202 +2,108 @@
Package: `catalog/domain-packages/gelios` Package: `catalog/domain-packages/gelios`
Status: `v1.0.0`, source-evidenced and product-required. Status: `v0.1.0` source-evidenced and product-required slice for the Robot2B client context and the Gelios provider.
## Purpose and boundary ## Purpose and boundary
The package describes Gelios fleet, telemetry, sensor, spatial, report and The package describes the meaning of Gelios fleet, telemetry, sensor, spatial, report and command concepts. It is not a Gelios client, token store, telemetry database, Cesium implementation or Engine workflow.
command meanings. It is not a provider client, credential store, runtime
database, renderer implementation, NDC L1 agent or NDC L2 workflow.
Gelios is the first live provider used to prove the common provider-package The practical audit established a wide REST read surface: 107 currently visible units, 105 with `lastMsg`, a visible administrative account, unit groups, telemetry variants, sensor catalogues, command-template catalogues, geozone groups and report templates. The REST OpenAPI documents 295 operations. This ontology therefore separates three things that must not be conflated:
model. Runtime transport, collection cadence and semantic mapping belong to an
isolated NDC L2 connection instance. External Data Plane persists and delivers
provider-neutral Data Products. Ontology Core describes meaning only.
The matching contract/data artifact is 1. documented provider capability;
`platform/packages/external-provider-contract/providers/gelios/v1`. Adding a 2. actual runtime read evidence for the current account;
second account creates another connection instance with different opaque 3. approved Robot2B product scope.
credential references; it does not create another ontology package, Platform
service or custom node.
## Provider authentication boundary The 95-unit legacy Engine snapshot is a temporary compatibility reference only. It is not the final business scope. The provider's current groups also do not resolve that scope automatically. A Robot2B owner must approve an allowlist or equivalent rule represented by `gelios.access_scope`.
Gelios issues exactly two provider secret artifacts: an access token and a
refresh token. The current NDC L2 `httpBearerAuth` request binding uses the
access token. Automatic refresh has not been proven in the deployed runtime, so
the matching provider package records refresh as `operator_managed`; neither
token value belongs in Ontology, a workflow graph, MCP, Ops or a trace.
A credential label such as `read access` is operator metadata, not a Gelios
token scope. `gelios.units.current.read` is classified as read because the
approved workflow transport is `GET /api/v1/units`. The label does not create a
separate read token or constrain other rights that Gelios may have granted to
the same access token. The scoped Data Product writer credential later in the
runtime path is an internal NDC/External Data Plane capability, not a third
Gelios token.
## Dynamic connection scope
`gelios.access_scope` is derived from two inputs on every collection run:
1. safe-read capabilities selected in the connection profile;
2. entities currently visible to the bound provider credential.
For `gelios.units.current.read`, NDC L2 processes every valid unit returned by
`GET /api/v1/units`. Static unit IDs, owner allowlists and provider-group business
filters are forbidden at collection boundary. New units visible to the same
credential appear automatically. Foundry and other Data Product consumers own
layer selection, sorting, filtering and visibility.
Technical limits such as a 5000-fact batch ceiling, retry policy and response
validation protect runtime health; they are not business entity filters.
## Canonical value contracts ## Canonical value contracts
### Unit and device ### Unit and device
`gelios.unit` is a stable provider subject. The published v1 source ID is `gelios.unit` is the stable business subject. The durable key is a namespaced provider identity such as `gelios.unit:<sourceUnitId>`, never a renderer entity ID or a display name. A unit can use a `gelios.tracker_device`, belong to multiple `gelios.unit_group` records and expose sensor, fuel, maintenance and custom-field configurations.
`gelios-unit-<sourceUnitId>` inside immutable `(providerId, connectionId)`
scope; a renderer entity ID or display name is never identity. The provider ID
suffix is preserved as an exact string (including leading zeros) and is never
parsed or renumbered. A unit can use a
`gelios.tracker_device`, belong to multiple `gelios.unit_group` records and
expose sensor, fuel, maintenance and custom-field configurations.
Hardware IDs, IMEI, phones, address, decrypt-related fields, raw `params` and Hardware IDs, IMEI, phones and decrypt-related fields are restricted or secret-class data. They are not part of the default Studio contract.
unclassified sensor payloads are excluded by the v1 field policy. New provider
fields remain dropped until they are evidenced, classified and introduced by a
new package/ontology revision.
### Current telemetry and position ### Current telemetry
`gelios.telemetry_snapshot` is normalized by the versioned NDC L2 mapping. `gelios.telemetry_snapshot` is the normalized current state produced by the Gateway, with at least:
`gelios.position_fix` is its time-qualified spatial portion; it is not a pin or
other renderer object.
The first approved output is exactly:
```text ```text
Data Product: fleet.positions.current.v1 unitSubjectId
version: 1.0.0 observedAt
ontology revision: ontology.map.moving_object.v1 receivedAt
semantic type: map.moving_object position: { latitude, longitude, height?, course?, speed?, satellites? }
delivery: snapshot+patch operationalStatus
history: sampled, latest entity per 60-second bucket, 90 days sensorReadings[]
counterValues[]
qualityFlags[]
source: gelios-rest
``` ```
Allowed Data Product fields are: The source supports a raw and a pure last-message variant. Pure data may include sensors, counters and accumulated fields. Raw messages and dynamic `params` remain behind restricted policy and may be retained separately for diagnostics only. `gelios.position_fix` is the time-qualified position portion used by spatial consumers; it is not itself a pin.
```text
course_degrees
display_name
elevation_meters
geometry
hdop
horizontal_accuracy_meters
object_kind
operational_status
position_source
position_valid
quality_flags
satellite_count
speed_kph
```
All attribute names are snake_case. GeoJSON `geometry` is omitted when valid
coordinates are unavailable, but the unit remains in the product with
`position_valid=false`, `operational_status=no_position` and appropriate
`quality_flags`. If the provider has no last-message timestamp, the collection
receive time becomes the explicit `observedAt` fallback so the credential-visible
unit is not silently dropped.
### Sensors and operational semantics ### Sensors and operational semantics
`gelios.sensor_definition`, `gelios.sensor_conversion` and `gelios.sensor_definition`, `gelios.sensor_conversion` and `gelios.sensor_reading` separate stable device configuration, calibration and time-series readings. Fuel profile, maintenance plan and custom field are independent configuration concepts. No assumed sensor meaning should be added to analytics until its message parameter, measure and conversion semantics are approved.
`gelios.sensor_reading` separate device configuration, calibration and
time-series values. Fuel profile, maintenance plan and custom field are
independent concepts. A sensor meaning cannot enter a Data Product until its
message parameter, unit, conversion, classification and field policy are
approved.
### Spatial binding ### Spatial binding and Cesium pins
Gelios is a source domain. The provider-neutral relation is: Gelios is a source domain; Cesium is a renderer adapter. The binding is deliberately provider-neutral:
```text ```text
gelios.unit + gelios.position_fix gelios.unit
-> map.moving_object -> map.moving_object (stable spatial subject)
-> fleet.positions.current.v1 gelios.position_fix
-> Foundry Data Product binding -> map.moving_object (current position update)
-> map layers, selection and telemetry panel map.moving_object
-> map.pin + map.label + map.visibility_rule
-> map.renderer_adapter (Cesium today, replaceable later)
``` ```
The publish boundary contains stable facts only. It cannot contain provider The future map payload selects a stable `subjectId`, position, normalized operational status, observed time and approved label fields. It must not contain Gelios credentials, raw payloads or Cesium transient entity references. This is the replacement boundary for the current M-map node-to-Cesium coupling.
credentials, tenant/connection claims supplied by the workflow, raw payloads,
pixel values, styles or transient renderer identities. The synthetic contract The Map Page resolves the selected Studio profile to the shared map.pin contract. The initial accepted presentation is an elevated spike — ground anchor, stem, outlined head and label — but that visual configuration remains on the Studio side. Gelios publishes data for a stable subject; it never publishes Cesium primitive settings, pin pixels or a renderer entity ID.
fixture is
[`examples/gelios-map-moving-object.fixture.json`](../examples/gelios-map-moving-object.fixture.json). The proposed renderer-neutral payload is in [`examples/gelios-map-moving-object.fixture.json`](../examples/gelios-map-moving-object.fixture.json). It uses synthetic coordinates only: it specifies the interface between Gateway and Map View, not a live trike position and not a Cesium entity.
### Commands ### Commands
`gelios.command_template` and `gelios.command_group` are read-only catalogue `gelios.command_template` and `gelios.command_group` are read-only catalogue concepts. `gelios.command_dispatch`, `gelios.command_delivery` and `gelios.command_audit` are a separate red domain.
concepts. `gelios.command_dispatch`, `gelios.command_delivery` and
`gelios.command_audit` are a separate red domain.
No collection run, workflow, map click or autonomous agent may create a No collection run, workflow, map click or autonomous agent may create a dispatch. A future dispatch requires an explicit human action, approved scope, selected unit/group and template/parameters, confirmation, an audit record and delivery-state reconciliation. This ontology package does not grant or test a write permission.
dispatch. A future command path requires explicit human intent, confirmation,
separate authorization, immutable audit and delivery reconciliation. This
ontology and the Gelios v1 provider package expose no command transport.
## Canonical runtime flow ## Future data plane
The intended flow is:
```text ```text
Gelios safe-read API Gelios REST OAuth
-> opaque Gelios access credential in NDC L2 Credentials -> server-side Gelios Gateway
-> isolated NDC L2 connection workflow -> scope filter + field policy + normalizer
-> versioned semantic mapping -> durable telemetry/configuration storage
-> NDC Data Product Publish through a native scoped writer credential -> NDC level-2 workflow / internal data contract
-> External Data Plane -> Map View binding / Cesium renderer adapter
-> scoped snapshot + patch stream
-> Foundry Data Product binding
``` ```
There is no Gelios-specific Platform service or database in this model. `gelios.collection_run` and `gelios.ingestion_cursor` define collection governance independently of the transport implementation. They support bounded incremental history ingestion later: one approved unit scope, limited time window, field whitelist, volume limit and durable checkpoint. They do not imply that all history should be collected now.
Provider transport never enters External Data Plane or Foundry. `gelios.collection_run`
and `gelios.ingestion_cursor` describe audit and resumability independently of Database technology is intentionally not chosen by Ontology Core. The proposed platform data-plane boundary and its validation gates are documented in [`platform/docs/ADR_GELIOS_DATA_PLANE.md`](../../../docs/ADR_GELIOS_DATA_PLANE.md). The ontology supplies the record boundaries needed by that ADR.
physical storage.
## Guardrails ## Guardrails
- Ontology Core never stores credentials, account state, runtime snapshots, - Ontology Core never stores secrets, runtime snapshots, full raw telemetry or large geometry datasets.
full raw telemetry or bulk geometry. - Current provider-account visibility is not the approved Robot2B scope; collection must enforce an owner-approved scope before every run.
- Gelios authentication has exactly access and refresh artifacts; credential - GET alone does not make a route safe: login-as, temporary tokens, configuration and WLN download are excluded from telemetry collection.
labels and workflow classifications must not be reinterpreted as additional - Geozones require paging, filtering, volume controls and map LOD.
provider token types or scopes. - Command send and mutation routes remain red even if the account has documented access.
- Every safe-read collection includes all valid entities returned for the - Map selection and pins target stable domain subjects, not provider payload IDs or renderer objects.
bound credential; unit allowlists and group filters are not collection
scope.
- A read method alone does not make a capability safe: impersonation,
temporary credential issuance, configuration mutation and downloads remain
excluded until separately classified.
- Dynamic provider fields are dropped until evidenced and classified.
- Geozones and history require bounded loading, paging/cursors, retention and
volume controls.
- Command send and mutation capabilities remain red even when provider access
exists.
- Foundry targets stable domain subjects and Data Product fields, not provider
payloads or renderer identities.
## Evidence and evolution ## Evidence and next implementation decisions
Evidence is limited to official Gelios API documentation, safe-read results, Evidence is limited to official REST OpenAPI, safe GET results, historical Engine donor inspection and the MAP package. The next decisions outside this ontology package are:
historical NDC L2 donor inspection and provider-neutral map contracts. The
Robot2B pilot counts (107 credential-visible units and a 95-unit legacy
snapshot) are retained only as historical evidence of response shape and
cardinality. They do not define canonical IDs, collection allowlists or product
scope.
Extension order is mandatory: establish safe provider evidence, extend 1. owner-approved Robot2B allowlist/rule;
capability/field classification, update ontology and mapping revision, add 2. field policy for Studio, analytics and raw retention;
synthetic fixtures and contract tests, then deploy an updated NDC L2 workflow. 3. data-plane ADR for realtime telemetry and history;
4. Gelios Gateway implementation boundary and deployment location;
5. fixture and renderer-adapter contract for Map View pins and labels.

52
services/ontology-core/examples/gelios-map-moving-object.fixture.json
View File

@ -1,33 +1,25 @@
{ {
"schemaVersion": "nodedc.data-product.publish/v1", "contractVersion": "0.1.0",
"batch": { "kind": "map.moving_object.current_position",
"runId": "gelios-ontology-fixture-20260716", "source": "gelios",
"sequence": 0, "subjectId": "gelios.unit:example-unit-id",
"idempotencyKey": "gelios-ontology-fixture-20260716.batch-0" "observedAt": "2026-07-13T00:00:00.000Z",
"receivedAt": "2026-07-13T00:00:03.000Z",
"position": {
"latitude": 55.000001,
"longitude": 37.000001,
"heightMeters": 0,
"courseDegrees": 90,
"speedKph": 0,
"satellites": 0
}, },
"facts": [ "operationalStatus": "unknown",
{ "display": {
"sourceId": "gelios-unit-1001", "label": "Robot2B unit",
"semanticType": "map.moving_object", "visibilityProfile": "fleet-default"
"observedAt": "2026-07-16T12:00:00.000Z", },
"geometry": { "policy": {
"type": "Point", "scopeApproved": true,
"coordinates": [37.6173, 55.7558] "fieldSet": "studio-current-position-v1"
}, }
"attributes": {
"course_degrees": 90,
"display_name": "Synthetic Gelios unit",
"elevation_meters": 0,
"hdop": 0.8,
"horizontal_accuracy_meters": 4.2,
"object_kind": "tracked_unit",
"operational_status": "active",
"position_source": "gelios",
"position_valid": true,
"quality_flags": [],
"satellite_count": 11,
"speed_kph": 0
}
}
]
} }