Compare commits
4 Commits
567f1550ab
...
31e078d6e5
| Author | SHA1 | Date |
|---|---|---|
Codex
|
31e078d6e5 | |
|
Codex
|
2dd6e33a54 | |
|
Codex
|
a0a4d36fa2 | |
|
Codex
|
3415674e76 |
|
|
@ -49,4 +49,4 @@ Ontology Core живёт в `services/ontology-core` как docs-first сема
|
|||
|
||||
Map Gateway живёт в `services/map-gateway` как общий platform boundary для provider credentials, безопасного asset endpoint exchange, tile/3D Tiles proxy и persistent offline TileCache. Runtime cache не является source artifact и хранится в named volume/object storage, а не в Git.
|
||||
|
||||
Внешние бизнес-поставщики подключаются по `packages/external-provider-contract`: adapter конкретного API принадлежит изолированному NDC Agent L2 workflow, а Platform даёт один provider-neutral External Data Plane для raw retention, canonical facts, current/history projections и scoped read products. Connection instance несёт tenant scope, credential reference и collection profile, но не secret value. Provider/domain mapping, entity filtering и renderer logic не попадают в Platform Data Plane; writer capability привязана к immutable EDP binding и выдаётся только отдельному Engine provisioner, не shared internal bearer. Подробный канон — `docs/ADR_L2_OWNED_EXTERNAL_CONNECTORS.md`.
|
||||
Внешние бизнес-поставщики подключаются по `packages/external-provider-contract`: adapter конкретного API принадлежит изолированному NDC L2 workflow, а Platform даёт один provider-neutral External Data Plane для raw retention, canonical facts, current/history projections и scoped read products. Connection instance несёт tenant scope, credential reference и collection profile, но не secret value. Provider/domain mapping, entity filtering и renderer logic не попадают в Platform Data Plane. Provider-issued credentials сохраняются в native NDC L2 Credentials; внутренний scoped publish grant генерируется и сохраняется generic control plane внутри native credential boundary, EDP получает только digest, а MCP видит только opaque reference/status. Legacy plaintext provisioning и managed digest-only ensure включаются независимо и по умолчанию закрыты. Подробный канон — `docs/ADR_L2_OWNED_EXTERNAL_CONNECTORS.md`.
|
||||
|
|
|
|||
|
|
@ -1,274 +1,238 @@
|
|||
# ADR: L2-owned external connectors and provider-neutral Data Plane
|
||||
# ADR: внешние коннекторы принадлежат NDC L2 workflow
|
||||
|
||||
Статус: **accepted**.
|
||||
Дата: 2026-07-14.
|
||||
Дата: 2026-07-16.
|
||||
Владелец решения: NODE.DC Platform.
|
||||
|
||||
## Контекст
|
||||
|
||||
NODE.DC — платформа разработки. Подключение внешнего API не должно создавать
|
||||
ещё один provider-specific сервис, в котором навсегда зашиты логика клиента,
|
||||
нормализация и правила отображения. Иначе каждый новый account или новый тип
|
||||
объекта превращается в изменение Platform runtime.
|
||||
NODE.DC состоит из двух разных слоёв:
|
||||
|
||||
Нужна одна повторяемая форма: Platform даёт безопасные границы, контракт
|
||||
хранения и выдачи данных; изолированный workflow NDC Agent L2 является
|
||||
адаптером конкретного API и остаётся редактируемым через предоставленный
|
||||
Codex/Engine MCP.
|
||||
- платформенный слой даёт NDC L1, NDC L2, Ontology, credentials, Data Products
|
||||
и Foundry-boundaries;
|
||||
- пользовательские автоматизации собирают из этих возможностей конкретную
|
||||
бизнес-логику.
|
||||
|
||||
Это решение заменяет provider-adapter часть
|
||||
`ADR_EXTERNAL_PROVIDER_DATA_PLANE.md` и все provider-specific runtime
|
||||
предписания `ADR_GELIOS_DATA_PLANE.md`. Они сохраняются как исторические
|
||||
черновики, но не являются каноном для новой реализации.
|
||||
Новый account, provider credential, tenant, расписание или интерфейс не должны создавать
|
||||
новый platform service и не должны требовать изменения NODE.DC source. Команда
|
||||
NODE.DC подключается только для нового поставщика либо для расширения
|
||||
подтверждённых capabilities и ontology уже поддержанного поставщика.
|
||||
|
||||
Gelios — первый живой источник и acceptance-кейс этого канона, а не отдельная
|
||||
архитектурная ветка. Документ заменяет provider-runtime решения из
|
||||
`ADR_EXTERNAL_PROVIDER_DATA_PLANE.md` и `ADR_GELIOS_DATA_PLANE.md`; те
|
||||
документы остаются только историей решения.
|
||||
|
||||
## Решение
|
||||
|
||||
### 1. Адаптер внешнего API принадлежит L2
|
||||
### 1. Пакет поставщика создаётся один раз
|
||||
|
||||
Один изолированный L2 workflow представляет одно connection instance и
|
||||
является единственным местом для:
|
||||
Каждый поддержанный поставщик получает один версионируемый **provider package**.
|
||||
Он является знанием платформы о внешнем API и содержит:
|
||||
|
||||
- обращения к API поставщика через ссылку на credential из Engine;
|
||||
- выбора read-capability, cursor/pagination, rate limit, retry и расписания;
|
||||
- получения raw envelope, разбиения большого ответа на native batch-операции;
|
||||
- семантического mapping к версии ontology и формирования canonical facts;
|
||||
- idempotency key, watermark и lifecycle источника;
|
||||
- deploy/run/execution/trace через Engine MCP.
|
||||
- стабильный `providerId` и версии подтверждённого API;
|
||||
- credential contract без значения секрета;
|
||||
- каталог safe-read capabilities, pagination/rate-limit metadata и response
|
||||
shapes;
|
||||
- mappings из provider fields в версии Ontology;
|
||||
- совместимые Data Products и scrubbed contract fixtures;
|
||||
- явно отделённый каталог команд без активного command transport.
|
||||
|
||||
L2 не хранит значение provider secret или writer token: он использует только
|
||||
Engine-bound credential references. Plaintext writer token выдаётся trusted
|
||||
provisioner ровно один раз при create/rotate binding и затем хранится только в
|
||||
Engine credential store, привязанном к этому L2; External Data Plane хранит
|
||||
лишь его hash. Provisioner читает отдельный runner-owned secret file, который
|
||||
read-only монтируется только в EDP и позднее в dedicated Engine provisioner,
|
||||
работающий под выделенным UID/GID `11006`; это не shared `.env`, не
|
||||
`NODEDC_INTERNAL_ACCESS_TOKEN` и не provider credential. До deployment этого
|
||||
provisioner `EXTERNAL_DATA_PLANE_PROVISIONING_ENABLED=false`, поэтому issuance
|
||||
routes fail closed. Token не попадает в L2 graph/profile, contract artifact,
|
||||
execution log/trace, raw envelope, Foundry или UI. Command/write-capabilities
|
||||
не получают transport route в read workflow.
|
||||
Пакет не является runtime service, scheduler, базой данных или customer
|
||||
configuration. Он расширяется только когда фактически подтверждён новый API,
|
||||
новый тип данных или новая ontology revision.
|
||||
|
||||
Для новой учётной записи создаётся новый instance/profile этого workflow с
|
||||
другой credential reference и connection configuration. Это не требует
|
||||
изменения Platform source. Если поставщик раскрывает новый объект или поле,
|
||||
сначала расширяется ontology/capability contract, затем изменяется именно
|
||||
соответствующий L2 workflow.
|
||||
Одна учётная запись поставщика задаётся данными, а не кодом:
|
||||
|
||||
### 2. Platform Data Plane нейтрален к provider и домену
|
||||
`provider package + credential reference + connection profile + NDC L2 workflow instance`
|
||||
|
||||
Platform предоставляет один versioned **External Data Plane**. Он принимает
|
||||
batch по внутреннему аутентифицированному контракту, хранит raw envelope с
|
||||
retention/provenance, canonical facts, current/history projections и отдаёт
|
||||
scoped data products/realtime updates. В нём запрещены:
|
||||
Поэтому второй account или другая provider credential pair создаёт ещё один credential/profile и
|
||||
workflow instance. Платформенный source, Data Plane и Foundry при этом не
|
||||
меняются.
|
||||
|
||||
- ветвления по названию provider, customer, account, vehicle или renderer;
|
||||
- provider field mapping, unit allowlist, business filtering и visual rules;
|
||||
- provider token, прямой browser-to-provider доступ и command transport.
|
||||
### 2. NDC L1 проектирует, NDC L2 workflow исполняет
|
||||
|
||||
Его canonical write-форма, которую Data Plane валидирует и сохраняет после
|
||||
авторизации, содержит только нейтральные элементы:
|
||||
NDC L1 получает пользовательское намерение, проверяет доступные capabilities и
|
||||
Ontology, проектирует или изменяет разрешённый NDC L2 workflow через NDC MCP и
|
||||
анализирует execution evidence.
|
||||
|
||||
```text
|
||||
source: providerId + tenantId + connectionId
|
||||
contract: contractVersion + ontologyRevision + dataProductId
|
||||
batch: runId + sequence + idempotencyKey + receivedAt
|
||||
raw: optional restricted ref + contentType + hash + retention policy
|
||||
facts: stable sourceId + semanticType + observedAt + attributes + geometry?
|
||||
```
|
||||
Один изолированный NDC L2 workflow представляет один connection instance и
|
||||
владеет исполняемой механикой:
|
||||
|
||||
Для нового L2 writer-а это не wire-форма. Он вызывает
|
||||
`POST /internal/data-plane/v1/data-products/:dataProductId/publish` с bearer
|
||||
writer token и формой `nodedc.data-product.publish/v1`, в которой вообще нет
|
||||
`source`, contract metadata, transport или persistence policy. Любой caller
|
||||
scope и scope headers запрещены.
|
||||
- safe-read вызовами provider API через credential reference;
|
||||
- pagination, cursor, retry, rate limit, batch size и collection cadence;
|
||||
- проверкой response shape и разбиением ответа на items;
|
||||
- mapping к точной ontology revision;
|
||||
- idempotency, watermark и публикацией canonical facts;
|
||||
- private workflow state, когда он действительно нужен.
|
||||
|
||||
External Data Plane разрешает token в собственный immutable writer binding
|
||||
`(tenantId, connectionId, providerId, allowedDataProductIds, expiresAt)`,
|
||||
проверяет active/non-expired binding, совпадение `providerId` и разрешение
|
||||
`dataProductId`, затем сам materializes canonical `source` с tenant/connection
|
||||
и только после этого валидирует и сохраняет batch. Присланный caller scope
|
||||
отклоняется, а не merge/override-ится. Изменение tenant, connection, provider,
|
||||
allowed data products или срока требует нового binding; для существующего
|
||||
binding допустимы только token rotation и revoke.
|
||||
Provider-specific mapping сначала реализуется штатными workflow nodes и малым
|
||||
Code-преобразователем. Это позволяет проверить реальный API без преждевременного
|
||||
создания custom nodes. В custom NDC nodes переносится только повторившаяся и
|
||||
доказанная **платформенная boundary-механика**, а не уникальная логика
|
||||
поставщика.
|
||||
|
||||
Выпуск, rotation и revoke binding принимаются только от dedicated provisioning
|
||||
principal с runner-owned secret file; он не монтируется в shared `.env` или L2.
|
||||
Shared legacy bearer не может создавать writer capability.
|
||||
### 3. Credentials являются данными connection instance
|
||||
|
||||
Идентификатор источника стабилен в пределах `(providerId, connectionId,
|
||||
sourceId)`. Отсутствие объекта в очередном ответе помечается lifecycle-state
|
||||
или временем последнего наблюдения; запись и её история не удаляются.
|
||||
Gelios выдаёт ровно два provider secrets: access token и refresh token. Текущий
|
||||
HTTP request binding `httpBearerAuth` использует access token. Автоматический
|
||||
refresh в принятом runtime не доказан, поэтому provider package фиксирует его
|
||||
как `operator_managed`; ни один из token values не попадает в graph, package,
|
||||
Ontology, Data Plane, Foundry, execution logs, MCP или Ops. Новый account или
|
||||
provider-issued token pair означает новые native credential records и
|
||||
connection instance, но не новую Platform-сущность.
|
||||
|
||||
### 3. Граница данных и интерфейса
|
||||
Локальное имя credential (например, с суффиксом `read access`) не является
|
||||
provider scope. В target-контракте Read-классификацию задаёт allowlisted
|
||||
method/endpoint в capability catalog и Engine workflow policy; тот же access
|
||||
token нельзя называть отдельным «read token» или «write token» только из-за
|
||||
label. Сейчас deployed Engine safe-ref policy ограничивает generic HTTP
|
||||
credential только по host, но ещё не связывает его с package/version и exact
|
||||
method/path. Поэтому Gelios `GET /api/v1/units` пока является декларативной
|
||||
capability, а не завершённой runtime-security гарантией; canonical acceptance
|
||||
требует capability-bound Engine policy и её MCP proof.
|
||||
|
||||
```text
|
||||
Provider API (safe read)
|
||||
-> Engine provider-credential reference
|
||||
-> L2 connector instance (adapter + mapping + batches)
|
||||
-> Engine writer-credential reference
|
||||
-> External Data Plane Data Product publish (scope materialized server-side)
|
||||
-> External Data Plane (generic persistence + projections)
|
||||
-> scoped data product / realtime stream
|
||||
-> Foundry binding / user interface
|
||||
```
|
||||
Writer capability для публикации Data Product — отдельный внутренний native NDC
|
||||
L2 credential, не третий Gelios token. Generic Engine/Platform control plane
|
||||
генерирует capability внутри native credential boundary, сохраняет её там же и
|
||||
передаёт EDP только SHA-256 digest вместе с точным provider, connection и
|
||||
набором Data Products. Пользователь и MCP получают только opaque
|
||||
reference/status. Capability не записывается в graph, provider package, env,
|
||||
file, Ops или trace и не копируется через пользовательский UI.
|
||||
|
||||
Ontology Core описывает semantic types, связи, capability catalog и mapping
|
||||
versions, но не хранит runtime payloads или секреты. Foundry получает только
|
||||
approved data product и решает presentation: pins, visibility, layers and
|
||||
filters. Оно не получает provider endpoint или credential.
|
||||
Широкий credential sink, resolver/daemon с произвольной записью secret и любой
|
||||
provider-specific credential service запрещены. Нужна одна узкая операция
|
||||
`ensure data-product publish grant`: scope выводится из granted L1→L2 target,
|
||||
зарегистрированного connection profile и разрешённого Data Product; issuance,
|
||||
rotation и native binding должны быть idempotent, CAS/crash-safe и auditable.
|
||||
Engine генерирует capability внутри credential boundary и передаёт EDP только
|
||||
SHA-256 digest через idempotent binding key + generation; EDP не возвращает
|
||||
plaintext. Новая generation создаётся до переключения node reference, а старая
|
||||
отзывается только после успешного bind/acceptance.
|
||||
Старые ручные EDP `POST/rotate` endpoints, которые возвращают capability один
|
||||
раз, включаются отдельным legacy-флагом и не входят в canonical acceptance.
|
||||
Digest-only managed endpoint имеет независимый флаг и принимает только
|
||||
Ed25519-signed Engine service requests: exact audience/method/request target/raw
|
||||
body hash входят в подпись, timestamp ограничен по skew, nonce защищён bounded
|
||||
fail-closed replay cache. Legacy provisioner bearer на managed routes не
|
||||
действует. EDP получает только deployment public key; matching private key
|
||||
остаётся только внутри Engine server boundary. Endpoint остаётся выключенным,
|
||||
пока key provisioning, Engine signer и exact native credential binding policy
|
||||
не пройдут runtime acceptance.
|
||||
В deployed Engine MCP этой операции пока нет — это текущий platform gap перед
|
||||
canonical publish proof, а не действие пользователя. Это решение не заявляет
|
||||
автоматический refresh Gelios: до отдельного runtime proof он остаётся
|
||||
`operator_managed`.
|
||||
|
||||
### 4. Полнота без entity allowlist
|
||||
### 4. External Data Plane нейтрален к поставщику
|
||||
|
||||
Collection profile выбирает разрешённые **read-capabilities**, а не список
|
||||
конкретных объектов. Если выбранный read endpoint возвращает все доступные
|
||||
источнику сущности, L2 передаёт все валидные элементы. Новые сущности
|
||||
добавляются автоматически; скрытие/отображение — задача data product/UI, а не
|
||||
сбора. Технические лимиты существуют только как размер batch, backpressure,
|
||||
quota и защита от повреждённого ответа, но не как бизнес-фильтр по ID.
|
||||
Platform предоставляет один versioned External Data Plane. Он принимает
|
||||
canonical facts через scoped writer binding, хранит current/history
|
||||
projections и публикует snapshot/patch contracts. В нём запрещены:
|
||||
|
||||
### 5. Масштабирование и native NDC Agent nodes
|
||||
- ветвления по provider, customer, account, entity или renderer;
|
||||
- provider field mapping и бизнес-фильтрация;
|
||||
- provider credential, endpoint, schedule или command transport;
|
||||
- caller-supplied tenant/connection scope.
|
||||
|
||||
L2 не запускает отдельный execution на каждую сущность. Он использует native
|
||||
nodes NDC Agent: HTTP Request, Split Out, Split In Batches/Loop Over Items,
|
||||
Aggregate, Postgres (когда нужен private workflow state), Set/If/Merge,
|
||||
Schedule Trigger and Respond to Webhook. Code node допустим только как малый
|
||||
преобразователь boundary-shape, когда эквивалентной native node нет; он не
|
||||
становится скрытым сервисом или provider database.
|
||||
`NDC Data Product Publish` отправляет только
|
||||
`nodedc.data-product.publish/v1`. External Data Plane materializes immutable
|
||||
scope из writer binding, проверяет разрешённый Data Product и его ontology
|
||||
revision, затем сохраняет batch.
|
||||
|
||||
Shared telemetry/history не записывается L2 напрямую в физические таблицы
|
||||
Platform Postgres: это создало бы coupling к schema. Direct Postgres допустим
|
||||
для private state конкретного workflow или отдельной project-owned базы.
|
||||
Общий контур использует только External Data Plane contract и bulk batches.
|
||||
EDP runtime импортирует только package subpath
|
||||
`@nodedc/external-provider-contract/data-plane`. Его deploy artifact и image
|
||||
содержат только provider-neutral wire validators; `providers/gelios`, mappings,
|
||||
fixtures и tests туда не входят. Изменение или добавление provider package не
|
||||
пересобирает и не перезапускает EDP: каталог поставщиков разворачивается через
|
||||
Engine/Ontology/control-plane путь отдельно.
|
||||
|
||||
### 5.1. Каталог custom nodes NODE.DC
|
||||
Collection cadence, history cadence и presentation cadence независимы:
|
||||
|
||||
Встроенные узлы runtime сохраняют свои штатные названия (`HTTP Request`,
|
||||
`Schedule Trigger`, `Loop Over Items` и т. п.). Любой узел, код которого
|
||||
принадлежит NODE.DC, обязан одновременно выполнять три условия:
|
||||
- NDC L2 забирает источник с частотой, нужной бизнес-задаче;
|
||||
- current projection принимает каждое валидное изменение;
|
||||
- declarative history policy может хранить все точки или sampling;
|
||||
- Foundry читает snapshot+patch и отдельно ограничивает частоту render.
|
||||
|
||||
- видимое имя начинается с `NDC `;
|
||||
- runtime type принадлежит package namespace `n8n-nodes-ndc.*`;
|
||||
- package-level test отклоняет публикацию узла без этого префикса и namespace.
|
||||
Для разной частоты БД и интерфейса не создаются второй provider sink, отдельный
|
||||
gateway или параллельный прямой push в renderer.
|
||||
|
||||
Первый канонический набор состоит из:
|
||||
### 5. Полнота означает все сущности разрешённой capability
|
||||
|
||||
- `NDC Data Product Publish` — runtime write boundary L2 → Data Plane;
|
||||
- `NDC Data Product Read` — scoped current snapshot read по reader grant;
|
||||
- `NDC Foundry Binding` — deploy/control-plane связь Data Product с
|
||||
`Application → Page → typed slot`, но не транспорт каждого realtime tick.
|
||||
Connection profile выбирает safe-read capabilities, а не зашитый в Platform
|
||||
список entity IDs. Если разрешённый endpoint возвращает все доступные credential
|
||||
сущности, NDC L2 обрабатывает каждый валидный item. Новый объект появляется
|
||||
автоматически.
|
||||
|
||||
`NDC Foundry Output` не используется как техническое имя: оно ошибочно
|
||||
подразумевает прямой transport L2 → renderer. Provider-specific adapter,
|
||||
mapping и collection profile остаются логикой конкретного L2 workflow на
|
||||
штатных nodes; они не оформляются как custom NODE.DC nodes и не добавляют
|
||||
provider branch в Data Plane или Foundry.
|
||||
Пользовательская фильтрация, слои и видимость находятся после сбора — в
|
||||
automation/Data Product/Foundry. Технические ограничения допустимы только как
|
||||
pagination, quota, batch size, backpressure и защита от повреждённого ответа.
|
||||
|
||||
`NDC Foundry Binding` отправляет только
|
||||
`nodedc.foundry.binding-upsert/v1`. Её постоянный L2 credential — отдельный
|
||||
revocable `ndc_fndbg_*` workload grant с allowlist точных
|
||||
`Application → Page → Binding → Slot → Data Product` targets. Короткоживущая
|
||||
`fnd1.*` capability интерактивного Foundry MCP, browser session, EDP
|
||||
writer/reader grant и `NODEDC_INTERNAL_ACCESS_TOKEN` для этой ноды запрещены.
|
||||
### 6. Custom NDC nodes ограничены платформенными границами
|
||||
|
||||
Runtime node не содержит URL, endpoint, provider ID, tenant ID, connection ID,
|
||||
token, contract version или history cadence. Узел выбирает только разрешённый
|
||||
Data Product; всё остальное materializes из opaque writer/reader binding и
|
||||
immutable Data Product definition.
|
||||
Первый канонический набор:
|
||||
|
||||
Package `n8n-nodes-ndc` устанавливается как штатный private community package в
|
||||
`/home/node/.n8n/nodes/node_modules`, чтобы n8n `PackageDirectoryLoader`
|
||||
сохранил namespace `n8n-nodes-ndc.*`. `~/.n8n/custom` и
|
||||
`N8N_CUSTOM_EXTENSIONS` запрещены для этого пакета: они создают namespace
|
||||
`CUSTOM.*`. Ручное копирование или `npm install` в живом container, patch
|
||||
Engine core и подмена built-in node запрещены. Production использует
|
||||
проверенный offline tarball, immutable release, atomic current/previous switch,
|
||||
read-only mount и restart всех n8n execution processes; acceptance завершается
|
||||
только когда Engine schema и MCP catalog возвращают точные package-qualified
|
||||
types.
|
||||
- `NDC Data Product Publish` — NDC L2 → External Data Plane;
|
||||
- `NDC Data Product Read` — scoped snapshot/patch read;
|
||||
- `NDC Foundry Binding` — control-plane связь Data Product с
|
||||
`Application → Page → typed slot`.
|
||||
|
||||
### 5.2. Realtime delivery contract
|
||||
Эти nodes не содержат provider ID, tenant ID, connection ID, endpoint, token,
|
||||
mapping или cadence. Runtime package сохраняет технический namespace
|
||||
`n8n-nodes-ndc.*`, но пользовательские документы и интерфейсы используют
|
||||
только терминологию NDC.
|
||||
|
||||
Новый writer использует:
|
||||
`NDC Foundry Binding` не транспортирует каждый realtime tick. Он создаёт
|
||||
постоянную связь интерфейса с Data Product; Foundry затем читает его
|
||||
snapshot+patch contract.
|
||||
|
||||
```text
|
||||
POST /internal/data-plane/v1/data-products/:dataProductId/publish
|
||||
Authorization: Bearer <opaque writer binding>
|
||||
```
|
||||
### 7. Реальный Gelios acceptance-кейс
|
||||
|
||||
Body имеет schema `nodedc.data-product.publish/v1` и содержит только batch
|
||||
identity и canonical facts. Data Plane одной транзакцией:
|
||||
Первая версия provider package должна доказать путь:
|
||||
|
||||
1. проверяет idempotency;
|
||||
2. обновляет current projection только более новым или изменившимся фактом;
|
||||
3. применяет declarative history policy (`none`, `all`, `sampled`);
|
||||
4. добавляет changed facts в durable patch outbox;
|
||||
5. фиксирует монотонный cursor.
|
||||
`Gelios safe read → все доступные units → map.moving_object facts →
|
||||
fleet.positions.current.v1 → Foundry map`
|
||||
|
||||
Reader grant получает snapshot `nodedc.data-product.snapshot/v1`, затем
|
||||
подключается к SSE stream с `after=<snapshot.cursor>` или `Last-Event-ID`.
|
||||
Каждый outbox event имеет schema `nodedc.data-product.patch/v1`. Если cursor
|
||||
уже удалён retention policy, stream отвечает `409 resync_required`, и consumer
|
||||
повторяет snapshot. Browser не получает Data Plane token или scope: Foundry BFF
|
||||
разрешает persisted binding и читает server-only reader grant.
|
||||
Canonical product использует ontology revision
|
||||
`ontology.map.moving_object.v1` и только объявленные snake_case поля. NDC L2
|
||||
не передаёт caller scope и не собирает собственный batch envelope вокруг
|
||||
`NDC Data Product Publish`.
|
||||
|
||||
`snapshot+patch` в v1 является **bounded product contract**. Current projection
|
||||
одного scoped Data Product не может содержать больше 5000 entity keys
|
||||
`(sourceId, semanticType)`. Snapshot возвращает целую согласованную projection
|
||||
и один barrier cursor из одной `REPEATABLE READ` transaction. Query-параметр
|
||||
`limit` — только защитный потолок, а не размер страницы: если полная projection
|
||||
не помещается, Data Plane отвечает `413 data_product_snapshot_limit_exceeded`
|
||||
и consumer не начинает patch stream с неполной базой.
|
||||
Acceptance выполняется по порядку:
|
||||
|
||||
Наивная pagination current snapshot по `sourceId`, offset или независимо
|
||||
полученным page cursors запрещена: изменения между страницами могут быть
|
||||
пропущены или продублированы относительно patch cursor. Product с ожидаемой
|
||||
cardinality выше 5000 обязан **до включения realtime** выбрать один из двух
|
||||
отдельно версионируемых контрактов:
|
||||
1. Platform выполняет generic `ensure data-product publish grant`, выпускает
|
||||
scoped EDP binding и атомарно сохраняет capability в native NDC L2
|
||||
Credentials;
|
||||
2. NDC MCP видит только совместимый opaque writer credential reference/status;
|
||||
3. применить подтверждённый graph patch без legacy intake;
|
||||
4. validate/preflight;
|
||||
5. один успешный manual run и проверка execution/trace/Data Product;
|
||||
6. только затем отдельным изменением добавить Schedule Trigger;
|
||||
7. после доказанного snapshot+patch подключить Foundry binding.
|
||||
|
||||
- stable partitioning в несколько Data Products, где partition key является
|
||||
частью definition, а каждый partition имеет собственный полный snapshot и
|
||||
независимый patch cursor;
|
||||
- новый query delivery contract с явно определёнными snapshot barrier,
|
||||
continuation cursor, query scope и правилами перехода к patch stream.
|
||||
### 8. Capacity и topology
|
||||
|
||||
Такой query contract не входит в `nodedc.data-product.snapshot/v1` и не может
|
||||
быть имитирован полем `nextPageCursor`. До его отдельного утверждения runtime и
|
||||
`NDC Data Product Read` работают только с bounded products до 5000 сущностей.
|
||||
|
||||
### 6. Порядок первой реализации
|
||||
|
||||
1. Версионировать нейтральный intake/read contract и поднять External Data
|
||||
Plane без provider-specific logic.
|
||||
2. Пересобрать существующий L2 proof в native-node pipeline: safe read →
|
||||
split/batch → semantic mapping → generic append → safe summary. Никаких
|
||||
provider ID filters и provider gateway endpoint.
|
||||
3. Выполнить manual run, проверить execution/trace, idempotency and current
|
||||
projection; только затем включить Schedule Trigger.
|
||||
4. Provision immutable writer binding, one-time place its token into an opaque
|
||||
Engine credential, then switch the L2 to
|
||||
`/data-products/:dataProductId/publish` and remove caller scope/headers and
|
||||
the shared legacy credential.
|
||||
5. Подключить Foundry к data product current positions. История, геозоны и
|
||||
новые сущности добавляются отдельными L2 collection profiles and ontology
|
||||
revisions.
|
||||
Количество одновременно активных NDC L2 проектов ограничивается фактическими
|
||||
ресурсами железа и профилем нагрузки. Пока capacity проверяется оператором и не
|
||||
автоматизируется. Канон не объявляет отдельные worker/webhook generations или
|
||||
high-load topology, которых ещё нет в принятом runtime.
|
||||
|
||||
## Последствия
|
||||
|
||||
- `services/<provider>-gateway` не является шаблоном для новых интеграций.
|
||||
Существующий experimental code не расширяется и не деплоится как часть этого
|
||||
решения.
|
||||
- Data Plane может иметь собственную БД/Timescale/PostGIS implementation, но
|
||||
физическая схема остаётся внутренней деталью neutral service, а не API для
|
||||
L2 или Foundry.
|
||||
- Inline raw запрещён в L2 → Data Plane v1: до отдельного raw-vault допускается
|
||||
только restricted ref/hash либо отсутствие raw envelope. Retention reference
|
||||
вычисляется по серверному acceptance time, а не по timestamp из batch; Data
|
||||
Plane выполняет bounded retention sweeps после готовности сервиса и по таймеру.
|
||||
- Обычный Codex Desktop получает ровно те L2 grants, которые выданы владельцем
|
||||
connection/workflow, и может безопасно развивать adapter только в этой
|
||||
границе.
|
||||
- `POST /internal/data-plane/v1/intake` остаётся временным migration-only route:
|
||||
он выключен по умолчанию (`EXTERNAL_DATA_PLANE_LEGACY_INTAKE_ENABLED=false`),
|
||||
требует `NODEDC_INTERNAL_ACCESS_TOKEN`, canonical scoped batch и совпадающие
|
||||
scope headers. Новый или migrated L2 не получает этот shared token и не
|
||||
использует fallback в legacy route. После миграции всех writers он удаляется.
|
||||
- `services/<provider>-gateway` не является частью канона и не создаётся для
|
||||
новых provider integrations. Существующий self-hosted Gelios Gateway и его
|
||||
Timescale volume остаются frozen compatibility contour, воспроизводятся из
|
||||
source/deploy и не изменяются новым L2 pilot без отдельного решения.
|
||||
- Новый account или provider-issued access/refresh pair — configuration change,
|
||||
а не platform release.
|
||||
- Новый provider или неподдержанная capability — versioned provider/Ontology
|
||||
change с contract tests.
|
||||
- Существующий legacy L1/SDK workflow является frozen compatibility boundary:
|
||||
новый L2 pilot его не редактирует, не отзывает его credential и не ставит его
|
||||
вывод условием acceptance.
|
||||
- Legacy `/internal/data-plane/v1/intake` остаётся выключенным migration-only
|
||||
route и не используется новыми NDC L2 workflows.
|
||||
- Команды устройствам остаются отдельным red-domain контуром с явным
|
||||
подтверждением, scope, idempotency и аудитом.
|
||||
- Foundry развивается после доказанного Data Product path; provider API и
|
||||
credentials в Foundry не попадают.
|
||||
|
|
|
|||
|
|
@ -28,7 +28,7 @@ The worker receives only a pairing-bound Hub URL. The Hub verifies that the pair
|
|||
|
||||
It intentionally does not expose filesystem paths, evidence ledgers, credentials, raw payloads, live telemetry, databases, command dispatch, workflow mutation or Studio controls.
|
||||
|
||||
## Boundary for Gelios and future data services
|
||||
## Boundary for external providers and runtime data
|
||||
|
||||
Ontology Core describes the canonical meanings and constraints:
|
||||
|
||||
|
|
@ -36,15 +36,21 @@ Ontology Core describes the canonical meanings and constraints:
|
|||
gelios.unit -> gelios.telemetry_snapshot -> gelios.position_fix -> map.moving_object
|
||||
```
|
||||
|
||||
It does **not** serve the Gelios database or provider API. The future Gelios Gateway/Data API is a distinct capability with its own scope checks, data contract and transport. That capability may later be supplied to an AI Workspace run as another dynamic MCP server. Ontology then gives the assistant the names, relations, guardrails and allowed data-contract route; the data capability enforces access and returns live data.
|
||||
It does **not** serve provider runtime data or call a provider API. Provider
|
||||
access and mapping belong to the granted NDC L2 workflow; shared current and
|
||||
history products belong to the provider-neutral External Data Plane. A scoped
|
||||
Data Product read capability may later be supplied to an AI Workspace run as a
|
||||
separate dynamic MCP server. Ontology supplies meanings, relations, guardrails
|
||||
and the allowed contract route; the data capability enforces access and returns
|
||||
live data.
|
||||
|
||||
This preserves one-way responsibility:
|
||||
|
||||
- Ontology Core: semantics, contracts, aliases, guardrails and context advice.
|
||||
- Gelios Gateway/Data API: access-scoped telemetry and history reads.
|
||||
- External Data Plane: access-scoped Data Product snapshots and updates.
|
||||
- Command Gateway: separate red-domain command route, explicit confirmation and audit.
|
||||
- Engine L2: workflow orchestration using granted capabilities.
|
||||
- Studio: later presentation consumer, outside this implementation.
|
||||
- NDC L2 workflow: provider access, collection and mapping using granted capabilities.
|
||||
- Foundry: presentation consumer, outside this implementation.
|
||||
|
||||
## Live catalog updates without AI Workspace rule redeploy
|
||||
|
||||
|
|
|
|||
|
|
@ -75,10 +75,20 @@ EXTERNAL_DATA_PLANE_WRITER_BINDING_MAX_TTL_DAYS=90
|
|||
EXTERNAL_DATA_PLANE_MAX_FUTURE_SKEW_SECONDS=300
|
||||
EXTERNAL_DATA_PLANE_RETENTION_SWEEP_MS=3600000
|
||||
EXTERNAL_DATA_PLANE_LEGACY_INTAKE_ENABLED=false
|
||||
# The writer-provisioner secret is not an env value. The root-owned deploy
|
||||
# runner keeps it in /volume1/docker/nodedc-platform/secrets/external-data-plane-provisioner/
|
||||
# and mounts it
|
||||
# only into External Data Plane and the future dedicated Engine provisioner.
|
||||
# Internal control-plane writer/reader binding issuance; keep false until the
|
||||
# atomic NDC L2 ensure-grant operation is deployed. Legacy path returns a
|
||||
# plaintext capability and must never be exposed to users.
|
||||
EXTERNAL_DATA_PLANE_PROVISIONING_ENABLED=false
|
||||
# Digest-only managed writer-binding ensure. It accepts only signed Engine
|
||||
# service requests; the legacy provisioner bearer is deliberately invalid here.
|
||||
# Keep false until the matching Engine private key is provisioned. The trust
|
||||
# directory is mounted read-only into EDP and contains only `public-key.pem`.
|
||||
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONING_ENABLED=false
|
||||
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_SERVICE_ID=nodedc-engine
|
||||
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_KEY_ID=engine-edp-managed-provisioner-v1
|
||||
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_AUDIENCE=nodedc-external-data-plane.managed-provisioning.v1
|
||||
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_MAX_SKEW_SECONDS=60
|
||||
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_REPLAY_CACHE_MAX_ENTRIES=10000
|
||||
|
||||
# notification core
|
||||
NOTIFICATION_PG_DB=nodedc_notifications
|
||||
|
|
@ -119,8 +129,10 @@ AI_WORKSPACE_HUB_FALLBACK_URLS=
|
|||
AI_WORKSPACE_ONTOLOGY_MCP_PUBLIC_URL=
|
||||
ONTOLOGY_CORE_HOST_BIND=127.0.0.1:18104
|
||||
|
||||
# Gelios Gateway — storage/read service. Provider credentials belong to the
|
||||
# protected Engine Collector and are never configured in this service.
|
||||
# Gelios Gateway — frozen legacy storage/read compatibility service. Provider
|
||||
# credentials belong to the protected Engine Collector and are never configured
|
||||
# in this service. Keep this contour reproducible; do not use it as a template
|
||||
# for new providers.
|
||||
GELIOS_TIMESCALE_IMAGE=timescale/timescaledb-ha:pg16.14-ts2.28.2-all
|
||||
GELIOS_PG_DB=nodedc_gelios
|
||||
GELIOS_PG_USER=nodedc_gelios
|
||||
|
|
@ -128,8 +140,6 @@ GELIOS_PG_PASS=change-me-generate-with-infra-scripts-init-dev-env
|
|||
# URL-encode reserved characters in GELIOS_PG_PASS when forming this URL.
|
||||
GELIOS_DATABASE_URL=postgresql://nodedc_gelios:change-me-generate-with-infra-scripts-init-dev-env@gelios-postgres:5432/nodedc_gelios
|
||||
GELIOS_GATEWAY_HOST_BIND=127.0.0.1:18105
|
||||
# Explicit tenant and connection identifiers are deployment configuration;
|
||||
# do not encode a customer or pilot name in source defaults.
|
||||
GELIOS_TENANT_ID=replace-with-tenant-id
|
||||
GELIOS_CONNECTION_ID=gelios-connection-id
|
||||
# `allowlist` accepts only GELIOS_ALLOWED_UNIT_IDS. `all` accepts every unit
|
||||
|
|
|
|||
|
|
@ -97,8 +97,15 @@ Engine-owned activator may select it after exact MCP schema acceptance.
|
|||
|
||||
The paired Engine activation is built by
|
||||
`build-engine-n8n-private-extension-artifact.mjs`. It deliberately does not
|
||||
copy the dirty Engine `docker-compose.yml` and does not build an image. Instead
|
||||
it installs a narrowly scoped Compose override plus a strict transition
|
||||
copy from or write generated files into the dirty Engine worktree, and it does
|
||||
not build an image. A fresh transition id is mandatory and previously issued
|
||||
ids are rejected:
|
||||
|
||||
```bash
|
||||
node infra/deploy-runner/build-engine-n8n-private-extension-artifact.mjs 20260717-004
|
||||
```
|
||||
|
||||
The builder emits a narrowly scoped Compose override plus a strict transition
|
||||
descriptor. On apply, the runner validates the staged release again, verifies
|
||||
that the running n8n container and the NAS-local `2.3.2` tag resolve to the
|
||||
same immutable image ID, and extracts the package into the root-owned,
|
||||
|
|
@ -110,10 +117,15 @@ read-only Engine release tree:
|
|||
|
||||
The override sets `N8N_USER_FOLDER=/home/node`, which is required because the
|
||||
actual Engine service runs as root while the canonical community package path
|
||||
is below `/home/node/.n8n`. It enables loading but disables reinstall, mounts
|
||||
only the exact release read-only, and uses both Compose `pull_policy: never`
|
||||
and `docker compose up --pull never`. No registry access, lifecycle script,
|
||||
database `installed_packages` row or custom-extension loader is involved.
|
||||
is below `/home/node/.n8n`. The live runtime contract remains the successfully
|
||||
deployed generation-003 contract and does not set `NODE_PATH`. Only the
|
||||
runner's isolated `node -e` package-loader probe temporarily initializes the
|
||||
dependency tree bundled inside the exact n8n base image; this reproduces n8n's
|
||||
own loader without redefining the live Compose state. The override enables
|
||||
loading but disables reinstall, mounts only the exact release read-only, and
|
||||
uses both Compose `pull_policy: never` and `docker compose up --pull never`.
|
||||
No registry access, lifecycle script, database `installed_packages` row or
|
||||
custom-extension loader is involved.
|
||||
|
||||
The runner pins the exact Engine service topology observed in source and
|
||||
rejects an added worker/webhook generation. Only the single actual `n8n`
|
||||
|
|
@ -140,20 +152,106 @@ PYTHONDONTWRITEBYTECODE=1 \
|
|||
```
|
||||
|
||||
For `platform` artifacts, the allowlist includes the versioned Ontology Core,
|
||||
the legacy Gelios experiment and the provider-neutral External Data Plane
|
||||
sources. An External Data Plane artifact builds only its image and starts
|
||||
`external-data-plane-postgres` plus `external-data-plane`; it never contains a
|
||||
provider credential, provider endpoint, collection schedule or command
|
||||
transport. Database credentials remain root-owned live `.env.synology`
|
||||
the frozen legacy Gelios compatibility service, and the provider-neutral
|
||||
External Data Plane sources. The Gelios service remains reproducible only to
|
||||
protect its existing database/workflow; it is not a template for a provider
|
||||
integration. An External Data Plane artifact builds only its image and
|
||||
force-recreates only `external-data-plane`; the already healthy
|
||||
`external-data-plane-postgres` container and its Timescale volume are an
|
||||
independent deploy prerequisite and are never selected by an EDP application
|
||||
artifact. The reviewed Compose source pins the Timescale image, named volume
|
||||
and target, internal database-only network, absence of database host ports,
|
||||
healthy dependency, localhost-only EDP bind and the two read-only
|
||||
provisioner/trust mounts in the reviewed Compose source. The runner does not
|
||||
reinterpret version-dependent `docker compose config` JSON as a second deploy
|
||||
schema. Its canonical enforcement remains the artifact/path allowlist plus
|
||||
hard-coded build command, selected service set, runtime-secret preparation and
|
||||
health acceptance. Post-apply acceptance also requires
|
||||
`database=ready`; a first-rollout failure removes only the candidate EDP
|
||||
container without volumes and restores the source overlay. It never contains a
|
||||
provider credential, provider endpoint,
|
||||
collection schedule or command
|
||||
transport. Its contract payload is an exact provider-neutral runtime subset;
|
||||
`providers/*`, mappings, fixtures and tests are excluded and do not trigger an
|
||||
EDP rebuild. Database credentials remain root-owned live `.env.synology`
|
||||
configuration and must not reuse `NODEDC_INTERNAL_ACCESS_TOKEN`.
|
||||
|
||||
The External Data Plane writer-provisioner credential is different: on the
|
||||
first relevant Platform apply, the root-owned runner creates
|
||||
On the first relevant Platform apply, the root-owned runner creates
|
||||
`/volume1/docker/nodedc-platform/secrets/external-data-plane-provisioner/token`
|
||||
atomically in a dedicated UID/GID `11006` directory (directory `0500`, token
|
||||
`0400`). It is never an `.env` value or an artifact member; Compose mounts it
|
||||
read-only only into External Data Plane and, when implemented, its dedicated
|
||||
Engine provisioner running under the same restricted identity.
|
||||
`0400`). It is never an `.env` value or an artifact member and is mounted
|
||||
read-only only into External Data Plane.
|
||||
|
||||
Manual one-time binding issuance and digest-only managed writer ensure are
|
||||
independently disabled by default. The target control-plane operation generates
|
||||
and stores the capability inside native NDC L2 Credentials and sends only its
|
||||
digest to EDP; users and MCP consumers receive only an opaque compatible
|
||||
reference/status. The runner-owned bearer above authenticates only legacy
|
||||
plaintext issuance and is intentionally rejected by managed ensure/revoke.
|
||||
Managed requests use a deployment Ed25519 Engine service key; EDP mounts only
|
||||
the public-key trust directory read-only. On every reviewed Engine apply and on
|
||||
an EDP runtime apply, this runner creates or validates one matching Ed25519 pair:
|
||||
the Engine-only private key is `root:root 0400`, while the EDP trust copy is
|
||||
`root:11006 0440`. Public-only crash state, key mismatch, a non-Ed25519 key,
|
||||
symlinks and permissive modes fail closed. `plan` discloses both paths without
|
||||
printing key material. The private key must not be broadened into an L2 graph,
|
||||
MCP surface, artifact or shared-token boundary.
|
||||
|
||||
The reviewed Engine source candidate has dedicated server-derived MCP
|
||||
plan/apply handling for the exact `ndcDataProductWriterApi` + Data Product
|
||||
Publish tuple. It is separate from the generic HTTP safe-ref path and accepts no
|
||||
caller-provided provider/scope/credential identity, capability, generation or
|
||||
service URL. The production managed flag remains false during staging. In the
|
||||
explicitly confirmed activation window it is set to true immediately before
|
||||
the Platform EDP artifact; runner acceptance then requires EDP `/healthz` to
|
||||
report managed provisioning `enabled`. At that point the old Engine still has
|
||||
no signer mount, so the endpoint remains usable only after the separately
|
||||
accepted Engine artifact. Root/UI transfer is emergency
|
||||
self-hosted diagnostics only, not the user journey or acceptance
|
||||
path. A provider-specific daemon remains forbidden.
|
||||
|
||||
Build the narrow Engine source artifact with:
|
||||
|
||||
```bash
|
||||
node infra/deploy-runner/build-engine-data-product-publish-grant-artifact.mjs \
|
||||
engine-data-product-publish-grant-YYYYMMDD-NNN
|
||||
```
|
||||
|
||||
The builder includes exactly the pinned provider-security catalog, the
|
||||
Publish-grant service, Engine Agent scope/gateway wiring, the existing n8n
|
||||
adapter and the reviewed additive backend runtime overlay. It deliberately
|
||||
excludes base Compose, frontend/dist, runtime data, tests, native credentials,
|
||||
the NDC L2 process and the legacy generic credential-sink route/core. The
|
||||
runner fixes the runtime action to `nodedc-backend` with `--no-deps` and
|
||||
`--pull never`; n8n, nginx app, database services and volumes are not selected.
|
||||
Acceptance requires backend health, the active immutable backend identity and
|
||||
the exact additive mount inventory. Any failure restores the touched source and
|
||||
recreates only the previous verified backend runtime.
|
||||
|
||||
Existing Engine Agents created before Publish-grant support store the original
|
||||
nine scopes as the complete developer profile. They must not require a new
|
||||
agent, setup command or device credential when the profile gains a server-owned
|
||||
capability. Build the compatibility artifact that introduces the durable named
|
||||
`full-developer` profile with:
|
||||
|
||||
```bash
|
||||
node infra/deploy-runner/build-engine-agent-full-grant-migration-artifact.mjs \
|
||||
engine-agent-full-grant-migration-YYYYMMDD-NNN
|
||||
```
|
||||
|
||||
This follow-up slice contains exactly
|
||||
`nodedc-source/server/engineAgents/store.js`. The runner pins its predecessor to
|
||||
the successfully applied Publish generation, requires the installed Publish
|
||||
overlay and active immutable backend, recreates only `nodedc-backend`, and
|
||||
proves the exact candidate SHA, named profile and current expanded runtime
|
||||
scope view. Store schema v1 is migrated atomically to v2 only when a grant
|
||||
contains the complete legacy nine-scope developer bundle. From then on the
|
||||
profile name is the authorization authority and its scope list is derived on
|
||||
every read, so capabilities deliberately added to `full-developer` immediately
|
||||
apply to existing full grants without token or store migrations. Partial grants
|
||||
become `custom` and remain exact; they are never elevated. The artifact does not
|
||||
touch UI, n8n, credentials, agent tokens, workflow graphs, databases or runtime
|
||||
payloads.
|
||||
|
||||
`module-foundry` is an independent, authenticated application component. Its
|
||||
artifact contains source and compose infrastructure only; its live
|
||||
|
|
@ -190,12 +288,17 @@ verified; it does not alter NAS routes, VPN, DNS, or Tailscale.
|
|||
Install or update the root-owned live runner on Synology:
|
||||
|
||||
```bash
|
||||
# First verify that no deploy process is active and state/deploy.lock is absent.
|
||||
sudo install -o root -g root -m 0755 \
|
||||
/volume1/docker/nodedc-deploy/runner-install/nodedc-deploy \
|
||||
/usr/local/sbin/nodedc-deploy
|
||||
sudo /usr/local/sbin/nodedc-deploy verify-install
|
||||
```
|
||||
|
||||
Runner promotion is a standalone admin step, never an app-overlay artifact and
|
||||
never part of a running apply. A Python process already executing the old file
|
||||
keeps the old code in memory; always invoke a fresh verified process afterward.
|
||||
|
||||
Normal service deploys must still use explicit artifacts:
|
||||
|
||||
```bash
|
||||
|
|
|
|||
|
|
@ -0,0 +1,121 @@
|
|||
#!/usr/bin/env node
|
||||
import { createHash } from 'node:crypto'
|
||||
import { spawnSync } from 'node:child_process'
|
||||
import { cp, lstat, mkdir, mkdtemp, readFile, rm, writeFile } from 'node:fs/promises'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { dirname, join, resolve } from 'node:path'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
|
||||
const here = dirname(fileURLToPath(import.meta.url))
|
||||
const platformRoot = resolve(here, '../..')
|
||||
const workspaceRoot = resolve(platformRoot, '..')
|
||||
const engineRoot = resolve(workspaceRoot, 'NODEDC_ENGINE_INFRA')
|
||||
const artifactRoot = resolve(process.env.NODEDC_DEPLOY_ARTIFACT_DIR || join(workspaceRoot, 'deploy-artifacts'))
|
||||
const [patchId = '', ...extra] = process.argv.slice(2)
|
||||
const storeRelativePath = 'nodedc-source/server/engineAgents/store.js'
|
||||
const predecessorSha256 = '52daa43499d6d9a97fe7ffa891edb9212b7791e733f91dd3ca686d42739b7e9a'
|
||||
const targetSha256 = '2e62654c2dc12905efcc83a9dff45a818dd7b47924a10600160835c4416540e9'
|
||||
const previouslyIssuedPatchIds = new Set([
|
||||
'engine-agent-full-grant-migration-20260717-001',
|
||||
])
|
||||
|
||||
if (extra.length || !/^engine-agent-full-grant-migration-\d{8}-\d{3}$/.test(patchId)) {
|
||||
throw new Error('usage: build-engine-agent-full-grant-migration-artifact.mjs <fresh-patch-id>')
|
||||
}
|
||||
if (previouslyIssuedPatchIds.has(patchId)) {
|
||||
throw new Error('engine_agent_full_grant_migration_patch_id_already_issued')
|
||||
}
|
||||
|
||||
const source = join(engineRoot, storeRelativePath)
|
||||
const sourceInfo = await lstat(source)
|
||||
if (sourceInfo.isSymbolicLink() || !sourceInfo.isFile()) throw new Error('engine_agent_store_source_unsafe')
|
||||
const sourceBytes = await readFile(source)
|
||||
if (digest(sourceBytes) !== targetSha256) throw new Error('engine_agent_store_target_sha256_mismatch')
|
||||
const sourceText = sourceBytes.toString('utf8')
|
||||
for (const required of [
|
||||
'const STORE_VERSION = 2',
|
||||
"export const ENGINE_AGENT_FULL_DEVELOPER_PROFILE = 'full-developer'",
|
||||
"export const ENGINE_AGENT_CUSTOM_PROFILE = 'custom'",
|
||||
'const LEGACY_FULL_DEVELOPER_SCOPES = Object.freeze([',
|
||||
"'engine:l2:data-product-publish-grant:plan'",
|
||||
"'engine:l2:data-product-publish-grant:write'",
|
||||
'const migrateLegacyFullDeveloper = sourceVersion === 1',
|
||||
'LEGACY_FULL_DEVELOPER_SCOPES.every((scope) => scopes.includes(scope))',
|
||||
'profile === ENGINE_AGENT_FULL_DEVELOPER_PROFILE ? [...ENGINE_AGENT_SCOPES] : scopes',
|
||||
"throw new Error('engine_agent_store_version_unsupported')",
|
||||
]) {
|
||||
if (!sourceText.includes(required)) throw new Error(`engine_agent_store_contract_missing:${required}`)
|
||||
}
|
||||
if (/gelios|robot2b/i.test(sourceText)) throw new Error('engine_agent_store_provider_logic_forbidden')
|
||||
run('node', ['--check', source])
|
||||
|
||||
await mkdir(artifactRoot, { recursive: true })
|
||||
const artifact = join(artifactRoot, `nodedc-${patchId}.tgz`)
|
||||
await assertFresh(artifact)
|
||||
const stage = await mkdtemp(join(tmpdir(), 'nodedc-engine-agent-grant-migration-'))
|
||||
try {
|
||||
const destination = join(stage, 'payload', storeRelativePath)
|
||||
await mkdir(dirname(destination), { recursive: true })
|
||||
await cp(source, destination, { force: false, verbatimSymlinks: true })
|
||||
await writeFile(
|
||||
join(stage, 'manifest.env'),
|
||||
`id=${patchId}\ncomponent=engine\ntype=app-overlay\n`,
|
||||
'utf8',
|
||||
)
|
||||
await writeFile(join(stage, 'files.txt'), `${storeRelativePath}\n`, 'utf8')
|
||||
run('python3', ['-c', canonicalTarScript(), artifact, stage])
|
||||
const artifactBytes = await readFile(artifact)
|
||||
console.log(JSON.stringify({
|
||||
ok: true,
|
||||
patchId,
|
||||
artifact,
|
||||
sha256: digest(artifactBytes),
|
||||
entries: [storeRelativePath],
|
||||
predecessorSha256,
|
||||
targetSha256,
|
||||
services: ['nodedc-backend'],
|
||||
excluded: ['n8n', 'app', 'databases', 'credentials', 'runtime-data'],
|
||||
}, null, 2))
|
||||
} finally {
|
||||
await rm(stage, { recursive: true, force: true })
|
||||
}
|
||||
|
||||
async function assertFresh(target) {
|
||||
try {
|
||||
await lstat(target)
|
||||
} catch (error) {
|
||||
if (error?.code === 'ENOENT') return
|
||||
throw error
|
||||
}
|
||||
throw new Error('engine_agent_full_grant_migration_artifact_already_exists')
|
||||
}
|
||||
|
||||
function canonicalTarScript() {
|
||||
return [
|
||||
'import gzip, io, pathlib, sys, tarfile',
|
||||
'root=pathlib.Path(sys.argv[2])',
|
||||
"with open(sys.argv[1], 'wb') as out:",
|
||||
" with gzip.GzipFile(filename='', mode='wb', fileobj=out, compresslevel=9, mtime=0) as gz:",
|
||||
" with tarfile.open(fileobj=gz, mode='w', format=tarfile.PAX_FORMAT) as tar:",
|
||||
" for top in ('manifest.env','files.txt','payload'):",
|
||||
" p=root/top; paths=[p] + (sorted(p.rglob('*')) if p.is_dir() else [])",
|
||||
' for x in paths:',
|
||||
" info=tar.gettarinfo(str(x), arcname=x.relative_to(root).as_posix())",
|
||||
" info.uid=info.gid=0; info.uname=info.gname='root'; info.mtime=0; info.mode=0o755 if info.isdir() else 0o644",
|
||||
" with (open(x,'rb') if info.isfile() else io.BytesIO()) as src: tar.addfile(info, src if info.isfile() else None)",
|
||||
].join('\n')
|
||||
}
|
||||
|
||||
function run(command, args) {
|
||||
const result = spawnSync(command, args, {
|
||||
encoding: 'utf8',
|
||||
stdio: ['ignore', 'pipe', 'pipe'],
|
||||
maxBuffer: 16 * 1024 * 1024,
|
||||
})
|
||||
if (result.status !== 0) throw new Error(`${command}_failed:${result.stderr || result.stdout}`)
|
||||
return result
|
||||
}
|
||||
|
||||
function digest(bytes) {
|
||||
return createHash('sha256').update(bytes).digest('hex')
|
||||
}
|
||||
|
|
@ -0,0 +1,156 @@
|
|||
#!/usr/bin/env node
|
||||
import { createHash } from 'node:crypto'
|
||||
import { spawnSync } from 'node:child_process'
|
||||
import { lstat, mkdir, mkdtemp, readFile, rm, writeFile } from 'node:fs/promises'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { dirname, join, resolve } from 'node:path'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
|
||||
const here = dirname(fileURLToPath(import.meta.url))
|
||||
const platformRoot = resolve(here, '../..')
|
||||
const workspaceRoot = resolve(platformRoot, '..')
|
||||
const artifactRoot = resolve(process.env.NODEDC_DEPLOY_ARTIFACT_DIR || join(workspaceRoot, 'deploy-artifacts'))
|
||||
const sourceArtifact = resolve(
|
||||
process.env.NODEDC_ENGINE_CREDENTIAL_SINK_RECOVERY_SOURCE
|
||||
|| '/Volumes/docker/nodedc-deploy/applied/nodedc-engine-credential-sink-20260716-001.tgz',
|
||||
)
|
||||
const [patchId = '', ...extra] = process.argv.slice(2)
|
||||
|
||||
if (extra.length || !/^engine-credential-sink-recovery-\d{8}-\d{3}$/.test(patchId)) {
|
||||
throw new Error('usage: build-engine-credential-sink-recovery-artifact.mjs <fresh-recovery-patch-id>')
|
||||
}
|
||||
|
||||
const sourceArtifactSha256 = '0a96add05fe59db8f490927f66e07a84490474a7afb3ef7de51e1d6fd96f86a2'
|
||||
const sourceManifest = 'id=engine-credential-sink-20260716-001\ncomponent=engine\ntype=app-overlay\n'
|
||||
const entries = Object.freeze([
|
||||
'nodedc-source/server/credentialPolicies/ndcPrivateNode.js',
|
||||
'nodedc-source/server/credentialSink',
|
||||
'nodedc-source/server/index.js',
|
||||
'nodedc-source/server/routes/engineAgentGateway.js',
|
||||
'nodedc-source/server/routes/engineCredentialSink.js',
|
||||
'nodedc-source/server/routes/n8n.js',
|
||||
'nodedc-source/server/routes/ndcAgentMcp.js',
|
||||
'nodedc-source/services/backend/credential-sink/docker-compose.immutable-runtime.yml',
|
||||
])
|
||||
const payloadSha256 = Object.freeze({
|
||||
'nodedc-source/server/credentialPolicies/ndcPrivateNode.js': '723874a02dc7b8a68b22ff2304431cfe64f28933cedf5f1e6cda79b2e1cf704a',
|
||||
'nodedc-source/server/credentialSink/core.js': '9f0facc41fd398fcd955cffdd486abb126cdcd756c87ffde667fcbe00e2c41d3',
|
||||
'nodedc-source/server/credentialSink/requestAuth.js': 'f8c9237c3e6f4219dee0d4f956d6e97f76f5bd7ba8a6fd0b4fb21aceffa38138',
|
||||
'nodedc-source/server/credentialSink/store.js': 'c78dc285a973b6acd8a2330f0310935ad08720b2905454492f292d235abf12c0',
|
||||
'nodedc-source/server/credentialSink/vendor/engine-credential-sink.mjs': 'b4800eead9bf94793ff1280d06e34aad6b37ef055a9d7f8d83d7fb5f9bd66d8b',
|
||||
'nodedc-source/server/index.js': 'b2b790b02839570d967a2ca68b00e2724485a99389ac9b3a589a1b22302a36b8',
|
||||
'nodedc-source/server/routes/engineAgentGateway.js': 'e3450a4e1d5318dbac37627b67804d62804e27e2032c9e90478a1e739cea6d3d',
|
||||
'nodedc-source/server/routes/engineCredentialSink.js': '9cbb69dbc8cbe6181cd5b0170fe9c4d717b0a173866ca98cba3e3766c0bab94e',
|
||||
'nodedc-source/server/routes/n8n.js': '783d822e2457d82e890f43bc00c7e33822077dc2841f0511a89ecb210fd36d48',
|
||||
'nodedc-source/server/routes/ndcAgentMcp.js': 'fbb3342b1a617b956d5b3a6a40d5111aa107c1176b4ff89c37b104995bada081',
|
||||
'nodedc-source/services/backend/credential-sink/docker-compose.immutable-runtime.yml': '944fa64b08255eb8207b93fd327aebb98ecd9400d37d25fcfa8e3a040ee44afe',
|
||||
})
|
||||
|
||||
const sourceInfo = await lstat(sourceArtifact)
|
||||
if (sourceInfo.isSymbolicLink() || !sourceInfo.isFile()) {
|
||||
throw new Error('engine_credential_sink_recovery_source_unsafe')
|
||||
}
|
||||
if (digest(await readFile(sourceArtifact)) !== sourceArtifactSha256) {
|
||||
throw new Error('engine_credential_sink_recovery_source_sha256_mismatch')
|
||||
}
|
||||
|
||||
await mkdir(artifactRoot, { recursive: true })
|
||||
const artifact = join(artifactRoot, `nodedc-${patchId}.tgz`)
|
||||
await assertArtifactTargetFresh(artifact)
|
||||
const stage = await mkdtemp(join(tmpdir(), 'nodedc-engine-credential-sink-recovery-'))
|
||||
|
||||
try {
|
||||
run('python3', [
|
||||
'-c', verifiedExtractScript(), sourceArtifact, stage,
|
||||
sourceArtifactSha256, sourceManifest, JSON.stringify(entries), JSON.stringify(payloadSha256),
|
||||
])
|
||||
await writeFile(
|
||||
join(stage, 'manifest.env'),
|
||||
`id=${patchId}\ncomponent=engine\ntype=app-overlay\n`,
|
||||
'utf8',
|
||||
)
|
||||
await writeFile(join(stage, 'files.txt'), `${entries.join('\n')}\n`, 'utf8')
|
||||
run('python3', ['-c', canonicalTarScript(), artifact, stage])
|
||||
console.log(JSON.stringify({
|
||||
ok: true,
|
||||
patchId,
|
||||
artifact,
|
||||
sha256: digest(await readFile(artifact)),
|
||||
sourceArtifact,
|
||||
sourceArtifactSha256,
|
||||
entries,
|
||||
payloadSha256,
|
||||
changed: ['manifest.env:id'],
|
||||
}, null, 2))
|
||||
} finally {
|
||||
await rm(stage, { recursive: true, force: true })
|
||||
}
|
||||
|
||||
async function assertArtifactTargetFresh(target) {
|
||||
try {
|
||||
await lstat(target)
|
||||
} catch (error) {
|
||||
if (error?.code === 'ENOENT') return
|
||||
throw error
|
||||
}
|
||||
throw new Error('engine_credential_sink_recovery_artifact_already_exists')
|
||||
}
|
||||
|
||||
function verifiedExtractScript() {
|
||||
return [
|
||||
'import hashlib,json,pathlib,sys,tarfile',
|
||||
'source=pathlib.Path(sys.argv[1]); stage=pathlib.Path(sys.argv[2])',
|
||||
'expected_archive=sys.argv[3]; expected_manifest=sys.argv[4].encode()',
|
||||
'entries=json.loads(sys.argv[5]); hashes=json.loads(sys.argv[6])',
|
||||
"raw=source.read_bytes()",
|
||||
"assert hashlib.sha256(raw).hexdigest()==expected_archive, 'source-sha256'",
|
||||
"expected_files={'manifest.env','files.txt'}|{'payload/'+name for name in hashes}",
|
||||
"expected_dirs={'payload'}",
|
||||
"for name in hashes:",
|
||||
" p=pathlib.PurePosixPath('payload/'+name)",
|
||||
" expected_dirs.update(str(parent) for parent in p.parents if str(parent)!='.')",
|
||||
"with tarfile.open(source,'r:gz') as tar:",
|
||||
" members=tar.getmembers(); names={member.name for member in members}",
|
||||
" assert names==expected_files|expected_dirs, 'source-member-set'",
|
||||
" for member in members:",
|
||||
" p=pathlib.PurePosixPath(member.name)",
|
||||
" assert not p.is_absolute() and '..' not in p.parts and '\\\\' not in member.name, 'unsafe-member'",
|
||||
" assert member.isdir() if member.name in expected_dirs else member.isfile(), 'unsafe-member-type'",
|
||||
" assert tar.extractfile('manifest.env').read()==expected_manifest, 'source-manifest'",
|
||||
" expected_list=('\\n'.join(entries)+'\\n').encode()",
|
||||
" assert tar.extractfile('files.txt').read()==expected_list, 'source-files-list'",
|
||||
" for name,wanted in hashes.items():",
|
||||
" data=tar.extractfile('payload/'+name).read()",
|
||||
" assert hashlib.sha256(data).hexdigest()==wanted, 'payload-sha256:'+name",
|
||||
" target=stage/'payload'/name; target.parent.mkdir(parents=True,exist_ok=True); target.write_bytes(data)",
|
||||
].join('\n')
|
||||
}
|
||||
|
||||
function canonicalTarScript() {
|
||||
return [
|
||||
'import gzip,io,pathlib,sys,tarfile',
|
||||
'root=pathlib.Path(sys.argv[2])',
|
||||
"with open(sys.argv[1],'wb') as out:",
|
||||
" with gzip.GzipFile(filename='',mode='wb',fileobj=out,compresslevel=9,mtime=0) as gz:",
|
||||
" with tarfile.open(fileobj=gz,mode='w',format=tarfile.PAX_FORMAT) as tar:",
|
||||
" for top in ('manifest.env','files.txt','payload'):",
|
||||
" p=root/top; paths=[p]+(sorted(p.rglob('*')) if p.is_dir() else [])",
|
||||
" for x in paths:",
|
||||
" info=tar.gettarinfo(str(x),arcname=x.relative_to(root).as_posix())",
|
||||
" info.uid=info.gid=0; info.uname=info.gname='root'; info.mtime=0; info.mode=0o755 if info.isdir() else 0o644",
|
||||
" with (open(x,'rb') if info.isfile() else io.BytesIO()) as src: tar.addfile(info,src if info.isfile() else None)",
|
||||
].join('\n')
|
||||
}
|
||||
|
||||
function run(command, args) {
|
||||
const result = spawnSync(command, args, {
|
||||
encoding: 'utf8',
|
||||
stdio: ['ignore', 'pipe', 'pipe'],
|
||||
maxBuffer: 128 * 1024 * 1024,
|
||||
})
|
||||
if (result.status !== 0) throw new Error(`${command}_failed:${result.stderr || result.stdout}`)
|
||||
}
|
||||
|
||||
function digest(bytes) {
|
||||
return createHash('sha256').update(bytes).digest('hex')
|
||||
}
|
||||
|
|
@ -0,0 +1,252 @@
|
|||
#!/usr/bin/env node
|
||||
import { createHash } from 'node:crypto'
|
||||
import { spawnSync } from 'node:child_process'
|
||||
import { cp, lstat, mkdir, mkdtemp, readFile, readdir, rm, writeFile } from 'node:fs/promises'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { dirname, join, relative, resolve } from 'node:path'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
|
||||
const here = dirname(fileURLToPath(import.meta.url))
|
||||
const platformRoot = resolve(here, '../..')
|
||||
const workspaceRoot = resolve(platformRoot, '..')
|
||||
const engineRoot = resolve(workspaceRoot, 'NODEDC_ENGINE_INFRA')
|
||||
const artifactRoot = resolve(process.env.NODEDC_DEPLOY_ARTIFACT_DIR || join(workspaceRoot, 'deploy-artifacts'))
|
||||
const [patchId = '', ...extra] = process.argv.slice(2)
|
||||
const previouslyIssuedPatchIds = new Set([
|
||||
'engine-data-product-publish-grant-20260716-001',
|
||||
'engine-data-product-publish-grant-20260717-001',
|
||||
'engine-data-product-publish-grant-20260717-002',
|
||||
])
|
||||
|
||||
if (extra.length || !/^engine-data-product-publish-grant-\d{8}-\d{3}$/.test(patchId)) {
|
||||
throw new Error('usage: build-engine-data-product-publish-grant-artifact.mjs <fresh-patch-id>')
|
||||
}
|
||||
if (previouslyIssuedPatchIds.has(patchId)) throw new Error('engine_publish_grant_patch_id_already_issued')
|
||||
|
||||
// Deliberately exclude frontend/dist, runtime data, tests, every credential
|
||||
// value, and the already-installed credential-sink implementation. The sink
|
||||
// remains mounted and byte-frozen as the predecessor domain; this artifact
|
||||
// adds the exact Data Product Publish grant flow beside it.
|
||||
const entries = Object.freeze([
|
||||
'nodedc-source/server/assets/provider-packages/v1/catalog.json',
|
||||
'nodedc-source/server/dataProductPublishGrant',
|
||||
'nodedc-source/server/engineAgents/store.js',
|
||||
'nodedc-source/server/routes/engineAgentGateway.js',
|
||||
'nodedc-source/server/routes/n8n.js',
|
||||
'nodedc-source/services/backend/data-product-publish-grant/docker-compose.immutable-runtime.yml',
|
||||
])
|
||||
const ignoredBasenames = new Set(['.DS_Store', '.git', 'node_modules'])
|
||||
const credentialSinkPredecessorSha256 = Object.freeze({
|
||||
'docker-compose.yml': '258cebb64ff1943c939655cc55bdce00fc5c4dced67ec291d84d6df066ace50e',
|
||||
'nodedc-source/server/credentialPolicies/ndcPrivateNode.js': '723874a02dc7b8a68b22ff2304431cfe64f28933cedf5f1e6cda79b2e1cf704a',
|
||||
'nodedc-source/server/credentialSink/core.js': '9f0facc41fd398fcd955cffdd486abb126cdcd756c87ffde667fcbe00e2c41d3',
|
||||
'nodedc-source/server/credentialSink/requestAuth.js': 'f8c9237c3e6f4219dee0d4f956d6e97f76f5bd7ba8a6fd0b4fb21aceffa38138',
|
||||
'nodedc-source/server/credentialSink/store.js': 'c78dc285a973b6acd8a2330f0310935ad08720b2905454492f292d235abf12c0',
|
||||
'nodedc-source/server/credentialSink/vendor/engine-credential-sink.mjs': 'b4800eead9bf94793ff1280d06e34aad6b37ef055a9d7f8d83d7fb5f9bd66d8b',
|
||||
'nodedc-source/server/index.js': 'b2b790b02839570d967a2ca68b00e2724485a99389ac9b3a589a1b22302a36b8',
|
||||
'nodedc-source/server/routes/engineCredentialSink.js': '9cbb69dbc8cbe6181cd5b0170fe9c4d717b0a173866ca98cba3e3766c0bab94e',
|
||||
'nodedc-source/server/routes/ndcAgentMcp.js': 'fbb3342b1a617b956d5b3a6a40d5111aa107c1176b4ff89c37b104995bada081',
|
||||
'nodedc-source/services/backend/credential-sink/docker-compose.immutable-runtime.yml': '944fa64b08255eb8207b93fd327aebb98ecd9400d37d25fcfa8e3a040ee44afe',
|
||||
})
|
||||
const publishGrantRuntimeOverride = [
|
||||
'services:',
|
||||
' nodedc-backend:',
|
||||
' user: "0:0"',
|
||||
' environment:',
|
||||
' ENGINE_DATA_PLANE_BASE_URL: http://external-data-plane:18106',
|
||||
' ENGINE_EDP_MANAGED_PROVISIONER_PRIVATE_KEY_FILE: /run/nodedc-secrets/engine-edp-managed-provisioner-private.pem',
|
||||
' ENGINE_CONTROL_PLANE_PUBLISH_GRANT_ROOT: /var/lib/nodedc-control-plane/publish-grants',
|
||||
' volumes:',
|
||||
' - type: bind',
|
||||
' source: /volume2/nodedc-demo/nodedc-control-plane/publish-grants',
|
||||
' target: /var/lib/nodedc-control-plane/publish-grants',
|
||||
' bind:',
|
||||
' create_host_path: false',
|
||||
' - type: bind',
|
||||
' source: /volume1/docker/nodedc-platform/secrets/engine-edp-managed-provisioner/private-key.pem',
|
||||
' target: /run/nodedc-secrets/engine-edp-managed-provisioner-private.pem',
|
||||
' read_only: true',
|
||||
' bind:',
|
||||
' create_host_path: false',
|
||||
'',
|
||||
].join('\n')
|
||||
|
||||
await assertSourceBoundary()
|
||||
await mkdir(artifactRoot, { recursive: true })
|
||||
const artifact = join(artifactRoot, `nodedc-${patchId}.tgz`)
|
||||
await assertArtifactTargetFresh(artifact)
|
||||
const stage = await mkdtemp(join(tmpdir(), 'nodedc-engine-publish-grant-artifact-'))
|
||||
const payload = join(stage, 'payload')
|
||||
|
||||
try {
|
||||
await mkdir(payload, { recursive: true })
|
||||
for (const entry of entries) {
|
||||
await copySafe(resolve(engineRoot, entry), join(payload, entry))
|
||||
}
|
||||
await writeFile(
|
||||
join(stage, 'manifest.env'),
|
||||
`id=${patchId}\ncomponent=engine\ntype=app-overlay\n`,
|
||||
'utf8',
|
||||
)
|
||||
await writeFile(join(stage, 'files.txt'), `${entries.join('\n')}\n`, 'utf8')
|
||||
run('python3', ['-c', canonicalTarScript(), artifact, stage])
|
||||
const sha256 = digest(await readFile(artifact))
|
||||
console.log(JSON.stringify({
|
||||
ok: true,
|
||||
patchId,
|
||||
artifact,
|
||||
sha256,
|
||||
entries,
|
||||
excluded: [
|
||||
'docker-compose.yml',
|
||||
'nodedc-source/server/index.js',
|
||||
'nodedc-source/server/credentialPolicies/ndcPrivateNode.js',
|
||||
'nodedc-source/server/credentialSink',
|
||||
'nodedc-source/server/routes/engineCredentialSink.js',
|
||||
'nodedc-source/server/middleware/demoAccess.js',
|
||||
'nodedc-source/server/routes/ndcAgentMcp.js',
|
||||
'nodedc-source/server/data',
|
||||
'nodedc-source/dist',
|
||||
'nodedc-source/server/tests',
|
||||
],
|
||||
preservedPredecessor: Object.keys(credentialSinkPredecessorSha256),
|
||||
}, null, 2))
|
||||
} finally {
|
||||
await rm(stage, { recursive: true, force: true })
|
||||
}
|
||||
|
||||
async function assertSourceBoundary() {
|
||||
for (const [relativePath, expectedSha256] of Object.entries(credentialSinkPredecessorSha256)) {
|
||||
const bytes = await readFile(join(engineRoot, relativePath))
|
||||
if (digest(bytes) !== expectedSha256) {
|
||||
throw new Error(`engine_credential_sink_predecessor_drift:${relativePath}`)
|
||||
}
|
||||
}
|
||||
const runtimeOverridePath = join(
|
||||
engineRoot,
|
||||
'nodedc-source/services/backend/data-product-publish-grant/docker-compose.immutable-runtime.yml',
|
||||
)
|
||||
if (await readFile(runtimeOverridePath, 'utf8') !== publishGrantRuntimeOverride) {
|
||||
throw new Error('engine_publish_grant_runtime_override_mismatch')
|
||||
}
|
||||
|
||||
const indexSource = await readFile(join(engineRoot, 'nodedc-source/server/index.js'), 'utf8')
|
||||
if (!indexSource.includes("app.use('/api/engine-agent-mcp', engineAgentMcpRouter)")) {
|
||||
throw new Error('engine_agent_mcp_mount_missing')
|
||||
}
|
||||
if (
|
||||
!indexSource.includes("import engineCredentialSinkRouter from './routes/engineCredentialSink.js'")
|
||||
|| !indexSource.includes("app.use('/internal/engine-credential-sink', express.json({")
|
||||
) throw new Error('engine_credential_sink_predecessor_mount_missing')
|
||||
|
||||
const gateway = await readFile(join(engineRoot, 'nodedc-source/server/routes/engineAgentGateway.js'), 'utf8')
|
||||
for (const tool of [
|
||||
'engine_plan_data_product_publish_grant',
|
||||
'engine_apply_data_product_publish_grant',
|
||||
'engine_accept_data_product_publish_grant',
|
||||
'engine_rollback_data_product_publish_grant',
|
||||
]) {
|
||||
if (!gateway.includes(tool)) throw new Error(`engine_publish_grant_tool_missing:${tool}`)
|
||||
}
|
||||
|
||||
const n8nRoute = await readFile(join(engineRoot, 'nodedc-source/server/routes/n8n.js'), 'utf8')
|
||||
if (n8nRoute.includes("from '../credentialSink/")) {
|
||||
throw new Error('engine_publish_grant_depends_on_legacy_sink')
|
||||
}
|
||||
if (!n8nRoute.includes('engineDataProductPublishGrantN8nAdapter')) {
|
||||
throw new Error('engine_publish_grant_native_adapter_missing')
|
||||
}
|
||||
if (!n8nRoute.includes('engineCredentialSinkN8nAdapter')) {
|
||||
throw new Error('engine_credential_sink_native_adapter_missing')
|
||||
}
|
||||
|
||||
const grantDirectory = join(engineRoot, 'nodedc-source/server/dataProductPublishGrant')
|
||||
const grantFiles = (await readdir(grantDirectory, { withFileTypes: true }))
|
||||
.filter((entry) => entry.isFile())
|
||||
.map((entry) => entry.name)
|
||||
.sort()
|
||||
const expectedGrantFiles = [
|
||||
'acceptance.js',
|
||||
'providerCatalog.js',
|
||||
'service.js',
|
||||
'signedDataPlaneClient.js',
|
||||
'store.js',
|
||||
]
|
||||
if (JSON.stringify(grantFiles) !== JSON.stringify(expectedGrantFiles)) {
|
||||
throw new Error('engine_publish_grant_source_set_mismatch')
|
||||
}
|
||||
for (const entry of entries) {
|
||||
const source = resolve(engineRoot, entry)
|
||||
const info = await lstat(source)
|
||||
if (info.isSymbolicLink() || (!info.isFile() && !info.isDirectory())) {
|
||||
throw new Error(`engine_artifact_source_unsafe:${entry}`)
|
||||
}
|
||||
}
|
||||
for (const source of [
|
||||
...expectedGrantFiles.map((name) => join(grantDirectory, name)),
|
||||
join(engineRoot, 'nodedc-source/server/routes/engineAgentGateway.js'),
|
||||
join(engineRoot, 'nodedc-source/server/routes/n8n.js'),
|
||||
join(engineRoot, 'nodedc-source/server/routes/ndcAgentMcp.js'),
|
||||
join(engineRoot, 'nodedc-source/server/engineAgents/store.js'),
|
||||
]) run('node', ['--check', source])
|
||||
}
|
||||
|
||||
async function assertArtifactTargetFresh(target) {
|
||||
try {
|
||||
await lstat(target)
|
||||
} catch (error) {
|
||||
if (error?.code === 'ENOENT') return
|
||||
throw error
|
||||
}
|
||||
throw new Error('engine_publish_grant_artifact_already_exists')
|
||||
}
|
||||
|
||||
async function copySafe(source, destination) {
|
||||
const info = await lstat(source)
|
||||
if (info.isSymbolicLink()) throw new Error(`source_symlink_rejected:${source}`)
|
||||
if (info.isFile()) {
|
||||
await mkdir(dirname(destination), { recursive: true })
|
||||
await cp(source, destination, { force: false, verbatimSymlinks: true })
|
||||
return
|
||||
}
|
||||
if (!info.isDirectory()) throw new Error(`source_type_rejected:${source}`)
|
||||
await mkdir(destination, { recursive: true })
|
||||
for (const entry of await readdir(source, { withFileTypes: true })) {
|
||||
if (ignoredBasenames.has(entry.name) || entry.name.startsWith('.env')) continue
|
||||
const childSource = join(source, entry.name)
|
||||
if (entry.isSymbolicLink()) {
|
||||
throw new Error(`source_symlink_rejected:${relative(engineRoot, childSource)}`)
|
||||
}
|
||||
await copySafe(childSource, join(destination, entry.name))
|
||||
}
|
||||
}
|
||||
|
||||
function canonicalTarScript() {
|
||||
return [
|
||||
'import gzip, io, pathlib, sys, tarfile',
|
||||
'root=pathlib.Path(sys.argv[2])',
|
||||
"with open(sys.argv[1], 'wb') as out:",
|
||||
" with gzip.GzipFile(filename='', mode='wb', fileobj=out, compresslevel=9, mtime=0) as gz:",
|
||||
" with tarfile.open(fileobj=gz, mode='w', format=tarfile.PAX_FORMAT) as tar:",
|
||||
" for top in ('manifest.env','files.txt','payload'):",
|
||||
" p=root/top; paths=[p] + (sorted(p.rglob('*')) if p.is_dir() else [])",
|
||||
' for x in paths:',
|
||||
" info=tar.gettarinfo(str(x), arcname=x.relative_to(root).as_posix())",
|
||||
" info.uid=info.gid=0; info.uname=info.gname='root'; info.mtime=0; info.mode=0o755 if info.isdir() else 0o644",
|
||||
" with (open(x,'rb') if info.isfile() else io.BytesIO()) as src: tar.addfile(info, src if info.isfile() else None)",
|
||||
].join('\n')
|
||||
}
|
||||
|
||||
function run(command, args) {
|
||||
const result = spawnSync(command, args, {
|
||||
encoding: 'utf8',
|
||||
stdio: ['ignore', 'pipe', 'pipe'],
|
||||
maxBuffer: 128 * 1024 * 1024,
|
||||
})
|
||||
if (result.status !== 0) throw new Error(`${command}_failed:${result.stderr || result.stdout}`)
|
||||
return result
|
||||
}
|
||||
|
||||
function digest(bytes) {
|
||||
return createHash('sha256').update(bytes).digest('hex')
|
||||
}
|
||||
|
|
@ -1,7 +1,7 @@
|
|||
#!/usr/bin/env node
|
||||
import { createHash } from "node:crypto";
|
||||
import { spawnSync } from "node:child_process";
|
||||
import { cp, mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
|
||||
import { cp, lstat, mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
|
||||
import { createRequire, Module } from "node:module";
|
||||
import { tmpdir } from "node:os";
|
||||
import { dirname, join, resolve } from "node:path";
|
||||
|
|
@ -24,8 +24,10 @@ const n8nVersion = "2.3.2";
|
|||
const baseImage = "docker.n8n.io/n8nio/n8n:2.3.2";
|
||||
const architecture = "amd64";
|
||||
const generatedAt = "2026-07-15T21:51:41.000Z";
|
||||
const activationId = "engine-n8n-private-extension-20260716-003";
|
||||
const rollbackId = "engine-n8n-private-extension-rollback-20260716-003";
|
||||
const previouslyIssuedTransitionIds = new Set(["20260715-002", "20260716-003"]);
|
||||
const transitionId = readTransitionId(process.argv.slice(2), process.env.NODEDC_N8N_TRANSITION_ID);
|
||||
const activationId = `engine-n8n-private-extension-${transitionId}`;
|
||||
const rollbackId = `engine-n8n-private-extension-rollback-${transitionId}`;
|
||||
|
||||
const transitionRoot = "nodedc-source/services/n8n/private-extensions";
|
||||
const descriptorRel = `${transitionRoot}/ndc-activation.json`;
|
||||
|
|
@ -62,6 +64,8 @@ const credentialModules = [
|
|||
];
|
||||
|
||||
await mkdir(artifactRoot, { recursive: true });
|
||||
await assertArtifactTargetFresh(join(artifactRoot, `nodedc-${activationId}.tgz`));
|
||||
await assertArtifactTargetFresh(join(artifactRoot, `nodedc-${rollbackId}.tgz`));
|
||||
assertSha(await readFile(stageArtifact), stageArtifactSha256, "staging artifact");
|
||||
assertEngineBaseline(await readFile(join(engineRoot, "docker-compose.yml"), "utf8"));
|
||||
|
||||
|
|
@ -125,13 +129,15 @@ try {
|
|||
const rollbackDescriptor = descriptor("rollback-inactive", [], [], releaseId);
|
||||
const override = composeOverride();
|
||||
|
||||
await writeJson(join(engineRoot, nodesCatalogRel), activeNodes);
|
||||
await writeJson(join(engineRoot, credentialsCatalogRel), activeCredentials);
|
||||
await writeJson(join(engineRoot, metaRel), activeMeta);
|
||||
await writeJson(join(engineRoot, descriptorRel), activationDescriptor);
|
||||
await writeFile(join(engineRoot, overrideRel), override, "utf8");
|
||||
await cp(join(packageRoot, "dist/icons/ndc.svg"), join(engineRoot, iconRel), { force: true });
|
||||
await cp(join(packageRoot, "dist/icons/ndc.dark.svg"), join(engineRoot, darkIconRel), { force: true });
|
||||
const generatedRoot = join(work, "generated-engine-payload");
|
||||
await writeJson(join(generatedRoot, nodesCatalogRel), activeNodes);
|
||||
await writeJson(join(generatedRoot, credentialsCatalogRel), activeCredentials);
|
||||
await writeJson(join(generatedRoot, metaRel), activeMeta);
|
||||
await writeJson(join(generatedRoot, descriptorRel), activationDescriptor);
|
||||
await writeFile(join(generatedRoot, overrideRel), override, "utf8");
|
||||
await mkdir(join(generatedRoot, iconRoot), { recursive: true });
|
||||
await cp(join(packageRoot, "dist/icons/ndc.svg"), join(generatedRoot, iconRel), { force: false });
|
||||
await cp(join(packageRoot, "dist/icons/ndc.dark.svg"), join(generatedRoot, darkIconRel), { force: false });
|
||||
|
||||
const activationEntries = [
|
||||
descriptorRel,
|
||||
|
|
@ -144,7 +150,7 @@ try {
|
|||
];
|
||||
const activationArtifact = await buildArtifact(work, activationId, activationEntries, async (payload) => {
|
||||
for (const rel of activationEntries) {
|
||||
await cp(join(engineRoot, rel), join(payload, rel), { recursive: true, force: false });
|
||||
await cp(join(generatedRoot, rel), join(payload, rel), { recursive: true, force: false });
|
||||
}
|
||||
});
|
||||
|
||||
|
|
@ -159,6 +165,7 @@ try {
|
|||
|
||||
console.log(JSON.stringify({
|
||||
ok: true,
|
||||
transitionId,
|
||||
releaseId,
|
||||
packageSha256,
|
||||
nodeTypes: expectedNodeTypes,
|
||||
|
|
@ -192,6 +199,42 @@ function descriptor(action, nodeTypes, credentialTypes, expectedCurrent) {
|
|||
};
|
||||
}
|
||||
|
||||
function readTransitionId(args, environmentValue) {
|
||||
if (args.length > 1) throw new Error("transition_id_argument_count_invalid");
|
||||
const argumentValue = args[0] || "";
|
||||
const envValue = String(environmentValue || "").trim();
|
||||
if (argumentValue && envValue && argumentValue !== envValue) {
|
||||
throw new Error("transition_id_sources_conflict");
|
||||
}
|
||||
const value = argumentValue || envValue;
|
||||
if (!value) throw new Error("transition_id_required");
|
||||
const match = /^(\d{4})(\d{2})(\d{2})-([0-9]{3})$/.exec(value);
|
||||
if (!match || match[4] === "000") throw new Error("transition_id_invalid");
|
||||
const year = Number(match[1]);
|
||||
const month = Number(match[2]);
|
||||
const day = Number(match[3]);
|
||||
const parsed = new Date(Date.UTC(year, month - 1, day));
|
||||
if (parsed.getUTCFullYear() !== year
|
||||
|| parsed.getUTCMonth() !== month - 1
|
||||
|| parsed.getUTCDate() !== day) {
|
||||
throw new Error("transition_id_invalid");
|
||||
}
|
||||
if (previouslyIssuedTransitionIds.has(value)) {
|
||||
throw new Error("transition_id_already_issued");
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
async function assertArtifactTargetFresh(path) {
|
||||
try {
|
||||
await lstat(path);
|
||||
} catch (error) {
|
||||
if (error?.code === "ENOENT") return;
|
||||
throw error;
|
||||
}
|
||||
throw new Error("transition_artifact_already_exists");
|
||||
}
|
||||
|
||||
function composeOverride() {
|
||||
const health = "const http=require('http');const req=http.get('http://127.0.0.1:5678/healthz/readiness',r=>{r.resume();process.exit(r.statusCode===200?0:1)});req.on('error',()=>process.exit(1));req.setTimeout(4000,()=>{req.destroy();process.exit(1)});";
|
||||
return [
|
||||
|
|
|
|||
|
|
@ -1,14 +1,14 @@
|
|||
#!/usr/bin/env node
|
||||
import { createHash } from "node:crypto";
|
||||
import { spawnSync } from "node:child_process";
|
||||
import { cp, lstat, mkdir, mkdtemp, readdir, rm, writeFile } from "node:fs/promises";
|
||||
import { cp, lstat, mkdir, mkdtemp, readFile, readdir, rm, writeFile } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { dirname, join, relative, resolve } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
const scriptDir = dirname(fileURLToPath(import.meta.url));
|
||||
const platformRoot = resolve(scriptDir, "../..");
|
||||
const artifactDir = resolve(scriptDir, "../deploy-artifacts");
|
||||
const artifactDir = resolve(process.env.NODEDC_DEPLOY_ARTIFACT_DIR || resolve(scriptDir, "../deploy-artifacts"));
|
||||
const [patchId = "external-data-plane-20260714-001", ...extra] = process.argv.slice(2);
|
||||
|
||||
if (extra.length || !/^[A-Za-z0-9._-]{1,96}$/.test(patchId)) {
|
||||
|
|
@ -18,13 +18,21 @@ if (extra.length || !/^[A-Za-z0-9._-]{1,96}$/.test(patchId)) {
|
|||
const files = [
|
||||
["infra/synology/docker-compose.external-data-plane.yml", "platform/docker-compose.external-data-plane.yml"],
|
||||
["services/external-data-plane", "platform/services/external-data-plane"],
|
||||
["packages/external-provider-contract", "platform/packages/external-provider-contract"],
|
||||
["packages/external-provider-contract/package.json", "platform/packages/external-provider-contract/package.json"],
|
||||
["packages/external-provider-contract/src/contract-version.mjs", "platform/packages/external-provider-contract/src/contract-version.mjs"],
|
||||
["packages/external-provider-contract/src/data-plane.mjs", "platform/packages/external-provider-contract/src/data-plane.mjs"],
|
||||
["packages/external-provider-contract/src/data-product.mjs", "platform/packages/external-provider-contract/src/data-product.mjs"],
|
||||
["packages/external-provider-contract/src/intake-batch.mjs", "platform/packages/external-provider-contract/src/intake-batch.mjs"],
|
||||
["packages/external-provider-contract/src/sensitive-field-policy.mjs", "platform/packages/external-provider-contract/src/sensitive-field-policy.mjs"],
|
||||
];
|
||||
const ignoredBasenames = new Set([".DS_Store", ".git", "node_modules"]);
|
||||
const ignoredDirectoryNames = new Set(["test"]);
|
||||
const stage = await mkdtemp(join(tmpdir(), "nodedc-external-data-plane-artifact-"));
|
||||
const payload = join(stage, "payload");
|
||||
const target = join(artifactDir, `nodedc-platform-${patchId}.tgz`);
|
||||
|
||||
await assertSourceBoundary();
|
||||
|
||||
try {
|
||||
await mkdir(payload, { recursive: true });
|
||||
for (const [sourceRelative, destinationRelative] of files) {
|
||||
|
|
@ -34,19 +42,66 @@ try {
|
|||
await writeFile(join(stage, "files.txt"), `${files.map(([, destination]) => destination).join("\n")}\n`, "utf8");
|
||||
await mkdir(artifactDir, { recursive: true });
|
||||
|
||||
const tar = spawnSync("python3", ["-c", [
|
||||
"import sys, tarfile",
|
||||
"with tarfile.open(sys.argv[1], 'w:gz', format=tarfile.PAX_FORMAT) as archive:",
|
||||
" [archive.add(name, arcname=name, recursive=True) for name in ('manifest.env', 'files.txt', 'payload')]",
|
||||
].join("\n"), target], { cwd: stage, encoding: "utf8" });
|
||||
const tar = spawnSync("python3", ["-c", canonicalTarScript(), target, stage], {
|
||||
encoding: "utf8",
|
||||
maxBuffer: 128 * 1024 * 1024,
|
||||
});
|
||||
if (tar.status !== 0) throw new Error(`tar_failed:${tar.stderr || tar.stdout}`);
|
||||
|
||||
const digest = createHash("sha256").update(await (await import("node:fs/promises")).readFile(target)).digest("hex");
|
||||
console.log(JSON.stringify({ ok: true, patchId, artifact: target, sha256: digest }, null, 2));
|
||||
const digest = createHash("sha256").update(await readFile(target)).digest("hex");
|
||||
console.log(JSON.stringify({
|
||||
ok: true,
|
||||
patchId,
|
||||
artifact: target,
|
||||
sha256: digest,
|
||||
entries: files.map(([, destination]) => destination),
|
||||
excluded: [
|
||||
".env*",
|
||||
"node_modules",
|
||||
"services/external-data-plane/test",
|
||||
"private-key.pem",
|
||||
"secrets",
|
||||
],
|
||||
}, null, 2));
|
||||
} finally {
|
||||
await rm(stage, { recursive: true, force: true });
|
||||
}
|
||||
|
||||
async function assertSourceBoundary() {
|
||||
const compose = await readFile(
|
||||
resolve(platformRoot, "infra/synology/docker-compose.external-data-plane.yml"),
|
||||
"utf8",
|
||||
);
|
||||
for (const fragment of [
|
||||
"source: /volume1/docker/nodedc-platform/trust/engine-managed-provisioner",
|
||||
"target: /run/nodedc-trust/engine-managed-provisioner",
|
||||
"EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_PUBLIC_KEY_FILE: /run/nodedc-trust/engine-managed-provisioner/public-key.pem",
|
||||
"create_host_path: false",
|
||||
]) {
|
||||
if (!compose.includes(fragment)) throw new Error(`platform_compose_boundary_missing:${fragment}`);
|
||||
}
|
||||
if (
|
||||
compose.includes("private-key.pem")
|
||||
|| compose.includes("ENGINE_EDP_MANAGED_PROVISIONER_PRIVATE_KEY_FILE")
|
||||
) throw new Error("platform_artifact_private_key_boundary_violation");
|
||||
}
|
||||
|
||||
function canonicalTarScript() {
|
||||
return [
|
||||
"import gzip, io, pathlib, sys, tarfile",
|
||||
"root=pathlib.Path(sys.argv[2])",
|
||||
"with open(sys.argv[1], 'wb') as out:",
|
||||
" with gzip.GzipFile(filename='', mode='wb', fileobj=out, compresslevel=9, mtime=0) as gz:",
|
||||
" with tarfile.open(fileobj=gz, mode='w', format=tarfile.PAX_FORMAT) as tar:",
|
||||
" for top in ('manifest.env','files.txt','payload'):",
|
||||
" p=root/top; paths=[p] + (sorted(p.rglob('*')) if p.is_dir() else [])",
|
||||
" for x in paths:",
|
||||
" info=tar.gettarinfo(str(x), arcname=x.relative_to(root).as_posix())",
|
||||
" info.uid=info.gid=0; info.uname=info.gname='root'; info.mtime=0; info.mode=0o755 if info.isdir() else 0o644",
|
||||
" with (open(x,'rb') if info.isfile() else io.BytesIO()) as src: tar.addfile(info, src if info.isfile() else None)",
|
||||
].join("\n");
|
||||
}
|
||||
|
||||
async function copySafe(source, destination) {
|
||||
const sourceStat = await lstat(source);
|
||||
if (sourceStat.isSymbolicLink()) throw new Error(`source_symlink_rejected:${source}`);
|
||||
|
|
@ -60,6 +115,7 @@ async function copySafe(source, destination) {
|
|||
await mkdir(destination, { recursive: true });
|
||||
for (const entry of await readdir(source, { withFileTypes: true })) {
|
||||
if (ignoredBasenames.has(entry.name) || entry.name.startsWith(".env")) continue;
|
||||
if (entry.isDirectory() && ignoredDirectoryNames.has(entry.name)) continue;
|
||||
const childSource = join(source, entry.name);
|
||||
const childDestination = join(destination, entry.name);
|
||||
if (entry.isSymbolicLink()) throw new Error(`source_symlink_rejected:${relative(platformRoot, childSource)}`);
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load Diff
|
|
@ -0,0 +1,189 @@
|
|||
#!/usr/bin/env python3
|
||||
import hashlib
|
||||
import importlib.machinery
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import tarfile
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from unittest import mock
|
||||
|
||||
|
||||
SCRIPT_DIR = Path(__file__).resolve().parent
|
||||
BUILDER = SCRIPT_DIR / "build-engine-agent-full-grant-migration-artifact.mjs"
|
||||
RUNNER_PATH = SCRIPT_DIR / "nodedc-deploy"
|
||||
PATCH_ID = "engine-agent-full-grant-migration-20990101-001"
|
||||
STORE_REL = "nodedc-source/server/engineAgents/store.js"
|
||||
PREDECESSOR_SHA256 = "52daa43499d6d9a97fe7ffa891edb9212b7791e733f91dd3ca686d42739b7e9a"
|
||||
TARGET_SHA256 = "2e62654c2dc12905efcc83a9dff45a818dd7b47924a10600160835c4416540e9"
|
||||
|
||||
|
||||
def load_runner():
|
||||
loader = importlib.machinery.SourceFileLoader(
|
||||
"nodedc_agent_grant_migration_runner_under_test",
|
||||
str(RUNNER_PATH),
|
||||
)
|
||||
spec = importlib.util.spec_from_loader(loader.name, loader)
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
RUNNER = load_runner()
|
||||
|
||||
|
||||
class EngineAgentFullGrantMigrationArtifactTest(unittest.TestCase):
|
||||
def build(self, artifact_dir, patch_id=PATCH_ID):
|
||||
environment = os.environ.copy()
|
||||
environment["NODEDC_DEPLOY_ARTIFACT_DIR"] = str(artifact_dir)
|
||||
result = subprocess.run(
|
||||
["node", str(BUILDER), patch_id],
|
||||
check=True,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
env=environment,
|
||||
)
|
||||
return json.loads(result.stdout)
|
||||
|
||||
def test_artifact_is_exact_reproducible_and_runner_accepted(self):
|
||||
with tempfile.TemporaryDirectory(prefix="nodedc-agent-grant-migration-") as directory:
|
||||
root = Path(directory)
|
||||
first_dir = root / "first"
|
||||
second_dir = root / "second"
|
||||
first_dir.mkdir()
|
||||
second_dir.mkdir()
|
||||
first = self.build(first_dir)
|
||||
second = self.build(second_dir)
|
||||
first_artifact = Path(first["artifact"])
|
||||
second_artifact = Path(second["artifact"])
|
||||
|
||||
self.assertEqual(first["entries"], [STORE_REL])
|
||||
self.assertEqual(first["predecessorSha256"], PREDECESSOR_SHA256)
|
||||
self.assertEqual(first["targetSha256"], TARGET_SHA256)
|
||||
self.assertEqual(first["sha256"], hashlib.sha256(first_artifact.read_bytes()).hexdigest())
|
||||
self.assertEqual(first["sha256"], second["sha256"])
|
||||
self.assertEqual(first_artifact.read_bytes(), second_artifact.read_bytes())
|
||||
|
||||
with tarfile.open(first_artifact, "r:gz") as archive:
|
||||
members = archive.getmembers()
|
||||
names = {member.name for member in members}
|
||||
manifest = archive.extractfile("manifest.env").read().decode("utf-8")
|
||||
files = archive.extractfile("files.txt").read().decode("utf-8")
|
||||
store = archive.extractfile(f"payload/{STORE_REL}").read()
|
||||
self.assertEqual(
|
||||
manifest,
|
||||
f"id={PATCH_ID}\ncomponent=engine\ntype=app-overlay\n",
|
||||
)
|
||||
self.assertEqual(files, f"{STORE_REL}\n")
|
||||
self.assertEqual(hashlib.sha256(store).hexdigest(), TARGET_SHA256)
|
||||
self.assertFalse(any(member.issym() or member.islnk() for member in members))
|
||||
self.assertFalse(any(
|
||||
name.startswith("payload/nodedc-source/server/data/")
|
||||
or name.startswith("payload/nodedc-source/dist/")
|
||||
or name.startswith("payload/nodedc-source/server/credentialSink/")
|
||||
for name in names
|
||||
))
|
||||
|
||||
with tempfile.TemporaryDirectory(prefix="nodedc-agent-grant-runner-") as work:
|
||||
manifest_value, entries, _payload = RUNNER.load_artifact(
|
||||
first_artifact,
|
||||
Path(work),
|
||||
)
|
||||
self.assertEqual(manifest_value["id"], PATCH_ID)
|
||||
self.assertEqual(tuple(entries), RUNNER.ENGINE_AGENT_FULL_GRANT_MIGRATION_ARTIFACT_ENTRIES)
|
||||
self.assertTrue(RUNNER.is_engine_agent_full_grant_migration_slice("engine", entries))
|
||||
self.assertEqual(RUNNER.component_services("engine", entries), ("nodedc-backend",))
|
||||
self.assertEqual(RUNNER.component_builds("engine", entries), ())
|
||||
self.assertEqual(
|
||||
RUNNER.component_healthchecks("engine", entries, ("nodedc-backend",)),
|
||||
(
|
||||
"http://127.0.0.1:3001/health",
|
||||
"http://127.0.0.1:3001/internal/engine-credential-sink/v1/health",
|
||||
),
|
||||
)
|
||||
|
||||
def test_predecessor_and_runtime_acceptance_are_exact(self):
|
||||
with tempfile.TemporaryDirectory(prefix="nodedc-agent-grant-predecessor-") as directory:
|
||||
root = Path(directory)
|
||||
store = root / STORE_REL
|
||||
store.parent.mkdir(parents=True)
|
||||
store.write_text("predecessor", encoding="utf-8")
|
||||
override = root / RUNNER.ENGINE_DATA_PRODUCT_PUBLISH_GRANT_OVERRIDE_REL
|
||||
override.parent.mkdir(parents=True)
|
||||
override.write_text(
|
||||
RUNNER.expected_engine_data_product_publish_grant_override(),
|
||||
encoding="utf-8",
|
||||
)
|
||||
original_root = RUNNER.COMPONENTS["engine"]["payload_root"]
|
||||
RUNNER.COMPONENTS["engine"]["payload_root"] = root
|
||||
try:
|
||||
with mock.patch.object(RUNNER, "sha256_file", return_value=PREDECESSOR_SHA256):
|
||||
self.assertEqual(
|
||||
RUNNER.preflight_engine_agent_full_grant_migration_predecessor(),
|
||||
PREDECESSOR_SHA256,
|
||||
)
|
||||
with mock.patch.object(RUNNER, "sha256_file", return_value="0" * 64):
|
||||
with self.assertRaisesRegex(RUNNER.DeployError, "predecessor drift"):
|
||||
RUNNER.preflight_engine_agent_full_grant_migration_predecessor()
|
||||
finally:
|
||||
RUNNER.COMPONENTS["engine"]["payload_root"] = original_root
|
||||
|
||||
expected = f"store-v2-full-developer:{TARGET_SHA256}"
|
||||
with mock.patch.object(
|
||||
RUNNER,
|
||||
"engine_backend_container_id",
|
||||
return_value="container-id",
|
||||
):
|
||||
with mock.patch.object(
|
||||
RUNNER,
|
||||
"run_engine_backend_probe",
|
||||
return_value=expected,
|
||||
) as probe:
|
||||
self.assertEqual(RUNNER.accept_engine_agent_full_grant_migration_runtime(), expected)
|
||||
arguments, label = probe.call_args.args[:2]
|
||||
self.assertEqual(label, "agent full grant migration")
|
||||
self.assertEqual(arguments[0:2], ("node", "--input-type=module"))
|
||||
self.assertIn(TARGET_SHA256, arguments)
|
||||
self.assertEqual(probe.call_args.kwargs["container_id"], "container-id")
|
||||
|
||||
def test_builder_rejects_invalid_id_and_overwrite(self):
|
||||
with tempfile.TemporaryDirectory(prefix="nodedc-agent-grant-negative-") as directory:
|
||||
root = Path(directory)
|
||||
environment = {**os.environ, "NODEDC_DEPLOY_ARTIFACT_DIR": str(root)}
|
||||
invalid = subprocess.run(
|
||||
["node", str(BUILDER), "engine-data-product-publish-grant-20990101-001"],
|
||||
check=False,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
env=environment,
|
||||
)
|
||||
self.assertNotEqual(invalid.returncode, 0)
|
||||
self.assertIn("fresh-patch-id", invalid.stderr)
|
||||
|
||||
issued = subprocess.run(
|
||||
["node", str(BUILDER), "engine-agent-full-grant-migration-20260717-001"],
|
||||
check=False,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
env=environment,
|
||||
)
|
||||
self.assertNotEqual(issued.returncode, 0)
|
||||
self.assertIn("patch_id_already_issued", issued.stderr)
|
||||
|
||||
self.build(root)
|
||||
duplicate = subprocess.run(
|
||||
["node", str(BUILDER), PATCH_ID],
|
||||
check=False,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
env=environment,
|
||||
)
|
||||
self.assertNotEqual(duplicate.returncode, 0)
|
||||
self.assertIn("artifact_already_exists", duplicate.stderr)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main(verbosity=2)
|
||||
|
|
@ -0,0 +1,147 @@
|
|||
#!/usr/bin/env python3
|
||||
import hashlib
|
||||
import importlib.machinery
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import tarfile
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
SCRIPT_DIR = Path(__file__).resolve().parent
|
||||
BUILDER = SCRIPT_DIR / "build-engine-credential-sink-recovery-artifact.mjs"
|
||||
RUNNER_PATH = SCRIPT_DIR / "nodedc-deploy"
|
||||
SOURCE = Path("/Volumes/docker/nodedc-deploy/applied/nodedc-engine-credential-sink-20260716-001.tgz")
|
||||
SOURCE_SHA256 = "0a96add05fe59db8f490927f66e07a84490474a7afb3ef7de51e1d6fd96f86a2"
|
||||
PATCH_ID = "engine-credential-sink-recovery-20990101-001"
|
||||
|
||||
|
||||
def load_runner():
|
||||
loader = importlib.machinery.SourceFileLoader(
|
||||
"nodedc_recovery_runner_under_test",
|
||||
str(RUNNER_PATH),
|
||||
)
|
||||
spec = importlib.util.spec_from_loader(loader.name, loader)
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
RUNNER = load_runner()
|
||||
|
||||
|
||||
class EngineCredentialSinkRecoveryArtifactTest(unittest.TestCase):
|
||||
def build(self, artifact_dir, patch_id=PATCH_ID, source=SOURCE):
|
||||
environment = os.environ.copy()
|
||||
environment["NODEDC_DEPLOY_ARTIFACT_DIR"] = str(artifact_dir)
|
||||
environment["NODEDC_ENGINE_CREDENTIAL_SINK_RECOVERY_SOURCE"] = str(source)
|
||||
result = subprocess.run(
|
||||
["node", str(BUILDER), patch_id],
|
||||
check=True,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
env=environment,
|
||||
)
|
||||
return json.loads(result.stdout)
|
||||
|
||||
def test_recovery_changes_only_manifest_id_and_passes_runner_policy(self):
|
||||
self.assertTrue(SOURCE.is_file())
|
||||
self.assertEqual(hashlib.sha256(SOURCE.read_bytes()).hexdigest(), SOURCE_SHA256)
|
||||
with tempfile.TemporaryDirectory(prefix="nodedc-engine-sink-recovery-") as directory:
|
||||
root = Path(directory)
|
||||
first_dir = root / "first"
|
||||
second_dir = root / "second"
|
||||
first_dir.mkdir()
|
||||
second_dir.mkdir()
|
||||
first = self.build(first_dir)
|
||||
second = self.build(second_dir)
|
||||
first_artifact = Path(first["artifact"])
|
||||
second_artifact = Path(second["artifact"])
|
||||
|
||||
self.assertEqual(first["sourceArtifactSha256"], SOURCE_SHA256)
|
||||
self.assertEqual(first["changed"], ["manifest.env:id"])
|
||||
self.assertEqual(first["sha256"], second["sha256"])
|
||||
self.assertEqual(first_artifact.read_bytes(), second_artifact.read_bytes())
|
||||
|
||||
with tarfile.open(SOURCE, "r:gz") as old, tarfile.open(first_artifact, "r:gz") as new:
|
||||
old_files = old.extractfile("files.txt").read()
|
||||
new_files = new.extractfile("files.txt").read()
|
||||
self.assertEqual(old_files, new_files)
|
||||
self.assertEqual(
|
||||
new.extractfile("manifest.env").read().decode("utf-8"),
|
||||
f"id={PATCH_ID}\ncomponent=engine\ntype=app-overlay\n",
|
||||
)
|
||||
for relative_path, expected_sha256 in first["payloadSha256"].items():
|
||||
old_bytes = old.extractfile(f"payload/{relative_path}").read()
|
||||
new_bytes = new.extractfile(f"payload/{relative_path}").read()
|
||||
self.assertEqual(old_bytes, new_bytes)
|
||||
self.assertEqual(hashlib.sha256(new_bytes).hexdigest(), expected_sha256)
|
||||
|
||||
with tempfile.TemporaryDirectory(prefix="nodedc-recovery-runner-") as work:
|
||||
manifest, entries, _payload = RUNNER.load_artifact(
|
||||
first_artifact,
|
||||
Path(work),
|
||||
)
|
||||
self.assertEqual(manifest["id"], PATCH_ID)
|
||||
self.assertEqual(tuple(entries), RUNNER.ENGINE_CREDENTIAL_SINK_ARTIFACT_ENTRIES)
|
||||
self.assertEqual(RUNNER.component_services("engine", entries), ("nodedc-backend",))
|
||||
|
||||
def test_recovery_rejects_tampered_source_invalid_id_and_overwrite(self):
|
||||
with tempfile.TemporaryDirectory(prefix="nodedc-engine-sink-recovery-negative-") as directory:
|
||||
root = Path(directory)
|
||||
tampered = root / "tampered.tgz"
|
||||
shutil.copyfile(SOURCE, tampered)
|
||||
with tampered.open("ab") as stream:
|
||||
stream.write(b"tampered")
|
||||
|
||||
environment = os.environ.copy()
|
||||
environment["NODEDC_DEPLOY_ARTIFACT_DIR"] = str(root / "artifacts")
|
||||
environment["NODEDC_ENGINE_CREDENTIAL_SINK_RECOVERY_SOURCE"] = str(tampered)
|
||||
result = subprocess.run(
|
||||
["node", str(BUILDER), PATCH_ID],
|
||||
check=False,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
env=environment,
|
||||
)
|
||||
self.assertNotEqual(result.returncode, 0)
|
||||
self.assertIn("source_sha256_mismatch", result.stderr)
|
||||
|
||||
invalid = subprocess.run(
|
||||
["node", str(BUILDER), "engine-credential-sink-20260716-001"],
|
||||
check=False,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
env={
|
||||
**os.environ,
|
||||
"NODEDC_DEPLOY_ARTIFACT_DIR": str(root / "invalid"),
|
||||
"NODEDC_ENGINE_CREDENTIAL_SINK_RECOVERY_SOURCE": str(SOURCE),
|
||||
},
|
||||
)
|
||||
self.assertNotEqual(invalid.returncode, 0)
|
||||
self.assertIn("fresh-recovery-patch-id", invalid.stderr)
|
||||
|
||||
artifact_dir = root / "duplicate"
|
||||
artifact_dir.mkdir()
|
||||
self.build(artifact_dir)
|
||||
duplicate = subprocess.run(
|
||||
["node", str(BUILDER), PATCH_ID],
|
||||
check=False,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
env={
|
||||
**os.environ,
|
||||
"NODEDC_DEPLOY_ARTIFACT_DIR": str(artifact_dir),
|
||||
"NODEDC_ENGINE_CREDENTIAL_SINK_RECOVERY_SOURCE": str(SOURCE),
|
||||
},
|
||||
)
|
||||
self.assertNotEqual(duplicate.returncode, 0)
|
||||
self.assertIn("artifact_already_exists", duplicate.stderr)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main(verbosity=2)
|
||||
|
|
@ -1,6 +1,8 @@
|
|||
#!/usr/bin/env python3
|
||||
import importlib.machinery
|
||||
import importlib.util
|
||||
import hashlib
|
||||
import inspect
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
|
|
@ -8,6 +10,7 @@ import tarfile
|
|||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from unittest import mock
|
||||
|
||||
|
||||
SCRIPT_DIR = Path(__file__).resolve().parent
|
||||
|
|
@ -20,6 +23,18 @@ STAGE_ARTIFACT = (
|
|||
/ "infra/deploy-artifacts"
|
||||
/ "nodedc-n8n-private-extension-n8n-nodes-ndc-release-20260716-003.tgz"
|
||||
)
|
||||
TRANSITION_ID = "20260717-004"
|
||||
SEALED_RELEASE_ID = "0.1.2-05e4b38b14b4a019"
|
||||
SEALED_PACKAGE_SHA256 = "05e4b38b14b4a019ce1f6eee27b9e320094cb3560903bd68074966b3a1267af5"
|
||||
BUILDER_ENGINE_PATHS = (
|
||||
"nodedc-source/server/assets/n8n/schema/v2.3.2/nodes.catalog.json",
|
||||
"nodedc-source/server/assets/n8n/schema/v2.3.2/credentials.catalog.json",
|
||||
"nodedc-source/server/assets/n8n/schema/v2.3.2/meta.json",
|
||||
"nodedc-source/server/assets/n8n/icons/ndc.svg",
|
||||
"nodedc-source/server/assets/n8n/icons/ndc.dark.svg",
|
||||
"nodedc-source/services/n8n/private-extensions/ndc-activation.json",
|
||||
"nodedc-source/services/n8n/private-extensions/docker-compose.ndc-private-extension.yml",
|
||||
)
|
||||
EXPECTED_NODES = [
|
||||
"n8n-nodes-ndc.ndcDataProductPublish",
|
||||
"n8n-nodes-ndc.ndcDataProductRead",
|
||||
|
|
@ -30,6 +45,14 @@ EXPECTED_CREDENTIALS = [
|
|||
"ndcDataProductReaderApi",
|
||||
"ndcFoundryBindingApi",
|
||||
]
|
||||
PRIVATE_EXTENSION_CANON_FUNCTION_SHA256 = {
|
||||
"expected_engine_n8n_compose_override": "8ee93f3e7c407f5557c711119236449a440d73a7fcac1bbfa50a09e6a13c1cff",
|
||||
"engine_n8n_package_loader_probe_script": "2b3b3c96fad6e5e48ebea74426b21bc7eea6b6fe0e786b537da3366d18aedd9b",
|
||||
"engine_n8n_private_loader_catalog": "a2ae389b4ef215900cf967b863d01056bd2a8eec55554566d0f6005e4e0bcbac",
|
||||
"validate_engine_n8n_base_compose_source": "adacc5b20e52684cbc427362ddba5a6d2e13ef2091a89859e2b8f7bbf2e20458",
|
||||
"preflight_engine_n8n_transition": "da3cec853e3e3e20df4d1e301aab98b77815f93ec3d68b0b61c84c6e6a3fb335",
|
||||
"accept_engine_n8n_runtime": "1a2614a465161e3833e84aca402817682da1a538bdb2aecc4af66b665497338d",
|
||||
}
|
||||
|
||||
|
||||
def load_runner():
|
||||
|
|
@ -40,6 +63,32 @@ def load_runner():
|
|||
return module
|
||||
|
||||
|
||||
def snapshot_engine_builder_paths():
|
||||
snapshot = {}
|
||||
for relative in BUILDER_ENGINE_PATHS:
|
||||
path = ENGINE_ROOT / relative
|
||||
try:
|
||||
path_stat = path.lstat()
|
||||
except FileNotFoundError:
|
||||
snapshot[relative] = None
|
||||
continue
|
||||
if path.is_symlink():
|
||||
content = ("symlink", os.readlink(path))
|
||||
elif path.is_file():
|
||||
content = ("file", path.read_bytes())
|
||||
else:
|
||||
content = ("other", None)
|
||||
snapshot[relative] = (
|
||||
path_stat.st_mode,
|
||||
path_stat.st_uid,
|
||||
path_stat.st_gid,
|
||||
path_stat.st_ino,
|
||||
path_stat.st_mtime_ns,
|
||||
content,
|
||||
)
|
||||
return snapshot
|
||||
|
||||
|
||||
RUNNER = load_runner()
|
||||
|
||||
|
||||
|
|
@ -49,14 +98,21 @@ class EngineN8nPrivateExtensionTest(unittest.TestCase):
|
|||
cls.temporary = tempfile.TemporaryDirectory(prefix="nodedc-engine-n8n-policy-")
|
||||
cls.root = Path(cls.temporary.name)
|
||||
cls.results = []
|
||||
cls.engine_snapshot_before = snapshot_engine_builder_paths()
|
||||
for index in range(2):
|
||||
output = cls.root / f"build-{index}"
|
||||
output.mkdir()
|
||||
env = os.environ.copy()
|
||||
env.pop("NODEDC_N8N_TRANSITION_ID", None)
|
||||
env["NODEDC_DEPLOY_ARTIFACT_DIR"] = str(output)
|
||||
env["NODEDC_N8N_EXTENSION_STAGE_ARTIFACT"] = str(STAGE_ARTIFACT)
|
||||
command = ["node", str(BUILDER_PATH)]
|
||||
if index == 0:
|
||||
command.append(TRANSITION_ID)
|
||||
else:
|
||||
env["NODEDC_N8N_TRANSITION_ID"] = TRANSITION_ID
|
||||
result = subprocess.run(
|
||||
["node", str(BUILDER_PATH)],
|
||||
command,
|
||||
cwd=PLATFORM_ROOT,
|
||||
env=env,
|
||||
check=True,
|
||||
|
|
@ -64,6 +120,7 @@ class EngineN8nPrivateExtensionTest(unittest.TestCase):
|
|||
text=True,
|
||||
)
|
||||
cls.results.append(json.loads(result.stdout))
|
||||
cls.engine_snapshot_after = snapshot_engine_builder_paths()
|
||||
|
||||
@classmethod
|
||||
def tearDownClass(cls):
|
||||
|
|
@ -81,6 +138,69 @@ class EngineN8nPrivateExtensionTest(unittest.TestCase):
|
|||
self.assertEqual(first[4:8], b"\0\0\0\0")
|
||||
self.assertEqual(first[3] & 0x08, 0)
|
||||
|
||||
def test_successful_generation_003_runner_functions_are_frozen(self):
|
||||
actual = {
|
||||
name: hashlib.sha256(inspect.getsource(getattr(RUNNER, name)).encode("utf-8")).hexdigest()
|
||||
for name in PRIVATE_EXTENSION_CANON_FUNCTION_SHA256
|
||||
}
|
||||
self.assertEqual(actual, PRIVATE_EXTENSION_CANON_FUNCTION_SHA256)
|
||||
|
||||
def test_explicit_transition_id_does_not_mutate_sealed_release_identity(self):
|
||||
expected_ids = {
|
||||
"activation": f"engine-n8n-private-extension-{TRANSITION_ID}",
|
||||
"rollback": f"engine-n8n-private-extension-rollback-{TRANSITION_ID}",
|
||||
}
|
||||
for result in self.results:
|
||||
self.assertEqual(result["transitionId"], TRANSITION_ID)
|
||||
self.assertEqual(result["releaseId"], SEALED_RELEASE_ID)
|
||||
self.assertEqual(result["packageSha256"], SEALED_PACKAGE_SHA256)
|
||||
for kind, expected_id in expected_ids.items():
|
||||
self.assertEqual(result[kind]["id"], expected_id)
|
||||
self.assertEqual(Path(result[kind]["artifact"]).name, f"nodedc-{expected_id}.tgz")
|
||||
|
||||
def test_builder_does_not_mutate_engine_source(self):
|
||||
self.assertEqual(self.engine_snapshot_after, self.engine_snapshot_before)
|
||||
|
||||
def test_transition_id_rejects_missing_invalid_and_previously_issued_values(self):
|
||||
cases = (
|
||||
([], "transition_id_required"),
|
||||
(["20260716-003"], "transition_id_already_issued"),
|
||||
(["engine-n8n-private-extension-20260717-004"], "transition_id_invalid"),
|
||||
(["20260230-004"], "transition_id_invalid"),
|
||||
(["20260717-000"], "transition_id_invalid"),
|
||||
([TRANSITION_ID, "unexpected-second-id"], "transition_id_argument_count_invalid"),
|
||||
)
|
||||
for arguments, expected_error in cases:
|
||||
with self.subTest(arguments=arguments):
|
||||
env = os.environ.copy()
|
||||
env.pop("NODEDC_N8N_TRANSITION_ID", None)
|
||||
result = subprocess.run(
|
||||
["node", str(BUILDER_PATH), *arguments],
|
||||
cwd=PLATFORM_ROOT,
|
||||
env=env,
|
||||
check=False,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
self.assertNotEqual(result.returncode, 0)
|
||||
self.assertIn(expected_error, result.stderr)
|
||||
|
||||
def test_existing_transition_artifact_is_never_overwritten(self):
|
||||
env = os.environ.copy()
|
||||
env.pop("NODEDC_N8N_TRANSITION_ID", None)
|
||||
env["NODEDC_DEPLOY_ARTIFACT_DIR"] = str(self.artifact(0, "activation").parent)
|
||||
env["NODEDC_N8N_EXTENSION_STAGE_ARTIFACT"] = str(STAGE_ARTIFACT)
|
||||
result = subprocess.run(
|
||||
["node", str(BUILDER_PATH), TRANSITION_ID],
|
||||
cwd=PLATFORM_ROOT,
|
||||
env=env,
|
||||
check=False,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
self.assertNotEqual(result.returncode, 0)
|
||||
self.assertIn("transition_artifact_already_exists", result.stderr)
|
||||
|
||||
def test_activation_and_rollback_pass_strict_runner_policy(self):
|
||||
expected = {"activation": ("activate", 7), "rollback": ("rollback-inactive", 4)}
|
||||
for kind, (action, entry_count) in expected.items():
|
||||
|
|
@ -137,12 +257,213 @@ class EngineN8nPrivateExtensionTest(unittest.TestCase):
|
|||
)
|
||||
override = (payload / RUNNER.ENGINE_N8N_COMPOSE_OVERRIDE_REL).read_text()
|
||||
self.assertEqual(override, RUNNER.expected_engine_n8n_compose_override(descriptor))
|
||||
self.assertEqual(
|
||||
hashlib.sha256(override.encode("utf-8")).hexdigest(),
|
||||
"a29e2f92b87dda59da2b4e3ce250bc9705ed9fa551c0fbe4e95464fca5caa3e1",
|
||||
)
|
||||
self.assertIn("pull_policy: never", override)
|
||||
self.assertIn("N8N_USER_FOLDER: /home/node", override)
|
||||
self.assertNotIn("NODE_PATH", override)
|
||||
self.assertIn(":/home/node/.n8n/nodes/node_modules/n8n-nodes-ndc:ro", override)
|
||||
self.assertNotIn("N8N_CUSTOM_EXTENSIONS", override)
|
||||
self.assertNotIn("build:", override)
|
||||
|
||||
def test_generation_003_override_is_accepted_as_installed_canon(self):
|
||||
original_root = RUNNER.COMPONENTS["engine"]["payload_root"]
|
||||
original_compose = RUNNER.COMPONENTS["engine"]["compose_root"]
|
||||
try:
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
work = Path(directory)
|
||||
extracted = work / "artifact"
|
||||
extracted.mkdir()
|
||||
_manifest, _entries, payload = RUNNER.load_artifact(
|
||||
self.artifact(0, "activation"),
|
||||
extracted,
|
||||
)
|
||||
engine_root = work / "engine"
|
||||
engine_root.mkdir()
|
||||
(engine_root / "docker-compose.yml").write_text("services:\n", encoding="utf-8")
|
||||
for relative in (
|
||||
RUNNER.ENGINE_N8N_TRANSITION_DESCRIPTOR_REL,
|
||||
RUNNER.ENGINE_N8N_COMPOSE_OVERRIDE_REL,
|
||||
):
|
||||
target = engine_root / relative
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
target.write_bytes((payload / relative).read_bytes())
|
||||
RUNNER.COMPONENTS["engine"]["payload_root"] = engine_root
|
||||
RUNNER.COMPONENTS["engine"]["compose_root"] = engine_root
|
||||
compose_files = RUNNER.component_compose_files("engine")
|
||||
self.assertEqual(compose_files, (
|
||||
engine_root / "docker-compose.yml",
|
||||
engine_root / RUNNER.ENGINE_N8N_COMPOSE_OVERRIDE_REL,
|
||||
))
|
||||
finally:
|
||||
RUNNER.COMPONENTS["engine"]["payload_root"] = original_root
|
||||
RUNNER.COMPONENTS["engine"]["compose_root"] = original_compose
|
||||
|
||||
def test_loader_probe_pins_n8n_node_path(self):
|
||||
container = {
|
||||
"Mounts": [{
|
||||
"Destination": RUNNER.ENGINE_N8N_RUNTIME_PACKAGE_PATH,
|
||||
"RW": False,
|
||||
}],
|
||||
}
|
||||
completed = subprocess.CompletedProcess(
|
||||
args=[],
|
||||
returncode=0,
|
||||
stdout=json.dumps({
|
||||
"packageName": "n8n-nodes-ndc",
|
||||
"nodes": ["ndcDataProductPublish", "ndcDataProductRead", "ndcFoundryBinding"],
|
||||
"credentials": EXPECTED_CREDENTIALS,
|
||||
}),
|
||||
stderr="",
|
||||
)
|
||||
with (
|
||||
mock.patch.object(RUNNER, "inspect_container", return_value=container),
|
||||
mock.patch.object(RUNNER.subprocess, "run", return_value=completed) as run,
|
||||
):
|
||||
catalog = RUNNER.engine_n8n_private_loader_catalog("n8n-container")
|
||||
|
||||
command = run.call_args.args[0]
|
||||
self.assertEqual(command[:-1], [
|
||||
str(RUNNER.DOCKER),
|
||||
"exec",
|
||||
"n8n-container",
|
||||
"node",
|
||||
"-e",
|
||||
])
|
||||
probe_script = command[-1]
|
||||
self.assertIn(
|
||||
f"process.env.NODE_PATH='{RUNNER.ENGINE_N8N_NODE_MODULES_PATH}'",
|
||||
probe_script,
|
||||
)
|
||||
self.assertIn("Module._initPaths()", probe_script)
|
||||
self.assertIn("PackageDirectoryLoader", probe_script)
|
||||
self.assertEqual(catalog, {
|
||||
"node_types": EXPECTED_NODES,
|
||||
"credential_types": EXPECTED_CREDENTIALS,
|
||||
})
|
||||
|
||||
def test_loader_probe_failure_emits_only_allowlisted_diagnostic(self):
|
||||
container = {
|
||||
"Mounts": [{
|
||||
"Destination": RUNNER.ENGINE_N8N_RUNTIME_PACKAGE_PATH,
|
||||
"RW": False,
|
||||
}],
|
||||
}
|
||||
cases = (
|
||||
("MODULE_NOT_FOUND", "MODULE_NOT_FOUND"),
|
||||
("Bearer secret-that-must-not-be-logged", "PROBE_ERROR"),
|
||||
)
|
||||
for probe_stderr, expected_category in cases:
|
||||
with (
|
||||
self.subTest(stderr=probe_stderr),
|
||||
mock.patch.object(RUNNER, "inspect_container", return_value=container),
|
||||
mock.patch.object(
|
||||
RUNNER.subprocess,
|
||||
"run",
|
||||
return_value=subprocess.CompletedProcess(
|
||||
args=[],
|
||||
returncode=1,
|
||||
stdout="",
|
||||
stderr=probe_stderr,
|
||||
),
|
||||
),
|
||||
):
|
||||
with self.assertRaises(RUNNER.DeployError) as raised:
|
||||
RUNNER.engine_n8n_private_loader_catalog("n8n-container")
|
||||
message = str(raised.exception)
|
||||
self.assertIn(f"category={expected_category}", message)
|
||||
self.assertNotIn("secret-that-must-not-be-logged", message)
|
||||
|
||||
def test_active_runtime_acceptance_does_not_require_node_path(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
_manifest, _entries, payload = RUNNER.load_artifact(
|
||||
self.artifact(0, "activation"),
|
||||
Path(directory),
|
||||
)
|
||||
descriptor = RUNNER.read_engine_n8n_transition_descriptor(
|
||||
payload / RUNNER.ENGINE_N8N_TRANSITION_DESCRIPTOR_REL
|
||||
)
|
||||
container = {
|
||||
"State": {"Status": "running", "StartedAt": "2026-07-16T00:00:00Z"},
|
||||
"Image": "sha256:base-image",
|
||||
"Config": {
|
||||
"Env": [
|
||||
"N8N_USER_FOLDER=/home/node",
|
||||
"N8N_COMMUNITY_PACKAGES_ENABLED=true",
|
||||
"N8N_COMMUNITY_PACKAGES_PREVENT_LOADING=false",
|
||||
"N8N_REINSTALL_MISSING_PACKAGES=false",
|
||||
],
|
||||
"Labels": {
|
||||
"nodedc.n8n-private-extension.release": descriptor["releaseId"],
|
||||
"nodedc.n8n-private-extension.package-sha256": descriptor["packageSha256"],
|
||||
},
|
||||
},
|
||||
"Mounts": [{
|
||||
"Destination": RUNNER.ENGINE_N8N_RUNTIME_PACKAGE_PATH,
|
||||
"Source": f"/volume2/nodedc-demo/{descriptor['sealedReleaseRelativePath']}",
|
||||
"RW": False,
|
||||
}],
|
||||
"RestartCount": 0,
|
||||
}
|
||||
log_result = subprocess.CompletedProcess(args=[], returncode=0, stdout="", stderr="")
|
||||
with (
|
||||
mock.patch.object(
|
||||
RUNNER,
|
||||
"inspect_engine_n8n_base_image",
|
||||
return_value={"id": "sha256:base-image"},
|
||||
),
|
||||
mock.patch.object(RUNNER, "engine_n8n_container_ids", return_value=("n8n-container",)),
|
||||
mock.patch.object(RUNNER, "inspect_container", return_value=container),
|
||||
mock.patch.object(RUNNER, "wait_engine_n8n_readiness"),
|
||||
mock.patch.object(RUNNER, "engine_n8n_version", return_value=RUNNER.ENGINE_N8N_VERSION),
|
||||
mock.patch.object(RUNNER.subprocess, "run", return_value=log_result),
|
||||
mock.patch.object(
|
||||
RUNNER,
|
||||
"engine_n8n_private_loader_catalog",
|
||||
return_value={
|
||||
"node_types": EXPECTED_NODES,
|
||||
"credential_types": EXPECTED_CREDENTIALS,
|
||||
},
|
||||
),
|
||||
mock.patch.object(RUNNER, "validate_engine_n8n_catalog_payload"),
|
||||
mock.patch.object(RUNNER.time, "sleep"),
|
||||
):
|
||||
accepted = RUNNER.accept_engine_n8n_runtime(descriptor)
|
||||
self.assertEqual(accepted["node_types"], EXPECTED_NODES)
|
||||
|
||||
def test_private_extension_transition_does_not_enter_publish_grant_domain(self):
|
||||
entries = (RUNNER.ENGINE_N8N_TRANSITION_DESCRIPTOR_REL,)
|
||||
descriptor = {"action": "rollback-inactive"}
|
||||
self.assertFalse(RUNNER.touches_engine_data_product_publish_grant(entries))
|
||||
self.assertEqual(RUNNER.component_services("engine", entries), ("n8n",))
|
||||
with (
|
||||
mock.patch.object(RUNNER, "current_engine_n8n_transition_descriptor", return_value=descriptor),
|
||||
mock.patch.object(RUNNER, "ensure_engine_edp_managed_provisioner_keypair") as keypair,
|
||||
mock.patch.object(RUNNER, "ensure_engine_publish_grant_private_state") as private_state,
|
||||
):
|
||||
RUNNER.prepare_component_runtime("engine", entries)
|
||||
keypair.assert_not_called()
|
||||
private_state.assert_not_called()
|
||||
|
||||
def test_generic_engine_compose_artifact_keeps_legacy_service_selection(self):
|
||||
self.assertEqual(
|
||||
RUNNER.component_services("engine", ("docker-compose.yml",)),
|
||||
RUNNER.COMPONENTS["engine"]["services"],
|
||||
)
|
||||
self.assertEqual(
|
||||
RUNNER.component_services(
|
||||
"engine",
|
||||
("docker-compose.yml", "nodedc-source/server/dataProductPublishGrant/store.js"),
|
||||
),
|
||||
RUNNER.COMPONENTS["engine"]["services"],
|
||||
)
|
||||
self.assertFalse(RUNNER.is_engine_data_product_publish_grant_slice(
|
||||
"engine",
|
||||
("docker-compose.yml", "nodedc-source/server/dataProductPublishGrant/store.js"),
|
||||
))
|
||||
|
||||
def test_unknown_descriptor_key_is_rejected(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
work = Path(directory)
|
||||
|
|
|
|||
|
|
@ -0,0 +1,156 @@
|
|||
#!/usr/bin/env python3
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import tarfile
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
SCRIPT_DIR = Path(__file__).resolve().parent
|
||||
BUILDER = SCRIPT_DIR / "build-engine-data-product-publish-grant-artifact.mjs"
|
||||
EXPECTED_ENTRIES = [
|
||||
"nodedc-source/server/assets/provider-packages/v1/catalog.json",
|
||||
"nodedc-source/server/dataProductPublishGrant",
|
||||
"nodedc-source/server/engineAgents/store.js",
|
||||
"nodedc-source/server/routes/engineAgentGateway.js",
|
||||
"nodedc-source/server/routes/n8n.js",
|
||||
"nodedc-source/services/backend/data-product-publish-grant/docker-compose.immutable-runtime.yml",
|
||||
]
|
||||
PATCH_ID = "engine-data-product-publish-grant-20990101-001"
|
||||
SUCCESSFUL_CREDENTIAL_SINK_INDEX_SHA256 = (
|
||||
"b2b790b02839570d967a2ca68b00e2724485a99389ac9b3a589a1b22302a36b8"
|
||||
)
|
||||
|
||||
|
||||
class EnginePublishGrantArtifactTest(unittest.TestCase):
|
||||
def build(self, artifact_dir, patch_id):
|
||||
environment = os.environ.copy()
|
||||
environment["NODEDC_DEPLOY_ARTIFACT_DIR"] = str(artifact_dir)
|
||||
result = subprocess.run(
|
||||
["node", str(BUILDER), patch_id],
|
||||
check=True,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
env=environment,
|
||||
)
|
||||
return json.loads(result.stdout)
|
||||
|
||||
def test_artifact_is_narrow_secret_free_and_deterministic(self):
|
||||
with tempfile.TemporaryDirectory(prefix="nodedc-engine-publish-artifact-") as directory:
|
||||
root = Path(directory)
|
||||
first_dir = root / "first"
|
||||
second_dir = root / "second"
|
||||
first_dir.mkdir()
|
||||
second_dir.mkdir()
|
||||
first = self.build(first_dir, PATCH_ID)
|
||||
artifact = Path(first["artifact"])
|
||||
first_bytes = artifact.read_bytes()
|
||||
second = self.build(second_dir, PATCH_ID)
|
||||
second_bytes = Path(second["artifact"]).read_bytes()
|
||||
|
||||
self.assertEqual(first["entries"], EXPECTED_ENTRIES)
|
||||
self.assertEqual(first["sha256"], hashlib.sha256(first_bytes).hexdigest())
|
||||
self.assertEqual(first["sha256"], second["sha256"])
|
||||
self.assertEqual(first_bytes, second_bytes)
|
||||
|
||||
with tarfile.open(artifact, "r:gz") as archive:
|
||||
names = {member.name for member in archive.getmembers()}
|
||||
files = archive.extractfile("files.txt").read().decode("utf-8").splitlines()
|
||||
manifest = archive.extractfile("manifest.env").read().decode("utf-8")
|
||||
n8n_source = archive.extractfile(
|
||||
"payload/nodedc-source/server/routes/n8n.js"
|
||||
).read().decode("utf-8")
|
||||
gateway_source = archive.extractfile(
|
||||
"payload/nodedc-source/server/routes/engineAgentGateway.js"
|
||||
).read().decode("utf-8")
|
||||
runtime_override = archive.extractfile(
|
||||
"payload/nodedc-source/services/backend/data-product-publish-grant/"
|
||||
"docker-compose.immutable-runtime.yml"
|
||||
).read().decode("utf-8")
|
||||
|
||||
self.assertEqual(files, EXPECTED_ENTRIES)
|
||||
self.assertEqual(
|
||||
manifest,
|
||||
f"id={PATCH_ID}\ncomponent=engine\ntype=app-overlay\n",
|
||||
)
|
||||
forbidden_roots = (
|
||||
"payload/docker-compose.yml",
|
||||
"payload/nodedc-source/server/index.js",
|
||||
"payload/nodedc-source/server/credentialPolicies/ndcPrivateNode.js",
|
||||
"payload/nodedc-source/server/credentialSink/",
|
||||
"payload/nodedc-source/server/routes/engineCredentialSink.js",
|
||||
"payload/nodedc-source/server/middleware/demoAccess.js",
|
||||
"payload/nodedc-source/server/routes/ndcAgentMcp.js",
|
||||
"payload/nodedc-source/server/tests/",
|
||||
"payload/nodedc-source/dist/",
|
||||
"payload/nodedc-source/server/data/",
|
||||
)
|
||||
self.assertFalse(any(
|
||||
name == root.rstrip("/") or name.startswith(root)
|
||||
for name in names
|
||||
for root in forbidden_roots
|
||||
))
|
||||
self.assertNotIn("from '../credentialSink/", n8n_source)
|
||||
self.assertIn("engineCredentialSinkN8nAdapter", n8n_source)
|
||||
self.assertIn("engineDataProductPublishGrantN8nAdapter", n8n_source)
|
||||
for tool in (
|
||||
"engine_plan_data_product_publish_grant",
|
||||
"engine_apply_data_product_publish_grant",
|
||||
"engine_accept_data_product_publish_grant",
|
||||
"engine_rollback_data_product_publish_grant",
|
||||
):
|
||||
self.assertIn(tool, gateway_source)
|
||||
self.assertIn(" nodedc-backend:", runtime_override)
|
||||
self.assertNotIn(" n8n:", runtime_override)
|
||||
self.assertIn(
|
||||
"source: /volume2/nodedc-demo/nodedc-control-plane/publish-grants",
|
||||
runtime_override,
|
||||
)
|
||||
self.assertNotIn(
|
||||
"source: /volume2/nodedc-demo/nodedc-control-plane\n",
|
||||
runtime_override,
|
||||
)
|
||||
self.assertIn("read_only: true", runtime_override)
|
||||
self.assertEqual(runtime_override.count("create_host_path: false"), 2)
|
||||
|
||||
def test_builder_requires_a_fresh_never_issued_patch_id(self):
|
||||
with tempfile.TemporaryDirectory(prefix="nodedc-engine-publish-id-") as directory:
|
||||
artifact_dir = Path(directory)
|
||||
environment = os.environ.copy()
|
||||
environment["NODEDC_DEPLOY_ARTIFACT_DIR"] = str(artifact_dir)
|
||||
cases = (
|
||||
([], "fresh-patch-id"),
|
||||
(["engine-data-product-publish-grant-20260716-001"], "patch_id_already_issued"),
|
||||
(["engine-data-product-publish-grant-20260717-001"], "patch_id_already_issued"),
|
||||
(["engine-data-product-publish-grant-20260717-002"], "patch_id_already_issued"),
|
||||
(["engine-publish-grant-unit-001"], "fresh-patch-id"),
|
||||
)
|
||||
for arguments, expected in cases:
|
||||
with self.subTest(arguments=arguments):
|
||||
result = subprocess.run(
|
||||
["node", str(BUILDER), *arguments],
|
||||
check=False,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
env=environment,
|
||||
)
|
||||
self.assertNotEqual(result.returncode, 0)
|
||||
self.assertIn(expected, result.stderr)
|
||||
|
||||
self.build(artifact_dir, PATCH_ID)
|
||||
duplicate = subprocess.run(
|
||||
["node", str(BUILDER), PATCH_ID],
|
||||
check=False,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
env=environment,
|
||||
)
|
||||
self.assertNotEqual(duplicate.returncode, 0)
|
||||
self.assertIn("artifact_already_exists", duplicate.stderr)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main(verbosity=2)
|
||||
|
|
@ -0,0 +1,88 @@
|
|||
#!/usr/bin/env python3
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import tarfile
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
SCRIPT_DIR = Path(__file__).resolve().parent
|
||||
BUILDER = SCRIPT_DIR / "build-external-data-plane-artifact.mjs"
|
||||
EXPECTED_ENTRIES = [
|
||||
"platform/docker-compose.external-data-plane.yml",
|
||||
"platform/services/external-data-plane",
|
||||
"platform/packages/external-provider-contract/package.json",
|
||||
"platform/packages/external-provider-contract/src/contract-version.mjs",
|
||||
"platform/packages/external-provider-contract/src/data-plane.mjs",
|
||||
"platform/packages/external-provider-contract/src/data-product.mjs",
|
||||
"platform/packages/external-provider-contract/src/intake-batch.mjs",
|
||||
"platform/packages/external-provider-contract/src/sensitive-field-policy.mjs",
|
||||
]
|
||||
|
||||
|
||||
class ExternalDataPlaneArtifactTest(unittest.TestCase):
|
||||
def build(self, artifact_dir, patch_id):
|
||||
environment = os.environ.copy()
|
||||
environment["NODEDC_DEPLOY_ARTIFACT_DIR"] = str(artifact_dir)
|
||||
result = subprocess.run(
|
||||
["node", str(BUILDER), patch_id],
|
||||
check=True,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
env=environment,
|
||||
)
|
||||
return json.loads(result.stdout)
|
||||
|
||||
def test_artifact_is_runtime_only_public_trust_and_deterministic(self):
|
||||
with tempfile.TemporaryDirectory(prefix="nodedc-edp-artifact-") as directory:
|
||||
artifact_dir = Path(directory)
|
||||
first = self.build(artifact_dir, "edp-managed-publish-unit-001")
|
||||
artifact = Path(first["artifact"])
|
||||
first_bytes = artifact.read_bytes()
|
||||
second = self.build(artifact_dir, "edp-managed-publish-unit-001")
|
||||
second_bytes = Path(second["artifact"]).read_bytes()
|
||||
|
||||
self.assertEqual(first["entries"], EXPECTED_ENTRIES)
|
||||
self.assertEqual(first["sha256"], hashlib.sha256(first_bytes).hexdigest())
|
||||
self.assertEqual(first["sha256"], second["sha256"])
|
||||
self.assertEqual(first_bytes, second_bytes)
|
||||
|
||||
with tarfile.open(artifact, "r:gz") as archive:
|
||||
members = archive.getmembers()
|
||||
names = {member.name for member in members}
|
||||
files = archive.extractfile("files.txt").read().decode("utf-8").splitlines()
|
||||
manifest = archive.extractfile("manifest.env").read().decode("utf-8")
|
||||
compose = archive.extractfile(
|
||||
"payload/platform/docker-compose.external-data-plane.yml"
|
||||
).read().decode("utf-8")
|
||||
regular_payloads = [
|
||||
archive.extractfile(member).read()
|
||||
for member in members
|
||||
if member.isfile()
|
||||
]
|
||||
|
||||
self.assertEqual(files, EXPECTED_ENTRIES)
|
||||
self.assertEqual(
|
||||
manifest,
|
||||
"id=edp-managed-publish-unit-001\ncomponent=platform\ntype=app-overlay\n",
|
||||
)
|
||||
self.assertFalse(any(
|
||||
name.startswith("payload/platform/services/external-data-plane/test/")
|
||||
or "/node_modules/" in name
|
||||
or Path(name).name.startswith(".env")
|
||||
or Path(name).name == "private-key.pem"
|
||||
for name in names
|
||||
))
|
||||
self.assertIn(
|
||||
"source: /volume1/docker/nodedc-platform/trust/engine-managed-provisioner",
|
||||
compose,
|
||||
)
|
||||
self.assertNotIn("private-key.pem", compose)
|
||||
self.assertNotIn(b"-----BEGIN PRIVATE KEY-----", b"\n".join(regular_payloads))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main(verbosity=2)
|
||||
|
|
@ -0,0 +1,926 @@
|
|||
#!/usr/bin/env python3
|
||||
import importlib.machinery
|
||||
import importlib.util
|
||||
import hashlib
|
||||
import inspect
|
||||
import json
|
||||
import stat
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from unittest import mock
|
||||
|
||||
|
||||
SCRIPT_DIR = Path(__file__).resolve().parent
|
||||
RUNNER_PATH = SCRIPT_DIR / "nodedc-deploy"
|
||||
|
||||
|
||||
def load_runner():
|
||||
loader = importlib.machinery.SourceFileLoader(
|
||||
"nodedc_platform_deploy_under_test",
|
||||
str(RUNNER_PATH),
|
||||
)
|
||||
spec = importlib.util.spec_from_loader(loader.name, loader)
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
RUNNER = load_runner()
|
||||
|
||||
LEGACY_CREDENTIAL_RUNTIME_CANON_FUNCTION_SHA256 = {
|
||||
"build_canonical_engine_backend_rootfs": "7f8fca6380632d717ee38a3d58b5bbc58d34b0e6c81e2764a759ddd63bfd07d0",
|
||||
"engine_backend_container_id": "08dc25b6fee37e8c0d942df91fc575d6766b99c306ad06c2c14afded8de3b375",
|
||||
"engine_backend_immutable_runtime_is_current": "9c6b9a73ac5004830b1f06f56bfc1f25336d11001a97a5de7a04388e10a20990",
|
||||
"engine_backend_node_modules_tree_sha256": "01749704e1624d247bdf41242aa71961f5e5bc092ffa906b69cdb911907b70c4",
|
||||
"engine_backend_rootfs_member_allowed": "62342e5891a1ea885f1ff02169a100f9ca94464eb7dfb62a7cbe67750ebcbb91",
|
||||
"engine_backend_rootfs_toolchain_link_is_omitted": "83772bc1a8aa654e5c1d0757d43b650ee978dc73aef7ae8fac567dbacc8c7c49",
|
||||
"engine_backend_tool_versions": "c21d7c11ee8c40ac5e3e5c6c24c441d1ce52c9dd2c22fc1d8f26e121138fa78a",
|
||||
"ensure_engine_backend_release_image": "f65a5f8573b911980f252dc3beed88cd6ee14e0f0c519c333ea90b78c8910953",
|
||||
"expected_engine_credential_backend_override": "b7c9db0059a4d262b8fe61dc0745b8ee1a4b75d56d227c39ba08115c7e414486",
|
||||
"inspect_engine_backend_derived_image": "2336ce8068bd321b6c8ac6e62b430dbcbf991201cb9b4178589294d1baf7e818",
|
||||
"preflight_engine_credential_sink_ready": "3f084ade6346644c86aa9540fbcf70b3a44e32c7add0876935f45177569aad6d",
|
||||
"prepare_engine_credential_backend_runtime": "0def464639767dcf85acd7978459b2f5c1ebf963424dd401d4c29aae7bb1aaa5",
|
||||
"quarantine_engine_backend_partial_runtime": "57e455f874c681815cf42bf529d0b8620ae8317e82551d784cbd6dae913603d2",
|
||||
"read_engine_backend_runtime_metadata": "7ee0ec2c72eb699db4b6a7d3d8d912f1b0c0c724c2a054eee223fba3ec0a1cb2",
|
||||
"run_compose": "8425528d4934f82e79ebf3551c4246505e4a3cdf3f450949d7618a4312fee169",
|
||||
"run_engine_backend_probe": "f0db7210b64cf3c80c1f0a00165f27dcab6962781a69dc2050d84a5373cedaa9",
|
||||
"touches_engine_credential_sink": "5d0e0d680d1a8508fb1ac21d8c8b3aa92466fe8b219d2ddc1fe5411f94fca912",
|
||||
"validate_engine_backend_activation_marker": "66c4d5d9b0b608c030c2c9c97e2e46e1f1634b7ba024dc107c15478091765483",
|
||||
"validate_engine_backend_dependencies": "ca1383dffd5a6397ecfeedcc8f88ab59e90a33d5089590f0249f789b05ed29e2",
|
||||
"validate_engine_backend_mounts": "5614267d1d0f734dd68ed7565292a53f34777b6dc3640137034f6db093ef7431",
|
||||
"validate_engine_backend_runtime_override_file": "eae1d32ab75cc59916dbd1d4cd73a072c156abf9014d8ef234fcd2333748139d",
|
||||
"validate_engine_credential_sink_slice": "87e6571baf47c30af120da32a327a88164415bf65b198e7190c28df7365bcfd5",
|
||||
}
|
||||
|
||||
|
||||
class CanonicalPlatformRegistryTest(unittest.TestCase):
|
||||
def test_legacy_credential_runtime_functions_match_last_successful_canon(self):
|
||||
actual = {
|
||||
name: hashlib.sha256(
|
||||
inspect.getsource(getattr(RUNNER, name)).encode("utf-8"),
|
||||
).hexdigest()
|
||||
for name in LEGACY_CREDENTIAL_RUNTIME_CANON_FUNCTION_SHA256
|
||||
}
|
||||
self.assertEqual(actual, LEGACY_CREDENTIAL_RUNTIME_CANON_FUNCTION_SHA256)
|
||||
|
||||
def test_publish_mount_extension_delegates_baseline_inventory_to_legacy_guard(self):
|
||||
baseline_mounts = [
|
||||
{
|
||||
"Type": "bind",
|
||||
"Source": "/baseline/app",
|
||||
"Destination": "/app",
|
||||
"RW": True,
|
||||
},
|
||||
]
|
||||
publish_mounts = [
|
||||
{
|
||||
"Type": "bind",
|
||||
"Source": str(RUNNER.ENGINE_PUBLISH_GRANT_STATE_PATH),
|
||||
"Destination": RUNNER.ENGINE_PUBLISH_GRANT_CONTAINER_PATH,
|
||||
"RW": True,
|
||||
},
|
||||
{
|
||||
"Type": "bind",
|
||||
"Source": str(RUNNER.ENGINE_EDP_MANAGED_PROVISIONER_PRIVATE_KEY_FILE),
|
||||
"Destination": RUNNER.ENGINE_EDP_PRIVATE_KEY_CONTAINER_PATH,
|
||||
"RW": False,
|
||||
},
|
||||
]
|
||||
container = {"Mounts": [*baseline_mounts, *publish_mounts]}
|
||||
with tempfile.TemporaryDirectory(prefix="nodedc-publish-mounts-") as directory:
|
||||
root = Path(directory)
|
||||
override = root / RUNNER.ENGINE_DATA_PRODUCT_PUBLISH_GRANT_OVERRIDE_REL
|
||||
override.parent.mkdir(parents=True)
|
||||
override.write_text(
|
||||
RUNNER.expected_engine_data_product_publish_grant_override(),
|
||||
encoding="utf-8",
|
||||
)
|
||||
original_root = RUNNER.COMPONENTS["engine"]["payload_root"]
|
||||
RUNNER.COMPONENTS["engine"]["payload_root"] = root
|
||||
try:
|
||||
with mock.patch.object(
|
||||
RUNNER,
|
||||
"validate_engine_backend_mounts",
|
||||
) as legacy_guard:
|
||||
RUNNER.validate_engine_backend_mounts_for_installed_runtime(
|
||||
container,
|
||||
True,
|
||||
)
|
||||
legacy_guard.assert_called_once_with(
|
||||
{"Mounts": baseline_mounts},
|
||||
True,
|
||||
)
|
||||
|
||||
wrong_source = {
|
||||
"Mounts": [
|
||||
*baseline_mounts,
|
||||
{**publish_mounts[0], "Source": "/wrong"},
|
||||
publish_mounts[1],
|
||||
],
|
||||
}
|
||||
with self.assertRaisesRegex(
|
||||
RUNNER.DeployError,
|
||||
"Publish mount barrier mismatch",
|
||||
):
|
||||
RUNNER.validate_engine_backend_mounts_for_installed_runtime(
|
||||
wrong_source,
|
||||
True,
|
||||
)
|
||||
finally:
|
||||
RUNNER.COMPONENTS["engine"]["payload_root"] = original_root
|
||||
|
||||
def test_publish_mount_extension_accepts_exact_complete_runtime_inventory(self):
|
||||
baseline = {
|
||||
"/app": (True, "/engine/source"),
|
||||
"/app/deploy/docker-compose.yml": (False, "/engine/docker-compose.yml"),
|
||||
"/seed-api": (False, "/engine/seed-api"),
|
||||
"/seed-data": (False, "/engine/seed-data"),
|
||||
"/app/node_modules": (False, str(RUNNER.ENGINE_CREDENTIAL_BACKEND_NODE_MODULES_DIR)),
|
||||
"/app/server/data": (True, "/engine/data"),
|
||||
"/app/server/storage": (True, "/engine/storage"),
|
||||
"/app/server/logs": (True, "/engine/logs"),
|
||||
"/var/run/docker.sock": (True, "/var/run/docker.sock"),
|
||||
}
|
||||
mounts = [
|
||||
{
|
||||
"Type": "bind",
|
||||
"Source": source,
|
||||
"Destination": destination,
|
||||
"RW": writable,
|
||||
}
|
||||
for destination, (writable, source) in baseline.items()
|
||||
]
|
||||
mounts.extend((
|
||||
{
|
||||
"Type": "bind",
|
||||
"Source": str(RUNNER.ENGINE_PUBLISH_GRANT_STATE_PATH),
|
||||
"Destination": RUNNER.ENGINE_PUBLISH_GRANT_CONTAINER_PATH,
|
||||
"RW": True,
|
||||
},
|
||||
{
|
||||
"Type": "bind",
|
||||
"Source": str(RUNNER.ENGINE_EDP_MANAGED_PROVISIONER_PRIVATE_KEY_FILE),
|
||||
"Destination": RUNNER.ENGINE_EDP_PRIVATE_KEY_CONTAINER_PATH,
|
||||
"RW": False,
|
||||
},
|
||||
))
|
||||
with tempfile.TemporaryDirectory(prefix="nodedc-publish-full-mounts-") as directory:
|
||||
root = Path(directory)
|
||||
override = root / RUNNER.ENGINE_DATA_PRODUCT_PUBLISH_GRANT_OVERRIDE_REL
|
||||
override.parent.mkdir(parents=True)
|
||||
override.write_text(
|
||||
RUNNER.expected_engine_data_product_publish_grant_override(),
|
||||
encoding="utf-8",
|
||||
)
|
||||
original_root = RUNNER.COMPONENTS["engine"]["payload_root"]
|
||||
RUNNER.COMPONENTS["engine"]["payload_root"] = root
|
||||
try:
|
||||
RUNNER.validate_engine_backend_mounts_for_installed_runtime(
|
||||
{"Mounts": mounts},
|
||||
True,
|
||||
)
|
||||
finally:
|
||||
RUNNER.COMPONENTS["engine"]["payload_root"] = original_root
|
||||
|
||||
def test_absent_publish_overlay_uses_legacy_mount_guard_unchanged(self):
|
||||
container = {"Mounts": [{"Destination": "/app"}]}
|
||||
with tempfile.TemporaryDirectory(prefix="nodedc-no-publish-mounts-") as directory:
|
||||
original_root = RUNNER.COMPONENTS["engine"]["payload_root"]
|
||||
RUNNER.COMPONENTS["engine"]["payload_root"] = Path(directory)
|
||||
try:
|
||||
with mock.patch.object(
|
||||
RUNNER,
|
||||
"validate_engine_backend_mounts",
|
||||
) as legacy_guard:
|
||||
RUNNER.validate_engine_backend_mounts_for_installed_runtime(
|
||||
container,
|
||||
False,
|
||||
)
|
||||
legacy_guard.assert_called_once_with(container, False)
|
||||
finally:
|
||||
RUNNER.COMPONENTS["engine"]["payload_root"] = original_root
|
||||
|
||||
def test_runner_does_not_reinterpret_compose_schema(self):
|
||||
self.assertFalse(hasattr(RUNNER, "render_compose_model"))
|
||||
self.assertFalse(hasattr(RUNNER, "validate_platform_compose_model"))
|
||||
self.assertFalse(hasattr(RUNNER, "validate_engine_compose_model"))
|
||||
self.assertFalse(hasattr(RUNNER, "preflight_platform_compose_candidate"))
|
||||
self.assertFalse(hasattr(RUNNER, "preflight_engine_compose_candidate"))
|
||||
|
||||
def test_platform_edp_change_recreates_only_application(self):
|
||||
entries = (
|
||||
"platform/docker-compose.external-data-plane.yml",
|
||||
"platform/services/external-data-plane",
|
||||
)
|
||||
services = RUNNER.component_services("platform", entries)
|
||||
self.assertEqual(services, ("external-data-plane",))
|
||||
self.assertNotIn("external-data-plane-postgres", services)
|
||||
|
||||
builds = RUNNER.component_builds("platform", entries)
|
||||
self.assertEqual(len(builds), 1)
|
||||
build_root, build_args = builds[0]
|
||||
self.assertEqual(
|
||||
build_root,
|
||||
Path("/volume1/docker/nodedc-platform/platform"),
|
||||
)
|
||||
self.assertEqual(
|
||||
build_args,
|
||||
(
|
||||
"build",
|
||||
"--no-cache",
|
||||
"-f",
|
||||
"services/external-data-plane/Dockerfile",
|
||||
"-t",
|
||||
RUNNER.EXTERNAL_DATA_PLANE_IMAGE,
|
||||
".",
|
||||
),
|
||||
)
|
||||
|
||||
def test_provider_catalog_is_metadata_only(self):
|
||||
entries = (
|
||||
"platform/packages/external-provider-contract/providers/gelios/v1/provider.json",
|
||||
)
|
||||
self.assertFalse(RUNNER.touches_external_data_plane_files(entries))
|
||||
self.assertEqual(RUNNER.component_services("platform", entries), ())
|
||||
self.assertEqual(RUNNER.component_builds("platform", entries), ())
|
||||
self.assertEqual(RUNNER.component_healthchecks("platform", entries), ())
|
||||
|
||||
with (
|
||||
mock.patch.object(RUNNER, "run_build") as build,
|
||||
mock.patch.object(RUNNER, "prepare_component_runtime") as prepare,
|
||||
mock.patch.object(RUNNER, "run_compose") as compose,
|
||||
mock.patch.object(RUNNER, "healthcheck_url") as healthcheck,
|
||||
):
|
||||
RUNNER.run_component_runtime("platform", entries, ())
|
||||
RUNNER.run_healthchecks("platform", entries, ())
|
||||
|
||||
build.assert_not_called()
|
||||
prepare.assert_not_called()
|
||||
compose.assert_not_called()
|
||||
healthcheck.assert_not_called()
|
||||
|
||||
def test_generic_engine_compose_change_keeps_legacy_service_selection(self):
|
||||
services = RUNNER.component_services(
|
||||
"engine",
|
||||
(
|
||||
"docker-compose.yml",
|
||||
"nodedc-source/server/index.js",
|
||||
"nodedc-source/server/dataProductPublishGrant/store.js",
|
||||
),
|
||||
)
|
||||
self.assertEqual(services, ("nodedc-backend", "app"))
|
||||
self.assertNotIn("n8n-postgres", services)
|
||||
self.assertNotIn("postgresql", services)
|
||||
self.assertNotIn("n8n", services)
|
||||
|
||||
def test_engine_runtime_prepares_managed_private_state(self):
|
||||
with (
|
||||
mock.patch.object(
|
||||
RUNNER,
|
||||
"ensure_engine_edp_managed_provisioner_keypair",
|
||||
) as keypair,
|
||||
mock.patch.object(
|
||||
RUNNER,
|
||||
"ensure_engine_publish_grant_private_state",
|
||||
) as private_state,
|
||||
):
|
||||
RUNNER.prepare_component_runtime(
|
||||
"engine",
|
||||
RUNNER.ENGINE_DATA_PRODUCT_PUBLISH_GRANT_ARTIFACT_ENTRIES,
|
||||
)
|
||||
|
||||
keypair.assert_called_once_with()
|
||||
private_state.assert_called_once_with()
|
||||
|
||||
def test_partial_publish_path_has_no_publish_runtime_side_effects(self):
|
||||
entries = ("nodedc-source/server/dataProductPublishGrant/store.js",)
|
||||
self.assertTrue(RUNNER.touches_engine_data_product_publish_grant(entries))
|
||||
self.assertFalse(RUNNER.is_engine_data_product_publish_grant_slice(
|
||||
"engine",
|
||||
entries,
|
||||
))
|
||||
with (
|
||||
mock.patch.object(
|
||||
RUNNER,
|
||||
"ensure_engine_edp_managed_provisioner_keypair",
|
||||
) as keypair,
|
||||
mock.patch.object(
|
||||
RUNNER,
|
||||
"ensure_engine_publish_grant_private_state",
|
||||
) as private_state,
|
||||
):
|
||||
RUNNER.prepare_component_runtime("engine", entries)
|
||||
keypair.assert_not_called()
|
||||
private_state.assert_not_called()
|
||||
|
||||
def test_publish_predecessor_guard_checks_every_pinned_file_before_mutation(self):
|
||||
with tempfile.TemporaryDirectory(prefix="nodedc-publish-predecessor-") as directory:
|
||||
root = Path(directory)
|
||||
files = {
|
||||
"docker-compose.yml": b"services: {}\n",
|
||||
"nodedc-source/server/routes/n8n.js": b"export const baseline = true\n",
|
||||
}
|
||||
expected = {}
|
||||
for relative_path, value in files.items():
|
||||
path = root / relative_path
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_bytes(value)
|
||||
expected[relative_path] = hashlib.sha256(value).hexdigest()
|
||||
|
||||
original_root = RUNNER.COMPONENTS["engine"]["payload_root"]
|
||||
original_hashes = RUNNER.ENGINE_DATA_PRODUCT_PUBLISH_GRANT_PREDECESSOR_SHA256
|
||||
RUNNER.COMPONENTS["engine"]["payload_root"] = root
|
||||
RUNNER.ENGINE_DATA_PRODUCT_PUBLISH_GRANT_PREDECESSOR_SHA256 = expected
|
||||
try:
|
||||
self.assertEqual(
|
||||
RUNNER.preflight_engine_data_product_publish_grant_predecessor(),
|
||||
{
|
||||
"mode": "credential-sink-initial-transition",
|
||||
"compose_sha256": expected["docker-compose.yml"],
|
||||
"changed_paths": (),
|
||||
},
|
||||
)
|
||||
(root / "nodedc-source/server/routes/n8n.js").write_bytes(b"drift\n")
|
||||
with self.assertRaisesRegex(
|
||||
RUNNER.DeployError,
|
||||
"path=nodedc-source/server/routes/n8n.js",
|
||||
):
|
||||
RUNNER.preflight_engine_data_product_publish_grant_predecessor()
|
||||
finally:
|
||||
RUNNER.ENGINE_DATA_PRODUCT_PUBLISH_GRANT_PREDECESSOR_SHA256 = original_hashes
|
||||
RUNNER.COMPONENTS["engine"]["payload_root"] = original_root
|
||||
|
||||
def test_installed_publish_update_preserves_foundation_and_allows_only_grant_source_changes(self):
|
||||
with tempfile.TemporaryDirectory(prefix="nodedc-publish-update-") as directory:
|
||||
base = Path(directory)
|
||||
root = base / "installed"
|
||||
payload = base / "payload"
|
||||
root.mkdir()
|
||||
payload.mkdir()
|
||||
|
||||
compose_bytes = b"services: {}\n"
|
||||
compose = root / "docker-compose.yml"
|
||||
compose.write_bytes(compose_bytes)
|
||||
expected = {
|
||||
"docker-compose.yml": hashlib.sha256(compose_bytes).hexdigest(),
|
||||
}
|
||||
shared_files = {
|
||||
"nodedc-source/server/assets/provider-packages/v1/catalog.json": "{}\n",
|
||||
"nodedc-source/server/engineAgents/store.js": "export const profile = 'full-developer'\n",
|
||||
"nodedc-source/server/routes/engineAgentGateway.js": "\n".join((
|
||||
"engine_plan_data_product_publish_grant",
|
||||
"engine_apply_data_product_publish_grant",
|
||||
"engine_accept_data_product_publish_grant",
|
||||
"engine_rollback_data_product_publish_grant",
|
||||
)),
|
||||
"nodedc-source/server/routes/n8n.js": (
|
||||
"engineCredentialSinkN8nAdapter\n"
|
||||
"engineDataProductPublishGrantN8nAdapter\n"
|
||||
),
|
||||
RUNNER.ENGINE_DATA_PRODUCT_PUBLISH_GRANT_OVERRIDE_REL: (
|
||||
RUNNER.expected_engine_data_product_publish_grant_override()
|
||||
),
|
||||
}
|
||||
for name in RUNNER.ENGINE_DATA_PRODUCT_PUBLISH_GRANT_SOURCE_FILES:
|
||||
shared_files[f"nodedc-source/server/dataProductPublishGrant/{name}"] = (
|
||||
f"export const source = '{name}'\n"
|
||||
)
|
||||
for relative_path, value in shared_files.items():
|
||||
for destination in (root, payload):
|
||||
path = destination / relative_path
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(value, encoding="utf-8")
|
||||
acceptance_rel = "nodedc-source/server/dataProductPublishGrant/acceptance.js"
|
||||
(payload / acceptance_rel).write_text(
|
||||
"export const source = 'acceptance-fixed'\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
original_root = RUNNER.COMPONENTS["engine"]["payload_root"]
|
||||
original_hashes = RUNNER.ENGINE_DATA_PRODUCT_PUBLISH_GRANT_PREDECESSOR_SHA256
|
||||
RUNNER.COMPONENTS["engine"]["payload_root"] = root
|
||||
RUNNER.ENGINE_DATA_PRODUCT_PUBLISH_GRANT_PREDECESSOR_SHA256 = expected
|
||||
try:
|
||||
self.assertEqual(
|
||||
RUNNER.preflight_engine_data_product_publish_grant_predecessor(payload),
|
||||
{
|
||||
"mode": "installed-source-update",
|
||||
"compose_sha256": expected["docker-compose.yml"],
|
||||
"changed_paths": (acceptance_rel,),
|
||||
},
|
||||
)
|
||||
|
||||
gateway_rel = "nodedc-source/server/routes/engineAgentGateway.js"
|
||||
(payload / gateway_rel).write_text("cross-boundary change\n", encoding="utf-8")
|
||||
with self.assertRaisesRegex(
|
||||
RUNNER.DeployError,
|
||||
"crosses its source boundary",
|
||||
):
|
||||
RUNNER.preflight_engine_data_product_publish_grant_predecessor(payload)
|
||||
|
||||
(payload / gateway_rel).write_text(shared_files[gateway_rel], encoding="utf-8")
|
||||
compose.write_text("drift\n", encoding="utf-8")
|
||||
with self.assertRaisesRegex(
|
||||
RUNNER.DeployError,
|
||||
"foundation drift",
|
||||
):
|
||||
RUNNER.preflight_engine_data_product_publish_grant_predecessor(payload)
|
||||
finally:
|
||||
RUNNER.ENGINE_DATA_PRODUCT_PUBLISH_GRANT_PREDECESSOR_SHA256 = original_hashes
|
||||
RUNNER.COMPONENTS["engine"]["payload_root"] = original_root
|
||||
|
||||
def test_edp_healthcheck_requires_database_and_managed_provisioning(self):
|
||||
checks = RUNNER.component_healthchecks(
|
||||
"platform",
|
||||
("platform/services/external-data-plane/src/server.mjs",),
|
||||
("external-data-plane",),
|
||||
)
|
||||
self.assertEqual(
|
||||
checks,
|
||||
(RUNNER.external_data_plane_healthcheck(),),
|
||||
)
|
||||
|
||||
def test_edp_runtime_never_probes_unselected_platform_services(self):
|
||||
entries = ("platform/services/external-data-plane/src/server.mjs",)
|
||||
services = ("external-data-plane",)
|
||||
with mock.patch.object(RUNNER, "healthcheck_url") as healthcheck:
|
||||
RUNNER.run_healthchecks("platform", entries, services)
|
||||
|
||||
healthcheck.assert_called_once_with(
|
||||
RUNNER.external_data_plane_healthcheck(),
|
||||
)
|
||||
|
||||
def test_json_healthcheck_accepts_expected_contract(self):
|
||||
class Response:
|
||||
status = 200
|
||||
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, *_args):
|
||||
return False
|
||||
|
||||
def read(self, _limit):
|
||||
return json.dumps({
|
||||
"ok": True,
|
||||
"database": "ready",
|
||||
}).encode("utf-8")
|
||||
|
||||
with mock.patch.object(
|
||||
RUNNER.NO_REDIRECT_OPENER,
|
||||
"open",
|
||||
return_value=Response(),
|
||||
):
|
||||
RUNNER.healthcheck_url({
|
||||
"url": "http://127.0.0.1:18106/healthz",
|
||||
"expected_json": {
|
||||
"ok": True,
|
||||
"database": "ready",
|
||||
},
|
||||
})
|
||||
|
||||
def test_json_healthcheck_accepts_restored_edp_baseline_contract(self):
|
||||
class Response:
|
||||
status = 200
|
||||
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, *_args):
|
||||
return False
|
||||
|
||||
def read(self, _limit):
|
||||
return json.dumps({
|
||||
"ok": True,
|
||||
"service": "nodedc-external-data-plane",
|
||||
"database": "ready",
|
||||
}).encode("utf-8")
|
||||
|
||||
with mock.patch.object(
|
||||
RUNNER.NO_REDIRECT_OPENER,
|
||||
"open",
|
||||
return_value=Response(),
|
||||
):
|
||||
RUNNER.healthcheck_url(
|
||||
RUNNER.external_data_plane_healthcheck(require_managed=False),
|
||||
)
|
||||
|
||||
def test_json_healthcheck_does_not_accept_redirect(self):
|
||||
class Response:
|
||||
status = 200
|
||||
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, *_args):
|
||||
return False
|
||||
|
||||
def read(self, _limit):
|
||||
return json.dumps({
|
||||
"ok": True,
|
||||
"database": "ready",
|
||||
}).encode("utf-8")
|
||||
|
||||
url = "http://127.0.0.1:18106/healthz"
|
||||
redirect = RUNNER.urllib.error.HTTPError(
|
||||
url,
|
||||
302,
|
||||
"Found",
|
||||
{},
|
||||
None,
|
||||
)
|
||||
with (
|
||||
mock.patch.object(
|
||||
RUNNER.NO_REDIRECT_OPENER,
|
||||
"open",
|
||||
side_effect=(redirect, Response()),
|
||||
) as opener,
|
||||
mock.patch.object(RUNNER.time, "sleep"),
|
||||
):
|
||||
RUNNER.healthcheck_url({
|
||||
"url": url,
|
||||
"expected_json": {
|
||||
"ok": True,
|
||||
"database": "ready",
|
||||
},
|
||||
})
|
||||
|
||||
self.assertEqual(opener.call_count, 2)
|
||||
|
||||
def test_publish_grant_healthcheck_does_not_enter_n8n_transition(self):
|
||||
services = ("nodedc-backend",)
|
||||
with (
|
||||
mock.patch.object(RUNNER, "accept_engine_n8n_runtime") as accept,
|
||||
mock.patch.object(RUNNER, "healthcheck_compose_service") as compose_health,
|
||||
mock.patch.object(RUNNER, "healthcheck_url") as healthcheck,
|
||||
mock.patch.object(
|
||||
RUNNER,
|
||||
"preflight_engine_credential_backend_runtime",
|
||||
return_value={"mode": "verified-derived-retry"},
|
||||
) as backend_preflight,
|
||||
):
|
||||
RUNNER.run_healthchecks(
|
||||
"engine",
|
||||
RUNNER.ENGINE_DATA_PRODUCT_PUBLISH_GRANT_ARTIFACT_ENTRIES,
|
||||
services,
|
||||
)
|
||||
|
||||
accept.assert_not_called()
|
||||
compose_health.assert_called_once_with("engine", "nodedc-backend")
|
||||
healthcheck.assert_called_once_with(
|
||||
"http://127.0.0.1:3001/health",
|
||||
)
|
||||
backend_preflight.assert_called_once_with()
|
||||
|
||||
def test_engine_ui_healthcheck_requires_selected_app(self):
|
||||
grant_entries = RUNNER.ENGINE_DATA_PRODUCT_PUBLISH_GRANT_ARTIFACT_ENTRIES
|
||||
self.assertEqual(
|
||||
RUNNER.component_healthchecks(
|
||||
"engine",
|
||||
grant_entries,
|
||||
("nodedc-backend",),
|
||||
),
|
||||
("http://127.0.0.1:3001/health",),
|
||||
)
|
||||
self.assertEqual(
|
||||
RUNNER.component_healthchecks(
|
||||
"engine",
|
||||
("docker-compose.yml",),
|
||||
("nodedc-backend", "app"),
|
||||
),
|
||||
(
|
||||
"http://127.0.0.1:8080/",
|
||||
"http://127.0.0.1:3001/health",
|
||||
),
|
||||
)
|
||||
|
||||
def test_optional_edp_overlay_is_omitted_until_installed(self):
|
||||
with tempfile.TemporaryDirectory(
|
||||
prefix="nodedc-compose-files-",
|
||||
) as directory:
|
||||
temporary = Path(directory)
|
||||
base = temporary / "docker-compose.platform-http.yml"
|
||||
edp = temporary / "docker-compose.external-data-plane.yml"
|
||||
base.write_text(
|
||||
"name: nodedc-platform\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
original = RUNNER.COMPONENTS["platform"]["compose_files"]
|
||||
RUNNER.COMPONENTS["platform"]["compose_files"] = (base, edp)
|
||||
try:
|
||||
self.assertEqual(
|
||||
RUNNER.component_compose_files("platform"),
|
||||
(base,),
|
||||
)
|
||||
finally:
|
||||
RUNNER.COMPONENTS["platform"]["compose_files"] = original
|
||||
|
||||
|
||||
class PlatformOverlayRollbackTest(unittest.TestCase):
|
||||
def test_engine_runtime_failure_restores_source_and_recreates_exact_services(self):
|
||||
with tempfile.TemporaryDirectory(prefix="nodedc-engine-runtime-rollback-") as directory:
|
||||
temporary = Path(directory)
|
||||
root = temporary / "engine"
|
||||
backup = temporary / "backup"
|
||||
runner_tmp = temporary / "tmp"
|
||||
compose = root / "docker-compose.yml"
|
||||
backend = root / "nodedc-source/server/index.js"
|
||||
compose.parent.mkdir(parents=True)
|
||||
backend.parent.mkdir(parents=True)
|
||||
backup.mkdir()
|
||||
runner_tmp.mkdir()
|
||||
compose.write_text("name: baseline\n", encoding="utf-8")
|
||||
backend.write_text("export const version = 1;\n", encoding="utf-8")
|
||||
entries = ["docker-compose.yml", "nodedc-source/server/index.js"]
|
||||
services = ("n8n", "nodedc-backend", "app")
|
||||
RUNNER.create_backup(root, backup, entries, include_nginx_html=False)
|
||||
compose.write_text("name: candidate\n", encoding="utf-8")
|
||||
backend.write_text("export const version = 2;\n", encoding="utf-8")
|
||||
|
||||
def assert_restored(component, rollback_entries, rollback_services):
|
||||
self.assertEqual(component, "engine")
|
||||
self.assertEqual(rollback_entries, entries)
|
||||
self.assertEqual(rollback_services, services)
|
||||
self.assertEqual(compose.read_text(encoding="utf-8"), "name: baseline\n")
|
||||
self.assertEqual(
|
||||
backend.read_text(encoding="utf-8"),
|
||||
"export const version = 1;\n",
|
||||
)
|
||||
|
||||
with (
|
||||
mock.patch.object(RUNNER, "TMP_DIR", runner_tmp),
|
||||
mock.patch.object(
|
||||
RUNNER,
|
||||
"run_component_runtime",
|
||||
side_effect=assert_restored,
|
||||
) as runtime,
|
||||
mock.patch.object(RUNNER, "run_healthchecks") as healthchecks,
|
||||
):
|
||||
status = RUNNER.rollback_engine_apply(
|
||||
root,
|
||||
backup,
|
||||
entries,
|
||||
"test-stamp",
|
||||
runtime_started=True,
|
||||
applied_services=services,
|
||||
)
|
||||
|
||||
self.assertEqual(status, "source+runtime-restored:2")
|
||||
runtime.assert_called_once()
|
||||
healthchecks.assert_called_once_with("engine", entries, services)
|
||||
|
||||
def test_candidate_only_service_cleanup_never_requests_volume_removal(self):
|
||||
with (
|
||||
mock.patch.object(RUNNER, "component_compose_root", return_value=Path("/live/platform")),
|
||||
mock.patch.object(RUNNER, "compose_base_cmd", return_value=["docker", "compose"]),
|
||||
mock.patch.object(RUNNER.subprocess, "run") as run,
|
||||
):
|
||||
RUNNER.stop_and_remove_compose_services("platform", ("external-data-plane",))
|
||||
|
||||
command = run.call_args.args[0]
|
||||
self.assertEqual(
|
||||
command,
|
||||
["docker", "compose", "rm", "--stop", "--force", "external-data-plane"],
|
||||
)
|
||||
self.assertNotIn("--volumes", command)
|
||||
|
||||
def test_failed_first_edp_activation_removes_containers_without_deleting_volume(self):
|
||||
with tempfile.TemporaryDirectory(prefix="nodedc-platform-edp-rollback-") as directory:
|
||||
temporary = Path(directory)
|
||||
root = temporary / "root"
|
||||
backup = temporary / "backup"
|
||||
runner_tmp = temporary / "tmp"
|
||||
overlay = root / RUNNER.PLATFORM_EXTERNAL_DATA_PLANE_COMPOSE_REL
|
||||
overlay.parent.mkdir(parents=True)
|
||||
backup.mkdir()
|
||||
runner_tmp.mkdir()
|
||||
entries = [RUNNER.PLATFORM_EXTERNAL_DATA_PLANE_COMPOSE_REL]
|
||||
RUNNER.create_backup(root, backup, entries, include_nginx_html=False)
|
||||
overlay.write_text("name: nodedc-platform\n", encoding="utf-8")
|
||||
|
||||
with (
|
||||
mock.patch.object(RUNNER, "TMP_DIR", runner_tmp),
|
||||
mock.patch.object(RUNNER, "stop_and_remove_compose_services") as remove,
|
||||
mock.patch.object(RUNNER, "run_component_runtime") as runtime,
|
||||
mock.patch.object(RUNNER, "run_healthchecks") as healthchecks,
|
||||
):
|
||||
status = RUNNER.rollback_platform_apply(
|
||||
root,
|
||||
backup,
|
||||
entries,
|
||||
"test-stamp",
|
||||
runtime_started=True,
|
||||
applied_services=("external-data-plane",),
|
||||
)
|
||||
|
||||
remove.assert_called_once_with(
|
||||
"platform",
|
||||
("external-data-plane",),
|
||||
)
|
||||
runtime.assert_not_called()
|
||||
healthchecks.assert_not_called()
|
||||
self.assertFalse(overlay.exists())
|
||||
self.assertEqual(status, "source+runtime-restored:1")
|
||||
|
||||
def test_candidate_cleanup_failure_still_restores_source_before_reporting_failure(self):
|
||||
with tempfile.TemporaryDirectory(prefix="nodedc-platform-edp-rollback-") as directory:
|
||||
temporary = Path(directory)
|
||||
root = temporary / "root"
|
||||
backup = temporary / "backup"
|
||||
runner_tmp = temporary / "tmp"
|
||||
overlay = root / RUNNER.PLATFORM_EXTERNAL_DATA_PLANE_COMPOSE_REL
|
||||
overlay.parent.mkdir(parents=True)
|
||||
backup.mkdir()
|
||||
runner_tmp.mkdir()
|
||||
entries = [RUNNER.PLATFORM_EXTERNAL_DATA_PLANE_COMPOSE_REL]
|
||||
RUNNER.create_backup(root, backup, entries, include_nginx_html=False)
|
||||
overlay.write_text("name: rejected\n", encoding="utf-8")
|
||||
|
||||
with (
|
||||
mock.patch.object(RUNNER, "TMP_DIR", runner_tmp),
|
||||
mock.patch.object(
|
||||
RUNNER,
|
||||
"stop_and_remove_compose_services",
|
||||
side_effect=RuntimeError("simulated cleanup failure"),
|
||||
),
|
||||
):
|
||||
with self.assertRaisesRegex(RUNNER.DeployError, "cleanup failed after source restore"):
|
||||
RUNNER.rollback_platform_apply(
|
||||
root,
|
||||
backup,
|
||||
entries,
|
||||
"test-stamp",
|
||||
runtime_started=True,
|
||||
applied_services=("external-data-plane",),
|
||||
)
|
||||
self.assertFalse(overlay.exists())
|
||||
|
||||
def test_runtime_failure_rebuilds_recreates_and_accepts_restored_overlay(self):
|
||||
with tempfile.TemporaryDirectory(prefix="nodedc-platform-runtime-rollback-") as directory:
|
||||
temporary = Path(directory)
|
||||
root = temporary / "root"
|
||||
backup = temporary / "backup"
|
||||
runner_tmp = temporary / "tmp"
|
||||
gateway = root / "platform/gelios-gateway"
|
||||
gateway.mkdir(parents=True)
|
||||
backup.mkdir()
|
||||
runner_tmp.mkdir()
|
||||
source = gateway / "server.mjs"
|
||||
source.write_text("export const version = 1;\n", encoding="utf-8")
|
||||
entries = ["platform/gelios-gateway"]
|
||||
RUNNER.create_backup(root, backup, entries, include_nginx_html=False)
|
||||
source.write_text("export const version = 2;\n", encoding="utf-8")
|
||||
|
||||
def assert_restored_runtime(component, rollback_entries, services):
|
||||
self.assertEqual(component, "platform")
|
||||
self.assertEqual(rollback_entries, entries)
|
||||
self.assertEqual(services, ("gelios-postgres", "gelios-gateway"))
|
||||
self.assertEqual(source.read_text(encoding="utf-8"), "export const version = 1;\n")
|
||||
|
||||
with (
|
||||
mock.patch.object(RUNNER, "TMP_DIR", runner_tmp),
|
||||
mock.patch.object(
|
||||
RUNNER,
|
||||
"run_component_runtime",
|
||||
side_effect=assert_restored_runtime,
|
||||
) as runtime,
|
||||
mock.patch.object(RUNNER, "run_healthchecks") as healthchecks,
|
||||
):
|
||||
status = RUNNER.rollback_platform_apply(
|
||||
root,
|
||||
backup,
|
||||
entries,
|
||||
"test-stamp",
|
||||
runtime_started=True,
|
||||
applied_services=("gelios-postgres", "gelios-gateway"),
|
||||
)
|
||||
|
||||
self.assertEqual(status, "source+runtime-restored:1")
|
||||
runtime.assert_called_once()
|
||||
healthchecks.assert_called_once_with(
|
||||
"platform",
|
||||
entries,
|
||||
("gelios-postgres", "gelios-gateway"),
|
||||
)
|
||||
|
||||
def test_existing_edp_rollback_accepts_only_restored_edp(self):
|
||||
with tempfile.TemporaryDirectory(prefix="nodedc-platform-edp-runtime-rollback-") as directory:
|
||||
temporary = Path(directory)
|
||||
root = temporary / "root"
|
||||
backup = temporary / "backup"
|
||||
runner_tmp = temporary / "tmp"
|
||||
service = root / "platform/services/external-data-plane"
|
||||
service.mkdir(parents=True)
|
||||
backup.mkdir()
|
||||
runner_tmp.mkdir()
|
||||
source = service / "server.mjs"
|
||||
source.write_text("export const version = 1;\n", encoding="utf-8")
|
||||
entries = ["platform/services/external-data-plane"]
|
||||
RUNNER.create_backup(root, backup, entries, include_nginx_html=False)
|
||||
source.write_text("export const version = 2;\n", encoding="utf-8")
|
||||
|
||||
with (
|
||||
mock.patch.object(RUNNER, "TMP_DIR", runner_tmp),
|
||||
mock.patch.object(RUNNER, "run_component_runtime") as runtime,
|
||||
mock.patch.object(RUNNER, "healthcheck_url") as healthcheck,
|
||||
):
|
||||
status = RUNNER.rollback_platform_apply(
|
||||
root,
|
||||
backup,
|
||||
entries,
|
||||
"test-stamp",
|
||||
runtime_started=True,
|
||||
applied_services=("external-data-plane",),
|
||||
)
|
||||
|
||||
self.assertEqual(status, "source+runtime-restored:1")
|
||||
self.assertEqual(
|
||||
source.read_text(encoding="utf-8"),
|
||||
"export const version = 1;\n",
|
||||
)
|
||||
runtime.assert_called_once_with(
|
||||
"platform",
|
||||
entries,
|
||||
("external-data-plane",),
|
||||
)
|
||||
healthcheck.assert_called_once_with(
|
||||
RUNNER.external_data_plane_healthcheck(require_managed=False),
|
||||
)
|
||||
|
||||
def test_restores_existing_overlay_and_removes_new_paths(self):
|
||||
with tempfile.TemporaryDirectory(prefix="nodedc-platform-rollback-test-") as directory:
|
||||
temporary = Path(directory)
|
||||
root = temporary / "root"
|
||||
backup = temporary / "backup"
|
||||
runner_tmp = temporary / "tmp"
|
||||
root.mkdir()
|
||||
backup.mkdir()
|
||||
runner_tmp.mkdir()
|
||||
|
||||
compose = root / "platform/docker-compose.platform-http.yml"
|
||||
gateway = root / "platform/gelios-gateway"
|
||||
new_service = root / "platform/services/new-service"
|
||||
unrelated = root / "platform/unrelated.txt"
|
||||
compose.parent.mkdir(parents=True)
|
||||
gateway.mkdir()
|
||||
compose.write_text("name: nodedc-platform\n", encoding="utf-8")
|
||||
executable = gateway / "run.sh"
|
||||
executable.write_text("#!/bin/sh\nexit 0\n", encoding="utf-8")
|
||||
executable.chmod(0o755)
|
||||
(gateway / "server.mjs").write_text("export const version = 1;\n", encoding="utf-8")
|
||||
unrelated.write_text("untouched\n", encoding="utf-8")
|
||||
|
||||
entries = [
|
||||
"platform/docker-compose.platform-http.yml",
|
||||
"platform/gelios-gateway",
|
||||
"platform/services/new-service",
|
||||
]
|
||||
RUNNER.create_backup(root, backup, entries, include_nginx_html=False)
|
||||
|
||||
compose.write_text("name: broken\n", encoding="utf-8")
|
||||
(gateway / "server.mjs").write_text("export const version = 2;\n", encoding="utf-8")
|
||||
executable.chmod(0o644)
|
||||
new_service.mkdir(parents=True)
|
||||
(new_service / "bad.mjs").write_text("broken\n", encoding="utf-8")
|
||||
|
||||
with mock.patch.object(RUNNER, "TMP_DIR", runner_tmp):
|
||||
restored = RUNNER.restore_platform_overlay(root, backup, entries, "test-stamp")
|
||||
|
||||
self.assertEqual(restored, len(entries))
|
||||
self.assertEqual(compose.read_text(encoding="utf-8"), "name: nodedc-platform\n")
|
||||
self.assertEqual(
|
||||
(gateway / "server.mjs").read_text(encoding="utf-8"),
|
||||
"export const version = 1;\n",
|
||||
)
|
||||
self.assertTrue(stat.S_IMODE((gateway / "run.sh").stat().st_mode) & stat.S_IXUSR)
|
||||
self.assertFalse(new_service.exists())
|
||||
self.assertEqual(unrelated.read_text(encoding="utf-8"), "untouched\n")
|
||||
|
||||
def test_rejects_inconsistent_backup_partition_before_restore(self):
|
||||
with tempfile.TemporaryDirectory(prefix="nodedc-platform-rollback-test-") as directory:
|
||||
temporary = Path(directory)
|
||||
root = temporary / "root"
|
||||
backup = temporary / "backup"
|
||||
runner_tmp = temporary / "tmp"
|
||||
target = root / "platform/docker-compose.platform-http.yml"
|
||||
target.parent.mkdir(parents=True)
|
||||
target.write_text("original\n", encoding="utf-8")
|
||||
backup.mkdir()
|
||||
runner_tmp.mkdir()
|
||||
entries = ["platform/docker-compose.platform-http.yml"]
|
||||
RUNNER.create_backup(root, backup, entries, include_nginx_html=False)
|
||||
(backup / "missing-files.txt").write_text(
|
||||
"platform/docker-compose.platform-http.yml\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
target.write_text("candidate\n", encoding="utf-8")
|
||||
|
||||
with mock.patch.object(RUNNER, "TMP_DIR", runner_tmp):
|
||||
with self.assertRaisesRegex(RUNNER.DeployError, "backup entry set mismatch"):
|
||||
RUNNER.restore_platform_overlay(root, backup, entries, "test-stamp")
|
||||
self.assertEqual(target.read_text(encoding="utf-8"), "candidate\n")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main(verbosity=2)
|
||||
|
|
@ -192,9 +192,8 @@ services:
|
|||
retries: 3
|
||||
start_period: 10s
|
||||
|
||||
# External-provider telemetry is isolated from Platform identity state. The
|
||||
# database is not published; only the Gateway is exposed on the local host
|
||||
# for operator checks and Engine development.
|
||||
# Frozen legacy Gelios compatibility contour. Keep it reproducible and
|
||||
# isolated; new provider integrations use the provider-neutral data plane.
|
||||
gelios-postgres:
|
||||
image: ${GELIOS_TIMESCALE_IMAGE:-timescale/timescaledb-ha:pg16.14-ts2.28.2-all}
|
||||
restart: unless-stopped
|
||||
|
|
|
|||
|
|
@ -97,9 +97,20 @@ EXTERNAL_DATA_PLANE_WRITER_BINDING_MAX_TTL_DAYS=90
|
|||
EXTERNAL_DATA_PLANE_MAX_FUTURE_SKEW_SECONDS=300
|
||||
EXTERNAL_DATA_PLANE_RETENTION_SWEEP_MS=3600000
|
||||
EXTERNAL_DATA_PLANE_LEGACY_INTAKE_ENABLED=false
|
||||
# The writer-provisioner secret is a root-owned file under
|
||||
# /volume1/docker/nodedc-platform/secrets/external-data-plane-provisioner/,
|
||||
# never a shared .env value.
|
||||
# Internal control-plane writer/reader binding issuance; keep false until the
|
||||
# atomic NDC L2 ensure-grant operation is deployed. Legacy path returns a
|
||||
# plaintext capability and must never be exposed to users.
|
||||
EXTERNAL_DATA_PLANE_PROVISIONING_ENABLED=false
|
||||
# Digest-only managed writer-binding ensure. It accepts only signed Engine
|
||||
# service requests; the legacy provisioner bearer is deliberately invalid here.
|
||||
# Keep false until the matching Engine private key is provisioned. The trust
|
||||
# directory is mounted read-only into EDP and contains only `public-key.pem`.
|
||||
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONING_ENABLED=false
|
||||
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_SERVICE_ID=nodedc-engine
|
||||
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_KEY_ID=engine-edp-managed-provisioner-v1
|
||||
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_AUDIENCE=nodedc-external-data-plane.managed-provisioning.v1
|
||||
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_MAX_SKEW_SECONDS=60
|
||||
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_REPLAY_CACHE_MAX_ENTRIES=10000
|
||||
|
||||
NOTIFICATION_PG_DB=nodedc_notifications
|
||||
NOTIFICATION_PG_USER=nodedc_notifications
|
||||
|
|
@ -132,8 +143,9 @@ AI_WORKSPACE_ONTOLOGY_MCP_ENABLED=true
|
|||
AI_WORKSPACE_ONTOLOGY_MCP_PUBLIC_URL=
|
||||
ONTOLOGY_CORE_HOST_BIND=127.0.0.1:18104
|
||||
|
||||
# Gelios Gateway — storage/read service. Provider credentials stay in the
|
||||
# protected Engine Collector. Intake remains disabled until scope is approved.
|
||||
# Gelios Gateway — frozen legacy storage/read compatibility service. Provider
|
||||
# credentials stay in the protected Engine Collector. Do not extend this
|
||||
# provider-specific contour for new providers.
|
||||
GELIOS_TIMESCALE_IMAGE=timescale/timescaledb-ha:pg16.14-ts2.28.2-all
|
||||
GELIOS_PG_DB=nodedc_gelios
|
||||
GELIOS_PG_USER=nodedc_gelios
|
||||
|
|
@ -141,8 +153,6 @@ GELIOS_PG_PASS=replace-with-random-synology-secret
|
|||
# URL-encode reserved characters in GELIOS_PG_PASS when forming this URL.
|
||||
GELIOS_DATABASE_URL=postgresql://nodedc_gelios:replace-with-url-encoded-synology-secret@gelios-postgres:5432/nodedc_gelios
|
||||
GELIOS_GATEWAY_HOST_BIND=127.0.0.1:18105
|
||||
# Explicit tenant and connection identifiers are deployment configuration;
|
||||
# do not encode a customer or pilot name in source defaults.
|
||||
GELIOS_TENANT_ID=replace-with-tenant-id
|
||||
GELIOS_CONNECTION_ID=gelios-connection-id
|
||||
# `all` accepts every unit returned by this approved tenant + connection.
|
||||
|
|
|
|||
|
|
@ -202,7 +202,13 @@ TASKER_SYNC_SOURCE=1 ./infra/synology/deploy-current.sh
|
|||
|
||||
Если emergency-fix был сделан прямо на Synology в этих env-файлах, перенести sanitized-значение в `.env.synology.example`/docs, а секрет оставить только в live env.
|
||||
|
||||
### Gelios: сбор всех units доверенного подключения
|
||||
### Gelios: frozen legacy compatibility contour
|
||||
|
||||
`gelios-postgres` и `gelios-gateway` — уже существующий self-hosted контур с
|
||||
собственным persistent Timescale volume. Он остаётся полностью воспроизводимым
|
||||
в source/deploy, но не развивается и не используется как шаблон для новых
|
||||
провайдеров. Его удаление или миграция требуют отдельного подтверждённого плана;
|
||||
обычный Platform/EDP deploy не удаляет volume и не изменяет старый workflow.
|
||||
|
||||
`GELIOS_UNIT_SCOPE=all` означает **все текущие и будущие units только одной уже
|
||||
настроенной пары `GELIOS_TENANT_ID` + `GELIOS_CONNECTION_ID`**. Это не wildcard
|
||||
|
|
@ -210,17 +216,71 @@ TASKER_SYNC_SOURCE=1 ./infra/synology/deploy-current.sh
|
|||
Пропавший из очередного ответа unit не удаляется: его stable provider ID и
|
||||
`last_seen_at` сохраняются; видимость на карте — отдельная логика витрины.
|
||||
|
||||
Перед каноническим `nodedc-deploy apply` включить политику на Synology (скрипт
|
||||
создаёт backup и не выводит секреты):
|
||||
Перед узким legacy apply включить политику на Synology (скрипт создаёт backup и
|
||||
не выводит секреты):
|
||||
|
||||
```bash
|
||||
sudo bash /volume1/docker/nodedc-deploy/inbox/prepare-gelios-all-units-env.sh
|
||||
```
|
||||
|
||||
Затем применить узкий Platform-артефакт, собранный с `--gateway-only`, через
|
||||
`nodedc-deploy`. В его plan должны быть только `gelios-postgres` и
|
||||
`gelios-gateway`; общий `docker-compose` в такой архив не входит. После apply
|
||||
Gateway начинает принимать весь состав units этого подключения.
|
||||
В plan такого Platform-артефакта должны быть только `gelios-postgres` и
|
||||
`gelios-gateway`; общий compose в узкий архив не входит.
|
||||
|
||||
## EDP publish grant: platform-owned credential
|
||||
|
||||
`NDC Data Product Writer API` — историческое имя native credential type для
|
||||
внутреннего scoped EDP publish grant. Это не Gelios access/refresh token и не
|
||||
право записи в Gelios.
|
||||
|
||||
Целевой путь полностью служебный: control plane NDC по разрешённому connection
|
||||
profile генерирует capability внутри native NDC L2 Credentials, передаёт EDP
|
||||
только digest для scoped binding и возвращает MCP только opaque compatible
|
||||
reference/status.
|
||||
Пользователь не читает, не копирует и не вводит этот внутренний секрет.
|
||||
|
||||
EDP уже имеет localhost-only digest-only managed endpoint. Managed endpoint и
|
||||
legacy plaintext provisioning имеют раздельные флаги и по умолчанию выключены.
|
||||
Runner-owned provisioner bearer относится только к legacy plaintext routes и
|
||||
на managed ensure/revoke намеренно не действует. Managed caller contract —
|
||||
Ed25519-signed Engine service request с exact audience, method, request target,
|
||||
raw-body SHA-256, timestamp и одноразовым nonce; EDP хранит только public key и
|
||||
fail-closed replay cache. Ни bearer, ни signing private key нельзя открывать
|
||||
workflow, MCP или переиспользовать как shared credential. В source-кандидате
|
||||
Engine добавлен отдельный server-derived plan/apply для exact пары
|
||||
`ndcDataProductWriterApi` + custom Publish node; он не расширяет generic HTTP
|
||||
safe-ref policy и не принимает provider/scope/credential identity, capability,
|
||||
generation или service URL от MCP. До отдельного Engine deploy и runtime proof
|
||||
текущий deployed MCP этого контракта не имеет. Поэтому отсутствие
|
||||
совместимого reference — платформенный runtime gap, а не действие пользователя.
|
||||
Ручной перенос capability из отдельно включаемого legacy endpoint через root/UI допустим только как аварийная
|
||||
диагностика self-hosted контура; он не является acceptance-путём и не должен
|
||||
закрепляться в пользовательской автоматизации.
|
||||
|
||||
Compose монтирует read-only trust directory
|
||||
`/volume1/docker/nodedc-platform/trust/engine-managed-provisioner` в EDP; в нём
|
||||
ожидается один regular, non-symlink файл `public-key.pem` с Ed25519 SPKI public
|
||||
key. Файл не должен быть group/world-writable и должен читаться runtime UID
|
||||
`11006`. Private key в этот каталог, Platform artifact, `.env` или EDP
|
||||
контейнер не попадает. Canonical deploy runner теперь атомарно создаёт или
|
||||
проверяет matching Ed25519 pair перед relevant Engine/EDP Compose apply: private
|
||||
key остаётся `root:root 0400`, public trust — `root:11006 0440`; public-only
|
||||
crash state, mismatch, symlink и неверный тип ключа отклоняются до запуска.
|
||||
`EXTERNAL_DATA_PLANE_MANAGED_PROVISIONING_ENABLED` остаётся `false` на staging.
|
||||
В отдельно подтверждённом activation window его переводят в `true`
|
||||
непосредственно перед Platform EDP artifact; runner принимает deploy только
|
||||
если `/healthz` явно вернул managed status `enabled`. Старый Engine в этот
|
||||
момент ещё не имеет signer mount, поэтому usable caller появляется лишь после
|
||||
следующего отдельно принятого Engine artifact.
|
||||
EDP application artifact пересобирает и force-recreate только
|
||||
`external-data-plane`. Уже работающий `external-data-plane-postgres` и его
|
||||
Timescale volume runner не выбирает и не перезапускает; healthy database —
|
||||
обязательная deploy prerequisite.
|
||||
Canonical service/key/audience заданы в `.env.synology.example`; wire headers и
|
||||
canonical signing payload описаны в `packages/external-provider-contract/README.md`.
|
||||
|
||||
До появления opaque compatible reference и успешного manual publish proof
|
||||
Schedule Trigger не включать. Frozen legacy workflow и его credentials этот
|
||||
процесс не изменяет.
|
||||
|
||||
## AI Hub relay-only deploy
|
||||
|
||||
|
|
|
|||
|
|
@ -38,7 +38,7 @@ echo "== ontology core image build =="
|
|||
cd "${PLATFORM_DIR}/ontology-core"
|
||||
"${DOCKER_BIN}" build --no-cache -t nodedc/ontology-core:local .
|
||||
|
||||
echo "== gelios gateway image build =="
|
||||
echo "== frozen legacy gelios gateway image build =="
|
||||
cd "${PLATFORM_DIR}/gelios-gateway"
|
||||
"${DOCKER_BIN}" build --no-cache -t nodedc/gelios-gateway:local .
|
||||
|
||||
|
|
@ -53,7 +53,7 @@ echo "== ontology core health check =="
|
|||
"${DOCKER_BIN}" exec nodedc-platform-ontology-core-1 sh -lc \
|
||||
'node -e '"'"'fetch("http://127.0.0.1:18104/healthz").then(async (response) => { console.log(await response.text()); process.exit(response.ok ? 0 : 1); }).catch((error) => { console.error(error); process.exit(1); })'"'"''
|
||||
|
||||
echo "== gelios gateway health check =="
|
||||
echo "== frozen legacy gelios gateway health check =="
|
||||
"${DOCKER_BIN}" exec nodedc-platform-gelios-gateway-1 sh -lc \
|
||||
'node -e '"'"'fetch("http://127.0.0.1:18105/healthz").then(async (response) => { console.log(await response.text()); process.exit(response.ok ? 0 : 1); }).catch((error) => { console.error(error); process.exit(1); })'"'"''
|
||||
|
||||
|
|
|
|||
|
|
@ -51,12 +51,23 @@ services:
|
|||
# Migration-only shared-token routes stay closed in a canonical install.
|
||||
EXTERNAL_DATA_PLANE_LEGACY_INTAKE_ENABLED: ${EXTERNAL_DATA_PLANE_LEGACY_INTAKE_ENABLED:-false}
|
||||
NODEDC_INTERNAL_ACCESS_TOKEN: ${NODEDC_INTERNAL_ACCESS_TOKEN:-}
|
||||
# Runner-owned secret file. This credential must never live in the
|
||||
# shared .env.synology that is inherited by unrelated services.
|
||||
# Legacy manual one-time issuance uses this EDP-only bearer. Managed
|
||||
# Engine ensure/revoke never accepts it and uses the Ed25519 boundary
|
||||
# below. Neither credential is exposed to a graph, MCP client or provider.
|
||||
EXTERNAL_DATA_PLANE_PROVISIONER_TOKEN_FILE: /run/nodedc-secrets/external-data-plane-provisioner-token
|
||||
# This stays false until the dedicated Engine provisioner is deployed and
|
||||
# receives the same read-only secret mount.
|
||||
EXTERNAL_DATA_PLANE_PROVISIONING_ENABLED: "false"
|
||||
# Disabled until the atomic native NDC L2 ensure-grant control-plane
|
||||
# operation is present. This legacy API returns plaintext capabilities;
|
||||
# a root/UI transfer is emergency diagnostics only.
|
||||
EXTERNAL_DATA_PLANE_PROVISIONING_ENABLED: ${EXTERNAL_DATA_PLANE_PROVISIONING_ENABLED:-false}
|
||||
# Digest-only, idempotent writer-binding ensure path. Enable only after
|
||||
# Engine owns the matching private key and the public trust file exists.
|
||||
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONING_ENABLED: ${EXTERNAL_DATA_PLANE_MANAGED_PROVISIONING_ENABLED:-false}
|
||||
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_PUBLIC_KEY_FILE: /run/nodedc-trust/engine-managed-provisioner/public-key.pem
|
||||
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_SERVICE_ID: ${EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_SERVICE_ID:-nodedc-engine}
|
||||
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_KEY_ID: ${EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_KEY_ID:-engine-edp-managed-provisioner-v1}
|
||||
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_AUDIENCE: ${EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_AUDIENCE:-nodedc-external-data-plane.managed-provisioning.v1}
|
||||
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_MAX_SKEW_SECONDS: ${EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_MAX_SKEW_SECONDS:-60}
|
||||
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_REPLAY_CACHE_MAX_ENTRIES: ${EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_REPLAY_CACHE_MAX_ENTRIES:-10000}
|
||||
volumes:
|
||||
- type: bind
|
||||
source: /volume1/docker/nodedc-platform/secrets/external-data-plane-provisioner/token
|
||||
|
|
@ -64,6 +75,12 @@ services:
|
|||
read_only: true
|
||||
bind:
|
||||
create_host_path: false
|
||||
- type: bind
|
||||
source: /volume1/docker/nodedc-platform/trust/engine-managed-provisioner
|
||||
target: /run/nodedc-trust/engine-managed-provisioner
|
||||
read_only: true
|
||||
bind:
|
||||
create_host_path: false
|
||||
expose:
|
||||
- "18106"
|
||||
ports:
|
||||
|
|
|
|||
|
|
@ -184,8 +184,8 @@ services:
|
|||
networks:
|
||||
- engine
|
||||
|
||||
# The provider database is private. Gateway is the only component that can
|
||||
# reach it; the host bind is loopback-only for audited operator diagnostics.
|
||||
# Frozen legacy Gelios compatibility contour. Keep it reproducible and
|
||||
# isolated; new provider integrations use the provider-neutral data plane.
|
||||
gelios-postgres:
|
||||
image: ${GELIOS_TIMESCALE_IMAGE:-timescale/timescaledb-ha:pg16.14-ts2.28.2-all}
|
||||
restart: unless-stopped
|
||||
|
|
|
|||
|
|
@ -67,7 +67,7 @@ echo "== ontology core health check =="
|
|||
"${DOCKER_BIN}" exec nodedc-platform-ontology-core-1 sh -lc \
|
||||
'node -e '"'"'fetch("http://127.0.0.1:18104/healthz").then(async (response) => { console.log(await response.text()); process.exit(response.ok ? 0 : 1); }).catch((error) => { console.error(error); process.exit(1); })'"'"''
|
||||
|
||||
echo "== gelios gateway health check =="
|
||||
echo "== frozen legacy gelios gateway health check =="
|
||||
"${DOCKER_BIN}" exec nodedc-platform-gelios-gateway-1 sh -lc \
|
||||
'node -e '"'"'fetch("http://127.0.0.1:18105/healthz").then(async (response) => { console.log(await response.text()); process.exit(response.ok ? 0 : 1); }).catch((error) => { console.error(error); process.exit(1); })'"'"''
|
||||
|
||||
|
|
|
|||
|
|
@ -5,7 +5,7 @@
|
|||
исполняемый connector code. `src/index.mjs` даёт dependency-free проверку
|
||||
минимальных v1 contracts; она запускается через `npm run check`.
|
||||
|
||||
Каждый L2 connector instance должен поставлять совместимые versioned
|
||||
Каждый NDC L2 connector instance должен поставлять совместимые versioned
|
||||
артефакты:
|
||||
|
||||
```text
|
||||
|
|
@ -21,16 +21,47 @@ command-catalog (metadata only until a red command gateway is approved)
|
|||
```
|
||||
|
||||
`provider-manifest` — декларативный template-артефакт: provider ID, ontology
|
||||
revision, L2 template version, capability catalog и data-product IDs. Он не
|
||||
revision, NDC L2 template version, capability catalog и data-product IDs. Он не
|
||||
является tenant connection, не содержит endpoint/URL, credential reference,
|
||||
secret, ручной scope или executable provider code. Concrete non-secret
|
||||
connection profile принадлежит и версионируется вместе с конкретным L2
|
||||
workflow; Engine только привязывает к нему opaque credential reference и grant.
|
||||
connection profile принадлежит и версионируется вместе с конкретным NDC L2
|
||||
workflow; caller привязывает только opaque provider credential reference, а
|
||||
внутренний workload grant разрешает control plane.
|
||||
|
||||
## Versioned provider packages
|
||||
|
||||
`providers/<provider>/<major>` — устанавливаемая contract/data единица, а не
|
||||
runtime service и не custom node. Она объединяет manifest, auth-mode metadata,
|
||||
capability catalog, field policies, collection profiles, Data Product
|
||||
definitions, semantic mapping contracts, provider-neutral NDC L2 template
|
||||
descriptor и synthetic fixtures. Package не содержит tenant/account state,
|
||||
credential values или исполняемый provider daemon.
|
||||
|
||||
`instantiateL2Connection` создаёт произвольное число connection instances из
|
||||
одного immutable package. Каждый instance получает собственные tenant,
|
||||
connection и opaque provider credential reference, но не требует изменения
|
||||
Platform source. Внутренние workload bindings объявляются отдельно как
|
||||
`control_plane_managed` и не являются caller input. Канонический collection
|
||||
scope — `all_visible_to_credential`: выбранная
|
||||
read-capability передаёт все entities, которые provider возвращает для
|
||||
привязанного credential. Unit allowlist и provider-group filter на collection
|
||||
boundary запрещены; сортировка и видимость принадлежат Data Product consumer и
|
||||
Foundry.
|
||||
|
||||
Первый production-shaped package — `providers/gelios/v1`. Он фиксирует реальный
|
||||
Gelios REST `GET /api/v1/units` transport contract и точный output
|
||||
`fleet.positions.current.v1@1.0.0` с revision
|
||||
`ontology.map.moving_object.v1`. Все Data Product fields используют snake_case.
|
||||
Его optional `tokenLifecycle` точно описывает два provider-issued artifacts —
|
||||
`access` и `refresh`: request использует `access`, а refresh пока имеет режим
|
||||
`operator_managed`. Это metadata без secret values и без заявления о
|
||||
реализованном автоматическом refresh.
|
||||
|
||||
## Проверяемые v1 contracts
|
||||
|
||||
- `Connection` — provider instance, tenant scope и *ссылка* на credential в
|
||||
Engine. Любые token/secret/password-like поля запрещены.
|
||||
- `Connection` — provider instance, tenant scope и *ссылка* на provider
|
||||
credential в NDC L2 Credentials. Любые token/secret/password-like поля
|
||||
запрещены; publisher представлен system-managed declaration/status без ref.
|
||||
- `Collection Profile` — явная policy сбора. `manual` не может скрыто содержать
|
||||
polling interval; `realtime` требует interval не чаще одного раза в секунду.
|
||||
- `Data Product` — нормализованный versioned output с semantic types, полями и
|
||||
|
|
@ -38,18 +69,18 @@ workflow; Engine только привязывает к нему opaque credenti
|
|||
- `Intake Batch` — canonical **scoped** record, который External Data Plane
|
||||
валидирует и сохраняет: source, contract revision, idempotency, restricted
|
||||
raw envelope и canonical facts. Его `source` содержит `providerId`,
|
||||
`tenantId` и `connectionId`. Writer-bound L2 request намеренно не является
|
||||
`tenantId` и `connectionId`. Writer-bound NDC L2 request намеренно не является
|
||||
готовым `Intake Batch`: Data Plane сначала materializes scope и лишь затем
|
||||
применяет этот contract. В canonical record нет provider field mapping,
|
||||
entity allowlist, token или renderer data. Inline `raw.payload` в v1
|
||||
запрещён: если нужна provenance-ссылка, connector передаёт restricted
|
||||
`raw.ref` вместе с hash. Отдельный raw-vault может быть добавлен только
|
||||
отдельным ADR и не становится частью L2 → Data Plane wire boundary.
|
||||
отдельным ADR и не становится частью NDC L2 → Data Plane wire boundary.
|
||||
- `NDC Foundry Binding` — адресует data product только в конкретную цепочку
|
||||
`Foundry Application → Page → approved slot`; `templateId` можно сохранить
|
||||
как дополнительную типизацию, но он не заменяет `applicationId` и `pageId`.
|
||||
В binding запрещены provider transport, endpoint и credential reference.
|
||||
- `Provider Manifest` — статическое описание L2 connector template, capability
|
||||
- `Provider Manifest` — статическое описание NDC L2 connector template, capability
|
||||
catalog и ontology/data-product contracts. Оно не может содержать tenant,
|
||||
connection, credential, secret или provider transport.
|
||||
|
||||
|
|
@ -94,13 +125,14 @@ PostgreSQL `integer` `0..2147483647`. Эти ограничения нельзя
|
|||
|
||||
Manifest, connection/collection profiles, data-product definition и Foundry
|
||||
binding используют fail-closed allowed-key schemas. Неизвестные поля, а также
|
||||
secret-like имена или значения (включая `ndc_edpwb_`/`ndc_edprb_`) отклоняются
|
||||
secret-like имена или значения (включая `ndc_edpwb_`, `ndc_edprb_` и
|
||||
`ndc_edppr_`) отклоняются
|
||||
на общей границе.
|
||||
|
||||
Все private custom nodes NODE.DC поставляются package
|
||||
`platform/packages/n8n-nodes-ndc`. Их display name обязан начинаться с `NDC `,
|
||||
а runtime type — с `n8n-nodes-ndc.`; package называется строго
|
||||
`n8n-nodes-ndc`. Provider-specific adapters остаются L2 workflow logic и не
|
||||
`n8n-nodes-ndc`. Provider-specific adapters остаются NDC L2 workflow logic и не
|
||||
становятся custom nodes или ветками Data Plane. Эти инварианты проверяются
|
||||
package test.
|
||||
|
||||
|
|
@ -110,76 +142,72 @@ declarative provider artifact и не runtime transport: команда соде
|
|||
application/page/binding/data-product projection и idempotency key, а право на
|
||||
операцию извлекается Foundry из отдельного opaque workload grant.
|
||||
|
||||
## Engine opaque credential sink
|
||||
## Native NDC L2 Credentials
|
||||
|
||||
`src/engine-credential-sink.mjs` задаёт dependency-free server-to-server v1
|
||||
границу для доставки трёх workload capabilities в Engine Credentials:
|
||||
Provider auth и внутренние workload capabilities используют один общий
|
||||
секретный boundary — native Credentials NDC L2, — но являются разными
|
||||
credential domains. После сохранения ядро владеет secret, а graph и MCP
|
||||
используют только opaque reference.
|
||||
|
||||
- `external-data-plane.writer` → точная нода
|
||||
`n8n-nodes-ndc.ndcDataProductPublish` / `ndcDataProductWriterApi`;
|
||||
- `external-data-plane.reader` → точная нода
|
||||
`n8n-nodes-ndc.ndcDataProductRead` / `ndcDataProductReaderApi`;
|
||||
- `foundry.binding` → точная нода
|
||||
`n8n-nodes-ndc.ndcFoundryBinding` / `ndcFoundryBindingApi`.
|
||||
Gelios выдаёт ровно access token и refresh token. Текущий credential type
|
||||
`httpBearerAuth` использует access token для HTTP request. Автоматический обмен
|
||||
refresh → access в текущем runtime не доказан и поэтому не заявлен: package
|
||||
фиксирует `refreshMode: operator_managed`. Название credential с текстом вроде
|
||||
`read access` является лишь локальной меткой; read-классификацию задают
|
||||
разрешённые endpoint/method в capability catalog и workflow policy, а не scope
|
||||
самого access token. В deployed Engine exact method/path policy ещё не
|
||||
подключена: текущая HTTP safe-ref граница проверяет host. Поэтому catalog
|
||||
classification остаётся декларативной до capability-bound Engine/MCP proof.
|
||||
|
||||
Provision request фиксирует `workflowId`, `workflowRevision`, `nodeId`, runtime
|
||||
node type, credential type, grant ID, expiry и issuer policy hash. Aggregate
|
||||
`transaction.policyHash` детерминированно считается по всему secret-free
|
||||
descriptor; подмена любой цели или policy ломает валидацию. Единственное поле,
|
||||
которое переносит plaintext capability, — `bindings[].material.value`; request
|
||||
нельзя писать в логи, traces, очередь или audit.
|
||||
Для общего Data Product transport используются `ndcDataProductWriterApi`,
|
||||
`ndcDataProductReaderApi` и `ndcFoundryBindingApi`. Это внутренние NDC
|
||||
capabilities, не Gelios tokens и не часть Gelios token lifecycle. Их значения
|
||||
не являются частью provider package и не передаются между workflow nodes как
|
||||
данные.
|
||||
|
||||
Provision и rollback envelopes действуют не более 15 минут: sink отклоняет
|
||||
истёкшие запросы и допускает максимум 60 секунд положительного clock skew.
|
||||
Receipt повторно проверяет freshness относительно собственного `processedAt`,
|
||||
чтобы старый запрос нельзя было применить или подтвердить через replay.
|
||||
Connection caller передаёт только provider credential reference. Publisher
|
||||
role остаётся credential binding для `NDC Data Product Publish`, но template
|
||||
маркирует его `management: control_plane_managed`: ни writer secret, ни writer
|
||||
reference не входят в connection parameters. Instantiation возвращает
|
||||
декларативный `systemBindings.publisher` с desired state и unresolved status;
|
||||
дальше его разрешает trusted control plane.
|
||||
|
||||
Каждый binding содержит `capabilityDigest = sha256(material.value)`: Engine
|
||||
самостоятельно хеширует полученный plaintext и сравнивает digest. Aggregate
|
||||
`policyHash` включает этот digest и issuer identity, а provision transaction
|
||||
несёт обязательную Ed25519 attestation. Engine принимает её только по
|
||||
allowlisted `serviceId:keyId`; заменить capability и пересчитать обычный hash
|
||||
без приватного issuer key невозможно.
|
||||
Canonical managed EDP writer capability генерируется и сохраняется внутри
|
||||
Engine credential boundary, а EDP получает только digest. Старые manual
|
||||
writer/reader и Foundry issuance paths могут возвращать capability один раз
|
||||
trusted control-plane caller и остаются отдельно закрытыми compatibility
|
||||
границами; MCP и пользователь получают только opaque reference/status.
|
||||
Запрещены password input
|
||||
как пользовательский acceptance-путь, graph parameters, connection/profile
|
||||
files, env, Ops, logs и traces.
|
||||
|
||||
Sink обязан выполнять `rollback-all`: сначала проверить весь request и точное
|
||||
состояние graph, затем создать credentials в staging, атомарно привязать весь
|
||||
набор и только после commit вернуть opaque `credentialRef`. При любой ошибке
|
||||
новые credentials удаляются, а прежние bindings остаются без изменений.
|
||||
`rollback-failed` означает карантин и ручное восстановление, но никогда не
|
||||
возвращает частичные credential refs.
|
||||
Broad credential sink/resolver/daemon и provider-specific provisioning
|
||||
запрещены. Требуется узкий generic `ensure data-product publish grant` contract
|
||||
с server-derived scope, idempotency, CAS/crash recovery и audit. Текущий
|
||||
deployed Engine MCP этой операции ещё не имеет; root/UI transfer допустим
|
||||
только как emergency self-hosted diagnostics, не как user journey.
|
||||
|
||||
Receipt, rollback request/receipt и audit имеют отдельные strict schemas. Они
|
||||
не способны вернуть capability material; audit хранит только hash opaque
|
||||
credential reference. Explicit rollback адресует committed transaction через
|
||||
`transactionId`, `policyHash` и hash committed receipt, поэтому не может
|
||||
случайно откатить другой набор. Реализация sink принадлежит Engine и не даёт
|
||||
Platform/Codex доступа к Engine core, runtime files или plaintext credential
|
||||
storage.
|
||||
|
||||
## Engine private-extension management
|
||||
## NDC L2 private-extension management
|
||||
|
||||
`src/engine-private-extension.mjs` задаёт строгую Platform-side границу для
|
||||
Engine-owned активации проверенного `n8n-nodes-ndc` release. Контракт не
|
||||
устанавливает package и не меняет Engine: он фиксирует async
|
||||
NDC L2-owned активации проверенного `n8n-nodes-ndc` release. Контракт не
|
||||
устанавливает package и не меняет NDC L2: он фиксирует async
|
||||
`plan -> apply receipt -> operation/status` protocol, где `apply` обязан
|
||||
быстро вернуть `queued` и `operationId`, а долгий recreate/acceptance
|
||||
отслеживается отдельно.
|
||||
|
||||
Активация и rollback требуют отдельной глобальной capability
|
||||
`engine.private-extension.manage`. Обычные L1/L2 grants её не дают. Чтение
|
||||
`engine.private-extension.manage`. Обычные NDC L1/NDC L2 grants её не дают. Чтение
|
||||
состояния допускает `engine.private-extension.read` или manage-capability.
|
||||
Запрос выбирает только allowlisted package, digest-bound `releaseId` и
|
||||
`packageSha256`; caller не передаёт host path, package bytes, Compose service,
|
||||
shell command или credential material. Plan живёт не более 15 минут, является
|
||||
single-use и применяется только с тем же idempotency key и plan hash.
|
||||
|
||||
Runtime transition для n8n 2.3.2 зафиксирован как community-package loader по
|
||||
`/home/node/.n8n/nodes/node_modules/n8n-nodes-ndc`, а не как
|
||||
`N8N_CUSTOM_EXTENSIONS` или `CUSTOM.*` loader:
|
||||
Runtime transition использует platform-managed community-package loader для
|
||||
immutable `n8n-nodes-ndc` release; private runtime paths и environment settings
|
||||
не являются частью публичного provider contract:
|
||||
|
||||
- `N8N_COMMUNITY_PACKAGES_ENABLED=true`,
|
||||
`N8N_COMMUNITY_PACKAGES_PREVENT_LOADING=false`,
|
||||
`N8N_REINSTALL_MISSING_PACKAGES=false`;
|
||||
- Deploy/Run quiesced and execution queue drained before the version switch;
|
||||
- read-only mount and atomic current/recovery state;
|
||||
- force-recreate main, every worker and every webhook instance as one version
|
||||
|
|
@ -189,24 +217,25 @@ Runtime transition для n8n 2.3.2 зафиксирован как community-pa
|
|||
|
||||
Любая ошибка после switch запускает automatic rollback и повторный
|
||||
force-recreate/acceptance. Ошибка самого rollback переводит runtime в
|
||||
`quarantined`. Immutable release и существующие Engine Credentials
|
||||
`quarantined`. Immutable release и существующие NDC L2 Credentials
|
||||
сохраняются. Для первой активации предыдущим проверенным состоянием является
|
||||
`n8n-nodes-ndc.inactive/v1`: rollback в этот baseline удаляет package из
|
||||
loader surface, но не удаляет credentials.
|
||||
|
||||
Public Ops gateway уже умеет прозрачно передавать эти операции через
|
||||
`/engine/mcp`, если Engine реализует соответствующие MCP tools. Так как
|
||||
`/engine/mcp`, если NDC L2 реализует соответствующие MCP tools. Так как
|
||||
gateway имеет 30-second upstream timeout, side effect остаётся асинхронным;
|
||||
отдельный public REST proxy для management boundary не требуется.
|
||||
|
||||
Пример `examples/gelios-positions-current.v1.mjs` — provider-specific fixture
|
||||
без customer, tenant identity или credential material.
|
||||
`providers/gelios/v1/fixtures` содержит synthetic provider response и ожидаемый
|
||||
publish contract без customer, tenant identity или credential material.
|
||||
|
||||
## Ownership
|
||||
|
||||
- `platform/packages/external-provider-contract` — общий контракт и schemas.
|
||||
- NDC Agent L2 — provider API adapter: fetch, pagination, batching,
|
||||
semantic mapping, collection profile и ссылка на credential в Engine.
|
||||
- NDC L2 — provider API adapter: fetch, pagination, batching,
|
||||
semantic mapping, collection profile и ссылка на credential в NDC L2
|
||||
Credentials.
|
||||
- Platform External Data Plane — provider-neutral intake, raw retention,
|
||||
canonical facts, current/history projections и scoped read products. Он не
|
||||
знает provider fields, customer/business filters или renderer rules.
|
||||
|
|
@ -222,46 +251,100 @@ gateway имеет 30-second upstream timeout, side effect остаётся ас
|
|||
## Mandatory connection boundary
|
||||
|
||||
`connection` принадлежит одному tenant/client context и содержит только
|
||||
ссылку на секрет, утверждённый capability scope, collection profile, field
|
||||
policy и retention policy. Во всех runtime records Data Plane сохраняет минимум
|
||||
ссылку на provider secret, утверждённый capability scope, collection profile,
|
||||
field policy и retention policy. Во всех runtime records Data Plane сохраняет минимум
|
||||
`tenant_id`, `connection_id`, `provider_id`, `observed_at`, `received_at` и
|
||||
provenance/version там, где это применимо.
|
||||
|
||||
Writer token — отдельный EDP runtime credential, а не поле `Connection`,
|
||||
profile, provider manifest или L2 graph.
|
||||
Provider credential reference текущего Gelios request адресует access token;
|
||||
наличие provider-issued refresh token описывается `tokenLifecycle`, но его value
|
||||
не входит в `Connection`. Writer capability — отдельный внутренний EDP runtime
|
||||
credential, а не Gelios token и не caller-supplied поле `Connection`, profile,
|
||||
provider manifest или NDC L2 graph. Connection artifact содержит только
|
||||
system-managed publisher declaration/status без credential reference.
|
||||
|
||||
## Scoped writer binding
|
||||
|
||||
Writer binding — EDP-owned runtime security state, а не versioned artifact
|
||||
provider manifest или connection profile. Он фиксирует `tenantId`,
|
||||
`connectionId`, `providerId`, `allowedDataProductIds`, active/revoked state и
|
||||
`expiresAt`. L2 не может редактировать binding или задавать его scope;
|
||||
`expiresAt`. NDC L2 не может редактировать binding или задавать его scope;
|
||||
изменение любого scope-поля либо TTL создаёт новый binding, а прежний binding
|
||||
можно только rotate/revoke.
|
||||
|
||||
Новый L2 вызывает `POST /internal/data-plane/v1/intake/writer-bound` с
|
||||
`Authorization: Bearer <writer-token>` и unscoped envelope. В `source`
|
||||
разрешён только `providerId`; `tenantId`, `connectionId` и
|
||||
`x-nodedc-*-id` headers запрещены. EDP проверяет token, binding, provider и
|
||||
`contract.dataProductId`, затем materializes immutable canonical scope и
|
||||
сохраняет обычный `Intake Batch`. Caller-provided scope отклоняется, а не
|
||||
доверяется и не объединяется с binding.
|
||||
Binding создаёт, сохраняет и привязывает только trusted control plane. Caller
|
||||
connection instance не принимает `writerCredentialRef`; publisher role в L2
|
||||
template является обязательным `control_plane_managed` binding requirement.
|
||||
|
||||
Plaintext writer token возвращается только trusted provisioner при создании
|
||||
или rotation. Provisioner помещает его непосредственно в opaque Engine
|
||||
credential, доступный назначенному L2. EDP хранит только hash token и binding
|
||||
metadata; token запрещён в connection/profile/manifest, L2 graph, logs/traces,
|
||||
raw payload, Foundry и UI.
|
||||
Новый NDC L2 вызывает
|
||||
`POST /internal/data-plane/v1/data-products/:dataProductId/publish` с opaque
|
||||
writer capability и envelope `nodedc.data-product.publish/v1`. В body есть
|
||||
только batch identity и canonical facts; `source`, provider, tenant,
|
||||
connection, ontology revision, product version и persistence policy запрещены.
|
||||
EDP проверяет binding и path `dataProductId`, materializes immutable scope и
|
||||
contract из server-owned registries, затем сохраняет canonical batch.
|
||||
Caller-provided scope отклоняется, а не доверяется и не объединяется с
|
||||
binding.
|
||||
|
||||
Provisioning routes принимают только отдельный secret file, созданный
|
||||
root-owned deploy runner:
|
||||
`/volume1/docker/nodedc-platform/secrets/external-data-plane-provisioner/token`.
|
||||
Он read-only монтируется только в EDP и позднее — в dedicated Engine
|
||||
provisioner, работающий под выделенным UID/GID `11006`; это не `.env` value,
|
||||
не `NODEDC_INTERNAL_ACCESS_TOKEN` и не provider credential. Пока generic Engine
|
||||
provisioner не развёрнут, `EXTERNAL_DATA_PLANE_PROVISIONING_ENABLED=false`, а
|
||||
create/rotate/revoke routes отвечают `503` и не делают fallback к shared
|
||||
internal bearer.
|
||||
Plaintext writer capability запрещена в provider package, connection/profile,
|
||||
NDC L2 graph, files, environment, logs/traces, raw payload, Foundry и MCP.
|
||||
В canonical пути она генерируется и сохраняется внутри native Engine credential
|
||||
boundary; EDP получает только SHA-256 digest, а workflow — только opaque
|
||||
reference/status. Старые ручные EDP create/rotate endpoints с one-time
|
||||
plaintext response остаются отдельно выключенным compatibility-механизмом и не
|
||||
являются пользовательским или acceptance-путём.
|
||||
|
||||
Legacy manual one-time issuance использует отдельный EDP-only runner-managed
|
||||
bearer. Digest-only managed ensure этот bearer не принимает: Engine подписывает
|
||||
каждый control-plane запрос отдельным Ed25519 service key, EDP хранит только
|
||||
публичный ключ. Оба пути имеют независимые флаги и по умолчанию выключены.
|
||||
Generic `ensure data-product publish grant` генерирует и сохраняет capability в
|
||||
native Engine credential boundary, передаёт EDP её digest и возвращает только
|
||||
opaque ref/status. Shared internal bearer и Gelios access/refresh token для
|
||||
этого запрещены. Deployed Engine MCP умеет читать
|
||||
opaque refs и bind только HTTP Request credentials; native
|
||||
`ndcDataProductWriterApi` к custom Publish node этим policy не привязывается.
|
||||
Поэтому ensure + exact private-node bind остаются реальным platform gap.
|
||||
Root/UI перенос — только аварийная диагностика.
|
||||
|
||||
Control-plane EDP boundary — idempotent
|
||||
`PUT /internal/data-plane/v1/writer-bindings/by-key/:bindingKey`. Engine создаёт
|
||||
capability внутри native credential boundary и отправляет только её SHA-256
|
||||
`capabilityDigest`, immutable scope, `generation` и expiry. Одинаковый
|
||||
`bindingKey + generation + request hash` возвращает тот же binding без нового
|
||||
secret; несовпадающий retry получает `409`. Rotation использует новую
|
||||
generation/credential, поэтому старый binding можно оставить активным до
|
||||
успешного node bind и затем явно revoke.
|
||||
Managed revoke выполняется идемпотентно по точным `bindingKey + generation`
|
||||
через managed-only route; для него не открывается legacy plaintext API.
|
||||
Managed route включается только через
|
||||
`EXTERNAL_DATA_PLANE_MANAGED_PROVISIONING_ENABLED`; legacy plaintext routes —
|
||||
через отдельный `EXTERNAL_DATA_PLANE_PROVISIONING_ENABLED`.
|
||||
|
||||
Managed auth wire contract —
|
||||
`nodedc.external-data-plane.managed-provisioner-request/v1`. Engine отправляет
|
||||
ровно по одному header:
|
||||
`x-nodedc-engine-service-id`, `x-nodedc-engine-key-id`,
|
||||
`x-nodedc-request-audience`, `x-nodedc-request-timestamp`,
|
||||
`x-nodedc-request-nonce`, `x-nodedc-content-sha256` и
|
||||
`x-nodedc-request-signature`. Timestamp — canonical UTC ISO с миллисекундами,
|
||||
nonce и signature — unpadded canonical base64url, body hash — lowercase hex
|
||||
SHA-256 от точных переданных bytes. Подписываются UTF-8 bytes результата
|
||||
`JSON.stringify` объекта с полями строго в порядке `schemaVersion`, `audience`,
|
||||
`serviceId`, `keyId`, `method`, `path`, `timestamp`, `nonce`, `bodySha256`.
|
||||
`method` и request target `path` (включая query) должны совпадать побайтно.
|
||||
Canonical identity: service `nodedc-engine`, key
|
||||
`engine-edp-managed-provisioner-v1`, audience
|
||||
`nodedc-external-data-plane.managed-provisioning.v1`. EDP проверяет bounded
|
||||
clock skew и одноразовый nonce в bounded fail-closed replay cache; bearer
|
||||
fallback для managed routes отсутствует, любой `Authorization` header на них
|
||||
отклоняется.
|
||||
|
||||
EDP импортирует только `@nodedc/external-provider-contract/data-plane`.
|
||||
Artifact/image allowlist для этого subpath содержит contract version, intake,
|
||||
publish/snapshot/patch validators и scrub policy; provider packages, mappings,
|
||||
fixtures и tests физически не входят в EDP runtime и не являются причиной его
|
||||
restart.
|
||||
|
||||
## Legacy intake migration
|
||||
|
||||
|
|
@ -271,22 +354,22 @@ existing writers. Он принимает только canonical scoped `Intake
|
|||
`x-nodedc-tenant-id` и `x-nodedc-connection-id`. Writer token этот route не
|
||||
заменяет.
|
||||
|
||||
Для миграции: создать writer binding, один раз записать выданный token в Engine
|
||||
credential, переключить L2 на `/intake/writer-bound`, удалить tenant/connection
|
||||
из body и headers, проверить успешный intake, затем удалить у L2 legacy
|
||||
credential. Новый или migrated writer не должен fallback-иться на legacy route
|
||||
после ошибки writer-bound intake. Когда migrated все writers, legacy route и
|
||||
его shared-token access удаляются.
|
||||
Для миграции: Platform обеспечивает scoped writer binding и native credential,
|
||||
затем через Engine MCP новый workflow branch переключается на Data Product
|
||||
publish route, удаляет tenant/connection из body и headers и проверяет
|
||||
успешный publish. Существующий legacy workflow/credential остаётся неизменным,
|
||||
пока владелец отдельно не решит его вывести; новый writer не должен
|
||||
fallback-иться на legacy route после ошибки publish.
|
||||
|
||||
## Intake boundary
|
||||
|
||||
L2 отправляет в общий Data Plane `Intake Batch`, а не SQL-запрос в общие
|
||||
NDC L2 отправляет в общий Data Plane publish request, а не SQL-запрос в общие
|
||||
таблицы. Platform проверяет boundary и сохраняет raw/history/current
|
||||
projections; semantic mapping остаётся в L2. Список конкретных source IDs не
|
||||
projections; semantic mapping остаётся в NDC L2. Список конкретных source IDs не
|
||||
является частью connection scope: scope выбирает read-capability API, а
|
||||
visibility решает data-product consumer.
|
||||
|
||||
Inline `raw.payload` запрещён: L2 не может записать в Data Plane полный ответ
|
||||
Inline `raw.payload` запрещён: NDC L2 не может записать в Data Plane полный ответ
|
||||
provider-а или произвольную строку. Пока отдельный raw-vault не утверждён,
|
||||
batch либо не содержит `raw`, либо содержит только restricted `raw.ref` и hash.
|
||||
Secret-like keys и распознаваемые bearer/JWT/writer-token values в любом участке
|
||||
|
|
@ -299,4 +382,6 @@ service делает sweep expired envelopes при старте и по рас
|
|||
Capabilities классифицируются как `read`, `metadata`, `write`, `destructive`
|
||||
или `unknown`. Только `read` и согласованные `metadata` могут попасть в
|
||||
collector. `write` и `destructive` остаются каталогизированными, но не имеют
|
||||
transport route в read adapter.
|
||||
transport route в read adapter. Эта классификация относится к разрешённому
|
||||
method/endpoint workflow, а не утверждает, что provider access token имеет
|
||||
read-only scope.
|
||||
|
|
|
|||
|
|
@ -8,15 +8,14 @@ export const geliosPositionsCurrentExample = Object.freeze({
|
|||
version: "1.0.0",
|
||||
ontology: {
|
||||
packageId: "gelios",
|
||||
revision: "gelios.positions.v1",
|
||||
revision: "ontology.gelios.v1",
|
||||
},
|
||||
l2Template: {
|
||||
id: "gelios.positions.current",
|
||||
id: "gelios.positions.current.l2.v1",
|
||||
version: "1.0.0",
|
||||
},
|
||||
capabilities: [
|
||||
{ id: "gelios.units.current.read", classification: "read" },
|
||||
{ id: "gelios.units.command.write", classification: "write" },
|
||||
],
|
||||
dataProductIds: ["fleet.positions.current.v1"],
|
||||
},
|
||||
|
|
@ -25,10 +24,10 @@ export const geliosPositionsCurrentExample = Object.freeze({
|
|||
id: "gelios.sample-fleet",
|
||||
providerId: "gelios",
|
||||
tenantId: "sample-tenant",
|
||||
credentialRef: { owner: "engine", reference: "engine-credential-reference" },
|
||||
credentialRef: { owner: "ndc_l2_credentials", reference: "ndc-credref:provider-example-0001" },
|
||||
scope: {
|
||||
capabilityIds: ["gelios.units.current.read"],
|
||||
fieldPolicyId: "fleet.position.display.v1",
|
||||
fieldPolicyId: "gelios.positions.current.fields.v1",
|
||||
},
|
||||
},
|
||||
collectionProfile: {
|
||||
|
|
@ -36,7 +35,7 @@ export const geliosPositionsCurrentExample = Object.freeze({
|
|||
id: "gelios.positions.realtime",
|
||||
connectionId: "gelios.sample-fleet",
|
||||
mode: "realtime",
|
||||
schedule: { intervalMs: 15000 },
|
||||
schedule: { intervalMs: 10000 },
|
||||
capabilityIds: ["gelios.units.current.read"],
|
||||
dataProductId: "fleet.positions.current.v1",
|
||||
},
|
||||
|
|
|
|||
|
|
@ -3,8 +3,12 @@
|
|||
"version": "0.1.0",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"exports": "./src/index.mjs",
|
||||
"exports": {
|
||||
".": "./src/index.mjs",
|
||||
"./data-plane": "./src/data-plane.mjs",
|
||||
"./providers/*": "./providers/*/index.mjs"
|
||||
},
|
||||
"scripts": {
|
||||
"check": "node test/contract.test.mjs && node test/data-product.test.mjs && node test/engine-credential-sink.test.mjs && node test/engine-private-extension.test.mjs"
|
||||
"check": "node test/contract.test.mjs && node test/data-product.test.mjs && node test/provider-package.test.mjs && node test/engine-private-extension.test.mjs"
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -0,0 +1,105 @@
|
|||
# Gelios provider package v1
|
||||
|
||||
Это versioned contract/data package для первого живого provider connection в
|
||||
NODE.DC. Он не является runtime service, custom node, account configuration или
|
||||
хранилищем telemetry. Provider-specific request и mapping исполняются внутри
|
||||
изолированного NDC L2 workflow; Platform получает только provider-neutral Data
|
||||
Product.
|
||||
|
||||
Package фиксирует:
|
||||
|
||||
- manifest `gelios.provider.v1@1.0.0`;
|
||||
- точный provider token lifecycle: Gelios выдаёт `access` и `refresh`, а
|
||||
HTTP request использует `access`;
|
||||
- safe-read capability `gelios.units.current.read`
|
||||
(`GET https://api.geliospro.com/api/v1/units` with an opaque bearer
|
||||
credential);
|
||||
- dynamic scope `all_visible_to_credential` без unit/group allowlist;
|
||||
- realtime profile с интервалом 10 секунд и manual profile;
|
||||
- fail-closed field policy: неизвестные и dynamic fields отбрасываются до
|
||||
классификации, hardware IDs, phone/address, raw params и sensor payloads не
|
||||
публикуются;
|
||||
- mapping в `map.moving_object`;
|
||||
- стабильный source ID `gelios-unit-<sourceUnitId>` с точным сохранением
|
||||
строкового provider ID;
|
||||
- explicit derivation rules для `position_valid`, `operational_status` и
|
||||
`quality_flags`, включая `no_position` и stale threshold 60 секунд;
|
||||
- точный Data Product `fleet.positions.current.v1@1.0.0`, revision
|
||||
`ontology.map.moving_object.v1`, с 13 разрешёнными snake_case fields;
|
||||
- provider-neutral NDC L2 template descriptor из пяти boundary steps:
|
||||
trigger → request → extract → map → `NDC Data Product Publish`;
|
||||
- publish credential binding с `management: control_plane_managed`, без
|
||||
caller-supplied writer reference;
|
||||
- synthetic source/publish fixture, включая видимый объект без координат.
|
||||
|
||||
## Authentication boundary
|
||||
|
||||
Gelios выдаёт ровно два provider secret artifact: access token и refresh
|
||||
token. Это не отдельные read/write tokens. Текущий NDC L2 HTTP binding типа
|
||||
`httpBearerAuth` подставляет access token в `Authorization`; автоматический
|
||||
refresh этим runtime пока не доказан, поэтому package явно фиксирует
|
||||
`refreshMode: operator_managed`. Ни access, ни refresh value не попадает в
|
||||
package, graph, fixture, MCP, Ops или trace.
|
||||
|
||||
Название native credential, например `Gelios — Robot2B — read access`, является
|
||||
только операторской меткой. `read` в `gelios.units.current.read` —
|
||||
классификация разрешённого workflow endpoint `GET /api/v1/units`, а не scope
|
||||
access token. Метка credential не создаёт отдельный «read token» и не сужает
|
||||
права, которые фактически выдал Gelios.
|
||||
|
||||
Unit без last message не исчезает: mapping использует collection receive time
|
||||
как `observedAt` fallback и публикует subject без geometry с
|
||||
`position_valid=false`/`operational_status=no_position`.
|
||||
|
||||
Префикс source ID для v1 — строго `gelios-unit-`. Сокращённый `unit-` не
|
||||
является каноническим: он создал бы второй entity key рядом с уже сохранённым
|
||||
Gelios subject. Значение provider ID после префикса преобразуется только в
|
||||
строку; ведущие нули и остальные значимые символы не нормализуются.
|
||||
|
||||
`fleet.positions.current.v1` ограничен 5000 current entities. Если credential
|
||||
видит больше, NDC L2 не должен отбрасывать «лишние» units: package/profile
|
||||
требует остановить Deploy до новой partitioned Data Product revision. Полное
|
||||
автоматическое enforcement этого правила остаётся gate будущего
|
||||
package-to-graph compiler/runtime validator.
|
||||
|
||||
Один package обслуживает любое число accounts. Для каждого account вызывается
|
||||
`instantiateL2Connection` с новым `tenantId`, `connectionId`, provider
|
||||
credential reference. Это единственная credential reference, которую передаёт
|
||||
caller: она адресует Gelios access binding в native NDC L2 Credentials.
|
||||
Внутреннюю capability для публикации в External Data Plane выдаёт и сохраняет
|
||||
control plane; caller не передаёт writer secret или `writerCredentialRef`.
|
||||
Результат instantiation содержит только provider ref и декларативный
|
||||
`systemBindings.publisher` со статусом `unresolved`, который Platform должна
|
||||
разрешить в native credential binding. Эта capability не выдаётся Gelios, не
|
||||
является третьим Gelios token и не описывает права provider account. Добавление
|
||||
account не меняет Platform source.
|
||||
|
||||
Расширение Gelios происходит новой версией package: сначала подтверждаются
|
||||
provider capability/fields, затем обновляются ontology, field policy, mapping,
|
||||
fixtures и tests. Пользовательские workflow, которые уже покрываются
|
||||
существующим catalog, не требуют участия разработчика Platform.
|
||||
|
||||
Технический runtime type publish boundary остаётся
|
||||
`n8n-nodes-ndc.ndcDataProductPublish`; это идентификатор общего NDC package, а
|
||||
не Gelios-specific node.
|
||||
|
||||
## Текущая готовность
|
||||
|
||||
Package, ontology links и synthetic fixtures проверены в source. Production
|
||||
Engine L2 target использует зарегистрированный Gelios Bearer credential на
|
||||
точном safe-read endpoint и managed `ndcDataProductWriterApi` generation g3;
|
||||
оба binding записаны без раскрытия secret values. Canonical graph содержит
|
||||
provider read, extraction, mapping и native `NDC Data Product Publish`.
|
||||
|
||||
Fresh production execution `881722` завершился успешно и опубликовал
|
||||
`fleet.positions.current.v1`; managed writer g3 принят и имеет state `current`.
|
||||
Это доказывает manual provider read → canonical Data Product path. Внутренняя
|
||||
writer capability остаётся NODE.DC capability и не может быть заменена access
|
||||
или refresh token Gelios.
|
||||
|
||||
Не доказаны: автоматический refresh, Schedule Trigger 10 секунд, несколько
|
||||
последовательных scheduled cycles, retry/backoff, sampled history/retention на
|
||||
continuous path и Foundry snapshot/patch → moving pins E2E. Provider package
|
||||
остаётся declarative: MCP package discovery и package/profile → L2 graph
|
||||
compiler пока отсутствуют, поэтому первый graph собран через общие Engine MCP
|
||||
graph tools.
|
||||
|
|
@ -0,0 +1,74 @@
|
|||
import { DATA_PRODUCT_PUBLISH_SCHEMA_VERSION } from "../../../../src/data-product.mjs";
|
||||
|
||||
export const geliosUnitsCurrentFixtureV1 = Object.freeze({
|
||||
sourceResponse: {
|
||||
units: [{
|
||||
id: 1001,
|
||||
name: "Synthetic unit 1001",
|
||||
hw_id: "restricted-fixture-value",
|
||||
phone: "restricted-fixture-value",
|
||||
lastMsg: {
|
||||
time: 1784203200,
|
||||
lat: "55.7558",
|
||||
lon: "37.6173",
|
||||
speed: 24.5,
|
||||
course: 91,
|
||||
height: 164,
|
||||
gps_valid: true,
|
||||
sats: 11,
|
||||
hdop: 0.8,
|
||||
accuracy: 4.2,
|
||||
params: "restricted-fixture-value",
|
||||
},
|
||||
}, {
|
||||
id: "001002",
|
||||
name: "Synthetic unit 1002",
|
||||
}],
|
||||
},
|
||||
collectionContext: {
|
||||
receivedAt: "2026-07-16T12:00:00.000Z",
|
||||
},
|
||||
expectedPublish: {
|
||||
schemaVersion: DATA_PRODUCT_PUBLISH_SCHEMA_VERSION,
|
||||
batch: {
|
||||
runId: "gelios-fixture-run-20260716",
|
||||
sequence: 0,
|
||||
idempotencyKey: "gelios-fixture-run-20260716.batch-0",
|
||||
},
|
||||
facts: [{
|
||||
sourceId: "gelios-unit-1001",
|
||||
semanticType: "map.moving_object",
|
||||
observedAt: "2026-07-16T12:00:00.000Z",
|
||||
geometry: {
|
||||
type: "Point",
|
||||
coordinates: [37.6173, 55.7558],
|
||||
},
|
||||
attributes: {
|
||||
course_degrees: 91,
|
||||
display_name: "Synthetic unit 1001",
|
||||
elevation_meters: 164,
|
||||
hdop: 0.8,
|
||||
horizontal_accuracy_meters: 4.2,
|
||||
object_kind: "tracked_unit",
|
||||
operational_status: "active",
|
||||
position_source: "gelios",
|
||||
position_valid: true,
|
||||
quality_flags: [],
|
||||
satellite_count: 11,
|
||||
speed_kph: 24.5,
|
||||
},
|
||||
}, {
|
||||
sourceId: "gelios-unit-001002",
|
||||
semanticType: "map.moving_object",
|
||||
observedAt: "2026-07-16T12:00:00.000Z",
|
||||
attributes: {
|
||||
display_name: "Synthetic unit 1002",
|
||||
object_kind: "tracked_unit",
|
||||
operational_status: "no_position",
|
||||
position_source: "gelios",
|
||||
position_valid: false,
|
||||
quality_flags: ["missing_geometry"],
|
||||
},
|
||||
}],
|
||||
},
|
||||
});
|
||||
|
|
@ -0,0 +1,11 @@
|
|||
export {
|
||||
GELIOS_POSITIONS_DATA_PRODUCT_ID,
|
||||
GELIOS_POSITIONS_DATA_PRODUCT_VERSION,
|
||||
GELIOS_POSITIONS_ONTOLOGY_REVISION,
|
||||
GELIOS_PROVIDER_PACKAGE_ID,
|
||||
GELIOS_PROVIDER_PACKAGE_VERSION,
|
||||
GELIOS_UNIT_SOURCE_ID_PREFIX,
|
||||
geliosProviderPackageV1,
|
||||
} from "./package.mjs";
|
||||
|
||||
export { geliosUnitsCurrentFixtureV1 } from "./fixtures/units-current.fixture.mjs";
|
||||
|
|
@ -0,0 +1,374 @@
|
|||
import {
|
||||
L2_TEMPLATE_SCHEMA_VERSION,
|
||||
PROVIDER_PACKAGE_SCHEMA_VERSION,
|
||||
SEMANTIC_MAPPING_SCHEMA_VERSION,
|
||||
} from "../../../src/provider-package.mjs";
|
||||
|
||||
export const GELIOS_PROVIDER_PACKAGE_ID = "gelios.provider.v1";
|
||||
export const GELIOS_PROVIDER_PACKAGE_VERSION = "1.0.0";
|
||||
export const GELIOS_POSITIONS_DATA_PRODUCT_ID = "fleet.positions.current.v1";
|
||||
export const GELIOS_POSITIONS_DATA_PRODUCT_VERSION = "1.0.0";
|
||||
export const GELIOS_POSITIONS_ONTOLOGY_REVISION = "ontology.map.moving_object.v1";
|
||||
export const GELIOS_UNIT_SOURCE_ID_PREFIX = "gelios-unit-";
|
||||
|
||||
const targetFields = Object.freeze([
|
||||
"course_degrees",
|
||||
"display_name",
|
||||
"elevation_meters",
|
||||
"geometry",
|
||||
"hdop",
|
||||
"horizontal_accuracy_meters",
|
||||
"object_kind",
|
||||
"operational_status",
|
||||
"position_source",
|
||||
"position_valid",
|
||||
"quality_flags",
|
||||
"satellite_count",
|
||||
"speed_kph",
|
||||
]);
|
||||
|
||||
export const geliosProviderPackageV1 = deepFreeze({
|
||||
schemaVersion: PROVIDER_PACKAGE_SCHEMA_VERSION,
|
||||
id: GELIOS_PROVIDER_PACKAGE_ID,
|
||||
providerId: "gelios",
|
||||
version: GELIOS_PROVIDER_PACKAGE_VERSION,
|
||||
manifest: {
|
||||
id: "gelios.provider.manifest.v1",
|
||||
providerId: "gelios",
|
||||
version: GELIOS_PROVIDER_PACKAGE_VERSION,
|
||||
ontology: {
|
||||
packageId: "gelios",
|
||||
revision: "ontology.gelios.v1",
|
||||
},
|
||||
authModeIds: ["gelios.rest-bearer.v1"],
|
||||
capabilityIds: ["gelios.units.current.read"],
|
||||
fieldPolicyIds: ["gelios.positions.current.fields.v1"],
|
||||
collectionProfileIds: [
|
||||
"gelios.positions.current.realtime.v1",
|
||||
"gelios.positions.current.manual.v1",
|
||||
],
|
||||
dataProductIds: [GELIOS_POSITIONS_DATA_PRODUCT_ID],
|
||||
mappingContractIds: ["gelios.units.to.fleet.positions.current.v1"],
|
||||
l2TemplateIds: ["gelios.positions.current.l2.v1"],
|
||||
},
|
||||
authModes: [{
|
||||
id: "gelios.rest-bearer.v1",
|
||||
kind: "api_token",
|
||||
credentialOwner: "ndc_l2_credentials",
|
||||
bindingCardinality: "one_per_connection",
|
||||
transport: {
|
||||
placement: "header",
|
||||
name: "Authorization",
|
||||
},
|
||||
tokenLifecycle: {
|
||||
artifacts: ["access", "refresh"],
|
||||
requestArtifact: "access",
|
||||
refreshMode: "operator_managed",
|
||||
},
|
||||
secretHandling: {
|
||||
store: "ndc_l2_credentials",
|
||||
exposeToGraph: false,
|
||||
persistInPackage: false,
|
||||
},
|
||||
}],
|
||||
capabilities: [{
|
||||
id: "gelios.units.current.read",
|
||||
classification: "read",
|
||||
status: "implemented",
|
||||
authModeId: "gelios.rest-bearer.v1",
|
||||
request: {
|
||||
method: "GET",
|
||||
baseUrl: "https://api.geliospro.com",
|
||||
path: "/api/v1/units",
|
||||
query: {},
|
||||
response: {
|
||||
collectionPaths: ["units", "data", "data.units", "result", "$"],
|
||||
pagination: "single_bounded_response",
|
||||
},
|
||||
},
|
||||
entityScope: {
|
||||
mode: "all_visible_to_credential",
|
||||
refresh: "each_collection_run",
|
||||
businessEntityFilter: "forbidden",
|
||||
},
|
||||
}],
|
||||
fieldPolicies: [{
|
||||
id: "gelios.positions.current.fields.v1",
|
||||
version: "1.0.0",
|
||||
dataProductId: GELIOS_POSITIONS_DATA_PRODUCT_ID,
|
||||
targetFields: [...targetFields],
|
||||
unknownSourceFields: "drop",
|
||||
dynamicSourceFields: "drop_until_classified",
|
||||
restrictedSourcePaths: [
|
||||
"decrypt_key",
|
||||
"hw_id",
|
||||
"imei",
|
||||
"info",
|
||||
"lmsg.address",
|
||||
"lmsg.params",
|
||||
"lmsg.sensors",
|
||||
"lastMsg.address",
|
||||
"lastMsg.params",
|
||||
"lastMsg.sensors",
|
||||
"lastMessage.address",
|
||||
"lastMessage.params",
|
||||
"lastMessage.sensors",
|
||||
"phone",
|
||||
"phone2",
|
||||
"sensors",
|
||||
],
|
||||
}],
|
||||
collectionProfiles: [{
|
||||
id: "gelios.positions.current.realtime.v1",
|
||||
version: "1.0.0",
|
||||
mode: "realtime",
|
||||
schedule: { intervalMs: 10000 },
|
||||
capabilityIds: ["gelios.units.current.read"],
|
||||
dataProductId: GELIOS_POSITIONS_DATA_PRODUCT_ID,
|
||||
mappingContractId: "gelios.units.to.fleet.positions.current.v1",
|
||||
fieldPolicyId: "gelios.positions.current.fields.v1",
|
||||
l2TemplateId: "gelios.positions.current.l2.v1",
|
||||
entityScope: {
|
||||
mode: "all_visible_to_credential",
|
||||
refresh: "each_collection_run",
|
||||
businessEntityFilter: "forbidden",
|
||||
},
|
||||
batching: { maxFacts: 5000 },
|
||||
cardinality: {
|
||||
maxCurrentEntities: 5000,
|
||||
onExceed: "require_partitioned_data_product",
|
||||
},
|
||||
retry: {
|
||||
maxAttempts: 4,
|
||||
backoff: "exponential_with_jitter",
|
||||
},
|
||||
}, {
|
||||
id: "gelios.positions.current.manual.v1",
|
||||
version: "1.0.0",
|
||||
mode: "manual",
|
||||
capabilityIds: ["gelios.units.current.read"],
|
||||
dataProductId: GELIOS_POSITIONS_DATA_PRODUCT_ID,
|
||||
mappingContractId: "gelios.units.to.fleet.positions.current.v1",
|
||||
fieldPolicyId: "gelios.positions.current.fields.v1",
|
||||
l2TemplateId: "gelios.positions.current.l2.v1",
|
||||
entityScope: {
|
||||
mode: "all_visible_to_credential",
|
||||
refresh: "each_collection_run",
|
||||
businessEntityFilter: "forbidden",
|
||||
},
|
||||
batching: { maxFacts: 5000 },
|
||||
cardinality: {
|
||||
maxCurrentEntities: 5000,
|
||||
onExceed: "require_partitioned_data_product",
|
||||
},
|
||||
retry: {
|
||||
maxAttempts: 4,
|
||||
backoff: "exponential_with_jitter",
|
||||
},
|
||||
}],
|
||||
dataProducts: [{
|
||||
id: GELIOS_POSITIONS_DATA_PRODUCT_ID,
|
||||
version: GELIOS_POSITIONS_DATA_PRODUCT_VERSION,
|
||||
ontologyRevision: GELIOS_POSITIONS_ONTOLOGY_REVISION,
|
||||
deliveryMode: "snapshot+patch",
|
||||
semanticTypes: ["map.moving_object"],
|
||||
fields: [...targetFields],
|
||||
history: {
|
||||
mode: "sampled",
|
||||
intervalMs: 60000,
|
||||
strategy: "latest-per-entity-per-bucket",
|
||||
retentionDays: 90,
|
||||
},
|
||||
}],
|
||||
mappingContracts: [{
|
||||
schemaVersion: SEMANTIC_MAPPING_SCHEMA_VERSION,
|
||||
id: "gelios.units.to.fleet.positions.current.v1",
|
||||
version: "1.0.0",
|
||||
sourceCapabilityId: "gelios.units.current.read",
|
||||
fieldPolicyId: "gelios.positions.current.fields.v1",
|
||||
target: {
|
||||
dataProductId: GELIOS_POSITIONS_DATA_PRODUCT_ID,
|
||||
version: GELIOS_POSITIONS_DATA_PRODUCT_VERSION,
|
||||
ontologyRevision: GELIOS_POSITIONS_ONTOLOGY_REVISION,
|
||||
semanticType: "map.moving_object",
|
||||
},
|
||||
derivations: {
|
||||
position_valid: {
|
||||
kind: "boolean_rule",
|
||||
rules: [
|
||||
"geometry.coordinates_finite_wgs84",
|
||||
"provider.gps_valid_not_false",
|
||||
],
|
||||
default: false,
|
||||
},
|
||||
operational_status: {
|
||||
kind: "ordered_rules",
|
||||
rules: [
|
||||
"position_invalid.no_position",
|
||||
"observed_age_gt_limit.stale",
|
||||
"otherwise.active",
|
||||
],
|
||||
default: "active",
|
||||
parameters: { staleAfterMs: 60000 },
|
||||
},
|
||||
quality_flags: {
|
||||
kind: "flag_set",
|
||||
rules: [
|
||||
"geometry_missing.missing_geometry",
|
||||
"provider_gps_false.gps_invalid",
|
||||
"satellite_count_zero.no_satellites",
|
||||
"observed_age_gt_limit.stale",
|
||||
],
|
||||
default: [],
|
||||
parameters: { staleAfterMs: 60000 },
|
||||
},
|
||||
},
|
||||
fact: {
|
||||
sourceId: {
|
||||
strategy: "first_non_empty",
|
||||
paths: ["id", "unit_id", "unitId"],
|
||||
coerce: "string",
|
||||
prefix: GELIOS_UNIT_SOURCE_ID_PREFIX,
|
||||
},
|
||||
semanticType: { constant: "map.moving_object" },
|
||||
observedAt: {
|
||||
strategy: "first_non_empty",
|
||||
paths: [
|
||||
"lmsg.time",
|
||||
"lmsg.ts",
|
||||
"lmsg.timestamp",
|
||||
"last_msg.time",
|
||||
"last_msg.timestamp",
|
||||
"lastMsg.time",
|
||||
"lastMessage.time",
|
||||
"last_msg_time",
|
||||
"lmsg_time",
|
||||
],
|
||||
coerce: "unix_or_iso_timestamp",
|
||||
fallback: "collection_received_at",
|
||||
},
|
||||
geometry: {
|
||||
type: "Point",
|
||||
longitude: {
|
||||
strategy: "first_non_empty",
|
||||
paths: ["lmsg.lon", "lmsg.lng", "lmsg.longitude", "last_msg.lon", "lastMsg.lon", "lastMsg.lng", "lastMsg.longitude", "lastMessage.lon", "lastMessage.lng", "lastMessage.longitude", "lon", "lng", "longitude"],
|
||||
coerce: "number",
|
||||
},
|
||||
latitude: {
|
||||
strategy: "first_non_empty",
|
||||
paths: ["lmsg.lat", "lmsg.latitude", "last_msg.lat", "lastMsg.lat", "lastMsg.latitude", "lastMessage.lat", "lastMessage.latitude", "lat", "latitude"],
|
||||
coerce: "number",
|
||||
},
|
||||
omitIfInvalid: true,
|
||||
},
|
||||
attributes: {
|
||||
course_degrees: {
|
||||
strategy: "first_non_empty",
|
||||
paths: ["lmsg.course", "lmsg.angle", "last_msg.course", "lastMsg.course", "lastMsg.angle", "lastMessage.course", "lastMessage.angle", "course"],
|
||||
coerce: "number",
|
||||
omitIfMissing: true,
|
||||
},
|
||||
display_name: {
|
||||
strategy: "first_non_empty",
|
||||
paths: ["name", "unit_name", "title", "label"],
|
||||
coerce: "string",
|
||||
},
|
||||
elevation_meters: {
|
||||
strategy: "first_non_empty",
|
||||
paths: ["lmsg.height", "lmsg.altitude", "last_msg.height", "lastMsg.height", "lastMsg.altitude", "lastMessage.height", "lastMessage.altitude", "height", "altitude"],
|
||||
coerce: "number",
|
||||
omitIfMissing: true,
|
||||
},
|
||||
hdop: {
|
||||
strategy: "first_non_empty",
|
||||
paths: ["lmsg.hdop", "last_msg.hdop", "lastMsg.hdop", "lastMessage.hdop", "gps_hdop", "hdop"],
|
||||
coerce: "number",
|
||||
omitIfMissing: true,
|
||||
},
|
||||
horizontal_accuracy_meters: {
|
||||
strategy: "first_non_empty",
|
||||
paths: ["lmsg.accuracy", "last_msg.accuracy", "lastMsg.accuracy", "lastMessage.accuracy", "gps_accuracy_m", "accuracy_m", "accuracy"],
|
||||
coerce: "number",
|
||||
omitIfMissing: true,
|
||||
},
|
||||
object_kind: { constant: "tracked_unit" },
|
||||
operational_status: { derive: "operational_status" },
|
||||
position_source: { constant: "gelios" },
|
||||
position_valid: { derive: "position_valid" },
|
||||
quality_flags: { derive: "quality_flags" },
|
||||
satellite_count: {
|
||||
strategy: "first_non_empty",
|
||||
paths: ["lmsg.sats", "lmsg.satellites", "last_msg.sats", "lastMsg.sats", "lastMsg.satellites", "lastMessage.sats", "lastMessage.satellites", "gps_sats", "sats", "satellites"],
|
||||
coerce: "number",
|
||||
omitIfMissing: true,
|
||||
},
|
||||
speed_kph: {
|
||||
strategy: "first_non_empty",
|
||||
paths: ["lmsg.speed", "last_msg.speed", "lastMsg.speed", "lastMessage.speed", "speed"],
|
||||
coerce: "number",
|
||||
omitIfMissing: true,
|
||||
},
|
||||
},
|
||||
},
|
||||
}],
|
||||
l2Templates: [{
|
||||
schemaVersion: L2_TEMPLATE_SCHEMA_VERSION,
|
||||
id: "gelios.positions.current.l2.v1",
|
||||
version: "1.0.0",
|
||||
runtime: "ndc_l2",
|
||||
instanceMode: "one_connection_per_workflow",
|
||||
connectionParameters: [
|
||||
"tenant_id",
|
||||
"connection_id",
|
||||
"collection_profile_id",
|
||||
"provider_credential_ref",
|
||||
],
|
||||
credentialBindings: [{
|
||||
role: "provider",
|
||||
owner: "ndc_l2_credentials",
|
||||
management: "connection_input",
|
||||
authModeId: "gelios.rest-bearer.v1",
|
||||
}, {
|
||||
role: "publisher",
|
||||
owner: "ndc_l2_credentials",
|
||||
management: "control_plane_managed",
|
||||
nodeType: "n8n-nodes-ndc.ndcDataProductPublish",
|
||||
}],
|
||||
steps: [{
|
||||
id: "collection.trigger",
|
||||
kind: "collection_trigger",
|
||||
collectionProfileDriven: true,
|
||||
}, {
|
||||
id: "provider.fetch",
|
||||
kind: "provider_request",
|
||||
capabilityId: "gelios.units.current.read",
|
||||
}, {
|
||||
id: "provider.extract-items",
|
||||
kind: "extract_items",
|
||||
capabilityId: "gelios.units.current.read",
|
||||
}, {
|
||||
id: "ontology.map",
|
||||
kind: "semantic_mapping",
|
||||
mappingContractId: "gelios.units.to.fleet.positions.current.v1",
|
||||
}, {
|
||||
id: "data-product.publish",
|
||||
kind: "data_product_publish",
|
||||
dataProductId: GELIOS_POSITIONS_DATA_PRODUCT_ID,
|
||||
nodeType: "n8n-nodes-ndc.ndcDataProductPublish",
|
||||
}],
|
||||
invariants: {
|
||||
oneConnectionPerInstance: true,
|
||||
providerSecretByReferenceOnly: true,
|
||||
publisherBindingControlPlaneManaged: true,
|
||||
callerScopeForbidden: true,
|
||||
providerSpecificServiceForbidden: true,
|
||||
},
|
||||
}],
|
||||
});
|
||||
|
||||
function deepFreeze(value) {
|
||||
if (!value || typeof value !== "object" || Object.isFrozen(value)) return value;
|
||||
Object.freeze(value);
|
||||
for (const child of Object.values(value)) deepFreeze(child);
|
||||
return value;
|
||||
}
|
||||
|
|
@ -0,0 +1 @@
|
|||
export const EXTERNAL_PROVIDER_CONTRACT_VERSION = "nodedc.external-provider-contract/v1";
|
||||
|
|
@ -0,0 +1,10 @@
|
|||
export { EXTERNAL_PROVIDER_CONTRACT_VERSION } from "./contract-version.mjs";
|
||||
export { validateIntakeBatch } from "./intake-batch.mjs";
|
||||
export {
|
||||
DATA_PRODUCT_PATCH_SCHEMA_VERSION,
|
||||
DATA_PRODUCT_PUBLISH_SCHEMA_VERSION,
|
||||
DATA_PRODUCT_SNAPSHOT_SCHEMA_VERSION,
|
||||
validateDataProductPatch,
|
||||
validateDataProductPublish,
|
||||
validateDataProductSnapshot,
|
||||
} from "./data-product.mjs";
|
||||
|
|
@ -2,11 +2,11 @@ export const DATA_PRODUCT_PUBLISH_SCHEMA_VERSION = "nodedc.data-product.publish/
|
|||
export const DATA_PRODUCT_SNAPSHOT_SCHEMA_VERSION = "nodedc.data-product.snapshot/v1";
|
||||
export const DATA_PRODUCT_PATCH_SCHEMA_VERSION = "nodedc.data-product.patch/v1";
|
||||
|
||||
import { SECRET_LIKE_KEY, SECRET_LIKE_VALUE } from "./sensitive-field-policy.mjs";
|
||||
|
||||
const IDENTIFIER = /^[a-z][a-z0-9._:-]{2,127}$/;
|
||||
const SEMVER = /^\d+\.\d+\.\d+(?:[-+][a-z0-9.-]+)?$/i;
|
||||
const CURSOR = /^(?:0|[1-9]\d*)$/;
|
||||
const SECRET_LIKE_KEY = /(token|secret|password|authorization|access[_-]?token|refresh[_-]?token|api[_-]?key)/i;
|
||||
const SECRET_LIKE_VALUE = /(?:ndc_edp(?:wb|rb)_[A-Za-z0-9_-]*|[?&](?:token|secret|password|authorization|access[_-]?token|refresh[_-]?token|api[_-]?key)=|(?:bearer|basic)\s+\S+|eyJ[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+)/i;
|
||||
const MAX_BATCH_SEQUENCE = 2_147_483_647;
|
||||
const MAX_FACT_ATTRIBUTES_BYTES = 64 * 1024;
|
||||
|
||||
|
|
|
|||
|
|
@ -1,662 +0,0 @@
|
|||
import { createHash, verify as verifySignature } from "node:crypto";
|
||||
|
||||
export const ENGINE_CREDENTIAL_SINK_PROVISION_SCHEMA_VERSION = "nodedc.engine.credential-sink.provision/v1";
|
||||
export const ENGINE_CREDENTIAL_SINK_RECEIPT_SCHEMA_VERSION = "nodedc.engine.credential-sink.receipt/v1";
|
||||
export const ENGINE_CREDENTIAL_SINK_ROLLBACK_SCHEMA_VERSION = "nodedc.engine.credential-sink.rollback/v1";
|
||||
export const ENGINE_CREDENTIAL_SINK_ROLLBACK_RECEIPT_SCHEMA_VERSION = "nodedc.engine.credential-sink.rollback-receipt/v1";
|
||||
export const ENGINE_CREDENTIAL_SINK_AUDIT_SCHEMA_VERSION = "nodedc.engine.credential-sink.audit/v1";
|
||||
|
||||
const HASH = /^sha256:[a-f0-9]{64}$/;
|
||||
const IDENTIFIER = /^[a-z][a-z0-9._:-]{2,127}$/;
|
||||
const OPAQUE_ID = /^[A-Za-z0-9][A-Za-z0-9._:-]{2,159}$/;
|
||||
const NODE_TYPE = /^[a-z][A-Za-z0-9.-]{2,159}$/;
|
||||
const CREDENTIAL_TYPE = /^[a-z][A-Za-z0-9]{2,127}$/;
|
||||
const CREDENTIAL_REFERENCE = /^[A-Za-z0-9][A-Za-z0-9_-]{5,159}$/;
|
||||
const REASON_CODE = /^[a-z][a-z0-9_.:-]{2,127}$/;
|
||||
const ED25519_SIGNATURE = /^[A-Za-z0-9_-]{86}$/;
|
||||
const SECRET_LIKE_KEY = /(token|secret|password|authorization|access[_-]?token|refresh[_-]?token|api[_-]?key|material|value)/i;
|
||||
const SECRET_LIKE_VALUE = /(?:ndc_(?:edp(?:wb|rb)|fndbg)_[A-Za-z0-9_-]+|(?:bearer|basic)\s+\S+|eyJ[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+)/i;
|
||||
const MAX_BINDINGS = 32;
|
||||
const MAX_REQUEST_LIFETIME_MS = 15 * 60 * 1000;
|
||||
const MAX_REQUEST_CLOCK_SKEW_MS = 60 * 1000;
|
||||
|
||||
const CAPABILITY_SPECS = Object.freeze({
|
||||
"external-data-plane.writer": Object.freeze({
|
||||
nodeType: "n8n-nodes-ndc.ndcDataProductPublish",
|
||||
credentialType: "ndcDataProductWriterApi",
|
||||
materialPattern: /^ndc_edpwb_[A-Za-z0-9_-]{43}$/,
|
||||
}),
|
||||
"external-data-plane.reader": Object.freeze({
|
||||
nodeType: "n8n-nodes-ndc.ndcDataProductRead",
|
||||
credentialType: "ndcDataProductReaderApi",
|
||||
materialPattern: /^ndc_edprb_[A-Za-z0-9_-]{43}$/,
|
||||
}),
|
||||
"foundry.binding": Object.freeze({
|
||||
nodeType: "n8n-nodes-ndc.ndcFoundryBinding",
|
||||
credentialType: "ndcFoundryBindingApi",
|
||||
materialPattern: /^ndc_fndbg_[A-Za-z0-9_-]{43}$/,
|
||||
}),
|
||||
});
|
||||
|
||||
export const ENGINE_CREDENTIAL_SINK_CAPABILITY_TYPES = Object.freeze(Object.keys(CAPABILITY_SPECS));
|
||||
|
||||
const PROVISION_KEYS = new Set(["schemaVersion", "transaction", "bindings"]);
|
||||
const TRANSACTION_KEYS = new Set([
|
||||
"id",
|
||||
"idempotencyKey",
|
||||
"requestedAt",
|
||||
"requestExpiresAt",
|
||||
"policyHash",
|
||||
"failureMode",
|
||||
"issuer",
|
||||
"attestation",
|
||||
]);
|
||||
const ISSUER_KEYS = new Set(["serviceId", "keyId"]);
|
||||
const ATTESTATION_KEYS = new Set(["algorithm", "signature"]);
|
||||
const BINDING_KEYS = new Set([
|
||||
"bindingId",
|
||||
"capabilityType",
|
||||
"grantId",
|
||||
"target",
|
||||
"expiresAt",
|
||||
"policyHash",
|
||||
"capabilityDigest",
|
||||
"material",
|
||||
]);
|
||||
const TARGET_KEYS = new Set([
|
||||
"workflowId",
|
||||
"workflowRevision",
|
||||
"nodeId",
|
||||
"nodeType",
|
||||
"credentialType",
|
||||
]);
|
||||
const MATERIAL_KEYS = new Set(["format", "value"]);
|
||||
const RECEIPT_KEYS = new Set([
|
||||
"schemaVersion",
|
||||
"transactionId",
|
||||
"idempotencyKey",
|
||||
"outcome",
|
||||
"policyHash",
|
||||
"processedAt",
|
||||
"credentials",
|
||||
"rollback",
|
||||
"errorCode",
|
||||
]);
|
||||
const RECEIPT_CREDENTIAL_KEYS = new Set([
|
||||
"bindingId",
|
||||
"capabilityType",
|
||||
"grantId",
|
||||
"target",
|
||||
"credentialRef",
|
||||
"expiresAt",
|
||||
"policyHash",
|
||||
"capabilityDigest",
|
||||
"disposition",
|
||||
]);
|
||||
const RECEIPT_ROLLBACK_KEYS = new Set(["status", "completedAt"]);
|
||||
const ROLLBACK_REQUEST_KEYS = new Set(["schemaVersion", "rollback"]);
|
||||
const ROLLBACK_KEYS = new Set([
|
||||
"id",
|
||||
"idempotencyKey",
|
||||
"transactionId",
|
||||
"requestedAt",
|
||||
"requestExpiresAt",
|
||||
"policyHash",
|
||||
"committedReceiptHash",
|
||||
"reasonCode",
|
||||
]);
|
||||
const ROLLBACK_RECEIPT_KEYS = new Set([
|
||||
"schemaVersion",
|
||||
"rollbackId",
|
||||
"transactionId",
|
||||
"outcome",
|
||||
"policyHash",
|
||||
"committedReceiptHash",
|
||||
"processedAt",
|
||||
"errorCode",
|
||||
]);
|
||||
const AUDIT_KEYS = new Set([
|
||||
"schemaVersion",
|
||||
"eventId",
|
||||
"transactionId",
|
||||
"operationId",
|
||||
"operation",
|
||||
"outcome",
|
||||
"occurredAt",
|
||||
"policyHash",
|
||||
"principal",
|
||||
"targets",
|
||||
"reasonCode",
|
||||
]);
|
||||
const PRINCIPAL_KEYS = new Set(["serviceId", "fingerprint"]);
|
||||
const AUDIT_TARGET_KEYS = new Set([
|
||||
"bindingId",
|
||||
"capabilityType",
|
||||
"grantId",
|
||||
"target",
|
||||
"expiresAt",
|
||||
"policyHash",
|
||||
"capabilityDigest",
|
||||
"credentialRefHash",
|
||||
]);
|
||||
|
||||
/**
|
||||
* Validates the only request allowed to carry plaintext workload capability
|
||||
* material across the trusted Platform -> Engine server boundary. Callers and
|
||||
* receivers must never log, trace, persist or return this request body.
|
||||
*/
|
||||
export function validateEngineCredentialSinkProvision(value, { now = Date.now(), issuerPublicKeys } = {}) {
|
||||
const errors = [];
|
||||
const nowMs = normalizeNow(now, errors);
|
||||
if (!isPlainObject(value)) return result(["credentialSinkProvision_must_be_object"]);
|
||||
if (value.schemaVersion !== ENGINE_CREDENTIAL_SINK_PROVISION_SCHEMA_VERSION) errors.push("schemaVersion_mismatch");
|
||||
rejectUnknownKeys(value, PROVISION_KEYS, "credentialSinkProvision", errors);
|
||||
|
||||
if (!isPlainObject(value.transaction)) {
|
||||
errors.push("transaction_must_be_object");
|
||||
} else {
|
||||
rejectUnknownKeys(value.transaction, TRANSACTION_KEYS, "transaction", errors);
|
||||
requiredOpaqueId(value.transaction.id, "transaction.id", errors);
|
||||
requiredOpaqueId(value.transaction.idempotencyKey, "transaction.idempotencyKey", errors);
|
||||
requiredTimestamp(value.transaction.requestedAt, "transaction.requestedAt", errors);
|
||||
requiredTimestamp(value.transaction.requestExpiresAt, "transaction.requestExpiresAt", errors);
|
||||
requiredHash(value.transaction.policyHash, "transaction.policyHash", errors);
|
||||
if (value.transaction.failureMode !== "rollback-all") errors.push("transaction.failureMode_must_be_rollback-all");
|
||||
validateIssuer(value.transaction.issuer, "transaction.issuer", errors);
|
||||
validateAttestation(value.transaction.attestation, "transaction.attestation", errors);
|
||||
validateRequestWindow(value.transaction.requestedAt, value.transaction.requestExpiresAt, nowMs, errors);
|
||||
}
|
||||
|
||||
if (!Array.isArray(value.bindings) || value.bindings.length === 0) {
|
||||
errors.push("bindings_must_be_nonempty_array");
|
||||
} else if (value.bindings.length > MAX_BINDINGS) {
|
||||
errors.push("bindings_limit_exceeded");
|
||||
} else {
|
||||
const bindingIds = new Set();
|
||||
const targets = new Set();
|
||||
value.bindings.forEach((binding, index) => {
|
||||
validateProvisionBinding(binding, index, value.transaction?.requestedAt, errors);
|
||||
if (!isPlainObject(binding)) return;
|
||||
if (bindingIds.has(binding.bindingId)) errors.push("bindings_bindingId_must_be_unique");
|
||||
bindingIds.add(binding.bindingId);
|
||||
const targetKey = targetIdentity(binding.target);
|
||||
if (targetKey && targets.has(targetKey)) errors.push("bindings_target_credential_must_be_unique");
|
||||
if (targetKey) targets.add(targetKey);
|
||||
});
|
||||
}
|
||||
|
||||
if (isPlainObject(value.transaction) && HASH.test(String(value.transaction.policyHash || ""))) {
|
||||
const expectedHash = computeEngineCredentialSinkPolicyHash(value);
|
||||
if (value.transaction.policyHash !== expectedHash) errors.push("transaction.policyHash_mismatch");
|
||||
verifyProvisionAttestation(value.transaction, issuerPublicKeys, errors);
|
||||
}
|
||||
return result(errors);
|
||||
}
|
||||
|
||||
/**
|
||||
* Computes the aggregate policy digest over the complete secret-free request
|
||||
* descriptor. Plaintext `material` is excluded by construction, while its
|
||||
* high-entropy capability digest is included; changing a target, grant,
|
||||
* capability, expiry, individual policy hash or transaction envelope changes
|
||||
* the aggregate digest and invalidates the issuer attestation.
|
||||
*/
|
||||
export function computeEngineCredentialSinkPolicyHash(value) {
|
||||
const descriptor = {
|
||||
schemaVersion: value?.schemaVersion,
|
||||
transaction: isPlainObject(value?.transaction) ? {
|
||||
id: value.transaction.id,
|
||||
idempotencyKey: value.transaction.idempotencyKey,
|
||||
requestedAt: value.transaction.requestedAt,
|
||||
requestExpiresAt: value.transaction.requestExpiresAt,
|
||||
failureMode: value.transaction.failureMode,
|
||||
issuer: value.transaction.issuer,
|
||||
} : value?.transaction,
|
||||
bindings: Array.isArray(value?.bindings)
|
||||
? value.bindings.map(secretFreeBindingDescriptor)
|
||||
: value?.bindings,
|
||||
};
|
||||
return `sha256:${createHash("sha256").update(stableJson(descriptor), "utf8").digest("hex")}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Receipt is intentionally incapable of carrying credential material. When a
|
||||
* request is supplied, the validator also proves the sink committed the exact
|
||||
* requested workflow/node/type set without target substitution.
|
||||
*/
|
||||
export function validateEngineCredentialSinkReceipt(value, { request, issuerPublicKeys } = {}) {
|
||||
const errors = [];
|
||||
if (!isPlainObject(value)) return result(["credentialSinkReceipt_must_be_object"]);
|
||||
if (value.schemaVersion !== ENGINE_CREDENTIAL_SINK_RECEIPT_SCHEMA_VERSION) errors.push("schemaVersion_mismatch");
|
||||
rejectUnknownKeys(value, RECEIPT_KEYS, "credentialSinkReceipt", errors);
|
||||
requiredOpaqueId(value.transactionId, "transactionId", errors);
|
||||
requiredOpaqueId(value.idempotencyKey, "idempotencyKey", errors);
|
||||
requiredHash(value.policyHash, "policyHash", errors);
|
||||
requiredTimestamp(value.processedAt, "processedAt", errors);
|
||||
const outcomes = new Set(["committed", "rolled-back", "rejected", "rollback-failed"]);
|
||||
if (!outcomes.has(value.outcome)) errors.push("outcome_invalid");
|
||||
|
||||
if (!Array.isArray(value.credentials)) {
|
||||
errors.push("credentials_must_be_array");
|
||||
} else {
|
||||
const bindingIds = new Set();
|
||||
const refs = new Set();
|
||||
value.credentials.forEach((credential, index) => {
|
||||
validateReceiptCredential(credential, index, errors);
|
||||
if (!isPlainObject(credential)) return;
|
||||
if (bindingIds.has(credential.bindingId)) errors.push("credentials_bindingId_must_be_unique");
|
||||
bindingIds.add(credential.bindingId);
|
||||
if (refs.has(credential.credentialRef)) errors.push("credentials_credentialRef_must_be_unique");
|
||||
refs.add(credential.credentialRef);
|
||||
});
|
||||
}
|
||||
validateReceiptOutcome(value, errors);
|
||||
if (containsSecretLikeMaterial(value)) errors.push("receipt_must_not_contain_secret_material");
|
||||
if (request !== undefined) compareReceiptToProvisionRequest(value, request, issuerPublicKeys, errors);
|
||||
return result(errors);
|
||||
}
|
||||
|
||||
export function computeEngineCredentialSinkReceiptHash(value) {
|
||||
return `sha256:${createHash("sha256").update(stableJson(value), "utf8").digest("hex")}`;
|
||||
}
|
||||
|
||||
export function validateEngineCredentialSinkRollback(value, { now = Date.now() } = {}) {
|
||||
const errors = [];
|
||||
const nowMs = normalizeNow(now, errors);
|
||||
if (!isPlainObject(value)) return result(["credentialSinkRollback_must_be_object"]);
|
||||
if (value.schemaVersion !== ENGINE_CREDENTIAL_SINK_ROLLBACK_SCHEMA_VERSION) errors.push("schemaVersion_mismatch");
|
||||
rejectUnknownKeys(value, ROLLBACK_REQUEST_KEYS, "credentialSinkRollback", errors);
|
||||
if (!isPlainObject(value.rollback)) {
|
||||
errors.push("rollback_must_be_object");
|
||||
return result(errors);
|
||||
}
|
||||
rejectUnknownKeys(value.rollback, ROLLBACK_KEYS, "rollback", errors);
|
||||
requiredOpaqueId(value.rollback.id, "rollback.id", errors);
|
||||
requiredOpaqueId(value.rollback.idempotencyKey, "rollback.idempotencyKey", errors);
|
||||
requiredOpaqueId(value.rollback.transactionId, "rollback.transactionId", errors);
|
||||
requiredTimestamp(value.rollback.requestedAt, "rollback.requestedAt", errors);
|
||||
requiredTimestamp(value.rollback.requestExpiresAt, "rollback.requestExpiresAt", errors);
|
||||
requiredHash(value.rollback.policyHash, "rollback.policyHash", errors);
|
||||
requiredHash(value.rollback.committedReceiptHash, "rollback.committedReceiptHash", errors);
|
||||
requiredReason(value.rollback.reasonCode, "rollback.reasonCode", errors);
|
||||
validateRequestWindow(value.rollback.requestedAt, value.rollback.requestExpiresAt, nowMs, errors);
|
||||
if (containsSecretLikeMaterial(value)) errors.push("rollback_must_not_contain_secret_material");
|
||||
return result(errors);
|
||||
}
|
||||
|
||||
export function validateEngineCredentialSinkRollbackReceipt(value, { request } = {}) {
|
||||
const errors = [];
|
||||
if (!isPlainObject(value)) return result(["credentialSinkRollbackReceipt_must_be_object"]);
|
||||
if (value.schemaVersion !== ENGINE_CREDENTIAL_SINK_ROLLBACK_RECEIPT_SCHEMA_VERSION) errors.push("schemaVersion_mismatch");
|
||||
rejectUnknownKeys(value, ROLLBACK_RECEIPT_KEYS, "credentialSinkRollbackReceipt", errors);
|
||||
requiredOpaqueId(value.rollbackId, "rollbackId", errors);
|
||||
requiredOpaqueId(value.transactionId, "transactionId", errors);
|
||||
requiredHash(value.policyHash, "policyHash", errors);
|
||||
requiredHash(value.committedReceiptHash, "committedReceiptHash", errors);
|
||||
requiredTimestamp(value.processedAt, "processedAt", errors);
|
||||
if (!new Set(["rolled-back", "rejected", "rollback-failed"]).has(value.outcome)) errors.push("outcome_invalid");
|
||||
if (value.outcome === "rolled-back") {
|
||||
if (value.errorCode !== undefined) errors.push("errorCode_forbidden_for_success");
|
||||
} else {
|
||||
requiredReason(value.errorCode, "errorCode", errors);
|
||||
}
|
||||
if (containsSecretLikeMaterial(value)) errors.push("rollbackReceipt_must_not_contain_secret_material");
|
||||
if (request !== undefined && isPlainObject(request?.rollback)) {
|
||||
if (!validateEngineCredentialSinkRollback(request, { now: value.processedAt }).ok) {
|
||||
errors.push("request_invalid_for_rollback_receipt_comparison");
|
||||
}
|
||||
if (value.rollbackId !== request.rollback.id) errors.push("rollbackId_request_mismatch");
|
||||
if (value.transactionId !== request.rollback.transactionId) errors.push("transactionId_request_mismatch");
|
||||
if (value.policyHash !== request.rollback.policyHash) errors.push("policyHash_request_mismatch");
|
||||
if (value.committedReceiptHash !== request.rollback.committedReceiptHash) {
|
||||
errors.push("committedReceiptHash_request_mismatch");
|
||||
}
|
||||
}
|
||||
return result(errors);
|
||||
}
|
||||
|
||||
export function validateEngineCredentialSinkAudit(value) {
|
||||
const errors = [];
|
||||
if (!isPlainObject(value)) return result(["credentialSinkAudit_must_be_object"]);
|
||||
if (value.schemaVersion !== ENGINE_CREDENTIAL_SINK_AUDIT_SCHEMA_VERSION) errors.push("schemaVersion_mismatch");
|
||||
rejectUnknownKeys(value, AUDIT_KEYS, "credentialSinkAudit", errors);
|
||||
requiredOpaqueId(value.eventId, "eventId", errors);
|
||||
requiredOpaqueId(value.transactionId, "transactionId", errors);
|
||||
requiredOpaqueId(value.operationId, "operationId", errors);
|
||||
if (!new Set(["provision", "rollback"]).has(value.operation)) errors.push("operation_invalid");
|
||||
if (!new Set(["committed", "rolled-back", "rejected", "rollback-failed"]).has(value.outcome)) errors.push("outcome_invalid");
|
||||
if (value.operation === "provision" && value.outcome === "rolled-back" && !value.reasonCode) {
|
||||
errors.push("reasonCode_required");
|
||||
}
|
||||
if (value.operation === "rollback" && value.outcome === "committed") errors.push("rollback_outcome_invalid");
|
||||
requiredTimestamp(value.occurredAt, "occurredAt", errors);
|
||||
requiredHash(value.policyHash, "policyHash", errors);
|
||||
if (!isPlainObject(value.principal)) {
|
||||
errors.push("principal_must_be_object");
|
||||
} else {
|
||||
rejectUnknownKeys(value.principal, PRINCIPAL_KEYS, "principal", errors);
|
||||
requiredIdentifier(value.principal.serviceId, "principal.serviceId", errors);
|
||||
requiredHash(value.principal.fingerprint, "principal.fingerprint", errors);
|
||||
}
|
||||
if (!Array.isArray(value.targets)) {
|
||||
errors.push("targets_must_be_array");
|
||||
} else {
|
||||
value.targets.forEach((target, index) => validateAuditTarget(target, index, errors));
|
||||
}
|
||||
if (value.reasonCode !== undefined) requiredReason(value.reasonCode, "reasonCode", errors);
|
||||
if (new Set(["rejected", "rollback-failed"]).has(value.outcome) && value.reasonCode === undefined) {
|
||||
errors.push("reasonCode_required");
|
||||
}
|
||||
if (containsSecretLikeMaterial(value)) errors.push("audit_must_not_contain_secret_material");
|
||||
return result(errors);
|
||||
}
|
||||
|
||||
/** Returns audit-safe target descriptors; material and credential refs cannot escape. */
|
||||
export function engineCredentialSinkAuditTargets(request, receipt) {
|
||||
const refByBinding = new Map(
|
||||
Array.isArray(receipt?.credentials)
|
||||
? receipt.credentials.map((item) => [item.bindingId, item.credentialRef])
|
||||
: [],
|
||||
);
|
||||
return Array.isArray(request?.bindings) ? request.bindings.map((binding) => {
|
||||
const descriptor = secretFreeBindingDescriptor(binding);
|
||||
const credentialRef = refByBinding.get(binding.bindingId);
|
||||
return credentialRef
|
||||
? { ...descriptor, credentialRefHash: sha256Value(credentialRef) }
|
||||
: descriptor;
|
||||
}) : [];
|
||||
}
|
||||
|
||||
function validateProvisionBinding(value, index, requestedAt, errors) {
|
||||
const path = `bindings[${index}]`;
|
||||
if (!isPlainObject(value)) {
|
||||
errors.push(`${path}_must_be_object`);
|
||||
return;
|
||||
}
|
||||
rejectUnknownKeys(value, BINDING_KEYS, path, errors);
|
||||
requiredIdentifier(value.bindingId, `${path}.bindingId`, errors);
|
||||
const spec = CAPABILITY_SPECS[value.capabilityType];
|
||||
if (!spec) errors.push(`${path}.capabilityType_invalid`);
|
||||
requiredOpaqueId(value.grantId, `${path}.grantId`, errors);
|
||||
validateTarget(value.target, path, errors);
|
||||
requiredTimestamp(value.expiresAt, `${path}.expiresAt`, errors);
|
||||
requiredHash(value.policyHash, `${path}.policyHash`, errors);
|
||||
requiredHash(value.capabilityDigest, `${path}.capabilityDigest`, errors);
|
||||
if (isTimestamp(requestedAt) && isTimestamp(value.expiresAt) && Date.parse(value.expiresAt) <= Date.parse(requestedAt)) {
|
||||
errors.push(`${path}.expiresAt_must_be_after_requestedAt`);
|
||||
}
|
||||
if (spec && isPlainObject(value.target)) {
|
||||
if (value.target.nodeType !== spec.nodeType) errors.push(`${path}.target.nodeType_capability_mismatch`);
|
||||
if (value.target.credentialType !== spec.credentialType) errors.push(`${path}.target.credentialType_capability_mismatch`);
|
||||
}
|
||||
if (!isPlainObject(value.material)) {
|
||||
errors.push(`${path}.material_must_be_object`);
|
||||
} else {
|
||||
rejectUnknownKeys(value.material, MATERIAL_KEYS, `${path}.material`, errors);
|
||||
if (value.material.format !== "opaque-bearer") errors.push(`${path}.material.format_must_be_opaque-bearer`);
|
||||
if (!spec || typeof value.material.value !== "string" || !spec.materialPattern.test(value.material.value)) {
|
||||
errors.push(`${path}.material.value_invalid_for_capability`);
|
||||
} else if (value.capabilityDigest !== computeEngineCredentialCapabilityDigest(value.material.value)) {
|
||||
errors.push(`${path}.capabilityDigest_material_mismatch`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function validateTarget(value, path, errors) {
|
||||
if (!isPlainObject(value)) {
|
||||
errors.push(`${path}.target_must_be_object`);
|
||||
return;
|
||||
}
|
||||
rejectUnknownKeys(value, TARGET_KEYS, `${path}.target`, errors);
|
||||
requiredOpaqueId(value.workflowId, `${path}.target.workflowId`, errors);
|
||||
requiredOpaqueId(value.workflowRevision, `${path}.target.workflowRevision`, errors);
|
||||
requiredOpaqueId(value.nodeId, `${path}.target.nodeId`, errors);
|
||||
if (typeof value.nodeType !== "string" || !NODE_TYPE.test(value.nodeType)) errors.push(`${path}.target.nodeType_invalid`);
|
||||
if (typeof value.credentialType !== "string" || !CREDENTIAL_TYPE.test(value.credentialType)) {
|
||||
errors.push(`${path}.target.credentialType_invalid`);
|
||||
}
|
||||
}
|
||||
|
||||
function validateReceiptCredential(value, index, errors) {
|
||||
const path = `credentials[${index}]`;
|
||||
if (!isPlainObject(value)) {
|
||||
errors.push(`${path}_must_be_object`);
|
||||
return;
|
||||
}
|
||||
rejectUnknownKeys(value, RECEIPT_CREDENTIAL_KEYS, path, errors);
|
||||
requiredIdentifier(value.bindingId, `${path}.bindingId`, errors);
|
||||
if (!CAPABILITY_SPECS[value.capabilityType]) errors.push(`${path}.capabilityType_invalid`);
|
||||
requiredOpaqueId(value.grantId, `${path}.grantId`, errors);
|
||||
validateTarget(value.target, path, errors);
|
||||
if (typeof value.credentialRef !== "string" || !CREDENTIAL_REFERENCE.test(value.credentialRef)) {
|
||||
errors.push(`${path}.credentialRef_invalid`);
|
||||
}
|
||||
requiredTimestamp(value.expiresAt, `${path}.expiresAt`, errors);
|
||||
requiredHash(value.policyHash, `${path}.policyHash`, errors);
|
||||
requiredHash(value.capabilityDigest, `${path}.capabilityDigest`, errors);
|
||||
if (!new Set(["created", "reused", "rotated"]).has(value.disposition)) errors.push(`${path}.disposition_invalid`);
|
||||
}
|
||||
|
||||
function validateReceiptOutcome(value, errors) {
|
||||
if (!isPlainObject(value.rollback)) {
|
||||
errors.push("rollback_must_be_object");
|
||||
return;
|
||||
}
|
||||
rejectUnknownKeys(value.rollback, RECEIPT_ROLLBACK_KEYS, "rollback", errors);
|
||||
const expectedRollback = {
|
||||
committed: "not-required",
|
||||
"rolled-back": "complete",
|
||||
rejected: "not-started",
|
||||
"rollback-failed": "incomplete",
|
||||
}[value.outcome];
|
||||
if (expectedRollback && value.rollback.status !== expectedRollback) errors.push("rollback.status_outcome_mismatch");
|
||||
if (new Set(["complete", "incomplete"]).has(value.rollback.status)) {
|
||||
requiredTimestamp(value.rollback.completedAt, "rollback.completedAt", errors);
|
||||
} else if (value.rollback.completedAt !== undefined) {
|
||||
errors.push("rollback.completedAt_not_allowed");
|
||||
}
|
||||
if (value.outcome === "committed") {
|
||||
if (!Array.isArray(value.credentials) || value.credentials.length === 0) errors.push("committed_credentials_required");
|
||||
if (value.errorCode !== undefined) errors.push("errorCode_forbidden_for_success");
|
||||
} else {
|
||||
if (Array.isArray(value.credentials) && value.credentials.length !== 0) errors.push("noncommitted_credentials_must_be_empty");
|
||||
requiredReason(value.errorCode, "errorCode", errors);
|
||||
}
|
||||
}
|
||||
|
||||
function compareReceiptToProvisionRequest(receipt, request, issuerPublicKeys, errors) {
|
||||
const requestValidation = validateEngineCredentialSinkProvision(request, {
|
||||
now: receipt.processedAt,
|
||||
issuerPublicKeys,
|
||||
});
|
||||
if (!requestValidation.ok) {
|
||||
errors.push("request_invalid_for_receipt_comparison");
|
||||
return;
|
||||
}
|
||||
if (receipt.transactionId !== request.transaction.id) errors.push("transactionId_request_mismatch");
|
||||
if (receipt.idempotencyKey !== request.transaction.idempotencyKey) errors.push("idempotencyKey_request_mismatch");
|
||||
if (receipt.policyHash !== request.transaction.policyHash) errors.push("policyHash_request_mismatch");
|
||||
if (receipt.outcome !== "committed") return;
|
||||
if (receipt.credentials.length !== request.bindings.length) errors.push("credentials_request_count_mismatch");
|
||||
const requested = new Map(request.bindings.map((binding) => [binding.bindingId, secretFreeBindingDescriptor(binding)]));
|
||||
for (const credential of receipt.credentials) {
|
||||
const expected = requested.get(credential.bindingId);
|
||||
if (!expected || stableJson({
|
||||
bindingId: credential.bindingId,
|
||||
capabilityType: credential.capabilityType,
|
||||
grantId: credential.grantId,
|
||||
target: credential.target,
|
||||
expiresAt: credential.expiresAt,
|
||||
policyHash: credential.policyHash,
|
||||
capabilityDigest: credential.capabilityDigest,
|
||||
}) !== stableJson(expected)) {
|
||||
errors.push("credentials_request_target_mismatch");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function validateAuditTarget(value, index, errors) {
|
||||
const path = `targets[${index}]`;
|
||||
if (!isPlainObject(value)) {
|
||||
errors.push(`${path}_must_be_object`);
|
||||
return;
|
||||
}
|
||||
rejectUnknownKeys(value, AUDIT_TARGET_KEYS, path, errors);
|
||||
requiredIdentifier(value.bindingId, `${path}.bindingId`, errors);
|
||||
if (!CAPABILITY_SPECS[value.capabilityType]) errors.push(`${path}.capabilityType_invalid`);
|
||||
requiredOpaqueId(value.grantId, `${path}.grantId`, errors);
|
||||
validateTarget(value.target, path, errors);
|
||||
requiredTimestamp(value.expiresAt, `${path}.expiresAt`, errors);
|
||||
requiredHash(value.policyHash, `${path}.policyHash`, errors);
|
||||
requiredHash(value.capabilityDigest, `${path}.capabilityDigest`, errors);
|
||||
if (value.credentialRefHash !== undefined) requiredHash(value.credentialRefHash, `${path}.credentialRefHash`, errors);
|
||||
}
|
||||
|
||||
function secretFreeBindingDescriptor(value) {
|
||||
return {
|
||||
bindingId: value?.bindingId,
|
||||
capabilityType: value?.capabilityType,
|
||||
grantId: value?.grantId,
|
||||
target: value?.target,
|
||||
expiresAt: value?.expiresAt,
|
||||
policyHash: value?.policyHash,
|
||||
capabilityDigest: value?.capabilityDigest,
|
||||
};
|
||||
}
|
||||
|
||||
export function computeEngineCredentialCapabilityDigest(value) {
|
||||
return sha256Value(value);
|
||||
}
|
||||
|
||||
function validateIssuer(value, path, errors) {
|
||||
if (!isPlainObject(value)) {
|
||||
errors.push(`${path}_must_be_object`);
|
||||
return;
|
||||
}
|
||||
rejectUnknownKeys(value, ISSUER_KEYS, path, errors);
|
||||
requiredIdentifier(value.serviceId, `${path}.serviceId`, errors);
|
||||
requiredOpaqueId(value.keyId, `${path}.keyId`, errors);
|
||||
}
|
||||
|
||||
function validateAttestation(value, path, errors) {
|
||||
if (!isPlainObject(value)) {
|
||||
errors.push(`${path}_must_be_object`);
|
||||
return;
|
||||
}
|
||||
rejectUnknownKeys(value, ATTESTATION_KEYS, path, errors);
|
||||
if (value.algorithm !== "Ed25519") errors.push(`${path}.algorithm_must_be_Ed25519`);
|
||||
if (typeof value.signature !== "string" || !ED25519_SIGNATURE.test(value.signature)) {
|
||||
errors.push(`${path}.signature_invalid`);
|
||||
}
|
||||
}
|
||||
|
||||
function verifyProvisionAttestation(transaction, issuerPublicKeys, errors) {
|
||||
if (!isPlainObject(transaction?.issuer) || !isPlainObject(transaction?.attestation)) return;
|
||||
if (transaction.attestation.algorithm !== "Ed25519" || !ED25519_SIGNATURE.test(String(transaction.attestation.signature || ""))) return;
|
||||
const keyIdentity = `${transaction.issuer.serviceId}:${transaction.issuer.keyId}`;
|
||||
const publicKey = isPlainObject(issuerPublicKeys) && Object.hasOwn(issuerPublicKeys, keyIdentity)
|
||||
? issuerPublicKeys[keyIdentity]
|
||||
: undefined;
|
||||
if (!publicKey) {
|
||||
errors.push("transaction.issuer_public_key_required");
|
||||
return;
|
||||
}
|
||||
let valid = false;
|
||||
try {
|
||||
valid = verifySignature(
|
||||
null,
|
||||
Buffer.from(String(transaction.policyHash), "utf8"),
|
||||
publicKey,
|
||||
Buffer.from(transaction.attestation.signature, "base64url"),
|
||||
);
|
||||
} catch {
|
||||
valid = false;
|
||||
}
|
||||
if (!valid) errors.push("transaction.attestation_invalid");
|
||||
}
|
||||
|
||||
function validateRequestWindow(requestedAt, requestExpiresAt, nowMs, errors) {
|
||||
if (!isTimestamp(requestedAt) || !isTimestamp(requestExpiresAt)) return;
|
||||
const requested = Date.parse(requestedAt);
|
||||
const expires = Date.parse(requestExpiresAt);
|
||||
if (expires <= requested) errors.push("requestExpiresAt_must_be_after_requestedAt");
|
||||
if (expires - requested > MAX_REQUEST_LIFETIME_MS) errors.push("request_lifetime_exceeds_15_minutes");
|
||||
if (Number.isFinite(nowMs)) {
|
||||
if (requested > nowMs + MAX_REQUEST_CLOCK_SKEW_MS) errors.push("requestedAt_exceeds_clock_skew");
|
||||
if (expires <= nowMs) errors.push("request_expired");
|
||||
}
|
||||
}
|
||||
|
||||
function normalizeNow(value, errors) {
|
||||
const normalized = value instanceof Date ? value.getTime() : typeof value === "string" ? Date.parse(value) : Number(value);
|
||||
if (!Number.isFinite(normalized)) {
|
||||
errors.push("validation_now_invalid");
|
||||
return Number.NaN;
|
||||
}
|
||||
return normalized;
|
||||
}
|
||||
|
||||
function targetIdentity(value) {
|
||||
if (!isPlainObject(value)) return "";
|
||||
return [value.workflowId, value.workflowRevision, value.nodeId, value.nodeType, value.credentialType].join("\u0000");
|
||||
}
|
||||
|
||||
function requiredIdentifier(value, path, errors) {
|
||||
if (typeof value !== "string" || !IDENTIFIER.test(value)) errors.push(`${path}_invalid`);
|
||||
}
|
||||
|
||||
function requiredOpaqueId(value, path, errors) {
|
||||
if (typeof value !== "string" || !OPAQUE_ID.test(value)) errors.push(`${path}_invalid`);
|
||||
}
|
||||
|
||||
function requiredHash(value, path, errors) {
|
||||
if (typeof value !== "string" || !HASH.test(value)) errors.push(`${path}_invalid`);
|
||||
}
|
||||
|
||||
function requiredReason(value, path, errors) {
|
||||
if (typeof value !== "string" || !REASON_CODE.test(value)) errors.push(`${path}_invalid`);
|
||||
}
|
||||
|
||||
function requiredTimestamp(value, path, errors) {
|
||||
if (!isTimestamp(value)) errors.push(`${path}_invalid_timestamp`);
|
||||
}
|
||||
|
||||
function isTimestamp(value) {
|
||||
return typeof value === "string" && !Number.isNaN(Date.parse(value)) && new Date(value).toISOString() === value;
|
||||
}
|
||||
|
||||
function sha256Value(value) {
|
||||
return `sha256:${createHash("sha256").update(String(value), "utf8").digest("hex")}`;
|
||||
}
|
||||
|
||||
function stableJson(value) {
|
||||
return JSON.stringify(sortValue(value));
|
||||
}
|
||||
|
||||
function sortValue(value) {
|
||||
if (Array.isArray(value)) return value.map(sortValue);
|
||||
if (!isPlainObject(value)) return value;
|
||||
return Object.fromEntries(Object.keys(value).sort().map((key) => [key, sortValue(value[key])]));
|
||||
}
|
||||
|
||||
function containsSecretLikeMaterial(value) {
|
||||
if (typeof value === "string") return SECRET_LIKE_VALUE.test(value);
|
||||
if (Array.isArray(value)) return value.some(containsSecretLikeMaterial);
|
||||
if (!isPlainObject(value)) return false;
|
||||
return Object.entries(value).some(([key, child]) => SECRET_LIKE_KEY.test(key) || containsSecretLikeMaterial(child));
|
||||
}
|
||||
|
||||
function isPlainObject(value) {
|
||||
return Boolean(value) && typeof value === "object" && !Array.isArray(value);
|
||||
}
|
||||
|
||||
function rejectUnknownKeys(value, allowed, path, errors) {
|
||||
if (!isPlainObject(value)) return;
|
||||
for (const key of Object.keys(value)) {
|
||||
if (!allowed.has(key)) errors.push(`${path}.${key}_not_allowed`);
|
||||
}
|
||||
}
|
||||
|
||||
function result(errors) {
|
||||
const uniqueErrors = [...new Set(errors)];
|
||||
return Object.freeze({ ok: uniqueErrors.length === 0, errors: Object.freeze(uniqueErrors) });
|
||||
}
|
||||
|
|
@ -1,6 +1,23 @@
|
|||
export const EXTERNAL_PROVIDER_CONTRACT_VERSION = "nodedc.external-provider-contract/v1";
|
||||
import { EXTERNAL_PROVIDER_CONTRACT_VERSION } from "./contract-version.mjs";
|
||||
|
||||
export { EXTERNAL_PROVIDER_CONTRACT_VERSION } from "./contract-version.mjs";
|
||||
export { validateIntakeBatch } from "./intake-batch.mjs";
|
||||
export const FOUNDRY_BINDING_UPSERT_SCHEMA_VERSION = "nodedc.foundry.binding-upsert/v1";
|
||||
|
||||
import {
|
||||
SECRET_LIKE_KEY,
|
||||
SECRET_LIKE_VALUE,
|
||||
} from "./sensitive-field-policy.mjs";
|
||||
|
||||
export {
|
||||
L2_CONNECTION_INSTANCE_SCHEMA_VERSION,
|
||||
L2_TEMPLATE_SCHEMA_VERSION,
|
||||
PROVIDER_PACKAGE_SCHEMA_VERSION,
|
||||
SEMANTIC_MAPPING_SCHEMA_VERSION,
|
||||
instantiateL2Connection,
|
||||
validateProviderPackage,
|
||||
} from "./provider-package.mjs";
|
||||
|
||||
export {
|
||||
DATA_PRODUCT_PATCH_SCHEMA_VERSION,
|
||||
DATA_PRODUCT_PUBLISH_SCHEMA_VERSION,
|
||||
|
|
@ -10,24 +27,6 @@ export {
|
|||
validateDataProductSnapshot,
|
||||
} from "./data-product.mjs";
|
||||
|
||||
export {
|
||||
ENGINE_CREDENTIAL_SINK_AUDIT_SCHEMA_VERSION,
|
||||
ENGINE_CREDENTIAL_SINK_CAPABILITY_TYPES,
|
||||
ENGINE_CREDENTIAL_SINK_PROVISION_SCHEMA_VERSION,
|
||||
ENGINE_CREDENTIAL_SINK_RECEIPT_SCHEMA_VERSION,
|
||||
ENGINE_CREDENTIAL_SINK_ROLLBACK_RECEIPT_SCHEMA_VERSION,
|
||||
ENGINE_CREDENTIAL_SINK_ROLLBACK_SCHEMA_VERSION,
|
||||
computeEngineCredentialCapabilityDigest,
|
||||
computeEngineCredentialSinkPolicyHash,
|
||||
computeEngineCredentialSinkReceiptHash,
|
||||
engineCredentialSinkAuditTargets,
|
||||
validateEngineCredentialSinkAudit,
|
||||
validateEngineCredentialSinkProvision,
|
||||
validateEngineCredentialSinkReceipt,
|
||||
validateEngineCredentialSinkRollback,
|
||||
validateEngineCredentialSinkRollbackReceipt,
|
||||
} from "./engine-credential-sink.mjs";
|
||||
|
||||
export {
|
||||
ENGINE_PRIVATE_EXTENSION_APPLY_RECEIPT_SCHEMA_VERSION,
|
||||
ENGINE_PRIVATE_EXTENSION_APPLY_REQUEST_SCHEMA_VERSION,
|
||||
|
|
@ -55,16 +54,11 @@ export {
|
|||
const COLLECTION_MODES = new Set(["realtime", "manual", "weekly", "history"]);
|
||||
const DELIVERY_MODES = new Set(["snapshot", "snapshot+patch", "query"]);
|
||||
const CAPABILITY_CLASSIFICATIONS = new Set(["read", "metadata", "write", "destructive", "unknown"]);
|
||||
const SECRET_LIKE_KEY = /(token|secret|password|authorization|access[_-]?token|refresh[_-]?token|api[_-]?key)/i;
|
||||
const SECRET_LIKE_REFERENCE = /(?:[?&](?:token|secret|password|authorization|access[_-]?token|refresh[_-]?token|api[_-]?key)=|(?:bearer|basic)\s+)/i;
|
||||
const SECRET_LIKE_VALUE = /(?:ndc_edp(?:wb|rb)_[A-Za-z0-9_-]*|[?&](?:token|secret|password|authorization|access[_-]?token|refresh[_-]?token|api[_-]?key)=|(?:bearer|basic)\s+\S+|eyJ[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+)/i;
|
||||
const IDENTIFIER = /^[a-z][a-z0-9._:-]{2,127}$/;
|
||||
const FOUNDRY_APPLICATION_ID = /^[0-9a-f]{8}-[0-9a-f]{4}-[1-5][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/i;
|
||||
const FOUNDRY_PAGE_ID = /^[a-z0-9][a-z0-9-]{0,79}$/;
|
||||
const FOUNDRY_SLOT_ID = /^[A-Za-z0-9][A-Za-z0-9-]{0,79}$/;
|
||||
const CONTRACT_VERSION = /^\d+\.\d+\.\d+(?:[-+][a-z0-9.-]+)?$/i;
|
||||
const MAX_BATCH_SEQUENCE = 2_147_483_647;
|
||||
const MAX_FACT_ATTRIBUTES_BYTES = 64 * 1024;
|
||||
|
||||
const PROVIDER_MANIFEST_KEYS = new Set(["schemaVersion", "id", "providerId", "version", "ontology", "l2Template", "capabilities", "dataProductIds"]);
|
||||
const CONNECTION_PROFILE_KEYS = new Set(["schemaVersion", "id", "providerId", "tenantId", "credentialRef", "scope"]);
|
||||
|
|
@ -159,8 +153,8 @@ export function validateConnectionProfile(value) {
|
|||
requiredString(value?.credentialRef?.owner, "credentialRef.owner", errors);
|
||||
requiredString(value?.credentialRef?.reference, "credentialRef.reference", errors);
|
||||
|
||||
if (value?.credentialRef?.owner && value.credentialRef.owner !== "engine") {
|
||||
errors.push("credentialRef.owner_must_be_engine");
|
||||
if (value?.credentialRef?.owner && value.credentialRef.owner !== "ndc_l2_credentials") {
|
||||
errors.push("credentialRef.owner_must_be_ndc_l2_credentials");
|
||||
}
|
||||
if (containsSecretLikeMaterial(value)) errors.push("profile_must_not_contain_secret_material");
|
||||
if (value?.scope !== undefined) {
|
||||
|
|
@ -301,45 +295,6 @@ export function validateFoundryBindingUpsert(value) {
|
|||
return result(errors);
|
||||
}
|
||||
|
||||
/**
|
||||
* Provider-neutral batch written by an L2 connector to External Data Plane.
|
||||
* The payload deliberately describes source facts rather than any provider
|
||||
* field names, customer entities or renderer representation.
|
||||
*/
|
||||
export function validateIntakeBatch(value) {
|
||||
const errors = baseErrors(value, "intakeBatch");
|
||||
rejectUnknownKeys(value, new Set(["schemaVersion", "source", "contract", "batch", "raw", "facts"]), "intakeBatch", errors);
|
||||
rejectUnknownKeys(value?.source, new Set(["providerId", "tenantId", "connectionId"]), "source", errors);
|
||||
rejectUnknownKeys(value?.contract, new Set(["dataProductId", "ontologyRevision", "version"]), "contract", errors);
|
||||
rejectUnknownKeys(value?.batch, new Set(["runId", "sequence", "idempotencyKey", "receivedAt"]), "batch", errors);
|
||||
requiredIdentifier(value?.source?.providerId, "source.providerId", errors);
|
||||
requiredIdentifier(value?.source?.tenantId, "source.tenantId", errors);
|
||||
requiredIdentifier(value?.source?.connectionId, "source.connectionId", errors);
|
||||
requiredIdentifier(value?.contract?.dataProductId, "contract.dataProductId", errors);
|
||||
requiredIdentifier(value?.contract?.ontologyRevision, "contract.ontologyRevision", errors);
|
||||
requiredString(value?.contract?.version, "contract.version", errors);
|
||||
if (value?.contract?.version && !CONTRACT_VERSION.test(value.contract.version)) {
|
||||
errors.push("contract.version_must_be_semver");
|
||||
}
|
||||
requiredIdentifier(value?.batch?.runId, "batch.runId", errors);
|
||||
requiredIdentifier(value?.batch?.idempotencyKey, "batch.idempotencyKey", errors);
|
||||
if (!Number.isInteger(value?.batch?.sequence) || value.batch.sequence < 0 || value.batch.sequence > MAX_BATCH_SEQUENCE) {
|
||||
errors.push("batch.sequence_must_be_integer_0_to_2147483647");
|
||||
}
|
||||
requiredIsoTimestamp(value?.batch?.receivedAt, "batch.receivedAt", errors);
|
||||
|
||||
if (!Array.isArray(value?.facts) || value.facts.length === 0) {
|
||||
errors.push("facts_must_be_nonempty_array");
|
||||
} else {
|
||||
value.facts.forEach((fact, index) => validateFact(fact, `facts[${index}]`, errors));
|
||||
}
|
||||
|
||||
if (value?.raw !== undefined) validateRawEnvelope(value.raw, errors);
|
||||
if (containsSecretLikeMaterial(value)) errors.push("intake_must_not_contain_secret_material");
|
||||
|
||||
return result(errors);
|
||||
}
|
||||
|
||||
export function assertValid(validator, value) {
|
||||
const validation = validator(value);
|
||||
if (!validation.ok) throw new Error(`external_provider_contract_invalid:${validation.errors.join(",")}`);
|
||||
|
|
@ -394,66 +349,6 @@ function validateUniqueIdentifierArray(value, path, errors) {
|
|||
if (new Set(value).size !== value.length) errors.push(`${path}_must_not_contain_duplicates`);
|
||||
}
|
||||
|
||||
function requiredIsoTimestamp(value, path, errors) {
|
||||
if (typeof value !== "string" || Number.isNaN(Date.parse(value))) errors.push(`${path}_invalid_timestamp`);
|
||||
}
|
||||
|
||||
function validateFact(value, path, errors) {
|
||||
if (!isPlainObject(value)) {
|
||||
errors.push(`${path}_must_be_object`);
|
||||
return;
|
||||
}
|
||||
rejectUnknownKeys(value, new Set(["sourceId", "semanticType", "observedAt", "attributes", "geometry"]), path, errors);
|
||||
requiredIdentifier(value.sourceId, `${path}.sourceId`, errors);
|
||||
requiredIdentifier(value.semanticType, `${path}.semanticType`, errors);
|
||||
requiredIsoTimestamp(value.observedAt, `${path}.observedAt`, errors);
|
||||
if (value.attributes !== undefined) {
|
||||
if (!isPlainObject(value.attributes)) {
|
||||
errors.push(`${path}.attributes_must_be_object`);
|
||||
} else if (serializedByteLength(value.attributes) > MAX_FACT_ATTRIBUTES_BYTES) {
|
||||
errors.push(`${path}.attributes_size_exceeded`);
|
||||
}
|
||||
}
|
||||
if (value.geometry !== undefined) validatePointGeometry(value.geometry, `${path}.geometry`, errors);
|
||||
}
|
||||
|
||||
function validatePointGeometry(value, path, errors) {
|
||||
if (!isPlainObject(value) || value.type !== "Point" || !Array.isArray(value.coordinates) || value.coordinates.length !== 2) {
|
||||
errors.push(`${path}_must_be_geojson_point`);
|
||||
return;
|
||||
}
|
||||
rejectUnknownKeys(value, new Set(["type", "coordinates"]), path, errors);
|
||||
if (!value.coordinates.every((coordinate) => typeof coordinate === "number" && Number.isFinite(coordinate))) {
|
||||
errors.push(`${path}_coordinates_must_be_finite_numbers`);
|
||||
return;
|
||||
}
|
||||
const [longitude, latitude] = value.coordinates;
|
||||
if (longitude < -180 || longitude > 180) errors.push(`${path}.longitude_out_of_range`);
|
||||
if (latitude < -90 || latitude > 90) errors.push(`${path}.latitude_out_of_range`);
|
||||
}
|
||||
|
||||
function validateRawEnvelope(value, errors) {
|
||||
if (!isPlainObject(value)) {
|
||||
errors.push("raw_must_be_object");
|
||||
return;
|
||||
}
|
||||
rejectUnknownKeys(value, new Set(["contentType", "payload", "hash", "ref", "retentionDays"]), "raw", errors);
|
||||
requiredString(value.contentType, "raw.contentType", errors);
|
||||
if (value.payload === undefined && value.ref === undefined) errors.push("raw_requires_payload_or_ref");
|
||||
if (value.payload !== undefined) errors.push("raw.inline_payload_not_supported");
|
||||
if (value.payload === undefined) requiredString(value.hash, "raw.hash", errors);
|
||||
if (value.hash !== undefined) requiredString(value.hash, "raw.hash", errors);
|
||||
if (value.ref !== undefined) {
|
||||
requiredString(value.ref, "raw.ref", errors);
|
||||
if (typeof value.ref === "string" && SECRET_LIKE_REFERENCE.test(value.ref)) {
|
||||
errors.push("raw.ref_must_not_contain_secret_material");
|
||||
}
|
||||
}
|
||||
if (value.retentionDays !== undefined && (!Number.isInteger(value.retentionDays) || value.retentionDays < 1)) {
|
||||
errors.push("raw.retentionDays_must_be_positive_integer");
|
||||
}
|
||||
}
|
||||
|
||||
function isPlainObject(value) {
|
||||
return Boolean(value) && typeof value === "object" && !Array.isArray(value);
|
||||
}
|
||||
|
|
@ -465,14 +360,6 @@ function rejectUnknownKeys(value, allowedKeys, path, errors) {
|
|||
}
|
||||
}
|
||||
|
||||
function serializedByteLength(value) {
|
||||
try {
|
||||
return Buffer.byteLength(JSON.stringify(value));
|
||||
} catch {
|
||||
return Number.POSITIVE_INFINITY;
|
||||
}
|
||||
}
|
||||
|
||||
function containsSecretLikeMaterial(value) {
|
||||
if (typeof value === "string") return SECRET_LIKE_VALUE.test(value);
|
||||
if (Array.isArray(value)) return value.some(containsSecretLikeMaterial);
|
||||
|
|
|
|||
|
|
@ -0,0 +1,152 @@
|
|||
import { EXTERNAL_PROVIDER_CONTRACT_VERSION } from "./contract-version.mjs";
|
||||
import {
|
||||
SECRET_LIKE_KEY,
|
||||
SECRET_LIKE_REFERENCE,
|
||||
SECRET_LIKE_VALUE,
|
||||
} from "./sensitive-field-policy.mjs";
|
||||
|
||||
const IDENTIFIER = /^[a-z][a-z0-9._:-]{2,127}$/;
|
||||
const CONTRACT_VERSION = /^\d+\.\d+\.\d+(?:[-+][a-z0-9.-]+)?$/i;
|
||||
const MAX_BATCH_SEQUENCE = 2_147_483_647;
|
||||
const MAX_FACT_ATTRIBUTES_BYTES = 64 * 1024;
|
||||
|
||||
/** Provider-neutral batch written by an L2 connector to External Data Plane. */
|
||||
export function validateIntakeBatch(value) {
|
||||
const errors = baseErrors(value, "intakeBatch");
|
||||
rejectUnknownKeys(value, new Set(["schemaVersion", "source", "contract", "batch", "raw", "facts"]), "intakeBatch", errors);
|
||||
rejectUnknownKeys(value?.source, new Set(["providerId", "tenantId", "connectionId"]), "source", errors);
|
||||
rejectUnknownKeys(value?.contract, new Set(["dataProductId", "ontologyRevision", "version"]), "contract", errors);
|
||||
rejectUnknownKeys(value?.batch, new Set(["runId", "sequence", "idempotencyKey", "receivedAt"]), "batch", errors);
|
||||
requiredIdentifier(value?.source?.providerId, "source.providerId", errors);
|
||||
requiredIdentifier(value?.source?.tenantId, "source.tenantId", errors);
|
||||
requiredIdentifier(value?.source?.connectionId, "source.connectionId", errors);
|
||||
requiredIdentifier(value?.contract?.dataProductId, "contract.dataProductId", errors);
|
||||
requiredIdentifier(value?.contract?.ontologyRevision, "contract.ontologyRevision", errors);
|
||||
requiredString(value?.contract?.version, "contract.version", errors);
|
||||
if (value?.contract?.version && !CONTRACT_VERSION.test(value.contract.version)) {
|
||||
errors.push("contract.version_must_be_semver");
|
||||
}
|
||||
requiredIdentifier(value?.batch?.runId, "batch.runId", errors);
|
||||
requiredIdentifier(value?.batch?.idempotencyKey, "batch.idempotencyKey", errors);
|
||||
if (!Number.isInteger(value?.batch?.sequence) || value.batch.sequence < 0 || value.batch.sequence > MAX_BATCH_SEQUENCE) {
|
||||
errors.push("batch.sequence_must_be_integer_0_to_2147483647");
|
||||
}
|
||||
requiredIsoTimestamp(value?.batch?.receivedAt, "batch.receivedAt", errors);
|
||||
|
||||
if (!Array.isArray(value?.facts) || value.facts.length === 0) {
|
||||
errors.push("facts_must_be_nonempty_array");
|
||||
} else {
|
||||
value.facts.forEach((fact, index) => validateFact(fact, `facts[${index}]`, errors));
|
||||
}
|
||||
|
||||
if (value?.raw !== undefined) validateRawEnvelope(value.raw, errors);
|
||||
if (containsSecretLikeMaterial(value)) errors.push("intake_must_not_contain_secret_material");
|
||||
|
||||
return result(errors);
|
||||
}
|
||||
|
||||
function baseErrors(value, label) {
|
||||
const errors = [];
|
||||
if (!isPlainObject(value)) return [`${label}_must_be_object`];
|
||||
if (value.schemaVersion !== EXTERNAL_PROVIDER_CONTRACT_VERSION) errors.push("schemaVersion_mismatch");
|
||||
return errors;
|
||||
}
|
||||
|
||||
function requiredString(value, path, errors) {
|
||||
if (typeof value !== "string" || !value.trim()) errors.push(`${path}_required`);
|
||||
}
|
||||
|
||||
function requiredIdentifier(value, path, errors) {
|
||||
if (typeof value !== "string" || !IDENTIFIER.test(value)) errors.push(`${path}_invalid`);
|
||||
}
|
||||
|
||||
function requiredIsoTimestamp(value, path, errors) {
|
||||
if (typeof value !== "string" || Number.isNaN(Date.parse(value))) errors.push(`${path}_invalid_timestamp`);
|
||||
}
|
||||
|
||||
function validateFact(value, path, errors) {
|
||||
if (!isPlainObject(value)) {
|
||||
errors.push(`${path}_must_be_object`);
|
||||
return;
|
||||
}
|
||||
rejectUnknownKeys(value, new Set(["sourceId", "semanticType", "observedAt", "attributes", "geometry"]), path, errors);
|
||||
requiredIdentifier(value.sourceId, `${path}.sourceId`, errors);
|
||||
requiredIdentifier(value.semanticType, `${path}.semanticType`, errors);
|
||||
requiredIsoTimestamp(value.observedAt, `${path}.observedAt`, errors);
|
||||
if (value.attributes !== undefined) {
|
||||
if (!isPlainObject(value.attributes)) {
|
||||
errors.push(`${path}.attributes_must_be_object`);
|
||||
} else if (serializedByteLength(value.attributes) > MAX_FACT_ATTRIBUTES_BYTES) {
|
||||
errors.push(`${path}.attributes_size_exceeded`);
|
||||
}
|
||||
}
|
||||
if (value.geometry !== undefined) validatePointGeometry(value.geometry, `${path}.geometry`, errors);
|
||||
}
|
||||
|
||||
function validatePointGeometry(value, path, errors) {
|
||||
if (!isPlainObject(value) || value.type !== "Point" || !Array.isArray(value.coordinates) || value.coordinates.length !== 2) {
|
||||
errors.push(`${path}_must_be_geojson_point`);
|
||||
return;
|
||||
}
|
||||
rejectUnknownKeys(value, new Set(["type", "coordinates"]), path, errors);
|
||||
if (!value.coordinates.every((coordinate) => typeof coordinate === "number" && Number.isFinite(coordinate))) {
|
||||
errors.push(`${path}_coordinates_must_be_finite_numbers`);
|
||||
return;
|
||||
}
|
||||
const [longitude, latitude] = value.coordinates;
|
||||
if (longitude < -180 || longitude > 180) errors.push(`${path}.longitude_out_of_range`);
|
||||
if (latitude < -90 || latitude > 90) errors.push(`${path}.latitude_out_of_range`);
|
||||
}
|
||||
|
||||
function validateRawEnvelope(value, errors) {
|
||||
if (!isPlainObject(value)) {
|
||||
errors.push("raw_must_be_object");
|
||||
return;
|
||||
}
|
||||
rejectUnknownKeys(value, new Set(["contentType", "payload", "hash", "ref", "retentionDays"]), "raw", errors);
|
||||
requiredString(value.contentType, "raw.contentType", errors);
|
||||
if (value.payload === undefined && value.ref === undefined) errors.push("raw_requires_payload_or_ref");
|
||||
if (value.payload !== undefined) errors.push("raw.inline_payload_not_supported");
|
||||
if (value.payload === undefined) requiredString(value.hash, "raw.hash", errors);
|
||||
if (value.hash !== undefined) requiredString(value.hash, "raw.hash", errors);
|
||||
if (value.ref !== undefined) {
|
||||
requiredString(value.ref, "raw.ref", errors);
|
||||
if (typeof value.ref === "string" && SECRET_LIKE_REFERENCE.test(value.ref)) {
|
||||
errors.push("raw.ref_must_not_contain_secret_material");
|
||||
}
|
||||
}
|
||||
if (value.retentionDays !== undefined && (!Number.isInteger(value.retentionDays) || value.retentionDays < 1)) {
|
||||
errors.push("raw.retentionDays_must_be_positive_integer");
|
||||
}
|
||||
}
|
||||
|
||||
function isPlainObject(value) {
|
||||
return Boolean(value) && typeof value === "object" && !Array.isArray(value);
|
||||
}
|
||||
|
||||
function rejectUnknownKeys(value, allowedKeys, path, errors) {
|
||||
if (!isPlainObject(value)) return;
|
||||
for (const key of Object.keys(value)) {
|
||||
if (!allowedKeys.has(key)) errors.push(`${path}.${key}_not_allowed`);
|
||||
}
|
||||
}
|
||||
|
||||
function serializedByteLength(value) {
|
||||
try {
|
||||
return Buffer.byteLength(JSON.stringify(value));
|
||||
} catch {
|
||||
return Number.POSITIVE_INFINITY;
|
||||
}
|
||||
}
|
||||
|
||||
function containsSecretLikeMaterial(value) {
|
||||
if (typeof value === "string") return SECRET_LIKE_VALUE.test(value);
|
||||
if (Array.isArray(value)) return value.some(containsSecretLikeMaterial);
|
||||
if (!isPlainObject(value)) return false;
|
||||
return Object.entries(value).some(([key, child]) => SECRET_LIKE_KEY.test(key) || containsSecretLikeMaterial(child));
|
||||
}
|
||||
|
||||
function result(errors) {
|
||||
const uniqueErrors = [...new Set(errors)];
|
||||
return Object.freeze({ ok: uniqueErrors.length === 0, errors: Object.freeze(uniqueErrors) });
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
|
|
@ -0,0 +1,16 @@
|
|||
export const NDC_SECRET_CAPABILITY_SOURCE =
|
||||
String.raw`ndc_(?:edp(?:wb|rb|pr)|fndbg)_[A-Za-z0-9_-]+`;
|
||||
|
||||
export const SECRET_LIKE_KEY =
|
||||
/(token|secret|password|authorization|access[_-]?token|refresh[_-]?token|api[_-]?key)/i;
|
||||
|
||||
export const SECRET_LIKE_MATERIAL_KEY =
|
||||
/(token|secret|password|authorization|access[_-]?token|refresh[_-]?token|api[_-]?key|material|value)/i;
|
||||
|
||||
export const SECRET_LIKE_REFERENCE =
|
||||
/(?:[?&](?:token|secret|password|authorization|access[_-]?token|refresh[_-]?token|api[_-]?key)=|(?:bearer|basic)\s+)/i;
|
||||
|
||||
export const SECRET_LIKE_VALUE = new RegExp(
|
||||
`(?:${NDC_SECRET_CAPABILITY_SOURCE}|[?&](?:token|secret|password|authorization|access[_-]?token|refresh[_-]?token|api[_-]?key)=|(?:bearer|basic)\\s+\\S+|eyJ[A-Za-z0-9_-]+\\.[A-Za-z0-9_-]+\\.[A-Za-z0-9_-]+)`,
|
||||
"i",
|
||||
);
|
||||
|
|
@ -15,6 +15,10 @@ import { geliosPositionsCurrentExample } from "../examples/gelios-positions-curr
|
|||
|
||||
assert.equal(validateProviderManifest(geliosPositionsCurrentExample.providerManifest).ok, true);
|
||||
assert.equal(validateConnectionProfile(geliosPositionsCurrentExample.connection).ok, true);
|
||||
assert.equal(validateConnectionProfile({
|
||||
...geliosPositionsCurrentExample.connection,
|
||||
credentialRef: { ...geliosPositionsCurrentExample.connection.credentialRef, owner: "engine" },
|
||||
}).errors.includes("credentialRef.owner_must_be_ndc_l2_credentials"), true);
|
||||
assert.equal(validateCollectionProfile(geliosPositionsCurrentExample.collectionProfile).ok, true);
|
||||
assert.equal(validateDataProduct(geliosPositionsCurrentExample.dataProduct).ok, true);
|
||||
assert.equal(validateFoundryBinding(geliosPositionsCurrentExample.foundryBinding).ok, true);
|
||||
|
|
@ -108,6 +112,10 @@ assert.equal(validateIntakeBatch({
|
|||
...neutralIntakeBatch,
|
||||
facts: [{ ...neutralIntakeBatch.facts[0], attributes: { metadata: "ndc_edprb_abcdefghijklmnopqrstuvwxyz0123456789ABCDE" } }],
|
||||
}).errors.includes("intake_must_not_contain_secret_material"), true);
|
||||
assert.equal(validateIntakeBatch({
|
||||
...neutralIntakeBatch,
|
||||
facts: [{ ...neutralIntakeBatch.facts[0], attributes: { metadata: `ndc_edppr_${"P".repeat(43)}` } }],
|
||||
}).errors.includes("intake_must_not_contain_secret_material"), true);
|
||||
assert.equal(validateIntakeBatch({
|
||||
...neutralIntakeBatch,
|
||||
raw: { contentType: "application/json", hash: "prefix ndc_edpwb_abcdefghijklmnopqrstuvwxyz0123456789ABCDE suffix", ref: "restricted://raw/fixture" },
|
||||
|
|
|
|||
|
|
@ -34,6 +34,10 @@ assert.equal(validateDataProductPublish({
|
|||
...publish,
|
||||
facts: [{ ...fact, attributes: { accessToken: "must-never-cross-the-boundary" } }],
|
||||
}).errors.includes("publish_must_not_contain_secret_material"), true);
|
||||
assert.equal(validateDataProductPublish({
|
||||
...publish,
|
||||
facts: [{ ...fact, attributes: { status: `ndc_edppr_${"P".repeat(43)}` } }],
|
||||
}).errors.includes("publish_must_not_contain_secret_material"), true);
|
||||
assert.equal(validateDataProductPublish({
|
||||
...publish,
|
||||
facts: [fact, { ...fact, observedAt: "2026-07-15T10:00:01.000Z" }],
|
||||
|
|
@ -85,6 +89,10 @@ assert.equal(validateDataProductSnapshot({
|
|||
...snapshot,
|
||||
facts: [{ ...canonicalFact, attributes: { status: "ndc_edprb_forbidden-reader-token" } }],
|
||||
}).errors.includes("snapshot_must_not_contain_secret_material"), true);
|
||||
assert.equal(validateDataProductSnapshot({
|
||||
...snapshot,
|
||||
facts: [{ ...canonicalFact, attributes: { status: `ndc_edppr_${"P".repeat(43)}` } }],
|
||||
}).errors.includes("snapshot_must_not_contain_secret_material"), true);
|
||||
|
||||
const patch = {
|
||||
schemaVersion: DATA_PRODUCT_PATCH_SCHEMA_VERSION,
|
||||
|
|
|
|||
|
|
@ -1,242 +0,0 @@
|
|||
import assert from "node:assert/strict";
|
||||
import { generateKeyPairSync, sign } from "node:crypto";
|
||||
|
||||
import {
|
||||
ENGINE_CREDENTIAL_SINK_AUDIT_SCHEMA_VERSION,
|
||||
ENGINE_CREDENTIAL_SINK_PROVISION_SCHEMA_VERSION,
|
||||
ENGINE_CREDENTIAL_SINK_RECEIPT_SCHEMA_VERSION,
|
||||
ENGINE_CREDENTIAL_SINK_ROLLBACK_RECEIPT_SCHEMA_VERSION,
|
||||
ENGINE_CREDENTIAL_SINK_ROLLBACK_SCHEMA_VERSION,
|
||||
computeEngineCredentialCapabilityDigest,
|
||||
computeEngineCredentialSinkPolicyHash,
|
||||
computeEngineCredentialSinkReceiptHash,
|
||||
engineCredentialSinkAuditTargets,
|
||||
validateEngineCredentialSinkAudit,
|
||||
validateEngineCredentialSinkProvision,
|
||||
validateEngineCredentialSinkReceipt,
|
||||
validateEngineCredentialSinkRollback,
|
||||
validateEngineCredentialSinkRollbackReceipt,
|
||||
} from "../src/index.mjs";
|
||||
|
||||
const hash = (character) => `sha256:${character.repeat(64)}`;
|
||||
const issuer = { serviceId: "platform.external-data-plane", keyId: "edp-issuer-20260715-001" };
|
||||
const { publicKey: issuerPublicKey, privateKey: issuerPrivateKey } = generateKeyPairSync("ed25519");
|
||||
const issuerPublicKeys = { [`${issuer.serviceId}:${issuer.keyId}`]: issuerPublicKey };
|
||||
const target = (nodeId, nodeType, credentialType) => ({
|
||||
workflowId: "WCb62yGL8v",
|
||||
workflowRevision: "revision-20260715-001",
|
||||
nodeId,
|
||||
nodeType,
|
||||
credentialType,
|
||||
});
|
||||
|
||||
const request = {
|
||||
schemaVersion: ENGINE_CREDENTIAL_SINK_PROVISION_SCHEMA_VERSION,
|
||||
transaction: {
|
||||
id: "credential-transaction-20260715-001",
|
||||
idempotencyKey: "credential-transaction-20260715-001",
|
||||
requestedAt: "2026-07-15T18:00:00.000Z",
|
||||
requestExpiresAt: "2026-07-15T18:10:00.000Z",
|
||||
policyHash: hash("0"),
|
||||
failureMode: "rollback-all",
|
||||
issuer,
|
||||
attestation: { algorithm: "Ed25519", signature: "A".repeat(86) },
|
||||
},
|
||||
bindings: [
|
||||
{
|
||||
bindingId: "positions-writer",
|
||||
capabilityType: "external-data-plane.writer",
|
||||
grantId: "writer-grant-001",
|
||||
target: target(
|
||||
"publish-node-001",
|
||||
"n8n-nodes-ndc.ndcDataProductPublish",
|
||||
"ndcDataProductWriterApi",
|
||||
),
|
||||
expiresAt: "2026-08-15T18:00:00.000Z",
|
||||
policyHash: hash("1"),
|
||||
material: { format: "opaque-bearer", value: `ndc_edpwb_${"A".repeat(43)}` },
|
||||
},
|
||||
{
|
||||
bindingId: "positions-reader",
|
||||
capabilityType: "external-data-plane.reader",
|
||||
grantId: "reader-grant-001",
|
||||
target: target(
|
||||
"read-node-001",
|
||||
"n8n-nodes-ndc.ndcDataProductRead",
|
||||
"ndcDataProductReaderApi",
|
||||
),
|
||||
expiresAt: "2026-08-15T18:00:00.000Z",
|
||||
policyHash: hash("2"),
|
||||
material: { format: "opaque-bearer", value: `ndc_edprb_${"B".repeat(43)}` },
|
||||
},
|
||||
{
|
||||
bindingId: "positions-foundry",
|
||||
capabilityType: "foundry.binding",
|
||||
grantId: "foundry-grant-001",
|
||||
target: target(
|
||||
"foundry-node-001",
|
||||
"n8n-nodes-ndc.ndcFoundryBinding",
|
||||
"ndcFoundryBindingApi",
|
||||
),
|
||||
expiresAt: "2026-08-15T18:00:00.000Z",
|
||||
policyHash: hash("3"),
|
||||
material: { format: "opaque-bearer", value: `ndc_fndbg_${"C".repeat(43)}` },
|
||||
},
|
||||
],
|
||||
};
|
||||
for (const binding of request.bindings) {
|
||||
binding.capabilityDigest = computeEngineCredentialCapabilityDigest(binding.material.value);
|
||||
}
|
||||
attest(request);
|
||||
const provisionValidationNow = "2026-07-15T18:00:01.000Z";
|
||||
|
||||
assert.equal(validateEngineCredentialSinkProvision(request, { now: provisionValidationNow, issuerPublicKeys }).ok, true);
|
||||
assert.equal(validateEngineCredentialSinkProvision(request, {
|
||||
now: provisionValidationNow,
|
||||
}).errors.includes("transaction.issuer_public_key_required"), true);
|
||||
assert.equal(validateEngineCredentialSinkProvision({
|
||||
...request,
|
||||
transaction: { ...request.transaction, policyHash: hash("f") },
|
||||
}, { now: provisionValidationNow, issuerPublicKeys }).errors.includes("transaction.policyHash_mismatch"), true);
|
||||
assert.equal(validateEngineCredentialSinkProvision({
|
||||
...request,
|
||||
bindings: [{
|
||||
...request.bindings[0],
|
||||
target: { ...request.bindings[0].target, nodeType: "n8n-nodes-ndc.ndcDataProductRead" },
|
||||
}],
|
||||
}, { now: provisionValidationNow, issuerPublicKeys }).errors.includes("bindings[0].target.nodeType_capability_mismatch"), true);
|
||||
assert.equal(validateEngineCredentialSinkProvision({
|
||||
...request,
|
||||
bindings: [{
|
||||
...request.bindings[0],
|
||||
material: { format: "opaque-bearer", value: request.bindings[1].material.value },
|
||||
}],
|
||||
}, { now: provisionValidationNow, issuerPublicKeys }).errors.includes("bindings[0].material.value_invalid_for_capability"), true);
|
||||
|
||||
const swappedCapabilityRequest = structuredClone(request);
|
||||
swappedCapabilityRequest.bindings[0].material.value = `ndc_edpwb_${"D".repeat(43)}`;
|
||||
assert.equal(validateEngineCredentialSinkProvision(swappedCapabilityRequest, {
|
||||
now: provisionValidationNow,
|
||||
issuerPublicKeys,
|
||||
}).errors.includes("bindings[0].capabilityDigest_material_mismatch"), true);
|
||||
|
||||
const resignedByAttackerRequest = structuredClone(swappedCapabilityRequest);
|
||||
resignedByAttackerRequest.bindings[0].capabilityDigest = computeEngineCredentialCapabilityDigest(
|
||||
resignedByAttackerRequest.bindings[0].material.value,
|
||||
);
|
||||
resignedByAttackerRequest.transaction.policyHash = computeEngineCredentialSinkPolicyHash(resignedByAttackerRequest);
|
||||
assert.equal(validateEngineCredentialSinkProvision(resignedByAttackerRequest, {
|
||||
now: provisionValidationNow,
|
||||
issuerPublicKeys,
|
||||
}).errors.includes("transaction.attestation_invalid"), true);
|
||||
|
||||
const staleRequest = structuredClone(request);
|
||||
staleRequest.transaction.requestedAt = "2020-01-01T00:00:00.000Z";
|
||||
staleRequest.transaction.requestExpiresAt = "2020-01-01T00:10:00.000Z";
|
||||
attest(staleRequest);
|
||||
assert.equal(validateEngineCredentialSinkProvision(staleRequest, { now: provisionValidationNow, issuerPublicKeys }).errors.includes("request_expired"), true);
|
||||
|
||||
const futureRequest = structuredClone(request);
|
||||
futureRequest.transaction.requestedAt = "2026-07-15T18:02:00.000Z";
|
||||
futureRequest.transaction.requestExpiresAt = "2026-07-15T18:12:00.000Z";
|
||||
attest(futureRequest);
|
||||
assert.equal(validateEngineCredentialSinkProvision(futureRequest, { now: provisionValidationNow, issuerPublicKeys }).errors.includes("requestedAt_exceeds_clock_skew"), true);
|
||||
|
||||
const credentials = request.bindings.map((binding, index) => ({
|
||||
bindingId: binding.bindingId,
|
||||
capabilityType: binding.capabilityType,
|
||||
grantId: binding.grantId,
|
||||
target: binding.target,
|
||||
credentialRef: `engcred_${index + 1}_opaque_reference`,
|
||||
expiresAt: binding.expiresAt,
|
||||
policyHash: binding.policyHash,
|
||||
capabilityDigest: binding.capabilityDigest,
|
||||
disposition: "created",
|
||||
}));
|
||||
const receipt = {
|
||||
schemaVersion: ENGINE_CREDENTIAL_SINK_RECEIPT_SCHEMA_VERSION,
|
||||
transactionId: request.transaction.id,
|
||||
idempotencyKey: request.transaction.idempotencyKey,
|
||||
outcome: "committed",
|
||||
policyHash: request.transaction.policyHash,
|
||||
processedAt: "2026-07-15T18:00:02.000Z",
|
||||
credentials,
|
||||
rollback: { status: "not-required" },
|
||||
};
|
||||
assert.equal(validateEngineCredentialSinkReceipt(receipt, { request, issuerPublicKeys }).ok, true);
|
||||
assert.equal(validateEngineCredentialSinkReceipt({
|
||||
...receipt,
|
||||
credentials: [{ ...credentials[0], target: { ...credentials[0].target, nodeId: "wrong-node" } }, ...credentials.slice(1)],
|
||||
}, { request, issuerPublicKeys }).errors.includes("credentials_request_target_mismatch"), true);
|
||||
assert.equal(validateEngineCredentialSinkReceipt({
|
||||
...receipt,
|
||||
capability: request.bindings[0].material.value,
|
||||
}).errors.includes("credentialSinkReceipt.capability_not_allowed"), true);
|
||||
assert.equal(validateEngineCredentialSinkReceipt({
|
||||
...receipt,
|
||||
outcome: "rolled-back",
|
||||
credentials: credentials.slice(0, 1),
|
||||
rollback: { status: "complete", completedAt: "2026-07-15T18:00:02.000Z" },
|
||||
errorCode: "engine_binding_failed",
|
||||
}).errors.includes("noncommitted_credentials_must_be_empty"), true);
|
||||
|
||||
const committedReceiptHash = computeEngineCredentialSinkReceiptHash(receipt);
|
||||
const rollbackRequest = {
|
||||
schemaVersion: ENGINE_CREDENTIAL_SINK_ROLLBACK_SCHEMA_VERSION,
|
||||
rollback: {
|
||||
id: "credential-rollback-20260715-001",
|
||||
idempotencyKey: "credential-rollback-20260715-001",
|
||||
transactionId: request.transaction.id,
|
||||
requestedAt: "2026-07-15T18:05:00.000Z",
|
||||
requestExpiresAt: "2026-07-15T18:10:00.000Z",
|
||||
policyHash: request.transaction.policyHash,
|
||||
committedReceiptHash,
|
||||
reasonCode: "operator_requested",
|
||||
},
|
||||
};
|
||||
assert.equal(validateEngineCredentialSinkRollback(rollbackRequest, { now: "2026-07-15T18:05:01.000Z" }).ok, true);
|
||||
const staleRollbackRequest = structuredClone(rollbackRequest);
|
||||
staleRollbackRequest.rollback.requestedAt = "2020-01-01T00:00:00.000Z";
|
||||
staleRollbackRequest.rollback.requestExpiresAt = "2020-01-01T00:10:00.000Z";
|
||||
assert.equal(validateEngineCredentialSinkRollback(staleRollbackRequest, { now: "2026-07-15T18:05:01.000Z" }).errors.includes("request_expired"), true);
|
||||
const rollbackReceipt = {
|
||||
schemaVersion: ENGINE_CREDENTIAL_SINK_ROLLBACK_RECEIPT_SCHEMA_VERSION,
|
||||
rollbackId: rollbackRequest.rollback.id,
|
||||
transactionId: rollbackRequest.rollback.transactionId,
|
||||
outcome: "rolled-back",
|
||||
policyHash: rollbackRequest.rollback.policyHash,
|
||||
committedReceiptHash,
|
||||
processedAt: "2026-07-15T18:05:01.000Z",
|
||||
};
|
||||
assert.equal(validateEngineCredentialSinkRollbackReceipt(rollbackReceipt, { request: rollbackRequest }).ok, true);
|
||||
|
||||
const audit = {
|
||||
schemaVersion: ENGINE_CREDENTIAL_SINK_AUDIT_SCHEMA_VERSION,
|
||||
eventId: "credential-audit-20260715-001",
|
||||
transactionId: request.transaction.id,
|
||||
operationId: request.transaction.id,
|
||||
operation: "provision",
|
||||
outcome: "committed",
|
||||
occurredAt: receipt.processedAt,
|
||||
policyHash: request.transaction.policyHash,
|
||||
principal: { serviceId: "platform.credential-provisioner", fingerprint: hash("a") },
|
||||
targets: engineCredentialSinkAuditTargets(request, receipt),
|
||||
};
|
||||
assert.equal(validateEngineCredentialSinkAudit(audit).ok, true);
|
||||
assert.equal(JSON.stringify(audit).includes("ndc_edpwb_"), false);
|
||||
assert.equal(JSON.stringify(audit).includes("engcred_1_opaque_reference"), false);
|
||||
assert.equal(validateEngineCredentialSinkAudit({
|
||||
...audit,
|
||||
authorization: `Bearer ${request.bindings[0].material.value}`,
|
||||
}).errors.includes("audit_must_not_contain_secret_material"), true);
|
||||
|
||||
function attest(value) {
|
||||
value.transaction.policyHash = computeEngineCredentialSinkPolicyHash(value);
|
||||
value.transaction.attestation.signature = sign(
|
||||
null,
|
||||
Buffer.from(value.transaction.policyHash, "utf8"),
|
||||
issuerPrivateKey,
|
||||
).toString("base64url");
|
||||
}
|
||||
|
||||
console.log("external-provider credential sink contract: ok");
|
||||
|
|
@ -132,6 +132,16 @@ assert.equal(
|
|||
}).errors.includes("transition_policy_mismatch"),
|
||||
true,
|
||||
);
|
||||
assert.equal(
|
||||
validateEnginePrivateExtensionPlan({
|
||||
...activatePlan,
|
||||
transition: {
|
||||
...transition(),
|
||||
loaderEnvironment: { ...transition().loaderEnvironment, NODE_PATH: "/tmp/untrusted-node-modules" },
|
||||
},
|
||||
}).errors.includes("transition.loaderEnvironment.NODE_PATH_not_allowed"),
|
||||
true,
|
||||
);
|
||||
|
||||
const applyRequest = {
|
||||
schemaVersion: ENGINE_PRIVATE_EXTENSION_APPLY_REQUEST_SCHEMA_VERSION,
|
||||
|
|
|
|||
|
|
@ -0,0 +1,464 @@
|
|||
import assert from "node:assert/strict";
|
||||
import { readFile } from "node:fs/promises";
|
||||
import {
|
||||
L2_CONNECTION_INSTANCE_SCHEMA_VERSION,
|
||||
instantiateL2Connection,
|
||||
validateDataProductPublish,
|
||||
validateProviderPackage,
|
||||
} from "../src/index.mjs";
|
||||
import {
|
||||
GELIOS_POSITIONS_DATA_PRODUCT_ID,
|
||||
GELIOS_POSITIONS_DATA_PRODUCT_VERSION,
|
||||
GELIOS_POSITIONS_ONTOLOGY_REVISION,
|
||||
GELIOS_UNIT_SOURCE_ID_PREFIX,
|
||||
geliosProviderPackageV1,
|
||||
geliosUnitsCurrentFixtureV1,
|
||||
} from "../providers/gelios/v1/index.mjs";
|
||||
import { normalizeDataProductDefinition } from "../../../services/external-data-plane/src/data-product-policy.mjs";
|
||||
|
||||
const expectedFields = [
|
||||
"course_degrees",
|
||||
"display_name",
|
||||
"elevation_meters",
|
||||
"geometry",
|
||||
"hdop",
|
||||
"horizontal_accuracy_meters",
|
||||
"object_kind",
|
||||
"operational_status",
|
||||
"position_source",
|
||||
"position_valid",
|
||||
"quality_flags",
|
||||
"satellite_count",
|
||||
"speed_kph",
|
||||
];
|
||||
|
||||
assert.deepEqual(validateProviderPackage(geliosProviderPackageV1), { ok: true, errors: [] });
|
||||
|
||||
const geliosAuth = geliosProviderPackageV1.authModes[0];
|
||||
const geliosUnitsRead = geliosProviderPackageV1.capabilities[0];
|
||||
assert.equal(geliosAuth.id, "gelios.rest-bearer.v1");
|
||||
assert.deepEqual(geliosAuth.transport, { placement: "header", name: "Authorization" });
|
||||
assert.deepEqual(geliosAuth.tokenLifecycle, {
|
||||
artifacts: ["access", "refresh"],
|
||||
requestArtifact: "access",
|
||||
refreshMode: "operator_managed",
|
||||
});
|
||||
assert.equal(geliosAuth.tokenLifecycle.artifacts.includes("writer"), false);
|
||||
assert.equal(geliosUnitsRead.request.baseUrl, "https://api.geliospro.com");
|
||||
assert.equal(geliosUnitsRead.request.path, "/api/v1/units");
|
||||
assert.deepEqual(geliosUnitsRead.request.query, {});
|
||||
|
||||
const product = geliosProviderPackageV1.dataProducts[0];
|
||||
const registeredProduct = JSON.parse(await readFile(new URL(
|
||||
"../../../services/external-data-plane/definitions/fleet.positions.current.v1.json",
|
||||
import.meta.url,
|
||||
), "utf8"));
|
||||
assert.equal(product.id, GELIOS_POSITIONS_DATA_PRODUCT_ID);
|
||||
assert.equal(product.version, GELIOS_POSITIONS_DATA_PRODUCT_VERSION);
|
||||
assert.equal(product.ontologyRevision, GELIOS_POSITIONS_ONTOLOGY_REVISION);
|
||||
assert.equal(product.id, "fleet.positions.current.v1");
|
||||
assert.equal(product.version, "1.0.0");
|
||||
assert.equal(product.ontologyRevision, "ontology.map.moving_object.v1");
|
||||
assert.deepEqual(product.fields, expectedFields);
|
||||
assert.deepEqual(product, registeredProduct);
|
||||
assert.doesNotThrow(() => normalizeDataProductDefinition(product));
|
||||
assert.equal(product.fields.every((field) => /^[a-z][a-z0-9_]*$/.test(field)), true);
|
||||
|
||||
const mapping = geliosProviderPackageV1.mappingContracts[0];
|
||||
assert.equal(mapping.target.dataProductId, product.id);
|
||||
assert.equal(mapping.target.version, product.version);
|
||||
assert.equal(mapping.target.ontologyRevision, product.ontologyRevision);
|
||||
assert.deepEqual(
|
||||
[...Object.keys(mapping.fact.attributes), "geometry"].sort(),
|
||||
[...product.fields].sort(),
|
||||
);
|
||||
assert.equal(mapping.derivations.operational_status.parameters.staleAfterMs, 60000);
|
||||
assert.deepEqual(mapping.derivations.quality_flags.default, []);
|
||||
assert.equal(mapping.fact.observedAt.fallback, "collection_received_at");
|
||||
assert.equal(GELIOS_UNIT_SOURCE_ID_PREFIX, "gelios-unit-");
|
||||
assert.equal(mapping.fact.sourceId.prefix, GELIOS_UNIT_SOURCE_ID_PREFIX);
|
||||
assert.equal(mapping.fact.sourceId.coerce, "string");
|
||||
assert.equal(mapping.fact.geometry.longitude.paths.includes("lastMsg.lon"), true);
|
||||
assert.equal(mapping.fact.geometry.latitude.paths.includes("lastMsg.lat"), true);
|
||||
assert.equal(mapping.fact.attributes.speed_kph.paths.includes("lastMsg.speed"), true);
|
||||
|
||||
assert.equal(validateDataProductPublish(geliosUnitsCurrentFixtureV1.expectedPublish).ok, true);
|
||||
assert.equal(
|
||||
geliosUnitsCurrentFixtureV1.sourceResponse.units.length,
|
||||
geliosUnitsCurrentFixtureV1.expectedPublish.facts.length,
|
||||
"every entity returned for the bound credential remains in the publish fixture",
|
||||
);
|
||||
assert.deepEqual(
|
||||
geliosUnitsCurrentFixtureV1.expectedPublish.facts.map((fact) => fact.sourceId),
|
||||
geliosUnitsCurrentFixtureV1.sourceResponse.units.map(
|
||||
(unit) => `${GELIOS_UNIT_SOURCE_ID_PREFIX}${String(unit.id)}`,
|
||||
),
|
||||
);
|
||||
assert.equal(
|
||||
geliosUnitsCurrentFixtureV1.expectedPublish.facts[1].sourceId,
|
||||
"gelios-unit-001002",
|
||||
"provider ID strings must remain byte-for-byte stable after the canonical prefix",
|
||||
);
|
||||
assert.equal(geliosUnitsCurrentFixtureV1.expectedPublish.facts[1].geometry, undefined);
|
||||
assert.equal(geliosUnitsCurrentFixtureV1.expectedPublish.facts[1].attributes.operational_status, "no_position");
|
||||
assert.equal(geliosUnitsCurrentFixtureV1.sourceResponse.units[1].lastMsg, undefined);
|
||||
assert.equal(
|
||||
geliosUnitsCurrentFixtureV1.expectedPublish.facts[1].observedAt,
|
||||
geliosUnitsCurrentFixtureV1.collectionContext.receivedAt,
|
||||
);
|
||||
|
||||
const realtimeProfileId = "gelios.positions.current.realtime.v1";
|
||||
assert.equal(
|
||||
geliosProviderPackageV1.collectionProfiles[0].cardinality.onExceed,
|
||||
"require_partitioned_data_product",
|
||||
);
|
||||
const accountA = instantiateL2Connection(geliosProviderPackageV1, {
|
||||
tenantId: "tenant-alpha",
|
||||
connectionId: "gelios-account-alpha",
|
||||
collectionProfileId: realtimeProfileId,
|
||||
providerCredentialRef: "ndc-credref:provider-alpha-0001",
|
||||
});
|
||||
const accountB = instantiateL2Connection(geliosProviderPackageV1, {
|
||||
tenantId: "tenant-beta",
|
||||
connectionId: "gelios-account-beta",
|
||||
collectionProfileId: realtimeProfileId,
|
||||
providerCredentialRef: "ndc-credref:provider-beta-0001",
|
||||
});
|
||||
|
||||
assert.equal(accountA.schemaVersion, L2_CONNECTION_INSTANCE_SCHEMA_VERSION);
|
||||
assert.equal(accountA.package.id, accountB.package.id);
|
||||
assert.equal(accountA.package.version, accountB.package.version);
|
||||
assert.equal(accountA.l2TemplateId, accountB.l2TemplateId);
|
||||
assert.notEqual(accountA.connectionId, accountB.connectionId);
|
||||
assert.notEqual(accountA.credentialRefs.provider.reference, accountB.credentialRefs.provider.reference);
|
||||
assert.equal(accountA.credentialRefs.provider.owner, "ndc_l2_credentials");
|
||||
assert.deepEqual(Object.keys(accountA.credentialRefs), ["provider"]);
|
||||
assert.deepEqual(accountA.systemBindings.publisher, {
|
||||
role: "publisher",
|
||||
owner: "ndc_l2_credentials",
|
||||
management: "control_plane_managed",
|
||||
nodeType: "n8n-nodes-ndc.ndcDataProductPublish",
|
||||
desiredState: "bound",
|
||||
status: "unresolved",
|
||||
});
|
||||
assert.equal(accountA.scope.mode, "all_visible_to_credential");
|
||||
assert.equal(accountA.scope.refresh, "each_collection_run");
|
||||
assert.equal(Object.isFrozen(accountA), true);
|
||||
assert.equal(Object.isFrozen(accountA.credentialRefs), true);
|
||||
assert.equal(Object.isFrozen(accountA.systemBindings), true);
|
||||
|
||||
const withEntityAllowlist = structuredClone(geliosProviderPackageV1);
|
||||
withEntityAllowlist.capabilities[0].entityScope.unitIds = ["gelios-unit-1001"];
|
||||
assert.equal(
|
||||
validateProviderPackage(withEntityAllowlist).errors.includes("capabilities[0].entityScope.unitIds_not_allowed"),
|
||||
true,
|
||||
);
|
||||
|
||||
const withGroupFilter = structuredClone(geliosProviderPackageV1);
|
||||
withGroupFilter.collectionProfiles[0].entityScope.businessEntityFilter = "provider_group";
|
||||
assert.equal(
|
||||
validateProviderPackage(withGroupFilter).errors.includes("collectionProfiles[0].entityScope.businessEntityFilter_must_be_forbidden"),
|
||||
true,
|
||||
);
|
||||
|
||||
const withWrongRevision = structuredClone(geliosProviderPackageV1);
|
||||
withWrongRevision.mappingContracts[0].target.ontologyRevision = "gelios.positions.v1";
|
||||
assert.equal(
|
||||
validateProviderPackage(withWrongRevision).errors.includes("mappingContract.gelios.units.to.fleet.positions.current.v1.target.ontologyRevision_must_match_data_product"),
|
||||
true,
|
||||
);
|
||||
|
||||
const withMissingDerivation = structuredClone(geliosProviderPackageV1);
|
||||
delete withMissingDerivation.mappingContracts[0].derivations.position_valid;
|
||||
assert.equal(
|
||||
validateProviderPackage(withMissingDerivation).errors.includes("mappingContracts[0].derive_position_valid_missing_definition"),
|
||||
true,
|
||||
);
|
||||
|
||||
const withWrongFactSemanticType = structuredClone(geliosProviderPackageV1);
|
||||
withWrongFactSemanticType.mappingContracts[0].fact.semanticType.constant = "map.other";
|
||||
assert.equal(
|
||||
validateProviderPackage(withWrongFactSemanticType).errors.includes(
|
||||
"mappingContract.gelios.units.to.fleet.positions.current.v1.fact.semanticType_must_equal_target_constant",
|
||||
),
|
||||
true,
|
||||
);
|
||||
|
||||
const nonGeospatialPackage = structuredClone(geliosProviderPackageV1);
|
||||
nonGeospatialPackage.dataProducts[0].fields = ["display_name"];
|
||||
nonGeospatialPackage.fieldPolicies[0].targetFields = ["display_name"];
|
||||
delete nonGeospatialPackage.mappingContracts[0].fact.geometry;
|
||||
nonGeospatialPackage.mappingContracts[0].fact.attributes = {
|
||||
display_name: { strategy: "first_non_empty", paths: ["name"], coerce: "string" },
|
||||
};
|
||||
delete nonGeospatialPackage.mappingContracts[0].derivations;
|
||||
assert.deepEqual(validateProviderPackage(nonGeospatialPackage), { ok: true, errors: [] });
|
||||
|
||||
for (const restrictedPath of ["phone", "phone.number", "lmsg"]) {
|
||||
const withRestrictedMappingPath = structuredClone(geliosProviderPackageV1);
|
||||
withRestrictedMappingPath.mappingContracts[0].fact.attributes.display_name.paths = [restrictedPath];
|
||||
assert.equal(
|
||||
validateProviderPackage(withRestrictedMappingPath).errors.includes(
|
||||
"mappingContract.gelios.units.to.fleet.positions.current.v1.fact_source_path_restricted",
|
||||
),
|
||||
true,
|
||||
);
|
||||
}
|
||||
|
||||
const withFieldPolicyProductMismatch = structuredClone(geliosProviderPackageV1);
|
||||
const secondProduct = structuredClone(withFieldPolicyProductMismatch.dataProducts[0]);
|
||||
secondProduct.id = "fleet.positions.alternate.v1";
|
||||
withFieldPolicyProductMismatch.dataProducts.push(secondProduct);
|
||||
withFieldPolicyProductMismatch.manifest.dataProductIds.push(secondProduct.id);
|
||||
withFieldPolicyProductMismatch.fieldPolicies[0].dataProductId = secondProduct.id;
|
||||
assert.equal(
|
||||
validateProviderPackage(withFieldPolicyProductMismatch).errors.includes(
|
||||
"mappingContract.gelios.units.to.fleet.positions.current.v1.field_policy_data_product_mismatch",
|
||||
),
|
||||
true,
|
||||
);
|
||||
|
||||
const withCamelCaseField = structuredClone(geliosProviderPackageV1);
|
||||
withCamelCaseField.dataProducts[0].fields[12] = "speedKph";
|
||||
assert.equal(validateProviderPackage(withCamelCaseField).ok, false);
|
||||
|
||||
const withSecretField = structuredClone(geliosProviderPackageV1);
|
||||
withSecretField.dataProducts[0].fields[12] = "api_token";
|
||||
withSecretField.fieldPolicies[0].targetFields[12] = "api_token";
|
||||
withSecretField.mappingContracts[0].fact.attributes.api_token = { constant: "metadata-only" };
|
||||
delete withSecretField.mappingContracts[0].fact.attributes.speed_kph;
|
||||
const secretFieldValidation = validateProviderPackage(withSecretField);
|
||||
assert.equal(secretFieldValidation.errors.includes("dataProducts[0].fields_must_not_contain_secret_fields"), true);
|
||||
assert.equal(secretFieldValidation.errors.includes("fieldPolicies[0].targetFields_must_not_contain_secret_fields"), true);
|
||||
assert.equal(secretFieldValidation.errors.includes("mappingContracts[0].fact.attributes.api_token_secret_field_not_allowed"), true);
|
||||
|
||||
const withRuntimeAccountState = structuredClone(geliosProviderPackageV1);
|
||||
withRuntimeAccountState.tenantId = "must-not-live-in-package";
|
||||
assert.equal(
|
||||
validateProviderPackage(withRuntimeAccountState).errors.includes("providerPackage_must_not_contain_connection_instance_state"),
|
||||
true,
|
||||
);
|
||||
|
||||
const withPrimitiveCapability = structuredClone(geliosProviderPackageV1);
|
||||
withPrimitiveCapability.capabilities.push(null);
|
||||
assert.doesNotThrow(() => validateProviderPackage(withPrimitiveCapability));
|
||||
assert.equal(validateProviderPackage(withPrimitiveCapability).ok, false);
|
||||
|
||||
const withMalformedNestedArrays = structuredClone(geliosProviderPackageV1);
|
||||
withMalformedNestedArrays.collectionProfiles[0].capabilityIds = {};
|
||||
withMalformedNestedArrays.l2Templates[0].credentialBindings = {};
|
||||
withMalformedNestedArrays.l2Templates[0].steps = {};
|
||||
assert.doesNotThrow(() => validateProviderPackage(withMalformedNestedArrays));
|
||||
assert.equal(validateProviderPackage(withMalformedNestedArrays).ok, false);
|
||||
|
||||
const withCredentialInStaticQuery = structuredClone(geliosProviderPackageV1);
|
||||
withCredentialInStaticQuery.authModes[0].transport = { placement: "query", name: "token" };
|
||||
withCredentialInStaticQuery.capabilities[0].request.query.token = "plain-provider-key";
|
||||
assert.equal(
|
||||
validateProviderPackage(withCredentialInStaticQuery).errors.includes("capabilities[0].request.query.token_credential_parameter_not_allowed"),
|
||||
true,
|
||||
);
|
||||
assert.equal(
|
||||
validateProviderPackage(withCredentialInStaticQuery).errors.includes("capability.gelios.units.current.read.request.query_must_not_embed_credential_parameter"),
|
||||
true,
|
||||
);
|
||||
|
||||
const withCredentialInBaseUrl = structuredClone(geliosProviderPackageV1);
|
||||
withCredentialInBaseUrl.capabilities[0].request.baseUrl = "https://user:pass@api.geliospro.com/?token=forbidden";
|
||||
assert.equal(validateProviderPackage(withCredentialInBaseUrl).ok, false);
|
||||
|
||||
for (const unsafePath of ["/ok\nX-Evil: yes", "/back\\slash", "/query?x=1"]) {
|
||||
const withUnsafeRequestPath = structuredClone(geliosProviderPackageV1);
|
||||
withUnsafeRequestPath.capabilities[0].request.path = unsafePath;
|
||||
assert.equal(validateProviderPackage(withUnsafeRequestPath).ok, false);
|
||||
}
|
||||
|
||||
for (const unsafeCollectionPath of ["__proto__.polluted", "data.items[0]", "constructor.prototype"]) {
|
||||
const withUnsafeCollectionPath = structuredClone(geliosProviderPackageV1);
|
||||
withUnsafeCollectionPath.capabilities[0].request.response.collectionPaths = [unsafeCollectionPath];
|
||||
assert.equal(validateProviderPackage(withUnsafeCollectionPath).ok, false);
|
||||
}
|
||||
|
||||
const withUnsafeMappingPath = structuredClone(geliosProviderPackageV1);
|
||||
withUnsafeMappingPath.mappingContracts[0].fact.attributes.display_name.paths = ["__proto__.polluted"];
|
||||
assert.equal(validateProviderPackage(withUnsafeMappingPath).ok, false);
|
||||
|
||||
const withUnsafeCollectionCapability = structuredClone(geliosProviderPackageV1);
|
||||
withUnsafeCollectionCapability.capabilities[0].classification = "write";
|
||||
assert.equal(
|
||||
validateProviderPackage(withUnsafeCollectionCapability).errors.includes("collectionProfile.gelios.positions.current.realtime.v1.capabilityIds_must_reference_implemented_safe_read"),
|
||||
true,
|
||||
);
|
||||
|
||||
const withUnexecutedSecondCapability = structuredClone(geliosProviderPackageV1);
|
||||
const secondCapability = structuredClone(withUnexecutedSecondCapability.capabilities[0]);
|
||||
secondCapability.id = "gelios.units.metadata.read";
|
||||
withUnexecutedSecondCapability.capabilities.push(secondCapability);
|
||||
withUnexecutedSecondCapability.manifest.capabilityIds.push(secondCapability.id);
|
||||
for (const profile of withUnexecutedSecondCapability.collectionProfiles) {
|
||||
profile.capabilityIds.push(secondCapability.id);
|
||||
}
|
||||
assert.equal(
|
||||
validateProviderPackage(withUnexecutedSecondCapability).errors.includes(
|
||||
"collectionProfile.gelios.positions.current.realtime.v1.capabilityIds_must_select_one_v1_read",
|
||||
),
|
||||
true,
|
||||
);
|
||||
|
||||
const withReorderedSteps = structuredClone(geliosProviderPackageV1);
|
||||
[withReorderedSteps.l2Templates[0].steps[0], withReorderedSteps.l2Templates[0].steps[1]] = [
|
||||
withReorderedSteps.l2Templates[0].steps[1],
|
||||
withReorderedSteps.l2Templates[0].steps[0],
|
||||
];
|
||||
assert.equal(
|
||||
validateProviderPackage(withReorderedSteps).errors.includes("l2Templates[0].steps_sequence_invalid"),
|
||||
true,
|
||||
);
|
||||
|
||||
const withUnknownConnectionParameters = structuredClone(geliosProviderPackageV1);
|
||||
withUnknownConnectionParameters.l2Templates[0].connectionParameters = ["foo"];
|
||||
assert.equal(
|
||||
validateProviderPackage(withUnknownConnectionParameters).errors.includes(
|
||||
"l2Templates[0].connectionParameters_must_match_v1_contract",
|
||||
),
|
||||
true,
|
||||
);
|
||||
|
||||
const withCredentialAuthMismatch = structuredClone(geliosProviderPackageV1);
|
||||
const secondAuthMode = structuredClone(withCredentialAuthMismatch.authModes[0]);
|
||||
secondAuthMode.id = "gelios.rest-bearer.alternate.v1";
|
||||
withCredentialAuthMismatch.authModes.push(secondAuthMode);
|
||||
withCredentialAuthMismatch.manifest.authModeIds.push(secondAuthMode.id);
|
||||
withCredentialAuthMismatch.l2Templates[0].credentialBindings[0].authModeId = secondAuthMode.id;
|
||||
assert.equal(
|
||||
validateProviderPackage(withCredentialAuthMismatch).errors.includes(
|
||||
"l2Template.gelios.positions.current.l2.v1.provider_credential_auth_mode_mismatch",
|
||||
),
|
||||
true,
|
||||
);
|
||||
|
||||
const withCallerManagedPublisher = structuredClone(geliosProviderPackageV1);
|
||||
withCallerManagedPublisher.l2Templates[0].credentialBindings[1].management = "connection_input";
|
||||
assert.equal(
|
||||
validateProviderPackage(withCallerManagedPublisher).errors.includes(
|
||||
"l2Templates[0].credentialBindings[1].publisher_binding_shape_invalid",
|
||||
),
|
||||
true,
|
||||
);
|
||||
|
||||
const withUnknownTokenArtifact = structuredClone(geliosProviderPackageV1);
|
||||
withUnknownTokenArtifact.authModes[0].tokenLifecycle.artifacts.push("writer");
|
||||
assert.equal(
|
||||
validateProviderPackage(withUnknownTokenArtifact).errors.includes(
|
||||
"authModes[0].tokenLifecycle.artifacts_invalid",
|
||||
),
|
||||
true,
|
||||
);
|
||||
|
||||
const withRefreshUsedForRequest = structuredClone(geliosProviderPackageV1);
|
||||
withRefreshUsedForRequest.authModes[0].tokenLifecycle.requestArtifact = "refresh";
|
||||
assert.equal(
|
||||
validateProviderPackage(withRefreshUsedForRequest).errors.includes(
|
||||
"authModes[0].tokenLifecycle.requestArtifact_must_be_access",
|
||||
),
|
||||
true,
|
||||
);
|
||||
|
||||
const withUnmanagedRefreshArtifact = structuredClone(geliosProviderPackageV1);
|
||||
withUnmanagedRefreshArtifact.authModes[0].tokenLifecycle.refreshMode = "not_applicable";
|
||||
assert.equal(
|
||||
validateProviderPackage(withUnmanagedRefreshArtifact).errors.includes(
|
||||
"authModes[0].tokenLifecycle.refreshMode_required_for_refresh_artifact",
|
||||
),
|
||||
true,
|
||||
);
|
||||
|
||||
for (const history of [{
|
||||
mode: "none",
|
||||
intervalMs: 60000,
|
||||
strategy: "latest-per-entity-per-bucket",
|
||||
retentionDays: 90,
|
||||
}, {
|
||||
mode: "sampled",
|
||||
intervalMs: 24 * 60 * 60 * 1000 + 1,
|
||||
strategy: "latest-per-entity-per-bucket",
|
||||
retentionDays: 90,
|
||||
}, {
|
||||
mode: "all",
|
||||
retentionDays: 3651,
|
||||
}]) {
|
||||
const withInvalidHistory = structuredClone(geliosProviderPackageV1);
|
||||
withInvalidHistory.dataProducts[0].history = history;
|
||||
assert.equal(validateProviderPackage(withInvalidHistory).ok, false);
|
||||
assert.throws(() => normalizeDataProductDefinition(withInvalidHistory.dataProducts[0]));
|
||||
}
|
||||
|
||||
const withBrokenArtifactChain = structuredClone(geliosProviderPackageV1);
|
||||
withBrokenArtifactChain.l2Templates[0].steps[4].dataProductId = "unrelated.product.v1";
|
||||
assert.equal(
|
||||
validateProviderPackage(withBrokenArtifactChain).errors.includes("collectionProfile.gelios.positions.current.realtime.v1.template_data_product_mismatch"),
|
||||
true,
|
||||
);
|
||||
|
||||
const withSecretNamedDerivationParameter = structuredClone(geliosProviderPackageV1);
|
||||
withSecretNamedDerivationParameter.mappingContracts[0].derivations.position_valid.parameters = {
|
||||
accessToken: "opaque-provider-value",
|
||||
};
|
||||
assert.equal(
|
||||
validateProviderPackage(withSecretNamedDerivationParameter).errors.includes(
|
||||
"mappingContracts[0].derivations.position_valid.parameters.accessToken_secret_field_not_allowed",
|
||||
),
|
||||
true,
|
||||
);
|
||||
|
||||
const withObjectConstant = structuredClone(geliosProviderPackageV1);
|
||||
withObjectConstant.mappingContracts[0].fact.attributes.object_kind.constant = {
|
||||
accessToken: "opaque-provider-value",
|
||||
};
|
||||
assert.equal(
|
||||
validateProviderPackage(withObjectConstant).errors.includes(
|
||||
"mappingContracts[0].fact.attributes.object_kind.constant_must_be_scalar_or_scalar_array",
|
||||
),
|
||||
true,
|
||||
);
|
||||
|
||||
assert.throws(() => instantiateL2Connection(geliosProviderPackageV1, {
|
||||
tenantId: "tenant-alpha",
|
||||
connectionId: "gelios-account-alpha",
|
||||
collectionProfileId: realtimeProfileId,
|
||||
providerCredentialRef: "Bearer plaintext-must-not-cross-boundary",
|
||||
}), /l2_connection_invalid/);
|
||||
|
||||
assert.throws(() => instantiateL2Connection(geliosProviderPackageV1, {
|
||||
tenantId: "tenant-alpha",
|
||||
connectionId: "gelios-account-alpha",
|
||||
collectionProfileId: realtimeProfileId,
|
||||
providerCredentialRef: "ndc-credref:provider-alpha-0001",
|
||||
writerCredentialRef: "ndc-credref:caller-writer-forbidden",
|
||||
}), /options\.writerCredentialRef_not_allowed/);
|
||||
|
||||
const serializedPackage = JSON.stringify(geliosProviderPackageV1);
|
||||
assert.equal(serializedPackage.includes('"prefix":"unit-"'), false);
|
||||
assert.equal(serializedPackage.includes("unitIds"), false);
|
||||
assert.equal(serializedPackage.includes("tenant-alpha"), false);
|
||||
assert.equal(serializedPackage.includes("gelios-gateway"), false);
|
||||
assert.equal(serializedPackage.includes("writer_credential_ref"), false);
|
||||
assert.equal(serializedPackage.includes("writerCapabilityByReferenceOnly"), false);
|
||||
assert.equal(serializedPackage.includes("publisherBindingControlPlaneManaged\":true"), true);
|
||||
assert.equal(serializedPackage.includes("providerSpecificServiceForbidden\":true"), true);
|
||||
assert.deepEqual(
|
||||
geliosProviderPackageV1.l2Templates[0].steps.map((step) => step.kind),
|
||||
[
|
||||
"collection_trigger",
|
||||
"provider_request",
|
||||
"extract_items",
|
||||
"semantic_mapping",
|
||||
"data_product_publish",
|
||||
],
|
||||
);
|
||||
assert.equal(
|
||||
geliosProviderPackageV1.l2Templates[0].steps.at(-1).nodeType,
|
||||
"n8n-nodes-ndc.ndcDataProductPublish",
|
||||
);
|
||||
|
||||
console.log("external-provider-contract provider-package: ok");
|
||||
|
|
@ -2,7 +2,12 @@ FROM node:20-alpine AS deps
|
|||
|
||||
WORKDIR /workspace
|
||||
|
||||
COPY packages/external-provider-contract ./packages/external-provider-contract
|
||||
COPY packages/external-provider-contract/package.json ./packages/external-provider-contract/package.json
|
||||
COPY packages/external-provider-contract/src/contract-version.mjs ./packages/external-provider-contract/src/contract-version.mjs
|
||||
COPY packages/external-provider-contract/src/data-plane.mjs ./packages/external-provider-contract/src/data-plane.mjs
|
||||
COPY packages/external-provider-contract/src/data-product.mjs ./packages/external-provider-contract/src/data-product.mjs
|
||||
COPY packages/external-provider-contract/src/intake-batch.mjs ./packages/external-provider-contract/src/intake-batch.mjs
|
||||
COPY packages/external-provider-contract/src/sensitive-field-policy.mjs ./packages/external-provider-contract/src/sensitive-field-policy.mjs
|
||||
COPY services/external-data-plane/package.json services/external-data-plane/package-lock.json ./services/external-data-plane/
|
||||
|
||||
WORKDIR /workspace/services/external-data-plane
|
||||
|
|
@ -21,8 +26,8 @@ COPY --from=deps /workspace/packages/external-provider-contract /packages/extern
|
|||
COPY services/external-data-plane/src ./src
|
||||
COPY services/external-data-plane/definitions ./definitions
|
||||
|
||||
# A dedicated numeric identity can read only the EDP provisioning secret mount;
|
||||
# it is intentionally not the shared Node/Map Gateway uid/gid (1000).
|
||||
# A dedicated numeric identity owns the legacy EDP bearer boundary and reads the
|
||||
# separate public Engine trust mount; it is not the shared service uid/gid.
|
||||
RUN addgroup -S -g 11006 nodedc-edp && adduser -S -D -H -u 11006 -G nodedc-edp nodedc-edp
|
||||
|
||||
USER 11006:11006
|
||||
|
|
|
|||
|
|
@ -6,9 +6,10 @@
|
|||
"scripts": {
|
||||
"start": "node src/server.mjs",
|
||||
"dev": "node --watch src/server.mjs",
|
||||
"check": "node --check src/server.mjs && node --check src/schema.mjs && node --check src/config.mjs && node --check src/intake-policy.mjs && node --check src/writer-binding.mjs && node --check src/reader-binding.mjs && node --check src/data-product-policy.mjs && node --check src/data-product-delivery.mjs && node --check src/definitions.mjs",
|
||||
"test": "node test/config.test.mjs && node test/intake-policy.test.mjs && node test/writer-binding.test.mjs && node test/reader-binding.test.mjs && node test/data-product-policy.test.mjs && node test/definitions.test.mjs",
|
||||
"test:integration": "node test/data-product-delivery.integration.test.mjs && node test/api.integration.test.mjs"
|
||||
"check": "node --check src/server.mjs && node --check src/schema.mjs && node --check src/config.mjs && node --check src/intake-policy.mjs && node --check src/writer-binding.mjs && node --check src/reader-binding.mjs && node --check src/data-product-policy.mjs && node --check src/data-product-delivery.mjs && node --check src/definitions.mjs && node --check src/managed-provisioner-auth.mjs",
|
||||
"test": "node test/config.test.mjs && node test/managed-provisioner-auth.test.mjs && node test/intake-policy.test.mjs && node test/writer-binding.test.mjs && node test/reader-binding.test.mjs && node test/data-product-policy.test.mjs && node test/definitions.test.mjs",
|
||||
"test:integration": "node test/data-product-delivery.integration.test.mjs && node test/api.integration.test.mjs",
|
||||
"test:all": "npm test && npm run test:integration"
|
||||
},
|
||||
"dependencies": {
|
||||
"@nodedc/external-provider-contract": "file:../../packages/external-provider-contract",
|
||||
|
|
|
|||
|
|
@ -1,14 +1,54 @@
|
|||
import { readFileSync } from "node:fs";
|
||||
import {
|
||||
loadManagedProvisionerPublicKeyFile,
|
||||
validateManagedProvisionerAudience,
|
||||
validateManagedProvisionerIdentity,
|
||||
} from "./managed-provisioner-auth.mjs";
|
||||
|
||||
export function readConfig(env = process.env) {
|
||||
const provisionerApiEnabled = boolean(env.EXTERNAL_DATA_PLANE_PROVISIONING_ENABLED, false);
|
||||
const managedProvisionerApiEnabled = boolean(env.EXTERNAL_DATA_PLANE_MANAGED_PROVISIONING_ENABLED, false);
|
||||
const managedProvisionerMaxSkewSeconds = integer(
|
||||
env.EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_MAX_SKEW_SECONDS,
|
||||
60,
|
||||
5,
|
||||
300,
|
||||
);
|
||||
const config = {
|
||||
port: integer(env.PORT, 18106, 1, 65535),
|
||||
databaseUrl: required(env.EXTERNAL_DATA_PLANE_DATABASE_URL, "EXTERNAL_DATA_PLANE_DATABASE_URL"),
|
||||
databasePoolSize: integer(env.EXTERNAL_DATA_PLANE_DATABASE_POOL_SIZE, 10, 1, 50),
|
||||
internalAccessToken: optional(env.NODEDC_INTERNAL_ACCESS_TOKEN),
|
||||
provisionerApiEnabled,
|
||||
managedProvisionerApiEnabled,
|
||||
provisionerAccessToken: provisionerApiEnabled ? secretFile(env.EXTERNAL_DATA_PLANE_PROVISIONER_TOKEN_FILE) : "",
|
||||
managedProvisionerPublicKey: managedProvisionerApiEnabled
|
||||
? loadManagedProvisionerPublicKeyFile(env.EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_PUBLIC_KEY_FILE)
|
||||
: null,
|
||||
managedProvisionerServiceId: managedProvisionerApiEnabled
|
||||
? validateManagedProvisionerIdentity(
|
||||
required(env.EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_SERVICE_ID, "EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_SERVICE_ID"),
|
||||
"service_id",
|
||||
)
|
||||
: "",
|
||||
managedProvisionerKeyId: managedProvisionerApiEnabled
|
||||
? validateManagedProvisionerIdentity(
|
||||
required(env.EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_KEY_ID, "EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_KEY_ID"),
|
||||
"key_id",
|
||||
)
|
||||
: "",
|
||||
managedProvisionerAudience: managedProvisionerApiEnabled
|
||||
? validateManagedProvisionerAudience(
|
||||
required(env.EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_AUDIENCE, "EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_AUDIENCE"),
|
||||
)
|
||||
: "",
|
||||
managedProvisionerMaxSkewMs: managedProvisionerMaxSkewSeconds * 1000,
|
||||
managedProvisionerReplayCacheMaxEntries: integer(
|
||||
env.EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_REPLAY_CACHE_MAX_ENTRIES,
|
||||
10_000,
|
||||
100,
|
||||
100_000,
|
||||
),
|
||||
rawRetentionDays: integer(env.EXTERNAL_DATA_PLANE_RAW_RETENTION_DAYS, 14, 1, 3650),
|
||||
maxBatchBytes: integer(env.EXTERNAL_DATA_PLANE_MAX_BATCH_BYTES, 5 * 1024 * 1024, 1024, 50 * 1024 * 1024),
|
||||
maxFactsPerPublish: integer(env.EXTERNAL_DATA_PLANE_MAX_FACTS_PER_PUBLISH, 5000, 1, 100_000),
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@ import { createHash, randomUUID } from "node:crypto";
|
|||
import {
|
||||
DATA_PRODUCT_PATCH_SCHEMA_VERSION,
|
||||
DATA_PRODUCT_SNAPSHOT_SCHEMA_VERSION,
|
||||
} from "@nodedc/external-provider-contract";
|
||||
} from "@nodedc/external-provider-contract/data-plane";
|
||||
|
||||
export async function loadDataProductDefinition(db, dataProductId, { activeOnly = true } = {}) {
|
||||
const result = await db.query(
|
||||
|
|
|
|||
|
|
@ -0,0 +1,318 @@
|
|||
import { createHash, createPublicKey, timingSafeEqual, verify as verifySignature } from "node:crypto";
|
||||
import { closeSync, constants as fsConstants, fstatSync, lstatSync, openSync, readFileSync } from "node:fs";
|
||||
import { isAbsolute } from "node:path";
|
||||
|
||||
export const MANAGED_PROVISIONER_SIGNATURE_SCHEMA = "nodedc.external-data-plane.managed-provisioner-request/v1";
|
||||
|
||||
export const MANAGED_PROVISIONER_HEADERS = Object.freeze({
|
||||
serviceId: "x-nodedc-engine-service-id",
|
||||
keyId: "x-nodedc-engine-key-id",
|
||||
audience: "x-nodedc-request-audience",
|
||||
timestamp: "x-nodedc-request-timestamp",
|
||||
nonce: "x-nodedc-request-nonce",
|
||||
bodySha256: "x-nodedc-content-sha256",
|
||||
signature: "x-nodedc-request-signature",
|
||||
});
|
||||
|
||||
const ISO_TIMESTAMP = /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z$/;
|
||||
const BASE64URL = /^[A-Za-z0-9_-]+$/;
|
||||
const IDENTITY = /^[a-z][a-z0-9._:-]{2,127}$/i;
|
||||
const AUDIENCE = /^[a-z][a-z0-9._:/-]{2,255}$/i;
|
||||
|
||||
export function loadManagedProvisionerPublicKeyFile(pathValue) {
|
||||
const path = String(pathValue ?? "").trim();
|
||||
if (!path) throw configError("external_data_plane_managed_provisioner_public_key_file_required");
|
||||
if (!isAbsolute(path)) throw configError("external_data_plane_managed_provisioner_public_key_file_must_be_absolute");
|
||||
|
||||
let stat;
|
||||
try {
|
||||
stat = lstatSync(path);
|
||||
} catch {
|
||||
throw configError("external_data_plane_managed_provisioner_public_key_file_unreadable");
|
||||
}
|
||||
if (!stat.isFile() || stat.isSymbolicLink()) {
|
||||
throw configError("external_data_plane_managed_provisioner_public_key_file_must_be_regular");
|
||||
}
|
||||
if ((stat.mode & 0o022) !== 0) {
|
||||
throw configError("external_data_plane_managed_provisioner_public_key_file_permissions_invalid");
|
||||
}
|
||||
if (stat.size < 1 || stat.size > 8192) {
|
||||
throw configError("external_data_plane_managed_provisioner_public_key_file_size_invalid");
|
||||
}
|
||||
|
||||
let descriptor;
|
||||
try {
|
||||
descriptor = openSync(path, fsConstants.O_RDONLY | fsConstants.O_NOFOLLOW);
|
||||
} catch {
|
||||
throw configError("external_data_plane_managed_provisioner_public_key_file_unreadable");
|
||||
}
|
||||
let pem;
|
||||
try {
|
||||
const openedStat = fstatSync(descriptor);
|
||||
if (!openedStat.isFile()) {
|
||||
throw configError("external_data_plane_managed_provisioner_public_key_file_must_be_regular");
|
||||
}
|
||||
if ((openedStat.mode & 0o022) !== 0) {
|
||||
throw configError("external_data_plane_managed_provisioner_public_key_file_permissions_invalid");
|
||||
}
|
||||
if (openedStat.size < 1 || openedStat.size > 8192) {
|
||||
throw configError("external_data_plane_managed_provisioner_public_key_file_size_invalid");
|
||||
}
|
||||
pem = readFileSync(descriptor, "utf8");
|
||||
} catch (error) {
|
||||
if (String(error?.code || "").startsWith("external_data_plane_managed_provisioner_")) throw error;
|
||||
throw configError("external_data_plane_managed_provisioner_public_key_file_unreadable");
|
||||
} finally {
|
||||
closeSync(descriptor);
|
||||
}
|
||||
const normalizedPem = pem.replace(/\r\n/g, "\n").trim();
|
||||
if (!/^-----BEGIN PUBLIC KEY-----\n[A-Za-z0-9+/=\n]+\n-----END PUBLIC KEY-----$/.test(normalizedPem)) {
|
||||
throw configError("external_data_plane_managed_provisioner_public_key_file_format_invalid");
|
||||
}
|
||||
|
||||
let publicKey;
|
||||
try {
|
||||
publicKey = createPublicKey(normalizedPem);
|
||||
} catch {
|
||||
throw configError("external_data_plane_managed_provisioner_public_key_file_format_invalid");
|
||||
}
|
||||
if (publicKey.type !== "public" || publicKey.asymmetricKeyType !== "ed25519") {
|
||||
throw configError("external_data_plane_managed_provisioner_public_key_algorithm_invalid");
|
||||
}
|
||||
return publicKey;
|
||||
}
|
||||
|
||||
export function validateManagedProvisionerIdentity(value, name) {
|
||||
const normalized = String(value ?? "").trim();
|
||||
if (!IDENTITY.test(normalized)) throw configError(`external_data_plane_managed_provisioner_${name}_invalid`);
|
||||
return normalized;
|
||||
}
|
||||
|
||||
export function validateManagedProvisionerAudience(value) {
|
||||
const normalized = String(value ?? "").trim();
|
||||
if (!AUDIENCE.test(normalized)) throw configError("external_data_plane_managed_provisioner_audience_invalid");
|
||||
return normalized;
|
||||
}
|
||||
|
||||
export function managedProvisionerSigningPayload({
|
||||
audience,
|
||||
serviceId,
|
||||
keyId,
|
||||
method,
|
||||
path,
|
||||
timestamp,
|
||||
nonce,
|
||||
bodySha256,
|
||||
}) {
|
||||
return JSON.stringify({
|
||||
schemaVersion: MANAGED_PROVISIONER_SIGNATURE_SCHEMA,
|
||||
audience,
|
||||
serviceId,
|
||||
keyId,
|
||||
method,
|
||||
path,
|
||||
timestamp,
|
||||
nonce,
|
||||
bodySha256,
|
||||
});
|
||||
}
|
||||
|
||||
export function sha256RawBody(rawBody) {
|
||||
if (!Buffer.isBuffer(rawBody)) throw new TypeError("raw_body_buffer_required");
|
||||
return createHash("sha256").update(rawBody).digest("hex");
|
||||
}
|
||||
|
||||
export class ManagedProvisionerReplayCache {
|
||||
constructor({ maxEntries }) {
|
||||
if (!Number.isInteger(maxEntries) || maxEntries < 1) throw new TypeError("replay_cache_max_entries_invalid");
|
||||
this.maxEntries = maxEntries;
|
||||
this.entries = new Map();
|
||||
}
|
||||
|
||||
consume(key, expiresAt, now = Date.now()) {
|
||||
this.prune(now);
|
||||
const existing = this.entries.get(key);
|
||||
if (existing !== undefined && existing > now) {
|
||||
throw authError(409, "managed_provisioner_request_replayed");
|
||||
}
|
||||
if (this.entries.size >= this.maxEntries) {
|
||||
throw authError(503, "managed_provisioner_replay_cache_exhausted");
|
||||
}
|
||||
this.entries.set(key, expiresAt);
|
||||
}
|
||||
|
||||
prune(now = Date.now()) {
|
||||
for (const [key, expiresAt] of this.entries) {
|
||||
if (expiresAt <= now) this.entries.delete(key);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export function createManagedProvisionerRequestVerifier({
|
||||
publicKey,
|
||||
serviceId,
|
||||
keyId,
|
||||
audience,
|
||||
maxSkewMs,
|
||||
replayCache,
|
||||
now = Date.now,
|
||||
}) {
|
||||
if (publicKey?.type !== "public" || publicKey?.asymmetricKeyType !== "ed25519") {
|
||||
throw configError("external_data_plane_managed_provisioner_public_key_algorithm_invalid");
|
||||
}
|
||||
if (!IDENTITY.test(serviceId)) throw configError("external_data_plane_managed_provisioner_service_id_invalid");
|
||||
if (!IDENTITY.test(keyId)) throw configError("external_data_plane_managed_provisioner_key_id_invalid");
|
||||
if (!AUDIENCE.test(audience)) throw configError("external_data_plane_managed_provisioner_audience_invalid");
|
||||
if (!Number.isInteger(maxSkewMs) || maxSkewMs < 1) {
|
||||
throw configError("external_data_plane_managed_provisioner_max_skew_invalid");
|
||||
}
|
||||
if (!(replayCache instanceof ManagedProvisionerReplayCache)) {
|
||||
throw configError("external_data_plane_managed_provisioner_replay_cache_invalid");
|
||||
}
|
||||
|
||||
return function verifyManagedProvisionerRequest(req) {
|
||||
const requestServiceId = exactHeader(req, MANAGED_PROVISIONER_HEADERS.serviceId);
|
||||
const requestKeyId = exactHeader(req, MANAGED_PROVISIONER_HEADERS.keyId);
|
||||
const requestAudience = exactHeader(req, MANAGED_PROVISIONER_HEADERS.audience);
|
||||
const timestamp = exactHeader(req, MANAGED_PROVISIONER_HEADERS.timestamp);
|
||||
const nonce = exactHeader(req, MANAGED_PROVISIONER_HEADERS.nonce);
|
||||
const declaredBodySha256 = exactHeader(req, MANAGED_PROVISIONER_HEADERS.bodySha256);
|
||||
const encodedSignature = exactHeader(req, MANAGED_PROVISIONER_HEADERS.signature);
|
||||
if (hasHeader(req, "authorization")) {
|
||||
throw authError(401, "managed_provisioner_authorization_header_forbidden");
|
||||
}
|
||||
|
||||
if (!safeEqualText(requestServiceId, serviceId)
|
||||
|| !safeEqualText(requestKeyId, keyId)
|
||||
|| !safeEqualText(requestAudience, audience)) {
|
||||
throw authError(401, "managed_provisioner_identity_mismatch");
|
||||
}
|
||||
|
||||
const timestampMs = parseTimestamp(timestamp);
|
||||
const verificationTime = Number(now());
|
||||
if (!Number.isFinite(verificationTime) || Math.abs(verificationTime - timestampMs) >= maxSkewMs) {
|
||||
throw authError(401, "managed_provisioner_timestamp_outside_window");
|
||||
}
|
||||
const nonceBytes = decodeCanonicalBase64Url(nonce, null, "managed_provisioner_nonce_invalid");
|
||||
if (nonceBytes.length < 16 || nonceBytes.length > 96) throw authError(401, "managed_provisioner_nonce_invalid");
|
||||
if (!/^[a-f0-9]{64}$/.test(declaredBodySha256)) {
|
||||
throw authError(401, "managed_provisioner_body_hash_invalid");
|
||||
}
|
||||
|
||||
const method = exactMethod(req);
|
||||
const path = exactPath(req);
|
||||
const rawBody = exactRawBody(req);
|
||||
const actualBodySha256 = sha256RawBody(rawBody);
|
||||
if (!safeEqualText(declaredBodySha256, actualBodySha256)) {
|
||||
throw authError(401, "managed_provisioner_body_hash_mismatch");
|
||||
}
|
||||
|
||||
const signature = decodeCanonicalBase64Url(encodedSignature, 64, "managed_provisioner_signature_invalid");
|
||||
const payload = managedProvisionerSigningPayload({
|
||||
audience: requestAudience,
|
||||
serviceId: requestServiceId,
|
||||
keyId: requestKeyId,
|
||||
method,
|
||||
path,
|
||||
timestamp,
|
||||
nonce,
|
||||
bodySha256: declaredBodySha256,
|
||||
});
|
||||
if (!verifySignature(null, Buffer.from(payload, "utf8"), publicKey, signature)) {
|
||||
throw authError(401, "managed_provisioner_signature_invalid");
|
||||
}
|
||||
|
||||
const replayKey = `${requestServiceId}\0${requestKeyId}\0${nonce}`;
|
||||
replayCache.consume(replayKey, timestampMs + maxSkewMs, verificationTime);
|
||||
return Object.freeze({ serviceId: requestServiceId, keyId: requestKeyId });
|
||||
};
|
||||
}
|
||||
|
||||
function exactHeader(req, name) {
|
||||
const rawHeaders = req?.rawHeaders;
|
||||
if (Array.isArray(rawHeaders)) {
|
||||
const values = [];
|
||||
for (let index = 0; index < rawHeaders.length; index += 2) {
|
||||
if (String(rawHeaders[index]).toLowerCase() === name) values.push(rawHeaders[index + 1]);
|
||||
}
|
||||
if (values.length !== 1 || typeof values[0] !== "string" || !values[0]) {
|
||||
throw authError(401, "managed_provisioner_header_invalid");
|
||||
}
|
||||
return values[0];
|
||||
}
|
||||
|
||||
const value = req?.headers?.[name];
|
||||
if (typeof value !== "string" || !value) throw authError(401, "managed_provisioner_header_invalid");
|
||||
return value;
|
||||
}
|
||||
|
||||
function hasHeader(req, name) {
|
||||
if (Array.isArray(req?.rawHeaders)) {
|
||||
for (let index = 0; index < req.rawHeaders.length; index += 2) {
|
||||
if (String(req.rawHeaders[index]).toLowerCase() === name) return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
return Object.hasOwn(req?.headers || {}, name);
|
||||
}
|
||||
|
||||
function exactMethod(req) {
|
||||
const method = String(req?.method ?? "");
|
||||
if (!/^[A-Z]{3,10}$/.test(method)) throw authError(401, "managed_provisioner_method_invalid");
|
||||
return method;
|
||||
}
|
||||
|
||||
function exactPath(req) {
|
||||
const path = String(req?.originalUrl ?? "");
|
||||
if (!path.startsWith("/") || path.length > 2048 || /[\u0000-\u0020#]/.test(path)) {
|
||||
throw authError(401, "managed_provisioner_path_invalid");
|
||||
}
|
||||
return path;
|
||||
}
|
||||
|
||||
function exactRawBody(req) {
|
||||
if (req?.rawBodyCaptured === true && Buffer.isBuffer(req.rawBody)) return req.rawBody;
|
||||
const contentLength = req?.headers?.["content-length"];
|
||||
const transferEncoding = req?.headers?.["transfer-encoding"];
|
||||
if ((contentLength === undefined || contentLength === "0") && transferEncoding === undefined) {
|
||||
return Buffer.alloc(0);
|
||||
}
|
||||
throw authError(401, "managed_provisioner_raw_body_unavailable");
|
||||
}
|
||||
|
||||
function parseTimestamp(value) {
|
||||
if (!ISO_TIMESTAMP.test(value)) throw authError(401, "managed_provisioner_timestamp_invalid");
|
||||
const timestampMs = Date.parse(value);
|
||||
if (!Number.isFinite(timestampMs) || new Date(timestampMs).toISOString() !== value) {
|
||||
throw authError(401, "managed_provisioner_timestamp_invalid");
|
||||
}
|
||||
return timestampMs;
|
||||
}
|
||||
|
||||
function decodeCanonicalBase64Url(value, expectedBytes, errorCode) {
|
||||
if (!BASE64URL.test(value)) throw authError(401, errorCode);
|
||||
let decoded;
|
||||
try {
|
||||
decoded = Buffer.from(value, "base64url");
|
||||
} catch {
|
||||
throw authError(401, errorCode);
|
||||
}
|
||||
if ((expectedBytes !== null && decoded.length !== expectedBytes) || decoded.toString("base64url") !== value) {
|
||||
throw authError(401, errorCode);
|
||||
}
|
||||
return decoded;
|
||||
}
|
||||
|
||||
function safeEqualText(left, right) {
|
||||
const leftBuffer = Buffer.from(String(left), "utf8");
|
||||
const rightBuffer = Buffer.from(String(right), "utf8");
|
||||
return leftBuffer.length === rightBuffer.length && timingSafeEqual(leftBuffer, rightBuffer);
|
||||
}
|
||||
|
||||
function authError(status, code) {
|
||||
return Object.assign(new Error(code), { status, code });
|
||||
}
|
||||
|
||||
function configError(code) {
|
||||
return Object.assign(new Error(code), { code });
|
||||
}
|
||||
|
|
@ -65,6 +65,9 @@ export async function migrate(pool) {
|
|||
create table if not exists external_data_plane_writer_bindings (
|
||||
id uuid primary key,
|
||||
token_hash text not null unique,
|
||||
binding_key text,
|
||||
request_hash text,
|
||||
generation integer not null default 1,
|
||||
tenant_id text not null,
|
||||
connection_id text not null,
|
||||
provider_id text not null,
|
||||
|
|
@ -74,6 +77,7 @@ export async function migrate(pool) {
|
|||
created_at timestamptz not null default now(),
|
||||
rotated_at timestamptz,
|
||||
revoked_at timestamptz,
|
||||
check (generation > 0),
|
||||
check (
|
||||
case when jsonb_typeof(allowed_data_product_ids) = 'array'
|
||||
then jsonb_array_length(allowed_data_product_ids) > 0
|
||||
|
|
@ -82,6 +86,37 @@ export async function migrate(pool) {
|
|||
)
|
||||
)
|
||||
`);
|
||||
await pool.query("alter table external_data_plane_writer_bindings add column if not exists binding_key text");
|
||||
await pool.query("alter table external_data_plane_writer_bindings add column if not exists request_hash text");
|
||||
await pool.query("alter table external_data_plane_writer_bindings add column if not exists generation integer not null default 1");
|
||||
await pool.query(`
|
||||
do $$
|
||||
begin
|
||||
if not exists (
|
||||
select 1 from pg_constraint
|
||||
where conrelid = 'external_data_plane_writer_bindings'::regclass
|
||||
and conname = 'external_data_plane_writer_bindings_generation_positive_ck'
|
||||
) then
|
||||
alter table external_data_plane_writer_bindings
|
||||
add constraint external_data_plane_writer_bindings_generation_positive_ck
|
||||
check (generation > 0);
|
||||
end if;
|
||||
if not exists (
|
||||
select 1 from pg_constraint
|
||||
where conrelid = 'external_data_plane_writer_bindings'::regclass
|
||||
and conname = 'external_data_plane_writer_bindings_managed_metadata_ck'
|
||||
) then
|
||||
alter table external_data_plane_writer_bindings
|
||||
add constraint external_data_plane_writer_bindings_managed_metadata_ck
|
||||
check (
|
||||
(binding_key is null and request_hash is null)
|
||||
or (binding_key is not null and request_hash ~ '^[a-f0-9]{64}$')
|
||||
);
|
||||
end if;
|
||||
end
|
||||
$$
|
||||
`);
|
||||
await pool.query("create unique index if not exists external_data_plane_writer_bindings_managed_key_idx on external_data_plane_writer_bindings (binding_key, generation) where binding_key is not null");
|
||||
await pool.query("create index if not exists external_data_plane_writer_bindings_active_idx on external_data_plane_writer_bindings (active, expires_at)");
|
||||
|
||||
await pool.query(`
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@ import express from "express";
|
|||
import { createHash, randomUUID, timingSafeEqual } from "node:crypto";
|
||||
import { createServer } from "node:http";
|
||||
import { Pool } from "pg";
|
||||
import { validateDataProductPublish, validateIntakeBatch } from "@nodedc/external-provider-contract";
|
||||
import { validateDataProductPublish, validateIntakeBatch } from "@nodedc/external-provider-contract/data-plane";
|
||||
import { readConfig } from "./config.mjs";
|
||||
import {
|
||||
loadDataProductDefinition,
|
||||
|
|
@ -18,6 +18,10 @@ import {
|
|||
import { normalizeDataProductDefinition, safeDataProductDefinition } from "./data-product-policy.mjs";
|
||||
import { reconcileDataProductDefinitions } from "./definitions.mjs";
|
||||
import { assertBatchTimeBounds, rawRetentionExpiry } from "./intake-policy.mjs";
|
||||
import {
|
||||
createManagedProvisionerRequestVerifier,
|
||||
ManagedProvisionerReplayCache,
|
||||
} from "./managed-provisioner-auth.mjs";
|
||||
import {
|
||||
assertReaderProduct,
|
||||
createReaderToken,
|
||||
|
|
@ -31,11 +35,26 @@ import {
|
|||
hashWriterToken,
|
||||
materializeDataProductPublish,
|
||||
materializeWriterBoundBatch,
|
||||
normalizeManagedWriterBindingRequest,
|
||||
normalizeWriterBindingRequest,
|
||||
safeWriterBinding,
|
||||
writerBindingRequestHash,
|
||||
} from "./writer-binding.mjs";
|
||||
|
||||
const config = readConfig();
|
||||
const managedProvisionerReplayCache = config.managedProvisionerApiEnabled
|
||||
? new ManagedProvisionerReplayCache({ maxEntries: config.managedProvisionerReplayCacheMaxEntries })
|
||||
: null;
|
||||
const verifyManagedProvisionerRequest = config.managedProvisionerApiEnabled
|
||||
? createManagedProvisionerRequestVerifier({
|
||||
publicKey: config.managedProvisionerPublicKey,
|
||||
serviceId: config.managedProvisionerServiceId,
|
||||
keyId: config.managedProvisionerKeyId,
|
||||
audience: config.managedProvisionerAudience,
|
||||
maxSkewMs: config.managedProvisionerMaxSkewMs,
|
||||
replayCache: managedProvisionerReplayCache,
|
||||
})
|
||||
: null;
|
||||
const pool = new Pool({ connectionString: config.databaseUrl, max: config.databasePoolSize });
|
||||
const app = express();
|
||||
const httpServer = createServer(app);
|
||||
|
|
@ -47,7 +66,19 @@ let shuttingDown = false;
|
|||
let shutdownPromise = null;
|
||||
|
||||
app.disable("x-powered-by");
|
||||
app.use(express.json({ limit: config.maxBatchBytes }));
|
||||
app.use((req, _res, next) => {
|
||||
req.rawBody = Buffer.alloc(0);
|
||||
req.rawBodyCaptured = false;
|
||||
next();
|
||||
});
|
||||
app.use(express.json({
|
||||
limit: config.maxBatchBytes,
|
||||
inflate: false,
|
||||
verify(req, _res, buffer) {
|
||||
req.rawBody = Buffer.from(buffer);
|
||||
req.rawBodyCaptured = true;
|
||||
},
|
||||
}));
|
||||
|
||||
app.get("/healthz", asyncRoute(async (_req, res) => {
|
||||
await pool.query("select 1");
|
||||
|
|
@ -59,10 +90,12 @@ app.get("/healthz", asyncRoute(async (_req, res) => {
|
|||
providerLogic: "absent",
|
||||
commandTransport: "absent",
|
||||
writerBindings: "supported",
|
||||
managedWriterBindings: "digest+idempotent-generation",
|
||||
readerBindings: "supported",
|
||||
dataProductDelivery: "snapshot+durable-patch",
|
||||
legacyIntake: config.legacyIntakeEnabled ? "migration-only" : "disabled",
|
||||
writerBindingProvisioning: config.provisionerApiEnabled ? "enabled" : "disabled",
|
||||
legacyCapabilityProvisioning: config.provisionerApiEnabled ? "enabled" : "disabled",
|
||||
managedWriterBindingProvisioning: config.managedProvisionerApiEnabled ? "enabled" : "disabled",
|
||||
rawRetentionSweep: {
|
||||
mode: "server-scheduled",
|
||||
lastSweepAt: lastRetentionSweepAt,
|
||||
|
|
@ -134,6 +167,128 @@ app.post("/internal/data-plane/v1/intake/writer-bound", requireLegacyIntake, req
|
|||
res.status(result.idempotent ? 200 : 201).json({ ok: true, ...result });
|
||||
}));
|
||||
|
||||
app.put("/internal/data-plane/v1/writer-bindings/by-key/:bindingKey", requireManagedProvisionerApi, asyncRoute(async (req, res) => {
|
||||
const bindingKey = requireIdentifier(req.params.bindingKey, "managed_writer_binding_key_invalid");
|
||||
const policy = normalizeManagedWriterBindingRequest(req.body, {
|
||||
maxTtlDays: config.writerBindingMaxTtlDays,
|
||||
});
|
||||
await assertRegisteredProductIds(policy.allowedDataProductIds);
|
||||
const requestHash = writerBindingRequestHash(policy);
|
||||
const client = await pool.connect();
|
||||
let response;
|
||||
try {
|
||||
await client.query("begin");
|
||||
await client.query("select pg_advisory_xact_lock(hashtextextended($1, 0))", [bindingKey]);
|
||||
const existing = await client.query(
|
||||
`select id, binding_key as "bindingKey", request_hash as "requestHash", generation,
|
||||
tenant_id as "tenantId", connection_id as "connectionId", provider_id as "providerId",
|
||||
allowed_data_product_ids as "allowedDataProductIds", expires_at as "expiresAt",
|
||||
active, created_at as "createdAt", rotated_at as "rotatedAt", revoked_at as "revokedAt"
|
||||
from external_data_plane_writer_bindings
|
||||
where binding_key = $1 and generation = $2
|
||||
for update`,
|
||||
[bindingKey, policy.generation],
|
||||
);
|
||||
if (existing.rowCount) {
|
||||
const binding = existing.rows[0];
|
||||
if (binding.requestHash !== requestHash) {
|
||||
throw httpError(409, "managed_writer_binding_request_conflict");
|
||||
}
|
||||
if (binding.active !== true || new Date(binding.expiresAt) <= new Date()) {
|
||||
throw httpError(409, "managed_writer_binding_generation_inactive");
|
||||
}
|
||||
response = { status: 200, idempotent: true, binding };
|
||||
} else {
|
||||
const inserted = await client.query(
|
||||
`insert into external_data_plane_writer_bindings (
|
||||
id, token_hash, binding_key, request_hash, generation,
|
||||
tenant_id, connection_id, provider_id, allowed_data_product_ids, expires_at
|
||||
) values ($1, $2, $3, $4, $5, $6, $7, $8, $9::jsonb, $10)
|
||||
returning id, binding_key as "bindingKey", generation,
|
||||
tenant_id as "tenantId", connection_id as "connectionId", provider_id as "providerId",
|
||||
allowed_data_product_ids as "allowedDataProductIds", expires_at as "expiresAt",
|
||||
active, created_at as "createdAt", rotated_at as "rotatedAt", revoked_at as "revokedAt"`,
|
||||
[
|
||||
randomUUID(),
|
||||
policy.capabilityDigest,
|
||||
bindingKey,
|
||||
requestHash,
|
||||
policy.generation,
|
||||
policy.tenantId,
|
||||
policy.connectionId,
|
||||
policy.providerId,
|
||||
JSON.stringify(policy.allowedDataProductIds),
|
||||
policy.expiresAt,
|
||||
],
|
||||
);
|
||||
response = { status: 201, idempotent: false, binding: inserted.rows[0] };
|
||||
}
|
||||
await client.query("commit");
|
||||
} catch (error) {
|
||||
await client.query("rollback");
|
||||
if (error?.code === "23505") {
|
||||
throw httpError(409, "managed_writer_binding_capability_digest_conflict");
|
||||
}
|
||||
throw error;
|
||||
} finally {
|
||||
client.release();
|
||||
}
|
||||
res.set("Cache-Control", "no-store, max-age=0");
|
||||
res.status(response.status).json({
|
||||
ok: true,
|
||||
idempotent: response.idempotent,
|
||||
writerBinding: safeWriterBinding(response.binding),
|
||||
});
|
||||
}));
|
||||
|
||||
app.post("/internal/data-plane/v1/writer-bindings/by-key/:bindingKey/generations/:generation/revoke", requireManagedProvisionerApi, asyncRoute(async (req, res) => {
|
||||
const bindingKey = requireIdentifier(req.params.bindingKey, "managed_writer_binding_key_invalid");
|
||||
const generation = requirePositiveInteger(req.params.generation, "managed_writer_binding_generation_invalid");
|
||||
const client = await pool.connect();
|
||||
let binding;
|
||||
let idempotent;
|
||||
try {
|
||||
await client.query("begin");
|
||||
await client.query("select pg_advisory_xact_lock(hashtextextended($1, 0))", [bindingKey]);
|
||||
const existing = await client.query(
|
||||
`select id, binding_key as "bindingKey", generation,
|
||||
tenant_id as "tenantId", connection_id as "connectionId", provider_id as "providerId",
|
||||
allowed_data_product_ids as "allowedDataProductIds", expires_at as "expiresAt",
|
||||
active, created_at as "createdAt", rotated_at as "rotatedAt", revoked_at as "revokedAt"
|
||||
from external_data_plane_writer_bindings
|
||||
where binding_key = $1 and generation = $2
|
||||
for update`,
|
||||
[bindingKey, generation],
|
||||
);
|
||||
if (!existing.rowCount) throw httpError(404, "managed_writer_binding_not_found");
|
||||
if (existing.rows[0].active === true) {
|
||||
const revoked = await client.query(
|
||||
`update external_data_plane_writer_bindings
|
||||
set active = false, revoked_at = now()
|
||||
where binding_key = $1 and generation = $2 and active = true
|
||||
returning id, binding_key as "bindingKey", generation,
|
||||
tenant_id as "tenantId", connection_id as "connectionId", provider_id as "providerId",
|
||||
allowed_data_product_ids as "allowedDataProductIds", expires_at as "expiresAt",
|
||||
active, created_at as "createdAt", rotated_at as "rotatedAt", revoked_at as "revokedAt"`,
|
||||
[bindingKey, generation],
|
||||
);
|
||||
binding = revoked.rows[0];
|
||||
idempotent = false;
|
||||
} else {
|
||||
binding = existing.rows[0];
|
||||
idempotent = true;
|
||||
}
|
||||
await client.query("commit");
|
||||
} catch (error) {
|
||||
await client.query("rollback");
|
||||
throw error;
|
||||
} finally {
|
||||
client.release();
|
||||
}
|
||||
res.set("Cache-Control", "no-store, max-age=0");
|
||||
res.json({ ok: true, idempotent, writerBinding: safeWriterBinding(binding) });
|
||||
}));
|
||||
|
||||
app.post("/internal/data-plane/v1/writer-bindings", requireProvisionerApi, asyncRoute(async (req, res) => {
|
||||
const policy = normalizeWriterBindingRequest(req.body, {
|
||||
maxTtlDays: config.writerBindingMaxTtlDays,
|
||||
|
|
@ -172,7 +327,7 @@ app.post("/internal/data-plane/v1/writer-bindings/:bindingId/rotate", requirePro
|
|||
const result = await pool.query(
|
||||
`update external_data_plane_writer_bindings
|
||||
set token_hash = $2, rotated_at = now()
|
||||
where id = $1 and active = true and expires_at > now()
|
||||
where id = $1 and binding_key is null and active = true and expires_at > now()
|
||||
returning id, tenant_id as "tenantId", connection_id as "connectionId",
|
||||
provider_id as "providerId", allowed_data_product_ids as "allowedDataProductIds",
|
||||
expires_at as "expiresAt", active, created_at as "createdAt",
|
||||
|
|
@ -188,7 +343,7 @@ app.post("/internal/data-plane/v1/writer-bindings/:bindingId/revoke", requirePro
|
|||
const result = await pool.query(
|
||||
`update external_data_plane_writer_bindings
|
||||
set active = false, revoked_at = now()
|
||||
where id = $1 and active = true
|
||||
where id = $1 and binding_key is null and active = true
|
||||
returning id, tenant_id as "tenantId", connection_id as "connectionId",
|
||||
provider_id as "providerId", allowed_data_product_ids as "allowedDataProductIds",
|
||||
expires_at as "expiresAt", active, created_at as "createdAt",
|
||||
|
|
@ -564,6 +719,21 @@ function requireLegacyIntake(_req, _res, next) {
|
|||
|
||||
function requireProvisionerApi(req, _res, next) {
|
||||
if (!config.provisionerApiEnabled) return next(httpError(503, "provisioner_api_disabled"));
|
||||
return requireProvisionerCredential(req, next);
|
||||
}
|
||||
|
||||
function requireManagedProvisionerApi(req, _res, next) {
|
||||
if (!config.managedProvisionerApiEnabled) return next(httpError(503, "managed_provisioner_api_disabled"));
|
||||
if (!verifyManagedProvisionerRequest) return next(httpError(503, "managed_provisioner_api_not_configured"));
|
||||
try {
|
||||
req.managedProvisionerIdentity = verifyManagedProvisionerRequest(req);
|
||||
return next();
|
||||
} catch (error) {
|
||||
return next(error);
|
||||
}
|
||||
}
|
||||
|
||||
function requireProvisionerCredential(req, next) {
|
||||
if (!config.provisionerAccessToken) return next(httpError(503, "provisioner_api_not_configured"));
|
||||
const value = bearerToken(req);
|
||||
if (!value || !safeEqual(value, config.provisionerAccessToken)) return next(httpError(401, "provisioner_unauthorized"));
|
||||
|
|
@ -697,6 +867,14 @@ function requireIdentifier(value, code) {
|
|||
return normalized;
|
||||
}
|
||||
|
||||
function requirePositiveInteger(value, code) {
|
||||
const normalized = String(value ?? "");
|
||||
if (!/^[1-9]\d*$/.test(normalized)) throw httpError(400, code);
|
||||
const number = Number(normalized);
|
||||
if (!Number.isSafeInteger(number) || number > 2_147_483_647) throw httpError(400, code);
|
||||
return number;
|
||||
}
|
||||
|
||||
function parseCursor(value) {
|
||||
const normalized = String(value ?? "");
|
||||
if (!/^(?:0|[1-9]\d*)$/.test(normalized)) throw httpError(400, "data_product_cursor_invalid");
|
||||
|
|
|
|||
|
|
@ -4,7 +4,15 @@ const IDENTIFIER = /^[a-z][a-z0-9._:-]{2,127}$/;
|
|||
const TOKEN_PREFIX = "ndc_edpwb_";
|
||||
const SECRET_LIKE_KEY = /(token|secret|password|access[_-]?token|refresh[_-]?token|api[_-]?key)/i;
|
||||
const BINDING_REQUEST_KEYS = new Set(["source", "allowedDataProductIds", "expiresAt"]);
|
||||
const MANAGED_BINDING_REQUEST_KEYS = new Set([
|
||||
"source",
|
||||
"allowedDataProductIds",
|
||||
"expiresAt",
|
||||
"generation",
|
||||
"capabilityDigest",
|
||||
]);
|
||||
const BINDING_SOURCE_KEYS = new Set(["tenantId", "connectionId", "providerId"]);
|
||||
const SHA256_DIGEST = /^[a-f0-9]{64}$/;
|
||||
|
||||
/**
|
||||
* Produces an opaque, high-entropy capability. The plaintext is returned only
|
||||
|
|
@ -59,6 +67,57 @@ export function normalizeWriterBindingRequest(value, { now = new Date(), maxTtlD
|
|||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Validates the zero-touch control-plane form. The native credential secret is
|
||||
* generated and stored inside Engine; EDP receives only its SHA-256 digest.
|
||||
* `generation` makes a retry address the same immutable binding generation.
|
||||
*/
|
||||
export function normalizeManagedWriterBindingRequest(value, { now = new Date(), maxTtlDays = 90 } = {}) {
|
||||
if (!isPlainObject(value) || !isPlainObject(value.source)) {
|
||||
throw writerBindingError("managed_writer_binding_request_invalid");
|
||||
}
|
||||
if (containsSecretLikeKey(value)) {
|
||||
throw writerBindingError("managed_writer_binding_secret_material_forbidden");
|
||||
}
|
||||
if (!hasOnlyKeys(value, MANAGED_BINDING_REQUEST_KEYS) || !hasOnlyKeys(value.source, BINDING_SOURCE_KEYS)) {
|
||||
throw writerBindingError("managed_writer_binding_request_fields_invalid");
|
||||
}
|
||||
|
||||
const scope = normalizeWriterBindingRequest({
|
||||
source: value.source,
|
||||
allowedDataProductIds: value.allowedDataProductIds,
|
||||
expiresAt: value.expiresAt,
|
||||
}, { now, maxTtlDays });
|
||||
const generation = Number(value.generation);
|
||||
if (!Number.isSafeInteger(generation) || generation < 1 || generation > 2_147_483_647) {
|
||||
throw writerBindingError("managed_writer_binding_generation_invalid");
|
||||
}
|
||||
const capabilityDigest = String(value.capabilityDigest || "").toLowerCase();
|
||||
if (!SHA256_DIGEST.test(capabilityDigest)) {
|
||||
throw writerBindingError("managed_writer_binding_capability_digest_invalid");
|
||||
}
|
||||
|
||||
return Object.freeze({
|
||||
...scope,
|
||||
allowedDataProductIds: Object.freeze([...scope.allowedDataProductIds].sort()),
|
||||
generation,
|
||||
capabilityDigest,
|
||||
});
|
||||
}
|
||||
|
||||
export function writerBindingRequestHash(policy) {
|
||||
const canonical = JSON.stringify({
|
||||
tenantId: policy.tenantId,
|
||||
connectionId: policy.connectionId,
|
||||
providerId: policy.providerId,
|
||||
allowedDataProductIds: [...policy.allowedDataProductIds].sort(),
|
||||
expiresAt: new Date(policy.expiresAt).toISOString(),
|
||||
generation: policy.generation,
|
||||
capabilityDigest: policy.capabilityDigest,
|
||||
});
|
||||
return createHash("sha256").update(canonical, "utf8").digest("hex");
|
||||
}
|
||||
|
||||
/**
|
||||
* Converts a caller-provided, deliberately unscoped intake envelope into the
|
||||
* canonical scoped form. Caller scope is rejected, never trusted or merged.
|
||||
|
|
@ -136,6 +195,8 @@ export function materializeDataProductPublish(value, binding, definition, dataPr
|
|||
export function safeWriterBinding(binding) {
|
||||
return {
|
||||
id: binding.id,
|
||||
...(binding.bindingKey ? { bindingKey: binding.bindingKey } : {}),
|
||||
...(Number.isInteger(Number(binding.generation)) ? { generation: Number(binding.generation) } : {}),
|
||||
tenantId: binding.tenantId,
|
||||
connectionId: binding.connectionId,
|
||||
providerId: binding.providerId,
|
||||
|
|
|
|||
|
|
@ -1,10 +1,16 @@
|
|||
import assert from "node:assert/strict";
|
||||
import { spawn } from "node:child_process";
|
||||
import { createHash, generateKeyPairSync, randomBytes, sign } from "node:crypto";
|
||||
import { mkdtemp, rm, writeFile } from "node:fs/promises";
|
||||
import net from "node:net";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { validateDataProductPatch, validateDataProductSnapshot } from "@nodedc/external-provider-contract";
|
||||
import { validateDataProductPatch, validateDataProductSnapshot } from "@nodedc/external-provider-contract/data-plane";
|
||||
import {
|
||||
MANAGED_PROVISIONER_HEADERS,
|
||||
managedProvisionerSigningPayload,
|
||||
sha256RawBody,
|
||||
} from "../src/managed-provisioner-auth.mjs";
|
||||
|
||||
const databaseUrl = process.env.EXTERNAL_DATA_PLANE_TEST_DATABASE_URL;
|
||||
if (!databaseUrl) throw new Error("EXTERNAL_DATA_PLANE_TEST_DATABASE_URL_required");
|
||||
|
|
@ -13,7 +19,17 @@ const baseUrl = `http://127.0.0.1:${port}`;
|
|||
const directory = await mkdtemp(join(tmpdir(), "nodedc-edp-api-test-"));
|
||||
const provisionerSecret = "nodedc_edp_integration_provisioner_secret_0123456789abcdef";
|
||||
const secretPath = join(directory, "provisioner-token");
|
||||
const managedProvisionerServiceId = "nodedc-engine";
|
||||
const managedProvisionerKeyId = "engine-edp-managed-provisioner-v1";
|
||||
const managedProvisionerAudience = "nodedc-external-data-plane.managed-provisioning.v1";
|
||||
const managedProvisionerPublicKeyPath = join(directory, "engine-public-key.pem");
|
||||
const { privateKey: managedProvisionerPrivateKey, publicKey: managedProvisionerPublicKey } = generateKeyPairSync("ed25519");
|
||||
await writeFile(secretPath, `${provisionerSecret}\n`, { mode: 0o600 });
|
||||
await writeFile(
|
||||
managedProvisionerPublicKeyPath,
|
||||
managedProvisionerPublicKey.export({ type: "spki", format: "pem" }),
|
||||
{ mode: 0o600 },
|
||||
);
|
||||
|
||||
const child = spawn(process.execPath, ["src/server.mjs"], {
|
||||
cwd: new URL("..", import.meta.url),
|
||||
|
|
@ -22,7 +38,12 @@ const child = spawn(process.execPath, ["src/server.mjs"], {
|
|||
PORT: String(port),
|
||||
EXTERNAL_DATA_PLANE_DATABASE_URL: databaseUrl,
|
||||
EXTERNAL_DATA_PLANE_PROVISIONING_ENABLED: "true",
|
||||
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONING_ENABLED: "true",
|
||||
EXTERNAL_DATA_PLANE_PROVISIONER_TOKEN_FILE: secretPath,
|
||||
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_PUBLIC_KEY_FILE: managedProvisionerPublicKeyPath,
|
||||
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_SERVICE_ID: managedProvisionerServiceId,
|
||||
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_KEY_ID: managedProvisionerKeyId,
|
||||
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_AUDIENCE: managedProvisionerAudience,
|
||||
EXTERNAL_DATA_PLANE_RETENTION_SWEEP_MS: "60000",
|
||||
EXTERNAL_DATA_PLANE_STREAM_POLL_MS: "250",
|
||||
EXTERNAL_DATA_PLANE_STREAM_HEARTBEAT_MS: "5000",
|
||||
|
|
@ -52,10 +73,139 @@ try {
|
|||
|
||||
const expiry = new Date(Date.now() + 24 * 60 * 60 * 1000).toISOString();
|
||||
const scope = { tenantId: "api-tenant", connectionId: "api-connection", providerId: "api-provider" };
|
||||
const managedToken = "ndc_edpwb_managed_api_test_0123456789abcdefghijklmnopqrstuvwxyz";
|
||||
const managedBody = {
|
||||
source: scope,
|
||||
allowedDataProductIds: [productId],
|
||||
expiresAt: expiry,
|
||||
generation: 1,
|
||||
capabilityDigest: createHash("sha256").update(managedToken, "utf8").digest("hex"),
|
||||
};
|
||||
const managedPath = "/internal/data-plane/v1/writer-bindings/by-key/engine.api-connection.positions";
|
||||
const bearerOnlyManaged = await fetch(`${baseUrl}${managedPath}`, {
|
||||
method: "PUT",
|
||||
headers: { Authorization: `Bearer ${provisionerSecret}`, "Content-Type": "application/json" },
|
||||
body: JSON.stringify(managedBody),
|
||||
});
|
||||
assert.equal(bearerOnlyManaged.status, 401);
|
||||
assert.equal((await bearerOnlyManaged.json()).error, "managed_provisioner_header_invalid");
|
||||
|
||||
const managedResults = await Promise.all([
|
||||
rawJsonRequest(managedPath, {
|
||||
method: "PUT",
|
||||
managedSignature: true,
|
||||
body: managedBody,
|
||||
}),
|
||||
rawJsonRequest(managedPath, {
|
||||
method: "PUT",
|
||||
managedSignature: true,
|
||||
body: managedBody,
|
||||
}),
|
||||
]);
|
||||
assert.deepEqual(managedResults.map(({ value }) => value.idempotent).sort(), [false, true]);
|
||||
assert.equal(managedResults[0].value.writerBinding.id, managedResults[1].value.writerBinding.id);
|
||||
assert.equal("token" in managedResults[0].value, false);
|
||||
assert.equal(JSON.stringify(managedResults[0].value).includes(managedBody.capabilityDigest), false);
|
||||
const managedCatalog = await jsonRequest("/internal/data-plane/v1/writer/data-products", { token: managedToken });
|
||||
assert.deepEqual(managedCatalog.dataProducts.map((value) => value.id), [productId]);
|
||||
const replayBody = JSON.stringify(managedBody);
|
||||
const replayHeaders = signedManagedHeaders(managedPath, "PUT", Buffer.from(replayBody, "utf8"));
|
||||
const firstReplayAttempt = await fetch(`${baseUrl}${managedPath}`, {
|
||||
method: "PUT", headers: { ...replayHeaders, "Content-Type": "application/json" }, body: replayBody,
|
||||
});
|
||||
assert.equal(firstReplayAttempt.status, 200);
|
||||
const rejectedReplay = await fetch(`${baseUrl}${managedPath}`, {
|
||||
method: "PUT", headers: { ...replayHeaders, "Content-Type": "application/json" }, body: replayBody,
|
||||
});
|
||||
assert.equal(rejectedReplay.status, 409);
|
||||
assert.equal((await rejectedReplay.json()).error, "managed_provisioner_request_replayed");
|
||||
|
||||
const tamperHeaders = signedManagedHeaders(managedPath, "PUT", Buffer.from(replayBody, "utf8"));
|
||||
const tamperedBody = await fetch(`${baseUrl}${managedPath}`, {
|
||||
method: "PUT",
|
||||
headers: { ...tamperHeaders, "Content-Type": "application/json" },
|
||||
body: `${replayBody} `,
|
||||
});
|
||||
assert.equal(tamperedBody.status, 401);
|
||||
assert.equal((await tamperedBody.json()).error, "managed_provisioner_body_hash_mismatch");
|
||||
|
||||
const pathHeaders = signedManagedHeaders(managedPath, "PUT", Buffer.from(replayBody, "utf8"));
|
||||
const tamperedPath = await fetch(`${baseUrl}${managedPath}?unexpected=true`, {
|
||||
method: "PUT", headers: { ...pathHeaders, "Content-Type": "application/json" }, body: replayBody,
|
||||
});
|
||||
assert.equal(tamperedPath.status, 401);
|
||||
assert.equal((await tamperedPath.json()).error, "managed_provisioner_signature_invalid");
|
||||
|
||||
const audienceHeaders = signedManagedHeaders(managedPath, "PUT", Buffer.from(replayBody, "utf8"));
|
||||
audienceHeaders[MANAGED_PROVISIONER_HEADERS.audience] = "other-audience";
|
||||
const wrongAudience = await fetch(`${baseUrl}${managedPath}`, {
|
||||
method: "PUT", headers: { ...audienceHeaders, "Content-Type": "application/json" }, body: replayBody,
|
||||
});
|
||||
assert.equal(wrongAudience.status, 401);
|
||||
assert.equal((await wrongAudience.json()).error, "managed_provisioner_identity_mismatch");
|
||||
|
||||
const managedConflict = await signedFetch(managedPath, {
|
||||
method: "PUT",
|
||||
body: {
|
||||
...managedBody,
|
||||
capabilityDigest: createHash("sha256").update(`${managedToken}-different`, "utf8").digest("hex"),
|
||||
},
|
||||
});
|
||||
assert.equal(managedConflict.status, 409);
|
||||
assert.equal((await managedConflict.json()).error, "managed_writer_binding_request_conflict");
|
||||
|
||||
const legacyManagedRevoke = await fetch(
|
||||
`${baseUrl}/internal/data-plane/v1/writer-bindings/${managedResults[0].value.writerBinding.id}/revoke`,
|
||||
{ method: "POST", headers: { Authorization: `Bearer ${provisionerSecret}` } },
|
||||
);
|
||||
assert.equal(legacyManagedRevoke.status, 404);
|
||||
|
||||
const managedTokenGeneration2 = `${managedToken}_generation_2`;
|
||||
const managedGeneration2 = await jsonRequest(
|
||||
"/internal/data-plane/v1/writer-bindings/by-key/engine.api-connection.positions",
|
||||
{
|
||||
method: "PUT",
|
||||
managedSignature: true,
|
||||
body: {
|
||||
...managedBody,
|
||||
generation: 2,
|
||||
capabilityDigest: createHash("sha256").update(managedTokenGeneration2, "utf8").digest("hex"),
|
||||
},
|
||||
},
|
||||
);
|
||||
assert.equal(managedGeneration2.writerBinding.generation, 2);
|
||||
assert.deepEqual(
|
||||
(await jsonRequest("/internal/data-plane/v1/writer/data-products", { token: managedTokenGeneration2 })).dataProducts.map((value) => value.id),
|
||||
[productId],
|
||||
);
|
||||
const managedRevoke = await jsonRequest(
|
||||
"/internal/data-plane/v1/writer-bindings/by-key/engine.api-connection.positions/generations/1/revoke",
|
||||
{ method: "POST", managedSignature: true },
|
||||
);
|
||||
assert.equal(managedRevoke.idempotent, false);
|
||||
assert.equal(managedRevoke.writerBinding.active, false);
|
||||
const managedRevokeRetry = await jsonRequest(
|
||||
"/internal/data-plane/v1/writer-bindings/by-key/engine.api-connection.positions/generations/1/revoke",
|
||||
{ method: "POST", managedSignature: true },
|
||||
);
|
||||
assert.equal(managedRevokeRetry.idempotent, true);
|
||||
const rejectedManagedGeneration1 = await fetch(`${baseUrl}/internal/data-plane/v1/writer/data-products`, {
|
||||
headers: { Authorization: `Bearer ${managedToken}` },
|
||||
});
|
||||
assert.equal(rejectedManagedGeneration1.status, 401);
|
||||
|
||||
const manualWriterBody = { source: scope, allowedDataProductIds: [productId], expiresAt: expiry };
|
||||
const signedOnlyManualProvisioning = await signedFetch("/internal/data-plane/v1/writer-bindings", {
|
||||
method: "POST",
|
||||
body: manualWriterBody,
|
||||
});
|
||||
assert.equal(signedOnlyManualProvisioning.status, 401);
|
||||
assert.equal((await signedOnlyManualProvisioning.json()).error, "provisioner_unauthorized");
|
||||
|
||||
const writerIssuance = await rawJsonRequest("/internal/data-plane/v1/writer-bindings", {
|
||||
method: "POST",
|
||||
token: provisionerSecret,
|
||||
body: { source: scope, allowedDataProductIds: [productId], expiresAt: expiry },
|
||||
body: manualWriterBody,
|
||||
});
|
||||
const readerIssuance = await rawJsonRequest("/internal/data-plane/v1/reader-bindings", {
|
||||
method: "POST",
|
||||
|
|
@ -176,26 +326,66 @@ function publish(runId, observedAt, longitude) {
|
|||
};
|
||||
}
|
||||
|
||||
async function jsonRequest(path, { method = "GET", token, body } = {}) {
|
||||
const { response, value } = await rawJsonRequest(path, { method, token, body });
|
||||
async function jsonRequest(path, { method = "GET", token, managedSignature = false, body } = {}) {
|
||||
const { response, value } = await rawJsonRequest(path, { method, token, managedSignature, body });
|
||||
if (!response.ok) throw new Error(`request_failed:${response.status}:${value.error}`);
|
||||
return value;
|
||||
}
|
||||
|
||||
async function rawJsonRequest(path, { method = "GET", token, body } = {}) {
|
||||
async function rawJsonRequest(path, { method = "GET", token, managedSignature = false, body } = {}) {
|
||||
const rawBody = body === undefined ? Buffer.alloc(0) : Buffer.from(JSON.stringify(body), "utf8");
|
||||
const response = await fetch(`${baseUrl}${path}`, {
|
||||
method,
|
||||
headers: {
|
||||
...(token ? { Authorization: `Bearer ${token}` } : {}),
|
||||
...(body ? { "Content-Type": "application/json" } : {}),
|
||||
...(managedSignature ? signedManagedHeaders(path, method, rawBody) : {}),
|
||||
...(body === undefined ? {} : { "Content-Type": "application/json" }),
|
||||
},
|
||||
body: body ? JSON.stringify(body) : undefined,
|
||||
body: body === undefined ? undefined : rawBody,
|
||||
});
|
||||
const value = await response.json();
|
||||
if (!response.ok) throw new Error(`request_failed:${response.status}:${value.error}`);
|
||||
return { response, value };
|
||||
}
|
||||
|
||||
async function signedFetch(path, { method, body } = {}) {
|
||||
const rawBody = body === undefined ? Buffer.alloc(0) : Buffer.from(JSON.stringify(body), "utf8");
|
||||
return fetch(`${baseUrl}${path}`, {
|
||||
method,
|
||||
headers: {
|
||||
...signedManagedHeaders(path, method, rawBody),
|
||||
...(body === undefined ? {} : { "Content-Type": "application/json" }),
|
||||
},
|
||||
body: body === undefined ? undefined : rawBody,
|
||||
});
|
||||
}
|
||||
|
||||
function signedManagedHeaders(path, method, rawBody) {
|
||||
const timestamp = new Date().toISOString();
|
||||
const nonce = randomBytes(24).toString("base64url");
|
||||
const bodySha256 = sha256RawBody(rawBody);
|
||||
const payload = managedProvisionerSigningPayload({
|
||||
audience: managedProvisionerAudience,
|
||||
serviceId: managedProvisionerServiceId,
|
||||
keyId: managedProvisionerKeyId,
|
||||
method,
|
||||
path,
|
||||
timestamp,
|
||||
nonce,
|
||||
bodySha256,
|
||||
});
|
||||
const signature = sign(null, Buffer.from(payload, "utf8"), managedProvisionerPrivateKey).toString("base64url");
|
||||
return {
|
||||
[MANAGED_PROVISIONER_HEADERS.serviceId]: managedProvisionerServiceId,
|
||||
[MANAGED_PROVISIONER_HEADERS.keyId]: managedProvisionerKeyId,
|
||||
[MANAGED_PROVISIONER_HEADERS.audience]: managedProvisionerAudience,
|
||||
[MANAGED_PROVISIONER_HEADERS.timestamp]: timestamp,
|
||||
[MANAGED_PROVISIONER_HEADERS.nonce]: nonce,
|
||||
[MANAGED_PROVISIONER_HEADERS.bodySha256]: bodySha256,
|
||||
[MANAGED_PROVISIONER_HEADERS.signature]: signature,
|
||||
};
|
||||
}
|
||||
|
||||
function assertOneTimeCapabilityResponse(response) {
|
||||
assert.equal(response.headers.get("cache-control"), "no-store, max-age=0");
|
||||
assert.equal(response.headers.get("pragma"), "no-cache");
|
||||
|
|
|
|||
|
|
@ -1,4 +1,5 @@
|
|||
import assert from "node:assert/strict";
|
||||
import { generateKeyPairSync } from "node:crypto";
|
||||
import { mkdtemp, rm, writeFile } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
|
|
@ -11,9 +12,12 @@ const base = {
|
|||
|
||||
const directory = await mkdtemp(join(tmpdir(), "nodedc-edp-config-"));
|
||||
const secretPath = join(directory, "provisioner-token");
|
||||
const publicKeyPath = join(directory, "engine-public-key.pem");
|
||||
const secret = "provisioner_secret_is_separate_from_legacy_internal_token_123456";
|
||||
try {
|
||||
await writeFile(secretPath, `${secret}\n`, { encoding: "utf8", mode: 0o600 });
|
||||
const { publicKey } = generateKeyPairSync("ed25519");
|
||||
await writeFile(publicKeyPath, publicKey.export({ type: "spki", format: "pem" }), { mode: 0o600 });
|
||||
const config = readConfig({
|
||||
...base,
|
||||
EXTERNAL_DATA_PLANE_PROVISIONING_ENABLED: "true",
|
||||
|
|
@ -22,6 +26,8 @@ try {
|
|||
EXTERNAL_DATA_PLANE_RETENTION_SWEEP_MS: "60000",
|
||||
});
|
||||
assert.equal(config.provisionerAccessToken, secret);
|
||||
assert.equal(config.provisionerApiEnabled, true);
|
||||
assert.equal(config.managedProvisionerApiEnabled, false);
|
||||
assert.equal(config.maxFutureSkewSeconds, 120);
|
||||
assert.equal(config.retentionSweepMs, 60000);
|
||||
assert.equal(config.maxPatchBytes, 262144);
|
||||
|
|
@ -39,10 +45,49 @@ try {
|
|||
EXTERNAL_DATA_PLANE_PROVISIONING_ENABLED: "true",
|
||||
EXTERNAL_DATA_PLANE_PROVISIONER_TOKEN_FILE: join(directory, "missing"),
|
||||
}), /external_data_plane_provisioner_token_file_unreadable/);
|
||||
const managedConfig = readConfig({
|
||||
...base,
|
||||
EXTERNAL_DATA_PLANE_PROVISIONING_ENABLED: "false",
|
||||
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONING_ENABLED: "true",
|
||||
EXTERNAL_DATA_PLANE_PROVISIONER_TOKEN_FILE: join(directory, "missing-token-is-ignored"),
|
||||
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_PUBLIC_KEY_FILE: publicKeyPath,
|
||||
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_SERVICE_ID: "nodedc-engine",
|
||||
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_KEY_ID: "engine-edp-managed-provisioner-v1",
|
||||
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_AUDIENCE: "nodedc-external-data-plane.managed-provisioning.v1",
|
||||
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_MAX_SKEW_SECONDS: "45",
|
||||
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_REPLAY_CACHE_MAX_ENTRIES: "500",
|
||||
});
|
||||
assert.equal(managedConfig.provisionerApiEnabled, false);
|
||||
assert.equal(managedConfig.managedProvisionerApiEnabled, true);
|
||||
assert.equal(managedConfig.provisionerAccessToken, "");
|
||||
assert.equal(managedConfig.managedProvisionerPublicKey.asymmetricKeyType, "ed25519");
|
||||
assert.equal(managedConfig.managedProvisionerServiceId, "nodedc-engine");
|
||||
assert.equal(managedConfig.managedProvisionerKeyId, "engine-edp-managed-provisioner-v1");
|
||||
assert.equal(managedConfig.managedProvisionerAudience, "nodedc-external-data-plane.managed-provisioning.v1");
|
||||
assert.equal(managedConfig.managedProvisionerMaxSkewMs, 45_000);
|
||||
assert.equal(managedConfig.managedProvisionerReplayCacheMaxEntries, 500);
|
||||
assert.throws(() => readConfig({
|
||||
...base,
|
||||
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONING_ENABLED: "true",
|
||||
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_PUBLIC_KEY_FILE: join(directory, "missing"),
|
||||
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_SERVICE_ID: "nodedc-engine",
|
||||
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_KEY_ID: "engine-edp-managed-provisioner-v1",
|
||||
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_AUDIENCE: "nodedc-external-data-plane.managed-provisioning.v1",
|
||||
}), /external_data_plane_managed_provisioner_public_key_file_unreadable/);
|
||||
assert.throws(() => readConfig({
|
||||
...base,
|
||||
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONING_ENABLED: "true",
|
||||
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_PUBLIC_KEY_FILE: publicKeyPath,
|
||||
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_SERVICE_ID: "invalid service id",
|
||||
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_KEY_ID: "engine-edp-managed-provisioner-v1",
|
||||
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_AUDIENCE: "nodedc-external-data-plane.managed-provisioning.v1",
|
||||
}), /external_data_plane_managed_provisioner_service_id_invalid/);
|
||||
assert.equal(readConfig({
|
||||
...base,
|
||||
EXTERNAL_DATA_PLANE_PROVISIONING_ENABLED: "false",
|
||||
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONING_ENABLED: "false",
|
||||
EXTERNAL_DATA_PLANE_PROVISIONER_TOKEN_FILE: join(directory, "missing"),
|
||||
EXTERNAL_DATA_PLANE_MANAGED_PROVISIONER_PUBLIC_KEY_FILE: join(directory, "missing"),
|
||||
}).provisionerAccessToken, "");
|
||||
} finally {
|
||||
await rm(directory, { recursive: true, force: true });
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
import assert from "node:assert/strict";
|
||||
import { Pool } from "pg";
|
||||
import { validateDataProductPatch, validateDataProductSnapshot } from "@nodedc/external-provider-contract";
|
||||
import { validateDataProductPatch, validateDataProductSnapshot } from "@nodedc/external-provider-contract/data-plane";
|
||||
import {
|
||||
loadDataProductDefinition,
|
||||
persistDataProductDefinition,
|
||||
|
|
|
|||
|
|
@ -0,0 +1,217 @@
|
|||
import assert from "node:assert/strict";
|
||||
import {
|
||||
generateKeyPairSync,
|
||||
randomBytes,
|
||||
sign,
|
||||
} from "node:crypto";
|
||||
import { chmod, mkdtemp, rm, symlink, writeFile } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import {
|
||||
createManagedProvisionerRequestVerifier,
|
||||
loadManagedProvisionerPublicKeyFile,
|
||||
MANAGED_PROVISIONER_HEADERS,
|
||||
managedProvisionerSigningPayload,
|
||||
ManagedProvisionerReplayCache,
|
||||
sha256RawBody,
|
||||
} from "../src/managed-provisioner-auth.mjs";
|
||||
|
||||
const serviceId = "nodedc-engine";
|
||||
const keyId = "engine-edp-managed-provisioner-v1";
|
||||
const audience = "nodedc-external-data-plane.managed-provisioning.v1";
|
||||
const nowMs = Date.parse("2026-07-16T12:00:00.000Z");
|
||||
const { privateKey, publicKey } = generateKeyPairSync("ed25519");
|
||||
let nonceSequence = 0;
|
||||
|
||||
const replayCache = new ManagedProvisionerReplayCache({ maxEntries: 100 });
|
||||
const verify = createManagedProvisionerRequestVerifier({
|
||||
publicKey,
|
||||
serviceId,
|
||||
keyId,
|
||||
audience,
|
||||
maxSkewMs: 60_000,
|
||||
replayCache,
|
||||
now: () => nowMs,
|
||||
});
|
||||
|
||||
const valid = signedRequest();
|
||||
assert.deepEqual(verify(valid), { serviceId, keyId });
|
||||
assert.throws(() => verify(valid), hasCode("managed_provisioner_request_replayed", 409));
|
||||
|
||||
const whitespaceTamper = signedRequest();
|
||||
whitespaceTamper.rawBody = Buffer.from(`${whitespaceTamper.rawBody.toString("utf8")} `, "utf8");
|
||||
assert.throws(() => verify(whitespaceTamper), hasCode("managed_provisioner_body_hash_mismatch", 401));
|
||||
|
||||
const methodTamper = signedRequest();
|
||||
methodTamper.method = "POST";
|
||||
assert.throws(() => verify(methodTamper), hasCode("managed_provisioner_signature_invalid", 401));
|
||||
|
||||
const pathTamper = signedRequest();
|
||||
pathTamper.originalUrl = `${pathTamper.originalUrl}?unexpected=true`;
|
||||
assert.throws(() => verify(pathTamper), hasCode("managed_provisioner_signature_invalid", 401));
|
||||
|
||||
const audienceTamper = signedRequest();
|
||||
setHeader(audienceTamper, MANAGED_PROVISIONER_HEADERS.audience, "other-audience");
|
||||
assert.throws(() => verify(audienceTamper), hasCode("managed_provisioner_identity_mismatch", 401));
|
||||
|
||||
const serviceTamper = signedRequest();
|
||||
setHeader(serviceTamper, MANAGED_PROVISIONER_HEADERS.serviceId, "other-engine");
|
||||
assert.throws(() => verify(serviceTamper), hasCode("managed_provisioner_identity_mismatch", 401));
|
||||
const keyTamper = signedRequest();
|
||||
setHeader(keyTamper, MANAGED_PROVISIONER_HEADERS.keyId, "other-key-v1");
|
||||
assert.throws(() => verify(keyTamper), hasCode("managed_provisioner_identity_mismatch", 401));
|
||||
|
||||
const forbiddenBearer = signedRequest();
|
||||
forbiddenBearer.headers.authorization = "Bearer value-that-must-not-appear-in-errors";
|
||||
forbiddenBearer.rawHeaders.push("Authorization", forbiddenBearer.headers.authorization);
|
||||
assert.throws(
|
||||
() => verify(forbiddenBearer),
|
||||
(error) => hasCode("managed_provisioner_authorization_header_forbidden", 401)(error)
|
||||
&& !error.message.includes("value-that-must-not-appear-in-errors"),
|
||||
);
|
||||
|
||||
const stale = signedRequest({ timestamp: new Date(nowMs - 60_000).toISOString() });
|
||||
assert.throws(() => verify(stale), hasCode("managed_provisioner_timestamp_outside_window", 401));
|
||||
const future = signedRequest({ timestamp: new Date(nowMs + 60_000).toISOString() });
|
||||
assert.throws(() => verify(future), hasCode("managed_provisioner_timestamp_outside_window", 401));
|
||||
const invalidTimestamp = signedRequest();
|
||||
setHeader(invalidTimestamp, MANAGED_PROVISIONER_HEADERS.timestamp, "2026-07-16T12:00:00Z");
|
||||
assert.throws(() => verify(invalidTimestamp), hasCode("managed_provisioner_timestamp_invalid", 401));
|
||||
|
||||
const duplicateHeader = signedRequest();
|
||||
duplicateHeader.rawHeaders.push(MANAGED_PROVISIONER_HEADERS.nonce, header(duplicateHeader, MANAGED_PROVISIONER_HEADERS.nonce));
|
||||
assert.throws(() => verify(duplicateHeader), hasCode("managed_provisioner_header_invalid", 401));
|
||||
|
||||
const uncaptured = signedRequest();
|
||||
uncaptured.rawBodyCaptured = false;
|
||||
uncaptured.headers["content-length"] = String(uncaptured.rawBody.length);
|
||||
assert.throws(() => verify(uncaptured), hasCode("managed_provisioner_raw_body_unavailable", 401));
|
||||
|
||||
const badSignature = signedRequest();
|
||||
setHeader(badSignature, MANAGED_PROVISIONER_HEADERS.signature, randomBytes(64).toString("base64url"));
|
||||
assert.throws(() => verify(badSignature), hasCode("managed_provisioner_signature_invalid", 401));
|
||||
const paddedSignature = signedRequest();
|
||||
setHeader(
|
||||
paddedSignature,
|
||||
MANAGED_PROVISIONER_HEADERS.signature,
|
||||
`${header(paddedSignature, MANAGED_PROVISIONER_HEADERS.signature)}=`,
|
||||
);
|
||||
assert.throws(() => verify(paddedSignature), hasCode("managed_provisioner_signature_invalid", 401));
|
||||
const uppercaseHash = signedRequest();
|
||||
setHeader(
|
||||
uppercaseHash,
|
||||
MANAGED_PROVISIONER_HEADERS.bodySha256,
|
||||
header(uppercaseHash, MANAGED_PROVISIONER_HEADERS.bodySha256).toUpperCase(),
|
||||
);
|
||||
assert.throws(() => verify(uppercaseHash), hasCode("managed_provisioner_body_hash_invalid", 401));
|
||||
|
||||
const cache = new ManagedProvisionerReplayCache({ maxEntries: 1 });
|
||||
cache.consume("first", nowMs + 1000, nowMs);
|
||||
assert.throws(() => cache.consume("second", nowMs + 1000, nowMs), hasCode("managed_provisioner_replay_cache_exhausted", 503));
|
||||
assert.throws(() => cache.consume("first", nowMs + 1000, nowMs), hasCode("managed_provisioner_request_replayed", 409));
|
||||
cache.consume("second", nowMs + 2000, nowMs + 1000);
|
||||
|
||||
const directory = await mkdtemp(join(tmpdir(), "nodedc-edp-managed-auth-"));
|
||||
try {
|
||||
const publicKeyPath = join(directory, "engine-public-key.pem");
|
||||
const rsaPublicKeyPath = join(directory, "rsa-public-key.pem");
|
||||
const privateKeyPath = join(directory, "engine-private-key.pem");
|
||||
const symlinkPath = join(directory, "engine-public-key-link.pem");
|
||||
await writeFile(publicKeyPath, publicKey.export({ type: "spki", format: "pem" }), { mode: 0o600 });
|
||||
assert.equal(loadManagedProvisionerPublicKeyFile(publicKeyPath).asymmetricKeyType, "ed25519");
|
||||
assert.throws(
|
||||
() => loadManagedProvisionerPublicKeyFile("relative-public-key.pem"),
|
||||
/external_data_plane_managed_provisioner_public_key_file_must_be_absolute/,
|
||||
);
|
||||
assert.throws(
|
||||
() => loadManagedProvisionerPublicKeyFile(join(directory, "missing.pem")),
|
||||
/external_data_plane_managed_provisioner_public_key_file_unreadable/,
|
||||
);
|
||||
|
||||
await symlink(publicKeyPath, symlinkPath);
|
||||
assert.throws(
|
||||
() => loadManagedProvisionerPublicKeyFile(symlinkPath),
|
||||
/external_data_plane_managed_provisioner_public_key_file_must_be_regular/,
|
||||
);
|
||||
|
||||
await chmod(publicKeyPath, 0o622);
|
||||
assert.throws(
|
||||
() => loadManagedProvisionerPublicKeyFile(publicKeyPath),
|
||||
/external_data_plane_managed_provisioner_public_key_file_permissions_invalid/,
|
||||
);
|
||||
await chmod(publicKeyPath, 0o600);
|
||||
|
||||
const { publicKey: rsaPublicKey } = generateKeyPairSync("rsa", { modulusLength: 2048 });
|
||||
await writeFile(rsaPublicKeyPath, rsaPublicKey.export({ type: "spki", format: "pem" }), { mode: 0o600 });
|
||||
assert.throws(
|
||||
() => loadManagedProvisionerPublicKeyFile(rsaPublicKeyPath),
|
||||
/external_data_plane_managed_provisioner_public_key_algorithm_invalid/,
|
||||
);
|
||||
|
||||
await writeFile(privateKeyPath, privateKey.export({ type: "pkcs8", format: "pem" }), { mode: 0o600 });
|
||||
assert.throws(
|
||||
() => loadManagedProvisionerPublicKeyFile(privateKeyPath),
|
||||
/external_data_plane_managed_provisioner_public_key_file_format_invalid/,
|
||||
);
|
||||
} finally {
|
||||
await rm(directory, { recursive: true, force: true });
|
||||
}
|
||||
|
||||
console.log("external-data-plane managed provisioner auth: ok");
|
||||
|
||||
function signedRequest({
|
||||
method = "PUT",
|
||||
path = "/internal/data-plane/v1/writer-bindings/by-key/engine.test.positions",
|
||||
rawBody = Buffer.from('{"generation":1}', "utf8"),
|
||||
timestamp = new Date(nowMs).toISOString(),
|
||||
nonce = nextNonce(),
|
||||
} = {}) {
|
||||
const bodySha256 = sha256RawBody(rawBody);
|
||||
const signature = sign(null, Buffer.from(managedProvisionerSigningPayload({
|
||||
audience,
|
||||
serviceId,
|
||||
keyId,
|
||||
method,
|
||||
path,
|
||||
timestamp,
|
||||
nonce,
|
||||
bodySha256,
|
||||
}), "utf8"), privateKey).toString("base64url");
|
||||
const headers = {
|
||||
[MANAGED_PROVISIONER_HEADERS.serviceId]: serviceId,
|
||||
[MANAGED_PROVISIONER_HEADERS.keyId]: keyId,
|
||||
[MANAGED_PROVISIONER_HEADERS.audience]: audience,
|
||||
[MANAGED_PROVISIONER_HEADERS.timestamp]: timestamp,
|
||||
[MANAGED_PROVISIONER_HEADERS.nonce]: nonce,
|
||||
[MANAGED_PROVISIONER_HEADERS.bodySha256]: bodySha256,
|
||||
[MANAGED_PROVISIONER_HEADERS.signature]: signature,
|
||||
"content-length": String(rawBody.length),
|
||||
};
|
||||
return {
|
||||
method,
|
||||
originalUrl: path,
|
||||
headers,
|
||||
rawHeaders: Object.entries(headers).flat(),
|
||||
rawBody,
|
||||
rawBodyCaptured: true,
|
||||
};
|
||||
}
|
||||
|
||||
function nextNonce() {
|
||||
nonceSequence += 1;
|
||||
return Buffer.from(`managed-auth-test-${String(nonceSequence).padStart(6, "0")}`).toString("base64url");
|
||||
}
|
||||
|
||||
function header(req, name) {
|
||||
return req.headers[name];
|
||||
}
|
||||
|
||||
function setHeader(req, name, value) {
|
||||
req.headers[name] = value;
|
||||
const index = req.rawHeaders.findIndex((candidate) => String(candidate).toLowerCase() === name);
|
||||
req.rawHeaders[index + 1] = value;
|
||||
}
|
||||
|
||||
function hasCode(code, status) {
|
||||
return (error) => error?.code === code && error?.status === status;
|
||||
}
|
||||
|
|
@ -1,12 +1,14 @@
|
|||
import assert from "node:assert/strict";
|
||||
import { validateIntakeBatch } from "../../../packages/external-provider-contract/src/index.mjs";
|
||||
import { validateIntakeBatch } from "../../../packages/external-provider-contract/src/data-plane.mjs";
|
||||
import {
|
||||
createWriterToken,
|
||||
hashWriterToken,
|
||||
materializeDataProductPublish,
|
||||
materializeWriterBoundBatch,
|
||||
normalizeManagedWriterBindingRequest,
|
||||
normalizeWriterBindingRequest,
|
||||
safeWriterBinding,
|
||||
writerBindingRequestHash,
|
||||
} from "../src/writer-binding.mjs";
|
||||
|
||||
const now = new Date("2026-07-15T12:00:00.000Z");
|
||||
|
|
@ -34,6 +36,39 @@ assert.match(token, /^ndc_edpwb_[A-Za-z0-9_-]{40,}$/);
|
|||
assert.equal(hashWriterToken(token), hashWriterToken(token));
|
||||
assert.notEqual(hashWriterToken(token), hashWriterToken(`${token}x`));
|
||||
|
||||
const managedPolicy = normalizeManagedWriterBindingRequest({
|
||||
...request,
|
||||
allowedDataProductIds: ["fleet.positions.current.v1", "asset.status.current.v1"],
|
||||
generation: 1,
|
||||
capabilityDigest: hashWriterToken(token),
|
||||
}, { now, maxTtlDays: 90 });
|
||||
assert.deepEqual(managedPolicy.allowedDataProductIds, [
|
||||
"asset.status.current.v1",
|
||||
"fleet.positions.current.v1",
|
||||
]);
|
||||
assert.equal(managedPolicy.generation, 1);
|
||||
assert.equal(managedPolicy.capabilityDigest, hashWriterToken(token));
|
||||
assert.equal(writerBindingRequestHash(managedPolicy), writerBindingRequestHash({
|
||||
...managedPolicy,
|
||||
allowedDataProductIds: [...managedPolicy.allowedDataProductIds].reverse(),
|
||||
}));
|
||||
assert.throws(() => normalizeManagedWriterBindingRequest({
|
||||
...request,
|
||||
generation: 0,
|
||||
capabilityDigest: hashWriterToken(token),
|
||||
}, { now, maxTtlDays: 90 }), /managed_writer_binding_generation_invalid/);
|
||||
assert.throws(() => normalizeManagedWriterBindingRequest({
|
||||
...request,
|
||||
generation: 1,
|
||||
capabilityDigest: "not-a-digest",
|
||||
}, { now, maxTtlDays: 90 }), /managed_writer_binding_capability_digest_invalid/);
|
||||
assert.throws(() => normalizeManagedWriterBindingRequest({
|
||||
...request,
|
||||
generation: 1,
|
||||
capabilityDigest: hashWriterToken(token),
|
||||
token,
|
||||
}, { now, maxTtlDays: 90 }), /managed_writer_binding_secret_material_forbidden/);
|
||||
|
||||
const unscopedBatch = {
|
||||
schemaVersion: "nodedc.external-provider-contract/v1",
|
||||
source: { providerId: "example-provider" },
|
||||
|
|
|
|||
|
|
@ -1,19 +1,19 @@
|
|||
{
|
||||
"version": "0.1.0",
|
||||
"updatedAt": "2026-07-13",
|
||||
"version": "1.0.0",
|
||||
"updatedAt": "2026-07-16",
|
||||
"aliases": [
|
||||
{ "alias": "gelios", "canonicalId": "gelios.integration" },
|
||||
{ "alias": "gelius", "canonicalId": "gelios.integration" },
|
||||
{ "alias": "helius", "canonicalId": "gelios.integration" },
|
||||
{ "alias": "hel i os", "canonicalId": "gelios.integration" },
|
||||
{ "alias": "гелиос", "canonicalId": "gelios.integration" },
|
||||
{ "alias": "трайк robot2b", "canonicalId": "gelios.unit" },
|
||||
{ "alias": "отслеживаемый объект гелиос", "canonicalId": "gelios.unit" },
|
||||
{ "alias": "юнит гелиос", "canonicalId": "gelios.unit" },
|
||||
{ "alias": "группа юнитов гелиос", "canonicalId": "gelios.unit_group" },
|
||||
{ "alias": "последняя телеметрия гелиос", "canonicalId": "gelios.telemetry_snapshot" },
|
||||
{ "alias": "сырая телеметрия гелиос", "canonicalId": "gelios.raw_telemetry_message" },
|
||||
{ "alias": "позиция трайка", "canonicalId": "gelios.position_fix" },
|
||||
{ "alias": "статус трайка", "canonicalId": "gelios.operational_status" },
|
||||
{ "alias": "позиция объекта гелиос", "canonicalId": "gelios.position_fix" },
|
||||
{ "alias": "статус объекта гелиос", "canonicalId": "gelios.operational_status" },
|
||||
{ "alias": "датчик гелиос", "canonicalId": "gelios.sensor_definition" },
|
||||
{ "alias": "показание датчика", "canonicalId": "gelios.sensor_reading" },
|
||||
{ "alias": "калибровка датчика", "canonicalId": "gelios.sensor_conversion" },
|
||||
|
|
@ -23,7 +23,7 @@
|
|||
{ "alias": "шаблон команды гелиос", "canonicalId": "gelios.command_template" },
|
||||
{ "alias": "диспетчеризация команды гелиос", "canonicalId": "gelios.command_dispatch" },
|
||||
{ "alias": "аудит команды гелиос", "canonicalId": "gelios.command_audit" },
|
||||
{ "alias": "контур robot2b", "canonicalId": "gelios.access_scope" },
|
||||
{ "alias": "область доступа credential гелиос", "canonicalId": "gelios.access_scope" },
|
||||
{ "alias": "курсор загрузки гелиос", "canonicalId": "gelios.ingestion_cursor" }
|
||||
]
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,19 +1,19 @@
|
|||
{
|
||||
"version": "0.1.0",
|
||||
"updatedAt": "2026-07-13",
|
||||
"version": "1.0.0",
|
||||
"updatedAt": "2026-07-16",
|
||||
"entities": [
|
||||
{ "id": "gelios.integration", "name": "Gelios Integration", "surface": "platform", "status": ["product-required", "source-evidenced"], "authority": "Gelios Gateway", "summary": "Server-side provider integration boundary for Gelios REST OAuth and legacy compatibility. It owns no UI, map renderer or workflow secret." },
|
||||
{ "id": "gelios.access_scope", "name": "Gelios Access Scope", "surface": "platform", "status": ["product-required", "source-evidenced"], "authority": "Robot2B owner + Gelios Gateway", "summary": "Approved collection boundary expressed as an allowlist or owner rule for units, groups and allowed capabilities; distinct from broad provider-account visibility." },
|
||||
{ "id": "gelios.unit", "name": "Gelios Unit", "surface": "domain", "status": ["source-evidenced"], "authority": "Gelios Pro", "summary": "Tracked operational object, including a Robot2B trike when its stable source unit ID is inside the approved scope." },
|
||||
{ "id": "gelios.unit_group", "name": "Gelios Unit Group", "surface": "domain", "status": ["source-evidenced"], "authority": "Gelios Pro", "summary": "Provider-managed grouping of units. It is evidence for navigation and access, not an automatic definition of the Robot2B business scope." },
|
||||
{ "id": "gelios.integration", "name": "Gelios Integration", "surface": "platform", "status": ["product-required", "source-evidenced"], "authority": "NDC L2 connection", "summary": "One tenant-scoped Gelios provider connection backed by one opaque NDC L2 credential reference. It owns no UI, renderer, secret value or provider-specific Platform service." },
|
||||
{ "id": "gelios.access_scope", "name": "Gelios Access Scope", "surface": "platform", "status": ["product-required", "source-evidenced"], "authority": "Gelios credential visibility + NDC L2 connection policy", "summary": "Dynamic collection boundary: selected safe-read capabilities include every entity visible to the bound credential on each run. Entity allowlists and provider-group business filters are not part of collection scope." },
|
||||
{ "id": "gelios.unit", "name": "Gelios Unit", "surface": "domain", "status": ["source-evidenced"], "authority": "Gelios Pro", "summary": "Tracked operational object with a stable provider source ID. Every unit returned for the bound credential is eligible for canonical mapping." },
|
||||
{ "id": "gelios.unit_group", "name": "Gelios Unit Group", "surface": "domain", "status": ["source-evidenced"], "authority": "Gelios Pro", "summary": "Provider-managed grouping of units used as domain data and optional consumer metadata, never as an implicit collection filter." },
|
||||
{ "id": "gelios.tracker_device", "name": "Tracker Device", "surface": "domain", "status": ["source-evidenced"], "authority": "Gelios Pro", "summary": "Hardware tracker associated with a unit, carrying device type/manufacturer and restricted hardware identity fields." },
|
||||
{ "id": "gelios.telemetry_snapshot", "name": "Telemetry Snapshot", "surface": "telemetry", "status": ["source-evidenced", "product-required"], "authority": "Gelios Gateway", "summary": "Normalized current telemetry state for one unit at observed and received time; derived from the Gelios pure last message contract." },
|
||||
{ "id": "gelios.telemetry_snapshot", "name": "Telemetry Snapshot", "surface": "telemetry", "status": ["source-evidenced", "product-required"], "authority": "NDC L2 semantic mapping", "summary": "Normalized current telemetry state for one unit at observed and received time; derived from the versioned Gelios last-message mapping contract." },
|
||||
{ "id": "gelios.raw_telemetry_message", "name": "Raw Telemetry Message", "surface": "telemetry", "status": ["source-evidenced"], "authority": "Gelios Pro", "summary": "Provider hardware message preserved only behind restricted raw-data policy; it is not the default Studio or analytics contract." },
|
||||
{ "id": "gelios.position_fix", "name": "Position Fix", "surface": "telemetry", "status": ["source-evidenced", "product-required"], "authority": "Gelios Gateway", "summary": "Time-qualified geographic position with latitude, longitude, height, course, speed, satellite and quality attributes for one unit." },
|
||||
{ "id": "gelios.position_fix", "name": "Position Fix", "surface": "telemetry", "status": ["source-evidenced", "product-required"], "authority": "NDC L2 semantic mapping", "summary": "Time-qualified geographic position with latitude, longitude, height, course, speed, satellite and quality attributes for one unit." },
|
||||
{ "id": "gelios.telemetry_parameter", "name": "Telemetry Parameter", "surface": "telemetry", "status": ["source-evidenced"], "authority": "Gelios Pro", "summary": "Name/value parameter received with a message. Parameters are dynamic and require a namespaced registry and whitelist before product exposure." },
|
||||
{ "id": "gelios.operational_status", "name": "Operational Status", "surface": "telemetry", "status": ["source-evidenced", "product-required"], "authority": "Gelios Gateway", "summary": "Derived unit state such as online, offline, moving, parked, no-position or low-GPS; distinct from raw provider message fields." },
|
||||
{ "id": "gelios.operational_status", "name": "Operational Status", "surface": "telemetry", "status": ["source-evidenced", "product-required"], "authority": "NDC L2 semantic mapping", "summary": "Derived unit state such as active, stale, moving, parked, no-position or low-GPS; distinct from raw provider message fields." },
|
||||
{ "id": "gelios.sensor_definition", "name": "Sensor Definition", "surface": "domain", "status": ["source-evidenced"], "authority": "Gelios Pro", "summary": "Declared sensor on a unit with message parameter, type, unit of measure, visibility and optional fuel semantics." },
|
||||
{ "id": "gelios.sensor_reading", "name": "Sensor Reading", "surface": "telemetry", "status": ["source-evidenced", "product-required"], "authority": "Gelios Gateway", "summary": "Time-qualified normalized reading of a defined sensor, retaining both numeric value and human-readable representation when permitted." },
|
||||
{ "id": "gelios.sensor_reading", "name": "Sensor Reading", "surface": "telemetry", "status": ["source-evidenced", "product-required"], "authority": "NDC L2 semantic mapping", "summary": "Time-qualified normalized reading of a defined sensor, retaining both numeric value and human-readable representation only when permitted by a versioned field policy." },
|
||||
{ "id": "gelios.sensor_conversion", "name": "Sensor Conversion", "surface": "domain", "status": ["source-evidenced"], "authority": "Gelios Pro", "summary": "Calibration/conversion rule or row for translating raw sensor values into operational measures." },
|
||||
{ "id": "gelios.fuel_profile", "name": "Fuel Profile", "surface": "domain", "status": ["source-evidenced"], "authority": "Gelios Pro", "summary": "Fuel and consumption configuration associated with a unit or fuel sensor; use only after sensor semantics are approved." },
|
||||
{ "id": "gelios.maintenance_plan", "name": "Maintenance Plan", "surface": "domain", "status": ["source-evidenced"], "authority": "Gelios Pro", "summary": "Planned maintenance configuration for a unit, separate from historical maintenance events." },
|
||||
|
|
@ -22,13 +22,13 @@
|
|||
{ "id": "gelios.geozone_group", "name": "Gelios Geozone Group", "surface": "spatial", "status": ["source-evidenced"], "authority": "Gelios Pro", "summary": "Provider grouping of geozones." },
|
||||
{ "id": "gelios.geopoint", "name": "Gelios Geopoint", "surface": "spatial", "status": ["source-evidenced"], "authority": "Gelios Pro", "summary": "Named provider spatial point; it must remain distinct from a live unit position." },
|
||||
{ "id": "gelios.report_template", "name": "Gelios Report Template", "surface": "analytics", "status": ["source-evidenced"], "authority": "Gelios Pro", "summary": "Reusable provider report definition available to the account or scope." },
|
||||
{ "id": "gelios.report_result", "name": "Gelios Report Result", "surface": "analytics", "status": ["source-evidenced"], "authority": "Gelios Pro + Gelios Gateway", "summary": "Bounded result, table, graphic or map output produced from an approved report request; not a default realtime stream." },
|
||||
{ "id": "gelios.collection_run", "name": "Gelios Collection Run", "surface": "platform", "status": ["product-required"], "authority": "Gelios Gateway", "summary": "Auditable read-only collection attempt with scope, endpoint family, volume controls and outcome; it contains no token or raw payload." },
|
||||
{ "id": "gelios.ingestion_cursor", "name": "Gelios Ingestion Cursor", "surface": "platform", "status": ["product-required"], "authority": "Gelios Gateway + telemetry storage", "summary": "Checkpoint and watermark controlling resumable historical or incremental ingestion for an approved unit and data family." },
|
||||
{ "id": "gelios.report_result", "name": "Gelios Report Result", "surface": "analytics", "status": ["source-evidenced"], "authority": "Gelios Pro + NDC L2 semantic mapping", "summary": "Bounded result, table, graphic or map output produced from an approved report request; not a default realtime stream." },
|
||||
{ "id": "gelios.collection_run", "name": "Gelios Collection Run", "surface": "platform", "status": ["product-required"], "authority": "NDC L2 workflow", "summary": "Auditable safe-read collection attempt with credential-visible entity scope, endpoint family, volume controls and outcome; it contains no credential value or raw payload." },
|
||||
{ "id": "gelios.ingestion_cursor", "name": "Gelios Ingestion Cursor", "surface": "platform", "status": ["product-required"], "authority": "NDC L2 workflow state + External Data Plane", "summary": "Checkpoint and watermark controlling resumable historical or incremental ingestion for a connection and data family." },
|
||||
{ "id": "gelios.command_template", "name": "Gelios Command Template", "surface": "command", "status": ["source-evidenced", "product-required"], "authority": "Gelios Pro", "summary": "Read-only catalogued command template assigned to a unit. Template content is red-domain configuration, not a telemetry field." },
|
||||
{ "id": "gelios.command_group", "name": "Gelios Command Group", "surface": "command", "status": ["source-evidenced", "product-required"], "authority": "Gelios Pro", "summary": "Provider grouping of command templates, managed separately from operational unit groups." },
|
||||
{ "id": "gelios.command_dispatch", "name": "Gelios Command Dispatch", "surface": "command", "status": ["product-required"], "authority": "Future Command Gateway", "summary": "Explicit, user-confirmed request to send a command to one approved unit or approved group. It must never be created by background ingestion or map rendering." },
|
||||
{ "id": "gelios.command_delivery", "name": "Gelios Command Delivery", "surface": "command", "status": ["product-required", "source-evidenced"], "authority": "Gelios Pro + Future Command Gateway", "summary": "Provider delivery/task status associated with a command dispatch." },
|
||||
{ "id": "gelios.command_audit", "name": "Gelios Command Audit", "surface": "command", "status": ["product-required"], "authority": "Future Command Gateway", "summary": "Immutable governance record of an explicit command decision, confirmation, target, outcome and actor; separate from provider command history." }
|
||||
{ "id": "gelios.command_dispatch", "name": "Gelios Command Dispatch", "surface": "command", "status": ["future-concept"], "authority": "Unimplemented NDC red-domain command boundary", "summary": "Future explicit, user-confirmed request to send a command to one approved unit or approved group. It must never be created by background ingestion or map rendering." },
|
||||
{ "id": "gelios.command_delivery", "name": "Gelios Command Delivery", "surface": "command", "status": ["future-concept", "source-evidenced"], "authority": "Gelios Pro; future NDC red-domain reconciliation", "summary": "Provider delivery/task status that may be reconciled with a future governed command dispatch; no command transport exists in the current package." },
|
||||
{ "id": "gelios.command_audit", "name": "Gelios Command Audit", "surface": "command", "status": ["future-concept"], "authority": "Unimplemented NDC red-domain command boundary", "summary": "Future immutable governance record of an explicit command decision, confirmation, target, outcome and actor; separate from provider command history." }
|
||||
]
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
{
|
||||
"version": "0.1.0",
|
||||
"updatedAt": "2026-07-13",
|
||||
"version": "1.0.0",
|
||||
"updatedAt": "2026-07-16",
|
||||
"sourceRoots": [
|
||||
{
|
||||
"surface": "gelios-rest",
|
||||
|
|
@ -8,9 +8,9 @@
|
|||
"mode": "OpenAPI inspection and safe read-only runtime audit; no write routes executed"
|
||||
},
|
||||
{
|
||||
"surface": "engine",
|
||||
"surface": "ndc-l2-legacy-evidence",
|
||||
"path": "/Users/dcconstructions/Downloads/mnt/NODEDC/NODEDC_ENGINE_INFRA/nodedc-source",
|
||||
"mode": "Read-only MMAP workflow, Gelios node and historical snapshot donor inspection"
|
||||
"mode": "Read-only historical NDC L2 donor, Gelios mapping and synthetic-safe field-shape inspection"
|
||||
},
|
||||
{
|
||||
"surface": "module-studio",
|
||||
|
|
@ -41,9 +41,9 @@
|
|||
"baselineDocs": ["docs/GELIOS_DOMAIN_ONTOLOGY.md"],
|
||||
"restrictions": [
|
||||
"Do not copy access/refresh tokens, passwords, hardware decrypt keys or runtime snapshots into Ontology Core.",
|
||||
"Do not make the current Engine Gelios node, Cesium entities or provider group names canonical domain identities.",
|
||||
"Do not make a current NDC L2 workflow node, renderer entities or provider group names canonical domain identities.",
|
||||
"Do not invoke command send, create, update, delete or purge routes as ontology evidence.",
|
||||
"Do not treat the 107 currently visible provider-account units or the 95 legacy snapshot units as the final Robot2B scope without owner approval.",
|
||||
"Robot2B pilot counts (107 credential-visible units and 95 legacy snapshot units) are historical evidence only; canonical collection scope is dynamically all entities visible to the bound credential.",
|
||||
"Do not bulk-download history, media or full geozone geometry before collection policy and storage architecture are approved."
|
||||
]
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
{
|
||||
"version": "0.1.0",
|
||||
"updatedAt": "2026-07-13",
|
||||
"version": "1.0.0",
|
||||
"updatedAt": "2026-07-16",
|
||||
"rules": [
|
||||
{
|
||||
"id": "guardrail.gelios.ontology_not_runtime_store",
|
||||
|
|
@ -8,10 +8,16 @@
|
|||
"summary": "Ontology Core describes Gelios meanings and contracts only. It must not contain access/refresh tokens, raw messages, live positions, provider command text, hardware decrypt keys or bulk geometry.",
|
||||
"entityIds": ["gelios.integration", "gelios.telemetry_snapshot", "gelios.raw_telemetry_message", "gelios.position_fix", "gelios.command_template", "gelios.geozone"]
|
||||
},
|
||||
{
|
||||
"id": "guardrail.gelios.provider_auth_pair_not_capability_scope",
|
||||
"severity": "error",
|
||||
"summary": "Gelios provider authentication consists of access and refresh tokens only. Credential labels and safe-read classifications are workflow policy, not provider token scope; internal External Data Plane capabilities are not Gelios tokens.",
|
||||
"entityIds": ["gelios.integration", "gelios.access_scope", "gelios.collection_run"]
|
||||
},
|
||||
{
|
||||
"id": "guardrail.gelios.scope_precedes_collection",
|
||||
"severity": "error",
|
||||
"summary": "A Gelios collection run must resolve an approved Robot2B access scope before addressing units. Provider-account visibility and provider group membership are not sufficient product scope on their own.",
|
||||
"summary": "A Gelios safe-read collection run must process every valid entity returned for the bound credential and selected capability. Static unit allowlists and provider-group business filters are forbidden at collection boundary; consumer visibility is handled downstream.",
|
||||
"entityIds": ["gelios.integration", "gelios.access_scope", "gelios.collection_run", "gelios.unit", "gelios.unit_group"]
|
||||
},
|
||||
{
|
||||
|
|
@ -41,7 +47,7 @@
|
|||
{
|
||||
"id": "guardrail.gelios.history_requires_cursor",
|
||||
"severity": "warning",
|
||||
"summary": "Any future history ingestion must be bounded by unit scope, time window, field whitelist and durable cursor. It must not turn a historical API family into an uncontrolled account dump.",
|
||||
"summary": "Any future history ingestion must be bounded by connection, credential-visible entity scope, time window, field policy and durable cursor. It must not turn a historical API family into an uncontrolled account dump.",
|
||||
"entityIds": ["gelios.collection_run", "gelios.ingestion_cursor", "gelios.raw_telemetry_message", "gelios.report_result"]
|
||||
}
|
||||
],
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
{
|
||||
"id": "gelios",
|
||||
"version": "0.1.0",
|
||||
"updatedAt": "2026-07-13",
|
||||
"version": "1.0.0",
|
||||
"updatedAt": "2026-07-16",
|
||||
"status": "product-required/source-evidenced",
|
||||
"summary": "Gelios Pro telemetry, fleet, spatial, reporting and command-domain ontology for the Robot2B client context. The package describes canonical meanings and contracts; it does not store credentials or runtime telemetry."
|
||||
"summary": "Provider-neutral Gelios Pro telemetry, fleet, spatial, reporting and command-domain ontology. It describes canonical meanings and contracts; it does not store credentials, account scope or runtime telemetry."
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,23 +1,23 @@
|
|||
{
|
||||
"version": "0.1.0",
|
||||
"updatedAt": "2026-07-13",
|
||||
"version": "1.0.0",
|
||||
"updatedAt": "2026-07-16",
|
||||
"relations": [
|
||||
{ "id": "gelios.integration.is_provider_connection", "from": ["gelios.integration"], "to": ["integration.connection"], "status": "product-required", "summary": "A Gelios integration is a tenant-scoped instance of the common external provider connection contract." },
|
||||
{ "id": "gelios.access_scope.specializes_integration_scope", "from": ["gelios.access_scope"], "to": ["integration.access_scope"], "status": "product-required", "summary": "Robot2B/Gelios object approval specializes the common provider access-scope contract." },
|
||||
{ "id": "gelios.access_scope.specializes_integration_scope", "from": ["gelios.access_scope"], "to": ["integration.access_scope"], "status": "product-required", "summary": "Gelios credential-visible capability scope specializes the common provider access-scope contract." },
|
||||
{ "id": "gelios.collection_run.is_integration_collection_run", "from": ["gelios.collection_run"], "to": ["integration.collection_run"], "status": "product-required", "summary": "Gelios collection runs inherit the bounded collection and audit lifecycle of external integrations." },
|
||||
{ "id": "gelios.raw_message.is_integration_raw_envelope", "from": ["gelios.raw_telemetry_message"], "to": ["integration.raw_envelope"], "status": "product-required", "summary": "A Gelios raw message is a restricted provider envelope, not a UI read model." },
|
||||
{ "id": "gelios.unit.is_integration_canonical_subject", "from": ["gelios.unit"], "to": ["integration.canonical_subject"], "status": "product-required", "summary": "A Gelios unit is a stable canonical subject produced by the provider adapter." },
|
||||
{ "id": "gelios.command_template.is_integration_command_capability", "from": ["gelios.command_template"], "to": ["integration.command_capability"], "status": "product-required", "summary": "Gelios command templates are catalogued provider command capabilities only." },
|
||||
{ "id": "gelios.command_dispatch.is_integration_command_intent", "from": ["gelios.command_dispatch"], "to": ["integration.command_intent"], "status": "future-concept", "summary": "Any future Gelios command dispatch must become an explicitly governed generic command intent." },
|
||||
{ "id": "gelios.command_audit.is_integration_command_audit", "from": ["gelios.command_audit"], "to": ["integration.command_audit"], "status": "future-concept", "summary": "Gelios command audit specializes the generic red-domain audit contract." },
|
||||
{ "id": "gelios.integration.enforces_scope", "from": ["gelios.integration"], "to": ["gelios.access_scope"], "status": "product-required", "summary": "The Gateway applies an approved Robot2B scope before reading or publishing data." },
|
||||
{ "id": "gelios.access_scope.includes_unit", "from": ["gelios.access_scope"], "to": ["gelios.unit", "gelios.unit_group"], "status": "product-required", "summary": "An approved scope can include explicit units and approved group criteria, but group membership alone is not authoritative." },
|
||||
{ "id": "gelios.integration.enforces_scope", "from": ["gelios.integration"], "to": ["gelios.access_scope"], "status": "product-required", "summary": "NDC L2 resolves selected safe-read capabilities and current provider visibility from the bound credential on every collection run." },
|
||||
{ "id": "gelios.access_scope.includes_unit", "from": ["gelios.access_scope"], "to": ["gelios.unit", "gelios.unit_group"], "status": "product-required", "summary": "Scope dynamically includes every unit and group record returned by the selected capability; it contains no static entity allowlist." },
|
||||
{ "id": "gelios.unit.belongs_to_group", "from": ["gelios.unit"], "to": ["gelios.unit_group"], "status": "source-evidenced", "summary": "A provider unit can belong to one or more provider unit groups." },
|
||||
{ "id": "gelios.unit.uses_tracker_device", "from": ["gelios.unit"], "to": ["gelios.tracker_device"], "status": "source-evidenced", "summary": "A tracked unit is associated with a hardware tracker device." },
|
||||
{ "id": "gelios.unit.has_telemetry_snapshot", "from": ["gelios.unit"], "to": ["gelios.telemetry_snapshot"], "status": "product-required", "summary": "The Gateway maintains normalized current telemetry for an approved unit." },
|
||||
{ "id": "gelios.unit.has_telemetry_snapshot", "from": ["gelios.unit"], "to": ["gelios.telemetry_snapshot"], "status": "product-required", "summary": "The versioned NDC L2 mapping produces normalized current telemetry for every valid returned unit." },
|
||||
{ "id": "gelios.raw_message.normalizes_to_snapshot", "from": ["gelios.raw_telemetry_message"], "to": ["gelios.telemetry_snapshot"], "status": "product-required", "summary": "Restricted raw provider telemetry can be normalized into the stable snapshot contract." },
|
||||
{ "id": "gelios.telemetry_snapshot.has_position_fix", "from": ["gelios.telemetry_snapshot"], "to": ["gelios.position_fix"], "status": "product-required", "summary": "A current telemetry snapshot can carry one time-qualified position fix." },
|
||||
{ "id": "gelios.telemetry_snapshot.has_operational_status", "from": ["gelios.telemetry_snapshot"], "to": ["gelios.operational_status"], "status": "product-required", "summary": "Gateway derives normalized operational state from telemetry and data-quality policy." },
|
||||
{ "id": "gelios.telemetry_snapshot.has_operational_status", "from": ["gelios.telemetry_snapshot"], "to": ["gelios.operational_status"], "status": "product-required", "summary": "The semantic mapping derives normalized operational state from telemetry and data-quality policy." },
|
||||
{ "id": "gelios.telemetry_snapshot.has_parameter", "from": ["gelios.telemetry_snapshot"], "to": ["gelios.telemetry_parameter"], "status": "source-evidenced", "summary": "A provider snapshot can contain dynamic telemetry parameters." },
|
||||
{ "id": "gelios.unit.has_sensor", "from": ["gelios.unit"], "to": ["gelios.sensor_definition"], "status": "source-evidenced", "summary": "A unit exposes zero or more sensor definitions." },
|
||||
{ "id": "gelios.sensor_definition.has_conversion", "from": ["gelios.sensor_definition"], "to": ["gelios.sensor_conversion"], "status": "source-evidenced", "summary": "A sensor can define conversion/calibration rows." },
|
||||
|
|
@ -29,14 +29,14 @@
|
|||
{ "id": "gelios.geozone.belongs_to_group", "from": ["gelios.geozone"], "to": ["gelios.geozone_group"], "status": "source-evidenced", "summary": "A provider geozone can belong to a provider geozone group." },
|
||||
{ "id": "gelios.report_result.uses_template", "from": ["gelios.report_result"], "to": ["gelios.report_template"], "status": "source-evidenced", "summary": "A report result is produced from an approved report template and bounded request." },
|
||||
{ "id": "gelios.collection_run.checkpoints_with_cursor", "from": ["gelios.collection_run"], "to": ["gelios.ingestion_cursor"], "status": "product-required", "summary": "Collection runs advance resumable ingestion only through a durable cursor." },
|
||||
{ "id": "gelios.collection_run.collects_unit", "from": ["gelios.collection_run"], "to": ["gelios.unit"], "status": "product-required", "summary": "A read-only collection run records the approved unit scope it addressed." },
|
||||
{ "id": "gelios.collection_run.collects_unit", "from": ["gelios.collection_run"], "to": ["gelios.unit"], "status": "product-required", "summary": "A safe-read collection run records every valid credential-visible unit returned by the selected capability." },
|
||||
{ "id": "gelios.unit.has_command_template", "from": ["gelios.unit"], "to": ["gelios.command_template"], "status": "source-evidenced", "summary": "Command templates are catalogued per unit through read-only routes." },
|
||||
{ "id": "gelios.command_template.belongs_to_group", "from": ["gelios.command_template"], "to": ["gelios.command_group"], "status": "source-evidenced", "summary": "A command template can belong to a provider command group." },
|
||||
{ "id": "gelios.command_dispatch.uses_template", "from": ["gelios.command_dispatch"], "to": ["gelios.command_template"], "status": "product-required", "summary": "A dispatch may reference a reviewed command template." },
|
||||
{ "id": "gelios.command_dispatch.targets_scope", "from": ["gelios.command_dispatch"], "to": ["gelios.unit", "gelios.unit_group", "gelios.access_scope"], "status": "product-required", "summary": "A dispatch target must resolve to an approved unit or explicitly approved group in scope." },
|
||||
{ "id": "gelios.command_dispatch.has_delivery", "from": ["gelios.command_dispatch"], "to": ["gelios.command_delivery"], "status": "product-required", "summary": "A dispatch is associated with delivery/task state rather than assumed successful on request creation." },
|
||||
{ "id": "gelios.command_dispatch.has_audit", "from": ["gelios.command_dispatch"], "to": ["gelios.command_audit"], "status": "product-required", "summary": "Every dispatch requires immutable user-confirmation and outcome audit." },
|
||||
{ "id": "gelios.unit.is_map_moving_object", "from": ["gelios.unit"], "to": ["map.moving_object"], "status": "product-required", "summary": "An approved Gelios unit is rendered through the provider-neutral moving-object contract, never as a Cesium entity identity." },
|
||||
{ "id": "gelios.command_dispatch.uses_template", "from": ["gelios.command_dispatch"], "to": ["gelios.command_template"], "status": "future-concept", "summary": "A future dispatch may reference a reviewed command template." },
|
||||
{ "id": "gelios.command_dispatch.targets_scope", "from": ["gelios.command_dispatch"], "to": ["gelios.unit", "gelios.unit_group", "gelios.access_scope"], "status": "future-concept", "summary": "A future dispatch target must resolve to an approved unit or explicitly approved group in scope." },
|
||||
{ "id": "gelios.command_dispatch.has_delivery", "from": ["gelios.command_dispatch"], "to": ["gelios.command_delivery"], "status": "future-concept", "summary": "A future dispatch is associated with delivery/task state rather than assumed successful on request creation." },
|
||||
{ "id": "gelios.command_dispatch.has_audit", "from": ["gelios.command_dispatch"], "to": ["gelios.command_audit"], "status": "future-concept", "summary": "Every future dispatch requires immutable user-confirmation and outcome audit." },
|
||||
{ "id": "gelios.unit.is_map_moving_object", "from": ["gelios.unit"], "to": ["map.moving_object"], "status": "product-required", "summary": "A valid credential-visible Gelios unit is exposed through the provider-neutral moving-object contract, never as a renderer entity identity." },
|
||||
{ "id": "gelios.position_fix.positions_map_subject", "from": ["gelios.position_fix"], "to": ["map.moving_object"], "status": "product-required", "summary": "A normalized position fix updates the stable map subject bound to the Gelios unit." },
|
||||
{ "id": "gelios.geozone.is_map_zone", "from": ["gelios.geozone"], "to": ["map.zone"], "status": "product-required", "summary": "An approved Gelios geozone is exposed to the map through the generic zone contract." },
|
||||
{ "id": "gelios.geopoint.is_map_place_target", "from": ["gelios.geopoint"], "to": ["map.place_target"], "status": "product-required", "summary": "An approved Gelios geopoint is exposed through a provider-neutral place target." }
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
{
|
||||
"version": "0.1.0",
|
||||
"updatedAt": "2026-07-13",
|
||||
"version": "1.0.0",
|
||||
"updatedAt": "2026-07-16",
|
||||
"aliases": [
|
||||
{"alias":"коннектор","canonicalId":"integration.connection","language":"ru","status":"product-required"},
|
||||
{"alias":"connector","canonicalId":"integration.connection","language":"en","status":"product-required"},
|
||||
|
|
|
|||
|
|
@ -1,21 +1,21 @@
|
|||
{
|
||||
"version": "0.1.0",
|
||||
"updatedAt": "2026-07-13",
|
||||
"version": "1.0.0",
|
||||
"updatedAt": "2026-07-16",
|
||||
"entities": [
|
||||
{"id":"integration.provider","name":"External Provider","surface":"integration","status":["product-required"],"authority":"Platform External Provider Data Plane","summary":"A third-party product or API family integrated through an app-owned adapter, not a tenant or a renderer."},
|
||||
{"id":"integration.connection","name":"Provider Connection","surface":"integration","status":["product-required"],"authority":"Provider Gateway","summary":"A tenant-scoped configured connection to one provider account or endpoint; it contains policy and secret references, never secret values."},
|
||||
{"id":"integration.credential_reference","name":"Provider Credential Reference","surface":"integration","status":["product-required"],"authority":"Provider Gateway / secret store","summary":"Opaque reference to a server-side secret or token lifecycle, not a token, login or password."},
|
||||
{"id":"integration.access_scope","name":"Integration Access Scope","surface":"integration","status":["product-required"],"authority":"Client owner + Provider Gateway","summary":"Approved product boundary for source accounts, objects, fields and permitted operations; provider visibility alone is insufficient."},
|
||||
{"id":"integration.capability","name":"Provider Capability","surface":"integration","status":["product-required"],"authority":"Provider Adapter","summary":"Versioned provider operation or feature classified as read, metadata, write, destructive or unknown."},
|
||||
{"id":"integration.field_definition","name":"Provider Field Definition","surface":"integration","status":["product-required"],"authority":"Provider Adapter + Ontology Core","summary":"A documented source field, its semantics, sensitivity, units and normalisation mapping; it is not a live value."},
|
||||
{"id":"integration.collection_profile","name":"Collection Profile","surface":"integration","status":["product-required"],"authority":"Provider Gateway","summary":"Approved rate, paging, cursor, capability and field selection for a connection; prevents uncontrolled account-wide polling."},
|
||||
{"id":"integration.retention_policy","name":"Integration Retention Policy","surface":"integration","status":["product-required"],"authority":"Client owner + Provider Gateway","summary":"Lifecycle rule for current state, raw envelope, normalised history, aggregates and backups."},
|
||||
{"id":"integration.collection_run","name":"Integration Collection Run","surface":"integration","status":["product-required"],"authority":"Provider Gateway","summary":"Auditable bounded execution of a safe-read collection profile with cursor, response metrics and outcome."},
|
||||
{"id":"integration.raw_envelope","name":"Raw Provider Envelope","surface":"integration","status":["product-required"],"authority":"Provider Gateway","summary":"Restricted provenance-preserving representation of a safe-read provider response, stored in a bounded cold layer rather than exposed to UI."},
|
||||
{"id":"integration.canonical_subject","name":"Canonical Integrated Subject","surface":"integration","status":["product-required"],"authority":"Provider Domain Adapter","summary":"Stable domain subject normalised from a provider object and suitable for relations, analytics and interfaces."},
|
||||
{"id":"integration.read_model","name":"Integration Read Model","surface":"integration","status":["product-required"],"authority":"Provider Gateway","summary":"Scoped, versioned current, history or aggregate projection delivered to approved consumers without raw provider bypass."},
|
||||
{"id":"integration.realtime_channel","name":"Integration Realtime Channel","surface":"integration","status":["product-required"],"authority":"Provider Gateway","summary":"Internal bounded event or subscription contract for projection changes; its emission rate is independent from provider collection rate."},
|
||||
{"id":"integration.command_capability","name":"Provider Command Capability","surface":"integration","status":["product-required"],"authority":"Provider Adapter","summary":"Catalogued command metadata, parameters and risk classification. It is not permission or a send route."},
|
||||
{"id":"integration.provider","name":"External Provider","surface":"integration","status":["product-required"],"authority":"Versioned provider package + Ontology Core","summary":"A third-party product or API family integrated through NDC L2, not a tenant, runtime service or renderer."},
|
||||
{"id":"integration.connection","name":"Provider Connection","surface":"integration","status":["product-required"],"authority":"NDC L2 connection","summary":"A tenant-scoped instance of one versioned provider package for one provider account; it contains policy and opaque credential references, never secret values."},
|
||||
{"id":"integration.credential_reference","name":"Provider Credential Reference","surface":"integration","status":["product-required"],"authority":"NDC L2 Credentials","summary":"Opaque reference to an NDC L2-managed provider credential lifecycle, not a credential value, login or password."},
|
||||
{"id":"integration.access_scope","name":"Integration Access Scope","surface":"integration","status":["product-required"],"authority":"Provider credential visibility + NDC L2 connection profile","summary":"Selected safe-read capabilities and fields applied to every valid entity visible to the bound credential. Static entity allowlists and provider-group business filters are not collection scope."},
|
||||
{"id":"integration.capability","name":"Provider Capability","surface":"integration","status":["product-required"],"authority":"Versioned provider package","summary":"Versioned provider operation or feature classified as read, metadata, write, destructive or unknown."},
|
||||
{"id":"integration.field_definition","name":"Provider Field Definition","surface":"integration","status":["product-required"],"authority":"Versioned provider package + Ontology Core","summary":"A documented source field, its semantics, sensitivity, units and normalisation mapping; it is not a live value."},
|
||||
{"id":"integration.collection_profile","name":"Collection Profile","surface":"integration","status":["product-required"],"authority":"NDC L2 connection profile","summary":"Rate, paging, cursor, safe capability, field policy and batching selection for one connection; every valid entity returned for the credential is processed."},
|
||||
{"id":"integration.retention_policy","name":"Integration Retention Policy","surface":"integration","status":["product-required"],"authority":"External Data Plane Data Product definition","summary":"Lifecycle rule for current state, normalised history, aggregates and any separately approved raw reference."},
|
||||
{"id":"integration.collection_run","name":"Integration Collection Run","surface":"integration","status":["product-required"],"authority":"NDC L2 workflow","summary":"Auditable bounded execution of a safe-read collection profile with credential-visible entity scope, cursor, response metrics and outcome."},
|
||||
{"id":"integration.raw_envelope","name":"Raw Provider Envelope","surface":"integration","status":["product-required"],"authority":"Restricted raw-data policy","summary":"Optional provenance reference to a separately approved bounded raw layer; inline provider payload is not a Data Product publish field."},
|
||||
{"id":"integration.canonical_subject","name":"Canonical Integrated Subject","surface":"integration","status":["product-required"],"authority":"NDC L2 semantic mapping + Ontology Core","summary":"Stable domain subject normalised from a provider object and suitable for relations, analytics and interfaces."},
|
||||
{"id":"integration.read_model","name":"Integration Read Model","surface":"integration","status":["product-required"],"authority":"External Data Plane","summary":"Scoped, versioned Data Product current/history projection delivered to approved consumers without provider transport or raw bypass."},
|
||||
{"id":"integration.realtime_channel","name":"Integration Realtime Channel","surface":"integration","status":["product-required"],"authority":"External Data Plane","summary":"Bounded snapshot-and-patch contract for Data Product changes; its delivery and history cadence are independent from provider collection rate."},
|
||||
{"id":"integration.command_capability","name":"Provider Command Capability","surface":"integration","status":["product-required"],"authority":"Versioned provider package","summary":"Catalogued command metadata, parameters and risk classification. It is not permission or a send route."},
|
||||
{"id":"integration.command_intent","name":"Provider Command Intent","surface":"integration","status":["future-concept"],"authority":"Future Provider Command Gateway","summary":"A future explicitly confirmed, scoped and idempotent request to execute a catalogued command; no read collector may create it."},
|
||||
{"id":"integration.command_audit","name":"Provider Command Audit","surface":"integration","status":["future-concept"],"authority":"Future Provider Command Gateway","summary":"Immutable approval, execution and delivery audit for a red-domain command lifecycle."}
|
||||
]
|
||||
|
|
|
|||
|
|
@ -2,11 +2,11 @@
|
|||
"sourceRoots": ["platform/docs", "platform/packages", "platform/services"],
|
||||
"ledgers": [
|
||||
{
|
||||
"id": "external_provider_data_plane_adr",
|
||||
"path": "../../docs/ADR_EXTERNAL_PROVIDER_DATA_PLANE.md",
|
||||
"id": "l2_owned_external_connectors_adr",
|
||||
"path": "../../docs/ADR_L2_OWNED_EXTERNAL_CONNECTORS.md",
|
||||
"entityIds": ["integration.provider", "integration.connection", "integration.access_scope", "integration.capability", "integration.collection_profile", "integration.read_model", "integration.command_capability"]
|
||||
}
|
||||
],
|
||||
"baselineDocs": ["../../docs/ADR_EXTERNAL_PROVIDER_DATA_PLANE.md", "../../packages/external-provider-contract/README.md"],
|
||||
"baselineDocs": ["../../docs/ADR_L2_OWNED_EXTERNAL_CONNECTORS.md", "../../packages/external-provider-contract/README.md"],
|
||||
"restrictions": ["The package is a semantic contract. Provider secrets, runtime data and command transport remain outside Ontology Core."]
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,11 +1,11 @@
|
|||
{
|
||||
"version": "0.1.0",
|
||||
"updatedAt": "2026-07-13",
|
||||
"version": "1.0.0",
|
||||
"updatedAt": "2026-07-16",
|
||||
"rules": [
|
||||
{"id":"guardrail.integration.ontology_is_not_data_plane","severity":"error","summary":"Ontology Core may describe providers and contracts but must not store secret values, live payloads, tenant telemetry, raw envelopes or provider execution state.","entityIds":["integration.provider","integration.connection","integration.credential_reference","integration.raw_envelope","integration.read_model"]},
|
||||
{"id":"guardrail.integration.scope_precedes_collection","severity":"error","summary":"A collection run must resolve tenant scope, field policy and bounded collection profile before it addresses a provider. Account visibility does not define product scope.","entityIds":["integration.connection","integration.access_scope","integration.collection_profile","integration.collection_run"]},
|
||||
{"id":"guardrail.integration.read_model_not_raw_bypass","severity":"error","summary":"L2 workflows, interfaces and renderer adapters consume scoped read models or realtime channels, never provider credentials, raw envelopes or direct adapter database access.","entityIds":["integration.credential_reference","integration.raw_envelope","integration.read_model","integration.realtime_channel"]},
|
||||
{"id":"guardrail.integration.commands_are_separate_red_domain","severity":"error","summary":"Cataloguing a command does not activate it. Read collectors, L2 workflows, maps and AI Workspace may not create command intent or delivery. A later command gateway requires explicit human confirmation, scope, idempotency and audit.","entityIds":["integration.command_capability","integration.command_intent","integration.command_audit","integration.collection_run"]},
|
||||
{"id":"guardrail.integration.scope_precedes_collection","severity":"error","summary":"A collection run must resolve tenant/connection, safe capabilities, field policy and bounded profile before calling a provider, then process every valid entity returned for the bound credential. Static entity allowlists and provider-group business filters are forbidden at collection boundary.","entityIds":["integration.connection","integration.access_scope","integration.collection_profile","integration.collection_run"]},
|
||||
{"id":"guardrail.integration.read_model_not_raw_bypass","severity":"error","summary":"NDC L2 workflows, interfaces and renderer adapters consume scoped Data Products or realtime channels, never provider credentials, raw envelopes or direct adapter database access.","entityIds":["integration.credential_reference","integration.raw_envelope","integration.read_model","integration.realtime_channel"]},
|
||||
{"id":"guardrail.integration.commands_are_separate_red_domain","severity":"error","summary":"Cataloguing a command does not activate it. Read collectors, NDC L2 workflows, maps and AI Workspace may not create command intent or delivery. A later command boundary requires explicit human confirmation, scope, idempotency and audit.","entityIds":["integration.command_capability","integration.command_intent","integration.command_audit","integration.collection_run"]},
|
||||
{"id":"guardrail.integration.full_catalog_not_full_rate","severity":"warning","summary":"Every provider field may be catalogued, but enabling continuous collection requires an explicit collection, retention and rate policy. Do not turn a capability catalog into unbounded account polling.","entityIds":["integration.capability","integration.field_definition","integration.collection_profile","integration.retention_policy"]}
|
||||
],
|
||||
"blockedConflations": [
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
{
|
||||
"id": "integration",
|
||||
"version": "0.1.0",
|
||||
"updatedAt": "2026-07-13",
|
||||
"version": "1.0.0",
|
||||
"updatedAt": "2026-07-16",
|
||||
"status": "product-required/source-evidenced",
|
||||
"summary": "Provider-neutral external integration ontology: tenant-scoped connections, capabilities, collection, controlled storage projections and a separated red command domain."
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,13 +1,13 @@
|
|||
{
|
||||
"version": "0.1.0",
|
||||
"updatedAt": "2026-07-13",
|
||||
"version": "1.0.0",
|
||||
"updatedAt": "2026-07-16",
|
||||
"relations": [
|
||||
{"id":"integration.provider.offers_connection","from":["integration.provider"],"to":["integration.connection"],"status":"product-required","summary":"A provider is connected through one or more tenant-scoped connection instances."},
|
||||
{"id":"integration.connection.references_credential","from":["integration.connection"],"to":["integration.credential_reference"],"status":"product-required","summary":"A connection uses an opaque server-side credential reference rather than exposing a secret."},
|
||||
{"id":"integration.connection.enforces_scope","from":["integration.connection"],"to":["integration.access_scope"],"status":"product-required","summary":"Every source request is bounded by an approved product scope."},
|
||||
{"id":"integration.connection.enforces_scope","from":["integration.connection"],"to":["integration.access_scope"],"status":"product-required","summary":"Every source request is bounded by selected safe capabilities and field policy while entity membership is dynamically all valid objects visible to the bound credential."},
|
||||
{"id":"integration.provider.declares_capability","from":["integration.provider"],"to":["integration.capability","integration.command_capability"],"status":"product-required","summary":"Adapter manifests catalogue source read and command capabilities independently from their activation."},
|
||||
{"id":"integration.capability.exposes_field","from":["integration.capability"],"to":["integration.field_definition"],"status":"product-required","summary":"A safe-read capability documents the fields it may return before a collection profile enables them."},
|
||||
{"id":"integration.collection_profile.selects_capability","from":["integration.collection_profile"],"to":["integration.capability","integration.field_definition"],"status":"product-required","summary":"A profile selects bounded capabilities and fields rather than polling every API feature continuously."},
|
||||
{"id":"integration.collection_profile.selects_capability","from":["integration.collection_profile"],"to":["integration.capability","integration.field_definition"],"status":"product-required","summary":"A profile selects bounded safe capabilities and fields rather than polling every API feature; it does not select business entity IDs."},
|
||||
{"id":"integration.collection_profile.applies_retention","from":["integration.collection_profile"],"to":["integration.retention_policy"],"status":"product-required","summary":"Collection and storage lifecycle are explicit per connection profile."},
|
||||
{"id":"integration.collection_run.uses_connection","from":["integration.collection_run"],"to":["integration.connection","integration.collection_profile"],"status":"product-required","summary":"A collection run records which connection and bounded policy it used."},
|
||||
{"id":"integration.collection_run.records_raw_envelope","from":["integration.collection_run"],"to":["integration.raw_envelope"],"status":"product-required","summary":"Safe-read response provenance is retained only according to the active policy."},
|
||||
|
|
|
|||
|
|
@ -2,108 +2,202 @@
|
|||
|
||||
Package: `catalog/domain-packages/gelios`
|
||||
|
||||
Status: `v0.1.0` source-evidenced and product-required slice for the Robot2B client context and the Gelios provider.
|
||||
Status: `v1.0.0`, source-evidenced and product-required.
|
||||
|
||||
## Purpose and boundary
|
||||
|
||||
The package describes the meaning of Gelios fleet, telemetry, sensor, spatial, report and command concepts. It is not a Gelios client, token store, telemetry database, Cesium implementation or Engine workflow.
|
||||
The package describes Gelios fleet, telemetry, sensor, spatial, report and
|
||||
command meanings. It is not a provider client, credential store, runtime
|
||||
database, renderer implementation, NDC L1 agent or NDC L2 workflow.
|
||||
|
||||
The practical audit established a wide REST read surface: 107 currently visible units, 105 with `lastMsg`, a visible administrative account, unit groups, telemetry variants, sensor catalogues, command-template catalogues, geozone groups and report templates. The REST OpenAPI documents 295 operations. This ontology therefore separates three things that must not be conflated:
|
||||
Gelios is the first live provider used to prove the common provider-package
|
||||
model. Runtime transport, collection cadence and semantic mapping belong to an
|
||||
isolated NDC L2 connection instance. External Data Plane persists and delivers
|
||||
provider-neutral Data Products. Ontology Core describes meaning only.
|
||||
|
||||
1. documented provider capability;
|
||||
2. actual runtime read evidence for the current account;
|
||||
3. approved Robot2B product scope.
|
||||
The matching contract/data artifact is
|
||||
`platform/packages/external-provider-contract/providers/gelios/v1`. Adding a
|
||||
second account creates another connection instance with different opaque
|
||||
credential references; it does not create another ontology package, Platform
|
||||
service or custom node.
|
||||
|
||||
The 95-unit legacy Engine snapshot is a temporary compatibility reference only. It is not the final business scope. The provider's current groups also do not resolve that scope automatically. A Robot2B owner must approve an allowlist or equivalent rule represented by `gelios.access_scope`.
|
||||
## Provider authentication boundary
|
||||
|
||||
Gelios issues exactly two provider secret artifacts: an access token and a
|
||||
refresh token. The current NDC L2 `httpBearerAuth` request binding uses the
|
||||
access token. Automatic refresh has not been proven in the deployed runtime, so
|
||||
the matching provider package records refresh as `operator_managed`; neither
|
||||
token value belongs in Ontology, a workflow graph, MCP, Ops or a trace.
|
||||
|
||||
A credential label such as `read access` is operator metadata, not a Gelios
|
||||
token scope. `gelios.units.current.read` is classified as read because the
|
||||
approved workflow transport is `GET /api/v1/units`. The label does not create a
|
||||
separate read token or constrain other rights that Gelios may have granted to
|
||||
the same access token. The scoped Data Product writer credential later in the
|
||||
runtime path is an internal NDC/External Data Plane capability, not a third
|
||||
Gelios token.
|
||||
|
||||
## Dynamic connection scope
|
||||
|
||||
`gelios.access_scope` is derived from two inputs on every collection run:
|
||||
|
||||
1. safe-read capabilities selected in the connection profile;
|
||||
2. entities currently visible to the bound provider credential.
|
||||
|
||||
For `gelios.units.current.read`, NDC L2 processes every valid unit returned by
|
||||
`GET /api/v1/units`. Static unit IDs, owner allowlists and provider-group business
|
||||
filters are forbidden at collection boundary. New units visible to the same
|
||||
credential appear automatically. Foundry and other Data Product consumers own
|
||||
layer selection, sorting, filtering and visibility.
|
||||
|
||||
Technical limits such as a 5000-fact batch ceiling, retry policy and response
|
||||
validation protect runtime health; they are not business entity filters.
|
||||
|
||||
## Canonical value contracts
|
||||
|
||||
### Unit and device
|
||||
|
||||
`gelios.unit` is the stable business subject. The durable key is a namespaced provider identity such as `gelios.unit:<sourceUnitId>`, never a renderer entity ID or a display name. A unit can use a `gelios.tracker_device`, belong to multiple `gelios.unit_group` records and expose sensor, fuel, maintenance and custom-field configurations.
|
||||
`gelios.unit` is a stable provider subject. The published v1 source ID is
|
||||
`gelios-unit-<sourceUnitId>` inside immutable `(providerId, connectionId)`
|
||||
scope; a renderer entity ID or display name is never identity. The provider ID
|
||||
suffix is preserved as an exact string (including leading zeros) and is never
|
||||
parsed or renumbered. A unit can use a
|
||||
`gelios.tracker_device`, belong to multiple `gelios.unit_group` records and
|
||||
expose sensor, fuel, maintenance and custom-field configurations.
|
||||
|
||||
Hardware IDs, IMEI, phones and decrypt-related fields are restricted or secret-class data. They are not part of the default Studio contract.
|
||||
Hardware IDs, IMEI, phones, address, decrypt-related fields, raw `params` and
|
||||
unclassified sensor payloads are excluded by the v1 field policy. New provider
|
||||
fields remain dropped until they are evidenced, classified and introduced by a
|
||||
new package/ontology revision.
|
||||
|
||||
### Current telemetry
|
||||
### Current telemetry and position
|
||||
|
||||
`gelios.telemetry_snapshot` is the normalized current state produced by the Gateway, with at least:
|
||||
`gelios.telemetry_snapshot` is normalized by the versioned NDC L2 mapping.
|
||||
`gelios.position_fix` is its time-qualified spatial portion; it is not a pin or
|
||||
other renderer object.
|
||||
|
||||
The first approved output is exactly:
|
||||
|
||||
```text
|
||||
unitSubjectId
|
||||
observedAt
|
||||
receivedAt
|
||||
position: { latitude, longitude, height?, course?, speed?, satellites? }
|
||||
operationalStatus
|
||||
sensorReadings[]
|
||||
counterValues[]
|
||||
qualityFlags[]
|
||||
source: gelios-rest
|
||||
Data Product: fleet.positions.current.v1
|
||||
version: 1.0.0
|
||||
ontology revision: ontology.map.moving_object.v1
|
||||
semantic type: map.moving_object
|
||||
delivery: snapshot+patch
|
||||
history: sampled, latest entity per 60-second bucket, 90 days
|
||||
```
|
||||
|
||||
The source supports a raw and a pure last-message variant. Pure data may include sensors, counters and accumulated fields. Raw messages and dynamic `params` remain behind restricted policy and may be retained separately for diagnostics only. `gelios.position_fix` is the time-qualified position portion used by spatial consumers; it is not itself a pin.
|
||||
Allowed Data Product fields are:
|
||||
|
||||
```text
|
||||
course_degrees
|
||||
display_name
|
||||
elevation_meters
|
||||
geometry
|
||||
hdop
|
||||
horizontal_accuracy_meters
|
||||
object_kind
|
||||
operational_status
|
||||
position_source
|
||||
position_valid
|
||||
quality_flags
|
||||
satellite_count
|
||||
speed_kph
|
||||
```
|
||||
|
||||
All attribute names are snake_case. GeoJSON `geometry` is omitted when valid
|
||||
coordinates are unavailable, but the unit remains in the product with
|
||||
`position_valid=false`, `operational_status=no_position` and appropriate
|
||||
`quality_flags`. If the provider has no last-message timestamp, the collection
|
||||
receive time becomes the explicit `observedAt` fallback so the credential-visible
|
||||
unit is not silently dropped.
|
||||
|
||||
### Sensors and operational semantics
|
||||
|
||||
`gelios.sensor_definition`, `gelios.sensor_conversion` and `gelios.sensor_reading` separate stable device configuration, calibration and time-series readings. Fuel profile, maintenance plan and custom field are independent configuration concepts. No assumed sensor meaning should be added to analytics until its message parameter, measure and conversion semantics are approved.
|
||||
`gelios.sensor_definition`, `gelios.sensor_conversion` and
|
||||
`gelios.sensor_reading` separate device configuration, calibration and
|
||||
time-series values. Fuel profile, maintenance plan and custom field are
|
||||
independent concepts. A sensor meaning cannot enter a Data Product until its
|
||||
message parameter, unit, conversion, classification and field policy are
|
||||
approved.
|
||||
|
||||
### Spatial binding and Cesium pins
|
||||
### Spatial binding
|
||||
|
||||
Gelios is a source domain; Cesium is a renderer adapter. The binding is deliberately provider-neutral:
|
||||
Gelios is a source domain. The provider-neutral relation is:
|
||||
|
||||
```text
|
||||
gelios.unit
|
||||
-> map.moving_object (stable spatial subject)
|
||||
gelios.position_fix
|
||||
-> map.moving_object (current position update)
|
||||
map.moving_object
|
||||
-> map.pin + map.label + map.visibility_rule
|
||||
-> map.renderer_adapter (Cesium today, replaceable later)
|
||||
gelios.unit + gelios.position_fix
|
||||
-> map.moving_object
|
||||
-> fleet.positions.current.v1
|
||||
-> Foundry Data Product binding
|
||||
-> map layers, selection and telemetry panel
|
||||
```
|
||||
|
||||
The future map payload selects a stable `subjectId`, position, normalized operational status, observed time and approved label fields. It must not contain Gelios credentials, raw payloads or Cesium transient entity references. This is the replacement boundary for the current M-map node-to-Cesium coupling.
|
||||
|
||||
The Map Page resolves the selected Studio profile to the shared map.pin contract. The initial accepted presentation is an elevated spike — ground anchor, stem, outlined head and label — but that visual configuration remains on the Studio side. Gelios publishes data for a stable subject; it never publishes Cesium primitive settings, pin pixels or a renderer entity ID.
|
||||
|
||||
The proposed renderer-neutral payload is in [`examples/gelios-map-moving-object.fixture.json`](../examples/gelios-map-moving-object.fixture.json). It uses synthetic coordinates only: it specifies the interface between Gateway and Map View, not a live trike position and not a Cesium entity.
|
||||
The publish boundary contains stable facts only. It cannot contain provider
|
||||
credentials, tenant/connection claims supplied by the workflow, raw payloads,
|
||||
pixel values, styles or transient renderer identities. The synthetic contract
|
||||
fixture is
|
||||
[`examples/gelios-map-moving-object.fixture.json`](../examples/gelios-map-moving-object.fixture.json).
|
||||
|
||||
### Commands
|
||||
|
||||
`gelios.command_template` and `gelios.command_group` are read-only catalogue concepts. `gelios.command_dispatch`, `gelios.command_delivery` and `gelios.command_audit` are a separate red domain.
|
||||
`gelios.command_template` and `gelios.command_group` are read-only catalogue
|
||||
concepts. `gelios.command_dispatch`, `gelios.command_delivery` and
|
||||
`gelios.command_audit` are a separate red domain.
|
||||
|
||||
No collection run, workflow, map click or autonomous agent may create a dispatch. A future dispatch requires an explicit human action, approved scope, selected unit/group and template/parameters, confirmation, an audit record and delivery-state reconciliation. This ontology package does not grant or test a write permission.
|
||||
No collection run, workflow, map click or autonomous agent may create a
|
||||
dispatch. A future command path requires explicit human intent, confirmation,
|
||||
separate authorization, immutable audit and delivery reconciliation. This
|
||||
ontology and the Gelios v1 provider package expose no command transport.
|
||||
|
||||
## Future data plane
|
||||
|
||||
The intended flow is:
|
||||
## Canonical runtime flow
|
||||
|
||||
```text
|
||||
Gelios REST OAuth
|
||||
-> server-side Gelios Gateway
|
||||
-> scope filter + field policy + normalizer
|
||||
-> durable telemetry/configuration storage
|
||||
-> NDC level-2 workflow / internal data contract
|
||||
-> Map View binding / Cesium renderer adapter
|
||||
Gelios safe-read API
|
||||
-> opaque Gelios access credential in NDC L2 Credentials
|
||||
-> isolated NDC L2 connection workflow
|
||||
-> versioned semantic mapping
|
||||
-> NDC Data Product Publish through a native scoped writer credential
|
||||
-> External Data Plane
|
||||
-> scoped snapshot + patch stream
|
||||
-> Foundry Data Product binding
|
||||
```
|
||||
|
||||
`gelios.collection_run` and `gelios.ingestion_cursor` define collection governance independently of the transport implementation. They support bounded incremental history ingestion later: one approved unit scope, limited time window, field whitelist, volume limit and durable checkpoint. They do not imply that all history should be collected now.
|
||||
|
||||
Database technology is intentionally not chosen by Ontology Core. The proposed platform data-plane boundary and its validation gates are documented in [`platform/docs/ADR_GELIOS_DATA_PLANE.md`](../../../docs/ADR_GELIOS_DATA_PLANE.md). The ontology supplies the record boundaries needed by that ADR.
|
||||
There is no Gelios-specific Platform service or database in this model.
|
||||
Provider transport never enters External Data Plane or Foundry. `gelios.collection_run`
|
||||
and `gelios.ingestion_cursor` describe audit and resumability independently of
|
||||
physical storage.
|
||||
|
||||
## Guardrails
|
||||
|
||||
- Ontology Core never stores secrets, runtime snapshots, full raw telemetry or large geometry datasets.
|
||||
- Current provider-account visibility is not the approved Robot2B scope; collection must enforce an owner-approved scope before every run.
|
||||
- GET alone does not make a route safe: login-as, temporary tokens, configuration and WLN download are excluded from telemetry collection.
|
||||
- Geozones require paging, filtering, volume controls and map LOD.
|
||||
- Command send and mutation routes remain red even if the account has documented access.
|
||||
- Map selection and pins target stable domain subjects, not provider payload IDs or renderer objects.
|
||||
- Ontology Core never stores credentials, account state, runtime snapshots,
|
||||
full raw telemetry or bulk geometry.
|
||||
- Gelios authentication has exactly access and refresh artifacts; credential
|
||||
labels and workflow classifications must not be reinterpreted as additional
|
||||
provider token types or scopes.
|
||||
- Every safe-read collection includes all valid entities returned for the
|
||||
bound credential; unit allowlists and group filters are not collection
|
||||
scope.
|
||||
- A read method alone does not make a capability safe: impersonation,
|
||||
temporary credential issuance, configuration mutation and downloads remain
|
||||
excluded until separately classified.
|
||||
- Dynamic provider fields are dropped until evidenced and classified.
|
||||
- Geozones and history require bounded loading, paging/cursors, retention and
|
||||
volume controls.
|
||||
- Command send and mutation capabilities remain red even when provider access
|
||||
exists.
|
||||
- Foundry targets stable domain subjects and Data Product fields, not provider
|
||||
payloads or renderer identities.
|
||||
|
||||
## Evidence and next implementation decisions
|
||||
## Evidence and evolution
|
||||
|
||||
Evidence is limited to official REST OpenAPI, safe GET results, historical Engine donor inspection and the MAP package. The next decisions outside this ontology package are:
|
||||
Evidence is limited to official Gelios API documentation, safe-read results,
|
||||
historical NDC L2 donor inspection and provider-neutral map contracts. The
|
||||
Robot2B pilot counts (107 credential-visible units and a 95-unit legacy
|
||||
snapshot) are retained only as historical evidence of response shape and
|
||||
cardinality. They do not define canonical IDs, collection allowlists or product
|
||||
scope.
|
||||
|
||||
1. owner-approved Robot2B allowlist/rule;
|
||||
2. field policy for Studio, analytics and raw retention;
|
||||
3. data-plane ADR for realtime telemetry and history;
|
||||
4. Gelios Gateway implementation boundary and deployment location;
|
||||
5. fixture and renderer-adapter contract for Map View pins and labels.
|
||||
Extension order is mandatory: establish safe provider evidence, extend
|
||||
capability/field classification, update ontology and mapping revision, add
|
||||
synthetic fixtures and contract tests, then deploy an updated NDC L2 workflow.
|
||||
|
|
|
|||
|
|
@ -1,25 +1,33 @@
|
|||
{
|
||||
"contractVersion": "0.1.0",
|
||||
"kind": "map.moving_object.current_position",
|
||||
"source": "gelios",
|
||||
"subjectId": "gelios.unit:example-unit-id",
|
||||
"observedAt": "2026-07-13T00:00:00.000Z",
|
||||
"receivedAt": "2026-07-13T00:00:03.000Z",
|
||||
"position": {
|
||||
"latitude": 55.000001,
|
||||
"longitude": 37.000001,
|
||||
"heightMeters": 0,
|
||||
"courseDegrees": 90,
|
||||
"speedKph": 0,
|
||||
"satellites": 0
|
||||
"schemaVersion": "nodedc.data-product.publish/v1",
|
||||
"batch": {
|
||||
"runId": "gelios-ontology-fixture-20260716",
|
||||
"sequence": 0,
|
||||
"idempotencyKey": "gelios-ontology-fixture-20260716.batch-0"
|
||||
},
|
||||
"operationalStatus": "unknown",
|
||||
"display": {
|
||||
"label": "Robot2B unit",
|
||||
"visibilityProfile": "fleet-default"
|
||||
},
|
||||
"policy": {
|
||||
"scopeApproved": true,
|
||||
"fieldSet": "studio-current-position-v1"
|
||||
}
|
||||
"facts": [
|
||||
{
|
||||
"sourceId": "gelios-unit-1001",
|
||||
"semanticType": "map.moving_object",
|
||||
"observedAt": "2026-07-16T12:00:00.000Z",
|
||||
"geometry": {
|
||||
"type": "Point",
|
||||
"coordinates": [37.6173, 55.7558]
|
||||
},
|
||||
"attributes": {
|
||||
"course_degrees": 90,
|
||||
"display_name": "Synthetic Gelios unit",
|
||||
"elevation_meters": 0,
|
||||
"hdop": 0.8,
|
||||
"horizontal_accuracy_meters": 4.2,
|
||||
"object_kind": "tracked_unit",
|
||||
"operational_status": "active",
|
||||
"position_source": "gelios",
|
||||
"position_valid": true,
|
||||
"quality_flags": [],
|
||||
"satellite_count": 11,
|
||||
"speed_kph": 0
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
|
|||
Loading…
Reference in New Issue