Compare commits
No commits in common. "567f1550abc2117db0636b0166ff9a25ed0c1088" and "e527812826acd4d0c227a09dabfd9d7140a0af92" have entirely different histories.
567f1550ab
...
e527812826
|
|
@ -16,9 +16,6 @@ build/
|
|||
.next/
|
||||
coverage/
|
||||
|
||||
# canonical deploy packages are transferred through the NAS inbox, not Git
|
||||
infra/deploy-artifacts/
|
||||
|
||||
# logs
|
||||
*.log
|
||||
logs/
|
||||
|
|
|
|||
|
|
@ -48,5 +48,3 @@ AI Workspace Assistant живёт в `services/ai-workspace-assistant` как о
|
|||
Ontology Core живёт в `services/ontology-core` как docs-first семантический слой для canonical entities, relations, aliases, guardrails, evidence, первого resolver MVP между OPS и ENGINE и policy MVP для NDC Core Assistant access. Он не владеет доменными БД HUB/OPS/ENGINE и не заменяет Launcher/HUB roles или OPS Gateway enforcement.
|
||||
|
||||
Map Gateway живёт в `services/map-gateway` как общий platform boundary для provider credentials, безопасного asset endpoint exchange, tile/3D Tiles proxy и persistent offline TileCache. Runtime cache не является source artifact и хранится в named volume/object storage, а не в Git.
|
||||
|
||||
Внешние бизнес-поставщики подключаются по `packages/external-provider-contract`: adapter конкретного API принадлежит изолированному NDC Agent L2 workflow, а Platform даёт один provider-neutral External Data Plane для raw retention, canonical facts, current/history projections и scoped read products. Connection instance несёт tenant scope, credential reference и collection profile, но не secret value. Provider/domain mapping, entity filtering и renderer logic не попадают в Platform Data Plane; writer capability привязана к immutable EDP binding и выдаётся только отдельному Engine provisioner, не shared internal bearer. Подробный канон — `docs/ADR_L2_OWNED_EXTERNAL_CONNECTORS.md`.
|
||||
|
|
|
|||
|
|
@ -1,143 +0,0 @@
|
|||
# ADR: канон External Provider Data Plane
|
||||
|
||||
Статус: **superseded**.
|
||||
Дата: 2026-07-13.
|
||||
Владелец решения: NODE.DC Platform.
|
||||
|
||||
> Заменено 2026-07-14 документом
|
||||
> [`ADR_L2_OWNED_EXTERNAL_CONNECTORS.md`](ADR_L2_OWNED_EXTERNAL_CONNECTORS.md).
|
||||
> Этот черновик неверно помещал provider adapter, normalisation и collection
|
||||
> policy в `services/<provider>-gateway`. Новое правило: adapter принадлежит
|
||||
> изолированному L2 workflow; Platform Data Plane остаётся provider-neutral.
|
||||
|
||||
## Контекст
|
||||
|
||||
Клиент может подключить к NODE.DC любой внешний продукт: телеметрию, ERP,
|
||||
роботов, энергетику, BIM-систему или иной источник данных. Gelios Pro для
|
||||
Gelios — первый конкретный поставщик для проверки канона, а не исключительная
|
||||
архитектурная ветка. Нельзя превращать Engine workflow, Ontology Core или общую БД Platform
|
||||
в место, куда попадают токены, raw payloads и частная логика каждого API.
|
||||
|
||||
Нужна повторяемая форма, в которой новый provider добавляется как отдельный
|
||||
adapter, но получает общие правила scope, секретов, collection, хранения,
|
||||
read-model, аудита и безопасной публикации в NDC.
|
||||
|
||||
## Решение
|
||||
|
||||
1. В `platform/packages/external-provider-contract` живёт общий versioned
|
||||
контракт интеграции. Это не runtime и не база данных. Он задаёт форму
|
||||
`provider`, `connection`, `capability catalog`, `credential reference`,
|
||||
`access scope`, `field policy`, `collection profile`, `retention policy`,
|
||||
`read model` и красный command-domain.
|
||||
2. Каждый provider получает самостоятельный app-owned adapter в
|
||||
`platform/services/<provider>-gateway`. Первый экземпляр —
|
||||
`platform/services/gelios-gateway`. Adapter владеет intake contract, rate
|
||||
budget, normalisation, collection policy, storage и своим internal
|
||||
read/realtime API. Сам provider secret остаётся в Engine Credentials и
|
||||
доступен только назначенному защищённому execution workflow.
|
||||
3. Один provider service может обслуживать много клиентов. Каждая строка,
|
||||
cursor, audit-event и read-model обязана иметь `tenant_id` и
|
||||
`connection_id`; видимость provider account сама по себе не является
|
||||
продуктовым scope. Для клиента с отдельными требованиями изоляции допустим
|
||||
отдельный deployment/database profile без изменения контракта.
|
||||
4. База принадлежит adapter-сервису, а не Ontology Core, Engine, Tasker или
|
||||
общему Platform Postgres. Для пространственно-временного Gelios-кейса
|
||||
базой служит PostgreSQL 16 + TimescaleDB + PostGIS (`gelios-postgres`).
|
||||
Другой provider может выбрать иной storage engine только через явный ADR,
|
||||
сохранив внешний контракт.
|
||||
5. Ontology Core хранит только provider-neutral и provider-specific смыслы,
|
||||
связи, правила и контракты. Он не хранит credentials, runtime telemetry,
|
||||
customer raw payloads или renderer objects. Enforcement остаётся в
|
||||
gateway/adapters. Engine Credentials хранит provider secret в границе
|
||||
специально назначенного execution workflow; значение не сериализуется в
|
||||
граф, ontology, логи, read-model или UI.
|
||||
6. Engine L2 Collector использует credential reference и получает разрешённые
|
||||
данные поставщика. Он передаёт в adapter только аутентифицированный нормализованный
|
||||
intake payload. Остальные L2 workflow получают исключительно scoped
|
||||
internal API/event contract и не читают gateway DB напрямую.
|
||||
|
||||
## Каноническая форма нового подключения
|
||||
|
||||
```text
|
||||
Client / tenant
|
||||
-> Engine Credential + protected Collector workflow
|
||||
-> provider connection instance
|
||||
-> provider adapter (safe intake policy + normalizer)
|
||||
-> provider-owned storage and projections
|
||||
-> internal read/realtime contract
|
||||
-> L2 workflow / approved interface binding
|
||||
-> renderer adapter
|
||||
```
|
||||
|
||||
Каждый новый provider добавляет только свой adapter package, capability
|
||||
catalog, schema mappings, scrubbed fixtures и domain ontology package. Он не
|
||||
добавляет отдельную схему доступа к Engine/Studio и не создаёт прямой путь из
|
||||
browser в provider API.
|
||||
|
||||
## Полнота данных без неконтролируемого объёма
|
||||
|
||||
«Предусмотреть все данные» означает каталогизировать каждую provider
|
||||
capability и поле, а не опрашивать весь account на максимальной частоте.
|
||||
Collection profile явно решает, какие safe-read capabilities, поля, scope и
|
||||
частота включены в конкретной connection instance.
|
||||
|
||||
| Слой | Что хранится | Режим |
|
||||
| --- | --- | --- |
|
||||
| Capability catalog | documented endpoint/read-field/command capability и его риск | versioned source + ontology |
|
||||
| Inventory/configuration | units, devices, groups, sensors, custom definitions | медленный reconcile |
|
||||
| Current projection | последняя разрешённая позиция, состояние и display fields | idempotent upsert |
|
||||
| Event history | нормализованные события и approved measurements | append-only, partitioned |
|
||||
| Raw envelope | полный safe-read ответ с provenance и hash | restricted cold layer, retention-bound |
|
||||
| Aggregates/features | rollups и признаки для аналитики/предиктива | derived, replaceable |
|
||||
|
||||
Raw envelope не выдаётся UI и не становится таблицей «всё в JSON навсегда».
|
||||
Вначале он может быть compressed/partitioned storage с метаданными в БД; при
|
||||
реальном объёме переносится в object storage, а PostgreSQL хранит immutable
|
||||
index, hash, policy и ссылку. Retention, raw depth и downsampling утверждаются
|
||||
после замера сообщений/сек, размера payload, требуемой истории, RPO/RTO и
|
||||
стоимости. Так инженер может запросить ранее не показанное поле из
|
||||
каталога/архива, не раздувая горячую read-модель.
|
||||
|
||||
## Realtime и интерфейс
|
||||
|
||||
Частота provider collection, обновления `current projection` и выдачи в UI —
|
||||
три разные настройки. Например, map consumer может получать выбранную
|
||||
read-модель раз в 3 секунды, но это не даёт ему права опрашивать Gelios раз в
|
||||
3 секунды или создавать отдельный polling loop на каждого зрителя.
|
||||
|
||||
Gateway сначала обновляет одну current projection и публикует change event.
|
||||
L2/Map binding затем может sampling/throttle этот поток по утверждённой
|
||||
настройке интерфейса. Источник истины для live state — gateway storage, не
|
||||
долгоживущий workflow и не Cesium session.
|
||||
|
||||
## Commands: моделируются, но не подключаются
|
||||
|
||||
Command templates, параметры, delivery states и audit входят в capability
|
||||
catalog и ontology полностью. Read adapter не содержит send route и не
|
||||
использует command capability. В будущем command execution создаётся только
|
||||
как отдельный `provider-command-gateway`/red-domain deployment с явным
|
||||
человеческим подтверждением, role/scope check, idempotency, audit и отдельным
|
||||
security review. До этого команда не может быть отправлена из collector, L2,
|
||||
Map или AI Workspace.
|
||||
|
||||
## Обязательные артефакты каждого adapter
|
||||
|
||||
- `provider manifest`: provider id, adapter version, auth modes, rate limits;
|
||||
- capability and field catalog: read/write classification, source evidence,
|
||||
pagination and error semantics;
|
||||
- connection profile: tenant, secret reference, approved scope, field policy,
|
||||
collection and retention profile;
|
||||
- normalised contract and migrations; scrubbed fixtures and contract tests;
|
||||
- health/metrics/audit without secrets or raw personal data;
|
||||
- ontology package with stable subjects, relations and guardrails;
|
||||
- internal read/realtime API contract; no browser/provider bypass.
|
||||
|
||||
## Не решено этим ADR
|
||||
|
||||
- конкретный deployment topology и HA/PITR target для каждого volume;
|
||||
- выбор object storage после real-volume measurement;
|
||||
- L2 stream execution contract и Module Studio binding implementation;
|
||||
- параметры first Gelios collection profile и owner-approved connection scope.
|
||||
|
||||
Gelios-specific применение этого решения описано в
|
||||
`docs/ADR_GELIOS_DATA_PLANE.md`.
|
||||
|
|
@ -1,107 +0,0 @@
|
|||
# ADR: Gelios adapter в External Provider Data Plane
|
||||
|
||||
Статус: **superseded**.
|
||||
Дата: 2026-07-13.
|
||||
Владелец решения: NODE.DC Platform.
|
||||
|
||||
> Заменено 2026-07-14 документом
|
||||
> [`ADR_L2_OWNED_EXTERNAL_CONNECTORS.md`](ADR_L2_OWNED_EXTERNAL_CONNECTORS.md).
|
||||
> Gelios остаётся domain/ontology примером, но не получает отдельный
|
||||
> provider-owned gateway: fetch, mapping и collection policy живут в его L2
|
||||
> connector instance; Platform предоставляет нейтральный Data Plane.
|
||||
|
||||
## Контекст
|
||||
|
||||
Gelios Pro поставляет непрерывную телеметрию, конфигурацию устройств и пространственные данные. В текущем доступе подтверждены 107 видимых units, из них 105 с `lastMsg`; окончательный connection scope должен быть зафиксирован allowlist-ом владельца. Это не данные Ontology Core и не данные Tasker. Они требуют отдельного контура для текущего состояния, истории, геозапросов, аналитики и будущих прогнозов.
|
||||
|
||||
Gelios является первым provider adapter, который подчиняется общему
|
||||
`docs/ADR_EXTERNAL_PROVIDER_DATA_PLANE.md`: connection instance хранит
|
||||
connection scope/policy, Engine Credentials владеет provider secret, gateway
|
||||
владеет storage/read-моделью, а ontology описывает значения, но не runtime
|
||||
data.
|
||||
|
||||
Физические trike-команды существуют в домене, но **не входят в ingestion или тестирование**. Для них позднее потребуется отдельный, ручной и аудируемый контур.
|
||||
|
||||
## Решение
|
||||
|
||||
1. Создать отдельный сервис `platform/services/gelios-gateway` как storage/read gateway:
|
||||
- не хранит и не получает provider token;
|
||||
- принимает только аутентифицированный safe-read intake от защищённого Engine L2 Collector workflow;
|
||||
- применяет allowlist и field policy до записи;
|
||||
- нормализует ответ в сущности пакета `gelios`;
|
||||
- публикует read-модель для Engine L2 workflow и Map View.
|
||||
2. В Engine появляется отдельный Gelios Credential и NDC Agent L2 collection profile:
|
||||
- credential скрывает access/refresh pair и его lifecycle;
|
||||
- credential привязан только к намеренно пошаренному Collector workflow;
|
||||
- Collector содержит allowlist read-capabilities и не имеет command transport;
|
||||
- ни секрет, ни provider response без нормализации не передаются в ontology, UI, Gateway или другие workflow.
|
||||
3. Выделить сервису собственную БД `gelios-postgres`, не деля её с Tasker, Authentik, Notification Core или Ontology Core. В будущей multi-tenant форме эта БД обслуживает несколько Gelios connection instances, но все records разделены `tenant_id` и `connection_id`.
|
||||
4. Базовый движок: PostgreSQL 16 с расширениями TimescaleDB и PostGIS.
|
||||
- Timescale hypertable хранит временные ряды и автоматически делит их по времени.
|
||||
- PostGIS хранит нормализованные точки и геометрии зон, а не renderer-объекты Cesium.
|
||||
- Данный выбор покрывает транзакционную конфигурацию, realtime upsert, исторические запросы, SQL-аналитику и географию одним контуром.
|
||||
5. **Не** использовать RabbitMQ Tasker как общую шину Gelios. Если измерения покажут, что прямой writer или число независимых потребителей не справляются, добавить в Gelios-контур NATS JetStream с durable pull consumers. Он даст replay, acknowledgement и контролируемое удержание сообщений.
|
||||
6. Engine level-2 Collector — единственная точка provider access; остальные workflow являются потребителями read-модели. Ontology Core остаётся только словарём и контрактами.
|
||||
|
||||
## Целевой поток
|
||||
|
||||
```text
|
||||
Gelios REST (safe read only)
|
||||
-> NDC Agent L2 collection profile + protected credential
|
||||
-> authenticated normalized intake
|
||||
-> Gelios Gateway: scope -> field policy -> normalizer
|
||||
-> gelios-postgres: current state + immutable telemetry history
|
||||
-> [при необходимости] NATS JetStream
|
||||
-> `fleet.positions.current.v1` read API / realtime subscription
|
||||
-> NDC workflow level 2
|
||||
-> Map View semantic binding
|
||||
-> Cesium renderer adapter
|
||||
```
|
||||
|
||||
Ни один шаг не получает права отправить команду устройству. Красный command-domain находится вне этого потока; metadata каталога команд сохраняется, но send transport не создаётся.
|
||||
|
||||
## Модель хранения v0
|
||||
|
||||
| Слой | Назначение | Минимальные записи |
|
||||
| --- | --- | --- |
|
||||
| Контроль | граница и повторяемость сбора | `access_scope`, `collection_run`, `ingestion_cursor`, endpoint/response metrics |
|
||||
| Каталог | стабильные сущности и их конфигурация | `unit`, `unit_group`, `tracker_device`, sensor/maintenance/custom-field definitions |
|
||||
| Current state | одна актуальная запись на unit для карты и интерфейса | `unit_current`, `position_fix`, approved operational status |
|
||||
| History | неизменяемые события с временем наблюдения и получения | `telemetry_snapshot`, selective `sensor_reading`, restricted raw payload reference |
|
||||
| Spatial | геометрии для запросов, не Cesium graphics | point/track/geozone with SRID 4326 |
|
||||
| Analytics | роллапы и признаки, не запросы по всему raw | hourly/daily aggregates, feature sets, model runs |
|
||||
| Audit | попытки, ошибки, политика, будущие команды | collection audit; separate command audit later |
|
||||
|
||||
`unit_current` обновляется idempotently по `unitSubjectId`. История записывается append-only с ключом дедупликации, включающим provider unit id, observed time и fingerprint сообщения. Все временные таблицы имеют `observed_at` и `received_at`: задержка поставщика не должна переписывать фактическое время на карте.
|
||||
|
||||
## Индексы и жизненный цикл
|
||||
|
||||
- Основной путь истории: `(unit_subject_id, observed_at DESC)`.
|
||||
- Пространственный индекс только для нормализованной geography/geometry; рендер-кэши в БД не храним.
|
||||
- Сырые `params`/raw messages — restricted, отдельно от публичной Studio read-model.
|
||||
- Политики retention, downsampling и резервного копирования должны быть утверждены до запуска history backfill. Они зависят от фактических msg/s, размера payload, нужной глубины истории, RPO/RTO и стоимости хранения.
|
||||
- Для предиктива держать recent/raw слой и отдельные часовые/дневные агрегаты. Timescale continuous aggregates позволяют сохранять длительную агрегированную историю после сокращения raw при корректно согласованных refresh и retention политиках.
|
||||
|
||||
## Нулевая итерация без лишней инфраструктуры
|
||||
|
||||
1. Зафиксировать owner-approved allowlist и видимые поля.
|
||||
2. Сделать только safe-read Engine Collector с ограничением по scope, rate limit, paging и cursor.
|
||||
3. В течение согласованного окна измерить: сообщений/сек, размер ответа, lag, дубликаты, задержку записи и нагрузку запросов карты.
|
||||
4. На фактах включить Timescale hypertables, PostGIS и retention policy; после этого решить, нужен ли JetStream сразу.
|
||||
5. Подать только `fleet.positions.current.v1`/approved current position в Map View. Историю и raw не отдавать в renderer напрямую.
|
||||
|
||||
## Что не решено этим ADR
|
||||
|
||||
- окончательное правило connection scope;
|
||||
- частота polling/возможность provider push;
|
||||
- сроки хранения raw, нормализованной истории и агрегатов;
|
||||
- RPO/RTO, репликация и production backup plan;
|
||||
- допуск к ручному command gateway. До отдельного решения отправка команд запрещена.
|
||||
|
||||
## Обоснование и источники
|
||||
|
||||
- [Timescale hypertables](https://docs.timescale.com/use-timescale/latest/hypertables/) — временные таблицы PostgreSQL автоматически партиционируются по времени.
|
||||
- [Timescale self-hosted installation](https://docs.timescale.com/self-hosted/latest/install/) — расширение разворачивается как self-hosted PostgreSQL-контур; production требует backup/PITR и HA-плана.
|
||||
- [Retention и continuous aggregates](https://docs.timescale.com/use-timescale/latest/data-retention/data-retention-with-continuous-aggregates/) — raw и агрегаты требуют согласованных lifecycle-политик.
|
||||
- [PostGIS](https://postgis.net/docs/en/) — PostgreSQL-расширение для spatial types и GiST R-tree индексов.
|
||||
- [NATS JetStream consumers](https://docs.nats.io/nats-concepts/jetstream/consumers) — durable consumers дают acknowledgement, повторную доставку и recovery; рекомендуются pull consumers для новых масштабируемых обработчиков.
|
||||
|
|
@ -1,274 +0,0 @@
|
|||
# ADR: L2-owned external connectors and provider-neutral Data Plane
|
||||
|
||||
Статус: **accepted**.
|
||||
Дата: 2026-07-14.
|
||||
Владелец решения: NODE.DC Platform.
|
||||
|
||||
## Контекст
|
||||
|
||||
NODE.DC — платформа разработки. Подключение внешнего API не должно создавать
|
||||
ещё один provider-specific сервис, в котором навсегда зашиты логика клиента,
|
||||
нормализация и правила отображения. Иначе каждый новый account или новый тип
|
||||
объекта превращается в изменение Platform runtime.
|
||||
|
||||
Нужна одна повторяемая форма: Platform даёт безопасные границы, контракт
|
||||
хранения и выдачи данных; изолированный workflow NDC Agent L2 является
|
||||
адаптером конкретного API и остаётся редактируемым через предоставленный
|
||||
Codex/Engine MCP.
|
||||
|
||||
Это решение заменяет provider-adapter часть
|
||||
`ADR_EXTERNAL_PROVIDER_DATA_PLANE.md` и все provider-specific runtime
|
||||
предписания `ADR_GELIOS_DATA_PLANE.md`. Они сохраняются как исторические
|
||||
черновики, но не являются каноном для новой реализации.
|
||||
|
||||
## Решение
|
||||
|
||||
### 1. Адаптер внешнего API принадлежит L2
|
||||
|
||||
Один изолированный L2 workflow представляет одно connection instance и
|
||||
является единственным местом для:
|
||||
|
||||
- обращения к API поставщика через ссылку на credential из Engine;
|
||||
- выбора read-capability, cursor/pagination, rate limit, retry и расписания;
|
||||
- получения raw envelope, разбиения большого ответа на native batch-операции;
|
||||
- семантического mapping к версии ontology и формирования canonical facts;
|
||||
- idempotency key, watermark и lifecycle источника;
|
||||
- deploy/run/execution/trace через Engine MCP.
|
||||
|
||||
L2 не хранит значение provider secret или writer token: он использует только
|
||||
Engine-bound credential references. Plaintext writer token выдаётся trusted
|
||||
provisioner ровно один раз при create/rotate binding и затем хранится только в
|
||||
Engine credential store, привязанном к этому L2; External Data Plane хранит
|
||||
лишь его hash. Provisioner читает отдельный runner-owned secret file, который
|
||||
read-only монтируется только в EDP и позднее в dedicated Engine provisioner,
|
||||
работающий под выделенным UID/GID `11006`; это не shared `.env`, не
|
||||
`NODEDC_INTERNAL_ACCESS_TOKEN` и не provider credential. До deployment этого
|
||||
provisioner `EXTERNAL_DATA_PLANE_PROVISIONING_ENABLED=false`, поэтому issuance
|
||||
routes fail closed. Token не попадает в L2 graph/profile, contract artifact,
|
||||
execution log/trace, raw envelope, Foundry или UI. Command/write-capabilities
|
||||
не получают transport route в read workflow.
|
||||
|
||||
Для новой учётной записи создаётся новый instance/profile этого workflow с
|
||||
другой credential reference и connection configuration. Это не требует
|
||||
изменения Platform source. Если поставщик раскрывает новый объект или поле,
|
||||
сначала расширяется ontology/capability contract, затем изменяется именно
|
||||
соответствующий L2 workflow.
|
||||
|
||||
### 2. Platform Data Plane нейтрален к provider и домену
|
||||
|
||||
Platform предоставляет один versioned **External Data Plane**. Он принимает
|
||||
batch по внутреннему аутентифицированному контракту, хранит raw envelope с
|
||||
retention/provenance, canonical facts, current/history projections и отдаёт
|
||||
scoped data products/realtime updates. В нём запрещены:
|
||||
|
||||
- ветвления по названию provider, customer, account, vehicle или renderer;
|
||||
- provider field mapping, unit allowlist, business filtering и visual rules;
|
||||
- provider token, прямой browser-to-provider доступ и command transport.
|
||||
|
||||
Его canonical write-форма, которую Data Plane валидирует и сохраняет после
|
||||
авторизации, содержит только нейтральные элементы:
|
||||
|
||||
```text
|
||||
source: providerId + tenantId + connectionId
|
||||
contract: contractVersion + ontologyRevision + dataProductId
|
||||
batch: runId + sequence + idempotencyKey + receivedAt
|
||||
raw: optional restricted ref + contentType + hash + retention policy
|
||||
facts: stable sourceId + semanticType + observedAt + attributes + geometry?
|
||||
```
|
||||
|
||||
Для нового L2 writer-а это не wire-форма. Он вызывает
|
||||
`POST /internal/data-plane/v1/data-products/:dataProductId/publish` с bearer
|
||||
writer token и формой `nodedc.data-product.publish/v1`, в которой вообще нет
|
||||
`source`, contract metadata, transport или persistence policy. Любой caller
|
||||
scope и scope headers запрещены.
|
||||
|
||||
External Data Plane разрешает token в собственный immutable writer binding
|
||||
`(tenantId, connectionId, providerId, allowedDataProductIds, expiresAt)`,
|
||||
проверяет active/non-expired binding, совпадение `providerId` и разрешение
|
||||
`dataProductId`, затем сам materializes canonical `source` с tenant/connection
|
||||
и только после этого валидирует и сохраняет batch. Присланный caller scope
|
||||
отклоняется, а не merge/override-ится. Изменение tenant, connection, provider,
|
||||
allowed data products или срока требует нового binding; для существующего
|
||||
binding допустимы только token rotation и revoke.
|
||||
|
||||
Выпуск, rotation и revoke binding принимаются только от dedicated provisioning
|
||||
principal с runner-owned secret file; он не монтируется в shared `.env` или L2.
|
||||
Shared legacy bearer не может создавать writer capability.
|
||||
|
||||
Идентификатор источника стабилен в пределах `(providerId, connectionId,
|
||||
sourceId)`. Отсутствие объекта в очередном ответе помечается lifecycle-state
|
||||
или временем последнего наблюдения; запись и её история не удаляются.
|
||||
|
||||
### 3. Граница данных и интерфейса
|
||||
|
||||
```text
|
||||
Provider API (safe read)
|
||||
-> Engine provider-credential reference
|
||||
-> L2 connector instance (adapter + mapping + batches)
|
||||
-> Engine writer-credential reference
|
||||
-> External Data Plane Data Product publish (scope materialized server-side)
|
||||
-> External Data Plane (generic persistence + projections)
|
||||
-> scoped data product / realtime stream
|
||||
-> Foundry binding / user interface
|
||||
```
|
||||
|
||||
Ontology Core описывает semantic types, связи, capability catalog и mapping
|
||||
versions, но не хранит runtime payloads или секреты. Foundry получает только
|
||||
approved data product и решает presentation: pins, visibility, layers and
|
||||
filters. Оно не получает provider endpoint или credential.
|
||||
|
||||
### 4. Полнота без entity allowlist
|
||||
|
||||
Collection profile выбирает разрешённые **read-capabilities**, а не список
|
||||
конкретных объектов. Если выбранный read endpoint возвращает все доступные
|
||||
источнику сущности, L2 передаёт все валидные элементы. Новые сущности
|
||||
добавляются автоматически; скрытие/отображение — задача data product/UI, а не
|
||||
сбора. Технические лимиты существуют только как размер batch, backpressure,
|
||||
quota и защита от повреждённого ответа, но не как бизнес-фильтр по ID.
|
||||
|
||||
### 5. Масштабирование и native NDC Agent nodes
|
||||
|
||||
L2 не запускает отдельный execution на каждую сущность. Он использует native
|
||||
nodes NDC Agent: HTTP Request, Split Out, Split In Batches/Loop Over Items,
|
||||
Aggregate, Postgres (когда нужен private workflow state), Set/If/Merge,
|
||||
Schedule Trigger and Respond to Webhook. Code node допустим только как малый
|
||||
преобразователь boundary-shape, когда эквивалентной native node нет; он не
|
||||
становится скрытым сервисом или provider database.
|
||||
|
||||
Shared telemetry/history не записывается L2 напрямую в физические таблицы
|
||||
Platform Postgres: это создало бы coupling к schema. Direct Postgres допустим
|
||||
для private state конкретного workflow или отдельной project-owned базы.
|
||||
Общий контур использует только External Data Plane contract и bulk batches.
|
||||
|
||||
### 5.1. Каталог custom nodes NODE.DC
|
||||
|
||||
Встроенные узлы runtime сохраняют свои штатные названия (`HTTP Request`,
|
||||
`Schedule Trigger`, `Loop Over Items` и т. п.). Любой узел, код которого
|
||||
принадлежит NODE.DC, обязан одновременно выполнять три условия:
|
||||
|
||||
- видимое имя начинается с `NDC `;
|
||||
- runtime type принадлежит package namespace `n8n-nodes-ndc.*`;
|
||||
- package-level test отклоняет публикацию узла без этого префикса и namespace.
|
||||
|
||||
Первый канонический набор состоит из:
|
||||
|
||||
- `NDC Data Product Publish` — runtime write boundary L2 → Data Plane;
|
||||
- `NDC Data Product Read` — scoped current snapshot read по reader grant;
|
||||
- `NDC Foundry Binding` — deploy/control-plane связь Data Product с
|
||||
`Application → Page → typed slot`, но не транспорт каждого realtime tick.
|
||||
|
||||
`NDC Foundry Output` не используется как техническое имя: оно ошибочно
|
||||
подразумевает прямой transport L2 → renderer. Provider-specific adapter,
|
||||
mapping и collection profile остаются логикой конкретного L2 workflow на
|
||||
штатных nodes; они не оформляются как custom NODE.DC nodes и не добавляют
|
||||
provider branch в Data Plane или Foundry.
|
||||
|
||||
`NDC Foundry Binding` отправляет только
|
||||
`nodedc.foundry.binding-upsert/v1`. Её постоянный L2 credential — отдельный
|
||||
revocable `ndc_fndbg_*` workload grant с allowlist точных
|
||||
`Application → Page → Binding → Slot → Data Product` targets. Короткоживущая
|
||||
`fnd1.*` capability интерактивного Foundry MCP, browser session, EDP
|
||||
writer/reader grant и `NODEDC_INTERNAL_ACCESS_TOKEN` для этой ноды запрещены.
|
||||
|
||||
Runtime node не содержит URL, endpoint, provider ID, tenant ID, connection ID,
|
||||
token, contract version или history cadence. Узел выбирает только разрешённый
|
||||
Data Product; всё остальное materializes из opaque writer/reader binding и
|
||||
immutable Data Product definition.
|
||||
|
||||
Package `n8n-nodes-ndc` устанавливается как штатный private community package в
|
||||
`/home/node/.n8n/nodes/node_modules`, чтобы n8n `PackageDirectoryLoader`
|
||||
сохранил namespace `n8n-nodes-ndc.*`. `~/.n8n/custom` и
|
||||
`N8N_CUSTOM_EXTENSIONS` запрещены для этого пакета: они создают namespace
|
||||
`CUSTOM.*`. Ручное копирование или `npm install` в живом container, patch
|
||||
Engine core и подмена built-in node запрещены. Production использует
|
||||
проверенный offline tarball, immutable release, atomic current/previous switch,
|
||||
read-only mount и restart всех n8n execution processes; acceptance завершается
|
||||
только когда Engine schema и MCP catalog возвращают точные package-qualified
|
||||
types.
|
||||
|
||||
### 5.2. Realtime delivery contract
|
||||
|
||||
Новый writer использует:
|
||||
|
||||
```text
|
||||
POST /internal/data-plane/v1/data-products/:dataProductId/publish
|
||||
Authorization: Bearer <opaque writer binding>
|
||||
```
|
||||
|
||||
Body имеет schema `nodedc.data-product.publish/v1` и содержит только batch
|
||||
identity и canonical facts. Data Plane одной транзакцией:
|
||||
|
||||
1. проверяет idempotency;
|
||||
2. обновляет current projection только более новым или изменившимся фактом;
|
||||
3. применяет declarative history policy (`none`, `all`, `sampled`);
|
||||
4. добавляет changed facts в durable patch outbox;
|
||||
5. фиксирует монотонный cursor.
|
||||
|
||||
Reader grant получает snapshot `nodedc.data-product.snapshot/v1`, затем
|
||||
подключается к SSE stream с `after=<snapshot.cursor>` или `Last-Event-ID`.
|
||||
Каждый outbox event имеет schema `nodedc.data-product.patch/v1`. Если cursor
|
||||
уже удалён retention policy, stream отвечает `409 resync_required`, и consumer
|
||||
повторяет snapshot. Browser не получает Data Plane token или scope: Foundry BFF
|
||||
разрешает persisted binding и читает server-only reader grant.
|
||||
|
||||
`snapshot+patch` в v1 является **bounded product contract**. Current projection
|
||||
одного scoped Data Product не может содержать больше 5000 entity keys
|
||||
`(sourceId, semanticType)`. Snapshot возвращает целую согласованную projection
|
||||
и один barrier cursor из одной `REPEATABLE READ` transaction. Query-параметр
|
||||
`limit` — только защитный потолок, а не размер страницы: если полная projection
|
||||
не помещается, Data Plane отвечает `413 data_product_snapshot_limit_exceeded`
|
||||
и consumer не начинает patch stream с неполной базой.
|
||||
|
||||
Наивная pagination current snapshot по `sourceId`, offset или независимо
|
||||
полученным page cursors запрещена: изменения между страницами могут быть
|
||||
пропущены или продублированы относительно patch cursor. Product с ожидаемой
|
||||
cardinality выше 5000 обязан **до включения realtime** выбрать один из двух
|
||||
отдельно версионируемых контрактов:
|
||||
|
||||
- stable partitioning в несколько Data Products, где partition key является
|
||||
частью definition, а каждый partition имеет собственный полный snapshot и
|
||||
независимый patch cursor;
|
||||
- новый query delivery contract с явно определёнными snapshot barrier,
|
||||
continuation cursor, query scope и правилами перехода к patch stream.
|
||||
|
||||
Такой query contract не входит в `nodedc.data-product.snapshot/v1` и не может
|
||||
быть имитирован полем `nextPageCursor`. До его отдельного утверждения runtime и
|
||||
`NDC Data Product Read` работают только с bounded products до 5000 сущностей.
|
||||
|
||||
### 6. Порядок первой реализации
|
||||
|
||||
1. Версионировать нейтральный intake/read contract и поднять External Data
|
||||
Plane без provider-specific logic.
|
||||
2. Пересобрать существующий L2 proof в native-node pipeline: safe read →
|
||||
split/batch → semantic mapping → generic append → safe summary. Никаких
|
||||
provider ID filters и provider gateway endpoint.
|
||||
3. Выполнить manual run, проверить execution/trace, idempotency and current
|
||||
projection; только затем включить Schedule Trigger.
|
||||
4. Provision immutable writer binding, one-time place its token into an opaque
|
||||
Engine credential, then switch the L2 to
|
||||
`/data-products/:dataProductId/publish` and remove caller scope/headers and
|
||||
the shared legacy credential.
|
||||
5. Подключить Foundry к data product current positions. История, геозоны и
|
||||
новые сущности добавляются отдельными L2 collection profiles and ontology
|
||||
revisions.
|
||||
|
||||
## Последствия
|
||||
|
||||
- `services/<provider>-gateway` не является шаблоном для новых интеграций.
|
||||
Существующий experimental code не расширяется и не деплоится как часть этого
|
||||
решения.
|
||||
- Data Plane может иметь собственную БД/Timescale/PostGIS implementation, но
|
||||
физическая схема остаётся внутренней деталью neutral service, а не API для
|
||||
L2 или Foundry.
|
||||
- Inline raw запрещён в L2 → Data Plane v1: до отдельного raw-vault допускается
|
||||
только restricted ref/hash либо отсутствие raw envelope. Retention reference
|
||||
вычисляется по серверному acceptance time, а не по timestamp из batch; Data
|
||||
Plane выполняет bounded retention sweeps после готовности сервиса и по таймеру.
|
||||
- Обычный Codex Desktop получает ровно те L2 grants, которые выданы владельцем
|
||||
connection/workflow, и может безопасно развивать adapter только в этой
|
||||
границе.
|
||||
- `POST /internal/data-plane/v1/intake` остаётся временным migration-only route:
|
||||
он выключен по умолчанию (`EXTERNAL_DATA_PLANE_LEGACY_INTAKE_ENABLED=false`),
|
||||
требует `NODEDC_INTERNAL_ACCESS_TOKEN`, canonical scoped batch и совпадающие
|
||||
scope headers. Новый или migrated L2 не получает этот shared token и не
|
||||
использует fallback в legacy route. После миграции всех writers он удаляется.
|
||||
|
|
@ -1,76 +0,0 @@
|
|||
# AI Workspace ↔ Ontology Core MCP
|
||||
|
||||
## Purpose
|
||||
|
||||
This is the read-only semantic path for a NODE.DC AI Workspace run:
|
||||
|
||||
```text
|
||||
Codex worker
|
||||
-> dynamic per-run MCP configuration
|
||||
-> public AI Workspace Hub (pairing-bound route)
|
||||
-> internal Ontology Core MCP
|
||||
-> freshly loaded ontology catalog and domain packages
|
||||
```
|
||||
|
||||
`ai-workspace-assistant` creates this MCP server entry at run-profile time for every Hub-connected executor when `AI_WORKSPACE_ONTOLOGY_MCP_ENABLED=true`. It is a platform runtime grant (`ontology:catalog:read`), not a user-owned installer secret and not a hard-coded copy of ontology rules in a prompt.
|
||||
|
||||
The worker receives only a pairing-bound Hub URL. The Hub verifies that the pairing agent is online and replaces any external authorization with the platform-internal token before forwarding to `ontology-core`. Ontology Core is internal-only and has no host port or reverse-proxy route.
|
||||
|
||||
## Read-only MCP surface
|
||||
|
||||
`nodedc_ontology` exposes only these tools:
|
||||
|
||||
- `ontology_status` — catalog counts, domain packages and a safe catalog hash;
|
||||
- `ontology_search` — canonical entities, relations and aliases;
|
||||
- `ontology_get_entity` — entity definition, aliases, relations and guardrails;
|
||||
- `ontology_get_guardrails` — semantic/safety rules and blocked conflations;
|
||||
- `ontology_resolve_context` — semantic route advice with source identifiers removed.
|
||||
|
||||
It intentionally does not expose filesystem paths, evidence ledgers, credentials, raw payloads, live telemetry, databases, command dispatch, workflow mutation or Studio controls.
|
||||
|
||||
## Boundary for Gelios and future data services
|
||||
|
||||
Ontology Core describes the canonical meanings and constraints:
|
||||
|
||||
```text
|
||||
gelios.unit -> gelios.telemetry_snapshot -> gelios.position_fix -> map.moving_object
|
||||
```
|
||||
|
||||
It does **not** serve the Gelios database or provider API. The future Gelios Gateway/Data API is a distinct capability with its own scope checks, data contract and transport. That capability may later be supplied to an AI Workspace run as another dynamic MCP server. Ontology then gives the assistant the names, relations, guardrails and allowed data-contract route; the data capability enforces access and returns live data.
|
||||
|
||||
This preserves one-way responsibility:
|
||||
|
||||
- Ontology Core: semantics, contracts, aliases, guardrails and context advice.
|
||||
- Gelios Gateway/Data API: access-scoped telemetry and history reads.
|
||||
- Command Gateway: separate red-domain command route, explicit confirmation and audit.
|
||||
- Engine L2: workflow orchestration using granted capabilities.
|
||||
- Studio: later presentation consumer, outside this implementation.
|
||||
|
||||
## Live catalog updates without AI Workspace rule redeploy
|
||||
|
||||
`loadCatalog()` runs for each Ontology MCP tool call; it does not cache the merged catalog. Therefore a new or edited domain package is visible to existing AI Workspace rules as soon as the catalog content is updated in the running Ontology Core container or mounted runtime catalog directory.
|
||||
|
||||
For a catalog-only release, update/restart **Ontology Core only**. The AI Workspace Assistant, Hub and Codex worker configuration do not need a rules redeploy because the MCP tool names and route stay stable. `NODEDC_ONTOLOGY_CATALOG_ROOT` may point Ontology Core at a separately managed catalog directory when a runtime-mounted catalog is required.
|
||||
|
||||
A Hub/Assistant rollout is needed only if the MCP transport, authorization contract or tool definitions themselves change.
|
||||
|
||||
## Safety properties
|
||||
|
||||
- The MCP server accepts only internal bearer authentication from AI Hub.
|
||||
- A browser `Origin` is rejected by default; allow specific origins only through `ONTOLOGY_MCP_ALLOWED_ORIGINS` if an intentional browser transport is introduced.
|
||||
- The Hub does not pass browser cookies or external `Authorization` downstream.
|
||||
- The Hub route is pairing-bound and is unavailable when the worker is offline.
|
||||
- All current tools are read-only. There is no generic ontology write API.
|
||||
|
||||
## Validation
|
||||
|
||||
```text
|
||||
cd platform/services/ontology-core
|
||||
npm run validate
|
||||
npm run smoke:mcp
|
||||
|
||||
cd ../../
|
||||
node --check services/ai-workspace-assistant/src/server.mjs
|
||||
node --check services/ai-workspace-hub/src/server.mjs
|
||||
docker compose --env-file infra/.env -f infra/docker-compose.dev.yml config
|
||||
```
|
||||
|
|
@ -51,35 +51,6 @@ NODEDC_INTERNAL_ACCESS_TOKEN=change-me-generate-with-infra-scripts-init-dev-env
|
|||
COOKIE_DOMAIN=.local.nodedc
|
||||
COOKIE_SECURE=false
|
||||
|
||||
# External Data Plane — provider-neutral storage owned by the Platform. This
|
||||
# password is a database credential only; never reuse NODEDC_INTERNAL_ACCESS_TOKEN.
|
||||
EXTERNAL_DATA_PLANE_TIMESCALE_IMAGE=timescale/timescaledb-ha:pg16.14-ts2.28.2-all
|
||||
EXTERNAL_DATA_PLANE_PG_DB=nodedc_data_plane
|
||||
EXTERNAL_DATA_PLANE_PG_USER=nodedc_data_plane
|
||||
EXTERNAL_DATA_PLANE_PG_PASS=change-me-generate-with-infra-scripts-init-dev-env
|
||||
EXTERNAL_DATA_PLANE_HOST_BIND=127.0.0.1:18106
|
||||
EXTERNAL_DATA_PLANE_DATABASE_POOL_SIZE=10
|
||||
EXTERNAL_DATA_PLANE_RAW_RETENTION_DAYS=14
|
||||
EXTERNAL_DATA_PLANE_MAX_BATCH_BYTES=5242880
|
||||
EXTERNAL_DATA_PLANE_MAX_FACTS_PER_PUBLISH=5000
|
||||
EXTERNAL_DATA_PLANE_MAX_ATTRIBUTES_BYTES_PER_FACT=65536
|
||||
EXTERNAL_DATA_PLANE_MAX_PATCH_OPERATIONS=500
|
||||
EXTERNAL_DATA_PLANE_MAX_PATCH_BYTES=262144
|
||||
EXTERNAL_DATA_PLANE_PATCH_RETENTION_MS=3600000
|
||||
EXTERNAL_DATA_PLANE_RECEIPT_RETENTION_MS=604800000
|
||||
EXTERNAL_DATA_PLANE_RETENTION_DELETE_LIMIT=10000
|
||||
EXTERNAL_DATA_PLANE_STREAM_HEARTBEAT_MS=20000
|
||||
EXTERNAL_DATA_PLANE_STREAM_POLL_MS=1000
|
||||
EXTERNAL_DATA_PLANE_MAX_READER_STREAMS=10
|
||||
EXTERNAL_DATA_PLANE_WRITER_BINDING_MAX_TTL_DAYS=90
|
||||
EXTERNAL_DATA_PLANE_MAX_FUTURE_SKEW_SECONDS=300
|
||||
EXTERNAL_DATA_PLANE_RETENTION_SWEEP_MS=3600000
|
||||
EXTERNAL_DATA_PLANE_LEGACY_INTAKE_ENABLED=false
|
||||
# The writer-provisioner secret is not an env value. The root-owned deploy
|
||||
# runner keeps it in /volume1/docker/nodedc-platform/secrets/external-data-plane-provisioner/
|
||||
# and mounts it
|
||||
# only into External Data Plane and the future dedicated Engine provisioner.
|
||||
|
||||
# notification core
|
||||
NOTIFICATION_PG_DB=nodedc_notifications
|
||||
NOTIFICATION_PG_USER=nodedc_notifications
|
||||
|
|
@ -95,15 +66,8 @@ NODEDC_AI_WORKSPACE_ASSISTANT_URL=http://ai-workspace-assistant:18082
|
|||
AI_WORKSPACE_OPS_ENTITLEMENT_URL=http://host.docker.internal:4100/api/internal/v1/ai-workspace/entitlements
|
||||
AI_WORKSPACE_OPS_ENTITLEMENT_TOKEN=replace-with-ops-agent-gateway-internal-token
|
||||
AI_WORKSPACE_OPS_ENTITLEMENT_REQUIRED=false
|
||||
# Add Module Foundry only after its domain, Launcher handoff and Authentik group
|
||||
# are verified. Preserve existing adapters when adding this JSON member:
|
||||
# {"module-foundry":{"url":"https://<foundry-domain>/api/ai-workspace/entitlements","required":false}}
|
||||
# The generic adapter reuses the existing NODE.DC internal server credential;
|
||||
# never define a separate Foundry token for a browser or worker.
|
||||
AI_WORKSPACE_ENTITLEMENT_ADAPTERS_JSON=
|
||||
AI_WORKSPACE_ASSISTANT_ACTION_RELAY_ENABLED=true
|
||||
AI_WORKSPACE_ASSISTANT_ACTION_RELAY_ID=local-dev
|
||||
AI_WORKSPACE_ONTOLOGY_MCP_ENABLED=true
|
||||
|
||||
# AI Workspace Hub for downloaded Codex workers.
|
||||
# Default local development may use the deployed AI Hub only as a relay for remote Codex workers.
|
||||
|
|
@ -114,32 +78,6 @@ AI_WORKSPACE_HUB_HOST_BIND=127.0.0.1:18081
|
|||
AI_WORKSPACE_HUB_PUBLIC_URL=wss://ai-hub.nodedc.ru/api/ai-workspace/hub
|
||||
AI_WORKSPACE_HUB_INTERNAL_URL=https://ai-hub.nodedc.ru
|
||||
AI_WORKSPACE_HUB_FALLBACK_URLS=
|
||||
# The worker receives a per-run, pairing-bound read-only Ontology MCP URL through AI Hub.
|
||||
# Do not put Ontology Core tokens or catalog paths in worker configuration.
|
||||
AI_WORKSPACE_ONTOLOGY_MCP_PUBLIC_URL=
|
||||
ONTOLOGY_CORE_HOST_BIND=127.0.0.1:18104
|
||||
|
||||
# Gelios Gateway — storage/read service. Provider credentials belong to the
|
||||
# protected Engine Collector and are never configured in this service.
|
||||
GELIOS_TIMESCALE_IMAGE=timescale/timescaledb-ha:pg16.14-ts2.28.2-all
|
||||
GELIOS_PG_DB=nodedc_gelios
|
||||
GELIOS_PG_USER=nodedc_gelios
|
||||
GELIOS_PG_PASS=change-me-generate-with-infra-scripts-init-dev-env
|
||||
# URL-encode reserved characters in GELIOS_PG_PASS when forming this URL.
|
||||
GELIOS_DATABASE_URL=postgresql://nodedc_gelios:change-me-generate-with-infra-scripts-init-dev-env@gelios-postgres:5432/nodedc_gelios
|
||||
GELIOS_GATEWAY_HOST_BIND=127.0.0.1:18105
|
||||
# Explicit tenant and connection identifiers are deployment configuration;
|
||||
# do not encode a customer or pilot name in source defaults.
|
||||
GELIOS_TENANT_ID=replace-with-tenant-id
|
||||
GELIOS_CONNECTION_ID=gelios-connection-id
|
||||
# `allowlist` accepts only GELIOS_ALLOWED_UNIT_IDS. `all` accepts every unit
|
||||
# returned by this already-approved tenant + connection, including future units.
|
||||
GELIOS_UNIT_SCOPE=allowlist
|
||||
GELIOS_ALLOWED_UNIT_IDS=
|
||||
GELIOS_INTAKE_ENABLED=false
|
||||
GELIOS_RAW_RETENTION_DAYS=14
|
||||
# Presentation state only; it does not change provider collection cadence.
|
||||
GELIOS_POSITION_STALE_AFTER_MS=300000
|
||||
|
||||
# map gateway — keep the actual ion token only in local/staging environment files or Docker secrets.
|
||||
# Never copy it into workflow metadata, Git, frontend code, or a runtime cache key.
|
||||
|
|
|
|||
|
|
@ -78,11 +78,18 @@ Generated Authentik bootstrap credentials are stored only in `infra/.env`.
|
|||
|
||||
## Map Gateway и offline TileCache
|
||||
|
||||
`map-gateway` добавлен как общий платформенный сервис на `127.0.0.1:18103`. На NAS его mutable live-cache лежит в `/volume1/docker/nodedc-platform/map-gateway/live-tile-cache`; read-only offline snapshot — рядом в `offline-snapshot`. Это host bind mounts, поэтому папки видны через SMB как `nodedc-platform/map-gateway/`, но не попадают в Git, Docker image или deployment artifact.
|
||||
`map-gateway` добавлен как общий платформенный сервис на `127.0.0.1:18103`. Его mutable live-cache монтируется как внешний named Docker volume `nodedc-platform_map-live-tile-cache`; offline snapshot — как `nodedc-platform_map-offline-snapshot`. Они не пишутся в `platform/`, не попадают в Git и не включаются в Docker image.
|
||||
|
||||
Обе папки создаёт root-owned `nodedc-deploy` при первом Map Gateway artifact. Agent не создаёт их напрямую через SMB и не кладёт в artifact. `docker compose down -v` их не удаляет. Очистка допустима только отдельной явно согласованной root-операцией при остановленном Gateway.
|
||||
Оба тома намеренно объявлены `external`: `docker compose down -v` не должен уничтожать уже записанные тайлы. На чистой машине они создаются один раз до первого запуска:
|
||||
|
||||
Для реального ion terrain/buildings положите `CESIUM_ION_TOKEN` только в неотслеживаемый `infra/.env` или deployment secret. Browser получает лишь публичный provider URL через same-origin Foundry proxy; Gateway добавляет asset credential только в исходящем private request. Детали API, cache modes и production access boundary описаны в `services/map-gateway/README.md`.
|
||||
```bash
|
||||
docker volume create nodedc-platform_map-live-tile-cache
|
||||
docker volume create nodedc-platform_map-offline-snapshot
|
||||
```
|
||||
|
||||
Очистка кэша — только осознанной командой `docker volume rm nodedc-platform_map-live-tile-cache` при остановленном Gateway.
|
||||
|
||||
Для реального ion terrain/buildings положите `CESIUM_ION_TOKEN` только в неотслеживаемый `infra/.env` или deployment secret. Studio получает asset-scoped endpoint token, а master token остаётся внутри Gateway. Детали API, cache modes и production access boundary описаны в `services/map-gateway/README.md`.
|
||||
|
||||
5. Bootstrap local Authentik groups and OIDC applications:
|
||||
|
||||
|
|
|
|||
|
|
@ -27,13 +27,6 @@ GROUP_SPECS = [
|
|||
("nodedc:taskmanager:admin", False),
|
||||
("nodedc:taskmanager:user", False),
|
||||
("nodedc:bim:access", False),
|
||||
# Module Foundry roles travel through the existing Authentik `groups`
|
||||
# claim and Launcher handoff. `access` remains a backward-compatible
|
||||
# member role until every existing assignment is migrated.
|
||||
("nodedc:module-foundry:admin", False),
|
||||
("nodedc:module-foundry:user", False),
|
||||
("nodedc:module-foundry:blocked", False),
|
||||
("nodedc:module-foundry:access", False),
|
||||
]
|
||||
|
||||
APP_SPECS = [
|
||||
|
|
@ -128,11 +121,7 @@ def ensure_user_groups(groups):
|
|||
user.groups.add(authentik_admins)
|
||||
|
||||
for name in groups:
|
||||
# The bootstrap owner must remain capable of entering Foundry after
|
||||
# the first provisioning run. Blocking is an explicit operator action,
|
||||
# never a default membership of the bootstrap principal.
|
||||
if name != "nodedc:module-foundry:blocked":
|
||||
user.groups.add(groups[name])
|
||||
user.groups.add(groups[name])
|
||||
return user
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -28,164 +28,6 @@ Supported components in this source:
|
|||
- `tasker`
|
||||
- `ops-agents`
|
||||
- `bim-viewer`
|
||||
- `n8n-private-extension`
|
||||
- `module-foundry`
|
||||
- `proxy-contur`
|
||||
- `dc-amd-proxy`
|
||||
|
||||
`n8n-private-extension` is a staging-only trust boundary for reviewed offline
|
||||
n8n private-node releases. Its artifact may contain exactly one digest-bound
|
||||
`n8n-nodes-ndc` release with `package.tgz`, `release.json` and
|
||||
`rollback.json`. The runner validates the inner npm tarball, rejects lifecycle
|
||||
scripts and runtime dependencies, refuses to overwrite an existing release,
|
||||
and seals the installed release root-owned/read-only under:
|
||||
|
||||
```text
|
||||
/volume1/docker/nodedc-platform/n8n-private-extensions/releases/n8n-nodes-ndc/<version>-<sha256-prefix>
|
||||
```
|
||||
|
||||
This component has no Compose file, service, container mutation or activation
|
||||
side effect. In particular, staging does **not** make the node visible to n8n.
|
||||
Activation remains an Engine-owned change: mount the reviewed immutable release
|
||||
at `/home/node/.n8n/nodes/node_modules/n8n-nodes-ndc`, atomically switch between
|
||||
verified releases, restart every n8n process, and accept only after MCP exposes
|
||||
the package-qualified `n8n-nodes-ndc.*` schemas. The Platform runner cannot
|
||||
cross that boundary and never runs `npm install` in a live container.
|
||||
|
||||
Build a verified offline release artifact:
|
||||
|
||||
```bash
|
||||
node infra/deploy-runner/build-n8n-private-extension-artifact.mjs \
|
||||
n8n-nodes-ndc-release-YYYYMMDD-NNN
|
||||
```
|
||||
|
||||
The builder is byte-reproducible and accepts exactly the three reviewed NDC
|
||||
runtime types:
|
||||
|
||||
- `n8n-nodes-ndc.ndcDataProductPublish`
|
||||
- `n8n-nodes-ndc.ndcDataProductRead`
|
||||
- `n8n-nodes-ndc.ndcFoundryBinding`
|
||||
|
||||
Their three opaque capability credential schemas are
|
||||
`ndcDataProductWriterApi`, `ndcDataProductReaderApi` and
|
||||
`ndcFoundryBindingApi`. A node description containing `usableAsTool` is
|
||||
rejected because n8n 2.3.2 would synthesize an additional `*Tool` runtime type
|
||||
and violate the exact-three activation contract. Run the positive and negative
|
||||
release-policy suite before publishing an artifact:
|
||||
|
||||
```bash
|
||||
PYTHONDONTWRITEBYTECODE=1 \
|
||||
python3 infra/deploy-runner/test_n8n_private_extension.py
|
||||
```
|
||||
|
||||
Release/rollback manifests use schema v2. Before a first activation, the
|
||||
Engine-owned activator must verify and record the current inactive state. That
|
||||
`verified_inactive` state is an allowed rollback baseline when no previous
|
||||
verified immutable release exists; later upgrades prefer the previous verified
|
||||
release. Rollback never deletes or mutates a staged release.
|
||||
|
||||
The historical `0.1.0` release remains immutable and must not be overwritten.
|
||||
Release `0.1.1-994756958861518e` is retained as rejected/inactive: its three
|
||||
node descriptions used `usableAsTool`, so n8n 2.3.2 exposed six NDC runtime
|
||||
types instead of the required three. It must not be activated, overwritten or
|
||||
deleted.
|
||||
|
||||
The corrected candidate is package version `0.1.2`, built with patch id
|
||||
`n8n-nodes-ndc-release-20260716-003`. It receives a new digest-bound release
|
||||
directory and remains inert after staging; only a separately reviewed
|
||||
Engine-owned activator may select it after exact MCP schema acceptance.
|
||||
|
||||
The paired Engine activation is built by
|
||||
`build-engine-n8n-private-extension-artifact.mjs`. It deliberately does not
|
||||
copy the dirty Engine `docker-compose.yml` and does not build an image. Instead
|
||||
it installs a narrowly scoped Compose override plus a strict transition
|
||||
descriptor. On apply, the runner validates the staged release again, verifies
|
||||
that the running n8n container and the NAS-local `2.3.2` tag resolve to the
|
||||
same immutable image ID, and extracts the package into the root-owned,
|
||||
read-only Engine release tree:
|
||||
|
||||
```text
|
||||
/volume2/nodedc-demo/n8n-private-extensions/releases/n8n-nodes-ndc/0.1.2-05e4b38b14b4a019/package
|
||||
```
|
||||
|
||||
The override sets `N8N_USER_FOLDER=/home/node`, which is required because the
|
||||
actual Engine service runs as root while the canonical community package path
|
||||
is below `/home/node/.n8n`. It enables loading but disables reinstall, mounts
|
||||
only the exact release read-only, and uses both Compose `pull_policy: never`
|
||||
and `docker compose up --pull never`. No registry access, lifecycle script,
|
||||
database `installed_packages` row or custom-extension loader is involved.
|
||||
|
||||
The runner pins the exact Engine service topology observed in source and
|
||||
rejects an added worker/webhook generation. Only the single actual `n8n`
|
||||
service is force-recreated with `--no-deps`; the
|
||||
Postgres service, `.n8n` data, encryption key and credentials remain intact.
|
||||
The apply gate verifies readiness, the running image/version, sealed mount,
|
||||
loader environment, package-loader node/credential sets, scoped loader logs,
|
||||
restart stability and content-exact pinned Engine MCP catalogs. The runner pins
|
||||
both the complete 434/385 inactive baseline and the reviewed 437/388
|
||||
activation catalogs, so a same-count substitution of any built-in schema is
|
||||
rejected. Any gate failure after
|
||||
mutation automatically restores the pre-apply catalogs/descriptor and
|
||||
force-recreates the previous verified runtime. Staged, sealed and failed
|
||||
releases are retained. The separate rollback artifact returns the first
|
||||
activation to the verified inactive 434-node/385-credential catalog baseline.
|
||||
|
||||
Run both policy suites before publishing the Engine pair:
|
||||
|
||||
```bash
|
||||
PYTHONDONTWRITEBYTECODE=1 \
|
||||
python3 infra/deploy-runner/test_n8n_private_extension.py
|
||||
PYTHONDONTWRITEBYTECODE=1 \
|
||||
python3 infra/deploy-runner/test_engine_n8n_private_extension.py
|
||||
```
|
||||
|
||||
For `platform` artifacts, the allowlist includes the versioned Ontology Core,
|
||||
the legacy Gelios experiment and the provider-neutral External Data Plane
|
||||
sources. An External Data Plane artifact builds only its image and starts
|
||||
`external-data-plane-postgres` plus `external-data-plane`; it never contains a
|
||||
provider credential, provider endpoint, collection schedule or command
|
||||
transport. Database credentials remain root-owned live `.env.synology`
|
||||
configuration and must not reuse `NODEDC_INTERNAL_ACCESS_TOKEN`.
|
||||
|
||||
The External Data Plane writer-provisioner credential is different: on the
|
||||
first relevant Platform apply, the root-owned runner creates
|
||||
`/volume1/docker/nodedc-platform/secrets/external-data-plane-provisioner/token`
|
||||
atomically in a dedicated UID/GID `11006` directory (directory `0500`, token
|
||||
`0400`). It is never an `.env` value or an artifact member; Compose mounts it
|
||||
read-only only into External Data Plane and, when implemented, its dedicated
|
||||
Engine provisioner running under the same restricted identity.
|
||||
|
||||
`module-foundry` is an independent, authenticated application component. Its
|
||||
artifact contains source and compose infrastructure only; its live
|
||||
`/volume1/docker/nodedc-platform/module-foundry/source/.env` is root-owned and
|
||||
never enters an artifact. The component reuses the existing internal platform
|
||||
credential for Launcher handoff validation and requires that runtime
|
||||
configuration before its first `apply`.
|
||||
|
||||
The Foundry ↔ Map Gateway signing key is not an application or `.env` setting.
|
||||
On the first relevant `platform` or `module-foundry` apply, the root-owned
|
||||
runner creates `/volume1/docker/nodedc-platform/secrets/map-gateway-admin-secret`
|
||||
atomically (root:gid 1000, mode `0640`). Both containers receive that file only
|
||||
as a read-only mount. The value is never printed, backed up with source,
|
||||
included in an artifact, or administered through Foundry.
|
||||
|
||||
`proxy-contur` is the canonical VPN egress for selected Map Gateway provider
|
||||
hosts. Its existing root-owned `PROXY_TOKEN` is copied by the runner into
|
||||
`/volume1/docker/nodedc-platform/secrets/map-egress-proxy-token` with
|
||||
`root:gid 1000`, mode `0640`, then mounted read-only only into Map Gateway.
|
||||
The value is neither printed nor contained in an artifact, Foundry setting, or
|
||||
browser response. Apply the `proxy-contur` artifact before the Platform Map
|
||||
Gateway artifact: it creates the private `nodedc-map-egress` Docker network.
|
||||
|
||||
`dc-amd-proxy` is the separate, staged connector for the neighbouring AMD VPN
|
||||
machine. Its active artifact attaches only to the private `nodedc-map-egress`
|
||||
network, exposes a narrow NAS-LAN pairing port, and has no direct provider
|
||||
egress. The runner preserves a `0700`, service-user-owned runtime directory
|
||||
for the one-time paired connector credential and synchronizes the existing
|
||||
private Map Gateway egress credential as a read-only file. Neither value is
|
||||
ever placed in an artifact, `.env`, browser response, or runner output. The
|
||||
separate Platform switch is applied only after the connector and pairing are
|
||||
verified; it does not alter NAS routes, VPN, DNS, or Tailscale.
|
||||
|
||||
Install or update the root-owned live runner on Synology:
|
||||
|
||||
|
|
|
|||
|
|
@ -1,44 +0,0 @@
|
|||
#!/usr/bin/env node
|
||||
import { createHash } from "node:crypto";
|
||||
import { spawnSync } from "node:child_process";
|
||||
import { cp, lstat, mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
|
||||
import { dirname, join, resolve } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
const scriptDir = dirname(fileURLToPath(import.meta.url));
|
||||
const platformRoot = resolve(scriptDir, "../..");
|
||||
const artifactDir = resolve(scriptDir, "../deploy-artifacts");
|
||||
const [patchId = "dc-amd-proxy-bootstrap-20260715-001", ...extra] = process.argv.slice(2);
|
||||
if (extra.length || !/^[A-Za-z0-9._-]{1,96}$/.test(patchId)) throw new Error("usage: build-dc-amd-proxy-artifact.mjs [patch-id]");
|
||||
|
||||
const files = [
|
||||
["services/dc-amd-proxy/Dockerfile", "Dockerfile"],
|
||||
["services/dc-amd-proxy/README.md", "README.md"],
|
||||
["services/dc-amd-proxy/docker-compose.yml", "docker-compose.yml"],
|
||||
["services/dc-amd-proxy/package.json", "package.json"],
|
||||
["services/dc-amd-proxy/server.mjs", "server.mjs"],
|
||||
];
|
||||
const stage = await mkdtemp(join(tmpdir(), "nodedc-dc-amd-proxy-artifact-"));
|
||||
const payload = join(stage, "payload");
|
||||
const target = join(artifactDir, `nodedc-${patchId}.tgz`);
|
||||
|
||||
try {
|
||||
await mkdir(payload, { recursive: true });
|
||||
for (const [source, destination] of files) await copySafe(resolve(platformRoot, source), join(payload, destination));
|
||||
await writeFile(join(stage, "manifest.env"), `id=${patchId}\ncomponent=dc-amd-proxy\ntype=app-overlay\n`, "utf8");
|
||||
await writeFile(join(stage, "files.txt"), `${files.map(([, destination]) => destination).join("\n")}\n`, "utf8");
|
||||
await mkdir(artifactDir, { recursive: true });
|
||||
const tar = spawnSync("python3", ["-c", "import sys,tarfile\nwith tarfile.open(sys.argv[1],'w:gz',format=tarfile.PAX_FORMAT) as a:\n [a.add(n,arcname=n,recursive=True) for n in ('manifest.env','files.txt','payload')]", target], { cwd: stage, encoding: "utf8" });
|
||||
if (tar.status !== 0) throw new Error(`tar_failed:${tar.stderr || tar.stdout}`);
|
||||
console.log(JSON.stringify({ ok: true, patchId, artifact: target, sha256: createHash("sha256").update(await readFile(target)).digest("hex") }, null, 2));
|
||||
} finally {
|
||||
await rm(stage, { recursive: true, force: true });
|
||||
}
|
||||
|
||||
async function copySafe(source, destination) {
|
||||
const info = await lstat(source);
|
||||
if (info.isSymbolicLink() || !info.isFile()) throw new Error(`source_file_rejected:${source}`);
|
||||
await mkdir(dirname(destination), { recursive: true });
|
||||
await cp(source, destination, { force: true });
|
||||
}
|
||||
|
|
@ -1,344 +0,0 @@
|
|||
#!/usr/bin/env node
|
||||
import { createHash } from "node:crypto";
|
||||
import { spawnSync } from "node:child_process";
|
||||
import { cp, mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
|
||||
import { createRequire, Module } from "node:module";
|
||||
import { tmpdir } from "node:os";
|
||||
import { dirname, join, resolve } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
const here = dirname(fileURLToPath(import.meta.url));
|
||||
const platformRoot = resolve(here, "../..");
|
||||
const engineRoot = resolve(platformRoot, "../NODEDC_ENGINE_INFRA");
|
||||
const artifactRoot = resolve(process.env.NODEDC_DEPLOY_ARTIFACT_DIR || join(here, "../deploy-artifacts"));
|
||||
const stageArtifact = resolve(
|
||||
process.env.NODEDC_N8N_EXTENSION_STAGE_ARTIFACT
|
||||
|| join(artifactRoot, "nodedc-n8n-private-extension-n8n-nodes-ndc-release-20260716-003.tgz"),
|
||||
);
|
||||
|
||||
const stageArtifactSha256 = "4601c16c57e5182996adf18d0163837511b27ab3f7cfdf97eac679418ee2d078";
|
||||
const releaseId = "0.1.2-05e4b38b14b4a019";
|
||||
const packageVersion = "0.1.2";
|
||||
const packageSha256 = "05e4b38b14b4a019ce1f6eee27b9e320094cb3560903bd68074966b3a1267af5";
|
||||
const n8nVersion = "2.3.2";
|
||||
const baseImage = "docker.n8n.io/n8nio/n8n:2.3.2";
|
||||
const architecture = "amd64";
|
||||
const generatedAt = "2026-07-15T21:51:41.000Z";
|
||||
const activationId = "engine-n8n-private-extension-20260716-003";
|
||||
const rollbackId = "engine-n8n-private-extension-rollback-20260716-003";
|
||||
|
||||
const transitionRoot = "nodedc-source/services/n8n/private-extensions";
|
||||
const descriptorRel = `${transitionRoot}/ndc-activation.json`;
|
||||
const overrideRel = `${transitionRoot}/docker-compose.ndc-private-extension.yml`;
|
||||
const schemaRoot = "nodedc-source/server/assets/n8n/schema/v2.3.2";
|
||||
const nodesCatalogRel = `${schemaRoot}/nodes.catalog.json`;
|
||||
const credentialsCatalogRel = `${schemaRoot}/credentials.catalog.json`;
|
||||
const metaRel = `${schemaRoot}/meta.json`;
|
||||
const iconRoot = "nodedc-source/server/assets/n8n/icons";
|
||||
const iconRel = `${iconRoot}/ndc.svg`;
|
||||
const darkIconRel = `${iconRoot}/ndc.dark.svg`;
|
||||
const sealedReleaseRelativePath = `n8n-private-extensions/releases/n8n-nodes-ndc/${releaseId}/package`;
|
||||
const runtimePackagePath = "/home/node/.n8n/nodes/node_modules/n8n-nodes-ndc";
|
||||
|
||||
const expectedNodeTypes = [
|
||||
"n8n-nodes-ndc.ndcDataProductPublish",
|
||||
"n8n-nodes-ndc.ndcDataProductRead",
|
||||
"n8n-nodes-ndc.ndcFoundryBinding",
|
||||
];
|
||||
const expectedCredentialTypes = [
|
||||
"ndcDataProductWriterApi",
|
||||
"ndcDataProductReaderApi",
|
||||
"ndcFoundryBindingApi",
|
||||
];
|
||||
const nodeModules = [
|
||||
["dist/nodes/NdcDataProductPublish/NdcDataProductPublish.node.js", "NdcDataProductPublish"],
|
||||
["dist/nodes/NdcDataProductRead/NdcDataProductRead.node.js", "NdcDataProductRead"],
|
||||
["dist/nodes/NdcFoundryBinding/NdcFoundryBinding.node.js", "NdcFoundryBinding"],
|
||||
];
|
||||
const credentialModules = [
|
||||
["dist/credentials/NdcDataProductWriterApi.credentials.js", "NdcDataProductWriterApi"],
|
||||
["dist/credentials/NdcDataProductReaderApi.credentials.js", "NdcDataProductReaderApi"],
|
||||
["dist/credentials/NdcFoundryBindingApi.credentials.js", "NdcFoundryBindingApi"],
|
||||
];
|
||||
|
||||
await mkdir(artifactRoot, { recursive: true });
|
||||
assertSha(await readFile(stageArtifact), stageArtifactSha256, "staging artifact");
|
||||
assertEngineBaseline(await readFile(join(engineRoot, "docker-compose.yml"), "utf8"));
|
||||
|
||||
const work = await mkdtemp(join(tmpdir(), "nodedc-engine-n8n-sealed-"));
|
||||
try {
|
||||
extractArchive(stageArtifact, work);
|
||||
const stagedRelease = join(work, "payload", "releases", "n8n-nodes-ndc", releaseId);
|
||||
const release = JSON.parse(await readFile(join(stagedRelease, "release.json"), "utf8"));
|
||||
assertRelease(release);
|
||||
assertSha(await readFile(join(stagedRelease, "package.tgz")), packageSha256, "private package");
|
||||
|
||||
const unpacked = join(work, "unpacked");
|
||||
await mkdir(unpacked);
|
||||
extractArchive(join(stagedRelease, "package.tgz"), unpacked);
|
||||
const packageRoot = join(unpacked, "package");
|
||||
const packageJson = JSON.parse(await readFile(join(packageRoot, "package.json"), "utf8"));
|
||||
assertPackage(packageJson);
|
||||
|
||||
const devNodeModules = join(platformRoot, "packages", "n8n-nodes-ndc", "node_modules");
|
||||
const nodePath = String(process.env.NODE_PATH || "").split(":").filter(Boolean);
|
||||
if (!nodePath.includes(devNodeModules)) nodePath.unshift(devNodeModules);
|
||||
process.env.NODE_PATH = nodePath.join(":");
|
||||
Module._initPaths();
|
||||
const packageRequire = createRequire(join(packageRoot, "package.json"));
|
||||
const privateNodes = nodeModules.map(([path, className], index) => {
|
||||
const NodeClass = packageRequire(join(packageRoot, path))[className];
|
||||
if (typeof NodeClass !== "function") throw new Error(`node_class_missing:${className}`);
|
||||
const description = structuredClone(new NodeClass().description);
|
||||
description.name = expectedNodeTypes[index];
|
||||
description.icon = { light: "file:ndc.svg", dark: "file:ndc.dark.svg" };
|
||||
if (Object.prototype.hasOwnProperty.call(description, "usableAsTool")) {
|
||||
throw new Error(`tool_variant_forbidden:${description.name}`);
|
||||
}
|
||||
return description;
|
||||
});
|
||||
const privateCredentials = credentialModules.map(([path, className]) => {
|
||||
const CredentialClass = packageRequire(join(packageRoot, path))[className];
|
||||
if (typeof CredentialClass !== "function") throw new Error(`credential_class_missing:${className}`);
|
||||
const description = structuredClone(new CredentialClass());
|
||||
description.icon = { light: "file:ndc.svg", dark: "file:ndc.dark.svg" };
|
||||
return description;
|
||||
});
|
||||
assertExact(privateNodes.map((item) => item.name), expectedNodeTypes, "node types");
|
||||
assertExact(privateCredentials.map((item) => item.name), expectedCredentialTypes, "credential types");
|
||||
|
||||
const baselineNodes = JSON.parse(gitFile(nodesCatalogRel));
|
||||
const baselineCredentials = JSON.parse(gitFile(credentialsCatalogRel));
|
||||
const baselineMeta = JSON.parse(gitFile(metaRel));
|
||||
assertBaselineCatalogs(baselineNodes, baselineCredentials, baselineMeta);
|
||||
const activeNodes = [...baselineNodes, ...privateNodes];
|
||||
const activeCredentials = [...baselineCredentials, ...privateCredentials];
|
||||
const activeMeta = {
|
||||
n8nVersion,
|
||||
generatedAt,
|
||||
source: `n8n-core+n8n-nodes-ndc@${packageVersion}`,
|
||||
nodeCount: activeNodes.length,
|
||||
credentialCount: activeCredentials.length,
|
||||
};
|
||||
|
||||
const activationDescriptor = descriptor("activate", expectedNodeTypes, expectedCredentialTypes, "verified_inactive");
|
||||
const rollbackDescriptor = descriptor("rollback-inactive", [], [], releaseId);
|
||||
const override = composeOverride();
|
||||
|
||||
await writeJson(join(engineRoot, nodesCatalogRel), activeNodes);
|
||||
await writeJson(join(engineRoot, credentialsCatalogRel), activeCredentials);
|
||||
await writeJson(join(engineRoot, metaRel), activeMeta);
|
||||
await writeJson(join(engineRoot, descriptorRel), activationDescriptor);
|
||||
await writeFile(join(engineRoot, overrideRel), override, "utf8");
|
||||
await cp(join(packageRoot, "dist/icons/ndc.svg"), join(engineRoot, iconRel), { force: true });
|
||||
await cp(join(packageRoot, "dist/icons/ndc.dark.svg"), join(engineRoot, darkIconRel), { force: true });
|
||||
|
||||
const activationEntries = [
|
||||
descriptorRel,
|
||||
overrideRel,
|
||||
nodesCatalogRel,
|
||||
credentialsCatalogRel,
|
||||
metaRel,
|
||||
iconRel,
|
||||
darkIconRel,
|
||||
];
|
||||
const activationArtifact = await buildArtifact(work, activationId, activationEntries, async (payload) => {
|
||||
for (const rel of activationEntries) {
|
||||
await cp(join(engineRoot, rel), join(payload, rel), { recursive: true, force: false });
|
||||
}
|
||||
});
|
||||
|
||||
const rollbackEntries = [descriptorRel, nodesCatalogRel, credentialsCatalogRel, metaRel];
|
||||
const rollbackArtifact = await buildArtifact(work, rollbackId, rollbackEntries, async (payload) => {
|
||||
await writeJson(join(payload, descriptorRel), rollbackDescriptor);
|
||||
await mkdir(join(payload, schemaRoot), { recursive: true });
|
||||
await writeFile(join(payload, nodesCatalogRel), `${JSON.stringify(baselineNodes, null, 2)}\n`, "utf8");
|
||||
await writeFile(join(payload, credentialsCatalogRel), `${JSON.stringify(baselineCredentials, null, 2)}\n`, "utf8");
|
||||
await writeFile(join(payload, metaRel), `${JSON.stringify(baselineMeta, null, 2)}\n`, "utf8");
|
||||
});
|
||||
|
||||
console.log(JSON.stringify({
|
||||
ok: true,
|
||||
releaseId,
|
||||
packageSha256,
|
||||
nodeTypes: expectedNodeTypes,
|
||||
credentialTypes: expectedCredentialTypes,
|
||||
activation: activationArtifact,
|
||||
rollback: rollbackArtifact,
|
||||
}, null, 2));
|
||||
} finally {
|
||||
await rm(work, { recursive: true, force: true });
|
||||
}
|
||||
|
||||
function descriptor(action, nodeTypes, credentialTypes, expectedCurrent) {
|
||||
return {
|
||||
schemaVersion: "nodedc.engine-n8n-private-extension-transition/v1",
|
||||
action,
|
||||
releaseId,
|
||||
packageVersion,
|
||||
packageSha256,
|
||||
n8nVersion,
|
||||
baseImage,
|
||||
baseImageArchitecture: architecture,
|
||||
baseImageIdentityPolicy: "running-container-and-local-tag-must-match",
|
||||
sealedReleaseRelativePath,
|
||||
composeOverride: overrideRel,
|
||||
runtimePackagePath,
|
||||
topologyServices: ["n8n"],
|
||||
expectedCurrent,
|
||||
expectedNodeTypes: nodeTypes,
|
||||
expectedCredentialTypes: credentialTypes,
|
||||
rollbackBaseline: "verified_inactive",
|
||||
};
|
||||
}
|
||||
|
||||
function composeOverride() {
|
||||
const health = "const http=require('http');const req=http.get('http://127.0.0.1:5678/healthz/readiness',r=>{r.resume();process.exit(r.statusCode===200?0:1)});req.on('error',()=>process.exit(1));req.setTimeout(4000,()=>{req.destroy();process.exit(1)});";
|
||||
return [
|
||||
"services:",
|
||||
" n8n:",
|
||||
` image: ${baseImage}`,
|
||||
" platform: linux/amd64",
|
||||
" pull_policy: never",
|
||||
" environment:",
|
||||
" N8N_USER_FOLDER: /home/node",
|
||||
" N8N_COMMUNITY_PACKAGES_ENABLED: \"true\"",
|
||||
" N8N_COMMUNITY_PACKAGES_PREVENT_LOADING: \"false\"",
|
||||
" N8N_REINSTALL_MISSING_PACKAGES: \"false\"",
|
||||
" volumes:",
|
||||
` - /volume2/nodedc-demo/${sealedReleaseRelativePath}:${runtimePackagePath}:ro`,
|
||||
" healthcheck:",
|
||||
` test: ${JSON.stringify(["CMD", "node", "-e", health])}`,
|
||||
" interval: 10s",
|
||||
" timeout: 5s",
|
||||
" retries: 30",
|
||||
" start_period: 30s",
|
||||
" labels:",
|
||||
` nodedc.n8n-private-extension.release: ${releaseId}`,
|
||||
` nodedc.n8n-private-extension.package-sha256: ${packageSha256}`,
|
||||
"",
|
||||
].join("\n");
|
||||
}
|
||||
|
||||
async function buildArtifact(workRoot, id, entries, populate) {
|
||||
const stage = join(workRoot, id);
|
||||
const payload = join(stage, "payload");
|
||||
await mkdir(payload, { recursive: true });
|
||||
await populate(payload);
|
||||
await writeFile(join(stage, "manifest.env"), `id=${id}\ncomponent=engine\ntype=app-overlay\n`, "utf8");
|
||||
await writeFile(join(stage, "files.txt"), `${entries.join("\n")}\n`, "utf8");
|
||||
const artifact = join(artifactRoot, `nodedc-${id}.tgz`);
|
||||
run("python3", ["-c", canonicalTarScript(), artifact, stage]);
|
||||
return { id, artifact, sha256: sha(await readFile(artifact)), entries };
|
||||
}
|
||||
|
||||
function assertRelease(value) {
|
||||
if (value?.releaseId !== releaseId
|
||||
|| value?.package?.name !== "n8n-nodes-ndc"
|
||||
|| value?.package?.version !== packageVersion
|
||||
|| value?.package?.sha256 !== packageSha256
|
||||
|| value?.storage?.relativePath !== `releases/n8n-nodes-ndc/${releaseId}`) {
|
||||
throw new Error("staged_release_identity_mismatch");
|
||||
}
|
||||
}
|
||||
|
||||
function assertPackage(value) {
|
||||
if (value.name !== "n8n-nodes-ndc" || value.version !== packageVersion || value.private !== true) {
|
||||
throw new Error("package_identity_mismatch");
|
||||
}
|
||||
if (value.dependencies !== undefined) throw new Error("runtime_dependencies_forbidden");
|
||||
for (const name of ["preinstall", "install", "postinstall", "prepare", "prepack", "postpack"]) {
|
||||
if (value.scripts?.[name] !== undefined) throw new Error(`lifecycle_forbidden:${name}`);
|
||||
}
|
||||
assertExact(value.n8n?.nodes, nodeModules.map(([path]) => path), "package nodes");
|
||||
assertExact(value.n8n?.credentials, credentialModules.map(([path]) => path), "package credentials");
|
||||
}
|
||||
|
||||
function assertBaselineCatalogs(nodes, credentials, meta) {
|
||||
if (!Array.isArray(nodes) || nodes.length !== 434 || nodes.some((item) => String(item?.name || "").startsWith("n8n-nodes-ndc."))) {
|
||||
throw new Error("baseline_node_catalog_mismatch");
|
||||
}
|
||||
if (!Array.isArray(credentials) || credentials.length !== 385
|
||||
|| credentials.some((item) => expectedCredentialTypes.includes(String(item?.name || "")))) {
|
||||
throw new Error("baseline_credential_catalog_mismatch");
|
||||
}
|
||||
if (meta?.n8nVersion !== n8nVersion || meta?.nodeCount !== 434 || meta?.credentialCount !== 385) {
|
||||
throw new Error("baseline_meta_mismatch");
|
||||
}
|
||||
}
|
||||
|
||||
function assertEngineBaseline(compose) {
|
||||
const exactImage = `image: docker.n8n.io/n8nio/n8n:\${N8N_IMAGE_TAG:-${n8nVersion}}`;
|
||||
if (!compose.includes(exactImage)) throw new Error("engine_n8n_version_mismatch");
|
||||
if ((compose.match(/^ n8n:\s*$/gm) || []).length !== 1) throw new Error("engine_n8n_topology_mismatch");
|
||||
if (/^ n8n-(?:worker|webhook)|^ (?:worker|webhook):/gm.test(compose)) throw new Error("unexpected_n8n_process_service");
|
||||
if (compose.includes("N8N_CUSTOM_EXTENSIONS") || compose.includes("CUSTOM.")) throw new Error("custom_extension_loader_forbidden");
|
||||
}
|
||||
|
||||
function assertExact(actual, expected, label) {
|
||||
if (!Array.isArray(actual) || JSON.stringify(actual) !== JSON.stringify(expected)) {
|
||||
throw new Error(`${label.replaceAll(" ", "_")}_mismatch`);
|
||||
}
|
||||
}
|
||||
|
||||
function assertSha(bytes, expected, label) {
|
||||
const actual = sha(bytes);
|
||||
if (actual !== expected) throw new Error(`${label.replaceAll(" ", "_")}_sha256_mismatch:${actual}`);
|
||||
}
|
||||
|
||||
function gitFile(rel) {
|
||||
return run("git", ["show", `HEAD:${rel}`], engineRoot).stdout;
|
||||
}
|
||||
|
||||
async function writeJson(path, value) {
|
||||
await mkdir(dirname(path), { recursive: true });
|
||||
await writeFile(path, `${JSON.stringify(value, null, 2)}\n`, "utf8");
|
||||
}
|
||||
|
||||
function extractArchive(archive, destination) {
|
||||
const script = [
|
||||
"import pathlib, sys, tarfile",
|
||||
"src=pathlib.Path(sys.argv[1]); dst=pathlib.Path(sys.argv[2]).resolve()",
|
||||
"with tarfile.open(src, 'r:gz') as tf:",
|
||||
" for m in tf:",
|
||||
" p=pathlib.PurePosixPath(m.name)",
|
||||
" if p.is_absolute() or '..' in p.parts or any(x.startswith('._') for x in p.parts) or not (m.isfile() or m.isdir()): raise SystemExit('unsafe archive member')",
|
||||
" target=dst.joinpath(*p.parts)",
|
||||
" target.mkdir(parents=True, exist_ok=True) if m.isdir() else target.parent.mkdir(parents=True, exist_ok=True)",
|
||||
" if m.isfile():",
|
||||
" source=tf.extractfile(m)",
|
||||
" with open(target, 'xb') as out: out.write(source.read())",
|
||||
].join("\n");
|
||||
run("python3", ["-c", script, archive, destination]);
|
||||
}
|
||||
|
||||
function canonicalTarScript() {
|
||||
return [
|
||||
"import gzip, io, pathlib, sys, tarfile",
|
||||
"root=pathlib.Path(sys.argv[2])",
|
||||
"with open(sys.argv[1], 'wb') as out:",
|
||||
" with gzip.GzipFile(filename='', mode='wb', fileobj=out, compresslevel=9, mtime=0) as gz:",
|
||||
" with tarfile.open(fileobj=gz, mode='w', format=tarfile.PAX_FORMAT) as tar:",
|
||||
" for top in ('manifest.env','files.txt','payload'):",
|
||||
" p=root/top; paths=[p] + (sorted(p.rglob('*')) if p.is_dir() else [])",
|
||||
" for x in paths:",
|
||||
" info=tar.gettarinfo(str(x), arcname=x.relative_to(root).as_posix())",
|
||||
" info.uid=info.gid=0; info.uname=info.gname='root'; info.mtime=0; info.mode=0o755 if info.isdir() else 0o644",
|
||||
" with (open(x,'rb') if info.isfile() else io.BytesIO()) as src: tar.addfile(info, src if info.isfile() else None)",
|
||||
].join("\n");
|
||||
}
|
||||
|
||||
function sha(bytes) {
|
||||
return createHash("sha256").update(bytes).digest("hex");
|
||||
}
|
||||
|
||||
function run(command, args, cwd) {
|
||||
const result = spawnSync(command, args, {
|
||||
cwd,
|
||||
encoding: "utf8",
|
||||
maxBuffer: 128 * 1024 * 1024,
|
||||
stdio: ["ignore", "pipe", "pipe"],
|
||||
});
|
||||
if (result.status !== 0) throw new Error(`${command}_failed:${result.stderr || result.stdout}`);
|
||||
return result;
|
||||
}
|
||||
|
|
@ -1,68 +0,0 @@
|
|||
#!/usr/bin/env node
|
||||
import { createHash } from "node:crypto";
|
||||
import { spawnSync } from "node:child_process";
|
||||
import { cp, lstat, mkdir, mkdtemp, readdir, rm, writeFile } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { dirname, join, relative, resolve } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
const scriptDir = dirname(fileURLToPath(import.meta.url));
|
||||
const platformRoot = resolve(scriptDir, "../..");
|
||||
const artifactDir = resolve(scriptDir, "../deploy-artifacts");
|
||||
const [patchId = "external-data-plane-20260714-001", ...extra] = process.argv.slice(2);
|
||||
|
||||
if (extra.length || !/^[A-Za-z0-9._-]{1,96}$/.test(patchId)) {
|
||||
throw new Error("usage: build-external-data-plane-artifact.mjs [patch-id]");
|
||||
}
|
||||
|
||||
const files = [
|
||||
["infra/synology/docker-compose.external-data-plane.yml", "platform/docker-compose.external-data-plane.yml"],
|
||||
["services/external-data-plane", "platform/services/external-data-plane"],
|
||||
["packages/external-provider-contract", "platform/packages/external-provider-contract"],
|
||||
];
|
||||
const ignoredBasenames = new Set([".DS_Store", ".git", "node_modules"]);
|
||||
const stage = await mkdtemp(join(tmpdir(), "nodedc-external-data-plane-artifact-"));
|
||||
const payload = join(stage, "payload");
|
||||
const target = join(artifactDir, `nodedc-platform-${patchId}.tgz`);
|
||||
|
||||
try {
|
||||
await mkdir(payload, { recursive: true });
|
||||
for (const [sourceRelative, destinationRelative] of files) {
|
||||
await copySafe(resolve(platformRoot, sourceRelative), join(payload, destinationRelative));
|
||||
}
|
||||
await writeFile(join(stage, "manifest.env"), `id=${patchId}\ncomponent=platform\ntype=app-overlay\n`, "utf8");
|
||||
await writeFile(join(stage, "files.txt"), `${files.map(([, destination]) => destination).join("\n")}\n`, "utf8");
|
||||
await mkdir(artifactDir, { recursive: true });
|
||||
|
||||
const tar = spawnSync("python3", ["-c", [
|
||||
"import sys, tarfile",
|
||||
"with tarfile.open(sys.argv[1], 'w:gz', format=tarfile.PAX_FORMAT) as archive:",
|
||||
" [archive.add(name, arcname=name, recursive=True) for name in ('manifest.env', 'files.txt', 'payload')]",
|
||||
].join("\n"), target], { cwd: stage, encoding: "utf8" });
|
||||
if (tar.status !== 0) throw new Error(`tar_failed:${tar.stderr || tar.stdout}`);
|
||||
|
||||
const digest = createHash("sha256").update(await (await import("node:fs/promises")).readFile(target)).digest("hex");
|
||||
console.log(JSON.stringify({ ok: true, patchId, artifact: target, sha256: digest }, null, 2));
|
||||
} finally {
|
||||
await rm(stage, { recursive: true, force: true });
|
||||
}
|
||||
|
||||
async function copySafe(source, destination) {
|
||||
const sourceStat = await lstat(source);
|
||||
if (sourceStat.isSymbolicLink()) throw new Error(`source_symlink_rejected:${source}`);
|
||||
if (sourceStat.isFile()) {
|
||||
await mkdir(dirname(destination), { recursive: true });
|
||||
await cp(source, destination, { force: true, verbatimSymlinks: true });
|
||||
return;
|
||||
}
|
||||
if (!sourceStat.isDirectory()) throw new Error(`source_type_rejected:${source}`);
|
||||
|
||||
await mkdir(destination, { recursive: true });
|
||||
for (const entry of await readdir(source, { withFileTypes: true })) {
|
||||
if (ignoredBasenames.has(entry.name) || entry.name.startsWith(".env")) continue;
|
||||
const childSource = join(source, entry.name);
|
||||
const childDestination = join(destination, entry.name);
|
||||
if (entry.isSymbolicLink()) throw new Error(`source_symlink_rejected:${relative(platformRoot, childSource)}`);
|
||||
await copySafe(childSource, childDestination);
|
||||
}
|
||||
}
|
||||
|
|
@ -1,109 +0,0 @@
|
|||
#!/usr/bin/env node
|
||||
import { createHash } from "node:crypto";
|
||||
import { spawnSync } from "node:child_process";
|
||||
import { cp, lstat, mkdir, mkdtemp, readdir, rm, writeFile } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { dirname, join, relative, resolve } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
const scriptDir = dirname(fileURLToPath(import.meta.url));
|
||||
const platformRoot = resolve(scriptDir, "../..");
|
||||
const artifactDir = resolve(scriptDir, "../deploy-artifacts");
|
||||
const args = process.argv.slice(2);
|
||||
const gatewayOnly = args.includes("--gateway-only");
|
||||
const positionalArgs = args.filter((argument) => argument !== "--gateway-only");
|
||||
if (positionalArgs.length > 1) {
|
||||
throw new Error("usage: build-gelios-data-plane-artifact.mjs [patch-id] [--gateway-only]");
|
||||
}
|
||||
const patchId = positionalArgs[0] || "gelios-data-plane-20260713-001";
|
||||
|
||||
if (!/^[A-Za-z0-9._-]{1,96}$/.test(patchId)) {
|
||||
throw new Error("patch_id_must_contain_only_letters_digits_dot_underscore_hyphen");
|
||||
}
|
||||
|
||||
const fullDataPlaneFiles = [
|
||||
["infra/synology/docker-compose.platform-http.yml", "platform/docker-compose.platform-http.yml"],
|
||||
["services/ontology-core", "platform/ontology-core"],
|
||||
["services/ai-workspace-hub", "platform/ai-workspace-hub"],
|
||||
["services/ai-workspace-assistant", "platform/ai-workspace-assistant"],
|
||||
["services/gelios-gateway", "platform/gelios-gateway"],
|
||||
];
|
||||
// A policy/code update to an already deployed Gelios data plane must not
|
||||
// carry the common Platform compose file. The deploy runner treats that file
|
||||
// as a whole-Platform change. The existing Gelios service already uses the
|
||||
// live Platform env_file, so this overlay can safely recreate only Gelios.
|
||||
const files = gatewayOnly
|
||||
? [["services/gelios-gateway", "platform/gelios-gateway"]]
|
||||
: fullDataPlaneFiles;
|
||||
|
||||
const ignoredBasenames = new Set([".DS_Store", ".git", "node_modules"]);
|
||||
const stage = await mkdtemp(join(tmpdir(), "nodedc-gelios-artifact-"));
|
||||
const payload = join(stage, "payload");
|
||||
const target = join(artifactDir, `nodedc-platform-${patchId}.tgz`);
|
||||
|
||||
try {
|
||||
await mkdir(payload, { recursive: true });
|
||||
|
||||
for (const [sourceRelative, destinationRelative] of files) {
|
||||
const source = resolve(platformRoot, sourceRelative);
|
||||
const destination = join(payload, destinationRelative);
|
||||
await copySafe(source, destination);
|
||||
}
|
||||
|
||||
if (!gatewayOnly) {
|
||||
// The Assistant imports the deterministic catalog as a local sibling at
|
||||
// runtime. Its production Dockerfile therefore expects this directory
|
||||
// inside the Assistant build context. Keep the deploy artifact equivalent
|
||||
// to the canonical Synology staging layout without making a second source
|
||||
// copy in the repository.
|
||||
await copySafe(
|
||||
resolve(platformRoot, "services/ontology-core"),
|
||||
join(payload, "platform/ai-workspace-assistant/ontology-core"),
|
||||
);
|
||||
// This nested copy is source-only build input for the Assistant. The nested
|
||||
// service Dockerfile is neither used nor allowed by the production runner.
|
||||
await rm(join(payload, "platform/ai-workspace-assistant/ontology-core/Dockerfile"), { force: true });
|
||||
}
|
||||
|
||||
await writeFile(join(stage, "manifest.env"), `id=${patchId}\ncomponent=platform\ntype=app-overlay\n`, "utf8");
|
||||
await writeFile(join(stage, "files.txt"), `${files.map(([, destination]) => destination).join("\n")}\n`, "utf8");
|
||||
await mkdir(artifactDir, { recursive: true });
|
||||
|
||||
// macOS bsdtar includes AppleDouble sidecar files for extended attributes.
|
||||
// The root runner rejects those as unexpected members, so use stdlib tarfile
|
||||
// to generate a portable, data-only archive instead.
|
||||
const tar = spawnSync("python3", ["-c", [
|
||||
"import sys, tarfile",
|
||||
"with tarfile.open(sys.argv[1], 'w:gz', format=tarfile.PAX_FORMAT) as archive:",
|
||||
" [archive.add(name, arcname=name, recursive=True) for name in ('manifest.env', 'files.txt', 'payload')]",
|
||||
].join("\n"), target], {
|
||||
cwd: stage,
|
||||
encoding: "utf8",
|
||||
});
|
||||
if (tar.status !== 0) throw new Error(`tar_failed:${tar.stderr || tar.stdout}`);
|
||||
|
||||
const digest = createHash("sha256").update(await (await import("node:fs/promises")).readFile(target)).digest("hex");
|
||||
console.log(JSON.stringify({ ok: true, patchId, gatewayOnly, artifact: target, sha256: digest }, null, 2));
|
||||
} finally {
|
||||
await rm(stage, { recursive: true, force: true });
|
||||
}
|
||||
|
||||
async function copySafe(source, destination) {
|
||||
const sourceStat = await lstat(source);
|
||||
if (sourceStat.isSymbolicLink()) throw new Error(`source_symlink_rejected:${source}`);
|
||||
if (sourceStat.isFile()) {
|
||||
await mkdir(dirname(destination), { recursive: true });
|
||||
await cp(source, destination, { force: true, verbatimSymlinks: true });
|
||||
return;
|
||||
}
|
||||
if (!sourceStat.isDirectory()) throw new Error(`source_type_rejected:${source}`);
|
||||
|
||||
await mkdir(destination, { recursive: true });
|
||||
for (const entry of await readdir(source, { withFileTypes: true })) {
|
||||
if (ignoredBasenames.has(entry.name) || entry.name.startsWith(".env")) continue;
|
||||
const childSource = join(source, entry.name);
|
||||
const childDestination = join(destination, entry.name);
|
||||
if (entry.isSymbolicLink()) throw new Error(`source_symlink_rejected:${relative(platformRoot, childSource)}`);
|
||||
await copySafe(childSource, childDestination);
|
||||
}
|
||||
}
|
||||
|
|
@ -1,85 +0,0 @@
|
|||
#!/usr/bin/env node
|
||||
import { createHash } from "node:crypto";
|
||||
import { spawnSync } from "node:child_process";
|
||||
import { cp, lstat, mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { dirname, join, resolve } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
const scriptDir = dirname(fileURLToPath(import.meta.url));
|
||||
const platformRoot = resolve(scriptDir, "../..");
|
||||
const launcherRoot = resolve(platformRoot, "../../data/nodedc_launcher");
|
||||
const artifactDir = resolve(scriptDir, "../deploy-artifacts");
|
||||
const patchId = process.argv[2] || "module-foundry-hub-registration-20260714-001";
|
||||
const profile = process.argv[3] || "registration";
|
||||
|
||||
if (!/^[A-Za-z0-9._-]{1,96}$/.test(patchId)) {
|
||||
throw new Error("patch_id_must_contain_only_letters_digits_dot_underscore_hyphen");
|
||||
}
|
||||
|
||||
// DCPLATFORM-21: the production runner overlays this payload onto the existing
|
||||
// Launcher source and recreates only the `launcher` service. Keep corrective
|
||||
// patches to their exact reviewed file set instead of re-sending unrelated
|
||||
// registration sources.
|
||||
const registrationFiles = [
|
||||
"server/authentik-sync.mjs",
|
||||
"server/control-plane-store.mjs",
|
||||
"server/dev-server.mjs",
|
||||
"src/entities/service/types.ts",
|
||||
];
|
||||
|
||||
const filesByProfile = {
|
||||
registration: registrationFiles,
|
||||
"handoff-fix": ["server/dev-server.mjs"],
|
||||
};
|
||||
|
||||
const files = filesByProfile[profile];
|
||||
if (!files) {
|
||||
throw new Error(`unknown_profile:${profile}`);
|
||||
}
|
||||
|
||||
const stage = await mkdtemp(join(tmpdir(), "nodedc-launcher-foundry-artifact-"));
|
||||
const payloadRoot = join(stage, "payload");
|
||||
const artifact = join(artifactDir, `launcher-${patchId}.tgz`);
|
||||
const checksum = `${artifact}.sha256`;
|
||||
|
||||
try {
|
||||
await mkdir(payloadRoot, { recursive: true });
|
||||
|
||||
for (const relativePath of files) {
|
||||
const source = resolve(launcherRoot, relativePath);
|
||||
const destination = join(payloadRoot, relativePath);
|
||||
const stat = await lstat(source);
|
||||
if (!stat.isFile() || stat.isSymbolicLink()) {
|
||||
throw new Error(`source_file_rejected:${relativePath}`);
|
||||
}
|
||||
await mkdir(dirname(destination), { recursive: true });
|
||||
await cp(source, destination, { force: true, verbatimSymlinks: true });
|
||||
}
|
||||
|
||||
await writeFile(
|
||||
join(stage, "manifest.env"),
|
||||
`id=${patchId}\ncomponent=launcher\ntype=app-overlay\n`,
|
||||
"utf8",
|
||||
);
|
||||
await writeFile(join(stage, "files.txt"), `${files.join("\n")}\n`, "utf8");
|
||||
await mkdir(artifactDir, { recursive: true });
|
||||
|
||||
// macOS bsdtar may emit AppleDouble `._*` sidecars. DCPLATFORM-21 rejects
|
||||
// them, therefore create a data-only POSIX archive through stdlib tarfile.
|
||||
const tar = spawnSync("python3", ["-c", [
|
||||
"import sys, tarfile",
|
||||
"with tarfile.open(sys.argv[1], 'w:gz', format=tarfile.PAX_FORMAT) as archive:",
|
||||
" [archive.add(name, arcname=name, recursive=True) for name in ('manifest.env', 'files.txt', 'payload')]",
|
||||
].join("\n"), artifact], {
|
||||
cwd: stage,
|
||||
encoding: "utf8",
|
||||
});
|
||||
if (tar.status !== 0) throw new Error(`tar_failed:${tar.stderr || tar.stdout}`);
|
||||
|
||||
const digest = createHash("sha256").update(await readFile(artifact)).digest("hex");
|
||||
await writeFile(checksum, `${digest} ${artifact.split("/").at(-1)}\n`, "utf8");
|
||||
console.log(JSON.stringify({ ok: true, patchId, profile, artifact, checksum, sha256: digest, files }, null, 2));
|
||||
} finally {
|
||||
await rm(stage, { recursive: true, force: true });
|
||||
}
|
||||
|
|
@ -1,49 +0,0 @@
|
|||
#!/usr/bin/env node
|
||||
import { createHash } from "node:crypto";
|
||||
import { spawnSync } from "node:child_process";
|
||||
import { cp, lstat, mkdir, mkdtemp, readdir, readFile, rm, writeFile } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { dirname, join, relative, resolve } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
const scriptDir = dirname(fileURLToPath(import.meta.url));
|
||||
const platformRoot = resolve(scriptDir, "../..");
|
||||
const artifactDir = resolve(scriptDir, "../deploy-artifacts");
|
||||
const [patchId = "platform-map-gateway-20260714-001", ...extra] = process.argv.slice(2);
|
||||
if (extra.length || !/^[A-Za-z0-9._-]{1,96}$/.test(patchId)) throw new Error("usage: build-map-gateway-artifact.mjs [patch-id]");
|
||||
|
||||
const files = [
|
||||
["infra/synology/docker-compose.platform-http.yml", "platform/docker-compose.platform-http.yml"],
|
||||
["services/map-gateway", "platform/services/map-gateway"],
|
||||
];
|
||||
const ignored = new Set([".DS_Store", ".git", "node_modules"]);
|
||||
const stage = await mkdtemp(join(tmpdir(), "nodedc-map-gateway-artifact-"));
|
||||
const payload = join(stage, "payload");
|
||||
const target = join(artifactDir, `nodedc-platform-${patchId}.tgz`);
|
||||
|
||||
try {
|
||||
await mkdir(payload, { recursive: true });
|
||||
for (const [source, destination] of files) await copySafe(resolve(platformRoot, source), join(payload, destination));
|
||||
await writeFile(join(stage, "manifest.env"), `id=${patchId}\ncomponent=platform\ntype=app-overlay\n`, "utf8");
|
||||
await writeFile(join(stage, "files.txt"), `${files.map(([, destination]) => destination).join("\n")}\n`, "utf8");
|
||||
await mkdir(artifactDir, { recursive: true });
|
||||
const tar = spawnSync("python3", ["-c", "import sys,tarfile\nwith tarfile.open(sys.argv[1],'w:gz',format=tarfile.PAX_FORMAT) as a:\n [a.add(n,arcname=n,recursive=True) for n in ('manifest.env','files.txt','payload')]", target], { cwd: stage, encoding: "utf8" });
|
||||
if (tar.status !== 0) throw new Error(`tar_failed:${tar.stderr || tar.stdout}`);
|
||||
console.log(JSON.stringify({ ok: true, patchId, artifact: target, sha256: createHash("sha256").update(await readFile(target)).digest("hex") }, null, 2));
|
||||
} finally {
|
||||
await rm(stage, { recursive: true, force: true });
|
||||
}
|
||||
|
||||
async function copySafe(source, destination) {
|
||||
const info = await lstat(source);
|
||||
if (info.isSymbolicLink()) throw new Error(`source_symlink_rejected:${source}`);
|
||||
if (info.isFile()) { await mkdir(dirname(destination), { recursive: true }); await cp(source, destination, { force: true }); return; }
|
||||
if (!info.isDirectory()) throw new Error(`source_type_rejected:${source}`);
|
||||
await mkdir(destination, { recursive: true });
|
||||
for (const entry of await readdir(source, { withFileTypes: true })) {
|
||||
if (ignored.has(entry.name) || entry.name.startsWith(".env")) continue;
|
||||
const child = join(source, entry.name);
|
||||
if (entry.isSymbolicLink()) throw new Error(`source_symlink_rejected:${relative(platformRoot, child)}`);
|
||||
await copySafe(child, join(destination, entry.name));
|
||||
}
|
||||
}
|
||||
|
|
@ -1,85 +0,0 @@
|
|||
#!/usr/bin/env node
|
||||
import { createHash } from "node:crypto";
|
||||
import { spawnSync } from "node:child_process";
|
||||
import { cp, lstat, mkdir, mkdtemp, readdir, readFile, rm, writeFile } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { dirname, join, relative, resolve } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
const scriptDir = dirname(fileURLToPath(import.meta.url));
|
||||
const platformRoot = resolve(scriptDir, "../..");
|
||||
const workspaceRoot = resolve(platformRoot, "..");
|
||||
const foundryRoot = resolve(workspaceRoot, "NODEDC_DESIGN_GUIDELINE");
|
||||
const artifactDir = resolve(scriptDir, "../deploy-artifacts");
|
||||
const patchId = process.argv[2] || "module-foundry-bootstrap-20260714-001";
|
||||
|
||||
if (!/^[A-Za-z0-9._-]{1,96}$/.test(patchId)) {
|
||||
throw new Error("patch_id_must_contain_only_letters_digits_dot_underscore_hyphen");
|
||||
}
|
||||
|
||||
const files = [
|
||||
".dockerignore",
|
||||
".env.example",
|
||||
".gitignore",
|
||||
"Dockerfile",
|
||||
"package.json",
|
||||
"package-lock.json",
|
||||
"tsconfig.base.json",
|
||||
"infra/docker-compose.module-foundry.yml",
|
||||
"apps",
|
||||
"packages",
|
||||
"registry",
|
||||
"runtime-seed",
|
||||
"scripts",
|
||||
"server",
|
||||
];
|
||||
const ignoredBasenames = new Set([".DS_Store", ".git", "node_modules", "runtime-data", "dist"]);
|
||||
const stage = await mkdtemp(join(tmpdir(), "nodedc-module-foundry-artifact-"));
|
||||
const payload = join(stage, "payload");
|
||||
const target = join(artifactDir, `nodedc-module-foundry-${patchId}.tgz`);
|
||||
|
||||
try {
|
||||
await mkdir(payload, { recursive: true });
|
||||
for (const sourceRelative of files) {
|
||||
await copySafe(resolve(foundryRoot, sourceRelative), join(payload, sourceRelative));
|
||||
}
|
||||
|
||||
await writeFile(join(stage, "manifest.env"), `id=${patchId}\ncomponent=module-foundry\ntype=app-overlay\n`, "utf8");
|
||||
await writeFile(join(stage, "files.txt"), `${files.join("\n")}\n`, "utf8");
|
||||
await mkdir(artifactDir, { recursive: true });
|
||||
|
||||
const tar = spawnSync("python3", ["-c", [
|
||||
"import sys, tarfile",
|
||||
"with tarfile.open(sys.argv[1], 'w:gz', format=tarfile.PAX_FORMAT) as archive:",
|
||||
" [archive.add(name, arcname=name, recursive=True) for name in ('manifest.env', 'files.txt', 'payload')]",
|
||||
].join("\n"), target], {
|
||||
cwd: stage,
|
||||
encoding: "utf8",
|
||||
});
|
||||
if (tar.status !== 0) throw new Error(`tar_failed:${tar.stderr || tar.stdout}`);
|
||||
|
||||
const digest = createHash("sha256").update(await readFile(target)).digest("hex");
|
||||
console.log(JSON.stringify({ ok: true, patchId, artifact: target, sha256: digest }, null, 2));
|
||||
} finally {
|
||||
await rm(stage, { recursive: true, force: true });
|
||||
}
|
||||
|
||||
async function copySafe(source, destination) {
|
||||
const sourceStat = await lstat(source);
|
||||
if (sourceStat.isSymbolicLink()) throw new Error(`source_symlink_rejected:${relative(foundryRoot, source)}`);
|
||||
if (sourceStat.isFile()) {
|
||||
await mkdir(dirname(destination), { recursive: true });
|
||||
await cp(source, destination, { force: true, verbatimSymlinks: true });
|
||||
return;
|
||||
}
|
||||
if (!sourceStat.isDirectory()) throw new Error(`source_type_rejected:${source}`);
|
||||
|
||||
await mkdir(destination, { recursive: true });
|
||||
for (const entry of await readdir(source, { withFileTypes: true })) {
|
||||
if (ignoredBasenames.has(entry.name) || entry.name.startsWith(".env") || entry.name.endsWith(".tsbuildinfo")) continue;
|
||||
const childSource = join(source, entry.name);
|
||||
const childDestination = join(destination, entry.name);
|
||||
if (entry.isSymbolicLink()) throw new Error(`source_symlink_rejected:${relative(foundryRoot, childSource)}`);
|
||||
await copySafe(childSource, childDestination);
|
||||
}
|
||||
}
|
||||
|
|
@ -1,286 +0,0 @@
|
|||
#!/usr/bin/env node
|
||||
import { createHash } from "node:crypto";
|
||||
import { createRequire } from "node:module";
|
||||
import { spawnSync } from "node:child_process";
|
||||
import { cp, mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { dirname, join, resolve } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
const scriptDir = dirname(fileURLToPath(import.meta.url));
|
||||
const platformRoot = resolve(scriptDir, "../..");
|
||||
const packageRoot = resolve(platformRoot, "packages/n8n-nodes-ndc");
|
||||
const artifactDir = resolve(process.env.NODEDC_DEPLOY_ARTIFACT_DIR || resolve(scriptDir, "../deploy-artifacts"));
|
||||
const requireModule = createRequire(import.meta.url);
|
||||
const expectedPackageVersion = "0.1.2";
|
||||
const [patchId = "n8n-nodes-ndc-release-20260716-003", ...extra] = process.argv.slice(2);
|
||||
const expectedRuntimeNodes = [
|
||||
{
|
||||
file: "dist/nodes/NdcDataProductPublish/NdcDataProductPublish.node.js",
|
||||
exportName: "NdcDataProductPublish",
|
||||
name: "ndcDataProductPublish",
|
||||
},
|
||||
{
|
||||
file: "dist/nodes/NdcDataProductRead/NdcDataProductRead.node.js",
|
||||
exportName: "NdcDataProductRead",
|
||||
name: "ndcDataProductRead",
|
||||
},
|
||||
{
|
||||
file: "dist/nodes/NdcFoundryBinding/NdcFoundryBinding.node.js",
|
||||
exportName: "NdcFoundryBinding",
|
||||
name: "ndcFoundryBinding",
|
||||
},
|
||||
];
|
||||
const expectedN8nNodes = expectedRuntimeNodes.map((node) => node.file);
|
||||
const expectedRuntimeNodeTypes = expectedRuntimeNodes.map((node) => `n8n-nodes-ndc.${node.name}`);
|
||||
const expectedN8nCredentials = [
|
||||
"dist/credentials/NdcDataProductWriterApi.credentials.js",
|
||||
"dist/credentials/NdcDataProductReaderApi.credentials.js",
|
||||
"dist/credentials/NdcFoundryBindingApi.credentials.js",
|
||||
];
|
||||
const expectedCredentialTypes = [
|
||||
"ndcDataProductWriterApi",
|
||||
"ndcDataProductReaderApi",
|
||||
"ndcFoundryBindingApi",
|
||||
];
|
||||
|
||||
if (extra.length || !/^[A-Za-z0-9._-]{1,96}$/.test(patchId)) {
|
||||
throw new Error("usage: build-n8n-private-extension-artifact.mjs [patch-id]");
|
||||
}
|
||||
|
||||
const packageJson = JSON.parse(await readFile(join(packageRoot, "package.json"), "utf8"));
|
||||
assertPackageSourcePolicy(packageJson);
|
||||
run("npm", ["test"], packageRoot);
|
||||
run("npm", ["run", "lint"], packageRoot);
|
||||
assertRuntimeNodePolicy(packageJson);
|
||||
|
||||
const stage = await mkdtemp(join(tmpdir(), "nodedc-n8n-private-extension-"));
|
||||
const packDir = join(stage, "pack");
|
||||
const payload = join(stage, "payload");
|
||||
|
||||
try {
|
||||
await mkdir(packDir, { recursive: true });
|
||||
const pack = run("npm", ["pack", "--ignore-scripts", "--json", "--pack-destination", packDir], packageRoot);
|
||||
const packResult = JSON.parse(pack.stdout);
|
||||
if (!Array.isArray(packResult) || packResult.length !== 1) throw new Error("npm_pack_result_invalid");
|
||||
const metadata = packResult[0];
|
||||
assertPackedFilePolicy(metadata, packageJson);
|
||||
|
||||
const packedPath = join(packDir, metadata.filename);
|
||||
const canonicalPackedPath = join(packDir, "n8n-nodes-ndc.canonical.tgz");
|
||||
const canonicalPackageScript = [
|
||||
"import gzip, io, sys, tarfile",
|
||||
"members = []",
|
||||
"with tarfile.open(sys.argv[1], 'r:gz') as source:",
|
||||
" for original in source:",
|
||||
" if not (original.isfile() or original.isdir()):",
|
||||
" raise SystemExit('unsupported npm package member')",
|
||||
" content = b''",
|
||||
" if original.isfile():",
|
||||
" extracted = source.extractfile(original)",
|
||||
" if extracted is None:",
|
||||
" raise SystemExit('unreadable npm package member')",
|
||||
" content = extracted.read()",
|
||||
" members.append((original.name, original.isdir(), content))",
|
||||
"with open(sys.argv[2], 'wb') as output:",
|
||||
" with gzip.GzipFile(filename='', mode='wb', fileobj=output, compresslevel=9, mtime=0) as compressed:",
|
||||
" with tarfile.open(fileobj=compressed, mode='w', format=tarfile.PAX_FORMAT) as target:",
|
||||
" for name, is_dir, content in sorted(members, key=lambda item: item[0]):",
|
||||
" member = tarfile.TarInfo(name)",
|
||||
" member.uid = member.gid = 0",
|
||||
" member.uname = member.gname = 'root'",
|
||||
" member.mtime = 0",
|
||||
" member.mode = 0o755 if is_dir else 0o644",
|
||||
" member.type = tarfile.DIRTYPE if is_dir else tarfile.REGTYPE",
|
||||
" member.size = 0 if is_dir else len(content)",
|
||||
" target.addfile(member, None if is_dir else io.BytesIO(content))",
|
||||
].join("\n");
|
||||
run("python3", ["-c", canonicalPackageScript, packedPath, canonicalPackedPath], stage);
|
||||
const packageBytes = await readFile(canonicalPackedPath);
|
||||
const packageSha256 = createHash("sha256").update(packageBytes).digest("hex");
|
||||
const releaseId = `${packageJson.version}-${packageSha256.slice(0, 16)}`;
|
||||
const relativeReleasePath = `releases/n8n-nodes-ndc/${releaseId}`;
|
||||
const releaseDir = join(payload, relativeReleasePath);
|
||||
await mkdir(releaseDir, { recursive: true });
|
||||
await cp(canonicalPackedPath, join(releaseDir, "package.tgz"), { force: false });
|
||||
|
||||
const rollbackBaselinePolicy = {
|
||||
allowed: [
|
||||
"previous_verified_immutable_release",
|
||||
"verified_inactive",
|
||||
],
|
||||
firstActivation: "verified_inactive",
|
||||
requiresPreActivationVerification: true,
|
||||
};
|
||||
const release = {
|
||||
schemaVersion: "nodedc.n8n-private-extension-release/v2",
|
||||
releaseId,
|
||||
package: {
|
||||
name: "n8n-nodes-ndc",
|
||||
version: packageJson.version,
|
||||
sha256: packageSha256,
|
||||
bytes: packageBytes.byteLength,
|
||||
runtimeTypePrefix: "n8n-nodes-ndc.",
|
||||
},
|
||||
storage: {
|
||||
relativePath: relativeReleasePath,
|
||||
immutable: true,
|
||||
},
|
||||
activation: {
|
||||
owner: "engine",
|
||||
status: "blocked_pending_engine_owned_mount",
|
||||
requiredCommunityPackagePath: "/home/node/.n8n/nodes/node_modules/n8n-nodes-ndc",
|
||||
requiresAtomicReleaseSwitch: true,
|
||||
requiresAllN8nProcessesRestart: true,
|
||||
requiresMcpSchemaAcceptance: true,
|
||||
rollbackBaselinePolicy,
|
||||
},
|
||||
};
|
||||
const rollback = {
|
||||
schemaVersion: "nodedc.n8n-private-extension-rollback/v2",
|
||||
releaseId,
|
||||
packageSha256,
|
||||
mode: "engine-owned-atomic-release-switch",
|
||||
baselinePolicy: rollbackBaselinePolicy,
|
||||
steps: [
|
||||
"select_verified_previous_release_or_preverified_inactive_baseline",
|
||||
"switch_engine_owned_mount_atomically",
|
||||
"restart_all_n8n_processes",
|
||||
"verify_mcp_schema_state_matches_selected_baseline",
|
||||
],
|
||||
forbidden: [
|
||||
"delete_active_release",
|
||||
"mutate_engine_core",
|
||||
"live_npm_install",
|
||||
],
|
||||
};
|
||||
await writeFile(join(releaseDir, "release.json"), `${JSON.stringify(release, null, 2)}\n`, "utf8");
|
||||
await writeFile(join(releaseDir, "rollback.json"), `${JSON.stringify(rollback, null, 2)}\n`, "utf8");
|
||||
await writeFile(join(stage, "manifest.env"), `id=${patchId}\ncomponent=n8n-private-extension\ntype=app-overlay\n`, "utf8");
|
||||
await writeFile(join(stage, "files.txt"), `${relativeReleasePath}\n`, "utf8");
|
||||
|
||||
await mkdir(artifactDir, { recursive: true });
|
||||
const target = join(artifactDir, `nodedc-n8n-private-extension-${patchId}.tgz`);
|
||||
const tarScript = [
|
||||
"import gzip, os, pathlib, sys, tarfile",
|
||||
"root = pathlib.Path(sys.argv[2])",
|
||||
"def clean(info):",
|
||||
" info.uid = info.gid = 0",
|
||||
" info.uname = info.gname = 'root'",
|
||||
" info.mtime = 0",
|
||||
" info.mode = 0o755 if info.isdir() else 0o644",
|
||||
" return info",
|
||||
"with open(sys.argv[1], 'wb') as output:",
|
||||
" with gzip.GzipFile(filename='', mode='wb', fileobj=output, compresslevel=9, mtime=0) as compressed:",
|
||||
" with tarfile.open(fileobj=compressed, mode='w', format=tarfile.PAX_FORMAT) as archive:",
|
||||
" for top in ('manifest.env', 'files.txt', 'payload'):",
|
||||
" path = root / top",
|
||||
" archive.add(path, arcname=top, recursive=False, filter=clean)",
|
||||
" if path.is_dir():",
|
||||
" for child in sorted(path.rglob('*'), key=lambda item: item.as_posix()):",
|
||||
" archive.add(child, arcname=child.relative_to(root).as_posix(), recursive=False, filter=clean)",
|
||||
].join("\n");
|
||||
run("python3", ["-c", tarScript, target, stage], stage);
|
||||
|
||||
const artifactSha256 = createHash("sha256").update(await readFile(target)).digest("hex");
|
||||
console.log(JSON.stringify({
|
||||
ok: true,
|
||||
patchId,
|
||||
artifact: target,
|
||||
artifactSha256,
|
||||
releaseId,
|
||||
packageSha256,
|
||||
nodeTypes: expectedRuntimeNodeTypes,
|
||||
credentialTypes: expectedCredentialTypes,
|
||||
activation: "blocked_pending_engine_owned_mount",
|
||||
}, null, 2));
|
||||
} finally {
|
||||
await rm(stage, { recursive: true, force: true });
|
||||
}
|
||||
|
||||
function assertPackageSourcePolicy(value) {
|
||||
if (value.name !== "n8n-nodes-ndc") throw new Error("package_name_invalid");
|
||||
if (value.version !== expectedPackageVersion) throw new Error("package_version_invalid");
|
||||
if (value.private !== true) throw new Error("package_must_remain_private");
|
||||
if (value.dependencies !== undefined) throw new Error("runtime_dependencies_forbidden");
|
||||
for (const lifecycle of ["preinstall", "install", "postinstall", "prepack", "prepare", "postpack"]) {
|
||||
if (value.scripts?.[lifecycle] !== undefined) throw new Error(`lifecycle_script_forbidden:${lifecycle}`);
|
||||
}
|
||||
assertExactRegistration(value.n8n?.nodes, expectedN8nNodes, "n8n_nodes");
|
||||
assertExactRegistration(value.n8n?.credentials, expectedN8nCredentials, "n8n_credentials");
|
||||
}
|
||||
|
||||
function assertRuntimeNodePolicy(packageJson) {
|
||||
const observedTypes = [];
|
||||
for (const expected of expectedRuntimeNodes) {
|
||||
const loaded = requireModule(join(packageRoot, expected.file));
|
||||
const NodeClass = loaded?.[expected.exportName];
|
||||
if (typeof NodeClass !== "function") throw new Error(`runtime_node_export_missing:${expected.exportName}`);
|
||||
const description = new NodeClass()?.description;
|
||||
if (!description || description.name !== expected.name) {
|
||||
throw new Error(`runtime_node_name_mismatch:${expected.exportName}`);
|
||||
}
|
||||
if ("usableAsTool" in description) {
|
||||
throw new Error(`runtime_tool_variant_forbidden:${expected.name}`);
|
||||
}
|
||||
observedTypes.push(`${packageJson.name}.${description.name}`);
|
||||
}
|
||||
if (JSON.stringify(observedTypes) !== JSON.stringify(expectedRuntimeNodeTypes)) {
|
||||
throw new Error("runtime_node_types_mismatch");
|
||||
}
|
||||
}
|
||||
|
||||
function assertExactRegistration(actual, expected, label) {
|
||||
if (!Array.isArray(actual)) throw new Error(`${label}_missing`);
|
||||
if (actual.length !== new Set(actual).size) throw new Error(`${label}_duplicate`);
|
||||
if (actual.length !== expected.length || expected.some((value) => !actual.includes(value))) {
|
||||
throw new Error(`${label}_mismatch`);
|
||||
}
|
||||
}
|
||||
|
||||
function assertPackedFilePolicy(metadata, sourcePackage) {
|
||||
if (metadata.name !== sourcePackage.name || metadata.version !== sourcePackage.version) {
|
||||
throw new Error("npm_pack_identity_mismatch");
|
||||
}
|
||||
if (!Number.isSafeInteger(metadata.size) || metadata.size < 1024 || metadata.size > 32 * 1024 * 1024) {
|
||||
throw new Error("npm_pack_size_invalid");
|
||||
}
|
||||
if (!Array.isArray(metadata.files) || metadata.files.length > 512) throw new Error("npm_pack_file_list_invalid");
|
||||
const paths = new Set();
|
||||
for (const file of metadata.files) {
|
||||
if (!file || typeof file.path !== "string" || paths.has(file.path)) throw new Error("npm_pack_file_invalid");
|
||||
paths.add(file.path);
|
||||
if (!(file.path === "README.md" || file.path === "package.json" || file.path.startsWith("dist/"))) {
|
||||
throw new Error(`npm_pack_path_forbidden:${file.path}`);
|
||||
}
|
||||
const parts = file.path.split("/");
|
||||
if (
|
||||
file.path.includes("\\")
|
||||
|| parts.some((part) => !part || part === "." || part === ".." || part.startsWith("."))
|
||||
|| file.mode !== 0o644
|
||||
) {
|
||||
throw new Error(`npm_pack_path_unsafe:${file.path}`);
|
||||
}
|
||||
}
|
||||
assertExactRegistration(
|
||||
[...paths].filter((value) => /^dist\/nodes\/.+\.node\.js$/.test(value)),
|
||||
expectedN8nNodes,
|
||||
"npm_pack_nodes",
|
||||
);
|
||||
assertExactRegistration(
|
||||
[...paths].filter((value) => /^dist\/credentials\/.+\.credentials\.js$/.test(value)),
|
||||
expectedN8nCredentials,
|
||||
"npm_pack_credentials",
|
||||
);
|
||||
for (const registered of [...sourcePackage.n8n.nodes, ...sourcePackage.n8n.credentials]) {
|
||||
if (!paths.has(registered)) throw new Error(`npm_pack_registration_missing:${registered}`);
|
||||
}
|
||||
}
|
||||
|
||||
function run(command, args, cwd) {
|
||||
const result = spawnSync(command, args, { cwd, encoding: "utf8", stdio: ["ignore", "pipe", "pipe"] });
|
||||
if (result.status !== 0) {
|
||||
throw new Error(`${command}_failed:${result.stderr || result.stdout}`);
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
|
@ -1,44 +0,0 @@
|
|||
#!/usr/bin/env node
|
||||
import { createHash } from "node:crypto";
|
||||
import { spawnSync } from "node:child_process";
|
||||
import { cp, lstat, mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
|
||||
import { dirname, join, resolve } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
const scriptDir = dirname(fileURLToPath(import.meta.url));
|
||||
const platformRoot = resolve(scriptDir, "../..");
|
||||
const artifactDir = resolve(scriptDir, "../deploy-artifacts");
|
||||
const [patchId = "cesium-egress-20260715-012", ...extra] = process.argv.slice(2);
|
||||
if (extra.length || !/^[A-Za-z0-9._-]{1,96}$/.test(patchId)) throw new Error("usage: build-proxy-contur-artifact.mjs [patch-id]");
|
||||
|
||||
const files = [
|
||||
["services/proxy-contur/Dockerfile", "Dockerfile"],
|
||||
["services/proxy-contur/README.md", "README.md"],
|
||||
["services/proxy-contur/docker-compose.yml", "docker-compose.yml"],
|
||||
["services/proxy-contur/package.json", "package.json"],
|
||||
["services/proxy-contur/server.js", "server.js"],
|
||||
];
|
||||
const stage = await mkdtemp(join(tmpdir(), "nodedc-proxy-contur-artifact-"));
|
||||
const payload = join(stage, "payload");
|
||||
const target = join(artifactDir, `nodedc-proxy-contur-${patchId}.tgz`);
|
||||
|
||||
try {
|
||||
await mkdir(payload, { recursive: true });
|
||||
for (const [source, destination] of files) await copySafe(resolve(platformRoot, source), join(payload, destination));
|
||||
await writeFile(join(stage, "manifest.env"), `id=${patchId}\ncomponent=proxy-contur\ntype=app-overlay\n`, "utf8");
|
||||
await writeFile(join(stage, "files.txt"), `${files.map(([, destination]) => destination).join("\n")}\n`, "utf8");
|
||||
await mkdir(artifactDir, { recursive: true });
|
||||
const tar = spawnSync("python3", ["-c", "import sys,tarfile\nwith tarfile.open(sys.argv[1],'w:gz',format=tarfile.PAX_FORMAT) as a:\n [a.add(n,arcname=n,recursive=True) for n in ('manifest.env','files.txt','payload')]", target], { cwd: stage, encoding: "utf8" });
|
||||
if (tar.status !== 0) throw new Error(`tar_failed:${tar.stderr || tar.stdout}`);
|
||||
console.log(JSON.stringify({ ok: true, patchId, artifact: target, sha256: createHash("sha256").update(await readFile(target)).digest("hex") }, null, 2));
|
||||
} finally {
|
||||
await rm(stage, { recursive: true, force: true });
|
||||
}
|
||||
|
||||
async function copySafe(source, destination) {
|
||||
const info = await lstat(source);
|
||||
if (info.isSymbolicLink() || !info.isFile()) throw new Error(`source_file_rejected:${source}`);
|
||||
await mkdir(dirname(destination), { recursive: true });
|
||||
await cp(source, destination, { force: true });
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
|
|
@ -1,285 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
import importlib.machinery
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import tarfile
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
SCRIPT_DIR = Path(__file__).resolve().parent
|
||||
PLATFORM_ROOT = SCRIPT_DIR.parent.parent
|
||||
ENGINE_ROOT = PLATFORM_ROOT.parent / "NODEDC_ENGINE_INFRA"
|
||||
RUNNER_PATH = SCRIPT_DIR / "nodedc-deploy"
|
||||
BUILDER_PATH = SCRIPT_DIR / "build-engine-n8n-private-extension-artifact.mjs"
|
||||
STAGE_ARTIFACT = (
|
||||
PLATFORM_ROOT
|
||||
/ "infra/deploy-artifacts"
|
||||
/ "nodedc-n8n-private-extension-n8n-nodes-ndc-release-20260716-003.tgz"
|
||||
)
|
||||
EXPECTED_NODES = [
|
||||
"n8n-nodes-ndc.ndcDataProductPublish",
|
||||
"n8n-nodes-ndc.ndcDataProductRead",
|
||||
"n8n-nodes-ndc.ndcFoundryBinding",
|
||||
]
|
||||
EXPECTED_CREDENTIALS = [
|
||||
"ndcDataProductWriterApi",
|
||||
"ndcDataProductReaderApi",
|
||||
"ndcFoundryBindingApi",
|
||||
]
|
||||
|
||||
|
||||
def load_runner():
|
||||
loader = importlib.machinery.SourceFileLoader("nodedc_engine_deploy_under_test", str(RUNNER_PATH))
|
||||
spec = importlib.util.spec_from_loader(loader.name, loader)
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
RUNNER = load_runner()
|
||||
|
||||
|
||||
class EngineN8nPrivateExtensionTest(unittest.TestCase):
|
||||
@classmethod
|
||||
def setUpClass(cls):
|
||||
cls.temporary = tempfile.TemporaryDirectory(prefix="nodedc-engine-n8n-policy-")
|
||||
cls.root = Path(cls.temporary.name)
|
||||
cls.results = []
|
||||
for index in range(2):
|
||||
output = cls.root / f"build-{index}"
|
||||
output.mkdir()
|
||||
env = os.environ.copy()
|
||||
env["NODEDC_DEPLOY_ARTIFACT_DIR"] = str(output)
|
||||
env["NODEDC_N8N_EXTENSION_STAGE_ARTIFACT"] = str(STAGE_ARTIFACT)
|
||||
result = subprocess.run(
|
||||
["node", str(BUILDER_PATH)],
|
||||
cwd=PLATFORM_ROOT,
|
||||
env=env,
|
||||
check=True,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
cls.results.append(json.loads(result.stdout))
|
||||
|
||||
@classmethod
|
||||
def tearDownClass(cls):
|
||||
cls.temporary.cleanup()
|
||||
|
||||
def artifact(self, build_index, kind):
|
||||
return Path(self.results[build_index][kind]["artifact"])
|
||||
|
||||
def test_engine_artifacts_are_byte_reproducible(self):
|
||||
for kind in ("activation", "rollback"):
|
||||
first = self.artifact(0, kind).read_bytes()
|
||||
second = self.artifact(1, kind).read_bytes()
|
||||
self.assertEqual(first, second)
|
||||
self.assertEqual(self.results[0][kind]["sha256"], self.results[1][kind]["sha256"])
|
||||
self.assertEqual(first[4:8], b"\0\0\0\0")
|
||||
self.assertEqual(first[3] & 0x08, 0)
|
||||
|
||||
def test_activation_and_rollback_pass_strict_runner_policy(self):
|
||||
expected = {"activation": ("activate", 7), "rollback": ("rollback-inactive", 4)}
|
||||
for kind, (action, entry_count) in expected.items():
|
||||
with self.subTest(kind=kind), tempfile.TemporaryDirectory() as directory:
|
||||
manifest, entries, payload = RUNNER.load_artifact(
|
||||
self.artifact(0, kind),
|
||||
Path(directory),
|
||||
)
|
||||
descriptor = RUNNER.read_engine_n8n_transition_descriptor(
|
||||
payload / RUNNER.ENGINE_N8N_TRANSITION_DESCRIPTOR_REL
|
||||
)
|
||||
self.assertEqual(manifest["component"], "engine")
|
||||
self.assertEqual(descriptor["action"], action)
|
||||
self.assertEqual(len(entries), entry_count)
|
||||
self.assertEqual(RUNNER.component_services("engine", entries), ("n8n",))
|
||||
self.assertEqual(RUNNER.component_builds("engine", entries), ())
|
||||
self.assertFalse(RUNNER.component_publish_dist("engine", entries))
|
||||
|
||||
def test_activation_catalog_is_exact_three_without_tool_variants(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
_manifest, _entries, payload = RUNNER.load_artifact(
|
||||
self.artifact(0, "activation"),
|
||||
Path(directory),
|
||||
)
|
||||
nodes = json.loads((payload / RUNNER.ENGINE_N8N_NODES_CATALOG_REL).read_text())
|
||||
credentials = json.loads((payload / RUNNER.ENGINE_N8N_CREDENTIALS_CATALOG_REL).read_text())
|
||||
private_nodes = [item for item in nodes if item.get("name", "").startswith("n8n-nodes-ndc.")]
|
||||
private_credentials = [item for item in credentials if item.get("name") in EXPECTED_CREDENTIALS]
|
||||
self.assertEqual([item["name"] for item in private_nodes], EXPECTED_NODES)
|
||||
self.assertEqual([item["name"] for item in private_credentials], EXPECTED_CREDENTIALS)
|
||||
self.assertTrue(all("usableAsTool" not in item for item in private_nodes))
|
||||
self.assertEqual((len(nodes), len(credentials)), (437, 388))
|
||||
|
||||
def test_rollback_catalog_restores_verified_inactive_baseline(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
_manifest, _entries, payload = RUNNER.load_artifact(
|
||||
self.artifact(0, "rollback"),
|
||||
Path(directory),
|
||||
)
|
||||
nodes = json.loads((payload / RUNNER.ENGINE_N8N_NODES_CATALOG_REL).read_text())
|
||||
credentials = json.loads((payload / RUNNER.ENGINE_N8N_CREDENTIALS_CATALOG_REL).read_text())
|
||||
self.assertEqual([item for item in nodes if item.get("name", "").startswith("n8n-nodes-ndc.")], [])
|
||||
self.assertEqual([item for item in credentials if item.get("name") in EXPECTED_CREDENTIALS], [])
|
||||
self.assertEqual((len(nodes), len(credentials)), (434, 385))
|
||||
|
||||
def test_compose_override_is_runner_derived_and_offline(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
_manifest, _entries, payload = RUNNER.load_artifact(
|
||||
self.artifact(0, "activation"),
|
||||
Path(directory),
|
||||
)
|
||||
descriptor = RUNNER.read_engine_n8n_transition_descriptor(
|
||||
payload / RUNNER.ENGINE_N8N_TRANSITION_DESCRIPTOR_REL
|
||||
)
|
||||
override = (payload / RUNNER.ENGINE_N8N_COMPOSE_OVERRIDE_REL).read_text()
|
||||
self.assertEqual(override, RUNNER.expected_engine_n8n_compose_override(descriptor))
|
||||
self.assertIn("pull_policy: never", override)
|
||||
self.assertIn("N8N_USER_FOLDER: /home/node", override)
|
||||
self.assertIn(":/home/node/.n8n/nodes/node_modules/n8n-nodes-ndc:ro", override)
|
||||
self.assertNotIn("N8N_CUSTOM_EXTENSIONS", override)
|
||||
self.assertNotIn("build:", override)
|
||||
|
||||
def test_unknown_descriptor_key_is_rejected(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
work = Path(directory)
|
||||
RUNNER.safe_extract(self.artifact(0, "activation"), work)
|
||||
descriptor_path = work / "payload" / RUNNER.ENGINE_N8N_TRANSITION_DESCRIPTOR_REL
|
||||
descriptor = json.loads(descriptor_path.read_text())
|
||||
descriptor["unexpected"] = True
|
||||
descriptor_path.write_text(json.dumps(descriptor))
|
||||
with self.assertRaisesRegex(RUNNER.DeployError, "keys mismatch"):
|
||||
RUNNER.validate_engine_n8n_transition(
|
||||
work / "payload",
|
||||
RUNNER.parse_files_list(work / "files.txt"),
|
||||
)
|
||||
|
||||
def test_core_catalog_substitution_is_rejected_even_when_counts_match(self):
|
||||
cases = (
|
||||
("activation", RUNNER.ENGINE_N8N_NODES_CATALOG_REL, "node"),
|
||||
("activation", RUNNER.ENGINE_N8N_CREDENTIALS_CATALOG_REL, "credential"),
|
||||
("rollback", RUNNER.ENGINE_N8N_NODES_CATALOG_REL, "node"),
|
||||
("rollback", RUNNER.ENGINE_N8N_CREDENTIALS_CATALOG_REL, "credential"),
|
||||
)
|
||||
for kind, relative_path, label in cases:
|
||||
with self.subTest(kind=kind, catalog=label), tempfile.TemporaryDirectory() as directory:
|
||||
work = Path(directory)
|
||||
RUNNER.safe_extract(self.artifact(0, kind), work)
|
||||
catalog_path = work / "payload" / relative_path
|
||||
catalog = json.loads(catalog_path.read_text())
|
||||
if label == "node":
|
||||
candidate = next(
|
||||
item for item in catalog
|
||||
if not item.get("name", "").startswith("n8n-nodes-ndc.")
|
||||
)
|
||||
candidate["name"] = "n8n-nodes-unreviewed.hiddenNode"
|
||||
else:
|
||||
candidate = next(
|
||||
item for item in catalog
|
||||
if item.get("name") not in EXPECTED_CREDENTIALS
|
||||
)
|
||||
candidate["name"] = "unreviewedCredential"
|
||||
catalog_path.write_text(json.dumps(catalog, indent=2) + "\n")
|
||||
with self.assertRaisesRegex(RUNNER.DeployError, "catalog sha256 mismatch"):
|
||||
RUNNER.validate_engine_n8n_transition(
|
||||
work / "payload",
|
||||
RUNNER.parse_files_list(work / "files.txt"),
|
||||
)
|
||||
|
||||
def test_old_engine_artifact_is_rejected_by_descriptor_gate(self):
|
||||
old = PLATFORM_ROOT / "infra/deploy-artifacts/nodedc-engine-n8n-private-extension-20260715-002.tgz"
|
||||
if not old.is_file():
|
||||
self.skipTest("rejected historical artifact not present")
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
with self.assertRaisesRegex(RUNNER.DeployError, "canonical transition descriptor"):
|
||||
RUNNER.load_artifact(old, Path(directory))
|
||||
|
||||
def test_engine_source_keeps_base_compose_and_separate_override(self):
|
||||
original_root = RUNNER.COMPONENTS["engine"]["payload_root"]
|
||||
original_compose = RUNNER.COMPONENTS["engine"]["compose_root"]
|
||||
try:
|
||||
RUNNER.COMPONENTS["engine"]["payload_root"] = ENGINE_ROOT
|
||||
RUNNER.COMPONENTS["engine"]["compose_root"] = ENGINE_ROOT
|
||||
RUNNER.validate_engine_n8n_base_compose_source()
|
||||
finally:
|
||||
RUNNER.COMPONENTS["engine"]["payload_root"] = original_root
|
||||
RUNNER.COMPONENTS["engine"]["compose_root"] = original_compose
|
||||
|
||||
def test_additional_n8n_runtime_service_is_rejected(self):
|
||||
original_root = RUNNER.COMPONENTS["engine"]["payload_root"]
|
||||
original_compose = RUNNER.COMPONENTS["engine"]["compose_root"]
|
||||
try:
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
engine_root = Path(directory)
|
||||
compose = (ENGINE_ROOT / "docker-compose.yml").read_text()
|
||||
compose += (
|
||||
"\n n8n-worker-2:\n"
|
||||
" image: docker.n8n.io/n8nio/n8n:${N8N_IMAGE_TAG:-2.3.2}\n"
|
||||
)
|
||||
(engine_root / "docker-compose.yml").write_text(compose)
|
||||
RUNNER.COMPONENTS["engine"]["payload_root"] = engine_root
|
||||
RUNNER.COMPONENTS["engine"]["compose_root"] = engine_root
|
||||
with self.assertRaisesRegex(RUNNER.DeployError, "exact service topology mismatch"):
|
||||
RUNNER.validate_engine_n8n_base_compose_source()
|
||||
finally:
|
||||
RUNNER.COMPONENTS["engine"]["payload_root"] = original_root
|
||||
RUNNER.COMPONENTS["engine"]["compose_root"] = original_compose
|
||||
|
||||
def test_sealed_release_exact_set_includes_implicit_directories(self):
|
||||
release_relative = Path(
|
||||
"payload/releases/n8n-nodes-ndc/0.1.2-05e4b38b14b4a019"
|
||||
)
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
work = Path(directory)
|
||||
outer = work / "outer"
|
||||
RUNNER.safe_extract(STAGE_ARTIFACT, outer)
|
||||
staged_release = outer / release_relative
|
||||
sealed_release = work / "sealed"
|
||||
sealed_release.mkdir()
|
||||
for filename in ("package.tgz", "release.json", "rollback.json"):
|
||||
(sealed_release / filename).write_bytes(
|
||||
(staged_release / filename).read_bytes()
|
||||
)
|
||||
|
||||
expected_paths = {
|
||||
"package",
|
||||
"package.tgz",
|
||||
"release.json",
|
||||
"rollback.json",
|
||||
}
|
||||
with tarfile.open(staged_release / "package.tgz", "r:gz") as archive:
|
||||
for member in archive:
|
||||
RUNNER.add_engine_n8n_sealed_member_paths(
|
||||
expected_paths,
|
||||
member.name,
|
||||
)
|
||||
target = sealed_release.joinpath(*Path(member.name).parts)
|
||||
if member.isdir():
|
||||
target.mkdir(parents=True, exist_ok=True)
|
||||
continue
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
source = archive.extractfile(member)
|
||||
self.assertIsNotNone(source)
|
||||
target.write_bytes(source.read())
|
||||
|
||||
actual_paths = {
|
||||
path.relative_to(sealed_release).as_posix()
|
||||
for path in sealed_release.rglob("*")
|
||||
}
|
||||
self.assertEqual(actual_paths, expected_paths)
|
||||
self.assertIn("package/dist/nodes", expected_paths)
|
||||
self.assertIn("package/dist/credentials", expected_paths)
|
||||
|
||||
def test_tar_members_have_no_appledouble_or_special_types(self):
|
||||
for kind in ("activation", "rollback"):
|
||||
with tarfile.open(self.artifact(0, kind), "r:gz") as archive:
|
||||
for member in archive:
|
||||
self.assertFalse(Path(member.name).name.startswith("._"))
|
||||
self.assertTrue(member.isfile() or member.isdir())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main(verbosity=2)
|
||||
|
|
@ -1,260 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
import hashlib
|
||||
import importlib.machinery
|
||||
import importlib.util
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import tarfile
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
SCRIPT_DIR = Path(__file__).resolve().parent
|
||||
PLATFORM_ROOT = SCRIPT_DIR.parent.parent
|
||||
RUNNER_PATH = SCRIPT_DIR / "nodedc-deploy"
|
||||
BUILDER_PATH = SCRIPT_DIR / "build-n8n-private-extension-artifact.mjs"
|
||||
PATCH_ID = "n8n-nodes-ndc-release-20260716-003"
|
||||
EXPECTED_NODES = [
|
||||
"dist/nodes/NdcDataProductPublish/NdcDataProductPublish.node.js",
|
||||
"dist/nodes/NdcDataProductRead/NdcDataProductRead.node.js",
|
||||
"dist/nodes/NdcFoundryBinding/NdcFoundryBinding.node.js",
|
||||
]
|
||||
EXPECTED_NODE_TYPES = [
|
||||
"n8n-nodes-ndc.ndcDataProductPublish",
|
||||
"n8n-nodes-ndc.ndcDataProductRead",
|
||||
"n8n-nodes-ndc.ndcFoundryBinding",
|
||||
]
|
||||
EXPECTED_CREDENTIALS = [
|
||||
"dist/credentials/NdcDataProductWriterApi.credentials.js",
|
||||
"dist/credentials/NdcDataProductReaderApi.credentials.js",
|
||||
"dist/credentials/NdcFoundryBindingApi.credentials.js",
|
||||
]
|
||||
EXPECTED_CREDENTIAL_TYPES = [
|
||||
"ndcDataProductWriterApi",
|
||||
"ndcDataProductReaderApi",
|
||||
"ndcFoundryBindingApi",
|
||||
]
|
||||
|
||||
|
||||
def load_runner():
|
||||
loader = importlib.machinery.SourceFileLoader("nodedc_deploy_under_test", str(RUNNER_PATH))
|
||||
spec = importlib.util.spec_from_loader(loader.name, loader)
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
RUNNER = load_runner()
|
||||
|
||||
|
||||
class N8nPrivateExtensionPolicyTest(unittest.TestCase):
|
||||
@classmethod
|
||||
def setUpClass(cls):
|
||||
cls.temporary = tempfile.TemporaryDirectory(prefix="nodedc-extension-policy-")
|
||||
cls.root = Path(cls.temporary.name)
|
||||
cls.artifacts = []
|
||||
cls.results = []
|
||||
for index in range(2):
|
||||
output_dir = cls.root / f"build-{index}"
|
||||
output_dir.mkdir()
|
||||
environment = os.environ.copy()
|
||||
environment["NODEDC_DEPLOY_ARTIFACT_DIR"] = str(output_dir)
|
||||
command = ["node", str(BUILDER_PATH)]
|
||||
if index == 0:
|
||||
command.append(PATCH_ID)
|
||||
result = subprocess.run(
|
||||
command,
|
||||
cwd=PLATFORM_ROOT,
|
||||
env=environment,
|
||||
check=True,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
metadata = json.loads(result.stdout)
|
||||
artifact = Path(metadata["artifact"])
|
||||
cls.results.append(metadata)
|
||||
cls.artifacts.append(artifact)
|
||||
|
||||
@classmethod
|
||||
def tearDownClass(cls):
|
||||
cls.temporary.cleanup()
|
||||
|
||||
def test_runner_source_compiles(self):
|
||||
compile(RUNNER_PATH.read_text(encoding="utf-8"), str(RUNNER_PATH), "exec")
|
||||
|
||||
def test_builder_is_byte_reproducible(self):
|
||||
first = self.artifacts[0].read_bytes()
|
||||
second = self.artifacts[1].read_bytes()
|
||||
self.assertEqual(first, second)
|
||||
self.assertEqual(self.results[0]["artifactSha256"], self.results[1]["artifactSha256"])
|
||||
self.assertEqual(self.results[0]["patchId"], PATCH_ID)
|
||||
self.assertEqual(self.results[0]["nodeTypes"], EXPECTED_NODE_TYPES)
|
||||
self.assertEqual(self.results[0]["credentialTypes"], EXPECTED_CREDENTIAL_TYPES)
|
||||
self.assertEqual(first[4:8], b"\0\0\0\0", "gzip mtime must be zero")
|
||||
self.assertEqual(first[3] & 0x08, 0, "gzip header must not carry a host filename")
|
||||
|
||||
inner = self._inner_package_bytes()
|
||||
self.assertEqual(inner[4:8], b"\0\0\0\0", "inner package gzip mtime must be zero")
|
||||
self.assertEqual(inner[3] & 0x08, 0, "inner package gzip header must not carry a host filename")
|
||||
|
||||
def test_positive_release_passes_runner_policy(self):
|
||||
with tempfile.TemporaryDirectory(prefix="nodedc-extension-load-") as directory:
|
||||
manifest, entries, _payload = RUNNER.load_artifact(self.artifacts[0], Path(directory))
|
||||
self.assertEqual(manifest["component"], "n8n-private-extension")
|
||||
self.assertEqual(len(entries), 1)
|
||||
self.assertRegex(entries[0], r"^releases/n8n-nodes-ndc/0\.1\.2-[a-f0-9]{16}$")
|
||||
|
||||
with tarfile.open(self.artifacts[0], "r:gz") as archive:
|
||||
names = archive.getnames()
|
||||
self.assertFalse(any(Path(name).name.startswith("._") for name in names))
|
||||
self.assertFalse(any(".." in Path(name).parts for name in names))
|
||||
|
||||
def test_release_has_exactly_three_non_tool_runtime_types(self):
|
||||
package = self._inner_package_bytes()
|
||||
with tarfile.open(fileobj=io.BytesIO(package), mode="r:gz") as archive:
|
||||
package_json_member = archive.getmember("package/package.json")
|
||||
package_json = json.loads(archive.extractfile(package_json_member).read())
|
||||
self.assertEqual(package_json["version"], "0.1.2")
|
||||
self.assertEqual(package_json["n8n"]["nodes"], EXPECTED_NODES)
|
||||
self.assertEqual(package_json["n8n"]["credentials"], EXPECTED_CREDENTIALS)
|
||||
for node_path in EXPECTED_NODES:
|
||||
source = archive.extractfile(archive.getmember(f"package/{node_path}")).read().decode("utf-8")
|
||||
self.assertNotRegex(source, r"\busableAsTool\b")
|
||||
|
||||
def test_inner_package_stream_and_members_are_canonical(self):
|
||||
package = self._inner_package_bytes()
|
||||
self.assertEqual(package[4:8], b"\0\0\0\0")
|
||||
self.assertEqual(package[3] & 0x08, 0)
|
||||
with tarfile.open(fileobj=io.BytesIO(package), mode="r:gz") as archive:
|
||||
members = archive.getmembers()
|
||||
self.assertEqual([member.name for member in members], sorted(member.name for member in members))
|
||||
for member in members:
|
||||
self.assertEqual((member.uid, member.gid, member.uname, member.gname), (0, 0, "root", "root"))
|
||||
self.assertEqual(member.mtime, 0)
|
||||
self.assertEqual(member.mode, 0o755 if member.isdir() else 0o644)
|
||||
|
||||
def test_v2_first_activation_uses_verified_inactive_baseline(self):
|
||||
with tempfile.TemporaryDirectory(prefix="nodedc-extension-v2-") as directory:
|
||||
work = Path(directory)
|
||||
RUNNER.safe_extract(self.artifacts[0], work)
|
||||
entries = RUNNER.parse_files_list(work / "files.txt")
|
||||
release_dir = work / "payload" / entries[0]
|
||||
release_path = release_dir / "release.json"
|
||||
rollback_path = release_dir / "rollback.json"
|
||||
release = json.loads(release_path.read_text(encoding="utf-8"))
|
||||
rollback = json.loads(rollback_path.read_text(encoding="utf-8"))
|
||||
|
||||
expected_policy = {
|
||||
"allowed": ["previous_verified_immutable_release", "verified_inactive"],
|
||||
"firstActivation": "verified_inactive",
|
||||
"requiresPreActivationVerification": True,
|
||||
}
|
||||
self.assertEqual(release["schemaVersion"], "nodedc.n8n-private-extension-release/v2")
|
||||
self.assertEqual(release["package"]["version"], "0.1.2")
|
||||
self.assertEqual(release["activation"]["rollbackBaselinePolicy"], expected_policy)
|
||||
self.assertEqual(rollback["schemaVersion"], "nodedc.n8n-private-extension-rollback/v2")
|
||||
self.assertEqual(rollback["baselinePolicy"], expected_policy)
|
||||
|
||||
rollback["baselinePolicy"]["requiresPreActivationVerification"] = False
|
||||
rollback_path.write_text(json.dumps(rollback), encoding="utf-8")
|
||||
with self.assertRaisesRegex(RUNNER.DeployError, "rollback manifest mismatch"):
|
||||
RUNNER.validate_n8n_private_extension_release(work / "payload", entries)
|
||||
|
||||
def test_outer_appledouble_and_traversal_are_rejected(self):
|
||||
with self.assertRaisesRegex(RUNNER.DeployError, "unexpected tar member"):
|
||||
RUNNER.validate_tar_member(tarfile.TarInfo("._manifest.env"))
|
||||
with self.assertRaisesRegex(RUNNER.DeployError, "path escape rejected"):
|
||||
RUNNER.validate_tar_member(tarfile.TarInfo("payload/../escape"))
|
||||
|
||||
def test_extra_custom_node_is_rejected(self):
|
||||
def mutate(package_json):
|
||||
package_json["n8n"]["nodes"].append("dist/nodes/Extra/Extra.node.js")
|
||||
|
||||
package_path, release = self._tampered_package(
|
||||
mutate,
|
||||
{"package/dist/nodes/Extra/Extra.node.js": b"module.exports = {};\n"},
|
||||
)
|
||||
with self.assertRaisesRegex(RUNNER.DeployError, "n8n.nodes registration mismatch"):
|
||||
RUNNER.validate_n8n_package_tarball(package_path, release)
|
||||
|
||||
def test_lifecycle_script_is_rejected(self):
|
||||
def mutate(package_json):
|
||||
package_json.setdefault("scripts", {})["install"] = "echo forbidden"
|
||||
|
||||
package_path, release = self._tampered_package(mutate)
|
||||
with self.assertRaisesRegex(RUNNER.DeployError, "lifecycle script is forbidden"):
|
||||
RUNNER.validate_n8n_package_tarball(package_path, release)
|
||||
|
||||
def test_inner_appledouble_is_rejected(self):
|
||||
package_path, release = self._tampered_package(
|
||||
lambda _package_json: None,
|
||||
{"package/dist/._NdcDataProductPublish.node.js": b"forbidden\n"},
|
||||
)
|
||||
with self.assertRaisesRegex(RUNNER.DeployError, "hidden member rejected"):
|
||||
RUNNER.validate_n8n_package_tarball(package_path, release)
|
||||
|
||||
def test_tool_variant_marker_is_rejected_by_runner(self):
|
||||
target = EXPECTED_NODES[0]
|
||||
package_path, release = self._tampered_package(
|
||||
lambda _package_json: None,
|
||||
file_mutations={target: lambda source: source + b"\n// usableAsTool: true\n"},
|
||||
)
|
||||
with self.assertRaisesRegex(RUNNER.DeployError, "would generate a tool variant"):
|
||||
RUNNER.validate_n8n_package_tarball(package_path, release)
|
||||
|
||||
def _tampered_package(self, mutate, extra_files=None, file_mutations=None):
|
||||
package_bytes = self._inner_package_bytes()
|
||||
members = []
|
||||
with tarfile.open(fileobj=io.BytesIO(package_bytes), mode="r:gz") as archive:
|
||||
for member in archive:
|
||||
if not member.isfile():
|
||||
continue
|
||||
source = archive.extractfile(member)
|
||||
members.append((member.name, source.read()))
|
||||
|
||||
rewritten = []
|
||||
for name, content in members:
|
||||
if name == "package/package.json":
|
||||
package_json = json.loads(content)
|
||||
mutate(package_json)
|
||||
content = (json.dumps(package_json, sort_keys=True) + "\n").encode("utf-8")
|
||||
package_rel = name.removeprefix("package/")
|
||||
if package_rel in (file_mutations or {}):
|
||||
content = file_mutations[package_rel](content)
|
||||
rewritten.append((name, content))
|
||||
rewritten.extend((extra_files or {}).items())
|
||||
|
||||
target = self.root / f"tampered-{len(list(self.root.glob('tampered-*.tgz')))}.tgz"
|
||||
with tarfile.open(target, "w:gz", format=tarfile.PAX_FORMAT) as archive:
|
||||
for name, content in rewritten:
|
||||
member = tarfile.TarInfo(name)
|
||||
member.mode = 0o644
|
||||
member.mtime = 0
|
||||
member.size = len(content)
|
||||
archive.addfile(member, io.BytesIO(content))
|
||||
|
||||
with tarfile.open(target, "r:gz") as archive:
|
||||
package_member = next(member for member in archive if member.name == "package/package.json")
|
||||
version = json.loads(archive.extractfile(package_member).read())["version"]
|
||||
release = {
|
||||
"package": {
|
||||
"version": version,
|
||||
"sha256": hashlib.sha256(target.read_bytes()).hexdigest(),
|
||||
"bytes": target.stat().st_size,
|
||||
}
|
||||
}
|
||||
return target, release
|
||||
|
||||
def _inner_package_bytes(self):
|
||||
with tarfile.open(self.artifacts[0], "r:gz") as archive:
|
||||
member = next(item for item in archive if item.name.endswith("/package.tgz"))
|
||||
source = archive.extractfile(member)
|
||||
return source.read()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main(verbosity=2)
|
||||
|
|
@ -152,11 +152,9 @@ services:
|
|||
AI_WORKSPACE_HUB_FALLBACK_URLS: ${AI_WORKSPACE_HUB_FALLBACK_URLS:-}
|
||||
AI_WORKSPACE_ASSISTANT_ACTION_RELAY_ENABLED: ${AI_WORKSPACE_ASSISTANT_ACTION_RELAY_ENABLED:-true}
|
||||
AI_WORKSPACE_ASSISTANT_ACTION_RELAY_ID: ${AI_WORKSPACE_ASSISTANT_ACTION_RELAY_ID:-local-dev}
|
||||
AI_WORKSPACE_ONTOLOGY_MCP_ENABLED: ${AI_WORKSPACE_ONTOLOGY_MCP_ENABLED:-true}
|
||||
AI_WORKSPACE_OPS_ENTITLEMENT_URL: ${AI_WORKSPACE_OPS_ENTITLEMENT_URL:-}
|
||||
AI_WORKSPACE_OPS_ENTITLEMENT_TOKEN: ${AI_WORKSPACE_OPS_ENTITLEMENT_TOKEN:-}
|
||||
AI_WORKSPACE_OPS_ENTITLEMENT_REQUIRED: ${AI_WORKSPACE_OPS_ENTITLEMENT_REQUIRED:-false}
|
||||
AI_WORKSPACE_ENTITLEMENT_ADAPTERS_JSON: ${AI_WORKSPACE_ENTITLEMENT_ADAPTERS_JSON:-}
|
||||
AI_WORKSPACE_OPS_GATEWAY_BASE_URL: ${AI_WORKSPACE_OPS_GATEWAY_BASE_URL:-}
|
||||
AI_WORKSPACE_OPS_DEFAULT_WORKSPACE_SLUG: ${AI_WORKSPACE_OPS_DEFAULT_WORKSPACE_SLUG:-}
|
||||
AI_WORKSPACE_OPS_DEFAULT_PROJECT_ID: ${AI_WORKSPACE_OPS_DEFAULT_PROJECT_ID:-}
|
||||
|
|
@ -170,86 +168,6 @@ services:
|
|||
ai-workspace-postgres:
|
||||
condition: service_healthy
|
||||
|
||||
ontology-core:
|
||||
image: nodedc/ontology-core:local
|
||||
build:
|
||||
context: ../services/ontology-core
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
NODE_ENV: production
|
||||
PORT: 18104
|
||||
NODEDC_INTERNAL_ACCESS_TOKEN: ${NODEDC_INTERNAL_ACCESS_TOKEN:-}
|
||||
expose:
|
||||
- "18104"
|
||||
healthcheck:
|
||||
test:
|
||||
- CMD
|
||||
- node
|
||||
- -e
|
||||
- "fetch('http://127.0.0.1:18104/healthz').then((response) => process.exit(response.ok ? 0 : 1)).catch(() => process.exit(1))"
|
||||
interval: 30s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
start_period: 10s
|
||||
|
||||
# External-provider telemetry is isolated from Platform identity state. The
|
||||
# database is not published; only the Gateway is exposed on the local host
|
||||
# for operator checks and Engine development.
|
||||
gelios-postgres:
|
||||
image: ${GELIOS_TIMESCALE_IMAGE:-timescale/timescaledb-ha:pg16.14-ts2.28.2-all}
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
POSTGRES_DB: ${GELIOS_PG_DB:-nodedc_gelios}
|
||||
POSTGRES_USER: ${GELIOS_PG_USER:-nodedc_gelios}
|
||||
POSTGRES_PASSWORD: ${GELIOS_PG_PASS:-change-me-generate-with-infra-scripts-init-dev-env}
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -d $${POSTGRES_DB} -U $${POSTGRES_USER}"]
|
||||
interval: 30s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
start_period: 30s
|
||||
volumes:
|
||||
- gelios-database:/home/postgres/pgdata/data
|
||||
networks:
|
||||
- gelios-data
|
||||
|
||||
gelios-gateway:
|
||||
image: nodedc/gelios-gateway:local
|
||||
build:
|
||||
context: ../services/gelios-gateway
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
NODE_ENV: production
|
||||
PORT: 18105
|
||||
DATABASE_URL: ${GELIOS_DATABASE_URL:-postgresql://nodedc_gelios:change-me-generate-with-infra-scripts-init-dev-env@gelios-postgres:5432/nodedc_gelios}
|
||||
NODEDC_INTERNAL_ACCESS_TOKEN: ${NODEDC_INTERNAL_ACCESS_TOKEN:-}
|
||||
GELIOS_TENANT_ID: ${GELIOS_TENANT_ID:-robot2b}
|
||||
GELIOS_CONNECTION_ID: ${GELIOS_CONNECTION_ID:-gelios-primary}
|
||||
GELIOS_UNIT_SCOPE: ${GELIOS_UNIT_SCOPE:-allowlist}
|
||||
GELIOS_ALLOWED_UNIT_IDS: ${GELIOS_ALLOWED_UNIT_IDS:-}
|
||||
GELIOS_INTAKE_ENABLED: ${GELIOS_INTAKE_ENABLED:-false}
|
||||
GELIOS_RAW_RETENTION_DAYS: ${GELIOS_RAW_RETENTION_DAYS:-14}
|
||||
expose:
|
||||
- "18105"
|
||||
ports:
|
||||
- "${GELIOS_GATEWAY_HOST_BIND:-127.0.0.1:18105}:18105"
|
||||
depends_on:
|
||||
gelios-postgres:
|
||||
condition: service_healthy
|
||||
healthcheck:
|
||||
test:
|
||||
- CMD
|
||||
- node
|
||||
- -e
|
||||
- "fetch('http://127.0.0.1:18105/healthz').then((response) => process.exit(response.ok ? 0 : 1)).catch(() => process.exit(1))"
|
||||
interval: 30s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
start_period: 15s
|
||||
networks:
|
||||
- default
|
||||
- gelios-data
|
||||
|
||||
ai-workspace-hub:
|
||||
image: nodedc/ai-workspace-hub:local
|
||||
build:
|
||||
|
|
@ -262,14 +180,10 @@ services:
|
|||
AI_WORKSPACE_HUB_WS_PATH: /api/ai-workspace/hub
|
||||
NODEDC_INTERNAL_ACCESS_TOKEN: ${NODEDC_INTERNAL_ACCESS_TOKEN:-}
|
||||
NODEDC_AI_WORKSPACE_ASSISTANT_URL: http://ai-workspace-assistant:18082
|
||||
NODEDC_ONTOLOGY_CORE_URL: http://ontology-core:18104
|
||||
expose:
|
||||
- "18081"
|
||||
ports:
|
||||
- "${AI_WORKSPACE_HUB_HOST_BIND:-127.0.0.1:18081}:18081"
|
||||
depends_on:
|
||||
ontology-core:
|
||||
condition: service_healthy
|
||||
|
||||
map-gateway:
|
||||
image: nodedc/map-gateway:local
|
||||
|
|
@ -280,10 +194,6 @@ services:
|
|||
NODE_ENV: production
|
||||
PORT: 18103
|
||||
CESIUM_ION_TOKEN: ${CESIUM_ION_TOKEN:-}
|
||||
# Foundry signs its private token-admin requests with the existing
|
||||
# Platform internal credential. The value stays outside this compose
|
||||
# file; this line only wires the local development runtime.
|
||||
NODEDC_INTERNAL_ACCESS_TOKEN: ${NODEDC_INTERNAL_ACCESS_TOKEN:-}
|
||||
CESIUM_ION_ASSET_ALLOWLIST: ${CESIUM_ION_ASSET_ALLOWLIST:-1,2,96188}
|
||||
MAP_GATEWAY_UPSTREAM_ALLOWLIST: ${MAP_GATEWAY_UPSTREAM_ALLOWLIST:-api.cesium.com,assets.ion.cesium.com,tile.openstreetmap.org,dev.virtualearth.net,ecn.t0.tiles.virtualearth.net,ecn.t1.tiles.virtualearth.net,ecn.t2.tiles.virtualearth.net,ecn.t3.tiles.virtualearth.net}
|
||||
# Offline access is selected explicitly with the cache profile, rather
|
||||
|
|
@ -326,7 +236,6 @@ volumes:
|
|||
authentik-certs:
|
||||
notification-database:
|
||||
ai-workspace-database:
|
||||
gelios-database:
|
||||
caddy-data:
|
||||
caddy-config:
|
||||
# Cache data is runtime state, not Compose lifecycle state. These volumes are
|
||||
|
|
@ -338,7 +247,3 @@ volumes:
|
|||
map-offline-snapshot:
|
||||
external: true
|
||||
name: ${NODEDC_MAP_OFFLINE_SNAPSHOT_VOLUME:-nodedc-platform_map-offline-snapshot}
|
||||
|
||||
networks:
|
||||
gelios-data:
|
||||
internal: true
|
||||
|
|
|
|||
|
|
@ -93,7 +93,6 @@ run_sh_quiet() {
|
|||
section "Static syntax"
|
||||
run_cmd "Hub server syntax" node --check "$HUB_DIR/src/server.mjs"
|
||||
run_cmd "Assistant server syntax" node --check "$ASSISTANT_DIR/src/server.mjs"
|
||||
run_cmd "Ontology MCP server syntax" node --check "$ONTOLOGY_DIR/src/mcp-server.mjs"
|
||||
run_cmd "Local environment safety checker syntax" node --check "$INFRA_DIR/scripts/check-local-environment-safety.mjs"
|
||||
run_sh "Shell scripts syntax" "$PLATFORM_ROOT" \
|
||||
"sh -n infra/scripts/init-dev-env.sh infra/scripts/check-local-test-system.sh infra/scripts/check-ai-workspace-topology.sh infra/scripts/check-ai-workspace-config-contract.sh infra/scripts/check-ai-workspace-release-gates.sh"
|
||||
|
|
@ -129,10 +128,8 @@ fi
|
|||
|
||||
section "AI Workspace smokes"
|
||||
run_sh_quiet "Assistant run-profile smoke" "$ASSISTANT_DIR" "npm run smoke:run-profile"
|
||||
run_sh_quiet "AI Hub Ontology MCP proxy smoke" "$HUB_DIR" "npm run smoke:ontology-mcp-proxy"
|
||||
if [ -d "$ONTOLOGY_DIR" ]; then
|
||||
run_sh_quiet "Ontology assistant caller smoke" "$ONTOLOGY_DIR" "npm run smoke:assistant-caller"
|
||||
run_sh_quiet "Ontology MCP smoke" "$ONTOLOGY_DIR" "npm run smoke:mcp"
|
||||
else
|
||||
skip "Ontology assistant caller smoke (missing $ONTOLOGY_DIR)"
|
||||
fi
|
||||
|
|
|
|||
|
|
@ -33,24 +33,6 @@ TASK_INTERNAL_LOGOUT_URL=https://ops.nodedc.ru/api/internal/nodedc/logout/
|
|||
NODEDC_ENGINE_DOCKER_NETWORK=nodedc-demo_default
|
||||
NODEDC_ENGINE_INTERNAL_URL=http://nodedc-demo-nodedc-backend-1:3001
|
||||
|
||||
# Platform Map Gateway — private service and one shared persistent TileCache.
|
||||
# Optional one-time legacy bootstrap only. Preferred operation: an authorized
|
||||
# Foundry admin saves the token through Page Library → Platform settings; it is
|
||||
# then stored in the private NAS cache volume and never exposed to Foundry UI,
|
||||
# deploy artifact or browser env. Do not paste a real token into this example.
|
||||
CESIUM_ION_TOKEN=
|
||||
CESIUM_ION_ASSET_ALLOWLIST=1,2,96188
|
||||
MAP_CACHE_MODE=readwrite
|
||||
MAP_CACHE_MAX_MB=20480
|
||||
MAP_GATEWAY_HOST_BIND=127.0.0.1:18103
|
||||
# Created by the root-owned deploy runner on first Map Gateway rollout.
|
||||
# These are NAS paths, visible through SMB as nodedc-platform/map-gateway/.
|
||||
MAP_LIVE_CACHE_HOST_DIR=/volume1/docker/nodedc-platform/map-gateway/live-tile-cache
|
||||
MAP_OFFLINE_SNAPSHOT_HOST_DIR=/volume1/docker/nodedc-platform/map-gateway/offline-snapshot
|
||||
MAP_GATEWAY_UPSTREAM_ALLOWLIST=api.cesium.com,assets.ion.cesium.com,tile.openstreetmap.org,dev.virtualearth.net,ecn.t0.tiles.virtualearth.net,ecn.t1.tiles.virtualearth.net,ecn.t2.tiles.virtualearth.net,ecn.t3.tiles.virtualearth.net
|
||||
MAP_GATEWAY_LEGACY_CACHE_HOSTS=
|
||||
MAP_GATEWAY_OFFLINE_PROVIDER_ALLOWLIST=
|
||||
|
||||
LAUNCHER_OIDC_ISSUER=https://id.nodedc.ru/application/o/launcher/
|
||||
LAUNCHER_OIDC_CLIENT_ID=nodedc-launcher
|
||||
LAUNCHER_OIDC_CLIENT_SECRET=replace-with-random-synology-secret
|
||||
|
|
@ -73,34 +55,6 @@ SESSION_SECRET=replace-with-random-synology-secret
|
|||
COOKIE_DOMAIN=.nas.nodedc
|
||||
COOKIE_SECURE=false
|
||||
|
||||
# External Data Plane — provider-neutral storage. Generate a distinct database
|
||||
# password; it must never reuse NODEDC_INTERNAL_ACCESS_TOKEN or a provider key.
|
||||
EXTERNAL_DATA_PLANE_TIMESCALE_IMAGE=timescale/timescaledb-ha:pg16.14-ts2.28.2-all
|
||||
EXTERNAL_DATA_PLANE_PG_DB=nodedc_data_plane
|
||||
EXTERNAL_DATA_PLANE_PG_USER=nodedc_data_plane
|
||||
EXTERNAL_DATA_PLANE_PG_PASS=replace-with-random-synology-secret
|
||||
EXTERNAL_DATA_PLANE_HOST_BIND=127.0.0.1:18106
|
||||
EXTERNAL_DATA_PLANE_DATABASE_POOL_SIZE=10
|
||||
EXTERNAL_DATA_PLANE_RAW_RETENTION_DAYS=14
|
||||
EXTERNAL_DATA_PLANE_MAX_BATCH_BYTES=5242880
|
||||
EXTERNAL_DATA_PLANE_MAX_FACTS_PER_PUBLISH=5000
|
||||
EXTERNAL_DATA_PLANE_MAX_ATTRIBUTES_BYTES_PER_FACT=65536
|
||||
EXTERNAL_DATA_PLANE_MAX_PATCH_OPERATIONS=500
|
||||
EXTERNAL_DATA_PLANE_MAX_PATCH_BYTES=262144
|
||||
EXTERNAL_DATA_PLANE_PATCH_RETENTION_MS=3600000
|
||||
EXTERNAL_DATA_PLANE_RECEIPT_RETENTION_MS=604800000
|
||||
EXTERNAL_DATA_PLANE_RETENTION_DELETE_LIMIT=10000
|
||||
EXTERNAL_DATA_PLANE_STREAM_HEARTBEAT_MS=20000
|
||||
EXTERNAL_DATA_PLANE_STREAM_POLL_MS=1000
|
||||
EXTERNAL_DATA_PLANE_MAX_READER_STREAMS=10
|
||||
EXTERNAL_DATA_PLANE_WRITER_BINDING_MAX_TTL_DAYS=90
|
||||
EXTERNAL_DATA_PLANE_MAX_FUTURE_SKEW_SECONDS=300
|
||||
EXTERNAL_DATA_PLANE_RETENTION_SWEEP_MS=3600000
|
||||
EXTERNAL_DATA_PLANE_LEGACY_INTAKE_ENABLED=false
|
||||
# The writer-provisioner secret is a root-owned file under
|
||||
# /volume1/docker/nodedc-platform/secrets/external-data-plane-provisioner/,
|
||||
# never a shared .env value.
|
||||
|
||||
NOTIFICATION_PG_DB=nodedc_notifications
|
||||
NOTIFICATION_PG_USER=nodedc_notifications
|
||||
NOTIFICATION_PG_PASS=replace-with-random-synology-secret
|
||||
|
|
@ -114,41 +68,9 @@ NODEDC_AI_WORKSPACE_ASSISTANT_URL=http://ai-workspace-assistant:18082
|
|||
AI_WORKSPACE_OPS_ENTITLEMENT_URL=http://172.22.0.222:18190/api/internal/v1/ai-workspace/entitlements
|
||||
AI_WORKSPACE_OPS_ENTITLEMENT_TOKEN=replace-with-ops-agent-gateway-internal-token
|
||||
AI_WORKSPACE_OPS_ENTITLEMENT_REQUIRED=false
|
||||
# Set only after Foundry passes Launcher handoff and Authentik access checks.
|
||||
# Keep every existing entitlement adapter in this same JSON object. The adapter
|
||||
# uses the existing NODE.DC internal service credential; no Foundry browser or
|
||||
# worker secret is required.
|
||||
# {"module-foundry":{"url":"https://foundry.nodedc.ru/api/ai-workspace/entitlements","required":false}}
|
||||
AI_WORKSPACE_ENTITLEMENT_ADAPTERS_JSON=
|
||||
|
||||
AI_WORKSPACE_HUB_TOKEN=replace-with-random-synology-secret
|
||||
AI_WORKSPACE_HUB_HOST_BIND=0.0.0.0:18081
|
||||
AI_WORKSPACE_HUB_PUBLIC_URL=wss://ai-hub.nodedc.ru/api/ai-workspace/hub
|
||||
AI_WORKSPACE_HUB_INTERNAL_URL=https://ai-hub.nodedc.ru
|
||||
AI_WORKSPACE_HUB_FALLBACK_URLS=
|
||||
AI_WORKSPACE_ONTOLOGY_MCP_ENABLED=true
|
||||
# Optional public HTTP origin for the dynamic worker MCP URL. Normally it is
|
||||
# derived from AI_WORKSPACE_HUB_PUBLIC_URL, so leave this empty.
|
||||
AI_WORKSPACE_ONTOLOGY_MCP_PUBLIC_URL=
|
||||
ONTOLOGY_CORE_HOST_BIND=127.0.0.1:18104
|
||||
|
||||
# Gelios Gateway — storage/read service. Provider credentials stay in the
|
||||
# protected Engine Collector. Intake remains disabled until scope is approved.
|
||||
GELIOS_TIMESCALE_IMAGE=timescale/timescaledb-ha:pg16.14-ts2.28.2-all
|
||||
GELIOS_PG_DB=nodedc_gelios
|
||||
GELIOS_PG_USER=nodedc_gelios
|
||||
GELIOS_PG_PASS=replace-with-random-synology-secret
|
||||
# URL-encode reserved characters in GELIOS_PG_PASS when forming this URL.
|
||||
GELIOS_DATABASE_URL=postgresql://nodedc_gelios:replace-with-url-encoded-synology-secret@gelios-postgres:5432/nodedc_gelios
|
||||
GELIOS_GATEWAY_HOST_BIND=127.0.0.1:18105
|
||||
# Explicit tenant and connection identifiers are deployment configuration;
|
||||
# do not encode a customer or pilot name in source defaults.
|
||||
GELIOS_TENANT_ID=replace-with-tenant-id
|
||||
GELIOS_CONNECTION_ID=gelios-connection-id
|
||||
# `all` accepts every unit returned by this approved tenant + connection.
|
||||
GELIOS_UNIT_SCOPE=allowlist
|
||||
GELIOS_ALLOWED_UNIT_IDS=
|
||||
GELIOS_INTAKE_ENABLED=false
|
||||
GELIOS_RAW_RETENTION_DAYS=14
|
||||
# Presentation state only; it does not change provider collection cadence.
|
||||
GELIOS_POSITION_STALE_AFTER_MS=300000
|
||||
|
|
|
|||
|
|
@ -202,26 +202,6 @@ TASKER_SYNC_SOURCE=1 ./infra/synology/deploy-current.sh
|
|||
|
||||
Если emergency-fix был сделан прямо на Synology в этих env-файлах, перенести sanitized-значение в `.env.synology.example`/docs, а секрет оставить только в live env.
|
||||
|
||||
### Gelios: сбор всех units доверенного подключения
|
||||
|
||||
`GELIOS_UNIT_SCOPE=all` означает **все текущие и будущие units только одной уже
|
||||
настроенной пары `GELIOS_TENANT_ID` + `GELIOS_CONNECTION_ID`**. Это не wildcard
|
||||
на другие tenants или providers. Provider credential остаётся только в Engine.
|
||||
Пропавший из очередного ответа unit не удаляется: его stable provider ID и
|
||||
`last_seen_at` сохраняются; видимость на карте — отдельная логика витрины.
|
||||
|
||||
Перед каноническим `nodedc-deploy apply` включить политику на Synology (скрипт
|
||||
создаёт backup и не выводит секреты):
|
||||
|
||||
```bash
|
||||
sudo bash /volume1/docker/nodedc-deploy/inbox/prepare-gelios-all-units-env.sh
|
||||
```
|
||||
|
||||
Затем применить узкий Platform-артефакт, собранный с `--gateway-only`, через
|
||||
`nodedc-deploy`. В его plan должны быть только `gelios-postgres` и
|
||||
`gelios-gateway`; общий `docker-compose` в такой архив не входит. После apply
|
||||
Gateway начинает принимать весь состав units этого подключения.
|
||||
|
||||
## AI Hub relay-only deploy
|
||||
|
||||
Для локального тестирования с внешней Codex-машиной нельзя деплоить Launcher, Engine, Ops, Authentik или Tasker. Единственная допустимая удалённая точка в этой схеме — `ai-workspace-hub`, потому что удалённый worker должен иметь публичный WebSocket/HTTP relay.
|
||||
|
|
|
|||
|
|
@ -34,28 +34,12 @@ echo "== ai workspace assistant image build =="
|
|||
cd "${PLATFORM_DIR}/ai-workspace-assistant"
|
||||
"${DOCKER_BIN}" build --no-cache -t nodedc/ai-workspace-assistant:local .
|
||||
|
||||
echo "== ontology core image build =="
|
||||
cd "${PLATFORM_DIR}/ontology-core"
|
||||
"${DOCKER_BIN}" build --no-cache -t nodedc/ontology-core:local .
|
||||
|
||||
echo "== gelios gateway image build =="
|
||||
cd "${PLATFORM_DIR}/gelios-gateway"
|
||||
"${DOCKER_BIN}" build --no-cache -t nodedc/gelios-gateway:local .
|
||||
|
||||
echo "== platform services recreate =="
|
||||
cd "${PLATFORM_DIR}"
|
||||
"${DOCKER_BIN}" compose \
|
||||
--env-file "${ENV_FILE}" \
|
||||
-f "${COMPOSE_FILE}" \
|
||||
up -d --force-recreate reverse-proxy authentik-server authentik-worker notification-postgres notification-core ai-workspace-postgres ontology-core ai-workspace-assistant ai-workspace-hub gelios-postgres gelios-gateway launcher
|
||||
|
||||
echo "== ontology core health check =="
|
||||
"${DOCKER_BIN}" exec nodedc-platform-ontology-core-1 sh -lc \
|
||||
'node -e '"'"'fetch("http://127.0.0.1:18104/healthz").then(async (response) => { console.log(await response.text()); process.exit(response.ok ? 0 : 1); }).catch((error) => { console.error(error); process.exit(1); })'"'"''
|
||||
|
||||
echo "== gelios gateway health check =="
|
||||
"${DOCKER_BIN}" exec nodedc-platform-gelios-gateway-1 sh -lc \
|
||||
'node -e '"'"'fetch("http://127.0.0.1:18105/healthz").then(async (response) => { console.log(await response.text()); process.exit(response.ok ? 0 : 1); }).catch((error) => { console.error(error); process.exit(1); })'"'"''
|
||||
up -d --force-recreate reverse-proxy authentik-server authentik-worker notification-postgres notification-core ai-workspace-postgres ai-workspace-assistant ai-workspace-hub launcher
|
||||
|
||||
echo "== notification core health check =="
|
||||
"${DOCKER_BIN}" exec nodedc-platform-notification-core-1 sh -lc \
|
||||
|
|
|
|||
|
|
@ -48,7 +48,6 @@ rsync_dir "${NAS_ROOT}/platform/authentik/" "${BACKUP_DIR}/files/platform/authen
|
|||
rsync_dir "${NAS_ROOT}/platform/notification-core/" "${BACKUP_DIR}/files/platform/notification-core/"
|
||||
rsync_dir "${NAS_ROOT}/platform/ai-workspace-hub/" "${BACKUP_DIR}/files/platform/ai-workspace-hub/"
|
||||
rsync_dir "${NAS_ROOT}/platform/ai-workspace-assistant/" "${BACKUP_DIR}/files/platform/ai-workspace-assistant/"
|
||||
rsync_dir "${NAS_ROOT}/platform/ontology-core/" "${BACKUP_DIR}/files/platform/ontology-core/"
|
||||
rsync_dir "${NAS_ROOT}/bim-viewer/source/server/data/" "${BACKUP_DIR}/files/bim-viewer/server-data/"
|
||||
|
||||
rsync_file "${NAS_ROOT}/platform/.env.synology" "${BACKUP_DIR}/files/platform/"
|
||||
|
|
@ -82,7 +81,6 @@ Contains:
|
|||
- Notification Core source/config: platform/notification-core, platform compose/env
|
||||
- AI Workspace Hub source/config: platform/ai-workspace-hub, platform compose/env
|
||||
- AI Workspace Assistant source/config: platform/ai-workspace-assistant, platform compose/env
|
||||
- Ontology Core source/catalog: platform/ontology-core, platform compose/env
|
||||
- Tasker runtime config: tasker/plane-app/.env.synology, compose, Synology override
|
||||
- Ops Agents Gateway runtime config: ops-agents/.env, compose
|
||||
- BIM Viewer runtime data/config: bim-viewer/source/server/data, .env, compose
|
||||
|
|
|
|||
|
|
@ -84,22 +84,6 @@ rsync -av "${RSYNC_METADATA_ARGS[@]}" --delete \
|
|||
"${PLATFORM_REPO}/services/ontology-core/" \
|
||||
"${NAS_ROOT}/platform/ai-workspace-assistant/ontology-core/"
|
||||
|
||||
mkdir -p "${NAS_ROOT}/platform/ontology-core"
|
||||
rsync -av "${RSYNC_METADATA_ARGS[@]}" --delete \
|
||||
--exclude='node_modules/' \
|
||||
--exclude='.env' \
|
||||
--exclude='.env.*' \
|
||||
"${PLATFORM_REPO}/services/ontology-core/" \
|
||||
"${NAS_ROOT}/platform/ontology-core/"
|
||||
|
||||
mkdir -p "${NAS_ROOT}/platform/gelios-gateway"
|
||||
rsync -av "${RSYNC_METADATA_ARGS[@]}" --delete \
|
||||
--exclude='node_modules/' \
|
||||
--exclude='.env' \
|
||||
--exclude='.env.*' \
|
||||
"${PLATFORM_REPO}/services/gelios-gateway/" \
|
||||
"${NAS_ROOT}/platform/gelios-gateway/"
|
||||
|
||||
if [[ "${SYNC_AUTHENTIK_TEMPLATES}" == "1" ]]; then
|
||||
mkdir -p "${NAS_ROOT}/authentik/custom-templates"
|
||||
rsync -av "${RSYNC_METADATA_ARGS[@]}" --delete \
|
||||
|
|
@ -256,20 +240,7 @@ fi
|
|||
|
||||
cat <<'EOF'
|
||||
|
||||
Synced source to the NAS mount and staged the root-owned runner candidate.
|
||||
|
||||
For a canonical deploy, do not use the legacy direct runtime scripts below.
|
||||
An administrator first installs/verifies the staged runner, then plans and
|
||||
applies a data-only artifact from /volume1/docker/nodedc-deploy/inbox:
|
||||
|
||||
sudo install -o root -g root -m 0755 \
|
||||
/volume1/docker/nodedc-deploy/runner-install/nodedc-deploy \
|
||||
/usr/local/sbin/nodedc-deploy
|
||||
sudo /usr/local/sbin/nodedc-deploy verify-install
|
||||
sudo /usr/local/sbin/nodedc-deploy plan /volume1/docker/nodedc-deploy/inbox/<artifact>.tgz
|
||||
sudo /usr/local/sbin/nodedc-deploy apply /volume1/docker/nodedc-deploy/inbox/<artifact>.tgz
|
||||
|
||||
Legacy direct runtime scripts (emergency/manual recovery only):
|
||||
Synced files to NAS mount. Run on Synology to apply runtime changes:
|
||||
|
||||
cd /volume1/docker/nodedc-platform/platform
|
||||
sudo bash apply-current-runtime.sh
|
||||
|
|
@ -305,14 +276,14 @@ cd /volume1/docker/nodedc-platform/platform
|
|||
sudo /usr/local/bin/docker compose \
|
||||
--env-file /volume1/docker/nodedc-platform/platform/.env.synology \
|
||||
-f /volume1/docker/nodedc-platform/platform/docker-compose.platform-http.yml \
|
||||
up -d --build --force-recreate --no-deps ontology-core ai-workspace-hub ai-workspace-assistant notification-core launcher
|
||||
up -d --build --force-recreate --no-deps ai-workspace-hub ai-workspace-assistant notification-core launcher
|
||||
|
||||
Optional Platform/Auth infra apply, only after deliberate compose/proxy/Auth templates changes:
|
||||
|
||||
sudo /usr/local/bin/docker compose \
|
||||
--env-file /volume1/docker/nodedc-platform/platform/.env.synology \
|
||||
-f /volume1/docker/nodedc-platform/platform/docker-compose.platform-http.yml \
|
||||
up -d --build --force-recreate --no-deps reverse-proxy authentik-server authentik-worker ontology-core ai-workspace-hub ai-workspace-assistant notification-core launcher
|
||||
up -d --build --force-recreate --no-deps reverse-proxy authentik-server authentik-worker ai-workspace-hub ai-workspace-assistant notification-core launcher
|
||||
|
||||
After Authentik template rollout, clear global Brand custom CSS from the live DB.
|
||||
NODE.DC login CSS must be template-scoped, not stored in Brand.branding_custom_css:
|
||||
|
|
|
|||
|
|
@ -1,96 +0,0 @@
|
|||
name: nodedc-platform
|
||||
|
||||
# This is intentionally an overlay, not a provider service. It is composed
|
||||
# with docker-compose.platform-http.yml by the canonical deploy runner.
|
||||
services:
|
||||
external-data-plane-postgres:
|
||||
image: ${EXTERNAL_DATA_PLANE_TIMESCALE_IMAGE:-timescale/timescaledb-ha:pg16.14-ts2.28.2-all}
|
||||
restart: unless-stopped
|
||||
env_file:
|
||||
- ${NODEDC_SYNOLOGY_ENV_FILE:-.env.synology}
|
||||
environment:
|
||||
POSTGRES_DB: ${EXTERNAL_DATA_PLANE_PG_DB:-nodedc_data_plane}
|
||||
POSTGRES_USER: ${EXTERNAL_DATA_PLANE_PG_USER:-nodedc_data_plane}
|
||||
POSTGRES_PASSWORD: ${EXTERNAL_DATA_PLANE_PG_PASS:?external data plane database password required}
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -d $${POSTGRES_DB} -U $${POSTGRES_USER}"]
|
||||
interval: 30s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
start_period: 30s
|
||||
volumes:
|
||||
- external-data-plane-database:/home/postgres/pgdata/data
|
||||
networks:
|
||||
- external-data-plane
|
||||
|
||||
external-data-plane:
|
||||
image: nodedc/external-data-plane:local
|
||||
restart: unless-stopped
|
||||
env_file:
|
||||
- ${NODEDC_SYNOLOGY_ENV_FILE:-.env.synology}
|
||||
environment:
|
||||
NODE_ENV: production
|
||||
PORT: 18106
|
||||
EXTERNAL_DATA_PLANE_DATABASE_URL: postgresql://${EXTERNAL_DATA_PLANE_PG_USER:-nodedc_data_plane}:${EXTERNAL_DATA_PLANE_PG_PASS:?external data plane database password required}@external-data-plane-postgres:5432/${EXTERNAL_DATA_PLANE_PG_DB:-nodedc_data_plane}
|
||||
EXTERNAL_DATA_PLANE_DATABASE_POOL_SIZE: ${EXTERNAL_DATA_PLANE_DATABASE_POOL_SIZE:-10}
|
||||
EXTERNAL_DATA_PLANE_RAW_RETENTION_DAYS: ${EXTERNAL_DATA_PLANE_RAW_RETENTION_DAYS:-14}
|
||||
EXTERNAL_DATA_PLANE_MAX_BATCH_BYTES: ${EXTERNAL_DATA_PLANE_MAX_BATCH_BYTES:-5242880}
|
||||
EXTERNAL_DATA_PLANE_MAX_FACTS_PER_PUBLISH: ${EXTERNAL_DATA_PLANE_MAX_FACTS_PER_PUBLISH:-5000}
|
||||
EXTERNAL_DATA_PLANE_MAX_ATTRIBUTES_BYTES_PER_FACT: ${EXTERNAL_DATA_PLANE_MAX_ATTRIBUTES_BYTES_PER_FACT:-65536}
|
||||
EXTERNAL_DATA_PLANE_MAX_PATCH_OPERATIONS: ${EXTERNAL_DATA_PLANE_MAX_PATCH_OPERATIONS:-500}
|
||||
EXTERNAL_DATA_PLANE_MAX_PATCH_BYTES: ${EXTERNAL_DATA_PLANE_MAX_PATCH_BYTES:-262144}
|
||||
EXTERNAL_DATA_PLANE_PATCH_RETENTION_MS: ${EXTERNAL_DATA_PLANE_PATCH_RETENTION_MS:-3600000}
|
||||
EXTERNAL_DATA_PLANE_RECEIPT_RETENTION_MS: ${EXTERNAL_DATA_PLANE_RECEIPT_RETENTION_MS:-604800000}
|
||||
EXTERNAL_DATA_PLANE_RETENTION_DELETE_LIMIT: ${EXTERNAL_DATA_PLANE_RETENTION_DELETE_LIMIT:-10000}
|
||||
EXTERNAL_DATA_PLANE_STREAM_HEARTBEAT_MS: ${EXTERNAL_DATA_PLANE_STREAM_HEARTBEAT_MS:-20000}
|
||||
EXTERNAL_DATA_PLANE_STREAM_POLL_MS: ${EXTERNAL_DATA_PLANE_STREAM_POLL_MS:-1000}
|
||||
EXTERNAL_DATA_PLANE_MAX_READER_STREAMS: ${EXTERNAL_DATA_PLANE_MAX_READER_STREAMS:-10}
|
||||
EXTERNAL_DATA_PLANE_WRITER_BINDING_MAX_TTL_DAYS: ${EXTERNAL_DATA_PLANE_WRITER_BINDING_MAX_TTL_DAYS:-90}
|
||||
EXTERNAL_DATA_PLANE_MAX_FUTURE_SKEW_SECONDS: ${EXTERNAL_DATA_PLANE_MAX_FUTURE_SKEW_SECONDS:-300}
|
||||
EXTERNAL_DATA_PLANE_RETENTION_SWEEP_MS: ${EXTERNAL_DATA_PLANE_RETENTION_SWEEP_MS:-3600000}
|
||||
# Migration-only shared-token routes stay closed in a canonical install.
|
||||
EXTERNAL_DATA_PLANE_LEGACY_INTAKE_ENABLED: ${EXTERNAL_DATA_PLANE_LEGACY_INTAKE_ENABLED:-false}
|
||||
NODEDC_INTERNAL_ACCESS_TOKEN: ${NODEDC_INTERNAL_ACCESS_TOKEN:-}
|
||||
# Runner-owned secret file. This credential must never live in the
|
||||
# shared .env.synology that is inherited by unrelated services.
|
||||
EXTERNAL_DATA_PLANE_PROVISIONER_TOKEN_FILE: /run/nodedc-secrets/external-data-plane-provisioner-token
|
||||
# This stays false until the dedicated Engine provisioner is deployed and
|
||||
# receives the same read-only secret mount.
|
||||
EXTERNAL_DATA_PLANE_PROVISIONING_ENABLED: "false"
|
||||
volumes:
|
||||
- type: bind
|
||||
source: /volume1/docker/nodedc-platform/secrets/external-data-plane-provisioner/token
|
||||
target: /run/nodedc-secrets/external-data-plane-provisioner-token
|
||||
read_only: true
|
||||
bind:
|
||||
create_host_path: false
|
||||
expose:
|
||||
- "18106"
|
||||
ports:
|
||||
- "${EXTERNAL_DATA_PLANE_HOST_BIND:-127.0.0.1:18106}:18106"
|
||||
depends_on:
|
||||
external-data-plane-postgres:
|
||||
condition: service_healthy
|
||||
healthcheck:
|
||||
test:
|
||||
- CMD
|
||||
- node
|
||||
- -e
|
||||
- "fetch('http://127.0.0.1:18106/healthz').then((response) => process.exit(response.ok ? 0 : 1)).catch(() => process.exit(1))"
|
||||
interval: 30s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
start_period: 15s
|
||||
networks:
|
||||
- engine
|
||||
- external-data-plane
|
||||
|
||||
networks:
|
||||
engine:
|
||||
external: true
|
||||
name: ${NODEDC_ENGINE_DOCKER_NETWORK:-nodedc-demo_default}
|
||||
external-data-plane:
|
||||
internal: true
|
||||
|
||||
volumes:
|
||||
external-data-plane-database:
|
||||
|
|
@ -142,8 +142,6 @@ services:
|
|||
AI_WORKSPACE_HUB_INTERNAL_URL: ${AI_WORKSPACE_HUB_INTERNAL_URL:-https://ai-hub.nodedc.ru}
|
||||
AI_WORKSPACE_HUB_TOKEN: ${AI_WORKSPACE_HUB_TOKEN:?ai workspace hub token required}
|
||||
AI_WORKSPACE_HUB_FALLBACK_URLS: ${AI_WORKSPACE_HUB_FALLBACK_URLS:-}
|
||||
AI_WORKSPACE_ONTOLOGY_MCP_ENABLED: ${AI_WORKSPACE_ONTOLOGY_MCP_ENABLED:-true}
|
||||
AI_WORKSPACE_ENTITLEMENT_ADAPTERS_JSON: ${AI_WORKSPACE_ENTITLEMENT_ADAPTERS_JSON:-}
|
||||
NDC_ONTOLOGY_LAUNCHER_BASE_URL: http://launcher:5173
|
||||
expose:
|
||||
- "18082"
|
||||
|
|
@ -156,160 +154,6 @@ services:
|
|||
- identity
|
||||
- engine
|
||||
|
||||
ontology-core:
|
||||
image: nodedc/ontology-core:local
|
||||
build:
|
||||
context: ./ontology-core
|
||||
restart: unless-stopped
|
||||
env_file:
|
||||
- ${NODEDC_SYNOLOGY_ENV_FILE:-.env.synology}
|
||||
environment:
|
||||
NODE_ENV: production
|
||||
PORT: 18104
|
||||
NODEDC_INTERNAL_ACCESS_TOKEN: ${NODEDC_INTERNAL_ACCESS_TOKEN:-}
|
||||
expose:
|
||||
- "18104"
|
||||
ports:
|
||||
- "${ONTOLOGY_CORE_HOST_BIND:-127.0.0.1:18104}:18104"
|
||||
healthcheck:
|
||||
test:
|
||||
- CMD
|
||||
- node
|
||||
- -e
|
||||
- "fetch('http://127.0.0.1:18104/healthz').then((response) => process.exit(response.ok ? 0 : 1)).catch(() => process.exit(1))"
|
||||
interval: 30s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
start_period: 10s
|
||||
networks:
|
||||
- engine
|
||||
|
||||
# The provider database is private. Gateway is the only component that can
|
||||
# reach it; the host bind is loopback-only for audited operator diagnostics.
|
||||
gelios-postgres:
|
||||
image: ${GELIOS_TIMESCALE_IMAGE:-timescale/timescaledb-ha:pg16.14-ts2.28.2-all}
|
||||
restart: unless-stopped
|
||||
env_file:
|
||||
- ${NODEDC_SYNOLOGY_ENV_FILE:-.env.synology}
|
||||
environment:
|
||||
POSTGRES_DB: ${GELIOS_PG_DB:-nodedc_gelios}
|
||||
POSTGRES_USER: ${GELIOS_PG_USER:-nodedc_gelios}
|
||||
POSTGRES_PASSWORD: ${GELIOS_PG_PASS:?gelios database password required}
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -d $${POSTGRES_DB} -U $${POSTGRES_USER}"]
|
||||
interval: 30s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
start_period: 30s
|
||||
volumes:
|
||||
- gelios-database:/home/postgres/pgdata/data
|
||||
networks:
|
||||
- gelios-data
|
||||
|
||||
gelios-gateway:
|
||||
image: nodedc/gelios-gateway:local
|
||||
build:
|
||||
context: ./gelios-gateway
|
||||
restart: unless-stopped
|
||||
env_file:
|
||||
- ${NODEDC_SYNOLOGY_ENV_FILE:-.env.synology}
|
||||
environment:
|
||||
NODE_ENV: production
|
||||
PORT: 18105
|
||||
DATABASE_URL: ${GELIOS_DATABASE_URL:?gelios database URL required}
|
||||
NODEDC_INTERNAL_ACCESS_TOKEN: ${NODEDC_INTERNAL_ACCESS_TOKEN:-}
|
||||
GELIOS_TENANT_ID: ${GELIOS_TENANT_ID:?gelios tenant id required}
|
||||
GELIOS_CONNECTION_ID: ${GELIOS_CONNECTION_ID:?gelios connection id required}
|
||||
GELIOS_UNIT_SCOPE: ${GELIOS_UNIT_SCOPE:-allowlist}
|
||||
GELIOS_ALLOWED_UNIT_IDS: ${GELIOS_ALLOWED_UNIT_IDS:-}
|
||||
GELIOS_INTAKE_ENABLED: ${GELIOS_INTAKE_ENABLED:-false}
|
||||
GELIOS_RAW_RETENTION_DAYS: ${GELIOS_RAW_RETENTION_DAYS:-14}
|
||||
GELIOS_POSITION_STALE_AFTER_MS: ${GELIOS_POSITION_STALE_AFTER_MS:-300000}
|
||||
expose:
|
||||
- "18105"
|
||||
ports:
|
||||
- "${GELIOS_GATEWAY_HOST_BIND:-127.0.0.1:18105}:18105"
|
||||
depends_on:
|
||||
gelios-postgres:
|
||||
condition: service_healthy
|
||||
healthcheck:
|
||||
test:
|
||||
- CMD
|
||||
- node
|
||||
- -e
|
||||
- "fetch('http://127.0.0.1:18105/healthz').then((response) => process.exit(response.ok ? 0 : 1)).catch(() => process.exit(1))"
|
||||
interval: 30s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
start_period: 15s
|
||||
networks:
|
||||
- engine
|
||||
- gelios-data
|
||||
|
||||
# One private Platform gateway and one NAS-resident persistent cache serve
|
||||
# every Foundry Application/Map Page instance. It is not published to a browser.
|
||||
map-gateway:
|
||||
image: nodedc/map-gateway:local
|
||||
build:
|
||||
context: ../../services/map-gateway
|
||||
restart: unless-stopped
|
||||
env_file:
|
||||
- ${NODEDC_SYNOLOGY_ENV_FILE:-.env.synology}
|
||||
environment:
|
||||
NODE_ENV: production
|
||||
PORT: 18103
|
||||
# Optional legacy bootstrap. The canonical mutable value, when an
|
||||
# authorized Foundry admin saves it, is the private NAS secret file
|
||||
# inside MAP_CACHE_DIR and never a deploy artifact or browser env.
|
||||
CESIUM_ION_TOKEN: ${CESIUM_ION_TOKEN:-}
|
||||
# Runner-owned file mount; the secret itself is never an env value.
|
||||
NODEDC_MAP_GATEWAY_ADMIN_SECRET_FILE: /run/nodedc-secrets/map-gateway-admin-secret
|
||||
# DC AMD Proxy preserves the AMD workstation's VPN-only Cesium route.
|
||||
# The fixed NAS LAN listener is authenticated by the same runner-managed
|
||||
# file token; neither token is a browser or Foundry env.
|
||||
MAP_GATEWAY_EGRESS_URL: http://172.22.0.222:8790
|
||||
MAP_GATEWAY_EGRESS_PROXY_TOKEN_FILE: /run/nodedc-secrets/map-egress-proxy-token
|
||||
CESIUM_ION_ASSET_ALLOWLIST: ${CESIUM_ION_ASSET_ALLOWLIST:-1,2,96188}
|
||||
MAP_CACHE_DIR: /var/lib/nodedc-map-live-cache
|
||||
MAP_OFFLINE_SNAPSHOT_DIR: /var/lib/nodedc-map-offline-snapshot
|
||||
MAP_CACHE_MODE: ${MAP_CACHE_MODE:-readwrite}
|
||||
MAP_CACHE_MAX_MB: ${MAP_CACHE_MAX_MB:-20480}
|
||||
MAP_GATEWAY_ALLOW_ANONYMOUS: "false"
|
||||
MAP_GATEWAY_TRUSTED_SUBJECT_HEADER: x-nodedc-user-id
|
||||
MAP_GATEWAY_UPSTREAM_ALLOWLIST: ${MAP_GATEWAY_UPSTREAM_ALLOWLIST:-api.cesium.com,assets.ion.cesium.com,tile.openstreetmap.org,dev.virtualearth.net,ecn.t0.tiles.virtualearth.net,ecn.t1.tiles.virtualearth.net,ecn.t2.tiles.virtualearth.net,ecn.t3.tiles.virtualearth.net}
|
||||
MAP_GATEWAY_LEGACY_CACHE_HOSTS: ${MAP_GATEWAY_LEGACY_CACHE_HOSTS:-}
|
||||
MAP_GATEWAY_OFFLINE_PROVIDER_ALLOWLIST: ${MAP_GATEWAY_OFFLINE_PROVIDER_ALLOWLIST:-}
|
||||
volumes:
|
||||
- type: bind
|
||||
source: ${MAP_LIVE_CACHE_HOST_DIR:-/volume1/docker/nodedc-platform/map-gateway/live-tile-cache}
|
||||
target: /var/lib/nodedc-map-live-cache
|
||||
- type: bind
|
||||
source: ${MAP_OFFLINE_SNAPSHOT_HOST_DIR:-/volume1/docker/nodedc-platform/map-gateway/offline-snapshot}
|
||||
target: /var/lib/nodedc-map-offline-snapshot
|
||||
read_only: true
|
||||
- type: bind
|
||||
source: /volume1/docker/nodedc-platform/secrets/map-gateway-admin-secret
|
||||
target: /run/nodedc-secrets/map-gateway-admin-secret
|
||||
read_only: true
|
||||
- type: bind
|
||||
source: /volume1/docker/nodedc-platform/secrets/map-egress-proxy-token
|
||||
target: /run/nodedc-secrets/map-egress-proxy-token
|
||||
read_only: true
|
||||
# Operator-only loopback binding for the root-owned deploy healthcheck.
|
||||
# Foundry reaches the service over the private Docker network; browsers
|
||||
# never receive this address.
|
||||
ports:
|
||||
- "${MAP_GATEWAY_HOST_BIND:-127.0.0.1:18103}:18103"
|
||||
healthcheck:
|
||||
test: ["CMD", "node", "-e", "fetch('http://127.0.0.1:18103/healthz',{headers:{'x-nodedc-user-id':'healthcheck'}}).then((response)=>process.exit(response.ok?0:1)).catch(()=>process.exit(1))"]
|
||||
interval: 30s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
start_period: 15s
|
||||
networks:
|
||||
- engine
|
||||
- map-egress
|
||||
|
||||
ai-workspace-hub:
|
||||
image: nodedc/ai-workspace-hub:local
|
||||
build:
|
||||
|
|
@ -323,15 +167,11 @@ services:
|
|||
AI_WORKSPACE_HUB_TOKEN: ${AI_WORKSPACE_HUB_TOKEN:?ai workspace hub token required}
|
||||
NODEDC_INTERNAL_ACCESS_TOKEN: ${NODEDC_INTERNAL_ACCESS_TOKEN:-}
|
||||
NODEDC_AI_WORKSPACE_ASSISTANT_URL: http://ai-workspace-assistant:18082
|
||||
NODEDC_ONTOLOGY_CORE_URL: http://ontology-core:18104
|
||||
AI_WORKSPACE_HUB_WS_PATH: /api/ai-workspace/hub
|
||||
expose:
|
||||
- "18081"
|
||||
ports:
|
||||
- "${AI_WORKSPACE_HUB_HOST_BIND:-0.0.0.0:18081}:18081"
|
||||
depends_on:
|
||||
ontology-core:
|
||||
condition: service_healthy
|
||||
networks:
|
||||
- edge
|
||||
- engine
|
||||
|
|
@ -416,11 +256,6 @@ networks:
|
|||
name: ${NODEDC_ENGINE_DOCKER_NETWORK:-nodedc-demo_default}
|
||||
identity:
|
||||
internal: true
|
||||
gelios-data:
|
||||
internal: true
|
||||
map-egress:
|
||||
external: true
|
||||
name: nodedc-map-egress
|
||||
|
||||
volumes:
|
||||
authentik-database:
|
||||
|
|
@ -428,6 +263,5 @@ volumes:
|
|||
authentik-certs:
|
||||
notification-database:
|
||||
ai-workspace-database:
|
||||
gelios-database:
|
||||
caddy-data:
|
||||
caddy-config:
|
||||
|
|
|
|||
|
|
@ -1,39 +0,0 @@
|
|||
#!/bin/sh
|
||||
set -eu
|
||||
|
||||
ENV_FILE=${1:-/volume1/docker/nodedc-platform/platform/.env.synology}
|
||||
|
||||
if [ ! -f "$ENV_FILE" ]; then
|
||||
echo "ERROR: env file not found: $ENV_FILE" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if grep -q '^EXTERNAL_DATA_PLANE_PG_PASS=' "$ENV_FILE"; then
|
||||
echo "External Data Plane database credential already exists; no change made."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if ! command -v openssl >/dev/null 2>&1; then
|
||||
echo "ERROR: openssl is required to generate the database password." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
umask 077
|
||||
TMP_FILE=$(mktemp "${ENV_FILE}.external-data-plane.XXXXXX")
|
||||
trap 'rm -f "$TMP_FILE"' EXIT HUP INT TERM
|
||||
PASSWORD=$(openssl rand -hex 32)
|
||||
|
||||
cp "$ENV_FILE" "$TMP_FILE"
|
||||
printf '\n# External Data Plane — generated locally; do not reuse an internal API token.\n' >> "$TMP_FILE"
|
||||
printf 'EXTERNAL_DATA_PLANE_TIMESCALE_IMAGE=timescale/timescaledb-ha:pg16.14-ts2.28.2-all\n' >> "$TMP_FILE"
|
||||
printf 'EXTERNAL_DATA_PLANE_PG_DB=nodedc_data_plane\n' >> "$TMP_FILE"
|
||||
printf 'EXTERNAL_DATA_PLANE_PG_USER=nodedc_data_plane\n' >> "$TMP_FILE"
|
||||
printf 'EXTERNAL_DATA_PLANE_PG_PASS=%s\n' "$PASSWORD" >> "$TMP_FILE"
|
||||
printf 'EXTERNAL_DATA_PLANE_HOST_BIND=127.0.0.1:18106\n' >> "$TMP_FILE"
|
||||
printf 'EXTERNAL_DATA_PLANE_DATABASE_POOL_SIZE=10\n' >> "$TMP_FILE"
|
||||
printf 'EXTERNAL_DATA_PLANE_RAW_RETENTION_DAYS=14\n' >> "$TMP_FILE"
|
||||
printf 'EXTERNAL_DATA_PLANE_MAX_BATCH_BYTES=5242880\n' >> "$TMP_FILE"
|
||||
|
||||
mv "$TMP_FILE" "$ENV_FILE"
|
||||
trap - EXIT HUP INT TERM
|
||||
echo "External Data Plane database configuration was added. The generated password was not printed."
|
||||
|
|
@ -1,81 +0,0 @@
|
|||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# This script is intentionally run on the Synology host, before the canonical
|
||||
# platform artifact apply. It changes collection policy only; provider
|
||||
# credentials remain in the protected Engine credential store.
|
||||
NAS_ROOT="${NAS_ROOT:-/volume1/docker/nodedc-platform}"
|
||||
PLATFORM_ENV="${PLATFORM_ENV:-${NAS_ROOT}/platform/.env.synology}"
|
||||
BACKUP_ROOT="${BACKUP_ROOT:-/volume1/docker/nodedc-deploy/backups/gelios-policy}"
|
||||
TIMESTAMP="${TIMESTAMP:-$(date +%Y%m%d-%H%M%S)}"
|
||||
BACKUP_DIR="${BACKUP_DIR:-${BACKUP_ROOT}/${TIMESTAMP}}"
|
||||
|
||||
read_env() {
|
||||
local file="$1"
|
||||
local key="$2"
|
||||
awk -F= -v key="${key}" '
|
||||
$0 ~ "^[[:space:]]*#" { next }
|
||||
$1 == key {
|
||||
value = substr($0, index($0, "=") + 1)
|
||||
gsub(/^[[:space:]]+|[[:space:]]+$/, "", value)
|
||||
gsub(/^"|"$/, "", value)
|
||||
print value
|
||||
exit
|
||||
}
|
||||
' "${file}"
|
||||
}
|
||||
|
||||
set_env_value() {
|
||||
local file="$1"
|
||||
local key="$2"
|
||||
local value="$3"
|
||||
local tmp
|
||||
tmp="$(mktemp "${file}.tmp.XXXXXX")"
|
||||
awk -v key="${key}" -v value="${value}" '
|
||||
BEGIN { replaced = 0 }
|
||||
$0 ~ "^[[:space:]]*#" { print; next }
|
||||
index($0, key "=") == 1 {
|
||||
print key "=" value
|
||||
replaced = 1
|
||||
next
|
||||
}
|
||||
{ print }
|
||||
END {
|
||||
if (!replaced) print key "=" value
|
||||
}
|
||||
' "${file}" > "${tmp}"
|
||||
chmod 600 "${tmp}"
|
||||
mv "${tmp}" "${file}"
|
||||
}
|
||||
|
||||
if [[ ! -f "${PLATFORM_ENV}" ]]; then
|
||||
echo "missing Platform env: ${PLATFORM_ENV}" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
tenant_id="$(read_env "${PLATFORM_ENV}" GELIOS_TENANT_ID || true)"
|
||||
connection_id="$(read_env "${PLATFORM_ENV}" GELIOS_CONNECTION_ID || true)"
|
||||
if [[ -z "${tenant_id}" || -z "${connection_id}" || "${tenant_id}" == replace-with-* || "${connection_id}" == replace-with-* ]]; then
|
||||
echo "GELIOS_TENANT_ID and GELIOS_CONNECTION_ID must be configured before enabling all-unit collection" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
mkdir -p "${BACKUP_DIR}"
|
||||
backup_path="${BACKUP_DIR}/platform.env.synology.before-gelios-all-units"
|
||||
cp "${PLATFORM_ENV}" "${backup_path}"
|
||||
chmod 600 "${backup_path}"
|
||||
|
||||
# `all` is constrained to the one tenant + connection validated above; it is
|
||||
# not a wildcard across Gelios tenants or other providers. Existing stable
|
||||
# unit rows are never deleted by this policy.
|
||||
set_env_value "${PLATFORM_ENV}" GELIOS_UNIT_SCOPE "all"
|
||||
set_env_value "${PLATFORM_ENV}" GELIOS_INTAKE_ENABLED "true"
|
||||
|
||||
cat <<EOF
|
||||
gelios-all-units-env-prep-ok
|
||||
backup_path=${backup_path}
|
||||
updated_platform_env=${PLATFORM_ENV}
|
||||
unit_scope=all
|
||||
intake_enabled=true
|
||||
secrets_printed=false
|
||||
EOF
|
||||
|
|
@ -63,14 +63,6 @@ echo "== ai workspace assistant health check =="
|
|||
"${DOCKER_BIN}" exec nodedc-platform-ai-workspace-assistant-1 sh -lc \
|
||||
'node -e '"'"'fetch("http://127.0.0.1:18082/healthz").then(async (response) => { console.log(await response.text()); process.exit(response.ok ? 0 : 1); }).catch((error) => { console.error(error); process.exit(1); })'"'"''
|
||||
|
||||
echo "== ontology core health check =="
|
||||
"${DOCKER_BIN}" exec nodedc-platform-ontology-core-1 sh -lc \
|
||||
'node -e '"'"'fetch("http://127.0.0.1:18104/healthz").then(async (response) => { console.log(await response.text()); process.exit(response.ok ? 0 : 1); }).catch((error) => { console.error(error); process.exit(1); })'"'"''
|
||||
|
||||
echo "== gelios gateway health check =="
|
||||
"${DOCKER_BIN}" exec nodedc-platform-gelios-gateway-1 sh -lc \
|
||||
'node -e '"'"'fetch("http://127.0.0.1:18105/healthz").then(async (response) => { console.log(await response.text()); process.exit(response.ok ? 0 : 1); }).catch((error) => { console.error(error); process.exit(1); })'"'"''
|
||||
|
||||
echo "== ai workspace assistant hub target check =="
|
||||
"${DOCKER_BIN}" exec nodedc-platform-ai-workspace-assistant-1 sh -lc \
|
||||
'test "$AI_WORKSPACE_HUB_PUBLIC_URL" = "wss://ai-hub.nodedc.ru/api/ai-workspace/hub" && test -z "$AI_WORKSPACE_HUB_FALLBACK_URLS" && echo ai-workspace-prod-hub-target-ok'
|
||||
|
|
|
|||
|
|
@ -1,302 +0,0 @@
|
|||
# External Provider Contract
|
||||
|
||||
Этот package — единственное общее место для формы внешних интеграций NODE.DC.
|
||||
Он не содержит provider secrets, customer records, runtime payloads или
|
||||
исполняемый connector code. `src/index.mjs` даёт dependency-free проверку
|
||||
минимальных v1 contracts; она запускается через `npm run check`.
|
||||
|
||||
Каждый L2 connector instance должен поставлять совместимые versioned
|
||||
артефакты:
|
||||
|
||||
```text
|
||||
provider-manifest
|
||||
connection-profile
|
||||
capability-catalog
|
||||
field-catalog
|
||||
collection-profile
|
||||
retention-profile
|
||||
semantic-mapping-contract
|
||||
read-model-contract
|
||||
command-catalog (metadata only until a red command gateway is approved)
|
||||
```
|
||||
|
||||
`provider-manifest` — декларативный template-артефакт: provider ID, ontology
|
||||
revision, L2 template version, capability catalog и data-product IDs. Он не
|
||||
является tenant connection, не содержит endpoint/URL, credential reference,
|
||||
secret, ручной scope или executable provider code. Concrete non-secret
|
||||
connection profile принадлежит и версионируется вместе с конкретным L2
|
||||
workflow; Engine только привязывает к нему opaque credential reference и grant.
|
||||
|
||||
## Проверяемые v1 contracts
|
||||
|
||||
- `Connection` — provider instance, tenant scope и *ссылка* на credential в
|
||||
Engine. Любые token/secret/password-like поля запрещены.
|
||||
- `Collection Profile` — явная policy сбора. `manual` не может скрыто содержать
|
||||
polling interval; `realtime` требует interval не чаще одного раза в секунду.
|
||||
- `Data Product` — нормализованный versioned output с semantic types, полями и
|
||||
внутренней аудиторией.
|
||||
- `Intake Batch` — canonical **scoped** record, который External Data Plane
|
||||
валидирует и сохраняет: source, contract revision, idempotency, restricted
|
||||
raw envelope и canonical facts. Его `source` содержит `providerId`,
|
||||
`tenantId` и `connectionId`. Writer-bound L2 request намеренно не является
|
||||
готовым `Intake Batch`: Data Plane сначала materializes scope и лишь затем
|
||||
применяет этот contract. В canonical record нет provider field mapping,
|
||||
entity allowlist, token или renderer data. Inline `raw.payload` в v1
|
||||
запрещён: если нужна provenance-ссылка, connector передаёт restricted
|
||||
`raw.ref` вместе с hash. Отдельный raw-vault может быть добавлен только
|
||||
отдельным ADR и не становится частью L2 → Data Plane wire boundary.
|
||||
- `NDC Foundry Binding` — адресует data product только в конкретную цепочку
|
||||
`Foundry Application → Page → approved slot`; `templateId` можно сохранить
|
||||
как дополнительную типизацию, но он не заменяет `applicationId` и `pageId`.
|
||||
В binding запрещены provider transport, endpoint и credential reference.
|
||||
- `Provider Manifest` — статическое описание L2 connector template, capability
|
||||
catalog и ontology/data-product contracts. Оно не может содержать tenant,
|
||||
connection, credential, secret или provider transport.
|
||||
|
||||
## Data Product delivery contracts
|
||||
|
||||
Provider-neutral runtime использует отдельные wire schemas:
|
||||
|
||||
- `nodedc.data-product.publish/v1` — unscoped publish request от
|
||||
`NDC Data Product Publish`; содержит только batch identity и canonical facts;
|
||||
- `nodedc.data-product.snapshot/v1` — согласованный current snapshot с cursor;
|
||||
- `nodedc.data-product.patch/v1` — committed upsert operations из durable
|
||||
outbox с `previousCursor`/`cursor`.
|
||||
|
||||
Publish request не может задавать provider, tenant, connection, endpoint,
|
||||
receivedAt, ontology revision, version или persistence policy. Data Plane
|
||||
materializes эти значения из opaque writer binding и зарегистрированного Data
|
||||
Product definition. Snapshot/stream читаются только по отдельному opaque reader
|
||||
binding; shared internal bearer и caller-provided scope headers являются legacy
|
||||
и не используются новыми nodes/Foundry runtime.
|
||||
|
||||
`snapshot+patch` v1 — bounded contract: один scoped Data Product содержит не
|
||||
более 5000 current entity keys `(sourceId, semanticType)`. Snapshot обязан быть
|
||||
полным и привязанным к одному repeatable-read cursor. Параметр `limit` задаёт
|
||||
защитный ceiling, не page size: превышение возвращает
|
||||
`413 data_product_snapshot_limit_exceeded`, а reader не имеет права начинать
|
||||
stream с усечённой базой. `nextPageCursor` зарезервирован для будущего отдельно
|
||||
версионируемого query contract и текущим bounded runtime не выдаётся.
|
||||
|
||||
Для большей cardinality definition заранее раскладывается по стабильным
|
||||
partition Data Products с независимыми snapshot/patch cursors либо использует
|
||||
будущий query contract с единым snapshot barrier и continuation semantics.
|
||||
Offset/source-ID pagination поверх меняющегося current snapshot запрещена:
|
||||
между страницами она способна потерять или задублировать изменения относительно
|
||||
patch cursor.
|
||||
|
||||
Для каждого canonical fact действует одинаковый hard ceiling: сериализованный
|
||||
`attributes` не больше 64 KiB как на publish/intake входе, так и в
|
||||
snapshot/patch выходе. GeoJSON `Point` принимает longitude только в
|
||||
`[-180, 180]`, latitude в `[-90, 90]`; batch sequence ограничен диапазоном
|
||||
PostgreSQL `integer` `0..2147483647`. Эти ограничения нельзя ослабить опциями
|
||||
конкретного caller-а.
|
||||
|
||||
Manifest, connection/collection profiles, data-product definition и Foundry
|
||||
binding используют fail-closed allowed-key schemas. Неизвестные поля, а также
|
||||
secret-like имена или значения (включая `ndc_edpwb_`/`ndc_edprb_`) отклоняются
|
||||
на общей границе.
|
||||
|
||||
Все private custom nodes NODE.DC поставляются package
|
||||
`platform/packages/n8n-nodes-ndc`. Их display name обязан начинаться с `NDC `,
|
||||
а runtime type — с `n8n-nodes-ndc.`; package называется строго
|
||||
`n8n-nodes-ndc`. Provider-specific adapters остаются L2 workflow logic и не
|
||||
становятся custom nodes или ветками Data Plane. Эти инварианты проверяются
|
||||
package test.
|
||||
|
||||
Control-plane команда `NDC Foundry Binding` имеет отдельную replay-safe schema
|
||||
`nodedc.foundry.binding-upsert/v1` (`validateFoundryBindingUpsert`). Это не
|
||||
declarative provider artifact и не runtime transport: команда содержит только
|
||||
application/page/binding/data-product projection и idempotency key, а право на
|
||||
операцию извлекается Foundry из отдельного opaque workload grant.
|
||||
|
||||
## Engine opaque credential sink
|
||||
|
||||
`src/engine-credential-sink.mjs` задаёт dependency-free server-to-server v1
|
||||
границу для доставки трёх workload capabilities в Engine Credentials:
|
||||
|
||||
- `external-data-plane.writer` → точная нода
|
||||
`n8n-nodes-ndc.ndcDataProductPublish` / `ndcDataProductWriterApi`;
|
||||
- `external-data-plane.reader` → точная нода
|
||||
`n8n-nodes-ndc.ndcDataProductRead` / `ndcDataProductReaderApi`;
|
||||
- `foundry.binding` → точная нода
|
||||
`n8n-nodes-ndc.ndcFoundryBinding` / `ndcFoundryBindingApi`.
|
||||
|
||||
Provision request фиксирует `workflowId`, `workflowRevision`, `nodeId`, runtime
|
||||
node type, credential type, grant ID, expiry и issuer policy hash. Aggregate
|
||||
`transaction.policyHash` детерминированно считается по всему secret-free
|
||||
descriptor; подмена любой цели или policy ломает валидацию. Единственное поле,
|
||||
которое переносит plaintext capability, — `bindings[].material.value`; request
|
||||
нельзя писать в логи, traces, очередь или audit.
|
||||
|
||||
Provision и rollback envelopes действуют не более 15 минут: sink отклоняет
|
||||
истёкшие запросы и допускает максимум 60 секунд положительного clock skew.
|
||||
Receipt повторно проверяет freshness относительно собственного `processedAt`,
|
||||
чтобы старый запрос нельзя было применить или подтвердить через replay.
|
||||
|
||||
Каждый binding содержит `capabilityDigest = sha256(material.value)`: Engine
|
||||
самостоятельно хеширует полученный plaintext и сравнивает digest. Aggregate
|
||||
`policyHash` включает этот digest и issuer identity, а provision transaction
|
||||
несёт обязательную Ed25519 attestation. Engine принимает её только по
|
||||
allowlisted `serviceId:keyId`; заменить capability и пересчитать обычный hash
|
||||
без приватного issuer key невозможно.
|
||||
|
||||
Sink обязан выполнять `rollback-all`: сначала проверить весь request и точное
|
||||
состояние graph, затем создать credentials в staging, атомарно привязать весь
|
||||
набор и только после commit вернуть opaque `credentialRef`. При любой ошибке
|
||||
новые credentials удаляются, а прежние bindings остаются без изменений.
|
||||
`rollback-failed` означает карантин и ручное восстановление, но никогда не
|
||||
возвращает частичные credential refs.
|
||||
|
||||
Receipt, rollback request/receipt и audit имеют отдельные strict schemas. Они
|
||||
не способны вернуть capability material; audit хранит только hash opaque
|
||||
credential reference. Explicit rollback адресует committed transaction через
|
||||
`transactionId`, `policyHash` и hash committed receipt, поэтому не может
|
||||
случайно откатить другой набор. Реализация sink принадлежит Engine и не даёт
|
||||
Platform/Codex доступа к Engine core, runtime files или plaintext credential
|
||||
storage.
|
||||
|
||||
## Engine private-extension management
|
||||
|
||||
`src/engine-private-extension.mjs` задаёт строгую Platform-side границу для
|
||||
Engine-owned активации проверенного `n8n-nodes-ndc` release. Контракт не
|
||||
устанавливает package и не меняет Engine: он фиксирует async
|
||||
`plan -> apply receipt -> operation/status` protocol, где `apply` обязан
|
||||
быстро вернуть `queued` и `operationId`, а долгий recreate/acceptance
|
||||
отслеживается отдельно.
|
||||
|
||||
Активация и rollback требуют отдельной глобальной capability
|
||||
`engine.private-extension.manage`. Обычные L1/L2 grants её не дают. Чтение
|
||||
состояния допускает `engine.private-extension.read` или manage-capability.
|
||||
Запрос выбирает только allowlisted package, digest-bound `releaseId` и
|
||||
`packageSha256`; caller не передаёт host path, package bytes, Compose service,
|
||||
shell command или credential material. Plan живёт не более 15 минут, является
|
||||
single-use и применяется только с тем же idempotency key и plan hash.
|
||||
|
||||
Runtime transition для n8n 2.3.2 зафиксирован как community-package loader по
|
||||
`/home/node/.n8n/nodes/node_modules/n8n-nodes-ndc`, а не как
|
||||
`N8N_CUSTOM_EXTENSIONS` или `CUSTOM.*` loader:
|
||||
|
||||
- `N8N_COMMUNITY_PACKAGES_ENABLED=true`,
|
||||
`N8N_COMMUNITY_PACKAGES_PREVENT_LOADING=false`,
|
||||
`N8N_REINSTALL_MISSING_PACKAGES=false`;
|
||||
- Deploy/Run quiesced and execution queue drained before the version switch;
|
||||
- read-only mount and atomic current/recovery state;
|
||||
- force-recreate main, every worker and every webhook instance as one version
|
||||
barrier; hot reload запрещён;
|
||||
- acceptance требует единый generation и точный набор трёх package-qualified
|
||||
node schemas и трёх credential schemas.
|
||||
|
||||
Любая ошибка после switch запускает automatic rollback и повторный
|
||||
force-recreate/acceptance. Ошибка самого rollback переводит runtime в
|
||||
`quarantined`. Immutable release и существующие Engine Credentials
|
||||
сохраняются. Для первой активации предыдущим проверенным состоянием является
|
||||
`n8n-nodes-ndc.inactive/v1`: rollback в этот baseline удаляет package из
|
||||
loader surface, но не удаляет credentials.
|
||||
|
||||
Public Ops gateway уже умеет прозрачно передавать эти операции через
|
||||
`/engine/mcp`, если Engine реализует соответствующие MCP tools. Так как
|
||||
gateway имеет 30-second upstream timeout, side effect остаётся асинхронным;
|
||||
отдельный public REST proxy для management boundary не требуется.
|
||||
|
||||
Пример `examples/gelios-positions-current.v1.mjs` — provider-specific fixture
|
||||
без customer, tenant identity или credential material.
|
||||
|
||||
## Ownership
|
||||
|
||||
- `platform/packages/external-provider-contract` — общий контракт и schemas.
|
||||
- NDC Agent L2 — provider API adapter: fetch, pagination, batching,
|
||||
semantic mapping, collection profile и ссылка на credential в Engine.
|
||||
- Platform External Data Plane — provider-neutral intake, raw retention,
|
||||
canonical facts, current/history projections и scoped read products. Он не
|
||||
знает provider fields, customer/business filters или renderer rules.
|
||||
- `platform/services/ontology-core/catalog/domain-packages/<provider>` —
|
||||
семантика, отношения и guardrails, но не runtime data.
|
||||
- Foundry/interface bindings — consumers scoped data product; они не получают
|
||||
provider transport, endpoint или credential reference.
|
||||
|
||||
`services/<provider>-gateway` — устаревший experimental path и не является
|
||||
шаблоном новых integration services. Канон описан в
|
||||
`docs/ADR_L2_OWNED_EXTERNAL_CONNECTORS.md`.
|
||||
|
||||
## Mandatory connection boundary
|
||||
|
||||
`connection` принадлежит одному tenant/client context и содержит только
|
||||
ссылку на секрет, утверждённый capability scope, collection profile, field
|
||||
policy и retention policy. Во всех runtime records Data Plane сохраняет минимум
|
||||
`tenant_id`, `connection_id`, `provider_id`, `observed_at`, `received_at` и
|
||||
provenance/version там, где это применимо.
|
||||
|
||||
Writer token — отдельный EDP runtime credential, а не поле `Connection`,
|
||||
profile, provider manifest или L2 graph.
|
||||
|
||||
## Scoped writer binding
|
||||
|
||||
Writer binding — EDP-owned runtime security state, а не versioned artifact
|
||||
provider manifest или connection profile. Он фиксирует `tenantId`,
|
||||
`connectionId`, `providerId`, `allowedDataProductIds`, active/revoked state и
|
||||
`expiresAt`. L2 не может редактировать binding или задавать его scope;
|
||||
изменение любого scope-поля либо TTL создаёт новый binding, а прежний binding
|
||||
можно только rotate/revoke.
|
||||
|
||||
Новый L2 вызывает `POST /internal/data-plane/v1/intake/writer-bound` с
|
||||
`Authorization: Bearer <writer-token>` и unscoped envelope. В `source`
|
||||
разрешён только `providerId`; `tenantId`, `connectionId` и
|
||||
`x-nodedc-*-id` headers запрещены. EDP проверяет token, binding, provider и
|
||||
`contract.dataProductId`, затем materializes immutable canonical scope и
|
||||
сохраняет обычный `Intake Batch`. Caller-provided scope отклоняется, а не
|
||||
доверяется и не объединяется с binding.
|
||||
|
||||
Plaintext writer token возвращается только trusted provisioner при создании
|
||||
или rotation. Provisioner помещает его непосредственно в opaque Engine
|
||||
credential, доступный назначенному L2. EDP хранит только hash token и binding
|
||||
metadata; token запрещён в connection/profile/manifest, L2 graph, logs/traces,
|
||||
raw payload, Foundry и UI.
|
||||
|
||||
Provisioning routes принимают только отдельный secret file, созданный
|
||||
root-owned deploy runner:
|
||||
`/volume1/docker/nodedc-platform/secrets/external-data-plane-provisioner/token`.
|
||||
Он read-only монтируется только в EDP и позднее — в dedicated Engine
|
||||
provisioner, работающий под выделенным UID/GID `11006`; это не `.env` value,
|
||||
не `NODEDC_INTERNAL_ACCESS_TOKEN` и не provider credential. Пока generic Engine
|
||||
provisioner не развёрнут, `EXTERNAL_DATA_PLANE_PROVISIONING_ENABLED=false`, а
|
||||
create/rotate/revoke routes отвечают `503` и не делают fallback к shared
|
||||
internal bearer.
|
||||
|
||||
## Legacy intake migration
|
||||
|
||||
`POST /internal/data-plane/v1/intake` — временный compatibility route для
|
||||
existing writers. Он принимает только canonical scoped `Intake Batch` под
|
||||
`NODEDC_INTERNAL_ACCESS_TOKEN` и требует совпадения body scope с
|
||||
`x-nodedc-tenant-id` и `x-nodedc-connection-id`. Writer token этот route не
|
||||
заменяет.
|
||||
|
||||
Для миграции: создать writer binding, один раз записать выданный token в Engine
|
||||
credential, переключить L2 на `/intake/writer-bound`, удалить tenant/connection
|
||||
из body и headers, проверить успешный intake, затем удалить у L2 legacy
|
||||
credential. Новый или migrated writer не должен fallback-иться на legacy route
|
||||
после ошибки writer-bound intake. Когда migrated все writers, legacy route и
|
||||
его shared-token access удаляются.
|
||||
|
||||
## Intake boundary
|
||||
|
||||
L2 отправляет в общий Data Plane `Intake Batch`, а не SQL-запрос в общие
|
||||
таблицы. Platform проверяет boundary и сохраняет raw/history/current
|
||||
projections; semantic mapping остаётся в L2. Список конкретных source IDs не
|
||||
является частью connection scope: scope выбирает read-capability API, а
|
||||
visibility решает data-product consumer.
|
||||
|
||||
Inline `raw.payload` запрещён: L2 не может записать в Data Plane полный ответ
|
||||
provider-а или произвольную строку. Пока отдельный raw-vault не утверждён,
|
||||
batch либо не содержит `raw`, либо содержит только restricted `raw.ref` и hash.
|
||||
Secret-like keys и распознаваемые bearer/JWT/writer-token values в любом участке
|
||||
canonical batch, включая `facts[].attributes`, отклоняются. Retention raw
|
||||
reference вычисляется от server acceptance time, а не от `batch.receivedAt`;
|
||||
service делает sweep expired envelopes при старте и по расписанию.
|
||||
|
||||
## Safety classification
|
||||
|
||||
Capabilities классифицируются как `read`, `metadata`, `write`, `destructive`
|
||||
или `unknown`. Только `read` и согласованные `metadata` могут попасть в
|
||||
collector. `write` и `destructive` остаются каталогизированными, но не имеют
|
||||
transport route в read adapter.
|
||||
|
|
@ -1,76 +0,0 @@
|
|||
import { EXTERNAL_PROVIDER_CONTRACT_VERSION } from "../src/index.mjs";
|
||||
|
||||
export const geliosPositionsCurrentExample = Object.freeze({
|
||||
providerManifest: {
|
||||
schemaVersion: EXTERNAL_PROVIDER_CONTRACT_VERSION,
|
||||
id: "gelios.positions.current",
|
||||
providerId: "gelios",
|
||||
version: "1.0.0",
|
||||
ontology: {
|
||||
packageId: "gelios",
|
||||
revision: "gelios.positions.v1",
|
||||
},
|
||||
l2Template: {
|
||||
id: "gelios.positions.current",
|
||||
version: "1.0.0",
|
||||
},
|
||||
capabilities: [
|
||||
{ id: "gelios.units.current.read", classification: "read" },
|
||||
{ id: "gelios.units.command.write", classification: "write" },
|
||||
],
|
||||
dataProductIds: ["fleet.positions.current.v1"],
|
||||
},
|
||||
connection: {
|
||||
schemaVersion: EXTERNAL_PROVIDER_CONTRACT_VERSION,
|
||||
id: "gelios.sample-fleet",
|
||||
providerId: "gelios",
|
||||
tenantId: "sample-tenant",
|
||||
credentialRef: { owner: "engine", reference: "engine-credential-reference" },
|
||||
scope: {
|
||||
capabilityIds: ["gelios.units.current.read"],
|
||||
fieldPolicyId: "fleet.position.display.v1",
|
||||
},
|
||||
},
|
||||
collectionProfile: {
|
||||
schemaVersion: EXTERNAL_PROVIDER_CONTRACT_VERSION,
|
||||
id: "gelios.positions.realtime",
|
||||
connectionId: "gelios.sample-fleet",
|
||||
mode: "realtime",
|
||||
schedule: { intervalMs: 15000 },
|
||||
capabilityIds: ["gelios.units.current.read"],
|
||||
dataProductId: "fleet.positions.current.v1",
|
||||
},
|
||||
dataProduct: {
|
||||
schemaVersion: EXTERNAL_PROVIDER_CONTRACT_VERSION,
|
||||
id: "fleet.positions.current.v1",
|
||||
version: "1.0.0",
|
||||
delivery: { mode: "snapshot+patch" },
|
||||
semanticTypes: ["map.moving_object"],
|
||||
fields: [
|
||||
"course_degrees",
|
||||
"display_name",
|
||||
"elevation_meters",
|
||||
"geometry",
|
||||
"hdop",
|
||||
"horizontal_accuracy_meters",
|
||||
"object_kind",
|
||||
"operational_status",
|
||||
"position_source",
|
||||
"position_valid",
|
||||
"quality_flags",
|
||||
"satellite_count",
|
||||
"speed_kph",
|
||||
],
|
||||
access: { audience: "internal" },
|
||||
},
|
||||
foundryBinding: {
|
||||
schemaVersion: EXTERNAL_PROVIDER_CONTRACT_VERSION,
|
||||
id: "fleet.operations-map.moving-objects",
|
||||
dataProductId: "fleet.positions.current.v1",
|
||||
applicationId: "11111111-1111-4111-8111-111111111111",
|
||||
pageId: "map",
|
||||
templateId: "map",
|
||||
slotId: "points",
|
||||
semanticType: "map.moving_object",
|
||||
},
|
||||
});
|
||||
|
|
@ -1,10 +0,0 @@
|
|||
{
|
||||
"name": "@nodedc/external-provider-contract",
|
||||
"version": "0.1.0",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"exports": "./src/index.mjs",
|
||||
"scripts": {
|
||||
"check": "node test/contract.test.mjs && node test/data-product.test.mjs && node test/engine-credential-sink.test.mjs && node test/engine-private-extension.test.mjs"
|
||||
}
|
||||
}
|
||||
|
|
@ -1,197 +0,0 @@
|
|||
export const DATA_PRODUCT_PUBLISH_SCHEMA_VERSION = "nodedc.data-product.publish/v1";
|
||||
export const DATA_PRODUCT_SNAPSHOT_SCHEMA_VERSION = "nodedc.data-product.snapshot/v1";
|
||||
export const DATA_PRODUCT_PATCH_SCHEMA_VERSION = "nodedc.data-product.patch/v1";
|
||||
|
||||
const IDENTIFIER = /^[a-z][a-z0-9._:-]{2,127}$/;
|
||||
const SEMVER = /^\d+\.\d+\.\d+(?:[-+][a-z0-9.-]+)?$/i;
|
||||
const CURSOR = /^(?:0|[1-9]\d*)$/;
|
||||
const SECRET_LIKE_KEY = /(token|secret|password|authorization|access[_-]?token|refresh[_-]?token|api[_-]?key)/i;
|
||||
const SECRET_LIKE_VALUE = /(?:ndc_edp(?:wb|rb)_[A-Za-z0-9_-]*|[?&](?:token|secret|password|authorization|access[_-]?token|refresh[_-]?token|api[_-]?key)=|(?:bearer|basic)\s+\S+|eyJ[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+)/i;
|
||||
const MAX_BATCH_SEQUENCE = 2_147_483_647;
|
||||
const MAX_FACT_ATTRIBUTES_BYTES = 64 * 1024;
|
||||
|
||||
/**
|
||||
* Wire form accepted from an NDC Data Product Publish node.
|
||||
*
|
||||
* Scope, provider identity, product version, ontology revision and storage
|
||||
* policy are deliberately absent: the Data Plane materializes them from the
|
||||
* opaque writer grant and its product registry.
|
||||
*/
|
||||
export function validateDataProductPublish(value, { maxFacts = 5000, maxAttributesBytes = 64 * 1024 } = {}) {
|
||||
const errors = [];
|
||||
const attributesCeiling = Number.isInteger(maxAttributesBytes) && maxAttributesBytes > 0
|
||||
? Math.min(maxAttributesBytes, MAX_FACT_ATTRIBUTES_BYTES)
|
||||
: MAX_FACT_ATTRIBUTES_BYTES;
|
||||
if (!isPlainObject(value)) return result(["publish_must_be_object"]);
|
||||
if (value.schemaVersion !== DATA_PRODUCT_PUBLISH_SCHEMA_VERSION) errors.push("schemaVersion_mismatch");
|
||||
rejectUnknownKeys(value, new Set(["schemaVersion", "batch", "facts"]), "publish", errors);
|
||||
|
||||
if (!isPlainObject(value.batch)) {
|
||||
errors.push("batch_must_be_object");
|
||||
} else {
|
||||
rejectUnknownKeys(value.batch, new Set(["runId", "sequence", "idempotencyKey"]), "batch", errors);
|
||||
requiredIdentifier(value.batch.runId, "batch.runId", errors);
|
||||
requiredIdentifier(value.batch.idempotencyKey, "batch.idempotencyKey", errors);
|
||||
if (!Number.isInteger(value.batch.sequence) || value.batch.sequence < 0 || value.batch.sequence > MAX_BATCH_SEQUENCE) {
|
||||
errors.push("batch.sequence_must_be_integer_0_to_2147483647");
|
||||
}
|
||||
}
|
||||
|
||||
if (!Array.isArray(value.facts) || value.facts.length === 0) {
|
||||
errors.push("facts_must_be_nonempty_array");
|
||||
} else if (value.facts.length > maxFacts) {
|
||||
errors.push("facts_limit_exceeded");
|
||||
} else {
|
||||
const entityKeys = new Set();
|
||||
value.facts.forEach((fact, index) => {
|
||||
validateFact(fact, `facts[${index}]`, errors, { maxAttributesBytes: attributesCeiling });
|
||||
if (!isPlainObject(fact) || typeof fact.sourceId !== "string" || typeof fact.semanticType !== "string") return;
|
||||
const entityKey = `${fact.sourceId}\u0000${fact.semanticType}`;
|
||||
if (entityKeys.has(entityKey)) errors.push("facts_duplicate_entity_key");
|
||||
entityKeys.add(entityKey);
|
||||
});
|
||||
}
|
||||
|
||||
if (containsSecretLikeMaterial(value)) errors.push("publish_must_not_contain_secret_material");
|
||||
return result(errors);
|
||||
}
|
||||
|
||||
export function validateDataProductSnapshot(value) {
|
||||
const errors = envelopeErrors(value, DATA_PRODUCT_SNAPSHOT_SCHEMA_VERSION, "snapshot");
|
||||
rejectUnknownKeys(value, new Set(["schemaVersion", "dataProduct", "generatedAt", "cursor", "facts", "nextPageCursor"]), "snapshot", errors);
|
||||
requiredCursor(value?.cursor, "cursor", errors);
|
||||
requiredIsoTimestamp(value?.generatedAt, "generatedAt", errors);
|
||||
if (!Array.isArray(value?.facts)) {
|
||||
errors.push("facts_must_be_array");
|
||||
} else {
|
||||
value.facts.forEach((fact, index) => validateCanonicalFact(fact, `facts[${index}]`, errors));
|
||||
}
|
||||
if (value?.nextPageCursor !== undefined) requiredString(value.nextPageCursor, "nextPageCursor", errors);
|
||||
if (containsSecretLikeMaterial(value)) errors.push("snapshot_must_not_contain_secret_material");
|
||||
return result(errors);
|
||||
}
|
||||
|
||||
export function validateDataProductPatch(value) {
|
||||
const errors = envelopeErrors(value, DATA_PRODUCT_PATCH_SCHEMA_VERSION, "patch");
|
||||
rejectUnknownKeys(value, new Set(["schemaVersion", "dataProduct", "cursor", "previousCursor", "emittedAt", "operations"]), "patch", errors);
|
||||
requiredCursor(value?.cursor, "cursor", errors);
|
||||
requiredCursor(value?.previousCursor, "previousCursor", errors);
|
||||
requiredIsoTimestamp(value?.emittedAt, "emittedAt", errors);
|
||||
if (!Array.isArray(value?.operations) || value.operations.length === 0) {
|
||||
errors.push("operations_must_be_nonempty_array");
|
||||
} else {
|
||||
value.operations.forEach((operation, index) => {
|
||||
if (!isPlainObject(operation) || operation.op !== "upsert") {
|
||||
errors.push(`operations[${index}].op_must_be_upsert`);
|
||||
return;
|
||||
}
|
||||
rejectUnknownKeys(operation, new Set(["op", "fact"]), `operations[${index}]`, errors);
|
||||
validateCanonicalFact(operation.fact, `operations[${index}].fact`, errors);
|
||||
});
|
||||
}
|
||||
if (containsSecretLikeMaterial(value)) errors.push("patch_must_not_contain_secret_material");
|
||||
return result(errors);
|
||||
}
|
||||
|
||||
function envelopeErrors(value, schemaVersion, label) {
|
||||
const errors = [];
|
||||
if (!isPlainObject(value)) return [`${label}_must_be_object`];
|
||||
if (value.schemaVersion !== schemaVersion) errors.push("schemaVersion_mismatch");
|
||||
if (!isPlainObject(value.dataProduct)) {
|
||||
errors.push("dataProduct_must_be_object");
|
||||
} else {
|
||||
rejectUnknownKeys(value.dataProduct, new Set(["id", "version"]), "dataProduct", errors);
|
||||
requiredIdentifier(value.dataProduct.id, "dataProduct.id", errors);
|
||||
requiredString(value.dataProduct.version, "dataProduct.version", errors);
|
||||
if (value.dataProduct.version && !SEMVER.test(value.dataProduct.version)) errors.push("dataProduct.version_must_be_semver");
|
||||
}
|
||||
return errors;
|
||||
}
|
||||
|
||||
function validateFact(value, path, errors, { maxAttributesBytes, canonical = false }) {
|
||||
if (!isPlainObject(value)) {
|
||||
errors.push(`${path}_must_be_object`);
|
||||
return;
|
||||
}
|
||||
const allowedKeys = new Set(["sourceId", "semanticType", "observedAt", "attributes", "geometry"]);
|
||||
if (canonical) allowedKeys.add("receivedAt");
|
||||
rejectUnknownKeys(value, allowedKeys, path, errors);
|
||||
requiredIdentifier(value.sourceId, `${path}.sourceId`, errors);
|
||||
requiredIdentifier(value.semanticType, `${path}.semanticType`, errors);
|
||||
requiredIsoTimestamp(value.observedAt, `${path}.observedAt`, errors);
|
||||
if (value.attributes !== undefined) {
|
||||
if (!isPlainObject(value.attributes)) {
|
||||
errors.push(`${path}.attributes_must_be_object`);
|
||||
} else if (serializedByteLength(value.attributes) > maxAttributesBytes) {
|
||||
errors.push(`${path}.attributes_size_exceeded`);
|
||||
}
|
||||
}
|
||||
if (value.geometry !== undefined) validatePointGeometry(value.geometry, `${path}.geometry`, errors);
|
||||
}
|
||||
|
||||
function validateCanonicalFact(value, path, errors) {
|
||||
validateFact(value, path, errors, { maxAttributesBytes: 64 * 1024, canonical: true });
|
||||
if (!isPlainObject(value)) return;
|
||||
requiredIsoTimestamp(value.receivedAt, `${path}.receivedAt`, errors);
|
||||
}
|
||||
|
||||
function validatePointGeometry(value, path, errors) {
|
||||
if (!isPlainObject(value) || value.type !== "Point" || !Array.isArray(value.coordinates) || value.coordinates.length !== 2) {
|
||||
errors.push(`${path}_must_be_geojson_point`);
|
||||
return;
|
||||
}
|
||||
rejectUnknownKeys(value, new Set(["type", "coordinates"]), path, errors);
|
||||
if (!value.coordinates.every((coordinate) => typeof coordinate === "number" && Number.isFinite(coordinate))) {
|
||||
errors.push(`${path}_coordinates_must_be_finite_numbers`);
|
||||
return;
|
||||
}
|
||||
const [longitude, latitude] = value.coordinates;
|
||||
if (longitude < -180 || longitude > 180) errors.push(`${path}.longitude_out_of_range`);
|
||||
if (latitude < -90 || latitude > 90) errors.push(`${path}.latitude_out_of_range`);
|
||||
}
|
||||
|
||||
function rejectUnknownKeys(value, allowed, path, errors) {
|
||||
if (!isPlainObject(value)) return;
|
||||
for (const key of Object.keys(value)) {
|
||||
if (!allowed.has(key)) errors.push(`${path}.${key}_not_allowed`);
|
||||
}
|
||||
}
|
||||
|
||||
function requiredIdentifier(value, path, errors) {
|
||||
if (typeof value !== "string" || !IDENTIFIER.test(value)) errors.push(`${path}_invalid`);
|
||||
}
|
||||
|
||||
function requiredString(value, path, errors) {
|
||||
if (typeof value !== "string" || !value.trim()) errors.push(`${path}_required`);
|
||||
}
|
||||
|
||||
function requiredCursor(value, path, errors) {
|
||||
if (typeof value !== "string" || !CURSOR.test(value)) errors.push(`${path}_invalid`);
|
||||
}
|
||||
|
||||
function requiredIsoTimestamp(value, path, errors) {
|
||||
if (typeof value !== "string" || Number.isNaN(Date.parse(value))) errors.push(`${path}_invalid_timestamp`);
|
||||
}
|
||||
|
||||
function isPlainObject(value) {
|
||||
return Boolean(value) && typeof value === "object" && !Array.isArray(value);
|
||||
}
|
||||
|
||||
function serializedByteLength(value) {
|
||||
try {
|
||||
return Buffer.byteLength(JSON.stringify(value));
|
||||
} catch {
|
||||
return Number.POSITIVE_INFINITY;
|
||||
}
|
||||
}
|
||||
|
||||
function containsSecretLikeMaterial(value) {
|
||||
if (typeof value === "string") return SECRET_LIKE_VALUE.test(value);
|
||||
if (Array.isArray(value)) return value.some(containsSecretLikeMaterial);
|
||||
if (!isPlainObject(value)) return false;
|
||||
return Object.entries(value).some(([key, child]) => SECRET_LIKE_KEY.test(key) || containsSecretLikeMaterial(child));
|
||||
}
|
||||
|
||||
function result(errors) {
|
||||
return Object.freeze({ ok: errors.length === 0, errors: Object.freeze([...new Set(errors)]) });
|
||||
}
|
||||
|
|
@ -1,662 +0,0 @@
|
|||
import { createHash, verify as verifySignature } from "node:crypto";
|
||||
|
||||
export const ENGINE_CREDENTIAL_SINK_PROVISION_SCHEMA_VERSION = "nodedc.engine.credential-sink.provision/v1";
|
||||
export const ENGINE_CREDENTIAL_SINK_RECEIPT_SCHEMA_VERSION = "nodedc.engine.credential-sink.receipt/v1";
|
||||
export const ENGINE_CREDENTIAL_SINK_ROLLBACK_SCHEMA_VERSION = "nodedc.engine.credential-sink.rollback/v1";
|
||||
export const ENGINE_CREDENTIAL_SINK_ROLLBACK_RECEIPT_SCHEMA_VERSION = "nodedc.engine.credential-sink.rollback-receipt/v1";
|
||||
export const ENGINE_CREDENTIAL_SINK_AUDIT_SCHEMA_VERSION = "nodedc.engine.credential-sink.audit/v1";
|
||||
|
||||
const HASH = /^sha256:[a-f0-9]{64}$/;
|
||||
const IDENTIFIER = /^[a-z][a-z0-9._:-]{2,127}$/;
|
||||
const OPAQUE_ID = /^[A-Za-z0-9][A-Za-z0-9._:-]{2,159}$/;
|
||||
const NODE_TYPE = /^[a-z][A-Za-z0-9.-]{2,159}$/;
|
||||
const CREDENTIAL_TYPE = /^[a-z][A-Za-z0-9]{2,127}$/;
|
||||
const CREDENTIAL_REFERENCE = /^[A-Za-z0-9][A-Za-z0-9_-]{5,159}$/;
|
||||
const REASON_CODE = /^[a-z][a-z0-9_.:-]{2,127}$/;
|
||||
const ED25519_SIGNATURE = /^[A-Za-z0-9_-]{86}$/;
|
||||
const SECRET_LIKE_KEY = /(token|secret|password|authorization|access[_-]?token|refresh[_-]?token|api[_-]?key|material|value)/i;
|
||||
const SECRET_LIKE_VALUE = /(?:ndc_(?:edp(?:wb|rb)|fndbg)_[A-Za-z0-9_-]+|(?:bearer|basic)\s+\S+|eyJ[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+)/i;
|
||||
const MAX_BINDINGS = 32;
|
||||
const MAX_REQUEST_LIFETIME_MS = 15 * 60 * 1000;
|
||||
const MAX_REQUEST_CLOCK_SKEW_MS = 60 * 1000;
|
||||
|
||||
const CAPABILITY_SPECS = Object.freeze({
|
||||
"external-data-plane.writer": Object.freeze({
|
||||
nodeType: "n8n-nodes-ndc.ndcDataProductPublish",
|
||||
credentialType: "ndcDataProductWriterApi",
|
||||
materialPattern: /^ndc_edpwb_[A-Za-z0-9_-]{43}$/,
|
||||
}),
|
||||
"external-data-plane.reader": Object.freeze({
|
||||
nodeType: "n8n-nodes-ndc.ndcDataProductRead",
|
||||
credentialType: "ndcDataProductReaderApi",
|
||||
materialPattern: /^ndc_edprb_[A-Za-z0-9_-]{43}$/,
|
||||
}),
|
||||
"foundry.binding": Object.freeze({
|
||||
nodeType: "n8n-nodes-ndc.ndcFoundryBinding",
|
||||
credentialType: "ndcFoundryBindingApi",
|
||||
materialPattern: /^ndc_fndbg_[A-Za-z0-9_-]{43}$/,
|
||||
}),
|
||||
});
|
||||
|
||||
export const ENGINE_CREDENTIAL_SINK_CAPABILITY_TYPES = Object.freeze(Object.keys(CAPABILITY_SPECS));
|
||||
|
||||
const PROVISION_KEYS = new Set(["schemaVersion", "transaction", "bindings"]);
|
||||
const TRANSACTION_KEYS = new Set([
|
||||
"id",
|
||||
"idempotencyKey",
|
||||
"requestedAt",
|
||||
"requestExpiresAt",
|
||||
"policyHash",
|
||||
"failureMode",
|
||||
"issuer",
|
||||
"attestation",
|
||||
]);
|
||||
const ISSUER_KEYS = new Set(["serviceId", "keyId"]);
|
||||
const ATTESTATION_KEYS = new Set(["algorithm", "signature"]);
|
||||
const BINDING_KEYS = new Set([
|
||||
"bindingId",
|
||||
"capabilityType",
|
||||
"grantId",
|
||||
"target",
|
||||
"expiresAt",
|
||||
"policyHash",
|
||||
"capabilityDigest",
|
||||
"material",
|
||||
]);
|
||||
const TARGET_KEYS = new Set([
|
||||
"workflowId",
|
||||
"workflowRevision",
|
||||
"nodeId",
|
||||
"nodeType",
|
||||
"credentialType",
|
||||
]);
|
||||
const MATERIAL_KEYS = new Set(["format", "value"]);
|
||||
const RECEIPT_KEYS = new Set([
|
||||
"schemaVersion",
|
||||
"transactionId",
|
||||
"idempotencyKey",
|
||||
"outcome",
|
||||
"policyHash",
|
||||
"processedAt",
|
||||
"credentials",
|
||||
"rollback",
|
||||
"errorCode",
|
||||
]);
|
||||
const RECEIPT_CREDENTIAL_KEYS = new Set([
|
||||
"bindingId",
|
||||
"capabilityType",
|
||||
"grantId",
|
||||
"target",
|
||||
"credentialRef",
|
||||
"expiresAt",
|
||||
"policyHash",
|
||||
"capabilityDigest",
|
||||
"disposition",
|
||||
]);
|
||||
const RECEIPT_ROLLBACK_KEYS = new Set(["status", "completedAt"]);
|
||||
const ROLLBACK_REQUEST_KEYS = new Set(["schemaVersion", "rollback"]);
|
||||
const ROLLBACK_KEYS = new Set([
|
||||
"id",
|
||||
"idempotencyKey",
|
||||
"transactionId",
|
||||
"requestedAt",
|
||||
"requestExpiresAt",
|
||||
"policyHash",
|
||||
"committedReceiptHash",
|
||||
"reasonCode",
|
||||
]);
|
||||
const ROLLBACK_RECEIPT_KEYS = new Set([
|
||||
"schemaVersion",
|
||||
"rollbackId",
|
||||
"transactionId",
|
||||
"outcome",
|
||||
"policyHash",
|
||||
"committedReceiptHash",
|
||||
"processedAt",
|
||||
"errorCode",
|
||||
]);
|
||||
const AUDIT_KEYS = new Set([
|
||||
"schemaVersion",
|
||||
"eventId",
|
||||
"transactionId",
|
||||
"operationId",
|
||||
"operation",
|
||||
"outcome",
|
||||
"occurredAt",
|
||||
"policyHash",
|
||||
"principal",
|
||||
"targets",
|
||||
"reasonCode",
|
||||
]);
|
||||
const PRINCIPAL_KEYS = new Set(["serviceId", "fingerprint"]);
|
||||
const AUDIT_TARGET_KEYS = new Set([
|
||||
"bindingId",
|
||||
"capabilityType",
|
||||
"grantId",
|
||||
"target",
|
||||
"expiresAt",
|
||||
"policyHash",
|
||||
"capabilityDigest",
|
||||
"credentialRefHash",
|
||||
]);
|
||||
|
||||
/**
|
||||
* Validates the only request allowed to carry plaintext workload capability
|
||||
* material across the trusted Platform -> Engine server boundary. Callers and
|
||||
* receivers must never log, trace, persist or return this request body.
|
||||
*/
|
||||
export function validateEngineCredentialSinkProvision(value, { now = Date.now(), issuerPublicKeys } = {}) {
|
||||
const errors = [];
|
||||
const nowMs = normalizeNow(now, errors);
|
||||
if (!isPlainObject(value)) return result(["credentialSinkProvision_must_be_object"]);
|
||||
if (value.schemaVersion !== ENGINE_CREDENTIAL_SINK_PROVISION_SCHEMA_VERSION) errors.push("schemaVersion_mismatch");
|
||||
rejectUnknownKeys(value, PROVISION_KEYS, "credentialSinkProvision", errors);
|
||||
|
||||
if (!isPlainObject(value.transaction)) {
|
||||
errors.push("transaction_must_be_object");
|
||||
} else {
|
||||
rejectUnknownKeys(value.transaction, TRANSACTION_KEYS, "transaction", errors);
|
||||
requiredOpaqueId(value.transaction.id, "transaction.id", errors);
|
||||
requiredOpaqueId(value.transaction.idempotencyKey, "transaction.idempotencyKey", errors);
|
||||
requiredTimestamp(value.transaction.requestedAt, "transaction.requestedAt", errors);
|
||||
requiredTimestamp(value.transaction.requestExpiresAt, "transaction.requestExpiresAt", errors);
|
||||
requiredHash(value.transaction.policyHash, "transaction.policyHash", errors);
|
||||
if (value.transaction.failureMode !== "rollback-all") errors.push("transaction.failureMode_must_be_rollback-all");
|
||||
validateIssuer(value.transaction.issuer, "transaction.issuer", errors);
|
||||
validateAttestation(value.transaction.attestation, "transaction.attestation", errors);
|
||||
validateRequestWindow(value.transaction.requestedAt, value.transaction.requestExpiresAt, nowMs, errors);
|
||||
}
|
||||
|
||||
if (!Array.isArray(value.bindings) || value.bindings.length === 0) {
|
||||
errors.push("bindings_must_be_nonempty_array");
|
||||
} else if (value.bindings.length > MAX_BINDINGS) {
|
||||
errors.push("bindings_limit_exceeded");
|
||||
} else {
|
||||
const bindingIds = new Set();
|
||||
const targets = new Set();
|
||||
value.bindings.forEach((binding, index) => {
|
||||
validateProvisionBinding(binding, index, value.transaction?.requestedAt, errors);
|
||||
if (!isPlainObject(binding)) return;
|
||||
if (bindingIds.has(binding.bindingId)) errors.push("bindings_bindingId_must_be_unique");
|
||||
bindingIds.add(binding.bindingId);
|
||||
const targetKey = targetIdentity(binding.target);
|
||||
if (targetKey && targets.has(targetKey)) errors.push("bindings_target_credential_must_be_unique");
|
||||
if (targetKey) targets.add(targetKey);
|
||||
});
|
||||
}
|
||||
|
||||
if (isPlainObject(value.transaction) && HASH.test(String(value.transaction.policyHash || ""))) {
|
||||
const expectedHash = computeEngineCredentialSinkPolicyHash(value);
|
||||
if (value.transaction.policyHash !== expectedHash) errors.push("transaction.policyHash_mismatch");
|
||||
verifyProvisionAttestation(value.transaction, issuerPublicKeys, errors);
|
||||
}
|
||||
return result(errors);
|
||||
}
|
||||
|
||||
/**
|
||||
* Computes the aggregate policy digest over the complete secret-free request
|
||||
* descriptor. Plaintext `material` is excluded by construction, while its
|
||||
* high-entropy capability digest is included; changing a target, grant,
|
||||
* capability, expiry, individual policy hash or transaction envelope changes
|
||||
* the aggregate digest and invalidates the issuer attestation.
|
||||
*/
|
||||
export function computeEngineCredentialSinkPolicyHash(value) {
|
||||
const descriptor = {
|
||||
schemaVersion: value?.schemaVersion,
|
||||
transaction: isPlainObject(value?.transaction) ? {
|
||||
id: value.transaction.id,
|
||||
idempotencyKey: value.transaction.idempotencyKey,
|
||||
requestedAt: value.transaction.requestedAt,
|
||||
requestExpiresAt: value.transaction.requestExpiresAt,
|
||||
failureMode: value.transaction.failureMode,
|
||||
issuer: value.transaction.issuer,
|
||||
} : value?.transaction,
|
||||
bindings: Array.isArray(value?.bindings)
|
||||
? value.bindings.map(secretFreeBindingDescriptor)
|
||||
: value?.bindings,
|
||||
};
|
||||
return `sha256:${createHash("sha256").update(stableJson(descriptor), "utf8").digest("hex")}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Receipt is intentionally incapable of carrying credential material. When a
|
||||
* request is supplied, the validator also proves the sink committed the exact
|
||||
* requested workflow/node/type set without target substitution.
|
||||
*/
|
||||
export function validateEngineCredentialSinkReceipt(value, { request, issuerPublicKeys } = {}) {
|
||||
const errors = [];
|
||||
if (!isPlainObject(value)) return result(["credentialSinkReceipt_must_be_object"]);
|
||||
if (value.schemaVersion !== ENGINE_CREDENTIAL_SINK_RECEIPT_SCHEMA_VERSION) errors.push("schemaVersion_mismatch");
|
||||
rejectUnknownKeys(value, RECEIPT_KEYS, "credentialSinkReceipt", errors);
|
||||
requiredOpaqueId(value.transactionId, "transactionId", errors);
|
||||
requiredOpaqueId(value.idempotencyKey, "idempotencyKey", errors);
|
||||
requiredHash(value.policyHash, "policyHash", errors);
|
||||
requiredTimestamp(value.processedAt, "processedAt", errors);
|
||||
const outcomes = new Set(["committed", "rolled-back", "rejected", "rollback-failed"]);
|
||||
if (!outcomes.has(value.outcome)) errors.push("outcome_invalid");
|
||||
|
||||
if (!Array.isArray(value.credentials)) {
|
||||
errors.push("credentials_must_be_array");
|
||||
} else {
|
||||
const bindingIds = new Set();
|
||||
const refs = new Set();
|
||||
value.credentials.forEach((credential, index) => {
|
||||
validateReceiptCredential(credential, index, errors);
|
||||
if (!isPlainObject(credential)) return;
|
||||
if (bindingIds.has(credential.bindingId)) errors.push("credentials_bindingId_must_be_unique");
|
||||
bindingIds.add(credential.bindingId);
|
||||
if (refs.has(credential.credentialRef)) errors.push("credentials_credentialRef_must_be_unique");
|
||||
refs.add(credential.credentialRef);
|
||||
});
|
||||
}
|
||||
validateReceiptOutcome(value, errors);
|
||||
if (containsSecretLikeMaterial(value)) errors.push("receipt_must_not_contain_secret_material");
|
||||
if (request !== undefined) compareReceiptToProvisionRequest(value, request, issuerPublicKeys, errors);
|
||||
return result(errors);
|
||||
}
|
||||
|
||||
export function computeEngineCredentialSinkReceiptHash(value) {
|
||||
return `sha256:${createHash("sha256").update(stableJson(value), "utf8").digest("hex")}`;
|
||||
}
|
||||
|
||||
export function validateEngineCredentialSinkRollback(value, { now = Date.now() } = {}) {
|
||||
const errors = [];
|
||||
const nowMs = normalizeNow(now, errors);
|
||||
if (!isPlainObject(value)) return result(["credentialSinkRollback_must_be_object"]);
|
||||
if (value.schemaVersion !== ENGINE_CREDENTIAL_SINK_ROLLBACK_SCHEMA_VERSION) errors.push("schemaVersion_mismatch");
|
||||
rejectUnknownKeys(value, ROLLBACK_REQUEST_KEYS, "credentialSinkRollback", errors);
|
||||
if (!isPlainObject(value.rollback)) {
|
||||
errors.push("rollback_must_be_object");
|
||||
return result(errors);
|
||||
}
|
||||
rejectUnknownKeys(value.rollback, ROLLBACK_KEYS, "rollback", errors);
|
||||
requiredOpaqueId(value.rollback.id, "rollback.id", errors);
|
||||
requiredOpaqueId(value.rollback.idempotencyKey, "rollback.idempotencyKey", errors);
|
||||
requiredOpaqueId(value.rollback.transactionId, "rollback.transactionId", errors);
|
||||
requiredTimestamp(value.rollback.requestedAt, "rollback.requestedAt", errors);
|
||||
requiredTimestamp(value.rollback.requestExpiresAt, "rollback.requestExpiresAt", errors);
|
||||
requiredHash(value.rollback.policyHash, "rollback.policyHash", errors);
|
||||
requiredHash(value.rollback.committedReceiptHash, "rollback.committedReceiptHash", errors);
|
||||
requiredReason(value.rollback.reasonCode, "rollback.reasonCode", errors);
|
||||
validateRequestWindow(value.rollback.requestedAt, value.rollback.requestExpiresAt, nowMs, errors);
|
||||
if (containsSecretLikeMaterial(value)) errors.push("rollback_must_not_contain_secret_material");
|
||||
return result(errors);
|
||||
}
|
||||
|
||||
export function validateEngineCredentialSinkRollbackReceipt(value, { request } = {}) {
|
||||
const errors = [];
|
||||
if (!isPlainObject(value)) return result(["credentialSinkRollbackReceipt_must_be_object"]);
|
||||
if (value.schemaVersion !== ENGINE_CREDENTIAL_SINK_ROLLBACK_RECEIPT_SCHEMA_VERSION) errors.push("schemaVersion_mismatch");
|
||||
rejectUnknownKeys(value, ROLLBACK_RECEIPT_KEYS, "credentialSinkRollbackReceipt", errors);
|
||||
requiredOpaqueId(value.rollbackId, "rollbackId", errors);
|
||||
requiredOpaqueId(value.transactionId, "transactionId", errors);
|
||||
requiredHash(value.policyHash, "policyHash", errors);
|
||||
requiredHash(value.committedReceiptHash, "committedReceiptHash", errors);
|
||||
requiredTimestamp(value.processedAt, "processedAt", errors);
|
||||
if (!new Set(["rolled-back", "rejected", "rollback-failed"]).has(value.outcome)) errors.push("outcome_invalid");
|
||||
if (value.outcome === "rolled-back") {
|
||||
if (value.errorCode !== undefined) errors.push("errorCode_forbidden_for_success");
|
||||
} else {
|
||||
requiredReason(value.errorCode, "errorCode", errors);
|
||||
}
|
||||
if (containsSecretLikeMaterial(value)) errors.push("rollbackReceipt_must_not_contain_secret_material");
|
||||
if (request !== undefined && isPlainObject(request?.rollback)) {
|
||||
if (!validateEngineCredentialSinkRollback(request, { now: value.processedAt }).ok) {
|
||||
errors.push("request_invalid_for_rollback_receipt_comparison");
|
||||
}
|
||||
if (value.rollbackId !== request.rollback.id) errors.push("rollbackId_request_mismatch");
|
||||
if (value.transactionId !== request.rollback.transactionId) errors.push("transactionId_request_mismatch");
|
||||
if (value.policyHash !== request.rollback.policyHash) errors.push("policyHash_request_mismatch");
|
||||
if (value.committedReceiptHash !== request.rollback.committedReceiptHash) {
|
||||
errors.push("committedReceiptHash_request_mismatch");
|
||||
}
|
||||
}
|
||||
return result(errors);
|
||||
}
|
||||
|
||||
export function validateEngineCredentialSinkAudit(value) {
|
||||
const errors = [];
|
||||
if (!isPlainObject(value)) return result(["credentialSinkAudit_must_be_object"]);
|
||||
if (value.schemaVersion !== ENGINE_CREDENTIAL_SINK_AUDIT_SCHEMA_VERSION) errors.push("schemaVersion_mismatch");
|
||||
rejectUnknownKeys(value, AUDIT_KEYS, "credentialSinkAudit", errors);
|
||||
requiredOpaqueId(value.eventId, "eventId", errors);
|
||||
requiredOpaqueId(value.transactionId, "transactionId", errors);
|
||||
requiredOpaqueId(value.operationId, "operationId", errors);
|
||||
if (!new Set(["provision", "rollback"]).has(value.operation)) errors.push("operation_invalid");
|
||||
if (!new Set(["committed", "rolled-back", "rejected", "rollback-failed"]).has(value.outcome)) errors.push("outcome_invalid");
|
||||
if (value.operation === "provision" && value.outcome === "rolled-back" && !value.reasonCode) {
|
||||
errors.push("reasonCode_required");
|
||||
}
|
||||
if (value.operation === "rollback" && value.outcome === "committed") errors.push("rollback_outcome_invalid");
|
||||
requiredTimestamp(value.occurredAt, "occurredAt", errors);
|
||||
requiredHash(value.policyHash, "policyHash", errors);
|
||||
if (!isPlainObject(value.principal)) {
|
||||
errors.push("principal_must_be_object");
|
||||
} else {
|
||||
rejectUnknownKeys(value.principal, PRINCIPAL_KEYS, "principal", errors);
|
||||
requiredIdentifier(value.principal.serviceId, "principal.serviceId", errors);
|
||||
requiredHash(value.principal.fingerprint, "principal.fingerprint", errors);
|
||||
}
|
||||
if (!Array.isArray(value.targets)) {
|
||||
errors.push("targets_must_be_array");
|
||||
} else {
|
||||
value.targets.forEach((target, index) => validateAuditTarget(target, index, errors));
|
||||
}
|
||||
if (value.reasonCode !== undefined) requiredReason(value.reasonCode, "reasonCode", errors);
|
||||
if (new Set(["rejected", "rollback-failed"]).has(value.outcome) && value.reasonCode === undefined) {
|
||||
errors.push("reasonCode_required");
|
||||
}
|
||||
if (containsSecretLikeMaterial(value)) errors.push("audit_must_not_contain_secret_material");
|
||||
return result(errors);
|
||||
}
|
||||
|
||||
/** Returns audit-safe target descriptors; material and credential refs cannot escape. */
|
||||
export function engineCredentialSinkAuditTargets(request, receipt) {
|
||||
const refByBinding = new Map(
|
||||
Array.isArray(receipt?.credentials)
|
||||
? receipt.credentials.map((item) => [item.bindingId, item.credentialRef])
|
||||
: [],
|
||||
);
|
||||
return Array.isArray(request?.bindings) ? request.bindings.map((binding) => {
|
||||
const descriptor = secretFreeBindingDescriptor(binding);
|
||||
const credentialRef = refByBinding.get(binding.bindingId);
|
||||
return credentialRef
|
||||
? { ...descriptor, credentialRefHash: sha256Value(credentialRef) }
|
||||
: descriptor;
|
||||
}) : [];
|
||||
}
|
||||
|
||||
function validateProvisionBinding(value, index, requestedAt, errors) {
|
||||
const path = `bindings[${index}]`;
|
||||
if (!isPlainObject(value)) {
|
||||
errors.push(`${path}_must_be_object`);
|
||||
return;
|
||||
}
|
||||
rejectUnknownKeys(value, BINDING_KEYS, path, errors);
|
||||
requiredIdentifier(value.bindingId, `${path}.bindingId`, errors);
|
||||
const spec = CAPABILITY_SPECS[value.capabilityType];
|
||||
if (!spec) errors.push(`${path}.capabilityType_invalid`);
|
||||
requiredOpaqueId(value.grantId, `${path}.grantId`, errors);
|
||||
validateTarget(value.target, path, errors);
|
||||
requiredTimestamp(value.expiresAt, `${path}.expiresAt`, errors);
|
||||
requiredHash(value.policyHash, `${path}.policyHash`, errors);
|
||||
requiredHash(value.capabilityDigest, `${path}.capabilityDigest`, errors);
|
||||
if (isTimestamp(requestedAt) && isTimestamp(value.expiresAt) && Date.parse(value.expiresAt) <= Date.parse(requestedAt)) {
|
||||
errors.push(`${path}.expiresAt_must_be_after_requestedAt`);
|
||||
}
|
||||
if (spec && isPlainObject(value.target)) {
|
||||
if (value.target.nodeType !== spec.nodeType) errors.push(`${path}.target.nodeType_capability_mismatch`);
|
||||
if (value.target.credentialType !== spec.credentialType) errors.push(`${path}.target.credentialType_capability_mismatch`);
|
||||
}
|
||||
if (!isPlainObject(value.material)) {
|
||||
errors.push(`${path}.material_must_be_object`);
|
||||
} else {
|
||||
rejectUnknownKeys(value.material, MATERIAL_KEYS, `${path}.material`, errors);
|
||||
if (value.material.format !== "opaque-bearer") errors.push(`${path}.material.format_must_be_opaque-bearer`);
|
||||
if (!spec || typeof value.material.value !== "string" || !spec.materialPattern.test(value.material.value)) {
|
||||
errors.push(`${path}.material.value_invalid_for_capability`);
|
||||
} else if (value.capabilityDigest !== computeEngineCredentialCapabilityDigest(value.material.value)) {
|
||||
errors.push(`${path}.capabilityDigest_material_mismatch`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function validateTarget(value, path, errors) {
|
||||
if (!isPlainObject(value)) {
|
||||
errors.push(`${path}.target_must_be_object`);
|
||||
return;
|
||||
}
|
||||
rejectUnknownKeys(value, TARGET_KEYS, `${path}.target`, errors);
|
||||
requiredOpaqueId(value.workflowId, `${path}.target.workflowId`, errors);
|
||||
requiredOpaqueId(value.workflowRevision, `${path}.target.workflowRevision`, errors);
|
||||
requiredOpaqueId(value.nodeId, `${path}.target.nodeId`, errors);
|
||||
if (typeof value.nodeType !== "string" || !NODE_TYPE.test(value.nodeType)) errors.push(`${path}.target.nodeType_invalid`);
|
||||
if (typeof value.credentialType !== "string" || !CREDENTIAL_TYPE.test(value.credentialType)) {
|
||||
errors.push(`${path}.target.credentialType_invalid`);
|
||||
}
|
||||
}
|
||||
|
||||
function validateReceiptCredential(value, index, errors) {
|
||||
const path = `credentials[${index}]`;
|
||||
if (!isPlainObject(value)) {
|
||||
errors.push(`${path}_must_be_object`);
|
||||
return;
|
||||
}
|
||||
rejectUnknownKeys(value, RECEIPT_CREDENTIAL_KEYS, path, errors);
|
||||
requiredIdentifier(value.bindingId, `${path}.bindingId`, errors);
|
||||
if (!CAPABILITY_SPECS[value.capabilityType]) errors.push(`${path}.capabilityType_invalid`);
|
||||
requiredOpaqueId(value.grantId, `${path}.grantId`, errors);
|
||||
validateTarget(value.target, path, errors);
|
||||
if (typeof value.credentialRef !== "string" || !CREDENTIAL_REFERENCE.test(value.credentialRef)) {
|
||||
errors.push(`${path}.credentialRef_invalid`);
|
||||
}
|
||||
requiredTimestamp(value.expiresAt, `${path}.expiresAt`, errors);
|
||||
requiredHash(value.policyHash, `${path}.policyHash`, errors);
|
||||
requiredHash(value.capabilityDigest, `${path}.capabilityDigest`, errors);
|
||||
if (!new Set(["created", "reused", "rotated"]).has(value.disposition)) errors.push(`${path}.disposition_invalid`);
|
||||
}
|
||||
|
||||
function validateReceiptOutcome(value, errors) {
|
||||
if (!isPlainObject(value.rollback)) {
|
||||
errors.push("rollback_must_be_object");
|
||||
return;
|
||||
}
|
||||
rejectUnknownKeys(value.rollback, RECEIPT_ROLLBACK_KEYS, "rollback", errors);
|
||||
const expectedRollback = {
|
||||
committed: "not-required",
|
||||
"rolled-back": "complete",
|
||||
rejected: "not-started",
|
||||
"rollback-failed": "incomplete",
|
||||
}[value.outcome];
|
||||
if (expectedRollback && value.rollback.status !== expectedRollback) errors.push("rollback.status_outcome_mismatch");
|
||||
if (new Set(["complete", "incomplete"]).has(value.rollback.status)) {
|
||||
requiredTimestamp(value.rollback.completedAt, "rollback.completedAt", errors);
|
||||
} else if (value.rollback.completedAt !== undefined) {
|
||||
errors.push("rollback.completedAt_not_allowed");
|
||||
}
|
||||
if (value.outcome === "committed") {
|
||||
if (!Array.isArray(value.credentials) || value.credentials.length === 0) errors.push("committed_credentials_required");
|
||||
if (value.errorCode !== undefined) errors.push("errorCode_forbidden_for_success");
|
||||
} else {
|
||||
if (Array.isArray(value.credentials) && value.credentials.length !== 0) errors.push("noncommitted_credentials_must_be_empty");
|
||||
requiredReason(value.errorCode, "errorCode", errors);
|
||||
}
|
||||
}
|
||||
|
||||
function compareReceiptToProvisionRequest(receipt, request, issuerPublicKeys, errors) {
|
||||
const requestValidation = validateEngineCredentialSinkProvision(request, {
|
||||
now: receipt.processedAt,
|
||||
issuerPublicKeys,
|
||||
});
|
||||
if (!requestValidation.ok) {
|
||||
errors.push("request_invalid_for_receipt_comparison");
|
||||
return;
|
||||
}
|
||||
if (receipt.transactionId !== request.transaction.id) errors.push("transactionId_request_mismatch");
|
||||
if (receipt.idempotencyKey !== request.transaction.idempotencyKey) errors.push("idempotencyKey_request_mismatch");
|
||||
if (receipt.policyHash !== request.transaction.policyHash) errors.push("policyHash_request_mismatch");
|
||||
if (receipt.outcome !== "committed") return;
|
||||
if (receipt.credentials.length !== request.bindings.length) errors.push("credentials_request_count_mismatch");
|
||||
const requested = new Map(request.bindings.map((binding) => [binding.bindingId, secretFreeBindingDescriptor(binding)]));
|
||||
for (const credential of receipt.credentials) {
|
||||
const expected = requested.get(credential.bindingId);
|
||||
if (!expected || stableJson({
|
||||
bindingId: credential.bindingId,
|
||||
capabilityType: credential.capabilityType,
|
||||
grantId: credential.grantId,
|
||||
target: credential.target,
|
||||
expiresAt: credential.expiresAt,
|
||||
policyHash: credential.policyHash,
|
||||
capabilityDigest: credential.capabilityDigest,
|
||||
}) !== stableJson(expected)) {
|
||||
errors.push("credentials_request_target_mismatch");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function validateAuditTarget(value, index, errors) {
|
||||
const path = `targets[${index}]`;
|
||||
if (!isPlainObject(value)) {
|
||||
errors.push(`${path}_must_be_object`);
|
||||
return;
|
||||
}
|
||||
rejectUnknownKeys(value, AUDIT_TARGET_KEYS, path, errors);
|
||||
requiredIdentifier(value.bindingId, `${path}.bindingId`, errors);
|
||||
if (!CAPABILITY_SPECS[value.capabilityType]) errors.push(`${path}.capabilityType_invalid`);
|
||||
requiredOpaqueId(value.grantId, `${path}.grantId`, errors);
|
||||
validateTarget(value.target, path, errors);
|
||||
requiredTimestamp(value.expiresAt, `${path}.expiresAt`, errors);
|
||||
requiredHash(value.policyHash, `${path}.policyHash`, errors);
|
||||
requiredHash(value.capabilityDigest, `${path}.capabilityDigest`, errors);
|
||||
if (value.credentialRefHash !== undefined) requiredHash(value.credentialRefHash, `${path}.credentialRefHash`, errors);
|
||||
}
|
||||
|
||||
function secretFreeBindingDescriptor(value) {
|
||||
return {
|
||||
bindingId: value?.bindingId,
|
||||
capabilityType: value?.capabilityType,
|
||||
grantId: value?.grantId,
|
||||
target: value?.target,
|
||||
expiresAt: value?.expiresAt,
|
||||
policyHash: value?.policyHash,
|
||||
capabilityDigest: value?.capabilityDigest,
|
||||
};
|
||||
}
|
||||
|
||||
export function computeEngineCredentialCapabilityDigest(value) {
|
||||
return sha256Value(value);
|
||||
}
|
||||
|
||||
function validateIssuer(value, path, errors) {
|
||||
if (!isPlainObject(value)) {
|
||||
errors.push(`${path}_must_be_object`);
|
||||
return;
|
||||
}
|
||||
rejectUnknownKeys(value, ISSUER_KEYS, path, errors);
|
||||
requiredIdentifier(value.serviceId, `${path}.serviceId`, errors);
|
||||
requiredOpaqueId(value.keyId, `${path}.keyId`, errors);
|
||||
}
|
||||
|
||||
function validateAttestation(value, path, errors) {
|
||||
if (!isPlainObject(value)) {
|
||||
errors.push(`${path}_must_be_object`);
|
||||
return;
|
||||
}
|
||||
rejectUnknownKeys(value, ATTESTATION_KEYS, path, errors);
|
||||
if (value.algorithm !== "Ed25519") errors.push(`${path}.algorithm_must_be_Ed25519`);
|
||||
if (typeof value.signature !== "string" || !ED25519_SIGNATURE.test(value.signature)) {
|
||||
errors.push(`${path}.signature_invalid`);
|
||||
}
|
||||
}
|
||||
|
||||
function verifyProvisionAttestation(transaction, issuerPublicKeys, errors) {
|
||||
if (!isPlainObject(transaction?.issuer) || !isPlainObject(transaction?.attestation)) return;
|
||||
if (transaction.attestation.algorithm !== "Ed25519" || !ED25519_SIGNATURE.test(String(transaction.attestation.signature || ""))) return;
|
||||
const keyIdentity = `${transaction.issuer.serviceId}:${transaction.issuer.keyId}`;
|
||||
const publicKey = isPlainObject(issuerPublicKeys) && Object.hasOwn(issuerPublicKeys, keyIdentity)
|
||||
? issuerPublicKeys[keyIdentity]
|
||||
: undefined;
|
||||
if (!publicKey) {
|
||||
errors.push("transaction.issuer_public_key_required");
|
||||
return;
|
||||
}
|
||||
let valid = false;
|
||||
try {
|
||||
valid = verifySignature(
|
||||
null,
|
||||
Buffer.from(String(transaction.policyHash), "utf8"),
|
||||
publicKey,
|
||||
Buffer.from(transaction.attestation.signature, "base64url"),
|
||||
);
|
||||
} catch {
|
||||
valid = false;
|
||||
}
|
||||
if (!valid) errors.push("transaction.attestation_invalid");
|
||||
}
|
||||
|
||||
function validateRequestWindow(requestedAt, requestExpiresAt, nowMs, errors) {
|
||||
if (!isTimestamp(requestedAt) || !isTimestamp(requestExpiresAt)) return;
|
||||
const requested = Date.parse(requestedAt);
|
||||
const expires = Date.parse(requestExpiresAt);
|
||||
if (expires <= requested) errors.push("requestExpiresAt_must_be_after_requestedAt");
|
||||
if (expires - requested > MAX_REQUEST_LIFETIME_MS) errors.push("request_lifetime_exceeds_15_minutes");
|
||||
if (Number.isFinite(nowMs)) {
|
||||
if (requested > nowMs + MAX_REQUEST_CLOCK_SKEW_MS) errors.push("requestedAt_exceeds_clock_skew");
|
||||
if (expires <= nowMs) errors.push("request_expired");
|
||||
}
|
||||
}
|
||||
|
||||
function normalizeNow(value, errors) {
|
||||
const normalized = value instanceof Date ? value.getTime() : typeof value === "string" ? Date.parse(value) : Number(value);
|
||||
if (!Number.isFinite(normalized)) {
|
||||
errors.push("validation_now_invalid");
|
||||
return Number.NaN;
|
||||
}
|
||||
return normalized;
|
||||
}
|
||||
|
||||
function targetIdentity(value) {
|
||||
if (!isPlainObject(value)) return "";
|
||||
return [value.workflowId, value.workflowRevision, value.nodeId, value.nodeType, value.credentialType].join("\u0000");
|
||||
}
|
||||
|
||||
function requiredIdentifier(value, path, errors) {
|
||||
if (typeof value !== "string" || !IDENTIFIER.test(value)) errors.push(`${path}_invalid`);
|
||||
}
|
||||
|
||||
function requiredOpaqueId(value, path, errors) {
|
||||
if (typeof value !== "string" || !OPAQUE_ID.test(value)) errors.push(`${path}_invalid`);
|
||||
}
|
||||
|
||||
function requiredHash(value, path, errors) {
|
||||
if (typeof value !== "string" || !HASH.test(value)) errors.push(`${path}_invalid`);
|
||||
}
|
||||
|
||||
function requiredReason(value, path, errors) {
|
||||
if (typeof value !== "string" || !REASON_CODE.test(value)) errors.push(`${path}_invalid`);
|
||||
}
|
||||
|
||||
function requiredTimestamp(value, path, errors) {
|
||||
if (!isTimestamp(value)) errors.push(`${path}_invalid_timestamp`);
|
||||
}
|
||||
|
||||
function isTimestamp(value) {
|
||||
return typeof value === "string" && !Number.isNaN(Date.parse(value)) && new Date(value).toISOString() === value;
|
||||
}
|
||||
|
||||
function sha256Value(value) {
|
||||
return `sha256:${createHash("sha256").update(String(value), "utf8").digest("hex")}`;
|
||||
}
|
||||
|
||||
function stableJson(value) {
|
||||
return JSON.stringify(sortValue(value));
|
||||
}
|
||||
|
||||
function sortValue(value) {
|
||||
if (Array.isArray(value)) return value.map(sortValue);
|
||||
if (!isPlainObject(value)) return value;
|
||||
return Object.fromEntries(Object.keys(value).sort().map((key) => [key, sortValue(value[key])]));
|
||||
}
|
||||
|
||||
function containsSecretLikeMaterial(value) {
|
||||
if (typeof value === "string") return SECRET_LIKE_VALUE.test(value);
|
||||
if (Array.isArray(value)) return value.some(containsSecretLikeMaterial);
|
||||
if (!isPlainObject(value)) return false;
|
||||
return Object.entries(value).some(([key, child]) => SECRET_LIKE_KEY.test(key) || containsSecretLikeMaterial(child));
|
||||
}
|
||||
|
||||
function isPlainObject(value) {
|
||||
return Boolean(value) && typeof value === "object" && !Array.isArray(value);
|
||||
}
|
||||
|
||||
function rejectUnknownKeys(value, allowed, path, errors) {
|
||||
if (!isPlainObject(value)) return;
|
||||
for (const key of Object.keys(value)) {
|
||||
if (!allowed.has(key)) errors.push(`${path}.${key}_not_allowed`);
|
||||
}
|
||||
}
|
||||
|
||||
function result(errors) {
|
||||
const uniqueErrors = [...new Set(errors)];
|
||||
return Object.freeze({ ok: uniqueErrors.length === 0, errors: Object.freeze(uniqueErrors) });
|
||||
}
|
||||
|
|
@ -1,721 +0,0 @@
|
|||
import { createHash } from "node:crypto";
|
||||
|
||||
export const ENGINE_PRIVATE_EXTENSION_PLAN_REQUEST_SCHEMA_VERSION =
|
||||
"nodedc.engine.private-extension.plan-request/v1";
|
||||
export const ENGINE_PRIVATE_EXTENSION_PLAN_SCHEMA_VERSION =
|
||||
"nodedc.engine.private-extension.plan/v1";
|
||||
export const ENGINE_PRIVATE_EXTENSION_APPLY_REQUEST_SCHEMA_VERSION =
|
||||
"nodedc.engine.private-extension.apply-request/v1";
|
||||
export const ENGINE_PRIVATE_EXTENSION_APPLY_RECEIPT_SCHEMA_VERSION =
|
||||
"nodedc.engine.private-extension.apply-receipt/v1";
|
||||
export const ENGINE_PRIVATE_EXTENSION_OPERATION_SCHEMA_VERSION =
|
||||
"nodedc.engine.private-extension.operation/v1";
|
||||
export const ENGINE_PRIVATE_EXTENSION_STATUS_SCHEMA_VERSION =
|
||||
"nodedc.engine.private-extension.status/v1";
|
||||
|
||||
export const ENGINE_PRIVATE_EXTENSION_MANAGE_CAPABILITY = "engine.private-extension.manage";
|
||||
export const ENGINE_PRIVATE_EXTENSION_READ_CAPABILITY = "engine.private-extension.read";
|
||||
export const ENGINE_PRIVATE_EXTENSION_PACKAGE_NAME = "n8n-nodes-ndc";
|
||||
export const ENGINE_PRIVATE_EXTENSION_INACTIVE_BASELINE = "n8n-nodes-ndc.inactive/v1";
|
||||
|
||||
export const ENGINE_PRIVATE_EXTENSION_NODE_TYPES = Object.freeze([
|
||||
"n8n-nodes-ndc.ndcDataProductPublish",
|
||||
"n8n-nodes-ndc.ndcDataProductRead",
|
||||
"n8n-nodes-ndc.ndcFoundryBinding",
|
||||
]);
|
||||
|
||||
export const ENGINE_PRIVATE_EXTENSION_CREDENTIAL_TYPES = Object.freeze([
|
||||
"ndcDataProductWriterApi",
|
||||
"ndcDataProductReaderApi",
|
||||
"ndcFoundryBindingApi",
|
||||
]);
|
||||
export const ENGINE_PRIVATE_EXTENSION_CREDENTIAL_SCHEMAS = ENGINE_PRIVATE_EXTENSION_CREDENTIAL_TYPES;
|
||||
|
||||
const PLAN_REQUEST_KEYS = new Set([
|
||||
"schemaVersion",
|
||||
"requestId",
|
||||
"idempotencyKey",
|
||||
"action",
|
||||
"requestedAt",
|
||||
"requestExpiresAt",
|
||||
"expectedCurrentGeneration",
|
||||
"target",
|
||||
]);
|
||||
const RELEASE_STATE_KEYS = new Set(["kind", "packageName", "releaseId", "packageSha256"]);
|
||||
const INACTIVE_STATE_KEYS = new Set(["kind", "packageName", "baselineId"]);
|
||||
const PREVIOUS_STATE_TARGET_KEYS = new Set(["kind", "packageName"]);
|
||||
const PLAN_KEYS = new Set([
|
||||
"schemaVersion",
|
||||
"planId",
|
||||
"planHash",
|
||||
"requestId",
|
||||
"idempotencyKey",
|
||||
"action",
|
||||
"createdAt",
|
||||
"expiresAt",
|
||||
"singleUse",
|
||||
"requiredCapability",
|
||||
"expectedCurrentGeneration",
|
||||
"nextGeneration",
|
||||
"currentState",
|
||||
"targetState",
|
||||
"recoveryState",
|
||||
"actions",
|
||||
"transition",
|
||||
"acceptance",
|
||||
"failurePolicy",
|
||||
]);
|
||||
const TRANSITION_KEYS = new Set([
|
||||
"mountMode",
|
||||
"loaderMode",
|
||||
"loaderPath",
|
||||
"loaderEnvironment",
|
||||
"quiesceMode",
|
||||
"stateSwitch",
|
||||
"runtimeAction",
|
||||
"scope",
|
||||
"requireUniformGeneration",
|
||||
"hotReload",
|
||||
"preserveCredentials",
|
||||
]);
|
||||
const LOADER_ENVIRONMENT_KEYS = new Set([
|
||||
"N8N_COMMUNITY_PACKAGES_ENABLED",
|
||||
"N8N_COMMUNITY_PACKAGES_PREVENT_LOADING",
|
||||
"N8N_REINSTALL_MISSING_PACKAGES",
|
||||
]);
|
||||
const ACCEPTANCE_SPEC_KEYS = new Set([
|
||||
"mode",
|
||||
"nodeTypes",
|
||||
"credentialSchemas",
|
||||
"requireUniformGeneration",
|
||||
]);
|
||||
const FAILURE_POLICY_KEYS = new Set([
|
||||
"mode",
|
||||
"rollbackFailureOutcome",
|
||||
"preserveImmutableRelease",
|
||||
"preserveCredentials",
|
||||
]);
|
||||
const APPLY_REQUEST_KEYS = new Set([
|
||||
"schemaVersion",
|
||||
"planId",
|
||||
"planHash",
|
||||
"idempotencyKey",
|
||||
"confirmedAt",
|
||||
]);
|
||||
const APPLY_RECEIPT_KEYS = new Set([
|
||||
"schemaVersion",
|
||||
"operationId",
|
||||
"planId",
|
||||
"planHash",
|
||||
"action",
|
||||
"acceptedAt",
|
||||
"state",
|
||||
]);
|
||||
const OPERATION_KEYS = new Set([
|
||||
"schemaVersion",
|
||||
"operationId",
|
||||
"planId",
|
||||
"planHash",
|
||||
"action",
|
||||
"state",
|
||||
"outcome",
|
||||
"phase",
|
||||
"expectedCurrentGeneration",
|
||||
"nextGeneration",
|
||||
"targetState",
|
||||
"recoveryState",
|
||||
"effectiveState",
|
||||
"runtime",
|
||||
"acceptance",
|
||||
"updatedAt",
|
||||
"errorCode",
|
||||
]);
|
||||
const RUNTIME_KEYS = new Set(["mode", "expectedInstances", "readyInstances", "generation"]);
|
||||
const ACCEPTANCE_REPORT_KEYS = new Set([
|
||||
"state",
|
||||
"nodeTypes",
|
||||
"credentialSchemas",
|
||||
"uniformGeneration",
|
||||
]);
|
||||
const OBSERVATION_KEYS = new Set(["expected", "observed"]);
|
||||
const STATUS_KEYS = new Set([
|
||||
"schemaVersion",
|
||||
"packageName",
|
||||
"generation",
|
||||
"health",
|
||||
"currentState",
|
||||
"previousState",
|
||||
"activeOperationId",
|
||||
"runtime",
|
||||
"acceptance",
|
||||
"updatedAt",
|
||||
"errorCode",
|
||||
]);
|
||||
|
||||
const RELEASE_ID = /^\d+\.\d+\.\d+-[a-f0-9]{16}$/;
|
||||
const SHA256 = /^[a-f0-9]{64}$/;
|
||||
const HASH = /^sha256:[a-f0-9]{64}$/;
|
||||
const OPAQUE_ID = /^[A-Za-z0-9][A-Za-z0-9._:-]{2,159}$/;
|
||||
const REASON_CODE = /^[a-z][a-z0-9_.:-]{2,127}$/;
|
||||
const MAX_REQUEST_LIFETIME_MS = 15 * 60 * 1000;
|
||||
const MAX_CLOCK_SKEW_MS = 60 * 1000;
|
||||
|
||||
const ACTIVATE_ACTIONS = Object.freeze([
|
||||
"verify_staged_immutable_release",
|
||||
"prepare_sealed_package_tree",
|
||||
"verify_community_package_loader_policy",
|
||||
"quiesce_deploy_run_and_drain_queue",
|
||||
"record_recovery_state",
|
||||
"atomic_switch_current",
|
||||
"force_recreate_main_workers_webhooks_as_version_barrier",
|
||||
"verify_exact_runtime_acceptance",
|
||||
"commit_active_state",
|
||||
"resume_deploy_run",
|
||||
]);
|
||||
const ROLLBACK_ACTIONS = Object.freeze([
|
||||
"verify_previous_activation_state",
|
||||
"verify_community_package_loader_policy",
|
||||
"quiesce_deploy_run_and_drain_queue",
|
||||
"record_recovery_state",
|
||||
"atomic_switch_current_to_previous",
|
||||
"force_recreate_main_workers_webhooks_as_version_barrier",
|
||||
"verify_exact_runtime_acceptance",
|
||||
"commit_rolled_back_state",
|
||||
"resume_deploy_run",
|
||||
]);
|
||||
|
||||
const NON_TERMINAL_STATES = new Set([
|
||||
"queued",
|
||||
"preparing",
|
||||
"switching",
|
||||
"recreating",
|
||||
"accepting",
|
||||
"rolling-back",
|
||||
]);
|
||||
const TERMINAL_STATE_OUTCOMES = Object.freeze({
|
||||
active: "committed",
|
||||
rejected: "rejected",
|
||||
quarantined: "quarantined",
|
||||
});
|
||||
|
||||
export function authorizeEnginePrivateExtensionOperation(operation, grantedCapabilities) {
|
||||
const capabilities = new Set(Array.isArray(grantedCapabilities) ? grantedCapabilities : []);
|
||||
const requiredCapability = operation === "status"
|
||||
? ENGINE_PRIVATE_EXTENSION_READ_CAPABILITY
|
||||
: ENGINE_PRIVATE_EXTENSION_MANAGE_CAPABILITY;
|
||||
const authorized = operation === "status"
|
||||
? capabilities.has(ENGINE_PRIVATE_EXTENSION_READ_CAPABILITY)
|
||||
|| capabilities.has(ENGINE_PRIVATE_EXTENSION_MANAGE_CAPABILITY)
|
||||
: capabilities.has(ENGINE_PRIVATE_EXTENSION_MANAGE_CAPABILITY);
|
||||
return Object.freeze({
|
||||
ok: authorized,
|
||||
requiredCapability,
|
||||
errors: Object.freeze(authorized ? [] : ["engine_private_extension_capability_required"]),
|
||||
});
|
||||
}
|
||||
|
||||
export function validateEnginePrivateExtensionPlanRequest(value, options = {}) {
|
||||
const errors = [];
|
||||
validatePlanRequestBody(value, options.now, errors);
|
||||
addAuthorizationErrors("plan", options.grantedCapabilities, errors);
|
||||
return result(errors);
|
||||
}
|
||||
|
||||
export function computeEnginePrivateExtensionPlanHash(value) {
|
||||
if (!isPlainObject(value)) return "";
|
||||
const descriptor = { ...value };
|
||||
delete descriptor.planHash;
|
||||
return "sha256:" + createHash("sha256").update(stableJson(descriptor), "utf8").digest("hex");
|
||||
}
|
||||
|
||||
export function validateEnginePrivateExtensionPlan(value, { request } = {}) {
|
||||
const errors = [];
|
||||
if (!isPlainObject(value)) return result(["enginePrivateExtensionPlan_must_be_object"]);
|
||||
if (value.schemaVersion !== ENGINE_PRIVATE_EXTENSION_PLAN_SCHEMA_VERSION) errors.push("schemaVersion_mismatch");
|
||||
rejectUnknownKeys(value, PLAN_KEYS, "enginePrivateExtensionPlan", errors);
|
||||
requiredOpaqueId(value.planId, "planId", errors);
|
||||
requiredHash(value.planHash, "planHash", errors);
|
||||
requiredOpaqueId(value.requestId, "requestId", errors);
|
||||
requiredOpaqueId(value.idempotencyKey, "idempotencyKey", errors);
|
||||
if (!new Set(["activate", "rollback"]).has(value.action)) errors.push("action_invalid");
|
||||
requiredTimestamp(value.createdAt, "createdAt", errors);
|
||||
requiredTimestamp(value.expiresAt, "expiresAt", errors);
|
||||
validateWindow(value.createdAt, value.expiresAt, Date.parse(value.createdAt), errors, "plan");
|
||||
if (value.singleUse !== true) errors.push("singleUse_must_be_true");
|
||||
if (value.requiredCapability !== ENGINE_PRIVATE_EXTENSION_MANAGE_CAPABILITY) {
|
||||
errors.push("requiredCapability_mismatch");
|
||||
}
|
||||
requiredGeneration(value.expectedCurrentGeneration, "expectedCurrentGeneration", errors);
|
||||
requiredGeneration(value.nextGeneration, "nextGeneration", errors);
|
||||
if (Number.isInteger(value.expectedCurrentGeneration)
|
||||
&& value.nextGeneration !== value.expectedCurrentGeneration + 1) {
|
||||
errors.push("nextGeneration_must_increment_current_generation");
|
||||
}
|
||||
validateActivationState(value.currentState, "currentState", errors);
|
||||
validateActivationState(value.targetState, "targetState", errors);
|
||||
validateActivationState(value.recoveryState, "recoveryState", errors);
|
||||
if (!sameValue(value.currentState, value.recoveryState)) errors.push("recoveryState_must_equal_currentState");
|
||||
if (value.action === "activate" && value.targetState?.kind !== "release") {
|
||||
errors.push("activate_targetState_must_be_release");
|
||||
}
|
||||
if (sameValue(value.currentState, value.targetState)) errors.push("targetState_must_differ_from_currentState");
|
||||
validateExactArray(
|
||||
value.actions,
|
||||
value.action === "rollback" ? ROLLBACK_ACTIONS : ACTIVATE_ACTIONS,
|
||||
"actions",
|
||||
errors,
|
||||
);
|
||||
validateTransition(value.transition, errors);
|
||||
validateAcceptanceSpec(value.acceptance, value.targetState, errors);
|
||||
validateFailurePolicy(value.failurePolicy, errors);
|
||||
if (HASH.test(String(value.planHash || "")) && value.planHash !== computeEnginePrivateExtensionPlanHash(value)) {
|
||||
errors.push("planHash_mismatch");
|
||||
}
|
||||
if (request !== undefined) comparePlanToRequest(value, request, errors);
|
||||
return result(errors);
|
||||
}
|
||||
|
||||
export function validateEnginePrivateExtensionApplyRequest(value, { plan, now = Date.now(), grantedCapabilities } = {}) {
|
||||
const errors = [];
|
||||
if (!isPlainObject(value)) return result(["enginePrivateExtensionApplyRequest_must_be_object"]);
|
||||
if (value.schemaVersion !== ENGINE_PRIVATE_EXTENSION_APPLY_REQUEST_SCHEMA_VERSION) errors.push("schemaVersion_mismatch");
|
||||
rejectUnknownKeys(value, APPLY_REQUEST_KEYS, "enginePrivateExtensionApplyRequest", errors);
|
||||
requiredOpaqueId(value.planId, "planId", errors);
|
||||
requiredHash(value.planHash, "planHash", errors);
|
||||
requiredOpaqueId(value.idempotencyKey, "idempotencyKey", errors);
|
||||
requiredTimestamp(value.confirmedAt, "confirmedAt", errors);
|
||||
addAuthorizationErrors("apply", grantedCapabilities, errors);
|
||||
const nowMs = normalizeNow(now, errors);
|
||||
if (isTimestamp(value.confirmedAt) && Date.parse(value.confirmedAt) > nowMs + MAX_CLOCK_SKEW_MS) {
|
||||
errors.push("confirmedAt_exceeds_clock_skew");
|
||||
}
|
||||
if (plan !== undefined) {
|
||||
const planValidation = validateEnginePrivateExtensionPlan(plan);
|
||||
if (!planValidation.ok) errors.push("plan_invalid_for_apply");
|
||||
if (value.planId !== plan?.planId) errors.push("planId_plan_mismatch");
|
||||
if (value.planHash !== plan?.planHash) errors.push("planHash_plan_mismatch");
|
||||
if (value.idempotencyKey !== plan?.idempotencyKey) errors.push("idempotencyKey_plan_mismatch");
|
||||
if (isTimestamp(plan?.expiresAt) && nowMs > Date.parse(plan.expiresAt)) errors.push("plan_expired");
|
||||
if (isTimestamp(value.confirmedAt) && isTimestamp(plan?.expiresAt)
|
||||
&& Date.parse(value.confirmedAt) > Date.parse(plan.expiresAt)) {
|
||||
errors.push("confirmedAt_after_plan_expiresAt");
|
||||
}
|
||||
if (isTimestamp(value.confirmedAt) && isTimestamp(plan?.createdAt)
|
||||
&& Date.parse(value.confirmedAt) < Date.parse(plan.createdAt)) {
|
||||
errors.push("confirmedAt_before_plan_createdAt");
|
||||
}
|
||||
}
|
||||
return result(errors);
|
||||
}
|
||||
|
||||
export function validateEnginePrivateExtensionApplyReceipt(value, { plan } = {}) {
|
||||
const errors = [];
|
||||
if (!isPlainObject(value)) return result(["enginePrivateExtensionApplyReceipt_must_be_object"]);
|
||||
if (value.schemaVersion !== ENGINE_PRIVATE_EXTENSION_APPLY_RECEIPT_SCHEMA_VERSION) errors.push("schemaVersion_mismatch");
|
||||
rejectUnknownKeys(value, APPLY_RECEIPT_KEYS, "enginePrivateExtensionApplyReceipt", errors);
|
||||
requiredOpaqueId(value.operationId, "operationId", errors);
|
||||
requiredOpaqueId(value.planId, "planId", errors);
|
||||
requiredHash(value.planHash, "planHash", errors);
|
||||
if (!new Set(["activate", "rollback"]).has(value.action)) errors.push("action_invalid");
|
||||
requiredTimestamp(value.acceptedAt, "acceptedAt", errors);
|
||||
if (value.state !== "queued") errors.push("apply_receipt_state_must_be_queued");
|
||||
if (plan !== undefined) {
|
||||
if (value.planId !== plan?.planId) errors.push("planId_plan_mismatch");
|
||||
if (value.planHash !== plan?.planHash) errors.push("planHash_plan_mismatch");
|
||||
if (value.action !== plan?.action) errors.push("action_plan_mismatch");
|
||||
if (isTimestamp(value.acceptedAt) && isTimestamp(plan?.expiresAt)
|
||||
&& Date.parse(value.acceptedAt) > Date.parse(plan.expiresAt)) {
|
||||
errors.push("acceptedAt_after_plan_expiresAt");
|
||||
}
|
||||
}
|
||||
return result(errors);
|
||||
}
|
||||
|
||||
export function validateEnginePrivateExtensionOperation(value) {
|
||||
const errors = [];
|
||||
if (!isPlainObject(value)) return result(["enginePrivateExtensionOperation_must_be_object"]);
|
||||
if (value.schemaVersion !== ENGINE_PRIVATE_EXTENSION_OPERATION_SCHEMA_VERSION) errors.push("schemaVersion_mismatch");
|
||||
rejectUnknownKeys(value, OPERATION_KEYS, "enginePrivateExtensionOperation", errors);
|
||||
requiredOpaqueId(value.operationId, "operationId", errors);
|
||||
requiredOpaqueId(value.planId, "planId", errors);
|
||||
requiredHash(value.planHash, "planHash", errors);
|
||||
if (!new Set(["activate", "rollback"]).has(value.action)) errors.push("action_invalid");
|
||||
if (!new Set([...NON_TERMINAL_STATES, "active", "rolled-back", "rejected", "quarantined"]).has(value.state)) {
|
||||
errors.push("state_invalid");
|
||||
}
|
||||
if (!new Set([
|
||||
"pending",
|
||||
"committed",
|
||||
"automatically-rolled-back",
|
||||
"explicitly-rolled-back",
|
||||
"rejected",
|
||||
"quarantined",
|
||||
]).has(value.outcome)) errors.push("outcome_invalid");
|
||||
if (!new Set([
|
||||
"queued",
|
||||
"prepare",
|
||||
"switch",
|
||||
"force-recreate",
|
||||
"acceptance",
|
||||
"rollback",
|
||||
"complete",
|
||||
]).has(value.phase)) errors.push("phase_invalid");
|
||||
requiredGeneration(value.expectedCurrentGeneration, "expectedCurrentGeneration", errors);
|
||||
requiredGeneration(value.nextGeneration, "nextGeneration", errors);
|
||||
if (Number.isInteger(value.expectedCurrentGeneration)
|
||||
&& value.nextGeneration !== value.expectedCurrentGeneration + 1) {
|
||||
errors.push("nextGeneration_must_increment_current_generation");
|
||||
}
|
||||
validateActivationState(value.targetState, "targetState", errors);
|
||||
validateActivationState(value.recoveryState, "recoveryState", errors);
|
||||
validateActivationState(value.effectiveState, "effectiveState", errors);
|
||||
if (sameValue(value.targetState, value.recoveryState)) errors.push("targetState_must_differ_from_recoveryState");
|
||||
validateRuntimeReport(value.runtime, errors);
|
||||
validateAcceptanceReport(value.acceptance, value.effectiveState, errors);
|
||||
requiredTimestamp(value.updatedAt, "updatedAt", errors);
|
||||
validateOperationOutcome(value, errors);
|
||||
return result(errors);
|
||||
}
|
||||
|
||||
export function validateEnginePrivateExtensionStatus(value) {
|
||||
const errors = [];
|
||||
if (!isPlainObject(value)) return result(["enginePrivateExtensionStatus_must_be_object"]);
|
||||
if (value.schemaVersion !== ENGINE_PRIVATE_EXTENSION_STATUS_SCHEMA_VERSION) errors.push("schemaVersion_mismatch");
|
||||
rejectUnknownKeys(value, STATUS_KEYS, "enginePrivateExtensionStatus", errors);
|
||||
if (value.packageName !== ENGINE_PRIVATE_EXTENSION_PACKAGE_NAME) errors.push("packageName_mismatch");
|
||||
requiredGeneration(value.generation, "generation", errors);
|
||||
if (!new Set(["ready", "transitioning", "quarantined"]).has(value.health)) errors.push("health_invalid");
|
||||
validateActivationState(value.currentState, "currentState", errors);
|
||||
validateActivationState(value.previousState, "previousState", errors);
|
||||
if (value.activeOperationId !== undefined) requiredOpaqueId(value.activeOperationId, "activeOperationId", errors);
|
||||
validateRuntimeReport(value.runtime, errors);
|
||||
validateAcceptanceReport(value.acceptance, value.currentState, errors);
|
||||
requiredTimestamp(value.updatedAt, "updatedAt", errors);
|
||||
if (value.runtime?.generation !== value.generation) errors.push("runtime_generation_mismatch");
|
||||
if (value.health === "ready") {
|
||||
if (value.activeOperationId !== undefined) errors.push("ready_status_must_not_have_active_operation");
|
||||
if (value.acceptance?.state !== "accepted") errors.push("ready_status_requires_acceptance");
|
||||
if (value.errorCode !== undefined) errors.push("ready_status_must_not_have_errorCode");
|
||||
} else if (value.health === "transitioning") {
|
||||
if (value.activeOperationId === undefined) errors.push("transitioning_status_requires_active_operation");
|
||||
} else {
|
||||
requiredReason(value.errorCode, "errorCode", errors);
|
||||
}
|
||||
return result(errors);
|
||||
}
|
||||
|
||||
function validatePlanRequestBody(value, now, errors) {
|
||||
if (!isPlainObject(value)) {
|
||||
errors.push("enginePrivateExtensionPlanRequest_must_be_object");
|
||||
return;
|
||||
}
|
||||
if (value.schemaVersion !== ENGINE_PRIVATE_EXTENSION_PLAN_REQUEST_SCHEMA_VERSION) errors.push("schemaVersion_mismatch");
|
||||
rejectUnknownKeys(value, PLAN_REQUEST_KEYS, "enginePrivateExtensionPlanRequest", errors);
|
||||
requiredOpaqueId(value.requestId, "requestId", errors);
|
||||
requiredOpaqueId(value.idempotencyKey, "idempotencyKey", errors);
|
||||
if (!new Set(["activate", "rollback"]).has(value.action)) errors.push("action_invalid");
|
||||
requiredTimestamp(value.requestedAt, "requestedAt", errors);
|
||||
requiredTimestamp(value.requestExpiresAt, "requestExpiresAt", errors);
|
||||
requiredGeneration(value.expectedCurrentGeneration, "expectedCurrentGeneration", errors);
|
||||
validateWindow(value.requestedAt, value.requestExpiresAt, normalizeNow(now, errors), errors, "request");
|
||||
validateRequestTarget(value.target, value.action, errors);
|
||||
}
|
||||
|
||||
function validateRequestTarget(value, action, errors) {
|
||||
if (!isPlainObject(value)) {
|
||||
errors.push("target_must_be_object");
|
||||
return;
|
||||
}
|
||||
if (action === "activate") {
|
||||
validateActivationState(value, "target", errors);
|
||||
if (value.kind !== "release") errors.push("activate_target_must_be_release");
|
||||
return;
|
||||
}
|
||||
rejectUnknownKeys(value, PREVIOUS_STATE_TARGET_KEYS, "target", errors);
|
||||
if (value.kind !== "previous-state") errors.push("rollback_target_must_be_previous-state");
|
||||
if (value.packageName !== ENGINE_PRIVATE_EXTENSION_PACKAGE_NAME) errors.push("target.packageName_mismatch");
|
||||
}
|
||||
|
||||
function validateActivationState(value, path, errors) {
|
||||
if (!isPlainObject(value)) {
|
||||
errors.push(path + "_must_be_object");
|
||||
return;
|
||||
}
|
||||
if (value.kind === "release") {
|
||||
rejectUnknownKeys(value, RELEASE_STATE_KEYS, path, errors);
|
||||
if (value.packageName !== ENGINE_PRIVATE_EXTENSION_PACKAGE_NAME) errors.push(path + ".packageName_mismatch");
|
||||
if (typeof value.releaseId !== "string" || !RELEASE_ID.test(value.releaseId)) {
|
||||
errors.push(path + ".releaseId_invalid");
|
||||
}
|
||||
if (typeof value.packageSha256 !== "string" || !SHA256.test(value.packageSha256)) {
|
||||
errors.push(path + ".packageSha256_invalid");
|
||||
} else if (typeof value.releaseId === "string"
|
||||
&& RELEASE_ID.test(value.releaseId)
|
||||
&& !value.releaseId.endsWith("-" + value.packageSha256.slice(0, 16))) {
|
||||
errors.push(path + ".releaseId_digest_mismatch");
|
||||
}
|
||||
return;
|
||||
}
|
||||
if (value.kind === "inactive-baseline") {
|
||||
rejectUnknownKeys(value, INACTIVE_STATE_KEYS, path, errors);
|
||||
if (value.packageName !== ENGINE_PRIVATE_EXTENSION_PACKAGE_NAME) errors.push(path + ".packageName_mismatch");
|
||||
if (value.baselineId !== ENGINE_PRIVATE_EXTENSION_INACTIVE_BASELINE) errors.push(path + ".baselineId_mismatch");
|
||||
return;
|
||||
}
|
||||
errors.push(path + ".kind_invalid");
|
||||
}
|
||||
|
||||
function validateTransition(value, errors) {
|
||||
if (!isPlainObject(value)) {
|
||||
errors.push("transition_must_be_object");
|
||||
return;
|
||||
}
|
||||
rejectUnknownKeys(value, TRANSITION_KEYS, "transition", errors);
|
||||
const expected = {
|
||||
mountMode: "read-only",
|
||||
loaderMode: "community-package",
|
||||
loaderPath: "/home/node/.n8n/nodes/node_modules/n8n-nodes-ndc",
|
||||
loaderEnvironment: {
|
||||
N8N_COMMUNITY_PACKAGES_ENABLED: "true",
|
||||
N8N_COMMUNITY_PACKAGES_PREVENT_LOADING: "false",
|
||||
N8N_REINSTALL_MISSING_PACKAGES: "false",
|
||||
},
|
||||
quiesceMode: "block-deploy-run-and-drain-queue",
|
||||
stateSwitch: "atomic-current-previous",
|
||||
runtimeAction: "force-recreate",
|
||||
scope: "main-workers-webhooks",
|
||||
requireUniformGeneration: true,
|
||||
hotReload: false,
|
||||
preserveCredentials: true,
|
||||
};
|
||||
if (isPlainObject(value.loaderEnvironment)) {
|
||||
rejectUnknownKeys(value.loaderEnvironment, LOADER_ENVIRONMENT_KEYS, "transition.loaderEnvironment", errors);
|
||||
}
|
||||
if (!sameValue(value, expected)) errors.push("transition_policy_mismatch");
|
||||
}
|
||||
|
||||
function validateAcceptanceSpec(value, targetState, errors) {
|
||||
if (!isPlainObject(value)) {
|
||||
errors.push("acceptance_must_be_object");
|
||||
return;
|
||||
}
|
||||
rejectUnknownKeys(value, ACCEPTANCE_SPEC_KEYS, "acceptance", errors);
|
||||
if (value.mode !== "exact") errors.push("acceptance.mode_must_be_exact");
|
||||
const expectedNodes = expectedTypes(targetState, ENGINE_PRIVATE_EXTENSION_NODE_TYPES);
|
||||
const expectedCredentials = expectedTypes(targetState, ENGINE_PRIVATE_EXTENSION_CREDENTIAL_TYPES);
|
||||
validateExactArray(value.nodeTypes, expectedNodes, "acceptance.nodeTypes", errors);
|
||||
validateExactArray(value.credentialSchemas, expectedCredentials, "acceptance.credentialSchemas", errors);
|
||||
if (value.requireUniformGeneration !== true) errors.push("acceptance.requireUniformGeneration_must_be_true");
|
||||
}
|
||||
|
||||
function validateFailurePolicy(value, errors) {
|
||||
if (!isPlainObject(value)) {
|
||||
errors.push("failurePolicy_must_be_object");
|
||||
return;
|
||||
}
|
||||
rejectUnknownKeys(value, FAILURE_POLICY_KEYS, "failurePolicy", errors);
|
||||
const expected = {
|
||||
mode: "automatic-rollback",
|
||||
rollbackFailureOutcome: "quarantined",
|
||||
preserveImmutableRelease: true,
|
||||
preserveCredentials: true,
|
||||
};
|
||||
if (!sameValue(value, expected)) errors.push("failurePolicy_mismatch");
|
||||
}
|
||||
|
||||
function validateRuntimeReport(value, errors) {
|
||||
if (!isPlainObject(value)) {
|
||||
errors.push("runtime_must_be_object");
|
||||
return;
|
||||
}
|
||||
rejectUnknownKeys(value, RUNTIME_KEYS, "runtime", errors);
|
||||
if (value.mode !== "force-recreate") errors.push("runtime.mode_must_be_force-recreate");
|
||||
requiredPositiveInteger(value.expectedInstances, "runtime.expectedInstances", errors);
|
||||
requiredNonnegativeInteger(value.readyInstances, "runtime.readyInstances", errors);
|
||||
if (Number.isInteger(value.expectedInstances) && Number.isInteger(value.readyInstances)
|
||||
&& value.readyInstances > value.expectedInstances) errors.push("runtime.readyInstances_exceeds_expectedInstances");
|
||||
requiredGeneration(value.generation, "runtime.generation", errors);
|
||||
}
|
||||
|
||||
function validateAcceptanceReport(value, effectiveState, errors) {
|
||||
if (!isPlainObject(value)) {
|
||||
errors.push("acceptance_must_be_object");
|
||||
return;
|
||||
}
|
||||
rejectUnknownKeys(value, ACCEPTANCE_REPORT_KEYS, "acceptance", errors);
|
||||
if (!new Set(["pending", "accepted", "rejected"]).has(value.state)) errors.push("acceptance.state_invalid");
|
||||
const expectedNodes = expectedTypes(effectiveState, ENGINE_PRIVATE_EXTENSION_NODE_TYPES);
|
||||
const expectedCredentials = expectedTypes(effectiveState, ENGINE_PRIVATE_EXTENSION_CREDENTIAL_TYPES);
|
||||
validateObservation(value.nodeTypes, expectedNodes, "acceptance.nodeTypes", value.state, errors);
|
||||
validateObservation(value.credentialSchemas, expectedCredentials, "acceptance.credentialSchemas", value.state, errors);
|
||||
if (typeof value.uniformGeneration !== "boolean") errors.push("acceptance.uniformGeneration_must_be_boolean");
|
||||
if (value.state === "accepted" && value.uniformGeneration !== true) {
|
||||
errors.push("accepted_runtime_requires_uniform_generation");
|
||||
}
|
||||
}
|
||||
|
||||
function validateObservation(value, expected, path, state, errors) {
|
||||
if (!isPlainObject(value)) {
|
||||
errors.push(path + "_must_be_object");
|
||||
return;
|
||||
}
|
||||
rejectUnknownKeys(value, OBSERVATION_KEYS, path, errors);
|
||||
validateExactArray(value.expected, expected, path + ".expected", errors);
|
||||
if (!Array.isArray(value.observed) || value.observed.some((item) => typeof item !== "string")) {
|
||||
errors.push(path + ".observed_must_be_string_array");
|
||||
return;
|
||||
}
|
||||
if (new Set(value.observed).size !== value.observed.length) errors.push(path + ".observed_must_be_unique");
|
||||
if (state === "accepted" && !sameValue(value.observed, expected)) errors.push(path + ".observed_exact_set_required");
|
||||
}
|
||||
|
||||
function validateOperationOutcome(value, errors) {
|
||||
if (NON_TERMINAL_STATES.has(value.state)) {
|
||||
if (value.outcome !== "pending") errors.push("nonterminal_operation_outcome_must_be_pending");
|
||||
return;
|
||||
}
|
||||
if (value.state === "rolled-back") {
|
||||
const expected = value.action === "rollback" ? "explicitly-rolled-back" : "automatically-rolled-back";
|
||||
if (value.outcome !== expected) errors.push("rolled_back_outcome_mismatch");
|
||||
} else if (TERMINAL_STATE_OUTCOMES[value.state] !== value.outcome) {
|
||||
errors.push("terminal_operation_outcome_mismatch");
|
||||
}
|
||||
if (value.state === "active" || value.state === "rolled-back") {
|
||||
if (value.acceptance?.state !== "accepted") errors.push("successful_terminal_state_requires_acceptance");
|
||||
if (value.runtime?.readyInstances !== value.runtime?.expectedInstances) {
|
||||
errors.push("successful_terminal_state_requires_all_instances_ready");
|
||||
}
|
||||
if (value.runtime?.generation !== value.nextGeneration) {
|
||||
errors.push("successful_terminal_state_runtime_generation_mismatch");
|
||||
}
|
||||
if (value.errorCode !== undefined && value.state === "active") errors.push("active_state_must_not_have_errorCode");
|
||||
if (value.state === "active" && value.action !== "activate") errors.push("active_state_action_mismatch");
|
||||
const expectedEffective = value.state === "active"
|
||||
? value.targetState
|
||||
: value.action === "rollback" ? value.targetState : value.recoveryState;
|
||||
if (!sameValue(value.effectiveState, expectedEffective)) errors.push("effectiveState_terminal_mismatch");
|
||||
if (value.state === "rolled-back" && value.action === "activate") {
|
||||
requiredReason(value.errorCode, "errorCode", errors);
|
||||
}
|
||||
if (value.state === "rolled-back" && value.action === "rollback" && value.errorCode !== undefined) {
|
||||
errors.push("explicit_rollback_must_not_have_errorCode");
|
||||
}
|
||||
} else if (value.state === "rejected" || value.state === "quarantined") {
|
||||
requiredReason(value.errorCode, "errorCode", errors);
|
||||
}
|
||||
}
|
||||
|
||||
function comparePlanToRequest(plan, request, errors) {
|
||||
const requestErrors = [];
|
||||
validatePlanRequestBody(request, plan.createdAt, requestErrors);
|
||||
if (requestErrors.length) errors.push("request_invalid_for_plan_comparison");
|
||||
if (plan.requestId !== request?.requestId) errors.push("requestId_request_mismatch");
|
||||
if (plan.idempotencyKey !== request?.idempotencyKey) errors.push("idempotencyKey_request_mismatch");
|
||||
if (plan.action !== request?.action) errors.push("action_request_mismatch");
|
||||
if (plan.expectedCurrentGeneration !== request?.expectedCurrentGeneration) {
|
||||
errors.push("expectedCurrentGeneration_request_mismatch");
|
||||
}
|
||||
if (request?.action === "activate" && !sameValue(plan.targetState, request?.target)) {
|
||||
errors.push("targetState_request_mismatch");
|
||||
}
|
||||
if (isTimestamp(request?.requestExpiresAt) && isTimestamp(plan.expiresAt)
|
||||
&& Date.parse(plan.expiresAt) > Date.parse(request.requestExpiresAt)) {
|
||||
errors.push("plan_expiresAt_exceeds_request");
|
||||
}
|
||||
}
|
||||
|
||||
function addAuthorizationErrors(operation, grantedCapabilities, errors) {
|
||||
errors.push(...authorizeEnginePrivateExtensionOperation(operation, grantedCapabilities).errors);
|
||||
}
|
||||
|
||||
function validateWindow(start, end, nowMs, errors, label) {
|
||||
if (!isTimestamp(start) || !isTimestamp(end) || !Number.isFinite(nowMs)) return;
|
||||
const startMs = Date.parse(start);
|
||||
const endMs = Date.parse(end);
|
||||
if (endMs <= startMs) errors.push(label + "_expiresAt_must_be_after_start");
|
||||
if (endMs - startMs > MAX_REQUEST_LIFETIME_MS) errors.push(label + "_lifetime_exceeds_15_minutes");
|
||||
if (startMs > nowMs + MAX_CLOCK_SKEW_MS) errors.push(label + "_start_exceeds_clock_skew");
|
||||
if (endMs < nowMs) errors.push(label + "_expired");
|
||||
}
|
||||
|
||||
function normalizeNow(value, errors) {
|
||||
const candidate = value === undefined ? Date.now() : value;
|
||||
const milliseconds = typeof candidate === "number" ? candidate : Date.parse(candidate);
|
||||
if (!Number.isFinite(milliseconds)) {
|
||||
errors.push("now_invalid");
|
||||
return Number.NaN;
|
||||
}
|
||||
return milliseconds;
|
||||
}
|
||||
|
||||
function requiredGeneration(value, path, errors) {
|
||||
requiredNonnegativeInteger(value, path, errors);
|
||||
}
|
||||
|
||||
function requiredPositiveInteger(value, path, errors) {
|
||||
if (!Number.isInteger(value) || value < 1) errors.push(path + "_must_be_positive_integer");
|
||||
}
|
||||
|
||||
function requiredNonnegativeInteger(value, path, errors) {
|
||||
if (!Number.isInteger(value) || value < 0) errors.push(path + "_must_be_nonnegative_integer");
|
||||
}
|
||||
|
||||
function requiredOpaqueId(value, path, errors) {
|
||||
if (typeof value !== "string" || !OPAQUE_ID.test(value)) errors.push(path + "_invalid");
|
||||
}
|
||||
|
||||
function requiredReason(value, path, errors) {
|
||||
if (typeof value !== "string" || !REASON_CODE.test(value)) errors.push(path + "_invalid");
|
||||
}
|
||||
|
||||
function requiredHash(value, path, errors) {
|
||||
if (typeof value !== "string" || !HASH.test(value)) errors.push(path + "_invalid");
|
||||
}
|
||||
|
||||
function requiredTimestamp(value, path, errors) {
|
||||
if (!isTimestamp(value)) errors.push(path + "_invalid_timestamp");
|
||||
}
|
||||
|
||||
function isTimestamp(value) {
|
||||
return typeof value === "string" && Number.isFinite(Date.parse(value));
|
||||
}
|
||||
|
||||
function expectedTypes(state, releaseTypes) {
|
||||
return state?.kind === "release" ? [...releaseTypes] : [];
|
||||
}
|
||||
|
||||
function validateExactArray(actual, expected, path, errors) {
|
||||
if (!Array.isArray(actual)) {
|
||||
errors.push(path + "_must_be_array");
|
||||
return;
|
||||
}
|
||||
if (!sameValue(actual, [...expected])) errors.push(path + "_mismatch");
|
||||
}
|
||||
|
||||
function rejectUnknownKeys(value, allowedKeys, path, errors) {
|
||||
if (!isPlainObject(value)) return;
|
||||
for (const key of Object.keys(value)) {
|
||||
if (!allowedKeys.has(key)) errors.push(path + "." + key + "_not_allowed");
|
||||
}
|
||||
}
|
||||
|
||||
function isPlainObject(value) {
|
||||
return Boolean(value) && typeof value === "object" && !Array.isArray(value);
|
||||
}
|
||||
|
||||
function sameValue(left, right) {
|
||||
return stableJson(left) === stableJson(right);
|
||||
}
|
||||
|
||||
function stableJson(value) {
|
||||
if (Array.isArray(value)) return "[" + value.map(stableJson).join(",") + "]";
|
||||
if (isPlainObject(value)) {
|
||||
return "{" + Object.keys(value).sort().map((key) => JSON.stringify(key) + ":" + stableJson(value[key])).join(",") + "}";
|
||||
}
|
||||
return JSON.stringify(value);
|
||||
}
|
||||
|
||||
function result(errors) {
|
||||
const unique = [...new Set(errors)];
|
||||
return Object.freeze({ ok: unique.length === 0, errors: Object.freeze(unique) });
|
||||
}
|
||||
|
|
@ -1,481 +0,0 @@
|
|||
export const EXTERNAL_PROVIDER_CONTRACT_VERSION = "nodedc.external-provider-contract/v1";
|
||||
export const FOUNDRY_BINDING_UPSERT_SCHEMA_VERSION = "nodedc.foundry.binding-upsert/v1";
|
||||
|
||||
export {
|
||||
DATA_PRODUCT_PATCH_SCHEMA_VERSION,
|
||||
DATA_PRODUCT_PUBLISH_SCHEMA_VERSION,
|
||||
DATA_PRODUCT_SNAPSHOT_SCHEMA_VERSION,
|
||||
validateDataProductPatch,
|
||||
validateDataProductPublish,
|
||||
validateDataProductSnapshot,
|
||||
} from "./data-product.mjs";
|
||||
|
||||
export {
|
||||
ENGINE_CREDENTIAL_SINK_AUDIT_SCHEMA_VERSION,
|
||||
ENGINE_CREDENTIAL_SINK_CAPABILITY_TYPES,
|
||||
ENGINE_CREDENTIAL_SINK_PROVISION_SCHEMA_VERSION,
|
||||
ENGINE_CREDENTIAL_SINK_RECEIPT_SCHEMA_VERSION,
|
||||
ENGINE_CREDENTIAL_SINK_ROLLBACK_RECEIPT_SCHEMA_VERSION,
|
||||
ENGINE_CREDENTIAL_SINK_ROLLBACK_SCHEMA_VERSION,
|
||||
computeEngineCredentialCapabilityDigest,
|
||||
computeEngineCredentialSinkPolicyHash,
|
||||
computeEngineCredentialSinkReceiptHash,
|
||||
engineCredentialSinkAuditTargets,
|
||||
validateEngineCredentialSinkAudit,
|
||||
validateEngineCredentialSinkProvision,
|
||||
validateEngineCredentialSinkReceipt,
|
||||
validateEngineCredentialSinkRollback,
|
||||
validateEngineCredentialSinkRollbackReceipt,
|
||||
} from "./engine-credential-sink.mjs";
|
||||
|
||||
export {
|
||||
ENGINE_PRIVATE_EXTENSION_APPLY_RECEIPT_SCHEMA_VERSION,
|
||||
ENGINE_PRIVATE_EXTENSION_APPLY_REQUEST_SCHEMA_VERSION,
|
||||
ENGINE_PRIVATE_EXTENSION_CREDENTIAL_SCHEMAS,
|
||||
ENGINE_PRIVATE_EXTENSION_CREDENTIAL_TYPES,
|
||||
ENGINE_PRIVATE_EXTENSION_INACTIVE_BASELINE,
|
||||
ENGINE_PRIVATE_EXTENSION_MANAGE_CAPABILITY,
|
||||
ENGINE_PRIVATE_EXTENSION_NODE_TYPES,
|
||||
ENGINE_PRIVATE_EXTENSION_OPERATION_SCHEMA_VERSION,
|
||||
ENGINE_PRIVATE_EXTENSION_PACKAGE_NAME,
|
||||
ENGINE_PRIVATE_EXTENSION_PLAN_REQUEST_SCHEMA_VERSION,
|
||||
ENGINE_PRIVATE_EXTENSION_PLAN_SCHEMA_VERSION,
|
||||
ENGINE_PRIVATE_EXTENSION_READ_CAPABILITY,
|
||||
ENGINE_PRIVATE_EXTENSION_STATUS_SCHEMA_VERSION,
|
||||
authorizeEnginePrivateExtensionOperation,
|
||||
computeEnginePrivateExtensionPlanHash,
|
||||
validateEnginePrivateExtensionApplyReceipt,
|
||||
validateEnginePrivateExtensionApplyRequest,
|
||||
validateEnginePrivateExtensionOperation,
|
||||
validateEnginePrivateExtensionPlan,
|
||||
validateEnginePrivateExtensionPlanRequest,
|
||||
validateEnginePrivateExtensionStatus,
|
||||
} from "./engine-private-extension.mjs";
|
||||
|
||||
const COLLECTION_MODES = new Set(["realtime", "manual", "weekly", "history"]);
|
||||
const DELIVERY_MODES = new Set(["snapshot", "snapshot+patch", "query"]);
|
||||
const CAPABILITY_CLASSIFICATIONS = new Set(["read", "metadata", "write", "destructive", "unknown"]);
|
||||
const SECRET_LIKE_KEY = /(token|secret|password|authorization|access[_-]?token|refresh[_-]?token|api[_-]?key)/i;
|
||||
const SECRET_LIKE_REFERENCE = /(?:[?&](?:token|secret|password|authorization|access[_-]?token|refresh[_-]?token|api[_-]?key)=|(?:bearer|basic)\s+)/i;
|
||||
const SECRET_LIKE_VALUE = /(?:ndc_edp(?:wb|rb)_[A-Za-z0-9_-]*|[?&](?:token|secret|password|authorization|access[_-]?token|refresh[_-]?token|api[_-]?key)=|(?:bearer|basic)\s+\S+|eyJ[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+)/i;
|
||||
const IDENTIFIER = /^[a-z][a-z0-9._:-]{2,127}$/;
|
||||
const FOUNDRY_APPLICATION_ID = /^[0-9a-f]{8}-[0-9a-f]{4}-[1-5][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/i;
|
||||
const FOUNDRY_PAGE_ID = /^[a-z0-9][a-z0-9-]{0,79}$/;
|
||||
const FOUNDRY_SLOT_ID = /^[A-Za-z0-9][A-Za-z0-9-]{0,79}$/;
|
||||
const CONTRACT_VERSION = /^\d+\.\d+\.\d+(?:[-+][a-z0-9.-]+)?$/i;
|
||||
const MAX_BATCH_SEQUENCE = 2_147_483_647;
|
||||
const MAX_FACT_ATTRIBUTES_BYTES = 64 * 1024;
|
||||
|
||||
const PROVIDER_MANIFEST_KEYS = new Set(["schemaVersion", "id", "providerId", "version", "ontology", "l2Template", "capabilities", "dataProductIds"]);
|
||||
const CONNECTION_PROFILE_KEYS = new Set(["schemaVersion", "id", "providerId", "tenantId", "credentialRef", "scope"]);
|
||||
const COLLECTION_PROFILE_KEYS = new Set(["schemaVersion", "id", "connectionId", "dataProductId", "mode", "schedule", "capabilityIds"]);
|
||||
const DATA_PRODUCT_KEYS = new Set(["schemaVersion", "id", "version", "delivery", "semanticTypes", "fields", "access"]);
|
||||
const FOUNDRY_BINDING_KEYS = new Set(["schemaVersion", "id", "dataProductId", "applicationId", "pageId", "templateId", "slotId", "semanticType"]);
|
||||
const FOUNDRY_BINDING_UPSERT_KEYS = new Set(["schemaVersion", "applicationId", "pageId", "idempotencyKey", "binding"]);
|
||||
const FOUNDRY_BINDING_UPSERT_BINDING_KEYS = new Set(["id", "dataProductId", "slotId", "semanticTypes", "fieldProjection"]);
|
||||
|
||||
/**
|
||||
* Provider-neutral, versioned description of an L2 connector template.
|
||||
*
|
||||
* This is a declarative package artifact, not a tenant connection or an
|
||||
* executable adapter. It may catalogue write capabilities, but it never
|
||||
* grants or transports them. Runtime API requests, secrets and connection
|
||||
* scope remain outside this manifest.
|
||||
*/
|
||||
export function validateProviderManifest(value) {
|
||||
const errors = baseErrors(value, "providerManifest");
|
||||
rejectUnknownKeys(value, PROVIDER_MANIFEST_KEYS, "providerManifest", errors);
|
||||
requiredIdentifier(value?.id, "id", errors);
|
||||
requiredIdentifier(value?.providerId, "providerId", errors);
|
||||
requiredString(value?.version, "version", errors);
|
||||
if (value?.version && !CONTRACT_VERSION.test(value.version)) {
|
||||
errors.push("version_must_be_semver");
|
||||
}
|
||||
|
||||
if (!isPlainObject(value?.ontology)) {
|
||||
errors.push("ontology_must_be_object");
|
||||
} else {
|
||||
rejectUnknownKeys(value.ontology, new Set(["packageId", "revision"]), "ontology", errors);
|
||||
}
|
||||
requiredIdentifier(value?.ontology?.packageId, "ontology.packageId", errors);
|
||||
requiredIdentifier(value?.ontology?.revision, "ontology.revision", errors);
|
||||
if (!isPlainObject(value?.l2Template)) {
|
||||
errors.push("l2Template_must_be_object");
|
||||
} else {
|
||||
rejectUnknownKeys(value.l2Template, new Set(["id", "version"]), "l2Template", errors);
|
||||
}
|
||||
requiredIdentifier(value?.l2Template?.id, "l2Template.id", errors);
|
||||
requiredString(value?.l2Template?.version, "l2Template.version", errors);
|
||||
if (value?.l2Template?.version && !CONTRACT_VERSION.test(value.l2Template.version)) {
|
||||
errors.push("l2Template.version_must_be_semver");
|
||||
}
|
||||
|
||||
if (!Array.isArray(value?.capabilities) || value.capabilities.length === 0) {
|
||||
errors.push("capabilities_must_be_nonempty_array");
|
||||
} else {
|
||||
value.capabilities.forEach((capability, index) => {
|
||||
if (!isPlainObject(capability)) {
|
||||
errors.push(`capabilities[${index}]_must_be_object`);
|
||||
return;
|
||||
}
|
||||
rejectUnknownKeys(capability, new Set(["id", "classification"]), `capabilities[${index}]`, errors);
|
||||
requiredIdentifier(capability?.id, `capabilities[${index}].id`, errors);
|
||||
if (!CAPABILITY_CLASSIFICATIONS.has(capability?.classification)) {
|
||||
errors.push(`capabilities[${index}].classification_invalid`);
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
if (!Array.isArray(value?.dataProductIds) || value.dataProductIds.length === 0) {
|
||||
errors.push("dataProductIds_must_be_nonempty_array");
|
||||
} else {
|
||||
value.dataProductIds.forEach((dataProductId, index) => {
|
||||
requiredIdentifier(dataProductId, `dataProductIds[${index}]`, errors);
|
||||
});
|
||||
}
|
||||
|
||||
if (value?.tenantId !== undefined || value?.connectionId !== undefined || value?.credentialRef !== undefined) {
|
||||
errors.push("manifest_must_not_contain_connection_runtime_state");
|
||||
}
|
||||
if (value?.endpoint !== undefined || value?.url !== undefined || value?.host !== undefined) {
|
||||
errors.push("manifest_must_not_contain_provider_transport");
|
||||
}
|
||||
if (containsSecretLikeMaterial(value)) errors.push("manifest_must_not_contain_secret_material");
|
||||
|
||||
return result(errors);
|
||||
}
|
||||
|
||||
export function validateConnectionProfile(value) {
|
||||
const errors = baseErrors(value, "connection");
|
||||
rejectUnknownKeys(value, CONNECTION_PROFILE_KEYS, "connection", errors);
|
||||
requiredIdentifier(value?.id, "id", errors);
|
||||
requiredIdentifier(value?.providerId, "providerId", errors);
|
||||
requiredIdentifier(value?.tenantId, "tenantId", errors);
|
||||
if (!isPlainObject(value?.credentialRef)) {
|
||||
errors.push("credentialRef_must_be_object");
|
||||
} else {
|
||||
rejectUnknownKeys(value.credentialRef, new Set(["owner", "reference"]), "credentialRef", errors);
|
||||
}
|
||||
requiredString(value?.credentialRef?.owner, "credentialRef.owner", errors);
|
||||
requiredString(value?.credentialRef?.reference, "credentialRef.reference", errors);
|
||||
|
||||
if (value?.credentialRef?.owner && value.credentialRef.owner !== "engine") {
|
||||
errors.push("credentialRef.owner_must_be_engine");
|
||||
}
|
||||
if (containsSecretLikeMaterial(value)) errors.push("profile_must_not_contain_secret_material");
|
||||
if (value?.scope !== undefined) {
|
||||
if (!isPlainObject(value.scope)) {
|
||||
errors.push("scope_must_be_object");
|
||||
} else {
|
||||
rejectUnknownKeys(value.scope, new Set(["capabilityIds", "fieldPolicyId", "retentionPolicyId", "collectionProfileIds"]), "scope", errors);
|
||||
validateOptionalIdentifierArray(value.scope.capabilityIds, "scope.capabilityIds", errors);
|
||||
validateOptionalIdentifierArray(value.scope.collectionProfileIds, "scope.collectionProfileIds", errors);
|
||||
if (value.scope.fieldPolicyId !== undefined) requiredIdentifier(value.scope.fieldPolicyId, "scope.fieldPolicyId", errors);
|
||||
if (value.scope.retentionPolicyId !== undefined) requiredIdentifier(value.scope.retentionPolicyId, "scope.retentionPolicyId", errors);
|
||||
}
|
||||
}
|
||||
|
||||
return result(errors);
|
||||
}
|
||||
|
||||
export function validateCollectionProfile(value) {
|
||||
const errors = baseErrors(value, "collectionProfile");
|
||||
rejectUnknownKeys(value, COLLECTION_PROFILE_KEYS, "collectionProfile", errors);
|
||||
requiredIdentifier(value?.id, "id", errors);
|
||||
requiredIdentifier(value?.connectionId, "connectionId", errors);
|
||||
requiredIdentifier(value?.dataProductId, "dataProductId", errors);
|
||||
|
||||
if (!COLLECTION_MODES.has(value?.mode)) errors.push("mode_must_be_realtime_manual_weekly_or_history");
|
||||
if (!Array.isArray(value?.capabilityIds) || value.capabilityIds.length === 0) {
|
||||
errors.push("capabilityIds_must_be_nonempty_array");
|
||||
} else {
|
||||
value.capabilityIds.forEach((capabilityId, index) => requiredIdentifier(capabilityId, `capabilityIds[${index}]`, errors));
|
||||
}
|
||||
|
||||
const intervalMs = value?.schedule?.intervalMs;
|
||||
if (value?.schedule !== undefined) {
|
||||
if (!isPlainObject(value.schedule)) {
|
||||
errors.push("schedule_must_be_object");
|
||||
} else {
|
||||
rejectUnknownKeys(value.schedule, new Set(["intervalMs"]), "schedule", errors);
|
||||
}
|
||||
}
|
||||
if (value?.mode === "realtime") {
|
||||
if (!Number.isInteger(intervalMs) || intervalMs < 1000) errors.push("realtime_schedule_intervalMs_must_be_integer_gte_1000");
|
||||
} else if (value?.mode === "manual") {
|
||||
if (intervalMs !== undefined) errors.push("manual_profile_must_not_define_intervalMs");
|
||||
}
|
||||
|
||||
if (containsSecretLikeMaterial(value)) errors.push("collectionProfile_must_not_contain_secret_material");
|
||||
|
||||
return result(errors);
|
||||
}
|
||||
|
||||
export function validateDataProduct(value) {
|
||||
const errors = baseErrors(value, "dataProduct");
|
||||
rejectUnknownKeys(value, DATA_PRODUCT_KEYS, "dataProduct", errors);
|
||||
requiredIdentifier(value?.id, "id", errors);
|
||||
requiredString(value?.version, "version", errors);
|
||||
|
||||
if (!isPlainObject(value?.delivery)) {
|
||||
errors.push("delivery_must_be_object");
|
||||
} else {
|
||||
rejectUnknownKeys(value.delivery, new Set(["mode"]), "delivery", errors);
|
||||
}
|
||||
if (!DELIVERY_MODES.has(value?.delivery?.mode)) errors.push("delivery.mode_must_be_snapshot_snapshot+patch_or_query");
|
||||
if (!Array.isArray(value?.semanticTypes) || value.semanticTypes.length === 0) {
|
||||
errors.push("semanticTypes_must_be_nonempty_array");
|
||||
} else {
|
||||
value.semanticTypes.forEach((semanticType, index) => requiredIdentifier(semanticType, `semanticTypes[${index}]`, errors));
|
||||
}
|
||||
if (!Array.isArray(value?.fields) || value.fields.length === 0) {
|
||||
errors.push("fields_must_be_nonempty_array");
|
||||
} else {
|
||||
value.fields.forEach((field, index) => requiredIdentifier(field, `fields[${index}]`, errors));
|
||||
}
|
||||
if (!isPlainObject(value?.access)) {
|
||||
errors.push("access_must_be_object");
|
||||
} else {
|
||||
rejectUnknownKeys(value.access, new Set(["audience"]), "access", errors);
|
||||
}
|
||||
if (value?.access?.audience !== "internal") errors.push("access.audience_must_be_internal");
|
||||
if (containsSecretLikeMaterial(value)) errors.push("dataProduct_must_not_contain_secret_material");
|
||||
|
||||
return result(errors);
|
||||
}
|
||||
|
||||
export function validateFoundryBinding(value) {
|
||||
const errors = baseErrors(value, "foundryBinding");
|
||||
rejectUnknownKeys(value, FOUNDRY_BINDING_KEYS, "foundryBinding", errors);
|
||||
requiredIdentifier(value?.id, "id", errors);
|
||||
requiredIdentifier(value?.dataProductId, "dataProductId", errors);
|
||||
if (typeof value?.applicationId !== "string" || !FOUNDRY_APPLICATION_ID.test(value.applicationId)) {
|
||||
errors.push("applicationId_invalid");
|
||||
}
|
||||
if (typeof value?.pageId !== "string" || !FOUNDRY_PAGE_ID.test(value.pageId)) errors.push("pageId_invalid");
|
||||
if (value?.templateId !== undefined) requiredIdentifier(value.templateId, "templateId", errors);
|
||||
if (typeof value?.slotId !== "string" || !FOUNDRY_SLOT_ID.test(value.slotId)) errors.push("slotId_invalid");
|
||||
requiredIdentifier(value?.semanticType, "semanticType", errors);
|
||||
|
||||
if (containsSecretLikeMaterial(value)) errors.push("binding_must_not_contain_secret_material");
|
||||
if (value?.providerId !== undefined || value?.credentialRef !== undefined || value?.endpoint !== undefined) {
|
||||
errors.push("binding_must_reference_data_product_not_provider_transport");
|
||||
}
|
||||
|
||||
return result(errors);
|
||||
}
|
||||
|
||||
/**
|
||||
* Replay-safe control-plane command emitted by `NDC Foundry Binding`.
|
||||
*
|
||||
* This is deliberately separate from the declarative Foundry binding artifact
|
||||
* above: the command carries an idempotency key and can express a safe
|
||||
* semantic/field projection, while authorization is materialized exclusively
|
||||
* from the opaque workload grant at the receiving service.
|
||||
*/
|
||||
export function validateFoundryBindingUpsert(value) {
|
||||
const errors = [];
|
||||
if (!isPlainObject(value)) return result(["foundryBindingUpsert_must_be_object"]);
|
||||
if (value.schemaVersion !== FOUNDRY_BINDING_UPSERT_SCHEMA_VERSION) errors.push("schemaVersion_mismatch");
|
||||
rejectUnknownKeys(value, FOUNDRY_BINDING_UPSERT_KEYS, "foundryBindingUpsert", errors);
|
||||
if (typeof value.applicationId !== "string" || !FOUNDRY_APPLICATION_ID.test(value.applicationId)) {
|
||||
errors.push("applicationId_invalid");
|
||||
}
|
||||
if (typeof value.pageId !== "string" || !FOUNDRY_PAGE_ID.test(value.pageId)) errors.push("pageId_invalid");
|
||||
requiredIdentifier(value.idempotencyKey, "idempotencyKey", errors);
|
||||
|
||||
if (!isPlainObject(value.binding)) {
|
||||
errors.push("binding_must_be_object");
|
||||
} else {
|
||||
rejectUnknownKeys(value.binding, FOUNDRY_BINDING_UPSERT_BINDING_KEYS, "binding", errors);
|
||||
requiredIdentifier(value.binding.id, "binding.id", errors);
|
||||
requiredIdentifier(value.binding.dataProductId, "binding.dataProductId", errors);
|
||||
if (typeof value.binding.slotId !== "string" || !FOUNDRY_SLOT_ID.test(value.binding.slotId)) {
|
||||
errors.push("binding.slotId_invalid");
|
||||
}
|
||||
validateRequiredUniqueIdentifierArray(value.binding.semanticTypes, "binding.semanticTypes", errors);
|
||||
validateUniqueIdentifierArray(value.binding.fieldProjection, "binding.fieldProjection", errors);
|
||||
}
|
||||
|
||||
if (containsSecretLikeMaterial(value)) errors.push("binding_must_not_contain_secret_material");
|
||||
return result(errors);
|
||||
}
|
||||
|
||||
/**
|
||||
* Provider-neutral batch written by an L2 connector to External Data Plane.
|
||||
* The payload deliberately describes source facts rather than any provider
|
||||
* field names, customer entities or renderer representation.
|
||||
*/
|
||||
export function validateIntakeBatch(value) {
|
||||
const errors = baseErrors(value, "intakeBatch");
|
||||
rejectUnknownKeys(value, new Set(["schemaVersion", "source", "contract", "batch", "raw", "facts"]), "intakeBatch", errors);
|
||||
rejectUnknownKeys(value?.source, new Set(["providerId", "tenantId", "connectionId"]), "source", errors);
|
||||
rejectUnknownKeys(value?.contract, new Set(["dataProductId", "ontologyRevision", "version"]), "contract", errors);
|
||||
rejectUnknownKeys(value?.batch, new Set(["runId", "sequence", "idempotencyKey", "receivedAt"]), "batch", errors);
|
||||
requiredIdentifier(value?.source?.providerId, "source.providerId", errors);
|
||||
requiredIdentifier(value?.source?.tenantId, "source.tenantId", errors);
|
||||
requiredIdentifier(value?.source?.connectionId, "source.connectionId", errors);
|
||||
requiredIdentifier(value?.contract?.dataProductId, "contract.dataProductId", errors);
|
||||
requiredIdentifier(value?.contract?.ontologyRevision, "contract.ontologyRevision", errors);
|
||||
requiredString(value?.contract?.version, "contract.version", errors);
|
||||
if (value?.contract?.version && !CONTRACT_VERSION.test(value.contract.version)) {
|
||||
errors.push("contract.version_must_be_semver");
|
||||
}
|
||||
requiredIdentifier(value?.batch?.runId, "batch.runId", errors);
|
||||
requiredIdentifier(value?.batch?.idempotencyKey, "batch.idempotencyKey", errors);
|
||||
if (!Number.isInteger(value?.batch?.sequence) || value.batch.sequence < 0 || value.batch.sequence > MAX_BATCH_SEQUENCE) {
|
||||
errors.push("batch.sequence_must_be_integer_0_to_2147483647");
|
||||
}
|
||||
requiredIsoTimestamp(value?.batch?.receivedAt, "batch.receivedAt", errors);
|
||||
|
||||
if (!Array.isArray(value?.facts) || value.facts.length === 0) {
|
||||
errors.push("facts_must_be_nonempty_array");
|
||||
} else {
|
||||
value.facts.forEach((fact, index) => validateFact(fact, `facts[${index}]`, errors));
|
||||
}
|
||||
|
||||
if (value?.raw !== undefined) validateRawEnvelope(value.raw, errors);
|
||||
if (containsSecretLikeMaterial(value)) errors.push("intake_must_not_contain_secret_material");
|
||||
|
||||
return result(errors);
|
||||
}
|
||||
|
||||
export function assertValid(validator, value) {
|
||||
const validation = validator(value);
|
||||
if (!validation.ok) throw new Error(`external_provider_contract_invalid:${validation.errors.join(",")}`);
|
||||
return value;
|
||||
}
|
||||
|
||||
function baseErrors(value, label) {
|
||||
const errors = [];
|
||||
if (!isPlainObject(value)) return [`${label}_must_be_object`];
|
||||
if (value.schemaVersion !== EXTERNAL_PROVIDER_CONTRACT_VERSION) {
|
||||
errors.push("schemaVersion_mismatch");
|
||||
}
|
||||
return errors;
|
||||
}
|
||||
|
||||
function result(errors) {
|
||||
const uniqueErrors = [...new Set(errors)];
|
||||
return Object.freeze({ ok: uniqueErrors.length === 0, errors: Object.freeze(uniqueErrors) });
|
||||
}
|
||||
|
||||
function requiredString(value, path, errors) {
|
||||
if (typeof value !== "string" || !value.trim()) errors.push(`${path}_required`);
|
||||
}
|
||||
|
||||
function requiredIdentifier(value, path, errors) {
|
||||
if (typeof value !== "string" || !IDENTIFIER.test(value)) errors.push(`${path}_invalid`);
|
||||
}
|
||||
|
||||
function validateOptionalIdentifierArray(value, path, errors) {
|
||||
if (value === undefined) return;
|
||||
if (!Array.isArray(value)) {
|
||||
errors.push(`${path}_must_be_array`);
|
||||
return;
|
||||
}
|
||||
value.forEach((item, index) => requiredIdentifier(item, `${path}[${index}]`, errors));
|
||||
}
|
||||
|
||||
function validateRequiredUniqueIdentifierArray(value, path, errors) {
|
||||
if (!Array.isArray(value) || value.length === 0) {
|
||||
errors.push(`${path}_must_be_nonempty_array`);
|
||||
return;
|
||||
}
|
||||
validateUniqueIdentifierArray(value, path, errors);
|
||||
}
|
||||
|
||||
function validateUniqueIdentifierArray(value, path, errors) {
|
||||
if (!Array.isArray(value)) {
|
||||
errors.push(`${path}_must_be_array`);
|
||||
return;
|
||||
}
|
||||
value.forEach((item, index) => requiredIdentifier(item, `${path}[${index}]`, errors));
|
||||
if (new Set(value).size !== value.length) errors.push(`${path}_must_not_contain_duplicates`);
|
||||
}
|
||||
|
||||
function requiredIsoTimestamp(value, path, errors) {
|
||||
if (typeof value !== "string" || Number.isNaN(Date.parse(value))) errors.push(`${path}_invalid_timestamp`);
|
||||
}
|
||||
|
||||
function validateFact(value, path, errors) {
|
||||
if (!isPlainObject(value)) {
|
||||
errors.push(`${path}_must_be_object`);
|
||||
return;
|
||||
}
|
||||
rejectUnknownKeys(value, new Set(["sourceId", "semanticType", "observedAt", "attributes", "geometry"]), path, errors);
|
||||
requiredIdentifier(value.sourceId, `${path}.sourceId`, errors);
|
||||
requiredIdentifier(value.semanticType, `${path}.semanticType`, errors);
|
||||
requiredIsoTimestamp(value.observedAt, `${path}.observedAt`, errors);
|
||||
if (value.attributes !== undefined) {
|
||||
if (!isPlainObject(value.attributes)) {
|
||||
errors.push(`${path}.attributes_must_be_object`);
|
||||
} else if (serializedByteLength(value.attributes) > MAX_FACT_ATTRIBUTES_BYTES) {
|
||||
errors.push(`${path}.attributes_size_exceeded`);
|
||||
}
|
||||
}
|
||||
if (value.geometry !== undefined) validatePointGeometry(value.geometry, `${path}.geometry`, errors);
|
||||
}
|
||||
|
||||
function validatePointGeometry(value, path, errors) {
|
||||
if (!isPlainObject(value) || value.type !== "Point" || !Array.isArray(value.coordinates) || value.coordinates.length !== 2) {
|
||||
errors.push(`${path}_must_be_geojson_point`);
|
||||
return;
|
||||
}
|
||||
rejectUnknownKeys(value, new Set(["type", "coordinates"]), path, errors);
|
||||
if (!value.coordinates.every((coordinate) => typeof coordinate === "number" && Number.isFinite(coordinate))) {
|
||||
errors.push(`${path}_coordinates_must_be_finite_numbers`);
|
||||
return;
|
||||
}
|
||||
const [longitude, latitude] = value.coordinates;
|
||||
if (longitude < -180 || longitude > 180) errors.push(`${path}.longitude_out_of_range`);
|
||||
if (latitude < -90 || latitude > 90) errors.push(`${path}.latitude_out_of_range`);
|
||||
}
|
||||
|
||||
function validateRawEnvelope(value, errors) {
|
||||
if (!isPlainObject(value)) {
|
||||
errors.push("raw_must_be_object");
|
||||
return;
|
||||
}
|
||||
rejectUnknownKeys(value, new Set(["contentType", "payload", "hash", "ref", "retentionDays"]), "raw", errors);
|
||||
requiredString(value.contentType, "raw.contentType", errors);
|
||||
if (value.payload === undefined && value.ref === undefined) errors.push("raw_requires_payload_or_ref");
|
||||
if (value.payload !== undefined) errors.push("raw.inline_payload_not_supported");
|
||||
if (value.payload === undefined) requiredString(value.hash, "raw.hash", errors);
|
||||
if (value.hash !== undefined) requiredString(value.hash, "raw.hash", errors);
|
||||
if (value.ref !== undefined) {
|
||||
requiredString(value.ref, "raw.ref", errors);
|
||||
if (typeof value.ref === "string" && SECRET_LIKE_REFERENCE.test(value.ref)) {
|
||||
errors.push("raw.ref_must_not_contain_secret_material");
|
||||
}
|
||||
}
|
||||
if (value.retentionDays !== undefined && (!Number.isInteger(value.retentionDays) || value.retentionDays < 1)) {
|
||||
errors.push("raw.retentionDays_must_be_positive_integer");
|
||||
}
|
||||
}
|
||||
|
||||
function isPlainObject(value) {
|
||||
return Boolean(value) && typeof value === "object" && !Array.isArray(value);
|
||||
}
|
||||
|
||||
function rejectUnknownKeys(value, allowedKeys, path, errors) {
|
||||
if (!isPlainObject(value)) return;
|
||||
for (const key of Object.keys(value)) {
|
||||
if (!allowedKeys.has(key)) errors.push(`${path}.${key}_not_allowed`);
|
||||
}
|
||||
}
|
||||
|
||||
function serializedByteLength(value) {
|
||||
try {
|
||||
return Buffer.byteLength(JSON.stringify(value));
|
||||
} catch {
|
||||
return Number.POSITIVE_INFINITY;
|
||||
}
|
||||
}
|
||||
|
||||
function containsSecretLikeMaterial(value) {
|
||||
if (typeof value === "string") return SECRET_LIKE_VALUE.test(value);
|
||||
if (Array.isArray(value)) return value.some(containsSecretLikeMaterial);
|
||||
if (!isPlainObject(value)) return false;
|
||||
return Object.entries(value).some(([key, child]) => SECRET_LIKE_KEY.test(key) || containsSecretLikeMaterial(child));
|
||||
}
|
||||
|
|
@ -1,267 +0,0 @@
|
|||
import assert from "node:assert/strict";
|
||||
import {
|
||||
EXTERNAL_PROVIDER_CONTRACT_VERSION,
|
||||
FOUNDRY_BINDING_UPSERT_SCHEMA_VERSION,
|
||||
assertValid,
|
||||
validateCollectionProfile,
|
||||
validateConnectionProfile,
|
||||
validateDataProduct,
|
||||
validateFoundryBinding,
|
||||
validateFoundryBindingUpsert,
|
||||
validateIntakeBatch,
|
||||
validateProviderManifest,
|
||||
} from "../src/index.mjs";
|
||||
import { geliosPositionsCurrentExample } from "../examples/gelios-positions-current.v1.mjs";
|
||||
|
||||
assert.equal(validateProviderManifest(geliosPositionsCurrentExample.providerManifest).ok, true);
|
||||
assert.equal(validateConnectionProfile(geliosPositionsCurrentExample.connection).ok, true);
|
||||
assert.equal(validateCollectionProfile(geliosPositionsCurrentExample.collectionProfile).ok, true);
|
||||
assert.equal(validateDataProduct(geliosPositionsCurrentExample.dataProduct).ok, true);
|
||||
assert.equal(validateFoundryBinding(geliosPositionsCurrentExample.foundryBinding).ok, true);
|
||||
|
||||
const foundryBindingUpsert = {
|
||||
schemaVersion: FOUNDRY_BINDING_UPSERT_SCHEMA_VERSION,
|
||||
applicationId: "11111111-1111-4111-8111-111111111111",
|
||||
pageId: "map",
|
||||
idempotencyKey: "foundry-binding-0123456789abcdef0123456789abcdef",
|
||||
binding: {
|
||||
id: "fleet-live-points",
|
||||
dataProductId: "fleet.positions.current.v1",
|
||||
slotId: "points",
|
||||
semanticTypes: ["map.moving_object"],
|
||||
fieldProjection: ["name", "speed", "course"],
|
||||
},
|
||||
};
|
||||
assert.equal(validateFoundryBindingUpsert(foundryBindingUpsert).ok, true);
|
||||
assert.equal(validateFoundryBindingUpsert({
|
||||
...foundryBindingUpsert,
|
||||
binding: { ...foundryBindingUpsert.binding, semanticTypes: ["map.moving_object", "map.moving_object"] },
|
||||
}).errors.includes("binding.semanticTypes_must_not_contain_duplicates"), true);
|
||||
assert.equal(validateFoundryBindingUpsert({
|
||||
...foundryBindingUpsert,
|
||||
binding: { ...foundryBindingUpsert.binding, accessToken: "forbidden" },
|
||||
}).errors.includes("binding.accessToken_not_allowed"), true);
|
||||
assert.equal(validateFoundryBindingUpsert({
|
||||
...foundryBindingUpsert,
|
||||
binding: { ...foundryBindingUpsert.binding, fieldProjection: ["ndc_edprb_forbidden-reader-token"] },
|
||||
}).errors.includes("binding_must_not_contain_secret_material"), true);
|
||||
|
||||
const attributesWithSerializedSize = (size) => ({
|
||||
blob: "x".repeat(size - Buffer.byteLength(JSON.stringify({ blob: "" }))),
|
||||
});
|
||||
|
||||
const neutralIntakeBatch = {
|
||||
schemaVersion: EXTERNAL_PROVIDER_CONTRACT_VERSION,
|
||||
source: {
|
||||
providerId: "example-provider",
|
||||
tenantId: "sample-tenant",
|
||||
connectionId: "sample-connection",
|
||||
},
|
||||
contract: {
|
||||
dataProductId: "fleet.positions.current.v1",
|
||||
ontologyRevision: "ontology.example-fleet.v1",
|
||||
version: "1.0.0",
|
||||
},
|
||||
batch: {
|
||||
runId: "run-20260714-001",
|
||||
sequence: 0,
|
||||
idempotencyKey: "run-20260714-001.batch-0",
|
||||
receivedAt: "2026-07-14T18:00:00.000Z",
|
||||
},
|
||||
raw: {
|
||||
contentType: "application/json",
|
||||
hash: "sha256:example",
|
||||
ref: "restricted://raw/run-20260714-001",
|
||||
retentionDays: 14,
|
||||
},
|
||||
facts: [{
|
||||
sourceId: "source-object-42",
|
||||
semanticType: "map.moving_object",
|
||||
observedAt: "2026-07-14T17:59:58.000Z",
|
||||
geometry: { type: "Point", coordinates: [37.6173, 55.7558] },
|
||||
attributes: { status: "active" },
|
||||
}],
|
||||
};
|
||||
|
||||
assert.equal(validateIntakeBatch(neutralIntakeBatch).ok, true);
|
||||
assert.equal(validateIntakeBatch({
|
||||
...neutralIntakeBatch,
|
||||
raw: { contentType: "application/json", payload: { safe: "fixture" } },
|
||||
}).errors.includes("raw.inline_payload_not_supported"), true);
|
||||
assert.equal(validateIntakeBatch({
|
||||
...neutralIntakeBatch,
|
||||
raw: { contentType: "application/json", payload: "unsafe-unstructured-inline-raw" },
|
||||
}).errors.includes("raw.inline_payload_not_supported"), true);
|
||||
assert.equal(validateIntakeBatch({
|
||||
...neutralIntakeBatch,
|
||||
raw: { contentType: "application/json", payload: { providerAccessToken: "must-never-be-stored" } },
|
||||
}).errors.includes("intake_must_not_contain_secret_material"), true);
|
||||
assert.equal(validateIntakeBatch({
|
||||
...neutralIntakeBatch,
|
||||
facts: [{ ...neutralIntakeBatch.facts[0], attributes: { authorization: "Bearer must-never-be-stored" } }],
|
||||
}).errors.includes("intake_must_not_contain_secret_material"), true);
|
||||
assert.equal(validateIntakeBatch({
|
||||
...neutralIntakeBatch,
|
||||
facts: [{ ...neutralIntakeBatch.facts[0], attributes: { metadata: "ndc_edpwb_abcdefghijklmnopqrstuvwxyz0123456789ABCDE" } }],
|
||||
}).errors.includes("intake_must_not_contain_secret_material"), true);
|
||||
assert.equal(validateIntakeBatch({
|
||||
...neutralIntakeBatch,
|
||||
facts: [{ ...neutralIntakeBatch.facts[0], attributes: { metadata: "ndc_edprb_abcdefghijklmnopqrstuvwxyz0123456789ABCDE" } }],
|
||||
}).errors.includes("intake_must_not_contain_secret_material"), true);
|
||||
assert.equal(validateIntakeBatch({
|
||||
...neutralIntakeBatch,
|
||||
raw: { contentType: "application/json", hash: "prefix ndc_edpwb_abcdefghijklmnopqrstuvwxyz0123456789ABCDE suffix", ref: "restricted://raw/fixture" },
|
||||
}).errors.includes("intake_must_not_contain_secret_material"), true);
|
||||
assert.equal(validateIntakeBatch({
|
||||
...neutralIntakeBatch,
|
||||
facts: [{ ...neutralIntakeBatch.facts[0], attributes: { metadata: "Bearer opaque-secret-material" } }],
|
||||
}).errors.includes("intake_must_not_contain_secret_material"), true);
|
||||
assert.equal(validateIntakeBatch({
|
||||
...neutralIntakeBatch,
|
||||
raw: { contentType: "application/json", hash: "sha256:fixture", ref: "restricted://raw?access_token=must-never-be-stored" },
|
||||
}).errors.includes("raw.ref_must_not_contain_secret_material"), true);
|
||||
assert.equal(validateIntakeBatch({
|
||||
...neutralIntakeBatch,
|
||||
facts: [{ ...neutralIntakeBatch.facts[0], sourceId: "" }],
|
||||
}).errors.includes("facts[0].sourceId_invalid"), true);
|
||||
assert.equal(validateIntakeBatch({
|
||||
...neutralIntakeBatch,
|
||||
batch: { ...neutralIntakeBatch.batch, sequence: 2_147_483_647 },
|
||||
}).ok, true);
|
||||
assert.equal(validateIntakeBatch({
|
||||
...neutralIntakeBatch,
|
||||
batch: { ...neutralIntakeBatch.batch, sequence: 2_147_483_648 },
|
||||
}).errors.includes("batch.sequence_must_be_integer_0_to_2147483647"), true);
|
||||
assert.equal(validateIntakeBatch({
|
||||
...neutralIntakeBatch,
|
||||
facts: [{ ...neutralIntakeBatch.facts[0], attributes: attributesWithSerializedSize(64 * 1024) }],
|
||||
}).ok, true);
|
||||
assert.equal(validateIntakeBatch({
|
||||
...neutralIntakeBatch,
|
||||
facts: [{ ...neutralIntakeBatch.facts[0], attributes: attributesWithSerializedSize((64 * 1024) + 1) }],
|
||||
}).errors.includes("facts[0].attributes_size_exceeded"), true);
|
||||
assert.equal(validateIntakeBatch({
|
||||
...neutralIntakeBatch,
|
||||
facts: [{ ...neutralIntakeBatch.facts[0], geometry: { type: "Point", coordinates: [180.0001, 55.7558] } }],
|
||||
}).errors.includes("facts[0].geometry.longitude_out_of_range"), true);
|
||||
assert.equal(validateIntakeBatch({
|
||||
...neutralIntakeBatch,
|
||||
facts: [{ ...neutralIntakeBatch.facts[0], geometry: { type: "Point", coordinates: [37.6173, -90.0001] } }],
|
||||
}).errors.includes("facts[0].geometry.latitude_out_of_range"), true);
|
||||
assert.equal(validateIntakeBatch({
|
||||
...neutralIntakeBatch,
|
||||
debug: true,
|
||||
}).errors.includes("intakeBatch.debug_not_allowed"), true);
|
||||
|
||||
assert.equal(validateConnectionProfile({
|
||||
...geliosPositionsCurrentExample.connection,
|
||||
apiToken: "must-never-be-here",
|
||||
}).ok, false);
|
||||
assert.equal(validateConnectionProfile({
|
||||
...geliosPositionsCurrentExample.connection,
|
||||
credentialRef: {
|
||||
...geliosPositionsCurrentExample.connection.credentialRef,
|
||||
reference: "ndc_edpwb_not-allowed-even-in-a-non-secret-field",
|
||||
},
|
||||
}).errors.includes("profile_must_not_contain_secret_material"), true);
|
||||
assert.equal(validateConnectionProfile({
|
||||
...geliosPositionsCurrentExample.connection,
|
||||
credentialRef: {
|
||||
...geliosPositionsCurrentExample.connection.credentialRef,
|
||||
reference: "opaque-reference?access_token=must-not-cross-the-boundary",
|
||||
},
|
||||
}).errors.includes("profile_must_not_contain_secret_material"), true);
|
||||
assert.equal(validateConnectionProfile({
|
||||
...geliosPositionsCurrentExample.connection,
|
||||
credentialRef: { ...geliosPositionsCurrentExample.connection.credentialRef, namespace: "unexpected" },
|
||||
}).errors.includes("credentialRef.namespace_not_allowed"), true);
|
||||
assert.equal(validateConnectionProfile({
|
||||
...geliosPositionsCurrentExample.connection,
|
||||
scope: { ...geliosPositionsCurrentExample.connection.scope, providerSelector: "unexpected" },
|
||||
}).errors.includes("scope.providerSelector_not_allowed"), true);
|
||||
assert.equal(validateCollectionProfile({
|
||||
...geliosPositionsCurrentExample.collectionProfile,
|
||||
mode: "manual",
|
||||
}).errors.includes("manual_profile_must_not_define_intervalMs"), true);
|
||||
assert.equal(validateCollectionProfile({
|
||||
...geliosPositionsCurrentExample.collectionProfile,
|
||||
schedule: { intervalMs: 3000, jitterMs: 100 },
|
||||
}).errors.includes("schedule.jitterMs_not_allowed"), true);
|
||||
assert.equal(validateCollectionProfile({
|
||||
...geliosPositionsCurrentExample.collectionProfile,
|
||||
capabilityIds: ["ndc_edprb_forbidden-reader-token"],
|
||||
}).errors.includes("collectionProfile_must_not_contain_secret_material"), true);
|
||||
assert.equal(validateFoundryBinding({
|
||||
...geliosPositionsCurrentExample.foundryBinding,
|
||||
providerId: "gelios",
|
||||
}).errors.includes("binding_must_reference_data_product_not_provider_transport"), true);
|
||||
assert.equal(validateFoundryBinding({
|
||||
...geliosPositionsCurrentExample.foundryBinding,
|
||||
applicationId: undefined,
|
||||
}).errors.includes("applicationId_invalid"), true);
|
||||
assert.equal(validateFoundryBinding({
|
||||
...geliosPositionsCurrentExample.foundryBinding,
|
||||
pageId: undefined,
|
||||
}).errors.includes("pageId_invalid"), true);
|
||||
assert.equal(validateFoundryBinding({
|
||||
...geliosPositionsCurrentExample.foundryBinding,
|
||||
templateId: undefined,
|
||||
}).ok, true);
|
||||
assert.equal(validateFoundryBinding({
|
||||
...geliosPositionsCurrentExample.foundryBinding,
|
||||
rendererOptions: {},
|
||||
}).errors.includes("foundryBinding.rendererOptions_not_allowed"), true);
|
||||
assert.equal(validateFoundryBinding({
|
||||
...geliosPositionsCurrentExample.foundryBinding,
|
||||
semanticType: "ndc_edprb_forbidden-reader-token",
|
||||
}).errors.includes("binding_must_not_contain_secret_material"), true);
|
||||
|
||||
assert.equal(validateProviderManifest({
|
||||
...geliosPositionsCurrentExample.providerManifest,
|
||||
tenantId: "must-not-live-in-provider-manifest",
|
||||
}).errors.includes("manifest_must_not_contain_connection_runtime_state"), true);
|
||||
assert.equal(validateProviderManifest({
|
||||
...geliosPositionsCurrentExample.providerManifest,
|
||||
apiToken: "must-never-be-here",
|
||||
}).errors.includes("manifest_must_not_contain_secret_material"), true);
|
||||
assert.equal(validateProviderManifest({
|
||||
...geliosPositionsCurrentExample.providerManifest,
|
||||
endpoint: "https://must-live-in-l2-template.invalid",
|
||||
}).errors.includes("manifest_must_not_contain_provider_transport"), true);
|
||||
assert.equal(validateProviderManifest({
|
||||
...geliosPositionsCurrentExample.providerManifest,
|
||||
capabilities: [{ id: "gelios.units.current.read", classification: "not-a-class" }],
|
||||
}).errors.includes("capabilities[0].classification_invalid"), true);
|
||||
assert.equal(validateProviderManifest({
|
||||
...geliosPositionsCurrentExample.providerManifest,
|
||||
metadata: {},
|
||||
}).errors.includes("providerManifest.metadata_not_allowed"), true);
|
||||
assert.equal(validateProviderManifest({
|
||||
...geliosPositionsCurrentExample.providerManifest,
|
||||
ontology: { ...geliosPositionsCurrentExample.providerManifest.ontology, transport: "unexpected" },
|
||||
}).errors.includes("ontology.transport_not_allowed"), true);
|
||||
assert.equal(validateProviderManifest({
|
||||
...geliosPositionsCurrentExample.providerManifest,
|
||||
capabilities: [{ id: "gelios.units.current.read", classification: "read", endpoint: "/units" }],
|
||||
}).errors.includes("capabilities[0].endpoint_not_allowed"), true);
|
||||
assert.equal(validateProviderManifest({
|
||||
...geliosPositionsCurrentExample.providerManifest,
|
||||
dataProductIds: ["ndc_edpwb_forbidden-writer-token"],
|
||||
}).errors.includes("manifest_must_not_contain_secret_material"), true);
|
||||
|
||||
assert.equal(validateDataProduct({
|
||||
...geliosPositionsCurrentExample.dataProduct,
|
||||
delivery: { mode: "snapshot+patch", transport: "sse" },
|
||||
}).errors.includes("delivery.transport_not_allowed"), true);
|
||||
|
||||
assert.throws(() => assertValid(validateDataProduct, {
|
||||
schemaVersion: EXTERNAL_PROVIDER_CONTRACT_VERSION,
|
||||
id: "fleet.positions.current.v1",
|
||||
version: "1.0.0",
|
||||
delivery: { mode: "snapshot" },
|
||||
semanticTypes: [],
|
||||
fields: [],
|
||||
access: { audience: "public" },
|
||||
}));
|
||||
|
||||
console.log("external-provider-contract: ok");
|
||||
|
|
@ -1,111 +0,0 @@
|
|||
import assert from "node:assert/strict";
|
||||
import {
|
||||
DATA_PRODUCT_PATCH_SCHEMA_VERSION,
|
||||
DATA_PRODUCT_PUBLISH_SCHEMA_VERSION,
|
||||
DATA_PRODUCT_SNAPSHOT_SCHEMA_VERSION,
|
||||
validateDataProductPatch,
|
||||
validateDataProductPublish,
|
||||
validateDataProductSnapshot,
|
||||
} from "../src/index.mjs";
|
||||
|
||||
const fact = {
|
||||
sourceId: "unit-42",
|
||||
semanticType: "map.moving_object",
|
||||
observedAt: "2026-07-15T10:00:00.000Z",
|
||||
attributes: { status: "online" },
|
||||
geometry: { type: "Point", coordinates: [37.6173, 55.7558] },
|
||||
};
|
||||
|
||||
const attributesWithSerializedSize = (size) => ({
|
||||
blob: "x".repeat(size - Buffer.byteLength(JSON.stringify({ blob: "" }))),
|
||||
});
|
||||
|
||||
const publish = {
|
||||
schemaVersion: DATA_PRODUCT_PUBLISH_SCHEMA_VERSION,
|
||||
batch: { runId: "execution-42", sequence: 0, idempotencyKey: "execution-42.node-01.chunk-0" },
|
||||
facts: [fact],
|
||||
};
|
||||
assert.equal(validateDataProductPublish(publish).ok, true);
|
||||
assert.equal(validateDataProductPublish({
|
||||
...publish,
|
||||
source: { tenantId: "forged" },
|
||||
}).errors.includes("publish.source_not_allowed"), true);
|
||||
assert.equal(validateDataProductPublish({
|
||||
...publish,
|
||||
facts: [{ ...fact, attributes: { accessToken: "must-never-cross-the-boundary" } }],
|
||||
}).errors.includes("publish_must_not_contain_secret_material"), true);
|
||||
assert.equal(validateDataProductPublish({
|
||||
...publish,
|
||||
facts: [fact, { ...fact, observedAt: "2026-07-15T10:00:01.000Z" }],
|
||||
}).errors.includes("facts_duplicate_entity_key"), true);
|
||||
assert.equal(validateDataProductPublish({
|
||||
...publish,
|
||||
batch: { ...publish.batch, sequence: 2_147_483_647 },
|
||||
}).ok, true);
|
||||
assert.equal(validateDataProductPublish({
|
||||
...publish,
|
||||
batch: { ...publish.batch, sequence: 2_147_483_648 },
|
||||
}).errors.includes("batch.sequence_must_be_integer_0_to_2147483647"), true);
|
||||
assert.equal(validateDataProductPublish({
|
||||
...publish,
|
||||
facts: [{ ...fact, attributes: attributesWithSerializedSize(64 * 1024) }],
|
||||
}).ok, true);
|
||||
assert.equal(validateDataProductPublish({
|
||||
...publish,
|
||||
facts: [{ ...fact, attributes: attributesWithSerializedSize((64 * 1024) + 1) }],
|
||||
}).errors.includes("facts[0].attributes_size_exceeded"), true);
|
||||
assert.equal(validateDataProductPublish({
|
||||
...publish,
|
||||
facts: [{ ...fact, attributes: attributesWithSerializedSize((64 * 1024) + 1) }],
|
||||
}, { maxAttributesBytes: 1024 * 1024 }).errors.includes("facts[0].attributes_size_exceeded"), true);
|
||||
assert.equal(validateDataProductPublish({
|
||||
...publish,
|
||||
facts: [{ ...fact, geometry: { type: "Point", coordinates: [-180.0001, 55.7558] } }],
|
||||
}).errors.includes("facts[0].geometry.longitude_out_of_range"), true);
|
||||
assert.equal(validateDataProductPublish({
|
||||
...publish,
|
||||
facts: [{ ...fact, geometry: { type: "Point", coordinates: [37.6173, 90.0001] } }],
|
||||
}).errors.includes("facts[0].geometry.latitude_out_of_range"), true);
|
||||
|
||||
const canonicalFact = { ...fact, receivedAt: "2026-07-15T10:00:01.000Z" };
|
||||
const snapshot = {
|
||||
schemaVersion: DATA_PRODUCT_SNAPSHOT_SCHEMA_VERSION,
|
||||
dataProduct: { id: "fleet.positions.current.v1", version: "1.0.0" },
|
||||
generatedAt: "2026-07-15T10:00:01.000Z",
|
||||
cursor: "12",
|
||||
facts: [canonicalFact],
|
||||
};
|
||||
assert.equal(validateDataProductSnapshot(snapshot).ok, true);
|
||||
assert.equal(validateDataProductSnapshot({ ...snapshot, transport: "sse" }).errors.includes("snapshot.transport_not_allowed"), true);
|
||||
assert.equal(validateDataProductSnapshot({
|
||||
...snapshot,
|
||||
facts: [{ ...canonicalFact, attributes: attributesWithSerializedSize((64 * 1024) + 1) }],
|
||||
}).errors.includes("facts[0].attributes_size_exceeded"), true);
|
||||
assert.equal(validateDataProductSnapshot({
|
||||
...snapshot,
|
||||
facts: [{ ...canonicalFact, attributes: { status: "ndc_edprb_forbidden-reader-token" } }],
|
||||
}).errors.includes("snapshot_must_not_contain_secret_material"), true);
|
||||
|
||||
const patch = {
|
||||
schemaVersion: DATA_PRODUCT_PATCH_SCHEMA_VERSION,
|
||||
dataProduct: { id: "fleet.positions.current.v1", version: "1.0.0" },
|
||||
cursor: "13",
|
||||
previousCursor: "12",
|
||||
emittedAt: "2026-07-15T10:00:02.000Z",
|
||||
operations: [{ op: "upsert", fact: canonicalFact }],
|
||||
};
|
||||
assert.equal(validateDataProductPatch(patch).ok, true);
|
||||
assert.equal(validateDataProductPatch({
|
||||
...patch,
|
||||
operations: [{ op: "delete", fact: canonicalFact }],
|
||||
}).errors.includes("operations[0].op_must_be_upsert"), true);
|
||||
assert.equal(validateDataProductPatch({
|
||||
...patch,
|
||||
operations: [{ op: "upsert", fact: { ...canonicalFact, attributes: attributesWithSerializedSize((64 * 1024) + 1) } }],
|
||||
}).errors.includes("operations[0].fact.attributes_size_exceeded"), true);
|
||||
assert.equal(validateDataProductPatch({
|
||||
...patch,
|
||||
operations: [{ op: "upsert", fact: { ...canonicalFact, attributes: { status: "ndc_edpwb_forbidden-writer-token" } } }],
|
||||
}).errors.includes("patch_must_not_contain_secret_material"), true);
|
||||
|
||||
console.log("data-product-contract: ok");
|
||||
|
|
@ -1,242 +0,0 @@
|
|||
import assert from "node:assert/strict";
|
||||
import { generateKeyPairSync, sign } from "node:crypto";
|
||||
|
||||
import {
|
||||
ENGINE_CREDENTIAL_SINK_AUDIT_SCHEMA_VERSION,
|
||||
ENGINE_CREDENTIAL_SINK_PROVISION_SCHEMA_VERSION,
|
||||
ENGINE_CREDENTIAL_SINK_RECEIPT_SCHEMA_VERSION,
|
||||
ENGINE_CREDENTIAL_SINK_ROLLBACK_RECEIPT_SCHEMA_VERSION,
|
||||
ENGINE_CREDENTIAL_SINK_ROLLBACK_SCHEMA_VERSION,
|
||||
computeEngineCredentialCapabilityDigest,
|
||||
computeEngineCredentialSinkPolicyHash,
|
||||
computeEngineCredentialSinkReceiptHash,
|
||||
engineCredentialSinkAuditTargets,
|
||||
validateEngineCredentialSinkAudit,
|
||||
validateEngineCredentialSinkProvision,
|
||||
validateEngineCredentialSinkReceipt,
|
||||
validateEngineCredentialSinkRollback,
|
||||
validateEngineCredentialSinkRollbackReceipt,
|
||||
} from "../src/index.mjs";
|
||||
|
||||
const hash = (character) => `sha256:${character.repeat(64)}`;
|
||||
const issuer = { serviceId: "platform.external-data-plane", keyId: "edp-issuer-20260715-001" };
|
||||
const { publicKey: issuerPublicKey, privateKey: issuerPrivateKey } = generateKeyPairSync("ed25519");
|
||||
const issuerPublicKeys = { [`${issuer.serviceId}:${issuer.keyId}`]: issuerPublicKey };
|
||||
const target = (nodeId, nodeType, credentialType) => ({
|
||||
workflowId: "WCb62yGL8v",
|
||||
workflowRevision: "revision-20260715-001",
|
||||
nodeId,
|
||||
nodeType,
|
||||
credentialType,
|
||||
});
|
||||
|
||||
const request = {
|
||||
schemaVersion: ENGINE_CREDENTIAL_SINK_PROVISION_SCHEMA_VERSION,
|
||||
transaction: {
|
||||
id: "credential-transaction-20260715-001",
|
||||
idempotencyKey: "credential-transaction-20260715-001",
|
||||
requestedAt: "2026-07-15T18:00:00.000Z",
|
||||
requestExpiresAt: "2026-07-15T18:10:00.000Z",
|
||||
policyHash: hash("0"),
|
||||
failureMode: "rollback-all",
|
||||
issuer,
|
||||
attestation: { algorithm: "Ed25519", signature: "A".repeat(86) },
|
||||
},
|
||||
bindings: [
|
||||
{
|
||||
bindingId: "positions-writer",
|
||||
capabilityType: "external-data-plane.writer",
|
||||
grantId: "writer-grant-001",
|
||||
target: target(
|
||||
"publish-node-001",
|
||||
"n8n-nodes-ndc.ndcDataProductPublish",
|
||||
"ndcDataProductWriterApi",
|
||||
),
|
||||
expiresAt: "2026-08-15T18:00:00.000Z",
|
||||
policyHash: hash("1"),
|
||||
material: { format: "opaque-bearer", value: `ndc_edpwb_${"A".repeat(43)}` },
|
||||
},
|
||||
{
|
||||
bindingId: "positions-reader",
|
||||
capabilityType: "external-data-plane.reader",
|
||||
grantId: "reader-grant-001",
|
||||
target: target(
|
||||
"read-node-001",
|
||||
"n8n-nodes-ndc.ndcDataProductRead",
|
||||
"ndcDataProductReaderApi",
|
||||
),
|
||||
expiresAt: "2026-08-15T18:00:00.000Z",
|
||||
policyHash: hash("2"),
|
||||
material: { format: "opaque-bearer", value: `ndc_edprb_${"B".repeat(43)}` },
|
||||
},
|
||||
{
|
||||
bindingId: "positions-foundry",
|
||||
capabilityType: "foundry.binding",
|
||||
grantId: "foundry-grant-001",
|
||||
target: target(
|
||||
"foundry-node-001",
|
||||
"n8n-nodes-ndc.ndcFoundryBinding",
|
||||
"ndcFoundryBindingApi",
|
||||
),
|
||||
expiresAt: "2026-08-15T18:00:00.000Z",
|
||||
policyHash: hash("3"),
|
||||
material: { format: "opaque-bearer", value: `ndc_fndbg_${"C".repeat(43)}` },
|
||||
},
|
||||
],
|
||||
};
|
||||
for (const binding of request.bindings) {
|
||||
binding.capabilityDigest = computeEngineCredentialCapabilityDigest(binding.material.value);
|
||||
}
|
||||
attest(request);
|
||||
const provisionValidationNow = "2026-07-15T18:00:01.000Z";
|
||||
|
||||
assert.equal(validateEngineCredentialSinkProvision(request, { now: provisionValidationNow, issuerPublicKeys }).ok, true);
|
||||
assert.equal(validateEngineCredentialSinkProvision(request, {
|
||||
now: provisionValidationNow,
|
||||
}).errors.includes("transaction.issuer_public_key_required"), true);
|
||||
assert.equal(validateEngineCredentialSinkProvision({
|
||||
...request,
|
||||
transaction: { ...request.transaction, policyHash: hash("f") },
|
||||
}, { now: provisionValidationNow, issuerPublicKeys }).errors.includes("transaction.policyHash_mismatch"), true);
|
||||
assert.equal(validateEngineCredentialSinkProvision({
|
||||
...request,
|
||||
bindings: [{
|
||||
...request.bindings[0],
|
||||
target: { ...request.bindings[0].target, nodeType: "n8n-nodes-ndc.ndcDataProductRead" },
|
||||
}],
|
||||
}, { now: provisionValidationNow, issuerPublicKeys }).errors.includes("bindings[0].target.nodeType_capability_mismatch"), true);
|
||||
assert.equal(validateEngineCredentialSinkProvision({
|
||||
...request,
|
||||
bindings: [{
|
||||
...request.bindings[0],
|
||||
material: { format: "opaque-bearer", value: request.bindings[1].material.value },
|
||||
}],
|
||||
}, { now: provisionValidationNow, issuerPublicKeys }).errors.includes("bindings[0].material.value_invalid_for_capability"), true);
|
||||
|
||||
const swappedCapabilityRequest = structuredClone(request);
|
||||
swappedCapabilityRequest.bindings[0].material.value = `ndc_edpwb_${"D".repeat(43)}`;
|
||||
assert.equal(validateEngineCredentialSinkProvision(swappedCapabilityRequest, {
|
||||
now: provisionValidationNow,
|
||||
issuerPublicKeys,
|
||||
}).errors.includes("bindings[0].capabilityDigest_material_mismatch"), true);
|
||||
|
||||
const resignedByAttackerRequest = structuredClone(swappedCapabilityRequest);
|
||||
resignedByAttackerRequest.bindings[0].capabilityDigest = computeEngineCredentialCapabilityDigest(
|
||||
resignedByAttackerRequest.bindings[0].material.value,
|
||||
);
|
||||
resignedByAttackerRequest.transaction.policyHash = computeEngineCredentialSinkPolicyHash(resignedByAttackerRequest);
|
||||
assert.equal(validateEngineCredentialSinkProvision(resignedByAttackerRequest, {
|
||||
now: provisionValidationNow,
|
||||
issuerPublicKeys,
|
||||
}).errors.includes("transaction.attestation_invalid"), true);
|
||||
|
||||
const staleRequest = structuredClone(request);
|
||||
staleRequest.transaction.requestedAt = "2020-01-01T00:00:00.000Z";
|
||||
staleRequest.transaction.requestExpiresAt = "2020-01-01T00:10:00.000Z";
|
||||
attest(staleRequest);
|
||||
assert.equal(validateEngineCredentialSinkProvision(staleRequest, { now: provisionValidationNow, issuerPublicKeys }).errors.includes("request_expired"), true);
|
||||
|
||||
const futureRequest = structuredClone(request);
|
||||
futureRequest.transaction.requestedAt = "2026-07-15T18:02:00.000Z";
|
||||
futureRequest.transaction.requestExpiresAt = "2026-07-15T18:12:00.000Z";
|
||||
attest(futureRequest);
|
||||
assert.equal(validateEngineCredentialSinkProvision(futureRequest, { now: provisionValidationNow, issuerPublicKeys }).errors.includes("requestedAt_exceeds_clock_skew"), true);
|
||||
|
||||
const credentials = request.bindings.map((binding, index) => ({
|
||||
bindingId: binding.bindingId,
|
||||
capabilityType: binding.capabilityType,
|
||||
grantId: binding.grantId,
|
||||
target: binding.target,
|
||||
credentialRef: `engcred_${index + 1}_opaque_reference`,
|
||||
expiresAt: binding.expiresAt,
|
||||
policyHash: binding.policyHash,
|
||||
capabilityDigest: binding.capabilityDigest,
|
||||
disposition: "created",
|
||||
}));
|
||||
const receipt = {
|
||||
schemaVersion: ENGINE_CREDENTIAL_SINK_RECEIPT_SCHEMA_VERSION,
|
||||
transactionId: request.transaction.id,
|
||||
idempotencyKey: request.transaction.idempotencyKey,
|
||||
outcome: "committed",
|
||||
policyHash: request.transaction.policyHash,
|
||||
processedAt: "2026-07-15T18:00:02.000Z",
|
||||
credentials,
|
||||
rollback: { status: "not-required" },
|
||||
};
|
||||
assert.equal(validateEngineCredentialSinkReceipt(receipt, { request, issuerPublicKeys }).ok, true);
|
||||
assert.equal(validateEngineCredentialSinkReceipt({
|
||||
...receipt,
|
||||
credentials: [{ ...credentials[0], target: { ...credentials[0].target, nodeId: "wrong-node" } }, ...credentials.slice(1)],
|
||||
}, { request, issuerPublicKeys }).errors.includes("credentials_request_target_mismatch"), true);
|
||||
assert.equal(validateEngineCredentialSinkReceipt({
|
||||
...receipt,
|
||||
capability: request.bindings[0].material.value,
|
||||
}).errors.includes("credentialSinkReceipt.capability_not_allowed"), true);
|
||||
assert.equal(validateEngineCredentialSinkReceipt({
|
||||
...receipt,
|
||||
outcome: "rolled-back",
|
||||
credentials: credentials.slice(0, 1),
|
||||
rollback: { status: "complete", completedAt: "2026-07-15T18:00:02.000Z" },
|
||||
errorCode: "engine_binding_failed",
|
||||
}).errors.includes("noncommitted_credentials_must_be_empty"), true);
|
||||
|
||||
const committedReceiptHash = computeEngineCredentialSinkReceiptHash(receipt);
|
||||
const rollbackRequest = {
|
||||
schemaVersion: ENGINE_CREDENTIAL_SINK_ROLLBACK_SCHEMA_VERSION,
|
||||
rollback: {
|
||||
id: "credential-rollback-20260715-001",
|
||||
idempotencyKey: "credential-rollback-20260715-001",
|
||||
transactionId: request.transaction.id,
|
||||
requestedAt: "2026-07-15T18:05:00.000Z",
|
||||
requestExpiresAt: "2026-07-15T18:10:00.000Z",
|
||||
policyHash: request.transaction.policyHash,
|
||||
committedReceiptHash,
|
||||
reasonCode: "operator_requested",
|
||||
},
|
||||
};
|
||||
assert.equal(validateEngineCredentialSinkRollback(rollbackRequest, { now: "2026-07-15T18:05:01.000Z" }).ok, true);
|
||||
const staleRollbackRequest = structuredClone(rollbackRequest);
|
||||
staleRollbackRequest.rollback.requestedAt = "2020-01-01T00:00:00.000Z";
|
||||
staleRollbackRequest.rollback.requestExpiresAt = "2020-01-01T00:10:00.000Z";
|
||||
assert.equal(validateEngineCredentialSinkRollback(staleRollbackRequest, { now: "2026-07-15T18:05:01.000Z" }).errors.includes("request_expired"), true);
|
||||
const rollbackReceipt = {
|
||||
schemaVersion: ENGINE_CREDENTIAL_SINK_ROLLBACK_RECEIPT_SCHEMA_VERSION,
|
||||
rollbackId: rollbackRequest.rollback.id,
|
||||
transactionId: rollbackRequest.rollback.transactionId,
|
||||
outcome: "rolled-back",
|
||||
policyHash: rollbackRequest.rollback.policyHash,
|
||||
committedReceiptHash,
|
||||
processedAt: "2026-07-15T18:05:01.000Z",
|
||||
};
|
||||
assert.equal(validateEngineCredentialSinkRollbackReceipt(rollbackReceipt, { request: rollbackRequest }).ok, true);
|
||||
|
||||
const audit = {
|
||||
schemaVersion: ENGINE_CREDENTIAL_SINK_AUDIT_SCHEMA_VERSION,
|
||||
eventId: "credential-audit-20260715-001",
|
||||
transactionId: request.transaction.id,
|
||||
operationId: request.transaction.id,
|
||||
operation: "provision",
|
||||
outcome: "committed",
|
||||
occurredAt: receipt.processedAt,
|
||||
policyHash: request.transaction.policyHash,
|
||||
principal: { serviceId: "platform.credential-provisioner", fingerprint: hash("a") },
|
||||
targets: engineCredentialSinkAuditTargets(request, receipt),
|
||||
};
|
||||
assert.equal(validateEngineCredentialSinkAudit(audit).ok, true);
|
||||
assert.equal(JSON.stringify(audit).includes("ndc_edpwb_"), false);
|
||||
assert.equal(JSON.stringify(audit).includes("engcred_1_opaque_reference"), false);
|
||||
assert.equal(validateEngineCredentialSinkAudit({
|
||||
...audit,
|
||||
authorization: `Bearer ${request.bindings[0].material.value}`,
|
||||
}).errors.includes("audit_must_not_contain_secret_material"), true);
|
||||
|
||||
function attest(value) {
|
||||
value.transaction.policyHash = computeEngineCredentialSinkPolicyHash(value);
|
||||
value.transaction.attestation.signature = sign(
|
||||
null,
|
||||
Buffer.from(value.transaction.policyHash, "utf8"),
|
||||
issuerPrivateKey,
|
||||
).toString("base64url");
|
||||
}
|
||||
|
||||
console.log("external-provider credential sink contract: ok");
|
||||
|
|
@ -1,370 +0,0 @@
|
|||
import assert from "node:assert/strict";
|
||||
|
||||
import {
|
||||
ENGINE_PRIVATE_EXTENSION_APPLY_RECEIPT_SCHEMA_VERSION,
|
||||
ENGINE_PRIVATE_EXTENSION_APPLY_REQUEST_SCHEMA_VERSION,
|
||||
ENGINE_PRIVATE_EXTENSION_CREDENTIAL_TYPES,
|
||||
ENGINE_PRIVATE_EXTENSION_INACTIVE_BASELINE,
|
||||
ENGINE_PRIVATE_EXTENSION_MANAGE_CAPABILITY,
|
||||
ENGINE_PRIVATE_EXTENSION_NODE_TYPES,
|
||||
ENGINE_PRIVATE_EXTENSION_OPERATION_SCHEMA_VERSION,
|
||||
ENGINE_PRIVATE_EXTENSION_PACKAGE_NAME,
|
||||
ENGINE_PRIVATE_EXTENSION_PLAN_REQUEST_SCHEMA_VERSION,
|
||||
ENGINE_PRIVATE_EXTENSION_PLAN_SCHEMA_VERSION,
|
||||
ENGINE_PRIVATE_EXTENSION_READ_CAPABILITY,
|
||||
ENGINE_PRIVATE_EXTENSION_STATUS_SCHEMA_VERSION,
|
||||
authorizeEnginePrivateExtensionOperation,
|
||||
computeEnginePrivateExtensionPlanHash,
|
||||
validateEnginePrivateExtensionApplyReceipt,
|
||||
validateEnginePrivateExtensionApplyRequest,
|
||||
validateEnginePrivateExtensionOperation,
|
||||
validateEnginePrivateExtensionPlan,
|
||||
validateEnginePrivateExtensionPlanRequest,
|
||||
validateEnginePrivateExtensionStatus,
|
||||
} from "../src/index.mjs";
|
||||
|
||||
const digest = "03413c6f3c706a1f6a9597a1cf1730504e9719228d0c77fdfb93bbdcc57a4cf3";
|
||||
const release = Object.freeze({
|
||||
kind: "release",
|
||||
packageName: ENGINE_PRIVATE_EXTENSION_PACKAGE_NAME,
|
||||
releaseId: "0.1.0-03413c6f3c706a1f",
|
||||
packageSha256: digest,
|
||||
});
|
||||
const inactive = Object.freeze({
|
||||
kind: "inactive-baseline",
|
||||
packageName: ENGINE_PRIVATE_EXTENSION_PACKAGE_NAME,
|
||||
baselineId: ENGINE_PRIVATE_EXTENSION_INACTIVE_BASELINE,
|
||||
});
|
||||
const manage = [ENGINE_PRIVATE_EXTENSION_MANAGE_CAPABILITY];
|
||||
|
||||
const activateRequest = {
|
||||
schemaVersion: ENGINE_PRIVATE_EXTENSION_PLAN_REQUEST_SCHEMA_VERSION,
|
||||
requestId: "extension-request-20260715-001",
|
||||
idempotencyKey: "extension-request-20260715-001",
|
||||
action: "activate",
|
||||
requestedAt: "2026-07-15T18:00:00.000Z",
|
||||
requestExpiresAt: "2026-07-15T18:10:00.000Z",
|
||||
expectedCurrentGeneration: 0,
|
||||
target: release,
|
||||
};
|
||||
|
||||
assert.deepEqual(
|
||||
validateEnginePrivateExtensionPlanRequest(activateRequest, {
|
||||
now: "2026-07-15T18:00:01.000Z",
|
||||
grantedCapabilities: manage,
|
||||
}),
|
||||
{ ok: true, errors: [] },
|
||||
);
|
||||
assert.equal(
|
||||
validateEnginePrivateExtensionPlanRequest(activateRequest, {
|
||||
now: "2026-07-15T18:00:01.000Z",
|
||||
grantedCapabilities: ["engine.l2.deploy"],
|
||||
}).errors.includes("engine_private_extension_capability_required"),
|
||||
true,
|
||||
);
|
||||
assert.equal(authorizeEnginePrivateExtensionOperation("status", [ENGINE_PRIVATE_EXTENSION_READ_CAPABILITY]).ok, true);
|
||||
assert.equal(authorizeEnginePrivateExtensionOperation("apply", [ENGINE_PRIVATE_EXTENSION_READ_CAPABILITY]).ok, false);
|
||||
|
||||
const activatePlan = withPlanHash({
|
||||
schemaVersion: ENGINE_PRIVATE_EXTENSION_PLAN_SCHEMA_VERSION,
|
||||
planId: "extension-plan-20260715-001",
|
||||
planHash: hash("0"),
|
||||
requestId: activateRequest.requestId,
|
||||
idempotencyKey: activateRequest.idempotencyKey,
|
||||
action: "activate",
|
||||
createdAt: "2026-07-15T18:00:01.000Z",
|
||||
expiresAt: "2026-07-15T18:09:00.000Z",
|
||||
singleUse: true,
|
||||
requiredCapability: ENGINE_PRIVATE_EXTENSION_MANAGE_CAPABILITY,
|
||||
expectedCurrentGeneration: 0,
|
||||
nextGeneration: 1,
|
||||
currentState: inactive,
|
||||
targetState: release,
|
||||
recoveryState: inactive,
|
||||
actions: [
|
||||
"verify_staged_immutable_release",
|
||||
"prepare_sealed_package_tree",
|
||||
"verify_community_package_loader_policy",
|
||||
"quiesce_deploy_run_and_drain_queue",
|
||||
"record_recovery_state",
|
||||
"atomic_switch_current",
|
||||
"force_recreate_main_workers_webhooks_as_version_barrier",
|
||||
"verify_exact_runtime_acceptance",
|
||||
"commit_active_state",
|
||||
"resume_deploy_run",
|
||||
],
|
||||
transition: transition(),
|
||||
acceptance: acceptanceSpec(release),
|
||||
failurePolicy: failurePolicy(),
|
||||
});
|
||||
|
||||
assert.deepEqual(validateEnginePrivateExtensionPlan(activatePlan, { request: activateRequest }), { ok: true, errors: [] });
|
||||
assert.equal(
|
||||
validateEnginePrivateExtensionPlan({ ...activatePlan, nextGeneration: 2 }).errors.includes("nextGeneration_must_increment_current_generation"),
|
||||
true,
|
||||
);
|
||||
assert.equal(
|
||||
validateEnginePrivateExtensionPlan({ ...activatePlan, transition: { ...transition(), runtimeAction: "restart" } })
|
||||
.errors.includes("transition_policy_mismatch"),
|
||||
true,
|
||||
);
|
||||
assert.equal(
|
||||
validateEnginePrivateExtensionPlan({
|
||||
...activatePlan,
|
||||
transition: { ...transition(), loaderMode: "custom-extension", loaderPath: "N8N_CUSTOM_EXTENSIONS" },
|
||||
}).errors.includes("transition_policy_mismatch"),
|
||||
true,
|
||||
);
|
||||
assert.equal(
|
||||
validateEnginePrivateExtensionPlan({
|
||||
...activatePlan,
|
||||
transition: { ...transition(), hotReload: true },
|
||||
}).errors.includes("transition_policy_mismatch"),
|
||||
true,
|
||||
);
|
||||
assert.equal(
|
||||
validateEnginePrivateExtensionPlan({
|
||||
...activatePlan,
|
||||
transition: {
|
||||
...transition(),
|
||||
loaderEnvironment: { ...transition().loaderEnvironment, N8N_REINSTALL_MISSING_PACKAGES: "true" },
|
||||
},
|
||||
}).errors.includes("transition_policy_mismatch"),
|
||||
true,
|
||||
);
|
||||
|
||||
const applyRequest = {
|
||||
schemaVersion: ENGINE_PRIVATE_EXTENSION_APPLY_REQUEST_SCHEMA_VERSION,
|
||||
planId: activatePlan.planId,
|
||||
planHash: activatePlan.planHash,
|
||||
idempotencyKey: activatePlan.idempotencyKey,
|
||||
confirmedAt: "2026-07-15T18:00:02.000Z",
|
||||
};
|
||||
assert.deepEqual(validateEnginePrivateExtensionApplyRequest(applyRequest, {
|
||||
plan: activatePlan,
|
||||
now: "2026-07-15T18:00:02.000Z",
|
||||
grantedCapabilities: manage,
|
||||
}), { ok: true, errors: [] });
|
||||
assert.equal(validateEnginePrivateExtensionApplyRequest(applyRequest, {
|
||||
plan: activatePlan,
|
||||
now: "2026-07-15T18:10:00.000Z",
|
||||
grantedCapabilities: manage,
|
||||
}).errors.includes("plan_expired"), true);
|
||||
|
||||
const receipt = {
|
||||
schemaVersion: ENGINE_PRIVATE_EXTENSION_APPLY_RECEIPT_SCHEMA_VERSION,
|
||||
operationId: "extension-operation-20260715-001",
|
||||
planId: activatePlan.planId,
|
||||
planHash: activatePlan.planHash,
|
||||
action: "activate",
|
||||
acceptedAt: "2026-07-15T18:00:02.100Z",
|
||||
state: "queued",
|
||||
};
|
||||
assert.deepEqual(validateEnginePrivateExtensionApplyReceipt(receipt, { plan: activatePlan }), { ok: true, errors: [] });
|
||||
assert.equal(validateEnginePrivateExtensionApplyReceipt({ ...receipt, state: "active" })
|
||||
.errors.includes("apply_receipt_state_must_be_queued"), true);
|
||||
|
||||
const activeOperation = {
|
||||
schemaVersion: ENGINE_PRIVATE_EXTENSION_OPERATION_SCHEMA_VERSION,
|
||||
operationId: receipt.operationId,
|
||||
planId: activatePlan.planId,
|
||||
planHash: activatePlan.planHash,
|
||||
action: "activate",
|
||||
state: "active",
|
||||
outcome: "committed",
|
||||
phase: "complete",
|
||||
expectedCurrentGeneration: 0,
|
||||
nextGeneration: 1,
|
||||
targetState: release,
|
||||
recoveryState: inactive,
|
||||
effectiveState: release,
|
||||
runtime: runtime(1, 2, 2),
|
||||
acceptance: acceptanceReport(release, "accepted"),
|
||||
updatedAt: "2026-07-15T18:00:32.000Z",
|
||||
};
|
||||
assert.deepEqual(validateEnginePrivateExtensionOperation(activeOperation), { ok: true, errors: [] });
|
||||
const unexpectedSchema = structuredClone(activeOperation);
|
||||
unexpectedSchema.acceptance.nodeTypes.observed.push("n8n-nodes-ndc.unreviewedNode");
|
||||
assert.equal(validateEnginePrivateExtensionOperation(unexpectedSchema).errors
|
||||
.includes("acceptance.nodeTypes.observed_exact_set_required"), true);
|
||||
|
||||
const automaticRollback = {
|
||||
...activeOperation,
|
||||
state: "rolled-back",
|
||||
outcome: "automatically-rolled-back",
|
||||
phase: "complete",
|
||||
effectiveState: inactive,
|
||||
acceptance: acceptanceReport(inactive, "accepted"),
|
||||
errorCode: "runtime_acceptance_failed",
|
||||
};
|
||||
assert.deepEqual(validateEnginePrivateExtensionOperation(automaticRollback), { ok: true, errors: [] });
|
||||
assert.equal(
|
||||
validateEnginePrivateExtensionOperation({ ...automaticRollback, errorCode: undefined })
|
||||
.errors.includes("errorCode_invalid"),
|
||||
true,
|
||||
);
|
||||
|
||||
const activeStatus = {
|
||||
schemaVersion: ENGINE_PRIVATE_EXTENSION_STATUS_SCHEMA_VERSION,
|
||||
packageName: ENGINE_PRIVATE_EXTENSION_PACKAGE_NAME,
|
||||
generation: 1,
|
||||
health: "ready",
|
||||
currentState: release,
|
||||
previousState: inactive,
|
||||
runtime: runtime(1, 2, 2),
|
||||
acceptance: acceptanceReport(release, "accepted"),
|
||||
updatedAt: "2026-07-15T18:00:33.000Z",
|
||||
};
|
||||
assert.deepEqual(validateEnginePrivateExtensionStatus(activeStatus), { ok: true, errors: [] });
|
||||
|
||||
const rollbackRequest = {
|
||||
schemaVersion: ENGINE_PRIVATE_EXTENSION_PLAN_REQUEST_SCHEMA_VERSION,
|
||||
requestId: "extension-rollback-request-20260715-001",
|
||||
idempotencyKey: "extension-rollback-request-20260715-001",
|
||||
action: "rollback",
|
||||
requestedAt: "2026-07-15T18:05:00.000Z",
|
||||
requestExpiresAt: "2026-07-15T18:15:00.000Z",
|
||||
expectedCurrentGeneration: 1,
|
||||
target: { kind: "previous-state", packageName: ENGINE_PRIVATE_EXTENSION_PACKAGE_NAME },
|
||||
};
|
||||
assert.deepEqual(validateEnginePrivateExtensionPlanRequest(rollbackRequest, {
|
||||
now: "2026-07-15T18:05:01.000Z",
|
||||
grantedCapabilities: manage,
|
||||
}), { ok: true, errors: [] });
|
||||
|
||||
const rollbackPlan = withPlanHash({
|
||||
schemaVersion: ENGINE_PRIVATE_EXTENSION_PLAN_SCHEMA_VERSION,
|
||||
planId: "extension-rollback-plan-20260715-001",
|
||||
planHash: hash("0"),
|
||||
requestId: rollbackRequest.requestId,
|
||||
idempotencyKey: rollbackRequest.idempotencyKey,
|
||||
action: "rollback",
|
||||
createdAt: "2026-07-15T18:05:01.000Z",
|
||||
expiresAt: "2026-07-15T18:14:00.000Z",
|
||||
singleUse: true,
|
||||
requiredCapability: ENGINE_PRIVATE_EXTENSION_MANAGE_CAPABILITY,
|
||||
expectedCurrentGeneration: 1,
|
||||
nextGeneration: 2,
|
||||
currentState: release,
|
||||
targetState: inactive,
|
||||
recoveryState: release,
|
||||
actions: [
|
||||
"verify_previous_activation_state",
|
||||
"verify_community_package_loader_policy",
|
||||
"quiesce_deploy_run_and_drain_queue",
|
||||
"record_recovery_state",
|
||||
"atomic_switch_current_to_previous",
|
||||
"force_recreate_main_workers_webhooks_as_version_barrier",
|
||||
"verify_exact_runtime_acceptance",
|
||||
"commit_rolled_back_state",
|
||||
"resume_deploy_run",
|
||||
],
|
||||
transition: transition(),
|
||||
acceptance: acceptanceSpec(inactive),
|
||||
failurePolicy: failurePolicy(),
|
||||
});
|
||||
assert.deepEqual(validateEnginePrivateExtensionPlan(rollbackPlan, { request: rollbackRequest }), { ok: true, errors: [] });
|
||||
|
||||
const explicitRollbackOperation = {
|
||||
...activeOperation,
|
||||
operationId: "extension-operation-rollback-20260715-001",
|
||||
planId: rollbackPlan.planId,
|
||||
planHash: rollbackPlan.planHash,
|
||||
action: "rollback",
|
||||
state: "rolled-back",
|
||||
outcome: "explicitly-rolled-back",
|
||||
expectedCurrentGeneration: 1,
|
||||
nextGeneration: 2,
|
||||
targetState: inactive,
|
||||
recoveryState: release,
|
||||
effectiveState: inactive,
|
||||
runtime: runtime(2, 2, 2),
|
||||
acceptance: acceptanceReport(inactive, "accepted"),
|
||||
};
|
||||
assert.deepEqual(validateEnginePrivateExtensionOperation(explicitRollbackOperation), { ok: true, errors: [] });
|
||||
|
||||
const quarantinedStatus = {
|
||||
...activeStatus,
|
||||
health: "quarantined",
|
||||
activeOperationId: "extension-operation-failed-20260715-001",
|
||||
acceptance: acceptanceReport(release, "rejected"),
|
||||
errorCode: "automatic_rollback_failed",
|
||||
};
|
||||
assert.deepEqual(validateEnginePrivateExtensionStatus(quarantinedStatus), { ok: true, errors: [] });
|
||||
|
||||
const pathInjection = structuredClone(activateRequest);
|
||||
pathInjection.target.releaseId = "../../runtime";
|
||||
assert.equal(validateEnginePrivateExtensionPlanRequest(pathInjection, {
|
||||
now: "2026-07-15T18:00:01.000Z",
|
||||
grantedCapabilities: manage,
|
||||
}).errors.includes("target.releaseId_invalid"), true);
|
||||
|
||||
const commandInjection = { ...activateRequest, command: "docker exec runtime npm install" };
|
||||
assert.equal(validateEnginePrivateExtensionPlanRequest(commandInjection, {
|
||||
now: "2026-07-15T18:00:01.000Z",
|
||||
grantedCapabilities: manage,
|
||||
}).errors.includes("enginePrivateExtensionPlanRequest.command_not_allowed"), true);
|
||||
|
||||
console.log("external-provider Engine private-extension contract: ok");
|
||||
|
||||
function transition() {
|
||||
return {
|
||||
mountMode: "read-only",
|
||||
loaderMode: "community-package",
|
||||
loaderPath: "/home/node/.n8n/nodes/node_modules/n8n-nodes-ndc",
|
||||
loaderEnvironment: {
|
||||
N8N_COMMUNITY_PACKAGES_ENABLED: "true",
|
||||
N8N_COMMUNITY_PACKAGES_PREVENT_LOADING: "false",
|
||||
N8N_REINSTALL_MISSING_PACKAGES: "false",
|
||||
},
|
||||
quiesceMode: "block-deploy-run-and-drain-queue",
|
||||
stateSwitch: "atomic-current-previous",
|
||||
runtimeAction: "force-recreate",
|
||||
scope: "main-workers-webhooks",
|
||||
requireUniformGeneration: true,
|
||||
hotReload: false,
|
||||
preserveCredentials: true,
|
||||
};
|
||||
}
|
||||
|
||||
function failurePolicy() {
|
||||
return {
|
||||
mode: "automatic-rollback",
|
||||
rollbackFailureOutcome: "quarantined",
|
||||
preserveImmutableRelease: true,
|
||||
preserveCredentials: true,
|
||||
};
|
||||
}
|
||||
|
||||
function acceptanceSpec(state) {
|
||||
return {
|
||||
mode: "exact",
|
||||
nodeTypes: state.kind === "release" ? [...ENGINE_PRIVATE_EXTENSION_NODE_TYPES] : [],
|
||||
credentialSchemas: state.kind === "release" ? [...ENGINE_PRIVATE_EXTENSION_CREDENTIAL_TYPES] : [],
|
||||
requireUniformGeneration: true,
|
||||
};
|
||||
}
|
||||
|
||||
function acceptanceReport(state, status) {
|
||||
const nodes = state.kind === "release" ? [...ENGINE_PRIVATE_EXTENSION_NODE_TYPES] : [];
|
||||
const credentials = state.kind === "release" ? [...ENGINE_PRIVATE_EXTENSION_CREDENTIAL_TYPES] : [];
|
||||
return {
|
||||
state: status,
|
||||
nodeTypes: { expected: nodes, observed: status === "pending" ? [] : nodes },
|
||||
credentialSchemas: { expected: credentials, observed: status === "pending" ? [] : credentials },
|
||||
uniformGeneration: status === "accepted",
|
||||
};
|
||||
}
|
||||
|
||||
function runtime(generation, expectedInstances, readyInstances) {
|
||||
return { mode: "force-recreate", generation, expectedInstances, readyInstances };
|
||||
}
|
||||
|
||||
function withPlanHash(plan) {
|
||||
plan.planHash = computeEnginePrivateExtensionPlanHash(plan);
|
||||
return plan;
|
||||
}
|
||||
|
||||
function hash(character) {
|
||||
return "sha256:" + character.repeat(64);
|
||||
}
|
||||
|
|
@ -1,133 +0,0 @@
|
|||
# n8n-nodes-ndc
|
||||
|
||||
Private NODE.DC node package for provider-neutral L2 workflows. It extends n8n
|
||||
through the supported private-node mechanism and does not patch Engine or n8n
|
||||
core.
|
||||
|
||||
Every custom node has an `NDC ` display-name prefix. Runtime types are package
|
||||
qualified:
|
||||
|
||||
- `n8n-nodes-ndc.ndcDataProductPublish` — publishes canonical facts into an
|
||||
approved Data Product.
|
||||
- `n8n-nodes-ndc.ndcDataProductRead` — reads the current snapshot of an
|
||||
approved Data Product.
|
||||
- `n8n-nodes-ndc.ndcFoundryBinding` — creates or updates a declarative Foundry
|
||||
page-slot binding. This is a control-plane operation, not a runtime data
|
||||
transport.
|
||||
|
||||
## Security and scope
|
||||
|
||||
Workflow parameters contain no service URL, provider, tenant, connection, raw
|
||||
secret, or credential selector. Product selection is loaded from the catalog
|
||||
visible to the selected opaque capability. Service locations are operator-owned
|
||||
runtime overrides (the package defaults to the canonical internal service
|
||||
names, so the first deployment needs no extra Engine environment):
|
||||
|
||||
- `NDC_DATA_PLANE_BASE_URL`
|
||||
- `NDC_FOUNDRY_BASE_URL`
|
||||
|
||||
The defaults are `http://external-data-plane:18106` and
|
||||
`http://nodedc-module-foundry:3333`. They are provider-neutral Platform service
|
||||
addresses, not workflow configuration.
|
||||
|
||||
The three credential types contain only one password-protected opaque
|
||||
capability. Writer, reader, and Foundry capabilities are intentionally distinct.
|
||||
Provider identity, product version, ontology revision, persistence policy, and
|
||||
tenant scope are materialized by the receiving service from the grant. The node
|
||||
never accepts them from a workflow.
|
||||
|
||||
## Fixed routes
|
||||
|
||||
- `GET /internal/data-plane/v1/writer/data-products`
|
||||
- `GET /internal/data-plane/v1/reader/data-products`
|
||||
- `POST /internal/data-plane/v1/data-products/:dataProductId/publish`
|
||||
- `GET /internal/data-plane/v1/data-products/:dataProductId/snapshot`
|
||||
- `GET /internal/foundry/v1/data-products`
|
||||
- `POST /internal/foundry/v1/data-product-bindings`
|
||||
|
||||
There is no fallback to a legacy intake route. `NDC Foundry Binding` uses the
|
||||
dedicated internal control-plane endpoint implemented by Foundry and fails
|
||||
closed when that route or its scoped workload grant is unavailable. Its
|
||||
canonical Map entity-stream slot defaults to `points`.
|
||||
|
||||
The binding command uses the versioned
|
||||
`nodedc.foundry.binding-upsert/v1` schema. Its opaque `ndc_fndbg_*` workload
|
||||
grant is separate from the short-lived Foundry MCP capability used by AI
|
||||
Workspace and from the shared Platform service token. The receiving service
|
||||
materializes actor, owner and exact application/page/binding/product scope from
|
||||
that grant; none of those authorization claims are accepted from headers or
|
||||
workflow data.
|
||||
|
||||
`NDC Data Product Read` supports only the bounded
|
||||
`nodedc.data-product.snapshot/v1` contract. Its `Page Size` parameter is a
|
||||
safety ceiling, not pagination: the complete scoped current projection must fit
|
||||
within 5000 entity keys. A larger product returns
|
||||
`data_product_snapshot_limit_exceeded`; the node must not assemble independent
|
||||
pages or start a patch stream from an incomplete snapshot. Large products need
|
||||
stable partition Data Products or a separately versioned query contract with a
|
||||
single snapshot barrier.
|
||||
|
||||
## Publish input
|
||||
|
||||
Each incoming item must be either a canonical fact or `{ "fact": <fact> }`:
|
||||
|
||||
```json
|
||||
{
|
||||
"sourceId": "fleet.unit.42",
|
||||
"semanticType": "map.moving_object",
|
||||
"observedAt": "2026-07-15T12:00:00.000Z",
|
||||
"attributes": {},
|
||||
"geometry": { "type": "Point", "coordinates": [37.61, 55.75] }
|
||||
}
|
||||
```
|
||||
|
||||
The node emits `nodedc.data-product.publish/v1`, enforces the 5000-fact and
|
||||
64-KiB-per-attributes limits, rejects secret-like material in attributes, and
|
||||
derives stable run/idempotency identifiers from workflow execution context.
|
||||
|
||||
## Build and verification
|
||||
|
||||
```sh
|
||||
npm install
|
||||
npm test
|
||||
npm run lint
|
||||
```
|
||||
|
||||
The package must be installed as an n8n private community package under the
|
||||
runtime user's community-package root:
|
||||
|
||||
```text
|
||||
/home/node/.n8n/nodes/node_modules/n8n-nodes-ndc
|
||||
```
|
||||
|
||||
The parent `/home/node/.n8n/nodes/package.json` owns the pinned package
|
||||
registration. `~/.n8n/custom` and `N8N_CUSTOM_EXTENSIONS` are deliberately not
|
||||
used: n8n loads those through the `CUSTOM` namespace, which would destroy the
|
||||
required `n8n-nodes-ndc.*` runtime types.
|
||||
|
||||
## Deployment boundary
|
||||
|
||||
The Platform MCP catalog bridge preserves package-qualified custom runtime
|
||||
types, but Engine can expose these schemas only after this private package is
|
||||
installed through the community-package path and the bridge slice is deployed.
|
||||
Production installation uses a verified offline tarball, an immutable
|
||||
root-owned release, an atomic current/previous switch and a read-only mount; it
|
||||
does not run `npm install` inside a live container. This package deliberately
|
||||
does not patch Engine or n8n core.
|
||||
|
||||
The Platform-side release builder is:
|
||||
|
||||
```sh
|
||||
node ../../infra/deploy-runner/build-n8n-private-extension-artifact.mjs \
|
||||
n8n-nodes-ndc-release-YYYYMMDD-NNN
|
||||
```
|
||||
|
||||
Its canonical deploy component only stages and seals a digest-bound release
|
||||
under `/volume1/docker/nodedc-platform/n8n-private-extensions`; it does not
|
||||
activate anything in Engine. Activation is intentionally blocked until an
|
||||
Engine-owned deployment slice provides the exact read-only community-package
|
||||
mount, atomic release selection, all-process restart and MCP schema acceptance.
|
||||
The v2 rollback contract accepts a pre-activation, explicitly verified inactive
|
||||
baseline for the first activation; subsequent upgrades prefer a previous
|
||||
verified immutable release. The staged v1 `0.1.0` release is never overwritten:
|
||||
this contract is published as a separate `0.1.1` release.
|
||||
|
|
@ -1,34 +0,0 @@
|
|||
import type {
|
||||
IAuthenticateGeneric,
|
||||
Icon,
|
||||
ICredentialType,
|
||||
INodeProperties,
|
||||
} from 'n8n-workflow';
|
||||
|
||||
export class NdcDataProductReaderApi implements ICredentialType {
|
||||
name = 'ndcDataProductReaderApi';
|
||||
displayName = 'NDC Data Product Reader API';
|
||||
icon: Icon = {
|
||||
light: 'file:../icons/ndc.svg',
|
||||
dark: 'file:../icons/ndc.dark.svg',
|
||||
};
|
||||
documentationUrl = '';
|
||||
properties: INodeProperties[] = [
|
||||
{
|
||||
displayName: 'Opaque Reader Capability',
|
||||
name: 'capability',
|
||||
type: 'string',
|
||||
typeOptions: { password: true },
|
||||
default: '',
|
||||
required: true,
|
||||
},
|
||||
];
|
||||
authenticate: IAuthenticateGeneric = {
|
||||
type: 'generic',
|
||||
properties: {
|
||||
headers: {
|
||||
Authorization: '=Bearer {{$credentials.capability}}',
|
||||
},
|
||||
},
|
||||
};
|
||||
}
|
||||
|
|
@ -1,34 +0,0 @@
|
|||
import type {
|
||||
IAuthenticateGeneric,
|
||||
Icon,
|
||||
ICredentialType,
|
||||
INodeProperties,
|
||||
} from 'n8n-workflow';
|
||||
|
||||
export class NdcDataProductWriterApi implements ICredentialType {
|
||||
name = 'ndcDataProductWriterApi';
|
||||
displayName = 'NDC Data Product Writer API';
|
||||
icon: Icon = {
|
||||
light: 'file:../icons/ndc.svg',
|
||||
dark: 'file:../icons/ndc.dark.svg',
|
||||
};
|
||||
documentationUrl = '';
|
||||
properties: INodeProperties[] = [
|
||||
{
|
||||
displayName: 'Opaque Writer Capability',
|
||||
name: 'capability',
|
||||
type: 'string',
|
||||
typeOptions: { password: true },
|
||||
default: '',
|
||||
required: true,
|
||||
},
|
||||
];
|
||||
authenticate: IAuthenticateGeneric = {
|
||||
type: 'generic',
|
||||
properties: {
|
||||
headers: {
|
||||
Authorization: '=Bearer {{$credentials.capability}}',
|
||||
},
|
||||
},
|
||||
};
|
||||
}
|
||||
|
|
@ -1,34 +0,0 @@
|
|||
import type {
|
||||
IAuthenticateGeneric,
|
||||
Icon,
|
||||
ICredentialType,
|
||||
INodeProperties,
|
||||
} from 'n8n-workflow';
|
||||
|
||||
export class NdcFoundryBindingApi implements ICredentialType {
|
||||
name = 'ndcFoundryBindingApi';
|
||||
displayName = 'NDC Foundry Binding API';
|
||||
icon: Icon = {
|
||||
light: 'file:../icons/ndc.svg',
|
||||
dark: 'file:../icons/ndc.dark.svg',
|
||||
};
|
||||
documentationUrl = '';
|
||||
properties: INodeProperties[] = [
|
||||
{
|
||||
displayName: 'Opaque Binding Capability',
|
||||
name: 'capability',
|
||||
type: 'string',
|
||||
typeOptions: { password: true },
|
||||
default: '',
|
||||
required: true,
|
||||
},
|
||||
];
|
||||
authenticate: IAuthenticateGeneric = {
|
||||
type: 'generic',
|
||||
properties: {
|
||||
headers: {
|
||||
Authorization: '=Bearer {{$credentials.capability}}',
|
||||
},
|
||||
},
|
||||
};
|
||||
}
|
||||
|
|
@ -1,45 +0,0 @@
|
|||
import { configWithoutCloudSupport } from '@n8n/node-cli/eslint';
|
||||
|
||||
export default [
|
||||
...configWithoutCloudSupport,
|
||||
{
|
||||
files: ['package.json'],
|
||||
rules: {
|
||||
// This is a private, unlicensed Platform extension, not a community package.
|
||||
'@n8n/community-nodes/valid-author': 'off',
|
||||
'@n8n/community-nodes/require-mit-license': 'off',
|
||||
// Keep the tested n8n 2.x runtime boundary instead of the community
|
||||
// marketplace's mandatory wildcard peer range.
|
||||
'@n8n/community-nodes/valid-peer-dependencies': 'off',
|
||||
'n8n-nodes-base/community-package-json-author-missing': 'off',
|
||||
'n8n-nodes-base/community-package-json-license-not-default': 'off',
|
||||
},
|
||||
},
|
||||
{
|
||||
files: ['./credentials/**/*.ts'],
|
||||
rules: {
|
||||
// Internal capability checks need operator-owned URLs unavailable to a
|
||||
// static credential test. The nodes test them through scoped catalogs.
|
||||
'@n8n/community-nodes/credential-test-required': 'off',
|
||||
'@n8n/community-nodes/credential-documentation-url': 'off',
|
||||
'n8n-nodes-base/cred-class-field-documentation-url-not-http-url': 'off',
|
||||
},
|
||||
},
|
||||
{
|
||||
files: ['./nodes/**/*.ts'],
|
||||
rules: {
|
||||
// Writes are deliberately fail-closed. continueOnFail would disguise a
|
||||
// rejected publish/binding as a successful control-plane operation.
|
||||
'@n8n/community-nodes/require-continue-on-fail': 'off',
|
||||
// The Engine schema contract exposes exactly three package-qualified
|
||||
// workflow nodes. Marking them as tools makes n8n 2.3.2 synthesize three
|
||||
// extra *Tool runtime types and breaks that exact catalog boundary.
|
||||
'@n8n/community-nodes/node-usable-as-tool': 'off',
|
||||
// The stock dynamic-options copy links to vendor documentation in the
|
||||
// product UI. NDC nodes keep that technical runtime brand out of every
|
||||
// user-visible label and description.
|
||||
'n8n-nodes-base/node-param-description-missing-from-dynamic-options': 'off',
|
||||
'n8n-nodes-base/node-param-description-wrong-for-dynamic-options': 'off',
|
||||
},
|
||||
},
|
||||
];
|
||||
|
|
@ -1,6 +0,0 @@
|
|||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 96 96" role="img" aria-label="NDC">
|
||||
<g transform="translate(6 23.5) scale(.9)" fill="#a98aff">
|
||||
<path d="M52.8 23.61 46.92 33.76 41.05 23.61H52.8m18-10.39H23.06l23.86 41.33Z"/>
|
||||
<path d="M31.28 33.13 18.11 10.34 75.73 10.34 62.59 33.13 74.28 33.13 93.22 0 0 0 19.61 33.13 31.28 33.13"/>
|
||||
</g>
|
||||
</svg>
|
||||
|
Before Width: | Height: | Size: 362 B |
|
|
@ -1,6 +0,0 @@
|
|||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 96 96" role="img" aria-label="NDC">
|
||||
<g transform="translate(6 23.5) scale(.9)" fill="#6f50bd">
|
||||
<path d="M52.8 23.61 46.92 33.76 41.05 23.61H52.8m18-10.39H23.06l23.86 41.33Z"/>
|
||||
<path d="M31.28 33.13 18.11 10.34 75.73 10.34 62.59 33.13 74.28 33.13 93.22 0 0 0 19.61 33.13 31.28 33.13"/>
|
||||
</g>
|
||||
</svg>
|
||||
|
Before Width: | Height: | Size: 362 B |
|
|
@ -1,105 +0,0 @@
|
|||
import type {
|
||||
IExecuteFunctions,
|
||||
ILoadOptionsFunctions,
|
||||
INodeExecutionData,
|
||||
INodeType,
|
||||
INodeTypeDescription,
|
||||
} from 'n8n-workflow';
|
||||
import { NodeConnectionTypes, NodeOperationError } from 'n8n-workflow';
|
||||
|
||||
import {
|
||||
DATA_PRODUCT_BASE_PATH,
|
||||
DATA_PLANE_DEFAULT_BASE_URL,
|
||||
DATA_PLANE_BASE_URL_ENV,
|
||||
DATA_PRODUCT_WRITER_CATALOG_PATH,
|
||||
NDC_DATA_PRODUCT_WRITER_CREDENTIAL,
|
||||
NDC_NODE_ICON,
|
||||
} from '../shared/constants';
|
||||
import { buildPublishPayload, encodeIdentifierPath } from '../shared/contracts';
|
||||
import { safeHttpError, safeInputError } from '../shared/errors';
|
||||
import { loadDataProductOptions, ndcRequest, serviceBaseUrl } from '../shared/http';
|
||||
|
||||
export class NdcDataProductPublish implements INodeType {
|
||||
description: INodeTypeDescription = {
|
||||
displayName: 'NDC Data Product Publish',
|
||||
name: 'ndcDataProductPublish',
|
||||
icon: NDC_NODE_ICON,
|
||||
group: ['output'],
|
||||
version: 1,
|
||||
subtitle: '={{$parameter["dataProductId"]}}',
|
||||
description: 'Publish canonical facts through a scoped NDC Data Product grant',
|
||||
defaults: { name: 'NDC Data Product Publish' },
|
||||
inputs: [NodeConnectionTypes.Main],
|
||||
outputs: [NodeConnectionTypes.Main],
|
||||
credentials: [{ name: NDC_DATA_PRODUCT_WRITER_CREDENTIAL, required: true }],
|
||||
properties: [
|
||||
{
|
||||
displayName: 'Data Product Name or ID',
|
||||
name: 'dataProductId',
|
||||
type: 'options',
|
||||
typeOptions: { loadOptionsMethod: 'getDataProducts' },
|
||||
default: '',
|
||||
required: true,
|
||||
},
|
||||
{
|
||||
displayName: 'Batch Sequence',
|
||||
name: 'sequence',
|
||||
type: 'number',
|
||||
typeOptions: { minValue: 0, numberStepSize: 1 },
|
||||
default: 0,
|
||||
required: true,
|
||||
description: 'Sequence number when one execution publishes several batches',
|
||||
},
|
||||
],
|
||||
};
|
||||
|
||||
methods = {
|
||||
loadOptions: {
|
||||
async getDataProducts(this: ILoadOptionsFunctions) {
|
||||
return loadDataProductOptions(
|
||||
this,
|
||||
NDC_DATA_PRODUCT_WRITER_CREDENTIAL,
|
||||
DATA_PLANE_BASE_URL_ENV,
|
||||
DATA_PLANE_DEFAULT_BASE_URL,
|
||||
DATA_PRODUCT_WRITER_CATALOG_PATH,
|
||||
);
|
||||
},
|
||||
},
|
||||
};
|
||||
|
||||
async execute(this: IExecuteFunctions): Promise<INodeExecutionData[][]> {
|
||||
const inputItems = this.getInputData();
|
||||
const dataProductId = this.getNodeParameter('dataProductId', 0) as string;
|
||||
const sequence = this.getNodeParameter('sequence', 0, 0) as number;
|
||||
let body;
|
||||
let encodedProductId;
|
||||
try {
|
||||
body = buildPublishPayload(
|
||||
inputItems,
|
||||
this.getExecutionId(),
|
||||
String(this.getWorkflow().id ?? 'workflow'),
|
||||
this.getNode().id,
|
||||
dataProductId,
|
||||
sequence,
|
||||
);
|
||||
encodedProductId = encodeIdentifierPath(dataProductId, 'dataProductId');
|
||||
} catch (error) {
|
||||
throw new NodeOperationError(this.getNode(), safeInputError(error, 'ndc_data_product_publish_input_invalid'));
|
||||
}
|
||||
|
||||
try {
|
||||
const baseUrl = serviceBaseUrl(DATA_PLANE_BASE_URL_ENV, DATA_PLANE_DEFAULT_BASE_URL);
|
||||
const response = await ndcRequest(this, NDC_DATA_PRODUCT_WRITER_CREDENTIAL, {
|
||||
method: 'POST',
|
||||
url: `${baseUrl}${DATA_PRODUCT_BASE_PATH}/${encodedProductId}/publish`,
|
||||
body,
|
||||
});
|
||||
return [[{
|
||||
json: response as INodeExecutionData['json'],
|
||||
pairedItem: inputItems.map((_item, index) => ({ item: index })),
|
||||
}]];
|
||||
} catch (error) {
|
||||
throw new NodeOperationError(this.getNode(), safeHttpError(error, 'ndc_data_product_publish_failed'));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -1,99 +0,0 @@
|
|||
import type {
|
||||
IExecuteFunctions,
|
||||
ILoadOptionsFunctions,
|
||||
INodeExecutionData,
|
||||
INodeType,
|
||||
INodeTypeDescription,
|
||||
} from 'n8n-workflow';
|
||||
import { NodeConnectionTypes, NodeOperationError } from 'n8n-workflow';
|
||||
|
||||
import {
|
||||
DATA_PRODUCT_BASE_PATH,
|
||||
DATA_PRODUCT_READER_CATALOG_PATH,
|
||||
DATA_PRODUCT_SNAPSHOT_SUFFIX,
|
||||
DATA_PLANE_DEFAULT_BASE_URL,
|
||||
DATA_PLANE_BASE_URL_ENV,
|
||||
NDC_DATA_PRODUCT_READER_CREDENTIAL,
|
||||
NDC_NODE_ICON,
|
||||
} from '../shared/constants';
|
||||
import { normalizeSnapshotFacts, snapshotReadParameters } from '../shared/contracts';
|
||||
import { safeHttpError, safeInputError } from '../shared/errors';
|
||||
import { loadDataProductOptions, ndcRequest, serviceBaseUrl } from '../shared/http';
|
||||
|
||||
export class NdcDataProductRead implements INodeType {
|
||||
description: INodeTypeDescription = {
|
||||
displayName: 'NDC Data Product Read',
|
||||
name: 'ndcDataProductRead',
|
||||
icon: NDC_NODE_ICON,
|
||||
group: ['input'],
|
||||
version: 1,
|
||||
subtitle: '={{$parameter["dataProductId"]}}',
|
||||
description: 'Read a scoped NDC Data Product snapshot',
|
||||
defaults: { name: 'NDC Data Product Read' },
|
||||
inputs: [NodeConnectionTypes.Main],
|
||||
outputs: [NodeConnectionTypes.Main],
|
||||
credentials: [{ name: NDC_DATA_PRODUCT_READER_CREDENTIAL, required: true }],
|
||||
properties: [
|
||||
{
|
||||
displayName: 'Data Product Name or ID',
|
||||
name: 'dataProductId',
|
||||
type: 'options',
|
||||
typeOptions: { loadOptionsMethod: 'getDataProducts' },
|
||||
default: '',
|
||||
required: true,
|
||||
},
|
||||
{
|
||||
displayName: 'Page Size',
|
||||
name: 'pageSize',
|
||||
type: 'number',
|
||||
typeOptions: { minValue: 1, maxValue: 5000, numberStepSize: 1 },
|
||||
default: 1000,
|
||||
required: true,
|
||||
description: 'Maximum number of snapshot facts to return',
|
||||
},
|
||||
],
|
||||
};
|
||||
|
||||
methods = {
|
||||
loadOptions: {
|
||||
async getDataProducts(this: ILoadOptionsFunctions) {
|
||||
return loadDataProductOptions(
|
||||
this,
|
||||
NDC_DATA_PRODUCT_READER_CREDENTIAL,
|
||||
DATA_PLANE_BASE_URL_ENV,
|
||||
DATA_PLANE_DEFAULT_BASE_URL,
|
||||
DATA_PRODUCT_READER_CATALOG_PATH,
|
||||
);
|
||||
},
|
||||
},
|
||||
};
|
||||
|
||||
async execute(this: IExecuteFunctions): Promise<INodeExecutionData[][]> {
|
||||
const dataProductId = this.getNodeParameter('dataProductId', 0) as string;
|
||||
const pageSize = this.getNodeParameter('pageSize', 0, 1000) as number;
|
||||
let parameters;
|
||||
try {
|
||||
parameters = snapshotReadParameters(dataProductId, pageSize);
|
||||
} catch (error) {
|
||||
throw new NodeOperationError(this.getNode(), safeInputError(error, 'ndc_data_product_read_input_invalid'));
|
||||
}
|
||||
|
||||
try {
|
||||
const baseUrl = serviceBaseUrl(DATA_PLANE_BASE_URL_ENV, DATA_PLANE_DEFAULT_BASE_URL);
|
||||
const response = await ndcRequest(this, NDC_DATA_PRODUCT_READER_CREDENTIAL, {
|
||||
method: 'GET',
|
||||
url: `${baseUrl}${DATA_PRODUCT_BASE_PATH}/${parameters.encodedProductId}${DATA_PRODUCT_SNAPSHOT_SUFFIX}`,
|
||||
qs: {
|
||||
limit: parameters.pageSize,
|
||||
},
|
||||
});
|
||||
const facts = normalizeSnapshotFacts(response);
|
||||
return [facts.map((fact) => ({ json: fact, pairedItem: { item: 0 } }))];
|
||||
} catch (error) {
|
||||
const safe = error instanceof Error && error.message === 'data_product_snapshot_invalid'
|
||||
? safeInputError(error, 'ndc_data_product_snapshot_invalid')
|
||||
: safeHttpError(error, 'ndc_data_product_read_failed');
|
||||
throw new NodeOperationError(this.getNode(), safe);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -1,140 +0,0 @@
|
|||
import type {
|
||||
IExecuteFunctions,
|
||||
ILoadOptionsFunctions,
|
||||
INodeExecutionData,
|
||||
INodeType,
|
||||
INodeTypeDescription,
|
||||
} from 'n8n-workflow';
|
||||
import { NodeConnectionTypes, NodeOperationError } from 'n8n-workflow';
|
||||
|
||||
import {
|
||||
FOUNDRY_BINDING_PATH,
|
||||
FOUNDRY_BASE_URL_ENV,
|
||||
FOUNDRY_CATALOG_PATH,
|
||||
FOUNDRY_DEFAULT_BASE_URL,
|
||||
NDC_FOUNDRY_BINDING_CREDENTIAL,
|
||||
NDC_NODE_ICON,
|
||||
} from '../shared/constants';
|
||||
import { buildFoundryBindingPayload } from '../shared/contracts';
|
||||
import { safeHttpError, safeInputError } from '../shared/errors';
|
||||
import { loadDataProductOptions, ndcRequest, serviceBaseUrl } from '../shared/http';
|
||||
|
||||
export class NdcFoundryBinding implements INodeType {
|
||||
description: INodeTypeDescription = {
|
||||
displayName: 'NDC Foundry Binding',
|
||||
name: 'ndcFoundryBinding',
|
||||
icon: NDC_NODE_ICON,
|
||||
group: ['output'],
|
||||
version: 1,
|
||||
subtitle: '={{$parameter["dataProductId"]}}',
|
||||
description: 'Bind an approved Data Product to a Foundry page slot through the control plane',
|
||||
defaults: { name: 'NDC Foundry Binding' },
|
||||
inputs: [NodeConnectionTypes.Main],
|
||||
outputs: [NodeConnectionTypes.Main],
|
||||
credentials: [{ name: NDC_FOUNDRY_BINDING_CREDENTIAL, required: true }],
|
||||
properties: [
|
||||
{
|
||||
displayName: 'Uses the dedicated Foundry control-plane endpoint and a separate scoped workload grant. It fails closed if either is unavailable and never sends runtime facts or provider credentials to Foundry.',
|
||||
name: 'availabilityNotice',
|
||||
type: 'notice',
|
||||
default: '',
|
||||
},
|
||||
{
|
||||
displayName: 'Application ID',
|
||||
name: 'applicationId',
|
||||
type: 'string',
|
||||
default: '',
|
||||
required: true,
|
||||
},
|
||||
{
|
||||
displayName: 'Page ID',
|
||||
name: 'pageId',
|
||||
type: 'string',
|
||||
default: '',
|
||||
required: true,
|
||||
},
|
||||
{
|
||||
displayName: 'Binding ID',
|
||||
name: 'bindingId',
|
||||
type: 'string',
|
||||
default: '',
|
||||
required: true,
|
||||
},
|
||||
{
|
||||
displayName: 'Data Product Name or ID',
|
||||
name: 'dataProductId',
|
||||
type: 'options',
|
||||
typeOptions: { loadOptionsMethod: 'getDataProducts' },
|
||||
default: '',
|
||||
required: true,
|
||||
},
|
||||
{
|
||||
displayName: 'Slot ID',
|
||||
name: 'slotId',
|
||||
type: 'string',
|
||||
default: 'points',
|
||||
required: true,
|
||||
},
|
||||
{
|
||||
displayName: 'Semantic Types',
|
||||
name: 'semanticTypes',
|
||||
type: 'string',
|
||||
typeOptions: { multipleValues: true, multipleValueButtonText: 'Add Semantic Type' },
|
||||
default: [],
|
||||
required: true,
|
||||
},
|
||||
{
|
||||
displayName: 'Field Projection',
|
||||
name: 'fieldProjection',
|
||||
type: 'string',
|
||||
typeOptions: { multipleValues: true, multipleValueButtonText: 'Add Field' },
|
||||
default: [],
|
||||
},
|
||||
],
|
||||
};
|
||||
|
||||
methods = {
|
||||
loadOptions: {
|
||||
async getDataProducts(this: ILoadOptionsFunctions) {
|
||||
return loadDataProductOptions(
|
||||
this,
|
||||
NDC_FOUNDRY_BINDING_CREDENTIAL,
|
||||
FOUNDRY_BASE_URL_ENV,
|
||||
FOUNDRY_DEFAULT_BASE_URL,
|
||||
FOUNDRY_CATALOG_PATH,
|
||||
);
|
||||
},
|
||||
},
|
||||
};
|
||||
|
||||
async execute(this: IExecuteFunctions): Promise<INodeExecutionData[][]> {
|
||||
let body;
|
||||
try {
|
||||
body = buildFoundryBindingPayload({
|
||||
applicationId: this.getNodeParameter('applicationId', 0) as string,
|
||||
pageId: this.getNodeParameter('pageId', 0) as string,
|
||||
binding: {
|
||||
id: this.getNodeParameter('bindingId', 0) as string,
|
||||
dataProductId: this.getNodeParameter('dataProductId', 0) as string,
|
||||
slotId: this.getNodeParameter('slotId', 0) as string,
|
||||
semanticTypes: this.getNodeParameter('semanticTypes', 0, []) as string[],
|
||||
fieldProjection: this.getNodeParameter('fieldProjection', 0, []) as string[],
|
||||
},
|
||||
});
|
||||
} catch (error) {
|
||||
throw new NodeOperationError(this.getNode(), safeInputError(error, 'ndc_foundry_binding_input_invalid'));
|
||||
}
|
||||
|
||||
try {
|
||||
const baseUrl = serviceBaseUrl(FOUNDRY_BASE_URL_ENV, FOUNDRY_DEFAULT_BASE_URL);
|
||||
const response = await ndcRequest(this, NDC_FOUNDRY_BINDING_CREDENTIAL, {
|
||||
method: 'POST',
|
||||
url: `${baseUrl}${FOUNDRY_BINDING_PATH}`,
|
||||
body,
|
||||
});
|
||||
return [[{ json: response as INodeExecutionData['json'], pairedItem: { item: 0 } }]];
|
||||
} catch (error) {
|
||||
throw new NodeOperationError(this.getNode(), safeHttpError(error, 'ndc_foundry_binding_unavailable'));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -1,23 +0,0 @@
|
|||
export const DATA_PRODUCT_PUBLISH_SCHEMA_VERSION = 'nodedc.data-product.publish/v1';
|
||||
export const FOUNDRY_BINDING_UPSERT_SCHEMA_VERSION = 'nodedc.foundry.binding-upsert/v1';
|
||||
|
||||
export const DATA_PRODUCT_WRITER_CATALOG_PATH = '/internal/data-plane/v1/writer/data-products';
|
||||
export const DATA_PRODUCT_READER_CATALOG_PATH = '/internal/data-plane/v1/reader/data-products';
|
||||
export const DATA_PRODUCT_BASE_PATH = '/internal/data-plane/v1/data-products';
|
||||
export const DATA_PRODUCT_SNAPSHOT_SUFFIX = '/snapshot';
|
||||
export const FOUNDRY_BINDING_PATH = '/internal/foundry/v1/data-product-bindings';
|
||||
export const FOUNDRY_CATALOG_PATH = '/internal/foundry/v1/data-products';
|
||||
|
||||
export const DATA_PLANE_BASE_URL_ENV = 'NDC_DATA_PLANE_BASE_URL';
|
||||
export const FOUNDRY_BASE_URL_ENV = 'NDC_FOUNDRY_BASE_URL';
|
||||
export const DATA_PLANE_DEFAULT_BASE_URL = 'http://external-data-plane:18106';
|
||||
export const FOUNDRY_DEFAULT_BASE_URL = 'http://nodedc-module-foundry:3333';
|
||||
|
||||
export const NDC_DATA_PRODUCT_WRITER_CREDENTIAL = 'ndcDataProductWriterApi';
|
||||
export const NDC_DATA_PRODUCT_READER_CREDENTIAL = 'ndcDataProductReaderApi';
|
||||
export const NDC_FOUNDRY_BINDING_CREDENTIAL = 'ndcFoundryBindingApi';
|
||||
|
||||
export const NDC_NODE_ICON = {
|
||||
light: 'file:../../icons/ndc.svg',
|
||||
dark: 'file:../../icons/ndc.dark.svg',
|
||||
} as const;
|
||||
|
|
@ -1,238 +0,0 @@
|
|||
import { createHash } from 'node:crypto';
|
||||
|
||||
import type { IDataObject, INodeExecutionData, INodeListSearchItems } from 'n8n-workflow';
|
||||
|
||||
import {
|
||||
DATA_PRODUCT_PUBLISH_SCHEMA_VERSION,
|
||||
FOUNDRY_BINDING_UPSERT_SCHEMA_VERSION,
|
||||
} from './constants';
|
||||
|
||||
const IDENTIFIER = /^[a-z][a-z0-9._:-]{2,127}$/;
|
||||
const MAX_FACTS = 5000;
|
||||
const MAX_ATTRIBUTES_BYTES = 64 * 1024;
|
||||
const SECRET_LIKE_KEY = /(token|secret|password|authorization|access[_-]?token|refresh[_-]?token|api[_-]?key)/i;
|
||||
const SECRET_LIKE_VALUE = /(?:ndc_edp(?:wb|rb)_[A-Za-z0-9_-]*|(?:bearer|basic)\s+\S+|eyJ[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+)/i;
|
||||
|
||||
export interface NdcFact extends IDataObject {
|
||||
sourceId: string;
|
||||
semanticType: string;
|
||||
observedAt: string;
|
||||
attributes?: IDataObject;
|
||||
geometry?: {
|
||||
type: 'Point';
|
||||
coordinates: [number, number];
|
||||
};
|
||||
}
|
||||
|
||||
export interface NdcPublishPayload extends IDataObject {
|
||||
schemaVersion: typeof DATA_PRODUCT_PUBLISH_SCHEMA_VERSION;
|
||||
batch: {
|
||||
runId: string;
|
||||
sequence: number;
|
||||
idempotencyKey: string;
|
||||
};
|
||||
facts: NdcFact[];
|
||||
}
|
||||
|
||||
export interface NdcFoundryBindingInput extends IDataObject {
|
||||
applicationId: string;
|
||||
pageId: string;
|
||||
binding: {
|
||||
id: string;
|
||||
dataProductId: string;
|
||||
slotId: string;
|
||||
semanticTypes: string[];
|
||||
fieldProjection: string[];
|
||||
};
|
||||
}
|
||||
|
||||
export function normalizeBaseUrl(value: unknown): string {
|
||||
const normalized = String(value ?? '').trim().replace(/\/+$/, '');
|
||||
if (!/^https?:\/\/[^\s]+$/i.test(normalized)) throw new Error('ndc_internal_api_base_url_invalid');
|
||||
return normalized;
|
||||
}
|
||||
|
||||
export function requireIdentifier(value: unknown, field: string): string {
|
||||
const normalized = String(value ?? '').trim();
|
||||
if (!IDENTIFIER.test(normalized)) throw new Error(`${field}_invalid`);
|
||||
return normalized;
|
||||
}
|
||||
|
||||
export function encodeIdentifierPath(value: unknown, field: string): string {
|
||||
return encodeURIComponent(requireIdentifier(value, field));
|
||||
}
|
||||
|
||||
export function factsFromItems(items: INodeExecutionData[]): NdcFact[] {
|
||||
if (!items.length) throw new Error('facts_required');
|
||||
if (items.length > MAX_FACTS) throw new Error('facts_limit_exceeded');
|
||||
const entityKeys = new Set<string>();
|
||||
return items.map((item, index) => {
|
||||
const source = isObject(item.json.fact) ? item.json.fact : item.json;
|
||||
const fact: NdcFact = {
|
||||
sourceId: requireIdentifier(source.sourceId, `facts_${index}_sourceId`),
|
||||
semanticType: requireIdentifier(source.semanticType, `facts_${index}_semanticType`),
|
||||
observedAt: requireIsoTimestamp(source.observedAt, `facts_${index}_observedAt`),
|
||||
};
|
||||
const entityKey = `${fact.sourceId}\u0000${fact.semanticType}`;
|
||||
if (entityKeys.has(entityKey)) throw new Error('facts_duplicate_entity_key');
|
||||
entityKeys.add(entityKey);
|
||||
if (source.attributes !== undefined) {
|
||||
if (!isObject(source.attributes)) throw new Error(`facts_${index}_attributes_invalid`);
|
||||
if (Buffer.byteLength(JSON.stringify(source.attributes)) > MAX_ATTRIBUTES_BYTES) {
|
||||
throw new Error(`facts_${index}_attributes_size_exceeded`);
|
||||
}
|
||||
if (containsSecretLikeMaterial(source.attributes)) throw new Error(`facts_${index}_attributes_secret_material_forbidden`);
|
||||
fact.attributes = source.attributes;
|
||||
}
|
||||
if (source.geometry !== undefined) fact.geometry = normalizePoint(source.geometry, index);
|
||||
return fact;
|
||||
});
|
||||
}
|
||||
|
||||
export function buildPublishPayload(
|
||||
items: INodeExecutionData[],
|
||||
executionId: string,
|
||||
workflowId: string,
|
||||
nodeId: string,
|
||||
dataProductId: string,
|
||||
sequence: number,
|
||||
): NdcPublishPayload {
|
||||
if (!Number.isInteger(sequence) || sequence < 0 || sequence > 2_147_483_647) throw new Error('batch_sequence_invalid');
|
||||
requireIdentifier(dataProductId, 'dataProductId');
|
||||
const runId = `run-${sha256(`${workflowId}|${executionId}`).slice(0, 48)}`;
|
||||
const idempotencyKey = `publish-${sha256(
|
||||
`${workflowId}|${executionId}|${nodeId}|${dataProductId}|${sequence}`,
|
||||
)}`;
|
||||
return {
|
||||
schemaVersion: DATA_PRODUCT_PUBLISH_SCHEMA_VERSION,
|
||||
batch: { runId, sequence, idempotencyKey },
|
||||
facts: factsFromItems(items),
|
||||
};
|
||||
}
|
||||
|
||||
export function normalizeSnapshotFacts(value: unknown): IDataObject[] {
|
||||
if (!isObject(value) || !Array.isArray(value.facts)) throw new Error('data_product_snapshot_invalid');
|
||||
return value.facts.map((fact, index) => {
|
||||
if (!isObject(fact)) throw new Error(`snapshot_fact_${index}_invalid`);
|
||||
return fact;
|
||||
});
|
||||
}
|
||||
|
||||
export function snapshotReadParameters(
|
||||
dataProductId: unknown,
|
||||
pageSize: unknown,
|
||||
): { encodedProductId: string; pageSize: number } {
|
||||
const normalizedPageSize = Number(pageSize);
|
||||
if (!Number.isInteger(normalizedPageSize) || normalizedPageSize < 1 || normalizedPageSize > 5000) {
|
||||
throw new Error('page_size_invalid');
|
||||
}
|
||||
return {
|
||||
encodedProductId: encodeIdentifierPath(dataProductId, 'dataProductId'),
|
||||
pageSize: normalizedPageSize,
|
||||
};
|
||||
}
|
||||
|
||||
export function dataProductOptions(value: unknown): INodeListSearchItems[] {
|
||||
if (!isObject(value)) throw new Error('data_product_catalog_invalid');
|
||||
const rows = Array.isArray(value.dataProducts)
|
||||
? value.dataProducts
|
||||
: Array.isArray(value.products)
|
||||
? value.products
|
||||
: [];
|
||||
return rows.flatMap((row) => {
|
||||
if (typeof row === 'string') {
|
||||
return IDENTIFIER.test(row) ? [{ name: row, value: row }] : [];
|
||||
}
|
||||
if (!isObject(row)) return [];
|
||||
const id = String(row.id ?? row.dataProductId ?? '').trim();
|
||||
if (!IDENTIFIER.test(id)) return [];
|
||||
const version = String(row.version ?? '').trim();
|
||||
const label = String(row.label ?? row.name ?? id).trim() || id;
|
||||
return [{ name: version ? `${label} · ${version}` : label, value: id }];
|
||||
});
|
||||
}
|
||||
|
||||
export function buildFoundryBindingPayload(input: NdcFoundryBindingInput): IDataObject {
|
||||
const applicationId = requireFoundryApplicationId(input.applicationId);
|
||||
const pageId = requireFoundryPageId(input.pageId);
|
||||
const bindingId = requireIdentifier(input.binding.id, 'binding_id');
|
||||
const dataProductId = requireIdentifier(input.binding.dataProductId, 'dataProductId');
|
||||
const slotId = requireFoundrySlotId(input.binding.slotId);
|
||||
const semanticTypes = uniqueIdentifiers(input.binding.semanticTypes, 'semanticTypes');
|
||||
if (!semanticTypes.length) throw new Error('semanticTypes_required');
|
||||
const fieldProjection = uniqueIdentifiers(input.binding.fieldProjection, 'fieldProjection', true);
|
||||
const binding = { id: bindingId, dataProductId, slotId, semanticTypes, fieldProjection };
|
||||
const digest = createHash('sha256')
|
||||
.update(JSON.stringify({ applicationId, pageId, binding }))
|
||||
.digest('hex')
|
||||
.slice(0, 32);
|
||||
return {
|
||||
schemaVersion: FOUNDRY_BINDING_UPSERT_SCHEMA_VERSION,
|
||||
applicationId,
|
||||
pageId,
|
||||
idempotencyKey: `foundry-binding-${digest}`,
|
||||
binding,
|
||||
};
|
||||
}
|
||||
|
||||
function requireFoundryApplicationId(value: unknown): string {
|
||||
const normalized = String(value ?? '').trim();
|
||||
if (!/^[0-9a-f]{8}-[0-9a-f]{4}-[1-5][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/i.test(normalized)) {
|
||||
throw new Error('applicationId_invalid');
|
||||
}
|
||||
return normalized;
|
||||
}
|
||||
|
||||
function requireFoundryPageId(value: unknown): string {
|
||||
const normalized = String(value ?? '').trim();
|
||||
if (!/^[a-z0-9][a-z0-9-]{0,79}$/.test(normalized)) throw new Error('pageId_invalid');
|
||||
return normalized;
|
||||
}
|
||||
|
||||
function requireFoundrySlotId(value: unknown): string {
|
||||
const normalized = String(value ?? '').trim();
|
||||
if (!/^[A-Za-z0-9][A-Za-z0-9-]{0,79}$/.test(normalized)) throw new Error('slotId_invalid');
|
||||
return normalized;
|
||||
}
|
||||
|
||||
function normalizePoint(value: unknown, index: number): NdcFact['geometry'] {
|
||||
if (!isObject(value) || value.type !== 'Point' || !Array.isArray(value.coordinates) || value.coordinates.length !== 2) {
|
||||
throw new Error(`facts_${index}_geometry_invalid`);
|
||||
}
|
||||
const [longitude, latitude] = value.coordinates;
|
||||
if (![longitude, latitude].every((coordinate) => typeof coordinate === 'number' && Number.isFinite(coordinate))) {
|
||||
throw new Error(`facts_${index}_geometry_coordinates_invalid`);
|
||||
}
|
||||
if ((longitude as number) < -180 || (longitude as number) > 180 || (latitude as number) < -90 || (latitude as number) > 90) {
|
||||
throw new Error(`facts_${index}_geometry_coordinates_out_of_range`);
|
||||
}
|
||||
return { type: 'Point', coordinates: [longitude as number, latitude as number] };
|
||||
}
|
||||
|
||||
function requireIsoTimestamp(value: unknown, field: string): string {
|
||||
const normalized = String(value ?? '').trim();
|
||||
if (!normalized || Number.isNaN(Date.parse(normalized))) throw new Error(`${field}_invalid`);
|
||||
return normalized;
|
||||
}
|
||||
|
||||
function uniqueIdentifiers(value: unknown, field: string, allowEmpty = false): string[] {
|
||||
if (!Array.isArray(value)) throw new Error(`${field}_invalid`);
|
||||
const normalized = [...new Set(value.map((item) => requireIdentifier(item, field)))];
|
||||
if (!allowEmpty && !normalized.length) throw new Error(`${field}_required`);
|
||||
return normalized;
|
||||
}
|
||||
|
||||
function sha256(value: string): string {
|
||||
return createHash('sha256').update(value).digest('hex');
|
||||
}
|
||||
|
||||
function containsSecretLikeMaterial(value: unknown): boolean {
|
||||
if (typeof value === 'string') return SECRET_LIKE_VALUE.test(value);
|
||||
if (Array.isArray(value)) return value.some(containsSecretLikeMaterial);
|
||||
if (!isObject(value)) return false;
|
||||
return Object.entries(value).some(([key, child]) => SECRET_LIKE_KEY.test(key) || containsSecretLikeMaterial(child));
|
||||
}
|
||||
|
||||
function isObject(value: unknown): value is IDataObject {
|
||||
return Boolean(value) && typeof value === 'object' && !Array.isArray(value);
|
||||
}
|
||||
|
|
@ -1,17 +0,0 @@
|
|||
export function safeInputError(error: unknown, fallback: string): Error {
|
||||
const message = error instanceof Error ? error.message : '';
|
||||
const normalized = /^[a-z0-9_.:-]{3,160}$/i.test(message) ? message : fallback;
|
||||
return new Error(normalized);
|
||||
}
|
||||
|
||||
export function safeHttpError(error: unknown, fallback: string): Error {
|
||||
const source = error && typeof error === 'object' ? error as Record<string, unknown> : {};
|
||||
const response = source.response && typeof source.response === 'object'
|
||||
? source.response as Record<string, unknown>
|
||||
: {};
|
||||
const candidate = Number(source.statusCode ?? source.status ?? response.statusCode ?? response.status);
|
||||
const suffix = Number.isInteger(candidate) && candidate >= 100 && candidate <= 599
|
||||
? `_http_${candidate}`
|
||||
: '';
|
||||
return new Error(`${fallback}${suffix}`);
|
||||
}
|
||||
|
|
@ -1,40 +0,0 @@
|
|||
import type {
|
||||
IExecuteFunctions,
|
||||
ILoadOptionsFunctions,
|
||||
IHttpRequestOptions,
|
||||
INodeListSearchItems,
|
||||
} from 'n8n-workflow';
|
||||
|
||||
import { dataProductOptions, normalizeBaseUrl } from './contracts';
|
||||
|
||||
type NdcHttpContext = IExecuteFunctions | ILoadOptionsFunctions;
|
||||
|
||||
export function serviceBaseUrl(environmentVariable: string, defaultBaseUrl: string): string {
|
||||
return normalizeBaseUrl(process.env[environmentVariable] || defaultBaseUrl);
|
||||
}
|
||||
|
||||
export async function ndcRequest(
|
||||
context: NdcHttpContext,
|
||||
credentialName: string,
|
||||
options: IHttpRequestOptions,
|
||||
): Promise<unknown> {
|
||||
return context.helpers.httpRequestWithAuthentication.call(context, credentialName, {
|
||||
...options,
|
||||
json: true,
|
||||
});
|
||||
}
|
||||
|
||||
export async function loadDataProductOptions(
|
||||
context: ILoadOptionsFunctions,
|
||||
credentialName: string,
|
||||
baseUrlEnvironmentVariable: string,
|
||||
defaultBaseUrl: string,
|
||||
catalogPath: string,
|
||||
): Promise<INodeListSearchItems[]> {
|
||||
const baseUrl = serviceBaseUrl(baseUrlEnvironmentVariable, defaultBaseUrl);
|
||||
const response = await ndcRequest(context, credentialName, {
|
||||
method: 'GET',
|
||||
url: `${baseUrl}${catalogPath}`,
|
||||
});
|
||||
return dataProductOptions(response);
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
|
|
@ -1,43 +0,0 @@
|
|||
{
|
||||
"name": "n8n-nodes-ndc",
|
||||
"version": "0.1.2",
|
||||
"description": "Private NODE.DC nodes for scoped data products and Foundry bindings.",
|
||||
"private": true,
|
||||
"license": "UNLICENSED",
|
||||
"keywords": [
|
||||
"n8n-community-node-package",
|
||||
"nodedc",
|
||||
"ndc"
|
||||
],
|
||||
"scripts": {
|
||||
"build": "n8n-node build",
|
||||
"lint": "n8n-node lint",
|
||||
"test": "npm run build && node test/package-policy.test.cjs"
|
||||
},
|
||||
"files": [
|
||||
"dist",
|
||||
"README.md"
|
||||
],
|
||||
"n8n": {
|
||||
"n8nNodesApiVersion": 1,
|
||||
"strict": false,
|
||||
"credentials": [
|
||||
"dist/credentials/NdcDataProductWriterApi.credentials.js",
|
||||
"dist/credentials/NdcDataProductReaderApi.credentials.js",
|
||||
"dist/credentials/NdcFoundryBindingApi.credentials.js"
|
||||
],
|
||||
"nodes": [
|
||||
"dist/nodes/NdcDataProductPublish/NdcDataProductPublish.node.js",
|
||||
"dist/nodes/NdcDataProductRead/NdcDataProductRead.node.js",
|
||||
"dist/nodes/NdcFoundryBinding/NdcFoundryBinding.node.js"
|
||||
]
|
||||
},
|
||||
"devDependencies": {
|
||||
"@n8n/node-cli": "0.39.3",
|
||||
"n8n-workflow": "2.3.1",
|
||||
"typescript": "5.9.3"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"n8n-workflow": ">=2.3.1 <3"
|
||||
}
|
||||
}
|
||||
|
|
@ -1,347 +0,0 @@
|
|||
const assert = require('node:assert/strict');
|
||||
const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
const { pathToFileURL } = require('node:url');
|
||||
|
||||
const packageRoot = path.resolve(__dirname, '..');
|
||||
const packageJson = require(path.join(packageRoot, 'package.json'));
|
||||
|
||||
const nodeSpecs = [
|
||||
['NdcDataProductPublish', 'ndcDataProductPublish', 'NDC Data Product Publish'],
|
||||
['NdcDataProductRead', 'ndcDataProductRead', 'NDC Data Product Read'],
|
||||
['NdcFoundryBinding', 'ndcFoundryBinding', 'NDC Foundry Binding'],
|
||||
];
|
||||
const credentialSpecs = [
|
||||
['NdcDataProductWriterApi', 'ndcDataProductWriterApi'],
|
||||
['NdcDataProductReaderApi', 'ndcDataProductReaderApi'],
|
||||
['NdcFoundryBindingApi', 'ndcFoundryBindingApi'],
|
||||
];
|
||||
|
||||
const forbiddenNodeParameter = /(url|provider|tenant|connection|token|secret|password|credential)/i;
|
||||
const forbiddenBindingKey = /(provider|tenant|connection|endpoint|url|credential|token|secret|payload)/i;
|
||||
const forbiddenProductBrand = /n8n(?:-nodes-ndc)?/i;
|
||||
const ndcNodeIcon = {
|
||||
light: 'file:../../icons/ndc.svg',
|
||||
dark: 'file:../../icons/ndc.dark.svg',
|
||||
};
|
||||
|
||||
async function main() {
|
||||
assert.equal(packageJson.name, 'n8n-nodes-ndc');
|
||||
assert.equal(packageJson.dependencies, undefined, 'private NDC nodes must not add runtime supply-chain dependencies');
|
||||
for (const lifecycle of ['preinstall', 'install', 'postinstall']) {
|
||||
assert.equal(packageJson.scripts?.[lifecycle], undefined, `${lifecycle} lifecycle script is forbidden`);
|
||||
}
|
||||
assert.equal(packageJson.n8n.nodes.length, nodeSpecs.length, 'every registered custom node must be covered by policy');
|
||||
assert.deepEqual(
|
||||
[...packageJson.n8n.nodes].sort(),
|
||||
nodeSpecs
|
||||
.map(([className]) => `dist/nodes/${className}/${className}.node.js`)
|
||||
.sort(),
|
||||
'registered custom nodes and policy specs must stay in lockstep',
|
||||
);
|
||||
const nodes = new Map();
|
||||
|
||||
for (const [className, internalName, displayName] of nodeSpecs) {
|
||||
const modulePath = path.join(packageRoot, 'dist', 'nodes', className, `${className}.node.js`);
|
||||
const NodeClass = require(modulePath)[className];
|
||||
const node = new NodeClass();
|
||||
nodes.set(className, node);
|
||||
assert.match(node.description.displayName, /^NDC /);
|
||||
assert.match(node.description.defaults.name, /^NDC /);
|
||||
assert.equal(node.description.displayName, displayName);
|
||||
assert.equal(node.description.defaults.name, displayName);
|
||||
assert.deepEqual(node.description.icon, ndcNodeIcon);
|
||||
assert.match(node.description.name, /^ndc[A-Z]/);
|
||||
assert.equal(node.description.name, internalName);
|
||||
assert.equal(`${packageJson.name}.${node.description.name}`, `${packageJson.name}.${internalName}`);
|
||||
assert.notEqual(
|
||||
node.description.usableAsTool,
|
||||
true,
|
||||
`${className} must not generate an additional *Tool runtime type`,
|
||||
);
|
||||
for (const property of node.description.properties) {
|
||||
assert.doesNotMatch(property.name, forbiddenNodeParameter, `${className}.${property.name} is transport-specific`);
|
||||
}
|
||||
assert.equal(node.description.credentials.length, 1);
|
||||
assert.match(node.description.credentials[0].name, /^ndc[A-Z]/);
|
||||
assertProductSurfaceHasNdcBrand(node.description, className);
|
||||
}
|
||||
|
||||
for (const [className, internalName] of credentialSpecs) {
|
||||
const modulePath = path.join(packageRoot, 'dist', 'credentials', `${className}.credentials.js`);
|
||||
const CredentialClass = require(modulePath)[className];
|
||||
const credential = new CredentialClass();
|
||||
assert.match(credential.displayName, /^NDC /);
|
||||
assert.equal(credential.name, internalName);
|
||||
assert.deepEqual(credential.icon, {
|
||||
light: 'file:../icons/ndc.svg',
|
||||
dark: 'file:../icons/ndc.dark.svg',
|
||||
});
|
||||
assert.deepEqual(credential.properties.map((property) => property.name), ['capability']);
|
||||
assert.equal(credential.properties[0].typeOptions.password, true);
|
||||
assert.equal(credential.authenticate.properties.headers.Authorization, '=Bearer {{$credentials.capability}}');
|
||||
assertProductSurfaceHasNdcBrand({
|
||||
displayName: credential.displayName,
|
||||
documentationUrl: credential.documentationUrl,
|
||||
properties: credential.properties,
|
||||
}, className);
|
||||
}
|
||||
|
||||
for (const icon of ['ndc.svg', 'ndc.dark.svg']) {
|
||||
const iconPath = path.join(packageRoot, 'dist', 'icons', icon);
|
||||
assert.equal(fs.existsSync(iconPath), true, `${icon} was not copied`);
|
||||
assert.match(fs.readFileSync(iconPath, 'utf8'), /aria-label="NDC"/, `${icon} must expose the NDC brand`);
|
||||
}
|
||||
|
||||
const contracts = require(path.join(packageRoot, 'dist', 'nodes', 'shared', 'contracts.js'));
|
||||
const constants = require(path.join(packageRoot, 'dist', 'nodes', 'shared', 'constants.js'));
|
||||
const http = require(path.join(packageRoot, 'dist', 'nodes', 'shared', 'http.js'));
|
||||
const externalContract = await import(pathToFileURL(
|
||||
path.resolve(packageRoot, '..', 'external-provider-contract', 'src', 'index.mjs'),
|
||||
).href);
|
||||
const items = [
|
||||
{
|
||||
json: {
|
||||
sourceId: 'fleet.unit.42',
|
||||
semanticType: 'map.moving_object',
|
||||
observedAt: '2026-07-15T12:00:00.000Z',
|
||||
attributes: { speedKph: 12 },
|
||||
geometry: { type: 'Point', coordinates: [37.61, 55.75] },
|
||||
},
|
||||
},
|
||||
];
|
||||
const publish = contracts.buildPublishPayload(
|
||||
items,
|
||||
'execution-42',
|
||||
'workflow-7',
|
||||
'node-3',
|
||||
'fleet.positions.current.v1',
|
||||
0,
|
||||
);
|
||||
assert.deepEqual(externalContract.validateDataProductPublish(publish), { ok: true, errors: [] });
|
||||
assert.deepEqual(
|
||||
publish,
|
||||
contracts.buildPublishPayload(items, 'execution-42', 'workflow-7', 'node-3', 'fleet.positions.current.v1', 0),
|
||||
'same execution chunk must produce the same idempotency envelope',
|
||||
);
|
||||
assert.notEqual(
|
||||
publish.batch.idempotencyKey,
|
||||
contracts.buildPublishPayload(items, 'execution-42', 'workflow-7', 'node-3', 'fleet.positions.current.v2', 0).batch.idempotencyKey,
|
||||
);
|
||||
assert.throws(
|
||||
() => contracts.buildPublishPayload(
|
||||
[{ json: { ...items[0].json, attributes: { accessToken: 'forbidden' } } }],
|
||||
'execution-42',
|
||||
'workflow-7',
|
||||
'node-3',
|
||||
'fleet.positions.current.v1',
|
||||
0,
|
||||
),
|
||||
/secret_material_forbidden/,
|
||||
);
|
||||
|
||||
const foundry = contracts.buildFoundryBindingPayload({
|
||||
applicationId: '11111111-1111-4111-8111-111111111111',
|
||||
pageId: 'map',
|
||||
binding: {
|
||||
id: 'fleet-live-points',
|
||||
dataProductId: 'fleet.positions.current.v1',
|
||||
slotId: 'points',
|
||||
semanticTypes: ['map.moving_object'],
|
||||
fieldProjection: ['coordinates.longitude', 'coordinates.latitude'],
|
||||
},
|
||||
});
|
||||
assertNoForbiddenKeys(foundry, forbiddenBindingKey);
|
||||
assert.equal(externalContract.validateFoundryBindingUpsert(foundry).ok, true);
|
||||
assert.equal(foundry.schemaVersion, externalContract.FOUNDRY_BINDING_UPSERT_SCHEMA_VERSION);
|
||||
assert.equal(foundry.binding.dataProductId, 'fleet.positions.current.v1');
|
||||
assert.equal(foundry.binding.slotId, 'points');
|
||||
assert.match(foundry.idempotencyKey, /^foundry-binding-[a-f0-9]{32}$/);
|
||||
|
||||
assert.equal(constants.DATA_PRODUCT_WRITER_CATALOG_PATH, '/internal/data-plane/v1/writer/data-products');
|
||||
assert.equal(constants.DATA_PRODUCT_READER_CATALOG_PATH, '/internal/data-plane/v1/reader/data-products');
|
||||
assert.equal(constants.DATA_PRODUCT_BASE_PATH, '/internal/data-plane/v1/data-products');
|
||||
assert.equal(constants.DATA_PLANE_DEFAULT_BASE_URL, 'http://external-data-plane:18106');
|
||||
assert.equal(constants.FOUNDRY_DEFAULT_BASE_URL, 'http://nodedc-module-foundry:3333');
|
||||
const previousDefaultProbe = process.env.NDC_DATA_PLANE_BASE_URL;
|
||||
delete process.env.NDC_DATA_PLANE_BASE_URL;
|
||||
assert.equal(
|
||||
http.serviceBaseUrl(constants.DATA_PLANE_BASE_URL_ENV, constants.DATA_PLANE_DEFAULT_BASE_URL),
|
||||
'http://external-data-plane:18106',
|
||||
);
|
||||
restoreEnvironment('NDC_DATA_PLANE_BASE_URL', previousDefaultProbe);
|
||||
|
||||
const readProperties = nodes.get('NdcDataProductRead').description.properties.map((property) => property.name);
|
||||
assert.equal(readProperties.includes('cursor'), false, 'snapshot endpoint has no cursor input');
|
||||
assert.equal(readProperties.includes('history'), false, 'history must stay hidden until its endpoint exists');
|
||||
const pageSizeProperty = nodes.get('NdcDataProductRead').description.properties.find((property) => property.name === 'pageSize');
|
||||
assert.equal(pageSizeProperty.typeOptions.minValue, 1);
|
||||
assert.equal(pageSizeProperty.typeOptions.maxValue, 5000, 'bounded snapshot v1 must not expose more than 5000 facts');
|
||||
assert.equal(pageSizeProperty.default <= pageSizeProperty.typeOptions.maxValue, true);
|
||||
const slotProperty = nodes.get('NdcFoundryBinding').description.properties.find((property) => property.name === 'slotId');
|
||||
assert.equal(slotProperty.default, 'points');
|
||||
|
||||
await assertNodeHttpContracts(nodes, externalContract, constants);
|
||||
|
||||
console.log('n8n-nodes-ndc package policy: ok');
|
||||
}
|
||||
|
||||
function assertProductSurfaceHasNdcBrand(surface, className) {
|
||||
const visibleStrings = [];
|
||||
collectVisibleStrings(surface, visibleStrings);
|
||||
for (const value of visibleStrings) {
|
||||
assert.doesNotMatch(value, forbiddenProductBrand, `${className} leaks a technical package/runtime brand: ${value}`);
|
||||
}
|
||||
}
|
||||
|
||||
function collectVisibleStrings(value, output, key = '') {
|
||||
if (typeof value === 'string') {
|
||||
if (!['name', 'type', 'value'].includes(key)) output.push(value);
|
||||
return;
|
||||
}
|
||||
if (Array.isArray(value)) {
|
||||
for (const item of value) collectVisibleStrings(item, output, key);
|
||||
return;
|
||||
}
|
||||
if (!value || typeof value !== 'object') return;
|
||||
for (const [childKey, childValue] of Object.entries(value)) {
|
||||
if (['credentials', 'icon'].includes(childKey)) continue;
|
||||
collectVisibleStrings(childValue, output, childKey);
|
||||
}
|
||||
}
|
||||
|
||||
async function assertNodeHttpContracts(nodes, externalContract, constants) {
|
||||
const previousDataPlaneUrl = process.env.NDC_DATA_PLANE_BASE_URL;
|
||||
const previousFoundryUrl = process.env.NDC_FOUNDRY_BASE_URL;
|
||||
process.env.NDC_DATA_PLANE_BASE_URL = 'http://data-plane.test/';
|
||||
process.env.NDC_FOUNDRY_BASE_URL = 'http://foundry.test/';
|
||||
try {
|
||||
const publishCalls = [];
|
||||
const publishContext = executionContext({
|
||||
parameters: { dataProductId: 'fleet.positions.current.v1', sequence: 0 },
|
||||
input: [{ json: canonicalFact() }],
|
||||
response: { ok: true, publishedFactCount: 1 },
|
||||
calls: publishCalls,
|
||||
});
|
||||
const publishResult = await nodes.get('NdcDataProductPublish').execute.call(publishContext);
|
||||
assert.equal(publishCalls[0].credentialName, 'ndcDataProductWriterApi');
|
||||
assert.equal(publishCalls[0].options.method, 'POST');
|
||||
assert.equal(
|
||||
publishCalls[0].options.url,
|
||||
'http://data-plane.test/internal/data-plane/v1/data-products/fleet.positions.current.v1/publish',
|
||||
);
|
||||
assert.equal(externalContract.validateDataProductPublish(publishCalls[0].options.body).ok, true);
|
||||
assert.equal(publishResult[0][0].json.publishedFactCount, 1);
|
||||
|
||||
const readCalls = [];
|
||||
const readContext = executionContext({
|
||||
parameters: { dataProductId: 'fleet.positions.current.v1', pageSize: 1000 },
|
||||
response: {
|
||||
schemaVersion: 'nodedc.data-product.snapshot/v1',
|
||||
dataProduct: { id: 'fleet.positions.current.v1', version: '1.0.0' },
|
||||
generatedAt: '2026-07-15T12:00:01.000Z',
|
||||
cursor: '1',
|
||||
facts: [{ ...canonicalFact(), receivedAt: '2026-07-15T12:00:00.100Z' }],
|
||||
},
|
||||
calls: readCalls,
|
||||
});
|
||||
const readResult = await nodes.get('NdcDataProductRead').execute.call(readContext);
|
||||
assert.equal(readCalls[0].credentialName, 'ndcDataProductReaderApi');
|
||||
assert.equal(readCalls[0].options.method, 'GET');
|
||||
assert.equal(
|
||||
readCalls[0].options.url,
|
||||
'http://data-plane.test/internal/data-plane/v1/data-products/fleet.positions.current.v1/snapshot',
|
||||
);
|
||||
assert.deepEqual(readCalls[0].options.qs, { limit: 1000 });
|
||||
assert.equal(readResult[0][0].json.sourceId, 'fleet.unit.42');
|
||||
|
||||
const foundryCalls = [];
|
||||
const foundryContext = executionContext({
|
||||
parameters: {
|
||||
applicationId: '11111111-1111-4111-8111-111111111111',
|
||||
pageId: 'map',
|
||||
bindingId: 'fleet-live-points',
|
||||
dataProductId: 'fleet.positions.current.v1',
|
||||
slotId: 'points',
|
||||
semanticTypes: ['map.moving_object'],
|
||||
fieldProjection: ['coordinates.longitude'],
|
||||
},
|
||||
response: { ok: true, binding: { id: 'fleet-live-points' } },
|
||||
calls: foundryCalls,
|
||||
});
|
||||
await nodes.get('NdcFoundryBinding').execute.call(foundryContext);
|
||||
assert.equal(foundryCalls[0].credentialName, 'ndcFoundryBindingApi');
|
||||
assert.equal(foundryCalls[0].options.method, 'POST');
|
||||
assert.equal(foundryCalls[0].options.url, 'http://foundry.test/internal/foundry/v1/data-product-bindings');
|
||||
assertNoForbiddenKeys(foundryCalls[0].options.body, forbiddenBindingKey);
|
||||
|
||||
const catalogCalls = [];
|
||||
const catalogContext = {
|
||||
helpers: {
|
||||
async httpRequestWithAuthentication(credentialName, options) {
|
||||
catalogCalls.push({ credentialName, options });
|
||||
return { dataProducts: [{ id: 'fleet.positions.current.v1', version: '1.0.0' }] };
|
||||
},
|
||||
},
|
||||
};
|
||||
const writerOptions = await nodes.get('NdcDataProductPublish').methods.loadOptions.getDataProducts.call(catalogContext);
|
||||
const readerOptions = await nodes.get('NdcDataProductRead').methods.loadOptions.getDataProducts.call(catalogContext);
|
||||
assert.deepEqual(writerOptions, [{ name: 'fleet.positions.current.v1 · 1.0.0', value: 'fleet.positions.current.v1' }]);
|
||||
assert.deepEqual(readerOptions, writerOptions);
|
||||
assert.equal(catalogCalls[0].options.url, `http://data-plane.test${constants.DATA_PRODUCT_WRITER_CATALOG_PATH}`);
|
||||
assert.equal(catalogCalls[1].options.url, `http://data-plane.test${constants.DATA_PRODUCT_READER_CATALOG_PATH}`);
|
||||
} finally {
|
||||
restoreEnvironment('NDC_DATA_PLANE_BASE_URL', previousDataPlaneUrl);
|
||||
restoreEnvironment('NDC_FOUNDRY_BASE_URL', previousFoundryUrl);
|
||||
}
|
||||
}
|
||||
|
||||
function executionContext({ parameters, input = [{ json: {} }], response, calls }) {
|
||||
return {
|
||||
getNodeParameter(name, _index, defaultValue) {
|
||||
return Object.prototype.hasOwnProperty.call(parameters, name) ? parameters[name] : defaultValue;
|
||||
},
|
||||
getInputData() { return input; },
|
||||
getExecutionId() { return 'execution-42'; },
|
||||
getWorkflow() { return { id: 'workflow-7' }; },
|
||||
getNode() { return { id: 'node-3' }; },
|
||||
helpers: {
|
||||
async httpRequestWithAuthentication(credentialName, options) {
|
||||
calls.push({ credentialName, options });
|
||||
return response;
|
||||
},
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
function canonicalFact() {
|
||||
return {
|
||||
sourceId: 'fleet.unit.42',
|
||||
semanticType: 'map.moving_object',
|
||||
observedAt: '2026-07-15T12:00:00.000Z',
|
||||
attributes: { speedKph: 12 },
|
||||
geometry: { type: 'Point', coordinates: [37.61, 55.75] },
|
||||
};
|
||||
}
|
||||
|
||||
function restoreEnvironment(name, value) {
|
||||
if (value === undefined) delete process.env[name];
|
||||
else process.env[name] = value;
|
||||
}
|
||||
|
||||
function assertNoForbiddenKeys(value, pattern, pathPrefix = 'body') {
|
||||
if (Array.isArray(value)) {
|
||||
value.forEach((entry, index) => assertNoForbiddenKeys(entry, pattern, `${pathPrefix}[${index}]`));
|
||||
return;
|
||||
}
|
||||
if (!value || typeof value !== 'object') return;
|
||||
for (const [key, child] of Object.entries(value)) {
|
||||
assert.doesNotMatch(key, pattern, `${pathPrefix}.${key} contains transport material`);
|
||||
assertNoForbiddenKeys(child, pattern, `${pathPrefix}.${key}`);
|
||||
}
|
||||
}
|
||||
|
||||
main().catch((error) => {
|
||||
console.error(error);
|
||||
process.exitCode = 1;
|
||||
});
|
||||
|
|
@ -1,29 +0,0 @@
|
|||
{
|
||||
"compilerOptions": {
|
||||
"strict": true,
|
||||
"module": "commonjs",
|
||||
"moduleResolution": "node",
|
||||
"target": "es2019",
|
||||
"lib": ["es2019", "es2020", "es2022.error"],
|
||||
"removeComments": true,
|
||||
"useUnknownInCatchVariables": false,
|
||||
"forceConsistentCasingInFileNames": true,
|
||||
"noImplicitAny": true,
|
||||
"noImplicitReturns": true,
|
||||
"noUnusedLocals": true,
|
||||
"strictNullChecks": true,
|
||||
"preserveConstEnums": true,
|
||||
"esModuleInterop": true,
|
||||
"resolveJsonModule": true,
|
||||
"incremental": false,
|
||||
"declaration": true,
|
||||
"sourceMap": true,
|
||||
"skipLibCheck": true,
|
||||
"outDir": "./dist/"
|
||||
},
|
||||
"include": [
|
||||
"credentials/**/*",
|
||||
"nodes/**/*",
|
||||
"package.json"
|
||||
]
|
||||
}
|
||||
|
|
@ -467,27 +467,9 @@ function normalizeTypeName(value) {
|
|||
function fullNodeType(value) {
|
||||
const raw = cleanString(value, 240)
|
||||
if (!raw) return ''
|
||||
if (isPackageQualifiedNodeType(raw)) return raw
|
||||
return raw.startsWith('n8n-nodes-base.') ? raw : `n8n-nodes-base.${raw}`
|
||||
}
|
||||
|
||||
function isPackageQualifiedNodeType(value) {
|
||||
return /^(?:@[a-z0-9._-]+\/)?n8n-nodes-[a-z0-9._-]+\.[a-z0-9._-]+$/i.test(cleanString(value, 240))
|
||||
}
|
||||
|
||||
function catalogNodeRuntimeType(node) {
|
||||
const explicit = [node?.runtimeType, node?.fullType, node?.publicType, node?.type]
|
||||
.map((value) => cleanString(value, 240))
|
||||
.find(isPackageQualifiedNodeType)
|
||||
if (explicit) return explicit
|
||||
const packageName = cleanString(node?.packageName || node?.package, 160)
|
||||
const name = cleanString(node?.name, 160)
|
||||
if (name && /^(?:@[a-z0-9._-]+\/)?n8n-nodes-[a-z0-9._-]+$/i.test(packageName)) {
|
||||
return `${packageName}.${name}`
|
||||
}
|
||||
return fullNodeType(node?.publicType || name)
|
||||
}
|
||||
|
||||
function cleanWebhookSegment(value, fallback) {
|
||||
const out = cleanString(value, 240)
|
||||
.toLowerCase()
|
||||
|
|
@ -556,8 +538,7 @@ function compactNodeDefinition(node, maxProperties = 80) {
|
|||
const properties = Array.isArray(node?.properties) ? node.properties.slice(0, maxProperties) : []
|
||||
return {
|
||||
name: node?.name || '',
|
||||
publicType: node?.publicType || node?.name || '',
|
||||
fullType: catalogNodeRuntimeType(node),
|
||||
fullType: fullNodeType(node?.name || ''),
|
||||
displayName: node?.displayName || node?.name || '',
|
||||
description: node?.description || '',
|
||||
group: node?.group || [],
|
||||
|
|
@ -741,8 +722,7 @@ async function handleSearchNodes(args) {
|
|||
.slice(0, limit)
|
||||
.map(({ node }) => ({
|
||||
name: node?.name || '',
|
||||
publicType: node?.publicType || node?.name || '',
|
||||
fullType: catalogNodeRuntimeType(node),
|
||||
fullType: fullNodeType(node?.name || ''),
|
||||
displayName: node?.displayName || node?.name || '',
|
||||
description: node?.description || '',
|
||||
group: node?.group || [],
|
||||
|
|
@ -757,8 +737,7 @@ async function handleGetNodeDefinition(args) {
|
|||
const catalog = await loadNodeCatalog(args.schemaVersion)
|
||||
const node = catalog.find((item) => (
|
||||
String(item?.name || '') === nodeType ||
|
||||
String(item?.publicType || '') === nodeType ||
|
||||
catalogNodeRuntimeType(item) === cleanString(args.nodeType || args.type || args.name)
|
||||
fullNodeType(item?.name || '') === cleanString(args.nodeType || args.type || args.name)
|
||||
))
|
||||
if (!node) throw new Error(`node_not_found:${nodeType}`)
|
||||
return { ok: true, node: compactNodeDefinition(node, Number(args.maxProperties || 80) || 80) }
|
||||
|
|
|
|||
|
|
@ -15,7 +15,7 @@ const SUPPORTED_CONNECTION_MODES = new Set(["hub", "direct"]);
|
|||
const SUPPORTED_EXECUTOR_STATUSES = new Set(["unknown", "online", "offline", "checking", "error"]);
|
||||
const SUPPORTED_THREAD_STATES = new Set(["active", "archived"]);
|
||||
const SUPPORTED_MESSAGE_ROLES = new Set(["user", "assistant", "system", "tool"]);
|
||||
const SUPPORTED_TOOL_PACKS = new Set(["engine", "ops", "ndc-agent-core", "ontology", "deploy", "docs"]);
|
||||
const SUPPORTED_TOOL_PACKS = new Set(["engine", "ops", "ndc-agent-core", "deploy", "docs"]);
|
||||
const SUPPORTED_RUN_STATUSES = new Set(["running", "completed", "failed", "timeout"]);
|
||||
const AI_WORKSPACE_BRIDGE_PACKAGE_NAME = "@nodedc/ai-workspace-bridge";
|
||||
const AI_WORKSPACE_BRIDGE_PACKAGE_BIN = "ai-workspace-bridge";
|
||||
|
|
@ -81,26 +81,6 @@ const APP_ROUTING_CATALOG = [
|
|||
requiredScopes: ["engine:workspace:read"],
|
||||
deniedText: ACCESS_DENIED_TEXT,
|
||||
},
|
||||
{
|
||||
appId: "ontology",
|
||||
appTitle: "NODE.DC Ontology Core",
|
||||
surface: "global",
|
||||
skillId: "ontology-context",
|
||||
whenToUse: [
|
||||
"canonical entities",
|
||||
"aliases",
|
||||
"relations",
|
||||
"semantic guardrails",
|
||||
"data contracts",
|
||||
"cross-contour context",
|
||||
"Gelios domain model",
|
||||
],
|
||||
actionNamespaces: [],
|
||||
actionIdPrefixes: [],
|
||||
mcpServerNames: ["nodedc_ontology"],
|
||||
requiredScopes: ["ontology:catalog:read"],
|
||||
deniedText: ACCESS_DENIED_TEXT,
|
||||
},
|
||||
{
|
||||
appId: "ops",
|
||||
appTitle: "NODE.DC Ops / Tasker",
|
||||
|
|
@ -2133,23 +2113,18 @@ async function buildRunProfile({ owner, thread, executor, ownerSettings, bridgeP
|
|||
|| thread.originSurface
|
||||
|| "global";
|
||||
const targetContexts = isPlainObject(context.contexts) ? context.contexts : {};
|
||||
let enabledToolPacks = mergeToolPacks(
|
||||
const enabledToolPacks = mergeToolPacks(
|
||||
ownerSettings?.enabledToolPacks,
|
||||
thread.enabledToolPacks,
|
||||
bridgePayload?.enabledToolPacks
|
||||
);
|
||||
const grantResolution = await resolveRunAppGrants({ owner, context, ownerSettings });
|
||||
const ontologyGrant = ontologyMcpGrantForExecutor(executor);
|
||||
const resolvedAppGrants = ontologyGrant
|
||||
? { ...grantResolution.appGrants, ontology: ontologyGrant }
|
||||
: grantResolution.appGrants;
|
||||
if (ontologyGrant) enabledToolPacks = mergeToolPacks(enabledToolPacks, ["ontology"]);
|
||||
const appGrants = summarizeRunAppGrants({ appGrants: resolvedAppGrants });
|
||||
const mcpServers = runProfileMcpServersFromAppGrants(ownerSettings, resolvedAppGrants);
|
||||
const appGrants = summarizeRunAppGrants({ appGrants: grantResolution.appGrants });
|
||||
const mcpServers = runProfileMcpServersFromAppGrants(ownerSettings, grantResolution.appGrants);
|
||||
const mcpServerNames = mcpServers.map((server) => server.serverName).filter(Boolean);
|
||||
const assistantActions = await assistantActionToolProfileForRun();
|
||||
const appCatalog = buildRunProfileAppCatalog({
|
||||
appGrants: resolvedAppGrants,
|
||||
appGrants: grantResolution.appGrants,
|
||||
mcpServers,
|
||||
assistantActions,
|
||||
});
|
||||
|
|
@ -2173,7 +2148,6 @@ async function buildRunProfile({ owner, thread, executor, ownerSettings, bridgeP
|
|||
deniedAppIds: appAccess.deniedAppIds,
|
||||
notGrantedAppIds: appAccess.notGrantedAppIds,
|
||||
entitlementAdapters: grantResolution.diagnostics,
|
||||
ontologyMcp: ontologyMcpDiagnostic(executor, ontologyGrant),
|
||||
mcpServerNames,
|
||||
requiredMcpServerNames,
|
||||
assistantActionIds: assistantActions.actionIds,
|
||||
|
|
@ -2215,41 +2189,6 @@ async function buildRunProfile({ owner, thread, executor, ownerSettings, bridgeP
|
|||
return runProfile;
|
||||
}
|
||||
|
||||
function ontologyMcpGrantForExecutor(executor) {
|
||||
if (!config.ontologyMcpEnabled) return null;
|
||||
if (executor?.connectionMode !== "hub") return null;
|
||||
const pairingCode = cleanPairingCode(executor?.pairingCode);
|
||||
if (!pairingCode || !config.ontologyMcpPublicBaseUrl) return null;
|
||||
return {
|
||||
appId: "ontology",
|
||||
appTitle: "NODE.DC Ontology Core",
|
||||
surface: "global",
|
||||
source: "platform-runtime",
|
||||
status: "granted",
|
||||
scopes: ["ontology:catalog:read"],
|
||||
mcpServers: [{
|
||||
appId: "ontology",
|
||||
appTitle: "NODE.DC Ontology Core",
|
||||
serverName: "nodedc_ontology",
|
||||
url: `${config.ontologyMcpPublicBaseUrl}/api/ai-workspace/hub/v1/ontology-mcp/${encodeURIComponent(pairingCode)}/mcp`,
|
||||
required: false,
|
||||
startupTimeoutSec: 20,
|
||||
toolTimeoutSec: 60,
|
||||
httpHeaders: {
|
||||
Accept: "application/json, text/event-stream",
|
||||
"MCP-Protocol-Version": "2025-06-18",
|
||||
},
|
||||
}],
|
||||
};
|
||||
}
|
||||
|
||||
function ontologyMcpDiagnostic(executor, grant) {
|
||||
if (grant) return { status: "granted", serverName: "nodedc_ontology", readOnly: true };
|
||||
if (!config.ontologyMcpEnabled) return { status: "disabled", readOnly: true };
|
||||
if (executor?.connectionMode !== "hub") return { status: "unavailable_for_direct_executor", readOnly: true };
|
||||
return { status: "pairing_or_public_hub_url_required", readOnly: true };
|
||||
}
|
||||
|
||||
async function assistantActionToolProfileForRun() {
|
||||
const gatewayUrl = assistantActionGatewayUrlForRun();
|
||||
const gatewayToken = optionalString(
|
||||
|
|
@ -2772,10 +2711,6 @@ function buildRunProfilePolicyPrompt({ context, diagnostics, assistantActions })
|
|||
"- Interpret the user's natural-language request first; call assistant actions only after selecting a structured action id.",
|
||||
"- Read assistant actions may execute after structured action selection. Privileged/write assistant actions require preview, explicit user confirmation, then execute.",
|
||||
"- Ops card actions advertised in this run are valid assistant actions: use ops.card.list_recent for reading cards, ops.card.create for creating cards, and ops.card.add_comment for comments instead of refusing because direct Ops MCP tools are absent.",
|
||||
...(diagnostics.mcpServerNames.includes("nodedc_ontology") ? [
|
||||
"- The nodedc_ontology MCP server is the live, read-only semantic source for canonical entities, aliases, relations and guardrails. Use it before inventing workflow names, data contracts or cross-contour bindings.",
|
||||
"- Ontology Core does not expose telemetry, databases, credentials, command dispatch or Studio controls. Request those only through separately granted application capabilities.",
|
||||
] : []),
|
||||
"- Destructive assistant actions are forbidden; offer safe alternatives such as block/disable instead of delete.",
|
||||
"- MCP tokens and headers are runtime secrets and must never be printed in public answers.",
|
||||
];
|
||||
|
|
@ -4530,11 +4465,6 @@ function readConfig() {
|
|||
optionalString(process.env.NDC_AI_WORKSPACE_OPS_GATEWAY_BASE_URL) ||
|
||||
"";
|
||||
const normalizedSetupGatewayUrl = setupGatewayUrl.replace(/\/+$/, "");
|
||||
const ontologyMcpPublicBaseUrl = cleanHttpEndpoint(
|
||||
optionalString(process.env.AI_WORKSPACE_ONTOLOGY_MCP_PUBLIC_URL) ||
|
||||
optionalString(process.env.NDC_AI_WORKSPACE_ONTOLOGY_MCP_PUBLIC_URL) ||
|
||||
httpUrlFromWebSocketUrl(hubWebSocketUrl)
|
||||
);
|
||||
const bridgePackageSpec =
|
||||
optionalString(process.env.AI_WORKSPACE_BRIDGE_PACKAGE_SPEC) ||
|
||||
optionalString(process.env.NDC_AI_WORKSPACE_BRIDGE_PACKAGE_SPEC) ||
|
||||
|
|
@ -4575,16 +4505,6 @@ function readConfig() {
|
|||
),
|
||||
hubInternalAccessToken:
|
||||
explicitHubAccessToken || (isDeployedPublicHubUrl(hubInternalHttpUrl) ? "" : sharedInternalAccessToken),
|
||||
ontologyMcpEnabled: isTruthy(
|
||||
optionalString(process.env.AI_WORKSPACE_ONTOLOGY_MCP_ENABLED) ||
|
||||
optionalString(process.env.NDC_AI_WORKSPACE_ONTOLOGY_MCP_ENABLED) ||
|
||||
"false"
|
||||
) && !isFalsy(
|
||||
optionalString(process.env.AI_WORKSPACE_ONTOLOGY_MCP_ENABLED) ||
|
||||
optionalString(process.env.NDC_AI_WORKSPACE_ONTOLOGY_MCP_ENABLED) ||
|
||||
"false"
|
||||
),
|
||||
ontologyMcpPublicBaseUrl,
|
||||
ontologyLauncherBaseUrl: ontologyLauncherBaseUrl.replace(/\/+$/, ""),
|
||||
launcherInternalAccessToken,
|
||||
opsGatewayBaseUrl: opsGatewayBaseUrl.replace(/\/+$/, ""),
|
||||
|
|
|
|||
|
|
@ -3457,27 +3457,9 @@ function normalizeTypeName(value) {
|
|||
function fullNodeType(value) {
|
||||
const raw = cleanString(value, 240)
|
||||
if (!raw) return ''
|
||||
if (isPackageQualifiedNodeType(raw)) return raw
|
||||
return raw.startsWith('n8n-nodes-base.') ? raw : `n8n-nodes-base.${raw}`
|
||||
}
|
||||
|
||||
function isPackageQualifiedNodeType(value) {
|
||||
return /^(?:@[a-z0-9._-]+\/)?n8n-nodes-[a-z0-9._-]+\.[a-z0-9._-]+$/i.test(cleanString(value, 240))
|
||||
}
|
||||
|
||||
function catalogNodeRuntimeType(node) {
|
||||
const explicit = [node?.runtimeType, node?.fullType, node?.publicType, node?.type]
|
||||
.map((value) => cleanString(value, 240))
|
||||
.find(isPackageQualifiedNodeType)
|
||||
if (explicit) return explicit
|
||||
const packageName = cleanString(node?.packageName || node?.package, 160)
|
||||
const name = cleanString(node?.name, 160)
|
||||
if (name && /^(?:@[a-z0-9._-]+\/)?n8n-nodes-[a-z0-9._-]+$/i.test(packageName)) {
|
||||
return `${packageName}.${name}`
|
||||
}
|
||||
return fullNodeType(node?.publicType || name)
|
||||
}
|
||||
|
||||
function cleanWebhookSegment(value, fallback) {
|
||||
const out = cleanString(value, 240)
|
||||
.toLowerCase()
|
||||
|
|
@ -3546,8 +3528,7 @@ function compactNodeDefinition(node, maxProperties = 80) {
|
|||
const properties = Array.isArray(node?.properties) ? node.properties.slice(0, maxProperties) : []
|
||||
return {
|
||||
name: node?.name || '',
|
||||
publicType: node?.publicType || node?.name || '',
|
||||
fullType: catalogNodeRuntimeType(node),
|
||||
fullType: fullNodeType(node?.name || ''),
|
||||
displayName: node?.displayName || node?.name || '',
|
||||
description: node?.description || '',
|
||||
group: node?.group || [],
|
||||
|
|
@ -3731,8 +3712,7 @@ async function handleSearchNodes(args) {
|
|||
.slice(0, limit)
|
||||
.map(({ node }) => ({
|
||||
name: node?.name || '',
|
||||
publicType: node?.publicType || node?.name || '',
|
||||
fullType: catalogNodeRuntimeType(node),
|
||||
fullType: fullNodeType(node?.name || ''),
|
||||
displayName: node?.displayName || node?.name || '',
|
||||
description: node?.description || '',
|
||||
group: node?.group || [],
|
||||
|
|
@ -3747,8 +3727,7 @@ async function handleGetNodeDefinition(args) {
|
|||
const catalog = await loadNodeCatalog(args.schemaVersion)
|
||||
const node = catalog.find((item) => (
|
||||
String(item?.name || '') === nodeType ||
|
||||
String(item?.publicType || '') === nodeType ||
|
||||
catalogNodeRuntimeType(item) === cleanString(args.nodeType || args.type || args.name)
|
||||
fullNodeType(item?.name || '') === cleanString(args.nodeType || args.type || args.name)
|
||||
))
|
||||
if (!node) throw new Error(`node_not_found:${nodeType}`)
|
||||
return { ok: true, node: compactNodeDefinition(node, Number(args.maxProperties || 80) || 80) }
|
||||
|
|
|
|||
|
|
@ -5,8 +5,7 @@
|
|||
"type": "module",
|
||||
"scripts": {
|
||||
"start": "node src/server.mjs",
|
||||
"dev": "node --watch src/server.mjs",
|
||||
"smoke:ontology-mcp-proxy": "node src/scripts/smoke-ontology-mcp-proxy.mjs"
|
||||
"dev": "node --watch src/server.mjs"
|
||||
},
|
||||
"dependencies": {
|
||||
"express": "^5.2.1",
|
||||
|
|
|
|||
|
|
@ -1,129 +0,0 @@
|
|||
#!/usr/bin/env node
|
||||
|
||||
import assert from 'node:assert/strict'
|
||||
import { spawn } from 'node:child_process'
|
||||
import { once } from 'node:events'
|
||||
import { createServer } from 'node:http'
|
||||
import path from 'node:path'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
import { WebSocket } from 'ws'
|
||||
import { createOntologyMcpServer } from '../../../ontology-core/src/mcp-server.mjs'
|
||||
|
||||
const INTERNAL_TOKEN = 'hub-ontology-mcp-smoke-token'
|
||||
const PAIRING_CODE = 'ONTOLOGYSMOKE1'
|
||||
const currentFile = fileURLToPath(import.meta.url)
|
||||
const hubRoot = path.resolve(path.dirname(currentFile), '..', '..')
|
||||
|
||||
const ontologyPort = await availablePort()
|
||||
const hubPort = await availablePort()
|
||||
const ontologyServer = createOntologyMcpServer({
|
||||
internalTokens: [INTERNAL_TOKEN],
|
||||
allowedOrigins: [],
|
||||
maxBodyBytes: 1024 * 1024,
|
||||
})
|
||||
const hubProcess = spawn(process.execPath, ['src/server.mjs'], {
|
||||
cwd: hubRoot,
|
||||
env: {
|
||||
...process.env,
|
||||
PORT: String(hubPort),
|
||||
AI_WORKSPACE_HUB_TOKEN: INTERNAL_TOKEN,
|
||||
NODEDC_INTERNAL_ACCESS_TOKEN: INTERNAL_TOKEN,
|
||||
NODEDC_ONTOLOGY_CORE_URL: `http://127.0.0.1:${ontologyPort}`,
|
||||
NODEDC_AI_WORKSPACE_ASSISTANT_URL: 'http://127.0.0.1:1',
|
||||
},
|
||||
stdio: ['ignore', 'pipe', 'pipe'],
|
||||
})
|
||||
let hubOutput = ''
|
||||
hubProcess.stdout.on('data', (chunk) => { hubOutput += String(chunk) })
|
||||
hubProcess.stderr.on('data', (chunk) => { hubOutput += String(chunk) })
|
||||
let worker = null
|
||||
|
||||
try {
|
||||
await new Promise((resolve) => ontologyServer.listen(ontologyPort, '127.0.0.1', resolve))
|
||||
await waitForHub(`http://127.0.0.1:${hubPort}/healthz`, hubProcess)
|
||||
|
||||
worker = new WebSocket(
|
||||
`ws://127.0.0.1:${hubPort}/api/ai-workspace/hub?pairingCode=${PAIRING_CODE}&machineName=ontology-smoke`,
|
||||
)
|
||||
await once(worker, 'open')
|
||||
worker.send(JSON.stringify({
|
||||
type: 'hello',
|
||||
agentVersion: 'smoke',
|
||||
protocolVersion: 'ai-workspace-bridge/v1',
|
||||
capabilities: ['smoke'],
|
||||
}))
|
||||
|
||||
const response = await fetch(
|
||||
`http://127.0.0.1:${hubPort}/api/ai-workspace/hub/v1/ontology-mcp/${PAIRING_CODE}/mcp`,
|
||||
{
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'content-type': 'application/json',
|
||||
accept: 'application/json, text/event-stream',
|
||||
'mcp-protocol-version': '2025-06-18',
|
||||
},
|
||||
body: JSON.stringify({
|
||||
jsonrpc: '2.0',
|
||||
id: 1,
|
||||
method: 'tools/call',
|
||||
params: {
|
||||
name: 'ontology_get_entity',
|
||||
arguments: { term: 'helius' },
|
||||
},
|
||||
}),
|
||||
},
|
||||
)
|
||||
const payload = await response.json()
|
||||
assert.equal(response.ok, true)
|
||||
assert.equal(response.headers.get('mcp-protocol-version'), '2025-06-18')
|
||||
assert.equal(payload.result.structuredContent.entity.id, 'gelios.integration')
|
||||
|
||||
worker.close()
|
||||
await once(worker, 'close')
|
||||
const offline = await fetch(
|
||||
`http://127.0.0.1:${hubPort}/api/ai-workspace/hub/v1/ontology-mcp/${PAIRING_CODE}/mcp`,
|
||||
{ method: 'POST', headers: { 'content-type': 'application/json' }, body: '{}' },
|
||||
)
|
||||
assert.equal(offline.status, 503)
|
||||
|
||||
console.log(JSON.stringify({
|
||||
ok: true,
|
||||
checks: [
|
||||
'pairing_bound_hub_proxy',
|
||||
'internal_bearer_replaced_by_hub',
|
||||
'ontology_mcp_tool_response',
|
||||
'mcp_protocol_header_forwarded',
|
||||
'offline_pairing_is_rejected',
|
||||
],
|
||||
}, null, 2))
|
||||
} finally {
|
||||
if (worker && worker.readyState === WebSocket.OPEN) worker.close()
|
||||
if (hubProcess.exitCode === null && !hubProcess.killed) {
|
||||
hubProcess.kill('SIGTERM')
|
||||
await once(hubProcess, 'exit').catch(() => {})
|
||||
}
|
||||
await new Promise((resolve, reject) => ontologyServer.close((error) => error ? reject(error) : resolve()))
|
||||
if (hubProcess.exitCode && hubProcess.exitCode !== 0) {
|
||||
throw new Error(`hub_smoke_child_failed:${hubOutput.slice(-2000)}`)
|
||||
}
|
||||
}
|
||||
|
||||
async function availablePort() {
|
||||
const server = createServer()
|
||||
await new Promise((resolve) => server.listen(0, '127.0.0.1', resolve))
|
||||
const port = server.address().port
|
||||
await new Promise((resolve, reject) => server.close((error) => error ? reject(error) : resolve()))
|
||||
return port
|
||||
}
|
||||
|
||||
async function waitForHub(url, processRef) {
|
||||
for (let attempt = 0; attempt < 40; attempt += 1) {
|
||||
if (processRef.exitCode !== null) throw new Error('hub_smoke_child_exited_early')
|
||||
try {
|
||||
const response = await fetch(url)
|
||||
if (response.ok) return
|
||||
} catch {}
|
||||
await new Promise((resolve) => setTimeout(resolve, 100))
|
||||
}
|
||||
throw new Error('hub_smoke_health_timeout')
|
||||
}
|
||||
|
|
@ -34,7 +34,6 @@ app.get("/healthz", (_req, res) => {
|
|||
agentsOnline: Array.from(agentsByCode.values()).filter(isAgentOnline).length,
|
||||
assistantRelays: assistantRelaysById.size,
|
||||
internalApiConfigured: config.internalAccessTokens.length > 0,
|
||||
ontologyMcpProxyConfigured: Boolean(config.ontologyCoreUrl && config.ontologyCoreAccessToken),
|
||||
});
|
||||
});
|
||||
|
||||
|
|
@ -86,7 +85,6 @@ app.post("/api/ai-workspace/hub/v1/assistant-relays/:relayId/poll", requireInter
|
|||
app.post("/api/ai-workspace/hub/v1/assistant-relays/:relayId/results/:callId", requireInternalApi, asyncRoute(completeAssistantRelayCall));
|
||||
|
||||
app.use("/api/ai-workspace/hub/v1/ndc-agent-mcp/:pairingCode", asyncRoute(proxyNdcAgentMcp));
|
||||
app.use("/api/ai-workspace/hub/v1/ontology-mcp/:pairingCode", asyncRoute(proxyOntologyMcp));
|
||||
|
||||
app.use((error, _req, res, _next) => {
|
||||
const status = Number(error?.status || 500);
|
||||
|
|
@ -290,60 +288,6 @@ async function proxyNdcAgentMcp(req, res) {
|
|||
res.send(text);
|
||||
}
|
||||
|
||||
async function proxyOntologyMcp(req, res) {
|
||||
if (!config.ontologyCoreUrl || !config.ontologyCoreAccessToken) {
|
||||
res.status(503).json({ ok: false, error: "ontology_mcp_proxy_not_configured" });
|
||||
return;
|
||||
}
|
||||
const pairingCode = cleanPairingCode(req.params.pairingCode);
|
||||
const agent = agentsByCode.get(pairingCode);
|
||||
if (!agent || !isAgentOnline(agent)) {
|
||||
res.status(503).json({ ok: false, error: "bridge_agent_offline" });
|
||||
return;
|
||||
}
|
||||
|
||||
const marker = `/api/ai-workspace/hub/v1/ontology-mcp/${encodeURIComponent(req.params.pairingCode)}`;
|
||||
const suffix = String(req.originalUrl || "").startsWith(marker)
|
||||
? String(req.originalUrl || "").slice(marker.length)
|
||||
: String(req.url || "");
|
||||
const targetUrl = `${config.ontologyCoreUrl.replace(/\/+$/, "")}${suffix || "/mcp"}`;
|
||||
await proxyInternalMcpRequest(req, res, targetUrl, config.ontologyCoreAccessToken);
|
||||
}
|
||||
|
||||
async function proxyInternalMcpRequest(req, res, targetUrl, accessToken) {
|
||||
const method = String(req.method || "GET").toUpperCase();
|
||||
const hasBody = !["GET", "HEAD"].includes(method);
|
||||
const upstream = await fetch(targetUrl, {
|
||||
method,
|
||||
redirect: "manual",
|
||||
headers: {
|
||||
Accept: String(req.headers.accept || "application/json, text/event-stream"),
|
||||
Authorization: `Bearer ${accessToken}`,
|
||||
...forwardMcpHeaders(req),
|
||||
...(hasBody ? { "Content-Type": "application/json" } : {}),
|
||||
},
|
||||
...(hasBody ? { body: JSON.stringify(req.body || {}) } : {}),
|
||||
});
|
||||
const contentType = upstream.headers.get("content-type") || "application/json; charset=utf-8";
|
||||
const text = await upstream.text();
|
||||
res.status(upstream.status);
|
||||
res.setHeader("content-type", contentType);
|
||||
for (const header of ["cache-control", "mcp-protocol-version", "mcp-session-id", "vary"]) {
|
||||
const value = upstream.headers.get(header);
|
||||
if (value) res.setHeader(header, value);
|
||||
}
|
||||
res.send(text);
|
||||
}
|
||||
|
||||
function forwardMcpHeaders(req) {
|
||||
const headers = {};
|
||||
for (const name of ["mcp-protocol-version", "mcp-session-id", "last-event-id"]) {
|
||||
const value = cleanString(req.headers[name], 1000);
|
||||
if (value) headers[name] = value;
|
||||
}
|
||||
return headers;
|
||||
}
|
||||
|
||||
async function proxyAssistantActions(req, res) {
|
||||
if (!config.assistantInternalUrl || !config.assistantInternalAccessToken) {
|
||||
res.status(503).json({ ok: false, error: "assistant_action_proxy_not_configured" });
|
||||
|
|
@ -849,13 +793,6 @@ function readConfig() {
|
|||
1000,
|
||||
).replace(/\/+$/, ""),
|
||||
assistantInternalAccessToken: cleanString(process.env.NODEDC_INTERNAL_ACCESS_TOKEN, 1000),
|
||||
ontologyCoreUrl: cleanString(
|
||||
process.env.NODEDC_ONTOLOGY_CORE_URL ||
|
||||
process.env.NDC_ONTOLOGY_CORE_URL ||
|
||||
"http://ontology-core:18104",
|
||||
1000,
|
||||
).replace(/\/+$/, ""),
|
||||
ontologyCoreAccessToken: cleanString(process.env.NODEDC_INTERNAL_ACCESS_TOKEN, 1000),
|
||||
};
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -1,3 +0,0 @@
|
|||
# Stable LAN address of the AMD Windows host. This is not the OpenVPN address.
|
||||
AMD_CONNECTOR_BIND_IP=172.22.0.183
|
||||
AMD_CONNECTOR_PORT=8791
|
||||
|
|
@ -1,2 +0,0 @@
|
|||
.env
|
||||
runtime/
|
||||
|
|
@ -1,12 +0,0 @@
|
|||
FROM node:20-alpine
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
COPY package.json ./
|
||||
COPY server.mjs ./
|
||||
|
||||
USER node
|
||||
|
||||
EXPOSE 8791
|
||||
|
||||
CMD ["node", "server.mjs"]
|
||||
|
|
@ -1,84 +0,0 @@
|
|||
# DC AMD Connector
|
||||
|
||||
DC AMD Connector runs on the adjacent AMD Windows machine. It is a restricted
|
||||
HTTP `CONNECT` proxy for the NODE.DC map path:
|
||||
|
||||
```text
|
||||
NAS DC AMD Proxy -> 172.22.0.183:8791 -> AMD VPN -> Cesium / Bing
|
||||
```
|
||||
|
||||
It is not a general-purpose proxy:
|
||||
|
||||
- only authenticated `CONNECT` requests are accepted;
|
||||
- only port `443` is accepted;
|
||||
- only Cesium Ion and required Bing imagery hosts are accepted;
|
||||
- ordinary HTTP proxy requests are rejected;
|
||||
- the Ion bearer token stays inside the end-to-end TLS connection from NAS to
|
||||
Cesium and is not processed or logged by this connector.
|
||||
|
||||
The Windows host owns the VPN. The NAS uses only the stable LAN address in
|
||||
`.env`, never an OpenVPN adapter address or a changing public VPN exit IP.
|
||||
|
||||
## First run on the AMD machine
|
||||
|
||||
1. Start Docker Desktop and wait for its engine to be running.
|
||||
2. Open **PowerShell as Administrator** and run `install.ps1` from this
|
||||
package folder. It verifies source checksums, copies the package to
|
||||
`C:\\NODEDC\\dc-amd-connector`, creates a local random access token without
|
||||
printing it, binds port `8791` only to `172.22.0.183`, scopes Windows
|
||||
Firewall to NAS `172.22.0.222`, constrains the container to one CPU, 256 MB,
|
||||
64 processes and 4096 file descriptors, and starts the container. It also
|
||||
adds a launcher to the current user's Windows Startup folder. At the next
|
||||
user sign-in it uses the Docker Desktop CLI when available, otherwise its
|
||||
installed application; the `unless-stopped` restart policy then restores
|
||||
this container.
|
||||
|
||||
```powershell
|
||||
powershell -NoProfile -ExecutionPolicy Bypass -File .\\install.ps1
|
||||
```
|
||||
|
||||
3. Do not send the runtime secret in chat or put it into a source file. The
|
||||
next NAS patch will provision the same value in a root-owned secret file
|
||||
through an operator-only transfer.
|
||||
|
||||
Docker Desktop on the WSL 2 backend is a desktop application, not an
|
||||
unattended Windows boot daemon. Thus this recovery guarantee starts after the
|
||||
configured Windows user signs in. Before a sign-in, the map must use its
|
||||
offline/tile-cache path. No Windows route, VPN, DNS, proxy setting or other
|
||||
application's traffic is modified.
|
||||
|
||||
## Operations and migration
|
||||
|
||||
Run these commands from an elevated PowerShell window:
|
||||
|
||||
```powershell
|
||||
cd C:\\NODEDC\\dc-amd-connector
|
||||
docker compose ps
|
||||
docker compose logs --tail 100
|
||||
docker compose restart
|
||||
```
|
||||
|
||||
`docker compose down` is an intentional stop and removes the container; restore
|
||||
it with `docker compose up -d`. The auto-start launcher is removed by
|
||||
`uninstall.ps1`.
|
||||
|
||||
To move to a different adjacent Windows host: install this same package there
|
||||
with its stable LAN address (`-BindAddress`) and verify its VPN tunnel first.
|
||||
It creates a fresh local access secret. Only then may the separate NAS pairing
|
||||
patch be changed to the new host; never run two active pairings against the
|
||||
same NAS endpoint.
|
||||
|
||||
## Verification
|
||||
|
||||
The connector health check uses the local runtime secret inside the container.
|
||||
It is expected to become `healthy` without making any external request. The
|
||||
following local check proves a permitted TLS tunnel through the current AMD VPN
|
||||
without revealing the access token:
|
||||
|
||||
```powershell
|
||||
docker exec dc-amd-connector node -e "const fs=require('fs'),net=require('net');const t=fs.readFileSync('/run/dc-amd-secrets/connector-access','utf8').trim();const s=net.connect(8791,'127.0.0.1',()=>s.write('CONNECT api.cesium.com:443 HTTP/1.1\r\nHost: api.cesium.com:443\r\nProxy-Authorization: Bearer '+t+'\r\n\r\n'));s.once('data',d=>{const v=d.toString('ascii');console.log(v.split('\r\n')[0]);s.destroy();process.exit(v.startsWith('HTTP/1.1 200')?0:1)});s.on('error',e=>{console.error(e.message);process.exit(1)})"
|
||||
```
|
||||
|
||||
Expected output is `HTTP/1.1 200 Connection Established`. If the AMD VPN is
|
||||
off, this test must fail; it must never silently move live Cesium traffic to
|
||||
the NAS.
|
||||
|
|
@ -1,10 +0,0 @@
|
|||
36283c8f926cd5a2fdfec0adda20e1d30fefb47e8fb6f4c6a41ebdfdb8034357 ./.env.example
|
||||
7e4d35564197224edd4899aae9525c5b3c0c904371bb2e635dcef7950bc776cd ./.gitignore
|
||||
3d7464d3a7b97b3b9be1036dcf77a9fd31cd841043a64069870e001a7f526a7b ./Dockerfile
|
||||
c967a21fbb804a61e467fd207702f328ccc5ed933beaf98a476490f033f6dfa4 ./README.md
|
||||
08708bd47ddf85dadda552eeb2ac1bc8f8d1e5d119305ef41bb0327b4dbac36b ./VERSION
|
||||
b00bbf2ec6b56d834b04543d4e65bda7eff97296d4d255df48c28d71369a3f0e ./docker-compose.yml
|
||||
608e3c75b69567376019b6f9fdf274a5cf9afecdde07348b6f02c89b688e672a ./install.ps1
|
||||
33d4c94fdec3b47bba25039306423dd9dea50d4b36014a5a34768e0a6a7ba174 ./package.json
|
||||
e08d0f33b4749289a5adf897b8a4ef876d3ebad9aa377d992e9d33b1060fe698 ./server.mjs
|
||||
f9bb0a9d862e9f78fd533eeb2a0750108392d73f6173aec2ccc8053120638e01 ./uninstall.ps1
|
||||
|
|
@ -1 +0,0 @@
|
|||
dc-amd-connector-20260715-004
|
||||
|
|
@ -1,47 +0,0 @@
|
|||
services:
|
||||
dc-amd-connector:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: Dockerfile
|
||||
container_name: dc-amd-connector
|
||||
image: nodedc/dc-amd-connector:local
|
||||
restart: unless-stopped
|
||||
# This is a byte-forwarding gateway, not a map renderer. These are circuit
|
||||
# breakers for the shared workstation, not normal throughput limits.
|
||||
cpus: "1.00"
|
||||
mem_limit: 256m
|
||||
pids_limit: 64
|
||||
ulimits:
|
||||
nofile:
|
||||
soft: 4096
|
||||
hard: 4096
|
||||
env_file:
|
||||
- .env
|
||||
environment:
|
||||
PORT: "8791"
|
||||
AMD_CONNECTOR_ACCESS_TOKEN_FILE: /run/dc-amd-secrets/connector-access
|
||||
# Bind only to the stable LAN address of the AMD host, never all Windows
|
||||
# interfaces and never the OpenVPN adapter address.
|
||||
ports:
|
||||
- "${AMD_CONNECTOR_BIND_IP:?set AMD_CONNECTOR_BIND_IP}:${AMD_CONNECTOR_PORT:-8791}:8791"
|
||||
volumes:
|
||||
- type: bind
|
||||
source: ./runtime/connector-access
|
||||
target: /run/dc-amd-secrets/connector-access
|
||||
read_only: true
|
||||
read_only: true
|
||||
tmpfs:
|
||||
- /tmp:rw,noexec,nosuid,size=8m
|
||||
security_opt:
|
||||
- no-new-privileges:true
|
||||
cap_drop:
|
||||
- ALL
|
||||
healthcheck:
|
||||
test:
|
||||
- CMD-SHELL
|
||||
- >
|
||||
node -e "const fs=require('fs'),http=require('http');const t=fs.readFileSync('/run/dc-amd-secrets/connector-access','utf8').trim();const r=http.get({host:'127.0.0.1',port:8791,path:'/healthz',headers:{'proxy-authorization':'Bearer '+t}},x=>process.exit(x.statusCode===200?0:1));r.on('error',()=>process.exit(1))"
|
||||
interval: 30s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
start_period: 10s
|
||||
|
|
@ -1,170 +0,0 @@
|
|||
[CmdletBinding()]
|
||||
param(
|
||||
[string]$Destination = 'C:\NODEDC\dc-amd-connector',
|
||||
[string]$NasAddress = '172.22.0.222',
|
||||
[string]$BindAddress = '172.22.0.183',
|
||||
[int]$Port = 8791,
|
||||
[bool]$EnableAutoStart = $true
|
||||
)
|
||||
|
||||
Set-StrictMode -Version Latest
|
||||
$ErrorActionPreference = 'Stop'
|
||||
|
||||
function Require-Administrator {
|
||||
$identity = [Security.Principal.WindowsIdentity]::GetCurrent()
|
||||
$principal = New-Object Security.Principal.WindowsPrincipal($identity)
|
||||
if (-not $principal.IsInRole([Security.Principal.WindowsBuiltInRole]::Administrator)) {
|
||||
throw 'Run this installer from an elevated PowerShell window.'
|
||||
}
|
||||
}
|
||||
|
||||
function Test-PackageChecksums([string]$Root) {
|
||||
$manifest = Join-Path $Root 'SHA256SUMS'
|
||||
if (-not (Test-Path -LiteralPath $manifest -PathType Leaf)) {
|
||||
throw "Checksum manifest not found: $manifest"
|
||||
}
|
||||
|
||||
foreach ($line in Get-Content -LiteralPath $manifest) {
|
||||
if ([string]::IsNullOrWhiteSpace($line)) { continue }
|
||||
$parts = $line -split '\s{2,}', 2
|
||||
if ($parts.Count -ne 2 -or $parts[0] -notmatch '^[a-f0-9]{64}$') {
|
||||
throw "Invalid checksum line: $line"
|
||||
}
|
||||
$relative = $parts[1] -replace '^\./', ''
|
||||
$file = Join-Path $Root $relative
|
||||
$actual = (Get-FileHash -LiteralPath $file -Algorithm SHA256).Hash.ToLowerInvariant()
|
||||
if ($actual -ne $parts[0]) { throw "Checksum mismatch: $relative" }
|
||||
}
|
||||
}
|
||||
|
||||
function Get-DockerDesktopStartCommand([string]$DockerCliPath) {
|
||||
& $DockerCliPath desktop start --help 2>$null | Out-Null
|
||||
if ($LASTEXITCODE -eq 0) {
|
||||
return "`"$DockerCliPath`" desktop start --detach"
|
||||
}
|
||||
|
||||
$dockerRoot = Split-Path (Split-Path (Split-Path $DockerCliPath -Parent) -Parent) -Parent
|
||||
$candidates = @(
|
||||
(Join-Path $dockerRoot 'Docker Desktop.exe'),
|
||||
(Join-Path $env:ProgramFiles 'Docker\Docker\Docker Desktop.exe'),
|
||||
(Join-Path $env:LOCALAPPDATA 'Programs\Docker\Docker\Docker Desktop.exe')
|
||||
) | Select-Object -Unique
|
||||
$desktopExe = $candidates | Where-Object { Test-Path -LiteralPath $_ -PathType Leaf } | Select-Object -First 1
|
||||
if (-not $desktopExe) {
|
||||
throw 'Unable to find Docker Desktop start command or Docker Desktop.exe for restart recovery.'
|
||||
}
|
||||
return "start `"`" `"$desktopExe`""
|
||||
}
|
||||
|
||||
function New-DockerDesktopAutoStart([string]$StartCommand) {
|
||||
$startup = [Environment]::GetFolderPath([Environment+SpecialFolder]::Startup)
|
||||
if ([string]::IsNullOrWhiteSpace($startup)) { throw 'Unable to resolve the current user Startup folder.' }
|
||||
|
||||
$launcher = Join-Path $startup 'NODE.DC DC AMD Connector - Docker Desktop.cmd'
|
||||
if (Test-Path -LiteralPath $launcher) {
|
||||
throw "Docker Desktop auto-start launcher already exists: $launcher"
|
||||
}
|
||||
|
||||
@(
|
||||
'@echo off',
|
||||
'rem NODE.DC DC AMD Connector - starts Docker Desktop after this user signs in.',
|
||||
$StartCommand
|
||||
) | Set-Content -LiteralPath $launcher -Encoding ascii
|
||||
return $launcher
|
||||
}
|
||||
|
||||
Require-Administrator
|
||||
$source = Split-Path -Parent $PSCommandPath
|
||||
if (-not (Test-Path -LiteralPath (Join-Path $source 'VERSION') -PathType Leaf)) {
|
||||
throw 'Run install.ps1 from the DC AMD Connector package folder.'
|
||||
}
|
||||
Test-PackageChecksums $source
|
||||
|
||||
if (Test-Path -LiteralPath $Destination) {
|
||||
throw "Destination already exists; refusing to overwrite: $Destination"
|
||||
}
|
||||
|
||||
$null = [Net.IPAddress]::Parse($NasAddress)
|
||||
$null = [Net.IPAddress]::Parse($BindAddress)
|
||||
if ($Port -lt 1 -or $Port -gt 65535) {
|
||||
throw "Port must be between 1 and 65535: $Port"
|
||||
}
|
||||
|
||||
$dockerCli = (Get-Command docker -CommandType Application -ErrorAction Stop | Select-Object -First 1).Path
|
||||
if ([string]::IsNullOrWhiteSpace($dockerCli) -or -not (Test-Path -LiteralPath $dockerCli -PathType Leaf)) {
|
||||
throw 'Docker CLI executable could not be resolved.'
|
||||
}
|
||||
& $dockerCli version --format '{{.Server.Os}}/{{.Server.Arch}} {{.Server.Version}}' | Out-Null
|
||||
if ($LASTEXITCODE -ne 0) {
|
||||
throw 'Docker Desktop engine is not available. Start it and wait for Engine running.'
|
||||
}
|
||||
$dockerDesktopStartCommand = Get-DockerDesktopStartCommand $dockerCli
|
||||
|
||||
$destinationCreated = $false
|
||||
$firewallCreated = $false
|
||||
$autoStartLauncher = $null
|
||||
$parent = Split-Path -Parent $Destination
|
||||
try {
|
||||
New-Item -ItemType Directory -Force $parent | Out-Null
|
||||
Copy-Item -LiteralPath $source -Destination $Destination -Recurse
|
||||
$destinationCreated = $true
|
||||
Test-PackageChecksums $Destination
|
||||
Set-Location $Destination
|
||||
|
||||
New-Item -ItemType Directory -Force runtime | Out-Null
|
||||
$tokenPath = Join-Path $Destination 'runtime\connector-access'
|
||||
if (Test-Path -LiteralPath $tokenPath) { throw "Runtime token already exists: $tokenPath" }
|
||||
$bytes = New-Object byte[] 48
|
||||
$rng = [System.Security.Cryptography.RandomNumberGenerator]::Create()
|
||||
try { $rng.GetBytes($bytes) } finally { $rng.Dispose() }
|
||||
$token = [Convert]::ToBase64String($bytes).TrimEnd('=').Replace('+', '-').Replace('/', '_')
|
||||
[IO.File]::WriteAllText($tokenPath, $token, [Text.UTF8Encoding]::new($false))
|
||||
|
||||
Copy-Item -LiteralPath (Join-Path $Destination '.env.example') -Destination (Join-Path $Destination '.env')
|
||||
@(
|
||||
"AMD_CONNECTOR_BIND_IP=$BindAddress",
|
||||
"AMD_CONNECTOR_PORT=$Port"
|
||||
) | Set-Content -LiteralPath (Join-Path $Destination '.env') -Encoding ascii
|
||||
|
||||
$ruleName = 'NODE.DC DC AMD Connector (NAS only)'
|
||||
if (Get-NetFirewallRule -DisplayName $ruleName -ErrorAction SilentlyContinue) {
|
||||
throw "Firewall rule already exists; refusing to replace: $ruleName"
|
||||
}
|
||||
New-NetFirewallRule -DisplayName $ruleName -Direction Inbound -Action Allow -Protocol TCP -Profile Any -LocalAddress $BindAddress -LocalPort $Port -RemoteAddress $NasAddress | Out-Null
|
||||
$firewallCreated = $true
|
||||
|
||||
if ($EnableAutoStart) {
|
||||
$autoStartLauncher = New-DockerDesktopAutoStart $dockerDesktopStartCommand
|
||||
}
|
||||
|
||||
& $dockerCli compose up -d --build
|
||||
$state = ''
|
||||
foreach ($attempt in 1..30) {
|
||||
$state = & $dockerCli inspect dc-amd-connector --format '{{.State.Status}} health={{if .State.Health}}{{.State.Health.Status}}{{end}}'
|
||||
if ($LASTEXITCODE -ne 0) { throw 'Container inspection failed after Docker Compose apply.' }
|
||||
if ($state -eq 'running health=healthy') { break }
|
||||
Start-Sleep -Seconds 2
|
||||
}
|
||||
if ($state -ne 'running health=healthy') {
|
||||
throw "Connector did not become healthy: $state"
|
||||
}
|
||||
|
||||
Write-Host "DC AMD Connector installed: $state"
|
||||
Write-Host "Bound to $BindAddress`:$Port; inbound firewall scope is NAS $NasAddress."
|
||||
if ($autoStartLauncher) {
|
||||
Write-Host "Docker Desktop will start at the next sign-in through: $autoStartLauncher"
|
||||
}
|
||||
Write-Host 'The connector access token remains only in runtime\connector-access. Do not print or send it in chat.'
|
||||
}
|
||||
catch {
|
||||
if ($autoStartLauncher -and (Test-Path -LiteralPath $autoStartLauncher)) {
|
||||
Remove-Item -LiteralPath $autoStartLauncher -Force -ErrorAction SilentlyContinue
|
||||
}
|
||||
if ($firewallCreated) {
|
||||
Remove-NetFirewallRule -DisplayName 'NODE.DC DC AMD Connector (NAS only)' -ErrorAction SilentlyContinue
|
||||
}
|
||||
if ($destinationCreated -and (Test-Path -LiteralPath $Destination)) {
|
||||
Remove-Item -LiteralPath $Destination -Recurse -Force -ErrorAction SilentlyContinue
|
||||
}
|
||||
throw
|
||||
}
|
||||
|
|
@ -1,9 +0,0 @@
|
|||
{
|
||||
"name": "dc-amd-connector",
|
||||
"version": "0.1.0",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"start": "node server.mjs"
|
||||
}
|
||||
}
|
||||
|
|
@ -1,130 +0,0 @@
|
|||
import { createServer } from "node:http";
|
||||
import { readFile } from "node:fs/promises";
|
||||
import { connect } from "node:net";
|
||||
import { timingSafeEqual } from "node:crypto";
|
||||
|
||||
const allowedHosts = new Set([
|
||||
"api.cesium.com",
|
||||
"assets.ion.cesium.com",
|
||||
"dev.virtualearth.net",
|
||||
"ecn.t0.tiles.virtualearth.net",
|
||||
"ecn.t1.tiles.virtualearth.net",
|
||||
"ecn.t2.tiles.virtualearth.net",
|
||||
"ecn.t3.tiles.virtualearth.net",
|
||||
]);
|
||||
const config = await readConfig();
|
||||
|
||||
const server = createServer((request, response) => {
|
||||
if (!isAuthorized(request.headers["proxy-authorization"])) return writeJson(response, 401, { ok: false, error: "connector_unauthorized" }, { "proxy-authenticate": "Bearer" });
|
||||
if (request.method === "GET" && request.url === "/healthz") return writeJson(response, 200, { ok: true, service: "dc-amd-connector", mode: "restricted-connect", allowedHosts: allowedHosts.size });
|
||||
return writeJson(response, 405, { ok: false, error: "connect_only" });
|
||||
});
|
||||
|
||||
server.on("connect", (request, clientSocket, head) => {
|
||||
if (!isAuthorized(request.headers["proxy-authorization"])) return rejectTunnel(clientSocket, 407, "Proxy Authentication Required", { "Proxy-Authenticate": "Bearer" });
|
||||
|
||||
let target;
|
||||
try {
|
||||
target = parseConnectTarget(request.url || "");
|
||||
} catch (error) {
|
||||
log("warn", "connect_rejected", { reason: error.message });
|
||||
return rejectTunnel(clientSocket, 403, "Forbidden");
|
||||
}
|
||||
|
||||
const upstream = connect({ host: target.host, port: target.port });
|
||||
let settled = false;
|
||||
const timeout = setTimeout(() => upstream.destroy(new Error("upstream_connect_timeout")), config.connectTimeoutMs);
|
||||
clientSocket.setTimeout(config.idleTimeoutMs, () => clientSocket.destroy());
|
||||
upstream.setTimeout(config.idleTimeoutMs, () => upstream.destroy());
|
||||
|
||||
upstream.once("connect", () => {
|
||||
settled = true;
|
||||
clearTimeout(timeout);
|
||||
clientSocket.write("HTTP/1.1 200 Connection Established\r\nProxy-Agent: dc-amd-connector\r\n\r\n");
|
||||
if (head?.length) upstream.write(head);
|
||||
clientSocket.pipe(upstream);
|
||||
upstream.pipe(clientSocket);
|
||||
log("info", "connect_established", { host: target.host, port: target.port });
|
||||
});
|
||||
upstream.once("error", (error) => {
|
||||
clearTimeout(timeout);
|
||||
if (!settled) {
|
||||
log("warn", "connect_failed", { host: target.host, port: target.port, reason: sanitizeError(error) });
|
||||
rejectTunnel(clientSocket, 502, "Bad Gateway");
|
||||
}
|
||||
});
|
||||
clientSocket.once("error", () => upstream.destroy());
|
||||
clientSocket.once("close", () => upstream.destroy());
|
||||
upstream.once("close", () => clientSocket.destroy());
|
||||
});
|
||||
|
||||
server.on("clientError", (_error, socket) => rejectTunnel(socket, 400, "Bad Request"));
|
||||
server.listen(config.port, "0.0.0.0", () => log("info", "connector_started", { port: config.port, allowedHosts: allowedHosts.size }));
|
||||
|
||||
for (const signal of ["SIGINT", "SIGTERM"]) process.on(signal, () => server.close(() => process.exit(0)));
|
||||
|
||||
async function readConfig() {
|
||||
const port = parsePort(process.env.PORT, 8791);
|
||||
const tokenFile = String(process.env.AMD_CONNECTOR_ACCESS_TOKEN_FILE || "/run/dc-amd-secrets/connector-access").trim();
|
||||
const token = String(await readFile(tokenFile, "utf8")).trim();
|
||||
if (!/^[A-Za-z0-9_-]{48,256}$/.test(token)) throw new Error("connector_access_token_invalid");
|
||||
return {
|
||||
port,
|
||||
token: Buffer.from(token, "utf8"),
|
||||
connectTimeoutMs: parseDuration(process.env.AMD_CONNECTOR_CONNECT_TIMEOUT_SECONDS, 20),
|
||||
idleTimeoutMs: parseDuration(process.env.AMD_CONNECTOR_IDLE_TIMEOUT_SECONDS, 90),
|
||||
};
|
||||
}
|
||||
|
||||
function parsePort(raw, fallback) {
|
||||
const value = Number(String(raw || fallback).trim());
|
||||
if (!Number.isInteger(value) || value < 1024 || value > 65535) throw new Error("connector_port_invalid");
|
||||
return value;
|
||||
}
|
||||
|
||||
function parseDuration(raw, fallbackSeconds) {
|
||||
const seconds = Number(String(raw || fallbackSeconds).trim());
|
||||
if (!Number.isInteger(seconds) || seconds < 1 || seconds > 600) throw new Error("connector_timeout_invalid");
|
||||
return seconds * 1000;
|
||||
}
|
||||
|
||||
function isAuthorized(rawHeader) {
|
||||
const match = /^Bearer\s+([A-Za-z0-9_-]{48,256})$/i.exec(String(rawHeader || "").trim());
|
||||
if (!match) return false;
|
||||
const candidate = Buffer.from(match[1], "utf8");
|
||||
return candidate.length === config.token.length && timingSafeEqual(candidate, config.token);
|
||||
}
|
||||
|
||||
function parseConnectTarget(raw) {
|
||||
const match = /^([A-Za-z0-9.-]{1,253}):(443)$/.exec(String(raw || "").trim());
|
||||
if (!match) throw new Error("connect_target_invalid");
|
||||
const host = match[1].toLowerCase();
|
||||
if (!allowedHosts.has(host)) throw new Error("connect_target_not_allowed");
|
||||
return { host, port: Number(match[2]) };
|
||||
}
|
||||
|
||||
function rejectTunnel(socket, status, message, headers = {}) {
|
||||
if (!socket || socket.destroyed) return;
|
||||
const extra = Object.entries(headers).map(([name, value]) => `${name}: ${value}\r\n`).join("");
|
||||
socket.end(`HTTP/1.1 ${status} ${message}\r\n${extra}Connection: close\r\nContent-Length: 0\r\n\r\n`);
|
||||
}
|
||||
|
||||
function writeJson(response, status, body, headers = {}) {
|
||||
const payload = JSON.stringify(body);
|
||||
response.writeHead(status, {
|
||||
"content-type": "application/json; charset=utf-8",
|
||||
"content-length": Buffer.byteLength(payload),
|
||||
"cache-control": "no-store",
|
||||
...headers,
|
||||
});
|
||||
response.end(payload);
|
||||
}
|
||||
|
||||
function sanitizeError(error) {
|
||||
const message = String(error?.message || "connector_error");
|
||||
return message.replace(/[^A-Za-z0-9_.:-]/g, "_").slice(0, 120);
|
||||
}
|
||||
|
||||
function log(level, event, fields = {}) {
|
||||
console.log(JSON.stringify({ ts: new Date().toISOString(), level, event, ...fields }));
|
||||
}
|
||||
|
|
@ -1,30 +0,0 @@
|
|||
[CmdletBinding()]
|
||||
param(
|
||||
[string]$Destination = 'C:\NODEDC\dc-amd-connector'
|
||||
)
|
||||
|
||||
Set-StrictMode -Version Latest
|
||||
$ErrorActionPreference = 'Stop'
|
||||
|
||||
function Require-Administrator {
|
||||
$identity = [Security.Principal.WindowsIdentity]::GetCurrent()
|
||||
$principal = New-Object Security.Principal.WindowsPrincipal($identity)
|
||||
if (-not $principal.IsInRole([Security.Principal.WindowsBuiltInRole]::Administrator)) {
|
||||
throw 'Run this uninstaller from an elevated PowerShell window.'
|
||||
}
|
||||
}
|
||||
|
||||
Require-Administrator
|
||||
$docker = Get-Command docker -CommandType Application -ErrorAction SilentlyContinue | Select-Object -First 1
|
||||
if ($docker -and (Test-Path -LiteralPath (Join-Path $Destination 'docker-compose.yml'))) {
|
||||
Push-Location $Destination
|
||||
try { & $docker.Path compose down --remove-orphans } finally { Pop-Location }
|
||||
}
|
||||
|
||||
Remove-NetFirewallRule -DisplayName 'NODE.DC DC AMD Connector (NAS only)' -ErrorAction SilentlyContinue
|
||||
$startup = [Environment]::GetFolderPath([Environment+SpecialFolder]::Startup)
|
||||
$launcher = Join-Path $startup 'NODE.DC DC AMD Connector - Docker Desktop.cmd'
|
||||
Remove-Item -LiteralPath $launcher -Force -ErrorAction SilentlyContinue
|
||||
Remove-Item -LiteralPath $Destination -Recurse -Force -ErrorAction SilentlyContinue
|
||||
|
||||
Write-Host 'DC AMD Connector removed. Docker Desktop and unrelated containers were not changed.'
|
||||
|
|
@ -1,12 +0,0 @@
|
|||
FROM node:20-alpine
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
COPY package.json ./
|
||||
COPY server.mjs ./
|
||||
|
||||
USER node
|
||||
|
||||
EXPOSE 8790
|
||||
|
||||
CMD ["node", "server.mjs"]
|
||||
|
|
@ -1,770 +0,0 @@
|
|||
# DC AMD Proxy — production operations and recovery canon
|
||||
|
||||
This document is the reproducible operations contract for the restricted
|
||||
Cesium/Bing egress path through the neighbouring Windows AMD workstation. It
|
||||
describes the implementation that is in source now. It is not a proposal for
|
||||
turning the NAS or the workstation into a general-purpose proxy.
|
||||
|
||||
The two non-negotiable boundaries are:
|
||||
|
||||
1. the NAS keeps its own routes, DNS, VPN, Tailscale and Internet connectivity
|
||||
unchanged;
|
||||
2. only approved Cesium/Bing HTTPS traffic may leave through the AMD host.
|
||||
|
||||
Do not solve an incident by adding a system proxy, default route, transparent
|
||||
NAT, browser-visible token, permissive CONNECT endpoint, `0.0.0.0` Windows
|
||||
bind, or direct NAS provider fallback.
|
||||
|
||||
## 1. Canonical architecture
|
||||
|
||||
```text
|
||||
Browser
|
||||
-> Foundry BFF
|
||||
-> Platform Map Gateway
|
||||
|-- warm object: NAS live/offline cache -> browser
|
||||
`-- cold/refresh object:
|
||||
HTTP + x-proxy-token
|
||||
-> NAS <NAS_LAN_IP>:<NAS_PROXY_PORT> dc-amd-proxy
|
||||
-> authenticated HTTP CONNECT
|
||||
-> AMD <AMD_LAN_IP>:<CONNECTOR_PORT> dc-amd-connector
|
||||
-> AMD/Docker DNS
|
||||
-> Windows VPN/default egress route
|
||||
-> approved Cesium/Bing host:443
|
||||
<- end-to-end provider TLS is terminated by dc-amd-proxy
|
||||
<- streamed provider response
|
||||
-> browser and, when enabled, atomic NAS cache fill
|
||||
```
|
||||
|
||||
The AMD connector is a byte-forwarding CONNECT boundary. It does not parse an
|
||||
Ion bearer and cannot read the provider TLS stream. `dc-amd-proxy` establishes
|
||||
TLS through that tunnel, validates the provider certificate and SNI, follows
|
||||
only approved redirects, and returns the response to Map Gateway.
|
||||
|
||||
This distinction is operationally important:
|
||||
|
||||
- a **warm cache hit does not traverse AMD, VPN or Cesium**;
|
||||
- a **cold miss or refresh does traverse AMD and depends on the workstation's
|
||||
Docker Desktop, DNS, routing and VPN**;
|
||||
- objects already cached on NAS must remain usable when AMD or its VPN is
|
||||
unavailable;
|
||||
- `dc-amd-proxy` is not in the browser-to-NAS cache read path.
|
||||
|
||||
### Current components and ownership
|
||||
|
||||
| Component | Runtime | Canonical state | Responsibility |
|
||||
| --- | --- | --- | --- |
|
||||
| Map Gateway | Synology Docker | `/volume1/docker/nodedc-platform/platform/services/map-gateway` | Provider credential scope, cache-first reads, streaming cache fill and private egress calls |
|
||||
| `dc-amd-proxy` | Synology Docker | `/volume1/docker/dc-amd-proxy` | Request authorization, target allowlist, CONNECT/TLS pool, redirect/retry policy and telemetry |
|
||||
| Proxy pair state | Synology bind mount | `/volume1/docker/dc-amd-proxy/runtime/connector-access` | Paired connector credential; persistent across container recreation |
|
||||
| Map egress credential | Synology root-owned file | `/volume1/docker/nodedc-platform/secrets/map-egress-proxy-token` | Authenticates Map Gateway to `dc-amd-proxy` |
|
||||
| `dc-amd-connector` | AMD Windows Docker Desktop | `C:\NODEDC\dc-amd-connector` by default | Restricted authenticated CONNECT to approved hosts on port 443 |
|
||||
| Connector credential | AMD local file | `C:\NODEDC\dc-amd-connector\runtime\connector-access` | Generated locally once; never printed or stored in source |
|
||||
| Docker Desktop launcher | Current Windows user's Startup folder | `NODE.DC DC AMD Connector - Docker Desktop.cmd` | Starts Docker Desktop after that user signs in |
|
||||
|
||||
Current source defaults are NAS listener port `8790`, AMD connector port
|
||||
`8791`, eight active sockets and four idle sockets per origin. IP addresses
|
||||
are deployment values and must be treated as configuration, not universal
|
||||
constants.
|
||||
|
||||
## 2. Trust, secrets and traffic boundaries
|
||||
|
||||
There are three different credentials. They are not interchangeable.
|
||||
|
||||
1. The Cesium Ion credential belongs to Map Gateway provider configuration.
|
||||
It must not be placed in Foundry browser code, an artifact or this proxy's
|
||||
Compose file.
|
||||
2. The Map egress credential authenticates Map Gateway to `dc-amd-proxy` using
|
||||
`x-proxy-token`. The canonical runner synchronizes it into the root-owned
|
||||
file mounted read-only into the two services.
|
||||
3. The connector credential authenticates `dc-amd-proxy` to the Windows
|
||||
connector using `Proxy-Authorization: Bearer ...`. The Windows installer
|
||||
generates it locally; the one-time pair request stores the same value in
|
||||
the NAS runtime bind mount.
|
||||
|
||||
No command in this runbook prints, copies to chat, or accepts a secret as a
|
||||
command-line argument.
|
||||
|
||||
### Current runner dependency that must not be mistaken for a traffic route
|
||||
|
||||
The deploy runner currently seeds the shared Map egress credential from the
|
||||
root-owned `PROXY_TOKEN` in `/volume1/docker/proxy-contur/.env`. Therefore a
|
||||
valid, safely permissioned historical `proxy-contur` environment file is a
|
||||
**deploy-time precondition** for `dc-amd-proxy` and Map Gateway applies.
|
||||
|
||||
This does **not** mean Cesium traffic passes through `proxy-contur`. Runtime
|
||||
traffic goes directly from Map Gateway to `dc-amd-proxy`, then to the AMD
|
||||
connector. Removing this legacy secret-seeding dependency is an explicit
|
||||
evolution item later in this document.
|
||||
|
||||
### NAS network boundary
|
||||
|
||||
`dc-amd-proxy` uses `network_mode: host` so the one-time pair endpoint can see
|
||||
the real AMD source address. Synology's published-port bridge path would hide
|
||||
that source behind Docker NAT. Host mode does not make the service broad:
|
||||
|
||||
- the process binds only `<NAS_LAN_IP>:<NAS_PROXY_PORT>`;
|
||||
- `/api/pair` accepts only `<AMD_LAN_IP>` and only while unpaired;
|
||||
- `/proxy/cesium/fetch` accepts only `GET`/`HEAD`, requires the private Map
|
||||
egress credential and validates the complete target URL;
|
||||
- `/healthz` and `/status` expose safe state/counters but no credentials or
|
||||
target URLs;
|
||||
- the container is read-only, drops all capabilities, enables
|
||||
`no-new-privileges`, and has only a small no-exec tmpfs plus the two explicit
|
||||
bind mounts.
|
||||
|
||||
The current Map Gateway URL is a NAS-LAN address, not a public hostname and
|
||||
not a Docker service discovery name.
|
||||
|
||||
### Windows network boundary
|
||||
|
||||
The connector publishes only `<AMD_LAN_IP>:<CONNECTOR_PORT>`. The installer
|
||||
creates one inbound Windows Firewall rule with all of these restrictions:
|
||||
|
||||
- TCP only;
|
||||
- local address `<AMD_LAN_IP>`;
|
||||
- local port `<CONNECTOR_PORT>`;
|
||||
- remote address `<NAS_LAN_IP>`;
|
||||
- no bind to the VPN adapter and no bind to all interfaces.
|
||||
|
||||
The connector accepts only authenticated `CONNECT <allowed-host>:443`.
|
||||
Ordinary forward-proxy HTTP methods, other ports, IP literals and arbitrary
|
||||
hosts are rejected. Its container is read-only, drops capabilities and is
|
||||
bounded to one CPU, 256 MB memory, 64 processes and 4096 file descriptors.
|
||||
These are circuit breakers for a shared production workstation, not bandwidth
|
||||
throttles.
|
||||
|
||||
The installer does not change the Windows default route, DNS, system proxy,
|
||||
VPN configuration or unrelated firewall rules. The connector's DNS lookup and
|
||||
TCP connection occur inside Docker Desktop on the AMD host and therefore use
|
||||
the egress path exposed to Docker by Windows/VPN.
|
||||
|
||||
### Exact provider allowlist
|
||||
|
||||
Both NAS proxy and Windows connector currently allow only:
|
||||
|
||||
- `api.cesium.com`
|
||||
- `assets.ion.cesium.com`
|
||||
- `dev.virtualearth.net`
|
||||
- `ecn.t0.tiles.virtualearth.net`
|
||||
- `ecn.t1.tiles.virtualearth.net`
|
||||
- `ecn.t2.tiles.virtualearth.net`
|
||||
- `ecn.t3.tiles.virtualearth.net`
|
||||
|
||||
Adding a provider hostname requires one reviewed change to both allowlists,
|
||||
updated smoke coverage, a new Windows connector package, a new NAS app-overlay
|
||||
artifact and the normal plan/apply acceptance. Never add a wildcard such as
|
||||
`*.cesium.com` or `*.virtualearth.net`.
|
||||
|
||||
## 3. Request lifecycle
|
||||
|
||||
For a cache miss, the sequence is:
|
||||
|
||||
1. Map Gateway checks the NAS cache before acquiring provider credentials or
|
||||
contacting egress.
|
||||
2. Map Gateway calls
|
||||
`http://<NAS_LAN_IP>:<NAS_PROXY_PORT>/proxy/cesium/fetch?url=<encoded-url>`
|
||||
and authenticates with the private Map egress credential.
|
||||
3. `dc-amd-proxy` validates method, URL scheme, host, port and credentials.
|
||||
4. A per-origin HTTP/1.1 keep-alive agent either reuses an existing TLS tunnel,
|
||||
opens a new one, or queues behind the bounded active-socket limit.
|
||||
5. For a new connection, the NAS opens an authenticated CONNECT request to
|
||||
`<AMD_LAN_IP>:<CONNECTOR_PORT>`.
|
||||
6. The connector validates its credential, exact hostname and port 443, then
|
||||
resolves/connects through the AMD network/VPN.
|
||||
7. The NAS completes TLS over the byte tunnel and validates the provider
|
||||
certificate.
|
||||
8. Response headers and body stream back to Map Gateway. Client cancellation
|
||||
propagates through queue, CONNECT, TLS, header wait, redirect drain, retry
|
||||
and response streaming.
|
||||
9. Map Gateway can stream the first bytes to the caller while its independent
|
||||
branch completes an atomic cache fill. A failed/partial fill must not become
|
||||
a valid cache object.
|
||||
|
||||
Provider headers forwarded upstream are deliberately narrow: `Accept`,
|
||||
`Range`, `If-None-Match`, `If-Modified-Since`, scoped provider authorization
|
||||
and the Map referer. `Accept-Encoding` is set to `identity`. Hop-by-hop headers
|
||||
are not forwarded back downstream.
|
||||
|
||||
### Credential scope across redirects
|
||||
|
||||
- same-origin redirects retain the provider `Authorization` header;
|
||||
- cross-origin redirects remove `Authorization` before the next request;
|
||||
- every redirect target is parsed again through the exact HTTPS/443/host
|
||||
allowlist;
|
||||
- redirect bodies are drained with abort and idle-timeout protection;
|
||||
- the chain is bounded and terminates with
|
||||
`amd_upstream_redirect_limit` rather than looping indefinitely.
|
||||
|
||||
This is why the Ion API bearer cannot leak when the API redirects to the
|
||||
assets or Bing origin.
|
||||
|
||||
### Pool, retry and timeout policy
|
||||
|
||||
| Control | Current default | Meaning |
|
||||
| --- | ---: | --- |
|
||||
| Active sockets | 8 per origin | Maximum concurrent provider tunnels for each approved origin |
|
||||
| Idle sockets | 4 per origin | Reusable warm TLS tunnels retained for each origin |
|
||||
| TCP keep-alive initial delay | 30 seconds | Socket keep-alive probe setting; actual idle lifetime is also controlled by provider/server socket closure |
|
||||
| Connector/TLS timeout | 20 seconds | Bounds NAS-to-AMD CONNECT and provider TLS setup |
|
||||
| Upstream header timeout | 30 seconds | Bounds one attempt until provider response headers |
|
||||
| Response/redirect body idle timeout | 30 seconds | Resets on body progress; does not cap a healthy long stream |
|
||||
| Slow request threshold | 2 seconds | Emits safe slow-request telemetry; does not cancel the request |
|
||||
| Windows connector upstream connect timeout | 20 seconds | Bounds AMD-to-provider TCP connect |
|
||||
| Windows connector tunnel idle timeout | 90 seconds | Closes a tunnel with no traffic |
|
||||
|
||||
The NAS duration values accept 1–120 seconds and pool sizes accept 1–32. Do
|
||||
not increase them to hide a route/VPN problem: first interpret queue,
|
||||
connection, TTFB and body timings separately.
|
||||
|
||||
There is at most one transport retry. It is allowed only when all conditions
|
||||
are true:
|
||||
|
||||
- method is idempotent `GET` or `HEAD`;
|
||||
- the failed attempt used a previously reused keep-alive socket;
|
||||
- failure is `ECONNRESET`, `EPIPE`, `ETIMEDOUT` or `ECONNABORTED`;
|
||||
- the request has not already retried and the client has not aborted.
|
||||
|
||||
A failure on a newly opened socket is not blindly replayed. A retry may open a
|
||||
replacement socket after the stale reused socket is discarded. Both attempts
|
||||
are included in telemetry and terminal accounting.
|
||||
|
||||
### Runtime configuration reference
|
||||
|
||||
All changes to these values require a reviewed Compose/source artifact; do not
|
||||
inject an ad-hoc live container environment.
|
||||
|
||||
| NAS proxy variable | Default | Contract |
|
||||
| --- | --- | --- |
|
||||
| `PORT` | `8790` | Internal/listener port |
|
||||
| `DC_AMD_PROXY_BIND_ADDRESS` | deployment NAS LAN IP | Exact listener address; IPv4 only |
|
||||
| `DC_AMD_CONNECTOR_HOST` | deployment AMD LAN IP | Stable connector LAN address; IPv4 only |
|
||||
| `DC_AMD_CONNECTOR_PORT` | `8791` | Windows published connector port |
|
||||
| `DC_AMD_PAIR_ALLOWED_SOURCE` | deployment AMD LAN IP | Only source allowed to perform first pair |
|
||||
| `DC_AMD_CONNECTOR_TOKEN_FILE` | `/var/lib/dc-amd-proxy/connector-access` | Persistent paired connector state |
|
||||
| `DC_AMD_MAP_EGRESS_TOKEN_FILE` | `/run/nodedc-secrets/map-egress-proxy-token` | Read-only Map Gateway authentication file |
|
||||
| `DC_AMD_CONNECT_TIMEOUT_SECONDS` | `20` | CONNECT and TLS setup timeout |
|
||||
| `DC_AMD_UPSTREAM_TIMEOUT_SECONDS` | `30` | Per-attempt response-header timeout |
|
||||
| `DC_AMD_BODY_IDLE_TIMEOUT_SECONDS` | `30` | Progress-based response/redirect body timeout |
|
||||
| `DC_AMD_POOL_MAX_SOCKETS` | `8` | Active sockets per origin |
|
||||
| `DC_AMD_POOL_MAX_FREE_SOCKETS` | `4` | Idle sockets per origin |
|
||||
| `DC_AMD_SLOW_REQUEST_SECONDS` | `2` | Safe slow-event threshold |
|
||||
|
||||
The Windows Compose `.env` contains only `AMD_CONNECTOR_BIND_IP` and
|
||||
`AMD_CONNECTOR_PORT`. The connector server also supports bounded
|
||||
`AMD_CONNECTOR_CONNECT_TIMEOUT_SECONDS` (default 20) and
|
||||
`AMD_CONNECTOR_IDLE_TIMEOUT_SECONDS` (default 90); changing them should be a
|
||||
versioned connector-package change.
|
||||
|
||||
## 4. Reproducible initial installation
|
||||
|
||||
Replace every angle-bracket placeholder before executing a command. Do not
|
||||
paste a credential into any placeholder.
|
||||
|
||||
### 4.1 Preconditions
|
||||
|
||||
- AMD workstation has a stable LAN address reserved in DHCP or statically
|
||||
configured.
|
||||
- NAS and AMD can reach each other on the trusted LAN.
|
||||
- Docker Desktop is installed and its Linux engine is running.
|
||||
- The intended Windows user can sign in after a reboot.
|
||||
- The VPN is connected and exposes its route to Docker Desktop workloads.
|
||||
- The reviewed Windows connector package contains a valid `SHA256SUMS`.
|
||||
- The canonical NAS runner is installed at
|
||||
`/usr/local/sbin/nodedc-deploy`.
|
||||
- No other host is using the selected connector LAN address/port.
|
||||
|
||||
### 4.2 Install the Windows connector
|
||||
|
||||
Extract the reviewed connector package to a temporary local folder. Open
|
||||
PowerShell **as Administrator**, enter that folder and run:
|
||||
|
||||
```powershell
|
||||
powershell.exe -NoProfile -ExecutionPolicy Bypass -File ".\install.ps1" `
|
||||
-Destination "C:\NODEDC\dc-amd-connector" `
|
||||
-NasAddress "<NAS_LAN_IP>" `
|
||||
-BindAddress "<AMD_LAN_IP>" `
|
||||
-Port <CONNECTOR_PORT> `
|
||||
-EnableAutoStart $true
|
||||
```
|
||||
|
||||
The installer is fail-closed. It verifies `SHA256SUMS`, refuses to overwrite an
|
||||
existing destination or firewall rule, generates a random connector credential
|
||||
without printing it, builds/starts the container, waits for `healthy`, creates
|
||||
the NAS-only firewall rule and writes the current-user Docker Desktop Startup
|
||||
launcher. On failure it removes only the state it created.
|
||||
|
||||
Verify locally without displaying the credential:
|
||||
|
||||
```powershell
|
||||
Set-Location "C:\NODEDC\dc-amd-connector"
|
||||
docker compose ps
|
||||
docker inspect dc-amd-connector --format '{{.State.Status}} health={{if .State.Health}}{{.State.Health.Status}}{{end}} restart={{.HostConfig.RestartPolicy.Name}}'
|
||||
docker compose logs --tail 100
|
||||
```
|
||||
|
||||
Expected state is `running health=healthy` and restart policy
|
||||
`unless-stopped`.
|
||||
|
||||
This safe CONNECT probe reads the credential only inside the container and
|
||||
prints only the HTTP status line:
|
||||
|
||||
```powershell
|
||||
docker exec dc-amd-connector node -e "const fs=require('fs'),net=require('net');const t=fs.readFileSync('/run/dc-amd-secrets/connector-access','utf8').trim();const s=net.connect(8791,'127.0.0.1',()=>s.write('CONNECT api.cesium.com:443 HTTP/1.1\r\nHost: api.cesium.com:443\r\nProxy-Authorization: Bearer '+t+'\r\n\r\n'));s.once('data',d=>{const v=d.toString('ascii');console.log(v.split('\r\n')[0]);s.destroy();process.exit(v.startsWith('HTTP/1.1 200')?0:1)});s.on('error',e=>{console.error(e.message);process.exit(1)})"
|
||||
```
|
||||
|
||||
Expected output is `HTTP/1.1 200 Connection Established`. This proves that the
|
||||
connector can currently reach the API host; it does not validate an Ion token
|
||||
or prove the VPN exit country.
|
||||
|
||||
### 4.3 Build and transfer the NAS artifact
|
||||
|
||||
From a clean `platform` repository checkout on the trusted build Mac:
|
||||
|
||||
```bash
|
||||
node infra/deploy-runner/build-dc-amd-proxy-artifact.mjs \
|
||||
dc-amd-proxy-<CHANGE_SLUG>-<YYYYMMDD>-<NNN>
|
||||
|
||||
cd infra/deploy-artifacts
|
||||
shasum -a 256 nodedc-dc-amd-proxy-<CHANGE_SLUG>-<YYYYMMDD>-<NNN>.tgz \
|
||||
> nodedc-dc-amd-proxy-<CHANGE_SLUG>-<YYYYMMDD>-<NNN>.tgz.sha256
|
||||
```
|
||||
|
||||
Copy exactly the `.tgz` and `.tgz.sha256` through SMB into:
|
||||
|
||||
```text
|
||||
/volume1/docker/nodedc-deploy/inbox
|
||||
```
|
||||
|
||||
The builder includes only `Dockerfile`, `README.md`, `docker-compose.yml`,
|
||||
`package.json` and `server.mjs`. Runtime, `.env`, credentials, logs, Windows
|
||||
packages and this operations document are not deploy payload members.
|
||||
|
||||
### 4.4 Canonical NAS plan/apply
|
||||
|
||||
On Synology, from the canonical inbox:
|
||||
|
||||
```bash
|
||||
cd /volume1/docker/nodedc-deploy/inbox
|
||||
|
||||
sha256sum -c \
|
||||
nodedc-dc-amd-proxy-<CHANGE_SLUG>-<YYYYMMDD>-<NNN>.tgz.sha256
|
||||
|
||||
sudo /usr/local/sbin/nodedc-deploy verify-install
|
||||
|
||||
sudo /usr/local/sbin/nodedc-deploy plan \
|
||||
/volume1/docker/nodedc-deploy/inbox/nodedc-dc-amd-proxy-<CHANGE_SLUG>-<YYYYMMDD>-<NNN>.tgz
|
||||
```
|
||||
|
||||
Accept the plan only when all are true:
|
||||
|
||||
- `component=dc-amd-proxy`
|
||||
- `type=app-overlay`
|
||||
- `payload_root=/volume1/docker/dc-amd-proxy`
|
||||
- `services=dc-amd-proxy`
|
||||
- `runtime_secret=runner-synced:.../map-egress-proxy-token`
|
||||
- `runtime_state=runner-prepared:/volume1/docker/dc-amd-proxy/runtime`
|
||||
- `state=new`
|
||||
- the file list is exactly the five allowlisted source files.
|
||||
|
||||
Then apply the exact planned filename; never use “latest” discovery:
|
||||
|
||||
```bash
|
||||
sudo /usr/local/sbin/nodedc-deploy apply \
|
||||
/volume1/docker/nodedc-deploy/inbox/nodedc-dc-amd-proxy-<CHANGE_SLUG>-<YYYYMMDD>-<NNN>.tgz
|
||||
```
|
||||
|
||||
The runner preserves/prepares the private pair-state directory, synchronizes
|
||||
the Map egress credential, creates a source backup, builds with Compose,
|
||||
force-recreates only `dc-amd-proxy`, waits for container health, moves the
|
||||
artifact to `applied`, and records the result. A new service should report
|
||||
`state=awaiting_pair`.
|
||||
|
||||
### 4.5 Pair Windows to NAS
|
||||
|
||||
Pair only after NAS status says `awaiting_pair`. Place the reviewed pairing
|
||||
tool on the AMD machine and run it from an elevated PowerShell window:
|
||||
|
||||
```powershell
|
||||
powershell.exe -NoProfile -ExecutionPolicy Bypass `
|
||||
-File ".\pair-dc-amd-proxy.ps1" `
|
||||
-NasAddress "<NAS_LAN_IP>" `
|
||||
-Port <NAS_PROXY_PORT>
|
||||
```
|
||||
|
||||
Expected output:
|
||||
|
||||
```text
|
||||
AMD connector paired with NAS. No secret value was displayed.
|
||||
```
|
||||
|
||||
The pair call is a one-time HTTP LAN handoff with the Windows proxy explicitly
|
||||
disabled. Its security boundary is the trusted LAN plus exact AMD source IP.
|
||||
After the first successful pair, later attempts return `409 already_paired`.
|
||||
|
||||
Verify on NAS:
|
||||
|
||||
```bash
|
||||
curl -fsS http://<NAS_LAN_IP>:<NAS_PROXY_PORT>/status
|
||||
|
||||
sudo /usr/local/bin/docker inspect dc-amd-proxy \
|
||||
--format 'status={{.State.Status}} health={{if .State.Health}}{{.State.Health.Status}}{{end}} network={{.HostConfig.NetworkMode}} read_only={{.HostConfig.ReadonlyRootfs}} restart={{.HostConfig.RestartPolicy.Name}}'
|
||||
```
|
||||
|
||||
Expected properties are `state=paired`, `forwarding=amd_connector_only`,
|
||||
`directEgress=false`, container `healthy`, network `host`, read-only `true`,
|
||||
and restart `unless-stopped`.
|
||||
|
||||
Finally verify one real Map Gateway request using the product health workflow.
|
||||
Do not construct a shell command that reads or prints either private runtime
|
||||
credential.
|
||||
|
||||
For a first installation, switch Map Gateway to
|
||||
`http://<NAS_LAN_IP>:<NAS_PROXY_PORT>` only through a separate reviewed
|
||||
Platform artifact and only after the proxy is paired and its tunnel test
|
||||
passes. This separation keeps a failed connector bootstrap from breaking an
|
||||
already-working provider route.
|
||||
|
||||
## 5. Restart and reboot behavior
|
||||
|
||||
### Synology
|
||||
|
||||
The NAS container has `restart: unless-stopped`. A Docker daemon or NAS reboot
|
||||
restarts it. Pair state survives because `./runtime` is a bind mount outside
|
||||
the container. The Map egress credential is a separate root-owned bind mount.
|
||||
|
||||
If an operator intentionally runs `docker compose down`, the container is
|
||||
removed and restart policy cannot recreate it. Restore it from the canonical
|
||||
source directory:
|
||||
|
||||
```bash
|
||||
cd /volume1/docker/dc-amd-proxy
|
||||
sudo /usr/local/bin/docker compose -p dc-amd-proxy up -d --build --no-deps dc-amd-proxy
|
||||
```
|
||||
|
||||
Use that command only for recovery of the already-deployed version. Source
|
||||
changes still require a runner artifact.
|
||||
|
||||
### Windows AMD host
|
||||
|
||||
Docker Desktop on the WSL 2 backend is not guaranteed to be an unattended
|
||||
Windows boot service. The installed launcher starts it when the configured
|
||||
Windows user signs in. After Docker Engine starts, `restart: unless-stopped`
|
||||
restores the existing connector container.
|
||||
|
||||
Therefore the actual recovery sequence is:
|
||||
|
||||
1. Windows boots;
|
||||
2. the configured user signs in;
|
||||
3. Startup launches Docker Desktop;
|
||||
4. Docker Engine becomes ready;
|
||||
5. the existing connector container restarts;
|
||||
6. the VPN must be connected and usable from Docker;
|
||||
7. cold Cesium requests recover.
|
||||
|
||||
Before user sign-in, or while Docker/VPN is unavailable, live misses fail and
|
||||
Map Gateway must rely on existing NAS cache. If the connector container was
|
||||
removed with `docker compose down`, restart policy alone cannot recreate it;
|
||||
run from elevated PowerShell:
|
||||
|
||||
```powershell
|
||||
Set-Location "C:\NODEDC\dc-amd-connector"
|
||||
docker compose up -d --build
|
||||
```
|
||||
|
||||
## 6. Routine operations and observability
|
||||
|
||||
### NAS commands
|
||||
|
||||
```bash
|
||||
curl -fsS http://<NAS_LAN_IP>:<NAS_PROXY_PORT>/status
|
||||
|
||||
sudo /usr/local/bin/docker inspect dc-amd-proxy \
|
||||
--format 'status={{.State.Status}} health={{if .State.Health}}{{.State.Health.Status}}{{end}} restart={{.HostConfig.RestartPolicy.Name}}'
|
||||
|
||||
sudo /usr/local/bin/docker logs --tail 200 dc-amd-proxy
|
||||
|
||||
cd /volume1/docker/dc-amd-proxy
|
||||
sudo /usr/local/bin/docker compose -p dc-amd-proxy ps
|
||||
```
|
||||
|
||||
### Windows commands
|
||||
|
||||
```powershell
|
||||
Set-Location "C:\NODEDC\dc-amd-connector"
|
||||
docker compose ps
|
||||
docker compose logs --tail 200
|
||||
docker inspect dc-amd-connector --format '{{.State.Status}} health={{if .State.Health}}{{.State.Health.Status}}{{end}} restart={{.HostConfig.RestartPolicy.Name}}'
|
||||
```
|
||||
|
||||
### What `/status` means
|
||||
|
||||
`/healthz` and `/status` currently return the same safe operational body. They
|
||||
prove process/local state, not end-to-end provider reachability.
|
||||
|
||||
Important fields:
|
||||
|
||||
- `state`: `awaiting_pair` or `paired`;
|
||||
- `forwarding`: disabled until paired, then `amd_connector_only`;
|
||||
- `directEgress`: must always be `false`;
|
||||
- `pool.activeSockets`, `idleSockets`, `queuedRequests`, `maxQueuedRequests`;
|
||||
- `pool.byOrigin`: safe active/idle/queued counts per allowlisted origin;
|
||||
- `metrics.requests`, `terminalRequests`, `inFlightRequests`;
|
||||
- `completedResponses`, `failedRequests`, `clientAborts`, `streamFailures`;
|
||||
- `openedTunnels`, `tunnelFailures`, `reusedSocketRequests`, `retries`;
|
||||
- complete and partial byte counts;
|
||||
- average/max queue, connection, TTFB, attempt, retry and total duration;
|
||||
- per-host counters and the last short transport error.
|
||||
|
||||
At all times:
|
||||
|
||||
```text
|
||||
requests = terminalRequests + inFlightRequests
|
||||
```
|
||||
|
||||
At quiescence, `inFlightRequests` should return to zero. A violation indicates
|
||||
a lifecycle/accounting defect. Counters are process-local and reset after a
|
||||
container restart; pair state does not reset.
|
||||
|
||||
Interpret latency by stage:
|
||||
|
||||
| Symptom | Likely stage |
|
||||
| --- | --- |
|
||||
| High `queuedRequests` / `queueMs` | Per-origin eight-socket pool saturated or requests not terminating |
|
||||
| High `connectionMs` / tunnel failures | NAS-to-AMD LAN, Windows connector, DNS, CONNECT or TLS setup |
|
||||
| Low connection but high `ttfbMs` | VPN/provider latency or provider throttling |
|
||||
| Low TTFB but high total duration | Slow/stalled response body or downstream backpressure |
|
||||
| Rising `partialBytes` / `streamFailures` | Provider/VPN body died after headers |
|
||||
| Rising `clientAborts` | Browser/navigation cancellation or caller timeout; not automatically an upstream failure |
|
||||
| Low `reusedSocketRequests` with high `openedTunnels` | Keep-alive churn, route instability or origin fan-out |
|
||||
| Retry spikes | Reused sockets are being reset; one bounded recovery is working but transport is unstable |
|
||||
|
||||
JSON logs intentionally contain event names, approved hostname, short error
|
||||
code and numeric timing only. They must not include a full URL, query string,
|
||||
Ion bearer, connector credential or Map egress credential. Normal fast tile
|
||||
requests are not logged individually. Useful events include:
|
||||
|
||||
- `cesium_egress_tunnel_opened`
|
||||
- `cesium_egress_reused_socket_retry`
|
||||
- `cesium_egress_retry_completed`
|
||||
- `cesium_egress_retry_failed`
|
||||
- `cesium_egress_slow_or_failed`
|
||||
- `cesium_egress_stream_failed`
|
||||
- `cesium_egress_request_failed`
|
||||
- `amd_connector_paired`
|
||||
|
||||
A warm Map Gateway cache hit should not increase `dc-amd-proxy` request
|
||||
counters. If it does, audit the cache-first ordering before tuning the proxy.
|
||||
|
||||
## 7. Failure matrix
|
||||
|
||||
| Observable result | Probable cause | Safe action |
|
||||
| --- | --- | --- |
|
||||
| NAS status unavailable | Container stopped/unhealthy or wrong NAS bind address | Inspect container/Compose state and logs; do not change NAS routes |
|
||||
| `state=awaiting_pair` | No durable connector pair state | Verify intended AMD host, then run the one-time pairing tool |
|
||||
| Pair HTTP 403 `pair_source_not_allowed` | Request did not originate from configured AMD LAN IP | Verify stable AMD IP, NAS host-mode listener and no proxy/NAT in the pair call |
|
||||
| Pair HTTP 409 `already_paired` | Pair state already exists | Do not overwrite it; use the controlled migration/rotation procedure |
|
||||
| `map_egress_unauthorized` | Map Gateway and NAS proxy do not share the runner-synced Map egress credential | Re-run canonical preflight/deploy investigation; never copy a token into `.env` by hand |
|
||||
| `amd_connector_not_paired` | NAS runtime pair file absent/invalid | Pair from the allowed AMD source; inspect ownership only through canonical runner checks |
|
||||
| Connector refused/timeout | Windows off, no user sign-in, Docker stopped, firewall/IP mismatch or connector unhealthy | Restore Windows/Docker/connector; warm cache should continue |
|
||||
| `amd_connector_rejected` | Connector credential mismatch or CONNECT rejected | Stop; do not retry pairing blindly. Treat as pair-state/migration incident |
|
||||
| `cesium_target_not_allowed` | Provider redirected/requested an unreviewed host or wrong scheme/port | Capture hostname safely, review necessity, update both allowlists through source |
|
||||
| TLS timeout/certificate error | VPN path, DNS, time, CA or provider interception problem | Verify AMD route/VPN and NAS container time/CA; never disable TLS verification in production |
|
||||
| `amd_upstream_timeout` | No provider headers within attempt timeout | Compare connection vs TTFB metrics; verify VPN/provider rather than increasing timeout first |
|
||||
| Body/redirect idle timeout | Stream stopped making progress | Check VPN packet loss/provider; partial object must not enter Map cache |
|
||||
| Provider HTTP 401/403 | Ion token, asset permission, referer restriction or provider account issue | Diagnose Map Gateway/provider configuration; transport is functioning if status is passed through |
|
||||
| Provider HTTP 429 | Provider rate/throughput policy | Reduce cold fan-out and use cache; do not add unbounded sockets/retries |
|
||||
| Rendering survives but cold areas fail | Expected cache-first degradation while AMD/VPN is down | Restore egress; preserve existing cache and avoid destructive cache resets |
|
||||
| Queue grows and never drains | Saturation, hung lifecycle or insufficient abort propagation | Inspect in-flight/accounting invariant, body idle errors and client aborts before changing pool size |
|
||||
| AMD host rebooted and does not recover | No Windows sign-in, Startup launcher missing, Docker not ready, container removed or VPN disconnected | Follow the explicit Windows reboot sequence |
|
||||
|
||||
### Important VPN caveat
|
||||
|
||||
The connector trusts the route Docker Desktop receives from Windows. It does
|
||||
not cryptographically prove that a request used a particular VPN or country.
|
||||
If Windows allows direct Internet egress when the VPN is off, the connector
|
||||
may use that direct AMD-host route. It will still never fall back to direct NAS
|
||||
egress, but that is a different guarantee.
|
||||
|
||||
If VPN-only provider egress is mandatory, enforce it on the AMD host/VPN with
|
||||
an outbound kill switch scoped to the connector workload/approved destinations
|
||||
and test it with VPN on and off. The current installer creates only an inbound
|
||||
NAS-only firewall rule; it does not install this outbound policy.
|
||||
|
||||
## 8. Migration to another AMD machine
|
||||
|
||||
Migration has two independent identities: stable LAN endpoint and connector
|
||||
credential. A clean install on another machine creates a new credential, while
|
||||
the current NAS pair endpoint is intentionally write-once. Changing only the
|
||||
IP in Compose is therefore insufficient.
|
||||
|
||||
### Required zero-guesswork migration sequence
|
||||
|
||||
1. Keep the old connector active while preparing the replacement.
|
||||
2. Assign the replacement a reserved `<NEW_AMD_LAN_IP>` and verify VPN/Docker
|
||||
routing.
|
||||
3. Install the reviewed connector package on the replacement using
|
||||
`<NEW_AMD_LAN_IP>` and the same NAS address/connector port.
|
||||
4. Verify local connector health and the safe CONNECT probe.
|
||||
5. Create a reviewed source change that updates NAS connector host and allowed
|
||||
pair source to `<NEW_AMD_LAN_IP>`.
|
||||
6. Add a runner-supported, audited pair-rotation transition that can retire the
|
||||
old NAS pair state and admit exactly one new pair. The transition must back
|
||||
up state, never print either credential, fail closed and be idempotent.
|
||||
7. Build, checksum, SMB-transfer, `plan`, and `apply` the exact versioned
|
||||
artifact/runner change through the deploy canon.
|
||||
8. Pair from the replacement machine, verify an end-to-end cold request and
|
||||
confirm `directEgress=false` plus clean accounting.
|
||||
9. Disable the old connector, then uninstall it only after acceptance.
|
||||
|
||||
The current implementation does **not yet provide step 6 as a first-class
|
||||
runner operation**. Until it exists, a fresh-secret migration is not fully
|
||||
canonical. Do not work around this by deleting
|
||||
`/volume1/docker/dc-amd-proxy/runtime/connector-access`, copying the credential
|
||||
through chat/SMB, weakening `/api/pair`, or running two active pair sources.
|
||||
|
||||
If an urgent migration is required before controlled rotation exists, open an
|
||||
Ops change and implement/review that transition first. This limitation is
|
||||
preferable to an undocumented secret reset.
|
||||
|
||||
To remove a retired connector from Windows after the new path is accepted:
|
||||
|
||||
```powershell
|
||||
powershell.exe -NoProfile -ExecutionPolicy Bypass `
|
||||
-File "C:\NODEDC\dc-amd-connector\uninstall.ps1" `
|
||||
-Destination "C:\NODEDC\dc-amd-connector"
|
||||
```
|
||||
|
||||
The uninstaller removes only this container, its NAS-only firewall rule,
|
||||
Startup launcher and installation directory. It does not remove Docker Desktop
|
||||
or unrelated containers.
|
||||
|
||||
## 9. Deploy and rollback canon
|
||||
|
||||
### Source/change acceptance before packaging
|
||||
|
||||
Run from the service directory in a clean checkout:
|
||||
|
||||
```bash
|
||||
node --check server.mjs
|
||||
npm run test:connection-pool
|
||||
docker build --no-cache -t nodedc/dc-amd-proxy:<VERIFY_TAG> .
|
||||
git status --short
|
||||
```
|
||||
|
||||
The smoke suite must cover pool reuse/saturation, aborts in each lifecycle
|
||||
stage, exact terminal accounting, one safe retry, body idle timeout, redirect
|
||||
credential boundaries and absence of secret markers in logs/status.
|
||||
|
||||
Commit source and documentation together. Build an artifact from the commit,
|
||||
not from an unknown dirty tree. The normal deployment sequence is:
|
||||
|
||||
```text
|
||||
source review -> tests -> commit -> artifact -> checksum -> SMB inbox
|
||||
-> verify-install -> exact plan -> human plan review -> exact apply
|
||||
-> container health -> proxy status -> end-to-end Map acceptance
|
||||
```
|
||||
|
||||
Never put `.env`, `runtime`, tokens, logs, backup archives, Windows package
|
||||
state or shell hooks inside a `dc-amd-proxy` app-overlay.
|
||||
|
||||
### What the runner records
|
||||
|
||||
Before applying, the runner creates:
|
||||
|
||||
```text
|
||||
/volume1/docker/nodedc-deploy/backups/<BACKUP_ID>/
|
||||
```
|
||||
|
||||
with manifest, file lists and `source-before.tgz`. Successful artifacts move
|
||||
to `/volume1/docker/nodedc-deploy/applied`; failed artifacts move to
|
||||
`/volume1/docker/nodedc-deploy/failed`. Applied and failed JSONL state lives
|
||||
under `/volume1/docker/nodedc-deploy/state`.
|
||||
|
||||
For this component the current runner does **not** perform generic automatic
|
||||
source rollback after a failed post-copy build/health check. The backup is
|
||||
evidence/recovery input, not permission for an improvised live extraction.
|
||||
|
||||
### Canonical forward rollback
|
||||
|
||||
Use a new patch ID and package the known-good service source from a separate
|
||||
clean worktree. Do not reset or overwrite the active developer worktree:
|
||||
|
||||
```bash
|
||||
git worktree add "../NODEDC-rollback" <KNOWN_GOOD_COMMIT>
|
||||
cd "../NODEDC-rollback"
|
||||
|
||||
node infra/deploy-runner/build-dc-amd-proxy-artifact.mjs \
|
||||
dc-amd-proxy-rollback-<YYYYMMDD>-<NNN>
|
||||
|
||||
cd infra/deploy-artifacts
|
||||
shasum -a 256 nodedc-dc-amd-proxy-rollback-<YYYYMMDD>-<NNN>.tgz \
|
||||
> nodedc-dc-amd-proxy-rollback-<YYYYMMDD>-<NNN>.tgz.sha256
|
||||
```
|
||||
|
||||
Transfer the exact pair to the inbox and run the same checksum,
|
||||
`verify-install`, exact `plan`, review and exact `apply` sequence. The runtime
|
||||
pair-state directory remains outside the source overlay and must be preserved.
|
||||
|
||||
If the failed version changed a runtime contract, Compose contract or secret
|
||||
format, stop and design an explicit rollback transition instead of assuming a
|
||||
source-only forward rollback is safe.
|
||||
|
||||
## 10. Capacity and product behavior
|
||||
|
||||
The default eight active sockets are **per approved origin**, not global. A
|
||||
Cesium session can use more than one origin, but every origin remains bounded.
|
||||
Twenty users do not overwrite one another's transport state; they share the
|
||||
pool and may queue. Cache objects are owned by Map Gateway, not by this proxy.
|
||||
|
||||
The proxy has no tile index, no cache eviction and no “do not overwrite cache”
|
||||
switch. Those belong to Map Gateway. It streams bytes and records transport
|
||||
telemetry. Consequently:
|
||||
|
||||
- proxy/VPN speed affects only cold/refresh acquisition;
|
||||
- NAS cache read speed should be independent from AMD/VPN speed;
|
||||
- increasing pool size cannot fix slow warm cache reads;
|
||||
- cache fill concurrency and capacity policy must be tuned in Map Gateway;
|
||||
- a browser reload should reuse NAS cache even if local browser cache is empty.
|
||||
|
||||
## 11. Known limitations and evolution path
|
||||
|
||||
The current path is production-usable but not yet a general infrastructure
|
||||
egress product. Track these changes explicitly rather than accumulating
|
||||
one-off live fixes:
|
||||
|
||||
1. **Controlled pair rotation/migration.** Add runner-owned one-time rotation,
|
||||
revocation and rollback state; optionally support overlapping old/new
|
||||
credentials for a bounded cutover window.
|
||||
2. **Remove the legacy `proxy-contur` secret source.** Generate/manage a
|
||||
dedicated Map egress credential as first-class runner state.
|
||||
3. **VPN-route enforcement.** Add a tested Windows outbound kill switch or a
|
||||
dedicated egress appliance so VPN-off cannot become direct AMD egress.
|
||||
4. **Unattended host recovery.** Replace user-sign-in-dependent Docker Desktop
|
||||
with a managed Windows service, signed installer or dedicated Linux egress
|
||||
node if 24/7 cold-cache availability is required.
|
||||
5. **Pair-channel hardening.** Replace source-IP-only HTTP pairing with a
|
||||
short-lived nonce plus authenticated/encrypted handoff or mutual TLS.
|
||||
6. **Two-level health.** Keep local `/healthz`, add a rate-limited synthetic
|
||||
egress readiness check that proves DNS/CONNECT/TLS without consuming or
|
||||
exposing the master Ion credential.
|
||||
7. **Durable metrics.** Export the safe counters to the platform metrics stack;
|
||||
process-local `/status` currently resets on restart.
|
||||
8. **Central allowlist contract.** Generate identical proxy/connector host
|
||||
policies from one reviewed provider contract and test that they cannot
|
||||
drift.
|
||||
9. **Circuit breaking and backoff.** Add per-origin fail-fast state for a dead
|
||||
AMD/VPN path so many cold misses do not occupy all queues until timeout.
|
||||
10. **High availability.** Define an explicit active/standby connector model,
|
||||
health-based selection and credential isolation before adding a second
|
||||
workstation.
|
||||
11. **Signed Windows distribution/update.** Provide a versioned package builder,
|
||||
signature verification and in-place upgrade/rollback rather than manual
|
||||
archive handling.
|
||||
12. **SLOs and load tests.** Establish cache-hit, cold-TTFB, queue-depth,
|
||||
abort-rate and recovery-time objectives using realistic concurrent Map
|
||||
Page sessions.
|
||||
|
||||
Until those changes land, preserve the narrow boundary: official allowlisted
|
||||
HTTPS only, no direct NAS fallback, no global workstation proxy, no secret in
|
||||
source, and every NAS mutation through a reviewed data-only artifact plus the
|
||||
canonical runner.
|
||||
|
|
@ -1,84 +0,0 @@
|
|||
# DC AMD Proxy
|
||||
|
||||
Полный воспроизводимый install/recovery/migration-контракт находится в
|
||||
[`OPERATIONS.md`](./OPERATIONS.md). Этот README фиксирует краткую runtime-модель;
|
||||
операционные изменения должны одновременно обновлять runbook.
|
||||
|
||||
`dc-amd-proxy` is the dedicated, controllable boundary between NODE.DC Map
|
||||
Gateway and the adjacent AMD VPN machine. It is deliberately independent from
|
||||
the historic `proxy-contur` service.
|
||||
|
||||
After pairing it has no direct Cesium egress: every approved Cesium/Bing
|
||||
request is tunnelled through the authenticated AMD connector
|
||||
(`172.22.0.183:8791`). The NAS always addresses this stable LAN endpoint;
|
||||
the AMD host alone owns its changing VPN exit.
|
||||
|
||||
This is a single restricted transport path, not a per-tile relay chain. The
|
||||
NAS keeps a small, bounded pool of HTTP/1.1 TLS tunnels per approved upstream
|
||||
host, so adjacent tile requests reuse the existing `NAS → AMD → VPN` channel.
|
||||
The pool is limited to eight active and four idle sockets per host by default;
|
||||
it cannot become a general-purpose proxy or consume an unbounded amount of the
|
||||
workstation's network resources.
|
||||
|
||||
`/status` exposes only safe operational counters: active/idle/queued sockets
|
||||
per approved origin, the retained peak queue, logical/in-flight/terminal
|
||||
requests, tunnels opened, reused sockets, bounded retries, complete and partial
|
||||
response bytes, separate queue/CONNECT+TLS/TTFB/retry/total timing, body-stream
|
||||
failures, client aborts, and the last short error code. Every accepted egress
|
||||
request is accounted exactly once even when the browser disconnects while it
|
||||
is queued, connecting, following a redirect, waiting for headers, or retrying.
|
||||
The status never contains an Ion token, connector token, full URL, or query
|
||||
parameters.
|
||||
|
||||
An upstream `Authorization` header is retained only across same-origin
|
||||
redirects. A redirect from the Cesium API origin to an assets or Bing origin
|
||||
is followed only after that header is removed, so an API/master bearer cannot
|
||||
cross provider credential boundaries.
|
||||
|
||||
Only an idempotent `GET`/`HEAD` that loses an already reused keep-alive socket
|
||||
is retried, once. The failed reused-socket attempt and the terminal retry
|
||||
attempt are logged separately with numeric transport timings and no URL or
|
||||
credential fields. A response body that makes no progress for 30 seconds is
|
||||
terminated and counted as a stream failure, so a stalled VPN connection cannot
|
||||
occupy one of the eight per-origin sockets forever. These limits are local to
|
||||
Cesium egress and do not modify routing or connectivity of the AMD workstation.
|
||||
|
||||
## Pairing and operational state
|
||||
|
||||
The active service has an explicitly narrow first-pair contract:
|
||||
|
||||
- `POST /api/pair` accepts exactly one connector access token and only from
|
||||
the configured AMD LAN source address;
|
||||
- the AMD operator invokes a supplied local script; it reads the token inside
|
||||
the AMD installation and sends it without printing it or putting it in chat;
|
||||
- the NAS writes the paired value only to its private runtime directory, never
|
||||
to `.env`, SMB inbox, a deploy artifact, browser code, or logs;
|
||||
- the Map Gateway fetch contract (`/proxy/cesium/fetch`) remains private to
|
||||
the NAS LAN address and still requires the runner-synchronised map egress
|
||||
token;
|
||||
- the service follows only the fixed Cesium Ion/Bing host allowlist and cannot
|
||||
fall back to direct NAS Internet egress.
|
||||
|
||||
## Deployment order
|
||||
|
||||
1. Deploy this service and confirm `/healthz` reports `awaiting_pair`.
|
||||
Map Gateway remains on its existing egress during this step.
|
||||
2. On AMD, run the supplied one-time pairing script. It sends the locally-held
|
||||
connector secret directly to the NAS and reports no secret value.
|
||||
3. Verify the private Map Gateway request through this service.
|
||||
4. Apply the separate, minimal Map Gateway switch patch. It changes only its
|
||||
private egress hostname from the historical service to `dc-amd-proxy`.
|
||||
|
||||
The initial 001 deployment was intentionally inert. It is superseded by this
|
||||
active configuration; it did not alter NAS routing, the NAS VPN/Tailscale
|
||||
configuration, or the Windows workstation's VPN configuration.
|
||||
|
||||
## Why host networking is narrow here
|
||||
|
||||
Synology's Docker published-port proxy replaces the caller address before it
|
||||
reaches a bridged container. That makes a strict one-time AMD source-address
|
||||
check impossible. This service therefore uses the host network namespace but
|
||||
binds its own HTTP listener only to `172.22.0.222:8790`; it is not bound to all
|
||||
NAS interfaces. No host route, VPN, DNS, Tailscale setting or firewall rule is
|
||||
changed. The forwarding code still has one hard-coded outbound path only:
|
||||
the authenticated AMD connector.
|
||||
|
|
@ -1,47 +0,0 @@
|
|||
services:
|
||||
dc-amd-proxy:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: Dockerfile
|
||||
container_name: dc-amd-proxy
|
||||
image: nodedc/dc-amd-proxy:local
|
||||
restart: unless-stopped
|
||||
# Preserve the real AMD source address for the one-time pairing request.
|
||||
# The process itself binds only to the NAS LAN address below; this does not
|
||||
# change NAS routes, VPN, DNS, Tailscale, or any other host service.
|
||||
network_mode: host
|
||||
environment:
|
||||
PORT: "8790"
|
||||
DC_AMD_PROXY_BIND_ADDRESS: ${DC_AMD_PROXY_BIND_ADDRESS:-172.22.0.222}
|
||||
DC_AMD_CONNECTOR_HOST: ${DC_AMD_CONNECTOR_HOST:-172.22.0.183}
|
||||
DC_AMD_CONNECTOR_PORT: ${DC_AMD_CONNECTOR_PORT:-8791}
|
||||
DC_AMD_PAIR_ALLOWED_SOURCE: ${DC_AMD_PAIR_ALLOWED_SOURCE:-172.22.0.183}
|
||||
DC_AMD_CONNECTOR_TOKEN_FILE: /var/lib/dc-amd-proxy/connector-access
|
||||
DC_AMD_MAP_EGRESS_TOKEN_FILE: /run/nodedc-secrets/map-egress-proxy-token
|
||||
DC_AMD_BODY_IDLE_TIMEOUT_SECONDS: ${DC_AMD_BODY_IDLE_TIMEOUT_SECONDS:-30}
|
||||
volumes:
|
||||
- type: bind
|
||||
source: /volume1/docker/nodedc-platform/secrets/map-egress-proxy-token
|
||||
target: /run/nodedc-secrets/map-egress-proxy-token
|
||||
read_only: true
|
||||
- type: bind
|
||||
source: ./runtime
|
||||
target: /var/lib/dc-amd-proxy
|
||||
read_only: true
|
||||
tmpfs:
|
||||
- /tmp:rw,noexec,nosuid,size=8m
|
||||
security_opt:
|
||||
- no-new-privileges:true
|
||||
cap_drop:
|
||||
- ALL
|
||||
healthcheck:
|
||||
test:
|
||||
- CMD-SHELL
|
||||
- >
|
||||
node -e "fetch('http://'+process.env.DC_AMD_PROXY_BIND_ADDRESS+':8790/healthz')
|
||||
.then(r => process.exit(r.ok ? 0 : 1))
|
||||
.catch(() => process.exit(1))"
|
||||
interval: 30s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
start_period: 10s
|
||||
|
|
@ -1,10 +0,0 @@
|
|||
{
|
||||
"name": "dc-amd-proxy",
|
||||
"version": "0.1.0",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"start": "node server.mjs",
|
||||
"test:connection-pool": "node scripts/smoke-connection-pool.mjs"
|
||||
}
|
||||
}
|
||||
|
|
@ -1,499 +0,0 @@
|
|||
import assert from "node:assert/strict";
|
||||
import { execFile as execFileCallback, spawn } from "node:child_process";
|
||||
import { once } from "node:events";
|
||||
import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
|
||||
import { createServer as createHttpServer, request as createHttpRequest } from "node:http";
|
||||
import { createServer as createHttpsServer } from "node:https";
|
||||
import { connect, createServer as createNetServer } from "node:net";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { promisify } from "node:util";
|
||||
|
||||
const execFile = promisify(execFileCallback);
|
||||
const root = await mkdtemp(join(tmpdir(), "nodedc-amd-pool-smoke-"));
|
||||
const upstreamPort = await freePort();
|
||||
const connectorPort = await freePort();
|
||||
const proxyPort = await freePort();
|
||||
const upstreamSockets = new Set();
|
||||
const connectorSockets = new Set();
|
||||
let upstream;
|
||||
let connector;
|
||||
let proxy;
|
||||
let proxyOutput = "";
|
||||
let tunnelCount = 0;
|
||||
let upstreamRequests = 0;
|
||||
let retryAttempts = 0;
|
||||
let nonReusedResetAttempts = 0;
|
||||
let retryLimitAttempts = 0;
|
||||
let retryAbortAttempts = 0;
|
||||
let queuedAbortAttempts = 0;
|
||||
let nextConnectorDelayMs = 15;
|
||||
const redirectCredentials = {};
|
||||
let holdReleased = false;
|
||||
let releaseHold;
|
||||
let markHoldStarted;
|
||||
let markPreHeaderSeen;
|
||||
let markPreHeaderClosed;
|
||||
let markIdleClosed;
|
||||
let markDelayedConnectorSeen;
|
||||
let markDelayedConnectorClosed;
|
||||
let markRedirectSeen;
|
||||
let markRedirectClosed;
|
||||
let markRetryAbortSecondSeen;
|
||||
let markRetryAbortSecondClosed;
|
||||
const holdRelease = new Promise((resolve) => { releaseHold = resolve; });
|
||||
const holdStarted = new Promise((resolve) => { markHoldStarted = resolve; });
|
||||
const preHeaderSeen = new Promise((resolve) => { markPreHeaderSeen = resolve; });
|
||||
const preHeaderClosed = new Promise((resolve) => { markPreHeaderClosed = resolve; });
|
||||
const idleClosed = new Promise((resolve) => { markIdleClosed = resolve; });
|
||||
const delayedConnectorSeen = new Promise((resolve) => { markDelayedConnectorSeen = resolve; });
|
||||
const delayedConnectorClosed = new Promise((resolve) => { markDelayedConnectorClosed = resolve; });
|
||||
const redirectSeen = new Promise((resolve) => { markRedirectSeen = resolve; });
|
||||
const redirectClosed = new Promise((resolve) => { markRedirectClosed = resolve; });
|
||||
const retryAbortSecondSeen = new Promise((resolve) => { markRetryAbortSecondSeen = resolve; });
|
||||
const retryAbortSecondClosed = new Promise((resolve) => { markRetryAbortSecondClosed = resolve; });
|
||||
|
||||
try {
|
||||
const keyPath = join(root, "key.pem");
|
||||
const certificatePath = join(root, "certificate.pem");
|
||||
await execFile("openssl", [
|
||||
"req", "-x509", "-newkey", "rsa:2048", "-nodes",
|
||||
"-keyout", keyPath,
|
||||
"-out", certificatePath,
|
||||
"-subj", "/CN=api.cesium.com",
|
||||
"-days", "1",
|
||||
]);
|
||||
|
||||
upstream = createHttpsServer({ key: await readFile(keyPath), cert: await readFile(certificatePath) }, (request, response) => {
|
||||
upstreamRequests += 1;
|
||||
if (request.url === "/retry") {
|
||||
retryAttempts += 1;
|
||||
if (retryAttempts === 1) {
|
||||
setTimeout(() => request.socket.destroy(), 25);
|
||||
return;
|
||||
}
|
||||
response.writeHead(200, { "content-type": "text/plain" });
|
||||
response.end("retry-ok");
|
||||
return;
|
||||
}
|
||||
if (request.url?.startsWith("/hold/")) {
|
||||
markHoldStarted();
|
||||
const finish = () => {
|
||||
response.writeHead(200, { "content-type": "text/plain" });
|
||||
response.end(`hold-${request.url.slice("/hold/".length)}`);
|
||||
};
|
||||
if (holdReleased) finish();
|
||||
else void holdRelease.then(finish);
|
||||
return;
|
||||
}
|
||||
if (request.url === "/preheader") {
|
||||
markPreHeaderSeen();
|
||||
request.socket.once("close", markPreHeaderClosed);
|
||||
return;
|
||||
}
|
||||
if (request.url === "/redirect-hold") {
|
||||
markRedirectSeen();
|
||||
request.socket.once("close", markRedirectClosed);
|
||||
response.writeHead(302, { location: "https://api.cesium.com/redirect-target", "content-type": "text/plain" });
|
||||
response.write("redirect-partial");
|
||||
return;
|
||||
}
|
||||
if (request.url === "/redirect-same-origin") {
|
||||
redirectCredentials.sameSource = request.headers.authorization;
|
||||
response.writeHead(302, { location: "/same-origin-target" });
|
||||
response.end("same-origin-redirect");
|
||||
return;
|
||||
}
|
||||
if (request.url === "/same-origin-target") {
|
||||
redirectCredentials.sameTarget = request.headers.authorization;
|
||||
response.writeHead(200, { "content-type": "text/plain" });
|
||||
response.end("same-origin-ok");
|
||||
return;
|
||||
}
|
||||
if (request.url === "/redirect-assets-origin") {
|
||||
redirectCredentials.assetsSource = request.headers.authorization;
|
||||
response.writeHead(302, { location: "https://assets.ion.cesium.com/assets-origin-target" });
|
||||
response.end("assets-origin-redirect");
|
||||
return;
|
||||
}
|
||||
if (request.url === "/assets-origin-target") {
|
||||
redirectCredentials.assetsHost = request.headers.host;
|
||||
redirectCredentials.assetsTarget = request.headers.authorization;
|
||||
response.writeHead(200, { "content-type": "text/plain" });
|
||||
response.end("assets-origin-ok");
|
||||
return;
|
||||
}
|
||||
if (request.url === "/redirect-bing-origin") {
|
||||
redirectCredentials.bingSource = request.headers.authorization;
|
||||
response.writeHead(302, { location: "https://dev.virtualearth.net/bing-origin-target" });
|
||||
response.end("bing-origin-redirect");
|
||||
return;
|
||||
}
|
||||
if (request.url === "/bing-origin-target") {
|
||||
redirectCredentials.bingHost = request.headers.host;
|
||||
redirectCredentials.bingTarget = request.headers.authorization;
|
||||
response.writeHead(200, { "content-type": "text/plain" });
|
||||
response.end("bing-origin-ok");
|
||||
return;
|
||||
}
|
||||
if (request.url === "/non-reused-reset") {
|
||||
nonReusedResetAttempts += 1;
|
||||
setTimeout(() => request.socket.destroy(), 25);
|
||||
return;
|
||||
}
|
||||
if (request.url === "/queued-abort") {
|
||||
queuedAbortAttempts += 1;
|
||||
response.writeHead(200, { "content-type": "text/plain" });
|
||||
response.end("queue-abort-reached-upstream");
|
||||
return;
|
||||
}
|
||||
if (request.url === "/retry-limit") {
|
||||
retryLimitAttempts += 1;
|
||||
setTimeout(() => request.socket.destroy(), 25);
|
||||
return;
|
||||
}
|
||||
if (request.url === "/retry-abort") {
|
||||
retryAbortAttempts += 1;
|
||||
if (retryAbortAttempts === 1) {
|
||||
setTimeout(() => request.socket.destroy(), 25);
|
||||
return;
|
||||
}
|
||||
markRetryAbortSecondSeen();
|
||||
request.socket.once("close", markRetryAbortSecondClosed);
|
||||
return;
|
||||
}
|
||||
if (request.url === "/idle") {
|
||||
request.socket.once("close", markIdleClosed);
|
||||
response.writeHead(200, { "content-type": "text/plain" });
|
||||
response.write("partial");
|
||||
return;
|
||||
}
|
||||
const responseNumber = upstreamRequests;
|
||||
const finish = () => {
|
||||
response.writeHead(200, { "content-type": "text/plain" });
|
||||
response.end(`tile-${responseNumber}`);
|
||||
};
|
||||
if (request.url?.startsWith("/tile/0?")) setTimeout(finish, 25);
|
||||
else finish();
|
||||
});
|
||||
upstream.on("connection", (socket) => {
|
||||
upstreamSockets.add(socket);
|
||||
socket.once("close", () => upstreamSockets.delete(socket));
|
||||
});
|
||||
upstream.listen(upstreamPort, "127.0.0.1");
|
||||
await once(upstream, "listening");
|
||||
|
||||
connector = createHttpServer((_request, response) => {
|
||||
response.writeHead(405);
|
||||
response.end();
|
||||
});
|
||||
connector.on("connection", (socket) => {
|
||||
connectorSockets.add(socket);
|
||||
socket.once("close", () => connectorSockets.delete(socket));
|
||||
});
|
||||
connector.on("connect", (_request, clientSocket, head) => {
|
||||
tunnelCount += 1;
|
||||
// The fake connector deliberately ignores the requested hostname and
|
||||
// connects to a local TLS fixture. Production still enforces the strict
|
||||
// Cesium/Bing allowlist before opening this tunnel.
|
||||
const connectorDelayMs = nextConnectorDelayMs;
|
||||
nextConnectorDelayMs = 15;
|
||||
if (connectorDelayMs > 100) {
|
||||
markDelayedConnectorSeen();
|
||||
clientSocket.once("close", markDelayedConnectorClosed);
|
||||
}
|
||||
const target = connect(upstreamPort, "127.0.0.1");
|
||||
target.once("connect", () => {
|
||||
setTimeout(() => {
|
||||
if (clientSocket.destroyed || target.destroyed) return;
|
||||
clientSocket.write("HTTP/1.1 200 Connection Established\r\n\r\n");
|
||||
if (head.length) target.write(head);
|
||||
clientSocket.pipe(target);
|
||||
target.pipe(clientSocket);
|
||||
}, connectorDelayMs);
|
||||
});
|
||||
target.once("error", () => clientSocket.destroy());
|
||||
clientSocket.once("error", () => target.destroy());
|
||||
clientSocket.once("close", () => target.destroy());
|
||||
});
|
||||
connector.listen(connectorPort, "127.0.0.1");
|
||||
await once(connector, "listening");
|
||||
|
||||
const connectorToken = `connector-secret-${"a".repeat(48)}`;
|
||||
const mapToken = "map-pool-smoke-secret";
|
||||
const providerToken = "provider-secret-marker";
|
||||
const connectorTokenFile = join(root, "connector-token");
|
||||
const mapTokenFile = join(root, "map-token");
|
||||
await writeFile(connectorTokenFile, `${connectorToken}\n`, { mode: 0o600 });
|
||||
await writeFile(mapTokenFile, `${mapToken}\n`, { mode: 0o600 });
|
||||
|
||||
proxy = spawn(process.execPath, ["server.mjs"], {
|
||||
cwd: new URL("..", import.meta.url),
|
||||
env: {
|
||||
...process.env,
|
||||
NODE_ENV: "test",
|
||||
PORT: String(proxyPort),
|
||||
DC_AMD_PROXY_BIND_ADDRESS: "127.0.0.1",
|
||||
DC_AMD_CONNECTOR_HOST: "127.0.0.1",
|
||||
DC_AMD_CONNECTOR_PORT: String(connectorPort),
|
||||
DC_AMD_PAIR_ALLOWED_SOURCE: "127.0.0.1",
|
||||
DC_AMD_CONNECTOR_TOKEN_FILE: connectorTokenFile,
|
||||
DC_AMD_MAP_EGRESS_TOKEN_FILE: mapTokenFile,
|
||||
DC_AMD_PROXY_TEST_ALLOW_INSECURE_TLS: "true",
|
||||
DC_AMD_POOL_MAX_SOCKETS: "1",
|
||||
DC_AMD_POOL_MAX_FREE_SOCKETS: "1",
|
||||
DC_AMD_SLOW_REQUEST_SECONDS: "30",
|
||||
DC_AMD_UPSTREAM_TIMEOUT_SECONDS: "2",
|
||||
DC_AMD_BODY_IDLE_TIMEOUT_SECONDS: "1",
|
||||
},
|
||||
stdio: ["ignore", "pipe", "pipe"],
|
||||
});
|
||||
proxy.stdout.on("data", (chunk) => { proxyOutput += String(chunk); });
|
||||
proxy.stderr.on("data", (chunk) => { proxyOutput += String(chunk); });
|
||||
await waitForProxy(proxyPort, proxy, () => proxyOutput);
|
||||
|
||||
for (let index = 0; index < 4; index += 1) {
|
||||
const target = `https://api.cesium.com/tile/${index}${index === 0 ? `?access_token=${providerToken}` : ""}`;
|
||||
const response = await proxyFetch(proxyPort, mapToken, target);
|
||||
assert.equal(response.status, 200);
|
||||
assert.equal(await response.text(), `tile-${index + 1}`);
|
||||
}
|
||||
assert.equal(upstreamRequests, 4);
|
||||
assert.equal(tunnelCount, 1);
|
||||
|
||||
const retryResponse = await proxyFetch(proxyPort, mapToken, "https://api.cesium.com/retry");
|
||||
assert.equal(retryResponse.status, 200);
|
||||
assert.equal(await retryResponse.text(), "retry-ok");
|
||||
assert.equal(retryAttempts, 2);
|
||||
|
||||
const redirectAuthorization = "Bearer master-api-secret-marker";
|
||||
const redirectHeaders = { "x-nodedc-cesium-authorization": redirectAuthorization };
|
||||
const sameOriginResponse = await proxyFetch(proxyPort, mapToken, "https://api.cesium.com/redirect-same-origin", undefined, redirectHeaders);
|
||||
assert.equal(sameOriginResponse.status, 200);
|
||||
assert.equal(await sameOriginResponse.text(), "same-origin-ok");
|
||||
const assetsOriginResponse = await proxyFetch(proxyPort, mapToken, "https://api.cesium.com/redirect-assets-origin", undefined, redirectHeaders);
|
||||
assert.equal(assetsOriginResponse.status, 200);
|
||||
assert.equal(await assetsOriginResponse.text(), "assets-origin-ok");
|
||||
const bingOriginResponse = await proxyFetch(proxyPort, mapToken, "https://api.cesium.com/redirect-bing-origin", undefined, redirectHeaders);
|
||||
assert.equal(bingOriginResponse.status, 200);
|
||||
assert.equal(await bingOriginResponse.text(), "bing-origin-ok");
|
||||
assert.equal(redirectCredentials.sameSource, redirectAuthorization);
|
||||
assert.equal(redirectCredentials.sameTarget, redirectAuthorization);
|
||||
assert.equal(redirectCredentials.assetsSource, redirectAuthorization);
|
||||
assert.equal(redirectCredentials.assetsTarget, undefined);
|
||||
assert.equal(redirectCredentials.assetsHost, "assets.ion.cesium.com");
|
||||
assert.equal(redirectCredentials.bingSource, redirectAuthorization);
|
||||
assert.equal(redirectCredentials.bingTarget, undefined);
|
||||
assert.equal(redirectCredentials.bingHost, "dev.virtualearth.net");
|
||||
|
||||
const held = Array.from({ length: 4 }, (_, index) => (
|
||||
rawProxyRequest(proxyPort, mapToken, `https://api.cesium.com/hold/${index}`).promise
|
||||
));
|
||||
await withTimeout(holdStarted, 1_000, "first saturated request did not reach upstream");
|
||||
const queuedAbortRequest = rawProxyRequest(proxyPort, mapToken, "https://api.cesium.com/queued-abort");
|
||||
const saturatedStatus = await waitFor(async () => {
|
||||
const status = await (await fetch(`http://127.0.0.1:${proxyPort}/status`)).json();
|
||||
return status.transport.pool.queuedRequests >= 4 ? status : null;
|
||||
}, 1_000, "agent queue never reached saturation");
|
||||
assert.equal(saturatedStatus.transport.pool.maxQueuedRequests >= 4, true);
|
||||
queuedAbortRequest.abort();
|
||||
await assert.rejects(withTimeout(queuedAbortRequest.promise, 1_000, "queued abort did not reject client request"));
|
||||
await delay(100);
|
||||
holdReleased = true;
|
||||
releaseHold();
|
||||
const heldResults = await withTimeout(Promise.all(held), 2_000, "saturated requests did not drain");
|
||||
for (const [index, result] of heldResults.entries()) {
|
||||
assert.equal(result.status, 200);
|
||||
assert.equal(result.body, `hold-${index}`);
|
||||
}
|
||||
assert.equal(queuedAbortAttempts, 0);
|
||||
|
||||
const redirectAbortRequest = rawProxyRequest(proxyPort, mapToken, "https://api.cesium.com/redirect-hold");
|
||||
await withTimeout(redirectSeen, 1_000, "redirect request did not reach upstream");
|
||||
redirectAbortRequest.abort();
|
||||
await assert.rejects(withTimeout(redirectAbortRequest.promise, 1_000, "redirect abort did not reject client request"));
|
||||
await withTimeout(redirectClosed, 1_000, "redirect abort did not close redirect body socket");
|
||||
|
||||
const preHeaderRequest = rawProxyRequest(proxyPort, mapToken, "https://api.cesium.com/preheader");
|
||||
await withTimeout(preHeaderSeen, 1_000, "pre-header request did not reach upstream");
|
||||
preHeaderRequest.abort();
|
||||
await assert.rejects(withTimeout(preHeaderRequest.promise, 1_000, "pre-header abort did not reject client request"));
|
||||
await withTimeout(preHeaderClosed, 1_000, "pre-header abort did not close upstream socket");
|
||||
|
||||
// With no pooled socket left, delay the next CONNECT response and abort the
|
||||
// browser-side request while that tunnel is still being established.
|
||||
nextConnectorDelayMs = 500;
|
||||
const connectAbortRequest = rawProxyRequest(proxyPort, mapToken, "https://api.cesium.com/connect-abort");
|
||||
await withTimeout(delayedConnectorSeen, 1_000, "CONNECT-abort request did not reach connector");
|
||||
connectAbortRequest.abort();
|
||||
await assert.rejects(withTimeout(connectAbortRequest.promise, 1_000, "CONNECT abort did not reject client request"));
|
||||
await withTimeout(delayedConnectorClosed, 1_000, "CONNECT abort did not close connector socket");
|
||||
await waitFor(async () => {
|
||||
const status = await (await fetch(`http://127.0.0.1:${proxyPort}/status`)).json();
|
||||
return status.transport.metrics.inFlightRequests === 0 ? status : null;
|
||||
}, 1_000, "CONNECT abort was not terminally accounted");
|
||||
|
||||
// The abort above removed the pooled socket. A reset on the next freshly
|
||||
// opened socket must fail directly, never trigger the safe reused-socket retry.
|
||||
const nonReusedResponse = await proxyFetch(proxyPort, mapToken, "https://api.cesium.com/non-reused-reset");
|
||||
assert.equal(nonReusedResponse.status, 502);
|
||||
assert.equal(nonReusedResetAttempts, 1);
|
||||
|
||||
const warmResponse = await proxyFetch(proxyPort, mapToken, "https://api.cesium.com/warm");
|
||||
assert.equal(warmResponse.status, 200);
|
||||
await warmResponse.text();
|
||||
|
||||
const retryAbortRequest = rawProxyRequest(proxyPort, mapToken, "https://api.cesium.com/retry-abort");
|
||||
await withTimeout(retryAbortSecondSeen, 1_000, "retry abort did not reach its second attempt");
|
||||
retryAbortRequest.abort();
|
||||
await assert.rejects(withTimeout(retryAbortRequest.promise, 1_000, "retry abort did not reject client request"));
|
||||
await withTimeout(retryAbortSecondClosed, 1_000, "retry abort did not close second-attempt socket");
|
||||
assert.equal(retryAbortAttempts, 2);
|
||||
|
||||
const retryLimitWarmResponse = await proxyFetch(proxyPort, mapToken, "https://api.cesium.com/warm-after-retry-abort");
|
||||
assert.equal(retryLimitWarmResponse.status, 200);
|
||||
await retryLimitWarmResponse.text();
|
||||
|
||||
// First failure is on a reused socket and is retried once. The second
|
||||
// failure is terminal: retry remains strictly bounded to two attempts.
|
||||
const retryLimitResponse = await proxyFetch(proxyPort, mapToken, "https://api.cesium.com/retry-limit");
|
||||
assert.equal(retryLimitResponse.status, 502);
|
||||
assert.equal(retryLimitAttempts, 2);
|
||||
|
||||
const idleResponse = await proxyFetch(proxyPort, mapToken, "https://api.cesium.com/idle");
|
||||
assert.equal(idleResponse.status, 200);
|
||||
await assert.rejects(withTimeout(idleResponse.text(), 2_000, "idle body timeout did not terminate downstream"));
|
||||
await withTimeout(idleClosed, 1_000, "idle timeout did not close upstream socket");
|
||||
|
||||
const status = await (await fetch(`http://127.0.0.1:${proxyPort}/status`)).json();
|
||||
const metrics = status.transport.metrics;
|
||||
assert.equal(metrics.requests, 22);
|
||||
assert.equal(metrics.terminalRequests, 22);
|
||||
assert.equal(metrics.inFlightRequests, 0);
|
||||
assert.equal(metrics.completedResponses, 14);
|
||||
assert.equal(metrics.failedRequests, 3);
|
||||
assert.equal(metrics.retries, 3);
|
||||
assert.equal(metrics.streamFailures, 1);
|
||||
assert.equal(metrics.clientAborts, 5);
|
||||
assert.equal(metrics.responseBytes > 0, true);
|
||||
assert.equal(metrics.partialBytes >= Buffer.byteLength("partial"), true);
|
||||
assert.equal(metrics.reusedSocketRequests >= 8, true);
|
||||
assert.equal(metrics.timing.queueMs.max >= 100, true);
|
||||
assert.equal(metrics.timing.connectionMs.max > 0, true);
|
||||
assert.equal(metrics.timing.ttfbMs.max >= 20, true);
|
||||
assert.equal(metrics.timing.retryMs.max >= 25, true);
|
||||
assert.equal(status.transport.pool.queuedRequests, 0);
|
||||
assert.equal(status.transport.pool.maxQueuedRequests >= 4, true);
|
||||
assert.equal(status.transport.pool.maxSockets, 1);
|
||||
assert.equal(status.transport.metrics.byHost["api.cesium.com"].terminalRequests, 22);
|
||||
assert.equal(proxyOutput.includes('"event":"cesium_egress_reused_socket_retry"'), true);
|
||||
assert.equal(proxyOutput.includes('"event":"cesium_egress_retry_completed"'), true);
|
||||
assert.equal(proxyOutput.includes('"event":"cesium_egress_retry_failed"'), true);
|
||||
for (const secret of [connectorToken, mapToken, providerToken, redirectAuthorization]) assert.equal(proxyOutput.includes(secret), false);
|
||||
console.log("ok: aborts propagate, queue peaks are retained, retries are bounded, timings are separated, and terminal accounting is exact");
|
||||
} catch (error) {
|
||||
if (proxyOutput) console.error(proxyOutput);
|
||||
throw error;
|
||||
} finally {
|
||||
if (proxy && proxy.exitCode === null && proxy.signalCode === null) {
|
||||
proxy.kill("SIGTERM");
|
||||
try {
|
||||
await withTimeout(once(proxy, "exit"), 2_000, "proxy did not stop after SIGTERM");
|
||||
} catch {
|
||||
proxy.kill("SIGKILL");
|
||||
await withTimeout(once(proxy, "exit"), 2_000, "proxy did not stop after SIGKILL").catch(() => undefined);
|
||||
}
|
||||
}
|
||||
await closeServer(connector, connectorSockets);
|
||||
await closeServer(upstream, upstreamSockets);
|
||||
await rm(root, { recursive: true, force: true });
|
||||
}
|
||||
|
||||
function proxyFetch(port, token, target, signal, extraHeaders = {}) {
|
||||
return fetch(`http://127.0.0.1:${port}/proxy/cesium/fetch?url=${encodeURIComponent(target)}`, {
|
||||
headers: { "x-proxy-token": token, ...extraHeaders },
|
||||
signal,
|
||||
});
|
||||
}
|
||||
|
||||
function rawProxyRequest(port, token, target) {
|
||||
let request;
|
||||
const promise = new Promise((resolve, reject) => {
|
||||
request = createHttpRequest({
|
||||
host: "127.0.0.1",
|
||||
port,
|
||||
method: "GET",
|
||||
path: `/proxy/cesium/fetch?url=${encodeURIComponent(target)}`,
|
||||
headers: { "x-proxy-token": token },
|
||||
agent: false,
|
||||
}, (response) => {
|
||||
const chunks = [];
|
||||
response.on("data", (chunk) => chunks.push(chunk));
|
||||
response.once("end", () => resolve({ status: response.statusCode, body: Buffer.concat(chunks).toString("utf8") }));
|
||||
response.once("error", reject);
|
||||
});
|
||||
request.once("error", reject);
|
||||
request.end();
|
||||
});
|
||||
// Register a rejection observer immediately; the test asserts the same
|
||||
// original promise after it deliberately destroys the request.
|
||||
void promise.catch(() => undefined);
|
||||
return {
|
||||
promise,
|
||||
abort() { request.destroy(new Error("fixture_client_abort")); },
|
||||
};
|
||||
}
|
||||
|
||||
async function freePort() {
|
||||
const server = createNetServer();
|
||||
server.listen(0, "127.0.0.1");
|
||||
await once(server, "listening");
|
||||
const address = server.address();
|
||||
assert(address && typeof address === "object");
|
||||
const closed = once(server, "close");
|
||||
server.close();
|
||||
await closed;
|
||||
return address.port;
|
||||
}
|
||||
|
||||
async function waitForProxy(port, child, output) {
|
||||
for (let attempt = 0; attempt < 80; attempt += 1) {
|
||||
if (child.exitCode !== null) throw new Error(`proxy_exited:${child.exitCode}:${output()}`);
|
||||
try {
|
||||
const response = await fetch(`http://127.0.0.1:${port}/healthz`);
|
||||
if (response.ok) return;
|
||||
} catch { /* service is still starting */ }
|
||||
await delay(50);
|
||||
}
|
||||
throw new Error(`proxy_start_timeout:${output()}`);
|
||||
}
|
||||
|
||||
async function waitFor(read, timeoutMs, message) {
|
||||
const startedAt = Date.now();
|
||||
while (Date.now() - startedAt < timeoutMs) {
|
||||
const value = await read();
|
||||
if (value) return value;
|
||||
await delay(20);
|
||||
}
|
||||
throw new Error(message);
|
||||
}
|
||||
|
||||
function withTimeout(promise, timeoutMs, message) {
|
||||
let timeout;
|
||||
const deadline = new Promise((_, reject) => {
|
||||
timeout = setTimeout(() => reject(new Error(message)), timeoutMs);
|
||||
});
|
||||
return Promise.race([promise, deadline]).finally(() => clearTimeout(timeout));
|
||||
}
|
||||
|
||||
function delay(milliseconds) {
|
||||
return new Promise((resolve) => setTimeout(resolve, milliseconds));
|
||||
}
|
||||
|
||||
async function closeServer(server, sockets) {
|
||||
for (const socket of sockets) socket.destroy();
|
||||
if (!server?.listening) return;
|
||||
const closed = once(server, "close");
|
||||
server.close();
|
||||
await withTimeout(closed, 2_000, "fixture server did not close").catch(() => undefined);
|
||||
}
|
||||
|
|
@ -1,987 +0,0 @@
|
|||
import { timingSafeEqual } from "node:crypto";
|
||||
import { link, mkdir, open, readFile, unlink } from "node:fs/promises";
|
||||
import { createServer } from "node:http";
|
||||
import https from "node:https";
|
||||
import { connect as connectNet } from "node:net";
|
||||
import { dirname } from "node:path";
|
||||
import { connect as connectTls } from "node:tls";
|
||||
|
||||
const allowedHosts = new Set([
|
||||
"api.cesium.com",
|
||||
"assets.ion.cesium.com",
|
||||
"dev.virtualearth.net",
|
||||
"ecn.t0.tiles.virtualearth.net",
|
||||
"ecn.t1.tiles.virtualearth.net",
|
||||
"ecn.t2.tiles.virtualearth.net",
|
||||
"ecn.t3.tiles.virtualearth.net",
|
||||
]);
|
||||
const connectionTimingSymbol = Symbol("nodedc.connection-timing");
|
||||
const config = await readConfig();
|
||||
const transport = createTransport(config);
|
||||
|
||||
const server = createServer(async (request, response) => {
|
||||
const url = new URL(request.url || "/", "http://localhost");
|
||||
try {
|
||||
if (request.method === "GET" && ["/healthz", "/status"].includes(url.pathname)) return writeJson(response, 200, await statusBody());
|
||||
if (request.method === "POST" && url.pathname === "/api/pair") return await pairConnector(request, response);
|
||||
if (["GET", "HEAD"].includes(request.method || "") && url.pathname === "/proxy/cesium/fetch") return await forwardCesiumRequest(request, response, url);
|
||||
return writeJson(response, 404, { ok: false, error: "not_found" });
|
||||
} catch (error) {
|
||||
if (response.destroyed || response.headersSent || response.writableEnded) {
|
||||
if (!response.writableEnded) response.destroy();
|
||||
return;
|
||||
}
|
||||
const errorCode = safeError(error);
|
||||
const status = Number(error?.statusCode || 502);
|
||||
console.warn(JSON.stringify({ event: "cesium_egress_request_failed", error: errorCode, status, ...telemetryLogFields(error?.telemetry) }));
|
||||
return writeJson(response, status, { ok: false, error: errorCode }, { "x-nodedc-egress-error": errorCode });
|
||||
}
|
||||
});
|
||||
|
||||
server.listen(config.port, config.bindAddress, () => console.log(JSON.stringify({ event: "dc_amd_proxy_started", bindAddress: config.bindAddress, port: config.port, connector: `${config.connectorHost}:${config.connectorPort}`, directEgress: false })));
|
||||
for (const signal of ["SIGINT", "SIGTERM"]) process.on(signal, () => server.close(() => transport.close(() => process.exit(0))));
|
||||
|
||||
async function readConfig() {
|
||||
const nodeEnv = String(process.env.NODE_ENV || "production").trim();
|
||||
return {
|
||||
port: parsePort(process.env.PORT, 8790),
|
||||
bindAddress: parseIpv4(process.env.DC_AMD_PROXY_BIND_ADDRESS || "172.22.0.222", "bind_address"),
|
||||
connectorHost: parseIpv4(process.env.DC_AMD_CONNECTOR_HOST || "172.22.0.183", "connector_host"),
|
||||
connectorPort: parsePort(process.env.DC_AMD_CONNECTOR_PORT, 8791),
|
||||
pairAllowedSource: parseIpv4(process.env.DC_AMD_PAIR_ALLOWED_SOURCE || "172.22.0.183", "pair_allowed_source"),
|
||||
connectorTokenFile: String(process.env.DC_AMD_CONNECTOR_TOKEN_FILE || "/var/lib/dc-amd-proxy/connector-access").trim(),
|
||||
mapToken: Buffer.from(await readSecretFile(process.env.DC_AMD_MAP_EGRESS_TOKEN_FILE || "/run/nodedc-secrets/map-egress-proxy-token", "map_egress_token"), "utf8"),
|
||||
connectTimeoutMs: parseDuration(process.env.DC_AMD_CONNECT_TIMEOUT_SECONDS, 20),
|
||||
upstreamTimeoutMs: parseDuration(process.env.DC_AMD_UPSTREAM_TIMEOUT_SECONDS, 30),
|
||||
bodyIdleTimeoutMs: parseDuration(process.env.DC_AMD_BODY_IDLE_TIMEOUT_SECONDS, 30),
|
||||
poolMaxSockets: parsePoolSize(process.env.DC_AMD_POOL_MAX_SOCKETS, 8),
|
||||
poolMaxFreeSockets: parsePoolSize(process.env.DC_AMD_POOL_MAX_FREE_SOCKETS, 4),
|
||||
slowRequestMs: parseDuration(process.env.DC_AMD_SLOW_REQUEST_SECONDS, 2),
|
||||
allowInsecureTls: nodeEnv === "test" && parseBoolean(process.env.DC_AMD_PROXY_TEST_ALLOW_INSECURE_TLS, false),
|
||||
};
|
||||
}
|
||||
|
||||
async function statusBody() {
|
||||
const paired = Boolean(await readConnectorToken());
|
||||
return {
|
||||
ok: true,
|
||||
service: "dc-amd-proxy",
|
||||
state: paired ? "paired" : "awaiting_pair",
|
||||
forwarding: paired ? "amd_connector_only" : "disabled",
|
||||
directEgress: false,
|
||||
connector: `${config.connectorHost}:${config.connectorPort}`,
|
||||
transport: transport.status(),
|
||||
};
|
||||
}
|
||||
|
||||
async function pairConnector(request, response) {
|
||||
if (normalizeAddress(request.socket.remoteAddress) !== config.pairAllowedSource) return writeJson(response, 403, { ok: false, error: "pair_source_not_allowed" });
|
||||
if (await readConnectorToken()) return writeJson(response, 409, { ok: false, error: "already_paired" });
|
||||
const body = await readJsonBody(request, 1024);
|
||||
const token = String(body?.connectorAccessToken || "").trim();
|
||||
if (!isSecret(token)) return writeJson(response, 400, { ok: false, error: "connector_access_token_invalid" });
|
||||
await persistConnectorToken(token);
|
||||
console.log(JSON.stringify({ event: "amd_connector_paired" }));
|
||||
return writeJson(response, 201, { ok: true, state: "paired" });
|
||||
}
|
||||
|
||||
async function forwardCesiumRequest(request, response, requestUrl) {
|
||||
if (!isMapAuthorized(request.headers["x-proxy-token"])) return writeJson(response, 401, { ok: false, error: "map_egress_unauthorized" });
|
||||
const target = parseTarget(requestUrl.searchParams.get("url") || "");
|
||||
const startedAt = Date.now();
|
||||
const terminal = transport.beginRequest(target.hostname);
|
||||
const controller = new AbortController();
|
||||
let telemetry = emptyTelemetry();
|
||||
let upstream;
|
||||
let responseBytes = 0;
|
||||
let upstreamEnded = false;
|
||||
let downstreamFinished = false;
|
||||
let clientAborted = false;
|
||||
let terminalFinished = false;
|
||||
let bodyIdleTimer;
|
||||
let abortUpstream;
|
||||
const clearBodyIdleTimer = () => {
|
||||
if (bodyIdleTimer) clearTimeout(bodyIdleTimer);
|
||||
bodyIdleTimer = undefined;
|
||||
};
|
||||
const cleanup = () => {
|
||||
clearBodyIdleTimer();
|
||||
request.off("aborted", onClientAbort);
|
||||
response.off("close", onClientAbort);
|
||||
if (abortUpstream) controller.signal.removeEventListener("abort", abortUpstream);
|
||||
};
|
||||
const finish = (outcome) => {
|
||||
if (terminalFinished) return false;
|
||||
terminalFinished = terminal.complete({
|
||||
...outcome,
|
||||
durationMs: Date.now() - startedAt,
|
||||
bytes: responseBytes,
|
||||
telemetry,
|
||||
});
|
||||
if (terminalFinished) cleanup();
|
||||
return terminalFinished;
|
||||
};
|
||||
function onClientAbort() {
|
||||
if (downstreamFinished || terminalFinished) return;
|
||||
clientAborted = true;
|
||||
if (!controller.signal.aborted) controller.abort(abortError());
|
||||
// Before headers, requestOnce owns the active/queued ClientRequest and its
|
||||
// signal. The await/catch below records its complete attempt telemetry.
|
||||
if (upstream) finish({ type: "client-abort" });
|
||||
}
|
||||
const armBodyIdleTimer = () => {
|
||||
clearBodyIdleTimer();
|
||||
bodyIdleTimer = setTimeout(() => {
|
||||
const error = typedError("amd_upstream_body_idle_timeout", 502);
|
||||
error.code = "ETIMEDOUT";
|
||||
upstream.destroy(error);
|
||||
}, config.bodyIdleTimeoutMs);
|
||||
bodyIdleTimer.unref?.();
|
||||
};
|
||||
request.once("aborted", onClientAbort);
|
||||
response.once("close", onClientAbort);
|
||||
|
||||
try {
|
||||
const connectorToken = await readConnectorToken();
|
||||
if (clientAborted || controller.signal.aborted) {
|
||||
finish({ type: "client-abort" });
|
||||
return;
|
||||
}
|
||||
if (!connectorToken) {
|
||||
finish({ type: "request-failure", errorCode: "amd_connector_not_paired" });
|
||||
return writeJson(response, 503, { ok: false, error: "amd_connector_not_paired" }, { "x-nodedc-egress-error": "amd_connector_not_paired" });
|
||||
}
|
||||
const result = await requestThroughAmd(target, request.method || "GET", upstreamHeaders(request), controller.signal);
|
||||
upstream = result.upstream;
|
||||
telemetry = result.telemetry;
|
||||
if (clientAborted || controller.signal.aborted) {
|
||||
upstream.destroy();
|
||||
finish({ type: "client-abort" });
|
||||
return;
|
||||
}
|
||||
abortUpstream = () => {
|
||||
if (!upstreamEnded && !upstream.destroyed) upstream.destroy(abortError());
|
||||
};
|
||||
controller.signal.addEventListener("abort", abortUpstream, { once: true });
|
||||
if (controller.signal.aborted) abortUpstream();
|
||||
|
||||
upstream.on("data", (chunk) => {
|
||||
responseBytes += chunk.length;
|
||||
armBodyIdleTimer();
|
||||
});
|
||||
upstream.once("end", () => {
|
||||
upstreamEnded = true;
|
||||
clearBodyIdleTimer();
|
||||
});
|
||||
upstream.once("error", (error) => {
|
||||
upstreamEnded = true;
|
||||
clearBodyIdleTimer();
|
||||
if (!finish({ type: clientAborted ? "client-abort" : "stream-failure", errorCode: safeTransportError(error) })) return;
|
||||
if (!clientAborted) {
|
||||
console.warn(JSON.stringify({ event: "cesium_egress_stream_failed", host: target.hostname, error: safeTransportError(error), bytes: responseBytes }));
|
||||
}
|
||||
if (!response.destroyed) response.destroy();
|
||||
});
|
||||
armBodyIdleTimer();
|
||||
response.writeHead(upstream.statusCode || 502, safeResponseHeaders(upstream.headers));
|
||||
response.once("finish", () => {
|
||||
downstreamFinished = true;
|
||||
clearBodyIdleTimer();
|
||||
const durationMs = Date.now() - startedAt;
|
||||
const status = Number(upstream.statusCode || 502);
|
||||
if (!finish({ type: "response", status })) return;
|
||||
// Normal tile traffic is intentionally not logged per object. Slow or
|
||||
// failing responses are sufficient to diagnose VPN/transport regressions
|
||||
// without creating a noisy, credential-bearing request log.
|
||||
if (durationMs >= config.slowRequestMs || status >= 400) {
|
||||
console.warn(JSON.stringify({
|
||||
event: "cesium_egress_slow_or_failed",
|
||||
host: target.hostname,
|
||||
status,
|
||||
durationMs,
|
||||
queueMs: telemetry.queueMs,
|
||||
connectionMs: telemetry.connectionMs,
|
||||
ttfbMs: telemetry.ttfbMs,
|
||||
retryMs: telemetry.retryMs,
|
||||
attempts: telemetry.attempts,
|
||||
bytes: responseBytes,
|
||||
reusedSocket: telemetry.reusedSocket,
|
||||
retries: telemetry.retries,
|
||||
}));
|
||||
}
|
||||
});
|
||||
upstream.pipe(response);
|
||||
} catch (error) {
|
||||
telemetry = mergeTelemetry(telemetry, error?.telemetry);
|
||||
if (clientAborted || controller.signal.aborted) {
|
||||
finish({ type: "client-abort" });
|
||||
return;
|
||||
}
|
||||
if (upstream && !upstream.destroyed) upstream.destroy();
|
||||
finish({ type: "request-failure", errorCode: safeTransportError(error) });
|
||||
error.telemetry = telemetry;
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
async function requestThroughAmd(initialTarget, method, headers, signal) {
|
||||
let target = initialTarget;
|
||||
let requestHeaders = { ...headers };
|
||||
let aggregate = emptyTelemetry();
|
||||
for (let redirects = 0; redirects <= 5; redirects += 1) {
|
||||
let result;
|
||||
try {
|
||||
result = await requestWithSafeRetry(target, method, requestHeaders, signal);
|
||||
} catch (error) {
|
||||
error.telemetry = mergeTelemetry(aggregate, error?.telemetry);
|
||||
throw error;
|
||||
}
|
||||
const { upstream } = result;
|
||||
aggregate = mergeTelemetry(aggregate, result.telemetry);
|
||||
const statusCode = Number(upstream.statusCode || 502);
|
||||
const location = Array.isArray(upstream.headers.location) ? upstream.headers.location[0] : upstream.headers.location;
|
||||
if ([301, 302, 303, 307, 308].includes(statusCode) && location) {
|
||||
try {
|
||||
aggregate.attemptMs += await drainRedirect(upstream, signal);
|
||||
const redirectedTarget = parseTarget(new URL(location, target).toString());
|
||||
if (redirectedTarget.origin !== target.origin) requestHeaders = withoutAuthorization(requestHeaders);
|
||||
target = redirectedTarget;
|
||||
} catch (error) {
|
||||
aggregate.attemptMs += Math.max(0, Number(error?.redirectDrainMs || 0));
|
||||
error.telemetry = aggregate;
|
||||
throw error;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
return { upstream, telemetry: aggregate };
|
||||
}
|
||||
const error = typedError("amd_upstream_redirect_limit", 502);
|
||||
error.telemetry = aggregate;
|
||||
throw error;
|
||||
}
|
||||
|
||||
function withoutAuthorization(headers) {
|
||||
const output = { ...headers };
|
||||
delete output.authorization;
|
||||
return output;
|
||||
}
|
||||
|
||||
function drainRedirect(upstream, signal) {
|
||||
return new Promise((resolve, reject) => {
|
||||
const startedAt = Date.now();
|
||||
let finished = false;
|
||||
let idleTimer;
|
||||
const cleanup = () => {
|
||||
if (idleTimer) clearTimeout(idleTimer);
|
||||
signal.removeEventListener("abort", onAbort);
|
||||
upstream.off("data", armIdleTimer);
|
||||
upstream.off("end", onEnd);
|
||||
upstream.off("aborted", onUpstreamAbort);
|
||||
upstream.off("error", onError);
|
||||
};
|
||||
const complete = () => {
|
||||
if (finished) return;
|
||||
finished = true;
|
||||
cleanup();
|
||||
resolve(Date.now() - startedAt);
|
||||
};
|
||||
const fail = (error) => {
|
||||
if (finished) return;
|
||||
finished = true;
|
||||
cleanup();
|
||||
error.redirectDrainMs = Date.now() - startedAt;
|
||||
upstream.destroy();
|
||||
reject(error);
|
||||
};
|
||||
const armIdleTimer = () => {
|
||||
if (idleTimer) clearTimeout(idleTimer);
|
||||
idleTimer = setTimeout(() => {
|
||||
const error = typedError("amd_upstream_redirect_body_idle_timeout", 502);
|
||||
error.code = "ETIMEDOUT";
|
||||
fail(error);
|
||||
}, config.bodyIdleTimeoutMs);
|
||||
idleTimer.unref?.();
|
||||
};
|
||||
const onAbort = () => fail(abortReason(signal));
|
||||
const onEnd = () => complete();
|
||||
const onUpstreamAbort = () => {
|
||||
const error = typedError("amd_upstream_redirect_aborted", 502);
|
||||
error.code = "ECONNRESET";
|
||||
fail(error);
|
||||
};
|
||||
const onError = (error) => fail(error);
|
||||
signal.addEventListener("abort", onAbort, { once: true });
|
||||
upstream.on("data", armIdleTimer);
|
||||
upstream.once("end", onEnd);
|
||||
upstream.once("aborted", onUpstreamAbort);
|
||||
upstream.once("error", onError);
|
||||
if (signal.aborted) onAbort();
|
||||
else {
|
||||
armIdleTimer();
|
||||
upstream.resume();
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
async function requestWithSafeRetry(target, method, headers, signal) {
|
||||
let retries = 0;
|
||||
let aggregate = emptyTelemetry();
|
||||
while (true) {
|
||||
try {
|
||||
const result = await requestOnce(target, method, headers, signal);
|
||||
aggregate = mergeTelemetry(aggregate, result.telemetry);
|
||||
aggregate.retries = retries;
|
||||
if (retries > 0) {
|
||||
console.warn(JSON.stringify({
|
||||
event: "cesium_egress_retry_completed",
|
||||
host: target.hostname,
|
||||
attempt: retries + 1,
|
||||
status: Number(result.upstream.statusCode || 502),
|
||||
...telemetryLogFields(result.telemetry),
|
||||
totalAttemptMs: aggregate.attemptMs,
|
||||
}));
|
||||
}
|
||||
return { upstream: result.upstream, telemetry: aggregate };
|
||||
} catch (error) {
|
||||
const failedAttemptTelemetry = error?.telemetry;
|
||||
aggregate = mergeTelemetry(aggregate, failedAttemptTelemetry);
|
||||
if (signal.aborted || retries >= 1 || !["GET", "HEAD"].includes(method) || error?.reusedSocket !== true || !isRetryableSocketError(error)) {
|
||||
if (retries > 0) {
|
||||
console.warn(JSON.stringify({
|
||||
event: "cesium_egress_retry_failed",
|
||||
host: target.hostname,
|
||||
attempt: retries + 1,
|
||||
error: safeTransportError(error),
|
||||
...telemetryLogFields(failedAttemptTelemetry),
|
||||
totalAttemptMs: aggregate.attemptMs,
|
||||
}));
|
||||
}
|
||||
error.telemetry = aggregate;
|
||||
throw error;
|
||||
}
|
||||
retries += 1;
|
||||
aggregate.retryMs += Number(error?.telemetry?.attemptMs || 0);
|
||||
aggregate.retries = retries;
|
||||
const errorCode = safeTransportError(error);
|
||||
transport.recordRetry(target.hostname, errorCode);
|
||||
console.warn(JSON.stringify({
|
||||
event: "cesium_egress_reused_socket_retry",
|
||||
host: target.hostname,
|
||||
attempt: retries,
|
||||
error: errorCode,
|
||||
...telemetryLogFields(failedAttemptTelemetry),
|
||||
}));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function requestOnce(target, method, headers, externalSignal) {
|
||||
return new Promise((resolve, reject) => {
|
||||
const startedAt = Date.now();
|
||||
let socketAssignedAt = null;
|
||||
let socketTiming = null;
|
||||
let reusedSocket = false;
|
||||
let settled = false;
|
||||
let timedOut = false;
|
||||
const pendingConnectionTiming = { startedAt: null, readyAt: null };
|
||||
const attemptController = new AbortController();
|
||||
const abortAttempt = () => {
|
||||
if (!attemptController.signal.aborted) attemptController.abort(abortReason(externalSignal));
|
||||
};
|
||||
if (externalSignal.aborted) abortAttempt();
|
||||
else externalSignal.addEventListener("abort", abortAttempt, { once: true });
|
||||
const client = https.request({
|
||||
protocol: "https:",
|
||||
hostname: target.hostname,
|
||||
port: 443,
|
||||
method,
|
||||
path: `${target.pathname || "/"}${target.search || ""}`,
|
||||
headers,
|
||||
agent: transport.agent,
|
||||
signal: attemptController.signal,
|
||||
nodedcAbortSignal: attemptController.signal,
|
||||
nodedcConnectionTiming: pendingConnectionTiming,
|
||||
});
|
||||
transport.observeQueuePeak();
|
||||
queueMicrotask(() => transport.observeQueuePeak());
|
||||
const timeoutError = typedError("amd_upstream_timeout", 502);
|
||||
timeoutError.code = "ETIMEDOUT";
|
||||
const timeout = setTimeout(() => {
|
||||
timedOut = true;
|
||||
if (!attemptController.signal.aborted) attemptController.abort(timeoutError);
|
||||
}, config.upstreamTimeoutMs);
|
||||
timeout.unref?.();
|
||||
const cleanup = () => {
|
||||
clearTimeout(timeout);
|
||||
externalSignal.removeEventListener("abort", abortAttempt);
|
||||
};
|
||||
const attemptTelemetry = (endedAt, receivedHeaders, error) => {
|
||||
const failedConnectionTiming = error?.[connectionTimingSymbol];
|
||||
const timing = socketTiming || failedConnectionTiming || (pendingConnectionTiming.startedAt ? pendingConnectionTiming : null);
|
||||
const newConnection = !reusedSocket && timing;
|
||||
const queueEnd = newConnection?.startedAt || socketAssignedAt || endedAt;
|
||||
const connectionMs = newConnection ? Math.max(0, Number(newConnection.readyAt || endedAt) - Number(newConnection.startedAt || endedAt)) : 0;
|
||||
return {
|
||||
queueMs: Math.max(0, Number(queueEnd) - startedAt),
|
||||
connectionMs,
|
||||
ttfbMs: receivedHeaders && socketAssignedAt ? Math.max(0, endedAt - socketAssignedAt) : 0,
|
||||
attemptMs: Math.max(0, endedAt - startedAt),
|
||||
retryMs: 0,
|
||||
attempts: 1,
|
||||
retries: 0,
|
||||
reusedSocket,
|
||||
};
|
||||
};
|
||||
client.once("socket", (socket) => {
|
||||
socketAssignedAt = Date.now();
|
||||
socketTiming = socket[connectionTimingSymbol] || null;
|
||||
const priorAssignments = Number(socketTiming?.assignments || 0);
|
||||
reusedSocket = client.reusedSocket === true || priorAssignments > 0;
|
||||
if (socketTiming) socketTiming.assignments = priorAssignments + 1;
|
||||
});
|
||||
client.once("error", (rawError) => {
|
||||
if (settled) return;
|
||||
settled = true;
|
||||
cleanup();
|
||||
const error = timedOut && !externalSignal.aborted ? timeoutError : externalSignal.aborted ? abortReason(externalSignal) : rawError;
|
||||
if (rawError?.[connectionTimingSymbol] && !error[connectionTimingSymbol]) error[connectionTimingSymbol] = rawError[connectionTimingSymbol];
|
||||
error.reusedSocket = reusedSocket || client.reusedSocket === true;
|
||||
error.telemetry = attemptTelemetry(Date.now(), false, error);
|
||||
reject(error);
|
||||
});
|
||||
client.once("response", (upstream) => {
|
||||
if (settled) {
|
||||
upstream.destroy();
|
||||
return;
|
||||
}
|
||||
settled = true;
|
||||
cleanup();
|
||||
const endedAt = Date.now();
|
||||
resolve({
|
||||
upstream,
|
||||
telemetry: attemptTelemetry(endedAt, true),
|
||||
});
|
||||
});
|
||||
client.end();
|
||||
transport.observeQueuePeak();
|
||||
});
|
||||
}
|
||||
|
||||
function emptyTelemetry() {
|
||||
return { queueMs: 0, connectionMs: 0, ttfbMs: 0, attemptMs: 0, retryMs: 0, attempts: 0, retries: 0, reusedSocket: false };
|
||||
}
|
||||
|
||||
function telemetryLogFields(telemetry) {
|
||||
if (!telemetry) return {};
|
||||
return {
|
||||
queueMs: Math.max(0, Number(telemetry.queueMs || 0)),
|
||||
connectionMs: Math.max(0, Number(telemetry.connectionMs || 0)),
|
||||
ttfbMs: Math.max(0, Number(telemetry.ttfbMs || 0)),
|
||||
attemptMs: Math.max(0, Number(telemetry.attemptMs || 0)),
|
||||
retryMs: Math.max(0, Number(telemetry.retryMs || 0)),
|
||||
attempts: Math.max(0, Number(telemetry.attempts || 0)),
|
||||
retries: Math.max(0, Number(telemetry.retries || 0)),
|
||||
reusedSocket: telemetry.reusedSocket === true,
|
||||
};
|
||||
}
|
||||
|
||||
function mergeTelemetry(left, right) {
|
||||
const merged = { ...emptyTelemetry(), ...(left || {}) };
|
||||
if (!right) return merged;
|
||||
for (const name of ["queueMs", "connectionMs", "ttfbMs", "attemptMs", "retryMs", "attempts", "retries"]) {
|
||||
if (name === "retries") merged[name] = Math.max(Number(merged[name] || 0), Number(right[name] || 0));
|
||||
else merged[name] += Math.max(0, Number(right[name] || 0));
|
||||
}
|
||||
merged.reusedSocket ||= right.reusedSocket === true;
|
||||
return merged;
|
||||
}
|
||||
|
||||
function createTransport(runtimeConfig) {
|
||||
const metrics = {
|
||||
openedTunnels: 0,
|
||||
tunnelFailures: 0,
|
||||
requests: 0,
|
||||
terminalRequests: 0,
|
||||
inFlightRequests: 0,
|
||||
completedResponses: 0,
|
||||
failedRequests: 0,
|
||||
slowRequests: 0,
|
||||
responseBytes: 0,
|
||||
partialBytes: 0,
|
||||
reusedSocketRequests: 0,
|
||||
retries: 0,
|
||||
streamFailures: 0,
|
||||
clientAborts: 0,
|
||||
maxQueuedRequests: 0,
|
||||
timing: {
|
||||
queueMs: { total: 0, max: 0 },
|
||||
connectionMs: { total: 0, max: 0 },
|
||||
ttfbMs: { total: 0, max: 0 },
|
||||
attemptMs: { total: 0, max: 0 },
|
||||
retryMs: { total: 0, max: 0 },
|
||||
durationMs: { total: 0, max: 0 },
|
||||
},
|
||||
byHost: {},
|
||||
lastFailure: null,
|
||||
lastFailureAt: null,
|
||||
};
|
||||
const agent = new https.Agent({
|
||||
keepAlive: true,
|
||||
keepAliveMsecs: 30_000,
|
||||
maxSockets: runtimeConfig.poolMaxSockets,
|
||||
maxFreeSockets: runtimeConfig.poolMaxFreeSockets,
|
||||
scheduling: "lifo",
|
||||
});
|
||||
// https.Agent maintains a separate pool per origin. A Cesium/Bing tile
|
||||
// therefore reuses an already authenticated NAS -> AMD -> VPN -> TLS path
|
||||
// for its host instead of paying a fresh CONNECT and TLS handshake.
|
||||
agent.createConnection = (options, callback) => {
|
||||
const hostname = String(options.servername || options.hostname || options.host || "").toLowerCase();
|
||||
const signal = options.nodedcAbortSignal;
|
||||
const pendingConnectionTiming = options.nodedcConnectionTiming;
|
||||
void (async () => {
|
||||
if (!allowedHosts.has(hostname)) throw typedError("cesium_target_not_allowed", 403);
|
||||
const connectorToken = await readConnectorToken();
|
||||
if (!connectorToken) throw typedError("amd_connector_not_paired", 503);
|
||||
const startedAt = Date.now();
|
||||
if (pendingConnectionTiming) pendingConnectionTiming.startedAt = startedAt;
|
||||
let connectedAt = startedAt;
|
||||
try {
|
||||
const tunnel = await openConnectorTunnel({ hostname }, connectorToken, signal);
|
||||
connectedAt = Date.now();
|
||||
const secureSocket = await openTlsTunnel(tunnel, hostname, signal);
|
||||
const readyAt = Date.now();
|
||||
if (pendingConnectionTiming) pendingConnectionTiming.readyAt = readyAt;
|
||||
secureSocket[connectionTimingSymbol] = { startedAt, readyAt, assignments: 0 };
|
||||
metrics.openedTunnels += 1;
|
||||
console.log(JSON.stringify({
|
||||
event: "cesium_egress_tunnel_opened",
|
||||
host: hostname,
|
||||
connectorMs: connectedAt - startedAt,
|
||||
tlsMs: readyAt - connectedAt,
|
||||
}));
|
||||
callback(null, secureSocket);
|
||||
} catch (error) {
|
||||
const readyAt = Date.now();
|
||||
if (pendingConnectionTiming) pendingConnectionTiming.readyAt = readyAt;
|
||||
error[connectionTimingSymbol] = { startedAt, readyAt, assignments: 0 };
|
||||
throw error;
|
||||
}
|
||||
})().catch((error) => {
|
||||
const errorCode = safeError(error);
|
||||
if (error?.code !== "ABORT_ERR") {
|
||||
metrics.tunnelFailures += 1;
|
||||
metrics.lastFailure = errorCode;
|
||||
metrics.lastFailureAt = new Date().toISOString();
|
||||
}
|
||||
callback(error);
|
||||
});
|
||||
return undefined;
|
||||
};
|
||||
|
||||
return {
|
||||
agent,
|
||||
close(callback) { agent.destroy(); callback(); },
|
||||
beginRequest(host) {
|
||||
const hostMetrics = perHostMetrics(metrics, host);
|
||||
metrics.requests += 1;
|
||||
hostMetrics.requests += 1;
|
||||
metrics.inFlightRequests += 1;
|
||||
hostMetrics.inFlightRequests += 1;
|
||||
let finished = false;
|
||||
return {
|
||||
complete({ type, status = 0, durationMs = 0, bytes = 0, telemetry = emptyTelemetry(), errorCode = "" }) {
|
||||
if (finished) return false;
|
||||
finished = true;
|
||||
metrics.terminalRequests += 1;
|
||||
metrics.inFlightRequests = Math.max(0, metrics.inFlightRequests - 1);
|
||||
hostMetrics.terminalRequests += 1;
|
||||
hostMetrics.inFlightRequests = Math.max(0, hostMetrics.inFlightRequests - 1);
|
||||
const safeBytes = Math.max(0, Number(bytes || 0));
|
||||
if (telemetry.reusedSocket) {
|
||||
metrics.reusedSocketRequests += 1;
|
||||
hostMetrics.reusedSocketRequests += 1;
|
||||
}
|
||||
observe(metrics.timing.queueMs, telemetry.queueMs);
|
||||
observe(metrics.timing.connectionMs, telemetry.connectionMs);
|
||||
observe(metrics.timing.ttfbMs, telemetry.ttfbMs);
|
||||
observe(metrics.timing.attemptMs, telemetry.attemptMs);
|
||||
observe(metrics.timing.retryMs, telemetry.retryMs);
|
||||
observe(metrics.timing.durationMs, durationMs);
|
||||
if (type === "response") {
|
||||
metrics.completedResponses += 1;
|
||||
hostMetrics.completedResponses += 1;
|
||||
metrics.responseBytes += safeBytes;
|
||||
hostMetrics.bytes += safeBytes;
|
||||
} else {
|
||||
metrics.partialBytes += safeBytes;
|
||||
hostMetrics.partialBytes += safeBytes;
|
||||
}
|
||||
if (type === "client-abort") {
|
||||
metrics.clientAborts += 1;
|
||||
hostMetrics.clientAborts += 1;
|
||||
}
|
||||
const failure = type === "stream-failure" || type === "request-failure" || type === "response" && Number(status) >= 400;
|
||||
if (failure) {
|
||||
metrics.failedRequests += 1;
|
||||
hostMetrics.failedRequests += 1;
|
||||
if (type === "stream-failure") {
|
||||
metrics.streamFailures += 1;
|
||||
hostMetrics.streamFailures += 1;
|
||||
}
|
||||
metrics.lastFailure = errorCode || `upstream_http_${status}`;
|
||||
metrics.lastFailureAt = new Date().toISOString();
|
||||
hostMetrics.lastTransportError = errorCode || `upstream_http_${status}`;
|
||||
}
|
||||
if (type !== "client-abort" && durationMs >= runtimeConfig.slowRequestMs) metrics.slowRequests += 1;
|
||||
return true;
|
||||
},
|
||||
};
|
||||
},
|
||||
recordRetry(host, errorCode) {
|
||||
metrics.retries += 1;
|
||||
const hostMetrics = perHostMetrics(metrics, host);
|
||||
hostMetrics.retries += 1;
|
||||
hostMetrics.lastTransportError = errorCode;
|
||||
},
|
||||
observeQueuePeak() {
|
||||
const queuedRequests = socketCount(agent.requests);
|
||||
metrics.maxQueuedRequests = Math.max(metrics.maxQueuedRequests, queuedRequests);
|
||||
},
|
||||
status() {
|
||||
const queuedRequests = socketCount(agent.requests);
|
||||
metrics.maxQueuedRequests = Math.max(metrics.maxQueuedRequests, queuedRequests);
|
||||
return {
|
||||
pool: {
|
||||
activeSockets: socketCount(agent.sockets),
|
||||
idleSockets: socketCount(agent.freeSockets),
|
||||
queuedRequests,
|
||||
maxQueuedRequests: metrics.maxQueuedRequests,
|
||||
maxSockets: runtimeConfig.poolMaxSockets,
|
||||
maxFreeSockets: runtimeConfig.poolMaxFreeSockets,
|
||||
byOrigin: poolByOrigin(agent),
|
||||
},
|
||||
metrics: publicTransportMetrics(metrics),
|
||||
};
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
function perHostMetrics(metrics, host) {
|
||||
const safeHost = allowedHosts.has(String(host || "").toLowerCase()) ? String(host).toLowerCase() : "unknown";
|
||||
metrics.byHost[safeHost] ||= {
|
||||
requests: 0,
|
||||
terminalRequests: 0,
|
||||
inFlightRequests: 0,
|
||||
completedResponses: 0,
|
||||
failedRequests: 0,
|
||||
reusedSocketRequests: 0,
|
||||
retries: 0,
|
||||
streamFailures: 0,
|
||||
clientAborts: 0,
|
||||
bytes: 0,
|
||||
partialBytes: 0,
|
||||
lastTransportError: null,
|
||||
};
|
||||
return metrics.byHost[safeHost];
|
||||
}
|
||||
|
||||
function observe(bucket, rawValue) {
|
||||
const value = Math.max(0, Number(rawValue || 0));
|
||||
bucket.total += value;
|
||||
bucket.max = Math.max(bucket.max, value);
|
||||
}
|
||||
|
||||
function publicTransportMetrics(metrics) {
|
||||
const average = (bucket) => metrics.terminalRequests ? Math.round(bucket.total / metrics.terminalRequests) : 0;
|
||||
return {
|
||||
openedTunnels: metrics.openedTunnels,
|
||||
tunnelFailures: metrics.tunnelFailures,
|
||||
requests: metrics.requests,
|
||||
terminalRequests: metrics.terminalRequests,
|
||||
inFlightRequests: metrics.inFlightRequests,
|
||||
completedResponses: metrics.completedResponses,
|
||||
failedRequests: metrics.failedRequests,
|
||||
slowRequests: metrics.slowRequests,
|
||||
responseBytes: metrics.responseBytes,
|
||||
partialBytes: metrics.partialBytes,
|
||||
reusedSocketRequests: metrics.reusedSocketRequests,
|
||||
retries: metrics.retries,
|
||||
streamFailures: metrics.streamFailures,
|
||||
clientAborts: metrics.clientAborts,
|
||||
timing: {
|
||||
queueMs: { average: average(metrics.timing.queueMs), max: metrics.timing.queueMs.max },
|
||||
connectionMs: { average: average(metrics.timing.connectionMs), max: metrics.timing.connectionMs.max },
|
||||
ttfbMs: { average: average(metrics.timing.ttfbMs), max: metrics.timing.ttfbMs.max },
|
||||
attemptMs: { average: average(metrics.timing.attemptMs), max: metrics.timing.attemptMs.max },
|
||||
retryMs: { average: average(metrics.timing.retryMs), max: metrics.timing.retryMs.max },
|
||||
durationMs: { average: average(metrics.timing.durationMs), max: metrics.timing.durationMs.max },
|
||||
},
|
||||
byHost: metrics.byHost,
|
||||
lastFailure: metrics.lastFailure,
|
||||
lastFailureAt: metrics.lastFailureAt,
|
||||
};
|
||||
}
|
||||
|
||||
function poolByOrigin(agent) {
|
||||
const origins = {};
|
||||
for (const [key, sockets] of Object.entries(agent.sockets)) poolOrigin(origins, key).active += sockets.length;
|
||||
for (const [key, sockets] of Object.entries(agent.freeSockets)) poolOrigin(origins, key).idle += sockets.length;
|
||||
for (const [key, requests] of Object.entries(agent.requests)) poolOrigin(origins, key).queued += requests.length;
|
||||
return origins;
|
||||
}
|
||||
|
||||
function poolOrigin(origins, agentKey) {
|
||||
const host = String(agentKey || "").split(":")[0].toLowerCase();
|
||||
const safeHost = allowedHosts.has(host) ? host : "unknown";
|
||||
origins[safeHost] ||= { active: 0, idle: 0, queued: 0 };
|
||||
return origins[safeHost];
|
||||
}
|
||||
|
||||
function openConnectorTunnel(target, connectorToken, signal) {
|
||||
return new Promise((resolve, reject) => {
|
||||
if (signal?.aborted) {
|
||||
reject(abortReason(signal));
|
||||
return;
|
||||
}
|
||||
const socket = connectNet({ host: config.connectorHost, port: config.connectorPort });
|
||||
let buffer = Buffer.alloc(0);
|
||||
let finished = false;
|
||||
const timeoutError = typedError("amd_connector_timeout", 502);
|
||||
timeoutError.code = "ETIMEDOUT";
|
||||
const timeout = setTimeout(() => fail(timeoutError), config.connectTimeoutMs);
|
||||
timeout.unref?.();
|
||||
const cleanup = () => {
|
||||
clearTimeout(timeout);
|
||||
signal?.removeEventListener("abort", onAbort);
|
||||
socket.off("connect", onConnect);
|
||||
socket.off("error", fail);
|
||||
socket.off("data", onData);
|
||||
};
|
||||
const finish = (value) => {
|
||||
if (finished) return;
|
||||
finished = true;
|
||||
cleanup();
|
||||
resolve(value);
|
||||
};
|
||||
const fail = (error) => {
|
||||
if (finished) return;
|
||||
finished = true;
|
||||
cleanup();
|
||||
socket.destroy();
|
||||
reject(error);
|
||||
};
|
||||
const onAbort = () => fail(abortReason(signal));
|
||||
const onConnect = () => socket.write(`CONNECT ${target.hostname}:443 HTTP/1.1\r\nHost: ${target.hostname}:443\r\nProxy-Authorization: Bearer ${connectorToken}\r\n\r\n`);
|
||||
const onData = (chunk) => {
|
||||
buffer = Buffer.concat([buffer, chunk]);
|
||||
if (buffer.length > 16 * 1024) return fail(typedError("amd_connector_response_invalid", 502));
|
||||
const end = buffer.indexOf("\r\n\r\n");
|
||||
if (end < 0) return;
|
||||
if (!/^HTTP\/1\.[01] 200\b/.test(buffer.subarray(0, end).toString("ascii"))) return fail(typedError("amd_connector_rejected", 502));
|
||||
const remainder = buffer.subarray(end + 4);
|
||||
if (remainder.length) socket.unshift(remainder);
|
||||
finish(socket);
|
||||
};
|
||||
signal?.addEventListener("abort", onAbort, { once: true });
|
||||
socket.once("connect", onConnect);
|
||||
socket.on("error", fail);
|
||||
socket.on("data", onData);
|
||||
});
|
||||
}
|
||||
|
||||
function openTlsTunnel(socket, hostname, signal) {
|
||||
return new Promise((resolve, reject) => {
|
||||
if (signal?.aborted) {
|
||||
socket.destroy();
|
||||
reject(abortReason(signal));
|
||||
return;
|
||||
}
|
||||
const secureSocket = connectTls({ socket, servername: hostname, ALPNProtocols: ["http/1.1"], rejectUnauthorized: !config.allowInsecureTls });
|
||||
let finished = false;
|
||||
const timeoutError = typedError("amd_upstream_tls_timeout", 502);
|
||||
timeoutError.code = "ETIMEDOUT";
|
||||
const timeout = setTimeout(() => fail(timeoutError), config.connectTimeoutMs);
|
||||
timeout.unref?.();
|
||||
const cleanup = () => {
|
||||
clearTimeout(timeout);
|
||||
signal?.removeEventListener("abort", onAbort);
|
||||
secureSocket.off("secureConnect", finish);
|
||||
secureSocket.off("error", fail);
|
||||
};
|
||||
const finish = () => {
|
||||
if (finished) return;
|
||||
finished = true;
|
||||
cleanup();
|
||||
resolve(secureSocket);
|
||||
};
|
||||
const fail = (error) => {
|
||||
if (finished) return;
|
||||
finished = true;
|
||||
cleanup();
|
||||
secureSocket.destroy();
|
||||
reject(error);
|
||||
};
|
||||
const onAbort = () => fail(abortReason(signal));
|
||||
signal?.addEventListener("abort", onAbort, { once: true });
|
||||
secureSocket.once("secureConnect", finish);
|
||||
secureSocket.once("error", fail);
|
||||
});
|
||||
}
|
||||
|
||||
function upstreamHeaders(request) {
|
||||
// Do not force `Connection: close`: the shared Agent owns a deliberately
|
||||
// bounded keep-alive pool for the approved upstream hosts.
|
||||
const headers = { accept: safeHeader(request.headers.accept, "*/*"), "accept-encoding": "identity" };
|
||||
for (const name of ["range", "if-none-match", "if-modified-since"]) {
|
||||
const value = safeHeader(request.headers[name], "");
|
||||
if (value) headers[name] = value;
|
||||
}
|
||||
const authorization = safeHeader(request.headers["x-nodedc-cesium-authorization"], "");
|
||||
if (authorization) headers.authorization = authorization;
|
||||
const referer = safeHeader(request.headers["x-nodedc-map-referer"], "");
|
||||
if (referer) headers.referer = referer;
|
||||
return headers;
|
||||
}
|
||||
|
||||
function safeResponseHeaders(headers) {
|
||||
const output = {};
|
||||
const skipped = new Set(["connection", "keep-alive", "proxy-authenticate", "proxy-authorization", "te", "trailer", "transfer-encoding", "upgrade"]);
|
||||
for (const [name, raw] of Object.entries(headers)) {
|
||||
if (skipped.has(name.toLowerCase()) || raw === undefined) continue;
|
||||
const value = Array.isArray(raw) ? raw.join(", ") : String(raw);
|
||||
if (value.length <= 8192 && !/[\r\n\u0000]/.test(value)) output[name] = value;
|
||||
}
|
||||
return output;
|
||||
}
|
||||
|
||||
function parseTarget(value) {
|
||||
let target;
|
||||
try { target = new URL(String(value)); } catch { throw typedError("cesium_target_invalid", 400); }
|
||||
if (target.protocol !== "https:" || target.username || target.password || target.port && target.port !== "443" || !allowedHosts.has(target.hostname.toLowerCase())) throw typedError("cesium_target_not_allowed", 403);
|
||||
target.hostname = target.hostname.toLowerCase();
|
||||
return target;
|
||||
}
|
||||
|
||||
async function readConnectorToken() {
|
||||
try {
|
||||
const token = (await readFile(config.connectorTokenFile, "utf8")).trim();
|
||||
return isSecret(token) ? token : null;
|
||||
} catch (error) {
|
||||
if (error?.code === "ENOENT") return null;
|
||||
throw typedError("connector_pair_state_unreadable", 500);
|
||||
}
|
||||
}
|
||||
|
||||
async function persistConnectorToken(token) {
|
||||
await mkdir(dirname(config.connectorTokenFile), { recursive: true, mode: 0o700 });
|
||||
const temporary = `${config.connectorTokenFile}.${process.pid}.${Date.now()}.tmp`;
|
||||
let handle;
|
||||
try {
|
||||
handle = await open(temporary, "wx", 0o600);
|
||||
await handle.writeFile(`${token}\n`, "ascii");
|
||||
await handle.sync();
|
||||
await handle.close();
|
||||
handle = null;
|
||||
// link(2) is the compare-and-set: unlike rename(), it cannot overwrite an
|
||||
// existing pair state. Two concurrent pair attempts therefore produce one
|
||||
// durable secret and one harmless 409 response.
|
||||
try {
|
||||
await link(temporary, config.connectorTokenFile);
|
||||
} catch (error) {
|
||||
if (error?.code === "EEXIST") throw typedError("already_paired", 409);
|
||||
throw error;
|
||||
}
|
||||
} finally {
|
||||
if (handle) await handle.close();
|
||||
await unlink(temporary).catch(() => {});
|
||||
}
|
||||
}
|
||||
|
||||
async function readSecretFile(path, label) {
|
||||
let value;
|
||||
try { value = (await readFile(String(path), "utf8")).trim(); } catch { throw new Error(`${label}_unreadable`); }
|
||||
if (!value || value.length > 4096 || /[\u0000-\u001f\u007f\s]/.test(value)) throw new Error(`${label}_invalid`);
|
||||
return value;
|
||||
}
|
||||
|
||||
function isMapAuthorized(raw) {
|
||||
if (Array.isArray(raw)) return false;
|
||||
const candidate = Buffer.from(String(raw || ""), "utf8");
|
||||
return candidate.length === config.mapToken.length && timingSafeEqual(candidate, config.mapToken);
|
||||
}
|
||||
|
||||
function readJsonBody(request, maxBytes) {
|
||||
return new Promise((resolve, reject) => {
|
||||
const chunks = [];
|
||||
let bytes = 0;
|
||||
request.on("data", (chunk) => {
|
||||
bytes += chunk.length;
|
||||
if (bytes > maxBytes) return reject(typedError("pair_payload_too_large", 413));
|
||||
chunks.push(chunk);
|
||||
});
|
||||
request.once("error", reject);
|
||||
request.once("end", () => {
|
||||
try { resolve(JSON.parse(Buffer.concat(chunks).toString("utf8"))); } catch { reject(typedError("pair_payload_invalid", 400)); }
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
function writeJson(response, status, body, extraHeaders = {}) {
|
||||
const payload = JSON.stringify(body);
|
||||
response.writeHead(status, {
|
||||
"content-type": "application/json; charset=utf-8",
|
||||
"content-length": Buffer.byteLength(payload),
|
||||
"cache-control": "no-store",
|
||||
...extraHeaders,
|
||||
});
|
||||
response.end(payload);
|
||||
}
|
||||
|
||||
function parsePort(raw, fallback) {
|
||||
const value = Number(String(raw || fallback).trim());
|
||||
if (!Number.isInteger(value) || value < 1024 || value > 65535) throw new Error("invalid_port");
|
||||
return value;
|
||||
}
|
||||
|
||||
function parseDuration(raw, fallbackSeconds) {
|
||||
const value = Number(String(raw || fallbackSeconds).trim());
|
||||
if (!Number.isInteger(value) || value < 1 || value > 120) throw new Error("invalid_duration");
|
||||
return value * 1000;
|
||||
}
|
||||
|
||||
function parsePoolSize(raw, fallback) {
|
||||
const value = Number(String(raw || fallback).trim());
|
||||
if (!Number.isInteger(value) || value < 1 || value > 32) throw new Error("invalid_pool_size");
|
||||
return value;
|
||||
}
|
||||
|
||||
function socketCount(table) {
|
||||
return Object.values(table).reduce((total, sockets) => total + (Array.isArray(sockets) ? sockets.length : 0), 0);
|
||||
}
|
||||
|
||||
function parseIpv4(raw, label) {
|
||||
const value = String(raw || "").trim();
|
||||
const octets = value.split(".").map(Number);
|
||||
if (octets.length !== 4 || octets.some((part) => !Number.isInteger(part) || part < 0 || part > 255)) throw new Error(`${label}_invalid`);
|
||||
return value;
|
||||
}
|
||||
|
||||
function normalizeAddress(value) { return String(value || "").replace(/^::ffff:/, ""); }
|
||||
function safeHeader(raw, fallback) {
|
||||
if (Array.isArray(raw)) return fallback;
|
||||
const value = String(raw || "").trim();
|
||||
return value.length <= 4096 && !/[\r\n\u0000]/.test(value) ? value : fallback;
|
||||
}
|
||||
function parseBoolean(value, fallback) { return value === undefined || value === "" ? fallback : ["1", "true", "yes", "on"].includes(String(value).trim().toLowerCase()); }
|
||||
function isSecret(value) { return /^[A-Za-z0-9_-]{48,256}$/.test(value); }
|
||||
function typedError(message, statusCode) { const error = new Error(message); error.statusCode = statusCode; return error; }
|
||||
function abortError() { const error = typedError("amd_client_aborted", 499); error.code = "ABORT_ERR"; return error; }
|
||||
function abortReason(signal) { return signal?.reason instanceof Error ? signal.reason : abortError(); }
|
||||
function safeError(error) { return String(error?.message || "dc_amd_proxy_error").replace(/[^A-Za-z0-9_.:-]/g, "_").slice(0, 120); }
|
||||
function isRetryableSocketError(error) { return new Set(["ECONNRESET", "EPIPE", "ETIMEDOUT", "ECONNABORTED"]).has(String(error?.code || "").toUpperCase()); }
|
||||
function safeTransportError(error) {
|
||||
const code = String(error?.code || "").trim().toLowerCase();
|
||||
if (/^[a-z0-9_]{1,48}$/.test(code)) return `amd_upstream_${code}`;
|
||||
return safeError(error);
|
||||
}
|
||||
|
|
@ -1,32 +0,0 @@
|
|||
FROM node:20-alpine AS deps
|
||||
|
||||
WORKDIR /workspace
|
||||
|
||||
COPY packages/external-provider-contract ./packages/external-provider-contract
|
||||
COPY services/external-data-plane/package.json services/external-data-plane/package-lock.json ./services/external-data-plane/
|
||||
|
||||
WORKDIR /workspace/services/external-data-plane
|
||||
RUN npm ci --omit=dev
|
||||
|
||||
FROM node:20-alpine AS runner
|
||||
|
||||
ENV NODE_ENV=production
|
||||
ENV PORT=18106
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
COPY --from=deps /workspace/services/external-data-plane/package.json /workspace/services/external-data-plane/package-lock.json ./
|
||||
COPY --from=deps /workspace/services/external-data-plane/node_modules ./node_modules
|
||||
COPY --from=deps /workspace/packages/external-provider-contract /packages/external-provider-contract
|
||||
COPY services/external-data-plane/src ./src
|
||||
COPY services/external-data-plane/definitions ./definitions
|
||||
|
||||
# A dedicated numeric identity can read only the EDP provisioning secret mount;
|
||||
# it is intentionally not the shared Node/Map Gateway uid/gid (1000).
|
||||
RUN addgroup -S -g 11006 nodedc-edp && adduser -S -D -H -u 11006 -G nodedc-edp nodedc-edp
|
||||
|
||||
USER 11006:11006
|
||||
|
||||
EXPOSE 18106
|
||||
|
||||
CMD ["node", "src/server.mjs"]
|
||||
|
|
@ -1,30 +0,0 @@
|
|||
{
|
||||
"id": "fleet.positions.current.v1",
|
||||
"version": "1.0.0",
|
||||
"ontologyRevision": "ontology.map.moving_object.v1",
|
||||
"deliveryMode": "snapshot+patch",
|
||||
"semanticTypes": [
|
||||
"map.moving_object"
|
||||
],
|
||||
"fields": [
|
||||
"course_degrees",
|
||||
"display_name",
|
||||
"elevation_meters",
|
||||
"geometry",
|
||||
"hdop",
|
||||
"horizontal_accuracy_meters",
|
||||
"object_kind",
|
||||
"operational_status",
|
||||
"position_source",
|
||||
"position_valid",
|
||||
"quality_flags",
|
||||
"satellite_count",
|
||||
"speed_kph"
|
||||
],
|
||||
"history": {
|
||||
"mode": "sampled",
|
||||
"intervalMs": 60000,
|
||||
"strategy": "latest-per-entity-per-bucket",
|
||||
"retentionDays": 90
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
|
|
@ -1,18 +0,0 @@
|
|||
{
|
||||
"name": "@nodedc/external-data-plane",
|
||||
"version": "0.1.0",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"start": "node src/server.mjs",
|
||||
"dev": "node --watch src/server.mjs",
|
||||
"check": "node --check src/server.mjs && node --check src/schema.mjs && node --check src/config.mjs && node --check src/intake-policy.mjs && node --check src/writer-binding.mjs && node --check src/reader-binding.mjs && node --check src/data-product-policy.mjs && node --check src/data-product-delivery.mjs && node --check src/definitions.mjs",
|
||||
"test": "node test/config.test.mjs && node test/intake-policy.test.mjs && node test/writer-binding.test.mjs && node test/reader-binding.test.mjs && node test/data-product-policy.test.mjs && node test/definitions.test.mjs",
|
||||
"test:integration": "node test/data-product-delivery.integration.test.mjs && node test/api.integration.test.mjs"
|
||||
},
|
||||
"dependencies": {
|
||||
"@nodedc/external-provider-contract": "file:../../packages/external-provider-contract",
|
||||
"express": "^5.2.1",
|
||||
"pg": "^8.18.0"
|
||||
}
|
||||
}
|
||||
|
|
@ -1,75 +0,0 @@
|
|||
import { readFileSync } from "node:fs";
|
||||
|
||||
export function readConfig(env = process.env) {
|
||||
const provisionerApiEnabled = boolean(env.EXTERNAL_DATA_PLANE_PROVISIONING_ENABLED, false);
|
||||
const config = {
|
||||
port: integer(env.PORT, 18106, 1, 65535),
|
||||
databaseUrl: required(env.EXTERNAL_DATA_PLANE_DATABASE_URL, "EXTERNAL_DATA_PLANE_DATABASE_URL"),
|
||||
databasePoolSize: integer(env.EXTERNAL_DATA_PLANE_DATABASE_POOL_SIZE, 10, 1, 50),
|
||||
internalAccessToken: optional(env.NODEDC_INTERNAL_ACCESS_TOKEN),
|
||||
provisionerApiEnabled,
|
||||
provisionerAccessToken: provisionerApiEnabled ? secretFile(env.EXTERNAL_DATA_PLANE_PROVISIONER_TOKEN_FILE) : "",
|
||||
rawRetentionDays: integer(env.EXTERNAL_DATA_PLANE_RAW_RETENTION_DAYS, 14, 1, 3650),
|
||||
maxBatchBytes: integer(env.EXTERNAL_DATA_PLANE_MAX_BATCH_BYTES, 5 * 1024 * 1024, 1024, 50 * 1024 * 1024),
|
||||
maxFactsPerPublish: integer(env.EXTERNAL_DATA_PLANE_MAX_FACTS_PER_PUBLISH, 5000, 1, 100_000),
|
||||
maxAttributesBytesPerFact: integer(env.EXTERNAL_DATA_PLANE_MAX_ATTRIBUTES_BYTES_PER_FACT, 64 * 1024, 256, 1024 * 1024),
|
||||
maxPatchOperations: integer(env.EXTERNAL_DATA_PLANE_MAX_PATCH_OPERATIONS, 500, 1, 5000),
|
||||
maxPatchBytes: integer(env.EXTERNAL_DATA_PLANE_MAX_PATCH_BYTES, 256 * 1024, 16 * 1024, 5 * 1024 * 1024),
|
||||
patchRetentionMs: integer(env.EXTERNAL_DATA_PLANE_PATCH_RETENTION_MS, 60 * 60 * 1000, 60 * 1000, 7 * 24 * 60 * 60 * 1000),
|
||||
receiptRetentionMs: integer(env.EXTERNAL_DATA_PLANE_RECEIPT_RETENTION_MS, 7 * 24 * 60 * 60 * 1000, 24 * 60 * 60 * 1000, 90 * 24 * 60 * 60 * 1000),
|
||||
retentionDeleteLimit: integer(env.EXTERNAL_DATA_PLANE_RETENTION_DELETE_LIMIT, 10_000, 100, 100_000),
|
||||
streamHeartbeatMs: integer(env.EXTERNAL_DATA_PLANE_STREAM_HEARTBEAT_MS, 20_000, 5_000, 60_000),
|
||||
streamPollMs: integer(env.EXTERNAL_DATA_PLANE_STREAM_POLL_MS, 1_000, 250, 10_000),
|
||||
maxReaderStreams: integer(env.EXTERNAL_DATA_PLANE_MAX_READER_STREAMS, 10, 1, 1000),
|
||||
writerBindingMaxTtlDays: integer(env.EXTERNAL_DATA_PLANE_WRITER_BINDING_MAX_TTL_DAYS, 90, 1, 365),
|
||||
maxFutureSkewSeconds: integer(env.EXTERNAL_DATA_PLANE_MAX_FUTURE_SKEW_SECONDS, 300, 0, 86400),
|
||||
retentionSweepMs: integer(env.EXTERNAL_DATA_PLANE_RETENTION_SWEEP_MS, 60 * 60 * 1000, 60 * 1000, 24 * 60 * 60 * 1000),
|
||||
legacyIntakeEnabled: boolean(env.EXTERNAL_DATA_PLANE_LEGACY_INTAKE_ENABLED, false),
|
||||
};
|
||||
if (config.internalAccessToken && config.provisionerAccessToken && config.internalAccessToken === config.provisionerAccessToken) {
|
||||
throw new Error("provisioner_token_must_differ_from_internal_token");
|
||||
}
|
||||
return config;
|
||||
}
|
||||
|
||||
function required(value, name) {
|
||||
const normalized = optional(value);
|
||||
if (!normalized) throw new Error(`${name}_required`);
|
||||
return normalized;
|
||||
}
|
||||
|
||||
function optional(value) {
|
||||
const normalized = String(value ?? "").trim();
|
||||
return normalized || "";
|
||||
}
|
||||
|
||||
function integer(value, fallback, min, max) {
|
||||
const candidate = optional(value);
|
||||
if (!candidate) return fallback;
|
||||
const number = Number.parseInt(candidate, 10);
|
||||
if (!Number.isInteger(number) || number < min || number > max) throw new Error(`invalid_integer:${candidate}`);
|
||||
return number;
|
||||
}
|
||||
|
||||
function boolean(value, fallback) {
|
||||
const normalized = optional(value).toLowerCase();
|
||||
if (!normalized) return fallback;
|
||||
if (normalized === "true") return true;
|
||||
if (normalized === "false") return false;
|
||||
throw new Error(`invalid_boolean:${normalized}`);
|
||||
}
|
||||
|
||||
function secretFile(value) {
|
||||
const path = optional(value);
|
||||
if (!path) return "";
|
||||
let secret = "";
|
||||
try {
|
||||
secret = readFileSync(path, "utf8").trim();
|
||||
} catch {
|
||||
throw new Error("external_data_plane_provisioner_token_file_unreadable");
|
||||
}
|
||||
if (!/^[A-Za-z0-9_-]{48,256}$/.test(secret)) {
|
||||
throw new Error("external_data_plane_provisioner_token_file_invalid");
|
||||
}
|
||||
return secret;
|
||||
}
|
||||
|
|
@ -1,527 +0,0 @@
|
|||
import { createHash, randomUUID } from "node:crypto";
|
||||
import {
|
||||
DATA_PRODUCT_PATCH_SCHEMA_VERSION,
|
||||
DATA_PRODUCT_SNAPSHOT_SCHEMA_VERSION,
|
||||
} from "@nodedc/external-provider-contract";
|
||||
|
||||
export async function loadDataProductDefinition(db, dataProductId, { activeOnly = true } = {}) {
|
||||
const result = await db.query(
|
||||
`select id, version, ontology_revision as "ontologyRevision",
|
||||
delivery_mode as "deliveryMode", semantic_types as "semanticTypes",
|
||||
fields, history_policy as "historyPolicy", active,
|
||||
created_at as "createdAt", updated_at as "updatedAt"
|
||||
from external_data_plane_products
|
||||
where id = $1 ${activeOnly ? "and active = true" : ""}`,
|
||||
[dataProductId],
|
||||
);
|
||||
return result.rows[0] || null;
|
||||
}
|
||||
|
||||
export async function persistDataProductDefinition(db, definition) {
|
||||
const result = await db.query(
|
||||
`insert into external_data_plane_products (
|
||||
id, version, ontology_revision, delivery_mode, semantic_types, fields, history_policy
|
||||
) values ($1, $2, $3, $4, $5::jsonb, $6::jsonb, $7::jsonb)
|
||||
on conflict (id) do update set
|
||||
active = true,
|
||||
updated_at = now()
|
||||
where external_data_plane_products.version = excluded.version
|
||||
and external_data_plane_products.ontology_revision = excluded.ontology_revision
|
||||
and external_data_plane_products.delivery_mode = excluded.delivery_mode
|
||||
and external_data_plane_products.semantic_types = excluded.semantic_types
|
||||
and external_data_plane_products.fields = excluded.fields
|
||||
and external_data_plane_products.history_policy = excluded.history_policy
|
||||
returning id, version, ontology_revision as "ontologyRevision",
|
||||
delivery_mode as "deliveryMode", semantic_types as "semanticTypes",
|
||||
fields, history_policy as "historyPolicy", active,
|
||||
created_at as "createdAt", updated_at as "updatedAt"`,
|
||||
[
|
||||
definition.id,
|
||||
definition.version,
|
||||
definition.ontologyRevision,
|
||||
definition.deliveryMode,
|
||||
JSON.stringify(definition.semanticTypes),
|
||||
JSON.stringify(definition.fields),
|
||||
JSON.stringify(definition.history),
|
||||
],
|
||||
);
|
||||
if (!result.rowCount) throw deliveryError("data_product_version_is_immutable", 409);
|
||||
return result.rows[0];
|
||||
}
|
||||
|
||||
export async function persistDataProductPublish(pool, batch, definition, {
|
||||
maxPatchOperations = 500,
|
||||
maxPatchBytes = 256 * 1024,
|
||||
} = {}) {
|
||||
assertPublishMatchesDefinition(batch, definition);
|
||||
const requestFingerprint = publishFingerprint(batch);
|
||||
const client = await pool.connect();
|
||||
try {
|
||||
await client.query("begin");
|
||||
const batchId = randomUUID();
|
||||
const insertedBatch = await client.query(
|
||||
`insert into external_data_plane_batches (
|
||||
id, tenant_id, connection_id, provider_id, data_product_id, contract_version,
|
||||
ontology_revision, run_id, sequence, idempotency_key, received_at, fact_count,
|
||||
request_fingerprint
|
||||
) values ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13)
|
||||
on conflict (tenant_id, connection_id, provider_id, data_product_id, idempotency_key)
|
||||
do nothing
|
||||
returning id`,
|
||||
[
|
||||
batchId, batch.source.tenantId, batch.source.connectionId, batch.source.providerId,
|
||||
batch.contract.dataProductId, batch.contract.version, batch.contract.ontologyRevision,
|
||||
batch.batch.runId, batch.batch.sequence, batch.batch.idempotencyKey,
|
||||
batch.batch.receivedAt, batch.facts.length, requestFingerprint,
|
||||
],
|
||||
);
|
||||
if (!insertedBatch.rowCount) {
|
||||
const existing = await client.query(
|
||||
`select id as "batchId", fact_count as "publishedFactCount",
|
||||
current_updated_count as "currentUpdatedCount",
|
||||
history_inserted_count as "historyInsertedCount",
|
||||
patch_operation_count as "patchOperationCount",
|
||||
delivery_cursor::text as cursor, received_at as "acceptedAt",
|
||||
request_fingerprint as "requestFingerprint"
|
||||
from external_data_plane_batches
|
||||
where tenant_id = $1 and connection_id = $2 and provider_id = $3
|
||||
and data_product_id = $4 and idempotency_key = $5`,
|
||||
scopeValues(batch, batch.batch.idempotencyKey),
|
||||
);
|
||||
if (!existing.rowCount || existing.rows[0].requestFingerprint !== requestFingerprint) {
|
||||
throw deliveryError("idempotency_key_reused", 409);
|
||||
}
|
||||
await client.query("commit");
|
||||
return { ...normalizeReceipt(existing.rows[0]), idempotent: true };
|
||||
}
|
||||
|
||||
const records = batch.facts.map((fact) => ({
|
||||
source_id: fact.sourceId,
|
||||
semantic_type: fact.semanticType,
|
||||
observed_at: fact.observedAt,
|
||||
received_at: batch.batch.receivedAt,
|
||||
attributes: fact.attributes || {},
|
||||
geometry: fact.geometry || null,
|
||||
fingerprint: fingerprint(fact),
|
||||
}));
|
||||
|
||||
const updated = await client.query(
|
||||
`insert into external_data_plane_current (
|
||||
tenant_id, connection_id, provider_id, data_product_id, source_id, semantic_type,
|
||||
observed_at, received_at, attributes, geometry, fingerprint, updated_at
|
||||
)
|
||||
select $1, $2, $3, $4, item.source_id, item.semantic_type,
|
||||
item.observed_at, item.received_at, coalesce(item.attributes, '{}'::jsonb),
|
||||
case when item.geometry is null then null
|
||||
else ST_SetSRID(ST_MakePoint(
|
||||
(item.geometry->'coordinates'->>0)::double precision,
|
||||
(item.geometry->'coordinates'->>1)::double precision
|
||||
), 4326)::geography end,
|
||||
item.fingerprint, now()
|
||||
from jsonb_to_recordset($5::jsonb) as item(
|
||||
source_id text, semantic_type text, observed_at timestamptz, received_at timestamptz,
|
||||
attributes jsonb, geometry jsonb, fingerprint text
|
||||
)
|
||||
on conflict (tenant_id, connection_id, provider_id, data_product_id, source_id, semantic_type)
|
||||
do update set
|
||||
observed_at = excluded.observed_at,
|
||||
received_at = excluded.received_at,
|
||||
attributes = excluded.attributes,
|
||||
geometry = excluded.geometry,
|
||||
fingerprint = excluded.fingerprint,
|
||||
updated_at = now()
|
||||
where excluded.observed_at > external_data_plane_current.observed_at
|
||||
or (excluded.observed_at = external_data_plane_current.observed_at
|
||||
and excluded.fingerprint <> external_data_plane_current.fingerprint)
|
||||
returning source_id as "sourceId", semantic_type as "semanticType",
|
||||
observed_at as "observedAt", received_at as "receivedAt", attributes,
|
||||
case when geometry is null then null
|
||||
else jsonb_build_object('type', 'Point', 'coordinates', jsonb_build_array(
|
||||
ST_X(geometry::geometry), ST_Y(geometry::geometry)
|
||||
)) end as geometry,
|
||||
fingerprint`,
|
||||
[
|
||||
batch.source.tenantId, batch.source.connectionId, batch.source.providerId,
|
||||
batch.contract.dataProductId, JSON.stringify(records),
|
||||
],
|
||||
);
|
||||
|
||||
const historyInsertedCount = await persistHistory(client, batch, definition.historyPolicy, records);
|
||||
const operations = updated.rows.map((fact) => ({ op: "upsert", fact: publicFact(fact) }));
|
||||
const chunks = definition.deliveryMode === "snapshot+patch"
|
||||
? chunkOperations(operations, maxPatchOperations, maxPatchBytes)
|
||||
: [];
|
||||
const cursor = chunks.length ? await persistPatchChunks(client, batch, definition, batchId, chunks) : await currentCursor(client, batch);
|
||||
|
||||
await client.query(
|
||||
`update external_data_plane_batches set
|
||||
current_updated_count = $2,
|
||||
history_inserted_count = $3,
|
||||
patch_operation_count = $4,
|
||||
delivery_cursor = $5
|
||||
where id = $1`,
|
||||
[batchId, updated.rowCount, historyInsertedCount, operations.length, cursor],
|
||||
);
|
||||
await client.query("commit");
|
||||
return normalizeReceipt({
|
||||
batchId,
|
||||
idempotent: false,
|
||||
publishedFactCount: batch.facts.length,
|
||||
currentUpdatedCount: updated.rowCount,
|
||||
historyInsertedCount,
|
||||
patchOperationCount: operations.length,
|
||||
cursor: String(cursor),
|
||||
acceptedAt: batch.batch.receivedAt,
|
||||
});
|
||||
} catch (error) {
|
||||
await client.query("rollback");
|
||||
throw error;
|
||||
} finally {
|
||||
client.release();
|
||||
}
|
||||
}
|
||||
|
||||
export async function readDataProductSnapshot(pool, binding, definition, { limit = 5000 } = {}) {
|
||||
const client = await pool.connect();
|
||||
try {
|
||||
await client.query("begin isolation level repeatable read read only");
|
||||
const cursor = await currentCursor(client, binding, definition.id);
|
||||
const rows = await client.query(
|
||||
`select source_id as "sourceId", semantic_type as "semanticType",
|
||||
observed_at as "observedAt", received_at as "receivedAt", attributes,
|
||||
case when geometry is null then null
|
||||
else jsonb_build_object('type', 'Point', 'coordinates', jsonb_build_array(
|
||||
ST_X(geometry::geometry), ST_Y(geometry::geometry)
|
||||
)) end as geometry
|
||||
from external_data_plane_current
|
||||
where tenant_id = $1 and connection_id = $2 and provider_id = $3 and data_product_id = $4
|
||||
order by source_id asc, semantic_type asc
|
||||
limit $5`,
|
||||
[binding.tenantId, binding.connectionId, binding.providerId, definition.id, limit + 1],
|
||||
);
|
||||
if (rows.rowCount > limit) throw deliveryError("data_product_snapshot_limit_exceeded", 413);
|
||||
await client.query("commit");
|
||||
return {
|
||||
schemaVersion: DATA_PRODUCT_SNAPSHOT_SCHEMA_VERSION,
|
||||
dataProduct: { id: definition.id, version: definition.version },
|
||||
generatedAt: new Date().toISOString(),
|
||||
cursor: String(cursor),
|
||||
facts: rows.rows.map(publicFact),
|
||||
};
|
||||
} catch (error) {
|
||||
await client.query("rollback");
|
||||
throw error;
|
||||
} finally {
|
||||
client.release();
|
||||
}
|
||||
}
|
||||
|
||||
export async function readPatchEvents(pool, binding, definition, after, { limit = 100 } = {}) {
|
||||
const client = await pool.connect();
|
||||
try {
|
||||
await client.query("begin isolation level repeatable read read only");
|
||||
const current = await currentCursor(client, binding, definition.id);
|
||||
if (after > current) throw deliveryError("resync_required", 409);
|
||||
const floor = await patchFloor(client, binding, definition.id);
|
||||
if (after < floor - 1n) throw deliveryError("resync_required", 409);
|
||||
const rows = await client.query(
|
||||
`select cursor::text, previous_cursor::text as "previousCursor",
|
||||
operations, emitted_at as "emittedAt"
|
||||
from external_data_plane_patch_outbox
|
||||
where tenant_id = $1 and connection_id = $2 and provider_id = $3
|
||||
and data_product_id = $4 and cursor > $5
|
||||
order by cursor asc limit $6`,
|
||||
[binding.tenantId, binding.connectionId, binding.providerId, definition.id, after.toString(), limit],
|
||||
);
|
||||
await client.query("commit");
|
||||
return rows.rows.map((row) => ({
|
||||
schemaVersion: DATA_PRODUCT_PATCH_SCHEMA_VERSION,
|
||||
dataProduct: { id: definition.id, version: definition.version },
|
||||
previousCursor: row.previousCursor,
|
||||
cursor: row.cursor,
|
||||
emittedAt: new Date(row.emittedAt).toISOString(),
|
||||
operations: row.operations,
|
||||
}));
|
||||
} catch (error) {
|
||||
await client.query("rollback");
|
||||
throw error;
|
||||
} finally {
|
||||
client.release();
|
||||
}
|
||||
}
|
||||
|
||||
export async function prunePatchOutbox(db, { retentionMs, limit = 10_000 }) {
|
||||
const cutoff = new Date(Date.now() - retentionMs);
|
||||
const result = await db.query(
|
||||
`with expired as (
|
||||
select tenant_id, connection_id, provider_id, data_product_id, cursor
|
||||
from external_data_plane_patch_outbox
|
||||
where emitted_at < $1
|
||||
order by emitted_at asc
|
||||
limit $2
|
||||
)
|
||||
delete from external_data_plane_patch_outbox as target
|
||||
using expired
|
||||
where target.tenant_id = expired.tenant_id
|
||||
and target.connection_id = expired.connection_id
|
||||
and target.provider_id = expired.provider_id
|
||||
and target.data_product_id = expired.data_product_id
|
||||
and target.cursor = expired.cursor`,
|
||||
[cutoff, limit],
|
||||
);
|
||||
return result.rowCount;
|
||||
}
|
||||
|
||||
export async function pruneDataProductHistory(db, { limit = 10_000 } = {}) {
|
||||
const result = await db.query(
|
||||
`with expired as (
|
||||
select history.tenant_id, history.connection_id, history.provider_id,
|
||||
history.data_product_id, history.source_id, history.semantic_type, history.bucket_start
|
||||
from external_data_plane_history as history
|
||||
join external_data_plane_products as product on product.id = history.data_product_id
|
||||
where history.bucket_start < now() - (
|
||||
greatest(1, coalesce((product.history_policy->>'retentionDays')::integer, 1)) * interval '1 day'
|
||||
)
|
||||
order by history.bucket_start asc
|
||||
limit $1
|
||||
)
|
||||
delete from external_data_plane_history as target
|
||||
using expired
|
||||
where target.tenant_id = expired.tenant_id
|
||||
and target.connection_id = expired.connection_id
|
||||
and target.provider_id = expired.provider_id
|
||||
and target.data_product_id = expired.data_product_id
|
||||
and target.source_id = expired.source_id
|
||||
and target.semantic_type = expired.semantic_type
|
||||
and target.bucket_start = expired.bucket_start`,
|
||||
[limit],
|
||||
);
|
||||
return result.rowCount;
|
||||
}
|
||||
|
||||
export async function pruneBatchReceipts(db, { retentionMs, limit = 10_000 }) {
|
||||
const cutoff = new Date(Date.now() - retentionMs);
|
||||
const result = await db.query(
|
||||
`with expired as (
|
||||
select batch.id
|
||||
from external_data_plane_batches as batch
|
||||
where batch.created_at < $1
|
||||
and not exists (select 1 from external_data_plane_patch_outbox as patch where patch.batch_id = batch.id)
|
||||
and not exists (select 1 from external_data_plane_raw_envelopes as raw where raw.batch_id = batch.id)
|
||||
order by batch.created_at asc
|
||||
limit $2
|
||||
)
|
||||
delete from external_data_plane_batches as target
|
||||
using expired
|
||||
where target.id = expired.id`,
|
||||
[cutoff, limit],
|
||||
);
|
||||
return result.rowCount;
|
||||
}
|
||||
|
||||
async function persistHistory(client, batch, policy, facts) {
|
||||
if (!facts.length || policy?.mode === "none") return 0;
|
||||
const intervalMs = policy?.mode === "sampled" ? Number(policy.intervalMs) : 1;
|
||||
const records = facts.map((fact) => ({
|
||||
...fact,
|
||||
bucket_start: new Date(Math.floor(new Date(fact.observed_at).getTime() / intervalMs) * intervalMs).toISOString(),
|
||||
}));
|
||||
const result = await client.query(
|
||||
`insert into external_data_plane_history (
|
||||
tenant_id, connection_id, provider_id, data_product_id, source_id, semantic_type,
|
||||
bucket_start, observed_at, received_at, attributes, geometry, fingerprint, updated_at
|
||||
)
|
||||
select $1, $2, $3, $4, item.source_id, item.semantic_type,
|
||||
item.bucket_start, item.observed_at, item.received_at, coalesce(item.attributes, '{}'::jsonb),
|
||||
case when item.geometry is null then null
|
||||
else ST_SetSRID(ST_MakePoint(
|
||||
(item.geometry->'coordinates'->>0)::double precision,
|
||||
(item.geometry->'coordinates'->>1)::double precision
|
||||
), 4326)::geography end,
|
||||
item.fingerprint, now()
|
||||
from jsonb_to_recordset($5::jsonb) as item(
|
||||
source_id text, semantic_type text, bucket_start timestamptz,
|
||||
observed_at timestamptz, received_at timestamptz,
|
||||
attributes jsonb, geometry jsonb, fingerprint text
|
||||
)
|
||||
on conflict (tenant_id, connection_id, provider_id, data_product_id, source_id, semantic_type, bucket_start)
|
||||
do update set
|
||||
observed_at = excluded.observed_at,
|
||||
received_at = excluded.received_at,
|
||||
attributes = excluded.attributes,
|
||||
geometry = excluded.geometry,
|
||||
fingerprint = excluded.fingerprint,
|
||||
updated_at = now()
|
||||
where excluded.observed_at >= external_data_plane_history.observed_at`,
|
||||
[
|
||||
batch.source.tenantId, batch.source.connectionId, batch.source.providerId,
|
||||
batch.contract.dataProductId,
|
||||
JSON.stringify(records),
|
||||
],
|
||||
);
|
||||
return result.rowCount;
|
||||
}
|
||||
|
||||
async function persistPatchChunks(client, batch, definition, batchId, chunks) {
|
||||
const endCursorResult = await client.query(
|
||||
`insert into external_data_plane_delivery_state (
|
||||
tenant_id, connection_id, provider_id, data_product_id, current_cursor
|
||||
) values ($1, $2, $3, $4, $5)
|
||||
on conflict (tenant_id, connection_id, provider_id, data_product_id)
|
||||
do update set
|
||||
current_cursor = external_data_plane_delivery_state.current_cursor + excluded.current_cursor,
|
||||
updated_at = now()
|
||||
returning current_cursor`,
|
||||
[
|
||||
batch.source.tenantId, batch.source.connectionId, batch.source.providerId,
|
||||
definition.id, chunks.length,
|
||||
],
|
||||
);
|
||||
const end = BigInt(endCursorResult.rows[0].current_cursor);
|
||||
const start = end - BigInt(chunks.length) + 1n;
|
||||
for (let index = 0; index < chunks.length; index += 1) {
|
||||
const cursor = start + BigInt(index);
|
||||
await client.query(
|
||||
`insert into external_data_plane_patch_outbox (
|
||||
tenant_id, connection_id, provider_id, data_product_id,
|
||||
cursor, previous_cursor, batch_id, operations
|
||||
) values ($1, $2, $3, $4, $5, $6, $7, $8::jsonb)`,
|
||||
[
|
||||
batch.source.tenantId, batch.source.connectionId, batch.source.providerId,
|
||||
definition.id, cursor.toString(), (cursor - 1n).toString(), batchId,
|
||||
JSON.stringify(chunks[index]),
|
||||
],
|
||||
);
|
||||
}
|
||||
return end;
|
||||
}
|
||||
|
||||
async function currentCursor(db, scope, explicitProductId) {
|
||||
const dataProductId = explicitProductId || scope.contract.dataProductId;
|
||||
const result = await db.query(
|
||||
`select current_cursor from external_data_plane_delivery_state
|
||||
where tenant_id = $1 and connection_id = $2 and provider_id = $3 and data_product_id = $4`,
|
||||
[scope.tenantId || scope.source.tenantId, scope.connectionId || scope.source.connectionId,
|
||||
scope.providerId || scope.source.providerId, dataProductId],
|
||||
);
|
||||
return result.rowCount ? BigInt(result.rows[0].current_cursor) : 0n;
|
||||
}
|
||||
|
||||
async function patchFloor(db, binding, dataProductId) {
|
||||
const result = await db.query(
|
||||
`select min(cursor) as floor from external_data_plane_patch_outbox
|
||||
where tenant_id = $1 and connection_id = $2 and provider_id = $3 and data_product_id = $4`,
|
||||
[binding.tenantId, binding.connectionId, binding.providerId, dataProductId],
|
||||
);
|
||||
if (result.rows[0]?.floor !== null && result.rows[0]?.floor !== undefined) return BigInt(result.rows[0].floor);
|
||||
const current = await currentCursor(db, binding, dataProductId);
|
||||
return current + 1n;
|
||||
}
|
||||
|
||||
function scopeValues(batch, tail) {
|
||||
return [
|
||||
batch.source.tenantId, batch.source.connectionId, batch.source.providerId,
|
||||
batch.contract.dataProductId, tail,
|
||||
];
|
||||
}
|
||||
|
||||
function normalizeReceipt(value) {
|
||||
return {
|
||||
batchId: value.batchId,
|
||||
idempotent: value.idempotent === true,
|
||||
publishedFactCount: Number(value.publishedFactCount || 0),
|
||||
currentUpdatedCount: Number(value.currentUpdatedCount || 0),
|
||||
historyInsertedCount: Number(value.historyInsertedCount || 0),
|
||||
patchOperationCount: Number(value.patchOperationCount || 0),
|
||||
cursor: String(value.cursor || "0"),
|
||||
acceptedAt: new Date(value.acceptedAt).toISOString(),
|
||||
};
|
||||
}
|
||||
|
||||
function publicFact(fact) {
|
||||
const value = {
|
||||
sourceId: fact.sourceId,
|
||||
semanticType: fact.semanticType,
|
||||
observedAt: new Date(fact.observedAt).toISOString(),
|
||||
receivedAt: new Date(fact.receivedAt).toISOString(),
|
||||
attributes: fact.attributes || {},
|
||||
};
|
||||
if (fact.geometry) value.geometry = fact.geometry;
|
||||
return value;
|
||||
}
|
||||
|
||||
function chunkOperations(values, maxOperations, maxBytes) {
|
||||
const result = [];
|
||||
let current = [];
|
||||
let currentBytes = 2;
|
||||
for (const value of values) {
|
||||
const serializedBytes = Buffer.byteLength(JSON.stringify(value));
|
||||
if (serializedBytes + 2 > maxBytes) throw deliveryError("data_product_patch_operation_size_exceeded", 413);
|
||||
const separatorBytes = current.length ? 1 : 0;
|
||||
if (current.length && (current.length >= maxOperations || currentBytes + separatorBytes + serializedBytes > maxBytes)) {
|
||||
result.push(current);
|
||||
current = [];
|
||||
currentBytes = 2;
|
||||
}
|
||||
current.push(value);
|
||||
currentBytes += (current.length > 1 ? 1 : 0) + serializedBytes;
|
||||
}
|
||||
if (current.length) result.push(current);
|
||||
return result;
|
||||
}
|
||||
|
||||
function fingerprint(value) {
|
||||
return createHash("sha256").update(stableJson(value)).digest("hex");
|
||||
}
|
||||
|
||||
function publishFingerprint(batch) {
|
||||
return fingerprint({
|
||||
schemaVersion: batch.schemaVersion,
|
||||
source: batch.source,
|
||||
contract: batch.contract,
|
||||
batch: {
|
||||
runId: batch.batch.runId,
|
||||
sequence: batch.batch.sequence,
|
||||
idempotencyKey: batch.batch.idempotencyKey,
|
||||
},
|
||||
facts: batch.facts,
|
||||
});
|
||||
}
|
||||
|
||||
export function assertPublishMatchesDefinition(batch, definition) {
|
||||
const semanticTypes = new Set(Array.isArray(definition?.semanticTypes) ? definition.semanticTypes : []);
|
||||
const fields = new Set(Array.isArray(definition?.fields) ? definition.fields : []);
|
||||
const entityKeys = new Set();
|
||||
for (const fact of batch?.facts || []) {
|
||||
if (!semanticTypes.has(fact.semanticType)) throw deliveryError("data_product_semantic_type_forbidden", 422);
|
||||
const entityKey = `${fact.sourceId}\u0000${fact.semanticType}`;
|
||||
if (entityKeys.has(entityKey)) throw deliveryError("data_product_duplicate_entity_key", 422);
|
||||
entityKeys.add(entityKey);
|
||||
for (const attribute of Object.keys(fact.attributes || {})) {
|
||||
if (!fields.has(attribute) && !fields.has(`attributes.${attribute}`)) {
|
||||
throw deliveryError("data_product_field_forbidden", 422);
|
||||
}
|
||||
}
|
||||
if (fact.geometry !== undefined && !geometryDeclared(fields)) {
|
||||
throw deliveryError("data_product_geometry_forbidden", 422);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function geometryDeclared(fields) {
|
||||
return fields.has("geometry")
|
||||
|| fields.has("coordinates")
|
||||
|| (fields.has("longitude") && fields.has("latitude"));
|
||||
}
|
||||
|
||||
function stableJson(value) {
|
||||
if (Array.isArray(value)) return `[${value.map(stableJson).join(",")}]`;
|
||||
if (value && typeof value === "object") {
|
||||
return `{${Object.keys(value).sort().map((key) => `${JSON.stringify(key)}:${stableJson(value[key])}`).join(",")}}`;
|
||||
}
|
||||
return JSON.stringify(value);
|
||||
}
|
||||
|
||||
function deliveryError(code, status) {
|
||||
return Object.assign(new Error(code), { status, code });
|
||||
}
|
||||
|
|
@ -1,118 +0,0 @@
|
|||
const IDENTIFIER = /^[a-z][a-z0-9._:-]{2,127}$/;
|
||||
const SEMVER = /^\d+\.\d+\.\d+(?:[-+][a-z0-9.-]+)?$/i;
|
||||
const DELIVERY_MODES = new Set(["snapshot", "snapshot+patch", "query"]);
|
||||
const HISTORY_MODES = new Set(["none", "all", "sampled"]);
|
||||
const DEFINITION_KEYS = new Set([
|
||||
"id", "version", "ontologyRevision", "deliveryMode", "semanticTypes", "fields", "history",
|
||||
]);
|
||||
const HISTORY_KEYS = new Set(["mode", "intervalMs", "retentionDays", "strategy"]);
|
||||
|
||||
export function normalizeDataProductDefinition(value) {
|
||||
if (!isPlainObject(value) || !hasOnlyKeys(value, DEFINITION_KEYS)) {
|
||||
throw policyError("data_product_definition_invalid");
|
||||
}
|
||||
const id = identifier(value.id);
|
||||
const version = string(value.version);
|
||||
const ontologyRevision = identifier(value.ontologyRevision);
|
||||
const deliveryMode = string(value.deliveryMode);
|
||||
const semanticTypes = identifierSet(
|
||||
value.semanticTypes,
|
||||
"data_product_definition_semantic_types_invalid",
|
||||
"data_product_definition_semantic_types_duplicate",
|
||||
);
|
||||
const fields = identifierSet(
|
||||
value.fields,
|
||||
"data_product_definition_fields_invalid",
|
||||
"data_product_definition_fields_duplicate",
|
||||
);
|
||||
if (!id || !SEMVER.test(version) || !ontologyRevision || !DELIVERY_MODES.has(deliveryMode)) {
|
||||
throw policyError("data_product_definition_identity_invalid");
|
||||
}
|
||||
if (!semanticTypes.length || !fields.length) throw policyError("data_product_definition_shape_invalid");
|
||||
|
||||
const history = normalizeHistoryPolicy(value.history);
|
||||
return Object.freeze({ id, version, ontologyRevision, deliveryMode, semanticTypes, fields, history });
|
||||
}
|
||||
|
||||
export function normalizeHistoryPolicy(value = { mode: "none" }) {
|
||||
if (!isPlainObject(value) || !hasOnlyKeys(value, HISTORY_KEYS)) throw policyError("history_policy_invalid");
|
||||
const mode = string(value.mode);
|
||||
if (!HISTORY_MODES.has(mode)) throw policyError("history_policy_mode_invalid");
|
||||
const retentionDays = integer(value.retentionDays, mode === "none" ? 1 : 90, 1, 3650);
|
||||
if (mode === "none") {
|
||||
if (value.intervalMs !== undefined || value.strategy !== undefined) throw policyError("history_policy_none_has_sampling_fields");
|
||||
return Object.freeze({ mode, retentionDays });
|
||||
}
|
||||
if (mode === "all") {
|
||||
if (value.intervalMs !== undefined || value.strategy !== undefined) throw policyError("history_policy_all_has_sampling_fields");
|
||||
return Object.freeze({ mode, retentionDays });
|
||||
}
|
||||
|
||||
const intervalMs = integer(value.intervalMs, 60_000, 1_000, 24 * 60 * 60 * 1000);
|
||||
const strategy = string(value.strategy || "latest-per-entity-per-bucket");
|
||||
if (strategy !== "latest-per-entity-per-bucket") throw policyError("history_policy_strategy_invalid");
|
||||
return Object.freeze({ mode, intervalMs, strategy, retentionDays });
|
||||
}
|
||||
|
||||
export function safeDataProductDefinition(row) {
|
||||
return {
|
||||
id: row.id,
|
||||
version: row.version,
|
||||
ontologyRevision: row.ontologyRevision,
|
||||
deliveryMode: row.deliveryMode,
|
||||
semanticTypes: array(row.semanticTypes),
|
||||
fields: array(row.fields),
|
||||
history: isPlainObject(row.historyPolicy) ? row.historyPolicy : {},
|
||||
active: row.active === true,
|
||||
createdAt: iso(row.createdAt),
|
||||
updatedAt: iso(row.updatedAt),
|
||||
};
|
||||
}
|
||||
|
||||
function policyError(code) {
|
||||
return Object.assign(new Error(code), { status: 400, code });
|
||||
}
|
||||
|
||||
function identifier(value) {
|
||||
const normalized = string(value);
|
||||
return IDENTIFIER.test(normalized) ? normalized : "";
|
||||
}
|
||||
|
||||
function identifierSet(value, invalidCode, duplicateCode) {
|
||||
if (!Array.isArray(value)) throw policyError(invalidCode);
|
||||
|
||||
const normalized = value.map((entry) => {
|
||||
const result = identifier(entry);
|
||||
if (!result) throw policyError(invalidCode);
|
||||
return result;
|
||||
});
|
||||
if (new Set(normalized).size !== normalized.length) throw policyError(duplicateCode);
|
||||
|
||||
return Object.freeze(normalized.sort());
|
||||
}
|
||||
|
||||
function integer(value, fallback, min, max) {
|
||||
const number = value === undefined ? fallback : Number(value);
|
||||
if (!Number.isInteger(number) || number < min || number > max) throw policyError("history_policy_number_invalid");
|
||||
return number;
|
||||
}
|
||||
|
||||
function string(value) {
|
||||
return typeof value === "string" ? value.trim() : "";
|
||||
}
|
||||
|
||||
function array(value) {
|
||||
return Array.isArray(value) ? value : [];
|
||||
}
|
||||
|
||||
function iso(value) {
|
||||
return value ? new Date(value).toISOString() : undefined;
|
||||
}
|
||||
|
||||
function isPlainObject(value) {
|
||||
return Boolean(value) && typeof value === "object" && !Array.isArray(value);
|
||||
}
|
||||
|
||||
function hasOnlyKeys(value, allowed) {
|
||||
return Object.keys(value).every((key) => allowed.has(key));
|
||||
}
|
||||
|
|
@ -1,43 +0,0 @@
|
|||
import { lstat, readFile, readdir } from "node:fs/promises";
|
||||
import { basename, join, resolve } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
import { normalizeDataProductDefinition } from "./data-product-policy.mjs";
|
||||
import { persistDataProductDefinition } from "./data-product-delivery.mjs";
|
||||
|
||||
const bundledDefinitionsDir = fileURLToPath(new URL("../definitions/", import.meta.url));
|
||||
|
||||
export async function loadDataProductDefinitions(directory = bundledDefinitionsDir) {
|
||||
const root = resolve(directory);
|
||||
const entries = (await readdir(root, { withFileTypes: true }))
|
||||
.filter((entry) => entry.name.endsWith(".json"))
|
||||
.sort((left, right) => left.name.localeCompare(right.name));
|
||||
const definitions = [];
|
||||
for (const entry of entries) {
|
||||
if (!entry.isFile() || entry.isSymbolicLink()) throw new Error("data_product_definition_file_invalid");
|
||||
const path = join(root, entry.name);
|
||||
const metadata = await lstat(path);
|
||||
if (!metadata.isFile() || metadata.isSymbolicLink() || metadata.size < 2 || metadata.size > 256 * 1024) {
|
||||
throw new Error("data_product_definition_file_invalid");
|
||||
}
|
||||
let value;
|
||||
try {
|
||||
value = JSON.parse(await readFile(path, "utf8"));
|
||||
} catch {
|
||||
throw new Error("data_product_definition_json_invalid");
|
||||
}
|
||||
const definition = normalizeDataProductDefinition(value);
|
||||
if (`${definition.id}.json` !== basename(path)) throw new Error("data_product_definition_filename_mismatch");
|
||||
definitions.push(definition);
|
||||
}
|
||||
if (new Set(definitions.map((definition) => definition.id)).size !== definitions.length) {
|
||||
throw new Error("data_product_definition_duplicate");
|
||||
}
|
||||
return Object.freeze(definitions);
|
||||
}
|
||||
|
||||
export async function reconcileDataProductDefinitions(db, directory = bundledDefinitionsDir) {
|
||||
const definitions = await loadDataProductDefinitions(directory);
|
||||
for (const definition of definitions) await persistDataProductDefinition(db, definition);
|
||||
return definitions;
|
||||
}
|
||||
|
|
@ -1,37 +0,0 @@
|
|||
export function assertBatchTimeBounds(batch, { now = new Date(), maxFutureSkewSeconds = 300 } = {}) {
|
||||
const acceptedAt = validDate(now, "intake_policy_clock_invalid");
|
||||
const maxFutureAt = new Date(acceptedAt.getTime() + Number(maxFutureSkewSeconds) * 1000);
|
||||
if (!Number.isInteger(maxFutureSkewSeconds) || maxFutureSkewSeconds < 0) {
|
||||
throw intakePolicyError("intake_policy_future_skew_invalid");
|
||||
}
|
||||
|
||||
const receivedAt = validDate(batch?.batch?.receivedAt, "batch_received_at_invalid");
|
||||
if (receivedAt > maxFutureAt) throw intakePolicyError("batch_received_at_too_far_in_future");
|
||||
|
||||
for (const fact of batch?.facts || []) {
|
||||
const observedAt = validDate(fact?.observedAt, "fact_observed_at_invalid");
|
||||
if (observedAt > maxFutureAt) throw intakePolicyError("fact_observed_at_too_far_in_future");
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Retention is based on the server's acceptance time, never on provider or L2
|
||||
* timestamps supplied in a batch.
|
||||
*/
|
||||
export function rawRetentionExpiry({ now = new Date(), rawRetentionDays }) {
|
||||
const acceptedAt = validDate(now, "intake_policy_clock_invalid");
|
||||
if (!Number.isInteger(rawRetentionDays) || rawRetentionDays < 1) {
|
||||
throw intakePolicyError("raw_retention_days_invalid");
|
||||
}
|
||||
return new Date(acceptedAt.getTime() + rawRetentionDays * 24 * 60 * 60 * 1000);
|
||||
}
|
||||
|
||||
export function intakePolicyError(code) {
|
||||
return Object.assign(new Error(code), { status: 422, code });
|
||||
}
|
||||
|
||||
function validDate(value, code) {
|
||||
const parsed = value instanceof Date ? new Date(value.getTime()) : new Date(String(value ?? ""));
|
||||
if (Number.isNaN(parsed.getTime())) throw intakePolicyError(code);
|
||||
return parsed;
|
||||
}
|
||||
|
|
@ -1,90 +0,0 @@
|
|||
import { createHash, randomBytes } from "node:crypto";
|
||||
|
||||
const IDENTIFIER = /^[a-z][a-z0-9._:-]{2,127}$/;
|
||||
const TOKEN_PREFIX = "ndc_edprb_";
|
||||
const SECRET_LIKE_KEY = /(token|secret|password|authorization|access[_-]?token|refresh[_-]?token|api[_-]?key)/i;
|
||||
const REQUEST_KEYS = new Set(["source", "allowedDataProductIds", "expiresAt"]);
|
||||
const SOURCE_KEYS = new Set(["tenantId", "connectionId", "providerId"]);
|
||||
|
||||
export function createReaderToken() {
|
||||
return `${TOKEN_PREFIX}${randomBytes(32).toString("base64url")}`;
|
||||
}
|
||||
|
||||
export function hashReaderToken(token) {
|
||||
return createHash("sha256").update(String(token), "utf8").digest("hex");
|
||||
}
|
||||
|
||||
export function normalizeReaderBindingRequest(value, { now = new Date(), maxTtlDays = 90 } = {}) {
|
||||
if (!isPlainObject(value) || !isPlainObject(value.source)) throw readerError("reader_binding_request_invalid");
|
||||
if (containsSecretLikeKey(value)) throw readerError("reader_binding_request_secret_material_forbidden");
|
||||
if (!hasOnlyKeys(value, REQUEST_KEYS) || !hasOnlyKeys(value.source, SOURCE_KEYS)) {
|
||||
throw readerError("reader_binding_request_fields_invalid");
|
||||
}
|
||||
const tenantId = identifier(value.source.tenantId);
|
||||
const connectionId = identifier(value.source.connectionId);
|
||||
const providerId = identifier(value.source.providerId);
|
||||
const allowedDataProductIds = uniqueIdentifiers(value.allowedDataProductIds);
|
||||
if (!tenantId || !connectionId || !providerId || !allowedDataProductIds.length) {
|
||||
throw readerError("reader_binding_scope_invalid");
|
||||
}
|
||||
const expiresAt = new Date(String(value.expiresAt || ""));
|
||||
const maxExpiresAt = new Date(now.getTime() + maxTtlDays * 24 * 60 * 60 * 1000);
|
||||
if (Number.isNaN(expiresAt.getTime()) || expiresAt <= now || expiresAt > maxExpiresAt) {
|
||||
throw readerError("reader_binding_expiry_invalid");
|
||||
}
|
||||
return Object.freeze({ tenantId, connectionId, providerId, allowedDataProductIds, expiresAt: expiresAt.toISOString() });
|
||||
}
|
||||
|
||||
export function assertReaderProduct(binding, dataProductId, now = new Date()) {
|
||||
if (!isPlainObject(binding) || binding.active !== true || new Date(binding.expiresAt) <= now) {
|
||||
throw readerError("reader_binding_inactive", 401);
|
||||
}
|
||||
const normalized = identifier(dataProductId);
|
||||
if (!normalized || !uniqueIdentifiers(binding.allowedDataProductIds).includes(normalized)) {
|
||||
throw readerError("reader_binding_data_product_forbidden", 403);
|
||||
}
|
||||
return normalized;
|
||||
}
|
||||
|
||||
export function safeReaderBinding(binding) {
|
||||
return {
|
||||
id: binding.id,
|
||||
tenantId: binding.tenantId,
|
||||
connectionId: binding.connectionId,
|
||||
providerId: binding.providerId,
|
||||
allowedDataProductIds: uniqueIdentifiers(binding.allowedDataProductIds),
|
||||
active: binding.active === true,
|
||||
expiresAt: new Date(binding.expiresAt).toISOString(),
|
||||
createdAt: binding.createdAt ? new Date(binding.createdAt).toISOString() : undefined,
|
||||
rotatedAt: binding.rotatedAt ? new Date(binding.rotatedAt).toISOString() : undefined,
|
||||
revokedAt: binding.revokedAt ? new Date(binding.revokedAt).toISOString() : undefined,
|
||||
};
|
||||
}
|
||||
|
||||
function readerError(code, status = 400) {
|
||||
return Object.assign(new Error(code), { status, code });
|
||||
}
|
||||
|
||||
function identifier(value) {
|
||||
const normalized = typeof value === "string" ? value.trim() : "";
|
||||
return IDENTIFIER.test(normalized) ? normalized : "";
|
||||
}
|
||||
|
||||
function uniqueIdentifiers(value) {
|
||||
if (!Array.isArray(value)) return [];
|
||||
return [...new Set(value.map(identifier).filter(Boolean))];
|
||||
}
|
||||
|
||||
function containsSecretLikeKey(value) {
|
||||
if (Array.isArray(value)) return value.some(containsSecretLikeKey);
|
||||
if (!isPlainObject(value)) return false;
|
||||
return Object.entries(value).some(([key, child]) => SECRET_LIKE_KEY.test(key) || containsSecretLikeKey(child));
|
||||
}
|
||||
|
||||
function isPlainObject(value) {
|
||||
return Boolean(value) && typeof value === "object" && !Array.isArray(value);
|
||||
}
|
||||
|
||||
function hasOnlyKeys(value, allowed) {
|
||||
return Object.keys(value).every((key) => allowed.has(key));
|
||||
}
|
||||
|
|
@ -1,206 +0,0 @@
|
|||
export async function migrate(pool) {
|
||||
await pool.query("create extension if not exists timescaledb");
|
||||
await pool.query("create extension if not exists postgis");
|
||||
|
||||
await pool.query(`
|
||||
create table if not exists external_data_plane_products (
|
||||
id text primary key,
|
||||
version text not null,
|
||||
ontology_revision text not null,
|
||||
delivery_mode text not null,
|
||||
semantic_types jsonb not null,
|
||||
fields jsonb not null,
|
||||
history_policy jsonb not null,
|
||||
active boolean not null default true,
|
||||
created_at timestamptz not null default now(),
|
||||
updated_at timestamptz not null default now(),
|
||||
check (jsonb_typeof(semantic_types) = 'array' and jsonb_array_length(semantic_types) > 0),
|
||||
check (jsonb_typeof(fields) = 'array' and jsonb_array_length(fields) > 0)
|
||||
)
|
||||
`);
|
||||
|
||||
await pool.query(`
|
||||
create table if not exists external_data_plane_batches (
|
||||
id uuid primary key,
|
||||
tenant_id text not null,
|
||||
connection_id text not null,
|
||||
provider_id text not null,
|
||||
data_product_id text not null,
|
||||
contract_version text not null,
|
||||
ontology_revision text not null,
|
||||
run_id text not null,
|
||||
sequence integer not null,
|
||||
idempotency_key text not null,
|
||||
received_at timestamptz not null,
|
||||
fact_count integer not null default 0,
|
||||
inserted_fact_count integer not null default 0,
|
||||
created_at timestamptz not null default now(),
|
||||
unique (tenant_id, connection_id, provider_id, data_product_id, idempotency_key)
|
||||
)
|
||||
`);
|
||||
await pool.query("create index if not exists external_data_plane_batches_scope_idx on external_data_plane_batches (tenant_id, connection_id, data_product_id, received_at desc)");
|
||||
await pool.query("alter table external_data_plane_batches add column if not exists current_updated_count integer not null default 0");
|
||||
await pool.query("alter table external_data_plane_batches add column if not exists history_inserted_count integer not null default 0");
|
||||
await pool.query("alter table external_data_plane_batches add column if not exists patch_operation_count integer not null default 0");
|
||||
await pool.query("alter table external_data_plane_batches add column if not exists delivery_cursor bigint");
|
||||
await pool.query("alter table external_data_plane_batches add column if not exists request_fingerprint text");
|
||||
|
||||
await pool.query(`
|
||||
create table if not exists external_data_plane_raw_envelopes (
|
||||
id uuid primary key,
|
||||
batch_id uuid not null unique references external_data_plane_batches(id) on delete cascade,
|
||||
payload_hash text not null,
|
||||
content_type text not null,
|
||||
payload jsonb,
|
||||
payload_ref text,
|
||||
payload_bytes integer,
|
||||
received_at timestamptz not null,
|
||||
expires_at timestamptz not null,
|
||||
check (payload is not null or payload_ref is not null)
|
||||
)
|
||||
`);
|
||||
await pool.query("create index if not exists external_data_plane_raw_expiry_idx on external_data_plane_raw_envelopes (expires_at)");
|
||||
|
||||
await pool.query(`
|
||||
create table if not exists external_data_plane_writer_bindings (
|
||||
id uuid primary key,
|
||||
token_hash text not null unique,
|
||||
tenant_id text not null,
|
||||
connection_id text not null,
|
||||
provider_id text not null,
|
||||
allowed_data_product_ids jsonb not null,
|
||||
expires_at timestamptz not null,
|
||||
active boolean not null default true,
|
||||
created_at timestamptz not null default now(),
|
||||
rotated_at timestamptz,
|
||||
revoked_at timestamptz,
|
||||
check (
|
||||
case when jsonb_typeof(allowed_data_product_ids) = 'array'
|
||||
then jsonb_array_length(allowed_data_product_ids) > 0
|
||||
else false
|
||||
end
|
||||
)
|
||||
)
|
||||
`);
|
||||
await pool.query("create index if not exists external_data_plane_writer_bindings_active_idx on external_data_plane_writer_bindings (active, expires_at)");
|
||||
|
||||
await pool.query(`
|
||||
create table if not exists external_data_plane_reader_bindings (
|
||||
id uuid primary key,
|
||||
token_hash text not null unique,
|
||||
tenant_id text not null,
|
||||
connection_id text not null,
|
||||
provider_id text not null,
|
||||
allowed_data_product_ids jsonb not null,
|
||||
expires_at timestamptz not null,
|
||||
active boolean not null default true,
|
||||
created_at timestamptz not null default now(),
|
||||
rotated_at timestamptz,
|
||||
revoked_at timestamptz,
|
||||
check (
|
||||
case when jsonb_typeof(allowed_data_product_ids) = 'array'
|
||||
then jsonb_array_length(allowed_data_product_ids) > 0
|
||||
else false
|
||||
end
|
||||
)
|
||||
)
|
||||
`);
|
||||
await pool.query("create index if not exists external_data_plane_reader_bindings_active_idx on external_data_plane_reader_bindings (active, expires_at)");
|
||||
|
||||
await pool.query(`
|
||||
create table if not exists external_data_plane_facts (
|
||||
id uuid not null,
|
||||
batch_id uuid not null references external_data_plane_batches(id) on delete cascade,
|
||||
tenant_id text not null,
|
||||
connection_id text not null,
|
||||
provider_id text not null,
|
||||
data_product_id text not null,
|
||||
source_id text not null,
|
||||
semantic_type text not null,
|
||||
observed_at timestamptz not null,
|
||||
received_at timestamptz not null,
|
||||
attributes jsonb not null default '{}'::jsonb,
|
||||
geometry geography(Point, 4326),
|
||||
fingerprint text not null,
|
||||
primary key (id, observed_at),
|
||||
unique (tenant_id, connection_id, provider_id, data_product_id, source_id, semantic_type, observed_at, fingerprint)
|
||||
)
|
||||
`);
|
||||
await pool.query("select create_hypertable('external_data_plane_facts', 'observed_at', if_not_exists => true, migrate_data => true)");
|
||||
await pool.query("create index if not exists external_data_plane_facts_source_time_idx on external_data_plane_facts (tenant_id, connection_id, data_product_id, source_id, observed_at desc)");
|
||||
await pool.query("create index if not exists external_data_plane_facts_geometry_idx on external_data_plane_facts using gist (geometry)");
|
||||
|
||||
await pool.query(`
|
||||
create table if not exists external_data_plane_current (
|
||||
tenant_id text not null,
|
||||
connection_id text not null,
|
||||
provider_id text not null,
|
||||
data_product_id text not null,
|
||||
source_id text not null,
|
||||
semantic_type text not null,
|
||||
observed_at timestamptz not null,
|
||||
received_at timestamptz not null,
|
||||
attributes jsonb not null default '{}'::jsonb,
|
||||
geometry geography(Point, 4326),
|
||||
fingerprint text not null,
|
||||
updated_at timestamptz not null default now(),
|
||||
primary key (tenant_id, connection_id, provider_id, data_product_id, source_id, semantic_type)
|
||||
)
|
||||
`);
|
||||
await pool.query("create index if not exists external_data_plane_current_scope_idx on external_data_plane_current (tenant_id, connection_id, data_product_id, observed_at desc)");
|
||||
await pool.query("create index if not exists external_data_plane_current_geometry_idx on external_data_plane_current using gist (geometry)");
|
||||
|
||||
await pool.query(`
|
||||
create table if not exists external_data_plane_delivery_state (
|
||||
tenant_id text not null,
|
||||
connection_id text not null,
|
||||
provider_id text not null,
|
||||
data_product_id text not null,
|
||||
current_cursor bigint not null default 0,
|
||||
updated_at timestamptz not null default now(),
|
||||
primary key (tenant_id, connection_id, provider_id, data_product_id)
|
||||
)
|
||||
`);
|
||||
|
||||
await pool.query(`
|
||||
create table if not exists external_data_plane_patch_outbox (
|
||||
tenant_id text not null,
|
||||
connection_id text not null,
|
||||
provider_id text not null,
|
||||
data_product_id text not null,
|
||||
cursor bigint not null,
|
||||
previous_cursor bigint not null,
|
||||
batch_id uuid not null references external_data_plane_batches(id) on delete cascade,
|
||||
operations jsonb not null,
|
||||
emitted_at timestamptz not null default now(),
|
||||
primary key (tenant_id, connection_id, provider_id, data_product_id, cursor),
|
||||
check (jsonb_typeof(operations) = 'array' and jsonb_array_length(operations) > 0)
|
||||
)
|
||||
`);
|
||||
await pool.query("create index if not exists external_data_plane_patch_outbox_retention_idx on external_data_plane_patch_outbox (emitted_at)");
|
||||
|
||||
await pool.query(`
|
||||
create table if not exists external_data_plane_history (
|
||||
tenant_id text not null,
|
||||
connection_id text not null,
|
||||
provider_id text not null,
|
||||
data_product_id text not null,
|
||||
source_id text not null,
|
||||
semantic_type text not null,
|
||||
bucket_start timestamptz not null,
|
||||
observed_at timestamptz not null,
|
||||
received_at timestamptz not null,
|
||||
attributes jsonb not null default '{}'::jsonb,
|
||||
geometry geography(Point, 4326),
|
||||
fingerprint text not null,
|
||||
updated_at timestamptz not null default now(),
|
||||
primary key (
|
||||
tenant_id, connection_id, provider_id, data_product_id,
|
||||
source_id, semantic_type, bucket_start
|
||||
)
|
||||
)
|
||||
`);
|
||||
await pool.query("select create_hypertable('external_data_plane_history', 'bucket_start', if_not_exists => true, migrate_data => true)");
|
||||
await pool.query("create index if not exists external_data_plane_history_source_time_idx on external_data_plane_history (tenant_id, connection_id, data_product_id, source_id, bucket_start desc)");
|
||||
await pool.query("create index if not exists external_data_plane_history_geometry_idx on external_data_plane_history using gist (geometry)");
|
||||
}
|
||||
|
|
@ -1,822 +0,0 @@
|
|||
import express from "express";
|
||||
import { createHash, randomUUID, timingSafeEqual } from "node:crypto";
|
||||
import { createServer } from "node:http";
|
||||
import { Pool } from "pg";
|
||||
import { validateDataProductPublish, validateIntakeBatch } from "@nodedc/external-provider-contract";
|
||||
import { readConfig } from "./config.mjs";
|
||||
import {
|
||||
loadDataProductDefinition,
|
||||
assertPublishMatchesDefinition,
|
||||
persistDataProductDefinition,
|
||||
persistDataProductPublish,
|
||||
pruneBatchReceipts,
|
||||
pruneDataProductHistory,
|
||||
prunePatchOutbox,
|
||||
readDataProductSnapshot,
|
||||
readPatchEvents,
|
||||
} from "./data-product-delivery.mjs";
|
||||
import { normalizeDataProductDefinition, safeDataProductDefinition } from "./data-product-policy.mjs";
|
||||
import { reconcileDataProductDefinitions } from "./definitions.mjs";
|
||||
import { assertBatchTimeBounds, rawRetentionExpiry } from "./intake-policy.mjs";
|
||||
import {
|
||||
assertReaderProduct,
|
||||
createReaderToken,
|
||||
hashReaderToken,
|
||||
normalizeReaderBindingRequest,
|
||||
safeReaderBinding,
|
||||
} from "./reader-binding.mjs";
|
||||
import { migrate } from "./schema.mjs";
|
||||
import {
|
||||
createWriterToken,
|
||||
hashWriterToken,
|
||||
materializeDataProductPublish,
|
||||
materializeWriterBoundBatch,
|
||||
normalizeWriterBindingRequest,
|
||||
safeWriterBinding,
|
||||
} from "./writer-binding.mjs";
|
||||
|
||||
const config = readConfig();
|
||||
const pool = new Pool({ connectionString: config.databaseUrl, max: config.databasePoolSize });
|
||||
const app = express();
|
||||
const httpServer = createServer(app);
|
||||
let retentionSweepTimer = null;
|
||||
let lastRetentionSweepAt = null;
|
||||
const activeReaderStreams = new Map();
|
||||
const activeStreamResponses = new Set();
|
||||
let shuttingDown = false;
|
||||
let shutdownPromise = null;
|
||||
|
||||
app.disable("x-powered-by");
|
||||
app.use(express.json({ limit: config.maxBatchBytes }));
|
||||
|
||||
app.get("/healthz", asyncRoute(async (_req, res) => {
|
||||
await pool.query("select 1");
|
||||
res.json({
|
||||
ok: true,
|
||||
service: "nodedc-external-data-plane",
|
||||
database: "ready",
|
||||
internalApiConfigured: Boolean(config.internalAccessToken),
|
||||
providerLogic: "absent",
|
||||
commandTransport: "absent",
|
||||
writerBindings: "supported",
|
||||
readerBindings: "supported",
|
||||
dataProductDelivery: "snapshot+durable-patch",
|
||||
legacyIntake: config.legacyIntakeEnabled ? "migration-only" : "disabled",
|
||||
writerBindingProvisioning: config.provisionerApiEnabled ? "enabled" : "disabled",
|
||||
rawRetentionSweep: {
|
||||
mode: "server-scheduled",
|
||||
lastSweepAt: lastRetentionSweepAt,
|
||||
},
|
||||
});
|
||||
}));
|
||||
|
||||
app.put("/internal/data-plane/v1/data-products/:dataProductId", requireProvisionerApi, asyncRoute(async (req, res) => {
|
||||
const definition = normalizeDataProductDefinition({ ...req.body, id: req.params.dataProductId });
|
||||
const saved = await persistDataProductDefinition(pool, definition);
|
||||
res.json({ ok: true, dataProduct: safeDataProductDefinition(saved) });
|
||||
}));
|
||||
|
||||
app.post("/internal/data-plane/v1/data-products/:dataProductId/publish", requireWriterBinding, asyncRoute(async (req, res) => {
|
||||
if (hasScopeHeaders(req)) throw httpError(400, "data_product_publish_scope_headers_forbidden");
|
||||
const validation = validateDataProductPublish(req.body, {
|
||||
maxFacts: config.maxFactsPerPublish,
|
||||
maxAttributesBytes: config.maxAttributesBytesPerFact,
|
||||
});
|
||||
if (!validation.ok) throw httpError(422, validation.errors[0] || "invalid_data_product_publish");
|
||||
const dataProductId = requireIdentifier(req.params.dataProductId, "data_product_id_invalid");
|
||||
const definition = await loadDataProductDefinition(pool, dataProductId);
|
||||
if (!definition) throw httpError(404, "data_product_not_found");
|
||||
const batch = materializeDataProductPublish(req.body, req.writerBinding, definition, dataProductId);
|
||||
const canonicalValidation = validateIntakeBatch(batch);
|
||||
if (!canonicalValidation.ok) throw httpError(422, "materialized_publish_invalid");
|
||||
assertBatchTimeBounds(batch, { maxFutureSkewSeconds: config.maxFutureSkewSeconds });
|
||||
const receipt = await persistDataProductPublish(pool, batch, definition, {
|
||||
maxPatchOperations: config.maxPatchOperations,
|
||||
maxPatchBytes: config.maxPatchBytes,
|
||||
});
|
||||
res.status(receipt.idempotent ? 200 : 201).json({ ok: true, ...receipt });
|
||||
}));
|
||||
|
||||
app.get("/internal/data-plane/v1/writer/data-products", requireWriterBinding, asyncRoute(async (req, res) => {
|
||||
const dataProducts = await listGrantedProducts(req.writerBinding);
|
||||
res.json({ ok: true, dataProducts });
|
||||
}));
|
||||
|
||||
app.get("/internal/data-plane/v1/reader/data-products", requireReaderBinding, asyncRoute(async (req, res) => {
|
||||
const dataProducts = await listGrantedProducts(req.readerBinding);
|
||||
res.json({ ok: true, dataProducts });
|
||||
}));
|
||||
|
||||
app.post("/internal/data-plane/v1/intake", requireLegacyIntake, requireInternalApi, asyncRoute(async (req, res) => {
|
||||
const validation = validateIntakeBatch(req.body);
|
||||
if (!validation.ok) throw httpError(422, "invalid_intake_batch");
|
||||
assertBatchTimeBounds(req.body, { maxFutureSkewSeconds: config.maxFutureSkewSeconds });
|
||||
await assertLegacyBatchProduct(req.body);
|
||||
const scope = requireScope(req);
|
||||
if (scope.tenantId !== req.body.source.tenantId || scope.connectionId !== req.body.source.connectionId) {
|
||||
throw httpError(403, "scope_mismatch");
|
||||
}
|
||||
|
||||
const result = await persistBatch(req.body);
|
||||
res.status(result.idempotent ? 200 : 201).json({ ok: true, ...result });
|
||||
}));
|
||||
|
||||
app.post("/internal/data-plane/v1/intake/writer-bound", requireLegacyIntake, requireWriterBinding, asyncRoute(async (req, res) => {
|
||||
const batch = materializeWriterBoundBatch(req.body, req.writerBinding, {
|
||||
hasScopeHeaders: hasScopeHeaders(req),
|
||||
});
|
||||
const validation = validateIntakeBatch(batch);
|
||||
if (!validation.ok) throw httpError(422, "invalid_intake_batch");
|
||||
assertBatchTimeBounds(batch, { maxFutureSkewSeconds: config.maxFutureSkewSeconds });
|
||||
await assertLegacyBatchProduct(batch);
|
||||
|
||||
const result = await persistBatch(batch);
|
||||
res.status(result.idempotent ? 200 : 201).json({ ok: true, ...result });
|
||||
}));
|
||||
|
||||
app.post("/internal/data-plane/v1/writer-bindings", requireProvisionerApi, asyncRoute(async (req, res) => {
|
||||
const policy = normalizeWriterBindingRequest(req.body, {
|
||||
maxTtlDays: config.writerBindingMaxTtlDays,
|
||||
});
|
||||
await assertRegisteredProductIds(policy.allowedDataProductIds);
|
||||
const token = createWriterToken();
|
||||
const bindingId = randomUUID();
|
||||
const result = await pool.query(
|
||||
`insert into external_data_plane_writer_bindings (
|
||||
id, token_hash, tenant_id, connection_id, provider_id,
|
||||
allowed_data_product_ids, expires_at
|
||||
) values ($1, $2, $3, $4, $5, $6::jsonb, $7)
|
||||
returning id, tenant_id as "tenantId", connection_id as "connectionId",
|
||||
provider_id as "providerId", allowed_data_product_ids as "allowedDataProductIds",
|
||||
expires_at as "expiresAt", active, created_at as "createdAt",
|
||||
rotated_at as "rotatedAt", revoked_at as "revokedAt"`,
|
||||
[
|
||||
bindingId,
|
||||
hashWriterToken(token),
|
||||
policy.tenantId,
|
||||
policy.connectionId,
|
||||
policy.providerId,
|
||||
JSON.stringify(policy.allowedDataProductIds),
|
||||
policy.expiresAt,
|
||||
],
|
||||
);
|
||||
// The token is intentionally returned exactly once to a trusted provisioner.
|
||||
// It must be placed directly into an opaque Engine credential reference and
|
||||
// must never be logged, stored in L2 graph data or shown to a consumer.
|
||||
sendOneTimeCapability(res, 201, { ok: true, writerBinding: safeWriterBinding(result.rows[0]), token });
|
||||
}));
|
||||
|
||||
app.post("/internal/data-plane/v1/writer-bindings/:bindingId/rotate", requireProvisionerApi, asyncRoute(async (req, res) => {
|
||||
const bindingId = requireUuid(req.params.bindingId, "writer_binding_id_invalid");
|
||||
const token = createWriterToken();
|
||||
const result = await pool.query(
|
||||
`update external_data_plane_writer_bindings
|
||||
set token_hash = $2, rotated_at = now()
|
||||
where id = $1 and active = true and expires_at > now()
|
||||
returning id, tenant_id as "tenantId", connection_id as "connectionId",
|
||||
provider_id as "providerId", allowed_data_product_ids as "allowedDataProductIds",
|
||||
expires_at as "expiresAt", active, created_at as "createdAt",
|
||||
rotated_at as "rotatedAt", revoked_at as "revokedAt"`,
|
||||
[bindingId, hashWriterToken(token)],
|
||||
);
|
||||
if (!result.rowCount) throw httpError(404, "writer_binding_not_found_or_inactive");
|
||||
sendOneTimeCapability(res, 200, { ok: true, writerBinding: safeWriterBinding(result.rows[0]), token });
|
||||
}));
|
||||
|
||||
app.post("/internal/data-plane/v1/writer-bindings/:bindingId/revoke", requireProvisionerApi, asyncRoute(async (req, res) => {
|
||||
const bindingId = requireUuid(req.params.bindingId, "writer_binding_id_invalid");
|
||||
const result = await pool.query(
|
||||
`update external_data_plane_writer_bindings
|
||||
set active = false, revoked_at = now()
|
||||
where id = $1 and active = true
|
||||
returning id, tenant_id as "tenantId", connection_id as "connectionId",
|
||||
provider_id as "providerId", allowed_data_product_ids as "allowedDataProductIds",
|
||||
expires_at as "expiresAt", active, created_at as "createdAt",
|
||||
rotated_at as "rotatedAt", revoked_at as "revokedAt"`,
|
||||
[bindingId],
|
||||
);
|
||||
if (!result.rowCount) throw httpError(404, "writer_binding_not_found_or_inactive");
|
||||
res.json({ ok: true, writerBinding: safeWriterBinding(result.rows[0]) });
|
||||
}));
|
||||
|
||||
app.post("/internal/data-plane/v1/reader-bindings", requireProvisionerApi, asyncRoute(async (req, res) => {
|
||||
const policy = normalizeReaderBindingRequest(req.body, {
|
||||
maxTtlDays: config.writerBindingMaxTtlDays,
|
||||
});
|
||||
await assertRegisteredProductIds(policy.allowedDataProductIds);
|
||||
const token = createReaderToken();
|
||||
const bindingId = randomUUID();
|
||||
const result = await pool.query(
|
||||
`insert into external_data_plane_reader_bindings (
|
||||
id, token_hash, tenant_id, connection_id, provider_id,
|
||||
allowed_data_product_ids, expires_at
|
||||
) values ($1, $2, $3, $4, $5, $6::jsonb, $7)
|
||||
returning id, tenant_id as "tenantId", connection_id as "connectionId",
|
||||
provider_id as "providerId", allowed_data_product_ids as "allowedDataProductIds",
|
||||
expires_at as "expiresAt", active, created_at as "createdAt",
|
||||
rotated_at as "rotatedAt", revoked_at as "revokedAt"`,
|
||||
[
|
||||
bindingId,
|
||||
hashReaderToken(token),
|
||||
policy.tenantId,
|
||||
policy.connectionId,
|
||||
policy.providerId,
|
||||
JSON.stringify(policy.allowedDataProductIds),
|
||||
policy.expiresAt,
|
||||
],
|
||||
);
|
||||
sendOneTimeCapability(res, 201, { ok: true, readerBinding: safeReaderBinding(result.rows[0]), token });
|
||||
}));
|
||||
|
||||
app.post("/internal/data-plane/v1/reader-bindings/:bindingId/rotate", requireProvisionerApi, asyncRoute(async (req, res) => {
|
||||
const bindingId = requireUuid(req.params.bindingId, "reader_binding_id_invalid");
|
||||
const token = createReaderToken();
|
||||
const result = await pool.query(
|
||||
`update external_data_plane_reader_bindings
|
||||
set token_hash = $2, rotated_at = now()
|
||||
where id = $1 and active = true and expires_at > now()
|
||||
returning id, tenant_id as "tenantId", connection_id as "connectionId",
|
||||
provider_id as "providerId", allowed_data_product_ids as "allowedDataProductIds",
|
||||
expires_at as "expiresAt", active, created_at as "createdAt",
|
||||
rotated_at as "rotatedAt", revoked_at as "revokedAt"`,
|
||||
[bindingId, hashReaderToken(token)],
|
||||
);
|
||||
if (!result.rowCount) throw httpError(404, "reader_binding_not_found_or_inactive");
|
||||
sendOneTimeCapability(res, 200, { ok: true, readerBinding: safeReaderBinding(result.rows[0]), token });
|
||||
}));
|
||||
|
||||
app.post("/internal/data-plane/v1/reader-bindings/:bindingId/revoke", requireProvisionerApi, asyncRoute(async (req, res) => {
|
||||
const bindingId = requireUuid(req.params.bindingId, "reader_binding_id_invalid");
|
||||
const result = await pool.query(
|
||||
`update external_data_plane_reader_bindings
|
||||
set active = false, revoked_at = now()
|
||||
where id = $1 and active = true
|
||||
returning id, tenant_id as "tenantId", connection_id as "connectionId",
|
||||
provider_id as "providerId", allowed_data_product_ids as "allowedDataProductIds",
|
||||
expires_at as "expiresAt", active, created_at as "createdAt",
|
||||
rotated_at as "rotatedAt", revoked_at as "revokedAt"`,
|
||||
[bindingId],
|
||||
);
|
||||
if (!result.rowCount) throw httpError(404, "reader_binding_not_found_or_inactive");
|
||||
res.json({ ok: true, readerBinding: safeReaderBinding(result.rows[0]) });
|
||||
}));
|
||||
|
||||
app.get("/internal/data-plane/v1/data-products/:dataProductId/snapshot", requireReaderBinding, asyncRoute(async (req, res) => {
|
||||
if (hasScopeHeaders(req)) throw httpError(400, "data_product_read_scope_headers_forbidden");
|
||||
const dataProductId = assertReaderProduct(req.readerBinding, req.params.dataProductId);
|
||||
const definition = await loadDataProductDefinition(pool, dataProductId);
|
||||
if (!definition) throw httpError(404, "data_product_not_found");
|
||||
const limit = boundedLimit(req.query.limit, 5000, 1, 5000);
|
||||
const snapshot = await readDataProductSnapshot(pool, req.readerBinding, definition, { limit });
|
||||
res.json(snapshot);
|
||||
}));
|
||||
|
||||
app.get("/internal/data-plane/v1/data-products/:dataProductId/stream", requireReaderBinding, asyncRoute(async (req, res) => {
|
||||
if (shuttingDown) throw httpError(503, "service_shutting_down");
|
||||
if (hasScopeHeaders(req)) throw httpError(400, "data_product_read_scope_headers_forbidden");
|
||||
const dataProductId = assertReaderProduct(req.readerBinding, req.params.dataProductId);
|
||||
const definition = await loadDataProductDefinition(pool, dataProductId);
|
||||
if (!definition) throw httpError(404, "data_product_not_found");
|
||||
if (definition.deliveryMode !== "snapshot+patch") throw httpError(409, "data_product_stream_not_supported");
|
||||
const streamCount = activeReaderStreams.get(req.readerBinding.id) || 0;
|
||||
if (streamCount >= config.maxReaderStreams) throw httpError(429, "reader_stream_limit_exceeded");
|
||||
activeReaderStreams.set(req.readerBinding.id, streamCount + 1);
|
||||
activeStreamResponses.add(res);
|
||||
|
||||
let poll = null;
|
||||
let heartbeat = null;
|
||||
let polling = false;
|
||||
let heartbeatWriting = false;
|
||||
let cleanedUp = false;
|
||||
const cleanup = () => {
|
||||
if (cleanedUp) return;
|
||||
cleanedUp = true;
|
||||
if (poll) clearInterval(poll);
|
||||
if (heartbeat) clearInterval(heartbeat);
|
||||
activeStreamResponses.delete(res);
|
||||
const remaining = Math.max(0, (activeReaderStreams.get(req.readerBinding.id) || 1) - 1);
|
||||
if (remaining) activeReaderStreams.set(req.readerBinding.id, remaining);
|
||||
else activeReaderStreams.delete(req.readerBinding.id);
|
||||
};
|
||||
req.once("close", cleanup);
|
||||
res.once("close", cleanup);
|
||||
|
||||
try {
|
||||
let cursor = parseCursor(req.get("last-event-id") || req.query.after || "0");
|
||||
const initialEvents = await readPatchEvents(pool, req.readerBinding, definition, cursor);
|
||||
if (res.destroyed || cleanedUp) return;
|
||||
res.status(200);
|
||||
res.set({
|
||||
"Cache-Control": "no-cache, no-transform",
|
||||
"Content-Type": "text/event-stream",
|
||||
Connection: "keep-alive",
|
||||
"X-Accel-Buffering": "no",
|
||||
});
|
||||
res.flushHeaders();
|
||||
await writeSseFrame(res, ": nodedc-data-product-stream\n\n");
|
||||
await writeSseFrame(res, `event: nodedc.data-product.ready.v1\ndata: ${JSON.stringify({
|
||||
schemaVersion: "nodedc.data-product.ready/v1",
|
||||
dataProductId,
|
||||
cursor: cursor.toString(),
|
||||
emittedAt: new Date().toISOString(),
|
||||
})}\n\n`);
|
||||
for (const event of initialEvents) {
|
||||
await writePatchEvent(res, event);
|
||||
cursor = BigInt(event.cursor);
|
||||
}
|
||||
|
||||
const pump = async () => {
|
||||
if (polling || res.writableEnded || res.destroyed || shuttingDown) return;
|
||||
polling = true;
|
||||
try {
|
||||
await assertReaderStreamAccess(req.readerBinding, dataProductId);
|
||||
const events = await readPatchEvents(pool, req.readerBinding, definition, cursor);
|
||||
for (const event of events) {
|
||||
await writePatchEvent(res, event);
|
||||
cursor = BigInt(event.cursor);
|
||||
}
|
||||
} catch (error) {
|
||||
const code = safeErrorCode(error);
|
||||
if (!res.writableEnded && !res.destroyed) {
|
||||
await writeSseFrame(res, `event: error\ndata: ${JSON.stringify({ ok: false, error: code })}\n\n`).catch(() => {});
|
||||
res.end();
|
||||
}
|
||||
} finally {
|
||||
polling = false;
|
||||
}
|
||||
};
|
||||
poll = setInterval(() => { void pump(); }, config.streamPollMs);
|
||||
poll.unref();
|
||||
heartbeat = setInterval(() => {
|
||||
if (polling || heartbeatWriting || res.writableEnded || res.destroyed) return;
|
||||
heartbeatWriting = true;
|
||||
void writeSseFrame(res, `: heartbeat ${Date.now()}\n\n`)
|
||||
.catch(() => { if (!res.writableEnded) res.end(); })
|
||||
.finally(() => { heartbeatWriting = false; });
|
||||
}, config.streamHeartbeatMs);
|
||||
heartbeat.unref();
|
||||
} catch (error) {
|
||||
cleanup();
|
||||
if (res.headersSent) {
|
||||
if (!res.writableEnded) res.end();
|
||||
return;
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
}));
|
||||
|
||||
app.get("/internal/data-plane/v1/data-products/:dataProductId/current", requireLegacyIntake, requireInternalApi, asyncRoute(async (req, res) => {
|
||||
const scope = requireScope(req);
|
||||
const dataProductId = String(req.params.dataProductId || "");
|
||||
if (!isIdentifier(dataProductId)) throw httpError(400, "data_product_id_invalid");
|
||||
const limit = boundedLimit(req.query.limit, 200, 1, 1000);
|
||||
const rows = await pool.query(
|
||||
`select
|
||||
provider_id as "providerId", data_product_id as "dataProductId",
|
||||
source_id as "sourceId", semantic_type as "semanticType",
|
||||
observed_at as "observedAt", received_at as "receivedAt", attributes,
|
||||
case when geometry is null then null
|
||||
else jsonb_build_object('type', 'Point', 'coordinates', jsonb_build_array(
|
||||
ST_X(geometry::geometry), ST_Y(geometry::geometry)
|
||||
)) end as geometry
|
||||
from external_data_plane_current
|
||||
where tenant_id = $1 and connection_id = $2 and data_product_id = $3
|
||||
order by observed_at desc, source_id asc
|
||||
limit $4`,
|
||||
[scope.tenantId, scope.connectionId, dataProductId, limit],
|
||||
);
|
||||
res.json({
|
||||
ok: true,
|
||||
dataProductId,
|
||||
tenantId: scope.tenantId,
|
||||
connectionId: scope.connectionId,
|
||||
facts: rows.rows,
|
||||
});
|
||||
}));
|
||||
|
||||
app.get("/internal/data-plane/v1/status", requireLegacyIntake, requireInternalApi, asyncRoute(async (req, res) => {
|
||||
const scope = requireScope(req);
|
||||
const result = await pool.query(
|
||||
`select count(*)::integer as "currentFactCount", max(received_at) as "lastReceivedAt"
|
||||
from external_data_plane_current where tenant_id = $1 and connection_id = $2`,
|
||||
[scope.tenantId, scope.connectionId],
|
||||
);
|
||||
res.json({ ok: true, ...scope, ...result.rows[0], providerLogic: "absent", commandTransport: "absent" });
|
||||
}));
|
||||
|
||||
app.use((error, _req, res, _next) => {
|
||||
const status = Number(error?.status || 500);
|
||||
const publicStatus = status >= 400 && status < 600 ? status : 500;
|
||||
console.error(JSON.stringify({ event: "external_data_plane_error", error: safeErrorCode(error), status: publicStatus }));
|
||||
res.status(publicStatus).json({ ok: false, error: publicStatus >= 500 ? "internal_error" : safeErrorCode(error) });
|
||||
});
|
||||
|
||||
await migrate(pool);
|
||||
const bundledDataProducts = await reconcileDataProductDefinitions(pool);
|
||||
retentionSweepTimer = setInterval(() => {
|
||||
void sweepRetention().catch((error) => {
|
||||
console.error(JSON.stringify({ event: "external_data_plane_retention_sweep_failed", error: safeErrorCode(error) }));
|
||||
});
|
||||
}, config.retentionSweepMs);
|
||||
retentionSweepTimer.unref();
|
||||
|
||||
httpServer.listen(config.port, "0.0.0.0", () => {
|
||||
console.log(`NODE.DC External Data Plane listening on http://0.0.0.0:${config.port}`);
|
||||
console.log(JSON.stringify({ event: "external_data_plane_definitions_ready", count: bundledDataProducts.length }));
|
||||
setImmediate(() => {
|
||||
void sweepRetention().catch((error) => {
|
||||
console.error(JSON.stringify({ event: "external_data_plane_retention_sweep_failed", error: safeErrorCode(error) }));
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
process.on("SIGTERM", shutdown);
|
||||
process.on("SIGINT", shutdown);
|
||||
|
||||
async function persistBatch(batch) {
|
||||
const client = await pool.connect();
|
||||
try {
|
||||
await client.query("begin");
|
||||
const existing = await client.query(
|
||||
`select id, fact_count as "factCount", inserted_fact_count as "insertedFactCount"
|
||||
from external_data_plane_batches
|
||||
where tenant_id = $1 and connection_id = $2 and provider_id = $3
|
||||
and data_product_id = $4 and idempotency_key = $5
|
||||
for update`,
|
||||
[
|
||||
batch.source.tenantId, batch.source.connectionId, batch.source.providerId,
|
||||
batch.contract.dataProductId, batch.batch.idempotencyKey,
|
||||
],
|
||||
);
|
||||
if (existing.rowCount) {
|
||||
await client.query("commit");
|
||||
return { batchId: existing.rows[0].id, idempotent: true, ...existing.rows[0] };
|
||||
}
|
||||
|
||||
const batchId = randomUUID();
|
||||
await client.query(
|
||||
`insert into external_data_plane_batches (
|
||||
id, tenant_id, connection_id, provider_id, data_product_id, contract_version,
|
||||
ontology_revision, run_id, sequence, idempotency_key, received_at, fact_count
|
||||
) values ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12)`,
|
||||
[
|
||||
batchId, batch.source.tenantId, batch.source.connectionId, batch.source.providerId,
|
||||
batch.contract.dataProductId, batch.contract.version, batch.contract.ontologyRevision,
|
||||
batch.batch.runId, batch.batch.sequence, batch.batch.idempotencyKey,
|
||||
batch.batch.receivedAt, batch.facts.length,
|
||||
],
|
||||
);
|
||||
|
||||
if (batch.raw) await persistRawEnvelope(client, batchId, batch);
|
||||
|
||||
const records = batch.facts.map((fact) => ({
|
||||
id: randomUUID(),
|
||||
source_id: fact.sourceId,
|
||||
semantic_type: fact.semanticType,
|
||||
observed_at: fact.observedAt,
|
||||
attributes: fact.attributes || {},
|
||||
geometry: fact.geometry || null,
|
||||
fingerprint: hash(fact),
|
||||
}));
|
||||
const inserted = await client.query(
|
||||
`insert into external_data_plane_facts (
|
||||
id, batch_id, tenant_id, connection_id, provider_id, data_product_id,
|
||||
source_id, semantic_type, observed_at, received_at, attributes, geometry, fingerprint
|
||||
)
|
||||
select item.id::uuid, $1, $2, $3, $4, $5,
|
||||
item.source_id, item.semantic_type, item.observed_at, $6,
|
||||
coalesce(item.attributes, '{}'::jsonb),
|
||||
case when item.geometry is null then null
|
||||
else ST_SetSRID(ST_MakePoint(
|
||||
(item.geometry->'coordinates'->>0)::double precision,
|
||||
(item.geometry->'coordinates'->>1)::double precision
|
||||
), 4326)::geography end,
|
||||
item.fingerprint
|
||||
from jsonb_to_recordset($7::jsonb) as item(
|
||||
id text, source_id text, semantic_type text, observed_at timestamptz,
|
||||
attributes jsonb, geometry jsonb, fingerprint text
|
||||
)
|
||||
on conflict do nothing
|
||||
returning id`,
|
||||
[
|
||||
batchId, batch.source.tenantId, batch.source.connectionId, batch.source.providerId,
|
||||
batch.contract.dataProductId, batch.batch.receivedAt, JSON.stringify(records),
|
||||
],
|
||||
);
|
||||
await client.query(
|
||||
`insert into external_data_plane_current (
|
||||
tenant_id, connection_id, provider_id, data_product_id, source_id, semantic_type,
|
||||
observed_at, received_at, attributes, geometry, fingerprint, updated_at
|
||||
)
|
||||
select tenant_id, connection_id, provider_id, data_product_id, source_id, semantic_type,
|
||||
observed_at, received_at, attributes, geometry, fingerprint, now()
|
||||
from external_data_plane_facts where batch_id = $1
|
||||
on conflict (tenant_id, connection_id, provider_id, data_product_id, source_id, semantic_type)
|
||||
do update set
|
||||
observed_at = excluded.observed_at,
|
||||
received_at = excluded.received_at,
|
||||
attributes = excluded.attributes,
|
||||
geometry = excluded.geometry,
|
||||
fingerprint = excluded.fingerprint,
|
||||
updated_at = now()
|
||||
where excluded.observed_at >= external_data_plane_current.observed_at`,
|
||||
[batchId],
|
||||
);
|
||||
await client.query(
|
||||
"update external_data_plane_batches set inserted_fact_count = $2 where id = $1",
|
||||
[batchId, inserted.rowCount],
|
||||
);
|
||||
await client.query("commit");
|
||||
return { batchId, idempotent: false, factCount: batch.facts.length, insertedFactCount: inserted.rowCount };
|
||||
} catch (error) {
|
||||
await client.query("rollback");
|
||||
throw error;
|
||||
} finally {
|
||||
client.release();
|
||||
}
|
||||
}
|
||||
|
||||
async function persistRawEnvelope(client, batchId, batch) {
|
||||
const serialized = batch.raw.payload === undefined ? null : JSON.stringify(batch.raw.payload);
|
||||
const expiresAt = rawRetentionExpiry({ rawRetentionDays: config.rawRetentionDays });
|
||||
await client.query(
|
||||
`insert into external_data_plane_raw_envelopes (
|
||||
id, batch_id, payload_hash, content_type, payload, payload_ref, payload_bytes, received_at, expires_at
|
||||
) values ($1, $2, $3, $4, $5::jsonb, $6, $7, $8, $9)`,
|
||||
[
|
||||
randomUUID(), batchId, batch.raw.hash || hash(batch.raw.payload), batch.raw.contentType, serialized, batch.raw.ref || null,
|
||||
serialized ? Buffer.byteLength(serialized) : null, batch.batch.receivedAt, expiresAt,
|
||||
],
|
||||
);
|
||||
}
|
||||
|
||||
function requireInternalApi(req, _res, next) {
|
||||
if (!config.internalAccessToken) return next(httpError(503, "internal_api_not_configured"));
|
||||
const value = bearerToken(req);
|
||||
if (!value || !safeEqual(value, config.internalAccessToken)) return next(httpError(401, "unauthorized"));
|
||||
return next();
|
||||
}
|
||||
|
||||
function requireLegacyIntake(_req, _res, next) {
|
||||
if (!config.legacyIntakeEnabled) return next(httpError(410, "legacy_intake_disabled"));
|
||||
return next();
|
||||
}
|
||||
|
||||
function requireProvisionerApi(req, _res, next) {
|
||||
if (!config.provisionerApiEnabled) return next(httpError(503, "provisioner_api_disabled"));
|
||||
if (!config.provisionerAccessToken) return next(httpError(503, "provisioner_api_not_configured"));
|
||||
const value = bearerToken(req);
|
||||
if (!value || !safeEqual(value, config.provisionerAccessToken)) return next(httpError(401, "provisioner_unauthorized"));
|
||||
return next();
|
||||
}
|
||||
|
||||
function requireWriterBinding(req, _res, next) {
|
||||
return resolveWriterBinding(req)
|
||||
.then((binding) => {
|
||||
req.writerBinding = binding;
|
||||
next();
|
||||
})
|
||||
.catch(next);
|
||||
}
|
||||
|
||||
function requireReaderBinding(req, _res, next) {
|
||||
return resolveReaderBinding(req)
|
||||
.then((binding) => {
|
||||
req.readerBinding = binding;
|
||||
next();
|
||||
})
|
||||
.catch(next);
|
||||
}
|
||||
|
||||
async function resolveWriterBinding(req) {
|
||||
const token = bearerToken(req);
|
||||
if (!token) throw httpError(401, "writer_binding_unauthorized");
|
||||
const result = await pool.query(
|
||||
`select id, tenant_id as "tenantId", connection_id as "connectionId",
|
||||
provider_id as "providerId", allowed_data_product_ids as "allowedDataProductIds",
|
||||
expires_at as "expiresAt", active, created_at as "createdAt",
|
||||
rotated_at as "rotatedAt", revoked_at as "revokedAt"
|
||||
from external_data_plane_writer_bindings
|
||||
where token_hash = $1 and active = true and expires_at > now()`,
|
||||
[hashWriterToken(token)],
|
||||
);
|
||||
if (!result.rowCount) throw httpError(401, "writer_binding_unauthorized");
|
||||
return result.rows[0];
|
||||
}
|
||||
|
||||
async function resolveReaderBinding(req) {
|
||||
const token = bearerToken(req);
|
||||
if (!token) throw httpError(401, "reader_binding_unauthorized");
|
||||
const result = await pool.query(
|
||||
`select id, token_hash as "tokenHash", tenant_id as "tenantId", connection_id as "connectionId",
|
||||
provider_id as "providerId", allowed_data_product_ids as "allowedDataProductIds",
|
||||
expires_at as "expiresAt", active, created_at as "createdAt",
|
||||
rotated_at as "rotatedAt", revoked_at as "revokedAt"
|
||||
from external_data_plane_reader_bindings
|
||||
where token_hash = $1 and active = true and expires_at > now()`,
|
||||
[hashReaderToken(token)],
|
||||
);
|
||||
if (!result.rowCount) throw httpError(401, "reader_binding_unauthorized");
|
||||
return result.rows[0];
|
||||
}
|
||||
|
||||
async function assertReaderStreamAccess(binding, dataProductId) {
|
||||
const result = await pool.query(
|
||||
`select binding.id
|
||||
from external_data_plane_reader_bindings as binding
|
||||
join external_data_plane_products as product
|
||||
on product.id = $3 and product.active = true
|
||||
where binding.id = $1 and binding.token_hash = $2
|
||||
and binding.active = true and binding.expires_at > now()
|
||||
and binding.allowed_data_product_ids ? $3`,
|
||||
[binding.id, binding.tokenHash, dataProductId],
|
||||
);
|
||||
if (!result.rowCount) throw httpError(401, "reader_stream_access_revoked");
|
||||
}
|
||||
|
||||
async function assertLegacyBatchProduct(batch) {
|
||||
const definition = await loadDataProductDefinition(pool, batch.contract.dataProductId);
|
||||
if (!definition) throw httpError(422, "legacy_data_product_not_registered");
|
||||
if (definition.version !== batch.contract.version || definition.ontologyRevision !== batch.contract.ontologyRevision) {
|
||||
throw httpError(422, "legacy_data_product_contract_mismatch");
|
||||
}
|
||||
assertPublishMatchesDefinition(batch, definition);
|
||||
}
|
||||
|
||||
async function listGrantedProducts(binding) {
|
||||
const allowed = Array.isArray(binding.allowedDataProductIds) ? binding.allowedDataProductIds : [];
|
||||
if (!allowed.length) return [];
|
||||
const result = await pool.query(
|
||||
`select id, version, ontology_revision as "ontologyRevision",
|
||||
delivery_mode as "deliveryMode", semantic_types as "semanticTypes",
|
||||
fields, history_policy as "historyPolicy", active,
|
||||
created_at as "createdAt", updated_at as "updatedAt"
|
||||
from external_data_plane_products
|
||||
where active = true and id = any($1::text[])
|
||||
order by id asc`,
|
||||
[allowed],
|
||||
);
|
||||
return result.rows.map(safeDataProductDefinition);
|
||||
}
|
||||
|
||||
async function assertRegisteredProductIds(dataProductIds) {
|
||||
const result = await pool.query(
|
||||
"select id from external_data_plane_products where active = true and id = any($1::text[])",
|
||||
[dataProductIds],
|
||||
);
|
||||
const registered = new Set(result.rows.map((row) => row.id));
|
||||
if (dataProductIds.some((id) => !registered.has(id))) throw httpError(422, "binding_data_product_not_registered");
|
||||
}
|
||||
|
||||
function requireScope(req) {
|
||||
const tenantId = String(req.get("x-nodedc-tenant-id") || "");
|
||||
const connectionId = String(req.get("x-nodedc-connection-id") || "");
|
||||
if (!isIdentifier(tenantId) || !isIdentifier(connectionId)) throw httpError(400, "scope_headers_required");
|
||||
return { tenantId, connectionId };
|
||||
}
|
||||
|
||||
function hasScopeHeaders(req) {
|
||||
return Object.hasOwn(req.headers, "x-nodedc-tenant-id") || Object.hasOwn(req.headers, "x-nodedc-connection-id");
|
||||
}
|
||||
|
||||
function bearerToken(req) {
|
||||
return String(req.get("authorization") || "").replace(/^Bearer\s+/i, "");
|
||||
}
|
||||
|
||||
function requireUuid(value, code) {
|
||||
const normalized = String(value || "");
|
||||
if (!/^[0-9a-f]{8}-[0-9a-f]{4}-[1-5][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/i.test(normalized)) {
|
||||
throw httpError(400, code);
|
||||
}
|
||||
return normalized;
|
||||
}
|
||||
|
||||
function requireIdentifier(value, code) {
|
||||
const normalized = String(value || "");
|
||||
if (!isIdentifier(normalized)) throw httpError(400, code);
|
||||
return normalized;
|
||||
}
|
||||
|
||||
function parseCursor(value) {
|
||||
const normalized = String(value ?? "");
|
||||
if (!/^(?:0|[1-9]\d*)$/.test(normalized)) throw httpError(400, "data_product_cursor_invalid");
|
||||
const cursor = BigInt(normalized);
|
||||
if (cursor > 9_223_372_036_854_775_807n) throw httpError(400, "data_product_cursor_invalid");
|
||||
return cursor;
|
||||
}
|
||||
|
||||
function writePatchEvent(res, event) {
|
||||
return writeSseFrame(res, `id: ${event.cursor}\nevent: nodedc.data-product.patch.v1\ndata: ${JSON.stringify(event)}\n\n`);
|
||||
}
|
||||
|
||||
function writeSseFrame(res, frame) {
|
||||
if (res.writableEnded || res.destroyed) return Promise.reject(httpError(499, "reader_stream_closed"));
|
||||
if (res.write(frame)) return Promise.resolve();
|
||||
return new Promise((resolve, reject) => {
|
||||
const cleanup = () => {
|
||||
res.off("drain", onDrain);
|
||||
res.off("close", onClose);
|
||||
res.off("error", onError);
|
||||
};
|
||||
const onDrain = () => { cleanup(); resolve(); };
|
||||
const onClose = () => { cleanup(); reject(httpError(499, "reader_stream_closed")); };
|
||||
const onError = (error) => { cleanup(); reject(error); };
|
||||
res.once("drain", onDrain);
|
||||
res.once("close", onClose);
|
||||
res.once("error", onError);
|
||||
});
|
||||
}
|
||||
|
||||
function sendOneTimeCapability(res, status, payload) {
|
||||
// Create/rotate is the only boundary where a capability exists in
|
||||
// plaintext. A trusted provisioner must consume the response in memory and
|
||||
// place it directly into its opaque destination; intermediaries must never
|
||||
// cache or persist it.
|
||||
res.set({
|
||||
"Cache-Control": "no-store, max-age=0",
|
||||
Pragma: "no-cache",
|
||||
Expires: "0",
|
||||
});
|
||||
return res.status(status).json(payload);
|
||||
}
|
||||
|
||||
function safeEqual(left, right) {
|
||||
const leftBuffer = Buffer.from(left);
|
||||
const rightBuffer = Buffer.from(right);
|
||||
return leftBuffer.length === rightBuffer.length && timingSafeEqual(leftBuffer, rightBuffer);
|
||||
}
|
||||
|
||||
function boundedLimit(value, fallback, min, max) {
|
||||
const candidate = Number.parseInt(String(value ?? ""), 10);
|
||||
if (!Number.isInteger(candidate)) return fallback;
|
||||
return Math.max(min, Math.min(max, candidate));
|
||||
}
|
||||
|
||||
function hash(value) {
|
||||
return createHash("sha256").update(JSON.stringify(value)).digest("hex");
|
||||
}
|
||||
|
||||
function isIdentifier(value) {
|
||||
return /^[a-z][a-z0-9._:-]{2,127}$/i.test(value);
|
||||
}
|
||||
|
||||
function asyncRoute(handler) {
|
||||
return (req, res, next) => Promise.resolve(handler(req, res, next)).catch(next);
|
||||
}
|
||||
|
||||
function httpError(status, code) {
|
||||
return Object.assign(new Error(code), { status, code });
|
||||
}
|
||||
|
||||
function safeErrorCode(error) {
|
||||
return String(error?.code || error?.message || "internal_error").replace(/[^a-z0-9_.:-]/gi, "_").slice(0, 120);
|
||||
}
|
||||
|
||||
async function sweepExpiredRawEnvelopes() {
|
||||
const result = await pool.query(
|
||||
`with expired as (
|
||||
select id from external_data_plane_raw_envelopes
|
||||
where expires_at < now()
|
||||
order by expires_at asc
|
||||
limit $1
|
||||
)
|
||||
delete from external_data_plane_raw_envelopes as target
|
||||
using expired where target.id = expired.id`,
|
||||
[config.retentionDeleteLimit],
|
||||
);
|
||||
return result.rowCount;
|
||||
}
|
||||
|
||||
async function sweepRetention() {
|
||||
const rawEnvelopeCount = await sweepExpiredRawEnvelopes();
|
||||
const patchEventCount = await prunePatchOutbox(pool, {
|
||||
retentionMs: config.patchRetentionMs,
|
||||
limit: config.retentionDeleteLimit,
|
||||
});
|
||||
const historyFactCount = await pruneDataProductHistory(pool, { limit: config.retentionDeleteLimit });
|
||||
const batchReceiptCount = await pruneBatchReceipts(pool, {
|
||||
retentionMs: config.receiptRetentionMs,
|
||||
limit: config.retentionDeleteLimit,
|
||||
});
|
||||
lastRetentionSweepAt = new Date().toISOString();
|
||||
return { rawEnvelopeCount, patchEventCount, historyFactCount, batchReceiptCount };
|
||||
}
|
||||
|
||||
async function shutdown() {
|
||||
if (shutdownPromise) return shutdownPromise;
|
||||
shutdownPromise = (async () => {
|
||||
shuttingDown = true;
|
||||
if (retentionSweepTimer) clearInterval(retentionSweepTimer);
|
||||
for (const response of activeStreamResponses) {
|
||||
if (!response.writableEnded) response.end();
|
||||
}
|
||||
const closed = new Promise((resolve) => httpServer.close(resolve));
|
||||
httpServer.closeIdleConnections?.();
|
||||
const forceClose = setTimeout(() => httpServer.closeAllConnections?.(), 250);
|
||||
await Promise.race([closed, new Promise((resolve) => setTimeout(resolve, 2_000))]);
|
||||
clearTimeout(forceClose);
|
||||
httpServer.closeAllConnections?.();
|
||||
await pool.end();
|
||||
})();
|
||||
return shutdownPromise;
|
||||
}
|
||||
|
|
@ -1,182 +0,0 @@
|
|||
import { createHash, randomBytes } from "node:crypto";
|
||||
|
||||
const IDENTIFIER = /^[a-z][a-z0-9._:-]{2,127}$/;
|
||||
const TOKEN_PREFIX = "ndc_edpwb_";
|
||||
const SECRET_LIKE_KEY = /(token|secret|password|access[_-]?token|refresh[_-]?token|api[_-]?key)/i;
|
||||
const BINDING_REQUEST_KEYS = new Set(["source", "allowedDataProductIds", "expiresAt"]);
|
||||
const BINDING_SOURCE_KEYS = new Set(["tenantId", "connectionId", "providerId"]);
|
||||
|
||||
/**
|
||||
* Produces an opaque, high-entropy capability. The plaintext is returned only
|
||||
* to the trusted provisioning caller; persistence uses its SHA-256 digest.
|
||||
*/
|
||||
export function createWriterToken() {
|
||||
return `${TOKEN_PREFIX}${randomBytes(32).toString("base64url")}`;
|
||||
}
|
||||
|
||||
export function hashWriterToken(token) {
|
||||
return createHash("sha256").update(String(token), "utf8").digest("hex");
|
||||
}
|
||||
|
||||
export function normalizeWriterBindingRequest(value, { now = new Date(), maxTtlDays = 90 } = {}) {
|
||||
if (!isPlainObject(value) || !isPlainObject(value.source)) {
|
||||
throw writerBindingError("writer_binding_request_invalid");
|
||||
}
|
||||
if (containsSecretLikeKey(value)) {
|
||||
throw writerBindingError("writer_binding_request_secret_material_forbidden");
|
||||
}
|
||||
if (value.endpoint !== undefined || value.url !== undefined || value.host !== undefined) {
|
||||
throw writerBindingError("writer_binding_request_transport_forbidden");
|
||||
}
|
||||
if (!hasOnlyKeys(value, BINDING_REQUEST_KEYS) || !hasOnlyKeys(value.source, BINDING_SOURCE_KEYS)) {
|
||||
throw writerBindingError("writer_binding_request_fields_invalid");
|
||||
}
|
||||
|
||||
const tenantId = normalizeIdentifier(value.source.tenantId);
|
||||
const connectionId = normalizeIdentifier(value.source.connectionId);
|
||||
const providerId = normalizeIdentifier(value.source.providerId);
|
||||
if (!tenantId || !connectionId || !providerId) {
|
||||
throw writerBindingError("writer_binding_scope_invalid");
|
||||
}
|
||||
|
||||
const allowedDataProductIds = uniqueIdentifiers(value.allowedDataProductIds);
|
||||
if (!allowedDataProductIds.length) {
|
||||
throw writerBindingError("writer_binding_data_products_invalid");
|
||||
}
|
||||
|
||||
const expiresAt = new Date(String(value.expiresAt || ""));
|
||||
const maxExpiresAt = new Date(now.getTime() + maxTtlDays * 24 * 60 * 60 * 1000);
|
||||
if (Number.isNaN(expiresAt.getTime()) || expiresAt <= now || expiresAt > maxExpiresAt) {
|
||||
throw writerBindingError("writer_binding_expiry_invalid");
|
||||
}
|
||||
|
||||
return Object.freeze({
|
||||
tenantId,
|
||||
connectionId,
|
||||
providerId,
|
||||
allowedDataProductIds,
|
||||
expiresAt: expiresAt.toISOString(),
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Converts a caller-provided, deliberately unscoped intake envelope into the
|
||||
* canonical scoped form. Caller scope is rejected, never trusted or merged.
|
||||
*/
|
||||
export function materializeWriterBoundBatch(value, binding, { hasScopeHeaders = false, now = new Date() } = {}) {
|
||||
if (!isPlainObject(value) || !isPlainObject(value.source) || !isPlainObject(binding)) {
|
||||
throw writerBindingError("writer_bound_intake_invalid");
|
||||
}
|
||||
if (hasScopeHeaders || value.source.tenantId !== undefined || value.source.connectionId !== undefined) {
|
||||
throw writerBindingError("writer_bound_scope_forbidden");
|
||||
}
|
||||
if (binding.active !== true || !bindingIsCurrent(binding, now)) {
|
||||
throw writerBindingError("writer_binding_inactive");
|
||||
}
|
||||
|
||||
const providerId = normalizeIdentifier(value.source.providerId);
|
||||
const tenantId = normalizeIdentifier(binding.tenantId);
|
||||
const connectionId = normalizeIdentifier(binding.connectionId);
|
||||
if (!providerId || !tenantId || !connectionId || providerId !== binding.providerId) {
|
||||
throw writerBindingError("writer_binding_provider_forbidden");
|
||||
}
|
||||
|
||||
const dataProductId = normalizeIdentifier(value.contract?.dataProductId);
|
||||
const allowedDataProductIds = uniqueIdentifiers(binding.allowedDataProductIds);
|
||||
if (!dataProductId || !allowedDataProductIds.includes(dataProductId)) {
|
||||
throw writerBindingError("writer_binding_data_product_forbidden");
|
||||
}
|
||||
|
||||
return {
|
||||
...value,
|
||||
source: { providerId, tenantId, connectionId },
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Materializes the provider-neutral Data Product publish wire form. Unlike the
|
||||
* legacy writer-bound intake, the caller cannot send provider identity,
|
||||
* contract metadata, receivedAt or any scope at all.
|
||||
*/
|
||||
export function materializeDataProductPublish(value, binding, definition, dataProductId, { now = new Date() } = {}) {
|
||||
if (!isPlainObject(value) || !isPlainObject(binding) || !isPlainObject(definition)) {
|
||||
throw writerBindingError("data_product_publish_invalid");
|
||||
}
|
||||
if (binding.active !== true || !bindingIsCurrent(binding, now)) {
|
||||
throw writerBindingError("writer_binding_inactive");
|
||||
}
|
||||
const normalizedProductId = normalizeIdentifier(dataProductId);
|
||||
const allowedDataProductIds = uniqueIdentifiers(binding.allowedDataProductIds);
|
||||
if (!normalizedProductId || normalizedProductId !== definition.id || !allowedDataProductIds.includes(normalizedProductId)) {
|
||||
throw writerBindingError("writer_binding_data_product_forbidden");
|
||||
}
|
||||
const tenantId = normalizeIdentifier(binding.tenantId);
|
||||
const connectionId = normalizeIdentifier(binding.connectionId);
|
||||
const providerId = normalizeIdentifier(binding.providerId);
|
||||
if (!tenantId || !connectionId || !providerId) throw writerBindingError("writer_binding_scope_invalid");
|
||||
|
||||
return {
|
||||
schemaVersion: "nodedc.external-provider-contract/v1",
|
||||
source: { tenantId, connectionId, providerId },
|
||||
contract: {
|
||||
dataProductId: normalizedProductId,
|
||||
ontologyRevision: definition.ontologyRevision,
|
||||
version: definition.version,
|
||||
},
|
||||
batch: {
|
||||
runId: value.batch?.runId,
|
||||
sequence: value.batch?.sequence,
|
||||
idempotencyKey: value.batch?.idempotencyKey,
|
||||
receivedAt: now.toISOString(),
|
||||
},
|
||||
facts: value.facts,
|
||||
};
|
||||
}
|
||||
|
||||
export function safeWriterBinding(binding) {
|
||||
return {
|
||||
id: binding.id,
|
||||
tenantId: binding.tenantId,
|
||||
connectionId: binding.connectionId,
|
||||
providerId: binding.providerId,
|
||||
allowedDataProductIds: uniqueIdentifiers(binding.allowedDataProductIds),
|
||||
active: binding.active === true,
|
||||
expiresAt: new Date(binding.expiresAt).toISOString(),
|
||||
createdAt: binding.createdAt ? new Date(binding.createdAt).toISOString() : undefined,
|
||||
rotatedAt: binding.rotatedAt ? new Date(binding.rotatedAt).toISOString() : undefined,
|
||||
revokedAt: binding.revokedAt ? new Date(binding.revokedAt).toISOString() : undefined,
|
||||
};
|
||||
}
|
||||
|
||||
export function writerBindingError(code) {
|
||||
return Object.assign(new Error(code), { status: 400, code });
|
||||
}
|
||||
|
||||
function bindingIsCurrent(binding, now) {
|
||||
const expiresAt = new Date(binding.expiresAt);
|
||||
return !Number.isNaN(expiresAt.getTime()) && expiresAt > now;
|
||||
}
|
||||
|
||||
function normalizeIdentifier(value) {
|
||||
const normalized = typeof value === "string" ? value.trim() : "";
|
||||
return IDENTIFIER.test(normalized) ? normalized : "";
|
||||
}
|
||||
|
||||
function uniqueIdentifiers(value) {
|
||||
if (!Array.isArray(value)) return [];
|
||||
return [...new Set(value.map(normalizeIdentifier).filter(Boolean))];
|
||||
}
|
||||
|
||||
function isPlainObject(value) {
|
||||
return Boolean(value) && typeof value === "object" && !Array.isArray(value);
|
||||
}
|
||||
|
||||
function containsSecretLikeKey(value) {
|
||||
if (Array.isArray(value)) return value.some(containsSecretLikeKey);
|
||||
if (!isPlainObject(value)) return false;
|
||||
return Object.entries(value).some(([key, child]) => SECRET_LIKE_KEY.test(key) || containsSecretLikeKey(child));
|
||||
}
|
||||
|
||||
function hasOnlyKeys(value, allowedKeys) {
|
||||
return Object.keys(value).every((key) => allowedKeys.has(key));
|
||||
}
|
||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue