feat(map): add persistent gateway and ontology package

This commit is contained in:
Codex
2026-07-13 17:14:34 +03:00
parent d196d4b0c7
commit e527812826
21 changed files with 1323 additions and 9 deletions
+6
View File
@@ -0,0 +1,6 @@
node_modules/
npm-debug.log
.env
.env.*
data/
volumes/
+20
View File
@@ -0,0 +1,20 @@
FROM node:20-alpine AS runner
ENV NODE_ENV=production
ENV PORT=18103
ENV MAP_CACHE_DIR=/var/lib/nodedc-map-cache
WORKDIR /app
RUN apk add --no-cache su-exec
COPY package.json ./
COPY src ./src
COPY docker-entrypoint.sh /usr/local/bin/nodedc-map-gateway
RUN chmod +x /usr/local/bin/nodedc-map-gateway
EXPOSE 18103
ENTRYPOINT ["/usr/local/bin/nodedc-map-gateway"]
CMD ["node", "src/server.mjs"]
+87
View File
@@ -0,0 +1,87 @@
# NODE.DC Map Gateway
`map-gateway` — единый платформенный boundary для map providers. Он не является частью Engine и не принадлежит отдельному Module Studio application.
## Что делает
- хранит master `CESIUM_ION_TOKEN` только в process environment;
- выдаёт браузеру только asset-scoped endpoint token для allowlisted ion assets;
- проксирует и кэширует tile/3D Tiles/terrain resources через `GET /api/map/cache?url=<https-url>`;
- может отдать перенесённый локальный Engine snapshot через `MAP_GATEWAY_LEGACY_CACHE_HOSTS`, но никогда не догружает для него cache miss из сети;
- работает с persistent volume, а не с Git или Docker image layer;
- при upstream outage отдаёт ранее сохранённый cache object; offline режим включается только для providers, явно разрешённых их лицензией;
- защищает proxy allowlist-ом upstream hosts, HTTPS-only правилом, лимитами размера и token-free cache key;
- ведёт лёгкий persistent cache index для LRU eviction и health/stats.
## API
```text
GET /healthz
GET /api/map/ion/assets/:assetId/endpoint
GET|HEAD /api/map/cache?url=<encoded-upstream-url>
```
`/api/map/ion/assets/:assetId/endpoint` разрешает только `CESIUM_ION_ASSET_ALLOWLIST`. Ответ содержит runtime asset token, а не master token. Browser использует endpoint вместе с Cesium `DefaultProxy`, направляющим resource requests в `/api/map/cache`.
Ion endpoint metadata также сохраняется в persistent volume с файловыми правами `0600`. Это нужно для cold start в offline: Cesium получает сохранённый asset URL и asset-scoped token, а proxy отвечает только из ранее записанного cache. В metadata никогда не записывается master token.
## Cache modes
- `readwrite` — свежий cache hit отдаётся сразу; stale object обновляется online; при upstream failure возвращается stale copy;
- `readonly` — использует cache, но не записывает новый;
- `offline` — никогда не идёт наружу; cache miss возвращает `504`.
`offline` дополнительно требует, чтобы hostname был явно указан в `MAP_GATEWAY_OFFLINE_PROVIDER_ALLOWLIST`. По умолчанию список пуст: Gateway не превращает Cesium ion или public OSM tile service в offline distribution. Это осознанная provider policy, а не техническая ошибка cache.
Range requests пока transparently проксируются и не записываются как partial cache object. Полные GET responses кэшируются. Это безопасная начальная граница для terrain/3D Tiles; отдельным следующим шагом добавляется range-aware chunk cache после замеров реальных GOS/3D Tiles assets.
## Persistent storage
Platform Compose использует два внешних named volume: mutable `nodedc-platform_map-live-tile-cache` монтируется в `/var/lib/nodedc-map-live-cache`, read-only snapshot `nodedc-platform_map-offline-snapshot` — в `/var/lib/nodedc-map-offline-snapshot`. Их содержимое намеренно исключено из Git. Внешний volume не удаляется через `docker compose down -v`; очистка требует явного `docker volume rm` при остановленном Gateway. При необходимости offline seed доставляется отдельным export/import artifact, а не коммитом runtime cache.
Кэш не способен создать участок карты, который никогда не был скачан. Для целевых офлайн-регионов нужен отдельный licensed dataset/provider и его `seed/prefetch` job: он заранее проходит разрешённые assets, zoom-диапазоны и географические области, записывает их в тот же persistent volume и формирует проверяемый export/import artifact. Эта задача не смешивается с runtime gateway и не помещает binary tiles в историю репозитория.
### Legacy Engine cache: локальный перенос без изменения Engine
В локальной Studio-песочнице существующий Engine cache можно скопировать в persistent volume как неизменяемый входной snapshot. Исходная директория Engine при этом только читается: import job не удаляет, не переименовывает и не изменяет ни одного её файла.
```bash
npm run audit:engine-cache -- /absolute/path/to/nodedc-data/api/cesium/manifest.json
```
Команда audit выдаёт агрегированный отчёт: hosts, число объектов, declared/on-disk размер и отсутствующие файлы. Сам перенос выполняется отдельно, после остановки gateway на время записи index:
```bash
npm run import:engine-cache -- /absolute/path/to/nodedc-data/api/cesium/manifest.json /persistent/map-tile-cache
```
По умолчанию действует режим `no-overwrite`: если объект уже присутствует в новом cache, он остаётся без изменений. `--dry-run` выводит план без копирования, `--refresh` разрешает осознанно заменить уже импортированный объект. Legacy `http` URLs нормализуются до `https`, чтобы совпасть с HTTPS-only Gateway; сами файлы, content-type, etag, timestamp и provenance сохраняются в новом index/import report.
Это sandbox-механизм переноса уже существующей локальной реализации. Для внешней коммерческой публикации потребуется отдельная provider/offline policy; она не блокирует текущую локальную миграцию, но и не должна смешиваться с ней. Runtime cache по-прежнему не попадает в Git или Docker image layer.
## Security and access
В local dev допускается `MAP_GATEWAY_ALLOW_ANONYMOUS=true`. В production он должен быть `false`; Caddy/Auth BFF проверяет Authentik session и передаёт очищенный trusted subject header в private platform network. Нельзя открывать gateway напрямую наружу и нельзя принимать user-provided upstream hosts вне allowlist.
## Required environment
```text
CESIUM_ION_TOKEN=
CESIUM_ION_ASSET_ALLOWLIST=1,96188
MAP_GATEWAY_UPSTREAM_ALLOWLIST=api.cesium.com,assets.ion.cesium.com,tile.openstreetmap.org
MAP_GATEWAY_LEGACY_CACHE_HOSTS=ecn.t0.tiles.virtualearth.net,ecn.t1.tiles.virtualearth.net,ecn.t2.tiles.virtualearth.net,ecn.t3.tiles.virtualearth.net
MAP_GATEWAY_OFFLINE_PROVIDER_ALLOWLIST=
MAP_CACHE_DIR=/var/lib/nodedc-map-cache
MAP_CACHE_MODE=readwrite
```
# Cache topology
`MAP_CACHE_DIR` is the mutable **live** cache. In the Platform compose profile it
is mounted as `map-live-tile-cache` and is the only store that can receive new
upstream objects.
`MAP_OFFLINE_SNAPSHOT_DIR` is optional and mounted read-only as
`map-offline-snapshot`. It is populated only by the controlled Engine cache
import. Requests marked with the `offline` cache profile are served exclusively
from this snapshot; a missing tile returns a cache miss and never reaches an
upstream provider.
+11
View File
@@ -0,0 +1,11 @@
#!/bin/sh
set -eu
# Docker creates a fresh named volume as root. Prepare only the mutable live
# cache before dropping privileges; the imported offline snapshot stays read-only.
if [ -d "${MAP_CACHE_DIR:-}" ]; then
mkdir -p "$MAP_CACHE_DIR"
chown -R node:node "$MAP_CACHE_DIR"
fi
exec su-exec node "$@"
+15
View File
@@ -0,0 +1,15 @@
{
"name": "@nodedc/map-gateway",
"version": "0.1.0",
"private": true,
"type": "module",
"scripts": {
"start": "node src/server.mjs",
"dev": "node --watch src/server.mjs",
"audit:engine-cache": "node scripts/audit-engine-cache.mjs",
"import:engine-cache": "node scripts/import-engine-cache.mjs"
},
"engines": {
"node": ">=20"
}
}
@@ -0,0 +1,91 @@
#!/usr/bin/env node
import { access, readFile, stat } from "node:fs/promises";
import { dirname, isAbsolute, join, resolve } from "node:path";
const manifestPath = resolve(process.argv[2] || "");
if (!process.argv[2]) {
console.error("Usage: npm run audit:engine-cache -- /absolute/path/to/cesium/manifest.json");
process.exitCode = 1;
} else {
await audit(manifestPath);
}
async function audit(path) {
let manifest;
try {
manifest = JSON.parse(await readFile(path, "utf8"));
} catch (error) {
throw new Error(`Cannot read legacy cache manifest: ${error instanceof Error ? error.message : "unknown error"}`);
}
const root = dirname(path);
const entries = Object.entries(manifest?.entries || {});
const providers = new Map();
let bytesFromManifest = 0;
let presentFiles = 0;
let missingFiles = 0;
let bytesOnDisk = 0;
for (const [rawUrl, entry] of entries) {
let hostname = "invalid-url";
try { hostname = new URL(rawUrl).hostname.toLowerCase() || "invalid-url"; } catch {}
const provider = providers.get(hostname) || { entries: 0, bytes: 0, presentFiles: 0, missingFiles: 0 };
const entryBytes = Math.max(0, Number(entry?.size || 0));
provider.entries += 1;
provider.bytes += entryBytes;
bytesFromManifest += entryBytes;
const relativeFile = String(entry?.path || "");
const candidate = resolve(root, relativeFile);
if (!relativeFile || !candidate.startsWith(`${root}/`) || isAbsolute(relativeFile) || !(await fileExists(candidate))) {
provider.missingFiles += 1;
missingFiles += 1;
providers.set(hostname, provider);
continue;
}
const fileInfo = await stat(candidate);
presentFiles += 1;
provider.presentFiles += 1;
bytesOnDisk += fileInfo.size;
providers.set(hostname, provider);
}
const report = {
kind: "nodedc-legacy-cesium-cache-audit",
generatedAt: new Date().toISOString(),
manifestPath: path,
manifestVersion: manifest?.version ?? null,
entries: entries.length,
declaredBytes: bytesFromManifest,
declaredMegabytes: toMegabytes(bytesFromManifest),
presentFiles,
missingFiles,
bytesOnDisk,
megabytesOnDisk: toMegabytes(bytesOnDisk),
providers: [...providers.entries()]
.map(([hostname, value]) => ({ hostname, ...value, megabytes: toMegabytes(value.bytes) }))
.sort((left, right) => right.bytes - left.bytes),
migrationPolicy: {
automaticImport: false,
reason: "Audit only. Import requires an explicit licence/provider decision for every source host and must preserve required attribution.",
target: "Map Gateway persistent volume or approved object storage; never Git history or a Docker image layer."
}
};
process.stdout.write(`${JSON.stringify(report, null, 2)}\n`);
}
async function fileExists(path) {
try {
await access(path);
return true;
} catch {
return false;
}
}
function toMegabytes(bytes) {
return Number((bytes / 1024 / 1024).toFixed(2));
}
@@ -0,0 +1,173 @@
#!/usr/bin/env node
import { createHash } from "node:crypto";
import { copyFile, mkdir, readFile, rename, stat, writeFile } from "node:fs/promises";
import { dirname, isAbsolute, join, resolve } from "node:path";
const args = new Set(process.argv.slice(2));
const positional = process.argv.slice(2).filter((value) => !value.startsWith("--"));
const [manifestArgument, cacheArgument] = positional;
const dryRun = args.has("--dry-run");
const refresh = args.has("--refresh");
if (!manifestArgument || !cacheArgument) {
console.error("Usage: npm run import:engine-cache -- /absolute/path/to/manifest.json /persistent/map-cache [--dry-run] [--refresh]");
process.exitCode = 1;
} else {
await importEngineCache(resolve(manifestArgument), resolve(cacheArgument), { dryRun, refresh });
}
async function importEngineCache(manifestPath, cacheDir, options) {
const manifest = JSON.parse(await readFile(manifestPath, "utf8"));
const sourceRoot = dirname(manifestPath);
const sourceEntries = Object.entries(manifest?.entries || {});
const indexPath = join(cacheDir, "index.json");
const index = await readIndex(indexPath);
const reindexed = reindexLegacyEntries(index);
const report = {
kind: "nodedc-engine-cache-import",
importedAt: new Date().toISOString(),
sourceManifest: manifestPath,
sourceManifestVersion: manifest?.version ?? null,
mode: options.dryRun ? "dry-run" : options.refresh ? "refresh" : "no-overwrite",
totalEntries: sourceEntries.length,
imported: 0,
alreadyPresent: 0,
missingSourceFiles: 0,
invalidEntries: 0,
reindexed,
bytesImported: 0,
providers: {},
};
if (!options.dryRun) await mkdir(join(cacheDir, "objects"), { recursive: true });
for (const [rawUrl, source] of sourceEntries) {
const normalized = normalizeLegacyUrl(rawUrl);
const relativeSource = String(source?.path || "");
const sourcePath = resolve(sourceRoot, relativeSource);
if (!normalized || !relativeSource || isAbsolute(relativeSource) || !sourcePath.startsWith(`${sourceRoot}/`)) {
report.invalidEntries += 1;
continue;
}
let sourceInfo;
try {
sourceInfo = await stat(sourcePath);
if (!sourceInfo.isFile()) throw new Error("not_file");
} catch {
report.missingSourceFiles += 1;
continue;
}
const key = createHash("sha256").update(canonicalCacheUrl(normalized)).digest("hex");
const targetFile = join("objects", key.slice(0, 2), `${key}.bin`);
const targetPath = join(cacheDir, targetFile);
const existsInIndex = Boolean(index.entries[key]);
const existsOnDisk = await fileExists(targetPath);
if (!options.refresh && (existsInIndex || existsOnDisk)) {
report.alreadyPresent += 1;
continue;
}
const hostname = new URL(normalized).hostname.toLowerCase();
report.providers[hostname] = (report.providers[hostname] || 0) + 1;
const savedAt = Number(source?.savedAt || Date.now());
const entry = {
key,
file: targetFile,
bytes: sourceInfo.size,
contentType: String(source?.contentType || "application/octet-stream"),
etag: String(source?.etag || ""),
savedAt,
lastAccessAt: Date.now(),
// Imported records are deliberately treated as fresh in the sandbox.
// A refresh is always explicit through nodedc_cache_refresh=1.
expiresAt: Date.now() + 3650 * 24 * 60 * 60 * 1000,
source: "engine-legacy-cache",
sourceUrl: normalized,
};
if (!options.dryRun) {
await mkdir(dirname(targetPath), { recursive: true });
await copyFile(sourcePath, targetPath);
index.entries[key] = entry;
}
report.imported += 1;
report.bytesImported += sourceInfo.size;
}
if (!options.dryRun) {
await writeJsonAtomically(indexPath, index);
const importsDir = join(cacheDir, "imports");
await mkdir(importsDir, { recursive: true });
await writeJsonAtomically(join(importsDir, "engine-legacy-cache-import.json"), report);
}
process.stdout.write(`${JSON.stringify(report, null, 2)}\n`);
}
function normalizeLegacyUrl(rawUrl) {
try {
const target = new URL(rawUrl);
// The Engine cache predates the HTTPS-only Gateway. The same tile hosts
// support HTTPS, so normalize only the protocol; path and provider stay intact.
if (target.protocol === "http:") target.protocol = "https:";
if (target.protocol !== "https:") return null;
return target;
} catch {
return null;
}
}
function canonicalCacheUrl(target) {
const url = new URL(target.toString());
if (/^ecn\.t[0-3]\.tiles\.virtualearth\.net$/i.test(url.hostname)) {
url.hostname = "ecn.tiles.virtualearth.net";
}
for (const key of [...url.searchParams.keys()]) {
if (["access_token", "accesstoken", "iontoken", "token", "key", "apikey", "api_key", "signature", "sig"].includes(key.toLowerCase())) {
url.searchParams.delete(key);
}
}
url.searchParams.sort();
return url.toString();
}
function reindexLegacyEntries(index) {
let reindexed = 0;
for (const [currentKey, entry] of Object.entries(index.entries)) {
if (entry?.source !== "engine-legacy-cache" || !entry?.sourceUrl) continue;
let source;
try { source = new URL(String(entry.sourceUrl)); } catch { continue; }
const nextKey = createHash("sha256").update(canonicalCacheUrl(source)).digest("hex");
if (nextKey === currentKey || index.entries[nextKey]) continue;
index.entries[nextKey] = { ...entry, key: nextKey };
delete index.entries[currentKey];
reindexed += 1;
}
return reindexed;
}
async function readIndex(indexPath) {
try {
const parsed = JSON.parse(await readFile(indexPath, "utf8"));
if (parsed?.version === 1 && parsed.entries && typeof parsed.entries === "object") return parsed;
} catch (error) {
if (error?.code !== "ENOENT") throw error;
}
return { version: 1, entries: {} };
}
async function fileExists(path) {
try {
return (await stat(path)).isFile();
} catch {
return false;
}
}
async function writeJsonAtomically(path, value) {
const temp = `${path}.${process.pid}.tmp`;
await writeFile(temp, `${JSON.stringify(value)}\n`, "utf8");
await rename(temp, path);
}
+533
View File
@@ -0,0 +1,533 @@
import { createHash } from "node:crypto";
import { createReadStream, createWriteStream } from "node:fs";
import { mkdir, readFile, rename, rm, stat, writeFile } from "node:fs/promises";
import { createServer } from "node:http";
import { dirname, join } from "node:path";
import { Readable, Transform } from "node:stream";
import { pipeline } from "node:stream/promises";
const config = readConfig();
const liveCache = createCacheStore("live", config.cacheDir, true);
const offlineSnapshot = config.offlineSnapshotDir ? createCacheStore("offline-snapshot", config.offlineSnapshotDir, false) : null;
const inflightWrites = new Map();
await initialiseCacheStore(liveCache);
if (offlineSnapshot) await initialiseCacheStore(offlineSnapshot);
const server = createServer(async (request, response) => {
applyCors(request, response);
if (request.method === "OPTIONS") return response.writeHead(204).end();
try {
const requestUrl = new URL(request.url || "/", `http://${request.headers.host || "127.0.0.1"}`);
if (!isAllowedRequest(request)) return writeJson(response, 401, { ok: false, error: "map_gateway_auth_required" });
if (requestUrl.pathname === "/healthz" && request.method === "GET") {
return writeJson(response, 200, {
ok: true,
service: "nodedc-map-gateway",
cache: await cacheStats(liveCache),
offlineSnapshot: offlineSnapshot ? await cacheStats(offlineSnapshot) : null,
ionConfigured: Boolean(config.cesiumIonToken),
assetAllowlist: [...config.assetAllowlist].map(Number).sort((left, right) => left - right),
anonymousAccess: config.allowAnonymous,
});
}
const ionAssetMatch = requestUrl.pathname.match(/^\/api\/map\/ion\/assets\/(\d+)\/endpoint$/);
if (ionAssetMatch && request.method === "GET") {
return await serveIonEndpoint(response, ionAssetMatch[1]);
}
if (requestUrl.pathname === "/api/map/cache" && ["GET", "HEAD"].includes(request.method || "")) {
return await serveCachedUpstream(request, response, requestUrl);
}
return writeJson(response, 404, { ok: false, error: "map_gateway_route_not_found" });
} catch (error) {
// A client can cancel a streamed cache response after its headers have
// already been sent. Do not try to turn that into JSON: Node would throw
// ERR_HTTP_HEADERS_SENT and take the entire gateway process down.
if (response.headersSent || response.writableEnded) {
if (!response.writableEnded) response.destroy();
return;
}
const status = Number(error?.statusCode || 500);
return writeJson(response, Number.isInteger(status) && status >= 400 && status < 600 ? status : 500, {
ok: false,
error: error instanceof Error ? error.message : "map_gateway_error",
});
}
});
server.listen(config.port, "0.0.0.0", () => {
console.log(`NODE.DC Map Gateway listening on http://0.0.0.0:${config.port}`);
console.log(`Live map cache: ${config.cacheDir} (${config.mode}, max ${Math.round(config.maxCacheBytes / 1024 / 1024)} MB)`);
if (offlineSnapshot) console.log(`Offline map snapshot: ${config.offlineSnapshotDir} (read-only)`);
});
process.on("SIGTERM", () => server.close());
process.on("SIGINT", () => server.close());
function readConfig() {
const mode = String(process.env.MAP_CACHE_MODE || "readwrite").trim().toLowerCase();
if (!["readwrite", "readonly", "offline"].includes(mode)) throw new Error("invalid_map_cache_mode");
const allowlist = parseList(process.env.MAP_GATEWAY_UPSTREAM_ALLOWLIST || "api.cesium.com,assets.ion.cesium.com,tile.openstreetmap.org,dev.virtualearth.net,ecn.t0.tiles.virtualearth.net,ecn.t1.tiles.virtualearth.net,ecn.t2.tiles.virtualearth.net,ecn.t3.tiles.virtualearth.net");
return {
port: parsePositiveInt(process.env.PORT, 18103),
cacheDir: String(process.env.MAP_CACHE_DIR || "/var/lib/nodedc-map-cache").trim(),
offlineSnapshotDir: String(process.env.MAP_OFFLINE_SNAPSHOT_DIR || "").trim(),
mode,
maxCacheBytes: parsePositiveInt(process.env.MAP_CACHE_MAX_MB, 20480) * 1024 * 1024,
maxObjectBytes: parsePositiveInt(process.env.MAP_CACHE_MAX_OBJECT_MB, 128) * 1024 * 1024,
defaultTtlMs: parsePositiveInt(process.env.MAP_CACHE_DEFAULT_TTL_SECONDS, 604800) * 1000,
upstreamTimeoutMs: parsePositiveInt(process.env.MAP_GATEWAY_UPSTREAM_TIMEOUT_SECONDS, 30) * 1000,
cesiumIonToken: String(process.env.CESIUM_ION_TOKEN || "").trim(),
assetAllowlist: new Set(parseList(process.env.CESIUM_ION_ASSET_ALLOWLIST || "1,2,96188")),
upstreamAllowlist: new Set(allowlist),
legacyCacheHosts: new Set(parseList(process.env.MAP_GATEWAY_LEGACY_CACHE_HOSTS || "")),
offlineProviderAllowlist: new Set(parseList(process.env.MAP_GATEWAY_OFFLINE_PROVIDER_ALLOWLIST || "")),
corsOrigins: new Set(parseList(process.env.MAP_GATEWAY_CORS_ORIGIN || "http://127.0.0.1:3333,http://localhost:3333")),
allowAnonymous: parseBoolean(process.env.MAP_GATEWAY_ALLOW_ANONYMOUS, process.env.NODE_ENV !== "production"),
trustedSubjectHeader: String(process.env.MAP_GATEWAY_TRUSTED_SUBJECT_HEADER || "x-nodedc-user-id").toLowerCase(),
};
}
function parseList(value) {
return String(value || "").split(",").map((part) => part.trim().toLowerCase()).filter(Boolean);
}
function parsePositiveInt(value, fallback) {
const parsed = Number.parseInt(String(value || ""), 10);
return Number.isInteger(parsed) && parsed > 0 ? parsed : fallback;
}
function parseBoolean(value, fallback) {
if (value === undefined || value === "") return fallback;
return ["1", "true", "yes", "on"].includes(String(value).trim().toLowerCase());
}
function applyCors(request, response) {
const origin = String(request.headers.origin || "");
if (!origin || (!config.corsOrigins.has("*") && !config.corsOrigins.has(origin))) return;
response.setHeader("access-control-allow-origin", config.corsOrigins.has("*") ? "*" : origin);
response.setHeader("access-control-allow-methods", "GET, HEAD, OPTIONS");
response.setHeader("access-control-allow-headers", "Range, Content-Type");
response.setHeader("vary", "Origin");
}
function isAllowedRequest(request) {
return config.allowAnonymous || Boolean(request.headers[config.trustedSubjectHeader]);
}
async function serveIonEndpoint(response, assetId) {
if (!config.assetAllowlist.has(assetId)) return writeJson(response, 403, { ok: false, error: "cesium_asset_not_allowed" });
const cached = await readIonEndpointCache(assetId);
if (config.mode === "offline") {
if (!cached) return writeJson(response, 504, { ok: false, error: "cesium_ion_offline_endpoint_miss" });
if (!isOfflineProviderAllowed(cached.url)) return writeJson(response, 409, { ok: false, error: "map_provider_offline_not_permitted" });
return writeJson(response, 200, { ok: true, ...cached, cache: "offline-endpoint-hit" });
}
if (!config.cesiumIonToken) {
if (cached) return writeJson(response, 200, { ok: true, ...cached, cache: "cached-endpoint-no-master-token" });
return writeJson(response, 503, { ok: false, error: "cesium_ion_not_configured" });
}
try {
const endpoint = await fetchWithTimeout(`https://api.cesium.com/v1/assets/${assetId}/endpoint`, {
headers: { authorization: `Bearer ${config.cesiumIonToken}` },
});
if (!endpoint.ok) throw gatewayError("cesium_ion_endpoint_unavailable", endpoint.status || 502);
const body = await endpoint.json();
const next = body.type === "IMAGERY" && body.externalType === "BING"
? {
assetId,
type: body.type,
externalType: "BING",
options: {
url: validateUpstream(String(body.options?.url || "")).toString(),
key: String(body.options?.key || ""),
mapStyle: String(body.options?.mapStyle || "Aerial"),
},
attributions: Array.isArray(body.attributions) ? body.attributions : [],
savedAt: Date.now(),
}
: {
assetId,
type: body.type,
url: validateUpstream(String(body.url || "")).toString(),
accessToken: String(body.accessToken || ""),
attributions: Array.isArray(body.attributions) ? body.attributions : [],
savedAt: Date.now(),
};
if (!(next.externalType === "BING" ? next.options?.key : next.accessToken)) throw gatewayError("cesium_ion_endpoint_invalid", 502);
if (config.mode === "readwrite") await writeIonEndpointCache(next);
return writeJson(response, 200, { ok: true, ...next, cache: "ion-endpoint-online" });
} catch (error) {
if (cached) return writeJson(response, 200, { ok: true, ...cached, cache: "stale-ion-endpoint" });
throw error;
}
}
async function serveCachedUpstream(request, response, requestUrl) {
const rawTarget = requestUrl.searchParams.get("url");
if (!rawTarget) return writeJson(response, 400, { ok: false, error: "map_cache_url_required" });
const target = validateUpstream(rawTarget);
const cacheProfile = String(target.searchParams.get("nodedc_cache_profile") || "live").toLowerCase();
target.searchParams.delete("nodedc_cache_profile");
if (!new Set(["live", "offline"]).has(cacheProfile)) return writeJson(response, 400, { ok: false, error: "map_cache_profile_invalid" });
const store = cacheProfile === "offline" ? offlineSnapshot : liveCache;
if (!store) return writeJson(response, 409, { ok: false, error: "map_offline_snapshot_not_configured" });
const cacheMode = String(target.searchParams.get("nodedc_cache_mode") || "hybrid").toLowerCase();
target.searchParams.delete("nodedc_cache_mode");
if (!new Set(["hybrid", "passthrough"]).has(cacheMode)) return writeJson(response, 400, { ok: false, error: "map_cache_mode_invalid" });
const isLegacyCacheHost = config.legacyCacheHosts.has(target.hostname.toLowerCase());
const forceRefresh = target.searchParams.get("nodedc_cache_refresh") === "1";
target.searchParams.delete("nodedc_cache_refresh");
const cacheKey = createHash("sha256").update(canonicalCacheUrl(target)).digest("hex");
// "Live" without cache is still routed through Gateway — it is never a
// browser-side direct request — but it neither reads from nor writes to the
// persistent store. The immutable offline profile intentionally has no such
// bypass mode.
if (cacheMode === "passthrough" && cacheProfile === "live") {
return proxyUncachedRequest(request, response, target, "live-pass-through");
}
const cached = await getCachedEntry(store, cacheKey);
const fresh = cached && cached.expiresAt > Date.now();
if (cached && (!forceRefresh || !store.mutable) && (fresh || !store.mutable || config.mode !== "readwrite")) {
return serveCachedFile(request, response, store, cached, fresh ? `${store.name}-hit` : `${store.name}-stale`);
}
// A migrated Engine cache is a local sandbox snapshot, not an instruction to
// fetch arbitrary historic providers when an object is missing. The profile
// is selected by the visual adapter; the browser can never turn it into a
// new upstream fetch by changing a setting.
if (cacheProfile === "offline") return writeJson(response, 504, { ok: false, error: "map_offline_snapshot_miss" });
if (isLegacyCacheHost) return writeJson(response, 504, { ok: false, error: "map_legacy_cache_miss" });
if (config.mode === "offline") {
if (!isOfflineProviderAllowed(target.toString())) return writeJson(response, 409, { ok: false, error: "map_provider_offline_not_permitted" });
return writeJson(response, 504, { ok: false, error: "map_cache_offline_miss" });
}
if (config.mode === "readonly") {
return proxyUncachedRequest(request, response, target, "pass-through-readonly");
}
if (String(request.headers.range || "").trim()) {
return proxyRangeRequest(request, response, target, cached, store);
}
try {
const entry = await fetchAndCache(target, cacheKey, store);
return serveCachedFile(request, response, store, entry, forceRefresh ? "live-refresh-record" : "live-record");
} catch (error) {
if (cached) return serveCachedFile(request, response, store, cached, "live-stale-upstream-error");
throw error;
}
}
function isOfflineProviderAllowed(rawTarget) {
if (!config.offlineProviderAllowlist.size) return false;
try {
const target = new URL(rawTarget);
return config.offlineProviderAllowlist.has(target.hostname.toLowerCase());
} catch {
return false;
}
}
function validateUpstream(rawTarget) {
if (rawTarget.length > 8192) throw gatewayError("map_cache_url_too_long", 400);
let target;
try { target = new URL(rawTarget); } catch { throw gatewayError("invalid_map_cache_url", 400); }
const hostname = target.hostname.toLowerCase();
// Bing's metadata endpoint still returns HTTP tile templates. They are not
// allowed through as HTTP: upgrade only the known Bing tile hosts before
// the standard HTTPS and allowlist checks. This keeps the browser and the
// Gateway on encrypted transport while making the provider contract work.
if (target.protocol === "http:" && /^(ecn\.t[0-3]\.tiles\.virtualearth\.net|ecn\.tiles\.virtualearth\.net)$/.test(hostname)) {
target.protocol = "https:";
}
if (target.protocol !== "https:") throw gatewayError("map_cache_https_required", 400);
if (target.username || target.password || (!config.upstreamAllowlist.has(hostname) && !config.legacyCacheHosts.has(hostname))) {
throw gatewayError("map_cache_upstream_not_allowed", 403);
}
return target;
}
function canonicalCacheUrl(target) {
const url = new URL(target.toString());
// The old Engine snapshot distributes identical Bing tile semantics across
// t0…t3. A viewer can choose a different subdomain for the same quadkey, so
// use one logical host for the cache key while preserving the original URL
// only as provenance in the imported index entry.
if (/^ecn\.t[0-3]\.tiles\.virtualearth\.net$/i.test(url.hostname)) {
url.hostname = "ecn.tiles.virtualearth.net";
}
for (const key of [...url.searchParams.keys()]) {
if (["access_token", "accesstoken", "iontoken", "token", "key", "apikey", "api_key", "signature", "sig"].includes(key.toLowerCase())) {
url.searchParams.delete(key);
}
}
url.searchParams.sort();
return url.toString();
}
async function fetchAndCache(target, cacheKey, store) {
const inflightKey = `${store.name}:${cacheKey}`;
const existing = inflightWrites.get(inflightKey);
if (existing) return existing;
const task = downloadAndCache(target, cacheKey, store);
inflightWrites.set(inflightKey, task);
// Several Cesium requests can share one in-flight download. Keep a rejection
// observer on the shared promise, then rethrow to each HTTP caller below.
// A 4xx/5xx tile response must never become an unhandled rejection that stops
// the gateway process.
task.catch(() => undefined);
try {
return await task;
} finally {
if (inflightWrites.get(inflightKey) === task) inflightWrites.delete(inflightKey);
}
}
async function downloadAndCache(target, cacheKey, store) {
const upstream = await fetchWithTimeout(target, { headers: { accept: "application/json, application/octet-stream, image/*, */*;q=0.5" } });
if (!upstream.ok || !upstream.body) throw gatewayError("map_upstream_unavailable", upstream.status || 502);
const expectedBytes = Number.parseInt(upstream.headers.get("content-length") || "", 10);
if (Number.isFinite(expectedBytes) && expectedBytes > config.maxObjectBytes) throw gatewayError("map_cache_object_too_large", 413);
const relativePath = join("objects", cacheKey.slice(0, 2), `${cacheKey}.bin`);
const filePath = join(store.dir, relativePath);
const tempPath = `${filePath}.${process.pid}.${Date.now()}.tmp`;
await mkdir(dirname(filePath), { recursive: true });
let bytes = 0;
const limit = new Transform({
transform(chunk, _encoding, callback) {
bytes += chunk.length;
if (bytes > config.maxObjectBytes) return callback(gatewayError("map_cache_object_too_large", 413));
callback(null, chunk);
},
});
try {
await pipeline(Readable.fromWeb(upstream.body), limit, createWriteStream(tempPath, { flags: "wx" }));
await rename(tempPath, filePath);
} catch (error) {
await rm(tempPath, { force: true });
throw error;
}
const now = Date.now();
const entry = {
key: cacheKey,
file: relativePath,
bytes,
contentType: upstream.headers.get("content-type") || "application/octet-stream",
etag: upstream.headers.get("etag") || null,
savedAt: now,
lastAccessAt: now,
expiresAt: now + responseTtl(upstream.headers.get("cache-control")),
};
store.index.entries[cacheKey] = entry;
await evictCache(store);
await writeCacheIndex(store);
return entry;
}
function responseTtl(cacheControl) {
const match = String(cacheControl || "").match(/max-age=(\d+)/i);
if (!match) return config.defaultTtlMs;
return Math.min(Number.parseInt(match[1], 10) * 1000, config.defaultTtlMs);
}
async function proxyRangeRequest(request, response, target, cached, store) {
const upstream = await fetchWithTimeout(target, { headers: { range: String(request.headers.range), accept: "*/*" } });
if (!upstream.ok || !upstream.body) {
if (cached) return serveCachedFile(request, response, store, cached, "stale-range-error");
throw gatewayError("map_upstream_range_unavailable", upstream.status || 502);
}
copyUpstreamHeaders(response, upstream.headers);
response.setHeader("x-nodedc-map-cache", "range-pass-through");
response.writeHead(upstream.status);
await pipeline(Readable.fromWeb(upstream.body), response);
}
async function proxyUncachedRequest(request, response, target, cacheState) {
const upstream = await fetchWithTimeout(target, {
headers: {
accept: String(request.headers.accept || "application/json, application/octet-stream, image/*, */*;q=0.5"),
...(request.headers.range ? { range: String(request.headers.range) } : {}),
},
});
if (!upstream.ok || !upstream.body) throw gatewayError("map_upstream_unavailable", upstream.status || 502);
copyUpstreamHeaders(response, upstream.headers);
response.setHeader("x-nodedc-map-cache", cacheState);
response.writeHead(upstream.status);
if (request.method === "HEAD") return response.end();
await pipeline(Readable.fromWeb(upstream.body), response);
}
async function serveCachedFile(request, response, store, entry, state) {
const filePath = join(store.dir, entry.file);
const info = await stat(filePath);
const range = parseRange(request.headers.range, info.size);
response.setHeader("content-type", entry.contentType);
response.setHeader("accept-ranges", "bytes");
response.setHeader("cache-control", "public, max-age=60");
response.setHeader("x-nodedc-map-cache", state);
response.setHeader("x-nodedc-map-cache-age", String(Math.max(0, Math.round((Date.now() - entry.savedAt) / 1000))));
if (store.mutable) {
entry.lastAccessAt = Date.now();
store.index.entries[entry.key] = entry;
void writeCacheIndex(store);
}
if (!range) {
response.setHeader("content-length", info.size);
response.writeHead(200);
if (request.method === "HEAD") return response.end();
return pipeline(createReadStream(filePath), response);
}
response.setHeader("content-length", range.end - range.start + 1);
response.setHeader("content-range", `bytes ${range.start}-${range.end}/${info.size}`);
response.writeHead(206);
if (request.method === "HEAD") return response.end();
return pipeline(createReadStream(filePath, range), response);
}
function parseRange(header, size) {
if (!header) return null;
const match = String(header).match(/^bytes=(\d*)-(\d*)$/);
if (!match) return null;
const start = match[1] ? Number.parseInt(match[1], 10) : 0;
const end = match[2] ? Number.parseInt(match[2], 10) : size - 1;
if (!Number.isInteger(start) || !Number.isInteger(end) || start < 0 || end < start || end >= size) return null;
return { start, end };
}
async function getCachedEntry(store, cacheKey) {
const entry = store.index.entries[cacheKey];
if (!entry) return null;
try {
const info = await stat(join(store.dir, entry.file));
if (!info.isFile()) throw new Error("not_a_file");
return entry;
} catch {
if (store.mutable) {
delete store.index.entries[cacheKey];
await writeCacheIndex(store);
}
return null;
}
}
function createCacheStore(name, dir, mutable) {
return {
name,
dir,
mutable,
objectsDir: join(dir, "objects"),
ionEndpointsDir: join(dir, "ion-endpoints"),
indexPath: join(dir, "index.json"),
index: { version: 1, entries: {} },
pendingIndexWrite: Promise.resolve(),
};
}
async function initialiseCacheStore(store) {
if (store.mutable) {
await mkdir(store.objectsDir, { recursive: true });
await mkdir(store.ionEndpointsDir, { recursive: true });
}
store.index = await readCacheIndex(store);
}
async function readCacheIndex(store) {
try {
const parsed = JSON.parse(await readFile(store.indexPath, "utf8"));
if (parsed?.version === 1 && parsed.entries && typeof parsed.entries === "object") return parsed;
} catch (error) {
if (error?.code !== "ENOENT") console.warn("Map cache index ignored: invalid file");
}
return { version: 1, entries: {} };
}
function ionEndpointPath(assetId) {
return join(liveCache.ionEndpointsDir, `${assetId}.json`);
}
async function readIonEndpointCache(assetId) {
try {
const value = JSON.parse(await readFile(ionEndpointPath(assetId), "utf8"));
if (value?.assetId === assetId && ((value?.url && value?.accessToken) || (value?.externalType === "BING" && value?.options?.url && value?.options?.key))) return value;
} catch (error) {
if (error?.code !== "ENOENT") console.warn("Ion endpoint cache ignored: invalid file");
}
return null;
}
async function writeIonEndpointCache(endpoint) {
const target = ionEndpointPath(endpoint.assetId);
const temp = `${target}.${process.pid}.tmp`;
await writeFile(temp, `${JSON.stringify(endpoint)}\n`, { encoding: "utf8", mode: 0o600 });
await rename(temp, target);
}
function writeCacheIndex(store) {
if (!store.mutable) return Promise.resolve();
store.pendingIndexWrite = store.pendingIndexWrite.then(async () => {
const tempPath = `${store.indexPath}.${process.pid}.tmp`;
await writeFile(tempPath, `${JSON.stringify(store.index)}\n`, "utf8");
await rename(tempPath, store.indexPath);
}).catch((error) => console.warn("Map cache index write failed", error instanceof Error ? error.message : "unknown"));
return store.pendingIndexWrite;
}
async function evictCache(store) {
let entries = Object.values(store.index.entries);
let size = entries.reduce((sum, entry) => sum + Number(entry.bytes || 0), 0);
if (size <= config.maxCacheBytes) return;
entries = entries.sort((left, right) => Number(left.lastAccessAt || 0) - Number(right.lastAccessAt || 0));
for (const entry of entries) {
if (size <= config.maxCacheBytes) break;
await rm(join(store.dir, entry.file), { force: true });
delete store.index.entries[entry.key];
size -= Number(entry.bytes || 0);
}
}
async function cacheStats(store) {
const entries = Object.values(store.index.entries);
const bytes = entries.reduce((sum, entry) => sum + Number(entry.bytes || 0), 0);
return { name: store.name, mode: store.mutable ? config.mode : "readonly", entries: entries.length, bytes, maxBytes: store.mutable ? config.maxCacheBytes : null, persistent: true };
}
async function fetchWithTimeout(target, options = {}) {
const controller = new AbortController();
const timeout = setTimeout(() => controller.abort(), config.upstreamTimeoutMs);
try { return await fetch(target, { ...options, signal: controller.signal, redirect: "follow" }); }
catch (error) { throw gatewayError(error?.name === "AbortError" ? "map_upstream_timeout" : "map_upstream_fetch_failed", 502); }
finally { clearTimeout(timeout); }
}
function copyUpstreamHeaders(response, headers) {
for (const name of ["content-type", "content-length", "content-range", "accept-ranges", "etag", "last-modified"]) {
const value = headers.get(name);
if (value) response.setHeader(name, value);
}
}
function writeJson(response, status, value) {
response.writeHead(status, { "content-type": "application/json; charset=utf-8", "cache-control": "no-store" });
response.end(JSON.stringify(value));
}
function gatewayError(message, statusCode) {
const error = new Error(message);
error.statusCode = statusCode;
return error;
}