feat(map): add persistent gateway and ontology package
This commit is contained in:
@@ -0,0 +1,6 @@
|
||||
node_modules/
|
||||
npm-debug.log
|
||||
.env
|
||||
.env.*
|
||||
data/
|
||||
volumes/
|
||||
@@ -0,0 +1,20 @@
|
||||
FROM node:20-alpine AS runner
|
||||
|
||||
ENV NODE_ENV=production
|
||||
ENV PORT=18103
|
||||
ENV MAP_CACHE_DIR=/var/lib/nodedc-map-cache
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
RUN apk add --no-cache su-exec
|
||||
|
||||
COPY package.json ./
|
||||
COPY src ./src
|
||||
COPY docker-entrypoint.sh /usr/local/bin/nodedc-map-gateway
|
||||
|
||||
RUN chmod +x /usr/local/bin/nodedc-map-gateway
|
||||
|
||||
EXPOSE 18103
|
||||
|
||||
ENTRYPOINT ["/usr/local/bin/nodedc-map-gateway"]
|
||||
CMD ["node", "src/server.mjs"]
|
||||
@@ -0,0 +1,87 @@
|
||||
# NODE.DC Map Gateway
|
||||
|
||||
`map-gateway` — единый платформенный boundary для map providers. Он не является частью Engine и не принадлежит отдельному Module Studio application.
|
||||
|
||||
## Что делает
|
||||
|
||||
- хранит master `CESIUM_ION_TOKEN` только в process environment;
|
||||
- выдаёт браузеру только asset-scoped endpoint token для allowlisted ion assets;
|
||||
- проксирует и кэширует tile/3D Tiles/terrain resources через `GET /api/map/cache?url=<https-url>`;
|
||||
- может отдать перенесённый локальный Engine snapshot через `MAP_GATEWAY_LEGACY_CACHE_HOSTS`, но никогда не догружает для него cache miss из сети;
|
||||
- работает с persistent volume, а не с Git или Docker image layer;
|
||||
- при upstream outage отдаёт ранее сохранённый cache object; offline режим включается только для providers, явно разрешённых их лицензией;
|
||||
- защищает proxy allowlist-ом upstream hosts, HTTPS-only правилом, лимитами размера и token-free cache key;
|
||||
- ведёт лёгкий persistent cache index для LRU eviction и health/stats.
|
||||
|
||||
## API
|
||||
|
||||
```text
|
||||
GET /healthz
|
||||
GET /api/map/ion/assets/:assetId/endpoint
|
||||
GET|HEAD /api/map/cache?url=<encoded-upstream-url>
|
||||
```
|
||||
|
||||
`/api/map/ion/assets/:assetId/endpoint` разрешает только `CESIUM_ION_ASSET_ALLOWLIST`. Ответ содержит runtime asset token, а не master token. Browser использует endpoint вместе с Cesium `DefaultProxy`, направляющим resource requests в `/api/map/cache`.
|
||||
|
||||
Ion endpoint metadata также сохраняется в persistent volume с файловыми правами `0600`. Это нужно для cold start в offline: Cesium получает сохранённый asset URL и asset-scoped token, а proxy отвечает только из ранее записанного cache. В metadata никогда не записывается master token.
|
||||
|
||||
## Cache modes
|
||||
|
||||
- `readwrite` — свежий cache hit отдаётся сразу; stale object обновляется online; при upstream failure возвращается stale copy;
|
||||
- `readonly` — использует cache, но не записывает новый;
|
||||
- `offline` — никогда не идёт наружу; cache miss возвращает `504`.
|
||||
|
||||
`offline` дополнительно требует, чтобы hostname был явно указан в `MAP_GATEWAY_OFFLINE_PROVIDER_ALLOWLIST`. По умолчанию список пуст: Gateway не превращает Cesium ion или public OSM tile service в offline distribution. Это осознанная provider policy, а не техническая ошибка cache.
|
||||
|
||||
Range requests пока transparently проксируются и не записываются как partial cache object. Полные GET responses кэшируются. Это безопасная начальная граница для terrain/3D Tiles; отдельным следующим шагом добавляется range-aware chunk cache после замеров реальных GOS/3D Tiles assets.
|
||||
|
||||
## Persistent storage
|
||||
|
||||
Platform Compose использует два внешних named volume: mutable `nodedc-platform_map-live-tile-cache` монтируется в `/var/lib/nodedc-map-live-cache`, read-only snapshot `nodedc-platform_map-offline-snapshot` — в `/var/lib/nodedc-map-offline-snapshot`. Их содержимое намеренно исключено из Git. Внешний volume не удаляется через `docker compose down -v`; очистка требует явного `docker volume rm` при остановленном Gateway. При необходимости offline seed доставляется отдельным export/import artifact, а не коммитом runtime cache.
|
||||
|
||||
Кэш не способен создать участок карты, который никогда не был скачан. Для целевых офлайн-регионов нужен отдельный licensed dataset/provider и его `seed/prefetch` job: он заранее проходит разрешённые assets, zoom-диапазоны и географические области, записывает их в тот же persistent volume и формирует проверяемый export/import artifact. Эта задача не смешивается с runtime gateway и не помещает binary tiles в историю репозитория.
|
||||
|
||||
### Legacy Engine cache: локальный перенос без изменения Engine
|
||||
|
||||
В локальной Studio-песочнице существующий Engine cache можно скопировать в persistent volume как неизменяемый входной snapshot. Исходная директория Engine при этом только читается: import job не удаляет, не переименовывает и не изменяет ни одного её файла.
|
||||
|
||||
```bash
|
||||
npm run audit:engine-cache -- /absolute/path/to/nodedc-data/api/cesium/manifest.json
|
||||
```
|
||||
|
||||
Команда audit выдаёт агрегированный отчёт: hosts, число объектов, declared/on-disk размер и отсутствующие файлы. Сам перенос выполняется отдельно, после остановки gateway на время записи index:
|
||||
|
||||
```bash
|
||||
npm run import:engine-cache -- /absolute/path/to/nodedc-data/api/cesium/manifest.json /persistent/map-tile-cache
|
||||
```
|
||||
|
||||
По умолчанию действует режим `no-overwrite`: если объект уже присутствует в новом cache, он остаётся без изменений. `--dry-run` выводит план без копирования, `--refresh` разрешает осознанно заменить уже импортированный объект. Legacy `http` URLs нормализуются до `https`, чтобы совпасть с HTTPS-only Gateway; сами файлы, content-type, etag, timestamp и provenance сохраняются в новом index/import report.
|
||||
|
||||
Это sandbox-механизм переноса уже существующей локальной реализации. Для внешней коммерческой публикации потребуется отдельная provider/offline policy; она не блокирует текущую локальную миграцию, но и не должна смешиваться с ней. Runtime cache по-прежнему не попадает в Git или Docker image layer.
|
||||
|
||||
## Security and access
|
||||
|
||||
В local dev допускается `MAP_GATEWAY_ALLOW_ANONYMOUS=true`. В production он должен быть `false`; Caddy/Auth BFF проверяет Authentik session и передаёт очищенный trusted subject header в private platform network. Нельзя открывать gateway напрямую наружу и нельзя принимать user-provided upstream hosts вне allowlist.
|
||||
|
||||
## Required environment
|
||||
|
||||
```text
|
||||
CESIUM_ION_TOKEN=
|
||||
CESIUM_ION_ASSET_ALLOWLIST=1,96188
|
||||
MAP_GATEWAY_UPSTREAM_ALLOWLIST=api.cesium.com,assets.ion.cesium.com,tile.openstreetmap.org
|
||||
MAP_GATEWAY_LEGACY_CACHE_HOSTS=ecn.t0.tiles.virtualearth.net,ecn.t1.tiles.virtualearth.net,ecn.t2.tiles.virtualearth.net,ecn.t3.tiles.virtualearth.net
|
||||
MAP_GATEWAY_OFFLINE_PROVIDER_ALLOWLIST=
|
||||
MAP_CACHE_DIR=/var/lib/nodedc-map-cache
|
||||
MAP_CACHE_MODE=readwrite
|
||||
```
|
||||
# Cache topology
|
||||
|
||||
`MAP_CACHE_DIR` is the mutable **live** cache. In the Platform compose profile it
|
||||
is mounted as `map-live-tile-cache` and is the only store that can receive new
|
||||
upstream objects.
|
||||
|
||||
`MAP_OFFLINE_SNAPSHOT_DIR` is optional and mounted read-only as
|
||||
`map-offline-snapshot`. It is populated only by the controlled Engine cache
|
||||
import. Requests marked with the `offline` cache profile are served exclusively
|
||||
from this snapshot; a missing tile returns a cache miss and never reaches an
|
||||
upstream provider.
|
||||
@@ -0,0 +1,11 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
|
||||
# Docker creates a fresh named volume as root. Prepare only the mutable live
|
||||
# cache before dropping privileges; the imported offline snapshot stays read-only.
|
||||
if [ -d "${MAP_CACHE_DIR:-}" ]; then
|
||||
mkdir -p "$MAP_CACHE_DIR"
|
||||
chown -R node:node "$MAP_CACHE_DIR"
|
||||
fi
|
||||
|
||||
exec su-exec node "$@"
|
||||
@@ -0,0 +1,15 @@
|
||||
{
|
||||
"name": "@nodedc/map-gateway",
|
||||
"version": "0.1.0",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"start": "node src/server.mjs",
|
||||
"dev": "node --watch src/server.mjs",
|
||||
"audit:engine-cache": "node scripts/audit-engine-cache.mjs",
|
||||
"import:engine-cache": "node scripts/import-engine-cache.mjs"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=20"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,91 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
import { access, readFile, stat } from "node:fs/promises";
|
||||
import { dirname, isAbsolute, join, resolve } from "node:path";
|
||||
|
||||
const manifestPath = resolve(process.argv[2] || "");
|
||||
if (!process.argv[2]) {
|
||||
console.error("Usage: npm run audit:engine-cache -- /absolute/path/to/cesium/manifest.json");
|
||||
process.exitCode = 1;
|
||||
} else {
|
||||
await audit(manifestPath);
|
||||
}
|
||||
|
||||
async function audit(path) {
|
||||
let manifest;
|
||||
try {
|
||||
manifest = JSON.parse(await readFile(path, "utf8"));
|
||||
} catch (error) {
|
||||
throw new Error(`Cannot read legacy cache manifest: ${error instanceof Error ? error.message : "unknown error"}`);
|
||||
}
|
||||
|
||||
const root = dirname(path);
|
||||
const entries = Object.entries(manifest?.entries || {});
|
||||
const providers = new Map();
|
||||
let bytesFromManifest = 0;
|
||||
let presentFiles = 0;
|
||||
let missingFiles = 0;
|
||||
let bytesOnDisk = 0;
|
||||
|
||||
for (const [rawUrl, entry] of entries) {
|
||||
let hostname = "invalid-url";
|
||||
try { hostname = new URL(rawUrl).hostname.toLowerCase() || "invalid-url"; } catch {}
|
||||
const provider = providers.get(hostname) || { entries: 0, bytes: 0, presentFiles: 0, missingFiles: 0 };
|
||||
const entryBytes = Math.max(0, Number(entry?.size || 0));
|
||||
provider.entries += 1;
|
||||
provider.bytes += entryBytes;
|
||||
bytesFromManifest += entryBytes;
|
||||
|
||||
const relativeFile = String(entry?.path || "");
|
||||
const candidate = resolve(root, relativeFile);
|
||||
if (!relativeFile || !candidate.startsWith(`${root}/`) || isAbsolute(relativeFile) || !(await fileExists(candidate))) {
|
||||
provider.missingFiles += 1;
|
||||
missingFiles += 1;
|
||||
providers.set(hostname, provider);
|
||||
continue;
|
||||
}
|
||||
|
||||
const fileInfo = await stat(candidate);
|
||||
presentFiles += 1;
|
||||
provider.presentFiles += 1;
|
||||
bytesOnDisk += fileInfo.size;
|
||||
providers.set(hostname, provider);
|
||||
}
|
||||
|
||||
const report = {
|
||||
kind: "nodedc-legacy-cesium-cache-audit",
|
||||
generatedAt: new Date().toISOString(),
|
||||
manifestPath: path,
|
||||
manifestVersion: manifest?.version ?? null,
|
||||
entries: entries.length,
|
||||
declaredBytes: bytesFromManifest,
|
||||
declaredMegabytes: toMegabytes(bytesFromManifest),
|
||||
presentFiles,
|
||||
missingFiles,
|
||||
bytesOnDisk,
|
||||
megabytesOnDisk: toMegabytes(bytesOnDisk),
|
||||
providers: [...providers.entries()]
|
||||
.map(([hostname, value]) => ({ hostname, ...value, megabytes: toMegabytes(value.bytes) }))
|
||||
.sort((left, right) => right.bytes - left.bytes),
|
||||
migrationPolicy: {
|
||||
automaticImport: false,
|
||||
reason: "Audit only. Import requires an explicit licence/provider decision for every source host and must preserve required attribution.",
|
||||
target: "Map Gateway persistent volume or approved object storage; never Git history or a Docker image layer."
|
||||
}
|
||||
};
|
||||
|
||||
process.stdout.write(`${JSON.stringify(report, null, 2)}\n`);
|
||||
}
|
||||
|
||||
async function fileExists(path) {
|
||||
try {
|
||||
await access(path);
|
||||
return true;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
function toMegabytes(bytes) {
|
||||
return Number((bytes / 1024 / 1024).toFixed(2));
|
||||
}
|
||||
@@ -0,0 +1,173 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
import { createHash } from "node:crypto";
|
||||
import { copyFile, mkdir, readFile, rename, stat, writeFile } from "node:fs/promises";
|
||||
import { dirname, isAbsolute, join, resolve } from "node:path";
|
||||
|
||||
const args = new Set(process.argv.slice(2));
|
||||
const positional = process.argv.slice(2).filter((value) => !value.startsWith("--"));
|
||||
const [manifestArgument, cacheArgument] = positional;
|
||||
const dryRun = args.has("--dry-run");
|
||||
const refresh = args.has("--refresh");
|
||||
|
||||
if (!manifestArgument || !cacheArgument) {
|
||||
console.error("Usage: npm run import:engine-cache -- /absolute/path/to/manifest.json /persistent/map-cache [--dry-run] [--refresh]");
|
||||
process.exitCode = 1;
|
||||
} else {
|
||||
await importEngineCache(resolve(manifestArgument), resolve(cacheArgument), { dryRun, refresh });
|
||||
}
|
||||
|
||||
async function importEngineCache(manifestPath, cacheDir, options) {
|
||||
const manifest = JSON.parse(await readFile(manifestPath, "utf8"));
|
||||
const sourceRoot = dirname(manifestPath);
|
||||
const sourceEntries = Object.entries(manifest?.entries || {});
|
||||
const indexPath = join(cacheDir, "index.json");
|
||||
const index = await readIndex(indexPath);
|
||||
const reindexed = reindexLegacyEntries(index);
|
||||
const report = {
|
||||
kind: "nodedc-engine-cache-import",
|
||||
importedAt: new Date().toISOString(),
|
||||
sourceManifest: manifestPath,
|
||||
sourceManifestVersion: manifest?.version ?? null,
|
||||
mode: options.dryRun ? "dry-run" : options.refresh ? "refresh" : "no-overwrite",
|
||||
totalEntries: sourceEntries.length,
|
||||
imported: 0,
|
||||
alreadyPresent: 0,
|
||||
missingSourceFiles: 0,
|
||||
invalidEntries: 0,
|
||||
reindexed,
|
||||
bytesImported: 0,
|
||||
providers: {},
|
||||
};
|
||||
|
||||
if (!options.dryRun) await mkdir(join(cacheDir, "objects"), { recursive: true });
|
||||
|
||||
for (const [rawUrl, source] of sourceEntries) {
|
||||
const normalized = normalizeLegacyUrl(rawUrl);
|
||||
const relativeSource = String(source?.path || "");
|
||||
const sourcePath = resolve(sourceRoot, relativeSource);
|
||||
if (!normalized || !relativeSource || isAbsolute(relativeSource) || !sourcePath.startsWith(`${sourceRoot}/`)) {
|
||||
report.invalidEntries += 1;
|
||||
continue;
|
||||
}
|
||||
|
||||
let sourceInfo;
|
||||
try {
|
||||
sourceInfo = await stat(sourcePath);
|
||||
if (!sourceInfo.isFile()) throw new Error("not_file");
|
||||
} catch {
|
||||
report.missingSourceFiles += 1;
|
||||
continue;
|
||||
}
|
||||
|
||||
const key = createHash("sha256").update(canonicalCacheUrl(normalized)).digest("hex");
|
||||
const targetFile = join("objects", key.slice(0, 2), `${key}.bin`);
|
||||
const targetPath = join(cacheDir, targetFile);
|
||||
const existsInIndex = Boolean(index.entries[key]);
|
||||
const existsOnDisk = await fileExists(targetPath);
|
||||
if (!options.refresh && (existsInIndex || existsOnDisk)) {
|
||||
report.alreadyPresent += 1;
|
||||
continue;
|
||||
}
|
||||
|
||||
const hostname = new URL(normalized).hostname.toLowerCase();
|
||||
report.providers[hostname] = (report.providers[hostname] || 0) + 1;
|
||||
const savedAt = Number(source?.savedAt || Date.now());
|
||||
const entry = {
|
||||
key,
|
||||
file: targetFile,
|
||||
bytes: sourceInfo.size,
|
||||
contentType: String(source?.contentType || "application/octet-stream"),
|
||||
etag: String(source?.etag || ""),
|
||||
savedAt,
|
||||
lastAccessAt: Date.now(),
|
||||
// Imported records are deliberately treated as fresh in the sandbox.
|
||||
// A refresh is always explicit through nodedc_cache_refresh=1.
|
||||
expiresAt: Date.now() + 3650 * 24 * 60 * 60 * 1000,
|
||||
source: "engine-legacy-cache",
|
||||
sourceUrl: normalized,
|
||||
};
|
||||
|
||||
if (!options.dryRun) {
|
||||
await mkdir(dirname(targetPath), { recursive: true });
|
||||
await copyFile(sourcePath, targetPath);
|
||||
index.entries[key] = entry;
|
||||
}
|
||||
report.imported += 1;
|
||||
report.bytesImported += sourceInfo.size;
|
||||
}
|
||||
|
||||
if (!options.dryRun) {
|
||||
await writeJsonAtomically(indexPath, index);
|
||||
const importsDir = join(cacheDir, "imports");
|
||||
await mkdir(importsDir, { recursive: true });
|
||||
await writeJsonAtomically(join(importsDir, "engine-legacy-cache-import.json"), report);
|
||||
}
|
||||
process.stdout.write(`${JSON.stringify(report, null, 2)}\n`);
|
||||
}
|
||||
|
||||
function normalizeLegacyUrl(rawUrl) {
|
||||
try {
|
||||
const target = new URL(rawUrl);
|
||||
// The Engine cache predates the HTTPS-only Gateway. The same tile hosts
|
||||
// support HTTPS, so normalize only the protocol; path and provider stay intact.
|
||||
if (target.protocol === "http:") target.protocol = "https:";
|
||||
if (target.protocol !== "https:") return null;
|
||||
return target;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
function canonicalCacheUrl(target) {
|
||||
const url = new URL(target.toString());
|
||||
if (/^ecn\.t[0-3]\.tiles\.virtualearth\.net$/i.test(url.hostname)) {
|
||||
url.hostname = "ecn.tiles.virtualearth.net";
|
||||
}
|
||||
for (const key of [...url.searchParams.keys()]) {
|
||||
if (["access_token", "accesstoken", "iontoken", "token", "key", "apikey", "api_key", "signature", "sig"].includes(key.toLowerCase())) {
|
||||
url.searchParams.delete(key);
|
||||
}
|
||||
}
|
||||
url.searchParams.sort();
|
||||
return url.toString();
|
||||
}
|
||||
|
||||
function reindexLegacyEntries(index) {
|
||||
let reindexed = 0;
|
||||
for (const [currentKey, entry] of Object.entries(index.entries)) {
|
||||
if (entry?.source !== "engine-legacy-cache" || !entry?.sourceUrl) continue;
|
||||
let source;
|
||||
try { source = new URL(String(entry.sourceUrl)); } catch { continue; }
|
||||
const nextKey = createHash("sha256").update(canonicalCacheUrl(source)).digest("hex");
|
||||
if (nextKey === currentKey || index.entries[nextKey]) continue;
|
||||
index.entries[nextKey] = { ...entry, key: nextKey };
|
||||
delete index.entries[currentKey];
|
||||
reindexed += 1;
|
||||
}
|
||||
return reindexed;
|
||||
}
|
||||
|
||||
async function readIndex(indexPath) {
|
||||
try {
|
||||
const parsed = JSON.parse(await readFile(indexPath, "utf8"));
|
||||
if (parsed?.version === 1 && parsed.entries && typeof parsed.entries === "object") return parsed;
|
||||
} catch (error) {
|
||||
if (error?.code !== "ENOENT") throw error;
|
||||
}
|
||||
return { version: 1, entries: {} };
|
||||
}
|
||||
|
||||
async function fileExists(path) {
|
||||
try {
|
||||
return (await stat(path)).isFile();
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
async function writeJsonAtomically(path, value) {
|
||||
const temp = `${path}.${process.pid}.tmp`;
|
||||
await writeFile(temp, `${JSON.stringify(value)}\n`, "utf8");
|
||||
await rename(temp, path);
|
||||
}
|
||||
@@ -0,0 +1,533 @@
|
||||
import { createHash } from "node:crypto";
|
||||
import { createReadStream, createWriteStream } from "node:fs";
|
||||
import { mkdir, readFile, rename, rm, stat, writeFile } from "node:fs/promises";
|
||||
import { createServer } from "node:http";
|
||||
import { dirname, join } from "node:path";
|
||||
import { Readable, Transform } from "node:stream";
|
||||
import { pipeline } from "node:stream/promises";
|
||||
|
||||
const config = readConfig();
|
||||
const liveCache = createCacheStore("live", config.cacheDir, true);
|
||||
const offlineSnapshot = config.offlineSnapshotDir ? createCacheStore("offline-snapshot", config.offlineSnapshotDir, false) : null;
|
||||
const inflightWrites = new Map();
|
||||
|
||||
await initialiseCacheStore(liveCache);
|
||||
if (offlineSnapshot) await initialiseCacheStore(offlineSnapshot);
|
||||
|
||||
const server = createServer(async (request, response) => {
|
||||
applyCors(request, response);
|
||||
if (request.method === "OPTIONS") return response.writeHead(204).end();
|
||||
|
||||
try {
|
||||
const requestUrl = new URL(request.url || "/", `http://${request.headers.host || "127.0.0.1"}`);
|
||||
if (!isAllowedRequest(request)) return writeJson(response, 401, { ok: false, error: "map_gateway_auth_required" });
|
||||
|
||||
if (requestUrl.pathname === "/healthz" && request.method === "GET") {
|
||||
return writeJson(response, 200, {
|
||||
ok: true,
|
||||
service: "nodedc-map-gateway",
|
||||
cache: await cacheStats(liveCache),
|
||||
offlineSnapshot: offlineSnapshot ? await cacheStats(offlineSnapshot) : null,
|
||||
ionConfigured: Boolean(config.cesiumIonToken),
|
||||
assetAllowlist: [...config.assetAllowlist].map(Number).sort((left, right) => left - right),
|
||||
anonymousAccess: config.allowAnonymous,
|
||||
});
|
||||
}
|
||||
|
||||
const ionAssetMatch = requestUrl.pathname.match(/^\/api\/map\/ion\/assets\/(\d+)\/endpoint$/);
|
||||
if (ionAssetMatch && request.method === "GET") {
|
||||
return await serveIonEndpoint(response, ionAssetMatch[1]);
|
||||
}
|
||||
|
||||
if (requestUrl.pathname === "/api/map/cache" && ["GET", "HEAD"].includes(request.method || "")) {
|
||||
return await serveCachedUpstream(request, response, requestUrl);
|
||||
}
|
||||
|
||||
return writeJson(response, 404, { ok: false, error: "map_gateway_route_not_found" });
|
||||
} catch (error) {
|
||||
// A client can cancel a streamed cache response after its headers have
|
||||
// already been sent. Do not try to turn that into JSON: Node would throw
|
||||
// ERR_HTTP_HEADERS_SENT and take the entire gateway process down.
|
||||
if (response.headersSent || response.writableEnded) {
|
||||
if (!response.writableEnded) response.destroy();
|
||||
return;
|
||||
}
|
||||
const status = Number(error?.statusCode || 500);
|
||||
return writeJson(response, Number.isInteger(status) && status >= 400 && status < 600 ? status : 500, {
|
||||
ok: false,
|
||||
error: error instanceof Error ? error.message : "map_gateway_error",
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
server.listen(config.port, "0.0.0.0", () => {
|
||||
console.log(`NODE.DC Map Gateway listening on http://0.0.0.0:${config.port}`);
|
||||
console.log(`Live map cache: ${config.cacheDir} (${config.mode}, max ${Math.round(config.maxCacheBytes / 1024 / 1024)} MB)`);
|
||||
if (offlineSnapshot) console.log(`Offline map snapshot: ${config.offlineSnapshotDir} (read-only)`);
|
||||
});
|
||||
|
||||
process.on("SIGTERM", () => server.close());
|
||||
process.on("SIGINT", () => server.close());
|
||||
|
||||
function readConfig() {
|
||||
const mode = String(process.env.MAP_CACHE_MODE || "readwrite").trim().toLowerCase();
|
||||
if (!["readwrite", "readonly", "offline"].includes(mode)) throw new Error("invalid_map_cache_mode");
|
||||
const allowlist = parseList(process.env.MAP_GATEWAY_UPSTREAM_ALLOWLIST || "api.cesium.com,assets.ion.cesium.com,tile.openstreetmap.org,dev.virtualearth.net,ecn.t0.tiles.virtualearth.net,ecn.t1.tiles.virtualearth.net,ecn.t2.tiles.virtualearth.net,ecn.t3.tiles.virtualearth.net");
|
||||
return {
|
||||
port: parsePositiveInt(process.env.PORT, 18103),
|
||||
cacheDir: String(process.env.MAP_CACHE_DIR || "/var/lib/nodedc-map-cache").trim(),
|
||||
offlineSnapshotDir: String(process.env.MAP_OFFLINE_SNAPSHOT_DIR || "").trim(),
|
||||
mode,
|
||||
maxCacheBytes: parsePositiveInt(process.env.MAP_CACHE_MAX_MB, 20480) * 1024 * 1024,
|
||||
maxObjectBytes: parsePositiveInt(process.env.MAP_CACHE_MAX_OBJECT_MB, 128) * 1024 * 1024,
|
||||
defaultTtlMs: parsePositiveInt(process.env.MAP_CACHE_DEFAULT_TTL_SECONDS, 604800) * 1000,
|
||||
upstreamTimeoutMs: parsePositiveInt(process.env.MAP_GATEWAY_UPSTREAM_TIMEOUT_SECONDS, 30) * 1000,
|
||||
cesiumIonToken: String(process.env.CESIUM_ION_TOKEN || "").trim(),
|
||||
assetAllowlist: new Set(parseList(process.env.CESIUM_ION_ASSET_ALLOWLIST || "1,2,96188")),
|
||||
upstreamAllowlist: new Set(allowlist),
|
||||
legacyCacheHosts: new Set(parseList(process.env.MAP_GATEWAY_LEGACY_CACHE_HOSTS || "")),
|
||||
offlineProviderAllowlist: new Set(parseList(process.env.MAP_GATEWAY_OFFLINE_PROVIDER_ALLOWLIST || "")),
|
||||
corsOrigins: new Set(parseList(process.env.MAP_GATEWAY_CORS_ORIGIN || "http://127.0.0.1:3333,http://localhost:3333")),
|
||||
allowAnonymous: parseBoolean(process.env.MAP_GATEWAY_ALLOW_ANONYMOUS, process.env.NODE_ENV !== "production"),
|
||||
trustedSubjectHeader: String(process.env.MAP_GATEWAY_TRUSTED_SUBJECT_HEADER || "x-nodedc-user-id").toLowerCase(),
|
||||
};
|
||||
}
|
||||
|
||||
function parseList(value) {
|
||||
return String(value || "").split(",").map((part) => part.trim().toLowerCase()).filter(Boolean);
|
||||
}
|
||||
|
||||
function parsePositiveInt(value, fallback) {
|
||||
const parsed = Number.parseInt(String(value || ""), 10);
|
||||
return Number.isInteger(parsed) && parsed > 0 ? parsed : fallback;
|
||||
}
|
||||
|
||||
function parseBoolean(value, fallback) {
|
||||
if (value === undefined || value === "") return fallback;
|
||||
return ["1", "true", "yes", "on"].includes(String(value).trim().toLowerCase());
|
||||
}
|
||||
|
||||
function applyCors(request, response) {
|
||||
const origin = String(request.headers.origin || "");
|
||||
if (!origin || (!config.corsOrigins.has("*") && !config.corsOrigins.has(origin))) return;
|
||||
response.setHeader("access-control-allow-origin", config.corsOrigins.has("*") ? "*" : origin);
|
||||
response.setHeader("access-control-allow-methods", "GET, HEAD, OPTIONS");
|
||||
response.setHeader("access-control-allow-headers", "Range, Content-Type");
|
||||
response.setHeader("vary", "Origin");
|
||||
}
|
||||
|
||||
function isAllowedRequest(request) {
|
||||
return config.allowAnonymous || Boolean(request.headers[config.trustedSubjectHeader]);
|
||||
}
|
||||
|
||||
async function serveIonEndpoint(response, assetId) {
|
||||
if (!config.assetAllowlist.has(assetId)) return writeJson(response, 403, { ok: false, error: "cesium_asset_not_allowed" });
|
||||
const cached = await readIonEndpointCache(assetId);
|
||||
if (config.mode === "offline") {
|
||||
if (!cached) return writeJson(response, 504, { ok: false, error: "cesium_ion_offline_endpoint_miss" });
|
||||
if (!isOfflineProviderAllowed(cached.url)) return writeJson(response, 409, { ok: false, error: "map_provider_offline_not_permitted" });
|
||||
return writeJson(response, 200, { ok: true, ...cached, cache: "offline-endpoint-hit" });
|
||||
}
|
||||
if (!config.cesiumIonToken) {
|
||||
if (cached) return writeJson(response, 200, { ok: true, ...cached, cache: "cached-endpoint-no-master-token" });
|
||||
return writeJson(response, 503, { ok: false, error: "cesium_ion_not_configured" });
|
||||
}
|
||||
|
||||
try {
|
||||
const endpoint = await fetchWithTimeout(`https://api.cesium.com/v1/assets/${assetId}/endpoint`, {
|
||||
headers: { authorization: `Bearer ${config.cesiumIonToken}` },
|
||||
});
|
||||
if (!endpoint.ok) throw gatewayError("cesium_ion_endpoint_unavailable", endpoint.status || 502);
|
||||
const body = await endpoint.json();
|
||||
const next = body.type === "IMAGERY" && body.externalType === "BING"
|
||||
? {
|
||||
assetId,
|
||||
type: body.type,
|
||||
externalType: "BING",
|
||||
options: {
|
||||
url: validateUpstream(String(body.options?.url || "")).toString(),
|
||||
key: String(body.options?.key || ""),
|
||||
mapStyle: String(body.options?.mapStyle || "Aerial"),
|
||||
},
|
||||
attributions: Array.isArray(body.attributions) ? body.attributions : [],
|
||||
savedAt: Date.now(),
|
||||
}
|
||||
: {
|
||||
assetId,
|
||||
type: body.type,
|
||||
url: validateUpstream(String(body.url || "")).toString(),
|
||||
accessToken: String(body.accessToken || ""),
|
||||
attributions: Array.isArray(body.attributions) ? body.attributions : [],
|
||||
savedAt: Date.now(),
|
||||
};
|
||||
if (!(next.externalType === "BING" ? next.options?.key : next.accessToken)) throw gatewayError("cesium_ion_endpoint_invalid", 502);
|
||||
if (config.mode === "readwrite") await writeIonEndpointCache(next);
|
||||
return writeJson(response, 200, { ok: true, ...next, cache: "ion-endpoint-online" });
|
||||
} catch (error) {
|
||||
if (cached) return writeJson(response, 200, { ok: true, ...cached, cache: "stale-ion-endpoint" });
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
async function serveCachedUpstream(request, response, requestUrl) {
|
||||
const rawTarget = requestUrl.searchParams.get("url");
|
||||
if (!rawTarget) return writeJson(response, 400, { ok: false, error: "map_cache_url_required" });
|
||||
const target = validateUpstream(rawTarget);
|
||||
const cacheProfile = String(target.searchParams.get("nodedc_cache_profile") || "live").toLowerCase();
|
||||
target.searchParams.delete("nodedc_cache_profile");
|
||||
if (!new Set(["live", "offline"]).has(cacheProfile)) return writeJson(response, 400, { ok: false, error: "map_cache_profile_invalid" });
|
||||
const store = cacheProfile === "offline" ? offlineSnapshot : liveCache;
|
||||
if (!store) return writeJson(response, 409, { ok: false, error: "map_offline_snapshot_not_configured" });
|
||||
const cacheMode = String(target.searchParams.get("nodedc_cache_mode") || "hybrid").toLowerCase();
|
||||
target.searchParams.delete("nodedc_cache_mode");
|
||||
if (!new Set(["hybrid", "passthrough"]).has(cacheMode)) return writeJson(response, 400, { ok: false, error: "map_cache_mode_invalid" });
|
||||
const isLegacyCacheHost = config.legacyCacheHosts.has(target.hostname.toLowerCase());
|
||||
const forceRefresh = target.searchParams.get("nodedc_cache_refresh") === "1";
|
||||
target.searchParams.delete("nodedc_cache_refresh");
|
||||
const cacheKey = createHash("sha256").update(canonicalCacheUrl(target)).digest("hex");
|
||||
|
||||
// "Live" without cache is still routed through Gateway — it is never a
|
||||
// browser-side direct request — but it neither reads from nor writes to the
|
||||
// persistent store. The immutable offline profile intentionally has no such
|
||||
// bypass mode.
|
||||
if (cacheMode === "passthrough" && cacheProfile === "live") {
|
||||
return proxyUncachedRequest(request, response, target, "live-pass-through");
|
||||
}
|
||||
const cached = await getCachedEntry(store, cacheKey);
|
||||
const fresh = cached && cached.expiresAt > Date.now();
|
||||
|
||||
if (cached && (!forceRefresh || !store.mutable) && (fresh || !store.mutable || config.mode !== "readwrite")) {
|
||||
return serveCachedFile(request, response, store, cached, fresh ? `${store.name}-hit` : `${store.name}-stale`);
|
||||
}
|
||||
|
||||
// A migrated Engine cache is a local sandbox snapshot, not an instruction to
|
||||
// fetch arbitrary historic providers when an object is missing. The profile
|
||||
// is selected by the visual adapter; the browser can never turn it into a
|
||||
// new upstream fetch by changing a setting.
|
||||
if (cacheProfile === "offline") return writeJson(response, 504, { ok: false, error: "map_offline_snapshot_miss" });
|
||||
if (isLegacyCacheHost) return writeJson(response, 504, { ok: false, error: "map_legacy_cache_miss" });
|
||||
|
||||
if (config.mode === "offline") {
|
||||
if (!isOfflineProviderAllowed(target.toString())) return writeJson(response, 409, { ok: false, error: "map_provider_offline_not_permitted" });
|
||||
return writeJson(response, 504, { ok: false, error: "map_cache_offline_miss" });
|
||||
}
|
||||
|
||||
if (config.mode === "readonly") {
|
||||
return proxyUncachedRequest(request, response, target, "pass-through-readonly");
|
||||
}
|
||||
|
||||
if (String(request.headers.range || "").trim()) {
|
||||
return proxyRangeRequest(request, response, target, cached, store);
|
||||
}
|
||||
|
||||
try {
|
||||
const entry = await fetchAndCache(target, cacheKey, store);
|
||||
return serveCachedFile(request, response, store, entry, forceRefresh ? "live-refresh-record" : "live-record");
|
||||
} catch (error) {
|
||||
if (cached) return serveCachedFile(request, response, store, cached, "live-stale-upstream-error");
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
function isOfflineProviderAllowed(rawTarget) {
|
||||
if (!config.offlineProviderAllowlist.size) return false;
|
||||
try {
|
||||
const target = new URL(rawTarget);
|
||||
return config.offlineProviderAllowlist.has(target.hostname.toLowerCase());
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
function validateUpstream(rawTarget) {
|
||||
if (rawTarget.length > 8192) throw gatewayError("map_cache_url_too_long", 400);
|
||||
let target;
|
||||
try { target = new URL(rawTarget); } catch { throw gatewayError("invalid_map_cache_url", 400); }
|
||||
const hostname = target.hostname.toLowerCase();
|
||||
|
||||
// Bing's metadata endpoint still returns HTTP tile templates. They are not
|
||||
// allowed through as HTTP: upgrade only the known Bing tile hosts before
|
||||
// the standard HTTPS and allowlist checks. This keeps the browser and the
|
||||
// Gateway on encrypted transport while making the provider contract work.
|
||||
if (target.protocol === "http:" && /^(ecn\.t[0-3]\.tiles\.virtualearth\.net|ecn\.tiles\.virtualearth\.net)$/.test(hostname)) {
|
||||
target.protocol = "https:";
|
||||
}
|
||||
if (target.protocol !== "https:") throw gatewayError("map_cache_https_required", 400);
|
||||
if (target.username || target.password || (!config.upstreamAllowlist.has(hostname) && !config.legacyCacheHosts.has(hostname))) {
|
||||
throw gatewayError("map_cache_upstream_not_allowed", 403);
|
||||
}
|
||||
return target;
|
||||
}
|
||||
|
||||
function canonicalCacheUrl(target) {
|
||||
const url = new URL(target.toString());
|
||||
// The old Engine snapshot distributes identical Bing tile semantics across
|
||||
// t0…t3. A viewer can choose a different subdomain for the same quadkey, so
|
||||
// use one logical host for the cache key while preserving the original URL
|
||||
// only as provenance in the imported index entry.
|
||||
if (/^ecn\.t[0-3]\.tiles\.virtualearth\.net$/i.test(url.hostname)) {
|
||||
url.hostname = "ecn.tiles.virtualearth.net";
|
||||
}
|
||||
for (const key of [...url.searchParams.keys()]) {
|
||||
if (["access_token", "accesstoken", "iontoken", "token", "key", "apikey", "api_key", "signature", "sig"].includes(key.toLowerCase())) {
|
||||
url.searchParams.delete(key);
|
||||
}
|
||||
}
|
||||
url.searchParams.sort();
|
||||
return url.toString();
|
||||
}
|
||||
|
||||
async function fetchAndCache(target, cacheKey, store) {
|
||||
const inflightKey = `${store.name}:${cacheKey}`;
|
||||
const existing = inflightWrites.get(inflightKey);
|
||||
if (existing) return existing;
|
||||
const task = downloadAndCache(target, cacheKey, store);
|
||||
inflightWrites.set(inflightKey, task);
|
||||
// Several Cesium requests can share one in-flight download. Keep a rejection
|
||||
// observer on the shared promise, then rethrow to each HTTP caller below.
|
||||
// A 4xx/5xx tile response must never become an unhandled rejection that stops
|
||||
// the gateway process.
|
||||
task.catch(() => undefined);
|
||||
try {
|
||||
return await task;
|
||||
} finally {
|
||||
if (inflightWrites.get(inflightKey) === task) inflightWrites.delete(inflightKey);
|
||||
}
|
||||
}
|
||||
|
||||
async function downloadAndCache(target, cacheKey, store) {
|
||||
const upstream = await fetchWithTimeout(target, { headers: { accept: "application/json, application/octet-stream, image/*, */*;q=0.5" } });
|
||||
if (!upstream.ok || !upstream.body) throw gatewayError("map_upstream_unavailable", upstream.status || 502);
|
||||
const expectedBytes = Number.parseInt(upstream.headers.get("content-length") || "", 10);
|
||||
if (Number.isFinite(expectedBytes) && expectedBytes > config.maxObjectBytes) throw gatewayError("map_cache_object_too_large", 413);
|
||||
|
||||
const relativePath = join("objects", cacheKey.slice(0, 2), `${cacheKey}.bin`);
|
||||
const filePath = join(store.dir, relativePath);
|
||||
const tempPath = `${filePath}.${process.pid}.${Date.now()}.tmp`;
|
||||
await mkdir(dirname(filePath), { recursive: true });
|
||||
let bytes = 0;
|
||||
const limit = new Transform({
|
||||
transform(chunk, _encoding, callback) {
|
||||
bytes += chunk.length;
|
||||
if (bytes > config.maxObjectBytes) return callback(gatewayError("map_cache_object_too_large", 413));
|
||||
callback(null, chunk);
|
||||
},
|
||||
});
|
||||
try {
|
||||
await pipeline(Readable.fromWeb(upstream.body), limit, createWriteStream(tempPath, { flags: "wx" }));
|
||||
await rename(tempPath, filePath);
|
||||
} catch (error) {
|
||||
await rm(tempPath, { force: true });
|
||||
throw error;
|
||||
}
|
||||
|
||||
const now = Date.now();
|
||||
const entry = {
|
||||
key: cacheKey,
|
||||
file: relativePath,
|
||||
bytes,
|
||||
contentType: upstream.headers.get("content-type") || "application/octet-stream",
|
||||
etag: upstream.headers.get("etag") || null,
|
||||
savedAt: now,
|
||||
lastAccessAt: now,
|
||||
expiresAt: now + responseTtl(upstream.headers.get("cache-control")),
|
||||
};
|
||||
store.index.entries[cacheKey] = entry;
|
||||
await evictCache(store);
|
||||
await writeCacheIndex(store);
|
||||
return entry;
|
||||
}
|
||||
|
||||
function responseTtl(cacheControl) {
|
||||
const match = String(cacheControl || "").match(/max-age=(\d+)/i);
|
||||
if (!match) return config.defaultTtlMs;
|
||||
return Math.min(Number.parseInt(match[1], 10) * 1000, config.defaultTtlMs);
|
||||
}
|
||||
|
||||
async function proxyRangeRequest(request, response, target, cached, store) {
|
||||
const upstream = await fetchWithTimeout(target, { headers: { range: String(request.headers.range), accept: "*/*" } });
|
||||
if (!upstream.ok || !upstream.body) {
|
||||
if (cached) return serveCachedFile(request, response, store, cached, "stale-range-error");
|
||||
throw gatewayError("map_upstream_range_unavailable", upstream.status || 502);
|
||||
}
|
||||
copyUpstreamHeaders(response, upstream.headers);
|
||||
response.setHeader("x-nodedc-map-cache", "range-pass-through");
|
||||
response.writeHead(upstream.status);
|
||||
await pipeline(Readable.fromWeb(upstream.body), response);
|
||||
}
|
||||
|
||||
async function proxyUncachedRequest(request, response, target, cacheState) {
|
||||
const upstream = await fetchWithTimeout(target, {
|
||||
headers: {
|
||||
accept: String(request.headers.accept || "application/json, application/octet-stream, image/*, */*;q=0.5"),
|
||||
...(request.headers.range ? { range: String(request.headers.range) } : {}),
|
||||
},
|
||||
});
|
||||
if (!upstream.ok || !upstream.body) throw gatewayError("map_upstream_unavailable", upstream.status || 502);
|
||||
copyUpstreamHeaders(response, upstream.headers);
|
||||
response.setHeader("x-nodedc-map-cache", cacheState);
|
||||
response.writeHead(upstream.status);
|
||||
if (request.method === "HEAD") return response.end();
|
||||
await pipeline(Readable.fromWeb(upstream.body), response);
|
||||
}
|
||||
|
||||
async function serveCachedFile(request, response, store, entry, state) {
|
||||
const filePath = join(store.dir, entry.file);
|
||||
const info = await stat(filePath);
|
||||
const range = parseRange(request.headers.range, info.size);
|
||||
response.setHeader("content-type", entry.contentType);
|
||||
response.setHeader("accept-ranges", "bytes");
|
||||
response.setHeader("cache-control", "public, max-age=60");
|
||||
response.setHeader("x-nodedc-map-cache", state);
|
||||
response.setHeader("x-nodedc-map-cache-age", String(Math.max(0, Math.round((Date.now() - entry.savedAt) / 1000))));
|
||||
if (store.mutable) {
|
||||
entry.lastAccessAt = Date.now();
|
||||
store.index.entries[entry.key] = entry;
|
||||
void writeCacheIndex(store);
|
||||
}
|
||||
|
||||
if (!range) {
|
||||
response.setHeader("content-length", info.size);
|
||||
response.writeHead(200);
|
||||
if (request.method === "HEAD") return response.end();
|
||||
return pipeline(createReadStream(filePath), response);
|
||||
}
|
||||
response.setHeader("content-length", range.end - range.start + 1);
|
||||
response.setHeader("content-range", `bytes ${range.start}-${range.end}/${info.size}`);
|
||||
response.writeHead(206);
|
||||
if (request.method === "HEAD") return response.end();
|
||||
return pipeline(createReadStream(filePath, range), response);
|
||||
}
|
||||
|
||||
function parseRange(header, size) {
|
||||
if (!header) return null;
|
||||
const match = String(header).match(/^bytes=(\d*)-(\d*)$/);
|
||||
if (!match) return null;
|
||||
const start = match[1] ? Number.parseInt(match[1], 10) : 0;
|
||||
const end = match[2] ? Number.parseInt(match[2], 10) : size - 1;
|
||||
if (!Number.isInteger(start) || !Number.isInteger(end) || start < 0 || end < start || end >= size) return null;
|
||||
return { start, end };
|
||||
}
|
||||
|
||||
async function getCachedEntry(store, cacheKey) {
|
||||
const entry = store.index.entries[cacheKey];
|
||||
if (!entry) return null;
|
||||
try {
|
||||
const info = await stat(join(store.dir, entry.file));
|
||||
if (!info.isFile()) throw new Error("not_a_file");
|
||||
return entry;
|
||||
} catch {
|
||||
if (store.mutable) {
|
||||
delete store.index.entries[cacheKey];
|
||||
await writeCacheIndex(store);
|
||||
}
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
function createCacheStore(name, dir, mutable) {
|
||||
return {
|
||||
name,
|
||||
dir,
|
||||
mutable,
|
||||
objectsDir: join(dir, "objects"),
|
||||
ionEndpointsDir: join(dir, "ion-endpoints"),
|
||||
indexPath: join(dir, "index.json"),
|
||||
index: { version: 1, entries: {} },
|
||||
pendingIndexWrite: Promise.resolve(),
|
||||
};
|
||||
}
|
||||
|
||||
async function initialiseCacheStore(store) {
|
||||
if (store.mutable) {
|
||||
await mkdir(store.objectsDir, { recursive: true });
|
||||
await mkdir(store.ionEndpointsDir, { recursive: true });
|
||||
}
|
||||
store.index = await readCacheIndex(store);
|
||||
}
|
||||
|
||||
async function readCacheIndex(store) {
|
||||
try {
|
||||
const parsed = JSON.parse(await readFile(store.indexPath, "utf8"));
|
||||
if (parsed?.version === 1 && parsed.entries && typeof parsed.entries === "object") return parsed;
|
||||
} catch (error) {
|
||||
if (error?.code !== "ENOENT") console.warn("Map cache index ignored: invalid file");
|
||||
}
|
||||
return { version: 1, entries: {} };
|
||||
}
|
||||
|
||||
function ionEndpointPath(assetId) {
|
||||
return join(liveCache.ionEndpointsDir, `${assetId}.json`);
|
||||
}
|
||||
|
||||
async function readIonEndpointCache(assetId) {
|
||||
try {
|
||||
const value = JSON.parse(await readFile(ionEndpointPath(assetId), "utf8"));
|
||||
if (value?.assetId === assetId && ((value?.url && value?.accessToken) || (value?.externalType === "BING" && value?.options?.url && value?.options?.key))) return value;
|
||||
} catch (error) {
|
||||
if (error?.code !== "ENOENT") console.warn("Ion endpoint cache ignored: invalid file");
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
async function writeIonEndpointCache(endpoint) {
|
||||
const target = ionEndpointPath(endpoint.assetId);
|
||||
const temp = `${target}.${process.pid}.tmp`;
|
||||
await writeFile(temp, `${JSON.stringify(endpoint)}\n`, { encoding: "utf8", mode: 0o600 });
|
||||
await rename(temp, target);
|
||||
}
|
||||
|
||||
function writeCacheIndex(store) {
|
||||
if (!store.mutable) return Promise.resolve();
|
||||
store.pendingIndexWrite = store.pendingIndexWrite.then(async () => {
|
||||
const tempPath = `${store.indexPath}.${process.pid}.tmp`;
|
||||
await writeFile(tempPath, `${JSON.stringify(store.index)}\n`, "utf8");
|
||||
await rename(tempPath, store.indexPath);
|
||||
}).catch((error) => console.warn("Map cache index write failed", error instanceof Error ? error.message : "unknown"));
|
||||
return store.pendingIndexWrite;
|
||||
}
|
||||
|
||||
async function evictCache(store) {
|
||||
let entries = Object.values(store.index.entries);
|
||||
let size = entries.reduce((sum, entry) => sum + Number(entry.bytes || 0), 0);
|
||||
if (size <= config.maxCacheBytes) return;
|
||||
entries = entries.sort((left, right) => Number(left.lastAccessAt || 0) - Number(right.lastAccessAt || 0));
|
||||
for (const entry of entries) {
|
||||
if (size <= config.maxCacheBytes) break;
|
||||
await rm(join(store.dir, entry.file), { force: true });
|
||||
delete store.index.entries[entry.key];
|
||||
size -= Number(entry.bytes || 0);
|
||||
}
|
||||
}
|
||||
|
||||
async function cacheStats(store) {
|
||||
const entries = Object.values(store.index.entries);
|
||||
const bytes = entries.reduce((sum, entry) => sum + Number(entry.bytes || 0), 0);
|
||||
return { name: store.name, mode: store.mutable ? config.mode : "readonly", entries: entries.length, bytes, maxBytes: store.mutable ? config.maxCacheBytes : null, persistent: true };
|
||||
}
|
||||
|
||||
async function fetchWithTimeout(target, options = {}) {
|
||||
const controller = new AbortController();
|
||||
const timeout = setTimeout(() => controller.abort(), config.upstreamTimeoutMs);
|
||||
try { return await fetch(target, { ...options, signal: controller.signal, redirect: "follow" }); }
|
||||
catch (error) { throw gatewayError(error?.name === "AbortError" ? "map_upstream_timeout" : "map_upstream_fetch_failed", 502); }
|
||||
finally { clearTimeout(timeout); }
|
||||
}
|
||||
|
||||
function copyUpstreamHeaders(response, headers) {
|
||||
for (const name of ["content-type", "content-length", "content-range", "accept-ranges", "etag", "last-modified"]) {
|
||||
const value = headers.get(name);
|
||||
if (value) response.setHeader(name, value);
|
||||
}
|
||||
}
|
||||
|
||||
function writeJson(response, status, value) {
|
||||
response.writeHead(status, { "content-type": "application/json; charset=utf-8", "cache-control": "no-store" });
|
||||
response.end(JSON.stringify(value));
|
||||
}
|
||||
|
||||
function gatewayError(message, statusCode) {
|
||||
const error = new Error(message);
|
||||
error.statusCode = statusCode;
|
||||
return error;
|
||||
}
|
||||
Reference in New Issue
Block a user