Author SHA1 Message Date
Codex 49813d7839 fix(map-gateway): unlock warm layers and local station search 2026-08-09 20:27:04 +03:00
6 changed files with 289 additions and 25 deletions
+14 -5
View File
@@ -32,7 +32,7 @@ GET /api/map/reference-sources/v1/profiles/transport-stations.v1/search?q=<exact
Reference-station route возвращает `nodedc.map-reference.snapshot/v1` и только Reference-station route возвращает `nodedc.map-reference.snapshot/v1` и только
два онтологических типа: `map.station` и `map.terminal`. Категории v1 — два онтологических типа: `map.station` и `map.terminal`. Категории v1 —
`metro`, `railway_station`, `railway_terminal`. Проекция allowlist-ит имена, `metro`, `railway_station`, `railway_terminal`. Проекция allowlist-ит имена,
operator/network, `uic_ref`, wheelchair и геометрию; raw OSM tags, upstream operator/network, локализованные/альтернативные имена, `uic_ref`, wheelchair и геометрию; raw OSM tags, upstream
endpoint и provider credentials в snapshot отсутствуют. endpoint и provider credentials в snapshot отсутствуют.
Seed проверяется на старте и покрывает Москву без сетевого запроса. Для bbox за Seed проверяется на старте и покрывает Москву без сетевого запроса. Для bbox за
@@ -48,9 +48,11 @@ upstream не повреждает уже сохранённые cells; `complet
upstream endpoint или payload. upstream endpoint или payload.
Поиск вне уже загруженного viewport вызывается только явным Enter пользователя. Поиск вне уже загруженного viewport вызывается только явным Enter пользователя.
Gateway сначала ищет в seed и persistent search index, затем при miss выполняет Gateway при старте восстанавливает индекс из seed, persistent search index и
точный OSM name lookup. Upstream selector использует глобально индексированные всех сохранённых spatial cells. Локальное совпадение возвращается сразу и не
`name`/`name:ru`, а найденные элементы повторно проходят fail-closed проверку ждёт Overpass; только полный miss выполняет точный OSM name lookup. Upstream
selector использует глобально индексированные `name`, `name:ru`, `name:en`,
`official_name` и `alt_name`, а найденные элементы повторно проходят fail-closed проверку
`railway=station|halt`. Это не autocomplete и не глобальный regex scan. `railway=station|halt`. Это не autocomplete и не глобальный regex scan.
Нормализованный результат атомарно пополняет persistent search index; после Нормализованный результат атомарно пополняет persistent search index; после
перелёта обычный bbox-контур загружает полную spatial cell. Одинаковые запросы перелёта обычный bbox-контур загружает полную spatial cell. Одинаковые запросы
@@ -68,7 +70,7 @@ search counters.
`/api/map/ion/assets/:assetId/endpoint` разрешает только `CESIUM_ION_ASSET_ALLOWLIST`. Ответ не содержит credential: Browser использует публичный provider URL вместе с Cesium `DefaultProxy`, направляющим resource requests в `/api/map/cache`. Gateway валидирует scope URL, удаляет любые credential query parameters из browser request и добавляет соответствующий server-side credential только перед обращением к provider. `/api/map/ion/assets/:assetId/endpoint` разрешает только `CESIUM_ION_ASSET_ALLOWLIST`. Ответ не содержит credential: Browser использует публичный provider URL вместе с Cesium `DefaultProxy`, направляющим resource requests в `/api/map/cache`. Gateway валидирует scope URL, удаляет любые credential query parameters из browser request и добавляет соответствующий server-side credential только перед обращением к provider.
Ion endpoint metadata с asset credential сохраняется только в private persistent storage с файловыми правами `0600`. Это нужно для cold start и offline: Browser по-прежнему получает только sanitised URL, а Gateway отвечает из ранее записанного cache. В metadata никогда не записывается master token; NAS backup этого каталога считается service-sensitive. Ion endpoint metadata с asset credential сохраняется только в private persistent storage с файловыми правами `0600`. Это нужно для cold start и offline: Browser по-прежнему получает только sanitised URL, а Gateway отвечает из ранее записанного cache. Если asset credential истёк, а upstream/VPN недоступен, Gateway всё равно возвращает сохранённый публичный URL без credential: cache lookup выполняется раньше credential injection, поэтому warm `layer.json`, `tileset.json` и дочерние объекты остаются доступны, а настоящий miss по-прежнему fails closed. В metadata никогда не записывается master token; NAS backup этого каталога считается service-sensitive.
## Cache modes ## Cache modes
@@ -80,6 +82,13 @@ Ion endpoint metadata с asset credential сохраняется только в
Первый cold miss начинает стримиться клиенту сразу после provider headers и одновременно записывается в private temp-файл. Только полностью полученный object публикуется через atomic rename и durable `index.json`; partial/error никогда не становится cache entry. Одинаковые параллельные misses делят один upstream download, а отмена одного browser request не прерывает server-owned fill для других инстансов. Warm hit не меняет index. Параллельные новые объекты объединяются в минимальное число полных index snapshots вместо одного O(N) rewrite на каждый tile. Первый cold miss начинает стримиться клиенту сразу после provider headers и одновременно записывается в private temp-файл. Только полностью полученный object публикуется через atomic rename и durable `index.json`; partial/error никогда не становится cache entry. Одинаковые параллельные misses делят один upstream download, а отмена одного browser request не прерывает server-owned fill для других инстансов. Warm hit не меняет index. Параллельные новые объекты объединяются в минимальное число полных index snapshots вместо одного O(N) rewrite на каждый tile.
Новые index entries получают безопасные `resourceKind`/`assetId` без URL и
credentials. `/healthz` агрегирует число и размер imagery, terrain, buildings,
metadata и legacy 3D/binary объектов. Старые hash-only entries классифицируются
только по надёжному content type; `application/octet-stream` честно остаётся
`3d-binary`, потому что ретроспективно приписать его конкретному Ion asset без
сохранённого URL невозможно.
`offline` дополнительно требует, чтобы hostname был явно указан в `MAP_GATEWAY_OFFLINE_PROVIDER_ALLOWLIST`. По умолчанию список пуст: Gateway не превращает Cesium ion или public OSM tile service в offline distribution. Это осознанная provider policy, а не техническая ошибка cache. `offline` дополнительно требует, чтобы hostname был явно указан в `MAP_GATEWAY_OFFLINE_PROVIDER_ALLOWLIST`. По умолчанию список пуст: Gateway не превращает Cesium ion или public OSM tile service в offline distribution. Это осознанная provider policy, а не техническая ошибка cache.
Range requests пока transparently проксируются и не записываются как partial cache object. Полные GET responses кэшируются. Это безопасная начальная граница для terrain/3D Tiles; отдельным следующим шагом добавляется range-aware chunk cache после замеров реальных GOS/3D Tiles assets. Range requests пока transparently проксируются и не записываются как partial cache object. Полные GET responses кэшируются. Это безопасная начальная граница для terrain/3D Tiles; отдельным следующим шагом добавляется range-aware chunk cache после замеров реальных GOS/3D Tiles assets.
@@ -63,12 +63,13 @@ function normalizeFeature(feature, generatedAt, index) {
const identity = nativeId.match(/^(node|way|relation)\/([1-9]\d*)$/); const identity = nativeId.match(/^(node|way|relation)\/([1-9]\d*)$/);
if (!identity) throw new Error(`station_seed_feature_${index}_identity_invalid`); if (!identity) throw new Error(`station_seed_feature_${index}_identity_invalid`);
const attributes = compact({ const attributes = compact({
name: firstString(properties.name, properties["name:ru"], properties.official_name, properties.loc_name), name: firstString(properties["name:ru"], properties.name, properties.official_name, properties.loc_name),
category, category,
network: optionalString(properties.network), network: optionalString(properties.network),
operator: optionalString(properties.operator), operator: optionalString(properties.operator),
official_name: optionalString(properties.official_name), official_name: optionalString(properties.official_name),
local_name: optionalString(properties.loc_name), local_name: optionalString(properties.loc_name),
alternate_names: alternateNames(properties),
uic_ref: optionalString(properties.uic_ref), uic_ref: optionalString(properties.uic_ref),
wheelchair: optionalString(properties.wheelchair), wheelchair: optionalString(properties.wheelchair),
}); });
@@ -101,6 +102,28 @@ function optionalString(value) {
return normalized && normalized.length <= 256 ? normalized : undefined; return normalized && normalized.length <= 256 ? normalized : undefined;
} }
function alternateNames(properties) {
const primary = firstString(properties["name:ru"], properties.name, properties.official_name, properties.loc_name);
const unique = new Map();
for (const value of [
properties.name,
properties["name:ru"],
properties["name:en"],
properties.official_name,
properties.loc_name,
properties.short_name,
properties.old_name,
...String(properties.alt_name || "").split(";"),
]) {
const normalized = optionalString(value);
if (!normalized || normalized.toLocaleLowerCase("ru") === primary?.toLocaleLowerCase("ru")) continue;
const key = normalized.toLocaleLowerCase("ru");
if (!unique.has(key)) unique.set(key, normalized);
}
const values = [...unique.values()].slice(0, 16);
return values.length ? values : undefined;
}
function compact(value) { function compact(value) {
return Object.fromEntries(Object.entries(value).filter(([, item]) => item !== undefined)); return Object.fromEntries(Object.entries(value).filter(([, item]) => item !== undefined));
} }
@@ -1,9 +1,10 @@
import assert from "node:assert/strict"; import assert from "node:assert/strict";
import { createHash } from "node:crypto";
import { once } from "node:events"; import { once } from "node:events";
import { mkdtemp, mkdir, rm, writeFile } from "node:fs/promises"; import { mkdtemp, mkdir, rm, writeFile } from "node:fs/promises";
import { createServer } from "node:net"; import { createServer } from "node:net";
import { tmpdir } from "node:os"; import { tmpdir } from "node:os";
import { join } from "node:path"; import { dirname, join } from "node:path";
import { spawn } from "node:child_process"; import { spawn } from "node:child_process";
const root = await mkdtemp(join(tmpdir(), "nodedc-map-gateway-credential-smoke-")); const root = await mkdtemp(join(tmpdir(), "nodedc-map-gateway-credential-smoke-"));
@@ -13,7 +14,27 @@ let child;
try { try {
await mkdir(join(cacheDir, "ion-endpoints"), { recursive: true }); await mkdir(join(cacheDir, "ion-endpoints"), { recursive: true });
await writeFile(join(cacheDir, "index.json"), `${JSON.stringify({ version: 1, entries: {} })}\n`); const buildingsUrl = "https://assets.ion.cesium.com/us-east-1/asset_depot/96188/OpenStreetMap/CWT/2025-04-01/tileset.json?v=expired";
const buildingsKey = createHash("sha256").update(buildingsUrl).digest("hex");
const buildingsRelativePath = join("objects", buildingsKey.slice(0, 2), `${buildingsKey}.bin`);
const buildingsBody = JSON.stringify({ asset: { version: "1.1" }, root: { geometricError: 0 } });
await mkdir(dirname(join(cacheDir, buildingsRelativePath)), { recursive: true });
await writeFile(join(cacheDir, buildingsRelativePath), buildingsBody);
await writeFile(join(cacheDir, "index.json"), `${JSON.stringify({
version: 1,
entries: {
[buildingsKey]: {
key: buildingsKey,
file: buildingsRelativePath,
bytes: Buffer.byteLength(buildingsBody),
contentType: "application/json",
etag: null,
savedAt: Date.now(),
lastAccessAt: Date.now(),
expiresAt: Date.now() - 1,
},
},
})}\n`);
await writeFile(join(cacheDir, "ion-endpoints", "1.json"), `${JSON.stringify({ await writeFile(join(cacheDir, "ion-endpoints", "1.json"), `${JSON.stringify({
assetId: "1", assetId: "1",
type: "TERRAIN", type: "TERRAIN",
@@ -38,7 +59,7 @@ try {
await writeFile(join(cacheDir, "ion-endpoints", "96188.json"), `${JSON.stringify({ await writeFile(join(cacheDir, "ion-endpoints", "96188.json"), `${JSON.stringify({
assetId: "96188", assetId: "96188",
type: "3DTILES", type: "3DTILES",
url: "https://assets.ion.cesium.com/us-east-1/asset_depot/96188/OpenStreetMap/CWT/2025-04-01/tileset.json?v=expired", url: buildingsUrl,
accessToken: expiredBuildingsToken, accessToken: expiredBuildingsToken,
attributions: [], attributions: [],
savedAt: Date.now(), savedAt: Date.now(),
@@ -48,11 +69,13 @@ try {
cwd: new URL("..", import.meta.url), cwd: new URL("..", import.meta.url),
env: { env: {
...process.env, ...process.env,
NODE_ENV: "test",
PORT: String(port), PORT: String(port),
MAP_CACHE_DIR: cacheDir, MAP_CACHE_DIR: cacheDir,
MAP_GATEWAY_ALLOW_ANONYMOUS: "true", MAP_GATEWAY_ALLOW_ANONYMOUS: "true",
MAP_CACHE_MODE: "readwrite", MAP_CACHE_MODE: "readwrite",
CESIUM_ION_TOKEN: "", CESIUM_ION_TOKEN: "configured-master-token",
CESIUM_ION_API_BASE_URL: `http://127.0.0.1:${port}/`,
CESIUM_ION_ASSET_ALLOWLIST: "1,2,96188", CESIUM_ION_ASSET_ALLOWLIST: "1,2,96188",
MAP_GATEWAY_UPSTREAM_ALLOWLIST: "assets.ion.cesium.com,dev.virtualearth.net", MAP_GATEWAY_UPSTREAM_ALLOWLIST: "assets.ion.cesium.com,dev.virtualearth.net",
}, },
@@ -73,10 +96,16 @@ try {
} }
const expired = await fetch(`http://127.0.0.1:${port}/api/map/ion/assets/96188/endpoint`); const expired = await fetch(`http://127.0.0.1:${port}/api/map/ion/assets/96188/endpoint`);
const expiredRaw = await expired.text(); const expiredRaw = await expired.text();
assert.equal(expired.status, 503); assert.equal(expired.status, 200);
assert.equal(expiredRaw.includes(expiredBuildingsToken), false); assert.equal(expiredRaw.includes(expiredBuildingsToken), false);
assert.equal(expiredRaw.includes("accessToken"), false); assert.equal(expiredRaw.includes("accessToken"), false);
console.log("ok: endpoint responses contain no credentials and a known-expired Ion JWT is never served"); assert.equal(JSON.parse(expiredRaw).cache, "ion-endpoint-stale-upstream-error");
const cachedRoot = await fetch(`http://127.0.0.1:${port}/api/map/cache?url=${encodeURIComponent(`${buildingsUrl}&nodedc_client_revision=2`)}`);
assert.equal(cachedRoot.status, 200);
assert.equal(cachedRoot.headers.get("x-nodedc-map-cache"), "live-cache-stale");
assert.deepEqual(await cachedRoot.json(), JSON.parse(buildingsBody));
console.log("ok: endpoint responses contain no credentials and expired metadata still unlocks warm cache during upstream outage");
} finally { } finally {
if (child && !child.killed) { if (child && !child.killed) {
child.kill("SIGTERM"); child.kill("SIGTERM");
@@ -40,6 +40,10 @@ export async function createReferenceStationSource({
const recentSearches = new Map(); const recentSearches = new Map();
const indexedFacts = new Map(seed.facts.map((fact) => [fact.sourceId, fact])); const indexedFacts = new Map(seed.facts.map((fact) => [fact.sourceId, fact]));
for (const fact of await readSearchIndex(searchIndexFile)) indexedFacts.set(fact.sourceId, fact); for (const fact of await readSearchIndex(searchIndexFile)) indexedFacts.set(fact.sourceId, fact);
// Cells are the durable record of everything the user has already viewed.
// Rehydrate them on restart so a warm viewport remains searchable without
// waiting for Overpass or revisiting the same coordinates first.
for (const fact of await readCachedCellFacts(cellRoot)) indexedFacts.set(fact.sourceId, fact);
const pendingFetches = []; const pendingFetches = [];
let activeFetches = 0; let activeFetches = 0;
let upstreamRequests = 0; let upstreamRequests = 0;
@@ -51,6 +55,16 @@ export async function createReferenceStationSource({
let searchFailures = 0; let searchFailures = 0;
let upstreamStartQueue = Promise.resolve(); let upstreamStartQueue = Promise.resolve();
let nextUpstreamStartAt = 0; let nextUpstreamStartAt = 0;
let searchIndexWriteQueue = Promise.resolve();
function persistIndexedFacts() {
// Serialize complete snapshots. Concurrent viewport-cell fills may finish
// in either order; taking the values inside the queued task prevents a
// later write from dropping facts indexed by an earlier completion.
const task = searchIndexWriteQueue.then(() => writeSearchIndex(searchIndexFile, [...indexedFacts.values()]));
searchIndexWriteQueue = task.catch(() => undefined);
return task;
}
async function snapshot({ bbox } = {}) { async function snapshot({ bbox } = {}) {
const normalizedBbox = bbox ? normalizeBbox(bbox) : null; const normalizedBbox = bbox ? normalizeBbox(bbox) : null;
@@ -109,8 +123,11 @@ export async function createReferenceStationSource({
const normalizedLimit = Math.max(1, Math.min(MAX_SEARCH_RESULTS, Number(limit) || 12)); const normalizedLimit = Math.max(1, Math.min(MAX_SEARCH_RESULTS, Number(limit) || 12));
const localFacts = searchIndexedFacts(indexedFacts.values(), normalizedQuery, normalizedLimit); const localFacts = searchIndexedFacts(indexedFacts.values(), normalizedQuery, normalizedLimit);
let upstreamFacts = []; let upstreamFacts = [];
let complete = !fetchEnabled; // Cache/seed hits are terminal. The previous implementation still waited
if (fetchEnabled && normalizedQuery.length >= 3) { // for a global Overpass request and turned an instant local result into a
// 30-second error whenever the public service was degraded.
let complete = localFacts.length > 0 || !fetchEnabled;
if (fetchEnabled && localFacts.length === 0 && normalizedQuery.length >= 3) {
const cachedSearch = recentSearches.get(normalizedQuery); const cachedSearch = recentSearches.get(normalizedQuery);
if (cachedSearch && Date.now() - cachedSearch.storedAt < SEARCH_CACHE_TTL_MS) { if (cachedSearch && Date.now() - cachedSearch.storedAt < SEARCH_CACHE_TTL_MS) {
upstreamFacts = cachedSearch.facts; upstreamFacts = cachedSearch.facts;
@@ -123,7 +140,7 @@ export async function createReferenceStationSource({
try { try {
const facts = await fetchSearch(endpoint, displayQuery, normalizedLimit, normalizedTimeoutMs, fetchImpl); const facts = await fetchSearch(endpoint, displayQuery, normalizedLimit, normalizedTimeoutMs, fetchImpl);
indexFacts(facts); indexFacts(facts);
await writeSearchIndex(searchIndexFile, [...indexedFacts.values()]); await persistIndexedFacts();
recentSearches.set(normalizedQuery, { storedAt: Date.now(), facts }); recentSearches.set(normalizedQuery, { storedAt: Date.now(), facts });
lastRefreshAt = new Date().toISOString(); lastRefreshAt = new Date().toISOString();
return { facts, complete: true }; return { facts, complete: true };
@@ -179,6 +196,7 @@ export async function createReferenceStationSource({
try { try {
const document = await fetchCell(endpoint, cell, normalizedTimeoutMs, fetchImpl); const document = await fetchCell(endpoint, cell, normalizedTimeoutMs, fetchImpl);
await writeCell(cellRoot, cell.key, document); await writeCell(cellRoot, cell.key, document);
indexFacts(document.facts);
lastRefreshAt = document.generatedAt; lastRefreshAt = document.generatedAt;
return document; return document;
} catch (error) { } catch (error) {
@@ -239,6 +257,7 @@ export async function createReferenceStationSource({
fetchEnabled, fetchEnabled,
cellDegrees: normalizedCellDegrees, cellDegrees: normalizedCellDegrees,
cachedCellCount, cachedCellCount,
indexedFactCount: indexedFacts.size,
upstreamRequests, upstreamRequests,
upstreamFailures, upstreamFailures,
searchRequests, searchRequests,
@@ -283,10 +302,16 @@ function validateFact(value) {
|| !Number.isFinite(coordinates[1]) || coordinates[1] < -90 || coordinates[1] > 90) { || !Number.isFinite(coordinates[1]) || coordinates[1] < -90 || coordinates[1] > 90) {
throw sourceError("reference_station_geometry_invalid"); throw sourceError("reference_station_geometry_invalid");
} }
const allowedAttributes = new Set(["name", "category", "network", "operator", "official_name", "local_name", "uic_ref", "wheelchair"]); const allowedAttributes = new Set(["name", "category", "network", "operator", "official_name", "local_name", "alternate_names", "uic_ref", "wheelchair"]);
if (Object.keys(value.attributes).some((key) => !allowedAttributes.has(key))) { if (Object.keys(value.attributes).some((key) => !allowedAttributes.has(key))) {
throw sourceError("reference_station_attribute_not_allowed"); throw sourceError("reference_station_attribute_not_allowed");
} }
if (value.attributes.alternate_names !== undefined
&& (!Array.isArray(value.attributes.alternate_names)
|| value.attributes.alternate_names.length > 16
|| value.attributes.alternate_names.some((name) => !optionalString(name)))) {
throw sourceError("reference_station_alternate_names_invalid");
}
if (value.semanticType === "map.terminal" !== (value.attributes.category === "railway_terminal")) { if (value.semanticType === "map.terminal" !== (value.attributes.category === "railway_terminal")) {
throw sourceError("reference_station_semantic_type_mismatch"); throw sourceError("reference_station_semantic_type_mismatch");
} }
@@ -342,7 +367,7 @@ async function fetchSearch(endpoint, query, limit, timeoutMs, fetchImpl) {
// whole planet by railway first is several orders of magnitude slower and // whole planet by railway first is several orders of magnitude slower and
// routinely times out; station semantics are therefore verified locally // routinely times out; station semantics are therefore verified locally
// by the same fail-closed normalizer used by viewport cells. // by the same fail-closed normalizer used by viewport cells.
const overpassQuery = `[out:json][timeout:25];(nwr["name"="${expression}"];nwr["name:ru"="${expression}"];);out center tags ${resultLimit};`; const overpassQuery = `[out:json][timeout:25];(nwr["name"="${expression}"];nwr["name:ru"="${expression}"];nwr["name:en"="${expression}"];nwr["official_name"="${expression}"];nwr["alt_name"="${expression}"];);out center tags ${resultLimit};`;
const response = await fetchImpl(endpoint, { const response = await fetchImpl(endpoint, {
method: "POST", method: "POST",
headers: { headers: {
@@ -388,6 +413,16 @@ function normalizeOverpassElement(value, observedAt) {
const name = firstString(value.tags["name:ru"], value.tags.name, value.tags.official_name, value.tags.loc_name); const name = firstString(value.tags["name:ru"], value.tags.name, value.tags.official_name, value.tags.loc_name);
if (!name) throw sourceError("reference_station_upstream_name_required"); if (!name) throw sourceError("reference_station_upstream_name_required");
const category = classifyStation(value.tags, name); const category = classifyStation(value.tags, name);
const alternateNames = uniqueStrings(
value.tags.name,
value.tags["name:ru"],
value.tags["name:en"],
value.tags.official_name,
value.tags.loc_name,
value.tags.short_name,
value.tags.old_name,
...String(value.tags.alt_name || "").split(";"),
).filter((candidate) => normalizeSearchValue(candidate) !== normalizeSearchValue(name)).slice(0, 16);
return validateFact({ return validateFact({
sourceId: `osm.${value.type}.${value.id}`, sourceId: `osm.${value.type}.${value.id}`,
semanticType: category === "railway_terminal" ? "map.terminal" : "map.station", semanticType: category === "railway_terminal" ? "map.terminal" : "map.station",
@@ -400,6 +435,7 @@ function normalizeOverpassElement(value, observedAt) {
operator: optionalString(value.tags.operator), operator: optionalString(value.tags.operator),
official_name: optionalString(value.tags.official_name), official_name: optionalString(value.tags.official_name),
local_name: optionalString(value.tags.loc_name), local_name: optionalString(value.tags.loc_name),
alternate_names: alternateNames.length ? alternateNames : undefined,
uic_ref: optionalString(value.tags.uic_ref), uic_ref: optionalString(value.tags.uic_ref),
wheelchair: optionalString(value.tags.wheelchair), wheelchair: optionalString(value.tags.wheelchair),
}), }),
@@ -484,6 +520,7 @@ function searchIndexedFacts(values, query, limit) {
fact.attributes.name, fact.attributes.name,
fact.attributes.official_name, fact.attributes.official_name,
fact.attributes.local_name, fact.attributes.local_name,
...(fact.attributes.alternate_names ?? []),
fact.attributes.uic_ref, fact.attributes.uic_ref,
], query); ], query);
return rank === null ? [] : [{ fact, rank }]; return rank === null ? [] : [{ fact, rank }];
@@ -562,6 +599,24 @@ async function readSearchIndex(file) {
} }
} }
async function readCachedCellFacts(root) {
let names;
try {
names = (await readdir(root)).filter((name) => name.endsWith(".json")).sort();
} catch {
return [];
}
const facts = new Map();
for (const name of names) {
const cell = await readCell(root, name.slice(0, -5));
for (const fact of cell?.facts ?? []) {
facts.set(fact.sourceId, fact);
if (facts.size >= MAX_FACTS) return [...facts.values()];
}
}
return [...facts.values()];
}
async function writeSearchIndex(file, facts) { async function writeSearchIndex(file, facts) {
const normalizedFacts = deduplicateFacts(facts).slice(-MAX_FACTS); const normalizedFacts = deduplicateFacts(facts).slice(-MAX_FACTS);
const document = { const document = {
@@ -618,6 +673,17 @@ function optionalString(value) {
return normalized && normalized.length <= 256 ? normalized : undefined; return normalized && normalized.length <= 256 ? normalized : undefined;
} }
function uniqueStrings(...values) {
const unique = new Map();
for (const value of values) {
const normalized = optionalString(value);
if (!normalized) continue;
const key = normalizeSearchValue(normalized);
if (!unique.has(key)) unique.set(key, normalized);
}
return [...unique.values()];
}
function compact(value) { function compact(value) {
return Object.fromEntries(Object.entries(value).filter(([, item]) => item !== undefined)); return Object.fromEntries(Object.entries(value).filter(([, item]) => item !== undefined));
} }
+90 -8
View File
@@ -114,6 +114,7 @@ const server = createServer(async (request, response) => {
referenceSources: { referenceSources: {
transportStations: await referenceStationSource.status(), transportStations: await referenceStationSource.status(),
}, },
providerCache: await ionProviderCacheStatus(),
}); });
} }
@@ -592,15 +593,18 @@ async function serveIonEndpoint(request, response, assetId) {
const cachedState = cached ? ionEndpointCacheState(cached) : null; const cachedState = cached ? ionEndpointCacheState(cached) : null;
if (config.mode === "offline") { if (config.mode === "offline") {
if (!cached) return writeJson(response, 504, { ok: false, error: "cesium_ion_offline_endpoint_miss" }); if (!cached) return writeJson(response, 504, { ok: false, error: "cesium_ion_offline_endpoint_miss" });
if (!cachedState.usable) return writeJson(response, 504, { ok: false, error: "cesium_ion_offline_endpoint_expired" });
if (!isOfflineProviderAllowed(ionEndpointUrl(cached))) return writeJson(response, 409, { ok: false, error: "map_provider_offline_not_permitted" }); if (!isOfflineProviderAllowed(ionEndpointUrl(cached))) return writeJson(response, 409, { ok: false, error: "map_provider_offline_not_permitted" });
rememberIonEndpoint(cached); rememberIonEndpoint(cached);
return writeJson(response, 200, { ok: true, ...publicIonEndpoint(cached), cache: "offline-endpoint-hit" }); // The browser receives only a sanitised provider URL. An expired private
// asset credential is irrelevant for a cache-only request because cache
// lookup happens before credential injection and an offline miss never
// reaches upstream.
return writeJson(response, 200, { ok: true, ...publicIonEndpoint(cached), cache: "offline-public-endpoint-hit" });
} }
if (!activeCesiumIonToken) { if (!activeCesiumIonToken) {
if (cached && cachedState.usable) { if (cached) {
rememberIonEndpoint(cached); rememberIonEndpoint(cached);
return writeJson(response, 200, { ok: true, ...publicIonEndpoint(cached), cache: "cached-endpoint-no-master-token" }); return writeJson(response, 200, { ok: true, ...publicIonEndpoint(cached), cache: "cached-public-endpoint-no-master-token" });
} }
return writeJson(response, 503, { ok: false, error: "cesium_ion_not_configured" }); return writeJson(response, 503, { ok: false, error: "cesium_ion_not_configured" });
} }
@@ -616,8 +620,20 @@ async function serveIonEndpoint(request, response, assetId) {
}); });
} }
const next = await refreshIonEndpoint(assetId, ionReferer); try {
return writeJson(response, 200, { ok: true, ...publicIonEndpoint(next), cache: "ion-endpoint-online" }); const next = await refreshIonEndpoint(assetId, ionReferer);
return writeJson(response, 200, { ok: true, ...publicIonEndpoint(next), cache: "ion-endpoint-online" });
} catch (error) {
if (!cached) throw error;
// Endpoint credentials may expire while the AMD/VPN route is offline.
// Returning the cached *public* URL lets Cesium request layer.json or
// tileset.json and consume every warm object. A real miss still reaches
// injectGatewayCredentials(), attempts refresh and fails closed.
rememberIonEndpoint(cached);
const errorCode = safeDiagnosticCode(error?.message || "cesium_ion_endpoint_refresh_failed");
console.warn(JSON.stringify({ event: "map_ion_endpoint_public_fallback", assetId, error: errorCode }));
return writeJson(response, 200, { ok: true, ...publicIonEndpoint(cached), cache: "ion-endpoint-stale-upstream-error" });
}
} }
function scheduleIonEndpointRefresh(assetId, ionReferer = "") { function scheduleIonEndpointRefresh(assetId, ionReferer = "") {
@@ -882,11 +898,11 @@ async function prepareStreamingCacheFill(target, cacheKey, store, ionReferer = "
} }
const [clientBody, cacheBody] = upstream.body.tee(); const [clientBody, cacheBody] = upstream.body.tee();
const cacheCommit = commitCacheBody(cacheBody, upstream.headers, cacheKey, store); const cacheCommit = commitCacheBody(cacheBody, upstream.headers, cacheKey, store, cacheResourceIdentity(target));
return { status: upstream.status, headers: upstream.headers, clientBody, cacheCommit }; return { status: upstream.status, headers: upstream.headers, clientBody, cacheCommit };
} }
async function commitCacheBody(cacheBody, upstreamHeaders, cacheKey, store) { async function commitCacheBody(cacheBody, upstreamHeaders, cacheKey, store, resourceIdentity = null) {
const relativePath = join("objects", cacheKey.slice(0, 2), `${cacheKey}.bin`); const relativePath = join("objects", cacheKey.slice(0, 2), `${cacheKey}.bin`);
const filePath = join(store.dir, relativePath); const filePath = join(store.dir, relativePath);
const tempPath = `${filePath}.${process.pid}.${Date.now()}.tmp`; const tempPath = `${filePath}.${process.pid}.${Date.now()}.tmp`;
@@ -916,6 +932,8 @@ async function commitCacheBody(cacheBody, upstreamHeaders, cacheKey, store) {
savedAt: now, savedAt: now,
lastAccessAt: now, lastAccessAt: now,
expiresAt: now + responseTtl(upstreamHeaders.get("cache-control")), expiresAt: now + responseTtl(upstreamHeaders.get("cache-control")),
...(resourceIdentity?.resourceKind ? { resourceKind: resourceIdentity.resourceKind } : {}),
...(resourceIdentity?.assetId ? { assetId: resourceIdentity.assetId } : {}),
}; };
store.index.entries[cacheKey] = entry; store.index.entries[cacheKey] = entry;
await releaseCacheCapacity(store, bytes); await releaseCacheCapacity(store, bytes);
@@ -1317,6 +1335,14 @@ function isCacheCapacityError(error) {
async function cacheStats(store) { async function cacheStats(store) {
const entries = Object.values(store.index.entries); const entries = Object.values(store.index.entries);
const bytes = cacheBytes(store); const bytes = cacheBytes(store);
const byResourceKind = {};
for (const entry of entries) {
const resourceKind = cacheEntryResourceKind(entry);
const aggregate = byResourceKind[resourceKind] ?? { entries: 0, bytes: 0 };
aggregate.entries += 1;
aggregate.bytes += Number(entry?.bytes || 0);
byResourceKind[resourceKind] = aggregate;
}
return { return {
name: store.name, name: store.name,
mode: store.mutable ? config.mode : "readonly", mode: store.mutable ? config.mode : "readonly",
@@ -1326,9 +1352,65 @@ async function cacheStats(store) {
maxBytes: store.mutable ? config.maxCacheBytes : null, maxBytes: store.mutable ? config.maxCacheBytes : null,
atCapacity: store.mutable ? bytes >= config.maxCacheBytes : false, atCapacity: store.mutable ? bytes >= config.maxCacheBytes : false,
persistent: true, persistent: true,
byResourceKind,
}; };
} }
function cacheResourceIdentity(rawTarget) {
let target;
try {
target = stripCredentialQueryParameters(new URL(String(rawTarget)));
} catch {
return null;
}
if (/(^|\.)virtualearth\.net$/i.test(target.hostname) || target.hostname === "tile.openstreetmap.org") {
return { resourceKind: "imagery", assetId: target.hostname === "tile.openstreetmap.org" ? "osm" : "2" };
}
const matched = [...activeIonEndpoints.values()]
.filter((endpoint) => endpoint.externalType !== "BING" && endpoint.url)
.map((endpoint) => ({ endpoint, scopeLength: endpointResourceScopeLength(target, endpoint.url) }))
.filter(({ scopeLength }) => scopeLength >= 0)
.sort((left, right) => right.scopeLength - left.scopeLength)[0]?.endpoint;
if (!matched) return null;
const assetId = String(matched.assetId);
const resourceKind = assetId === "1" ? "terrain" : assetId === "96188" ? "buildings" : "ion-other";
return { resourceKind, assetId };
}
function cacheEntryResourceKind(entry) {
if (["imagery", "terrain", "buildings", "ion-other"].includes(entry?.resourceKind)) return entry.resourceKind;
const contentType = String(entry?.contentType || "").toLowerCase();
if (contentType.startsWith("image/")) return "imagery";
if (contentType.includes("quantized-mesh")) return "terrain";
// Old runtime entries predate safe provider provenance. With no stored URL
// an octet-stream cannot honestly be attributed to one asset, so expose it
// as legacy 3D/binary rather than pretending every object is an OSM house.
if (contentType.includes("application/octet-stream")) return "3d-binary";
if (contentType.includes("json") || contentType.startsWith("text/")) return "metadata";
return "unknown";
}
async function ionProviderCacheStatus() {
return Promise.all(canonicalCesiumAssetIds.map(async (assetId) => {
const endpoint = await readIonEndpointCache(assetId);
if (!endpoint) return { assetId: Number(assetId), cachedEndpoint: false, credentialUsable: false, entryPointCached: false };
const candidateUrls = assetId === "1"
? [new URL("layer.json", endpoint.url).toString()]
: assetId === "96188" ? [endpoint.url] : [];
const entryPointCached = candidateUrls.some((url) => {
const key = createHash("sha256").update(canonicalCacheUrl(new URL(url))).digest("hex");
return Boolean(liveCache.index.entries[key]);
});
return {
assetId: Number(assetId),
type: endpoint.type,
cachedEndpoint: true,
credentialUsable: ionEndpointCacheState(endpoint).usable,
entryPointCached,
};
}));
}
function shouldUseCesiumEgress(target) { function shouldUseCesiumEgress(target) {
if (!config.mapEgress.url) return false; if (!config.mapEgress.url) return false;
try { try {
@@ -81,6 +81,61 @@ test("bbox fetch uses one cached spatial cell and never exposes upstream payload
assert.equal(cached.schemaVersion, "nodedc.map-reference-cell/v1"); assert.equal(cached.schemaVersion, "nodedc.map-reference-cell/v1");
}); });
test("seed and viewed cells resolve locally without waiting for global search", async () => {
const cacheDir = await mkdtemp(join(tmpdir(), "nodedc-reference-stations-"));
let calls = 0;
const source = await createReferenceStationSource({
seedFile,
cacheDir,
cellDegrees: 1,
fetchImpl: async () => {
calls += 1;
return new Response(JSON.stringify({
elements: [{
type: "node",
id: 987654399,
lat: 59.93,
lon: 30.31,
tags: {
railway: "station",
station: "subway",
name: "Petrogradskaya",
"name:ru": "Петроградская",
"name:en": "Petrogradskaya station",
},
}],
}), { status: 200, headers: { "content-type": "application/json" } });
},
});
const seedSnapshot = await source.snapshot();
const seedName = seedSnapshot.facts[0].attributes.name;
const seedSearch = await source.search({ query: seedName, limit: 8 });
assert.equal(seedSearch.complete, true);
assert.ok(seedSearch.facts.length > 0);
assert.equal(calls, 0);
await source.snapshot({ bbox: [30, 59, 31, 60] });
const viewedSearch = await source.search({ query: "Petrogradskaya station", limit: 8 });
assert.equal(viewedSearch.complete, true);
assert.equal(viewedSearch.facts[0].sourceId, "osm.node.987654399");
assert.equal(calls, 1);
let restartedCalls = 0;
const restarted = await createReferenceStationSource({
seedFile,
cacheDir,
fetchImpl: async () => {
restartedCalls += 1;
throw new Error("warm viewed cell must not call upstream");
},
});
const restartedSearch = await restarted.search({ query: "Petrogradskaya station", limit: 8 });
assert.equal(restartedSearch.complete, true);
assert.equal(restartedSearch.facts[0].sourceId, "osm.node.987654399");
assert.equal(restartedCalls, 0);
});
test("viewport cells use bounded concurrency and deduplicate inflight fetches", async () => { test("viewport cells use bounded concurrency and deduplicate inflight fetches", async () => {
const cacheDir = await mkdtemp(join(tmpdir(), "nodedc-reference-stations-")); const cacheDir = await mkdtemp(join(tmpdir(), "nodedc-reference-stations-"));
let active = 0; let active = 0;