diff --git a/infra/deploy-runner/README.md b/infra/deploy-runner/README.md index 7ccf280..ad47396 100644 --- a/infra/deploy-runner/README.md +++ b/infra/deploy-runner/README.md @@ -81,9 +81,347 @@ Supported components in this source: - `n8n-private-extension` - `module-foundry` - `device-plane` +- `gitea` - `proxy-contur` - `dc-amd-proxy` +## Fresh Gitea installation + +`gitea` is a one-time, fresh-install-only component. Its artifact contains +only the reviewed Compose file and strict deployment descriptor; it contains +no database, repository, user, token, key, hook, runtime data or secret. Build +the deterministic artifact locally with: + +```bash +node infra/deploy-runner/build-gitea-fresh-install-artifact.mjs \ + gitea-fresh-install-YYYYMMDD-NNN +``` + +The runtime is fixed to Gitea `1.27.1-rootless` by exact digest and +`linux/amd64`. The runner never pulls it: the exact image must first be loaded +through a separately reviewed trusted image-acquisition step. Compose is +invoked with `--pull never --no-deps`, uses `network_mode: none`, publishes no +ports, and serves only the host-mounted Unix socket +`/volume1/docker/nodedc-gitea/socket/gitea.sock`. It exposes no SSH port, uses +fresh SQLite, enables no LFS, and mounts two runner-generated secrets by URI. +Registration and password Basic auth are disabled, while MFA is enforced; +hooks, webhooks, migrations, packages, Actions, OAuth2/OpenID, federation and +update checks are disabled. HTTP Git with PAT remains the intended trusted +client path; PAT clone/push acceptance belongs to the later reviewed +user/repository restoration transition, not this empty installation. LFS may +only be enabled by that later transition after its secret and restart behavior +has been accepted against the pinned runtime. + +Plan and apply fail closed unless all live prerequisites are already true: + +- `/volume1/docker/nodedc-gitea`, the `nodedc-gitea` project, and any stale + `nodedc-gitea_internal` network are absent; TCP/4022 is closed and Docker + publishes neither TCP/3000 nor TCP/4022; +- Docker Server is exactly 24.0.2, Compose is at least 2.20.1, and the exact + pinned image ID and RepoDigest are local for linux/amd64 with image user + `1000:1000`; +- the separate root-owned Nginx bridge + `/usr/local/etc/nginx/conf.d/http.nodedc-gitea-uds.conf` matches its exact + reviewed bytes, owner/mode and SHA-256; Nginx 1.23.1 configuration validates, + and only its root/uid-1023 processes own `127.0.0.1:3000`. The bridge proxies + to the Unix socket, preserves Host/X-Real-IP/XFF/XFP, rate-limits login and + bounds concurrent requests without limiting Git request bodies; +- DSM reverse-proxy UUID `5bc46027-0307-4261-af7e-4f94a3c508c9` persistently + and in generated nginx config routes `git.dcserve.ru:443` to + `127.0.0.1:3000`; the generated vhost must also retain its exact host guard + and canonical Host/X-Real-IP/XFF/XFP forwarding directives, so the UDS + bridge never trusts a client-supplied `X-Real-IP`; +- the emergency broad INPUT DROP for TCP/3000 has been removed, while the + legacy `172.22.0.222:3000` OUTPUT DROP and TCP/4022 INPUT DROP remain; +- any legacy container named `gitea` is stopped with restart policy `no`. + +Changing DSM reverse proxy/firewall rules, acquiring evidence from the old +root, restoring trusted users/repositories, and loading the pinned image are +separate reviewed transitions. This component never reads or mounts +`/volume1/docker/gitea` and never starts the legacy container. Failed apply +stops/removes only the `nodedc-gitea` candidate, proves that it is absent, and +quarantines the complete newly created root (including data/config/socket/secrets) for +evidence, and restores the absent source/root state without deleting runtime +evidence. If candidate absence cannot be proven, rollback preserves the root in +place and records reconciliation-required instead of disconnecting live bind +mounts. + +Run the policy suite before publishing: + +```bash +PYTHONDONTWRITEBYTECODE=1 \ + python3 infra/deploy-runner/test_gitea_fresh_install.py -v +``` + +## Gitea incident salvage (clean-state recovery) + +The additive `gitea-incident-salvage` slice is the reviewed recovery design for +incident `gitea-20260814`. It does not upgrade or mount the compromised state. +The candidate starts with a new SQLite database, new config, new runtime +secrets, new email addresses and new password hashes. The exact v2 decision +bundle is embedded as inert, digest-bound data: + +- 2 new active identities (`dctouch`, admin; `SILVER`, non-admin), each with a + new root-contained one-time password and mandatory password change; +- 8 new locked identities with random discarded passwords, inactive, + restricted and prohibited from login; +- 45 exact repositories: 32 private `dctouch` repositories and 13 public + `SILVER` repositories. The other 962 legacy users and 2013 legacy repository + rows are not imported into the clean database. + +The successor v3 descriptor embeds the owner-confirmed, canonical +`confirmed-disposition-v1.json` and the additive +`confirmed-closure-disposition-v1.json`. Both are bound to the exact snapshot, +database, identity decision, reference manifest, unsupported-state report, +schema catalog and semantic-topics hashes. The original disposition's 105 +reference decisions are explicit rows, not namespace wildcards: 85 normal +heads, 4 tags and 4 wiki heads are eligible for later live reconstruction; the +exact 5 pull refs and 7 remote refs are sealed archive-only evidence and can +never be auto-promoted. The closure disposition authorizes only bounded +plan-time inventory and records the target policy for access, collaboration, +issue/PR metadata, attachments, releases, labels, projects, units, Packages +and Actions. Observed hashes do not become activation authority, and the new +closure-report digest is deliberately unpinned until separate review. + +Build the deterministic review artifact locally with: + +```bash +node infra/deploy-runner/build-gitea-incident-salvage-artifact.mjs \ + gitea-incident-salvage-subrelation-closure-20260814-006 +``` + +The original v1 descriptor and `...salvage-20260814-001.tgz` remain immutable. +The exact `...salvage-disposition-20260814-002.tgz` also remains immutable and +must not be retried: its plan failed safely because it assumed every nonempty +`repository.topics` TEXT value was a JSON array. A bounded immutable-database +classification proved all 45 kept rows are instead exact four-byte canonical +JSON `null`, Gitea's semantic empty representation. The `...topics-...-003` +artifact is the forward fix with a new patch id and digest. +That exact `...topics-...-003` artifact is now also an immutable predecessor. +The local pre-audit `...closure-...-004` draft must not be staged: review found +orphan-join and nullable-Actions gaps. The local +`...closure-reviewed-...-005` artifact is also stale and must not be staged: +its primary closure was fail-closed, but it retained a broad polymorphic- +subrelation blocker. The additive `...subrelation-closure-...-006` successor +closes the concrete comment/history/merger relations in the pinned snapshot +schema and is the only artifact produced by the current builder. +The successor builder refuses to overwrite an existing artifact basename and +publishes from an exclusively created same-directory temporary file using an +atomic no-clobber hard link. A failed build removes only its private temporary +file. The disposition is sorted, compact ASCII JSON with one terminal LF. + +The runtime pin is +`docker.gitea.com/gitea:1.27.2-rootless@sha256:7de5f49a…3abd2c`, with exact +linux/amd64 OCI image ID `sha256:272085a8…bc5c4c6`. It runs as `1000:1000`, +uses the direct Gitea binary, `network_mode: none`, no Docker ports, no SSH and +no LFS. Candidate restart policy is `no`. The existing reviewed Nginx bridge +cannot reach the candidate while the socket parent is mode `0700`; publication +is an explicit post-acceptance change to `0750`, followed by public TLS and +smart-HTTP checks. Only after those checks may restart policy become +`unless-stopped`. The transition never changes or enforces two-factor +authentication. + +The snapshot boundary is exact: UUID +`f5a3fe3a-93ea-bb4d-847f-6221a6bcbc9f`, immutable SQLite SHA-256 +`8db9e74a…1a13052`, and the 972/2058 decisions must recompute from the database +row by row. The runner enumerates only each v2 `repo_relative_path` and +`wiki_relative_path`. It rejects symlinks, special files, hard links, NOCOW, +alternates, shallow/replace/grafts/worktrees, promisor state, unknown object +material and incomplete pack/index pairs. One preloaded mount inventory also +guards the `objects` and `refs` roots plus every traversed directory and file: +all must remain on the trusted snapshot device, outside nested mountpoints and, +for directories, outside nested Btrfs subvolumes. Traversal errors fail closed. +Forbidden-path probes (`objects/info/alternates`, grafts, replace refs and the +other listed sentinels) lstat each path component without following links and +validate every ancestor before probing its child; a symlink or mount boundary +therefore stops the plan before any lookup can escape the snapshot tree. + +Only three exact derived-file classes observed in the bounded 45-repository +inventory are excluded: dumb-HTTP `objects/info/packs`, +`objects/pack/pack-.bitmap` with its exact complete pack/index pair, and +monolithic `objects/info/commit-graph`. Each must be a bounded regular, +single-link, non-NOCOW file. The reference manifest records its exact path, +class and byte size, but the runner neither reads nor copies the excluded +bytes. Unobserved `.rev`, MIDX and split commit-graph layouts stay fail-closed; +`.promisor`, `.mtimes`, `.keep`, unknown and temporary lookalikes are not +classified as disposable accelerators. + +The same inventory found receive-pack quarantine directories matching only an +exact direct child `objects/tmp_objdir-incoming-[A-Za-z0-9]{6}`. A match must +retain the reviewed uid/gid `1000:1000`, mode `0755`, single-link and non-NOCOW +directory metadata. Its exact path and lstat fields are recorded, then the +subtree is pruned without reading, hashing, copying or reflinking contents; the +sealed read-only incident snapshot retains the forensic copy. Any name, +metadata, mount or subvolume drift remains a hard stop. A reviewed reference/OID +manifest is required; raw HEAD/refs files are never copied. The intended data +path is same-filesystem per-file `--reflink=always`, with no byte-copy fallback +and a bounded exclusive-allocation gate. + +This slice is intentionally **NO-GO for apply** at the current review point. +`plan` inventories and prints the exact ref-manifest digest, byte length and +canonical compact JSON. The JSON is the reviewable, hash-bound reference/OID +artifact: it contains approved repository identities, ref names/OIDs, object +totals and stat-bound derived/quarantine exclusions, but no Git object bytes, +hooks or config. The same plan prints the canonical unsupported-state report +and a separate incident-closure report, each with its SHA-256, byte length and +compact ASCII JSON. The closure report is evidence only: its disposition keeps +`expectedSha256` and `expectedBytes` null and retains an explicit review-pin +blocker. + +The reviewed plan evidence bound by the disposition is reference manifest +`9cddaf0e…a311` (36,010 bytes), unsupported-state report +`4b2cecf8…106ac` (119,468 bytes), and schema catalog +`b5e3b677…b6db`. The report has exact 45 repository rows with empty +`schema_missing`, `schema_mismatch`, and `anomalies` arrays. These facts close +only the owner's material-state decision; fsck/reachability, physical blob +inventory, sanitized archive creation and candidate acceptance remain separate +verifiers. + +The unsupported-state report has one deterministic row for every one of the +45 kept repository IDs, including explicit zeroes. It separates direct +repository-relation counts, directional base/head PR counts, repo-unit type +counts, text-metadata presence booleans and denormalized numeric hints; it does +not add overlapping relations into a misleading grand total. LFS rows are +reported per repository with validated logical byte sums and distinct-OID +aggregates, including size conflicts and sharing with non-kept repositories. +Every association sharing a kept OID must also have an integer repository ID +that resolves to the exact snapshot `repository` table; invalid and orphan +owners make shared/unique byte aggregates unavailable and add a data-anomaly +blocker. +Attachments are reported per repository with validated logical byte sums and +explicit, overlapping issue/comment/release/unlinked link splits. Neither +category claims that a physical blob exists; no title, body, note, comment, +config, URL, token, key, secret, hash payload or blob content is selected or +exported. + +The report also embeds the exact `PRAGMA table_xinfo` catalog and explicit +label-to-table-to-repository-column map for a code-owned allowlist of relevant +tables. `sqlite_master` must prove an ordinary, non-virtual table before any +row query; SQLite 3.37+ adds `PRAGMA table_list` ordinary-main-table proof. On +an older compatible SQLite, the weaker object-kind capability is recorded and +adds a coverage blocker. `query_only=ON` and `trusted_schema=OFF` are read back +before the snapshot connection is accepted. Only column identifiers and +structural metadata are exported, never DDL, defaults or row values. The v3 +closure inventory follows code-owned, schema-attested primary joins for the +issue/PR dependency tables (comments, assignees, content histories, labels, issue +users/watches, reactions, reviews/state, dependencies, notifications, +stopwatches, tracked time and auto-merge), plus projects/boards, releases, +attachments, Packages and Actions. It emits only IDs required to prove joins, +actor classes, counts, declared sizes and SQLite byte lengths; it never selects +or prints text bodies, names, URLs, package metadata, workflow payloads, +credentials, tokens, secrets or logs. Package and Actions target features stay +disabled and no legacy row is importable. The successor additionally closes +every concrete FK-like comment field in the pinned schema (labels, projects, +milestones, tracked time, assignees, dependency, review and cross-reference), +content-history-to-comment, review-team and pull-merger provenance. Target +repository and local actor IDs are classified through the exact 2058/972 +decisions. Migrated external-author IDs are never treated as local users and +are represented only by presence counts and author-name byte lengths. A +nonzero comment-assignee or review-reviewer team must resolve to an exact team +and organization; because no organization/team allowlist exists, its row, +team and organization IDs enter a sealed HOLD inventory and add the narrowly +scoped `issue-pr-team-mapping-hold` blocker. Orphans, conflicting user/team +identities, invalid cross-reference pairs and schema/type drift fail closed. +The report is still not a physical-file or sanitized-archive verifier, so +those separate blockers remain explicit. +The schema-catalog digest is recorded as observed evidence but deliberately +remains unpinned as reviewed activation authority. Schema absence/type drift, +invalid numeric/OID state or aggregate overflow is a hard error. `apply` fails +before creating `/volume1/docker/nodedc-gitea` until all of the following are +registered: + +- root-owned Btrfs proof that the named snapshot UUID is read-only; +- exact stopped/restart-`no` legacy container image ID, image ref and sole + `/volume1/docker/gitea:/data` bind inspect; +- a separately verified exact reference/OID manifest digest, fsck, + reachability and selected-object reconstruction; +- immutable sanitized archives for issues, PRs, comments, releases, labels, + projects and repository descriptions, with no legacy row import; +- physical attachment inventory and reachable LFS pointer/OID/size/SHA proof; +- kept-user mapping before any collaboration recreation; and +- package/Actions schema plus physical zero closure and target unit-policy + acceptance. + +All 25 direct repository relations have explicit dispositions, including the +15 exact zero-and-drop categories. Pull base/head counts remain directional +views of the same five PRs. All repository numeric hints are dropped and +recomputed. Repo-unit types 1–10 are total: clean types 1–5 and 8 are recreated +once per repository; external wiki/tracker are forbidden; Packages (9) and +Actions (10) are absent in the target and remain globally disabled. Legacy +unit config is never imported. The topics verifier accepts exact JSON `null` +as semantic empty state, or a canonical JSON array of sorted unique lowercase +names matching `^[a-z0-9][-.a-z0-9]*$` with at most 35 UTF-8 bytes. SQL NULL, +whitespace/case variants, quoted `"null"`, objects, numbers, booleans and +noncanonical arrays fail closed. The current snapshot must prove exactly 45 +serialized JSON nulls, zero serialized arrays, zero material topics and zero +`repo_topic` rows; target cache is rebuilt from relations. + +The activation hook remains frozen until these blockers are closed and the +full clean-database recreation, reflink reconstruction, doctor/fsck, +credential-table zeroing, rollback and public acceptance path has its own +runtime smoke. The current reviewed source identities are: + +- runner `nodedc-deploy` SHA-256 + `c766985aa02fe911fa5a873717276f3503d9cecded448e41dc6945030dcbce9e`; +- deterministic builder SHA-256 + `b92076583aeab854015a05deb9d942c8dfcf3ca475fd3d52115d402d6f61acca`; +- salvage policy test SHA-256 + `99b96b6179286a78e7d53ee6a923fc092e3471a8cece382a472449d455ad212c`; +- fresh-install policy test SHA-256 + `20653c640142c25e7d2ef712048acc7889c56e1d519e2d03b5cad4127e48947c`; +- confirmed disposition SHA-256 + `0a066724bcf6e4933133db6cab6cc273393e3c262dd00dda0bbf9ceebd84f78c`; +- confirmed closure disposition SHA-256 + `7ed66d9848268431a703fe24b22c41afbaa7c5ff48949604d6fc448d93e0d243`; +- successor v3 descriptor SHA-256 + `9b98eb1a1640fd5569cf051a621837379b167eff4527313a43a0a851e7cc181a`; +- deterministic local review artifact + `nodedc-gitea-gitea-incident-salvage-subrelation-closure-20260814-006.tgz` + SHA-256 + `d3e598cf892b1371912dcd9cef64caa991dd23613e48fe979d0f95d493c23d53`. + +Only after that exact runner is separately reviewed, promoted to the +root-owned runner path and its installed SHA-256 is re-attested may the exact +artifact be staged into the canonical inbox for a canonical `plan` command. +Staging is evidence collection only; it grants no authority to run `apply`. +The legacy identity is now pinned to the observed ref `gitea/gitea:latest` +and immutable image ID +`sha256:bf95d9a45ce4fe38b027d051cdc4a4bc531513489fa6244af4074efbb1c376d6`. +The mutable tag is provenance only. Every plan re-inspects exact name +`/gitea`, stopped state, restart `no`, and the sole RW bind +`/volume1/docker/gitea:/data`; drift is a hard stop. + +The successor plan must report these exact remaining blockers: + +- `attachment-physical-verifier-pending`; +- `candidate-root-activation-hard-frozen`; +- `closure-report-review-pin-pending`; +- `collaboration-kept-user-mapping-verifier-pending`; +- `forensic-ref-archive-verifier-pending`; +- `issue-pr-metadata-sanitized-archive-verifier-pending`; +- `lfs-reachable-pointer-physical-verifier-pending`; +- `package-action-physical-closure-verifier-pending`; +- `reference-manifest-fsck-reachability-verifier-pending`; +- `repository-object-reconstruction-verifier-pending`; +- `target-unit-policy-acceptance-pending`; +- `unsupported-schema-catalog-verifier-pending`. + +Any other blocker or identity drift is also a stop. `apply` remains prohibited +and hard-frozen before root creation until a subsequent reviewed runner +revision closes every blocker and implements and smoke-tests the complete +activation/rollback path. The successor files must be independently reviewed +before any runner promotion or inbox staging. + +Run the bounded policy suite with: + +```bash +PYTHONDONTWRITEBYTECODE=1 \ + python3 infra/deploy-runner/test_gitea_incident_salvage.py -v +``` + +The frozen local cutoff passes 54 incident-salvage tests and 26 fresh-install +tests, plus Python compilation and `git diff --check`. + `n8n-private-extension` is a staging-only trust boundary for reviewed offline n8n private-node releases. Its artifact may contain exactly one digest-bound `n8n-nodes-ndc` release with `package.tgz`, `release.json` and diff --git a/infra/deploy-runner/build-gitea-fresh-install-artifact.mjs b/infra/deploy-runner/build-gitea-fresh-install-artifact.mjs new file mode 100644 index 0000000..c0a8b8f --- /dev/null +++ b/infra/deploy-runner/build-gitea-fresh-install-artifact.mjs @@ -0,0 +1,222 @@ +#!/usr/bin/env node +import { createHash } from "node:crypto"; +import { spawnSync } from "node:child_process"; +import { + cp, + lstat, + mkdir, + mkdtemp, + readFile, + rm, + writeFile, +} from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { dirname, join, resolve } from "node:path"; +import { fileURLToPath } from "node:url"; + +const scriptDir = dirname(fileURLToPath(import.meta.url)); +const fixtureRoot = resolve(scriptDir, "fixtures/gitea"); +const artifactDir = resolve( + process.env.NODEDC_DEPLOY_ARTIFACT_DIR + || resolve(scriptDir, "../deploy-artifacts"), +); +const [patchId = "gitea-fresh-install-20260813-001", ...extra] = + process.argv.slice(2); + +if (extra.length || !/^[A-Za-z0-9._-]{1,96}$/.test(patchId)) { + throw new Error("usage: build-gitea-fresh-install-artifact.mjs [patch-id]"); +} + +const composeRelative = "docker-compose.gitea.yml"; +const descriptorRelative = "deployment/gitea-fresh-install-v1.json"; +const files = [composeRelative, descriptorRelative]; +const stage = await mkdtemp(join(tmpdir(), "nodedc-gitea-artifact-")); +const payload = join(stage, "payload"); +const target = join(artifactDir, `nodedc-gitea-${patchId}.tgz`); + +await assertFixtureContract(); + +try { + await mkdir(payload, { recursive: true }); + for (const relative of files) { + const source = resolve(fixtureRoot, relative); + const sourceStat = await lstat(source); + if (sourceStat.isSymbolicLink() || !sourceStat.isFile()) { + throw new Error(`gitea_fixture_type_rejected:${relative}`); + } + const destination = join(payload, relative); + await mkdir(dirname(destination), { recursive: true }); + await cp(source, destination, { force: true, verbatimSymlinks: true }); + } + await writeFile( + join(stage, "manifest.env"), + `id=${patchId}\ncomponent=gitea\ntype=app-overlay\n`, + "utf8", + ); + await writeFile(join(stage, "files.txt"), `${files.join("\n")}\n`, "utf8"); + await mkdir(artifactDir, { recursive: true }); + + const tar = spawnSync( + "python3", + ["-c", canonicalTarScript(), target, stage], + { encoding: "utf8", maxBuffer: 16 * 1024 * 1024 }, + ); + if (tar.status !== 0) { + throw new Error(`tar_failed:${tar.stderr || tar.stdout}`); + } + + const digest = sha256(await readFile(target)); + console.log(JSON.stringify({ + ok: true, + patchId, + artifact: target, + sha256: digest, + component: "gitea", + entries: files, + services: ["gitea"], + image: "docker.gitea.com/gitea:1.27.1-rootless@sha256:89dc3c214b3992e5bb01e05ad21139d7a8b302d3ea3d8942d3f7e904e92af148", + installMode: "fresh-only", + database: "fresh-sqlite-only", + lfs: "disabled-pending-reviewed-restore-transition", + transport: "unix:/run/gitea/gitea.sock", + networkMode: "none", + minimumComposeVersion: "2.20.1", + preserved: ["legacy-gitea-root-unread-and-untouched"], + excluded: [ + "secrets", + "runtime-data", + "database", + "repositories", + "users", + "tokens", + "ssh-keys", + "hooks", + ], + }, null, 2)); +} finally { + await rm(stage, { recursive: true, force: true }); +} + +async function assertFixtureContract() { + const composeBytes = await readFile(resolve(fixtureRoot, composeRelative)); + const compose = composeBytes.toString("utf8"); + const descriptor = JSON.parse( + await readFile(resolve(fixtureRoot, descriptorRelative), "utf8"), + ); + const expectedImage = + "docker.gitea.com/gitea:1.27.1-rootless@sha256:89dc3c214b3992e5bb01e05ad21139d7a8b302d3ea3d8942d3f7e904e92af148"; + const required = [ + `image: ${expectedImage}`, + "platform: linux/amd64", + "pull_policy: never", + "network_mode: none", + 'user: "1000:1000"', + "stop_grace_period: 30s", + "driver: json-file", + 'max-size: "10m"', + 'max-file: "3"', + "GITEA__server__PROTOCOL: http+unix", + "GITEA__server__HTTP_ADDR: /run/gitea/gitea.sock", + 'GITEA__server__UNIX_SOCKET_PERMISSION: "0666"', + "GITEA__server__LOCAL_ROOT_URL: http://unix/", + 'GITEA__server__DISABLE_SSH: "true"', + 'GITEA__server__LFS_START_SERVER: "false"', + 'GITEA__server__LFS_ALLOW_PURE_SSH: "false"', + "GITEA__security__SECRET_KEY_URI: file:/run/secrets/gitea_secret_key", + "GITEA__security__INTERNAL_TOKEN_URI: file:/run/secrets/gitea_internal_token", + "GITEA__security__TWO_FACTOR_AUTH: enforced", + 'GITEA__security__REVERSE_PROXY_LIMIT: "1"', + "GITEA__security__ALLOWED_HOST_LIST: loopback", + "GITEA__security__REVERSE_PROXY_TRUSTED_PROXIES: 127.0.0.0/8,::1/128", + 'GITEA__service__DISABLE_REGISTRATION: "true"', + 'GITEA__service__ENABLE_REVERSE_PROXY_AUTHENTICATION: "false"', + 'GITEA__service__ENABLE_REVERSE_PROXY_AUTHENTICATION_API: "false"', + 'GITEA__service__ENABLE_REVERSE_PROXY_AUTO_REGISTRATION: "false"', + 'GITEA__service__ENABLE_BASIC_AUTHENTICATION: "false"', + 'GITEA__admin__DISABLE_REGULAR_ORG_CREATION: "true"', + "GITEA__admin__USER_DISABLED_FEATURES: deletion,manage_ssh_keys,manage_gpg_keys,change_username", + 'GITEA__security__DISABLE_GIT_HOOKS: "true"', + 'GITEA__security__DISABLE_WEBHOOKS: "true"', + 'GITEA__repository__DISABLE_MIGRATIONS: "true"', + 'GITEA__packages__ENABLED: "false"', + 'GITEA__oauth2__ENABLED: "false"', + 'GITEA__openid__ENABLE_OPENID_SIGNIN: "false"', + 'GITEA__cron.update_checker__ENABLED: "false"', + "source: /volume1/docker/nodedc-gitea/socket", + "target: /run/gitea", + "create_host_path: false", + "read_only: true", + "no-new-privileges:true", + ]; + for (const fragment of required) { + if (!compose.includes(fragment)) { + throw new Error(`gitea_compose_boundary_missing:${fragment}`); + } + } + for (const forbidden of [ + "4022", + "2222:2222", + "0.0.0.0:3000", + "ports:", + "networks:", + "/var/run/docker.sock", + "/volume1/docker/gitea", + "privileged: true", + "pull_policy: always", + "__FILE", + "GITEA__security__SECRET_KEY:", + "GITEA__security__INTERNAL_TOKEN:", + "GITEA__server__LFS_JWT_SECRET:", + "GITEA__server__LFS_JWT_SECRET_URI", + "gitea_lfs_jwt_secret", + "lfs-jwt-secret", + "GITEA__server__REVERSE_PROXY_LIMIT", + "GITEA__server__REVERSE_PROXY_TRUSTED_PROXIES", + "GITEA__security__ENABLE_REVERSE_PROXY_AUTHENTICATION", + "GITEA__security__ENABLE_REVERSE_PROXY_AUTHENTICATION_API", + "GITEA__security__ENABLE_REVERSE_PROXY_AUTO_REGISTRATION", + "GITEA__service__DISABLE_REGULAR_ORG_CREATION", + "GITEA__service__USER_DISABLED_FEATURES", + ]) { + if (compose.includes(forbidden)) { + throw new Error(`gitea_compose_boundary_violation:${forbidden}`); + } + } + if ( + descriptor.schemaVersion !== "nodedc.gitea.fresh-install.v1" + || descriptor.action !== "fresh-install" + || descriptor.component !== "gitea" + || descriptor.compose?.sha256 !== sha256(composeBytes) + || descriptor.runtime?.image !== expectedImage + || descriptor.runtime?.minimumComposeVersion !== "2.20.1" + || descriptor.runtime?.lfs !== "disabled-pending-reviewed-restore-transition" + || descriptor.runtime?.transport !== "unix:/run/gitea/gitea.sock" + || descriptor.runtime?.networkMode !== "none" + || descriptor.runtime?.logging !== "bounded-json-file-10m-x3" + || descriptor.runtime?.stopGracePeriod !== "30s" + || descriptor.trust?.artifactSecrets !== "forbidden" + || descriptor.trust?.legacyRootAccess !== "forbidden" + ) { + throw new Error("gitea_descriptor_contract_mismatch"); + } +} + +function sha256(bytes) { + return createHash("sha256").update(bytes).digest("hex"); +} + +function canonicalTarScript() { + return [ + "import gzip,io,pathlib,sys,tarfile", + "root=pathlib.Path(sys.argv[2])", + "with open(sys.argv[1],'wb') as out:", + " with gzip.GzipFile(filename='',mode='wb',fileobj=out,compresslevel=9,mtime=0) as gz:", + " with tarfile.open(fileobj=gz,mode='w',format=tarfile.PAX_FORMAT) as tar:", + " for top in ('manifest.env','files.txt','payload'):", + " p=root/top; paths=[p]+(sorted(p.rglob('*')) if p.is_dir() else [])", + " for x in paths:", + " info=tar.gettarinfo(str(x),arcname=x.relative_to(root).as_posix())", + " info.uid=info.gid=0; info.uname=info.gname='root'; info.mtime=0; info.mode=0o755 if info.isdir() else 0o644", + " with (open(x,'rb') if info.isfile() else io.BytesIO()) as src: tar.addfile(info,src if info.isfile() else None)", + ].join("\n"); +} diff --git a/infra/deploy-runner/build-gitea-incident-salvage-artifact.mjs b/infra/deploy-runner/build-gitea-incident-salvage-artifact.mjs new file mode 100644 index 0000000..ce43663 --- /dev/null +++ b/infra/deploy-runner/build-gitea-incident-salvage-artifact.mjs @@ -0,0 +1,322 @@ +#!/usr/bin/env node +import { createHash } from "node:crypto"; +import { spawnSync } from "node:child_process"; +import { + cp, + link, + lstat, + mkdir, + mkdtemp, + readFile, + rm, + writeFile, +} from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { basename, dirname, join, resolve } from "node:path"; +import { fileURLToPath } from "node:url"; + +const scriptDir = dirname(fileURLToPath(import.meta.url)); +const fixtureRoot = resolve(scriptDir, "fixtures/gitea-salvage"); +const decisionRoot = resolve( + scriptDir, + "../../../security-incidents/gitea-20260814/confirmed-decisions-v2", +); +const artifactDir = resolve( + process.env.NODEDC_DEPLOY_ARTIFACT_DIR + || resolve(scriptDir, "../deploy-artifacts"), +); +const [patchId = "gitea-incident-salvage-subrelation-closure-20260814-006", ...extra] = + process.argv.slice(2); + +if (extra.length || !/^[A-Za-z0-9._-]{1,96}$/.test(patchId)) { + throw new Error( + "usage: build-gitea-incident-salvage-artifact.mjs [patch-id]", + ); +} + +const composeRelative = "docker-compose.gitea.yml"; +const descriptorRelative = "deployment/gitea-incident-salvage-v3.json"; +const decisionPrefix = "deployment/gitea-incident-salvage"; +const dispositionRelative = `${decisionPrefix}/confirmed-disposition-v1.json`; +const closureDispositionRelative = + `${decisionPrefix}/confirmed-closure-disposition-v1.json`; +const descriptorSha256 = + "9b98eb1a1640fd5569cf051a621837379b167eff4527313a43a0a851e7cc181a"; +const dispositionSha256 = + "0a066724bcf6e4933133db6cab6cc273393e3c262dd00dda0bbf9ceebd84f78c"; +const closureDispositionSha256 = + "7ed66d9848268431a703fe24b22c41afbaa7c5ff48949604d6fc448d93e0d243"; +const decisionFiles = [ + ["confirmed-decision.json", "dc9528462624158eb44218d37cc7054d551ca2d7ded562592982aa3f34c9fc2a"], + ["users.decisions.csv", "e3b82f1073a86eea9e567edff062dd1689d21ec0edcf3ed92e844da9351ee8b6"], + ["repositories.decisions.csv", "76b4bae2ab5cec490330c19bfc5ae9429abf7636705ae028c1c64fd54a6a0493"], +]; +const files = [ + composeRelative, + descriptorRelative, + dispositionRelative, + closureDispositionRelative, + ...decisionFiles.map(([name]) => `${decisionPrefix}/${name}`), +]; +const stage = await mkdtemp(join(tmpdir(), "nodedc-gitea-salvage-artifact-")); +const payload = join(stage, "payload"); +const target = join(artifactDir, `nodedc-gitea-${patchId}.tgz`); +const targetTemporary = join( + artifactDir, + `.${basename(target)}.${process.pid}.tmp`, +); + +await assertSourceContract(); + +try { + await mkdir(payload, { recursive: true }); + for (const relative of [ + composeRelative, + descriptorRelative, + dispositionRelative, + closureDispositionRelative, + ]) { + await copyRegularFile(resolve(fixtureRoot, relative), join(payload, relative), relative); + } + for (const [name] of decisionFiles) { + await copyRegularFile( + resolve(decisionRoot, name), + join(payload, decisionPrefix, name), + `${decisionPrefix}/${name}`, + ); + } + await writeFile( + join(stage, "manifest.env"), + `id=${patchId}\ncomponent=gitea\ntype=app-overlay\n`, + "utf8", + ); + await writeFile(join(stage, "files.txt"), `${files.join("\n")}\n`, "utf8"); + await mkdir(artifactDir, { recursive: true }); + + const tar = spawnSync( + "python3", + ["-c", canonicalTarScript(), targetTemporary, stage], + { encoding: "utf8", maxBuffer: 16 * 1024 * 1024 }, + ); + if (tar.status !== 0) { + throw new Error(`tar_failed:${tar.stderr || tar.stdout}`); + } + try { + await link(targetTemporary, target); + } catch (error) { + if (error?.code === "EEXIST") { + throw new Error("gitea_salvage_artifact_target_already_exists"); + } + throw error; + } + + console.log(JSON.stringify({ + ok: true, + patchId, + artifact: target, + sha256: sha256(await readFile(target)), + component: "gitea", + transition: "clean-state-incident-salvage", + entries: files, + services: ["gitea"], + image: "docker.gitea.com/gitea:1.27.2-rootless@sha256:7de5f49ada687b8c8d2938f547cdb7634839764ba51f297457bae35cee3abd2c", + decisions: { + users: { active: 2, locked: 8, delete: 962 }, + repositories: { keep: 45, delete: 2013 }, + }, + disposition: { + sha256: dispositionSha256, + forensicRefs: 105, + liveRefs: 93, + archiveOnlyRefs: 12, + }, + closureDisposition: { + sha256: closureDispositionSha256, + state: "plan-report-review-pending", + }, + stateBoundary: "new-database-config-secrets-identities", + repositoryBoundary: "exact-45-object-and-validated-ref-material-only", + networkMode: "none", + transport: "unix:/run/gitea/gitea.sock", + stagePolicy: "after-exact-runner-promotion-plan-only", + applyPolicy: "hard-frozen-before-root-creation", + }, null, 2)); +} finally { + await rm(targetTemporary, { force: true }); + await rm(stage, { recursive: true, force: true }); +} + +async function copyRegularFile(source, destination, label) { + const sourceStat = await lstat(source); + if (sourceStat.isSymbolicLink() || !sourceStat.isFile()) { + throw new Error(`gitea_salvage_source_type_rejected:${label}`); + } + await mkdir(dirname(destination), { recursive: true }); + await cp(source, destination, { force: true, verbatimSymlinks: true }); +} + +function canonicalJson(value) { + if (Array.isArray(value)) { + return value.map(canonicalJson); + } + if (value && typeof value === "object") { + return Object.fromEntries( + Object.keys(value).sort().map((key) => [key, canonicalJson(value[key])]), + ); + } + return value; +} + +async function assertSourceContract() { + for (const [name, expected] of decisionFiles) { + const bytes = await readFile(resolve(decisionRoot, name)); + if (sha256(bytes) !== expected) { + throw new Error(`gitea_salvage_decision_digest_mismatch:${name}`); + } + } + const composeBytes = await readFile(resolve(fixtureRoot, composeRelative)); + const compose = composeBytes.toString("utf8"); + const descriptorBytes = await readFile( + resolve(fixtureRoot, descriptorRelative), + ); + const descriptor = JSON.parse(descriptorBytes.toString("utf8")); + if (sha256(descriptorBytes) !== descriptorSha256) { + throw new Error("gitea_salvage_descriptor_digest_mismatch"); + } + const dispositionBytes = await readFile( + resolve(fixtureRoot, dispositionRelative), + ); + const disposition = JSON.parse(dispositionBytes.toString("utf8")); + if (sha256(dispositionBytes) !== dispositionSha256) { + throw new Error("gitea_salvage_disposition_digest_mismatch"); + } + const canonicalDisposition = Buffer.from( + `${JSON.stringify(canonicalJson(disposition))}\n`, + "utf8", + ); + if (!dispositionBytes.equals(canonicalDisposition)) { + throw new Error("gitea_salvage_disposition_not_canonical"); + } + const closureDispositionBytes = await readFile( + resolve(fixtureRoot, closureDispositionRelative), + ); + const closureDisposition = JSON.parse(closureDispositionBytes.toString("utf8")); + if (sha256(closureDispositionBytes) !== closureDispositionSha256) { + throw new Error("gitea_salvage_closure_disposition_digest_mismatch"); + } + const canonicalClosureDisposition = Buffer.from( + `${JSON.stringify(canonicalJson(closureDisposition))}\n`, + "utf8", + ); + if (!closureDispositionBytes.equals(canonicalClosureDisposition)) { + throw new Error("gitea_salvage_closure_disposition_not_canonical"); + } + const expectedImage = + "docker.gitea.com/gitea:1.27.2-rootless@sha256:7de5f49ada687b8c8d2938f547cdb7634839764ba51f297457bae35cee3abd2c"; + for (const required of [ + `image: ${expectedImage}`, + "platform: linux/amd64", + "pull_policy: never", + "network_mode: none", + 'user: "1000:1000"', + "entrypoint:", + "- /usr/local/bin/gitea", + "- /etc/gitea/app.ini", + "read_only: true", + "cap_drop:", + "- ALL", + "no-new-privileges:true", + "source: /volume1/docker/nodedc-gitea/data", + "target: /data", + "source: /volume1/docker/nodedc-gitea/config", + "target: /etc/gitea", + "source: /volume1/docker/nodedc-gitea/socket", + "target: /run/gitea", + "create_host_path: false", + ]) { + if (!compose.includes(required)) { + throw new Error(`gitea_salvage_compose_boundary_missing:${required}`); + } + } + for (const forbidden of [ + "ports:", + "networks:", + "/var/run/docker.sock", + "/volume1/docker/gitea", + "privileged: true", + "4022", + "TWO_FACTOR_AUTH", + "LFS_JWT_SECRET", + ]) { + if (compose.includes(forbidden)) { + throw new Error(`gitea_salvage_compose_boundary_violation:${forbidden}`); + } + } + if ( + descriptor.schemaVersion !== "nodedc.gitea.incident-salvage.v3" + || descriptor.action !== "clean-state-salvage" + || descriptor.compose?.sha256 !== sha256(composeBytes) + || descriptor.runtime?.image !== expectedImage + || descriptor.runtime?.networkMode !== "none" + || descriptor.runtime?.database !== "new-sqlite-1.27.2-only" + || descriptor.disposition?.file !== dispositionRelative + || descriptor.disposition?.sha256 !== dispositionSha256 + || descriptor.closureDisposition?.file !== closureDispositionRelative + || descriptor.closureDisposition?.sha256 !== closureDispositionSha256 + || descriptor.closureDisposition?.predecessorArtifactSha256 + !== "d6870b5583a2f329eadb4e6cda65fdf4d271532df5ffbf8bfb1403968a434672" + || descriptor.trust?.legacyDatabaseImported !== false + || descriptor.trust?.legacyCredentialsImported !== false + || descriptor.identity?.twoFactorAuthentication + !== "not-configured-by-transition" + ) { + throw new Error("gitea_salvage_descriptor_contract_mismatch"); + } + if ( + disposition.schemaVersion !== "nodedc.gitea.incident-disposition.v1" + || disposition.incidentId !== "gitea-20260814" + || disposition.activation?.allowedOperation !== "canonical-plan-only" + || disposition.activation?.applyFrozen !== true + || disposition.activation?.freezeBoundary + !== "before-candidate-root-creation" + || disposition.referencePolicy?.forensicScope?.allDiscoveredRefs !== 105 + || disposition.referencePolicy?.exactDecisions?.length !== 105 + || disposition.referencePolicy?.liveRestore?.totalRefs !== 93 + || disposition.referencePolicy?.archiveOnly?.totalRefs !== 12 + ) { + throw new Error("gitea_salvage_disposition_contract_mismatch"); + } + if ( + closureDisposition.schemaVersion + !== "nodedc.gitea.incident-closure-disposition.v1" + || closureDisposition.incidentId !== "gitea-20260814" + || closureDisposition.activation?.allowedOperation !== "canonical-plan-only" + || closureDisposition.activation?.applyFrozen !== true + || closureDisposition.closureReport?.expectedSha256 !== null + || closureDisposition.predecessor?.dispositionSha256 !== dispositionSha256 + || closureDisposition.remainingBlockers + ?.includes("closure-report-review-pin-pending") !== true + ) { + throw new Error("gitea_salvage_closure_disposition_contract_mismatch"); + } +} + +function sha256(bytes) { + return createHash("sha256").update(bytes).digest("hex"); +} + +function canonicalTarScript() { + return [ + "import gzip,io,pathlib,sys,tarfile", + "root=pathlib.Path(sys.argv[2])", + "with open(sys.argv[1],'xb') as out:", + " with gzip.GzipFile(filename='',mode='wb',fileobj=out,compresslevel=9,mtime=0) as gz:", + " with tarfile.open(fileobj=gz,mode='w',format=tarfile.PAX_FORMAT) as tar:", + " for top in ('manifest.env','files.txt','payload'):", + " p=root/top; paths=[p]+(sorted(p.rglob('*')) if p.is_dir() else [])", + " for x in paths:", + " info=tar.gettarinfo(str(x),arcname=x.relative_to(root).as_posix())", + " info.uid=info.gid=0; info.uname=info.gname='root'; info.mtime=0; info.mode=0o755 if info.isdir() else 0o644", + " with (open(x,'rb') if info.isfile() else io.BytesIO()) as src: tar.addfile(info,src if info.isfile() else None)", + ].join("\n"); +} diff --git a/infra/deploy-runner/fixtures/gitea-salvage/deployment/gitea-incident-salvage-v1.json b/infra/deploy-runner/fixtures/gitea-salvage/deployment/gitea-incident-salvage-v1.json new file mode 100644 index 0000000..23dc71f --- /dev/null +++ b/infra/deploy-runner/fixtures/gitea-salvage/deployment/gitea-incident-salvage-v1.json @@ -0,0 +1,112 @@ +{ + "schemaVersion": "nodedc.gitea.incident-salvage.v1", + "action": "clean-state-salvage", + "component": "gitea", + "incidentId": "gitea-20260814", + "installRoot": "/volume1/docker/nodedc-gitea", + "compose": { + "file": "docker-compose.gitea.yml", + "project": "nodedc-gitea", + "service": "gitea", + "sha256": "2f031d5bfff4f42c73cabd8c94487ec3e4f1e1b0a96d3b7eec958904f735908a" + }, + "runtime": { + "image": "docker.gitea.com/gitea:1.27.2-rootless@sha256:7de5f49ada687b8c8d2938f547cdb7634839764ba51f297457bae35cee3abd2c", + "imageId": "sha256:272085a806e6d182352cdb011c0ebab1d2efc7ec45247de84de5659c7bc5c4c6", + "repoDigest": "docker.gitea.com/gitea@sha256:7de5f49ada687b8c8d2938f547cdb7634839764ba51f297457bae35cee3abd2c", + "platform": "linux/amd64", + "pullPolicy": "never", + "entrypoint": "/usr/local/bin/gitea", + "runAs": "1000:1000", + "transport": "unix:/run/gitea/gitea.sock", + "networkMode": "none", + "ssh": "disabled-no-published-port", + "lfs": "unsupported-hard-stop", + "database": "new-sqlite-1.27.2-only", + "logging": "bounded-json-file-10m-x3", + "stopGracePeriod": "30s", + "candidateRestartPolicy": "no", + "acceptedRestartPolicy": "unless-stopped" + }, + "decision": { + "schema": "nodedc.gitea.incident-decision/v2", + "manifestFile": "deployment/gitea-incident-salvage/confirmed-decision.json", + "manifestSha256": "dc9528462624158eb44218d37cc7054d551ca2d7ded562592982aa3f34c9fc2a", + "usersFile": "deployment/gitea-incident-salvage/users.decisions.csv", + "usersSha256": "e3b82f1073a86eea9e567edff062dd1689d21ec0edcf3ed92e844da9351ee8b6", + "repositoriesFile": "deployment/gitea-incident-salvage/repositories.decisions.csv", + "repositoriesSha256": "76b4bae2ab5cec490330c19bfc5ae9429abf7636705ae028c1c64fd54a6a0493", + "users": { + "active": 2, + "locked": 8, + "delete": 962 + }, + "repositories": { + "keep": 45, + "delete": 2013, + "dctouchPrivate": 32, + "silverPublic": 13 + } + }, + "snapshot": { + "root": "/volume1/.nodedc-security-snapshots/docker-gitea-incident-20260814", + "uuid": "f5a3fe3a-93ea-bb4d-847f-6221a6bcbc9f", + "readOnlyRequired": true, + "database": "gitea/gitea/gitea.db", + "databaseBytes": 182681600, + "databaseSha256": "8db9e74a5641662a808d8252c5d6c9de43fe9efd8687634bfbff2d4361a13052", + "repositoriesRoot": "gitea/git/repositories" + }, + "trust": { + "legacyRootMounted": false, + "legacyDatabaseImported": false, + "legacyConfigImported": false, + "legacyCredentialsImported": false, + "artifactSecrets": "forbidden", + "runtimeSecrets": "runner-generated", + "repositorySelection": "exact-v2-rows-only", + "repositoryCopy": "same-filesystem-per-file-reflink-object-and-validated-ref-material-only", + "forbiddenRepositoryMaterial": [ + "hooks", + "config", + "alternates", + "http-alternates", + "shallow", + "replace", + "grafts", + "worktrees", + "commondir", + "lfs" + ], + "unsupportedDatabaseState": "zero-material-rows-required-before-apply" + }, + "identity": { + "preserveNumericUserIds": false, + "preserveNumericRepositoryIds": false, + "oldToNewIdMapping": "/volume1/docker/nodedc-gitea/audit/identity-map.json", + "emails": "new-local-noreply-only", + "passwordHashes": "never-imported", + "activeBootstrapCredentials": "root-only-runner-state-require-change", + "lockedCredentials": "fresh-random-discarded-and-login-prohibited", + "twoFactorAuthentication": "not-configured-by-transition" + }, + "externalPrerequisites": { + "legacyContainer": "exact-inspect-pin-required-stopped-restart-no", + "publicProxy": "reviewed-nginx-uds-bridge-and-dsm-loopback-upstream", + "firewall": "legacy-isolation-retained-and-loopback-3000-ready", + "candidatePublicationGate": "socket-parent-uid1000-gid1023-mode0700", + "publishAction": "runner-chmod-socket-parent-0750-after-internal-acceptance", + "mustPassBeforeApply": true + }, + "acceptance": { + "users": "exact-10-row-by-row", + "repositories": "exact-45-row-by-row-and-visibility", + "credentialsSessionsKeysWebhooksActions": "zero", + "git": "fsck-all-and-public-smart-http-smoke", + "registration": "closed", + "ssh": "disabled", + "network": "none-no-ports", + "publicHttps": "required-before-ledger-success" + }, + "rollback": "stop-remove-candidate-prove-absent-quarantine-new-root-never-start-legacy" +} diff --git a/infra/deploy-runner/fixtures/gitea-salvage/deployment/gitea-incident-salvage-v2.json b/infra/deploy-runner/fixtures/gitea-salvage/deployment/gitea-incident-salvage-v2.json new file mode 100644 index 0000000..1d8b22c --- /dev/null +++ b/infra/deploy-runner/fixtures/gitea-salvage/deployment/gitea-incident-salvage-v2.json @@ -0,0 +1,117 @@ +{ + "schemaVersion": "nodedc.gitea.incident-salvage.v2", + "action": "clean-state-salvage", + "component": "gitea", + "incidentId": "gitea-20260814", + "installRoot": "/volume1/docker/nodedc-gitea", + "compose": { + "file": "docker-compose.gitea.yml", + "project": "nodedc-gitea", + "service": "gitea", + "sha256": "2f031d5bfff4f42c73cabd8c94487ec3e4f1e1b0a96d3b7eec958904f735908a" + }, + "runtime": { + "image": "docker.gitea.com/gitea:1.27.2-rootless@sha256:7de5f49ada687b8c8d2938f547cdb7634839764ba51f297457bae35cee3abd2c", + "imageId": "sha256:272085a806e6d182352cdb011c0ebab1d2efc7ec45247de84de5659c7bc5c4c6", + "repoDigest": "docker.gitea.com/gitea@sha256:7de5f49ada687b8c8d2938f547cdb7634839764ba51f297457bae35cee3abd2c", + "platform": "linux/amd64", + "pullPolicy": "never", + "entrypoint": "/usr/local/bin/gitea", + "runAs": "1000:1000", + "transport": "unix:/run/gitea/gitea.sock", + "networkMode": "none", + "ssh": "disabled-no-published-port", + "lfs": "disabled-pending-reachable-pointer-physical-sha-verifier", + "database": "new-sqlite-1.27.2-only", + "logging": "bounded-json-file-10m-x3", + "stopGracePeriod": "30s", + "candidateRestartPolicy": "no", + "acceptedRestartPolicy": "unless-stopped" + }, + "decision": { + "schema": "nodedc.gitea.incident-decision/v2", + "manifestFile": "deployment/gitea-incident-salvage/confirmed-decision.json", + "manifestSha256": "dc9528462624158eb44218d37cc7054d551ca2d7ded562592982aa3f34c9fc2a", + "usersFile": "deployment/gitea-incident-salvage/users.decisions.csv", + "usersSha256": "e3b82f1073a86eea9e567edff062dd1689d21ec0edcf3ed92e844da9351ee8b6", + "repositoriesFile": "deployment/gitea-incident-salvage/repositories.decisions.csv", + "repositoriesSha256": "76b4bae2ab5cec490330c19bfc5ae9429abf7636705ae028c1c64fd54a6a0493", + "users": { + "active": 2, + "locked": 8, + "delete": 962 + }, + "repositories": { + "keep": 45, + "delete": 2013, + "dctouchPrivate": 32, + "silverPublic": 13 + } + }, + "disposition": { + "schema": "nodedc.gitea.incident-disposition.v1", + "file": "deployment/gitea-incident-salvage/confirmed-disposition-v1.json", + "sha256": "0a066724bcf6e4933133db6cab6cc273393e3c262dd00dda0bbf9ceebd84f78c" + }, + "snapshot": { + "root": "/volume1/.nodedc-security-snapshots/docker-gitea-incident-20260814", + "uuid": "f5a3fe3a-93ea-bb4d-847f-6221a6bcbc9f", + "readOnlyRequired": true, + "database": "gitea/gitea/gitea.db", + "databaseBytes": 182681600, + "databaseSha256": "8db9e74a5641662a808d8252c5d6c9de43fe9efd8687634bfbff2d4361a13052", + "repositoriesRoot": "gitea/git/repositories" + }, + "trust": { + "legacyRootMounted": false, + "legacyDatabaseImported": false, + "legacyConfigImported": false, + "legacyCredentialsImported": false, + "artifactSecrets": "forbidden", + "runtimeSecrets": "runner-generated", + "repositorySelection": "exact-v2-rows-only", + "repositoryCopy": "same-filesystem-per-file-reflink-object-and-validated-ref-material-only", + "forbiddenRepositoryMaterial": [ + "hooks", + "config", + "alternates", + "http-alternates", + "shallow", + "replace", + "grafts", + "worktrees", + "commondir", + "lfs" + ], + "unsupportedDatabaseState": "confirmed-disposition-bound-no-legacy-row-import-verifiers-pending" + }, + "identity": { + "preserveNumericUserIds": false, + "preserveNumericRepositoryIds": false, + "oldToNewIdMapping": "/volume1/docker/nodedc-gitea/audit/identity-map.json", + "emails": "new-local-noreply-only", + "passwordHashes": "never-imported", + "activeBootstrapCredentials": "root-only-runner-state-require-change", + "lockedCredentials": "fresh-random-discarded-and-login-prohibited", + "twoFactorAuthentication": "not-configured-by-transition" + }, + "externalPrerequisites": { + "legacyContainer": "exact-inspect-pin-required-stopped-restart-no", + "publicProxy": "reviewed-nginx-uds-bridge-and-dsm-loopback-upstream", + "firewall": "legacy-isolation-retained-and-loopback-3000-ready", + "candidatePublicationGate": "socket-parent-uid1000-gid1023-mode0700", + "publishAction": "runner-chmod-socket-parent-0750-after-internal-acceptance", + "mustPassBeforeApply": true + }, + "acceptance": { + "users": "exact-10-row-by-row", + "repositories": "exact-45-row-by-row-and-visibility", + "credentialsSessionsKeysWebhooksActions": "zero", + "git": "fsck-all-and-public-smart-http-smoke", + "registration": "closed", + "ssh": "disabled", + "network": "none-no-ports", + "publicHttps": "required-before-ledger-success" + }, + "rollback": "stop-remove-candidate-prove-absent-quarantine-new-root-never-start-legacy" +} diff --git a/infra/deploy-runner/fixtures/gitea-salvage/deployment/gitea-incident-salvage-v3.json b/infra/deploy-runner/fixtures/gitea-salvage/deployment/gitea-incident-salvage-v3.json new file mode 100644 index 0000000..1aaefb6 --- /dev/null +++ b/infra/deploy-runner/fixtures/gitea-salvage/deployment/gitea-incident-salvage-v3.json @@ -0,0 +1,123 @@ +{ + "schemaVersion": "nodedc.gitea.incident-salvage.v3", + "action": "clean-state-salvage", + "component": "gitea", + "incidentId": "gitea-20260814", + "installRoot": "/volume1/docker/nodedc-gitea", + "compose": { + "file": "docker-compose.gitea.yml", + "project": "nodedc-gitea", + "service": "gitea", + "sha256": "2f031d5bfff4f42c73cabd8c94487ec3e4f1e1b0a96d3b7eec958904f735908a" + }, + "runtime": { + "image": "docker.gitea.com/gitea:1.27.2-rootless@sha256:7de5f49ada687b8c8d2938f547cdb7634839764ba51f297457bae35cee3abd2c", + "imageId": "sha256:272085a806e6d182352cdb011c0ebab1d2efc7ec45247de84de5659c7bc5c4c6", + "repoDigest": "docker.gitea.com/gitea@sha256:7de5f49ada687b8c8d2938f547cdb7634839764ba51f297457bae35cee3abd2c", + "platform": "linux/amd64", + "pullPolicy": "never", + "entrypoint": "/usr/local/bin/gitea", + "runAs": "1000:1000", + "transport": "unix:/run/gitea/gitea.sock", + "networkMode": "none", + "ssh": "disabled-no-published-port", + "lfs": "disabled-pending-reachable-pointer-physical-sha-verifier", + "database": "new-sqlite-1.27.2-only", + "logging": "bounded-json-file-10m-x3", + "stopGracePeriod": "30s", + "candidateRestartPolicy": "no", + "acceptedRestartPolicy": "unless-stopped" + }, + "decision": { + "schema": "nodedc.gitea.incident-decision/v2", + "manifestFile": "deployment/gitea-incident-salvage/confirmed-decision.json", + "manifestSha256": "dc9528462624158eb44218d37cc7054d551ca2d7ded562592982aa3f34c9fc2a", + "usersFile": "deployment/gitea-incident-salvage/users.decisions.csv", + "usersSha256": "e3b82f1073a86eea9e567edff062dd1689d21ec0edcf3ed92e844da9351ee8b6", + "repositoriesFile": "deployment/gitea-incident-salvage/repositories.decisions.csv", + "repositoriesSha256": "76b4bae2ab5cec490330c19bfc5ae9429abf7636705ae028c1c64fd54a6a0493", + "users": { + "active": 2, + "locked": 8, + "delete": 962 + }, + "repositories": { + "keep": 45, + "delete": 2013, + "dctouchPrivate": 32, + "silverPublic": 13 + } + }, + "disposition": { + "schema": "nodedc.gitea.incident-disposition.v1", + "file": "deployment/gitea-incident-salvage/confirmed-disposition-v1.json", + "sha256": "0a066724bcf6e4933133db6cab6cc273393e3c262dd00dda0bbf9ceebd84f78c" + }, + "snapshot": { + "root": "/volume1/.nodedc-security-snapshots/docker-gitea-incident-20260814", + "uuid": "f5a3fe3a-93ea-bb4d-847f-6221a6bcbc9f", + "readOnlyRequired": true, + "database": "gitea/gitea/gitea.db", + "databaseBytes": 182681600, + "databaseSha256": "8db9e74a5641662a808d8252c5d6c9de43fe9efd8687634bfbff2d4361a13052", + "repositoriesRoot": "gitea/git/repositories" + }, + "trust": { + "legacyRootMounted": false, + "legacyDatabaseImported": false, + "legacyConfigImported": false, + "legacyCredentialsImported": false, + "artifactSecrets": "forbidden", + "runtimeSecrets": "runner-generated", + "repositorySelection": "exact-v2-rows-only", + "repositoryCopy": "same-filesystem-per-file-reflink-object-and-validated-ref-material-only", + "forbiddenRepositoryMaterial": [ + "hooks", + "config", + "alternates", + "http-alternates", + "shallow", + "replace", + "grafts", + "worktrees", + "commondir", + "lfs" + ], + "unsupportedDatabaseState": "closure-disposition-bound-plan-report-review-and-verifiers-pending" + }, + "identity": { + "preserveNumericUserIds": false, + "preserveNumericRepositoryIds": false, + "oldToNewIdMapping": "/volume1/docker/nodedc-gitea/audit/identity-map.json", + "emails": "new-local-noreply-only", + "passwordHashes": "never-imported", + "activeBootstrapCredentials": "root-only-runner-state-require-change", + "lockedCredentials": "fresh-random-discarded-and-login-prohibited", + "twoFactorAuthentication": "not-configured-by-transition" + }, + "externalPrerequisites": { + "legacyContainer": "exact-inspect-pin-required-stopped-restart-no", + "publicProxy": "reviewed-nginx-uds-bridge-and-dsm-loopback-upstream", + "firewall": "legacy-isolation-retained-and-loopback-3000-ready", + "candidatePublicationGate": "socket-parent-uid1000-gid1023-mode0700", + "publishAction": "runner-chmod-socket-parent-0750-after-internal-acceptance", + "mustPassBeforeApply": true + }, + "acceptance": { + "users": "exact-10-row-by-row", + "repositories": "exact-45-row-by-row-and-visibility", + "credentialsSessionsKeysWebhooksActions": "zero", + "git": "fsck-all-and-public-smart-http-smoke", + "registration": "closed", + "ssh": "disabled", + "network": "none-no-ports", + "publicHttps": "required-before-ledger-success" + }, + "rollback": "stop-remove-candidate-prove-absent-quarantine-new-root-never-start-legacy", + "closureDisposition": { + "file": "deployment/gitea-incident-salvage/confirmed-closure-disposition-v1.json", + "predecessorArtifactSha256": "d6870b5583a2f329eadb4e6cda65fdf4d271532df5ffbf8bfb1403968a434672", + "schema": "nodedc.gitea.incident-closure-disposition.v1", + "sha256": "7ed66d9848268431a703fe24b22c41afbaa7c5ff48949604d6fc448d93e0d243" + } +} diff --git a/infra/deploy-runner/fixtures/gitea-salvage/deployment/gitea-incident-salvage/confirmed-closure-disposition-v1.json b/infra/deploy-runner/fixtures/gitea-salvage/deployment/gitea-incident-salvage/confirmed-closure-disposition-v1.json new file mode 100644 index 0000000..e0ebe57 --- /dev/null +++ b/infra/deploy-runner/fixtures/gitea-salvage/deployment/gitea-incident-salvage/confirmed-closure-disposition-v1.json @@ -0,0 +1 @@ +{"activation":{"allowedOperation":"canonical-plan-only","applyFrozen":true,"freezeBoundary":"before-candidate-root-creation"},"authority":{"policyScope":"access-collaboration-issue-pr-attachment-release-label-project-unit-package-action-closure","source":"owner-instruction-in-current-incident-thread","state":"confirmed-policy-evidence-review-pending"},"closureReport":{"expectedBytes":null,"expectedSha256":null,"reviewState":"canonical-plan-output-unreviewed","schema":"nodedc.gitea.salvage-closure-inventory/v1"},"incidentId":"gitea-20260814","policies":{"accessCollaboration":{"accessCache":"DROP_RESET_RECOMPUTE","actorEvidence":"old-user-id-mode-and-kept-deleted-class-only","collaboration":"RECREATE_ONLY_KEPT_ACTORS_AFTER_REVIEWED_OLD_TO_NEW_ID_MAP","deletedActors":962,"keptActors":10,"legacyRowsImported":false,"sensitiveUserPayloadExported":false},"attachments":{"databaseManifest":"ID_UUID_RELATIONS_DECLARED_SIZE_ONLY","filenameOrContentExported":false,"legacyRowsImported":false,"physicalDisposition":"VERIFY_BYTES_THEN_SANITIZED_IMMUTABLE_ARCHIVE_ONLY","physicalPresenceClaimed":false},"issuesPullRequestsMetadata":{"archive":"SANITIZED_IMMUTABLE_ARCHIVE_ONLY","legacyRowsImported":false,"payloadExportedInPlan":false,"planEvidence":"PER_REPOSITORY_COUNTS_ACTOR_CLASSES_TEXT_BYTE_LENGTHS_ONLY","subrelationClosure":{"commentHistoryMerger":"SCHEMA_BOUND_EXACT_RELATION_COUNTS_AND_CLASSES","externalAuthors":"PRESENCE_AND_NAME_BYTE_LENGTHS_ONLY_NO_LOCAL_USER_MAPPING","legacyRowsImported":false,"teamRelations":"EXACT_ROW_TEAM_ORG_IDS_SEALED_HOLD_AND_BLOCK_IF_PRESENT"},"tables":["comment","issue","issue_assignees","issue_content_history","issue_dependency","issue_label","issue_user","issue_watch","notification","project","project_board","project_issue","pull_auto_merge","pull_request","reaction","review","review_state","stopwatch","tracked_time"]},"packagesActions":{"actionsTarget":"DISABLED","legacyPayloadSecretTokenLogImported":false,"packageTarget":"DISABLED","physicalDisposition":"DROP_ONLY_AFTER_SCHEMA_RELATION_AND_PHYSICAL_CLOSURE_VERIFIER","planEvidence":"RELATIONSHIP_COUNTS_AND_SAFE_DECLARED_SIZES_ONLY"},"releasesLabelsProjects":{"archive":"SANITIZED_IMMUTABLE_ARCHIVE_ONLY","legacyRowsImported":false,"payloadExportedInPlan":false},"repositoryState":{"cachedAccessCountersStarsWatches":"DROP_RESET_RECOMPUTE","legacyHooksWebhooksKeysTokensSessionsCredentialsSecrets":"IMPORT_ZERO","topics":"VERIFIED_SEMANTIC_EMPTY_REBUILD_ZERO"},"units":{"actionsType10":"DISABLED_ZERO_TARGET_ROWS","disabledTypes":[6,7,9,10],"enabledCleanTypes":[1,2,3,4,5,8],"legacyConfigImported":false,"legacyRowsImported":false,"packagesType9":"DISABLED_ZERO_TARGET_ROWS"}},"predecessor":{"artifactSha256":"d6870b5583a2f329eadb4e6cda65fdf4d271532df5ffbf8bfb1403968a434672","dispositionFile":"deployment/gitea-incident-salvage/confirmed-disposition-v1.json","dispositionSha256":"0a066724bcf6e4933133db6cab6cc273393e3c262dd00dda0bbf9ceebd84f78c"},"remainingBlockers":["attachment-physical-verifier-pending","candidate-root-activation-hard-frozen","closure-report-review-pin-pending","collaboration-kept-user-mapping-verifier-pending","forensic-ref-archive-verifier-pending","issue-pr-metadata-sanitized-archive-verifier-pending","lfs-reachable-pointer-physical-verifier-pending","package-action-physical-closure-verifier-pending","reference-manifest-fsck-reachability-verifier-pending","repository-object-reconstruction-verifier-pending","target-unit-policy-acceptance-pending","unsupported-schema-catalog-verifier-pending"],"schemaVersion":"nodedc.gitea.incident-closure-disposition.v1","scope":{"deletedRepositories":2013,"deletedUsers":962,"keptRepositories":45,"keptUsers":10},"sourceEvidence":{"databaseSha256":"8db9e74a5641662a808d8252c5d6c9de43fe9efd8687634bfbff2d4361a13052","identityDecisionManifestSha256":"dc9528462624158eb44218d37cc7054d551ca2d7ded562592982aa3f34c9fc2a","referenceManifestSha256":"9cddaf0e4d4cf22dd264a6ae589ccc50d29e07f85c55e9d34b14627cecb8a311","semanticTopicsSha256":"df6e3612186234bfcf3c172ef4e0fff933baaa691a510f9780ebf9e21c8d4d05","snapshotUuid":"f5a3fe3a-93ea-bb4d-847f-6221a6bcbc9f","unsupportedRepositoryReportSha256":"4b2cecf88c62fc5c4a43419885e88a01c9f9aac03133afb19dae0a7caef106ac","unsupportedSchemaCatalogSha256":"b5e3b6776926c4f1627fafd882362ed0ef986bfc86fc6ac6507a43976531b6db"}} diff --git a/infra/deploy-runner/fixtures/gitea-salvage/deployment/gitea-incident-salvage/confirmed-disposition-v1.json b/infra/deploy-runner/fixtures/gitea-salvage/deployment/gitea-incident-salvage/confirmed-disposition-v1.json new file mode 100644 index 0000000..3c6fe8e --- /dev/null +++ b/infra/deploy-runner/fixtures/gitea-salvage/deployment/gitea-incident-salvage/confirmed-disposition-v1.json @@ -0,0 +1 @@ +{"activation":{"allowedOperation":"canonical-plan-only","applyFrozen":true,"freezeBoundary":"before-candidate-root-creation"},"confirmation":{"confirmationRecordedAt":"2026-08-14T15:16:04Z","confirmationScope":"exact-10-users-45-repositories;heads-tags-live;pull-remote-sealed-archive;non-git-state-policy;no-2fa-change","confirmationSource":"owner-instruction-in-current-incident-thread","confirmedBy":"dctouch"},"incidentId":"gitea-20260814","referencePolicy":{"archiveOnly":{"immutableArchiveRequired":true,"neverAutoPromote":true,"pullRefs":5,"remoteRefs":7,"totalRefs":12},"exactDecisions":[{"disposition":"LIVE_RESTORE","name":"refs/heads/master","oid":"85b2768c0aa9e9b6f1419bf4414eeefd05e60735","oldRepositoryId":1,"repositoryPath":"dctouch/rv_2023_51.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/master","oid":"f06d474d01859c6a1b82822529a27fa0363c5b05","oldRepositoryId":1,"repositoryPath":"dctouch/rv_2023_51.wiki.git","wiki":true},{"disposition":"LIVE_RESTORE","name":"refs/heads/main","oid":"fd0c1050c2971651e47ed3981ef963842c5aded3","oldRepositoryId":3,"repositoryPath":"dctouch/dc_oled_prez.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/character_refactoring","oid":"e91ebb03227c8545cd950cfa455db27866ffd8b1","oldRepositoryId":4,"repositoryPath":"dctouch/dcmocap_rv.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/master","oid":"e91ebb03227c8545cd950cfa455db27866ffd8b1","oldRepositoryId":4,"repositoryPath":"dctouch/dcmocap_rv.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/vivemocap","oid":"f3f351867e1aebe8128f860a7930ad894e3a13db","oldRepositoryId":4,"repositoryPath":"dctouch/dcmocap_rv.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/MoCap","oid":"ba0c845f637919e998d6f50cddd6c3d963bdb8bf","oldRepositoryId":7,"repositoryPath":"dctouch/rv_2024_52.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/Trainer-Rulers","oid":"915ae33db2f94baed0a78826e30752f8f5b054cd","oldRepositoryId":7,"repositoryPath":"dctouch/rv_2024_52.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/add_vosk_binaries","oid":"e5436386629ebd1f78cd2c31532e0f1cf7dbefac","oldRepositoryId":7,"repositoryPath":"dctouch/rv_2024_52.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/character_refactor","oid":"0cff92b59f17b17e9f8f7429f04ace7290bb47b3","oldRepositoryId":7,"repositoryPath":"dctouch/rv_2024_52.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/master","oid":"130799aed535f213613ad1a1b2cd5d190eb0093a","oldRepositoryId":7,"repositoryPath":"dctouch/rv_2024_52.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/rewrite_voice_chat","oid":"57eb2ba5c3a25a6363f639157e5945eb8c5bb377","oldRepositoryId":7,"repositoryPath":"dctouch/rv_2024_52.git","wiki":false},{"disposition":"SEALED_ARCHIVE_ONLY","name":"refs/pull/18/head","oid":"14824aa8ae6034447b564e394d8dbeba5fab6e83","oldRepositoryId":7,"repositoryPath":"dctouch/rv_2024_52.git","wiki":false},{"disposition":"SEALED_ARCHIVE_ONLY","name":"refs/pull/19/head","oid":"e5436386629ebd1f78cd2c31532e0f1cf7dbefac","oldRepositoryId":7,"repositoryPath":"dctouch/rv_2024_52.git","wiki":false},{"disposition":"SEALED_ARCHIVE_ONLY","name":"refs/pull/20/head","oid":"57eb2ba5c3a25a6363f639157e5945eb8c5bb377","oldRepositoryId":7,"repositoryPath":"dctouch/rv_2024_52.git","wiki":false},{"disposition":"SEALED_ARCHIVE_ONLY","name":"refs/pull/51/head","oid":"0cff92b59f17b17e9f8f7429f04ace7290bb47b3","oldRepositoryId":7,"repositoryPath":"dctouch/rv_2024_52.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/main","oid":"a439f754315f3c52ddf10663dcaf25bc5144d660","oldRepositoryId":8,"repositoryPath":"dctouch/rv_1_2.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/ohrana","oid":"806aa643b20dfe597353f788f72db7d0b0155296","oldRepositoryId":8,"repositoryPath":"dctouch/rv_1_2.git","wiki":false},{"disposition":"SEALED_ARCHIVE_ONLY","name":"refs/pull/16/head","oid":"fa910f61c597e49c9f4e8492c7bf7247668e6763","oldRepositoryId":8,"repositoryPath":"dctouch/rv_1_2.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/master","oid":"6441e04b88b2f213ec7109a9ee2ff82cbe03b20b","oldRepositoryId":9,"repositoryPath":"dctouch/dcv.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/Fix_packaging_DCDroneNetworkSubsystem","oid":"94084967333e93812033d126225e1a4552a8d1e8","oldRepositoryId":16,"repositoryPath":"dctouch/rt_supercam.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/done","oid":"c56e4c90d0b8e59ec498aa2a76e6ae06d2ae55af","oldRepositoryId":16,"repositoryPath":"dctouch/rt_supercam.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/drone_pitch_dmx","oid":"879ff070690829341202c79cc8bec9b5908ac138","oldRepositoryId":16,"repositoryPath":"dctouch/rt_supercam.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/main","oid":"b1bbc9b28a981134a9dc5336281c05b3a8a8d178","oldRepositoryId":16,"repositoryPath":"dctouch/rt_supercam.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/master","oid":"e0af297683a706da11bd383e5ad69d2b3d23181c","oldRepositoryId":16,"repositoryPath":"dctouch/rt_supercam.wiki.git","wiki":true},{"disposition":"LIVE_RESTORE","name":"refs/heads/main","oid":"8c15fda13791e053bcf61c177bb78b8b7cb08e9a","oldRepositoryId":18,"repositoryPath":"dctouch/rt_drone_scam.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/main","oid":"4e8185696973c2a63c2fa2205af6e753f946cafc","oldRepositoryId":19,"repositoryPath":"dctouch/rt_jsb_flysimulator.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/main","oid":"6460f84ac9c6ac06a8ea69f91d7a5fe98daef490","oldRepositoryId":23,"repositoryPath":"dctouch/dmx_test.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/main","oid":"f51a3f6cc37e7865b59e0251c241b28c6bcbeb56","oldRepositoryId":26,"repositoryPath":"dctouch/rt_drone_communication.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/main","oid":"d92f7957883d3150652ced81dfbeacc64eefedc9","oldRepositoryId":28,"repositoryPath":"dctouch/dcv_pers_props.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/main","oid":"e90804f3d94dced39c6ccc69f70322cedd51a9da","oldRepositoryId":30,"repositoryPath":"dctouch/rt_navigation.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/main","oid":"1d20ac3c88900a60109e20063c27082ed8a0440e","oldRepositoryId":35,"repositoryPath":"dctouch/dc_vr_gaussian_splat.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/main","oid":"5bc98b98f84a02b9222f595f9e4de89ef872ce10","oldRepositoryId":37,"repositoryPath":"dctouch/dcue_tuio.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/54","oid":"ac570d6e3e11d0086069f0023d3962c287b3f357","oldRepositoryId":38,"repositoryPath":"dctouch/dc_mosmap_2025.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/main","oid":"2fa6b0f8993da6bcf6a60da1cb18efb7f396d568","oldRepositoryId":38,"repositoryPath":"dctouch/dc_mosmap_2025.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/master","oid":"b9b2ff0da739aeb4c478d5f0285ba1bdabcf970f","oldRepositoryId":38,"repositoryPath":"dctouch/dc_mosmap_2025.wiki.git","wiki":true},{"disposition":"LIVE_RESTORE","name":"refs/heads/main","oid":"80c9c1f14cf5d64cf0e4e3aa38fec544c5568edf","oldRepositoryId":41,"repositoryPath":"dctouch/dc_lcc_leningradskiy.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/main","oid":"a06685f9de2299a706446068e92df59c91cc549b","oldRepositoryId":43,"repositoryPath":"dctouch/dc_gaussian_sretenka.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/5.6","oid":"688c79341ebd700251e71847dfca1f8565ac245a","oldRepositoryId":44,"repositoryPath":"dctouch/dc_hover.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/main","oid":"c5a6809a7be70f5ccf7e79a067259761e7032d19","oldRepositoryId":44,"repositoryPath":"dctouch/dc_hover.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/new-new-try","oid":"80c3b9b7be0e73e89ba31d895060d086b5d82f9c","oldRepositoryId":44,"repositoryPath":"dctouch/dc_hover.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/new-try","oid":"1de60e4b2ce935189ca60bb53a65cf111ab49f32","oldRepositoryId":44,"repositoryPath":"dctouch/dc_hover.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/main","oid":"086d649c22de24155145b03cd142cd7dcdaa05c4","oldRepositoryId":46,"repositoryPath":"dctouch/rv_2024_52_final.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/main","oid":"531e2d04c27106685b145b047de050d59284775a","oldRepositoryId":47,"repositoryPath":"dctouch/dc_vr_template_54.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/main","oid":"0626b46156ec85a13919110bb8b4ed951a2998dc","oldRepositoryId":48,"repositoryPath":"dctouch/rv_2024_52_final_full.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/MoCap","oid":"ba0c845f637919e998d6f50cddd6c3d963bdb8bf","oldRepositoryId":49,"repositoryPath":"dctouch/rv_2024_52_fix.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/Trainer-Rulers","oid":"915ae33db2f94baed0a78826e30752f8f5b054cd","oldRepositoryId":49,"repositoryPath":"dctouch/rv_2024_52_fix.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/add_vosk_binaries","oid":"e5436386629ebd1f78cd2c31532e0f1cf7dbefac","oldRepositoryId":49,"repositoryPath":"dctouch/rv_2024_52_fix.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/character_refactor","oid":"f65a389f4d7f31aa3d08c7e7fe905ef8c9ec43e3","oldRepositoryId":49,"repositoryPath":"dctouch/rv_2024_52_fix.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/master","oid":"17cf59d456a7a4145857042093c5ab744faa3fbf","oldRepositoryId":49,"repositoryPath":"dctouch/rv_2024_52_fix.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/tmp","oid":"b06cb21fe21a61dd7deb0c1171294b213ffdb757","oldRepositoryId":49,"repositoryPath":"dctouch/rv_2024_52_fix.git","wiki":false},{"disposition":"SEALED_ARCHIVE_ONLY","name":"refs/remotes/origin/HEAD","oid":"130799aed535f213613ad1a1b2cd5d190eb0093a","oldRepositoryId":49,"repositoryPath":"dctouch/rv_2024_52_fix.git","wiki":false},{"disposition":"SEALED_ARCHIVE_ONLY","name":"refs/remotes/origin/MoCap","oid":"ba0c845f637919e998d6f50cddd6c3d963bdb8bf","oldRepositoryId":49,"repositoryPath":"dctouch/rv_2024_52_fix.git","wiki":false},{"disposition":"SEALED_ARCHIVE_ONLY","name":"refs/remotes/origin/Trainer-Rulers","oid":"915ae33db2f94baed0a78826e30752f8f5b054cd","oldRepositoryId":49,"repositoryPath":"dctouch/rv_2024_52_fix.git","wiki":false},{"disposition":"SEALED_ARCHIVE_ONLY","name":"refs/remotes/origin/add_vosk_binaries","oid":"e5436386629ebd1f78cd2c31532e0f1cf7dbefac","oldRepositoryId":49,"repositoryPath":"dctouch/rv_2024_52_fix.git","wiki":false},{"disposition":"SEALED_ARCHIVE_ONLY","name":"refs/remotes/origin/character_refactor","oid":"0cff92b59f17b17e9f8f7429f04ace7290bb47b3","oldRepositoryId":49,"repositoryPath":"dctouch/rv_2024_52_fix.git","wiki":false},{"disposition":"SEALED_ARCHIVE_ONLY","name":"refs/remotes/origin/master","oid":"130799aed535f213613ad1a1b2cd5d190eb0093a","oldRepositoryId":49,"repositoryPath":"dctouch/rv_2024_52_fix.git","wiki":false},{"disposition":"SEALED_ARCHIVE_ONLY","name":"refs/remotes/origin/rewrite_voice_chat","oid":"57eb2ba5c3a25a6363f639157e5945eb8c5bb377","oldRepositoryId":49,"repositoryPath":"dctouch/rv_2024_52_fix.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/main","oid":"4994ab862d1a9825053dea939439cf2f318e06c6","oldRepositoryId":51,"repositoryPath":"dctouch/dc_lcc_lug_52.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/master","oid":"fe77460a8de2ba3b4430b57815f557d7ad43ad45","oldRepositoryId":51,"repositoryPath":"dctouch/dc_lcc_lug_52.wiki.git","wiki":true},{"disposition":"LIVE_RESTORE","name":"refs/heads/main","oid":"d4b0cacbaba4b3c5314cb9a207a24a7a98a2c6d0","oldRepositoryId":52,"repositoryPath":"dctouch/dc_lcc_lug_54.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/pixel-streaming","oid":"af1e8d141d5e7979e2dee0ce43ed4e4b3e08bf39","oldRepositoryId":52,"repositoryPath":"dctouch/dc_lcc_lug_54.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/main","oid":"5927ebbcb5e8c32bd7f0b537bd2cc2b3dd624872","oldRepositoryId":53,"repositoryPath":"dctouch/dc_drm_system.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/main","oid":"33c1ecca70b0ff8f66b26c38e078191bf06428d3","oldRepositoryId":55,"repositoryPath":"dctouch/dc_vr_villa.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/main","oid":"694c54c23fec9a626bf7f6975e0741a06434850a","oldRepositoryId":56,"repositoryPath":"dctouch/dc_moscowmap_high_land.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/main","oid":"d933a92238f2ae882e6625f72906766cd8f10939","oldRepositoryId":57,"repositoryPath":"dctouch/dc_vr_gaussianply_astra.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/main","oid":"6c646eeeb62559cbd80677096e797abe95f24120","oldRepositoryId":59,"repositoryPath":"dctouch/rjd_railverse_2.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/feature/aip-integration","oid":"bc528844a5646942ceafa66b3f8f86deb5ff5b0d","oldRepositoryId":60,"repositoryPath":"dctouch/node_dc.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/feature/auth-deploy-access-control","oid":"0e625a5af03153d6cb90dfd31e23dada4da5af79","oldRepositoryId":60,"repositoryPath":"dctouch/node_dc.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/master","oid":"2fb58d6a8a17130f44f5efe2597314b118cdc7ff","oldRepositoryId":60,"repositoryPath":"dctouch/node_dc.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/ndc-integration","oid":"68d5888585926a4bb5169fd6f1c4306b61aef4b0","oldRepositoryId":60,"repositoryPath":"dctouch/node_dc.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/main","oid":"fa07bb1f0e5092d328df977f66775b2266d75e92","oldRepositoryId":61,"repositoryPath":"silver/nodedc_1c.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/CROSSASSISTANT","oid":"763b90f3ba6df6ee07b57e5267d5dff5a025ed0c","oldRepositoryId":62,"repositoryPath":"silver/nodedc_taskmanager.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/ai-workspaces-npm","oid":"43e5f570e5c7a7e2d51d2427685e23c99e578cc4","oldRepositoryId":62,"repositoryPath":"silver/nodedc_taskmanager.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/master","oid":"70559910b1518aa578da52b6c14c42a0603b0a4e","oldRepositoryId":62,"repositoryPath":"silver/nodedc_taskmanager.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/ai-workspaces-npm","oid":"24d01fc813176f3437c7a1d44b173597ee1f1e94","oldRepositoryId":63,"repositoryPath":"silver/nodedc_launcher.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/main","oid":"5b23410549b5718638fd5333b135f09b05d58a0d","oldRepositoryId":63,"repositoryPath":"silver/nodedc_launcher.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/CROSSASSISTANT","oid":"6244b44c6eeb76a0589c92a541612f71668218b7","oldRepositoryId":64,"repositoryPath":"silver/nodedc_platform.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/ai-workspaces-npm","oid":"1aceabee4498ec81fb3ebd62e1258e3fd6b40e72","oldRepositoryId":64,"repositoryPath":"silver/nodedc_platform.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/codex/map-gateway-offline-coverage-20260809","oid":"49813d7839969f0b8812912c1177e41906482778","oldRepositoryId":64,"repositoryPath":"silver/nodedc_platform.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/codex/module-foundry-deploy-recovery-20260809-001","oid":"b511e92cf979f84300de5f9c29809690792c4238","oldRepositoryId":64,"repositoryPath":"silver/nodedc_platform.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/main","oid":"07224c6f0d97b289cf510bbbc6b8189c98b93f7e","oldRepositoryId":64,"repositoryPath":"silver/nodedc_platform.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/CROSSASSISTANT","oid":"c0a6fac55ff49a02efa4c75735c26e37273d3a3d","oldRepositoryId":65,"repositoryPath":"silver/nodedc_taskmanager_codexapi.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/main","oid":"d3ab1e3fb7a615083e2f618ef10dd9e546123156","oldRepositoryId":65,"repositoryPath":"silver/nodedc_taskmanager_codexapi.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/CROSSASSISTANT","oid":"352e21cdeab9955a21b0f106d1077fc7fcdda3c5","oldRepositoryId":66,"repositoryPath":"dctouch/nodedc_engine_infra.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/NDC_AI_SPACE","oid":"0a46a4ca0332a4a7a2ac677a30c0e8d356be9d79","oldRepositoryId":66,"repositoryPath":"dctouch/nodedc_engine_infra.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/ai-workspaces-npm","oid":"d427272c7e245b46a568e9ae6113ed4357fe8b59","oldRepositoryId":66,"repositoryPath":"dctouch/nodedc_engine_infra.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/beta","oid":"850974405de3ac4b994219ea80485fb6bb6cf22e","oldRepositoryId":66,"repositoryPath":"dctouch/nodedc_engine_infra.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/codex-lfs-smoke-test-20260518102242","oid":"05208f31287a0132e77ac0d24bab2611e49c8ddf","oldRepositoryId":66,"repositoryPath":"dctouch/nodedc_engine_infra.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/main","oid":"1337588ef08313ef19e84d800fb66ff282c50ed5","oldRepositoryId":66,"repositoryPath":"dctouch/nodedc_engine_infra.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/tags/alpha-live-2026-05-18","oid":"6746ef723a452f138902b7d532494ec74429d5dd","oldRepositoryId":66,"repositoryPath":"dctouch/nodedc_engine_infra.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/tags/backup/main-before-ndc-ai-space-main-20260606-183501","oid":"05208f31287a0132e77ac0d24bab2611e49c8ddf","oldRepositoryId":66,"repositoryPath":"dctouch/nodedc_engine_infra.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/tags/backup/ndc-ai-space-before-main-20260606-183501","oid":"0a46a4ca0332a4a7a2ac677a30c0e8d356be9d79","oldRepositoryId":66,"repositoryPath":"dctouch/nodedc_engine_infra.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/beam","oid":"1a78dd0dc814586713eb2a8a9c2bf0e5da7927c7","oldRepositoryId":67,"repositoryPath":"silver/nodedc_bim_viewer.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/main","oid":"9aaf82862fdcc63a8887eee61ffc2d3aebe13a8b","oldRepositoryId":67,"repositoryPath":"silver/nodedc_bim_viewer.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/main","oid":"bb7c54e9caf6080ba6085597574654e576a1e480","oldRepositoryId":68,"repositoryPath":"silver/nodedc_site.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/noadmin","oid":"2c07dc2df81e6633b4822914fc1450621e462acc","oldRepositoryId":68,"repositoryPath":"silver/nodedc_site.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/main","oid":"089bc2406fc98af4ba8e3de182c7c0f3c8f06521","oldRepositoryId":69,"repositoryPath":"silver/nodedc_seo.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/master","oid":"4c6f133ff1dba5975e45b3c7ab332c40eaf96ab8","oldRepositoryId":71,"repositoryPath":"silver/dc_cms.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/codex/foundry-map-panel-layout-20260809","oid":"88cbef1aba7425c2467706dd8cf6dc6c13eb8b2d","oldRepositoryId":72,"repositoryPath":"silver/nodedc_design_guideline.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/main","oid":"6c1265b8b6bb7e6884dbadbc47d25ee92102f3ce","oldRepositoryId":72,"repositoryPath":"silver/nodedc_design_guideline.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/feat/simulation-polygon-s0","oid":"a19053dfd47577fb66cc63f605ba29ea2314f6aa","oldRepositoryId":73,"repositoryPath":"silver/nodedc_mission_core.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/feat/simulation-polygon-s1","oid":"6b810952a490e3a6050269bcbc0a2f7dc5c2ff09","oldRepositoryId":73,"repositoryPath":"silver/nodedc_mission_core.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/heads/main","oid":"7fba39a6297e4a1792c67eccab317731abaeab72","oldRepositoryId":73,"repositoryPath":"silver/nodedc_mission_core.git","wiki":false},{"disposition":"LIVE_RESTORE","name":"refs/tags/mission-core-m0-k1-realtime-2026-07-16","oid":"87e68214915f3be56d401c35b40fe43448a6f692","oldRepositoryId":73,"repositoryPath":"silver/nodedc_mission_core.git","wiki":false}],"executionAuthority":"exact-decisions-one-to-one-observed-reference-manifest","forensicScope":{"allDiscoveredRefs":105,"preserveNameOidEvidence":true},"headInvariants":{"allowedMissingTargets":[{"head":"refs/heads/main","oldRepositoryId":70,"repositoryPath":"silver/nodedc_aegis.git","wiki":false},{"head":"refs/heads/main","oldRepositoryId":2119,"repositoryPath":"silver/nodedc_device_core.git","wiki":false}],"objectFormat":"sha1","stores":49,"symbolicHeads":49,"targetsPresentAmongLiveRefs":47},"liveRestore":{"copyLegacyHeadOrRefFiles":false,"mainRepositoryHeads":85,"mainRepositoryTags":4,"totalRefs":93,"wikiHeads":4},"unknownNamespacePolicy":"reject"},"remainingBlockers":["attachment-physical-verifier-pending","candidate-root-activation-hard-frozen","collaboration-kept-user-mapping-verifier-pending","forensic-ref-archive-verifier-pending","issue-pr-metadata-sanitized-archive-verifier-pending","lfs-reachable-pointer-physical-verifier-pending","package-action-physical-closure-verifier-pending","reference-manifest-fsck-reachability-verifier-pending","repository-object-reconstruction-verifier-pending","target-unit-policy-acceptance-pending","unsupported-schema-catalog-verifier-pending"],"repositoryStatePolicy":{"attachments":{"declaredRows":230,"disposition":"PHYSICAL_VERIFY_THEN_SANITIZED_ARCHIVE","legacyRowsImported":false,"physicalPresenceClaimed":false},"directRelations":[{"disposition":"SANITIZED_ARCHIVE_THEN_RECOMPUTE","label":"access_grants","sourceCount":32},{"disposition":"ASSERT_ZERO_AND_DROP","label":"action_runners","sourceCount":0},{"disposition":"ASSERT_ZERO_AND_DROP","label":"action_runs","sourceCount":0},{"disposition":"ASSERT_ZERO_AND_DROP","label":"action_schedules","sourceCount":0},{"disposition":"ASSERT_ZERO_AND_DROP","label":"action_secrets","sourceCount":0},{"disposition":"ASSERT_ZERO_AND_DROP","label":"action_variables","sourceCount":0},{"disposition":"PHYSICAL_VERIFY_THEN_SANITIZED_ARCHIVE","label":"attachments","sourceCount":230},{"disposition":"SANITIZED_ARCHIVE_THEN_RECREATE_AFTER_KEPT_USER_MAPPING","label":"collaborators","sourceCount":32},{"disposition":"ASSERT_ZERO_AND_DROP","label":"deploy_keys","sourceCount":0},{"disposition":"SANITIZED_ARCHIVE_ONLY","label":"issues","sourceCount":867},{"disposition":"SANITIZED_ARCHIVE_ONLY","label":"labels","sourceCount":7},{"disposition":"ASSERT_ZERO_AND_DROP","label":"lfs_locks","sourceCount":0},{"disposition":"REACHABILITY_AND_PHYSICAL_VERIFY_THEN_RESTORE","label":"lfs_objects","sourceCount":573941},{"disposition":"ASSERT_ZERO_AND_DROP","label":"milestones","sourceCount":0},{"disposition":"ASSERT_ZERO_AND_DROP","label":"mirrors","sourceCount":0},{"disposition":"ASSERT_ZERO_AND_DROP","label":"packages","sourceCount":0},{"disposition":"ASSERT_ZERO_AND_DROP","label":"protected_branches","sourceCount":0},{"disposition":"SANITIZED_ARCHIVE_ONLY_DIRECTIONAL_PROJECTION","label":"pull_requests_base","sourceCount":5},{"disposition":"SANITIZED_ARCHIVE_ONLY_DIRECTIONAL_PROJECTION","label":"pull_requests_head","sourceCount":5},{"disposition":"ASSERT_ZERO_AND_DROP","label":"push_mirrors","sourceCount":0},{"disposition":"SANITIZED_ARCHIVE_ONLY","label":"releases","sourceCount":4},{"disposition":"ASSERT_ZERO_AND_DROP","label":"stars","sourceCount":0},{"disposition":"ASSERT_ZERO_AND_DROP","label":"topics","sourceCount":0},{"disposition":"DROP_AND_REGENERATE","label":"watches","sourceCount":45},{"disposition":"ASSERT_ZERO_AND_DROP","label":"webhooks","sourceCount":0}],"legacySecretsCredentialsSessionsKeysIntegrations":"ZERO_NEVER_IMPORT","lfs":{"associationRows":573941,"disposition":"DISABLED_UNTIL_REACHABLE_POINTER_PHYSICAL_SHA_VERIFIER","legacyRowsImported":false,"physicalPresenceClaimed":false},"metadataArchive":{"categories":["issues","pull-requests","comments","releases","labels","projects","repository-description"],"legacyRowsImported":false,"mode":"SANITIZED_IMMUTABLE_ARCHIVE_ONLY"},"numericHints":[{"disposition":"DROP_AND_RECOMPUTE","name":"lfs_size","sourceNonzero":true},{"disposition":"DROP_AND_RECOMPUTE","name":"num_action_runs","sourceNonzero":false},{"disposition":"DROP_AND_RECOMPUTE","name":"num_issues","sourceNonzero":true},{"disposition":"DROP_AND_RECOMPUTE","name":"num_milestones","sourceNonzero":false},{"disposition":"DROP_AND_RECOMPUTE","name":"num_projects","sourceNonzero":false},{"disposition":"DROP_AND_RECOMPUTE","name":"num_pulls","sourceNonzero":true},{"disposition":"DROP_AND_RECOMPUTE","name":"num_stars","sourceNonzero":false},{"disposition":"DROP_AND_RECOMPUTE","name":"num_watches","sourceNonzero":true}],"relationSemantics":{"attachmentLinkCountsMayOverlap":true,"pullBaseAndHeadAreDirectionalProjections":true},"schemaOnlyDependencyGroups":[{"disposition":"SANITIZED_ARCHIVE_ONLY_AFTER_SCHEMA_AND_JOIN_VERIFIER","group":"issue-pr-release-project","tables":["comment","issue_assignees","issue_content_history","issue_dependency","issue_label","issue_user","issue_watch","notification","project","project_board","project_issue","pull_auto_merge","reaction","review","review_state","stopwatch","tracked_time"]},{"disposition":"ASSERT_ZERO_AND_DROP_AFTER_SCHEMA_AND_PHYSICAL_CLOSURE_VERIFIER","group":"packages","tables":["package_blob","package_file","package_property","package_version"]},{"disposition":"ASSERT_ZERO_AND_DROP_AFTER_SCHEMA_AND_PHYSICAL_CLOSURE_VERIFIER","group":"actions","tables":["action_artifact","action_run_index","action_run_job","action_task"]}],"sourceNonzeroCategories":["access_grants","attachments","collaborators","issues","labels","lfs_objects","pull_requests_base","pull_requests_head","releases","repo_units","repository_hint:lfs_size","repository_hint:num_issues","repository_hint:num_pulls","repository_hint:num_watches","repository_metadata:description","repository_metadata:topics","watches"],"textMetadata":[{"disposition":"ASSERT_ZERO_AND_DROP","name":"avatar","sourceRepositories":0},{"disposition":"SANITIZED_ARCHIVE_ONLY","name":"description","sourceRepositories":1},{"disposition":"ASSERT_ZERO_AND_DROP","name":"original_url","sourceRepositories":0},{"disposition":"VERIFY_SEMANTIC_EMPTY_THEN_REBUILD_FROM_RELATIONS","name":"topics","sourceRepositories":45},{"disposition":"ASSERT_ZERO_AND_DROP","name":"website","sourceRepositories":0}],"topics":{"acceptedLegacyEncoding":"canonical-json-null-or-canonical-json-array","canonicalLowercaseSortedUnique":true,"expectedMaterialRepositories":0,"expectedRelationalRows":0,"expectedSerializedArrays":0,"expectedSerializedNulls":45,"expectedTopics":0,"legacyRowsImported":false,"maxUtf8BytesPerTopic":35,"relationalAuthority":"repo_topic-join-topic","targetCache":"rebuild-from-relations","topicNamePattern":"^[a-z0-9][-.a-z0-9]*$"},"units":{"actionsGloballyDisabled":true,"legacyConfigImported":false,"legacyRowsImported":false,"packagesGloballyDisabled":true,"rows":[{"disposition":"CREATE_CLEAN_DEFAULT","name":"Code","sourceCount":45,"targetCount":45,"type":1},{"disposition":"CREATE_CLEAN_DEFAULT","name":"Issues","sourceCount":45,"targetCount":45,"type":2},{"disposition":"CREATE_CLEAN_DEFAULT","name":"Pull Requests","sourceCount":45,"targetCount":45,"type":3},{"disposition":"CREATE_CLEAN_DEFAULT","name":"Releases","sourceCount":45,"targetCount":45,"type":4},{"disposition":"CREATE_CLEAN_DEFAULT","name":"Wiki","sourceCount":45,"targetCount":45,"type":5},{"disposition":"ASSERT_SOURCE_ZERO_AND_FORBID_TARGET","name":"External Wiki","sourceCount":0,"targetCount":0,"type":6},{"disposition":"ASSERT_SOURCE_ZERO_AND_FORBID_TARGET","name":"External Tracker","sourceCount":0,"targetCount":0,"type":7},{"disposition":"CREATE_CLEAN_DEFAULT","name":"Projects","sourceCount":45,"targetCount":45,"type":8},{"disposition":"DROP_AND_DISABLE_TARGET","name":"Packages","sourceCount":45,"targetCount":0,"type":9},{"disposition":"ASSERT_SOURCE_ZERO_AND_DISABLE_TARGET","name":"Actions","sourceCount":0,"targetCount":0,"type":10}],"targetRowTotal":270,"targetUniquePerRepositoryAndType":true}},"schemaVersion":"nodedc.gitea.incident-disposition.v1","scope":{"keptRepositories":45,"keptUsers":10,"referenceDecisionRows":105,"repositoryStores":49},"sourceEvidence":{"databaseSha256":"8db9e74a5641662a808d8252c5d6c9de43fe9efd8687634bfbff2d4361a13052","identityDecisionManifestSha256":"dc9528462624158eb44218d37cc7054d551ca2d7ded562592982aa3f34c9fc2a","referenceManifestBytes":36010,"referenceManifestReviewState":"observed-unreviewed","referenceManifestSha256":"9cddaf0e4d4cf22dd264a6ae589ccc50d29e07f85c55e9d34b14627cecb8a311","snapshotUuid":"f5a3fe3a-93ea-bb4d-847f-6221a6bcbc9f","unsupportedRepositoryReportBytes":119468,"unsupportedRepositoryReportSha256":"4b2cecf88c62fc5c4a43419885e88a01c9f9aac03133afb19dae0a7caef106ac","unsupportedSchemaCatalogReviewState":"observed-unreviewed","unsupportedSchemaCatalogSha256":"b5e3b6776926c4f1627fafd882362ed0ef986bfc86fc6ac6507a43976531b6db"}} diff --git a/infra/deploy-runner/fixtures/gitea-salvage/docker-compose.gitea.yml b/infra/deploy-runner/fixtures/gitea-salvage/docker-compose.gitea.yml new file mode 100644 index 0000000..36d7efc --- /dev/null +++ b/infra/deploy-runner/fixtures/gitea-salvage/docker-compose.gitea.yml @@ -0,0 +1,62 @@ +name: nodedc-gitea + +services: + gitea: + image: docker.gitea.com/gitea:1.27.2-rootless@sha256:7de5f49ada687b8c8d2938f547cdb7634839764ba51f297457bae35cee3abd2c + platform: linux/amd64 + pull_policy: never + network_mode: none + user: "1000:1000" + entrypoint: + - /usr/local/bin/gitea + command: + - web + - --config + - /etc/gitea/app.ini + restart: "no" + stop_grace_period: 30s + init: true + read_only: true + cap_drop: + - ALL + security_opt: + - no-new-privileges:true + pids_limit: 512 + logging: + driver: json-file + options: + max-size: "10m" + max-file: "3" + environment: + USER: git + HOME: /data/gitea + GITEA_WORK_DIR: /data/gitea + GITEA_CUSTOM: /data/gitea + volumes: + - type: bind + source: /volume1/docker/nodedc-gitea/data + target: /data + bind: + create_host_path: false + - type: bind + source: /volume1/docker/nodedc-gitea/config + target: /etc/gitea + read_only: true + bind: + create_host_path: false + - type: bind + source: /volume1/docker/nodedc-gitea/socket + target: /run/gitea + bind: + create_host_path: false + secrets: + - gitea_secret_key + - gitea_internal_token + tmpfs: + - /tmp:rw,noexec,nosuid,nodev,size=256m + +secrets: + gitea_secret_key: + file: /volume1/docker/nodedc-gitea/secrets/secret-key + gitea_internal_token: + file: /volume1/docker/nodedc-gitea/secrets/internal-token diff --git a/infra/deploy-runner/fixtures/gitea/deployment/gitea-fresh-install-v1.json b/infra/deploy-runner/fixtures/gitea/deployment/gitea-fresh-install-v1.json new file mode 100644 index 0000000..e42f5f5 --- /dev/null +++ b/infra/deploy-runner/fixtures/gitea/deployment/gitea-fresh-install-v1.json @@ -0,0 +1,42 @@ +{ + "schemaVersion": "nodedc.gitea.fresh-install.v1", + "action": "fresh-install", + "component": "gitea", + "installRoot": "/volume1/docker/nodedc-gitea", + "compose": { + "file": "docker-compose.gitea.yml", + "project": "nodedc-gitea", + "service": "gitea", + "sha256": "25868a40996c405543b4627d06499b68f43556e607d839e969a400b0bc0ddadb" + }, + "runtime": { + "image": "docker.gitea.com/gitea:1.27.1-rootless@sha256:89dc3c214b3992e5bb01e05ad21139d7a8b302d3ea3d8942d3f7e904e92af148", + "platform": "linux/amd64", + "pullPolicy": "never", + "minimumComposeVersion": "2.20.1", + "transport": "unix:/run/gitea/gitea.sock", + "socketBind": "/volume1/docker/nodedc-gitea/socket:/run/gitea", + "ssh": "disabled-no-published-port", + "database": "fresh-sqlite-only", + "lfs": "disabled-pending-reviewed-restore-transition", + "networkMode": "none", + "logging": "bounded-json-file-10m-x3", + "stopGracePeriod": "30s" + }, + "trust": { + "artifactSecrets": "forbidden", + "runtimeSecrets": "runner-managed-file-mounts", + "legacyRootAccess": "forbidden", + "legacyDatabaseImport": "forbidden", + "legacyRepositoryImport": "forbidden" + }, + "reverseProxyPrerequisite": { + "managedOutsideArtifact": true, + "requiredDsmUpstream": "127.0.0.1:3000", + "requiredNginxBridge": "/usr/local/etc/nginx/conf.d/http.nodedc-gitea-uds.conf", + "requiredNginxBridgeSha256": "164f37a12a4f91e656cf20bd5b109978d16d723bdfde236653722aaf820780c9", + "requiredUnixUpstream": "/volume1/docker/nodedc-gitea/socket/gitea.sock", + "mustBeCompletedBeforeApply": true + }, + "rollback": "stop-candidate-preserve-fresh-runtime-state-and-restore-source" +} diff --git a/infra/deploy-runner/fixtures/gitea/docker-compose.gitea.yml b/infra/deploy-runner/fixtures/gitea/docker-compose.gitea.yml new file mode 100644 index 0000000..b42c00b --- /dev/null +++ b/infra/deploy-runner/fixtures/gitea/docker-compose.gitea.yml @@ -0,0 +1,128 @@ +name: nodedc-gitea + +services: + gitea: + image: docker.gitea.com/gitea:1.27.1-rootless@sha256:89dc3c214b3992e5bb01e05ad21139d7a8b302d3ea3d8942d3f7e904e92af148 + platform: linux/amd64 + pull_policy: never + network_mode: none + user: "1000:1000" + restart: unless-stopped + stop_grace_period: 30s + init: true + read_only: true + cap_drop: + - ALL + security_opt: + - no-new-privileges:true + pids_limit: 512 + logging: + driver: json-file + options: + max-size: "10m" + max-file: "3" + environment: + GITEA_WORK_DIR: /var/lib/gitea + GITEA__database__DB_TYPE: sqlite3 + GITEA__database__PATH: /var/lib/gitea/data/gitea.db + GITEA__server__DOMAIN: git.dcserve.ru + GITEA__server__ROOT_URL: https://git.dcserve.ru/ + GITEA__server__PROTOCOL: http+unix + GITEA__server__HTTP_ADDR: /run/gitea/gitea.sock + GITEA__server__UNIX_SOCKET_PERMISSION: "0666" + GITEA__server__LOCAL_ROOT_URL: http://unix/ + GITEA__server__DISABLE_SSH: "true" + GITEA__server__START_SSH_SERVER: "false" + GITEA__server__SSH_CREATE_AUTHORIZED_KEYS_FILE: "false" + GITEA__server__LFS_START_SERVER: "false" + GITEA__server__LFS_ALLOW_PURE_SSH: "false" + GITEA__server__OFFLINE_MODE: "true" + GITEA__server__LANDING_PAGE: login + GITEA__security__INSTALL_LOCK: "true" + GITEA__security__SECRET_KEY_URI: file:/run/secrets/gitea_secret_key + GITEA__security__INTERNAL_TOKEN_URI: file:/run/secrets/gitea_internal_token + GITEA__security__REVERSE_PROXY_LIMIT: "1" + GITEA__security__REVERSE_PROXY_TRUSTED_PROXIES: 127.0.0.0/8,::1/128 + GITEA__security__DISABLE_GIT_HOOKS: "true" + GITEA__security__DISABLE_WEBHOOKS: "true" + GITEA__security__IMPORT_LOCAL_PATHS: "false" + GITEA__security__ONLY_ALLOW_PUSH_IF_GITEA_ENVIRONMENT_SET: "true" + GITEA__security__PASSWORD_HASH_ALGO: argon2 + GITEA__security__MIN_PASSWORD_LENGTH: "16" + GITEA__security__PASSWORD_COMPLEXITY: lower,upper,digit,spec + GITEA__security__TWO_FACTOR_AUTH: enforced + GITEA__security__DISABLE_QUERY_AUTH_TOKEN: "true" + GITEA__security__ALLOWED_HOST_LIST: loopback + GITEA__service__DISABLE_REGISTRATION: "true" + GITEA__service__REQUIRE_SIGNIN_VIEW: "true" + GITEA__service__SHOW_REGISTRATION_BUTTON: "false" + GITEA__service__DEFAULT_KEEP_EMAIL_PRIVATE: "true" + GITEA__service__DEFAULT_ALLOW_CREATE_ORGANIZATION: "false" + GITEA__service__DEFAULT_USER_IS_RESTRICTED: "true" + GITEA__service__DEFAULT_USER_VISIBILITY: private + GITEA__service__ALLOWED_USER_VISIBILITY_MODES: private + GITEA__service__DEFAULT_ORG_VISIBILITY: private + GITEA__service__ENABLE_REVERSE_PROXY_AUTHENTICATION: "false" + GITEA__service__ENABLE_REVERSE_PROXY_AUTHENTICATION_API: "false" + GITEA__service__ENABLE_REVERSE_PROXY_AUTO_REGISTRATION: "false" + GITEA__service__ENABLE_NOTIFY_MAIL: "false" + GITEA__service__ENABLE_BASIC_AUTHENTICATION: "false" + GITEA__admin__DISABLE_REGULAR_ORG_CREATION: "true" + GITEA__admin__USER_DISABLED_FEATURES: deletion,manage_ssh_keys,manage_gpg_keys,change_username + GITEA__repository__FORCE_PRIVATE: "true" + GITEA__repository__DEFAULT_PRIVATE: private + GITEA__repository__USER_MAX_CREATION_LIMIT: "0" + GITEA__repository__ORG_MAX_CREATION_LIMIT: "0" + GITEA__repository__ENABLE_PUSH_CREATE_USER: "false" + GITEA__repository__ENABLE_PUSH_CREATE_ORG: "false" + GITEA__repository__DISABLE_MIGRATIONS: "true" + GITEA__repository__ALLOW_ADOPTION_OF_UNADOPTED_REPOSITORIES: "false" + GITEA__repository__ALLOW_DELETION_OF_UNADOPTED_REPOSITORIES: "false" + GITEA__repository__DISABLE_HTTP_GIT: "false" + GITEA__repository.upload__ENABLED: "false" + GITEA__attachment__ENABLED: "false" + GITEA__actions__ENABLED: "false" + GITEA__packages__ENABLED: "false" + GITEA__oauth2__ENABLED: "false" + GITEA__oauth2_client__ENABLE_AUTO_REGISTRATION: "false" + GITEA__openid__ENABLE_OPENID_SIGNIN: "false" + GITEA__openid__ENABLE_OPENID_SIGNUP: "false" + GITEA__federation__ENABLED: "false" + GITEA__mailer__ENABLED: "false" + GITEA__session__COOKIE_SECURE: "true" + GITEA__session__SAME_SITE: strict + GITEA__api__ENABLE_SWAGGER: "false" + GITEA__migrations__ALLOW_LOCALNETWORKS: "false" + GITEA__migrations__SKIP_TLS_VERIFY: "false" + GITEA__cors__ENABLED: "false" + GITEA__metrics__ENABLED: "false" + GITEA__cron.update_checker__ENABLED: "false" + GITEA__log__MODE: console + GITEA__log__LEVEL: Info + volumes: + - type: bind + source: /volume1/docker/nodedc-gitea/data + target: /var/lib/gitea + bind: + create_host_path: false + - type: bind + source: /volume1/docker/nodedc-gitea/config + target: /etc/gitea + bind: + create_host_path: false + - type: bind + source: /volume1/docker/nodedc-gitea/socket + target: /run/gitea + bind: + create_host_path: false + secrets: + - gitea_secret_key + - gitea_internal_token + tmpfs: + - /tmp:rw,noexec,nosuid,nodev,size=256m + +secrets: + gitea_secret_key: + file: /volume1/docker/nodedc-gitea/secrets/secret-key + gitea_internal_token: + file: /volume1/docker/nodedc-gitea/secrets/internal-token diff --git a/infra/deploy-runner/nodedc-deploy b/infra/deploy-runner/nodedc-deploy index 657e9a5..e273b6b 100755 --- a/infra/deploy-runner/nodedc-deploy +++ b/infra/deploy-runner/nodedc-deploy @@ -1,8 +1,13 @@ #!/usr/bin/env python3 import argparse +import array import base64 +import csv +import fcntl import grp import hashlib +import http.client +import io import json import os import pwd @@ -10,6 +15,7 @@ import re import secrets import shutil import socket +import sqlite3 import stat import subprocess import sys @@ -1782,6 +1788,579 @@ ENGINE_BASE_COMPOSE_SERVICES = ( "app", ) +GITEA_ROOT = Path("/volume1/docker/nodedc-gitea") +GITEA_COMPOSE_PROJECT = "nodedc-gitea" +GITEA_SERVICE = "gitea" +GITEA_COMPOSE_REL = "docker-compose.gitea.yml" +GITEA_FRESH_INSTALL_DESCRIPTOR_REL = ( + "deployment/gitea-fresh-install-v1.json" +) +GITEA_FRESH_INSTALL_ENTRIES = ( + GITEA_COMPOSE_REL, + GITEA_FRESH_INSTALL_DESCRIPTOR_REL, +) +GITEA_IMAGE = ( + "docker.gitea.com/gitea:1.27.1-rootless@" + "sha256:89dc3c214b3992e5bb01e05ad21139d7a8b302d3ea3d8942d3f7e904e92af148" +) +GITEA_IMAGE_ID = ( + "sha256:89dc3c214b3992e5bb01e05ad21139d7a8b302d3ea3d8942d3f7e904e92af148" +) +GITEA_REPO_DIGEST = ( + "docker.gitea.com/gitea@" + "sha256:89dc3c214b3992e5bb01e05ad21139d7a8b302d3ea3d8942d3f7e904e92af148" +) +GITEA_COMPOSE_SHA256 = ( + "25868a40996c405543b4627d06499b68f43556e607d839e969a400b0bc0ddadb" +) +GITEA_MINIMUM_COMPOSE_VERSION = (2, 20, 1) +GITEA_REQUIRED_DOCKER_VERSION = "24.0.2" +GITEA_RUNTIME_UID = 1000 +GITEA_RUNTIME_GID = 1000 +GITEA_NGINX_WORKER_UID = 1023 +GITEA_NGINX_GID = 1023 +GITEA_DATA_DIR = GITEA_ROOT / "data" +GITEA_CONFIG_DIR = GITEA_ROOT / "config" +GITEA_SOCKET_DIR = GITEA_ROOT / "socket" +GITEA_SOCKET_FILE = GITEA_SOCKET_DIR / "gitea.sock" +GITEA_SECRET_DIR = GITEA_ROOT / "secrets" +GITEA_SECRET_KEY_FILE = GITEA_SECRET_DIR / "secret-key" +GITEA_INTERNAL_TOKEN_FILE = GITEA_SECRET_DIR / "internal-token" +GITEA_HTTP_PORT = 3000 +GITEA_DISABLED_SSH_HOST_PORT = 4022 +GITEA_FORBIDDEN_LEGACY_NETWORK = "nodedc-gitea_internal" +GITEA_LEGACY_CONTAINER = "gitea" +GITEA_LEGACY_REVERSE_PROXY_IP = "172.22.0.222" +GITEA_REVERSE_PROXY_CONFIG = Path("/usr/syno/etc/www/ReverseProxy.json") +GITEA_REVERSE_PROXY_UUID = "5bc46027-0307-4261-af7e-4f94a3c508c9" +GITEA_REVERSE_PROXY_DESCRIPTION = "Gittea" +GITEA_REVERSE_PROXY_GENERATED_CONFIG = Path( + "/usr/local/etc/nginx/sites-available/" + "82fc9da8-f8a2-48d1-8ea5-6ba8dc8299d8.w3conf" +) +GITEA_NGINX = Path("/usr/bin/nginx") +GITEA_NGINX_VERSION = "nginx version: nginx/1.23.1" +GITEA_NGINX_MAIN_CONFIG = Path("/etc/nginx/nginx.conf.run") +GITEA_NGINX_BRIDGE_CONFIG = Path( + "/usr/local/etc/nginx/conf.d/http.nodedc-gitea-uds.conf" +) +GITEA_NGINX_BRIDGE_CONTENT = ( + "map $uri $nodedc_gitea_login_limit_key {\n" + " default \"\";\n" + " /user/login $http_x_real_ip;\n" + "}\n" + "\n" + "limit_req_zone $nodedc_gitea_login_limit_key " + "zone=nodedc_gitea_login:10m rate=10r/m;\n" + "limit_conn_zone $http_x_real_ip zone=nodedc_gitea_conn:10m;\n" + "\n" + "server {\n" + " listen 127.0.0.1:3000;\n" + " server_name nodedc-gitea-uds.internal;\n" + "\n" + " location / {\n" + " limit_req zone=nodedc_gitea_login burst=10 nodelay;\n" + " limit_conn nodedc_gitea_conn 40;\n" + " proxy_http_version 1.1;\n" + " proxy_set_header Host $http_host;\n" + " proxy_set_header X-Real-IP $http_x_real_ip;\n" + " proxy_set_header X-Forwarded-For $http_x_forwarded_for;\n" + " proxy_set_header X-Forwarded-Proto $http_x_forwarded_proto;\n" + " proxy_pass http://unix:/volume1/docker/nodedc-gitea/socket/gitea.sock:;\n" + " }\n" + "}\n" +) +GITEA_NGINX_BRIDGE_SHA256 = ( + "164f37a12a4f91e656cf20bd5b109978d16d723bdfde236653722aaf820780c9" +) +GITEA_IPTABLES = Path("/usr/bin/iptables") +GITEA_SECRET_RE = re.compile(r"^[A-Za-z0-9_-]{64,256}$") +GITEA_EXPECTED_ENVIRONMENT = { + "GITEA_WORK_DIR": "/var/lib/gitea", + "GITEA__database__DB_TYPE": "sqlite3", + "GITEA__database__PATH": "/var/lib/gitea/data/gitea.db", + "GITEA__server__DOMAIN": "git.dcserve.ru", + "GITEA__server__ROOT_URL": "https://git.dcserve.ru/", + "GITEA__server__PROTOCOL": "http+unix", + "GITEA__server__HTTP_ADDR": "/run/gitea/gitea.sock", + "GITEA__server__UNIX_SOCKET_PERMISSION": "0666", + "GITEA__server__LOCAL_ROOT_URL": "http://unix/", + "GITEA__server__DISABLE_SSH": "true", + "GITEA__server__START_SSH_SERVER": "false", + "GITEA__server__SSH_CREATE_AUTHORIZED_KEYS_FILE": "false", + "GITEA__server__LFS_START_SERVER": "false", + "GITEA__server__LFS_ALLOW_PURE_SSH": "false", + "GITEA__server__OFFLINE_MODE": "true", + "GITEA__server__LANDING_PAGE": "login", + "GITEA__security__INSTALL_LOCK": "true", + "GITEA__security__SECRET_KEY_URI": "file:/run/secrets/gitea_secret_key", + "GITEA__security__INTERNAL_TOKEN_URI": ( + "file:/run/secrets/gitea_internal_token" + ), + "GITEA__security__REVERSE_PROXY_LIMIT": "1", + "GITEA__security__REVERSE_PROXY_TRUSTED_PROXIES": "127.0.0.0/8,::1/128", + "GITEA__security__DISABLE_GIT_HOOKS": "true", + "GITEA__security__DISABLE_WEBHOOKS": "true", + "GITEA__security__IMPORT_LOCAL_PATHS": "false", + "GITEA__security__ONLY_ALLOW_PUSH_IF_GITEA_ENVIRONMENT_SET": "true", + "GITEA__security__PASSWORD_HASH_ALGO": "argon2", + "GITEA__security__MIN_PASSWORD_LENGTH": "16", + "GITEA__security__PASSWORD_COMPLEXITY": "lower,upper,digit,spec", + "GITEA__security__TWO_FACTOR_AUTH": "enforced", + "GITEA__security__DISABLE_QUERY_AUTH_TOKEN": "true", + "GITEA__security__ALLOWED_HOST_LIST": "loopback", + "GITEA__service__DISABLE_REGISTRATION": "true", + "GITEA__service__REQUIRE_SIGNIN_VIEW": "true", + "GITEA__service__SHOW_REGISTRATION_BUTTON": "false", + "GITEA__service__DEFAULT_KEEP_EMAIL_PRIVATE": "true", + "GITEA__service__DEFAULT_ALLOW_CREATE_ORGANIZATION": "false", + "GITEA__service__DEFAULT_USER_IS_RESTRICTED": "true", + "GITEA__service__DEFAULT_USER_VISIBILITY": "private", + "GITEA__service__ALLOWED_USER_VISIBILITY_MODES": "private", + "GITEA__service__DEFAULT_ORG_VISIBILITY": "private", + "GITEA__service__ENABLE_REVERSE_PROXY_AUTHENTICATION": "false", + "GITEA__service__ENABLE_REVERSE_PROXY_AUTHENTICATION_API": "false", + "GITEA__service__ENABLE_REVERSE_PROXY_AUTO_REGISTRATION": "false", + "GITEA__service__ENABLE_NOTIFY_MAIL": "false", + "GITEA__service__ENABLE_BASIC_AUTHENTICATION": "false", + "GITEA__admin__DISABLE_REGULAR_ORG_CREATION": "true", + "GITEA__admin__USER_DISABLED_FEATURES": ( + "deletion,manage_ssh_keys,manage_gpg_keys,change_username" + ), + "GITEA__repository__FORCE_PRIVATE": "true", + "GITEA__repository__DEFAULT_PRIVATE": "private", + "GITEA__repository__USER_MAX_CREATION_LIMIT": "0", + "GITEA__repository__ORG_MAX_CREATION_LIMIT": "0", + "GITEA__repository__ENABLE_PUSH_CREATE_USER": "false", + "GITEA__repository__ENABLE_PUSH_CREATE_ORG": "false", + "GITEA__repository__DISABLE_MIGRATIONS": "true", + "GITEA__repository__ALLOW_ADOPTION_OF_UNADOPTED_REPOSITORIES": "false", + "GITEA__repository__ALLOW_DELETION_OF_UNADOPTED_REPOSITORIES": "false", + "GITEA__repository__DISABLE_HTTP_GIT": "false", + "GITEA__repository.upload__ENABLED": "false", + "GITEA__attachment__ENABLED": "false", + "GITEA__actions__ENABLED": "false", + "GITEA__packages__ENABLED": "false", + "GITEA__oauth2__ENABLED": "false", + "GITEA__oauth2_client__ENABLE_AUTO_REGISTRATION": "false", + "GITEA__openid__ENABLE_OPENID_SIGNIN": "false", + "GITEA__openid__ENABLE_OPENID_SIGNUP": "false", + "GITEA__federation__ENABLED": "false", + "GITEA__mailer__ENABLED": "false", + "GITEA__session__COOKIE_SECURE": "true", + "GITEA__session__SAME_SITE": "strict", + "GITEA__api__ENABLE_SWAGGER": "false", + "GITEA__migrations__ALLOW_LOCALNETWORKS": "false", + "GITEA__migrations__SKIP_TLS_VERIFY": "false", + "GITEA__cors__ENABLED": "false", + "GITEA__metrics__ENABLED": "false", + "GITEA__cron.update_checker__ENABLED": "false", + "GITEA__log__MODE": "console", + "GITEA__log__LEVEL": "Info", +} + +# Incident salvage is deliberately additive to the fresh-install component. +# It never imports the compromised SQLite database, config, home, credentials, +# hooks or arbitrary repository trees. Only the exact v2 decision rows and +# validated Git object/ref material may cross the read-only snapshot boundary. +GITEA_SALVAGE_DESCRIPTOR_REL = ( + "deployment/gitea-incident-salvage-v3.json" +) +GITEA_SALVAGE_DECISION_PREFIX = "deployment/gitea-incident-salvage" +GITEA_SALVAGE_DISPOSITION_REL = ( + f"{GITEA_SALVAGE_DECISION_PREFIX}/confirmed-disposition-v1.json" +) +GITEA_SALVAGE_CLOSURE_DISPOSITION_REL = ( + f"{GITEA_SALVAGE_DECISION_PREFIX}/confirmed-closure-disposition-v1.json" +) +GITEA_SALVAGE_DECISION_MANIFEST_REL = ( + f"{GITEA_SALVAGE_DECISION_PREFIX}/confirmed-decision.json" +) +GITEA_SALVAGE_USERS_REL = ( + f"{GITEA_SALVAGE_DECISION_PREFIX}/users.decisions.csv" +) +GITEA_SALVAGE_REPOSITORIES_REL = ( + f"{GITEA_SALVAGE_DECISION_PREFIX}/repositories.decisions.csv" +) +GITEA_SALVAGE_ENTRIES = ( + GITEA_COMPOSE_REL, + GITEA_SALVAGE_DESCRIPTOR_REL, + GITEA_SALVAGE_DISPOSITION_REL, + GITEA_SALVAGE_CLOSURE_DISPOSITION_REL, + GITEA_SALVAGE_DECISION_MANIFEST_REL, + GITEA_SALVAGE_USERS_REL, + GITEA_SALVAGE_REPOSITORIES_REL, +) +GITEA_SALVAGE_IMAGE = ( + "docker.gitea.com/gitea:1.27.2-rootless@" + "sha256:7de5f49ada687b8c8d2938f547cdb7634839764ba51f297457bae35cee3abd2c" +) +GITEA_SALVAGE_IMAGE_ID = ( + "sha256:272085a806e6d182352cdb011c0ebab1d2efc7ec45247de84de5659c7bc5c4c6" +) +GITEA_SALVAGE_REPO_DIGEST = ( + "docker.gitea.com/gitea@" + "sha256:7de5f49ada687b8c8d2938f547cdb7634839764ba51f297457bae35cee3abd2c" +) +GITEA_SALVAGE_COMPOSE_SHA256 = ( + "2f031d5bfff4f42c73cabd8c94487ec3e4f1e1b0a96d3b7eec958904f735908a" +) +GITEA_SALVAGE_DESCRIPTOR_SHA256 = ( + "9b98eb1a1640fd5569cf051a621837379b167eff4527313a43a0a851e7cc181a" +) +GITEA_SALVAGE_DISPOSITION_SHA256 = ( + "0a066724bcf6e4933133db6cab6cc273393e3c262dd00dda0bbf9ceebd84f78c" +) +GITEA_SALVAGE_CLOSURE_DISPOSITION_SHA256 = ( + "7ed66d9848268431a703fe24b22c41afbaa7c5ff48949604d6fc448d93e0d243" +) +GITEA_SALVAGE_DECISION_MANIFEST_SHA256 = ( + "dc9528462624158eb44218d37cc7054d551ca2d7ded562592982aa3f34c9fc2a" +) +GITEA_SALVAGE_USERS_SHA256 = ( + "e3b82f1073a86eea9e567edff062dd1689d21ec0edcf3ed92e844da9351ee8b6" +) +GITEA_SALVAGE_REPOSITORIES_SHA256 = ( + "76b4bae2ab5cec490330c19bfc5ae9429abf7636705ae028c1c64fd54a6a0493" +) +GITEA_SALVAGE_SNAPSHOT_ROOT = Path( + "/volume1/.nodedc-security-snapshots/docker-gitea-incident-20260814" +) +GITEA_SALVAGE_SNAPSHOT_UUID = "f5a3fe3a-93ea-bb4d-847f-6221a6bcbc9f" +GITEA_SALVAGE_SNAPSHOT_DATABASE = ( + GITEA_SALVAGE_SNAPSHOT_ROOT / "gitea/gitea/gitea.db" +) +GITEA_SALVAGE_SNAPSHOT_DATABASE_BYTES = 182681600 +GITEA_SALVAGE_SNAPSHOT_DATABASE_SHA256 = ( + "8db9e74a5641662a808d8252c5d6c9de43fe9efd8687634bfbff2d4361a13052" +) +GITEA_SALVAGE_SNAPSHOT_REPOSITORIES = ( + GITEA_SALVAGE_SNAPSHOT_ROOT / "gitea/git/repositories" +) +GITEA_SALVAGE_BTRFS = Path("/usr/sbin/btrfs") +GITEA_SALVAGE_LEGACY_ROOT = Path("/volume1/docker/gitea") +GITEA_SALVAGE_MAINTENANCE_CONTAINER = "nodedc-gitea-salvage-maintenance" +GITEA_SALVAGE_REPOSITORY_QUARANTINE = GITEA_ROOT / "quarantine/repositories" +GITEA_SALVAGE_BOOTSTRAP_DIR = GITEA_ROOT / "bootstrap" +GITEA_SALVAGE_AUDIT_DIR = GITEA_ROOT / "audit" +GITEA_SALVAGE_IDENTITY_MAP = GITEA_SALVAGE_AUDIT_DIR / "identity-map.json" +GITEA_SALVAGE_DCTOUCH_PASSWORD_FILE = ( + GITEA_SALVAGE_BOOTSTRAP_DIR / "dctouch.password" +) +GITEA_SALVAGE_SILVER_PASSWORD_FILE = ( + GITEA_SALVAGE_BOOTSTRAP_DIR / "silver.password" +) +GITEA_SALVAGE_APP_INI = GITEA_CONFIG_DIR / "app.ini" +GITEA_SALVAGE_DB = GITEA_DATA_DIR / "gitea/gitea.db" +GITEA_SALVAGE_REPOSITORY_ROOT = GITEA_DATA_DIR / "git/repositories" +GITEA_SALVAGE_USER_COUNTS = { + "KEEP_ACTIVE": 2, + "KEEP_LOCKED": 8, + "DELETE": 962, +} +GITEA_SALVAGE_REPOSITORY_COUNTS = {"KEEP": 45, "DELETE": 2013} +GITEA_SALVAGE_ACTIVE_USERS = {1: "dctouch", 16: "SILVER"} +GITEA_SALVAGE_LOCKED_USERS = { + 2: "KOPYLOV", + 4: "kalininUN", + 6: "arturHITECA", + 7: "uePATSUKEVICH", + 11: "sadenov-a", + 15: "KKK", + 17: "ayoauo", + 18: "ayoayuoo", +} +# Exact stopped-container identity, reviewed from root-owned Docker inspect. +# The mutable `latest` text is provenance only; the immutable image ID and full +# stopped/mount topology below are the execution authority. +GITEA_SALVAGE_EXPECTED_LEGACY_IMAGE = "gitea/gitea:latest" +GITEA_SALVAGE_EXPECTED_LEGACY_IMAGE_ID = ( + "sha256:bf95d9a45ce4fe38b027d051cdc4a4bc531513489fa6244af4074efbb1c376d6" +) +GITEA_SALVAGE_EXPECTED_REF_MANIFEST_SHA256 = None +GITEA_SALVAGE_DISPOSITION_REFERENCE_MANIFEST_SHA256 = ( + "9cddaf0e4d4cf22dd264a6ae589ccc50d29e07f85c55e9d34b14627cecb8a311" +) +GITEA_SALVAGE_EXPECTED_UNSUPPORTED_REPORT_SHA256 = ( + "4b2cecf88c62fc5c4a43419885e88a01c9f9aac03133afb19dae0a7caef106ac" +) +GITEA_SALVAGE_DISPOSITION_UNSUPPORTED_SCHEMA_SHA256 = ( + "b5e3b6776926c4f1627fafd882362ed0ef986bfc86fc6ac6507a43976531b6db" +) +GITEA_SALVAGE_DISPOSITION_TOPICS_SHA256 = ( + "df6e3612186234bfcf3c172ef4e0fff933baaa691a510f9780ebf9e21c8d4d05" +) +GITEA_SALVAGE_CLOSURE_PREDECESSOR_ARTIFACT_SHA256 = ( + "d6870b5583a2f329eadb4e6cda65fdf4d271532df5ffbf8bfb1403968a434672" +) +GITEA_SALVAGE_CLOSURE_PREDECESSOR_DISPOSITION_SHA256 = ( + "0a066724bcf6e4933133db6cab6cc273393e3c262dd00dda0bbf9ceebd84f78c" +) +GITEA_SALVAGE_CLOSURE_MAX_ROWS_PER_RELATION = 2_000_000 +GITEA_SALVAGE_CLOSURE_MAX_TEXT_BYTES_PER_FIELD = 64 * 1024 * 1024 +GITEA_SALVAGE_CLOSURE_ACTOR_RELATIONS = ( + ("access_cache", "access", "DROP_CACHE_RECOMPUTE"), + ("collaborations", "collaboration", "RECREATE_KEPT_ACTOR_AFTER_ID_MAP"), +) +GITEA_SALVAGE_CLOSURE_ISSUE_DEPENDENTS = ( + ( + "comments", + "comment", + "poster_id", + ("content", "patch", "old_title", "new_title", "old_ref", "new_ref"), + ), + ("issue_assignees", "issue_assignees", "assignee_id", ()), + ( + "issue_content_histories", + "issue_content_history", + "poster_id", + ("content_text",), + ), + ("issue_labels", "issue_label", None, ()), + ("issue_users", "issue_user", "uid", ()), + ("issue_watches", "issue_watch", "user_id", ()), + ("reviews", "review", "reviewer_id", ("content",)), + ("stopwatches", "stopwatch", "user_id", ()), + ("tracked_times", "tracked_time", "user_id", ()), +) +GITEA_SALVAGE_CLOSURE_METRICS = tuple( + sorted( + { + "action_artifacts", + "action_run_indexes", + "action_run_jobs", + "action_runners", + "action_runs", + "action_schedules", + "action_secrets", + "action_tasks", + "action_variables", + "comments", + "issue_assignees", + "issue_content_histories", + "issue_dependencies", + "issue_labels", + "issue_users", + "issue_watches", + "issues_ordinary", + "labels", + "milestones", + "notifications", + "package_blobs", + "package_files", + "package_properties", + "package_versions", + "packages", + "project_boards", + "project_issue_links", + "projects", + "pull_auto_merges", + "pull_request_wrappers", + "reactions", + "releases", + "repo_units", + "review_states", + "reviews", + "stopwatches", + "tracked_times", + } + ) +) +GITEA_SALVAGE_CLOSURE_TEXT_COLUMNS = { + "comments": ("content", "new_ref", "new_title", "old_ref", "old_title", "patch"), + "issue_content_histories": ("content_text",), + "issues_ordinary": ("content", "name"), + "labels": ("color", "description", "name"), + "milestones": ("content", "name"), + "project_boards": ("color", "title"), + "projects": ("description", "title"), + "pull_auto_merges": ("merge_style", "message"), + "pull_request_wrappers": ("content", "name"), + "releases": ("note", "tag_name", "target", "title"), + "repo_units": ("config",), + "review_states": ("updated_files",), + "reviews": ("content",), +} +GITEA_SALVAGE_CLOSURE_NUMERIC_COLUMNS = { + "action_artifacts": ("file_compressed_size", "file_size"), + "action_tasks": ("log_length", "log_size"), +} +GITEA_SALVAGE_CLOSURE_ISSUE_SUBRELATION_COUNTS = ( + "comment_assignee", + "comment_assignee_team", + "comment_cross_reference", + "comment_cross_reference_comment", + "comment_current_milestone", + "comment_current_project", + "comment_dependent_issue", + "comment_label", + "comment_old_milestone", + "comment_old_project", + "comment_resolve_doer", + "comment_review", + "comment_tracked_time", + "content_history_comment", + "content_history_issue", + "pull_merger", + "review_reviewer_team", +) +GITEA_SALVAGE_CLOSURE_ISSUE_SUBRELATION_ACTORS = ( + "comment_assignee", + "comment_resolve_doer", + "pull_merger", +) +GITEA_SALVAGE_CLOSURE_ISSUE_SUBRELATION_REPOSITORIES = ( + "comment_cross_reference", + "comment_current_milestone", + "comment_current_project", + "comment_dependent_issue", + "comment_label", + "comment_old_milestone", + "comment_old_project", +) +GITEA_SALVAGE_CLOSURE_ISSUE_SUBRELATION_NULLABLE_COLUMNS = ( + "comment.assignee_id", + "comment.dependent_issue_id", + "comment.label_id", + "comment.milestone_id", + "comment.old_milestone_id", + "comment.old_project_id", + "comment.original_author_id", + "comment.project_id", + "comment.ref_action", + "comment.ref_comment_id", + "comment.ref_is_pull", + "comment.ref_issue_id", + "comment.ref_repo_id", + "comment.resolve_doer_id", + "comment.review_id", + "comment.time_id", + "issue_content_history.comment_id", + "pull_request.merger_id", + "review.original_author_id", +) +GITEA_SALVAGE_CLOSURE_EXTERNAL_AUTHOR_SOURCES = ("comment", "review") +GITEA_SALVAGE_DISPOSITION_REMAINING_BLOCKERS = ( + "attachment-physical-verifier-pending", + "candidate-root-activation-hard-frozen", + "collaboration-kept-user-mapping-verifier-pending", + "forensic-ref-archive-verifier-pending", + "issue-pr-metadata-sanitized-archive-verifier-pending", + "lfs-reachable-pointer-physical-verifier-pending", + "package-action-physical-closure-verifier-pending", + "reference-manifest-fsck-reachability-verifier-pending", + "repository-object-reconstruction-verifier-pending", + "target-unit-policy-acceptance-pending", + "unsupported-schema-catalog-verifier-pending", +) +GITEA_SALVAGE_CLOSURE_REMAINING_BLOCKERS = ( + "attachment-physical-verifier-pending", + "candidate-root-activation-hard-frozen", + "closure-report-review-pin-pending", + "collaboration-kept-user-mapping-verifier-pending", + "forensic-ref-archive-verifier-pending", + "issue-pr-metadata-sanitized-archive-verifier-pending", + "lfs-reachable-pointer-physical-verifier-pending", + "package-action-physical-closure-verifier-pending", + "reference-manifest-fsck-reachability-verifier-pending", + "repository-object-reconstruction-verifier-pending", + "target-unit-policy-acceptance-pending", + "unsupported-schema-catalog-verifier-pending", +) +GITEA_SALVAGE_FS_IOC_GETFLAGS = 0x80086601 +GITEA_SALVAGE_FS_NOCOW_FL = 0x00800000 +GITEA_SALVAGE_EXCLUSIVE_ALLOCATION_BUDGET_BYTES = 512 * 1024 * 1024 +GITEA_SALVAGE_DERIVED_INFO_PACKS_MAX_BYTES = 1024 * 1024 +GITEA_SALVAGE_DERIVED_PACK_BITMAP_MAX_BYTES = 2 * 1024 * 1024 +GITEA_SALVAGE_DERIVED_COMMIT_GRAPH_MAX_BYTES = 1024 * 1024 +GITEA_SALVAGE_EVIDENCE_MAX_BYTES = 8 * 1024 * 1024 +GITEA_SALVAGE_UNSUPPORTED_SIZE_PER_ROW_MAX_BYTES = 16 * 1024**4 +GITEA_SALVAGE_UNSUPPORTED_SIZE_TOTAL_MAX_BYTES = (1 << 63) - 1 +GITEA_SALVAGE_EXPECTED_UNSUPPORTED_SCHEMA_SHA256 = None +GITEA_SALVAGE_UNSUPPORTED_REPOSITORY_TABLES = ( + ("issues", "issue", "repo_id"), + ("pull_requests_base", "pull_request", "base_repo_id"), + ("pull_requests_head", "pull_request", "head_repo_id"), + ("releases", "release", "repo_id"), + ("attachments", "attachment", "repo_id"), + ("collaborators", "collaboration", "repo_id"), + ("deploy_keys", "deploy_key", "repo_id"), + ("webhooks", "webhook", "repo_id"), + ("protected_branches", "protected_branch", "repo_id"), + ("milestones", "milestone", "repo_id"), + ("labels", "label", "repo_id"), + ("stars", "star", "repo_id"), + ("watches", "watch", "repo_id"), + ("access_grants", "access", "repo_id"), + ("topics", "repo_topic", "repo_id"), + ("mirrors", "mirror", "repo_id"), + ("push_mirrors", "push_mirror", "repo_id"), + ("lfs_objects", "lfs_meta_object", "repository_id"), + ("lfs_locks", "lfs_lock", "repo_id"), + ("packages", "package", "repo_id"), + ("action_runs", "action_run", "repo_id"), + ("action_schedules", "action_schedule", "repo_id"), + ("action_runners", "action_runner", "repo_id"), + ("action_variables", "action_variable", "repo_id"), + ("action_secrets", "secret", "repo_id"), +) +GITEA_SALVAGE_UNSUPPORTED_REPOSITORY_TEXT_METADATA = ( + "description", + "website", + "original_url", + "topics", + "avatar", +) +GITEA_SALVAGE_UNSUPPORTED_REPOSITORY_NUMERIC_HINTS = ( + "num_watches", + "num_stars", + "num_issues", + "num_pulls", + "num_milestones", + "num_projects", + "num_action_runs", + "lfs_size", +) +# This is a code-owned, schema-only review surface. Table and column names from +# the compromised database never become SQL identifiers. Direct repository +# relations above are counted now; the tables below expose the exact schema +# needed to design a later, reviewed issue/PR/package/Actions dependency closure. +GITEA_SALVAGE_UNSUPPORTED_SCHEMA_TABLES = tuple( + sorted( + { + "repository", + "repo_unit", + *(table for _label, table, _column in GITEA_SALVAGE_UNSUPPORTED_REPOSITORY_TABLES), + "action_artifact", + "action_run_index", + "action_run_job", + "action_task", + "attachment", + "comment", + "issue_assignees", + "issue_content_history", + "issue_dependency", + "issue_label", + "issue_user", + "issue_watch", + "notification", + "package_blob", + "package_file", + "package_property", + "package_version", + "project", + "project_board", + "project_issue", + "pull_auto_merge", + "reaction", + "review", + "review_state", + "stopwatch", + "tracked_time", + } + ) +) + COMPONENTS = { "engine": { "payload_root": Path("/volume2/nodedc-demo"), @@ -1921,6 +2500,15 @@ COMPONENTS = { # separately registers the third stateless service. "services": ("device-control-core", "device-gateway"), }, + "gitea": { + "payload_root": GITEA_ROOT, + "compose_root": GITEA_ROOT, + "compose_project": GITEA_COMPOSE_PROJECT, + "compose_files": (GITEA_ROOT / GITEA_COMPOSE_REL,), + "bootstrap_root": True, + "compose_no_deps": True, + "services": (GITEA_SERVICE,), + }, "proxy-contur": { "payload_root": Path("/volume1/docker/proxy-contur"), "compose_root": Path("/volume1/docker/proxy-contur"), @@ -3501,6 +4089,8 @@ def denied_payload_path(component, rel): "services/device-backhaul-target/Dockerfile", ): pass + elif component == "gitea" and rel == GITEA_COMPOSE_REL: + pass elif component == "proxy-contur" and rel in ( "Dockerfile", "docker-compose.yml", @@ -3651,6 +4241,30 @@ def denied_payload_path(component, rel): "services/device-manager/Dockerfile.bak", )): return "device-plane backup file" + elif component == "gitea": + runtime_prefixes = ( + "data", + "config", + "secrets", + "repositories", + "users", + "tokens", + "hooks", + "logs", + "backups", + ) + if rel == ".env" or rel.startswith(".env."): + return "gitea env file" + if rel.startswith(( + "docker-compose.gitea.yml.bak", + "deployment/gitea-fresh-install-v1.json.bak", + "deployment/gitea-incident-salvage-v1.json.bak", + "deployment/gitea-incident-salvage/confirmed-disposition-v1.json.bak", + "deployment/gitea-incident-salvage/confirmed-decision.json.bak", + "deployment/gitea-incident-salvage/users.decisions.csv.bak", + "deployment/gitea-incident-salvage/repositories.decisions.csv.bak", + )): + return "gitea backup file" elif component == "n8n-private-extension": # An extension release is inert data at this boundary. Activation is # Engine-owned and must never be smuggled into the artifact as a script, @@ -3951,6 +4565,10 @@ def allowed_payload_path(component, rel): )): return True + if component == "gitea": + if rel in (*GITEA_FRESH_INSTALL_ENTRIES, *GITEA_SALVAGE_ENTRIES): + return True + if component == "n8n-private-extension": parts = PurePosixPath(rel).parts if ( @@ -4099,6 +4717,1340 @@ def require_exact_json_keys(value, expected, label): die(f"{label} keys mismatch: expected={sorted(expected)} actual={sorted(actual)}") +def expected_gitea_fresh_install_descriptor(): + return { + "schemaVersion": "nodedc.gitea.fresh-install.v1", + "action": "fresh-install", + "component": "gitea", + "installRoot": str(GITEA_ROOT), + "compose": { + "file": GITEA_COMPOSE_REL, + "project": GITEA_COMPOSE_PROJECT, + "service": GITEA_SERVICE, + "sha256": GITEA_COMPOSE_SHA256, + }, + "runtime": { + "image": GITEA_IMAGE, + "platform": "linux/amd64", + "pullPolicy": "never", + "minimumComposeVersion": "2.20.1", + "transport": "unix:/run/gitea/gitea.sock", + "socketBind": ( + "/volume1/docker/nodedc-gitea/socket:/run/gitea" + ), + "ssh": "disabled-no-published-port", + "database": "fresh-sqlite-only", + "lfs": "disabled-pending-reviewed-restore-transition", + "networkMode": "none", + "logging": "bounded-json-file-10m-x3", + "stopGracePeriod": "30s", + }, + "trust": { + "artifactSecrets": "forbidden", + "runtimeSecrets": "runner-managed-file-mounts", + "legacyRootAccess": "forbidden", + "legacyDatabaseImport": "forbidden", + "legacyRepositoryImport": "forbidden", + }, + "reverseProxyPrerequisite": { + "managedOutsideArtifact": True, + "requiredDsmUpstream": "127.0.0.1:3000", + "requiredNginxBridge": str(GITEA_NGINX_BRIDGE_CONFIG), + "requiredNginxBridgeSha256": GITEA_NGINX_BRIDGE_SHA256, + "requiredUnixUpstream": str(GITEA_SOCKET_FILE), + "mustBeCompletedBeforeApply": True, + }, + "rollback": ( + "stop-candidate-preserve-fresh-runtime-state-and-restore-source" + ), + } + + +def validate_gitea_fresh_install_payload(payload_dir, entries): + if tuple(entries) != GITEA_FRESH_INSTALL_ENTRIES: + die("Gitea fresh-install artifact entry set mismatch") + descriptor = read_strict_json( + payload_dir / GITEA_FRESH_INSTALL_DESCRIPTOR_REL, + "Gitea fresh-install descriptor", + max_bytes=16 * 1024, + ) + if descriptor != expected_gitea_fresh_install_descriptor(): + die("Gitea fresh-install descriptor mismatch") + + compose = payload_dir / GITEA_COMPOSE_REL + try: + compose_stat = compose.lstat() + except FileNotFoundError: + die("Gitea fresh-install Compose file is missing") + if stat.S_ISLNK(compose_stat.st_mode) or not stat.S_ISREG(compose_stat.st_mode): + die("Gitea fresh-install Compose file is unsafe") + if sha256_file(compose) != GITEA_COMPOSE_SHA256: + die("Gitea fresh-install Compose digest mismatch") + try: + compose_text = compose.read_text(encoding="utf-8") + except UnicodeDecodeError: + die("Gitea fresh-install Compose file is not utf-8") + if compose_text.count(GITEA_IMAGE) != 1: + die("Gitea fresh-install image identity mismatch") + for required in ( + "platform: linux/amd64", + "pull_policy: never", + "network_mode: none", + 'user: "1000:1000"', + "stop_grace_period: 30s", + "driver: json-file", + 'max-size: "10m"', + 'max-file: "3"', + "GITEA__server__PROTOCOL: http+unix", + "GITEA__server__HTTP_ADDR: /run/gitea/gitea.sock", + 'GITEA__server__UNIX_SOCKET_PERMISSION: "0666"', + "GITEA__server__LOCAL_ROOT_URL: http://unix/", + 'GITEA__server__DISABLE_SSH: "true"', + 'GITEA__server__LFS_START_SERVER: "false"', + 'GITEA__server__LFS_ALLOW_PURE_SSH: "false"', + "GITEA__security__SECRET_KEY_URI: file:/run/secrets/gitea_secret_key", + "GITEA__security__INTERNAL_TOKEN_URI: file:/run/secrets/gitea_internal_token", + "GITEA__security__TWO_FACTOR_AUTH: enforced", + 'GITEA__security__REVERSE_PROXY_LIMIT: "1"', + "GITEA__security__ALLOWED_HOST_LIST: loopback", + "GITEA__security__REVERSE_PROXY_TRUSTED_PROXIES: 127.0.0.0/8,::1/128", + 'GITEA__service__DISABLE_REGISTRATION: "true"', + 'GITEA__service__ENABLE_REVERSE_PROXY_AUTHENTICATION: "false"', + 'GITEA__service__ENABLE_REVERSE_PROXY_AUTHENTICATION_API: "false"', + 'GITEA__service__ENABLE_REVERSE_PROXY_AUTO_REGISTRATION: "false"', + 'GITEA__service__ENABLE_BASIC_AUTHENTICATION: "false"', + 'GITEA__admin__DISABLE_REGULAR_ORG_CREATION: "true"', + "GITEA__admin__USER_DISABLED_FEATURES: deletion,manage_ssh_keys,manage_gpg_keys,change_username", + 'GITEA__security__DISABLE_GIT_HOOKS: "true"', + 'GITEA__security__DISABLE_WEBHOOKS: "true"', + 'GITEA__repository__DISABLE_MIGRATIONS: "true"', + 'GITEA__packages__ENABLED: "false"', + 'GITEA__oauth2__ENABLED: "false"', + 'GITEA__openid__ENABLE_OPENID_SIGNIN: "false"', + 'GITEA__cron.update_checker__ENABLED: "false"', + "source: /volume1/docker/nodedc-gitea/socket", + "target: /run/gitea", + "create_host_path: false", + "read_only: true", + "no-new-privileges:true", + ): + if required not in compose_text: + die(f"Gitea fresh-install Compose boundary missing: {required}") + for forbidden in ( + "4022", + "2222:2222", + "0.0.0.0:3000", + "ports:", + "networks:", + "/var/run/docker.sock", + "/volume1/docker/gitea", + "privileged: true", + "pull_policy: always", + "__FILE", + "GITEA__security__SECRET_KEY:", + "GITEA__security__INTERNAL_TOKEN:", + "GITEA__server__LFS_JWT_SECRET:", + "GITEA__server__LFS_JWT_SECRET_URI", + "gitea_lfs_jwt_secret", + "lfs-jwt-secret", + "GITEA__server__REVERSE_PROXY_LIMIT", + "GITEA__server__REVERSE_PROXY_TRUSTED_PROXIES", + "GITEA__security__ENABLE_REVERSE_PROXY_AUTHENTICATION", + "GITEA__security__ENABLE_REVERSE_PROXY_AUTHENTICATION_API", + "GITEA__security__ENABLE_REVERSE_PROXY_AUTO_REGISTRATION", + "GITEA__service__DISABLE_REGULAR_ORG_CREATION", + "GITEA__service__USER_DISABLED_FEATURES", + ): + if forbidden in compose_text: + die(f"Gitea fresh-install Compose boundary violation: {forbidden}") + return descriptor + + +def canonical_gitea_salvage_record_sha256(record): + encoded = json.dumps( + record, + ensure_ascii=False, + sort_keys=True, + separators=(",", ":"), + ).encode("utf-8") + return hashlib.sha256(encoded).hexdigest() + + +def canonical_gitea_salvage_evidence(value, label): + try: + encoded = json.dumps( + value, + ensure_ascii=True, + sort_keys=True, + separators=(",", ":"), + allow_nan=False, + ).encode("utf-8") + except (TypeError, ValueError): + die(f"Gitea salvage {label} is not canonical JSON") + if len(encoded) > GITEA_SALVAGE_EVIDENCE_MAX_BYTES: + die(f"Gitea salvage {label} exceeds the evidence byte limit") + return { + "bytes": len(encoded), + "json": encoded.decode("utf-8"), + "sha256": hashlib.sha256(encoded).hexdigest(), + } + + +def read_gitea_salvage_csv(path, expected_header, expected_rows, label): + try: + path_stat = path.lstat() + except FileNotFoundError: + die(f"{label} is missing") + if ( + stat.S_ISLNK(path_stat.st_mode) + or not stat.S_ISREG(path_stat.st_mode) + or path_stat.st_size < len(",".join(expected_header)) + or path_stat.st_size > 8 * 1024 * 1024 + ): + die(f"{label} is unsafe") + try: + raw = path.read_text(encoding="utf-8") + except (OSError, UnicodeDecodeError): + die(f"{label} is not readable utf-8") + if raw.startswith("\ufeff") or "\x00" in raw or not raw.endswith("\n"): + die(f"{label} has invalid text framing") + try: + reader = csv.DictReader(io.StringIO(raw, newline=""), strict=True) + if tuple(reader.fieldnames or ()) != tuple(expected_header): + die(f"{label} header mismatch") + rows = list(reader) + except csv.Error as exc: + die(f"{label} is invalid csv: {exc}") + if len(rows) != expected_rows: + die(f"{label} row count mismatch") + for index, row in enumerate(rows, 2): + if None in row or set(row) != set(expected_header): + die(f"{label} row {index} shape mismatch") + for key, value in row.items(): + if ( + not isinstance(value, str) + or len(value.encode("utf-8")) > 1024 + or any(ord(char) < 0x20 for char in value) + ): + die(f"{label} row {index} field {key} is unsafe") + return rows + + +def validate_gitea_salvage_incident_disposition(payload_dir): + path = payload_dir / GITEA_SALVAGE_DISPOSITION_REL + try: + path_stat = path.lstat() + raw = path.read_bytes() + except (FileNotFoundError, OSError): + die("Gitea salvage confirmed disposition is missing or unreadable") + if ( + stat.S_ISLNK(path_stat.st_mode) + or not stat.S_ISREG(path_stat.st_mode) + or path_stat.st_size < 2 + or path_stat.st_size > 64 * 1024 + ): + die("Gitea salvage confirmed disposition is unsafe") + if sha256_file(path) != GITEA_SALVAGE_DISPOSITION_SHA256: + die("Gitea salvage confirmed disposition digest mismatch") + disposition = read_strict_json( + path, + "Gitea salvage confirmed disposition", + max_bytes=64 * 1024, + ) + try: + canonical = ( + json.dumps( + disposition, + ensure_ascii=True, + sort_keys=True, + separators=(",", ":"), + allow_nan=False, + ).encode("utf-8") + + b"\n" + ) + except (TypeError, ValueError): + die("Gitea salvage confirmed disposition is not canonical JSON") + if raw != canonical: + die("Gitea salvage confirmed disposition byte framing is not canonical") + + require_exact_json_keys( + disposition, + { + "activation", + "confirmation", + "incidentId", + "referencePolicy", + "remainingBlockers", + "repositoryStatePolicy", + "schemaVersion", + "scope", + "sourceEvidence", + }, + "Gitea salvage confirmed disposition", + ) + if ( + disposition.get("schemaVersion") + != "nodedc.gitea.incident-disposition.v1" + or disposition.get("incidentId") != "gitea-20260814" + or disposition.get("confirmation") + != { + "confirmationRecordedAt": "2026-08-14T15:16:04Z", + "confirmationScope": ( + "exact-10-users-45-repositories;heads-tags-live;" + "pull-remote-sealed-archive;non-git-state-policy;" + "no-2fa-change" + ), + "confirmationSource": "owner-instruction-in-current-incident-thread", + "confirmedBy": "dctouch", + } + or disposition.get("scope") + != { + "keptRepositories": 45, + "keptUsers": 10, + "referenceDecisionRows": 105, + "repositoryStores": 49, + } + ): + die("Gitea salvage confirmed disposition identity mismatch") + + source = disposition.get("sourceEvidence") + expected_source = { + "databaseSha256": GITEA_SALVAGE_SNAPSHOT_DATABASE_SHA256, + "identityDecisionManifestSha256": ( + GITEA_SALVAGE_DECISION_MANIFEST_SHA256 + ), + "referenceManifestBytes": 36010, + "referenceManifestReviewState": "observed-unreviewed", + "referenceManifestSha256": ( + GITEA_SALVAGE_DISPOSITION_REFERENCE_MANIFEST_SHA256 + ), + "snapshotUuid": GITEA_SALVAGE_SNAPSHOT_UUID, + "unsupportedRepositoryReportBytes": 119468, + "unsupportedRepositoryReportSha256": ( + GITEA_SALVAGE_EXPECTED_UNSUPPORTED_REPORT_SHA256 + ), + "unsupportedSchemaCatalogReviewState": "observed-unreviewed", + "unsupportedSchemaCatalogSha256": ( + GITEA_SALVAGE_DISPOSITION_UNSUPPORTED_SCHEMA_SHA256 + ), + } + if source != expected_source: + die("Gitea salvage confirmed disposition source mismatch") + + reference_policy = disposition.get("referencePolicy") + require_exact_json_keys( + reference_policy, + { + "archiveOnly", + "exactDecisions", + "executionAuthority", + "forensicScope", + "headInvariants", + "liveRestore", + "unknownNamespacePolicy", + }, + "Gitea salvage confirmed reference policy", + ) + if ( + reference_policy.get("executionAuthority") + != "exact-decisions-one-to-one-observed-reference-manifest" + or reference_policy.get("unknownNamespacePolicy") != "reject" + or reference_policy.get("forensicScope") + != {"allDiscoveredRefs": 105, "preserveNameOidEvidence": True} + or reference_policy.get("liveRestore") + != { + "copyLegacyHeadOrRefFiles": False, + "mainRepositoryHeads": 85, + "mainRepositoryTags": 4, + "totalRefs": 93, + "wikiHeads": 4, + } + or reference_policy.get("archiveOnly") + != { + "immutableArchiveRequired": True, + "neverAutoPromote": True, + "pullRefs": 5, + "remoteRefs": 7, + "totalRefs": 12, + } + or reference_policy.get("headInvariants") + != { + "allowedMissingTargets": [ + { + "head": "refs/heads/main", + "oldRepositoryId": 70, + "repositoryPath": "silver/nodedc_aegis.git", + "wiki": False, + }, + { + "head": "refs/heads/main", + "oldRepositoryId": 2119, + "repositoryPath": "silver/nodedc_device_core.git", + "wiki": False, + }, + ], + "objectFormat": "sha1", + "stores": 49, + "symbolicHeads": 49, + "targetsPresentAmongLiveRefs": 47, + } + ): + die("Gitea salvage confirmed reference policy mismatch") + + exact_decisions = reference_policy.get("exactDecisions") + if not isinstance(exact_decisions, list) or len(exact_decisions) != 105: + die("Gitea salvage exact reference decision count mismatch") + previous_identity = None + reference_identities = set() + store_identities = set() + reference_counts = { + "main_heads": 0, + "main_tags": 0, + "pull": 0, + "remote": 0, + "wiki_heads": 0, + } + for record in exact_decisions: + require_exact_json_keys( + record, + { + "disposition", + "name", + "oid", + "oldRepositoryId", + "repositoryPath", + "wiki", + }, + "Gitea salvage exact reference decision", + ) + old_repo_id = record.get("oldRepositoryId") + repository_path = record.get("repositoryPath") + name = record.get("name") + oid = record.get("oid") + wiki = record.get("wiki") + if not isinstance(repository_path, str): + die("Gitea salvage exact reference path is invalid") + try: + validate_posix_path(repository_path) + except DeployError: + die("Gitea salvage exact reference path is invalid") + identity = ( + old_repo_id, + int(wiki) if isinstance(wiki, bool) else -1, + repository_path, + name, + oid, + ) + if ( + not isinstance(old_repo_id, int) + or old_repo_id <= 0 + or not isinstance(wiki, bool) + or len(PurePosixPath(repository_path).parts) != 2 + or not isinstance(name, str) + or not gitea_salvage_refname_is_safe(name) + or not isinstance(oid, str) + or not re.fullmatch(r"[a-f0-9]{40}", oid) + or previous_identity is not None + and identity <= previous_identity + or identity in reference_identities + ): + die("Gitea salvage exact reference decision is invalid") + previous_identity = identity + reference_identities.add(identity) + store_identities.add((old_repo_id, repository_path, wiki)) + + if wiki and name.startswith("refs/heads/"): + reference_counts["wiki_heads"] += 1 + expected_disposition = "LIVE_RESTORE" + elif not wiki and name.startswith("refs/heads/"): + reference_counts["main_heads"] += 1 + expected_disposition = "LIVE_RESTORE" + elif not wiki and name.startswith("refs/tags/"): + reference_counts["main_tags"] += 1 + expected_disposition = "LIVE_RESTORE" + elif not wiki and name.startswith("refs/pull/"): + reference_counts["pull"] += 1 + expected_disposition = "SEALED_ARCHIVE_ONLY" + elif not wiki and name.startswith("refs/remotes/"): + reference_counts["remote"] += 1 + expected_disposition = "SEALED_ARCHIVE_ONLY" + else: + die("Gitea salvage exact reference namespace is rejected") + if record.get("disposition") != expected_disposition: + die("Gitea salvage exact reference disposition mismatch") + if ( + reference_counts + != { + "main_heads": 85, + "main_tags": 4, + "pull": 5, + "remote": 7, + "wiki_heads": 4, + } + or len(store_identities) != 47 + or { + old_repo_id + for old_repo_id, _path, wiki in store_identities + if wiki + } + != {1, 16, 38, 51} + ): + die("Gitea salvage exact reference partition mismatch") + + state_policy = disposition.get("repositoryStatePolicy") + require_exact_json_keys( + state_policy, + { + "attachments", + "directRelations", + "legacySecretsCredentialsSessionsKeysIntegrations", + "lfs", + "metadataArchive", + "numericHints", + "relationSemantics", + "schemaOnlyDependencyGroups", + "sourceNonzeroCategories", + "textMetadata", + "topics", + "units", + }, + "Gitea salvage confirmed repository-state policy", + ) + expected_nonzero = [ + "access_grants", + "attachments", + "collaborators", + "issues", + "labels", + "lfs_objects", + "pull_requests_base", + "pull_requests_head", + "releases", + "repo_units", + "repository_hint:lfs_size", + "repository_hint:num_issues", + "repository_hint:num_pulls", + "repository_hint:num_watches", + "repository_metadata:description", + "repository_metadata:topics", + "watches", + ] + if state_policy.get("sourceNonzeroCategories") != expected_nonzero: + die("Gitea salvage source nonzero-category partition mismatch") + + direct_counts = { + "access_grants": 32, + "action_runners": 0, + "action_runs": 0, + "action_schedules": 0, + "action_secrets": 0, + "action_variables": 0, + "attachments": 230, + "collaborators": 32, + "deploy_keys": 0, + "issues": 867, + "labels": 7, + "lfs_locks": 0, + "lfs_objects": 573941, + "milestones": 0, + "mirrors": 0, + "packages": 0, + "protected_branches": 0, + "pull_requests_base": 5, + "pull_requests_head": 5, + "push_mirrors": 0, + "releases": 4, + "stars": 0, + "topics": 0, + "watches": 45, + "webhooks": 0, + } + nonzero_dispositions = { + "access_grants": "SANITIZED_ARCHIVE_THEN_RECOMPUTE", + "attachments": "PHYSICAL_VERIFY_THEN_SANITIZED_ARCHIVE", + "collaborators": ( + "SANITIZED_ARCHIVE_THEN_RECREATE_AFTER_KEPT_USER_MAPPING" + ), + "issues": "SANITIZED_ARCHIVE_ONLY", + "labels": "SANITIZED_ARCHIVE_ONLY", + "lfs_objects": "REACHABILITY_AND_PHYSICAL_VERIFY_THEN_RESTORE", + "pull_requests_base": ( + "SANITIZED_ARCHIVE_ONLY_DIRECTIONAL_PROJECTION" + ), + "pull_requests_head": ( + "SANITIZED_ARCHIVE_ONLY_DIRECTIONAL_PROJECTION" + ), + "releases": "SANITIZED_ARCHIVE_ONLY", + "watches": "DROP_AND_REGENERATE", + } + expected_direct_relations = [ + { + "disposition": ( + "ASSERT_ZERO_AND_DROP" + if source_count == 0 + else nonzero_dispositions[label] + ), + "label": label, + "sourceCount": source_count, + } + for label, source_count in sorted(direct_counts.items()) + ] + if state_policy.get("directRelations") != expected_direct_relations: + die("Gitea salvage direct-relation disposition partition mismatch") + + expected_schema_groups = [ + { + "disposition": ( + "SANITIZED_ARCHIVE_ONLY_AFTER_SCHEMA_AND_JOIN_VERIFIER" + ), + "group": "issue-pr-release-project", + "tables": [ + "comment", + "issue_assignees", + "issue_content_history", + "issue_dependency", + "issue_label", + "issue_user", + "issue_watch", + "notification", + "project", + "project_board", + "project_issue", + "pull_auto_merge", + "reaction", + "review", + "review_state", + "stopwatch", + "tracked_time", + ], + }, + { + "disposition": ( + "ASSERT_ZERO_AND_DROP_AFTER_SCHEMA_AND_PHYSICAL_CLOSURE_VERIFIER" + ), + "group": "packages", + "tables": [ + "package_blob", + "package_file", + "package_property", + "package_version", + ], + }, + { + "disposition": ( + "ASSERT_ZERO_AND_DROP_AFTER_SCHEMA_AND_PHYSICAL_CLOSURE_VERIFIER" + ), + "group": "actions", + "tables": [ + "action_artifact", + "action_run_index", + "action_run_job", + "action_task", + ], + }, + ] + if state_policy.get("schemaOnlyDependencyGroups") != expected_schema_groups: + die("Gitea salvage schema-only dependency disposition mismatch") + covered_tables = [ + table + for group in expected_schema_groups + for table in group["tables"] + ] + if len(covered_tables) != 25 or len(set(covered_tables)) != 25: + die("Gitea salvage schema-only dependency partition is not total") + + expected_units = { + "actionsGloballyDisabled": True, + "legacyConfigImported": False, + "legacyRowsImported": False, + "packagesGloballyDisabled": True, + "rows": [ + { + "disposition": disposition_name, + "name": name, + "sourceCount": source_count, + "targetCount": target_count, + "type": unit_type, + } + for ( + unit_type, + name, + source_count, + target_count, + disposition_name, + ) in ( + (1, "Code", 45, 45, "CREATE_CLEAN_DEFAULT"), + (2, "Issues", 45, 45, "CREATE_CLEAN_DEFAULT"), + (3, "Pull Requests", 45, 45, "CREATE_CLEAN_DEFAULT"), + (4, "Releases", 45, 45, "CREATE_CLEAN_DEFAULT"), + (5, "Wiki", 45, 45, "CREATE_CLEAN_DEFAULT"), + ( + 6, + "External Wiki", + 0, + 0, + "ASSERT_SOURCE_ZERO_AND_FORBID_TARGET", + ), + ( + 7, + "External Tracker", + 0, + 0, + "ASSERT_SOURCE_ZERO_AND_FORBID_TARGET", + ), + (8, "Projects", 45, 45, "CREATE_CLEAN_DEFAULT"), + (9, "Packages", 45, 0, "DROP_AND_DISABLE_TARGET"), + ( + 10, + "Actions", + 0, + 0, + "ASSERT_SOURCE_ZERO_AND_DISABLE_TARGET", + ), + ) + ], + "targetRowTotal": 270, + "targetUniquePerRepositoryAndType": True, + } + if state_policy.get("units") != expected_units: + die("Gitea salvage repo-unit disposition mismatch") + + expected_numeric_hints = [ + { + "disposition": "DROP_AND_RECOMPUTE", + "name": name, + "sourceNonzero": name + in {"lfs_size", "num_issues", "num_pulls", "num_watches"}, + } + for name in ( + "lfs_size", + "num_action_runs", + "num_issues", + "num_milestones", + "num_projects", + "num_pulls", + "num_stars", + "num_watches", + ) + ] + if ( + state_policy.get("attachments") + != { + "declaredRows": 230, + "disposition": "PHYSICAL_VERIFY_THEN_SANITIZED_ARCHIVE", + "legacyRowsImported": False, + "physicalPresenceClaimed": False, + } + or state_policy.get("lfs") + != { + "associationRows": 573941, + "disposition": ( + "DISABLED_UNTIL_REACHABLE_POINTER_PHYSICAL_SHA_VERIFIER" + ), + "legacyRowsImported": False, + "physicalPresenceClaimed": False, + } + or state_policy.get("metadataArchive") + != { + "categories": [ + "issues", + "pull-requests", + "comments", + "releases", + "labels", + "projects", + "repository-description", + ], + "legacyRowsImported": False, + "mode": "SANITIZED_IMMUTABLE_ARCHIVE_ONLY", + } + or state_policy.get("numericHints") != expected_numeric_hints + or state_policy.get("relationSemantics") + != { + "attachmentLinkCountsMayOverlap": True, + "pullBaseAndHeadAreDirectionalProjections": True, + } + or state_policy.get("textMetadata") + != [ + { + "disposition": "ASSERT_ZERO_AND_DROP", + "name": "avatar", + "sourceRepositories": 0, + }, + { + "disposition": "SANITIZED_ARCHIVE_ONLY", + "name": "description", + "sourceRepositories": 1, + }, + { + "disposition": "ASSERT_ZERO_AND_DROP", + "name": "original_url", + "sourceRepositories": 0, + }, + { + "disposition": ( + "VERIFY_SEMANTIC_EMPTY_THEN_REBUILD_FROM_RELATIONS" + ), + "name": "topics", + "sourceRepositories": 45, + }, + { + "disposition": "ASSERT_ZERO_AND_DROP", + "name": "website", + "sourceRepositories": 0, + }, + ] + or state_policy.get("topics") + != { + "acceptedLegacyEncoding": ( + "canonical-json-null-or-canonical-json-array" + ), + "canonicalLowercaseSortedUnique": True, + "expectedMaterialRepositories": 0, + "expectedRelationalRows": 0, + "expectedSerializedArrays": 0, + "expectedSerializedNulls": 45, + "expectedTopics": 0, + "legacyRowsImported": False, + "maxUtf8BytesPerTopic": 35, + "relationalAuthority": "repo_topic-join-topic", + "targetCache": "rebuild-from-relations", + "topicNamePattern": r"^[a-z0-9][-.a-z0-9]*$", + } + or state_policy.get( + "legacySecretsCredentialsSessionsKeysIntegrations" + ) + != "ZERO_NEVER_IMPORT" + ): + die("Gitea salvage repository-state disposition mismatch") + + if disposition.get("activation") != { + "allowedOperation": "canonical-plan-only", + "applyFrozen": True, + "freezeBoundary": "before-candidate-root-creation", + }: + die("Gitea salvage disposition activation policy mismatch") + if disposition.get("remainingBlockers") != list( + GITEA_SALVAGE_DISPOSITION_REMAINING_BLOCKERS + ): + die("Gitea salvage disposition blocker contract mismatch") + return disposition + + +def validate_gitea_salvage_closure_disposition(payload_dir): + path = payload_dir / GITEA_SALVAGE_CLOSURE_DISPOSITION_REL + try: + path_stat = path.lstat() + raw = path.read_bytes() + except (FileNotFoundError, OSError): + die("Gitea salvage closure disposition is missing or unreadable") + if ( + stat.S_ISLNK(path_stat.st_mode) + or not stat.S_ISREG(path_stat.st_mode) + or path_stat.st_size < 2 + or path_stat.st_size > 64 * 1024 + or sha256_file(path) != GITEA_SALVAGE_CLOSURE_DISPOSITION_SHA256 + ): + die("Gitea salvage closure disposition identity is unsafe") + disposition = read_strict_json( + path, + "Gitea salvage closure disposition", + max_bytes=64 * 1024, + ) + try: + canonical = ( + json.dumps( + disposition, + ensure_ascii=True, + sort_keys=True, + separators=(",", ":"), + allow_nan=False, + ).encode("utf-8") + + b"\n" + ) + except (TypeError, ValueError): + die("Gitea salvage closure disposition is not canonical JSON") + if raw != canonical: + die("Gitea salvage closure disposition byte framing is not canonical") + require_exact_json_keys( + disposition, + { + "activation", + "authority", + "closureReport", + "incidentId", + "policies", + "predecessor", + "remainingBlockers", + "schemaVersion", + "scope", + "sourceEvidence", + }, + "Gitea salvage closure disposition", + ) + if ( + disposition.get("schemaVersion") + != "nodedc.gitea.incident-closure-disposition.v1" + or disposition.get("incidentId") != "gitea-20260814" + or disposition.get("activation") + != { + "allowedOperation": "canonical-plan-only", + "applyFrozen": True, + "freezeBoundary": "before-candidate-root-creation", + } + or disposition.get("authority") + != { + "policyScope": ( + "access-collaboration-issue-pr-attachment-release-label-" + "project-unit-package-action-closure" + ), + "source": "owner-instruction-in-current-incident-thread", + "state": "confirmed-policy-evidence-review-pending", + } + or disposition.get("scope") + != { + "deletedRepositories": 2013, + "deletedUsers": 962, + "keptRepositories": 45, + "keptUsers": 10, + } + or disposition.get("predecessor") + != { + "artifactSha256": GITEA_SALVAGE_CLOSURE_PREDECESSOR_ARTIFACT_SHA256, + "dispositionFile": GITEA_SALVAGE_DISPOSITION_REL, + "dispositionSha256": ( + GITEA_SALVAGE_CLOSURE_PREDECESSOR_DISPOSITION_SHA256 + ), + } + or disposition.get("sourceEvidence") + != { + "databaseSha256": GITEA_SALVAGE_SNAPSHOT_DATABASE_SHA256, + "identityDecisionManifestSha256": ( + GITEA_SALVAGE_DECISION_MANIFEST_SHA256 + ), + "referenceManifestSha256": ( + GITEA_SALVAGE_DISPOSITION_REFERENCE_MANIFEST_SHA256 + ), + "semanticTopicsSha256": GITEA_SALVAGE_DISPOSITION_TOPICS_SHA256, + "snapshotUuid": GITEA_SALVAGE_SNAPSHOT_UUID, + "unsupportedRepositoryReportSha256": ( + GITEA_SALVAGE_EXPECTED_UNSUPPORTED_REPORT_SHA256 + ), + "unsupportedSchemaCatalogSha256": ( + GITEA_SALVAGE_DISPOSITION_UNSUPPORTED_SCHEMA_SHA256 + ), + } + or disposition.get("closureReport") + != { + "expectedBytes": None, + "expectedSha256": None, + "reviewState": "canonical-plan-output-unreviewed", + "schema": "nodedc.gitea.salvage-closure-inventory/v1", + } + or disposition.get("remainingBlockers") + != list(GITEA_SALVAGE_CLOSURE_REMAINING_BLOCKERS) + ): + die("Gitea salvage closure disposition contract mismatch") + + policies = disposition.get("policies") + require_exact_json_keys( + policies, + { + "accessCollaboration", + "attachments", + "issuesPullRequestsMetadata", + "packagesActions", + "releasesLabelsProjects", + "repositoryState", + "units", + }, + "Gitea salvage closure policies", + ) + if ( + policies["accessCollaboration"].get("accessCache") + != "DROP_RESET_RECOMPUTE" + or policies["accessCollaboration"].get("collaboration") + != "RECREATE_ONLY_KEPT_ACTORS_AFTER_REVIEWED_OLD_TO_NEW_ID_MAP" + or policies["accessCollaboration"].get("legacyRowsImported") is not False + or policies["attachments"].get("databaseManifest") + != "ID_UUID_RELATIONS_DECLARED_SIZE_ONLY" + or policies["attachments"].get("legacyRowsImported") is not False + or policies["issuesPullRequestsMetadata"].get("archive") + != "SANITIZED_IMMUTABLE_ARCHIVE_ONLY" + or policies["issuesPullRequestsMetadata"].get("legacyRowsImported") + is not False + or policies["issuesPullRequestsMetadata"].get("subrelationClosure") + != { + "commentHistoryMerger": ( + "SCHEMA_BOUND_EXACT_RELATION_COUNTS_AND_CLASSES" + ), + "externalAuthors": ( + "PRESENCE_AND_NAME_BYTE_LENGTHS_ONLY_NO_LOCAL_USER_MAPPING" + ), + "legacyRowsImported": False, + "teamRelations": ( + "EXACT_ROW_TEAM_ORG_IDS_SEALED_HOLD_AND_BLOCK_IF_PRESENT" + ), + } + or policies["packagesActions"].get("actionsTarget") != "DISABLED" + or policies["packagesActions"].get("packageTarget") != "DISABLED" + or policies["packagesActions"].get("legacyPayloadSecretTokenLogImported") + is not False + or policies["repositoryState"].get( + "legacyHooksWebhooksKeysTokensSessionsCredentialsSecrets" + ) + != "IMPORT_ZERO" + or policies["units"].get("enabledCleanTypes") != [1, 2, 3, 4, 5, 8] + or policies["units"].get("disabledTypes") != [6, 7, 9, 10] + or policies["units"].get("legacyConfigImported") is not False + or policies["units"].get("legacyRowsImported") is not False + ): + die("Gitea salvage closure policy mismatch") + return disposition + +def validate_gitea_salvage_decision_bundle(payload_dir): + manifest_path = payload_dir / GITEA_SALVAGE_DECISION_MANIFEST_REL + users_path = payload_dir / GITEA_SALVAGE_USERS_REL + repositories_path = payload_dir / GITEA_SALVAGE_REPOSITORIES_REL + for path, expected, label in ( + ( + manifest_path, + GITEA_SALVAGE_DECISION_MANIFEST_SHA256, + "Gitea salvage decision manifest", + ), + (users_path, GITEA_SALVAGE_USERS_SHA256, "Gitea salvage user decisions"), + ( + repositories_path, + GITEA_SALVAGE_REPOSITORIES_SHA256, + "Gitea salvage repository decisions", + ), + ): + try: + path_stat = path.lstat() + except FileNotFoundError: + die(f"{label} is missing") + if stat.S_ISLNK(path_stat.st_mode) or not stat.S_ISREG(path_stat.st_mode): + die(f"{label} is unsafe") + if sha256_file(path) != expected: + die(f"{label} digest mismatch") + + manifest = read_strict_json( + manifest_path, + "Gitea salvage decision manifest", + max_bytes=32 * 1024, + ) + require_exact_json_keys( + manifest, + { + "confirmation_scope", + "confirmed_at", + "confirmed_by", + "counts", + "execution_policy", + "repositories_file", + "repositories_file_sha256", + "schema", + "source", + "users_file", + "users_file_sha256", + }, + "Gitea salvage decision manifest", + ) + if ( + manifest.get("schema") != "nodedc.gitea.incident-decision/v2" + or manifest.get("confirmed_by") != "dctouch" + or manifest.get("repositories_file") != "repositories.decisions.csv" + or manifest.get("users_file") != "users.decisions.csv" + or manifest.get("repositories_file_sha256") + != GITEA_SALVAGE_REPOSITORIES_SHA256 + or manifest.get("users_file_sha256") != GITEA_SALVAGE_USERS_SHA256 + or manifest.get("counts") + != { + "repositories_delete": 2013, + "repositories_keep": 45, + "repositories_total": 2058, + "users_delete": 962, + "users_keep_active": 2, + "users_keep_locked": 8, + "users_total": 972, + } + or manifest.get("execution_policy") + != { + "explicit_rows_only": True, + "two_factor_authentication": ( + "not changed; owner will configure it manually" + ), + "wildcard_or_owner_only_execution_forbidden": True, + } + ): + die("Gitea salvage decision manifest contract mismatch") + source = manifest.get("source") + if ( + not isinstance(source, dict) + or source.get("snapshot_uuid") != GITEA_SALVAGE_SNAPSHOT_UUID + or source.get("database_sha256") + != GITEA_SALVAGE_SNAPSHOT_DATABASE_SHA256 + ): + die("Gitea salvage decision source mismatch") + + user_header = ( + "decision", + "user_id", + "owner", + "lower_owner", + "desired_active", + "desired_restricted", + "desired_admin", + "current_active", + "current_restricted", + "current_admin", + "repository_count", + "record_sha256", + ) + users = read_gitea_salvage_csv( + users_path, + user_header, + 972, + "Gitea salvage user decisions", + ) + user_ids = set() + user_counts = {name: 0 for name in GITEA_SALVAGE_USER_COUNTS} + previous_user_id = 0 + for row in users: + try: + user_id = int(row["user_id"]) + current_values = tuple( + int(row[key]) + for key in ( + "current_active", + "current_restricted", + "current_admin", + "repository_count", + ) + ) + except ValueError: + die("Gitea salvage user decision has a non-integer field") + if ( + user_id <= previous_user_id + or user_id in user_ids + or not re.fullmatch(r"[A-Za-z0-9_.-]{1,40}", row["owner"]) + or row["lower_owner"] != row["owner"].casefold() + or row["record_sha256"] + != row["record_sha256"].lower() + or not re.fullmatch(r"[a-f0-9]{64}", row["record_sha256"]) + or any(value < 0 for value in current_values) + ): + die("Gitea salvage user decision identity mismatch") + previous_user_id = user_id + user_ids.add(user_id) + expected_name = GITEA_SALVAGE_ACTIVE_USERS.get( + user_id, + GITEA_SALVAGE_LOCKED_USERS.get(user_id), + ) + if user_id in GITEA_SALVAGE_ACTIVE_USERS: + expected = ( + "KEEP_ACTIVE", + "1", + "0", + "1" if user_id == 1 else "0", + ) + elif user_id in GITEA_SALVAGE_LOCKED_USERS: + expected = ("KEEP_LOCKED", "0", "1", "0") + else: + expected = ("DELETE", "", "", "") + if ( + (row["decision"], row["desired_active"], row["desired_restricted"], row["desired_admin"]) + != expected + or expected_name is not None + and row["owner"] != expected_name + ): + die("Gitea salvage user disposition mismatch") + user_counts[row["decision"]] += 1 + if user_counts != GITEA_SALVAGE_USER_COUNTS: + die("Gitea salvage user disposition count mismatch") + + repository_header = ( + "decision", + "repo_id", + "owner_id", + "owner", + "lower_owner", + "slug", + "lower_slug", + "repo_relative_path", + "wiki_relative_path", + "record_sha256", + ) + repositories = read_gitea_salvage_csv( + repositories_path, + repository_header, + 2058, + "Gitea salvage repository decisions", + ) + repository_ids = set() + repository_paths = set() + repository_counts = {name: 0 for name in GITEA_SALVAGE_REPOSITORY_COUNTS} + previous_repository_id = 0 + kept_by_owner = {} + for row in repositories: + try: + repo_id = int(row["repo_id"]) + owner_id = int(row["owner_id"]) + except ValueError: + die("Gitea salvage repository decision has a non-integer field") + expected_repo_path = f"{row['lower_owner']}/{row['lower_slug']}.git" + expected_wiki_path = f"{row['lower_owner']}/{row['lower_slug']}.wiki.git" + for relative in (row["repo_relative_path"], row["wiki_relative_path"]): + validate_posix_path(relative) + if len(PurePosixPath(relative).parts) != 2: + die("Gitea salvage repository path depth mismatch") + if ( + repo_id <= previous_repository_id + or repo_id in repository_ids + or not re.fullmatch(r"[A-Za-z0-9_.-]{1,100}", row["slug"]) + or row["lower_owner"] != row["owner"].casefold() + or row["lower_slug"] != row["slug"].casefold() + or row["repo_relative_path"] != expected_repo_path + or row["wiki_relative_path"] != expected_wiki_path + or row["repo_relative_path"] in repository_paths + or row["wiki_relative_path"] in repository_paths + or not re.fullmatch(r"[a-f0-9]{64}", row["record_sha256"]) + ): + die("Gitea salvage repository identity mismatch") + previous_repository_id = repo_id + repository_ids.add(repo_id) + repository_paths.update((row["repo_relative_path"], row["wiki_relative_path"])) + expected_owner = GITEA_SALVAGE_ACTIVE_USERS.get(owner_id) + expected_decision = "KEEP" if expected_owner == row["owner"] else "DELETE" + if row["decision"] != expected_decision: + die("Gitea salvage repository disposition mismatch") + repository_counts[row["decision"]] += 1 + if row["decision"] == "KEEP": + kept_by_owner[row["owner"]] = kept_by_owner.get(row["owner"], 0) + 1 + if ( + repository_counts != GITEA_SALVAGE_REPOSITORY_COUNTS + or kept_by_owner != {"dctouch": 32, "SILVER": 13} + ): + die("Gitea salvage repository disposition count mismatch") + return { + "manifest": manifest, + "users": users, + "repositories": repositories, + "kept_users": [row for row in users if row["decision"] != "DELETE"], + "kept_repositories": [ + row for row in repositories if row["decision"] == "KEEP" + ], + } + + +def validate_gitea_incident_salvage_payload(payload_dir, entries): + if tuple(entries) != GITEA_SALVAGE_ENTRIES: + die("Gitea incident-salvage artifact entry set mismatch") + descriptor_path = payload_dir / GITEA_SALVAGE_DESCRIPTOR_REL + if sha256_file(descriptor_path) != GITEA_SALVAGE_DESCRIPTOR_SHA256: + die("Gitea incident-salvage descriptor digest mismatch") + descriptor = read_strict_json( + descriptor_path, + "Gitea incident-salvage descriptor", + max_bytes=64 * 1024, + ) + if ( + descriptor.get("schemaVersion") + != "nodedc.gitea.incident-salvage.v3" + or descriptor.get("action") != "clean-state-salvage" + or descriptor.get("component") != "gitea" + or descriptor.get("incidentId") != "gitea-20260814" + or (descriptor.get("compose") or {}).get("sha256") + != GITEA_SALVAGE_COMPOSE_SHA256 + or (descriptor.get("runtime") or {}).get("image") + != GITEA_SALVAGE_IMAGE + or (descriptor.get("runtime") or {}).get("imageId") + != GITEA_SALVAGE_IMAGE_ID + or (descriptor.get("runtime") or {}).get("repoDigest") + != GITEA_SALVAGE_REPO_DIGEST + or (descriptor.get("disposition") or {}).get("schema") + != "nodedc.gitea.incident-disposition.v1" + or (descriptor.get("disposition") or {}).get("file") + != GITEA_SALVAGE_DISPOSITION_REL + or (descriptor.get("disposition") or {}).get("sha256") + != GITEA_SALVAGE_DISPOSITION_SHA256 + or (descriptor.get("closureDisposition") or {}).get("schema") + != "nodedc.gitea.incident-closure-disposition.v1" + or (descriptor.get("closureDisposition") or {}).get("file") + != GITEA_SALVAGE_CLOSURE_DISPOSITION_REL + or (descriptor.get("closureDisposition") or {}).get("sha256") + != GITEA_SALVAGE_CLOSURE_DISPOSITION_SHA256 + or (descriptor.get("closureDisposition") or {}).get( + "predecessorArtifactSha256" + ) + != GITEA_SALVAGE_CLOSURE_PREDECESSOR_ARTIFACT_SHA256 + or (descriptor.get("identity") or {}).get("twoFactorAuthentication") + != "not-configured-by-transition" + or (descriptor.get("identity") or {}).get("preserveNumericUserIds") + is not False + or (descriptor.get("identity") or {}).get("preserveNumericRepositoryIds") + is not False + ): + die("Gitea incident-salvage descriptor contract mismatch") + + compose = payload_dir / GITEA_COMPOSE_REL + try: + compose_stat = compose.lstat() + compose_text = compose.read_text(encoding="utf-8") + except (FileNotFoundError, OSError, UnicodeDecodeError): + die("Gitea incident-salvage Compose file is missing or invalid") + if ( + stat.S_ISLNK(compose_stat.st_mode) + or not stat.S_ISREG(compose_stat.st_mode) + or sha256_file(compose) != GITEA_SALVAGE_COMPOSE_SHA256 + or compose_text.count(GITEA_SALVAGE_IMAGE) != 1 + ): + die("Gitea incident-salvage Compose identity mismatch") + for required in ( + "platform: linux/amd64", + "pull_policy: never", + "network_mode: none", + 'user: "1000:1000"', + "- /usr/local/bin/gitea", + "- /etc/gitea/app.ini", + 'restart: "no"', + "read_only: true", + "- ALL", + "no-new-privileges:true", + "source: /volume1/docker/nodedc-gitea/data", + "target: /data", + "source: /volume1/docker/nodedc-gitea/config", + "target: /etc/gitea", + "source: /volume1/docker/nodedc-gitea/socket", + "target: /run/gitea", + "create_host_path: false", + ): + if required not in compose_text: + die(f"Gitea incident-salvage Compose boundary missing: {required}") + for forbidden in ( + "ports:", + "networks:", + "/var/run/docker.sock", + "/volume1/docker/gitea", + "privileged: true", + "4022", + "TWO_FACTOR_AUTH", + "LFS_JWT_SECRET", + "restart: unless-stopped", + ): + if forbidden in compose_text: + die(f"Gitea incident-salvage Compose boundary violation: {forbidden}") + disposition = validate_gitea_salvage_incident_disposition(payload_dir) + closure_disposition = validate_gitea_salvage_closure_disposition(payload_dir) + decisions = validate_gitea_salvage_decision_bundle(payload_dir) + return { + "closure_disposition": closure_disposition, + "descriptor": descriptor, + "disposition": disposition, + **decisions, + } + + def validate_n8n_package_tarball(package_path, release): package_stat = package_path.lstat() if stat.S_ISLNK(package_stat.st_mode) or not stat.S_ISREG(package_stat.st_mode): @@ -8962,6 +10914,10 @@ def load_artifact(artifact, work_dir): die(f"files.txt entry missing in payload: {rel}") validate_payload_tree(manifest["component"], payload_dir, entries) + if is_gitea_fresh_install_slice(manifest["component"], entries): + validate_gitea_fresh_install_payload(payload_dir, entries) + elif is_gitea_incident_salvage_slice(manifest["component"], entries): + validate_gitea_incident_salvage_payload(payload_dir, entries) if is_platform_device_core_hub_trust_slice( manifest["component"], entries, @@ -9461,6 +11417,29 @@ def component_root(component): return COMPONENTS[component]["payload_root"] +def is_gitea_fresh_install_slice(component, entries): + return ( + component == "gitea" + and entries is not None + and tuple(entries) == GITEA_FRESH_INSTALL_ENTRIES + ) + + +def is_gitea_incident_salvage_slice(component, entries): + return ( + component == "gitea" + and entries is not None + and tuple(entries) == GITEA_SALVAGE_ENTRIES + ) + + +def is_gitea_bootstrap_slice(component, entries): + return is_gitea_fresh_install_slice( + component, + entries, + ) or is_gitea_incident_salvage_slice(component, entries) + + def is_platform_device_core_hub_trust_slice(component, entries): return ( component == "platform" @@ -17286,6 +19265,6126 @@ def inspect_optional_local_image(image_ref, label): return image_id +def gitea_compose_project_container_ids(): + result = subprocess.run( + [ + str(DOCKER), + "container", + "ls", + "--all", + "--filter", + f"label=com.docker.compose.project={GITEA_COMPOSE_PROJECT}", + "--format", + "{{.ID}}", + ], + check=False, + capture_output=True, + text=True, + ) + if result.returncode != 0: + die("Gitea fresh-install container inventory failed") + container_ids = [ + line.strip() + for line in result.stdout.splitlines() + if line.strip() + ] + if any(not re.fullmatch(r"[a-f0-9]{12,64}", value) for value in container_ids): + die("Gitea fresh-install container inventory is invalid") + return tuple(container_ids) + + +def inspect_gitea_docker_compose_version(): + result = subprocess.run( + [str(DOCKER), "compose", "version", "--short"], + check=False, + capture_output=True, + text=True, + ) + raw = result.stdout.strip() + match = re.fullmatch( + r"v?(\d+)\.(\d+)\.(\d+)(?:[-+][A-Za-z0-9.-]+)?", + raw, + ) + if result.returncode != 0 or match is None: + die("Gitea fresh-install Docker Compose version is unavailable or invalid") + version = tuple(int(part) for part in match.groups()) + if version < GITEA_MINIMUM_COMPOSE_VERSION: + die( + "Gitea fresh-install requires Docker Compose >= " + + ".".join(str(part) for part in GITEA_MINIMUM_COMPOSE_VERSION) + ) + return raw + + +def inspect_gitea_docker_server_version(): + result = subprocess.run( + [str(DOCKER), "version", "--format", "{{.Server.Version}}"], + check=False, + capture_output=True, + text=True, + ) + version = result.stdout.strip() + if result.returncode != 0 or version != GITEA_REQUIRED_DOCKER_VERSION: + die( + "Gitea fresh-install requires exact Docker Server " + f"{GITEA_REQUIRED_DOCKER_VERSION}" + ) + return version + + +def validate_gitea_compose_schema(compose_path): + result = subprocess.run( + [ + str(DOCKER), + "compose", + "--project-name", + GITEA_COMPOSE_PROJECT, + "--file", + str(compose_path), + "config", + "--quiet", + ], + cwd=str(compose_path.parent), + check=False, + capture_output=True, + text=True, + ) + if result.returncode != 0: + die("Gitea fresh-install Compose schema/config validation failed") + + +def inspect_gitea_local_image(): + images = docker_json( + ["image", "inspect", GITEA_IMAGE], + "Gitea pinned image inspect", + ) + if not isinstance(images, list) or len(images) != 1 or not isinstance(images[0], dict): + die("Gitea pinned image inspect shape mismatch") + image = images[0] + if ( + image.get("Id") != GITEA_IMAGE_ID + or tuple(image.get("RepoDigests") or ()) != (GITEA_REPO_DIGEST,) + or image.get("Architecture") != "amd64" + or image.get("Os") != "linux" + or (image.get("Config") or {}).get("User") != "1000:1000" + ): + die("Gitea pinned image identity/repo-digest/platform/user mismatch") + return GITEA_IMAGE_ID + + +def gitea_loopback_listener_inodes(port): + expected_local = f"0100007F:{port:04X}" + try: + lines = Path("/proc/net/tcp").read_text(encoding="ascii").splitlines() + except (OSError, UnicodeDecodeError): + die("Gitea Nginx listener inventory is unavailable") + inodes = set() + for line in lines[1:]: + fields = line.split() + if len(fields) < 10: + die("Gitea Nginx listener inventory is invalid") + if fields[1] == expected_local and fields[3] == "0A": + if not fields[9].isdigit() or fields[9] == "0": + die("Gitea Nginx listener inode is invalid") + inodes.add(fields[9]) + if not inodes: + die(f"Gitea Nginx loopback listener is missing: {port}") + return inodes + + +def gitea_listener_process_owners(inodes): + pending = set(inodes) + owners = {} + try: + process_entries = tuple(Path("/proc").iterdir()) + except OSError: + die("Gitea Nginx process inventory is unavailable") + for process in process_entries: + if not process.name.isdigit(): + continue + try: + process_stat = process.stat() + comm = (process / "comm").read_text(encoding="ascii").strip() + descriptors = tuple((process / "fd").iterdir()) + except FileNotFoundError: + continue + except (OSError, UnicodeDecodeError): + die("Gitea Nginx process ownership inventory failed") + matched = set() + for descriptor in descriptors: + try: + target = os.readlink(descriptor) + except FileNotFoundError: + continue + except OSError: + die("Gitea Nginx descriptor ownership inventory failed") + match = re.fullmatch(r"socket:\[(\d+)\]", target) + if match and match.group(1) in inodes: + matched.add(match.group(1)) + for inode in matched: + owners.setdefault(inode, set()).add((comm, process_stat.st_uid)) + pending.discard(inode) + if pending: + die("Gitea Nginx listener owner is unproven") + return owners + + +def validate_gitea_nginx_listener(): + inodes = gitea_loopback_listener_inodes(GITEA_HTTP_PORT) + owners = gitea_listener_process_owners(inodes) + allowed = {("nginx", 0), ("nginx", GITEA_NGINX_WORKER_UID)} + if any(not values or not values.issubset(allowed) for values in owners.values()): + die("Gitea TCP/3000 listener is not exclusively owned by Nginx") + return { + "address": "127.0.0.1:3000", + "owners": sorted({owner for values in owners.values() for owner in values}), + } + + +def validate_gitea_nginx_bridge_prerequisite(): + try: + bridge_stat = GITEA_NGINX_BRIDGE_CONFIG.lstat() + bridge = GITEA_NGINX_BRIDGE_CONFIG.read_bytes() + except OSError: + die("Gitea Nginx UDS bridge config is missing or unreadable") + if ( + stat.S_ISLNK(bridge_stat.st_mode) + or not stat.S_ISREG(bridge_stat.st_mode) + or bridge_stat.st_uid != 0 + or bridge_stat.st_gid != 0 + or stat.S_IMODE(bridge_stat.st_mode) != 0o644 + or bridge != GITEA_NGINX_BRIDGE_CONTENT.encode("utf-8") + or hashlib.sha256(bridge).hexdigest() != GITEA_NGINX_BRIDGE_SHA256 + ): + die("Gitea Nginx UDS bridge identity/metadata mismatch") + version = subprocess.run( + [str(GITEA_NGINX), "-v"], + check=False, + capture_output=True, + text=True, + ) + version_text = "\n".join( + value.strip() for value in (version.stdout, version.stderr) if value.strip() + ) + if version.returncode != 0 or version_text != GITEA_NGINX_VERSION: + die("Gitea Nginx version mismatch") + config_test = subprocess.run( + [str(GITEA_NGINX), "-t", "-c", str(GITEA_NGINX_MAIN_CONFIG)], + check=False, + capture_output=True, + text=True, + timeout=30, + ) + if config_test.returncode != 0: + die("Gitea Nginx effective config test failed") + listener = validate_gitea_nginx_listener() + return { + "config": str(GITEA_NGINX_BRIDGE_CONFIG), + "sha256": GITEA_NGINX_BRIDGE_SHA256, + "version": GITEA_NGINX_VERSION.removeprefix("nginx version: "), + "listener": listener, + "unix_upstream": str(GITEA_SOCKET_FILE), + } + + +def validate_gitea_no_docker_port_publications(): + result = subprocess.run( + [str(DOCKER), "container", "ls", "--format", "{{.ID}}"], + check=False, + capture_output=True, + text=True, + ) + if result.returncode != 0: + die("Gitea running Docker container inventory failed") + container_ids = tuple( + line.strip() for line in result.stdout.splitlines() if line.strip() + ) + if any(not re.fullmatch(r"[a-f0-9]{12,64}", value) for value in container_ids): + die("Gitea running Docker container inventory is invalid") + if not container_ids: + return "no-running-docker-publications-3000-4022" + containers = docker_json( + ["inspect", *container_ids], + "Gitea running Docker publication inspect", + ) + if not isinstance(containers, list) or len(containers) != len(container_ids): + die("Gitea running Docker publication inventory shape mismatch") + forbidden_ports = {str(GITEA_HTTP_PORT), str(GITEA_DISABLED_SSH_HOST_PORT)} + for container in containers: + if not isinstance(container, dict): + die("Gitea running Docker publication inventory is invalid") + bindings = (container.get("HostConfig") or {}).get("PortBindings") or {} + if not isinstance(bindings, dict): + die("Gitea running Docker publication bindings are invalid") + for values in bindings.values(): + if values is None: + continue + if not isinstance(values, list): + die("Gitea running Docker publication bindings are invalid") + for value in values: + if not isinstance(value, dict): + die("Gitea running Docker publication bindings are invalid") + if str(value.get("HostPort") or "") in forbidden_ports: + die("Docker must not publish Gitea TCP/3000 or TCP/4022") + return "no-running-docker-publications-3000-4022" + + +def validate_gitea_legacy_candidate_network_absent(): + result = subprocess.run( + [str(DOCKER), "network", "inspect", GITEA_FORBIDDEN_LEGACY_NETWORK], + check=False, + capture_output=True, + text=True, + ) + if result.returncode == 0: + die("Gitea forbidden legacy candidate network still exists") + error = result.stderr.strip() + if ( + result.returncode != 1 + or result.stdout.strip() + or not re.search(r"(?:No such network|network .+ not found)", error) + ): + die("Gitea forbidden legacy candidate network absence is unproven") + return "absent" + + +def inspect_gitea_builtin_none_network(): + networks = docker_json( + ["network", "inspect", "none"], + "Gitea built-in none network inspect", + ) + if ( + not isinstance(networks, list) + or len(networks) != 1 + or not isinstance(networks[0], dict) + ): + die("Gitea built-in none network inspect shape mismatch") + network = networks[0] + network_id = network.get("Id") + if ( + network.get("Name") != "none" + or not isinstance(network_id, str) + or not re.fullmatch(r"[a-f0-9]{64}", network_id) + or network.get("Scope") != "local" + or network.get("Driver") != "null" + or network.get("Internal") is not False + or network.get("Attachable") is not False + or network.get("Ingress") is not False + or network.get("ConfigOnly") is not False + ): + die("Gitea built-in none network identity mismatch") + return network + + +def gitea_generated_reverse_proxy_server_block(generated): + marker = "server_name git.dcserve.ru ;" + if generated.count(marker) != 1: + die("Gitea DSM generated reverse-proxy server identity mismatch") + marker_index = generated.index(marker) + server_index = generated.rfind("server {", 0, marker_index) + if server_index < 0: + die("Gitea DSM generated reverse-proxy server block is missing") + brace_index = generated.index("{", server_index) + depth = 0 + for index in range(brace_index, len(generated)): + character = generated[index] + if character == "{": + depth += 1 + elif character == "}": + depth -= 1 + if depth == 0: + return generated[server_index:index + 1] + die("Gitea DSM generated reverse-proxy server block is invalid") + + +def validate_gitea_reverse_proxy_prerequisite(): + try: + config_stat = GITEA_REVERSE_PROXY_CONFIG.lstat() + except FileNotFoundError: + die("Gitea DSM reverse-proxy persistent config is missing") + if ( + stat.S_ISLNK(config_stat.st_mode) + or not stat.S_ISREG(config_stat.st_mode) + or config_stat.st_uid != 0 + or config_stat.st_gid != 0 + or config_stat.st_size < 64 + or config_stat.st_size > 8 * 1024 * 1024 + ): + die("Gitea DSM reverse-proxy persistent config is unsafe") + document = read_strict_json( + GITEA_REVERSE_PROXY_CONFIG, + "Gitea DSM reverse-proxy persistent config", + max_bytes=8 * 1024 * 1024, + ) + rule = document.get(GITEA_REVERSE_PROXY_UUID) if isinstance(document, dict) else None + expected = { + "backend": {"fqdn": "127.0.0.1", "port": 3000, "protocol": 0}, + "customize_headers": [], + "description": GITEA_REVERSE_PROXY_DESCRIPTION, + "frontend": { + "acl": None, + "fqdn": "git.dcserve.ru", + "https": {"hsts": False}, + "port": 443, + "protocol": 1, + }, + "proxy_connect_timeout": 60, + "proxy_http_version": 1, + "proxy_intercept_errors": False, + "proxy_read_timeout": 60, + "proxy_send_timeout": 60, + } + if rule != expected: + die( + "Gitea DSM reverse-proxy prerequisite mismatch; apply the " + "separate reviewed proxy transition first" + ) + + try: + generated_stat = GITEA_REVERSE_PROXY_GENERATED_CONFIG.lstat() + generated = GITEA_REVERSE_PROXY_GENERATED_CONFIG.read_text( + encoding="utf-8" + ) + except (FileNotFoundError, UnicodeDecodeError): + die("Gitea DSM generated reverse-proxy config is missing or invalid") + server_block = gitea_generated_reverse_proxy_server_block(generated) + generated_headers = {} + for name, value in re.findall( + r"(?m)^\s*proxy_set_header\s+([A-Za-z0-9-]+)\s+([^;]+);\s*$", + server_block, + ): + generated_headers.setdefault(name.casefold(), []).append(value.strip()) + expected_headers = { + "host": ["$http_host"], + "x-real-ip": ["$remote_addr"], + "x-forwarded-for": ["$proxy_add_x_forwarded_for"], + "x-forwarded-proto": ["$scheme"], + } + host_guard = re.search( + r"if\s*\(\s*\$host\s+!~\s+\"\(\^git\.dcserve\.ru\$\)\"\s*\)" + r"\s*\{\s*return\s+404\s*;\s*\}", + server_block, + ) + if ( + stat.S_ISLNK(generated_stat.st_mode) + or not stat.S_ISREG(generated_stat.st_mode) + or generated_stat.st_uid != 0 + or generated_stat.st_size < 64 + or generated_stat.st_size > 4 * 1024 * 1024 + or server_block.count("proxy_pass http://127.0.0.1:3000;") != 1 + or "proxy_pass http://172.22.0.222:3000;" in server_block + or any( + generated_headers.get(name) != values + for name, values in expected_headers.items() + ) + or host_guard is None + ): + die("Gitea DSM generated reverse-proxy prerequisite mismatch") + return { + "persistent_rule": GITEA_REVERSE_PROXY_UUID, + "upstream": "http://127.0.0.1:3000", + "generated_config": str(GITEA_REVERSE_PROXY_GENERATED_CONFIG), + } + + +def gitea_iptables_rule_present(arguments): + result = subprocess.run( + [str(GITEA_IPTABLES), "-w", "5", "-C", *arguments], + check=False, + capture_output=True, + text=True, + ) + if result.returncode not in (0, 1): + die("Gitea firewall prerequisite inventory failed") + return result.returncode == 0 + + +def validate_gitea_firewall_prerequisite(): + if not GITEA_IPTABLES.is_file(): + die("Gitea firewall prerequisite command is missing") + if gitea_iptables_rule_present( + ("INPUT", "-p", "tcp", "--dport", "3000", "-j", "DROP") + ): + die( + "Gitea loopback TCP/3000 is still blocked; apply the separate " + "reviewed firewall transition first" + ) + required = ( + ( + "OUTPUT", + "-o", + "lo", + "-d", + f"{GITEA_LEGACY_REVERSE_PROXY_IP}/32", + "-p", + "tcp", + "--dport", + "3000", + "-j", + "DROP", + ), + ("INPUT", "-p", "tcp", "--dport", "4022", "-j", "DROP"), + ) + if not all(gitea_iptables_rule_present(arguments) for arguments in required): + die("Gitea legacy firewall isolation prerequisite is incomplete") + return { + "loopback_3000": "not-blocked-by-emergency-input-drop", + "legacy_172_22_0_222_3000": "output-drop-present", + "legacy_4022": "input-drop-present", + } + + +def validate_legacy_gitea_container_isolation(): + result = subprocess.run( + [str(DOCKER), "container", "inspect", GITEA_LEGACY_CONTAINER], + check=False, + capture_output=True, + text=True, + ) + if result.returncode != 0: + return "absent" + try: + containers = json.loads(result.stdout) + except json.JSONDecodeError: + die("legacy Gitea container isolation inventory is invalid") + if ( + not isinstance(containers, list) + or len(containers) != 1 + or not isinstance(containers[0], dict) + ): + die("legacy Gitea container isolation inventory shape mismatch") + container = containers[0] + if ( + (container.get("State") or {}).get("Running") is not False + or (container.get("HostConfig") or {}).get("RestartPolicy", {}).get("Name") + != "no" + ): + die("legacy Gitea container is not stopped with restart disabled") + return "stopped-restart-disabled" + + +def preflight_gitea_fresh_install(): + try: + GITEA_ROOT.lstat() + except FileNotFoundError: + pass + else: + die( + "Gitea fresh-install root already exists; updates, imports, and " + "partial-state reuse are forbidden" + ) + if gitea_compose_project_container_ids(): + die("Gitea fresh-install Compose project already has containers") + legacy_candidate_network = validate_gitea_legacy_candidate_network_absent() + assert_loopback_tcp_port_closed(GITEA_DISABLED_SSH_HOST_PORT) + nginx_bridge = validate_gitea_nginx_bridge_prerequisite() + reverse_proxy = validate_gitea_reverse_proxy_prerequisite() + firewall = validate_gitea_firewall_prerequisite() + legacy_container = validate_legacy_gitea_container_isolation() + docker_publications = validate_gitea_no_docker_port_publications() + docker_version = inspect_gitea_docker_server_version() + builtin_none_network = inspect_gitea_builtin_none_network() + compose_version = inspect_gitea_docker_compose_version() + image_id = inspect_gitea_local_image() + return { + "mode": "fresh-root-absent", + "image_id": image_id, + "transport": "unix:/run/gitea/gitea.sock", + "network_mode": "none", + "ssh_host_port": "closed:4022/tcp", + "reverse_proxy_prerequisite": "127.0.0.1:3000", + "nginx_bridge": nginx_bridge, + "docker_version": docker_version, + "builtin_none_network_id": builtin_none_network["Id"], + "compose_version": compose_version, + "reverse_proxy": reverse_proxy, + "firewall": firewall, + "docker_publications": docker_publications, + "legacy_candidate_network": legacy_candidate_network, + "legacy_container": legacy_container, + } + + +def inspect_gitea_salvage_local_image(): + images = docker_json( + ["image", "inspect", GITEA_SALVAGE_IMAGE], + "Gitea salvage pinned image inspect", + ) + if not isinstance(images, list) or len(images) != 1 or not isinstance(images[0], dict): + die("Gitea salvage pinned image inspect shape mismatch") + image = images[0] + config = image.get("Config") or {} + if ( + image.get("Id") != GITEA_SALVAGE_IMAGE_ID + or tuple(image.get("RepoDigests") or ()) != (GITEA_SALVAGE_REPO_DIGEST,) + or image.get("Architecture") != "amd64" + or image.get("Os") != "linux" + or config.get("User") != "1000:1000" + ): + die("Gitea salvage pinned image identity/repo-digest/platform/user mismatch") + return GITEA_SALVAGE_IMAGE_ID + + +def docker_named_container_inspect_fail_closed(name, label): + result = subprocess.run( + [str(DOCKER), "container", "inspect", name], + check=False, + capture_output=True, + text=True, + ) + if result.returncode == 0: + try: + containers = json.loads(result.stdout) + except json.JSONDecodeError: + die(f"{label} returned invalid JSON") + if ( + not isinstance(containers, list) + or len(containers) != 1 + or not isinstance(containers[0], dict) + ): + die(f"{label} shape mismatch") + return containers[0] + + stdout_raw = result.stdout.strip() + if not stdout_raw: + stdout_category = "empty" + stdout_proves_empty = True + elif stdout_raw == "[]": + stdout_category = "json-empty-list" + stdout_proves_empty = True + else: + try: + stdout_json = json.loads(stdout_raw) + except json.JSONDecodeError: + stdout_category = "non-json" + stdout_proves_empty = False + else: + stdout_category = f"json-{type(stdout_json).__name__}" + stdout_proves_empty = False + + error = result.stderr.strip() + exact_no_such = { + f"No such object: {name}": "exact-no-such-object", + f"No such container: {name}": "exact-no-such-container", + f"Error: No such object: {name}": "exact-no-such-object", + f"Error: No such container: {name}": "exact-no-such-container", + f"Error response from daemon: No such object: {name}": ( + "exact-no-such-object" + ), + f"Error response from daemon: No such container: {name}": ( + "exact-no-such-container" + ), + } + stderr_category = exact_no_such.get(error) + if ( + result.returncode == 1 + and stdout_proves_empty + and stderr_category is not None + ): + return None + if stderr_category is None: + lowered = error.lower() + if not error: + stderr_category = "empty" + elif "permission denied" in lowered or "access is denied" in lowered: + stderr_category = "permission-denied" + elif "cannot connect" in lowered or "daemon" in lowered: + stderr_category = "daemon-error" + elif "timed out" in lowered or "deadline exceeded" in lowered: + stderr_category = "timeout" + else: + stderr_category = "other" + die( + f"{label} absence is unproven: rc={result.returncode} " + f"stdout={stdout_category} stderr={stderr_category}" + ) + + +def validate_gitea_salvage_legacy_container(): + if ( + GITEA_SALVAGE_EXPECTED_LEGACY_IMAGE is None + or GITEA_SALVAGE_EXPECTED_LEGACY_IMAGE_ID is None + ): + die( + "Gitea salvage legacy container image/mount identity is not " + "reviewed and pinned" + ) + container = docker_named_container_inspect_fail_closed( + GITEA_LEGACY_CONTAINER, + "Gitea salvage legacy container inspect", + ) + if container is None: + die("Gitea salvage requires the exact stopped legacy container evidence") + config = container.get("Config") or {} + host = container.get("HostConfig") or {} + mounts = container.get("Mounts") or [] + expected_mounts = { + (str(GITEA_SALVAGE_LEGACY_ROOT), "/data", "bind", True), + } + actual_mounts = set() + for mount in mounts: + if not isinstance(mount, dict) or mount.get("RW") is not True: + die("Gitea salvage legacy container mount inventory is invalid") + actual_mounts.add( + ( + mount.get("Source"), + mount.get("Destination"), + mount.get("Type"), + mount.get("RW"), + ) + ) + if ( + container.get("Name") != "/gitea" + or len(mounts) != 1 + or (container.get("State") or {}).get("Running") is not False + or (host.get("RestartPolicy") or {}).get("Name") != "no" + or config.get("Image") != GITEA_SALVAGE_EXPECTED_LEGACY_IMAGE + or container.get("Image") != GITEA_SALVAGE_EXPECTED_LEGACY_IMAGE_ID + or actual_mounts != expected_mounts + ): + die("Gitea salvage legacy container identity/isolation mismatch") + return { + "container_id": container.get("Id"), + "image": config.get("Image"), + "image_id": container.get("Image"), + "mounts": sorted(actual_mounts), + "name": container.get("Name"), + "state": "stopped-restart-no", + } + + +def gitea_salvage_mountpoints(): + try: + lines = Path("/proc/self/mountinfo").read_text(encoding="utf-8").splitlines() + except (OSError, UnicodeError): + die("Gitea salvage cannot attest process mount boundaries") + mountpoints = set() + for line in lines: + fields = line.split() + if len(fields) < 10 or "-" not in fields: + die("Gitea salvage process mount inventory is malformed") + raw = fields[4] + if re.search(r"\\(?!(?:040|011|012|134))", raw): + die("Gitea salvage process mount inventory escape is malformed") + try: + decoded = re.sub( + r"\\([0-7]{3})", + lambda match: chr(int(match.group(1), 8)), + raw, + ) + except (TypeError, ValueError): + die("Gitea salvage process mount inventory escape is malformed") + if not decoded.startswith("/"): + die("Gitea salvage process mount inventory path is unsafe") + mountpoints.add(os.path.normpath(decoded)) + if not mountpoints: + die("Gitea salvage process mount inventory is empty") + return mountpoints + + +def validate_gitea_salvage_path_chain( + root, + target, + final_mode, + label, + allow_missing_final=False, + trusted_device=None, + mountpoints=None, +): + try: + relative = target.relative_to(root) + except (TypeError, ValueError): + die(f"Gitea salvage {label} escapes its trusted root") + if not relative.parts or any(part in ("", ".", "..") for part in relative.parts): + die(f"Gitea salvage {label} path is invalid") + try: + root_stat = root.lstat() + except FileNotFoundError: + die(f"Gitea salvage {label} trusted root is missing") + if stat.S_ISLNK(root_stat.st_mode) or not stat.S_ISDIR(root_stat.st_mode): + die(f"Gitea salvage {label} trusted root is unsafe") + if trusted_device is None: + trusted_device = root_stat.st_dev + elif root_stat.st_dev != trusted_device: + die(f"Gitea salvage {label} trusted root crosses a filesystem boundary") + if mountpoints is None: + mountpoints = gitea_salvage_mountpoints() + current = root + for index, part in enumerate(relative.parts): + current = current / part + final = index == len(relative.parts) - 1 + try: + current_stat = current.lstat() + except FileNotFoundError: + if final and allow_missing_final: + return None + die(f"Gitea salvage {label} is missing") + if stat.S_ISLNK(current_stat.st_mode): + die(f"Gitea salvage {label} path contains a symlink") + if current_stat.st_dev != trusted_device: + die(f"Gitea salvage {label} crosses a filesystem boundary") + if os.path.normpath(str(current)) in mountpoints: + die(f"Gitea salvage {label} crosses a mount boundary") + # Every Btrfs subvolume root has inode 256. The trusted root itself is + # the reviewed snapshot; seeing another inode-256 directory below it + # means the parent ro property does not attest the nested subvolume. + if stat.S_ISDIR(current_stat.st_mode) and current_stat.st_ino == 256: + die(f"Gitea salvage {label} crosses a nested subvolume boundary") + if final: + if not final_mode(current_stat.st_mode): + die(f"Gitea salvage {label} has an unsafe type") + elif not stat.S_ISDIR(current_stat.st_mode): + die(f"Gitea salvage {label} parent has an unsafe type") + return target + + +def validate_gitea_salvage_internal_entry( + path, + path_stat, + trusted_device, + mountpoints, + expected_mode, + label, +): + if stat.S_ISLNK(path_stat.st_mode) or not expected_mode(path_stat.st_mode): + die(f"Gitea salvage {label} has an unsafe type: {path}") + if path_stat.st_dev != trusted_device: + die(f"Gitea salvage {label} crosses a filesystem boundary: {path}") + if os.path.normpath(str(path)) in mountpoints: + die(f"Gitea salvage {label} crosses a mount boundary: {path}") + if stat.S_ISDIR(path_stat.st_mode) and path_stat.st_ino == 256: + die(f"Gitea salvage {label} crosses a nested subvolume boundary: {path}") + + +def probe_gitea_salvage_path_no_follow( + root, + relative, + trusted_device, + mountpoints, + label, + final_mode=lambda _mode: True, +): + parts = PurePosixPath(relative).parts + if ( + not parts + or PurePosixPath(relative).is_absolute() + or any(part in ("", ".", "..") for part in parts) + ): + die(f"Gitea salvage {label} probe path is invalid") + current = root + for index, part in enumerate(parts): + current = current / part + try: + current_stat = current.lstat() + except FileNotFoundError: + return None + except OSError as exc: + error_number = exc.errno if isinstance(exc.errno, int) else "unknown" + die(f"Gitea salvage {label} probe failed: errno={error_number}") + validate_gitea_salvage_internal_entry( + current, + current_stat, + trusted_device, + mountpoints, + stat.S_ISDIR if index < len(parts) - 1 else final_mode, + label, + ) + return current_stat + + +def validate_gitea_salvage_snapshot_boundary(): + snapshot_parent = GITEA_SALVAGE_SNAPSHOT_ROOT.parent + try: + parent_stat = snapshot_parent.lstat() + snapshot_stat = GITEA_SALVAGE_SNAPSHOT_ROOT.lstat() + except FileNotFoundError: + die("Gitea salvage snapshot root is missing") + if ( + stat.S_ISLNK(parent_stat.st_mode) + or not stat.S_ISDIR(parent_stat.st_mode) + or parent_stat.st_uid != 0 + or parent_stat.st_gid != 0 + or stat.S_IMODE(parent_stat.st_mode) != 0o700 + or + stat.S_ISLNK(snapshot_stat.st_mode) + or not stat.S_ISDIR(snapshot_stat.st_mode) + or snapshot_stat.st_uid != 0 + or snapshot_stat.st_gid != 0 + ): + die("Gitea salvage snapshot root metadata mismatch") + if not GITEA_SALVAGE_BTRFS.is_file(): + die("Gitea salvage btrfs command is missing") + show = subprocess.run( + [str(GITEA_SALVAGE_BTRFS), "subvolume", "show", str(GITEA_SALVAGE_SNAPSHOT_ROOT)], + check=False, + capture_output=True, + text=True, + ) + if show.returncode != 0: + die("Gitea salvage snapshot subvolume inventory failed") + uuids = re.findall( + r"(?mi)^\s*UUID:\s*([0-9a-f-]{36})\s*$", + show.stdout, + ) + if uuids != [GITEA_SALVAGE_SNAPSHOT_UUID]: + die("Gitea salvage snapshot UUID mismatch") + readonly = subprocess.run( + [ + str(GITEA_SALVAGE_BTRFS), + "property", + "get", + str(GITEA_SALVAGE_SNAPSHOT_ROOT), + "ro", + ], + check=False, + capture_output=True, + text=True, + ) + if readonly.returncode != 0 or readonly.stdout.strip() != "ro=true": + die("Gitea salvage snapshot is not attestably read-only") + for path, expected_mode, label in ( + (GITEA_SALVAGE_SNAPSHOT_DATABASE, stat.S_ISREG, "database"), + (GITEA_SALVAGE_SNAPSHOT_REPOSITORIES, stat.S_ISDIR, "repository root"), + ): + validate_gitea_salvage_path_chain( + GITEA_SALVAGE_SNAPSHOT_ROOT, + path, + expected_mode, + f"snapshot {label}", + ) + database_stat = GITEA_SALVAGE_SNAPSHOT_DATABASE.lstat() + if ( + database_stat.st_size != GITEA_SALVAGE_SNAPSHOT_DATABASE_BYTES + or sha256_file(GITEA_SALVAGE_SNAPSHOT_DATABASE) + != GITEA_SALVAGE_SNAPSHOT_DATABASE_SHA256 + ): + die("Gitea salvage snapshot database identity mismatch") + return { + "root": str(GITEA_SALVAGE_SNAPSHOT_ROOT), + "uuid": GITEA_SALVAGE_SNAPSHOT_UUID, + "readonly": True, + "database_sha256": GITEA_SALVAGE_SNAPSHOT_DATABASE_SHA256, + } + + +def gitea_salvage_sqlite_connection(database): + connection = sqlite3.connect( + f"file:{database}?mode=ro&immutable=1", + uri=True, + ) + connection.row_factory = sqlite3.Row + connection.execute("PRAGMA query_only=ON") + connection.execute("PRAGMA trusted_schema=OFF") + query_only = connection.execute("PRAGMA query_only").fetchone() + trusted_schema = connection.execute("PRAGMA trusted_schema").fetchone() + if ( + query_only is None + or len(query_only) != 1 + or query_only[0] != 1 + or trusted_schema is None + or len(trusted_schema) != 1 + or trusted_schema[0] != 0 + ): + connection.close() + die("Gitea salvage SQLite read-only safety PRAGMA mismatch") + return connection + + +def gitea_salvage_table_columns(connection, table): + if not re.fullmatch(r"[a-z][a-z0-9_]{0,63}", table): + die("Gitea salvage schema table name is unsafe") + rows = connection.execute(f'PRAGMA table_info("{table}")').fetchall() + return {str(row[1]) for row in rows} + + +def gitea_salvage_sqlite_version(connection): + try: + value = connection.execute("SELECT sqlite_version()").fetchone()[0] + except (sqlite3.DatabaseError, TypeError, IndexError): + die("Gitea salvage SQLite version inventory failed") + match = re.fullmatch(r"([0-9]+)\.([0-9]+)\.([0-9]+)", str(value)) + if match is None: + die("Gitea salvage SQLite version inventory is invalid") + return str(value), tuple(int(part) for part in match.groups()) + + +def gitea_salvage_table_schema( + connection, + table, + table_list_supported, + allowed_tables=None, +): + registry = ( + GITEA_SALVAGE_UNSUPPORTED_SCHEMA_TABLES + if allowed_tables is None + else allowed_tables + ) + if ( + table not in registry + or not re.fullmatch(r"[a-z][a-z0-9_]{0,63}", table) + ): + die("Gitea salvage schema table is outside the review registry") + try: + object_rows = connection.execute( + "SELECT type,sql FROM sqlite_master WHERE name=? ORDER BY type", + (table,), + ).fetchall() + table_list_rows = ( + connection.execute(f'PRAGMA table_list("{table}")').fetchall() + if table_list_supported + else [] + ) + except sqlite3.DatabaseError: + die("Gitea salvage schema-only inventory failed") + object_type = None + object_sql = None + if len(object_rows) == 1 and len(object_rows[0]) == 2: + object_type = object_rows[0][0] + object_sql = object_rows[0][1] + ordinary_sql = ( + isinstance(object_sql, str) + and re.match( + r"\A\s*CREATE\s+TABLE(?:\s+IF\s+NOT\s+EXISTS)?\s+", + object_sql, + flags=re.IGNORECASE, + ) + is not None + ) + virtual_sql = ( + isinstance(object_sql, str) + and re.match( + r"\A\s*CREATE\s+VIRTUAL\s+TABLE(?:\s+IF\s+NOT\s+EXISTS)?\s+", + object_sql, + flags=re.IGNORECASE, + ) + is not None + ) + safe_sqlite_schema_object = object_type == "table" and ordinary_sql + safe_table_list_object = ( + len(table_list_rows) == 1 + and len(table_list_rows[0]) == 6 + and tuple(table_list_rows[0][0:3]) == ("main", table, "table") + and isinstance(table_list_rows[0][3], int) + and table_list_rows[0][3] >= 0 + and table_list_rows[0][4] in (0, 1) + and table_list_rows[0][5] in (0, 1) + ) + safe_object = safe_sqlite_schema_object and ( + safe_table_list_object if table_list_supported else True + ) + rows = [] + if safe_object: + try: + rows = connection.execute(f'PRAGMA table_xinfo("{table}")').fetchall() + except sqlite3.DatabaseError: + die("Gitea salvage schema-only inventory failed") + columns = [] + seen = set() + for row in rows: + if len(row) != 7: + die("Gitea salvage schema-only inventory shape mismatch") + cid = row[0] + name = str(row[1]) + declared_type = str(row[2] or "") + not_null = row[3] + primary_key = row[5] + hidden = row[6] + if ( + not isinstance(cid, int) + or cid < 0 + or not re.fullmatch(r"[A-Za-z_][A-Za-z0-9_]{0,127}", name) + or name in seen + or len(declared_type) > 64 + or re.fullmatch(r"[A-Za-z0-9_(), ]*", declared_type) is None + or not_null not in (0, 1) + or not isinstance(primary_key, int) + or primary_key < 0 + or hidden not in (0, 1, 2, 3) + ): + die("Gitea salvage schema-only inventory is unsafe") + seen.add(name) + columns.append( + { + "cid": cid, + "hidden": int(hidden), + "name": name, + "not_null": int(not_null), + "primary_key": int(primary_key), + "type": declared_type, + } + ) + exists = bool(object_rows or table_list_rows) + sqlite_schema_kind = None + sqlite_schema_virtual = None + if object_type is not None: + sqlite_schema_kind = object_type + sqlite_schema_virtual = 1 if virtual_sql else 0 + table_list = None + if len(table_list_rows) == 1 and len(table_list_rows[0]) == 6: + table_list = { + "columns": table_list_rows[0][3], + "schema": table_list_rows[0][0], + "strict": table_list_rows[0][5], + "type": table_list_rows[0][2], + "without_rowid": table_list_rows[0][4], + } + if ( + table_list_supported + and safe_object + and table_list["columns"] != len(columns) + ): + die("Gitea salvage schema-only column count mismatch") + return { + "columns": columns, + "exists": exists, + "ordinary_main_table": safe_object, + "object_kind_attestation": ( + "sqlite-master-and-table-list" + if table_list_supported + else "sqlite-master-nonvirtual-table" + ), + "sqlite_schema_kind": sqlite_schema_kind, + "sqlite_schema_virtual": sqlite_schema_virtual, + "table": table, + "table_list": table_list, + } + + +def gitea_salvage_unsupported_schema_catalog(connection): + sqlite_version, sqlite_version_tuple = gitea_salvage_sqlite_version(connection) + table_list_supported = sqlite_version_tuple >= (3, 37, 0) + catalog = { + "database_sha256": GITEA_SALVAGE_SNAPSHOT_DATABASE_SHA256, + "schema": "nodedc.gitea.salvage-unsupported-schema/v1", + "sqlite_version": sqlite_version, + "table_list_supported": table_list_supported, + "tables": [ + gitea_salvage_table_schema( + connection, + table, + table_list_supported, + ) + for table in GITEA_SALVAGE_UNSUPPORTED_SCHEMA_TABLES + ], + } + canonical = canonical_gitea_salvage_evidence( + catalog, + "unsupported-state schema catalog", + ) + return {"catalog": catalog, **canonical} + + +def gitea_salvage_schema_columns_by_name(schema): + return {column["name"]: column for column in schema["columns"]} + + +def gitea_salvage_declared_type_has_integer_affinity(declared_type): + return "INT" in declared_type.upper() + + +def gitea_salvage_declared_type_has_text_affinity(declared_type): + normalized = declared_type.upper() + return any(token in normalized for token in ("CHAR", "CLOB", "TEXT")) + + +def gitea_salvage_parse_semantic_topics(value): + if not isinstance(value, str): + die("Gitea salvage repository topics value is not text") + try: + encoded = value.encode("utf-8") + except UnicodeEncodeError: + die("Gitea salvage repository topics value is not valid UTF-8") + if len(encoded) > 16 * 1024: + die("Gitea salvage repository topics value exceeds the byte limit") + # Gitea persists a nil topics slice as the canonical JSON literal `null`. + # The incident snapshot has exactly this four-byte text representation for + # all 45 kept repositories. It is semantic empty state, not a topic value. + # Accept only the exact canonical literal; whitespace, case variants, + # quoted strings and SQL NULL remain rejected below/by the caller. + if value == "null": + return () + try: + parsed = json.loads(value) + except (json.JSONDecodeError, RecursionError): + die("Gitea salvage repository topics JSON is invalid") + if not isinstance(parsed, list) or len(parsed) > 256: + die("Gitea salvage repository topics JSON is not a bounded array") + try: + canonical = json.dumps( + parsed, + ensure_ascii=True, + separators=(",", ":"), + allow_nan=False, + ) + except (TypeError, ValueError): + die("Gitea salvage repository topics JSON is not canonical") + if value != canonical: + die("Gitea salvage repository topics JSON is not canonical") + topics = [] + seen = set() + for topic in parsed: + if not isinstance(topic, str): + die("Gitea salvage repository topic is not text") + try: + topic_bytes = topic.encode("utf-8") + except UnicodeEncodeError: + die("Gitea salvage repository topic is not valid UTF-8") + if ( + not topic + or len(topic_bytes) > 35 + or topic != topic.lower() + or re.fullmatch(r"[a-z0-9][-.a-z0-9]*", topic) is None + ): + die("Gitea salvage repository topic is unsafe") + if topic in seen: + die("Gitea salvage repository topics contain a duplicate") + seen.add(topic) + topics.append(topic) + if topics != sorted(topics): + die("Gitea salvage repository topics are not sorted") + return tuple(topics) + + +def gitea_salvage_semantic_topics_inventory(connection, kept_repositories): + normalized = [] + seen = set() + for record in kept_repositories: + try: + repo_id = int(record["repo_id"]) + except (KeyError, TypeError, ValueError): + die("Gitea salvage semantic-topics repository identity is invalid") + if repo_id <= 0 or repo_id in seen: + die("Gitea salvage semantic-topics repository identity is invalid") + seen.add(repo_id) + normalized.append(repo_id) + normalized.sort() + if not normalized: + die("Gitea salvage semantic-topics repository set is empty") + placeholders = ",".join("?" for _ in normalized) + try: + rows = connection.execute( + f"SELECT id,topics,typeof(topics) FROM repository " + f"WHERE id IN ({placeholders}) ORDER BY id", + normalized, + ).fetchall() + except sqlite3.DatabaseError: + die("Gitea salvage semantic-topics query failed") + if len(rows) != len(normalized): + die("Gitea salvage semantic-topics repository set is incomplete") + repositories = [] + total_topics = 0 + material_repositories = 0 + serialized_arrays = 0 + serialized_nulls = 0 + for expected_repo_id, row in zip(normalized, rows): + repo_id, value, value_type = row + if ( + repo_id != expected_repo_id + or value_type != "text" + ): + die("Gitea salvage semantic-topics row is invalid") + topics = gitea_salvage_parse_semantic_topics(value) + if value == "null": + encoding = "json-null" + serialized_nulls += 1 + else: + encoding = "json-array" + serialized_arrays += 1 + count = len(topics) + total_topics += count + material_repositories += int(bool(count)) + repositories.append( + { + "encoding": encoding, + "material": bool(count), + "old_repo_id": repo_id, + "topic_count": count, + } + ) + evidence = { + "database_sha256": GITEA_SALVAGE_SNAPSHOT_DATABASE_SHA256, + "decision_manifest_sha256": GITEA_SALVAGE_DECISION_MANIFEST_SHA256, + "material_repositories": material_repositories, + "repositories": repositories, + "schema": "nodedc.gitea.salvage-semantic-topics/v2", + "serialized_arrays": serialized_arrays, + "serialized_nulls": serialized_nulls, + "snapshot_uuid": GITEA_SALVAGE_SNAPSHOT_UUID, + "topics": total_topics, + } + canonical = canonical_gitea_salvage_evidence( + evidence, + "semantic topics evidence", + ) + return {"evidence": evidence, **canonical} + + +def bind_gitea_salvage_decisions_to_snapshot(decisions): + connection = gitea_salvage_sqlite_connection(GITEA_SALVAGE_SNAPSHOT_DATABASE) + try: + quick = [row[0] for row in connection.execute("PRAGMA quick_check").fetchall()] + if quick != ["ok"]: + die("Gitea salvage snapshot SQLite quick_check failed") + user_columns = gitea_salvage_table_columns(connection, "user") + repository_columns = gitea_salvage_table_columns(connection, "repository") + required_user_columns = { + "id", + "name", + "lower_name", + "created_unix", + "updated_unix", + "is_admin", + "is_active", + "is_restricted", + } + required_repository_columns = { + "id", + "owner_id", + "owner_name", + "name", + "lower_name", + "created_unix", + "updated_unix", + "is_private", + "is_archived", + "is_mirror", + "is_fork", + } + if ( + not required_user_columns.issubset(user_columns) + or not required_repository_columns.issubset(repository_columns) + ): + die("Gitea salvage snapshot identity schema mismatch") + user_rows = connection.execute( + "SELECT id,name,lower_name,created_unix,updated_unix,is_admin," + "is_active,is_restricted FROM user ORDER BY id" + ).fetchall() + repository_rows = connection.execute( + "SELECT id,owner_id,owner_name,name,lower_name,created_unix," + "updated_unix,is_private,is_archived,is_mirror,is_fork " + "FROM repository ORDER BY id" + ).fetchall() + if len(user_rows) != 972 or len(repository_rows) != 2058: + die("Gitea salvage snapshot identity row count mismatch") + repositories_per_user = {} + lower_owner_by_id = {} + for row in user_rows: + lower_owner_by_id[int(row["id"])] = str(row["lower_name"]) + for row in repository_rows: + owner_id = int(row["owner_id"]) + repositories_per_user[owner_id] = repositories_per_user.get(owner_id, 0) + 1 + decision_users = {int(row["user_id"]): row for row in decisions["users"]} + decision_repositories = { + int(row["repo_id"]): row for row in decisions["repositories"] + } + if ( + set(decision_users) != {int(row["id"]) for row in user_rows} + or set(decision_repositories) + != {int(row["id"]) for row in repository_rows} + ): + die("Gitea salvage decision bundle is not a full database partition") + for row in user_rows: + user_id = int(row["id"]) + fingerprint = { + "user_id": user_id, + "owner": str(row["name"]), + "lower_owner": str(row["lower_name"]), + "created_raw": str(row["created_unix"]), + "updated_raw": str(row["updated_unix"]), + "repository_count": repositories_per_user.get(user_id, 0), + "is_admin": int(row["is_admin"]), + "is_active": int(row["is_active"]), + "is_restricted": int(row["is_restricted"]), + } + if ( + canonical_gitea_salvage_record_sha256(fingerprint) + != decision_users[user_id]["record_sha256"] + ): + die(f"Gitea salvage user fingerprint mismatch: {user_id}") + for row in repository_rows: + repo_id = int(row["id"]) + owner_id = int(row["owner_id"]) + decision = decision_repositories[repo_id] + fingerprint = { + "repo_id": repo_id, + "owner_id": owner_id, + "owner": str(row["owner_name"]), + "slug": str(row["name"]), + "lower_owner": lower_owner_by_id.get(owner_id), + "lower_slug": str(row["lower_name"]), + "repo_relative_path": decision["repo_relative_path"], + "created_raw": str(row["created_unix"]), + "updated_raw": str(row["updated_unix"]), + "is_private": int(row["is_private"]), + "is_archived": int(row["is_archived"]), + "is_mirror": int(row["is_mirror"]), + "is_fork": int(row["is_fork"]), + } + if ( + canonical_gitea_salvage_record_sha256(fingerprint) + != decision["record_sha256"] + ): + die(f"Gitea salvage repository fingerprint mismatch: {repo_id}") + if decision["decision"] == "KEEP" and ( + int(row["is_private"]) != (1 if owner_id == 1 else 0) + or int(row["is_archived"]) != 0 + or int(row["is_mirror"]) != 0 + or int(row["is_fork"]) != 0 + ): + die(f"Gitea salvage kept repository state mismatch: {repo_id}") + unsupported = gitea_salvage_unsupported_state_inventory( + connection, + decisions["kept_repositories"], + ) + topics = gitea_salvage_semantic_topics_inventory( + connection, + decisions["kept_repositories"], + ) + closure = gitea_salvage_incident_closure_inventory( + connection, + decisions, + unsupported, + topics, + ) + return { + "closure": closure, + "quick_check": "ok", + "users": len(user_rows), + "repositories": len(repository_rows), + "topics": topics, + "unsupported": unsupported, + } + finally: + connection.close() + + +def gitea_salvage_unsupported_state_inventory(connection, kept_repositories): + normalized = [] + seen_repo_ids = set() + for row in kept_repositories: + if not isinstance(row, dict): + die("Gitea salvage kept-repository evidence input is invalid") + try: + repo_id = int(row["repo_id"]) + owner = str(row["owner"]) + slug = str(row["slug"]) + except (KeyError, TypeError, ValueError): + die("Gitea salvage kept-repository evidence input is invalid") + if ( + repo_id <= 0 + or repo_id in seen_repo_ids + or not re.fullmatch(r"[A-Za-z0-9_.-]{1,255}", owner) + or not re.fullmatch(r"[A-Za-z0-9_.-]{1,255}", slug) + ): + die("Gitea salvage kept-repository evidence identity is invalid") + seen_repo_ids.add(repo_id) + normalized.append({"old_repo_id": repo_id, "owner": owner, "slug": slug}) + normalized.sort(key=lambda item: item["old_repo_id"]) + if not normalized: + die("Gitea salvage kept-repository evidence set is empty") + + kept_repo_ids = tuple(item["old_repo_id"] for item in normalized) + placeholders = ",".join("?" for _ in kept_repo_ids) + schema_evidence = gitea_salvage_unsupported_schema_catalog(connection) + schemas = { + schema["table"]: schema + for schema in schema_evidence["catalog"]["tables"] + } + schema_columns = { + table: gitea_salvage_schema_columns_by_name(schema) + for table, schema in schemas.items() + } + missing_schema = set() + mismatched_schema = set() + anomalies = [] + per_repository = [] + by_repo_id = {} + count_labels = tuple( + label for label, _table, _column in GITEA_SALVAGE_UNSUPPORTED_REPOSITORY_TABLES + ) + for identity in normalized: + record = { + **identity, + "attachment_links": { + label: {"logical_bytes": None, "rows": None} + for label in ("comment", "issue", "multi_link", "release", "unlinked") + }, + "attachments": { + "association_rows": None, + "logical_bytes": None, + }, + "counts": {label: None for label in count_labels}, + "lfs": { + "association_logical_bytes": None, + "association_rows": None, + "distinct_oids": None, + }, + "repo_unit_types": None, + "repository_hints": { + column: None + for column in GITEA_SALVAGE_UNSUPPORTED_REPOSITORY_NUMERIC_HINTS + }, + "repository_metadata_present": { + column: None + for column in GITEA_SALVAGE_UNSUPPORTED_REPOSITORY_TEXT_METADATA + }, + } + per_repository.append(record) + by_repo_id[identity["old_repo_id"]] = record + + def require_column(table, column, affinity=None): + schema = schemas[table] + columns = schema_columns[table] + if schema["exists"] and not schema["ordinary_main_table"]: + mismatched_schema.add(f"{table}:ordinary-main-table") + return False + if not schema["exists"] or column not in columns: + missing_schema.add(f"{table}.{column}") + return False + declared_type = columns[column]["type"] + if affinity == "integer" and not gitea_salvage_declared_type_has_integer_affinity( + declared_type + ): + mismatched_schema.add(f"{table}.{column}:integer-affinity") + return False + if affinity == "text" and not gitea_salvage_declared_type_has_text_affinity( + declared_type + ): + mismatched_schema.add(f"{table}.{column}:text-affinity") + return False + return True + + metric_available = {} + for label, table, column in GITEA_SALVAGE_UNSUPPORTED_REPOSITORY_TABLES: + available = require_column(table, column, "integer") + metric_available[label] = available + if not available: + continue + try: + invalid_repo_ids = int( + connection.execute( + f'SELECT COUNT(*) FROM "{table}" ' + f'WHERE "{column}" IN ({placeholders}) ' + f'AND typeof("{column}") != \'integer\'', + kept_repo_ids, + ).fetchone()[0] + ) + grouped = connection.execute( + f'SELECT "{column}",COUNT(*) FROM "{table}" ' + f'WHERE "{column}" IN ({placeholders}) GROUP BY "{column}"', + kept_repo_ids, + ).fetchall() + except sqlite3.DatabaseError: + die("Gitea salvage direct-relation evidence query failed") + if invalid_repo_ids: + anomalies.append(f"{table}.{column}:invalid_repo_ids={invalid_repo_ids}") + metric_available[label] = False + continue + for repo_id, count in grouped: + if not isinstance(repo_id, int) or repo_id not in by_repo_id: + die("Gitea salvage direct-relation grouping escaped the keep set") + by_repo_id[repo_id]["counts"][label] = int(count) + for record in per_repository: + if record["counts"][label] is None: + record["counts"][label] = 0 + + repo_unit_requirements = [ + require_column("repo_unit", column, "integer") + for column in ("repo_id", "type") + ] + repo_unit_ready = all(repo_unit_requirements) + if repo_unit_ready: + try: + unit_rows = connection.execute( + f"SELECT repo_id,type,COUNT(*) FROM repo_unit " + f"WHERE repo_id IN ({placeholders}) GROUP BY repo_id,type", + kept_repo_ids, + ).fetchall() + except sqlite3.DatabaseError: + die("Gitea salvage repo-unit evidence query failed") + unit_types = {repo_id: {} for repo_id in kept_repo_ids} + for repo_id, unit_type, count in unit_rows: + if ( + not isinstance(repo_id, int) + or repo_id not in by_repo_id + or not isinstance(unit_type, int) + or unit_type < 0 + ): + anomalies.append("repo_unit.repo_id/type:invalid_group") + repo_unit_ready = False + break + unit_types[repo_id][str(unit_type)] = int(count) + if repo_unit_ready: + for repo_id, values in unit_types.items(): + by_repo_id[repo_id]["repo_unit_types"] = values + + repository_id_ready = require_column("repository", "id", "integer") + for column in GITEA_SALVAGE_UNSUPPORTED_REPOSITORY_TEXT_METADATA: + column_ready = require_column("repository", column, "text") + if not repository_id_ready or not column_ready: + continue + try: + invalid = int( + connection.execute( + f'SELECT COUNT(*) FROM repository WHERE id IN ({placeholders}) ' + f'AND typeof("{column}") NOT IN (\'null\',\'text\')', + kept_repo_ids, + ).fetchone()[0] + ) + rows = connection.execute( + f'SELECT id,CASE WHEN "{column}" IS NULL OR "{column}" = \'\' ' + f'THEN 0 ELSE 1 END FROM repository WHERE id IN ({placeholders})', + kept_repo_ids, + ).fetchall() + except sqlite3.DatabaseError: + die("Gitea salvage repository metadata evidence query failed") + if invalid: + anomalies.append(f"repository.{column}:invalid_values={invalid}") + continue + for repo_id, present in rows: + if repo_id not in by_repo_id or present not in (0, 1): + die("Gitea salvage repository metadata grouping is invalid") + by_repo_id[repo_id]["repository_metadata_present"][column] = bool(present) + + for column in GITEA_SALVAGE_UNSUPPORTED_REPOSITORY_NUMERIC_HINTS: + column_ready = require_column("repository", column, "integer") + if not repository_id_ready or not column_ready: + continue + try: + rows = connection.execute( + f'SELECT id,"{column}",typeof("{column}") FROM repository ' + f'WHERE id IN ({placeholders})', + kept_repo_ids, + ).fetchall() + except sqlite3.DatabaseError: + die("Gitea salvage repository hint evidence query failed") + valid = True + for repo_id, value, value_type in rows: + if ( + repo_id not in by_repo_id + or value_type not in ("integer", "null") + or (value is not None and (not isinstance(value, int) or value < 0)) + ): + anomalies.append(f"repository.{column}:invalid_value") + valid = False + break + if valid: + for repo_id, value, _value_type in rows: + by_repo_id[repo_id]["repository_hints"][column] = ( + 0 if value is None else int(value) + ) + + lfs_requirements = [ + require_column("lfs_meta_object", "repository_id", "integer"), + require_column("lfs_meta_object", "oid", "text"), + require_column("lfs_meta_object", "size", "integer"), + ] + lfs_required = all(lfs_requirements) and repository_id_ready + lfs_aggregate = { + "association_logical_bytes": None, + "association_rows": None, + "distinct_oids": None, + "invalid_oid_rows": None, + "invalid_related_repository_rows": None, + "invalid_related_size_rows": None, + "invalid_size_rows": None, + "non_kept_shared_logical_bytes": None, + "non_kept_shared_oids": None, + "physical_presence": "not-inventoried", + "orphan_related_repository_rows": None, + "size_conflict_oids": None, + "unique_logical_bytes": None, + } + if lfs_required: + try: + invalid_oid_rows = int( + connection.execute( + f"SELECT COUNT(*) FROM lfs_meta_object " + f"WHERE repository_id IN ({placeholders}) AND (" + "typeof(oid) != 'text' OR length(oid) != 64 OR " + "oid GLOB '*[^0-9a-f]*')", + kept_repo_ids, + ).fetchone()[0] + ) + invalid_size_rows = int( + connection.execute( + f"SELECT COUNT(*) FROM lfs_meta_object " + f"WHERE repository_id IN ({placeholders}) AND (" + "typeof(size) != 'integer' OR size < 0 OR size > ?)", + (*kept_repo_ids, GITEA_SALVAGE_UNSUPPORTED_SIZE_PER_ROW_MAX_BYTES), + ).fetchone()[0] + ) + invalid_related_size_rows = int( + connection.execute( + f"SELECT COUNT(*) FROM lfs_meta_object WHERE oid IN (" + f"SELECT oid FROM lfs_meta_object " + f"WHERE repository_id IN ({placeholders})) AND (" + "typeof(size) != 'integer' OR size < 0 OR size > ?)", + (*kept_repo_ids, GITEA_SALVAGE_UNSUPPORTED_SIZE_PER_ROW_MAX_BYTES), + ).fetchone()[0] + ) + invalid_related_repository_rows = int( + connection.execute( + f"SELECT COUNT(*) FROM lfs_meta_object WHERE oid IN (" + f"SELECT oid FROM lfs_meta_object " + f"WHERE repository_id IN ({placeholders})) AND (" + "typeof(repository_id) != 'integer' OR repository_id <= 0)", + kept_repo_ids, + ).fetchone()[0] + ) + orphan_related_repository_rows = int( + connection.execute( + f"SELECT COUNT(*) FROM lfs_meta_object AS related WHERE oid IN (" + f"SELECT oid FROM lfs_meta_object " + f"WHERE repository_id IN ({placeholders})) " + "AND typeof(related.repository_id) = 'integer' " + "AND related.repository_id > 0 AND NOT EXISTS (" + "SELECT 1 FROM repository " + "WHERE repository.id=related.repository_id)", + kept_repo_ids, + ).fetchone()[0] + ) + except sqlite3.DatabaseError: + die("Gitea salvage LFS validation query failed") + lfs_aggregate["invalid_oid_rows"] = invalid_oid_rows + lfs_aggregate["invalid_related_repository_rows"] = ( + invalid_related_repository_rows + ) + lfs_aggregate["invalid_related_size_rows"] = invalid_related_size_rows + lfs_aggregate["invalid_size_rows"] = invalid_size_rows + lfs_aggregate["orphan_related_repository_rows"] = ( + orphan_related_repository_rows + ) + if invalid_oid_rows: + anomalies.append(f"lfs_meta_object.oid:invalid_rows={invalid_oid_rows}") + if invalid_size_rows: + anomalies.append(f"lfs_meta_object.size:invalid_rows={invalid_size_rows}") + if invalid_related_size_rows: + anomalies.append( + "lfs_meta_object.size:" + f"invalid_related_rows={invalid_related_size_rows}" + ) + if invalid_related_repository_rows: + anomalies.append( + "lfs_meta_object.repository_id:" + f"invalid_related_rows={invalid_related_repository_rows}" + ) + if orphan_related_repository_rows: + anomalies.append( + "lfs_meta_object.repository_id:" + f"orphan_related_rows={orphan_related_repository_rows}" + ) + if not any( + ( + invalid_oid_rows, + invalid_size_rows, + invalid_related_size_rows, + invalid_related_repository_rows, + orphan_related_repository_rows, + ) + ): + try: + grouped = connection.execute( + f"SELECT repository_id,COUNT(*),COUNT(DISTINCT oid)," + f"COALESCE(SUM(size),0) FROM lfs_meta_object " + f"WHERE repository_id IN ({placeholders}) GROUP BY repository_id", + kept_repo_ids, + ).fetchall() + unique = connection.execute( + f"SELECT COUNT(*),COALESCE(SUM(max_size),0) FROM (" + f"SELECT oid,MAX(size) AS max_size FROM lfs_meta_object " + f"WHERE repository_id IN ({placeholders}) GROUP BY oid)", + kept_repo_ids, + ).fetchone() + conflicts = int( + connection.execute( + f"SELECT COUNT(*) FROM (SELECT oid FROM lfs_meta_object " + f"WHERE oid IN (SELECT oid FROM lfs_meta_object " + f"WHERE repository_id IN ({placeholders})) " + f"GROUP BY oid HAVING MIN(size) != MAX(size))", + kept_repo_ids, + ).fetchone()[0] + ) + shared = connection.execute( + f"SELECT COUNT(*),COALESCE(SUM(kept_size),0) FROM (" + f"SELECT kept.oid,MAX(kept.size) AS kept_size " + f"FROM lfs_meta_object AS kept " + f"WHERE kept.repository_id IN ({placeholders}) AND EXISTS (" + f"SELECT 1 FROM lfs_meta_object AS other " + f"WHERE other.oid=kept.oid AND other.repository_id NOT IN ({placeholders})" + f") GROUP BY kept.oid)", + (*kept_repo_ids, *kept_repo_ids), + ).fetchone() + except sqlite3.DatabaseError: + die("Gitea salvage LFS aggregate query failed") + association_rows = 0 + association_bytes = 0 + distinct_oids_per_repo = 0 + for repo_id, rows, distinct_oids, logical_bytes in grouped: + if repo_id not in by_repo_id: + die("Gitea salvage LFS grouping escaped the keep set") + values = (rows, distinct_oids, logical_bytes) + if any(not isinstance(value, int) or value < 0 for value in values): + die("Gitea salvage LFS aggregate is invalid") + if logical_bytes > GITEA_SALVAGE_UNSUPPORTED_SIZE_TOTAL_MAX_BYTES: + die("Gitea salvage LFS aggregate exceeds the byte limit") + by_repo_id[repo_id]["lfs"] = { + "association_logical_bytes": logical_bytes, + "association_rows": rows, + "distinct_oids": distinct_oids, + } + association_rows += rows + association_bytes += logical_bytes + distinct_oids_per_repo += distinct_oids + for record in per_repository: + if record["lfs"]["association_rows"] is None: + record["lfs"] = { + "association_logical_bytes": 0, + "association_rows": 0, + "distinct_oids": 0, + } + unique_oids, unique_bytes = unique + shared_oids, shared_bytes = shared + for value in ( + association_rows, + association_bytes, + distinct_oids_per_repo, + unique_oids, + unique_bytes, + shared_oids, + shared_bytes, + conflicts, + ): + if not isinstance(value, int) or value < 0: + die("Gitea salvage LFS aggregate is invalid") + if max(association_bytes, unique_bytes, shared_bytes) > ( + GITEA_SALVAGE_UNSUPPORTED_SIZE_TOTAL_MAX_BYTES + ): + die("Gitea salvage LFS aggregate exceeds the byte limit") + lfs_aggregate.update( + { + "association_logical_bytes": association_bytes, + "association_rows": association_rows, + "distinct_oids": unique_oids, + "non_kept_shared_logical_bytes": shared_bytes, + "non_kept_shared_oids": shared_oids, + "size_conflict_oids": conflicts, + "unique_logical_bytes": unique_bytes, + } + ) + if conflicts: + anomalies.append(f"lfs_meta_object.size:conflict_oids={conflicts}") + + attachment_requirements = [ + require_column("attachment", column, "integer") + for column in ("id", "repo_id", "issue_id", "release_id", "comment_id", "size") + ] + attachment_required = all(attachment_requirements) + attachment_aggregate = { + "association_rows": None, + "invalid_association_rows": None, + "invalid_size_rows": None, + "link_splits": { + label: {"logical_bytes": None, "rows": None} + for label in ("comment", "issue", "multi_link", "release", "unlinked") + }, + "logical_bytes": None, + "physical_presence": "not-inventoried", + } + if attachment_required: + try: + invalid_associations = int( + connection.execute( + f"SELECT COUNT(*) FROM attachment WHERE repo_id IN ({placeholders}) " + "AND (typeof(repo_id) != 'integer' OR " + "typeof(issue_id) NOT IN ('integer','null') OR " + "typeof(release_id) NOT IN ('integer','null') OR " + "typeof(comment_id) NOT IN ('integer','null') OR " + "COALESCE(issue_id,0) < 0 OR COALESCE(release_id,0) < 0 OR " + "COALESCE(comment_id,0) < 0)", + kept_repo_ids, + ).fetchone()[0] + ) + invalid_sizes = int( + connection.execute( + f"SELECT COUNT(*) FROM attachment WHERE repo_id IN ({placeholders}) " + "AND (typeof(size) != 'integer' OR size < 0 OR size > ?)", + (*kept_repo_ids, GITEA_SALVAGE_UNSUPPORTED_SIZE_PER_ROW_MAX_BYTES), + ).fetchone()[0] + ) + except sqlite3.DatabaseError: + die("Gitea salvage attachment validation query failed") + attachment_aggregate["invalid_association_rows"] = invalid_associations + attachment_aggregate["invalid_size_rows"] = invalid_sizes + if invalid_associations: + anomalies.append(f"attachment.association:invalid_rows={invalid_associations}") + if invalid_sizes: + anomalies.append(f"attachment.size:invalid_rows={invalid_sizes}") + if not invalid_associations and not invalid_sizes: + try: + grouped = connection.execute( + f"SELECT repo_id,COUNT(*),COALESCE(SUM(size),0) " + f"FROM attachment WHERE repo_id IN ({placeholders}) GROUP BY repo_id", + kept_repo_ids, + ).fetchall() + except sqlite3.DatabaseError: + die("Gitea salvage attachment aggregate query failed") + association_rows = 0 + logical_bytes = 0 + for repo_id, rows, size_bytes in grouped: + if ( + repo_id not in by_repo_id + or not isinstance(rows, int) + or rows < 0 + or not isinstance(size_bytes, int) + or size_bytes < 0 + or size_bytes > GITEA_SALVAGE_UNSUPPORTED_SIZE_TOTAL_MAX_BYTES + ): + die("Gitea salvage attachment aggregate is invalid") + by_repo_id[repo_id]["attachments"] = { + "association_rows": rows, + "logical_bytes": size_bytes, + } + association_rows += rows + logical_bytes += size_bytes + for record in per_repository: + if record["attachments"]["association_rows"] is None: + record["attachments"] = { + "association_rows": 0, + "logical_bytes": 0, + } + if logical_bytes > GITEA_SALVAGE_UNSUPPORTED_SIZE_TOTAL_MAX_BYTES: + die("Gitea salvage attachment aggregate exceeds the byte limit") + attachment_aggregate["association_rows"] = association_rows + attachment_aggregate["logical_bytes"] = logical_bytes + link_predicates = { + "comment": "COALESCE(comment_id,0) > 0", + "issue": "COALESCE(issue_id,0) > 0", + "multi_link": "((COALESCE(issue_id,0) > 0) + (COALESCE(release_id,0) > 0) + (COALESCE(comment_id,0) > 0)) > 1", + "release": "COALESCE(release_id,0) > 0", + "unlinked": "COALESCE(issue_id,0) = 0 AND COALESCE(release_id,0) = 0 AND COALESCE(comment_id,0) = 0", + } + for label, predicate in link_predicates.items(): + try: + split_rows = connection.execute( + f"SELECT repo_id,COUNT(*),COALESCE(SUM(size),0) " + f"FROM attachment WHERE repo_id IN ({placeholders}) " + f"AND ({predicate}) GROUP BY repo_id", + kept_repo_ids, + ).fetchall() + except sqlite3.DatabaseError: + die("Gitea salvage attachment link-split query failed") + total_rows = 0 + total_bytes = 0 + for repo_id, rows, size_bytes in split_rows: + if ( + repo_id not in by_repo_id + or not isinstance(rows, int) + or rows < 0 + or not isinstance(size_bytes, int) + or size_bytes < 0 + ): + die("Gitea salvage attachment link split is invalid") + by_repo_id[repo_id]["attachment_links"][label] = { + "logical_bytes": size_bytes, + "rows": rows, + } + total_rows += rows + total_bytes += size_bytes + for record in per_repository: + if record["attachment_links"][label]["rows"] is None: + record["attachment_links"][label] = { + "logical_bytes": 0, + "rows": 0, + } + if total_bytes > GITEA_SALVAGE_UNSUPPORTED_SIZE_TOTAL_MAX_BYTES: + die("Gitea salvage attachment link split exceeds the byte limit") + attachment_aggregate["link_splits"][label] = { + "logical_bytes": total_bytes, + "rows": total_rows, + } + + direct_totals = { + label: ( + sum(record["counts"][label] for record in per_repository) + if metric_available[label] + and all(record["counts"][label] is not None for record in per_repository) + else None + ) + for label in count_labels + } + metadata_totals = { + column: ( + sum( + 1 + for record in per_repository + if record["repository_metadata_present"][column] is True + ) + if all( + record["repository_metadata_present"][column] is not None + for record in per_repository + ) + else None + ) + for column in GITEA_SALVAGE_UNSUPPORTED_REPOSITORY_TEXT_METADATA + } + nonzero_categories = sorted( + [label for label, value in direct_totals.items() if isinstance(value, int) and value] + + [ + f"repository_metadata:{column}" + for column, value in metadata_totals.items() + if isinstance(value, int) and value + ] + + ( + ["repo_units"] + if repo_unit_ready + and any(record["repo_unit_types"] for record in per_repository) + else [] + ) + + [ + f"repository_hint:{column}" + for column in GITEA_SALVAGE_UNSUPPORTED_REPOSITORY_NUMERIC_HINTS + if any( + isinstance(record["repository_hints"][column], int) + and record["repository_hints"][column] > 0 + for record in per_repository + ) + ] + ) + schema_review_matches = ( + GITEA_SALVAGE_EXPECTED_UNSUPPORTED_SCHEMA_SHA256 is not None + and schema_evidence["sha256"] + == GITEA_SALVAGE_EXPECTED_UNSUPPORTED_SCHEMA_SHA256 + ) + coverage_blockers = [ + "issue-pull-dependent-closure-unreviewed", + "package-blob-closure-unreviewed", + "actions-artifact-closure-unreviewed", + "lfs-physical-object-inventory-unreviewed", + "attachment-physical-object-inventory-unreviewed", + ] + if not schema_review_matches: + coverage_blockers.append("unsupported-schema-catalog-unreviewed") + if not schema_evidence["catalog"]["table_list_supported"]: + coverage_blockers.append("sqlite-table-list-object-kind-unavailable") + report = { + "aggregates": { + "attachments": attachment_aggregate, + "direct_relation_counts": direct_totals, + "lfs": lfs_aggregate, + "repository_metadata_presence": metadata_totals, + }, + "anomalies": sorted(set(anomalies)), + "coverage": { + "attachments": "database-metadata-only-no-physical-presence-claim", + "direct_repository_relations": "counted-per-kept-repository", + "lfs": "database-associations-and-logical-bytes-no-physical-presence-claim", + "repository_hints": "denormalized-non-authoritative", + "schema_only_unreviewed_tables": [ + "action_artifact", + "action_run_index", + "action_run_job", + "action_task", + "comment", + "issue_assignees", + "issue_content_history", + "issue_dependency", + "issue_label", + "issue_user", + "issue_watch", + "notification", + "package_blob", + "package_file", + "package_property", + "package_version", + "project", + "project_board", + "project_issue", + "pull_auto_merge", + "reaction", + "review", + "review_state", + "stopwatch", + "tracked_time", + ], + }, + "coverage_blockers": sorted(coverage_blockers), + "database_sha256": GITEA_SALVAGE_SNAPSHOT_DATABASE_SHA256, + "decision_manifest_sha256": GITEA_SALVAGE_DECISION_MANIFEST_SHA256, + "direct_relation_contract": sorted( + ( + { + "label": label, + "repository_column": column, + "table": table, + } + for label, table, column in GITEA_SALVAGE_UNSUPPORTED_REPOSITORY_TABLES + ), + key=lambda item: item["label"], + ), + "kept_repository_ids": list(kept_repo_ids), + "material_present": bool(nonzero_categories), + "nonzero_categories": nonzero_categories, + "per_repository": per_repository, + "schema": "nodedc.gitea.salvage-unsupported-state/v2", + "schema_catalog": schema_evidence["catalog"], + "schema_catalog_sha256": schema_evidence["sha256"], + "schema_mismatch": sorted(mismatched_schema), + "schema_missing": sorted(missing_schema), + "schema_review": { + "expected_sha256": GITEA_SALVAGE_EXPECTED_UNSUPPORTED_SCHEMA_SHA256, + "matches": schema_review_matches, + }, + "snapshot_uuid": GITEA_SALVAGE_SNAPSHOT_UUID, + } + canonical = canonical_gitea_salvage_evidence( + report, + "unsupported-state report", + ) + return {"report": report, **canonical} + + +def gitea_salvage_closure_actor_class(user_id, user_classes, label): + if ( + not isinstance(user_id, int) + or isinstance(user_id, bool) + or user_id < 0 + ): + die(f"Gitea salvage closure {label} actor identity is invalid") + if user_id == 0: + return "system-or-external" + actor_class = user_classes.get(user_id) + if actor_class is None: + die(f"Gitea salvage closure {label} actor is outside user decisions") + return actor_class + + +def gitea_salvage_closure_metric_template(label): + return { + "actor_classes": { + "deleted": 0, + "kept": 0, + "system-or-external": 0, + }, + "logical_bytes": 0, + "numeric_totals": { + column: 0 + for column in GITEA_SALVAGE_CLOSURE_NUMERIC_COLUMNS.get(label, ()) + }, + "rows": 0, + "text_bytes": { + column: 0 + for column in GITEA_SALVAGE_CLOSURE_TEXT_COLUMNS.get(label, ()) + }, + } + + +def gitea_salvage_closure_add_metric( + record, + label, + actor_class=None, + logical_bytes=0, + numeric_values=None, + text_bytes=None, +): + metric = record["closure"][label] + metric["rows"] += 1 + if actor_class is not None: + if actor_class not in metric["actor_classes"]: + die("Gitea salvage closure actor classification is invalid") + metric["actor_classes"][actor_class] += 1 + if ( + not isinstance(logical_bytes, int) + or isinstance(logical_bytes, bool) + or logical_bytes < 0 + or logical_bytes > GITEA_SALVAGE_UNSUPPORTED_SIZE_PER_ROW_MAX_BYTES + ): + die("Gitea salvage closure logical-byte aggregate is invalid") + metric["logical_bytes"] += logical_bytes + if metric["logical_bytes"] > GITEA_SALVAGE_UNSUPPORTED_SIZE_TOTAL_MAX_BYTES: + die("Gitea salvage closure logical-byte aggregate exceeds the limit") + for column, value in (numeric_values or {}).items(): + if ( + column not in metric["numeric_totals"] + or not isinstance(value, int) + or isinstance(value, bool) + or value < 0 + or value > GITEA_SALVAGE_UNSUPPORTED_SIZE_PER_ROW_MAX_BYTES + ): + die("Gitea salvage closure declared numeric value is invalid") + metric["numeric_totals"][column] += value + if metric["numeric_totals"][column] > ( + GITEA_SALVAGE_UNSUPPORTED_SIZE_TOTAL_MAX_BYTES + ): + die("Gitea salvage closure declared numeric total exceeds the limit") + for column, value in (text_bytes or {}).items(): + if ( + column not in metric["text_bytes"] + or not isinstance(value, int) + or isinstance(value, bool) + or value < 0 + or value > GITEA_SALVAGE_CLOSURE_MAX_TEXT_BYTES_PER_FIELD + ): + die("Gitea salvage closure text-byte aggregate is invalid") + metric["text_bytes"][column] += value + if metric["text_bytes"][column] > ( + GITEA_SALVAGE_UNSUPPORTED_SIZE_TOTAL_MAX_BYTES + ): + die("Gitea salvage closure text-byte aggregate exceeds the limit") + + +def gitea_salvage_incident_closure_inventory( + connection, + decisions, + unsupported, + topics, +): + if ( + unsupported.get("sha256") + != GITEA_SALVAGE_EXPECTED_UNSUPPORTED_REPORT_SHA256 + or (unsupported.get("report") or {}).get("schema_catalog_sha256") + != GITEA_SALVAGE_DISPOSITION_UNSUPPORTED_SCHEMA_SHA256 + or topics.get("sha256") != GITEA_SALVAGE_DISPOSITION_TOPICS_SHA256 + ): + die("Gitea salvage closure predecessor evidence mismatch") + report = unsupported["report"] + if ( + report.get("database_sha256") + != GITEA_SALVAGE_SNAPSHOT_DATABASE_SHA256 + or report.get("decision_manifest_sha256") + != GITEA_SALVAGE_DECISION_MANIFEST_SHA256 + or report.get("snapshot_uuid") != GITEA_SALVAGE_SNAPSHOT_UUID + or report.get("schema_missing") != [] + or report.get("schema_mismatch") != [] + or report.get("anomalies") != [] + ): + die("Gitea salvage closure source report is not clean") + + user_classes = {} + for row in decisions.get("users", []): + try: + user_id = int(row["user_id"]) + except (KeyError, TypeError, ValueError): + die("Gitea salvage closure user decision is invalid") + decision = row.get("decision") + if ( + user_id <= 0 + or user_id in user_classes + or decision not in {"KEEP_ACTIVE", "KEEP_LOCKED", "DELETE"} + ): + die("Gitea salvage closure user decision is invalid") + user_classes[user_id] = ( + "deleted" if decision == "DELETE" else "kept" + ) + if ( + len(user_classes) != 972 + or sum(value == "kept" for value in user_classes.values()) != 10 + or sum(value == "deleted" for value in user_classes.values()) != 962 + ): + die("Gitea salvage closure user partition mismatch") + + kept_repo_ids = [] + all_repo_ids = set() + for row in decisions.get("repositories", []): + try: + repo_id = int(row["repo_id"]) + except (KeyError, TypeError, ValueError): + die("Gitea salvage closure repository decision is invalid") + if repo_id <= 0 or repo_id in all_repo_ids: + die("Gitea salvage closure repository decision is invalid") + all_repo_ids.add(repo_id) + if row.get("decision") == "KEEP": + kept_repo_ids.append(repo_id) + elif row.get("decision") != "DELETE": + die("Gitea salvage closure repository decision is invalid") + kept_repo_ids.sort() + if len(all_repo_ids) != 2058 or len(kept_repo_ids) != 45: + die("Gitea salvage closure repository partition mismatch") + if report.get("kept_repository_ids") != kept_repo_ids: + die("Gitea salvage closure kept-repository evidence mismatch") + kept_repo_set = set(kept_repo_ids) + placeholders = ",".join("?" for _ in kept_repo_ids) + + schemas = { + row["table"]: row + for row in (report.get("schema_catalog") or {}).get("tables", []) + if isinstance(row, dict) and isinstance(row.get("table"), str) + } + required_schema = {} + closure_extra_schema = {} + + def require_columns(table, columns): + required_schema.setdefault(table, set()).update(columns) + schema = schemas.get(table) + if ( + not isinstance(schema, dict) + or schema.get("ordinary_main_table") is not True + ): + die(f"Gitea salvage closure table is not an ordinary table: {table}") + available = { + column.get("name") + for column in schema.get("columns", []) + if isinstance(column, dict) + } + if not set(columns).issubset(available): + die(f"Gitea salvage closure schema is missing required columns: {table}") + + def require_extra_columns(table, columns): + if table != "team": + die("Gitea salvage closure extra schema table is outside the registry") + table_list_supported = bool( + (report.get("schema_catalog") or {}).get("table_list_supported") + ) + schema = gitea_salvage_table_schema( + connection, + table, + table_list_supported, + allowed_tables={"team"}, + ) + available = { + column.get("name") + for column in schema.get("columns", []) + if isinstance(column, dict) + } + if ( + schema.get("ordinary_main_table") is not True + or not set(columns).issubset(available) + ): + die(f"Gitea salvage closure extra schema is unsafe: {table}") + required_schema.setdefault(table, set()).update(columns) + closure_extra_schema[table] = schema + + def fetch_rows(label, sql, parameters): + try: + rows = connection.execute(sql, parameters).fetchall() + except sqlite3.DatabaseError: + die(f"Gitea salvage closure {label} query failed") + if len(rows) > GITEA_SALVAGE_CLOSURE_MAX_ROWS_PER_RELATION: + die(f"Gitea salvage closure {label} exceeds the row limit") + return rows + + per_repository = [ + { + "closure": { + label: gitea_salvage_closure_metric_template(label) + for label in GITEA_SALVAGE_CLOSURE_METRICS + }, + "issue_states": { + "ordinary_closed": 0, + "ordinary_open": 0, + "pull_wrapper_closed": 0, + "pull_wrapper_open": 0, + }, + "old_repo_id": repo_id, + "pull_states": {"merged": 0, "unmerged": 0}, + } + for repo_id in kept_repo_ids + ] + by_repo_id = {row["old_repo_id"]: row for row in per_repository} + actor_relations = [] + query_contract = [] + + for label, table, target_disposition in GITEA_SALVAGE_CLOSURE_ACTOR_RELATIONS: + require_columns(table, ("id", "repo_id", "user_id", "mode")) + rows = fetch_rows( + label, + f'SELECT id,repo_id,user_id,mode,typeof(id),typeof(repo_id),' + f'typeof(user_id),typeof(mode) FROM "{table}" ' + f'WHERE repo_id IN ({placeholders}) ORDER BY repo_id,user_id,id', + kept_repo_ids, + ) + seen_pairs = set() + for row in rows: + relation_id, repo_id, user_id, mode = row[0:4] + if ( + tuple(row[4:8]) != ("integer", "integer", "integer", "integer") + or not all( + isinstance(value, int) and not isinstance(value, bool) + for value in (relation_id, repo_id, user_id, mode) + ) + or relation_id <= 0 + or repo_id not in kept_repo_set + or user_id <= 0 + or not 0 <= mode <= 5 + or (repo_id, user_id) in seen_pairs + ): + die(f"Gitea salvage closure {label} row is invalid") + seen_pairs.add((repo_id, user_id)) + actor_class = gitea_salvage_closure_actor_class( + user_id, + user_classes, + label, + ) + actor_relations.append( + { + "actor_class": actor_class, + "disposition": ( + target_disposition + if label == "access_cache" or actor_class == "kept" + else "DROP_DELETED_ACTOR" + ), + "legacy_mode": mode, + "old_relation_id": relation_id, + "old_repo_id": repo_id, + "old_user_id": user_id, + "relation": label, + } + ) + query_contract.append( + { + "join": "direct-repository-and-full-user-decision-partition", + "label": label, + "output": "ids-mode-actor-class-disposition-no-user-payload", + "table": table, + } + ) + + require_columns( + "issue", + ( + "id", + "repo_id", + "poster_id", + "name", + "content", + "is_pull", + "is_closed", + "milestone_id", + ), + ) + issue_rows = fetch_rows( + "issues", + f"SELECT id,repo_id,poster_id,is_pull,is_closed,milestone_id," + "typeof(id),typeof(repo_id),typeof(poster_id),typeof(is_pull)," + "typeof(is_closed),typeof(milestone_id),typeof(name)," + "CASE WHEN name IS NULL THEN 0 ELSE length(CAST(name AS BLOB)) END," + "typeof(content),CASE WHEN content IS NULL THEN 0 " + "ELSE length(CAST(content AS BLOB)) END FROM issue " + f"WHERE repo_id IN ({placeholders}) ORDER BY repo_id,id", + kept_repo_ids, + ) + issue_to_repo = {} + issue_is_pull = {} + issue_milestones = [] + for row in issue_rows: + ( + issue_id, + repo_id, + poster_id, + is_pull, + is_closed, + milestone_id, + id_type, + repo_type, + poster_type, + pull_type, + closed_type, + milestone_type, + name_type, + name_bytes, + content_type, + content_bytes, + ) = row + if ( + (id_type, repo_type, poster_type, pull_type, closed_type) + != ("integer", "integer", "integer", "integer", "integer") + or milestone_type not in ("integer", "null") + or name_type not in ("null", "text") + or content_type not in ("null", "text") + or not all( + isinstance(value, int) and not isinstance(value, bool) + for value in ( + issue_id, + repo_id, + poster_id, + is_pull, + is_closed, + name_bytes, + content_bytes, + ) + ) + or issue_id <= 0 + or issue_id in issue_to_repo + or repo_id not in kept_repo_set + or is_pull not in (0, 1) + or is_closed not in (0, 1) + or milestone_id is not None + and ( + not isinstance(milestone_id, int) + or isinstance(milestone_id, bool) + or milestone_id < 0 + ) + ): + die("Gitea salvage closure issue row is invalid") + issue_to_repo[issue_id] = repo_id + issue_is_pull[issue_id] = bool(is_pull) + if milestone_id: + issue_milestones.append((repo_id, issue_id, milestone_id)) + label = "pull_request_wrappers" if is_pull else "issues_ordinary" + state_label = ( + "pull_wrapper_closed" + if is_pull and is_closed + else "pull_wrapper_open" + if is_pull + else "ordinary_closed" + if is_closed + else "ordinary_open" + ) + by_repo_id[repo_id]["issue_states"][state_label] += 1 + actor_class = gitea_salvage_closure_actor_class( + poster_id, + user_classes, + "issue poster", + ) + gitea_salvage_closure_add_metric( + by_repo_id[repo_id], + label, + actor_class, + text_bytes={"content": content_bytes, "name": name_bytes}, + ) + query_contract.append( + { + "join": "issue.repo_id-in-exact-kept-set", + "label": "issues-and-pull-wrappers", + "output": "counts-actor-classes-name-content-byte-lengths", + "table": "issue", + } + ) + + require_columns( + "pull_request", + ("id", "issue_id", "base_repo_id", "head_repo_id", "has_merged"), + ) + pull_rows = fetch_rows( + "pull requests", + f"SELECT p.id,p.issue_id,p.base_repo_id,p.head_repo_id,p.has_merged," + "typeof(p.id),typeof(p.issue_id),typeof(p.base_repo_id)," + "typeof(p.head_repo_id),typeof(p.has_merged) FROM pull_request AS p " + "INNER JOIN issue AS i ON i.id=p.issue_id " + f"WHERE i.repo_id IN ({placeholders}) ORDER BY i.repo_id,p.id", + kept_repo_ids, + ) + pull_to_repo = {} + pull_issue_ids = set() + pull_head_partitions = {"deleted": 0, "kept": 0} + for row in pull_rows: + pull_id, issue_id, base_repo_id, head_repo_id, has_merged = row[0:5] + if ( + tuple(row[5:10]) + != ("integer", "integer", "integer", "integer", "integer") + or not all( + isinstance(value, int) and not isinstance(value, bool) + for value in ( + pull_id, + issue_id, + base_repo_id, + head_repo_id, + has_merged, + ) + ) + or pull_id <= 0 + or pull_id in pull_to_repo + or issue_id not in issue_to_repo + or not issue_is_pull[issue_id] + or issue_id in pull_issue_ids + or base_repo_id != issue_to_repo[issue_id] + or head_repo_id not in all_repo_ids + or has_merged not in (0, 1) + ): + die("Gitea salvage closure pull-request row is invalid") + pull_to_repo[pull_id] = base_repo_id + pull_issue_ids.add(issue_id) + by_repo_id[base_repo_id]["pull_states"][ + "merged" if has_merged else "unmerged" + ] += 1 + pull_head_partitions[ + "kept" if head_repo_id in kept_repo_set else "deleted" + ] += 1 + if pull_issue_ids != { + issue_id for issue_id, is_pull in issue_is_pull.items() if is_pull + }: + die("Gitea salvage closure pull-wrapper relation is incomplete") + query_contract.append( + { + "join": "pull_request.issue_id-to-kept-issue-base-repo-exact", + "label": "pull-request-identity", + "output": "counts-and-head-repository-partition-no-branch-text", + "table": "pull_request", + } + ) + + comment_to_repo = {} + comment_to_issue = {} + content_history_to_issue = {} + review_to_issue = {} + tracked_time_to_issue = {} + issue_label_pairs = [] + for label, table, actor_column, text_columns in ( + GITEA_SALVAGE_CLOSURE_ISSUE_DEPENDENTS + ): + required = ["id", "issue_id", *text_columns] + if actor_column is not None: + required.append(actor_column) + if table == "issue_label": + required.append("label_id") + require_columns(table, required) + select_parts = [ + "d.id AS relation_id", + "d.issue_id AS issue_id", + "i.repo_id AS repo_id", + "typeof(d.id) AS relation_id_type", + "typeof(d.issue_id) AS issue_id_type", + "typeof(i.repo_id) AS repo_id_type", + ] + if actor_column is not None: + select_parts.extend( + ( + f'd."{actor_column}" AS actor_id', + f'typeof(d."{actor_column}") AS actor_id_type', + ) + ) + if table == "issue_label": + select_parts.extend( + ( + "d.label_id AS label_id", + "typeof(d.label_id) AS label_id_type", + ) + ) + for column in text_columns: + select_parts.extend( + ( + f'typeof(d."{column}") AS "{column}_type"', + f'CASE WHEN d."{column}" IS NULL THEN 0 ' + f'ELSE length(CAST(d."{column}" AS BLOB)) ' + f'END AS "{column}_bytes"', + ) + ) + rows = fetch_rows( + label, + f'SELECT {",".join(select_parts)} FROM "{table}" AS d ' + "INNER JOIN issue AS i ON i.id=d.issue_id " + f"WHERE i.repo_id IN ({placeholders}) ORDER BY i.repo_id,d.id", + kept_repo_ids, + ) + seen_ids = set() + for row in rows: + relation_id = row["relation_id"] + issue_id = row["issue_id"] + repo_id = row["repo_id"] + if ( + (row["relation_id_type"], row["issue_id_type"], row["repo_id_type"]) + != ("integer", "integer", "integer") + or not all( + isinstance(value, int) and not isinstance(value, bool) + for value in (relation_id, issue_id, repo_id) + ) + or relation_id <= 0 + or relation_id in seen_ids + or issue_to_repo.get(issue_id) != repo_id + ): + die(f"Gitea salvage closure {label} row is invalid") + seen_ids.add(relation_id) + actor_class = None + if actor_column is not None: + if row["actor_id_type"] != "integer": + die(f"Gitea salvage closure {label} actor is invalid") + actor_class = gitea_salvage_closure_actor_class( + row["actor_id"], + user_classes, + label, + ) + byte_values = {} + for column in text_columns: + if row[f"{column}_type"] not in ("null", "text"): + die(f"Gitea salvage closure {label} text type is invalid") + byte_values[column] = row[f"{column}_bytes"] + gitea_salvage_closure_add_metric( + by_repo_id[repo_id], + label, + actor_class, + text_bytes=byte_values, + ) + if table == "comment": + comment_to_repo[relation_id] = repo_id + comment_to_issue[relation_id] = issue_id + elif table == "issue_content_history": + content_history_to_issue[relation_id] = issue_id + elif table == "issue_label": + if ( + row["label_id_type"] != "integer" + or not isinstance(row["label_id"], int) + or isinstance(row["label_id"], bool) + or row["label_id"] <= 0 + ): + die("Gitea salvage closure issue-label row is invalid") + issue_label_pairs.append( + (repo_id, issue_id, row["label_id"]) + ) + elif table == "review": + review_to_issue[relation_id] = issue_id + elif table == "tracked_time": + tracked_time_to_issue[relation_id] = issue_id + query_contract.append( + { + "join": f"{table}.issue_id-to-kept-issue", + "label": label, + "output": ( + "counts-actor-classes-and-text-byte-lengths-no-payload" + ), + "table": table, + } + ) + + require_columns( + "reaction", + ("id", "issue_id", "comment_id", "user_id"), + ) + reaction_rows = fetch_rows( + "reactions", + "SELECT r.id,r.issue_id,r.comment_id,r.user_id," + "typeof(r.id),typeof(r.issue_id),typeof(r.comment_id),typeof(r.user_id)," + "direct_issue.repo_id,comment_issue.repo_id " + "FROM reaction AS r " + "LEFT JOIN issue AS direct_issue ON direct_issue.id=r.issue_id " + "LEFT JOIN comment AS c ON c.id=r.comment_id " + "LEFT JOIN issue AS comment_issue ON comment_issue.id=c.issue_id " + f"WHERE direct_issue.repo_id IN ({placeholders}) " + f"OR comment_issue.repo_id IN ({placeholders}) ORDER BY r.id", + (*kept_repo_ids, *kept_repo_ids), + ) + seen_reactions = set() + for row in reaction_rows: + reaction_id, issue_id, comment_id, user_id = row[0:4] + direct_repo, comment_repo = row[8:10] + if ( + tuple(row[4:8]) != ("integer", "integer", "integer", "integer") + or not all( + isinstance(value, int) and not isinstance(value, bool) + for value in (reaction_id, issue_id, comment_id, user_id) + ) + or reaction_id <= 0 + or reaction_id in seen_reactions + or issue_id < 0 + or comment_id < 0 + ): + die("Gitea salvage closure reaction row is invalid") + candidate_repos = { + value + for value in (direct_repo, comment_repo) + if isinstance(value, int) and value in kept_repo_set + } + if len(candidate_repos) != 1: + die("Gitea salvage closure reaction association is ambiguous") + repo_id = candidate_repos.pop() + if issue_id and issue_to_repo.get(issue_id) != repo_id: + die("Gitea salvage closure reaction issue association is invalid") + if comment_id and comment_to_repo.get(comment_id) != repo_id: + die("Gitea salvage closure reaction comment association is invalid") + seen_reactions.add(reaction_id) + actor_class = gitea_salvage_closure_actor_class( + user_id, + user_classes, + "reaction", + ) + gitea_salvage_closure_add_metric( + by_repo_id[repo_id], + "reactions", + actor_class, + ) + query_contract.append( + { + "join": "reaction.issue-or-comment-to-kept-issue-exact-one-repo", + "label": "reactions", + "output": "counts-and-actor-classes-no-reaction-text", + "table": "reaction", + } + ) + + require_columns("review_state", ("id", "pull_id", "user_id", "updated_files")) + review_state_rows = fetch_rows( + "review states", + "SELECT s.id,s.pull_id,s.user_id,p.base_repo_id," + "typeof(s.id),typeof(s.pull_id),typeof(s.user_id),typeof(p.base_repo_id)," + "typeof(s.updated_files),CASE WHEN s.updated_files IS NULL THEN 0 " + "ELSE length(CAST(s.updated_files AS BLOB)) END " + "FROM review_state AS s INNER JOIN pull_request AS p ON p.id=s.pull_id " + "INNER JOIN issue AS i ON i.id=p.issue_id " + f"WHERE i.repo_id IN ({placeholders}) ORDER BY i.repo_id,s.id", + kept_repo_ids, + ) + seen_review_states = set() + for row in review_state_rows: + state_id, pull_id, user_id, repo_id = row[0:4] + if ( + tuple(row[4:8]) != ("integer", "integer", "integer", "integer") + or row[8] not in ("null", "text") + or not all( + isinstance(value, int) and not isinstance(value, bool) + for value in (state_id, pull_id, user_id, repo_id, row[9]) + ) + or state_id <= 0 + or state_id in seen_review_states + or pull_to_repo.get(pull_id) != repo_id + ): + die("Gitea salvage closure review-state row is invalid") + seen_review_states.add(state_id) + actor_class = gitea_salvage_closure_actor_class( + user_id, + user_classes, + "review state", + ) + gitea_salvage_closure_add_metric( + by_repo_id[repo_id], + "review_states", + actor_class, + text_bytes={"updated_files": row[9]}, + ) + query_contract.append( + { + "join": "review_state.pull_id-to-kept-pull-wrapper", + "label": "review-states", + "output": "counts-actor-classes-updated-files-byte-length-no-payload", + "table": "review_state", + } + ) + + require_columns( + "issue_dependency", + ("id", "user_id", "issue_id", "dependency_id"), + ) + dependency_rows = fetch_rows( + "issue dependencies", + "SELECT d.id,d.user_id,d.issue_id,d.dependency_id,source.repo_id," + "target.repo_id,typeof(d.id),typeof(d.user_id),typeof(d.issue_id)," + "typeof(d.dependency_id) FROM issue_dependency AS d " + "INNER JOIN issue AS source ON source.id=d.issue_id " + "LEFT JOIN issue AS target ON target.id=d.dependency_id " + f"WHERE source.repo_id IN ({placeholders}) ORDER BY source.repo_id,d.id", + kept_repo_ids, + ) + seen_dependencies = set() + cross_repo_dependencies = 0 + for row in dependency_rows: + relation_id, user_id, issue_id, dependency_id, repo_id, target_repo_id = row[0:6] + if ( + tuple(row[6:10]) != ("integer", "integer", "integer", "integer") + or not all( + isinstance(value, int) and not isinstance(value, bool) + for value in row[0:6] + ) + or relation_id <= 0 + or relation_id in seen_dependencies + or issue_to_repo.get(issue_id) != repo_id + or target_repo_id not in all_repo_ids + ): + die("Gitea salvage closure issue-dependency row is invalid") + seen_dependencies.add(relation_id) + cross_repo_dependencies += int(repo_id != target_repo_id) + actor_class = gitea_salvage_closure_actor_class( + user_id, + user_classes, + "issue dependency", + ) + gitea_salvage_closure_add_metric( + by_repo_id[repo_id], + "issue_dependencies", + actor_class, + ) + query_contract.append( + { + "join": "issue_dependency.issue_id-to-kept-source-issue", + "label": "issue-dependencies", + "output": "counts-actor-classes-cross-repo-count-no-text", + "table": "issue_dependency", + } + ) + + direct_text_contracts = ( + ("labels", "label", None, ("name", "description", "color")), + ("milestones", "milestone", None, ("name", "content")), + ( + "projects", + "project", + "creator_id", + ("title", "description"), + ), + ( + "releases", + "release", + "publisher_id", + ("tag_name", "target", "title", "note"), + ), + ) + label_to_repo = {} + milestone_to_repo = {} + project_to_repo = {} + release_to_repo = {} + for label, table, actor_column, text_columns in direct_text_contracts: + required = ["id", "repo_id", *text_columns] + if actor_column is not None: + required.append(actor_column) + require_columns(table, required) + select_parts = [ + "id", + "repo_id", + "typeof(id) AS id_type", + "typeof(repo_id) AS repo_id_type", + ] + if actor_column is not None: + select_parts.extend( + ( + f'"{actor_column}" AS actor_id', + f'typeof("{actor_column}") AS actor_id_type', + ) + ) + for column in text_columns: + select_parts.extend( + ( + f'typeof("{column}") AS "{column}_type"', + f'CASE WHEN "{column}" IS NULL THEN 0 ' + f'ELSE length(CAST("{column}" AS BLOB)) ' + f'END AS "{column}_bytes"', + ) + ) + rows = fetch_rows( + label, + f'SELECT {",".join(select_parts)} FROM "{table}" ' + f"WHERE repo_id IN ({placeholders}) ORDER BY repo_id,id", + kept_repo_ids, + ) + seen_ids = set() + for row in rows: + relation_id = row["id"] + repo_id = row["repo_id"] + if ( + (row["id_type"], row["repo_id_type"]) + != ("integer", "integer") + or not isinstance(relation_id, int) + or isinstance(relation_id, bool) + or relation_id <= 0 + or relation_id in seen_ids + or repo_id not in kept_repo_set + ): + die(f"Gitea salvage closure {label} row is invalid") + seen_ids.add(relation_id) + actor_class = None + if actor_column is not None: + if row["actor_id_type"] != "integer": + die(f"Gitea salvage closure {label} actor is invalid") + actor_class = gitea_salvage_closure_actor_class( + row["actor_id"], + user_classes, + label, + ) + byte_values = {} + for column in text_columns: + if row[f"{column}_type"] not in ("null", "text"): + die(f"Gitea salvage closure {label} text type is invalid") + byte_values[column] = row[f"{column}_bytes"] + gitea_salvage_closure_add_metric( + by_repo_id[repo_id], + label, + actor_class, + text_bytes=byte_values, + ) + if table == "label": + label_to_repo[relation_id] = repo_id + elif table == "milestone": + milestone_to_repo[relation_id] = repo_id + elif table == "project": + project_to_repo[relation_id] = repo_id + elif table == "release": + release_to_repo[relation_id] = repo_id + query_contract.append( + { + "join": f"{table}.repo_id-in-exact-kept-set", + "label": label, + "output": "counts-actor-classes-text-byte-lengths-no-payload", + "table": table, + } + ) + + for repo_id, _issue_id, label_id in issue_label_pairs: + if label_to_repo.get(label_id) != repo_id: + die("Gitea salvage closure issue-label repository relation is invalid") + for repo_id, _issue_id, milestone_id in issue_milestones: + if milestone_to_repo.get(milestone_id) != repo_id: + die("Gitea salvage closure issue-milestone repository relation is invalid") + + require_columns( + "project_board", + ("id", "project_id", "title", "color"), + ) + board_rows = fetch_rows( + "project boards", + "SELECT b.id,b.project_id,p.repo_id,typeof(b.id),typeof(b.project_id)," + "typeof(p.repo_id),typeof(b.title),CASE WHEN b.title IS NULL THEN 0 " + "ELSE length(CAST(b.title AS BLOB)) END,typeof(b.color)," + "CASE WHEN b.color IS NULL THEN 0 ELSE length(CAST(b.color AS BLOB)) END " + "FROM project_board AS b INNER JOIN project AS p ON p.id=b.project_id " + f"WHERE p.repo_id IN ({placeholders}) ORDER BY p.repo_id,b.id", + kept_repo_ids, + ) + board_to_repo = {} + for row in board_rows: + board_id, project_id, repo_id = row[0:3] + if ( + tuple(row[3:6]) != ("integer", "integer", "integer") + or row[6] not in ("null", "text") + or row[8] not in ("null", "text") + or not all( + isinstance(value, int) and not isinstance(value, bool) + for value in (board_id, project_id, repo_id, row[7], row[9]) + ) + or board_id <= 0 + or board_id in board_to_repo + or project_to_repo.get(project_id) != repo_id + ): + die("Gitea salvage closure project-board row is invalid") + board_to_repo[board_id] = repo_id + gitea_salvage_closure_add_metric( + by_repo_id[repo_id], + "project_boards", + text_bytes={"color": row[9], "title": row[7]}, + ) + query_contract.append( + { + "join": "project_board.project_id-to-kept-project", + "label": "project-boards", + "output": "counts-title-color-byte-lengths-no-payload", + "table": "project_board", + } + ) + + require_columns( + "project_issue", + ("id", "issue_id", "project_id", "project_board_id"), + ) + project_issue_rows = fetch_rows( + "project issue links", + "SELECT pi.id,pi.issue_id,pi.project_id,pi.project_board_id," + "i.repo_id,p.repo_id,typeof(pi.id),typeof(pi.issue_id)," + "typeof(pi.project_id),typeof(pi.project_board_id) " + "FROM project_issue AS pi LEFT JOIN issue AS i ON i.id=pi.issue_id " + "LEFT JOIN project AS p ON p.id=pi.project_id " + f"WHERE i.repo_id IN ({placeholders}) OR p.repo_id IN ({placeholders}) " + "ORDER BY pi.id", + (*kept_repo_ids, *kept_repo_ids), + ) + seen_project_issue = set() + for row in project_issue_rows: + link_id, issue_id, project_id, board_id, issue_repo, project_repo = row[0:6] + if ( + tuple(row[6:10]) != ("integer", "integer", "integer", "integer") + or not all( + isinstance(value, int) and not isinstance(value, bool) + for value in row[0:6] + ) + or link_id <= 0 + or link_id in seen_project_issue + or issue_repo != project_repo + or issue_to_repo.get(issue_id) != issue_repo + or project_to_repo.get(project_id) != project_repo + or board_id > 0 and board_to_repo.get(board_id) != project_repo + or board_id < 0 + ): + die("Gitea salvage closure project-issue relation is invalid") + seen_project_issue.add(link_id) + gitea_salvage_closure_add_metric( + by_repo_id[project_repo], + "project_issue_links", + ) + query_contract.append( + { + "join": "project_issue-to-same-kept-repository-issue-project-board", + "label": "project-issue-links", + "output": "counts-only-no-project-or-issue-text", + "table": "project_issue", + } + ) + + # Close the concrete FK-like comment/history/pull subrelations present in + # the pinned snapshot schema. Optional legacy INTEGER fields may be SQL + # NULL or zero; both are treated as absent, but NULL representation counts + # remain explicit. No comment type, body, path, ref text or commit value is + # selected. Team IDs are existence-checked and sealed as HOLD because the + # incident decision has no organization/team allowlist. + subrelation_per_repo = { + repo_id: { + "actor_classes": { + label: { + "deleted": 0, + "kept": 0, + "system-or-external": 0, + } + for label in GITEA_SALVAGE_CLOSURE_ISSUE_SUBRELATION_ACTORS + }, + "counts": { + label: 0 + for label in GITEA_SALVAGE_CLOSURE_ISSUE_SUBRELATION_COUNTS + }, + "external_author_provenance": { + source: { + "id_without_name": 0, + "name_bytes": 0, + "name_without_id": 0, + "rows_with_id": 0, + "rows_with_name": 0, + } + for source in GITEA_SALVAGE_CLOSURE_EXTERNAL_AUTHOR_SOURCES + }, + "null_encodings": { + column: 0 + for column in ( + GITEA_SALVAGE_CLOSURE_ISSUE_SUBRELATION_NULLABLE_COLUMNS + ) + }, + "old_repo_id": repo_id, + "target_repository_classes": { + label: {"deleted": 0, "global": 0, "kept": 0} + for label in ( + GITEA_SALVAGE_CLOSURE_ISSUE_SUBRELATION_REPOSITORIES + ) + }, + } + for repo_id in kept_repo_ids + } + subrelation_holds = [] + + def add_external_author_provenance( + repo_id, + source, + external_id, + external_id_type, + external_name_type, + external_name_bytes, + ): + if source not in GITEA_SALVAGE_CLOSURE_EXTERNAL_AUTHOR_SOURCES: + die("Gitea salvage external-author source is invalid") + if external_id_type == "null" and external_id is None: + subrelation_per_repo[repo_id]["null_encodings"][ + f"{source}.original_author_id" + ] += 1 + external_id = 0 + if ( + external_id_type not in ("integer", "null") + or not isinstance(external_id, int) + or isinstance(external_id, bool) + or external_id < 0 + or external_name_type not in ("null", "text") + or not isinstance(external_name_bytes, int) + or isinstance(external_name_bytes, bool) + or external_name_bytes < 0 + or external_name_bytes + > GITEA_SALVAGE_CLOSURE_MAX_TEXT_BYTES_PER_FIELD + ): + die("Gitea salvage external-author provenance is invalid") + record = subrelation_per_repo[repo_id]["external_author_provenance"][ + source + ] + has_id = external_id > 0 + has_name = external_name_bytes > 0 + record["rows_with_id"] += int(has_id) + record["rows_with_name"] += int(has_name) + record["id_without_name"] += int(has_id and not has_name) + record["name_without_id"] += int(has_name and not has_id) + record["name_bytes"] += external_name_bytes + if record["name_bytes"] > GITEA_SALVAGE_UNSUPPORTED_SIZE_TOTAL_MAX_BYTES: + die("Gitea salvage external-author byte total exceeds the limit") + + def add_subrelation_count(repo_id, label): + if repo_id not in subrelation_per_repo or label not in ( + GITEA_SALVAGE_CLOSURE_ISSUE_SUBRELATION_COUNTS + ): + die("Gitea salvage issue subrelation count is invalid") + subrelation_per_repo[repo_id]["counts"][label] += 1 + + def add_subrelation_actor(repo_id, label, actor_id): + actor_class = gitea_salvage_closure_actor_class( + actor_id, + user_classes, + label, + ) + subrelation_per_repo[repo_id]["actor_classes"][label][actor_class] += 1 + + def add_subrelation_target_repo(repo_id, label, target_repo_id): + if target_repo_id == 0: + target_class = "global" + elif target_repo_id in kept_repo_set: + target_class = "kept" + elif target_repo_id in all_repo_ids: + target_class = "deleted" + else: + die("Gitea salvage issue subrelation target repository is invalid") + subrelation_per_repo[repo_id]["target_repository_classes"][label][ + target_class + ] += 1 + + comment_optional_columns = ( + "assignee_id", + "assignee_team_id", + "dependent_issue_id", + "label_id", + "milestone_id", + "old_milestone_id", + "old_project_id", + "original_author_id", + "project_id", + "ref_action", + "ref_comment_id", + "ref_is_pull", + "ref_issue_id", + "ref_repo_id", + "resolve_doer_id", + "review_id", + "time_id", + ) + require_columns( + "comment", + ("id", "issue_id", "original_author", *comment_optional_columns), + ) + require_extra_columns("team", ("id", "org_id")) + comment_select = [ + "c.id AS comment_id", + "c.issue_id AS source_issue_id", + "source.repo_id AS source_repo_id", + "typeof(c.id) AS comment_id_type", + "typeof(c.issue_id) AS source_issue_id_type", + "typeof(source.repo_id) AS source_repo_id_type", + ] + for column in comment_optional_columns: + comment_select.extend( + ( + f'c."{column}" AS "{column}"', + f'typeof(c."{column}") AS "{column}_type"', + ) + ) + comment_select.extend( + ( + "label_target.id AS label_target_id", + "label_target.repo_id AS label_target_repo_id", + "old_project.id AS old_project_target_id", + "old_project.repo_id AS old_project_target_repo_id", + "current_project.id AS project_target_id", + "current_project.repo_id AS project_target_repo_id", + "old_milestone.id AS old_milestone_target_id", + "old_milestone.repo_id AS old_milestone_target_repo_id", + "current_milestone.id AS milestone_target_id", + "current_milestone.repo_id AS milestone_target_repo_id", + "tracked.id AS tracked_target_id", + "tracked.issue_id AS tracked_target_issue_id", + "team_target.id AS team_target_id", + "dependent.id AS dependent_target_id", + "dependent.repo_id AS dependent_target_repo_id", + "review_target.id AS review_target_id", + "review_target.issue_id AS review_target_issue_id", + "ref_issue.id AS ref_issue_target_id", + "ref_issue.repo_id AS ref_issue_target_repo_id", + "ref_issue.is_pull AS ref_issue_target_is_pull", + "ref_comment.id AS ref_comment_target_id", + "ref_comment.issue_id AS ref_comment_target_issue_id", + "team_target.org_id AS team_target_org_id", + "typeof(c.original_author) AS original_author_type", + "CASE WHEN c.original_author IS NULL THEN 0 " + "ELSE length(CAST(c.original_author AS BLOB)) END " + "AS original_author_bytes", + ) + ) + comment_relation_rows = fetch_rows( + "comment polymorphic subrelations", + f"SELECT {','.join(comment_select)} FROM comment AS c " + "INNER JOIN issue AS source ON source.id=c.issue_id " + "LEFT JOIN label AS label_target ON label_target.id=c.label_id " + "LEFT JOIN project AS old_project ON old_project.id=c.old_project_id " + "LEFT JOIN project AS current_project ON current_project.id=c.project_id " + "LEFT JOIN milestone AS old_milestone " + "ON old_milestone.id=c.old_milestone_id " + "LEFT JOIN milestone AS current_milestone " + "ON current_milestone.id=c.milestone_id " + "LEFT JOIN tracked_time AS tracked ON tracked.id=c.time_id " + "LEFT JOIN team AS team_target ON team_target.id=c.assignee_team_id " + "LEFT JOIN issue AS dependent ON dependent.id=c.dependent_issue_id " + "LEFT JOIN review AS review_target ON review_target.id=c.review_id " + "LEFT JOIN issue AS ref_issue ON ref_issue.id=c.ref_issue_id " + "LEFT JOIN comment AS ref_comment ON ref_comment.id=c.ref_comment_id " + f"WHERE source.repo_id IN ({placeholders}) ORDER BY source.repo_id,c.id", + kept_repo_ids, + ) + + def optional_comment_integer(row, column, repo_id): + value_type = row[f"{column}_type"] + value = row[column] + if value_type == "null" and value is None: + if column == "assignee_team_id": + die("Gitea salvage comment assignee-team identity is NULL") + subrelation_per_repo[repo_id]["null_encodings"][ + f"comment.{column}" + ] += 1 + return 0 + if ( + value_type != "integer" + or not isinstance(value, int) + or isinstance(value, bool) + or value < 0 + ): + die("Gitea salvage comment subrelation value is invalid") + return value + + def validate_scoped_comment_target( + repo_id, + label, + target_id, + joined_id, + joined_repo_id, + allow_global=False, + ): + if target_id == 0: + return + if ( + joined_id != target_id + or not isinstance(joined_repo_id, int) + or isinstance(joined_repo_id, bool) + or ( + joined_repo_id != repo_id + and not (allow_global and joined_repo_id == 0) + ) + ): + die(f"Gitea salvage {label} relation is invalid") + add_subrelation_count(repo_id, label) + add_subrelation_target_repo(repo_id, label, joined_repo_id) + + seen_comment_relations = set() + for row in comment_relation_rows: + comment_id = row["comment_id"] + issue_id = row["source_issue_id"] + repo_id = row["source_repo_id"] + if ( + ( + row["comment_id_type"], + row["source_issue_id_type"], + row["source_repo_id_type"], + ) + != ("integer", "integer", "integer") + or not all( + isinstance(value, int) and not isinstance(value, bool) + for value in (comment_id, issue_id, repo_id) + ) + or comment_id <= 0 + or comment_id in seen_comment_relations + or comment_to_issue.get(comment_id) != issue_id + or issue_to_repo.get(issue_id) != repo_id + ): + die("Gitea salvage comment subrelation source is invalid") + seen_comment_relations.add(comment_id) + values = { + column: optional_comment_integer(row, column, repo_id) + for column in comment_optional_columns + } + if values["assignee_id"] and values["assignee_team_id"]: + die("Gitea salvage comment assignee identities conflict") + add_external_author_provenance( + repo_id, + "comment", + values["original_author_id"], + row["original_author_id_type"], + row["original_author_type"], + row["original_author_bytes"], + ) + + validate_scoped_comment_target( + repo_id, + "comment_label", + values["label_id"], + row["label_target_id"], + row["label_target_repo_id"], + allow_global=True, + ) + for label, column, id_key, repo_key, allow_global in ( + ( + "comment_old_project", + "old_project_id", + "old_project_target_id", + "old_project_target_repo_id", + True, + ), + ( + "comment_current_project", + "project_id", + "project_target_id", + "project_target_repo_id", + True, + ), + ( + "comment_old_milestone", + "old_milestone_id", + "old_milestone_target_id", + "old_milestone_target_repo_id", + False, + ), + ( + "comment_current_milestone", + "milestone_id", + "milestone_target_id", + "milestone_target_repo_id", + False, + ), + ): + validate_scoped_comment_target( + repo_id, + label, + values[column], + row[id_key], + row[repo_key], + allow_global=allow_global, + ) + + time_id = values["time_id"] + if time_id: + if ( + row["tracked_target_id"] != time_id + or row["tracked_target_issue_id"] != issue_id + or tracked_time_to_issue.get(time_id) != issue_id + ): + die("Gitea salvage comment tracked-time relation is invalid") + add_subrelation_count(repo_id, "comment_tracked_time") + + for label, column in ( + ("comment_assignee", "assignee_id"), + ("comment_resolve_doer", "resolve_doer_id"), + ): + actor_id = values[column] + if actor_id: + add_subrelation_count(repo_id, label) + add_subrelation_actor(repo_id, label, actor_id) + + team_id = values["assignee_team_id"] + if team_id: + team_org_id = row["team_target_org_id"] + if ( + row["team_target_id"] != team_id + or not isinstance(team_org_id, int) + or isinstance(team_org_id, bool) + or team_org_id <= 0 + ): + die("Gitea salvage comment assignee-team relation is orphaned") + add_subrelation_count(repo_id, "comment_assignee_team") + subrelation_holds.append( + { + "kind": "comment-assignee-team-mapping", + "old_org_identity_class": ( + gitea_salvage_closure_actor_class( + team_org_id, + user_classes, + "comment assignee-team organization", + ) + ), + "old_org_id": team_org_id, + "old_repo_id": repo_id, + "old_row_id": comment_id, + "old_team_id": team_id, + "source_table": "comment", + } + ) + + dependent_id = values["dependent_issue_id"] + if dependent_id: + target_repo_id = row["dependent_target_repo_id"] + if ( + row["dependent_target_id"] != dependent_id + or not isinstance(target_repo_id, int) + or isinstance(target_repo_id, bool) + or target_repo_id not in all_repo_ids + ): + die("Gitea salvage comment dependent-issue relation is invalid") + add_subrelation_count(repo_id, "comment_dependent_issue") + add_subrelation_target_repo( + repo_id, + "comment_dependent_issue", + target_repo_id, + ) + + review_id = values["review_id"] + if review_id: + if ( + row["review_target_id"] != review_id + or row["review_target_issue_id"] != issue_id + or review_to_issue.get(review_id) != issue_id + ): + die("Gitea salvage comment review relation is invalid") + add_subrelation_count(repo_id, "comment_review") + + ref_repo_id = values["ref_repo_id"] + ref_issue_id = values["ref_issue_id"] + ref_comment_id = values["ref_comment_id"] + ref_action = values["ref_action"] + ref_is_pull = values["ref_is_pull"] + if ref_action not in (0, 1, 2, 3) or ref_is_pull not in (0, 1): + die("Gitea salvage comment cross-reference state is invalid") + if ref_repo_id or ref_issue_id or ref_comment_id: + if ( + ref_repo_id <= 0 + or ref_issue_id <= 0 + or ref_repo_id not in all_repo_ids + or row["ref_issue_target_id"] != ref_issue_id + or row["ref_issue_target_repo_id"] != ref_repo_id + or row["ref_issue_target_is_pull"] != ref_is_pull + ): + die("Gitea salvage comment cross-reference relation is invalid") + if ref_comment_id and ( + row["ref_comment_target_id"] != ref_comment_id + or row["ref_comment_target_issue_id"] != ref_issue_id + ): + die("Gitea salvage comment cross-reference comment is invalid") + add_subrelation_count(repo_id, "comment_cross_reference") + add_subrelation_target_repo( + repo_id, + "comment_cross_reference", + ref_repo_id, + ) + if ref_comment_id: + add_subrelation_count( + repo_id, + "comment_cross_reference_comment", + ) + elif ref_action or ref_is_pull: + die("Gitea salvage empty comment cross-reference state is invalid") + + if seen_comment_relations != set(comment_to_issue): + die("Gitea salvage comment subrelation coverage is incomplete") + + require_columns( + "review", + ( + "id", + "issue_id", + "reviewer_id", + "reviewer_team_id", + "original_author_id", + "original_author", + ), + ) + review_relation_rows = fetch_rows( + "review team and external-author relations", + "SELECT r.id,r.issue_id,i.repo_id,r.reviewer_id,r.reviewer_team_id," + "r.original_author_id,typeof(r.id),typeof(r.issue_id)," + "typeof(i.repo_id),typeof(r.reviewer_team_id)," + "typeof(r.reviewer_id),typeof(r.original_author_id)," + "team_target.id,team_target.org_id," + "typeof(r.original_author),CASE WHEN r.original_author IS NULL THEN 0 " + "ELSE length(CAST(r.original_author AS BLOB)) END " + "FROM review AS r INNER JOIN issue AS i ON i.id=r.issue_id " + "LEFT JOIN team AS team_target ON team_target.id=r.reviewer_team_id " + f"WHERE i.repo_id IN ({placeholders}) ORDER BY i.repo_id,r.id", + kept_repo_ids, + ) + seen_review_relations = set() + for row in review_relation_rows: + ( + review_id, + issue_id, + repo_id, + reviewer_id, + reviewer_team_id, + external_author_id, + ) = row[0:6] + if ( + tuple(row[6:11]) + != ("integer", "integer", "integer", "integer", "integer") + or row[11] not in ("integer", "null") + or not all( + isinstance(value, int) and not isinstance(value, bool) + for value in ( + review_id, + issue_id, + repo_id, + reviewer_id, + reviewer_team_id, + ) + ) + or review_id <= 0 + or review_id in seen_review_relations + or reviewer_team_id < 0 + or reviewer_id < 0 + or reviewer_id and reviewer_team_id + or review_to_issue.get(review_id) != issue_id + or issue_to_repo.get(issue_id) != repo_id + ): + die("Gitea salvage review subrelation source is invalid") + seen_review_relations.add(review_id) + add_external_author_provenance( + repo_id, + "review", + external_author_id, + row[11], + row[14], + row[15], + ) + if reviewer_team_id: + team_id, team_org_id = row[12:14] + if ( + team_id != reviewer_team_id + or not isinstance(team_org_id, int) + or isinstance(team_org_id, bool) + or team_org_id <= 0 + ): + die("Gitea salvage review reviewer-team relation is orphaned") + add_subrelation_count(repo_id, "review_reviewer_team") + subrelation_holds.append( + { + "kind": "review-reviewer-team-mapping", + "old_org_identity_class": ( + gitea_salvage_closure_actor_class( + team_org_id, + user_classes, + "reviewer-team organization", + ) + ), + "old_org_id": team_org_id, + "old_repo_id": repo_id, + "old_row_id": review_id, + "old_team_id": reviewer_team_id, + "source_table": "review", + } + ) + if seen_review_relations != set(review_to_issue): + die("Gitea salvage review subrelation coverage is incomplete") + + require_columns( + "issue_content_history", + ("id", "issue_id", "comment_id"), + ) + history_relation_rows = fetch_rows( + "content-history comment relations", + "SELECT h.id,h.issue_id,h.comment_id,i.repo_id,typeof(h.id)," + "typeof(h.issue_id),typeof(h.comment_id),typeof(i.repo_id) " + "FROM issue_content_history AS h " + "INNER JOIN issue AS i ON i.id=h.issue_id " + f"WHERE i.repo_id IN ({placeholders}) ORDER BY i.repo_id,h.id", + kept_repo_ids, + ) + seen_history_relations = set() + for row in history_relation_rows: + history_id, issue_id, comment_id, repo_id = row[0:4] + if ( + tuple(row[4:6]) != ("integer", "integer") + or row[6] not in ("integer", "null") + or row[7] != "integer" + or not all( + isinstance(value, int) and not isinstance(value, bool) + for value in (history_id, issue_id, repo_id) + ) + or history_id <= 0 + or history_id in seen_history_relations + or issue_to_repo.get(issue_id) != repo_id + ): + die("Gitea salvage content-history subrelation is invalid") + seen_history_relations.add(history_id) + if content_history_to_issue.get(history_id) != issue_id: + die("Gitea salvage content-history coverage is incomplete") + if comment_id is None: + subrelation_per_repo[repo_id]["null_encodings"][ + "issue_content_history.comment_id" + ] += 1 + comment_id = 0 + if ( + not isinstance(comment_id, int) + or isinstance(comment_id, bool) + or comment_id < 0 + ): + die("Gitea salvage content-history comment identity is invalid") + if comment_id: + if comment_to_issue.get(comment_id) != issue_id: + die("Gitea salvage content-history comment relation is invalid") + add_subrelation_count(repo_id, "content_history_comment") + else: + add_subrelation_count(repo_id, "content_history_issue") + if seen_history_relations != set(content_history_to_issue): + die("Gitea salvage content-history subrelation coverage is incomplete") + + require_columns( + "pull_request", + ("id", "base_repo_id", "has_merged", "merger_id"), + ) + merger_rows = fetch_rows( + "pull merger relations", + "SELECT p.id,p.base_repo_id,p.has_merged,p.merger_id,typeof(p.id)," + "typeof(p.base_repo_id),typeof(p.has_merged),typeof(p.merger_id) " + "FROM pull_request AS p INNER JOIN issue AS i ON i.id=p.issue_id " + f"WHERE i.repo_id IN ({placeholders}) ORDER BY p.base_repo_id,p.id", + kept_repo_ids, + ) + seen_mergers = set() + for row in merger_rows: + pull_id, repo_id, has_merged, merger_id = row[0:4] + if ( + tuple(row[4:7]) != ("integer", "integer", "integer") + or row[7] not in ("integer", "null") + or not all( + isinstance(value, int) and not isinstance(value, bool) + for value in (pull_id, repo_id, has_merged) + ) + or pull_id <= 0 + or pull_id in seen_mergers + or pull_to_repo.get(pull_id) != repo_id + or has_merged not in (0, 1) + ): + die("Gitea salvage pull merger subrelation is invalid") + seen_mergers.add(pull_id) + if merger_id is None: + subrelation_per_repo[repo_id]["null_encodings"][ + "pull_request.merger_id" + ] += 1 + merger_id = 0 + if ( + not isinstance(merger_id, int) + or isinstance(merger_id, bool) + or merger_id < 0 + or (not has_merged and merger_id != 0) + ): + die("Gitea salvage pull merger identity is invalid") + if has_merged: + add_subrelation_count(repo_id, "pull_merger") + add_subrelation_actor(repo_id, "pull_merger", merger_id) + if seen_mergers != set(pull_to_repo): + die("Gitea salvage pull merger subrelation coverage is incomplete") + + issue_pr_subrelations = { + "aggregates": { + "actor_classes": { + label: { + actor_class: sum( + record["actor_classes"][label][actor_class] + for record in subrelation_per_repo.values() + ) + for actor_class in ("deleted", "kept", "system-or-external") + } + for label in GITEA_SALVAGE_CLOSURE_ISSUE_SUBRELATION_ACTORS + }, + "counts": { + label: sum( + record["counts"][label] + for record in subrelation_per_repo.values() + ) + for label in GITEA_SALVAGE_CLOSURE_ISSUE_SUBRELATION_COUNTS + }, + "external_author_provenance": { + source: { + field: sum( + record["external_author_provenance"][source][field] + for record in subrelation_per_repo.values() + ) + for field in ( + "id_without_name", + "name_bytes", + "name_without_id", + "rows_with_id", + "rows_with_name", + ) + } + for source in GITEA_SALVAGE_CLOSURE_EXTERNAL_AUTHOR_SOURCES + }, + "null_encodings": { + column: sum( + record["null_encodings"][column] + for record in subrelation_per_repo.values() + ) + for column in ( + GITEA_SALVAGE_CLOSURE_ISSUE_SUBRELATION_NULLABLE_COLUMNS + ) + }, + "target_repository_classes": { + label: { + target_class: sum( + record["target_repository_classes"][label][target_class] + for record in subrelation_per_repo.values() + ) + for target_class in ("deleted", "global", "kept") + } + for label in ( + GITEA_SALVAGE_CLOSURE_ISSUE_SUBRELATION_REPOSITORIES + ) + }, + }, + "conditional_hold_blockers": ( + ["issue-pr-team-mapping-hold"] + if subrelation_holds + else [] + ), + "extra_schema_coverage": [ + closure_extra_schema[table] + for table in sorted(closure_extra_schema) + ], + "holds": sorted( + subrelation_holds, + key=lambda row: ( + row["kind"], + row["old_repo_id"], + row["old_row_id"], + row["old_team_id"], + ), + ), + "per_repository": [ + subrelation_per_repo[repo_id] + for repo_id in kept_repo_ids + ], + "schema": "nodedc.gitea.salvage-issue-pr-subrelation-closure/v1", + } + query_contract.extend( + ( + { + "join": ( + "comment-optional-identities-to-exact-label-project-" + "milestone-time-team-issue-review-and-xref-targets" + ), + "label": "comment-polymorphic-subrelations", + "output": ( + "counts-classes-null-encodings-external-author-byte-" + "lengths-and-team-hold-ids-only" + ), + "table": "comment", + }, + { + "join": "issue_content_history.comment_id-to-same-kept-issue", + "label": "content-history-comment-relations", + "output": "issue-vs-comment-history-counts-only", + "table": "issue_content_history", + }, + { + "join": ( + "review.reviewer_team_id-to-existing-team-and-external-" + "author-presence-classification" + ), + "label": "review-team-and-external-author-relations", + "output": "counts-byte-lengths-and-exact-team-hold-ids-only", + "table": "review", + }, + { + "join": "pull_request.merger_id-to-full-user-decision-partition", + "label": "pull-merger-relations", + "output": "counts-and-actor-classes-no-commit-or-user-payload", + "table": "pull_request", + }, + ) + ) + + require_columns( + "pull_auto_merge", + ("id", "pull_id", "doer_id", "merge_style", "message"), + ) + auto_merge_rows = fetch_rows( + "pull auto merges", + "SELECT a.id,a.pull_id,a.doer_id,p.base_repo_id,typeof(a.id)," + "typeof(a.pull_id),typeof(a.doer_id),typeof(p.base_repo_id)," + "typeof(a.merge_style),CASE WHEN a.merge_style IS NULL THEN 0 " + "ELSE length(CAST(a.merge_style AS BLOB)) END,typeof(a.message)," + "CASE WHEN a.message IS NULL THEN 0 ELSE length(CAST(a.message AS BLOB)) END " + "FROM pull_auto_merge AS a INNER JOIN pull_request AS p ON p.id=a.pull_id " + "INNER JOIN issue AS i ON i.id=p.issue_id " + f"WHERE i.repo_id IN ({placeholders}) ORDER BY i.repo_id,a.id", + kept_repo_ids, + ) + seen_auto_merges = set() + for row in auto_merge_rows: + merge_id, pull_id, doer_id, repo_id = row[0:4] + if ( + tuple(row[4:8]) != ("integer", "integer", "integer", "integer") + or row[8] not in ("null", "text") + or row[10] not in ("null", "text") + or not all( + isinstance(value, int) and not isinstance(value, bool) + for value in (merge_id, pull_id, doer_id, repo_id, row[9], row[11]) + ) + or merge_id <= 0 + or merge_id in seen_auto_merges + or pull_to_repo.get(pull_id) != repo_id + ): + die("Gitea salvage closure pull-auto-merge row is invalid") + seen_auto_merges.add(merge_id) + actor_class = gitea_salvage_closure_actor_class( + doer_id, + user_classes, + "pull auto merge", + ) + gitea_salvage_closure_add_metric( + by_repo_id[repo_id], + "pull_auto_merges", + actor_class, + text_bytes={"merge_style": row[9], "message": row[11]}, + ) + query_contract.append( + { + "join": "pull_auto_merge.pull_id-to-kept-pull-wrapper", + "label": "pull-auto-merges", + "output": "counts-actor-classes-text-byte-lengths-no-payload", + "table": "pull_auto_merge", + } + ) + + require_columns( + "notification", + ("id", "user_id", "repo_id", "issue_id", "comment_id"), + ) + notification_rows = fetch_rows( + "notifications", + f"SELECT id,user_id,repo_id,issue_id,comment_id,typeof(id)," + "typeof(user_id),typeof(repo_id),typeof(issue_id),typeof(comment_id) " + f"FROM notification WHERE repo_id IN ({placeholders}) ORDER BY repo_id,id", + kept_repo_ids, + ) + seen_notifications = set() + for row in notification_rows: + notification_id, user_id, repo_id, issue_id, comment_id = row[0:5] + if ( + tuple(row[5:8]) != ("integer", "integer", "integer") + or row[8] not in ("integer", "null") + or row[9] not in ("integer", "null") + or not all( + value is None + or isinstance(value, int) and not isinstance(value, bool) + for value in row[0:5] + ) + or notification_id <= 0 + or notification_id in seen_notifications + or repo_id not in kept_repo_set + or (issue_id or 0) < 0 + or (comment_id or 0) < 0 + or issue_id and issue_to_repo.get(issue_id) != repo_id + or comment_id and comment_to_repo.get(comment_id) != repo_id + ): + die("Gitea salvage closure notification row is invalid") + seen_notifications.add(notification_id) + actor_class = gitea_salvage_closure_actor_class( + user_id, + user_classes, + "notification", + ) + gitea_salvage_closure_add_metric( + by_repo_id[repo_id], + "notifications", + actor_class, + ) + query_contract.append( + { + "join": "notification.repo_id-and-optional-kept-issue-comment", + "label": "notifications", + "output": "counts-and-actor-classes-no-commit-or-content", + "table": "notification", + } + ) + + require_columns("repo_unit", ("id", "repo_id", "type", "config")) + unit_rows = fetch_rows( + "repo units", + f"SELECT id,repo_id,type,typeof(id),typeof(repo_id),typeof(type)," + "typeof(config),CASE WHEN config IS NULL THEN 0 " + "ELSE length(CAST(config AS BLOB)) END FROM repo_unit " + f"WHERE repo_id IN ({placeholders}) ORDER BY repo_id,type,id", + kept_repo_ids, + ) + unit_types_by_repo = {repo_id: {} for repo_id in kept_repo_ids} + seen_units = set() + for row in unit_rows: + unit_id, repo_id, unit_type = row[0:3] + if ( + tuple(row[3:6]) != ("integer", "integer", "integer") + or row[6] not in ("null", "text") + or not all( + isinstance(value, int) and not isinstance(value, bool) + for value in (unit_id, repo_id, unit_type, row[7]) + ) + or unit_id <= 0 + or unit_id in seen_units + or repo_id not in kept_repo_set + or not 1 <= unit_type <= 10 + or unit_type in unit_types_by_repo[repo_id] + ): + die("Gitea salvage closure repo-unit row is invalid") + seen_units.add(unit_id) + unit_types_by_repo[repo_id][str(unit_type)] = 1 + gitea_salvage_closure_add_metric( + by_repo_id[repo_id], + "repo_units", + text_bytes={"config": row[7]}, + ) + for record in per_repository: + record["unit_types"] = unit_types_by_repo[record["old_repo_id"]] + query_contract.append( + { + "join": "repo_unit.repo_id-in-exact-kept-set", + "label": "repo-units", + "output": "type-counts-and-config-byte-length-no-config-payload", + "table": "repo_unit", + } + ) + + require_columns( + "attachment", + ( + "id", + "uuid", + "repo_id", + "issue_id", + "release_id", + "uploader_id", + "comment_id", + "size", + ), + ) + attachment_rows = fetch_rows( + "attachment manifest", + f"SELECT id,uuid,repo_id,issue_id,release_id,uploader_id,comment_id,size," + "typeof(id),typeof(uuid),typeof(repo_id),typeof(issue_id)," + "typeof(release_id),typeof(uploader_id),typeof(comment_id),typeof(size) " + f"FROM attachment WHERE repo_id IN ({placeholders}) ORDER BY repo_id,id", + kept_repo_ids, + ) + attachment_manifest = [] + seen_attachment_ids = set() + seen_attachment_uuids = set() + attachment_link_classes = { + "comment": 0, + "issue": 0, + "multi-link": 0, + "release": 0, + "unlinked": 0, + } + for row in attachment_rows: + ( + attachment_id, + attachment_uuid, + repo_id, + issue_id, + release_id, + uploader_id, + comment_id, + declared_size, + ) = row[0:8] + if ( + row[8] != "integer" + or row[9] != "text" + or row[10] != "integer" + or any(value not in ("integer", "null") for value in row[11:15]) + or row[15] != "integer" + or not all( + value is None + or isinstance(value, int) and not isinstance(value, bool) + for value in ( + attachment_id, + repo_id, + issue_id, + release_id, + uploader_id, + comment_id, + declared_size, + ) + ) + or attachment_id <= 0 + or attachment_id in seen_attachment_ids + or not isinstance(attachment_uuid, str) + or re.fullmatch( + r"[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}", + attachment_uuid, + ) + is None + or attachment_uuid in seen_attachment_uuids + or repo_id not in kept_repo_set + or (issue_id or 0) < 0 + or (release_id or 0) < 0 + or (comment_id or 0) < 0 + or declared_size < 0 + or declared_size > GITEA_SALVAGE_UNSUPPORTED_SIZE_PER_ROW_MAX_BYTES + ): + die("Gitea salvage closure attachment manifest row is invalid") + links = [] + if issue_id: + if issue_to_repo.get(issue_id) != repo_id: + die("Gitea salvage closure attachment issue relation is invalid") + links.append("issue") + if release_id: + if release_to_repo.get(release_id) != repo_id: + die("Gitea salvage closure attachment release relation is invalid") + links.append("release") + if comment_id: + if comment_to_repo.get(comment_id) != repo_id: + die("Gitea salvage closure attachment comment relation is invalid") + links.append("comment") + link_class = ( + "unlinked" + if not links + else "multi-link" + if len(links) > 1 + else links[0] + ) + actor_class = gitea_salvage_closure_actor_class( + uploader_id, + user_classes, + "attachment uploader", + ) + attachment_link_classes[link_class] += 1 + seen_attachment_ids.add(attachment_id) + seen_attachment_uuids.add(attachment_uuid) + attachment_manifest.append( + { + "content_hash": "unavailable-in-schema", + "declared_size": declared_size, + "disposition": "PHYSICAL_VERIFY_THEN_SANITIZED_ARCHIVE", + "link_class": link_class, + "link_ids": { + "comment_id": int(comment_id or 0), + "issue_id": int(issue_id or 0), + "release_id": int(release_id or 0), + }, + "old_attachment_id": attachment_id, + "old_repo_id": repo_id, + "uploader_class": actor_class, + "uuid": attachment_uuid, + } + ) + query_contract.append( + { + "join": "attachment.repo-and-exact-issue-release-comment-relations", + "label": "attachment-manifest", + "output": ( + "id-uuid-relations-size-uploader-class-no-name-or-file-bytes" + ), + "table": "attachment", + } + ) + + # Package rows are intentionally reduced to relationship identifiers and + # declared blob sizes. Names, versions, metadata_json and property + # name/value payloads are never selected because Packages remain disabled. + require_columns("package", ("id", "repo_id", "owner_id")) + package_rows = fetch_rows( + "packages", + f"SELECT id,repo_id,owner_id,typeof(id),typeof(repo_id),typeof(owner_id) " + f"FROM package WHERE repo_id IN ({placeholders}) ORDER BY repo_id,id", + kept_repo_ids, + ) + package_to_repo = {} + for row in package_rows: + package_id, repo_id, owner_id = row[0:3] + if ( + tuple(row[3:6]) != ("integer", "integer", "integer") + or not all( + isinstance(value, int) and not isinstance(value, bool) + for value in (package_id, repo_id, owner_id) + ) + or package_id <= 0 + or package_id in package_to_repo + or repo_id not in kept_repo_set + or owner_id <= 0 + ): + die("Gitea salvage closure package row is invalid") + package_to_repo[package_id] = repo_id + gitea_salvage_closure_add_metric(by_repo_id[repo_id], "packages") + query_contract.append( + { + "join": "package.repo_id-in-exact-kept-set", + "label": "packages", + "output": "counts-and-relationship-ids-no-name-or-metadata", + "table": "package", + } + ) + + require_columns("package_version", ("id", "package_id", "creator_id")) + package_version_rows = fetch_rows( + "package versions", + "SELECT v.id,v.package_id,v.creator_id,p.repo_id,typeof(v.id)," + "typeof(v.package_id),typeof(v.creator_id),typeof(p.repo_id) " + "FROM package_version AS v INNER JOIN package AS p ON p.id=v.package_id " + f"WHERE p.repo_id IN ({placeholders}) ORDER BY p.repo_id,v.id", + kept_repo_ids, + ) + package_version_to_repo = {} + for row in package_version_rows: + version_id, package_id, creator_id, repo_id = row[0:4] + if ( + tuple(row[4:8]) != ("integer", "integer", "integer", "integer") + or not all( + isinstance(value, int) and not isinstance(value, bool) + for value in (version_id, package_id, creator_id, repo_id) + ) + or version_id <= 0 + or version_id in package_version_to_repo + or package_to_repo.get(package_id) != repo_id + ): + die("Gitea salvage closure package-version row is invalid") + package_version_to_repo[version_id] = repo_id + actor_class = gitea_salvage_closure_actor_class( + creator_id, + user_classes, + "package version creator", + ) + gitea_salvage_closure_add_metric( + by_repo_id[repo_id], + "package_versions", + actor_class, + ) + query_contract.append( + { + "join": "package_version.package_id-to-kept-package", + "label": "package-versions", + "output": "counts-and-creator-class-no-version-or-metadata", + "table": "package_version", + } + ) + + require_columns("package_file", ("id", "version_id", "blob_id")) + package_file_rows = fetch_rows( + "package files", + "SELECT f.id,f.version_id,f.blob_id,p.repo_id,typeof(f.id)," + "typeof(f.version_id),typeof(f.blob_id),typeof(p.repo_id) " + "FROM package_file AS f " + "INNER JOIN package_version AS v ON v.id=f.version_id " + "INNER JOIN package AS p ON p.id=v.package_id " + f"WHERE p.repo_id IN ({placeholders}) ORDER BY p.repo_id,f.id", + kept_repo_ids, + ) + package_file_to_repo = {} + package_file_blob = {} + repo_blob_ids = {repo_id: set() for repo_id in kept_repo_ids} + for row in package_file_rows: + file_id, version_id, blob_id, repo_id = row[0:4] + if ( + tuple(row[4:8]) != ("integer", "integer", "integer", "integer") + or not all( + isinstance(value, int) and not isinstance(value, bool) + for value in (file_id, version_id, blob_id, repo_id) + ) + or file_id <= 0 + or file_id in package_file_to_repo + or blob_id <= 0 + or package_version_to_repo.get(version_id) != repo_id + ): + die("Gitea salvage closure package-file row is invalid") + package_file_to_repo[file_id] = repo_id + package_file_blob[file_id] = blob_id + repo_blob_ids[repo_id].add(blob_id) + gitea_salvage_closure_add_metric(by_repo_id[repo_id], "package_files") + query_contract.append( + { + "join": "package_file.version_id-to-kept-package-version", + "label": "package-files", + "output": "counts-and-blob-relationship-ids-no-file-name", + "table": "package_file", + } + ) + + require_columns("package_blob", ("id", "size")) + all_package_blob_ids = sorted( + {blob_id for values in repo_blob_ids.values() for blob_id in values} + ) + package_blob_sizes = {} + if all_package_blob_ids: + blob_placeholders = ",".join("?" for _ in all_package_blob_ids) + package_blob_rows = fetch_rows( + "package blobs", + f"SELECT id,size,typeof(id),typeof(size) FROM package_blob " + f"WHERE id IN ({blob_placeholders}) ORDER BY id", + tuple(all_package_blob_ids), + ) + for blob_id, blob_size, id_type, size_type in package_blob_rows: + if ( + (id_type, size_type) != ("integer", "integer") + or not isinstance(blob_id, int) + or isinstance(blob_id, bool) + or not isinstance(blob_size, int) + or isinstance(blob_size, bool) + or blob_id <= 0 + or blob_id in package_blob_sizes + or blob_size < 0 + or blob_size > GITEA_SALVAGE_UNSUPPORTED_SIZE_PER_ROW_MAX_BYTES + ): + die("Gitea salvage closure package-blob row is invalid") + package_blob_sizes[blob_id] = blob_size + if set(package_blob_sizes) != set(all_package_blob_ids): + die("Gitea salvage closure package blob relation is incomplete") + for repo_id in kept_repo_ids: + for blob_id in sorted(repo_blob_ids[repo_id]): + gitea_salvage_closure_add_metric( + by_repo_id[repo_id], + "package_blobs", + logical_bytes=package_blob_sizes[blob_id], + ) + query_contract.append( + { + "join": "package_blob.id-to-distinct-kept-package-file-blob-per-repository", + "label": "package-blobs", + "output": "association-counts-and-declared-logical-bytes-no-hashes", + "table": "package_blob", + } + ) + + require_columns("package_property", ("id", "ref_type", "ref_id")) + property_targets = { + 0: package_version_to_repo, + 1: package_file_to_repo, + 2: package_to_repo, + } + all_property_target_ids = sorted( + { + target_id + for target_map in property_targets.values() + for target_id in target_map + } + ) + if all_property_target_ids: + target_placeholders = ",".join("?" for _ in all_property_target_ids) + invalid_property_types = fetch_rows( + "package property reference types", + f"SELECT COUNT(*) FROM package_property WHERE " + f"ref_id IN ({target_placeholders}) AND (" + "typeof(ref_type) != 'integer' OR ref_type NOT IN (0,1,2))", + tuple(all_property_target_ids), + )[0][0] + if ( + not isinstance(invalid_property_types, int) + or isinstance(invalid_property_types, bool) + or invalid_property_types != 0 + ): + die("Gitea salvage closure package-property type is invalid") + property_clauses = [] + property_parameters = [] + for ref_type in (0, 1, 2): + target_ids = sorted(property_targets[ref_type]) + if not target_ids: + continue + property_clauses.append( + f"(ref_type=? AND ref_id IN ({','.join('?' for _ in target_ids)}))" + ) + property_parameters.extend((ref_type, *target_ids)) + package_property_rows = [] + if property_clauses: + package_property_rows = fetch_rows( + "package properties", + "SELECT id,ref_type,ref_id,typeof(id),typeof(ref_type)," + "typeof(ref_id) FROM package_property WHERE " + + " OR ".join(property_clauses) + + " ORDER BY id", + tuple(property_parameters), + ) + seen_properties = set() + for row in package_property_rows: + property_id, ref_type, ref_id = row[0:3] + if ( + tuple(row[3:6]) != ("integer", "integer", "integer") + or not all( + isinstance(value, int) and not isinstance(value, bool) + for value in (property_id, ref_type, ref_id) + ) + or property_id <= 0 + or property_id in seen_properties + or ref_type not in property_targets + or ref_id not in property_targets[ref_type] + ): + die("Gitea salvage closure package-property row is invalid") + seen_properties.add(property_id) + repo_id = property_targets[ref_type][ref_id] + gitea_salvage_closure_add_metric( + by_repo_id[repo_id], + "package_properties", + ) + query_contract.append( + { + "join": ( + "package_property.ref_type-0-version-1-file-2-package-" + "to-exact-kept-package-closure" + ), + "label": "package-properties", + "output": "counts-and-reference-types-no-property-name-or-value", + "table": "package_property", + } + ) + + # Actions are disabled in the target. The report therefore inventories + # only relationship/count/size metadata and never selects event payloads, + # workflow content, variables, secrets, runner tokens or task logs. + action_run_to_repo = {} + + def inventory_direct_action_table( + label, + table, + extra_columns=(), + actor_column=None, + logical_byte_columns=(), + declared_numeric_columns=(), + ): + required = ("id", "repo_id", *extra_columns) + require_columns(table, required) + select_columns = ["id", "repo_id", *extra_columns] + select_parts = [ + *[f'"{column}"' for column in select_columns], + *[f'typeof("{column}") AS "{column}_type"' for column in select_columns], + ] + rows = fetch_rows( + label, + f'SELECT {",".join(select_parts)} FROM "{table}" ' + f"WHERE repo_id IN ({placeholders}) ORDER BY repo_id,id", + kept_repo_ids, + ) + seen_ids = set() + normalized_rows = [] + for row in rows: + values = {column: row[column] for column in select_columns} + if any(row[f"{column}_type"] != "integer" for column in select_columns): + die(f"Gitea salvage closure {label} row type is invalid") + if any( + not isinstance(value, int) or isinstance(value, bool) + for value in values.values() + ): + die(f"Gitea salvage closure {label} row is invalid") + relation_id = values["id"] + repo_id = values["repo_id"] + if ( + relation_id <= 0 + or relation_id in seen_ids + or repo_id not in kept_repo_set + ): + die(f"Gitea salvage closure {label} row is invalid") + seen_ids.add(relation_id) + actor_class = None + if actor_column is not None: + actor_class = gitea_salvage_closure_actor_class( + values[actor_column], + user_classes, + label, + ) + logical_bytes = 0 + numeric_values = {} + for column in declared_numeric_columns: + value = values[column] + if ( + value < 0 + or value > GITEA_SALVAGE_UNSUPPORTED_SIZE_PER_ROW_MAX_BYTES + ): + die(f"Gitea salvage closure {label} declared value is invalid") + numeric_values[column] = value + for column in logical_byte_columns: + value = values[column] + logical_bytes += value + if logical_bytes > GITEA_SALVAGE_UNSUPPORTED_SIZE_PER_ROW_MAX_BYTES: + die(f"Gitea salvage closure {label} size exceeds the row limit") + gitea_salvage_closure_add_metric( + by_repo_id[repo_id], + label, + actor_class, + logical_bytes=logical_bytes, + numeric_values=numeric_values, + ) + normalized_rows.append(values) + query_contract.append( + { + "join": f"{table}.repo_id-in-exact-kept-set", + "label": label.replace("_", "-"), + "output": ( + "counts-actor-class-and-declared-size-metadata-only-" + "no-action-payload-secret-token-or-log" + ), + "table": table, + } + ) + return normalized_rows + + for values in inventory_direct_action_table( + "action_runs", + "action_run", + ("trigger_user_id",), + actor_column="trigger_user_id", + ): + action_run_to_repo[values["id"]] = values["repo_id"] + inventory_direct_action_table( + "action_schedules", + "action_schedule", + ("trigger_user_id",), + actor_column="trigger_user_id", + ) + inventory_direct_action_table("action_runners", "action_runner") + inventory_direct_action_table("action_variables", "action_variable") + inventory_direct_action_table("action_secrets", "secret") + + artifact_rows = inventory_direct_action_table( + "action_artifacts", + "action_artifact", + ("run_id", "file_size", "file_compressed_size"), + logical_byte_columns=("file_size",), + declared_numeric_columns=("file_size", "file_compressed_size"), + ) + for values in artifact_rows: + run_id = values["run_id"] + if run_id <= 0 or action_run_to_repo.get(run_id) != values["repo_id"]: + die("Gitea salvage closure action-artifact run relation is invalid") + if action_run_to_repo: + action_run_ids = sorted(action_run_to_repo) + action_run_placeholders = ",".join("?" for _ in action_run_ids) + for table in ("action_artifact", "action_run_job"): + missed = fetch_rows( + f"{table} indirect rows", + f'SELECT COUNT(*) FROM "{table}" WHERE ' + f"run_id IN ({action_run_placeholders}) " + f"AND (typeof(repo_id) != 'integer' " + f"OR repo_id NOT IN ({placeholders}))", + (*action_run_ids, *kept_repo_ids), + )[0][0] + if not isinstance(missed, int) or isinstance(missed, bool) or missed: + die(f"Gitea salvage closure {table} indirect relation is invalid") + + action_job_to_repo = {} + job_rows = inventory_direct_action_table( + "action_run_jobs", + "action_run_job", + ("run_id",), + ) + for values in job_rows: + run_id = values["run_id"] + if run_id <= 0 or action_run_to_repo.get(run_id) != values["repo_id"]: + die("Gitea salvage closure action-job run relation is invalid") + action_job_to_repo[values["id"]] = values["repo_id"] + + task_rows = inventory_direct_action_table( + "action_tasks", + "action_task", + ("job_id", "log_length", "log_size"), + logical_byte_columns=("log_size",), + declared_numeric_columns=("log_length", "log_size"), + ) + for values in task_rows: + job_id = values["job_id"] + if ( + job_id <= 0 + or action_job_to_repo.get(job_id) != values["repo_id"] + or values["log_length"] < 0 + ): + die("Gitea salvage closure action-task job relation is invalid") + if action_job_to_repo: + action_job_ids = sorted(action_job_to_repo) + action_job_placeholders = ",".join("?" for _ in action_job_ids) + missed = fetch_rows( + "action task indirect rows", + f"SELECT COUNT(*) FROM action_task WHERE " + f"job_id IN ({action_job_placeholders}) " + f"AND (typeof(repo_id) != 'integer' " + f"OR repo_id NOT IN ({placeholders}))", + (*action_job_ids, *kept_repo_ids), + )[0][0] + if not isinstance(missed, int) or isinstance(missed, bool) or missed: + die("Gitea salvage closure action_task indirect relation is invalid") + + require_columns("action_run_index", ("group_id", "max_index")) + action_index_rows = fetch_rows( + "action run indexes", + f"SELECT group_id,max_index,typeof(group_id),typeof(max_index) " + f"FROM action_run_index WHERE group_id IN ({placeholders}) " + "ORDER BY group_id", + kept_repo_ids, + ) + seen_action_indexes = set() + for group_id, max_index, group_type, max_type in action_index_rows: + if ( + (group_type, max_type) != ("integer", "integer") + or not isinstance(group_id, int) + or isinstance(group_id, bool) + or not isinstance(max_index, int) + or isinstance(max_index, bool) + or group_id not in kept_repo_set + or group_id in seen_action_indexes + or max_index < 0 + ): + die("Gitea salvage closure action-run-index row is invalid") + seen_action_indexes.add(group_id) + gitea_salvage_closure_add_metric( + by_repo_id[group_id], + "action_run_indexes", + ) + query_contract.append( + { + "join": ( + "action_run_index.group_id-as-gitea-resource-index-" + "repository-id-in-exact-kept-set" + ), + "label": "action-run-indexes", + "output": "counts-only-no-action-payload", + "table": "action_run_index", + } + ) + + metric_aggregates = {} + for label in GITEA_SALVAGE_CLOSURE_METRICS: + aggregate = gitea_salvage_closure_metric_template(label) + for record in per_repository: + metric = record["closure"][label] + aggregate["rows"] += metric["rows"] + aggregate["logical_bytes"] += metric["logical_bytes"] + for actor_class, count in metric["actor_classes"].items(): + aggregate["actor_classes"][actor_class] += count + for column, value in metric["numeric_totals"].items(): + aggregate["numeric_totals"][column] += value + for column, byte_count in metric["text_bytes"].items(): + aggregate["text_bytes"][column] += byte_count + if aggregate["logical_bytes"] > GITEA_SALVAGE_UNSUPPORTED_SIZE_TOTAL_MAX_BYTES: + die("Gitea salvage closure aggregate byte count exceeds the limit") + if any( + value > GITEA_SALVAGE_UNSUPPORTED_SIZE_TOTAL_MAX_BYTES + for value in aggregate["text_bytes"].values() + ): + die("Gitea salvage closure aggregate text-byte count exceeds the limit") + if any( + value > GITEA_SALVAGE_UNSUPPORTED_SIZE_TOTAL_MAX_BYTES + for value in aggregate["numeric_totals"].values() + ): + die("Gitea salvage closure aggregate numeric total exceeds the limit") + metric_aggregates[label] = aggregate + + actor_relation_counts = { + relation: { + "deleted": sum( + row["actor_class"] == "deleted" + for row in actor_relations + if row["relation"] == relation + ), + "kept": sum( + row["actor_class"] == "kept" + for row in actor_relations + if row["relation"] == relation + ), + "rows": sum(row["relation"] == relation for row in actor_relations), + } + for relation, _table, _disposition in GITEA_SALVAGE_CLOSURE_ACTOR_RELATIONS + } + unit_type_counts = { + str(unit_type): sum( + record["unit_types"].get(str(unit_type), 0) + for record in per_repository + ) + for unit_type in range(1, 11) + } + issue_state_totals = { + state: sum(record["issue_states"][state] for record in per_repository) + for state in ( + "ordinary_closed", + "ordinary_open", + "pull_wrapper_closed", + "pull_wrapper_open", + ) + } + pull_state_totals = { + state: sum(record["pull_states"][state] for record in per_repository) + for state in ("merged", "unmerged") + } + direct_counts = (report.get("aggregates") or {}).get("direct_relation_counts") + attachment_aggregate = (report.get("aggregates") or {}).get("attachments") + cross_checks = { + "access_grants": actor_relation_counts["access_cache"]["rows"], + "action_runners": metric_aggregates["action_runners"]["rows"], + "action_runs": metric_aggregates["action_runs"]["rows"], + "action_schedules": metric_aggregates["action_schedules"]["rows"], + "action_secrets": metric_aggregates["action_secrets"]["rows"], + "action_variables": metric_aggregates["action_variables"]["rows"], + "attachments": len(attachment_manifest), + "collaborators": actor_relation_counts["collaborations"]["rows"], + "issues": ( + metric_aggregates["issues_ordinary"]["rows"] + + metric_aggregates["pull_request_wrappers"]["rows"] + ), + "labels": metric_aggregates["labels"]["rows"], + "milestones": metric_aggregates["milestones"]["rows"], + "packages": metric_aggregates["packages"]["rows"], + "pull_requests_base": len(pull_to_repo), + "releases": metric_aggregates["releases"]["rows"], + } + if not isinstance(direct_counts, dict) or any( + direct_counts.get(label) != value + for label, value in cross_checks.items() + ): + die("Gitea salvage closure predecessor count cross-check failed") + if ( + direct_counts.get("pull_requests_head") != len(pull_to_repo) + or sum(pull_head_partitions.values()) != len(pull_to_repo) + or not isinstance(attachment_aggregate, dict) + or attachment_aggregate.get("association_rows") != len(attachment_manifest) + or attachment_aggregate.get("logical_bytes") + != sum(row["declared_size"] for row in attachment_manifest) + ): + die("Gitea salvage closure predecessor relation cross-check failed") + + attachment_summary = { + "declared_logical_bytes": sum( + row["declared_size"] for row in attachment_manifest + ), + "link_classes": attachment_link_classes, + "physical_presence": "not-inventoried", + "rows": len(attachment_manifest), + "unique_uuids": len(seen_attachment_uuids), + } + closure_report = { + "actor_relation_counts": actor_relation_counts, + "actor_relations": actor_relations, + "aggregates": metric_aggregates, + "attachment_manifest": attachment_manifest, + "attachment_summary": attachment_summary, + "coverage": { + "access_collaboration": ( + "exact-id-mode-and-full-kept-deleted-user-partition" + ), + "actions": ( + "relationship-count-size-metadata-only-no-payload-" + "target-disabled" + ), + "attachments": ( + "database-identifier-relation-size-manifest-only-" + "physical-verifier-pending" + ), + "issues_pull_requests": ( + "schema-bound-primary-dependent-and-concrete-polymorphic-" + "subrelations-counts-classes-text-byte-lengths-only" + ), + "packages": ( + "relationship-count-size-metadata-only-no-payload-" + "target-disabled" + ), + "projects_releases_labels_units": ( + "relationship-counts-text-byte-lengths-and-unit-types-only" + ), + }, + "cross_repo_issue_dependencies": cross_repo_dependencies, + "database_sha256": GITEA_SALVAGE_SNAPSHOT_DATABASE_SHA256, + "decision_manifest_sha256": GITEA_SALVAGE_DECISION_MANIFEST_SHA256, + "kept_repository_ids": kept_repo_ids, + "issue_state_totals": issue_state_totals, + "issue_pr_subrelations": issue_pr_subrelations, + "per_repository": per_repository, + "privacy_contract": { + "attachment_identifier": "strict-lowercase-uuid-only", + "payload_values_selected": False, + "prohibited_exports": [ + "action-event-or-workflow-payload", + "action-log-or-storage-path", + "email-password-token-key-or-secret", + "issue-comment-review-release-or-project-text", + "package-name-version-metadata-hash-property-name-or-value", + ], + "text_evidence": "sqlite-byte-lengths-only", + }, + "pull_head_repository_partition": pull_head_partitions, + "pull_state_totals": pull_state_totals, + "query_contract": sorted(query_contract, key=lambda item: item["label"]), + "remaining_blockers": sorted( + [ + "attachment-physical-verifier-pending", + "closure-report-review-pin-pending", + "collaboration-kept-user-mapping-verifier-pending", + "issue-pr-metadata-sanitized-archive-verifier-pending", + "package-action-physical-closure-verifier-pending", + "target-unit-policy-acceptance-pending", + ] + + issue_pr_subrelations["conditional_hold_blockers"] + ), + "schema": "nodedc.gitea.salvage-closure-inventory/v1", + "schema_catalog_sha256": report["schema_catalog_sha256"], + "schema_coverage": [ + { + "columns": sorted(columns), + "table": table, + } + for table, columns in sorted(required_schema.items()) + ], + "scope": { + "all_repositories": len(all_repo_ids), + "all_users": len(user_classes), + "deleted_repositories": len(all_repo_ids) - len(kept_repo_ids), + "deleted_users": sum( + value == "deleted" for value in user_classes.values() + ), + "kept_repositories": len(kept_repo_ids), + "kept_users": sum(value == "kept" for value in user_classes.values()), + }, + "snapshot_uuid": GITEA_SALVAGE_SNAPSHOT_UUID, + "source_evidence": { + "predecessor_artifact_sha256": ( + GITEA_SALVAGE_CLOSURE_PREDECESSOR_ARTIFACT_SHA256 + ), + "predecessor_disposition_sha256": ( + GITEA_SALVAGE_CLOSURE_PREDECESSOR_DISPOSITION_SHA256 + ), + "reference_manifest_sha256": ( + GITEA_SALVAGE_DISPOSITION_REFERENCE_MANIFEST_SHA256 + ), + "semantic_topics_sha256": topics["sha256"], + "unsupported_report_sha256": unsupported["sha256"], + "unsupported_schema_sha256": report["schema_catalog_sha256"], + }, + "target_policy": { + "access_cache": "RESET_AND_RECOMPUTE", + "actions": "DISABLED_DROP_ALL_LEGACY_ROWS_AND_PHYSICAL_STATE", + "attachments": "SANITIZED_ARCHIVE_ONLY_AFTER_PHYSICAL_VERIFIER", + "collaborations": "RECREATE_EXACT_KEPT_ACTORS_AFTER_ID_MAP", + "issues_pull_requests_and_dependents": "SANITIZED_ARCHIVE_ONLY", + "packages": "DISABLED_DROP_ALL_LEGACY_ROWS_AND_PHYSICAL_STATE", + "repository_counters_watches_stars": "RESET_AND_RECOMPUTE", + "secrets_integrations_credentials": "IMPORT_ZERO", + }, + "unit_type_counts": unit_type_counts, + } + canonical = canonical_gitea_salvage_evidence( + closure_report, + "incident closure report", + ) + return {"report": closure_report, **canonical} + +def gitea_salvage_refname_is_safe(refname): + try: + encoded_refname = refname.encode("utf-8") if isinstance(refname, str) else b"" + except UnicodeEncodeError: + return False + if ( + not isinstance(refname, str) + or not refname.startswith("refs/") + or len(encoded_refname) > 512 + or refname.endswith(("/", ".")) + or ".." in refname + or "@{" in refname + or "//" in refname + or any(char in refname for char in " ~^:?*[\\") + or any(ord(char) < 0x20 or ord(char) == 0x7F for char in refname) + ): + return False + parts = refname.split("/") + return all( + part + and not part.startswith(".") + and not part.endswith(".lock") + for part in parts + ) + + +def gitea_salvage_file_has_nocow(path): + flags = array.array("I", [0]) + try: + descriptor = os.open(path, os.O_RDONLY | getattr(os, "O_NOFOLLOW", 0)) + try: + fcntl.ioctl(descriptor, GITEA_SALVAGE_FS_IOC_GETFLAGS, flags, True) + finally: + os.close(descriptor) + except OSError: + die(f"Gitea salvage cannot attest file flags: {path}") + return bool(flags[0] & GITEA_SALVAGE_FS_NOCOW_FL) + + +def read_gitea_salvage_ref_file(path, label): + try: + path_stat = path.lstat() + raw = path.read_bytes() + except (FileNotFoundError, OSError): + die(f"Gitea salvage {label} is unreadable") + if ( + stat.S_ISLNK(path_stat.st_mode) + or not stat.S_ISREG(path_stat.st_mode) + or path_stat.st_nlink != 1 + or len(raw) > 1024 + or b"\x00" in raw + ): + die(f"Gitea salvage {label} is unsafe") + try: + return raw.decode("ascii") + except UnicodeDecodeError: + die(f"Gitea salvage {label} is not ASCII") + + +def gitea_salvage_walk_error(label): + def fail(error): + error_number = getattr(error, "errno", None) + if not isinstance(error_number, int): + error_number = "unknown" + die(f"Gitea salvage {label} traversal failed: errno={error_number}") + + return fail + + +def inventory_gitea_salvage_bare_repository( + path, + relative_path, + trusted_device, + mountpoints, +): + try: + repo_stat = path.lstat() + except FileNotFoundError: + return None + validate_gitea_salvage_internal_entry( + path, + repo_stat, + trusted_device, + mountpoints, + stat.S_ISDIR, + f"repository source {relative_path}", + ) + objects = path / "objects" + objects_stat = probe_gitea_salvage_path_no_follow( + path, + "objects", + trusted_device, + mountpoints, + f"repository objects root {relative_path}", + final_mode=stat.S_ISDIR, + ) + if objects_stat is None: + die(f"Gitea salvage repository objects are missing: {relative_path}") + + loose_refs_root = path / "refs" + loose_root_stat = probe_gitea_salvage_path_no_follow( + path, + "refs", + trusted_device, + mountpoints, + f"repository loose refs root {relative_path}", + final_mode=stat.S_ISDIR, + ) + repository_info = path / "info" + repository_info_stat = probe_gitea_salvage_path_no_follow( + path, + "info", + trusted_device, + mountpoints, + f"repository info root {relative_path}", + final_mode=stat.S_ISDIR, + ) + forbidden_probes = ( + (objects, "info/alternates"), + (objects, "info/http-alternates"), + (path, "shallow"), + (repository_info, "grafts") if repository_info_stat is not None else None, + (loose_refs_root, "replace") if loose_root_stat is not None else None, + (path, "worktrees"), + (path, "commondir"), + ) + for probe in forbidden_probes: + if probe is None: + continue + probe_root, forbidden = probe + if probe_gitea_salvage_path_no_follow( + probe_root, + forbidden, + trusted_device, + mountpoints, + f"repository forbidden path {relative_path}", + ) is not None: + displayed = (probe_root / forbidden).relative_to(path).as_posix() + die( + "Gitea salvage forbidden Git material present: " + f"{relative_path}/{displayed}" + ) + + object_files = 0 + object_bytes = 0 + oid_lengths = set() + pack_members = {} + pack_bitmaps = set() + excluded_derived_files = [] + excluded_quarantine_directories = [] + for root, directories, files in os.walk( + objects, + topdown=True, + onerror=gitea_salvage_walk_error( + f"repository objects {relative_path}" + ), + followlinks=False, + ): + root_path = Path(root) + root_stat = root_path.lstat() + validate_gitea_salvage_internal_entry( + root_path, + root_stat, + trusted_device, + mountpoints, + stat.S_ISDIR, + f"repository object directory {relative_path}", + ) + directories.sort() + files.sort() + for name in tuple(directories): + child = root_path / name + child_stat = child.lstat() + child_relative = child.relative_to(objects).as_posix() + validate_gitea_salvage_internal_entry( + child, + child_stat, + trusted_device, + mountpoints, + stat.S_ISDIR, + f"repository object directory {relative_path}", + ) + if ( + root_path == objects + and re.fullmatch( + r"tmp_objdir-incoming-[A-Za-z0-9]{6}", + name, + ) + ): + if ( + stat.S_IMODE(child_stat.st_mode) != 0o755 + or child_stat.st_uid != 1000 + or child_stat.st_gid != 1000 + or child_stat.st_nlink != 1 + or gitea_salvage_file_has_nocow(child) + ): + die( + "Gitea salvage receive quarantine directory is unsafe: " + f"{relative_path}/objects/{child_relative}" + ) + excluded_quarantine_directories.append( + { + "kind": "receive-pack-quarantine", + "lstat": { + "gid": child_stat.st_gid, + "mode": f"{stat.S_IMODE(child_stat.st_mode):04o}", + "nlink": child_stat.st_nlink, + "size": child_stat.st_size, + "uid": child_stat.st_uid, + }, + "path": f"objects/{child_relative}", + } + ) + directories.remove(name) + continue + if ( + child_relative not in {"info", "pack"} + and re.fullmatch(r"[a-f0-9]{2}", child_relative) is None + ): + die( + "Gitea salvage unexpected object directory: " + f"{relative_path}/objects/{child_relative}" + ) + for name in files: + child = root_path / name + child_stat = child.lstat() + validate_gitea_salvage_internal_entry( + child, + child_stat, + trusted_device, + mountpoints, + stat.S_ISREG, + f"repository object file {relative_path}", + ) + if ( + child_stat.st_nlink != 1 + or gitea_salvage_file_has_nocow(child) + ): + die(f"Gitea salvage object file is unsafe: {child}") + relative = child.relative_to(objects).as_posix() + if relative == "info/packs": + if child_stat.st_size > GITEA_SALVAGE_DERIVED_INFO_PACKS_MAX_BYTES: + die( + "Gitea salvage derived object cache is oversized: " + f"{relative_path}/objects/{relative}" + ) + excluded_derived_files.append( + { + "bytes": child_stat.st_size, + "kind": "dumb-http-pack-list", + "path": "objects/info/packs", + } + ) + continue + if relative == "info/commit-graph": + if child_stat.st_size > GITEA_SALVAGE_DERIVED_COMMIT_GRAPH_MAX_BYTES: + die( + "Gitea salvage derived object cache is oversized: " + f"{relative_path}/objects/{relative}" + ) + excluded_derived_files.append( + { + "bytes": child_stat.st_size, + "kind": "commit-graph", + "path": "objects/info/commit-graph", + } + ) + continue + bitmap = re.fullmatch( + r"pack/pack-([a-f0-9]{40}|[a-f0-9]{64})\.bitmap", + relative, + ) + if bitmap: + if child_stat.st_size > GITEA_SALVAGE_DERIVED_PACK_BITMAP_MAX_BYTES: + die( + "Gitea salvage derived object cache is oversized: " + f"{relative_path}/objects/{relative}" + ) + digest = bitmap.group(1) + pack_bitmaps.add(digest) + excluded_derived_files.append( + { + "bytes": child_stat.st_size, + "kind": "pack-bitmap", + "path": f"objects/{relative}", + } + ) + continue + loose = re.fullmatch(r"([a-f0-9]{2})/([a-f0-9]{38}|[a-f0-9]{62})", relative) + pack = re.fullmatch( + r"pack/pack-([a-f0-9]{40}|[a-f0-9]{64})\." + r"(pack|idx)", + relative, + ) + if loose: + oid_lengths.add(len(loose.group(1) + loose.group(2))) + elif pack: + oid_lengths.add(len(pack.group(1))) + pack_members.setdefault(pack.group(1), set()).add(pack.group(2)) + elif re.fullmatch( + r"pack/pack-([a-f0-9]{40}|[a-f0-9]{64})\.promisor", + relative, + ): + die( + "Gitea salvage promisor object material is forbidden: " + f"{relative_path}/objects/{relative}" + ) + else: + die( + "Gitea salvage unexpected object material: " + f"{relative_path}/objects/{relative}" + ) + object_files += 1 + object_bytes += child_stat.st_size + orphan_pack_bitmaps = sorted( + digest + for digest in pack_bitmaps + if pack_members.get(digest) != {"pack", "idx"} + ) + if orphan_pack_bitmaps: + die( + "Gitea salvage pack bitmap lacks a complete pack/index pair: " + f"{relative_path}/{orphan_pack_bitmaps[0]}" + ) + incomplete_packs = sorted( + digest + for digest, members in pack_members.items() + if members != {"pack", "idx"} + ) + if incomplete_packs: + die( + "Gitea salvage incomplete pack/index pair: " + f"{relative_path}/{incomplete_packs[0]}" + ) + if len(oid_lengths) > 1 or oid_lengths - {40, 64}: + die(f"Gitea salvage mixed/invalid object format: {relative_path}") + oid_length = next(iter(oid_lengths), 40) + oid_pattern = re.compile(rf"[a-f0-9]{{{oid_length}}}") + + refs = {} + packed_refs = path / "packed-refs" + try: + packed_refs_stat = packed_refs.lstat() + except FileNotFoundError: + packed_refs_stat = None + if packed_refs_stat is not None: + validate_gitea_salvage_internal_entry( + packed_refs, + packed_refs_stat, + trusted_device, + mountpoints, + stat.S_ISREG, + f"repository packed refs {relative_path}", + ) + packed = read_gitea_salvage_ref_file(packed_refs, f"{relative_path} packed-refs") + previous_ref = None + for line in packed.splitlines(): + if not line or line.startswith("#"): + continue + if line.startswith("^"): + if previous_ref is None or oid_pattern.fullmatch(line[1:]) is None: + die(f"Gitea salvage packed-refs peeled line is invalid: {relative_path}") + continue + fields = line.split(" ") + if ( + len(fields) != 2 + or oid_pattern.fullmatch(fields[0]) is None + or not gitea_salvage_refname_is_safe(fields[1]) + or fields[1] in refs + ): + die(f"Gitea salvage packed ref is invalid: {relative_path}") + refs[fields[1]] = fields[0] + previous_ref = fields[1] + if loose_root_stat is not None: + validate_gitea_salvage_internal_entry( + loose_refs_root, + loose_root_stat, + trusted_device, + mountpoints, + stat.S_ISDIR, + f"repository loose refs root {relative_path}", + ) + for root, directories, files in os.walk( + loose_refs_root, + topdown=True, + onerror=gitea_salvage_walk_error( + f"repository refs {relative_path}" + ), + followlinks=False, + ): + root_path = Path(root) + root_stat = root_path.lstat() + validate_gitea_salvage_internal_entry( + root_path, + root_stat, + trusted_device, + mountpoints, + stat.S_ISDIR, + f"repository loose ref directory {relative_path}", + ) + directories.sort() + files.sort() + for name in directories: + directory_path = root_path / name + child_stat = directory_path.lstat() + validate_gitea_salvage_internal_entry( + directory_path, + child_stat, + trusted_device, + mountpoints, + stat.S_ISDIR, + f"repository loose ref directory {relative_path}", + ) + for name in files: + ref_path = root_path / name + ref_path_stat = ref_path.lstat() + validate_gitea_salvage_internal_entry( + ref_path, + ref_path_stat, + trusted_device, + mountpoints, + stat.S_ISREG, + f"repository loose ref file {relative_path}", + ) + refname = ref_path.relative_to(path).as_posix() + value = read_gitea_salvage_ref_file( + ref_path, + f"{relative_path} loose ref", + ).strip() + if ( + not gitea_salvage_refname_is_safe(refname) + or oid_pattern.fullmatch(value) is None + ): + die(f"Gitea salvage loose ref is invalid: {relative_path}/{refname}") + refs[refname] = value + head_path = path / "HEAD" + try: + head_stat = head_path.lstat() + except FileNotFoundError: + die(f"Gitea salvage {relative_path} HEAD is unreadable") + validate_gitea_salvage_internal_entry( + head_path, + head_stat, + trusted_device, + mountpoints, + stat.S_ISREG, + f"repository HEAD {relative_path}", + ) + head_raw = read_gitea_salvage_ref_file(head_path, f"{relative_path} HEAD") + head_match = re.fullmatch(r"ref: (refs/[A-Za-z0-9._/-]+)\n?", head_raw) + if head_match is None or not gitea_salvage_refname_is_safe(head_match.group(1)): + die(f"Gitea salvage HEAD is not a safe symbolic ref: {relative_path}") + manifest_refs = [ + {"name": name, "oid": oid} + for name, oid in sorted(refs.items()) + ] + return { + "relative_path": relative_path, + "head": head_match.group(1), + "object_format": "sha1" if oid_length == 40 else "sha256", + "object_files": object_files, + "object_bytes": object_bytes, + "excluded_derived_files": sorted( + excluded_derived_files, + key=lambda item: item["path"], + ), + "excluded_derived_bytes": sum( + item["bytes"] for item in excluded_derived_files + ), + "excluded_quarantine_directories": sorted( + excluded_quarantine_directories, + key=lambda item: item["path"], + ), + "refs": manifest_refs, + } + + +def inventory_gitea_salvage_repository_refs(decisions): + repositories = [] + mountpoints = gitea_salvage_mountpoints() + repository_root_stat = GITEA_SALVAGE_SNAPSHOT_REPOSITORIES.lstat() + trusted_device = repository_root_stat.st_dev + for row in decisions["kept_repositories"]: + source = GITEA_SALVAGE_SNAPSHOT_REPOSITORIES / row["repo_relative_path"] + validate_gitea_salvage_path_chain( + GITEA_SALVAGE_SNAPSHOT_REPOSITORIES, + source, + stat.S_ISDIR, + f"repository source {row['repo_relative_path']}", + trusted_device=trusted_device, + mountpoints=mountpoints, + ) + inventory = inventory_gitea_salvage_bare_repository( + source, + row["repo_relative_path"], + trusted_device, + mountpoints, + ) + if inventory is None: + die(f"Gitea salvage kept repository source is missing: {row['repo_relative_path']}") + inventory["old_repo_id"] = int(row["repo_id"]) + inventory["owner"] = row["owner"] + inventory["slug"] = row["slug"] + inventory["wiki"] = False + repositories.append(inventory) + wiki_source = GITEA_SALVAGE_SNAPSHOT_REPOSITORIES / row["wiki_relative_path"] + wiki_path = validate_gitea_salvage_path_chain( + GITEA_SALVAGE_SNAPSHOT_REPOSITORIES, + wiki_source, + stat.S_ISDIR, + f"wiki source {row['wiki_relative_path']}", + allow_missing_final=True, + trusted_device=trusted_device, + mountpoints=mountpoints, + ) + if wiki_path is None: + continue + wiki = inventory_gitea_salvage_bare_repository( + wiki_source, + row["wiki_relative_path"], + trusted_device, + mountpoints, + ) + if wiki is None: + die(f"Gitea salvage wiki source disappeared: {row['wiki_relative_path']}") + wiki["old_repo_id"] = int(row["repo_id"]) + wiki["owner"] = row["owner"] + wiki["slug"] = row["slug"] + wiki["wiki"] = True + repositories.append(wiki) + manifest = { + "schema": "nodedc.gitea.salvage-refs/v2", + "snapshot_uuid": GITEA_SALVAGE_SNAPSHOT_UUID, + "database_sha256": GITEA_SALVAGE_SNAPSHOT_DATABASE_SHA256, + "repositories": repositories, + } + canonical = canonical_gitea_salvage_evidence( + manifest, + "reference manifest", + ) + return { + "manifest": manifest, + **canonical, + "repository_stores": len(repositories), + "object_files": sum(item["object_files"] for item in repositories), + "object_bytes": sum(item["object_bytes"] for item in repositories), + "excluded_derived_files": sum( + len(item["excluded_derived_files"]) for item in repositories + ), + "excluded_derived_bytes": sum( + item["excluded_derived_bytes"] for item in repositories + ), + "excluded_quarantine_directories": sum( + len(item["excluded_quarantine_directories"]) + for item in repositories + ), + "refs": sum(len(item["refs"]) for item in repositories), + } + + +def validate_gitea_salvage_disposition_evidence( + disposition, + refs, + unsupported, + topics, +): + source = disposition["sourceEvidence"] + report = unsupported["report"] + topic_evidence = topics["evidence"] + if ( + refs.get("sha256") != source["referenceManifestSha256"] + or refs.get("bytes") != source["referenceManifestBytes"] + or unsupported.get("sha256") + != source["unsupportedRepositoryReportSha256"] + or unsupported.get("bytes") + != source["unsupportedRepositoryReportBytes"] + or report.get("schema_catalog_sha256") + != source["unsupportedSchemaCatalogSha256"] + or report.get("database_sha256") + != source["databaseSha256"] + or report.get("decision_manifest_sha256") + != source["identityDecisionManifestSha256"] + or report.get("snapshot_uuid") != source["snapshotUuid"] + or report.get("schema_missing") != [] + or report.get("schema_mismatch") != [] + or report.get("anomalies") != [] + or not isinstance(report.get("kept_repository_ids"), list) + or len(report["kept_repository_ids"]) + != disposition["scope"]["keptRepositories"] + or len(report.get("per_repository", [])) + != disposition["scope"]["keptRepositories"] + ): + die("Gitea salvage disposition source evidence drift") + + state_policy = disposition["repositoryStatePolicy"] + if report.get("nonzero_categories") != state_policy[ + "sourceNonzeroCategories" + ]: + die("Gitea salvage nonzero-category evidence drift") + expected_direct_counts = { + row["label"]: row["sourceCount"] + for row in state_policy["directRelations"] + } + direct_counts = (report.get("aggregates") or {}).get( + "direct_relation_counts" + ) + if direct_counts != expected_direct_counts: + die("Gitea salvage direct-relation evidence drift") + + expected_unit_counts = { + str(row["type"]): row["sourceCount"] + for row in state_policy["units"]["rows"] + if row["sourceCount"] + } + actual_unit_counts = {} + seen_repo_ids = set() + for repository in report["per_repository"]: + old_repo_id = repository.get("old_repo_id") + unit_types = repository.get("repo_unit_types") + if ( + not isinstance(old_repo_id, int) + or old_repo_id <= 0 + or old_repo_id in seen_repo_ids + or not isinstance(unit_types, dict) + ): + die("Gitea salvage repo-unit evidence row is invalid") + seen_repo_ids.add(old_repo_id) + for unit_type, count in unit_types.items(): + if ( + unit_type not in expected_unit_counts + or not isinstance(count, int) + or isinstance(count, bool) + or count != 1 + ): + die("Gitea salvage repo-unit evidence drift") + actual_unit_counts[unit_type] = ( + actual_unit_counts.get(unit_type, 0) + count + ) + if actual_unit_counts != expected_unit_counts: + die("Gitea salvage repo-unit evidence drift") + + expected_metadata = { + row["name"]: row["sourceRepositories"] + for row in state_policy["textMetadata"] + } + if ( + (report.get("aggregates") or {}).get( + "repository_metadata_presence" + ) + != expected_metadata + ): + die("Gitea salvage repository metadata evidence drift") + expected_schema_only_tables = sorted( + table + for group in state_policy["schemaOnlyDependencyGroups"] + for table in group["tables"] + ) + if ( + sorted((report.get("coverage") or {}).get( + "schema_only_unreviewed_tables", + [], + )) + != expected_schema_only_tables + ): + die("Gitea salvage schema-only coverage evidence drift") + + topic_policy = state_policy["topics"] + topic_rows = topic_evidence.get("repositories") + topic_row_ids = [] + topic_row_arrays = 0 + topic_row_nulls = 0 + topic_row_material = 0 + topic_row_topics = 0 + if not isinstance(topic_rows, list): + die("Gitea salvage semantic-topics evidence rows are invalid") + for row in topic_rows: + if ( + not isinstance(row, dict) + or set(row) + != {"encoding", "material", "old_repo_id", "topic_count"} + or row.get("encoding") not in {"json-array", "json-null"} + or not isinstance(row.get("material"), bool) + or not isinstance(row.get("old_repo_id"), int) + or isinstance(row.get("old_repo_id"), bool) + or row["old_repo_id"] <= 0 + or not isinstance(row.get("topic_count"), int) + or isinstance(row.get("topic_count"), bool) + or not 0 <= row["topic_count"] <= 256 + or row["material"] != bool(row["topic_count"]) + or ( + row["encoding"] == "json-null" + and row["topic_count"] != 0 + ) + ): + die("Gitea salvage semantic-topics evidence row is invalid") + topic_row_ids.append(row["old_repo_id"]) + topic_row_arrays += int(row["encoding"] == "json-array") + topic_row_nulls += int(row["encoding"] == "json-null") + topic_row_material += int(row["material"]) + topic_row_topics += row["topic_count"] + if ( + topic_evidence.get("schema") + != "nodedc.gitea.salvage-semantic-topics/v2" + or topic_row_ids != report["kept_repository_ids"] + or topic_evidence.get("database_sha256") != source["databaseSha256"] + or topic_evidence.get("decision_manifest_sha256") + != source["identityDecisionManifestSha256"] + or topic_evidence.get("snapshot_uuid") != source["snapshotUuid"] + or len(topic_evidence.get("repositories", [])) + != disposition["scope"]["keptRepositories"] + or topic_evidence.get("serialized_arrays") + != topic_policy["expectedSerializedArrays"] + or topic_row_arrays != topic_evidence.get("serialized_arrays") + or topic_evidence.get("serialized_nulls") + != topic_policy["expectedSerializedNulls"] + or topic_row_nulls != topic_evidence.get("serialized_nulls") + or topic_evidence.get("material_repositories") + != topic_policy["expectedMaterialRepositories"] + or topic_row_material != topic_evidence.get("material_repositories") + or topic_evidence.get("topics") != topic_policy["expectedTopics"] + or topic_row_topics != topic_evidence.get("topics") + or direct_counts.get("topics") + != topic_policy["expectedRelationalRows"] + ): + die("Gitea salvage semantic-topics evidence drift") + + actual_decisions = [] + stores = refs.get("manifest", {}).get("repositories") + if not isinstance(stores, list): + die("Gitea salvage reference manifest shape mismatch") + live_head_targets = 0 + missing_head_targets = [] + for repository in stores: + if ( + not isinstance(repository, dict) + or not isinstance(repository.get("old_repo_id"), int) + or not isinstance(repository.get("relative_path"), str) + or not isinstance(repository.get("wiki"), bool) + or repository.get("object_format") != "sha1" + or not isinstance(repository.get("head"), str) + or not repository["head"].startswith("refs/heads/") + or not isinstance(repository.get("refs"), list) + ): + die("Gitea salvage reference store evidence is invalid") + live_names = set() + for ref in repository["refs"]: + name = ref.get("name") + oid = ref.get("oid") + if repository["wiki"] and isinstance(name, str) and name.startswith( + "refs/heads/" + ): + reference_disposition = "LIVE_RESTORE" + elif ( + not repository["wiki"] + and isinstance(name, str) + and ( + name.startswith("refs/heads/") + or name.startswith("refs/tags/") + ) + ): + reference_disposition = "LIVE_RESTORE" + elif ( + not repository["wiki"] + and isinstance(name, str) + and ( + name.startswith("refs/pull/") + or name.startswith("refs/remotes/") + ) + ): + reference_disposition = "SEALED_ARCHIVE_ONLY" + else: + die("Gitea salvage reference namespace is outside disposition") + if reference_disposition == "LIVE_RESTORE": + live_names.add(name) + actual_decisions.append( + { + "disposition": reference_disposition, + "name": name, + "oid": oid, + "oldRepositoryId": repository["old_repo_id"], + "repositoryPath": repository["relative_path"], + "wiki": repository["wiki"], + } + ) + if repository["head"] in live_names: + live_head_targets += 1 + else: + missing_head_targets.append( + { + "head": repository["head"], + "oldRepositoryId": repository["old_repo_id"], + "repositoryPath": repository["relative_path"], + "wiki": repository["wiki"], + } + ) + actual_decisions.sort( + key=lambda row: ( + row["oldRepositoryId"], + int(row["wiki"]), + row["repositoryPath"], + row["name"], + row["oid"], + ) + ) + reference_policy = disposition["referencePolicy"] + head_invariants = reference_policy["headInvariants"] + if ( + len(stores) != head_invariants["stores"] + or refs.get("refs") != len(actual_decisions) + or actual_decisions != reference_policy["exactDecisions"] + or live_head_targets != head_invariants["targetsPresentAmongLiveRefs"] + or sorted( + missing_head_targets, + key=lambda row: ( + row["oldRepositoryId"], + int(row["wiki"]), + row["repositoryPath"], + row["head"], + ), + ) + != head_invariants["allowedMissingTargets"] + ): + die("Gitea salvage exact reference disposition evidence mismatch") + + live_refs = sum( + row["disposition"] == "LIVE_RESTORE" for row in actual_decisions + ) + archive_refs = sum( + row["disposition"] == "SEALED_ARCHIVE_ONLY" + for row in actual_decisions + ) + if ( + live_refs != reference_policy["liveRestore"]["totalRefs"] + or archive_refs != reference_policy["archiveOnly"]["totalRefs"] + or live_refs + archive_refs + != reference_policy["forensicScope"]["allDiscoveredRefs"] + ): + die("Gitea salvage reference disposition totals mismatch") + + return { + "archive_only_refs": archive_refs, + "blockers": list(GITEA_SALVAGE_DISPOSITION_REMAINING_BLOCKERS), + "forensic_refs": len(actual_decisions), + "live_refs": live_refs, + "sha256": GITEA_SALVAGE_DISPOSITION_SHA256, + "topics": topics, + } + + +def validate_gitea_salvage_closure_evidence(disposition, closure): + report = closure.get("report") + if not isinstance(report, dict): + die("Gitea salvage closure evidence report is missing") + source = disposition["sourceEvidence"] + report_source = report.get("source_evidence") + issue_pr_subrelations = report.get("issue_pr_subrelations") + if not isinstance(issue_pr_subrelations, dict): + die("Gitea salvage issue/PR subrelation evidence is missing") + conditional_hold_blockers = issue_pr_subrelations.get( + "conditional_hold_blockers" + ) + expected_report_blockers = [ + "attachment-physical-verifier-pending", + "closure-report-review-pin-pending", + "collaboration-kept-user-mapping-verifier-pending", + "issue-pr-metadata-sanitized-archive-verifier-pending", + "package-action-physical-closure-verifier-pending", + "target-unit-policy-acceptance-pending", + ] + if isinstance(conditional_hold_blockers, list): + expected_report_blockers.extend(conditional_hold_blockers) + expected_report_blockers.sort() + if ( + report.get("schema") != disposition["closureReport"]["schema"] + or report.get("database_sha256") != source["databaseSha256"] + or report.get("decision_manifest_sha256") + != source["identityDecisionManifestSha256"] + or report.get("snapshot_uuid") != source["snapshotUuid"] + or report.get("schema_catalog_sha256") + != source["unsupportedSchemaCatalogSha256"] + or not isinstance(report_source, dict) + or report_source.get("predecessor_artifact_sha256") + != disposition["predecessor"]["artifactSha256"] + or report_source.get("predecessor_disposition_sha256") + != disposition["predecessor"]["dispositionSha256"] + or report_source.get("reference_manifest_sha256") + != source["referenceManifestSha256"] + or report_source.get("semantic_topics_sha256") + != source["semanticTopicsSha256"] + or report_source.get("unsupported_report_sha256") + != source["unsupportedRepositoryReportSha256"] + or report_source.get("unsupported_schema_sha256") + != source["unsupportedSchemaCatalogSha256"] + or report.get("scope") + != { + "all_repositories": 2058, + "all_users": 972, + "deleted_repositories": 2013, + "deleted_users": 962, + "kept_repositories": 45, + "kept_users": 10, + } + or len(report.get("kept_repository_ids", [])) != 45 + or len(report.get("per_repository", [])) != 45 + or report.get("remaining_blockers") != expected_report_blockers + ): + die("Gitea salvage closure source evidence drift") + + require_exact_json_keys( + issue_pr_subrelations, + { + "aggregates", + "conditional_hold_blockers", + "extra_schema_coverage", + "holds", + "per_repository", + "schema", + }, + "Gitea salvage issue/PR subrelation evidence", + ) + if issue_pr_subrelations["schema"] != ( + "nodedc.gitea.salvage-issue-pr-subrelation-closure/v1" + ): + die("Gitea salvage issue/PR subrelation schema drift") + + def validate_nonnegative_integer_map(value, keys, label): + if not isinstance(value, dict) or set(value) != set(keys): + die(f"Gitea salvage {label} shape is invalid") + if any( + not isinstance(item, int) + or isinstance(item, bool) + or item < 0 + or item > GITEA_SALVAGE_UNSUPPORTED_SIZE_TOTAL_MAX_BYTES + for item in value.values() + ): + die(f"Gitea salvage {label} value is invalid") + + class_keys = ("deleted", "kept", "system-or-external") + target_class_keys = ("deleted", "global", "kept") + external_fields = ( + "id_without_name", + "name_bytes", + "name_without_id", + "rows_with_id", + "rows_with_name", + ) + per_subrelations = issue_pr_subrelations.get("per_repository") + if ( + not isinstance(per_subrelations, list) + or len(per_subrelations) != 45 + or [row.get("old_repo_id") for row in per_subrelations] + != report["kept_repository_ids"] + ): + die("Gitea salvage issue/PR per-repository subrelations are invalid") + for row in per_subrelations: + if not isinstance(row, dict): + die("Gitea salvage issue/PR per-repository row is invalid") + require_exact_json_keys( + row, + { + "actor_classes", + "counts", + "external_author_provenance", + "null_encodings", + "old_repo_id", + "target_repository_classes", + }, + "Gitea salvage issue/PR per-repository row", + ) + validate_nonnegative_integer_map( + row["counts"], + GITEA_SALVAGE_CLOSURE_ISSUE_SUBRELATION_COUNTS, + "issue/PR subrelation count", + ) + validate_nonnegative_integer_map( + row["null_encodings"], + GITEA_SALVAGE_CLOSURE_ISSUE_SUBRELATION_NULLABLE_COLUMNS, + "issue/PR null encoding", + ) + actor_classes = row["actor_classes"] + if not isinstance(actor_classes, dict) or set(actor_classes) != set( + GITEA_SALVAGE_CLOSURE_ISSUE_SUBRELATION_ACTORS + ): + die("Gitea salvage issue/PR actor-class shape is invalid") + for label in actor_classes: + validate_nonnegative_integer_map( + actor_classes[label], + class_keys, + "issue/PR actor class", + ) + if sum(actor_classes[label].values()) != row["counts"][label]: + die("Gitea salvage issue/PR actor-class count drift") + target_classes = row["target_repository_classes"] + if not isinstance(target_classes, dict) or set(target_classes) != set( + GITEA_SALVAGE_CLOSURE_ISSUE_SUBRELATION_REPOSITORIES + ): + die("Gitea salvage issue/PR repository-class shape is invalid") + for label in target_classes: + validate_nonnegative_integer_map( + target_classes[label], + target_class_keys, + "issue/PR repository class", + ) + if sum(target_classes[label].values()) != row["counts"][label]: + die("Gitea salvage issue/PR repository-class count drift") + external = row["external_author_provenance"] + if not isinstance(external, dict) or set(external) != set( + GITEA_SALVAGE_CLOSURE_EXTERNAL_AUTHOR_SOURCES + ): + die("Gitea salvage external-author source shape is invalid") + for source in external: + validate_nonnegative_integer_map( + external[source], + external_fields, + "external-author provenance", + ) + if ( + external[source]["id_without_name"] + > external[source]["rows_with_id"] + or external[source]["name_without_id"] + > external[source]["rows_with_name"] + ): + die("Gitea salvage external-author provenance is inconsistent") + + expected_aggregates = { + "actor_classes": { + label: { + actor_class: sum( + row["actor_classes"][label][actor_class] + for row in per_subrelations + ) + for actor_class in class_keys + } + for label in GITEA_SALVAGE_CLOSURE_ISSUE_SUBRELATION_ACTORS + }, + "counts": { + label: sum(row["counts"][label] for row in per_subrelations) + for label in GITEA_SALVAGE_CLOSURE_ISSUE_SUBRELATION_COUNTS + }, + "external_author_provenance": { + source: { + field: sum( + row["external_author_provenance"][source][field] + for row in per_subrelations + ) + for field in external_fields + } + for source in GITEA_SALVAGE_CLOSURE_EXTERNAL_AUTHOR_SOURCES + }, + "null_encodings": { + column: sum( + row["null_encodings"][column] + for row in per_subrelations + ) + for column in GITEA_SALVAGE_CLOSURE_ISSUE_SUBRELATION_NULLABLE_COLUMNS + }, + "target_repository_classes": { + label: { + target_class: sum( + row["target_repository_classes"][label][target_class] + for row in per_subrelations + ) + for target_class in target_class_keys + } + for label in GITEA_SALVAGE_CLOSURE_ISSUE_SUBRELATION_REPOSITORIES + }, + } + if issue_pr_subrelations.get("aggregates") != expected_aggregates: + die("Gitea salvage issue/PR subrelation aggregate drift") + + extra_schema = issue_pr_subrelations.get("extra_schema_coverage") + if ( + not isinstance(extra_schema, list) + or len(extra_schema) != 1 + or not isinstance(extra_schema[0], dict) + or extra_schema[0].get("table") != "team" + or extra_schema[0].get("ordinary_main_table") is not True + or not {"id", "org_id"}.issubset( + { + column.get("name") + for column in extra_schema[0].get("columns", []) + if isinstance(column, dict) + } + ) + ): + die("Gitea salvage issue/PR extra schema coverage is invalid") + + holds = issue_pr_subrelations.get("holds") + if not isinstance(holds, list) or len(holds) > ( + GITEA_SALVAGE_CLOSURE_MAX_ROWS_PER_RELATION + ): + die("Gitea salvage issue/PR hold inventory is invalid") + seen_holds = set() + for row in holds: + if not isinstance(row, dict): + die("Gitea salvage issue/PR hold row is invalid") + require_exact_json_keys( + row, + { + "kind", + "old_org_identity_class", + "old_org_id", + "old_repo_id", + "old_row_id", + "old_team_id", + "source_table", + }, + "Gitea salvage issue/PR hold row", + ) + expected_source = { + "comment-assignee-team-mapping": "comment", + "review-reviewer-team-mapping": "review", + }.get(row["kind"]) + identity = ( + row["source_table"], + row["old_row_id"], + row["old_team_id"], + ) + if ( + expected_source is None + or row["source_table"] != expected_source + or row["old_org_identity_class"] not in {"deleted", "kept"} + or row["old_repo_id"] not in report["kept_repository_ids"] + or identity in seen_holds + or any( + not isinstance(row[key], int) + or isinstance(row[key], bool) + or row[key] <= 0 + for key in ( + "old_org_id", + "old_repo_id", + "old_row_id", + "old_team_id", + ) + ) + ): + die("Gitea salvage issue/PR hold row is invalid") + seen_holds.add(identity) + expected_conditional = ["issue-pr-team-mapping-hold"] if holds else [] + if conditional_hold_blockers != expected_conditional: + die("Gitea salvage issue/PR conditional hold blocker drift") + if len(holds) != ( + expected_aggregates["counts"]["comment_assignee_team"] + + expected_aggregates["counts"]["review_reviewer_team"] + ): + die("Gitea salvage issue/PR team hold count drift") + report_aggregates = report.get("aggregates") + history_aggregate = ( + report_aggregates.get("issue_content_histories") + if isinstance(report_aggregates, dict) + else None + ) + review_aggregate = ( + report_aggregates.get("reviews") + if isinstance(report_aggregates, dict) + else None + ) + pull_state_totals = report.get("pull_state_totals") + if ( + not isinstance(history_aggregate, dict) + or not isinstance(review_aggregate, dict) + or not isinstance(pull_state_totals, dict) + or not isinstance(history_aggregate.get("rows"), int) + or isinstance(history_aggregate.get("rows"), bool) + or not isinstance(review_aggregate.get("rows"), int) + or isinstance(review_aggregate.get("rows"), bool) + or not isinstance(pull_state_totals.get("merged"), int) + or isinstance(pull_state_totals.get("merged"), bool) + or ( + expected_aggregates["counts"]["content_history_comment"] + + expected_aggregates["counts"]["content_history_issue"] + != history_aggregate["rows"] + ) + or expected_aggregates["counts"]["pull_merger"] + != pull_state_totals["merged"] + or expected_aggregates["counts"]["review_reviewer_team"] + > review_aggregate["rows"] + ): + die("Gitea salvage issue/PR subrelation coverage invariant failed") + + actor_relations = report.get("actor_relations") + if not isinstance(actor_relations, list) or len(actor_relations) > 128: + die("Gitea salvage closure actor-relation evidence is invalid") + relation_identities = set() + for row in actor_relations: + if not isinstance(row, dict): + die("Gitea salvage closure actor-relation row is invalid") + require_exact_json_keys( + row, + { + "actor_class", + "disposition", + "legacy_mode", + "old_relation_id", + "old_repo_id", + "old_user_id", + "relation", + }, + "Gitea salvage closure actor-relation row", + ) + identity = (row["relation"], row["old_repo_id"], row["old_user_id"]) + expected = ( + "DROP_CACHE_RECOMPUTE" + if row["relation"] == "access_cache" + else "RECREATE_KEPT_ACTOR_AFTER_ID_MAP" + if row["relation"] == "collaborations" + and row["actor_class"] == "kept" + else "DROP_DELETED_ACTOR" + if row["relation"] == "collaborations" + and row["actor_class"] == "deleted" + else None + ) + if ( + identity in relation_identities + or row["relation"] not in {"access_cache", "collaborations"} + or row["actor_class"] not in {"deleted", "kept"} + or expected is None + or row["disposition"] != expected + or not isinstance(row["legacy_mode"], int) + or isinstance(row["legacy_mode"], bool) + or not 0 <= row["legacy_mode"] <= 5 + ): + die("Gitea salvage closure actor-relation disposition is invalid") + relation_identities.add(identity) + + attachment_summary = report.get("attachment_summary") + attachment_manifest = report.get("attachment_manifest") + if ( + not isinstance(attachment_summary, dict) + or not isinstance(attachment_manifest, list) + or any(not isinstance(row, dict) for row in attachment_manifest) + or attachment_summary.get("rows") != len(attachment_manifest) + or attachment_summary.get("unique_uuids") != len(attachment_manifest) + or attachment_summary.get("physical_presence") != "not-inventoried" + or attachment_summary.get("declared_logical_bytes") + != sum(row.get("declared_size", -1) for row in attachment_manifest) + ): + die("Gitea salvage closure attachment evidence is invalid") + for row in attachment_manifest: + if ( + not isinstance(row, dict) + or row.get("content_hash") != "unavailable-in-schema" + or row.get("disposition") + != "PHYSICAL_VERIFY_THEN_SANITIZED_ARCHIVE" + or row.get("link_class") + not in {"comment", "issue", "multi-link", "release", "unlinked"} + or not isinstance(row.get("declared_size"), int) + or isinstance(row.get("declared_size"), bool) + or row["declared_size"] < 0 + or re.fullmatch( + r"[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-" + r"[0-9a-f]{4}-[0-9a-f]{12}", + str(row.get("uuid")), + ) + is None + ): + die("Gitea salvage closure attachment manifest is invalid") + + if ( + report.get("unit_type_counts") + != { + "1": 45, + "2": 45, + "3": 45, + "4": 45, + "5": 45, + "6": 0, + "7": 0, + "8": 45, + "9": 45, + "10": 0, + } + or (report.get("privacy_contract") or {}).get("payload_values_selected") + is not False + or (report.get("target_policy") or {}).get("actions") + != "DISABLED_DROP_ALL_LEGACY_ROWS_AND_PHYSICAL_STATE" + or (report.get("target_policy") or {}).get("packages") + != "DISABLED_DROP_ALL_LEGACY_ROWS_AND_PHYSICAL_STATE" + or (report.get("target_policy") or {}).get( + "secrets_integrations_credentials" + ) + != "IMPORT_ZERO" + ): + die("Gitea salvage closure target disposition drift") + return { + "blockers": list(GITEA_SALVAGE_CLOSURE_REMAINING_BLOCKERS) + + expected_conditional, + "bytes": closure["bytes"], + "json": closure["json"], + "sha256": closure["sha256"], + } + +def preflight_gitea_incident_salvage(payload_dir, enforce_apply=False): + decisions = validate_gitea_incident_salvage_payload( + payload_dir, + GITEA_SALVAGE_ENTRIES, + ) + try: + GITEA_ROOT.lstat() + except FileNotFoundError: + pass + else: + die("Gitea salvage candidate root must be absent") + if gitea_compose_project_container_ids(): + die("Gitea salvage Compose project already has containers") + maintenance = docker_named_container_inspect_fail_closed( + GITEA_SALVAGE_MAINTENANCE_CONTAINER, + "Gitea salvage maintenance container inspect", + ) + if maintenance is not None: + die("Gitea salvage maintenance container already exists") + snapshot = validate_gitea_salvage_snapshot_boundary() + database = bind_gitea_salvage_decisions_to_snapshot(decisions) + refs = inventory_gitea_salvage_repository_refs(decisions) + disposition = validate_gitea_salvage_disposition_evidence( + decisions["disposition"], + refs, + database["unsupported"], + database["topics"], + ) + closure_disposition = validate_gitea_salvage_closure_evidence( + decisions["closure_disposition"], + database["closure"], + ) + blockers = sorted( + set(disposition["blockers"] + closure_disposition["blockers"]) + ) + legacy = None + if ( + GITEA_SALVAGE_EXPECTED_LEGACY_IMAGE is None + or GITEA_SALVAGE_EXPECTED_LEGACY_IMAGE_ID is None + ): + blockers.append("legacy-container-identity-not-reviewed") + else: + legacy = validate_gitea_salvage_legacy_container() + + common = { + "mode": "clean-state-exact-repository-material-only", + "snapshot": snapshot, + "database": database, + "disposition": disposition, + "closure_disposition": closure_disposition, + "legacy": legacy, + "refs": refs, + "blockers": blockers, + } + if not enforce_apply: + return common + die( + "Gitea salvage activation is frozen before candidate root creation: " + + ",".join(blockers) + ) + + def ensure_engine_node_intelligence_secret(): try: runtime_stat = ENGINE_NODE_INTELLIGENCE_RUNTIME_DIR.lstat() @@ -19113,8 +27212,14 @@ def plan_artifact(artifact): provider_catalog_preflight = None device_plane_postgres_preflight = None device_plane_foundation_recovery_preflight = None + gitea_preflight = None + gitea_salvage_preflight = None with tempfile.TemporaryDirectory(prefix="plan-", dir=TMP_DIR) as tmp: manifest, entries, payload_dir = load_artifact(artifact, Path(tmp)) + if is_gitea_fresh_install_slice(manifest["component"], entries): + validate_gitea_compose_schema(payload_dir / GITEA_COMPOSE_REL) + if is_gitea_incident_salvage_slice(manifest["component"], entries): + validate_gitea_compose_schema(payload_dir / GITEA_COMPOSE_REL) reject_failed_artifact_replay(manifest, sha) reject_terminal_engine_l2_failed_artifact(manifest, sha) reject_terminal_device_plane_foundation_artifact(manifest, sha) @@ -19361,6 +27466,13 @@ def plan_artifact(artifact): payload_dir ) ) + if is_gitea_fresh_install_slice(manifest["component"], entries): + gitea_preflight = preflight_gitea_fresh_install() + if is_gitea_incident_salvage_slice(manifest["component"], entries): + gitea_salvage_preflight = preflight_gitea_incident_salvage( + payload_dir, + enforce_apply=False, + ) component = manifest["component"] root = component_root(component) @@ -19632,6 +27744,149 @@ def plan_artifact(artifact): print(f"build_root={build_root}") print(f"build={' '.join((str(DOCKER),) + tuple(build_args))}") print(f"services={' '.join(services)}") + if gitea_preflight: + print("gitea_transition=fresh-install-only") + print(f"gitea_root={GITEA_ROOT}:required-absent") + print(f"gitea_image={GITEA_IMAGE}") + print(f"gitea_image_id={gitea_preflight['image_id']}") + print("gitea_platform=linux/amd64") + print("gitea_pull=never") + print(f"gitea_docker_version={gitea_preflight['docker_version']}") + print( + "gitea_compose_version=" + f"{gitea_preflight['compose_version']}" + ) + print("gitea_database=fresh-sqlite-users-0-repositories-0") + print("gitea_lfs=disabled-pending-reviewed-restore-transition") + print("gitea_transport=unix:/run/gitea/gitea.sock") + print(f"gitea_socket_host={GITEA_SOCKET_FILE}") + print("gitea_ssh=disabled-no-published-port") + print("gitea_network_mode=none") + print("gitea_docker_ports=none") + print(f"gitea_secret=runner-managed:{GITEA_SECRET_KEY_FILE}") + print(f"gitea_secret=runner-managed:{GITEA_INTERNAL_TOKEN_FILE}") + print("gitea_legacy_root=unread-unmounted-untouched") + print("gitea_legacy_container=never-started") + print( + "gitea_reverse_proxy=" + f"{gitea_preflight['reverse_proxy']['upstream']}" + ) + print( + "gitea_nginx_uds_bridge=" + f"{gitea_preflight['nginx_bridge']['sha256']}" + ) + print("gitea_firewall=legacy-isolated-loopback-3000-ready") + print( + "gitea_rollback=stop-remove-candidate-preserve-runtime-state-" + "quarantine-source" + ) + if gitea_salvage_preflight: + print("gitea_transition=incident-salvage-clean-state") + print(f"gitea_root={GITEA_ROOT}:required-absent") + print(f"gitea_image={GITEA_SALVAGE_IMAGE}") + print(f"gitea_image_id={GITEA_SALVAGE_IMAGE_ID}") + print(f"gitea_repo_digest={GITEA_SALVAGE_REPO_DIGEST}") + print("gitea_database=new-sqlite-no-legacy-import") + print("gitea_users=2-active-8-locked-new-credentials") + print("gitea_repositories=45-exact-object-and-ref-material-only") + print("gitea_repository_visibility=dctouch-private-32,silver-public-13") + print("gitea_two_factor_authentication=unchanged-not-configured") + print("gitea_candidate_publication=socket-parent-0700") + print("gitea_candidate_restart=no") + legacy = gitea_salvage_preflight["legacy"] + legacy_mount = legacy["mounts"][0] + print(f"gitea_legacy_container_name={legacy['name']}") + print(f"gitea_legacy_container_state={legacy['state']}") + print(f"gitea_legacy_image_ref={legacy['image']}") + print(f"gitea_legacy_image_id={legacy['image_id']}") + print( + "gitea_legacy_mount=" + f"{legacy_mount[0]}:{legacy_mount[1]}:{legacy_mount[2]}:" + f"{'rw' if legacy_mount[3] else 'ro'}" + ) + print( + "gitea_snapshot_uuid=" + f"{gitea_salvage_preflight['snapshot']['uuid']}" + ) + print( + "gitea_snapshot_database_sha256=" + f"{gitea_salvage_preflight['snapshot']['database_sha256']}" + ) + print( + "gitea_reference_manifest_sha256=" + f"{gitea_salvage_preflight['refs']['sha256']}" + ) + print( + "gitea_reference_manifest_bytes=" + f"{gitea_salvage_preflight['refs']['bytes']}" + ) + print( + "gitea_reference_manifest_json=" + f"{gitea_salvage_preflight['refs']['json']}" + ) + print( + "gitea_reference_inventory=" + f"stores:{gitea_salvage_preflight['refs']['repository_stores']}," + f"refs:{gitea_salvage_preflight['refs']['refs']}," + f"object_files:{gitea_salvage_preflight['refs']['object_files']}," + f"object_bytes:{gitea_salvage_preflight['refs']['object_bytes']}," + "excluded_derived_files:" + f"{gitea_salvage_preflight['refs']['excluded_derived_files']}," + "excluded_derived_bytes:" + f"{gitea_salvage_preflight['refs']['excluded_derived_bytes']}," + "excluded_quarantine_directories:" + f"{gitea_salvage_preflight['refs']['excluded_quarantine_directories']}" + ) + disposition = gitea_salvage_preflight["disposition"] + print(f"gitea_incident_disposition_sha256={disposition['sha256']}") + print( + "gitea_incident_disposition_refs=" + f"forensic:{disposition['forensic_refs']}," + f"live:{disposition['live_refs']}," + f"archive-only:{disposition['archive_only_refs']}" + ) + print( + "gitea_incident_disposition_policy=" + "heads-tags-and-wiki-heads-live;pull-remote-refs-archive-only;" + "legacy-metadata-sanitized-archive-only;no-legacy-row-import" + ) + topics = disposition["topics"] + print(f"gitea_semantic_topics_sha256={topics['sha256']}") + print(f"gitea_semantic_topics_bytes={topics['bytes']}") + print(f"gitea_semantic_topics_json={topics['json']}") + print( + "gitea_incident_disposition_remaining_blockers=" + + ",".join(disposition["blockers"]) + ) + unsupported = gitea_salvage_preflight["database"]["unsupported"] + print( + "gitea_unsupported_repository_report_sha256=" + f"{unsupported['sha256']}" + ) + print( + "gitea_unsupported_repository_report_bytes=" + f"{unsupported['bytes']}" + ) + print( + "gitea_unsupported_schema_catalog_sha256=" + f"{unsupported['report']['schema_catalog_sha256']}" + ) + print( + "gitea_unsupported_repository_report_json=" + f"{unsupported['json']}" + ) + closure = gitea_salvage_preflight["closure_disposition"] + print( + "gitea_incident_closure_disposition_sha256=" + f"{GITEA_SALVAGE_CLOSURE_DISPOSITION_SHA256}" + ) + print(f"gitea_incident_closure_report_sha256={closure['sha256']}") + print(f"gitea_incident_closure_report_bytes={closure['bytes']}") + print(f"gitea_incident_closure_report_json={closure['json']}") + print( + "gitea_apply_blockers=" + + (",".join(gitea_salvage_preflight["blockers"]) or "none") + ) if l2_closed_loop_preflight: descriptor = l2_closed_loop_preflight["descriptor"] print( @@ -22215,6 +30470,79 @@ def rollback_engine_credential_bridge( return "engine-sink-source-restored" +def rollback_gitea_fresh_install( + root, + backup_dir, + entries, + current_stamp, + runtime_started=False, +): + if root != GITEA_ROOT or not is_gitea_fresh_install_slice("gitea", entries): + die("Gitea fresh-install rollback called for unrelated artifact") + existing = read_backup_path_list(backup_dir / "existing-files.txt") + missing = read_backup_path_list(backup_dir / "missing-files.txt") + existing_set, missing_set = validate_backup_partition( + entries, + existing, + missing, + "Gitea fresh-install rollback", + ) + if existing_set or missing_set != set(GITEA_FRESH_INSTALL_ENTRIES): + die("Gitea fresh-install rollback predecessor must be absent") + + if runtime_started: + try: + stop_and_remove_compose_services("gitea", (GITEA_SERVICE,)) + except Exception: + # Never restore an absent source root while a bind-mounted + # candidate may still be live. Preserve the complete root in + # place and require explicit reconciliation. + raise ReconciliationRequired( + "gitea_candidate_stop_failed_root_preserved" + ) + + try: + candidate_ids = gitea_compose_project_container_ids() + except Exception: + raise ReconciliationRequired( + "gitea_candidate_absence_unproven_root_preserved" + ) + if candidate_ids: + raise ReconciliationRequired( + "gitea_candidate_still_present_root_preserved" + ) + + quarantined_root = None + if root.exists() or root.is_symlink(): + root_stat = root.lstat() + if stat.S_ISLNK(root_stat.st_mode) or not stat.S_ISDIR(root_stat.st_mode): + die("Gitea fresh-install rollback root is unsafe") + quarantined_root = root.with_name( + f"{root.name}.failed-{current_stamp}" + ) + if quarantined_root.exists() or quarantined_root.is_symlink(): + die("Gitea fresh-install rollback quarantine collision") + root.rename(quarantined_root) + + restore_overlay_source(root, backup_dir, entries, current_stamp) + if root.exists() or root.is_symlink(): + die("Gitea fresh-install rollback did not restore absent root") + assert_loopback_tcp_port_closed(GITEA_DISABLED_SSH_HOST_PORT) + validate_legacy_gitea_container_isolation() + if gitea_compose_project_container_ids(): + raise ReconciliationRequired( + "gitea_candidate_reappeared_after_quarantine" + ) + validate_gitea_legacy_candidate_network_absent() + validate_gitea_no_docker_port_publications() + # Data/config/secrets are intentionally retained in the isolated new root + # as incident evidence. No legacy path or legacy container is referenced. + return ( + "candidate-stopped-root-quarantined-runtime-state-preserved:" + + (quarantined_root.name if quarantined_root else "none-created") + ) + + def rollback_engine_node_intelligence( root, backup_dir, @@ -22405,6 +30733,44 @@ def run_compose(component, services, entries=None): ) +def run_gitea_fresh_install_compose(services, entries): + if ( + tuple(services) != (GITEA_SERVICE,) + or not is_gitea_fresh_install_slice("gitea", entries) + ): + die("Gitea fresh-install Compose service set mismatch") + cmd = [ + *compose_base_cmd("gitea"), + "up", + "-d", + "--force-recreate", + "--pull", + "never", + "--no-deps", + GITEA_SERVICE, + ] + try: + subprocess.run(cmd, cwd=str(GITEA_ROOT), check=True) + except subprocess.CalledProcessError: + subprocess.run( + [ + *compose_base_cmd("gitea"), + "logs", + "--no-color", + "--tail=180", + GITEA_SERVICE, + ], + cwd=str(GITEA_ROOT), + check=False, + ) + raise + subprocess.run( + [*compose_base_cmd("gitea"), "ps"], + cwd=str(GITEA_ROOT), + check=True, + ) + + def run_engine_node_intelligence_compose(services, entries): if not is_engine_node_intelligence_transition("engine", entries): die("Engine node-intelligence Compose called for unrelated artifact") @@ -22559,7 +30925,108 @@ def stop_and_remove_compose_services(component, services): ) +def create_gitea_runtime_secret(path, label): + if path.exists() or path.is_symlink(): + die(f"Gitea {label} secret already exists during fresh install") + value = secrets.token_urlsafe(64) + if not GITEA_SECRET_RE.fullmatch(value): + die(f"Gitea {label} secret generation failed") + temporary = path.with_name( + f".{path.name}.{os.getpid()}.{time.time_ns()}.tmp" + ) + descriptor = None + try: + descriptor = os.open( + str(temporary), + os.O_WRONLY | os.O_CREAT | os.O_EXCL, + 0o400, + ) + os.write(descriptor, f"{value}\n".encode("ascii")) + os.fsync(descriptor) + os.fchown(descriptor, GITEA_RUNTIME_UID, GITEA_RUNTIME_GID) + os.fchmod(descriptor, 0o400) + os.close(descriptor) + descriptor = None + os.replace(temporary, path) + fsync_directory(path.parent) + finally: + if descriptor is not None: + os.close(descriptor) + if temporary.exists(): + temporary.unlink() + + +def prepare_gitea_fresh_runtime(entries): + if not is_gitea_fresh_install_slice("gitea", entries): + die("Gitea runtime preparation called for unrelated artifact") + compose = GITEA_ROOT / GITEA_COMPOSE_REL + descriptor_path = GITEA_ROOT / GITEA_FRESH_INSTALL_DESCRIPTOR_REL + if ( + compose.is_symlink() + or not compose.is_file() + or sha256_file(compose) != GITEA_COMPOSE_SHA256 + ): + die("installed Gitea fresh-install Compose source mismatch") + descriptor = read_strict_json( + descriptor_path, + "installed Gitea fresh-install descriptor", + max_bytes=16 * 1024, + ) + if descriptor != expected_gitea_fresh_install_descriptor(): + die("installed Gitea fresh-install descriptor mismatch") + + root_stat = GITEA_ROOT.lstat() + if ( + stat.S_ISLNK(root_stat.st_mode) + or not stat.S_ISDIR(root_stat.st_mode) + or root_stat.st_uid != 0 + ): + die("Gitea fresh-install root is unsafe") + os.chown(GITEA_ROOT, 0, 0) + GITEA_ROOT.chmod(0o755) + for directory, uid, gid, mode, label in ( + ( + GITEA_DATA_DIR, + GITEA_RUNTIME_UID, + GITEA_RUNTIME_GID, + 0o750, + "data", + ), + ( + GITEA_CONFIG_DIR, + GITEA_RUNTIME_UID, + GITEA_RUNTIME_GID, + 0o750, + "config", + ), + ( + GITEA_SOCKET_DIR, + GITEA_RUNTIME_UID, + GITEA_NGINX_GID, + 0o750, + "socket", + ), + (GITEA_SECRET_DIR, 0, GITEA_RUNTIME_GID, 0o710, "secret"), + ): + if directory.exists() or directory.is_symlink(): + die(f"Gitea fresh-install {label} directory already exists") + directory.mkdir(parents=False, exist_ok=False) + os.chown(directory, uid, gid) + directory.chmod(mode) + create_gitea_runtime_secret(GITEA_SECRET_KEY_FILE, "global") + create_gitea_runtime_secret(GITEA_INTERNAL_TOKEN_FILE, "internal token") + + def prepare_component_runtime(component, entries=None): + if is_gitea_fresh_install_slice(component, entries): + prepare_gitea_fresh_runtime(entries) + return + if is_gitea_incident_salvage_slice(component, entries): + die( + "Gitea incident-salvage activation is frozen before candidate " + "root creation until every disposition verifier is implemented" + ) + if is_engine_data_product_publish_grant_slice(component, entries): ensure_engine_edp_managed_provisioner_keypair() ensure_engine_publish_grant_private_state() @@ -22765,7 +31232,11 @@ def run_component_runtime(component, entries, services): return run_build(component, entries) prepare_component_runtime(component, entries) - if is_engine_node_intelligence_transition(component, entries): + if is_gitea_fresh_install_slice(component, entries): + run_gitea_fresh_install_compose(services, entries) + elif is_gitea_incident_salvage_slice(component, entries): + die("Gitea incident-salvage runtime is not activation-ready") + elif is_engine_node_intelligence_transition(component, entries): run_engine_node_intelligence_compose(services, entries) else: run_compose(component, services, entries) @@ -22869,6 +31340,128 @@ def assert_loopback_tcp_port_open(port): connection.close() +def healthcheck_gitea_json(): + url = "http://127.0.0.1:3000/api/healthz" + last_error = None + for attempt in range(1, 61): + try: + request = urllib.request.Request( + url, + headers={"Accept": "application/json"}, + ) + with NO_REDIRECT_OPENER.open(request, timeout=10) as response: + raw = response.read(64 * 1024 + 1) + if not (200 <= response.status < 300): + last_error = f"HTTP {response.status}" + elif len(raw) > 64 * 1024: + last_error = "health response too large" + else: + try: + payload = json.loads(raw.decode("utf-8")) + except (UnicodeDecodeError, json.JSONDecodeError): + last_error = "health response is not json" + else: + database = ( + payload.get("checks", {}).get("database:ping") + if isinstance(payload, dict) + else None + ) + if ( + isinstance(payload, dict) + and payload.get("status") == "pass" + and isinstance(database, list) + and len(database) == 1 + and isinstance(database[0], dict) + and database[0].get("status") == "pass" + ): + return payload + last_error = "health response contract mismatch" + except urllib.error.HTTPError as exc: + last_error = f"HTTP {exc.code}" + except Exception as exc: + last_error = str(exc) + if attempt < 60: + time.sleep(5) + die(f"Gitea healthcheck failed for {url}: {last_error}") + + +def validate_gitea_socket_boundary(): + try: + parent_stat = GITEA_SOCKET_DIR.lstat() + socket_stat = GITEA_SOCKET_FILE.lstat() + except FileNotFoundError: + die("Gitea Unix socket boundary is missing") + if ( + GITEA_SOCKET_FILE.parent != GITEA_SOCKET_DIR + or stat.S_ISLNK(parent_stat.st_mode) + or not stat.S_ISDIR(parent_stat.st_mode) + or parent_stat.st_uid != GITEA_RUNTIME_UID + or parent_stat.st_gid != GITEA_NGINX_GID + or stat.S_IMODE(parent_stat.st_mode) != 0o750 + or not stat.S_ISSOCK(socket_stat.st_mode) + or socket_stat.st_uid != GITEA_RUNTIME_UID + or socket_stat.st_gid != GITEA_RUNTIME_GID + or stat.S_IMODE(socket_stat.st_mode) != 0o666 + ): + die("Gitea Unix socket identity/metadata mismatch") + return str(GITEA_SOCKET_FILE) + + +def healthcheck_gitea_uds_json(): + last_error = None + for attempt in range(1, 61): + connection = None + response = None + try: + connection = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) + connection.settimeout(10) + connection.connect(str(GITEA_SOCKET_FILE)) + connection.sendall( + b"GET /api/healthz HTTP/1.1\r\n" + b"Host: git.dcserve.ru\r\n" + b"Accept: application/json\r\n" + b"Connection: close\r\n\r\n" + ) + response = http.client.HTTPResponse(connection) + response.begin() + raw = response.read(64 * 1024 + 1) + if not (200 <= response.status < 300): + last_error = f"HTTP {response.status}" + elif len(raw) > 64 * 1024: + last_error = "health response too large" + else: + try: + payload = json.loads(raw.decode("utf-8")) + except (UnicodeDecodeError, json.JSONDecodeError): + last_error = "health response is not json" + else: + database = ( + payload.get("checks", {}).get("database:ping") + if isinstance(payload, dict) + else None + ) + if ( + isinstance(payload, dict) + and payload.get("status") == "pass" + and isinstance(database, list) + and len(database) == 1 + and isinstance(database[0], dict) + and database[0].get("status") == "pass" + ): + return payload + last_error = "health response contract mismatch" + except (OSError, http.client.HTTPException) as exc: + last_error = str(exc) + finally: + if response is not None: + response.close() + if connection is not None: + connection.close() + if attempt < 60: + time.sleep(5) + die(f"Gitea UDS healthcheck failed: {last_error}") + + def external_data_plane_healthcheck(require_managed=True): expected_json = { "ok": True, @@ -23311,6 +31904,395 @@ def container_environment(container, label): return environment +def validate_gitea_runtime_secret_metadata(): + try: + secret_dir_stat = GITEA_SECRET_DIR.lstat() + except FileNotFoundError: + die("Gitea runtime secret directory is missing") + if ( + stat.S_ISLNK(secret_dir_stat.st_mode) + or not stat.S_ISDIR(secret_dir_stat.st_mode) + or secret_dir_stat.st_uid != 0 + or secret_dir_stat.st_gid != GITEA_RUNTIME_GID + or stat.S_IMODE(secret_dir_stat.st_mode) != 0o710 + ): + die("Gitea runtime secret directory metadata mismatch") + values = [] + for path, label in ( + (GITEA_SECRET_KEY_FILE, "global"), + (GITEA_INTERNAL_TOKEN_FILE, "internal token"), + ): + try: + path_stat = path.lstat() + value = path.read_text(encoding="ascii").strip() + except (FileNotFoundError, UnicodeDecodeError): + die(f"Gitea {label} secret is unreadable") + if ( + stat.S_ISLNK(path_stat.st_mode) + or not stat.S_ISREG(path_stat.st_mode) + or path_stat.st_uid != GITEA_RUNTIME_UID + or path_stat.st_gid != GITEA_RUNTIME_GID + or stat.S_IMODE(path_stat.st_mode) != 0o400 + or path_stat.st_size > 512 + or not GITEA_SECRET_RE.fullmatch(value) + ): + die(f"Gitea {label} secret metadata mismatch") + values.append(value) + if len(set(values)) != len(values): + die("Gitea runtime secrets must be distinct") + return tuple(values) + + +def validate_gitea_runtime_directory_metadata(): + for path, uid, gid, mode, label in ( + (GITEA_ROOT, 0, 0, 0o755, "root"), + (GITEA_DATA_DIR, GITEA_RUNTIME_UID, GITEA_RUNTIME_GID, 0o750, "data"), + ( + GITEA_CONFIG_DIR, + GITEA_RUNTIME_UID, + GITEA_RUNTIME_GID, + 0o750, + "config", + ), + ( + GITEA_SOCKET_DIR, + GITEA_RUNTIME_UID, + GITEA_NGINX_GID, + 0o750, + "socket", + ), + ): + try: + path_stat = path.lstat() + except FileNotFoundError: + die(f"Gitea runtime {label} directory is missing") + if ( + stat.S_ISLNK(path_stat.st_mode) + or not stat.S_ISDIR(path_stat.st_mode) + or path_stat.st_uid != uid + or path_stat.st_gid != gid + or stat.S_IMODE(path_stat.st_mode) != mode + ): + die(f"Gitea runtime {label} directory metadata mismatch") + + +def parse_gitea_app_ini_explicit(raw): + allowed_root_keys = {"APP_NAME", "RUN_USER", "RUN_MODE", "WORK_PATH"} + root_values = {} + sections = {} + section_names = {} + current = None + for line_number, raw_line in enumerate(raw.splitlines(), start=1): + line = raw_line.strip() + if not line or line.startswith(("#", ";")): + continue + section_match = re.fullmatch(r"\[([A-Za-z0-9_.-]+)\]", line) + if section_match: + section = section_match.group(1) + normalized = section.casefold() + if normalized == "default" or normalized in section_names: + die("Gitea installed app.ini contains duplicate/unsafe section") + section_names[normalized] = section + sections[section] = {} + current = section + continue + setting_match = re.fullmatch(r"([A-Za-z0-9_.-]+)\s*=\s*(.*)", line) + if setting_match is None: + die(f"Gitea installed app.ini syntax mismatch at line {line_number}") + key, value = setting_match.groups() + destination = root_values if current is None else sections[current] + normalized_key = key.casefold() + if any(existing.casefold() == normalized_key for existing in destination): + die("Gitea installed app.ini contains duplicate option") + if current is None and key not in allowed_root_keys: + die("Gitea installed app.ini contains unsafe root-level option") + destination[key] = value.strip() + if not sections: + die("Gitea installed app.ini has no explicit sections") + return root_values, sections + + +def validate_gitea_installed_config(secret_values): + app_ini = GITEA_CONFIG_DIR / "app.ini" + try: + app_ini_stat = app_ini.lstat() + raw = app_ini.read_text(encoding="utf-8") + except (FileNotFoundError, UnicodeDecodeError): + die("Gitea installed app.ini is unreadable") + if ( + stat.S_ISLNK(app_ini_stat.st_mode) + or not stat.S_ISREG(app_ini_stat.st_mode) + or app_ini_stat.st_uid != GITEA_RUNTIME_UID + or app_ini_stat.st_mode & (stat.S_IWGRP | stat.S_IWOTH) + or app_ini_stat.st_size < 32 + or app_ini_stat.st_size > 1024 * 1024 + ): + die("Gitea installed app.ini metadata mismatch") + _root_values, sections = parse_gitea_app_ini_explicit(raw) + expected = { + ("database", "DB_TYPE"): "sqlite3", + ("database", "PATH"): "/var/lib/gitea/data/gitea.db", + ("server", "ROOT_URL"): "https://git.dcserve.ru/", + ("server", "PROTOCOL"): "http+unix", + ("server", "HTTP_ADDR"): "/run/gitea/gitea.sock", + ("server", "UNIX_SOCKET_PERMISSION"): "0666", + ("server", "LOCAL_ROOT_URL"): "http://unix/", + ("server", "DISABLE_SSH"): "true", + ("server", "START_SSH_SERVER"): "false", + ("server", "LFS_START_SERVER"): "false", + ("server", "LFS_ALLOW_PURE_SSH"): "false", + ("security", "INSTALL_LOCK"): "true", + ("security", "SECRET_KEY_URI"): "file:/run/secrets/gitea_secret_key", + ("security", "INTERNAL_TOKEN_URI"): ( + "file:/run/secrets/gitea_internal_token" + ), + ("security", "REVERSE_PROXY_LIMIT"): "1", + ("security", "REVERSE_PROXY_TRUSTED_PROXIES"): ( + "127.0.0.0/8,::1/128" + ), + ("security", "DISABLE_GIT_HOOKS"): "true", + ("security", "DISABLE_WEBHOOKS"): "true", + ("security", "IMPORT_LOCAL_PATHS"): "false", + ("security", "TWO_FACTOR_AUTH"): "enforced", + ("security", "DISABLE_QUERY_AUTH_TOKEN"): "true", + ("security", "ALLOWED_HOST_LIST"): "loopback", + ("service", "DISABLE_REGISTRATION"): "true", + ("service", "REQUIRE_SIGNIN_VIEW"): "true", + ("service", "DEFAULT_USER_IS_RESTRICTED"): "true", + ("service", "ENABLE_REVERSE_PROXY_AUTHENTICATION"): "false", + ("service", "ENABLE_REVERSE_PROXY_AUTHENTICATION_API"): "false", + ("service", "ENABLE_REVERSE_PROXY_AUTO_REGISTRATION"): "false", + ("service", "ENABLE_BASIC_AUTHENTICATION"): "false", + ("admin", "DISABLE_REGULAR_ORG_CREATION"): "true", + ("admin", "USER_DISABLED_FEATURES"): ( + "deletion,manage_ssh_keys,manage_gpg_keys,change_username" + ), + ("repository", "FORCE_PRIVATE"): "true", + ("repository", "USER_MAX_CREATION_LIMIT"): "0", + ("repository", "ORG_MAX_CREATION_LIMIT"): "0", + ("repository", "DISABLE_MIGRATIONS"): "true", + ("actions", "ENABLED"): "false", + ("packages", "ENABLED"): "false", + ("oauth2", "ENABLED"): "false", + ("openid", "ENABLE_OPENID_SIGNIN"): "false", + ("openid", "ENABLE_OPENID_SIGNUP"): "false", + ("federation", "ENABLED"): "false", + } + for (section, key), value in expected.items(): + if sections.get(section, {}).get(key) != value: + die(f"Gitea installed config mismatch: {section}.{key}") + for section, key in ( + ("security", "SECRET_KEY"), + ("security", "INTERNAL_TOKEN"), + ("server", "LFS_JWT_SECRET"), + ("server", "LFS_JWT_SECRET_URI"), + ): + if (sections.get(section, {}).get(key, "") or "").strip(): + die(f"Gitea installed config contains plaintext secret: {section}.{key}") + if any(value in raw for value in secret_values): + die("Gitea installed config contains runner-managed secret bytes") + + +def validate_gitea_fresh_sqlite(): + database = GITEA_DATA_DIR / "data" / "gitea.db" + try: + database_stat = database.lstat() + except FileNotFoundError: + die("Gitea fresh SQLite database is missing") + if ( + stat.S_ISLNK(database_stat.st_mode) + or not stat.S_ISREG(database_stat.st_mode) + or database_stat.st_uid != GITEA_RUNTIME_UID + or database_stat.st_mode & (stat.S_IWGRP | stat.S_IWOTH) + or database_stat.st_size < 4096 + ): + die("Gitea fresh SQLite database metadata mismatch") + connection = None + try: + connection = sqlite3.connect( + f"file:{database}?mode=ro", + uri=True, + timeout=10, + ) + connection.execute("PRAGMA query_only = ON") + integrity = connection.execute("PRAGMA quick_check(1)").fetchone() + tables = { + row[0] + for row in connection.execute( + "SELECT name FROM sqlite_master WHERE type = 'table'" + ) + } + if integrity != ("ok",) or not {"user", "repository"}.issubset(tables): + die("Gitea fresh SQLite schema/integrity mismatch") + counts = { + table: connection.execute( + f'SELECT COUNT(*) FROM "{table}"' + ).fetchone()[0] + for table in ("user", "repository") + } + except sqlite3.Error as exc: + die(f"Gitea fresh SQLite acceptance failed: {exc}") + finally: + if connection is not None: + connection.close() + if counts != {"user": 0, "repository": 0}: + die("Gitea fresh SQLite unexpectedly contains users or repositories") + return counts + + +def accept_gitea_fresh_install_runtime(): + container_id = compose_service_container_id("gitea", GITEA_SERVICE) + healthcheck_container(container_id) + validate_gitea_socket_boundary() + healthcheck_gitea_uds_json() + nginx_bridge = validate_gitea_nginx_bridge_prerequisite() + healthcheck_gitea_json() + validate_gitea_no_docker_port_publications() + assert_loopback_tcp_port_closed(GITEA_DISABLED_SSH_HOST_PORT) + legacy_container = validate_legacy_gitea_container_isolation() + + image_id = inspect_gitea_local_image() + container = inspect_container(container_id) + config = container.get("Config") or {} + host = container.get("HostConfig") or {} + labels = config.get("Labels") or {} + if ( + container.get("Image") != image_id + or config.get("Image") != GITEA_IMAGE + or config.get("User") != "1000:1000" + or labels.get("com.docker.compose.project") != GITEA_COMPOSE_PROJECT + or labels.get("com.docker.compose.service") != GITEA_SERVICE + ): + die("Gitea runtime identity mismatch") + environment = container_environment(container, "Gitea runtime") + for key, expected in GITEA_EXPECTED_ENVIRONMENT.items(): + if environment.get(key) != expected: + die(f"Gitea runtime environment mismatch: {key}") + for forbidden_key in ( + "GITEA__security__SECRET_KEY", + "GITEA__security__INTERNAL_TOKEN", + "GITEA__server__LFS_JWT_SECRET", + "GITEA__server__LFS_JWT_SECRET_URI", + "GITEA__security__SECRET_KEY__FILE", + "GITEA__security__INTERNAL_TOKEN__FILE", + "GITEA__server__LFS_JWT_SECRET__FILE", + ): + if forbidden_key in environment: + die(f"Gitea runtime plaintext secret environment rejected: {forbidden_key}") + + if ( + host.get("Privileged") is not False + or host.get("ReadonlyRootfs") is not True + or host.get("NetworkMode") != "none" + or (host.get("RestartPolicy") or {}).get("Name") != "unless-stopped" + or host.get("PidsLimit") != 512 + or host.get("Init") is not True + or config.get("StopTimeout") != 30 + or "ALL" not in set(host.get("CapDrop") or ()) + or not any( + value in ("no-new-privileges", "no-new-privileges:true") + for value in (host.get("SecurityOpt") or ()) + ) + ): + die("Gitea runtime hardening mismatch") + if host.get("LogConfig") != { + "Type": "json-file", + "Config": {"max-file": "3", "max-size": "10m"}, + }: + die("Gitea runtime bounded logging mismatch") + if (host.get("PortBindings") or {}) != {}: + die("Gitea runtime port publication mismatch") + tmpfs = host.get("Tmpfs") or {} + tmpfs_options = tmpfs.get("/tmp") or "" + if not all( + option in tmpfs_options + for option in ("rw", "noexec", "nosuid", "nodev", "size=268435456") + ): + die("Gitea runtime tmpfs boundary mismatch") + + expected_bind_mounts = { + (str(GITEA_DATA_DIR), "/var/lib/gitea", True), + (str(GITEA_CONFIG_DIR), "/etc/gitea", True), + (str(GITEA_SECRET_KEY_FILE), "/run/secrets/gitea_secret_key", False), + ( + str(GITEA_INTERNAL_TOKEN_FILE), + "/run/secrets/gitea_internal_token", + False, + ), + (str(GITEA_SOCKET_DIR), "/run/gitea", True), + } + bind_mounts = { + (mount.get("Source"), mount.get("Destination"), mount.get("RW")) + for mount in (container.get("Mounts") or ()) + if mount.get("Type") == "bind" + } + if bind_mounts != expected_bind_mounts: + die("Gitea runtime mount boundary mismatch") + + network_settings = container.get("NetworkSettings") or {} + attached_networks = network_settings.get("Networks") or {} + if set(attached_networks) != {"none"}: + die("Gitea runtime must not attach to a Docker network") + none_network = attached_networks["none"] + builtin_none = inspect_gitea_builtin_none_network() + if ( + not isinstance(none_network, dict) + or none_network.get("NetworkID") != builtin_none["Id"] + or not re.fullmatch( + r"[a-f0-9]{64}", + str(none_network.get("EndpointID") or ""), + ) + or any( + none_network.get(key) not in (None, "", 0, [], {}) + for key in ( + "IPAMConfig", + "Links", + "Aliases", + "DriverOpts", + "GwPriority", + "Gateway", + "IPAddress", + "MacAddress", + "IPPrefixLen", + "IPv6Gateway", + "GlobalIPv6Address", + "GlobalIPv6PrefixLen", + "DNSNames", + ) + ) + ): + die("Gitea Docker none-network endpoint is not isolated") + full_container_id = container.get("Id") + none_container = (builtin_none.get("Containers") or {}).get(full_container_id) + if ( + not isinstance(full_container_id, str) + or not re.fullmatch(r"[a-f0-9]{64}", full_container_id) + or not isinstance(none_container, dict) + or none_container.get("EndpointID") != none_network.get("EndpointID") + or any( + none_container.get(key) not in (None, "") + for key in ("MacAddress", "IPv4Address", "IPv6Address") + ) + ): + die("Gitea built-in none network membership mismatch") + validate_gitea_legacy_candidate_network_absent() + + validate_gitea_runtime_directory_metadata() + secret_values = validate_gitea_runtime_secret_metadata() + validate_gitea_installed_config(secret_values) + database_counts = validate_gitea_fresh_sqlite() + return { + "container_id": container_id, + "image_id": image_id, + "transport": str(GITEA_SOCKET_FILE), + "network_mode": "none", + "nginx_bridge": nginx_bridge, + "database": "fresh-sqlite", + "database_counts": database_counts, + "legacy_container": legacy_container, + } + + def validate_engine_node_intelligence_secret_mount(container, label): matches = [ mount @@ -23557,6 +32539,12 @@ process.stdout.write('engine-l2-closed-loop:0.7.0:cas+safe-profile+external-plan def run_healthchecks(component, entries=None, services=None): + if is_gitea_fresh_install_slice(component, entries): + if tuple(services or ()) != (GITEA_SERVICE,): + die("Gitea fresh-install service set mismatch") + accept_gitea_fresh_install_runtime() + return + if is_device_plane_control_core_release_slice(component, entries): if tuple(services or ()) != ("device-control-core",): die("Device Control Core release service set mismatch") @@ -24547,6 +33535,20 @@ def apply_artifact(artifact): with tempfile.TemporaryDirectory(prefix=f"apply-{current_stamp}-", dir=TMP_DIR) as tmp: work = Path(tmp) manifest, entries, payload_dir = load_artifact(artifact, work) + if is_gitea_fresh_install_slice( + manifest["component"], + entries, + ): + validate_gitea_compose_schema( + payload_dir / GITEA_COMPOSE_REL + ) + if is_gitea_incident_salvage_slice( + manifest["component"], + entries, + ): + validate_gitea_compose_schema( + payload_dir / GITEA_COMPOSE_REL + ) reject_failed_artifact_replay(manifest, sha) reject_terminal_engine_l2_failed_artifact(manifest, sha) reject_terminal_device_plane_foundation_artifact( @@ -24569,11 +33571,22 @@ def apply_artifact(artifact): bootstrap_root = bool(COMPONENTS[component].get("bootstrap_root")) services = component_services(component, entries) artifact_only = component_artifact_only(component) + defer_bootstrap_root = is_gitea_bootstrap_slice( + component, + entries, + ) if state_has_sha(sha): die(f"artifact sha already applied: {sha}") if state_has_patch_id(patch_id): die(f"patch id already applied: {patch_id}") + if is_gitea_fresh_install_slice(component, entries): + preflight_gitea_fresh_install() + if is_gitea_incident_salvage_slice(component, entries): + preflight_gitea_incident_salvage( + payload_dir, + enforce_apply=True, + ) if is_device_plane_postgres_bootstrap_slice( component, entries, @@ -24663,7 +33676,12 @@ def apply_artifact(artifact): ) ) if not root.is_dir(): - if bootstrap_root: + if bootstrap_root and defer_bootstrap_root: + # The fresh-install root itself is a mutation. Delay it + # until after backup and apply_started so every created + # path is covered by Gitea's automatic rollback. + pass + elif bootstrap_root: root.mkdir(parents=True, exist_ok=True) else: die(f"component payload root not found: {root}") @@ -24906,7 +33924,9 @@ def apply_artifact(artifact): if is_engine_provider_security_catalog_slice(component, entries): preflight_engine_provider_security_catalog_predecessor() if not artifact_only and not compose_root.is_dir(): - if bootstrap_root and is_relative_to(compose_root.resolve(strict=False), root.resolve(strict=False)): + if bootstrap_root and defer_bootstrap_root: + pass + elif bootstrap_root and is_relative_to(compose_root.resolve(strict=False), root.resolve(strict=False)): compose_root.mkdir(parents=True, exist_ok=True) else: die(f"component compose root not found: {compose_root}") @@ -25165,7 +34185,34 @@ def apply_artifact(artifact): and backup_dir is not None and root is not None ): - if transition_descriptor is not None and is_engine_n8n_transition(component, entries): + if is_gitea_fresh_install_slice(component, entries): + try: + restored_action = rollback_gitea_fresh_install( + root, + backup_dir, + entries, + current_stamp, + runtime_started=runtime_started, + ) + rollback_status = f"ok:gitea-fresh-install:{restored_action}" + print( + f"gitea-automatic-rollback={rollback_status}", + file=sys.stderr, + ) + except ReconciliationRequired as rollback_exc: + rollback_status = "deferred:reconciliation-required" + print( + "gitea-automatic-rollback=deferred:" + f"{rollback_exc}", + file=sys.stderr, + ) + except Exception as rollback_exc: + rollback_status = f"failed:{type(rollback_exc).__name__}" + print( + "gitea-automatic-rollback=failed", + file=sys.stderr, + ) + elif transition_descriptor is not None and is_engine_n8n_transition(component, entries): try: restored_action = restore_engine_n8n_transition( root, diff --git a/infra/deploy-runner/test_gitea_fresh_install.py b/infra/deploy-runner/test_gitea_fresh_install.py new file mode 100644 index 0000000..a55272c --- /dev/null +++ b/infra/deploy-runner/test_gitea_fresh_install.py @@ -0,0 +1,1038 @@ +#!/usr/bin/env python3 +import configparser +import hashlib +import importlib.machinery +import importlib.util +import inspect +import json +import os +import sqlite3 +import stat +import subprocess +import tarfile +import tempfile +import unittest +from pathlib import Path +from types import SimpleNamespace +from unittest import mock + + +SCRIPT_DIR = Path(__file__).resolve().parent +RUNNER_PATH = SCRIPT_DIR / "nodedc-deploy" +BUILDER_PATH = SCRIPT_DIR / "build-gitea-fresh-install-artifact.mjs" +FIXTURE_ROOT = SCRIPT_DIR / "fixtures" / "gitea" +COMPOSE_PATH = FIXTURE_ROOT / "docker-compose.gitea.yml" +DESCRIPTOR_PATH = ( + FIXTURE_ROOT / "deployment" / "gitea-fresh-install-v1.json" +) + + +def load_runner(): + loader = importlib.machinery.SourceFileLoader( + "nodedc_gitea_deploy_under_test", + str(RUNNER_PATH), + ) + spec = importlib.util.spec_from_loader(loader.name, loader) + module = importlib.util.module_from_spec(spec) + loader.exec_module(module) + return module + + +RUNNER = load_runner() + + +def completed(returncode=0, stdout="", stderr=""): + return subprocess.CompletedProcess([], returncode, stdout, stderr) + + +class GiteaFreshInstallArtifactTest(unittest.TestCase): + def test_fixture_and_descriptor_are_digest_bound_and_hardened(self): + compose = COMPOSE_PATH.read_text(encoding="utf-8") + descriptor = json.loads(DESCRIPTOR_PATH.read_text(encoding="utf-8")) + digest = hashlib.sha256(COMPOSE_PATH.read_bytes()).hexdigest() + self.assertEqual(digest, RUNNER.GITEA_COMPOSE_SHA256) + self.assertEqual(descriptor, RUNNER.expected_gitea_fresh_install_descriptor()) + self.assertIn(f"image: {RUNNER.GITEA_IMAGE}", compose) + for required in ( + "platform: linux/amd64", + "pull_policy: never", + "network_mode: none", + "GITEA__server__PROTOCOL: http+unix", + "GITEA__server__HTTP_ADDR: /run/gitea/gitea.sock", + 'GITEA__server__UNIX_SOCKET_PERMISSION: "0666"', + "GITEA__server__LOCAL_ROOT_URL: http://unix/", + "source: /volume1/docker/nodedc-gitea/socket", + "target: /run/gitea", + "create_host_path: false", + "stop_grace_period: 30s", + "driver: json-file", + 'max-size: "10m"', + 'max-file: "3"', + "GITEA__security__SECRET_KEY_URI: file:/run/secrets/gitea_secret_key", + "GITEA__security__INTERNAL_TOKEN_URI: file:/run/secrets/gitea_internal_token", + "GITEA__security__TWO_FACTOR_AUTH: enforced", + 'GITEA__server__LFS_START_SERVER: "false"', + 'GITEA__server__LFS_ALLOW_PURE_SSH: "false"', + "GITEA__security__ALLOWED_HOST_LIST: loopback", + 'GITEA__service__ENABLE_BASIC_AUTHENTICATION: "false"', + 'GITEA__repository__DISABLE_MIGRATIONS: "true"', + 'GITEA__packages__ENABLED: "false"', + ): + self.assertIn(required, compose) + for forbidden in ( + "__FILE", + "4022", + "0.0.0.0:3000", + "ports:", + "networks:", + "/var/run/docker.sock", + "/volume1/docker/gitea", + "privileged: true", + "GITEA__server__LFS_JWT_SECRET_URI", + "gitea_lfs_jwt_secret", + "lfs-jwt-secret", + "GITEA__server__REVERSE_PROXY_LIMIT", + "GITEA__security__ENABLE_REVERSE_PROXY_AUTHENTICATION", + "GITEA__service__DISABLE_REGULAR_ORG_CREATION", + ): + self.assertNotIn(forbidden, compose) + + def test_compose_schema_is_accepted_without_pull_or_start(self): + result = subprocess.run( + [ + "docker", + "compose", + "--project-name", + "nodedc-gitea-fixture-test", + "--file", + str(COMPOSE_PATH), + "config", + "--quiet", + ], + check=False, + capture_output=True, + text=True, + ) + if result.returncode == 127: + self.skipTest("docker compose is unavailable") + self.assertEqual(result.returncode, 0, result.stderr) + + def test_builder_is_deterministic_and_archive_is_data_only(self): + with tempfile.TemporaryDirectory(prefix="nodedc-gitea-builder-") as directory: + environment = dict(os.environ) + environment["NODEDC_DEPLOY_ARTIFACT_DIR"] = directory + command = ["node", str(BUILDER_PATH), "gitea-test-001"] + first = subprocess.run( + command, + env=environment, + check=True, + capture_output=True, + text=True, + ) + artifact = Path(json.loads(first.stdout)["artifact"]) + first_digest = hashlib.sha256(artifact.read_bytes()).hexdigest() + second = subprocess.run( + command, + env=environment, + check=True, + capture_output=True, + text=True, + ) + self.assertEqual( + first_digest, + hashlib.sha256(artifact.read_bytes()).hexdigest(), + ) + self.assertEqual(json.loads(second.stdout)["sha256"], first_digest) + with tarfile.open(artifact, "r:gz") as archive: + names = archive.getnames() + self.assertEqual( + names, + [ + "manifest.env", + "files.txt", + "payload", + "payload/deployment", + "payload/deployment/gitea-fresh-install-v1.json", + "payload/docker-compose.gitea.yml", + ], + ) + manifest = archive.extractfile("manifest.env").read().decode() + files = archive.extractfile("files.txt").read().decode() + self.assertEqual( + manifest, + "id=gitea-test-001\ncomponent=gitea\ntype=app-overlay\n", + ) + self.assertEqual( + files, + "docker-compose.gitea.yml\n" + "deployment/gitea-fresh-install-v1.json\n", + ) + + def test_payload_validator_accepts_exact_fixture_and_rejects_mutation(self): + with tempfile.TemporaryDirectory(prefix="nodedc-gitea-payload-") as directory: + payload = Path(directory) + (payload / "deployment").mkdir() + (payload / RUNNER.GITEA_COMPOSE_REL).write_bytes(COMPOSE_PATH.read_bytes()) + (payload / RUNNER.GITEA_FRESH_INSTALL_DESCRIPTOR_REL).write_bytes( + DESCRIPTOR_PATH.read_bytes() + ) + RUNNER.validate_gitea_fresh_install_payload( + payload, + RUNNER.GITEA_FRESH_INSTALL_ENTRIES, + ) + (payload / RUNNER.GITEA_COMPOSE_REL).write_text( + COMPOSE_PATH.read_text(encoding="utf-8") + "\n# mutation\n", + encoding="utf-8", + ) + with self.assertRaisesRegex( + RUNNER.DeployError, + "Compose digest mismatch", + ): + RUNNER.validate_gitea_fresh_install_payload( + payload, + RUNNER.GITEA_FRESH_INSTALL_ENTRIES, + ) + + +class GiteaFreshInstallRunnerTest(unittest.TestCase): + def test_registry_and_payload_boundary_are_exact(self): + component = RUNNER.COMPONENTS["gitea"] + self.assertEqual(component["payload_root"], RUNNER.GITEA_ROOT) + self.assertEqual(component["compose_project"], "nodedc-gitea") + self.assertEqual(component["services"], ("gitea",)) + self.assertTrue(component["bootstrap_root"]) + self.assertTrue(component["compose_no_deps"]) + for allowed in RUNNER.GITEA_FRESH_INSTALL_ENTRIES: + self.assertTrue(RUNNER.allowed_payload_path("gitea", allowed)) + for rejected in ( + ".env", + "data/gitea.db", + "config/app.ini", + "secrets/secret-key", + "repositories/org/repo.git", + "users/export.json", + "tokens/api-token", + "hooks/post-receive", + "docker-compose.yml", + ): + with self.assertRaises(RUNNER.DeployError): + RUNNER.allowed_payload_path("gitea", rejected) + + def test_runner_has_no_legacy_root_path_object_or_runtime_mount(self): + source = inspect.getsource(RUNNER.preflight_gitea_fresh_install) + prepare = inspect.getsource(RUNNER.prepare_gitea_fresh_runtime) + self.assertNotIn("GITEA_SALVAGE_LEGACY_ROOT", source) + self.assertNotIn("GITEA_SALVAGE_LEGACY_ROOT", prepare) + self.assertNotIn('source: /volume1/docker/gitea', COMPOSE_PATH.read_text()) + self.assertEqual(RUNNER.GITEA_SALVAGE_LEGACY_ROOT, Path("/volume1/docker/gitea")) + + def test_plan_and_apply_integrate_preflight_before_bootstrap(self): + plan_source = inspect.getsource(RUNNER.plan_artifact) + apply_source = inspect.getsource(RUNNER.apply_artifact) + self.assertIn("gitea_preflight = preflight_gitea_fresh_install()", plan_source) + preflight_index = apply_source.index("preflight_gitea_fresh_install()") + bootstrap_index = apply_source.index("if not root.is_dir()") + self.assertLess(preflight_index, bootstrap_index) + self.assertIn("defer_bootstrap_root = is_gitea_bootstrap_slice", apply_source) + self.assertLess( + apply_source.index("apply_started = True"), + apply_source.index("copy_payload_path(payload_dir, root, rel"), + ) + + def test_preflight_fails_closed_and_returns_exact_attestation(self): + with tempfile.TemporaryDirectory(prefix="nodedc-gitea-preflight-") as directory: + absent_root = Path(directory) / "absent" + with ( + mock.patch.object(RUNNER, "GITEA_ROOT", absent_root), + mock.patch.object( + RUNNER, + "gitea_compose_project_container_ids", + return_value=(), + ), + mock.patch.object( + RUNNER, + "validate_gitea_legacy_candidate_network_absent", + return_value="absent", + ), + mock.patch.object(RUNNER, "assert_loopback_tcp_port_closed") as port, + mock.patch.object( + RUNNER, + "validate_gitea_nginx_bridge_prerequisite", + return_value={"sha256": RUNNER.GITEA_NGINX_BRIDGE_SHA256}, + ), + mock.patch.object( + RUNNER, + "validate_gitea_reverse_proxy_prerequisite", + return_value={"upstream": "http://127.0.0.1:3000"}, + ), + mock.patch.object( + RUNNER, + "validate_gitea_firewall_prerequisite", + return_value={"legacy": "isolated"}, + ), + mock.patch.object( + RUNNER, + "validate_legacy_gitea_container_isolation", + return_value="stopped-restart-disabled", + ), + mock.patch.object( + RUNNER, + "validate_gitea_no_docker_port_publications", + return_value="none", + ), + mock.patch.object( + RUNNER, + "inspect_gitea_docker_server_version", + return_value="24.0.2", + ), + mock.patch.object( + RUNNER, + "inspect_gitea_builtin_none_network", + return_value={"Id": "d" * 64}, + ), + mock.patch.object( + RUNNER, + "inspect_gitea_docker_compose_version", + return_value="2.20.1-6047-g6817716", + ), + mock.patch.object( + RUNNER, + "inspect_gitea_local_image", + return_value=RUNNER.GITEA_IMAGE_ID, + ), + ): + result = RUNNER.preflight_gitea_fresh_install() + self.assertEqual(result["mode"], "fresh-root-absent") + self.assertEqual(result["image_id"], RUNNER.GITEA_IMAGE_ID) + self.assertEqual(result["legacy_container"], "stopped-restart-disabled") + self.assertEqual( + [call.args[0] for call in port.call_args_list], + [4022], + ) + + absent_root.mkdir() + with ( + mock.patch.object(RUNNER, "GITEA_ROOT", absent_root), + self.assertRaisesRegex( + RUNNER.DeployError, + "root already exists", + ), + ): + RUNNER.preflight_gitea_fresh_install() + + def test_local_image_gate_requires_exact_linux_amd64_rootless_user(self): + valid = [{ + "Id": RUNNER.GITEA_IMAGE_ID, + "RepoDigests": [RUNNER.GITEA_REPO_DIGEST], + "Architecture": "amd64", + "Os": "linux", + "Config": {"User": "1000:1000"}, + }] + with mock.patch.object(RUNNER, "docker_json", return_value=valid): + self.assertEqual( + RUNNER.inspect_gitea_local_image(), + RUNNER.GITEA_IMAGE_ID, + ) + invalid = json.loads(json.dumps(valid)) + invalid[0]["Config"]["User"] = "root" + with ( + mock.patch.object(RUNNER, "docker_json", return_value=invalid), + self.assertRaisesRegex(RUNNER.DeployError, "identity/repo-digest"), + ): + RUNNER.inspect_gitea_local_image() + wrong_identity = json.loads(json.dumps(valid)) + wrong_identity[0]["Id"] = "sha256:" + "b" * 64 + with ( + mock.patch.object(RUNNER, "docker_json", return_value=wrong_identity), + self.assertRaisesRegex(RUNNER.DeployError, "identity/repo-digest"), + ): + RUNNER.inspect_gitea_local_image() + wrong_digest = json.loads(json.dumps(valid)) + wrong_digest[0]["RepoDigests"] = ["docker.gitea.com/gitea@sha256:" + "c" * 64] + with ( + mock.patch.object(RUNNER, "docker_json", return_value=wrong_digest), + self.assertRaisesRegex(RUNNER.DeployError, "identity/repo-digest"), + ): + RUNNER.inspect_gitea_local_image() + + def test_docker_server_version_gate_is_exact(self): + with mock.patch.object( + RUNNER.subprocess, + "run", + return_value=completed(0, stdout="24.0.2\n"), + ): + self.assertEqual(RUNNER.inspect_gitea_docker_server_version(), "24.0.2") + for version in ("24.0.1", "24.0.2-synology", "25.0.0", ""): + with ( + mock.patch.object( + RUNNER.subprocess, + "run", + return_value=completed(0, stdout=version + "\n"), + ), + self.assertRaisesRegex(RUNNER.DeployError, "exact Docker Server"), + ): + RUNNER.inspect_gitea_docker_server_version() + + def test_firewall_gate_removes_only_broad_3000_drop_requirement(self): + fake_iptables = Path(__file__) + with ( + mock.patch.object(RUNNER, "GITEA_IPTABLES", fake_iptables), + mock.patch.object( + RUNNER.subprocess, + "run", + side_effect=[completed(1), completed(0), completed(0)], + ) as run, + ): + result = RUNNER.validate_gitea_firewall_prerequisite() + self.assertEqual(result["legacy_4022"], "input-drop-present") + commands = [call.args[0] for call in run.call_args_list] + self.assertIn( + [ + str(fake_iptables), "-w", "5", "-C", "INPUT", "-p", + "tcp", "--dport", "3000", "-j", "DROP", + ], + commands, + ) + self.assertTrue(any("172.22.0.222/32" in command for command in commands)) + + with ( + mock.patch.object(RUNNER, "GITEA_IPTABLES", fake_iptables), + mock.patch.object( + RUNNER.subprocess, + "run", + return_value=completed(0), + ), + self.assertRaisesRegex(RUNNER.DeployError, "still blocked"), + ): + RUNNER.validate_gitea_firewall_prerequisite() + + def test_legacy_candidate_network_absence_fails_closed_on_unexpected_error(self): + expected = completed( + 1, + stderr=f"Error: No such network: {RUNNER.GITEA_FORBIDDEN_LEGACY_NETWORK}\n", + ) + with mock.patch.object(RUNNER.subprocess, "run", return_value=expected): + self.assertEqual( + RUNNER.validate_gitea_legacy_candidate_network_absent(), + "absent", + ) + for result in ( + completed(0, stdout="[]\n"), + completed(1, stderr="permission denied\n"), + completed(2, stderr="daemon unavailable\n"), + ): + with ( + mock.patch.object(RUNNER.subprocess, "run", return_value=result), + self.assertRaises(RUNNER.DeployError), + ): + RUNNER.validate_gitea_legacy_candidate_network_absent() + + def test_builtin_none_network_identity_is_exact(self): + exact = [{ + "Name": "none", + "Id": "d" * 64, + "Scope": "local", + "Driver": "null", + "Internal": False, + "Attachable": False, + "Ingress": False, + "ConfigOnly": False, + "Containers": {}, + }] + with mock.patch.object(RUNNER, "docker_json", return_value=exact): + self.assertEqual( + RUNNER.inspect_gitea_builtin_none_network()["Id"], + "d" * 64, + ) + invalid = json.loads(json.dumps(exact)) + invalid[0]["Driver"] = "bridge" + with ( + mock.patch.object(RUNNER, "docker_json", return_value=invalid), + self.assertRaisesRegex(RUNNER.DeployError, "identity mismatch"), + ): + RUNNER.inspect_gitea_builtin_none_network() + + def test_reverse_proxy_gate_requires_exact_persistent_and_generated_rule(self): + exact = { + RUNNER.GITEA_REVERSE_PROXY_UUID: { + "backend": {"fqdn": "127.0.0.1", "port": 3000, "protocol": 0}, + "customize_headers": [], + "description": "Gittea", + "frontend": { + "acl": None, + "fqdn": "git.dcserve.ru", + "https": {"hsts": False}, + "port": 443, + "protocol": 1, + }, + "proxy_connect_timeout": 60, + "proxy_http_version": 1, + "proxy_intercept_errors": False, + "proxy_read_timeout": 60, + "proxy_send_timeout": 60, + } + } + generated = """ +server { + server_name git.dcserve.ru ; + if ( $host !~ "(^git.dcserve.ru$)" ) { return 404; } + location / { + proxy_set_header Host $http_host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_pass http://127.0.0.1:3000; + } +} +server { server_name other.example ; proxy_pass http://172.22.0.222:3000; } +""" + with tempfile.TemporaryDirectory(prefix="nodedc-gitea-proxy-") as directory: + root = Path(directory) + persistent = root / "ReverseProxy.json" + rendered = root / "gitea.w3conf" + persistent.write_text("{}", encoding="utf-8") + rendered.write_text(generated, encoding="utf-8") + stats = [ + SimpleNamespace( + st_mode=stat.S_IFREG | 0o644, + st_uid=0, + st_gid=0, + st_size=128, + ), + SimpleNamespace( + st_mode=stat.S_IFREG | 0o644, + st_uid=0, + st_gid=0, + st_size=len(generated), + ), + ] + with ( + mock.patch.object(RUNNER, "GITEA_REVERSE_PROXY_CONFIG", persistent), + mock.patch.object( + RUNNER, + "GITEA_REVERSE_PROXY_GENERATED_CONFIG", + rendered, + ), + mock.patch.object(RUNNER, "read_strict_json", return_value=exact), + mock.patch.object(Path, "lstat", side_effect=stats), + ): + result = RUNNER.validate_gitea_reverse_proxy_prerequisite() + self.assertEqual(result["upstream"], "http://127.0.0.1:3000") + + rendered.write_text( + generated.replace("$remote_addr;", "$http_x_real_ip;", 1), + encoding="utf-8", + ) + with ( + mock.patch.object(RUNNER, "GITEA_REVERSE_PROXY_CONFIG", persistent), + mock.patch.object( + RUNNER, + "GITEA_REVERSE_PROXY_GENERATED_CONFIG", + rendered, + ), + mock.patch.object(RUNNER, "read_strict_json", return_value=exact), + mock.patch.object(Path, "lstat", side_effect=stats), + self.assertRaisesRegex(RUNNER.DeployError, "prerequisite mismatch"), + ): + RUNNER.validate_gitea_reverse_proxy_prerequisite() + rendered.write_text(generated, encoding="utf-8") + + legacy = json.loads(json.dumps(exact)) + legacy[RUNNER.GITEA_REVERSE_PROXY_UUID]["backend"]["fqdn"] = ( + "172.22.0.222" + ) + with ( + mock.patch.object(RUNNER, "GITEA_REVERSE_PROXY_CONFIG", persistent), + mock.patch.object(RUNNER, "read_strict_json", return_value=legacy), + mock.patch.object(Path, "lstat", return_value=stats[0]), + self.assertRaisesRegex(RUNNER.DeployError, "prerequisite mismatch"), + ): + RUNNER.validate_gitea_reverse_proxy_prerequisite() + + def test_fresh_sqlite_gate_is_read_only_integrity_and_zero_counts(self): + with tempfile.TemporaryDirectory(prefix="nodedc-gitea-sqlite-") as directory: + data_root = Path(directory) + database = data_root / "data" / "gitea.db" + database.parent.mkdir() + connection = sqlite3.connect(database) + connection.executescript( + 'CREATE TABLE "user" (id INTEGER PRIMARY KEY);' + 'CREATE TABLE "repository" (id INTEGER PRIMARY KEY);' + ) + connection.commit() + connection.close() + metadata = SimpleNamespace( + st_mode=stat.S_IFREG | 0o600, + st_uid=1000, + st_size=max(database.stat().st_size, 4096), + ) + with ( + mock.patch.object(RUNNER, "GITEA_DATA_DIR", data_root), + mock.patch.object(Path, "lstat", return_value=metadata), + ): + self.assertEqual( + RUNNER.validate_gitea_fresh_sqlite(), + {"user": 0, "repository": 0}, + ) + connection = sqlite3.connect(database) + connection.execute('INSERT INTO "repository" DEFAULT VALUES') + connection.commit() + connection.close() + with ( + mock.patch.object(RUNNER, "GITEA_DATA_DIR", data_root), + mock.patch.object(Path, "lstat", return_value=metadata), + self.assertRaisesRegex(RUNNER.DeployError, "unexpectedly contains"), + ): + RUNNER.validate_gitea_fresh_sqlite() + + def test_nginx_uds_bridge_is_exact_and_listener_is_nginx_owned(self): + expected_fragments = ( + "listen 127.0.0.1:3000;", + "proxy_pass http://unix:/volume1/docker/nodedc-gitea/socket/gitea.sock:;", + "proxy_set_header Host $http_host;", + "proxy_set_header X-Real-IP $http_x_real_ip;", + "proxy_set_header X-Forwarded-For $http_x_forwarded_for;", + "proxy_set_header X-Forwarded-Proto $http_x_forwarded_proto;", + "/user/login $http_x_real_ip;", + "rate=10r/m;", + "limit_conn nodedc_gitea_conn 40;", + ) + self.assertTrue( + all(value in RUNNER.GITEA_NGINX_BRIDGE_CONTENT for value in expected_fragments) + ) + self.assertEqual( + hashlib.sha256(RUNNER.GITEA_NGINX_BRIDGE_CONTENT.encode()).hexdigest(), + RUNNER.GITEA_NGINX_BRIDGE_SHA256, + ) + with tempfile.TemporaryDirectory(prefix="nodedc-gitea-nginx-") as directory: + bridge = Path(directory) / "http.nodedc-gitea-uds.conf" + bridge.write_text(RUNNER.GITEA_NGINX_BRIDGE_CONTENT, encoding="utf-8") + metadata = SimpleNamespace( + st_mode=stat.S_IFREG | 0o644, + st_uid=0, + st_gid=0, + ) + with ( + mock.patch.object(RUNNER, "GITEA_NGINX_BRIDGE_CONFIG", bridge), + mock.patch.object(Path, "lstat", return_value=metadata), + mock.patch.object( + RUNNER.subprocess, + "run", + side_effect=[ + completed(0, stderr=RUNNER.GITEA_NGINX_VERSION + "\n"), + completed(0, stderr="syntax is ok\n"), + ], + ) as run, + mock.patch.object( + RUNNER, + "validate_gitea_nginx_listener", + return_value={"address": "127.0.0.1:3000"}, + ), + ): + result = RUNNER.validate_gitea_nginx_bridge_prerequisite() + self.assertEqual(result["sha256"], RUNNER.GITEA_NGINX_BRIDGE_SHA256) + self.assertEqual( + run.call_args_list[1].args[0], + [ + str(RUNNER.GITEA_NGINX), + "-t", + "-c", + str(RUNNER.GITEA_NGINX_MAIN_CONFIG), + ], + ) + bridge.write_text( + RUNNER.GITEA_NGINX_BRIDGE_CONTENT.replace( + "$http_x_forwarded_proto", + "$scheme", + ), + encoding="utf-8", + ) + with ( + mock.patch.object(RUNNER, "GITEA_NGINX_BRIDGE_CONFIG", bridge), + mock.patch.object(Path, "lstat", return_value=metadata), + self.assertRaisesRegex(RUNNER.DeployError, "identity/metadata"), + ): + RUNNER.validate_gitea_nginx_bridge_prerequisite() + + with ( + mock.patch.object( + RUNNER, + "gitea_loopback_listener_inodes", + return_value={"123"}, + ), + mock.patch.object( + RUNNER, + "gitea_listener_process_owners", + return_value={"123": {("nginx", 0), ("nginx", 1023)}}, + ), + ): + RUNNER.validate_gitea_nginx_listener() + with ( + mock.patch.object( + RUNNER, + "gitea_loopback_listener_inodes", + return_value={"123"}, + ), + mock.patch.object( + RUNNER, + "gitea_listener_process_owners", + return_value={"123": {("docker-proxy", 0)}}, + ), + self.assertRaisesRegex(RUNNER.DeployError, "not exclusively owned"), + ): + RUNNER.validate_gitea_nginx_listener() + + def test_installed_config_uses_uri_paths_and_never_secret_bytes(self): + parser = configparser.ConfigParser(interpolation=None) + parser.optionxform = str + for key, value in RUNNER.GITEA_EXPECTED_ENVIRONMENT.items(): + if not key.startswith("GITEA__"): + continue + _, section, option = key.split("__", 2) + if not parser.has_section(section): + parser.add_section(section) + parser.set(section, option, value) + with tempfile.TemporaryDirectory(prefix="nodedc-gitea-config-") as directory: + config_root = Path(directory) + app_ini = config_root / "app.ini" + with app_ini.open("w", encoding="utf-8") as output: + output.write("APP_NAME = NODE.DC Git\nRUN_USER = git\n\n") + parser.write(output) + metadata = SimpleNamespace( + st_mode=stat.S_IFREG | 0o600, + st_uid=1000, + st_size=app_ini.stat().st_size, + ) + secrets = ("secret-a" * 10, "secret-b" * 10) + with ( + mock.patch.object(RUNNER, "GITEA_CONFIG_DIR", config_root), + mock.patch.object(Path, "lstat", return_value=metadata), + ): + RUNNER.validate_gitea_installed_config(secrets) + parser.set("server", "LFS_JWT_SECRET", "unsafe-generated-secret") + with app_ini.open("w", encoding="utf-8") as output: + output.write("APP_NAME = NODE.DC Git\nRUN_USER = git\n\n") + parser.write(output) + with ( + mock.patch.object(RUNNER, "GITEA_CONFIG_DIR", config_root), + mock.patch.object(Path, "lstat", return_value=metadata), + self.assertRaisesRegex(RUNNER.DeployError, "plaintext secret"), + ): + RUNNER.validate_gitea_installed_config(secrets) + parser.remove_option("server", "LFS_JWT_SECRET") + with app_ini.open("w", encoding="utf-8") as output: + output.write("APP_NAME = NODE.DC Git\nRUN_USER = git\n\n") + parser.write(output) + with app_ini.open("a", encoding="utf-8") as output: + output.write(f"\n# leaked={secrets[0]}\n") + with ( + mock.patch.object(RUNNER, "GITEA_CONFIG_DIR", config_root), + mock.patch.object(Path, "lstat", return_value=metadata), + self.assertRaisesRegex(RUNNER.DeployError, "secret bytes"), + ): + RUNNER.validate_gitea_installed_config(secrets) + + def test_installed_config_rejects_duplicate_and_root_shadow_options(self): + sections = {} + for key, value in RUNNER.GITEA_EXPECTED_ENVIRONMENT.items(): + if key.startswith("GITEA__"): + _, section, option = key.split("__", 2) + sections.setdefault(section, []).append((option, value)) + raw = "APP_NAME = NODE.DC Git\nRUN_USER = git\n\n" + "\n".join( + "[" + section + "]\n" + "\n".join( + f"{option} = {value}" for option, value in values + ) + for section, values in sections.items() + ) + "\n" + RUNNER.parse_gitea_app_ini_explicit(raw) + with self.assertRaisesRegex(RUNNER.DeployError, "duplicate option"): + RUNNER.parse_gitea_app_ini_explicit( + raw.replace( + "[server]\n", + "[server]\nPROTOCOL = http\n", + 1, + ) + ) + with self.assertRaisesRegex(RUNNER.DeployError, "duplicate/unsafe section"): + RUNNER.parse_gitea_app_ini_explicit(raw + "\n[SERVER]\nROOT_URL=x\n") + with self.assertRaisesRegex(RUNNER.DeployError, "unsafe root-level"): + RUNNER.parse_gitea_app_ini_explicit( + "PROTOCOL = http\n" + raw + ) + with self.assertRaisesRegex(RUNNER.DeployError, "duplicate/unsafe section"): + RUNNER.parse_gitea_app_ini_explicit( + "[DEFAULT]\nPROTOCOL = http\n" + raw + ) + + def test_runner_generates_distinct_non_artifact_secret_files(self): + with tempfile.TemporaryDirectory(prefix="nodedc-gitea-secrets-") as directory: + root = Path(directory) + paths = [root / name for name in ("secret-key", "internal-token")] + with mock.patch.object(RUNNER.os, "fchown") as chown: + for index, path in enumerate(paths): + RUNNER.create_gitea_runtime_secret(path, f"test-{index}") + values = [path.read_text(encoding="ascii").strip() for path in paths] + self.assertEqual(len(set(values)), 2) + self.assertTrue(all(RUNNER.GITEA_SECRET_RE.fullmatch(value) for value in values)) + self.assertTrue( + all(stat.S_IMODE(path.stat().st_mode) == 0o400 for path in paths) + ) + self.assertEqual(chown.call_count, 2) + + def test_compose_runtime_is_no_pull_and_exact_service_only(self): + with ( + mock.patch.object( + RUNNER, + "compose_base_cmd", + return_value=["docker", "compose", "-f", "candidate.yml"], + ), + mock.patch.object(RUNNER.subprocess, "run", return_value=completed()) as run, + ): + RUNNER.run_gitea_fresh_install_compose( + ("gitea",), + RUNNER.GITEA_FRESH_INSTALL_ENTRIES, + ) + command = run.call_args_list[0].args[0] + self.assertEqual( + command[-7:], + [ + "up", "-d", "--force-recreate", "--pull", "never", + "--no-deps", "gitea", + ], + ) + self.assertNotIn("pull", command[:command.index("up")]) + + def test_health_dispatch_is_gitea_specific(self): + with mock.patch.object( + RUNNER, + "accept_gitea_fresh_install_runtime", + ) as accept: + RUNNER.run_healthchecks( + "gitea", + RUNNER.GITEA_FRESH_INSTALL_ENTRIES, + ("gitea",), + ) + accept.assert_called_once_with() + + def test_socket_boundary_and_no_docker_publications_are_exact(self): + parent = SimpleNamespace( + st_mode=stat.S_IFDIR | 0o750, + st_uid=1000, + st_gid=1023, + ) + uds = SimpleNamespace( + st_mode=stat.S_IFSOCK | 0o666, + st_uid=1000, + st_gid=1000, + ) + with mock.patch.object(Path, "lstat", side_effect=[parent, uds]): + self.assertEqual( + RUNNER.validate_gitea_socket_boundary(), + str(RUNNER.GITEA_SOCKET_FILE), + ) + bad = SimpleNamespace(**vars(uds)) + bad.st_mode = stat.S_IFREG | 0o666 + with ( + mock.patch.object(Path, "lstat", side_effect=[parent, bad]), + self.assertRaisesRegex(RUNNER.DeployError, "identity/metadata"), + ): + RUNNER.validate_gitea_socket_boundary() + + safe_container = {"HostConfig": {"PortBindings": {}}} + with ( + mock.patch.object( + RUNNER.subprocess, + "run", + return_value=completed(0, stdout="a" * 12 + "\n"), + ), + mock.patch.object(RUNNER, "docker_json", return_value=[safe_container]), + ): + RUNNER.validate_gitea_no_docker_port_publications() + published = { + "HostConfig": { + "PortBindings": { + "3000/tcp": [{"HostIp": "127.0.0.1", "HostPort": "3000"}] + } + } + } + with ( + mock.patch.object( + RUNNER.subprocess, + "run", + return_value=completed(0, stdout="a" * 12 + "\n"), + ), + mock.patch.object(RUNNER, "docker_json", return_value=[published]), + self.assertRaisesRegex(RUNNER.DeployError, "must not publish"), + ): + RUNNER.validate_gitea_no_docker_port_publications() + + def test_health_json_requires_pass_and_database_ping_pass(self): + response = mock.MagicMock() + response.status = 200 + response.read.return_value = json.dumps({ + "status": "pass", + "checks": {"database:ping": [{"status": "pass"}]}, + }).encode("utf-8") + response.__enter__.return_value = response + with mock.patch.object( + RUNNER.NO_REDIRECT_OPENER, + "open", + return_value=response, + ): + payload = RUNNER.healthcheck_gitea_json() + self.assertEqual(payload["status"], "pass") + + response.read.return_value = json.dumps({ + "status": "pass", + "checks": {"database:ping": [{"status": "fail"}]}, + }).encode("utf-8") + with ( + mock.patch.object( + RUNNER.NO_REDIRECT_OPENER, + "open", + return_value=response, + ), + mock.patch.object(RUNNER.time, "sleep"), + self.assertRaisesRegex(RUNNER.DeployError, "contract mismatch"), + ): + RUNNER.healthcheck_gitea_json() + + def test_uds_health_requires_database_ping_pass(self): + connection = mock.MagicMock() + response = mock.MagicMock() + response.status = 200 + response.read.return_value = json.dumps({ + "status": "pass", + "checks": {"database:ping": [{"status": "pass"}]}, + }).encode("utf-8") + with ( + mock.patch.object(RUNNER.socket, "socket", return_value=connection), + mock.patch.object(RUNNER.http.client, "HTTPResponse", return_value=response), + ): + payload = RUNNER.healthcheck_gitea_uds_json() + self.assertEqual(payload["status"], "pass") + connection.connect.assert_called_once_with(str(RUNNER.GITEA_SOCKET_FILE)) + self.assertIn(b"Host: git.dcserve.ru", connection.sendall.call_args.args[0]) + + response.read.return_value = json.dumps({ + "status": "pass", + "checks": {"database:ping": [{"status": "fail"}]}, + }).encode("utf-8") + with ( + mock.patch.object(RUNNER.socket, "socket", return_value=connection), + mock.patch.object(RUNNER.http.client, "HTTPResponse", return_value=response), + mock.patch.object(RUNNER.time, "sleep"), + self.assertRaisesRegex(RUNNER.DeployError, "contract mismatch"), + ): + RUNNER.healthcheck_gitea_uds_json() + + def test_rollback_stops_only_candidate_and_quarantines_new_state(self): + entries = RUNNER.GITEA_FRESH_INSTALL_ENTRIES + with tempfile.TemporaryDirectory(prefix="nodedc-gitea-rollback-") as directory: + root = Path(directory) / "nodedc-gitea" + backup = Path(directory) / "backup" + root.mkdir() + backup.mkdir() + data = root / "data" + config = root / "config" + secrets = root / "secrets" + socket_dir = root / "socket" + for path in (data, config, secrets, socket_dir): + path.mkdir() + (backup / "existing-files.txt").write_text("", encoding="utf-8") + (backup / "missing-files.txt").write_text( + "\n".join(entries) + "\n", + encoding="utf-8", + ) + with ( + mock.patch.object(RUNNER, "GITEA_ROOT", root), + mock.patch.object(RUNNER, "GITEA_DATA_DIR", data), + mock.patch.object(RUNNER, "GITEA_CONFIG_DIR", config), + mock.patch.object(RUNNER, "GITEA_SECRET_DIR", secrets), + mock.patch.object(RUNNER, "GITEA_SOCKET_DIR", socket_dir), + mock.patch.object( + RUNNER, + "stop_and_remove_compose_services", + ) as stop, + mock.patch.object(RUNNER, "restore_overlay_source") as restore, + mock.patch.object(RUNNER, "assert_loopback_tcp_port_closed"), + mock.patch.object( + RUNNER, + "validate_gitea_legacy_candidate_network_absent", + return_value="absent", + ), + mock.patch.object( + RUNNER, + "validate_gitea_no_docker_port_publications", + return_value="none", + ), + mock.patch.object( + RUNNER, + "validate_legacy_gitea_container_isolation", + return_value="stopped-restart-disabled", + ), + mock.patch.object( + RUNNER, + "gitea_compose_project_container_ids", + return_value=(), + ), + ): + result = RUNNER.rollback_gitea_fresh_install( + root, + backup, + entries, + "test-stamp", + runtime_started=True, + ) + stop.assert_called_once_with("gitea", ("gitea",)) + restore.assert_called_once_with(root, backup, entries, "test-stamp") + self.assertFalse(root.exists()) + retained = root.with_name("nodedc-gitea.failed-test-stamp") + self.assertTrue((retained / "data").is_dir()) + self.assertTrue((retained / "config").is_dir()) + self.assertTrue((retained / "secrets").is_dir()) + self.assertTrue((retained / "socket").is_dir()) + self.assertIn("runtime-state-preserved", result) + + def test_rollback_preserves_root_when_candidate_absence_is_unproven(self): + entries = RUNNER.GITEA_FRESH_INSTALL_ENTRIES + with tempfile.TemporaryDirectory(prefix="nodedc-gitea-reconcile-") as directory: + root = Path(directory) / "nodedc-gitea" + backup = Path(directory) / "backup" + root.mkdir() + backup.mkdir() + (backup / "existing-files.txt").write_text("", encoding="utf-8") + (backup / "missing-files.txt").write_text( + "\n".join(entries) + "\n", + encoding="utf-8", + ) + with ( + mock.patch.object(RUNNER, "GITEA_ROOT", root), + mock.patch.object( + RUNNER, + "stop_and_remove_compose_services", + ), + mock.patch.object( + RUNNER, + "gitea_compose_project_container_ids", + return_value=("a" * 12,), + ), + mock.patch.object(RUNNER, "restore_overlay_source") as restore, + self.assertRaisesRegex( + RUNNER.ReconciliationRequired, + "candidate_still_present", + ), + ): + RUNNER.rollback_gitea_fresh_install( + root, + backup, + entries, + "test-stamp", + runtime_started=True, + ) + self.assertTrue(root.is_dir()) + restore.assert_not_called() + + +if __name__ == "__main__": + unittest.main() diff --git a/infra/deploy-runner/test_gitea_incident_salvage.py b/infra/deploy-runner/test_gitea_incident_salvage.py new file mode 100644 index 0000000..5ab574f --- /dev/null +++ b/infra/deploy-runner/test_gitea_incident_salvage.py @@ -0,0 +1,3700 @@ +#!/usr/bin/env python3 +import csv +import hashlib +import importlib.machinery +import importlib.util +import inspect +import json +import os +import sqlite3 +import stat +import subprocess +import tarfile +import tempfile +import unittest +from pathlib import Path +from unittest import mock + + +SCRIPT_DIR = Path(__file__).resolve().parent +RUNNER_PATH = SCRIPT_DIR / "nodedc-deploy" +BUILDER_PATH = SCRIPT_DIR / "build-gitea-incident-salvage-artifact.mjs" +FIXTURE_ROOT = SCRIPT_DIR / "fixtures" / "gitea-salvage" +COMPOSE_PATH = FIXTURE_ROOT / "docker-compose.gitea.yml" +DESCRIPTOR_PATH = ( + FIXTURE_ROOT / "deployment" / "gitea-incident-salvage-v3.json" +) +LEGACY_DESCRIPTOR_PATH = ( + FIXTURE_ROOT / "deployment" / "gitea-incident-salvage-v1.json" +) +TOPICS_DESCRIPTOR_PATH = ( + FIXTURE_ROOT / "deployment" / "gitea-incident-salvage-v2.json" +) +DISPOSITION_PATH = ( + FIXTURE_ROOT + / "deployment" + / "gitea-incident-salvage" + / "confirmed-disposition-v1.json" +) +CLOSURE_DISPOSITION_PATH = ( + FIXTURE_ROOT + / "deployment" + / "gitea-incident-salvage" + / "confirmed-closure-disposition-v1.json" +) +DECISION_ROOT = ( + SCRIPT_DIR.parent.parent.parent + / "security-incidents" + / "gitea-20260814" + / "confirmed-decisions-v2" +) + + +def load_runner(): + loader = importlib.machinery.SourceFileLoader( + "nodedc_gitea_salvage_under_test", + str(RUNNER_PATH), + ) + spec = importlib.util.spec_from_loader(loader.name, loader) + module = importlib.util.module_from_spec(spec) + loader.exec_module(module) + return module + + +RUNNER = load_runner() + + +def completed(returncode=0, stdout="", stderr=""): + return subprocess.CompletedProcess([], returncode, stdout, stderr) + + +def inventory_repository(path, relative_path, trusted_device=None, mountpoints=None): + if trusted_device is None: + trusted_device = path.lstat().st_dev + if mountpoints is None: + mountpoints = set() + return RUNNER.inventory_gitea_salvage_bare_repository( + path, + relative_path, + trusted_device, + mountpoints, + ) + + +def unsupported_state_test_connection(): + connection = sqlite3.connect(":memory:") + table_columns = {} + for _label, table, column in RUNNER.GITEA_SALVAGE_UNSUPPORTED_REPOSITORY_TABLES: + table_columns.setdefault(table, set()).add(column) + for table, columns in table_columns.items(): + if table in {"attachment", "lfs_meta_object"}: + continue + fields = ", ".join(f'"{column}" INTEGER' for column in sorted(columns)) + extras = ", secret_sentinel TEXT" if table == "secret" else "" + connection.execute( + f'CREATE TABLE "{table}" (id INTEGER PRIMARY KEY, {fields}{extras})' + ) + connection.execute( + "CREATE TABLE lfs_meta_object (id INTEGER PRIMARY KEY, oid TEXT, " + "size INTEGER, repository_id INTEGER)" + ) + connection.execute( + "CREATE TABLE attachment (id INTEGER PRIMARY KEY, repo_id INTEGER, " + "issue_id INTEGER, release_id INTEGER, comment_id INTEGER, size INTEGER, " + "content_sentinel TEXT)" + ) + connection.execute( + "CREATE TABLE repo_unit (repo_id INTEGER, type INTEGER, config TEXT)" + ) + connection.execute( + "CREATE TABLE repository (id INTEGER PRIMARY KEY, description TEXT, " + "website TEXT, original_url TEXT, topics TEXT, avatar TEXT, " + "num_watches INTEGER, num_stars INTEGER, num_issues INTEGER, " + "num_pulls INTEGER, num_milestones INTEGER, num_projects INTEGER, " + "num_action_runs INTEGER, lfs_size INTEGER)" + ) + return connection + + +def disposition_evidence_fixture(): + disposition = json.loads(DISPOSITION_PATH.read_text(encoding="utf-8")) + grouped = {} + for decision in disposition["referencePolicy"]["exactDecisions"]: + key = ( + decision["oldRepositoryId"], + decision["repositoryPath"], + decision["wiki"], + ) + grouped.setdefault(key, []).append( + {"name": decision["name"], "oid": decision["oid"]} + ) + repositories = [] + for (repo_id, relative_path, wiki), refs_for_store in sorted(grouped.items()): + live_heads = [ + ref["name"] + for ref in refs_for_store + if ref["name"].startswith("refs/heads/") + ] + repositories.append( + { + "head": live_heads[0], + "object_format": "sha1", + "old_repo_id": repo_id, + "relative_path": relative_path, + "refs": refs_for_store, + "wiki": wiki, + } + ) + for missing in disposition["referencePolicy"]["headInvariants"][ + "allowedMissingTargets" + ]: + repositories.append( + { + "head": missing["head"], + "object_format": "sha1", + "old_repo_id": missing["oldRepositoryId"], + "relative_path": missing["repositoryPath"], + "refs": [], + "wiki": missing["wiki"], + } + ) + repositories.sort( + key=lambda row: ( + row["old_repo_id"], + int(row["wiki"]), + row["relative_path"], + ) + ) + refs = { + "bytes": disposition["sourceEvidence"]["referenceManifestBytes"], + "manifest": {"repositories": repositories}, + "refs": 105, + "sha256": RUNNER.GITEA_SALVAGE_DISPOSITION_REFERENCE_MANIFEST_SHA256, + } + + direct_counts = { + row["label"]: row["sourceCount"] + for row in disposition["repositoryStatePolicy"]["directRelations"] + } + report = { + "aggregates": { + "direct_relation_counts": direct_counts, + "repository_metadata_presence": { + row["name"]: row["sourceRepositories"] + for row in disposition["repositoryStatePolicy"]["textMetadata"] + }, + }, + "anomalies": [], + "coverage": { + "schema_only_unreviewed_tables": sorted( + table + for group in disposition["repositoryStatePolicy"][ + "schemaOnlyDependencyGroups" + ] + for table in group["tables"] + ), + }, + "database_sha256": RUNNER.GITEA_SALVAGE_SNAPSHOT_DATABASE_SHA256, + "decision_manifest_sha256": RUNNER.GITEA_SALVAGE_DECISION_MANIFEST_SHA256, + "kept_repository_ids": list(range(1, 46)), + "nonzero_categories": disposition["repositoryStatePolicy"][ + "sourceNonzeroCategories" + ], + "per_repository": [ + { + "old_repo_id": repo_id, + "repo_unit_types": { + "1": 1, + "2": 1, + "3": 1, + "4": 1, + "5": 1, + "8": 1, + "9": 1, + }, + } + for repo_id in range(1, 46) + ], + "schema_catalog_sha256": ( + RUNNER.GITEA_SALVAGE_DISPOSITION_UNSUPPORTED_SCHEMA_SHA256 + ), + "schema_mismatch": [], + "schema_missing": [], + "snapshot_uuid": RUNNER.GITEA_SALVAGE_SNAPSHOT_UUID, + } + unsupported = { + "bytes": disposition["sourceEvidence"][ + "unsupportedRepositoryReportBytes" + ], + "report": report, + "sha256": RUNNER.GITEA_SALVAGE_EXPECTED_UNSUPPORTED_REPORT_SHA256, + } + topic_evidence = { + "database_sha256": RUNNER.GITEA_SALVAGE_SNAPSHOT_DATABASE_SHA256, + "decision_manifest_sha256": RUNNER.GITEA_SALVAGE_DECISION_MANIFEST_SHA256, + "material_repositories": 0, + "repositories": [ + { + "encoding": "json-null", + "material": False, + "old_repo_id": repo_id, + "topic_count": 0, + } + for repo_id in range(1, 46) + ], + "schema": "nodedc.gitea.salvage-semantic-topics/v2", + "serialized_arrays": 0, + "serialized_nulls": 45, + "snapshot_uuid": RUNNER.GITEA_SALVAGE_SNAPSHOT_UUID, + "topics": 0, + } + topics = { + "bytes": 1, + "evidence": topic_evidence, + "json": "{}", + "sha256": "0" * 64, + } + return disposition, refs, unsupported, topics + + +def closure_decisions_fixture(): + with (DECISION_ROOT / "users.decisions.csv").open( + encoding="utf-8", + newline="", + ) as handle: + users = list(csv.DictReader(handle)) + with (DECISION_ROOT / "repositories.decisions.csv").open( + encoding="utf-8", + newline="", + ) as handle: + repositories = list(csv.DictReader(handle)) + return { + "users": users, + "repositories": repositories, + "kept_repositories": [ + row for row in repositories if row["decision"] == "KEEP" + ], + } + + +def closure_state_test_connection(): + connection = sqlite3.connect(":memory:") + connection.row_factory = sqlite3.Row + statements = ( + "CREATE TABLE access (id INTEGER PRIMARY KEY, repo_id INTEGER, " + "user_id INTEGER, mode INTEGER)", + "CREATE TABLE collaboration (id INTEGER PRIMARY KEY, repo_id INTEGER, " + "user_id INTEGER, mode INTEGER)", + "CREATE TABLE issue (id INTEGER PRIMARY KEY, repo_id INTEGER, " + "poster_id INTEGER, name TEXT, content TEXT, is_pull INTEGER, " + "is_closed INTEGER, milestone_id INTEGER)", + "CREATE TABLE pull_request (id INTEGER PRIMARY KEY, issue_id INTEGER, " + "base_repo_id INTEGER, head_repo_id INTEGER, has_merged INTEGER, " + "merger_id INTEGER)", + "CREATE TABLE comment (id INTEGER PRIMARY KEY, issue_id INTEGER, " + "poster_id INTEGER, original_author TEXT, original_author_id INTEGER, " + "label_id INTEGER, old_project_id INTEGER, project_id INTEGER, " + "old_milestone_id INTEGER, milestone_id INTEGER, time_id INTEGER, " + "assignee_id INTEGER, assignee_team_id INTEGER NOT NULL DEFAULT 0, " + "resolve_doer_id INTEGER, dependent_issue_id INTEGER, review_id INTEGER, " + "ref_repo_id INTEGER, ref_issue_id INTEGER, ref_comment_id INTEGER, " + "ref_action INTEGER, ref_is_pull INTEGER, content TEXT, patch TEXT, " + "old_title TEXT, new_title TEXT, old_ref TEXT, new_ref TEXT)", + "CREATE TABLE issue_assignees (id INTEGER PRIMARY KEY, issue_id INTEGER, " + "assignee_id INTEGER)", + "CREATE TABLE issue_content_history (id INTEGER PRIMARY KEY, " + "issue_id INTEGER, comment_id INTEGER, poster_id INTEGER, " + "content_text TEXT)", + "CREATE TABLE issue_label (id INTEGER PRIMARY KEY, issue_id INTEGER, " + "label_id INTEGER)", + "CREATE TABLE issue_user (id INTEGER PRIMARY KEY, issue_id INTEGER, " + "uid INTEGER)", + "CREATE TABLE issue_watch (id INTEGER PRIMARY KEY, issue_id INTEGER, " + "user_id INTEGER)", + "CREATE TABLE review (id INTEGER PRIMARY KEY, issue_id INTEGER, " + "reviewer_id INTEGER, reviewer_team_id INTEGER NOT NULL DEFAULT 0, " + "original_author TEXT, original_author_id INTEGER, content TEXT)", + "CREATE TABLE stopwatch (id INTEGER PRIMARY KEY, issue_id INTEGER, " + "user_id INTEGER)", + "CREATE TABLE tracked_time (id INTEGER PRIMARY KEY, issue_id INTEGER, " + "user_id INTEGER, time INTEGER)", + "CREATE TABLE reaction (id INTEGER PRIMARY KEY, issue_id INTEGER, " + "comment_id INTEGER, user_id INTEGER)", + "CREATE TABLE review_state (id INTEGER PRIMARY KEY, pull_id INTEGER, " + "user_id INTEGER, updated_files TEXT)", + "CREATE TABLE issue_dependency (id INTEGER PRIMARY KEY, user_id INTEGER, " + "issue_id INTEGER, dependency_id INTEGER)", + "CREATE TABLE label (id INTEGER PRIMARY KEY, repo_id INTEGER, name TEXT, " + "description TEXT, color TEXT)", + "CREATE TABLE milestone (id INTEGER PRIMARY KEY, repo_id INTEGER, " + "name TEXT, content TEXT)", + "CREATE TABLE project (id INTEGER PRIMARY KEY, repo_id INTEGER, " + "creator_id INTEGER, title TEXT, description TEXT)", + "CREATE TABLE release (id INTEGER PRIMARY KEY, repo_id INTEGER, " + "publisher_id INTEGER, tag_name TEXT, target TEXT, title TEXT, note TEXT)", + "CREATE TABLE project_board (id INTEGER PRIMARY KEY, project_id INTEGER, " + "title TEXT, color TEXT)", + "CREATE TABLE project_issue (id INTEGER PRIMARY KEY, issue_id INTEGER, " + "project_id INTEGER, project_board_id INTEGER)", + "CREATE TABLE team (id INTEGER PRIMARY KEY, org_id INTEGER NOT NULL)", + "CREATE TABLE pull_auto_merge (id INTEGER PRIMARY KEY, pull_id INTEGER, " + "doer_id INTEGER, merge_style TEXT, message TEXT)", + "CREATE TABLE notification (id INTEGER PRIMARY KEY, user_id INTEGER, " + "repo_id INTEGER, issue_id INTEGER, comment_id INTEGER)", + "CREATE TABLE repo_unit (id INTEGER PRIMARY KEY, repo_id INTEGER, " + "type INTEGER, config TEXT)", + "CREATE TABLE attachment (id INTEGER PRIMARY KEY, uuid UUID, " + "repo_id INTEGER, issue_id INTEGER, release_id INTEGER, " + "uploader_id INTEGER, comment_id INTEGER, size INTEGER, name TEXT)", + "CREATE TABLE package (id INTEGER PRIMARY KEY, repo_id INTEGER, " + "owner_id INTEGER, name TEXT)", + "CREATE TABLE package_version (id INTEGER PRIMARY KEY, " + "package_id INTEGER, creator_id INTEGER, version TEXT, metadata_json TEXT)", + "CREATE TABLE package_file (id INTEGER PRIMARY KEY, version_id INTEGER, " + "blob_id INTEGER, name TEXT)", + "CREATE TABLE package_blob (id INTEGER PRIMARY KEY, size INTEGER, " + "hash_sha256 TEXT)", + "CREATE TABLE package_property (id INTEGER PRIMARY KEY, ref_type INTEGER, " + "ref_id INTEGER, name TEXT, value TEXT)", + "CREATE TABLE action_run (id INTEGER PRIMARY KEY, repo_id INTEGER, " + "trigger_user_id INTEGER, event_payload TEXT)", + "CREATE TABLE action_schedule (id INTEGER PRIMARY KEY, repo_id INTEGER, " + "trigger_user_id INTEGER, content BLOB)", + "CREATE TABLE action_runner (id INTEGER PRIMARY KEY, repo_id INTEGER, " + "token_hash TEXT, token_salt TEXT)", + "CREATE TABLE action_variable (id INTEGER PRIMARY KEY, repo_id INTEGER, " + "data TEXT)", + "CREATE TABLE secret (id INTEGER PRIMARY KEY, repo_id INTEGER, data TEXT)", + "CREATE TABLE action_artifact (id INTEGER PRIMARY KEY, repo_id INTEGER, " + "run_id INTEGER, file_size INTEGER, file_compressed_size INTEGER, " + "storage_path TEXT)", + "CREATE TABLE action_run_job (id INTEGER PRIMARY KEY, repo_id INTEGER, " + "run_id INTEGER, workflow_payload BLOB)", + "CREATE TABLE action_task (id INTEGER PRIMARY KEY, repo_id INTEGER, " + "job_id INTEGER, log_length INTEGER, log_size INTEGER, token_hash TEXT, " + "log_filename TEXT)", + "CREATE TABLE action_run_index (group_id INTEGER PRIMARY KEY, " + "max_index INTEGER)", + ) + for statement in statements: + connection.execute(statement) + return connection + + +def populated_closure_evidence_fixture(): + connection = closure_state_test_connection() + decisions = closure_decisions_fixture() + kept_ids = sorted( + int(row["repo_id"]) for row in decisions["kept_repositories"] + ) + kept_user = next( + int(row["user_id"]) + for row in decisions["users"] + if row["decision"] == "KEEP_ACTIVE" + ) + deleted_user = next( + int(row["user_id"]) + for row in decisions["users"] + if row["decision"] == "DELETE" + ) + repo_id, second_repo_id = kept_ids[:2] + connection.executemany( + "INSERT INTO access VALUES (?,?,?,?)", + ((1, repo_id, kept_user, 2), (2, repo_id, deleted_user, 1)), + ) + connection.executemany( + "INSERT INTO collaboration VALUES (?,?,?,?)", + ((3, repo_id, kept_user, 2), (4, repo_id, deleted_user, 1)), + ) + connection.executemany( + "INSERT INTO issue VALUES (?,?,?,?,?,?,?,?)", + ( + ( + 100, + repo_id, + kept_user, + "SECRET_ISSUE_TITLE", + "SECRET_ISSUE_BODY", + 0, + 0, + 401, + ), + ( + 101, + repo_id, + deleted_user, + "SECRET_PULL_TITLE", + "SECRET_PULL_BODY", + 1, + 1, + 0, + ), + ), + ) + connection.execute( + "INSERT INTO pull_request " + "(id,issue_id,base_repo_id,head_repo_id,has_merged,merger_id) " + "VALUES (200,101,?,?,0,0)", + (repo_id, second_repo_id), + ) + connection.execute( + "INSERT INTO comment " + "(id,issue_id,poster_id,original_author,original_author_id,label_id," + "old_project_id,project_id,old_milestone_id,milestone_id,time_id," + "assignee_id,assignee_team_id,resolve_doer_id,dependent_issue_id," + "review_id,ref_repo_id,ref_issue_id,ref_comment_id,ref_action," + "ref_is_pull,content,patch,old_title,new_title,old_ref,new_ref) " + "VALUES (300,100,?,'SECRET_EXTERNAL_AUTHOR',987654,400,500,500," + "401,401,308,?,0,?,101,0,?,100,0,1,0,'SECRET_COMMENT'," + "'SECRET_PATCH','SECRET_OLD_TITLE','SECRET_NEW_TITLE'," + "'SECRET_OLD_REF','SECRET_NEW_REF')", + (kept_user, kept_user, deleted_user, repo_id), + ) + connection.execute( + "INSERT INTO issue_assignees VALUES (301,100,?)", + (deleted_user,), + ) + connection.execute( + "INSERT INTO issue_content_history " + "(id,issue_id,comment_id,poster_id,content_text) " + "VALUES (302,100,300,?,'SECRET_HISTORY')", + (kept_user,), + ) + connection.execute("INSERT INTO issue_label VALUES (303,100,400)") + connection.execute("INSERT INTO issue_user VALUES (304,100,?)", (kept_user,)) + connection.execute("INSERT INTO issue_watch VALUES (305,100,?)", (deleted_user,)) + connection.execute( + "INSERT INTO review " + "(id,issue_id,reviewer_id,reviewer_team_id,original_author," + "original_author_id,content) " + "VALUES (306,101,?,0,'SECRET_REVIEW_EXTERNAL_AUTHOR',456789," + "'SECRET_REVIEW')", + (kept_user,), + ) + connection.execute("INSERT INTO stopwatch VALUES (307,100,?)", (kept_user,)) + connection.execute("INSERT INTO tracked_time VALUES (308,100,?,15)", (kept_user,)) + connection.execute("INSERT INTO reaction VALUES (309,100,0,?)", (kept_user,)) + connection.execute( + "INSERT INTO review_state VALUES (310,200,?,'SECRET_UPDATED_FILES')", + (kept_user,), + ) + connection.execute( + "INSERT INTO issue_dependency VALUES (311,?,100,101)", + (kept_user,), + ) + connection.execute( + "INSERT INTO label VALUES (400,?,'SECRET_LABEL','SECRET_LABEL_DESC','abcdef')", + (repo_id,), + ) + connection.execute( + "INSERT INTO milestone VALUES (401,?,'SECRET_MILESTONE','SECRET_MILESTONE_BODY')", + (repo_id,), + ) + connection.execute( + "INSERT INTO project VALUES (500,?,?,'SECRET_PROJECT','SECRET_PROJECT_BODY')", + (repo_id, kept_user), + ) + connection.execute( + "INSERT INTO release VALUES (600,?,?,'v1','main','SECRET_RELEASE','SECRET_NOTE')", + (repo_id, kept_user), + ) + connection.execute( + "INSERT INTO project_board VALUES (501,500,'SECRET_BOARD','aabbcc')" + ) + connection.execute("INSERT INTO project_issue VALUES (502,100,500,501)") + connection.execute( + "INSERT INTO pull_auto_merge VALUES (503,200,?,'merge','SECRET_MERGE')", + (kept_user,), + ) + connection.execute( + "INSERT INTO notification VALUES (504,?,?,100,300)", + (kept_user, repo_id), + ) + unit_rows = [] + unit_id = 505 + for unit_repo_id in kept_ids: + for unit_type in (1, 2, 3, 4, 5, 8, 9): + unit_rows.append( + (unit_id, unit_repo_id, unit_type, "SECRET_UNIT_CONFIG") + ) + unit_id += 1 + connection.executemany( + "INSERT INTO repo_unit VALUES (?,?,?,?)", + unit_rows, + ) + connection.execute( + "INSERT INTO attachment VALUES (700,'12345678-1234-1234-1234-123456789abc'," + "?,100,0,?,300,123,'SECRET_ATTACHMENT_NAME')", + (repo_id, kept_user), + ) + + schema = RUNNER.gitea_salvage_unsupported_schema_catalog(connection) + direct_counts = { + label: 0 + for label, _table, _column in ( + RUNNER.GITEA_SALVAGE_UNSUPPORTED_REPOSITORY_TABLES + ) + } + direct_counts.update( + { + "access_grants": 2, + "attachments": 1, + "collaborators": 2, + "issues": 2, + "labels": 1, + "milestones": 1, + "pull_requests_base": 1, + "pull_requests_head": 1, + "releases": 1, + } + ) + report = { + "aggregates": { + "attachments": { + "association_rows": 1, + "logical_bytes": 123, + }, + "direct_relation_counts": direct_counts, + }, + "anomalies": [], + "database_sha256": RUNNER.GITEA_SALVAGE_SNAPSHOT_DATABASE_SHA256, + "decision_manifest_sha256": RUNNER.GITEA_SALVAGE_DECISION_MANIFEST_SHA256, + "kept_repository_ids": kept_ids, + "schema_catalog": schema["catalog"], + "schema_catalog_sha256": schema["sha256"], + "schema_mismatch": [], + "schema_missing": [], + "snapshot_uuid": RUNNER.GITEA_SALVAGE_SNAPSHOT_UUID, + } + unsupported = { + "report": report, + **RUNNER.canonical_gitea_salvage_evidence( + report, + "closure test predecessor report", + ), + } + topics = {"sha256": "1" * 64} + return connection, decisions, unsupported, topics, repo_id, deleted_user + + +def run_closure_inventory_fixture(connection, decisions, unsupported, topics): + with ( + mock.patch.object( + RUNNER, + "GITEA_SALVAGE_EXPECTED_UNSUPPORTED_REPORT_SHA256", + unsupported["sha256"], + ), + mock.patch.object( + RUNNER, + "GITEA_SALVAGE_DISPOSITION_UNSUPPORTED_SCHEMA_SHA256", + unsupported["report"]["schema_catalog_sha256"], + ), + mock.patch.object( + RUNNER, + "GITEA_SALVAGE_DISPOSITION_TOPICS_SHA256", + topics["sha256"], + ), + ): + return RUNNER.gitea_salvage_incident_closure_inventory( + connection, + decisions, + unsupported, + topics, + ) + +class GiteaIncidentSalvageArtifactTest(unittest.TestCase): + def test_fixture_is_exact_rootless_1272_clean_state_boundary(self): + compose = COMPOSE_PATH.read_text(encoding="utf-8") + descriptor = json.loads(DESCRIPTOR_PATH.read_text(encoding="utf-8")) + self.assertEqual( + hashlib.sha256(COMPOSE_PATH.read_bytes()).hexdigest(), + RUNNER.GITEA_SALVAGE_COMPOSE_SHA256, + ) + self.assertEqual( + hashlib.sha256(DESCRIPTOR_PATH.read_bytes()).hexdigest(), + RUNNER.GITEA_SALVAGE_DESCRIPTOR_SHA256, + ) + self.assertEqual( + hashlib.sha256(DISPOSITION_PATH.read_bytes()).hexdigest(), + RUNNER.GITEA_SALVAGE_DISPOSITION_SHA256, + ) + self.assertEqual( + hashlib.sha256(CLOSURE_DISPOSITION_PATH.read_bytes()).hexdigest(), + RUNNER.GITEA_SALVAGE_CLOSURE_DISPOSITION_SHA256, + ) + self.assertEqual( + hashlib.sha256(LEGACY_DESCRIPTOR_PATH.read_bytes()).hexdigest(), + "d0a0734e65ad182cbf17e83179a9e19058a1fc2e60c2905aff545483661f55dd", + ) + self.assertEqual( + json.loads(LEGACY_DESCRIPTOR_PATH.read_text(encoding="utf-8"))[ + "schemaVersion" + ], + "nodedc.gitea.incident-salvage.v1", + ) + self.assertEqual( + hashlib.sha256(TOPICS_DESCRIPTOR_PATH.read_bytes()).hexdigest(), + "879f4d761a062bbf967d79329436d4ae8ee1655bb63d0b93562d75146b0feaf3", + ) + self.assertEqual( + json.loads(TOPICS_DESCRIPTOR_PATH.read_text(encoding="utf-8"))[ + "schemaVersion" + ], + "nodedc.gitea.incident-salvage.v2", + ) + disposition = json.loads(DISPOSITION_PATH.read_text(encoding="utf-8")) + self.assertEqual( + descriptor["disposition"], + { + "schema": "nodedc.gitea.incident-disposition.v1", + "file": RUNNER.GITEA_SALVAGE_DISPOSITION_REL, + "sha256": RUNNER.GITEA_SALVAGE_DISPOSITION_SHA256, + }, + ) + self.assertTrue(disposition["activation"]["applyFrozen"]) + self.assertEqual( + descriptor["closureDisposition"], + { + "file": RUNNER.GITEA_SALVAGE_CLOSURE_DISPOSITION_REL, + "predecessorArtifactSha256": ( + RUNNER.GITEA_SALVAGE_CLOSURE_PREDECESSOR_ARTIFACT_SHA256 + ), + "schema": "nodedc.gitea.incident-closure-disposition.v1", + "sha256": RUNNER.GITEA_SALVAGE_CLOSURE_DISPOSITION_SHA256, + }, + ) + self.assertEqual( + disposition["remainingBlockers"], + list(RUNNER.GITEA_SALVAGE_DISPOSITION_REMAINING_BLOCKERS), + ) + self.assertEqual(descriptor["runtime"]["image"], RUNNER.GITEA_SALVAGE_IMAGE) + self.assertEqual( + descriptor["runtime"]["imageId"], + "sha256:272085a806e6d182352cdb011c0ebab1d2efc7ec45247de84de5659c7bc5c4c6", + ) + self.assertEqual( + descriptor["runtime"]["repoDigest"], + RUNNER.GITEA_SALVAGE_REPO_DIGEST, + ) + self.assertFalse(descriptor["identity"]["preserveNumericUserIds"]) + self.assertFalse(descriptor["identity"]["preserveNumericRepositoryIds"]) + self.assertIn("oldToNewIdMapping", descriptor["identity"]) + for required in ( + "network_mode: none", + 'restart: "no"', + 'user: "1000:1000"', + "- /usr/local/bin/gitea", + "- /etc/gitea/app.ini", + "read_only: true", + "- ALL", + "no-new-privileges:true", + "source: /volume1/docker/nodedc-gitea/data", + "target: /data", + "source: /volume1/docker/nodedc-gitea/config", + "target: /etc/gitea", + "source: /volume1/docker/nodedc-gitea/socket", + "target: /run/gitea", + "create_host_path: false", + ): + self.assertIn(required, compose) + for forbidden in ( + "ports:", + "networks:", + "/volume1/docker/gitea", + "/var/run/docker.sock", + "TWO_FACTOR_AUTH", + "LFS_JWT_SECRET", + "restart: unless-stopped", + ): + self.assertNotIn(forbidden, compose) + + def test_compose_schema_is_valid_without_pull_or_start(self): + result = subprocess.run( + [ + "docker", + "compose", + "--project-name", + "nodedc-gitea-salvage-fixture-test", + "--file", + str(COMPOSE_PATH), + "config", + "--quiet", + ], + check=False, + capture_output=True, + text=True, + ) + if result.returncode == 127: + self.skipTest("docker compose is unavailable") + self.assertEqual(result.returncode, 0, result.stderr) + + def test_decision_v2_bundle_is_exact_and_partitioned(self): + expected = { + "confirmed-decision.json": RUNNER.GITEA_SALVAGE_DECISION_MANIFEST_SHA256, + "users.decisions.csv": RUNNER.GITEA_SALVAGE_USERS_SHA256, + "repositories.decisions.csv": RUNNER.GITEA_SALVAGE_REPOSITORIES_SHA256, + } + for name, digest in expected.items(): + self.assertEqual( + hashlib.sha256((DECISION_ROOT / name).read_bytes()).hexdigest(), + digest, + ) + with tempfile.TemporaryDirectory(prefix="gitea-salvage-decision-") as directory: + payload = Path(directory) + for relative, source in ( + ( + RUNNER.GITEA_SALVAGE_DECISION_MANIFEST_REL, + DECISION_ROOT / "confirmed-decision.json", + ), + (RUNNER.GITEA_SALVAGE_USERS_REL, DECISION_ROOT / "users.decisions.csv"), + ( + RUNNER.GITEA_SALVAGE_REPOSITORIES_REL, + DECISION_ROOT / "repositories.decisions.csv", + ), + ): + target = payload / relative + target.parent.mkdir(parents=True, exist_ok=True) + target.write_bytes(source.read_bytes()) + decisions = RUNNER.validate_gitea_salvage_decision_bundle(payload) + self.assertEqual(len(decisions["users"]), 972) + self.assertEqual(len(decisions["kept_users"]), 10) + self.assertEqual(len(decisions["repositories"]), 2058) + self.assertEqual(len(decisions["kept_repositories"]), 45) + self.assertEqual( + {row["owner"] for row in decisions["kept_repositories"]}, + {"dctouch", "SILVER"}, + ) + + def test_incident_disposition_is_exact_hash_bound_plan_only_data(self): + with tempfile.TemporaryDirectory(prefix="gitea-salvage-disposition-") as directory: + payload = Path(directory) + target = payload / RUNNER.GITEA_SALVAGE_DISPOSITION_REL + target.parent.mkdir(parents=True, exist_ok=True) + target.write_bytes(DISPOSITION_PATH.read_bytes()) + disposition = RUNNER.validate_gitea_salvage_incident_disposition(payload) + self.assertEqual( + disposition["sourceEvidence"]["unsupportedRepositoryReportSha256"], + RUNNER.GITEA_SALVAGE_EXPECTED_UNSUPPORTED_REPORT_SHA256, + ) + self.assertEqual( + disposition["referencePolicy"]["liveRestore"]["totalRefs"], + 93, + ) + self.assertEqual( + disposition["referencePolicy"]["archiveOnly"]["totalRefs"], + 12, + ) + self.assertEqual( + len(disposition["referencePolicy"]["exactDecisions"]), + 105, + ) + unit_rows = disposition["repositoryStatePolicy"]["units"]["rows"] + self.assertEqual([row["type"] for row in unit_rows], list(range(1, 11))) + self.assertEqual( + [row["type"] for row in unit_rows if row["targetCount"]], + [1, 2, 3, 4, 5, 8], + ) + canonical = json.dumps( + disposition, + ensure_ascii=True, + sort_keys=True, + separators=(",", ":"), + allow_nan=False, + ).encode("utf-8") + b"\n" + self.assertEqual(DISPOSITION_PATH.read_bytes(), canonical) + target.write_bytes(DISPOSITION_PATH.read_bytes() + b"\n") + with self.assertRaisesRegex(RUNNER.DeployError, "digest mismatch"): + RUNNER.validate_gitea_salvage_incident_disposition(payload) + + def test_closure_disposition_is_additive_hash_bound_and_apply_frozen(self): + with tempfile.TemporaryDirectory( + prefix="gitea-salvage-closure-disposition-" + ) as directory: + payload = Path(directory) + target = payload / RUNNER.GITEA_SALVAGE_CLOSURE_DISPOSITION_REL + target.parent.mkdir(parents=True, exist_ok=True) + target.write_bytes(CLOSURE_DISPOSITION_PATH.read_bytes()) + disposition = RUNNER.validate_gitea_salvage_closure_disposition( + payload + ) + self.assertEqual( + disposition["sourceEvidence"], + { + "databaseSha256": ( + RUNNER.GITEA_SALVAGE_SNAPSHOT_DATABASE_SHA256 + ), + "identityDecisionManifestSha256": ( + RUNNER.GITEA_SALVAGE_DECISION_MANIFEST_SHA256 + ), + "referenceManifestSha256": ( + RUNNER.GITEA_SALVAGE_DISPOSITION_REFERENCE_MANIFEST_SHA256 + ), + "semanticTopicsSha256": ( + RUNNER.GITEA_SALVAGE_DISPOSITION_TOPICS_SHA256 + ), + "snapshotUuid": RUNNER.GITEA_SALVAGE_SNAPSHOT_UUID, + "unsupportedRepositoryReportSha256": ( + RUNNER.GITEA_SALVAGE_EXPECTED_UNSUPPORTED_REPORT_SHA256 + ), + "unsupportedSchemaCatalogSha256": ( + RUNNER.GITEA_SALVAGE_DISPOSITION_UNSUPPORTED_SCHEMA_SHA256 + ), + }, + ) + self.assertTrue(disposition["activation"]["applyFrozen"]) + self.assertIsNone(disposition["closureReport"]["expectedSha256"]) + self.assertIn( + "closure-report-review-pin-pending", + disposition["remainingBlockers"], + ) + self.assertNotIn( + "issue-pr-polymorphic-subrelation-verifier-pending", + disposition["remainingBlockers"], + ) + self.assertEqual( + disposition["policies"]["issuesPullRequestsMetadata"][ + "subrelationClosure" + ], + { + "commentHistoryMerger": ( + "SCHEMA_BOUND_EXACT_RELATION_COUNTS_AND_CLASSES" + ), + "externalAuthors": ( + "PRESENCE_AND_NAME_BYTE_LENGTHS_ONLY_NO_LOCAL_USER_MAPPING" + ), + "legacyRowsImported": False, + "teamRelations": ( + "EXACT_ROW_TEAM_ORG_IDS_SEALED_HOLD_AND_BLOCK_IF_PRESENT" + ), + }, + ) + target.write_bytes(CLOSURE_DISPOSITION_PATH.read_bytes() + b"\n") + with self.assertRaisesRegex(RUNNER.DeployError, "identity is unsafe"): + RUNNER.validate_gitea_salvage_closure_disposition(payload) + + def test_payload_validator_accepts_only_exact_files(self): + with tempfile.TemporaryDirectory(prefix="gitea-salvage-payload-") as directory: + payload = Path(directory) + for relative, source in ( + (RUNNER.GITEA_COMPOSE_REL, COMPOSE_PATH), + (RUNNER.GITEA_SALVAGE_DESCRIPTOR_REL, DESCRIPTOR_PATH), + (RUNNER.GITEA_SALVAGE_DISPOSITION_REL, DISPOSITION_PATH), + ( + RUNNER.GITEA_SALVAGE_CLOSURE_DISPOSITION_REL, + CLOSURE_DISPOSITION_PATH, + ), + ( + RUNNER.GITEA_SALVAGE_DECISION_MANIFEST_REL, + DECISION_ROOT / "confirmed-decision.json", + ), + (RUNNER.GITEA_SALVAGE_USERS_REL, DECISION_ROOT / "users.decisions.csv"), + ( + RUNNER.GITEA_SALVAGE_REPOSITORIES_REL, + DECISION_ROOT / "repositories.decisions.csv", + ), + ): + target = payload / relative + target.parent.mkdir(parents=True, exist_ok=True) + target.write_bytes(source.read_bytes()) + result = RUNNER.validate_gitea_incident_salvage_payload( + payload, + RUNNER.GITEA_SALVAGE_ENTRIES, + ) + self.assertEqual(len(result["kept_users"]), 10) + self.assertEqual(len(result["kept_repositories"]), 45) + self.assertEqual( + result["disposition"]["referencePolicy"]["liveRestore"]["totalRefs"], + 93, + ) + (payload / RUNNER.GITEA_SALVAGE_USERS_REL).write_bytes( + (DECISION_ROOT / "users.decisions.csv").read_bytes() + b"\n" + ) + with self.assertRaisesRegex(RUNNER.DeployError, "digest mismatch"): + RUNNER.validate_gitea_incident_salvage_payload( + payload, + RUNNER.GITEA_SALVAGE_ENTRIES, + ) + + def test_builder_is_deterministic_data_only_and_refuses_overwrite(self): + with ( + tempfile.TemporaryDirectory( + prefix="gitea-salvage-builder-a-" + ) as first_directory, + tempfile.TemporaryDirectory( + prefix="gitea-salvage-builder-b-" + ) as second_directory, + ): + command = ["node", str(BUILDER_PATH), "gitea-salvage-test-002"] + results = [] + for directory in (first_directory, second_directory): + environment = dict(os.environ) + environment["NODEDC_DEPLOY_ARTIFACT_DIR"] = directory + results.append( + subprocess.run( + command, + env=environment, + check=True, + capture_output=True, + text=True, + ) + ) + first_result = json.loads(results[0].stdout) + second_result = json.loads(results[1].stdout) + first_artifact = Path(first_result["artifact"]) + second_artifact = Path(second_result["artifact"]) + first_sha = hashlib.sha256(first_artifact.read_bytes()).hexdigest() + second_sha = hashlib.sha256(second_artifact.read_bytes()).hexdigest() + self.assertEqual(first_sha, second_sha) + self.assertEqual(first_sha, first_result["sha256"]) + self.assertEqual(second_sha, second_result["sha256"]) + self.assertEqual( + first_result["stagePolicy"], + "after-exact-runner-promotion-plan-only", + ) + self.assertEqual( + first_result["applyPolicy"], + "hard-frozen-before-root-creation", + ) + + overwrite_environment = dict(os.environ) + overwrite_environment["NODEDC_DEPLOY_ARTIFACT_DIR"] = first_directory + overwrite = subprocess.run( + command, + env=overwrite_environment, + check=False, + capture_output=True, + text=True, + ) + self.assertNotEqual(overwrite.returncode, 0) + self.assertIn( + "gitea_salvage_artifact_target_already_exists", + overwrite.stderr, + ) + self.assertEqual( + hashlib.sha256(first_artifact.read_bytes()).hexdigest(), + first_sha, + ) + self.assertEqual( + list(Path(first_directory).glob(".*.tmp")), + [], + ) + + with tarfile.open(first_artifact, "r:gz") as archive: + names = archive.getnames() + self.assertNotIn("payload/config", names) + self.assertNotIn("payload/data", names) + self.assertNotIn("payload/secrets", names) + self.assertIn( + "payload/deployment/gitea-incident-salvage/repositories.decisions.csv", + names, + ) + self.assertIn( + "payload/deployment/gitea-incident-salvage/confirmed-disposition-v1.json", + names, + ) + self.assertIn( + "payload/deployment/gitea-incident-salvage/confirmed-closure-disposition-v1.json", + names, + ) + self.assertIn( + "payload/deployment/gitea-incident-salvage-v3.json", + names, + ) + self.assertEqual( + archive.extractfile("files.txt").read().decode(), + "\n".join(RUNNER.GITEA_SALVAGE_ENTRIES) + "\n", + ) + disposition_bytes = archive.extractfile( + "payload/deployment/gitea-incident-salvage/" + "confirmed-disposition-v1.json" + ).read() + self.assertEqual(disposition_bytes, DISPOSITION_PATH.read_bytes()) + with tempfile.TemporaryDirectory( + prefix="gitea-salvage-load-artifact-" + ) as work_directory: + manifest, entries, _payload = RUNNER.load_artifact( + first_artifact, + Path(work_directory), + ) + self.assertEqual(manifest["component"], "gitea") + self.assertEqual(tuple(entries), RUNNER.GITEA_SALVAGE_ENTRIES) + + +class GiteaIncidentSalvageRunnerTest(unittest.TestCase): + def test_slice_is_additive_and_payload_boundary_is_exact(self): + self.assertTrue( + RUNNER.is_gitea_incident_salvage_slice( + "gitea", + RUNNER.GITEA_SALVAGE_ENTRIES, + ) + ) + self.assertFalse( + RUNNER.is_gitea_fresh_install_slice( + "gitea", + RUNNER.GITEA_SALVAGE_ENTRIES, + ) + ) + for relative in RUNNER.GITEA_SALVAGE_ENTRIES: + self.assertTrue(RUNNER.allowed_payload_path("gitea", relative)) + for rejected in ( + "data/gitea/gitea.db", + "config/app.ini", + "secrets/secret-key", + "deployment/gitea-incident-salvage/refs.decisions.csv", + "repositories/dctouch/repo.git/objects/aa/object", + ): + with self.assertRaises(RUNNER.DeployError): + RUNNER.allowed_payload_path("gitea", rejected) + + def test_apply_gates_before_root_creation(self): + source = inspect.getsource(RUNNER.apply_artifact) + preflight = source.index("preflight_gitea_incident_salvage(") + bootstrap = source.index("if not root.is_dir()") + mutation = source.index("apply_started = True") + self.assertLess(preflight, bootstrap) + self.assertLess(bootstrap, mutation) + self.assertIn("enforce_apply=True", source) + self.assertIn("defer_bootstrap_root = is_gitea_bootstrap_slice", source) + + def test_executed_apply_preflight_freezes_before_candidate_root_creation(self): + disposition, refs, unsupported, topics = disposition_evidence_fixture() + with tempfile.TemporaryDirectory( + prefix="gitea-salvage-apply-freeze-" + ) as directory: + root = Path(directory) + payload = root / "payload" + candidate = root / "candidate" + for relative, source in ( + (RUNNER.GITEA_COMPOSE_REL, COMPOSE_PATH), + (RUNNER.GITEA_SALVAGE_DESCRIPTOR_REL, DESCRIPTOR_PATH), + (RUNNER.GITEA_SALVAGE_DISPOSITION_REL, DISPOSITION_PATH), + ( + RUNNER.GITEA_SALVAGE_CLOSURE_DISPOSITION_REL, + CLOSURE_DISPOSITION_PATH, + ), + ( + RUNNER.GITEA_SALVAGE_DECISION_MANIFEST_REL, + DECISION_ROOT / "confirmed-decision.json", + ), + ( + RUNNER.GITEA_SALVAGE_USERS_REL, + DECISION_ROOT / "users.decisions.csv", + ), + ( + RUNNER.GITEA_SALVAGE_REPOSITORIES_REL, + DECISION_ROOT / "repositories.decisions.csv", + ), + ): + target = payload / relative + target.parent.mkdir(parents=True, exist_ok=True) + target.write_bytes(source.read_bytes()) + with ( + mock.patch.object(RUNNER, "GITEA_ROOT", candidate), + mock.patch.object( + RUNNER, + "gitea_compose_project_container_ids", + return_value=[], + ), + mock.patch.object( + RUNNER, + "docker_named_container_inspect_fail_closed", + return_value=None, + ), + mock.patch.object( + RUNNER, + "validate_gitea_salvage_snapshot_boundary", + return_value={ + "database_sha256": ( + RUNNER.GITEA_SALVAGE_SNAPSHOT_DATABASE_SHA256 + ), + "readonly": True, + "uuid": RUNNER.GITEA_SALVAGE_SNAPSHOT_UUID, + }, + ), + mock.patch.object( + RUNNER, + "bind_gitea_salvage_decisions_to_snapshot", + return_value={ + "closure": {}, + "topics": topics, + "unsupported": unsupported, + }, + ), + mock.patch.object( + RUNNER, + "validate_gitea_salvage_closure_evidence", + return_value={ + "blockers": list( + RUNNER.GITEA_SALVAGE_CLOSURE_REMAINING_BLOCKERS + ), + "bytes": 2, + "json": "{}", + "sha256": "1" * 64, + }, + ), + mock.patch.object( + RUNNER, + "inventory_gitea_salvage_repository_refs", + return_value=refs, + ), + mock.patch.object( + RUNNER, + "validate_gitea_salvage_legacy_container", + return_value={"state": "stopped-restart-no"}, + ), + self.assertRaisesRegex( + RUNNER.DeployError, + "activation is frozen before candidate root creation", + ), + ): + RUNNER.preflight_gitea_incident_salvage( + payload, + enforce_apply=True, + ) + self.assertFalse(candidate.exists()) + self.assertEqual(disposition["activation"]["applyFrozen"], True) + + def test_reviewed_legacy_and_disposition_evidence_pins_are_exact(self): + self.assertEqual( + RUNNER.GITEA_SALVAGE_EXPECTED_LEGACY_IMAGE, + "gitea/gitea:latest", + ) + self.assertEqual( + RUNNER.GITEA_SALVAGE_EXPECTED_LEGACY_IMAGE_ID, + "sha256:bf95d9a45ce4fe38b027d051cdc4a4bc531513489fa6244af4074efbb1c376d6", + ) + self.assertIsNone(RUNNER.GITEA_SALVAGE_EXPECTED_REF_MANIFEST_SHA256) + self.assertEqual( + RUNNER.GITEA_SALVAGE_DISPOSITION_REFERENCE_MANIFEST_SHA256, + "9cddaf0e4d4cf22dd264a6ae589ccc50d29e07f85c55e9d34b14627cecb8a311", + ) + self.assertEqual( + RUNNER.GITEA_SALVAGE_EXPECTED_UNSUPPORTED_REPORT_SHA256, + "4b2cecf88c62fc5c4a43419885e88a01c9f9aac03133afb19dae0a7caef106ac", + ) + self.assertEqual( + RUNNER.GITEA_SALVAGE_DISPOSITION_UNSUPPORTED_SCHEMA_SHA256, + "b5e3b6776926c4f1627fafd882362ed0ef986bfc86fc6ac6507a43976531b6db", + ) + # The source report itself remains byte-for-byte historical evidence; + # review closure is represented by the successor disposition contract. + self.assertIsNone(RUNNER.GITEA_SALVAGE_EXPECTED_UNSUPPORTED_SCHEMA_SHA256) + container = { + "Config": {"Image": "gitea/gitea:latest"}, + "HostConfig": {"RestartPolicy": {"Name": "no"}}, + "Image": RUNNER.GITEA_SALVAGE_EXPECTED_LEGACY_IMAGE_ID, + "Mounts": [ + { + "Destination": "/data", + "RW": True, + "Source": "/volume1/docker/gitea", + "Type": "bind", + } + ], + "Name": "/gitea", + "State": {"Running": False}, + } + with mock.patch.object( + RUNNER, + "docker_named_container_inspect_fail_closed", + return_value=container, + ): + result = RUNNER.validate_gitea_salvage_legacy_container() + self.assertEqual(result["state"], "stopped-restart-no") + preflight_source = inspect.getsource(RUNNER.preflight_gitea_incident_salvage) + self.assertLess( + preflight_source.index("validate_gitea_salvage_legacy_container()"), + preflight_source.index("if not enforce_apply"), + ) + for field, drift in ( + ("Name", "other"), + ("Image", "sha256:" + "0" * 64), + ("State", {"Running": True}), + ("Mounts", []), + ): + candidate = json.loads(json.dumps(container)) + candidate[field] = drift + with ( + self.subTest(field=field), + mock.patch.object( + RUNNER, + "docker_named_container_inspect_fail_closed", + return_value=candidate, + ), + self.assertRaisesRegex( + RUNNER.DeployError, + "identity/isolation mismatch", + ), + ): + RUNNER.validate_gitea_salvage_legacy_container() + + def test_confirmed_disposition_matches_exact_observed_evidence(self): + disposition, refs, unsupported, topics = disposition_evidence_fixture() + result = RUNNER.validate_gitea_salvage_disposition_evidence( + disposition, + refs, + unsupported, + topics, + ) + self.assertEqual(result["forensic_refs"], 105) + self.assertEqual(result["live_refs"], 93) + self.assertEqual(result["archive_only_refs"], 12) + self.assertEqual( + result["blockers"], + list(RUNNER.GITEA_SALVAGE_DISPOSITION_REMAINING_BLOCKERS), + ) + self.assertIn("unsupported-schema-catalog-verifier-pending", result["blockers"]) + self.assertIn("reference-manifest-fsck-reachability-verifier-pending", result["blockers"]) + + drift_cases = [] + drift_refs = json.loads(json.dumps(refs)) + drift_refs["manifest"]["repositories"][0]["refs"][0]["oid"] = "f" * 40 + drift_cases.append((drift_refs, unsupported, topics, "reference")) + drift_unsupported = json.loads(json.dumps(unsupported)) + drift_unsupported["report"]["aggregates"]["direct_relation_counts"][ + "issues" + ] += 1 + drift_cases.append((refs, drift_unsupported, topics, "direct-relation")) + drift_coverage = json.loads(json.dumps(unsupported)) + drift_coverage["report"]["coverage"][ + "schema_only_unreviewed_tables" + ].pop() + drift_cases.append((refs, drift_coverage, topics, "schema-only")) + drift_units = json.loads(json.dumps(unsupported)) + drift_units["report"]["per_repository"][0]["repo_unit_types"].pop("9") + drift_cases.append((refs, drift_units, topics, "repo-unit")) + drift_topics = json.loads(json.dumps(topics)) + drift_topics["evidence"]["serialized_nulls"] = 44 + drift_cases.append((refs, unsupported, drift_topics, "semantic-topics")) + drift_topic_row = json.loads(json.dumps(topics)) + drift_topic_row["evidence"]["repositories"][0]["encoding"] = ( + "json-array" + ) + drift_cases.append( + (refs, unsupported, drift_topic_row, "semantic-topics-row") + ) + for drift_refs, drift_report, drift_topic, label in drift_cases: + with self.subTest(label=label), self.assertRaises(RUNNER.DeployError): + RUNNER.validate_gitea_salvage_disposition_evidence( + disposition, + drift_refs, + drift_report, + drift_topic, + ) + + def test_semantic_topics_parser_accepts_only_canonical_gitea_topics(self): + self.assertEqual(RUNNER.gitea_salvage_parse_semantic_topics("null"), ()) + self.assertEqual(RUNNER.gitea_salvage_parse_semantic_topics("[]"), ()) + self.assertEqual( + RUNNER.gitea_salvage_parse_semantic_topics('["alpha","beta-2"]'), + ("alpha", "beta-2"), + ) + for value in ( + None, + "", + " null ", + "NULL", + '"null"', + "{}", + "0", + "true", + "false", + " [] ", + '["Beta"]', + '["beta","alpha"]', + '["alpha","alpha"]', + '["unsafe/topic"]', + '["' + "a" * 36 + '"]', + ): + with self.subTest(value=value), self.assertRaises(RUNNER.DeployError): + RUNNER.gitea_salvage_parse_semantic_topics(value) + + connection = sqlite3.connect(":memory:") + try: + connection.execute("CREATE TABLE repository (id INTEGER, topics TEXT)") + connection.executemany( + "INSERT INTO repository (id,topics) VALUES (?,?)", + ((7, "null"), (9, "null")), + ) + evidence = RUNNER.gitea_salvage_semantic_topics_inventory( + connection, + ({"repo_id": 7}, {"repo_id": 9}), + )["evidence"] + finally: + connection.close() + self.assertEqual(evidence["serialized_arrays"], 0) + self.assertEqual(evidence["serialized_nulls"], 2) + self.assertEqual( + [row["encoding"] for row in evidence["repositories"]], + ["json-null", "json-null"], + ) + self.assertEqual(evidence["material_repositories"], 0) + self.assertEqual(evidence["topics"], 0) + + def test_closure_inventory_is_per_repo_deterministic_and_payload_free(self): + ( + connection, + decisions, + unsupported, + topics, + repo_id, + _deleted_user, + ) = populated_closure_evidence_fixture() + try: + first = run_closure_inventory_fixture( + connection, + decisions, + unsupported, + topics, + ) + second = run_closure_inventory_fixture( + connection, + decisions, + unsupported, + topics, + ) + finally: + connection.close() + self.assertEqual(first["sha256"], second["sha256"]) + self.assertEqual(first["json"], second["json"]) + report = first["report"] + self.assertEqual(report["scope"]["kept_users"], 10) + self.assertEqual(report["scope"]["deleted_users"], 962) + self.assertEqual(len(report["per_repository"]), 45) + per_repo = { + row["old_repo_id"]: row for row in report["per_repository"] + } + self.assertEqual( + per_repo[repo_id]["closure"]["issues_ordinary"]["rows"], + 1, + ) + self.assertEqual( + per_repo[repo_id]["closure"]["pull_request_wrappers"]["rows"], + 1, + ) + self.assertEqual(report["aggregates"]["comments"]["rows"], 1) + self.assertEqual( + report["issue_state_totals"], + { + "ordinary_closed": 0, + "ordinary_open": 1, + "pull_wrapper_closed": 1, + "pull_wrapper_open": 0, + }, + ) + self.assertEqual(report["pull_state_totals"], {"merged": 0, "unmerged": 1}) + self.assertEqual( + report["aggregates"]["comments"]["text_bytes"]["content"], + len("SECRET_COMMENT"), + ) + self.assertEqual(report["aggregates"]["packages"]["rows"], 0) + self.assertEqual(report["aggregates"]["action_runs"]["rows"], 0) + self.assertEqual( + report["actor_relation_counts"]["collaborations"], + {"deleted": 1, "kept": 1, "rows": 2}, + ) + self.assertEqual( + { + row["actor_class"]: row["disposition"] + for row in report["actor_relations"] + if row["relation"] == "collaborations" + }, + { + "deleted": "DROP_DELETED_ACTOR", + "kept": "RECREATE_KEPT_ACTOR_AFTER_ID_MAP", + }, + ) + self.assertEqual(report["attachment_summary"]["rows"], 1) + self.assertEqual( + report["attachment_manifest"][0]["link_class"], + "multi-link", + ) + self.assertEqual( + report["attachment_manifest"][0]["content_hash"], + "unavailable-in-schema", + ) + for sentinel in ( + "SECRET_ISSUE_TITLE", + "SECRET_ISSUE_BODY", + "SECRET_COMMENT", + "SECRET_PATCH", + "SECRET_EXTERNAL_AUTHOR", + "SECRET_REVIEW_EXTERNAL_AUTHOR", + "SECRET_ATTACHMENT_NAME", + "SECRET_UNIT_CONFIG", + ): + self.assertNotIn(sentinel, first["json"]) + scalar_values = [] + + def collect_scalar_values(value): + if isinstance(value, dict): + for child in value.values(): + collect_scalar_values(child) + return + if isinstance(value, list): + for child in value: + collect_scalar_values(child) + return + scalar_values.append(value) + + collect_scalar_values(report) + self.assertNotIn(987654, scalar_values) + self.assertNotIn(456789, scalar_values) + self.assertFalse(report["privacy_contract"]["payload_values_selected"]) + schema_coverage = { + row["table"]: row["columns"] + for row in report["schema_coverage"] + } + self.assertEqual( + schema_coverage["secret"], + ["id", "repo_id"], + ) + self.assertEqual( + schema_coverage["package_property"], + ["id", "ref_id", "ref_type"], + ) + self.assertNotIn("token_hash", schema_coverage["action_runner"]) + self.assertNotIn("name", schema_coverage["attachment"]) + self.assertIn( + "closure-report-review-pin-pending", + report["remaining_blockers"], + ) + self.assertNotIn( + "issue-pr-polymorphic-subrelation-verifier-pending", + report["remaining_blockers"], + ) + subrelations = report["issue_pr_subrelations"] + self.assertEqual(subrelations["conditional_hold_blockers"], []) + self.assertEqual(subrelations["holds"], []) + self.assertEqual( + subrelations["aggregates"]["counts"]["comment_label"], + 1, + ) + self.assertEqual( + subrelations["aggregates"]["counts"]["comment_cross_reference"], + 1, + ) + self.assertEqual( + subrelations["aggregates"]["counts"]["content_history_comment"], + 1, + ) + self.assertEqual( + subrelations["aggregates"]["external_author_provenance"][ + "comment" + ]["rows_with_id"], + 1, + ) + self.assertEqual( + subrelations["extra_schema_coverage"][0]["table"], + "team", + ) + + def test_closure_evidence_validator_binds_policy_and_rejects_drift(self): + ( + connection, + decisions, + unsupported, + topics, + _repo_id, + _deleted_user, + ) = populated_closure_evidence_fixture() + try: + closure = run_closure_inventory_fixture( + connection, + decisions, + unsupported, + topics, + ) + finally: + connection.close() + disposition = json.loads( + CLOSURE_DISPOSITION_PATH.read_text(encoding="utf-8") + ) + disposition["sourceEvidence"]["semanticTopicsSha256"] = topics[ + "sha256" + ] + disposition["sourceEvidence"]["unsupportedRepositoryReportSha256"] = ( + unsupported["sha256"] + ) + disposition["sourceEvidence"]["unsupportedSchemaCatalogSha256"] = ( + unsupported["report"]["schema_catalog_sha256"] + ) + result = RUNNER.validate_gitea_salvage_closure_evidence( + disposition, + closure, + ) + self.assertEqual(result["sha256"], closure["sha256"]) + self.assertIn( + "closure-report-review-pin-pending", + result["blockers"], + ) + drift = json.loads(json.dumps(closure)) + deleted = next( + row + for row in drift["report"]["actor_relations"] + if row["relation"] == "collaborations" + and row["actor_class"] == "deleted" + ) + deleted["disposition"] = "RECREATE_KEPT_ACTOR_AFTER_ID_MAP" + with self.assertRaisesRegex( + RUNNER.DeployError, + "actor-relation disposition is invalid", + ): + RUNNER.validate_gitea_salvage_closure_evidence( + disposition, + drift, + ) + + def test_closure_inventory_fails_closed_on_actor_and_join_drift(self): + cases = ( + ( + "orphan-actor", + "UPDATE comment SET poster_id=999999 WHERE id=300", + "actor is outside user decisions", + ), + ( + "unsafe-attachment-uuid", + "UPDATE attachment SET uuid='12345678-1234-1234-1234-123456789ABC' " + "WHERE id=700", + "attachment manifest row is invalid", + ), + ( + "cross-repository-label", + "UPDATE label SET repo_id=3 WHERE id=400", + "issue-label repository relation is invalid", + ), + ( + "orphan-issue-dependency-target", + "UPDATE issue_dependency SET dependency_id=999999 WHERE id=311", + "issue-dependency row is invalid", + ), + ( + "orphan-project-link-target", + "UPDATE project_issue SET project_id=999999 WHERE id=502", + "project-issue relation is invalid", + ), + ) + for label, mutation, message in cases: + ( + connection, + decisions, + unsupported, + topics, + _repo_id, + _deleted_user, + ) = populated_closure_evidence_fixture() + try: + connection.execute(mutation) + with ( + self.subTest(label=label), + self.assertRaisesRegex(RUNNER.DeployError, message), + ): + run_closure_inventory_fixture( + connection, + decisions, + unsupported, + topics, + ) + finally: + connection.close() + + def test_closure_inventory_polymorphic_subrelations_fail_closed(self): + cases = ( + ( + "orphan-comment-label", + ("UPDATE comment SET label_id=999999 WHERE id=300",), + "comment_label relation is invalid", + ), + ( + "cross-repository-comment-project", + ( + "INSERT INTO project " + "(id,repo_id,creator_id,title,description) " + "VALUES (510,{second_repo_id},{kept_user},'SECRET','SECRET')", + "UPDATE comment SET project_id=510 WHERE id=300", + ), + "comment_current_project relation is invalid", + ), + ( + "wrong-issue-tracked-time", + ( + "INSERT INTO tracked_time VALUES (313,101,{kept_user},15)", + "UPDATE comment SET time_id=313 WHERE id=300", + ), + "comment tracked-time relation is invalid", + ), + ( + "orphan-dependent-issue", + ( + "UPDATE comment SET dependent_issue_id=999999 WHERE id=300", + ), + "comment dependent-issue relation is invalid", + ), + ( + "wrong-issue-review", + ("UPDATE comment SET review_id=306 WHERE id=300",), + "comment review relation is invalid", + ), + ( + "cross-reference-repository-mismatch", + ( + "UPDATE comment SET ref_repo_id={second_repo_id} WHERE id=300", + ), + "comment cross-reference relation is invalid", + ), + ( + "cross-reference-comment-mismatch", + ( + "INSERT INTO comment " + "(id,issue_id,poster_id,assignee_team_id,content) " + "VALUES (320,101,{kept_user},0,'SECRET_REF_COMMENT')", + "UPDATE comment SET ref_comment_id=320 WHERE id=300", + ), + "comment cross-reference comment is invalid", + ), + ( + "unknown-comment-assignee", + ("UPDATE comment SET assignee_id=999999 WHERE id=300",), + "actor is outside user decisions", + ), + ( + "orphan-history-comment", + ( + "UPDATE issue_content_history SET comment_id=999999 " + "WHERE id=302", + ), + "content-history comment relation is invalid", + ), + ( + "unmerged-pull-with-merger", + ( + "UPDATE pull_request SET merger_id={kept_user} WHERE id=200", + ), + "pull merger identity is invalid", + ), + ( + "merged-pull-unknown-merger", + ( + "UPDATE pull_request SET has_merged=1,merger_id=999999 " + "WHERE id=200", + ), + "actor is outside user decisions", + ), + ( + "orphan-comment-team", + ( + "UPDATE comment SET assignee_id=0,assignee_team_id=999999 " + "WHERE id=300", + ), + "comment assignee-team relation is orphaned", + ), + ( + "orphan-review-team", + ( + "UPDATE review SET reviewer_id=0,reviewer_team_id=999999 " + "WHERE id=306", + ), + "review reviewer-team relation is orphaned", + ), + ( + "conflicting-comment-assignee-identities", + ( + "INSERT INTO team VALUES (10,{kept_user})", + "UPDATE comment SET assignee_team_id=10 WHERE id=300", + ), + "comment assignee identities conflict", + ), + ( + "conflicting-reviewer-identities", + ( + "INSERT INTO team VALUES (10,{kept_user})", + "UPDATE review SET reviewer_team_id=10 WHERE id=306", + ), + "review subrelation source is invalid", + ), + ( + "team-organization-outside-decisions", + ( + "INSERT INTO team VALUES (10,999999)", + "UPDATE comment SET assignee_id=0,assignee_team_id=10 " + "WHERE id=300", + ), + "actor is outside user decisions", + ), + ( + "invalid-external-author-id", + ( + "UPDATE comment SET original_author_id='invalid' WHERE id=300", + ), + "comment subrelation value is invalid", + ), + ( + "invalid-cross-reference-action", + ("UPDATE comment SET ref_action=9 WHERE id=300",), + "comment cross-reference state is invalid", + ), + ) + for label, statements, message in cases: + ( + connection, + decisions, + unsupported, + topics, + repo_id, + _deleted_user, + ) = populated_closure_evidence_fixture() + kept_ids = sorted( + int(row["repo_id"]) + for row in decisions["kept_repositories"] + ) + kept_user = next( + int(row["user_id"]) + for row in decisions["users"] + if row["decision"] == "KEEP_ACTIVE" + ) + try: + for statement in statements: + connection.execute( + statement.format( + kept_user=kept_user, + repo_id=repo_id, + second_repo_id=kept_ids[1], + ) + ) + with ( + self.subTest(label=label), + self.assertRaisesRegex(RUNNER.DeployError, message), + ): + run_closure_inventory_fixture( + connection, + decisions, + unsupported, + topics, + ) + finally: + connection.close() + + def test_closure_inventory_seals_exact_team_mapping_holds(self): + ( + connection, + decisions, + unsupported, + topics, + repo_id, + deleted_user, + ) = populated_closure_evidence_fixture() + try: + connection.execute("INSERT INTO team VALUES (10,?)", (deleted_user,)) + connection.execute( + "UPDATE comment SET assignee_id=0,assignee_team_id=10 " + "WHERE id=300" + ) + connection.execute( + "UPDATE review SET reviewer_id=0,reviewer_team_id=10 " + "WHERE id=306" + ) + closure = run_closure_inventory_fixture( + connection, + decisions, + unsupported, + topics, + ) + finally: + connection.close() + subrelations = closure["report"]["issue_pr_subrelations"] + self.assertEqual( + subrelations["conditional_hold_blockers"], + ["issue-pr-team-mapping-hold"], + ) + self.assertEqual( + [row["kind"] for row in subrelations["holds"]], + [ + "comment-assignee-team-mapping", + "review-reviewer-team-mapping", + ], + ) + for row in subrelations["holds"]: + self.assertEqual(row["old_org_id"], deleted_user) + self.assertEqual(row["old_org_identity_class"], "deleted") + self.assertEqual(row["old_repo_id"], repo_id) + self.assertEqual(row["old_team_id"], 10) + self.assertIn( + "issue-pr-team-mapping-hold", + closure["report"]["remaining_blockers"], + ) + disposition = json.loads( + CLOSURE_DISPOSITION_PATH.read_text(encoding="utf-8") + ) + disposition["sourceEvidence"]["semanticTopicsSha256"] = topics[ + "sha256" + ] + disposition["sourceEvidence"][ + "unsupportedRepositoryReportSha256" + ] = unsupported["sha256"] + disposition["sourceEvidence"]["unsupportedSchemaCatalogSha256"] = ( + unsupported["report"]["schema_catalog_sha256"] + ) + validated = RUNNER.validate_gitea_salvage_closure_evidence( + disposition, + closure, + ) + self.assertIn("issue-pr-team-mapping-hold", validated["blockers"]) + drift = json.loads(json.dumps(closure)) + drift["report"]["issue_pr_subrelations"]["holds"][0][ + "kind" + ] = "unknown-team-mapping" + with self.assertRaisesRegex( + RUNNER.DeployError, + "hold row is invalid", + ): + RUNNER.validate_gitea_salvage_closure_evidence( + disposition, + drift, + ) + + def test_closure_inventory_attests_nullable_subrelation_encodings(self): + ( + connection, + decisions, + unsupported, + topics, + _repo_id, + _deleted_user, + ) = populated_closure_evidence_fixture() + try: + connection.execute( + "UPDATE comment SET original_author=NULL,original_author_id=NULL " + "WHERE id=300" + ) + connection.execute( + "UPDATE review SET original_author=NULL,original_author_id=NULL " + "WHERE id=306" + ) + connection.execute( + "UPDATE issue_content_history SET comment_id=NULL WHERE id=302" + ) + connection.execute( + "UPDATE pull_request SET merger_id=NULL WHERE id=200" + ) + report = run_closure_inventory_fixture( + connection, + decisions, + unsupported, + topics, + )["report"] + finally: + connection.close() + aggregates = report["issue_pr_subrelations"]["aggregates"] + self.assertEqual(aggregates["counts"]["content_history_comment"], 0) + self.assertEqual(aggregates["counts"]["content_history_issue"], 1) + self.assertEqual( + aggregates["null_encodings"]["comment.original_author_id"], + 1, + ) + self.assertEqual( + aggregates["null_encodings"]["review.original_author_id"], + 1, + ) + self.assertEqual( + aggregates["null_encodings"]["pull_request.merger_id"], + 1, + ) + + def test_closure_inventory_team_schema_gate_rejects_absence_and_views(self): + cases = ( + ("missing", ("DROP TABLE team",)), + ( + "view", + ( + "DROP TABLE team", + "CREATE VIEW team AS SELECT 1 AS id,1 AS org_id", + ), + ), + ( + "missing-org-id", + ( + "DROP TABLE team", + "CREATE TABLE team (id INTEGER PRIMARY KEY)", + ), + ), + ) + for label, statements in cases: + ( + connection, + decisions, + unsupported, + topics, + _repo_id, + _deleted_user, + ) = populated_closure_evidence_fixture() + try: + for statement in statements: + connection.execute(statement) + with ( + self.subTest(label=label), + self.assertRaisesRegex( + RUNNER.DeployError, + "extra schema is unsafe", + ), + ): + run_closure_inventory_fixture( + connection, + decisions, + unsupported, + topics, + ) + finally: + connection.close() + + def test_closure_inventory_packages_actions_are_metadata_only(self): + ( + connection, + decisions, + unsupported, + topics, + repo_id, + _deleted_user, + ) = populated_closure_evidence_fixture() + kept_user = next( + int(row["user_id"]) + for row in decisions["users"] + if row["decision"] == "KEEP_ACTIVE" + ) + try: + connection.execute( + "INSERT INTO package VALUES (900,?,?,'SECRET_PACKAGE_NAME')", + (repo_id, kept_user), + ) + connection.execute( + "INSERT INTO package_version VALUES " + "(901,900,?,'SECRET_VERSION','SECRET_PACKAGE_METADATA')", + (kept_user,), + ) + connection.execute( + "INSERT INTO package_blob VALUES (902,100,'SECRET_BLOB_HASH')" + ) + connection.execute( + "INSERT INTO package_file VALUES (903,901,902,'SECRET_FILE_NAME')" + ) + connection.executemany( + "INSERT INTO package_property VALUES (?,?,?,?,?)", + ( + (904, 2, 900, "SECRET_PROP", "SECRET_VALUE"), + (905, 0, 901, "SECRET_PROP", "SECRET_VALUE"), + (906, 1, 903, "SECRET_PROP", "SECRET_VALUE"), + ), + ) + connection.execute( + "INSERT INTO action_run VALUES " + "(1000,?,?,'SECRET_ACTION_EVENT_PAYLOAD')", + (repo_id, kept_user), + ) + connection.execute( + "INSERT INTO action_schedule VALUES " + "(1001,?,?,CAST('SECRET_WORKFLOW' AS BLOB))", + (repo_id, kept_user), + ) + connection.execute( + "INSERT INTO action_runner VALUES " + "(1002,?,'SECRET_RUNNER_TOKEN','SECRET_RUNNER_SALT')", + (repo_id,), + ) + connection.execute( + "INSERT INTO action_variable VALUES (1003,?,'SECRET_VARIABLE')", + (repo_id,), + ) + connection.execute( + "INSERT INTO secret VALUES (1004,?,'SECRET_ACTION_SECRET')", + (repo_id,), + ) + connection.execute( + "INSERT INTO action_artifact VALUES " + "(1005,?,1000,20,10,'SECRET_STORAGE_PATH')", + (repo_id,), + ) + connection.execute( + "INSERT INTO action_run_job VALUES " + "(1006,?,1000,CAST('SECRET_JOB_PAYLOAD' AS BLOB))", + (repo_id,), + ) + connection.execute( + "INSERT INTO action_task VALUES " + "(1007,?,1006,50,40,'SECRET_TASK_TOKEN','SECRET_LOG_PATH')", + (repo_id,), + ) + connection.execute( + "INSERT INTO action_run_index VALUES (?,1)", + (repo_id,), + ) + direct = unsupported["report"]["aggregates"][ + "direct_relation_counts" + ] + for label in ( + "action_runners", + "action_runs", + "action_schedules", + "action_secrets", + "action_variables", + "packages", + ): + direct[label] = 1 + unsupported.update( + RUNNER.canonical_gitea_salvage_evidence( + unsupported["report"], + "closure package action predecessor report", + ) + ) + evidence = run_closure_inventory_fixture( + connection, + decisions, + unsupported, + topics, + ) + connection.execute( + "INSERT INTO package_property VALUES " + "(907,9,900,'SECRET_UNKNOWN_PROP','SECRET_UNKNOWN_VALUE')" + ) + with self.assertRaisesRegex( + RUNNER.DeployError, + "package-property type is invalid", + ): + run_closure_inventory_fixture( + connection, + decisions, + unsupported, + topics, + ) + connection.execute("DELETE FROM package_property WHERE id=907") + deleted_repo_id = next( + int(row["repo_id"]) + for row in decisions["repositories"] + if row["decision"] == "DELETE" + ) + connection.execute( + "INSERT INTO action_artifact VALUES " + "(1008,?,1000,1,1,'SECRET_INDIRECT_PATH')", + (deleted_repo_id,), + ) + with self.assertRaisesRegex( + RUNNER.DeployError, + "action_artifact indirect relation is invalid", + ): + run_closure_inventory_fixture( + connection, + decisions, + unsupported, + topics, + ) + finally: + connection.close() + aggregates = evidence["report"]["aggregates"] + self.assertEqual(aggregates["packages"]["rows"], 1) + self.assertEqual(aggregates["package_versions"]["rows"], 1) + self.assertEqual(aggregates["package_files"]["rows"], 1) + self.assertEqual(aggregates["package_properties"]["rows"], 3) + self.assertEqual(aggregates["package_blobs"]["logical_bytes"], 100) + self.assertEqual(aggregates["action_artifacts"]["logical_bytes"], 20) + self.assertEqual( + aggregates["action_artifacts"]["numeric_totals"], + {"file_compressed_size": 10, "file_size": 20}, + ) + self.assertEqual(aggregates["action_tasks"]["logical_bytes"], 40) + self.assertEqual( + aggregates["action_tasks"]["numeric_totals"], + {"log_length": 50, "log_size": 40}, + ) + self.assertEqual(aggregates["action_run_indexes"]["rows"], 1) + for sentinel in ( + "SECRET_PACKAGE_NAME", + "SECRET_VERSION", + "SECRET_PACKAGE_METADATA", + "SECRET_BLOB_HASH", + "SECRET_PROP", + "SECRET_VALUE", + "SECRET_ACTION_EVENT_PAYLOAD", + "SECRET_WORKFLOW", + "SECRET_RUNNER_TOKEN", + "SECRET_VARIABLE", + "SECRET_ACTION_SECRET", + "SECRET_STORAGE_PATH", + "SECRET_JOB_PAYLOAD", + "SECRET_TASK_TOKEN", + "SECRET_LOG_PATH", + ): + self.assertNotIn(sentinel, evidence["json"]) + + def test_closure_inventory_actions_rejects_null_indirect_repository(self): + cases = ( + ( + "artifact", + ( + "INSERT INTO action_artifact VALUES " + "(1001,NULL,1000,1,1,'SECRET_NULL_ARTIFACT')" + ), + "action_artifact indirect relation is invalid", + ), + ( + "job", + ( + "INSERT INTO action_run_job VALUES " + "(1001,NULL,1000,CAST('SECRET_NULL_JOB' AS BLOB))" + ), + "action_run_job indirect relation is invalid", + ), + ( + "task", + ( + "INSERT INTO action_run_job VALUES " + "(1001,{repo_id},1000,CAST('SECRET_JOB' AS BLOB));" + "INSERT INTO action_task VALUES " + "(1002,NULL,1001,1,1,'SECRET_NULL_TASK','SECRET_NULL_LOG')" + ), + "action_task indirect relation is invalid", + ), + ) + for label, mutation, message in cases: + ( + connection, + decisions, + unsupported, + topics, + repo_id, + _deleted_user, + ) = populated_closure_evidence_fixture() + kept_user = next( + int(row["user_id"]) + for row in decisions["users"] + if row["decision"] == "KEEP_ACTIVE" + ) + try: + connection.execute( + "INSERT INTO action_run VALUES (1000,?,?,?)", + (repo_id, kept_user, "SECRET_RUN"), + ) + for statement in mutation.format(repo_id=repo_id).split(";"): + connection.execute(statement) + unsupported["report"]["aggregates"]["direct_relation_counts"][ + "action_runs" + ] = 1 + unsupported.update( + RUNNER.canonical_gitea_salvage_evidence( + unsupported["report"], + "closure null indirect repository predecessor report", + ) + ) + with ( + self.subTest(label=label), + self.assertRaisesRegex(RUNNER.DeployError, message), + ): + run_closure_inventory_fixture( + connection, + decisions, + unsupported, + topics, + ) + finally: + connection.close() + + def test_closure_inventory_actions_bounds_every_declared_numeric_value(self): + cases = ( + ( + "negative-compressed-size", + "INSERT INTO action_artifact VALUES " + "(1001,{repo_id},1000,1,-1,'SECRET_ARTIFACT')", + ), + ( + "oversized-log-length", + "INSERT INTO action_run_job VALUES " + "(1001,{repo_id},1000,CAST('SECRET_JOB' AS BLOB));" + "INSERT INTO action_task VALUES " + "(1002,{repo_id},1001,{oversized},1,'SECRET_TASK','SECRET_LOG')", + ), + ) + for label, mutation in cases: + ( + connection, + decisions, + unsupported, + topics, + repo_id, + _deleted_user, + ) = populated_closure_evidence_fixture() + kept_user = next( + int(row["user_id"]) + for row in decisions["users"] + if row["decision"] == "KEEP_ACTIVE" + ) + try: + connection.execute( + "INSERT INTO action_run VALUES (1000,?,?,?)", + (repo_id, kept_user, "SECRET_RUN"), + ) + for statement in mutation.format( + repo_id=repo_id, + oversized=( + RUNNER.GITEA_SALVAGE_UNSUPPORTED_SIZE_PER_ROW_MAX_BYTES + + 1 + ), + ).split(";"): + connection.execute(statement) + unsupported["report"]["aggregates"]["direct_relation_counts"][ + "action_runs" + ] = 1 + unsupported.update( + RUNNER.canonical_gitea_salvage_evidence( + unsupported["report"], + "closure numeric bounds predecessor report", + ) + ) + with ( + self.subTest(label=label), + self.assertRaisesRegex( + RUNNER.DeployError, + "declared value is invalid", + ), + ): + run_closure_inventory_fixture( + connection, + decisions, + unsupported, + topics, + ) + finally: + connection.close() + + def test_container_absence_parser_rejects_daemon_and_permission_errors(self): + with mock.patch.object( + RUNNER.subprocess, + "run", + return_value=completed( + 1, + stderr="Error response from daemon: No such container: candidate", + ), + ): + self.assertIsNone( + RUNNER.docker_named_container_inspect_fail_closed( + "candidate", + "candidate inspect", + ) + ) + for error in ( + "permission denied while trying to connect to the Docker daemon", + "context deadline exceeded", + "Cannot connect to the Docker daemon", + ): + with ( + self.subTest(error=error), + mock.patch.object( + RUNNER.subprocess, + "run", + return_value=completed(1, stdout="[]\n", stderr=error), + ), + ): + with self.assertRaisesRegex( + RUNNER.DeployError, + r"absence is unproven: rc=1 stdout=json-empty-list " + r"stderr=(?:permission-denied|timeout|daemon-error)", + ): + RUNNER.docker_named_container_inspect_fail_closed( + "candidate", + "candidate inspect", + ) + + def test_container_absence_parser_accepts_exact_json_empty_list_live_format_only(self): + for stderr in ( + "Error: No such object: candidate\n", + "Error response from daemon: No such container: candidate\n", + ): + with ( + self.subTest(stderr=stderr), + mock.patch.object( + RUNNER.subprocess, + "run", + return_value=completed(1, stdout="[]\n", stderr=stderr), + ), + ): + self.assertIsNone( + RUNNER.docker_named_container_inspect_fail_closed( + "candidate", + "candidate inspect", + ) + ) + + for stdout, stderr, expected_stdout, expected_stderr in ( + ( + "[]\n", + "Error: No such object: different-name\n", + "json-empty-list", + "other", + ), + ( + "[]\n", + "Error: No such object: candidate\nextra diagnostic\n", + "json-empty-list", + "other", + ), + ( + "[{}]\n", + "Error: No such object: candidate\n", + "json-list", + "exact-no-such-object", + ), + ( + "[ ]\n", + "Error: No such object: candidate\n", + "json-list", + "exact-no-such-object", + ), + ( + "not-json\n", + "supersecret unexpected failure\n", + "non-json", + "other", + ), + ): + with ( + self.subTest(stdout=stdout, stderr=stderr), + mock.patch.object( + RUNNER.subprocess, + "run", + return_value=completed(1, stdout=stdout, stderr=stderr), + ), + ): + with self.assertRaises(RUNNER.DeployError) as raised: + RUNNER.docker_named_container_inspect_fail_closed( + "candidate", + "candidate inspect", + ) + message = str(raised.exception) + self.assertIn(f"stdout={expected_stdout}", message) + self.assertIn(f"stderr={expected_stderr}", message) + self.assertNotIn("supersecret", message) + + def test_snapshot_gate_uses_proven_btrfs_property_form_and_accepts_contained_mode(self): + class FakePath: + def __init__(self, value, mode, uid=0, gid=0, size=0, parent=None): + self.value = value + self._stat = type( + "FakeStat", + (), + { + "st_mode": mode, + "st_uid": uid, + "st_gid": gid, + "st_size": size, + }, + )() + self.parent = parent + + def lstat(self): + return self._stat + + def __str__(self): + return self.value + + parent = FakePath("/snapshots", stat.S_IFDIR | 0o700) + root = FakePath("/snapshots/incident", stat.S_IFDIR | 0o755, parent=parent) + database = FakePath( + "/snapshots/incident/gitea/gitea/gitea.db", + stat.S_IFREG | 0o600, + uid=1000, + gid=1000, + size=RUNNER.GITEA_SALVAGE_SNAPSHOT_DATABASE_BYTES, + ) + repositories = FakePath( + "/snapshots/incident/gitea/git/repositories", + stat.S_IFDIR | 0o755, + uid=1000, + gid=1000, + ) + btrfs = mock.MagicMock() + btrfs.is_file.return_value = True + btrfs.__str__.return_value = "/usr/sbin/btrfs" + calls = [] + + def run(command, **_kwargs): + calls.append(command) + if command[1:3] == ["subvolume", "show"]: + return completed( + 0, + stdout=f"UUID: {RUNNER.GITEA_SALVAGE_SNAPSHOT_UUID}\n", + ) + if command[1:3] == ["property", "get"]: + return completed(0, stdout="ro=true\n") + raise AssertionError(command) + + with ( + mock.patch.object(RUNNER, "GITEA_SALVAGE_SNAPSHOT_ROOT", root), + mock.patch.object(RUNNER, "GITEA_SALVAGE_SNAPSHOT_DATABASE", database), + mock.patch.object( + RUNNER, + "GITEA_SALVAGE_SNAPSHOT_REPOSITORIES", + repositories, + ), + mock.patch.object(RUNNER, "GITEA_SALVAGE_BTRFS", btrfs), + mock.patch.object( + RUNNER, + "sha256_file", + return_value=RUNNER.GITEA_SALVAGE_SNAPSHOT_DATABASE_SHA256, + ), + mock.patch.object( + RUNNER, + "validate_gitea_salvage_path_chain", + side_effect=lambda _root, target, *_args, **_kwargs: target, + ), + mock.patch.object(RUNNER.subprocess, "run", side_effect=run), + ): + result = RUNNER.validate_gitea_salvage_snapshot_boundary() + self.assertTrue(result["readonly"]) + self.assertEqual( + calls[1], + ["/usr/sbin/btrfs", "property", "get", str(root), "ro"], + ) + + def test_path_chain_rejects_symlink_parent_and_allows_only_missing_final(self): + with tempfile.TemporaryDirectory(prefix="gitea-salvage-path-chain-") as directory: + base = Path(directory) + root = base / "root" + outside = base / "outside" + root.mkdir() + outside.mkdir() + (outside / "repo.git").mkdir() + (root / "owner").symlink_to(outside, target_is_directory=True) + with ( + mock.patch.object( + RUNNER, + "gitea_salvage_mountpoints", + return_value=set(), + ), + self.assertRaisesRegex(RUNNER.DeployError, "contains a symlink"), + ): + RUNNER.validate_gitea_salvage_path_chain( + root, + root / "owner" / "repo.git", + stat.S_ISDIR, + "repository source owner/repo.git", + ) + + (root / "owner").unlink() + (root / "owner").mkdir() + with mock.patch.object( + RUNNER, + "gitea_salvage_mountpoints", + return_value=set(), + ): + self.assertIsNone( + RUNNER.validate_gitea_salvage_path_chain( + root, + root / "owner" / "missing.wiki.git", + stat.S_ISDIR, + "wiki source owner/missing.wiki.git", + allow_missing_final=True, + ) + ) + + def test_path_chain_rejects_mount_filesystem_and_nested_subvolume_boundaries(self): + with tempfile.TemporaryDirectory(prefix="gitea-salvage-boundary-") as directory: + root = Path(directory) / "root" + owner = root / "owner" + repo = owner / "repo.git" + repo.mkdir(parents=True) + + with ( + mock.patch.object( + RUNNER, + "gitea_salvage_mountpoints", + return_value={str(owner)}, + ), + self.assertRaisesRegex(RUNNER.DeployError, "mount boundary"), + ): + RUNNER.validate_gitea_salvage_path_chain( + root, + repo, + stat.S_ISDIR, + "repository source owner/repo.git", + ) + + original_lstat = Path.lstat + real_owner_stat = owner.lstat() + + def lstat_with_owner(**changes): + values = { + name: getattr(real_owner_stat, name) + for name in dir(real_owner_stat) + if name.startswith("st_") + } + values.update(changes) + return type("BoundaryStat", (), values)() + + def nested_subvolume_lstat(path): + if path == owner: + return lstat_with_owner(st_ino=256) + return original_lstat(path) + + with ( + mock.patch.object( + RUNNER, + "gitea_salvage_mountpoints", + return_value=set(), + ), + mock.patch.object(Path, "lstat", nested_subvolume_lstat), + self.assertRaisesRegex(RUNNER.DeployError, "nested subvolume boundary"), + ): + RUNNER.validate_gitea_salvage_path_chain( + root, + repo, + stat.S_ISDIR, + "repository source owner/repo.git", + ) + + def foreign_device_lstat(path): + if path == owner: + return lstat_with_owner(st_dev=real_owner_stat.st_dev + 1) + return original_lstat(path) + + with ( + mock.patch.object( + RUNNER, + "gitea_salvage_mountpoints", + return_value=set(), + ), + mock.patch.object(Path, "lstat", foreign_device_lstat), + self.assertRaisesRegex(RUNNER.DeployError, "filesystem boundary"), + ): + RUNNER.validate_gitea_salvage_path_chain( + root, + repo, + stat.S_ISDIR, + "repository source owner/repo.git", + ) + + def test_ref_parser_reconstructs_manifest_and_rejects_alternates(self): + with tempfile.TemporaryDirectory(prefix="gitea-salvage-repo-") as directory: + repo = Path(directory) / "owner" / "repo.git" + object_file = repo / "objects" / "aa" / ("b" * 38) + ref_file = repo / "refs" / "heads" / "main" + object_file.parent.mkdir(parents=True) + ref_file.parent.mkdir(parents=True) + object_file.write_bytes(b"object") + oid = "a" * 40 + ref_file.write_text(oid + "\n", encoding="ascii") + (repo / "HEAD").write_text("ref: refs/heads/main\n", encoding="ascii") + with mock.patch.object( + RUNNER, + "gitea_salvage_file_has_nocow", + return_value=False, + ): + inventory = inventory_repository( + repo, + "owner/repo.git", + ) + self.assertEqual(inventory["head"], "refs/heads/main") + self.assertEqual(inventory["refs"], [{"name": "refs/heads/main", "oid": oid}]) + self.assertEqual(inventory["object_format"], "sha1") + alternates = repo / "objects" / "info" / "alternates" + alternates.parent.mkdir() + alternates.write_text("/evil\n", encoding="ascii") + with self.assertRaisesRegex(RUNNER.DeployError, "forbidden Git material"): + inventory_repository( + repo, + "owner/repo.git", + ) + + def test_review_evidence_ascii_escapes_unicode_and_rejects_surrogate_refname(self): + refname = "refs/heads/ветка-\u202ereview" + self.assertTrue(RUNNER.gitea_salvage_refname_is_safe(refname)) + evidence = RUNNER.canonical_gitea_salvage_evidence( + {"ref": refname}, + "test reference manifest", + ) + evidence["json"].encode("ascii") + self.assertNotIn("\u202e", evidence["json"]) + self.assertIn("\\u202e", evidence["json"]) + self.assertIn("\\u0432", evidence["json"]) + self.assertFalse( + RUNNER.gitea_salvage_refname_is_safe("refs/heads/unsafe-\udcff") + ) + + def test_internal_object_and_ref_walk_boundaries_fail_closed(self): + cases = ( + ("objects-root-device", "objects", "device"), + ("object-directory-subvolume", "objects/aa", "subvolume"), + ("object-file-mount", "objects/aa/" + "b" * 38, "mount"), + ("refs-root-device", "refs", "device"), + ("ref-directory-subvolume", "refs/heads", "subvolume"), + ("ref-file-mount", "refs/heads/main", "mount"), + ( + "quarantine-subvolume", + "objects/tmp_objdir-incoming-abc123", + "subvolume", + ), + ) + for label, target_relative, boundary in cases: + with self.subTest(label=label), tempfile.TemporaryDirectory( + prefix="gitea-salvage-internal-boundary-" + ) as directory: + repo = Path(directory) / "repo.git" + object_file = repo / "objects" / "aa" / ("b" * 38) + ref_file = repo / "refs" / "heads" / "main" + object_file.parent.mkdir(parents=True) + ref_file.parent.mkdir(parents=True) + object_file.write_bytes(b"object") + ref_file.write_text("a" * 40 + "\n", encoding="ascii") + (repo / "HEAD").write_text( + "ref: refs/heads/main\n", + encoding="ascii", + ) + if label == "quarantine-subvolume": + (repo / target_relative).mkdir() + target = repo / target_relative + trusted_device = repo.lstat().st_dev + mountpoints = {os.path.normpath(str(target))} if boundary == "mount" else set() + real_lstat = Path.lstat + + def boundary_lstat(path): + value = real_lstat(path) + if path != target or boundary == "mount": + return value + result = mock.Mock() + for attribute in ( + "st_mode", + "st_uid", + "st_gid", + "st_nlink", + "st_size", + "st_dev", + "st_ino", + ): + setattr(result, attribute, getattr(value, attribute)) + if boundary == "device": + result.st_dev = trusted_device + 1 + else: + result.st_ino = 256 + return result + + expected = { + "device": "crosses a filesystem boundary", + "mount": "crosses a mount boundary", + "subvolume": "crosses a nested subvolume boundary", + }[boundary] + with ( + mock.patch.object(Path, "lstat", boundary_lstat), + mock.patch.object( + RUNNER, + "gitea_salvage_file_has_nocow", + return_value=False, + ), + self.assertRaisesRegex(RUNNER.DeployError, expected), + ): + inventory_repository( + repo, + "repo.git", + trusted_device=trusted_device, + mountpoints=mountpoints, + ) + + def test_forbidden_path_probes_never_cross_unvalidated_ancestors(self): + cases = ( + ("objects/info", "objects/info/alternates"), + ("info", "info/grafts"), + ("refs", "refs/replace"), + ) + for boundary_kind in ("symlink", "mount"): + for ancestor_relative, forbidden_relative in cases: + with ( + self.subTest( + boundary_kind=boundary_kind, + ancestor=ancestor_relative, + ), + tempfile.TemporaryDirectory( + prefix="gitea-salvage-forbidden-ancestor-" + ) as directory, + ): + base = Path(directory) + repo = base / "repo.git" + object_file = repo / "objects" / "aa" / ("b" * 38) + object_file.parent.mkdir(parents=True) + object_file.write_bytes(b"object") + (repo / "HEAD").write_text( + "ref: refs/heads/main\n", + encoding="ascii", + ) + ancestor = repo / ancestor_relative + ancestor.parent.mkdir(parents=True, exist_ok=True) + if boundary_kind == "symlink": + outside = base / "outside" + outside.mkdir() + ancestor.symlink_to(outside, target_is_directory=True) + mountpoints = set() + expected = "has an unsafe type" + else: + ancestor.mkdir() + mountpoints = {os.path.normpath(str(ancestor))} + expected = "crosses a mount boundary" + forbidden = repo / forbidden_relative + real_lstat = Path.lstat + + def reject_nested_probe(path): + if path == forbidden: + raise AssertionError( + f"nested forbidden path was probed: {path}" + ) + return real_lstat(path) + + with ( + mock.patch.object(Path, "lstat", reject_nested_probe), + mock.patch.object( + RUNNER, + "gitea_salvage_file_has_nocow", + return_value=False, + ), + self.assertRaisesRegex(RUNNER.DeployError, expected), + ): + inventory_repository( + repo, + "repo.git", + mountpoints=mountpoints, + ) + + def test_object_boundary_requires_pack_idx_pair_and_rejects_promisor(self): + with tempfile.TemporaryDirectory(prefix="gitea-salvage-pack-") as directory: + repo = Path(directory) / "repo.git" + pack_root = repo / "objects" / "pack" + ref_root = repo / "refs" / "heads" + pack_root.mkdir(parents=True) + ref_root.mkdir(parents=True) + digest = "a" * 40 + oid = "b" * 40 + (pack_root / f"pack-{digest}.pack").write_bytes(b"pack") + (ref_root / "main").write_text(oid + "\n", encoding="ascii") + (repo / "HEAD").write_text("ref: refs/heads/main\n", encoding="ascii") + with ( + mock.patch.object( + RUNNER, + "gitea_salvage_file_has_nocow", + return_value=False, + ), + self.assertRaisesRegex(RUNNER.DeployError, "incomplete pack/index pair"), + ): + inventory_repository(repo, "repo.git") + (pack_root / f"pack-{digest}.idx").write_bytes(b"idx") + with mock.patch.object( + RUNNER, + "gitea_salvage_file_has_nocow", + return_value=False, + ): + inventory = inventory_repository( + repo, + "repo.git", + ) + self.assertEqual(inventory["object_files"], 2) + (pack_root / f"pack-{digest}.promisor").write_bytes(b"promisor") + with ( + mock.patch.object( + RUNNER, + "gitea_salvage_file_has_nocow", + return_value=False, + ), + self.assertRaisesRegex( + RUNNER.DeployError, + r"promisor object material is forbidden: repo\.git/objects/pack/" + r"pack-[a-f0-9]+\.promisor", + ), + ): + inventory_repository(repo, "repo.git") + + def test_object_info_packs_is_validated_excluded_derived_cache(self): + with tempfile.TemporaryDirectory(prefix="gitea-salvage-info-packs-") as directory: + repo = Path(directory) / "repo.git" + object_file = repo / "objects" / "aa" / ("b" * 38) + info_packs = repo / "objects" / "info" / "packs" + ref_file = repo / "refs" / "heads" / "main" + object_file.parent.mkdir(parents=True) + info_packs.parent.mkdir(parents=True) + ref_file.parent.mkdir(parents=True) + object_file.write_bytes(b"object") + info_packs.write_text( + f"P pack-{'c' * 40}.pack\n\n", + encoding="ascii", + ) + oid = "a" * 40 + ref_file.write_text(oid + "\n", encoding="ascii") + (repo / "HEAD").write_text("ref: refs/heads/main\n", encoding="ascii") + with mock.patch.object( + RUNNER, + "gitea_salvage_file_has_nocow", + return_value=False, + ): + inventory = inventory_repository( + repo, + "repo.git", + ) + self.assertEqual(inventory["object_files"], 1) + self.assertEqual(inventory["object_bytes"], len(b"object")) + self.assertEqual( + inventory["excluded_derived_files"], + [ + { + "bytes": len(f"P pack-{'c' * 40}.pack\n\n"), + "kind": "dumb-http-pack-list", + "path": "objects/info/packs", + } + ], + ) + self.assertEqual( + inventory["excluded_derived_bytes"], + len(f"P pack-{'c' * 40}.pack\n\n"), + ) + + def test_observed_bitmap_and_commit_graph_are_exact_bounded_exclusions(self): + with tempfile.TemporaryDirectory(prefix="gitea-salvage-derived-") as directory: + repo = Path(directory) / "repo.git" + pack_root = repo / "objects" / "pack" + info_root = repo / "objects" / "info" + ref_root = repo / "refs" / "heads" + pack_root.mkdir(parents=True) + info_root.mkdir(parents=True) + ref_root.mkdir(parents=True) + digest = "a" * 40 + oid = "b" * 40 + pack_bytes = b"pack" + idx_bytes = b"idx" + bitmap_bytes = b"bitmap" + graph_bytes = b"commit-graph" + info_packs_bytes = b"P pack-derived.pack\n\n" + (pack_root / f"pack-{digest}.pack").write_bytes(pack_bytes) + (pack_root / f"pack-{digest}.idx").write_bytes(idx_bytes) + bitmap = pack_root / f"pack-{digest}.bitmap" + commit_graph = info_root / "commit-graph" + info_packs = info_root / "packs" + bitmap.write_bytes(bitmap_bytes) + commit_graph.write_bytes(graph_bytes) + info_packs.write_bytes(info_packs_bytes) + (ref_root / "main").write_text(oid + "\n", encoding="ascii") + (repo / "HEAD").write_text("ref: refs/heads/main\n", encoding="ascii") + real_read_bytes = Path.read_bytes + + def reject_derived_reads(path): + if path in {bitmap, commit_graph, info_packs}: + raise AssertionError(f"derived bytes were read: {path}") + return real_read_bytes(path) + + with ( + mock.patch.object( + RUNNER, + "gitea_salvage_file_has_nocow", + return_value=False, + ), + mock.patch.object(Path, "read_bytes", reject_derived_reads), + ): + inventory = inventory_repository( + repo, + "repo.git", + ) + self.assertEqual(inventory["object_files"], 2) + self.assertEqual(inventory["object_bytes"], len(pack_bytes) + len(idx_bytes)) + self.assertEqual( + inventory["excluded_derived_files"], + [ + { + "bytes": len(graph_bytes), + "kind": "commit-graph", + "path": "objects/info/commit-graph", + }, + { + "bytes": len(info_packs_bytes), + "kind": "dumb-http-pack-list", + "path": "objects/info/packs", + }, + { + "bytes": len(bitmap_bytes), + "kind": "pack-bitmap", + "path": f"objects/pack/pack-{digest}.bitmap", + }, + ], + ) + self.assertEqual( + inventory["excluded_derived_bytes"], + len(bitmap_bytes) + len(graph_bytes) + len(info_packs_bytes), + ) + + def test_pack_bitmap_requires_exact_complete_pack_pair(self): + for present_members in (set(), {"pack"}, {"idx"}): + with self.subTest(present_members=present_members), tempfile.TemporaryDirectory( + prefix="gitea-salvage-orphan-bitmap-" + ) as directory: + repo = Path(directory) / "repo.git" + pack_root = repo / "objects" / "pack" + ref_root = repo / "refs" / "heads" + pack_root.mkdir(parents=True) + ref_root.mkdir(parents=True) + digest = "a" * 40 + oid = "b" * 40 + for member in present_members: + (pack_root / f"pack-{digest}.{member}").write_bytes(member.encode()) + (pack_root / f"pack-{digest}.bitmap").write_bytes(b"bitmap") + (ref_root / "main").write_text(oid + "\n", encoding="ascii") + (repo / "HEAD").write_text( + "ref: refs/heads/main\n", + encoding="ascii", + ) + with ( + mock.patch.object( + RUNNER, + "gitea_salvage_file_has_nocow", + return_value=False, + ), + self.assertRaisesRegex( + RUNNER.DeployError, + "pack bitmap lacks a complete pack/index pair", + ), + ): + inventory_repository(repo, "repo.git") + + def test_unobserved_accelerators_and_malformed_temporary_dirs_remain_blocked(self): + rejected = ( + "pack/pack-" + "a" * 40 + ".rev", + "pack/pack-" + "a" * 40 + ".mtimes", + "pack/pack-" + "a" * 40 + ".keep", + "pack/multi-pack-index", + ) + for relative in rejected: + with self.subTest(relative=relative), tempfile.TemporaryDirectory( + prefix="gitea-salvage-unobserved-derived-" + ) as directory: + repo = Path(directory) / "repo.git" + rejected_path = repo / "objects" / relative + rejected_path.parent.mkdir(parents=True) + rejected_path.write_bytes(b"unreviewed") + with ( + mock.patch.object( + RUNNER, + "gitea_salvage_file_has_nocow", + return_value=False, + ), + self.assertRaisesRegex( + RUNNER.DeployError, + "unexpected object material", + ), + ): + inventory_repository(repo, "repo.git") + + with tempfile.TemporaryDirectory(prefix="gitea-salvage-temp-object-") as directory: + repo = Path(directory) / "repo.git" + (repo / "objects" / "tmp_objdir-incoming-incident").mkdir(parents=True) + with self.assertRaisesRegex( + RUNNER.DeployError, + r"unexpected object directory: repo\.git/objects/" + r"tmp_objdir-incoming-incident", + ): + inventory_repository(repo, "repo.git") + + def test_exact_receive_quarantine_is_stat_bound_and_not_traversed(self): + with tempfile.TemporaryDirectory(prefix="gitea-salvage-quarantine-") as directory: + repo = Path(directory) / "repo.git" + object_file = repo / "objects" / "aa" / ("b" * 38) + quarantine = repo / "objects" / "tmp_objdir-incoming-aB09zZ" + ref_file = repo / "refs" / "heads" / "main" + object_file.parent.mkdir(parents=True) + quarantine.mkdir(parents=True) + ref_file.parent.mkdir(parents=True) + object_file.write_bytes(b"object") + # Deliberately invalid main-ODB material proves that os.walk prunes the + # approved receive-pack quarantine without reading or classifying it. + (quarantine / "pack").mkdir() + (quarantine / "pack" / "orphan.pack").write_bytes(b"ignored quarantine") + oid = "a" * 40 + ref_file.write_text(oid + "\n", encoding="ascii") + (repo / "HEAD").write_text("ref: refs/heads/main\n", encoding="ascii") + real_lstat = Path.lstat + + def pinned_owner_lstat(path): + value = real_lstat(path) + if path != quarantine: + return value + result = mock.Mock() + result.st_mode = value.st_mode + result.st_uid = 1000 + result.st_gid = 1000 + result.st_nlink = 1 + result.st_size = value.st_size + result.st_dev = value.st_dev + result.st_ino = value.st_ino + return result + + with ( + mock.patch.object(Path, "lstat", pinned_owner_lstat), + mock.patch.object( + RUNNER, + "gitea_salvage_file_has_nocow", + return_value=False, + ), + ): + inventory = inventory_repository( + repo, + "repo.git", + ) + self.assertEqual(inventory["object_files"], 1) + self.assertEqual( + inventory["excluded_quarantine_directories"], + [ + { + "kind": "receive-pack-quarantine", + "lstat": { + "gid": 1000, + "mode": "0755", + "nlink": 1, + "size": real_lstat(quarantine).st_size, + "uid": 1000, + }, + "path": "objects/tmp_objdir-incoming-aB09zZ", + } + ], + ) + + def test_exact_receive_quarantine_rejects_structural_drift(self): + for unsafe_kind in ("mode", "nocow"): + with self.subTest(unsafe_kind=unsafe_kind), tempfile.TemporaryDirectory( + prefix="gitea-salvage-quarantine-unsafe-" + ) as directory: + repo = Path(directory) / "repo.git" + quarantine = repo / "objects" / "tmp_objdir-incoming-abc123" + quarantine.mkdir(parents=True) + if unsafe_kind == "mode": + quarantine.chmod(0o700) + real_lstat = Path.lstat + + def pinned_owner_lstat(path): + value = real_lstat(path) + if path != quarantine: + return value + result = mock.Mock() + result.st_mode = value.st_mode + result.st_uid = 1000 + result.st_gid = 1000 + result.st_nlink = 1 + result.st_size = value.st_size + result.st_dev = value.st_dev + result.st_ino = value.st_ino + return result + + with ( + mock.patch.object(Path, "lstat", pinned_owner_lstat), + mock.patch.object( + RUNNER, + "gitea_salvage_file_has_nocow", + return_value=unsafe_kind == "nocow", + ), + self.assertRaisesRegex( + RUNNER.DeployError, + "receive quarantine directory is unsafe", + ), + ): + inventory_repository(repo, "repo.git") + + def test_observed_derived_exclusions_reject_unsafe_metadata(self): + for cache_kind in ("bitmap", "commit-graph"): + for unsafe_kind in ( + "directory", + "symlink", + "hardlink", + "fifo", + "nocow", + "oversized", + ): + with ( + self.subTest(cache_kind=cache_kind, unsafe_kind=unsafe_kind), + tempfile.TemporaryDirectory( + prefix="gitea-salvage-derived-unsafe-" + ) as directory, + ): + base = Path(directory) + repo = base / "repo.git" + pack_root = repo / "objects" / "pack" + info_root = repo / "objects" / "info" + ref_root = repo / "refs" / "heads" + pack_root.mkdir(parents=True) + info_root.mkdir(parents=True) + ref_root.mkdir(parents=True) + digest = "a" * 40 + oid = "b" * 40 + (pack_root / f"pack-{digest}.pack").write_bytes(b"pack") + (pack_root / f"pack-{digest}.idx").write_bytes(b"idx") + cache = ( + pack_root / f"pack-{digest}.bitmap" + if cache_kind == "bitmap" + else info_root / "commit-graph" + ) + if unsafe_kind == "directory": + cache.mkdir() + elif unsafe_kind == "symlink": + cache.symlink_to("/dev/null") + elif unsafe_kind == "hardlink": + source = base / "hardlink-source" + source.write_bytes(b"derived") + os.link(source, cache) + elif unsafe_kind == "fifo": + os.mkfifo(cache) + elif unsafe_kind == "oversized": + limit = ( + RUNNER.GITEA_SALVAGE_DERIVED_PACK_BITMAP_MAX_BYTES + if cache_kind == "bitmap" + else RUNNER.GITEA_SALVAGE_DERIVED_COMMIT_GRAPH_MAX_BYTES + ) + with cache.open("wb") as handle: + handle.truncate(limit + 1) + else: + cache.write_bytes(b"derived") + (ref_root / "main").write_text(oid + "\n", encoding="ascii") + (repo / "HEAD").write_text( + "ref: refs/heads/main\n", + encoding="ascii", + ) + + def has_nocow(path): + return unsafe_kind == "nocow" and path == cache + + expected = ( + "unexpected object directory" + if unsafe_kind == "directory" + else "derived object cache is oversized" + if unsafe_kind == "oversized" + else "has an unsafe type" + if unsafe_kind in {"symlink", "fifo"} + else "object file is unsafe" + ) + with ( + mock.patch.object( + RUNNER, + "gitea_salvage_file_has_nocow", + side_effect=has_nocow, + ), + self.assertRaisesRegex(RUNNER.DeployError, expected), + ): + inventory_repository( + repo, + "repo.git", + ) + + def test_object_info_packs_rejects_unsafe_metadata_variants(self): + for unsafe_kind in ( + "directory", + "symlink", + "hardlink", + "fifo", + "nocow", + "oversized", + ): + with self.subTest(unsafe_kind=unsafe_kind), tempfile.TemporaryDirectory( + prefix="gitea-salvage-info-packs-unsafe-" + ) as directory: + base = Path(directory) + repo = base / "repo.git" + object_file = repo / "objects" / "aa" / ("b" * 38) + info_packs = repo / "objects" / "info" / "packs" + ref_file = repo / "refs" / "heads" / "main" + object_file.parent.mkdir(parents=True) + info_packs.parent.mkdir(parents=True) + ref_file.parent.mkdir(parents=True) + object_file.write_bytes(b"object") + if unsafe_kind == "directory": + info_packs.mkdir() + elif unsafe_kind == "symlink": + info_packs.symlink_to("/dev/null") + elif unsafe_kind == "hardlink": + hardlink_source = base / "hardlink-source" + hardlink_source.write_bytes(b"derived") + os.link(hardlink_source, info_packs) + elif unsafe_kind == "fifo": + os.mkfifo(info_packs) + elif unsafe_kind == "oversized": + with info_packs.open("wb") as handle: + handle.truncate( + RUNNER.GITEA_SALVAGE_DERIVED_INFO_PACKS_MAX_BYTES + 1 + ) + else: + info_packs.write_bytes(b"derived") + oid = "a" * 40 + ref_file.write_text(oid + "\n", encoding="ascii") + (repo / "HEAD").write_text( + "ref: refs/heads/main\n", + encoding="ascii", + ) + expected = ( + "unexpected object directory" + if unsafe_kind == "directory" + else "derived object cache is oversized" + if unsafe_kind == "oversized" + else "has an unsafe type" + if unsafe_kind in {"symlink", "fifo"} + else "object file is unsafe" + ) + + def has_nocow(path): + return unsafe_kind == "nocow" and path == info_packs + + with ( + mock.patch.object( + RUNNER, + "gitea_salvage_file_has_nocow", + side_effect=has_nocow, + ), + self.assertRaisesRegex(RUNNER.DeployError, expected), + ): + inventory_repository( + repo, + "repo.git", + ) + + def test_object_walk_error_fails_closed(self): + with tempfile.TemporaryDirectory(prefix="gitea-salvage-object-walk-") as directory: + repo = Path(directory) / "repo.git" + (repo / "objects").mkdir(parents=True) + + def failing_walk( + root, + topdown=True, + onerror=None, + followlinks=False, + ): + self.assertEqual(Path(root), repo / "objects") + self.assertTrue(topdown) + self.assertFalse(followlinks) + self.assertIsNotNone(onerror) + onerror(OSError(5, "simulated I/O error", str(root))) + return () + + with ( + mock.patch.object(RUNNER.os, "walk", side_effect=failing_walk), + self.assertRaisesRegex( + RUNNER.DeployError, + r"repository objects repo\.git traversal failed: errno=5", + ), + ): + inventory_repository(repo, "repo.git") + + def test_loose_ref_walk_error_fails_closed(self): + with tempfile.TemporaryDirectory(prefix="gitea-salvage-ref-walk-") as directory: + repo = Path(directory) / "repo.git" + object_file = repo / "objects" / "aa" / ("b" * 38) + loose_refs = repo / "refs" + object_file.parent.mkdir(parents=True) + loose_refs.mkdir(parents=True) + object_file.write_bytes(b"object") + (repo / "HEAD").write_text("ref: refs/heads/main\n", encoding="ascii") + real_walk = RUNNER.os.walk + + def selective_walk( + root, + topdown=True, + onerror=None, + followlinks=False, + ): + if Path(root) == loose_refs: + self.assertIsNotNone(onerror) + onerror(PermissionError(13, "simulated denial", str(root))) + return () + return real_walk( + root, + topdown=topdown, + onerror=onerror, + followlinks=followlinks, + ) + + with ( + mock.patch.object( + RUNNER, + "gitea_salvage_file_has_nocow", + return_value=False, + ), + mock.patch.object( + RUNNER.os, + "walk", + side_effect=selective_walk, + ), + self.assertRaisesRegex( + RUNNER.DeployError, + r"repository refs repo\.git traversal failed: errno=13", + ), + ): + inventory_repository(repo, "repo.git") + + def test_object_boundary_rejects_symlink_special_and_nocow(self): + for unsafe_kind in ("symlink", "fifo", "nocow"): + with self.subTest(unsafe_kind=unsafe_kind), tempfile.TemporaryDirectory( + prefix="gitea-salvage-object-unsafe-" + ) as directory: + repo = Path(directory) / "repo.git" + object_root = repo / "objects" / "aa" + ref_root = repo / "refs" / "heads" + object_root.mkdir(parents=True) + ref_root.mkdir(parents=True) + object_file = object_root / ("b" * 38) + if unsafe_kind == "symlink": + object_file.symlink_to("/dev/null") + elif unsafe_kind == "fifo": + os.mkfifo(object_file) + else: + object_file.write_bytes(b"object") + oid = "a" * 40 + (ref_root / "main").write_text(oid + "\n", encoding="ascii") + (repo / "HEAD").write_text("ref: refs/heads/main\n", encoding="ascii") + with ( + mock.patch.object( + RUNNER, + "gitea_salvage_file_has_nocow", + return_value=unsafe_kind == "nocow", + ), + self.assertRaisesRegex( + RUNNER.DeployError, + "has an unsafe type" + if unsafe_kind in {"symlink", "fifo"} + else "object file is unsafe", + ), + ): + inventory_repository( + repo, + "repo.git", + ) + + def test_unsupported_state_report_is_per_repo_deterministic_and_size_bound(self): + connection = unsupported_state_test_connection() + try: + connection.executemany( + "INSERT INTO repository (id,description,website,original_url,topics," + "avatar,num_watches,num_stars,num_issues,num_pulls,num_milestones," + "num_projects,num_action_runs,lfs_size) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?)", + ( + (7, "kept", "", "", "", "", 1, 0, 1, 0, 0, 0, 0, 30), + (9, "", "", "", "", "", 0, 0, 0, 0, 0, 0, 0, 10), + (100, "ignored", "", "", "", "", 99, 0, 0, 0, 0, 0, 0, 99), + ), + ) + connection.executemany( + "INSERT INTO issue (id,repo_id) VALUES (?,?)", + ((1, 7), (2, 9), (3, 100)), + ) + connection.executemany( + "INSERT INTO repo_unit (repo_id,type,config) VALUES (?,?,?)", + ((7, 1, "DO-NOT-EXPORT-CONFIG"), (7, 2, "{}"), (100, 1, "{}")), + ) + connection.executemany( + "INSERT INTO lfs_meta_object (id,oid,size,repository_id) VALUES (?,?,?,?)", + ( + (1, "a" * 64, 10, 7), + (2, "a" * 64, 10, 9), + (3, "b" * 64, 20, 9), + (4, "a" * 64, 10, 100), + ), + ) + connection.executemany( + "INSERT INTO attachment " + "(id,repo_id,issue_id,release_id,comment_id,size,content_sentinel) " + "VALUES (?,?,?,?,?,?,?)", + ( + (1, 7, 11, 0, 21, 5, "DO-NOT-EXPORT-CONTENT"), + (2, 9, 0, 12, 0, 7, "safe-boundary"), + (3, 100, 0, 0, 0, 99, "ignored"), + ), + ) + kept = ( + {"repo_id": 9, "owner": "SILVER", "slug": "nine"}, + {"repo_id": 7, "owner": "dctouch", "slug": "seven"}, + ) + + def authorizer(action, _table, column, _database, _trigger): + if action == sqlite3.SQLITE_READ and column in { + "config", + "content_sentinel", + "secret_sentinel", + }: + return sqlite3.SQLITE_DENY + return sqlite3.SQLITE_OK + + connection.set_authorizer(authorizer) + first = RUNNER.gitea_salvage_unsupported_state_inventory(connection, kept) + second = RUNNER.gitea_salvage_unsupported_state_inventory( + connection, + tuple(reversed(kept)), + ) + finally: + connection.close() + + self.assertEqual(first["json"], second["json"]) + self.assertEqual(first["sha256"], second["sha256"]) + self.assertEqual(json.loads(first["json"]), first["report"]) + report = first["report"] + self.assertEqual(report["kept_repository_ids"], [7, 9]) + self.assertEqual( + [row["old_repo_id"] for row in report["per_repository"]], + [7, 9], + ) + by_id = {row["old_repo_id"]: row for row in report["per_repository"]} + self.assertEqual(by_id[7]["counts"]["issues"], 1) + self.assertEqual(by_id[9]["counts"]["issues"], 1) + self.assertEqual(by_id[7]["repo_unit_types"], {"1": 1, "2": 1}) + self.assertEqual(by_id[7]["lfs"]["association_logical_bytes"], 10) + self.assertEqual(by_id[9]["lfs"]["association_logical_bytes"], 30) + self.assertEqual(report["aggregates"]["lfs"]["association_rows"], 3) + self.assertEqual( + report["aggregates"]["lfs"]["association_logical_bytes"], + 40, + ) + self.assertEqual(report["aggregates"]["lfs"]["distinct_oids"], 2) + self.assertEqual(report["aggregates"]["lfs"]["unique_logical_bytes"], 30) + self.assertEqual(report["aggregates"]["lfs"]["non_kept_shared_oids"], 1) + self.assertEqual(report["aggregates"]["attachments"]["association_rows"], 2) + self.assertEqual(report["aggregates"]["attachments"]["logical_bytes"], 12) + self.assertEqual( + report["aggregates"]["attachments"]["link_splits"]["issue"], + {"logical_bytes": 5, "rows": 1}, + ) + self.assertEqual( + report["aggregates"]["attachments"]["link_splits"]["multi_link"], + {"logical_bytes": 5, "rows": 1}, + ) + self.assertTrue(report["material_present"]) + self.assertNotIn("material_total", first["json"]) + self.assertNotIn("DO-NOT-EXPORT", first["json"]) + self.assertIn("issue-pull-dependent-closure-unreviewed", report["coverage_blockers"]) + self.assertFalse(report["schema_review"]["matches"]) + self.assertIn( + {"label": "issues", "repository_column": "repo_id", "table": "issue"}, + report["direct_relation_contract"], + ) + + def test_unsupported_state_report_keeps_zeroes_and_fails_schema_closed(self): + connection = unsupported_state_test_connection() + try: + connection.execute( + "INSERT INTO repository (id,description,website,original_url,topics," + "avatar,num_watches,num_stars,num_issues,num_pulls,num_milestones," + "num_projects,num_action_runs,lfs_size) VALUES " + "(7,'','','','','',0,0,0,0,0,0,0,0)" + ) + connection.execute("DROP TABLE issue") + connection.execute("CREATE TABLE issue (id INTEGER PRIMARY KEY, repo_id TEXT)") + report = RUNNER.gitea_salvage_unsupported_state_inventory( + connection, + ({"repo_id": 7, "owner": "dctouch", "slug": "seven"},), + )["report"] + finally: + connection.close() + record = report["per_repository"][0] + self.assertIsNone(record["counts"]["issues"]) + self.assertEqual(record["counts"]["releases"], 0) + self.assertEqual(record["lfs"]["association_rows"], 0) + self.assertIn("issue.repo_id:integer-affinity", report["schema_mismatch"]) + self.assertFalse(report["material_present"]) + + def test_unsupported_state_report_never_queries_view_as_table(self): + connection = unsupported_state_test_connection() + try: + connection.execute( + "INSERT INTO repository (id,description,website,original_url,topics," + "avatar,num_watches,num_stars,num_issues,num_pulls,num_milestones," + "num_projects,num_action_runs,lfs_size) VALUES " + "(7,'','','','','',0,0,0,0,0,0,0,0)" + ) + connection.execute("DROP TABLE issue") + connection.execute( + "CREATE TABLE issue_backing (id INTEGER PRIMARY KEY, repo_id INTEGER, " + "content_sentinel TEXT)" + ) + connection.execute( + "INSERT INTO issue_backing VALUES (1,7,'DO-NOT-QUERY-VIEW-CONTENT')" + ) + connection.execute("CREATE VIEW issue AS SELECT * FROM issue_backing") + + def authorizer(action, table, column, _database, _trigger): + if action == sqlite3.SQLITE_READ and table == "issue_backing": + return sqlite3.SQLITE_DENY + return sqlite3.SQLITE_OK + + connection.set_authorizer(authorizer) + evidence = RUNNER.gitea_salvage_unsupported_state_inventory( + connection, + ({"repo_id": 7, "owner": "dctouch", "slug": "seven"},), + ) + finally: + connection.close() + report = evidence["report"] + self.assertIsNone(report["per_repository"][0]["counts"]["issues"]) + self.assertIn("issue:ordinary-main-table", report["schema_mismatch"]) + issue_schema = next( + item + for item in report["schema_catalog"]["tables"] + if item["table"] == "issue" + ) + self.assertFalse(issue_schema["ordinary_main_table"]) + self.assertEqual(issue_schema["sqlite_schema_kind"], "view") + self.assertNotIn("DO-NOT-QUERY", evidence["json"]) + + def test_schema_object_gate_has_safe_pre_table_list_fallback(self): + connection = unsupported_state_test_connection() + try: + table = RUNNER.gitea_salvage_table_schema( + connection, + "repository", + table_list_supported=False, + ) + connection.execute("DROP TABLE issue") + connection.execute("CREATE VIEW issue AS SELECT 1 AS id,7 AS repo_id") + view = RUNNER.gitea_salvage_table_schema( + connection, + "issue", + table_list_supported=False, + ) + finally: + connection.close() + self.assertTrue(table["ordinary_main_table"]) + self.assertIsNone(table["table_list"]) + self.assertEqual( + table["object_kind_attestation"], + "sqlite-master-nonvirtual-table", + ) + self.assertFalse(view["ordinary_main_table"]) + + def test_snapshot_sqlite_connection_reads_back_safety_pragmas(self): + with tempfile.TemporaryDirectory(prefix="gitea-salvage-sqlite-") as directory: + database = Path(directory) / "snapshot.db" + source = sqlite3.connect(database) + try: + source.execute("CREATE TABLE evidence (id INTEGER PRIMARY KEY)") + source.commit() + finally: + source.close() + connection = RUNNER.gitea_salvage_sqlite_connection(database) + try: + self.assertEqual(connection.execute("PRAGMA query_only").fetchone()[0], 1) + self.assertEqual( + connection.execute("PRAGMA trusted_schema").fetchone()[0], + 0, + ) + with self.assertRaises(sqlite3.OperationalError): + connection.execute("INSERT INTO evidence VALUES (1)") + finally: + connection.close() + + def test_unsupported_state_report_rejects_invalid_lfs_sizes_and_oids(self): + connection = unsupported_state_test_connection() + try: + connection.execute( + "INSERT INTO repository (id,description,website,original_url,topics," + "avatar,num_watches,num_stars,num_issues,num_pulls,num_milestones," + "num_projects,num_action_runs,lfs_size) VALUES " + "(7,'','','','','',0,0,0,0,0,0,0,0)" + ) + connection.executemany( + "INSERT INTO lfs_meta_object (id,oid,size,repository_id) VALUES (?,?,?,7)", + ((1, "A" * 64, 1), (2, "b" * 64, -1)), + ) + report = RUNNER.gitea_salvage_unsupported_state_inventory( + connection, + ({"repo_id": 7, "owner": "dctouch", "slug": "seven"},), + )["report"] + finally: + connection.close() + self.assertEqual(report["aggregates"]["lfs"]["invalid_oid_rows"], 1) + self.assertEqual(report["aggregates"]["lfs"]["invalid_size_rows"], 1) + self.assertIsNone(report["aggregates"]["lfs"]["association_logical_bytes"]) + self.assertTrue(any("lfs_meta_object.oid" in item for item in report["anomalies"])) + self.assertTrue(any("lfs_meta_object.size" in item for item in report["anomalies"])) + + def test_unsupported_state_report_rejects_invalid_related_lfs_size(self): + connection = unsupported_state_test_connection() + try: + connection.execute( + "INSERT INTO repository (id,description,website,original_url,topics," + "avatar,num_watches,num_stars,num_issues,num_pulls,num_milestones," + "num_projects,num_action_runs,lfs_size) VALUES " + "(7,'','','','','',0,0,0,0,0,0,0,0)" + ) + connection.execute( + "INSERT INTO repository (id,description,website,original_url,topics," + "avatar,num_watches,num_stars,num_issues,num_pulls,num_milestones," + "num_projects,num_action_runs,lfs_size) VALUES " + "(100,'','','','','',0,0,0,0,0,0,0,0)" + ) + connection.executemany( + "INSERT INTO lfs_meta_object (id,oid,size,repository_id) VALUES (?,?,?,?)", + ((1, "a" * 64, 10, 7), (2, "a" * 64, "invalid", 100)), + ) + report = RUNNER.gitea_salvage_unsupported_state_inventory( + connection, + ({"repo_id": 7, "owner": "dctouch", "slug": "seven"},), + )["report"] + finally: + connection.close() + lfs = report["aggregates"]["lfs"] + self.assertEqual(lfs["invalid_size_rows"], 0) + self.assertEqual(lfs["invalid_related_size_rows"], 1) + self.assertIsNone(lfs["unique_logical_bytes"]) + self.assertTrue(any("invalid_related_rows=1" in item for item in report["anomalies"])) + + def test_unsupported_state_report_rejects_invalid_and_orphan_lfs_owners(self): + connection = unsupported_state_test_connection() + try: + connection.execute( + "INSERT INTO repository (id,description,website,original_url,topics," + "avatar,num_watches,num_stars,num_issues,num_pulls,num_milestones," + "num_projects,num_action_runs,lfs_size) VALUES " + "(7,'','','','','',0,0,0,0,0,0,0,0)" + ) + connection.executemany( + "INSERT INTO lfs_meta_object (id,oid,size,repository_id) VALUES (?,?,?,?)", + ( + (1, "a" * 64, 10, 7), + (2, "a" * 64, 10, None), + (3, "a" * 64, 10, 999), + ), + ) + report = RUNNER.gitea_salvage_unsupported_state_inventory( + connection, + ({"repo_id": 7, "owner": "dctouch", "slug": "seven"},), + )["report"] + finally: + connection.close() + lfs = report["aggregates"]["lfs"] + self.assertEqual(lfs["invalid_related_repository_rows"], 1) + self.assertEqual(lfs["orphan_related_repository_rows"], 1) + self.assertIsNone(lfs["non_kept_shared_oids"]) + self.assertTrue(any("orphan_related_rows=1" in item for item in report["anomalies"])) + + def test_plan_source_exports_hash_bound_reviewable_evidence(self): + source = RUNNER_PATH.read_text(encoding="utf-8") + for required in ( + "gitea_reference_manifest_bytes=", + "gitea_reference_manifest_json=", + "gitea_legacy_container_name=", + "gitea_legacy_container_state=", + "gitea_legacy_image_ref=", + "gitea_legacy_image_id=", + "gitea_legacy_mount=", + "gitea_unsupported_repository_report_sha256=", + "gitea_unsupported_repository_report_bytes=", + "gitea_unsupported_schema_catalog_sha256=", + "gitea_unsupported_repository_report_json=", + "gitea_incident_disposition_sha256=", + "gitea_incident_disposition_refs=", + "gitea_incident_disposition_remaining_blockers=", + "gitea_incident_closure_disposition_sha256=", + "gitea_incident_closure_report_sha256=", + "gitea_incident_closure_report_bytes=", + "gitea_incident_closure_report_json=", + "reference-manifest-fsck-reachability-verifier-pending", + "unsupported-schema-catalog-verifier-pending", + "closure-report-review-pin-pending", + ): + self.assertIn(required, source) + + def test_salvage_runtime_cannot_activate_while_review_is_open(self): + with self.assertRaisesRegex(RUNNER.DeployError, "activation is frozen"): + RUNNER.prepare_component_runtime( + "gitea", + RUNNER.GITEA_SALVAGE_ENTRIES, + ) + + +if __name__ == "__main__": + unittest.main()