NODEDC_PLATFORM/services/ai-workspace-assistant/TEST_MATRIX.md

AI Workspace Test Matrix

This matrix defines what blocks a runtime deploy while the assistant control plane moves from prototype to a scalable multi-app system.

Gate -1: Topology Classification

Run this before interpreting any local AI Workspace result.

Command:

• infra/scripts/check-ai-workspace-topology.sh

Current Mac-local shape observed on 2026-06-13:

• task.local.nodedc resolves to 127.0.0.1 and returns local Plane/Tasker HTML through the local reverse proxy.
• Local containers are up for Tasker runtime, Ops Agent Gateway :4100, AI Workspace Hub :18081, and AI Workspace Assistant :18082.
• Local AI Workspace Hub reports agentsOnline: 0.
• Local AI Workspace Assistant still points at public ai-hub.nodedc.ru for Hub public/internal URLs.
• Local AI Workspace Assistant is not configured with a local Ops entitlement adapter URL.
• Engine dev server 127.0.0.1:5300 is not part of the currently running local process set unless started separately.
• Local Ops Agent Gateway points Tasker downstream to task.local.nodedc, but its public MCP URL can be a LAN address, not a universally reachable URL.

Progress:

• The test explicitly labels itself as one of: contract-only, local-ops-vertical, local-ui-only, hybrid-prod-bridge, or true-local-e2e.
• contract-only and local-ops-vertical are allowed before deploy and do not require a live Codex worker.
• true-local-e2e is claimed only when Engine, Platform Assistant, Hub, Ops Gateway, Tasker, and the Codex worker all use the same intended local/tunnel topology.

Regression:

• A hybrid test is treated as proof that the local end-to-end path works.
• A UI conversation is counted as a write-path pass while the active Codex worker is connected to public ai-hub.nodedc.ru and local Hub has zero online agents.
• A local run writes through a Gateway/Tasker target that is not the expected local Tasker runtime.
• A returned MCP URL is not reachable from the worker that will execute the run.

Ignore:

• task.local.nodedc returning HTML proves only local Tasker UI/proxy availability.
• Local Hub availability on 127.0.0.1:18081 proves only relay process health, not worker connectivity.
• Engine UI on 127.0.0.1:5300 proves only frontend/dev server availability unless its backend AI Workspace URLs point to the same local/tunnel control plane.

True local e2e prerequisites:

• Start Engine dev/runtime and point it to local Platform Assistant.
• Configure local Platform Assistant with local Ops entitlement adapter: AI_WORKSPACE_OPS_ENTITLEMENT_URL=http://host.docker.internal:4100/api/internal/v1/ai-workspace/entitlements plus the matching internal token.
• Make Platform Assistant and the executing Codex worker use the same Hub endpoint.
• Make NODEDC_AGENT_GATEWAY_PUBLIC_URL reachable from that worker, not only from the Mac host.
• Keep Ops Gateway NODEDC_TASKER_INTERNAL_URL pointed at the intended local Tasker runtime.
• If the worker is not on the Mac/local network, expose local Hub and Gateway through an explicit tunnel and mark the test as tunnel-local-e2e, not pure local.

Gate 0: Static Hygiene

Commands:

• All current predeploy gates: infra/scripts/check-ai-workspace-release-gates.sh
• Config contract: infra/scripts/check-ai-workspace-config-contract.sh
• Controlled Synology apply plan: infra/synology/check-ai-workspace-apply-plan.sh
• Platform: node --check src/server.mjs && npm run smoke:run-profile
• Ops Gateway: npm run check && npm run build
• Global: git diff --check

Progress:

• Code parses cleanly.
• TypeScript and build checks pass.
• No whitespace damage in tracked diffs.

Regression:

• Any syntax, build, type, or diff hygiene failure.

Ignore:

• NAS runtime not changing after source sync only. It is expected until container env and recreate are applied.

Gate 1: Entitlement Adapter Boundary

Commands:

• Platform: npm run smoke:run-profile

Progress:

• Adapter response with appGrants.ops.mcpServers becomes runProfile.toolProfile.mcpServers.
• Runtime profile keeps real MCP headers for the worker.
• Public run profile and diagnostics redact MCP headers and tokens.
• MCP server names are normalized and stable.

Regression:

• Dynamic MCP server disappears from the run profile.
• Runtime headers are dropped before worker dispatch.
• Public metadata contains a bearer token or raw authorization header.
• Adapter grants fail to override same-app static grants for the current run.

Ignore:

• This gate does not prove a live Codex worker can reach Ops. It only proves the Platform contract shape before dispatch.

Gate 2: Ops Gateway Vertical Smoke

Command:

• DATABASE_URL=... NODEDC_AGENT_GATEWAY_INTERNAL_TOKEN=... NODEDC_AI_WORKSPACE_RUN_TOKEN_TTL_SECONDS=43200 npm run smoke:gateway

Progress:

• Lifecycle internal auth is enforced.
• Owner lifecycle API creates and lists agent state.
• Preflight returns readiness without issuing a token.
• Entitlement call issues a short-lived run token only when target workspace/project context is present.
• Run token session reports grant_source=token and grant_scope=token.
• Run token is narrowed to the requested project and scope intersection.
• MCP tools/list exposes only tools allowed by the token grant.
• Cross-project or out-of-scope writes return 403.
• Write tools still require idempotency.

Regression:

• No-context entitlement issues a broad token.
• Token-scoped entitlement inherits unrelated project grants.
• Read-only run token can see write tools.
• Cross-project write reaches Tasker.
• Missing internal token is accepted.
• Existing long-lived agent token flow breaks unexpectedly.

Ignore:

• A Tasker boundary 403/503 in the final write leg can be acceptable in local smoke if the Gateway proved the request reached the downstream boundary with the expected auth context.
• Legacy long-lived manual tokens may still expose all agent grants by design; AI Workspace run tokens must not.

Gate 3: Worker Config Boundary

Progress:

• Worker creates per-run isolated CODEX_HOME/config.toml from runProfile.toolProfile.mcpServers.
• Dynamic run profile takes priority over install-time MCP config.
• Legacy install-time Ops MCP config is used only when no dynamic profile exists.
• Worker health/event output reports dynamic MCP names but not token values.

Regression:

• Worker merges stale install-time MCP credentials into a dynamic run.
• Worker prints MCP tokens in events, logs, health, or errors.
• A new app requires rebuilding the worker instead of adding an entitlement adapter/grant response.

Ignore:

• Existing local agent installation files are not the source of truth for new AI Workspace runs once dynamic profile is present.

Gate 4: Controlled Runtime Before Deploy

Scenarios:

• Engine user with Ops read grant opens an Ops-targeted run and can read the target card.
• Engine user with Ops write grant creates or updates in the target Ops project.
• Engine user without target Ops grant gets preflight denied and no token.
• Ops-origin run can call Engine-side context once the Engine entitlement adapter exists.
• Admin grants a user a new project permission; new AI Workspace run receives it without reinstalling the agent.
• Admin revokes a project permission; new run token loses it and existing source-linked token grants stop resolving through deleted grants.

Progress:

• New apps are onboarded by adding entitlement adapter config and app grant response shape.
• User access changes flow through Authentik/app ACL to entitlement preflight and run tokens.
• Agent installation remains a bridge/bootstrap layer, not a per-user ACL store.

Regression:

• Any app requires hard-coded MCP config in the installer.
• User ACL changes require reinstalling a worker.
• Run tokens remain valid with revoked source grants.
• One project grant leaks to another project or app.

Ignore:

• SSH access from the current workstation to Synology can time out; it is not evidence of app regression.
• Source files synced to NAS are not active runtime until env and container recreate are explicitly applied.