179 lines
8.0 KiB
Markdown
179 lines
8.0 KiB
Markdown
# AI Workspace Test Matrix
|
|
|
|
This matrix defines what blocks a runtime deploy while the assistant control plane moves from prototype to a scalable multi-app system.
|
|
|
|
## Gate -1: Topology Classification
|
|
|
|
Run this before interpreting any local AI Workspace result.
|
|
|
|
Command:
|
|
|
|
- `infra/scripts/check-ai-workspace-topology.sh`
|
|
|
|
Current Mac-local shape observed on 2026-06-13:
|
|
|
|
- `task.local.nodedc` resolves to `127.0.0.1` and returns local Plane/Tasker HTML through the local reverse proxy.
|
|
- Local containers are up for Tasker runtime, Ops Agent Gateway `:4100`, AI Workspace Hub `:18081`, and AI Workspace Assistant `:18082`.
|
|
- Local AI Workspace Hub reports `agentsOnline: 0`.
|
|
- Local AI Workspace Assistant still points at public `ai-hub.nodedc.ru` for Hub public/internal URLs.
|
|
- Local AI Workspace Assistant is not configured with a local Ops entitlement adapter URL.
|
|
- Engine dev server `127.0.0.1:5300` is not part of the currently running local process set unless started separately.
|
|
- Local Ops Agent Gateway points Tasker downstream to `task.local.nodedc`, but its public MCP URL can be a LAN address, not a universally reachable URL.
|
|
|
|
Progress:
|
|
|
|
- The test explicitly labels itself as one of: `contract-only`, `local-ops-vertical`, `local-ui-only`, `hybrid-prod-bridge`, or `true-local-e2e`.
|
|
- `contract-only` and `local-ops-vertical` are allowed before deploy and do not require a live Codex worker.
|
|
- `true-local-e2e` is claimed only when Engine, Platform Assistant, Hub, Ops Gateway, Tasker, and the Codex worker all use the same intended local/tunnel topology.
|
|
|
|
Regression:
|
|
|
|
- A hybrid test is treated as proof that the local end-to-end path works.
|
|
- A UI conversation is counted as a write-path pass while the active Codex worker is connected to public `ai-hub.nodedc.ru` and local Hub has zero online agents.
|
|
- A local run writes through a Gateway/Tasker target that is not the expected local Tasker runtime.
|
|
- A returned MCP URL is not reachable from the worker that will execute the run.
|
|
|
|
Ignore:
|
|
|
|
- `task.local.nodedc` returning HTML proves only local Tasker UI/proxy availability.
|
|
- Local Hub availability on `127.0.0.1:18081` proves only relay process health, not worker connectivity.
|
|
- Engine UI on `127.0.0.1:5300` proves only frontend/dev server availability unless its backend AI Workspace URLs point to the same local/tunnel control plane.
|
|
|
|
True local e2e prerequisites:
|
|
|
|
- Start Engine dev/runtime and point it to local Platform Assistant.
|
|
- Configure local Platform Assistant with local Ops entitlement adapter: `AI_WORKSPACE_OPS_ENTITLEMENT_URL=http://host.docker.internal:4100/api/internal/v1/ai-workspace/entitlements` plus the matching internal token.
|
|
- Make Platform Assistant and the executing Codex worker use the same Hub endpoint.
|
|
- Make `NODEDC_AGENT_GATEWAY_PUBLIC_URL` reachable from that worker, not only from the Mac host.
|
|
- Keep Ops Gateway `NODEDC_TASKER_INTERNAL_URL` pointed at the intended local Tasker runtime.
|
|
- If the worker is not on the Mac/local network, expose local Hub and Gateway through an explicit tunnel and mark the test as `tunnel-local-e2e`, not pure local.
|
|
|
|
## Gate 0: Static Hygiene
|
|
|
|
Commands:
|
|
|
|
- All current predeploy gates: `infra/scripts/check-ai-workspace-release-gates.sh`
|
|
- Config contract: `infra/scripts/check-ai-workspace-config-contract.sh`
|
|
- Controlled Synology apply plan: `infra/synology/check-ai-workspace-apply-plan.sh`
|
|
- Platform: `node --check src/server.mjs && npm run smoke:run-profile`
|
|
- Ops Gateway: `npm run check && npm run build`
|
|
- Global: `git diff --check`
|
|
|
|
Progress:
|
|
|
|
- Code parses cleanly.
|
|
- TypeScript and build checks pass.
|
|
- No whitespace damage in tracked diffs.
|
|
|
|
Regression:
|
|
|
|
- Any syntax, build, type, or diff hygiene failure.
|
|
|
|
Ignore:
|
|
|
|
- NAS runtime not changing after source sync only. It is expected until container env and recreate are applied.
|
|
|
|
## Gate 1: Entitlement Adapter Boundary
|
|
|
|
Commands:
|
|
|
|
- Platform: `npm run smoke:run-profile`
|
|
|
|
Progress:
|
|
|
|
- Adapter response with `appGrants.ops.mcpServers` becomes `runProfile.toolProfile.mcpServers`.
|
|
- Runtime profile keeps real MCP headers for the worker.
|
|
- Public run profile and diagnostics redact MCP headers and tokens.
|
|
- MCP server names are normalized and stable.
|
|
|
|
Regression:
|
|
|
|
- Dynamic MCP server disappears from the run profile.
|
|
- Runtime headers are dropped before worker dispatch.
|
|
- Public metadata contains a bearer token or raw authorization header.
|
|
- Adapter grants fail to override same-app static grants for the current run.
|
|
|
|
Ignore:
|
|
|
|
- This gate does not prove a live Codex worker can reach Ops. It only proves the Platform contract shape before dispatch.
|
|
|
|
## Gate 2: Ops Gateway Vertical Smoke
|
|
|
|
Command:
|
|
|
|
- `DATABASE_URL=... NODEDC_AGENT_GATEWAY_INTERNAL_TOKEN=... NODEDC_AI_WORKSPACE_RUN_TOKEN_TTL_SECONDS=43200 npm run smoke:gateway`
|
|
|
|
Progress:
|
|
|
|
- Lifecycle internal auth is enforced.
|
|
- Owner lifecycle API creates and lists agent state.
|
|
- Preflight returns readiness without issuing a token.
|
|
- Entitlement call issues a short-lived run token only when target workspace/project context is present.
|
|
- Run token session reports `grant_source=token` and `grant_scope=token`.
|
|
- Run token is narrowed to the requested project and scope intersection.
|
|
- MCP `tools/list` exposes only tools allowed by the token grant.
|
|
- Cross-project or out-of-scope writes return 403.
|
|
- Write tools still require idempotency.
|
|
|
|
Regression:
|
|
|
|
- No-context entitlement issues a broad token.
|
|
- Token-scoped entitlement inherits unrelated project grants.
|
|
- Read-only run token can see write tools.
|
|
- Cross-project write reaches Tasker.
|
|
- Missing internal token is accepted.
|
|
- Existing long-lived agent token flow breaks unexpectedly.
|
|
|
|
Ignore:
|
|
|
|
- A Tasker boundary 403/503 in the final write leg can be acceptable in local smoke if the Gateway proved the request reached the downstream boundary with the expected auth context.
|
|
- Legacy long-lived manual tokens may still expose all agent grants by design; AI Workspace run tokens must not.
|
|
|
|
## Gate 3: Worker Config Boundary
|
|
|
|
Progress:
|
|
|
|
- Worker creates per-run isolated `CODEX_HOME/config.toml` from `runProfile.toolProfile.mcpServers`.
|
|
- Dynamic run profile takes priority over install-time MCP config.
|
|
- Legacy install-time Ops MCP config is used only when no dynamic profile exists.
|
|
- Worker health/event output reports dynamic MCP names but not token values.
|
|
|
|
Regression:
|
|
|
|
- Worker merges stale install-time MCP credentials into a dynamic run.
|
|
- Worker prints MCP tokens in events, logs, health, or errors.
|
|
- A new app requires rebuilding the worker instead of adding an entitlement adapter/grant response.
|
|
|
|
Ignore:
|
|
|
|
- Existing local agent installation files are not the source of truth for new AI Workspace runs once dynamic profile is present.
|
|
|
|
## Gate 4: Controlled Runtime Before Deploy
|
|
|
|
Scenarios:
|
|
|
|
- Engine user with Ops read grant opens an Ops-targeted run and can read the target card.
|
|
- Engine user with Ops write grant creates or updates in the target Ops project.
|
|
- Engine user without target Ops grant gets preflight denied and no token.
|
|
- Ops-origin run can call Engine-side context once `AI_WORKSPACE_ENGINE_ENTITLEMENT_URL` points at Engine `/api/ai-workspace/internal/v1/entitlements`.
|
|
- Admin grants a user a new project permission; new AI Workspace run receives it without reinstalling the agent.
|
|
- Admin revokes a project permission; new run token loses it and existing source-linked token grants stop resolving through deleted grants.
|
|
|
|
Progress:
|
|
|
|
- New apps are onboarded by adding entitlement adapter config and app grant response shape.
|
|
- User access changes flow through Authentik/app ACL to entitlement preflight and run tokens.
|
|
- Agent installation remains a bridge/bootstrap layer, not a per-user ACL store.
|
|
|
|
Regression:
|
|
|
|
- Any app requires hard-coded MCP config in the installer.
|
|
- User ACL changes require reinstalling a worker.
|
|
- Run tokens remain valid with revoked source grants.
|
|
- One project grant leaks to another project or app.
|
|
|
|
Ignore:
|
|
|
|
- SSH access from the current workstation to Synology can time out; it is not evidence of app regression.
|
|
- Source files synced to NAS are not active runtime until env and container recreate are explicitly applied.
|