Автоматизировать агентную проверку GUI-прогонов и stage-loop

2026-05-09 11:44:02 +03:00
parent 7c77db2c8d
commit 931251d1eb
11 changed files with 2428 additions and 9 deletions
@@ -53,6 +53,78 @@ Pack artifacts live under:
 - `final_status.md`
 - `scenarios/<scenario_id>/...`

+## AGENT autorun save gate
+
+`scripts/save_agent_semantic_run.py` is a post-validation persistence tool, not a replay executor.
+The normal path is:
+
+1. build/update the truth-harness spec;
+2. run `python scripts/domain_truth_harness.py run-live --spec ... --output-dir artifacts/domain_runs/<run_id>`;
+3. inspect `truth_review.md`, `business_review.md`, `pack_state.json`, and `final_status.md`;
+4. save to GUI autoruns only with `python scripts/save_agent_semantic_run.py --spec ... --validated-run-dir artifacts/domain_runs/<run_id>`.
+
+The save gate requires:
+- `pack_state.final_status = accepted`;
+- `pack_state.acceptance_gate_passed = true`;
+- `truth_review.summary.overall_status = pass`;
+- `business_review.overall_business_status = pass`;
+- zero unresolved P0 and zero business-answer failures.
+
+If a pack must be saved as a deliberate manual draft before live acceptance, use
+`--allow-unvalidated --unvalidated-reason "<why this is intentionally not accepted>"`.
+That path is explicitly marked as unvalidated and must not be treated as semantic proof.
+
+## Stage-level AGENT loop
+
+`scripts/stage_agent_loop.py` wraps the domain pack loop into the development-stage workflow:
+
+1. take the current global/local stage manifest;
+2. run `scripts/domain_case_loop.py run-pack-loop` for that stage pack;
+3. let the loop iterate through pack replay, business-first analyst verdict, coder patch, and rerun until the objective gate is accepted, blocked, or a real user decision is required;
+4. if accepted, persist the validated AGENT pack into GUI autoruns through `scripts/save_agent_semantic_run.py --validated-run-dir`;
+5. write `stage_loop_summary.json` and `stage_loop_handoff.md` for the final human visual confirmation.
+
+The stage manifest schema is `docs/orchestration/schemas/stage_agent_loop_manifest.schema.json`.
+The default stage gate is intentionally stricter than a narrow case gate: `target_score = 88`, no unresolved P0/P1 repair targets, accepted analyst verdict, clean business usefulness, direct-answer, temporal-honesty, field-truth, and answer-layering flags.
+
+Canonical commands:
+
+```powershell
+python scripts/stage_agent_loop.py plan --manifest docs/orchestration/<stage_loop>.json
+python scripts/stage_agent_loop.py run --manifest docs/orchestration/<stage_loop>.json
+python scripts/stage_agent_loop.py summarize --manifest docs/orchestration/<stage_loop>.json
+```
+
+This is the intended path for “implement the stage, generate/check stage questions, analyze business answers, patch code, rerun, then ask the user for final visual confirmation”.
+
+## GUI run review bridge
+
+When a manual or GUI autorun already exists, `scripts/review_assistant_stage1_run.py` turns the run id into the same machine-readable review surface.
+
+Canonical command:
+
+```powershell
+python scripts/review_assistant_stage1_run.py assistant-stage1-<id> --print-summary
+```
+
+The script resolves:
+- `llm_normalizer/reports/assistant-stage1-<id>.md`;
+- `llm_normalizer/data/assistant_sessions/assistant-stage1-<id>-*.json`.
+
+It writes:
+- `artifacts/domain_runs/gui_run_reviews/assistant-stage1-<id>/run_review.json`;
+- `artifacts/domain_runs/gui_run_reviews/assistant-stage1-<id>/run_review.md`;
+- `conversation_pairs.json`;
+- `question_quality_review.json`;
+- `repair_targets.json`.
+
+This bridge is intentionally business-first:
+- the user's question and visible assistant answer are reviewed before route ids and debug fields;
+- noisy direct answers, missing first-line answers, technical garbage, and over-broad business answers become findings;
+- generated question packs get a deterministic quality review for follow-up density, direct questions, report-style analysis, domain diversity, duplicates, and weak business anchors.
+
+Use this bridge when the operator would otherwise say “чекни прогон `assistant-stage1-...`”. The expected next step is no longer manual eyeballing first; it is: review by id, inspect `run_review.md`, map `repair_targets.json` into the current stage loop, patch, and rerun.
+
 ## Placeholder contract

 Scenario questions can reference earlier step outputs with placeholders such as:
@@ -0,0 +1,72 @@
+{
+  "$schema": "http://json-schema.org/draft-07/schema#",
+  "title": "Stage Agent Loop Manifest",
+  "type": "object",
+  "additionalProperties": true,
+  "required": ["stage_id", "module_name", "title", "pack_manifest"],
+  "properties": {
+    "schema_version": {
+      "type": "string",
+      "enum": ["stage_agent_loop_manifest_v1"]
+    },
+    "stage_id": {
+      "type": "string",
+      "minLength": 1
+    },
+    "module_name": {
+      "type": "string",
+      "minLength": 1
+    },
+    "title": {
+      "type": "string",
+      "minLength": 1
+    },
+    "architecture_phase": {
+      "type": "string"
+    },
+    "agent_focus": {
+      "type": "string"
+    },
+    "current_stage_status": {
+      "type": "string"
+    },
+    "global_plan_refs": {
+      "type": "array",
+      "items": {
+        "type": "string"
+      }
+    },
+    "pack_manifest": {
+      "type": "string",
+      "description": "Path to a domain_case_loop run-pack manifest with scenarios for the stage gate."
+    },
+    "loop_id": {
+      "type": "string"
+    },
+    "target_score": {
+      "type": "integer",
+      "minimum": 0,
+      "maximum": 100,
+      "default": 88
+    },
+    "max_iterations": {
+      "type": "integer",
+      "minimum": 1,
+      "default": 6
+    },
+    "acceptance_invariants": {
+      "type": "array",
+      "items": {
+        "type": "string"
+      }
+    },
+    "save_autorun_on_accept": {
+      "type": "boolean",
+      "default": true
+    },
+    "manual_confirmation_required_after_accept": {
+      "type": "boolean",
+      "default": true
+    }
+  }
+}