АРЧ АП11 - Архитектура после регресса: + Архитектура: восстановить bridge от provenance выбранной позиции к VAT-периоду и закрыть phase10 replay
This commit is contained in:
+18
@@ -193,6 +193,24 @@ Still open after the accepted phase9 replay:
|
||||
- business answers are now semantically correct on this path, but some inventory list formatting still feels heavier and more mechanical than the target human style;
|
||||
- the next architecture slice should keep expanding saved-session proof across additional real user chains, while separately tightening answer presentation so exact routes do not feel template-driven even when the truth path is already correct.
|
||||
|
||||
Latest phase10 bridge-and-aggregate evidence after the manual replay recovery:
|
||||
|
||||
- live replay `address_truth_harness_phase10_manual_bridge_and_aggregate_mix_live_20260418_rerun8` is accepted end-to-end with `9/9` steps green;
|
||||
- the previously broken bridge `selected item purchase provenance -> VAT on purchase date` is now explicit instead of implicit:
|
||||
- the continuity layer derives the purchase month from the grounded provenance evidence;
|
||||
- the same session keeps `selected object` continuity instead of collapsing into generic root-only VAT arbitration;
|
||||
- the runtime now routes this follow-up as `vat_liability_confirmed_for_tax_period`, not as `forecast`, `unknown`, or generic clarification;
|
||||
- the same replay also proves that the neighboring aggregate fixes are live on the real assistant path:
|
||||
- top-customer-all-time now returns a direct business answer first;
|
||||
- top-year aggregate now returns a direct business answer first;
|
||||
- very-old-stock now prefers `inventory_aging_by_purchase_date` over a generic inventory snapshot;
|
||||
- this matters architecturally because the seam that used to exist only as ambient monolith behavior is now protected as an explicit carryover contract plus replay-backed acceptance path.
|
||||
|
||||
Still open after the accepted phase10 replay:
|
||||
|
||||
- the user-facing VAT explanation block is now correct and grounded, but some long exact answers still feel heavier than the target human product tone;
|
||||
- the next architecture slice should keep moving from repaired bridge authority into answer-shaping cleanup and broader saved-session replay coverage, not back into isolated wording tweaks.
|
||||
|
||||
## Ready Signal
|
||||
|
||||
The project can leave the current breakpoint when:
|
||||
|
||||
+240
@@ -0,0 +1,240 @@
|
||||
# 12 - Manual Run System Analysis `assistant-stage1-3NilqwT1G2` (2026-04-18)
|
||||
|
||||
## Purpose
|
||||
|
||||
This note analyzes the manual saved-session run `assistant-stage1-3NilqwT1G2`.
|
||||
|
||||
The goal is not to review isolated prompts.
|
||||
|
||||
The goal is to determine:
|
||||
|
||||
- which failures should improve as orchestration / continuity authority is repaired;
|
||||
- which failures are separate route or capability gaps;
|
||||
- which failures come from answer shaping and presentation rather than from routing.
|
||||
|
||||
## Evidence
|
||||
|
||||
Primary artifacts:
|
||||
|
||||
- `llm_normalizer/reports/assistant-stage1-3NilqwT1G2.md`
|
||||
- `llm_normalizer/reports/assistant-stage1-3NilqwT1G2.json`
|
||||
- `llm_normalizer/data/assistant_sessions/assistant-stage1-3NilqwT1G2-SAVED-001.json`
|
||||
|
||||
Relevant architecture context:
|
||||
|
||||
- `10 - regression_breakpoint_analysis_2026-04-17.md`
|
||||
- `11 - continuity_stabilization_plan_2026-04-17.md`
|
||||
- `docs/orchestration/active_domain_contract.json`
|
||||
|
||||
## High-Level Verdict
|
||||
|
||||
This run is bad for more than one reason.
|
||||
|
||||
It is not a single orchestration bug.
|
||||
|
||||
It is also not a pure capability-gap run.
|
||||
|
||||
The failure pattern is mixed:
|
||||
|
||||
1. Some turns fail because the right contour already exists, but orchestration or intent selection does not reach it.
|
||||
2. Some turns fail because extracted neighboring entities no longer share an explicit bridge contract.
|
||||
3. Some turns fail because the needed business capability family is only partially modeled.
|
||||
4. Some turns are technically routed but still feel bad because answer shape and presentation are too template-like.
|
||||
|
||||
## What Still Works
|
||||
|
||||
The following contours are still alive in this run:
|
||||
|
||||
- inventory root snapshot and historical date carryover;
|
||||
- selected-item supplier provenance;
|
||||
- selected-item sale trace;
|
||||
- confirmed receivables snapshot on a date;
|
||||
- VAT confirmed for the carried tax period after a debt-date question;
|
||||
- documents by counterparty;
|
||||
- inventory root return on later turns.
|
||||
|
||||
This matters because it means the runtime foundation is not dead.
|
||||
|
||||
The project did not lose all exact routes.
|
||||
|
||||
The main problem is that the system no longer enters the correct contour reliably under mixed real-session wording.
|
||||
|
||||
## Failure Families
|
||||
|
||||
### A. Contour exists, but the runtime does not enter it
|
||||
|
||||
These are the clearest orchestration / intent-resolution failures.
|
||||
|
||||
#### `Q12` - `прикинь какой ндс нам надо заплатить на февраль 2017`
|
||||
|
||||
- Actual behavior: fell into `living_chat`, `living_chat_response_source = llm_chat`.
|
||||
- Why this is bad: the system already has `vat_liability_confirmed_for_tax_period` as a real contour.
|
||||
- Why this is architecture, not wording only: the right family exists, but the lane authority does not protect it under colloquial phrasing.
|
||||
|
||||
#### `Q16` - `мы должны комуто денег на сегодня?`
|
||||
|
||||
- Actual behavior: `detected_intent = unknown`, partial refusal.
|
||||
- Why this is bad: `payables_confirmed_as_of_date` exists in recipe, capability, filter, compose, and navigation layers.
|
||||
- Why this is architecture: the contour is present, but the resolver / route arbitration fails to bind the phrase to the payables family.
|
||||
|
||||
#### `Q29` - `Есть ли остатки товара, которые закупались очень давно`
|
||||
|
||||
- Actual behavior: routed as `inventory_on_hand_as_of_date`, returned a generic current stock snapshot.
|
||||
- Why this is bad: the active domain contract explicitly contains the old-stock / aging path, and the runtime has `inventory_aging_by_purchase_date`.
|
||||
- Why this is architecture: generic inventory root wins over the more specific aging branch because precedence is wrong.
|
||||
|
||||
### B. Cross-domain bridge disappeared after extraction
|
||||
|
||||
These are the most important "neighbor entities used to live together" failures.
|
||||
|
||||
#### `Q10` - `ндс можешь прикинуть на дату покупки рабочей станции?`
|
||||
|
||||
- Actual behavior: `detected_intent = unknown`, no route, organization only, selected-item / purchase-date context dropped.
|
||||
- What this should have used: selected item -> purchase provenance bundle -> purchase date / period -> VAT contour.
|
||||
- Why this is architecture: the system can answer selected-item provenance and can answer VAT periods, but the bridge between those adjacent contours is not explicit.
|
||||
|
||||
#### `Q17` - `а нам?`
|
||||
|
||||
- Actual behavior: rewritten into a generic "our company / our base" question and lost the debt polarity follow-up.
|
||||
- What this should have used: `payables -> receivables` role-swap follow-up over the same date scope.
|
||||
- Why this is architecture: short polarity pivots need an explicit follow-up contract, not free-form reinterpretation.
|
||||
|
||||
#### `Q23` - `а по свк`
|
||||
|
||||
- Actual behavior: stayed inside `list_documents_by_counterparty`, but the short alias retarget is semantically weak and the answer quality is poor.
|
||||
- Why this is architecture: short retargets over an existing counterparty lane need a dedicated alias / retarget seam.
|
||||
|
||||
### C. Business family partially exists, but not at the right grain
|
||||
|
||||
These are not fixed by continuity alone.
|
||||
|
||||
#### `Q13` - `кто у нас самый доходный клиент за все время`
|
||||
|
||||
- Actual behavior: `detected_intent = unknown`, partial refusal.
|
||||
- What exists nearby: `customer_revenue_and_payments`.
|
||||
- What is missing in practice: stable ranking / all-time aggregate handling for company-level customer revenue questions under colloquial wording.
|
||||
|
||||
#### `Q18` - `какой у нас самый доходный год`
|
||||
|
||||
#### `Q19` - `а за 2017 мы скок заработали?`
|
||||
|
||||
#### `Q20` - `сколько вообще денег мы заработали за все время?`
|
||||
|
||||
- Actual behavior: broad refusal or partial refusal.
|
||||
- Why this matters: these are not random questions; they are business analytics asks at company/year/all-time aggregate grain.
|
||||
- Why this is not only orchestration: current nearby revenue contour is not enough to honestly cover company-year and all-time aggregate semantics.
|
||||
|
||||
### D. Route exists nearby, but wording-to-intent binding regressed
|
||||
|
||||
#### `Q11` - `а какой ндс мы должны сгрузить на март 2020?`
|
||||
|
||||
- Actual behavior: `detected_intent = unknown`.
|
||||
- What exists nearby: VAT confirmed liability contour.
|
||||
- Why this matters: colloquial accounting wording like `сгрузить НДС` should not fall out of contour if the product claims live accountant dialogue.
|
||||
|
||||
#### `Q25` - `что нам отгружать чепурнов? какой товар или услугу?`
|
||||
|
||||
- Actual behavior: partial refusal.
|
||||
- What exists nearby: counterparty documents and related shipment evidence.
|
||||
- Why this matters: the system has adjacent evidence contours but lacks a stable "counterparty -> shipped items/services" interpretation.
|
||||
|
||||
### E. User-facing quality defects even when routing is not fully broken
|
||||
|
||||
#### `Q2` - `расскажи что можешь интересного`
|
||||
|
||||
- The answer is cleaner than before, but still catalog-like and mechanically enumerated.
|
||||
|
||||
#### `Q14` / `Q15`
|
||||
|
||||
- These turns route correctly, but the answer shape still exposes internal "block" thinking and feels less human than the target product tone.
|
||||
|
||||
#### `Q28`
|
||||
|
||||
- The route works, but the result relies on fallback reasoning (`confirmed_balance_unavailable_fallback_to_heuristic_candidates` / `open_items_account_query_override_to_movements`), so business confidence should stay explicit.
|
||||
|
||||
## System Classification
|
||||
|
||||
This run should be read as:
|
||||
|
||||
- `orchestration / continuity authority failures`: `Q10`, `Q12`, `Q16`, `Q17`, `Q23`, `Q29`
|
||||
- `resolver / wording-to-intent failures`: `Q11`, `Q13`, `Q25`
|
||||
- `aggregate business contour gaps`: `Q18`, `Q19`, `Q20`
|
||||
- `answer-shape / presentation defects`: `Q2`, `Q14`, `Q15`, `Q28`
|
||||
|
||||
The important conclusion:
|
||||
|
||||
- a large share of the bad run is still architecture and orchestration;
|
||||
- but not all of it will heal automatically just by continuing continuity fixes.
|
||||
|
||||
## What The Old Monolith Likely Had
|
||||
|
||||
Before extraction, several neighboring decisions lived in one ambient runtime region:
|
||||
|
||||
- intent guess;
|
||||
- follow-up carryover;
|
||||
- selected-object reuse;
|
||||
- organization/date reuse;
|
||||
- recipe selection;
|
||||
- answer packaging.
|
||||
|
||||
That old shape was fragile, but it allowed adjacent capabilities to "borrow" context from one another without explicit contracts.
|
||||
|
||||
After extraction, those same borrow paths became seams.
|
||||
|
||||
The system now needs those seams to be first-class contracts.
|
||||
|
||||
Without that, optimization does not produce a stronger stone.
|
||||
|
||||
It produces split pieces whose joints are filled by ad hoc glue.
|
||||
|
||||
## Architecture Requirement
|
||||
|
||||
The extraction path is still the right direction only if every broken adjacency becomes explicit.
|
||||
|
||||
The missing explicit seams now visible in this run are:
|
||||
|
||||
- `inventory root <-> inventory aging`
|
||||
- `selected item / provenance bundle <-> VAT period ask`
|
||||
- `payables <-> receivables polarity flip`
|
||||
- `counterparty documents <-> short alias retarget`
|
||||
- `counterparty evidence <-> shipped goods/services summary`
|
||||
- `organization scope <-> company analytics aggregates`
|
||||
|
||||
Each seam needs:
|
||||
|
||||
1. an explicit transition / carryover contract;
|
||||
2. one clear intent-family arbitration rule;
|
||||
3. answer-shape expectations for the user-facing reply;
|
||||
4. replay coverage inside a saved-session scenario, not only in isolated unit cases.
|
||||
|
||||
## Final Conclusion
|
||||
|
||||
Yes, the architecture can become better than the old monolith.
|
||||
|
||||
But only under one condition:
|
||||
|
||||
- every extracted neighbor relationship that used to work implicitly must now be reintroduced as an explicit contract and guarded by scenario-level acceptance.
|
||||
|
||||
If that is done, the stone becomes better:
|
||||
|
||||
- less magical;
|
||||
- more inspectable;
|
||||
- safer to expand across domains.
|
||||
|
||||
If that is not done, the stone does not become better.
|
||||
|
||||
It becomes fragmented and cosmetically re-glued.
|
||||
|
||||
## Next Recommended Slice
|
||||
|
||||
The next phase should not be a random patch sweep.
|
||||
|
||||
It should be one focused replay pack built from this exact saved session with priority order:
|
||||
|
||||
1. `inventory -> selected item -> VAT purchase-date bridge`
|
||||
2. `direct VAT colloquial liability wording`
|
||||
3. `payables / receivables polarity swap`
|
||||
4. `inventory aging vs generic inventory precedence`
|
||||
5. `company analytics aggregates`
|
||||
|
||||
That pack should become the next acceptance gate before wider domain expansion.
|
||||
+265
@@ -0,0 +1,265 @@
|
||||
{
|
||||
"schema_version": "domain_truth_harness_spec_v1",
|
||||
"scenario_id": "address_truth_harness_phase10_manual_bridge_and_aggregate_mix",
|
||||
"domain": "address_phase10_manual_bridge_and_aggregate_mix",
|
||||
"title": "Phase 10 manual bridge and aggregate replay for orchestration recovery",
|
||||
"description": "Focused AGENT replay built from the manual saved session assistant-stage1-3NilqwT1G2. The scenario validates broken seams where neighboring contours used to cooperate implicitly: inventory -> selected item -> VAT purchase-date bridge, colloquial VAT liability wording, payables/receivables polarity flip, company-level revenue aggregates, and inventory aging versus generic stock snapshot precedence.",
|
||||
"bindings": {},
|
||||
"steps": [
|
||||
{
|
||||
"step_id": "step_01_inventory_historical_anchor",
|
||||
"title": "Historical inventory anchor on March 2016",
|
||||
"question": "остатки на март 2016",
|
||||
"allowed_reply_types": [
|
||||
"factual"
|
||||
],
|
||||
"expected_intents": [
|
||||
"inventory_on_hand_as_of_date"
|
||||
],
|
||||
"expected_recipe": "address_inventory_on_hand_as_of_date_v1",
|
||||
"required_filters": {
|
||||
"as_of_date": "2016-03-31",
|
||||
"period_from": "2016-03-01",
|
||||
"period_to": "2016-03-31"
|
||||
},
|
||||
"required_direct_answer_patterns_any": [
|
||||
"31\\.03\\.2016",
|
||||
"(?i)на складе"
|
||||
],
|
||||
"criticality": "critical",
|
||||
"semantic_tags": [
|
||||
"inventory_root",
|
||||
"historical_date_anchor"
|
||||
]
|
||||
},
|
||||
{
|
||||
"step_id": "step_02_selected_item_purchase_provenance",
|
||||
"title": "Selected workstation supplier / purchase provenance on the same date",
|
||||
"question": "По выбранному объекту \"Рабочая станция универсального специалиста (индивидуальное изготовление)\": где взяли это?",
|
||||
"allowed_reply_types": [
|
||||
"factual",
|
||||
"partial_coverage"
|
||||
],
|
||||
"expected_intents": [
|
||||
"inventory_purchase_provenance_for_item"
|
||||
],
|
||||
"expected_recipe": "address_inventory_purchase_provenance_for_item_v1",
|
||||
"required_filters": {
|
||||
"item": "Рабочая станция универсального специалиста (индивидуальное изготовление)",
|
||||
"as_of_date": "{{step_01_inventory_historical_anchor.filters.as_of_date}}"
|
||||
},
|
||||
"required_direct_answer_patterns_any": [
|
||||
"(?i)поставщик|закуп",
|
||||
"(?i)рабочая станция"
|
||||
],
|
||||
"criticality": "critical",
|
||||
"semantic_tags": [
|
||||
"selected_object",
|
||||
"inventory_provenance",
|
||||
"date_scope"
|
||||
]
|
||||
},
|
||||
{
|
||||
"step_id": "step_03_vat_on_purchase_date_bridge",
|
||||
"title": "VAT approximate on the purchase date of the selected item",
|
||||
"question": "ндс можешь прикинуть на дату покупки рабочей станции?",
|
||||
"allowed_reply_types": [
|
||||
"factual",
|
||||
"factual_with_explanation",
|
||||
"partial_coverage"
|
||||
],
|
||||
"expected_intents": [
|
||||
"vat_liability_confirmed_for_tax_period"
|
||||
],
|
||||
"required_direct_answer_patterns_any": [
|
||||
"(?i)ндс",
|
||||
"(?i)дата покупки|период|налоговый период|не удается точно"
|
||||
],
|
||||
"forbidden_direct_answer_patterns": [
|
||||
"(?i)mcp",
|
||||
"(?i)address_mode",
|
||||
"(?i)tool_gate_reason"
|
||||
],
|
||||
"criticality": "critical",
|
||||
"semantic_tags": [
|
||||
"bridge_inventory_to_vat",
|
||||
"selected_object",
|
||||
"purchase_date_bundle"
|
||||
]
|
||||
},
|
||||
{
|
||||
"step_id": "step_04_colloquial_vat_tax_period",
|
||||
"title": "Colloquial VAT liability wording on February 2017",
|
||||
"question": "прикинь какой ндс нам надо заплатить на февраль 2017",
|
||||
"allowed_reply_types": [
|
||||
"factual",
|
||||
"factual_with_explanation",
|
||||
"partial_coverage"
|
||||
],
|
||||
"expected_intents": [
|
||||
"vat_liability_confirmed_for_tax_period"
|
||||
],
|
||||
"expected_recipe": "address_vat_liability_confirmed_tax_period_v1",
|
||||
"required_filters": {
|
||||
"period_from": "2017-02-01",
|
||||
"period_to": "2017-02-28"
|
||||
},
|
||||
"required_direct_answer_patterns_any": [
|
||||
"(?i)ндс",
|
||||
"(?i)2017"
|
||||
],
|
||||
"forbidden_direct_answer_patterns": [
|
||||
"(?i)mcp",
|
||||
"(?i)address_mode",
|
||||
"(?i)tool_gate_reason"
|
||||
],
|
||||
"criticality": "critical",
|
||||
"semantic_tags": [
|
||||
"vat_colloquial_wording",
|
||||
"tax_period"
|
||||
]
|
||||
},
|
||||
{
|
||||
"step_id": "step_05_payables_today",
|
||||
"title": "Payables snapshot for today",
|
||||
"question": "мы должны комуто денег на сегодня?",
|
||||
"allowed_reply_types": [
|
||||
"factual",
|
||||
"partial_coverage"
|
||||
],
|
||||
"expected_intents": [
|
||||
"payables_confirmed_as_of_date",
|
||||
"list_payables_counterparties"
|
||||
],
|
||||
"required_direct_answer_patterns_any": [
|
||||
"(?i)кредитор|обязатель|долж",
|
||||
"(?i)сегодня|на .*20"
|
||||
],
|
||||
"forbidden_direct_answer_patterns": [
|
||||
"(?i)mcp",
|
||||
"(?i)address_mode",
|
||||
"(?i)tool_gate_reason"
|
||||
],
|
||||
"criticality": "critical",
|
||||
"semantic_tags": [
|
||||
"payables",
|
||||
"debt_polarity"
|
||||
]
|
||||
},
|
||||
{
|
||||
"step_id": "step_06_receivables_role_flip",
|
||||
"title": "Short polarity flip from payables to receivables",
|
||||
"question": "а нам?",
|
||||
"allowed_reply_types": [
|
||||
"factual",
|
||||
"partial_coverage"
|
||||
],
|
||||
"expected_intents": [
|
||||
"receivables_confirmed_as_of_date",
|
||||
"list_receivables_counterparties"
|
||||
],
|
||||
"required_direct_answer_patterns_any": [
|
||||
"(?i)дебитор|нам должны|должны нам",
|
||||
"(?i)сегодня|на .*20|не удается точно"
|
||||
],
|
||||
"forbidden_direct_answer_patterns": [
|
||||
"(?i)что относится к вашей компании",
|
||||
"(?i)mcp",
|
||||
"(?i)address_mode",
|
||||
"(?i)tool_gate_reason"
|
||||
],
|
||||
"criticality": "critical",
|
||||
"semantic_tags": [
|
||||
"receivables",
|
||||
"polarity_flip",
|
||||
"followup_short"
|
||||
]
|
||||
},
|
||||
{
|
||||
"step_id": "step_07_top_customer_all_time",
|
||||
"title": "Top customer by all-time revenue",
|
||||
"question": "кто у нас самый доходный клиент за все время",
|
||||
"allowed_reply_types": [
|
||||
"factual",
|
||||
"factual_with_explanation",
|
||||
"partial_coverage",
|
||||
"clarification_required"
|
||||
],
|
||||
"expected_intents": [
|
||||
"customer_revenue_and_payments"
|
||||
],
|
||||
"required_direct_answer_patterns_any": [
|
||||
"(?i)клиент|контрагент|доход|выруч",
|
||||
"(?i)не удается точно|за все время|самый доходный"
|
||||
],
|
||||
"forbidden_direct_answer_patterns": [
|
||||
"(?i)mcp",
|
||||
"(?i)address_mode",
|
||||
"(?i)tool_gate_reason"
|
||||
],
|
||||
"criticality": "critical",
|
||||
"semantic_tags": [
|
||||
"customer_analytics",
|
||||
"aggregate_all_time",
|
||||
"ranking"
|
||||
]
|
||||
},
|
||||
{
|
||||
"step_id": "step_08_top_year_revenue",
|
||||
"title": "Most profitable year",
|
||||
"question": "какой у нас самый доходный год",
|
||||
"allowed_reply_types": [
|
||||
"factual",
|
||||
"factual_with_explanation",
|
||||
"partial_coverage",
|
||||
"clarification_required"
|
||||
],
|
||||
"expected_intents": [
|
||||
"customer_revenue_and_payments"
|
||||
],
|
||||
"required_direct_answer_patterns_any": [
|
||||
"(?i)год|доход|выруч|прибыл",
|
||||
"(?i)не удается точно|самый доходный"
|
||||
],
|
||||
"forbidden_direct_answer_patterns": [
|
||||
"(?i)mcp",
|
||||
"(?i)address_mode",
|
||||
"(?i)tool_gate_reason"
|
||||
],
|
||||
"criticality": "critical",
|
||||
"semantic_tags": [
|
||||
"company_analytics",
|
||||
"aggregate_year",
|
||||
"ranking"
|
||||
]
|
||||
},
|
||||
{
|
||||
"step_id": "step_09_very_old_stock",
|
||||
"title": "Very old stock should prefer aging contour over generic inventory snapshot",
|
||||
"question": "Есть ли остатки товара, которые закупались очень давно",
|
||||
"allowed_reply_types": [
|
||||
"factual",
|
||||
"factual_with_explanation",
|
||||
"partial_coverage"
|
||||
],
|
||||
"expected_intents": [
|
||||
"inventory_aging_by_purchase_date"
|
||||
],
|
||||
"expected_recipe": "address_inventory_aging_by_purchase_date_v1",
|
||||
"required_direct_answer_patterns_any": [
|
||||
"(?i)стар|давно|закуп",
|
||||
"(?i)остат"
|
||||
],
|
||||
"forbidden_direct_answer_patterns": [
|
||||
"(?i)18\\.04\\.2026",
|
||||
"(?i)на складе подтверждено \\d+ позиц"
|
||||
],
|
||||
"criticality": "critical",
|
||||
"semantic_tags": [
|
||||
"inventory_aging",
|
||||
"very_old_stock",
|
||||
"precedence"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
Reference in New Issue
Block a user