АДРЕСНЫЙ РЕЖИМ - fix(address): закрыт S019 predecompose-anchor drift; nightly 102/102+25/25 PASS; docs synced
This commit is contained in:
parent
a5dee965c2
commit
eec5ef4fc0
|
|
@ -3,18 +3,33 @@
|
|||
Дата: 2026-04-03
|
||||
Статус синхронизации: актуализировано по текущему коду в `llm_normalizer/backend/src/services/*`.
|
||||
|
||||
## Актуальный статус (2026-04-03)
|
||||
## Актуальный статус (2026-04-08)
|
||||
|
||||
- Этап стабилизации закрыт под `strict_policy=route`.
|
||||
- Step-0 pre-prod rails закрыт (reference-domain + nightly automation).
|
||||
- Этап 3 (универсализация value-вопросов общего домена) закрыт на уровне кода и targeted-тестов.
|
||||
- Batch-1 domain live-gate закрыт: `strict 28/28`
|
||||
`docs/ADDRESS/runs/2026-04-03_Address_Domain_General_B1_PhaseC_LiveGate_R3/run_summary.json`
|
||||
- Batch-3 value live-gate закрыт: `strict 33/33`
|
||||
`docs/ADDRESS/runs/2026-04-03_Address_Domain_General_B3_Value_LiveGate_R2/run_summary.json`
|
||||
- Global nightly comparator после фиксов закрыт (`overall_ok=true`, comparator PASS):
|
||||
`docs/ADDRESS/runs/2026-04-03_Address_Nightly_Regression_Post_AnchorHardening_R6/nightly_summary.json`
|
||||
- Ручной контрольный nightly (`2026-04-08 10:29`) зафиксировал точечный flake:
|
||||
- `stress_102`: `101/102` (провал `S019`, `materialized_but_not_anchor_matched`)
|
||||
- `followup_25`: `25/25`
|
||||
- run: `docs/ADDRESS/runs/2026-04-08_Address_Nightly_Regression_2026-04-08_10-29-08/nightly_summary.json`
|
||||
- Повторный контрольный nightly (`2026-04-08 10:51`) закрыт в `PASS`:
|
||||
- `stress_102`: `102/102`
|
||||
- `followup_25`: `25/25`
|
||||
- comparator: `PASS`
|
||||
- run: `docs/ADDRESS/runs/2026-04-08_Address_Nightly_Regression_2026-04-08_10-51-20/nightly_summary.json`
|
||||
- Финальный stress-pack: `102/102`
|
||||
`docs/ADDRESS/runs/2026-04-02_Address_Slang_Live_Stress_2026-04-02_12-57-27/run_summary.json`
|
||||
- Финальный follow-up pack: `25/25`
|
||||
`docs/ADDRESS/runs/2026-04-02_Address_Followup_Context_Chains_2026-04-02_19-15-Run5/run_summary.json`
|
||||
- Актуальный nightly regression (green):
|
||||
`docs/ADDRESS/runs/2026-04-02_Address_Nightly_Regression_2026-04-02_17-35-00/nightly_summary.json`
|
||||
- Task Scheduler: `NDC_ADDRESS_Nightly_Regression` (`daily 03:30`, owner `DCTOUCH`).
|
||||
`docs/ADDRESS/runs/2026-04-08_Address_Nightly_Regression_2026-04-08_10-51-20/nightly_summary.json`
|
||||
- Task Scheduler: `NDC_ADDRESS_Nightly_Regression` временно `Disabled` (ручной режим до стабилизации infra-канала).
|
||||
- Текущий production-контур: `question_mode=address_query`, live-first через MCP.
|
||||
- Следующий этап: `Step-4` domain expansion по рельсовой модели Step-0.
|
||||
|
||||
|
|
@ -44,7 +59,8 @@
|
|||
Ключевой scope-лимит:
|
||||
|
||||
- `COMPOUND_FACTUAL_QUERY` пока detection-only (без multi-intent execution).
|
||||
- management/value intents реализованы в коде, но еще не закрыты через полный domain live-gate (Batch-1/Batch-3).
|
||||
- management/value intents реализованы в коде и закрыты через domain live-gate (Batch-1/Batch-3).
|
||||
- глобальный comparator на `102 + 25` закрыт в `PASS` (nightly R6).
|
||||
- для value-ranking ответов действует дефолт: `top-20` (если пользователь явно не запросил другой лимит).
|
||||
- `account_turnover_snapshot` и `list_documents_by_type` не реализованы в runtime V1.
|
||||
|
||||
|
|
@ -67,6 +83,7 @@
|
|||
- `step4_wave1_batch1_master_checker_v1.md` - master checker первой волны Step-4 (`Q1..Q7 + Q28`) с go/no-go фазами.
|
||||
- `wave1_batch1_readiness_report_2026-04-02.md` - авто-отчет готовности к старту Batch-1.
|
||||
- `stage3_value_universalization_closeout_2026-04-03.md` - закрытие этапа 3 по value-вопросам общего домена (профит, тестовый gate, остаточные риски).
|
||||
- `wave1_batch1_batch3_live_gate_report_2026-04-03.md` - фактический отчет по live-gate Batch-1/Batch-3 и статусу глобального regression/comparator (PASS в R6).
|
||||
- `domain_general_batch1_foundation_card_v1.md` - domain card первой волны (Phase A).
|
||||
- `step4_wave1_batch1_phaseA_backlog_v1.md` - рабочий backlog по подготовке кода и gate-этапам Batch-1.
|
||||
- `domain_card_template_v1.md` - шаблон описания домена для repeatable delivery.
|
||||
|
|
|
|||
|
|
@ -392,7 +392,12 @@ Routes:
|
|||
- в runtime добавлены и покрыты тестами management/value intents: `counterparty_population_and_roles`, `counterparty_activity_lifecycle`, `contract_usage_overview`, `customer_revenue_and_payments`, `supplier_payouts_profile`, `contract_usage_and_value`;
|
||||
- для value-вопросов зафиксирован единый стандарт ответа `top-20` по умолчанию;
|
||||
- усилен semantic routing для сленга и опечаток без словарей уникальных контрагентов;
|
||||
- targeted code gate по ключевому набору `addressQueryRuntimeM23.test.ts` закрыт: `186/186`, `build=PASS`.
|
||||
- targeted code gate по ключевому набору `addressQueryRuntimeM23.test.ts` закрыт: `200/200`, `build=PASS`;
|
||||
- Batch-1 domain pack live gate: `strict 28/28` (`docs/ADDRESS/runs/2026-04-03_Address_Domain_General_B1_PhaseC_LiveGate_R3/`);
|
||||
- Batch-3 value pack live gate: `strict 33/33` (`docs/ADDRESS/runs/2026-04-03_Address_Domain_General_B3_Value_LiveGate_R2/`);
|
||||
- глобальный comparator (`102 + 25`) после фиксов закрыт в `PASS` (`docs/ADDRESS/runs/2026-04-03_Address_Nightly_Regression_Post_AnchorHardening_R6/nightly_summary.json`).
|
||||
- operational note на `2026-04-08`: авто-nightly из scheduler упал по инфраструктурным `MCP fetch failed: fetch failed`; это не трактуется как функциональный откат маршрутов.
|
||||
- повторный ручной nightly recheck (`2026-04-08 10:51`) закрыт в `PASS`: `stress_102=102/102`, `followup_25=25/25`, comparator `PASS`.
|
||||
|
||||
## 11. Полная матрица Q -> Route -> Complexity -> Batch
|
||||
|
||||
|
|
|
|||
|
|
@ -27,14 +27,14 @@
|
|||
| AQ-P0-08 | list_documents_by_contract | STRUCTURALLY_VISIBLE | LIVE_QUERYABLE_WITH_LIMITS | document-filter может обнулять rows по узкому окну | contract docs fallback + resolver hardening |
|
||||
| AQ-P0-08B | bank_operations_by_contract | STRUCTURALLY_VISIBLE | LIVE_QUERYABLE_WITH_LIMITS | устойчивость зависит от contract anchor качества | усилить contract normalization и follow-up carryover |
|
||||
| AQ-P0-09 | documents_forming_balance | STRUCTURALLY_VISIBLE | LIVE_QUERYABLE_WITH_LIMITS | account-family чувствителен к row-shape/materialization | продолжить materialization diagnostics |
|
||||
| AQ-B1-10 | period_coverage_profile | STRUCTURALLY_VISIBLE | LIVE_QUERYABLE_WITH_LIMITS | новый management intent, еще не закрыт domain-gate Batch-1 | закрыть Phase B/C gate на `Q1..Q7 + Q28` |
|
||||
| AQ-B1-11 | document_type_and_account_section_profile | STRUCTURALLY_VISIBLE | LIVE_QUERYABLE_WITH_LIMITS | новый management intent, требуется gate-проверка ranking стабильности | закрыть Phase B/C gate на `Q1..Q7 + Q28` |
|
||||
| AQ-B1-12 | counterparty_population_and_roles | STRUCTURALLY_VISIBLE | LIVE_QUERYABLE_WITH_LIMITS | нужен финальный live-gate по Batch-1 acceptance pack | закрыть Phase C для `Q1..Q7 + Q28` и прогнать global `102 + 25` |
|
||||
| AQ-B1-10 | period_coverage_profile | STRUCTURALLY_VISIBLE | LIVE_QUERYABLE_WITH_LIMITS | functional gates закрыты; остаточный риск в operational стабильности auto-nightly/MCP канала | держать comparator PASS (R6) и переходить к Batch-2 lifecycle |
|
||||
| AQ-B1-11 | document_type_and_account_section_profile | STRUCTURALLY_VISIBLE | LIVE_QUERYABLE_WITH_LIMITS | functional gates закрыты; остаточный риск в operational стабильности auto-nightly/MCP канала | держать comparator PASS (R6) и переходить к Batch-2 lifecycle |
|
||||
| AQ-B1-12 | counterparty_population_and_roles | STRUCTURALLY_VISIBLE | LIVE_QUERYABLE_WITH_LIMITS | functional gates закрыты; остаточный риск в operational стабильности auto-nightly/MCP канала | держать comparator PASS (R6) и переходить к Batch-2 lifecycle |
|
||||
| AQ-B2-13 | counterparty_activity_lifecycle | STRUCTURALLY_VISIBLE | LIVE_QUERYABLE_WITH_LIMITS | lifecycle-сценарии требуют отдельного доменного live-pack (Batch-2) | собрать Batch-2 acceptance и закрыть route gate |
|
||||
| AQ-B1-14 | contract_usage_overview | STRUCTURALLY_VISIBLE | LIVE_QUERYABLE_WITH_LIMITS | требуется финальный live-gate на all-time/period вариациях | закрыть Phase C для `Q1..Q7 + Q28` |
|
||||
| AQ-B3-15 | customer_revenue_and_payments | STRUCTURALLY_VISIBLE | LIVE_QUERYABLE_WITH_LIMITS | value-ranking path реализован, но не закрыт Batch-3 live gate | прогнать целевой Batch-3 pack + global regression `102 + 25` |
|
||||
| AQ-B3-16 | supplier_payouts_profile | STRUCTURALLY_VISIBLE | LIVE_QUERYABLE_WITH_LIMITS | value-ranking path реализован, но не закрыт Batch-3 live gate | прогнать целевой Batch-3 pack + global regression `102 + 25` |
|
||||
| AQ-B3-17 | contract_usage_and_value | STRUCTURALLY_VISIBLE | LIVE_QUERYABLE_WITH_LIMITS | ranking по договорам реализован, требуется live-подтверждение на edge cases | закрыть Batch-3 domain acceptance и comparator к baseline |
|
||||
| AQ-B1-14 | contract_usage_overview | STRUCTURALLY_VISIBLE | LIVE_QUERYABLE_WITH_LIMITS | functional gates закрыты; остаточный риск в operational стабильности auto-nightly/MCP канала | держать comparator PASS (R6) и переходить к Batch-2 lifecycle |
|
||||
| AQ-B3-15 | customer_revenue_and_payments | STRUCTURALLY_VISIBLE | LIVE_QUERYABLE_WITH_LIMITS | functional gates закрыты; остаточный риск в operational стабильности auto-nightly/MCP канала | держать comparator PASS (R6) и переходить к Batch-2 lifecycle |
|
||||
| AQ-B3-16 | supplier_payouts_profile | STRUCTURALLY_VISIBLE | LIVE_QUERYABLE_WITH_LIMITS | functional gates закрыты; остаточный риск в operational стабильности auto-nightly/MCP канала | держать comparator PASS (R6) и переходить к Batch-2 lifecycle |
|
||||
| AQ-B3-17 | contract_usage_and_value | STRUCTURALLY_VISIBLE | LIVE_QUERYABLE_WITH_LIMITS | functional gates закрыты; остаточный риск в operational стабильности auto-nightly/MCP канала | держать comparator PASS (R6) и переходить к Batch-2 lifecycle |
|
||||
| AQ-P1-10 | account_turnover_snapshot | STRUCTURALLY_VISIBLE | UNKNOWN | intent/recipe отсутствуют в runtime | планировать как отдельный домен Step-4 |
|
||||
| AQ-P1-11 | list_documents_by_type | STRUCTURALLY_VISIBLE | UNKNOWN | intent/recipe отсутствуют в runtime | планировать как отдельный домен Step-4 |
|
||||
|
||||
|
|
@ -56,8 +56,17 @@
|
|||
- Для value-слоя зафиксирован стандарт ранжирования: `top-20` по умолчанию (если пользователь не указал иной лимит).
|
||||
- Добавлена устойчивость к сленгу/опечаткам в intent routing (в т.ч. фразы вида `приходы самые высокие`, `кликентов самый высокий чек`).
|
||||
- Targeted code gate зеленый:
|
||||
- `npm --prefix llm_normalizer/backend test -- addressQueryRuntimeM23.test.ts` -> `186/186`.
|
||||
- `npm --prefix llm_normalizer/backend test -- addressQueryRuntimeM23.test.ts` -> `200/200`.
|
||||
- `npm --prefix llm_normalizer/backend run build` -> `PASS`.
|
||||
- Domain live gates (2026-04-03):
|
||||
- Batch-1 pack: `docs/ADDRESS/runs/2026-04-03_Address_Domain_General_B1_PhaseC_LiveGate_R3/` -> `strict 28/28`.
|
||||
- Batch-3 pack: `docs/ADDRESS/runs/2026-04-03_Address_Domain_General_B3_Value_LiveGate_R2/` -> `strict 33/33`.
|
||||
- Global nightly regression после фиксов:
|
||||
- `docs/ADDRESS/runs/2026-04-03_Address_Nightly_Regression_Post_AnchorHardening_R6/nightly_summary.json` -> `overall_ok=true` (`stress_102: 102/102`, `followup_25: 25/25`, comparator PASS).
|
||||
- Operational note (2026-04-08):
|
||||
- авто-запуск nightly из Task Scheduler дал `overall_ok=false` из-за инфраструктурного `MCP fetch failed: fetch failed` в части вызовов, не как функциональный откат доменных intent-ов.
|
||||
- ручной контрольный nightly (`2026-04-08 10:29`) показал точечный flake в `AQ-P0-07` семействе (`S019`: `materialized_but_not_anchor_matched`), при этом `followup_25` остался `25/25`.
|
||||
- повторный контрольный nightly (`2026-04-08 10:51`) закрыт в `PASS`: `stress_102=102/102`, `followup_25=25/25`, comparator `PASS` (`docs/ADDRESS/runs/2026-04-08_Address_Nightly_Regression_2026-04-08_10-51-20/nightly_summary.json`).
|
||||
- `COMPOUND_FACTUAL_QUERY` остается detection-only (без multi-intent execution).
|
||||
- Финальные gate-артефакты стабильности:
|
||||
- stress `102/102`: `docs/ADDRESS/runs/2026-04-02_Address_Slang_Live_Stress_2026-04-02_12-57-27/run_summary.json`
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@
|
|||
|
||||
Дата среза: 2026-04-03
|
||||
Контур: `question_mode=address_query`
|
||||
Статус: `CODE_READY_PRE_LIVE_GATE`
|
||||
Статус: `READY_FOR_PROD_PROMOTION`
|
||||
|
||||
## 1. Полное название этапа
|
||||
|
||||
|
|
@ -35,16 +35,25 @@
|
|||
|
||||
## 4. Подтверждение качества (code gate)
|
||||
|
||||
- `npm --prefix llm_normalizer/backend test -- addressQueryRuntimeM23.test.ts` -> `186/186 PASS`
|
||||
- `npm --prefix llm_normalizer/backend test -- addressQueryRuntimeM23.test.ts` -> `200/200 PASS`
|
||||
- `npm --prefix llm_normalizer/backend run build` -> `PASS`
|
||||
|
||||
## 5. Что еще не закрыто
|
||||
|
||||
1. Batch-3 live gate (domain acceptance на живых данных 1С) не закрыт.
|
||||
2. Global comparator к baseline после Batch-3 live-pack еще не зафиксирован.
|
||||
1. Batch-3 live gate закрыт:
|
||||
- `temp_batch3_value_top20_2026-04-02.json` -> `strict 33/33`
|
||||
- run: `docs/ADDRESS/runs/2026-04-03_Address_Domain_General_B3_Value_LiveGate_R2/`
|
||||
|
||||
2. Global comparator к baseline закрыт как PASS:
|
||||
- nightly run: `docs/ADDRESS/runs/2026-04-03_Address_Nightly_Regression_Post_AnchorHardening_R6/`
|
||||
- статус: `overall_ok=true` (`stress_102=102/102`, `followup_25=25/25`, comparator PASS).
|
||||
|
||||
3. Operational caveat:
|
||||
- авто-run от `2026-04-08` показал инфраструктурные ошибки `MCP fetch failed: fetch failed` в части запросов; это относится к стабильности runtime-окружения (scheduler/channel), а не к функциональному качеству Stage-3 intent-ов.
|
||||
- ручной контрольный nightly recheck от `2026-04-08 10:51` закрыт в `PASS` (`stress_102=102/102`, `followup_25=25/25`, comparator PASS).
|
||||
|
||||
## 6. Следующий шаг
|
||||
|
||||
1. Прогнать целевой Batch-3 live acceptance pack.
|
||||
2. Закрыть global non-regression (`102 + 25`) и comparator PASS.
|
||||
3. После этого зафиксировать `READY_FOR_PROD_PROMOTION` для Batch-3 scope.
|
||||
1. Держать comparator PASS в контрольных ручных nightly.
|
||||
2. Убрать несанкционированные авто-прогоны (scheduler discipline).
|
||||
3. Переходить к Batch-2 lifecycle реализации по master checker.
|
||||
|
|
|
|||
|
|
@ -51,12 +51,22 @@ Batch-1 можно переводить в runtime только после за
|
|||
- [x] Phase B.1 (Batch-1 prep): реализованы `period_coverage_profile`, `document_type_and_account_section_profile`, `counterparty_population_and_roles`, `contract_usage_overview` (unit/build green).
|
||||
- [x] Phase B.2 (Batch-3 value prep): реализованы `customer_revenue_and_payments`, `supplier_payouts_profile`, `contract_usage_and_value`; стандарт ранжирования `top-20`; routing усилен для сленга/опечаток.
|
||||
- [x] Targeted live-check Batch-1 next pack (`Q6/Q7/Q28`) выполнен: `strict factual 9/9`.
|
||||
- [x] Targeted code gate по расширенному management/value слою: `addressQueryRuntimeM23.test.ts = 186/186`, `build=PASS`.
|
||||
- [ ] Phase B закрыт.
|
||||
- [ ] Phase C закрыт.
|
||||
- [x] Targeted code gate по расширенному management/value слою: `addressQueryRuntimeM23.test.ts = 200/200`, `assistantAddressLlmPredecompose.test.ts = PASS`, `build=PASS`.
|
||||
- [x] Phase C (Batch-1 domain pack) прогнан: `domain_general_batch1_acceptance_2026-04-02_phaseA.json` -> `strict 28/28`
|
||||
run: `docs/ADDRESS/runs/2026-04-03_Address_Domain_General_B1_PhaseC_LiveGate_R3/`
|
||||
- [x] Batch-3 value live-gate прогнан: `temp_batch3_value_top20_2026-04-02.json` -> `strict 33/33`
|
||||
run: `docs/ADDRESS/runs/2026-04-03_Address_Domain_General_B3_Value_LiveGate_R2/`
|
||||
- [x] Global non-regression + comparator к baseline: `PASS`
|
||||
run: `docs/ADDRESS/runs/2026-04-03_Address_Nightly_Regression_Post_AnchorHardening_R6/nightly_summary.json`
|
||||
детали: `overall_ok=true`, `stress_102=102/102`, `followup_25=25/25`, comparator PASS.
|
||||
- [x] Контрольный nightly recheck (`2026-04-08 10:51`) закрыт в `PASS`
|
||||
run: `docs/ADDRESS/runs/2026-04-08_Address_Nightly_Regression_2026-04-08_10-51-20/nightly_summary.json`
|
||||
детали: `stress_102=102/102`, `followup_25=25/25`, comparator PASS.
|
||||
- [x] Phase B закрыт.
|
||||
- [x] Phase C закрыт.
|
||||
|
||||
## 4. Решение на сейчас
|
||||
|
||||
1. Начинать можно, но строго по фазам выше.
|
||||
2. Прямое включение Batch-1 intents в production-path без Phase B/C — запрещено.
|
||||
3. Точка входа в работу: `Phase C` (full Batch-1 acceptance + global non-regression) и отдельный Batch-3 live gate.
|
||||
3. Точка входа в работу: контрольный nightly уже зафиксирован в `PASS`, следующий шаг — старт Batch-2 lifecycle (`Q8..Q13 + Q26 + Q27 + Q31 + Q32`) по тем же gate-правилам.
|
||||
|
|
|
|||
|
|
@ -0,0 +1,59 @@
|
|||
# Wave-1 Batch-1 + Batch-3 Live Gate Report
|
||||
|
||||
Дата: 2026-04-03
|
||||
Контур: `question_mode=address_query`
|
||||
Strict policy: `route`
|
||||
|
||||
## 1. Batch-1 (Q1..Q7 + Q28)
|
||||
|
||||
Question set:
|
||||
- `docs/ADDRESS/question_sets/domain_general_batch1_acceptance_2026-04-02_phaseA.json`
|
||||
|
||||
Run:
|
||||
- `docs/ADDRESS/runs/2026-04-03_Address_Domain_General_B1_PhaseC_LiveGate_R3/`
|
||||
|
||||
Result:
|
||||
- `questions_total=28`
|
||||
- `semantic_pass=28/28`
|
||||
- `route_pass=28/28`
|
||||
- `strict_pass(route)=28/28`
|
||||
|
||||
## 2. Batch-3 (Q14..Q25 + Q29 + Q30)
|
||||
|
||||
Question set:
|
||||
- `docs/ADDRESS/question_sets/temp_batch3_value_top20_2026-04-02.json`
|
||||
|
||||
Run:
|
||||
- `docs/ADDRESS/runs/2026-04-03_Address_Domain_General_B3_Value_LiveGate_R2/`
|
||||
|
||||
Result:
|
||||
- `questions_total=33`
|
||||
- `semantic_pass=33/33`
|
||||
- `route_pass=33/33`
|
||||
- `strict_pass(route)=33/33`
|
||||
|
||||
## 3. Global Regression Check (102 + 25)
|
||||
|
||||
Nightly bundle:
|
||||
- `docs/ADDRESS/runs/2026-04-03_Address_Nightly_Regression_Post_AnchorHardening_R6/`
|
||||
|
||||
Result:
|
||||
- `overall_ok=true`
|
||||
- `stress_102`: `102/102` (`strict_pass_rate=1.0`)
|
||||
- `followup_25`: `25/25` (`strict_pass_rate=1.0`)
|
||||
- comparator: `PASS` по обоим pack.
|
||||
|
||||
Comparator detail:
|
||||
- baseline comparator закрыт без деградации (`strict_pass_rate`/`route_pass_rate` сохранены).
|
||||
|
||||
## 4. Вывод
|
||||
|
||||
1. Domain live-gates для Batch-1 и Batch-3 закрыты.
|
||||
2. Глобальный comparator закрыт в PASS, блокер прод-промоушена снят.
|
||||
3. Следующий шаг: Batch-2 lifecycle pack по рельсовой схеме (design -> prep -> live gate -> global regression).
|
||||
|
||||
## 5. Operational note (2026-04-08)
|
||||
|
||||
- Автозапуск nightly от `2026-04-08 09:52:38` (Task Scheduler) дал `overall_ok=false` из-за инфраструктурного класса ошибок (`MCP fetch failed: fetch failed`) в части вызовов.
|
||||
- Это operational-инцидент окружения, а не функциональный откат закрытых Batch-1/Batch-3 intent-ов.
|
||||
- Ручной контрольный nightly recheck от `2026-04-08 10:51` закрыт в `PASS` (`stress_102=102/102`, `followup_25=25/25`, comparator PASS): `docs/ADDRESS/runs/2026-04-08_Address_Nightly_Regression_2026-04-08_10-51-20/nightly_summary.json`.
|
||||
|
|
@ -0,0 +1,33 @@
|
|||
# 2026-04-03_Address_Domain_General_B1_PhaseC_LiveGate
|
||||
|
||||
Generated at: 2026-04-03T08:48:23
|
||||
Questions file: X:\1C\NDC_1C\docs\ADDRESS\question_sets\domain_general_batch1_acceptance_2026-04-02_phaseA.json
|
||||
Backend URL: http://127.0.0.1:8787/api/assistant/message
|
||||
LLM: local / qwen2.5-14b-instruct-1m @ http://127.0.0.1:1234
|
||||
Strict policy: route
|
||||
|
||||
## Totals
|
||||
- questions_total: 28
|
||||
- ok_200_count: 28
|
||||
- semantic_pass_count: 21
|
||||
- semantic_pass_rate: 0.75
|
||||
- route_pass_count: 21
|
||||
- route_pass_rate: 0.75
|
||||
- strict_pass_count: 21
|
||||
- strict_pass_rate: 0.75
|
||||
- factual_count: 21
|
||||
- partial_coverage_count: 6
|
||||
- clarification_required_count: 1
|
||||
- http_error_count: 0
|
||||
- llm_decomposition_attempted_count: 28
|
||||
- llm_decomposition_applied_count: 14
|
||||
- llm_fallback_count: 0
|
||||
- llm_fallback_rate: 0.0
|
||||
- tool_gate_blocked_count: 0
|
||||
- tool_gate_blocked_rate: 0.0
|
||||
- avg_elapsed_ms: 6238.9
|
||||
|
||||
## Files
|
||||
- run_summary.json
|
||||
- full_live_results.json
|
||||
- failures_only.json
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
|
|
@ -0,0 +1,32 @@
|
|||
# Response Audit: 2026-04-03_Address_Domain_General_B1_PhaseC_LiveGate
|
||||
|
||||
| id | strict | route_health | reply_type | intent | limited_reason | question | assistant_first_line |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| B1_C001 | True | ok_or_factual | factual | period_coverage_profile | None | За какие годы в базе есть данные? | Профиль периодов базы собран (movement-based aggregate). |
|
||||
| B1_C002 | True | ok_or_factual | factual | period_coverage_profile | None | Какой год самый активный по количеству документов? | Профиль периодов базы собран (movement-based aggregate). |
|
||||
| B1_C003 | True | ok_or_factual | factual | period_coverage_profile | None | Какой месяц самый активный по количеству операций? | Профиль периодов базы собран (movement-based aggregate). |
|
||||
| B1_C004 | True | ok_or_factual | factual | document_type_and_account_section_profile | None | Какие типы документов используются чаще всего в базе? | Профиль типов документов и разделов учета собран (movement-based aggregate). |
|
||||
| B1_C005 | True | ok_or_factual | factual | document_type_and_account_section_profile | None | Какие разделы учета наиболее заполнены и какие почти не используются? | Профиль типов документов и разделов учета собран (movement-based aggregate). |
|
||||
| B1_C006 | True | ok_or_factual | factual | counterparty_population_and_roles | None | Сколько всего уникальных контрагентов в базе? | Профиль контрагентов собран (catalog + bank-doc activity aggregate). |
|
||||
| B1_C007 | True | ok_or_factual | factual | counterparty_population_and_roles | None | Сколько у нас заказчиков, поставщиков и смешанных контрагентов? | Профиль контрагентов собран (catalog + bank-doc activity aggregate). |
|
||||
| B1_C008 | True | ok_or_factual | factual | contract_usage_overview | None | Сколько всего договоров заведено и сколько из них реально использовались? | Профиль договорной базы собран (catalog + usage aggregate). |
|
||||
| B1_C009 | True | ok_or_factual | factual | period_coverage_profile | None | Покажи профиль данных по базе: диапазон лет, топ год и топ месяц активности. | Профиль периодов базы собран (movement-based aggregate). |
|
||||
| B1_C010 | False | likely_blocked_route | partial_coverage | list_documents_by_counterparty | empty_match | Сформируй сводку по типам документов и их доле в общем объеме. | В live-данных по текущему фильтру записи не найдены. |
|
||||
| B1_C011 | True | ok_or_factual | factual | counterparty_population_and_roles | None | Сколько контрагентов активны как клиенты и как поставщики? | Профиль контрагентов собран (catalog + bank-doc activity aggregate). |
|
||||
| B1_C012 | True | ok_or_factual | factual | contract_usage_overview | None | Дай обзор договорной базы: total договоров и used договоров. | Профиль договорной базы собран (catalog + usage aggregate). |
|
||||
| B1_N001 | False | blocked_clarification | clarification_required | None | None | за какие года база ваще живая? | Коротко: Наиболее вероятный механизм проблемы подтвержден частично и требует первичной проверки. Что сломано: |
|
||||
| B1_N002 | False | likely_blocked_route | partial_coverage | list_documents_by_counterparty | empty_match | какой год тут самый движовый по докам | В live-данных по текущему фильтру записи не найдены. |
|
||||
| B1_N003 | False | blocked_missing_anchor | partial_coverage | bank_operations_by_counterparty | missing_anchor | какой месяц прям пик по операциям? | Для точного адресного поиска не хватает обязательного якоря. |
|
||||
| B1_N004 | False | blocked_missing_anchor | partial_coverage | bank_operations_by_counterparty | missing_anchor | каких доков у нас больше всего крутится? | Для точного адресного поиска не хватает обязательного якоря. |
|
||||
| B1_N005 | True | ok_or_factual | factual | document_type_and_account_section_profile | None | какие разделы учета жирные, а какие мертвые? | Профиль типов документов и разделов учета собран (movement-based aggregate). |
|
||||
| B1_N006 | True | ok_or_factual | factual | counterparty_population_and_roles | None | скок уник контрагентов вообще есть | Профиль контрагентов собран (catalog + bank-doc activity aggregate). |
|
||||
| B1_N007 | True | ok_or_factual | factual | counterparty_population_and_roles | None | разбей контров на клиентов/поставщиков/смешанных | Профиль контрагентов собран (catalog + bank-doc activity aggregate). |
|
||||
| B1_N008 | True | ok_or_factual | factual | contract_usage_overview | None | договоров сколько всего и сколько реально юзались? | Профиль договорной базы собран (catalog + usage aggregate). |
|
||||
| B1_N009 | False | blocked_unsupported | partial_coverage | unknown | unsupported | покажи по базе years/top year/top month, только без воды | Этот запрос не подходит под address_query V1. |
|
||||
| B1_N010 | True | ok_or_factual | factual | contract_usage_overview | None | чо по договорной базе: total vs used | Профиль договорной базы собран (catalog + usage aggregate). |
|
||||
| B1_F001 | True | ok_or_factual | factual | period_coverage_profile | None | За какие годы в базе есть данные? | Профиль периодов базы собран (movement-based aggregate). |
|
||||
| B1_F002 | True | ok_or_factual | factual | period_coverage_profile | None | а какой год самый активный по документам? | Профиль периодов базы собран (movement-based aggregate). |
|
||||
| B1_F003 | False | blocked_missing_anchor | partial_coverage | bank_operations_by_counterparty | missing_anchor | а теперь месяц-пик по операциям | Для точного адресного поиска не хватает обязательного якоря. |
|
||||
| B1_F004 | True | ok_or_factual | factual | counterparty_population_and_roles | None | Сколько всего уникальных контрагентов в базе? | Профиль контрагентов собран (catalog + bank-doc activity aggregate). |
|
||||
| B1_F005 | True | ok_or_factual | factual | counterparty_population_and_roles | None | а теперь разбей их на заказчиков и поставщиков | Профиль контрагентов собран (catalog + bank-doc activity aggregate). |
|
||||
| B1_F006 | True | ok_or_factual | factual | contract_usage_overview | None | и сразу сколько всего договоров и сколько used | Профиль договорной базы собран (catalog + usage aggregate). |
|
||||
|
|
@ -0,0 +1,152 @@
|
|||
{
|
||||
"run_id": "2026-04-03_Address_Domain_General_B1_PhaseC_LiveGate",
|
||||
"generated_at": "2026-04-03T08:48:23",
|
||||
"source_questions_file": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\question_sets\\domain_general_batch1_acceptance_2026-04-02_phaseA.json",
|
||||
"backend_url": "http://127.0.0.1:8787/api/assistant/message",
|
||||
"llm_provider": "local",
|
||||
"llm_model": "qwen2.5-14b-instruct-1m",
|
||||
"llm_base_url": "http://127.0.0.1:1234",
|
||||
"strict_policy": "route",
|
||||
"totals": {
|
||||
"questions_total": 28,
|
||||
"ok_200_count": 28,
|
||||
"semantic_pass_count": 21,
|
||||
"semantic_pass_rate": 0.75,
|
||||
"route_pass_count": 21,
|
||||
"route_pass_rate": 0.75,
|
||||
"strict_pass_count": 21,
|
||||
"strict_pass_rate": 0.75,
|
||||
"factual_count": 21,
|
||||
"partial_coverage_count": 6,
|
||||
"clarification_required_count": 1,
|
||||
"http_error_count": 0,
|
||||
"llm_decomposition_attempted_count": 28,
|
||||
"llm_decomposition_applied_count": 14,
|
||||
"llm_fallback_count": 0,
|
||||
"llm_fallback_rate": 0.0,
|
||||
"tool_gate_blocked_count": 0,
|
||||
"tool_gate_blocked_rate": 0.0,
|
||||
"avg_elapsed_ms": 6238.9
|
||||
},
|
||||
"distributions": {
|
||||
"reply_type": {
|
||||
"factual": 21,
|
||||
"partial_coverage": 6,
|
||||
"clarification_required": 1
|
||||
},
|
||||
"actual_intent": {
|
||||
"period_coverage_profile": 6,
|
||||
"document_type_and_account_section_profile": 3,
|
||||
"counterparty_population_and_roles": 7,
|
||||
"contract_usage_overview": 5,
|
||||
"list_documents_by_counterparty": 2,
|
||||
"None": 1,
|
||||
"bank_operations_by_counterparty": 3,
|
||||
"unknown": 1
|
||||
},
|
||||
"actual_mode": {
|
||||
"address_query": 27,
|
||||
"None": 1
|
||||
},
|
||||
"mcp_call_status": {
|
||||
"matched_non_empty": 21,
|
||||
"materialized_but_not_anchor_matched": 3,
|
||||
"None": 1,
|
||||
"skipped": 3
|
||||
},
|
||||
"limited_reason_category": {
|
||||
"empty_match": 2,
|
||||
"missing_anchor": 3,
|
||||
"unsupported": 1
|
||||
},
|
||||
"route_health": {
|
||||
"ok_or_factual": 21,
|
||||
"likely_blocked_route": 2,
|
||||
"blocked_clarification": 1,
|
||||
"blocked_missing_anchor": 3,
|
||||
"blocked_unsupported": 1
|
||||
},
|
||||
"tool_gate_decision": {
|
||||
"run_address_lane": 28
|
||||
},
|
||||
"tool_gate_reason": {
|
||||
"address_mode_classifier_detected": 27,
|
||||
"llm_canonical_candidate_detected": 1
|
||||
}
|
||||
},
|
||||
"address_llm_predecompose_metrics": {
|
||||
"overall": {
|
||||
"llm_attempted": 28,
|
||||
"llm_applied": 14,
|
||||
"fallback_used": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"tool_gate_blocked": 0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"by_intent": {
|
||||
"period_coverage_profile": {
|
||||
"total": 6,
|
||||
"llm_attempted": 6,
|
||||
"llm_applied": 1,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"document_type_and_account_section_profile": {
|
||||
"total": 3,
|
||||
"llm_attempted": 3,
|
||||
"llm_applied": 1,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"counterparty_population_and_roles": {
|
||||
"total": 7,
|
||||
"llm_attempted": 7,
|
||||
"llm_applied": 3,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"contract_usage_overview": {
|
||||
"total": 5,
|
||||
"llm_attempted": 5,
|
||||
"llm_applied": 3,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"list_documents_by_counterparty": {
|
||||
"total": 2,
|
||||
"llm_attempted": 2,
|
||||
"llm_applied": 2,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"unknown": {
|
||||
"total": 2,
|
||||
"llm_attempted": 2,
|
||||
"llm_applied": 1,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"bank_operations_by_counterparty": {
|
||||
"total": 3,
|
||||
"llm_attempted": 3,
|
||||
"llm_applied": 3,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,33 @@
|
|||
# 2026-04-03_Address_Domain_General_B1_PhaseC_LiveGate_R2
|
||||
|
||||
Generated at: 2026-04-03T09:01:37
|
||||
Questions file: X:\1C\NDC_1C\docs\ADDRESS\question_sets\domain_general_batch1_acceptance_2026-04-02_phaseA.json
|
||||
Backend URL: http://127.0.0.1:8787/api/assistant/message
|
||||
LLM: local / qwen2.5-14b-instruct-1m @ http://127.0.0.1:1234
|
||||
Strict policy: route
|
||||
|
||||
## Totals
|
||||
- questions_total: 28
|
||||
- ok_200_count: 28
|
||||
- semantic_pass_count: 27
|
||||
- semantic_pass_rate: 0.9643
|
||||
- route_pass_count: 27
|
||||
- route_pass_rate: 0.9643
|
||||
- strict_pass_count: 27
|
||||
- strict_pass_rate: 0.9643
|
||||
- factual_count: 27
|
||||
- partial_coverage_count: 1
|
||||
- clarification_required_count: 0
|
||||
- http_error_count: 0
|
||||
- llm_decomposition_attempted_count: 28
|
||||
- llm_decomposition_applied_count: 10
|
||||
- llm_fallback_count: 0
|
||||
- llm_fallback_rate: 0.0
|
||||
- tool_gate_blocked_count: 0
|
||||
- tool_gate_blocked_rate: 0.0
|
||||
- avg_elapsed_ms: 5904.4
|
||||
|
||||
## Files
|
||||
- run_summary.json
|
||||
- full_live_results.json
|
||||
- failures_only.json
|
||||
|
|
@ -0,0 +1,185 @@
|
|||
[
|
||||
{
|
||||
"index": 10,
|
||||
"id": "B1_C010",
|
||||
"question": "Сформируй сводку по типам документов и их доле в общем объеме.",
|
||||
"session": null,
|
||||
"session_id": "asst-2026-04-03_Address_Domain_General_B1_PhaseC_LiveGate_R2-b1_c010",
|
||||
"status_code": 200,
|
||||
"ok": true,
|
||||
"elapsed_ms": 9458,
|
||||
"reply_type": "partial_coverage",
|
||||
"trace_id": "address-cDYRKzJtv2",
|
||||
"assistant_reply": "В live-данных по текущему фильтру записи не найдены.\nПричина: по указанному якорю и фильтрам в live-выборке нет строк.\nЧто нужно уточнить: уточните период или снимите часть фильтров.",
|
||||
"assistant_reply_first_line": "В live-данных по текущему фильтру записи не найдены.",
|
||||
"expected_intent": "document_type_and_account_section_profile",
|
||||
"actual_intent": "list_documents_by_counterparty",
|
||||
"intent_match": false,
|
||||
"expected_mode": "address_query",
|
||||
"actual_mode": "address_query",
|
||||
"mode_match": true,
|
||||
"expected_reply_type": null,
|
||||
"reply_match": true,
|
||||
"semantic_pass": false,
|
||||
"route_pass": false,
|
||||
"route_health": "likely_blocked_route",
|
||||
"strict_policy": "route",
|
||||
"strict_pass": false,
|
||||
"selected_recipe": "address_documents_by_counterparty_v1",
|
||||
"missing_required_filters": [],
|
||||
"match_failure_stage": "materialized_but_not_anchor_matched",
|
||||
"match_failure_reason": "counterparty_anchor_not_matched_in_materialized_rows",
|
||||
"rows_fetched": 1000,
|
||||
"rows_matched": 0,
|
||||
"mcp_call_status": "materialized_but_not_anchor_matched",
|
||||
"limited_reason_category": "empty_match",
|
||||
"llm_decomposition_attempted": true,
|
||||
"llm_decomposition_applied": true,
|
||||
"llm_decomposition_reason": "normalized_fragment_applied",
|
||||
"llm_canonical_candidate_detected": true,
|
||||
"fallback_rule_hit": null,
|
||||
"tool_gate_decision": "run_address_lane",
|
||||
"tool_gate_reason": "address_mode_classifier_detected",
|
||||
"predecompose_contract_intent": "list_documents_by_counterparty",
|
||||
"predecompose_contract_aggregation_profile": "list_lookup",
|
||||
"predecompose_contract_period_scope": "unspecified",
|
||||
"debug_payload": {
|
||||
"trace_id": "address-cDYRKzJtv2",
|
||||
"prompt_version": "address_query_runtime_v1",
|
||||
"schema_version": "address_query_runtime_v1",
|
||||
"fallback_type": "partial",
|
||||
"route_summary": null,
|
||||
"fragments": [],
|
||||
"requirements_extracted": [],
|
||||
"coverage_report": {
|
||||
"requirements_total": 0,
|
||||
"requirements_covered": 0,
|
||||
"requirements_uncovered": [],
|
||||
"requirements_partially_covered": [],
|
||||
"clarification_needed_for": [],
|
||||
"out_of_scope_requirements": []
|
||||
},
|
||||
"routes": [],
|
||||
"retrieval_status": [],
|
||||
"retrieval_results": [],
|
||||
"answer_grounding_check": {
|
||||
"status": "partial",
|
||||
"route_subject_match": true,
|
||||
"missing_requirements": [],
|
||||
"reasons": [
|
||||
"loose_by_anchor_detected",
|
||||
"document_list_signal_detected",
|
||||
"documents_by_counterparty_signal_detected"
|
||||
],
|
||||
"why_included_summary": [],
|
||||
"selection_reason_summary": []
|
||||
},
|
||||
"dropped_intent_segments": [],
|
||||
"detected_mode": "address_query",
|
||||
"detected_mode_confidence": "medium",
|
||||
"query_shape": "DOCUMENT_LIST",
|
||||
"query_shape_confidence": "medium",
|
||||
"detected_intent": "list_documents_by_counterparty",
|
||||
"detected_intent_confidence": "medium",
|
||||
"extracted_filters": {
|
||||
"sort": "period_desc",
|
||||
"limit": 20,
|
||||
"counterparty": "типам"
|
||||
},
|
||||
"missing_required_filters": [],
|
||||
"selected_recipe": "address_documents_by_counterparty_v1",
|
||||
"mcp_call_status_legacy": "materialized_but_not_matched",
|
||||
"account_scope_mode": "preferred",
|
||||
"account_scope_fallback_applied": false,
|
||||
"anchor_type": "counterparty",
|
||||
"anchor_value_raw": "типам",
|
||||
"anchor_value_resolved": "типам",
|
||||
"resolver_confidence": "medium",
|
||||
"ambiguity_count": 0,
|
||||
"match_failure_stage": "materialized_but_not_anchor_matched",
|
||||
"match_failure_reason": "counterparty_anchor_not_matched_in_materialized_rows",
|
||||
"mcp_call_status": "materialized_but_not_anchor_matched",
|
||||
"rows_fetched": 1000,
|
||||
"raw_rows_received": 1000,
|
||||
"rows_after_account_scope": 124,
|
||||
"rows_after_recipe_filter": 0,
|
||||
"rows_materialized": 124,
|
||||
"rows_matched": 0,
|
||||
"raw_row_keys_sample": [
|
||||
"Период",
|
||||
"Регистратор",
|
||||
"СчетДт",
|
||||
"СчетКт",
|
||||
"Сумма",
|
||||
"Контрагент",
|
||||
"Period",
|
||||
"Registrator",
|
||||
"Amount"
|
||||
],
|
||||
"materialization_drop_reason": "none",
|
||||
"account_token_raw": null,
|
||||
"account_token_normalized": null,
|
||||
"account_scope_fields_checked": [
|
||||
"account_dt",
|
||||
"account_kt",
|
||||
"registrator",
|
||||
"analytics"
|
||||
],
|
||||
"account_scope_match_strategy": "account_code_regex_plus_alias_map_v1",
|
||||
"account_scope_drop_reason": "not_applicable",
|
||||
"runtime_readiness": "LIVE_QUERYABLE_WITH_LIMITS",
|
||||
"limited_reason_category": "empty_match",
|
||||
"response_type": "LIMITED_WITH_REASON",
|
||||
"execution_lane": "address_query",
|
||||
"llm_decomposition_applied": true,
|
||||
"llm_decomposition_attempted": true,
|
||||
"llm_provider_used": "local",
|
||||
"llm_decomposition_trace_id": "cUU7oaPgWpZjwv",
|
||||
"llm_decomposition_effective_message": "Сформировать сводку по типам документов и их доле в общем объеме.",
|
||||
"llm_decomposition_reason": "normalized_fragment_applied",
|
||||
"llm_canonical_candidate_detected": true,
|
||||
"llm_predecompose_contract": {
|
||||
"schema_version": "address_llm_predecompose_contract_v1",
|
||||
"source_message": "Сформируй сводку по типам документов и их доле в общем объеме.",
|
||||
"canonical_message": "Сформировать сводку по типам документов и их доле в общем объеме.",
|
||||
"mode": "address_query",
|
||||
"mode_confidence": "medium",
|
||||
"query_shape": "DOCUMENT_LIST",
|
||||
"query_shape_confidence": "medium",
|
||||
"intent": "list_documents_by_counterparty",
|
||||
"intent_confidence": "medium",
|
||||
"entities": {
|
||||
"account": null,
|
||||
"counterparty": "типам",
|
||||
"contract": null,
|
||||
"document_type": null,
|
||||
"document_ref": null,
|
||||
"organization": null
|
||||
},
|
||||
"period": {
|
||||
"scope": "unspecified",
|
||||
"period_from": null,
|
||||
"period_to": null,
|
||||
"as_of_date": null,
|
||||
"has_explicit_period": false
|
||||
},
|
||||
"aggregation_profile": "list_lookup"
|
||||
},
|
||||
"fallback_rule_hit": null,
|
||||
"sanitized_user_message": "сформируй сводку по типам документов и их доле в общем объеме.",
|
||||
"tool_gate_decision": "run_address_lane",
|
||||
"tool_gate_reason": "address_mode_classifier_detected",
|
||||
"answer_structure_v11": null,
|
||||
"investigation_state_snapshot": null,
|
||||
"normalized": null,
|
||||
"normalizer_output": {
|
||||
"trace_id": "cUU7oaPgWpZjwv",
|
||||
"prompt_version": "normalizer_v2_0_2",
|
||||
"applied": true,
|
||||
"effective_message": "Сформировать сводку по типам документов и их доле в общем объеме."
|
||||
}
|
||||
},
|
||||
"error_code": null,
|
||||
"error_message": null
|
||||
}
|
||||
]
|
||||
File diff suppressed because it is too large
Load Diff
|
|
@ -0,0 +1,32 @@
|
|||
# Response Audit: 2026-04-03_Address_Domain_General_B1_PhaseC_LiveGate_R2
|
||||
|
||||
| id | strict | route_health | reply_type | intent | limited_reason | question | assistant_first_line |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| B1_C001 | True | ok_or_factual | factual | period_coverage_profile | None | За какие годы в базе есть данные? | Профиль периодов базы собран (movement-based aggregate). |
|
||||
| B1_C002 | True | ok_or_factual | factual | period_coverage_profile | None | Какой год самый активный по количеству документов? | Профиль периодов базы собран (movement-based aggregate). |
|
||||
| B1_C003 | True | ok_or_factual | factual | period_coverage_profile | None | Какой месяц самый активный по количеству операций? | Профиль периодов базы собран (movement-based aggregate). |
|
||||
| B1_C004 | True | ok_or_factual | factual | document_type_and_account_section_profile | None | Какие типы документов используются чаще всего в базе? | Профиль типов документов и разделов учета собран (movement-based aggregate). |
|
||||
| B1_C005 | True | ok_or_factual | factual | document_type_and_account_section_profile | None | Какие разделы учета наиболее заполнены и какие почти не используются? | Профиль типов документов и разделов учета собран (movement-based aggregate). |
|
||||
| B1_C006 | True | ok_or_factual | factual | counterparty_population_and_roles | None | Сколько всего уникальных контрагентов в базе? | Профиль контрагентов собран (catalog + bank-doc activity aggregate). |
|
||||
| B1_C007 | True | ok_or_factual | factual | counterparty_population_and_roles | None | Сколько у нас заказчиков, поставщиков и смешанных контрагентов? | Профиль контрагентов собран (catalog + bank-doc activity aggregate). |
|
||||
| B1_C008 | True | ok_or_factual | factual | contract_usage_overview | None | Сколько всего договоров заведено и сколько из них реально использовались? | Профиль договорной базы собран (catalog + usage aggregate). |
|
||||
| B1_C009 | True | ok_or_factual | factual | period_coverage_profile | None | Покажи профиль данных по базе: диапазон лет, топ год и топ месяц активности. | Профиль периодов базы собран (movement-based aggregate). |
|
||||
| B1_C010 | False | likely_blocked_route | partial_coverage | list_documents_by_counterparty | empty_match | Сформируй сводку по типам документов и их доле в общем объеме. | В live-данных по текущему фильтру записи не найдены. |
|
||||
| B1_C011 | True | ok_or_factual | factual | counterparty_population_and_roles | None | Сколько контрагентов активны как клиенты и как поставщики? | Профиль контрагентов собран (catalog + bank-doc activity aggregate). |
|
||||
| B1_C012 | True | ok_or_factual | factual | contract_usage_overview | None | Дай обзор договорной базы: total договоров и used договоров. | Профиль договорной базы собран (catalog + usage aggregate). |
|
||||
| B1_N001 | True | ok_or_factual | factual | period_coverage_profile | None | за какие года база ваще живая? | Профиль периодов базы собран (movement-based aggregate). |
|
||||
| B1_N002 | True | ok_or_factual | factual | period_coverage_profile | None | какой год тут самый движовый по докам | Профиль периодов базы собран (movement-based aggregate). |
|
||||
| B1_N003 | True | ok_or_factual | factual | period_coverage_profile | None | какой месяц прям пик по операциям? | Профиль периодов базы собран (movement-based aggregate). |
|
||||
| B1_N004 | True | ok_or_factual | factual | document_type_and_account_section_profile | None | каких доков у нас больше всего крутится? | Профиль типов документов и разделов учета собран (movement-based aggregate). |
|
||||
| B1_N005 | True | ok_or_factual | factual | document_type_and_account_section_profile | None | какие разделы учета жирные, а какие мертвые? | Профиль типов документов и разделов учета собран (movement-based aggregate). |
|
||||
| B1_N006 | True | ok_or_factual | factual | counterparty_population_and_roles | None | скок уник контрагентов вообще есть | Профиль контрагентов собран (catalog + bank-doc activity aggregate). |
|
||||
| B1_N007 | True | ok_or_factual | factual | counterparty_population_and_roles | None | разбей контров на клиентов/поставщиков/смешанных | Профиль контрагентов собран (catalog + bank-doc activity aggregate). |
|
||||
| B1_N008 | True | ok_or_factual | factual | contract_usage_overview | None | договоров сколько всего и сколько реально юзались? | Профиль договорной базы собран (catalog + usage aggregate). |
|
||||
| B1_N009 | True | ok_or_factual | factual | period_coverage_profile | None | покажи по базе years/top year/top month, только без воды | Профиль периодов базы собран (movement-based aggregate). |
|
||||
| B1_N010 | True | ok_or_factual | factual | contract_usage_overview | None | чо по договорной базе: total vs used | Профиль договорной базы собран (catalog + usage aggregate). |
|
||||
| B1_F001 | True | ok_or_factual | factual | period_coverage_profile | None | За какие годы в базе есть данные? | Профиль периодов базы собран (movement-based aggregate). |
|
||||
| B1_F002 | True | ok_or_factual | factual | period_coverage_profile | None | а какой год самый активный по документам? | Профиль периодов базы собран (movement-based aggregate). |
|
||||
| B1_F003 | True | ok_or_factual | factual | period_coverage_profile | None | а теперь месяц-пик по операциям | Профиль периодов базы собран (movement-based aggregate). |
|
||||
| B1_F004 | True | ok_or_factual | factual | counterparty_population_and_roles | None | Сколько всего уникальных контрагентов в базе? | Профиль контрагентов собран (catalog + bank-doc activity aggregate). |
|
||||
| B1_F005 | True | ok_or_factual | factual | counterparty_population_and_roles | None | а теперь разбей их на заказчиков и поставщиков | Профиль контрагентов собран (catalog + bank-doc activity aggregate). |
|
||||
| B1_F006 | True | ok_or_factual | factual | contract_usage_overview | None | и сразу сколько всего договоров и сколько used | Профиль договорной базы собран (catalog + usage aggregate). |
|
||||
|
|
@ -0,0 +1,121 @@
|
|||
{
|
||||
"run_id": "2026-04-03_Address_Domain_General_B1_PhaseC_LiveGate_R2",
|
||||
"generated_at": "2026-04-03T09:01:37",
|
||||
"source_questions_file": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\question_sets\\domain_general_batch1_acceptance_2026-04-02_phaseA.json",
|
||||
"backend_url": "http://127.0.0.1:8787/api/assistant/message",
|
||||
"llm_provider": "local",
|
||||
"llm_model": "qwen2.5-14b-instruct-1m",
|
||||
"llm_base_url": "http://127.0.0.1:1234",
|
||||
"strict_policy": "route",
|
||||
"totals": {
|
||||
"questions_total": 28,
|
||||
"ok_200_count": 28,
|
||||
"semantic_pass_count": 27,
|
||||
"semantic_pass_rate": 0.9643,
|
||||
"route_pass_count": 27,
|
||||
"route_pass_rate": 0.9643,
|
||||
"strict_pass_count": 27,
|
||||
"strict_pass_rate": 0.9643,
|
||||
"factual_count": 27,
|
||||
"partial_coverage_count": 1,
|
||||
"clarification_required_count": 0,
|
||||
"http_error_count": 0,
|
||||
"llm_decomposition_attempted_count": 28,
|
||||
"llm_decomposition_applied_count": 10,
|
||||
"llm_fallback_count": 0,
|
||||
"llm_fallback_rate": 0.0,
|
||||
"tool_gate_blocked_count": 0,
|
||||
"tool_gate_blocked_rate": 0.0,
|
||||
"avg_elapsed_ms": 5904.4
|
||||
},
|
||||
"distributions": {
|
||||
"reply_type": {
|
||||
"factual": 27,
|
||||
"partial_coverage": 1
|
||||
},
|
||||
"actual_intent": {
|
||||
"period_coverage_profile": 11,
|
||||
"document_type_and_account_section_profile": 4,
|
||||
"counterparty_population_and_roles": 7,
|
||||
"contract_usage_overview": 5,
|
||||
"list_documents_by_counterparty": 1
|
||||
},
|
||||
"actual_mode": {
|
||||
"address_query": 28
|
||||
},
|
||||
"mcp_call_status": {
|
||||
"matched_non_empty": 27,
|
||||
"materialized_but_not_anchor_matched": 1
|
||||
},
|
||||
"limited_reason_category": {
|
||||
"empty_match": 1
|
||||
},
|
||||
"route_health": {
|
||||
"ok_or_factual": 27,
|
||||
"likely_blocked_route": 1
|
||||
},
|
||||
"tool_gate_decision": {
|
||||
"run_address_lane": 28
|
||||
},
|
||||
"tool_gate_reason": {
|
||||
"address_mode_classifier_detected": 28
|
||||
}
|
||||
},
|
||||
"address_llm_predecompose_metrics": {
|
||||
"overall": {
|
||||
"llm_attempted": 28,
|
||||
"llm_applied": 10,
|
||||
"fallback_used": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"tool_gate_blocked": 0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"by_intent": {
|
||||
"period_coverage_profile": {
|
||||
"total": 11,
|
||||
"llm_attempted": 11,
|
||||
"llm_applied": 2,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"document_type_and_account_section_profile": {
|
||||
"total": 4,
|
||||
"llm_attempted": 4,
|
||||
"llm_applied": 1,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"counterparty_population_and_roles": {
|
||||
"total": 7,
|
||||
"llm_attempted": 7,
|
||||
"llm_applied": 3,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"contract_usage_overview": {
|
||||
"total": 5,
|
||||
"llm_attempted": 5,
|
||||
"llm_applied": 3,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"list_documents_by_counterparty": {
|
||||
"total": 1,
|
||||
"llm_attempted": 1,
|
||||
"llm_applied": 1,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,33 @@
|
|||
# 2026-04-03_Address_Domain_General_B1_PhaseC_LiveGate_R3
|
||||
|
||||
Generated at: 2026-04-03T09:05:41
|
||||
Questions file: X:\1C\NDC_1C\docs\ADDRESS\question_sets\domain_general_batch1_acceptance_2026-04-02_phaseA.json
|
||||
Backend URL: http://127.0.0.1:8787/api/assistant/message
|
||||
LLM: local / qwen2.5-14b-instruct-1m @ http://127.0.0.1:1234
|
||||
Strict policy: route
|
||||
|
||||
## Totals
|
||||
- questions_total: 28
|
||||
- ok_200_count: 28
|
||||
- semantic_pass_count: 28
|
||||
- semantic_pass_rate: 1.0
|
||||
- route_pass_count: 28
|
||||
- route_pass_rate: 1.0
|
||||
- strict_pass_count: 28
|
||||
- strict_pass_rate: 1.0
|
||||
- factual_count: 28
|
||||
- partial_coverage_count: 0
|
||||
- clarification_required_count: 0
|
||||
- http_error_count: 0
|
||||
- llm_decomposition_attempted_count: 28
|
||||
- llm_decomposition_applied_count: 10
|
||||
- llm_fallback_count: 0
|
||||
- llm_fallback_rate: 0.0
|
||||
- tool_gate_blocked_count: 0
|
||||
- tool_gate_blocked_rate: 0.0
|
||||
- avg_elapsed_ms: 5800.7
|
||||
|
||||
## Files
|
||||
- run_summary.json
|
||||
- full_live_results.json
|
||||
- failures_only.json
|
||||
|
|
@ -0,0 +1 @@
|
|||
[]
|
||||
File diff suppressed because it is too large
Load Diff
|
|
@ -0,0 +1,32 @@
|
|||
# Response Audit: 2026-04-03_Address_Domain_General_B1_PhaseC_LiveGate_R3
|
||||
|
||||
| id | strict | route_health | reply_type | intent | limited_reason | question | assistant_first_line |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| B1_C001 | True | ok_or_factual | factual | period_coverage_profile | None | За какие годы в базе есть данные? | Профиль периодов базы собран (movement-based aggregate). |
|
||||
| B1_C002 | True | ok_or_factual | factual | period_coverage_profile | None | Какой год самый активный по количеству документов? | Профиль периодов базы собран (movement-based aggregate). |
|
||||
| B1_C003 | True | ok_or_factual | factual | period_coverage_profile | None | Какой месяц самый активный по количеству операций? | Профиль периодов базы собран (movement-based aggregate). |
|
||||
| B1_C004 | True | ok_or_factual | factual | document_type_and_account_section_profile | None | Какие типы документов используются чаще всего в базе? | Профиль типов документов и разделов учета собран (movement-based aggregate). |
|
||||
| B1_C005 | True | ok_or_factual | factual | document_type_and_account_section_profile | None | Какие разделы учета наиболее заполнены и какие почти не используются? | Профиль типов документов и разделов учета собран (movement-based aggregate). |
|
||||
| B1_C006 | True | ok_or_factual | factual | counterparty_population_and_roles | None | Сколько всего уникальных контрагентов в базе? | Профиль контрагентов собран (catalog + bank-doc activity aggregate). |
|
||||
| B1_C007 | True | ok_or_factual | factual | counterparty_population_and_roles | None | Сколько у нас заказчиков, поставщиков и смешанных контрагентов? | Профиль контрагентов собран (catalog + bank-doc activity aggregate). |
|
||||
| B1_C008 | True | ok_or_factual | factual | contract_usage_overview | None | Сколько всего договоров заведено и сколько из них реально использовались? | Профиль договорной базы собран (catalog + usage aggregate). |
|
||||
| B1_C009 | True | ok_or_factual | factual | period_coverage_profile | None | Покажи профиль данных по базе: диапазон лет, топ год и топ месяц активности. | Профиль периодов базы собран (movement-based aggregate). |
|
||||
| B1_C010 | True | ok_or_factual | factual | document_type_and_account_section_profile | None | Сформируй сводку по типам документов и их доле в общем объеме. | Профиль типов документов и разделов учета собран (movement-based aggregate). |
|
||||
| B1_C011 | True | ok_or_factual | factual | counterparty_population_and_roles | None | Сколько контрагентов активны как клиенты и как поставщики? | Профиль контрагентов собран (catalog + bank-doc activity aggregate). |
|
||||
| B1_C012 | True | ok_or_factual | factual | contract_usage_overview | None | Дай обзор договорной базы: total договоров и used договоров. | Профиль договорной базы собран (catalog + usage aggregate). |
|
||||
| B1_N001 | True | ok_or_factual | factual | period_coverage_profile | None | за какие года база ваще живая? | Профиль периодов базы собран (movement-based aggregate). |
|
||||
| B1_N002 | True | ok_or_factual | factual | period_coverage_profile | None | какой год тут самый движовый по докам | Профиль периодов базы собран (movement-based aggregate). |
|
||||
| B1_N003 | True | ok_or_factual | factual | period_coverage_profile | None | какой месяц прям пик по операциям? | Профиль периодов базы собран (movement-based aggregate). |
|
||||
| B1_N004 | True | ok_or_factual | factual | document_type_and_account_section_profile | None | каких доков у нас больше всего крутится? | Профиль типов документов и разделов учета собран (movement-based aggregate). |
|
||||
| B1_N005 | True | ok_or_factual | factual | document_type_and_account_section_profile | None | какие разделы учета жирные, а какие мертвые? | Профиль типов документов и разделов учета собран (movement-based aggregate). |
|
||||
| B1_N006 | True | ok_or_factual | factual | counterparty_population_and_roles | None | скок уник контрагентов вообще есть | Профиль контрагентов собран (catalog + bank-doc activity aggregate). |
|
||||
| B1_N007 | True | ok_or_factual | factual | counterparty_population_and_roles | None | разбей контров на клиентов/поставщиков/смешанных | Профиль контрагентов собран (catalog + bank-doc activity aggregate). |
|
||||
| B1_N008 | True | ok_or_factual | factual | contract_usage_overview | None | договоров сколько всего и сколько реально юзались? | Профиль договорной базы собран (catalog + usage aggregate). |
|
||||
| B1_N009 | True | ok_or_factual | factual | period_coverage_profile | None | покажи по базе years/top year/top month, только без воды | Профиль периодов базы собран (movement-based aggregate). |
|
||||
| B1_N010 | True | ok_or_factual | factual | contract_usage_overview | None | чо по договорной базе: total vs used | Профиль договорной базы собран (catalog + usage aggregate). |
|
||||
| B1_F001 | True | ok_or_factual | factual | period_coverage_profile | None | За какие годы в базе есть данные? | Профиль периодов базы собран (movement-based aggregate). |
|
||||
| B1_F002 | True | ok_or_factual | factual | period_coverage_profile | None | а какой год самый активный по документам? | Профиль периодов базы собран (movement-based aggregate). |
|
||||
| B1_F003 | True | ok_or_factual | factual | period_coverage_profile | None | а теперь месяц-пик по операциям | Профиль периодов базы собран (movement-based aggregate). |
|
||||
| B1_F004 | True | ok_or_factual | factual | counterparty_population_and_roles | None | Сколько всего уникальных контрагентов в базе? | Профиль контрагентов собран (catalog + bank-doc activity aggregate). |
|
||||
| B1_F005 | True | ok_or_factual | factual | counterparty_population_and_roles | None | а теперь разбей их на заказчиков и поставщиков | Профиль контрагентов собран (catalog + bank-doc activity aggregate). |
|
||||
| B1_F006 | True | ok_or_factual | factual | contract_usage_overview | None | и сразу сколько всего договоров и сколько used | Профиль договорной базы собран (catalog + usage aggregate). |
|
||||
|
|
@ -0,0 +1,106 @@
|
|||
{
|
||||
"run_id": "2026-04-03_Address_Domain_General_B1_PhaseC_LiveGate_R3",
|
||||
"generated_at": "2026-04-03T09:05:41",
|
||||
"source_questions_file": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\question_sets\\domain_general_batch1_acceptance_2026-04-02_phaseA.json",
|
||||
"backend_url": "http://127.0.0.1:8787/api/assistant/message",
|
||||
"llm_provider": "local",
|
||||
"llm_model": "qwen2.5-14b-instruct-1m",
|
||||
"llm_base_url": "http://127.0.0.1:1234",
|
||||
"strict_policy": "route",
|
||||
"totals": {
|
||||
"questions_total": 28,
|
||||
"ok_200_count": 28,
|
||||
"semantic_pass_count": 28,
|
||||
"semantic_pass_rate": 1.0,
|
||||
"route_pass_count": 28,
|
||||
"route_pass_rate": 1.0,
|
||||
"strict_pass_count": 28,
|
||||
"strict_pass_rate": 1.0,
|
||||
"factual_count": 28,
|
||||
"partial_coverage_count": 0,
|
||||
"clarification_required_count": 0,
|
||||
"http_error_count": 0,
|
||||
"llm_decomposition_attempted_count": 28,
|
||||
"llm_decomposition_applied_count": 10,
|
||||
"llm_fallback_count": 0,
|
||||
"llm_fallback_rate": 0.0,
|
||||
"tool_gate_blocked_count": 0,
|
||||
"tool_gate_blocked_rate": 0.0,
|
||||
"avg_elapsed_ms": 5800.7
|
||||
},
|
||||
"distributions": {
|
||||
"reply_type": {
|
||||
"factual": 28
|
||||
},
|
||||
"actual_intent": {
|
||||
"period_coverage_profile": 11,
|
||||
"document_type_and_account_section_profile": 5,
|
||||
"counterparty_population_and_roles": 7,
|
||||
"contract_usage_overview": 5
|
||||
},
|
||||
"actual_mode": {
|
||||
"address_query": 28
|
||||
},
|
||||
"mcp_call_status": {
|
||||
"matched_non_empty": 28
|
||||
},
|
||||
"limited_reason_category": {},
|
||||
"route_health": {
|
||||
"ok_or_factual": 28
|
||||
},
|
||||
"tool_gate_decision": {
|
||||
"run_address_lane": 28
|
||||
},
|
||||
"tool_gate_reason": {
|
||||
"address_mode_classifier_detected": 28
|
||||
}
|
||||
},
|
||||
"address_llm_predecompose_metrics": {
|
||||
"overall": {
|
||||
"llm_attempted": 28,
|
||||
"llm_applied": 10,
|
||||
"fallback_used": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"tool_gate_blocked": 0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"by_intent": {
|
||||
"period_coverage_profile": {
|
||||
"total": 11,
|
||||
"llm_attempted": 11,
|
||||
"llm_applied": 2,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"document_type_and_account_section_profile": {
|
||||
"total": 5,
|
||||
"llm_attempted": 5,
|
||||
"llm_applied": 2,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"counterparty_population_and_roles": {
|
||||
"total": 7,
|
||||
"llm_attempted": 7,
|
||||
"llm_applied": 3,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"contract_usage_overview": {
|
||||
"total": 5,
|
||||
"llm_attempted": 5,
|
||||
"llm_applied": 3,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,33 @@
|
|||
# 2026-04-03_Address_Domain_General_B3_Value_LiveGate
|
||||
|
||||
Generated at: 2026-04-03T08:52:00
|
||||
Questions file: X:\1C\NDC_1C\docs\ADDRESS\question_sets\temp_batch3_value_top20_2026-04-02.json
|
||||
Backend URL: http://127.0.0.1:8787/api/assistant/message
|
||||
LLM: local / qwen2.5-14b-instruct-1m @ http://127.0.0.1:1234
|
||||
Strict policy: route
|
||||
|
||||
## Totals
|
||||
- questions_total: 33
|
||||
- ok_200_count: 33
|
||||
- semantic_pass_count: 32
|
||||
- semantic_pass_rate: 0.9697
|
||||
- route_pass_count: 32
|
||||
- route_pass_rate: 0.9697
|
||||
- strict_pass_count: 32
|
||||
- strict_pass_rate: 0.9697
|
||||
- factual_count: 32
|
||||
- partial_coverage_count: 1
|
||||
- clarification_required_count: 0
|
||||
- http_error_count: 0
|
||||
- llm_decomposition_attempted_count: 33
|
||||
- llm_decomposition_applied_count: 30
|
||||
- llm_fallback_count: 0
|
||||
- llm_fallback_rate: 0.0
|
||||
- tool_gate_blocked_count: 0
|
||||
- tool_gate_blocked_rate: 0.0
|
||||
- avg_elapsed_ms: 6369.7
|
||||
|
||||
## Files
|
||||
- run_summary.json
|
||||
- full_live_results.json
|
||||
- failures_only.json
|
||||
|
|
@ -0,0 +1,175 @@
|
|||
[
|
||||
{
|
||||
"index": 19,
|
||||
"id": "B3_N006",
|
||||
"question": "покажи топ-20 самых маленьких сделок по бюджету",
|
||||
"session": null,
|
||||
"session_id": "asst-2026-04-03_Address_Domain_General_B3_Value_LiveGate-b3_n006",
|
||||
"status_code": 200,
|
||||
"ok": true,
|
||||
"elapsed_ms": 5744,
|
||||
"reply_type": "partial_coverage",
|
||||
"trace_id": "address-ZXnVIUlkaj",
|
||||
"assistant_reply": "Этот запрос не подходит под address_query V1.\nПричина: intent пока не поддержан в address V1.\nЧто нужно уточнить: переформулируйте вопрос как адресный lookup по счету/контрагенту/договору.",
|
||||
"assistant_reply_first_line": "Этот запрос не подходит под address_query V1.",
|
||||
"expected_intent": "customer_revenue_and_payments",
|
||||
"actual_intent": "unknown",
|
||||
"intent_match": false,
|
||||
"expected_mode": "address_query",
|
||||
"actual_mode": "address_query",
|
||||
"mode_match": true,
|
||||
"expected_reply_type": "factual",
|
||||
"reply_match": false,
|
||||
"semantic_pass": false,
|
||||
"route_pass": false,
|
||||
"route_health": "blocked_unsupported",
|
||||
"strict_policy": "route",
|
||||
"strict_pass": false,
|
||||
"selected_recipe": null,
|
||||
"missing_required_filters": [],
|
||||
"match_failure_stage": "none",
|
||||
"match_failure_reason": null,
|
||||
"rows_fetched": 0,
|
||||
"rows_matched": 0,
|
||||
"mcp_call_status": "skipped",
|
||||
"limited_reason_category": "unsupported",
|
||||
"llm_decomposition_attempted": true,
|
||||
"llm_decomposition_applied": true,
|
||||
"llm_decomposition_reason": "normalized_fragment_applied",
|
||||
"llm_canonical_candidate_detected": true,
|
||||
"fallback_rule_hit": null,
|
||||
"tool_gate_decision": "run_address_lane",
|
||||
"tool_gate_reason": "address_mode_classifier_detected",
|
||||
"predecompose_contract_intent": "unknown",
|
||||
"predecompose_contract_aggregation_profile": "unknown",
|
||||
"predecompose_contract_period_scope": "unspecified",
|
||||
"debug_payload": {
|
||||
"trace_id": "address-ZXnVIUlkaj",
|
||||
"prompt_version": "address_query_runtime_v1",
|
||||
"schema_version": "address_query_runtime_v1",
|
||||
"fallback_type": "partial",
|
||||
"route_summary": null,
|
||||
"fragments": [],
|
||||
"requirements_extracted": [],
|
||||
"coverage_report": {
|
||||
"requirements_total": 0,
|
||||
"requirements_covered": 0,
|
||||
"requirements_uncovered": [],
|
||||
"requirements_partially_covered": [],
|
||||
"clarification_needed_for": [],
|
||||
"out_of_scope_requirements": []
|
||||
},
|
||||
"routes": [],
|
||||
"retrieval_status": [],
|
||||
"retrieval_results": [],
|
||||
"answer_grounding_check": {
|
||||
"status": "partial",
|
||||
"route_subject_match": true,
|
||||
"missing_requirements": [],
|
||||
"reasons": [
|
||||
"address_action_detected",
|
||||
"address_entity_detected",
|
||||
"shape_not_detected",
|
||||
"intent_not_supported_in_v1"
|
||||
],
|
||||
"why_included_summary": [],
|
||||
"selection_reason_summary": []
|
||||
},
|
||||
"dropped_intent_segments": [],
|
||||
"detected_mode": "address_query",
|
||||
"detected_mode_confidence": "high",
|
||||
"query_shape": "UNKNOWN",
|
||||
"query_shape_confidence": "low",
|
||||
"detected_intent": "unknown",
|
||||
"detected_intent_confidence": "low",
|
||||
"extracted_filters": {
|
||||
"sort": "period_desc",
|
||||
"limit": 20
|
||||
},
|
||||
"missing_required_filters": [],
|
||||
"selected_recipe": null,
|
||||
"mcp_call_status_legacy": "skipped",
|
||||
"account_scope_mode": "strict",
|
||||
"account_scope_fallback_applied": false,
|
||||
"anchor_type": "unknown",
|
||||
"anchor_value_raw": null,
|
||||
"anchor_value_resolved": null,
|
||||
"resolver_confidence": "low",
|
||||
"ambiguity_count": 0,
|
||||
"match_failure_stage": "none",
|
||||
"match_failure_reason": null,
|
||||
"mcp_call_status": "skipped",
|
||||
"rows_fetched": 0,
|
||||
"raw_rows_received": 0,
|
||||
"rows_after_account_scope": 0,
|
||||
"rows_after_recipe_filter": 0,
|
||||
"rows_materialized": 0,
|
||||
"rows_matched": 0,
|
||||
"raw_row_keys_sample": [],
|
||||
"materialization_drop_reason": "none",
|
||||
"account_token_raw": null,
|
||||
"account_token_normalized": null,
|
||||
"account_scope_fields_checked": [
|
||||
"account_dt",
|
||||
"account_kt",
|
||||
"registrator",
|
||||
"analytics"
|
||||
],
|
||||
"account_scope_match_strategy": "account_code_regex_plus_alias_map_v1",
|
||||
"account_scope_drop_reason": "not_applicable",
|
||||
"runtime_readiness": "DEEP_ONLY",
|
||||
"limited_reason_category": "unsupported",
|
||||
"response_type": "LIMITED_WITH_REASON",
|
||||
"execution_lane": "address_query",
|
||||
"llm_decomposition_applied": true,
|
||||
"llm_decomposition_attempted": true,
|
||||
"llm_provider_used": "local",
|
||||
"llm_decomposition_trace_id": "PZdRqjSLwUz7JK",
|
||||
"llm_decomposition_effective_message": "показать топ-20 сделок с минимальными значениями в бюджете",
|
||||
"llm_decomposition_reason": "normalized_fragment_applied",
|
||||
"llm_canonical_candidate_detected": true,
|
||||
"llm_predecompose_contract": {
|
||||
"schema_version": "address_llm_predecompose_contract_v1",
|
||||
"source_message": "покажи топ-20 самых маленьких сделок по бюджету",
|
||||
"canonical_message": "показать топ-20 сделок с минимальными значениями в бюджете",
|
||||
"mode": "address_query",
|
||||
"mode_confidence": "high",
|
||||
"query_shape": "UNKNOWN",
|
||||
"query_shape_confidence": "low",
|
||||
"intent": "unknown",
|
||||
"intent_confidence": "low",
|
||||
"entities": {
|
||||
"account": null,
|
||||
"counterparty": null,
|
||||
"contract": null,
|
||||
"document_type": null,
|
||||
"document_ref": null,
|
||||
"organization": null
|
||||
},
|
||||
"period": {
|
||||
"scope": "unspecified",
|
||||
"period_from": null,
|
||||
"period_to": null,
|
||||
"as_of_date": null,
|
||||
"has_explicit_period": false
|
||||
},
|
||||
"aggregation_profile": "unknown"
|
||||
},
|
||||
"fallback_rule_hit": null,
|
||||
"sanitized_user_message": "покажи топ-20 самых маленьких сделок по бюджету",
|
||||
"tool_gate_decision": "run_address_lane",
|
||||
"tool_gate_reason": "address_mode_classifier_detected",
|
||||
"answer_structure_v11": null,
|
||||
"investigation_state_snapshot": null,
|
||||
"normalized": null,
|
||||
"normalizer_output": {
|
||||
"trace_id": "PZdRqjSLwUz7JK",
|
||||
"prompt_version": "normalizer_v2_0_2",
|
||||
"applied": true,
|
||||
"effective_message": "показать топ-20 сделок с минимальными значениями в бюджете"
|
||||
}
|
||||
},
|
||||
"error_code": null,
|
||||
"error_message": null
|
||||
}
|
||||
]
|
||||
File diff suppressed because it is too large
Load Diff
|
|
@ -0,0 +1,37 @@
|
|||
# Response Audit: 2026-04-03_Address_Domain_General_B3_Value_LiveGate
|
||||
|
||||
| id | strict | route_health | reply_type | intent | limited_reason | question | assistant_first_line |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| B3_C001 | True | ok_or_factual | factual | customer_revenue_and_payments | None | Покажи топ-20 заказчиков по сумме поступлений за все время. | Собран профиль поступлений от заказчиков (bank-doc value aggregate). |
|
||||
| B3_C002 | True | ok_or_factual | factual | customer_revenue_and_payments | None | Покажи топ-20 заказчиков по сумме поступлений за 2020 год. | Собран профиль поступлений от заказчиков (bank-doc value aggregate). |
|
||||
| B3_C003 | True | ok_or_factual | factual | customer_revenue_and_payments | None | Покажи топ-20 заказчиков по количеству входящих платежных операций за все время. | Собран профиль поступлений от заказчиков (bank-doc value aggregate). |
|
||||
| B3_C004 | True | ok_or_factual | factual | customer_revenue_and_payments | None | Покажи топ-20 заказчиков по максимальной сумме одной входящей операции за все время. | Собран профиль поступлений от заказчиков (bank-doc value aggregate). |
|
||||
| B3_C005 | True | ok_or_factual | factual | customer_revenue_and_payments | None | Покажи топ-20 заказчиков по среднему чеку среди активных клиентов (минимум 3 входящие операции). | Собран профиль поступлений от заказчиков (bank-doc value aggregate). |
|
||||
| B3_C006 | True | ok_or_factual | factual | customer_revenue_and_payments | None | Покажи топ-20 самых крупных разовых сделок по поступлениям (дата, контрагент, документ, сумма). | Собран профиль поступлений от заказчиков (bank-doc value aggregate). |
|
||||
| B3_C007 | True | ok_or_factual | factual | customer_revenue_and_payments | None | Покажи топ-20 самых маленьких разовых сделок по поступлениям среди активных заказчиков. | Собран профиль поступлений от заказчиков (bank-doc value aggregate). |
|
||||
| B3_C008 | True | ok_or_factual | factual | supplier_payouts_profile | None | Покажи топ-20 поставщиков по сумме выплат за все время. | Собран профиль выплат поставщикам (bank-doc value aggregate). |
|
||||
| B3_C009 | True | ok_or_factual | factual | supplier_payouts_profile | None | Покажи топ-20 поставщиков по сумме выплат за 2020 год. | Собран профиль выплат поставщикам (bank-doc value aggregate). |
|
||||
| B3_C010 | True | ok_or_factual | factual | supplier_payouts_profile | None | Покажи топ-20 поставщиков по количеству исходящих платежных операций за все время. | Собран профиль выплат поставщикам (bank-doc value aggregate). |
|
||||
| B3_C011 | True | ok_or_factual | factual | supplier_payouts_profile | None | Покажи топ-20 самых крупных разовых выплат поставщикам (дата, контрагент, документ, сумма). | Собран профиль выплат поставщикам (bank-doc value aggregate). |
|
||||
| B3_C012 | True | ok_or_factual | factual | contract_usage_and_value | None | Покажи топ-20 договоров по сумме оборота за все время. | Собран профиль договоров по обороту/бюджету (bank-doc contract aggregate). |
|
||||
| B3_C013 | True | ok_or_factual | factual | contract_usage_and_value | None | Покажи топ-20 договоров с минимальным бюджетом среди активных договоров. | Собран профиль договоров по обороту/бюджету (bank-doc contract aggregate). |
|
||||
| B3_N001 | True | ok_or_factual | factual | customer_revenue_and_payments | None | какие клиенты самые доходные, выдай топ-20 | Собран профиль поступлений от заказчиков (bank-doc value aggregate). |
|
||||
| B3_N002 | True | ok_or_factual | factual | customer_revenue_and_payments | None | топ-20 заказчиков по деньгам за все время | Собран профиль поступлений от заказчиков (bank-doc value aggregate). |
|
||||
| B3_N003 | True | ok_or_factual | factual | customer_revenue_and_payments | None | за 20й год кто нам больше всего занес, топ-20 | Собран профиль поступлений от заказчиков (bank-doc value aggregate). |
|
||||
| B3_N004 | True | ok_or_factual | factual | customer_revenue_and_payments | None | кто платит чаще всего, дай топ-20 | Собран профиль поступлений от заказчиков (bank-doc value aggregate). |
|
||||
| B3_N005 | True | ok_or_factual | factual | customer_revenue_and_payments | None | покажи топ-20 самых жирных сделок по поступлениям | Собран профиль поступлений от заказчиков (bank-doc value aggregate). |
|
||||
| B3_N006 | False | blocked_unsupported | partial_coverage | unknown | unsupported | покажи топ-20 самых маленьких сделок по бюджету | Этот запрос не подходит под address_query V1. |
|
||||
| B3_N007 | True | ok_or_factual | factual | supplier_payouts_profile | None | кому мы больше всего сгрузили денег, топ-20 поставщиков | Собран профиль выплат поставщикам (bank-doc value aggregate). |
|
||||
| B3_N008 | True | ok_or_factual | factual | supplier_payouts_profile | None | топ-20 поставщиков по выплатам за все время | Собран профиль выплат поставщикам (bank-doc value aggregate). |
|
||||
| B3_N009 | True | ok_or_factual | factual | supplier_payouts_profile | None | за 2020 год кому ушло больше всего денег, топ-20 | Собран профиль выплат поставщикам (bank-doc value aggregate). |
|
||||
| B3_N010 | True | ok_or_factual | factual | supplier_payouts_profile | None | поставщики с максимальным числом выплат, топ-20 | Собран профиль выплат поставщикам (bank-doc value aggregate). |
|
||||
| B3_N011 | True | ok_or_factual | factual | contract_usage_and_value | None | договоры по обороту ранкни и дай топ-20 | Собран профиль договоров по обороту/бюджету (bank-doc contract aggregate). |
|
||||
| B3_N012 | True | ok_or_factual | factual | contract_usage_and_value | None | покажи топ-20 договоров с самым мелким бюджетом, но только активные | Собран профиль договоров по обороту/бюджету (bank-doc contract aggregate). |
|
||||
| B3_F001 | True | ok_or_factual | factual | customer_revenue_and_payments | None | Покажи топ-20 заказчиков по сумме поступлений за все время. | Собран профиль поступлений от заказчиков (bank-doc value aggregate). |
|
||||
| B3_F002 | True | ok_or_factual | factual | customer_revenue_and_payments | None | Теперь только за 2020 год, тоже топ-20. | Собран профиль поступлений от заказчиков (bank-doc value aggregate). |
|
||||
| B3_F003 | True | ok_or_factual | factual | customer_revenue_and_payments | None | И отдельно покажи топ-20 по частоте входящих платежей. | Собран профиль поступлений от заказчиков (bank-doc value aggregate). |
|
||||
| B3_F004 | True | ok_or_factual | factual | supplier_payouts_profile | None | Покажи топ-20 поставщиков по сумме выплат за все время. | Собран профиль выплат поставщикам (bank-doc value aggregate). |
|
||||
| B3_F005 | True | ok_or_factual | factual | supplier_payouts_profile | None | Теперь за 2020 год, тоже топ-20. | Собран профиль выплат поставщикам (bank-doc value aggregate). |
|
||||
| B3_F006 | True | ok_or_factual | factual | supplier_payouts_profile | None | И дай топ-20 поставщиков по количеству выплат. | Собран профиль выплат поставщикам (bank-doc value aggregate). |
|
||||
| B3_F007 | True | ok_or_factual | factual | customer_revenue_and_payments | None | Покажи топ-20 самых крупных разовых сделок по поступлениям. | Собран профиль поступлений от заказчиков (bank-doc value aggregate). |
|
||||
| B3_F008 | True | ok_or_factual | factual | customer_revenue_and_payments | None | А теперь топ-20 самых маленьких сделок по бюджету среди активных заказчиков. | Собран профиль поступлений от заказчиков (bank-doc value aggregate). |
|
||||
|
|
@ -0,0 +1,111 @@
|
|||
{
|
||||
"run_id": "2026-04-03_Address_Domain_General_B3_Value_LiveGate",
|
||||
"generated_at": "2026-04-03T08:52:00",
|
||||
"source_questions_file": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\question_sets\\temp_batch3_value_top20_2026-04-02.json",
|
||||
"backend_url": "http://127.0.0.1:8787/api/assistant/message",
|
||||
"llm_provider": "local",
|
||||
"llm_model": "qwen2.5-14b-instruct-1m",
|
||||
"llm_base_url": "http://127.0.0.1:1234",
|
||||
"strict_policy": "route",
|
||||
"totals": {
|
||||
"questions_total": 33,
|
||||
"ok_200_count": 33,
|
||||
"semantic_pass_count": 32,
|
||||
"semantic_pass_rate": 0.9697,
|
||||
"route_pass_count": 32,
|
||||
"route_pass_rate": 0.9697,
|
||||
"strict_pass_count": 32,
|
||||
"strict_pass_rate": 0.9697,
|
||||
"factual_count": 32,
|
||||
"partial_coverage_count": 1,
|
||||
"clarification_required_count": 0,
|
||||
"http_error_count": 0,
|
||||
"llm_decomposition_attempted_count": 33,
|
||||
"llm_decomposition_applied_count": 30,
|
||||
"llm_fallback_count": 0,
|
||||
"llm_fallback_rate": 0.0,
|
||||
"tool_gate_blocked_count": 0,
|
||||
"tool_gate_blocked_rate": 0.0,
|
||||
"avg_elapsed_ms": 6369.7
|
||||
},
|
||||
"distributions": {
|
||||
"reply_type": {
|
||||
"factual": 32,
|
||||
"partial_coverage": 1
|
||||
},
|
||||
"actual_intent": {
|
||||
"customer_revenue_and_payments": 17,
|
||||
"supplier_payouts_profile": 11,
|
||||
"contract_usage_and_value": 4,
|
||||
"unknown": 1
|
||||
},
|
||||
"actual_mode": {
|
||||
"address_query": 33
|
||||
},
|
||||
"mcp_call_status": {
|
||||
"matched_non_empty": 32,
|
||||
"skipped": 1
|
||||
},
|
||||
"limited_reason_category": {
|
||||
"unsupported": 1
|
||||
},
|
||||
"route_health": {
|
||||
"ok_or_factual": 32,
|
||||
"blocked_unsupported": 1
|
||||
},
|
||||
"tool_gate_decision": {
|
||||
"run_address_lane": 33
|
||||
},
|
||||
"tool_gate_reason": {
|
||||
"address_mode_classifier_detected": 33
|
||||
}
|
||||
},
|
||||
"address_llm_predecompose_metrics": {
|
||||
"overall": {
|
||||
"llm_attempted": 33,
|
||||
"llm_applied": 30,
|
||||
"fallback_used": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"tool_gate_blocked": 0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"by_intent": {
|
||||
"customer_revenue_and_payments": {
|
||||
"total": 17,
|
||||
"llm_attempted": 17,
|
||||
"llm_applied": 16,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"supplier_payouts_profile": {
|
||||
"total": 11,
|
||||
"llm_attempted": 11,
|
||||
"llm_applied": 9,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"contract_usage_and_value": {
|
||||
"total": 4,
|
||||
"llm_attempted": 4,
|
||||
"llm_applied": 4,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"unknown": {
|
||||
"total": 1,
|
||||
"llm_attempted": 1,
|
||||
"llm_applied": 1,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,33 @@
|
|||
# 2026-04-03_Address_Domain_General_B3_Value_LiveGate_R2
|
||||
|
||||
Generated at: 2026-04-03T08:56:58
|
||||
Questions file: X:\1C\NDC_1C\docs\ADDRESS\question_sets\temp_batch3_value_top20_2026-04-02.json
|
||||
Backend URL: http://127.0.0.1:8787/api/assistant/message
|
||||
LLM: local / qwen2.5-14b-instruct-1m @ http://127.0.0.1:1234
|
||||
Strict policy: route
|
||||
|
||||
## Totals
|
||||
- questions_total: 33
|
||||
- ok_200_count: 33
|
||||
- semantic_pass_count: 33
|
||||
- semantic_pass_rate: 1.0
|
||||
- route_pass_count: 33
|
||||
- route_pass_rate: 1.0
|
||||
- strict_pass_count: 33
|
||||
- strict_pass_rate: 1.0
|
||||
- factual_count: 33
|
||||
- partial_coverage_count: 0
|
||||
- clarification_required_count: 0
|
||||
- http_error_count: 0
|
||||
- llm_decomposition_attempted_count: 33
|
||||
- llm_decomposition_applied_count: 30
|
||||
- llm_fallback_count: 0
|
||||
- llm_fallback_rate: 0.0
|
||||
- tool_gate_blocked_count: 0
|
||||
- tool_gate_blocked_rate: 0.0
|
||||
- avg_elapsed_ms: 6383.8
|
||||
|
||||
## Files
|
||||
- run_summary.json
|
||||
- full_live_results.json
|
||||
- failures_only.json
|
||||
|
|
@ -0,0 +1 @@
|
|||
[]
|
||||
File diff suppressed because it is too large
Load Diff
|
|
@ -0,0 +1,37 @@
|
|||
# Response Audit: 2026-04-03_Address_Domain_General_B3_Value_LiveGate_R2
|
||||
|
||||
| id | strict | route_health | reply_type | intent | limited_reason | question | assistant_first_line |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| B3_C001 | True | ok_or_factual | factual | customer_revenue_and_payments | None | Покажи топ-20 заказчиков по сумме поступлений за все время. | Собран профиль поступлений от заказчиков (bank-doc value aggregate). |
|
||||
| B3_C002 | True | ok_or_factual | factual | customer_revenue_and_payments | None | Покажи топ-20 заказчиков по сумме поступлений за 2020 год. | Собран профиль поступлений от заказчиков (bank-doc value aggregate). |
|
||||
| B3_C003 | True | ok_or_factual | factual | customer_revenue_and_payments | None | Покажи топ-20 заказчиков по количеству входящих платежных операций за все время. | Собран профиль поступлений от заказчиков (bank-doc value aggregate). |
|
||||
| B3_C004 | True | ok_or_factual | factual | customer_revenue_and_payments | None | Покажи топ-20 заказчиков по максимальной сумме одной входящей операции за все время. | Собран профиль поступлений от заказчиков (bank-doc value aggregate). |
|
||||
| B3_C005 | True | ok_or_factual | factual | customer_revenue_and_payments | None | Покажи топ-20 заказчиков по среднему чеку среди активных клиентов (минимум 3 входящие операции). | Собран профиль поступлений от заказчиков (bank-doc value aggregate). |
|
||||
| B3_C006 | True | ok_or_factual | factual | customer_revenue_and_payments | None | Покажи топ-20 самых крупных разовых сделок по поступлениям (дата, контрагент, документ, сумма). | Собран профиль поступлений от заказчиков (bank-doc value aggregate). |
|
||||
| B3_C007 | True | ok_or_factual | factual | customer_revenue_and_payments | None | Покажи топ-20 самых маленьких разовых сделок по поступлениям среди активных заказчиков. | Собран профиль поступлений от заказчиков (bank-doc value aggregate). |
|
||||
| B3_C008 | True | ok_or_factual | factual | supplier_payouts_profile | None | Покажи топ-20 поставщиков по сумме выплат за все время. | Собран профиль выплат поставщикам (bank-doc value aggregate). |
|
||||
| B3_C009 | True | ok_or_factual | factual | supplier_payouts_profile | None | Покажи топ-20 поставщиков по сумме выплат за 2020 год. | Собран профиль выплат поставщикам (bank-doc value aggregate). |
|
||||
| B3_C010 | True | ok_or_factual | factual | supplier_payouts_profile | None | Покажи топ-20 поставщиков по количеству исходящих платежных операций за все время. | Собран профиль выплат поставщикам (bank-doc value aggregate). |
|
||||
| B3_C011 | True | ok_or_factual | factual | supplier_payouts_profile | None | Покажи топ-20 самых крупных разовых выплат поставщикам (дата, контрагент, документ, сумма). | Собран профиль выплат поставщикам (bank-doc value aggregate). |
|
||||
| B3_C012 | True | ok_or_factual | factual | contract_usage_and_value | None | Покажи топ-20 договоров по сумме оборота за все время. | Собран профиль договоров по обороту/бюджету (bank-doc contract aggregate). |
|
||||
| B3_C013 | True | ok_or_factual | factual | contract_usage_and_value | None | Покажи топ-20 договоров с минимальным бюджетом среди активных договоров. | Собран профиль договоров по обороту/бюджету (bank-doc contract aggregate). |
|
||||
| B3_N001 | True | ok_or_factual | factual | customer_revenue_and_payments | None | какие клиенты самые доходные, выдай топ-20 | Собран профиль поступлений от заказчиков (bank-doc value aggregate). |
|
||||
| B3_N002 | True | ok_or_factual | factual | customer_revenue_and_payments | None | топ-20 заказчиков по деньгам за все время | Собран профиль поступлений от заказчиков (bank-doc value aggregate). |
|
||||
| B3_N003 | True | ok_or_factual | factual | customer_revenue_and_payments | None | за 20й год кто нам больше всего занес, топ-20 | Собран профиль поступлений от заказчиков (bank-doc value aggregate). |
|
||||
| B3_N004 | True | ok_or_factual | factual | customer_revenue_and_payments | None | кто платит чаще всего, дай топ-20 | Собран профиль поступлений от заказчиков (bank-doc value aggregate). |
|
||||
| B3_N005 | True | ok_or_factual | factual | customer_revenue_and_payments | None | покажи топ-20 самых жирных сделок по поступлениям | Собран профиль поступлений от заказчиков (bank-doc value aggregate). |
|
||||
| B3_N006 | True | ok_or_factual | factual | customer_revenue_and_payments | None | покажи топ-20 самых маленьких сделок по бюджету | Собран профиль поступлений от заказчиков (bank-doc value aggregate). |
|
||||
| B3_N007 | True | ok_or_factual | factual | supplier_payouts_profile | None | кому мы больше всего сгрузили денег, топ-20 поставщиков | Собран профиль выплат поставщикам (bank-doc value aggregate). |
|
||||
| B3_N008 | True | ok_or_factual | factual | supplier_payouts_profile | None | топ-20 поставщиков по выплатам за все время | Собран профиль выплат поставщикам (bank-doc value aggregate). |
|
||||
| B3_N009 | True | ok_or_factual | factual | supplier_payouts_profile | None | за 2020 год кому ушло больше всего денег, топ-20 | Собран профиль выплат поставщикам (bank-doc value aggregate). |
|
||||
| B3_N010 | True | ok_or_factual | factual | supplier_payouts_profile | None | поставщики с максимальным числом выплат, топ-20 | Собран профиль выплат поставщикам (bank-doc value aggregate). |
|
||||
| B3_N011 | True | ok_or_factual | factual | contract_usage_and_value | None | договоры по обороту ранкни и дай топ-20 | Собран профиль договоров по обороту/бюджету (bank-doc contract aggregate). |
|
||||
| B3_N012 | True | ok_or_factual | factual | contract_usage_and_value | None | покажи топ-20 договоров с самым мелким бюджетом, но только активные | Собран профиль договоров по обороту/бюджету (bank-doc contract aggregate). |
|
||||
| B3_F001 | True | ok_or_factual | factual | customer_revenue_and_payments | None | Покажи топ-20 заказчиков по сумме поступлений за все время. | Собран профиль поступлений от заказчиков (bank-doc value aggregate). |
|
||||
| B3_F002 | True | ok_or_factual | factual | customer_revenue_and_payments | None | Теперь только за 2020 год, тоже топ-20. | Собран профиль поступлений от заказчиков (bank-doc value aggregate). |
|
||||
| B3_F003 | True | ok_or_factual | factual | customer_revenue_and_payments | None | И отдельно покажи топ-20 по частоте входящих платежей. | Собран профиль поступлений от заказчиков (bank-doc value aggregate). |
|
||||
| B3_F004 | True | ok_or_factual | factual | supplier_payouts_profile | None | Покажи топ-20 поставщиков по сумме выплат за все время. | Собран профиль выплат поставщикам (bank-doc value aggregate). |
|
||||
| B3_F005 | True | ok_or_factual | factual | supplier_payouts_profile | None | Теперь за 2020 год, тоже топ-20. | Собран профиль выплат поставщикам (bank-doc value aggregate). |
|
||||
| B3_F006 | True | ok_or_factual | factual | supplier_payouts_profile | None | И дай топ-20 поставщиков по количеству выплат. | Собран профиль выплат поставщикам (bank-doc value aggregate). |
|
||||
| B3_F007 | True | ok_or_factual | factual | customer_revenue_and_payments | None | Покажи топ-20 самых крупных разовых сделок по поступлениям. | Собран профиль поступлений от заказчиков (bank-doc value aggregate). |
|
||||
| B3_F008 | True | ok_or_factual | factual | customer_revenue_and_payments | None | А теперь топ-20 самых маленьких сделок по бюджету среди активных заказчиков. | Собран профиль поступлений от заказчиков (bank-doc value aggregate). |
|
||||
|
|
@ -0,0 +1,96 @@
|
|||
{
|
||||
"run_id": "2026-04-03_Address_Domain_General_B3_Value_LiveGate_R2",
|
||||
"generated_at": "2026-04-03T08:56:58",
|
||||
"source_questions_file": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\question_sets\\temp_batch3_value_top20_2026-04-02.json",
|
||||
"backend_url": "http://127.0.0.1:8787/api/assistant/message",
|
||||
"llm_provider": "local",
|
||||
"llm_model": "qwen2.5-14b-instruct-1m",
|
||||
"llm_base_url": "http://127.0.0.1:1234",
|
||||
"strict_policy": "route",
|
||||
"totals": {
|
||||
"questions_total": 33,
|
||||
"ok_200_count": 33,
|
||||
"semantic_pass_count": 33,
|
||||
"semantic_pass_rate": 1.0,
|
||||
"route_pass_count": 33,
|
||||
"route_pass_rate": 1.0,
|
||||
"strict_pass_count": 33,
|
||||
"strict_pass_rate": 1.0,
|
||||
"factual_count": 33,
|
||||
"partial_coverage_count": 0,
|
||||
"clarification_required_count": 0,
|
||||
"http_error_count": 0,
|
||||
"llm_decomposition_attempted_count": 33,
|
||||
"llm_decomposition_applied_count": 30,
|
||||
"llm_fallback_count": 0,
|
||||
"llm_fallback_rate": 0.0,
|
||||
"tool_gate_blocked_count": 0,
|
||||
"tool_gate_blocked_rate": 0.0,
|
||||
"avg_elapsed_ms": 6383.8
|
||||
},
|
||||
"distributions": {
|
||||
"reply_type": {
|
||||
"factual": 33
|
||||
},
|
||||
"actual_intent": {
|
||||
"customer_revenue_and_payments": 18,
|
||||
"supplier_payouts_profile": 11,
|
||||
"contract_usage_and_value": 4
|
||||
},
|
||||
"actual_mode": {
|
||||
"address_query": 33
|
||||
},
|
||||
"mcp_call_status": {
|
||||
"matched_non_empty": 33
|
||||
},
|
||||
"limited_reason_category": {},
|
||||
"route_health": {
|
||||
"ok_or_factual": 33
|
||||
},
|
||||
"tool_gate_decision": {
|
||||
"run_address_lane": 33
|
||||
},
|
||||
"tool_gate_reason": {
|
||||
"address_mode_classifier_detected": 33
|
||||
}
|
||||
},
|
||||
"address_llm_predecompose_metrics": {
|
||||
"overall": {
|
||||
"llm_attempted": 33,
|
||||
"llm_applied": 30,
|
||||
"fallback_used": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"tool_gate_blocked": 0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"by_intent": {
|
||||
"customer_revenue_and_payments": {
|
||||
"total": 18,
|
||||
"llm_attempted": 18,
|
||||
"llm_applied": 17,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"supplier_payouts_profile": {
|
||||
"total": 11,
|
||||
"llm_attempted": 11,
|
||||
"llm_applied": 9,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"contract_usage_and_value": {
|
||||
"total": 4,
|
||||
"llm_attempted": 4,
|
||||
"llm_applied": 4,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,33 @@
|
|||
# 2026-04-03_Address_Nightly_followup_25_09-53-20
|
||||
|
||||
Generated at: 2026-04-03T09:56:07
|
||||
Questions file: X:\1C\NDC_1C\docs\ADDRESS\question_sets\address_followup_context_chains_2026-04-02.json
|
||||
Backend URL: http://127.0.0.1:8787/api/assistant/message
|
||||
LLM: local / qwen2.5-14b-instruct-1m @ http://127.0.0.1:1234
|
||||
Strict policy: route
|
||||
|
||||
## Totals
|
||||
- questions_total: 25
|
||||
- ok_200_count: 25
|
||||
- semantic_pass_count: 22
|
||||
- semantic_pass_rate: 0.88
|
||||
- route_pass_count: 20
|
||||
- route_pass_rate: 0.8
|
||||
- strict_pass_count: 20
|
||||
- strict_pass_rate: 0.8
|
||||
- factual_count: 19
|
||||
- partial_coverage_count: 6
|
||||
- clarification_required_count: 0
|
||||
- http_error_count: 0
|
||||
- llm_decomposition_attempted_count: 25
|
||||
- llm_decomposition_applied_count: 24
|
||||
- llm_fallback_count: 0
|
||||
- llm_fallback_rate: 0.0
|
||||
- tool_gate_blocked_count: 0
|
||||
- tool_gate_blocked_rate: 0.0
|
||||
- avg_elapsed_ms: 6659.5
|
||||
|
||||
## Files
|
||||
- run_summary.json
|
||||
- full_live_results.json
|
||||
- failures_only.json
|
||||
|
|
@ -0,0 +1,936 @@
|
|||
[
|
||||
{
|
||||
"index": 13,
|
||||
"id": "F013",
|
||||
"question": "а на ту же дату по 62",
|
||||
"session": "chain_balance_c",
|
||||
"session_id": "asst-2026-04-03_Address_Nightly_followup_25_09-53-20-chain_balance_c",
|
||||
"status_code": 200,
|
||||
"ok": true,
|
||||
"elapsed_ms": 5072,
|
||||
"reply_type": "factual",
|
||||
"trace_id": "address-QJ5K59CgaG",
|
||||
"assistant_reply": "Собран drilldown документов, формирующих остаток по счету на указанную дату.\nДокументных строк отобрано: 4.\nСумма по отобранным движениям: 2545546.24.\n1. 2020-05-21T12:00:00Z | Поступление на расчетный счет 00000000023 от 21.05.2020 12:00:00 | 51 / 62.02 | 250546.24\n2. 2020-05-18T12:00:01Z | Реализация товаров и услуг 00000000002 от 18.05.2020 12:00:01 | 62.01 / 90.01.1 | 765000\n3. 2020-05-18T12:00:01Z | Реализация товаров и услуг 00000000002 от 18.05.2020 12:00:01 | 62.02 / 62.01 | 765000\n4. 2020-05-15T12:00:02Z | Поступление на расчетный счет 00000000022 от 15.05.2020 12:00:02 | 51 / 62.02 | 765000\nМожно уточнить выборку по контрагенту, договору или периоду.",
|
||||
"assistant_reply_first_line": "Собран drilldown документов, формирующих остаток по счету на указанную дату.",
|
||||
"expected_intent": "account_balance_snapshot",
|
||||
"actual_intent": "documents_forming_balance",
|
||||
"intent_match": false,
|
||||
"expected_mode": "address_query",
|
||||
"actual_mode": "address_query",
|
||||
"mode_match": true,
|
||||
"expected_reply_type": null,
|
||||
"reply_match": true,
|
||||
"semantic_pass": false,
|
||||
"route_pass": false,
|
||||
"route_health": "ok_or_factual",
|
||||
"strict_policy": "route",
|
||||
"strict_pass": false,
|
||||
"selected_recipe": "address_documents_forming_balance_v1",
|
||||
"missing_required_filters": [],
|
||||
"match_failure_stage": "none",
|
||||
"match_failure_reason": null,
|
||||
"rows_fetched": 4,
|
||||
"rows_matched": 4,
|
||||
"mcp_call_status": "matched_non_empty",
|
||||
"limited_reason_category": null,
|
||||
"llm_decomposition_attempted": true,
|
||||
"llm_decomposition_applied": true,
|
||||
"llm_decomposition_reason": "normalized_fragment_applied",
|
||||
"llm_canonical_candidate_detected": true,
|
||||
"fallback_rule_hit": null,
|
||||
"tool_gate_decision": "run_address_lane",
|
||||
"tool_gate_reason": "address_mode_classifier_detected",
|
||||
"predecompose_contract_intent": "documents_forming_balance",
|
||||
"predecompose_contract_aggregation_profile": "balance_snapshot",
|
||||
"predecompose_contract_period_scope": "as_of",
|
||||
"debug_payload": {
|
||||
"trace_id": "address-QJ5K59CgaG",
|
||||
"prompt_version": "address_query_runtime_v1",
|
||||
"schema_version": "address_query_runtime_v1",
|
||||
"fallback_type": "none",
|
||||
"route_summary": null,
|
||||
"fragments": [],
|
||||
"requirements_extracted": [],
|
||||
"coverage_report": {
|
||||
"requirements_total": 0,
|
||||
"requirements_covered": 0,
|
||||
"requirements_uncovered": [],
|
||||
"requirements_partially_covered": [],
|
||||
"clarification_needed_for": [],
|
||||
"out_of_scope_requirements": []
|
||||
},
|
||||
"routes": [],
|
||||
"retrieval_status": [],
|
||||
"retrieval_results": [],
|
||||
"answer_grounding_check": {
|
||||
"status": "grounded",
|
||||
"route_subject_match": true,
|
||||
"missing_requirements": [],
|
||||
"reasons": [
|
||||
"address_entity_detected",
|
||||
"document_list_signal_detected",
|
||||
"documents_by_account_drilldown_signal_detected",
|
||||
"as_of_date_from_followup_context",
|
||||
"period_from_followup_context",
|
||||
"address_followup_context_applied"
|
||||
],
|
||||
"why_included_summary": [],
|
||||
"selection_reason_summary": []
|
||||
},
|
||||
"dropped_intent_segments": [],
|
||||
"detected_mode": "address_query",
|
||||
"detected_mode_confidence": "medium",
|
||||
"query_shape": "DOCUMENT_LIST",
|
||||
"query_shape_confidence": "medium",
|
||||
"detected_intent": "documents_forming_balance",
|
||||
"detected_intent_confidence": "medium",
|
||||
"extracted_filters": {
|
||||
"sort": "period_desc",
|
||||
"limit": 20,
|
||||
"account": "62",
|
||||
"as_of_date": "2020-05-31",
|
||||
"period_from": "2020-05-01",
|
||||
"period_to": "2020-05-31"
|
||||
},
|
||||
"missing_required_filters": [],
|
||||
"selected_recipe": "address_documents_forming_balance_v1",
|
||||
"mcp_call_status_legacy": "matched_non_empty",
|
||||
"account_scope_mode": "strict",
|
||||
"account_scope_fallback_applied": false,
|
||||
"anchor_type": "account",
|
||||
"anchor_value_raw": "62",
|
||||
"anchor_value_resolved": "62",
|
||||
"resolver_confidence": "high",
|
||||
"ambiguity_count": 0,
|
||||
"match_failure_stage": "none",
|
||||
"match_failure_reason": null,
|
||||
"mcp_call_status": "matched_non_empty",
|
||||
"rows_fetched": 4,
|
||||
"raw_rows_received": 4,
|
||||
"rows_after_account_scope": 4,
|
||||
"rows_after_recipe_filter": 4,
|
||||
"rows_materialized": 4,
|
||||
"rows_matched": 4,
|
||||
"raw_row_keys_sample": [
|
||||
"Период",
|
||||
"Регистратор",
|
||||
"СчетДт",
|
||||
"СчетКт",
|
||||
"Сумма",
|
||||
"Period",
|
||||
"Registrator",
|
||||
"AccountDt",
|
||||
"AccountKt",
|
||||
"Amount"
|
||||
],
|
||||
"materialization_drop_reason": "none",
|
||||
"account_token_raw": "62",
|
||||
"account_token_normalized": "62",
|
||||
"account_scope_fields_checked": [
|
||||
"account_dt",
|
||||
"account_kt",
|
||||
"registrator",
|
||||
"analytics"
|
||||
],
|
||||
"account_scope_match_strategy": "account_code_regex_plus_alias_map_v1",
|
||||
"account_scope_drop_reason": "rows_remaining_after_scope_filter",
|
||||
"runtime_readiness": "LIVE_QUERYABLE_WITH_LIMITS",
|
||||
"limited_reason_category": null,
|
||||
"response_type": "FACTUAL_LIST",
|
||||
"execution_lane": "address_query",
|
||||
"llm_decomposition_applied": true,
|
||||
"llm_decomposition_attempted": true,
|
||||
"llm_provider_used": "local",
|
||||
"llm_decomposition_trace_id": "XUkxb5h2IQB9YC",
|
||||
"llm_decomposition_effective_message": "документы или проводки по счету 62 на ту же дату",
|
||||
"llm_decomposition_reason": "normalized_fragment_applied",
|
||||
"llm_canonical_candidate_detected": true,
|
||||
"llm_predecompose_contract": {
|
||||
"schema_version": "address_llm_predecompose_contract_v1",
|
||||
"source_message": "а на ту же дату по 62",
|
||||
"canonical_message": "документы или проводки по счету 62 на ту же дату",
|
||||
"mode": "address_query",
|
||||
"mode_confidence": "medium",
|
||||
"query_shape": "DOCUMENT_LIST",
|
||||
"query_shape_confidence": "medium",
|
||||
"intent": "documents_forming_balance",
|
||||
"intent_confidence": "medium",
|
||||
"entities": {
|
||||
"account": "62",
|
||||
"counterparty": null,
|
||||
"contract": null,
|
||||
"document_type": null,
|
||||
"document_ref": null,
|
||||
"organization": null
|
||||
},
|
||||
"period": {
|
||||
"scope": "as_of",
|
||||
"period_from": null,
|
||||
"period_to": null,
|
||||
"as_of_date": "2026-04-03",
|
||||
"has_explicit_period": true
|
||||
},
|
||||
"aggregation_profile": "balance_snapshot"
|
||||
},
|
||||
"fallback_rule_hit": null,
|
||||
"sanitized_user_message": "а на ту же дату по 62",
|
||||
"tool_gate_decision": "run_address_lane",
|
||||
"tool_gate_reason": "address_mode_classifier_detected",
|
||||
"answer_structure_v11": null,
|
||||
"investigation_state_snapshot": null,
|
||||
"normalized": null,
|
||||
"normalizer_output": {
|
||||
"trace_id": "XUkxb5h2IQB9YC",
|
||||
"prompt_version": "normalizer_v2_0_2",
|
||||
"applied": true,
|
||||
"effective_message": "документы или проводки по счету 62 на ту же дату"
|
||||
}
|
||||
},
|
||||
"error_code": null,
|
||||
"error_message": null
|
||||
},
|
||||
{
|
||||
"index": 17,
|
||||
"id": "F017",
|
||||
"question": "а теперь банковские операции",
|
||||
"session": "chain_contract_d",
|
||||
"session_id": "asst-2026-04-03_Address_Nightly_followup_25_09-53-20-chain_contract_d",
|
||||
"status_code": 200,
|
||||
"ok": true,
|
||||
"elapsed_ms": 9204,
|
||||
"reply_type": "partial_coverage",
|
||||
"trace_id": "address-9NUBRFG3Rt",
|
||||
"assistant_reply": "В live-данных по текущему фильтру записи не найдены.\nПричина: по указанному якорю и фильтрам в live-выборке нет строк.\nЧто нужно уточнить: уточните период или снимите часть фильтров.",
|
||||
"assistant_reply_first_line": "В live-данных по текущему фильтру записи не найдены.",
|
||||
"expected_intent": "bank_operations_by_contract",
|
||||
"actual_intent": "bank_operations_by_counterparty",
|
||||
"intent_match": false,
|
||||
"expected_mode": "address_query",
|
||||
"actual_mode": "address_query",
|
||||
"mode_match": true,
|
||||
"expected_reply_type": null,
|
||||
"reply_match": true,
|
||||
"semantic_pass": false,
|
||||
"route_pass": false,
|
||||
"route_health": "likely_blocked_route",
|
||||
"strict_policy": "route",
|
||||
"strict_pass": false,
|
||||
"selected_recipe": "address_bank_operations_by_counterparty_v1",
|
||||
"missing_required_filters": [],
|
||||
"match_failure_stage": "materialized_but_not_anchor_matched",
|
||||
"match_failure_reason": "counterparty_anchor_not_matched_in_materialized_rows",
|
||||
"rows_fetched": 20,
|
||||
"rows_matched": 0,
|
||||
"mcp_call_status": "materialized_but_not_anchor_matched",
|
||||
"limited_reason_category": "empty_match",
|
||||
"llm_decomposition_attempted": true,
|
||||
"llm_decomposition_applied": true,
|
||||
"llm_decomposition_reason": "fallback_rule_preferred_over_llm_candidate_anchor_quality",
|
||||
"llm_canonical_candidate_detected": true,
|
||||
"fallback_rule_hit": "bank_operations_counterparty_rewrite",
|
||||
"tool_gate_decision": "run_address_lane",
|
||||
"tool_gate_reason": "address_mode_classifier_detected",
|
||||
"predecompose_contract_intent": "bank_operations_by_counterparty",
|
||||
"predecompose_contract_aggregation_profile": "list_lookup",
|
||||
"predecompose_contract_period_scope": "unspecified",
|
||||
"debug_payload": {
|
||||
"trace_id": "address-9NUBRFG3Rt",
|
||||
"prompt_version": "address_query_runtime_v1",
|
||||
"schema_version": "address_query_runtime_v1",
|
||||
"fallback_type": "partial",
|
||||
"route_summary": null,
|
||||
"fragments": [],
|
||||
"requirements_extracted": [],
|
||||
"coverage_report": {
|
||||
"requirements_total": 0,
|
||||
"requirements_covered": 0,
|
||||
"requirements_uncovered": [],
|
||||
"requirements_partially_covered": [],
|
||||
"clarification_needed_for": [],
|
||||
"out_of_scope_requirements": []
|
||||
},
|
||||
"routes": [],
|
||||
"retrieval_status": [],
|
||||
"retrieval_results": [],
|
||||
"answer_grounding_check": {
|
||||
"status": "partial",
|
||||
"route_subject_match": true,
|
||||
"missing_requirements": [],
|
||||
"reasons": [
|
||||
"address_entity_detected",
|
||||
"object_signal_detected",
|
||||
"bank_ops_by_counterparty_signal_detected",
|
||||
"period_from_followup_context",
|
||||
"address_followup_context_applied"
|
||||
],
|
||||
"why_included_summary": [],
|
||||
"selection_reason_summary": []
|
||||
},
|
||||
"dropped_intent_segments": [],
|
||||
"detected_mode": "address_query",
|
||||
"detected_mode_confidence": "medium",
|
||||
"query_shape": "OBJECT_LOOKUP",
|
||||
"query_shape_confidence": "medium",
|
||||
"detected_intent": "bank_operations_by_counterparty",
|
||||
"detected_intent_confidence": "medium",
|
||||
"extracted_filters": {
|
||||
"sort": "period_desc",
|
||||
"limit": 20,
|
||||
"counterparty": "теперь",
|
||||
"period_from": "2020-01-01",
|
||||
"period_to": "2020-12-31"
|
||||
},
|
||||
"missing_required_filters": [],
|
||||
"selected_recipe": "address_bank_operations_by_counterparty_v1",
|
||||
"mcp_call_status_legacy": "materialized_but_not_matched",
|
||||
"account_scope_mode": "preferred",
|
||||
"account_scope_fallback_applied": false,
|
||||
"anchor_type": "counterparty",
|
||||
"anchor_value_raw": "теперь",
|
||||
"anchor_value_resolved": "теперь",
|
||||
"resolver_confidence": "medium",
|
||||
"ambiguity_count": 0,
|
||||
"match_failure_stage": "materialized_but_not_anchor_matched",
|
||||
"match_failure_reason": "counterparty_anchor_not_matched_in_materialized_rows",
|
||||
"mcp_call_status": "materialized_but_not_anchor_matched",
|
||||
"rows_fetched": 20,
|
||||
"raw_rows_received": 20,
|
||||
"rows_after_account_scope": 5,
|
||||
"rows_after_recipe_filter": 0,
|
||||
"rows_materialized": 5,
|
||||
"rows_matched": 0,
|
||||
"raw_row_keys_sample": [
|
||||
"Период",
|
||||
"Регистратор",
|
||||
"СчетДт",
|
||||
"СчетКт",
|
||||
"Сумма",
|
||||
"Контрагент",
|
||||
"Period",
|
||||
"Registrator",
|
||||
"Amount"
|
||||
],
|
||||
"materialization_drop_reason": "none",
|
||||
"account_token_raw": null,
|
||||
"account_token_normalized": null,
|
||||
"account_scope_fields_checked": [
|
||||
"account_dt",
|
||||
"account_kt",
|
||||
"registrator",
|
||||
"analytics"
|
||||
],
|
||||
"account_scope_match_strategy": "account_code_regex_plus_alias_map_v1",
|
||||
"account_scope_drop_reason": "not_applicable",
|
||||
"runtime_readiness": "LIVE_QUERYABLE_WITH_LIMITS",
|
||||
"limited_reason_category": "empty_match",
|
||||
"response_type": "LIMITED_WITH_REASON",
|
||||
"execution_lane": "address_query",
|
||||
"llm_decomposition_applied": true,
|
||||
"llm_decomposition_attempted": true,
|
||||
"llm_provider_used": "local",
|
||||
"llm_decomposition_trace_id": "psdW7A8T8Ot3eM",
|
||||
"llm_decomposition_effective_message": "банковские операции по контрагенту теперь",
|
||||
"llm_decomposition_reason": "fallback_rule_preferred_over_llm_candidate_anchor_quality",
|
||||
"llm_canonical_candidate_detected": true,
|
||||
"llm_predecompose_contract": {
|
||||
"schema_version": "address_llm_predecompose_contract_v1",
|
||||
"source_message": "а теперь банковские операции",
|
||||
"canonical_message": "банковские операции по контрагенту теперь",
|
||||
"mode": "address_query",
|
||||
"mode_confidence": "medium",
|
||||
"query_shape": "OBJECT_LOOKUP",
|
||||
"query_shape_confidence": "medium",
|
||||
"intent": "bank_operations_by_counterparty",
|
||||
"intent_confidence": "medium",
|
||||
"entities": {
|
||||
"account": null,
|
||||
"counterparty": "теперь",
|
||||
"contract": null,
|
||||
"document_type": null,
|
||||
"document_ref": null,
|
||||
"organization": null
|
||||
},
|
||||
"period": {
|
||||
"scope": "unspecified",
|
||||
"period_from": null,
|
||||
"period_to": null,
|
||||
"as_of_date": null,
|
||||
"has_explicit_period": false
|
||||
},
|
||||
"aggregation_profile": "list_lookup"
|
||||
},
|
||||
"fallback_rule_hit": "bank_operations_counterparty_rewrite",
|
||||
"sanitized_user_message": "а теперь банковские операции",
|
||||
"tool_gate_decision": "run_address_lane",
|
||||
"tool_gate_reason": "address_mode_classifier_detected",
|
||||
"answer_structure_v11": null,
|
||||
"investigation_state_snapshot": null,
|
||||
"normalized": null,
|
||||
"normalizer_output": {
|
||||
"trace_id": "psdW7A8T8Ot3eM",
|
||||
"prompt_version": "normalizer_v2_0_2",
|
||||
"applied": true,
|
||||
"effective_message": "банковские операции по контрагенту теперь"
|
||||
}
|
||||
},
|
||||
"error_code": null,
|
||||
"error_message": null
|
||||
},
|
||||
{
|
||||
"index": 18,
|
||||
"id": "F018",
|
||||
"question": "а теперь только за 2020",
|
||||
"session": "chain_contract_d",
|
||||
"session_id": "asst-2026-04-03_Address_Nightly_followup_25_09-53-20-chain_contract_d",
|
||||
"status_code": 200,
|
||||
"ok": true,
|
||||
"elapsed_ms": 8127,
|
||||
"reply_type": "partial_coverage",
|
||||
"trace_id": "address-QfYZ2SMoY9",
|
||||
"assistant_reply": "В live-данных по текущему фильтру записи не найдены.\nПричина: по указанному якорю и фильтрам в live-выборке нет строк.\nЧто нужно уточнить: уточните период или снимите часть фильтров.",
|
||||
"assistant_reply_first_line": "В live-данных по текущему фильтру записи не найдены.",
|
||||
"expected_intent": "bank_operations_by_contract",
|
||||
"actual_intent": "bank_operations_by_counterparty",
|
||||
"intent_match": false,
|
||||
"expected_mode": "address_query",
|
||||
"actual_mode": "address_query",
|
||||
"mode_match": true,
|
||||
"expected_reply_type": null,
|
||||
"reply_match": true,
|
||||
"semantic_pass": false,
|
||||
"route_pass": false,
|
||||
"route_health": "likely_blocked_route",
|
||||
"strict_policy": "route",
|
||||
"strict_pass": false,
|
||||
"selected_recipe": "address_bank_operations_by_counterparty_v1",
|
||||
"missing_required_filters": [],
|
||||
"match_failure_stage": "materialized_but_not_anchor_matched",
|
||||
"match_failure_reason": "counterparty_anchor_not_matched_in_materialized_rows",
|
||||
"rows_fetched": 20,
|
||||
"rows_matched": 0,
|
||||
"mcp_call_status": "materialized_but_not_anchor_matched",
|
||||
"limited_reason_category": "empty_match",
|
||||
"llm_decomposition_attempted": true,
|
||||
"llm_decomposition_applied": true,
|
||||
"llm_decomposition_reason": "normalized_fragment_applied",
|
||||
"llm_canonical_candidate_detected": true,
|
||||
"fallback_rule_hit": null,
|
||||
"tool_gate_decision": "run_address_lane",
|
||||
"tool_gate_reason": "llm_canonical_candidate_detected",
|
||||
"predecompose_contract_intent": "unknown",
|
||||
"predecompose_contract_aggregation_profile": "unknown",
|
||||
"predecompose_contract_period_scope": "year",
|
||||
"debug_payload": {
|
||||
"trace_id": "address-QfYZ2SMoY9",
|
||||
"prompt_version": "address_query_runtime_v1",
|
||||
"schema_version": "address_query_runtime_v1",
|
||||
"fallback_type": "partial",
|
||||
"route_summary": null,
|
||||
"fragments": [],
|
||||
"requirements_extracted": [],
|
||||
"coverage_report": {
|
||||
"requirements_total": 0,
|
||||
"requirements_covered": 0,
|
||||
"requirements_uncovered": [],
|
||||
"requirements_partially_covered": [],
|
||||
"clarification_needed_for": [],
|
||||
"out_of_scope_requirements": []
|
||||
},
|
||||
"routes": [],
|
||||
"retrieval_status": [],
|
||||
"retrieval_results": [],
|
||||
"answer_grounding_check": {
|
||||
"status": "partial",
|
||||
"route_subject_match": true,
|
||||
"missing_requirements": [],
|
||||
"reasons": [
|
||||
"no_address_or_deep_signal",
|
||||
"address_mode_from_followup_context",
|
||||
"shape_not_detected",
|
||||
"intent_not_supported_in_v1",
|
||||
"intent_from_followup_context",
|
||||
"counterparty_from_followup_context",
|
||||
"address_followup_context_applied"
|
||||
],
|
||||
"why_included_summary": [],
|
||||
"selection_reason_summary": []
|
||||
},
|
||||
"dropped_intent_segments": [],
|
||||
"detected_mode": "address_query",
|
||||
"detected_mode_confidence": "medium",
|
||||
"query_shape": "UNKNOWN",
|
||||
"query_shape_confidence": "low",
|
||||
"detected_intent": "bank_operations_by_counterparty",
|
||||
"detected_intent_confidence": "low",
|
||||
"extracted_filters": {
|
||||
"sort": "period_desc",
|
||||
"limit": 20,
|
||||
"period_from": "2020-01-01",
|
||||
"period_to": "2020-12-31",
|
||||
"counterparty": "теперь"
|
||||
},
|
||||
"missing_required_filters": [],
|
||||
"selected_recipe": "address_bank_operations_by_counterparty_v1",
|
||||
"mcp_call_status_legacy": "materialized_but_not_matched",
|
||||
"account_scope_mode": "preferred",
|
||||
"account_scope_fallback_applied": false,
|
||||
"anchor_type": "counterparty",
|
||||
"anchor_value_raw": "теперь",
|
||||
"anchor_value_resolved": "теперь",
|
||||
"resolver_confidence": "medium",
|
||||
"ambiguity_count": 0,
|
||||
"match_failure_stage": "materialized_but_not_anchor_matched",
|
||||
"match_failure_reason": "counterparty_anchor_not_matched_in_materialized_rows",
|
||||
"mcp_call_status": "materialized_but_not_anchor_matched",
|
||||
"rows_fetched": 20,
|
||||
"raw_rows_received": 20,
|
||||
"rows_after_account_scope": 5,
|
||||
"rows_after_recipe_filter": 0,
|
||||
"rows_materialized": 5,
|
||||
"rows_matched": 0,
|
||||
"raw_row_keys_sample": [
|
||||
"Период",
|
||||
"Регистратор",
|
||||
"СчетДт",
|
||||
"СчетКт",
|
||||
"Сумма",
|
||||
"Контрагент",
|
||||
"Period",
|
||||
"Registrator",
|
||||
"Amount"
|
||||
],
|
||||
"materialization_drop_reason": "none",
|
||||
"account_token_raw": null,
|
||||
"account_token_normalized": null,
|
||||
"account_scope_fields_checked": [
|
||||
"account_dt",
|
||||
"account_kt",
|
||||
"registrator",
|
||||
"analytics"
|
||||
],
|
||||
"account_scope_match_strategy": "account_code_regex_plus_alias_map_v1",
|
||||
"account_scope_drop_reason": "not_applicable",
|
||||
"runtime_readiness": "LIVE_QUERYABLE_WITH_LIMITS",
|
||||
"limited_reason_category": "empty_match",
|
||||
"response_type": "LIMITED_WITH_REASON",
|
||||
"execution_lane": "address_query",
|
||||
"llm_decomposition_applied": true,
|
||||
"llm_decomposition_attempted": true,
|
||||
"llm_provider_used": "local",
|
||||
"llm_decomposition_trace_id": "w5Jjz0rwUDZcOB",
|
||||
"llm_decomposition_effective_message": "требуется информация только за 2020 год",
|
||||
"llm_decomposition_reason": "normalized_fragment_applied",
|
||||
"llm_canonical_candidate_detected": true,
|
||||
"llm_predecompose_contract": {
|
||||
"schema_version": "address_llm_predecompose_contract_v1",
|
||||
"source_message": "а теперь только за 2020",
|
||||
"canonical_message": "требуется информация только за 2020 год",
|
||||
"mode": "unsupported",
|
||||
"mode_confidence": "low",
|
||||
"query_shape": "UNKNOWN",
|
||||
"query_shape_confidence": "low",
|
||||
"intent": "unknown",
|
||||
"intent_confidence": "low",
|
||||
"entities": {
|
||||
"account": null,
|
||||
"counterparty": null,
|
||||
"contract": null,
|
||||
"document_type": null,
|
||||
"document_ref": null,
|
||||
"organization": null
|
||||
},
|
||||
"period": {
|
||||
"scope": "year",
|
||||
"period_from": "2020-01-01",
|
||||
"period_to": "2020-12-31",
|
||||
"as_of_date": null,
|
||||
"has_explicit_period": true
|
||||
},
|
||||
"aggregation_profile": "unknown"
|
||||
},
|
||||
"fallback_rule_hit": null,
|
||||
"sanitized_user_message": "а теперь только за 2020",
|
||||
"tool_gate_decision": "run_address_lane",
|
||||
"tool_gate_reason": "llm_canonical_candidate_detected",
|
||||
"answer_structure_v11": null,
|
||||
"investigation_state_snapshot": null,
|
||||
"normalized": null,
|
||||
"normalizer_output": {
|
||||
"trace_id": "w5Jjz0rwUDZcOB",
|
||||
"prompt_version": "normalizer_v2_0_2",
|
||||
"applied": true,
|
||||
"effective_message": "требуется информация только за 2020 год"
|
||||
}
|
||||
},
|
||||
"error_code": null,
|
||||
"error_message": null
|
||||
},
|
||||
{
|
||||
"index": 20,
|
||||
"id": "F020",
|
||||
"question": "покажи документы по этому же договору",
|
||||
"session": "chain_contract_d",
|
||||
"session_id": "asst-2026-04-03_Address_Nightly_followup_25_09-53-20-chain_contract_d",
|
||||
"status_code": 200,
|
||||
"ok": true,
|
||||
"elapsed_ms": 9067,
|
||||
"reply_type": "partial_coverage",
|
||||
"trace_id": "address-0bf4dWjq4o",
|
||||
"assistant_reply": "В live-данных по текущему фильтру записи не найдены.\nПричина: по указанному якорю и фильтрам в live-выборке нет строк.\nЧто нужно уточнить: уточните период или снимите часть фильтров.",
|
||||
"assistant_reply_first_line": "В live-данных по текущему фильтру записи не найдены.",
|
||||
"expected_intent": "list_documents_by_contract",
|
||||
"actual_intent": "list_documents_by_contract",
|
||||
"intent_match": true,
|
||||
"expected_mode": "address_query",
|
||||
"actual_mode": "address_query",
|
||||
"mode_match": true,
|
||||
"expected_reply_type": null,
|
||||
"reply_match": true,
|
||||
"semantic_pass": true,
|
||||
"route_pass": false,
|
||||
"route_health": "likely_blocked_route",
|
||||
"strict_policy": "route",
|
||||
"strict_pass": false,
|
||||
"selected_recipe": "address_documents_by_contract_v1",
|
||||
"missing_required_filters": [],
|
||||
"match_failure_stage": "materialized_but_not_anchor_matched",
|
||||
"match_failure_reason": "contract_anchor_not_matched_in_materialized_rows",
|
||||
"rows_fetched": 1000,
|
||||
"rows_matched": 0,
|
||||
"mcp_call_status": "materialized_but_not_anchor_matched",
|
||||
"limited_reason_category": "empty_match",
|
||||
"llm_decomposition_attempted": true,
|
||||
"llm_decomposition_applied": true,
|
||||
"llm_decomposition_reason": "fallback_rule_preferred_over_llm_candidate_anchor_quality",
|
||||
"llm_canonical_candidate_detected": true,
|
||||
"fallback_rule_hit": "documents_contract_rewrite",
|
||||
"tool_gate_decision": "run_address_lane",
|
||||
"tool_gate_reason": "address_mode_classifier_detected",
|
||||
"predecompose_contract_intent": "list_documents_by_contract",
|
||||
"predecompose_contract_aggregation_profile": "list_lookup",
|
||||
"predecompose_contract_period_scope": "unspecified",
|
||||
"debug_payload": {
|
||||
"trace_id": "address-0bf4dWjq4o",
|
||||
"prompt_version": "address_query_runtime_v1",
|
||||
"schema_version": "address_query_runtime_v1",
|
||||
"fallback_type": "partial",
|
||||
"route_summary": null,
|
||||
"fragments": [],
|
||||
"requirements_extracted": [],
|
||||
"coverage_report": {
|
||||
"requirements_total": 0,
|
||||
"requirements_covered": 0,
|
||||
"requirements_uncovered": [],
|
||||
"requirements_partially_covered": [],
|
||||
"clarification_needed_for": [],
|
||||
"out_of_scope_requirements": []
|
||||
},
|
||||
"routes": [],
|
||||
"retrieval_status": [],
|
||||
"retrieval_results": [],
|
||||
"answer_grounding_check": {
|
||||
"status": "partial",
|
||||
"route_subject_match": true,
|
||||
"missing_requirements": [],
|
||||
"reasons": [
|
||||
"address_entity_detected",
|
||||
"document_list_signal_detected",
|
||||
"documents_by_contract_signal_detected"
|
||||
],
|
||||
"why_included_summary": [],
|
||||
"selection_reason_summary": []
|
||||
},
|
||||
"dropped_intent_segments": [],
|
||||
"detected_mode": "address_query",
|
||||
"detected_mode_confidence": "medium",
|
||||
"query_shape": "DOCUMENT_LIST",
|
||||
"query_shape_confidence": "medium",
|
||||
"detected_intent": "list_documents_by_contract",
|
||||
"detected_intent_confidence": "medium",
|
||||
"extracted_filters": {
|
||||
"sort": "period_desc",
|
||||
"limit": 20,
|
||||
"contract": "овору"
|
||||
},
|
||||
"missing_required_filters": [],
|
||||
"selected_recipe": "address_documents_by_contract_v1",
|
||||
"mcp_call_status_legacy": "materialized_but_not_matched",
|
||||
"account_scope_mode": "preferred",
|
||||
"account_scope_fallback_applied": false,
|
||||
"anchor_type": "contract",
|
||||
"anchor_value_raw": "овору",
|
||||
"anchor_value_resolved": "овору",
|
||||
"resolver_confidence": "medium",
|
||||
"ambiguity_count": 0,
|
||||
"match_failure_stage": "materialized_but_not_anchor_matched",
|
||||
"match_failure_reason": "contract_anchor_not_matched_in_materialized_rows",
|
||||
"mcp_call_status": "materialized_but_not_anchor_matched",
|
||||
"rows_fetched": 1000,
|
||||
"raw_rows_received": 1000,
|
||||
"rows_after_account_scope": 416,
|
||||
"rows_after_recipe_filter": 0,
|
||||
"rows_materialized": 416,
|
||||
"rows_matched": 0,
|
||||
"raw_row_keys_sample": [
|
||||
"Период",
|
||||
"Регистратор",
|
||||
"СчетДт",
|
||||
"СчетКт",
|
||||
"Сумма",
|
||||
"Period",
|
||||
"Registrator",
|
||||
"AccountDt",
|
||||
"AccountKt",
|
||||
"Amount"
|
||||
],
|
||||
"materialization_drop_reason": "none",
|
||||
"account_token_raw": null,
|
||||
"account_token_normalized": null,
|
||||
"account_scope_fields_checked": [
|
||||
"account_dt",
|
||||
"account_kt",
|
||||
"registrator",
|
||||
"analytics"
|
||||
],
|
||||
"account_scope_match_strategy": "account_code_regex_plus_alias_map_v1",
|
||||
"account_scope_drop_reason": "not_applicable",
|
||||
"runtime_readiness": "LIVE_QUERYABLE_WITH_LIMITS",
|
||||
"limited_reason_category": "empty_match",
|
||||
"response_type": "LIMITED_WITH_REASON",
|
||||
"execution_lane": "address_query",
|
||||
"llm_decomposition_applied": true,
|
||||
"llm_decomposition_attempted": true,
|
||||
"llm_provider_used": "local",
|
||||
"llm_decomposition_trace_id": "_rP6NyFCpQFXrO",
|
||||
"llm_decomposition_effective_message": "документы по договору овору",
|
||||
"llm_decomposition_reason": "fallback_rule_preferred_over_llm_candidate_anchor_quality",
|
||||
"llm_canonical_candidate_detected": true,
|
||||
"llm_predecompose_contract": {
|
||||
"schema_version": "address_llm_predecompose_contract_v1",
|
||||
"source_message": "покажи документы по этому же договору",
|
||||
"canonical_message": "документы по договору овору",
|
||||
"mode": "address_query",
|
||||
"mode_confidence": "medium",
|
||||
"query_shape": "DOCUMENT_LIST",
|
||||
"query_shape_confidence": "medium",
|
||||
"intent": "list_documents_by_contract",
|
||||
"intent_confidence": "medium",
|
||||
"entities": {
|
||||
"account": null,
|
||||
"counterparty": null,
|
||||
"contract": "овору",
|
||||
"document_type": null,
|
||||
"document_ref": null,
|
||||
"organization": null
|
||||
},
|
||||
"period": {
|
||||
"scope": "unspecified",
|
||||
"period_from": null,
|
||||
"period_to": null,
|
||||
"as_of_date": null,
|
||||
"has_explicit_period": false
|
||||
},
|
||||
"aggregation_profile": "list_lookup"
|
||||
},
|
||||
"fallback_rule_hit": "documents_contract_rewrite",
|
||||
"sanitized_user_message": "покажи документы по этому же договору",
|
||||
"tool_gate_decision": "run_address_lane",
|
||||
"tool_gate_reason": "address_mode_classifier_detected",
|
||||
"answer_structure_v11": null,
|
||||
"investigation_state_snapshot": null,
|
||||
"normalized": null,
|
||||
"normalizer_output": {
|
||||
"trace_id": "_rP6NyFCpQFXrO",
|
||||
"prompt_version": "normalizer_v2_0_2",
|
||||
"applied": true,
|
||||
"effective_message": "документы по договору овору"
|
||||
}
|
||||
},
|
||||
"error_code": null,
|
||||
"error_message": null
|
||||
},
|
||||
{
|
||||
"index": 25,
|
||||
"id": "F025",
|
||||
"question": "а документы по этому же договору за тот же период",
|
||||
"session": "chain_contract_e",
|
||||
"session_id": "asst-2026-04-03_Address_Nightly_followup_25_09-53-20-chain_contract_e",
|
||||
"status_code": 200,
|
||||
"ok": true,
|
||||
"elapsed_ms": 8312,
|
||||
"reply_type": "partial_coverage",
|
||||
"trace_id": "address-eQuO9SqbSw",
|
||||
"assistant_reply": "В live-данных по текущему фильтру записи не найдены.\nПричина: по указанному якорю и фильтрам в live-выборке нет строк.\nЧто нужно уточнить: уточните период или снимите часть фильтров.",
|
||||
"assistant_reply_first_line": "В live-данных по текущему фильтру записи не найдены.",
|
||||
"expected_intent": "list_documents_by_contract",
|
||||
"actual_intent": "list_documents_by_contract",
|
||||
"intent_match": true,
|
||||
"expected_mode": "address_query",
|
||||
"actual_mode": "address_query",
|
||||
"mode_match": true,
|
||||
"expected_reply_type": null,
|
||||
"reply_match": true,
|
||||
"semantic_pass": true,
|
||||
"route_pass": false,
|
||||
"route_health": "likely_blocked_route",
|
||||
"strict_policy": "route",
|
||||
"strict_pass": false,
|
||||
"selected_recipe": "address_documents_by_contract_v1",
|
||||
"missing_required_filters": [],
|
||||
"match_failure_stage": "materialized_but_not_anchor_matched",
|
||||
"match_failure_reason": "contract_anchor_not_matched_in_materialized_rows",
|
||||
"rows_fetched": 1000,
|
||||
"rows_matched": 0,
|
||||
"mcp_call_status": "materialized_but_not_anchor_matched",
|
||||
"limited_reason_category": "empty_match",
|
||||
"llm_decomposition_attempted": true,
|
||||
"llm_decomposition_applied": true,
|
||||
"llm_decomposition_reason": "normalized_fragment_applied",
|
||||
"llm_canonical_candidate_detected": true,
|
||||
"fallback_rule_hit": null,
|
||||
"tool_gate_decision": "run_address_lane",
|
||||
"tool_gate_reason": "address_mode_classifier_detected",
|
||||
"predecompose_contract_intent": "list_documents_by_contract",
|
||||
"predecompose_contract_aggregation_profile": "list_lookup",
|
||||
"predecompose_contract_period_scope": "unspecified",
|
||||
"debug_payload": {
|
||||
"trace_id": "address-eQuO9SqbSw",
|
||||
"prompt_version": "address_query_runtime_v1",
|
||||
"schema_version": "address_query_runtime_v1",
|
||||
"fallback_type": "partial",
|
||||
"route_summary": null,
|
||||
"fragments": [],
|
||||
"requirements_extracted": [],
|
||||
"coverage_report": {
|
||||
"requirements_total": 0,
|
||||
"requirements_covered": 0,
|
||||
"requirements_uncovered": [],
|
||||
"requirements_partially_covered": [],
|
||||
"clarification_needed_for": [],
|
||||
"out_of_scope_requirements": []
|
||||
},
|
||||
"routes": [],
|
||||
"retrieval_status": [],
|
||||
"retrieval_results": [],
|
||||
"answer_grounding_check": {
|
||||
"status": "partial",
|
||||
"route_subject_match": true,
|
||||
"missing_requirements": [],
|
||||
"reasons": [
|
||||
"address_entity_detected",
|
||||
"document_list_signal_detected",
|
||||
"documents_by_contract_signal_detected"
|
||||
],
|
||||
"why_included_summary": [],
|
||||
"selection_reason_summary": []
|
||||
},
|
||||
"dropped_intent_segments": [],
|
||||
"detected_mode": "address_query",
|
||||
"detected_mode_confidence": "medium",
|
||||
"query_shape": "DOCUMENT_LIST",
|
||||
"query_shape_confidence": "medium",
|
||||
"detected_intent": "list_documents_by_contract",
|
||||
"detected_intent_confidence": "medium",
|
||||
"extracted_filters": {
|
||||
"sort": "period_desc",
|
||||
"limit": 20,
|
||||
"contract": "за указанный период"
|
||||
},
|
||||
"missing_required_filters": [],
|
||||
"selected_recipe": "address_documents_by_contract_v1",
|
||||
"mcp_call_status_legacy": "materialized_but_not_matched",
|
||||
"account_scope_mode": "preferred",
|
||||
"account_scope_fallback_applied": false,
|
||||
"anchor_type": "contract",
|
||||
"anchor_value_raw": "за указанный период",
|
||||
"anchor_value_resolved": "за указанный период",
|
||||
"resolver_confidence": "medium",
|
||||
"ambiguity_count": 0,
|
||||
"match_failure_stage": "materialized_but_not_anchor_matched",
|
||||
"match_failure_reason": "contract_anchor_not_matched_in_materialized_rows",
|
||||
"mcp_call_status": "materialized_but_not_anchor_matched",
|
||||
"rows_fetched": 1000,
|
||||
"raw_rows_received": 1000,
|
||||
"rows_after_account_scope": 416,
|
||||
"rows_after_recipe_filter": 0,
|
||||
"rows_materialized": 416,
|
||||
"rows_matched": 0,
|
||||
"raw_row_keys_sample": [
|
||||
"Период",
|
||||
"Регистратор",
|
||||
"СчетДт",
|
||||
"СчетКт",
|
||||
"Сумма",
|
||||
"Period",
|
||||
"Registrator",
|
||||
"AccountDt",
|
||||
"AccountKt",
|
||||
"Amount"
|
||||
],
|
||||
"materialization_drop_reason": "none",
|
||||
"account_token_raw": null,
|
||||
"account_token_normalized": null,
|
||||
"account_scope_fields_checked": [
|
||||
"account_dt",
|
||||
"account_kt",
|
||||
"registrator",
|
||||
"analytics"
|
||||
],
|
||||
"account_scope_match_strategy": "account_code_regex_plus_alias_map_v1",
|
||||
"account_scope_drop_reason": "not_applicable",
|
||||
"runtime_readiness": "LIVE_QUERYABLE_WITH_LIMITS",
|
||||
"limited_reason_category": "empty_match",
|
||||
"response_type": "LIMITED_WITH_REASON",
|
||||
"execution_lane": "address_query",
|
||||
"llm_decomposition_applied": true,
|
||||
"llm_decomposition_attempted": true,
|
||||
"llm_provider_used": "local",
|
||||
"llm_decomposition_trace_id": "Iyq1kOgsRyIRq9",
|
||||
"llm_decomposition_effective_message": "документы по договору за указанный период",
|
||||
"llm_decomposition_reason": "normalized_fragment_applied",
|
||||
"llm_canonical_candidate_detected": true,
|
||||
"llm_predecompose_contract": {
|
||||
"schema_version": "address_llm_predecompose_contract_v1",
|
||||
"source_message": "а документы по этому же договору за тот же период",
|
||||
"canonical_message": "документы по договору за указанный период",
|
||||
"mode": "address_query",
|
||||
"mode_confidence": "medium",
|
||||
"query_shape": "DOCUMENT_LIST",
|
||||
"query_shape_confidence": "medium",
|
||||
"intent": "list_documents_by_contract",
|
||||
"intent_confidence": "medium",
|
||||
"entities": {
|
||||
"account": null,
|
||||
"counterparty": null,
|
||||
"contract": "за указанный период",
|
||||
"document_type": null,
|
||||
"document_ref": null,
|
||||
"organization": null
|
||||
},
|
||||
"period": {
|
||||
"scope": "unspecified",
|
||||
"period_from": null,
|
||||
"period_to": null,
|
||||
"as_of_date": null,
|
||||
"has_explicit_period": false
|
||||
},
|
||||
"aggregation_profile": "list_lookup"
|
||||
},
|
||||
"fallback_rule_hit": null,
|
||||
"sanitized_user_message": "а документы по этому же договору за тот же период",
|
||||
"tool_gate_decision": "run_address_lane",
|
||||
"tool_gate_reason": "address_mode_classifier_detected",
|
||||
"answer_structure_v11": null,
|
||||
"investigation_state_snapshot": null,
|
||||
"normalized": null,
|
||||
"normalizer_output": {
|
||||
"trace_id": "Iyq1kOgsRyIRq9",
|
||||
"prompt_version": "normalizer_v2_0_2",
|
||||
"applied": true,
|
||||
"effective_message": "документы по договору за указанный период"
|
||||
}
|
||||
},
|
||||
"error_code": null,
|
||||
"error_message": null
|
||||
}
|
||||
]
|
||||
File diff suppressed because it is too large
Load Diff
|
|
@ -0,0 +1,29 @@
|
|||
# Response Audit: 2026-04-03_Address_Nightly_followup_25_09-53-20
|
||||
|
||||
| id | strict | route_health | reply_type | intent | limited_reason | question | assistant_first_line |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| F001 | True | ok_or_factual | factual | list_documents_by_counterparty | None | Покажи документы по СВК за 2020 | Собран список документов по контрагенту (live address lane). |
|
||||
| F002 | True | ok_or_factual | factual | list_documents_by_counterparty | None | а теперь только за май 2020 | По окну 2020-05-01..2020-05-31 строк не найдено; показаны ближайшие доступные данные 2020-07-27..2021-11-10. |
|
||||
| F003 | True | ok_or_factual | factual | account_balance_snapshot | None | а по счету 60.01 на ту же дату | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| F004 | True | ok_or_factual | factual | account_balance_snapshot | None | а на ту же дату по 62.01 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| F005 | True | ok_or_factual | factual | documents_forming_balance | None | раскрой остаток по 62.01 документами на ту же дату | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| F006 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | Покажи банковские операции по СВК за 2020 | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| F007 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | а теперь только ноябрь 2020 | Период сохранен. Глубина live-выборки автоматически расширена до 1000 строк. |
|
||||
| F008 | True | ok_or_factual | factual | list_documents_by_counterparty | None | а теперь документы по нему | Период сохранен. Глубина live-выборки автоматически расширена до 1000 строк. |
|
||||
| F009 | True | ok_or_factual | factual | account_balance_snapshot | None | и остаток по 51 на ту же дату | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| F010 | True | ok_or_factual | factual | account_balance_snapshot | None | а теперь только май 2020 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| F011 | True | partial_non_blocking | partial_coverage | account_balance_snapshot | empty_match | Покажи сальдо по счету 60 на 2020-12-31 | В live-данных по текущему фильтру записи не найдены. |
|
||||
| F012 | True | ok_or_factual | factual | account_balance_snapshot | None | а теперь только май 2020 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| F013 | False | ok_or_factual | factual | documents_forming_balance | None | а на ту же дату по 62 | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| F014 | True | ok_or_factual | factual | documents_forming_balance | None | раскрой 62 документами на ту же дату | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| F015 | True | partial_non_blocking | partial_coverage | account_balance_snapshot | empty_match | вернись на 2020-12-31 по 60 | В live-данных по текущему фильтру записи не найдены. |
|
||||
| F016 | True | ok_or_factual | factual | list_documents_by_contract | None | Покажи документы по договору 19/15 за 2020 | Период сохранен. Глубина live-выборки автоматически расширена до 1000 строк. |
|
||||
| F017 | False | likely_blocked_route | partial_coverage | bank_operations_by_counterparty | empty_match | а теперь банковские операции | В live-данных по текущему фильтру записи не найдены. |
|
||||
| F018 | False | likely_blocked_route | partial_coverage | bank_operations_by_counterparty | empty_match | а теперь только за 2020 | В live-данных по текущему фильтру записи не найдены. |
|
||||
| F019 | True | ok_or_factual | factual | open_items_by_counterparty_or_contract | None | есть ли открытые позиции по договору 19/15 на 2020-12-31 | Период сохранен. Глубина live-выборки автоматически расширена до 1000 строк. |
|
||||
| F020 | False | likely_blocked_route | partial_coverage | list_documents_by_contract | empty_match | покажи документы по этому же договору | В live-данных по текущему фильтру записи не найдены. |
|
||||
| F021 | True | ok_or_factual | factual | bank_operations_by_contract | None | Покажи банковские операции по договору 19/15 | Собран список банковских операций по договору (live address lane). |
|
||||
| F022 | True | ok_or_factual | factual | open_items_by_counterparty_or_contract | None | а теперь открытые позиции по нему | Период сохранен. Глубина live-выборки автоматически расширена до 1000 строк. |
|
||||
| F023 | True | ok_or_factual | factual | open_items_by_counterparty_or_contract | None | а теперь на 2020-12-31 | Период сохранен. Глубина live-выборки автоматически расширена до 1000 строк. |
|
||||
| F024 | True | ok_or_factual | factual | account_balance_snapshot | None | а по счету 51 на ту же дату | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| F025 | False | likely_blocked_route | partial_coverage | list_documents_by_contract | empty_match | а документы по этому же договору за тот же период | В live-данных по текущему фильтру записи не найдены. |
|
||||
|
|
@ -0,0 +1,144 @@
|
|||
{
|
||||
"run_id": "2026-04-03_Address_Nightly_followup_25_09-53-20",
|
||||
"generated_at": "2026-04-03T09:56:07",
|
||||
"source_questions_file": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\question_sets\\address_followup_context_chains_2026-04-02.json",
|
||||
"backend_url": "http://127.0.0.1:8787/api/assistant/message",
|
||||
"llm_provider": "local",
|
||||
"llm_model": "qwen2.5-14b-instruct-1m",
|
||||
"llm_base_url": "http://127.0.0.1:1234",
|
||||
"strict_policy": "route",
|
||||
"totals": {
|
||||
"questions_total": 25,
|
||||
"ok_200_count": 25,
|
||||
"semantic_pass_count": 22,
|
||||
"semantic_pass_rate": 0.88,
|
||||
"route_pass_count": 20,
|
||||
"route_pass_rate": 0.8,
|
||||
"strict_pass_count": 20,
|
||||
"strict_pass_rate": 0.8,
|
||||
"factual_count": 19,
|
||||
"partial_coverage_count": 6,
|
||||
"clarification_required_count": 0,
|
||||
"http_error_count": 0,
|
||||
"llm_decomposition_attempted_count": 25,
|
||||
"llm_decomposition_applied_count": 24,
|
||||
"llm_fallback_count": 0,
|
||||
"llm_fallback_rate": 0.0,
|
||||
"tool_gate_blocked_count": 0,
|
||||
"tool_gate_blocked_rate": 0.0,
|
||||
"avg_elapsed_ms": 6659.5
|
||||
},
|
||||
"distributions": {
|
||||
"reply_type": {
|
||||
"factual": 19,
|
||||
"partial_coverage": 6
|
||||
},
|
||||
"actual_intent": {
|
||||
"list_documents_by_counterparty": 3,
|
||||
"account_balance_snapshot": 8,
|
||||
"documents_forming_balance": 3,
|
||||
"bank_operations_by_counterparty": 4,
|
||||
"list_documents_by_contract": 3,
|
||||
"open_items_by_counterparty_or_contract": 3,
|
||||
"bank_operations_by_contract": 1
|
||||
},
|
||||
"actual_mode": {
|
||||
"address_query": 25
|
||||
},
|
||||
"mcp_call_status": {
|
||||
"matched_non_empty": 19,
|
||||
"no_raw_rows": 2,
|
||||
"materialized_but_not_anchor_matched": 4
|
||||
},
|
||||
"limited_reason_category": {
|
||||
"empty_match": 6
|
||||
},
|
||||
"route_health": {
|
||||
"ok_or_factual": 19,
|
||||
"partial_non_blocking": 2,
|
||||
"likely_blocked_route": 4
|
||||
},
|
||||
"tool_gate_decision": {
|
||||
"run_address_lane": 25
|
||||
},
|
||||
"tool_gate_reason": {
|
||||
"address_mode_classifier_detected": 17,
|
||||
"llm_canonical_candidate_detected": 8
|
||||
}
|
||||
},
|
||||
"address_llm_predecompose_metrics": {
|
||||
"overall": {
|
||||
"llm_attempted": 25,
|
||||
"llm_applied": 24,
|
||||
"fallback_used": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"tool_gate_blocked": 0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"by_intent": {
|
||||
"list_documents_by_counterparty": {
|
||||
"total": 3,
|
||||
"llm_attempted": 3,
|
||||
"llm_applied": 3,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"account_balance_snapshot": {
|
||||
"total": 8,
|
||||
"llm_attempted": 8,
|
||||
"llm_applied": 7,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"documents_forming_balance": {
|
||||
"total": 3,
|
||||
"llm_attempted": 3,
|
||||
"llm_applied": 3,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"bank_operations_by_counterparty": {
|
||||
"total": 4,
|
||||
"llm_attempted": 4,
|
||||
"llm_applied": 4,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"list_documents_by_contract": {
|
||||
"total": 3,
|
||||
"llm_attempted": 3,
|
||||
"llm_applied": 3,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"open_items_by_counterparty_or_contract": {
|
||||
"total": 3,
|
||||
"llm_attempted": 3,
|
||||
"llm_applied": 3,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"bank_operations_by_contract": {
|
||||
"total": 1,
|
||||
"llm_attempted": 1,
|
||||
"llm_applied": 1,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,67 @@
|
|||
{
|
||||
"generated_at": "2026-04-03T09:56:07",
|
||||
"overall_pass": false,
|
||||
"baseline": {
|
||||
"summary_path": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\runs\\2026-04-02_Address_Followup_Context_Chains_2026-04-02_19-15-Run5\\run_summary.json",
|
||||
"run_id": "2026-04-02_Address_Followup_Context_Chains_2026-04-02_19-15-Run5",
|
||||
"questions_total": 25,
|
||||
"strict_pass_rate": 1.0,
|
||||
"route_pass_rate": 1.0,
|
||||
"execution_error_count": 0,
|
||||
"false_factual_rate": 0.0,
|
||||
"notes": [
|
||||
"execution_error_count derived as http_error_count + limited_reason_category.execution_error",
|
||||
"false_factual_rate derived from false_factual_count/questions_total"
|
||||
]
|
||||
},
|
||||
"candidate": {
|
||||
"summary_path": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\runs\\2026-04-03_Address_Nightly_Regression_Post_AnchorHardening\\2026-04-03_Address_Nightly_followup_25_09-53-20\\run_summary.json",
|
||||
"run_id": "2026-04-03_Address_Nightly_followup_25_09-53-20",
|
||||
"questions_total": 25,
|
||||
"strict_pass_rate": 0.8,
|
||||
"route_pass_rate": 0.8,
|
||||
"execution_error_count": 0,
|
||||
"false_factual_rate": 0.0,
|
||||
"notes": [
|
||||
"execution_error_count derived as http_error_count + limited_reason_category.execution_error",
|
||||
"false_factual_rate derived from false_factual_count/questions_total"
|
||||
]
|
||||
},
|
||||
"checks": [
|
||||
{
|
||||
"metric": "strict_pass_rate",
|
||||
"passed": false,
|
||||
"baseline": 1.0,
|
||||
"candidate": 0.8,
|
||||
"rule": "candidate >= baseline"
|
||||
},
|
||||
{
|
||||
"metric": "route_pass_rate",
|
||||
"passed": false,
|
||||
"baseline": 1.0,
|
||||
"candidate": 0.8,
|
||||
"rule": "candidate >= baseline"
|
||||
},
|
||||
{
|
||||
"metric": "execution_error_count",
|
||||
"passed": true,
|
||||
"baseline": 0,
|
||||
"candidate": 0,
|
||||
"rule": "candidate <= baseline"
|
||||
},
|
||||
{
|
||||
"metric": "false_factual_rate",
|
||||
"passed": true,
|
||||
"baseline": 0.0,
|
||||
"candidate": 0.0,
|
||||
"rule": "candidate <= baseline"
|
||||
},
|
||||
{
|
||||
"metric": "questions_total_match",
|
||||
"passed": true,
|
||||
"baseline": 25,
|
||||
"candidate": 25,
|
||||
"rule": "candidate == baseline"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -0,0 +1,23 @@
|
|||
{
|
||||
"generated_at": "2026-04-03T09:56:07",
|
||||
"total": 1,
|
||||
"passed": 1,
|
||||
"failed": 0,
|
||||
"results": [
|
||||
{
|
||||
"run_dir": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\runs\\2026-04-03_Address_Nightly_Regression_Post_AnchorHardening\\2026-04-03_Address_Nightly_followup_25_09-53-20",
|
||||
"valid": true,
|
||||
"errors": [],
|
||||
"warnings": [],
|
||||
"metrics": {
|
||||
"run_id": "2026-04-03_Address_Nightly_followup_25_09-53-20",
|
||||
"questions_total": 25,
|
||||
"route_pass_rate": 0.8,
|
||||
"strict_pass_rate": 0.8,
|
||||
"http_error_count": 0,
|
||||
"rows_count": 25,
|
||||
"failures_count": 5
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -0,0 +1,33 @@
|
|||
# 2026-04-03_Address_Nightly_stress_102_09-42-09
|
||||
|
||||
Generated at: 2026-04-03T09:53:20
|
||||
Questions file: X:\1C\NDC_1C\docs\ADDRESS\question_sets\address_slang_stress_full_2026-04-02.json
|
||||
Backend URL: http://127.0.0.1:8787/api/assistant/message
|
||||
LLM: local / qwen2.5-14b-instruct-1m @ http://127.0.0.1:1234
|
||||
Strict policy: route
|
||||
|
||||
## Totals
|
||||
- questions_total: 102
|
||||
- ok_200_count: 102
|
||||
- semantic_pass_count: 102
|
||||
- semantic_pass_rate: 1.0
|
||||
- route_pass_count: 90
|
||||
- route_pass_rate: 0.8824
|
||||
- strict_pass_count: 90
|
||||
- strict_pass_rate: 0.8824
|
||||
- factual_count: 85
|
||||
- partial_coverage_count: 17
|
||||
- clarification_required_count: 0
|
||||
- http_error_count: 0
|
||||
- llm_decomposition_attempted_count: 102
|
||||
- llm_decomposition_applied_count: 87
|
||||
- llm_fallback_count: 0
|
||||
- llm_fallback_rate: 0.0
|
||||
- tool_gate_blocked_count: 0
|
||||
- tool_gate_blocked_rate: 0.0
|
||||
- avg_elapsed_ms: 6577.5
|
||||
|
||||
## Files
|
||||
- run_summary.json
|
||||
- full_live_results.json
|
||||
- failures_only.json
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
|
|
@ -0,0 +1,106 @@
|
|||
# Response Audit: 2026-04-03_Address_Nightly_stress_102_09-42-09
|
||||
|
||||
| id | strict | route_health | reply_type | intent | limited_reason | question | assistant_first_line |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| S001 | False | likely_blocked_route | partial_coverage | list_documents_by_counterparty | empty_match | свк доки за 2020 покаж пж | В live-данных по текущему фильтру записи не найдены. |
|
||||
| S002 | True | ok_or_factual | factual | list_documents_by_counterparty | None | покеж все доки по свк за 20 год | Собран список документов по контрагенту (live address lane). |
|
||||
| S003 | True | ok_or_factual | factual | list_documents_by_counterparty | None | дай список документов по свк за 2020 | Собран список документов по контрагенту (live address lane). |
|
||||
| S004 | True | ok_or_factual | factual | list_documents_by_counterparty | None | какие у свк есть доки за 2020? | Собран список документов по контрагенту (live address lane). |
|
||||
| S005 | True | ok_or_factual | factual | list_documents_by_counterparty | None | че по свк по докам за 2020 год | Собран список документов по контрагенту (live address lane). |
|
||||
| S006 | True | ok_or_factual | factual | list_documents_by_counterparty | None | ёпт покажи доки по свк за 2020 | Собран список документов по контрагенту (live address lane). |
|
||||
| S007 | True | ok_or_factual | factual | list_documents_by_counterparty | None | бля, доки свк 2020 вывали | Собран список документов по контрагенту (live address lane). |
|
||||
| S008 | True | ok_or_factual | factual | list_documents_by_counterparty | None | svk docs for 2020 please | Собран список документов по контрагенту (live address lane). |
|
||||
| S009 | False | likely_blocked_route | partial_coverage | list_documents_by_counterparty | empty_match | docs by counterparty svk year 2020 | В live-данных по текущему фильтру записи не найдены. |
|
||||
| S010 | True | ok_or_factual | factual | list_documents_by_counterparty | None | по контрагенту свк покаж документы за 2020 | Собран список документов по контрагенту (live address lane). |
|
||||
| S011 | False | likely_blocked_route | partial_coverage | list_documents_by_counterparty | empty_match | плс доки по свк 2020 | В live-данных по текущему фильтру записи не найдены. |
|
||||
| S012 | True | ok_or_factual | factual | list_documents_by_counterparty | None | свк док за 2020 гони | Собран список документов по контрагенту (live address lane). |
|
||||
| S013 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | по свк банковские операции за 2020 покаж | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| S014 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | свк выписка банка 2020 | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| S015 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | платежи по свк за 2020 выведи | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| S016 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | оплаты по свк за 2020 | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| S017 | False | likely_blocked_route | partial_coverage | bank_operations_by_counterparty | empty_match | свк банк опер за 2020 | В live-данных по текущему фильтру записи не найдены. |
|
||||
| S018 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | че у свк по банку за 2020 | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| S019 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | свк списания/поступления за 2020 | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| S020 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | bank ops by svk for 2020 | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| S021 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | transactions by counterparty svk 2020 | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| S022 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | бля, по свк банковские платежки 2020 | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| S023 | True | ok_or_factual | factual | account_balance_snapshot | None | сальдо по счету 60.01 на дату 2020-07-31 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| S024 | True | ok_or_factual | factual | account_balance_snapshot | None | остаток по счету 60.01 на 2020.05 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| S025 | True | ok_or_factual | factual | account_balance_snapshot | None | покажи остаток по 60.01 за май 2020 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| S026 | True | ok_or_factual | factual | account_balance_snapshot | None | счет 60 остаток на май 2020 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| S027 | True | ok_or_factual | factual | account_balance_snapshot | None | баланс account 60.01 as of 2020-07-31 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| S028 | True | ok_or_factual | factual | account_balance_snapshot | None | 60.01 saldo на 31.07.2020 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| S029 | True | ok_or_factual | factual | account_balance_snapshot | None | сколько на счете 60 на конец 2020-05 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| S030 | True | ok_or_factual | factual | account_balance_snapshot | None | дай остаток 60.02 на 2020-12-31 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| S031 | True | partial_non_blocking | partial_coverage | account_balance_snapshot | empty_match | епт, сальдо по счету 62.01 на 2020-12-31 | В live-данных по текущему фильтру записи не найдены. |
|
||||
| S032 | True | ok_or_factual | factual | account_balance_snapshot | None | плс остаток по счету 51 на 2020-06-30 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| S033 | True | ok_or_factual | factual | account_balance_snapshot | None | как там по счету 60.01 на 2020.07? | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| S034 | True | ok_or_factual | factual | account_balance_snapshot | None | сальдо 60 на 2020 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| S035 | True | ok_or_factual | factual | documents_forming_balance | None | какие документы формируют остаток по счету 60.01 на 2020-07-31 | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| S036 | True | ok_or_factual | factual | documents_forming_balance | None | из чего состоит остаток по счету 60 на 2020.05 | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| S037 | True | ok_or_factual | factual | documents_forming_balance | None | раскрой остаток 60.01 по документам на конец июля 2020 | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| S038 | True | ok_or_factual | factual | documents_forming_balance | None | доки под остатком 60.01 на 2020-07-31 | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| S039 | True | ok_or_factual | factual | documents_forming_balance | None | documents forming balance for account 60.01 as of 2020-07-31 | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| S040 | True | ok_or_factual | factual | documents_forming_balance | None | бля, какие доки формируют остаток 60.01 на 2020-05-31 | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| S041 | True | partial_non_blocking | partial_coverage | documents_forming_balance | empty_match | по счету 62.01 раскрой остаток документами на 2020-12-31 | В live-данных по текущему фильтру записи не найдены. |
|
||||
| S042 | True | ok_or_factual | factual | documents_forming_balance | None | из чего остаток по счету 51 на 2020-06-30 | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| S043 | True | ok_or_factual | factual | list_documents_by_contract | None | покажи документы по договору 19/15 за 2020 | Период сохранен. Глубина live-выборки автоматически расширена до 1000 строк. |
|
||||
| S044 | True | ok_or_factual | factual | list_documents_by_contract | None | доки по дог. 19/15 за 2020 | Период сохранен. Глубина live-выборки автоматически расширена до 1000 строк. |
|
||||
| S045 | True | ok_or_factual | factual | list_documents_by_contract | None | какие документы по контракту 19/15 | Собран список документов по договору (live address lane). |
|
||||
| S046 | True | ok_or_factual | factual | list_documents_by_contract | None | docs by contract 19/15 year 2020 | Период сохранен. Глубина live-выборки автоматически расширена до 1000 строк. |
|
||||
| S047 | True | ok_or_factual | factual | list_documents_by_contract | None | бля, покажи по договору 19/15 доки | Собран список документов по договору (live address lane). |
|
||||
| S048 | True | ok_or_factual | factual | list_documents_by_contract | None | договор 19/15 документы | Собран список документов по договору (live address lane). |
|
||||
| S049 | False | likely_blocked_route | partial_coverage | list_documents_by_contract | empty_match | плс документы договор №19/15 | В live-данных по текущему фильтру записи не найдены. |
|
||||
| S050 | True | ok_or_factual | factual | list_documents_by_contract | None | покеж документы по 19/15 | Собран список документов по договору (live address lane). |
|
||||
| S051 | True | ok_or_factual | factual | bank_operations_by_contract | None | банковские операции по договору 19/15 | Собран список банковских операций по договору (live address lane). |
|
||||
| S052 | True | ok_or_factual | factual | bank_operations_by_contract | None | платежи по договору 19/15 | Собран список банковских операций по договору (live address lane). |
|
||||
| S053 | True | ok_or_factual | factual | bank_operations_by_contract | None | выписка по договору 19/15 | Собран список банковских операций по договору (live address lane). |
|
||||
| S054 | True | ok_or_factual | factual | bank_operations_by_contract | None | bank ops by contract 19/15 | Собран список банковских операций по договору (live address lane). |
|
||||
| S055 | True | ok_or_factual | factual | bank_operations_by_contract | None | ёпт, покажи банк операции по дог 19/15 | Собран список банковских операций по договору (live address lane). |
|
||||
| S056 | True | ok_or_factual | factual | bank_operations_by_contract | None | по контракту 19/15 платежки | Собран список банковских операций по договору (live address lane). |
|
||||
| S057 | True | ok_or_factual | factual | account_balance_snapshot | None | покажи сальдо по счету 60 за 2020 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| S058 | True | ok_or_factual | factual | account_balance_snapshot | None | а теперь только май 2020 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| S059 | True | ok_or_factual | factual | account_balance_snapshot | None | ок, теперь 2020.05 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| S060 | True | ok_or_factual | factual | account_balance_snapshot | None | теперь на 2020-07-31 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| X001 | False | likely_blocked_route | partial_coverage | list_documents_by_counterparty | empty_match | свк доки кинь за 2020 | В live-данных по текущему фильтру записи не найдены. |
|
||||
| X002 | True | ok_or_factual | factual | list_documents_by_counterparty | None | по свк все документы за 20й год | Собран список документов по контрагенту (live address lane). |
|
||||
| X003 | True | ok_or_factual | factual | list_documents_by_counterparty | None | пж вывали доки контрагент свк 2020 | Собран список документов по контрагенту (live address lane). |
|
||||
| X004 | False | likely_blocked_route | partial_coverage | list_documents_by_counterparty | empty_match | svk poka docy za 2020 | В live-данных по текущему фильтру записи не найдены. |
|
||||
| X005 | True | ok_or_factual | factual | list_documents_by_counterparty | None | какие доки по компании свк за 2020 | Собран список документов по контрагенту (live address lane). |
|
||||
| X006 | False | likely_blocked_route | partial_coverage | list_documents_by_counterparty | empty_match | бля покаж свк документы 2020 | В live-данных по текущему фильтру записи не найдены. |
|
||||
| X007 | False | likely_blocked_route | partial_coverage | bank_operations_by_counterparty | empty_match | свк банк операции за 2020 | В live-данных по текущему фильтру записи не найдены. |
|
||||
| X008 | False | likely_blocked_route | partial_coverage | bank_operations_by_counterparty | empty_match | по свк выписку банка за 2020 выведи | В live-данных по текущему фильтру записи не найдены. |
|
||||
| X009 | False | likely_blocked_route | partial_coverage | bank_operations_by_counterparty | empty_match | свк платежки за 2020 | В live-данных по текущему фильтру записи не найдены. |
|
||||
| X010 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | bank ops svk 2020 | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| X011 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | транзакции по свк за 2020 | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| X012 | False | blocked_missing_anchor | partial_coverage | bank_operations_by_counterparty | missing_anchor | ёпт по свк че по банку 2020 | Для точного адресного поиска не хватает обязательного якоря. |
|
||||
| X013 | True | ok_or_factual | factual | account_balance_snapshot | None | остаток по 60.01 на 2020-07-31 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| X014 | True | ok_or_factual | factual | account_balance_snapshot | None | сальдо 60.01 на 31.07.2020 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| X015 | True | ok_or_factual | factual | account_balance_snapshot | None | что на счете 60 на 2020.05 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| X016 | True | ok_or_factual | factual | account_balance_snapshot | None | дай остаток 62.01 на 2020 май | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| X017 | True | partial_non_blocking | partial_coverage | account_balance_snapshot | empty_match | saldo account 60 as of 2020-12-31 | В live-данных по текущему фильтру записи не найдены. |
|
||||
| X018 | True | ok_or_factual | factual | account_balance_snapshot | None | плиз остаток по счету 51 на 2020-06-30 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| X019 | True | partial_non_blocking | partial_coverage | account_balance_snapshot | empty_match | скока по 60.02 на конец 2020-12 | В live-данных по текущему фильтру записи не найдены. |
|
||||
| X020 | True | ok_or_factual | factual | account_balance_snapshot | None | бля сальдо 60.01 на 2020-05-31 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| X021 | True | ok_or_factual | factual | documents_forming_balance | None | раскрой остаток 60.01 по документам на конец июля 2020 | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| X022 | True | ok_or_factual | factual | documents_forming_balance | None | доки под остатком 60.01 на 2020-07-31 | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| X023 | True | ok_or_factual | factual | documents_forming_balance | None | из чего остаток 60.01 на 2020-07-31 | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| X024 | True | ok_or_factual | factual | documents_forming_balance | None | какие документы формируют остаток по 60.01 на 2020-07-31 | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| X025 | True | ok_or_factual | factual | documents_forming_balance | None | docs forming balance 60.01 as of 2020-07-31 | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| X026 | True | partial_non_blocking | partial_coverage | documents_forming_balance | empty_match | раскидай остаток 62.01 по докам на 2020-12-31 | В live-данных по текущему фильтру записи не найдены. |
|
||||
| X027 | True | ok_or_factual | factual | documents_forming_balance | None | по счету 51 раскрой остаток документами на 2020-06-30 | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| X028 | True | ok_or_factual | factual | documents_forming_balance | None | бля из чего состоит остаток по 60.01 на 2020-05-31 | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| X029 | True | ok_or_factual | factual | list_documents_by_contract | None | доки по договору 19/15 за 2020 | Период сохранен. Глубина live-выборки автоматически расширена до 1000 строк. |
|
||||
| X030 | True | ok_or_factual | factual | list_documents_by_contract | None | покажи документы контракт 19/15 | Собран список документов по договору (live address lane). |
|
||||
| X031 | True | ok_or_factual | factual | list_documents_by_contract | None | docs by contract 19/15 2020 | Период сохранен. Глубина live-выборки автоматически расширена до 1000 строк. |
|
||||
| X032 | True | ok_or_factual | factual | list_documents_by_contract | None | по дог 19/15 какие документы | Собран список документов по договору (live address lane). |
|
||||
| X033 | True | ok_or_factual | factual | list_documents_by_contract | None | договор 19/15 выведи доки | Собран список документов по договору (live address lane). |
|
||||
| X034 | True | ok_or_factual | factual | list_documents_by_contract | None | нах покажи документы по 19/15 | Собран список документов по договору (live address lane). |
|
||||
| X035 | True | ok_or_factual | factual | bank_operations_by_contract | None | банк операции по договору 19/15 | Собран список банковских операций по договору (live address lane). |
|
||||
| X036 | True | ok_or_factual | factual | bank_operations_by_contract | None | платежки по дог 19/15 | Собран список банковских операций по договору (live address lane). |
|
||||
| X037 | True | ok_or_factual | factual | bank_operations_by_contract | None | выписка по контракту 19/15 | Собран список банковских операций по договору (live address lane). |
|
||||
| X038 | True | ok_or_factual | factual | bank_operations_by_contract | None | bank ops by contract 19/15 | Собран список банковских операций по договору (live address lane). |
|
||||
| X039 | True | ok_or_factual | factual | bank_operations_by_contract | None | покажи банк опер по дог 19/15 пж | Собран список банковских операций по договору (live address lane). |
|
||||
| X040 | True | ok_or_factual | factual | bank_operations_by_contract | None | епт транзакции договор 19/15 | Собран список банковских операций по договору (live address lane). |
|
||||
| X041 | True | ok_or_factual | factual | account_balance_snapshot | None | покажи сальдо по счету 60 за 2020 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| X042 | True | ok_or_factual | factual | account_balance_snapshot | None | а теперь только май 2020 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
|
|
@ -0,0 +1,136 @@
|
|||
{
|
||||
"run_id": "2026-04-03_Address_Nightly_stress_102_09-42-09",
|
||||
"generated_at": "2026-04-03T09:53:20",
|
||||
"source_questions_file": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\question_sets\\address_slang_stress_full_2026-04-02.json",
|
||||
"backend_url": "http://127.0.0.1:8787/api/assistant/message",
|
||||
"llm_provider": "local",
|
||||
"llm_model": "qwen2.5-14b-instruct-1m",
|
||||
"llm_base_url": "http://127.0.0.1:1234",
|
||||
"strict_policy": "route",
|
||||
"totals": {
|
||||
"questions_total": 102,
|
||||
"ok_200_count": 102,
|
||||
"semantic_pass_count": 102,
|
||||
"semantic_pass_rate": 1.0,
|
||||
"route_pass_count": 90,
|
||||
"route_pass_rate": 0.8824,
|
||||
"strict_pass_count": 90,
|
||||
"strict_pass_rate": 0.8824,
|
||||
"factual_count": 85,
|
||||
"partial_coverage_count": 17,
|
||||
"clarification_required_count": 0,
|
||||
"http_error_count": 0,
|
||||
"llm_decomposition_attempted_count": 102,
|
||||
"llm_decomposition_applied_count": 87,
|
||||
"llm_fallback_count": 0,
|
||||
"llm_fallback_rate": 0.0,
|
||||
"tool_gate_blocked_count": 0,
|
||||
"tool_gate_blocked_rate": 0.0,
|
||||
"avg_elapsed_ms": 6577.5
|
||||
},
|
||||
"distributions": {
|
||||
"reply_type": {
|
||||
"partial_coverage": 17,
|
||||
"factual": 85
|
||||
},
|
||||
"actual_intent": {
|
||||
"list_documents_by_counterparty": 18,
|
||||
"bank_operations_by_counterparty": 16,
|
||||
"account_balance_snapshot": 26,
|
||||
"documents_forming_balance": 16,
|
||||
"list_documents_by_contract": 14,
|
||||
"bank_operations_by_contract": 12
|
||||
},
|
||||
"actual_mode": {
|
||||
"address_query": 102
|
||||
},
|
||||
"mcp_call_status": {
|
||||
"materialized_but_not_anchor_matched": 12,
|
||||
"matched_non_empty": 85,
|
||||
"no_raw_rows": 5
|
||||
},
|
||||
"limited_reason_category": {
|
||||
"empty_match": 16,
|
||||
"missing_anchor": 1
|
||||
},
|
||||
"route_health": {
|
||||
"likely_blocked_route": 11,
|
||||
"ok_or_factual": 85,
|
||||
"partial_non_blocking": 5,
|
||||
"blocked_missing_anchor": 1
|
||||
},
|
||||
"tool_gate_decision": {
|
||||
"run_address_lane": 102
|
||||
},
|
||||
"tool_gate_reason": {
|
||||
"address_mode_classifier_detected": 98,
|
||||
"llm_canonical_candidate_detected": 4
|
||||
}
|
||||
},
|
||||
"address_llm_predecompose_metrics": {
|
||||
"overall": {
|
||||
"llm_attempted": 102,
|
||||
"llm_applied": 87,
|
||||
"fallback_used": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"tool_gate_blocked": 0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"by_intent": {
|
||||
"list_documents_by_counterparty": {
|
||||
"total": 18,
|
||||
"llm_attempted": 18,
|
||||
"llm_applied": 17,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"bank_operations_by_counterparty": {
|
||||
"total": 16,
|
||||
"llm_attempted": 16,
|
||||
"llm_applied": 16,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"account_balance_snapshot": {
|
||||
"total": 26,
|
||||
"llm_attempted": 26,
|
||||
"llm_applied": 25,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"documents_forming_balance": {
|
||||
"total": 16,
|
||||
"llm_attempted": 16,
|
||||
"llm_applied": 11,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"list_documents_by_contract": {
|
||||
"total": 14,
|
||||
"llm_attempted": 14,
|
||||
"llm_applied": 10,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"bank_operations_by_contract": {
|
||||
"total": 12,
|
||||
"llm_attempted": 12,
|
||||
"llm_applied": 8,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,67 @@
|
|||
{
|
||||
"generated_at": "2026-04-03T09:53:20",
|
||||
"overall_pass": false,
|
||||
"baseline": {
|
||||
"summary_path": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\runs\\2026-04-02_Address_Slang_Live_Stress_2026-04-02_12-57-27\\run_summary.json",
|
||||
"run_id": "2026-04-02_Address_Slang_Live_Stress_2026-04-02_12-57-27",
|
||||
"questions_total": 102,
|
||||
"strict_pass_rate": 1.0,
|
||||
"route_pass_rate": 1.0,
|
||||
"execution_error_count": 0,
|
||||
"false_factual_rate": 0.0,
|
||||
"notes": [
|
||||
"execution_error_count derived as http_error_count + limited_reason_category.execution_error",
|
||||
"false_factual_rate derived from false_factual_count/questions_total"
|
||||
]
|
||||
},
|
||||
"candidate": {
|
||||
"summary_path": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\runs\\2026-04-03_Address_Nightly_Regression_Post_AnchorHardening\\2026-04-03_Address_Nightly_stress_102_09-42-09\\run_summary.json",
|
||||
"run_id": "2026-04-03_Address_Nightly_stress_102_09-42-09",
|
||||
"questions_total": 102,
|
||||
"strict_pass_rate": 0.8824,
|
||||
"route_pass_rate": 0.8824,
|
||||
"execution_error_count": 0,
|
||||
"false_factual_rate": 0.0,
|
||||
"notes": [
|
||||
"execution_error_count derived as http_error_count + limited_reason_category.execution_error",
|
||||
"false_factual_rate derived from false_factual_count/questions_total"
|
||||
]
|
||||
},
|
||||
"checks": [
|
||||
{
|
||||
"metric": "strict_pass_rate",
|
||||
"passed": false,
|
||||
"baseline": 1.0,
|
||||
"candidate": 0.8824,
|
||||
"rule": "candidate >= baseline"
|
||||
},
|
||||
{
|
||||
"metric": "route_pass_rate",
|
||||
"passed": false,
|
||||
"baseline": 1.0,
|
||||
"candidate": 0.8824,
|
||||
"rule": "candidate >= baseline"
|
||||
},
|
||||
{
|
||||
"metric": "execution_error_count",
|
||||
"passed": true,
|
||||
"baseline": 0,
|
||||
"candidate": 0,
|
||||
"rule": "candidate <= baseline"
|
||||
},
|
||||
{
|
||||
"metric": "false_factual_rate",
|
||||
"passed": true,
|
||||
"baseline": 0.0,
|
||||
"candidate": 0.0,
|
||||
"rule": "candidate <= baseline"
|
||||
},
|
||||
{
|
||||
"metric": "questions_total_match",
|
||||
"passed": true,
|
||||
"baseline": 102,
|
||||
"candidate": 102,
|
||||
"rule": "candidate == baseline"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -0,0 +1,23 @@
|
|||
{
|
||||
"generated_at": "2026-04-03T09:53:20",
|
||||
"total": 1,
|
||||
"passed": 1,
|
||||
"failed": 0,
|
||||
"results": [
|
||||
{
|
||||
"run_dir": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\runs\\2026-04-03_Address_Nightly_Regression_Post_AnchorHardening\\2026-04-03_Address_Nightly_stress_102_09-42-09",
|
||||
"valid": true,
|
||||
"errors": [],
|
||||
"warnings": [],
|
||||
"metrics": {
|
||||
"run_id": "2026-04-03_Address_Nightly_stress_102_09-42-09",
|
||||
"questions_total": 102,
|
||||
"route_pass_rate": 0.8824,
|
||||
"strict_pass_rate": 0.8824,
|
||||
"http_error_count": 0,
|
||||
"rows_count": 102,
|
||||
"failures_count": 12
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -0,0 +1,26 @@
|
|||
# 2026-04-03_Address_Nightly_Regression_Post_AnchorHardening
|
||||
|
||||
Generated at: 2026-04-03T09:56:07
|
||||
Dry run: False
|
||||
Strict policy: route
|
||||
Overall: FAIL
|
||||
|
||||
## Packs
|
||||
### stress_102
|
||||
- run_id: 2026-04-03_Address_Nightly_stress_102_09-42-09
|
||||
- runner_ok: True
|
||||
- validator_ok: True
|
||||
- comparator_ok: False
|
||||
- run_dir: X:\1C\NDC_1C\docs\ADDRESS\runs\2026-04-03_Address_Nightly_Regression_Post_AnchorHardening\2026-04-03_Address_Nightly_stress_102_09-42-09
|
||||
- errors:
|
||||
- comparator failed with exit code 1
|
||||
|
||||
### followup_25
|
||||
- run_id: 2026-04-03_Address_Nightly_followup_25_09-53-20
|
||||
- runner_ok: True
|
||||
- validator_ok: True
|
||||
- comparator_ok: False
|
||||
- run_dir: X:\1C\NDC_1C\docs\ADDRESS\runs\2026-04-03_Address_Nightly_Regression_Post_AnchorHardening\2026-04-03_Address_Nightly_followup_25_09-53-20
|
||||
- errors:
|
||||
- comparator failed with exit code 1
|
||||
|
||||
|
|
@ -0,0 +1,37 @@
|
|||
{
|
||||
"nightly_run_id": "2026-04-03_Address_Nightly_Regression_Post_AnchorHardening",
|
||||
"generated_at": "2026-04-03T09:56:07",
|
||||
"dry_run": false,
|
||||
"overall_ok": false,
|
||||
"strict_policy": "route",
|
||||
"packs": [
|
||||
{
|
||||
"pack": "stress_102",
|
||||
"run_id": "2026-04-03_Address_Nightly_stress_102_09-42-09",
|
||||
"questions_file": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\question_sets\\address_slang_stress_full_2026-04-02.json",
|
||||
"baseline_summary": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\runs\\2026-04-02_Address_Slang_Live_Stress_2026-04-02_12-57-27\\run_summary.json",
|
||||
"run_dir": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\runs\\2026-04-03_Address_Nightly_Regression_Post_AnchorHardening\\2026-04-03_Address_Nightly_stress_102_09-42-09",
|
||||
"runner_ok": true,
|
||||
"validator_ok": true,
|
||||
"comparator_ok": false,
|
||||
"errors": [
|
||||
"comparator failed with exit code 1"
|
||||
],
|
||||
"validator_report": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\runs\\2026-04-03_Address_Nightly_Regression_Post_AnchorHardening\\2026-04-03_Address_Nightly_stress_102_09-42-09_validator_report.json"
|
||||
},
|
||||
{
|
||||
"pack": "followup_25",
|
||||
"run_id": "2026-04-03_Address_Nightly_followup_25_09-53-20",
|
||||
"questions_file": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\question_sets\\address_followup_context_chains_2026-04-02.json",
|
||||
"baseline_summary": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\runs\\2026-04-02_Address_Followup_Context_Chains_2026-04-02_19-15-Run5\\run_summary.json",
|
||||
"run_dir": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\runs\\2026-04-03_Address_Nightly_Regression_Post_AnchorHardening\\2026-04-03_Address_Nightly_followup_25_09-53-20",
|
||||
"runner_ok": true,
|
||||
"validator_ok": true,
|
||||
"comparator_ok": false,
|
||||
"errors": [
|
||||
"comparator failed with exit code 1"
|
||||
],
|
||||
"validator_report": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\runs\\2026-04-03_Address_Nightly_Regression_Post_AnchorHardening\\2026-04-03_Address_Nightly_followup_25_09-53-20_validator_report.json"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -0,0 +1,33 @@
|
|||
# 2026-04-03_Address_Nightly_followup_25_10-11-30
|
||||
|
||||
Generated at: 2026-04-03T10:14:11
|
||||
Questions file: X:\1C\NDC_1C\docs\ADDRESS\question_sets\address_followup_context_chains_2026-04-02.json
|
||||
Backend URL: http://127.0.0.1:8787/api/assistant/message
|
||||
LLM: local / qwen2.5-14b-instruct-1m @ http://127.0.0.1:1234
|
||||
Strict policy: route
|
||||
|
||||
## Totals
|
||||
- questions_total: 25
|
||||
- ok_200_count: 25
|
||||
- semantic_pass_count: 24
|
||||
- semantic_pass_rate: 0.96
|
||||
- route_pass_count: 23
|
||||
- route_pass_rate: 0.92
|
||||
- strict_pass_count: 23
|
||||
- strict_pass_rate: 0.92
|
||||
- factual_count: 22
|
||||
- partial_coverage_count: 3
|
||||
- clarification_required_count: 0
|
||||
- http_error_count: 0
|
||||
- llm_decomposition_attempted_count: 25
|
||||
- llm_decomposition_applied_count: 23
|
||||
- llm_fallback_count: 0
|
||||
- llm_fallback_rate: 0.0
|
||||
- tool_gate_blocked_count: 0
|
||||
- tool_gate_blocked_rate: 0.0
|
||||
- avg_elapsed_ms: 6446.1
|
||||
|
||||
## Files
|
||||
- run_summary.json
|
||||
- full_live_results.json
|
||||
- failures_only.json
|
||||
|
|
@ -0,0 +1,369 @@
|
|||
[
|
||||
{
|
||||
"index": 13,
|
||||
"id": "F013",
|
||||
"question": "а на ту же дату по 62",
|
||||
"session": "chain_balance_c",
|
||||
"session_id": "asst-2026-04-03_Address_Nightly_followup_25_10-11-30-chain_balance_c",
|
||||
"status_code": 200,
|
||||
"ok": true,
|
||||
"elapsed_ms": 6097,
|
||||
"reply_type": "factual",
|
||||
"trace_id": "address-onSyvgHwCd",
|
||||
"assistant_reply": "Собран drilldown документов, формирующих остаток по счету на указанную дату.\nДокументных строк отобрано: 4.\nСумма по отобранным движениям: 2545546.24.\n1. 2020-05-21T12:00:00Z | Поступление на расчетный счет 00000000023 от 21.05.2020 12:00:00 | 51 / 62.02 | 250546.24\n2. 2020-05-18T12:00:01Z | Реализация товаров и услуг 00000000002 от 18.05.2020 12:00:01 | 62.01 / 90.01.1 | 765000\n3. 2020-05-18T12:00:01Z | Реализация товаров и услуг 00000000002 от 18.05.2020 12:00:01 | 62.02 / 62.01 | 765000\n4. 2020-05-15T12:00:02Z | Поступление на расчетный счет 00000000022 от 15.05.2020 12:00:02 | 51 / 62.02 | 765000\nМожно уточнить выборку по контрагенту, договору или периоду.",
|
||||
"assistant_reply_first_line": "Собран drilldown документов, формирующих остаток по счету на указанную дату.",
|
||||
"expected_intent": "account_balance_snapshot",
|
||||
"actual_intent": "documents_forming_balance",
|
||||
"intent_match": false,
|
||||
"expected_mode": "address_query",
|
||||
"actual_mode": "address_query",
|
||||
"mode_match": true,
|
||||
"expected_reply_type": null,
|
||||
"reply_match": true,
|
||||
"semantic_pass": false,
|
||||
"route_pass": false,
|
||||
"route_health": "ok_or_factual",
|
||||
"strict_policy": "route",
|
||||
"strict_pass": false,
|
||||
"selected_recipe": "address_documents_forming_balance_v1",
|
||||
"missing_required_filters": [],
|
||||
"match_failure_stage": "none",
|
||||
"match_failure_reason": null,
|
||||
"rows_fetched": 4,
|
||||
"rows_matched": 4,
|
||||
"mcp_call_status": "matched_non_empty",
|
||||
"limited_reason_category": null,
|
||||
"llm_decomposition_attempted": true,
|
||||
"llm_decomposition_applied": true,
|
||||
"llm_decomposition_reason": "normalized_fragment_applied",
|
||||
"llm_canonical_candidate_detected": true,
|
||||
"fallback_rule_hit": null,
|
||||
"tool_gate_decision": "run_address_lane",
|
||||
"tool_gate_reason": "address_mode_classifier_detected",
|
||||
"predecompose_contract_intent": "documents_forming_balance",
|
||||
"predecompose_contract_aggregation_profile": "balance_snapshot",
|
||||
"predecompose_contract_period_scope": "as_of",
|
||||
"debug_payload": {
|
||||
"trace_id": "address-onSyvgHwCd",
|
||||
"prompt_version": "address_query_runtime_v1",
|
||||
"schema_version": "address_query_runtime_v1",
|
||||
"fallback_type": "none",
|
||||
"route_summary": null,
|
||||
"fragments": [],
|
||||
"requirements_extracted": [],
|
||||
"coverage_report": {
|
||||
"requirements_total": 0,
|
||||
"requirements_covered": 0,
|
||||
"requirements_uncovered": [],
|
||||
"requirements_partially_covered": [],
|
||||
"clarification_needed_for": [],
|
||||
"out_of_scope_requirements": []
|
||||
},
|
||||
"routes": [],
|
||||
"retrieval_status": [],
|
||||
"retrieval_results": [],
|
||||
"answer_grounding_check": {
|
||||
"status": "grounded",
|
||||
"route_subject_match": true,
|
||||
"missing_requirements": [],
|
||||
"reasons": [
|
||||
"address_entity_detected",
|
||||
"document_list_signal_detected",
|
||||
"documents_by_account_drilldown_signal_detected",
|
||||
"as_of_date_from_followup_context",
|
||||
"period_from_followup_context",
|
||||
"address_followup_context_applied"
|
||||
],
|
||||
"why_included_summary": [],
|
||||
"selection_reason_summary": []
|
||||
},
|
||||
"dropped_intent_segments": [],
|
||||
"detected_mode": "address_query",
|
||||
"detected_mode_confidence": "medium",
|
||||
"query_shape": "DOCUMENT_LIST",
|
||||
"query_shape_confidence": "medium",
|
||||
"detected_intent": "documents_forming_balance",
|
||||
"detected_intent_confidence": "medium",
|
||||
"extracted_filters": {
|
||||
"sort": "period_desc",
|
||||
"limit": 20,
|
||||
"account": "62",
|
||||
"as_of_date": "2020-05-31",
|
||||
"period_from": "2020-05-01",
|
||||
"period_to": "2020-05-31"
|
||||
},
|
||||
"missing_required_filters": [],
|
||||
"selected_recipe": "address_documents_forming_balance_v1",
|
||||
"mcp_call_status_legacy": "matched_non_empty",
|
||||
"account_scope_mode": "strict",
|
||||
"account_scope_fallback_applied": false,
|
||||
"anchor_type": "account",
|
||||
"anchor_value_raw": "62",
|
||||
"anchor_value_resolved": "62",
|
||||
"resolver_confidence": "high",
|
||||
"ambiguity_count": 0,
|
||||
"match_failure_stage": "none",
|
||||
"match_failure_reason": null,
|
||||
"mcp_call_status": "matched_non_empty",
|
||||
"rows_fetched": 4,
|
||||
"raw_rows_received": 4,
|
||||
"rows_after_account_scope": 4,
|
||||
"rows_after_recipe_filter": 4,
|
||||
"rows_materialized": 4,
|
||||
"rows_matched": 4,
|
||||
"raw_row_keys_sample": [
|
||||
"Период",
|
||||
"Регистратор",
|
||||
"СчетДт",
|
||||
"СчетКт",
|
||||
"Сумма",
|
||||
"Period",
|
||||
"Registrator",
|
||||
"AccountDt",
|
||||
"AccountKt",
|
||||
"Amount"
|
||||
],
|
||||
"materialization_drop_reason": "none",
|
||||
"account_token_raw": "62",
|
||||
"account_token_normalized": "62",
|
||||
"account_scope_fields_checked": [
|
||||
"account_dt",
|
||||
"account_kt",
|
||||
"registrator",
|
||||
"analytics"
|
||||
],
|
||||
"account_scope_match_strategy": "account_code_regex_plus_alias_map_v1",
|
||||
"account_scope_drop_reason": "rows_remaining_after_scope_filter",
|
||||
"runtime_readiness": "LIVE_QUERYABLE_WITH_LIMITS",
|
||||
"limited_reason_category": null,
|
||||
"response_type": "FACTUAL_LIST",
|
||||
"execution_lane": "address_query",
|
||||
"llm_decomposition_applied": true,
|
||||
"llm_decomposition_attempted": true,
|
||||
"llm_provider_used": "local",
|
||||
"llm_decomposition_trace_id": "rm9H5yt7ud1b8q",
|
||||
"llm_decomposition_effective_message": "документы или проводки по счету 62 на ту же дату",
|
||||
"llm_decomposition_reason": "normalized_fragment_applied",
|
||||
"llm_canonical_candidate_detected": true,
|
||||
"llm_predecompose_contract": {
|
||||
"schema_version": "address_llm_predecompose_contract_v1",
|
||||
"source_message": "а на ту же дату по 62",
|
||||
"canonical_message": "документы или проводки по счету 62 на ту же дату",
|
||||
"mode": "address_query",
|
||||
"mode_confidence": "medium",
|
||||
"query_shape": "DOCUMENT_LIST",
|
||||
"query_shape_confidence": "medium",
|
||||
"intent": "documents_forming_balance",
|
||||
"intent_confidence": "medium",
|
||||
"entities": {
|
||||
"account": "62",
|
||||
"counterparty": null,
|
||||
"contract": null,
|
||||
"document_type": null,
|
||||
"document_ref": null,
|
||||
"organization": null
|
||||
},
|
||||
"period": {
|
||||
"scope": "as_of",
|
||||
"period_from": null,
|
||||
"period_to": null,
|
||||
"as_of_date": "2026-04-03",
|
||||
"has_explicit_period": true
|
||||
},
|
||||
"aggregation_profile": "balance_snapshot"
|
||||
},
|
||||
"fallback_rule_hit": null,
|
||||
"sanitized_user_message": "а на ту же дату по 62",
|
||||
"tool_gate_decision": "run_address_lane",
|
||||
"tool_gate_reason": "address_mode_classifier_detected",
|
||||
"answer_structure_v11": null,
|
||||
"investigation_state_snapshot": null,
|
||||
"normalized": null,
|
||||
"normalizer_output": {
|
||||
"trace_id": "rm9H5yt7ud1b8q",
|
||||
"prompt_version": "normalizer_v2_0_2",
|
||||
"applied": true,
|
||||
"effective_message": "документы или проводки по счету 62 на ту же дату"
|
||||
}
|
||||
},
|
||||
"error_code": null,
|
||||
"error_message": null
|
||||
},
|
||||
{
|
||||
"index": 25,
|
||||
"id": "F025",
|
||||
"question": "а документы по этому же договору за тот же период",
|
||||
"session": "chain_contract_e",
|
||||
"session_id": "asst-2026-04-03_Address_Nightly_followup_25_10-11-30-chain_contract_e",
|
||||
"status_code": 200,
|
||||
"ok": true,
|
||||
"elapsed_ms": 4953,
|
||||
"reply_type": "partial_coverage",
|
||||
"trace_id": "address-GBaeVr6I-M",
|
||||
"assistant_reply": "Для точного адресного поиска не хватает обязательного якоря.\nПричина: не хватает обязательных фильтров.\nЧто нужно уточнить: уточните: contract.",
|
||||
"assistant_reply_first_line": "Для точного адресного поиска не хватает обязательного якоря.",
|
||||
"expected_intent": "list_documents_by_contract",
|
||||
"actual_intent": "list_documents_by_contract",
|
||||
"intent_match": true,
|
||||
"expected_mode": "address_query",
|
||||
"actual_mode": "address_query",
|
||||
"mode_match": true,
|
||||
"expected_reply_type": null,
|
||||
"reply_match": true,
|
||||
"semantic_pass": true,
|
||||
"route_pass": false,
|
||||
"route_health": "blocked_missing_anchor",
|
||||
"strict_policy": "route",
|
||||
"strict_pass": false,
|
||||
"selected_recipe": "address_documents_by_contract_v1",
|
||||
"missing_required_filters": [
|
||||
"contract"
|
||||
],
|
||||
"match_failure_stage": "none",
|
||||
"match_failure_reason": null,
|
||||
"rows_fetched": 0,
|
||||
"rows_matched": 0,
|
||||
"mcp_call_status": "skipped",
|
||||
"limited_reason_category": "missing_anchor",
|
||||
"llm_decomposition_attempted": true,
|
||||
"llm_decomposition_applied": true,
|
||||
"llm_decomposition_reason": "normalized_fragment_applied",
|
||||
"llm_canonical_candidate_detected": true,
|
||||
"fallback_rule_hit": null,
|
||||
"tool_gate_decision": "run_address_lane",
|
||||
"tool_gate_reason": "address_mode_classifier_detected",
|
||||
"predecompose_contract_intent": "list_documents_by_contract",
|
||||
"predecompose_contract_aggregation_profile": "list_lookup",
|
||||
"predecompose_contract_period_scope": "unspecified",
|
||||
"debug_payload": {
|
||||
"trace_id": "address-GBaeVr6I-M",
|
||||
"prompt_version": "address_query_runtime_v1",
|
||||
"schema_version": "address_query_runtime_v1",
|
||||
"fallback_type": "partial",
|
||||
"route_summary": null,
|
||||
"fragments": [],
|
||||
"requirements_extracted": [],
|
||||
"coverage_report": {
|
||||
"requirements_total": 0,
|
||||
"requirements_covered": 0,
|
||||
"requirements_uncovered": [],
|
||||
"requirements_partially_covered": [],
|
||||
"clarification_needed_for": [],
|
||||
"out_of_scope_requirements": []
|
||||
},
|
||||
"routes": [],
|
||||
"retrieval_status": [],
|
||||
"retrieval_results": [],
|
||||
"answer_grounding_check": {
|
||||
"status": "partial",
|
||||
"route_subject_match": true,
|
||||
"missing_requirements": [],
|
||||
"reasons": [
|
||||
"address_entity_detected",
|
||||
"document_list_signal_detected",
|
||||
"documents_by_contract_signal_detected",
|
||||
"missing_required_filters"
|
||||
],
|
||||
"why_included_summary": [],
|
||||
"selection_reason_summary": []
|
||||
},
|
||||
"dropped_intent_segments": [],
|
||||
"detected_mode": "address_query",
|
||||
"detected_mode_confidence": "medium",
|
||||
"query_shape": "DOCUMENT_LIST",
|
||||
"query_shape_confidence": "medium",
|
||||
"detected_intent": "list_documents_by_contract",
|
||||
"detected_intent_confidence": "medium",
|
||||
"extracted_filters": {
|
||||
"sort": "period_desc",
|
||||
"limit": 20
|
||||
},
|
||||
"missing_required_filters": [
|
||||
"contract"
|
||||
],
|
||||
"selected_recipe": "address_documents_by_contract_v1",
|
||||
"mcp_call_status_legacy": "skipped",
|
||||
"account_scope_mode": "strict",
|
||||
"account_scope_fallback_applied": false,
|
||||
"anchor_type": "unknown",
|
||||
"anchor_value_raw": null,
|
||||
"anchor_value_resolved": null,
|
||||
"resolver_confidence": "low",
|
||||
"ambiguity_count": 0,
|
||||
"match_failure_stage": "none",
|
||||
"match_failure_reason": null,
|
||||
"mcp_call_status": "skipped",
|
||||
"rows_fetched": 0,
|
||||
"raw_rows_received": 0,
|
||||
"rows_after_account_scope": 0,
|
||||
"rows_after_recipe_filter": 0,
|
||||
"rows_materialized": 0,
|
||||
"rows_matched": 0,
|
||||
"raw_row_keys_sample": [],
|
||||
"materialization_drop_reason": "none",
|
||||
"account_token_raw": null,
|
||||
"account_token_normalized": null,
|
||||
"account_scope_fields_checked": [
|
||||
"account_dt",
|
||||
"account_kt",
|
||||
"registrator",
|
||||
"analytics"
|
||||
],
|
||||
"account_scope_match_strategy": "account_code_regex_plus_alias_map_v1",
|
||||
"account_scope_drop_reason": "not_applicable",
|
||||
"runtime_readiness": "LIVE_QUERYABLE_WITH_LIMITS",
|
||||
"limited_reason_category": "missing_anchor",
|
||||
"response_type": "LIMITED_WITH_REASON",
|
||||
"execution_lane": "address_query",
|
||||
"llm_decomposition_applied": true,
|
||||
"llm_decomposition_attempted": true,
|
||||
"llm_provider_used": "local",
|
||||
"llm_decomposition_trace_id": "vRr64JtkFKVyav",
|
||||
"llm_decomposition_effective_message": "документы по договору за указанный период",
|
||||
"llm_decomposition_reason": "normalized_fragment_applied",
|
||||
"llm_canonical_candidate_detected": true,
|
||||
"llm_predecompose_contract": {
|
||||
"schema_version": "address_llm_predecompose_contract_v1",
|
||||
"source_message": "а документы по этому же договору за тот же период",
|
||||
"canonical_message": "документы по договору за указанный период",
|
||||
"mode": "address_query",
|
||||
"mode_confidence": "medium",
|
||||
"query_shape": "DOCUMENT_LIST",
|
||||
"query_shape_confidence": "medium",
|
||||
"intent": "list_documents_by_contract",
|
||||
"intent_confidence": "medium",
|
||||
"entities": {
|
||||
"account": null,
|
||||
"counterparty": null,
|
||||
"contract": null,
|
||||
"document_type": null,
|
||||
"document_ref": null,
|
||||
"organization": null
|
||||
},
|
||||
"period": {
|
||||
"scope": "unspecified",
|
||||
"period_from": null,
|
||||
"period_to": null,
|
||||
"as_of_date": null,
|
||||
"has_explicit_period": false
|
||||
},
|
||||
"aggregation_profile": "list_lookup"
|
||||
},
|
||||
"fallback_rule_hit": null,
|
||||
"sanitized_user_message": "а документы по этому же договору за тот же период",
|
||||
"tool_gate_decision": "run_address_lane",
|
||||
"tool_gate_reason": "address_mode_classifier_detected",
|
||||
"answer_structure_v11": null,
|
||||
"investigation_state_snapshot": null,
|
||||
"normalized": null,
|
||||
"normalizer_output": {
|
||||
"trace_id": "vRr64JtkFKVyav",
|
||||
"prompt_version": "normalizer_v2_0_2",
|
||||
"applied": true,
|
||||
"effective_message": "документы по договору за указанный период"
|
||||
}
|
||||
},
|
||||
"error_code": null,
|
||||
"error_message": null
|
||||
}
|
||||
]
|
||||
File diff suppressed because it is too large
Load Diff
|
|
@ -0,0 +1,29 @@
|
|||
# Response Audit: 2026-04-03_Address_Nightly_followup_25_10-11-30
|
||||
|
||||
| id | strict | route_health | reply_type | intent | limited_reason | question | assistant_first_line |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| F001 | True | ok_or_factual | factual | list_documents_by_counterparty | None | Покажи документы по СВК за 2020 | Собран список документов по контрагенту (live address lane). |
|
||||
| F002 | True | ok_or_factual | factual | list_documents_by_counterparty | None | а теперь только за май 2020 | По окну 2020-05-01..2020-05-31 строк не найдено; показаны ближайшие доступные данные 2020-07-27..2021-11-10. |
|
||||
| F003 | True | ok_or_factual | factual | account_balance_snapshot | None | а по счету 60.01 на ту же дату | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| F004 | True | ok_or_factual | factual | account_balance_snapshot | None | а на ту же дату по 62.01 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| F005 | True | ok_or_factual | factual | documents_forming_balance | None | раскрой остаток по 62.01 документами на ту же дату | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| F006 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | Покажи банковские операции по СВК за 2020 | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| F007 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | а теперь только ноябрь 2020 | Период сохранен. Глубина live-выборки автоматически расширена до 1000 строк. |
|
||||
| F008 | True | ok_or_factual | factual | list_documents_by_counterparty | None | а теперь документы по нему | Период сохранен. Глубина live-выборки автоматически расширена до 1000 строк. |
|
||||
| F009 | True | ok_or_factual | factual | account_balance_snapshot | None | и остаток по 51 на ту же дату | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| F010 | True | ok_or_factual | factual | account_balance_snapshot | None | а теперь только май 2020 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| F011 | True | partial_non_blocking | partial_coverage | account_balance_snapshot | empty_match | Покажи сальдо по счету 60 на 2020-12-31 | В live-данных по текущему фильтру записи не найдены. |
|
||||
| F012 | True | ok_or_factual | factual | account_balance_snapshot | None | а теперь только май 2020 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| F013 | False | ok_or_factual | factual | documents_forming_balance | None | а на ту же дату по 62 | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| F014 | True | ok_or_factual | factual | documents_forming_balance | None | раскрой 62 документами на ту же дату | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| F015 | True | partial_non_blocking | partial_coverage | account_balance_snapshot | empty_match | вернись на 2020-12-31 по 60 | В live-данных по текущему фильтру записи не найдены. |
|
||||
| F016 | True | ok_or_factual | factual | list_documents_by_contract | None | Покажи документы по договору 19/15 за 2020 | Период сохранен. Глубина live-выборки автоматически расширена до 1000 строк. |
|
||||
| F017 | True | ok_or_factual | factual | bank_operations_by_contract | None | а теперь банковские операции | Период сохранен. Глубина live-выборки автоматически расширена до 1000 строк. |
|
||||
| F018 | True | ok_or_factual | factual | bank_operations_by_contract | None | а теперь только за 2020 | Период сохранен. Глубина live-выборки автоматически расширена до 1000 строк. |
|
||||
| F019 | True | ok_or_factual | factual | open_items_by_counterparty_or_contract | None | есть ли открытые позиции по договору 19/15 на 2020-12-31 | Период сохранен. Глубина live-выборки автоматически расширена до 1000 строк. |
|
||||
| F020 | True | ok_or_factual | factual | list_documents_by_contract | None | покажи документы по этому же договору | Собран список документов по договору (live address lane). |
|
||||
| F021 | True | ok_or_factual | factual | bank_operations_by_contract | None | Покажи банковские операции по договору 19/15 | Собран список банковских операций по договору (live address lane). |
|
||||
| F022 | True | ok_or_factual | factual | open_items_by_counterparty_or_contract | None | а теперь открытые позиции по нему | Период сохранен. Глубина live-выборки автоматически расширена до 1000 строк. |
|
||||
| F023 | True | ok_or_factual | factual | open_items_by_counterparty_or_contract | None | а теперь на 2020-12-31 | Период сохранен. Глубина live-выборки автоматически расширена до 1000 строк. |
|
||||
| F024 | True | ok_or_factual | factual | account_balance_snapshot | None | а по счету 51 на ту же дату | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| F025 | False | blocked_missing_anchor | partial_coverage | list_documents_by_contract | missing_anchor | а документы по этому же договору за тот же период | Для точного адресного поиска не хватает обязательного якоря. |
|
||||
|
|
@ -0,0 +1,145 @@
|
|||
{
|
||||
"run_id": "2026-04-03_Address_Nightly_followup_25_10-11-30",
|
||||
"generated_at": "2026-04-03T10:14:11",
|
||||
"source_questions_file": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\question_sets\\address_followup_context_chains_2026-04-02.json",
|
||||
"backend_url": "http://127.0.0.1:8787/api/assistant/message",
|
||||
"llm_provider": "local",
|
||||
"llm_model": "qwen2.5-14b-instruct-1m",
|
||||
"llm_base_url": "http://127.0.0.1:1234",
|
||||
"strict_policy": "route",
|
||||
"totals": {
|
||||
"questions_total": 25,
|
||||
"ok_200_count": 25,
|
||||
"semantic_pass_count": 24,
|
||||
"semantic_pass_rate": 0.96,
|
||||
"route_pass_count": 23,
|
||||
"route_pass_rate": 0.92,
|
||||
"strict_pass_count": 23,
|
||||
"strict_pass_rate": 0.92,
|
||||
"factual_count": 22,
|
||||
"partial_coverage_count": 3,
|
||||
"clarification_required_count": 0,
|
||||
"http_error_count": 0,
|
||||
"llm_decomposition_attempted_count": 25,
|
||||
"llm_decomposition_applied_count": 23,
|
||||
"llm_fallback_count": 0,
|
||||
"llm_fallback_rate": 0.0,
|
||||
"tool_gate_blocked_count": 0,
|
||||
"tool_gate_blocked_rate": 0.0,
|
||||
"avg_elapsed_ms": 6446.1
|
||||
},
|
||||
"distributions": {
|
||||
"reply_type": {
|
||||
"factual": 22,
|
||||
"partial_coverage": 3
|
||||
},
|
||||
"actual_intent": {
|
||||
"list_documents_by_counterparty": 3,
|
||||
"account_balance_snapshot": 8,
|
||||
"documents_forming_balance": 3,
|
||||
"bank_operations_by_counterparty": 2,
|
||||
"list_documents_by_contract": 3,
|
||||
"bank_operations_by_contract": 3,
|
||||
"open_items_by_counterparty_or_contract": 3
|
||||
},
|
||||
"actual_mode": {
|
||||
"address_query": 25
|
||||
},
|
||||
"mcp_call_status": {
|
||||
"matched_non_empty": 22,
|
||||
"no_raw_rows": 2,
|
||||
"skipped": 1
|
||||
},
|
||||
"limited_reason_category": {
|
||||
"empty_match": 2,
|
||||
"missing_anchor": 1
|
||||
},
|
||||
"route_health": {
|
||||
"ok_or_factual": 22,
|
||||
"partial_non_blocking": 2,
|
||||
"blocked_missing_anchor": 1
|
||||
},
|
||||
"tool_gate_decision": {
|
||||
"run_address_lane": 25
|
||||
},
|
||||
"tool_gate_reason": {
|
||||
"address_mode_classifier_detected": 17,
|
||||
"llm_canonical_candidate_detected": 8
|
||||
}
|
||||
},
|
||||
"address_llm_predecompose_metrics": {
|
||||
"overall": {
|
||||
"llm_attempted": 25,
|
||||
"llm_applied": 23,
|
||||
"fallback_used": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"tool_gate_blocked": 0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"by_intent": {
|
||||
"list_documents_by_counterparty": {
|
||||
"total": 3,
|
||||
"llm_attempted": 3,
|
||||
"llm_applied": 3,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"account_balance_snapshot": {
|
||||
"total": 8,
|
||||
"llm_attempted": 8,
|
||||
"llm_applied": 7,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"documents_forming_balance": {
|
||||
"total": 3,
|
||||
"llm_attempted": 3,
|
||||
"llm_applied": 3,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"bank_operations_by_counterparty": {
|
||||
"total": 2,
|
||||
"llm_attempted": 2,
|
||||
"llm_applied": 2,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"list_documents_by_contract": {
|
||||
"total": 3,
|
||||
"llm_attempted": 3,
|
||||
"llm_applied": 3,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"bank_operations_by_contract": {
|
||||
"total": 3,
|
||||
"llm_attempted": 3,
|
||||
"llm_applied": 2,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"open_items_by_counterparty_or_contract": {
|
||||
"total": 3,
|
||||
"llm_attempted": 3,
|
||||
"llm_applied": 3,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,67 @@
|
|||
{
|
||||
"generated_at": "2026-04-03T10:14:11",
|
||||
"overall_pass": false,
|
||||
"baseline": {
|
||||
"summary_path": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\runs\\2026-04-02_Address_Followup_Context_Chains_2026-04-02_19-15-Run5\\run_summary.json",
|
||||
"run_id": "2026-04-02_Address_Followup_Context_Chains_2026-04-02_19-15-Run5",
|
||||
"questions_total": 25,
|
||||
"strict_pass_rate": 1.0,
|
||||
"route_pass_rate": 1.0,
|
||||
"execution_error_count": 0,
|
||||
"false_factual_rate": 0.0,
|
||||
"notes": [
|
||||
"execution_error_count derived as http_error_count + limited_reason_category.execution_error",
|
||||
"false_factual_rate derived from false_factual_count/questions_total"
|
||||
]
|
||||
},
|
||||
"candidate": {
|
||||
"summary_path": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\runs\\2026-04-03_Address_Nightly_Regression_Post_AnchorHardening_R2\\2026-04-03_Address_Nightly_followup_25_10-11-30\\run_summary.json",
|
||||
"run_id": "2026-04-03_Address_Nightly_followup_25_10-11-30",
|
||||
"questions_total": 25,
|
||||
"strict_pass_rate": 0.92,
|
||||
"route_pass_rate": 0.92,
|
||||
"execution_error_count": 0,
|
||||
"false_factual_rate": 0.0,
|
||||
"notes": [
|
||||
"execution_error_count derived as http_error_count + limited_reason_category.execution_error",
|
||||
"false_factual_rate derived from false_factual_count/questions_total"
|
||||
]
|
||||
},
|
||||
"checks": [
|
||||
{
|
||||
"metric": "strict_pass_rate",
|
||||
"passed": false,
|
||||
"baseline": 1.0,
|
||||
"candidate": 0.92,
|
||||
"rule": "candidate >= baseline"
|
||||
},
|
||||
{
|
||||
"metric": "route_pass_rate",
|
||||
"passed": false,
|
||||
"baseline": 1.0,
|
||||
"candidate": 0.92,
|
||||
"rule": "candidate >= baseline"
|
||||
},
|
||||
{
|
||||
"metric": "execution_error_count",
|
||||
"passed": true,
|
||||
"baseline": 0,
|
||||
"candidate": 0,
|
||||
"rule": "candidate <= baseline"
|
||||
},
|
||||
{
|
||||
"metric": "false_factual_rate",
|
||||
"passed": true,
|
||||
"baseline": 0.0,
|
||||
"candidate": 0.0,
|
||||
"rule": "candidate <= baseline"
|
||||
},
|
||||
{
|
||||
"metric": "questions_total_match",
|
||||
"passed": true,
|
||||
"baseline": 25,
|
||||
"candidate": 25,
|
||||
"rule": "candidate == baseline"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -0,0 +1,23 @@
|
|||
{
|
||||
"generated_at": "2026-04-03T10:14:11",
|
||||
"total": 1,
|
||||
"passed": 1,
|
||||
"failed": 0,
|
||||
"results": [
|
||||
{
|
||||
"run_dir": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\runs\\2026-04-03_Address_Nightly_Regression_Post_AnchorHardening_R2\\2026-04-03_Address_Nightly_followup_25_10-11-30",
|
||||
"valid": true,
|
||||
"errors": [],
|
||||
"warnings": [],
|
||||
"metrics": {
|
||||
"run_id": "2026-04-03_Address_Nightly_followup_25_10-11-30",
|
||||
"questions_total": 25,
|
||||
"route_pass_rate": 0.92,
|
||||
"strict_pass_rate": 0.92,
|
||||
"http_error_count": 0,
|
||||
"rows_count": 25,
|
||||
"failures_count": 2
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -0,0 +1,33 @@
|
|||
# 2026-04-03_Address_Nightly_stress_102_10-00-47
|
||||
|
||||
Generated at: 2026-04-03T10:11:29
|
||||
Questions file: X:\1C\NDC_1C\docs\ADDRESS\question_sets\address_slang_stress_full_2026-04-02.json
|
||||
Backend URL: http://127.0.0.1:8787/api/assistant/message
|
||||
LLM: local / qwen2.5-14b-instruct-1m @ http://127.0.0.1:1234
|
||||
Strict policy: route
|
||||
|
||||
## Totals
|
||||
- questions_total: 102
|
||||
- ok_200_count: 102
|
||||
- semantic_pass_count: 102
|
||||
- semantic_pass_rate: 1.0
|
||||
- route_pass_count: 100
|
||||
- route_pass_rate: 0.9804
|
||||
- strict_pass_count: 100
|
||||
- strict_pass_rate: 0.9804
|
||||
- factual_count: 95
|
||||
- partial_coverage_count: 7
|
||||
- clarification_required_count: 0
|
||||
- http_error_count: 0
|
||||
- llm_decomposition_attempted_count: 102
|
||||
- llm_decomposition_applied_count: 77
|
||||
- llm_fallback_count: 0
|
||||
- llm_fallback_rate: 0.0
|
||||
- tool_gate_blocked_count: 0
|
||||
- tool_gate_blocked_rate: 0.0
|
||||
- avg_elapsed_ms: 6300.7
|
||||
|
||||
## Files
|
||||
- run_summary.json
|
||||
- full_live_results.json
|
||||
- failures_only.json
|
||||
|
|
@ -0,0 +1,367 @@
|
|||
[
|
||||
{
|
||||
"index": 9,
|
||||
"id": "S009",
|
||||
"question": "docs by counterparty svk year 2020",
|
||||
"session": null,
|
||||
"session_id": "asst-2026-04-03_Address_Nightly_stress_102_10-00-47-s009",
|
||||
"status_code": 200,
|
||||
"ok": true,
|
||||
"elapsed_ms": 8373,
|
||||
"reply_type": "partial_coverage",
|
||||
"trace_id": "address-kTchuCyYri",
|
||||
"assistant_reply": "В live-данных по текущему фильтру записи не найдены.\nПричина: по указанному якорю и фильтрам в live-выборке нет строк.\nЧто нужно уточнить: уточните период или снимите часть фильтров.",
|
||||
"assistant_reply_first_line": "В live-данных по текущему фильтру записи не найдены.",
|
||||
"expected_intent": "list_documents_by_counterparty",
|
||||
"actual_intent": "list_documents_by_counterparty",
|
||||
"intent_match": true,
|
||||
"expected_mode": "address_query",
|
||||
"actual_mode": "address_query",
|
||||
"mode_match": true,
|
||||
"expected_reply_type": null,
|
||||
"reply_match": true,
|
||||
"semantic_pass": true,
|
||||
"route_pass": false,
|
||||
"route_health": "likely_blocked_route",
|
||||
"strict_policy": "route",
|
||||
"strict_pass": false,
|
||||
"selected_recipe": "address_documents_by_counterparty_v1",
|
||||
"missing_required_filters": [],
|
||||
"match_failure_stage": "materialized_but_not_anchor_matched",
|
||||
"match_failure_reason": "counterparty_anchor_not_matched_in_materialized_rows",
|
||||
"rows_fetched": 20,
|
||||
"rows_matched": 0,
|
||||
"mcp_call_status": "materialized_but_not_anchor_matched",
|
||||
"limited_reason_category": "empty_match",
|
||||
"llm_decomposition_attempted": true,
|
||||
"llm_decomposition_applied": false,
|
||||
"llm_decomposition_reason": "normalized_fragment_rejected_anchor_degradation",
|
||||
"llm_canonical_candidate_detected": true,
|
||||
"fallback_rule_hit": null,
|
||||
"tool_gate_decision": "run_address_lane",
|
||||
"tool_gate_reason": "address_mode_classifier_detected",
|
||||
"predecompose_contract_intent": "list_documents_by_counterparty",
|
||||
"predecompose_contract_aggregation_profile": "list_lookup",
|
||||
"predecompose_contract_period_scope": "year",
|
||||
"debug_payload": {
|
||||
"trace_id": "address-kTchuCyYri",
|
||||
"prompt_version": "address_query_runtime_v1",
|
||||
"schema_version": "address_query_runtime_v1",
|
||||
"fallback_type": "partial",
|
||||
"route_summary": null,
|
||||
"fragments": [],
|
||||
"requirements_extracted": [],
|
||||
"coverage_report": {
|
||||
"requirements_total": 0,
|
||||
"requirements_covered": 0,
|
||||
"requirements_uncovered": [],
|
||||
"requirements_partially_covered": [],
|
||||
"clarification_needed_for": [],
|
||||
"out_of_scope_requirements": []
|
||||
},
|
||||
"routes": [],
|
||||
"retrieval_status": [],
|
||||
"retrieval_results": [],
|
||||
"answer_grounding_check": {
|
||||
"status": "partial",
|
||||
"route_subject_match": true,
|
||||
"missing_requirements": [],
|
||||
"reasons": [
|
||||
"address_entity_detected",
|
||||
"object_signal_detected",
|
||||
"documents_by_counterparty_signal_detected"
|
||||
],
|
||||
"why_included_summary": [],
|
||||
"selection_reason_summary": []
|
||||
},
|
||||
"dropped_intent_segments": [],
|
||||
"detected_mode": "address_query",
|
||||
"detected_mode_confidence": "medium",
|
||||
"query_shape": "OBJECT_LOOKUP",
|
||||
"query_shape_confidence": "medium",
|
||||
"detected_intent": "list_documents_by_counterparty",
|
||||
"detected_intent_confidence": "medium",
|
||||
"extracted_filters": {
|
||||
"sort": "period_desc",
|
||||
"limit": 20,
|
||||
"counterparty": "svk year 2020",
|
||||
"period_from": "2020-01-01",
|
||||
"period_to": "2020-12-31"
|
||||
},
|
||||
"missing_required_filters": [],
|
||||
"selected_recipe": "address_documents_by_counterparty_v1",
|
||||
"mcp_call_status_legacy": "materialized_but_not_matched",
|
||||
"account_scope_mode": "preferred",
|
||||
"account_scope_fallback_applied": false,
|
||||
"anchor_type": "counterparty",
|
||||
"anchor_value_raw": "svk year 2020",
|
||||
"anchor_value_resolved": "svk year 2020",
|
||||
"resolver_confidence": "medium",
|
||||
"ambiguity_count": 0,
|
||||
"match_failure_stage": "materialized_but_not_anchor_matched",
|
||||
"match_failure_reason": "counterparty_anchor_not_matched_in_materialized_rows",
|
||||
"mcp_call_status": "materialized_but_not_anchor_matched",
|
||||
"rows_fetched": 20,
|
||||
"raw_rows_received": 20,
|
||||
"rows_after_account_scope": 5,
|
||||
"rows_after_recipe_filter": 0,
|
||||
"rows_materialized": 5,
|
||||
"rows_matched": 0,
|
||||
"raw_row_keys_sample": [
|
||||
"Период",
|
||||
"Регистратор",
|
||||
"СчетДт",
|
||||
"СчетКт",
|
||||
"Сумма",
|
||||
"Контрагент",
|
||||
"Period",
|
||||
"Registrator",
|
||||
"Amount"
|
||||
],
|
||||
"materialization_drop_reason": "none",
|
||||
"account_token_raw": null,
|
||||
"account_token_normalized": null,
|
||||
"account_scope_fields_checked": [
|
||||
"account_dt",
|
||||
"account_kt",
|
||||
"registrator",
|
||||
"analytics"
|
||||
],
|
||||
"account_scope_match_strategy": "account_code_regex_plus_alias_map_v1",
|
||||
"account_scope_drop_reason": "not_applicable",
|
||||
"runtime_readiness": "LIVE_QUERYABLE_WITH_LIMITS",
|
||||
"limited_reason_category": "empty_match",
|
||||
"response_type": "LIMITED_WITH_REASON",
|
||||
"execution_lane": "address_query",
|
||||
"llm_decomposition_applied": false,
|
||||
"llm_decomposition_attempted": true,
|
||||
"llm_provider_used": "local",
|
||||
"llm_decomposition_trace_id": "hmwoYUtMeN4gu9",
|
||||
"llm_decomposition_effective_message": "docs by counterparty svk year 2020",
|
||||
"llm_decomposition_reason": "normalized_fragment_rejected_anchor_degradation",
|
||||
"llm_canonical_candidate_detected": true,
|
||||
"llm_predecompose_contract": {
|
||||
"schema_version": "address_llm_predecompose_contract_v1",
|
||||
"source_message": "docs by counterparty svk year 2020",
|
||||
"canonical_message": "docs by counterparty svk year 2020",
|
||||
"mode": "address_query",
|
||||
"mode_confidence": "medium",
|
||||
"query_shape": "OBJECT_LOOKUP",
|
||||
"query_shape_confidence": "medium",
|
||||
"intent": "list_documents_by_counterparty",
|
||||
"intent_confidence": "medium",
|
||||
"entities": {
|
||||
"account": null,
|
||||
"counterparty": "svk year 2020",
|
||||
"contract": null,
|
||||
"document_type": null,
|
||||
"document_ref": null,
|
||||
"organization": null
|
||||
},
|
||||
"period": {
|
||||
"scope": "year",
|
||||
"period_from": "2020-01-01",
|
||||
"period_to": "2020-12-31",
|
||||
"as_of_date": null,
|
||||
"has_explicit_period": true
|
||||
},
|
||||
"aggregation_profile": "list_lookup"
|
||||
},
|
||||
"fallback_rule_hit": null,
|
||||
"sanitized_user_message": "документы by counterparty svk year 2020",
|
||||
"tool_gate_decision": "run_address_lane",
|
||||
"tool_gate_reason": "address_mode_classifier_detected",
|
||||
"answer_structure_v11": null,
|
||||
"investigation_state_snapshot": null,
|
||||
"normalized": null,
|
||||
"normalizer_output": {
|
||||
"trace_id": "hmwoYUtMeN4gu9",
|
||||
"prompt_version": "normalizer_v2_0_2",
|
||||
"applied": false,
|
||||
"effective_message": "docs by counterparty svk year 2020"
|
||||
}
|
||||
},
|
||||
"error_code": null,
|
||||
"error_message": null
|
||||
},
|
||||
{
|
||||
"index": 64,
|
||||
"id": "X004",
|
||||
"question": "svk poka docy za 2020",
|
||||
"session": null,
|
||||
"session_id": "asst-2026-04-03_Address_Nightly_stress_102_10-00-47-x004",
|
||||
"status_code": 200,
|
||||
"ok": true,
|
||||
"elapsed_ms": 5441,
|
||||
"reply_type": "partial_coverage",
|
||||
"trace_id": "address-byPTMz6Jqq",
|
||||
"assistant_reply": "Для точного адресного поиска не хватает обязательного якоря.\nПричина: не хватает обязательных фильтров.\nЧто нужно уточнить: уточните: counterparty.",
|
||||
"assistant_reply_first_line": "Для точного адресного поиска не хватает обязательного якоря.",
|
||||
"expected_intent": "list_documents_by_counterparty",
|
||||
"actual_intent": "list_documents_by_counterparty",
|
||||
"intent_match": true,
|
||||
"expected_mode": "address_query",
|
||||
"actual_mode": "address_query",
|
||||
"mode_match": true,
|
||||
"expected_reply_type": null,
|
||||
"reply_match": true,
|
||||
"semantic_pass": true,
|
||||
"route_pass": false,
|
||||
"route_health": "blocked_missing_anchor",
|
||||
"strict_policy": "route",
|
||||
"strict_pass": false,
|
||||
"selected_recipe": "address_documents_by_counterparty_v1",
|
||||
"missing_required_filters": [
|
||||
"counterparty"
|
||||
],
|
||||
"match_failure_stage": "none",
|
||||
"match_failure_reason": null,
|
||||
"rows_fetched": 0,
|
||||
"rows_matched": 0,
|
||||
"mcp_call_status": "skipped",
|
||||
"limited_reason_category": "missing_anchor",
|
||||
"llm_decomposition_attempted": true,
|
||||
"llm_decomposition_applied": true,
|
||||
"llm_decomposition_reason": "normalized_fragment_applied",
|
||||
"llm_canonical_candidate_detected": true,
|
||||
"fallback_rule_hit": null,
|
||||
"tool_gate_decision": "run_address_lane",
|
||||
"tool_gate_reason": "address_mode_classifier_detected",
|
||||
"predecompose_contract_intent": "list_documents_by_counterparty",
|
||||
"predecompose_contract_aggregation_profile": "list_lookup",
|
||||
"predecompose_contract_period_scope": "year",
|
||||
"debug_payload": {
|
||||
"trace_id": "address-byPTMz6Jqq",
|
||||
"prompt_version": "address_query_runtime_v1",
|
||||
"schema_version": "address_query_runtime_v1",
|
||||
"fallback_type": "partial",
|
||||
"route_summary": null,
|
||||
"fragments": [],
|
||||
"requirements_extracted": [],
|
||||
"coverage_report": {
|
||||
"requirements_total": 0,
|
||||
"requirements_covered": 0,
|
||||
"requirements_uncovered": [],
|
||||
"requirements_partially_covered": [],
|
||||
"clarification_needed_for": [],
|
||||
"out_of_scope_requirements": []
|
||||
},
|
||||
"routes": [],
|
||||
"retrieval_status": [],
|
||||
"retrieval_results": [],
|
||||
"answer_grounding_check": {
|
||||
"status": "partial",
|
||||
"route_subject_match": true,
|
||||
"missing_requirements": [],
|
||||
"reasons": [
|
||||
"address_action_detected",
|
||||
"address_entity_detected",
|
||||
"document_list_signal_detected",
|
||||
"documents_by_counterparty_signal_detected",
|
||||
"missing_required_filters"
|
||||
],
|
||||
"why_included_summary": [],
|
||||
"selection_reason_summary": []
|
||||
},
|
||||
"dropped_intent_segments": [],
|
||||
"detected_mode": "address_query",
|
||||
"detected_mode_confidence": "high",
|
||||
"query_shape": "DOCUMENT_LIST",
|
||||
"query_shape_confidence": "medium",
|
||||
"detected_intent": "list_documents_by_counterparty",
|
||||
"detected_intent_confidence": "medium",
|
||||
"extracted_filters": {
|
||||
"sort": "period_desc",
|
||||
"limit": 20,
|
||||
"period_from": "2020-01-01",
|
||||
"period_to": "2020-12-31"
|
||||
},
|
||||
"missing_required_filters": [
|
||||
"counterparty"
|
||||
],
|
||||
"selected_recipe": "address_documents_by_counterparty_v1",
|
||||
"mcp_call_status_legacy": "skipped",
|
||||
"account_scope_mode": "strict",
|
||||
"account_scope_fallback_applied": false,
|
||||
"anchor_type": "unknown",
|
||||
"anchor_value_raw": null,
|
||||
"anchor_value_resolved": null,
|
||||
"resolver_confidence": "low",
|
||||
"ambiguity_count": 0,
|
||||
"match_failure_stage": "none",
|
||||
"match_failure_reason": null,
|
||||
"mcp_call_status": "skipped",
|
||||
"rows_fetched": 0,
|
||||
"raw_rows_received": 0,
|
||||
"rows_after_account_scope": 0,
|
||||
"rows_after_recipe_filter": 0,
|
||||
"rows_materialized": 0,
|
||||
"rows_matched": 0,
|
||||
"raw_row_keys_sample": [],
|
||||
"materialization_drop_reason": "none",
|
||||
"account_token_raw": null,
|
||||
"account_token_normalized": null,
|
||||
"account_scope_fields_checked": [
|
||||
"account_dt",
|
||||
"account_kt",
|
||||
"registrator",
|
||||
"analytics"
|
||||
],
|
||||
"account_scope_match_strategy": "account_code_regex_plus_alias_map_v1",
|
||||
"account_scope_drop_reason": "not_applicable",
|
||||
"runtime_readiness": "LIVE_QUERYABLE_WITH_LIMITS",
|
||||
"limited_reason_category": "missing_anchor",
|
||||
"response_type": "LIMITED_WITH_REASON",
|
||||
"execution_lane": "address_query",
|
||||
"llm_decomposition_applied": true,
|
||||
"llm_decomposition_attempted": true,
|
||||
"llm_provider_used": "local",
|
||||
"llm_decomposition_trace_id": "25AvO3598F9lLt",
|
||||
"llm_decomposition_effective_message": "Покажи документы за 2020 год.",
|
||||
"llm_decomposition_reason": "normalized_fragment_applied",
|
||||
"llm_canonical_candidate_detected": true,
|
||||
"llm_predecompose_contract": {
|
||||
"schema_version": "address_llm_predecompose_contract_v1",
|
||||
"source_message": "svk poka docy za 2020",
|
||||
"canonical_message": "Покажи документы за 2020 год.",
|
||||
"mode": "address_query",
|
||||
"mode_confidence": "high",
|
||||
"query_shape": "DOCUMENT_LIST",
|
||||
"query_shape_confidence": "medium",
|
||||
"intent": "list_documents_by_counterparty",
|
||||
"intent_confidence": "medium",
|
||||
"entities": {
|
||||
"account": null,
|
||||
"counterparty": null,
|
||||
"contract": null,
|
||||
"document_type": null,
|
||||
"document_ref": null,
|
||||
"organization": null
|
||||
},
|
||||
"period": {
|
||||
"scope": "year",
|
||||
"period_from": "2020-01-01",
|
||||
"period_to": "2020-12-31",
|
||||
"as_of_date": null,
|
||||
"has_explicit_period": true
|
||||
},
|
||||
"aggregation_profile": "list_lookup"
|
||||
},
|
||||
"fallback_rule_hit": null,
|
||||
"sanitized_user_message": "svk poka docy za 2020",
|
||||
"tool_gate_decision": "run_address_lane",
|
||||
"tool_gate_reason": "address_mode_classifier_detected",
|
||||
"answer_structure_v11": null,
|
||||
"investigation_state_snapshot": null,
|
||||
"normalized": null,
|
||||
"normalizer_output": {
|
||||
"trace_id": "25AvO3598F9lLt",
|
||||
"prompt_version": "normalizer_v2_0_2",
|
||||
"applied": true,
|
||||
"effective_message": "Покажи документы за 2020 год."
|
||||
}
|
||||
},
|
||||
"error_code": null,
|
||||
"error_message": null
|
||||
}
|
||||
]
|
||||
File diff suppressed because it is too large
Load Diff
|
|
@ -0,0 +1,106 @@
|
|||
# Response Audit: 2026-04-03_Address_Nightly_stress_102_10-00-47
|
||||
|
||||
| id | strict | route_health | reply_type | intent | limited_reason | question | assistant_first_line |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| S001 | True | ok_or_factual | factual | list_documents_by_counterparty | None | свк доки за 2020 покаж пж | Собран список документов по контрагенту (live address lane). |
|
||||
| S002 | True | ok_or_factual | factual | list_documents_by_counterparty | None | покеж все доки по свк за 20 год | Собран список документов по контрагенту (live address lane). |
|
||||
| S003 | True | ok_or_factual | factual | list_documents_by_counterparty | None | дай список документов по свк за 2020 | Собран список документов по контрагенту (live address lane). |
|
||||
| S004 | True | ok_or_factual | factual | list_documents_by_counterparty | None | какие у свк есть доки за 2020? | Собран список документов по контрагенту (live address lane). |
|
||||
| S005 | True | ok_or_factual | factual | list_documents_by_counterparty | None | че по свк по докам за 2020 год | Собран список документов по контрагенту (live address lane). |
|
||||
| S006 | True | ok_or_factual | factual | list_documents_by_counterparty | None | ёпт покажи доки по свк за 2020 | Собран список документов по контрагенту (live address lane). |
|
||||
| S007 | True | ok_or_factual | factual | list_documents_by_counterparty | None | бля, доки свк 2020 вывали | Собран список документов по контрагенту (live address lane). |
|
||||
| S008 | True | ok_or_factual | factual | list_documents_by_counterparty | None | svk docs for 2020 please | Собран список документов по контрагенту (live address lane). |
|
||||
| S009 | False | likely_blocked_route | partial_coverage | list_documents_by_counterparty | empty_match | docs by counterparty svk year 2020 | В live-данных по текущему фильтру записи не найдены. |
|
||||
| S010 | True | ok_or_factual | factual | list_documents_by_counterparty | None | по контрагенту свк покаж документы за 2020 | Собран список документов по контрагенту (live address lane). |
|
||||
| S011 | True | ok_or_factual | factual | list_documents_by_counterparty | None | плс доки по свк 2020 | Собран список документов по контрагенту (live address lane). |
|
||||
| S012 | True | ok_or_factual | factual | list_documents_by_counterparty | None | свк док за 2020 гони | Собран список документов по контрагенту (live address lane). |
|
||||
| S013 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | по свк банковские операции за 2020 покаж | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| S014 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | свк выписка банка 2020 | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| S015 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | платежи по свк за 2020 выведи | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| S016 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | оплаты по свк за 2020 | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| S017 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | свк банк опер за 2020 | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| S018 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | че у свк по банку за 2020 | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| S019 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | свк списания/поступления за 2020 | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| S020 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | bank ops by svk for 2020 | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| S021 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | transactions by counterparty svk 2020 | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| S022 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | бля, по свк банковские платежки 2020 | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| S023 | True | ok_or_factual | factual | account_balance_snapshot | None | сальдо по счету 60.01 на дату 2020-07-31 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| S024 | True | ok_or_factual | factual | account_balance_snapshot | None | остаток по счету 60.01 на 2020.05 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| S025 | True | ok_or_factual | factual | account_balance_snapshot | None | покажи остаток по 60.01 за май 2020 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| S026 | True | ok_or_factual | factual | account_balance_snapshot | None | счет 60 остаток на май 2020 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| S027 | True | ok_or_factual | factual | account_balance_snapshot | None | баланс account 60.01 as of 2020-07-31 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| S028 | True | ok_or_factual | factual | account_balance_snapshot | None | 60.01 saldo на 31.07.2020 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| S029 | True | ok_or_factual | factual | account_balance_snapshot | None | сколько на счете 60 на конец 2020-05 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| S030 | True | ok_or_factual | factual | account_balance_snapshot | None | дай остаток 60.02 на 2020-12-31 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| S031 | True | partial_non_blocking | partial_coverage | account_balance_snapshot | empty_match | епт, сальдо по счету 62.01 на 2020-12-31 | В live-данных по текущему фильтру записи не найдены. |
|
||||
| S032 | True | ok_or_factual | factual | account_balance_snapshot | None | плс остаток по счету 51 на 2020-06-30 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| S033 | True | ok_or_factual | factual | account_balance_snapshot | None | как там по счету 60.01 на 2020.07? | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| S034 | True | ok_or_factual | factual | account_balance_snapshot | None | сальдо 60 на 2020 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| S035 | True | ok_or_factual | factual | documents_forming_balance | None | какие документы формируют остаток по счету 60.01 на 2020-07-31 | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| S036 | True | ok_or_factual | factual | documents_forming_balance | None | из чего состоит остаток по счету 60 на 2020.05 | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| S037 | True | ok_or_factual | factual | documents_forming_balance | None | раскрой остаток 60.01 по документам на конец июля 2020 | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| S038 | True | ok_or_factual | factual | documents_forming_balance | None | доки под остатком 60.01 на 2020-07-31 | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| S039 | True | ok_or_factual | factual | documents_forming_balance | None | documents forming balance for account 60.01 as of 2020-07-31 | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| S040 | True | ok_or_factual | factual | documents_forming_balance | None | бля, какие доки формируют остаток 60.01 на 2020-05-31 | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| S041 | True | partial_non_blocking | partial_coverage | documents_forming_balance | empty_match | по счету 62.01 раскрой остаток документами на 2020-12-31 | В live-данных по текущему фильтру записи не найдены. |
|
||||
| S042 | True | ok_or_factual | factual | documents_forming_balance | None | из чего остаток по счету 51 на 2020-06-30 | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| S043 | True | ok_or_factual | factual | list_documents_by_contract | None | покажи документы по договору 19/15 за 2020 | Период сохранен. Глубина live-выборки автоматически расширена до 1000 строк. |
|
||||
| S044 | True | ok_or_factual | factual | list_documents_by_contract | None | доки по дог. 19/15 за 2020 | Период сохранен. Глубина live-выборки автоматически расширена до 1000 строк. |
|
||||
| S045 | True | ok_or_factual | factual | list_documents_by_contract | None | какие документы по контракту 19/15 | Собран список документов по договору (live address lane). |
|
||||
| S046 | True | ok_or_factual | factual | list_documents_by_contract | None | docs by contract 19/15 year 2020 | Период сохранен. Глубина live-выборки автоматически расширена до 1000 строк. |
|
||||
| S047 | True | ok_or_factual | factual | list_documents_by_contract | None | бля, покажи по договору 19/15 доки | Собран список документов по договору (live address lane). |
|
||||
| S048 | True | ok_or_factual | factual | list_documents_by_contract | None | договор 19/15 документы | Собран список документов по договору (live address lane). |
|
||||
| S049 | True | ok_or_factual | factual | list_documents_by_contract | None | плс документы договор №19/15 | Собран список документов по договору (live address lane). |
|
||||
| S050 | True | ok_or_factual | factual | list_documents_by_contract | None | покеж документы по 19/15 | Собран список документов по договору (live address lane). |
|
||||
| S051 | True | ok_or_factual | factual | bank_operations_by_contract | None | банковские операции по договору 19/15 | Собран список банковских операций по договору (live address lane). |
|
||||
| S052 | True | ok_or_factual | factual | bank_operations_by_contract | None | платежи по договору 19/15 | Собран список банковских операций по договору (live address lane). |
|
||||
| S053 | True | ok_or_factual | factual | bank_operations_by_contract | None | выписка по договору 19/15 | Собран список банковских операций по договору (live address lane). |
|
||||
| S054 | True | ok_or_factual | factual | bank_operations_by_contract | None | bank ops by contract 19/15 | Собран список банковских операций по договору (live address lane). |
|
||||
| S055 | True | ok_or_factual | factual | bank_operations_by_contract | None | ёпт, покажи банк операции по дог 19/15 | Собран список банковских операций по договору (live address lane). |
|
||||
| S056 | True | ok_or_factual | factual | bank_operations_by_contract | None | по контракту 19/15 платежки | Собран список банковских операций по договору (live address lane). |
|
||||
| S057 | True | ok_or_factual | factual | account_balance_snapshot | None | покажи сальдо по счету 60 за 2020 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| S058 | True | ok_or_factual | factual | account_balance_snapshot | None | а теперь только май 2020 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| S059 | True | ok_or_factual | factual | account_balance_snapshot | None | ок, теперь 2020.05 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| S060 | True | ok_or_factual | factual | account_balance_snapshot | None | теперь на 2020-07-31 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| X001 | True | ok_or_factual | factual | list_documents_by_counterparty | None | свк доки кинь за 2020 | Собран список документов по контрагенту (live address lane). |
|
||||
| X002 | True | ok_or_factual | factual | list_documents_by_counterparty | None | по свк все документы за 20й год | Собран список документов по контрагенту (live address lane). |
|
||||
| X003 | True | ok_or_factual | factual | list_documents_by_counterparty | None | пж вывали доки контрагент свк 2020 | Собран список документов по контрагенту (live address lane). |
|
||||
| X004 | False | blocked_missing_anchor | partial_coverage | list_documents_by_counterparty | missing_anchor | svk poka docy za 2020 | Для точного адресного поиска не хватает обязательного якоря. |
|
||||
| X005 | True | ok_or_factual | factual | list_documents_by_counterparty | None | какие доки по компании свк за 2020 | Собран список документов по контрагенту (live address lane). |
|
||||
| X006 | True | ok_or_factual | factual | list_documents_by_counterparty | None | бля покаж свк документы 2020 | Собран список документов по контрагенту (live address lane). |
|
||||
| X007 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | свк банк операции за 2020 | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| X008 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | по свк выписку банка за 2020 выведи | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| X009 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | свк платежки за 2020 | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| X010 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | bank ops svk 2020 | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| X011 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | транзакции по свк за 2020 | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| X012 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | ёпт по свк че по банку 2020 | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| X013 | True | ok_or_factual | factual | account_balance_snapshot | None | остаток по 60.01 на 2020-07-31 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| X014 | True | ok_or_factual | factual | account_balance_snapshot | None | сальдо 60.01 на 31.07.2020 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| X015 | True | ok_or_factual | factual | account_balance_snapshot | None | что на счете 60 на 2020.05 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| X016 | True | ok_or_factual | factual | account_balance_snapshot | None | дай остаток 62.01 на 2020 май | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| X017 | True | partial_non_blocking | partial_coverage | account_balance_snapshot | empty_match | saldo account 60 as of 2020-12-31 | В live-данных по текущему фильтру записи не найдены. |
|
||||
| X018 | True | ok_or_factual | factual | account_balance_snapshot | None | плиз остаток по счету 51 на 2020-06-30 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| X019 | True | partial_non_blocking | partial_coverage | account_balance_snapshot | empty_match | скока по 60.02 на конец 2020-12 | В live-данных по текущему фильтру записи не найдены. |
|
||||
| X020 | True | ok_or_factual | factual | account_balance_snapshot | None | бля сальдо 60.01 на 2020-05-31 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| X021 | True | ok_or_factual | factual | documents_forming_balance | None | раскрой остаток 60.01 по документам на конец июля 2020 | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| X022 | True | ok_or_factual | factual | documents_forming_balance | None | доки под остатком 60.01 на 2020-07-31 | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| X023 | True | ok_or_factual | factual | documents_forming_balance | None | из чего остаток 60.01 на 2020-07-31 | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| X024 | True | ok_or_factual | factual | documents_forming_balance | None | какие документы формируют остаток по 60.01 на 2020-07-31 | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| X025 | True | ok_or_factual | factual | documents_forming_balance | None | docs forming balance 60.01 as of 2020-07-31 | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| X026 | True | partial_non_blocking | partial_coverage | documents_forming_balance | empty_match | раскидай остаток 62.01 по докам на 2020-12-31 | В live-данных по текущему фильтру записи не найдены. |
|
||||
| X027 | True | ok_or_factual | factual | documents_forming_balance | None | по счету 51 раскрой остаток документами на 2020-06-30 | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| X028 | True | ok_or_factual | factual | documents_forming_balance | None | бля из чего состоит остаток по 60.01 на 2020-05-31 | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| X029 | True | ok_or_factual | factual | list_documents_by_contract | None | доки по договору 19/15 за 2020 | Период сохранен. Глубина live-выборки автоматически расширена до 1000 строк. |
|
||||
| X030 | True | ok_or_factual | factual | list_documents_by_contract | None | покажи документы контракт 19/15 | Собран список документов по договору (live address lane). |
|
||||
| X031 | True | ok_or_factual | factual | list_documents_by_contract | None | docs by contract 19/15 2020 | Период сохранен. Глубина live-выборки автоматически расширена до 1000 строк. |
|
||||
| X032 | True | ok_or_factual | factual | list_documents_by_contract | None | по дог 19/15 какие документы | Собран список документов по договору (live address lane). |
|
||||
| X033 | True | ok_or_factual | factual | list_documents_by_contract | None | договор 19/15 выведи доки | Собран список документов по договору (live address lane). |
|
||||
| X034 | True | ok_or_factual | factual | list_documents_by_contract | None | нах покажи документы по 19/15 | Собран список документов по договору (live address lane). |
|
||||
| X035 | True | ok_or_factual | factual | bank_operations_by_contract | None | банк операции по договору 19/15 | Собран список банковских операций по договору (live address lane). |
|
||||
| X036 | True | ok_or_factual | factual | bank_operations_by_contract | None | платежки по дог 19/15 | Собран список банковских операций по договору (live address lane). |
|
||||
| X037 | True | ok_or_factual | factual | bank_operations_by_contract | None | выписка по контракту 19/15 | Собран список банковских операций по договору (live address lane). |
|
||||
| X038 | True | ok_or_factual | factual | bank_operations_by_contract | None | bank ops by contract 19/15 | Собран список банковских операций по договору (live address lane). |
|
||||
| X039 | True | ok_or_factual | factual | bank_operations_by_contract | None | покажи банк опер по дог 19/15 пж | Собран список банковских операций по договору (live address lane). |
|
||||
| X040 | True | ok_or_factual | factual | bank_operations_by_contract | None | епт транзакции договор 19/15 | Собран список банковских операций по договору (live address lane). |
|
||||
| X041 | True | ok_or_factual | factual | account_balance_snapshot | None | покажи сальдо по счету 60 за 2020 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| X042 | True | ok_or_factual | factual | account_balance_snapshot | None | а теперь только май 2020 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
|
|
@ -0,0 +1,137 @@
|
|||
{
|
||||
"run_id": "2026-04-03_Address_Nightly_stress_102_10-00-47",
|
||||
"generated_at": "2026-04-03T10:11:29",
|
||||
"source_questions_file": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\question_sets\\address_slang_stress_full_2026-04-02.json",
|
||||
"backend_url": "http://127.0.0.1:8787/api/assistant/message",
|
||||
"llm_provider": "local",
|
||||
"llm_model": "qwen2.5-14b-instruct-1m",
|
||||
"llm_base_url": "http://127.0.0.1:1234",
|
||||
"strict_policy": "route",
|
||||
"totals": {
|
||||
"questions_total": 102,
|
||||
"ok_200_count": 102,
|
||||
"semantic_pass_count": 102,
|
||||
"semantic_pass_rate": 1.0,
|
||||
"route_pass_count": 100,
|
||||
"route_pass_rate": 0.9804,
|
||||
"strict_pass_count": 100,
|
||||
"strict_pass_rate": 0.9804,
|
||||
"factual_count": 95,
|
||||
"partial_coverage_count": 7,
|
||||
"clarification_required_count": 0,
|
||||
"http_error_count": 0,
|
||||
"llm_decomposition_attempted_count": 102,
|
||||
"llm_decomposition_applied_count": 77,
|
||||
"llm_fallback_count": 0,
|
||||
"llm_fallback_rate": 0.0,
|
||||
"tool_gate_blocked_count": 0,
|
||||
"tool_gate_blocked_rate": 0.0,
|
||||
"avg_elapsed_ms": 6300.7
|
||||
},
|
||||
"distributions": {
|
||||
"reply_type": {
|
||||
"factual": 95,
|
||||
"partial_coverage": 7
|
||||
},
|
||||
"actual_intent": {
|
||||
"list_documents_by_counterparty": 18,
|
||||
"bank_operations_by_counterparty": 16,
|
||||
"account_balance_snapshot": 26,
|
||||
"documents_forming_balance": 16,
|
||||
"list_documents_by_contract": 14,
|
||||
"bank_operations_by_contract": 12
|
||||
},
|
||||
"actual_mode": {
|
||||
"address_query": 102
|
||||
},
|
||||
"mcp_call_status": {
|
||||
"matched_non_empty": 95,
|
||||
"materialized_but_not_anchor_matched": 1,
|
||||
"no_raw_rows": 5,
|
||||
"skipped": 1
|
||||
},
|
||||
"limited_reason_category": {
|
||||
"empty_match": 6,
|
||||
"missing_anchor": 1
|
||||
},
|
||||
"route_health": {
|
||||
"ok_or_factual": 95,
|
||||
"likely_blocked_route": 1,
|
||||
"partial_non_blocking": 5,
|
||||
"blocked_missing_anchor": 1
|
||||
},
|
||||
"tool_gate_decision": {
|
||||
"run_address_lane": 102
|
||||
},
|
||||
"tool_gate_reason": {
|
||||
"address_mode_classifier_detected": 98,
|
||||
"llm_canonical_candidate_detected": 4
|
||||
}
|
||||
},
|
||||
"address_llm_predecompose_metrics": {
|
||||
"overall": {
|
||||
"llm_attempted": 102,
|
||||
"llm_applied": 77,
|
||||
"fallback_used": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"tool_gate_blocked": 0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"by_intent": {
|
||||
"list_documents_by_counterparty": {
|
||||
"total": 18,
|
||||
"llm_attempted": 18,
|
||||
"llm_applied": 12,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"bank_operations_by_counterparty": {
|
||||
"total": 16,
|
||||
"llm_attempted": 16,
|
||||
"llm_applied": 11,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"account_balance_snapshot": {
|
||||
"total": 26,
|
||||
"llm_attempted": 26,
|
||||
"llm_applied": 25,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"documents_forming_balance": {
|
||||
"total": 16,
|
||||
"llm_attempted": 16,
|
||||
"llm_applied": 11,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"list_documents_by_contract": {
|
||||
"total": 14,
|
||||
"llm_attempted": 14,
|
||||
"llm_applied": 10,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"bank_operations_by_contract": {
|
||||
"total": 12,
|
||||
"llm_attempted": 12,
|
||||
"llm_applied": 8,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,67 @@
|
|||
{
|
||||
"generated_at": "2026-04-03T10:11:30",
|
||||
"overall_pass": false,
|
||||
"baseline": {
|
||||
"summary_path": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\runs\\2026-04-02_Address_Slang_Live_Stress_2026-04-02_12-57-27\\run_summary.json",
|
||||
"run_id": "2026-04-02_Address_Slang_Live_Stress_2026-04-02_12-57-27",
|
||||
"questions_total": 102,
|
||||
"strict_pass_rate": 1.0,
|
||||
"route_pass_rate": 1.0,
|
||||
"execution_error_count": 0,
|
||||
"false_factual_rate": 0.0,
|
||||
"notes": [
|
||||
"execution_error_count derived as http_error_count + limited_reason_category.execution_error",
|
||||
"false_factual_rate derived from false_factual_count/questions_total"
|
||||
]
|
||||
},
|
||||
"candidate": {
|
||||
"summary_path": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\runs\\2026-04-03_Address_Nightly_Regression_Post_AnchorHardening_R2\\2026-04-03_Address_Nightly_stress_102_10-00-47\\run_summary.json",
|
||||
"run_id": "2026-04-03_Address_Nightly_stress_102_10-00-47",
|
||||
"questions_total": 102,
|
||||
"strict_pass_rate": 0.9804,
|
||||
"route_pass_rate": 0.9804,
|
||||
"execution_error_count": 0,
|
||||
"false_factual_rate": 0.0,
|
||||
"notes": [
|
||||
"execution_error_count derived as http_error_count + limited_reason_category.execution_error",
|
||||
"false_factual_rate derived from false_factual_count/questions_total"
|
||||
]
|
||||
},
|
||||
"checks": [
|
||||
{
|
||||
"metric": "strict_pass_rate",
|
||||
"passed": false,
|
||||
"baseline": 1.0,
|
||||
"candidate": 0.9804,
|
||||
"rule": "candidate >= baseline"
|
||||
},
|
||||
{
|
||||
"metric": "route_pass_rate",
|
||||
"passed": false,
|
||||
"baseline": 1.0,
|
||||
"candidate": 0.9804,
|
||||
"rule": "candidate >= baseline"
|
||||
},
|
||||
{
|
||||
"metric": "execution_error_count",
|
||||
"passed": true,
|
||||
"baseline": 0,
|
||||
"candidate": 0,
|
||||
"rule": "candidate <= baseline"
|
||||
},
|
||||
{
|
||||
"metric": "false_factual_rate",
|
||||
"passed": true,
|
||||
"baseline": 0.0,
|
||||
"candidate": 0.0,
|
||||
"rule": "candidate <= baseline"
|
||||
},
|
||||
{
|
||||
"metric": "questions_total_match",
|
||||
"passed": true,
|
||||
"baseline": 102,
|
||||
"candidate": 102,
|
||||
"rule": "candidate == baseline"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -0,0 +1,23 @@
|
|||
{
|
||||
"generated_at": "2026-04-03T10:11:29",
|
||||
"total": 1,
|
||||
"passed": 1,
|
||||
"failed": 0,
|
||||
"results": [
|
||||
{
|
||||
"run_dir": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\runs\\2026-04-03_Address_Nightly_Regression_Post_AnchorHardening_R2\\2026-04-03_Address_Nightly_stress_102_10-00-47",
|
||||
"valid": true,
|
||||
"errors": [],
|
||||
"warnings": [],
|
||||
"metrics": {
|
||||
"run_id": "2026-04-03_Address_Nightly_stress_102_10-00-47",
|
||||
"questions_total": 102,
|
||||
"route_pass_rate": 0.9804,
|
||||
"strict_pass_rate": 0.9804,
|
||||
"http_error_count": 0,
|
||||
"rows_count": 102,
|
||||
"failures_count": 2
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -0,0 +1,26 @@
|
|||
# 2026-04-03_Address_Nightly_Regression_Post_AnchorHardening_R2
|
||||
|
||||
Generated at: 2026-04-03T10:14:11
|
||||
Dry run: False
|
||||
Strict policy: route
|
||||
Overall: FAIL
|
||||
|
||||
## Packs
|
||||
### stress_102
|
||||
- run_id: 2026-04-03_Address_Nightly_stress_102_10-00-47
|
||||
- runner_ok: True
|
||||
- validator_ok: True
|
||||
- comparator_ok: False
|
||||
- run_dir: X:\1C\NDC_1C\docs\ADDRESS\runs\2026-04-03_Address_Nightly_Regression_Post_AnchorHardening_R2\2026-04-03_Address_Nightly_stress_102_10-00-47
|
||||
- errors:
|
||||
- comparator failed with exit code 1
|
||||
|
||||
### followup_25
|
||||
- run_id: 2026-04-03_Address_Nightly_followup_25_10-11-30
|
||||
- runner_ok: True
|
||||
- validator_ok: True
|
||||
- comparator_ok: False
|
||||
- run_dir: X:\1C\NDC_1C\docs\ADDRESS\runs\2026-04-03_Address_Nightly_Regression_Post_AnchorHardening_R2\2026-04-03_Address_Nightly_followup_25_10-11-30
|
||||
- errors:
|
||||
- comparator failed with exit code 1
|
||||
|
||||
|
|
@ -0,0 +1,37 @@
|
|||
{
|
||||
"nightly_run_id": "2026-04-03_Address_Nightly_Regression_Post_AnchorHardening_R2",
|
||||
"generated_at": "2026-04-03T10:14:11",
|
||||
"dry_run": false,
|
||||
"overall_ok": false,
|
||||
"strict_policy": "route",
|
||||
"packs": [
|
||||
{
|
||||
"pack": "stress_102",
|
||||
"run_id": "2026-04-03_Address_Nightly_stress_102_10-00-47",
|
||||
"questions_file": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\question_sets\\address_slang_stress_full_2026-04-02.json",
|
||||
"baseline_summary": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\runs\\2026-04-02_Address_Slang_Live_Stress_2026-04-02_12-57-27\\run_summary.json",
|
||||
"run_dir": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\runs\\2026-04-03_Address_Nightly_Regression_Post_AnchorHardening_R2\\2026-04-03_Address_Nightly_stress_102_10-00-47",
|
||||
"runner_ok": true,
|
||||
"validator_ok": true,
|
||||
"comparator_ok": false,
|
||||
"errors": [
|
||||
"comparator failed with exit code 1"
|
||||
],
|
||||
"validator_report": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\runs\\2026-04-03_Address_Nightly_Regression_Post_AnchorHardening_R2\\2026-04-03_Address_Nightly_stress_102_10-00-47_validator_report.json"
|
||||
},
|
||||
{
|
||||
"pack": "followup_25",
|
||||
"run_id": "2026-04-03_Address_Nightly_followup_25_10-11-30",
|
||||
"questions_file": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\question_sets\\address_followup_context_chains_2026-04-02.json",
|
||||
"baseline_summary": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\runs\\2026-04-02_Address_Followup_Context_Chains_2026-04-02_19-15-Run5\\run_summary.json",
|
||||
"run_dir": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\runs\\2026-04-03_Address_Nightly_Regression_Post_AnchorHardening_R2\\2026-04-03_Address_Nightly_followup_25_10-11-30",
|
||||
"runner_ok": true,
|
||||
"validator_ok": true,
|
||||
"comparator_ok": false,
|
||||
"errors": [
|
||||
"comparator failed with exit code 1"
|
||||
],
|
||||
"validator_report": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\runs\\2026-04-03_Address_Nightly_Regression_Post_AnchorHardening_R2\\2026-04-03_Address_Nightly_followup_25_10-11-30_validator_report.json"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -0,0 +1,33 @@
|
|||
# 2026-04-03_Address_Nightly_followup_25_10-31-38
|
||||
|
||||
Generated at: 2026-04-03T10:34:22
|
||||
Questions file: X:\1C\NDC_1C\docs\ADDRESS\question_sets\address_followup_context_chains_2026-04-02.json
|
||||
Backend URL: http://127.0.0.1:8787/api/assistant/message
|
||||
LLM: local / qwen2.5-14b-instruct-1m @ http://127.0.0.1:1234
|
||||
Strict policy: route
|
||||
|
||||
## Totals
|
||||
- questions_total: 25
|
||||
- ok_200_count: 25
|
||||
- semantic_pass_count: 24
|
||||
- semantic_pass_rate: 0.96
|
||||
- route_pass_count: 24
|
||||
- route_pass_rate: 0.96
|
||||
- strict_pass_count: 24
|
||||
- strict_pass_rate: 0.96
|
||||
- factual_count: 23
|
||||
- partial_coverage_count: 2
|
||||
- clarification_required_count: 0
|
||||
- http_error_count: 0
|
||||
- llm_decomposition_attempted_count: 25
|
||||
- llm_decomposition_applied_count: 23
|
||||
- llm_fallback_count: 0
|
||||
- llm_fallback_rate: 0.0
|
||||
- tool_gate_blocked_count: 0
|
||||
- tool_gate_blocked_rate: 0.0
|
||||
- avg_elapsed_ms: 6586.2
|
||||
|
||||
## Files
|
||||
- run_summary.json
|
||||
- full_live_results.json
|
||||
- failures_only.json
|
||||
|
|
@ -0,0 +1,192 @@
|
|||
[
|
||||
{
|
||||
"index": 13,
|
||||
"id": "F013",
|
||||
"question": "а на ту же дату по 62",
|
||||
"session": "chain_balance_c",
|
||||
"session_id": "asst-2026-04-03_Address_Nightly_followup_25_10-31-38-chain_balance_c",
|
||||
"status_code": 200,
|
||||
"ok": true,
|
||||
"elapsed_ms": 6104,
|
||||
"reply_type": "factual",
|
||||
"trace_id": "address-YXYl7jQC_c",
|
||||
"assistant_reply": "Собран drilldown документов, формирующих остаток по счету на указанную дату.\nДокументных строк отобрано: 4.\nСумма по отобранным движениям: 2545546.24.\n1. 2020-05-21T12:00:00Z | Поступление на расчетный счет 00000000023 от 21.05.2020 12:00:00 | 51 / 62.02 | 250546.24\n2. 2020-05-18T12:00:01Z | Реализация товаров и услуг 00000000002 от 18.05.2020 12:00:01 | 62.01 / 90.01.1 | 765000\n3. 2020-05-18T12:00:01Z | Реализация товаров и услуг 00000000002 от 18.05.2020 12:00:01 | 62.02 / 62.01 | 765000\n4. 2020-05-15T12:00:02Z | Поступление на расчетный счет 00000000022 от 15.05.2020 12:00:02 | 51 / 62.02 | 765000\nМожно уточнить выборку по контрагенту, договору или периоду.",
|
||||
"assistant_reply_first_line": "Собран drilldown документов, формирующих остаток по счету на указанную дату.",
|
||||
"expected_intent": "account_balance_snapshot",
|
||||
"actual_intent": "documents_forming_balance",
|
||||
"intent_match": false,
|
||||
"expected_mode": "address_query",
|
||||
"actual_mode": "address_query",
|
||||
"mode_match": true,
|
||||
"expected_reply_type": null,
|
||||
"reply_match": true,
|
||||
"semantic_pass": false,
|
||||
"route_pass": false,
|
||||
"route_health": "ok_or_factual",
|
||||
"strict_policy": "route",
|
||||
"strict_pass": false,
|
||||
"selected_recipe": "address_documents_forming_balance_v1",
|
||||
"missing_required_filters": [],
|
||||
"match_failure_stage": "none",
|
||||
"match_failure_reason": null,
|
||||
"rows_fetched": 4,
|
||||
"rows_matched": 4,
|
||||
"mcp_call_status": "matched_non_empty",
|
||||
"limited_reason_category": null,
|
||||
"llm_decomposition_attempted": true,
|
||||
"llm_decomposition_applied": true,
|
||||
"llm_decomposition_reason": "normalized_fragment_applied",
|
||||
"llm_canonical_candidate_detected": true,
|
||||
"fallback_rule_hit": null,
|
||||
"tool_gate_decision": "run_address_lane",
|
||||
"tool_gate_reason": "address_mode_classifier_detected",
|
||||
"predecompose_contract_intent": "documents_forming_balance",
|
||||
"predecompose_contract_aggregation_profile": "balance_snapshot",
|
||||
"predecompose_contract_period_scope": "as_of",
|
||||
"debug_payload": {
|
||||
"trace_id": "address-YXYl7jQC_c",
|
||||
"prompt_version": "address_query_runtime_v1",
|
||||
"schema_version": "address_query_runtime_v1",
|
||||
"fallback_type": "none",
|
||||
"route_summary": null,
|
||||
"fragments": [],
|
||||
"requirements_extracted": [],
|
||||
"coverage_report": {
|
||||
"requirements_total": 0,
|
||||
"requirements_covered": 0,
|
||||
"requirements_uncovered": [],
|
||||
"requirements_partially_covered": [],
|
||||
"clarification_needed_for": [],
|
||||
"out_of_scope_requirements": []
|
||||
},
|
||||
"routes": [],
|
||||
"retrieval_status": [],
|
||||
"retrieval_results": [],
|
||||
"answer_grounding_check": {
|
||||
"status": "grounded",
|
||||
"route_subject_match": true,
|
||||
"missing_requirements": [],
|
||||
"reasons": [
|
||||
"address_entity_detected",
|
||||
"document_list_signal_detected",
|
||||
"documents_by_account_drilldown_signal_detected",
|
||||
"as_of_date_from_followup_context",
|
||||
"period_from_followup_context",
|
||||
"address_followup_context_applied"
|
||||
],
|
||||
"why_included_summary": [],
|
||||
"selection_reason_summary": []
|
||||
},
|
||||
"dropped_intent_segments": [],
|
||||
"detected_mode": "address_query",
|
||||
"detected_mode_confidence": "medium",
|
||||
"query_shape": "DOCUMENT_LIST",
|
||||
"query_shape_confidence": "medium",
|
||||
"detected_intent": "documents_forming_balance",
|
||||
"detected_intent_confidence": "medium",
|
||||
"extracted_filters": {
|
||||
"sort": "period_desc",
|
||||
"limit": 20,
|
||||
"account": "62",
|
||||
"as_of_date": "2020-05-31",
|
||||
"period_from": "2020-05-01",
|
||||
"period_to": "2020-05-31"
|
||||
},
|
||||
"missing_required_filters": [],
|
||||
"selected_recipe": "address_documents_forming_balance_v1",
|
||||
"mcp_call_status_legacy": "matched_non_empty",
|
||||
"account_scope_mode": "strict",
|
||||
"account_scope_fallback_applied": false,
|
||||
"anchor_type": "account",
|
||||
"anchor_value_raw": "62",
|
||||
"anchor_value_resolved": "62",
|
||||
"resolver_confidence": "high",
|
||||
"ambiguity_count": 0,
|
||||
"match_failure_stage": "none",
|
||||
"match_failure_reason": null,
|
||||
"mcp_call_status": "matched_non_empty",
|
||||
"rows_fetched": 4,
|
||||
"raw_rows_received": 4,
|
||||
"rows_after_account_scope": 4,
|
||||
"rows_after_recipe_filter": 4,
|
||||
"rows_materialized": 4,
|
||||
"rows_matched": 4,
|
||||
"raw_row_keys_sample": [
|
||||
"Период",
|
||||
"Регистратор",
|
||||
"СчетДт",
|
||||
"СчетКт",
|
||||
"Сумма",
|
||||
"Period",
|
||||
"Registrator",
|
||||
"AccountDt",
|
||||
"AccountKt",
|
||||
"Amount"
|
||||
],
|
||||
"materialization_drop_reason": "none",
|
||||
"account_token_raw": "62",
|
||||
"account_token_normalized": "62",
|
||||
"account_scope_fields_checked": [
|
||||
"account_dt",
|
||||
"account_kt",
|
||||
"registrator",
|
||||
"analytics"
|
||||
],
|
||||
"account_scope_match_strategy": "account_code_regex_plus_alias_map_v1",
|
||||
"account_scope_drop_reason": "rows_remaining_after_scope_filter",
|
||||
"runtime_readiness": "LIVE_QUERYABLE_WITH_LIMITS",
|
||||
"limited_reason_category": null,
|
||||
"response_type": "FACTUAL_LIST",
|
||||
"execution_lane": "address_query",
|
||||
"llm_decomposition_applied": true,
|
||||
"llm_decomposition_attempted": true,
|
||||
"llm_provider_used": "local",
|
||||
"llm_decomposition_trace_id": "EwDn_CZS-xJQOL",
|
||||
"llm_decomposition_effective_message": "документы или проводки по счету 62 на ту же дату",
|
||||
"llm_decomposition_reason": "normalized_fragment_applied",
|
||||
"llm_canonical_candidate_detected": true,
|
||||
"llm_predecompose_contract": {
|
||||
"schema_version": "address_llm_predecompose_contract_v1",
|
||||
"source_message": "а на ту же дату по 62",
|
||||
"canonical_message": "документы или проводки по счету 62 на ту же дату",
|
||||
"mode": "address_query",
|
||||
"mode_confidence": "medium",
|
||||
"query_shape": "DOCUMENT_LIST",
|
||||
"query_shape_confidence": "medium",
|
||||
"intent": "documents_forming_balance",
|
||||
"intent_confidence": "medium",
|
||||
"entities": {
|
||||
"account": "62",
|
||||
"counterparty": null,
|
||||
"contract": null,
|
||||
"document_type": null,
|
||||
"document_ref": null,
|
||||
"organization": null
|
||||
},
|
||||
"period": {
|
||||
"scope": "as_of",
|
||||
"period_from": null,
|
||||
"period_to": null,
|
||||
"as_of_date": "2026-04-03",
|
||||
"has_explicit_period": true
|
||||
},
|
||||
"aggregation_profile": "balance_snapshot"
|
||||
},
|
||||
"fallback_rule_hit": null,
|
||||
"sanitized_user_message": "а на ту же дату по 62",
|
||||
"tool_gate_decision": "run_address_lane",
|
||||
"tool_gate_reason": "address_mode_classifier_detected",
|
||||
"answer_structure_v11": null,
|
||||
"investigation_state_snapshot": null,
|
||||
"normalized": null,
|
||||
"normalizer_output": {
|
||||
"trace_id": "EwDn_CZS-xJQOL",
|
||||
"prompt_version": "normalizer_v2_0_2",
|
||||
"applied": true,
|
||||
"effective_message": "документы или проводки по счету 62 на ту же дату"
|
||||
}
|
||||
},
|
||||
"error_code": null,
|
||||
"error_message": null
|
||||
}
|
||||
]
|
||||
File diff suppressed because it is too large
Load Diff
|
|
@ -0,0 +1,29 @@
|
|||
# Response Audit: 2026-04-03_Address_Nightly_followup_25_10-31-38
|
||||
|
||||
| id | strict | route_health | reply_type | intent | limited_reason | question | assistant_first_line |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| F001 | True | ok_or_factual | factual | list_documents_by_counterparty | None | Покажи документы по СВК за 2020 | Собран список документов по контрагенту (live address lane). |
|
||||
| F002 | True | ok_or_factual | factual | list_documents_by_counterparty | None | а теперь только за май 2020 | По окну 2020-05-01..2020-05-31 строк не найдено; показаны ближайшие доступные данные 2020-07-27..2021-11-10. |
|
||||
| F003 | True | ok_or_factual | factual | account_balance_snapshot | None | а по счету 60.01 на ту же дату | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| F004 | True | ok_or_factual | factual | account_balance_snapshot | None | а на ту же дату по 62.01 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| F005 | True | ok_or_factual | factual | documents_forming_balance | None | раскрой остаток по 62.01 документами на ту же дату | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| F006 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | Покажи банковские операции по СВК за 2020 | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| F007 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | а теперь только ноябрь 2020 | Период сохранен. Глубина live-выборки автоматически расширена до 1000 строк. |
|
||||
| F008 | True | ok_or_factual | factual | list_documents_by_counterparty | None | а теперь документы по нему | Период сохранен. Глубина live-выборки автоматически расширена до 1000 строк. |
|
||||
| F009 | True | ok_or_factual | factual | account_balance_snapshot | None | и остаток по 51 на ту же дату | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| F010 | True | ok_or_factual | factual | account_balance_snapshot | None | а теперь только май 2020 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| F011 | True | partial_non_blocking | partial_coverage | account_balance_snapshot | empty_match | Покажи сальдо по счету 60 на 2020-12-31 | В live-данных по текущему фильтру записи не найдены. |
|
||||
| F012 | True | ok_or_factual | factual | account_balance_snapshot | None | а теперь только май 2020 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| F013 | False | ok_or_factual | factual | documents_forming_balance | None | а на ту же дату по 62 | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| F014 | True | ok_or_factual | factual | documents_forming_balance | None | раскрой 62 документами на ту же дату | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| F015 | True | partial_non_blocking | partial_coverage | account_balance_snapshot | empty_match | вернись на 2020-12-31 по 60 | В live-данных по текущему фильтру записи не найдены. |
|
||||
| F016 | True | ok_or_factual | factual | list_documents_by_contract | None | Покажи документы по договору 19/15 за 2020 | Период сохранен. Глубина live-выборки автоматически расширена до 1000 строк. |
|
||||
| F017 | True | ok_or_factual | factual | bank_operations_by_contract | None | а теперь банковские операции | Период сохранен. Глубина live-выборки автоматически расширена до 1000 строк. |
|
||||
| F018 | True | ok_or_factual | factual | bank_operations_by_contract | None | а теперь только за 2020 | Период сохранен. Глубина live-выборки автоматически расширена до 1000 строк. |
|
||||
| F019 | True | ok_or_factual | factual | open_items_by_counterparty_or_contract | None | есть ли открытые позиции по договору 19/15 на 2020-12-31 | Период сохранен. Глубина live-выборки автоматически расширена до 1000 строк. |
|
||||
| F020 | True | ok_or_factual | factual | list_documents_by_contract | None | покажи документы по этому же договору | Собран список документов по договору (live address lane). |
|
||||
| F021 | True | ok_or_factual | factual | bank_operations_by_contract | None | Покажи банковские операции по договору 19/15 | Собран список банковских операций по договору (live address lane). |
|
||||
| F022 | True | ok_or_factual | factual | open_items_by_counterparty_or_contract | None | а теперь открытые позиции по нему | Период сохранен. Глубина live-выборки автоматически расширена до 1000 строк. |
|
||||
| F023 | True | ok_or_factual | factual | open_items_by_counterparty_or_contract | None | а теперь на 2020-12-31 | Период сохранен. Глубина live-выборки автоматически расширена до 1000 строк. |
|
||||
| F024 | True | ok_or_factual | factual | account_balance_snapshot | None | а по счету 51 на ту же дату | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| F025 | True | ok_or_factual | factual | list_documents_by_contract | None | а документы по этому же договору за тот же период | Собран список документов по договору (live address lane). |
|
||||
|
|
@ -0,0 +1,142 @@
|
|||
{
|
||||
"run_id": "2026-04-03_Address_Nightly_followup_25_10-31-38",
|
||||
"generated_at": "2026-04-03T10:34:22",
|
||||
"source_questions_file": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\question_sets\\address_followup_context_chains_2026-04-02.json",
|
||||
"backend_url": "http://127.0.0.1:8787/api/assistant/message",
|
||||
"llm_provider": "local",
|
||||
"llm_model": "qwen2.5-14b-instruct-1m",
|
||||
"llm_base_url": "http://127.0.0.1:1234",
|
||||
"strict_policy": "route",
|
||||
"totals": {
|
||||
"questions_total": 25,
|
||||
"ok_200_count": 25,
|
||||
"semantic_pass_count": 24,
|
||||
"semantic_pass_rate": 0.96,
|
||||
"route_pass_count": 24,
|
||||
"route_pass_rate": 0.96,
|
||||
"strict_pass_count": 24,
|
||||
"strict_pass_rate": 0.96,
|
||||
"factual_count": 23,
|
||||
"partial_coverage_count": 2,
|
||||
"clarification_required_count": 0,
|
||||
"http_error_count": 0,
|
||||
"llm_decomposition_attempted_count": 25,
|
||||
"llm_decomposition_applied_count": 23,
|
||||
"llm_fallback_count": 0,
|
||||
"llm_fallback_rate": 0.0,
|
||||
"tool_gate_blocked_count": 0,
|
||||
"tool_gate_blocked_rate": 0.0,
|
||||
"avg_elapsed_ms": 6586.2
|
||||
},
|
||||
"distributions": {
|
||||
"reply_type": {
|
||||
"factual": 23,
|
||||
"partial_coverage": 2
|
||||
},
|
||||
"actual_intent": {
|
||||
"list_documents_by_counterparty": 3,
|
||||
"account_balance_snapshot": 8,
|
||||
"documents_forming_balance": 3,
|
||||
"bank_operations_by_counterparty": 2,
|
||||
"list_documents_by_contract": 3,
|
||||
"bank_operations_by_contract": 3,
|
||||
"open_items_by_counterparty_or_contract": 3
|
||||
},
|
||||
"actual_mode": {
|
||||
"address_query": 25
|
||||
},
|
||||
"mcp_call_status": {
|
||||
"matched_non_empty": 23,
|
||||
"no_raw_rows": 2
|
||||
},
|
||||
"limited_reason_category": {
|
||||
"empty_match": 2
|
||||
},
|
||||
"route_health": {
|
||||
"ok_or_factual": 23,
|
||||
"partial_non_blocking": 2
|
||||
},
|
||||
"tool_gate_decision": {
|
||||
"run_address_lane": 25
|
||||
},
|
||||
"tool_gate_reason": {
|
||||
"address_mode_classifier_detected": 17,
|
||||
"llm_canonical_candidate_detected": 8
|
||||
}
|
||||
},
|
||||
"address_llm_predecompose_metrics": {
|
||||
"overall": {
|
||||
"llm_attempted": 25,
|
||||
"llm_applied": 23,
|
||||
"fallback_used": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"tool_gate_blocked": 0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"by_intent": {
|
||||
"list_documents_by_counterparty": {
|
||||
"total": 3,
|
||||
"llm_attempted": 3,
|
||||
"llm_applied": 3,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"account_balance_snapshot": {
|
||||
"total": 8,
|
||||
"llm_attempted": 8,
|
||||
"llm_applied": 7,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"documents_forming_balance": {
|
||||
"total": 3,
|
||||
"llm_attempted": 3,
|
||||
"llm_applied": 3,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"bank_operations_by_counterparty": {
|
||||
"total": 2,
|
||||
"llm_attempted": 2,
|
||||
"llm_applied": 2,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"list_documents_by_contract": {
|
||||
"total": 3,
|
||||
"llm_attempted": 3,
|
||||
"llm_applied": 3,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"bank_operations_by_contract": {
|
||||
"total": 3,
|
||||
"llm_attempted": 3,
|
||||
"llm_applied": 2,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"open_items_by_counterparty_or_contract": {
|
||||
"total": 3,
|
||||
"llm_attempted": 3,
|
||||
"llm_applied": 3,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,67 @@
|
|||
{
|
||||
"generated_at": "2026-04-03T10:34:23",
|
||||
"overall_pass": false,
|
||||
"baseline": {
|
||||
"summary_path": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\runs\\2026-04-02_Address_Followup_Context_Chains_2026-04-02_19-15-Run5\\run_summary.json",
|
||||
"run_id": "2026-04-02_Address_Followup_Context_Chains_2026-04-02_19-15-Run5",
|
||||
"questions_total": 25,
|
||||
"strict_pass_rate": 1.0,
|
||||
"route_pass_rate": 1.0,
|
||||
"execution_error_count": 0,
|
||||
"false_factual_rate": 0.0,
|
||||
"notes": [
|
||||
"execution_error_count derived as http_error_count + limited_reason_category.execution_error",
|
||||
"false_factual_rate derived from false_factual_count/questions_total"
|
||||
]
|
||||
},
|
||||
"candidate": {
|
||||
"summary_path": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\runs\\2026-04-03_Address_Nightly_Regression_Post_AnchorHardening_R4\\2026-04-03_Address_Nightly_followup_25_10-31-38\\run_summary.json",
|
||||
"run_id": "2026-04-03_Address_Nightly_followup_25_10-31-38",
|
||||
"questions_total": 25,
|
||||
"strict_pass_rate": 0.96,
|
||||
"route_pass_rate": 0.96,
|
||||
"execution_error_count": 0,
|
||||
"false_factual_rate": 0.0,
|
||||
"notes": [
|
||||
"execution_error_count derived as http_error_count + limited_reason_category.execution_error",
|
||||
"false_factual_rate derived from false_factual_count/questions_total"
|
||||
]
|
||||
},
|
||||
"checks": [
|
||||
{
|
||||
"metric": "strict_pass_rate",
|
||||
"passed": false,
|
||||
"baseline": 1.0,
|
||||
"candidate": 0.96,
|
||||
"rule": "candidate >= baseline"
|
||||
},
|
||||
{
|
||||
"metric": "route_pass_rate",
|
||||
"passed": false,
|
||||
"baseline": 1.0,
|
||||
"candidate": 0.96,
|
||||
"rule": "candidate >= baseline"
|
||||
},
|
||||
{
|
||||
"metric": "execution_error_count",
|
||||
"passed": true,
|
||||
"baseline": 0,
|
||||
"candidate": 0,
|
||||
"rule": "candidate <= baseline"
|
||||
},
|
||||
{
|
||||
"metric": "false_factual_rate",
|
||||
"passed": true,
|
||||
"baseline": 0.0,
|
||||
"candidate": 0.0,
|
||||
"rule": "candidate <= baseline"
|
||||
},
|
||||
{
|
||||
"metric": "questions_total_match",
|
||||
"passed": true,
|
||||
"baseline": 25,
|
||||
"candidate": 25,
|
||||
"rule": "candidate == baseline"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -0,0 +1,23 @@
|
|||
{
|
||||
"generated_at": "2026-04-03T10:34:23",
|
||||
"total": 1,
|
||||
"passed": 1,
|
||||
"failed": 0,
|
||||
"results": [
|
||||
{
|
||||
"run_dir": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\runs\\2026-04-03_Address_Nightly_Regression_Post_AnchorHardening_R4\\2026-04-03_Address_Nightly_followup_25_10-31-38",
|
||||
"valid": true,
|
||||
"errors": [],
|
||||
"warnings": [],
|
||||
"metrics": {
|
||||
"run_id": "2026-04-03_Address_Nightly_followup_25_10-31-38",
|
||||
"questions_total": 25,
|
||||
"route_pass_rate": 0.96,
|
||||
"strict_pass_rate": 0.96,
|
||||
"http_error_count": 0,
|
||||
"rows_count": 25,
|
||||
"failures_count": 1
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -0,0 +1,33 @@
|
|||
# 2026-04-03_Address_Nightly_stress_102_10-20-55
|
||||
|
||||
Generated at: 2026-04-03T10:31:38
|
||||
Questions file: X:\1C\NDC_1C\docs\ADDRESS\question_sets\address_slang_stress_full_2026-04-02.json
|
||||
Backend URL: http://127.0.0.1:8787/api/assistant/message
|
||||
LLM: local / qwen2.5-14b-instruct-1m @ http://127.0.0.1:1234
|
||||
Strict policy: route
|
||||
|
||||
## Totals
|
||||
- questions_total: 102
|
||||
- ok_200_count: 102
|
||||
- semantic_pass_count: 102
|
||||
- semantic_pass_rate: 1.0
|
||||
- route_pass_count: 101
|
||||
- route_pass_rate: 0.9902
|
||||
- strict_pass_count: 101
|
||||
- strict_pass_rate: 0.9902
|
||||
- factual_count: 96
|
||||
- partial_coverage_count: 6
|
||||
- clarification_required_count: 0
|
||||
- http_error_count: 0
|
||||
- llm_decomposition_attempted_count: 102
|
||||
- llm_decomposition_applied_count: 77
|
||||
- llm_fallback_count: 0
|
||||
- llm_fallback_rate: 0.0
|
||||
- tool_gate_blocked_count: 0
|
||||
- tool_gate_blocked_rate: 0.0
|
||||
- avg_elapsed_ms: 6301.0
|
||||
|
||||
## Files
|
||||
- run_summary.json
|
||||
- full_live_results.json
|
||||
- failures_only.json
|
||||
|
|
@ -0,0 +1,182 @@
|
|||
[
|
||||
{
|
||||
"index": 64,
|
||||
"id": "X004",
|
||||
"question": "svk poka docy za 2020",
|
||||
"session": null,
|
||||
"session_id": "asst-2026-04-03_Address_Nightly_stress_102_10-20-55-x004",
|
||||
"status_code": 200,
|
||||
"ok": true,
|
||||
"elapsed_ms": 5570,
|
||||
"reply_type": "partial_coverage",
|
||||
"trace_id": "address-_TgCfGEmPJ",
|
||||
"assistant_reply": "Для точного адресного поиска не хватает обязательного якоря.\nПричина: не хватает обязательных фильтров.\nЧто нужно уточнить: уточните: counterparty.",
|
||||
"assistant_reply_first_line": "Для точного адресного поиска не хватает обязательного якоря.",
|
||||
"expected_intent": "list_documents_by_counterparty",
|
||||
"actual_intent": "list_documents_by_counterparty",
|
||||
"intent_match": true,
|
||||
"expected_mode": "address_query",
|
||||
"actual_mode": "address_query",
|
||||
"mode_match": true,
|
||||
"expected_reply_type": null,
|
||||
"reply_match": true,
|
||||
"semantic_pass": true,
|
||||
"route_pass": false,
|
||||
"route_health": "blocked_missing_anchor",
|
||||
"strict_policy": "route",
|
||||
"strict_pass": false,
|
||||
"selected_recipe": "address_documents_by_counterparty_v1",
|
||||
"missing_required_filters": [
|
||||
"counterparty"
|
||||
],
|
||||
"match_failure_stage": "none",
|
||||
"match_failure_reason": null,
|
||||
"rows_fetched": 0,
|
||||
"rows_matched": 0,
|
||||
"mcp_call_status": "skipped",
|
||||
"limited_reason_category": "missing_anchor",
|
||||
"llm_decomposition_attempted": true,
|
||||
"llm_decomposition_applied": true,
|
||||
"llm_decomposition_reason": "normalized_fragment_applied",
|
||||
"llm_canonical_candidate_detected": true,
|
||||
"fallback_rule_hit": null,
|
||||
"tool_gate_decision": "run_address_lane",
|
||||
"tool_gate_reason": "address_mode_classifier_detected",
|
||||
"predecompose_contract_intent": "list_documents_by_counterparty",
|
||||
"predecompose_contract_aggregation_profile": "list_lookup",
|
||||
"predecompose_contract_period_scope": "year",
|
||||
"debug_payload": {
|
||||
"trace_id": "address-_TgCfGEmPJ",
|
||||
"prompt_version": "address_query_runtime_v1",
|
||||
"schema_version": "address_query_runtime_v1",
|
||||
"fallback_type": "partial",
|
||||
"route_summary": null,
|
||||
"fragments": [],
|
||||
"requirements_extracted": [],
|
||||
"coverage_report": {
|
||||
"requirements_total": 0,
|
||||
"requirements_covered": 0,
|
||||
"requirements_uncovered": [],
|
||||
"requirements_partially_covered": [],
|
||||
"clarification_needed_for": [],
|
||||
"out_of_scope_requirements": []
|
||||
},
|
||||
"routes": [],
|
||||
"retrieval_status": [],
|
||||
"retrieval_results": [],
|
||||
"answer_grounding_check": {
|
||||
"status": "partial",
|
||||
"route_subject_match": true,
|
||||
"missing_requirements": [],
|
||||
"reasons": [
|
||||
"address_action_detected",
|
||||
"address_entity_detected",
|
||||
"document_list_signal_detected",
|
||||
"documents_by_counterparty_signal_detected",
|
||||
"missing_required_filters"
|
||||
],
|
||||
"why_included_summary": [],
|
||||
"selection_reason_summary": []
|
||||
},
|
||||
"dropped_intent_segments": [],
|
||||
"detected_mode": "address_query",
|
||||
"detected_mode_confidence": "high",
|
||||
"query_shape": "DOCUMENT_LIST",
|
||||
"query_shape_confidence": "medium",
|
||||
"detected_intent": "list_documents_by_counterparty",
|
||||
"detected_intent_confidence": "medium",
|
||||
"extracted_filters": {
|
||||
"sort": "period_desc",
|
||||
"limit": 20,
|
||||
"period_from": "2020-01-01",
|
||||
"period_to": "2020-12-31"
|
||||
},
|
||||
"missing_required_filters": [
|
||||
"counterparty"
|
||||
],
|
||||
"selected_recipe": "address_documents_by_counterparty_v1",
|
||||
"mcp_call_status_legacy": "skipped",
|
||||
"account_scope_mode": "strict",
|
||||
"account_scope_fallback_applied": false,
|
||||
"anchor_type": "unknown",
|
||||
"anchor_value_raw": null,
|
||||
"anchor_value_resolved": null,
|
||||
"resolver_confidence": "low",
|
||||
"ambiguity_count": 0,
|
||||
"match_failure_stage": "none",
|
||||
"match_failure_reason": null,
|
||||
"mcp_call_status": "skipped",
|
||||
"rows_fetched": 0,
|
||||
"raw_rows_received": 0,
|
||||
"rows_after_account_scope": 0,
|
||||
"rows_after_recipe_filter": 0,
|
||||
"rows_materialized": 0,
|
||||
"rows_matched": 0,
|
||||
"raw_row_keys_sample": [],
|
||||
"materialization_drop_reason": "none",
|
||||
"account_token_raw": null,
|
||||
"account_token_normalized": null,
|
||||
"account_scope_fields_checked": [
|
||||
"account_dt",
|
||||
"account_kt",
|
||||
"registrator",
|
||||
"analytics"
|
||||
],
|
||||
"account_scope_match_strategy": "account_code_regex_plus_alias_map_v1",
|
||||
"account_scope_drop_reason": "not_applicable",
|
||||
"runtime_readiness": "LIVE_QUERYABLE_WITH_LIMITS",
|
||||
"limited_reason_category": "missing_anchor",
|
||||
"response_type": "LIMITED_WITH_REASON",
|
||||
"execution_lane": "address_query",
|
||||
"llm_decomposition_applied": true,
|
||||
"llm_decomposition_attempted": true,
|
||||
"llm_provider_used": "local",
|
||||
"llm_decomposition_trace_id": "pxvaQYvAuA2UjS",
|
||||
"llm_decomposition_effective_message": "Покажи документы за 2020 год.",
|
||||
"llm_decomposition_reason": "normalized_fragment_applied",
|
||||
"llm_canonical_candidate_detected": true,
|
||||
"llm_predecompose_contract": {
|
||||
"schema_version": "address_llm_predecompose_contract_v1",
|
||||
"source_message": "svk poka docy za 2020",
|
||||
"canonical_message": "Покажи документы за 2020 год.",
|
||||
"mode": "address_query",
|
||||
"mode_confidence": "high",
|
||||
"query_shape": "DOCUMENT_LIST",
|
||||
"query_shape_confidence": "medium",
|
||||
"intent": "list_documents_by_counterparty",
|
||||
"intent_confidence": "medium",
|
||||
"entities": {
|
||||
"account": null,
|
||||
"counterparty": null,
|
||||
"contract": null,
|
||||
"document_type": null,
|
||||
"document_ref": null,
|
||||
"organization": null
|
||||
},
|
||||
"period": {
|
||||
"scope": "year",
|
||||
"period_from": "2020-01-01",
|
||||
"period_to": "2020-12-31",
|
||||
"as_of_date": null,
|
||||
"has_explicit_period": true
|
||||
},
|
||||
"aggregation_profile": "list_lookup"
|
||||
},
|
||||
"fallback_rule_hit": null,
|
||||
"sanitized_user_message": "svk poka docy za 2020",
|
||||
"tool_gate_decision": "run_address_lane",
|
||||
"tool_gate_reason": "address_mode_classifier_detected",
|
||||
"answer_structure_v11": null,
|
||||
"investigation_state_snapshot": null,
|
||||
"normalized": null,
|
||||
"normalizer_output": {
|
||||
"trace_id": "pxvaQYvAuA2UjS",
|
||||
"prompt_version": "normalizer_v2_0_2",
|
||||
"applied": true,
|
||||
"effective_message": "Покажи документы за 2020 год."
|
||||
}
|
||||
},
|
||||
"error_code": null,
|
||||
"error_message": null
|
||||
}
|
||||
]
|
||||
File diff suppressed because it is too large
Load Diff
|
|
@ -0,0 +1,106 @@
|
|||
# Response Audit: 2026-04-03_Address_Nightly_stress_102_10-20-55
|
||||
|
||||
| id | strict | route_health | reply_type | intent | limited_reason | question | assistant_first_line |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| S001 | True | ok_or_factual | factual | list_documents_by_counterparty | None | свк доки за 2020 покаж пж | Собран список документов по контрагенту (live address lane). |
|
||||
| S002 | True | ok_or_factual | factual | list_documents_by_counterparty | None | покеж все доки по свк за 20 год | Собран список документов по контрагенту (live address lane). |
|
||||
| S003 | True | ok_or_factual | factual | list_documents_by_counterparty | None | дай список документов по свк за 2020 | Собран список документов по контрагенту (live address lane). |
|
||||
| S004 | True | ok_or_factual | factual | list_documents_by_counterparty | None | какие у свк есть доки за 2020? | Собран список документов по контрагенту (live address lane). |
|
||||
| S005 | True | ok_or_factual | factual | list_documents_by_counterparty | None | че по свк по докам за 2020 год | Собран список документов по контрагенту (live address lane). |
|
||||
| S006 | True | ok_or_factual | factual | list_documents_by_counterparty | None | ёпт покажи доки по свк за 2020 | Собран список документов по контрагенту (live address lane). |
|
||||
| S007 | True | ok_or_factual | factual | list_documents_by_counterparty | None | бля, доки свк 2020 вывали | Собран список документов по контрагенту (live address lane). |
|
||||
| S008 | True | ok_or_factual | factual | list_documents_by_counterparty | None | svk docs for 2020 please | Собран список документов по контрагенту (live address lane). |
|
||||
| S009 | True | ok_or_factual | factual | list_documents_by_counterparty | None | docs by counterparty svk year 2020 | Собран список документов по контрагенту (live address lane). |
|
||||
| S010 | True | ok_or_factual | factual | list_documents_by_counterparty | None | по контрагенту свк покаж документы за 2020 | Собран список документов по контрагенту (live address lane). |
|
||||
| S011 | True | ok_or_factual | factual | list_documents_by_counterparty | None | плс доки по свк 2020 | Собран список документов по контрагенту (live address lane). |
|
||||
| S012 | True | ok_or_factual | factual | list_documents_by_counterparty | None | свк док за 2020 гони | Собран список документов по контрагенту (live address lane). |
|
||||
| S013 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | по свк банковские операции за 2020 покаж | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| S014 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | свк выписка банка 2020 | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| S015 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | платежи по свк за 2020 выведи | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| S016 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | оплаты по свк за 2020 | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| S017 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | свк банк опер за 2020 | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| S018 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | че у свк по банку за 2020 | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| S019 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | свк списания/поступления за 2020 | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| S020 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | bank ops by svk for 2020 | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| S021 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | transactions by counterparty svk 2020 | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| S022 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | бля, по свк банковские платежки 2020 | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| S023 | True | ok_or_factual | factual | account_balance_snapshot | None | сальдо по счету 60.01 на дату 2020-07-31 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| S024 | True | ok_or_factual | factual | account_balance_snapshot | None | остаток по счету 60.01 на 2020.05 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| S025 | True | ok_or_factual | factual | account_balance_snapshot | None | покажи остаток по 60.01 за май 2020 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| S026 | True | ok_or_factual | factual | account_balance_snapshot | None | счет 60 остаток на май 2020 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| S027 | True | ok_or_factual | factual | account_balance_snapshot | None | баланс account 60.01 as of 2020-07-31 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| S028 | True | ok_or_factual | factual | account_balance_snapshot | None | 60.01 saldo на 31.07.2020 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| S029 | True | ok_or_factual | factual | account_balance_snapshot | None | сколько на счете 60 на конец 2020-05 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| S030 | True | ok_or_factual | factual | account_balance_snapshot | None | дай остаток 60.02 на 2020-12-31 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| S031 | True | partial_non_blocking | partial_coverage | account_balance_snapshot | empty_match | епт, сальдо по счету 62.01 на 2020-12-31 | В live-данных по текущему фильтру записи не найдены. |
|
||||
| S032 | True | ok_or_factual | factual | account_balance_snapshot | None | плс остаток по счету 51 на 2020-06-30 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| S033 | True | ok_or_factual | factual | account_balance_snapshot | None | как там по счету 60.01 на 2020.07? | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| S034 | True | ok_or_factual | factual | account_balance_snapshot | None | сальдо 60 на 2020 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| S035 | True | ok_or_factual | factual | documents_forming_balance | None | какие документы формируют остаток по счету 60.01 на 2020-07-31 | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| S036 | True | ok_or_factual | factual | documents_forming_balance | None | из чего состоит остаток по счету 60 на 2020.05 | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| S037 | True | ok_or_factual | factual | documents_forming_balance | None | раскрой остаток 60.01 по документам на конец июля 2020 | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| S038 | True | ok_or_factual | factual | documents_forming_balance | None | доки под остатком 60.01 на 2020-07-31 | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| S039 | True | ok_or_factual | factual | documents_forming_balance | None | documents forming balance for account 60.01 as of 2020-07-31 | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| S040 | True | ok_or_factual | factual | documents_forming_balance | None | бля, какие доки формируют остаток 60.01 на 2020-05-31 | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| S041 | True | partial_non_blocking | partial_coverage | documents_forming_balance | empty_match | по счету 62.01 раскрой остаток документами на 2020-12-31 | В live-данных по текущему фильтру записи не найдены. |
|
||||
| S042 | True | ok_or_factual | factual | documents_forming_balance | None | из чего остаток по счету 51 на 2020-06-30 | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| S043 | True | ok_or_factual | factual | list_documents_by_contract | None | покажи документы по договору 19/15 за 2020 | Период сохранен. Глубина live-выборки автоматически расширена до 1000 строк. |
|
||||
| S044 | True | ok_or_factual | factual | list_documents_by_contract | None | доки по дог. 19/15 за 2020 | Период сохранен. Глубина live-выборки автоматически расширена до 1000 строк. |
|
||||
| S045 | True | ok_or_factual | factual | list_documents_by_contract | None | какие документы по контракту 19/15 | Собран список документов по договору (live address lane). |
|
||||
| S046 | True | ok_or_factual | factual | list_documents_by_contract | None | docs by contract 19/15 year 2020 | Период сохранен. Глубина live-выборки автоматически расширена до 1000 строк. |
|
||||
| S047 | True | ok_or_factual | factual | list_documents_by_contract | None | бля, покажи по договору 19/15 доки | Собран список документов по договору (live address lane). |
|
||||
| S048 | True | ok_or_factual | factual | list_documents_by_contract | None | договор 19/15 документы | Собран список документов по договору (live address lane). |
|
||||
| S049 | True | ok_or_factual | factual | list_documents_by_contract | None | плс документы договор №19/15 | Собран список документов по договору (live address lane). |
|
||||
| S050 | True | ok_or_factual | factual | list_documents_by_contract | None | покеж документы по 19/15 | Собран список документов по договору (live address lane). |
|
||||
| S051 | True | ok_or_factual | factual | bank_operations_by_contract | None | банковские операции по договору 19/15 | Собран список банковских операций по договору (live address lane). |
|
||||
| S052 | True | ok_or_factual | factual | bank_operations_by_contract | None | платежи по договору 19/15 | Собран список банковских операций по договору (live address lane). |
|
||||
| S053 | True | ok_or_factual | factual | bank_operations_by_contract | None | выписка по договору 19/15 | Собран список банковских операций по договору (live address lane). |
|
||||
| S054 | True | ok_or_factual | factual | bank_operations_by_contract | None | bank ops by contract 19/15 | Собран список банковских операций по договору (live address lane). |
|
||||
| S055 | True | ok_or_factual | factual | bank_operations_by_contract | None | ёпт, покажи банк операции по дог 19/15 | Собран список банковских операций по договору (live address lane). |
|
||||
| S056 | True | ok_or_factual | factual | bank_operations_by_contract | None | по контракту 19/15 платежки | Собран список банковских операций по договору (live address lane). |
|
||||
| S057 | True | ok_or_factual | factual | account_balance_snapshot | None | покажи сальдо по счету 60 за 2020 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| S058 | True | ok_or_factual | factual | account_balance_snapshot | None | а теперь только май 2020 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| S059 | True | ok_or_factual | factual | account_balance_snapshot | None | ок, теперь 2020.05 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| S060 | True | ok_or_factual | factual | account_balance_snapshot | None | теперь на 2020-07-31 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| X001 | True | ok_or_factual | factual | list_documents_by_counterparty | None | свк доки кинь за 2020 | Собран список документов по контрагенту (live address lane). |
|
||||
| X002 | True | ok_or_factual | factual | list_documents_by_counterparty | None | по свк все документы за 20й год | Собран список документов по контрагенту (live address lane). |
|
||||
| X003 | True | ok_or_factual | factual | list_documents_by_counterparty | None | пж вывали доки контрагент свк 2020 | Собран список документов по контрагенту (live address lane). |
|
||||
| X004 | False | blocked_missing_anchor | partial_coverage | list_documents_by_counterparty | missing_anchor | svk poka docy za 2020 | Для точного адресного поиска не хватает обязательного якоря. |
|
||||
| X005 | True | ok_or_factual | factual | list_documents_by_counterparty | None | какие доки по компании свк за 2020 | Собран список документов по контрагенту (live address lane). |
|
||||
| X006 | True | ok_or_factual | factual | list_documents_by_counterparty | None | бля покаж свк документы 2020 | Собран список документов по контрагенту (live address lane). |
|
||||
| X007 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | свк банк операции за 2020 | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| X008 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | по свк выписку банка за 2020 выведи | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| X009 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | свк платежки за 2020 | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| X010 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | bank ops svk 2020 | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| X011 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | транзакции по свк за 2020 | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| X012 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | ёпт по свк че по банку 2020 | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| X013 | True | ok_or_factual | factual | account_balance_snapshot | None | остаток по 60.01 на 2020-07-31 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| X014 | True | ok_or_factual | factual | account_balance_snapshot | None | сальдо 60.01 на 31.07.2020 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| X015 | True | ok_or_factual | factual | account_balance_snapshot | None | что на счете 60 на 2020.05 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| X016 | True | ok_or_factual | factual | account_balance_snapshot | None | дай остаток 62.01 на 2020 май | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| X017 | True | partial_non_blocking | partial_coverage | account_balance_snapshot | empty_match | saldo account 60 as of 2020-12-31 | В live-данных по текущему фильтру записи не найдены. |
|
||||
| X018 | True | ok_or_factual | factual | account_balance_snapshot | None | плиз остаток по счету 51 на 2020-06-30 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| X019 | True | partial_non_blocking | partial_coverage | account_balance_snapshot | empty_match | скока по 60.02 на конец 2020-12 | В live-данных по текущему фильтру записи не найдены. |
|
||||
| X020 | True | ok_or_factual | factual | account_balance_snapshot | None | бля сальдо 60.01 на 2020-05-31 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| X021 | True | ok_or_factual | factual | documents_forming_balance | None | раскрой остаток 60.01 по документам на конец июля 2020 | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| X022 | True | ok_or_factual | factual | documents_forming_balance | None | доки под остатком 60.01 на 2020-07-31 | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| X023 | True | ok_or_factual | factual | documents_forming_balance | None | из чего остаток 60.01 на 2020-07-31 | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| X024 | True | ok_or_factual | factual | documents_forming_balance | None | какие документы формируют остаток по 60.01 на 2020-07-31 | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| X025 | True | ok_or_factual | factual | documents_forming_balance | None | docs forming balance 60.01 as of 2020-07-31 | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| X026 | True | partial_non_blocking | partial_coverage | documents_forming_balance | empty_match | раскидай остаток 62.01 по докам на 2020-12-31 | В live-данных по текущему фильтру записи не найдены. |
|
||||
| X027 | True | ok_or_factual | factual | documents_forming_balance | None | по счету 51 раскрой остаток документами на 2020-06-30 | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| X028 | True | ok_or_factual | factual | documents_forming_balance | None | бля из чего состоит остаток по 60.01 на 2020-05-31 | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| X029 | True | ok_or_factual | factual | list_documents_by_contract | None | доки по договору 19/15 за 2020 | Период сохранен. Глубина live-выборки автоматически расширена до 1000 строк. |
|
||||
| X030 | True | ok_or_factual | factual | list_documents_by_contract | None | покажи документы контракт 19/15 | Собран список документов по договору (live address lane). |
|
||||
| X031 | True | ok_or_factual | factual | list_documents_by_contract | None | docs by contract 19/15 2020 | Период сохранен. Глубина live-выборки автоматически расширена до 1000 строк. |
|
||||
| X032 | True | ok_or_factual | factual | list_documents_by_contract | None | по дог 19/15 какие документы | Собран список документов по договору (live address lane). |
|
||||
| X033 | True | ok_or_factual | factual | list_documents_by_contract | None | договор 19/15 выведи доки | Собран список документов по договору (live address lane). |
|
||||
| X034 | True | ok_or_factual | factual | list_documents_by_contract | None | нах покажи документы по 19/15 | Собран список документов по договору (live address lane). |
|
||||
| X035 | True | ok_or_factual | factual | bank_operations_by_contract | None | банк операции по договору 19/15 | Собран список банковских операций по договору (live address lane). |
|
||||
| X036 | True | ok_or_factual | factual | bank_operations_by_contract | None | платежки по дог 19/15 | Собран список банковских операций по договору (live address lane). |
|
||||
| X037 | True | ok_or_factual | factual | bank_operations_by_contract | None | выписка по контракту 19/15 | Собран список банковских операций по договору (live address lane). |
|
||||
| X038 | True | ok_or_factual | factual | bank_operations_by_contract | None | bank ops by contract 19/15 | Собран список банковских операций по договору (live address lane). |
|
||||
| X039 | True | ok_or_factual | factual | bank_operations_by_contract | None | покажи банк опер по дог 19/15 пж | Собран список банковских операций по договору (live address lane). |
|
||||
| X040 | True | ok_or_factual | factual | bank_operations_by_contract | None | епт транзакции договор 19/15 | Собран список банковских операций по договору (live address lane). |
|
||||
| X041 | True | ok_or_factual | factual | account_balance_snapshot | None | покажи сальдо по счету 60 за 2020 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| X042 | True | ok_or_factual | factual | account_balance_snapshot | None | а теперь только май 2020 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
|
|
@ -0,0 +1,135 @@
|
|||
{
|
||||
"run_id": "2026-04-03_Address_Nightly_stress_102_10-20-55",
|
||||
"generated_at": "2026-04-03T10:31:38",
|
||||
"source_questions_file": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\question_sets\\address_slang_stress_full_2026-04-02.json",
|
||||
"backend_url": "http://127.0.0.1:8787/api/assistant/message",
|
||||
"llm_provider": "local",
|
||||
"llm_model": "qwen2.5-14b-instruct-1m",
|
||||
"llm_base_url": "http://127.0.0.1:1234",
|
||||
"strict_policy": "route",
|
||||
"totals": {
|
||||
"questions_total": 102,
|
||||
"ok_200_count": 102,
|
||||
"semantic_pass_count": 102,
|
||||
"semantic_pass_rate": 1.0,
|
||||
"route_pass_count": 101,
|
||||
"route_pass_rate": 0.9902,
|
||||
"strict_pass_count": 101,
|
||||
"strict_pass_rate": 0.9902,
|
||||
"factual_count": 96,
|
||||
"partial_coverage_count": 6,
|
||||
"clarification_required_count": 0,
|
||||
"http_error_count": 0,
|
||||
"llm_decomposition_attempted_count": 102,
|
||||
"llm_decomposition_applied_count": 77,
|
||||
"llm_fallback_count": 0,
|
||||
"llm_fallback_rate": 0.0,
|
||||
"tool_gate_blocked_count": 0,
|
||||
"tool_gate_blocked_rate": 0.0,
|
||||
"avg_elapsed_ms": 6301.0
|
||||
},
|
||||
"distributions": {
|
||||
"reply_type": {
|
||||
"factual": 96,
|
||||
"partial_coverage": 6
|
||||
},
|
||||
"actual_intent": {
|
||||
"list_documents_by_counterparty": 18,
|
||||
"bank_operations_by_counterparty": 16,
|
||||
"account_balance_snapshot": 26,
|
||||
"documents_forming_balance": 16,
|
||||
"list_documents_by_contract": 14,
|
||||
"bank_operations_by_contract": 12
|
||||
},
|
||||
"actual_mode": {
|
||||
"address_query": 102
|
||||
},
|
||||
"mcp_call_status": {
|
||||
"matched_non_empty": 96,
|
||||
"no_raw_rows": 5,
|
||||
"skipped": 1
|
||||
},
|
||||
"limited_reason_category": {
|
||||
"empty_match": 5,
|
||||
"missing_anchor": 1
|
||||
},
|
||||
"route_health": {
|
||||
"ok_or_factual": 96,
|
||||
"partial_non_blocking": 5,
|
||||
"blocked_missing_anchor": 1
|
||||
},
|
||||
"tool_gate_decision": {
|
||||
"run_address_lane": 102
|
||||
},
|
||||
"tool_gate_reason": {
|
||||
"address_mode_classifier_detected": 98,
|
||||
"llm_canonical_candidate_detected": 4
|
||||
}
|
||||
},
|
||||
"address_llm_predecompose_metrics": {
|
||||
"overall": {
|
||||
"llm_attempted": 102,
|
||||
"llm_applied": 77,
|
||||
"fallback_used": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"tool_gate_blocked": 0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"by_intent": {
|
||||
"list_documents_by_counterparty": {
|
||||
"total": 18,
|
||||
"llm_attempted": 18,
|
||||
"llm_applied": 12,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"bank_operations_by_counterparty": {
|
||||
"total": 16,
|
||||
"llm_attempted": 16,
|
||||
"llm_applied": 11,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"account_balance_snapshot": {
|
||||
"total": 26,
|
||||
"llm_attempted": 26,
|
||||
"llm_applied": 25,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"documents_forming_balance": {
|
||||
"total": 16,
|
||||
"llm_attempted": 16,
|
||||
"llm_applied": 11,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"list_documents_by_contract": {
|
||||
"total": 14,
|
||||
"llm_attempted": 14,
|
||||
"llm_applied": 10,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"bank_operations_by_contract": {
|
||||
"total": 12,
|
||||
"llm_attempted": 12,
|
||||
"llm_applied": 8,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,67 @@
|
|||
{
|
||||
"generated_at": "2026-04-03T10:31:38",
|
||||
"overall_pass": false,
|
||||
"baseline": {
|
||||
"summary_path": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\runs\\2026-04-02_Address_Slang_Live_Stress_2026-04-02_12-57-27\\run_summary.json",
|
||||
"run_id": "2026-04-02_Address_Slang_Live_Stress_2026-04-02_12-57-27",
|
||||
"questions_total": 102,
|
||||
"strict_pass_rate": 1.0,
|
||||
"route_pass_rate": 1.0,
|
||||
"execution_error_count": 0,
|
||||
"false_factual_rate": 0.0,
|
||||
"notes": [
|
||||
"execution_error_count derived as http_error_count + limited_reason_category.execution_error",
|
||||
"false_factual_rate derived from false_factual_count/questions_total"
|
||||
]
|
||||
},
|
||||
"candidate": {
|
||||
"summary_path": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\runs\\2026-04-03_Address_Nightly_Regression_Post_AnchorHardening_R4\\2026-04-03_Address_Nightly_stress_102_10-20-55\\run_summary.json",
|
||||
"run_id": "2026-04-03_Address_Nightly_stress_102_10-20-55",
|
||||
"questions_total": 102,
|
||||
"strict_pass_rate": 0.9902,
|
||||
"route_pass_rate": 0.9902,
|
||||
"execution_error_count": 0,
|
||||
"false_factual_rate": 0.0,
|
||||
"notes": [
|
||||
"execution_error_count derived as http_error_count + limited_reason_category.execution_error",
|
||||
"false_factual_rate derived from false_factual_count/questions_total"
|
||||
]
|
||||
},
|
||||
"checks": [
|
||||
{
|
||||
"metric": "strict_pass_rate",
|
||||
"passed": false,
|
||||
"baseline": 1.0,
|
||||
"candidate": 0.9902,
|
||||
"rule": "candidate >= baseline"
|
||||
},
|
||||
{
|
||||
"metric": "route_pass_rate",
|
||||
"passed": false,
|
||||
"baseline": 1.0,
|
||||
"candidate": 0.9902,
|
||||
"rule": "candidate >= baseline"
|
||||
},
|
||||
{
|
||||
"metric": "execution_error_count",
|
||||
"passed": true,
|
||||
"baseline": 0,
|
||||
"candidate": 0,
|
||||
"rule": "candidate <= baseline"
|
||||
},
|
||||
{
|
||||
"metric": "false_factual_rate",
|
||||
"passed": true,
|
||||
"baseline": 0.0,
|
||||
"candidate": 0.0,
|
||||
"rule": "candidate <= baseline"
|
||||
},
|
||||
{
|
||||
"metric": "questions_total_match",
|
||||
"passed": true,
|
||||
"baseline": 102,
|
||||
"candidate": 102,
|
||||
"rule": "candidate == baseline"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -0,0 +1,23 @@
|
|||
{
|
||||
"generated_at": "2026-04-03T10:31:38",
|
||||
"total": 1,
|
||||
"passed": 1,
|
||||
"failed": 0,
|
||||
"results": [
|
||||
{
|
||||
"run_dir": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\runs\\2026-04-03_Address_Nightly_Regression_Post_AnchorHardening_R4\\2026-04-03_Address_Nightly_stress_102_10-20-55",
|
||||
"valid": true,
|
||||
"errors": [],
|
||||
"warnings": [],
|
||||
"metrics": {
|
||||
"run_id": "2026-04-03_Address_Nightly_stress_102_10-20-55",
|
||||
"questions_total": 102,
|
||||
"route_pass_rate": 0.9902,
|
||||
"strict_pass_rate": 0.9902,
|
||||
"http_error_count": 0,
|
||||
"rows_count": 102,
|
||||
"failures_count": 1
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -0,0 +1,26 @@
|
|||
# 2026-04-03_Address_Nightly_Regression_Post_AnchorHardening_R4
|
||||
|
||||
Generated at: 2026-04-03T10:34:23
|
||||
Dry run: False
|
||||
Strict policy: route
|
||||
Overall: FAIL
|
||||
|
||||
## Packs
|
||||
### stress_102
|
||||
- run_id: 2026-04-03_Address_Nightly_stress_102_10-20-55
|
||||
- runner_ok: True
|
||||
- validator_ok: True
|
||||
- comparator_ok: False
|
||||
- run_dir: X:\1C\NDC_1C\docs\ADDRESS\runs\2026-04-03_Address_Nightly_Regression_Post_AnchorHardening_R4\2026-04-03_Address_Nightly_stress_102_10-20-55
|
||||
- errors:
|
||||
- comparator failed with exit code 1
|
||||
|
||||
### followup_25
|
||||
- run_id: 2026-04-03_Address_Nightly_followup_25_10-31-38
|
||||
- runner_ok: True
|
||||
- validator_ok: True
|
||||
- comparator_ok: False
|
||||
- run_dir: X:\1C\NDC_1C\docs\ADDRESS\runs\2026-04-03_Address_Nightly_Regression_Post_AnchorHardening_R4\2026-04-03_Address_Nightly_followup_25_10-31-38
|
||||
- errors:
|
||||
- comparator failed with exit code 1
|
||||
|
||||
|
|
@ -0,0 +1,37 @@
|
|||
{
|
||||
"nightly_run_id": "2026-04-03_Address_Nightly_Regression_Post_AnchorHardening_R4",
|
||||
"generated_at": "2026-04-03T10:34:23",
|
||||
"dry_run": false,
|
||||
"overall_ok": false,
|
||||
"strict_policy": "route",
|
||||
"packs": [
|
||||
{
|
||||
"pack": "stress_102",
|
||||
"run_id": "2026-04-03_Address_Nightly_stress_102_10-20-55",
|
||||
"questions_file": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\question_sets\\address_slang_stress_full_2026-04-02.json",
|
||||
"baseline_summary": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\runs\\2026-04-02_Address_Slang_Live_Stress_2026-04-02_12-57-27\\run_summary.json",
|
||||
"run_dir": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\runs\\2026-04-03_Address_Nightly_Regression_Post_AnchorHardening_R4\\2026-04-03_Address_Nightly_stress_102_10-20-55",
|
||||
"runner_ok": true,
|
||||
"validator_ok": true,
|
||||
"comparator_ok": false,
|
||||
"errors": [
|
||||
"comparator failed with exit code 1"
|
||||
],
|
||||
"validator_report": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\runs\\2026-04-03_Address_Nightly_Regression_Post_AnchorHardening_R4\\2026-04-03_Address_Nightly_stress_102_10-20-55_validator_report.json"
|
||||
},
|
||||
{
|
||||
"pack": "followup_25",
|
||||
"run_id": "2026-04-03_Address_Nightly_followup_25_10-31-38",
|
||||
"questions_file": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\question_sets\\address_followup_context_chains_2026-04-02.json",
|
||||
"baseline_summary": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\runs\\2026-04-02_Address_Followup_Context_Chains_2026-04-02_19-15-Run5\\run_summary.json",
|
||||
"run_dir": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\runs\\2026-04-03_Address_Nightly_Regression_Post_AnchorHardening_R4\\2026-04-03_Address_Nightly_followup_25_10-31-38",
|
||||
"runner_ok": true,
|
||||
"validator_ok": true,
|
||||
"comparator_ok": false,
|
||||
"errors": [
|
||||
"comparator failed with exit code 1"
|
||||
],
|
||||
"validator_report": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\runs\\2026-04-03_Address_Nightly_Regression_Post_AnchorHardening_R4\\2026-04-03_Address_Nightly_followup_25_10-31-38_validator_report.json"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -0,0 +1,33 @@
|
|||
# 2026-04-03_Address_Nightly_followup_25_10-55-41
|
||||
|
||||
Generated at: 2026-04-03T10:58:22
|
||||
Questions file: X:\1C\NDC_1C\docs\ADDRESS\question_sets\address_followup_context_chains_2026-04-02.json
|
||||
Backend URL: http://127.0.0.1:8787/api/assistant/message
|
||||
LLM: local / qwen2.5-14b-instruct-1m @ http://127.0.0.1:1234
|
||||
Strict policy: route
|
||||
|
||||
## Totals
|
||||
- questions_total: 25
|
||||
- ok_200_count: 25
|
||||
- semantic_pass_count: 25
|
||||
- semantic_pass_rate: 1.0
|
||||
- route_pass_count: 25
|
||||
- route_pass_rate: 1.0
|
||||
- strict_pass_count: 25
|
||||
- strict_pass_rate: 1.0
|
||||
- factual_count: 23
|
||||
- partial_coverage_count: 2
|
||||
- clarification_required_count: 0
|
||||
- http_error_count: 0
|
||||
- llm_decomposition_attempted_count: 25
|
||||
- llm_decomposition_applied_count: 22
|
||||
- llm_fallback_count: 0
|
||||
- llm_fallback_rate: 0.0
|
||||
- tool_gate_blocked_count: 0
|
||||
- tool_gate_blocked_rate: 0.0
|
||||
- avg_elapsed_ms: 6421.8
|
||||
|
||||
## Files
|
||||
- run_summary.json
|
||||
- full_live_results.json
|
||||
- failures_only.json
|
||||
|
|
@ -0,0 +1 @@
|
|||
[]
|
||||
File diff suppressed because it is too large
Load Diff
|
|
@ -0,0 +1,29 @@
|
|||
# Response Audit: 2026-04-03_Address_Nightly_followup_25_10-55-41
|
||||
|
||||
| id | strict | route_health | reply_type | intent | limited_reason | question | assistant_first_line |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| F001 | True | ok_or_factual | factual | list_documents_by_counterparty | None | Покажи документы по СВК за 2020 | Собран список документов по контрагенту (live address lane). |
|
||||
| F002 | True | ok_or_factual | factual | list_documents_by_counterparty | None | а теперь только за май 2020 | По окну 2020-05-01..2020-05-31 строк не найдено; показаны ближайшие доступные данные 2020-07-27..2021-11-10. |
|
||||
| F003 | True | ok_or_factual | factual | account_balance_snapshot | None | а по счету 60.01 на ту же дату | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| F004 | True | ok_or_factual | factual | account_balance_snapshot | None | а на ту же дату по 62.01 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| F005 | True | ok_or_factual | factual | documents_forming_balance | None | раскрой остаток по 62.01 документами на ту же дату | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| F006 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | Покажи банковские операции по СВК за 2020 | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| F007 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | а теперь только ноябрь 2020 | Период сохранен. Глубина live-выборки автоматически расширена до 1000 строк. |
|
||||
| F008 | True | ok_or_factual | factual | list_documents_by_counterparty | None | а теперь документы по нему | Период сохранен. Глубина live-выборки автоматически расширена до 1000 строк. |
|
||||
| F009 | True | ok_or_factual | factual | account_balance_snapshot | None | и остаток по 51 на ту же дату | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| F010 | True | ok_or_factual | factual | account_balance_snapshot | None | а теперь только май 2020 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| F011 | True | partial_non_blocking | partial_coverage | account_balance_snapshot | empty_match | Покажи сальдо по счету 60 на 2020-12-31 | В live-данных по текущему фильтру записи не найдены. |
|
||||
| F012 | True | ok_or_factual | factual | account_balance_snapshot | None | а теперь только май 2020 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| F013 | True | ok_or_factual | factual | account_balance_snapshot | None | а на ту же дату по 62 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| F014 | True | ok_or_factual | factual | documents_forming_balance | None | раскрой 62 документами на ту же дату | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| F015 | True | partial_non_blocking | partial_coverage | account_balance_snapshot | empty_match | вернись на 2020-12-31 по 60 | В live-данных по текущему фильтру записи не найдены. |
|
||||
| F016 | True | ok_or_factual | factual | list_documents_by_contract | None | Покажи документы по договору 19/15 за 2020 | Период сохранен. Глубина live-выборки автоматически расширена до 1000 строк. |
|
||||
| F017 | True | ok_or_factual | factual | bank_operations_by_contract | None | а теперь банковские операции | Период сохранен. Глубина live-выборки автоматически расширена до 1000 строк. |
|
||||
| F018 | True | ok_or_factual | factual | bank_operations_by_contract | None | а теперь только за 2020 | Период сохранен. Глубина live-выборки автоматически расширена до 1000 строк. |
|
||||
| F019 | True | ok_or_factual | factual | open_items_by_counterparty_or_contract | None | есть ли открытые позиции по договору 19/15 на 2020-12-31 | Период сохранен. Глубина live-выборки автоматически расширена до 1000 строк. |
|
||||
| F020 | True | ok_or_factual | factual | list_documents_by_contract | None | покажи документы по этому же договору | Собран список документов по договору (live address lane). |
|
||||
| F021 | True | ok_or_factual | factual | bank_operations_by_contract | None | Покажи банковские операции по договору 19/15 | Собран список банковских операций по договору (live address lane). |
|
||||
| F022 | True | ok_or_factual | factual | open_items_by_counterparty_or_contract | None | а теперь открытые позиции по нему | Период сохранен. Глубина live-выборки автоматически расширена до 1000 строк. |
|
||||
| F023 | True | ok_or_factual | factual | open_items_by_counterparty_or_contract | None | а теперь на 2020-12-31 | Период сохранен. Глубина live-выборки автоматически расширена до 1000 строк. |
|
||||
| F024 | True | ok_or_factual | factual | account_balance_snapshot | None | а по счету 51 на ту же дату | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| F025 | True | ok_or_factual | factual | list_documents_by_contract | None | а документы по этому же договору за тот же период | Собран список документов по договору (live address lane). |
|
||||
|
|
@ -0,0 +1,142 @@
|
|||
{
|
||||
"run_id": "2026-04-03_Address_Nightly_followup_25_10-55-41",
|
||||
"generated_at": "2026-04-03T10:58:22",
|
||||
"source_questions_file": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\question_sets\\address_followup_context_chains_2026-04-02.json",
|
||||
"backend_url": "http://127.0.0.1:8787/api/assistant/message",
|
||||
"llm_provider": "local",
|
||||
"llm_model": "qwen2.5-14b-instruct-1m",
|
||||
"llm_base_url": "http://127.0.0.1:1234",
|
||||
"strict_policy": "route",
|
||||
"totals": {
|
||||
"questions_total": 25,
|
||||
"ok_200_count": 25,
|
||||
"semantic_pass_count": 25,
|
||||
"semantic_pass_rate": 1.0,
|
||||
"route_pass_count": 25,
|
||||
"route_pass_rate": 1.0,
|
||||
"strict_pass_count": 25,
|
||||
"strict_pass_rate": 1.0,
|
||||
"factual_count": 23,
|
||||
"partial_coverage_count": 2,
|
||||
"clarification_required_count": 0,
|
||||
"http_error_count": 0,
|
||||
"llm_decomposition_attempted_count": 25,
|
||||
"llm_decomposition_applied_count": 22,
|
||||
"llm_fallback_count": 0,
|
||||
"llm_fallback_rate": 0.0,
|
||||
"tool_gate_blocked_count": 0,
|
||||
"tool_gate_blocked_rate": 0.0,
|
||||
"avg_elapsed_ms": 6421.8
|
||||
},
|
||||
"distributions": {
|
||||
"reply_type": {
|
||||
"factual": 23,
|
||||
"partial_coverage": 2
|
||||
},
|
||||
"actual_intent": {
|
||||
"list_documents_by_counterparty": 3,
|
||||
"account_balance_snapshot": 9,
|
||||
"documents_forming_balance": 2,
|
||||
"bank_operations_by_counterparty": 2,
|
||||
"list_documents_by_contract": 3,
|
||||
"bank_operations_by_contract": 3,
|
||||
"open_items_by_counterparty_or_contract": 3
|
||||
},
|
||||
"actual_mode": {
|
||||
"address_query": 25
|
||||
},
|
||||
"mcp_call_status": {
|
||||
"matched_non_empty": 23,
|
||||
"no_raw_rows": 2
|
||||
},
|
||||
"limited_reason_category": {
|
||||
"empty_match": 2
|
||||
},
|
||||
"route_health": {
|
||||
"ok_or_factual": 23,
|
||||
"partial_non_blocking": 2
|
||||
},
|
||||
"tool_gate_decision": {
|
||||
"run_address_lane": 25
|
||||
},
|
||||
"tool_gate_reason": {
|
||||
"address_mode_classifier_detected": 16,
|
||||
"llm_canonical_candidate_detected": 9
|
||||
}
|
||||
},
|
||||
"address_llm_predecompose_metrics": {
|
||||
"overall": {
|
||||
"llm_attempted": 25,
|
||||
"llm_applied": 22,
|
||||
"fallback_used": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"tool_gate_blocked": 0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"by_intent": {
|
||||
"list_documents_by_counterparty": {
|
||||
"total": 3,
|
||||
"llm_attempted": 3,
|
||||
"llm_applied": 3,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"account_balance_snapshot": {
|
||||
"total": 9,
|
||||
"llm_attempted": 9,
|
||||
"llm_applied": 7,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"documents_forming_balance": {
|
||||
"total": 2,
|
||||
"llm_attempted": 2,
|
||||
"llm_applied": 2,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"bank_operations_by_counterparty": {
|
||||
"total": 2,
|
||||
"llm_attempted": 2,
|
||||
"llm_applied": 2,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"list_documents_by_contract": {
|
||||
"total": 3,
|
||||
"llm_attempted": 3,
|
||||
"llm_applied": 3,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"bank_operations_by_contract": {
|
||||
"total": 3,
|
||||
"llm_attempted": 3,
|
||||
"llm_applied": 2,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"open_items_by_counterparty_or_contract": {
|
||||
"total": 3,
|
||||
"llm_attempted": 3,
|
||||
"llm_applied": 3,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,67 @@
|
|||
{
|
||||
"generated_at": "2026-04-03T10:58:22",
|
||||
"overall_pass": true,
|
||||
"baseline": {
|
||||
"summary_path": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\runs\\2026-04-02_Address_Followup_Context_Chains_2026-04-02_19-15-Run5\\run_summary.json",
|
||||
"run_id": "2026-04-02_Address_Followup_Context_Chains_2026-04-02_19-15-Run5",
|
||||
"questions_total": 25,
|
||||
"strict_pass_rate": 1.0,
|
||||
"route_pass_rate": 1.0,
|
||||
"execution_error_count": 0,
|
||||
"false_factual_rate": 0.0,
|
||||
"notes": [
|
||||
"execution_error_count derived as http_error_count + limited_reason_category.execution_error",
|
||||
"false_factual_rate derived from false_factual_count/questions_total"
|
||||
]
|
||||
},
|
||||
"candidate": {
|
||||
"summary_path": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\runs\\2026-04-03_Address_Nightly_Regression_Post_AnchorHardening_R5\\2026-04-03_Address_Nightly_followup_25_10-55-41\\run_summary.json",
|
||||
"run_id": "2026-04-03_Address_Nightly_followup_25_10-55-41",
|
||||
"questions_total": 25,
|
||||
"strict_pass_rate": 1.0,
|
||||
"route_pass_rate": 1.0,
|
||||
"execution_error_count": 0,
|
||||
"false_factual_rate": 0.0,
|
||||
"notes": [
|
||||
"execution_error_count derived as http_error_count + limited_reason_category.execution_error",
|
||||
"false_factual_rate derived from false_factual_count/questions_total"
|
||||
]
|
||||
},
|
||||
"checks": [
|
||||
{
|
||||
"metric": "strict_pass_rate",
|
||||
"passed": true,
|
||||
"baseline": 1.0,
|
||||
"candidate": 1.0,
|
||||
"rule": "candidate >= baseline"
|
||||
},
|
||||
{
|
||||
"metric": "route_pass_rate",
|
||||
"passed": true,
|
||||
"baseline": 1.0,
|
||||
"candidate": 1.0,
|
||||
"rule": "candidate >= baseline"
|
||||
},
|
||||
{
|
||||
"metric": "execution_error_count",
|
||||
"passed": true,
|
||||
"baseline": 0,
|
||||
"candidate": 0,
|
||||
"rule": "candidate <= baseline"
|
||||
},
|
||||
{
|
||||
"metric": "false_factual_rate",
|
||||
"passed": true,
|
||||
"baseline": 0.0,
|
||||
"candidate": 0.0,
|
||||
"rule": "candidate <= baseline"
|
||||
},
|
||||
{
|
||||
"metric": "questions_total_match",
|
||||
"passed": true,
|
||||
"baseline": 25,
|
||||
"candidate": 25,
|
||||
"rule": "candidate == baseline"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -0,0 +1,23 @@
|
|||
{
|
||||
"generated_at": "2026-04-03T10:58:22",
|
||||
"total": 1,
|
||||
"passed": 1,
|
||||
"failed": 0,
|
||||
"results": [
|
||||
{
|
||||
"run_dir": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\runs\\2026-04-03_Address_Nightly_Regression_Post_AnchorHardening_R5\\2026-04-03_Address_Nightly_followup_25_10-55-41",
|
||||
"valid": true,
|
||||
"errors": [],
|
||||
"warnings": [],
|
||||
"metrics": {
|
||||
"run_id": "2026-04-03_Address_Nightly_followup_25_10-55-41",
|
||||
"questions_total": 25,
|
||||
"route_pass_rate": 1.0,
|
||||
"strict_pass_rate": 1.0,
|
||||
"http_error_count": 0,
|
||||
"rows_count": 25,
|
||||
"failures_count": 0
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -0,0 +1,33 @@
|
|||
# 2026-04-03_Address_Nightly_stress_102_10-44-57
|
||||
|
||||
Generated at: 2026-04-03T10:55:41
|
||||
Questions file: X:\1C\NDC_1C\docs\ADDRESS\question_sets\address_slang_stress_full_2026-04-02.json
|
||||
Backend URL: http://127.0.0.1:8787/api/assistant/message
|
||||
LLM: local / qwen2.5-14b-instruct-1m @ http://127.0.0.1:1234
|
||||
Strict policy: route
|
||||
|
||||
## Totals
|
||||
- questions_total: 102
|
||||
- ok_200_count: 102
|
||||
- semantic_pass_count: 102
|
||||
- semantic_pass_rate: 1.0
|
||||
- route_pass_count: 101
|
||||
- route_pass_rate: 0.9902
|
||||
- strict_pass_count: 101
|
||||
- strict_pass_rate: 0.9902
|
||||
- factual_count: 96
|
||||
- partial_coverage_count: 6
|
||||
- clarification_required_count: 0
|
||||
- http_error_count: 0
|
||||
- llm_decomposition_attempted_count: 102
|
||||
- llm_decomposition_applied_count: 76
|
||||
- llm_fallback_count: 0
|
||||
- llm_fallback_rate: 0.0
|
||||
- tool_gate_blocked_count: 0
|
||||
- tool_gate_blocked_rate: 0.0
|
||||
- avg_elapsed_ms: 6312.9
|
||||
|
||||
## Files
|
||||
- run_summary.json
|
||||
- full_live_results.json
|
||||
- failures_only.json
|
||||
|
|
@ -0,0 +1,188 @@
|
|||
[
|
||||
{
|
||||
"index": 64,
|
||||
"id": "X004",
|
||||
"question": "svk poka docy za 2020",
|
||||
"session": null,
|
||||
"session_id": "asst-2026-04-03_Address_Nightly_stress_102_10-44-57-x004",
|
||||
"status_code": 200,
|
||||
"ok": true,
|
||||
"elapsed_ms": 8433,
|
||||
"reply_type": "partial_coverage",
|
||||
"trace_id": "address-Pf1_TmW0wX",
|
||||
"assistant_reply": "В live-данных по текущему фильтру записи не найдены.\nПричина: по указанному якорю и фильтрам в live-выборке нет строк.\nЧто нужно уточнить: уточните период или снимите часть фильтров.",
|
||||
"assistant_reply_first_line": "В live-данных по текущему фильтру записи не найдены.",
|
||||
"expected_intent": "list_documents_by_counterparty",
|
||||
"actual_intent": "list_documents_by_counterparty",
|
||||
"intent_match": true,
|
||||
"expected_mode": "address_query",
|
||||
"actual_mode": "address_query",
|
||||
"mode_match": true,
|
||||
"expected_reply_type": null,
|
||||
"reply_match": true,
|
||||
"semantic_pass": true,
|
||||
"route_pass": false,
|
||||
"route_health": "likely_blocked_route",
|
||||
"strict_policy": "route",
|
||||
"strict_pass": false,
|
||||
"selected_recipe": "address_documents_by_counterparty_v1",
|
||||
"missing_required_filters": [],
|
||||
"match_failure_stage": "materialized_but_not_anchor_matched",
|
||||
"match_failure_reason": "counterparty_anchor_not_matched_in_materialized_rows",
|
||||
"rows_fetched": 20,
|
||||
"rows_matched": 0,
|
||||
"mcp_call_status": "materialized_but_not_anchor_matched",
|
||||
"limited_reason_category": "empty_match",
|
||||
"llm_decomposition_attempted": true,
|
||||
"llm_decomposition_applied": false,
|
||||
"llm_decomposition_reason": "normalized_fragment_rejected_anchor_degradation",
|
||||
"llm_canonical_candidate_detected": true,
|
||||
"fallback_rule_hit": null,
|
||||
"tool_gate_decision": "run_address_lane",
|
||||
"tool_gate_reason": "address_mode_classifier_detected",
|
||||
"predecompose_contract_intent": "list_documents_by_counterparty",
|
||||
"predecompose_contract_aggregation_profile": "list_lookup",
|
||||
"predecompose_contract_period_scope": "year",
|
||||
"debug_payload": {
|
||||
"trace_id": "address-Pf1_TmW0wX",
|
||||
"prompt_version": "address_query_runtime_v1",
|
||||
"schema_version": "address_query_runtime_v1",
|
||||
"fallback_type": "partial",
|
||||
"route_summary": null,
|
||||
"fragments": [],
|
||||
"requirements_extracted": [],
|
||||
"coverage_report": {
|
||||
"requirements_total": 0,
|
||||
"requirements_covered": 0,
|
||||
"requirements_uncovered": [],
|
||||
"requirements_partially_covered": [],
|
||||
"clarification_needed_for": [],
|
||||
"out_of_scope_requirements": []
|
||||
},
|
||||
"routes": [],
|
||||
"retrieval_status": [],
|
||||
"retrieval_results": [],
|
||||
"answer_grounding_check": {
|
||||
"status": "partial",
|
||||
"route_subject_match": true,
|
||||
"missing_requirements": [],
|
||||
"reasons": [
|
||||
"docs_or_bank_signal_detected",
|
||||
"anchor_like_token_detected",
|
||||
"shape_not_detected",
|
||||
"documents_by_counterparty_signal_detected"
|
||||
],
|
||||
"why_included_summary": [],
|
||||
"selection_reason_summary": []
|
||||
},
|
||||
"dropped_intent_segments": [],
|
||||
"detected_mode": "address_query",
|
||||
"detected_mode_confidence": "medium",
|
||||
"query_shape": "UNKNOWN",
|
||||
"query_shape_confidence": "low",
|
||||
"detected_intent": "list_documents_by_counterparty",
|
||||
"detected_intent_confidence": "medium",
|
||||
"extracted_filters": {
|
||||
"sort": "period_desc",
|
||||
"limit": 20,
|
||||
"counterparty": "poka",
|
||||
"period_from": "2020-01-01",
|
||||
"period_to": "2020-12-31"
|
||||
},
|
||||
"missing_required_filters": [],
|
||||
"selected_recipe": "address_documents_by_counterparty_v1",
|
||||
"mcp_call_status_legacy": "materialized_but_not_matched",
|
||||
"account_scope_mode": "preferred",
|
||||
"account_scope_fallback_applied": false,
|
||||
"anchor_type": "counterparty",
|
||||
"anchor_value_raw": "poka",
|
||||
"anchor_value_resolved": "poka",
|
||||
"resolver_confidence": "medium",
|
||||
"ambiguity_count": 0,
|
||||
"match_failure_stage": "materialized_but_not_anchor_matched",
|
||||
"match_failure_reason": "counterparty_anchor_not_matched_in_materialized_rows",
|
||||
"mcp_call_status": "materialized_but_not_anchor_matched",
|
||||
"rows_fetched": 20,
|
||||
"raw_rows_received": 20,
|
||||
"rows_after_account_scope": 5,
|
||||
"rows_after_recipe_filter": 0,
|
||||
"rows_materialized": 5,
|
||||
"rows_matched": 0,
|
||||
"raw_row_keys_sample": [
|
||||
"Период",
|
||||
"Регистратор",
|
||||
"СчетДт",
|
||||
"СчетКт",
|
||||
"Сумма",
|
||||
"Контрагент",
|
||||
"Period",
|
||||
"Registrator",
|
||||
"Amount"
|
||||
],
|
||||
"materialization_drop_reason": "none",
|
||||
"account_token_raw": null,
|
||||
"account_token_normalized": null,
|
||||
"account_scope_fields_checked": [
|
||||
"account_dt",
|
||||
"account_kt",
|
||||
"registrator",
|
||||
"analytics"
|
||||
],
|
||||
"account_scope_match_strategy": "account_code_regex_plus_alias_map_v1",
|
||||
"account_scope_drop_reason": "not_applicable",
|
||||
"runtime_readiness": "LIVE_QUERYABLE_WITH_LIMITS",
|
||||
"limited_reason_category": "empty_match",
|
||||
"response_type": "LIMITED_WITH_REASON",
|
||||
"execution_lane": "address_query",
|
||||
"llm_decomposition_applied": false,
|
||||
"llm_decomposition_attempted": true,
|
||||
"llm_provider_used": "local",
|
||||
"llm_decomposition_trace_id": "jf_iOhLTLgxz5Q",
|
||||
"llm_decomposition_effective_message": "svk poka docy za 2020",
|
||||
"llm_decomposition_reason": "normalized_fragment_rejected_anchor_degradation",
|
||||
"llm_canonical_candidate_detected": true,
|
||||
"llm_predecompose_contract": {
|
||||
"schema_version": "address_llm_predecompose_contract_v1",
|
||||
"source_message": "svk poka docy za 2020",
|
||||
"canonical_message": "svk poka docy za 2020",
|
||||
"mode": "address_query",
|
||||
"mode_confidence": "medium",
|
||||
"query_shape": "UNKNOWN",
|
||||
"query_shape_confidence": "low",
|
||||
"intent": "list_documents_by_counterparty",
|
||||
"intent_confidence": "medium",
|
||||
"entities": {
|
||||
"account": null,
|
||||
"counterparty": "poka",
|
||||
"contract": null,
|
||||
"document_type": null,
|
||||
"document_ref": null,
|
||||
"organization": null
|
||||
},
|
||||
"period": {
|
||||
"scope": "year",
|
||||
"period_from": "2020-01-01",
|
||||
"period_to": "2020-12-31",
|
||||
"as_of_date": null,
|
||||
"has_explicit_period": true
|
||||
},
|
||||
"aggregation_profile": "list_lookup"
|
||||
},
|
||||
"fallback_rule_hit": null,
|
||||
"sanitized_user_message": "svk poka доки za 2020",
|
||||
"tool_gate_decision": "run_address_lane",
|
||||
"tool_gate_reason": "address_mode_classifier_detected",
|
||||
"answer_structure_v11": null,
|
||||
"investigation_state_snapshot": null,
|
||||
"normalized": null,
|
||||
"normalizer_output": {
|
||||
"trace_id": "jf_iOhLTLgxz5Q",
|
||||
"prompt_version": "normalizer_v2_0_2",
|
||||
"applied": false,
|
||||
"effective_message": "svk poka docy za 2020"
|
||||
}
|
||||
},
|
||||
"error_code": null,
|
||||
"error_message": null
|
||||
}
|
||||
]
|
||||
File diff suppressed because it is too large
Load Diff
|
|
@ -0,0 +1,106 @@
|
|||
# Response Audit: 2026-04-03_Address_Nightly_stress_102_10-44-57
|
||||
|
||||
| id | strict | route_health | reply_type | intent | limited_reason | question | assistant_first_line |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| S001 | True | ok_or_factual | factual | list_documents_by_counterparty | None | свк доки за 2020 покаж пж | Собран список документов по контрагенту (live address lane). |
|
||||
| S002 | True | ok_or_factual | factual | list_documents_by_counterparty | None | покеж все доки по свк за 20 год | Собран список документов по контрагенту (live address lane). |
|
||||
| S003 | True | ok_or_factual | factual | list_documents_by_counterparty | None | дай список документов по свк за 2020 | Собран список документов по контрагенту (live address lane). |
|
||||
| S004 | True | ok_or_factual | factual | list_documents_by_counterparty | None | какие у свк есть доки за 2020? | Собран список документов по контрагенту (live address lane). |
|
||||
| S005 | True | ok_or_factual | factual | list_documents_by_counterparty | None | че по свк по докам за 2020 год | Собран список документов по контрагенту (live address lane). |
|
||||
| S006 | True | ok_or_factual | factual | list_documents_by_counterparty | None | ёпт покажи доки по свк за 2020 | Собран список документов по контрагенту (live address lane). |
|
||||
| S007 | True | ok_or_factual | factual | list_documents_by_counterparty | None | бля, доки свк 2020 вывали | Собран список документов по контрагенту (live address lane). |
|
||||
| S008 | True | ok_or_factual | factual | list_documents_by_counterparty | None | svk docs for 2020 please | Собран список документов по контрагенту (live address lane). |
|
||||
| S009 | True | ok_or_factual | factual | list_documents_by_counterparty | None | docs by counterparty svk year 2020 | Собран список документов по контрагенту (live address lane). |
|
||||
| S010 | True | ok_or_factual | factual | list_documents_by_counterparty | None | по контрагенту свк покаж документы за 2020 | Собран список документов по контрагенту (live address lane). |
|
||||
| S011 | True | ok_or_factual | factual | list_documents_by_counterparty | None | плс доки по свк 2020 | Собран список документов по контрагенту (live address lane). |
|
||||
| S012 | True | ok_or_factual | factual | list_documents_by_counterparty | None | свк док за 2020 гони | Собран список документов по контрагенту (live address lane). |
|
||||
| S013 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | по свк банковские операции за 2020 покаж | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| S014 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | свк выписка банка 2020 | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| S015 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | платежи по свк за 2020 выведи | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| S016 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | оплаты по свк за 2020 | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| S017 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | свк банк опер за 2020 | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| S018 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | че у свк по банку за 2020 | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| S019 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | свк списания/поступления за 2020 | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| S020 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | bank ops by svk for 2020 | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| S021 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | transactions by counterparty svk 2020 | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| S022 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | бля, по свк банковские платежки 2020 | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| S023 | True | ok_or_factual | factual | account_balance_snapshot | None | сальдо по счету 60.01 на дату 2020-07-31 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| S024 | True | ok_or_factual | factual | account_balance_snapshot | None | остаток по счету 60.01 на 2020.05 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| S025 | True | ok_or_factual | factual | account_balance_snapshot | None | покажи остаток по 60.01 за май 2020 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| S026 | True | ok_or_factual | factual | account_balance_snapshot | None | счет 60 остаток на май 2020 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| S027 | True | ok_or_factual | factual | account_balance_snapshot | None | баланс account 60.01 as of 2020-07-31 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| S028 | True | ok_or_factual | factual | account_balance_snapshot | None | 60.01 saldo на 31.07.2020 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| S029 | True | ok_or_factual | factual | account_balance_snapshot | None | сколько на счете 60 на конец 2020-05 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| S030 | True | ok_or_factual | factual | account_balance_snapshot | None | дай остаток 60.02 на 2020-12-31 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| S031 | True | partial_non_blocking | partial_coverage | account_balance_snapshot | empty_match | епт, сальдо по счету 62.01 на 2020-12-31 | В live-данных по текущему фильтру записи не найдены. |
|
||||
| S032 | True | ok_or_factual | factual | account_balance_snapshot | None | плс остаток по счету 51 на 2020-06-30 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| S033 | True | ok_or_factual | factual | account_balance_snapshot | None | как там по счету 60.01 на 2020.07? | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| S034 | True | ok_or_factual | factual | account_balance_snapshot | None | сальдо 60 на 2020 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| S035 | True | ok_or_factual | factual | documents_forming_balance | None | какие документы формируют остаток по счету 60.01 на 2020-07-31 | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| S036 | True | ok_or_factual | factual | documents_forming_balance | None | из чего состоит остаток по счету 60 на 2020.05 | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| S037 | True | ok_or_factual | factual | documents_forming_balance | None | раскрой остаток 60.01 по документам на конец июля 2020 | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| S038 | True | ok_or_factual | factual | documents_forming_balance | None | доки под остатком 60.01 на 2020-07-31 | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| S039 | True | ok_or_factual | factual | documents_forming_balance | None | documents forming balance for account 60.01 as of 2020-07-31 | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| S040 | True | ok_or_factual | factual | documents_forming_balance | None | бля, какие доки формируют остаток 60.01 на 2020-05-31 | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| S041 | True | partial_non_blocking | partial_coverage | documents_forming_balance | empty_match | по счету 62.01 раскрой остаток документами на 2020-12-31 | В live-данных по текущему фильтру записи не найдены. |
|
||||
| S042 | True | ok_or_factual | factual | documents_forming_balance | None | из чего остаток по счету 51 на 2020-06-30 | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| S043 | True | ok_or_factual | factual | list_documents_by_contract | None | покажи документы по договору 19/15 за 2020 | Период сохранен. Глубина live-выборки автоматически расширена до 1000 строк. |
|
||||
| S044 | True | ok_or_factual | factual | list_documents_by_contract | None | доки по дог. 19/15 за 2020 | Период сохранен. Глубина live-выборки автоматически расширена до 1000 строк. |
|
||||
| S045 | True | ok_or_factual | factual | list_documents_by_contract | None | какие документы по контракту 19/15 | Собран список документов по договору (live address lane). |
|
||||
| S046 | True | ok_or_factual | factual | list_documents_by_contract | None | docs by contract 19/15 year 2020 | Период сохранен. Глубина live-выборки автоматически расширена до 1000 строк. |
|
||||
| S047 | True | ok_or_factual | factual | list_documents_by_contract | None | бля, покажи по договору 19/15 доки | Собран список документов по договору (live address lane). |
|
||||
| S048 | True | ok_or_factual | factual | list_documents_by_contract | None | договор 19/15 документы | Собран список документов по договору (live address lane). |
|
||||
| S049 | True | ok_or_factual | factual | list_documents_by_contract | None | плс документы договор №19/15 | Собран список документов по договору (live address lane). |
|
||||
| S050 | True | ok_or_factual | factual | list_documents_by_contract | None | покеж документы по 19/15 | Собран список документов по договору (live address lane). |
|
||||
| S051 | True | ok_or_factual | factual | bank_operations_by_contract | None | банковские операции по договору 19/15 | Собран список банковских операций по договору (live address lane). |
|
||||
| S052 | True | ok_or_factual | factual | bank_operations_by_contract | None | платежи по договору 19/15 | Собран список банковских операций по договору (live address lane). |
|
||||
| S053 | True | ok_or_factual | factual | bank_operations_by_contract | None | выписка по договору 19/15 | Собран список банковских операций по договору (live address lane). |
|
||||
| S054 | True | ok_or_factual | factual | bank_operations_by_contract | None | bank ops by contract 19/15 | Собран список банковских операций по договору (live address lane). |
|
||||
| S055 | True | ok_or_factual | factual | bank_operations_by_contract | None | ёпт, покажи банк операции по дог 19/15 | Собран список банковских операций по договору (live address lane). |
|
||||
| S056 | True | ok_or_factual | factual | bank_operations_by_contract | None | по контракту 19/15 платежки | Собран список банковских операций по договору (live address lane). |
|
||||
| S057 | True | ok_or_factual | factual | account_balance_snapshot | None | покажи сальдо по счету 60 за 2020 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| S058 | True | ok_or_factual | factual | account_balance_snapshot | None | а теперь только май 2020 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| S059 | True | ok_or_factual | factual | account_balance_snapshot | None | ок, теперь 2020.05 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| S060 | True | ok_or_factual | factual | account_balance_snapshot | None | теперь на 2020-07-31 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| X001 | True | ok_or_factual | factual | list_documents_by_counterparty | None | свк доки кинь за 2020 | Собран список документов по контрагенту (live address lane). |
|
||||
| X002 | True | ok_or_factual | factual | list_documents_by_counterparty | None | по свк все документы за 20й год | Собран список документов по контрагенту (live address lane). |
|
||||
| X003 | True | ok_or_factual | factual | list_documents_by_counterparty | None | пж вывали доки контрагент свк 2020 | Собран список документов по контрагенту (live address lane). |
|
||||
| X004 | False | likely_blocked_route | partial_coverage | list_documents_by_counterparty | empty_match | svk poka docy za 2020 | В live-данных по текущему фильтру записи не найдены. |
|
||||
| X005 | True | ok_or_factual | factual | list_documents_by_counterparty | None | какие доки по компании свк за 2020 | Собран список документов по контрагенту (live address lane). |
|
||||
| X006 | True | ok_or_factual | factual | list_documents_by_counterparty | None | бля покаж свк документы 2020 | Собран список документов по контрагенту (live address lane). |
|
||||
| X007 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | свк банк операции за 2020 | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| X008 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | по свк выписку банка за 2020 выведи | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| X009 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | свк платежки за 2020 | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| X010 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | bank ops svk 2020 | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| X011 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | транзакции по свк за 2020 | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| X012 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | ёпт по свк че по банку 2020 | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| X013 | True | ok_or_factual | factual | account_balance_snapshot | None | остаток по 60.01 на 2020-07-31 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| X014 | True | ok_or_factual | factual | account_balance_snapshot | None | сальдо 60.01 на 31.07.2020 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| X015 | True | ok_or_factual | factual | account_balance_snapshot | None | что на счете 60 на 2020.05 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| X016 | True | ok_or_factual | factual | account_balance_snapshot | None | дай остаток 62.01 на 2020 май | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| X017 | True | partial_non_blocking | partial_coverage | account_balance_snapshot | empty_match | saldo account 60 as of 2020-12-31 | В live-данных по текущему фильтру записи не найдены. |
|
||||
| X018 | True | ok_or_factual | factual | account_balance_snapshot | None | плиз остаток по счету 51 на 2020-06-30 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| X019 | True | partial_non_blocking | partial_coverage | account_balance_snapshot | empty_match | скока по 60.02 на конец 2020-12 | В live-данных по текущему фильтру записи не найдены. |
|
||||
| X020 | True | ok_or_factual | factual | account_balance_snapshot | None | бля сальдо 60.01 на 2020-05-31 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| X021 | True | ok_or_factual | factual | documents_forming_balance | None | раскрой остаток 60.01 по документам на конец июля 2020 | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| X022 | True | ok_or_factual | factual | documents_forming_balance | None | доки под остатком 60.01 на 2020-07-31 | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| X023 | True | ok_or_factual | factual | documents_forming_balance | None | из чего остаток 60.01 на 2020-07-31 | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| X024 | True | ok_or_factual | factual | documents_forming_balance | None | какие документы формируют остаток по 60.01 на 2020-07-31 | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| X025 | True | ok_or_factual | factual | documents_forming_balance | None | docs forming balance 60.01 as of 2020-07-31 | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| X026 | True | partial_non_blocking | partial_coverage | documents_forming_balance | empty_match | раскидай остаток 62.01 по докам на 2020-12-31 | В live-данных по текущему фильтру записи не найдены. |
|
||||
| X027 | True | ok_or_factual | factual | documents_forming_balance | None | по счету 51 раскрой остаток документами на 2020-06-30 | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| X028 | True | ok_or_factual | factual | documents_forming_balance | None | бля из чего состоит остаток по 60.01 на 2020-05-31 | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| X029 | True | ok_or_factual | factual | list_documents_by_contract | None | доки по договору 19/15 за 2020 | Период сохранен. Глубина live-выборки автоматически расширена до 1000 строк. |
|
||||
| X030 | True | ok_or_factual | factual | list_documents_by_contract | None | покажи документы контракт 19/15 | Собран список документов по договору (live address lane). |
|
||||
| X031 | True | ok_or_factual | factual | list_documents_by_contract | None | docs by contract 19/15 2020 | Период сохранен. Глубина live-выборки автоматически расширена до 1000 строк. |
|
||||
| X032 | True | ok_or_factual | factual | list_documents_by_contract | None | по дог 19/15 какие документы | Собран список документов по договору (live address lane). |
|
||||
| X033 | True | ok_or_factual | factual | list_documents_by_contract | None | договор 19/15 выведи доки | Собран список документов по договору (live address lane). |
|
||||
| X034 | True | ok_or_factual | factual | list_documents_by_contract | None | нах покажи документы по 19/15 | Собран список документов по договору (live address lane). |
|
||||
| X035 | True | ok_or_factual | factual | bank_operations_by_contract | None | банк операции по договору 19/15 | Собран список банковских операций по договору (live address lane). |
|
||||
| X036 | True | ok_or_factual | factual | bank_operations_by_contract | None | платежки по дог 19/15 | Собран список банковских операций по договору (live address lane). |
|
||||
| X037 | True | ok_or_factual | factual | bank_operations_by_contract | None | выписка по контракту 19/15 | Собран список банковских операций по договору (live address lane). |
|
||||
| X038 | True | ok_or_factual | factual | bank_operations_by_contract | None | bank ops by contract 19/15 | Собран список банковских операций по договору (live address lane). |
|
||||
| X039 | True | ok_or_factual | factual | bank_operations_by_contract | None | покажи банк опер по дог 19/15 пж | Собран список банковских операций по договору (live address lane). |
|
||||
| X040 | True | ok_or_factual | factual | bank_operations_by_contract | None | епт транзакции договор 19/15 | Собран список банковских операций по договору (live address lane). |
|
||||
| X041 | True | ok_or_factual | factual | account_balance_snapshot | None | покажи сальдо по счету 60 за 2020 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| X042 | True | ok_or_factual | factual | account_balance_snapshot | None | а теперь только май 2020 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
|
|
@ -0,0 +1,134 @@
|
|||
{
|
||||
"run_id": "2026-04-03_Address_Nightly_stress_102_10-44-57",
|
||||
"generated_at": "2026-04-03T10:55:41",
|
||||
"source_questions_file": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\question_sets\\address_slang_stress_full_2026-04-02.json",
|
||||
"backend_url": "http://127.0.0.1:8787/api/assistant/message",
|
||||
"llm_provider": "local",
|
||||
"llm_model": "qwen2.5-14b-instruct-1m",
|
||||
"llm_base_url": "http://127.0.0.1:1234",
|
||||
"strict_policy": "route",
|
||||
"totals": {
|
||||
"questions_total": 102,
|
||||
"ok_200_count": 102,
|
||||
"semantic_pass_count": 102,
|
||||
"semantic_pass_rate": 1.0,
|
||||
"route_pass_count": 101,
|
||||
"route_pass_rate": 0.9902,
|
||||
"strict_pass_count": 101,
|
||||
"strict_pass_rate": 0.9902,
|
||||
"factual_count": 96,
|
||||
"partial_coverage_count": 6,
|
||||
"clarification_required_count": 0,
|
||||
"http_error_count": 0,
|
||||
"llm_decomposition_attempted_count": 102,
|
||||
"llm_decomposition_applied_count": 76,
|
||||
"llm_fallback_count": 0,
|
||||
"llm_fallback_rate": 0.0,
|
||||
"tool_gate_blocked_count": 0,
|
||||
"tool_gate_blocked_rate": 0.0,
|
||||
"avg_elapsed_ms": 6312.9
|
||||
},
|
||||
"distributions": {
|
||||
"reply_type": {
|
||||
"factual": 96,
|
||||
"partial_coverage": 6
|
||||
},
|
||||
"actual_intent": {
|
||||
"list_documents_by_counterparty": 18,
|
||||
"bank_operations_by_counterparty": 16,
|
||||
"account_balance_snapshot": 26,
|
||||
"documents_forming_balance": 16,
|
||||
"list_documents_by_contract": 14,
|
||||
"bank_operations_by_contract": 12
|
||||
},
|
||||
"actual_mode": {
|
||||
"address_query": 102
|
||||
},
|
||||
"mcp_call_status": {
|
||||
"matched_non_empty": 96,
|
||||
"no_raw_rows": 5,
|
||||
"materialized_but_not_anchor_matched": 1
|
||||
},
|
||||
"limited_reason_category": {
|
||||
"empty_match": 6
|
||||
},
|
||||
"route_health": {
|
||||
"ok_or_factual": 96,
|
||||
"partial_non_blocking": 5,
|
||||
"likely_blocked_route": 1
|
||||
},
|
||||
"tool_gate_decision": {
|
||||
"run_address_lane": 102
|
||||
},
|
||||
"tool_gate_reason": {
|
||||
"address_mode_classifier_detected": 98,
|
||||
"llm_canonical_candidate_detected": 4
|
||||
}
|
||||
},
|
||||
"address_llm_predecompose_metrics": {
|
||||
"overall": {
|
||||
"llm_attempted": 102,
|
||||
"llm_applied": 76,
|
||||
"fallback_used": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"tool_gate_blocked": 0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"by_intent": {
|
||||
"list_documents_by_counterparty": {
|
||||
"total": 18,
|
||||
"llm_attempted": 18,
|
||||
"llm_applied": 11,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"bank_operations_by_counterparty": {
|
||||
"total": 16,
|
||||
"llm_attempted": 16,
|
||||
"llm_applied": 11,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"account_balance_snapshot": {
|
||||
"total": 26,
|
||||
"llm_attempted": 26,
|
||||
"llm_applied": 25,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"documents_forming_balance": {
|
||||
"total": 16,
|
||||
"llm_attempted": 16,
|
||||
"llm_applied": 11,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"list_documents_by_contract": {
|
||||
"total": 14,
|
||||
"llm_attempted": 14,
|
||||
"llm_applied": 10,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"bank_operations_by_contract": {
|
||||
"total": 12,
|
||||
"llm_attempted": 12,
|
||||
"llm_applied": 8,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,67 @@
|
|||
{
|
||||
"generated_at": "2026-04-03T10:55:41",
|
||||
"overall_pass": false,
|
||||
"baseline": {
|
||||
"summary_path": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\runs\\2026-04-02_Address_Slang_Live_Stress_2026-04-02_12-57-27\\run_summary.json",
|
||||
"run_id": "2026-04-02_Address_Slang_Live_Stress_2026-04-02_12-57-27",
|
||||
"questions_total": 102,
|
||||
"strict_pass_rate": 1.0,
|
||||
"route_pass_rate": 1.0,
|
||||
"execution_error_count": 0,
|
||||
"false_factual_rate": 0.0,
|
||||
"notes": [
|
||||
"execution_error_count derived as http_error_count + limited_reason_category.execution_error",
|
||||
"false_factual_rate derived from false_factual_count/questions_total"
|
||||
]
|
||||
},
|
||||
"candidate": {
|
||||
"summary_path": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\runs\\2026-04-03_Address_Nightly_Regression_Post_AnchorHardening_R5\\2026-04-03_Address_Nightly_stress_102_10-44-57\\run_summary.json",
|
||||
"run_id": "2026-04-03_Address_Nightly_stress_102_10-44-57",
|
||||
"questions_total": 102,
|
||||
"strict_pass_rate": 0.9902,
|
||||
"route_pass_rate": 0.9902,
|
||||
"execution_error_count": 0,
|
||||
"false_factual_rate": 0.0,
|
||||
"notes": [
|
||||
"execution_error_count derived as http_error_count + limited_reason_category.execution_error",
|
||||
"false_factual_rate derived from false_factual_count/questions_total"
|
||||
]
|
||||
},
|
||||
"checks": [
|
||||
{
|
||||
"metric": "strict_pass_rate",
|
||||
"passed": false,
|
||||
"baseline": 1.0,
|
||||
"candidate": 0.9902,
|
||||
"rule": "candidate >= baseline"
|
||||
},
|
||||
{
|
||||
"metric": "route_pass_rate",
|
||||
"passed": false,
|
||||
"baseline": 1.0,
|
||||
"candidate": 0.9902,
|
||||
"rule": "candidate >= baseline"
|
||||
},
|
||||
{
|
||||
"metric": "execution_error_count",
|
||||
"passed": true,
|
||||
"baseline": 0,
|
||||
"candidate": 0,
|
||||
"rule": "candidate <= baseline"
|
||||
},
|
||||
{
|
||||
"metric": "false_factual_rate",
|
||||
"passed": true,
|
||||
"baseline": 0.0,
|
||||
"candidate": 0.0,
|
||||
"rule": "candidate <= baseline"
|
||||
},
|
||||
{
|
||||
"metric": "questions_total_match",
|
||||
"passed": true,
|
||||
"baseline": 102,
|
||||
"candidate": 102,
|
||||
"rule": "candidate == baseline"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -0,0 +1,23 @@
|
|||
{
|
||||
"generated_at": "2026-04-03T10:55:41",
|
||||
"total": 1,
|
||||
"passed": 1,
|
||||
"failed": 0,
|
||||
"results": [
|
||||
{
|
||||
"run_dir": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\runs\\2026-04-03_Address_Nightly_Regression_Post_AnchorHardening_R5\\2026-04-03_Address_Nightly_stress_102_10-44-57",
|
||||
"valid": true,
|
||||
"errors": [],
|
||||
"warnings": [],
|
||||
"metrics": {
|
||||
"run_id": "2026-04-03_Address_Nightly_stress_102_10-44-57",
|
||||
"questions_total": 102,
|
||||
"route_pass_rate": 0.9902,
|
||||
"strict_pass_rate": 0.9902,
|
||||
"http_error_count": 0,
|
||||
"rows_count": 102,
|
||||
"failures_count": 1
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -0,0 +1,24 @@
|
|||
# 2026-04-03_Address_Nightly_Regression_Post_AnchorHardening_R5
|
||||
|
||||
Generated at: 2026-04-03T10:58:22
|
||||
Dry run: False
|
||||
Strict policy: route
|
||||
Overall: FAIL
|
||||
|
||||
## Packs
|
||||
### stress_102
|
||||
- run_id: 2026-04-03_Address_Nightly_stress_102_10-44-57
|
||||
- runner_ok: True
|
||||
- validator_ok: True
|
||||
- comparator_ok: False
|
||||
- run_dir: X:\1C\NDC_1C\docs\ADDRESS\runs\2026-04-03_Address_Nightly_Regression_Post_AnchorHardening_R5\2026-04-03_Address_Nightly_stress_102_10-44-57
|
||||
- errors:
|
||||
- comparator failed with exit code 1
|
||||
|
||||
### followup_25
|
||||
- run_id: 2026-04-03_Address_Nightly_followup_25_10-55-41
|
||||
- runner_ok: True
|
||||
- validator_ok: True
|
||||
- comparator_ok: True
|
||||
- run_dir: X:\1C\NDC_1C\docs\ADDRESS\runs\2026-04-03_Address_Nightly_Regression_Post_AnchorHardening_R5\2026-04-03_Address_Nightly_followup_25_10-55-41
|
||||
|
||||
|
|
@ -0,0 +1,36 @@
|
|||
{
|
||||
"nightly_run_id": "2026-04-03_Address_Nightly_Regression_Post_AnchorHardening_R5",
|
||||
"generated_at": "2026-04-03T10:58:22",
|
||||
"dry_run": false,
|
||||
"overall_ok": false,
|
||||
"strict_policy": "route",
|
||||
"packs": [
|
||||
{
|
||||
"pack": "stress_102",
|
||||
"run_id": "2026-04-03_Address_Nightly_stress_102_10-44-57",
|
||||
"questions_file": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\question_sets\\address_slang_stress_full_2026-04-02.json",
|
||||
"baseline_summary": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\runs\\2026-04-02_Address_Slang_Live_Stress_2026-04-02_12-57-27\\run_summary.json",
|
||||
"run_dir": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\runs\\2026-04-03_Address_Nightly_Regression_Post_AnchorHardening_R5\\2026-04-03_Address_Nightly_stress_102_10-44-57",
|
||||
"runner_ok": true,
|
||||
"validator_ok": true,
|
||||
"comparator_ok": false,
|
||||
"errors": [
|
||||
"comparator failed with exit code 1"
|
||||
],
|
||||
"validator_report": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\runs\\2026-04-03_Address_Nightly_Regression_Post_AnchorHardening_R5\\2026-04-03_Address_Nightly_stress_102_10-44-57_validator_report.json"
|
||||
},
|
||||
{
|
||||
"pack": "followup_25",
|
||||
"run_id": "2026-04-03_Address_Nightly_followup_25_10-55-41",
|
||||
"questions_file": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\question_sets\\address_followup_context_chains_2026-04-02.json",
|
||||
"baseline_summary": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\runs\\2026-04-02_Address_Followup_Context_Chains_2026-04-02_19-15-Run5\\run_summary.json",
|
||||
"run_dir": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\runs\\2026-04-03_Address_Nightly_Regression_Post_AnchorHardening_R5\\2026-04-03_Address_Nightly_followup_25_10-55-41",
|
||||
"runner_ok": true,
|
||||
"validator_ok": true,
|
||||
"comparator_ok": true,
|
||||
"errors": [],
|
||||
"validator_report": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\runs\\2026-04-03_Address_Nightly_Regression_Post_AnchorHardening_R5\\2026-04-03_Address_Nightly_followup_25_10-55-41_validator_report.json",
|
||||
"comparator_report": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\runs\\2026-04-03_Address_Nightly_Regression_Post_AnchorHardening_R5\\2026-04-03_Address_Nightly_followup_25_10-55-41_comparator_report.json"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -0,0 +1,33 @@
|
|||
# 2026-04-03_Address_Nightly_followup_25_11-12-58
|
||||
|
||||
Generated at: 2026-04-03T11:15:38
|
||||
Questions file: X:\1C\NDC_1C\docs\ADDRESS\question_sets\address_followup_context_chains_2026-04-02.json
|
||||
Backend URL: http://127.0.0.1:8787/api/assistant/message
|
||||
LLM: local / qwen2.5-14b-instruct-1m @ http://127.0.0.1:1234
|
||||
Strict policy: route
|
||||
|
||||
## Totals
|
||||
- questions_total: 25
|
||||
- ok_200_count: 25
|
||||
- semantic_pass_count: 25
|
||||
- semantic_pass_rate: 1.0
|
||||
- route_pass_count: 25
|
||||
- route_pass_rate: 1.0
|
||||
- strict_pass_count: 25
|
||||
- strict_pass_rate: 1.0
|
||||
- factual_count: 23
|
||||
- partial_coverage_count: 2
|
||||
- clarification_required_count: 0
|
||||
- http_error_count: 0
|
||||
- llm_decomposition_attempted_count: 25
|
||||
- llm_decomposition_applied_count: 22
|
||||
- llm_fallback_count: 0
|
||||
- llm_fallback_rate: 0.0
|
||||
- tool_gate_blocked_count: 0
|
||||
- tool_gate_blocked_rate: 0.0
|
||||
- avg_elapsed_ms: 6419.9
|
||||
|
||||
## Files
|
||||
- run_summary.json
|
||||
- full_live_results.json
|
||||
- failures_only.json
|
||||
|
|
@ -0,0 +1 @@
|
|||
[]
|
||||
File diff suppressed because it is too large
Load Diff
|
|
@ -0,0 +1,29 @@
|
|||
# Response Audit: 2026-04-03_Address_Nightly_followup_25_11-12-58
|
||||
|
||||
| id | strict | route_health | reply_type | intent | limited_reason | question | assistant_first_line |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| F001 | True | ok_or_factual | factual | list_documents_by_counterparty | None | Покажи документы по СВК за 2020 | Собран список документов по контрагенту (live address lane). |
|
||||
| F002 | True | ok_or_factual | factual | list_documents_by_counterparty | None | а теперь только за май 2020 | По окну 2020-05-01..2020-05-31 строк не найдено; показаны ближайшие доступные данные 2020-07-27..2021-11-10. |
|
||||
| F003 | True | ok_or_factual | factual | account_balance_snapshot | None | а по счету 60.01 на ту же дату | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| F004 | True | ok_or_factual | factual | account_balance_snapshot | None | а на ту же дату по 62.01 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| F005 | True | ok_or_factual | factual | documents_forming_balance | None | раскрой остаток по 62.01 документами на ту же дату | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| F006 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | Покажи банковские операции по СВК за 2020 | Собран список банковских операций по контрагенту (live address lane). |
|
||||
| F007 | True | ok_or_factual | factual | bank_operations_by_counterparty | None | а теперь только ноябрь 2020 | Период сохранен. Глубина live-выборки автоматически расширена до 1000 строк. |
|
||||
| F008 | True | ok_or_factual | factual | list_documents_by_counterparty | None | а теперь документы по нему | Период сохранен. Глубина live-выборки автоматически расширена до 1000 строк. |
|
||||
| F009 | True | ok_or_factual | factual | account_balance_snapshot | None | и остаток по 51 на ту же дату | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| F010 | True | ok_or_factual | factual | account_balance_snapshot | None | а теперь только май 2020 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| F011 | True | partial_non_blocking | partial_coverage | account_balance_snapshot | empty_match | Покажи сальдо по счету 60 на 2020-12-31 | В live-данных по текущему фильтру записи не найдены. |
|
||||
| F012 | True | ok_or_factual | factual | account_balance_snapshot | None | а теперь только май 2020 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| F013 | True | ok_or_factual | factual | account_balance_snapshot | None | а на ту же дату по 62 | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| F014 | True | ok_or_factual | factual | documents_forming_balance | None | раскрой 62 документами на ту же дату | Собран drilldown документов, формирующих остаток по счету на указанную дату. |
|
||||
| F015 | True | partial_non_blocking | partial_coverage | account_balance_snapshot | empty_match | вернись на 2020-12-31 по 60 | В live-данных по текущему фильтру записи не найдены. |
|
||||
| F016 | True | ok_or_factual | factual | list_documents_by_contract | None | Покажи документы по договору 19/15 за 2020 | Период сохранен. Глубина live-выборки автоматически расширена до 1000 строк. |
|
||||
| F017 | True | ok_or_factual | factual | bank_operations_by_contract | None | а теперь банковские операции | Период сохранен. Глубина live-выборки автоматически расширена до 1000 строк. |
|
||||
| F018 | True | ok_or_factual | factual | bank_operations_by_contract | None | а теперь только за 2020 | Период сохранен. Глубина live-выборки автоматически расширена до 1000 строк. |
|
||||
| F019 | True | ok_or_factual | factual | open_items_by_counterparty_or_contract | None | есть ли открытые позиции по договору 19/15 на 2020-12-31 | Период сохранен. Глубина live-выборки автоматически расширена до 1000 строк. |
|
||||
| F020 | True | ok_or_factual | factual | list_documents_by_contract | None | покажи документы по этому же договору | Собран список документов по договору (live address lane). |
|
||||
| F021 | True | ok_or_factual | factual | bank_operations_by_contract | None | Покажи банковские операции по договору 19/15 | Собран список банковских операций по договору (live address lane). |
|
||||
| F022 | True | ok_or_factual | factual | open_items_by_counterparty_or_contract | None | а теперь открытые позиции по нему | Период сохранен. Глубина live-выборки автоматически расширена до 1000 строк. |
|
||||
| F023 | True | ok_or_factual | factual | open_items_by_counterparty_or_contract | None | а теперь на 2020-12-31 | Период сохранен. Глубина live-выборки автоматически расширена до 1000 строк. |
|
||||
| F024 | True | ok_or_factual | factual | account_balance_snapshot | None | а по счету 51 на ту же дату | Адресный срез по счету собран (по движениям live MCP). |
|
||||
| F025 | True | ok_or_factual | factual | list_documents_by_contract | None | а документы по этому же договору за тот же период | Собран список документов по договору (live address lane). |
|
||||
|
|
@ -0,0 +1,142 @@
|
|||
{
|
||||
"run_id": "2026-04-03_Address_Nightly_followup_25_11-12-58",
|
||||
"generated_at": "2026-04-03T11:15:38",
|
||||
"source_questions_file": "X:\\1C\\NDC_1C\\docs\\ADDRESS\\question_sets\\address_followup_context_chains_2026-04-02.json",
|
||||
"backend_url": "http://127.0.0.1:8787/api/assistant/message",
|
||||
"llm_provider": "local",
|
||||
"llm_model": "qwen2.5-14b-instruct-1m",
|
||||
"llm_base_url": "http://127.0.0.1:1234",
|
||||
"strict_policy": "route",
|
||||
"totals": {
|
||||
"questions_total": 25,
|
||||
"ok_200_count": 25,
|
||||
"semantic_pass_count": 25,
|
||||
"semantic_pass_rate": 1.0,
|
||||
"route_pass_count": 25,
|
||||
"route_pass_rate": 1.0,
|
||||
"strict_pass_count": 25,
|
||||
"strict_pass_rate": 1.0,
|
||||
"factual_count": 23,
|
||||
"partial_coverage_count": 2,
|
||||
"clarification_required_count": 0,
|
||||
"http_error_count": 0,
|
||||
"llm_decomposition_attempted_count": 25,
|
||||
"llm_decomposition_applied_count": 22,
|
||||
"llm_fallback_count": 0,
|
||||
"llm_fallback_rate": 0.0,
|
||||
"tool_gate_blocked_count": 0,
|
||||
"tool_gate_blocked_rate": 0.0,
|
||||
"avg_elapsed_ms": 6419.9
|
||||
},
|
||||
"distributions": {
|
||||
"reply_type": {
|
||||
"factual": 23,
|
||||
"partial_coverage": 2
|
||||
},
|
||||
"actual_intent": {
|
||||
"list_documents_by_counterparty": 3,
|
||||
"account_balance_snapshot": 9,
|
||||
"documents_forming_balance": 2,
|
||||
"bank_operations_by_counterparty": 2,
|
||||
"list_documents_by_contract": 3,
|
||||
"bank_operations_by_contract": 3,
|
||||
"open_items_by_counterparty_or_contract": 3
|
||||
},
|
||||
"actual_mode": {
|
||||
"address_query": 25
|
||||
},
|
||||
"mcp_call_status": {
|
||||
"matched_non_empty": 23,
|
||||
"no_raw_rows": 2
|
||||
},
|
||||
"limited_reason_category": {
|
||||
"empty_match": 2
|
||||
},
|
||||
"route_health": {
|
||||
"ok_or_factual": 23,
|
||||
"partial_non_blocking": 2
|
||||
},
|
||||
"tool_gate_decision": {
|
||||
"run_address_lane": 25
|
||||
},
|
||||
"tool_gate_reason": {
|
||||
"address_mode_classifier_detected": 16,
|
||||
"llm_canonical_candidate_detected": 9
|
||||
}
|
||||
},
|
||||
"address_llm_predecompose_metrics": {
|
||||
"overall": {
|
||||
"llm_attempted": 25,
|
||||
"llm_applied": 22,
|
||||
"fallback_used": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"tool_gate_blocked": 0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"by_intent": {
|
||||
"list_documents_by_counterparty": {
|
||||
"total": 3,
|
||||
"llm_attempted": 3,
|
||||
"llm_applied": 3,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"account_balance_snapshot": {
|
||||
"total": 9,
|
||||
"llm_attempted": 9,
|
||||
"llm_applied": 7,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"documents_forming_balance": {
|
||||
"total": 2,
|
||||
"llm_attempted": 2,
|
||||
"llm_applied": 2,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"bank_operations_by_counterparty": {
|
||||
"total": 2,
|
||||
"llm_attempted": 2,
|
||||
"llm_applied": 2,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"list_documents_by_contract": {
|
||||
"total": 3,
|
||||
"llm_attempted": 3,
|
||||
"llm_applied": 3,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"bank_operations_by_contract": {
|
||||
"total": 3,
|
||||
"llm_attempted": 3,
|
||||
"llm_applied": 2,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
},
|
||||
"open_items_by_counterparty_or_contract": {
|
||||
"total": 3,
|
||||
"llm_attempted": 3,
|
||||
"llm_applied": 3,
|
||||
"fallback_used": 0,
|
||||
"tool_gate_blocked": 0,
|
||||
"fallback_rate": 0.0,
|
||||
"gate_block_rate": 0.0
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue