Initial import NDC_1C
This commit is contained in:
@@ -0,0 +1,679 @@
|
||||
# Assistant Mode Global Status Report
|
||||
|
||||
Date: 2026-03-24
|
||||
Scope: `llm_normalizer` + Assistant Mode pipeline + retrieval/explainability contours
|
||||
Prepared for: architectural checkpoint and next-step planning
|
||||
|
||||
## 0) Executive Summary
|
||||
|
||||
Current system is no longer a raw route demo: we now have a working end-to-end assistant loop with decomposition, routing, retrieval, grounding, explainable response shaping, session logging, and regression tests.
|
||||
|
||||
At the same time, the system is still not a full accountant-grade investigation assistant. Main reason: data/model/retrieval unit depth is still below causal accounting reasoning depth in several domains.
|
||||
|
||||
Key status snapshot:
|
||||
|
||||
- Backend build/tests: `tsc` OK, `vitest` OK (`25/25` tests passed).
|
||||
- Explainable contract: implemented (`requirements`, `coverage_report`, `answer_grounding_check`, explainable reply sections).
|
||||
- Retrieval-layer upgrade: `executeHybrid` moved from `GUID-or-full-scan` to semantic profile + semantic narrowing.
|
||||
- Proven narrowing example: for bank mismatch query with accounts `51/60`, narrowing reduced records from `262` to `75`.
|
||||
- Proven limitation: for generic cross-entity chain query without explicit account scope, narrowing still wide (`262` to `242`), so answer quality can remain too broad.
|
||||
|
||||
---
|
||||
|
||||
## 1) Data Contour
|
||||
|
||||
### 1.1 How it works now
|
||||
|
||||
- Assistant retrieval reads local snapshot bundle from `docs/ARCH/2020экспорт`.
|
||||
- Main files currently loaded in executor:
|
||||
- `03_snapshot_fragment_problem_cases.json`
|
||||
- `04_samples_SpisanieSRaschetnogoScheta.json`
|
||||
- `05_samples_RealizaciyaTovarovUslug.json`
|
||||
- `06_samples_PostuplenieTovarovUslug.json`
|
||||
- `07_samples_DocumentJournals.json`
|
||||
- `08_samples_NDS_registers.json`
|
||||
- `09_samples_key_fields_Recorder_Ref_Supplier_Buyer_Responsible.json`
|
||||
- Data access is read-only snapshot, not live 1C state.
|
||||
|
||||
### 1.2 What works
|
||||
|
||||
- Documents/journals/register records are available with links and key attributes.
|
||||
- Counterparty/document linkage and part of relation topology are usable.
|
||||
- Enough depth exists for POC-level chain/risk analysis and explainable evidence pack.
|
||||
|
||||
### 1.3 Constraints
|
||||
|
||||
- Live truth is absent in assistant retrieval path (snapshot-only).
|
||||
- Lifecycle/status semantics are incomplete and partly heuristic.
|
||||
- Some accounting contexts are represented as flattened fields instead of normalized graph nodes.
|
||||
|
||||
### 1.4 What assistant cannot do because of this
|
||||
|
||||
- Guarantee real-time explanation of current accounting state.
|
||||
- Reliably prove deep causal accounting chains in all domains (especially where lifecycle semantics are implicit).
|
||||
|
||||
### 1.5 Symptoms already seen in dialogs
|
||||
|
||||
- Repeated top entities across semantically different but broad queries.
|
||||
- “Looks relevant” answers with weak differentiating evidence for some generic prompts.
|
||||
|
||||
### 1.6 Local changes needed
|
||||
|
||||
- Add richer field extraction/parsing from snapshot for account/document/lifecycle signals.
|
||||
- Enforce tighter domain-specific filters for low-specificity queries.
|
||||
|
||||
### 1.7 Architectural changes needed
|
||||
|
||||
- Add live data bridge layer for on-demand truth check (hybrid snapshot + live drilldown).
|
||||
- Add normalized accounting graph storage layer for causal traversal.
|
||||
|
||||
### 1.8 Priority
|
||||
|
||||
- `P0`: stronger retrieval constraints and lifecycle signal extraction.
|
||||
- `P1`: live bridge for targeted verification.
|
||||
- `P2`: full graph-backed data model.
|
||||
|
||||
---
|
||||
|
||||
## 2) Ontology / Domain Model Contour
|
||||
|
||||
### 2.1 How it works now
|
||||
|
||||
- Entity/relation semantics exist as retrieval profile vocabulary plus heuristic signal extraction.
|
||||
- Domain labels include bank/suppliers/customers/VAT/fixed_assets/deferred_expense/period_close/settlements.
|
||||
- Relation patterns include:
|
||||
- `payment_to_settlement`
|
||||
- `document_to_posting`
|
||||
- `statement_to_document`
|
||||
- `asset_card_to_depreciation`
|
||||
- `deferred_expense_to_writeoff`
|
||||
- `invoice_to_vat`
|
||||
- `contract_to_documents`
|
||||
- `receipt_to_stock_movement`
|
||||
|
||||
### 2.2 What works
|
||||
|
||||
- Query intent can be translated into semantic retrieval profile.
|
||||
- Basic anomaly vocabulary exists and affects ranking/explanation.
|
||||
|
||||
### 2.3 Constraints
|
||||
|
||||
- No explicit ontology graph engine with typed nodes/edges and reasoning rules.
|
||||
- Lifecycle model is heuristic (`created/posted/partially_linked/no_continuation/period_boundary`) rather than formal accounting state machine.
|
||||
|
||||
### 2.4 What assistant cannot do because of this
|
||||
|
||||
- Stable causal proofs for complex cross-domain reconciliation.
|
||||
- Deterministic explanation of “why exactly this stage is broken” across all domains.
|
||||
|
||||
### 2.5 Symptoms
|
||||
|
||||
- Explanation can still be structurally correct but semantically generic.
|
||||
- Retrieval unit can still drift toward “counterparty-heavy” answer shape.
|
||||
|
||||
### 2.6 Local changes needed
|
||||
|
||||
- Expand structured anomaly dictionary with accountant-facing defect classes.
|
||||
- Promote lifecycle markers from heuristics to explicit modeled states where possible.
|
||||
|
||||
### 2.7 Architectural changes needed
|
||||
|
||||
- Build ontology/lifecycle core as first-class subsystem.
|
||||
- Move from “labels on records” to “typed causal nodes and edges”.
|
||||
|
||||
### 2.8 Priority
|
||||
|
||||
- `P0`: anomaly taxonomy hardening + lifecycle schema hardening.
|
||||
- `P1`: typed ontology graph.
|
||||
- `P2`: rule engine over ontology.
|
||||
|
||||
---
|
||||
|
||||
## 3) Retrieval / Query Execution Contour
|
||||
|
||||
### 3.1 How it works now
|
||||
|
||||
- Deterministic routed executors:
|
||||
- `store_feature_risk`
|
||||
- `hybrid_store_plus_live`
|
||||
- `batch_refresh_then_store`
|
||||
- `store_canonical`
|
||||
- `live_mcp_drilldown`
|
||||
- `executeHybrid` now uses `semantic_retrieval_profile` and semantic narrowing when GUID is absent.
|
||||
- Retrieval result now carries richer context in items and summary (`query_subject`, profile, ranking basis, narrowing metrics).
|
||||
|
||||
### 3.2 What works
|
||||
|
||||
- No hard fallback to pure full scan in hybrid path for non-GUID queries.
|
||||
- Query with explicit accounting scope (`51/60`, wrong document closure) produces stronger narrowing and different ranking.
|
||||
- Evidence pack is richer and usable by explainable answer layer.
|
||||
|
||||
### 3.3 Constraints
|
||||
|
||||
- Generic prompts without explicit scope can still produce wide narrowed sets.
|
||||
- Retrieval top unit still often converges to counterparty-centric grouping.
|
||||
- Not all routes have equal semantic depth.
|
||||
|
||||
### 3.4 What assistant cannot do because of this
|
||||
|
||||
- Consistently deliver problem-node-first output in every query class.
|
||||
- Guarantee high differentiation for all semantically close prompts.
|
||||
|
||||
### 3.5 Symptoms
|
||||
|
||||
- For some queries, narrowing reduction is still modest (example `262 -> 242`).
|
||||
- Answers can remain “good but broad”.
|
||||
|
||||
### 3.6 Local changes needed
|
||||
|
||||
- Tighten mandatory intersections for generic bank/cross-entity prompts.
|
||||
- Add domain-specific minimum evidence thresholds before final top ranking.
|
||||
|
||||
### 3.7 Architectural changes needed
|
||||
|
||||
- Introduce explicit “problem cluster” retrieval unit.
|
||||
- Add cross-branch retrieval policy (neighbor contour checks).
|
||||
|
||||
### 3.8 Priority
|
||||
|
||||
- `P0`: further narrowing hardening + anti-generic ranking guards.
|
||||
- `P1`: problem-cluster retrieval unit.
|
||||
- `P2`: multi-branch investigation retrieval policy.
|
||||
|
||||
---
|
||||
|
||||
## 4) LLM Layer and Decomposition Contour
|
||||
|
||||
### 4.1 How it works now
|
||||
|
||||
- Prompt/schema baseline: `normalizer_v2_0_2`.
|
||||
- Deterministic v2 routing summary with fallback types.
|
||||
- Requirements extraction + coverage report + dropped-intent tracking are implemented.
|
||||
|
||||
### 4.2 What works
|
||||
|
||||
- Route and execution readiness are explicit.
|
||||
- Coverage and grounding diagnostics are available per turn.
|
||||
- Route mismatch blocking is now less false-positive for non-critical contextual tokens.
|
||||
|
||||
### 4.3 Constraints
|
||||
|
||||
- Requirement extraction remains coarse in many cases (often 1 requirement per fragment).
|
||||
- Transliteration/noisy mixed-language prompts still degrade in-scope detection.
|
||||
|
||||
### 4.4 What assistant cannot do because of this
|
||||
|
||||
- Fine-grained multi-requirement planning for complex accounting requests.
|
||||
- Fully robust handling of colloquial/translit business language.
|
||||
|
||||
### 4.5 Symptoms
|
||||
|
||||
- Some translit prompts fall into `out_of_scope/clarification`.
|
||||
- Partial semantic intent may be compressed in long multi-part prompts.
|
||||
|
||||
### 4.6 Local changes needed
|
||||
|
||||
- Expand language normalization and translit alias mapping before decomposition.
|
||||
- Improve requirement extraction granularity inside one fragment.
|
||||
|
||||
### 4.7 Architectural changes needed
|
||||
|
||||
- Add dedicated semantic parser layer before normalizer for business-language canonicalization.
|
||||
- Add requirement graph (instead of flat list) for planning/execution.
|
||||
|
||||
### 4.8 Priority
|
||||
|
||||
- `P0`: translit/business alias normalization.
|
||||
- `P1`: requirement graph extraction.
|
||||
- `P2`: adaptive decomposition policy.
|
||||
|
||||
---
|
||||
|
||||
## 5) Answer Synthesis / Explanation Contour
|
||||
|
||||
### 5.1 How it works now
|
||||
|
||||
- Reply types include:
|
||||
- `factual_with_explanation`
|
||||
- `partial_coverage`
|
||||
- `clarification_required`
|
||||
- `no_grounded_answer`
|
||||
- `route_mismatch_blocked`
|
||||
- others
|
||||
- Response includes explainable sections: result, why included, selection basis, risk signs, business meaning, limitations, next step.
|
||||
|
||||
### 5.2 What works
|
||||
|
||||
- Core explainable contract is implemented and stable.
|
||||
- Blocking logic prevents clearly mismatched subject answers.
|
||||
|
||||
### 5.3 Constraints
|
||||
|
||||
- Generic wording still appears when retrieval unit is broad.
|
||||
- Explanations are still largely template-driven for some routes.
|
||||
|
||||
### 5.4 What assistant cannot do because of this
|
||||
|
||||
- Deliver fully case-unique accountant-level narratives in all scenarios.
|
||||
|
||||
### 5.5 Symptoms
|
||||
|
||||
- Two semantically close broad prompts may yield similar explanatory skeleton.
|
||||
|
||||
### 5.6 Local changes needed
|
||||
|
||||
- Route-specific explanation templates with stronger domain phrasing.
|
||||
- Explicit “mechanism-of-failure” fields in retrieval result for composer.
|
||||
|
||||
### 5.7 Architectural changes needed
|
||||
|
||||
- Separate explanation planner from template renderer.
|
||||
- Add accountant-facing narrative policy with domain lexicon packs.
|
||||
|
||||
### 5.8 Priority
|
||||
|
||||
- `P0`: route-specific explanation enrichment.
|
||||
- `P1`: mechanism-level explanation fields.
|
||||
- `P2`: explanation planner subsystem.
|
||||
|
||||
---
|
||||
|
||||
## 6) Memory / State / Session Continuity Contour
|
||||
|
||||
### 6.1 How it works now
|
||||
|
||||
- Session-scoped conversation state is persisted.
|
||||
- One JSON file per session with turn-level human-readable + technical blocks.
|
||||
|
||||
### 6.2 What works
|
||||
|
||||
- Stable conversation history and replay.
|
||||
- Explicit per-turn decomposition and response auditability.
|
||||
|
||||
### 6.3 Constraints
|
||||
|
||||
- No robust investigation state model (hypotheses/open checks/resolution graph).
|
||||
- Context memory is conversational, not analytical.
|
||||
|
||||
### 6.4 What assistant cannot do because of this
|
||||
|
||||
- True multi-step investigative reasoning with hypothesis tracking.
|
||||
|
||||
### 6.5 Symptoms
|
||||
|
||||
- Follow-up can be coherent but not yet “investigation-driven”.
|
||||
|
||||
### 6.6 Local changes needed
|
||||
|
||||
- Add per-session `investigation_state` object (focus, active entities, open hypotheses, unresolved branches).
|
||||
|
||||
### 6.7 Architectural changes needed
|
||||
|
||||
- Add working-memory layer for research workflow, not only chat continuity.
|
||||
|
||||
### 6.8 Priority
|
||||
|
||||
- `P0`: investigation_state schema + persistence.
|
||||
- `P1`: branch tracking and hypothesis status transitions.
|
||||
- `P2`: multi-turn analytical planning engine.
|
||||
|
||||
---
|
||||
|
||||
## 7) Orchestration / Routing / Control Policy Contour
|
||||
|
||||
### 7.1 How it works now
|
||||
|
||||
- Deterministic routing with fallback (`none/out_of_scope/clarification/partial`).
|
||||
- Linear execution plan per turn.
|
||||
|
||||
### 7.2 What works
|
||||
|
||||
- Clear route decisions and no-route reasons.
|
||||
- Strong deterministic observability.
|
||||
|
||||
### 7.3 Constraints
|
||||
|
||||
- Mostly route-driven linear pipeline.
|
||||
- Limited iterative branch exploration initiated by system policy.
|
||||
|
||||
### 7.4 What assistant cannot do because of this
|
||||
|
||||
- Automatically run neighbor contour verification when primary evidence is weak.
|
||||
|
||||
### 7.5 Symptoms
|
||||
|
||||
- Reasonable direct answers, but limited self-initiated investigation depth.
|
||||
|
||||
### 7.6 Local changes needed
|
||||
|
||||
- Introduce confidence-driven secondary retrieval triggers.
|
||||
|
||||
### 7.7 Architectural changes needed
|
||||
|
||||
- Orchestration policy engine with iterative reasoning loops and stop criteria.
|
||||
|
||||
### 7.8 Priority
|
||||
|
||||
- `P0`: confidence-based secondary checks.
|
||||
- `P1`: branch exploration policy.
|
||||
- `P2`: full investigation orchestrator.
|
||||
|
||||
---
|
||||
|
||||
## 8) Quality / Observability / Eval Contour
|
||||
|
||||
### 8.1 How it works now
|
||||
|
||||
- Structured runtime logs (stdout JSON).
|
||||
- Trace storage and session logs.
|
||||
- Regression tests for API behavior, grounding and retrieval semantics.
|
||||
|
||||
### 8.2 What works
|
||||
|
||||
- Technical observability is strong for current stage.
|
||||
- Automated test baseline is green (`25/25`).
|
||||
|
||||
### 8.3 Constraints
|
||||
|
||||
- Limited accountant-utility evaluation metrics.
|
||||
- No broad canonical scenario benchmark with decision-quality scoring.
|
||||
|
||||
### 8.4 What assistant cannot do because of this
|
||||
|
||||
- Provide hard quantitative proof of business usefulness across accounting domains.
|
||||
|
||||
### 8.5 Symptoms
|
||||
|
||||
- Technical success may still exceed practical user-perceived success.
|
||||
|
||||
### 8.6 Local changes needed
|
||||
|
||||
- Add eval metrics:
|
||||
- retrieval differentiation rate
|
||||
- generic explanation rate
|
||||
- accountant actionability score
|
||||
- false confidence rate
|
||||
|
||||
### 8.7 Architectural changes needed
|
||||
|
||||
- Build domain eval harness with canonical accounting scenarios and target outcomes.
|
||||
|
||||
### 8.8 Priority
|
||||
|
||||
- `P0`: metric instrumentation for practical usefulness.
|
||||
- `P1`: canonical benchmark suite (bank/60/97/OS/VAT/period close/multi-intent/translit/follow-up).
|
||||
- `P2`: continuous quality dashboard.
|
||||
|
||||
---
|
||||
|
||||
## 9) Evolutionary Architecture Contour
|
||||
|
||||
### 9.1 Missing pieces (high impact)
|
||||
|
||||
- Ontology graph core.
|
||||
- Lifecycle engine.
|
||||
- Problem-cluster retrieval unit.
|
||||
- Investigation memory/state.
|
||||
- Orchestration policy engine for iterative checks.
|
||||
- Live verification bridge for source-of-truth escalation.
|
||||
|
||||
### 9.2 Current ceiling
|
||||
|
||||
- Without deeper ontology/lifecycle/state layers, system remains strong “explainable routed assistant”, but not full accountant investigation copilot.
|
||||
|
||||
### 9.3 Local vs architectural changes
|
||||
|
||||
- Local: better filters, better templates, more metrics, better parser.
|
||||
- Architectural: graph model, lifecycle engine, investigation state, multi-step orchestrator.
|
||||
|
||||
### 9.4 Priority
|
||||
|
||||
- `P0`: finish semantic retrieval hardening + practical eval metrics + investigation_state baseline.
|
||||
- `P1`: ontology/lifecycle formalization + problem-cluster retrieval.
|
||||
- `P2`: iterative orchestrator + live verification framework.
|
||||
|
||||
---
|
||||
|
||||
## 10) Answers to 12 Mandatory Questions
|
||||
|
||||
1. What data reaches assistant and where detail is lost:
|
||||
Data reaches from snapshot package with links/attributes; detail loss happens in flattening/grouping and lack of formal lifecycle semantics.
|
||||
|
||||
2. Full domain model exists:
|
||||
Partially. Semantic labels and patterns exist, formal ontology graph does not.
|
||||
|
||||
3. Primary retrieval unit:
|
||||
Mostly counterparty-grouped chain/risk clusters; not yet universal problem-node unit.
|
||||
|
||||
4. Real constraints and wide-scan risk:
|
||||
Constraints now executed in hybrid semantic profile, but generic queries can still remain broad.
|
||||
|
||||
5. What LLM receives before answer:
|
||||
Normalizer output + route summary + normalized retrieval payload + grounding/coverage diagnostics.
|
||||
|
||||
6. What is lost in decomposition:
|
||||
Fine-grained multi-requirement structure can still compress; translit/noisy input can lose intent quality.
|
||||
|
||||
7. Why explanation still generic in places:
|
||||
Broad retrieval unit + template-driven synthesis with limited mechanism-specific fields.
|
||||
|
||||
8. Can system explain mechanism (not only labels):
|
||||
Partially. Better than before, still constrained by retrieval evidence depth.
|
||||
|
||||
9. Working state/memory for multi-step analysis:
|
||||
Conversation memory exists; investigation memory model is missing.
|
||||
|
||||
10. Can system explore neighbor accounting branches automatically:
|
||||
Not yet as policy standard; mostly linear route execution.
|
||||
|
||||
11. How usefulness is measured:
|
||||
Technical pipeline quality is measured; accountant-facing utility metrics are not complete yet.
|
||||
|
||||
12. Missing architectural entities preventing next quality tier:
|
||||
Ontology graph, lifecycle engine, problem-cluster unit, investigation state, iterative orchestration.
|
||||
|
||||
---
|
||||
|
||||
## 11) Current Phase Status (Condensed)
|
||||
|
||||
- Phase status: `Functional MVP+` (explainable routed assistant with semantic retrieval upgrade).
|
||||
- Not yet: `Production accountant copilot`.
|
||||
- Immediate gate to next phase: tighten broad-query narrowing + add practical accountant eval metrics + investigation state schema.
|
||||
|
||||
---
|
||||
|
||||
## 12) Recommended Next Step Pack
|
||||
|
||||
### P0 (next iteration)
|
||||
|
||||
- Tighten generic-query semantic narrowing in hybrid route.
|
||||
- Add investigation state object in session model.
|
||||
- Add practical eval metrics (differentiation/actionability/generic-rate).
|
||||
|
||||
### P1 (after P0 stabilization)
|
||||
|
||||
- Formalize ontology + lifecycle layers.
|
||||
- Shift retrieval output from entity-heavy to problem-cluster-heavy for key domains.
|
||||
|
||||
### P2 (strategic)
|
||||
|
||||
- Add iterative orchestration with neighbor-branch verification.
|
||||
- Add live source-of-truth verification path for high-confidence conclusions.
|
||||
|
||||
---
|
||||
|
||||
## 13) Data Loss Map (Source to LLM)
|
||||
|
||||
This section is the explicit loss map requested for architecture decisions.
|
||||
|
||||
| Source Layer | Current Internal Representation | Lost/Weakened Signals | Observable Assistant Symptom | Required Fix Layer |
|
||||
|---|---|---|---|---|
|
||||
| 1C document/journal/register snapshot record | flattened `SnapshotRecord` + heuristic signal extraction | formal business status transitions, typed lifecycle stage semantics | explanation can be structurally correct but semantically generic | lifecycle model + ontology graph |
|
||||
| document + posting relation hints | relation pattern labels inferred by regex/rules | deterministic causal edge type and confidence | “close to right chain” answers without strict mechanism proof | typed relation graph + relation confidence |
|
||||
| account hints from query and record fields | `account_scope` and inferred `account_context` arrays | strong account-role semantics (main vs side context) | broad retrieval if account scope is not explicit | account-role policy in retrieval profile |
|
||||
| anomaly signs (`unknown links`, `zero guid`, etc.) | anomaly pattern tags (`missing_link`, `broken_lifecycle`, etc.) | accountant-grade defect class and business consequence mapping | same anomaly labels across semantically different defects | anomaly catalog and mapping engine |
|
||||
| session chat turns | conversation list + turn log | investigation branch state and hypothesis state | follow-up can be coherent but not deeply investigative | investigation_state subsystem |
|
||||
| snapshot-only truth | no guaranteed live verification step in assistant route | real-time status confirmation | high-quality but potentially stale conclusion in sensitive cases | live verification bridge |
|
||||
|
||||
### 13.1 Diagnostic implication
|
||||
|
||||
The dominant ceiling is not “weak wording” but “insufficiently structured causal context before synthesis”.
|
||||
|
||||
---
|
||||
|
||||
## 14) Query Class vs Required Architecture Depth
|
||||
|
||||
| User Query Class | Required Layers | Current Readiness | Ceiling Cause | Next Upgrade |
|
||||
|---|---|---|---|---|
|
||||
| simple factual object lookup | routing + canonical retrieval + basic grounding | medium/high | snapshot-only verification | optional live drilldown |
|
||||
| anomaly ranking (one contour) | semantic profile + risk retrieval + explainable synthesis | medium | anomaly semantics still heuristic | anomaly catalog hardening |
|
||||
| causal chain in one contour | relation patterns + chain retrieval + evidence pack | medium | retrieval unit still entity-heavy in broad prompts | problem-cluster unit |
|
||||
| cross-domain reconciliation | ontology + lifecycle + neighbor branch policy | low/medium | no formal cross-domain causal graph | ontology graph + branch policy |
|
||||
| period-close impact analysis | lifecycle + period-risk model + orchestration | low/medium | lifecycle model incomplete | lifecycle engine |
|
||||
| multi-step investigation with follow-up | investigation_state + orchestration loops + hypothesis tracking | low | memory is conversational, not investigative | investigation mode layer |
|
||||
| ambiguity-heavy/translit business language | semantic parser + alias normalization + decomposition guard | low/medium | parser limitations before routing | pre-normalization parser layer |
|
||||
|
||||
### 14.1 Decision implication
|
||||
|
||||
Prompt/model tuning alone cannot close low-readiness classes above; they are architecture-depth dependent.
|
||||
|
||||
---
|
||||
|
||||
## 15) Retrieval Unit Diagnosis (Core Bottleneck)
|
||||
|
||||
### 15.1 Current dominant unit
|
||||
|
||||
- Dominant unit in hybrid route is still often `counterparty group`, even after semantic narrowing.
|
||||
|
||||
### 15.2 Where this unit is acceptable
|
||||
|
||||
- quick ranking
|
||||
- initial risk surfacing
|
||||
- broad operational scanning
|
||||
|
||||
### 15.3 Where this unit breaks answer quality
|
||||
|
||||
- “what exactly is broken in chain”
|
||||
- “closed by wrong document type”
|
||||
- “which lifecycle stage is inconsistent”
|
||||
- “what blocks period close and why”
|
||||
|
||||
### 15.4 Target retrieval units (must become first-class)
|
||||
|
||||
- `document_conflict`
|
||||
- `broken_chain_segment`
|
||||
- `lifecycle_anomaly_node`
|
||||
- `unresolved_settlement_cluster`
|
||||
- `period_risk_cluster`
|
||||
- `cross_branch_inconsistency_cluster`
|
||||
|
||||
### 15.5 Transition plan
|
||||
|
||||
- Step 1 (`P0`): keep counterparty groups but add explicit `mechanism_of_failure` + `failed_expected_edge`.
|
||||
- Step 2 (`P1`): introduce mixed-unit ranking (problem cluster first, entity second).
|
||||
- Step 3 (`P2`): use problem-cluster as default answer unit for chain/anomaly/period-risk routes.
|
||||
|
||||
---
|
||||
|
||||
## 16) LLM Ceiling Boundaries (Not Solvable by Prompt Alone)
|
||||
|
||||
The following limitations remain even with stronger models/prompts unless architecture changes:
|
||||
|
||||
1. no formal lifecycle state machine on input -> model cannot produce deterministic lifecycle diagnosis;
|
||||
2. no typed causal graph edges -> model cannot consistently prove mechanism, only infer plausible narrative;
|
||||
3. entity-heavy retrieval unit -> model can explain “who is risky”, but not always “what exact mechanism broke”;
|
||||
4. missing investigation_state -> model cannot reliably manage long hypothesis trees across turns;
|
||||
5. no mandatory live verification gate -> model cannot guarantee real-time truth in high-stakes answers.
|
||||
|
||||
### 16.1 Governance rule
|
||||
|
||||
When limitations above are active, quality work must target data/model/orchestration layers first; LLM tuning is secondary.
|
||||
|
||||
---
|
||||
|
||||
## 17) Investigation Mode Specification (Required Next Architecture)
|
||||
|
||||
### 17.1 Minimal `investigation_state` schema
|
||||
|
||||
```json
|
||||
{
|
||||
"session_id": "asst-...",
|
||||
"focus": {
|
||||
"domain": "bank_settlements",
|
||||
"period": "2020-06",
|
||||
"primary_accounts": ["51", "60"]
|
||||
},
|
||||
"active_entities": [
|
||||
{ "type": "counterparty", "id": "..." },
|
||||
{ "type": "document", "id": "..." }
|
||||
],
|
||||
"open_hypotheses": [
|
||||
{
|
||||
"hypothesis_id": "H1",
|
||||
"statement": "closure performed by wrong document type",
|
||||
"status": "open",
|
||||
"evidence_for": [],
|
||||
"evidence_against": []
|
||||
}
|
||||
],
|
||||
"branches": [
|
||||
{
|
||||
"branch_id": "B1",
|
||||
"name": "bank->settlement",
|
||||
"status": "in_progress",
|
||||
"unresolved_reason": null
|
||||
}
|
||||
],
|
||||
"resolved_findings": [],
|
||||
"next_actions": []
|
||||
}
|
||||
```
|
||||
|
||||
### 17.2 Required branch lifecycle
|
||||
|
||||
- `open` -> `in_progress` -> `confirmed` or `rejected` -> `closed`
|
||||
|
||||
### 17.3 System-initiated branch rule (minimum)
|
||||
|
||||
If primary route confidence is high but mechanism evidence is weak, assistant should launch one neighbor branch check before final high-confidence conclusion.
|
||||
|
||||
---
|
||||
|
||||
## 18) Symptom to Root Cause to Required Layer
|
||||
|
||||
| Symptom | Root Cause | Required Layer |
|
||||
|---|---|---|
|
||||
| generic explanation despite “ok” reply | mechanism fields missing in retrieval payload | retrieval schema + answer planner |
|
||||
| similar answers for broad prompts | weak semantic narrowing for low-specificity queries | retrieval policy |
|
||||
| follow-up does not deepen analysis | no hypothesis/branch state | investigation_state |
|
||||
| strong dependence on explicit account hints | weak semantic parser/ontology grounding | parser + ontology |
|
||||
| lifecycle conclusions not stable | lifecycle semantics heuristic only | lifecycle engine |
|
||||
| high confidence on snapshot-only route | no live verification gate | live verification bridge |
|
||||
|
||||
---
|
||||
|
||||
## 19) Value-Impact Roadmap (Decision Table)
|
||||
|
||||
| Change | Complexity | Quality Gain | Accountant Usefulness Gain | Multi-step Investigation Gain | Priority |
|
||||
|---|---|---|---|---|---|
|
||||
| tighten generic semantic narrowing | low/medium | high | high | medium | P0 |
|
||||
| add `mechanism_of_failure` retrieval fields | medium | high | high | medium | P0 |
|
||||
| add `investigation_state` persistence | medium | medium/high | high | high | P0 |
|
||||
| add practical utility eval metrics | low/medium | medium | high | medium | P0 |
|
||||
| formalize anomaly catalog | medium | medium/high | high | medium | P1 |
|
||||
| ontology graph core | high | high | high | high | P1 |
|
||||
| lifecycle engine | high | high | high | high | P1 |
|
||||
| problem-cluster retrieval unit | high | high | high | high | P1 |
|
||||
| iterative orchestration engine | high | high | high | very high | P2 |
|
||||
| live verification bridge | high | medium/high | high | medium/high | P2 |
|
||||
|
||||
---
|
||||
|
||||
## 20) What Not To Do (Explicit Guardrails)
|
||||
|
||||
1. Do not attempt to solve mechanism-level quality only with prompt edits.
|
||||
2. Do not treat richer wording as substitute for stronger retrieval unit.
|
||||
3. Do not scale explanation templates without adding mechanism evidence fields.
|
||||
4. Do not equate long conversation history with investigation_state.
|
||||
5. Do not claim production-grade confidence without live verification path for critical answers.
|
||||
|
||||
@@ -0,0 +1,63 @@
|
||||
# Assistant Mode Global Status Appendix
|
||||
|
||||
Date: 2026-03-24
|
||||
|
||||
## A) Verification Commands
|
||||
|
||||
```powershell
|
||||
cd X:\1C\NDC_1C\llm_normalizer\backend
|
||||
npm.cmd run build
|
||||
npm.cmd run test
|
||||
```
|
||||
|
||||
Observed result:
|
||||
|
||||
- TypeScript build: success
|
||||
- Test suite: success (`25/25`)
|
||||
|
||||
## B) Retrieval Narrowing Evidence
|
||||
|
||||
### Case 1: bank mismatch with explicit account scope
|
||||
|
||||
- Session: `asst-FuRihiL5Bp`
|
||||
- Query subject: `bank_settlement_mismatch`
|
||||
- Source records: `262`
|
||||
- Filtered after narrowing: `75`
|
||||
- Semantic narrowing applied: `true`
|
||||
|
||||
### Case 2: generic cross-entity bank chain
|
||||
|
||||
- Session: `asst-j9spgqdY7k`
|
||||
- Query subject: `cross_entity_breakage`
|
||||
- Source records: `262`
|
||||
- Filtered after narrowing: `242`
|
||||
- Semantic narrowing applied: `true`
|
||||
|
||||
Interpretation:
|
||||
|
||||
- Semantic narrowing is active and effective for constrained accounting scope.
|
||||
- Generic prompts still need stronger narrowing policy.
|
||||
|
||||
## C) Key Implementation Anchors
|
||||
|
||||
- Semantic profile contract and builder:
|
||||
- `X:\1C\NDC_1C\llm_normalizer\backend\src\services\assistantDataLayer.ts`
|
||||
- Hybrid narrowing and enriched evidence pack:
|
||||
- `X:\1C\NDC_1C\llm_normalizer\backend\src\services\assistantDataLayer.ts`
|
||||
- API regression test for semantic narrowing:
|
||||
- `X:\1C\NDC_1C\llm_normalizer\backend\tests\assistantEndpoint.test.ts`
|
||||
|
||||
## D) v1.1 Report Reinforcement Checklist
|
||||
|
||||
All 4 requested reinforcements are now explicitly present in the main report:
|
||||
|
||||
1. Data-loss path map (`Source -> Internal -> Lost -> Symptom -> Fix layer`)
|
||||
2. Query-class vs architecture-depth matrix
|
||||
3. Dedicated retrieval-unit diagnosis block (current vs target units)
|
||||
4. Investigation mode schema and control-policy baseline
|
||||
|
||||
Also added:
|
||||
|
||||
- symptom -> root cause -> required layer matrix
|
||||
- value-impact roadmap table
|
||||
- explicit “what not to do” guardrails
|
||||
Reference in New Issue
Block a user