# v2.0.2 Eval Plan

## Objective
Validate stability/no-route/state-hardening changes in `normalizer_v2_0_2` and confirm operational readiness.

## Dataset Shape (20 labeled cases)

### Block A: Routable In-Scope (8 cases)
Expected:
- `message_in_scope=true`
- `predicted_clarification_required=false`
- `predicted_route_status=routed`
- no dominant `no_route`

### Block B: Legit Clarification (4 cases)
Expected:
- `message_in_scope=true`
- clarification required
- `route_status=no_route`
- `no_route_reason=insufficient_specificity`

### Block C: Out-of-Scope (4 cases)
Expected:
- `message_in_scope=false`
- `route_status=no_route`
- `no_route_reason=out_of_scope`
- fallback: `out_of_scope`

### Block D: Borderline / Soft Assumptions (4 cases)
Expected:
- `message_in_scope=true`
- routed
- `execution_readiness=executable_with_soft_assumptions`

## Required Labels per Case
- `expected_scope_in_scope`
- `expected_route_status`
- optional: `expected_no_route_reason`
- optional: `clarification_required`

## Metrics to Track
- `schema_validation_pass_rate`
- `scope_detection_accuracy`
- `route_resolution_accuracy`
- `no_route_precision`
- `false_no_route_rate`
- `execution_state_consistency_rate`
- `clarification_precision`
- `clarification_recall`

## Target Gates

Minimum acceptable:
- `schema_validation_pass_rate = 100`
- `execution_state_consistency_rate >= 95`
- `false_no_route_rate <= 10`

Preferred:
- `execution_state_consistency_rate = 100`
- `false_no_route_rate <= 5`

## Execution Command (example)

```bash
curl -X POST http://localhost:8787/api/eval/run \
  -H "Content-Type: application/json" \
  -d '{
    "mode": "single-pass-strict",
    "useMock": false,
    "caseSetFile": "eval_cases/normalizer_eval_v2_0_2_20cases.json",
    "normalizeConfig": {
      "promptVersion": "normalizer_v2_0_2",
      "schemaVersion": "v2_0_2",
      "model": "gpt-4o-mini",
      "temperature": 0,
      "maxOutputTokens": 900
    }
  }'
```

## Output Artifact
- Eval report JSON saved to: `data/eval_cases/eval-*.report.json`
- Must include new sections:
  - `route_eval`
  - `scope_eval`
  - `execution_state_eval`
  - v2.0.2 metrics in `metrics`
