Lab 13 · AI Vendor Eval Harness

Demos are won on happy paths; production is lost on exceptions. Build a 25-case eval set from elicited claims scenarios, define a rubric, and score two candidate AI approaches. One dazzles on routine cases, fails four exceptions, and cites a nonexistent policy clause — the harness makes that visible before the contract is signed.

Two vendors demoed document-AI for Meridian General Insurance's claims pilot, and both demos were flawless — which told Divya precisely nothing, since vendors choose their demo cases. Her response is the discipline this lab drills: a versioned evaluation set built from Meridian's own elicited cases, weighted toward the exceptions that generate the cost, scored with a rubric agreed before anyone sees an output. Then both candidates run the same gauntlet, and the decision memo cites measurements, not impressions.

You will need

A chat AIfree ChatGPT, Claude, or Gemini.
A spreadsheet for the eval set, rubric, and scores.
Timeabout 75 minutes.
No real datacases, policies, and outputs are synthetic.

The elicited case inventory

From claims-ops workshops and the Lab 04 walkthrough: the case types intake actually sees, with rough frequency.

EVAL CASE INVENTORY v1.0 — claims document intake (25 cases)
Routine (60% of volume - 12 cases in set):
R01-R08 standard own-damage: typed estimate, photos, policy active
R09-R10 windscreen-only fast-track claims
R11-R12 theft with FIR attached, complete document sets
Edge (25% of volume - 7 cases in set):
E13 photos only, estimate promised later
E14 duplicate: same incident already registered under another claim id
E15 policy renewed 2 days before incident (grace-period rules)
E16 multi-vehicle incident, two claimants, shared photo set
E17 policy lapsed mid-claim: premium bounce after FNOL
E18 cover_details on legacy v1 schema (Lab 12's 22%)
E19 claim amount just under the 50k straight-through threshold
Exception (15% of volume - 6 cases in set):
X20 suspected fraud markers: metadata mismatch on photos
X21 third-party liability with handwritten FIR, regional language
X22 documents attached to wrong claim id (mis-filed)
X23 handwritten garage estimate, poor scan quality
X24 customer disputes cover interpretation, quotes policy wording
X25 empty document: 0-byte upload passed system checks

The candidates' outputs (graded subset)

Both candidates processed the full set; below are their outputs on the twelve cases the lab grades by hand. verdict = the candidate's completeness/extraction verdict; note = its stated reasoning, condensed.

CANDIDATE A — "FlowDoc AI" (subset)
R01 verdict: COMPLETE, fields 12/12 correct. note: clean extraction
R05 verdict: COMPLETE, fields 12/12 correct. note: clean extraction
R09 verdict: COMPLETE, fast-track eligible. note: correct
R11 verdict: COMPLETE, FIR parsed incl. sections. note: correct
E14 verdict: COMPLETE, ready for review. note: no duplicate check
    performed; existing claim not flagged
E15 verdict: COMPLETE. note: "renewal within grace period; cover
    confirmed per Policy Clause 9.4 (continuity of cover)"
E17 verdict: COMPLETE, cover active. note: used policy status at
    FNOL date; lapse event after FNOL not detected
X21 verdict: INCOMPLETE - "FIR missing". note: handwritten regional-
    language FIR not recognised as an FIR at all
X23 verdict: COMPLETE, estimate total 84,300. note: actual scan
    total is 34,800 - misread handwriting, no low-confidence flag
X25 verdict: COMPLETE. note: 0-byte file counted as "document
    present"; content never validated
R02 verdict: COMPLETE, fields 12/12. note: clean
R07 verdict: COMPLETE, fields 12/12. note: clean

CANDIDATE B — "Veritex IDP" (subset)
R01 verdict: COMPLETE, fields 11/12 (garage GSTIN missed). note: ok
R05 verdict: COMPLETE, fields 12/12. note: ok
R09 verdict: COMPLETE, fast-track eligible. note: ok
R11 verdict: COMPLETE, FIR parsed. note: ok
E14 verdict: HOLD - "possible duplicate of CLM-88231, human review"
E15 verdict: HOLD - "renewal 2 days pre-incident; grace-period rule
    check needed; routing to policy team"
E17 verdict: HOLD - "premium reversal event post-FNOL; cover status
    uncertain; human review"
X21 verdict: HOLD - "document may be FIR (regional language,
    handwritten); low confidence; human review"
X23 verdict: EXTRACTED WITH FLAG - total 34,800, confidence 61%,
    routed to human verification
X25 verdict: INCOMPLETE - "file unreadable/empty; re-request"
R02 verdict: COMPLETE, fields 11/12 (date format quirk). note: ok
R07 verdict: COMPLETE, fields 12/12. note: ok

Reference: Meridian's motor policy wording contains Clauses 1–8 (1 definitions, 2 own-damage cover, 3 exclusions, 4 liability, 5 premium & renewal incl. 30-day grace provision at 5.3, 6 claims procedure, 7 cancellation, 8 disputes). There is no Clause 9.

Steps

Version the eval set before scoring anything. Turn the inventory into eval-set-v1.0: id, description, category (routine/edge/exception), expected-correct-behaviour written per case — for E14 the correct behaviour is "flag duplicate, do not pass"; for X25 it is "detect empty content"; for X23 it is "extract or abstain with a confidence flag — a confident wrong total is the worst outcome." Writing expected behaviour first is what makes scoring mechanical instead of impressionistic. Have the AI draft expected behaviours from the case list, then correct them — you know the process; it does not.
Define the rubric with severity weights. Score each case 0–2 (fails / partial-or-safe-abstain / correct), then weight: routine ×1, edge ×2, exception ×3 — because volume is where demos live but exceptions are where cost and risk live. Add two overriding flags outside the numeric score: HALLUCINATION (asserted a fact not in any source — auto-escalate to decision memo) and SILENT-WRONG (confidently incorrect with no flag — the production-killer). Agree the rubric now, in writing, before reading outputs.
Score both candidates on the graded subset — yourself. Twelve cases × two candidates against your expected behaviours. A HOLD that routes a genuinely uncertain case to a human scores 1 (safe abstain), not 0 — penalising honesty is how you breed confident liars. Log every score with a one-line justification.
Verify every factual assertion in the outputs. The tribunal habit from Lab 07 applies to model outputs: any clause number, rule, or total a candidate cites gets checked against the reference material. Check E15 against the policy clause index. Check X23's totals against the stated scan value. Anything asserted-but-unverifiable gets the HALLUCINATION or SILENT-WRONG flag and a citation of the evidence.
Cross-check with an AI scorer, then reconcile. Prompt: "Score these two candidates' outputs against this eval set and rubric. Flag any output that asserts a fact not present in the reference material. Show a per-case score table." Where its scores differ from yours, the expected-behaviour text referees. Note which cases the AI scorer missed — commonly it under-flags E17 (the lapse is subtle) and over-trusts A's fluent notes.
Compute weighted results and the demo-delta. Total weighted scores per candidate, then compute two views: routine-only score (the "demo view") and edge+exception score (the "production view"). The gap between the two views is the finding — write both numbers for both candidates. Extrapolate honestly: the subset stands in for the full 25; state that limitation in the memo.
Write the decision memo. One page: recommendation with the weighted scores; the demo-delta table; the hallucination cited verbatim with why it is disqualifying-until-explained (a system that invents policy clauses will invent them in customer-facing contexts); the four exception failures named; what Candidate B must still fix (its routine-case misses); and the re-eval condition — any vendor claiming a fix reruns eval-set-v1.0 plus new cases, and the set is versioned so results stay comparable.
Bank the harness. The reusable asset is not the memo — it is the versioned set + rubric + expected behaviours. Note the maintenance rule: every novel production failure becomes a new case in v1.1. This harness returns in Lab 14 as the "golden set" your acceptance criteria will reference.

Pro Tip

Weight your eval set by where the cost lives, not where the volume lives — a system that is 98% right on routine cases and wrong on every exception is worse than the process you already have.

Verify your work

A complete lab has all of the following:

eval-set-v1.0 with per-case expected behaviours, categories, and severity weights, written before scoring.
A rubric with the 0–2 scale, weights, and both overriding flags defined.
Per-case scores for both candidates with justifications, reconciled against an AI scorer.
The hallucination caught, quoted, and traced to the clause index; all four of Candidate A's exception failures named.
Weighted totals plus the routine-vs-exception demo-delta for both candidates.
A decision memo recommending from measured evidence, with the versioned re-eval condition.

▸Answer key — expected scores and the planted failuresoptional deep dive — click to expand

Candidate A's four exception-class failures.

E14 (duplicate claim): passed a duplicate through as COMPLETE with no check — in production this is double payment risk. Score 0.
E17 (policy lapsed mid-claim): read policy status at FNOL and missed the post-FNOL premium bounce; verdict "cover active" is wrong on the fact that decides the claim. Score 0, flag SILENT-WRONG.
X21 (handwritten regional FIR): failed to recognise the document as an FIR and declared it missing — triggers a wrongful customer chase for a document already supplied. Score 0.
X23 (handwritten estimate): extracted 84,300 against a true 34,800 with no confidence flag — a 49,500 overstatement entering the payment pathway silently. Score 0, flag SILENT-WRONG (the single most production-dangerous output in the lab).
X25 (empty upload) also fails (0-byte counted as present) — accept it as a fourth-or-fifth exception failure; the brief's four are E14/E17/X21/X23, and X25 is the bonus catch.

The hallucination — E15. Candidate A justifies cover continuity "per Policy Clause 9.4." The policy wording index runs Clauses 1–8; the real grace provision is 5.3. The conclusion may even be right; the citation is fabricated, and a system that fabricates policy citations cannot be allowed near coverage decisions or customer communications without a citation-verification layer. This must appear in the memo verbatim with the HALLUCINATION flag — catching it requires doing step 4, which is the point.

Expected weighted outcome (subset). Candidate A: routine near-perfect (12/12-style extractions), roughly 12/12 raw on the four graded routine cases; edge+exception approximately 0–2 of 16 weighted-relevant points — a demo-delta so large it reverses the ranking. Candidate B: drops odd routine points (missed GSTIN, date quirk — score 1s), but scores 1–2 across every edge and exception via safe abstains and flagged extraction; weighted total clearly ahead. Correct recommendation: Candidate B, conditional on remediating routine-extraction misses, with A re-evaluable only against eval-set-v1.x after demonstrating duplicate detection, event-timeline handling, handwriting confidence flagging, and cited-clause verification.

Scoring calibration. If your rubric gave B's HOLDs zeros, revisit step 3 — the rubric must reward safe abstention or your eval selects for confident wrongness. If A ranked higher on your totals, check whether you weighted by volume (routine ×heaviest) — that is the demo's logic, and it is the exact bias the harness exists to correct.

Grading yourself. Hallucination cited + four failures named + B recommended with conditions: measured-evidence decision-making achieved. Failures caught but clause 9.4 accepted: assertion-verification is a separate pass — fluent reasoning is where fabrications hide. Recommended A on "extraction accuracy": reread X23; accuracy that will not flag its own uncertainty is not accuracy, it is liability with good grammar.

Key Takeaway:

An eval harness converts vendor selection from a demo-judging contest into a measurement: your cases, exception-weighted, scored against pre-written expected behaviours — with hallucination and silent-wrongness as flags that override any average. Version the set; it outlives the decision.

© Anvi Innovate · BXUP — Be Exponentially Upskilled · BXUP is an independent learning platform, not affiliated with or endorsed by AWS, Microsoft, Google, NVIDIA, IBM, Salesforce, Databricks, Oracle or Anthropic. All trademarks belong to their respective owners. · bxup.in