Evaluate AI Vendors with an Eval Harness, Not Their Demos

Workflow area: AI Opportunity & Feasibility · Tier: Advanced · Hands-on time: ~50 minutes

Vendor demos are optimized artifacts. The inputs are curated, the edge cases are absent, and the one hallucination per hundred runs never appears on the projector. A BA who selects an AI tool from demos is signing requirements against evidence that was engineered to flatter. The fix is the same discipline you apply to any vendor claim — verify it yourself — industrialized: one eval set, every candidate, identical conditions, scores in a table before anyone falls in love.

When to use — and when not to

• Use it whenever an AI tool or model choice is on the tablevendor platforms, competing foundation models, or a vendor solution versus an internal build. Anything scored in a demo deserves scoring in a harness.
• Use it when the use case has exception cases that matterdemos live on the happy path; your eval set is where the messy 20% of real cases finally gets a vote.
• Use it before commercial negotiationmeasured quality gaps are leverage; "candidate B fails 4 of 6 exception scenarios" reprices a contract faster than any discount request.
• Skip it when the decision is not really about output qualityif integration constraints or data-residency rules already eliminate all but one candidate, document that and save the harness effort.
• Skip it if you cannot assemble real casesan eval set invented from imagination tests nothing; go run the elicitation and sampling work first (see the golden-dataset playbook, pb-48).
• Skip the DIY harness for regulated, high-stakes selectionsprocurement of a credit-decisioning model needs a formal validation team; your harness informs, it does not replace.

The recipe

1. 1.

   Build the scenario set from real cases. Pull 20–30 genuine cases from the process the tool will serve — tickets, claims, emails, documents — deliberately including the distribution that demos hide: roughly 60% routine, 25% edge cases, 15% known trouble (ambiguous inputs, missing fields, cases where the right answer is "escalate to a human"). Anonymize every case before it leaves your environment. For each case, record the input and the expected behavior, reviewed by the SME who owns the process.

2. 2.

   Write the rubric before you see any output. Define 3–5 scored dimensions and what a pass looks like on each. Locking the rubric first prevents the classic failure: unconsciously bending criteria toward the candidate you liked. A workable starting rubric:

   Copy

   Per case, score each candidate:
   - CORRECTNESS (0-2): matches expected outcome; 0 if any factual
     invention, regardless of how good the rest is.
   - COMPLETENESS (0-2): all required elements present.
   - SAFE FAILURE (0-2): on ambiguous/missing-data cases, does it
     say "cannot determine" or escalate — or does it guess?
   - FORMAT (0-1): output usable downstream without rework.
   Record separately: latency (s), cost per case (from provider
   pricing), and any hallucination as an automatic case FAIL.
   

3. 3.

   Run every candidate on identical inputs. For API-accessible models, a harness tool like promptfoo runs the whole set per candidate from one config file; for closed vendor platforms, run the cases through their trial workspace by hand and export the outputs. The non-negotiable: same cases, same phrasing, same day. If a vendor will not let you run your own cases in a trial, write that refusal into the evaluation as a finding.

4. 4.

   Score blind where possible. Strip candidate names from outputs and have two scorers (you plus an SME) apply the rubric independently, then reconcile disagreements — a disagreement usually means the rubric line is ambiguous, so fix the rubric and note it. A scoring summary worth producing:

   Copy

   Candidate A: 78% weighted score · 0 hallucinations ·
     3/5 safe-failure cases passed · $0.04/case · p95 6.2s
   Candidate B: 84% happy-path score BUT 2/5 safe-failure passed,
     1 hallucinated policy clause (case #17) · $0.09/case
   Finding: B wins demos; A wins the process. Recommend A, with
     case #17 attached to the memo.
   

5. 5.

   Write the decision memo from the scores. One page: recommendation, the score table, the two or three cases that decided it (quoted), the costs at projected volume, and what was NOT tested — this harness measures output quality, not integration effort, admin tooling, or vendor viability, so name those as separate evaluation lines. Version the eval set and archive it: it becomes your regression gate when the chosen vendor ships model updates.

Make it yours

• promptfoo for the free, repeatable path: one YAML file lists prompts, providers, and assertions; rerunning the full comparison after a vendor update is a single command. Ideal when candidates are API-accessible models.
• Spreadsheet harness for closed platforms: one row per case, one column block per candidate, rubric scores as columns — less automated, equally decisive, and finance-legible.
• Provider workbenches for quick head-to-heads: Anthropic and OpenAI workbenches let you run the same case against model tiers in minutes when the "vendors" are just model choices.
• At program scale: make the harness a procurement standard — no AI line item passes governance without an eval-set score sheet attached, and the eval sets accumulate into an organizational asset.

Quality check

• Cases are real, sampled, and SME-approvednot invented; the exception slice is present in realistic proportion.
• Rubric was frozen before scoringany mid-scoring rubric change is logged and applied to all candidates retroactively.
• Conditions were identicalsame inputs, same day, no vendor-tuned prompts for one candidate but not others.
• Hallucinations are dealbreakers, not deductionsone invented policy clause is cited by case number in the memo, never averaged away.
• Costs computed at projected volumeper-case cost times annual case count, from list prices, arithmetic shown.
• Limits of the harness are statedthe memo says what was not evaluated, so the decision inherits your honesty.