Vendor Bake-Off: Decide with Your Eval Set, Not Demos

Workflow area: AI Product Economics · Tier: Advanced · Hands-on time: ~45 minutes

The way AI vendor decisions actually get made is embarrassing: a stakeholder saw a demo, a leaderboard screenshot circulated, a discount was offered, and the architecture was chosen. Public benchmarks measure performance on public benchmarks — your support copilot's ability to handle your refund-policy edge cases, in your tone, at your latency budget is a different question that only your own eval set answers. The bake-off is the PM-owned decision instrument: same inputs, same grading, every candidate, numbers in a table.

When to use — and when not to

• Use it for any model or vendor selection with real money or roadmap attachedchoosing the model behind a feature, renewing an AI vendor contract, or evaluating a "switch to us, we're cheaper" pitch.
• Use it at model-refresh timewhen a provider ships a new version or deprecates yours, rerun the same set; the bake-off is reusable infrastructure, not a one-off.
• Use it to arbitrate build-vs-buythe vendor's product and your prompt-over-API prototype can run the same cases and be compared on the same table.
• Skip it before you have an eval set worth the namea bake-off over five cherry-picked examples is a demo with extra steps; if you haven't done pb-36/pb-37's work, do a lightweight version of it first.
• Skip pure leaderboard reasoningranking on public benchmarks is a screening signal for which 2–4 candidates to include, never the decision itself.

The recipe

1. 1.

   Fix the candidates and the constraints before running anything. Two to four candidates (more dilutes attention), and the constraint sheet: latency budget (e.g. p95 under 3s), cost ceiling per interaction, data-residency and DPA requirements, rate limits. A candidate that fails a hard constraint is out regardless of quality — write that down first so the bake-off can't be relitigated by demo enthusiasm.

2. 2.

   Assemble the eval set from real distribution plus hard cases. 30–60 cases minimum: representative inputs at their production frequency, plus your failure-taxonomy hits (pb-36) and edge cases with per-case pass criteria (pb-37). Every case carries an expected outcome or a rubric. Resist the urge to only test hard cases — you're buying performance on the distribution, not on the museum of horrors.

3. 3.

   Run the identical set across candidates. With promptfoo, one config runs all candidates over all cases; OpenRouter gives you many models behind one API key for the comparison:

   Copy

   # promptfooconfig.yaml — the shape of a bake-off
   prompts:
     - file://prompts/support_copilot.txt
   providers:
     - openrouter:anthropic/claude-sonnet-4.5
     - openrouter:openai/gpt-5.2
     - openrouter:google/gemini-3-pro
   tests:
     - file://cases/eval_set.yaml   # same cases, every provider
   

   Hold the prompt constant for the core run; if a candidate needs prompt adaptation to be viable, run that as a labeled second pass — "with tuned prompt" is a different (and fair) claim than "out of the box."

4. 4.

   Grade with the same standard, and audit the grader. Deterministic assertions where possible; a rubric-based LLM judge for subjective dimensions — the same judge for every candidate, and never a judge from the same family as a candidate without checking its bias. Spot-check 10–15 judge grades by hand before trusting the totals (pb-38 is the full discipline).

5. 5.

   Build the decision table. One row per candidate:

   Copy

   CANDIDATE       QUALITY   HIGH-SEV FAILS   p95 LAT   COST/1K INTER.
   Model A          91%          0             2.1s        $4.10
   Model B          93%          3             1.4s        $1.60
   Model C          84%          1             3.8s        $0.55
   

   The classic trap sits in that table: Model B wins headline accuracy while failing three high-severity cases (say, confident policy hallucinations). Severity-weight the failures — a candidate that's 2% better on average but fails catastrophically on the cases that trigger refunds or press coverage is not better.

6. 6.

   Write the memo from the table. One page: recommendation, the table, the severity analysis, constraint compliance, cost at projected volume (pb-46 math), and — mandatory — the revisit triggers: "re-run this bake-off when: our volume 10×es, candidate prices change >30%, a candidate ships a major version, or our failure taxonomy gains a new high-severity mode." Attach the eval config so the rerun is one command, not a project.

Make it yours

• No-code path: run candidates in their playgrounds over the same 30 cases pasted from a sheet, grade in the sheet with your rubric — slower, but the logic is identical and the decision table comes out the same.
• Vendor-product bake-off: for buying a product rather than a model, submit the same cases through each vendor's trial and grade the outputs; ask each vendor to run your cases live in the sales call — watching how they react to your eval set is itself diagnostic.
• Continuous bake-off: keep the config in git and re-run monthly against new model releases; the quality-vs-cost frontier this produces over time is pb-45's input.

Quality check

• Identical cases, identical grading, every candidateany asymmetry (tuned prompt, retries, cherry-picked run) is labeled or the comparison is void.
• The set covers the distribution, not just the horrorsproduction-representative cases dominate; hard cases are present and severity-tagged.
• Judge grades were spot-checked by hand before totals were believed, and judge-family bias was considered.
• Failures are severity-weighted in the decisionheadline accuracy never overrode high-severity failure counts without an explicit argument.
• Cost is computed at projected volume with real token countsnot the vendor's per-token price in isolation.
• The memo names its revisit triggers and the config is versioned for the rerun.