Write the Eval Set for Your AI Feature

Workflow area: Evals & AI Quality · Tier: Advanced · Hands-on time: ~45 minutes to learn, ~half a day for the first set

Most teams' first "eval set" is five examples in a Slack thread that everyone eyeballs after prompt changes. The real thing is different in kind: dozens of cases spanning the input distribution and every known failure mode, each with an explicit pass criterion, runnable on demand, so that "did the new prompt make things better?" has a numerical answer instead of a vibe. Engineering can build the harness; the content — which cases matter, what counts as passing, how severe each failure is — is product judgment. This playbook is how a PM authors that content properly.

When to use — and when not to

• Use it after pb-36's error analysisthe taxonomy tells you which failure modes need coverage; an eval set written before error analysis tests imagined failures.
• Use it when quality bars enter a PRD (pb-34)every bar needs the instrument that measures it, and this is the instrument.
• Use it before any bake-off (pb-33) or launch gate (pb-40)both consume the eval set; neither works without it.
• Skip the heavyweight version for a pre-launch feature with no tracesstart with a 15-case "starter set" from the spike (pb-35) and dogfooding, and grow it from production reality after launch.
• Don't confuse the eval set with monitoringevals are the controlled instrument you run on demand; production metrics (pb-41) are the field readings. You need both; this playbook is the first.

The recipe

1. 1.

   Draft the case skeleton from taxonomy + PRD. Coverage comes from two directions: every failure mode in pb-36's taxonomy gets 3–5 cases (real traces, scrubbed, make the best cases), and every promise in the PRD's requirements annex (pb-34) gets cases that test it. Add the distribution backbone: 15–20 representative everyday cases at production frequency, because a set of pure edge cases makes every score look apocalyptic.

2. 2.

   Write per-case pass criteria — the hard, valuable part. Each case states what passing means, mechanically checkable where possible:

   Copy

   # eval case — the shape that matters (promptfoo-style)
   - description: "stale-policy trap: 2023 vs 2024 refund policy"
     vars:
       user_query: "What's your refund window for annual plans?"
     assert:
       - type: contains        # current policy fact
         value: "60 days"
       - type: not-contains    # the stale figure = S1 fail
         value: "30 days"
       - type: llm-rubric
         value: "Answer cites the refund policy document and does
                 not invent conditions absent from it"
     metadata:
       severity: S1
       failure_mode: "stale-doc grounding"
       source: "trace #014, 2026-07 error analysis"
   

   Use deterministic assertions (contains / not-contains / regex / JSON-validity) wherever truth is crisp, and reserve LLM-rubric grading for genuinely subjective dimensions — every rubric you write here is a judge that pb-38 must later align.

3. 3.

   Generate synthetic edge cases with AI — then curate like an editor. For each failure mode, have Claude propose variants: "Generate 10 user queries likely to trigger stale-knowledge answers for this product; vary phrasing, indirection and language." Expect to keep half — delete duplicates, implausible cases, and anything whose pass criterion you can't defend. Synthetic cases are scaffolding; real traces are load-bearing. Keep the ratio visible in metadata.

4. 4.

   Tag severity and write the rubrics for subjective dimensions. Severity tags (S1 user-harming / S2 degrading / S3 cosmetic) are what make scores decision-ready — pb-33's trap of a model winning on average while failing S1 cases is only catchable if severity lives in the set. For subjective dimensions (tone, helpfulness), write a 3–4 level rubric with an example answer at each level; a rubric without anchored examples is a mood.

5. 5.

   Run it, read the failures, fix the set first. First run (promptfoo CLI, Braintrust, or the sheet-based manual path) always finds set bugs: ambiguous criteria, cases where the "expected" answer is itself wrong, assertions that pass on garbage. Read every failure and ask "is this a model failure or a case failure?" — budget a full revision pass before any score from the set is quoted.

6. 6.

   Version it like code, with ownership. The set lives in git (or Braintrust's versioned datasets): changes go through review, each version is dated, and scores always cite the set version they were measured on. You own the cases and the bars; engineering owns the harness and the CI wiring (pb-40). Schedule a monthly additions pass — new production failures (pb-36's next round) become new cases, which is how the set stays honest as the world drifts.

Make it yours

• The no-code path: the whole discipline works in a spreadsheet — case, input, pass criteria, severity, last-run result — graded by hand against pasted model outputs; slower per run, identical in rigor, and an easy later migration into promptfoo.
• Braintrust/LangSmith-native: seed the dataset directly from traces you labeled in pb-36's annotation pass, so every real case carries its trace lineage automatically.
• The golden subset: mark 10–15 cases as the "golden set" — the non-negotiables that gate merges in pb-51's governance contract — separate from the broader set used for exploration and bake-offs.

Quality check

• Every taxonomy failure mode has casescoverage is checkable by grouping cases on the failure_mode tag; a mode with zero cases is an open hole.
• The distribution backbone existseveryday representative cases outnumber any single edge-case cluster.
• Pass criteria are mechanical where truth is crispLLM-rubric grading is reserved for genuinely subjective dimensions, and each rubric has anchored examples.
• Severity tags are present and honestS1 means user-harming, not "annoys the PM."
• Real vs synthetic provenance is recordedand real traces anchor every high-severity mode.
• The set is versioned, owned and citedevery quoted score names its set version; changes are reviewed, and a monthly additions pass is on the calendar.