Write a 10-Item Eval Set for a Recurring Task

Tier: Core · Hands-on time: ~50 minutes

It takes an hour to build and pays out every time a model updates, a prompt changes, or someone asks "why do you trust this?"

When to use

• •
  You run the same AI task weekly or more (ticket triage, summary emails, extraction, code-review comments) and want to know — not feel — how reliable it is.
• •
  Before rolling a personal workflow out to your team.
• •
  When choosing between models or prompts for one task (pb-10 needs this as input).

Prerequisites

• •
  One specific recurring task with a prompt you already use (from your pb-05 library).
• •
  Ten real historical inputs for that task — actual past tickets/transcripts/emails, not invented ones.
• •
  Any spreadsheet tool.

The workflow

1. 1.

   Create the spreadsheet. Columns:

   Copy

   id | input (or link) | expected_outcome | must_include | must_not_include | score | notes
   

   One row per test case. expected_outcome is the reference answer you would accept; must_include / must_not_include are the mechanical checks that make grading fast and honest.

2. 2.

   Choose 10 inputs that span reality. The distribution matters more than the count:

   • •
     5 typical cases — the everyday shape of the task.
   • •
     3 hard cases — long, messy, ambiguous, multilingual; whatever historically causes trouble.
   • •
     2 trap cases — inputs where the right behavior is to decline, flag, or output "not found" (an empty transcript; a ticket that's actually spam). Traps measure trustworthiness, not capability.
3. 3.

   Write expected outcomes before running anything. For each row, fill expected_outcome and the include/exclude checks from the historical ground truth (you know what that ticket's real severity was). Writing expectations after seeing model output is the most common way evals lie to their authors.

4. 4.

   Run and grade. Feed each input through your standard prompt (fresh chat per item — no context bleed). Grade each row 0 / 0.5 / 1:

   • •
     1 — meets expected outcome, all must-includes present, no must-not-includes.
   • •
     0.5 — usable after minor human editing.
   • •
     0 — wrong, invented content, or a trap case answered instead of declined. Record one-line failure reasons in notes — these become prompt fixes.
5. 5.

   Read the score like an engineer. Sum the points (max 10):

   • •
     9–10: automate freely; spot-check occasionally.
   • •
     7–8.5: use with the human-review step your workflow already has; fix the noted failures.
   • •
     Below 7: the task, prompt, or model isn't ready — the notes column tells you which. Any trap-case failure caps your trust regardless of total: a 9/10 that invents answers for empty inputs is not a 9.
6. 6.

   Re-run on change. Date-stamp each run in a new score column. Re-run the same 10 rows whenever you change the prompt, the model updates, or quarterly — regressions from silent model updates are real and this is the only way you'll see them.

Tool variants

• Spreadsheet + manual runs (this playbook): zero infrastructure, teaches the discipline; right-sized for personal and small-team workflows.
• Promptfoo (open-source CLI): your same cases as a YAML file, run automatically across multiple models/prompts with assertion checks — the natural upgrade once re-running by hand gets old.
• Hosted eval dashboards (various AI platforms): team visibility, versioned datasets, LLM-judge scoring at scale; worth it when several people share one workflow.

Verification

• •
  All 10 expected outcomes were written before the first model run (the timestamps in your own edit history are the honesty check).
• •
  The set contains ≥3 genuinely hard cases and 2 traps — if every case scored 1, your set is too easy to be informative; add harder rows.
• •
  Failure notes are specific enough to act on ("merged two speakers' action items," not "bad output").
• •
  A second run on an unchanged setup scores within ±1 point — larger swings mean your grading criteria are too vague; tighten must-include lists.

Pitfalls

• Cherry-picked inputs. Ten clean typical cases produce a flattering, useless score. The 5/3/2 mix is the point.
• Grading on eloquence. Fluent wrong answers earn sympathy points. Grade against must_include mechanically first, prose impressions second.
• Moving the goalposts. Editing expected_outcome to match model output converts your eval into a rubber stamp. Expectations change only when the task changes.
• One-and-done. An eval run once is a snapshot; the compounding value is in the re-runs across model and prompt versions.
• Over-trusting an LLM judge. Judges share the models' blind spots and grade traps poorly. Human eyes on traps, always.

Restricted-environment variant

• •
  Build the eval from anonymized historical inputs: apply pb-03-style pseudonymization (CLIENT_A, PERSON_1) to real cases — structure and difficulty survive; identities don't. Never put raw client data in a spreadsheet that syncs to personal cloud storage.
• •
  Store the eval sheet in the project's approved repository, not personal drive — it contains a map of your client's data shapes.
• •
  Local path: run the same 10 cases against your Ollama model (pb-07) — the identical sheet then doubles as your local-vs-hosted comparison for pb-10 routing decisions.

Success criteria

• •
  A 10-row eval sheet exists for one recurring task: 5 typical / 3 hard / 2 traps, expectations written first.
• •
  At least two dated score columns (e.g., initial + after one prompt fix) showing you actually re-ran it.
• •
  One concrete prompt improvement traced directly to a failure note, with the score delta visible.
• •
  You can state your task's reliability as a number with evidence, not an anecdote — and you know exactly what it fails at.