Evals: Measuring AI Quality Honestly

Ask an engineer in 2023 whether their AI feature worked and you'd get a demo. Ask the same question in 2026 and a good engineer opens an eval dashboard: here are the 60 cases we test against, here's the pass rate on the last five prompt versions, here's the failure mode we're currently fixing, and here's the rollback plan if the next model upgrade regresses. That shift — from "it seems good" to "here is the evidence" — is the single most consistent differentiator employers named in our research across all fourteen role families. Product managers are asked in interviews to "walk through an eval set, failure modes, and rollback strategy." ML engineers are expected to ship eval harnesses before they ship models. QA roles now list "LLM-output evaluation" as a distinct discipline.

This module teaches you the craft: what an eval actually is, the three ways to grade one, why error analysis comes before metric design, and how to build your first eval set for a task from your own job — with a complete worked example you can copy.

What You'll Learn

• Why evals became the differentiatorwhat employers cite, and what interviewers actually probe for
• The anatomy of an evaltask set, grading criteria, pass bar, and how the three parts interact
• Three grading methodsprogrammatic checks, human rubric review, and LLM-as-judge, plus the calibration caveats that make the third one honest
• Error analysis firstwhy you read real failures before you design any metric
• A complete 15-item eval setemail summarization, shown in full, ready to adapt to your own recurring task
• Evals in daily workflowregression before every prompt or model change, and the CI mindset
• Honest metricswhy a 90% score on a bad eval set is worse than no score at all
• Free tooling pathsspreadsheet first, then promptfoo or DeepEval when you outgrow it

Why Evals Became the Differentiator

Between 2024 and 2026, most companies discovered the same thing the hard way: getting an LLM feature to sometimes work takes a weekend; knowing how often it works — and keeping it working through prompt edits, model upgrades, and new user behavior — takes engineering discipline that most teams didn't have. The teams that shipped reliable AI products all converged on the same practice: systematic evaluation.

Employers noticed. Our research across live 2025–2026 job postings found evals to be the most-cited skill differentiator across role families, and it shows up far beyond ML roles:

• Product managersinterview bars now include walking through an eval set, its failure modes, and a rollback strategy — treating eval design as a product skill, not a data science skill.
• Data science / ML engineeringpostings ask for "eval harnesses measuring accuracy, faithfulness, hallucination, and latency," with named tools like Ragas and Arize Phoenix.
• QA and testingLLM-output evaluation is listed as its own category, distinct from traditional test automation.
• Software engineeringcompanies with AI mandates (Coinbase's is public) require human review of AI-generated code — and review at scale needs criteria, which is eval thinking applied inline.

The reason evals cut across roles is simple: every AI system decision — which model, which prompt, whether to add retrieval, whether the new version is safe to ship — is unanswerable without measurement. The person in the room who can measure gets to make the call.

Vibes vs Evals

"Vibe checking" is what everyone does first: run a few prompts, eyeball the outputs, feel good, ship. It is not worthless — a vibe check catches catastrophic failures fast. But it fails as a decision-making tool for three reasons:

1. 1Tiny, biased samples. You test the inputs you think of, which are the easy, typical ones. Real users supply the weird ones.
2. 2No memory. When you tweak the prompt to fix case A, you don't re-test cases B through Z. LLM behavior is coupled: fixing one case silently breaks others, and vibes never catch the regression.
3. 3No shared standard. Your "looks good" and your teammate's "looks good" differ. Decisions made on vibes are decisions made on whoever spoke last.

An eval fixes all three: a fixed set of cases (including the weird ones), graded the same way every time, producing a number two people can argue about productively.

Anatomy of an Eval

Every eval, from a 10-row spreadsheet to a production harness, has three parts.

1. 1The task set. A collection of inputs your system should handle, each paired with what a good output looks like. The set must be representative (drawn from real usage, not imagination), diverse (typical cases, edge cases, and known failure cases), and stable (versioned, so scores are comparable over time). For a personal workflow eval, 10–20 cases is a genuine start; production features typically grow to 50–200+.
2. 2The grading criteria. For each case, an explicit statement of what "pass" means. This is where most first evals go wrong — "output is good" is not a criterion. "Output mentions the deadline if the input contains one" is. Good criteria are checkable: two reasonable people applying them to the same output reach the same verdict.
3. 3The pass bar. The threshold that converts a score into a decision: "≥ 90% overall, and 100% on the six safety-critical cases, or we don't ship the change." Without a pre-committed bar, teams rationalize whatever score they get. The bar belongs to the decision, not the metric — different actions (ship to everyone, ship behind a flag, roll back) can have different bars.

A useful mental model: the task set defines what you care about, the criteria define what good means, and the pass bar defines what you'll do about it.

Grading Methods

How you check each output determines how much you can trust the score. There are three methods, in increasing order of flexibility and decreasing order of reliability.

Programmatic checks

Code decides pass/fail: exact string match, regex, JSON schema validation, "contains X / does not contain Y", length bounds, or executing generated code against unit tests. Deterministic, free, instant, and perfectly consistent. Use them wherever the task has a checkable structure — classification labels, extracted fields, format compliance, code correctness. The discipline is to decompose fuzzy tasks into checkable parts: you can't programmatically grade "is this summary good?", but you can grade "is it under 100 words?", "does it contain the invoice number from the input?", and "does it avoid the phrase 'as an AI'?"

Rubric-based human review

A human grades each output against a written rubric — a checklist of criteria, each pass/fail or scored 1–5. This is the gold standard for quality judgments (tone, faithfulness, usefulness) and the only trustworthy method when stakes are high. Its costs are time and drift: humans get tired, and two reviewers disagree unless the rubric is sharp. Mitigate with small batches, concrete rubric language ("cites at least one number from the source" beats "is specific"), and periodic double-grading of a sample to measure reviewer agreement.

LLM-as-judge

A second model call grades the first model's output against your rubric. This scales human-like judgment to hundreds of cases per minute at token cost — which is why it has become the workhorse of production eval suites. But a judge is itself an unvalidated AI system until you calibrate it, and uncalibrated judges fail in known ways:

• Leniency and position bias. Judges tend to grade generously, and in pairwise comparisons can favor the first-listed or longer answer regardless of quality.
• Self-preference. Models tend to rate outputs from their own family higher.
• Criteria drift. A vague rubric prompt lets the judge substitute its own notion of "good" for yours.

The fix is a calibration loop: have a human grade 20–30 outputs first, run the judge on the same outputs, and measure agreement. If judge and human agree on ~90%+ of verdicts, the judge is usable for that task; if not, tighten the judge prompt (binary questions beat 1–10 scales; one criterion per judge call beats an omnibus "rate this") and re-check. Then keep spot-checking a sample of judge verdicts forever — agreement measured once is not agreement guaranteed.

Start With Error Analysis, Not Metrics

The most common beginner mistake is picking metrics first — grabbing a generic "hallucination score" or "coherence score" off a tool's shelf and dashboarding it. The practitioners who teach this best (Hamel Husain and Shreya Shankar, whose error-analysis method has become the de facto industry curriculum) insist on the opposite order: look at your data first.

The method is unglamorous and works:

1. 1Collect real traces. Gather 30–100 actual input/output pairs from your system — real usage, not synthetic tests.
2. 2Read them and annotate. For each one, write a short free-text note on what, if anything, went wrong. Don't categorize yet; just describe. ("Ignored the second question." "Invented a meeting date." "Tone too casual for a client.")
3. 3Cluster the notes into failure modes. Patterns emerge fast: maybe 40% of failures are missed instructions, 25% are fabricated details, 20% are formatting. This taxonomy — your system's actual failure modes, not a generic list — is the most valuable artifact in the whole process.
4. 4Build the eval from the taxonomy. Each failure mode becomes a slice of your task set (with the real failing examples as test cases) and a grading criterion targeting it.

This ordering guarantees your eval measures problems you actually have. It also gives you the interview-ready answer to "what are your failure modes?" — because you found them empirically instead of guessing.

Worked Example: A 15-Item Eval for Email Summarization

Here is a complete, real-shaped eval for a task nearly every professional has: summarize an email thread into a brief update with action items. The system under test takes a raw thread and must produce a ≤120-word summary plus a bulleted action list. Grading mixes programmatic checks (P) and judge/human rubric checks (R). Every case states its input character, what good looks like, and the pass criteria.

Notes on the design, because the shape is the lesson:

• •
  Cases 1–5 are the typical core; 6–12 are edges found by imagining and by reading real threads; 13–14 come straight from error analysis; 15 is a safety case with a hard bar.
• •
  The pass bar for this eval: ≥13/15 overall, with cases 5, 13, and 15 mandatory (numbers, fabrication, injection). A version scoring 14/15 that fails case 13 does not ship.
• •
  Roughly half the cases are programmatically checkable. That half runs free and instantly on every prompt tweak; the rubric half runs when you're deciding whether to adopt a change.

Evals in Your Workflow

An eval you run once is a report; an eval you run on every change is a safety net. The professional habit looks like this:

• Before any prompt change: run the eval on the current version to confirm the baseline, make the change, run again, compare per-case — not just the aggregate. A flat overall score can hide two fixes and two new breaks.
• Before any model change: model upgrades and "silent" provider updates are the most common cause of regressions. Never switch models on vibes; run the full set.
• The CI idea: teams wire the programmatic subset into continuous integration — the eval runs automatically on every commit that touches a prompt, and a score below the bar blocks the merge, exactly like failing unit tests. Prompt edits are deploys, and they regress like code.
• Version everything together: prompt, model ID, eval set, and scores. "v14: model X, prompt hash abc123, 14/15" is a sentence that ends arguments.

Honest Metrics: Why 90% Can Be Meaningless

A score is only as meaningful as the set it's computed on. The classic failure: a team reports "92% accuracy," leadership relaxes, and the feature fails in production week one — because the eval set was 50 easy cases the prompt was tuned against. Three honesty rules:

1. 1Hard cases or it doesn't count. If your set contains no cases your system currently fails, your eval measures nothing. A healthy eval sits well below 100% — headroom is information.
2. 2Don't tune on the test. If you iterate your prompt against the same cases you report scores on, you are overfitting to the eval. Keep a holdout slice you look at only when making adoption decisions, and refresh cases from production regularly.
3. 3Slice the score. Report per-category numbers (by failure mode, by input type), not just the aggregate. "95% overall, 60% on multi-owner threads" is actionable; "89% overall" is a shrug.

And the corollary for consuming other people's numbers: when a vendor claims "3x faster QA" or "40% fewer bugs," your first question is now professional instinct — what was the eval set, who graded it, and what was the bar?

Tooling: Spreadsheet First

Do not start with a framework. Start with a spreadsheet — columns: input, expected/criteria, output_v1, pass_v1, notes, then output_v2, pass_v2... A 15-case spreadsheet eval, graded by you against written criteria, is a real eval and will teach you more in an afternoon than a week of tool configuration. It also produces the artifact interviews ask about.

Graduate to tooling when you feel specific pain — re-running by hand, wanting LLM-as-judge at scale, or wiring into CI. Two solid open-source options (both free):

• promptfooconfig-file driven, strong for side-by-side prompt/model comparison, assertions (contains, regex, JSON validity, judge rubrics), and a local web viewer; runs in CI with a single command.
• DeepEvalpytest-style Python framework; write eval cases as tests, use built-in metrics (faithfulness, answer relevancy) or custom judges; natural fit if your team already lives in pytest.

RAG-specific evaluation (faithfulness to retrieved context, retrieval quality) adds its own metric family — Ragas is the common open-source choice there, and Module 5 covered the retrieval concepts these metrics score.

Exercises

1. 1Error analysis on your own usage. Collect 20 real outputs from an AI tool you already use for work (coding assistant suggestions, drafted emails, meeting summaries). Annotate each with a free-text note on what went wrong, if anything. Cluster the notes into 3–5 named failure modes. Deliverable: your failure-mode taxonomy with counts.
2. 2Build a 10-item eval set for a recurring task from your own job. Pick a task you'd actually delegate to AI (summarizing tickets, drafting status updates, writing test descriptions, translating requirements). Using the email-summarization table above as a template, write 10 cases: at least 5 typical, 3 edge cases, and 2 drawn from failures you found in Exercise 1. For each: input, what good looks like, pass criteria, and whether it's programmatically checkable. Then define your pass bar and mandatory cases. Deliverable: the spreadsheet.
3. 3Run and regress. Run your 10 cases against your preferred AI tool. Record per-case pass/fail. Change the prompt to fix your worst failure, re-run all ten, and write three sentences on what improved, what regressed, and whether you'd adopt the change under your pass bar.
4. 4Calibrate a judge (stretch). Write a judge prompt that grades one criterion from your eval (binary pass/fail with a one-sentence reason). Run it on 10 outputs you have already graded by hand, and compute agreement. If below 9/10, revise the judge prompt once and re-measure.

Further Reading

• •
  Anthropic — Define your success criteria and build evals — official guidance on criteria design and eval development
• •
  OpenAI — Evaluating model outputs — official evals guide covering graders and regression workflow
• •
  promptfoo documentation — open-source eval runner: assertions, judges, CI integration
• •
  DeepEval documentation — pytest-style LLM evaluation framework
• •
  Ragas documentation — RAG-specific evaluation metrics (faithfulness, context precision/recall)