Evals Hands-On III: Eval-Driven Development & Production

The 2026 production consensus is blunt: eval-first development has replaced prompt-first development. The teams shipping reliable AI features do not write a prompt, demo it, and then wonder about testing — they write the eval cases from the spec, watch them fail, and iterate the prompt until the cases pass, exactly the way test-driven development treats code. Everything in this module operationalizes that inversion: the eval stops being a report you generate and becomes the gate every change passes through.

What You'll Learn

• Eval-first developmentcases before prompt, and why the order matters
• Pre-merge eval runsa complete GitHub Actions workflow you can commit today
• Regression gatespass-rate floors, mandatory slices, and the baseline file
• Versioning golden setsappend-only cases, set versions, and the holdout rule in CI
• Offline evals vs online signalswhat each can and cannot tell you
• Drift detectionscheduled runs, provider updates, input drift, and judge drift
• Cost and samplingthe three-tier run structure and the arithmetic behind it
• Re-baseliningthe legitimate reasons, and the illegitimate one
• A case studya team catching a silent regression the aggregate score hid
• The eval maturity checklistten items to audit any team against

Eval-First: Cases Before Prompt

The inversion is simple to state. Given a new feature — say, Meridian (Module 9) wants the support bot to also handle plan downgrades — the eval-first order is:

1. 1Write the cases first. Read the downgrade policy, write 6–10 test cases including the edges (downgrade mid-cycle, downgrade with unpaid invoice, downgrade request that is really a cancellation), and mark the mandatory ones.
2. 2Run them and watch them fail. The current prompt knows nothing about downgrades; the red run proves your cases can detect absence. A case that passes before the feature exists is testing nothing.
3. 3Iterate the prompt until greenfull suite, not just the new slice, because the downgrade paragraph you added can regress the refund cases (Module 9's break-and-catch exercise was a rehearsal for exactly this).
4. 4Commit prompt and cases together. One PR, one review, one revert unit. A prompt change without eval changes is now the suspicious diff, not the normal one.

The payoff is the same as TDD's: the spec disagreement surfaces before the build ("wait — do downgrades take effect immediately or at cycle end?") instead of in production, and every future change inherits the safety net.

Pre-Merge: The CI Gate

Here is a complete GitHub Actions workflow for the Module 9 suite. It runs on every pull request that touches prompts or eval config, executes the full suite, and fails the PR on either of two conditions: any mandatory case fails, or the overall pass rate drops below the committed baseline.

# .github/workflows/prompt-evals.yml
name: prompt-evals

on:
  pull_request:
    paths:
      - "prompts/**"
      - "promptfooconfig.yaml"
      - "evals/**"

jobs:
  eval:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: 22

      - name: Run eval suite
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: |
          npx promptfoo@latest eval -c promptfooconfig.yaml \
            -o results.json --no-progress-bar || true
          # '|| true': we enforce our own gate below instead of
          # promptfoo's default any-failure exit code.

      - name: Gate 1 - mandatory cases must be 100%
        run: |
          MANDATORY_FAILS=$(jq '[.results.results[]
            | select((.testCase.description // "") | startswith("[mandatory]"))
            | select(.success == false)] | length' results.json)
          echo "Mandatory failures: $MANDATORY_FAILS"
          if [ "$MANDATORY_FAILS" -gt 0 ]; then
            echo "::error::$MANDATORY_FAILS mandatory case(s) failing - blocking merge"
            exit 1
          fi

      - name: Gate 2 - pass rate must not drop below baseline
        run: |
          PASS=$(jq '.results.stats.successes' results.json)
          FAIL=$(jq '.results.stats.failures' results.json)
          RATE=$(jq -n "$PASS / ($PASS + $FAIL)")
          BASELINE=$(jq '.pass_rate' evals/baseline.json)
          echo "Pass rate: $RATE  (baseline: $BASELINE)"
          BELOW=$(jq -n "$RATE < $BASELINE")
          if [ "$BELOW" = "true" ]; then
            echo "::error::Pass rate $RATE below baseline $BASELINE - blocking merge"
            exit 1
          fi

      - name: Upload full results
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: eval-results
          path: results.json

And the baseline file it reads, committed at evals/baseline.json:

{
  "pass_rate": 0.90,
  "eval_set_version": "v3",
  "prompt_commit": "9f31c2a",
  "model": "claude-sonnet-4-6",
  "recorded": "2026-08-05",
  "note": "Baseline after downgrade-policy launch. Mandatory slices gated separately at 100%."
}

Design decisions worth stealing even if your stack differs:

• The gate is yours, not the tool's. promptfoo exits non-zero on any assertion failure, but "any failure blocks" is usually the wrong policy for a healthy suite that deliberately sits below 100% (Module 6: headroom is information). The two-gate structure — mandatory slices absolute, aggregate against a floor — encodes the actual ship policy.
• The baseline is a reviewed file in the repo. Raising or lowering it is a visible diff in a PR, with a name attached — never a silent side effect of a green run.
• Results upload on every run, pass or fail, so reviewers read per-case diffs instead of a binary verdict.
• Secrets stay in CI secrets, and the job has a timeout — a hung provider call should fail the run, not burn the runner for six hours.

Versioning Golden Sets

The eval set changes over time — production failures become new cases, taxonomies grow slices, ambiguous cases get rewritten. Version it with three rules:

• Append-only caseswhen a case turns out to be wrong (you shipped Module 9's planted bugs for real), fix it in place only if it was never the basis of a reported score; otherwise add the corrected case and retire the old one explicitly. Silently editing a case rewrites history: every past score computed on it changes meaning.
• Named set versionstag the set (v3) whenever cases are added or retired, record the version in the baseline file, and never compare pass rates across set versions as if they were the same number. "88% on v3" and "91% on v2" are not a regression; they are different exams.
• The holdout stays out of CIModule 6's don't-tune-on-the-test rule has a CI corollary: the holdout slice does not run on every PR, where iteration would overfit to it. It runs at adoption decisions and scheduled checkpoints, by hand or by a separate infrequent workflow.

Offline Evals vs Online Signals

Everything so far is offline: fixed cases, run before ship, answering "did this change make the system better on inputs we chose?" Production adds online signals: user thumbs, edit-before-send rates, escalation rates, deflection, complaint keywords. The two disagree constantly, and both directions of disagreement are informative:

• Offline green, online souryour golden set no longer represents real traffic. Users found inputs you never imagined. Response: sample the traces behind the bad signal, run error analysis (Module 6), and grow the eval set with what you find. This is the standing pipeline: online signals select traces, humans annotate them, the golden set absorbs them.
• Online flat, offline reda regression on cases users rarely hit, or your eval is stricter than users are. Still investigate: "users didn't notice yet" is how silent failures compound, and mandatory slices exist precisely because some failures are unacceptable at any observed frequency.

The judge from Module 10 bridges the two worlds: run it on a sample of production traces daily, and its per-criterion pass rates become an online metric with offline-grade precision — provided you keep spot-checking its calibration on production-shaped data, not just the original 20 labels.

Drift: The Regressions Nobody Committed

CI gates catch changes you made. Production systems also degrade from changes nobody made:

• Provider driftmodel updates, both announced deprecations and silent serving changes, are the top cause of "nothing changed but everything changed." Mitigation: pin model versions where the provider allows it, and treat any model change — including "upgrading to the newer, better model" — as a full-suite adoption decision, never a config tweak.
• Input driftyour users change: new customer segments, new product features generating new ticket types, seasonal shifts. The golden set frozen in March measures March. Mitigation: the online-to-offline pipeline above, plus a monthly check that your set's category mix still roughly matches production's.
• Judge driftthe judge is a model too; if it floats on a provider's latest alias, its verdicts can shift while the system under test stands still, which reads as a fake regression (or worse, masks a real one). Mitigation: pin the judge model, re-calibrate on any judge change, and keep a small set of frozen output/verdict pairs the judge must reproduce.

The instrument for all three is the scheduled full run — nightly or weekly, on an unchanged prompt and set, alerting when results move. On an unchanged system, any movement is the signal: something drifted underneath you. A cron trigger (on: schedule) on the same workflow, running full instead of PR-scoped, is enough infrastructure.

What Eval Runs Cost, and Sampling Your Way Out

Evals cost tokens, and pretending otherwise is how suites get quietly disabled. The arithmetic for the Meridian suite: 14 cases × 2 providers × roughly 900 tokens per call, plus ~10 judge calls — under a cent per full run at Haiku-class prices, single-digit cents with a Sonnet-class grader. At that scale, run everything always. The arithmetic changes at production scale: a 500-case suite with judge grading across 3 candidate prompts is thousands of model calls per run, and judging 100% of production traffic can cost more than serving it.

The standard structure is three tiers:

1. 1Smoke tier — seconds, ~free. The programmatic subset (contains/javascript assertions) plus mandatory cases, on every commit during active iteration.
2. 2Full tier — minutes, cents to dollars. The whole suite including judge-graded cases, on every PR (the workflow above) and nightly.
3. 3Deep tier — scheduled or on-demand. Holdout slice, pairwise tournaments between candidate prompts, multi-provider matrices, production-trace judging. Weekly, or at adoption decisions.

For production monitoring, sample instead of judging everything: a few hundred traces per day, stratified rather than uniform — oversample the slices where failure is expensive (escalations, refund promises, injection-shaped inputs) and undersample the easy bulk. A stratified 2% sample with a calibrated judge beats a 100% sweep with an uncalibrated one, at a fiftieth of the cost.

When to Re-Baseline

The baseline number should move rarely and deliberately. Legitimate triggers:

• An intentional capability changeyou adopted a new model or overhauled the prompt, the full suite plus holdout confirmed the improvement, and the new floor reflects the new normal.
• An eval-set version bumpnew cases changed the denominator; the old floor is meaningless against the new set, so record a fresh baseline in the same PR that bumps the set version.
• A taxonomy changeyou split or merged slices, so per-slice floors need re-deriving.

The illegitimate trigger is the tempting one: the gate is red and the deadline is close. Lowering the floor to make a failing run pass is deleting a failing test to ship — it converts your safety net into paperwork. The honest alternatives are: fix the regression, consciously accept and document the trade-off in the baseline file's note field with a follow-up ticket, or descope the change. What distinguishes an engineering organization is not that the pressure never arrives; it is that the baseline file's git history shows every time someone answered it, and how.

Case Study: The Regression the Average Hid

A realistic composite of how this plays out — Meridian's team, six weeks after wiring up the workflow above.

The moral is not that the team failed — the system worked: a silent behavioral regression, invisible in demos and in the aggregate, was caught within 18 hours by a scheduled run, diagnosed by per-slice reporting, reverted by clean attribution, and converted into two new eval cases and a stronger gate. Without the nightly run and slice view, this ships to customers and gets discovered as a refund-fraud spike in week three; without prompt-and-cases-in-one-PR, the bisect takes days. Every mechanism in this module earned its place in that timeline.

The Eval Maturity Checklist

Audit any team — including your own — against these ten. Each item is binary; count the yeses.

• Golden set in version controlcases live in the repo as plain data, with named set versions and an append-only history, not in someone's spreadsheet.
• Checkable criteria on every caseeach case states pass conditions two people would apply identically; no case graded on vibes.
• Mandatory slices definedthe failures that block ship at any rate are named, and the gate treats them as absolute, separate from the aggregate.
• Pre-merge gate wiredno prompt or model change reaches production without an automatic eval run; prompt and cases merge in the same PR.
• Committed baseline with historythe pass-rate floor is a reviewed file whose every change has an author, a reason, and a date.
• Calibrated judgeevery LLM-graded criterion has measured human agreement (kappa reported, holdout respected) and a pinned judge model with re-calibration on change.
• Per-slice reportingresults are read by failure-mode slice, and slice floors exist, because aggregates average away exactly the regressions that matter.
• Scheduled runs on unchanged systemsnightly or weekly full runs exist to catch provider, input, and judge drift, and slice regressions alert someone who will look.
• Online-to-offline pipelineproduction signals select traces, humans annotate a scheduled sample, and new failure modes flow into the golden set monthly.
• Documented re-baseline policythe legitimate triggers are written down, and red-gate-near-deadline is explicitly not one of them.

Scoring honestly: 0–3 is demo-driven development with extra steps; 4–7 is where most teams shipping AI features in 2026 actually sit; 8–10 is the discipline this track has been building toward. The checklist is also a hiring instrument — asking "walk me through how a prompt change reaches production" and mapping the answer against these ten tells you more than any benchmark discussion.

Exercises

1. 1Wire the gate. Put the Module 9 suite in a repo, commit the workflow and baseline file above, and open a PR that deliberately regresses the injection case. Deliverable: a screenshot of the blocked PR and the passing fix PR.
2. 2Add the schedule and slice view. Extend the workflow with an on: schedule nightly trigger and a step that prints per-slice pass rates (group by a slice tag you add to case descriptions). Deliverable: one nightly run's slice table.
3. 3Run the maturity audit. Score a team you know — your own, or one you interview with — against the ten items, with one sentence of evidence per item. Deliverable: the scorecard and the two items you would fix first, with a week-one plan for each.
4. 4Cost model your suite. For your own Module 9-style suite, estimate tokens per full run, cost per tier, and monthly cost at PR + nightly + weekly-deep cadence. Decide where sampling belongs. Deliverable: the arithmetic on one page.

Further Reading

• •
  promptfoo documentation — CI/CD integration — GitHub Actions patterns, output formats, and the official action
• •
  OpenAI Cookbook — Eval-Driven System Design — an end-to-end worked example of the eval-first inversion (note: OpenAI's hosted Evals platform is being retired in late 2026 — learn the methodology, not that platform)
• •
  OpenAI — Evaluation best practices — regression-testing workflow guidance
• •
  Hamel Husain & Shreya Shankar — AI Evals for Engineers & PMs — the CI-and-monitoring weeks of the benchmark course this mini-track mirrors