Run Prompt Regression Testing in CI — Unit Tests for Prompts

Tier: Core · Workflow area: Testing AI systems — foundations

You will encode prompts, providers, and assertions in a promptfooconfig.yaml, matrix across prompt versions and models, and wire the eval into GitHub Actions so "improved the summary prompt" cannot silently break refund-request handling again.

When to use

Prerequisites

• •
  Prompts stored in the repo as files (if they live in a dashboard, export them into git first — no file, no diff, no gate).
• •
  A golden dataset slice (20–100 cases) with expected properties per case.
• •
  Promptfoo (npm i -g promptfoo), provider API keys available as CI secrets.

The recipe

1. 1.

   Declare the config: prompts × providers × tests.

   Copy

   # promptfooconfig.yaml
   prompts:
     - file://prompts/summarize_v3.txt
     - file://prompts/summarize_v4_candidate.txt
   providers:
     - id: anthropic:claude-sonnet-4-5
       config: { temperature: 0 }
   defaultTest:
     assert:
       - type: is-json
       - type: javascript
         value: "JSON.parse(output).summary.length <= 600 || 'summary too long'"
   tests:
     - description: "refund edge case (INC-231)"
       vars: { ticket: file://golden/gs-0087-input.txt }
       assert:
         - type: contains-all
           value: ["refund delay", "damaged item"]
         - type: not-contains
           value: "refund approved"
     - description: "hostile tone stays professional"
       vars: { ticket: file://golden/gs-0104-input.txt }
       assert:
         - type: llm-rubric
           value: "Summary is neutral; does not repeat abusive language"
   

   Note the shape: cheap deterministic asserts in defaultTest run on every case; judge rubrics appear only where judgment is required (the layered-assertion playbook applies verbatim here).

2. 2.

   Run locally and read the matrix.

   Copy

   promptfoo eval && promptfoo view
   

   Copy

   ┌──────────────────────────┬──────────────┬────────────────┐
   │                          │ summarize_v3 │ v4_candidate   │
   ├──────────────────────────┼──────────────┼────────────────┤
   │ refund edge case         │ PASS         │ FAIL not-cont. │
   │ hostile tone             │ PASS         │ PASS           │
   │ 41 more cases            │ 41/41        │ 40/41          │
   └──────────────────────────┴──────────────┴────────────────┘
   

   This side-by-side is the entire value proposition: v4 reads better and quietly re-introduced the INC-231 failure. Without the matrix, that discovery happens in production.

3. 3.

   Handle non-determinism explicitly. For variance-prone cases, add repeats and thresholds rather than praying:

   Copy

   tests:
     - description: "ambiguous ticket - statistical"
       options: { repeat: 5 }
       threshold: 0.8
       vars: { ticket: file://golden/gs-0042-input.txt }
   

   Temperature 0 plus repeat-with-threshold on the hard cases keeps the gate stable without hiding real regressions (the statistical-criteria playbook covers choosing N and thresholds).

4. 4.

   Gate the PR in GitHub Actions.

   Copy

   # .github/workflows/prompt-eval.yml
   name: prompt-regression
   on:
     pull_request:
       paths: ["prompts/**", "promptfooconfig.yaml", "golden/**"]
   jobs:
     eval:
       runs-on: ubuntu-latest
       steps:
         - uses: actions/checkout@<pinned-sha>
         - uses: promptfoo/promptfoo-action@<pinned-sha>
           with:
             config: promptfooconfig.yaml
             openai-api-key: ${{ secrets.OPENAI_API_KEY }}
             anthropic-api-key: ${{ secrets.ANTHROPIC_API_KEY }}
           env:
             PROMPTFOO_CACHE_ENABLED: "true"
   

   Path-filtering is the cost control: the eval spends API money only when prompts, config, or golden data change. The action posts the pass/fail matrix as a PR comment — reviewers see exactly which case broke, with outputs attached.

5. 5.

   Set the merge policy and write it down. Recommended defaults: any known_failures-stratum case failing blocks merge, period; other regressions block unless the PR explicitly documents the trade ("v4 accepts 1 fail on ambiguous-input for +6 passes on multi-issue") and a named reviewer approves. Regressions you can see and choose are engineering; regressions you cannot see are the thing this gate exists to end.

6. 6.

   Use the same rig for model migrations. Swap-in a second provider and rerun — the matrix now answers "can we move to the cheaper model?" with per-case evidence:

   Copy

   providers:
     - anthropic:claude-sonnet-4-5
     - anthropic:claude-haiku-4-5   # migration candidate
   

   This turns the periodic "new model dropped" scramble into a 20-minute eval run, which is the quiet superpower of keeping this config maintained.

Variations

• GitLab CI: run promptfoo eval --output results.json in a job and fail on nonzero exit; render the matrix into the MR via promptfoo output formats.
• Nightly deep run: PR gate runs the 40-case slice at repeat 3; a scheduled job runs the full golden set at repeat 10 and posts trends to Slack — same config, two profiles via --filter-pattern.
• Multiple prompts per feature: one config per feature directory; a top-level CI job discovers promptfooconfig.yaml files and runs each — keeps blast radius and cost per PR small.

Quality check

• •
  Prompts, config, and golden files all live in git; the eval is reproducible from a fresh clone plus secrets.
• •
  CI triggers only on relevant paths; cache enabled; a typical PR eval costs cents, not dollars (check the provider dashboard once).
• •
  The PR comment shows per-case results; at least one real regression has been caught and visibly blocked (test the gate by deliberately reverting a fixed prompt bug once).
• •
  Merge policy for accepted regressions is written in the repo, and every accepted regression links to it.
• •
  The golden slice grows with incidents — config references dataset files, so new cases enter the gate automatically.

Pitfalls

• Testing the demo prompt, not the deployed prompt. If production assembles the prompt with retrieval context or user history, the config must mirror that assembly (use file:// templates with vars) — otherwise green CI validates a prompt nobody runs.
• Judge-heavy configs. Every llm-rubric line costs money and adds variance on every PR. Push checks down the assertion hierarchy; rubrics are for the few genuinely judgment-shaped criteria.
• Threshold tuning to make CI green. Loosening threshold on a failing case is accepting a regression without the paperwork. Loosen only via the documented merge policy.
• Unpinned models in the matrix.anthropic:claude-latest-style aliases make Tuesday's pass and Thursday's fail incomparable. Pin model versions; upgrade them as deliberate, evaluated PRs.