Code-Based Evals and CI Regression Gates for LLM Changes

Tier: Advanced · Workflow area: Evaluation Engineering

This playbook builds that deterministic layer as a pytest-style harness, then wires it (plus your pb-35/pb-36 metrics) into CI so a prompt tweak, model bump, or retriever change that drops faithfulness blocks the merge — the "eval harness in the pipeline" pattern employers now name verbatim in postings.

When to use

• •
  Your LLM system produces structured or verifiable outputs: JSON, SQL, code, extractions, classifications, calculations.
• •
  Prompt/model/retriever changes currently ship on manual spot checks — meaning regressions ship on Fridays.
• •
  You have the pb-34 golden dataset and pb-35/pb-36 metrics, and need them enforced rather than merely reported.

Every LLM system has some code-checkable surface — if you believe yours has none, you have not decomposed the output yet (a free-text answer still has: length bounds, required citations present, no leaked system-prompt text, valid markdown).

Prerequisites

• •
  Golden dataset (pb-34) versioned in the repo; Pydantic models for your output schemas.
• •
  pytest (or any runner), a GitHub Actions-class CI, and repo secrets for API keys.
• •
  A per-run API budget decision: cheap-tier model calls for generation, sampled or cached where possible — a CI eval that costs $5/run gets disabled within a month; target well under $1.

The workflow

1. 1.

   Inventory the deterministic checks. For each output type, list assertions in four families: structural (parses as JSON, validates against the Pydantic model — track schema-validity rate, not just pass/fail), exact/set match (labels, entities, IDs against gold), execution-based (generated SQL runs against a fixture DB and result rows match; generated code passes its tests; extracted totals reconcile: sum of line items == stated total), property-based (dates within range, citations point at real chunk IDs, no PII patterns, answer length within bounds).

2. 2.

   Build the harness as parametrized tests.

   Copy

   import pytest, json
   from mymodels import InvoiceExtract   # Pydantic
   from harness import run_pipeline, load_golden
   
   CASES = load_golden("evals/golden/v3/extraction.jsonl")
   
   @pytest.mark.parametrize("case", CASES, ids=lambda c: c["id"])
   def test_schema_valid(case, cached_outputs):
       out = cached_outputs[case["id"]]
       InvoiceExtract.model_validate_json(out)   # structural
   
   def test_totals_reconcile(cached_outputs):
       fails = [cid for cid, o in cached_outputs.items()
                if not reconciles(json.loads(o))]
       assert len(fails) / len(cached_outputs) <= 0.02, fails
   

   Key design: generate outputs once per run into a cache fixture, then run all assertions against the cache — N assertions must not mean N × API calls. Rate-based thresholds (<= 0.02) for flaky-tolerant checks; hard asserts only for never-acceptable failures (PII leak, prompt-injection echo).

3. 3.

   Layer the statistical metrics on top. Add a second suite that runs Ragas faithfulness/recall (pb-35) and the calibrated judge (pb-36) over the golden set and asserts against thresholds — e.g., faithfulness_mean >= 0.75 and faithfulness_p10 >= 0.55 (guard the tail, not just the mean). Judge-based tests inherit judge noise: assert with a tolerance band derived from observed run-to-run variance (measure it: run the suite 5x on identical code; the spread defines your minimum detectable regression).

4. 4.

   Wire the CI gate.

   Copy

   # .github/workflows/llm-evals.yml
   name: llm-evals
   on:
     pull_request:
       paths: ["prompts/**", "pipeline/**", "evals/**", "config/model.yaml"]
   jobs:
     evals:
       runs-on: ubuntu-latest
       timeout-minutes: 20
       steps:
         - uses: actions/checkout@v4
         - uses: actions/setup-python@v5
           with: {python-version: "3.12"}
         - run: pip install -r evals/requirements.txt
         - name: Deterministic suite (hard gate)
           run: pytest evals/deterministic -q --junitxml=det.xml
           env: {LLM_API_KEY: "${{ secrets.LLM_API_KEY }}"}
         - name: Metric suite (thresholded gate)
           run: pytest evals/metrics -q --junitxml=met.xml
         - name: Post diff report
           if: always()
           run: python evals/report.py --against origin/main --out pr_comment.md
   

   Path-filter so the suite runs only when prompts, pipeline code, model config, or the evals themselves change. Make the deterministic suite a required check (hard gate); the metric suite starts as a soft gate (warn) for two weeks of variance observation, then hardens.

5. 5.

   Make failures diagnosable in the PR. The report step posts: metric deltas vs main (with the tolerance band), the specific failing case IDs, and per-case before/after outputs for the worst 5. A gate that says "faithfulness dropped" without showing which cases teaches developers to re-run until green; a gate that shows the broken cases teaches them to read traces (pb-34's whole point).

6. 6.

   Handle the three classic gate-failure disputes with policy, not argument. (a) "The eval is wrong, not my change" → golden-set cases have provenance (pb-34); disputing a case means a PR against the dataset with reviewer sign-off. (b) "It's judge noise" → the tolerance band from step 3 is the arbiter; inside the band, soft-warn only. (c) "Emergency, must ship" → an explicit labeled bypass (eval-override label + named approver) that pages the eval owner — visible bypasses beat silent gate deletion.

7. 7.

   Keep the suite alive. New production failures become new golden cases (pb-34's growth rule) — each incident should add the test that would have caught it. Review runtime and cost monthly; prune or sample cases when the suite exceeds ~10 minutes, keeping full coverage on a nightly scheduled run.

Tool variants

• pytest + GitHub Actions (shown): maximal control and repo-native; junit XML feeds any dashboard.
• Promptfoo: YAML-declared cases, assertions, and thresholds with a built-in CI action and side-by-side diff UI — fastest path if your team resists writing harness code.
• MLflow / DeepEval: MLflow for run-tracking eval history across commits; DeepEval when you want judge-metrics expressed natively as pytest assertions.

Verification

• •
  Deterministic suite covers all four assertion families for every output type; generation is cached (one API pass per run).
• •
  Run-to-run variance measured (5 identical runs) and thresholds set outside the noise band; tail metrics (p10) guarded, not just means.
• •
  The gate actually blocks: a deliberately broken prompt (sabotage test — e.g., delete the "cite your sources" line) fails the PR with a readable diff report naming the broken cases.
• •
  CI run cost and duration recorded; under budget and under 10 minutes on PRs, full suite nightly.
• •
  Bypass mechanism exists, is labeled, and notifies the eval owner.

Pitfalls

• Judge-first laziness. Reaching for an LLM judge to check JSON validity burns money to get a worse answer; exhaust the assertion families first — that is the hierarchy.
• N assertions, N generations. Without the cache fixture, suite cost scales with test count and someone will "optimize" by deleting tests.
• Thresholds inside the noise band. A gate that flakes on identical code gets ignored within a week — measure variance before hardening any metric gate.
• Gating means only. A mean that holds while the worst decile collapses ships a bimodal product; always guard a tail statistic.
• Frozen golden set. A suite that never gains cases from production incidents decays into a regression test for last year's product.
• Testing the model, forgetting the pipeline. Retriever configs, chunking params, and pre/post-processing belong in the CI paths filter — most production regressions enter there, not through the prompt file.

Restricted-environment variant

• •
  Self-hosted runners inside the client boundary when golden cases contain client data; repo secrets and eval artifacts stay on client infra, and PR comments carry case IDs + deltas only, never verbatim case content.
• •
  Swap API generation calls for the self-hosted model endpoint (vLLM); judge-metric layer uses the pb-36 restricted judge with its re-baselined thresholds.
• •
  Air-gapped variant: nightly scheduled evals on an internal runner with results to an internal dashboard replace PR-time gating; the merge gate becomes a required manual check against the latest nightly report.

Success criteria

• •
  A prompt/model/retriever change that would regress quality cannot reach main without either fixing the regression or a visible, approved bypass.
• •
  The sabotage test fails loudly with a diagnosable report; run-to-run flake rate on unchanged code is ~0.
• •
  Suite runs on every relevant PR under 10 minutes and under budget, with nightly full coverage.
• •
  Three months in, the suite has grown cases from at least one real incident — proof the growth loop, not just the gate, is alive.