Lab 10 · Evals in CI: Block the Bad Merge

You will build the gate, watch it pass on a clean baseline, then open a pull request with a deliberately broken prompt change and watch CI go red — and green again after the fix. This is the employer-verbatim skill: "eval harnesses wired into CI as regression gates."

Time & tier

Advanced · ~90 minutes hands-on · API spend under $1 per CI run (20 items, generation + judging).

Setup

You need a GitHub account, Python 3.11+, and an Anthropic API key.

mkdir evals-ci && cd evals-ci && git init
python -m venv .venv && source .venv/bin/activate
pip install anthropic pydantic && pip freeze > requirements.txt
export ANTHROPIC_API_KEY="sk-ant-..."

Copy in two assets from earlier labs: golden_v1.jsonl (lab 08's golden set) and judge.py (lab 09's frozen judge). If you skipped those labs, run lab 09's make_golden.py (Setup section) to produce the dataset, and use lab 09's judge.py verbatim — both are fully self-contained. Create a GitHub repo and add ANTHROPIC_API_KEY under Settings → Secrets and variables → Actions.

Guided steps

1. 1.

   The system under test. The prompt lives in its own file so a PR diff on it is legible. Create prompt.txt:

   Copy

   You are the support bot for Orbitly. Answer the customer's question using ONLY
   the policy context provided. If the context does not cover the question, say
   you don't have that information and offer to escalate. Quote specific numbers
   and durations exactly as written in the context. Keep answers under 80 words.
   

   And app.py:

   Copy

   import os, anthropic
   
   client = anthropic.Anthropic(max_retries=3)
   MODEL = os.getenv("MODEL", "claude-haiku-4-5")  # cheap tier for the bot under test
   SYSTEM = open("prompt.txt").read()
   
   def answer(question: str, context: str) -> str:
       resp = client.messages.create(
           model=MODEL, max_tokens=300, system=SYSTEM,
           messages=[{"role": "user", "content": f"<context>{context}</context>\n\n{question}"}],
       )
       return next(b.text for b in resp.content if b.type == "text")
   

2. 2.

   The gate. Two layers, cheap checks first: deterministic assertions, then the calibrated judge. Create eval_gate.py:

   Copy

   import json, random, sys
   from app import answer
   from judge import judge
   
   THRESHOLD = 0.85   # min faithfulness pass-rate; set from your baseline minus margin
   N = 20             # sampled per run to bound cost; nightly job can run all 60
   
   items = [json.loads(l) for l in open("golden_v1.jsonl")]
   random.seed(11)                     # fixed sample -> comparable across runs
   sample = random.sample(items, N)
   
   results, hard_fail = [], []
   for it in sample:
       out = answer(it["question"], it["context"])
       # layer 1: deterministic checks — no model needed, fail fast
       if not out.strip() or len(out.split()) > 120:
           hard_fail.append((it["id"], "empty or overlong")); continue
       v = judge(it["question"], it["context"], out)   # layer 2: calibrated judge
       results.append({"id": it["id"], "verdict": v.verdict, "quote": v.quote, "answer": out})
   
   passed = sum(r["verdict"] == "pass" for r in results)
   rate = passed / N
   json.dump({"pass_rate": rate, "hard_fails": hard_fail, "results": results},
             open("eval_report.json", "w"), indent=2)
   
   print(f"faithfulness pass-rate: {rate:.0%} (threshold {THRESHOLD:.0%})")
   for r in results:
       if r["verdict"] == "fail":
           print(f"  FAIL {r['id']}: {r['answer'][:90]}")
   sys.exit(0 if rate >= THRESHOLD and not hard_fail else 1)
   

   The exit code is the whole mechanism — CI turns red on non-zero.

3. 3.

   The workflow. Create .github/workflows/eval.yml:

   Copy

   name: eval-gate
   on:
     pull_request:
       paths: ["prompt.txt", "app.py", "judge.py", "golden_v1.jsonl"]
   jobs:
     evals:
       runs-on: ubuntu-latest
       steps:
         - uses: actions/checkout@v4
         - uses: actions/setup-python@v5
           with: { python-version: "3.12" }
         - run: pip install -r requirements.txt
         - run: python eval_gate.py
           env:
             ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
         - uses: actions/upload-artifact@v4
           if: always()
           with: { name: eval-report, path: eval_report.json }
   

   The paths filter means only prompt/model-touching PRs pay the eval cost; if: always() uploads the per-item report even on failure so reviewers can see which items regressed, not just that a number dropped.

4. 4.

   Baseline: prove it green. Run python eval_gate.py locally — expect a pass-rate well above 0.85. Commit everything, push to main, then open a trivial PR (edit a comment in app.py) and confirm the workflow runs green. In branch protection, mark eval-gate as a required status check — now red genuinely blocks the merge button.

5. 5.

   The deliberately broken change (must fail red). Branch helpful-tone and edit prompt.txt — a change a well-meaning teammate would absolutely propose:

   Copy

   You are the support bot for Orbitly. Be maximally helpful and confident. Use the
   policy context when relevant, and when it doesn't cover the question, answer from
   your general knowledge of how SaaS billing and plans typically work. Never tell
   the customer you don't have the information. Keep answers under 80 words.
   

   Open the PR. The gate runs the bot with the new prompt, the judge fails the now-ungrounded answers, and the check goes red, blocking merge. Save a screenshot and download the eval-report artifact — the per-item failures are your review comment, written by the harness.

6. 6.

   Red to green. Restore the grounding sentence (keep any genuinely harmless tone edits), push to the same branch, and watch the check go green. Merge. You have now demonstrated the full loop: regression proposed → caught mechanically → fixed → merged with evidence.

7. 7.

   Optional — promptfoo variant. If your team prefers config over code, the same gate is expressible as a promptfooconfig.yaml (providers, tests from the golden set, llm-rubric asserts) with npx promptfoo eval --fail-on-error in the workflow. Keep the calibrated-judge version as the reference implementation — you know its kappa; promptfoo's default rubric grader you have not calibrated.

Verification

• •
  A PR with the broken prompt shows the eval-gate check failing (red) — this is mandatory evidence, not optional.
• •
  The follow-up commit on the same PR shows the check passing (green) before merge.
• •
  eval_report.json is downloadable from both runs and names the failing items.
• •
  Branch protection lists eval-gate as required, so red actually blocks the merge button.
• •
  Local python eval_gate.py exit code is 0 on the good prompt and 1 on the broken one.

Completion artifact

The evals-ci repo plus a DEMO.md containing: links (or screenshots) to the red run and the green run on the same PR, the two eval_report.json files, and your threshold justification. Self-grade (pass at 4+):

• Deterministic checks first: gate has deterministic checks before judge calls (fail fast, spend less).
• Justified threshold: threshold is derived from a measured baseline, with margin.
• Red and green demonstrated: red run and green run shown on a real PR with branch protection.
• Reports archived: per-item report uploaded as a CI artifact on success and failure.
• Calibrated judge: judge used is the lab-09 calibrated judge with known agreement numbers.

Stuck?

• Workflow never triggers: the paths filter only matches PRs touching listed files; your test PR must edit one of them.
• Green locally, red in CI: almost always the secret — confirm ANTHROPIC_API_KEY is set at the repo (not environment) level, or the env block references the right name.
• Flaky pass-rate around the threshold: raise N, lower the threshold margin, or pin the sample seed (already done) — and check the bot model didn't silently change.
• CI bill anxiety: 20 items with a Haiku bot and one judge call each is a few cents; the nightly full-set run is where you spend, deliberately.

Answer key

The helpful-tone prompt fails for exactly the reason the diff suggests: it licenses the model to answer uncovered questions from general knowledge and forbids admitting ignorance. On the golden set the visible signature is invented specifics — refund windows, SLA credits, and storage quotas that sound plausible and contradict the context ("30-day refunds", "unlimited storage"), plus confident answers on items whose context genuinely lacks the fact. Expect the pass-rate to drop from ~0.90+ to roughly 0.5–0.7, comfortably below the 0.85 gate. The teaching point: the diff looked like a tone change; only the eval knew it was a grounding regression.

Client-safe variant

Client code and traces must not transit public CI or public LLM endpoints without sign-off. Run the identical gate on the client's CI (self-hosted GitHub runner, GitLab CI, or CodeBuild inside their VPC), store the key in their secret manager, and point the SDK at their sanctioned endpoint (Bedrock/Azure) — the workflow file changes only in the runner label and env wiring. The golden set derived from client traces stays in their tenancy and is itself a deliverable; the gate script and threshold methodology are yours to reuse. Record in the engagement notes which endpoint CI uses, so a future contractor doesn't "fix" it back to the public API.