"How do you evaluate it?" is the most-cited differentiator in applied-LLM hiring, and for a concrete reason: without an eval gate, every prompt tweak and model upgrade ships on vibes, and regressions surface as customer complaints.
Tier: Advanced · Workflow area: Building LLM Features
This playbook builds the production norm — a golden dataset plus layered assertions that run like a test suite, wired into CI so a broken prompt change fails the pipeline before deploy, demonstrated red and green.
pip install pytest anthropic — or npx promptfoo@latest init for the config-driven route.Curate a 30-case golden dataset from reality. Pull from production logs (pb-51's traces make this trivial), not imagination. Composition that catches regressions: ~15 representative cases, ~8 known past failures (the bugs you already fixed — your regression suite), ~7 edge/adversarial (empty input, wrong language, injection attempts, off-topic). Each case: input, expected (exact output, key facts, or a rubric), and assert_type. Store as JSONL in the repo, versioned with the prompts it tests.
Layer assertions cheapest-first. Deterministic checks catch most regressions for free; the judge handles what code cannot:
Write the harness as a pytest suite — it is just tests.
import json, pytest, anthropic
from app.llm import triage # the function under test (pb-44)
CASES = [json.loads(l) for l in open("evals/golden.jsonl")]
client = anthropic.Anthropic(timeout=30.0)
@pytest.mark.parametrize("case", CASES, ids=lambda c: c["id"])
def test_golden(case):
out = triage(case["input"])
if case["assert_type"] == "exact":
assert out.category == case["expected"]["category"]
elif case["assert_type"] == "contains":
assert all(s in out.summary for s in case["expected"]["must_contain"])
elif case["assert_type"] == "judge":
assert judge(case, out) >= 4
def judge(case, out) -> int:
resp = client.messages.parse(
model="claude-opus-4-8", max_tokens=512,
system="Score the response 1-5 against the rubric. Justify, then score. "
"A 4 means: meets every rubric criterion with minor style issues only.",
messages=[{"role": "user", "content":
f"Rubric: {case['rubric']}\nInput: {case['input']}\n"
f"Response: {out.model_dump_json()}"}],
output_format=JudgeScore) # {justification: str, score: int}
return resp.parsed_output.score
Calibrate the judge before trusting it. Hand-score 15 outputs yourself, run the judge on the same 15, and compare. Agreement below ~85% means the rubric is vague — tighten it with explicit criteria and an anchored example of a 3 vs a 5. An uncalibrated judge is noise with confidence. Re-calibrate whenever you change the judge's model or rubric.
Wire it into CI with a threshold, not perfection.
# .github/workflows/evals.yml
on:
pull_request:
paths: ["app/prompts/**", "app/llm/**", "evals/**"]
jobs:
evals:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: pip install -r requirements.txt
- run: pytest evals/ --tb=short -q
env:
ANTHROPIC_API_KEY: ${{ secrets.EVAL_API_KEY }}
Policy: tier-1/2 failures block merge outright; judge-tier score drops >10% vs the base branch block; small judge wobbles warn. Path-scoping keeps eval cost off unrelated PRs.
Prove red, then green. Deliberately break the prompt (delete the refusal instruction, or swap the schema) in a branch: CI must fail with a readable diff of which cases broke. Revert: green. Do this once in front of the team — it is the demo that makes evals-before-merge a norm instead of your hobby.
Feed the dataset from production forever. Every user-reported bad output becomes a labeled case in the next PR. The dataset's growth rate is the health metric of your eval culture.
The golden dataset is a spec that executes. Prompt engineering without it is refactoring without tests — you can do it, but you cannot know what you broke.
client.responses.parse(model="gpt-5.6", ..., text_format=JudgeScore) — some teams deliberately judge with a different provider than they generate with, to decorrelate errors.