An uncalibrated judge is an uncalibrated gauge on a pressure vessel: it blocks good releases and waves through bad ones with equal confidence. Before any LLM-as-judge may fail a build, you measure its agreement against human labels on a gold slice — and you control the biases judges are known to have.
Tier: Advanced · Workflow area: Testing AI systems — judges, agents & robustness
This playbook takes you from "we wrote a judge prompt that looks reasonable" to "we have an agreement report that says this judge may gate releases." The judge is a measurement instrument, and QA's job is the same as with any instrument: calibrate it, document its error bars, and schedule re-calibration.
You are about to wire a model-graded assertion into CI where a fail blocks a merge or release. Faithfulness, tone, helpfulness or policy-compliance verdicts feed a dashboard leadership acts on. You inherited a judge someone else wrote and nobody can say how often it is wrong.
The property is checkable deterministically — schema validity, regex, PII absence, length. Semantic similarity against a reference answer covers it. The judge is only exploratory (labeling for triage, not gating) — calibrate before promotion, not before experimentation.
Write the rubric before the judge prompt. Decide what "pass" means in words a new team member could apply. Prefer binary or 3-point scales — 1–10 scales collapse into noise around 6–8. Example rubric for a support-summary feature: Pass = every claim in the summary appears in the source ticket; no invented order numbers, dates or amounts; refund policy stated only if the ticket mentions it.
Hand-label the gold slice. Sample 50 outputs stratified across easy, hard and known-failure cases. Both labelers apply the rubric independently, then reconcile. Record inter-human agreement first:
Gold slice: 50 items
Labeler A vs Labeler B raw agreement: 46/50 (92%)
Reconciled labels: 34 pass, 16 fail
If two humans cannot exceed ~85% agreement, the rubric is ambiguous — fix the rubric, not the judge.
Write the judge prompt with structured output. Ask for the verdict last, after reasoning, and force JSON:
You are evaluating a customer-support summary against its source ticket.
Rubric:
- FAIL if the summary contains any claim not present in the ticket.
- FAIL if any number, date, or amount differs from the ticket.
- FAIL if refund policy is mentioned but the ticket never raises refunds.
- Otherwise PASS.
Ticket: {{ticket}}
Summary: {{summary}}
Think through each rubric line, quoting the ticket. Then output JSON:
{"reasoning": "...", "verdict": "PASS" | "FAIL"}
Judge the content only. Length, style and formatting are irrelevant.
Run the judge over the gold slice and compute agreement. A 20-line script is enough:
promptfoo eval -c judge-calibration.yaml -o judge-run.json
python compare_labels.py gold_labels.csv judge-run.json
Judge vs reconciled human labels (n=50)
Raw agreement: 44/50 (88%)
Agreement on FAILs: 12/16 (75%) <- the number that matters for a gate
Cohen's kappa: 0.72
False passes: 4 | False fails: 2
Weight agreement-on-fails over raw agreement: a gate exists to catch bad outputs, and a judge that passes everything scores 68% raw agreement here while being useless.
Probe the known biases. Re-run the slice with two controls: swap the order of any paired comparisons (position bias), and check verdict correlation with output length (verbosity bias — judges reward longer answers). If verdicts flip on swap or correlate with token count, add explicit rubric lines ("length is irrelevant") and re-run.
Read every disagreement. The 6 mismatches are the whole payoff. Typical findings: the judge misses subtle number changes (add an explicit "compare every number digit by digit" instruction), or humans mislabeled an item (fix the gold label). Iterate the judge prompt until agreement-on-fails clears your bar.
Set the bar, then wire the gate. A defensible starting policy: kappa ≥ 0.7 and agreement-on-fails ≥ 80% before the judge may block anything. Below that, the judge may only flag for human review. Version the judge prompt, model and threshold in git next to the eval config.
Schedule re-calibration. Re-run the gold slice whenever the judge model changes, the judge prompt changes, or quarterly — whichever comes first. Judges drift when providers silently update models; the gold slice is how you notice.
Never let the model under test judge itself with the same prompt family. Self-preference bias is well documented — use a different model family for the judge, or at minimum a different provider snapshot.
llm-rubric assert with the judge model pinned in defaultTest.options.provider; keep the calibration run as a separate config you re-run on schedule.evaluation_steps; calibration is a pytest run over the gold slice comparing metric.score to your labels.A judge earns gate power the way a new hire earns merge rights — by demonstrated agreement with people you trust, measured on work you already know the answer to, re-checked on a schedule. No agreement report, no gate.