An uncalibrated LLM judge is a random-number generator with good grammar — and it is grading your product. Judges systematically prefer the first answer shown, the longer answer, the better-formatted answer, and answers from their own model family.
Tier: Advanced · Workflow area: Evaluation Engineering
The 2026 best-practice literature is unambiguous: randomize order, take dual-order verdicts, length-match when needed, and continuously re-grade a human sample. This playbook builds a judge you can trust: rubric design, a formal bias audit, judge↔human agreement on a gold set (~200 traces), and the annotation ops — labeling guidelines, inter-annotator agreement, drift monitoring — that keep it trustworthy over time.
Skip for anything checkable by code: schema validity, exact answers, executable SQL. Judges are the fallback, not the default.
Write labeling guidelines before any prompt. A 1–2 page document humans can follow: the quality dimensions, a 1–5 (or pass/fail) scale with an anchored description and a real example per level, explicit edge-case rulings ("partially correct but confident → 2"), and what to ignore (formatting, unless formatting is the dimension). If humans cannot apply it consistently, no judge prompt will.
Build the human gold set with agreement stats. Two or more labelers independently label ~200 traces (stratified: all failure modes, all slices, ~30% clear passes). Compute inter-annotator agreement — Cohen's κ for two labelers (Krippendorff's α beyond two). κ < 0.6 means your guidelines are ambiguous: run a disagreement-resolution session, tighten the guidelines, relabel the disputed cases. Adjudicated labels = the gold set; final κ goes in the report.
Write the judge prompt from the guidelines. Structure: role + task; the rubric verbatim (same anchors as the human doc); the inputs (question, context if RAG, answer); required output — {"reasoning": "...", "verdict": ...} with reasoning before verdict; and 2–4 few-shot examples including at least one from each extreme level. For pairwise use, the same skeleton with responses A and B and verdict ∈ {A, B, tie}.
Run the bias audit. Four checks, each on 50+ items:
Measure judge↔human agreement and set the bar. Run the calibrated judge over the 200-trace gold set: report agreement rate and κ vs adjudicated labels, plus the confusion pattern (where does it disagree — which rubric levels, which failure modes?). Working bar: judge↔human κ within ~0.1 of human↔human κ. Below the bar, iterate the judge prompt against the confusion pattern (not against overall accuracy) — e.g., merge rubric levels the judge cannot distinguish.
Deploy with drift monitoring. In steady state: route 5–10% of judge-graded traces to human re-grade every cycle; track rolling judge↔human agreement on a dashboard; alert when it drops (judge-model version change, data drift, or your product drifting outside the rubric). Re-calibrate on the gold set whenever the judge model or prompt changes — a judge change is a metric change (pb-35's pin discipline).
Document the judge card. Model + version pin, prompt hash, rubric version, gold-set version, κ numbers, bias-audit results with dates, known blind spots, re-calibration triggers. This travels with every metric the judge produces.
A judge is calibrated when you can state its agreement with adjudicated human labels, its measured biases, and its blind spots — with numbers. Anything less is an opinion with an API bill.
Never bootstrap gold labels by having the judge label its own calibration set — circular calibration produces beautiful agreement with itself and none with reality.