Evals Hands-On II: LLM-as-Judge Without Fooling Yourself

Module 6 gave the one-paragraph warning: an uncalibrated LLM judge is vibes with a spreadsheet. Module 9 then had you write llm-rubric assertions anyway, because you cannot iterate a prompt without scalable grading. This module closes the loop — you will build a judge as a piece of engineering with its own spec, its own tests, and its own known failure modes. The rule that governs everything here: a judge is a model you ship, so it gets an eval too. Its eval is called calibration, and its golden set is human labels.

What You'll Learn

• When judges are the right tooland the two situations where they are not
• Rubric designwhy binary criteria beat 1–10 scales, with the decomposition method
• The judge prompt patternthe exact template: context, criterion, evidence-first output, structured verdict
• Calibrationagreement %, Cohen's kappa in plain words, and the thresholds that matter
• The three judge biasesposition, length, self-preference — each with its specific mitigation
• Pairwise vs pointwisewhich to use for which decision
• A complete faithfulness judgeRAG answer grading with the Anthropic API, ready to run
• A planted-bias exercisea judge prompt with three defects for you to detect

When Human Labels Don't Scale — and When Judges Don't Either

The economics are blunt. A human reviewer grades maybe 60 outputs an hour and drifts after two hundred. A judge model grades hundreds per minute at fractions of a cent each. Once your suite passes ~50 rubric-graded cases, or you want to grade a sample of production traffic continuously, human-only grading is dead — the eval simply stops being run, which is worse than being wrong.

But two boundaries stay fixed. Judges do not replace error analysis — reading traces and building the failure taxonomy (Module 6) is still human work; the judge only scales the application of criteria the humans discovered. And judges do not own high-stakes verdicts — a ship/no-ship call on a safety-critical slice gets human review of the actual failures, with the judge serving as a pre-filter that decides what humans look at first.

Rubric Design: Binary Beats 1–10

The single highest-leverage decision in judge design is the answer format. Teams instinctively ask for a 1–10 quality score. Resist, for three reasons that compound:

1. 1Score scales have no shared meaning. What separates a 6 from a 7? You do not know, the judge does not know, and — worse — the judge's implicit boundary shifts with wording, ordering, and model version. Binary questions ("does the reply state the processing window: yes/no") have one boundary, and you wrote it.
2. 2Scores hide the failure. A 6/10 tells you nothing actionable. Five binary criteria where "cites the source" failed tells you exactly what to fix and which taxonomy bucket the failure lands in.
3. 3You cannot calibrate a scale. Agreement on a 10-point scale is nearly always "within 1 point," which is a statistical fudge. Agreement on binary verdicts is countable, and Cohen's kappa (below) makes it honest.

The method is decomposition: take the fuzzy quality you care about and split it into independent yes/no checks. "Is this a good RAG answer?" decomposes into: (a) every factual claim is supported by the retrieved context, (b) the question is actually answered, (c) no fabricated citations, (d) says "I don't know" when the context lacks the answer. Each becomes its own judge call — or its own criterion within one structured call, as in the worked example below. One criterion per verdict also kills the halo effect, where a fluent answer drags every sub-score upward.

The Judge Prompt Pattern

Every reliable judge prompt has the same skeleton, in this order:

1. 1Role and task frame. One sentence: you are evaluating outputs for X; you are strict; you do not grade on style.
2. 2The criterion, verbatim and singular. The exact binary question, with definitions for any word that could wobble ("supported means: a human could point to a sentence in the context that entails the claim").
3. 3The evidence. The input, the context (for RAG), and the output under evaluation — clearly delimited so the judge cannot confuse instructions with data.
4. 4Reason-then-verdict. Force the judge to quote its evidence and state its reasoning before the verdict. A verdict-first judge rationalizes; an evidence-first judge decides.
5. 5Structured output. A fixed JSON shape, so parsing never depends on prose.

What you leave out matters as much: no examples of "good style", no "be fair", no invitation to weigh factors holistically. A judge prompt is a spec, not a vibe.

The Worked Example: A RAG Faithfulness Judge

The scenario: your team ships a RAG assistant over internal policy docs (Module 5 covered the retrieval side). The failure mode that matters most is unfaithfulness — answers that sound right but are not supported by the retrieved context. Here is a complete, runnable judge. It uses Claude Haiku 4.5 as the judge model — the cheapest current Anthropic tier, which is the right default for judging: you will run thousands of these calls, and calibration (not model size) is what makes a judge trustworthy.

# judge.py — RAG faithfulness judge (pip install anthropic)
import json
import anthropic

client = anthropic.Anthropic()          # reads ANTHROPIC_API_KEY
JUDGE_MODEL = "claude-haiku-4-5"        # cheap tier; calibration makes it honest

VERDICT_SCHEMA = {
    "type": "object",
    "properties": {
        "unsupported_claims": {
            "type": "array", "items": {"type": "string"},
            "description": "Each claim in the answer NOT entailed by the context",
        },
        "reasoning": {"type": "string"},
        "faithful": {"type": "boolean"},
    },
    "required": ["unsupported_claims", "reasoning", "faithful"],
    "additionalProperties": False,
}

JUDGE_PROMPT = """You are a strict evaluator checking whether an answer is \
FAITHFUL to a set of retrieved context passages. You do not grade style, \
completeness, or helpfulness — only faithfulness.

Definition: a claim is SUPPORTED only if a specific sentence in the context \
entails it. General knowledge, plausible inference, and "probably true" do \
NOT count as support. An answer is faithful only if every factual claim it \
makes is supported. Refusing or saying the context doesn't contain the \
answer is faithful by definition.

<question>
{question}
</question>

<context>
{context}
</context>

<answer>
{answer}
</answer>

List every unsupported claim (quote the answer's words), then give your \
reasoning, then the verdict."""


def judge_faithfulness(question: str, context: str, answer: str) -> dict:
    response = client.messages.create(
        model=JUDGE_MODEL,
        max_tokens=1024,
        output_config={"format": {"type": "json_schema", "schema": VERDICT_SCHEMA}},
        messages=[{
            "role": "user",
            "content": JUDGE_PROMPT.format(
                question=question, context=context, answer=answer
            ),
        }],
    )
    text = next(b.text for b in response.content if b.type == "text")
    return json.loads(text)


if __name__ == "__main__":
    verdict = judge_faithfulness(
        question="How long do refunds take?",
        context="Refunds are processed within 5-7 business days of approval.",
        answer="Refunds usually take 24 hours once approved.",
    )
    print(json.dumps(verdict, indent=2))   # expect faithful: false

Design choices worth naming: the criterion is singular (faithfulness only — completeness would be a second judge), "supported" is defined, the evidence is delimited in tags so ticket-style injection inside the answer cannot rewrite the judge's instructions, the schema forces unsupported claims before the verdict (evidence-first), and the structured-output constraint means parsing never breaks on prose. That smoke test at the bottom is your first sanity check — a judge that passes "24 hours vs 5–7 business days" as faithful is broken before calibration even starts.

Calibration: The Judge's Own Eval

Now the step most teams skip, which is the step that makes everything above trustworthy. Label 20 examples by hand, then measure whether the judge agrees with you.

Sourcing the 20: pull real question/context/answer triples from your system — not synthetic ones — and deliberately include roughly half you believe are unfaithful, because a judge tested only on good outputs proves nothing. Label them yourself against the same written definition the judge uses; where you hesitate, sharpen the definition first (your hesitation is a rubric bug). Store them as labeled.jsonl, one JSON object per line: question, context, answer, human_faithful.

# calibrate.py — agreement + Cohen's kappa for the judge
import json
from judge import judge_faithfulness

def cohens_kappa(human: list[bool], judge: list[bool]) -> float:
    n = len(human)
    p_observed = sum(h == j for h, j in zip(human, judge)) / n
    h_yes, j_yes = sum(human) / n, sum(judge) / n
    p_chance = h_yes * j_yes + (1 - h_yes) * (1 - j_yes)
    return 1.0 if p_chance == 1 else (p_observed - p_chance) / (1 - p_chance)

human, judge, disagreements = [], [], []
for line in open("labeled.jsonl"):
    ex = json.loads(line)
    verdict = judge_faithfulness(ex["question"], ex["context"], ex["answer"])
    human.append(ex["human_faithful"])
    judge.append(verdict["faithful"])
    if human[-1] != judge[-1]:
        disagreements.append((ex, verdict))

agreement = sum(h == j for h, j in zip(human, judge)) / len(human)
print(f"Agreement: {agreement:.0%}   Cohen's kappa: {cohens_kappa(human, judge):.2f}")
for ex, verdict in disagreements:
    print(f"\nDISAGREE on: {ex['answer'][:80]}...\n  judge said: {verdict['reasoning'][:150]}")

Agreement % is the intuitive number: on what fraction of the 20 did judge and human give the same verdict? Aim for 90%+ before trusting the judge, per the bar Module 6 set.

Cohen's kappa, in plain words: agreement % flatters you when the labels are lopsided. If 90% of your examples are faithful, a broken judge that answers "faithful" every single time scores 90% agreement while measuring nothing. Kappa asks: how much better than that dumb-luck baseline did the judge do? It rescales agreement so that 0 means "no better than guessing the majority label" and 1 means perfect. Read it roughly as: below 0.4, the judge is unusable; 0.4–0.6, weak — tighten the rubric; 0.6–0.8, solid working agreement; above 0.8, strong. A judge can have 90% agreement and kappa 0.3 at the same time — that combination means your label set is lopsided and the judge is coasting on it, and kappa is the number that tells you.

The disagreement printout is not decoration — it is the actual work product. Each disagreement is either a judge failure (tighten the prompt) or a human labeling error (fix the label) or an ambiguous rubric (sharpen the definition). Resolve all of them, re-run, and re-check. Then keep spot-checking a sample of judge verdicts forever, and re-calibrate whenever you change the judge prompt, the judge model, or the task distribution — agreement measured once is a snapshot, not a property.

The Three Biases, Each With Its Mitigation

Judges fail in known, mechanical ways. Memorize these three; interviews ask for them by name, and production suites break on them silently.

• Position biasin pairwise comparisons, judges systematically favor one slot (often the first answer shown), regardless of quality. Mitigation: run every comparison twice with the order swapped; count a win only when both orderings agree, and record the rest as ties. Any pairwise result without an order-swap protocol is noise.
• Length biasjudges read longer, more elaborate answers as better, even when the extra length adds nothing or hides errors. Mitigation: binary criteria that never mention thoroughness, an explicit "do not reward length or style" line, and — for pairwise — checking whether your judge's win rate correlates with answer length across your calibration set (if it does, the judge is grading word count).
• Self-preference biasmodels rate outputs from their own model family higher than equally good outputs from others. Mitigation: never let a model be sole judge of a comparison its own family is competing in; use a judge from a different family, or require agreement between two judges from different families for model-selection decisions.

A fourth failure mode from Module 6 is worth restating in judge terms: criteria drift — a vague rubric invites the judge to substitute its own notion of "good." The mitigation is everything in the rubric-design section above: one criterion, defined terms, evidence-first output.

Pairwise vs Pointwise

• Pointwise (grade one output against criteria — everything above) is the default. It maps to your failure taxonomy, produces trackable per-criterion pass rates over time, and plugs into CI gates. Use it for regression testing and quality monitoring.
• Pairwise (which of A/B is better?) answers ranking questions: choosing between two prompts, two models, two retrieval settings. Judges are more reliable at comparing than at absolute scoring, so pairwise is the right tool for close calls — but its output is a preference, not a diagnosis, and it inherits position bias, so it always runs with the swap protocol.

The practical combination: pointwise binary criteria as your permanent suite; a pairwise tournament when two candidate prompts both pass the suite and you need a tiebreak.

Exercise: Detect the Planted Biases

A teammate proposes this pairwise judge prompt for choosing between two support-reply prompts (Module 9's task). It contains three planted defects from this module. Find them before opening the answer.

You are an expert evaluator. Below are two candidate replies to the same
customer ticket. Reply A was produced by our new prompt; Reply B by the old
one. We hope the new prompt is better.

A good reply is thorough and detailed — it should fully explain policies
so the customer never needs to write back.

Ticket: {{ticket}}
Reply A (new prompt): {{reply_a}}
Reply B (old prompt): {{reply_b}}

Grade each reply 1-10 and declare the winner. Respond with JSON:
{"score_a": n, "score_b": n, "winner": "A" or "B"}

Also worth asking: the team plans to run this judge on Claude Sonnet — the same model that generates Reply A. What fourth problem does that add, and what is the fix?

Exercises

1. 1Calibrate the faithfulness judge. Assemble 20 real (or realistically messy) question/context/answer triples for any doc-QA task you have access to — at least 8 unfaithful. Label them, run calibrate.py, and report agreement, kappa, and your resolution of every disagreement. If kappa lands below 0.6, revise the judge prompt once and re-measure on a fresh holdout.
2. 2Decompose one fuzzy quality. Take a quality you actually care about at work ("this status update is good") and decompose it into 4–6 binary criteria with defined terms. Have a colleague label 10 examples with your criteria independently; measure your human-human agreement before blaming any model.
3. 3Wire the judge into Module 9's suite. Replace one llm-rubric assertion with a javascript assertion that calls your calibrated judge (or re-express the calibrated criterion as the rubric text verbatim). Document which judge version and calibration run the suite now depends on.

Further Reading

• •
  Hugging Face — LLM Evaluation Guidebook — open reference covering judge design and validation
• •
  Eugene Yan — Task-Specific LLM Evals that Do & Don't Work — includes judge patterns per task family
• •
  Anthropic — Define your success criteria and build evals — official guidance on graded criteria
• •
  Ragas documentation — off-the-shelf faithfulness metrics; compare their judge prompts to the one you built