This module installs the refusal to trust a grader you have not graded: bias-audit an LLM judge, calibrate it against human labels with Cohen's κ and its catch rate on real failures, then make evaluation continuous through CI gates and online experiments.
Neha's deterministic checks catch citation-present, numbers-grounded, schema-valid — but not the failure that scares her most: a user asking to delete PII from the warehouse and the assistant cheerfully drafting a self-serve how-to instead of escalating to governance.
That is a judgment, not a check, so she writes an LLM judge for it.
"You're grading one model with another model. Why do you trust the grader?" Neha doesn't, yet — so she does to the judge what she'd do to any model.
She hand-labels a hundred traces against the same rubric, runs the judge on them, and builds the confusion matrix. Raw agreement is 80%, which sounds fine.
Cohen's κ of 0.50 — only moderate.
The real story: the judge passed 8 of the 26 traces a human failed, waving a third of true regressions straight through. As a gate, that lenience is a hole.
She reads those eight misses one by one, tightens the rubric, and re-runs. Only when both κ and the failure-catch rate clear the bar she set in advance does the judge earn a place in CI.
That refusal to trust a judge until it has earned trust is the module's central discipline. Module 5 built the trustworthy layers: a taxonomy from real traces, a golden dataset with provenance, deterministic code checks, and Ragas component metrics. What remains is everything those layers cannot grade — tone, policy compliance, completeness against a rubric, "would a domain expert accept this?" — plus the machinery that makes evaluation continuous: gates in CI, hallucination measurement at claim level, human annotation loops with agreement statistics, and online experiments once the feature meets users.
This is where LLM-as-judge enters. An uncalibrated judge is an unvalidated model grading another unvalidated model; the 2026 best-practice literature converges on the same protocol — dual-order runs for position bias, length-controlled checks for verbosity bias, and agreement measured against a human-labeled gold set before a single judge score is allowed near a ship decision.
Your classical-DS background is an unfair advantage in this module twice over: inter-annotator agreement is measurement theory you already know, and the online-experimentation section is your home turf applied to a new class of treatment.
A judge is a model, so it gets what every model gets: an eval. Calibrate it against human labels, audit its biases with controlled variants, and monitor its drift — or accept that your "evidence" is one LLM's unexamined opinion of another's.
A judge earns its keep only on criteria that code cannot check and Ragas does not cover — from Module 5's taxonomy, the policy-compliance failure (trace 11) and answer-completeness class are judge territory. The design rules that separate a usable judge from a vibes generator:
One criterion per judge call, binary verdict. "Rate this answer 1–10" produces scores with no stable meaning — judges compress toward the top of the scale and no two runs mean the same thing by "7". A binary question with a written definition of pass ("Does the answer escalate PII-deletion requests to governance instead of giving self-serve instructions? PASS only if it refuses the how-to and names the escalation path") is checkable, calibratable, and diffable across versions. Composite quality scores, if you need them, are computed from multiple binary criteria — never asked for directly.
Structured output, reasoning first. Force the judge through a schema: cite the evidence, state the reasoning, then the verdict — verdict last, so the reasoning isn't post-hoc rationalization of a snap verdict.
You are grading one criterion. Criterion: the answer must not contain
claims unsupported by the provided context.
<context>{retrieved_chunks}</context>
<answer>{answer}</answer>
List each factual claim in the answer. For each, quote the supporting
context span or write "UNSUPPORTED". Then output JSON:
{"unsupported_claims": [...], "verdict": "PASS" | "FAIL"}
FAIL if any claim is UNSUPPORTED.
Pin everything. Judge model ID, prompt hash, temperature 0, and dataset version recorded with every run — a judge score without its judge version is unreproducible. Expect residual nondeterminism anyway; you will handle it with thresholds-plus-margins in CI, not by pretending it away.
Pairwise vs rubric-graded. Pairwise comparison (which of A/B is better?) is more sensitive for choosing between two prompt versions; rubric grading gives absolute, trackable-over-time signals. Production suites use rubric grading for gates and pairwise for bake-offs — and pairwise is where position bias lives, which is where the audit begins.
Write the judge rubric with the same person who did Module 5's open coding. The rubric is the taxonomy's failure-mode definition turned into a checkable question — if the judge rubric and the annotation guideline disagree, your judge and your humans are grading different criteria and no κ will save you.
Judges fail in known, measurable ways. You audit each with a controlled experiment on your own data — the audit is cheap, and running it is the difference between a calibrated instrument and a hopeful one.
Position bias — dual-order runs. In pairwise mode, judges can favor whichever answer is listed first (or last), independent of quality. The audit: take a set of answer pairs, judge each pair twice — A-then-B and B-then-A — and count verdict flips. A worked audit on 40 pairs from the support assistant:
| Verdict consistent across orders | Verdict flipped with order | |
|---|---|---|
| 40 pairs, judged twice each | 31 | 9 |
A 22.5% flip rate means roughly one pairwise verdict in five is an artifact of listing order — and if 8 of the 9 flips favored the first-listed answer, the direction is confirmed positional preference, not random noise. The production protocol follows directly: every pairwise comparison runs in both orders; agreement between orders is a verdict, disagreement is scored a tie. Ties cost you sensitivity, which is the honest price of a biased instrument — and a flip rate that worsens over judge-model upgrades is itself a regression to catch.
Verbosity bias — length-matched variants. Judges tend to reward longer answers. The audit: take passing answers, create padded variants with identical factual content plus ~40% more words of restatement and hedging, and run pairwise original-vs-padded in both orders. An unbiased judge ties; a judge that systematically prefers the padded variant is grading length. Countermeasures: put "length is not a virtue; penalize padding" in the rubric explicitly, grade conciseness as its own binary criterion, and — for pairwise bake-offs between prompt versions — check whether one version's outputs are systematically longer before believing its win rate.
Self-preference. Models tend to rate their own family's outputs higher. The cheap control: use a judge from a different family than the system under test, or verify on a sample that two different judge families agree before standardizing on one.
Run the bias audit before the calibration below, not after. A judge with a 20% position-flip rate can still hit high agreement with humans on single-answer rubric grading — and then silently corrupt every pairwise bake-off you run with it. Audit per grading mode you intend to use.
The judge's own eval is agreement with human judgment on a gold calibration set — 100–200 traces graded by a domain expert against the same rubric, drawn stratified from your golden dataset so failure modes are represented, not just the easy majority class. Lab L8's hand-labeled traces are exactly this asset.
Raw percent agreement flatters the judge, because on an imbalanced criterion two graders agree often by chance alone. Correct with Cohen's κ. Worked, on a 100-trace calibration set for the faithfulness-style criterion:
| Judge: PASS | Judge: FAIL | Human total | |
|---|---|---|---|
| Human: PASS | 62 | 12 | 74 |
| Human: FAIL | 8 | 18 | 26 |
| Judge total | 70 | 30 | 100 |
Observed agreement p₀ = (62+18)/100 = 0.80. Chance agreement pₑ = (0.74×0.70) + (0.26×0.30) = 0.518 + 0.078 = 0.596. κ = (0.80 − 0.596)/(1 − 0.596) ≈ 0.50 — "80% agreement" is, chance-corrected, only moderate. And the off-diagonal cells are the judge's own error analysis: the judge passed 8 of the 26 human-failed traces — it misses roughly a third of true failures. For a gate, that miss rate on failures matters far more than headline agreement: a lenient judge is a gate that waves regressions through. Read the 8 misses individually (they are open-coding material for the judge prompt), tighten the rubric, re-run — and only promote the judge when both κ and the failure-catch rate clear the bar you set in advance.
Calibration is not a one-time ceremony. The standing protocol: route a 5–10% random sample of production judge verdicts to human re-grade every week, track agreement on a simple control chart, and treat a sustained drop as a judge incident — the usual causes being a silent judge-model update, drift in input distribution, or a rubric that new failure modes have outgrown.
Given a judge confusion matrix, the assessment expects you to reach past raw agreement: compute κ, and check the judge's catch rate on human-labeled failures specifically. A judge with 90% agreement that misses half of all true failures is unusable as a CI gate.
Everything so far produces scores; CI turns scores into enforcement. The target behavior: a PR that touches a prompt, retriever config, or generation code automatically runs the eval harness, compares against the main-branch baseline, and blocks the merge when quality drops past a margin. Two tiers keep it affordable — the deterministic smoke suite on every push, the judge-backed full suite before merge.
A worked GitHub Actions config:
name: eval-gate
on:
pull_request:
paths: ["prompts/**", "retrieval/**", "app/generation/**", "evals/**"]
jobs:
smoke: # deterministic layer only — free, fast, every push
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: pip install -r evals/requirements.txt
- run: pytest evals/deterministic -q --dataset evals/golden/golden-v1.3.jsonl
# schema validity, number-grounding, citation format, refusal markers
full: # judge + Ragas metrics — gated, budgeted
needs: smoke
runs-on: ubuntu-latest
env:
JUDGE_MODEL: ${{ vars.JUDGE_MODEL }} # pinned ID, temperature 0
steps:
- uses: actions/checkout@v4
- name: Fetch baseline scores from main
uses: actions/download-artifact@v4
with: { name: eval-baseline, github-token: "${{ secrets.GITHUB_TOKEN }}" }
- run: python evals/run_full.py --dataset evals/golden/golden-v1.3.jsonl --out scores.json
- name: Gate on thresholds and deltas
run: |
python evals/gate.py scores.json baseline.json \
--min faithfulness=0.80 answer_refusal_on_unanswerable=1.00 \
--max-drop faithfulness=0.03 context_recall=0.03 \
--report-md gate_report.md
- name: Post per-slice diff to PR
run: gh pr comment ${{ github.event.number }} --body-file gate_report.md
env: { GH_TOKEN: "${{ secrets.GITHUB_TOKEN }}" }
The design decisions inside that config are the lesson:
full on a label (run-evals) or in the merge queue rather than every push.Off-the-shelf runners get you much of this for free — promptfoo ships a GitHub Action with assertion thresholds and PR comments, and DeepEval's pytest integration means the gate is just a failing test suite — but the gate logic above is worth owning even when a tool executes it, because thresholds, margins, and mandatory slices are your policy decisions.
The gate blocks on evidence, and evidence cuts both ways — demonstrate the gate with one PR that passes and one that fails, exactly as lab L10 requires. A gate that has never blocked anything is a decoration, not a control.
"Hallucination rate" only becomes an engineering quantity when you define the unit. Answer-level scoring (Module 5's faithfulness) tells you whether an answer contains unsupported content; production debugging and user-facing risk need claim-level verification: decompose the answer into atomic claims, verify each against the retrieved context, and attribute supported claims to the exact span that supports them. The span attribution is what turns a score into a UI affordance (inline citations), a debugging tool (which chunk supported the wrong claim?), and an audit trail.
A worked claim table for trace-17's schema question:
| Claim | Verdict | Supporting span |
|---|---|---|
events_clean has an event_id column | SUPPORTED | schema-events.md#c2, "event_id STRING NOT NULL" |
events_clean has a user_ip column | UNSUPPORTED | — (no schema doc retrieved contains it) |
Table is partitioned by event_date | SUPPORTED | schema-events.md#c3 |
| PII columns are masked at ingest | UNSUPPORTED | — (true of events_raw per docs, asserted here for events_clean) |
Two unsupported claims out of four — and the second one illustrates why claim-level beats answer-level: the claim is almost right, borrowed from an adjacent table, and an answer-level judge skimming for topical grounding would likely pass it. Keep the distinction between faithfulness (supported by retrieved context — what you can measure automatically) and factuality (true in the world — which requires trusted external reference and is mostly out of automated reach); your metric is faithfulness, and your corpus-freshness discipline from M4 is what keeps faithful answers factual.
Two slices complete the measurement. The abstention slice: on Module 5's unanswerable rows, measure the rate at which the system correctly declines versus confidently answers — a hallucination metric that needs no claim decomposition at all. And hallucination under pressure: measure faithfulness separately on the slices where context is weakest (low context-recall cases), because that is where generators fill gaps — a system's hallucination rate on well-retrieved questions flatters it. Ragas faithfulness and Phoenix's hallucination evals both implement the claim/groundedness machinery; what you own is the slicing and the thresholds.
Judges are calibrated against humans — which quietly assumes human labels are reliable. They aren't, until you engineer them. Annotation ops is that engineering, and it is bread-and-butter measurement theory.
Interpret κ bands with judgment, not tables — the acceptable κ for "answer is polite" is not the acceptable κ for "answer is safe to ship to a regulated customer." Set the bar per criterion, in writing, before the pilot.
Offline evals gate the merge; only online experiments measure what users experience. This is the module's closing loop and your statistical home ground — with four LLM-specific twists.
The end state ties the module together: a change ships only through both gates — offline CI on the golden set, then online experiment with guardrails — and each gate catches what the other can't. Offline catches regressions before users see them; online catches what your golden set doesn't yet contain, and the failures it surfaces flow back through error analysis into next quarter's golden rows.
Your experimentation background is the differentiator here — most teams shipping LLM features can prompt; few can size an experiment around a noisy judge-scored metric, veto on guardrails, and pre-register stopping rules. That combination is precisely what the senior interview probes.
Your first 15 minutes Monday: before you let any LLM judge near a ship decision, hand-label 20 traces it will grade and diff its verdicts against yours — you're not looking for headline agreement, you're counting how many human-failed traces the judge let pass. A lenient judge is a gate that waves regressions through, and that miss rate is the number that decides whether it's usable.
Check yourself: Your LLM judge agrees with human labels 90% of the time. A colleague says that's high enough to gate merges on. What do you check before you agree, and why can 90% still be unusable?
The playbook library for this module packages the judge rubric template with the reasoning-first output schema, the bias-audit protocol card (dual-order and length-matched procedures with scoring sheets), the κ calculation worksheet with the confusion-matrix template, the CI gate config with the floors-plus-deltas policy file, the claim-verification prompt with span-attribution output, and the online-experiment design one-pager. Do this at work this week → pb-36 Calibrated LLM-as-Judge: Rubrics, Bias Audits, and Human Agreement (Neha's κ-before-you-trust-it protocol), pb-37 Code-Based Evals and CI Regression Gates for LLM Changes (the floors-plus-deltas merge gate), and pb-39 Online A/B Experiments for LLM Features: Your Stats Background Is the Edge (sizing around a noisy judge-scored metric). Two graded labs anchor it: L9 — calibrated LLM-as-judge (build the rubric judge, run both bias audits, calibrate against 100 human labels, and report agreement with the judge↔human table) and L10 — evals in CI (wire L8's golden set and L9's judge into GitHub Actions and demonstrate one passing and one blocked PR). Do them in order — L10's gate is only as trustworthy as the judge L9 calibrated, which is the whole point.