Build a rubric-based LLM judge for support-bot answers, then do what most teams skip: prove the judge can be trusted. You will audit it for position bias (dual-order pairwise runs) and verbosity bias (length-matched variants), calibrate it against human labels, and report judge-to-human agreement with Cohen's kappa.
An uncalibrated judge is a random-number generator with confident prose — this lab is what separates "we use LLM-as-judge" from "we have a measurement instrument."
Advanced · ~90 minutes hands-on · API spend under $2.
You need Python 3.11+, an Anthropic API key, and:
mkdir lab-09 && cd lab-09 && python -m venv .venv && source .venv/bin/activate
pip install anthropic pydantic
export ANTHROPIC_API_KEY="sk-ant-..."
If you completed lab 08: copy your golden_v1.jsonl into this folder. Each item needs id, question, context, answer, human_label (pass/fail) — rename fields if yours differ.
If you skipped lab 08, generate the fallback golden set (60 labeled support-bot answers over ten fictional SaaS policies, fixed seed so everyone gets identical data). Save as make_golden.py and run it:
import json, random
random.seed(7)
TOPICS = [
("refund window", "Refunds are available within 14 days of purchase on Pro and Team plans. Free workspaces are not eligible for refunds.",
"How long do I have to request a refund?", "14 days from purchase, on Pro and Team plans", "30 days on any plan, including Free"),
("SSO availability", "SAML SSO is available on the Team plan only. Pro and Free plans support Google sign-in but not SAML.",
"Does the Pro plan include SAML SSO?", "no — SAML SSO is Team-plan only; Pro supports Google sign-in", "yes, SAML SSO is included on Pro"),
("data export", "Workspace admins can export all data as CSV or JSON. Exports are generated within 24 hours and download links expire after 7 days.",
"How long is an export download link valid?", "7 days after the export is generated", "links never expire once generated"),
("seat billing", "Adding a seat mid-cycle is prorated to the day. Removing a seat takes effect at the next renewal; no partial credits are issued.",
"If I remove a seat mid-cycle, do I get a credit?", "no — removal takes effect at next renewal with no partial credit", "yes, a prorated credit is applied immediately"),
("API limits", "The API allows 600 requests per minute per workspace. Exceeding the limit returns HTTP 429 with a Retry-After header.",
"What is the API rate limit?", "600 requests per minute per workspace", "60 requests per minute per user"),
("retention", "Deleted records are held in trash for 30 days, after which they are permanently purged and cannot be recovered.",
"Can support restore a record deleted 45 days ago?", "no — records are purged permanently after 30 days in trash", "yes, support can restore anything within 90 days"),
("uptime SLA", "The uptime SLA is 99.9% for Team plans, measured monthly. Credits are 10% of the monthly fee per 0.1% below target.",
"What SLA credit applies if uptime drops 0.2% below target?", "20% of the monthly fee (10% per 0.1% shortfall)", "a full month's fee is refunded"),
("trial length", "Trials last 21 days and include all Team features. No credit card is required to start a trial.",
"Do I need a credit card to start a trial?", "no — trials start without a card and run 21 days", "yes, a card is required and trials last 14 days"),
("2FA policy", "Two-factor authentication can be enforced workspace-wide by an admin. Enforcement gives existing members a 7-day grace period.",
"What happens to members when an admin enforces 2FA?", "they get a 7-day grace period to enroll", "they are locked out immediately until they enroll"),
("storage", "Each workspace includes 100 GB of storage on Pro and 1 TB on Team. Additional storage is billed at $5 per 100 GB per month.",
"How much storage does the Team plan include?", "1 TB, with extra storage at $5 per 100 GB/month", "unlimited storage at no extra cost"),
]
PASS_T = ["Per our policy: {f}.", "{F}. Let me know if anything else is unclear.", "Short answer: {f}."]
FAIL_T = ["Per our policy: {f}.", "{F}. Let me know if anything else is unclear.", "Short answer: {f}."]
rows, i = [], 0
for name, ctx, q, fact, wrong in TOPICS:
for t in range(3):
for text, label in [(fact, "pass"), (wrong, "fail")]:
tmpl = (PASS_T if label == "pass" else FAIL_T)[t]
rows.append({"id": f"g{i:03d}", "topic": name, "question": q, "context": ctx,
"answer": tmpl.format(f=text, F=text[0].upper() + text[1:]),
"human_label": label})
i += 1
random.shuffle(rows)
with open("golden_v1.jsonl", "w") as f:
for r in rows: f.write(json.dumps(r) + "\n")
print(len(rows), "items")
The human_label field plays the role of your 100 hand-labeled traces — in a real engagement you produce these yourself (lab 08) or with a domain expert.
The judge defaults to claude-opus-4-8. While iterating on plumbing you can export JUDGE_MODEL=claude-haiku-4-5 — a deliberate cost decision; run the final calibration on the stronger judge.
Write the rubric judge. A calibrated judge grades against explicit criteria, not vibes. Create judge.py:
import os, json, anthropic
from pydantic import BaseModel
from typing import Literal
client = anthropic.Anthropic(max_retries=3)
MODEL = os.getenv("JUDGE_MODEL", "claude-opus-4-8")
class Verdict(BaseModel):
quote: str # the context sentence the answer stands or falls on
reasoning: str
verdict: Literal["pass", "fail"]
RUBRIC = """You grade a support bot's answer against the policy context.
FAIL if any criterion fails:
1. Grounded: every factual claim is supported by the context. A claim the context
contradicts, or that the context does not cover, is a fail.
2. Responsive: the answer actually addresses the question asked.
3. No invented specifics: numbers, durations, and plan names must match the context.
Length, politeness, and formatting are NOT quality. A short correct answer passes;
a long fluent wrong answer fails. First find the deciding quote, then reason, then verdict."""
def judge(question: str, context: str, answer: str) -> Verdict:
resp = client.messages.parse(
model=MODEL, max_tokens=512,
system=RUBRIC,
messages=[{"role": "user", "content":
f"<context>{context}</context>\n<question>{question}</question>\n<answer>{answer}</answer>"}],
output_format=Verdict,
)
if resp.parsed_output is None:
raise ValueError(f"no parsed output ({resp.stop_reason})")
return resp.parsed_output
The forced quote field is a cheap grounding trick: a judge that must cite the deciding sentence hallucinates verdicts less.
Run it and calibrate against human labels. Create calibrate.py — judge all 60 items, then compute raw agreement and Cohen's kappa:
import json
from judge import judge
items = [json.loads(l) for l in open("golden_v1.jsonl")]
human, machine = [], []
for it in items:
v = judge(it["question"], it["context"], it["answer"])
human.append(it["human_label"]); machine.append(v.verdict)
print(it["id"], it["human_label"], "->", v.verdict)
def kappa(a, b):
n = len(a); po = sum(x == y for x, y in zip(a, b)) / n
pa = sum(x == "pass" for x in a) / n; pb = sum(x == "pass" for x in b) / n
pe = pa * pb + (1 - pa) * (1 - pb)
return (po - pe) / (1 - pe) if pe < 1 else 1.0
agree = sum(h == m for h, m in zip(human, machine)) / len(human)
print(f"agreement={agree:.2%} kappa={kappa(human, machine):.3f}")
Record both numbers. On this synthetic set expect agreement above 0.9 and kappa above 0.8; on real traces 0.8/0.6 is a respectable start. Read every disagreement — sometimes the judge is right and your human label is wrong; fix the label, note it, that is calibration working.
Position-bias audit (dual-order runs). Pairwise judges systematically prefer one slot. Build a pairwise judge (verdict: Literal["A", "B"], same rubric) and run all 30 same-question pass/fail pairs twice — (A=pass, B=fail) and (A=fail, B=pass). Score two numbers: accuracy (did it pick the grounded answer?) and swap consistency (same winner regardless of order). Consistency below ~0.9 means position bias; the standard mitigation is to always run both orders and accept only agreeing verdicts, escalating disagreements to a human.
Verbosity-bias audit (length-matched variants). Pad every fail answer to roughly 3x length with content-free politeness ("Thanks so much for reaching out — great question! ..." plus a restatement), changing no facts. Re-run the rubric judge on the padded fails and compute the flip rate: how often a wrong answer now passes because it got longer. Do the same for padded passes (flip to fail). Anything above ~10% is a verbosity bias you must prompt against ("length is not quality" is already in the rubric — strengthen it or add a matched-length instruction) and re-measure.
Write the bias-audit table. One markdown table in bias_audit.md: agreement, kappa, pairwise accuracy per order, swap consistency, verbosity flip rates (fail-to-pass and pass-to-fail), plus one paragraph on what you changed and the before/after numbers if you re-prompted.
Freeze the judge. Commit judge.py with a version comment (JUDGE_V=1.1, kappa=0.87 on golden_v1). Labs 10 and 15 import this exact judge — its calibration numbers are what make its downstream verdicts meaningful.
A judge is a model, so it gets a validation set. Agreement with humans, invariance under answer order, and invariance under padding are the minimum bar before its scores gate anything.
calibrate.py prints agreement and kappa over all 60 items; kappa ≥ 0.8 on the fallback set.judge.py (versioned rubric judge) + bias_audit.md (table plus mitigation notes) + the agreement/kappa numbers. Self-grade (pass at 4+):
parsed_output is None: check resp.stop_reason — max_tokens means bump the limit; a refusal means inspect the item.nan or wildly low despite high agreement: kappa collapses when one class dominates; check your pass/fail balance and report both metrics.time.sleep(0.5) between calls if needed.On client engagements the golden set is built from client traces, so it cannot touch a public endpoint without sign-off: point the same anthropic client at the client's sanctioned deployment (Amazon Bedrock via AnthropicBedrockMantle, or their gateway base URL) and keep golden_v1.jsonl, judge outputs, and the audit table inside the client tenancy. The judge prompt and audit methodology are your portable IP — the labeled data is theirs. If even a hosted enterprise endpoint is off the table, the same harness runs against a self-hosted open-weight judge (your lab 14 vLLM endpoint) — re-run the full calibration, because judge quality drops and the kappa number is the honest disclosure of by how much.