Lab 13 · Calibrated Judge for a Release Gate

A judge that gates releases is a test oracle, and an uncalibrated oracle is worse than none: it stamps PASS on regressions with the same confidence it stamps anything else. Priya's team wired a judge into CI in an afternoon and spent a quarter discovering it preferred long answers to correct ones. The discipline this lab drills is the calibration protocol: human labels first, judge second, agreement measured, biases diagnosed with controlled experiments, rubric fixed, agreement re-measured — and only past a numeric threshold does the judge earn gate authority. You will run the whole protocol on a support-answer quality judge, using traces we provide and labels you create.

Time, tier & cost

~120 minutes hands-on · Advanced tier (maps to module M7 and playbook PB-40). Cost: $0–2 (judge calls; a strong model is worth the rupees here — a weak judge fails calibration honestly, which is also instructive).

Setup

mkdir -p qa-lab-13 && cd qa-lab-13
python3 -m venv .venv && source .venv/bin/activate
pip install requests pyyaml scikit-learn

traces.yaml — 12 production-style traces (question, policy-correct answer summary, and the actual bot answer). Abbreviated fields shown; author all 12 rows as given:

# policy: refunds need photo + report within 48h of delivery;
# delivery 6am-11pm; KiranaPlus Rs 99/mo = free delivery + 5% cashback
- id: tr-01
  q: "Damaged atta bag delivered today, what do I do?"
  a: "So sorry! Please share a photo of the damaged bag and we'll process
      your refund - just make sure it's within 48 hours of delivery."
- id: tr-02
  q: "Refund for the eggs from last Tuesday?"
  a: "Of course! I've noted your refund request for the eggs and our team
      will process it within 3-5 business days."
- id: tr-03
  q: "Do you deliver at 5am?"
  a: "Our delivery window is 6am to 11pm, so the earliest slot would be
      6am. Would you like me to find the first slot tomorrow?"
- id: tr-04
  q: "Is KiranaPlus worth it?"
  a: "KiranaPlus is our premium membership program designed to reward our
      most valued customers. For a monthly fee you unlock a suite of
      benefits including complimentary delivery on every single order you
      place with us, regardless of cart size, plus an ongoing 5% cashback
      on all purchases which accumulates in your wallet automatically.
      Many customers who order more than twice a month find that the
      membership pays for itself very quickly. The fee is Rs 99 per month."
- id: tr-05
  q: "Milk was sour, refund pls"
  a: "Sorry to hear that! Refund initiated."
- id: tr-06
  q: "What are your delivery hours on Diwali?"
  a: "We deliver from 6am to 11pm every day including festivals, and you
      can schedule any slot you like on Diwali."
- id: tr-07
  q: "How do I cancel KiranaPlus?"
  a: "You can cancel anytime from Account > Memberships > Cancel. Your
      benefits continue until the end of the billing month."
- id: tr-08
  q: "Wrong item delivered - got dish soap instead of hand soap."
  a: "Apologies! Please share a quick photo of the item received and we'll
      arrange a refund or replacement within 48 hours of delivery."
- id: tr-09
  q: "Delivery fee kitna hai?"
  a: "Delivery fee Rs 29 hai, aur Rs 499 se upar ke cart par free delivery
      milti hai."
- id: tr-10
  q: "Can I get cashback in cash instead of wallet?"
  a: "Great question! Cashback is one of the most popular KiranaPlus
      benefits. The 5% cashback is credited to your KiranaCart wallet and
      can be used on any future order, which most customers find even more
      convenient than cash since it applies automatically at checkout.
      Currently we don't offer bank transfers of cashback amounts."
- id: tr-11
  q: "App charged me twice for one order!"
  a: "That sounds frustrating. Double charges usually auto-reverse within
      24 hours. If it doesn't, please share your order ID and I'll
      escalate to our payments team right away."
- id: tr-12
  q: "Are your onions fresh?"
  a: "Absolutely! All our produce is sourced daily from local farms and
      goes through a three-stage freshness check before dispatch."

rubric-v1.txt — the starter judge rubric (seeded; apply verbatim in step 3):

You are evaluating a support bot's answer for release readiness.
Score PASS or FAIL.

An answer PASSES if it is helpful and policy-compliant. Thorough,
detailed answers that fully explain the relevant policies and options
demonstrate higher quality. Short answers that skip explanation
usually indicate low effort.

When comparing against the ideal, the first consideration is whether
the answer covers everything the ideal answer would contain.

Question: {q}
Bot answer: {a}
Verdict (PASS/FAIL):

Steps

1. 1Hand-label the gold slice first — no judge in sight. For each trace, decide PASS/FAIL yourself against the policy comment at the top of traces.yaml, and write one sentence of reasoning per label into labels.yaml. Do this before any model sees anything: your labels are about to become the measuring stick, and a stick calibrated after the fact bends toward the judge. Traps to label carefully: tr-02 (48-hour window — last Tuesday is outside it, and no photo was requested), tr-05 (refund granted with no photo and no window check), tr-06 (policy says no scheduled slots on public holidays), tr-12 (freshness claims appear nowhere in policy — the bot is inventing).
2. 2Compute your label distribution. Count PASS vs FAIL. The seeded set is designed to land near 7 PASS / 5 FAIL; if you got 11/1 either way, re-read the policy line by line — the traces reward strictness. Freeze labels.yaml with a git commit; post-hoc label edits are the calibration sin the commit history exists to catch.
3. 3Run the v1 judge and measure agreement. Write judge.py: for each trace, fill rubric-v1.txt, call your judge model, parse PASS/FAIL, and write judge-v1.yaml. Then compute raw agreement and Cohen's kappa against your labels (sklearn.metrics.cohen_kappa_score). Record both in agreement-report.md under "v1." Expected: mediocre — the seeded biases misclassify a specific, diagnosable subset.
4. 4Diagnose disagreements with a per-trace autopsy. For every disagreement, log: your label, judge verdict, and the judge's implicit reason (rerun those traces asking the judge to explain). Look for the two seeded patterns: verbose-but-wrong answers passing (does tr-04's flattering wall of text pass despite burying the fee? does tr-10 pass while tr-05-style brevity fails?), and short-but-correct answers failing (tr-03, tr-09 are terse and fully correct — did either fail "low effort"?).
5. 5Confirm each bias with a controlled experiment, not vibes. Verbosity: take tr-05's terse answer and pad it with three sentences of warm fluff that add zero policy content; rerun — if the verdict flips FAIL→PASS on padding alone, verbosity bias is confirmed. Position/anchoring: the rubric's "first consideration is whether the answer covers everything the ideal would contain" anchors the judge on coverage-vs-ideal before correctness; test by asking the judge two orders — correctness-first vs coverage-first — on tr-04 and recording whether verdicts differ. Log both experiments with outputs in agreement-report.md.
6. 6Rewrite the rubric.rubric-v2.txt fixes: (a) strike the thoroughness/effort language — replace with "judge factual and policy correctness only; length is irrelevant; an answer that grants anything policy forbids, omits a required condition (photo, 48-hour window), or asserts facts not in policy FAILS"; (b) reorder criteria so hard policy violations are checked first and are individually sufficient to FAIL; (c) require the judge to output a one-line cited reason before the verdict (reason-then-verdict measurably stabilizes judgments — and gives you an audit trail); (d) include the policy facts verbatim in the rubric so the judge stops judging from its own imagination.
7. 7Re-measure, and gate the judge itself. Run v2 over all 12, compute agreement and kappa, and add the before/after table to agreement-report.md. Your pre-registered bar: ≥90% raw agreement (≥11/12) and kappa ≥0.75. Clear it and write the release line: "Judge v2 is authorized to gate; disagreement autopsies attached." Miss it and iterate v3 — the point of a threshold is that failing it has consequences.
8. 8Wire the earned gate. Express the calibrated judge as a Promptfoo llm-rubric assert (or Langfuse evaluator) referencing rubric-v2.txt, over the 12-trace golden set. Add the recalibration clause to the report: the judge re-runs this calibration whenever the rubric, judge model, or policy changes, and quarterly regardless — a judge is a model artifact and drifts like one.

Verify

• •
  labels.yaml committed before any judge run, 12 labels each with one-line reasoning.
• •
  v1 agreement and kappa recorded; every disagreement has an autopsy line.
• •
  Both bias experiments logged with actual model outputs (padding flip test; criteria-order test).
• •
  rubric-v2.txt implements all four fixes, including verbatim policy facts and reason-before-verdict.
• •
  Before/after table shows v2 ≥90% agreement and kappa ≥0.75, or an honest v3 iteration.
• •
  Recalibration triggers written into the report.

Self-grading rubric (pass = 8/12):