Lab 12 · LLM-as-Judge Eval Architecture

The change that motivates this lab looks harmless. A well-meaning engineer at Orbit edits the orbit-docs-copilot prompt to make answers "snappier," ships it on vibes, and for two weeks the copilot confidently invents SSO capabilities Orbit does not have — until a customer escalation lands on Anita's desk. Her post-incident design decision is the industry's: prompt changes are deploys, deploys need gates, and the gate is an eval. You will build the offline plane (golden set + CI judge) end to end, calibrate the judge against your own hand labels before trusting it, and design — on paper — the online plane that samples production. The seeded "bad" prompt is provided; your judge must catch it.

Time & tier

~120 minutes hands-on · Advanced tier (maps to module M7). ₹0 path: generator and judge both run locally via Ollama (different model families). Primary path: local generator + Claude as the judge (small API spend, stronger grading).

Setup

mkdir -p lab-12/goldens && cd lab-12
python -m venv .venv && source .venv/bin/activate
pip install anthropic ollama pydantic
ollama pull llama3.2          # generator (Llama family)
ollama pull qwen2.5:7b        # ₹0 judge (Qwen family — deliberately different)
# Primary judge path: export ANTHROPIC_API_KEY=...

Create prompt_v1.txt — the current production prompt, the behavior your golden set will encode:

You are the Orbit docs copilot. Answer the user's question using ONLY the
provided context. Quote or closely paraphrase the context for every claim.
If the context does not contain the answer, reply exactly:
"I don't have that in the Orbit docs - please contact support."
Keep answers under 120 words.

And prompt_v2.txt — the seeded regression you will test later. Read it now and notice how reasonable it sounds in a pull request:

You are the Orbit docs copilot. Be direct and snappy: answer in one short
sentence. If the context does not fully cover the question, use your general
knowledge of SaaS products to give your best guess - customers hate
"I don't know". Never mention the context or your limitations.

Guided steps

Step 1 — Build the app under test (15 min)

Create app.py. The generator is local (Llama family) so the ₹0 path is default; the prompt template is a runtime argument because the prompt is the artifact under version control and under test:

# app.py — the system under test: grounded Q&A over provided context
import ollama

def answer(question: str, context: str, prompt_path: str) -> str:
    system = open(prompt_path).read()
    user = f"Context:\n{context}\n\nQuestion: {question}"
    r = ollama.chat(model="llama3.2",
                    messages=[{"role": "system", "content": system},
                              {"role": "user", "content": user}])
    return r["message"]["content"].strip()

if __name__ == "__main__":
    ctx = "Orbit SSO supports SAML 2.0 and OIDC. SCIM provisioning is in beta."
    print(answer("Does Orbit support SCIM?", ctx, "prompt_v1.txt"))

Run it once with v1 and once with v2 on a question the context does not answer ("What is Orbit's uptime SLA?") and eyeball the difference — v1 refuses, v2 guesses.

✅ Checkpoint: v1 produces the exact refusal string on the SLA question; v2 produces a confident invented answer.

Step 2 — Generate the golden set, then calibrate 10 items by hand (25 min)

Create goldens/template.jsonl with this item shape, then use your assistant to help you write 30 items about a fictional product ("Orbit"), spread across four categories:

{"id": "g01", "category": "direct", "must_refuse": false,
 "context": "Orbit tenants can be pinned to the Mumbai or Frankfurt region.",
 "question": "Which regions can my Orbit tenant be pinned to?",
 "reference": "Mumbai or Frankfurt; the answer must come only from the context."}

Category quota: 10 direct (answer is one fact in context) · 8 multi (answer combines two context facts) · 7 refuse (context lacks the answer; must_refuse: true) · 5 near_miss (context contains a similar but different fact — e.g., context says SCIM is in beta, question asks if SCIM is generally available). The near-miss items are the hard cases that separate a real judge from a keyword matcher.

Generation prompt to your assistant: paste the template and quotas and ask for candidates — then you review all 30 and hand-write the expected behavior for 10 of them (2–3 per category) in a file goldens/hand_labels.md before ever running the judge. These 10 are your calibration set: your ground truth, written blind.

✅ Checkpoint: goldens/golden.jsonl has 30 items meeting the category quotas; hand_labels.md has your blind verdicts for 10 of them.

Step 3 — Build the judge with a structured verdict (25 min)

Create judge.py. Primary path: Claude judges (Anthropic family ≠ Llama generator) with a Pydantic-typed verdict; ₹0 path: Qwen judges via Ollama's structured output. Same schema either way:

# judge.py — cross-family judge, structured verdict
from pydantic import BaseModel

class Verdict(BaseModel):
    faithful: bool           # every claim supported by the context
    refused_correctly: bool  # refused if and only if context lacks the answer
    reasoning: str           # 2-3 sentences citing the evidence

RUBRIC = """Grade a support copilot's answer. Be strict.
Context: {context}
Question: {question}
Expected behavior: {reference} (must_refuse={must_refuse})
Candidate answer: {answer}
faithful = every factual claim is supported by the Context alone.
refused_correctly = it refused exactly when the Context lacks the answer."""

def judge_claude(item: dict, answer: str) -> Verdict:
    import anthropic
    client = anthropic.Anthropic()
    r = client.messages.parse(
        model="claude-opus-4-8", max_tokens=500,
        thinking={"type": "adaptive"},
        messages=[{"role": "user",
                   "content": RUBRIC.format(**item, answer=answer)}],
        output_format=Verdict)
    return r.parsed_output

def judge_ollama(item: dict, answer: str) -> Verdict:   # ₹0 path (Qwen family)
    import ollama
    r = ollama.chat(model="qwen2.5:7b",
                    messages=[{"role": "user",
                               "content": RUBRIC.format(**item, answer=answer)}],
                    format=Verdict.model_json_schema())
    return Verdict.model_validate_json(r["message"]["content"])

✅ Checkpoint: judging one hand-labeled item returns a Verdict object whose reasoning quotes or references the context.

Step 4 — Calibrate before you trust (15 min)

Run the generator (v1) plus the judge over your 10 calibration items and compare the judge's verdicts to hand_labels.md. Count agreements. The working threshold this lab uses: ≥8/10 agreement on pass/fail before the judge earns gate authority. Where you disagree, decide who is wrong — sometimes it is you (your hand label missed a subtle unfaithfulness), sometimes it is the judge (tighten RUBRIC and re-run). Record every disagreement and its resolution in calibration.md. This miniature is standing in for the production practice of large human-labeled calibration sets — small set, same discipline.

✅ Checkpoint: calibration.md records ≥8/10 agreement (after at most one rubric revision) with each disagreement resolved in writing.

Step 5 — Wire the gate and pass v1 (10 min)

Create gate.py: run all 30 goldens through app.py with the candidate prompt, judge each answer, and compute the pass rate (an item passes when faithful and refused_correctly). Gate policy: exit 0 if pass rate ≥ 0.90, else exit 1 and print the failing items. This is your CI contract — in a real pipeline this script runs on every PR that touches prompt_*.txt (promptfoo gives you the same shape with YAML instead of Python; use it if you prefer). Run python gate.py prompt_v1.txt and record the baseline pass rate.

✅ Checkpoint: v1 gates green with its pass rate and per-category breakdown printed; output saved to runs/v1_pass.log.

Step 6 — The seeded regression gets blocked (15 min)

Run python gate.py prompt_v2.txt. Watch which categories collapse: refuse items fail because v2 forbids saying "I don't know," and near_miss items fail because one-sentence "snappy" answers flatten the beta-vs-GA distinction. Save the run as runs/v2_blocked.log — this file, showing a non-zero exit and named failing items, is the artifact that makes the architecture argument for you: the regression was invisible to code review and obvious to the gate.

✅ Checkpoint: v2's pass rate lands well below 0.90, the gate exits non-zero, and the log shows refuse and near_miss as the collapsed categories.

Step 7 — Design the production plane on paper (15 min)

The CI plane catches regressions you can anticipate; production catches the ones you cannot. Write production-sampling.md (one page): sample ~5% of live traffic (100% is unaffordable with a frontier judge — note the distilled-judge option: train/prompt a small model against your calibrated judge's verdicts for cheap full-traffic scoring); judge asynchronously off the request path; alert on a 7-day rolling faithfulness drop of >5 points (synthetic threshold — state your own and justify it); define trace retention and PII handling for stored question/context/answer triples; and close the loop — production failures graduate into new golden items, which is how the golden set stays adversarial as usage drifts.

✅ Checkpoint: production-sampling.md covers sampling rate, async placement, alert threshold, PII/retention, and the failure-to-golden feedback loop.

Verification

• Golden set30 items in four categories meeting the quotas, including 5 near-miss traps; 10 hand-labeled blind.
• Calibration≥8/10 judge–human agreement documented, disagreements resolved in writing.
• Gate behaviorv1 exits 0 at ≥0.90; v2 exits non-zero with failing items named; both logs saved.
• Family separationgenerator and judge are from different model families on whichever path you ran.
• Online planethe production-sampling note exists and closes the failure-to-golden loop.

Completion artifact

The lab-12/ eval repo: app.py, both prompts, goldens/ (golden set + hand labels), judge.py, gate.py, calibration.md, runs/ (pass + blocked logs), production-sampling.md. Self-grading rubric (pass = 8/10):

• Blocked deploy (0–3): v2 blocked by the gate with category-level evidence in the log = 3.
• Calibration discipline (0–2): blind hand labels written before judging; agreement measured; disagreements adjudicated = 2.
• Golden-set quality (0–2): near-miss items genuinely near; refuse items genuinely unanswerable from context = 2.
• Cross-family judging (0–2): family separation implemented and the bias rationale stated in your own words = 2.
• Production design (0–1): sampling note is specific enough for a platform team to implement = 1.

Client-safe variant

The ₹0 path (Llama generator, Qwen judge, both via Ollama) sends nothing off-machine and runs inside any client network. For engagements: golden sets built from client documents are client data — store them in the client's repo, not yours; run the judge in the client-approved tenancy (Bedrock/Azure OpenAI — still honoring family separation from whatever generates); and treat production trace sampling as a data-protection decision requiring the client's sign-off on retention and PII redaction before the first trace is stored. The eval architecture — two planes, calibration, gate thresholds — is yours to reuse everywhere; the eval data never travels between clients.

Stuck?

• •
  Ollama structured output returns invalid JSON: retry once, and pin format=Verdict.model_json_schema() exactly — older Ollama versions need format="json" plus schema-in-prompt.
• •
  v1 fails the gate on refuse items: llama3.2 sometimes paraphrases the refusal string — judge on refusal behavior, not exact string match; loosen the rubric, not the prompt.
• •
  Judge agreement stuck at 6–7/10: your rubric is underspecified — add the failure definitions ("a hedged guess counts as not refusing") rather than re-rolling verdicts.
• •
  v2 sneaks past the gate: your golden set is too soft — check that refuse items truly have no answer in context and that near-miss items differ on a fact, not phrasing.
• •
  Claude call fails on parse: confirm the SDK is current and output_format=Verdict (not a JSON string); the verdict arrives on .parsed_output.

Answer key (don't read before attempting)

The seeded regression and the traps around it:

1. 1What v2 actually breaks. Two behaviors, both invisible to a reviewer skimming the diff: "use your general knowledge … give your best guess" destroys groundedness (fails faithful on refuse and near_miss items — the copilot now answers the SLA question with an invented number), and "never mention your limitations" outlaws the refusal path entirely (fails refused_correctly on all 7 refuse items). Expected v2 pass rate: roughly 50–70% depending on generator mood — decisively below the 0.90 gate.
2. 2Why the near-miss category exists. "SCIM is in beta" vs "SCIM is generally available" is one fact apart; v1's quote-the-context habit preserves the distinction, v2's one-sentence snappiness flattens it. A golden set without near-misses certifies keyword overlap, not faithfulness.
3. 3The self-preference trap you avoided. Same-family judging (Llama grading Llama, or Claude grading Claude) inflates scores precisely on stylistic-but-wrong answers — the v2 failure mode. Cross-family judging is the cheap structural defense; pairwise comparisons run in both orderings are the next layer when you compare prompts head-to-head.
4. 4The calibration step is the architecture. The gate's authority comes entirely from the measured judge–human agreement — skip it and the 0.90 threshold is theater. In production this becomes a standing human-labeled calibration set, re-measured whenever the judge model or rubric changes; the judge is itself a model deployment with its own regression risk.
5. 5What the one-pager should have caught. If your production note judges on the request path (latency), stores raw traces indefinitely (PII/retention), or samples 100% with a frontier judge (cost), those are the three standard design errors — the fixes are async, TTL+redaction, and sampling or a distilled judge, respectively.