An uncalibrated judge is a random number generator with gravitas. This playbook designs the two-plane LLM-as-judge architecture — an offline CI judge gating deploys, an online judge scoring sampled production traffic — with human-label calibration before any metric is trusted and a different model family judging than generating.
Tier: Advanced · Hands-on time: ~40 minutes
"LLM-as-judge evaluation" is verbatim language from the Amazon solutions-architect posting this track is built on, and the 2026 practitioner consensus has converged on a recognizable shape: hybrid metric-plus-judge-plus-human-triage systems, calibration sets of 500+ human labels before metrics are trusted, pairwise comparisons run in both orderings to cancel position bias, distilled cheap judges scoring 100% of traffic at roughly 1/30th the cost per practitioner reports, and a judge drawn from a different model family than the generator. The architect's job is to turn that consensus into a system design with gates, owners, and known failure modes.
Design the rubric before the judge. Each dimension gets level descriptions concrete enough that two humans agree most of the time — if humans can't agree on what a 2 versus a 4 looks like, no judge can be calibrated against their labels. Pilot the rubric on 20 outputs with two human raters and fix the dimensions where they diverge.
Build the calibration set as the first-class asset. Collect representative outputs (including deliberately bad ones), have humans label them against the rubric — the practitioner bar is 500+ labels for trustworthy calibration, so budget accordingly and grow toward it — and split into a tuning set (for judge prompt iteration) and a held-out set (for honest agreement measurement).
Draft the judge prompt and measure agreement, iteratively. Anita, building the judge for orbit-docs-copilot:
Draft an LLM judge prompt for scoring RAG copilot answers on our rubric
(attached: 4 dimensions, 1-5 levels with descriptions, 12 worked
examples from our calibration set).
Requirements:
- Score each dimension separately with a one-sentence justification
citing the specific answer text; no overall vibe score.
- The judge sees: question, retrieved context, answer. Faithfulness is
judged ONLY against the provided context, not world knowledge.
- Output as strict JSON matching the attached schema.
- Include explicit instructions against known judge biases: no reward
for length, no reward for confident tone, no penalty for honest
"the context does not cover this" refusals.
I will run this on a held-out human-labeled set and report agreement;
propose what to adjust if agreement is weak on specific dimensions.
Iterate on the tuning set only; report final agreement (per-dimension correlation or kappa against humans) from the held-out set. Below-bar agreement on a dimension means the rubric or prompt changes — not that the number ships anyway.
Design the offline plane as a CI gate. Every prompt, retrieval, or model change runs the judge over the golden set (pb-33); the merge blocks on regression past thresholds you set from the calibration work. For A/B-style comparisons, use pairwise judging with both orderings — position bias is a documented judge failure and running both directions cancels it.
Design the online plane with a cost-tiered judge. Sampled production traffic gets scored asynchronously; where 100%-traffic scoring is wanted, the practitioner pattern is a distilled or smaller judge (reported at roughly 1/30th the cost) with periodic agreement checks against the full judge. Scores land in the trace layer (Langfuse-class), feeding drift alerts and the pb-33 loop. Low-scoring samples route to human triage — the judge flags; humans still adjudicate the tail.
Keep the judge and generator in different families. Same-family judging risks correlated blind spots and self-preference; the consensus practice is a different model family for the judge, re-verified whenever either side is upgraded. Record the pairing in the design and re-run calibration on judge upgrades — a new judge model is a new judge.
Define the evals-to-guardrails path and ship the design. Persistent, well-calibrated failure patterns graduate from post-hoc scoring into pre-response guardrails where latency allows. The deliverable: two-plane diagram, rubric with agreement numbers, calibration-set ownership and refresh plan, CI gate thresholds with override rules, sampling and triage design, and the judge/generator pairing ADR.
Calibration sets built from production outputs contain whatever the production data contains — under client MSAs, sanitize or synthesize before any labeling tool or external judge sees them, and prefer client-tenancy judge endpoints (Bedrock or Azure OpenAI in the client subscription). The different-family rule still applies inside a single cloud: pick two model families available on the client's approved platform. Human labeling stays with client-cleared staff for confidential content. Fully local: promptfoo plus an Ollama-served judge model runs the whole two-plane pattern offline at reduced judge quality — state the quality trade-off in the design, and lean harder on human triage of the tail.
Calibrate before you trust — held-out human agreement, both-orderings pairwise, different judge family, a CI gate that has actually blocked a seeded regression — and let the cheap judge scale only as far as its audits reach.