Align an LLM Judge to Human Labels

Workflow area: Evals & AI Quality · Tier: Advanced · Hands-on time: ~45 minutes to learn, ~half a day to align

The seduction of LLM-as-judge is obvious: human grading doesn't scale, and a model can score ten thousand responses overnight. The trap is equally obvious once stated: you've replaced the question "is our AI good?" with "does our judge think our AI is good?" — and judges have systematic biases. They reward confident, polished, sycophantic answers; they under-punish subtle factual errors wrapped in good structure; they prefer longer responses. The only honest way to use a judge is to first measure it against human judgment — yours — and to know, in writing, the response types where it stays blind.

When to use — and when not to

• Use it for subjective dimensions at volumehelpfulness, tone, completeness, rubric adherence — where pb-37's deterministic assertions can't reach and human grading can't scale.
• Use it before any judge-scored number ships upwarda bake-off (pb-33) or launch gate (pb-40) scored by an unaligned judge inherits every bias of that judge, silently.
• Use it on a schedulerealign quarterly and after any judge-model change; judge behavior drifts across model versions like everything else.
• Skip judges where deterministic checks sufficefactual containment, format validity and citation resolution are cheaper and more trustworthy as string/regex/schema assertions; spend judges only on genuine subjectivity.
• Skip fully automating dimensions the alignment run proves the judge can't gradeif agreement stays low on a response type after iteration, that type keeps human review; the documentation of this boundary is a primary artifact, not an apology.

The recipe

1. 1.

   Build the labeled human sample first. Take 30–50 real outputs spanning quality levels (pb-36's coded traces are a head start) and label them yourself on the dimension in question, using a 3-level scale with written definitions — pass / borderline / fail beats 1–10, because humans can't be consistent at ten levels either. If two humans label independently and compare, disagreements sharpen the definitions before the judge ever sees them. Hide these labels from every judge run.

2. 2.

   Write the judge prompt like a rubric, not a vibe.

   Copy

   You are grading a support assistant's response for HELPFULNESS.
   
   Grade PASS if the response: resolves the user's actual question,
   is grounded in the provided context, and gives an actionable
   next step when one exists.
   Grade BORDERLINE if: partially resolves it, or resolves it with
   meaningful effort left to the user.
   Grade FAIL if: doesn't address the actual question, invents
   information, or is confidently wrong.
   
   Rules — these override your instincts:
   - Judge substance, not style. Polished-but-wrong is FAIL.
   - A short correct answer outranks a long padded one. Length
     is not quality.
   - Confidence is not correctness. Hedged-but-right beats
     assertive-but-wrong.
   - First state your grade, then TWO sentences of reasoning
     citing specifics from the response.
   
   [USER QUESTION] {{question}}
   [CONTEXT PROVIDED] {{context}}
   [RESPONSE TO GRADE] {{response}}
   

   The anti-bias rules aren't decoration — they target the judge's known failure modes: sycophancy-reward, length-reward, confidence-reward.

3. 3.

   Run the judge over the labeled sample and compute agreement. Score all 30–50 items, then build the confusion matrix in a sheet:

   Copy

   JUDGE vs HUMAN (n=40)              judge:PASS  BORDER  FAIL
   human: PASS  (18)                      16         2      0
   human: BORDERLINE (10)                  5         4      1
   human: FAIL  (12)                       3         2      7
   Raw agreement: 27/40 = 67.5% — NOT yet trustworthy.
   Worst cell: 3 human-FAILs judged PASS → read those three first.
   

   The off-diagonal cells matter more than the headline number — human-FAIL/judge-PASS is the dangerous cell, because those are the failures your judge will wave through at scale.

4. 4.

   Read every disagreement and iterate the judge — not your labels. For each mismatch, diagnose: is the judge fooled (sycophantic-but-wrong answer graded PASS — tighten the substance-over-style rule, add that example as an in-prompt anchor), or was your label wrong or your rubric ambiguous (fix the rubric, relabel deliberately, and note it)? Add the hardest two or three cases into the judge prompt as worked examples. Rerun. Two to four iterations typically moves agreement from the 60s to 85%+.

5. 5.

   Set the acceptance bar and document the blind spots. A common working bar: ≥85% raw agreement AND near-zero human-FAIL/judge-PASS on S1-severity content. Then write the trust memo — the artifact most teams skip:

   Copy

   JUDGE CARD — helpfulness-judge v4 (judge model: [model/version])
   Aligned: 2026-08 · Agreement: 88% (n=40) · FAIL→PASS leakage: 1/12
   TRUSTED FOR: routine helpfulness grading in CI and bake-offs.
   NOT TRUSTED FOR: terse-but-expert answers (judge under-grades),
   domain-specific factual subtleties (judge misses), any S1
   policy-accuracy call → these remain human-reviewed.
   Realign: quarterly, or on judge-model change.
   

6. 6.

   Deploy with drift guards. Wire the judge into pb-37's rubric assertions, keep the labeled sample as a canary set, and rerun alignment on it whenever the judge model or prompt changes — an 88%-aligned judge on last quarter's model version is an unvalidated judge today.

Make it yours

• promptfoo path: encode the judge as an llm-rubric grader and keep the labeled sample as a test file — alignment reruns become one command.
• Braintrust/Phoenix path: use built-in human-annotation queues to collect labels and the judge-evaluation views to track agreement over time — the plumbing improves, the discipline is identical.
• Panel judging for high-stakes dimensions: for S1-adjacent grading, run two different judge models and escalate disagreements to a human — cheaper than full human review, safer than one judge's blind spots.

Quality check

• Human labels came first and stayed hidden from every judge run — a judge tuned on visible answers is memorizing, not judging.
• Agreement is reported as a confusion matrix, not one number — and the human-FAIL/judge-PASS cell got read case-by-case.
• Iterations changed the judge for judge-failures and the rubric for rubric-failureslabel changes were deliberate and logged, never made to flatter the agreement score.
• The known fooling patterns were testedsycophantic-but-wrong and terse-but-correct items are in the labeled sample, and the judge's behavior on them is documented.
• The judge card existstrusted-for, not-trusted-for, alignment date, judge-model version, realignment trigger.
• A realignment canary is wired injudge-model or prompt changes automatically trigger a rerun on the labeled sample.