Synthesize Interview Debriefs Without Letting the Model Make the Hire

Workflow area: Hiring & Performance · Tier: Core · Hands-on time: ~30 minutes

A debrief packet is four scorecards written in four dialects: one interviewer's "strong hire" is backed by two observations, another's "no hire" is a paragraph of specifics, and the loudest voice in the room wins the discussion. AI is genuinely useful here — it reads all four cards without fatigue and notices that the highest rating has the thinnest evidence. It is also exactly where AI use in hiring gets legally serious: NYC Local Law 144 audits automated employment decision tools, the EU AI Act classifies AI in hiring as high-risk, and the design that keeps you on the right side of both is the same one that produces better hires — AI surfaces patterns, humans make the call.

When to use — and when not to

• Use it before every debrief meetingthe synthesis is pre-read material that turns the meeting from rating-averaging into evidence-examination.
• Use it when scorecards disagree sharplysplit panels are where evidence-vs-rating analysis earns its keep; the disagreement is usually about different rounds observing different things.
• Use it to audit your loop over timerun the same analysis across a quarter of debriefs and you'll find which round produces ratings without evidence (then fix it via pb-18).
• Skip any configuration where AI output determines the outcomeno auto-advancing candidates, no AI-generated ratings, no ranking candidates against each other by model. That's the line between an advisory tool and a regulated automated employment decision tool.
• Skip it on incomplete packetssynthesizing two of five scorecards produces confident nonsense about a candidate the model half-saw; chase the missing cards first.

The recipe

1. 1.

   Get the scorecards out clean. Export the completed cards from your ATS — ratings, free-text observations, rubric scores — for one candidate. Strip anything the analysis must not see: demographic details, compensation discussions, referral source, and other candidates' names. The model should receive observations and ratings, nothing else.

2. 2.

   Run the evidence-vs-rating analysis in your enterprise tenant.

   Copy

   You are an advisory analyst for an interview debrief. You do
   NOT rate the candidate or recommend an outcome. The panel
   decides; your job is to make their evidence easier to examine.
   
   Scorecards (role: [ROLE], rubric: [PASTE RUBRIC]):
   [PASTE SCORECARDS]
   
   Produce:
   1. EVIDENCE MAP — for each rubric dimension: what each
      interviewer actually observed (verbatim phrases), which
      round it came from.
   2. EVIDENCE-VS-RATING GAPS — every rating whose written
      evidence is thin, generic, or points the other way. Quote
      the rating and the evidence side by side.
   3. BIAS-PATTERN FLAGS — mark, with quotes: HALO (one strong
      moment echoed across unrelated dimensions), RECENCY (final
      round dominating), CONTRAST (comparisons to other people),
      and LANGUAGE SKEW (confidence adjectives replacing observed
      behavior). Flag patterns only — never infer anything about
      the candidate's identity or protected characteristics.
   4. CONTRADICTIONS — where interviewers observed opposite
      behaviors, so the panel can reconcile them.
   

3. 3.

   Read the output as questions, not verdicts. Good output looks like:

   Copy

   EVIDENCE-VS-RATING GAP — Interviewer C rated Technical 4/4;
   written evidence: "really sharp, great energy." No observed
   behavior cited. Compare Interviewer A, rated 2/4, who quotes
   the candidate's incorrect explanation of the cache
   invalidation in round 2.
   
   HALO FLAG — the phrase "great communicator" appears in 3
   cards across dimensions where communication wasn't assessed;
   all trace to the round-1 presentation.
   

   Each item becomes a debrief question — "C, what did you see that 'sharp' summarizes?" — not a conclusion. The model has no idea whether C saw something real; only the panel can ask.

4. 4.

   Run the debrief on evidence, decide by humans, record why. Circulate the synthesis as pre-read, open with the gaps and contradictions rather than the ratings, and let interviewers add the observations their cards under-recorded. The panel decides; the hiring manager records the decision and rationale in the ATS in their own words. The AI synthesis is an input you keep, not the rationale itself.

5. 5.

   Close the loop quarterly. Aggregate a quarter of syntheses (candidates de-identified) and look for structural findings: a round that consistently generates ratings without evidence, an interviewer whose cards are all adjectives, halo patterns that always start in the same round. Those findings feed interviewer coaching and pb-18's loop redesign — this is the legitimate, systemic use of the bias flags, as opposed to relitigating individual decisions.

Make it yours

• Small team or startup: with three interviewers you may skip the tooling and read the cards yourself — but the discipline transfers: open every debrief with "what did we observe?" before anyone says a rating, and write the decision rationale down even when it feels obvious.
• Large org or GCC: this workflow is enterprise-tenant-only — candidate data in a consumer AI tool is a compliance incident, full stop. Coordinate with HR and legal on where AI-assisted synthesis fits your jurisdiction's automated-decision rules (NYC LL144 bias audits, EU AI Act high-risk obligations, India's DPDP Act consent requirements for candidate data), and document that the tool advises while humans decide — that documentation is what an audit asks for.
• High-volume hiring: the synthesis prompt scales to batch use, but resist the efficiency temptation to let it shortlist — the moment output ordering influences who advances, you've built an automated employment decision tool and inherited its legal obligations. Keep it per-candidate, keep it advisory.

Quality check

• The model never produced a rating or recommendationreread outputs; if a "suggested outcome" ever appears, tighten the prompt and delete the run.
• Every flag carries a verbatim quotea bias flag without the phrase that triggered it is an accusation the panel can't examine.
• Gaps became questions in the actual meetinga synthesis nobody discusses is compliance theater; check that debrief notes reference it.
• The recorded rationale is human-writtenthe ATS decision field in the hiring manager's own words, not pasted model output; that field is what gets read in a dispute.
• Quarterly patterns drive loop fixes, not interviewer punishmentthe aggregate exists to repair rounds and rubrics; the first time it's used to shame an individual interviewer, your scorecards will turn vague in self-defense.