Redesign the Interview Loop for Candidates Who Code with AI — Because They All Do

Workflow area: Hiring & Performance · Tier: Core · Hands-on time: ~40 minutes

The uncomfortable audit result almost every team gets: paste your current coding question into a frontier model, and it produces a passing solution in one prompt. That means your loop now measures either "can the candidate use AI" (if you allow it) or "did the candidate practice pretending not to" (if you ban it and can't tell). Neither is what you meant to hire for. The fix is not banning AI or surrendering to it — it's deciding, round by round, what each round measures, and making the AI rules serve that measurement.

When to use — and when not to

• Use it before your next hiring cycle opensretrofitting rounds mid-pipeline creates inconsistent signal and unfair comparisons across candidates.
• Use it when interviewers report "everyone solves it now"that's not a stronger market; that's your problem set aging out.
• Use it when you've written AI fluency into the JDpb-17's must-haves are empty until a round actually assesses them.
• Skip wholesale redesign if your loop already separates signala well-run system-design round and a genuine debugging round may only need the single-prompt verification pass, not reconstruction.
• Skip building rounds around detectiondesigning the loop as an anti-cheating arms race measures stealth, not engineering; the policy side of covert use belongs in pb-19, not in your problem design.

The recipe

1. 1.

   Inventory what each current round actually measures. List every round with its intended signal (algorithms? design? collaboration?) and its honest signal now that AI exists. Be blunt: a take-home with no follow-up conversation currently measures access to a chatbot.

2. 2.

   Run the single-prompt audit on every problem. Paste each coding question, verbatim, into two frontier models. If a model produces a substantially correct solution in one prompt, the problem cannot anchor an AI-free round and is weak even in an AI-allowed one — the round becomes transcription. Log results per problem; this audit repeats quarterly, because model capability moves.

3. 3.

   Design the AI-allowed round to score the collaboration, not the artifact. The candidate uses your sanctioned tool (a pinned model in your interview platform — CoderPad and CodeSignal-class platforms now ship AI-enabled modes) on a problem where the model alone gets it wrong. Have AI help you build it:

   Copy

   Design a 60-minute AI-allowed interview exercise for a
   [LEVEL] engineer on [STACK]. Constraints:
   - A single prompt to a frontier model must NOT solve it: it
     needs context the candidate must supply, a deliberately
     underspecified requirement they must notice, and at least
     one place where typical AI output is subtly wrong
     (concurrency, edge cases, or a stale-API trap).
   - Produce: the exercise, the ideal-path walkthrough, and a
     scoring rubric with observable behaviors for: prompt
     quality and context-supplying; VALIDATION (does the
     candidate test/inspect AI output before building on it);
     debugging when AI output fails; and judgment about when
     to stop prompting and just write the code.
   

   A usable rubric row looks like:

   Copy

   VALIDATION — Strong: runs/tests AI output before integrating;
   spots the off-by-one in the pagination helper unprompted.
   Adequate: tests after integration when something breaks.
   Red flag: pastes, compiles, moves on — ships the planted bug.
   

   That red-flag row is the round's entire reason to exist: you are watching for the engineer who treats model output as finished work.

4. 4.

   Protect fundamentals with a deliberately AI-free round. One round — reasoning about code on a whiteboard or shared doc, reading unfamiliar code and predicting behavior, or a design conversation with shifting constraints — where the signal is what the candidate holds in their head. Say why in the invite: "this round is tool-free because we're assessing your reasoning, not your tooling." Candidates respect a stated reason; they resent an unexplained ban.

5. 5.

   Rebalance the loop and brief every interviewer. A typical shape: screen (AI-free reasoning) → AI-allowed build round → AI-free design → behavioral including a "tell me about a time AI misled you" probe. Retrain interviewers on the new rubrics — an interviewer who scores the AI-allowed round on lines typed instead of validation behavior silently reverts the redesign. Dry-run the new rounds on two internal engineers before a candidate ever sees them.

Make it yours

• Small team or startup: you can't afford four rounds — collapse to one AI-allowed build session (their real workflow, your real stack) plus one AI-free conversation about their choices. The single-prompt audit still applies to your one exercise.
• Large org or GCC: standardize exercises and rubrics across the site so debriefs compare like with like (pb-20 depends on this), version-control the problem bank, and rotate exercises on a schedule since leaked problems plus AI equals instant solves. Align the loop with the candidate-AI policy your platform settings enforce.
• Hiring seniors and staff: weight the AI-allowed round toward judgment — when they overrode the model, what they refused to delegate — and add an "review this AI-generated PR" exercise; it's the day job now, and it exposes rubber-stampers at every level.

Quality check

• Every problem passed the single-prompt audit this quartera problem that one-shots today invalidates every comparison your pipeline makes tomorrow.
• Rubrics score behaviors, not artifactsif two interviewers watching the same session couldn't cite the same observable moments, the rubric is adjectives.
• The loop's AI rules match the published candidate policyround-by-round rules that contradict what pb-19 told candidates is an integrity failure you created.
• AI-free rounds have a stated purpose"no AI because we said so" reads as fear; "no AI because this measures reasoning" reads as design.
• Signal actually separatesafter five loops, check the spread: if every candidate scores identically on the AI-allowed round, it's measuring the model, not the human.