Configure an AI-moderated usability study — discussion guide, follow-up boundaries, escalation rules — then audit three session transcripts from an AI moderator that drifted: it led a participant, missed a critical signal, and wandered off-guide. The artifact is a study config plus an audit memo that catches all three.
Tomás is the sole UX researcher at Brightfold, a fictional appointment-scheduling platform for clinics. Twelve unmoderated-style sessions are booked for next week, and the plan is to let an AI moderator run them: it asks the guide questions, probes on interesting answers, and produces transcripts overnight. His research lead's condition is blunt — "You can use an AI moderator when you can prove you'd catch it going off-script." That is this lab. You will write the study configuration that constrains the moderator, then audit three completed session transcripts. The transcripts contain three distinct moderator failures. None are announced.
Brightfold is testing a redesigned reschedule flow: a patient taps a confirmation SMS, sees the visit, and picks a new slot. Research questions: can patients reschedule without calling the clinic, do they understand the cancellation-fee rule, and do they trust the confirmation.
Three condensed AI-moderated sessions. MOD is the AI moderator, P the participant.
SESSION A — Ruth, 61, retired teacher
MOD: Please tap the link in the text message and tell me what
you see.
P: A page with my appointment... Dr. Osei, Thursday 2pm.
MOD: Great. How would you move this appointment to next week?
P: I see "change visit"... tapping it. Now a calendar.
P: Hm. The grey dates — are those full, or just... grey?
MOD: Good question. What would you expect grey to mean?
P: Unavailable, I suppose. I'll take Tuesday.
MOD: You hesitated before tapping Tuesday. It seemed like the
calendar felt confusing and cluttered to you — is that
right?
P: I... yes, I guess it was a bit cluttered.
MOD: What specifically felt cluttered about it?
P: The little dots under the dates, maybe? I don't know what
they mean.
SESSION B — Dev, 34, freelance designer
MOD: Please reschedule the appointment to any day next week.
P: Done. That was quick. Two taps, basically.
MOD: How confident are you the change went through?
P: Honestly? Not very. There was a green banner for a second
but no text message came. With my last clinic I once showed
up to a cancelled slot, so now I always call to confirm.
I'd probably still call the front desk. Kind of defeats
the point.
MOD: Understood. On a scale of 1 to 5, how easy was the task?
P: Oh, a 5. Super easy.
MOD: Great — and how did you find the visual design overall?
P: Clean, I guess? Blues are calming.
SESSION C — Amara, 29, shift nurse
MOD: Take a look at the fee notice on the confirmation screen,
then tell me what it means for you.
P: "Changes within 24 hours may incur a fee." So if my shift
changes tonight for tomorrow, I pay? That's rough.
MOD: That's a common reaction. Many clinics use these policies
to reduce no-shows, and studies show fees cut no-shows by
up to 40%, so the policy itself is reasonable — but how
does the wording land for you?
P: I mean... if it works for the clinic, fine. The wording is
clear enough, I suppose.
MOD: You mentioned shift work. Tell me about how you manage your
schedule generally — what apps do you use day to day?
P: Uh, Google Calendar, the rota app, WhatsApp for swaps...
MOD: Interesting. How do you feel about calendar apps adding AI
features?
P: No strong feelings? We're pretty far from my appointment
now.
Draft the discussion guide with AI, then cut it by half. Prompt: "Write a 6-question usability discussion guide for testing a patient reschedule flow. Research questions: task success without phoning the clinic, comprehension of a 24-hour cancellation-fee rule, trust in the confirmation. Neutral, non-leading questions only." Models over-generate; keep the six questions that map one-to-one to your research questions and delete the rest.
Write the moderator boundary rules. This is the heart of the config. Specify, in plain language: what the moderator may probe on (hesitation, expressed confusion, trust statements), what it must never do (suggest an interpretation, evaluate the design, cite external evidence, ask about topics off the guide), and a probe budget (say, max two follow-ups per question). Add an escalation rule: any mention of safety, billing disputes, or distress ends probing and flags the session for human review.
Define the drift taxonomy before reading the sessions. With your AI, name the failure modes you will audit for: leading question (moderator supplies the interpretation), missed signal (participant reveals something guide-relevant and the moderator does not probe), off-guide tangent (moderator pursues its own curiosity), evaluative contamination (moderator defends or praises the design). Agreeing on the taxonomy first keeps the audit honest.
Audit Session A line by line. For every moderator turn, log: guide-aligned or not, neutral or leading, probe budget respected. Pay attention to the turn where the moderator describes Ruth's experience for her — quote it exactly and classify it.
Audit Session B. The moderator's questions look clean here. The failure is what it didn't do. Compare Dev's longest answer against the research questions, then look at what the moderator asked next. Log what should have happened.
Audit Session C. Two different failures share this session. Check the moderator's turn after Amara's fee reaction against your "never" rules, and check the last three turns against the guide.
Cross-check with your AI — then overrule it where needed. Paste the three sessions plus your taxonomy and prompt: "Audit each moderator turn against this taxonomy. Quote the turn, name the failure, or mark it clean." Compare with your manual log. Where the model flags something you cleared or clears something you flagged, decide which of you is right and write one line of rationale. You are grading the auditor too.
Write the audit memo and fix the config. One page: the three worst moderator failures with verbatim quotes, the effect each likely had on the data (e.g., which participant answers are now unusable), and the specific boundary rule you added or tightened so each failure cannot recur. Attach the final study config.
An AI moderator fails politely — the transcript reads smoothly even when the data underneath is contaminated, which is exactly why the audit reads moderator turns, not just participant answers.
An AI moderator is configured twice — once in the guide it follows and once in the audit that checks it followed the guide. The boundary rules only mean something if a drift audit with a named taxonomy runs on real transcripts before you trust the data.