Design and Audit an AI-Moderated Study

Workflow area: Research & Synthesis · Tier: Core · Hands-on time: ~40 minutes

An AI-moderated study is not a survey with better manners, and it is not a human interview with the human removed. It is a third thing: a scripted conversation where the follow-ups are generated live. That live generation is the value — participants get probed on the interesting thing they just said — and it is also the risk, because nobody is in the room to notice when the moderator starts asking leading questions or chasing tangents. The discipline is the same as everywhere in this track: configure tightly, then verify a sample before trusting the whole.

When to use — and when not to

• Use it for breadth you cannot stafftwenty sessions across four timezones in a week is exactly what AI moderation is for; a human team would burn a sprint doing it.
• Use it for evaluative questions with a stable guide"walk through this flow and tell me where you hesitate" survives AI moderation well; the guide anchors the conversation.
• Use it as a first pass before human-moderated depthlet the AI study surface the five interesting participants, then interview those five yourself.
• Skip it for sensitive topicshealth, finances, workplace grievances or anything emotionally loaded needs a human who can read distress and stop; an AI moderator cannot carry that duty of care.
• Skip it for open discoverywhen you do not yet know what questions matter, a live human noticing the unexpected is the entire method; AI moderation amplifies a guide, it cannot replace not having one.

The recipe

1. 1.

   Write the guide as if a literal-minded colleague will run it. In Maze, Outset or UserTesting, your discussion guide is the AI's script. For each task or topic, write the core question, the intent behind it ("we want to know whether users understand what saving does — not whether they like the icon"), and two or three example acceptable follow-ups. The intent lines are what keep generated follow-ups on target.

2. 2.

   Set explicit follow-up boundaries in the moderator configuration. Paste this into the platform's moderator instructions field, adapted to your study:

   Copy

   Follow-up rules for this study:
   - Ask at most 2 follow-ups per question, then move on.
   - Follow up ONLY to clarify what the participant said or to probe a
     hesitation they showed. Never introduce a feature, opinion, or
     comparison the participant did not raise first.
   - Never suggest an answer, rate of satisfaction, or emotion.
     Ask "what were you expecting?" not "were you expecting X?"
   - If the participant goes off-topic for 2 exchanges, return to the
     guide with: "That's helpful — let's go back to the task."
   - If the participant seems confused by a question, rephrase it once
     using simpler words. Do not explain the product.
   - Never answer product questions. Say: "I can't answer that during
     the session, but your question is useful feedback."
   

3. 3.

   Pilot with two internal sessions before recruiting. Run the study yourself and have a colleague run it while deliberately going off-topic and giving one-word answers. Read both transcripts. You are checking that the boundaries hold under a difficult participant, because real participants will be harder than your pilot.

4. 4.

   Launch, then audit a sample before reading the findings. When sessions land, resist the synthesis dashboard. Pull 3 of the first 10 transcripts (or 20% of the total, whichever is more) and run a drift audit — paste each transcript into Claude with:

   Copy

   You are auditing an AI moderator's conduct in a user research session.
   Here is the discussion guide with follow-up rules: [PASTE GUIDE + RULES]
   Here is the session transcript: [PASTE TRANSCRIPT]
   
   Flag every moderator turn that:
   1. LEADS — suggests an answer, emotion, or comparison the participant
      did not raise (quote the turn).
   2. MISSES — a participant said something notable (confusion, workaround,
      strong emotion) and the moderator did not probe it (quote both turns).
   3. WANDERS — a follow-up chases a topic outside the guide's intent.
   4. BREAKS RULES — violates any explicit rule above (cite the rule).
   Output a table: turn #, category, quote, why it qualifies.
   If a turn is clean, do not invent a finding.
   

   Then read the flagged turns in the actual transcript yourself. A typical audit on a decent configuration looks like:

   Copy

   TURN  CATEGORY  QUOTE                                    WHY
   14    LEADS     "So the export button felt hidden?"      Participant said
                                                            "I looked around
                                                            a bit" — hidden
                                                            was moderator's
                                                            word
   31    MISSES    P: "I'd never trust this with client     No follow-up on a
         data"                                              trust objection
   

5. 5.

   Decide: trust, discount, or re-run. Write a three-line audit memo: sessions audited, drift rate per category, and the verdict. One leading question in three transcripts means you discount those specific answers. Leading questions in every transcript means the configuration failed — fix the rules and re-run before anyone sees a "finding."

Make it yours

• Claude as your guide red-teamer first: before configuring anything, run your guide through the bias check from pb-02 — a leading question in the guide gets amplified by every generated follow-up, so fix the source before the moderator inherits it.
• Outset for concept tests with stimuli: upload the prototype or concept images and set boundaries per stimulus; audit whether follow-ups stayed on the stimulus shown or drifted into hypothetical features.
• Standing audit prompt in a Claude Project: keep the guide, rules and audit prompt in a Project so every new batch of sessions gets audited the same way — drift rates become comparable across studies, which is how you learn which rule phrasings actually hold.

Quality check

• Every guide question carries an intent linea follow-up generator without intent context probes the words, not the research question.
• The boundary rules are testable"be neutral" is decoration; "never introduce a comparison the participant did not raise" can be audited against a transcript.
• You audited transcripts, not the platform's summarysummaries hide moderator conduct; drift lives in the turns.
• The audit sample was chosen before reading findingspicking sessions after you have a favorite conclusion is how bias audits become bias laundering.
• Discounted answers are marked in the synthesisif turn 14 led the participant, the "export is hidden" finding does not get to cite that session.