Use Synthetic Users as Hypotheses, Never Evidence

Workflow area: Research & Synthesis · Tier: Core · Hands-on time: ~30 minutes

The temptation is obvious: why recruit eight participants over two weeks when a prompt produces "user feedback" in eight seconds? Because the model is not sampling your users — it is sampling text about people vaguely like your users, smoothed toward agreeableness. NN/g's documented failure cases include synthetic users praising a concept that real users rejected in testing — the optimism false positive that green-lights the wrong build. Used as evidence, synthetic users are a research-shaped hallucination. Used as hypothesis generators, they are genuinely useful: they widen your question set, stress your assumptions, and cost nothing to consult. This playbook is the fence between those two uses.

When to use — and when not to

• Use it before writing a discussion guidesynthetic responses surface angles and objections you had not thought to ask about; every surprising one becomes a real question for real participants.
• Use it to pressure-test proto-personasrole-playing a persona exposes internal contradictions and missing attributes before recruitment money gets spent on the wrong screener.
• Use it to rehearse interviewspracticing follow-ups against a difficult synthetic participant is free moderator training.
• Skip it wherever a decision will be madeconcept validation, pricing reactions, feature prioritization, "would users want this": synthetic answers to these questions are noise wearing a persona's name tag.
• Skip it for underrepresented user groups especiallythe model's training data underrepresents them too; synthetic output here is stereotype extrapolation, and it is least reliable exactly where your research gaps are largest.

The recipe

1. 1.

   Write the boundary sentence before the first prompt. In your study plan, add one line: "Synthetic users in this project generate hypotheses and research questions only; no synthetic output may be cited as a finding, quoted in a report, or used to de-scope real research." This exists so a stakeholder skimming your artifacts never mistakes rehearsal for results.

2. 2.

   Condition the persona from real data, not vibes. Build the persona prompt from whatever genuine evidence exists — support themes, past research, analytics:

   Copy

   You will role-play a user persona for hypothesis generation.
   This is NOT user research; your answers are conjectures to test.
   
   Persona: mid-market HR manager, 8 years in role, runs onboarding
   for ~40 hires/quarter. Evidence base: our support data shows this
   segment complains about duplicate data entry and audit-trail gaps;
   past research shows low trust in automation for compliance tasks.
   
   Stay in character. Be concrete and specific. Do not be agreeable —
   if the persona would be indifferent or skeptical, say so and why.
   When you are extrapolating beyond the evidence base above, prefix
   the sentence with [GUESS].
   

3. 3.

   Interrogate the concept, hunting for questions — not verdicts. Present your concept and probe: "What would make you abandon this in week one?" "What's the workaround you'd use instead?" "What would you need to see before trusting it with compliance data?" Push past the first agreeable answer every time: "You're being polite. What's the objection you'd say to a colleague, not a vendor?"

4. 4.

   Harvest hypotheses into a testable list. Convert every interesting synthetic claim into a falsifiable hypothesis with a method attached:

   Copy

   SYNTHETIC CLAIM                        HYPOTHESIS TO TEST         METHOD
   "I'd never let it auto-send offer      Users distrust automation  Interview
   letters"                               at the send step, not      Qs 4-6
                                          the draft step
   "The dashboard is for my boss, not     Primary user ≠ report      Field obs.
   me" [GUESS]                            consumer                   + analytics
   

   The [GUESS] markers tell you which hypotheses rest on the thinnest air — often those are the most valuable to test, and the most dangerous to believe.

5. 5.

   Run the real research, then score the synthetic run. After real sessions, spend ten minutes comparing: which synthetic claims did real users confirm, contradict, or never mention? Log it. Over a few studies this calibration file teaches your team precisely where synthetic output tends to mislead for your product — turning a vague distrust into a documented, local error profile.

Make it yours

• Two-model disagreement mining: run the same persona in Claude and ChatGPT and diff the answers — where the models disagree is where your real research questions live, because at least one of them is confabulating.
• Panel mode for screener design: role-play three variants of the persona (skeptic, power user, first-timer) and use their divergent needs to sharpen recruitment criteria — if your screener cannot tell the three apart, it is too loose.
• Interview rehearsal mode: have the model play a rambling, tangent-prone participant and practice redirecting; then have it critique your moderation for leading questions afterward.

Quality check

• The boundary sentence is written where stakeholders will see itin the study plan, not in your head.
• Every persona prompt cites its evidence basea persona conditioned on nothing generates fiction conditioned on nothing.
• Outputs became hypotheses with methods attacheda synthetic claim without a planned test is a rumor you paid zero dollars to start believing.
• No synthetic quote appears in any report, deck or ticketnot even labeled; labels fall off as artifacts travel.
• The calibration log exists and gets updatedcomparing synthetic guesses to real findings after each study is what makes this practice compound instead of drift.