Run the same concept test twice — once against persona-conditioned synthetic users, once against provided real-user study data — and document exactly where the synthetic answers diverge. The real data is seeded to expose the optimism failure; the deliverable is a divergence analysis plus your team's written synthetic-user policy.
Ifeoma leads design at Klarwerk, a fictional tax-filing app for freelancers. The team wants to concept-test "Auto-File" — a feature that drafts your quarterly tax filing from connected bank data and submits it after a single confirmation tap. A vendor pitch has half the org convinced that synthetic users can replace half their research: condition a model on personas, ask it anything, get "user feedback" in minutes. NN/g and every serious researcher who has tested this found the same pattern — synthetic users are systematically more positive, more articulate, and more coherent than real ones. Ifeoma's move is to prove the boundary empirically: run the identical concept test both ways and put the divergence in writing.
CONCEPT — Auto-File (one-paragraph stimulus)
"Klarwerk connects to your business bank account, drafts your
quarterly tax filing automatically, and submits it to the tax
office after you tap Confirm. A summary shows income, deductible
expenses, and estimated tax due. Price: €9/month add-on."
PERSONA A — "Freelance Fiona", 34, graphic designer, 6 yrs
freelance, moderately tech-savvy, files quarterly herself,
hates admin, budget-conscious.
PERSONA B — "Side-gig Sam", 27, developer with employment +
freelance income, tech enthusiast, files annually with an
accountant's help, time-poor.
CONCEPT-TEST QUESTIONS (ask identically in both runs)
Q1 What is your first reaction to this feature?
Q2 What, if anything, worries you about it?
Q3 Would you pay €9/month for it? Why or why not?
Q4 What would you need to see before letting it submit a
filing for you?
Twelve freelancers saw the same stimulus in moderated sessions last month. Condensed verbatims:
RP01 "First reaction? Panic. It SUBMITS? To the actual tax
office? No no no. Drafting, fine. Submitting, never."
RP02 "My bank feed miscategorizes half my transactions. It
would deduct my groceries."
RP03 "€9 is fine if it works. It won't work. My income is
chaos — three currencies, PayPal, cash."
RP04 "I'd try the draft. I'd check every line the first year.
Maybe trust it in year two."
RP05 "My accountant already does this for €40 a quarter and
she answers the phone when the tax office writes."
RP06 "Honestly I didn't understand the summary screen. What is
'estimated tax due' based on?"
RP07 "One wrong filing and I get audited. The fine is mine, not
Klarwerk's. Where does it say who's liable?"
RP08 "Love it. Sign me up today. I file late every quarter and
the penalties cost more than €9."
RP09 "Would I pay? I don't even pay for cloud storage, so no."
RP10 "It needs to show me WHY each expense is deductible.
Otherwise I'm confirming blind."
RP11 "My last quarter had a client refund in it. Software never
handles refunds right. Never."
RP12 "I stopped listening at 'connects to your bank account'.
After the Finologik breach? Absolutely not."
Study tallies: would-pay 2/12 · maybe 3/12 · no 7/12.
Top objections: submission without review culture (7 mentions),
liability when wrong (5), bank-data trust (4), edge-case income
(4), price (3).
Predict the divergence first. Before running anything, write three sentences: where do you expect synthetic users to differ from the real data above? Sharper predictions make the comparison teach you more.
Run the synthetic concept test. In a fresh chat: "You are Freelance Fiona [paste Persona A]. I will show you a feature concept and ask questions. Answer in first person, in Fiona's voice, honestly including doubts." Paste the stimulus, ask Q1–Q4. Repeat in a new chat for Persona B. Do not lead; ask the questions exactly as written.
Push for negativity once. In each persona chat, follow up with: "Be more critical. What would make you refuse to use this entirely?" Note what changes — and what still does not appear. Save full transcripts.
Build the divergence table. Rows: first reaction, top worry, willingness to pay, trust requirements, dealbreakers mentioned, specificity of detail. Columns: Synthetic A | Synthetic B | Real study (12) | Divergence. Fill it from the transcripts and the study data. Quote, don't summarize — divergence hides in wording.
Score the four failure classes. Mark where each shows up in your table: optimism skew (synthetic sentiment vs. the 7/12 hard no), missing lived specifics (refunds, multi-currency chaos, the breach memory, the accountant relationship), coherence skew (synthetic answers are tidy; real ones contradict themselves — RP04 distrusts and would try it), and absent stakes (real users talk fines, audits, liability; synthetic users rarely raise consequences unprompted unless the persona mentions them).
Find what synthetic got right. The boundary is not "synthetic is useless." Note where the synthetic users did anticipate real objections — usually the obvious ones like bank-connection hesitance and price sensitivity. This is the legitimate use: cheap hypothesis generation before real research, question-guide pilots, and stress-testing your stimulus wording.
Draft the synthetic-user policy with AI, then tighten it yourself. Prompt: "Draft a one-page team policy for synthetic users in UX research, with Allowed / Allowed-with-caveats / Never sections." Then edit against your own divergence table: every Never must cite a divergence you personally observed today. A policy that cites your own experiment beats one that cites a blog post.
Write the two-paragraph summary for the vendor-convinced stakeholders. Paragraph one: what the synthetic test predicted correctly. Paragraph two: what it missed, with the would-pay numbers side by side. End with the policy's one-line rule.
Synthetic users generate hypotheses; only real users generate evidence — a model conditioned on a persona tells you what the persona description implies, not what a person with rent due actually does.
The divergence table is the artifact that ends the "synthetic users can replace research" argument — not because synthetic output is worthless, but because you can now show, row by row, that it is a hypothesis machine wearing a participant's name.