Run a Wizard-of-Oz Test for an AI Feature

Workflow area: Prototyping & Validation · Tier: Core · Hands-on time: ~40 minutes setup

AI features carry a double demand risk: users may not want the capability, and they may not tolerate the capability at the quality a real model will deliver. A Wizard-of-Oz test — the interface is real, the "AI" is a human (or canned responses) behind it — answers both before any model is wired in: what do users actually ask for, how do they react to answers, where does trust break. It's also the rare test that pays twice, because everything users type into the fake is exactly the input distribution your eval set (pb-37) will need. The non-negotiable is the disclosure protocol: simulating a product's intelligence for a test is a method; letting participants walk away deceived is a trust debt your product pays later.

When to use — and when not to

• Use it when demand for the AI capability is the open questionbefore pb-35's feasibility spike, before roadmap commitment. Feasibility doesn't matter if nobody uses it.
• Use it when you need the real input distributionwhat users actually type into an "ask anything" box is unknowable from workshops, and it makes or breaks scope.
• Use it to find the trust boundarywatching a user decide whether to act on the fake's answer tells you what confidence display and citation UX the real feature needs (feeding pb-49).
• Skip it when the feature already exists behind an APIif a prompt plus your sample data can produce real outputs in a day, test the real thing (pb-35); fake only what you can't cheaply make.
• Skip any design where deception persists after the sessionno undisclosed humans reading sensitive inputs, no launching "AI-powered" marketing on a manned backend. The curtain comes down at debrief, always.

The recipe

1. 1.

   Define the wizard's script and the demand signal. Write the hypothesis card (pb-25's format) for the AI feature — riskiest assumption, test, kill criteria — then decide the wizard mechanics: live human (highest fidelity, exhausting — for 5–8 deep sessions) or canned responses (pre-written answers to the 15 most likely queries plus honest fallbacks — scales to dozens of async participants). Write the wizard's rulebook so responses are consistent: tone, length, what it refuses ("I can only answer questions about your expense data"), how long the "thinking" delay runs — 3 to 8 seconds; instant answers break the illusion and skew trust readings.

2. 2.

   Build the front stage. In Lovable (or your pb-25 prototype), a chat or suggestion surface wired to the backstage: for live-wizard, a shared doc or Slack channel where the wizard types responses that appear in the UI; for canned mode, responses keyed to query patterns. Use ChatGPT beforehand to draft the canned set — then hand-edit every answer to the quality you believe a production model could plausibly hit, not the quality of your dreams. Overpromising quality in the fake inflates demand you'll never serve.

3. 3.

   Instrument everything typed and clicked. Log with timestamps: the user's exact query text, the response shown, whether the user acted on it (clicked through, copied it, followed the suggestion), retries and rephrasings, and abandonments. For async canned tests, a Google Form at the end captures the expectation reading: "What did you expect it to be able to answer? Where was it wrong? Would you use this weekly — for what?"

4. 4.

   Run sessions with disclosure at the honest boundary. Consent upfront covers recording and that "some features are simulated for research." During the task, the assistant behaves as the product would. At debrief, always: "The assistant's answers were written by a human/prepared in advance — we test this way before deciding whether to build the AI. Knowing that, does it change what you'd trust it with?" That last question is itself high-value data — it isolates the trust users grant the interface from the trust they'd grant a machine. A session log entry worth its weight:

   Copy

   P4, 14:32 — typed: "why is marketing over budget this month"
   → wizard: variance summary w/ top 3 line items
   → P4 copied the summary into an email WITHOUT verifying the
     numbers against the dashboard. At debrief: "if it's wrong
     once in front of my CFO, I'm done with it."
   

5. 5.

   Read out demand, trust, and the eval seed — separately. Three artifacts: the demand readout (did usage meet your kill criteria — repeated engaged use, stated willingness to rely, task completion via the assistant when a manual path existed?); the trust map (where users acted without verification, where they double-checked, what the debrief question moved); and the eval seed — every real query, deduplicated and clustered, with the response quality users accepted or rejected per cluster. That last file, versioned, is the day-one input distribution for pb-37's eval set and the quality bar for pb-34's probabilistic requirements. Wizard-of-Oz demand comes with a caveat to keep visible in the readout: your wizard's quality is an upper bound the model must actually reach — pb-35's spike tests whether it can.

Make it yours

• The painted-door lite: if even the wizard is too expensive, ship a real "Ask AI" button that opens "coming soon — what would you have asked?" with a text box. Weakest signal, near-zero cost, still harvests the query distribution.
• The degraded-quality arm: run a second canned set at deliberately mediocre quality (right answers, no sources, occasional hedge) for half your participants — the demand delta between arms is your quality-sensitivity estimate, the number pb-45's model-tier decision quietly depends on.
• B2B concierge mode: for enterprise, run it as a disclosed human-powered pilot — "analyst-backed beta, humans in the loop" — for two weeks. Fully honest, produces production-shaped logs, and customers often prefer it; several durable AI products started as exactly this.

Quality check

• Disclosure happened at every debriefno participant left believing they used a working AI; if any did, that's a protocol failure to fix before the next session, not a footnote.
• The wizard held the rulebookspot-check session logs for drift; a wizard who got helpfully smarter mid-study inflated the quality bar and contaminated the demand read.
• Demand claims cite behavior, not complimentsacted-on responses and repeat usage count; "this is cool" at debrief counts for nothing, and the readout says which is which.
• The quality caveat is on the readoutevery demand number carries "at wizard-level quality"; pb-35 decides whether that level is reachable before this evidence enters a roadmap argument.
• The eval seed is filed and versionedreal queries clustered, accepted-quality notes per cluster, stored where pb-37 will find it. If the test ends and this file doesn't exist, half the test's value evaporated.