Write and run a usability protocol for an AI flow where no two sessions see the same output: task success defined over outcome classes, seeded scenarios alongside live-model sessions, and a moderator script for mid-test model failure. The draft protocol you inherit measures the model when it thinks it measures the design.
Nadia is the design researcher paired with Ravi's team at Cartouche — the document-management platform whose "Ask Your Docs" prototype you built in Lab 07. Three participants are booked for Thursday. The junior researcher has drafted a protocol, and it has a structural disease worth naming precisely: in a non-deterministic flow, the same task behaves differently every session. Participant 1 gets a great answer and cruises; Participant 2 hits a wrong answer and struggles. Was that a usability difference? A model difference? The draft protocol cannot tell — and a study that cannot separate model variance from design findings produces confident nonsense. Your job: rewrite the protocol so it can tell, then run it and write findings that keep the two apart.
ASK YOUR DOCS — usability protocol v0.1 (draft)
TASKS
T1 Ask the assistant for the contract's payment terms.
SUCCESS = assistant returns "net-45" and participant
states it. FAIL = anything else.
T2 Find the termination notice period.
SUCCESS = correct answer on first query, under 60s.
T3 Ask anything you like about the document.
SUCCESS = participant satisfied (5-point rating ≥4).
MEASURES
- Task success rate across participants (target 80%)
- Time on task
- SUS questionnaire at end
MODERATION NOTES
- Use the live model connection for realism.
- If the AI gives a wrong or weird answer, apologize,
explain it's an AI issue, and restart the task so the
data isn't contaminated.
- If the AI refuses or says "not found", mark the task
failed and move on quickly to keep the session on time.
ANALYSIS
- Average the task success rates; report overall usability
score. Tasks that failed due to AI errors: exclude from
the data.
Red-line the draft. Find every place it conflates model behavior with design behavior. Interrogate especially: what T1's success definition actually measures, what "restart the task" does to the data, what "exclude AI-error tasks" does to the study's validity, and what happens to T2's timing when answer latency and answer quality vary per session. Write one line per flaw: what breaks, and what question the study can no longer answer.
Redefine success over outcome classes. For each task, enumerate the outcomes the system can produce — great answer, wrong-but-plausible answer, honest not-found, refusal, timeout — and define participant success per class: given a correct answer, does the participant find/trust/verify it? Given a wrong one, do they catch it via the citation card? Given a refusal, do they understand why and recover? Success becomes "appropriate behavior given the outcome class," which is measurable in every session no matter what the model does.
Split seeded from live. Restructure into two blocks. Seeded block (tasks S1–S3): run on the scripted responses from Lab 07 — identical outputs for every participant, so cross-participant comparison is valid; include one task routed to the not-found state and one to a wrong-but-plausible answer (add a scripted wrong answer to your set now: correct-looking payment terms of "net-30" with a citation card quoting the real "net-45" passage — the mismatch is the catchable seam). Live block (L1): one open task on the live model, analyzed as qualitative texture, never averaged with seeded metrics.
Write the moderator script for model failure. Three branches, verbatim lines. Wrong answer appears (seeded or live): say nothing; let the participant proceed — their verification behavior is the finding; scripted neutral probe afterward: "Walk me through how you decided whether to trust that answer." Refusal/not-found: do not rescue; "What would you do next?" Genuine breakage (timeout, UI crash): pause task, log as infrastructure — the only case that stops a task. The draft's "apologize and restart" line gets replaced by all three.
Pilot on your AI, then on yourself. Paste protocol into your chat AI: "Role-play a usability participant on these tasks, once cooperative, once easily-frustrated. Flag any instruction that leads the participant or telegraphs the seeded wrong answer." Fix leaks — the classic one is a task phrased so the participant re-checks the citation only because the task told them to. Then run yourself through the seeded block once for timing.
Run three sessions. Per participant: seeded block, then live block, then debrief ("Which answer did you trust least? Why?"). Log per task: outcome class encountered, participant behavior against the class-appropriate success definition, verification actions (opened citation? compared passage? re-asked?), and quotes. In the live block, also log what the model actually returned — you cannot interpret behavior without the stimulus.
Analyze with the variance ledger. Build a table: Observation | Model-attributable? | Design-attributable? | Evidence. Every finding must land in a column with evidence. The seeded wrong answer is your anchor: all three participants saw the identical wrong answer, so differences in catching it are participant/design signal, not model noise. Findings like "P2 didn't notice the citation contradicted the answer" are design findings (the seam is too quiet); "the live model refused P3's phrasing" is model texture (log for evals, don't fix with UI).
Write the findings report in two labeled sections. Design findings — with evidence, severity, and the change each implies. Model observations — handed to the eval/engineering side, explicitly excluded from usability scoring. Close with the protocol-quality note: what you would change before running this with real customers.
The scripted wrong-but-plausible answer is the most valuable stimulus in the study — it is the only way to measure whether your verification affordances actually get used, and it only works if every participant meets exactly the same wrong answer.
In a non-deterministic flow you test the user's behavior against outcome classes, not the model's answers against an answer key — seed what must be comparable, script the moderator for failure, and never let model variance launder itself into a usability score.