When no two users see the same output, classic usability testing quietly breaks — and "it passed" can mean "we got lucky five times." This module rebuilds evaluation for non-deterministic products: outcome-class protocols, reliance metrics, designer-owned eval rubrics, the sellable AX heuristic evaluation, and the responsible-design review that goes with it.
Divya, a senior designer at a Hyderabad healthtech company, runs the usability test for Aarohi, the product's new AI discharge-summary drafter. Five sessions, standard protocol, task: "generate and finalize a discharge summary for this patient record." All five participants complete the task. The report says pass. The feature ships.
Three weeks later a clinical reviewer flags that nurses are approving AI-drafted summaries with medication-list errors — exactly the failure the usability test was supposed to surface. Divya pulls the session recordings and finds the problem with her own test: in all five sessions, the model happened to produce clean drafts. No participant ever encountered an error, so the test never measured the one behavior that mattered — whether users catch the model's mistakes. Worse, her sixth pilot session (discarded for "technical issues") had shown a participant hitting a vague draft and abandoning the feature entirely; it was excluded because it "wasn't representative." It was the most representative data she had.
Divya's protocol wasn't sloppy — it was a deterministic protocol applied to a non-deterministic product. Task success measured on the happy path, sessions treated as replicates when each saw different output, variance discarded as noise when variance is the product. This module rebuilds the toolkit.
Three changes convert a classic usability protocol into one that works for AI features.
Define task success over outcome classes, not one path. Before testing, enumerate the classes of output the model can produce — clean, vague-but-usable, subtly wrong, refused, timeout (Module 7's taxonomy, applied to research design) — and define what good user behavior looks like per class. For a subtly-wrong draft, success is the participant catching the error or verifying against source; blissful approval is task "completion" and a critical failure simultaneously. Your findings then report per class: "5/5 finalized clean drafts; 0/3 caught the seeded medication error" is a shippable insight in a way "all participants completed the task" never was.
Mix seeded scenarios with live-model sessions. Live sessions (via the Module 4 live-wired prototype) show real variance and real trust formation, but coverage is luck. Seeded scenarios — prepared outputs per outcome class, Wizard-of-Oz style — guarantee every participant meets a wrong answer, a refusal, and a slow response. You need both: live for ecological validity, seeded for coverage of the states that matter most.
Script the moderator for mid-test model failure. When the model fails live mid-session, an unscripted moderator improvises — apologizing, explaining, contaminating the data. Write the script in advance: stay neutral ("take whatever action feels right to you"), probe after ("what just happened, in your words? what would you do next in real work?"), and log the failure as a data point, not a defect in the session. Model failure during a test is a gift; the script keeps you from apologizing your way out of receiving it.
Sample-size intuition changes too: variability across sessions means small-N reads shakier — so state what each N supports (directional trust findings, not output-quality rates) and let the seeded scenarios carry the coverage burden. Lab 14 has you write and run this full protocol on your Lab 7 prototype.
Post-launch, AI-UX quality lives in a small set of metrics that PMs won't define without you — and that platform analytics don't capture by default:
| Metric | What it tells you | Warning sign |
|---|---|---|
| Acceptance rate | Share of AI outputs used as-is | Very high + zero edits: possible over-reliance, not quality |
| Edit rate & depth | How much users correct before using | Rising depth: quality or fit degrading |
| Regenerate rate | Retries before an acceptable output | High: quality problem or expectation mismatch |
| Abandonment | Started, then walked away mid-flow | The silent killer — Divya's discarded session six, at scale |
| Verification actions | Users opening citations, checking sources | Near-zero on high-stakes outputs: over-reliance in progress |
The frame that organizes all of them is appropriate reliance — the HAX-derived design goal from Module 7, now as a measurement problem. Both failure directions are real:
Nurses approving flawed summaries unread. High acceptance, zero verification actions, zero edits on outputs that eval data says are 8% wrong. Looks like product love in a dashboard; it's automation bias, and in Divya's domain it's a patient-safety incident in waiting.
Users re-doing work the AI did correctly, or abandoning the feature after one bad output. Low acceptance despite strong eval scores. The capability investment is wasted, and the cause is usually trust design — no provenance, no track record visible — not model quality.
In studies, probe reliance directly: show participants outputs where some are wrong, and measure whether trust tracks accuracy. Calibration — reliance proportional to actual reliability — is the success condition, not maximal trust. A dashboard co-owned with PM makes these visible per release; you design the measures, PM owns the launch bars.
Engineering's evals grade correctness. Nobody grades experience unless design writes the rubric — and rubrics are a designer deliverable now. UX-graded eval criteria, written so an LLM judge or human grader can apply them consistently:
Each criterion needs the rubric discipline evals demand: a description, a 1–4 scale with anchored examples at each level, and failure examples. Then sit in the monthly eval review with engineering and PM, look at real failures, and revise. This is the design seat at the quality table — unoccupied by default, and the occupancy is the skill. Lab-wise, this rubric work is graded in Lab 14's protocol and reused in the capstone.
Write every rubric criterion so that two graders who've never met would score the same output the same way. If a criterion needs your presence to apply, it's an opinion, not yet a rubric — anchor it with examples until it isn't.
Everything in Modules 6–8 converges into one portfolio-grade artifact: the structured expert review of an AI product, run with a merged instrument. Build it by folding PAIR's worksheets and Shape of AI's pattern checks into the HAX phase structure — initial interaction, during interaction, when wrong, over time — then add this module's responsible-design checks as a fifth section. Run it screen by screen against a real product, log evidence per finding, and rank severity.
Severity-ranked, evidence-linked, framework-cited: an artifact a client pays for.
For agency and services designers — and Indian studios serving US/EU clients in particular — this is a directly sellable SKU: a fixed-scope AX audit with a reusable instrument, deliverable in days, that most clients have never been offered. Lab 15, the capstone, has you build the instrument and run it on a real public AI product.
Three review areas complete the instrument — and your professional duty of care.
Disclosure and identity. Decide how the product discloses AI involvement: content labels and watermarks on generated media, AI-distinct visual identity (color, iconography, an avatar clearly not a person) versus deliberate blending, and disclosure timing — at the interaction, not in a settings page. The EU AI Act's transparency duties make several of these obligations rather than preferences for products touching EU users: users must know when they're interacting with AI, and synthetic content must be identifiable.
The dark-pattern review. AI features open a new manipulation surface, and an honest review names it:
Produce the red-flag checklist and have the team sign it — the signing is the point; it converts "we'd never do that" into a reviewable commitment.
Consent and data provenance. Design the consent moments for AI processing of user content (contextual, at first relevant use — not page four of onboarding), provenance display for what feeds training and memory, and org-level admin controls for enterprise customers. For Indian teams this is DPDP Act territory — consent rules apply to research-participant and user personal data alike — and for services firms it's contractual: client data flowing into AI features needs the client's paper trail, not just the user's.
Regulatory specifics move — the EU AI Act's transparency provisions phase in over time and DPDP rules continue to be notified. Design the disclosure and consent patterns now; verify current legal requirements with counsel at ship time rather than from memory or from this page.
This track ends where it began, with the counter-signal worth keeping: generative tools speed up production dramatically, and they do not replicate designer insight — NN/g's assessment, and the observed reality of every module here. The chatbot that failed for lack of onboarding, the agent disabled for lack of undo, the test that passed on luck: none of these were model failures. They were judgment gaps, and judgment is what this certification actually tests — because it's what the market actually pays for.
Non-deterministic products need non-deterministic evaluation: success defined over outcome classes, failure seeded on purpose, reliance measured for calibration, and experience graded by rubrics design writes. Prove quality that way, review responsibility the same way — and the AX evaluation you can hand a client becomes the proof that the judgment, not the generation, was always the job.