Take an AI feature and its eval-failure list and design the complete failure-state kit — wrong-but-plausible, stale, partial, refused, timeout, hallucinated citation — with a distinct designed state for each, plus a confidence-display memo arguing when showing uncertainty helps and when it harms. An error banner on a wrong-but-plausible answer fails the lab.
Dario is staff designer at Meridiem, a fictional clinical-notes platform for physiotherapy clinics, whose AI feature "Visit Summary" drafts the after-visit summary from the therapist's dictated notes: what was treated, exercises assigned, progress against the care plan. The eval team just delivered their failure analysis, and it ends the fantasy that failure is one state: the feature fails six different ways, and five of them look nothing like an error. The most dangerous output the system produces is fluent, well-formatted, and wrong. A red banner cannot fix that, because nothing looks broken. This lab is where you design a distinct state for each failure class — and take a defensible position on the confidence-display question instead of sprinkling percentages on everything.
VISIT SUMMARY — behavior
Input: therapist's dictated note (auto-transcribed).
Output: structured summary — treatments performed, exercises
assigned (with reps/frequency), progress note, next visit.
Sent to the patient portal AFTER therapist approval.
EVAL-FAILURE LIST (from 400-case eval run)
F1 WRONG-BUT-PLAUSIBLE (6.5% of cases) — summary asserts a
treatment or rep count not in the dictation; fluent and
well-formed. Ex: dictation says "theraband rows, 2x12",
summary says "3x12"; dictation never mentions ultrasound,
summary includes "ultrasound therapy 10 min".
F2 STALE CONTEXT (3%) — summary references the PREVIOUS
visit's care plan when the plan changed today. Ex: keeps
"continue knee protocol" after therapist dictated a
switch to hip protocol.
F3 PARTIAL (5%) — transcription gaps produce a summary
missing whole sections; model silently omits rather
than flags. Ex: exercise section absent, no mention
that it's absent.
F4 REFUSED (1%) — model declines when dictation contains
ambiguous medication references (by policy). Currently
shows: "Something went wrong."
F5 TIMEOUT (2%) — generation exceeds 30s on long
dictations. Currently shows a spinner forever.
F6 HALLUCINATED CITATION (1.5%) — summary cites a care-plan
document or a prior-visit note that does not exist.
Ex: "as outlined in your Phase 3 plan (12 May)" — no
such document.
Sort the six failures by detectability before designing anything. Three bins: system-detectable (the product knows it failed: F4, F5, and F6 when the cited document can be checked against the record), system-suspectable (signals exist — transcription-confidence gaps for F3, plan-edit timestamps for F2), and system-blind (F1: nothing in the output distinguishes wrong from right). Write the bin next to each failure. This taxonomy drives every design decision after it: you cannot show an error you cannot detect.
Design the wrong-but-plausible state first — as verification affordances, not an error state. Since the system is blind to F1, the design target is the reviewing therapist, and the tool is comparison: side-by-side source-and-summary with every extractable claim in the summary linked to its dictation span; claims with no source span auto-highlighted as "not found in dictation — verify or remove"; numbers (reps, durations) rendered as tap-to-confirm chips. The therapist approves claim-by-claim for flagged items, not page-at-once. Write the microcopy for the unsourced-claim flag.
Design the stale and partial states from their signals. F2: when the care plan was edited during today's visit, the summary carries a banner-level notice at the plan-reference lines — "Care plan changed today at 14:32 — this line references the old plan. Update?" F3: transcription gaps become visible structure — "Exercises: [2 min of audio couldn't be transcribed — section may be incomplete]" with a replay-audio affordance. The shared principle: silence is the failure; the design makes the absence visible where it occurred.
Redesign the refusal and timeout. F4 is a policy refusal wearing a crash costume. Rewrite: what happened ("This dictation mentions medication changes, which Visit Summary doesn't draft by policy"), why it is a feature ("medication instructions need your exact wording"), and the path ("draft everything else and leave the medication section to you" — partial capability beats binary refusal). F5: kill the infinite spinner — honest progress ("Long dictation — about 40s"), a cancel that preserves the dictation, and a fallback ("start from the structured template instead").
Design the hallucinated-citation state. F6 is detectable by lookup: every document the summary cites gets checked against the record at render time. Cited-and-found: normal link. Cited-and-missing: the reference renders as a flagged claim — "cites a document we can't find — remove or correct before sending" — and blocks approval until resolved. Decide and document the blocking rule: this is patient-facing clinical text; a fabricated reference must be impossible to approve unseen.
Write the confidence-display decision memo. One page, taking positions: Where does showing uncertainty help? (Per-claim provenance flags and "not found in dictation" markers — they direct verification effort to specific spans.) Where does it harm? (A global "87% confident" score on the summary: uncalibrated, unactionable, and it trains page-level trust — therapists learn to approve high numbers, which is exactly how F1 sails through.) State your rule — e.g., show uncertainty only where it is attached to a specific verifiable claim and a specific action; never as a global score — and defend the accept-anyway case: what the design does when the therapist overrides a flag (log it; feed it to evals).
Stress-test the kit against compound failures. Ask your AI: "Given these six designed states, generate five realistic cases where two failures co-occur (e.g., stale + partial, wrong-but-plausible + hallucinated citation). For each, walk through what the UI shows." Fix collisions — two banners fighting, a blocked approval hiding an unsourced-claim flag, flag fatigue when many claims fire at once (design the >5-flags behavior: summarize flags into a review checklist instead of confetti).
Assemble the kit. Six states, each specified as: trigger (from the eval list), detectability bin, UI behavior, microcopy, and what the user can do next. Plus the confidence memo and the compound-failure notes. Format it as the document the eval team and engineers iterate against next quarter — failure rates change; your states should reference failure classes, not today's percentages.
The failure that ships most damage is the one that renders beautifully — wrong-but-plausible output needs verification affordances built into the reading experience, because by definition no banner will ever know to appear.
Failure is not a state, it is a taxonomy — sort every failure by what the system can know about it, tell the truth loudly where it can know, and rebuild the reading experience around verification where it cannot.