Your AI feature will hallucinate, refuse, go stale, get injected and time out — the only question is whether the product has a designed response or an accident. This playbook catalogs every failure mode with its user impact, detection signal and product response, and ships it as a PRD annex the whole team builds against.
Workflow area: Evals & AI Quality · Tier: Advanced · Hands-on time: ~40 minutes
Deterministic features fail into error states someone designed. AI features fail into outputs — fluent, confident, wrong — and if nobody designed the product's behavior for those moments, users get the raw failure at full strength. The failure-mode catalog is the document that turns "the model sometimes gets it wrong" from a disclaimer into engineering requirements: per mode, what the user experiences, how the system detects it, and exactly what the product does about it — fallback UX, escape hatches, human handoff. It is also, not incidentally, the document that answers the hardest AI-PM interview question: "what happens when your feature fails?"
Enumerate the modes — seeded, then brainstormed. Start from the standard five that apply to nearly every LLM feature — hallucination (confident fabrication), wrong refusal (declining benign requests), staleness (grounding on outdated knowledge), prompt injection (instructions smuggled through inputs), and latency/outage — then add your feature's specifics from pb-36's taxonomy and a structured brainstorm:
For this AI feature, enumerate failure modes beyond the standard
five (hallucination, wrong refusal, staleness, injection,
latency/outage). Consider: retrieval failures, multi-turn context
loss, harmful/off-brand tone, PII leakage in outputs, cost
runaway, degradation under non-English input, and modes specific
to this domain. For each: one-line description and a concrete
example of what the user would see.
Feature: [PASTE description, knowledge sources, user population,
actions the feature can take]
Curate the output — merge duplicates, cut the implausible, and keep only modes you can describe as something a user experiences.
Fill the catalog row by row. The schema that makes it engineering-consumable:
MODE: Stale-knowledge answer
USER IMPACT: acts on outdated policy → failed request, S1
FREQUENCY: 14% of failures (error analysis 2026-07)
DETECTION: retrieval-doc age check; eval regression on
stale-trap cases; complaint keyword monitor
PRODUCT RESPONSE: answers cite source + last-updated date;
retrieval filters superseded docs; if only stale docs match,
say so and link the doc instead of answering
EVAL COVERAGE: pb-37 cases #12-#16
OWNER: retrieval — eng; citation UX — design
The two columns teams skip are the two that matter: DETECTION (how will we know this is happening — a monitor, an eval, a complaint pattern; a mode with no detection signal is invisible in production) and OWNER (a row nobody owns is a wish).
Design the degradation UX per mode — the annex's second half. For each mode, specify what the user actually sees, at wireframe-note fidelity: low-confidence states show sources and hedged copy rather than fluent certainty; empty-retrieval states say "I couldn't find this in our docs" with a search link — never freestyle; timeout states fail into the non-AI path with an apology, not a spinner; repeated-failure sessions proactively offer the human handoff. Every state needs an escape hatch — the user's route to accomplishing their goal without the AI. Design owns these screens; the annex is where they're commissioned.
Run the adversarial pass on your own catalog. Fresh AI context: "Here is our failure-mode catalog. What failure modes are missing? Which detection signals would fail to fire in practice? Which product responses would make the user experience worse?" — the responses critique is the valuable one; over-eager fallbacks (hedging on every answer, handoff prompts after one hiccup) degrade the feature for the 90% of interactions that succeed.
Ship it as a PRD annex and wire the dependencies. Attach the catalog to the PRD next to pb-34's requirements annex. Then make the connections explicit: every S1 mode has eval cases (pb-37) — add any missing ones now; detection signals become monitoring requirements (pb-41); response procedures feed the rollout plan's kill criteria (pb-43) and the incident-response classes (pb-52). A catalog that doesn't cross-reference these is a standalone document waiting to go stale.
Review it live, quarterly. Each pb-36 error-analysis round updates frequencies, adds discovered modes and retires theoretical ones. The version history — "injection moved from theoretical to observed in v1.3" — is itself a quality narrative your team and your auditors will want.
The catalog itself is sensitive — it is a map of exactly how to make your product fail, including injection vectors. Keep it in internal-only workspaces, exclude it from public docs and vendor shares, and treat the injection rows with the same handling as security findings.
Users forgive an AI feature that fails gracefully and lose trust forever in one that fails confidently — the catalog is where you decide, mode by mode, that every failure lands on a designed surface with a detection signal, an owner and a way out.