Catalog Failure Modes and Fallbacks as Numbered Requirements

Workflow area: AI Requirements & Agent Redesign · Tier: Advanced · Hands-on time: ~45 minutes

Deterministic systems fail loudly — exceptions, error codes, timeouts. AI features fail politely: the answer arrives on time, well-formatted, and wrong. That inversion is why failure handling cannot be left to developers to improvise; it is requirements work. The BA who catalogs failure modes and specifies the response to each has written the part of the spec that determines whether the first production incident is a logged, handled event or a customer-facing disaster with no playbook.

When to use — and when not to

• Use it for every AI feature spec, after the happy path stabilizesthe catalog is the companion document to your eval-based acceptance criteria (pb-43); ship them together.
• Use it when reviewing a vendor's or team's draft specthe fastest expert review you can run is checking which of the standard failure modes have no requirement; there are usually three or more missing.
• Use it as input to oversight and incident designfailure modes with high impact become human-approval points (pb-45) and incident classes (pb-52); the catalog is the bridge artifact.
• Skip the exotic modes for low-stakes internal toolsan internal meeting summarizer needs hallucination and stale-data rows, not a nation-state threat model; scale rigor to blast radius.
• Skip it as a substitute for security reviewprompt injection and data exfiltration rows define required behavior; penetration testing and security architecture remain specialist work.
• Do not write rows you cannot testa failure mode without a plausible detection signal or test case is a risk-register entry, not a requirement; park it there honestly.

The recipe

1. 1.

   Start from the standard catalog, then localize it. The recurring failure classes for LLM features: fabrication (invented facts, citations, amounts), confident misclassification, unjustified refusal or over-escalation, stale knowledge (the model's world ended before your fiscal year), context overflow (long inputs silently truncated), prompt injection (instructions smuggled in via processed content), personal-data leakage in outputs, latency degradation, cost runaway, and upstream model-version drift. For each, ask the SME: what does this look like in OUR process, and what does it cost when it happens?

2. 2.

   Generate the localized catalog with AI, against your feature spec. Prompt:

   Copy

   You are a requirements analyst. Against the AI feature described
   below, produce a failure-mode catalog as a table:
   ID | Failure mode | Concrete example in THIS feature |
   Detection signal | User-visible behavior | Fallback |
   Human handoff | Logged data
   Cover at minimum: fabrication, confident misclassification,
   refusal/over-escalation, stale data, truncated context, prompt
   injection via processed content, personal-data leakage, latency
   > [X]s, cost > [Y]/case, model-version drift.
   Rules: examples must be realistic for this feature and audience.
   If a detection signal is not feasible, write DETECTION GAP —
   do not invent one.
   Feature spec: [PASTE]
   

   Review every row — the model will propose plausible detection signals that your architecture cannot actually produce; each of those becomes either a new technical requirement or an honest DETECTION GAP.

3. 3.

   Rewrite each row as numbered requirements. One failure mode yields a small family of clauses. The pattern:

   Copy

   FM-03  Fabricated reference data
   R-FM-03.1  The system SHALL validate every extracted policy
              number against the policy master before display;
              non-matching values render as "unverified" with
              the source snippet, never as data.
   R-FM-03.2  WHEN validation fails, the case SHALL route to the
              review queue within 60s with reason code FAB-REF.
   R-FM-03.3  The user-facing message SHALL NOT display the
              fabricated value as fact.
   R-FM-03.4  Each FAB-REF event SHALL be logged with input hash,
              model version, prompt version, and output — retained
              per the audit-log requirement (see pb-45 clauses).
   

   EARS phrasing (WHEN/WHILE/IF-THEN) keeps every clause atomic and testable — the same lint discipline as pb-19, applied to failure paths.

4. 4.

   Specify the fallback ladder, not a single fallback. For each high-impact mode, define graduated behavior: degrade (show the source document instead of the summary), divert (route to the human queue with context attached), disable (feature-level kill switch, who can pull it, what users see when it's off). The kill switch is a requirement with an owner and an SLA — write it, because pb-52's incident process will depend on it existing.

5. 5.

   Attach test cases and close the loop with the eval set. Every detection and fallback clause gets at least one test case: seeded fabrications, an injection string embedded in a processed document, a forced timeout. Add the adversarial cases to the golden set (pb-48) so regression runs exercise failure paths forever, and hand the catalog's high-impact rows to whoever owns the oversight spec (pb-45) and incident matrix (pb-52).

Make it yours

• Workshop format: run the catalog as a 90-minute pre-mortem with SME, developer, and support lead — "it's six months from now and this feature caused an incident; what happened?" — then convert the wall into rows.
• NIST AI RMF mapping for regulated contexts: tag each row with the RMF function it serves (Map/Measure/Manage) so the catalog doubles as audit evidence.
• Severity-weight the catalog at scale: score rows by likelihood × impact and let the score drive how much requirement machinery each mode gets — ten deep rows beat forty shallow ones.
• Vendor-feature variant: for bought features, the catalog becomes a questionnaire — "what is your detection signal for X?" — and unanswered rows become contract clauses or accepted, documented risks.

Quality check

• The standard classes are all present or consciously excludedevery skipped mode has a one-line justification, not an oversight.
• Examples are localizedeach row names what the failure looks like in this feature, with this data, for these users; generic rows teach nothing.
• Detection signals are realeach is technically producible in the actual architecture, or the row honestly says DETECTION GAP with an owner.
• User-visible behavior is specified for every modewhat the user sees during failure is never left to the front-end developer's imagination at 5pm.
• Fallbacks are requirements with SLAsrouting times, queue destinations, and the kill switch all have numbers and owners.
• Every row has a testseeded failure cases exist in the eval set and run on every regression cycle.