Graceful-Degradation Ladders for AI Features

Tier: Core · Hands-on time: ~30 minutes

The resilience-pattern literature this track draws on (AWS's Bedrock-with-gateway guidance, the Azure multi-region practitioner corpus) establishes the failure modes; the degradation-ladder framing is this track's synthesis of how a feature should absorb them. The architectural insight is that degradation is a product decision executed by infrastructure: which rungs exist, what quality each sacrifices, and what the user is told are choices that must be made with product owners at design time — because during an incident, nobody is convening that meeting.

When to use

• Any user-facing AI feature heading to productionthe ladder section belongs in its design doc alongside HA (pb-44).
• An incident review revealed all-or-nothing behaviorthe model hiccupped and the feature simply died.
• Cost guardrails need teeth: pb-43's budget alerts want a pre-agreed "run cheaper" mode to switch to.
• Skip AI when the feature is internal, low-stakes, and its honest fallback is "try again later" — a clear error message is a legitimate one-rung ladder; write the sentence and move on.

Prerequisites

• The feature's quality floor, from product: what is the worst experience that is still worth shipping to a user?
• Gateway fallback capability (LiteLLM-class config or equivalent) as the enforcement layer.
• A smaller candidate model and a caching layer as prospective rungs, with pb-46-style eval evidence for the model swap.
• A chat assistant to stress the ladder against failure scenarios.

The workflow

1. 1.

   Define the rungs top to bottom. A typical ladder: full experience (frontier model, full context); reduced (smaller model, or trimmed retrieval context, or shorter outputs); cached (serve semantically-matching previous answers, clearly marked as possibly stale); static (honest capability-down message with the non-AI path — search box, docs link, human handoff). Each rung gets a quality note and a cost note; not every feature needs all four.

2. 2.

   Specify the triggers per transition. What moves traffic down a rung: latency past the pb-44 budget, 429 cooldown active, error-rate threshold, cost guardrail breach (pb-43), or a manual switch. And — the forgotten half — what moves it back up: recovery probes, cooldown expiry, human decision. Hysteresis matters; a ladder that flaps between rungs every thirty seconds is its own outage.

3. 3.

   Design the UX states with product, now. Each rung has a user-facing truth: does the user see a "responses may be simpler right now" notice? Are cached answers labeled? Does the static rung apologize and route to a human? Rajesh, specifying the Meridian support copilot's ladder:

   Copy

   Draft the degradation ladder spec for a retail support copilot.
   Rungs: (1) frontier model + full RAG; (2) small model + top-3 chunks
   only; (3) semantic cache of previous answers, staleness-labeled;
   (4) static: "assistant unavailable" + order-status lookup form +
   human chat queue.
   
   For each rung and transition, specify: trigger condition and its
   metric source (gateway latency p95, 429 cooldown state, error rate,
   cost breach); recovery condition with hysteresis; the exact user-facing
   state (copy tone: honest, no fake confidence); what gets logged so we
   can count time-per-rung later; and which transitions page a human vs
   proceed automatically.
   
   Then attack it: find the scenario where this ladder makes the
   experience WORSE than failing fast, and the rung most likely to
   silently serve wrong answers.
   

   The attack matters — cached answers about changed prices, or a small model confidently wrong about policy, are ladder-induced incidents.

4. 4.

   Put the kill switches on the ladder. A per-feature kill switch is the ladder's bottom transition, executable by a human in seconds, mapped to the static rung (pb-38's kill-switch design generalizes to non-agentic features). Decide who may pull it and confirm the path from decision to effect is shorter than an incident call.

5. 5.

   Gate the model-swap rung with evals. The reduced rung is only legitimate if the smaller model's quality on your golden set (pb-33/pb-46) is measured and product has accepted the delta. An unevaluated fallback model is not a rung; it is a different, untested product.

6. 6.

   Implement in the gateway, not scattered app code. Fallback chains, cooldowns, and cache serving belong in the gateway configuration where they are observable and consistent across callers; the app layer owns only the UX states. Hand the config and its dashboards to the operations side with the transition log spec.

7. 7.

   Drill it before launch. Force each trigger in staging (mock 429s, latency injection, cost-breach flag) and watch the feature step down and recover, screenshotting each UX state. The drill record joins the design package.

Verify before you trust it

• Every transition has been forced at least once. Down and up, in staging, with the transition visible in gateway logs and the correct UX state rendered — untested rungs fail in the direction of most embarrassment.
• The cached rung is staleness-bounded. Verify cache entries carry timestamps and TTLs, staleness labels render, and price/policy-sensitive content classes are excluded from cache serving by rule, not hope.
• The reduced rung's quality delta is a measured number. Eval results for the smaller model on the golden set are attached to the design, with product's sign-off on the delta recorded.
• Flapping is bounded. Simulate an oscillating trigger and confirm hysteresis holds the rung stable; count transitions per hour and check it against the design's stated maximum.
• The kill switch is timed from human decision to user effect. Someone actually pulls it in staging and the stopwatch number goes in the runbook.

Restricted-environment variant

Degradation design is architecture, not data — the ladder spec drafts safely in any approved assistant with the feature described generically. Constraints bite at the rungs: client MSAs may fix the model list (no arbitrary small-model rung without approval), and cached answers derived from client data must live inside the client boundary with the client's retention rules applied to the cache. In client-tenancy deployments, implement the ladder in a gateway running inside their subscription; drills use synthetic traffic only. For the strictest environments, note that the static rung — honest unavailability plus the non-AI path — is always contract-safe, and designing it well is the highest-value hour of this playbook anyway.