Lab 11 · Quota-Aware Failover Design + Simulation

The Meridian Retail order-status agent has a launch date in the same week as a Meridian mega-sale, and Rajesh's reliability review has one finding in red: the design treats the LLM endpoint like a stateless web backend. It is not. LLM capacity fails in two distinct ways — quota exhaustion (429: the service is healthy, your allocation is spent) and service failure (5xx: the service is broken) — and a classic load balancer health check cannot tell them apart. Failing over on 429 to a region sharing the same quota pool solves nothing; retrying a 429 in-region burns latency against a wall that will not move for seconds. This lab makes you design the ladder, then run it against a scheduled storm.

Time & tier

~120 minutes hands-on · Advanced tier (maps to module M7). Runs at ~₹0 — the "providers" are local mock servers, the gateway is self-hosted LiteLLM (MIT-licensed), and no real model API is called at any point.

Setup

mkdir lab-11 && cd lab-11
python -m venv .venv && source .venv/bin/activate
pip install fastapi uvicorn "litellm[proxy]" openai

All capacity numbers in this lab are synthetic. Where your real design needs quota figures, the taught reflex applies: pull today's numbers from your provider's quota console and pricing pages — never from memory, and never from a model.

Guided steps

Step 1 — Draft the HA design doc before touching code (25 min)

Create ha-design.md with four sections, written for the Meridian ARB:

1. 1Failure taxonomy. A table with rows 429 with Retry-After, 429 without Retry-After, 5xx, timeout, and columns what it means · correct reaction · wrong-but-common reaction. Draft it yourself, then have your assistant red-team it.
2. 2Quota model. A synthetic per-region budget table (e.g., region A: 400K TPM / 600 RPM; region B: 200K TPM / 300 RPM — clearly invented) and one paragraph on why cross-region failover only helps if the regions draw on separate quota pools (separate allocations, or separate accounts — account sharding).
3. 3The degradation ladder. Four rungs with explicit UX per rung: frontier model (full answer) → small model (shorter, "fast mode" banner) → cached response (last known answer for repeatable intents, staleness label) → static fallback (deterministic self-service pointer). Every rung answers something; no rung shows a spinner that ends in an error.
4. 4Control placement. One diagram: clients → LLM gateway (where 429-vs-5xx intelligence lives) → providers. State why this cannot live in a plain load balancer: the LB sees connection health, not allocation state.

✅ Checkpoint: ha-design.md has all four sections; the failure-taxonomy table distinguishes the reaction to 429 from the reaction to 5xx.

Step 2 — Build the mock provider with a 429 schedule (20 min)

One FastAPI app plays two providers: mock-frontier (good answers, throttles on a fixed schedule) and mock-small (always up, faster, plainer). Create mock_provider.py:

# mock_provider.py — OpenAI-compatible mock; frontier 429s on a schedule
import time, itertools
from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse

app = FastAPI()
counters = {}

def reply(model, text):
    return {"id": "mock-1", "object": "chat.completion", "model": model,
            "choices": [{"index": 0, "finish_reason": "stop",
                         "message": {"role": "assistant", "content": text}}],
            "usage": {"prompt_tokens": 40, "completion_tokens": 25,
                      "total_tokens": 65}}

@app.post("/v1/chat/completions")
async def chat(req: Request):
    body = await req.json()
    model = body.get("model", "mock-frontier")
    n = next(counters.setdefault(model, itertools.count(1)))
    # Schedule: frontier throttles requests 4,5,6 of every block of 6.
    if model == "mock-frontier" and (n - 1) % 6 >= 3:
        return JSONResponse(
            status_code=429, headers={"Retry-After": "20"},
            content={"error": {"type": "rate_limit_error",
                     "message": "TPM quota exceeded for mock-frontier"}})
    time.sleep(0.15 if model == "mock-frontier" else 0.05)
    tag = "FRONTIER" if model == "mock-frontier" else "SMALL"
    return JSONResponse(reply(model, f"[{tag}] Order 8841 shipped 26 Jul, "
                                     f"ETA 30 Jul. (answer #{n})"))

Run it: uvicorn mock_provider:app --port 9001. Curl it a few times and watch the 4th–6th frontier calls return 429 with Retry-After: 20.

✅ Checkpoint: curl shows the deterministic pattern — three 200s, three 429s, repeating — and the 429 body says quota, not error.

Step 3 — Stand up the gateway, naive config first (20 min)

Create config_naive.yaml — the configuration the original design implied: retry the same model on failure, no fallback:

# config_naive.yaml — what "just add retries" looks like
model_list:
  - model_name: frontier
    litellm_params:
      model: openai/mock-frontier
      api_base: http://localhost:9001/v1
      api_key: mock
  - model_name: small
    litellm_params:
      model: openai/mock-small
      api_base: http://localhost:9001/v1
      api_key: mock
litellm_settings:
  num_retries: 3          # same-target retries — the trap
  request_timeout: 30

Start it: litellm --config config_naive.yaml --port 4000.

✅ Checkpoint: gateway is up on :4000 and forwards a request to the mock ([FRONTIER] … comes back).

Step 4 — Replay the workload and measure the naive baseline (15 min)

Create driver.py — 30 requests through the gateway, logging which rung answered and how long the user waited. It also carries the ladder's bottom rungs client-side:

# driver.py — replay 30 requests; log rung + user-visible latency
import csv, sys, time, openai
client = openai.OpenAI(base_url="http://localhost:4000", api_key="sk-local")
Q = "Where is order 8841?"
CACHE = {Q: "[CACHED] Order 8841: shipped 26 Jul (as of 09:10 IST)."}
STATIC = "[STATIC] High demand right now - track any order at meridian.example/track."
rows, counts = [], {}
for i in range(1, 31):
    t0, rung = time.time(), "static"
    try:
        r = client.chat.completions.create(
            model="frontier", messages=[{"role": "user", "content": Q}])
        text = r.choices[0].message.content
        rung = "frontier" if "[FRONTIER]" in text else "small"
    except Exception:
        rung, text = ("cached", CACHE[Q]) if Q in CACHE else ("static", STATIC)
    dt = round(time.time() - t0, 2)
    counts[rung] = counts.get(rung, 0) + 1
    rows.append([i, rung, dt, text[:48]])
with open(sys.argv[1], "w", newline="") as f:
    w = csv.writer(f); w.writerow(["req", "rung", "latency_s", "preview"])
    w.writerows(rows)
print(counts, "worst latency:", max(r[2] for r in rows), "s")

Run python driver.py naive.csv. Read the numbers like an SRE: every throttled request either burned ~3 same-target retries before erroring into the client-side cache, or waited out retry backoff. Note the worst user-visible latency and how many requests never got a model answer.

✅ Checkpoint: naive.csv exists; you can state the naive config's rung distribution and worst-case latency in one sentence.

Step 5 — Enable the ladder and measure again (20 min)

Create config_ladder.yaml — same models, but the gateway now expresses the design: no same-target retries on quota errors; fall to the next rung immediately:

# config_ladder.yaml — fallback-first: 429 means "go down a rung now"
model_list:
  - model_name: frontier
    litellm_params:
      model: openai/mock-frontier
      api_base: http://localhost:9001/v1
      api_key: mock
  - model_name: small
    litellm_params:
      model: openai/mock-small
      api_base: http://localhost:9001/v1
      api_key: mock
litellm_settings:
  num_retries: 0
  request_timeout: 30
  fallbacks:
    - {"frontier": ["small"]}

Restart the gateway with the new config, restart the mock (to reset counters), and run python driver.py ladder.csv. Compare: every request in the throttle window should now land on [SMALL] in well under a second, and no request should reach the client-side rungs at all.

✅ Checkpoint: ladder.csv shows ~half the requests answered by small, zero exceptions, and worst-case latency an order of magnitude below the naive run.

Step 6 — Kill the provider entirely and watch the bottom rungs (20 min)

Stop mock_provider.py mid-run (simulate a true 5xx/outage rather than a quota event) and run python driver.py outage.csv. Now the gateway has nowhere to go, the exception path fires, and the cached rung answers with its staleness label — the user still gets their order status, tagged with "as of 09:10 IST." Change Q to a question not in the cache and confirm the static rung catches it. Then write the closing section of ha-design.md: a three-row results table (naive / ladder / outage) with rung distribution and worst latency, plus the operational note that 429 recovery is scheduled (Retry-After says when capacity returns) while 5xx recovery is unknown (circuit-break and probe).

✅ Checkpoint: three CSVs exist; ha-design.md closes with the measured comparison and one paragraph distinguishing scheduled from unknown recovery.

Verification

• Design docha-design.md contains the failure taxonomy, synthetic quota model, four-rung ladder with UX states, and gateway placement rationale.
• Deterministic stormthe mock's 429 schedule is reproducible (three throttled per six), with Retry-After set.
• Measured contrastnaive.csv vs ladder.csv shows the same workload with materially different user-visible behavior, and you can explain why from the configs alone.
• Bottom rungs provenoutage.csv shows cached and static answers with no unhandled errors.

Completion artifact

The lab-11/ repo: ha-design.md, mock_provider.py, both gateway configs, driver.py, and the three simulation logs. Self-grading rubric (pass = 8/10):

• 429-vs-5xx clarity (0–3): taxonomy table has correct, distinct reactions per row, including the Retry-After nuance = 3.
• Ladder completeness (0–2): four rungs, each with an explicit UX state; no rung ends in a raw error = 2.
• Simulation fidelity (0–2): all three runs logged; rung distributions match the configs' predicted behavior = 2.
• Quota reasoning (0–2): design explains separate quota pools / account sharding as the precondition for useful regional failover = 2.
• Verification reflex (0–1): doc states that real quota and pricing figures must be read from the provider console today, not assumed = 1.

Client-safe variant

Nothing in this lab touches a real provider, so it runs unmodified inside any client network — the mocks and gateway are local processes. For a real engagement: the design doc's structure transfers as-is, but populate the quota model from the client's own provider consoles under their credentials, run LiteLLM (or the client's approved gateway/APIM equivalent) inside their tenancy, and keep mock-based simulation as the pre-production test — it is precisely the storm you cannot ethically generate against shared production quota.

Stuck?

• •
  Gateway returns 500 instead of falling back: check the fallbacks syntax matches your LiteLLM version (list of single-key dicts) and that num_retries is 0 in the ladder config.
• •
  Rung counts look wrong after re-runs: the mock's counters persist per process — restart uvicorn between measured runs to reset the schedule.
• •
  Everything answers [FRONTIER] in the naive run too: your driver retried until it left the throttle window — confirm the mock throttles 3-of-6 and that you ran exactly 30 requests.
• •
  openai client raises on 429 immediately: that is the naive path working as designed — the exception falls to the client-side cache rung; log it, don't fix it.
• •
  Port already in use: previous gateway/mock still running — pkill -f litellm; pkill -f uvicorn and restart.

Answer key (don't read before attempting)

The seeded traps and what they teach:

1. 1The naive config's num_retries: 3 is the primary trap. Same-target retries against a quota wall convert a capacity event into a latency event: the user pays for three doomed round-trips (plus backoff) and still gets no frontier answer. The mock's Retry-After: 20 makes this vivid — the wall's height was announced, and the retries ignored it.
2. 2The 429 body says quota, and that changes the failover math. Falling over to another region only helps if that region draws on a separate allocation. If your design doc's quota model let both regions share one pool, the whole regional story was decorative — this is why account sharding and per-region allocations appear in resilience patterns.
3. 3The outage run is a different disease. With the provider dead you get connection errors, not 429s: recovery time is unknown, so the correct control is a circuit breaker with probes — versus the 429 case where recovery is scheduled and the correct control is the ladder plus honoring Retry-After. If your closing paragraph gave both cases the same cure, revisit the taxonomy.
4. 4The cached rung must carry a staleness label.[CACHED] … (as of 09:10 IST) is the difference between graceful degradation and quietly lying about order status — a UX state is part of the architecture, not a frontend nicety.
5. 5What the lab deliberately simplified: one gateway instance (production wants the gateway itself HA), one intent (production caches per intent+entity), and client-side bottom rungs (production usually puts cached/static in the gateway or edge so every client benefits). Naming these gaps in your design doc is worth more than pretending the POC is the architecture.