LLM endpoints fail differently: the thing that takes you down is usually a quota, not an outage — and a naive load balancer cannot tell a 429 from a failure. This playbook designs quota-aware HA: gateway-layer routing intelligence, account sharding, cross-region inference, and capacity as a first-class design constraint.
Tier: Advanced · Hands-on time: ~40 minutes
The reference literature is unusually direct here. AWS's resilience-patterns guidance for Bedrock with an LLM gateway centers on quota exhaustion, account sharding, and cross-region inference profiles; Azure practitioner architectures document that Front Door cannot distinguish a 429 from a real failure — which is why the published pattern routes through APIM-layer intelligence — and that Azure OpenAI provides no automatic failover, so the application must own routing. Per-region TPM/RPM quotas are documented constraints, not folklore. Classic HA instincts transplant badly onto this surface; this playbook rebuilds them.
Write the two failure taxonomies side by side. Infrastructure failure (5xx, timeout: the endpoint is unhealthy — retry elsewhere immediately) versus capacity exhaustion (429: the endpoint is healthy and telling you to slow down — retrying the same endpoint harder makes it worse). Every downstream decision (retry policy, health checks, failover triggers) forks on this distinction, which is exactly what naive load balancers cannot see.
Budget quota like capacity, per region. Tabulate TPM/RPM per region/model against peak demand with headroom for retries and growth. If one region's quota cannot absorb another region's failed-over traffic on top of its own, your "multi-region HA" is a diagram, not a capability — this arithmetic is the design's honesty test.
Place the intelligence at the gateway. The gateway (LiteLLM-class or APIM pattern) is where 429-aware behavior lives: distinct handling per status class, token-bucket awareness, cooldown on quota-exhausted endpoints, priority tiers so interactive traffic pre-empts batch under pressure. DNS and L7 load balancers stay out of the decision — the published Azure pattern exists precisely because Front Door can't make it.
Design the capacity-diversity ladder. In escalating order: cross-region inference profiles where the provider offers them (verify current regional coverage in docs); account sharding to multiply per-account quotas (the AWS-documented pattern — with the ops tax of multiple accounts priced in); provisioned throughput for the guaranteed floor; and a second provider via the gateway as the last rung, with pb-46-style evals confirming the fallback model's quality is acceptable before an incident makes the swap for you.
Set retry and timeout budgets for LLM latency reality. Long generations mean generous timeouts; generous timeouts mean retry storms compound fast. Budget end-to-end: per-attempt timeout, max attempts across distinct endpoints for 5xx, backoff-and-respect for 429 (honor retry-after semantics where provided), and a total latency ceiling that hands off to pb-45's degradation ladder when exceeded.
Red-team the design with a quota-day scenario. Anita, hardening orbit-docs-copilot:
Attack this LLM HA design (attached: gateway config summary, two
regions with documented TPM quotas, account-sharding plan, fallback
provider) with five scenarios:
1. Marketing launch triples traffic at 10:00 IST — walk the 429
cascade minute by minute through my retry budgets.
2. Primary region hard-fails (real 5xx) — does failover traffic
plus native traffic exceed the secondary's quota? Show arithmetic.
3. A batch job saturates TPM at peak interactive hours — where does
priority tiering actually intervene?
4. The fallback provider is engaged for 6 hours — what quality and
cost deltas do users and finance see?
5. Retry amplification: every layer (SDK, gateway, app) retries —
compute worst-case request multiplication.
For each: which design element catches it, or FLAG if none does.
Simulate before you certify. Stand up the gateway with mock endpoints throwing scripted 429s and 5xxs; replay realistic traffic and measure user-visible behavior — latency distribution, error rate, fallback engagement. Ship the design with the simulation log attached; an HA claim without a drill is a wish.
Client engagements often pin you to one provider, one region, or client-subscription endpoints — which removes ladder rungs, not the discipline. Extract MSA constraints first, then budget honestly within them: single-region designs lean on provisioned throughput for the floor, aggressive priority tiering, and pb-45 degradation as the real resilience story — say so plainly in the design rather than gesturing at failover that contracts forbid. Simulation is fully restricted-safe: LiteLLM with mock endpoints and synthetic traffic runs on a laptop, no client data or live quota consumed. Where multi-account sharding is contemplated inside a client's organization, it needs their platform team's sign-off — put it in the design as a decision they own.
A 429 is a healthy endpoint saying slow down — route around it at the gateway, budget quota per region like the capacity it is, and believe no failover story until the simulation log shows it working.