Reliability, Evals & AI Cost Architecture

This is the hardening module: everything Modules 5 and 6 designed now has to survive contact with production traffic, regional quotas and a finance review. The industry's track record is the motivation — per FinOps Foundation ecosystem reporting, 73% of AI projects run over budget, even as 98% of practitioners now manage AI spend. The architects who avoid both fates design reliability and cost in, before launch week teaches the lesson the hard way.

The scenario — the 429 storm in launch week

Every element of Anita's fix list is a section of this module. The unifying idea: for LLM workloads, capacity, quality and cost are the same architecture problem — they share a gateway, they share instrumentation, and they fail together in launch week if any one of them was left to defaults.

What You'll Learn

• Quota-aware multi-region HAdesign routing that distinguishes 429 from 5xx, because naive load balancers can't; apply account sharding and cross-region inference profiles as capacity tools.
• Capacity planning under TPM/RPM quotasbudget per-region token throughput, weigh provisioned throughput against on-demand, and treat provider diversity as a capacity strategy.
• Graceful-degradation laddersdesign the frontier→small→cached→static fallback chain, kill switches, and retry budgets tuned for LLM latency and rate-limit semantics.
• Two-plane eval architecturean offline CI judge that gates deploys plus online sampled scoring, calibrated against human labels before either plane is trusted.
• The AI cost stacklayer prompt/prefix caching, semantic caching and confidence-gated routing, with per-layer hit-rate instrumentation and model choice as the biggest lever.
• TCO and launch forecastingbuild token unit-economics models with verified pricing, and forecast launch costs with explicit uncertainty bands instead of a single hopeful number.

Quota-aware HA — a 429 is not a 5xx

Classical HA assumes failure means a dead instance, and a load balancer solves it. LLM capacity breaks that model: the endpoint is healthy while its quota is exhausted, and the failure signal is a 429 status your L4/L7 infrastructure treats as just another response. The practitioner record on Azure is explicit: Front Door-class load balancing cannot distinguish a rate-limited backend from a failed one, and Azure OpenAI performs no automatic failover — the application must route. The same architecture lesson generalizes across providers; AWS's own resilience guidance for Bedrock prescribes gateway-layer intelligence for exactly this reason.

The design responses stack:

• Put the intelligence in a gateway, not a load balancer. An LLM gateway (LiteLLM-class, or APIM with policy logic on Azure) inspects response codes and routes on them: 429 means this deployment is out of capacity, try the next one; 5xx means this deployment is unhealthy, eject it. Practitioner APIM designs report high availability (one published pattern claims 99.95%) — treat the number as a vendor-adjacent claim, but the pattern as sound.
• Shard across accounts and regions. Quotas are typically per account per region — so multiple accounts and multiple regions are multiplicative capacity, not just redundancy. AWS's resilience patterns name account sharding explicitly.
• Use cross-region inference profiles where the provider offers them. Bedrock's cross-region inference spreads requests across regions under one profile — capacity pooling as a managed feature. Verify region pairs against residency constraints before enabling; convenient routing that crosses a compliance boundary is an incident of a different kind.
• Tune retries for rate-limit semantics. Retrying a 429 against the same target is a retry storm. Backoff must be honored, and the correct "retry" for an exhausted quota is usually a different region, account or model — which is a routing decision, which is why it lives in the gateway.

Capacity planning when capacity is a quota

LLM capacity planning is quota arithmetic. Providers enforce tokens-per-minute and requests-per-minute limits per model, per region (Azure OpenAI's per-region TPM/RPM quotas are the documented reference case) — so the capacity model is: expected requests × tokens per request, against the quota you actually hold, region by region.

• Budget tokens, not requests. A chatty RAG prompt with a fat context consumes quota far faster than request counts suggest. The tokens-per-interaction number from your cost model (below) is the same number that drives capacity — one model, two consumers.
• Provisioned throughput vs on-demand is an economics decision. Provisioned capacity buys guaranteed throughput at committed cost; on-demand is elastic until the quota wall. The crossover depends on utilization — steady high-volume traffic favors commitment, spiky launch-phase traffic favors on-demand with multi-region headroom. Model it; don't default it.
• Keep burst headroom explicit. Anita's storm was a marketing email. Launches, campaigns and seasonal peaks need a stated headroom multiple, and the gateway needs somewhere to route when it's exceeded — which is the degradation ladder's job.
• Provider diversity is a capacity strategy. A second provider is not only a resilience hedge; it is additional quota. The gateway abstraction is what makes it usable without rewriting the app.

The degradation ladder — designed, not improvised

When capacity runs out anyway — and launch week says it will — what users experience is a design decision made in advance. The ladder is the artifact:

Three design notes make the ladder real rather than aspirational:

• Each rung is pre-wired in the gatewayfallback order is configuration, exercised in a game day, not an incident-time improvisation.
• Kill switches are a product feature. The AI feature can be disabled per tenant and globally without a deploy — when the model misbehaves, when costs spike, or when a compliance question lands, "off" must be a setting.
• Timeout and retry budgets respect LLM latency. LLM calls are seconds, not milliseconds; a web-tier timeout tuned for microservices will kill healthy slow responses and double-spend tokens on retries. Budget end-to-end latency per rung, and make the UX degrade visibly and honestly — a labeled cached answer beats a spinner that dies.

Evals in production — the two-plane judge architecture

Reliability for AI features includes quality reliability: a model swap, a prompt tweak or corpus drift can degrade answers with every system green. The architecture answer is LLM-as-judge, named verbatim in the Amazon SA posting, deployed on two planes:

• Offline plane — the CI judge. A judge model scores every candidate change (prompt, model, retrieval config) against a labeled golden set, and a score regression blocks the deploy exactly like a failing test. This is the eval loop from Module 5, now generalized past RAG.
• Online plane — sampled production scoring. A judge scores a sample of live traffic continuously, feeding drift alerts and the dashboards the DevOps track operates. Sampling holds the cost line; the sample rate is a budget decision.

The 2026 practitioner consensus adds the calibration discipline that separates a judge from a vibe:

• Calibrate before you trust. Collect human labels (practitioner guidance says 500+) and measure judge–human agreement before the judge gates anything. An uncalibrated judge is a random number generator with authority.
• Judge from a different model family than the generatorsame-family judges share blind spots and flatter their siblings.
• Pairwise comparisons run both orderingsjudges exhibit position bias; A-vs-B and B-vs-A must both run.
• Distill a cheap judge for full coverage. Practitioner reports describe distilled evaluators at roughly 1/30th the cost scoring 100% of traffic — the expensive frontier judge calibrates the cheap one, the cheap one scales.
• Wire evals into observability. Trace tokens, cost, time-to-first-token, tool calls and eval scores together — trace-shaped, in your existing APM or a dedicated LLM-observability tool; you own the placement decision, DevOps owns the run-state.

from pydantic import BaseModel
from typing import Literal
import anthropic

class JudgeVerdict(BaseModel):
    faithfulness: int          # 1-5: grounded in the provided context?
    completeness: int          # 1-5: addresses the whole question?
    verdict: Literal["pass", "fail"]
    reasoning: str

client = anthropic.Anthropic()

def judge(question: str, context: str, answer: str) -> JudgeVerdict:
    resp = client.messages.parse(
        model="claude-opus-4-8",       # different family from the generator under test
        max_tokens=1000,
        thinking={"type": "adaptive"},
        system=(
            "You are a calibrated evaluation judge in a CI gate. Score strictly "
            "against the rubric. A 'fail' blocks the deploy; when uncertain, fail "
            "and explain — humans review failures, not passes."
        ),
        messages=[{
            "role": "user",
            "content": f"Question:\n{question}\n\nContext:\n{context}\n\nAnswer:\n{answer}",
        }],
        output_format=JudgeVerdict,
    )
    return resp.parsed_output   # CI compares against the calibrated baseline

The AI cost stack — caching, routing, tiering

AI cost is an architecture property, and the levers stack. Work them largest-first:

• Model choice is the biggest lever. The cost spread between frontier and small models for the same task runs around 50x — no cache or router recovers what over-modeling wastes. Right-size the default model per task; escalate on evidence.
• Prompt/prefix caching. Providers discount cached input tokens heavily — practitioner analyses put the input-cost reduction up to ~90%, with break-even around 1.4 reads of a cached prefix. Architectural consequence: structure prompts so the stable parts (system prompt, tool definitions, corpus preamble) form a cacheable prefix, and keep them stable.
• Semantic caching. Analyses of production query streams report roughly 31% of queries are semantically similar to prior ones — a semantic cache in the gateway serves those without a model call. Add an exact-match cache in front of it; the layers are complementary, not competing.
• Confidence-gated routing. A cheap model answers by default; low-confidence cases escalate to the frontier tier. Practitioner reports claim ~95% of frontier quality at 75–85% lower cost — keep the hedge, and validate against your own eval suite (the judge above is how you know your numbers).
• Instrument hit rates per layer. Exact cache, semantic cache, prefix cache, router escalation rate — each layer gets its own metric, because an uninstrumented cache silently decays and an unmeasured router silently over-escalates.

The gateway carries all of this — the same component doing quota-aware routing does the caching, the tiering and the spend metering, which is why it appears in every module of this track's second half.

TCO, token unit economics, and the launch forecast

Finance will ask two questions: what does this cost to run, and what will it cost next quarter? Both need models — with a discipline this track repeats deliberately: LLM pricing recall is unreliable by construction, so every rate in the model is verified against current calculators and price sheets, never taken from a model's memory or a stale blog post.

• The TCO model compares real alternativeshosted API vs provisioned throughput vs self-hosted — with sensitivity to context length and cache hit rate, and breakpoints marked: the volume at which the next option wins. (Self-hosting economics matter industry-wide: 55–80% of enterprise AI GPU spend now goes to inference, per FinOps ecosystem reporting.)
• Unit economics make cost governable. Cost per 100K tokens is the core FinOps-for-AI metric; from it derive cost per conversation, per document, per tenant. Design tagging, attribution and gateway metering so those numbers are measurable from day one — retrofitting attribution is many times harder than designing it in.
• The launch forecast carries uncertainty bands. Adoption curve × tokens per interaction × model mix × cache hit rate — each a range, not a point, yielding a best/expected/worst band that finance can plan against. Then review monthly against gateway actuals; the 73%-over-budget statistic describes teams that shipped a point estimate and stopped looking.

Before any cost model leaves your desk, run the verification pass:

• Every price has a source linkcalculator or current price sheet, checked this week, never model memory.
• Token counts are measured, not guessedfrom real prompts through a tokenizer, including system prompt and retrieved context.
• Cache assumptions are labeled as assumptionswith the hit-rate instrumentation that will replace them named.
• Breakpoints are explicitthe volume where provisioned beats on-demand, where self-hosting enters the conversation.
• The bands are honestthe worst case uses the worst plausible inputs, and the review cadence against actuals is written into the doc.

Where this lands in your week

• In design reviewevery LLM feature's HA section must answer "what happens on a 429?" with a routing answer, not a retry answer.
• In capacity planningthe quota table per region and account joins the capacity doc, and burst headroom is a stated multiple with a ladder behind it.
• In the deploy pipelinethe calibrated CI judge gates prompt and model changes this sprint; the online sample follows.
• In the cost reviewthe per-layer hit-rate dashboard and unit-economics numbers turn the monthly AI spend conversation from apology into engineering.
• In the launch checklistladder rungs tested, kill switch tested, forecast bands filed, actuals review scheduled.

Your playbooks & labs

This module has the track's biggest lab load, and every lab produces a production-shaped artifact: Lab 06 (TCO model for a RAG application) — the three-way cost model with verified pricing and breakpoints; Lab 11 (quota-aware failover design + simulation) — design the 429-vs-5xx ladder, then prove it against a LiteLLM-class gateway with mock providers throwing 429s; Lab 12 (LLM-as-judge eval architecture) — build the two-plane eval and show a deploy blocked by a seeded regression; and Lab 13 (AI cost architecture retrofit) — add caching layers and a two-tier router to a naive app and measure per-layer hit rates on a 200-request replay.

Companion playbooks: pb-41 (TCO Models with Pricing Verification & Cost-Review Narratives), pb-42 (AI-Workload Cost Modeling — Token Economics & Unit Costs), pb-43 (AI Feature Launch Cost Forecasting with Uncertainty Bands), pb-44 (Quota-Aware Multi-Region HA — 429 Is Not a 5xx), pb-45 (Graceful-Degradation Ladders for AI Features), pb-46 (LLM-as-Judge Evaluation Architecture — Two Planes, Calibrated), and pb-47 (The AI Cost Stack — Caching, Routing & Model Tiering).