The biggest AI cost savings are architectural, decided before launch: a layered cache stack, confidence-gated routing to cheaper models, and per-task model right-sizing. This playbook designs the cost stack as a stack — with per-layer hit-rate instrumentation so every claimed saving is a measured one.
Tier: Advanced · Hands-on time: ~40 minutes
The practitioner cost literature this track draws on reports the levers and their rough magnitudes: prompt/prefix caching cutting input costs by up to ~90% on cache hits with break-even at roughly 1.4 reads of a cached prefix; around 31% of production queries being semantically similar enough for a semantic cache to answer; confidence-gated routing holding ~95% of quality at a 75–85% cost reduction per practitioner reports; and a roughly 50x cost spread between frontier and small models on the same task — making model selection the single biggest lever. All of these are reported figures from other people's workloads. The design discipline is to layer the levers, instrument each layer, and let your own hit rates replace the literature's within a month of launch.
Order the layers by cheapness of a hit. The canonical stack, outermost first: exact-match cache (identical request, near-zero cost), semantic cache (embedding-similarity match against previous answers), provider prompt/prefix caching (stable system prompt and tool schemas priced at the cached rate), then routing (small model default, frontier escalation), then the model call itself. Each request falls through until something answers it; each layer's job is to keep requests off the more expensive layers below.
Size the prompt-cache opportunity from your profile. The stable-prefix share from pb-42 bounds the upside: a workload whose input is 60% stable system-prompt-and-schemas has real caching leverage; one dominated by unique user content has little. Check the provider's actual caching mechanics in current docs — minimum cacheable sizes, TTLs, write costs, and pricing differ by platform and change often — and compute break-even for your read pattern rather than assuming the literature's ~1.4-reads figure.
Design the semantic cache with honesty rules. Similarity threshold set conservatively and tuned with eval evidence; staleness TTLs per content class; exclusion rules for anything personalized, tenant-specific (pb-39), or volatile (prices, policies); and a served-from-cache marker in traces. The ~31% similar-queries figure is a reason to build the layer and instrument it — your workload's true rate is what the hit-rate dashboard says after two weeks.
Design routing as confidence-gated escalation. Small model answers by default; escalation to the frontier tier triggers on measurable signals — task class, low model confidence, judge-flagged complexity, retrieval ambiguity, or explicit user escalation. Sameer, retrofitting the GCC's document-processing pipeline:
Critique this cost-stack design before I take it to review.
Workload: document summarization + Q&A, 400K requests/month, input
tokens 58% stable prefix (measured). Design: exact cache (24h TTL) ->
semantic cache (0.92 cosine threshold, price/policy content excluded)
-> provider prompt caching on the stable prefix -> router: small model
default, escalate to frontier on (a) document > 30 pages, (b) judge
confidence score < threshold, (c) user retry.
Attack: 1. Which layer combinations interact badly (e.g. semantic
cache serving a small-model answer that deserved escalation)?
2. What does the escalation rate need to stay below for the economics
to hold — give me the formula, not a number. 3. Which single
instrumentation gap would make this stack unauditable? 4. What
quality regression would each layer hide, and which eval catches it?
Right-size models per task, gated by evals. For each task class, run the golden set against the candidate cheaper model (pb-46's judge scores the delta) and let the numbers decide the default tier. The ~50x spread means this is the biggest lever in the stack — and also the one that most needs eval evidence, because it is the one that changes answers.
Instrument per layer, from day one. Each layer emits hits, misses, latency added, and cost avoided (computed from verified rates per pb-41); the router emits escalation rate and reasons. This instrumentation is what turns the literature's hedged percentages into your dashboard's facts — and it is drastically cheaper to design in than to retrofit.
Model the savings with bands, then reconcile. Build the spreadsheet with pessimistic/expected/optimistic hit rates per layer (the literature's figures are the optimistic bound, not the plan), feed it into pb-43's forecast, and reconcile monthly against gateway actuals. Layers that underperform their band are findings: tune the threshold, or admit the layer isn't earning its complexity and remove it.
Caches concentrate client data — cached answers, embeddings for semantic matching, and stable prefixes that may contain client system prompts all live inside the client boundary, under the client's retention rules, with tenant scoping per pb-39. Extract MSA constraints first: approved model lists constrain the routing tiers, and client-tenancy endpoints (Bedrock or Azure OpenAI in the client subscription) support provider-side prompt caching only as their platform documents it — verify in their tenancy, not from public-cloud assumptions. The whole stack prototypes offline: LiteLLM caching plus Ollama-served small and large models exercises layering, routing logic, and instrumentation with synthetic traffic before anything touches client systems.
Layer the levers — exact, semantic, prefix, router — gate every quality-affecting one with evals, and instrument each layer so that within a month your dashboard's hit rates, not the literature's hedged percentages, are what the savings claim stands on.