Cut LLM Cost and Latency with Semantic Caching and Traffic Shaping

Tier: Advanced · Workflow area: Platform-for-AI: observability & cost

This playbook applies each lever where it pays, with the measurement discipline to prove it, and the false-hit guardrails semantic caching demands.

When to use

• •
  The pb-46 dashboards show repeated or near-duplicate prompts (support bots, RAG over stable corpora, retry storms).
• •
  Interactive TTFT degrades whenever batch/eval jobs run — you need tiers, not more GPUs.
• •
  Cost-per-request must come down and the prompt itself is already lean.

Skip semantic caching for low-volume, high-uniqueness traffic (agent transcripts, unique documents) — hit rates will be single-digit and the false-hit risk isn't worth it. Prefix caching and rate shaping still apply.

Prerequisites

• •
  pb-45 LiteLLM gateway and (if self-hosting) pb-43 vLLM; a Redis instance for gateway caching.
• •
  pb-46 metrics live — you cannot claim a win you cannot measure.
• •
  A written statement of which routes are cacheable at all (see the safety rules in step 3).

The workflow

1. 1.

   Take the free win: vLLM prefix caching. vLLM's automatic prefix caching reuses KV-cache blocks for shared prompt prefixes (system prompts, few-shot preambles, RAG boilerplate) — it cuts prefill compute and TTFT without changing response content, so it is always safe. Confirm it's enabled for your vLLM version (recent versions default on; older need --enable-prefix-caching), then structure prompts for it: stable prefix first, volatile content last — the same discipline as provider-side prompt caching for hosted APIs.

2. 2.

   Add exact-match response caching at the gateway. In LiteLLM:

   Copy

   litellm_settings:
     cache: true
     cache_params:
       type: redis
       host: os.environ/REDIS_HOST
       ttl: 600
   

   Exact-match caching is low-risk (identical request → identical stored response) and already absorbs retries and duplicate clicks. Scope it per-route/key rather than globally, and keep TTLs short where the underlying knowledge moves.

3. 3.

   Add semantic caching only where meaning-equivalent answers are acceptable. Semantic cache embeds the prompt and serves a stored response when similarity exceeds a threshold:

   Copy

   litellm_settings:
     cache: true
     cache_params:
       type: redis-semantic
       similarity_threshold: 0.92     # start strict; loosen with evidence
       redis_semantic_cache_embedding_model: text-embedding-3-small
   

   Safety rules: never on personalized or per-user-context routes (cross-user answer leakage), never on time-sensitive answers, always with the cache keyed within a tenant/team boundary, and always with a cache_hit flag emitted into pb-46 traces so hits are auditable. Verify config keys against your pinned LiteLLM version; GPTCache-class sidecars are the alternative if you need custom eviction or scoring.

4. 4.

   Shape traffic into tiers. Define two or three service tiers and enforce them with the mechanisms you already own: per-key rpm_limit/tpm_limit as token buckets (interactive keys get burst headroom; batch keys get throughput caps), the pb-44 inference gateway's priority/criticality policy so batch never queues ahead of chat, and scheduling batch/eval jobs off-peak. Token-bucket intuition: cap burn rate per tier so no single consumer can starve the pool even inside its monthly budget.

5. 5.

   Decide your hedging policy deliberately. Request hedging (duplicate a slow request, take the first response) tames tail latency on cheap idempotent calls — but every hedge doubles token cost and load, and hedging into a saturated pool makes the tail worse. If you enable client/gateway retries with tight timeouts as a hedge, cap hedge rate, exempt expensive long-generation routes, and watch the pb-46 cost dashboards for the hedge tax.

6. 6.

   Measure before/after like an SRE. For each lever: one-week baseline, enable for a segment, compare — cache hit rate, cost-per-request, P95 TTFT, and (for semantic cache) a sampled false-hit audit where a human judges whether served-from-cache answers were actually correct for the new prompt. Keep or kill each lever on evidence.

Tool variants

• LiteLLM caching (this playbook): exact + semantic in the gateway you already run — first choice.
• Kong AI Gateway: semantic caching plus rate limiting in the Kong plugin ecosystem if that's your gateway family.
• GPTCache-style library/sidecar: in-app semantic caching when you need custom similarity scoring or per-feature eviction; more code to own.
• Provider prompt caching (Anthropic/OpenAI): for hosted-API traffic, structuring prompts with stable prefixes and provider cache controls is the analog of step 1 — same prefix discipline, provider-side discount.

Verification

• •
  Prefix caching: measurable TTFT drop on long-system-prompt routes (vLLM logs/metrics show cache block reuse).
• •
  Exact cache: hit rate and cost delta visible in dashboards; cached responses flagged in traces.
• •
  Semantic cache: hit rate ≥ your worthwhile threshold (typically >20% on FAQ-like routes) and false-hit audit below your agreed tolerance (e.g. <1% of sampled hits).
• •
  Tier test: batch flood at full rate leaves interactive P95 TTFT within SLO.

Pitfalls

• Caching personalized answers. "What's my account status?" served from another user's cache entry is a data breach, not a performance win. Route-level allowlists, not global config.
• Threshold tuned on hit rate alone. Lowering similarity_threshold always raises hit rate and always raises false hits; only the human audit tells you the real trade.
• Invisible staleness. Cached answers outliving a knowledge-base update look like model failure. Purge caches on corpus deploys; keep TTLs short by default.
• Rate limits nobody rehearsed. Consumers must handle 429s with backoff; roll tiers out with drills (as in pb-45) or the first throttle event becomes an incident.

Restricted-environment variant

• •
  Keep Redis (cache contents = prompts and completions = potentially client data) inside the same compliance boundary as the gateway; encrypt at rest and scope access like a production datastore.
• •
  Per-client stacks get per-client caches — never share a semantic cache across client tenancies, whatever the hit-rate temptation.
• •
  If embeddings for semantic caching would leave the boundary (hosted embedding API), use a local embedding model instead; the embedding call is a data egress too.

Success criteria

• •
  Documented before/after showing cost-per-request and P95 TTFT improvements per lever, with the losing levers turned off.
• •
  Zero cross-user or cross-tenant cache hits by construction (key scoping reviewed, not assumed).
• •
  Interactive SLO holds during batch floods without adding hardware.
• •
  The false-hit audit is a standing weekly ritual with an owner, not a launch-week one-off.