Model Gateway: Routing, Budgets, and Semantic Caching

Tier: Advanced · Workflow area: Serving & LLMOps

A model gateway — the pattern Uber's GenAI Gateway made canonical — puts one OpenAI-compatible endpoint in front of all providers and centralizes routing, budgets, fallbacks, and caching. This playbook stands up a LiteLLM-style gateway with three specific policies: cheap-model-first cascades, per-team budgets, and a semantic cache, then measures what each saves.

When to use

• •
  Two or more apps/teams calling LLM APIs with no shared spend visibility or rate-limit management.
• •
  You want provider portability: swap claude-x for a self-hosted vLLM model (pb-46) without touching app code.
• •
  Monthly LLM spend has reached "someone asked about it in a meeting."

A single script calling a single model does not need a gateway — adopt it at the second consumer or the first budget question, whichever comes first.

Prerequisites

• •
  pip install 'litellm[proxy]' and Docker (for Redis).
• •
  API keys for at least two backends — e.g. one hosted provider plus your pb-46 vLLM endpoint.
• •
  A replayable sample of ~200 real (or realistic) requests from your workload, including the natural duplication rate — synthetic unique prompts will zero out the cache result.

The workflow

1. 1.

   Stand up the proxy with a model list. config.yaml maps public model names to backends; apps then call the gateway with an OpenAI client and a gateway-issued key:

   Copy

   model_list:
     - model_name: fast            # cheap tier
       litellm_params: {model: "anthropic/claude-haiku-4-5", api_key: os.environ/ANTHROPIC_API_KEY}
     - model_name: smart           # quality tier
       litellm_params: {model: "anthropic/claude-opus-4-8", api_key: os.environ/ANTHROPIC_API_KEY}
     - model_name: local           # your vLLM box
       litellm_params: {model: "openai/qwen2.5-7b", api_base: "http://vllm:8000/v1", api_key: "none"}
   

   Run litellm --config config.yaml, verify a completion through the proxy.

2. 2.

   Add provider fallback. Configure fallbacks so smart → fast → local on provider errors/timeouts (LiteLLM fallbacks router setting). Test by pointing smart at a wrong API key and confirming requests still succeed via the fallback, with the reroute visible in gateway logs.

3. 3.

   Implement the cascade. Cheap-model-first with escalation: route each request to fast; escalate to smart when a checkable condition fails — schema validation error, a self-reported confidence below threshold, or a task-type classifier flag. Implement as a thin client-side wrapper or gateway hook. The cascade needs a deterministic escalation trigger — "answer looks bad" is not one; "JSON failed to validate" is.

4. 4.

   Set per-team budgets. Issue each team (or app) its own virtual key with a monthly budget and rate limits via the gateway's key-management API. Set one team's budget deliberately low and verify the 429-on-exhaustion behavior — discovering budget semantics in production is the expensive way.

5. 5.

   Add semantic caching. Wire Redis as a semantic cache (embedding-similarity match above a threshold returns the cached response). Start conservative (similarity ≥0.97, cache only deterministic/low-temperature routes); paraphrase-level cache hits returning subtly wrong answers are worse than misses.

6. 6.

   Measure on the 200-request replay. Report: % served by fast without escalation, escalation rate, cache hit rate, fallback activations, and blended cost per request vs the all-smart baseline. Cascades on mixed workloads routinely cut spend 40–70%; your number depends on your escalation trigger's strictness.

Tool variants

• LiteLLM proxy (shown) — OSS, fastest path, huge provider matrix.
• Portkey / Kong AI gateway / cloud-provider gatewaysmanaged alternatives when ops-ownership is the constraint.
• Envoy AI Gatewaywhen your platform team already runs Envoy and wants gateway policy in the mesh.

Verification

• •
  One app switched from direct provider calls to the gateway with only a base-URL and key change.
• •
  Fallback fires on an induced provider failure; the event is attributable in logs.
• •
  Budget exhaustion returns a clean, documented error; spend per team is visible in the gateway's tracking.
• •
  Replay report includes cache false-hit inspection: sample 10 cache hits and confirm the cached answer was actually valid for the new request.

Pitfalls

• Semantic cache on creative/contextual routes. Two similar-looking prompts with different conversation history must not share an answer; scope the cache to stateless, deterministic endpoints.
• Cascade evaluation by cost alone. Always pair the cost number with a quality check on escalated-vs-non-escalated outputs; a cascade that never escalates is just a cheap model.
• The gateway as a new single point of failure. Note the availability story (multiple replicas, or client-side fallback to direct calls) even if you don't build it in the lab.
• Leaving the master key in app configs. Apps get virtual keys; the master key stays in your secrets manager.

Restricted-environment variant

Run the gateway and Redis inside the private network with only self-hosted backends (vLLM endpoints) — you keep unified budgets, routing, and caching with zero external calls. For client work, per-client virtual keys give you clean per-engagement cost attribution, which billing will thank you for.

Success criteria

• •
  Gateway serving two+ backends with fallback, cascade, budgets, and semantic cache all demonstrably active.
• •
  Replay-measured blended cost reduction with escalation and cache-hit rates reported honestly (including false-hit audit).
• •
  Per-team spend attribution working via virtual keys.
• •
  A config-only change (model swap) propagated to consumers with zero app-code edits — demonstrated.