Run a Multi-Provider LLM Gateway: Virtual Keys, Budgets, and Fallbacks

Tier: Advanced · Workflow area: Platform-for-AI: serving & gateways

This playbook deploys LiteLLM (self-hosted, MIT-licensed) as that layer, so no application ever holds a raw provider key and no team can silently blow the AI budget.

When to use

• •
  More than one team or app calls LLM APIs and provider keys are spreading through env vars and CI secrets.
• •
  You need per-team cost attribution and hard spend caps before the first surprise invoice, not after.
• •
  You want provider portability: swap or mix Anthropic/OpenAI/Bedrock/self-hosted vLLM without app changes.

Skip a gateway when exactly one app calls one provider under one budget — wrap the SDK and revisit at the second consumer.

Prerequisites

• •
  Docker/K8s to run litellm (the proxy image) plus a Postgres database (required for virtual keys and spend tracking; DATABASE_URL).
• •
  Provider credentials in a secrets manager; the pb-43 vLLM endpoint if you have one.
• •
  A LITELLM_MASTER_KEY (must start with sk-) — treat it like a cloud root credential.

The workflow

1. 1.

   Write the config: models, routing, fallbacks.

   Copy

   # litellm-config.yaml
   model_list:
     - model_name: chat-default            # user-facing alias
       litellm_params:
         model: anthropic/claude-sonnet-4-6
         api_key: os.environ/ANTHROPIC_API_KEY
     - model_name: chat-default            # same alias = load-balanced group
       litellm_params:
         model: bedrock/anthropic.claude-sonnet-4-6
         aws_region_name: us-east-1
     - model_name: chat-local
       litellm_params:
         model: hosted_vllm/Qwen/Qwen2.5-7B-Instruct
         api_base: http://vllm-qwen.ml.svc:8000/v1
   
   litellm_settings:
     fallbacks: [{"chat-default": ["chat-local"]}]
     num_retries: 2
     request_timeout: 120
   
   router_settings:
     routing_strategy: usage-based-routing
   
   general_settings:
     master_key: os.environ/LITELLM_MASTER_KEY
   

   Aliases are the contract with app teams; providers behind an alias can change without breaking anyone. Verify exact field names against the LiteLLM docs for your pinned version — the config surface evolves.

2. 2.

   Deploy with Postgres and pin the version. docker run/Helm with DATABASE_URL and the config mounted. Pin the image tag; gateway upgrades are change-managed like any other prod middleware.

3. 3.

   Mint virtual keys per team, with budgets attached.

   Copy

   curl http://gateway:4000/key/generate \
     -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
     -H 'Content-Type: application/json' \
     -d '{
       "team_id": "checkout-squad",
       "models": ["chat-default"],
       "max_budget": 200.0,
       "budget_duration": "30d",
       "tpm_limit": 200000,
       "rpm_limit": 500,
       "metadata": {"service": "support-bot", "env": "prod"}
     }'
   

   One key per service per environment. The models allowlist is your first guardrail: a team without chat-local access cannot route client data to the wrong tenancy even by accident.

4. 4.

   Point apps at the gateway. Apps use any OpenAI-compatible SDK with base_url=http://gateway:4000 and their virtual key. Confiscate raw provider keys from app configs — rotate them so old copies die.

5. 5.

   Prove the failure modes before your users find them. Three drills: (a) exhaust a test key's max_budget and confirm requests 4xx with a budget error; (b) exceed rpm_limit and confirm 429s; (c) break the primary provider (bad key in staging) and confirm fallback serves traffic — and that the fallback event is visible in logs/alerts, because silent fallback to a weaker model is a quality incident.

6. 6.

   Wire spend reporting. Per-key/team spend accrues in the database; export to your FinOps dashboards and alert at 80% of budget. This is the cost-per-team number your leadership asks for (feeds pb-46's observability layer).

Tool variants

• LiteLLM (this playbook): widest provider coverage, MIT license, self-hosted — the default free path and the one to learn on.
• Portkey: OSS core with a guardrails/observability-led feature set; strong choice when pb-49's gateway-level guardrails are your priority.
• Kong AI Gateway / Envoy AI Gateway: right when the platform team already operates that gateway family and wants LLM traffic under the same control plane.

Verification

• •
  All three failure drills (budget, rate limit, fallback) pass with observable, alerting evidence.
• •
  Spend report for a test team matches a hand-computed check from token counts and price sheets within rounding.
• •
  Grep of app repos/CI finds zero raw provider keys post-migration.
• •
  Gateway P95 overhead under load is single-digit milliseconds (LiteLLM benchmarks ~8ms at 1k RPS — measure your own).

Pitfalls

• Gateway as SPOF. You just centralized all AI traffic; run ≥2 replicas behind a Service/LB and load-test the gateway itself, not just the models.
• Config drift via the admin UI. UI-created keys and models bypass your GitOps trail. Manage config declaratively; restrict UI to read-only where possible.
• Budgets as the only control.max_budget stops spend after it happens up to the cap. Pair with tpm_limit/rpm_limit to cap burn rate, not just the monthly total.
• Silent model substitution. Fallback chains that cross quality tiers (Sonnet-class → 7B local) must page someone; users experiencing degraded answers with no alert is the worst failure mode.

Restricted-environment variant

This layer is the services-company superpower: route by data classification.

• •
  Client-restricted workloads get keys whose models list contains only in-VPC targets (client's Bedrock/Azure OpenAI tenancy, or the pb-43 vLLM). General workloads get hosted-provider aliases. Policy becomes enforceable config, not a wiki page.
• •
  Add gateway-level PII redaction (pb-49) in front of any hosted-provider alias as defense-in-depth.
• •
  Keep the gateway, database, and logs inside the client's compliance boundary for client-dedicated deployments; spend data itself can be commercially sensitive.

Success criteria

• •
  One endpoint, N providers: apps switched providers via config alias with zero code changes.
• •
  Every consuming service has its own budgeted, rate-limited, allowlisted key; finance gets per-team spend without asking you.
• •
  A provider outage in staging degraded to fallback with an alert, not an incident channel full of screenshots.
• •
  New team onboarding to AI capability is a key-generation call, not a procurement thread.