Lab 13 · Serve an open model + inference gateway

Load-test it and read the two numbers that define serving performance — time-to-first-token (TTFT) and tokens-per-second (TPS) — plus a cost figure.

This is capstone lab 1 of 3. The gateway you build here is the exact system Lab 14 instruments with tracing and guardrails, and Lab 15 exposes and gates via MCP + prompt CI. Build the config cleanly — you will live in it for two more labs.

Time & tier

Advanced · ~90 minutes hands-on. The ₹0 path uses Ollama (or vLLM CPU-mode) for the local model and a hosted API only as the fallback route (a few paise). A GPU serving path is desk-checked against docs.

Setup

Self-contained. Python 3.11+, and an Anthropic API key for the fallback route.

mkdir ai-platform && cd ai-platform
python -m venv .venv && source .venv/bin/activate
pip install "litellm[proxy]" httpx
export ANTHROPIC_API_KEY="sk-ant-..."

You need a local OpenAI-compatible model server. Pick the path that matches your hardware:

• ₹0 laptop path (recommended): Ollama.curl -fsSL https://ollama.com/install.sh | sh then ollama pull qwen2.5:0.5b && ollama serve. Ollama exposes an OpenAI-compatible API at http://localhost:11434/v1.
• vLLM CPU-mode.pip install vllm then vllm serve Qwen/Qwen2.5-0.5B-Instruct --device cpu --port 8001. Serves at http://localhost:8001/v1. CPU decode is slow but real — good for reading TTFT/TPS behaviour under batching.
• GPU / hosted path (desk-check). On a small GPU (Colab/Kaggle free tier or ~₹30–60/hr spot), the same vllm serve command without --device cpu gives production-shaped numbers. Verify the current vllm serve flags against the vLLM docs before renting GPU time.

Guided steps

1. 1.

   Confirm the local model answers. With Ollama running:

   Copy

   curl -s http://localhost:11434/v1/chat/completions \
     -H 'content-type: application/json' \
     -d '{"model":"qwen2.5:0.5b","messages":[{"role":"user","content":"say hi"}]}' \
     | python -m json.tool
   

   You should get a completion. This is your "backend model" — the thing the gateway sits in front of.

2. 2.

   Write the gateway config. This one file is the whole point of the lab: it declares the local model, a hosted fallback, a model allowlist, and the fallback routing. Create config.yaml:

   Copy

   model_list:
     - model_name: platform-small        # the name callers use
       litellm_params:
         model: openai/qwen2.5:0.5b       # openai/ prefix = OpenAI-compatible backend
         api_base: http://localhost:11434/v1
         api_key: "ollama"                # Ollama ignores it; LiteLLM requires a value
     - model_name: platform-fallback
       litellm_params:
         model: anthropic/claude-haiku-4-5
         api_key: os.environ/ANTHROPIC_API_KEY
   
   litellm_settings:
     fallbacks: [{"platform-small": ["platform-fallback"]}]
   
   general_settings:
     master_key: sk-master-CHANGEME       # admin key that mints virtual keys
   

   The fallbacks block is the reliability feature: if platform-small errors or is unreachable, the gateway retries the request on platform-fallback transparently — the caller's key and request are unchanged.

3. 3.

   Run the gateway.

   Copy

   litellm --config config.yaml --port 4000
   

   It comes up on http://localhost:4000, speaking the OpenAI API. Every request now flows: caller → LiteLLM (key check, budget, routing) → model.

4. 4.

   Mint a virtual key with a budget and a model allowlist. Callers never get the master key. Each team gets a scoped virtual key:

   Copy

   curl -s http://localhost:4000/key/generate \
     -H "Authorization: Bearer sk-master-CHANGEME" \
     -H "content-type: application/json" \
     -d '{"models": ["platform-small"], "max_budget": 0.50, "key_alias": "team-search"}' \
     | python -m json.tool
   

   The response contains a key like sk-.... It can call only platform-small, and once it has spent $0.50 the gateway refuses further requests. Save the key as TEAM_KEY.

5. 5.

   Call through the gateway with the virtual key.

   Copy

   curl -s http://localhost:4000/v1/chat/completions \
     -H "Authorization: Bearer $TEAM_KEY" -H 'content-type: application/json' \
     -d '{"model":"platform-small","messages":[{"role":"user","content":"one-line SRE tip"}]}'
   

   Then prove the allowlist bites: swap platform-small for platform-fallback in the body and confirm the virtual key is rejected — it isn't allowlisted for that model.

6. 6.

   Load-test it and read TTFT/TPS. Create loadtest.py — it fires N concurrent streaming requests and records time-to-first-token and tokens-per-second per request:

   Copy

   import asyncio, time, os, httpx
   KEY = os.environ["TEAM_KEY"]; URL = "http://localhost:4000/v1/chat/completions"
   async def one(client, i):
       body = {"model": "platform-small", "stream": True,
               "messages": [{"role": "user", "content": f"Summarize incident {i} in 2 lines"}]}
       start = time.perf_counter(); first = None; toks = 0
       async with client.stream("POST", URL, json=body,
               headers={"Authorization": f"Bearer {KEY}"}) as r:
           async for line in r.alines():
               if line.startswith("data:") and "content" in line:
                   if first is None: first = time.perf_counter()
                   toks += 1
       end = time.perf_counter()
       ttft = (first or end) - start
       tps = toks / (end - (first or end)) if first and end > first else 0
       return ttft, tps
   async def main(n=20):
       async with httpx.AsyncClient(timeout=60) as c:
           res = await asyncio.gather(*(one(c, i) for i in range(n)))
       ttfts = sorted(x[0] for x in res); tpss = [x[1] for x in res]
       print(f"n={n}  TTFT p50={ttfts[n//2]:.2f}s p95={ttfts[int(n*0.95)]:.2f}s  "
             f"mean TPS={sum(tpss)/n:.1f}")
   asyncio.run(main())
   

   Run python loadtest.py. Record the numbers — they are your artifact and the Lab 14 baseline. On CPU the TPS will be single-digit; that's expected and the point — you're learning to read the numbers, not to hit a target.

7. 7.

   Prove the fallback fires. Stop Ollama (Ctrl-C its terminal) and re-run one gateway request. It should still return a valid completion — served by platform-fallback — demonstrating the gateway's reliability layer. Restart Ollama afterward.

Verification

• •
  A virtual key calls platform-small successfully and is rejected when it tries platform-fallback (allowlist enforced).
• •
  loadtest.py prints p50/p95 TTFT and a mean TPS across 20 concurrent requests.
• •
  Stopping the local model still yields a valid completion via the fallback route.
• •
  The budget bites: lower max_budget to a tiny value, spend it, and confirm the gateway returns a budget-exceeded error.

Completion artifact

The ai-platform/ folder (config.yaml, loadtest.py) plus BASELINE.md recording the backend used, p50/p95 TTFT, mean TPS, and one line confirming the fallback and budget both fired. Keep this folder — Labs 14 and 15 build directly on it. Self-grade (pass at 4/5):

1. 1.
   Local model served behind an OpenAI-compatible endpoint.
2. 2.
   Gateway config declares the model, a fallback, and an allowlist.
3. 3.
   A budgeted, model-scoped virtual key works and its allowlist/budget are both enforced.
4. 4.
   Load-test evidence exists with real TTFT and TPS numbers.
5. 5.
   Fallback to the hosted model is proven by killing the local server.

Stuck?

• LiteLLM won't start / model not found: the openai/ prefix and api_base must both point at your backend; for Ollama the model name is qwen2.5:0.5b, for vLLM it's the HF repo id you served.
• Virtual key 401: you used the master key in the Authorization header of /v1/chat/completions instead of the minted virtual key. Master key is only for /key/generate.
• Fallback doesn't fire: confirm the fallbacks list uses the model_name values, and that the error is a connection error (a 400 from a bad request won't trigger a fallback).
• Load test all timeouts on CPU: lower concurrency to n=5; CPU decode serialises heavily and 20 concurrent 0.5B requests can saturate a laptop.

Client-safe variant

This architecture is the client-data-compliance answer for services companies: the gateway lets you keep all inference on a client-approved backend (a model served inside the client VPC, or Bedrock in their tenancy) while the fallback and every other feature stay identical. Point platform-small at the client's model, set the fallback to another client-sanctioned endpoint rather than the public API, and use per-team virtual keys to enforce which client's data each team's key may touch. Record "which backend serves which client" in BASELINE.md as a design decision — it's the artifact an MSA review will ask for.