This module flips your seat: you stop placing AI on the platform and start running AI as a platform workload — serving, routing, metering and observing requests that are long-running, KV-cache-bound and token-metered, on infrastructure whose familiar dashboards lie about all three.
The internal chat assistant had been live a week when the complaints started: replies that used to stream instantly now hung ten seconds before the first word appeared. Priya picked up the page and pulled the dashboard she'd trust for any other service — CPU across the two GPU replicas sat at a comfortable 55%, memory looked fine, the load balancer reported connections split evenly. By every signal she owned, the service was healthy. It was not.
The load balancer was least-connections, and least-connections saw a replica holding "one connection" and sent it more work — never mind that the one connection was a user generating a 4,000-token essay, its KV cache already 96% full, every new request now queued behind it. The other replica sat with a warm prefix cache that round-robin kept scattering traffic away from.
Priya had spent three years learning to trust CPU and connection count, and for this workload both were lying to her. Meera, reviewing it the next morning, put it flatly: "we deployed a new kind of physics on an old kind of dashboard."
Everything before this module was about using AI to run the platform. This module and the next are about the inverse — running AI on the platform — and it is the half of this track that turns a senior DevOps engineer into an AI platform engineer, the newest title on the ladder and the one that carries the wage premium. Job postings now name the components explicitly: inference gateways, LLM cost/latency observability, GPU scheduling, prompt/version pipelines. None of these are exotic. They are load balancing, multi-tenancy, metrics, and capacity management — your core trade — applied to a workload class with genuinely different physics.
The trust map from M1 still governs everything here, but your seat changes. Until now you were placing AI assistance on the map — deciding where a model may advise on your pipelines and incidents. From here on you are the operator of the map: the person who decides which teams get which models, at what budget, behind which guardrails, with what evidence trail. When the 76% of developers who won't trust AI in production ask "who's watching this thing?", the answer this module builds is: the platform, with the same rigor it watches everything else.
LLM serving is not a special snowflake — it is multi-tenant infrastructure with unusual physics: requests are long-running, partially stateful (KV cache), token-metered rather than request-metered, and bound by GPU memory rather than CPU. Every design decision in this module follows from those four properties.
Your first 15 minutes: open your LLM service's dashboard and check whether it graphs queue depth and KV-cache utilization at all — if it only shows CPU and connections, you are flying Priya's blind dashboard, and fixing that is step one of this module.
Start with the request lifecycle, because every platform decision downstream depends on it. An LLM request has two phases. Prefill processes the entire prompt in one compute-bound pass and produces the first output token. Decode then generates output one token at a time, each step reading the KV cache — the per-request attention state that lives in GPU memory for the request's whole lifetime. Prefill is compute-bound and fast per token; decode is memory-bandwidth-bound and slow per token; and the KV cache is the real capacity constraint, because GPU memory holds the model weights plus every in-flight request's cache.
This is why the metrics split in two. TTFT (time to first token) is dominated by prefill plus queueing — it is the felt latency of a streaming UI. TPS (output tokens per second, both per-request and aggregate) is the decode-side throughput number that determines how many concurrent users a GPU serves. A change that improves one routinely degrades the other: bigger batches raise aggregate TPS and raise TTFT; smaller --max-model-len frees KV memory for concurrency but caps context.
Now consider what your existing load balancer assumes: requests are short, uniform, stateless, and cheap to move. LLM requests are none of these. A request generating 4,000 tokens occupies its replica for tens of seconds; least-connections sees "1 connection" on a replica whose KV cache is 95% full and happily sends it more work; round-robin scatters requests with a common prefix across replicas, destroying prefix-cache hits that would have skipped most of prefill. Utilization signals lie too — a GPU at 60% compute may be completely full in the dimension that matters, KV-cache memory.
The load signal for LLM serving is queue depth and KV-cache utilization, not CPU or connection count. Any routing or autoscaling built on classic signals will look healthy while users see ten-second TTFTs.
vLLM is the default open serving engine — PagedAttention manages KV cache in fixed-size blocks the way an OS pages memory, and continuous batching admits new requests into the running batch as others finish. The DS/ML track teaches vLLM from the model owner's seat (quantization, eval parity). Your seat is different: you run it as a tenant-serving platform workload, which means a Deployment with correct resource semantics, health probes, and autoscaling — the same checklist as any service, with GPU-specific values.
apiVersion: apps/v1
kind: Deployment
metadata:
name: vllm-qwen-7b
namespace: llm-serving
labels: {app: vllm-qwen-7b}
spec:
replicas: 2
selector:
matchLabels: {app: vllm-qwen-7b}
template:
metadata:
labels: {app: vllm-qwen-7b}
spec:
containers:
- name: vllm
image: vllm/vllm-openai:latest # pin a digest in production
args:
- --model=Qwen/Qwen2.5-7B-Instruct-AWQ
- --max-model-len=8192 # unused context length is concurrency given away
- --gpu-memory-utilization=0.90
- --enable-prefix-caching
ports:
- containerPort: 8000
resources:
limits:
nvidia.com/gpu: "1" # GPUs are integer, non-overcommittable
readinessProbe:
httpGet: {path: /health, port: 8000}
initialDelaySeconds: 60 # model load takes minutes, not seconds
periodSeconds: 10
livenessProbe:
httpGet: {path: /health, port: 8000}
initialDelaySeconds: 300
periodSeconds: 30
volumeMounts:
- {name: model-cache, mountPath: /root/.cache/huggingface}
volumes:
- name: model-cache
persistentVolumeClaim: {claimName: model-cache-pvc}
Three platform-specific notes carry that manifest:
nvidia.com/gpu is requested in integers via the NVIDIA device plugin; there is no "request 0.5, limit 1" (fractionalization comes later, via MIG or time-slicing)./metrics, and the correct HPA input is vllm:num_requests_waiting (queue depth) or vllm:kv_cache_usage_perc, via the Prometheus Adapter or KEDA — not CPU, which barely moves on a GPU-bound pod. Scale-up must also respect the cold-start reality: a replica that takes four minutes to become ready is not a burst-absorber, which is why queue-based scaling with generous stabilization windows beats aggressive reactive scaling.At larger scale, the same physics push toward disaggregated serving, the idea at the core of llm-d (a CNCF Sandbox project built on vLLM): run prefill and decode on separate pools of GPUs sized independently, transfer the KV cache between them, and add cache-aware and load-aware scheduling across the fleet. You don't need llm-d for a team-scale deployment, but you should be able to explain why it exists: prefill and decode have opposite hardware profiles, so co-locating them means one always underuses the GPU. When an architect asks "should we adopt llm-d?", the platform-engineer answer is a capacity argument, not a fashion argument.
Client-data constraints make self-hosted serving a first-class requirement in services companies, not an optimization. When the MSA forbids sending client telemetry to public AI APIs, "vLLM inside the client's VPC behind your gateway" is the compliant architecture — which is why this module's lab runs the full stack self-hosted on free-tier infrastructure.
If vLLM is the backend, the Gateway API Inference Extension is how Kubernetes networking catches up with the physics above. It extends the standard Gateway API with inference-aware routing, and its v1 API has two resources with a deliberate ownership split — note that the original InferenceModel CRD was replaced by InferenceObjective at v1, so treat older blog posts with care:
x-gateway-inference-objective header naming the objective.apiVersion: inference.networking.k8s.io/v1
kind: InferencePool
metadata:
name: qwen-pool
namespace: llm-serving
spec:
selector:
matchLabels: {app: vllm-qwen-7b}
targetPortNumber: 8000
extensionRef:
name: qwen-epp # the Endpoint Picker deployment
---
apiVersion: inference.networking.x-k8s.io/v1alpha2
kind: InferenceObjective
metadata:
name: interactive-chat
namespace: llm-serving
spec:
priority: 10 # higher = served first under load
poolRef:
name: qwen-pool
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: llm-route
spec:
parentRefs:
- name: inference-gateway
rules:
- matches:
- path: {type: PathPrefix, value: /v1}
backendRefs:
- group: inference.networking.k8s.io
kind: InferencePool
name: qwen-pool
The pattern to internalize is the persona split, because it is the same split you already run for every other workload: the platform team owns the pool, the gateway, and the EPP (the "how traffic flows"); application teams own objectives and routes for their model consumers (the "what my traffic needs"). Priority is how you encode business reality into saturation behavior — the customer-facing chatbot outranks the nightly batch summarizer — and traffic splitting across pools gives you progressive model rollouts with the same HTTPRoute weight mechanics you use for canaries today. Envoy Gateway, kgateway, Istio, and GKE's Inference Gateway all implement the extension, so this is a portable skill, not a vendor feature.
Scenario questions in this domain hinge on the ownership split and the routing signal. If an answer routes LLM traffic by round-robin, autoscales on CPU, or has application teams editing the InferencePool, it is the planted distractor.
The inference gateway routes traffic to your self-hosted pool. One layer up sits the LLM gateway — a LiteLLM-class proxy that gives every application in the company one OpenAI-compatible URL and gives you, the platform team, the control plane: who may call which models, at what spend, with what failover. This is the component job postings mean by "inference gateway operations," and it is where multi-tenancy actually happens. LiteLLM (MIT-licensed, self-hostable, 100+ providers) is the reference implementation for the labs; Portkey and Kong AI Gateway are the named alternatives — the concepts transfer.
A minimal production shape:
# litellm config.yaml
model_list:
- model_name: chat-default # what tenants see
litellm_params:
model: hosted_vllm/Qwen/Qwen2.5-7B-Instruct-AWQ
api_base: http://qwen-pool.llm-serving.svc:8000/v1
- model_name: chat-frontier
litellm_params:
model: anthropic/claude-sonnet-4-5
api_key: os.environ/ANTHROPIC_API_KEY
router_settings:
num_retries: 2
fallbacks: [{"chat-default": ["chat-frontier"]}] # self-hosted down → hosted API
allowed_fails: 3
cooldown_time: 30
general_settings:
master_key: os.environ/LITELLM_MASTER_KEY
database_url: os.environ/DATABASE_URL # Postgres — required for virtual keys
Onto that base, three operational mechanisms:
Virtual keys are per-team credentials minted from the gateway, not provider keys handed around. POST /key/generate (authorized by the master key) creates a key scoped to specific models with its own limits. Provider API keys live only in the gateway's secret store; when a team's key leaks, you revoke one row in Postgres, not the org's Anthropic key.
curl -s http://litellm:4000/key/generate \
-H "Authorization: Bearer $LITELLM_MASTER_KEY" \
-d '{
"team_id": "checkout-squad",
"models": ["chat-default"],
"max_budget": 200.0,
"budget_duration": "30d",
"tpm_limit": 100000,
"rpm_limit": 500
}'
Budgets and rate limits turn cost management from a monthly forecasting exercise into an enforced control. max_budget with budget_duration hard-caps a team's monthly spend; tpm_limit/rpm_limit cap burst. This matters doubly for agentic consumers (M8): an agent stuck in a loop exhausts its own budget and gets a 429, instead of exhausting the org's. Spend aggregates automatically per key, user, and team — which is exactly the showback dimension your FinOps reporting (M6) needs.
Fallbacks are your reliability story. Provider outages, 429s, and context overflows reroute automatically — LiteLLM distinguishes general fallbacks, context-window fallbacks, and content-policy fallbacks, each a mapping from primary to an ordered fallback list, with cooldowns taking a failing deployment out of rotation. The trap to teach your tenants: a fallback model is a different model, and per the DS/ML track's rule it must be eval-qualified before the outage, or failover becomes a silent quality incident.
For India services teams, run gateway-level PII redaction (Presidio-class, wired as a pre-call guardrail — configured fully in M8) on routes that may carry client data. The gateway is the one choke point where "no client data reaches a public API" can be enforced and proven to a client auditor, which turns a compliance burden into a deliverable.
An unobserved LLM platform fails in two currencies at once — latency you can't attribute and spend you can't explain. The good news: this is a trace-and-metrics problem, and you already own the stack. Three layers:
Engine metrics (Prometheus). vLLM's /metrics endpoint exposes the inference-native series; scrape it like any exporter and alert on the saturation chain:
| Metric | What it tells you |
|---|---|
vllm:time_to_first_token_seconds | TTFT histogram — the user-felt latency; alert on p95 |
vllm:inter_token_latency_seconds | Decode-side smoothness; streaming UX degrades here first |
vllm:e2e_request_latency_seconds | Whole-request latency |
vllm:num_requests_waiting | Queue depth — your HPA signal and earliest saturation warning |
vllm:kv_cache_usage_perc | The real capacity dimension; sustained >0.9 means TTFT is about to spike |
vllm:prefix_cache_hits / vllm:prefix_cache_queries | Cache hit ratio — a routing-quality signal, not just a cost one |
vllm:generation_tokens / vllm:prompt_tokens | Throughput counters — aggregate TPS, and the denominator for GPU cost-per-token |
Request traces and cost (Langfuse). Metrics tell you the platform is slow; traces tell you whose requests, on which model, at what cost. Langfuse (open source, self-hostable, OpenTelemetry-based) ingests traces directly from LiteLLM — the gateway emits one span per request with model, tokens in/out, computed cost, latency, and the virtual key's team metadata. That single integration gives you cost-per-request, cost-per-team, and cost-per-feature dashboards without touching application code, because the gateway is the choke point through which every request already flows.
The alerts that matter. Translate the SLO discipline you already have: a TTFT p95 SLO per priority class (interactive vs. batch); a queue-depth page that fires before the TTFT one; per-team budget-burn alerts at 50/80/100% (the gateway enforces at 100%, but the 80% alert is what prevents the enforcement being a surprise); and a cost-per-request regression alert, because a prompt change that doubles output tokens is a cost incident wearing a latency costume. Note what's absent: model quality metrics. Quality evaluation belongs to the eval pipelines in M8 — the platform's observability job is latency, saturation, availability, and money.
Never alert on GPU compute utilization as a capacity signal. A decode-heavy workload shows modest compute while KV cache is exhausted; conversely 100% compute during prefill bursts is healthy. Queue depth and kv_cache_usage_perc are the truth; nvidia-smi utilization is trivia.
GPUs are the scarcest, most expensive resource you will schedule, and the default Kubernetes experience — integer allocation, no preemption, first-come-first-served — wastes them. Four mechanisms, in the order you'll need them:
MostAllocated scoring in the scheduler) so whole nodes free up and scale away. The tension to manage: tight packing concentrates blast radius, so pair it with PodDisruptionBudgets on serving replicas.Caching is the highest-leverage cost lever on an LLM platform — and the only one that can silently serve wrong answers. Keep the two layers distinct:
Traffic shaping completes the picture with tools you already know, re-aimed at token economics: token-bucket rate limits at the gateway (per-key TPM/RPM — tokens are the unit that maps to cost and GPU time, so TPM is the limit that matters); priority tiers via InferenceObjective so batch yields to interactive under load instead of an outage; and request hedging only for idempotent, non-streaming calls — a duplicated LLM request is not a duplicated 5ms GET, it is double the tokens and double the GPU seconds.
Prefix caching is a free win you enable; semantic caching is a correctness trade you govern. If you cannot answer "which cache entry served this response?" from your traces, the semantic cache is not ready for production traffic.
Priya's LLM pool autoscales on CPU and routes least-connections, yet users see ten-second TTFTs while the GPUs read 55% utilized. Name the two signals she should route and scale on instead — and say why CPU is lying to her.
/metrics and LiteLLM spend data, build one dashboard: TTFT p95, aggregate TPS, queue depth, KV-cache %, prefix-cache hit ratio, cost-per-request by team. Define two alerts (queue-depth saturation, budget burn 80%). Deliverable: dashboard JSON/screenshot plus alert rules.This module anchors Lab L13 — Serve an open model + inference gateway: deploy a small open model on vLLM (CPU-class, free-tier GPU, or hosted-API fallback — all three paths documented), front it with LiteLLM, mint virtual keys with budgets, configure a fallback to a hosted API, then load-test and read the TTFT/TPS curve. Artifact: the running stack, gateway config, and a latency/cost report. It also begins Lab L14 — LLM observability + guardrails (first half): instrument the L13 stack with Langfuse traces and the Prometheus dashboard from Exercise 5 — the guardrail layer lands in M8, which completes L14.
Three playbooks turn the module's builds into repeatable procedure:
The metrics-and-alerts pack, the GPU scheduling decision tree (pb-48), the semantic-cache eligibility matrix (pb-47), and the client-safe serving variant round out the set. Every lab runs at ₹0 except the optional GPU path (~₹500 total spot budget, with Ollama/Colab alternatives documented).