Deploy vLLM on Kubernetes: Your First Self-Hosted Model Service

Tier: Advanced · Workflow area: Platform-for-AI: serving & gateways

This playbook deploys vLLM as a proper Kubernetes service: OpenAI-compatible API, GPU scheduling, health probes, model cache, and the serving concepts (prefill vs decode, KV cache, continuous batching) you need to reason about its performance.

When to use

• •
  Client-data constraints or cost projections justify self-hosting an open model (Llama, Qwen, Mistral class).
• •
  You're building the internal "models as a service" layer other teams will consume through a gateway.
• •
  You need a lab-grade stack to learn serving operations before a production commitment.

Skip self-hosting when usage is low and data policy permits hosted APIs — a gateway in front of hosted providers (pb-45) is far less operational burden.

Prerequisites

• •
  A K8s cluster with at least one NVIDIA GPU node, device plugin or GPU Operator installed (kubectl describe node | grep nvidia.com/gpu shows capacity). No GPU? See the free-tier paths under Tool variants.
• •
  A Hugging Face token if the model is gated; a PVC-capable StorageClass for the model cache.
• •
  Mental model check: prefill processes the prompt (compute-bound, determines TTFT); decode generates tokens one step at a time (memory-bandwidth-bound, determines TPS); the KV cache holds attention state per request in GPU memory and is what actually limits concurrency; continuous batching is why vLLM sustains high throughput.

The workflow

1. 1.

   Start with a model that fits. Weights-plus-KV-cache must fit GPU memory: a 7–8B model in FP16 needs ~16GB plus cache headroom. Use a quantized (AWQ/GPTQ) variant on 16GB cards. Ask your assistant to sanity-check: "vRAM budget for Qwen2.5-7B-Instruct FP16 on one L4 with 8k context, ~8 concurrent requests?" — then verify against vLLM docs.

2. 2.

   Write the Deployment. Have your assistant draft it, then check every field below exists — these are the parts assistants routinely omit:

   Copy

   apiVersion: apps/v1
   kind: Deployment
   metadata: {name: vllm-qwen, labels: {app: vllm-qwen}}
   spec:
     replicas: 1
     selector: {matchLabels: {app: vllm-qwen}}
     template:
       metadata: {labels: {app: vllm-qwen}}
       spec:
         containers:
         - name: vllm
           image: vllm/vllm-openai:latest   # pin a version in prod
           command: ["vllm", "serve", "Qwen/Qwen2.5-7B-Instruct",
                     "--max-model-len", "8192"]
           env:
           - name: HUGGING_FACE_HUB_TOKEN
             valueFrom: {secretKeyRef: {name: hf-token, key: token}}
           ports: [{containerPort: 8000}]
           resources:
             limits: {nvidia.com/gpu: "1", memory: 20Gi, cpu: "8"}
             requests: {nvidia.com/gpu: "1", memory: 8Gi, cpu: "2"}
           readinessProbe:
             httpGet: {path: /health, port: 8000}
             initialDelaySeconds: 60
             periodSeconds: 5
           livenessProbe:
             httpGet: {path: /health, port: 8000}
             initialDelaySeconds: 60
             periodSeconds: 10
           volumeMounts:
           - {name: hf-cache, mountPath: /root/.cache/huggingface}
           - {name: shm, mountPath: /dev/shm}
         volumes:
         - name: hf-cache
           persistentVolumeClaim: {claimName: vllm-hf-cache}
         - name: shm
           emptyDir: {medium: Memory, sizeLimit: 2Gi}
   

   The PVC cache means restarts don't re-download 15GB of weights; the /dev/shm emptyDir is required for tensor-parallel and NCCL. Set generous probe delays — model load takes minutes.

3. 3.

   Expose it privately. ClusterIP Service on port 8000. Do not put an Ingress on it — the gateway (pb-44/pb-45) is the front door; raw model endpoints must not be directly reachable by clients.

4. 4.

   Smoke-test the OpenAI-compatible API.

   Copy

   kubectl port-forward svc/vllm-qwen 8000:8000
   curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \
     -d '{"model":"Qwen/Qwen2.5-7B-Instruct","messages":[{"role":"user","content":"Say OK"}],"max_tokens":8}'
   

5. 5.

   Scrape metrics and watch the two numbers that matter. vLLM exposes Prometheus metrics at /metrics. Chart vllm:num_requests_waiting (queue depth — your scaling signal) and GPU cache usage. HPA on CPU is meaningless for LLM pods; scale on queue depth or concurrency via custom metrics.

6. 6.

   Kill it and watch it recover. Delete the pod; confirm probes gate traffic until the model is loaded and no requests hit an unready pod. This failure drill is your acceptance test.

Tool variants

• No GPU / laptop path: Ollama locally, or the vLLM CPU image with a small quantized model — same API shape, radically lower throughput; fine for learning the ops. Colab/Kaggle free GPUs or spot GPU rentals (~₹30–60/hr) cover the real-GPU exercise on a small budget.
• NVIDIA Triton / TensorRT-LLM: more setup, best raw performance on NVIDIA; consider when you outgrow vLLM's latency.
• llm-d (CNCF Sandbox): disaggregated prefill/decode serving built around vLLM for large multi-node estates — the concept to know for where this architecture goes next.

Verification

• •
  curl /v1/chat/completions returns a completion; /health returns 200; /metrics scrapes.
• •
  Pod restart re-serves without re-downloading weights (check PVC hit in logs) and probes hold traffic until ready.
• •
  Under a 20-concurrent-request burst, no 5xx — requests queue (watch num_requests_waiting) rather than crash.
• •
  GPU memory utilization is high and stable — vLLM pre-allocates; that is by design, not a leak.

Pitfalls

• Trusting assistant-drafted manifests blindly. Common omissions: /dev/shm, HF token secret, probe delays long enough for model load. Diff against the checklist in step 2.
• --max-model-len defaulting huge. The default context length can eat your KV-cache budget and tank concurrency; set what your workloads need.
• latest image in prod. vLLM moves fast; pin versions and test upgrades — flags and defaults change between releases.
• Exposing port 8000 publicly. The raw endpoint has no auth, no quotas, no guardrails. Gateway or nothing.

Restricted-environment variant

Self-hosting is the restricted-environment play — the model runs inside your (or the client's) VPC and prompts never leave. Complete it properly:

• •
  Air-gapped or egress-limited clusters: pre-pull weights into the PVC via a job, or bake them into an internal image; block Hugging Face egress after.
• •
  Log discipline: vLLM request logging can capture prompts — configure log redaction/retention to match the client's data policy.
• •
  Put the client's compliance boundary in writing: model, weights license, cluster, and log destinations.

Success criteria

• •
  A self-hosted OpenAI-compatible endpoint served from your cluster, surviving pod kills without downtime beyond load time.
• •
  You can explain, using your own metrics, why TTFT and TPS change under load (prefill contention vs KV-cache pressure).
• •
  Weights-fit and concurrency estimates you made in step 1 roughly match observed behavior.
• •
  The endpoint is only reachable through the platform gateway path — verified by a failed direct external request.