Self-Host an Open-Weight Model with vLLM

Tier: Advanced · Workflow area: Serving & LLMOps

This playbook deploys a 7–8B model behind an OpenAI-compatible API and measures the two numbers that define serving quality: time-to-first-token (TTFT) and tokens-per-second (TPS) under concurrency.

When to use

• •
  Data can't leave your environment (client work, regulated data) but you need LLM capability.
• •
  API spend at volume justifies owning inference, and you need real numbers to make that case.
• •
  A serving benchmark is the prerequisite for the quantization study (pb-47) and gateway routing (pb-48).

If you need frontier-model quality or your volume is a few thousand requests a day, a hosted API is cheaper and better — self-hosting wins on privacy, unit economics at scale, and latency control, not on peak quality.

Prerequisites

• •
  One NVIDIA GPU with ≥16GB VRAM: a T4/P100 (Kaggle free tier, 30 GPU-h/week), or a T4/A10 spot instance ($0.10–0.40/hr in Indian regions). A 7–8B model in AWQ 4-bit fits a T4; FP16 needs ~16GB+ for weights alone.
• •
  pip install vllm (Linux + CUDA; use the official Docker image if the host is messy).
• •
  A load-testing tool: k6, locust, or vLLM's built-in benchmark CLI.

The workflow

1. 1.

   Pick the model to match the VRAM. On a 16GB T4, serve a 4-bit AWQ build of a 7–8B instruct model (e.g. Qwen/Qwen2.5-7B-Instruct-AWQ). On a 24GB A10, FP16 7–8B fits.

2. 2.

   Serve it.

   Copy

   vllm serve Qwen/Qwen2.5-7B-Instruct-AWQ \
     --quantization awq \
     --max-model-len 8192 \
     --gpu-memory-utilization 0.90 \
     --port 8000
   

   --max-model-len bounds context (longer = more KV-cache per request = fewer concurrent requests); --gpu-memory-utilization is how much VRAM vLLM may claim for weights + KV cache.

3. 3.

   Smoke-test via the OpenAI-compatible API. Any OpenAI client works with base_url="http://localhost:8000/v1" and a dummy key; hit /v1/chat/completions and confirm sane output and a populated usage block.

4. 4.

   Benchmark methodically. Fix a representative prompt set (your real task's length distribution, not "hello"). Sweep concurrency 1 → 4 → 16 → 32, and at each level record: median and p95 TTFT, per-request decode TPS, total system throughput (tok/s across all requests), and error/timeout rate. vLLM exports Prometheus metrics at /metrics (TTFT and e2e latency histograms) — scrape them rather than trusting client-side timing alone.

5. 5.

   Read the curves. Expect: system throughput climbs with concurrency (continuous batching working) until KV-cache saturates; TTFT degrades first as requests queue. The knee of the p95-TTFT-vs-concurrency curve is your capacity number — write it down as "N concurrent users at ≤X s p95 TTFT."

6. 6.

   Compute unit economics. (tokens served during the sweep ÷ GPU-hours × spot price) → ₹ or $ per million tokens. Put it next to hosted-API pricing for the closest comparable model — this table is the deliverable your manager cares about.

Tool variants

• vLLM Docker image (vllm/vllm-openai) — same flags, cleaner host; the right default on cloud VMs.
• Ollama / llama.cpplaptop-grade CPU/metal serving for development; no continuous batching, so don't benchmark capacity on it.
• TGI (Text Generation Inference)the main alternative engine; same benchmark protocol if your org standardizes on it.
• vllm bench servebuilt-in benchmark client as a cross-check on your k6/locust numbers.

Verification

• •
  Endpoint serves correct, non-truncated completions through an unmodified OpenAI client.
• •
  Benchmark table covers ≥4 concurrency levels with TTFT p50/p95, TPS, throughput, and error rate per level.
• •
  Server-side /metrics histograms roughly agree with client-side measurements (large gaps mean client-side queuing).
• •
  Cost-per-million-tokens computed and compared against at least one hosted-API price.

Pitfalls

• Benchmarking with toy prompts. Short prompts flatter TTFT and throughput; use your production length distribution.
• --max-model-len left at model default. A 32k default eats KV-cache budget and quietly halves concurrent capacity when your real prompts are 2k.
• Ignoring cold starts. First requests after model load are slow; warm up before measuring, but also record cold-start time — schedulers and spot restarts hit it.
• Spot-instance amnesia. Spot boxes die; if this becomes real infra, you need a restart story and health checks — note it in the report even if you don't build it.

Restricted-environment variant

Self-hosting is the restricted-environment answer — that is its point. For fully air-gapped setups, pre-download model weights (huggingface-cli download) on a connected machine and transfer; vLLM serves from a local path. Keep the endpoint on a private network and put authentication in front (vLLM's --api-key flag or a reverse proxy) before any colleague touches it.

Success criteria

• •
  A 7–8B model serving on hardware you sized correctly, benchmarked across a concurrency sweep.
• •
  A stated capacity number ("N concurrent @ ≤X s p95 TTFT") backed by data.
• •
  A cost table (self-hosted ₹/M tokens vs hosted API) you could defend in a platform review.
• •
  You can explain PagedAttention and continuous batching in two sentences each, from observed behavior.