Lab 14 · Serve It Yourself: vLLM + Quantization

You will also demonstrate the strategic payoff of quantization: a 7B model serving on a 16 GB GPU that could never hold it in fp16.

Time & tier

Advanced · ~2 hours hands-on (10–15 minutes of that is installs and model downloads) · $0 on Kaggle's free GPU, or ~$0.15–0.50/hr on a T4/A10 spot instance.

Setup

Create a Kaggle notebook: Accelerator = GPU T4 x2 (we use one; a note on two later), Internet = On. Then:

!pip install -q -U vllm httpx
import torch; print(torch.cuda.get_device_name(0))

Two things about the T4 to internalize before serving anything: it has no bfloat16 (every launch below passes --dtype half, or vLLM will refuse models whose config requests bf16), and it is an SM75 part, so vLLM falls back to the plain AWQ kernels (the faster awq_marlin path needs SM80+, i.e. A100/A10-class) — expect AWQ's throughput win to be modest here and its memory win to be the headline.

Quality set: reuse holdout.jsonl from lab 11 (80 analyst-note → JSON items). If you skipped lab 11, run its dataset-generator cell (Setup section) — it is fully self-contained and seeded; you only need the holdout file. Scoring is deterministic (JSON validity + field accuracy), so quality evaluation costs no API credit.

Guided steps

1. 1.

   Launch the fp16 baseline. Like-for-like comparison first: Qwen2.5-1.5B-Instruct (Apache-2.0, ~3.1 GB fp16) against its official AWQ build. Start the server as a subprocess and wait for health:

   Copy

   import subprocess, time, httpx, os, signal
   
   def launch(args):
       proc = subprocess.Popen(["vllm", "serve"] + args,
                               stdout=open("vllm.log", "w"), stderr=subprocess.STDOUT)
       for _ in range(180):
           try:
               if httpx.get("http://localhost:8000/health", timeout=2).status_code == 200:
                   return proc
           except httpx.HTTPError:
               time.sleep(5)
       raise RuntimeError("server never became healthy — read vllm.log")
   
   proc = launch(["Qwen/Qwen2.5-1.5B-Instruct", "--dtype", "half",
                  "--max-model-len", "4096", "--gpu-memory-utilization", "0.90",
                  "--port", "8000"])
   

   Record VRAM: torch.cuda.mem_get_info() from a second cell, or !nvidia-smi --query-gpu=memory.used --format=csv. Note that vLLM pre-allocates KV-cache up to gpu-memory-utilization, so read the weights number from vllm.log and the total from nvidia-smi — report both.

2. 2.

   Benchmark: TTFT and throughput. Two numbers, two regimes. TTFT is a single-stream, user-experience number; throughput is a saturated, cost number. Create the harness:

   Copy

   import asyncio, time, json, httpx, statistics
   
   URL = "http://localhost:8000/v1/chat/completions"
   PROMPT = [{"role": "user", "content": "Summarize why evaluation harnesses matter for RAG systems, in about 150 words."}]
   
   async def one_stream(client, model):
       t0 = time.perf_counter(); ttft = None; toks = 0
       async with client.stream("POST", URL, json={"model": model, "messages": PROMPT,
                                                   "max_tokens": 200, "stream": True}, timeout=120) as r:
           async for line in r.aiter_lines():
               if line.startswith("data: ") and line != "data: [DONE]":
                   delta = json.loads(line[6:])["choices"][0]["delta"]
                   if delta.get("content"):
                       toks += 1
                       ttft = ttft or (time.perf_counter() - t0)
       return ttft, toks, time.perf_counter() - t0
   
   async def bench(model, concurrency=8, rounds=10):
       async with httpx.AsyncClient() as c:
           await one_stream(c, model)                                   # warmup
           singles = [await one_stream(c, model) for _ in range(rounds)] # single-stream TTFT
           t0 = time.perf_counter()
           batch = await asyncio.gather(*[one_stream(c, model) for _ in range(concurrency * 2)])
           wall = time.perf_counter() - t0
       ttfts = sorted(s[0] for s in singles)
       print(f"{model}: TTFT p50={statistics.median(ttfts)*1000:.0f}ms "
             f"p90={ttfts[int(0.9*len(ttfts))]*1000:.0f}ms | "
             f"throughput@{concurrency}={sum(b[1] for b in batch)/wall:.0f} tok/s")
   
   await bench("Qwen/Qwen2.5-1.5B-Instruct")
   

   (Chunk-count ≈ token-count is fine for relative comparison; keep the method identical across runs.)

3. 3.

   Quality run. Same endpoint, the 80-item holdout, deterministic scoring:

   Copy

   import json, httpx
   
   def quality(model):
       items = [json.loads(l) for l in open("holdout.jsonl")]
       keys = ["metric", "segment", "direction", "change_pct", "confidence", "driver"]
       valid = fields = 0
       with httpx.Client(timeout=120) as c:
           for it in items:
               r = c.post(URL, json={"model": model, "messages": it["prompt"],
                                     "max_tokens": 150, "temperature": 0})
               text = r.json()["choices"][0]["message"]["content"]
               gold = json.loads(it["completion"][0]["content"])
               try:
                   pred = json.loads(text[text.index("{"): text.rindex("}") + 1])
                   valid += 1
                   fields += sum(pred.get(k) == gold[k] for k in keys) / len(keys)
               except (ValueError, json.JSONDecodeError):
                   pass
       print(f"{model}: json_valid={valid/len(items):.2%} field_acc={fields/len(items):.2%}")
   
   quality("Qwen/Qwen2.5-1.5B-Instruct")
   

4. 4.

   Swap in AWQ — like for like. Kill the server (proc.send_signal(signal.SIGINT); proc.wait()), relaunch quantized, re-run steps 2 and 3 identically:

   Copy

   proc = launch(["Qwen/Qwen2.5-1.5B-Instruct-AWQ", "--quantization", "awq",
                  "--dtype", "half", "--max-model-len", "4096",
                  "--gpu-memory-utilization", "0.90", "--port", "8000"])
   

   Expect: weights VRAM roughly a third of fp16, more room for KV-cache (hence better behavior under concurrency), TTFT similar or slightly worse on T4 (dequant overhead on old kernels), quality within a point or two on this structured task. Whatever you observe, that is the number that goes in the report — not the blog-post number.

5. 5.

   The capability unlock: 7B on a 16 GB card. Qwen2.5-7B-Instruct in fp16 is ~15 GB of weights — it does not fit a T4 with any KV-cache. Its AWQ build (~5.6 GB) does, comfortably:

   Copy

   proc = launch(["Qwen/Qwen2.5-7B-Instruct-AWQ", "--quantization", "awq",
                  "--dtype", "half", "--max-model-len", "4096",
                  "--gpu-memory-utilization", "0.92", "--port", "8000"])
   await bench("Qwen/Qwen2.5-7B-Instruct-AWQ"); quality("Qwen/Qwen2.5-7B-Instruct-AWQ")
   

   This row is why quantization is a capacity decision, not just a speed tweak: the question becomes "7B-AWQ vs 1.5B-fp16 at the same hardware cost," and your quality harness is the referee. (Kaggle gives you a second T4: --tensor-parallel-size 2 splits a model across both — try it as a stretch goal and note the inter-GPU overhead.)

6. 6.

   The benchmark report. One table in SERVING.md — model × {weights VRAM, total VRAM, TTFT p50/p90, tok/s @8 concurrent, JSON-valid %, field-acc %} for all three configurations — followed by a five-sentence recommendation: which config you would deploy for this extraction workload, at what expected cost per million tokens (use spot pricing for a T4 in your region and your measured tok/s), and what quality you are giving up or gaining. This memo format is the deliverable employers actually read.

Verification

• •
  All three configurations served and benchmarked with the identical harness and prompts.
• •
  TTFT reported as p50/p90 from ≥10 single-stream runs after warmup; throughput measured under concurrency, not sequentially.
• •
  Quality scored on the untouched 80-item holdout for all three; fp16-vs-AWQ delta at 1.5B is quantified (expect ≤ ~2 points here).
• •
  The 7B-AWQ configuration demonstrably runs on the 16 GB card, with VRAM evidence.
• •
  The recommendation cites your measured numbers, including a cost-per-million-token estimate.

Completion artifact

SERVING.md (benchmark table + recommendation memo) + the notebook + vllm.log excerpts showing each launch config. Self-grade (pass at 4+):

• Clean measurement: warmup excluded and percentiles used.
• Stated concurrency: throughput measured under stated concurrency.
• Identical quality harness: the quality harness is identical across configs.
• Memory broken out: memory numbers separate weights from KV-cache.
• A decision, not a table: recommendation converts measurements into a cost/quality decision.

Stuck?

• Server dies at startup with a dtype error: the model config requests bf16 — you forgot --dtype half on the T4.
• OOM on launch: lower --gpu-memory-utilization to 0.85 and --max-model-len to 2048; confirm no previous server process is still holding memory (!nvidia-smi).
• Health check never passes: first launch downloads several GB of weights — check vllm.log for download progress before assuming failure; Kaggle Internet must be On.
• Throughput looks identical fp16 vs AWQ: plausible on T4 (plain-AWQ kernels); the memory column and the 7B row still carry the lesson — say so in the memo rather than forcing a story.
• vllm install extremely slow: normal (~5–10 min); it pins its own torch build.

Client-safe variant

Self-hosting is itself the client-safe answer — this lab is the playbook for engagements where prompts and data may not leave the client's network. Run the identical stack on a GPU instance inside the client's cloud account (a single T4/A10 spot in ap-south-1 or asia-south1 serves the AWQ builds at trivial cost) with no egress to any model provider. Additions for client work: put the endpoint behind their auth (vLLM's --api-key flag plus network policy — never expose :8000 publicly), verify the model license (Qwen2.5 1.5B/7B are Apache-2.0) against their OSS policy, pin the exact model revision in the launch command for reproducibility, and hand over SERVING.md with cost-per-million-tokens computed at their instance pricing so the build-vs-API economics are theirs to defend.