Quantization Trade-off Study: AWQ vs GPTQ vs FP8

Tier: Advanced · Workflow area: Serving & LLMOps

The output is an eval-parity report that says which build to serve and what it costs in quality.

When to use

• •
  You serve (or plan to serve) an open-weight model and VRAM or throughput is the binding constraint.
• •
  Deciding between a bigger GPU and a smaller model representation — a purchasing decision that deserves data.
• •
  Directly after pb-46 (serving baseline) with the eval harness from your evals-in-CI work available.

Skip if you only ever call hosted APIs — quantization choices are the provider's problem there.

Prerequisites

• •
  The pb-46 vLLM setup and benchmark scripts.
• •
  Your task eval harness: 50–100 labeled examples with automated scoring (exact match, rubric judge, or Ragas metrics for RAG) — quality deltas of 1–3% are invisible without one.
• •
  Hardware note (verified against vLLM's support matrix): AWQ needs Turing or newer (T4 ✓); GPTQ runs on Volta+ (T4/P100 ✓); FP8 (W8A8) requires Ada Lovelace or Hopper (L4, L40S, H100) — it will not run on T4 or A10/Ampere. On Kaggle's T4, the study is AWQ vs GPTQ vs FP16.
• •
  Pre-quantized checkpoints from the Hub (search <model>-AWQ, <model>-GPTQ, -FP8) — quantizing yourself is a separate project.

The workflow

1. 1.

   Fix everything except the quantization. Same base model, same --max-model-len, same sampling parameters, same prompt set, same eval harness. One variable.

2. 2.

   Serve each build in turn (one at a time on a single GPU):

   Copy

   vllm serve <org>/<model>-AWQ  --quantization awq  --max-model-len 8192
   vllm serve <org>/<model>-GPTQ --quantization gptq --max-model-len 8192
   vllm serve <org>/<model>-FP8  --quantization fp8  --max-model-len 8192  # Ada/Hopper only
   # baseline: the FP16 model with no --quantization flag (needs ~2x VRAM)
   

3. 3.

   Measure VRAM honestly. Record nvidia-smi peak memory after warm-up at your benchmark concurrency — KV cache grows with load, so idle VRAM understates the footprint. Also record max concurrent requests before out-of-memory or queue collapse.

4. 4.

   Run the quality harness against each endpoint. Same 50–100 examples, 2–3 repeats if outputs are non-deterministic. Score per-example so you can inspect which items degrade — quantization damage is often concentrated in long-form reasoning and math, not spread evenly.

5. 5.

   Run the throughput protocol from pb-46 on each build: TTFT p50/p95 and system tok/s at fixed concurrency. Documented FP8 experience is ~25–30% throughput gain over FP16 on supported hardware; 4-bit builds mostly buy VRAM (bigger batch = more throughput at high concurrency), not single-stream speed.

6. 6.

   Assemble the parity report. One table: build × {quality score, Δ vs FP16, p95 TTFT, tok/s, peak VRAM, max concurrency}, plus 3 worked examples of the worst quality regressions. End with a recommendation and the condition that would reverse it ("AWQ; revisit if we add chain-of-math features where its Δ concentrated").

Tool variants

• LLM Compressor / GPTQModel / AutoAWQ if you must quantize a fine-tuned model yourself (e.g. your pb QLoRA merge) — budget a separate session.
• bitsandbytes on-the-fly quantization for quick VRAM relief in experiments; generally slower than pre-quantized AWQ/GPTQ kernels for serving.
• Quantized KV cache (vLLM option) — an orthogonal lever worth one extra benchmark column when context lengths are long.

Verification

• •
  Identical prompt set, sampling params, and harness across all builds — diffable configs prove it.
• •
  FP16 baseline included (even if measured on a bigger cloud GPU than you'd serve on) so deltas have an anchor.
• •
  VRAM measured under load, not idle; harness scores repeated and stable within noise.
• •
  Worst-regression examples inspected by eye — you can say what kind of items degrade, not just how much.

Pitfalls

• Comparing different fine-tunes.-AWQ uploads from random accounts are sometimes quantized from a different base revision; check the model card lineage or quantize from your own weights.
• FP8 on unsupported hardware. It errors or silently falls back — check the support matrix before renting the GPU.
• Judging quality by vibes on 5 prompts. The whole point is that small deltas need a harness; no harness, no study.
• Ignoring load-dependent effects. A build that wins at concurrency 1 can lose at 32 when its VRAM headroom runs out — always report at your target concurrency.

Restricted-environment variant

Runs entirely on your infra by construction. If eval examples contain client data, the rubric-judge scoring must also use a local model (serve a second small model, or score with the FP16 build judging the quantized ones — note the conflict of interest in the report). Kaggle free tier covers the full AWQ/GPTQ/FP16 study within a weekly GPU allowance; FP8 needs a rented L4/H100 spot box for an hour or two.

Success criteria

• •
  A parity table covering ≥3 builds with quality, latency, throughput, VRAM, and concurrency columns.
• •
  A recommendation with a stated reversal condition, tied to where the quality delta concentrated.
• •
  Reproducible scripts: anyone with the GPU can regenerate the table.
• •
  You can state your task's measured quality cost of 4-bit serving to one decimal place.