Serving, LLMOps & Leading the Transition

Everything the track has built — measured retrieval, eval gates, adapters, agents — becomes a production system in this module, and a production system has an operations bill someone must own. That someone is increasingly the senior DS/MLE: postings that name vLLM, quantization, and observability tooling in the same breath as "eval harnesses" are describing one job, the person who can serve a model, prove its quality hasn't drifted, attribute its cost, and roll it back when either goes wrong. The module closes with the skill that makes the rest legible upward: translating faithfulness curves and dollars-per-request into decisions an executive can actually make.

The through-line, one last time, is the eval harness from M5–M6. Quantization is a quality claim your harness verifies. Routing to a cheaper model is safe exactly when the golden set says so. A canary is promoted or rolled back on sampled eval scores. By the end of this module, the harness isn't a testing tool — it's the control system your serving infrastructure answers to.

What You'll Learn

• Self-hosting an open-weight model with vLLMPagedAttention, continuous batching, and an OpenAI-compatible endpoint you load-test for TTFT and throughput
• AWQ vs. GPTQ vs. FP8 quantizationhardware constraints, throughput effects, and quality verified with your own eval harness rather than leaderboard claims
• Model gatewayscheap-first routing, provider fallback, semantic caching, and per-team budgets
• Production observabilityOTel-based tracing, token-cost dashboards, and drift alerts from sampled online evals, with Phoenix, Langfuse, and MLflow as the self-hostable options
• Canary and rollback for prompts and modelstreating both as versioned deploys gated by evals
• Communicating performance and economics upwardmaking AI-system results legible to executives, and leading a team's transition

Self-Hosting with vLLM

The API-vs-self-host decision is an economics-and-constraints decision. APIs win on frontier quality, zero ops, and elastic scale. Self-hosting wins when data cannot leave your boundary (the default posture for client data in services work), when a tuned 7–8B model from M7 covers the task, or when sustained high volume makes per-token pricing worse than a GPU lease. What makes self-hosting viable at all is the serving engine, and vLLM has become the default answer — it's the engine named in job postings and the paved-path choice at major engineering organizations.

Two ideas do most of vLLM's work:

• PagedAttentionmanages the KV cache (the per-request memory that holds attention state, and the real capacity constraint in LLM serving) in fixed-size blocks, the way an OS pages virtual memory, eliminating the fragmentation that wastes most of a naïve server's VRAM.
• Continuous batchingadmits new requests into the running batch as others finish, instead of waiting for a full batch to drain, keeping the GPU saturated under real, ragged traffic.

Together they are why one mid-range GPU can serve a 7–8B model to a team.

Standing it up is deliberately anticlimactic:

pip install vllm

vllm serve Qwen/Qwen2.5-7B-Instruct-AWQ \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90 \
  --dtype half            # T4-class GPUs: fp16, no bfloat16

That exposes an OpenAI-compatible endpoint, so every client, gateway, and framework in your stack points at it with a base-URL change:

from openai import OpenAI  # pointed at your own server, not OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
resp = client.chat.completions.create(
    model="Qwen/Qwen2.5-7B-Instruct-AWQ",
    messages=[{"role": "user", "content": "Summarize this incident: ..."}],
)

Two flags deserve respect:

• --max-model-lenbounds per-request KV growth; set it to what your workload actually needs, because unused context length is concurrency you gave away.
• --gpu-memory-utilizationthe fraction of VRAM vLLM claims for weights plus KV cache; on a shared or small GPU, headroom prevents OOM at burst.

Then benchmark, because a server without a load test is a rumor. The metrics that matter: TTFT (time to first token — the felt latency for streaming UIs), inter-token latency, output tokens/second across the batch (system throughput), and request throughput at your target latency percentile. Use vLLM's bundled benchmark (vllm bench serve) or a k6/locust script against the endpoint, sweep concurrency from 1 to saturation, and record the curve — the knee of that curve is your capacity number, and it goes in the exec memo later, converted to dollars: GPU-hours per million tokens at your observed throughput versus the API price for equivalent quality.

Serving composes with M7: vLLM can mount LoRA adapters over a shared base (--enable-lora), so one deployed base model serves several task-tuned adapters — the deployment story that makes small fine-tunes operationally cheap. For CPU-only environments and laptop demos, Ollama or llama.cpp are the fallback path; the measurement discipline is identical.

Quantization: The Trade You Verify Yourself

Quantization shrinks weights (and sometimes activations) to lower precision, buying VRAM headroom — which becomes KV-cache space, which becomes concurrency — and often raw speed. The three formats you'll actually choose between for serving:

The pattern to internalize: published quality numbers for quantized checkpoints are averages over generic benchmarks, and your task is not a generic benchmark. Quantization error is uneven — it can be invisible on prose summarization and material on numeric reasoning, edge-case extraction, or your domain's rare vocabulary. So the M6 CI gate gets a new customer:

1. 1Freeze the FP16 baseline. Serve it, run the full eval harness against the endpoint, and freeze the scores.
2. 2Swap the checkpoint. The AWQ (or GPTQ/FP8) build — same server, same prompts, same harness.
3. 3Diff per-metric and per-slice. Exactly like the M7 before/after protocol: aggregate parity can hide a slice that collapsed.
4. 4Record the other side of the trade. Throughput, TTFT, and max concurrency at each precision, from the same run.

The deliverable is an eval-parity report — "AWQ holds within noise of FP16 on all slices except long-tail numeric extraction (−4 points), at 2.3x the concurrency on the same GPU" is a decision-ready sentence, and producing it is precisely the "benchmarking model performance" skill postings ask for. KV-cache quantization (FP8 cache) is a further headroom lever with the same verify-first rule.

Gateways: Routing, Fallback, Caching, Budgets

Once more than one model serves traffic — an API frontier model, your vLLM endpoint, a cheap tier for classification — you need a single choke point, or cost control and provider migration become archaeology. A model gateway (LiteLLM is the common self-hostable choice; large orgs build their own) gives every application one OpenAI-compatible URL and centralizes four concerns:

• Routing. Map task tiers to models: extraction and classification to a small cheap model, synthesis and multi-step reasoning to the frontier tier, your fine-tuned adapter for its specific task. The eval harness is what makes routing safe rather than reckless — a route change is qualified by running the golden set against the cheaper model and checking it clears that route's bar. Cascades (try cheap, escalate on low confidence or failed validation) push further: most requests never touch the expensive tier.
• Fallback. Provider outages and 429s reroute to a second model automatically. The M8 rule applies unchanged: a fallback model is a different model — eval it before you need it, or the outage becomes a silent quality incident.
• Caching. Two distinct layers. Provider-side prompt caching rewards stable-prefix design (system prompt and tool definitions frozen, volatile content last) with steep discounts on the cached span — an architectural habit, not a config flag. Gateway-side semantic caching returns a stored response when a new request embeds close to a previous one; it shines on repetitive internal analytics questions, but set the similarity threshold conservatively and exclude time-sensitive routes — a near-miss cache hit is a wrong answer served confidently.
• Budgets and keys. Per-team virtual keys with daily caps, so an agent gone loopy (M8) exhausts its own budget rather than the org's, and so cost attribution in the next section has clean dimensions to group by.

Observability: Traces, Cost, Drift

An unobserved LLM system fails silently in two currencies — quality drifts without alarms, and spend accrues without attribution. The instrumentation stack has three layers, and you should stand them up before the first production request, because retrofitting tracing after the incident is the expensive ordering.

Traces. Every request carries a span tree: retrieval step, each model call, each tool call, each agent hop — with model ID, prompt version, input/output/cached token counts, computed cost, latency, and stop reason on every LLM span, joined to your existing distributed traces by request ID. The ecosystem has converged on OpenTelemetry with the OpenInference semantic conventions for LLM spans, which keeps you portable across backends. The self-hostable options, all free:

The choice matters less than the fields: prompt version and model ID on every span, because every quality investigation starts with "what changed," and prompt edits don't show up in your deploy log unless you put them there.

Cost dashboards. From trace data, maintain cost-per-request and cost-per-day broken down by feature, route, model, and team — plus cache hit rate, because a caching regression looks exactly like a traffic spike until you split by cache-read tokens. Set budget alerts before the first invoice surprise. Your dashboarding skills transfer directly; the only new idea is that tokens are the metered unit and the gateway plus traces are the meter.

Drift alerts. Production quality drifts for reasons no code change explains — input mix shifts, a provider updates a model behind an alias, a corpus goes stale. The mechanism you already own scales to catch it: sample a few percent of production traces (plus everything user-flagged or validation-failed), run your M6 online eval suite over the sample on a schedule, and alert on metric movement — treating faithfulness-over-time as a monitored series with the same changepoint discipline you'd apply to any production metric. Route the alerting slice's traces into your annotation queue: this is the production→error-analysis→golden-dataset loop from M5, now running continuously. And keep the classical-ML monitoring you already do (feature drift, performance decay) in the same pane — most real systems are hybrids, and the LLM half doesn't excuse the GBM half.

Canary and Rollback for Prompts and Models

The final piece of discipline: a prompt change is a deploy, and a model swap is a migration. Both get the treatment code gets:

• Version everything. Prompts live in a registry (MLflow's prompt registry, Langfuse's prompt management, or git with hashes surfaced into trace attributes) — never inline in application code as anonymous strings. Model versions, adapter versions, and routing tables are pinned and change-controlled. The non-negotiable: given any production trace, you can name the exact prompt version and model that produced it.
• Gate offline, then canary online. The M6 CI gate runs first — golden set, thresholds, mandatory cases at 100%. What CI can't see is real traffic: input drift, weird encodings, the interaction between the new prompt and this week's queries. So promote through the gateway at a canary weight — a few percent of traffic on the new version — while the drift machinery evaluates sampled canary traces against control side by side, on quality metrics and cost/latency (a prompt edit that silently doubles output tokens is a cost incident wearing a quality costume).
• Make rollback boring. Promotion and rollback are the same operation — a routing-weight change at the gateway — which means rollback is instant, and auto-rollback is a rule: if canary eval scores drop below control by more than the pre-agreed margin, the weight goes to zero without a meeting. Because you defined the margin before the canary, the 2 a.m. decision is mechanical.

This loop — registry, offline gate, weighted canary, sampled online evals, automatic rollback — is the LLMOps answer to "how do you change anything in production without praying," and describing it end-to-end, with the eval harness as the arbiter at both gates, is the expected shape of the senior interview answer.

Communicating Performance and Economics Upward

The last skill is the one that determines whether any of this infrastructure gets funded: making AI-system performance legible to people who will never read a faithfulness score. Executives make resourcing and risk decisions; your job is to convert harness output and cost telemetry into exactly that — decisions.

• Translate metrics into consequences. "Faithfulness 0.87" means nothing upstairs; "roughly one in eight answers contains a claim our documents don't support, which is why the human-review gate stays on for billing questions" is a risk statement with a control attached. Do the translation yourself — if you leave it to the audience, they'll either ignore the metric or over-trust it. For every metric you report, attach the operational consequence and the mitigation.
• Report unit economics, not invoices. The board-legible number is cost per unit of business outcome — per resolved ticket, per report generated, per analyst-hour saved — trended over time and, where honest data exists, compared to the human-only baseline. Your traces give the numerator; instrumenting the denominator is ordinary analytics work you already know. Present the quality-cost frontier from your routing and quantization studies as the menu it is: "frontier-tier everywhere" and "cheap-tier everywhere" are both points on a curve, and the memo's job is to recommend a point and price the alternatives.
• Structure the memo for a decision. One page: the decision requested up front; three or four metrics with consequences attached; the cost trend and its driver; the top risk and its mitigation status; the ask. Charts follow the discipline you've always applied — one message per chart, axes that don't flatter. LLMs are good at drafting these memos from your dashboards and eval reports; the M1 rule is load-bearing here — every number in the memo is one you own, because the memo travels without you and your credibility travels with it.
• Lead the transition. Being the person who built this stack makes you the person others ask how to work now. What that means in practice: set the team's verification standards (what must be eval-gated, what human review is mandatory, what the sandbox rules are) and write them down; run error-analysis sessions as a team ritual so eval-first thinking spreads by practice rather than mandate; pair the skeptics with the enthusiasts on lab-sized internal projects; and keep an honest register of what didn't work and why — credibility with both your team and your leadership compounds on candor about failures, not on demo reels. The market context you carry into those conversations: this stack — evals, serving, observability, economics — is the differentiator employers repeatedly name, and teaching it to your team multiplies you rather than diluting you.

Check yourself: You swap an FP16 model for a 4-bit AWQ build to double concurrency, and the aggregate eval score holds within noise of the baseline. Are you clear to ship, and what is the decision-ready way to report the result?

Exercises

1. 1Serve and load-test. Deploy a 7–8B AWQ model on vLLM (Kaggle T4 or a spot GPU). Sweep concurrency with vllm bench serve or k6, and produce the TTFT/throughput curve with the knee marked. Deliverable: the benchmark report plus the exact serve command and flags, with one paragraph on what --max-model-len cost or bought you.
2. 2Quantization parity study. Run your full M6 harness against FP16 and AWQ builds of the same model on the same endpoint. Deliverable: the eval-parity report — per-slice quality deltas, throughput and concurrency gains, and a ship/don't-ship recommendation with the trade stated in one sentence.
3. 3Route it safely. Stand up a LiteLLM gateway over two tiers (your vLLM endpoint plus a cheap API model). Qualify one route for the cheap tier by running your golden set against it; wire a fallback and per-key budget. Deliverable: the gateway config, the qualifying eval run, and a demonstration of a budget cap triggering.
4. 4Instrument and catch the regression. Trace your M5–M8 app with Phoenix, Langfuse, or MLflow (prompt version and cost on every span), build the cost-per-day dashboard, then have a partner inject a regression (a degraded prompt or a mis-set temperature). Find it from traces and dashboards alone. Deliverable: the RCA memo with the trace evidence.
5. 5The executive memo. From your quantization study and routing costs, write the one-page memo: decision requested, three metrics with consequences, unit-economics trend, top risk with mitigation, and the ask. Have a non-technical reader confirm they could make the decision from it.

Your playbooks & labs

This module anchors the track's two closing labs. Lab L14 — Serve it yourself deploys a 7–8B open model on vLLM, load-tests TTFT and throughput, swaps in an AWQ build, and quantifies the quality-vs-throughput trade with your own eval harness — artifact: the serving config plus the benchmark and eval-parity report. Lab L15 — Observability capstone instruments the full L5–L12 application with Phoenix or MLflow tracing, runs online evals on sampled traces, builds the token-cost dashboard, and triages one injected production regression from traces alone — artifact: dashboards plus the regression RCA memo. Together with L11–L13 they complete the portfolio arc: a tuned, agentic, self-served, observed system with its economics documented. The playbooks provide the vLLM deployment and sizing worksheet (India-aware spot-GPU pricing paths), the quantization decision matrix, the gateway routing-table template with eval-qualification links, the observability field checklist, the canary/rollback runbook, and the executive memo template. Do this at work this week → pb-46 Self-Host an Open-Weight Model with vLLM (the load-tested endpoint behind Karthik's serving), pb-47 Quantization Trade-off Study: AWQ vs GPTQ vs FP8 (Neha's verify-on-your-own-harness parity report), and pb-52 Executive Communication of AI-System Performance (Priya's metric → consequence → mitigation framing). All labs fit the free tier: Kaggle T4 or low-cost spot instances, self-hosted OSS observability, cheap-tier judge models.

Further Reading

• •
  vLLM documentation — serving, PagedAttention, benchmarking, and LoRA adapter serving
• •
  vLLM — Quantization — supported formats and the hardware compatibility matrix
• •
  LiteLLM documentation — gateway deployment, routing, fallbacks, budgets, and virtual keys
• •
  Arize Phoenix documentation — OTel/OpenInference tracing and evals over traces
• •
  Langfuse documentation — tracing, cost tracking, prompt management, and annotation queues
• •
  MLflow — GenAI tracing — tracing and scorers inside the MLflow stack
• •
  OpenTelemetry documentation — the tracing substrate the LLM conventions build on
• •
  Anthropic — Prompt caching — prefix-caching mechanics and the stable-prefix design rule