Put Cost Guardrails on GPU and LLM Workloads

Tier: Advanced · Hands-on time: ~60 minutes

This playbook stands up the minimum viable cost-control layer for LLM and GPU workloads — per-team budgets, cost-per-request visibility, utilization tracking, and regression alerts — using gateway-native spend tracking rather than month-end billing archaeology.

When to use

• •
  Your platform now serves LLM traffic (hosted APIs, self-hosted vLLM, or both) and nobody can answer "what does one request cost, per team?".
• •
  GPU nodes exist in the cluster and their utilization is unknown or embarrassing.
• •
  After the first LLM bill surprise — or ideally, the week before the workload ships.

Skip the GPU half if you are purely on hosted APIs; skip the gateway half only if a gateway is genuinely impossible (then budget enforcement moves to provider-level limits, which are much cruder).

Prerequisites

• •
  An LLM gateway in the request path — LiteLLM (self-hosted, MIT) or Portkey-class — or a plan to insert one; this is where budgets and attribution live.
• •
  For GPU workloads: Prometheus + DCGM exporter (or NVIDIA GPU Operator metrics) and OpenCost for K8s cost allocation.
• •
  Provider price sheets for your models, and an agreed cost-allocation key (team, product, or client).
• •
  Comfort from Common Core with tokens, context windows, and why output tokens usually dominate cost.

The workflow

1. 1.

   Route all LLM traffic through the gateway with virtual keys. One virtual key per team/app — never shared provider keys. In LiteLLM: define teams, attach max_budget and budget_duration, set rpm/tpm limits. Attribution now exists at request time, not invoice time.

2. 2.

   Set budgets that fail loud, not silent. Configure soft alerts at 60/80% and a hard cap at 100% per key. Decide per workload whether hitting the cap should degrade (fallback to a cheaper model) or stop — write that choice into the gateway config, and document it where the owning team will find it.

3. 3.

   Instrument cost-per-request. Enable gateway spend logging and export to your observability stack (Langfuse for traces, or Prometheus metrics). Define the two SLI-like numbers that make LLM cost manageable: cost-per-request (p50/p95, per route) and tokens-per-request. Dashboards show them next to latency (TTFT/TPS) — cost regressions and latency regressions usually share a cause.

4. 4.

   Alert on regressions, not just totals. A monthly budget alert catches disasters late. Add: cost-per-request p95 doubling over 24h baseline; tokens-per-request jump on a single route (prompt change or retry loop); spend rate that would exhaust the monthly budget in under a week. Use AI to draft these alert rules from your metric names, then review thresholds yourself.

5. 5.

   Do the GPU side with utilization SLOs. From DCGM metrics, build the honest dashboard: allocation vs actual GPU utilization per namespace, cost per GPU-hour from OpenCost. Set a floor (e.g., "sustained <30% for 7 days triggers a review"), enable bin-packing improvements (time-slicing/MIG for small inference, Kueue for batch queueing), and route batch jobs to spot/preemptible pools.

6. 6.

   Generate the monthly AI-cost narrative. Feed the gateway spend export and GPU utilization summary into the pb-27 workflow: per-team narrative, unit economics (cost per 1K requests, per feature), verified against the provider invoice. The gateway numbers and the invoice will disagree slightly — reconcile and document why (caching, retries, billing lag).

Tool variants

• LiteLLM (default, self-hosted): virtual keys, team budgets, spend logs, model fallbacks — the full playbook runs on the free MIT core.
• Portkey / Kong AI Gateway: same concepts; Portkey adds guardrail hooks at the gateway, Kong suits orgs already on Kong for API management.
• Provider-native controls only (no gateway): per-key spend limits on the provider console + tagged API keys per team. Coarse, no fallbacks, no cost-per-request — treat as temporary.
• Langfuse (self-hosted) for trace-level cost: attach cost to individual traces/features when product asks "what does this AI feature cost per user?".

Verification

• •
  Fire a test: run a scripted loop against a low-budget test key and confirm the soft alert fires and the hard cap actually blocks or degrades as configured.
• •
  Cost-per-request dashboard shows sane numbers that reconcile with provider pricing math on a hand-checked sample of 10 requests.
• •
  Gateway monthly total vs provider invoice reconciles within an explained margin.
• •
  GPU dashboard: every GPU node attributable to a namespace/team; utilization floor alert tested against a known-idle pool.

Pitfalls

• Shared provider keys. One key for everything means no attribution, no per-team caps, and one leaked key drains the whole budget.
• Budget alerts without a degrade decision. Teams discover the hard cap in production at peak; decide degrade-vs-stop per workload up front.
• Ignoring output-token asymmetry. A prompt tweak that lengthens responses can triple cost at flat request volume — hence tokens-per-request alerts per route.
• Retry storms. Client retries on timeout multiply spend invisibly; alert on requests-per-user-action if you can, and cap retries at the gateway.
• GPU cost theater. Showing GPU node cost without utilization lets 20%-utilized clusters look "allocated"; always pair cost with utilization.

Restricted-environment variant

• •
  Services-company pattern: run LiteLLM inside the client VPC in front of the client's approved endpoints (Bedrock/Azure OpenAI private) — budgets, attribution, and logging stay in the client boundary, and no prompt content leaves it.
• •
  Redact spend-log payloads: keep token counts and costs, drop prompt/response bodies from any export that leaves the environment, or enable the gateway's logging-off/redaction mode for sensitive routes.
• •
  Free-tier lab path: Ollama serving a small model + LiteLLM in front of it reproduces every mechanism in this playbook at ₹0 — budgets, virtual keys, spend logs — before you touch paid providers.

Success criteria

• •
  Every LLM request attributable to a team and a budget; every GPU-hour attributable to a namespace.
• •
  A deliberately induced cost regression is caught by an alert within hours, not by the invoice within weeks.
• •
  You can state cost-per-request and cost-per-1K-requests for your top routes from a dashboard, not a spreadsheet.
• •
  The monthly AI-cost narrative ships with reconciled numbers and has survived a finance review.