Cost, Capacity & FinOps with AI

The Slack from finance lands at 09:12: "cloud bill's up 31% month-on-month — can someone explain before the board call at 2?" Divya has three hours and no appetite for spelunking Cost Explorer, so she pastes the daily-spend chart into an assistant and asks why. Ninety seconds later she has three fluent paragraphs: the spike, they say, is checkout-api's autoscaling group scaling out for early Diwali traffic, driven by a c5.4xlarge fleet that grew from four instances to twelve. It reads like something she'd have written herself.

It is also wrong in a way that would have survived the board call: checkout-api never scaled that week, there is no c5.4xlarge fleet in that account, and the real culprit is a NAT gateway data-processing charge that tripled after a PR rerouted the analytics workers through it.

The model didn't look anything up — it pattern-matched a plausible story onto a number. Divya only caught it because she reopened the billing console and the headline figure wouldn't reconcile. The five-minute check is the whole difference between a cost narrative and a confident fiction.

Cloud cost work is a strange fit for AI, and the strangeness is instructive. On one side, most FinOps toil is explanation — why did the bill jump, which team owns this spend, what does this rightsizing recommendation actually mean — and explanation over structured data is exactly what LLMs are good at. On the other side, cost data is where a hallucinated number does maximum damage with minimum visibility: a fabricated attribution in a cost narrative doesn't crash anything, it just quietly misdirects an engineering quarter or a client conversation. And unlike a wrong Terraform module, no plan command catches it.

So this module applies the track's verification-gate discipline to money. Four workflows: AI-explained cost anomalies where every number traces back to billing data; rightsizing as a review discipline where AI summarizes and humans validate against performance SLOs before anything resizes; forecasting with a strict division of labor — statistical models produce the forecast, LLMs critique and explain it, never the reverse; and the newest line item on your bill, GPU and LLM workload costs, where the FinOps discipline you already have meets a spend category that can double in a day because someone changed a prompt.

There's also a career logic to this module. Cost fluency is a differentiator for senior platform engineers — it's the language the business speaks — and the AI-workload cost material is where this track's two halves meet: the platform-for-AI skills of M7–M8 come with a bill, and the engineer who can serve a model and explain its unit economics is the one in the promotion conversation.

What You'll Learn

• Cost-anomaly explanationdetection from billing tooling, attribution narrative from AI, verification against the billing console before the story circulates
• FinOps reporting and unit economicsshowback narratives that scale across teams without a reporting headcount
• Rightsizing review disciplinewhat recommenders know, what they don't, and the SLO-validation gate between recommendation and change
• The forecasting division of laborwhy LLMs are the wrong tool for producing forecasts and the right tool for critiquing them
• GPU and LLM-workload cost managementtoken spend tracking, cost-per-request budgets, utilization economics, and cost regressions as first-class alerts
• Client-account constraintsrunning all of this where billing data is contractually sensitive

Cost Anomaly Explanation: Narrative with Receipts

The recurring FinOps fire drill: the daily spend chart steps up 30%, someone important asks why, and an engineer spends half a day spelunking through Cost Explorer groupings to produce three sentences. The detection half of this is already solved without LLMs — AWS Cost Anomaly Detection, Azure Cost Management alerts, GCP billing budgets, and FinOps platforms like Vantage or CloudZero all flag deviations against learned baselines. What stays expensive is the attribution story: which service, which team, which change, since when, one-off or run-rate. That story is a summarization-and-correlation task over structured data — an LLM sweet spot (UC-33).

The pipeline that keeps it honest:

1. Start from exported billing truth. Feed the model data, not access to vibes: the anomaly alert itself, plus a Cost and Usage Report (CUR) / cost-export slice grouped along the dimensions that answer attribution questions — service, linked account, region, and your cost-allocation tags (team, app, env). Add the change context the billing data can't see: deploy history, IaC merges to the affected account, scaling events for the window. If your tagging discipline is weak, this is where it bites — an untagged spend spike can only ever be attributed to "something in account 4711," and no model fixes that. Tag hygiene is to FinOps-AI what OTel semconv was to observability-AI in M4: the data quality floor.

2. Demand a structured, evidence-linked narrative. The prompt asks for a specific shape: what changed (dollar and percentage delta, which line items), when it started (with the granularity the export supports), the most likely driver ranked against alternatives, correlated change events, and — non-negotiable — the source rows/figures behind every claim. A workable output contract:

FINDING: NAT Gateway data-processing charges in ap-south-1 rose from
~$41/day to ~$312/day starting 2026-07-14.
EVIDENCE: CUR rows — usage type ApS1-NatGateway-Bytes, daily usage up
7.4x on 07-14; no other usage types in the account moved >10%.
CORRELATED CHANGE: PR #2119 merged 07-13 moved analytics workers to
private subnets (routing S3 traffic through NAT).
LIKELY DRIVER: worker S3 egress now traversing NAT instead of a
gateway VPC endpoint.
CHECK BEFORE CIRCULATING: confirm usage-type daily totals in the
billing console; confirm no VPC endpoint exists in that VPC.
PROPOSED FIX: add S3 gateway endpoint (no hourly charge for gateway
endpoints; verify current pricing for your region).

3. Verify against the console before the story leaves your desk. The verification gate for cost narratives is refreshingly cheap: the billing console is right there. Before an AI-drafted explanation goes to a manager, a client, or a decision, someone reproduces the headline numbers in Cost Explorer (or vendor equivalent) and confirms the correlated change actually exists. Two known failure modes make this non-optional. LLMs mangle arithmetic and units on large tables — sums drift, a lakh becomes a crore, daily and monthly figures get conflated. And they complete patterns: given a spike and a plausible-sounding service name, a model will confidently attribute spend to a resource that isn't in the export at all. The check takes five minutes; the wrong narrative in a client QBR costs considerably more.

FinOps narratives at scale

The same pattern, scheduled, becomes your reporting layer (UC-35): monthly showback narratives per team — spend versus last month, versus budget, top movers with drivers, commitment coverage, one action item — generated from the same tagged exports, in language a non-infra engineering manager acts on. This is low-risk, high-toil-value automation: the numbers come from the export (and can be templated deterministically, with the LLM writing only the connective narrative — a nice trick that shrinks the verification surface to prose), and the alternative was usually no per-team narrative, because nobody had the hours. Pair each narrative with unit economics where you have them — cost per order, per tenant, per 1K requests — because "₹18 lakh this month" invites panic while "cost per order fell 8% while orders grew 20%" invites correct conclusions. Unit-economics framing is also your best defense against the annual "why is the cloud bill growing" conversation being had in absolute numbers only.

Rightsizing: AI Proposes, Humans Validate Against SLOs

Every cloud and every FinOps tool now emits rightsizing recommendations — this instance to that size, this EBS volume to gp3, these idle resources to termination, this workload's requests/limits down 60%. The recommendations pile up unactioned because each one carries an implicit risk assessment someone has to do: is the headroom this recommender wants to reclaim actually unused, or is it there for a reason? That triage is where AI helps (UC-34) — and where the human gate is non-negotiable, because the recommender's blind spots are systematic.

What recommenders see is utilization history — typically averages over a lookback window of days to weeks. What they don't see:

The AI-in-the-loop workflow: feed the recommendation export plus workload context (utilization percentiles, SLOs, deploy topology, autoscaling config) to an assistant and ask it to triage, not endorse — bucketing each recommendation with its reasoning attached:

• Safe-applyclearly idle, stateless, redundant.
• Validate-firstsavings are real but a blind-spot flag is raised; the assistant must say which one, and name the specific validation the item needs.
• Rejectthe headroom is load-bearing.

This turns a 200-row CSV nobody reads into a reviewed queue with the risk thinking pre-drafted. The human owns every bucket assignment — the assistant's job is to make the review tractable, and its reasoning is checkable precisely because it must cite the utilization figures and constraints it used.

Then the gate that makes it engineering rather than optimism: validation against performance SLOs, not against the savings number. A rightsizing change ships like any risky change — load-test at the new size against SLO thresholds where you can, or roll out gradually (one replica, one AZ, canary traffic) with SLO burn-rate alerts watching, and an articulated rollback. The M4 material is the harness here: if you can't observe p95 latency and error rate per deployment slice, you can't validate a resize, and the honest sequencing is observability first, rightsizing second. Kubernetes-side, the same discipline applies to VPA recommendations and to consolidation tools like Karpenter or Cast.ai — automated bin-packing is a policy you review and bound (disruption budgets, exclusion lists for stateful/latency-critical workloads), not a switch you flip fleet-wide.

Forecasting: Models Forecast, LLMs Explain

Here is this module's deliberate anti-pattern lesson (UC-36), and it generalizes far beyond cost: do not use an LLM to produce a numerical forecast. It's worth being precise about why, because the failure is structural, not a prompt-engineering gap. Capacity and cost forecasting is extrapolation over time-series — trend, seasonality, changepoints, uncertainty intervals. That's the home turf of statistical methods (Prophet-class decomposition, ARIMA-family models, or your observability platform's built-in forecasting, e.g. PromQL's predict_linear for quick linear projections and vendor forecast monitors for seasonal ones). An LLM asked to "forecast next quarter's spend from this CSV" produces plausible-looking numbers with no model behind them — no decomposition, no honest uncertainty, no reproducibility. Same class of error as trusting a language model's arithmetic, at quarterly-planning stakes.

The division of labor that works:

• The statistical model produces the numberspoint forecast plus uncertainty intervals, from a pipeline you can rerun. Even a simple Prophet fit on daily tagged spend gets you decomposed trend/seasonality and honest intervals in a dozen lines of Python — which your assistant will happily write; writing the forecasting code is a great LLM task even though being the forecast is not.
• The LLM critiques the setupbefore you trust the model, have the assistant review the choices: is the history window long enough to capture the seasonality being claimed? Are known one-offs (a migration month, an incident's cost spike) polluting the training window? Are there scheduled step-changes — a product launch, a committed migration, a contract renewal — that no amount of history can predict and that must be added as assumptions? This assumption-surfacing is genuinely valuable and genuinely language-shaped work.
• The LLM explains the outputtranslate the forecast and its intervals for the budgeting audience: what's driving the trend, what the interval width means for how much buffer to hold, which assumptions would invalidate it. A forecast that ships with its assumptions stated is one that can be revisited intelligently when reality diverges; a bare number is just a future argument.
• The human owns the decisioncommitments (Reserved Instances, Savings Plans, committed-use discounts) are financial instruments purchased against forecast confidence. Commitment coverage decisions take the statistical forecast, the stated assumptions, and business judgment about what the model can't know. No part of that sentence contains "the LLM decided."

This pattern — deterministic engine for numbers, language model for critique and narrative, human for judgment — is the same one you saw with scanners in M2 (Checkov finds, AI explains) and recommenders above. Recognizing which role a tool should play is the trust-calibration skill this track keeps exercising from new angles.

GPU and LLM-Workload Costs: FinOps for AI

The bill your FinOps practice grew up on moves in slow, capacity-shaped curves. The AI-workload bill doesn't — and as your org adopts the M7–M8 platform capabilities (model serving, gateways, agents), this becomes your problem, because phase-1 job-posting research shows "LLM cost/latency observability" appearing verbatim in platform-role requirements. Two sub-bills, two disciplines (UC-37):

Token spend (API and gateway-routed usage). Its defining property: cost is a function of behavior, not capacity. A prompt-template change that doubles context length doubles that call's input cost with zero infrastructure change; an agent that retries in a loop can burn a week's budget overnight; a model-version swap repriced every call. Nothing in your capacity-shaped FinOps tooling catches any of this, so the controls live in the serving path — which, after M7, you operate:

• Meter at the gateway. A LiteLLM/Portkey-class gateway (or Langfuse-style tracing at the app layer) records tokens, model, and computed cost per request, attributed by virtual key to team/app/env — cost-allocation tags for tokens. This attribution is the foundation everything else stands on, exactly as CUR tags were above.
• Budget and rate-limit per key. Gateways enforce per-key budgets with hard cutoffs or alerts, plus model allowlists so an experiment can't quietly route to the most expensive frontier model. For agentic workloads, per-key budgets are a safety control as much as a financial one — a runaway loop hits its budget ceiling before it hits your invoice; this is the cost face of M8's guardrails material.
• Track cost-per-request as an SLI-like signal, and alert on regressions. Aggregate spend hides the story; cost-per-request (or per conversation, per document, per unit of your product) exposes it. A prompt or model change that regresses cost-per-request 40% should page or block like a latency regression — wire it into M8's prompt-rollout CI as a gate metric alongside eval scores. Cost regressions from behavior changes are the fastest-moving cost class you'll ever manage; treat them with alert-grade urgency, not month-end-report urgency.

GPU spend (self-hosted serving and batch). Here capacity thinking returns, with brutal unit prices and one dominating metric: utilization. A GPU node idling at 15% while its pod holds the whole device is the single most common AI-infra waste pattern, and the levers are M7's scheduling machinery viewed through a cost lens:

• Queueing and quota (Kueue) — batch work fills the gaps instead of provisioning new nodes.
• Fractional sharing (MIG, time-slicing) — small inference workloads stop hoarding whole devices.
• Right-shaped autoscalingscale on tokens-in-flight or queue depth, not CPU, via Karpenter-class provisioning.
• Spot/preemptible GPUsfor interruptible batch inference, with checkpointing.

Instrument DCGM utilization metrics next to allocation data (OpenCost-class tooling) so you can see cost and usage per namespace/team in one place; a GPU allocation report without utilization is a bill, not information.

And keep the strategic comparison live: self-hosted-vs-API is not a one-time decision. The break-even between "run vLLM on GPUs we pay for hourly" and "pay per token" shifts with your traffic volume, utilization achieved, and every provider price change — re-run the comparison quarterly with real numbers from your own gateway metering and GPU utilization data. This is also squarely an India-context decision: for services companies, client-data constraints may mandate self-hosting regardless of unit economics, which makes achieving high GPU utilization the difference between a compliant architecture and a compliant-but-indefensible bill. The FinOps Foundation's FinOps for AI guidance is tracking this domain as it stabilizes; the durable content is exactly what this section taught — attribute, budget, unit-cost, utilize.

Check Yourself

Finance asks you for next quarter's spend forecast. Why should the forecast number never come out of an LLM — and what are the LLM's two legitimate jobs in that pipeline?

Exercises

1. 1Anomaly autopsy with receipts. Take a real cost movement from the last quarter (any step-change ≥15% in some grouping). Export the relevant CUR/cost-export slice plus deploy history, run the structured-narrative prompt, then verify every figure and the correlated change in the billing console. Deliverable: the narrative, the verification notes, and any claim the console check corrected.
2. 2Showback narrative pilot. For two teams with decent tagging, generate this month's showback narrative (spend vs last month, top movers with drivers, one action item). Have each team's lead rate accuracy and usefulness. Deliverable: both narratives plus the leads' corrections — the corrections tell you where your tagging or context data is thin.
3. 3Rightsizing triage. Export your recommender's current recommendations (any cloud, any tool). Run the AI triage into safe-apply / validate-first / reject with reasoning. Manually audit ten items — do you agree with the bucket and the cited evidence? Deliverable: the triaged queue, your audit notes, and one validate-first item taken through an SLO-watched rollout plan (executed or written).
4. 4Forecast with a critique pass. Fit a simple statistical forecast (Prophet, or even predict_linear) on six-plus months of daily tagged spend for one service. Then run the LLM critique pass: history-window sufficiency, one-off pollution, unpredictable step-changes to add as assumptions. Deliverable: forecast with intervals, the critique findings, and a stakeholder-ready explanation paragraph.
5. 5Unit-cost an AI workload. For any LLM-backed feature you or your org runs (or the stack you build in the M7 labs): meter a week of usage, compute cost-per-request and its p95 (heavy conversations are your cost tail), and set a budget alert threshold. Deliverable: the unit-cost figures, the tail analysis, and the configured alert.

Your playbooks & labs

This module has no dedicated lab by design — its disciplines thread through the M7 labs, where cost becomes concrete: the serving and gateway lab (deploy an open model on vLLM, front it with LiteLLM) is where you configure virtual keys, per-team budgets, and provider fallbacks and produce a real latency/cost report; the LLM observability and guardrails lab (Langfuse tracing over that stack) is where tokens, cost-per-request, and cost-regression visibility get instrumented and proven.

Three playbooks give you this module's workflows in copy-adaptable form:

• pb-27 — Explain a Cloud Cost Anomaly, and Verify It Against Billing Datathe structured narrative contract plus the verify-before-circulating console checklist (and the scheduled showback generator).
• pb-28 — Turn Rightsizing Recommendations into a Reviewed, Safe Change Planthe AI triage rubric, the recommender blind-spot table, and the SLO-watched rollout gate.
• pb-30 — Put Cost Guardrails on GPU and LLM Workloadsgateway metering, per-key budgets, the cost-per-request regression alert, and the GPU-utilization worksheet.

Each ships a client-account variant covering billing-data sensitivity: client cost data is contract-restricted at most services companies, so the playbooks default to aggregated/redacted exports and client-tenancy or self-hosted models for anything containing account-level billing detail. (The forecast-critique workflow from the section above lives in pb-29 — Sanity-Check a Capacity Forecast with AI, Without Letting It Forecast.)

Further Reading

• •
  FinOps Foundation — FinOps Framework — the phase/capability vocabulary this module's workflows slot into
• •
  FinOps Foundation — FinOps for AI — the working-group view of AI-workload cost management as the practice stabilizes
• •
  AWS — Cost and Usage Reports (CUR) documentation — the export format your anomaly narratives should be built on
• •
  AWS — Cost Anomaly Detection — the detection layer upstream of AI explanation
• •
  Microsoft — Azure Cost Management documentation — exports, budgets, and anomaly alerts on Azure estates
• •
  OpenCost documentation — open-source Kubernetes cost allocation, including GPU visibility
• •
  Infracost documentation — cost estimation in IaC review, the pre-deploy end of this module's pipeline
• •
  Prophet documentation — decomposition-based forecasting with uncertainty intervals
• •
  Kueue documentation — job queueing and quota for GPU utilization
• •
  Karpenter documentation — consolidation and right-shaped node provisioning
• •
  LiteLLM — Budgets and rate limits — per-key budgets and spend tracking at the gateway
• •
  Langfuse documentation — token/cost tracing for the unit-cost worksheet