Lab 06 · TCO Model for a RAG Application

Anita's orbit-docs-copilot is approaching GA and the CFO wants a number. Industry surveys keep finding that most AI projects blow their budgets, and the root cause is rarely arithmetic — it is unverified inputs and missing sensitivity. One rule governs this whole lab: verify today's price — do not trust any number in this lab or from the model. LLM pricing recall is unreliable by construction; prices change faster than training data.

Time & tier

~90 minutes hands-on · Core tier (maps to module M7). Cost: ₹0 — a spreadsheet or Python, public pricing calculators, and a chat assistant.

Setup

Create rag-tco/ with prices.md (empty — your evidence file) and the model script below as tco_rag.py. The workload numbers are synthetic Orbit GA estimates; the prices are deliberately None and the script refuses to run until you fill them from live sources.

# tco_rag.py — three-way TCO model. EVERY price is a PLACEHOLDER.
# Fill each None from the provider's own calculator/price page TODAY,
# and record source URL + date in prices.md before running.

PRICES = {
    "api_input_per_1m": None,       # hosted API: $ / 1M input tokens
    "api_output_per_1m": None,      # hosted API: $ / 1M output tokens
    "api_cached_in_per_1m": None,   # $ / 1M cached (cache-read) input tokens
    "pt_unit_per_hour": None,       # provisioned throughput: $ / unit-hour
    "pt_unit_tokens_per_min": None, # vendor-stated tokens/min per unit
    "gpu_instance_per_hour": None,  # self-hosted: cloud GPU $ / hour
    "gpu_tokens_per_sec": None,     # benchmarked serving throughput
}
W = {
    "requests_per_day": 40_000,   # synthetic Orbit GA estimate
    "input_tokens": 6_000,        # system + question + retrieved context
    "output_tokens": 400,
    "cache_hit_rate": 0.40,       # fraction of input served from cache
    "peak_to_avg": 3.0,
}

missing = [k for k, v in PRICES.items() if v is None]
assert not missing, f"Fill from live calculators first: {missing}"

def hosted_api(w, p):
    din = w["requests_per_day"] * w["input_tokens"]
    cached, fresh = din * w["cache_hit_rate"], din * (1 - w["cache_hit_rate"])
    dout = w["requests_per_day"] * w["output_tokens"]
    return (fresh * p["api_input_per_1m"] + cached * p["api_cached_in_per_1m"]
            + dout * p["api_output_per_1m"]) / 1e6 * 30

def provisioned(w, p):
    peak_tpm = (w["requests_per_day"] / 86_400) * w["peak_to_avg"] * 60 \
               * (w["input_tokens"] + w["output_tokens"])
    units = max(1, -(-peak_tpm // p["pt_unit_tokens_per_min"]))
    return units * p["pt_unit_per_hour"] * 24 * 30

def self_hosted(w, p):
    peak_tps = (w["requests_per_day"] / 86_400) * w["peak_to_avg"] \
               * (w["input_tokens"] + w["output_tokens"])
    gpus = max(1, -(-peak_tps // p["gpu_tokens_per_sec"]))
    return gpus * p["gpu_instance_per_hour"] * 24 * 30

for name, fn in [("hosted API", hosted_api), ("provisioned PT", provisioned),
                 ("self-hosted GPU", self_hosted)]:
    print(f"{name:16s} ≈ ${fn(W, PRICES):,.0f}/month")

Guided steps

Step 1 — Pin the workload model (15 min)

Before touching prices, interrogate the synthetic workload with your assistant: which assumption moves cost the most? Sketch the arithmetic: daily input tokens dwarf output tokens in RAG (6,000 vs 400 here) because retrieved context rides along on every request — which is why context length and caching, not model choice alone, dominate RAG economics. Write your workload assumptions and their sources ("Orbit GA estimate — synthetic") at the top of prices.md. Adjust peak_to_avg if your judgment says B2B docs traffic is spikier than 3x.

✅ Checkpoint: prices.md opens with the workload table and a one-line justification per assumption.

Step 2 — Fill every price from a live source (20 min)

Open the actual pricing pages/calculators for one hosted API provider, that provider's (or a hyperscaler's) provisioned-throughput offering, and one cloud GPU instance family suitable for serving an open-weights model. Fill all seven PRICES values. For each, record in prices.md: value, exact source URL, and today's date. For gpu_tokens_per_sec, use a published serving benchmark for your chosen model/GPU pair and record that source too — never let the assistant "estimate" it.

Then run the audit that motivates this lab: ask your AI assistant what these prices are, and log its answers next to the verified ones. Any delta — and there will be deltas — goes in a short "AI price audit" section.

✅ Checkpoint: script runs (assertion passes); prices.md has 7 sourced prices, each dated, plus the AI-vs-calculator audit table.

Step 3 — Produce the base comparison (15 min)

Run tco_rag.py and copy the three monthly numbers into a comparison table with two extra rows you compute by hand: cost per request and cost per 1M tokens served (a unit-economics number finance can benchmark). Sanity-check the shape, not just the values: provisioned throughput bills for capacity whether used or not, so at modest utilization it should look expensive; self-hosted carries the GPU 24×7 plus the ops burden the script deliberately omits — add a memo line acknowledging the unmodeled engineer-time cost.

✅ Checkpoint: comparison table with monthly, per-request, and per-1M-token rows for all three options.

Step 4 — Sensitivity: context length and cache hit rate (20 min)

Re-run the model across a grid: input_tokens ∈ {2,000, 6,000, 12,000} × cache_hit_rate ∈ {0.0, 0.4, 0.7} (a small loop over W, or nine spreadsheet cells). Record the hosted-API cost for each cell and note where the ranking of the three options changes, not just the totals. Two findings should emerge from the arithmetic: retrieval design (how much context you stuff) is a first-order cost decision, and cache hit rate behaves like a discount on your dominant cost driver — which makes the caching architecture worth designing before scale, not after.

✅ Checkpoint: a 3×3 sensitivity table with the option ranking marked in every cell.

Step 5 — The recommendation memo with breakpoints (20 min)

Write memo.md, one page, CFO register: recommendation for Orbit's GA volume, and — the part most memos omit — the breakpoints: at what requests/day does provisioned throughput's utilization math beat pay-per-token (solve for the crossover using your verified prices)? At what sustained token volume does self-hosted break even before counting ops headcount, and what does adding a realistic ops cost do to that threshold? Close with a review trigger: "re-run this model when prices change or volume passes X" — pricing moves often enough that a TCO model has a shelf life, and saying so in the memo is what makes it credible.

✅ Checkpoint: memo.md with a recommendation, two computed breakpoints, the ops-cost caveat, and a review trigger.

Verification

• No unverified numbersall 7 prices carry source URL + date; the script's assertion passed only after real values went in.
• Audit performedthe AI-quoted vs calculator-verified price table exists, with deltas noted.
• Sensitivity realthe 3×3 grid shows at least one cell where the cheapest option differs from the base case (if not, your grid is too narrow — widen it and say so).
• Breakpoints computedcrossovers derived from your prices, shown with the arithmetic, not asserted.

Completion artifact

The rag-tco/ folder: tco_rag.py (filled), prices.md, the comparison and sensitivity tables, memo.md. Self-grading rubric (pass = 7/9):

• Price hygiene (0–3)sourced, dated, audited against AI recall = 3.
• Model correctness (0–2)per-request and per-1M-token unit costs computed correctly = 2.
• Sensitivity insight (0–2)ranking changes identified and explained in one line each = 2.
• Memo quality (0–2)breakpoints with arithmetic, ops caveat, review trigger = 2.

Client-safe variant

Cost models rarely contain client-confidential data — the workload numbers are the sensitive part. Under a strict MSA: keep real volumes in a local spreadsheet, run the AI-assisted structuring with rounded or indexed volumes (e.g., "N requests/day"), and never paste client contract pricing (negotiated discounts are confidential even when list prices are public). The memo format is a client deliverable; your prices.md evidence file is what makes it defensible in a commercial review.

Stuck?

• •
  The assertion won't pass: at least one None remains — gpu_tokens_per_sec is the one everyone forgets to source.
• •
  Provisioned throughput looks absurdly expensive: that is usually correct at 40k requests/day — low utilization is the finding, not a bug; state it in the memo.
• •
  You can't find a serving benchmark: pick a model/GPU pair that has published numbers rather than guessing for your preferred pair, and note the substitution.
• •
  The three options land within noise of each other: recompute at 10x volume — the ranking spreads, and that spread is your breakpoint section.