Build the three-way TCO model — RAG, fine-tune, prompt-only — for a product FAQ copilot with monthly content churn, 200K queries a month, and India data-residency constraints, then write the two-page recommendation memo with revisit triggers. The provided pricing sheet plants one stale price that flips the recommendation if used unverified.
Priya — the Chennai invoice-workflow PM — has her "add AI" mandate: a copilot answering customer questions about invoices, approvals, and policies from the company's help content. A vendor is pushing managed fine-tuning; her engineers mention RAG; someone asks why not just stuff the FAQ into the prompt. The honest answer is arithmetic plus constraints — and the arithmetic is only as good as the prices in it. One number in the pack below is stale enough to flip the whole decision, and finding it is the graded step.
SCENARIO CONSTRAINTS
- FAQ copilot over ~400 help articles; content churn: ~15% of
articles change EVERY MONTH (policy + pricing updates).
- Volume: 200,000 queries/month, growing ~10%/quarter.
- Latency budget: p95 under 4 seconds.
- Data residency: customer data must be processed in India region.
- Team: 2 engineers part-time; no ML-ops experience.
- Per-query token estimates (all approaches): ~300 output tokens.
Prompt-only (entire FAQ in context): ~12,000 input tokens/query.
RAG (top-4 chunks retrieved): ~2,000 input tokens/query.
Fine-tuned (short prompt): ~500 input tokens/query.
PRICING PACK - "compiled by the vendor, January 2025"
L1. Frontier-tier API model: $15.00 / 1M input, $75.00 / 1M output
L2. Mid-tier API model: $3.00 / 1M input, $15.00 / 1M output
L3. Small-tier API model: $5.00 / 1M input, $0.60 / 1M output
L4. Embeddings: $0.10 / 1M tokens
L5. Managed vector DB starter: $70 / month
L6. TuneServe managed fine-tune bundle: $1,800 / month flat
(hosting + one retrain per quarter; extra retrains $900 each;
hosted in TuneServe US-East region)
L7. India-region API hosting: available from major providers for
L1-L3 class models at standard API prices, per their pricing pages
Sanity-read the pricing pack before modeling. Two minutes, no spreadsheet: does anything look internally inconsistent? Rank the three model tiers by input price as listed. Note anything that violates how model pricing normally works, and keep the note — you will formally verify in step 4.
Build the TCO skeleton. One column per approach (prompt-only, RAG, fine-tune), rows for: monthly input token cost, output token cost, infrastructure (vector DB, embeddings refresh), retraining/content-update cost, and engineering-effort notes. Compute input tokens/month first: queries × tokens/query (prompt-only: 200K × 12K = 2.4B; RAG: 400M; fine-tune: 100M). Getting the token arithmetic right before pricing is half the model.
Fill in costs using the pack as given. Use the small tier (L3) for prompt-only and RAG (an FAQ bot rarely needs frontier), and L6 for fine-tune with monthly retrains — the scenario says 15% of content changes every month, so a quarterly-retrain bundle needs 8 extra retrains a year. Record the ranking this produces and the winner it implies. Write it down explicitly: "Using the vendor pack unverified, the cheapest viable option is ___."
Verify every API price against live pricing pages. Open the current pricing pages of at least two major model providers (OpenAI, Anthropic, or Google) and record today's small-tier and mid-tier prices with URLs and dates in a mini-ledger. Compare with L1–L4. One line in the pack is not just stale but wildly off current reality — identify it, quote the live number you found, and note the direction and rough magnitude of the error.
Re-run the model with verified prices. Correct the stale line and recompute all three columns. Compare the before/after ranking — the graded observation is whether the winner changed and precisely which line item flipped it. Then add the churn stress: what does fine-tune cost look like with 12 retrains/year versus RAG's near-zero content-update cost (re-embed changed articles: trivial at L4 prices)?
Apply the non-cost constraints as gates, not tiebreakers. Data residency: L6 is hosted US-East only — does the fine-tune option even clear the India-region requirement as quoted? Latency: all three approaches clear 4s p95 comfortably at these token counts. Team: no ML-ops experience is a standing cost against self-managed fine-tuning. Mark each approach PASS/FAIL/CONDITIONAL per constraint before any cost comparison is allowed to decide.
Write the two-page memo. Structure: recommendation up front with the monthly number; the constraint-gate table; the TCO table (verified prices, with your price ledger as an appendix line); why churn kills fine-tune for this workload regardless of price; what would change the answer. Include the honest disclosure: "the vendor pack's L3 price was stale by roughly an order of magnitude; unverified, it flips the recommendation to the vendor's bundle."
Write the revisit triggers. Five concrete tripwires, each with a threshold and an owner: query volume passes 1M/month (token economics shift); content churn drops to quarterly (fine-tune re-enters); an India-region managed fine-tune under $X/month appears; small-tier prices move ±50%; retrieval quality on the eval set drops below target (architecture, not price, becomes the problem). A decision memo without revisit triggers is a decision that silently goes stale — exactly like the price that almost fooled you.
Model prices change faster than any document that quotes them — every AI cost model needs a verified-on date per price, and any memo older than a quarter is due for re-verification before reuse.
A complete lab has all of the following:
Architecture decisions for AI features are TCO models gated by constraints — and the model is only as honest as its freshest price. Verify every number against a live page, apply residency and churn as gates before cost, and ship the memo with revisit triggers so it expires gracefully instead of silently.