Decide RAG vs Fine-Tune vs Prompt-Only with TCO Math

Workflow area: AI Product Economics · Tier: Advanced · Hands-on time: ~45 minutes

The architecture choice for grounding an AI feature is a product decision wearing an engineering costume. Fine-tuning gets proposed because it sounds serious; RAG gets proposed because it's fashionable; prompt-only gets dismissed because it sounds lazy — and all three reflexes are wrong in different situations. The variables that actually decide it are ones the PM owns: how fast the knowledge changes, how wrong the feature can afford to be, whose data it is, and what the cost curve looks like at real volume. Your job is to bring those variables to the table with numbers.

When to use — and when not to

• Use it when the feature depends on knowledge the base model lacksyour docs, your catalog, your policies, your customers' data.
• Use it when someone proposes fine-tuningthe memo either justifies the investment or saves a quarter of engineering time; both outcomes pay for the 45 minutes.
• Use it at renegotiation pointsvolume 10×'d, a model got cheaper, retrieval got worse; the decision has a shelf life and this playbook is the re-run.
• Skip it when prompt-only obviously sufficesif the knowledge fits in a few thousand tokens, changes rarely, and isn't customer-specific, put it in the system prompt and move on; write one paragraph saying why.
• Skip the memo when engineering has already prototyped both paths with evalsthen your job is auditing their numbers, not producing parallel ones.

The recipe

1. 1.

   Score the four decision variables first, before any costs. Fill this table honestly: knowledge volatility (changes hourly? monthly? never?), update cadence required (how stale can an answer be before it harms?), per-answer stakes (does a wrong answer cost a ticket or a lawsuit?), data rights (can this data legally be used to train a model, and can one customer's data ever influence another's answers?). The last one is a hard constraint, not a score: customer-confidential data generally rules fine-tuning on pooled data out immediately.

2. 2.

   Get the decision logic challenged by AI before the math. Paste into Claude:

   Copy

   Act as a skeptical staff engineer reviewing an architecture choice
   for grounding an AI feature.
   Feature: [WHAT IT ANSWERS/GENERATES, FOR WHOM]
   Knowledge source: [WHAT IT IS, SIZE, WHO OWNS IT]
   Volatility: [HOW OFTEN IT CHANGES + HOW FAST CHANGES MUST APPEAR]
   Stakes: [COST OF A WRONG/STALE ANSWER]
   Data rights: [TRAINING PERMITTED? CROSS-CUSTOMER ISOLATION NEEDED?]
   Volume: [REQUESTS/MONTH NOW AND AT 12-MONTH PROJECTION]
   
   For each of prompt-only, RAG, and fine-tune:
   1. Whether it's viable AT ALL given the constraints (data rights
      and staleness can eliminate options — say so).
   2. What it's best at here, and its dominant failure mode.
   3. The hidden costs PMs miss (RAG: chunking quality, index
      refresh pipeline, retrieval eval; fine-tune: data prep,
      re-training on every knowledge change AND base-model
      deprecation; prompt-only: context-window cost at volume).
   Recommend one, state the strongest argument AGAINST your
   recommendation, and name what evidence would change it.
   

   The "strongest argument against" line is the part to read twice — it's your memo's risks section.

3. 3.

   Build the TCO sheet at three volumes. One Google Sheet, three columns (prompt-only, RAG, fine-tune), rows for: per-request inference cost (tokens × current provider prices — pull them from the pricing pages today, they move), retrieval infra (vector store, embedding refresh) for RAG, training + hosting + re-training-per-update for fine-tune, and engineering time to build and to maintain (the row everyone omits). Compute at current volume, 5×, and 25×. Prompt-only's long-context costs scale linearly with volume and often cross RAG's curve embarrassingly early — the sheet finds that crossover.

4. 4.

   Audit the sheet yourself. Recompute two cells by hand: one inference cost (tokens per request × price per million — check the tokens estimate against a real logged request, not a guess) and the fine-tune re-training cadence (volatility says monthly? then 12 trainings/year, not the one-time cost the proposal quoted). If you can't defend a cell to a staff engineer, the memo isn't ready.

5. 5.

   Write the one-page memo. Structure: decision in one sentence → the four variables and which one dominated → TCO table with the crossover volume → what would reverse this decision (a price drop, a volume threshold, retrieval quality below X on the eval set) → review date. The reversal triggers are what make it a decision memo instead of an opinion.

Make it yours

• Hybrid check: the honest answer is often staged — prompt-only to launch, RAG when the knowledge outgrows the context window, fine-tune only for tone/format (not knowledge) if ever. Add a "sequencing" section proposing the staged path with its trigger points.
• Regulated-data variant: when the source data is regulated (health, financial), add a compliance row to the sheet — RAG with per-tenant indexes and audit logging usually wins on compliance cost alone; get the memo reviewed by whoever owns data protection.
• Eval-first variant: if you have a golden set (pb-40), prototype prompt-only and RAG in an afternoon and put actual eval scores in the memo — a measured 4-point accuracy gap beats every theoretical argument in the room.

Quality check

• Data rights were checked firstan architecture that's illegal for your data isn't cheap at any volume.
• Fine-tune costs include re-training cadenceone-time training cost quoted for volatile knowledge is the classic error; the memo multiplies by updates per year.
• The crossover volume is stated"RAG becomes cheaper than prompt-only above ~N requests/month" is the sentence that survives the meeting.
• Maintenance engineering is a line itemindex refresh pipelines and eval upkeep are recurring costs, not launch costs.
• Reversal triggers are numericthe memo names the price, volume, or eval score that reopens the decision, and a review date exists.