Choose the Adaptation Path: Prompt, RAG, Fine-Tune, or Distill

Tier: Core · Workflow area: Fine-tuning & Model Adaptation

This playbook is the decision gate you run before anyone spins up a GPU: a cheap-to-expensive ladder with an eval checkpoint at every rung, ending in a one-page cost-model memo that survives review.

When to use

• •
  A stakeholder or PM says "we should fine-tune a model" and you need to confirm or redirect that instinct with evidence.
• •
  An LLM feature underperforms and you must choose the next investment: better prompts, retrieval, tuning, or a distilled small model.
• •
  You are asked in a design review (or an interview) to defend RAG-vs-fine-tune trade-offs with numbers, not vibes.

Skip the full ladder when the failure is obviously knowledge freshness (yesterday's prices, this week's tickets) — that is retrieval by definition; go straight to RAG.

Prerequisites

• •
  A concrete task with 30–50 representative inputs and expected outputs (even rough ones) — you cannot choose an adaptation path for a task you cannot evaluate.
• •
  Access to one strong API model and one small open model (Groq/OpenRouter free tiers are enough).
• •
  A minimal eval script: exact-match or rubric scoring over your 30–50 cases (see pb-34 and pb-37 for building real harnesses).
• •
  45–60 minutes plus API spend under $2.

The workflow

1. 1.

   Classify the failure before touching any solution. For 20 failing examples, label each failure as one of: F1 knowledge (model lacks facts), F2 format/style (right content, wrong shape or tone), F3 reasoning/procedure (wrong steps), F4 latency/cost (works, too slow or expensive). Tally the labels — the majority class picks your ladder entry point.

2. 2.

   Rung 1 — prompt engineering (hours, ~$0). Rewrite the system prompt with explicit rules, add 3–5 few-shot examples covering your failure modes, and require structured output. Re-run the 30–50-case eval. In practice this closes most F2 and many F3 failures.

3. 3.

   Rung 2 — RAG (days, infra cost only). If F1 dominates, wire retrieval over the source documents and re-evaluate. Grade with faithfulness in mind: the fix must cite retrieved context, not memory. Do not proceed past this rung for knowledge failures — fine-tuning bakes in facts that go stale and cannot be updated per-document.

4. 4.

   Rung 3 — fine-tune (weeks, GPU + data-curation cost). Justified only when the eval still fails on F2/F3 after rungs 1–2 and you have or can curate ~1,000 quality examples (pb-28). Typical wins: rigid output schemas, house style, domain-specific procedures, small-model capability lift. Estimate cost as: curation hours × loaded rate + GPU hours + re-eval time (pb-33).

5. 5.

   Rung 4 — distill (after a working expensive solution). If the frontier-model solution passes evals but F4 (cost/latency) fails, distill it into a small model (pb-32). Distillation is a cost optimization on a proven pipeline, never a first move.

6. 6.

   Write the one-page memo. Sections: task and eval metric; failure-mode tally; score at each rung attempted; chosen path; monthly cost at 10x current volume for the top two candidate paths; revisit trigger (e.g., "re-run this decision if token prices drop 50% or volume grows 5x").

Tool variants

• API-first stack: run rungs 1–2 entirely on Anthropic/OpenAI/Gemini APIs with LiteLLM for uniform calls; free Gemini tier covers the eval volume.
• Open-weights stack: use a 7–8B model via Groq/OpenRouter free tiers as the "small model" baseline, so rung-3/4 projections use realistic small-model scores.
• Spreadsheet cost model: a simple sheet with rows = rungs, columns = build cost, monthly run cost at 1x/10x volume, eval score. This is the artifact reviewers actually read.

Verification

• •
  Failure-mode tally exists for at least 20 real failures, with the input text linked or pasted for each.
• •
  Each rung you attempted has a before/after score on the same fixed eval set — same cases, same grader.
• •
  The memo states monthly cost at 10x volume for at least two paths, with the arithmetic shown.
• •
  A colleague can reproduce your recommendation from the memo alone, without asking you anything.

Pitfalls

• Fine-tuning to inject facts. The model will sound more confident about your domain while still hallucinating specifics; knowledge belongs in retrieval where it can be updated and cited.
• Skipping rung 1 because it feels too simple. Serious prompt work with few-shot examples is the highest-ROI rung and the baseline every later rung must beat.
• Evaluating each rung on different examples. Cherry-picked demos make every technique look good; the fixed 30–50-case set is non-negotiable.
• Costing only GPU time. Data curation and honest re-evaluation dominate real fine-tune cost; a memo that omits them undercounts by 3–5x.
• Treating the decision as permanent. Model prices and capabilities shift quarterly in 2026; write the revisit trigger into the memo.

Restricted-environment variant

For client or regulated data:

• •
  Run rung comparisons with anonymized or synthetic look-alike examples first; only the winning path gets evaluated on real data inside the approved environment.
• •
  If client contracts bar external APIs, restrict the ladder to self-hosted open models (Ollama locally, vLLM on approved infra) — the ladder logic is unchanged, only the model pool shrinks.
• •
  Note in the memo which rungs were evaluated on synthetic data, so nobody mistakes a proxy score for a production score.

Success criteria

• •
  You produced a defensible adapt-or-not recommendation in under a day of effort and under $2 of API spend.
• •
  The recommendation is backed by scores on a fixed eval set at every rung attempted.
• •
  At least one expensive path (fine-tune or distill) was explicitly rejected or deferred with a cost figure attached.
• •
  Six weeks later, the revisit trigger in the memo still tells the team exactly when to re-open the decision.