Measured RAG Baseline — and Whether You Need RAG at All

Tier: Core · Workflow area: RAG & Retrieval Engineering

Every later retrieval upgrade (pb-22 through pb-26) is judged against this baseline.

When to use

• •
  Starting any grounded-QA/assistant project over an internal corpus (docs, wikis, reports, tickets).
• •
  An existing RAG system has no metrics and "feels off" — rebuild the measurement layer under it.
• •
  Someone asks "with 1M-token context windows, why do we need RAG?" — this playbook produces the answer for your corpus, with numbers.

Prerequisites

• •
  A corpus (100–10,000 documents) and realistic user questions — collect 30–50 from actual stakeholders; invented questions produce invented conclusions.
• •
  langchain or llama-index (either), pgvector/qdrant/faiss (all free/self-hostable), ragas, sentence-transformers.
• •
  Token counts of your corpus (count_tokens over a sample × doc count) — needed for the stuffing-vs-RAG math.

The workflow

1. 1.

   Decision gate: long-context vs retrieval. Compute three numbers: corpus tokens, expected queries/month, freshness cadence. Then run the comparison empirically on 20 questions:

   • •
     Stuffing: whole corpus (or the relevant subset) in context with prompt caching on the static prefix.
   • •
     RAG: the baseline pipeline below.

   Rules of thumb the numbers usually confirm: corpora under ~200k tokens with low query volume → stuffing + caching wins on simplicity and often accuracy; large corpora, high QPS, per-user ACLs, or minute-level freshness → retrieval wins on cost and latency. Cache-read pricing (~0.1×) changes the arithmetic — model it, don't assume. Record the decision and the numbers; revisit quarterly.

2. 2.

   Build the eval set before the pipeline. For each of the 30–50 questions, record: the answer (or key facts), and the source passages that support it (doc ID + span). This grounding set powers retrieval metrics forever after.

3. 3.

   Assemble the naive baseline. Deliberately boring: recursive character splitting at ~450 tokens with 15% overlap (the strong 2026 default — pb-22 tunes this properly), bge-small-en-v1.5 embeddings, cosine top-5, answer with a cited-generation prompt. Resist every upgrade urge; upgrades come later, measured.

4. 4.

   Instrument retrieval separately from generation. Log, per query: retrieved chunk IDs + scores, and compute against the grounding set:

   Copy

   # recall@k: did we retrieve any gold passage?
   recall_at_k = mean(any(c in gold[q] for c in retrieved[q][:k]) for q in questions)
   # MRR: how high did the first gold passage rank?
   

   Retrieval recall@5 is the ceiling on your whole system — if the passage isn't retrieved, no generation model can save you.

5. 5.

   Score end-to-end with RAGAS. Component metrics tell you where it breaks:

   Copy

   from ragas import evaluate
   from ragas.metrics import faithfulness, answer_relevancy, context_precision, context_recall
   from datasets import Dataset
   ds = Dataset.from_dict({"question": qs, "answer": answers,
                           "contexts": contexts, "ground_truth": truths})
   report = evaluate(ds, metrics=[faithfulness, answer_relevancy,
                                  context_precision, context_recall])
   

   Use a cheap-tier model as the RAGAS judge and pin its version. Read the four numbers as a diagnosis: low context_recall → retrieval problem (pb-22/23/24); high recall + low faithfulness → generation/grounding problem; low context_precision → you're retrieving noise.

6. 6.

   Freeze the baseline scorecard. One table: recall@5, MRR, the four RAGAS metrics, p95 latency, cost/query. This is the number every subsequent change must beat — commit it next to the code.

Tool variants

• Vector store: pgvector if you already run Postgres (one less system), Qdrant OSS for payload filtering you'll want in pb-25, FAISS for pure-local experiments. No paid vector DB is required at any point in this track.
• Framework: LlamaIndex is strongest for ingestion/index abstractions, LangChain for chain composition; the playbook is framework-agnostic — the metrics layer is yours either way.
• Long-context arm: any 1M-context flagship (e.g. claude-opus-4-8) with cache_control on the corpus prefix; verify cache hits via usage.cache_read_input_tokens before trusting the cost model.
• RAGAS alternatives: DeepEval, or hand-rolled judge prompts via pb-20's harness — the metric definitions matter more than the library.

Verification

• •
  Grounding set covers ≥30 real questions with passage-level provenance; at least 5 are "hard negatives" (plausible questions the corpus cannot answer — the system should say so).
• •
  Baseline scorecard computed and committed; rerunning it reproduces scores within noise.
• •
  The stuffing-vs-RAG decision is written down with corpus size, query volume, and measured accuracy/cost for both arms.
• •
  Unanswerable questions: the baseline declines (or flags low confidence) on ≥3 of the 5 hard negatives rather than hallucinating.

Pitfalls

• Synthetic-only eval questions. LLM-generated questions from your own chunks inflate recall (they mirror chunk vocabulary). Fine for smoke tests; decisions need real user questions.
• Judging end-to-end only. An answer score without retrieval metrics cannot distinguish "retriever missed" from "generator ignored" — you will tune the wrong component.
• Upgrading before baselining. Hybrid search + reranking + query rewriting installed on day one leaves you with no idea which of them pays rent. Naive first, measured upgrades after.
• Ignoring the freshness axis in the stuffing decision. Cached long-context is repriced every time the corpus changes; a daily-changing corpus quietly destroys the caching economics.
• Judge model drift in RAGAS. Same rule as pb-20: pin the judge, re-baseline when it changes.

Restricted-environment variant

• •
  Full-local stack: FAISS/pgvector + bge embeddings + a local 7–8B generator, with RAGAS pointed at a local judge — zero external calls, standard for client-data engagements.
• •
  The long-context arm shrinks locally (8–32k practical contexts on small models); note that this changes the decision gate's answer on-prem — retrieval wins earlier. Record both hosted and local decisions.
• •
  Lab budget: hosted arm on a free-tier/cheap-tier model with a public corpus stays under ₹250 (~$2–3), the track's per-lab design constraint.

Success criteria

• •
  A running baseline RAG pipeline whose every query logs retrieved IDs, scores, and metric-ready traces.
• •
  A frozen scorecard (retrieval + RAGAS + latency + cost) that later playbooks must beat to justify their complexity.
• •
  A written, numbers-backed answer to "why RAG (or why not) for this corpus."
• •
  Team agreement that no retrieval change merges without moving this scorecard — the cultural deliverable.