Every RAG pipeline decision a data engineer makes — chunk size, chunking strategy, embedding model, hybrid search, reranking — changes retrieval quality, and almost every team makes those decisions by vibes.
Tier: Core · Workflow area: Data for fine-tuning & evaluation
RAGAS turns them into measurements: build a golden retrieval set once, compute context precision and context recall (the retrieval half — the half the DE owns), and run the same harness across every pipeline variant so each choice is evidence-based. Indian job postings now name "documented RAGAS scores" verbatim; this playbook produces exactly that artifact, plus the CI gate that keeps scores from regressing.
Assemble a golden set of 50+ question–ground-truth pairs. Three sources, in order of value: real user questions with human-verified answers and the corpus passages that support them; questions hand-written by domain experts against specific documents; and RAGAS's synthetic testset generator to fill coverage gaps (gate its output — see the synthetic-data playbook). Each item: question, ground_truth answer, and reference contexts. Stratify across document types and difficulty; freeze it in version control alongside the pipeline code.
Instrument your pipeline to emit the evaluation trace. For each golden question, capture what the retriever returned before generation: question, retrieved_contexts (the actual chunks), answer (if evaluating end-to-end), ground_truth. This trace shape is RAGAS's input — and getting your pipeline to produce it cleanly is most of the integration work.
Score the retrieval metrics. Context precision (are retrieved chunks relevant — how much noise did you pull?) and context recall (did retrieval capture what's needed to answer — is the answer's support actually in the chunks?):
from ragas import evaluate
from ragas.metrics import context_precision, context_recall
from datasets import Dataset
ds = Dataset.from_dict({
"question": questions,
"contexts": retrieved_contexts, # list[list[str]] per question
"ground_truth": ground_truths,
})
report = evaluate(ds, metrics=[context_precision, context_recall])
print(report) # e.g. {'context_precision': 0.81, 'context_recall': 0.74}
Expected output: per-metric aggregates plus per-question rows. The per-question view is the debugging tool — sort ascending and read the worst ten retrievals; the failure pattern (wrong document, right document/wrong chunk, chunk truncated mid-answer) tells you which knob to turn.
Add the generation-side metrics for the full picture. Faithfulness (is the answer grounded in retrieved context?) and answer relevancy round out the standard four. As the DE, your contract is the retrieval pair — but reporting all four lets you demonstrate whether a bad answer is a retrieval problem (low recall) or a generation problem (low faithfulness on high recall), which ends the cross-team blame loop.
Run the same harness across variants — this is the whole point. Parameterize the pipeline (chunking strategy, chunk size, embedding model, dense vs hybrid vs hybrid+rerank, top-k) and evaluate each variant against the identical golden set:
for cfg in [fixed_512, recursive_512, structure_aware, parent_child]:
contexts = run_retrieval(cfg, questions)
scores[cfg.name] = evaluate(build_ds(contexts), metrics=[...])
Expected artifact: a decision table — variant × context precision/recall × latency × cost per query. "Structure-aware chunking: +0.09 recall over fixed-512 at equal precision" is a decision memo; "structure-aware felt better" is not.
Wire a CI regression gate. On every change to the ingestion/retrieval path, CI re-runs the harness and fails the build if context recall or precision drops more than an agreed tolerance (e.g., 2 points) against the stored baseline. Score history is logged per commit — retrieval quality becomes a tested property of the codebase, exactly like unit tests.
Keep the golden set alive. New document types, new user question shapes, and every production retrieval failure worth a postmortem become new golden items. Version the set; when it changes, re-baseline and note it — score movements must be attributable to either the pipeline or the set, never ambiguously both.
Know the four core RAGAS metrics and their split: context precision and context recall measure retrieval (the data engineer's half); faithfulness and answer relevancy measure generation. Low recall means fix chunking/embedding/search; low faithfulness with high recall means the problem isn't yours.
The golden set contains real client questions, answers, and document passages — it is client-confidential and lives in their repo, full stop. RAGAS's judge metrics send questions, contexts, and answers to an LLM, so the evaluation itself needs the same provider approval as the RAG system being measured; where that's restricted, configure RAGAS to use a client-approved endpoint or a self-hosted judge model via vLLM (scores shift slightly — re-baseline, and never mix judge models within one comparison). Run the harness and CI gate in client infrastructure, and deliver the decision table and score history as the engagement artifact: a client who can show documented RAGAS scores for every pipeline decision has inherited the discipline, not just the pipeline.