"The RAG bot is hallucinating" is a symptom, not a diagnosis. A RAG failure lives in one of three places — the retriever, the generator, or the contract between them — and each needs a different fix.
Tier: Core · Workflow area: Evaluation Engineering
Ragas (named verbatim in DS/ML job postings) gives you component-attributed metrics: faithfulness, answer relevancy, context precision, context recall. This playbook runs the four metrics on your golden dataset, teaches you to read each score against the right pipeline component, and goes one level deeper with claim-level verification — decomposing answers into atomic claims and checking support for each, which is how you measure hallucination instead of arguing about it.
Skip metric selection debates until pb-34's error analysis is done — the taxonomy tells you which of these metrics matters most for your failure Pareto.
question, retrieved_contexts (captured at runtime, not re-retrieved later), answer, and where available ground_truth / reference facts.ragas installed; an LLM for metric computation (a cheap-tier model handles it; budget ≤ $2 for ~50 cases) and optionally Phoenix or LangSmith for trace-linked results.Map the four metrics to components before running anything. Write this table into your report skeleton: faithfulness (is the answer supported by retrieved context?) → generator problem when low; answer relevancy (does the answer address the question?) → generator/prompt problem; context precision (are relevant chunks ranked above irrelevant ones?) → retriever/reranker problem; context recall (did retrieval fetch what the ground truth needs?) → retriever/chunking/embedder problem. Reading scores without this mapping is how teams "fix" the wrong component for a sprint.
Run the Ragas evaluation.
from ragas import evaluate
from ragas.metrics import (faithfulness, answer_relevancy,
context_precision, context_recall)
from datasets import Dataset
ds = Dataset.from_list(cases) # question, contexts, answer, ground_truth
report = evaluate(ds, metrics=[faithfulness, answer_relevancy,
context_precision, context_recall])
Pin the judge-LLM model and version in config — metric values are not comparable across judge models, so the pin is part of the metric definition.
Interpret with thresholds and the diagnostic grid. Working thresholds to start (tune to your domain): faithfulness < 0.70 → active hallucination investigation; context recall < 0.75 → retrieval is starving the generator. Diagnose combinations: low faithfulness + high recall = generator ignoring or overriding good context (prompt/model fix); low recall + high faithfulness = model faithfully answering from insufficient context — correct behavior, starved inputs (retrieval fix); low precision + high recall = right chunks buried in noise (reranker fix); low relevancy + high faithfulness = grounded but off-target (question-handling/prompt fix).
Drill into hallucination at claim level. Averages hide severity — one fabricated contract clause in an otherwise faithful answer can be the whole incident. For the 10–20 lowest-faithfulness cases: (a) prompt an LLM to decompose the answer into atomic factual claims; (b) for each claim, check support against the retrieved contexts — verdicts supported (quote the span), contradicted, or unsupported; (c) human-verify verdicts on 3–5 cases. Output per case: claim table with verdicts and supporting spans. Unsupported-claim rate (claims unsupported / total claims) is your precise hallucination number; the span attribution is what makes it reviewable.
Slice, then read traces. Break all metrics by pb-34's slice tags (intent, document type, question complexity). Then read the 5 worst traces per weak slice — the metric found where; the trace tells you what. Every eval run ends in trace reading or it ends in guessing.
Fix the indicated component and re-run. One change at a time (the pb-31 discipline): fix, re-run the identical dataset and judge pin, report deltas on all four metrics — a reranker that lifts precision can dent recall; the four-metric panel catches the trade.
Institutionalize. Commit the eval script, dataset version, judge pin, and thresholds; this exact bundle becomes the pb-37 CI regression gate, and the claim-level check becomes the escalation path when the gate trips on faithfulness.
Faithfulness low and context recall high means the generator is the culprit; both low means retrieval starved it — the entire value of Ragas is attribution, and attribution is what turns "it hallucinates" into a fixable ticket.
The judge model pin is part of your metric's identity — a self-hosted judge swap mid-quarter silently redefines every threshold; version it like a schema change.