Accept a RAG Feature — and Blame the Right Layer When It Fails

Tier: Advanced · Workflow area: Testing AI systems — judges, agents & robustness

You will run a RAG feature against an eval set, compute retrieval metrics and generation metrics separately, accept or reject against written thresholds, and file failures with a layer attribution the receiving engineer can act on immediately.

When to use

Prerequisites

• •
  An eval set of 30+ items: question, reference answer, and ideally the doc IDs that should be retrieved.
• •
  Tracing that captures the retrieved contexts per query — Langfuse instrumentation or your pipeline's own logging. If you cannot see what was retrieved, stop and add that first.
• •
  DeepEval (with its Ragas-style metrics) or Ragas installed; an LLM API key for the metric judges.

The workflow

1. 1.

   Instrument so every answer carries its contexts. Each eval run must record, per question: the retrieved chunks (with scores), the final prompt, and the answer. In Langfuse this is a retrieval span nested in the generation trace; in a bare pipeline, a JSONL with question, contexts[], answer, reference is enough.

2. 2.

   Pick the four metrics and what each blames. Write this table into the test plan — it is the routing logic:

   Metric	Question it answers	Low score blames
   Contextual recall	Did retrieval fetch the info the reference answer needs?	Retrieval
   Contextual precision	Are relevant chunks ranked above irrelevant ones?	Retrieval (ranking)
   Faithfulness	Is every claim in the answer supported by the contexts?	Generation
   Answer relevancy	Does the answer address the question asked?	Generation (prompt)

3. 3.

   Encode the run as pytest with DeepEval.

   Copy

   from deepeval import assert_test
   from deepeval.test_case import LLMTestCase
   from deepeval.metrics import (
       ContextualRecallMetric, ContextualPrecisionMetric,
       FaithfulnessMetric, AnswerRelevancyMetric)
   
   def test_rag_case(eval_item):
       tc = LLMTestCase(
           input=eval_item.question,
           actual_output=eval_item.answer,
           expected_output=eval_item.reference,
           retrieval_context=eval_item.contexts)
       assert_test(tc, [
           ContextualRecallMetric(threshold=0.8),
           ContextualPrecisionMetric(threshold=0.7),
           FaithfulnessMetric(threshold=0.85),
           AnswerRelevancyMetric(threshold=0.8)])
   

   Copy

   deepeval test run test_rag_acceptance.py
   

   Thresholds are yours to set from a baseline run, not copied from a tutorial — run once, look at the score distribution, put the bar where your known-good cases sit.

4. 4.

   Attribute each failure before filing anything. Use the decision order: recall low → retrieval bug (chunking, embedding model, index freshness, query rewriting). Recall fine but precision low → ranking bug (reranker, top-k). Contexts fine but faithfulness low → generation bug (the model invents or contradicts; tighten the prompt's grounding instructions). Everything high but relevancy low → the prompt answers a different question than asked. When metrics disagree with your reading, open the trace and check the contexts by eye — the metrics are judges, and judges have error bars.

5. 5.

   File layer-attributed bugs. The report template that makes this playbook pay off:

   Copy

   Title: RAG failure — retrieval recall — refund-policy questions
   Layer: RETRIEVAL (contextual recall 0.31 on 6/8 refund questions)
   Evidence: reference answer requires §4.2 of refund-policy.md;
     retrieved top-5 contains only marketing FAQ chunks.
     Trace: langfuse.example.com/trace/tr_8842
   Not a generation bug: faithfulness 0.94 — the model faithfully
     answered from the wrong documents.
   Suggested owner: search/indexing team.
   

6. 6.

   Gate the release on the aggregate. A workable acceptance line: ≥90% of eval items pass all four thresholds, zero faithfulness failures on the compliance-sensitive slice, and no metric's mean regressed more than 5 points vs the last accepted run. Wire the same pytest suite into CI so retriever and prompt changes both trigger it.

Variations

• Ragas directly: same four metrics natively, better for batch offline scoring of large trace exports; DeepEval wins when you want pytest semantics and JUnit XML in existing CI.
• Langfuse-native: attach the metrics as evaluators over sampled production traces for continuous acceptance — the offline eval set catches regressions pre-merge, the trace evaluators catch drift post-deploy.
• No reference answers yet: faithfulness and relevancy still work (they need only question, contexts, answer); run recall/precision on the subset you have labeled and grow the set from production traces.

Quality check

• Contexts captured per queryyou can open any failure and see exactly what was retrieved.
• Thresholds set from your own baseline run, written down with the date and dataset version.
• Every failure carries a layer attribution and the metric evidence for it.
• Compliance-sensitive slice (refunds, medical, legal — whatever yours is) held to a stricter faithfulness bar.
• Metric-judge spot-check done: you manually verified the metric's verdict on at least five failures.

Pitfalls

• Judging the answer only. End-to-end "answer quality" scores hide which layer broke — you will tune prompts for weeks to fix an indexing bug.
• Eval set from the docs, not the users. Questions generated from your own documents flatter recall; harvest real user phrasings from traces.
• Stale-index blindness. Recall failures after a docs update are freshness bugs, not chunking bugs — record index build time with every run.
• Treating metric scores as ground truth. These metrics use LLM judges internally; spot-check them like any judge (see the judge-calibration playbook).