Two disciplines keep a shipped RAG system trustworthy: a trace-driven triage method that turns "the bot gave a wrong answer" into a specific component diagnosis (retrieval-miss vs grounding-miss vs generation-miss), and an ingestion pipeline that keeps the index true to a living corpus — incremental upserts, dedup, document versioning, and drift monitoring.
Tier: Advanced · Workflow area: RAG Operations & Observability
This playbook installs both; they meet in the middle, because a large share of "model problems" are actually stale-index problems.
Adopt the triage taxonomy. Every bad answer gets exactly one primary label, assigned by walking the trace top-down:
| Label | Test on the trace | Fix lives in |
|---|---|---|
| Retrieval-miss | Gold/needed evidence absent from retrieved set | pb-22/23/24/25; or ingestion (below) |
| Grounding-miss | Evidence retrieved but answer ignores/contradicts it | Generation prompt, context formatting, reranker order |
| Generation-miss | Evidence present and used; answer still wrong (bad reasoning, wrong synthesis) | Model tier, prompt, decomposition |
| Ingestion-miss | Evidence not in the index at all (stale, deleted, never chunked) | This playbook, steps 4–6 |
| Refusal-miss | Question unanswerable; system answered anyway (or vice versa) | Abstention prompt + hard-negative evals |
Run the clinic. For each failure: open the trace, apply the tests in order, label, and file. Automate the first test — a script that checks whether any retrieved chunk contains the expected evidence (RAGAS context_recall per-case does this). After 30–60 cases you have a Pareto: most systems concentrate 60–80% of failures in one or two labels, which sets the roadmap. Re-run the distribution monthly; it shifts as you fix things.
# per-case component check with ragas
from ragas import SingleTurnSample
from ragas.metrics import LLMContextRecall, Faithfulness
sample = SingleTurnSample(user_input=q, retrieved_contexts=ctx,
response=ans, reference=gold_answer)
# low context_recall → retrieval/ingestion-miss; high recall + low faithfulness → grounding-miss
Wire failures back into evals. Every triaged case gets minimized and appended to the pb-21 grounding set's hard slice (same loop as pb-20 step 6). The clinic is not a one-off audit; it is the intake valve for your eval set.
Make ingestion incremental and idempotent. Diff the source of truth against index state on a schedule; upsert only changes:
for doc in source.changed_since(last_run):
chunks = chunk(doc) # pb-22 winner
point_ids = [uuid5(NAMESPACE, f"{doc.id}:{i}:{doc.version}") for i in range(len(chunks))]
qdrant.upsert("corpus", points=embed_and_wrap(chunks, point_ids, doc))
qdrant.delete("corpus", filter=Filter(must=[
FieldCondition(key="doc_id", match=MatchValue(value=doc.id)),
FieldCondition(key="version", match=MatchValue(value=doc.prev_version))]))
Deterministic point IDs (doc + position + version) make reruns idempotent. Delete-then-orphan is the classic bug: deleting the doc's old chunks must be part of the same transaction-shaped step as upserting the new ones, or searches interleave versions.
Dedup and version at ingestion. Near-duplicate chunks (boilerplate, mirrored docs, template pages) poison retrieval diversity — apply the pb-17 FAISS dedup pass at ingestion, keeping counts as metadata. Carry version and effective_date in payloads (pb-25 schema) so retrieval can prefer current versions and answer "as of March" questions where required.
Monitor corpus and query drift. Weekly job: (a) index freshness — max(source updated_at − index updated_at), alert over threshold; (b) embedding-space drift — centroid shift / distribution distance of new chunks vs the corpus, which flags new topics your eval set doesn't cover; (c) retrieval health — score distributions of top-1 hits trending down means queries are outrunning the corpus; (d) unanswered/low-confidence rate by topic cluster (pb-17 on recent queries) to find what users need that the corpus lacks.
Schedule the full-reindex escape hatch. Chunking, embedding-model, or schema changes (pb-22/23/25 decisions) require full re-embedding: blue-green it — build the new index alongside, run the pb-21 scorecard against both, cut over on green, keep the old index until the bake period ends.
Before touching any prompt or model in response to a bad answer, check the trace for whether the evidence was retrieved — and check the index for whether the evidence was ever ingested. Stale-index bugs wear a hallucination costume.
doc_id no longer exists in the source of truth.