Monitor RAG Pipeline Freshness and Retrieval Quality

Tier: Advanced · Workflow area: RAG data pipelines

The symptom shows up weeks later as users losing trust in answers. Data engineers already know the cure: observability on the data, not just the jobs. This playbook instruments the RAG data pipeline the way you instrument any pipeline — freshness SLAs, volume drift, failure-rate metrics — plus the one genuinely new instrument: canary queries that catch retrieval-quality regressions the day they happen.

When to reach for this

• •
  A RAG system is entering production and "how do we know it still works?" has no measured answer.
• •
  A silent failure already happened — stale index, missing source, degraded recall — and was found by users instead of alerts.
• •
  You run incremental sync (pb-36) and need its SLAs wired into the team's existing observability stack.
• •
  Retrieval quality must be defended over time, not just at launch eval (pb-28's golden set as a monitor, not a one-off).

The workflow

1. 1.

   Instrument the four data-side metric families. Emit these from the pipeline into whatever metrics store the team already uses (warehouse table + dashboard is fine):

   • •
     Freshness: per source, now() - max(applied event_ts) (pb-36's lag) and last-successful-run age per stage (parse, embed, sync).
   • •
     Volume/drift: chunk count, document count, chars-per-page (pb-27), chunk-token histogram per run — step changes here are parser or source breakages.
   • •
     Process health: embedding-batch failure and DLQ rates (pb-30), sync reconciliation drift (pb-36).
   • •
     Retrieval quality: canary-query pass rate — steps 2–4.
2. 2.

   Build the canary query set. Take 20–40 golden-set questions (pb-28) with stable, known-correct source documents, plus a few probes per critical source system ("does content from the HR wiki still retrieve at all?"). Store as data, not code:

   Copy

   {"canary_id": "hr-leave-01", "q": "How many privilege leave days accrue per year?",
    "expect_doc_id": "hr/leave-policy.md", "k": 10, "source": "hr-wiki"}
   

3. 3.

   Run canaries on a schedule against the live retrieval endpoint. Same entry point users hit (pb-31's shared function), so parser, index, filters, hybrid, and rerank are all inside the test boundary:

   Copy

   def run_canaries(canaries, search):
       results = []
       for c in canaries:
           hits = search(c["q"], k=c["k"])
           passed = any(h["doc_id"] == c["expect_doc_id"] for h in hits)
           results.append({**c, "passed": passed, "run_at": now_iso()})
       return results   # persist to canary_runs table
   

   The headline metric: canary pass rate (recall@10 over the canary set), tracked per run. Expected steady state: a flat line at your launch eval level (e.g. 92–97%). Alert on a drop of more than ~5 points sustained over two consecutive runs, and on any per-source probe going to zero — that is a whole source silently vanished.

4. 4.

   Make alerts diagnostic, not just loud. A canary failure plus which other metrics moved localizes the fault: pass-rate drop + chunk-count spike → chunking/parser change; drop confined to one source + freshness lag on that source → sync failure; drop after a deploy + no data metrics moved → retrieval code regression. Encode this triage table in the runbook next to the alert.

5. 5.

   Wire it into the existing stack — do not build a parallel one. Airflow/Dagster SLAs and sensors own freshness alerts; dbt-native observability (Elementary) or your data-observability platform (Monte Carlo class) owns volume/drift on the chunk tables; traces from Langfuse or Phoenix cover per-query retrieval latency and scores. The canary runner is one small scheduled job writing to one table that all of them can read.

6. 6.

   Review drift monthly, not just alerts daily. Slow decay — canary pass rate sliding 1 point a month as corpus composition shifts — never trips a threshold. A monthly look at the trend lines, plus refreshing canaries when documents legitimately change (a canary failing because its gold document was correctly updated is maintenance, not regression), keeps the monitor honest.

Verify before you trust it

• Fault-injection drilldeliberately break each layer in staging — pause sync (freshness alert must fire), delete a canary's gold document from the index (pass-rate alert must fire), corrupt the parser config (chars-per-page drift must fire). An alert that has never fired in a drill is a hypothesis, not a monitor.
• Alert latency measuredtime from injected fault to page is within the SLA you claim (a daily canary run means up-to-24h detection — say so explicitly, or run hourly).
• Canary maintenance path testedupdate one gold document legitimately, confirm the runbook's refresh procedure distinguishes "document changed" from "retrieval broke".
• Dashboard sanityone screen shows freshness per source, chunk-count trend, DLQ rate, and canary pass rate — if diagnosing requires five tools, on-call will skip it.

Pitfalls

• Monitoring jobs instead of data. Airflow showing green while the parser emits empty strings is the canonical RAG silent failure. Job success is necessary, data metrics are the point.
• Canaries hitting a test index or bypassing filters. Run them through the production retrieval function with a representative tenant/ACL context, or they certify a pipeline nobody uses.
• Alerting on single-run noise. One flaky canary out of 30 is a 3-point swing; require sustained deltas or quorum failures to page, or the alert gets muted within a month.
• A canary set that never changes. As the corpus and query mix evolve, the canaries measure yesterday's system. Add canaries with every major new source; retire ones whose documents churn constantly.
• LLM-quality metrics crowding out data metrics. Answer-quality evals (RAGAS-style) are valuable but confound model, prompt, and retrieval. Keep the data-side pass rate separate so regressions localize to your layer.

Tools

• Langfuse / Arize PhoenixOSS tracing and eval tracking for the query path: per-query latency, retrieval scores, and canary run history.
• Elementary (or Monte Carlo-class platforms)anomaly detection on chunk/document tables inside the dbt/warehouse estate; Elementary is the OSS, India-friendly default.
• Airflow / Dagster SLAsfreshness and job-health alerting where the team's pipelines already live.

Client-safe variant

Monitoring artifacts leak content in small ways — canary questions quote client documents, and failure alerts may embed retrieved text — so keep the canary set, run history, and dashboards inside the client boundary, and strip retrieved-passage payloads from anything that pages an external on-call channel (alert with IDs and metrics, never chunk text). The fault-injection drill doubles as a strong client deliverable: a recorded demonstration that stale-index and lost-source failures are detected within SLA is precisely the operational-readiness evidence their platform review wants, and it can be run entirely on synthetic canary documents seeded for that purpose.