RAG Evaluation Architecture — the Loop Is Part of the System

Tier: Core · Hands-on time: ~35 minutes

The posting evidence is verbatim: "RAGAS evaluation" appears in the job descriptions this track is built from, and the design principle it encodes is that evaluation is part of the system architecture, not a QA afterthought. The architect's deliverable here is an eval architecture design — what gets measured, where the gates sit, who owns the golden set — that the build team implements and the operations track runs.

When to use

• A RAG design is heading to review and its quality story is currently "we tried it and it seemed good."
• Retrieval or prompt changes keep shipping unmeasured and quality regressions are discovered by users.
• A client or ARB asks "how will you know it still works?" and the answer must be a diagram, not a promise.
• Skip AI when an eval loop already exists and the problem is a specific failing metric — debug that metric; don't redesign the loop around it.

Prerequisites

• Access to real or realistic user questionssupport tickets, search logs, or SME interviews — to seed the golden set.
• Ragas and promptfoo (or equivalents) as the candidate harness tools; Langfuse or Phoenix as the candidate trace/sampling layer.
• Your CI system's gate mechanicsthe eval must be able to block a merge.
• A chat assistant to scaffold the design and draft the golden set for human curation.

The workflow

1. 1.

   Define what "good" means for this system, in metrics. The Ragas-class core: faithfulness (answer grounded in retrieved context), context precision and recall (retrieval quality), answer relevance. Add workload-specific checks — citation presence, refusal-on-unknown behavior, tenant-scoping. Each metric gets a target and an owner.

2. 2.

   Design the golden set as a maintained asset. Size it honestly (30–100 items to start), source questions from real usage, have SMEs verify reference answers against source passages, and — critically — assign ownership and a refresh cadence. A golden set nobody owns is stale in one quarter.

3. 3.

   Draft the set with AI, curate by hand. Anita seeds orbit-docs-copilot's set from support tickets:

   Copy

   Here are 60 support tickets that our docs copilot should have been able
   to answer (attached, sanitized). Draft a golden evaluation set:
   
   - For each viable ticket: the user question (cleaned), the doc sections
     a correct answer must draw from, and a 2-3 sentence reference answer.
   - Mark every item where you are not certain the reference answer is
     fully supported by the cited sections — a human will verify all items,
     but flag the doubtful ones for extra scrutiny.
   - Include 8 out-of-scope questions where the correct behavior is
     refusal with a pointer, not an answer.
   

   Every reference answer is then human-verified against the docs. The out-of-scope items matter: refusal behavior regresses silently if it is never measured.

4. 4.

   Place the offline gate in CI. Any change to chunking, embeddings, prompts, retrieval config, or model version triggers the eval run; the merge blocks if a headline metric drops beyond a set threshold. Decide the thresholds and the override path (who may ship past a red eval, and what they must record) at design time.

5. 5.

   Design production sampling. CI catches what the golden set covers; production catches what it doesn't. Specify the sample rate, what gets scored (the same metrics where computable without references, plus judge-scored quality per pb-46), where scores land (Langfuse/Phoenix traces), and the PII handling for sampled prompts and answers.

6. 6.

   Wire drift alerts. Two drifts matter: quality drift (sampled metrics trending down — alert thresholds and an owner) and corpus drift (the golden set no longer resembling live questions — schedule a quarterly comparison of sampled production questions against golden-set coverage, feeding the refresh).

7. 7.

   Draw the loop and ship the design. One diagram: golden set → CI gate → deploy → production sampling → drift alerts → golden-set refresh, with owners on each edge. Plus the metric table (metric, target, threshold, gate location). Hand the operational dashboards to the platform/DevOps side — they run it; you designed it.

Verify before you trust it

• The gate actually blocks. Seed a deliberate regression (break the retriever's top-k, or corrupt a prompt) and confirm CI goes red and the merge is stopped. An eval that has never blocked anything is decoration.
• Golden-set answers are human-verified. Sample every AI-drafted reference answer against its cited sections before it enters the set; an unverified golden set launders the assistant's errors into your quality bar.
• Metrics reproduce within noise. Run the full eval twice on the same build; if headline metrics wobble more than your regression threshold, the threshold is meaningless — fix set size or scoring first.
• Sampling respects privacy rules. Confirm sampled production traces follow your retention and PII policy — traces are a personal-data surface (pb-49), and the eval design must say so.
• Thresholds are yours, not the tool's defaults. Every gate threshold traces to a decision you can defend in review, with the trade-off (strictness vs deploy friction) written down.

Restricted-environment variant

Golden sets built from client tickets carry client data — sanitize before any external assistant sees them, per the MSA. Where real tickets are off-limits entirely, generate synthetic questions from the document corpus structure with SME review standing in for user reality, and label the set's provenance honestly in the design. The eval loop itself runs fine inside restricted boundaries: Ragas and promptfoo are open source, judge calls can route to client-tenancy endpoints (Bedrock or Azure OpenAI in the client subscription), and a fully local loop with Ollama-served models works for the strictest classes — at reduced judge quality, which the design should state.