Designing RAG Systems

This module opens the second half of the track: not using AI to do architecture work, but architecting AI systems themselves. The demand signal is verbatim in the job postings — Amazon's Senior AI Solutions Architect JD names "RAG (Knowledge Bases, OpenSearch Serverless, Aurora pgvector)" explicitly — and all three hyperscalers now publish formal RAG reference architectures. Your job is not to memorize any one stack. It is to make the decisions the stacks force, in the right order, with evidence.

The scenario — Anita picks a platform without picking a side

Notice what Anita did not do: she did not let the framework debate run as a features argument, and she did not trust a single unverified capability claim. The platform decision is the highest-stakes RAG decision because it constrains every later one — and it is also the most reversible-looking decision that quietly isn't, which is why the exit-cost note is a first-class section of the ADR, not a footnote.

What You'll Learn

• The managed-vs-custom decisionscore Bedrock Knowledge Bases, Azure AI Search + Foundry, and Vertex AI Search against a LangChain/LlamaIndex + vector DB stack on control, cost, compliance, ops burden and exit cost, and ship the result as an ADR.
• Vector-store selectionchoose between co-located pgvector, OpenSearch Serverless, and dedicated vector databases using co-location, filtering, scale, cost floor and residency as the axes.
• Retrieval design decisionsset chunking strategy per document type, choose hybrid vs pure vector search, and budget a reranker's cost and latency — each validated by evals, not intuition.
• RAG security and tenancyenforce document-level authorization at retrieval time, choose index-per-tenant vs metadata filtering deliberately, and treat embeddings as a personal-data surface.
• GraphRAG and agentic retrievalrecognize when relationships and multi-hop questions justify the extra cost, and when they don't.
• Eval architecture as a design elementdesign the golden set, CI metrics gate, and production sampling loop into the system from day one, so every retrieval decision above has a referee.

The platform decision — managed, custom, or staged

The decision spectrum is now the official guidance shape: Google alone publishes five distinct RAG reference architectures, running from fully managed search through pgvector-on-AlloyDB builds to GKE-hosted custom pipelines and GraphRAG. The managed end — Bedrock Knowledge Bases, Azure AI Search with Foundry, Vertex AI Search — handles ingestion, chunking, embedding, retrieval and citation for you. The custom end — LangChain or LlamaIndex over a vector DB you operate — hands you every knob and every pager.

Score the two ends on five axes, and make the weights yours, not the AI's:

• Controlcustom wins. Managed services expose limited chunking, embedding and reranking knobs; if your eval results demand a retrieval strategy the service doesn't offer, you're stuck negotiating with a roadmap.
• Costdepends on shape, so model it, don't assume it. Managed pricing bundles convenience; custom trades service fees for engineering time and an ops rota. The honest comparison is a TCO model with verified prices (Module 7's discipline).
• Compliancemanaged services inherit the provider's certifications and region story, which helps; custom lets you pin every component to a region and prove data paths end to end, which some regulators and client MSAs demand. DPDP-scoped personal data makes residency and deletion paths a scoring criterion, not a tiebreaker.
• Ops burdenmanaged wins, decisively, for teams without a platform group. A custom RAG pipeline is a distributed system: ingestion jobs, embedding versioning, index maintenance, retrieval latency SLOs. Someone must own it at 2 a.m.
• Exit costthe axis teams forget. Managed exits mean re-ingesting the corpus, rebuilding chunking behavior, and re-validating quality elsewhere. Keep the corpus, chunking config and eval suite portable and provider-neutral, and the exit stays a project instead of a rewrite.

Here is the worked shape of Anita's matrix — your weights will differ, and that's the point:

The output is an ADR: context, options, weighted scores, the decision, and the consequences — including the exit instruments. A staged strategy is often the honest answer: launch managed, keep the corpus and evals portable, and revisit when eval results or costs hit a wall the managed knobs can't fix.

When you do use an assistant to draft the comparison, structure the prompt so verification is built in:

Draft a managed-vs-custom RAG platform comparison for a customer-facing
docs copilot (multi-tenant SaaS, India + EU customers, DPDP-scoped data).

Candidates: Bedrock Knowledge Bases | Azure AI Search + Foundry |
Vertex AI Search | LlamaIndex + pgvector (self-managed).

Rules:
1. Score axes: control, cost, compliance, ops burden, exit cost.
   Propose scores; I will set the weights myself.
2. For EVERY quantitative claim (limits, quotas, regions, prices),
   output a [VERIFY] tag with the exact vendor doc page to check.
   Do not present any number as fact.
3. List what each managed option does NOT let me configure.
4. End with an exit-cost note per option: what must be rebuilt to leave.

Choosing the vector store

The vector store decision is narrower than the platform decision but earns its own ADR, because it carries the residency and cost floor. Three families, three profiles:

• Co-located pgvector (Aurora, AlloyDB, or any Postgres)embeddings live next to your operational data. You get SQL joins between vectors and business tables, one backup story, one residency story, and a cost floor near zero for teams already running Postgres. Google's own reference architectures include exactly this embedding-alongside-operational-data pattern. For most product teams, this is the default until scale or search features force a move.
• OpenSearch Serverlessnamed verbatim in the Amazon AI SA posting for a reason: strong hybrid (keyword + vector) search, serverless scaling, and a natural fit when you're already in the AWS search ecosystem. The trade is another service in the architecture and a higher cost floor than reusing your existing database.
• Dedicated vector databases (Pinecone, Weaviate, Qdrant)purpose-built performance and features at scale, but a new vendor surface: a separate residency and DPA conversation, another exit cost, another bill. The OSS members of the family (Qdrant, Weaviate) self-host cleanly, which restores control at the price of ops.

Residency deserves one explicit paragraph in the ADR. DPDP today imposes no blanket localization, but the government can restrict transfers per-destination — so the design goal is restrictability: choose a store you can pin to a region, and keep the corpus and embeddings portable enough to re-home if a restriction lands. A portable store is a compliance instrument, not just an ops preference.

Retrieval design decisions, refereed by evals

Retrieval quality decisions look like tuning; at architecture level they are budget decisions, and every one of them is validated by eval deltas — never by intuition or a demo that happened to go well.

• Chunking follows document type. API references, policy PDFs, release notes and support tickets have different natural units; one chunk size across a mixed corpus is the most common silent quality killer. Decide chunking per document family, and record it as configuration that travels with the corpus.
• Hybrid vs pure vector. Pure vector search fumbles exact identifiers — SKUs, error codes, version numbers — that keyword search nails. If your golden set contains lookup-shaped questions (most product-docs corpora do), hybrid retrieval is usually where evals point.
• Rerankers buy quality with latency and money. A reranker re-scores the top-k on every query — a per-query cost and latency tax, forever. Budget it explicitly: what eval improvement justifies how many added milliseconds and what per-query spend? If the eval delta is marginal, the answer is no.

The discipline that makes all three tractable: change one knob, re-run the golden set, keep the change only if the metrics move. That requires the eval loop from the last section of this module to exist before you start tuning — which is exactly why it's a design element, not an afterthought.

Security and tenancy architecture

A RAG system is a new path to your documents, and it must not become a permission bypass or a tenant-boundary leak. Four decisions, made at design time:

• Document-level authorization. Retrieval must respect source ACLs: if a user can't open the document in the source system, the copilot must not quote it to them. That means propagating identity into the retrieval query and filtering at retrieval time — an architecture decision, not a prompt instruction.
• Tenant isolation model. Index-per-tenant makes cross-tenant retrieval structurally impossible and simplifies per-tenant deletion, at the cost of operational sprawl as tenants grow. A shared index with metadata filtering scales operationally but is one dropped filter away from a breach — if you choose it, the filter must be enforced in one mandatory code path, tested adversarially. For DPDP- or client-MSA-scoped data, bias toward structural isolation.
• PII in embeddings. The DPDP practitioner analysis is blunt: prompts, embeddings and model outputs are all personal-data surfaces. Embeddings can leak information about their source text, so they inherit the source's classification — which means retention, deletion and residency obligations attach to the vector store, not just the document store. Deleting a user's data means deleting their vectors too; design the mapping that makes that possible.
• Redaction before ingestion. A redaction gateway on the ingestion path — masking identifiers before embedding — shrinks every downstream obligation at once. It's far cheaper to keep PII out of the index than to chase it through one later.

Before a RAG design leaves review, walk this checklist:

• ACL paritycan any query retrieve a chunk its user couldn't open at the source? Prove no, with a test.
• Tenant boundaryis isolation structural (per-tenant index) or filtered? If filtered, is the filter unbypassable and adversarially tested?
• Deletion pathgiven one user's erasure request, can you enumerate and delete their chunks, embeddings and cached answers? In how long?
• Embedding classificationare vectors stored, backed up and access-controlled at the same tier as their source documents?
• Ingestion trustfor every ingestion source, can an attacker write to it? Each yes needs sanitization and provenance (the Security track's poisoning lesson applies here in full).

GraphRAG and agentic retrieval — priced honestly

Two advanced retrieval architectures earn their complexity only when the simple one demonstrably fails.

GraphRAG combines vector retrieval with a knowledge graph, and it matters when the questions are about relationships: org structures, bills of materials, regulatory cross-references — "which products are affected if this component changes?" is a graph traversal wearing a search costume. It has hyperscaler-canon status — Google's Architecture Center publishes a GraphRAG reference architecture — but the honest price is steep: you now own graph construction and maintenance as a pipeline, entity extraction quality as a new failure mode, and more moving parts per query.

Agentic retrieval puts a model in the retrieval loop — decomposing multi-hop questions, issuing follow-up queries, deciding when it has enough. It answers questions single-shot retrieval can't, and it multiplies token spend and latency per question while adding a new nondeterministic component to debug.

The decision rule for both: collect the eval failures first. If your golden set shows a class of relationship or multi-hop questions the plain pipeline keeps failing, you have evidence sized to a business case. If you can't produce those failing cases, you're buying complexity for a hypothesis.

The eval loop is part of the system

Every decision in this module was refereed by evals — which is only possible if the eval loop is designed into the system, with the same seriousness as the ingestion pipeline. The posting evidence names this directly ("RAGAS evaluation" appears verbatim in the JDs). The architecture has three parts:

• A golden set the humans own. 30–50 questions with reference answers, drawn from real user intent, curated — not generated and forgotten. AI can draft candidates; the architect and domain owners decide what makes the set.
• CI metrics that gate change. Ragas-class metrics — faithfulness, context precision, context recall — run against the golden set on every retrieval-design change: chunking, embedding model, hybrid weights, reranker. A regression blocks the merge exactly like a failing unit test.
• Production sampling and drift alerts. A sampled slice of live traffic scored continuously (Langfuse or Phoenix-class tooling — Module 7 covers the judge design; the DevOps track operates the dashboards). Corpus drift and query drift both degrade quality silently; sampling is how you notice before customers do.

AI helps you draft the golden set from the corpus — with the architect curating every item:

from pydantic import BaseModel
import anthropic

class GoldenItem(BaseModel):
    question: str
    reference_answer: str
    source_doc_ids: list[str]
    question_type: str  # "lookup" | "synthesis" | "multi-hop" | "out-of-corpus"

class GoldenSetDraft(BaseModel):
    items: list[GoldenItem]

client = anthropic.Anthropic()

resp = client.messages.parse(
    model="claude-opus-4-8",
    max_tokens=4000,
    thinking={"type": "adaptive"},
    system=(
        "Draft golden-set candidates for RAG evaluation. Ground every "
        "reference answer in the provided docs, cite source doc ids, and "
        "include out-of-corpus questions where the correct behavior is refusal. "
        "A human curator makes the final selection."
    ),
    messages=[{"role": "user", "content": corpus_excerpts}],
    output_format=GoldenSetDraft,
)
draft = resp.parsed_output  # architect curates before anything enters CI

Where this lands in your week

• At the whiteboardthe five-axis platform matrix replaces the framework flame war; you set weights, AI drafts scores, vendor docs settle claims.
• In the ADR queuethe platform decision and the vector-store decision each land as an ADR with an exit-cost note; six months from now, nobody re-litigates.
• In design reviewthe tenancy checklist (ACL parity, tenant boundary, deletion path, embedding classification, ingestion trust) becomes your standing RAG review rubric.
• In sprint planningthe golden set is a ticket in the first sprint, before any tuning ticket; CI eval gates land with the pipeline, not after it.
• On client callswhen a client asks "why not just use the managed thing?", you answer with the matrix and the exit instruments, not vibes.

Your playbooks & labs

Take this module into Lab 07 (RAG platform decision package) — for a multi-tenant SaaS scenario with DPDP-scoped data, you'll produce the full decision package: weighted matrix, reference architecture diagram, ADR and exit-cost note — and Lab 08 (build both RAG paths), where you'll stand up a managed knowledge base AND a LlamaIndex + pgvector pipeline on the same 50-document corpus and settle the argument with a Ragas scorecard, exactly as Anita did.

Companion playbooks: pb-28 (The RAG Platform Decision — Managed vs Custom, as an ADR), pb-29 (Vector Store Selection — pgvector, OpenSearch, or Dedicated), pb-30 (Retrieval Design Decisions, Validated by Evals Not Intuition), pb-31 (RAG Security & Tenancy Architecture), pb-32 (GraphRAG & Agentic Retrieval — When Relationships Matter), and pb-33 (RAG Evaluation Architecture — the Loop Is Part of the System).