The RAG Platform Decision — Managed vs Custom, as an ADR

Tier: Advanced · Hands-on time: ~40 minutes

This is the first decision every RAG design hits, and the official guidance is itself a spectrum: Google publishes five distinct RAG reference architectures running from fully managed to GKE-custom to GraphRAG, and Amazon's own senior AI SA job postings name Knowledge Bases explicitly. The decision is real on both sides — which is exactly why it belongs in an ADR with weights the architect owns, not the AI.

When to use

• A first customer-facing RAG workload is being designed and the platform choice will outlive the feature.
• A client RFP demands a GenAI architecture and you must defend managed-vs-custom in the review.
• An existing custom pipeline is fraying and someone is asking "why aren't we just using the managed thing?"
• Don't reach for this when the workload is a throwaway internal POC — pick the fastest path, label it POC-only, and record the real decision when production is on the table.

Prerequisites

• The workload facts: corpus size and churn rate, tenancy model, data-residency constraints, query volume, latency budget.
• A chat assistant with your org context (Claude Projects or equivalent) to scaffold the matrix and red-team the draft.
• Vendor documentation open for the two or three candidate platforms — every capability claim gets checked there.
• Your org's ADR template (MADR-style works).

The workflow

1. 1.

   Fix the decision dimensions before asking AI anything. The research-backed set: control over retrieval internals, cost shape, compliance and residency fit, ops burden, and exit cost. Add workload-specific dimensions; resist letting the assistant add its own later.

2. 2.

   Set the weights yourself. Weights encode business judgment — Anita, designing Orbit's orbit-docs-copilot, weights compliance and exit cost high because the copilot is multi-tenant and the platform decision is nearly irreversible once customer indexes exist. The AI never sets weights; it interrogates them.

3. 3.

   Scaffold the matrix with a constrained prompt.

   Copy

   You are helping me build a platform decision matrix for a customer-facing
   RAG copilot ("orbit-docs-copilot") at a B2B SaaS company.
   
   Candidates: (a) Bedrock Knowledge Bases, (b) Azure AI Search + Foundry,
   (c) custom LlamaIndex + pgvector on our existing Aurora Postgres.
   
   Dimensions and my weights: control 20, cost shape 15, compliance/residency 25,
   ops burden 15, exit cost 25.
   
   For each cell: 2-3 sentences of assessment plus a "VERIFY" line naming the
   exact vendor doc claim I need to confirm. Do not score the cells — I score.
   Where you are unsure whether a capability exists, say UNKNOWN, do not guess.
   

4. 4.

   Verify every capability cell. Chunking control, metadata filtering, access-control integration, region availability, pricing model — each claim gets a citation to a vendor doc page, checked today. Stale or invented platform capabilities are this track's signature failure mode, and RAG platforms change quarterly.

5. 5.

   Score the cells yourself, then make the AI argue the other side. Whichever option leads, prompt: "Argue that this choice is wrong for this workload. What breaks in 18 months?" The strongest counterarguments become ADR consequences.

6. 6.

   Price the exit explicitly. For the managed options: what leaves with you (source documents, metadata) and what does not (index format, chunking behavior, tuning). For custom: who maintains it when the two engineers who built it move on. Write the exit-cost paragraph as if you are the person executing the exit.

7. 7.

   Draft the ADR from the completed matrix. Context, options considered with scores, decision, consequences — including the losing options' strongest points and a named revisit trigger (corpus 10x, new residency rule, platform pricing change).

8. 8.

   Attach a one-page reference diagram of the chosen architecture so the ADR is reviewable by people who will never open the matrix.

Verify before you trust it

• Every matrix cell has a live citation. Open the vendor page for each capability claim; anything the AI marked UNKNOWN or you cannot find in docs is treated as absent from the platform.
• Pricing comes from the calculator, not the chat. Run the candidate workloads through each provider's pricing calculator with your real volume assumptions; never let an AI-recalled price into the matrix.
• Residency claims checked per region. Confirm the specific services are available in the regions your data must stay in — regional availability is a documented gap that AI summaries gloss over.
• The scores are yours. Audit the final matrix: if any numeric score originated in the assistant's output, re-derive it. AI critiques and structures; the human owns ratings and sign-off.
• The exit-cost section survives a hostile read. Have a colleague read only that section and ask: "could I actually execute this migration from what's written?"

Restricted-environment variant

When the decision is for a client engagement, extract the MSA's AI and data-handling constraints first — they can eliminate candidates outright (no cross-border processing, approved-vendor lists) before any matrix work. Describe the workload to the assistant abstractly: "a multi-tenant document copilot for a regulated retail client," no client names, corpus samples, or contract terms. If the client provides approved tenancy (Bedrock or Azure OpenAI in their subscription), run the scaffolding there. For a fully offline pass, a local model via Ollama can scaffold the matrix structure — the verification work is all vendor-doc reading, which needs no client data at all.