Lab 07 · RAG Platform Decision Package

Orbit's board has approved orbit-docs-copilot for GA, and Anita must now commit the platform: a managed RAG service from a hyperscaler, or a custom stack she controls. Every hyperscaler publishes reference architectures for both paths — the decision spectrum is the official guidance shape, not a vendor trick. What the guidance cannot do is weight Orbit's constraints. That is the architect's job, and it is not delegable to the AI.

Time & tier

~90 minutes hands-on · Core→Advanced tier (maps to module M5). Cost: ₹0 — design-artifact lab; a chat assistant (free tier or local Ollama), Mermaid for the diagram.

Setup

Create rag-decision/ with the scenario below as scenario.md, plus empty matrix.md, architecture.mmd, adr-rag-platform.md, exit-cost-note.md.

Orbit — orbit-docs-copilot platform decision: scenario constraints

Company: Orbit, B2B SaaS (workforce management), 140 tenant companies.
Workload: customer-facing docs copilot; tenants upload their own HR
  policy documents — India-scoped personal data is in scope, so DPDP
  Rules 2025 obligations (consent lineage, deletion on request, purpose
  limitation) attach to prompts, embeddings, and outputs.
Tenancy: hard requirement — tenant A's documents must never influence
  tenant B's answers; several enterprise tenants demand per-tenant
  deletion certificates.
Residency: Indian tenants' data stays in-region; two EU prospects will
  require EU processing within 12 months.
Team: Anita + 4 engineers; strong on Postgres/K8s/VPCs; nobody has run
  a search cluster; on-call rotation is already thin.
Traffic: ~40k requests/day at GA, 3x peak; growth plan 5x in 18 months.
Commercials: platform opex target under control of unit economics
  review each quarter; CFO allergic to open-ended per-query pricing
  without a cap; finance wants an exit story before any multi-year
  commitment.
Timeline: GA in one quarter; a failed re-platform later would cost two.

Guided steps

Step 1 — Extract decision drivers from the scenario (15 min)

Turn the scenario into 6–8 named decision criteria in matrix.md — e.g., tenant isolation & deletion capability, residency control, ops burden at team size, time to GA, cost shape & predictability, retrieval quality control, exit cost, ecosystem fit. Ask your assistant to critique your criteria list against the scenario ("what constraint have I not represented?") — but the list is yours. Every criterion must trace to a scenario line; delete any that exist because they are conventional rather than because Orbit needs them.

✅ Checkpoint: 6–8 criteria in matrix.md, each annotated with the scenario line that justifies it.

Step 2 — Set weights before the AI scores anything (15 min)

Assign each criterion a weight (total 100) and write one sentence per weight defending it from the scenario — the deletion/tenancy and residency constraints are regulatory, the team-size constraint is structural, so expect them to dominate. Do this before any option scoring: weights set after scores are rationalization, not analysis. Record the weight rationale in the matrix file; the ARB will read the rationale, not the arithmetic.

✅ Checkpoint: weights sum to 100, set and justified prior to any scoring, committed in matrix.md.

Step 3 — Score the options with the AI, verify every capability claim (25 min)

Define 3 options: (a) managed RAG (a hyperscaler knowledge-base/search offering end to end), (b) custom stack (LlamaIndex-class framework + pgvector on managed Postgres + your own eval loop), (c) hybrid (managed model endpoint, custom retrieval). Have the assistant draft 1–5 scores per criterion per option with a stated reason each. Then do the architect's pass: any score whose reason rests on a claimed platform capability — per-tenant deletion semantics, residency guarantees, isolation model, pricing shape — gets verified against the vendor's current documentation before you accept it, because capability claims about managed AI services are exactly where assistants hallucinate. Adjust scores you disagree with; mark every adjustment.

✅ Checkpoint: completed matrix with weighted totals, ≥2 cells marked "verified against docs on <date>", and your adjustments visible.

Step 4 — Draw the reference architecture for the winner (15 min)

For the winning option, produce architecture.mmd (Mermaid): ingestion path with consent/lineage capture point, per-tenant isolation mechanism made visible (metadata filter or index-per-tenant — name it on the edge), the deletion path (the DPDP-driven arrow most diagrams omit: request → embeddings + chunks + caches + traces), region boundary, and the eval loop as a first-class component. Generate with the AI, then verify each element against your matrix reasoning — the diagram must show the mechanisms your scores assumed exist.

✅ Checkpoint: rendering diagram whose isolation, deletion, and residency mechanisms match the claims your matrix scored.

Step 5 — Write the ADR and the exit-cost note (20 min)

Draft adr-rag-platform.md (MADR shape: context from the scenario, the three options with weighted totals, decision, consequences — including what you gave up). Then the artifact that separates a decision package from a slide: exit-cost-note.md. In one page: what is portable if Orbit must leave the chosen platform in 18 months (documents, eval sets, prompts, application code) versus what is rebuilt (embeddings — re-embedding cost at then-current prices, index structures, platform-specific orchestration, learned retrieval tuning), an order-of-magnitude effort estimate in engineer-weeks with your assumptions stated, and the two lock-in reducers you will adopt from day one (e.g., portable document store as source of truth, provider-neutral eval harness). Red-team the ADR once with a fresh adversarial prompt and fold in what survives.

✅ Checkpoint: ADR status "proposed" with consequences; exit-cost note quantifies portable-vs-rebuilt with stated assumptions.

Verification

• Traceabilityevery criterion and every weight cites a scenario line; nothing generic survives.
• Verification trailcapability-dependent scores show doc-verified marks with dates; at least one AI-drafted score was adjusted with a reason.
• Diagram honestyisolation, deletion, and residency mechanisms in the diagram match what the matrix assumed; the deletion path reaches embeddings and caches and traces.
• Exit realismthe note distinguishes portable from rebuilt and carries an effort estimate with assumptions, not a hand-wave.

Completion artifact

The rag-decision/ package: matrix, diagram, ADR, exit-cost note. Self-grading rubric (pass = 8/11):

• Criteria & weights (0–3)scenario-traced, justified before scoring = 3.
• Score verification (0–3)capability cells doc-verified, adjustments marked = 3.
• Architecture fidelity (0–2)all three regulatory mechanisms visible = 2.
• ADR quality (0–1)consequences include what was given up = 1.
• Exit-cost note (0–2)portable/rebuilt split with estimate and lock-in reducers = 2.

Client-safe variant

Platform decisions for clients involve commercial terms and client data classifications that must not enter public tools. Run this lab's synthetic scenario freely; on engagement, keep negotiated pricing and tenant names out of prompts (use indexed placeholders), run scoring assistance inside the client-approved tenancy or a local model, and verify capabilities against vendor docs from a logged-out browser so the trail is reproducible. The four-artifact package format is the client deliverable.

Stuck?

• •
  The AI's scores all cluster at 3–4: force ranking — "for this criterion, order the three options and justify the order" — then convert to scores yourself.
• •
  You can't verify a capability claim from docs: score it conservatively and record "unverified — pilot before commitment" in the ADR consequences; unverifiable claims are findings, not blockers.
• •
  The matrix produces a tie: your weights are avoiding the decision — revisit Step 2; regulatory constraints in this scenario are not weight-2 items.
• •
  The exit-cost note keeps growing: cap it at one page by keeping only costs that differ between the options — shared costs don't inform the decision.
• •
  The red-team pass produces nothing useful: run it in a fresh conversation with only the ADR text — an assistant that drafted the package will defend it instead of attacking it.