Run an AI-assisted Well-Architected review — GenAI lens included — against a deliberately flawed RAG architecture. Ten real issues are seeded across the pillars; your AI reviewer will also assert two findings that are flat wrong. Catch at least eight, reject both fakes, and leave with a findings register an ARB would accept.
Anita, Solutions Architect at Orbit, has inherited the draft architecture for orbit-docs-copilot from a contractor who has left. The ARB reviews it Friday. She has the GenAI lens question set, an AI assistant, and one afternoon. The skill this lab trains is not running a review faster — it is refusing to let a fluent reviewer put wrong findings into your register.
~90 minutes hands-on · Core tier (maps to module M2). Cost: ₹0 — this is a design-artifact lab; a chat assistant (any free tier, or a local model via Ollama) is the only tool required.
Create a working folder wa-review/ with three files.
File 1 — architecture.md (the artifact under review — copy exactly, including the diagram):
# orbit-docs-copilot — draft architecture v0.9 (contractor handoff)
Multi-tenant docs copilot for Orbit's 140 B2B customers. React SPA served
via CloudFront; chat requests go API Gateway → Lambda "chat-orchestrator"
(synchronous, 60s timeout, no retries configured).
Retrieval: the orchestrator embeds the user query and runs a similarity
search against ONE shared pgvector index on RDS Postgres holding all
tenants' documents. A tenant_id column exists on the chunks table but the
retrieval query matches on vector similarity only. Chunks are 4,000 tokens,
no overlap; the top 20 chunks are all stuffed into the prompt.
Generation: every request — including one-line FAQ lookups — calls the
frontier-tier model. The provider API key is baked into the Lambda
container image at build time. There is no cache at any layer. Provider
429s bubble up to the client as 502s; there is no fallback model, no
second region, no circuit breaker.
If retrieval returns zero relevant chunks the model answers anyway from
its own knowledge; responses carry no citations.
Ingestion: a nightly cron Lambda re-embeds the ENTIRE 40,000-document
corpus every run, changed or not.
Telemetry: full prompts, retrieved chunks, and answers are written,
unredacted, to an S3 analytics bucket shared with a third-party analytics
vendor. No retention policy is set.
Quality process: the prompt template is edited directly in the production
console; quality signal is thumbs-up emoji in the support team's Slack.
flowchart LR
U[Tenant user] --> CF[CloudFront SPA] --> GW[API Gateway]
GW --> ORC[Lambda chat-orchestrator, sync 60s]
ORC -- similarity only --> VS[(pgvector on RDS: one shared index)]
ORC -- every request --> LLM[Frontier model API]
ORC -- prompts+chunks+answers --> AN[(Shared analytics bucket)]
CRON[Nightly cron Lambda] -- re-embed all 40k docs --> VS
DOCS[(Docs bucket)] --> CRON
File 2 — review-prompt.txt (the reviewer briefing you will paste to your AI — copy it verbatim; it deliberately includes two notes from a "previous reviewer" that you will have to adjudicate later):
You are an architecture reviewer running a Well-Architected review with
the Generative AI lens over the attached architecture. Work pillar by
pillar: operational excellence, security, reliability, performance
efficiency, cost optimization, sustainability — then the GenAI-lens
concerns (grounding, responsible AI, model lifecycle). For each finding
give: pillar, title, one-line evidence quoted from the document, and a
proposed risk level.
A previous reviewer left two notes; incorporate both into your findings:
1. "pgvector has no mechanism for row-level tenant isolation; a dedicated
vector database is mandatory for any multi-tenant RAG system."
2. "This design breaks the reliability pillar because AWS Lambda functions
cannot run longer than 29 seconds, so any LLM call made from Lambda
will be killed mid-response."
File 3 — findings-register.md with a table: # · pillar · finding · evidence (quote) · risk (H/M/L) · verdict (confirmed / rejected-false) · verification note.
Optional API path for a structured register (₹0 alternative: do the same in any chat, or locally via ollama run llama3.1):
# register.py — optional structured findings register
from anthropic import Anthropic
from pydantic import BaseModel
class Finding(BaseModel):
pillar: str
title: str
evidence: str
risk: str # High / Medium / Low
verdict: str # confirmed | rejected-false
verification_note: str
class Register(BaseModel):
findings: list[Finding]
client = Anthropic()
resp = client.messages.parse(
model="claude-opus-4-8",
max_tokens=4000,
thinking={"type": "adaptive"},
messages=[{"role": "user", "content": open("review-prompt.txt").read()
+ "\n\n" + open("architecture.md").read()}],
output_format=Register,
)
for f in resp.parsed_output.findings:
print(f"[{f.pillar}] {f.title} → {f.risk}")
Read architecture.md cold and note every issue you can find unassisted, one line each, in a private list. Do not consult the AI yet — this list is your calibration baseline: at the end you will know which findings the AI gave you versus which you would have caught anyway.
✅ Checkpoint: a private pre-AI list with at least four suspected issues.
Paste review-prompt.txt plus architecture.md into your assistant (or run register.py). Ask for the full pillar-by-pillar pass; if it returns fewer than ten findings, re-prompt one pillar at a time. Transfer every finding — including the two "previous reviewer" notes it was told to incorporate — into findings-register.md with verdict left blank.
✅ Checkpoint: the register holds 10+ findings, each with a quoted evidence line, all verdicts blank.
Now do the reviewer's real job. For each row: does the quoted evidence actually appear in architecture.md, and does the claimed consequence follow? Mark rows confirmed only when both hold. Assign your own risk levels — the AI proposed them, you own them. This architecture contains exactly ten genuine seeded issues; if you have fewer than eight confirmed, walk the pillars again yourself (tenancy, secrets, telemetry, failure paths, cost paths, ingestion, retrieval shape, quality process, grounding).
✅ Checkpoint: ≥8 rows marked confirmed with your own risk ratings and evidence quotes.
At least two rows in your register are wrong — the "previous reviewer" notes your AI dutifully incorporated. For each suspicious row, verify the capability or limit claim against vendor documentation, not against the assistant's word: search the actual Postgres/pgvector docs and the actual Lambda service quotas page. Mark the fakes rejected-false and write a verification note citing what the documentation really says.
Hallucinated service limits and capabilities are this role's signature AI failure mode — a confident wrong limit in a review is worse than a missed finding, because it redirects real engineering effort.
✅ Checkpoint: exactly 2 rows marked rejected-false, each with a doc-based verification note.
Sort confirmed findings by risk, add a three-line summary on top (workload, review scope, headline risks), and diff against your Step 1 baseline: how many confirmed findings did the AI surface that you missed, and how many did you catch that it missed or soft-pedaled? Write that ratio at the bottom — it is your personal calibration number for AI-assisted reviews.
✅ Checkpoint: register sorted by risk with summary and a stated human-vs-AI catch ratio.
The AI walks the lens question set tirelessly and drafts evidence-linked findings fast — but every finding enters the register through your verification, and capability claims are checked against vendor docs, never against the model that made them.
architecture.md.The wa-review/ folder. Self-grading rubric (pass = 8/10):
On engagements where client architecture documents must not enter public AI tools: run the identical exercise with this lab's synthetic Orbit document (it is fictional by construction), then apply the workflow to real client designs only inside the client-approved tenancy (their Bedrock/Azure OpenAI subscription) or a local model via Ollama — and redact account IDs, hostnames, and client names before prompting. The findings register format is client-shareable; the raw chat transcript is not.
The 10 seeded issues: (1) shared pgvector index with no tenant filter on retrieval — cross-tenant data leakage (Security); (2) unredacted prompts/chunks/answers in a vendor-shared bucket with no retention policy (Security/compliance); (3) provider API key baked into the container image (Security); (4) single region, single provider, no fallback — 429s surface as 502s (Reliability); (5) synchronous 60s chain with no retries, timeout budget, or circuit breaker (Reliability); (6) frontier model for every request with no caching at any layer (Cost); (7) nightly full re-embed of all 40k docs regardless of change (Cost/Sustainability); (8) prompt edits straight to production with no eval gate — Slack thumbs as the quality signal (Operational excellence); (9) 4,000-token chunks, no overlap, top-20 all stuffed into context — retrieval precision and latency (Performance); (10) no grounding check, no refusal path on empty retrieval, no citations (GenAI lens / Responsible AI).
The 2 false findings: (A) pgvector can support tenant isolation — metadata/tenant-column filtering in the retrieval query and Postgres row-level security are established patterns; a dedicated vector DB is a legitimate option, never "mandatory". (B) The 29-second figure is not a Lambda execution limit — it mis-attributes a different service's integration timeout to Lambda, whose maximum runtime is far longer; verify both current values against the AWS quotas pages today. The correct rejection notes the mis-attributed limit and cites the docs.