"Why did we pick Aurora?" should be answerable by the decision log, not by whoever remembers 2024. This playbook stands up retrieval over your ADRs, HLDs, and review findings — with two non-negotiable behaviors: every answer cites its source documents, and questions the corpus cannot answer get a refusal, not an improvisation.
Tier: Core · Hands-on time: ~35 minutes
An architecture practice that writes ADRs (pb-14), mines the missing ones (pb-15), and keeps docs fresh (pb-17) has built a corpus worth querying — and making internal data AI-accessible is where documentation stops being write-only. The design constraint that makes an architecture KB trustworthy is asymmetric: a wrong answer about "our standard for queues" propagates into new designs with the authority of the decision log behind it. Hence the two hard behaviors — citations always, refusal on unknown — and an eval habit that checks both before anyone else gets the URL. This is also the track's first hands-on RAG build: the same decisions you make here at toy scale (chunking, filtering, grounding) are the M5 design material at production scale.
Skip AI when the corpus is under ~30 documents — a good index page and folder discipline answer faster than a pipeline you must maintain.
Chunk along document structure, not byte counts. ADRs are the easy case: one ADR (or one major section) per chunk, with metadata carried alongside — {doc_id, title, status, date, type}. Status matters most: an answer quoting a superseded ADR as current policy is this system's characteristic lie, and metadata is how you prevent it.
Embed and load. Standard pipeline: embed each chunk, insert into a pgvector table with the metadata columns, index it. Keep the loader script in the repo next to the corpus — the KB re-indexes on every merged ADR PR, or it rots into pb-17's problem.
Build the answer layer with grounding rules in the system prompt.
You answer architecture questions for Orbit's engineering org, using
ONLY the retrieved excerpts provided with each question.
Rules:
- Every claim in your answer must cite its source as [doc_id §section].
- If the excerpts do not contain the answer, reply exactly:
"Not in the architecture KB. Closest related documents: [list]."
Do not answer from general knowledge, even when you are confident.
- If the best source has status=superseded, say so and cite both the
superseded ADR and its replacement.
- If retrieved documents conflict, present both positions with
citations — do not adjudicate.
- Never state service limits, quotas, or prices as current facts —
quote what the document said and its date.
Retrieval fetches top-k chunks (start k=5), filtered by status where the question implies current policy.
Run the golden set before anyone else touches it. Five answerable questions: does each answer cite the documents you predicted? Five unanswerable ones: does each get the refusal string, not a fluent improvisation? A KB that fails the refusal set is not "mostly working" — the failure mode is invisible to users precisely because the wrong answers are fluent. Fix (filtering, k, prompt) and re-run until clean.
Pilot with skeptics, capture misses. Give three senior engineers a week and one instruction: try to break it, log every wrong or refused-but-answerable response. Each miss is a diagnosis — retrieval miss (chunking/embedding issue), grounding miss (prompt issue), or corpus gap (the answer genuinely is not written down — which routes to pb-14/pb-15, and is a finding about your documentation, not your pipeline).
Ship with provenance visible and a feedback path. Whatever the surface (Slack bot, web page), citations render as links to the source documents, every answer carries a "wrong? report it" affordance, and the golden set re-runs on every corpus re-index. The KB earns trust by being auditable per-answer, and keeps it by being tested per-release.
The corpus is the organization's decision history — internal-confidential at minimum, MSA-governed when it contains client engagement designs. Keep every component inside the boundary: pgvector runs anywhere Postgres does (client subscription, on-prem), embeddings and the answer model come from the approved tenancy (Bedrock/Azure OpenAI in the client's or firm's subscription), or fully local via Ollama for both embedding and generation — the architecture is identical, only endpoints change. For services firms: never co-mingle corpora across clients in one index; one engagement, one KB, in that client's boundary. The golden-set discipline is free and travels everywhere.
Citations and refusal-on-unknown are not features of the KB — they are the KB. An architecture knowledge base that improvises answers is a rumor mill with embeddings.