Build a Platform Docs Q&A Bot That Cites Sources and Refuses Unknowns

Tier: Core · Hands-on time: ~90 minutes

A RAG assistant over your platform docs deflects a real share of those tickets — but only if it behaves like a good platform engineer: cites its sources, says "I don't know" instead of inventing a kubectl command, and never answers from stale docs. This playbook builds that bot, puts refusal behavior and corpus freshness in from day one, and measures deflection honestly. DORA's AI Capabilities Model calls this capability "AI-accessible internal data" — it is one of the seven things that make org-level AI adoption actually pay off.

When to use

• •
  Your platform/DevEx team fields repeated how-do-I questions (deploy, get a database, set up alerts) answered somewhere in your docs.
• •
  Onboarding waves: new engineers or a newly onboarded client team need self-service answers with authoritative links.
• •
  As the retrieval backbone for platform-ticket deflection: classify incoming tickets, auto-answer the answerable fraction with citations, route the rest with context attached.

Do not build this if your docs are thin or mostly wrong — RAG amplifies your corpus, including its rot. Fix the top-20 documents first (this playbook's corpus audit shows you which ones).

Prerequisites

• •
  A docs corpus: Backstage TechDocs / wiki export / runbooks repo in Markdown (Markdown-native corpora work best).
• •
  An embedding + vector store path: pgvector or Chroma (both free/self-hostable), plus an LLM API or local model for generation.
• •
  Common Core RAG literacy assumed: chunking, embeddings, retrieval, and evals are not re-taught here.
• •
  A place the bot will live (Slack bot, Backstage plugin, or CLI) and a named owner for the corpus.

The workflow

1. 1.

   Audit and stamp the corpus first. Before embedding anything, run runbook-rot detection: for each doc, extract commands/versions/URLs and check them against current reality (does the CLI flag still exist, does the linked dashboard resolve, is the cluster name current?). AI does the extraction and comparison lists; you confirm. Tag every doc with last-verified date and an owner. Docs failing the audit get fixed or excluded — an authoritative-sounding bot serving stale runbooks is worse than no bot.

2. 2.

   Ingest with metadata that enables refusal. Chunk by heading; store source path, section anchor, last-verified, and doc owner with each chunk. These fields power citations ("per docs/deploy.md#canary, verified 2026-06") and staleness warnings at answer time.

3. 3.

   Constrain generation to retrieved context.

   Copy

   You answer internal platform questions ONLY from the provided context
   chunks. Rules:
   - Every answer must cite source file + section for each claim.
   - If the context does not contain the answer, reply exactly:
     "I don't have this in the platform docs" + the 2 closest docs +
     "ask #platform-help". Never construct commands or config from
     general knowledge.
   - If any cited chunk has last-verified older than 90 days, append a
     staleness warning naming the doc owner.
   - Prefer quoting exact commands from docs over paraphrasing them.
   

   The refusal string is a feature, not a failure: a wrong-but-confident helm upgrade command costs more than ten "I don't know"s.

4. 4.

   Build the eval set before launch: 20 questions, 5 unanswerable. From real ticket history: 15 questions your docs do answer (with expected source), and 5 your docs genuinely do not. Score: answer accuracy, citation correctness (the cited section really supports the answer), and refusal correctness (all 5 unanswerables refused). Gate launch on refusal correctness — it is the trust-critical metric, and re-run the set on every corpus or prompt change.

5. 5.

   Ship narrow, then widen. Launch to the platform team, then one friendly product team, in the channel where questions already arrive. Log every question, answer, citation, and a thumbs-up/down. Weekly triage of thumbs-down: wrong retrieval → chunking/metadata fix; wrong docs → docs fix (with the owner from step 1); should-have-refused → prompt fix. Each is a different repair; the logs tell you which.

6. 6.

   Wire the deflection loop and measure honestly. Connect to your ticket intake: classify incoming platform tickets, auto-answer only above a confidence threshold and always with citations, route everything else to a human with the retrieved context attached (making the human faster too). Report deflection as "tickets resolved by bot answer with no human follow-up" — not "tickets the bot replied to".

Tool variants

• pgvector/Chroma + LLM API + Slack bolt app: the default self-built path; a few hundred lines, fully ownable.
• Backstage TechDocs + RAG plugin: keeps answers inside the IDP with per-page anchors as citations; corpus audit workflow unchanged.
• Enterprise search (Glean-class) or vendor assistants: faster to ship; you still own steps 1, 4, and 6 — corpus hygiene, evals with unanswerables, and honest deflection metrics do not come in the box.
• Local-only variant: Ollama (8B-class) + Chroma runs the entire stack on one VM for restricted environments; expect to tighten the refusal prompt — smaller models freelance more.

Verification

• •
  Eval set green: 15/15 sourced answers with correct citations, 5/5 refusals on unanswerables.
• •
  Spot-check: click 10 citations from real usage — each cited section actually supports the answer given.
• •
  Staleness warnings fire on a deliberately expired test doc.
• •
  Deflection dashboard distinguishes resolved-by-bot from answered-by-bot; both trend lines visible.
• •
  The bot refuses a question about a system you deliberately excluded from the corpus.

Pitfalls

• Embedding rot instead of fixing it. The corpus audit is the unglamorous half of the build and the reason answers can be trusted.
• Skipping unanswerable evals. Bots tested only on answerable questions look great until the first invented runbook step reaches production.
• General-knowledge leakage. Models "help" by supplementing docs with plausible kubectl/terraform commands; the ONLY-from-context rule plus refusal-string testing catches this.
• Announcing org-wide on day one. Trust lost to early wrong answers does not come back; narrow launch with feedback triage first.
• Measuring replies as deflection. A bot that answers everything badly shows 100% engagement and negative value.

Restricted-environment variant

• •
  Platform docs often contain client architecture, hostnames, and credentials-adjacent detail: self-host end to end (Ollama/vLLM + LiteLLM + pgvector) inside the environment, or use the client's approved enterprise tenancy. Embedding APIs count as data egress too — check them against the MSA, not just the generation calls.
• •
  Per-client corpora: never mix client A's runbooks into client B's retrieval; separate collections with access control mapped to engagement membership.
• •
  Follow-the-sun India teams: the bot carries tribal knowledge across timezone handoffs — night-shift engineers get cited answers instead of waiting for the US team to wake; log those queries as candidate runbook gaps.

Success criteria

• •
  Measured deflection: a meaningful share of platform questions resolved by cited bot answers with no human follow-up, and interrupt load visibly down.
• •
  Zero invented commands/config in a month of logged answers (audited sample).
• •
  Corpus health improves as a side effect: every doc has an owner and a last-verified date, and thumbs-down triage drives real doc fixes weekly.
• •
  Engineers cite the bot's citations — the docs, not the bot, are the authority, and usage reflects that.