Lab 12 · Platform docs RAG bot

Then evaluate it against 20 questions, five of which are deliberately unanswerable, and score both answer accuracy and refusal correctness.

This is the Module 5 lab on operational knowledge. The DORA "AI-accessible internal data" capability only pays off if the bot refuses to make things up — a platform bot that invents a plat command is worse than no bot, because developers will run it.

Time & tier

Core · ~75 minutes hands-on. Runs at ₹0: pgvector in Docker, a local embedding model, Claude for grounded answers (a few paise per question).

Setup

Self-contained. Docker, Python 3.11+, and an Anthropic API key.

mkdir docs-rag && cd docs-rag
python -m venv .venv && source .venv/bin/activate
pip install psycopg[binary] sentence-transformers anthropic
docker run -d --name pgvec -e POSTGRES_PASSWORD=pw -p 5432:5432 pgvector/pgvector:pg16
export ANTHROPIC_API_KEY="sk-ant-..."

Generate a 12-document IDP doc set plus a 20-question eval set (15 answerable with a known source doc and expected keyword, 5 unanswerable). Save as gen_docs.py:

import json, pathlib
OUT = pathlib.Path("idp-docs"); OUT.mkdir(exist_ok=True)
DOCS = {
"golden-path-new-service":"# New service\nRun `plat create service <name> --template svc-golden-go --team <team>`. Names must be lowercase, max 30 chars, end in -svc or -api. catalog-info.yaml must set spec.owner.",
"provision-postgres":"# Postgres\nRequest with `plat create postgres --size {small|medium|large} --env {staging|prod}`. small=2vCPU/8GiB, medium=4vCPU/16GiB, large=8vCPU/32GiB. Prod DBs need a change ticket and get 7-day PITR.",
"deployments":"# Deploying to prod\nMerging to main deploys to staging. Promote with `plat promote <service> --to prod`, gated on: passing e2e, no active SEV1/SEV2, deploy window 08:00-17:00 IST Mon-Thu. Fridays frozen except break-glass.",
"rollbacks":"# Rollbacks\n`plat rollback <service> --env prod` reverts to the previous Argo CD revision, bypassing the window but paging the owner. DB migrations are NOT rolled back automatically.",
"secrets":"# Secrets\nSecrets live in Vault under secret/<team>/<service>/. Rotation: 90 days API keys, 30 days DB creds. Add with `plat secret set <service> KEY=value --env staging`. kubectl create secret is blocked by Kyverno.",
"ingress-dns":"# Ingress/DNS\nPublic hostnames are <service>.api.example-corp.com; cert-manager issues TLS. Internal use <service>.internal.example-corp.com. Wildcard/apex needs a platform ticket.",
"observability":"# Observability\nGolden-path services ship a Grafana dashboard, RED metrics via the OTel sidecar, and logs to Loki with 14-day retention. SLOs in slo.yaml generate burn-rate alerts at 2%/1h and 5%/6h. Custom metrics prefixed app_.",
"on-call":"# On-call\nSEV1 = customer-facing outage, page immediately. SEV2 = degraded, page business hours. Platform on-call handles cluster-wide issues; service alerts route to the owner via spec.owner.",
"resource-quotas":"# Quotas\nDefault quotas: staging 20 CPU/64 GiB, prod 60 CPU/192 GiB per team. Every container must set requests and limits or the pod is rejected. Guaranteed QoS needs justification.",
"ci-pipeline":"# CI\nStages: lint, unit, build, image scan (Trivy, fails on HIGH/CRITICAL), kubeconform, e2e on staging. Pipelines time out at 25 min. Flaky-test quarantine via .ci/quarantine.yaml with a linked ticket.",
"cost-showback":"# Cost\nOpenCost tags every namespace; monthly showback lands in #team-finops on the 3rd. Idle staging scales to zero nightly 22:00 IST unless annotated platform/keep-warm: true.",
"agent-access":"# AI agent access\nApproved AI assistants reach platform APIs only via the platform MCP server with scoped service accounts. Read by default; writes (restart/scale/promote) need a logged human approval. Agent tokens expire after 8 hours.",
}
for name, body in DOCS.items(): (OUT/f"{name}.md").write_text(body+"\n")
EVAL = [
 ("What command scaffolds a new service?","golden-path-new-service","plat create service"),
 ("What are the naming rules for services?","golden-path-new-service","lowercase"),
 ("How much memory does a medium Postgres have?","provision-postgres","16"),
 ("Do prod databases have point-in-time recovery?","provision-postgres","7-day"),
 ("What gates a promotion to prod?","deployments","e2e"),
 ("Can I deploy to prod on Friday?","deployments","frozen"),
 ("Does rollback also revert DB migrations?","rollbacks","not"),
 ("How do I add a secret for staging?","secrets","plat secret set"),
 ("How often must DB credentials rotate?","secrets","30"),
 ("What hostname do public services get?","ingress-dns","api.example-corp.com"),
 ("What log retention do services get?","observability","14-day"),
 ("When does a SEV2 page?","on-call","business hours"),
 ("What is the prod CPU quota per team?","resource-quotas","60"),
 ("What does the image scan fail on?","ci-pipeline","HIGH"),
 ("When are idle staging environments scaled to zero?","cost-showback","22:00"),
 ("How do I provision a Kafka topic?",None,None),
 ("What is the SSO session timeout for Backstage?",None,None),
 ("Which regions can I deploy prod workloads to?",None,None),
 ("What is the retention period for S3 access logs?",None,None),
 ("How do I request a static egress IP?",None,None),
]
pathlib.Path("docs_evalset.json").write_text(json.dumps(
    [{"q":q,"source":s,"keyword":k,"answerable":s is not None} for q,s,k in EVAL], indent=2))
print(f"wrote {len(DOCS)} docs and {len(EVAL)} eval questions")

Run python gen_docs.py. The five unanswerables (Kafka, SSO timeout, regions, S3 logs, egress IP) are topics a real platform has but this doc set does not — exactly the gaps that tempt a bot into hallucinating.

Guided steps

1. 1.

   Schema and ingest. Reuse the pattern from Lab 09. schema.sql:

   Copy

   CREATE EXTENSION IF NOT EXISTS vector;
   CREATE TABLE docs (name TEXT PRIMARY KEY, body TEXT, embedding VECTOR(384));
   

   Apply it, then ingest.py:

   Copy

   import glob, os, psycopg
   from sentence_transformers import SentenceTransformer
   m = SentenceTransformer("all-MiniLM-L6-v2")
   c = psycopg.connect("host=localhost user=postgres password=pw")
   for p in glob.glob("idp-docs/*.md"):
       body = open(p).read(); name = os.path.basename(p)[:-3]
       c.execute("INSERT INTO docs VALUES (%s,%s,%s) ON CONFLICT DO NOTHING",
                 (name, body, str(m.encode(body).tolist())))
   c.commit(); print("ingested")
   

2. 2.

   Retrieve with a relevance floor. The key to refusal-on-unknown is a similarity threshold: if the best doc isn't similar enough, retrieval returns nothing and the bot must say it doesn't know. Create bot.py:

   Copy

   import psycopg, anthropic
   from sentence_transformers import SentenceTransformer
   m = SentenceTransformer("all-MiniLM-L6-v2")
   c = psycopg.connect("host=localhost user=postgres password=pw")
   client = anthropic.Anthropic()
   FLOOR = 0.25  # minimum cosine similarity to consider a doc relevant
   SYSTEM = ("You answer questions about our internal platform using ONLY the provided "
             "docs. Cite the doc name in [brackets] for every claim. If the docs do not "
             "contain the answer, reply EXACTLY: 'I don't know — this isn't covered in the "
             "platform docs.' Never guess a command, size, or policy.")
   def ask(q):
       v = str(m.encode(q).tolist())
       rows = c.execute("SELECT name, body, 1-(embedding <=> %s) AS sim "
                        "FROM docs ORDER BY embedding <=> %s LIMIT 3", (v, v)).fetchall()
       rows = [r for r in rows if r[2] >= FLOOR]
       if not rows:
           return "I don't know — this isn't covered in the platform docs."
       ctx = "\n---\n".join(f"[{r[0]}]\n{r[1]}" for r in rows)
       return client.messages.create(model="claude-opus-4-8", max_tokens=400,
           system=SYSTEM,
           messages=[{"role":"user","content":f"Docs:\n{ctx}\n\nQuestion: {q}"}]
           ).content[0].text
   

   The relevance floor is a cheap, deterministic first line of defence; the system prompt's explicit refusal instruction is the second. You want both — the floor catches truly off-topic questions before spending a token, the prompt catches near-miss questions where a doc was retrieved but doesn't actually answer.

3. 3.

   Evaluate accuracy and refusal correctness together. Create eval.py:

   Copy

   import json
   from bot import ask
   ev = json.load(open("docs_evalset.json"))
   ans_ok = ref_ok = ans_n = ref_n = 0
   for case in ev:
       a = ask(case["q"]); refused = "I don't know" in a
       if case["answerable"]:
           ans_n += 1
           ok = (not refused) and case["keyword"].lower() in a.lower()
           ans_ok += ok
           print(f"[ANS ] {'OK ' if ok else 'MISS'} {case['q'][:50]}")
       else:
           ref_n += 1
           ref_ok += refused
           print(f"[REF ] {'OK ' if refused else 'HALLUC'} {case['q'][:50]}")
   print(f"\nAnswer accuracy: {ans_ok}/{ans_n}   Refusal correctness: {ref_ok}/{ref_n}")
   

   Run python eval.py. Target: answer accuracy ≥ 13/15 and refusal correctness 5/5. A single hallucinated answer to an unanswerable question is a failure that matters more than a missed answerable one.

Verification

• •
  ingest.py loads 12 docs.
• •
  python -c "from bot import ask; print(ask('How do I provision a Kafka topic?'))" returns the exact "I don't know" refusal.
• •
  eval.py prints per-question ANS/REF results plus totals.
• •
  Answer accuracy ≥ 13/15 and refusal correctness = 5/5.
• •
  Every answered question's response cites at least one doc name in brackets.

Completion artifact

The docs-rag/ folder plus EVAL-RESULTS.md recording answer accuracy, refusal correctness, and the relevance-floor value you settled on. Self-grade (pass at 4/5):

1. 1.
   Retrieval applies a similarity floor so off-topic questions retrieve nothing.
2. 2.
   Answers cite doc names; the system prompt forbids guessing.
3. 3.
   Eval scores answer accuracy and refusal correctness separately.
4. 4.
   Refusal correctness = 5/5 (no hallucinated answers to unanswerables).
5. 5.
   Answer accuracy ≥ 13/15.

Stuck?

• Bot answers a Kafka question by extrapolating from the Postgres doc: lower FLOOR isn't the fix — raise it so the weakly-related Postgres doc doesn't clear the bar, and strengthen the prompt's "never guess" clause.
• Refuses an answerable question: your floor is too high; the Postgres/quota docs sit around 0.3–0.5 similarity for on-topic questions, so 0.25 is a safe floor. Print the similarities to calibrate.
• Keyword-match eval marks a correct answer as MISS: the model paraphrased (e.g. "sixteen" vs "16"); this is an eval-brittleness lesson — note it, and prefer semantic checks for production evals.

Client-safe variant

Client platform docs are among the most sensitive things a services team touches — they encode the client's internal architecture. Embed and store them inside the client VPC, run the embedding model on a client node, and point answer synthesis at the client's sanctioned endpoint. The refusal behaviour is doubly important here: a bot that invents a client-specific command exposes the services team to a real incident. Add the client's own PII/redaction rules to the ingest step and record the relevance-floor and refusal-rate results as part of the engagement's AI-usage evidence.

Answer key

Two traps. First, the relevance floor and the prompt refusal are both required and catch different failures: without the floor, an off-topic question ("static egress IP") still retrieves the three nearest docs and hands them to the model, which may pattern-match "IP" to the ingress doc and confabulate; without the prompt instruction, a near-miss question ("SSO timeout") retrieves the agent-access doc — genuinely related, above the floor — and the model must recognise it doesn't actually answer the question. Removing either guard reliably produces a hallucination on the eval set. Second, the keyword-match eval is intentionally brittle: a model that answers "sixteen gibibytes" is correct but scored MISS by a literal "16" check. That's a teachable flaw — production docs-bot evals should use an LLM-judge or semantic match, and you should note in EVAL-RESULTS.md which "misses" were real versus eval artefacts.