Lab 09 · Incident-history RAG

You will embed 30 postmortems into pgvector, build a similarity query with recency and relevance ranking, and evaluate it against a labelled eval set so you know its recall@k before you ever trust it on-call.

This lab builds the incident-history RAG service that the incident-response module (M3, AI-augmented incident response) introduces; in the guided path it comes in Stage 4 (Institutionalize), after you've built the live triage copilot. The RAG service you build here is a component you could later wire into the triage copilot from Lab 06 and the bounded remediation agent in Lab 10.

Time & tier

Core · ~75 minutes hands-on. Runs at ₹0: pgvector in Docker, a local embedding model, and Claude only for the final answer synthesis (a few paise per query, or skip it entirely for the retrieval-only path).

Setup

Self-contained. You need Docker, Python 3.11+, and optionally an Anthropic API key for answer synthesis.

mkdir incident-rag && cd incident-rag
python -m venv .venv && source .venv/bin/activate
pip install psycopg[binary] sentence-transformers anthropic
docker run -d --name pgvec -e POSTGRES_PASSWORD=pw -p 5432:5432 pgvector/pgvector:pg16

You need the 30 postmortems and a ground-truth eval set. Save this generator as gen_postmortems.py and run it — it produces varied, realistic postmortems across ten root-cause families (pool exhaustion, cert expiry, bad deploy, DNS, disk-full, throttling, config drift, memory leak, queue lag, upstream rate-limiting), three per family, plus evalset.json mapping ten fresh incident descriptions to the postmortems that should be retrieved for each.

import json, random, pathlib
random.seed(42)
OUT = pathlib.Path("postmortems"); OUT.mkdir(exist_ok=True)
SERVICES = ["checkout-api","payments-worker","auth-svc","search-indexer","notify-svc",
            "inventory-db","edge-gateway","reporting-etl"]
CAUSES = [
 ("db-pool","connection pool exhaustion","HikariCP pool capped at 20; a retry storm from {svc} amplified load until every request queued","raised pool to 50, added a circuit breaker on the retry path"),
 ("cert-expiry","TLS certificate expiry","cert-manager renewal failed silently for 30 days; {svc} began rejecting TLS handshakes at expiry","alert on cert age > 75 days; fixed the broken ClusterIssuer"),
 ("bad-deploy","deployment regression","release {ver} changed pagination defaults, driving p99 latency up 8x on {svc}","rolled back to {prev}; added a latency gate to the deploy pipeline"),
 ("dns","DNS resolution failure","CoreDNS pods were OOMKilled during a node drain; NXDOMAIN storms cascaded into {svc}","raised CoreDNS memory limits; deployed NodeLocal DNSCache"),
 ("disk-full","disk exhaustion","debug logging left on after a previous incident filled the log volume on {svc} nodes; kubelet evicted pods","enforced log rotation with 7-day retention; disk alerts at 80%"),
 ("throttling","cloud API throttling","the cluster autoscaler hammered the cloud API after a spot interruption; scale-ups for {svc} were throttled 40 min","added exponential backoff and cached describe calls"),
 ("config-drift","configuration drift","a manual hotfix to the prod ConfigMap was reverted by the GitOps sync loop mid-incident, re-breaking {svc}","made Argo CD the only write path; documented a break-glass procedure"),
 ("memory-leak","memory leak","a client-library upgrade introduced gradual RSS growth in {svc}; pods OOMKilled roughly every 6 hours","pinned the library; added heap-profile diffing to CI"),
 ("queue-lag","queue consumer backlog","a partition rebalance halved effective consumer throughput; lag on the {svc} topic grew past the SLO","scaled consumers; alert on lag growth rate, not absolute lag"),
 ("rate-limit","upstream rate limiting","a third-party API silently reduced our quota; 429s from {svc} cascaded into checkout errors","added a local cache and graceful degradation for the dependency"),
]
DETECT = ["a customer support escalation","the p99 latency SLO burn-rate alert","a synthetic probe failure",
          "an on-call engineer noticing elevated 5xx","the error-budget fast-burn alert"]
eval_map = {}; n = 0
for key, frag, rc_tpl, fix in CAUSES:
    for _ in range(3):
        n += 1; pm = f"PM-{n:03d}"; svc = random.choice(SERVICES)
        ver, prev = f"v2.{random.randint(10,40)}.0", f"v2.{random.randint(1,9)}.0"
        sev = random.choice(["SEV1","SEV2","SEV2","SEV3"]); dur = random.choice([23,41,67,95,142,210])
        date = f"2026-{random.randint(1,6):02d}-{random.randint(1,28):02d}"
        rc = rc_tpl.format(svc=svc, ver=ver, prev=prev)
        (OUT/f"{pm}.md").write_text(
f"""# {pm}: {svc} {frag}
- Date: {date}  Severity: {sev}  Duration: {dur} minutes
- Services: {svc}
- Detected by: {random.choice(DETECT)}
## Impact
{sev} incident: elevated errors/latency on {svc} for {dur} minutes.
## Root cause
{rc.capitalize()}.
## Remediation
Immediate: {fix}. Follow-up: runbook updated; detection gap review scheduled.
""")
        eval_map.setdefault(key, []).append(pm)
QUERIES = {
 "db-pool":"checkout-api requests timing out, DB connection wait time climbing, retries piling up",
 "cert-expiry":"clients failing TLS handshake against edge-gateway, certificate presented is expired",
 "bad-deploy":"p99 latency jumped right after this morning's release of search-indexer",
 "dns":"intermittent NXDOMAIN errors across several services, CoreDNS pods restarting",
 "disk-full":"pods evicted on multiple nodes, node condition DiskPressure, log volume at 100%",
 "throttling":"autoscaler cannot add nodes, cloud provider API returning throttling errors",
 "config-drift":"fix applied on the prod configmap keeps reverting itself minutes later",
 "memory-leak":"notify-svc memory climbing steadily since a dependency bump, periodic OOMKills",
 "queue-lag":"consumer lag on the payments topic growing steadily after a broker rebalance",
 "rate-limit":"spike of 429 responses from a third-party API cascading into checkout failures",
}
pathlib.Path("evalset.json").write_text(json.dumps(
    [{"query": q, "relevant": eval_map[k]} for k, q in QUERIES.items()], indent=2))
print(f"wrote {n} postmortems and {len(QUERIES)} eval queries")

Run python gen_postmortems.py. You should see wrote 30 postmortems and 10 eval queries.

Guided steps

1. 1.

   Create the schema. Store one row per postmortem with its embedding and parsed metadata. Save schema.sql:

   Copy

   CREATE EXTENSION IF NOT EXISTS vector;
   CREATE TABLE postmortems (
     id TEXT PRIMARY KEY, body TEXT, incident_date DATE,
     embedding VECTOR(384));
   

   Apply it: docker exec -i pgvec psql -U postgres -d postgres < schema.sql.

2. 2.

   Embed and load. Create ingest.py:

   Copy

   import glob, re, psycopg
   from sentence_transformers import SentenceTransformer
   model = SentenceTransformer("all-MiniLM-L6-v2")
   conn = psycopg.connect("host=localhost user=postgres password=pw")
   for path in glob.glob("postmortems/*.md"):
       body = open(path).read()
       pm_id = re.search(r"PM-\d+", body).group()
       date = re.search(r"Date: (\d{4}-\d{2}-\d{2})", body).group(1)
       vec = model.encode(body).tolist()
       conn.execute("INSERT INTO postmortems VALUES (%s,%s,%s,%s) "
                    "ON CONFLICT (id) DO NOTHING", (pm_id, body, date, str(vec)))
   conn.commit(); print("ingested")
   

   Run python ingest.py.

3. 3.

   Build retrieval with recency-weighted ranking. Cosine similarity alone treats a two-year-old incident the same as last week's. Blend a small recency bonus. Create retrieve.py:

   Copy

   import sys, psycopg
   from sentence_transformers import SentenceTransformer
   model = SentenceTransformer("all-MiniLM-L6-v2")
   conn = psycopg.connect("host=localhost user=postgres password=pw")
   def search(query, k=3):
       vec = str(model.encode(query).tolist())
       rows = conn.execute(
         "SELECT id, incident_date, 1-(embedding <=> %s) AS sim, "
         "  GREATEST(0, 1 - (CURRENT_DATE - incident_date)/365.0) AS recency "
         "FROM postmortems ORDER BY embedding <=> %s LIMIT 10", (vec, vec)).fetchall()
       # re-rank: 0.85 relevance + 0.15 recency
       ranked = sorted(rows, key=lambda r: 0.85*r[2] + 0.15*r[3], reverse=True)
       return [r[0] for r in ranked[:k]]
   if __name__ == "__main__":
       print(search(sys.argv[1]))
   

   The <=> operator is pgvector's cosine distance; 1 - distance is similarity. Test it: python retrieve.py "TLS handshake failing, expired certificate" should return the three cert-expiry postmortems.

4. 4.

   Evaluate retrieval quality. A retriever you haven't measured is a liability on-call. Create eval.py:

   Copy

   import json
   from retrieve import search
   evalset = json.load(open("evalset.json"))
   tot_recall = tot_p = 0
   for case in evalset:
       got = set(search(case["query"], k=3)); rel = set(case["relevant"])
       hit = got & rel
       recall = len(hit)/len(rel); prec = len(hit)/len(got)
       tot_recall += recall; tot_p += prec
       print(f"{case['query'][:45]:45} recall@3={recall:.2f} prec@3={prec:.2f}")
   n = len(evalset)
   print(f"\nMEAN recall@3={tot_recall/n:.2f}  precision@3={tot_p/n:.2f}")
   

   Run python eval.py. You should see a mean recall@3 around 0.9 — most families cluster cleanly, one or two overlap.

5. 5.

   Add grounded answer synthesis (optional). Feed the retrieved postmortems to Claude to draft a "have we seen this before?" note. Create advise.py:

   Copy

   import sys, anthropic, psycopg
   from retrieve import search
   conn = psycopg.connect("host=localhost user=postgres password=pw")
   ids = search(sys.argv[1])
   docs = "\n---\n".join(conn.execute(
       "SELECT body FROM postmortems WHERE id = ANY(%s)", (ids,)).fetchall()[i][0]
       for i in range(len(ids)))
   msg = anthropic.Anthropic().messages.create(
       model="claude-opus-4-8", max_tokens=600,
       system=("You are an SRE assistant. Using ONLY the past postmortems provided, "
               "say whether this incident resembles a prior one and cite the PM-id. "
               "If none match, say so — do not speculate."),
       messages=[{"role":"user","content":f"Current incident: {sys.argv[1]}\n\nPast:\n{docs}"}])
   print(msg.content[0].text)
   

   The system prompt forces citation and permits "no match" — the two behaviours that keep this trustworthy on-call.

Verification

• •
  python ingest.py loads 30 rows (SELECT count(*) FROM postmortems; returns 30).
• •
  python retrieve.py "expired certificate, TLS handshake failing" returns three cert-expiry postmortems.
• •
  python eval.py prints a per-query table and a mean recall@3 ≥ 0.80.
• •
  The synthesis path cites at least one PM-id and never invents an incident not in the retrieved set.

Completion artifact

The incident-rag/ folder (schema.sql, ingest.py, retrieve.py, eval.py, optional advise.py) plus RESULTS.md recording your mean recall@3 and precision@3 and one sentence on the recency weight's effect. Self-grade (pass at 4/5):

1. 1.
   pgvector schema stores embeddings plus parsed metadata (date), not just text.
2. 2.
   Retrieval blends similarity with a recency signal, not similarity alone.
3. 3.
   Evaluation runs against the labelled set and reports recall@k and precision@k.
4. 4.
   Mean recall@3 ≥ 0.80.
5. 5.
   Synthesis (if built) cites PM-ids and can answer "no prior match".

Stuck?

• could not connect to server: the container needs a few seconds; retry, or docker logs pgvec.
• type "vector" does not exist: you skipped CREATE EXTENSION; re-apply schema.sql.
• First SentenceTransformer(...) is slow: it downloads ~90 MB once, then caches. On a truly offline India path, pre-download on any network then copy ~/.cache/huggingface.
• Recall lower than expected: confirm you re-rank the top-10 by the blended score, not just return the raw top-3 by distance.

Client-safe variant

For services-company engagements where client postmortems cannot leave the client network: everything here already runs locally — pgvector in the client VPC, the embedding model on a client-approved node, and answer synthesis pointed at the client's sanctioned endpoint (Bedrock inside their tenancy, or a local model via LiteLLM's OpenAI-compatible base URL) instead of the public Anthropic API. Redact customer identifiers from postmortems before embedding; record "can this incident text be embedded off-prem?" as an explicit decision in RESULTS.md.

Answer key

The trap is treating similarity as the whole answer. Two families in the generated set overlap semantically — a "retry storm" (db-pool) and a "cascade of 429s" (rate-limit) both mention retries and cascading failure — so a naive top-3-by-distance query occasionally pulls one wrong doc, capping precision below 1.0. That is realistic and correct: the lesson is that you measure it and report it, rather than assuming retrieval is perfect. The recency weight is deliberately small (0.15); crank it to 0.5 and watch older-but-more-relevant incidents fall out of the top-3 — evidence that recency is a tie-breaker, not a ranking primary.