Build Similar-Incident Retrieval: RAG over Your Incident History

Tier: Core · Hands-on time: ~60 minutes

This playbook builds retrieval-augmented lookup over your incident history: embed past postmortems and incident summaries, retrieve the most similar at triage time with recency and relevance ranking, and inject them into your triage/investigation prompts (pb-14/pb-15). Commercial equivalents exist — PagerDuty's SRE Agent "learns from past incidents" via shared agent memory, incident.io leans on its Catalog — but a pgvector table and fifty lines of Python teach you exactly what those products do and where similarity search misleads.

When to use

• •
  Your org has 30+ postmortems/incident writeups and repeat failure modes (cert expiry, connection-pool exhaustion, quota cliffs) that different on-callers rediscover from scratch.
• •
  During triage, to attach "past incidents like this one" context to the bundle before prompting.
• •
  Building institutional memory across a services company where engineers rotate between client accounts.

Not worth it below ~20 documents — grep and a well-named postmortem folder outperform embeddings at that scale.

Prerequisites

• •
  Incident corpus as Markdown/text: postmortems, or AI-generated summaries of incident channels (pb-19's factory output is ideal feedstock). Each document needs metadata: date, services involved, severity, root-cause tags.
• •
  Postgres with pgvector (Docker: pgvector/pgvector:pg16), or any vector store you already run.
• •
  An embedding model: hosted API, or a local sentence-transformer / Ollama embedding model for the restricted path.
• •
  Python (or TS) with your LLM SDK; Common Core RAG playbook completed is assumed.

The workflow

1. 1.

   Normalize the corpus into retrieval units. Chunk each postmortem into sections (summary, timeline, root cause, action items) rather than fixed windows — a hit on "root cause" text is far more diagnostic than a hit in a timeline. Store one row per section: (incident_id, section, text, embedding vector, date, services text[], severity).

2. 2.

   Embed and index. Batch-embed sections; create the pgvector index (ivfflat/hnsw, cosine distance). Sanity-check dimensionality and that re-embedding the same text yields distance ~0 — a cheap catch for pipeline bugs.

3. 3.

   Retrieve with a composite score, not raw similarity. At query time, embed the new incident's triage summary (alert text + suspected service) and rank with similarity blended with recency decay and a service-match boost — in SQL or in Python:

   Copy

   SELECT incident_id, section, text,
     (1 - (embedding <=> $1)) * exp(-extract(days from now()-date)/365.0)
       * (CASE WHEN services && $2 THEN 1.3 ELSE 1.0 END) AS score
   FROM incident_chunks ORDER BY score DESC LIMIT 8;
   

   Recency matters because your stack changes: a 3-year-old incident about a decommissioned system is similarity-bait.

4. 4.

   Present retrieval as candidates, then let the LLM compare — with citations. Deduplicate to the top 3 incidents and prompt:

   Copy

   Current incident: <triage summary>. Candidate past incidents: <retrieved
   sections with incident IDs>. For each candidate, state: SAME failure mode /
   SAME symptom, DIFFERENT cause / UNRELATED — quoting the retrieved text that
   justifies the judgment. If a past incident matches, list what fixed it then,
   citing its action-item text. Do not use knowledge beyond these excerpts.
   

   The SAME-symptom/DIFFERENT-cause distinction is the entire game — surface-similar incidents with different causes are how retrieval misleads responders.

5. 5.

   Wire it into triage and keep it fed. Add the top match (with its incident ID and match rationale) to your pb-14 bundle as === SIMILAR PAST INCIDENTS ===. Add an ingest step to your postmortem pipeline so every new (verified) postmortem is embedded on merge — a memory that stops learning decays into a museum.

Tool variants

• pgvector + LLM API (lab default): full control, runs anywhere, ~zero cost with local embeddings; the schema above is the whole product.
• PagerDuty SRE Agent shared memory / incident.io Catalog: managed equivalents that accumulate cross-incident context automatically. Evaluate them with this playbook's failure questions: can you see WHY an incident was deemed similar, and can you correct a bad match?
• Existing search stack (Elastic/OpenSearch with kNN): if you already run it for logs, add a postmortem index with dense vectors — one less system, and hybrid BM25+vector search handles exact error strings better than pure embeddings.

Verification

• •
  Retrieval quality check with known ground truth: pick 5 incidents whose true "sibling" incident you know; the sibling must appear in the top 3 for at least 4 of 5 queries. If not, fix chunking/scoring before trusting it in anger.
• •
  Citation audit on the comparison output: every SAME/DIFFERENT judgment quotes retrieved text that actually appears in the source postmortem — open the cited postmortem and confirm; a judgment citing text the model paraphrased into existence gets the whole match discarded.
• •
  The "what fixed it" answer traces to real action items in the cited postmortem, and you check whether those action items were actually completed — recommending a fix that was already applied and didn't stick is a retrieval-specific trap.
• •
  Recency weighting verified: query with a failure mode that occurred both last month and 3 years ago; last month's must rank first.

Pitfalls

• Symptom-level false positives. "High latency on checkout" embeds close to every other latency incident. Section-level chunking plus the SAME-cause/SAME-symptom forced distinction mitigates; blind trust in the top hit does not.
• Stale corpus. Post-migration (K8s move, cloud switch), old incidents mislead more than they help. Tag corpus eras and consider filtering pre-migration incidents for infra-level queries.
• Embedding secrets. Postmortems can contain tokens/customer data that redaction missed; whatever enters the vector DB is retrievable by anyone with query access. Sanitize at ingest, and ACL the retrieval endpoint like you ACL the postmortems themselves.
• Letting retrieval replace investigation. A past incident is a prior, not a diagnosis. It reorders your hypothesis list in pb-15; it never skips the discriminating test.

Restricted-environment variant

• •
  Client incident history is client-confidential: run embeddings locally (sentence-transformers or Ollama nomic-embed-text-class models) and keep pgvector inside the client boundary. Retrieval quality with good local embedding models is entirely adequate for this corpus size.
• •
  Per-client isolation is non-negotiable in services companies: one vector DB (or at minimum one schema + row-level security) per client. Cross-client retrieval — surfacing Client A's outage to Client B's responder — is an MSA breach delivered by SQL.
• •
  The comparison LLM call can also run on a local model; excerpts are short, so 8B-class models handle the SAME/DIFFERENT task well.

Success criteria

• •
  Ground-truth retrieval test passes (4/5 siblings in top 3) and stays passing as new incidents are ingested.
• •
  During a real or drilled triage, the similar-incident block changed the responder's first action at least once — and the citation audit confirmed the match was causal, not cosmetic.
• •
  New postmortems appear in the index within a day of merge without manual steps.
• •
  You can explain to a skeptical teammate exactly why the system surfaced a given incident, down to the section text and score components.