A knowledge base rots in three ways: articles go stale as the estate changes, near-duplicates split search traffic, and two articles quietly give opposite instructions. AI makes all three findable at scale — and reopen-rate data tells you which broken articles are actively hurting users right now.
Tier: Core · Workflow area: Knowledge management · ~35 minutes
This playbook runs a periodic health audit over a full KB export: embeddings to surface duplicates, an LLM pass to spot contradictions and staleness signals, and a join against ticket reopen data to rank which fixes-that-don't-fix deserve attention first.
Export the KB. Full text plus metadata: article ID, title, body, created/updated dates, view count, and linked-ticket IDs if your platform tracks them. CSV or JSON, one row per article.
Find duplicate clusters with embeddings. Locally, so nothing leaves your machine:
from sentence_transformers import SentenceTransformer, util
import csv
model = SentenceTransformer("all-MiniLM-L6-v2")
rows = list(csv.DictReader(open("kb-export.csv")))
texts = [r["title"] + "\n" + r["body"][:2000] for r in rows]
emb = model.encode(texts, convert_to_tensor=True)
pairs = util.paraphrase_mining_embeddings(emb, top_k=5)
for score, i, j in pairs:
if score > 0.80:
print(f"{score:.2f} | {rows[i]['id']} {rows[i]['title']}"
f" <-> {rows[j]['id']} {rows[j]['title']}")
Pairs above ~0.80 are merge candidates; above ~0.92 they are usually true duplicates. Tune against a handful you inspect by hand.
Run the contradiction pass on each cluster. For every candidate pair or cluster, one prompt:
These KB articles cover the same or overlapping topics. Compare them.
Report:
1. Same issue or genuinely different? (one line)
2. Contradictions: any step, setting, path or value where they
disagree — quote both versions exactly.
3. Which is more current/complete, judging only from the text and
updated dates given.
4. Recommendation: merge / retire one (which) / keep both (why).
Article A ([id], updated [date]): [body]
Article B ([id], updated [date]): [body]
Sample output worth acting on:
2. CONTRADICTION: A says "set MaxTokenSize to 48000"; B says
"65535". A step 4 disables the health service; B step 2 says the
health service must stay running.
4. Retire A (updated 2023-02, pre-migration values); fold its
screenshots section into B.
Sweep for staleness signals. Batch articles (10–20 per prompt) and ask the model to flag: references to retired products or versions you name ("Windows 10, Skype for Business, the old VPN"), dead-looking URLs, dates or version strings older than your threshold, and steps referencing menus that no longer exist in current builds you describe. Staleness flags are leads, not verdicts.
Join reopen data — the harm ranking. From ITSM reporting, pull tickets that linked each article and their reopen/reclassification rate. An article whose linked tickets reopen at twice the baseline is a documented fix that does not work — the most damaging article class you have. Rank the fix queue: contradictions on high-traffic articles first, high-reopen articles second, stale low-traffic last.
Assign and time-box the cleanup. Each flagged article gets an owner and one of four verdicts: retire, merge-into, correct, confirm-good. Record the verdict on the article; "confirm-good" resets its review date, so the next audit skips it.
Rerun quarterly and trend it. Duplicate-cluster count, contradiction count, and mean article age should all fall. Publish the three numbers; a KB with falling rot metrics is the credibility foundation for every AI feature you build on it.
A full KB export can contain embedded credentials, internal hostnames and customer references accumulated over years — run the embeddings step locally, and sanitize article bodies before sending any batch to an external assistant.
Embeddings find the duplicates, the LLM finds the contradictions, but reopen-rate data is what turns a hygiene chore into a ranked list of articles that are actively wasting your users' time.