An employee asks the internal policy assistant how much notice they must serve. The assistant answers "15 days", names a real circular, and sounds certain. The employee resigns on that basis and books a joining date with the next employer. HR then confirms the notice period is 60 days, and the assistant was quoting Circular 2022/07, a relaxation that the 2025 employment policy replaced. Nobody typed a wrong answer anywhere. The assistant retrieved a genuine company document, quoted it correctly, and was still wrong.
Any retrieval system over a shared drive that keeps its own history — which is every HR, finance and compliance drive in existence. Fires hardest where documents carry a version number in the filename rather than in the metadata, where withdrawn circulars are kept "for audit" in the same folder as live policy, and where the effective date sits inside the body text instead of in a field the retriever can read. On the P3 reference corpus there are 8 version pairs and 6 withdrawn circulars, all indexed together. Linked modules: P3 (Enterprise RAG assistant + eval harness), 3.6 (RAG that survives production), 3.7 (Evaluating LLM systems).
Similarity search has no idea what "current" means. A v1 chunk and a v2 chunk say almost the same words about the same rule, so both score highly and their order is decided by noise — chunk length, a stray heading, the exact phrasing of the question. On the P3 corpus, a plain top-5 lexical retriever placed the current version above the old one on only 6 of 16 superseded questions, a rate of 0.375. That is barely better than a coin toss, and a coin toss is what it is: nothing in the ranking function knows about time.
Three separate mistakes stack up to produce the incident. First, the effective date and the version were never attached to the chunk as fields, so no ranker could use them even if it wanted to. Second, the withdrawn circular was indexed exactly like live policy, with no flag saying it had been superseded. Third, the answer template said "according to company policy" instead of naming the document and its version, so the reader had no way to spot the problem before acting on it.
Confirm it in ten minutes. Take every question whose answer changed between versions, run retrieval, and print the top five chunks with their document id, version and effective date. Two things will be visible at once: how often both versions appear together, and how often the older one sits higher. Then read the answer text and check whether it names a version at all. If it does not, this failure has probably already happened to you and nobody reported it, because a confident wrong answer produces no error and no log line.
Immediate: recall the wrong guidance in writing and tell the affected employee before
anything else. Then rebuild the index with version, effective_date and superseded_by
as real metadata fields on every chunk, taken from the document header rather than guessed
from the filename.
Structural, in three parts. Add a recency preference to the ranker: when two chunks
describe the same rule, the one with the later effective date wins, and a chunk whose
superseded_by field is filled cannot outrank its own replacement. On the P3 corpus that
one change moved version preference from 0.375 to 0.938 on the same 16 cases, with no
change to the retriever itself. Next, make the answer cite the document, the version and
the effective date in the sentence a reader will read — "HR-EMPLOY v2, effective
2025-04-01, says 60 days" — because a citation the reader can check is what turns a silent
failure into a visible one. Last, when both versions are retrieved and they disagree,
answer from the current one and say plainly that an older document exists and no longer
applies. Hiding the conflict is not the same as resolving it.
Never solve this with a prompt line that says "prefer recent documents". The model cannot see a date you did not attach, and an instruction is not a mechanism.
Treat supersession as an ingestion problem, not a generation problem. Every document that enters the index carries an effective date and a superseded-by field on the day it lands, and the ingestion step refuses a document that has neither. Keep withdrawn circulars if audit requires them, but mark them at ingestion and exclude them from the default search scope, so retrieving one is a deliberate act.
Then measure it forever. A gold set that contains only questions with one stable answer cannot see this failure, so keep a superseded case class: questions whose answer changed between versions, with the expected document AND version written down. Report that class separately in every eval run. On the reference corpus it is the hardest class of all — the old version competes for the same shortlist slots — and a project whose superseded class is not reported separately has not been tested for the failure that actually costs people their notice pay.
Q: "Your RAG assistant quoted a real company document and still gave the wrong answer. Explain how that happens and what you would change." Strong answer: say first that similarity does not encode time, so a v1 chunk and a v2 chunk are near-identical to the retriever and their order is decided by noise. Give the mechanism in three parts — version and effective date attached as metadata at ingestion, a recency preference in the ranker so a superseded chunk can never outrank its replacement, and the version named in the answer so a reader can catch it. Quote a number if you have one from your own project. Close with the measurement: a superseded case class in the gold set, reported separately, because an aggregate score hides exactly this. If you finish by saying that a prompt instruction like "prefer newer documents" is not a fix, you have shown you know the difference between a mechanism and a wish.