The bot answers with confidence and a citation — for a rule that no longer exists. HR cut the notice period from 60 days to 30 last month, the current file says 30 everywhere, yet the bot still says "60 days [C11]". Opening the retrieved chunk makes it stranger: the chunk text does not match what the document says today.
A RAG script that was built once and kept running. The embeddings were computed
in an earlier session — saved to disk with np.save, pickled, or sitting in a
long-lived notebook variable — and the source document was edited afterwards.
Nobody re-ran the indexing stages, because nothing crashed and nothing looked
wrong. Common wherever the "build index" and "answer questions" code paths were
separated for speed. Linked module: 3.5 (RAG v1 - working pipeline).
Two stores have drifted apart: the text file on disk is current, but the chunk
list and its embeddings are a frozen snapshot from index-build time. Retrieval
and generation are both working correctly — over stale data — which is why no
error appears anywhere. Confirm in two minutes: print the retrieved chunk's
text and search for it in the current document; if it is missing or different,
the index is stale. A second probe: compare the document's modified time
(os.path.getmtime) with the index file's creation time — an index older than
its source is the smoking gun. Note what this is not: not a model problem, not
a chunking problem, not WR-034's grounding problem. The bot was perfectly
grounded — in last month's document.
Rebuild the derived data: re-run load, chunk, and embed on the current file, then re-ask the failing question and watch the new answer cite the new rule. Then remove the trap permanently: store the source file's hash (or modified time) alongside the saved index, and on every startup compare it with the current file — rebuild automatically when they differ, load the cache only when they match. Ten lines of code, and the failure class disappears.
Treat the index as derived data, never as a source of truth — like a compiled file, it must be rebuilt when the source changes. Save index metadata from day one: source file hash, build timestamp, and embedding model name (the model name also guards against WR-031's mixed-model trap). Keep one golden question whose answer changed in a recent document edit; if the bot ever gives the old answer, staleness is back. In any pipeline that updates documents on a schedule, make re-indexing part of that same schedule.
Q: "A user proves the bot's cited answer is not in the document, but your retrieval log and prompt both look correct. What do you check next?" Strong answer: index freshness — the chunks and embeddings are a snapshot, so compare the index build time against the document's modified time and diff the retrieved chunk against the current file. Close with the durable fix: hash the source into the index metadata and auto-rebuild on mismatch. Bonus for the phrase "the index is derived data" — it signals a data-pipeline mindset.