The RAG bot's answers are fluent and confident, sometimes even carrying citation markers — and wrong. The document says 12 days of casual leave; the bot says 15. The document says a 30-day notice period; the bot says "typically 60 to 90 days". Users who know the policy stop trusting the bot within a day, because it sounds most certain exactly when it is inventing.
A fresher's first RAG build, usually right after wiring retrieval to generation. Retrieval itself is healthy — the log shows the right chunks arriving with good scores. The prompt was assembled the quick way: chunks pasted in with no labels, no instruction about how to use them, question first or last at random. Small local models make it worse, but hosted models do it too. Linked module: 3.5 (RAG v1 - working pipeline).
The model treats the provided context as a suggestion, not as the source of truth. Given nothing that says otherwise, it blends the chunks with its training memory — and for common topics like leave policies, its memory of the internet's "typical" answer often wins. Confirm it in two steps. First, read the retrieval log for the failing question: if the correct chunk is present with a clear score, retrieval is not the suspect. Second, run a trap question — one where your document's answer differs from the world's default (an unusual leave count, an odd notice period). If the bot returns the internet-common number instead of your document's number, it is answering from memory. Common root causes: no grounding rule in the system prompt, unlabelled context the model cannot cite, and the context dumped after the question where attention fades.
Restructure the prompt, then verify. System rule first: "Answer using ONLY the context below. After each fact, add its chunk marker. If the context does not contain the answer, say: I don't know - this is not in the document." Label every chunk with an id like [C4], place the full context before the question, and keep top-k small so the relevant chunk is not buried. Then add a citation check to your test loop: for each answer, open the cited chunk and confirm the claim actually appears in it. Re-run the trap questions — your document's numbers should now win, with markers pointing at the right chunks.
Keep three to five trap questions where your document disagrees with the world's default, and run them after every prompt, model, or k change. Log the fully assembled prompt for every request, so "what did the model actually see" is a file read, not a guess. Treat any uncited claim in an answer as a failure even when it happens to be true — the habit you are protecting is checkability, not luck.
Q: "Your RAG system has perfect retrieval and still gives answers that contradict the source document. What is happening?" Strong answer: name the conflict between training memory and provided context, describe the trap-question probe (a fact where the document differs from the internet's default), and give the two-part exit — a grounding rule with labelled chunks in the prompt, plus a per-claim citation check in testing. Bonus for noting the model sounds most confident precisely when ungrounded.