A support engineer asks the internal policy assistant a plain question about leave encashment. The reply is helpful, and it also contains three salary band ranges and the variable pay pool percentage for the financial year, quoted from a finance-only sheet. The engineer screenshots it in a team group. Within a day the whole floor knows what each band pays. The assistant never crashed, never warned anybody, and every word it printed was true. The complaint that reaches you is not "the bot is inaccurate". It is from the finance head, and it is about a confidentiality breach.
Every enterprise assistant built over a shared drive, because shared drives always hold files that most staff may not read: salary sheets, disciplinary registers, penetration test findings, credential runbooks. Fires most often when a pilot built on one open department is pointed at the whole drive "to make it more useful", and when access control is described in the system prompt rather than enforced in the query. On the P3 reference corpus, 6 of 48 documents are restricted across three departments. Linked modules: P3 (Enterprise RAG assistant + eval harness), 3.6 (RAG that survives production), 3.11 (Safety, guardrails, DPDP).
This is an access-control failure wearing the costume of a retrieval quality issue, and the two need completely different responses. A quality issue is a number you improve. A breach is a release you stop.
The mechanism is simple and it is worth saying out loud: the restricted document was in the index, the user's role was not in the query, so the retriever had no reason to skip it. A confidential file is often the most quotable text in the whole drive — dense, specific, full of numbers — so it scores well on questions that merely brush against pay or discipline. On the P3 corpus, a flat top-5 retriever with no role filter pulled at least one restricted document into the context on 27 of 77 gold cases, a leak rate of 0.351. Structure-aware chunking on its own made it worse, not better, moving the rate to 0.442, because tighter chunks made the confidential schedule an even better lexical match. That result matters: a change that improves retrieval can increase a breach rate, and only a separate measurement will tell you.
The second mechanism is the popular non-fix. Teams add a line to the system prompt — "do not reveal confidential documents" — and the leak rate falls without reaching zero. It cannot reach zero, because the text is already in the context window by then, and every defence downstream of retrieval is advice rather than a wall. Confirm your own exposure in one command: run your gold set, and for each case list any retrieved chunk whose access label is not in the asking role's visible set. If that list is not empty, you have the same incident waiting.
Immediate: stop the assistant, do not "improve the prompt" while it is running. Pull the query logs and list every session where a restricted chunk entered a context window, then hand that list to whoever owns the data. That is a disclosure question, not an engineering one, and it belongs with the data owner on day one.
Structural: the filter moves into the retrieval call as a hard predicate. Every chunk
carries an access label at ingestion; every query carries the asking role; the store is
asked only for chunks whose label sits in that role's visible set. On the P3 corpus this
moved the leak rate from 0.351 to 0.000 on the same 77 cases, and left every other
retrieval number untouched — it costs nothing and it is not a trade-off. Add a second
check after retrieval that re-reads the label on each chunk and refuses to build a prompt
if any chunk fails, so that a future refactor that forgets the filter fails loudly instead
of leaking quietly. Then write the test: a role-scoped case set where a single restricted
chunk in any context is a build failure, not a score.
Decide at design time that role is a required argument of the retrieval function, with no default value. A function that can be called without a role will eventually be called without a role. Restricted documents get their label at ingestion from the document header and never from a folder name, because folders get reorganised and labels must not move when they do.
Then keep this out of the average. Access control is a gate, not a metric: the correct target is zero leaks, per release, and a leak rate of "only two percent" is a breach with a decimal point in front of it. Keep a cross-department case class in the gold set — questions phrased by a role that must not see the answer — and report it as pass or fail, never as a percentage next to your quality scores. India's DPDP Act treats salary and disciplinary records as personal data with a purpose limit, so the log of who asked what matters as much as the answer, and it must record the role that asked.
Q: "Your RAG assistant quoted a confidential finance document to a user who should not see it. What is your first move?" Strong answer: separate the two problems in the first sentence — this is an access-control breach, so the release stops and the data owner is told, and only then does the engineering start. Explain the mechanism: the document was in the index and the role was not in the query, so nothing in the retrieval path could skip it. Give the fix as a hard filter inside the retrieval call plus a post-retrieval check that fails the build, and say explicitly why a system-prompt instruction cannot work — by the time the model reads it, the confidential text is already in the context. Strongest answers add the counter-intuitive measurement: better chunking can raise the leak rate, which is why access must be measured separately from quality, and reported as zero or not-zero.