A user acted on the bot's advice and got hurt — financially, medically, or legally. The transcript shows a fluent, specific, confident answer in a regulated area: which medicine to take, whether to close a loan early, how to respond to a tax notice. No disclaimer, no refusal, no human in the path. The answer may even cite a retrieved chunk, but the harmful claim itself is not in the chunk — it was generated to fill the gap, in the same confident voice.
Any assistant whose corpus touches a regulated domain: bank and loan FAQs, insurance policies, hospital information pages, tax and legal help desks. The trap is gradual — the bot was built for policy questions ("what documents does KYC need"), works well, and then real users ask the next question ("should I break my fixed deposit to prepay this loan?"). Modules 3.5 and 3.6 taught the "I don't know" path for missing facts; this failure is different, because the facts were present and the model went beyond them into advice. Linked module: 3.11 (Safety, guardrails, DPDP).
Three gates were missing at once, and all three live outside the model. No domain gate: nothing classified the question as regulated-advice and routed it differently from a policy lookup. No refusal rule: the system prompt never drew the line between quoting a document and advising a person. No human review: the answer reached the user without any qualified check. Confirm it from the transcript the way module 3.5's grounding audit taught: put the answer beside its retrieved chunks and mark every claim. The chunks state facts; the harmful sentence is a recommendation that appears in none of them — a hallucination wearing a citation. Then check the traffic: count how many recent questions were advice-seeking rather than fact-seeking. That number is usually the surprise that ends the debate.
Build the domain gate first: classify incoming questions (keyword rules reach far, a small classifier reaches further) so anything medical, financial, or legal takes a separate path. On that path, three honest outcomes replace the generated verdict: answer with document facts plus a clear disclaimer, refuse with a route to a licensed human, or queue the draft for human review before sending. Add refusal rules to the system prompt — quote the document, never recommend a personal action — and back them with an output check in code that scans drafts for advice-shaped phrasing ("you should", dosage numbers, buy/sell wording), because module 3.11's rule holds here: the prompt lowers the odds, the code check catches the remainder. Rerun the incident question; the correct output is facts plus disclaimer, or a referral — never a verdict.
Decide at design time which question classes the bot may answer alone, which need disclaimers, and which always need a human — write it down and test it. Put advice-seeking questions in the golden set from module 3.7, so a regression run fails the moment a model or prompt change starts generating verdicts again. Review a sample of real transcripts weekly; users drift into advice territory long before dashboards show it. And keep the honest-refusal habit from module 3.5: in regulated flows, "please speak to a licensed advisor" is a correct answer, not a failure.
Q: "Your bank's FAQ bot told a customer to prepay their loan, and the customer lost money. The retrieval was accurate. What went wrong and what do you build?" Strong answer: separate facts from advice — the chunks held facts, the recommendation was generated, so this is a hallucinated verdict shipped without gates. Name the three missing controls: a domain gate classifying advice-seeking questions, refusal rules plus disclaimer paths, and human review on regulated answers. Strengthen it with testing: advice questions in the golden set, output checks for advice-shaped phrasing. Bonus: note that "talk to a licensed advisor" is a success case in a regulated flow, and that the same altitude applies to medical and legal domains.