Answers are vague, off-topic, or stitched together from unrelated parts of the corpus. Asked about sick leave, the bot talks about travel reimbursement rules. The team's first instinct is "the model is too small" and someone proposes upgrading to a bigger one — which changes nothing, because the new model gets fed the same irrelevant chunks.
A RAG pipeline where nobody prints what retrieval returned. Generation is the visible stage — its output is what users read — so it absorbs all the blame, while the retrieval step runs silently in the middle of the script. Most common in a fresher's first build over a messy or mixed corpus, right after the demo question (which happened to retrieve well) gave everyone false confidence. Linked module: 3.5 (RAG v1 - working pipeline).
A generator can only work with the chunks it receives; if the top-k is noise, the model either improvises or produces mush, and no model upgrade fixes that. The retrieval log is the boundary evidence that splits the system in half. Print, for each failing question: the query, the top-k chunk ids, their scores, and a preview of each chunk. Then read it. Low, nearly tied scores with off-topic chunks mean retrieval failed — the generator never saw the answer. From there, walk upstream in order: chunking (giant or mid-sentence chunks — WR-032's signatures), query wording versus corpus wording, whether the answer exists in the corpus at all, and whether the right document was even loaded. If the log instead shows the correct chunk arriving with a healthy score, only then is generation the suspect — that case is WR-034.
Fix upstream first. Re-chunk along document structure and re-embed, then test retrieval alone — no generation — with five questions whose answers you can point to in the source. Iterate until the correct chunk appears in the top-3 for at least four of the five. Only then reconnect generation and re-test the original failing questions. Finally, add a score floor so that when retrieval returns nothing relevant, the pipeline refuses instead of handing the model garbage to improvise from.
Log query, top-k ids, scores, and chunk previews for every request from day one — it is four lines of code and it converts every future "the bot is broken" report into a two-minute diagnosis. Keep a small golden set of question-to-chunk pairs and run it after any change to chunking, embedding model, or corpus. Debug the pipeline in stage order, upstream first; the model is the last suspect, not the first.
Q: "Users say your RAG bot gives nonsense answers. Walk me through your first ten minutes." Strong answer: pull the retrieval logs for the failing queries before touching anything else, because the log splits the system in half — noise in the top-k means an upstream problem (chunking, query mismatch, missing content), while healthy retrieval points at prompt or model. Bonus for saying a model cannot answer from evidence it never received, and for testing retrieval in isolation against known-answer questions.