The RAG bot keeps giving wrong or vague answers. The team responds where the text is: fourteen prompt rewrites, two model swaps, longer rules, more examples — and the end-to-end score barely moves. Every change helps one failing question and hurts another. Morale drops; someone proposes fine-tuning.
A pipeline where quality is tracked as one blended number: the share of questions answered correctly end to end. Nobody measures retrieval on its own, so nobody knows whether the right chunks ever reach the model. Common when the team formed around prompt engineering skills and thinks of the system as "the prompt", with retrieval as invisible plumbing. Linked module: 3.7 (Evaluating LLM systems); the sibling failure at debug time is WR-035.
A blended score cannot say which stage failed, and the team spent three weeks tuning the visible stage instead of the broken one. The instrument that settles it in minutes is stage separation: for every eval case, record the expected chunk id, then measure hit rate — the share of cases whose correct chunk lands in the top-k — with no generator involved. In the real incident behind this entry, hit rate came back 5/12: the model never received the right text for most questions, so no prompt could have fixed the answers. The matching check on the other stage is groundedness — whether the answer's claims are supported by whatever was retrieved — which had been fine all along.
Freeze the prompt. Fix retrieval as its own project: re-chunk along document structure, check query wording against corpus wording, re-embed, and rerun the hit-rate check alone until it clears a bar you set in advance (say 10/12). Only then reconnect generation and re-score end to end. In the source incident the original prompt — version one, untouched — passed almost everything once retrieval was healthy: three weeks of prompt work had been solving the wrong problem.
Every RAG eval run reports two numbers, never one: retrieval hit rate and answer groundedness. Store the expected chunk id in every eval case from day one so hit rate is always five lines of code away. Debug in stage order, upstream first, and let the numbers pick the suspect before any tuning starts. Re-measure hit rate after anything that touches the index — new chunking, new embedding model, edited documents — because those changes reset retrieval quality silently.
Q: "Your RAG system's accuracy is stuck despite weeks of prompt work. What do you check?" Strong answer: split the pipeline — measure retrieval hit rate against expected chunks separately from answer groundedness — and state the rule that a generator cannot answer from evidence it never received. Bonus for describing the eval-case design that makes this possible (expected chunk id per case) and for naming the pre-set bar retrieval must clear before any prompt tuning resumes.