The bot that answered in a few seconds during the pilot now takes fifteen to twenty seconds per question, and users have stopped waiting. On a laptop the fan spins up on every query. Nothing errors; nothing in the logs says slow. The team starts talking about buying a GPU or a faster vector database.
A RAG pipeline that grew on three axes at once: more documents in the store, a larger k ("retrieve 10 to be safe"), and longer chunks after someone widened the chunker "so no rule gets cut". Typical a month or two into adoption, once real traffic and real corpus size replace demo conditions. The proposed fixes on the table are all hardware; nobody has measured where the time goes. Linked module: 3.6 (RAG that survives production).
Measure before believing anyone — wrap each stage in time.perf_counter():
query embedding, store query, generation. On a local setup the numbers usually
land near: embedding the question ~80 ms, the vector store query ~15 ms even at
thousands of chunks, generation 14+ seconds. The store is almost never the
culprit at this scale; the prompt is. Generation time on a CPU-bound model
scales with the tokens it must read, and k=10 long chunks means the model chews
through thousands of context tokens before writing word one. The cost lives in
what you send, not in how you search. A second multiplier hides in traffic
shape: real users ask the same top questions repeatedly, and the pipeline pays
full price every time. Confirm by comparing per-stage timings at k=3 versus
k=10 — if generation dominates and grows with k, this is your failure.
Cheapest lever first. (1) Retrieve less: drop k back to 3 with good filters, and cap chunk length — a well-filtered top-3 beats an unfiltered top-10 on both speed and answer quality. (2) Cache finished answers keyed by the question for the repeat-heavy head of traffic; even an exact-string dict removes whole seconds for the most common questions, and a rephrase-tolerant cache (match on embedding similarity to past questions) catches more. (3) Where a wide retrieval is genuinely needed, rerank the shortlist and send only the best few chunks to the model — spend milliseconds on ordering to save seconds on reading. Re-measure after each lever; stop when the budget is met. Hardware is the last resort, not the first.
Log per-stage timing from day one, so growth shows up as a trend instead of a surprise. Treat k and chunk size as budget items that need justification, not safety margins to inflate. Put a latency check beside your quality eval — a regression in seconds is a regression, same as a wrong answer. Review the cache hit rate monthly; a falling rate means the question mix changed.
Q: "Your RAG endpoint got slow as the corpus grew. Walk me through your diagnosis and first three fixes." Strong answer: instrument stages before proposing anything, name generation-reads-the-prompt as the usual dominant cost, then order the levers — retrieve less and shorter, cache repeats, rerank to shrink the prompt — and explicitly defer hardware until measurements prove the cheap levers are exhausted.