RAG v1 - working pipeline
Tuesday afternoon, Nagpur. Tanvi runs HR operations at a logistics company with nine hundred drivers and staff. The same questions arrive all day: how many casual leaves, what counts as night allowance, when does notice period start. Her intern wired a chatbot to a model last month. It answered fast, and it invented a maternity rule that does not exist in the handbook. Her manager's order is blunt: every answer must quote the policy, or the bot goes. Today you build the version that quotes.
- "Chat with our documents" is the most-requested GenAI feature in Indian companies, and this module is that feature end to end.
- You build every moving part in plain Python, so when a framework hides these parts later, you can still debug them.
- Interviewers ask "what happens when the answer is not in the documents" — today you build the honest path they want to hear.
Module 3.4 ended one step short of the goal. Your search could find the right chunk, but a person still had to read it. RAG — retrieval-augmented generation — is a pipeline that finds the right chunks first, then has the model answer from them. The model never met your HR policy in training, and training it again for every policy edit would burn money. Retrieval is the cheaper road: keep the documents outside the model and fetch the right piece at question time.
The pipeline has five stages: load, chunk, embed, retrieve, generate. The first three run once, before any user shows up: read the document, cut it into chunks, and turn each chunk into a vector. The last two run for every question. Retrieval — searching the stored vectors for the chunks closest in meaning to a question — is the brute-force scan you built in 3.4. The winners are called the top-k — the k best-scoring chunks from a search — and k stays small, often three.
There is a wrong way and a right way to give a model your documents. The wrong way stuffs the whole handbook into every prompt. It strains the context window from 3.1, pays for thousands of unneeded tokens per question, and buries the one relevant paragraph under eighty pages of noise. The right way retrieves first and sends only the top-k chunks. The model reads a short, relevant prompt, and the cost stays flat whether the handbook has ten pages or ten thousand.
The prompt you assemble decides whether the answer can be trusted. Each retrieved chunk enters with a label like [C4], then the question, then one strict rule: answer only from this context. That rule enforces grounding — keeping every claim in the answer tied to the retrieved text. A citation — a marker naming which chunk a fact came from — lets anyone verify a claim in seconds. Without both, the model falls back on its training memory, and its answer can contradict the very document you gave it.
Sometimes the document holds no answer, and the pipeline must say so. The tell is in the retrieval scores: when the best score sits low and the top three nearly tie, nothing relevant was found. The honest response is a refusal — a plain statement that the document does not cover this. That habit sets up your first quality measure: for every answer, check whether each claim appears in the retrieved chunks. An answer whose claims all trace back is grounded; one that adds facts from nowhere is a hallucination in a confident voice.
Everything below runs offline on the Ollama setup from modules 3.1 and 3.4 — nomic-embed-text for embeddings, llama3.2:1b for generation. We grow one script, rag.py, five stages in five steps. First, the document: create hr-policy.txt yourself — twelve to fifteen short paragraphs of an imaginary company's HR policy (casual leave, sick leave, work hours, night allowance, notice period, travel reimbursement), roughly 6,000 to 8,000 characters, one blank line between paragraphs. Writing it yourself matters: you can only judge groundedness against a document you know completely.
Step 1 — load and chunk. Split on paragraphs, then merge tiny ones so every chunk can stand alone (the WR-032 lesson from module 3.4):
import requests
import numpy as np
policy = open("hr-policy.txt").read()
paras = [p.strip() for p in policy.split("\n\n") if p.strip()]
chunks, buf = [], ""
for p in paras:
buf = (buf + "\n" + p).strip()
if len(buf) >= 300:
chunks.append(buf)
buf = ""
if buf:
chunks.append(buf)
print(len(chunks), "chunks")
print("C0:", chunks[0][:60], "...")
Output: 14 chunks then C0: Casual leave: every employee earns 12 days of casual ... (your numbers and text will differ). The merge loop respects paragraph boundaries instead of slicing mid-sentence, and 300+ characters keeps each chunk meaningful on its own. The C0 label is not decoration — chunk ids become our citation markers later, so print them from the start.
Step 2 — embed and store. The "index" in RAG v1 is nothing fancier than the chunk list plus one normalised NumPy array — the exact setup from module 3.4:
def embed(texts):
r = requests.post("http://localhost:11434/api/embed",
json={"model": "nomic-embed-text", "input": texts},
timeout=120)
r.raise_for_status()
return np.array(r.json()["embeddings"])
index = embed(chunks)
index = index / np.linalg.norm(index, axis=1, keepdims=True)
print(index.shape)
Output: (14, 768) — one 768-number vector per chunk, all normalised so a plain dot product gives cosine similarity. This is the last offline stage. If you edit hr-policy.txt after this point, you must re-run steps 1 and 2, or you will be searching a snapshot of a document that no longer exists — that trap has its own War Room entry, WR-036.
Step 3 — retrieve top-3, and always print the log. The online half begins here — the question enters the pipeline at this stage:
def retrieve(question, k=3):
q = embed([question])[0]
q = q / np.linalg.norm(q)
scores = index @ q
top = np.argsort(scores)[::-1][:k]
return [(int(i), float(scores[i])) for i in top]
for i, s in retrieve("How many casual leaves do I get in a year?"):
print(f"C{i:<2d} score={s:.2f} {chunks[i][:48]}...")
Output:
C0 score=0.72 Casual leave: every employee earns 12 days of ca...
C3 score=0.58 Sick leave beyond two days needs a doctor's cert...
C11 score=0.35 Exit process: the notice period for confirmed st...
This printout is a retrieval log: the query's top-k chunk ids, their scores, and a preview of each chunk. Read it before you ever look at a generated answer. Here the log is healthy — the casual-leave chunk leads clearly, the runners-up trail off. When answers go wrong later, this log tells you in one glance whether retrieval or generation is to blame, and that single habit is WR-035's entire fix.
Step 4 — assemble the prompt and generate. Labelled context first, one strict rule, then the question — sent to the chat endpoint from module 3.2:
CHAT = "http://localhost:11434/api/chat"
RULES = ("Answer using ONLY the context below. After each fact, add its "
"chunk marker, like [C4]. If the context does not contain the "
"answer, reply exactly: I don't know - this is not in the document.")
def ask(question, k=3, floor=0.45):
hits = retrieve(question, k)
for i, s in hits:
print(f" log: C{i} {s:.2f}")
if hits[0][1] < floor:
return "I don't know - this is not in the document."
context = "\n\n".join(f"[C{i}] {chunks[i]}" for i, _ in hits)
msgs = [{"role": "system", "content": RULES},
{"role": "user",
"content": f"Context:\n{context}\n\nQuestion: {question}"}]
r = requests.post(CHAT, json={"model": "llama3.2:1b", "messages": msgs,
"stream": False}, timeout=120)
r.raise_for_status()
return r.json()["message"]["content"]
print(ask("How many casual leaves do I get in a year?"))
Output: the three log lines, then something like You earn 12 days of casual leave per year [C0]. Trace what the model received: not the handbook, not its training memory — three labelled chunks and a rule. The [C0] marker is the citation, and it is checkable: open chunk 0 and the claim is there. The floor check sits before generation on purpose — when retrieval finds nothing worth reading, the cheapest correct move is to skip the model call entirely.
Step 5 — the out-of-scope question. Ask something your policy genuinely does not cover:
print(ask("What is the office wifi password?"))
Output: three log lines like log: C7 0.31, log: C2 0.29, log: C12 0.27, then I don't know - this is not in the document. Low, nearly tied scores are the signature of "nothing relevant exists" — the same flat-band pattern you met in 3.4, now put to work. One warning from that module still applies: 0.45 is not a magic number. Scores are model- and corpus-relative, so the floor must be tuned on your own document — exercise 4 does exactly that.
Work inside your ai-lab environment with Ollama running and both nomic-embed-text and llama3.2:1b pulled. Each exercise ends with a check — you are done only when the check passes.
RAG over a document you choose. Pick a real document you know well — your college rulebook, hostel rules, a society's bye-laws, or your own class notes — and save it as plain text, at least 8,000 characters. Rebuild the five-stage script as ragme.py over it: paragraph-aware chunking, embed and normalise, retrieve(), ask() with the grounding rule, citation markers, and the floor. Ask five questions in your own words, phrased differently from the document's wording. Check: every answer carries at least one [C#] marker, and for each one you opened the cited chunk and confirmed the claim is actually in it; the retrieval log prints before every answer.
The grounding audit. Write ten questions for your document: seven answerable from it, three genuinely not. Run all ten through ask(). For each answerable one, read the answer next to the retrieved chunks and give a verdict: grounded (every claim appears in the chunks), or hallucinated (any claim is absent from them). For the three out-of-scope ones, record whether the refusal line appeared. Build a table in notes-3.5.md: question, top score, verdict. Check: the table has ten rows with verdicts; each out-of-scope question either produced the refusal or the failure is recorded with the chunk that fooled the model; one closing sentence names the most confident-sounding hallucination you caught.
Break retrieval on purpose. Copy ragme.py to ragbroken.py and sabotage retrieval two ways, one at a time: first force k=1 so the model gets a single chunk; then replace retrieve()'s result with three random chunk ids using random.sample(range(len(chunks)), 3) (keep the printed log honest so you can see what was sent). Re-run your seven answerable questions under each sabotage and record verdicts beside the healthy pipeline's verdicts from exercise 2. Check: notes-3.5.md has a three-column comparison — healthy, top-1, random — plus two sentences stating what you observed: answer quality followed retrieval quality, and which sabotage produced confident nonsense rather than refusals.
Tune the honesty gate. The floor of 0.45 was tuned on the lesson's document, not yours. Sweep floor from 0.30 to 0.60 in steps of 0.05, run your ten audit questions at each step, and count answered versus refused. Choose the floor that refuses all three out-of-scope questions while still answering at least six of the seven answerable ones. Then remove RULES entirely (system prompt empty, floor set to 0) and re-run the three out-of-scope questions. Check: notes-3.5.md records the sweep table and your chosen floor with its two counts, one transcript where the no-rules bot invented an answer that your gated bot refused, and one line explaining why your floor number cannot be copied to another corpus or model.
Three failures cover most first-month RAG pain. Each has a War Room entry with the full diagnosis path:
- WR-034 — the answer contradicts the document it cites. Retrieval delivered the right chunks, but the model answered from its own training memory; a grounding rule in the prompt plus a per-claim citation check is the exit.
- WR-035 — garbage retrieval poisoned a healthy model. The top-k was full of irrelevant chunks, so the model improvised; print and read the retrieval log before blaming generation, and fix upstream first.
- WR-036 — the policy changed, the bot kept quoting the old one. The document was edited but the embeddings were never rebuilt; the index is derived data, and it must be rebuilt whenever the source changes.
RAG feeds a model the right chunks of your documents, so answers come from the document instead of training memory. The pipeline has five stages — load, chunk, embed, retrieve, generate — and the first three run once while the last two run per question. The assembled prompt carries labelled context, one grounding rule, and the question, and every claim in the answer must cite a chunk marker. When the top retrieval scores sit low and nearly tied, the honest output is a refusal, not a guess. Quality has a first test you can run today: read each answer beside its retrieved chunks and mark every claim grounded or hallucinated.
- Bank and insurance helpdesks in India run RAG over policy wordings, so agents quote exact clauses to customers instead of guessing from memory.
- Logistics and railway support teams point RAG at SOP manuals; the citation marker tells a supervisor which section to verify in seconds.
- GCC internal assistants answer HR and IT questions from company wikis, and the "not in the document" path routes the question to a human ticket.
Answer these before opening the gate assessment. Each question names its kind.
type=output-prediction— Yourask()usesfloor=0.45. For one question,retrieve()returns[(6, 0.36), (1, 0.33), (9, 0.31)]. Predict the exact stringask()returns, and state which of the two Ollama endpoints was called for this question and which was not.type=debug— The bot answers "notice period is 60 days [C11]", but the current policy file says 30 days everywhere. The retrieval log looks healthy and the prompt rules are strict. The file was edited last week, andrag.pyloads embeddings it saved to disk a month ago. Name the cause and the fix that prevents a repeat.type=interpretation— A failing question's retrieval log shows scores 0.34, 0.33, 0.32, and all three chunks discuss travel reimbursement while the question asks about sick leave. What do the tight low scores and the off-topic chunks together tell you, and which stage do you inspect before touching the generation prompt?type=scenario-judgement— A teammate says the new model's context window fits your whole 80-page handbook, so retrieval should be deleted and the full text pasted into every prompt. Give your judgement and two concrete costs of that proposal that a one-question demo will not show.type=trade-off— Sending top-3 chunks versus top-10 chunks tollama3.2:1bfor a policy bot: give one concrete gain and one concrete cost of each choice.
Explain to a friend who studies commerce and has never coded: how can a chatbot answer only from the office handbook, show where each fact came from, and admit when the handbook does not cover something? Use at most 4 sentences, plain words only — "finds the few closest paragraphs" is allowed, "cosine similarity over embeddings" is not. If your friend can repeat back why the model receives a few chosen paragraphs rather than the whole book, you own this module.