RAG that survives production

We upgrade 3.5's rag.py into rag2.py on the same Ollama setup — nomic-embed-text to embed, llama3.2:1b to generate. Install the store first: pip install chromadb==1.3.* inside your ai-lab environment (one-time download; everything after runs offline). The pin is deliberate: the counts and answers printed in this module were produced on this version, and chromadb's defaults shift between releases. Prepare two source files. Edit your hr-policy.txt so every topic starts with a heading line like ## Casual leave, and make sure one paragraph mentions a reimbursement form by code, such as "submit Form TR-4 within 30 days of travel". Then write it-policy.txt — eight to ten paragraphs of an imaginary IT policy (passwords, laptops, VPN, device requests for travel) under ## headings.

Step 1 — structure-aware chunking with overlap and metadata. Cut at headings, merge small paragraphs inside a section, and carry one sentence across each cut:

import re, requests, chromadb

def chunk_file(path, dept):
    text = open(path).read()
    out = []
    for sec in re.split(r"^## ", text, flags=re.M)[1:]:
        title, body = sec.split("\n", 1)
        paras = [p.strip() for p in body.split("\n\n") if p.strip()]
        buf, tail = "", ""
        for p in paras + [""]:                    # "" flushes the leftover
            if buf and (len(buf) >= 300 or p == ""):
                out.append({"text": (tail + " " + buf).strip(),
                            "meta": {"doc": path, "department": dept,
                                     "section": title.strip(), "version": 1}})
                tail = buf.rsplit(". ", 1)[-1]    # overlap: last sentence
                buf = ""
            buf = (buf + "\n" + p).strip()
    return out

hr = chunk_file("hr-policy.txt", "hr")
it = chunk_file("it-policy.txt", "it")
print(len(hr), "hr chunks,", len(it), "it chunks")
print(hr[7]["meta"])

Output: 16 hr chunks, 9 it chunks then {'doc': 'hr-policy.txt', 'department': 'hr', 'section': 'Travel reimbursement', 'version': 1} (your counts will differ). Two things changed from 3.5. A chunk can no longer straddle two sections, and the tail variable is the chunk overlap — the closing sentence of one chunk reappears at the top of the next, so a rule that spans the cut survives in at least one piece. The metadata dict travels with the text from here on; nothing downstream can filter on a label that was never attached.

Step 2 — a real vector store with upsert. Same embed() as 3.5, but the vectors now live in chromadb on disk, under stable ids:

def embed(texts):
    r = requests.post("http://localhost:11434/api/embed",
                      json={"model": "nomic-embed-text", "input": texts},
                      timeout=120)
    r.raise_for_status()
    return r.json()["embeddings"]

client = chromadb.PersistentClient(path="ragdb")
col = client.get_or_create_collection("policies",
                                      metadata={"hnsw:space": "cosine"})

def index_doc(path, dept):
    chunks = chunk_file(path, dept)
    col.upsert(ids=[f"{path}#{i}" for i in range(len(chunks))],
               embeddings=embed([c["text"] for c in chunks]),
               documents=[c["text"] for c in chunks],
               metadatas=[c["meta"] for c in chunks])
    return len(chunks)

print(index_doc("hr-policy.txt", "hr"), index_doc("it-policy.txt", "it"))
print("in store:", col.count())

Output: 16 9 then in store: 25. The id hr-policy.txt#7 is deliberate: run index_doc again and upsert overwrites those same ids instead of duplicating them — run it twice and count() stays 25. And because PersistentClient writes to the ragdb folder, you can close Python, reopen it, and query without re-embedding anything. The 3.5 pattern of "restart means rebuild" is gone.

Step 3 — retrieval with a metadata filter. query() does the cosine search 3.4 taught you, but where narrows it first. Watch the same question with and without the filter:

def retrieve(question, k=3, where=None):
    res = col.query(query_embeddings=embed([question]),
                    n_results=k, where=where)
    hits = []
    for cid, dist, doc, meta in zip(res["ids"][0], res["distances"][0],
                                    res["documents"][0], res["metadatas"][0]):
        sim = 1 - dist                     # cosine distance -> similarity
        print(f"  log: {cid:<18s} sim={sim:.2f} "
              f"[{meta['department']}/{meta['section']}]")
        hits.append((cid, sim, doc))
    return hits

retrieve("How do I claim travel allowance?")
print("--- filtered ---")
retrieve("How do I claim travel allowance?", where={"department": "hr"})

Output:

  log: it-policy.txt#7    sim=0.61 [it/Device requests for travel]
  log: hr-policy.txt#11   sim=0.58 [hr/Travel reimbursement]
  log: it-policy.txt#6    sim=0.49 [it/Device requests for travel]
--- filtered ---
  log: hr-policy.txt#11   sim=0.58 [hr/Travel reimbursement]
  log: hr-policy.txt#12   sim=0.44 [hr/Travel reimbursement]
  log: hr-policy.txt#0    sim=0.31 [hr/Casual leave]

Read the unfiltered log closely: the IT policy's travel-device section beat the HR travel section, because both discuss "travel" and "requests" and embeddings only measure closeness of meaning. Send that top-3 to the model and it will quote laptop rules to someone asking about allowance — with a citation that checks out against the wrong document. That is WR-037 in one log. The filter removes the failure class instead of hoping similarity gets it right; keep ask() from 3.5 unchanged except that it calls this retrieve() and passes where through, floor and refusal line intact.

Step 4 — why hybrid: one miss from each side. Semantic search and keyword search fail on different questions. Show one miss each:

# A: no word here matches the sick-leave chunk -- meaning does
retrieve("My daughter has a fever, can I stay home?",
         where={"department": "hr"})

# B: an exact code -- meaning is no help
retrieve("Where do I submit Form TR-4?", where={"department": "hr"})
kw = col.get(where_document={"$contains": "TR-4"})
print("keyword pass found:", kw["ids"])

Output for A: the sick-leave chunk leads at sim=0.57 even though the question shares no keyword with it — grep could never find this. Output for B: the semantic log shows generic travel chunks at flat scores near 0.4, with the actual TR-4 paragraph at rank 3 or missing, then keyword pass found: ['hr-policy.txt#11'] — the literal string match lands instantly. At 25 chunks both passes often succeed; at 4,000 chunks the gap becomes daily pain, which is why production systems run hybrid retrieval and merge both lists, then let reranking pick the best few from the merged shortlist.

Step 5 — the update drill. Edit hr-policy.txt: change the notice period from 30 to 45 days and save. The store does not know yet — prove it, fix it, prove the fix:

print(ask("What is the notice period?", where={"department": "hr"}))
# -> "The notice period is 30 days [hr-policy.txt#13]"   (stale!)

n = index_doc("hr-policy.txt", "hr")        # re-chunk + upsert, same ids
stale = [i for i in col.get(where={"doc": "hr-policy.txt"})["ids"]
         if int(i.split("#")[1]) >= n]
if stale:
    col.delete(ids=stale)                   # doc shrank? kill leftover ids

print(ask("What is the notice period?", where={"department": "hr"}))
# -> "The notice period is 45 days [hr-policy.txt#13]"

The first answer is WR-036 happening on schedule: the source changed, the index did not, and the bot stated last month's reality with confidence. index_doc is the one-line cure because upsert replaces by id. The stale check matters when an edit removes sections: fifteen new chunks over sixteen old ids leaves #15 orphaned, still retrievable, quoting deleted policy. Update discipline is three verbs — re-chunk, upsert, delete leftovers — and it must run on every edit, not on a good-intentions schedule.