Lab 11 · Local RAG Chatbot over the KB

Farhan supports two client accounts at a Kochi MSP, and both contracts forbid client data leaving approved systems — which rules out pasting KB content into cloud chatbots. This lab is the pattern that squares that circle: a retrieval-augmented chatbot running entirely on his laptop, over a KB he controls, that answers with citations and — the harder discipline — refuses when the KB does not contain the answer. This is the vendor-independent skill underneath every service-desk AI product: once you have built chunk-embed-retrieve-ground by hand, every vendor demo becomes legible, and every DPDP or client-confidentiality constraint has a local answer.

The KB below contains two articles that quietly disagree with each other, and your test set contains questions the KB cannot answer. A bot that answers everything is broken; the deliverable is a bot that knows the difference.

Time & tier

~90 minutes hands-on · Core tier (maps to module M6). $0 — Ollama (free, local), Python 3.10+, pip install chromadb ollama. Models: ollama pull llama3.2 (~2GB) and ollama pull nomic-embed-text (~275MB). Any laptop with 8GB RAM works; nothing touches the network after the pulls.

Setup

Create a kb/ directory with six files, one article each. Author them with your AI chat and review per lab-05 — or copy this seed content and expand each to 10–15 lines:

kb01-password-reset.md
  Self-service reset at portal.fincorp.test/reset. Requires MFA
  enrollment. Locked accounts auto-unlock after 30 min; the desk
  can unlock sooner with identity verification.
kb02-mfa-new-phone.md
  Re-enroll at aka.ms/mfasetup from a trusted device. Desk can
  issue a temporary access pass valid 1 hour. Old phone should
  be removed from authenticator list after migration.
kb03-vpn-error-809.md
  Error 809: UDP 500/4500 blocked, common on hotel wifi. Fixes:
  wired/hotspot fallback, then AlwaysOn profile repair. Note:
  do NOT disable the firewall; use the profile repair tool.
kb04-shared-drive-access.md
  Access via portal request, manager + data-owner approval,
  provisioned within 4 business hours. Access is group-based;
  do not request individual NTFS grants.
kb05-vpn-split-tunnel.md
  As of the March change, all traffic routes through the VPN
  tunnel (full tunnel). Split tunneling was disabled estate-wide
  for DLP compliance. Users noticing slow video calls should
  use the office network or approved SD-WAN sites.
kb06-video-call-quality.md
  For poor Teams/Zoom quality on VPN: enable split tunneling in
  the VPN client settings so media traffic bypasses the tunnel.
  (Article last reviewed before the March network change.)

Save your test set as test-questions.txt:

Q1 How do I reset my password?
Q2 I got a new phone, how do I set up MFA again?
Q3 What does VPN error 809 mean and what should I try?
Q4 How long does shared drive access take after approval?
Q5 My video calls are choppy when I'm on the VPN - what
   should I do?
Q6 How do I request a new laptop?
Q7 What's the wifi password for the Pune office?
Q8 My account is locked AND I lost my phone - what do I do?
Q9 Should I disable my firewall to fix VPN error 809?
Q10 Can I get admin rights on my laptop?

Guided steps

1. 1Install and smoke-test the stack. Install Ollama, pull both models, and confirm ollama run llama3.2 "say ok" responds. Expected output: a local completion with no network dependency. If your machine struggles, llama3.2:1b is an acceptable fallback — note the substitution in your design note, because model size will show up in answer quality.
2. 2Write the ingest script. With your AI chat's help (review the code — lab-06 habits apply to Python too), write ingest.py: read kb/*.md, split each article into chunks (start at ~500 characters with 50 overlap, splitting on headings/paragraphs first), embed each chunk with nomic-embed-text, and store in a Chroma collection with metadata {source: filename}. Expected output: a printed count of chunks per file — with articles this size, mostly 1–3 chunks each.
3. 3Write the query pipeline with grounding rules.ask.py: embed the question, retrieve top-3 chunks, and build a prompt that is the heart of this lab: "Answer ONLY from the context below. Cite the source filename for every claim, in the form [kb0X]. If the context does not contain the answer, say exactly: 'The KB does not cover this — please raise a ticket.' Context: {chunks} Question: {question}". Expected output: Q1 answered with a [kb01] citation.
4. 4Run the ten questions and log verbatim results. Record each answer in rag-test-log.md with: cited sources, correct/incorrect, refused/should-have-refused. Do not tune anything mid-run — this is your baseline, and lab-12 needs it intact.
5. 5Score the refusals. Q6, Q7 and Q10 have no KB coverage. Expected output: the exact refusal string for all three. If the model answered from its general knowledge — laptop request procedures it invented, a plausible-sounding admin-rights policy — your grounding prompt is leaking. Tighten it (lower temperature, stronger instruction, explicit "do not use outside knowledge") and re-run only these three, logging before/after.
6. 6Face the contradiction. Q5 retrieves both kb05 and kb06 — which give opposite instructions about split tunneling. Read your bot's answer carefully: did it pick one silently, blend them into nonsense, or surface the conflict? The correct behavior for a support bot is to surface it: "the KB contains conflicting guidance [kb05][kb06] — kb05 reflects the March change." Your bot almost certainly did not do this unaided. Add a line to the prompt ("If retrieved sources conflict, say so and cite both") and re-test. Log both behaviors — the failure is a finding, and fixing the KB itself is lab-05's audit lesson arriving in RAG form.
7. 7Check the multi-article question. Q8 needs kb01 (unlock) and kb02 (lost phone/MFA) stitched together. Expected output: an answer citing both files. If only one was retrieved, note it — top-k retrieval breadth is a real design choice, and this is your first data point for lab-12's tuning.
8. 8Write the design note. One page, design-note.md: chunk size and why; top-k and why; the grounding prompt verbatim with each rule justified; refusal behavior evidence; how the contradiction was handled; and the two-sentence client-facing claim you can now honestly make about data residency ("KB content and questions never leave the laptop; models run locally").

Verify

• •
  ingest.py and ask.py run end-to-end locally; a fresh question round-trips in seconds with no network.
• •
  Q1–Q4 answered correctly, each with at least one accurate [kb0X] citation.
• •
  Q6, Q7, Q10 produce the exact refusal string — zero invented policy.
• •
  The kb05/kb06 conflict is surfaced with both citations after your prompt fix, and both before/after behaviors are logged.
• •
  Q8 cites both kb01 and kb02, or the retrieval gap is documented as a lab-12 input.
• •
  design-note.md justifies chunking, top-k, grounding and refusal with evidence from the test log, and states the data-residency claim.

Self-grading rubric (pass = 7/10): all three refusals exact (0–3) · citations accurate on answered questions (0–2) · contradiction surfaced post-fix with both behaviors logged (0–2) · multi-article question handled or honestly diagnosed (0–1) · design note defends every choice with test evidence (0–2).