Your First RAG Over Your Own Notes and Docs

Tier: Core · Hands-on time: ~60 minutes

That experience is what this playbook delivers, on a free local stack, with a hosted shortcut included.

When to use

• •
  You have a corpus you keep re-searching: project notes, runbooks, meeting minutes, policy docs, saved articles.
• •
  You want grounded answers with citations instead of the model's general knowledge.
• •
  You need the hands-on RAG literacy every AI-era job description now assumes.

Prerequisites

• •
  Working Ollama setup from pb-07 (a 7–9B instruct model pulled).
• •
  Docker installed (for the Open WebUI path), or willingness to use the desktop-app path.
• •
  10–30 of your own documents as Markdown, text, or PDF — real ones; toy corpora teach nothing.
• •
  Prepare 5 test questions you know the answers to, plus 2 questions you know are NOT answered in the corpus.

The workflow

1. 1.

   Start Open WebUI connected to Ollama.

   Copy

   docker run -d -p 3000:8080 \
     -e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
     -v open-webui:/app/backend/data \
     --name open-webui ghcr.io/open-webui/open-webui:main
   

   Open http://localhost:3000, create the local admin account (stored locally), and confirm your Ollama model appears in the model picker.

2. 2.

   Build the knowledge base. In Open WebUI: Workspace → Knowledge → create a collection ("my-notes") → upload your documents. The tool chunks and embeds them locally — this is the "indexing" half of RAG happening on your machine.

3. 3.

   Ask with retrieval on. In a new chat, reference the collection (the # collection picker) and ask one of your known-answer questions. Inspect the response's citations panel: which chunks were retrieved, from which files.

4. 4.

   Run your 7-question eval. Ask all 5 known-answer questions and both known-absent questions. Score each:

   • •
     Known-answer: Correct? Cited the right document? Or answered from general knowledge while ignoring your notes?
   • •
     Known-absent: Did it say "not in the provided documents" (correct) or confabulate an answer (the classic RAG failure)?
5. 5.

   Tune the two levers that matter. If answers miss obvious content: your documents may have poor headings/structure — retrieval works on chunks, so a 40-page unstructured PDF retrieves badly; split it or add headings and re-upload. If it confabulates on absent questions, add a system prompt to the model in Open WebUI:

   Copy

   Answer ONLY from the provided context. If the context does not contain
   the answer, reply exactly: "Not found in the provided documents."
   Always name the source document for each claim.
   

6. 6.

   Adopt or discard deliberately. Re-run the eval after tuning. If ≥5 of 7 pass, fold it into your routine for that corpus; if not, you've still learned precisely where RAG breaks — which is the professional skill this playbook is actually teaching.

Tool variants

• Open WebUI + Ollama (this playbook): fully local and free; the most transparent view of retrieval internals among the easy options.
• AnythingLLM (desktop app): no Docker required — point it at Ollama, drag in a folder of documents; slightly less visibility into chunks, gentler setup.
• Hosted "projects/knowledge" features: major assistants and notebook-style research tools accept uploaded documents and answer with citations — zero setup and stronger models, but your corpus leaves your machine; fine for non-sensitive notes, unusable for client data.

Verification

• •
  ≥4 of 5 known-answer questions answered correctly with citations pointing to the right file — an uncited correct answer may be general knowledge, which is a retrieval failure in disguise.
• •
  2 of 2 known-absent questions produce "not found" rather than confident invention.
• •
  Click one citation per answer and confirm the passage actually supports the claim.
• •
  Ask one question twice with different phrasing — materially different answers reveal retrieval instability worth knowing about.

Pitfalls

• Testing only easy questions. Questions whose keywords match document headings always pass. Include paraphrased questions that share no keywords with the source — that's where retrieval genuinely fails.
• Garbage corpus in, garbage RAG out. Scanned PDFs without OCR, slide decks that are all images, and giant unstructured exports retrieve terribly. Curate before you index.
• Blaming the model for retrieval misses. Swapping to a bigger model can't fix a chunk that was never retrieved. Diagnose with the citations panel first.
• Stale index. Edited documents don't re-index themselves — re-upload after significant changes, or your RAG confidently serves last month's runbook.
• Skipping the known-absent test. A RAG that never says "not found" is a liability precisely because it's fluent.

Restricted-environment variant

• •
  The Open WebUI + Ollama path is already fully local — documents, embeddings, and answers never leave your machine. This makes it suitable for client-confidential corpora where hosted knowledge features are prohibited.
• •
  Verify the loop is truly closed: in Open WebUI admin settings, confirm no external/web search or hosted-model connections are enabled before loading client documents.
• •
  Multi-client hygiene: one knowledge collection per client, and purge collections (and the Docker volume, if leaving the engagement) when off-boarded.
• •
  If even local installation is prohibited on the client machine, run the stack on your own hardware with client-approved, anonymized excerpts only (see pb-01's anonymization method).

Success criteria

• •
  A working local RAG over ≥10 of your real documents, built by you, running end-to-end offline.
• •
  Your 7-question eval sheet exists with pass/fail scores before and after tuning.
• •
  You can explain, from observed behavior, the difference between a generation failure and a retrieval failure — and demonstrated one of each.
• •
  A deliberate keep-or-discard decision, recorded with reasons.