Ask AI about your own wiki, tickets or contracts and it answers from nothing. This module explains retrieval-augmented generation end to end — the seven pipeline stages, what each one decides, the failure signatures you can spot as a user, and the access-control questions to ask.
Ask a frontier model about your company's leave policy, last quarter's incident postmortems, or the config conventions in your monorepo, and it will answer fluently — from nothing. The model was trained on the public internet up to some cutoff date; it has never seen your wiki, your tickets, or your contracts. Retrieval-Augmented Generation (RAG) is the industry's standard fix: fetch the relevant pieces of your data at question time, put them in front of the model, and instruct it to answer from those pieces — with citations. Nearly every "chat with our docs," "AI knowledge base," and "ask the codebase" product you'll touch this decade is RAG under the hood.
RAG doesn't make the model smarter — it changes what the model reads before it answers. Every RAG system is a search engine bolted to an LLM, and almost every RAG failure is a search failure wearing an AI costume.
Our Phase 1 employer research (2025–26 postings analysis) found RAG named in roughly 65% of applied-LLM job listings — making it the single most common applied-AI requirement after general API skills. And the postings aren't vague: data-engineering listings now spell out RAG sub-skills verbatim — "chunking strategy," "embedding selection," "hybrid search," "reranking," "RAGAS evaluation" — while "vector databases" has become boilerplate on par with SQL. Cloud-architect postings ask for RAG designs on managed stacks (AWS Bedrock Knowledge Bases, pgvector on Aurora); security postings ask about "securing RAG against data poisoning"; PM interview loops probe RAG-versus-fine-tuning trade-offs.
The reach matters as much as the number. Even if you never build a retrieval pipeline, you will almost certainly operate, evaluate, secure, or make decisions about one: the support engineer running a KB chatbot, the QA lead testing whether answers are grounded, the analyst deciding whether the sales team can trust the contract-search tool, the EM approving the budget for it. That's why this module targets literacy depth — enough to reason about every stage and catch failures — while the role tracks (data engineering, DS/ML, architecture) go implementation-deep.
Every RAG system, from a weekend project to an enterprise deployment, is the same seven-stage pipeline. The first four run offline when documents are added; the last three run online for every question.
OFFLINE (indexing) ONLINE (per query)
1. Ingest — collect & clean docs 5. Retrieve — find candidate chunks
2. Chunk — split into pieces 6. Rerank — reorder by true relevance
3. Embed — pieces → vectors 7. Generate — LLM answers from chunks,
4. Store — vectors + text in DB with citations
Walk one question through it. You ask an internal bot: "What's our on-call compensation policy?" The system embeds your question into a vector, searches the vector store for the chunks whose vectors sit closest to it (likely paragraphs from an HR PDF and a 2024 policy update page), optionally reranks those candidates so the truly relevant ones come first, then builds a prompt roughly shaped like: "Using only the following excerpts, answer the user's question. Cite which excerpt supports each claim." The model writes the answer; the UI shows it with source links.
Two properties fall out of this design and explain most of what you'll observe in practice:
Each stage embeds design decisions with visible consequences. You don't need to make these decisions yet — you need to recognize their fingerprints.
Documents must be split because embeddings and prompts work on pieces, not 80-page PDFs. Chunking is a trade-off triangle:
Common practice: a few hundred tokens per chunk, 10–20% overlap between neighbors, splitting along document structure (headings, sections) rather than raw character counts, and attaching metadata (source, title, date, access level) to every chunk. A newer refinement worth knowing by name — contextual retrieval — prepends a short generated summary of where the chunk sits in its document ("This excerpt is from the 2026 on-call policy, section 4: compensation") before embedding it, which measurably cuts retrieval misses.
An embedding model converts text into a vector — a list of numbers positioned so that semantically similar texts land near each other. "On-call pay" and "compensation for after-hours support" share almost no words but sit close in embedding space; that's what makes this better than keyword search alone. Named options employers actually use: OpenAI's text-embedding-3 family, Voyage AI (the provider Anthropic points its customers to), Google's Gemini embedding models, and open-weight models like the BGE/E5 families for self-hosted or data-residency-constrained stacks. They differ in quality, dimension size (storage cost), price, and language coverage.
Queries and documents must be embedded with the same model. Swapping or upgrading the embedding model means re-embedding the entire corpus — mixed-model vectors aren't comparable, and the failure is silent: retrieval just quietly gets worse.
The vector database stores chunk vectors and finds nearest neighbors fast. The market splits into dedicated vector DBs — Pinecone (managed), Qdrant, Weaviate, Milvus (open source), Chroma (lightweight, great for prototypes) — and vector extensions to databases you already run, led by pgvector for PostgreSQL.
pgvector is the accessible default for most teams — if your organization already runs Postgres, you get vector search inside the database you already operate, back up, and control access to, with no new vendor. Dedicated vector DBs earn their place at large scale (tens of millions of vectors, high query rates) or for specialized features — not as a reflex.
Pure vector similarity is weak exactly where professionals need strength: exact identifiers. Search "error TLS-4012" or a part number, and semantic search returns conceptually similar chunks while missing the one containing that literal string. Production systems therefore run hybrid search — classic keyword ranking (BM25) and vector similarity in parallel, with the result lists fused (commonly via reciprocal rank fusion). If a RAG tool you use nails concepts but whiffs on ticket numbers and error codes, you've just diagnosed a missing keyword leg.
Retrieval optimizes for speed over a huge corpus and returns, say, 20 plausible chunks. A reranker — a slower, more accurate model that reads the query and each chunk together (Cohere Rerank, Voyage rerankers, open-source cross-encoders) — re-scores those 20 and promotes the best 5 into the prompt. It's a cheap quality lever: better final chunks, fewer tokens sent to the LLM, no re-indexing required.
The final prompt says, in effect: answer only from these excerpts; cite which excerpt supports each claim; say "not found" if the excerpts don't contain the answer. Citations aren't decoration — they're the verification hook. An answer that cites chunk 3 lets you click chunk 3 and check; an uncited answer from a RAG system deserves the same skepticism as an uncited claim from a stranger.
RAG is one of three ways to get your knowledge into a model, and "which one" is now a standard interview question across roles.
| Approach | What it does | Best when | Watch out |
|---|---|---|---|
| RAG | Retrieves relevant pieces at query time | Large/changing corpora, freshness matters, citations and per-user access control needed | You're now operating a search system; quality depends on retrieval |
| Long context | Pastes entire documents into the prompt (1M-token windows make this real) | Corpus is small and stable — a handful of docs, a single contract review, one codebase question | You re-pay those input tokens every call; recall degrades on huge stuffed prompts; no access control story |
| Fine-tuning | Trains the model's weights on your examples | Teaching behavior — style, format, domain jargon, tool patterns | Terrible at facts: knowledge bakes in stale, no citations, no per-user permissions; re-training per update |
The clean heuristic assessors look for — fine-tuning teaches a model how to behave; RAG tells it what is true right now; long context is RAG-without-retrieval for corpora small enough to paste. "Our product docs change weekly and answers need citations" is a RAG answer, not a fine-tuning answer.
These combine: a fine-tuned model for tone, RAG for facts, long context for the one meeting transcript pasted into today's session.
You will spend more time using RAG systems than building them. These five failure signatures are what literacy is for.
The two cheapest probes for any RAG system you're handed — ask something you can verify and click through the citation; then ask something you know isn't in the corpus and see whether it says "not found" or improvises. A system that can't say "I don't know" can't be trusted when it "knows."
The property that separates a trustworthy RAG system from a fluent one is faithfulness (also called groundedness): is every claim in the answer actually supported by the retrieved chunks? Note what this doesn't ask — whether the answer is true in the world, only whether it's backed by the sources shown. A faithful answer from a stale index is faithfully wrong; that's why faithfulness and index freshness are separate checks.
Practitioner evaluation splits RAG quality into stage-shaped questions: did retrieval find the relevant chunks (context recall/precision)? Is the answer supported by them (faithfulness)? Does it actually address the question (answer relevance)? Frameworks like RAGAS — named verbatim in data-engineering job postings — score these automatically over test-question sets, typically using an LLM as the judge. At literacy level, the takeaway is the decomposition itself: when a RAG system disappoints, ask which stage failed before asking which model to blame. Module cc-6 turns this idea into a general evaluation discipline.
RAG creates a new data surface, and the most common serious mistake is treating the vector index as a harmless cache.
Before piloting any "chat with our documents" tool at work, get three questions answered — who can retrieve what (do source permissions carry over?), where do embeddings live (whose infrastructure, which region?), and what happens on deletion? A RAG deployment that can't answer all three is a data-leak incident on a timer.
Nothing builds the mental model like watching your own documents flow through the pipeline. Two free paths — do at least one; doing both teaches you what the no-code tools hide.
Google's NotebookLM (notebooklm.google, free tier) is a polished RAG system in a box. Create a notebook, upload 5–10 of your own documents — meeting notes, a project README, exported wiki pages — and ask questions. Every answer carries inline citations that jump to the exact source passage. You're watching stages 5–7 (retrieve, rerank, generate-with-citations) through a UI, with Google operating stages 1–4 invisibly. Use real work-adjacent documents (nothing confidential) so the failure modes have stakes.
The build-it path runs entirely on your laptop — no API key, no data leaving the machine. Install Ollama, pull one chat model and one embedding model (ollama pull llama3.2 and ollama pull nomic-embed-text), then pip install ollama chromadb:
import ollama, chromadb, glob
# OFFLINE: ingest → chunk → embed → store
db = chromadb.Client().create_collection("notes")
for path in glob.glob("notes/*.md"):
text = open(path).read()
chunks = [text[i:i+1200] for i in range(0, len(text), 1000)] # ~1200 chars, 200 overlap
for j, chunk in enumerate(chunks):
vec = ollama.embed(model="nomic-embed-text", input=chunk)["embeddings"][0]
db.add(ids=[f"{path}-{j}"], embeddings=[vec],
documents=[chunk], metadatas=[{"source": path}])
# ONLINE: retrieve → generate with citations
question = "What did we decide about the deployment schedule?"
qvec = ollama.embed(model="nomic-embed-text", input=question)["embeddings"][0]
hits = db.query(query_embeddings=[qvec], n_results=4)
context = "\n\n".join(f"[{m['source']}]\n{d}"
for d, m in zip(hits["documents"][0], hits["metadatas"][0]))
answer = ollama.chat(model="llama3.2", messages=[{
"role": "user",
"content": f"Answer ONLY from these excerpts, citing [source] for each claim. "
f"If the answer isn't in them, say so.\n\n{context}\n\nQuestion: {question}"
}])
print(answer["message"]["content"])
That's a complete RAG system minus reranking and hybrid search — which is exactly the point: you now know precisely which stages a "minimal" system skips, and where production systems spend their added complexity.