RAG Feature, Part 1: Chunking, Embeddings, and a Working Retrieval Pipeline

Tier: Advanced · Workflow area: Building LLM Features

This playbook builds the pipeline end to end on infrastructure you already run (Postgres + pgvector): deliberate chunking, embeddings, a retrieval endpoint, and grounded answer synthesis. Part 2 (pb-48) upgrades retrieval quality with hybrid search, reranking, and citations.

When to use

• •
  Search-and-answer over product docs, internal wikis, support knowledge, or policy documents.
• •
  The model's training data cannot know your content (private, recent, or fast-changing).
• •
  You are choosing between fine-tuning and RAG: for knowledge injection, RAG wins on freshness, auditability, and cost — start here.

Skip RAG when the whole corpus fits in the context window comfortably and queries are rare — just include the documents in the prompt with caching.

Prerequisites

• •
  Postgres 15+ with the pgvector extension (CREATE EXTENSION vector;) — no new vendor required.
• •
  pip install anthropic voyageai psycopg fastapi (or your embedding provider's SDK; OpenAI's text-embedding-3-small is the common alternative).
• •
  A real corpus slice: 50–500 documents. Building against toy data hides every real problem.
• •
  10 real user questions with known-correct source passages — your first eval set.

The workflow

1. 1.

   Chunk deliberately — and test two strategies, not zero. Chunking is the highest-leverage decision in the pipeline. Implement both and keep the winner:

   • •
     Strategy A — structural: split on headings/sections (Markdown/HTML structure), max ~800 tokens, no overlap. Best when documents are well-structured.
   • •
     Strategy B — fixed window: ~500 tokens with 15% overlap, splitting on paragraph boundaries. Robust default for messy content.

   Every chunk keeps metadata: doc_id, title, section, updated_at, source URL. You will need all of it for citations and freshness filtering.

2. 2.

   Embed and store.

   Copy

   CREATE TABLE chunks (
     id bigserial PRIMARY KEY,
     doc_id text NOT NULL,
     title text, section text, url text, updated_at timestamptz,
     content text NOT NULL,
     embedding vector(1024)          -- match your embedding model's dimension
   );
   CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops);
   

   Batch-embed chunks (embedding APIs take arrays — one call per 100 chunks, not per chunk). Record the embedding model name in a config table: mixing embeddings from different models in one column produces silent garbage.

3. 3.

   Retrieve top-k with a similarity floor.

   Copy

   def retrieve(query: str, k: int = 8) -> list[Chunk]:
       qvec = embed([query])[0]                    # SAME model as ingestion
       rows = db.execute(
           """SELECT id, title, section, url, content,
                     1 - (embedding <=> %s::vector) AS score
              FROM chunks ORDER BY embedding <=> %s::vector LIMIT %s""",
           (qvec, qvec, k)).fetchall()
       return [r for r in rows if r.score > 0.35]  # tune floor on your data
   

   The floor matters: with no floor, an off-topic question still "retrieves" the 8 least-irrelevant chunks and the model gamely answers from them.

4. 4.

   Synthesize with grounding rules. The prompt does three jobs: scope the model to the context, force refusal on empty retrieval, and demand source attribution:

   Copy

   import anthropic
   client = anthropic.Anthropic(timeout=30.0)
   
   def answer(query: str) -> dict:
       chunks = retrieve(query)
       if not chunks:
           return {"answer": None, "reason": "no_relevant_content"}
       context = "\n\n".join(
           f"[{i}] ({c.title} / {c.section})\n{c.content}" for i, c in enumerate(chunks))
       resp = client.messages.create(
           model="claude-opus-4-8", max_tokens=1500,
           system="Answer ONLY from the numbered context passages. Cite passage "
                  "numbers like [2] after each claim. If the context does not "
                  "contain the answer, say exactly: NOT_IN_CONTEXT.",
           messages=[{"role": "user",
                      "content": f"Context:\n{context}\n\nQuestion: {query}"}])
       text = resp.content[0].text
       return {"answer": None if "NOT_IN_CONTEXT" in text else text,
               "sources": [{"title": c.title, "url": c.url} for c in chunks]}
   

5. 5.

   Run the chunking bake-off. For your 10 eval questions, measure per strategy: retrieval hit rate (is the known-correct passage in top-k?) and answer correctness (does the answer match, judged by you). Keep the winning strategy; record the numbers — they are your baseline for pb-48.

6. 6.

   Wire ingestion for freshness. Re-chunk and re-embed on document update (hash content per chunk; only re-embed changed chunks). Stale RAG is worse than no RAG — it answers confidently from last quarter's policy.

Tool variants

• pgvector (shown): right default for backend teams — transactional ingestion, one database, SQL filtering alongside vectors.
• Qdrant (OSS) / Pinecone: dedicated vector stores; reach for them at >5–10M chunks or when you need heavy metadata-filtered ANN at scale.
• Embeddings: Voyage or OpenAI text-embedding-3-small are solid defaults; whatever you pick, pin the model + dimension in config.
• LlamaIndex / LangChain: fine as accelerators for loaders and chunkers, but build this hand-rolled version once — framework RAG failures are undebuggable without knowing what is under them.
• OpenAI generation: swap step 4's call for client.responses.create(model="gpt-5.6", ...) with the same grounding system prompt — the pattern is provider-neutral.

Verification

• •
  Retrieval hit rate ≥8/10 on your eval questions for the winning chunking strategy — measured, not vibes.
• •
  Ask three questions whose answers are not in the corpus: all three return NOT_IN_CONTEXT/refusal, not confident fabrication.
• •
  Ask about content you know changed recently: the answer reflects the current version (freshness path works).
• •
  Inspect retrieved chunks for two failed questions: confirm whether the failure is chunking (answer split across chunks), retrieval (right chunk exists, not returned), or synthesis (chunk retrieved, answer wrong) — and that you can tell which.

Pitfalls

• Chunking by fixed characters through tables and code blocks. Structure-blind splitting shreds exactly the content people search for. Respect document structure.
• Different embedding models at ingest vs query. Everything runs, nothing works, no errors. Pin it in config; assert it at startup.
• k as a quality dial. Raising k to fix retrieval stuffs more noise into the context and often makes answers worse. Fix retrieval quality (pb-48), don't widen the net.
• No similarity floor. Every question gets an answer, including the ones that should not.
• Skipping the eval set. Ten questions with known sources is one hour of work and the difference between engineering and guessing — every later change (pb-48, model swaps) is judged against it.

Restricted-environment variant

• Client tenancypgvector runs wherever Postgres runs; embeddings + generation via Bedrock or Azure OpenAI keep the whole pipeline inside the client's cloud. This combination passes most client data reviews.
• Fully localOllama serves both embedding models (e.g. nomic-embed-text) and generation; quality drops at synthesis, so lean harder on the refusal rule and keep answers extractive.
• DPDP/PIIcorpus documents may embed personal data — apply retention and access rules to the chunks table like any other datastore; embeddings of PII are still personal data.

Success criteria

• •
  A working endpoint answering real questions from your real corpus, with sources, refusing when it should.
• •
  A measured baseline: retrieval hit rate and answer correctness per chunking strategy, written down.
• •
  You can classify any bad answer as a chunking, retrieval, or synthesis failure from logs in under five minutes.
• •
  Ingestion keeps the index fresh without full re-embeds.