Stand up a baseline RAG pipeline over a synthetic company handbook — instrumented from the first commit. Deterministic retrieval metrics (recall@k, MRR) and Ragas generation metrics are wired in before you tune anything; then a 3-strategies × 3-sizes chunking matrix decides the configuration by evidence instead of vibes.
The corpus, the QA set, and the harness you build here are reused by Labs 06, 07, and 08 — keep the working directory.
~3 hours hands-on · Core tier · API spend ≤ $0.50 (Ragas judge calls on 10–24 samples with a cheap-tier model; retrieval metrics cost $0).
Python 3.10+, sentence-transformers (CPU is fine), numpy, ragas, langchain-openai (or the Gemini/Groq equivalent for the judge). First, generate the corpus: 12 policy documents for the fictional Meridian Analytics, each seeded with two verifiable numeric facts, plus qa.jsonl — 24 questions with ground truth and the exact gold sentence. Deterministic (seed 13), stdlib only.
"""make_corpus.py — synthetic policy corpus + QA eval set.
Writes corpus/*.md and qa.jsonl."""
import json
import os
import random
rng = random.Random(13)
os.makedirs("corpus", exist_ok=True)
SPECS = [
("travel", "Domestic Travel & Reimbursement Policy", [
("Hotel stays in metro cities are reimbursed up to Rs. {v} per night.",
"What is the per-night hotel reimbursement cap for metro cities?", "Rs. {v}", (3000, 9000, 500)),
("Flight bookings above Rs. {v} require prior VP approval.",
"Above what flight cost is VP approval required?", "Rs. {v}", (10000, 30000, 1000))]),
("leave", "Leave & Time-Off Policy", [
("Employees accrue {v} days of paid leave per calendar year.",
"How many paid leave days accrue per year?", "{v} days", (18, 30, 1)),
("Unused leave up to {v} days may be carried into the next year.",
"How many unused leave days can be carried forward?", "{v} days", (5, 15, 1))]),
("oncall", "On-Call Compensation Policy", [
("Weekend on-call shifts pay a flat allowance of Rs. {v} per day.",
"What is the weekend on-call allowance per day?", "Rs. {v}", (1500, 5000, 250)),
("An engineer may be on-call for at most {v} weeks per quarter.",
"What is the maximum on-call weeks per quarter?", "{v} weeks", (2, 6, 1))]),
("expense", "General Expense Policy", [
("Meal expenses during client visits are capped at Rs. {v} per day.",
"What is the daily meal cap during client visits?", "Rs. {v}", (800, 2500, 100)),
("Expense claims must be filed within {v} days of the spend date.",
"Within how many days must expense claims be filed?", "{v} days", (15, 60, 5))]),
("security", "Information Security Policy", [
("Laptops must auto-lock after {v} minutes of inactivity.",
"After how many idle minutes must laptops auto-lock?", "{v} minutes", (5, 15, 1)),
("Passwords must be rotated every {v} days for privileged accounts.",
"How often must privileged-account passwords be rotated?", "every {v} days", (30, 90, 15))]),
("data", "Data Retention & Privacy Policy", [
("Customer PII is retained for {v} months after contract termination.",
"How long is customer PII retained after contract termination?", "{v} months", (6, 36, 6)),
("Backup snapshots are kept for {v} days before automatic deletion.",
"How many days are backup snapshots kept?", "{v} days", (30, 120, 15))]),
("wfh", "Remote & Hybrid Work Policy", [
("Staff may work remotely up to {v} days per week without approval.",
"How many remote days per week need no approval?", "{v} days", (2, 4, 1)),
("A one-time home-office stipend of Rs. {v} is available to new joiners.",
"What is the one-time home-office stipend?", "Rs. {v}", (8000, 25000, 1000))]),
("procurement", "Procurement & Vendor Policy", [
("Purchases above Rs. {v} require three competitive quotes.",
"Above what purchase value are three quotes required?", "Rs. {v}", (50000, 200000, 25000)),
("Vendor invoices are paid on net-{v} terms from receipt date.",
"What are the standard vendor payment terms?", "net-{v}", (30, 60, 15))]),
("training", "Learning & Development Policy", [
("Each employee has an annual training budget of Rs. {v}.",
"What is the annual per-employee training budget?", "Rs. {v}", (15000, 60000, 5000)),
("Certification exam fees are reimbursed at {v} percent on a pass.",
"What percent of certification fees is reimbursed on a pass?", "{v} percent", (50, 100, 10))]),
("incident", "Incident Response Runbook", [
("Sev-1 incidents must be acknowledged within {v} minutes.",
"What is the Sev-1 acknowledgement SLA?", "{v} minutes", (5, 30, 5)),
("Post-incident reviews are due within {v} business days of resolution.",
"When are post-incident reviews due?", "{v} business days", (3, 10, 1))]),
("payroll", "Payroll & Benefits Handbook", [
("Salary is credited on the {v}th of every month.",
"On which day of the month is salary credited?", "{v}th", (25, 31, 1)),
("The referral bonus for senior engineering hires is Rs. {v}.",
"What is the referral bonus for senior engineering hires?", "Rs. {v}", (25000, 100000, 12500))]),
("assets", "IT Asset Management Policy", [
("Laptops are refreshed on a {v}-year replacement cycle.",
"What is the laptop replacement cycle?", "{v}-year", (3, 5, 1)),
("Lost devices must be reported to IT within {v} hours.",
"Within how many hours must lost devices be reported?", "{v} hours", (4, 24, 4))]),
]
FILLER = [
"This policy applies to all full-time employees, contractors, and interns of Meridian Analytics.",
"Exceptions require written approval from the department head and the HR business partner.",
"Managers are responsible for communicating this policy to their teams each quarter.",
"Violations may result in disciplinary action as described in the employee handbook.",
"This document is reviewed annually by the policy committee and updated as needed.",
"Questions about interpretation should be raised in the #people-ops channel first.",
"Reimbursement requests must include itemized receipts uploaded to the finance portal.",
"Approvals are recorded in the workflow tool and audited twice a year.",
"The finance team publishes a monthly digest of policy changes on the intranet.",
"Where local law conflicts with this policy, local law prevails.",
]
qa = []
for slug, title, facts in SPECS:
paras, fact_sentences = [], []
for sent, q, a, (lo, hi, step) in facts:
v = rng.randrange(lo, hi + 1, step)
fact_sentences.append(sent.format(v=v))
qa.append({"question": q, "ground_truth": a.format(v=v),
"gold_doc": f"{slug}.md", "gold_span": sent.format(v=v)})
for p in range(7):
chosen = rng.sample(FILLER, rng.randint(3, 5))
for fs in fact_sentences:
if rng.random() < 0.35 and fs not in " ".join(paras):
chosen.insert(rng.randrange(len(chosen) + 1), fs)
paras.append(" ".join(chosen))
for fs in fact_sentences:
if fs not in " ".join(paras):
paras[rng.randrange(len(paras))] += " " + fs
with open(f"corpus/{slug}.md", "w") as f:
f.write(f"# {title}\n\n" + "\n\n".join(paras) + "\n")
with open("qa.jsonl", "w") as f:
f.writelines(json.dumps(row) + "\n" for row in qa)
print(f"wrote {len(SPECS)} docs and {len(qa)} QA pairs")
Sanity-check: every gold_span appears exactly once in its gold_doc — write the three-line assertion loop yourself before proceeding; it is your indexing smoke test.
SentenceTransformer("all-MiniLM-L6-v2") (normalized), retrieve top-4 by cosine for each of the 24 questions. Before writing any generation code, compute recall@4 (a retrieved chunk comes from gold_doc and contains ground_truth) and MRR. This is the "instrumented from the first commit" discipline: retrieval quality is measurable for free, so measure it first.from ragas import evaluate, EvaluationDataset
from ragas.metrics import Faithfulness, AnswerRelevancy, LLMContextRecall, LLMContextPrecisionWithReference
from ragas.llms import LangchainLLMWrapper
from ragas.embeddings import LangchainEmbeddingsWrapper
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
judge = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o-mini", temperature=0))
emb = LangchainEmbeddingsWrapper(OpenAIEmbeddings(model="text-embedding-3-small"))
ds = EvaluationDataset.from_list(samples) # user_input, retrieved_contexts, response, reference
print(evaluate(ds, metrics=[Faithfulness(), AnswerRelevancy(),
LLMContextRecall(), LLMContextPrecisionWithReference()],
llm=judge, embeddings=emb))
The cheap-model path: gpt-4o-mini (or Gemini Flash via langchain-google-genai, or a Groq-hosted model) keeps 10 samples × 4 metrics around $0.10–0.20. Record all four scores as your baseline row. Interpretation contract: faithfulness < 0.70 means investigate hallucination; low context recall points at retrieval, low faithfulness with high recall points at the prompt.
3. Run the chunking matrix. Three chunkers × three sizes, scored on the deterministic metrics (no LLM cost, so run all nine):
"""chunk_bench.py — 3 chunkers x 3 sizes, deterministic retrieval scoring."""
import json, pathlib, re
import numpy as np
from sentence_transformers import SentenceTransformer
def chunk_fixed(text, size):
return [text[i:i + size] for i in range(0, len(text), size)]
def chunk_recursive(text, size):
out, buf, parts = [], "", []
for para in text.split("\n\n"):
parts.extend([para] if len(para) <= size else re.split(r"(?<=\.)\s+", para))
for p in parts:
if len(buf) + len(p) + 1 > size and buf:
out.append(buf)
buf = buf[-int(size * 0.15):] # 15% overlap tail
buf = (buf + " " + p).strip()
return out + ([buf] if buf else [])
def chunk_sentence(text, size):
sents = re.split(r"(?<=\.)\s+", text.replace("\n\n", " "))
out, buf = [], ""
for s in sents:
if len(buf) + len(s) + 1 > size and buf:
out.append(buf); buf = ""
buf = (buf + " " + s).strip()
return out + ([buf] if buf else [])
docs = {p.name: p.read_text() for p in sorted(pathlib.Path("corpus").glob("*.md"))}
qa = [json.loads(l) for l in open("qa.jsonl")]
model = SentenceTransformer("all-MiniLM-L6-v2")
K = 4
for name, fn in [("fixed", chunk_fixed), ("recursive", chunk_recursive),
("sentence", chunk_sentence)]:
for size in (300, 700, 1400):
chunks, src = [], []
for fname, text in docs.items():
for c in fn(text, size):
chunks.append(c); src.append(fname)
emb = model.encode(chunks, normalize_embeddings=True, show_progress_bar=False)
qemb = model.encode([r["question"] for r in qa], normalize_embeddings=True,
show_progress_bar=False)
sims = qemb @ emb.T
hits, rr = 0, 0.0
for qi, row in enumerate(qa):
top = np.argsort(-sims[qi])[:K]
ranks = [t + 1 for t, ci in enumerate(top)
if src[ci] == row["gold_doc"] and row["ground_truth"] in chunks[ci]]
if ranks:
hits += 1; rr += 1.0 / ranks[0]
print(f"{name:<10} {size:>5} chunks={len(chunks):>4} "
f"recall@4={hits/len(qa):.2f} mrr={rr/len(qa):.2f}")
rag_baseline.py with the config at the top and both metric layers runnable by command. Labs 06–08 build on this exact directory.Deterministic retrieval metrics are free and run on every config; LLM-judged metrics are budgeted and run on finalists — that two-tier design is what makes measured RAG affordable.
rag-experiment-report.md: baseline row, the 9-config table, Ragas scores for finalists, and the justified pick. Rubric (pass = 8/10):
all-MiniLM-L6-v2 needs one ~90 MB download; on Kaggle enable internet or attach the model as a dataset.EvaluationDataset/evaluate interface of ragas ≥0.2. Pin the version and re-check the metric class names.For client corpora: embeddings and the vector store run locally (sentence-transformers + FAISS/pgvector — nothing leaves the machine), and only the generation/judge calls need an approved endpoint; with a local Ollama judge the whole loop is on-prem. The chunking-matrix protocol is a client deliverable in itself — "we chose 300-char recursive chunks because recall@4 was 1.00 vs 0.88" is the sentence that wins architecture reviews.