Stand up the same 50-document corpus behind two RAG pipelines — a managed knowledge-base service on trial credits and a DIY stack (LlamaIndex + pgvector on a free Postgres) — then score both with Ragas on the same questions and settle the managed-vs-custom argument with measurements instead of opinions.
Lab 07 made the platform decision on paper; this lab is the empirical check. Anita's matrix scored "retrieval quality control" from vendor claims — now she builds both paths and measures. The corpus is generated deterministically, so your numbers are reproducible and the comparison is fair: same documents, same questions, different platforms.
~120 minutes hands-on · Advanced tier (maps to module M5). Cost: the DIY path is fully ₹0 (OSS + free-tier Postgres + local models via Ollama); the managed path uses a hyperscaler trial/free credits — if you have none, the documented fallback keeps the lab complete at ₹0.
Requirements: Python 3.11+, pip install llama-index llama-index-vector-stores-postgres llama-index-embeddings-ollama llama-index-llms-ollama ragas datasets, Ollama installed with nomic-embed-text and llama3.1 pulled, and a free Postgres with pgvector (Neon or Supabase free tier — create the database, enable the vector extension, note the connection string).
Create rag-both-ways/ and generate the corpus:
# make_corpus.py — deterministic 50-doc synthetic Orbit product corpus
import random, pathlib
random.seed(4207) # fixed seed: everyone's corpus is identical
OUT = pathlib.Path("corpus"); OUT.mkdir(exist_ok=True)
MODULES = ["Billing", "Webhooks", "SSO", "Audit Log", "API Keys",
"Data Export", "Roles", "Sandbox", "Rate Limits", "Alerts"]
VERBS = ["configure", "rotate", "disable", "export", "restrict"]
for i in range(50):
m, v = MODULES[i % 10], VERBS[i % 5]
limit = random.randint(3, 96)
days = random.choice([7, 30, 90, 365])
plan = random.choice(["Starter", "Growth", "Enterprise"])
mins = random.randint(2, 15)
slug = m.lower().replace(" ", "-")
doc = (
f"# Orbit {m} — how to {v} (doc {i+1:02d})\n\n"
f"Orbit {m} is available on the {plan} plan and above.\n\n"
f"## Key facts\n"
f"- Default retention for {m.lower()} records: {days} days.\n"
f"- Maximum {m.lower()} configurations per workspace: {limit}.\n"
f"- Required permission: `{slug}:{v}` (admin role).\n\n"
f"## Procedure\n"
f"1. Open Settings, then {m}.\n"
f"2. Choose '{v.title()}' and confirm with your passkey.\n"
f"3. Changes propagate to all tenant regions within {mins} minutes.\n"
)
(OUT / f"orbit-{slug}-{i+1:02d}.md").write_text(doc)
print("wrote 50 docs to ./corpus")
Create questions.csv with a header question,reference_answer and these 10 questions (fill reference_answer by reading your generated docs — the seed makes them identical for everyone, and filling them yourself is the point: ground truth is authored, never generated):
Which plan is required for Orbit SSO as described in doc 03?
What is the default retention window for billing records in doc 01?
What permission is required to rotate API keys?
How many webhook configurations can a workspace hold per doc 02?
Within how many minutes do audit log changes propagate in doc 04?
What is the procedure to disable SSO?
Which modules document an export permission in their key facts?
What retention window applies to alerts records in doc 10?
What must an admin confirm changes with, per the standard procedure?
Does any doc state a workspace limit above 90 configurations? Which?
Run python make_corpus.py, open three documents, and fill all 10 reference_answer cells from the actual text. Two questions are multi-doc or procedural on purpose — note which ones you expect to be hard for pure vector retrieval.
✅ Checkpoint: 50 files in corpus/, questions.csv fully filled with hand-authored ground truth.
Wire LlamaIndex to your free Postgres and local models — this whole path costs nothing and every knob is yours:
# diy_rag.py — LlamaIndex + pgvector (Neon/Supabase) + Ollama, ₹0
import os
from llama_index.core import (SimpleDirectoryReader, VectorStoreIndex,
StorageContext, Settings)
from llama_index.vector_stores.postgres import PGVectorStore
from llama_index.embeddings.ollama import OllamaEmbedding
from llama_index.llms.ollama import Ollama
Settings.embed_model = OllamaEmbedding(model_name="nomic-embed-text")
Settings.llm = Ollama(model="llama3.1", request_timeout=180)
store = PGVectorStore.from_params(
connection_string=os.environ["PG_URL"],
table_name="orbit_docs",
embed_dim=768, # must match the embedding model's output dimension
)
docs = SimpleDirectoryReader("corpus").load_data()
index = VectorStoreIndex.from_documents(
docs,
storage_context=StorageContext.from_defaults(vector_store=store),
)
qe = index.as_query_engine(similarity_top_k=4)
print(qe.query("What permission is required to rotate API keys?"))
Run it, then probe three questions interactively and read the retrieved chunks (resp.source_nodes), not just the answers. Record in notes-diy.md: chunking defaults used, top_k, embedding model/dimension, and rough end-to-end latency per question. If you prefer a frontier LLM for generation, any LlamaIndex-supported provider slots into Settings.llm — keep the embedding model fixed so retrieval stays comparable.
✅ Checkpoint: DIY pipeline answers questions from the corpus; notes-diy.md records the four configuration facts and observed latency.
On whichever hyperscaler you hold trial credits for, create the managed equivalent — the steps are deliberately generic because consoles change: (1) put the 50 docs in the platform's object store or upload them directly; (2) create a managed knowledge base / search index over them, accepting default chunking; (3) attach a model you have access to; (4) ask the same three probe questions in the test console and capture the retrieved passages. Record in notes-managed.md the same four facts you logged for DIY — and note which of them the console let you see or change, because that difference is the control argument made concrete. Before creating anything, check the service's pricing page for index or capacity-unit minimums — some managed vector backends bill hourly even when idle — and delete every resource when done.
No credits? Documented ₹0 fallback: build a second DIY variant as a "managed proxy" — defaults untouched, similarity_top_k=2, no visibility into chunks (discipline yourself to not read them) — and compare against your tuned DIY path. You lose vendor specifics but keep the methodology: two pipelines, same corpus, same questions, one scorecard.
✅ Checkpoint: managed (or proxy) pipeline answers the probe questions; notes-managed.md records configuration visibility per fact.
Run the same 10 questions through both pipelines and evaluate:
# eval_rag.py — one scorecard, both pipelines
import csv
from datasets import Dataset
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision
def score(query_fn, name):
rows = list(csv.DictReader(open("questions.csv")))
d = {"question": [], "answer": [], "contexts": [], "ground_truth": []}
for r in rows:
resp = query_fn(r["question"])
d["question"].append(r["question"])
d["answer"].append(str(resp))
d["contexts"].append([n.get_content() for n in resp.source_nodes])
d["ground_truth"].append(r["reference_answer"])
result = evaluate(Dataset.from_dict(d),
metrics=[faithfulness, answer_relevancy,
context_precision])
print(name, result)
return result
Ragas needs a judge LLM: point it at a capable model via its provider configuration (a Claude or other frontier judge gives the most reliable scores; a local llama3.1 judge keeps the lab at ₹0 but treat its absolute numbers as directional and compare only between your two pipelines). For the managed path, adapt query_fn to call the managed API and return answer + retrieved passages in the same shape. Save both metric sets.
✅ Checkpoint: faithfulness, answer relevancy, and context precision recorded for both pipelines on the identical question set.
Build scorecard.md with rows: the three Ragas metrics, median latency, cost per 1,000 queries (compute DIY from your setup — near-zero marginal; take managed from the pricing page today, never from memory or the model), setup time you actually spent, and a control row (which retrieval knobs each path exposed). Close with a five-line decision note: does the measurement support the Lab 07 paper decision, or does it change a matrix score? Cite the specific cells.
✅ Checkpoint: complete scorecard plus a decision note that explicitly confirms or amends the Lab 07 matrix.
Managed buys you speed and gives up visibility; DIY buys you every knob and bills you in operational attention. Neither claim means anything until the same corpus and the same questions produce two scorecards — the eval harness, not the vendor page, is the arbiter.
make_corpus.py re-run produces identical files (fixed seed); ground truth was authored by you from the docs.notes-managed.md.The rag-both-ways/ repo: corpus generator, both pipelines, questions.csv, eval script, both notes files, scorecard.md. Self-grading rubric (pass = 8/11):
The corpus here is synthetic, so this lab is safe anywhere. For a client-corpus rerun: generate a schema-faithful synthetic corpus the same way (the generator pattern adapts to any document family), keep embeddings and generation inside the client-approved tenancy or fully local via Ollama, and remember DPDP/MSA scope extends to embeddings and eval traces, not just documents — a Ragas run that ships client chunks to a third-party judge API is an exfiltration path. Local judge or in-tenancy judge for client data, always.
PGVectorStore connection fails: Neon/Supabase need SSL — append ?sslmode=require to the connection string, and confirm the vector extension is enabled.embed_dim — drop the table or use a new table_name.nomic-embed-text for embeddings so retrieval comparisons hold.pip install "ragas<1" "datasets<3" shapes vary by release) and check the current quickstart — eval libraries move fast.