Build the first stage of a RAG data platform: an ingestion pipeline over a mixed enterprise corpus — PDFs with tables, HTML with navigation boilerplate, an exact-duplicate page, and a scanned (image-only) document.
You will compare two parsers head-to-head on extraction fidelity using a golden-facts scorecard, discover that the parsers tie on raw fact recall but split 0-for-4 versus 4-for-4 on table row integrity (the metric that actually predicts retrieval quality), then wire dedup, OCR-fallback routing, and metadata capture into a pipeline that emits clean, retrieval-ready JSONL. Parsing is where enterprise RAG projects quietly fail — garbage extraction is unrecoverable by any amount of clever chunking or embedding downstream — and the lab's core discipline is treating parsers as swappable components judged by measured quality, never by reputation.
~120 minutes hands-on · Advanced tier (maps to module M5 · RAG Pipelines I — Ingestion, Chunking, Embeddings).
Fully local, $0. The corpus is generated by script, so every learner scores the same documents and no proprietary files are needed.
mkdir ingest-lab && cd ingest-lab
python3 -m venv .venv && source .venv/bin/activate
pip install reportlab pypdf pdfplumber beautifulsoup4 pillow
The two parsers under test are pypdf and pdfplumber — both light, both ubiquitous. In production you would run the same bake-off over Docling (fully open source, strong on layout), Unstructured (open-source core), or a managed parser like LlamaParse; the scorecard method is identical, only the contenders change.
Save make_corpus.py:
"""Generate a mixed enterprise corpus: PDFs with tables, HTML with boilerplate,
one exact-duplicate page, and one image-only 'scanned' PDF. Also emits golden_facts.json."""
import json, os
from reportlab.lib.pagesizes import A4
from reportlab.platypus import SimpleDocTemplate, Paragraph, Table, TableStyle, Spacer
from reportlab.lib.styles import getSampleStyleSheet
from reportlab.lib import colors
from PIL import Image, ImageDraw
OUT = os.path.join(os.path.dirname(os.path.abspath(__file__)), "corpus")
os.makedirs(OUT, exist_ok=True)
styles = getSampleStyleSheet()
facts = {}
# --- PDF 1: quarterly ops report with a table ---
rows1 = [["Region", "Uptime %", "Tickets", "MTTR (min)"],
["North", "99.94", "312", "38"], ["South", "99.71", "540", "61"],
["East", "99.88", "205", "44"], ["West", "98.92", "1288", "97"]]
doc = SimpleDocTemplate(f"{OUT}/ops_report_q2.pdf", pagesize=A4)
t = Table(rows1); t.setStyle(TableStyle([("GRID", (0,0), (-1,-1), 0.5, colors.grey)]))
doc.build([Paragraph("Quarterly Operations Report - Q2 2025", styles["Title"]),
Paragraph("West region missed its SLA due to repeated storage failures in the Pune data hall.", styles["Normal"]),
Spacer(1, 12), t])
facts["ops_report_q2.pdf"] = ["99.94", "1288", "Pune data hall", "MTTR"]
# --- PDF 2: pricing sheet with a wider table ---
rows2 = [["Plan", "Monthly (USD)", "Annual (USD)", "Included seats", "SLA"],
["Free", "0", "0", "3", "none"], ["Pro", "49", "490", "25", "99.5%"],
["Enterprise", "399", "3990", "unlimited", "99.9%"]]
doc = SimpleDocTemplate(f"{OUT}/pricing_sheet.pdf", pagesize=A4)
t = Table(rows2); t.setStyle(TableStyle([("GRID", (0,0), (-1,-1), 0.5, colors.grey)]))
doc.build([Paragraph("Product Pricing - effective 2025-07-01", styles["Title"]), t,
Paragraph("Enterprise contracts renew automatically unless cancelled 60 days before term end.", styles["Normal"])])
facts["pricing_sheet.pdf"] = ["399", "3990", "60 days", "99.9%"]
# --- HTML pages with nav/footer boilerplate ---
BOILER_TOP = "<nav><a href='/'>Home</a><a href='/docs'>Docs</a><a href='/pricing'>Pricing</a></nav>"
BOILER_BOT = "<footer>(c) 2025 Acme Analytics - Privacy - Terms - Careers</footer>"
pages = {
"runbook_ingest.html": ("Ingestion runbook",
"<h1>Ingestion pipeline runbook</h1><p>If the nightly load fails with error INGEST-504, "
"restart the loader with <code>make reload</code> and backfill from the last checkpoint. "
"Checkpoints are stored in the ops bucket and retained for 14 days.</p>"),
"faq_security.html": ("Security FAQ",
"<h1>Security FAQ</h1><p>Customer data is encrypted at rest with AES-256. "
"Access reviews run quarterly, and audit logs are retained for 400 days.</p>"),
"policy_retention.html": ("Retention policy",
"<h1>Data retention policy</h1><p>Raw event data is retained for 90 days, "
"aggregated metrics for 5 years. Deletion requests complete within 30 days.</p>"),
}
for fname, (title, body) in pages.items():
with open(f"{OUT}/{fname}", "w") as f:
f.write(f"<html><head><title>{title}</title></head><body>{BOILER_TOP}{body}{BOILER_BOT}</body></html>")
facts["runbook_ingest.html"] = ["INGEST-504", "make reload", "14 days"]
facts["faq_security.html"] = ["AES-256", "400 days"]
facts["policy_retention.html"] = ["90 days", "5 years", "30 days"]
# exact duplicate under a different name (mirrored URL)
with open(f"{OUT}/runbook_ingest_mirror.html", "w") as f:
f.write(open(f"{OUT}/runbook_ingest.html").read())
# --- 'scanned' page: text rendered as an image, embedded in a PDF (no text layer) ---
img = Image.new("RGB", (1240, 400), "white")
d = ImageDraw.Draw(img)
d.text((60, 60), "MEMO: Incident 2025-041\nRoot cause: expired TLS certificate on the ingest gateway.\nAction: rotate certs quarterly.", fill="black")
img.save(f"{OUT}/scan_memo.png")
from reportlab.lib.utils import ImageReader
from reportlab.pdfgen import canvas
c = canvas.Canvas(f"{OUT}/scanned_memo.pdf", pagesize=A4)
c.drawImage(ImageReader(f"{OUT}/scan_memo.png"), 30, 500, width=520, height=170)
c.save(); os.remove(f"{OUT}/scan_memo.png")
facts["scanned_memo.pdf"] = ["2025-041", "TLS certificate"]
with open(f"{OUT}/golden_facts.json", "w") as f:
json.dump(facts, f, indent=2)
print("corpus files:", sorted(os.listdir(OUT)))
Run python make_corpus.py — eight files: two table PDFs, three HTML pages, one HTML mirror, one scanned PDF, and golden_facts.json (the per-document facts a good parser must recover).
Build the pipeline skeleton. Save ingest.py — parsers, normalizer, scorecard, and the pipeline proper:
"""Ingestion pipeline: parse -> normalize -> dedup -> metadata -> JSONL, plus a parser scorecard."""
import hashlib, json, os, re
from datetime import datetime, timezone
CORPUS = os.path.join(os.path.dirname(os.path.abspath(__file__)), "corpus")
GOLD = json.load(open(f"{CORPUS}/golden_facts.json"))
def parse_pdf_pypdf(path):
from pypdf import PdfReader
return "\n".join((p.extract_text() or "") for p in PdfReader(path).pages)
def parse_pdf_plumber(path):
import pdfplumber
out = []
with pdfplumber.open(path) as pdf:
for page in pdf.pages:
out.append(page.extract_text() or "")
for table in page.extract_tables():
out.append("\n".join(" | ".join(c or "" for c in row) for row in table))
return "\n".join(out)
def parse_html(path):
from bs4 import BeautifulSoup
soup = BeautifulSoup(open(path).read(), "html.parser")
for tag in soup(["nav", "footer", "script", "style"]):
tag.decompose() # boilerplate stripping
return soup.get_text(separator="\n", strip=True)
def normalize(text):
return re.sub(r"\s+", " ", text).strip().lower()
def fact_recall(parse_fn):
results = {}
for fname, gold_facts in GOLD.items():
if not fname.endswith(".pdf"):
continue
text = parse_fn(os.path.join(CORPUS, fname))
hits = sum(1 for fact in gold_facts if fact.lower() in text.lower())
results[fname] = (hits, len(gold_facts), len(text))
return results
def row_integrity(parse_fn):
"""Are a table's cells still on ONE line, in order? The metric that predicts retrieval quality."""
rows = [["North","99.94","312","38"], ["South","99.71","540","61"],
["East","99.88","205","44"], ["West","98.92","1288","97"]]
lines = parse_fn(os.path.join(CORPUS, "ops_report_q2.pdf")).splitlines()
return sum(1 for r in rows if any(all(c in line for c in r) for line in lines)), len(rows)
if __name__ == "__main__":
print("=== parser scorecard (PDFs) ===")
for name, fn in [("pypdf", parse_pdf_pypdf), ("pdfplumber", parse_pdf_plumber)]:
for fname, (hits, total, chars) in fact_recall(fn).items():
print(f"{name:11s} {fname:22s} facts {hits}/{total} chars={chars}")
intact, total = row_integrity(fn)
print(f"{name:11s} table row integrity: {intact}/{total}")
print("\n=== pipeline run ===")
docs, seen = [], {}
for fname in sorted(os.listdir(CORPUS)):
path = os.path.join(CORPUS, fname)
if fname.endswith(".pdf"): text, dtype = parse_pdf_plumber(path), "pdf"
elif fname.endswith(".html"): text, dtype = parse_html(path), "html"
else: continue
content_hash = hashlib.sha256(normalize(text).encode()).hexdigest()[:16]
if len(normalize(text)) < 40:
print(f"OCR-FALLBACK {fname}: {len(normalize(text))} chars extracted -> route to OCR queue")
continue
if content_hash in seen:
print(f"DEDUPED {fname}: duplicate of {seen[content_hash]}")
continue
seen[content_hash] = fname
docs.append({"doc_id": content_hash, "source": fname, "doc_type": dtype,
"title": text.strip().splitlines()[0][:80],
"n_chars": len(text), "content_hash": content_hash,
"parsed_at": datetime.now(timezone.utc).isoformat(), "text": text})
with open(os.path.join(os.path.dirname(CORPUS), "documents.jsonl"), "w") as f:
for d in docs:
f.write(json.dumps(d) + "\n")
print(f"kept {len(docs)} unique parseable docs -> documents.jsonl")
Run the scorecard and read it carefully. python ingest.py. Expected fact recall: both parsers score 4/4 on ops_report_q2.pdf and 4/4 on pricing_sheet.pdf, and both score 0/2 on scanned_memo.pdf (0 chars — no text layer exists). If you stopped at fact recall you would call the parsers equivalent. Now the integrity lines: pypdf 0/4 rows intact, pdfplumber 4/4. Open the raw extractions to see why — pypdf emits each table cell on its own line (North, newline, 99.94, newline, 312...), so "West region uptime" and "98.92" end up many lines apart with no structural link; pdfplumber's extract_tables() reconstructs West | 98.92 | 1288 | 97 on one line. A retriever fed pypdf output can match the word "West" but can never return the row that answers "what was West's uptime?" — substring recall ties, retrieval quality does not.
Write the scorecard verdict. Create parser-scorecard.md with a table: parser × (fact recall, table row integrity, chars extracted, scanned-page behavior) and a one-paragraph verdict choosing pdfplumber for this corpus with the integrity metric as the deciding evidence. Add the caveat a senior engineer would: on a corpus of borderless or multi-column tables the ranking could flip — which is why the scorecard is re-run per corpus, and why swapping in Docling or Unstructured later is a config change plus a re-score, not a rewrite.
Trace the pipeline's routing decisions. The pipeline run prints two special-handling lines. DEDUPED runbook_ingest_mirror.html: duplicate of runbook_ingest.html — the mirror survives to the parser but dies at the content hash, because dedup runs on normalized parsed text, catching same-content-different-URL duplicates that a filename or byte-level hash would miss. OCR-FALLBACK scanned_memo.pdf: 0 chars extracted -> route to OCR queue — a length gate under 40 chars routes image-only documents to an OCR queue instead of silently indexing an empty string (the failure mode that produces "the bot doesn't know about the memo" tickets months later). Final count: kept 5 unique parseable docs.
Implement the OCR fallback stub. Add a parse_ocr function that, for the lab, records the routing decision honestly: append the scanned doc to ocr_queue.jsonl with {"source": ..., "reason": "no_text_layer", "queued_at": ...}. In production this queue feeds Tesseract (via pytesseract), a vision-capable LLM, or a managed OCR service — each with a per-page cost, which is exactly why you gate rather than OCR everything. Re-run and confirm the queue file holds one entry.
Audit the metadata against retrieval needs. Every record in documents.jsonl carries doc_id, source, doc_type, title, n_chars, content_hash, parsed_at. Check each field earns its place: content_hash powers incremental re-ingestion (unchanged doc → skip re-parse and re-embed downstream); source powers citations; title powers result display; parsed_at powers freshness monitoring; doc_type powers per-type chunking in the next lab. Add the one field this schema is missing for real deployments — an ACL/visibility tag (even a hardcoded "visibility": "internal") — because retrofitting access control onto an already-embedded corpus is a re-index, and this metadata is precisely what flows into vector-store filters later in the track.
Spot-check the HTML boilerplate stripping. Grep documents.jsonl for Careers and for the nav link text Pricing — zero boilerplate hits, because nav and footer were decomposed before text extraction. Then comment out the tag.decompose() line, re-run, and see every HTML document's text gain identical nav/footer noise — the noise that would otherwise become the most-retrieved "chunk" in the index since it appears in every document. Restore the line. This tiny experiment is the whole argument for parse-time cleaning over embed-time hoping.
Choose parsers with a measured scorecard on your own corpus, and score structure (row integrity), not just substring recall — then make dedup, OCR routing, and metadata capture part of the pipeline, because every downstream RAG stage inherits whatever ingestion lets through.
python make_corpus.py produces 8 files including golden_facts.json.documents.jsonl contains 5 records.ocr_queue.jsonl holds one entry with a reason field.documents.jsonl contains nav/footer boilerplate text, and every record has all seven metadata fields plus your added visibility tag.The ingest-lab/ repo: corpus generator, ingest.py, parser-scorecard.md, documents.jsonl, and ocr_queue.jsonl. Self-grading rubric (pass = 8/10):
pip install -U reportlab pillow) — version skew between them is common.extract_text and table settings to avoid overlap.documents.jsonl has 6+ records: the scanned memo leaked past the length gate — confirm the gate uses normalized length, threshold 40.Client documents are the highest-sensitivity input in this whole track — contracts, invoices, and internal memos routinely contain PII and commercially confidential terms. This lab's pipeline is safe by construction: pypdf, pdfplumber, BeautifulSoup, and the dedup/metadata logic are all local libraries with zero network calls, so the entire bake-off and ingestion run inside the client boundary. When production needs stronger parsing, prefer self-hosted open source (Docling, Unstructured's OSS core, Tesseract for OCR) before any managed parsing API; if a vision-LLM OCR fallback is justified, route it through the client's approved tenancy (Amazon Bedrock or Azure OpenAI) and scrub or mask identifiers first. Keep the parser scorecard as the client deliverable — it is evidence the pipeline was chosen on measured fidelity for their documents.
runbook_ingest_mirror.html is byte-identical content at a different path; only the normalized-content hash catches it. Skipping dedup would double-retrieve the runbook forever.scanned_memo.pdf has no text layer (both parsers extract 0 chars). Correct handling is gate-and-queue for OCR; indexing the empty string silently loses the document.decompose(), every HTML doc gains identical nav/footer text — cross-document repeated noise that pollutes retrieval for every query.