Build a Document Parsing Stage for RAG Ingestion

Tier: Foundation · Workflow area: RAG data pipelines

This playbook builds a parser stage the way you build any pipeline stage: swappable components behind a stable interface, with quality metrics that tell you when a parser is failing on your corpus.

When to reach for this

• •
  You are standing up ingestion for a new document corpus (contracts, manuals, reports, wiki exports).
• •
  Retrieval quality is poor and you suspect the text going into the index, not the search over it.
• •
  Your corpus mixes born-digital PDFs with scans and you need an OCR fallback path.
• •
  You are replacing an ad-hoc pdfminer script that silently mangles tables and multi-column layouts.

Skip the heavy machinery if your corpus is plain Markdown or HTML you control — a simple reader plus cleanup is enough.

The workflow

1. 1.

   Profile the corpus before choosing a parser. Count formats, page counts, and the scan share (pages with no extractable text layer):

   Copy

   import fitz  # pymupdf
   from pathlib import Path
   
   stats = {"digital": 0, "scanned": 0}
   for pdf in Path("corpus/").glob("**/*.pdf"):
       doc = fitz.open(pdf)
       text_pages = sum(1 for p in doc if len(p.get_text().strip()) > 50)
       stats["digital" if text_pages / max(len(doc), 1) > 0.5 else "scanned"] += 1
   print(stats)
   

   Expected output: a format census, e.g. {"digital": 412, "scanned": 37}. This decides whether OCR fallback is a corner case or a first-class path.

2. 2.

   Wrap parsers behind one interface. Treat the parser as a swappable component emitting a normalized record — never let downstream chunking depend on one library's output shape:

   Copy

   from docling.document_converter import DocumentConverter
   
   converter = DocumentConverter()  # local-first; LlamaParse/Unstructured API are hosted equivalents
   
   def parse(path: str) -> dict:
       result = converter.convert(path)
       return {
           "doc_id": path,
           "markdown": result.document.export_to_markdown(),
           "n_tables": len(result.document.tables),
           "parser": "docling-v2",
           "parse_status": "ok",
       }
   

   Expected output: Markdown with headings preserved and tables as pipe tables. Docling runs fully local (layout model included); the hosted equivalents are LlamaParse and the Unstructured.io API.

3. 3.

   Build a 20–30 document golden set with known answers. Pick documents that exercise the hard cases — multi-column layouts, tables with merged cells, scanned pages — and record ground truth you can check mechanically: known table totals, known section headings, known entity strings.

4. 4.

   Score each candidate parser on the golden set. The metric that proves this step worked is extraction accuracy on the golden set: percentage of known strings recovered, and table-cell accuracy (correct cells / total cells) on your reference tables.

   Copy

   def score(parsed: dict, expected_strings: list[str]) -> float:
       hits = sum(1 for s in expected_strings if s in parsed["markdown"])
       return hits / len(expected_strings)
   

   Expected output: a per-parser scorecard, e.g. docling: 0.94 strings / 0.88 table cells; pdfminer: 0.71 / 0.32. Pick with numbers, not vibes.

5. 5.

   Add the OCR fallback and dedup gate. Route pages with no text layer to OCR (Docling handles this with its OCR option; Tesseract is the fully local engine). Deduplicate at ingest with a normalized content hash so re-crawled or duplicated files never double-index:

   Copy

   import hashlib
   content_hash = hashlib.sha256(" ".join(md.split()).lower().encode()).hexdigest()
   

6. 6.

   Emit parse-quality metrics per run. Log per-document: parse status, characters extracted per page, table count, OCR-page share. A sudden drop in chars-per-page or table count on a re-run is your earliest signal that a source format changed. Verify the stage by re-running ingestion twice: document count in the store must be identical (dedup works) and golden-set accuracy must not regress.

Verify before you trust it

• •
  Golden-set string recovery at or above your threshold (0.9 is a reasonable bar for born-digital PDFs) and table-cell accuracy measured, not assumed.
• •
  Scanned-document path exercised: at least one known scan parses via OCR and its known strings are recovered.
• •
  Idempotency check: two consecutive ingestion runs produce zero new documents.
• •
  Spot-read five parsed documents end to end — metrics miss reading-order failures that are obvious to a human in seconds.

Pitfalls

• Judging parsers on easy documents. Any library handles single-column born-digital PDFs. Your scorecard must be built from your worst 10% — that is where parsers differ by 3x.
• Tables flattened to prose. Cell adjacency is meaning. If financial or spec tables matter, table-cell accuracy is a first-class metric, not a nice-to-have.
• Silent partial failures. A parser that returns 200 characters from a 40-page document "succeeded". Alert on chars-per-page outliers, not just exceptions.
• Headers, footers, and boilerplate indexed. Page furniture repeated 400 times dominates similarity search. Strip it at parse time, before chunking.
• OCR applied to everything. OCR on born-digital pages is slower and less accurate than the text layer. Route by text-layer detection, not globally.

Tools

• Doclingfully open-source, local-first layout-aware parsing with table structure and OCR; the recommended free path.
• Unstructured.ioOSS tier plus hosted API; broad format coverage including Office and email.
• LlamaIndex readers / LlamaParsereader integrations for quick starts; LlamaParse as the hosted heavy-duty option for complex layouts.

Client-safe variant

On client engagements, parsing is often the stage most likely to touch confidential documents, so default to fully local parsing: Docling and Tesseract run offline with no document leaving the client environment, and the golden-set scorecard works identically air-gapped. If a hosted parser (LlamaParse, Unstructured API) measurably outperforms on the client's document types, present the scorecard delta and get written approval naming the vendor and data categories before any client document crosses the wire; until then, build the golden set from public or synthetic look-alike documents so parser selection can proceed without exposure.