Extract Tables and Fields from PDFs and Scans

Tier: Core · Workflow area: Unstructured data processing

This playbook runs both against a small ground-truth set, picks per document type, and validates every extraction against known totals before anything lands.

When to reach for this

• •
  The source of record is PDF or image: statements, invoices, lab reports, government filings, scanned archives.
• •
  Tables matter — row/column structure must survive, not just the text.
• •
  You can obtain or construct ground truth for a sample (retyped tables, control totals, row counts).

If the documents are born-digital with a text layer and no tables, plain text extraction plus the corpus-extraction playbook is cheaper. If they're uniform forms at massive scale, a dedicated form-recognition service may beat both options here.

The workflow

1. 1.

   Build a 20-document ground-truth set. Pick documents spanning the real variety: clean digital PDFs, low-quality scans, multi-page tables, edge layouts. Manually capture the tables/fields you need, plus per-document control totals (e.g., "this statement's transactions sum to 4,53,210.50").

2. 2.

   Run the parser baseline. Docling (fully OSS) converts PDFs into structured Markdown/JSON with table geometry preserved:

   Copy

   from docling.document_converter import DocumentConverter
   
   converter = DocumentConverter()
   result = converter.convert("statement_2026_03.pdf")
   tables = result.document.tables      # structured rows/cols
   markdown = result.document.export_to_markdown()
   

   Score it against ground truth: cell-level accuracy for tables, exact-match for scalar fields.

3. 3.

   Run the vision-LLM variant on the same set. Send page images with a schema contract so output is typed, and loop page by page — per-page calls keep cost visible and failures isolated:

   Copy

   import anthropic, base64
   from pydantic import BaseModel
   
   class StatementPage(BaseModel):
       transactions: list[dict]     # date, description, amount
       page_total: float | None
       confidence: float
   
   client = anthropic.Anthropic()
   for page_num, png in enumerate(pdf_pages_as_png("statement_2026_03.pdf")):
       response = client.messages.parse(
           model="claude-opus-4-8",
           max_tokens=4096,
           thinking={"type": "adaptive"},
           messages=[{
               "role": "user",
               "content": [
                   {"type": "image", "source": {"type": "base64",
                       "media_type": "image/png",
                       "data": base64.standard_b64encode(png).decode()}},
                   {"type": "text", "text":
                       "Extract every transaction row from this page. "
                       "Copy amounts exactly as printed; never compute or infer them."},
               ],
           }],
           output_format=StatementPage,
       )
       save_page(page_num, response.parsed_output)
   

4. 4.

   Compare on an accuracy-vs-cost scorecard. For each document type: cell accuracy, field accuracy, cost per page, failure modes. Typical pattern — parsers win on clean digital tables at near-zero cost; vision LLMs win on scans, weird layouts, and handwriting at real per-page cost. A hybrid (parser first, vision LLM only for pages the parser flags or fails) is often the production answer.

5. 5.

   Validate every extraction against known totals — the gate. Layered like all extraction gates: schema (enforced by parse() or the parser's structure), allowed values/formats (dates parse, amounts numeric), reconciliation (extracted rows sum to the printed page/statement total; row count matches the printed count where available), confidence threshold, and a human-review queue for anything that fails. Reconciliation is the star here — a statement whose transactions sum to its own printed total is self-verifying:

   Copy

   def gate(page: StatementPage) -> str:
       s = sum(t["amount"] for t in page.transactions)
       if page.page_total is not None and abs(s - page.page_total) > 0.01:
           return "review"
       if page.confidence < 0.85:
           return "review"
       return "accept"
   

6. 6.

   Land with lineage. Warehouse rows carry source_file, page, extractor (docling | vision-llm | human), and gate_status, so a bad batch can be traced and selectively re-extracted.

Verify before you trust it

• •
  Ground-truth scorecard: cell-level table accuracy and field accuracy per document type, per extractor, before choosing the production path.
• •
  Reconciliation pass-rate on the full corpus — the percentage of documents whose internal totals check out without review.
• •
  Cost per 1,000 pages measured, not estimated, for the chosen configuration.
• •
  Review-queue dwell time: extraction that stalls two weeks behind a human queue isn't a pipeline yet.

Pitfalls

• Testing only on clean documents. The demo PDF parses perfectly; production accuracy is set by the worst 20% of scans. Build the ground-truth set from the ugly ones.
• Trusting model-computed totals. See the callout — reconciliation only works when the total is transcribed from the page and summed in your code.
• Whole-document prompts for long PDFs. Accuracy drops and cost balloons; page-level extraction with stitching is more reliable and restartable.
• Ignoring rotated/skewed scans. A deskew/preprocess step (or the parser's OCR options) can move accuracy more than any prompt change.
• One extractor for all document types. Route by type; the scorecard from step 4 exists to justify a hybrid.

Tools

• Doclingfully open-source parser with strong table structure; the free-path default and India-friendly.
• Vision-capable LLM (Claude/GPT)for scans, handwriting, and layouts parsers fumble; per-page cost, so gate its use.
• Unstructured.ioalternative parser with an OSS tier; useful second baseline in the bake-off.

Client-safe variant

Scanned client documents frequently carry signatures, bank details, and personal data, so treat this workflow as PII-handling by default. Process inside the client environment where possible (Docling runs fully offline, which makes the parser path trivially approvable; vision-LLM calls need explicit written sign-off naming the provider and retention terms). Strip or mask account numbers before any external call if the schema doesn't need them. Keep the ground-truth set and scorecard as client-side artifacts — they contain real document content — and hand off the reconciliation gate and review queue as part of the deliverable so accuracy remains monitored after the engagement ends.