Extract Structured Tables from a Document Corpus at Scale

Tier: Core · Workflow area: Unstructured data processing

The difference between a demo and a production pipeline is everything around the model call — the schema contract, the validation gate, and the sampled human QA that tells you the field-level accuracy you're actually shipping.

When to reach for this

• •
  Hundreds-to-millions of semi-uniform documents (invoices, POs, policies, tickets) need to become queryable warehouse rows.
• •
  The target schema is knowable in advance — you can name the fields, their types, and their allowed values.
• •
  Downstream consumers can tolerate a human-review queue for a minority of documents; nothing here should feed an automated payment run unreviewed.

Don't use this for one-off extractions of ten documents (do it by hand or in a chat session) or for documents where layout is the data (complex nested tables — see the PDF/table extraction playbook).

The workflow

1. 1.

   Write the schema as a Pydantic contract, including allowed values. Every enum you can close, close:

   Copy

   from pydantic import BaseModel
   from typing import Literal, Optional
   
   class InvoiceRecord(BaseModel):
       vendor_name: str
       invoice_number: str
       invoice_date: str            # ISO 8601
       currency: Literal["INR", "USD", "EUR", "GBP"]
       subtotal: float
       tax: float
       total: float
       payment_terms: Optional[str]
       confidence: float            # model's own 0-1 estimate
   

2. 2.

   Hand-label a 30-document golden sample first. Before any pipeline code, extract 30 docs manually into the same schema. This is your accuracy yardstick and your prompt-iteration test set.

3. 3.

   Extract with structured outputs, looping over the corpus. Batch processing is a loop over the API with retries and checkpointing — persist every result as you go so a crash doesn't restart the corpus:

   Copy

   import anthropic
   
   client = anthropic.Anthropic()
   
   for doc in corpus:                      # checkpoint: skip docs already extracted
       response = client.messages.parse(
           model="claude-opus-4-8",
           max_tokens=2048,
           thinking={"type": "adaptive"},
           messages=[{
               "role": "user",
               "content": (
                   "Extract the invoice fields from this document. "
                   "Use null for fields not present; never guess amounts.\n\n"
                   f"{doc.text}"
               ),
           }],
           output_format=InvoiceRecord,
       )
       save_raw_result(doc.id, response.parsed_output)
   

   Expected output per document: a validated InvoiceRecord instance in response.parsed_output — the SDK raises if the model's output doesn't match the schema, which is itself a signal worth logging.

4. 4.

   Run the validation gate before anything lands in the warehouse. Four layers, in order:

   • •
     Schema: already enforced by parse(); log and route any schema failures to review rather than retrying blindly.
   • •
     Allowed values: re-check enums and formats in code (dates parse, currency in the closed set).
   • •
     Business-rule checks: subtotal + tax == total within a tolerance; invoice_date not in the future; amounts non-negative.
   • •
     Confidence threshold + human-review queue: below-threshold records, business-rule failures, and schema failures all go to a review table with the source document attached. Everything else lands in staging.invoice_extractions.

   Copy

   def gate(rec: InvoiceRecord) -> str:
       if rec.currency not in {"INR", "USD", "EUR", "GBP"}: return "review"
       if abs(rec.subtotal + rec.tax - rec.total) > 0.01:   return "review"
       if rec.confidence < 0.85:                             return "review"
       return "accept"
   

5. 5.

   Measure field-level accuracy on the golden sample. Run the pipeline over your 30 labeled docs and score per field: vendor_name 100%, total 97%, payment_terms 83%. Publish this table — it tells consumers which columns to trust and tells you where to iterate on the prompt.

6. 6.

   Land accepted rows with lineage columns. Every warehouse row carries source_doc_id, extraction_model, prompt_version, extracted_at, and gate_status. When the prompt changes, you can re-extract selectively and diff.

Verify before you trust it

• •
  Golden-sample field accuracy meets the bar you agreed with consumers (typically ≥95% on critical fields) before any backfill.
• •
  Review-queue rate is stable and affordable — if 40% of documents route to review, the pipeline isn't production-ready; iterate on the prompt or the schema.
• •
  Reconciliation: for financial documents, sum of extracted totals over a period vs. the known ledger figure.
• •
  Drift check: weekly, re-score a fresh 20-doc sample; document formats change and accuracy decays silently.

Pitfalls

• Skipping the golden sample. Without hand-labeled ground truth you have no accuracy number, only vibes — and consumers will ask.
• Retrying schema failures until they pass. A document the model can't extract cleanly twice belongs in the review queue, not in a retry loop that eventually coerces garbage through.
• One mega-prompt for heterogeneous documents. Route by document type first (even a cheap classifier step); per-type prompts and schemas dramatically beat one universal extractor.
• No checkpointing. A 200K-document run that dies at 180K and restarts from zero doubles your bill.
• Confidence as the only gate. Self-reported confidence misses systematic errors (e.g., consistently reading the due date as the invoice date at confidence 0.95). Business-rule checks catch what confidence can't.

Tools

• Anthropic/OpenAI batch APIs + structured outputsschema-enforced extraction; corpora sized so a lab run stays in single-digit dollars.
• Great Expectationsthe warehouse-side half of the gate: expectations on the landed staging table (nullability, set membership, reconciliation).
• Langfusetrace and cost tracking per document; indispensable when you need to explain the extraction bill.

Client-safe variant

Client documents are usually the most sensitive artifacts on an engagement — invoices name vendors and amounts, contracts name terms. Before any extraction, confirm the client's AI policy in writing: which provider, which region, whether a zero-retention agreement is in place. Prefer running inside the client's own cloud account (their API keys, their warehouse) so documents and extractions never touch your infrastructure. If no external LLM is approved, the identical pipeline — schema contract, gate, golden sample, review queue — runs against a self-hosted model (vLLM on client GPUs) or the client warehouse's native AI functions; expect lower accuracy and budget a larger review queue. Redact or hash direct identifiers in anything you take off-site, including the accuracy report.