Lab 04 · Document-to-Table Pipeline

This is the workhorse pattern behind "~45% of data/analytics postings now mention AI": unstructured documents in, trustworthy rows out, with a measured error rate instead of a shrug.

Time & tier

~2.5 hours hands-on · Core tier · API spend ≤ $0.75 (50–100 cheap-tier calls).

Setup

Python 3.10+, fpdf2 (generation), pypdf (text extraction), pydantic, pandas, openai SDK. The generator writes 50 PDFs across three layout families with planted defects, plus truth.csv (the answer key — do not open until scoring). Deterministic, seed 11.

"""make_invoice_pdfs.py — 50 heterogeneous invoice PDFs + ground truth.
Requires: fpdf2. Writes pdfs/inv_00.pdf ... inv_49.pdf and truth.csv."""
import csv
import os
import random
from fpdf import FPDF

rng = random.Random(11)
os.makedirs("pdfs", exist_ok=True)

VENDORS = ["Kaveri Components Pvt Ltd", "Northwind Traders", "Zenith Office Supply",
           "Bluefin Logistics", "Orchid Print Works", "Mistry & Sons Hardware"]
DESCS = ["Steel brackets (box of 50)", "A4 paper ream", "Courier charges",
         "Toner cartridge", "Cat-6 cable 305m", "Machine oil 5L", "Pallet wrap roll"]
MONTHS = ["Jan", "Feb", "Mar", "Apr", "May", "Jun",
          "Jul", "Aug", "Sep", "Oct", "Nov", "Dec"]

rows = []
for i in range(50):
    vendor = rng.choice(VENDORS)
    inv_no = f"INV-{rng.randint(2026000, 2026999)}"
    y, m, d = 2026, rng.randint(1, 12), rng.randint(1, 28)
    date_iso = f"{y}-{m:02d}-{d:02d}"
    n_items = rng.randint(1, 4)
    items = [(rng.choice(DESCS), rng.randint(1, 12), round(rng.uniform(80, 4000), 2))
             for _ in range(n_items)]
    true_total = round(sum(q * p for _, q, p in items), 2)
    layout = i % 3
    date_txt = [date_iso, f"{d:02d}/{m:02d}/{y}", f"{d} {MONTHS[m-1]} {y}"][layout]
    po = f"PO-{rng.randint(10000, 99999)}"

    defect = "none"
    printed_total = true_total
    if i in (4, 17, 23, 31, 42, 48):     # arithmetic error: printed != sum(lines)
        printed_total = round(true_total + rng.choice([-1, 1]) * rng.uniform(50, 400), 2)
        defect = "total_mismatch"
    if i in (7, 19, 26, 38, 44):         # missing PO number
        po, defect = "", ("missing_po" if defect == "none" else defect + "+missing_po")

    pdf = FPDF()
    pdf.add_page()
    pdf.set_font("Helvetica", size=11)
    if layout == 0:                      # plain top-left block
        pdf.set_font(style="B", size=14); pdf.cell(0, 10, vendor, new_x="LMARGIN", new_y="NEXT")
        pdf.set_font(style="", size=11)
        pdf.cell(0, 8, f"Invoice No: {inv_no}   Date: {date_txt}", new_x="LMARGIN", new_y="NEXT")
        if po: pdf.cell(0, 8, f"PO Ref: {po}", new_x="LMARGIN", new_y="NEXT")
        pdf.ln(4)
        for desc, q, p in items:
            pdf.cell(0, 7, f"{desc:<32} qty {q} @ Rs.{p:,.2f}", new_x="LMARGIN", new_y="NEXT")
        pdf.ln(3); pdf.set_font(style="B")
        pdf.cell(0, 8, f"TOTAL DUE: Rs.{printed_total:,.2f}", new_x="LMARGIN", new_y="NEXT")
    elif layout == 1:                    # bordered table with header band
        pdf.set_fill_color(230, 230, 230)
        pdf.cell(0, 10, f"TAX INVOICE  {inv_no}", fill=True, new_x="LMARGIN", new_y="NEXT")
        pdf.cell(0, 8, f"Supplier: {vendor}", new_x="LMARGIN", new_y="NEXT")
        pdf.cell(0, 8, f"Invoice date: {date_txt}" + (f"   Buyer PO: {po}" if po else ""),
                 new_x="LMARGIN", new_y="NEXT")
        pdf.ln(2)
        pdf.cell(90, 7, "Description", border=1); pdf.cell(25, 7, "Qty", border=1)
        pdf.cell(35, 7, "Rate", border=1); pdf.cell(35, 7, "Amount", border=1,
                                                    new_x="LMARGIN", new_y="NEXT")
        for desc, q, p in items:
            pdf.cell(90, 7, desc, border=1); pdf.cell(25, 7, str(q), border=1)
            pdf.cell(35, 7, f"{p:,.2f}", border=1)
            pdf.cell(35, 7, f"{q * p:,.2f}", border=1, new_x="LMARGIN", new_y="NEXT")
        pdf.cell(150, 8, "Grand Total (INR)", border=1)
        pdf.cell(35, 8, f"{printed_total:,.2f}", border=1, new_x="LMARGIN", new_y="NEXT")
    else:                                # narrative style, numbers inline
        pdf.set_font(style="B", size=13); pdf.cell(0, 10, "PAYMENT REQUEST",
                                                   new_x="LMARGIN", new_y="NEXT")
        pdf.set_font(style="", size=11)
        body = (f"{vendor} requests payment against invoice {inv_no} dated {date_txt}"
                + (f" (purchase order {po})." if po else " (no PO on file)."))
        pdf.multi_cell(0, 7, body, new_x="LMARGIN", new_y="NEXT")
        pdf.ln(2)
        for desc, q, p in items:
            pdf.multi_cell(0, 7, f"Supplied {q} unit(s) of {desc} at {p:,.2f} rupees each.",
                           new_x="LMARGIN", new_y="NEXT")
        pdf.multi_cell(0, 7, f"The amount payable is {printed_total:,.2f} rupees.",
                       new_x="LMARGIN", new_y="NEXT")

    pdf.output(f"pdfs/inv_{i:02d}.pdf")
    rows.append([f"inv_{i:02d}.pdf", vendor, inv_no, date_iso, po,
                 n_items, true_total, printed_total, defect])

with open("truth.csv", "w", newline="") as f:
    w = csv.writer(f)
    w.writerow(["file", "vendor", "invoice_no", "date_iso", "po_number",
                "n_items", "line_sum", "printed_total", "defect"])
    w.writerows(rows)
print("wrote 50 PDFs + truth.csv")

Guided steps

1. 1Parse before you prompt. Extract raw text per PDF with pypdf (PdfReader(f).pages[0].extract_text()). Skim three files, one per layout family — notice the bordered table flattens into space-separated rows and the narrative layout hides amounts in prose. Extraction difficulty lives here, not in the LLM.
2. 2Define the row schema (Lab 03 pattern): vendor, invoice_no (INV-\d{7}), date (ISO), po_number (optional — model must return null, not invent), line_items (desc/qty/rate), printed_total. Add field_confidence: dict[str, float] and instruct the model to self-report per-field confidence in [0,1].
3. 3Extract with the layered harness from Lab 03 (validate → retry → constrained fallback), one call per document, temperature 0, raw text in the prompt. Log tokens.
4. 4Add deterministic cross-checksthis is where DS discipline beats prompt tinkering. For each row compute line_sum = sum(qty*rate) and flag abs(line_sum - printed_total) > 0.01. A mismatch means either your extraction is wrong or the document itself is wrong; both deserve human eyes. Normalize dates from all three styles to ISO in code you own, not in the LLM.
5. 5Route the QA sample. Send to human review: every cross-check flag, every null po_number, every field with self-reported confidence < 0.8, plus random rows to reach 10% minimum (5 docs). Review them against the actual PDFs and record corrections.
6. 6Score against truth.csv (open it only now): per-field accuracy (vendor, invoice_no, date, po, printed_total exact; line_sum within 0.01), plus two honesty metrics — how many of the six seeded total_mismatch documents your cross-check caught, and whether the five missing-PO documents produced nulls or hallucinated PO numbers.
7. 7Ship the tableinvoices.parquet (or CSV) with data columns, confidence columns, qa_status (auto/human-verified/corrected), and source_file lineage.

Verification

• •
  Field accuracy ≥95% on invoice_no, date, printed_total; 100% of missing-PO docs yield null (any invented PO-xxxxx is an automatic red flag on your prompt).
• •
  Cross-check catches ≥5 of the 6 seeded arithmetic-mismatch invoices without consulting truth.csv.
• •
  All three date styles normalized to ISO; per-layout accuracy reported separately (narrative layout is usually worst — say so with numbers).
• •
  QA sample ≥10% and biased toward flags, not purely random.

Completion artifact

extraction-report.md + the final table file. Self-grading rubric (pass = 8/10):

• Accuracy (0–3): headline per-field table vs ground truth, split by layout.
• Defect detection (0–3): seeded mismatches and missing-POs handled as specified.
• QA design (0–2): routing rules stated; corrections traceable to PDFs.
• Lineage & cost (0–2): every row traceable to its file; $/1k documents computed.

Stuck?

• Bordered-table text extracts scrambled. Keep the raw text but tell the model the columns ("rows are: description qty rate amount") — or swap in pdfplumber for layout-aware extraction and compare.
• Model "fixes" the arithmetic-mismatch invoices by returning the computed sum instead of the printed total. Your prompt must demand the total as printed; the cross-check, not the model, judges consistency.
• Confidence always 0.95. Ask for confidence per field with one sentence of justification; uniform confidence means the instruction is being pattern-matched, and you should weight your QA routing toward deterministic flags.

Client-safe variant

Real invoice streams are PII- and finance-sensitive: process only in the client's tenancy, mask GSTIN/bank fields before any LLM call that leaves a controlled environment, and keep the QA sample workflow — it becomes your contractual accuracy SLA evidence. This lab's synthetic generator is also the client-safe way to demo the pipeline in a sales conversation before any data-sharing agreement exists.

Answer key (don't read before attempting)

Seeded defects: invoices at indices 4, 17, 23, 31, 42, 48 print a grand total that differs from the line-item sum by Rs. 50–400 (direction random); invoices 7, 19, 26, 38, 44 have no PO number (layout 2 renders this as "no PO on file"). Layouts rotate i % 3: 0 = plain block with Rs. amounts, 1 = bordered table (space-flattened by pypdf), 2 = narrative prose with amounts in words-adjacent form. Dates: layout 0 ISO, layout 1 DD/MM/YYYY, layout 2 "D Mon YYYY". truth.csv carries both line_sum and printed_total so you can score extraction fidelity (matches printed) separately from document correctness (mismatch defect).