Document intelligence pipeline
Ujwala Deshmukh keeps the accounts at a freight firm in Kolhapur. Two years of invoices and delivery slips sit in a shared drive as phone photos and faint scans. Four people type them into the billing system by hand. Last month a total was keyed as fourteen lakh instead of one lakh, and the payment went out. There is no budget for a scanning vendor and no server to spare. She hands you the folder with one line: read these for me, and tell me when you are not sure.
- Document work is the most common paid AI job in Indian back offices. Every logistics firm, bank and hospital has a paper backlog, and somebody is hired every year to shrink it.
- This project makes you prove that you pick tools by cost and not by fashion. A student who can show what each method actually bought is a student who gets past the second interview round.
- It ends with a running pipeline, a measured accuracy table and a human queue. That is a story you can hold for ten minutes under questions, which no downloaded notebook can give you.
You are not building a model here. You are building a system that turns paper into rows a billing clerk can trust. Ujwala's folder holds two vendors, two layouts and a lot of bad photographs. Every technique you need was taught earlier in the course. The work is choosing, ordering, and proving what you chose.
One honest note on time. The header says 22 hours, which assumes the prerequisites are still warm; a realistic first-time solo range is 25 to 35 hours, and budgeting for that is a decision, not a failure.
Start by making the mess yourself. This module gives you a generator that writes the backlog and the true answers at the same moment. That gives you a gold set — a set of pages where you already know every correct field value. Without one you can measure nothing, and every claim you make is a feeling. With one you can measure field-level accuracy — the share of pages on which a named field came out exactly right, not merely came out.
That word "exactly" carries the project. A pipeline that returns a number for every page looks finished. Module 3.7 drew the line you need here. Returning something and returning the right thing are two different results, and only the second one pays a vendor correctly.
The tempting design is one prompt for every page. It reads well in a demo and it fails in three ways. It spends money on pages a four-line pattern could read for nothing. It gives you no reason you could show an auditor. And it ships the buyer's name, phone number and tax id to an outside company for no gain at all.
The design that survives is an extraction ladder — a fixed order of methods, cheapest and most auditable first, where a page stops at the rung that reads it well enough. Rung one is rules: patterns on the printed label line, free to run, and the pattern is its own explanation. Rung two is similarity from module 3.4: match a damaged value against a known list and keep the score. Rung three is a hosted or local model with the JSON contract discipline of module 3.2, used only on the pages that fell past the first two. Module 3.10 gave you the same rule in one line: use the cheapest tool that clears the bar.
Reading a field is not the same as believing it. An invoice is arithmetic wearing a layout, so the fields must agree with each other. Taxable value plus every tax line must equal the printed total. The tax rate must be one that exists in law, the date must be plausible, and one invoice number must appear once. A rule that fails raises a flag, and a flag is never allowed to pass in silence.
Now the decision that makes this a business system. Every extraction carries a confidence, and you must set a confidence gate — the score above which a page is posted with no person in the way. Everything under it goes to a review queue — the list of pages a human opens, corrects and signs. Both ends of that dial cost money. A wrong bill paid automatically costs thousands of rupees and weeks of chasing, while a review costs a few staff minutes, so you price both and sweep the gate exactly as module 1.5 taught.
These documents are made of personal data. Names, addresses, phone numbers and tax ids sit on every page, and India's DPDP Act binds whoever holds them. So redaction — swapping personal details for neutral tags before text is stored or sent — is part of the pipeline and not an afterthought. Module 3.11 set the standard, and this project asks you the harder question it raises. What may leave the building, to which vendor, for how long, and who decided?
Two production judgements close the build. This is a backlog, not a website, so module 4.9's answer is batch: one run over many pages, restartable, with no user waiting. And the thing that will break it is a vendor changing a template, which is drift wearing a new costume, so module 4.8's monitoring must report per vendor and never only on the average.
The defence is the assessment, and ruling 2 of this course applies here without softening. If you cannot explain your own ladder and change your own code while somebody watches, the project scores zero. That holds even when every number on the page is good.
Six artefacts, not a tutorial. Every method here was taught in Phase 3 or Phase 4; what follows is the standard your work is measured against. Read all six before you write a line of code.
Artefact 1 — the corpus generator. Save this as make_corpus.py, run it once from your repo root, and never edit it. It uses the Python standard library only: no numpy, no GPU, no paid OCR, and nothing to install. It writes the scanned text of every page and, beside it, the true field values that page was built from. If Pillow happens to be installed it also renders forty sample page images so you can see what you are working with, and if Pillow is missing the run is unaffected — the text corpus is the graded path.
"""make_corpus.py - run once, then never edit. Writes corpus/ and gold/labels.jsonl.
A scanned-document backlog from a Kolhapur logistics firm: two invoice vendors with
different layouts, plus proof-of-delivery slips. Seeded, so every student in the batch
works the identical backlog. Standard library only - no numpy, no GPU, no paid OCR.
"""
import json
import random
import re
from datetime import date, timedelta
from pathlib import Path
RNG = random.Random(20250211)
N_INV_A, N_INV_B, N_POD = 240, 200, 160
OUT = Path("corpus")
GOLD = Path("gold")
VEND_A = "SAHYADRI CARRIERS PVT LTD"
VEND_B = "BHIMA FREIGHT LINES"
GSTIN_A, GSTIN_B = "27AABCS1429L1ZP", "27AACCB8811M1Z4"
START = date(2024, 4, 1)
BUYERS = ["Kolhapur Steel Traders", "Panchganga Agro Mills", "Rankala Cotton Co",
"Ichalkaranji Textiles", "Shiroli Auto Parts", "Warana Dairy Supplies",
"Gadhinglaj Cement Depot", "Jaysingpur Paper Works", "Hupari Silver Mart",
"Kagal Poly Packs", "Radhanagari Timber", "Chandgad Rubber Traders"]
CONTACTS = ["R. Kamble", "S. Patil", "A. Mulla", "V. Jadhav", "D. Chougule",
"P. Sutar", "M. Kadam", "N. Bhosale", "T. Salunkhe", "G. Nadaf"]
STATES = [("Maharashtra", "27"), ("Karnataka", "29"), ("Goa", "30"),
("Gujarat", "24"), ("Telangana", "36")]
LINE_ITEMS = ["Freight - Pune route", "Freight - Belagavi route", "Loading charges",
"Unloading charges", "Detention charges", "Door delivery",
"Hamali", "Toll reimbursement", "Warehouse handling"]
DEVA_ADDR = ["पता : प्लॉट १४, एमआयडीसी शिरोली, कोल्हापूर ४१६१२२",
"पत्ता : ट्रान्सपोर्ट नगर, सोलापूर रोड, कोल्हापूर",
"टीप : माल पोहोचल्यावर पावती द्यावी"]
DEVA_POD = ["प्राप्तकर्ता स्वाक्षरी आवश्यक आहे",
"माल सुस्थितीत मिळाला"]
CONFUSE = {"0": "O", "O": "0", "1": "l", "l": "1", "5": "S", "S": "5",
"8": "B", "B": "8", "2": "Z", "I": "1", "6": "G", "G": "6"}
# a hand-written name comes back far worse than print, and in a different way
HAND = {"a": "o", "o": "a", "n": "m", "m": "rn", "u": "v", "r": "n", "e": "c",
"k": "h", "t": "f", "s": "5", "l": "i", "i": "l", "d": "cl",
"K": "H", "P": "R", "J": "I", "M": "N", "V": "U", "G": "C"}
# every noise event is counted, and the counts go into the receipt
tally = {k: 0 for k in [
"char_confusion_chars", "decimal_read_as_comma", "comma_lost", "space_in_amount",
"missing_invoice_date", "missing_gstin", "missing_place_of_supply",
"missing_consignment", "line_dropped", "adjacent_lines_swapped",
"stray_marks", "devanagari_lines", "handwriting_chars", "vendor_arithmetic_error"]}
def inr(n):
"""Indian digit grouping: 1234567 -> 12,34,567"""
s = str(int(abs(n)))
if len(s) <= 3:
head, tail = "", s
else:
head, tail = s[:-3], s[-3:]
parts = []
while len(head) > 2:
parts.insert(0, head[-2:])
head = head[:-2]
if head:
parts.insert(0, head)
head = ",".join(parts) + ","
return f"{head}{tail}.00"
def confuse(text, p):
out = []
for ch in text:
if ch in CONFUSE and RNG.random() < p:
out.append(CONFUSE[ch])
tally["char_confusion_chars"] += 1
else:
out.append(ch)
return "".join(out)
def handwrite(text, p):
"""Corrupt only the value after the colon - the printed label survives."""
if ":" not in text:
return text
label, val = text.split(":", 1)
out = []
for ch in val:
if ch in HAND and RNG.random() < p:
out.append(HAND[ch])
tally["handwriting_chars"] += 1
else:
out.append(ch)
return label + ":" + "".join(out)
def scan(lines, p_char=0.013, p_stray=0.045, p_drop=0.015, p_swap=0.05):
"""Turn clean printed lines into what a cheap scan-plus-OCR gives back."""
out = []
for ln in lines:
if ln.strip() and RNG.random() < p_drop:
tally["line_dropped"] += 1
continue
ln = confuse(ln, p_char)
ln = " " * RNG.randint(0, 4) + ln
if RNG.random() < p_stray:
ln = ln + RNG.choice([" .", " ·", " ~", " ,"])
tally["stray_marks"] += 1
out.append(ln.rstrip() if RNG.random() < 0.5 else ln)
for i in range(len(out) - 1):
if out[i].strip() and out[i + 1].strip() and RNG.random() < p_swap * 0.15:
out[i], out[i + 1] = out[i + 1], out[i]
tally["adjacent_lines_swapped"] += 1
return out
def money_noise(text, amount_str):
"""The expensive one: a decimal point read as a thousands comma."""
r = RNG.random()
if r < 0.045:
tally["decimal_read_as_comma"] += 1
return text.replace(amount_str, amount_str[:-3] + "," + amount_str[-2:])
if r < 0.085:
tally["comma_lost"] += 1
return text.replace(amount_str, amount_str.replace(",", "", 1))
if r < 0.120:
tally["space_in_amount"] += 1
return text.replace(amount_str, amount_str.replace(",", ", ", 1))
return text
def build_invoice_a(i, dt, buyer, state, items, taxable, cgst, sgst, total,
inv_no, cn_no, phone):
L = [VEND_A,
"Plot 14, MIDC Shiroli, Kolhapur 416122",
f"GSTIN : {GSTIN_A} Phone : {phone}",
"-" * 52,
"TAX INVOICE",
f"Invoice No : {inv_no}",
f"Invoice Date : {dt.strftime('%d/%m/%Y')}",
f"Consignment No : {cn_no}",
f"Buyer : {buyer}",
f"Place of Supply : {state[0]} ({state[1]})",
"-" * 52,
"Sr Description Qty Amount"]
for n, (desc, qty, amt) in enumerate(items, 1):
L.append(f"{n} {desc:<26}{qty:>3} {inr(amt):>12}")
L += ["-" * 52,
f"Taxable Value Rs. {inr(taxable):>14}",
f"CGST 9% Rs. {inr(cgst):>14}",
f"SGST 9% Rs. {inr(sgst):>14}",
f"Grand Total Rs. {inr(total):>14}",
"E & O E. Subject to Kolhapur jurisdiction"]
return L
def build_invoice_b(i, dt, buyer, state, items, taxable, igst, total,
inv_no, lr_no, phone):
L = [VEND_B,
"Transport Nagar, Solapur Road, Kolhapur",
f"GSTIN {GSTIN_B} Mob {phone}",
"==== TRANSPORT INVOICE ====",
f"INV# {inv_no}",
f"Dated {dt.strftime('%d-%b-%Y')}",
f"LR No {lr_no}",
f"Bill To: {buyer}",
f"Ship To State: {state[0]} ({state[1]})",
"Particulars Amount(INR)"]
for desc, qty, amt in items:
L.append(f"{desc:<32}{inr(amt):>12}")
L += [f"{'Sub Total':<32}{inr(taxable):>12}",
f"{'IGST @ 18%':<32}{inr(igst):>12}",
f"{'TOTAL PAYABLE':<32}{inr(total):>12}",
"Payment due in 30 days. Cheques favouring Bhima Freight Lines."]
return L
def build_pod(carrier, cn_no, inv_ref, dt, deliver_dt, receiver, pkgs, phone):
return ["PROOF OF DELIVERY",
f"Carrier : {carrier}",
f"Consignment No : {cn_no}",
f"Invoice Ref : {inv_ref}",
f"Despatched On : {dt.strftime('%d/%m/%Y')}",
f"Delivered On : {deliver_dt.strftime('%d/%m/%Y')}",
f"Packages : {pkgs}",
"Condition : OK",
f"Received By : {receiver}",
f"Contact : {phone}",
"Signature : ______________________"]
def drop_field(lines, prefix, key):
"""A field the scan lost completely - the truth still exists, the pixels do not."""
for idx, ln in enumerate(lines):
if ln.startswith(prefix):
lines.pop(idx)
tally[key] += 1
return True
return False
def write_doc(doc_id, lines):
(OUT / "text").mkdir(parents=True, exist_ok=True)
(OUT / "text" / f"{doc_id}.txt").write_text("\n".join(lines) + "\n", encoding="utf-8")
records, docs = [], []
inv_index = {}
# ---------- 1. vendor A invoices ----------
for i in range(N_INV_A):
dt = START + timedelta(days=RNG.randint(0, 300))
buyer, state = RNG.choice(BUYERS), STATES[0]
n_items = RNG.randint(1, 3)
items = [(RNG.choice(LINE_ITEMS), RNG.randint(1, 6), RNG.randrange(600, 90000, 50))
for _ in range(n_items)]
taxable = sum(a for _, _, a in items)
cgst = sgst = round(taxable * 0.09)
total = taxable + cgst + sgst
printed_total = total
if RNG.random() < 0.04: # the vendor's own arithmetic slip
printed_total = total + RNG.choice([-100, -50, 90, 200, 1000])
tally["vendor_arithmetic_error"] += 1
inv_no = f"SC/24-25/{1200 + i:05d}"
cn_no = f"CN{4400000 + RNG.randint(0, 99999)}"
phone = f"9{RNG.randint(200000000, 899999999)}"
lines = build_invoice_a(i, dt, buyer, state, items, taxable, cgst, sgst,
printed_total, inv_no, cn_no, phone)
if RNG.random() < 0.20:
lines.insert(2, RNG.choice(DEVA_ADDR)); tally["devanagari_lines"] += 1
if RNG.random() < 0.07: drop_field(lines, "Invoice Date", "missing_invoice_date")
if RNG.random() < 0.09: drop_field(lines, "GSTIN", "missing_gstin")
if RNG.random() < 0.11: drop_field(lines, "Place of Supply", "missing_place_of_supply")
if RNG.random() < 0.06: drop_field(lines, "Consignment No", "missing_consignment")
body = "\n".join(scan(lines))
body = money_noise(body, inr(printed_total))
doc_id = f"DOC{len(docs) + 1:05d}"
docs.append((doc_id, body.split("\n")))
inv_index[inv_no] = (doc_id, cn_no, dt, buyer)
records.append(dict(doc_id=doc_id, doc_type="invoice", vendor="sahyadri",
invoice_no=inv_no, invoice_date=dt.isoformat(),
consignment_no=cn_no, gstin=GSTIN_A, buyer=buyer,
place_of_supply=state[1], taxable_value=taxable,
tax_total=cgst + sgst, printed_total=printed_total,
arithmetic_consistent=(printed_total == total),
phone=phone, is_duplicate_of=None))
# ---------- 2. vendor B invoices ----------
for i in range(N_INV_B):
dt = START + timedelta(days=RNG.randint(0, 300))
buyer, state = RNG.choice(BUYERS), RNG.choice(STATES[1:])
n_items = RNG.randint(1, 3)
items = [(RNG.choice(LINE_ITEMS), RNG.randint(1, 6), RNG.randrange(600, 90000, 50))
for _ in range(n_items)]
taxable = sum(a for _, _, a in items)
igst = round(taxable * 0.18)
total = taxable + igst
printed_total = total
if RNG.random() < 0.04:
printed_total = total + RNG.choice([-100, -50, 90, 200, 1000])
tally["vendor_arithmetic_error"] += 1
inv_no = f"BFL-2024-{4400 + i}"
lr_no = f"{88000 + RNG.randint(0, 999)}"
phone = f"9{RNG.randint(200000000, 899999999)}"
lines = build_invoice_b(i, dt, buyer, state, items, taxable, igst,
printed_total, inv_no, lr_no, phone)
if RNG.random() < 0.20:
lines.insert(2, RNG.choice(DEVA_ADDR)); tally["devanagari_lines"] += 1
if RNG.random() < 0.07: drop_field(lines, "Dated", "missing_invoice_date")
if RNG.random() < 0.09: drop_field(lines, "GSTIN", "missing_gstin")
if RNG.random() < 0.11: drop_field(lines, "Ship To State", "missing_place_of_supply")
if RNG.random() < 0.06: drop_field(lines, "LR No", "missing_consignment")
body = "\n".join(scan(lines))
body = money_noise(body, inr(printed_total))
doc_id = f"DOC{len(docs) + 1:05d}"
docs.append((doc_id, body.split("\n")))
inv_index[inv_no] = (doc_id, lr_no, dt, buyer)
records.append(dict(doc_id=doc_id, doc_type="invoice", vendor="bhima",
invoice_no=inv_no, invoice_date=dt.isoformat(),
consignment_no=lr_no, gstin=GSTIN_B, buyer=buyer,
place_of_supply=state[1], taxable_value=taxable,
tax_total=igst, printed_total=printed_total,
arithmetic_consistent=(printed_total == total),
phone=phone, is_duplicate_of=None))
# ---------- 3. proof-of-delivery slips ----------
inv_keys = list(inv_index.keys())
for i in range(N_POD):
inv_no = RNG.choice(inv_keys)
_, cn_no, dt, buyer = inv_index[inv_no]
carrier = VEND_A if inv_no.startswith("SC/") else VEND_B
deliver_dt = dt + timedelta(days=RNG.randint(1, 9))
receiver = RNG.choice(CONTACTS)
pkgs = RNG.randint(1, 12)
phone = f"9{RNG.randint(200000000, 899999999)}"
lines = build_pod(carrier, cn_no, inv_no, dt, deliver_dt, receiver, pkgs, phone)
if RNG.random() < 0.25:
lines.insert(1, RNG.choice(DEVA_POD)); tally["devanagari_lines"] += 1
if RNG.random() < 0.09:
drop_field(lines, "Delivered On", "missing_invoice_date")
scanned = scan(lines, p_char=0.010)
# the receiver's name is handwritten, so it comes back far worse than print
for idx, ln in enumerate(scanned):
if "Received By" in ln:
scanned[idx] = handwrite(ln, 0.30)
doc_id = f"DOC{len(docs) + 1:05d}"
docs.append((doc_id, scanned))
records.append(dict(doc_id=doc_id, doc_type="pod", vendor=(
"sahyadri" if carrier == VEND_A else "bhima"), invoice_no=inv_no,
invoice_date=deliver_dt.isoformat(), consignment_no=cn_no, gstin=None,
buyer=buyer, place_of_supply=None, taxable_value=None, tax_total=None,
printed_total=None, arithmetic_consistent=None, phone=phone,
receiver_name=receiver, packages=pkgs, is_duplicate_of=None))
# ---------- 4. re-scans: the same paper through the scanner twice ----------
base_invoices = [r for r in records if r["doc_type"] == "invoice"]
n_dup = int(0.06 * len(base_invoices))
for src in RNG.sample(base_invoices, n_dup):
orig_lines = dict(docs)[src["doc_id"]]
clean = [re.sub(r"\s+$", "", ln) for ln in orig_lines]
body = "\n".join(scan(clean, p_char=0.014, p_stray=0.06))
doc_id = f"DOC{len(docs) + 1:05d}"
docs.append((doc_id, body.split("\n")))
dup = dict(src); dup["doc_id"] = doc_id; dup["is_duplicate_of"] = src["doc_id"]
records.append(dup)
# ---------- 5. write the corpus and the labels ----------
for doc_id, lines in docs:
write_doc(doc_id, lines)
GOLD.mkdir(exist_ok=True)
with (GOLD / "labels.jsonl").open("w", encoding="utf-8") as fh:
for r in records:
fh.write(json.dumps(r, ensure_ascii=False) + "\n")
# ---------- 6. optional: render page images if Pillow happens to be installed ----------
try:
from PIL import Image, ImageDraw
(OUT / "images").mkdir(exist_ok=True)
for doc_id, lines in docs[:40]:
img = Image.new("L", (760, 1000), 245)
d = ImageDraw.Draw(img)
for k, ln in enumerate(lines[:56]):
d.text((28 + RNG.randint(0, 3), 24 + k * 17), ln, fill=40)
img.save(OUT / "images" / f"{doc_id}.png")
img_note = f"{min(40, len(docs))} sample page images written to corpus/images/"
except ImportError:
img_note = "Pillow not installed - text corpus only, which is the graded path"
# ---------- 7. the receipt you paste into your README ----------
inv = [r for r in records if r["doc_type"] == "invoice"]
pod = [r for r in records if r["doc_type"] == "pod"]
n = len(records)
print("documents written :", n)
print(" invoices / PODs :", len(inv), "/", len(pod))
print(" vendor sahyadri / bhima :", sum(1 for r in records if r["vendor"] == "sahyadri"),
"/", sum(1 for r in records if r["vendor"] == "bhima"))
print("duplicate re-scans :", sum(1 for r in records if r["is_duplicate_of"]))
print("unique invoice numbers :", len({r["invoice_no"] for r in inv}))
print("arithmetic-broken invoices :", sum(1 for r in inv if r["arithmetic_consistent"] is False))
print("--- noise the scanner added, counted exactly ---")
for k in sorted(tally):
rate = tally[k] / n
print(f" {k:<26}: {tally[k]:>6} ({rate:.3f} per document)")
print("gold labels :", len(records), "rows ->", GOLD / "labels.jsonl")
print("images :", img_note)
On CPython 3.14 with no third-party packages the receipt reads: documents written 626, invoices / PODs 466 / 160, vendor sahyadri / bhima 333 / 293, duplicate re-scans 26, unique invoice numbers 440, arithmetic-broken invoices 10, then the noise counts — char_confusion_chars 670, line_dropped 181, stray_marks 458, devanagari_lines 143, handwriting_chars 236, adjacent_lines_swapped 84, missing_place_of_supply 49, missing_invoice_date 39, missing_gstin 37, missing_consignment 30, decimal_read_as_comma 20, space_in_amount 14, comma_lost 12, vendor_arithmetic_error 9. Read those lines as findings, not as decoration. There are 466 invoices but only 440 distinct invoice numbers, so 26 pages are the same paper scanned twice and a duplicate check is not optional. Ten invoices carry a total that does not equal subtotal plus tax, and nine of those are the vendor's own mistake while the tenth is a re-scan of one of them — meaning your arithmetic rule will find real errors on real bills, not only extraction errors. Twenty pages have the grand total's decimal point read as a comma, which is the exact failure that cost Ujwala fourteen lakh, and 236 characters of a hand-written receiver name came back wrong, which is where rules will die and similarity will earn its place. Paste the receipt into your README before you write an extractor, because a reviewer who sees these lines knows you looked at your data.
If your numbers are close but not identical, triage in a fixed order before touching the generator. Every count off by a little points at the interpreter: run python --version and compare against the CPython 3.14 named above, because the receipt was produced on it. One count wildly off while the rest agree points at stale state: run make_corpus.py fresh from the top in a new interpreter, never piece by piece, since seeded draws only reproduce in the original order. Only when both checks pass and the receipt still disagrees should you suspect your copy of the script — diff it character by character against the printed one.
Artefact 2 — the acceptance criteria. This is the whole marking scheme for the build stages, with nothing hidden behind it. Print it and tick rows.
# P2 ACCEPTANCE CRITERIA. A stage is DONE when every row is true.
STAGE 1 - CORPUS, GOLD SET, RULES BASELINE
1.1 make_corpus.py committed unedited; its receipt pasted into the
README with the Python version that produced it.
1.2 README states the field list you will extract, and for each
field one line saying what a wrong value would cost the firm.
1.3 A document dictionary: for each of the two invoice layouts and
the POD slip, where every field sits and what its label reads.
1.4 gold/labels.jsonl loaded by src/gold.py; the loader asserts
626 rows, 466 invoices, 160 PODs, 440 unique invoice numbers.
1.5 Rules-only baseline in src/rules.py, scored per field AND per
vendor, with counts of right / no-value / wrong, never one
accuracy number on its own.
1.6 The eval report distinguishes "returned nothing" from
"returned the wrong thing" for every field. A field with 0.00
accuracy is investigated in writing, not averaged away.
1.7 The money parser is tested on all five ugly forms:
1,42,367.00 · 1,42,367,00 · 142367.00 · 1,42, 367.00 · 32,00O.00
STAGE 2 - THE EXTRACTION LADDER
2.1 Every extractor returns the SAME contract object (artefact 4),
including a confidence and the rung that produced the value.
2.2 Rung 2 uses similarity against a known list for at least one
field, with the score kept as the confidence. Module 3.4's
embeddings or a character-similarity fallback, your choice,
and the choice is defended in one sentence.
2.3 Rung 3 calls a model with a strict JSON schema, validates the
reply against the contract, and retries or gives up - never
passes an unparsed reply downstream. Module 3.2's discipline.
2.4 A page reaches rung 3 only after rungs 1 and 2 failed. The
report states how many pages each rung kept.
2.5 Ladder accuracy is compared to the stage-1 baseline field by
field. Any field where the ladder did NOT beat rules is named,
and that field goes back to rules.
2.6 Cost table: pages sent to rung 3, tokens or seconds per page,
and the rupee or minute cost of one full backlog run.
STAGE 3 - VALIDATION, CONFIDENCE, THE HUMAN QUEUE
3.1 src/rules_business.py implements at least five checks: totals
reconcile, tax rate is legal, date is plausible and not in the
future, invoice number is unique, POD references a real
invoice.
3.2 Every check failure is a FLAG carrying the doc id, the rule
name and the two numbers that disagreed. No silent passes.
3.3 The duplicate check finds the 26 re-scans and reports them by
invoice number, not by file name.
3.4 cost_of_errors.md prices two things with a stated source or a
flagged estimate: one wrong auto-approval, and one human
review.
3.5 A gate sweep over at least 5 thresholds showing auto-approved,
wrongly auto-approved, reviewed, rejected and total cost.
3.6 The shipped gate is the cheapest row, it lives in config.py,
and the decision log carries the sentence that chose it.
3.7 queue.jsonl is produced: one row per page needing a human,
with the reason, sorted so the costliest pages come first.
STAGE 4 - BATCH PIPELINE, REPO, TESTS, DPDP
4.1 Repo matches artefact 3. Clean clone to a finished run in
three commands, proven in the README.
4.2 run_batch.py processes the whole corpus in one restartable
run, writes a manifest, and can resume without redoing pages.
4.3 README states why this is a batch job and not an API, in two
sentences, using module 4.9's reasoning.
4.4 pytest suite of at least 10 tests: money parser, one rule, one
business check, the contract shape, the redactor, and a
QUALITY GATE test that fails the build when per-vendor field
accuracy drops below your floor.
4.5 redact.py replaces buyer name, receiver name, phone, GSTIN and
address with stable tags. A test asserts no raw value survives
on a page containing all five.
4.6 No personal data reaches any log. A test greps the run log for
a phone pattern and a GSTIN pattern and asserts zero hits.
4.7 data_flow.md: what leaves the machine, what is removed first,
to which vendor, retention, and who decided. If nothing leaves,
that is written down as a decision too.
4.8 Dockerfile builds; the container completes a run over a small
sample; requirements pinned with ==.
STAGE 5 - MONITORING AND DEFENCE
5.1 monitoring_plan.md names at least five checks, each with a
threshold, a window, an owner and an action. At least three
are computed PER VENDOR.
5.2 The plan says exactly what a new vendor layout looks like in
the numbers, and what happens on the day one appears.
5.3 Canary pages: five known pages per vendor in the test suite,
failing the build when any vendor's canary drops.
5.4 decisions.md has 10+ dated entries, each with the option you
rejected and the reason.
5.5 Defence rehearsed against artefact 5, with two live
modifications timed.
Row 1.6 is the row that separates this project from a demo, and it is where most submissions quietly lose their credibility: a field that returns something for every page can still be right on none of them. Row 2.5 exists because students add rungs for the sake of the diagram — if the ladder does not beat plain rules on a field, the rung is decoration and it must come out. Row 3.2's flag format is what a reviewer will grep for at the start of the defence, so a flag that says only "failed" is worth nothing. Row 4.6 catches the failure that outlives every other fix, because a redacted API call and an unredacted log leak the same data to different people. Rows 5.1 to 5.3 are the ones that would have caught a vendor going to zero while the average held, which is the whole of WR-179.
Artefact 3 — the repo you must produce. Not a suggestion. A reviewer opens these paths by name.
doc-intel-pipeline/
README.md what it is, the 3-command run, the receipt,
the accuracy table BY VENDOR, known limits
requirements.txt every line pinned with ==
Dockerfile copies requirements first, then the code
config.py SEED, FIELDS, GATE, COST_WRONG, COST_REVIEW,
VENDOR_PATTERNS, LLM_ENABLED - every knob here
make_corpus.py given to you; run once; never edited
corpus/ gitignored. text/ and images/ land here
gold/ gitignored. labels.jsonl lands here
src/
contract.py the ExtractedField / DocumentRecord models
gold.py loads labels, asserts the corpus receipt
money.py parse_amount() - the five ugly forms
rules.py RUNG 1: per-vendor label patterns
similar.py RUNG 2: nearest known value + score
llm.py RUNG 3: JSON-schema call, validate, retry
ladder.py run the rungs in order, stop when good
rules_business.py arithmetic, tax, dates, uniqueness, POD link
confidence.py combine rung score + rule flags -> one number
queue.py build queue.jsonl, costliest page first
redact.py names, phones, GSTIN, address -> tags
evaluate.py field-level accuracy, per vendor, per rung
run_batch.py entry point: whole corpus, restartable
review_app.py optional: page text beside extracted fields
tests/
test_money.py the five ugly amount forms
test_rules.py one field on each of the two layouts
test_business.py a broken total is flagged, a good one is not
test_contract.py every extractor returns the same shape
test_redact.py no raw phone/GSTIN/name survives
test_logs.py the run log contains no personal data
test_quality.py THE GATE: per-vendor accuracy >= your floor
test_canary.py five known pages per vendor still parse
monitor/
per_vendor.py accuracy, no-value rate, page count by vendor
layout_watch.py alarm when a vendor's no-value rate jumps
docs/
document_dictionary.md both layouts, field by field
cost_of_errors.md the two prices and where they came from
data_flow.md what leaves, what is redacted, who decided
monitoring_plan.md checks, thresholds, owners, actions
decisions.md the dated log
Four paths in this tree are load-bearing. src/contract.py exists so that rung one, rung two and rung three are interchangeable: a reviewer who asks "can you turn the model off?" gets a one-line answer instead of a refactor. src/money.py is a separate file with its own test because one bad amount parser is the single most expensive bug in this whole domain, and it deserves to be tested alone. src/redact.py sits on the boundary so that no future call can skip it by accident, which is exactly the WR-180 lesson. And monitor/per_vendor.py exists because a metric with no vendor column beside it cannot see the failure that actually happens.
Artefact 4 — the extraction contract. Every extractor on every rung returns this shape, or it does not ship. This is module 3.2's contract discipline applied to your own code and not only to the model's reply.
# src/contract.py - the only shape any extractor may return.
from typing import Literal, Optional
from pydantic import BaseModel, Field, field_validator
Rung = Literal["rules", "similarity", "llm", "none"]
class ExtractedField(BaseModel):
"""One field, one page. The value AND the story of how it got here."""
name: str # "total_amount", "invoice_no", ...
value: Optional[str] = None # None means honestly not found
confidence: float = Field(ge=0.0, le=1.0) # 0.0 when value is None
rung: Rung # which rung produced this
evidence: Optional[str] = None # the raw line it came from
span: Optional[tuple[int, int]] = None # char offsets into page text
@field_validator("confidence")
@classmethod
def zero_when_empty(cls, v, info):
if info.data.get("value") is None and v != 0.0:
raise ValueError("no value means no confidence")
return v
class Flag(BaseModel):
"""A business rule that did not hold. Never a warning in a log."""
rule: str # "totals_reconcile"
detail: str # "13,550 + 2,439 != 1,59,890"
severity: Literal["block", "review"]
class DocumentRecord(BaseModel):
doc_id: str
doc_type: Literal["invoice", "pod", "unknown"]
vendor: Literal["sahyadri", "bhima", "unknown"]
fields: dict[str, ExtractedField]
flags: list[Flag] = []
page_confidence: float = Field(ge=0.0, le=1.0)
decision: Literal["auto_approve", "review", "reject"]
duplicate_of: Optional[str] = None
redacted: bool = False # True before anything leaves
@field_validator("decision")
@classmethod
def flags_block_auto(cls, v, info):
if v == "auto_approve":
if any(f.severity == "block" for f in info.data.get("flags", [])):
raise ValueError("a blocking flag can never auto-approve")
return v
Three things in this contract are doing real work. value: Optional[str] = None with the validator beneath it makes "I did not find it" a first-class answer with zero confidence, so your evaluation can separate silence from a wrong guess — which is acceptance row 1.6 enforced in code. rung is what lets you print the page count per rung at the defence, and without it you will be guessing about your own system. And flags_block_auto is the rule from WR-178 written where it cannot be forgotten: a confidence score alone may never open the gate, because confidence measures whether a value was found and a business rule measures whether it can be true.
Artefact 5 — the defence rubric. Forty marks, and the build stages cannot rescue a weak defence.
# P2 GRADING. Total 100.
BUILD - 60 marks, from the acceptance criteria above
Stage 1 corpus, gold set, rules baseline .......... 12
Stage 2 the ladder, measured against the baseline . 15
Stage 3 business rules, gate, human queue ......... 15
Stage 4 batch pipeline, repo, tests, DPDP ......... 12
Stage 5 per-vendor monitoring, decision log ....... 6
DEFENCE - 40 marks, live, 10 minutes, no notes on screen
D1 WALKTHROUGH (6)
Full marks: 90 seconds, four parts in order - problem,
approach, result, limit - ending with an offer to go deeper.
Half marks: right content, over three minutes, no offer.
Zero: a tool list, or a scroll through the repo.
D2 NUMBERS (8)
Full marks: corpus size, per-vendor accuracy for your worst
field, pages kept by each rung, the gate and its two prices,
the review queue length - each inside three seconds.
Half: two lookups needed.
Zero: cannot say what share of pages a human still opens.
D3 THE LADDER (8)
Full marks: for one named field, the accuracy at each rung,
the cost of each rung, and the sentence saying why rung 3 was
or was not worth it there.
Half: describes the rungs, no numbers per rung.
Zero: "the LLM handles the hard ones" with nothing behind it.
D4 THE GATE (6)
Full marks: names the two prices, shows the sweep, states the
chosen gate, and says what would move it.
Half: gate stated, prices not sourced.
Zero: a default threshold with no table behind it.
D5 DATA AND LIMITS (6)
Full marks: says what personal data exists, what leaves the
machine, what is redacted first, who decided - then volunteers
three limits before being asked, one about the corpus itself.
Zero: argues a real weakness is not a weakness.
D6 LIVE MODIFICATION (6)
Two changes, in your own code, while watched. Examples: move
the gate to a new cost ratio and re-run the sweep; break one
vendor's label pattern and show the per-vendor report catching
it; add a sixth business rule and make its test pass.
Full marks: names the file, the function, the number that will
move and its direction, edits it, runs it, reads the result,
and says whether the prediction was right.
Half: correct reasoning, cannot complete the edit.
Zero: asks for time, or describes the change in general words.
THE OVERRIDE, and it is not negotiable:
A project the student cannot explain and cannot modify scores
ZERO overall, whatever the accuracy table says and however
complete the repo looks. Build marks are released only after D1
to D6 are attempted. This is ruling 2 of the course: the defence
IS the assessment, because generated code cannot be detected and
an owner can always be questioned.
Read D3 twice, because it is the row this project exists for. Almost every student can say "rules first, then embeddings, then an LLM"; almost none can say that rules read the grand total right on 414 of 466 invoices and that the next rung bought nothing on that field. D2's last line fails more students than anything else, since a pipeline whose owner does not know how much human work remains has not been measured at all. D5 is where a compliance-shaped question meets an engineering answer, and "we send nothing outside" is a full-marks answer when you can show why. The override at the bottom is ruling 2 with no exception clause, and it is the same rule the panel uses in module 6.6.
Artefact 6 — one slice, done to standard. Two fields, worked end to end on the generated corpus. Every number below was measured on the 626 documents the generator writes, so you can reproduce them exactly. This is a bar, not an answer key: your patterns and your model will differ, and the shape of the work must not.
# WHICH RUNG EARNS ITS PLACE, FIELD BY FIELD. Measured, not assumed.
FIELD A - grand total, 466 invoices
RUNG 1, first attempt. One pattern for both layouts:
(?:Grand\s*Total|TOTAL\s*PAYABLE)[^\d]{0,12}([\d,]+)
parsed with float(text.replace(",", ""))
overall : right 184, no-value 269, wrong 13 acc 0.395
by vendor : bhima 0.880 | sahyadri 0.000
STOP AND READ THAT SECOND LINE. The aggregate looks mediocre.
One vendor is at absolute zero - the pattern allows 12 non-digit
characters between label and number, and that layout prints
"Grand Total Rs. 70,742.00" with about twenty.
It never matched once. This is WR-179 in your own repo on day one,
and an average would have hidden it.
Of the 13 wrong values, 5 were more than five times too large:
DOC00256 truth 1,42,367 read as 1,42,36,700 (x100)
DOC00377 truth 1,93,343 read as 1,93,34,300 (x100)
That is a decimal point scanned as a comma, then a comma-stripping
parser. Rs 5.35 crore of over-statement from five pages.
RUNG 1, second attempt. Widen the gap, allow the currency word,
and anchor the money shape on two decimal digits:
(?:Grand\s*T[o0]tal|T[O0]TAL\s*PAYABLE)\D{0,30}?([\d, ]+[.,]\d{2})
parsed with re.fullmatch(r"([\d,]+)[.,](\d{2})", s)
overall : right 414, no-value 29, wrong 23 acc 0.888
by vendor : sahyadri 0.879 | bhima 0.900
hundred-fold errors : 0
Cost of this rung: zero rupees, about 2 seconds for all 466 pages.
RUNG 2 on this field: no gain. There is no known list of totals to
match against, so similarity has nothing to compare with. Skipped,
and the reason is written down.
RUNG 3 on this field: 52 pages still unresolved (29 empty, 23
wrong). Send ONLY those 52 to the model, not all 466. Before you
do, price it: 52 pages against 466 is 11% of the corpus, so the
bill and the personal-data exposure both drop by nine tenths for
the same possible gain. Measure what it buys and write the number
down. If it recovers fewer than half of the 52, rules stay and the
rest go to the human queue - that is a legitimate result.
FIELD B - receiver name on 160 delivery slips
RUNG 1: read whatever follows "Received By :".
right 23, no-value 4, wrong 133 acc 0.144
The name is hand-written, so the scan returns "P. Sutan" for
"P. Sutar" and "A. Mvlla" for "A. Mulla". The pattern is perfect
and the answer is still wrong 133 times out of 160.
RUNG 2: match that damaged string against the 10 known consignee
contacts, keep the best similarity score as the confidence,
and return nothing below 0.62.
right 153, no-value 7, wrong 0 acc 0.956
Cost: a CPU second for the whole corpus, no model, no network.
Every page at or above 0.62 was correct in this run, so on this
field the score is a usable gate - which is a property of a short
closed list and must be re-checked on any open field.
RUNG 3 on this field: not needed. A model would cost money to do
worse than difflib did for nothing.
THE BUSINESS RULE, ON TOP OF BOTH
Check: taxable value + every tax line == printed total.
372 invoices had all three numbers extracted, so the check could
run. It raised 79 flags:
20 of them were wrongly-extracted totals - all 20 that existed
7 were invoices whose printed total is genuinely wrong
52 were pages where the subtotal or a tax line was damaged
Read those three lines carefully. Recall on bad totals was 20 of
20. But 52 of the 79 flags are not about the total at all - the
flag means "this document did not reconcile", never "the total is
wrong". Say that sentence at your defence before somebody says it
to you.
WHAT THIS SLICE DOES NOT PROVE
These numbers come from one seed and one generator, and the
generator's noise is kinder than a real phone photograph. The
0.62 similarity cut-off holds because the candidate list has ten
well-spaced names; on a list of two thousand consignees it will
not. Both facts belong in your README as limits, in your own
words, before a reviewer finds them.
Notice what this slice does that a tutorial never does: it reports a rung that was skipped and a rung that was not needed, each with a reason. That is what "use the cheapest tool that clears the bar" looks like when it is written down instead of said. Field A shows the aggregate hiding a total failure, which you now have in your own repo and can therefore defend from memory. Field B shows the opposite case, where a perfect pattern reads the wrong answer 133 times and a method costing nothing fixes it — proof that "add an LLM" is not the answer to every gap. And the business-rule block ends with an honest limit rather than a win, which is the sentence that earns marks in D5.
Five stages, five graded submissions. Work in a repo called doc-intel-pipeline from the first command, never in a loose notebook — exploration may live in notebooks/, but nothing there is graded and nothing there may be imported. Every stage ends with a commit whose message names the stage, and every stage adds entries to docs/decisions.md on the day you decide, not later.
Where a Check states a number that comes from the generator, your number must match it exactly. Where it states an accuracy, a band is given, because your patterns are your own. Any check you cannot meet is written up in the README as a known gap with your reasoning; an honest gap costs a mark, a hidden one costs the stage.
A note on tools before you start. Nothing in this project needs a GPU, a paid API or an internet connection to reach a passing grade. pytesseract with the Tesseract binary is welcome if it installs cleanly on your machine and you want to work from the rendered images, but it is not the graded path and it is not assumed anywhere: the generator hands you the scanned text directly, and the whole ladder can be built and defended from that text alone. If Tesseract will not install, write one line in your README saying so and move on. That is a correct engineering decision, not a shortcut.
Stage 1 — make the mess, build the gold set, and let the rules fail honestly. Create the repo skeleton from artefact 3 with .gitignore covering corpus/, gold/ and .venv/ before your first commit. Save and run make_corpus.py exactly as given, and paste its full receipt into the README under a heading naming your Python version. Then make the backlog yours — this step is mandatory. Write fifteen pages of a document type you actually know — rent receipts, tuition invoices, pharmacy bills, a shop's delivery challans, any domain you can defend — into corpus/mine/, with a hand-written gold label for each in gold/labels_mine.jsonl, and carry them through every later stage as their own row in the evaluation table. Do not touch the generated 626 pages or their gold file. The reason is blunt: every student in the batch works the identical seeded backlog, so without this your accuracy table matches every other portfolio's line for line and reads as a tutorial clone — these fifteen pages are what make your numbers yours, and the defence will ask you to explain your numbers on them. Open ten documents by hand — five from each vendor, plus three delivery slips — and write docs/document_dictionary.md describing where each field sits in each layout and what its printed label reads. Choose the fields you will extract and, for each, write one line saying what a wrong value costs the firm; that line is what will price your gate in stage 3. Build src/gold.py to load gold/labels.jsonl and assert the corpus receipt, so a corrupted download fails loudly. Write src/money.py with a parse_amount() that survives all five ugly forms, and test it before you use it anywhere. Then write src/rules.py: per-field, per-vendor label patterns, and score them with src/evaluate.py reporting right, no-value and wrong counts for every field, broken out by vendor. Check: your receipt matches the generator exactly on all six headline lines, including documents written 626, unique invoice numbers 440 and arithmetic-broken invoices 10; src/gold.py raises rather than warns when a count is off; parse_amount returns 142367 for all of 1,42,367.00, 1,42,367,00, 142367.00 and 1,42, 367.00, and returns None rather than a guess for 32,00O.00; your evaluation table has one row per field and per vendor with three counts and never a bare accuracy; at least one field on at least one vendor scores far worse than its partner, and you have written two sentences naming the layout difference that caused it; the README states in one sentence why "returned a value" and "returned the right value" are counted in different columns; and corpus/mine/ holds your fifteen personal pages with hand-written labels in gold/labels_mine.jsonl, named in the README with the domain they come from.
Stage 2 — build the ladder and make every rung prove it earned a place. Write src/contract.py from artefact 4 first, and make your stage-1 rules return that object, including the confidence and the rung. Add rung two in src/similar.py: for at least one field, match the extracted string against a known list and keep the similarity as the confidence. Use module 3.4's embeddings if you want the practice, or a character-similarity function if you want the speed; either is acceptable and the choice must be defended in one sentence about cost. Add rung three in src/llm.py: a call carrying a strict JSON schema, a validation of the reply against your contract, one retry on invalid JSON, and a clean give-up that returns a None value rather than a guess. Wire them in src/ladder.py so a page only reaches a rung when the rung above returned nothing or failed its check. Now measure: field-level accuracy per field, per vendor and per rung, against your stage-1 baseline, plus the count of pages each rung kept and the cost of a full run. Check: every extractor returns a DocumentRecord and a test proves the three rungs are interchangeable; a page with no value has confidence exactly zero and your validator rejects any other combination; your table shows, for each field, the accuracy after rung 1, after rung 2 and after rung 3, with the page count each rung kept; at least one field is documented as "rung 2 skipped" or "rung 3 not needed" with the reason, and at least one field improves by more than 0.3 from a rung that costs nothing; fewer than a quarter of all pages ever reach rung three, and your README states the exact percentage and what it saves; the cost table gives one number for a full backlog run in rupees or in minutes; and any field where the ladder failed to beat plain rules has been reverted to rules, with the reverted number still printed in the report.
Stage 3 — validate the arithmetic, price the mistakes, open the queue. Write src/rules_business.py with at least five checks: taxable value plus tax lines equals the printed total, the tax rate is one that exists in law, the invoice date is real and not in the future, the invoice number appears once across the corpus, and every delivery slip references an invoice that exists. Each failure produces a Flag carrying the rule name and the two values that disagreed. Then build src/confidence.py, which combines the rung score with the flags into one page confidence, and make a blocking flag impossible to auto-approve. Write docs/cost_of_errors.md pricing two things with a named source or a clearly flagged estimate: one wrong auto-approval and one human review. Sweep the gate over at least five thresholds, printing auto-approved, wrongly auto-approved, reviewed, rejected and total cost, and ship the cheapest row into config.py. Finish with src/queue.py writing queue.jsonl, one row per page a human must open, carrying the reason and sorted so the most expensive pages are first. Check: the duplicate rule finds exactly 26 re-scanned pages and reports them by invoice number, and your README explains why a file-name check would have found none of them; your arithmetic rule flags the invoices whose printed total is genuinely wrong and you state how many of your flags are about the total versus about some other damaged line; a page carrying a blocking flag cannot be auto-approved and a test proves it; cost_of_errors.md gives two rupee figures, each with a source or an explicit estimate flag, and the ratio between them is stated in one sentence; the sweep table has at least five rows and the shipped gate is the cheapest one, not the highest one, and one sentence explains why the tightest gate is more expensive than the winner; queue.jsonl exists with a reason on every row; and the decision-log entry for the gate names the option you rejected and the trigger that would make you re-run the sweep.
Stage 4 — turn it into a batch pipeline that a stranger can run, and keep the personal data where it belongs. Build run_batch.py to process the whole corpus in one restartable run: a manifest of what is done, resumption without redoing pages, and a summary written at the end. State in the README, in two sentences, why this is a batch job and not a web service, using module 4.9's reasoning about who is waiting. Write src/redact.py replacing buyer name, receiver name, phone number, GSTIN and address with stable tags before any text leaves the process, and make the redaction happen at the boundary so no future call can skip it. Fix the logs in the same pass: field names, lengths and outcomes, never values. Write docs/data_flow.md naming what leaves the machine, what is removed first, which vendor receives it, how long they keep it and who decided — and if your answer is that nothing leaves, write that down as a decision with its reason. Build the test suite to at least ten tests including the quality gate on per-vendor accuracy, then containerise and prove the container completes a run. Check: a classmate clones your repo on a fresh machine, follows three README commands, and gets a finished run without asking you a question; killing run_batch.py halfway and restarting it does not reprocess finished pages and does not lose any, and you demonstrate this on video or in front of a classmate; test_redact.py passes on a page containing all five categories and asserts no raw value survives; test_logs.py greps the run log for a phone pattern and a GSTIN pattern and asserts zero hits, and you prove it works by removing the redactor once, watching the test go red, and pasting that red output into the README; data_flow.md answers all five questions in plain words and names a person; the quality gate fails the build when you deliberately break one vendor's pattern; the container builds, completes a sample run and the image is under 1.5 GB; and requirements.txt is pinned with == on every line.
Working alone? Use the solo substitutes defined in P1: a clean container or a fresh OS user account stands in for the classmate's clone test here, and a recorded self-probe against module 6.6's depth ladder stands in for stage 5's classmate viva.
Stage 5 — watch it for the failure that will actually happen, then defend the whole thing. The thing that breaks this pipeline is not a slow statistical drift; it is one vendor changing a template on one Monday. Write monitor/per_vendor.py reporting, for every vendor and document type, field-level accuracy, the no-value rate and the page count behind them. Write monitor/layout_watch.py to alarm when a vendor's no-value rate on a required field crosses a threshold you choose and justify. Then write docs/monitoring_plan.md with at least five checks, each carrying a threshold, a window, an owner and an action, at least three of them per vendor, plus one paragraph saying exactly what a new vendor layout looks like in your numbers and what happens on the day one appears. Add tests/test_canary.py holding five known pages per vendor and failing the build when any vendor's canary drops. Bring docs/decisions.md to at least ten dated entries. Finally rehearse against artefact 5: record the 90-second walkthrough, have a classmate probe you for ten minutes using module 6.6's depth ladder, and take two live modifications with a timer running. Check: simulate the real failure — edit one vendor's label pattern to break it, run the batch, and show that your per-vendor report drops that vendor to near zero while the overall average moves only a little, then state both numbers in the README as evidence that the aggregate alone would not have alerted anyone; your monitoring plan's five checks each name a threshold, a window, an owner and an action, and three are computed per vendor; the canary test goes red for exactly the broken vendor; decisions.md has ten or more dated entries, each with the rejected option and the reason; your recorded walkthrough is under 110 seconds and ends with an offer to go deeper; you can state from memory the corpus size, your worst field's per-vendor accuracy, the share of pages that reached rung three and the share a human still opens; and for both live modifications you named the file, the function and the direction of the number before editing, then reported in one sentence whether your prediction was right.
Three failures that end document projects, all of them reachable from the stages above. Each has a War Room entry with the full diagnosis path:
- WR-178 — the total read as fourteen lakh. A decimal point scanned as a comma, a parser that stripped every comma, and a confidence score of 0.94 that measured only whether a value had been located; the fix is a money parser anchored on two decimal digits plus an arithmetic check that gates auto-approval no matter what the confidence says.
- WR-179 — the average that hid a vendor at zero. Overall accuracy held near 0.89 while one layout returned nothing at all on every page, because a weighted average cannot see a per-layout collapse; the fix is per-vendor reporting, per-vendor alarms and canary pages in the test suite.
- WR-180 — the invoices that left the country. Full page text including names, phone numbers and GSTINs posted to a hosted model with no redaction, no data-flow record and four months of unredacted logs; the fix is redaction at the boundary, safe logging, and a dated decision naming the vendor, the categories and the retention.
This project hands you a paper backlog, a generator that knows every true answer, and the standard your work is judged by. You own each decision, from the field list to a ladder of methods, a set of business rules, a priced gate and a human queue. The measured lesson is that the cheapest rung usually wins, and that a rung which cannot show a page count is decoration. Report accuracy per vendor and never only on the average, keep personal data out of your logs and out of other people's servers, and write down what you decided on the day you decided it. The defence is the assessment, so a project you cannot explain and cannot change while somebody watches scores zero.
- Logistics and freight firms across India run this exact loop on freight bills and delivery slips. One mis-read total moves real money, and the review queue is a real desk with real people at it.
- Bank and insurance back offices process KYC forms, cheques and claim papers the same way, and they hire freshers straight into extraction and review work every year.
- Hospital billing and government record teams face the harder version of this problem, with hand-written entries and mixed scripts on one page, which is why the delivery slips in your corpus carry both.
Answer these aloud before opening the gate assessment. Each question names its kind.
type=interpretation— A pipeline reports overall field accuracy of 0.888 and a review queue that has tripled for one transporter in a week. Interpret both facts together, say which number should never have been the alarm, and name the two columns you would add to the report before saying anything else.type=scenario-judgement— Your teammate proposes dropping the rules rung entirely, because the model reads every field correctly in a ten-page test and the code would be much shorter. Give your ruling, name the three costs the ten-page test did not measure, and say what evidence would change your mind.type=trade-off— Compare setting the confidence gate at 0.99 against setting it at the cheapest point on your sweep. Give one concrete gain and one concrete cost of each, and name the single business situation that would make 0.99 the right choice.type=debug— Every page returns a value for the grand total, no extractor raises an error, and the totals are silently wrong on about one page in twenty. Diagnose the two most likely causes from that pattern alone, name the two files you would open first, and give the one check that settles it.type=output-prediction— A student sends all 466 invoices to the model instead of only the 52 the rules could not read. Predict what happens to the run cost, to the number of pages carrying personal data outside the company, and to accuracy on the grand total, and say which of the three a reviewer will ask about first.
Explain to a friend studying commerce why a computer that reads an invoice must sometimes say "I am not sure" and hand the page to a person. Use an everyday comparison, such as a shop assistant who cannot read a faded price tag and calls the owner rather than guessing. Cover three things: that reading a number and reading the right number are different, that paying a wrong bill costs far more than asking a person to check, and that the names and phone numbers on the page must not be sent anywhere they are not needed. Use at most 4 sentences and no word your friend would need to look up.