An invoice pipeline is running clean. Every page produced a value for every field, no
extractor threw an error, and the confidence score on the offending page was 0.94 —
comfortably above the auto-approve gate. Two weeks later finance asks why a Kolhapur
freight vendor was paid ₹14,23,670 against a bill for ₹1,42,367. The extracted JSON is
still in the audit log, and it says "total": 1423670 with "confidence": 0.94. Nothing
in the pipeline logged a warning, because nothing in the pipeline was broken.
Batch document extraction over scanned or photographed invoices, usually the first
week the pipeline runs unattended. Common when the corpus carries Indian digit
grouping (1,42,367.00) and the amount parser was written as
float(text.replace(",", "")). Also fires on faint scans where a decimal point is read
as a comma, where a stray mark becomes a digit, or where a letter inside a number
(32,00O.00) truncates the match. Linked modules: P2 (Document intelligence pipeline),
3.2 (Working with LLM APIs properly), 1.5 (Evaluation & metrics deep dive).
Two separate things went wrong, and only naming both explains the loss.
First, the parser. 1,42,367.00 scanned as 1,42,367,00 — the decimal point picked up
as a comma, which is one of the most common OCR confusions on faint print. Stripping
every comma turns that into 14236700, then dividing by nothing gives ₹1,42,36,700, or
₹14,23,670 depending on where the paise land. The digits were all read correctly. Only
the separator moved, and a comma-stripping parser cannot tell a grouping comma from a
decimal one.
Second, the confidence score. It was 0.94 because the extractor was confident it had found the total line — the label matched, a number followed, the regex fired cleanly. Confidence measured whether a value was located, not whether the value was right. That distinction is the whole failure. A field can be located with certainty and still hold a number that is a hundred times too large, and no confidence score built from matching will ever notice, because from the matcher's point of view nothing went wrong.
Confirm it in two minutes: pull the raw text for the document from the audit log and look at the total line as scanned. If the amount ends in a comma-plus-two-digits rather than a point-plus-two-digits, this is your case. Then check whether the extracted subtotal plus the extracted tax lines equal the extracted total. On a real invoice they must. Here they will be out by roughly a factor of a hundred, and that arithmetic gap was sitting in the record from the first minute.
Fix the parser first: never strip separators blindly. Match the money shape explicitly
with a pattern that requires exactly two trailing digits after a point or a comma, and
treat the last separator as the decimal one:
re.fullmatch(r"([\d,]+)[.,](\d{2})", s) then strip commas from group 1 only. On one
reference corpus this single change moved grand-total accuracy from 0.395 to 0.888 and
removed every hundred-fold error.
Then add the check that would have caught it even with the bad parser. Business-rule validation reads the fields against each other: taxable value plus every tax line must equal the printed total; the total must be positive and within a sane band for the vendor; the invoice date must not be in the future. Any invoice failing arithmetic is flagged and routed to the human queue regardless of its confidence score. On the same reference corpus this rule caught 20 of the 20 wrongly-extracted totals that had complete field sets — a full catch — plus 7 invoices where the vendor's own arithmetic was wrong on paper. Re-run the whole backlog after the fix; a pipeline that ran wrong once has already written wrong rows.
Treat confidence and correctness as two different measurements and never let one stand in for the other. Confidence answers "did I find a value?"; a business rule answers "can this value be true?". Ship both, and make the auto-approve gate require both.
Write the arithmetic checks before the extractor, not after the incident — for invoices
the list is short and it never changes: totals reconcile, tax rate is one of the legal
ones, dates are plausible, invoice numbers are unique. Keep a small gold set of pages
where you know the true values, and score field-level accuracy on the value, not on
whether something was returned. And test the money parser on the ugly cases first:
1,42,367.00, 1,42,367,00, 142367.00, 1,42, 367.00, 32,00O.00. Every one of
those appears in real Indian invoice scans.
Q: "Your extraction reported 0.94 confidence and the value was a hundred times too large. How is that possible, and what would you add?" Strong answer: separate the two claims — the score measured that a value was located, not that it was correct, and those come apart exactly when the characters are read fine but a separator moves. Name the Indian digit-grouping case concretely. Then give the fix in two layers: a money parser that anchors on two decimal digits instead of stripping commas, and an independent arithmetic check (subtotal + tax = total) that gates auto-approval no matter what the confidence says. Finish with the honest note that the second layer is the one that generalises, because it catches errors you have not thought of yet.