A document pipeline reports overall field accuracy of 0.888 on the weekly run, roughly where it has been since launch. The dashboard is green. Meanwhile the accounts team says they are now correcting almost every bill from one particular transporter, and the review queue for that vendor has quietly tripled. Nobody can reconcile the two stories, because the pipeline's own numbers say nothing changed. Split the same run by vendor and the picture inverts: 0.900 for one vendor, 0.000 for the other — every single page returning "no value found" for the grand total.
Any extraction pipeline that serves more than one document layout, which in practice
means every real one. Fires hardest when a new vendor is onboarded, when an existing
vendor changes its billing software, or when a template is redesigned — a header moves,
a label gains a currency symbol, Grand Total Rs. 70,742.00 becomes
Total Payable ₹70,742.00. Also fires in reverse when one big vendor dominates the mix
and its good numbers hide everyone else's collapse. Linked modules: P2 (Document
intelligence pipeline), 4.8 (Monitoring), 3.7 (Evaluating LLM systems).
The aggregate is a weighted average, and a weighted average is exactly the wrong instrument for a per-layout failure. Two things must be separated.
The first is arithmetic. If one vendor supplies 209 of 466 invoices at 0.900 and the other supplies 257 at some value, the overall figure moves slowly. A vendor can go to absolute zero and drag the aggregate down by less than half, which still reads as a disappointing week rather than an emergency. The number that would have screamed was never computed.
The second is the shape of the failure. Rules-based extraction does not degrade
gracefully; it stops. A pattern like
(?:Grand\s*Total|TOTAL\s*PAYABLE)[^\d]{0,12}([\d,]+) allows at most twelve non-digit
characters between the label and the number. One layout prints
Grand Total Rs. 70,742.00 with about twenty characters of padding and
a currency word in between, so the pattern never fires at all — not once, on any page,
from that vendor. This is why the failure is total rather than partial, and it is also
why "no value found" is the honest signal that was available all along and was averaged
away.
Confirm it in five minutes. Group the last run's field results by vendor, by layout version, and by month, and print accuracy plus the "no value found" rate for each cell. A cell that is uniformly zero is a pattern that never matched, which is a layout problem. A cell that is noisy but non-zero is a scan-quality problem. The two need different fixes, and the aggregate tells you neither.
Immediate: re-run the affected vendor's backlog with a pattern that covers both layouts and a strict money parser. On the reference corpus, widening the label-to-number gap and allowing the currency word moved that vendor from 0.000 to 0.879 with no change to the other vendor's 0.900. Route every page extracted during the broken window back through the human queue; those rows were never right and no amount of downstream repair changes that.
Structural: make the monitoring per-segment by default. The weekly report must carry one row per vendor and per document type, each with field-level accuracy, the "no value found" rate, and the page count behind it — a metric with no denominator beside it can hide anything. Add two alarms that fire on the segment, not the average: any vendor whose "no value found" rate for a required field crosses a threshold you set, and any vendor whose page count moves sharply. Then add a canary: keep five known pages per vendor in a small gold set, run them on every deploy, and fail the build when any vendor's canary drops. That is the same idea as a quality-gate test in module 4.5, applied per layout.
Decide at design time that every accuracy number in this system is reported per vendor and per document type, and that the aggregate is a summary line, never an alarm. New vendors are not rare events; they are the normal business of a logistics or accounts team, so treat "a layout we have not seen" as a routine input rather than an incident.
Two habits make it cheap. Keep the gold set stratified so that every vendor has known pages in it from the day that vendor is onboarded. And log the reason a field was empty — pattern did not match, page unreadable, field genuinely absent from the paper — because those three look identical in an accuracy score and completely different in a diagnosis. Drift here rarely arrives as a slow statistical shift. It arrives as one vendor's new template, on one Monday, all at once.
Q: "Your extraction accuracy is stable at 0.89 but one team says quality has collapsed. Who is right?" Strong answer: both, and the aggregate is the reason they disagree. Explain that overall accuracy is a weighted average across layouts, so a total failure in one segment can look like a mild dip overall. Say what you would compute first: the same metric grouped by vendor and document type, with the page count and the "no value found" rate alongside. Name the giveaway — a uniformly zero cell means a pattern that never matched, so it is a layout change and not a scan-quality problem. Close with the fix that prevents a repeat: per-vendor alarms plus a stratified gold set with canary pages per vendor in the test suite.