Audit fifteen ERP copilot answers against the underlying exports the way no vendor teaches: verify each with SQL, classify every error — stale data, wrong aggregation, invented figure — and make the go/no-go call for wider rollout. Then flip roles: use AI to draft a month-end close pack where every figure traces to a query you ran.
Meenakshi is the finance-systems analyst at a Coimbatore manufacturer that just enabled an ERP copilot for the FP&A team, and the CFO's question is the right one: "Before forty people trust it, how often is it wrong, and wrong how?" Copilot answers over financial data fail in specific, classifiable ways, and the classification matters because each class has a different fix — stale data is a grounding problem, wrong aggregation is a semantics problem, an invented figure is a trust problem. Five of the fifteen answers below are wrong. The exports are the ground truth; the SQL is the audit.
~90 minutes hands-on · Core tier (maps to module M6 · Operating & Building on ERP/CRM Copilots).
DuckDB only, $0. Save the three exports. gl_balances.csv (sign convention: credits negative — revenue accounts carry minus):
account,name,period,amount
400000,Revenue,2026-05,-8450000
400000,Revenue,2026-06,-9120000
410000,Other income,2026-05,-150000
410000,Other income,2026-06,-180000
500000,COGS,2026-05,5070000
500000,COGS,2026-06,5472000
600000,Salaries,2026-05,1900000
600000,Salaries,2026-06,1900000
610000,Rent,2026-05,300000
610000,Rent,2026-06,300000
620000,Marketing,2026-05,450000
620000,Marketing,2026-06,610000
open_items.csv (AR as of 2026-06-30; cleared=Y means paid):
invoice,customer,amount,due_date,cleared
INV-1001,Alpha,250000,2026-06-15,N
INV-1002,Alpha,180000,2026-07-10,N
INV-1003,Beta,320000,2026-05-30,N
INV-1004,Beta,90000,2026-06-01,Y
INV-1005,Gamma,410000,2026-07-05,N
INV-1006,Gamma,75000,2026-06-20,N
INV-1007,Delta,130000,2026-04-25,N
INV-1008,Delta,220000,2026-07-15,N
INV-1009,Epsilon,60000,2026-06-28,N
INV-1010,Epsilon,45000,2026-05-15,Y
orders.csv — 20 orders: 6 open (values 120000, 150000, 90000, 110000, 140000, 110000), 5 shipped, 7 completed, 2 cancelled (author the remaining values yourself; only the open set's total matters to the audit). And the file under audit, copilot_answers.md:
A1: June revenue was Rs 9,120,000.
A2: Total operating expenses in June were Rs 2,810,000.
A3: Revenue grew 7.9% month-over-month.
A4: Marketing spend for June was Rs 450,000.
A5: Total open receivables stand at Rs 1,645,000.
A6: Overdue receivables total Rs 1,645,000.
A7: The largest single open invoice is Rs 410,000 (INV-1005, Gamma).
A8: The provision for doubtful debts is Rs 120,000.
A9: Delta has Rs 350,000 open across two invoices.
A10: There are 6 open orders worth Rs 720,000.
A11: 11 orders are still open.
A12: Gross margin for June was 40%.
A13: Total June income was Rs 9,300,000.
A14: Beta's overdue balance is Rs 410,000.
A15: 2 orders were cancelled.
Load everything: duckdb close.duckdb then create table gl as select * from read_csv_auto('gl_balances.csv'); and likewise for the other two.
Write the spot-check protocol before checking anything. protocol.md, five lines: every answer gets a verdict (CORRECT / WRONG / UNVERIFIABLE) backed by one SQL query pasted next to it; a wrong answer gets an error class (STALE — right figure, wrong period; WRONG-AGGREGATION — real rows, wrong filter or sum; INVENTED — figure appears nowhere in the data; WRONG-SCOPE — right metric, wrong entity or status set); no verdict from memory or eyeball — the query output is the verdict; ambiguous business terms get the strictest reasonable reading, noted.
Verify the P&L answers (A1–A4, A12–A13). One query pattern serves them all:
select account, name, amount from gl where period = '2026-06';
select round(100.0 * (-9120000 - -8450000) / -8450000, 1); -- growth check
Expected verdicts: A1 correct (Rs 9,120,000, sign convention noted); A2 correct (1,900,000 + 300,000 + 610,000 = 2,810,000); A3 correct (670,000 / 8,450,000 = 7.9%); A4 WRONG — STALE: 450,000 is May's marketing; June is 610,000 (the copilot answered from the prior period — the most common ERP copilot failure, and invisible unless you check the period); A12 correct ((9,120,000 − 5,472,000) / 9,120,000 = 40.0%, using revenue account only — note the definition you accepted); A13 correct (9,120,000 + 180,000).
Verify the receivables answers (A5–A9, A14). The queries:
select sum(amount) from open_items where cleared = 'N'; -- A5
select sum(amount) from open_items
where cleared = 'N' and due_date < date '2026-06-30'; -- A6
select customer, sum(amount) from open_items
where cleared = 'N' group by 1; -- A9, A14
Expected: A5 correct (1,645,000); A6 WRONG — WRONG-AGGREGATION: overdue is 835,000 (INV-1001/1003/1006/1007/1009); the copilot reported total open as overdue — the number is real, the filter is missing; A7 correct; A8 WRONG — INVENTED: no provision account, no 120,000 anywhere in any export — the figure is fabricated, and your query proof is a search that returns zero rows; A9 correct (130,000 + 220,000); A14 WRONG — WRONG-SCOPE: Beta's overdue is 320,000; 410,000 only emerges by including the cleared INV-1004 — paid invoices counted as receivable.
Verify the order answers (A10, A11, A15). Expected: A10 correct (6 open, 720,000); A11 WRONG — WRONG-SCOPE: 11 = open + shipped; "still open" is 6 — the copilot folded a status in silently; A15 correct.
Compile the findings and make the go/no-go call. findings.md: the 15-row verdict table (answer, verdict, class, proof query, correct figure), then the tally — 10 correct, 5 wrong, error classes: 1 STALE, 1 WRONG-AGGREGATION, 1 INVENTED, 2 WRONG-SCOPE. Now the decision, argued in one paragraph: a 33% error rate on financial questions is a no-go for unsupervised rollout — but the classes point to the remediation: STALE and WRONG-SCOPE are grounding/semantic-layer fixes (default period injection, status glossaries — the Lab 09 playbook applied to a copilot you do not control means constraining it to curated views); INVENTED is the class that mandates the human-verification rule permanently, because no configuration eliminates it. State the conditions under which you would re-test for a go: error rate under an agreed threshold on a fresh 15-question audit, zero INVENTED, every user trained to demand the source query.
Flip roles: draft the close pack with AI. Give the assistant the three exports and ask for a month-end close support pack: (a) a reconciliation checklist for June (P&L completeness, AR-to-GL tie-out, order-pipeline sanity) with a DuckDB query per check, and (b) a variance narrative May-to-June. The rule of the exercise: you run every query; a figure enters the pack only with its query output attached. Expected narrative content, each line traced: revenue +670,000 (+7.9%); other income +30,000; COGS +402,000 — flag that COGS held at exactly 60.0% of revenue both months (a ratio stability worth one narrative sentence); salaries and rent flat; marketing +160,000 (+35.6%) — the single variance needing a business explanation, mark it as the open question for the marketing controller.
Trace-audit your own pack. Reread the AI-drafted narrative hunting for exactly the five error classes you just cataloged — any figure you cannot re-derive with a pasted query gets cut or flagged. This is the closing discipline of the lab: the auditor's checklist you built for the copilot applies verbatim to your own AI-assisted work product. Finalize close-pack.md with the checklist, the narrative, and every supporting query inline.
close-pack.md has an attached query output; the marketing variance is flagged for business follow-up.Self-grading rubric (pass = 8/10): protocol written before auditing (0–1); all five errors caught with correct figures (0–3); error classification defensible with class-specific remediation (0–2); go/no-go argued from the evidence, not vibes (0–1); close pack fully traced, ratio observation included (0–2); self-trace-audit performed on your own pack (0–1).
A copilot over financial data is audited, not believed: verdict, query, error class, and a rollout decision that maps each class to its fix. And the same protocol turns around onto your own AI-drafted close pack — the figure that cannot show its query does not ship, no matter who or what wrote it.