Spot-Check ERP Copilot Answers Against the Database

Tier: Core · Workflow area: ERP Copilots & Agents · ~35 minutes

Vendors ship the copilot; nobody ships the accuracy number for your tenant. Copilot answers fail in specific, classifiable ways — stale data, wrong aggregation scope, misresolved entities, silently dropped filters, and occasionally an invented figure — and each error type has a different fix. The person who runs this protocol converts "people feel the copilot is pretty good" into an evidence-backed go/no-go, and becomes the natural owner of AI verification in the ERP landscape. This is the skill no vendor teaches, because it is the skill that audits them.

When to use

• Before widening a copilot rolloutfrom pilot group to a department, or from navigation-only into analytical use. The gate is a measured accuracy score, not adoption enthusiasm.
• After any change that shifts the copilot's groundrelease upgrades, config changes to key value sets, new company codes or entities. Rerun the protocol; accuracy is not a one-time property.
• When a copilot answer caused a visible missa wrong number reached a meeting. The protocol turns the incident into a classification, a fix and a regression test instead of a trust collapse.

When NOT to use

• To validate individual answers in daily workthat is pb-37's thirty-second habit. This playbook is the systematic audit that decides whether the copilot deserves standing trust for a use case.
• When you lack query access to the underlying dataverification requires an independent path (direct queries, exports, or trusted reports). No independent path, no audit — and that gap is itself a governance finding.
• On questions with no determinable ground truth"summarize this account's health" cannot be scored true/false. Audit factual questions; review qualitative outputs by editorial sampling instead.

The recipe

1. 1.

   Build the question sample from real usage, stratified by risk. Collect 15–25 real questions users ask the copilot (usage logs, or ask the pilot group). Stratify: transactional status lookups, aggregations (balances, totals, counts), period-scoped questions, and entity-resolution cases (ambiguous customer names, similar materials). Weight toward what feeds decisions — month-end figures over navigation trivia.

2. 2.

   Capture answers verbatim, with context. Run each question through the copilot and record: exact question, exact answer, timestamp, user role. Timestamp matters — ERP data moves, and a fair audit compares the answer against the data as of when it answered, or reruns both sides together.

3. 3.

   Write the independent verification query for each answer. For every question, get ground truth through a path that does not touch the copilot — direct SQL on the operational or reporting store, or the standard app/report:

   Copy

   -- Copilot said: "Vendor V-2201 has 4 open invoices totaling 312,400 INR"
   SELECT COUNT(*), SUM(gross_amount)
   FROM open_vendor_items
   WHERE vendor = 'V-2201' AND company_code = '1000';
   

   AI accelerates this step well — paste the question and your schema and ask for the verification query — but you confirm the query semantics, because a wrong verifier corrupts the whole audit. Watch the three classic scope mismatches: company code, document status set, and fiscal vs. calendar period.

4. 4.

   Classify every mismatch with a fixed taxonomy. Score each answer correct / incorrect / partially correct, and give every non-correct answer exactly one primary error class:

   • •
     Stale data — right logic, old snapshot; points at sync lag between the copilot's data source and the live system.
   • •
     Wrong aggregation scope — dropped or added a filter (company code, status, period); the most common class and often a phrasing-sensitivity issue.
   • •
     Entity misresolution — answered about the wrong customer/vendor/material among similar names.
   • •
     Invented figure — a number with no grounding in the data at all; rare, and an automatic escalation regardless of overall score.
   • •
     Right answer, misleading framing — technically correct, practically deceptive (right total, wrong currency implied; count of items vs. count of documents).
5. 5.

   Score, decide, and write the go/no-go. A worked example of the deliverable:

   metric	result
   questions audited	20
   correct	15 (75%)
   stale data	2
   wrong aggregation scope	2
   entity misresolution	1
   invented figure	0

   Then the decision, per use case: status lookups 9/9 — approve for general use; aggregations 6/10 — hold for decision-grade use, all figures require pb-37 verification until re-audit; escalate the two scope errors to the vendor with transcripts. Pre-commit your thresholds before scoring (the pb-33 discipline): for figures feeding decisions, anything below ~90% with zero invented figures is a hold.

6. 6.

   Institutionalize: recheck cadence and error-triggered reruns. Schedule a quarterly rerun of the same question set (it is now a regression suite), plus an immediate partial rerun after upgrades or config changes touching audited areas. Publish results where users see them — calibrated trust spreads faster than either hype or fear, and the published score is what makes "verify before you act on figures" a policy people accept rather than resist.

Make it yours

• SAP Joulemind the fiscal-period trap in period-scoped questions and the sold-to vs. ship-to entity distinction; verify against standard Fiori apps or CDS-view queries as your independent path.
• Dynamics 365 Copilotaudit record summaries against Dataverse queries; classify omissions in summaries (a dropped credit hold) as misleading-framing errors, they matter more than wrong totals in practice.
• Salesforce (Einstein/Agentforce)SOQL is your verification path; entity misresolution across similar account names is the class to over-sample, and grounding gaps feed directly into pb-39's supervision setup.
• BI chatbots over the warehousesame protocol; add a data-freshness column to the scorecard, because warehouse latency masquerades as copilot error and the fix (and the owner) is different.

Quality check

• Verification queries are independent and themselves reviewedground truth never came from the copilot's own path, and a second pair of eyes checked the verifiers' scope semantics. A biased verifier is worse than no audit.
• Thresholds were committed before scoringthe go/no-go bar exists in writing, dated before the results. Deciding the bar after seeing 75% is how 75% ships.
• Every error carries one classification and one ownerstale data goes to the integration team, scope errors to phrasing guidance or the vendor, misresolution to master-data quality (pb-49 territory). Unclassified errors fix nothing.
• The question set is versioned and rerun on a trigger listupgrades, config changes, quarterly clock. A one-time audit is a press release; the rerun is the control.