Natural-language analytics puts a data analyst in every BA's browser — and the wrong-query-plausible-chart is now the most dangerous artifact in the toolkit. This module installs the mandatory habits: inspect the query, check filters and dates, reproduce one number independently — then applies them to profiling, KPI trees, opportunity sizing and anomaly investigations.
David is a Process Excellence Lead at a US insurance enterprise, twelve years in, Lean Six Sigma black belt. His COO asks a simple question before a headcount decision: has claims-team throughput really fallen since the new intake system launched? David exports the ticket data, loads it into ChatGPT's data analysis, and asks. Ninety seconds later: a clean line chart showing throughput per handler down 14% quarter over quarter, with a fluent narrative about "post-migration productivity degradation." It matches the executive suspicion. It nearly goes into the deck.
Instead — because a colleague had been burned before — David runs the module's third habit: reproduce one number independently. His own pivot table says the drop is 6%, not 14%. Digging into the AI's generated code, he finds it counted tickets per handler while his pivot counted closed claims — and the new intake system had started splitting complex claims into multiple linked tickets. Worse, on a second pass he notices the real story is neither number: two senior handlers had moved to a new complex-claims queue, a mix shift that dragged the simple-queue average down while every individual handler's throughput was flat or up. The honest answer to the COO: throughput is stable; the metric definition broke; and the "drop" was a composition effect. A headcount decision very nearly got made on a fluent chart of the wrong quantity.
Everything about that near-miss generalizes. NL analytics tools — Excel Copilot, Gemini in Sheets, Power BI Copilot, ChatGPT and Claude data analysis — produce plausible output on every query, including the misconceived ones. The chart renders either way. The narrative writes either way. Nothing about wrongness looks different.
In AI-assisted analysis, confidence and formatting are constants, not signals. A wrong-query chart is pixel-identical to a right-query chart. The only defense is procedural: inspect, check, reproduce — every time, no exceptions for outputs that confirm what everyone already believed.
Every NL analysis, before its output enters any deliverable:
The same discipline governs text-to-SQL: give the model schema context (table and column descriptions — it cannot guess your naming conventions), and before first use of any generated query, cross-check its result against a figure you already trust — last month's official report, a number finance signs off on. Text-to-SQL is a genuine self-serve superpower for BAs; unverified text-to-SQL is a defect generator with a keyboard.
Every requirements document contains claims about data: "customer email is always captured," "each order has one invoice," "status transitions are sequential." Source systems disagree with those claims at rates that surprise everyone except whoever has profiled the data. Profiling turns assertion into evidence, and AI removes the excuse of not having a script handy: describe the extract, get a pandas or Power Query profiling pass — null rates, format distributions, duplicates, orphaned references, out-of-range values — plus a readable narrative.
The output is not a report to admire; it is requirements input:
customer_emailbecomes a data-quality requirement, a migration cleansing rule, and a design question ("what does the notification feature do for the 12%?")Profiling evidence also feeds forward: in Module 7 it becomes the backbone of the data-readiness assessment, where "does the data exist at usable quality" decides whether an AI use case lives. Profile early; every later artifact inherits the honesty.
Profile the extract you were actually given, not the schema documentation. The documentation says what the system was designed to hold; the profile says what fifteen years of production use actually put there. Requirements built on documentation alone are built on the design intent of a system nobody has met recently.
When the deliverable is measurement itself, AI drafts well and critiques better. From an objective ("reduce claims cycle time without hurting quality"), generate the KPI tree: the outcome metric, its drivers, definitions, formulas, source fields per metric. Then the differentiated move — before any data owner is consulted, run the gaming-risk critique per metric:
| Question per metric | The failure it catches |
|---|---|
| How would a rational team hit this number while missing its point? | Closing tickets fast by closing them wrong; splitting cases to boost counts |
| What behavior does this metric punish that we actually want? | Thorough handlers penalized on volume; honest re-opens hidden |
| What pairing metric keeps this one honest? | Cycle time paired with reopen rate; volume paired with quality sampling |
| Can the source field be manipulated upstream of the metric? | Status flipped early; timestamps edited; the field the metric trusts is hand-typed |
Models are genuinely good at this adversarial enumeration — it is a pattern-completion task over well-documented failure modes. The output specifies not just a dashboard but a defensible one, and then data owners validate that every source field exists, means what the spec assumes, and updates on the cadence the dashboard claims.
Benefit quantification is where analysis meets money, and where AI-drafted arithmetic must be auditable or absent. The contract for any sizing model this track's analysts produce:
AI helps draft the model and — used adversarially — helps attack it: "what assumptions does this model hide? which numbers lack sources?" But models also make arithmetic slips and unit errors with perfect confidence, so the Common Core rule stands: recompute the spine of any money math yourself. This artifact feeds directly into Module 7's honest business case, where an unmeasured baseline is a named killer.
"Automating intake will save 40% of effort" — no baseline measured, no source for 40%, point estimate, cost stack absent. Survives a kickoff slide; dies in finance review or, worse, survives finance review and dies in benefits realization.
Baseline: 11,400 intakes/quarter at 18 min median (event log, Q2). Benefit range: 22–41% handle-time reduction (pilot sample, n=120). Every step shown, sensitivity named (adoption rate), null result acknowledged. A reviewer can disagree with an assumption — because they can see it.
A metric moves; stakeholders demand a story; the worst possible response is the first plausible narrative — and AI will supply one instantly if asked "why did X drop?" The discipline is to make AI enumerate before it narrates. Build the standard hypothesis tree and test each branch against data before any conclusion travels:
AI's role: generate the tree exhaustively (it will add branches you'd forget — a policy change, a large single customer, a data-latency artifact), draft the test for each branch, and write the final narrative after the data has voted. Your role: run the tests, keep the discipline, and refuse to let the fluent first story skip the queue.
Everything in this module runs on free tiers — Google Sheets with Gemini, free ChatGPT/Claude data analysis on CSV extracts, and hand-rolled pivot tables as the independent check. Tooling is not the constraint. The habits are the constraint, and they cost only the discipline to run them when the deadline is loud.
NL analytics gives every analyst a data team and every wrong idea a beautiful chart. The three habits are non-negotiable: inspect the query, check filters and dates, reproduce one number independently — then let profiling evidence your requirements, gaming critiques harden your metrics, shown math carry your sizing, and hypothesis trees discipline your anomaly stories. Evidence over fluency, every time.