Analytics & Experimentation with Statistical Caution

The scenario — the number that was right about the wrong thing

Nisha is a PM at a Hyderabad-based edtech company, responsible for the learner-retention surface. Her company rolled out an analytics copilot over their product data, and it changed her mornings: instead of filing a data request and waiting two days, she typed "how did week-4 retention change for learners who used the new practice-quiz feature?" and had a chart in eleven seconds. The chart showed +9 points. She put it in the monthly business review deck with the caption "practice quizzes drive a 9-point retention lift."

The head of data science read the deck twenty minutes before the meeting and asked one question: "Did you look at the query?" Nisha hadn't. The generated query had done three things she didn't intend. It compared quiz-users against all users — a self-selection comparison, since engaged learners both try new features and retain better. It had silently scoped to a date range that excluded the exam-season cohort. And "used the feature" had been interpreted as "ever fired the feature's page-view event" — including learners who bounced in four seconds.

The number was computationally correct. The query was a faithful answer to a question Nisha hadn't meant to ask. That is the analytics version of hallucination — the wrong-query-plausible-chart failure mode — and it is more dangerous than a model inventing a fact, because the chart is real, reproducible, and wearing the authority of your own data.

What You'll Learn

• NL analytics across PostHog Max, Mixpanel Spark, and Amplitude agentswith the three mandatory habits
• The wrong-query-plausible-chart failure modeand the inspection ritual that catches it
• Metric-move investigations as a hypothesis treeseasonality, mix shift, instrumentation, real change
• NL-to-SQL self-servewarehouse queries with schema context and known-good cross-checks
• Tracking specsfeature spec to event schema in your org's taxonomy
• Experiment design and honest readoutsMDE math, peeking, and narratives that preserve uncertainty

The three mandatory habits

All three major product-analytics platforms now ship conversational agents — PostHog's Max, Mixpanel's Spark, Amplitude's AI agents — and warehouse-side, Snowflake Cortex and BigQuery's Gemini integration do the same for SQL. The self-serve win is real: questions that died in the data-team queue now get answered while the meeting is still happening. The discipline that must ride along is three habits, applied every single time:

1. 1Inspect the generated query. Every one of these tools shows you the query, filter set, or event definition behind the answer. Read it before you read the chart. You are checking one thing: is this the question I meant? — the right event, the right population, the right comparison.
2. 2Check date ranges and filters explicitly. Silent defaults are the most common divergence: last-30-days when you meant the quarter, completed-events-only, one platform, excluded internal users — or not. Nisha's excluded exam-season cohort lived here.
3. 3Reproduce one number manually before quoting anything upward. Before an AI-derived number enters a deck, an exec update, or a decision, rebuild one load-bearing figure the hard way — a hand-built funnel in the platform UI, a known-good dashboard, or a SQL query you wrote yourself. One manual reproduction validates the whole chain; zero validates nothing.

The inspection is faster when you know where translations actually break. Four divergence points account for most wrong-query incidents:

• Populationwho is being counted: all users vs. a cohort, self-selected feature-users vs. a comparable baseline, internal and test accounts included or excluded. Nisha's self-selection comparison lived here, and it is the most damaging class because it produces causal-sounding numbers.
• Event semanticswhich event stands in for the behavior: "used the feature" as page view, click, or completed action are three different products. The model picks whichever event name matches your phrasing best, not whichever one means what you meant.
• Window and grainthe date range, the bucketing (daily vs. weekly), and whether partial current periods are included. A chart ending on a half-complete week manufactures a decline every single time.
• The denominatorrates need both halves checked: retention of whom, conversion from what. A numerator moved by a definition change while the denominator stayed put is a phantom trend.

For NL-to-SQL against the warehouse, the same habits apply with one addition: give the model your actual schema and metric definitions as context (it cannot guess that active_user means "3+ sessions in 7 days" in your org), and cross-check the result against a known-good dashboard number before it travels. A query that returns a number is not a query that returns your metric.

Metric moves — investigate with a tree, not a vibe

"Retention dropped 3 points — find out why by Friday" is the highest-stakes analytics task a PM owns, and it is where AI assistance most needs structure. Handed the question raw, a model will do what juniors do: grab the first plausible correlate and narrate it into a cause. The structured alternative is a hypothesis tree — enumerate the branches first, then test each against data before any conclusion ships:

AI accelerates every branch — proposing sub-hypotheses you'd miss, drafting the segment queries, summarizing findings — and the tree keeps it honest: each branch produces an explicit supported / refuted / inconclusive verdict with the evidence attached, and instrumentation gets checked before anyone builds a narrative about user behavior. A remarkable fraction of urgent metric investigations end at "an event got renamed," and finding that out after presenting a behavioral story to leadership is a bad Tuesday.

The same join-the-evidence pattern powers churn and win-loss work: qualitative themes from Module 2's mining (tickets, calls) joined with quantitative cohorts into ranked driver hypotheses, each linked to both evidence types. AI is genuinely good at proposing the joins; the rank and the causal language stay yours.

Instrumentation — specs the data team will accept

Self-serve analytics is only as good as the events underneath, and tracking specs are a classic PM chore AI does well inside guardrails: feed the feature spec plus your org's event taxonomy and naming conventions, get back a draft schema — event names, properties, triggers — in your house style, plus a drift check against the existing scheme ("you have checkout_completed but this spec proposes purchase_done"). Data engineering reviews before anything ships. Tedious, verifiable, high-leverage — the exact profile of a task to delegate, with review.

Experiments — the math is cheap, the honesty is the job

Experiment design has always had a stats-literacy gap on product teams, and AI closes the mechanical half well: it will walk MDE and sample-size math step-by-step, flag ratio-metric pitfalls, propose guardrail metrics, and estimate run length from your traffic. Platforms like GrowthBook and Statsig embed similar assistance natively (GrowthBook is open source and self-hostable — the ₹0 lab path). Use all of it — and show the math in the design doc, because "the AI said 6 weeks" is not a defensible run length and step-by-step arithmetic is auditable by your data scientist.

Where AI actively works against you is interpretation. Models are agreeable, and an agreeable narrator of experiment results is a machine for overclaiming. The classic sins it will happily commit on your behalf:

• Peeking blessed as signalyou checked at day 6 of a 21-day run, the number looked good, and the AI drafts "early results are strongly positive." Sequential looks without correction inflate false positives; the readout date is part of the design.
• Segment fishing dressed as insightslice enough segments and something is always "significant." Twenty subgroup analyses at 95% confidence expect one false positive; the AI will narrate that one as the finding.
• Significance inflated into certainty"p < 0.05" becomes "the variant wins"; a confidence interval spanning 0.2% to 9% becomes "a 4.6% lift." The interval, novelty-effect caveats, and the difference between statistical and practical significance vanish in fluent summary.
• The null result buriedan honest "we cannot distinguish this from no effect" gets rewritten as directional encouragement, because models mirror the enthusiasm in the prompt.

The honest-readout pattern: have AI draft the stakeholder narrative under explicit instruction to preserve uncertainty — confidence intervals stated, caveats in the summary paragraph and not a footnote, segment findings labeled exploratory — then hand-rewrite the recommendation sentence yourself. You own the claim; the model drafts around it. Lab 6 hands you a deliberately flawed readout with all three planted sins and grades the rewrite.

Try it now

1. 1Audit yesterday's answer. Take the last AI-generated analytics answer you or a teammate used (or generate one now with your analytics tool). Open the query behind it and check event, population, filters, and date range against the question actually meant. Log any divergence.
2. 2Tree one metric move. Pick a metric on your product that moved recently. Write the four-branch hypothesis tree, use AI to draft the test for each branch, and run at least the instrumentation branch to a verdict.
3. 3Stress-test a readout. Feed your most recent experiment readout (or any public one) to your assistant with the instruction "find every place this overclaims: peeking, multiple comparisons, missing intervals, significance-to-certainty leaps." Rewrite the summary paragraph to survive that critique.