Ask Analytics Questions in Plain English — Then Inspect the Query

Workflow area: Analytics & Experimentation · Tier: Core · Hands-on time: ~35 minutes

Natural-language analytics is the single biggest self-serve unlock for PMs since dashboards — and its failure mode is uniquely dangerous. A wrong SQL query errors out; a wrong NL interpretation returns a beautiful, confident, plausible chart of the wrong thing. The model heard "signup conversion" and measured "signup button clicks." Nothing looks broken. That is the analytics equivalent of hallucination, and the fix is a routine, not a tool.

When to use — and when not to

• Use it for exploratory questions you'd otherwise queue for an analyst"what's D7 retention for users who used import?", "where does the checkout funnel leak on mobile?" — the ask-inspect-verify loop turns a two-day wait into ten minutes.
• Use it for first-pass segmentationcomparing cohorts, plans, geos or acquisition channels before you decide which cut deserves a proper analysis.
• Use it to learn your own event taxonomyinspecting generated queries teaches you which events and properties actually exist faster than any tracking-plan doc.
• Skip it for numbers going into board decks, pricing decisions or public claimsthose get a data-team review, full stop; NL analytics is for exploration, not for the numbers of record.
• Skip it when the tracking itself is suspectif instrumentation shipped last week or events were renamed, the AI will happily query the broken schema; fix the data before trusting any tool on top of it.

The recipe

1. 1.

   Ask the question with entities pinned down. Vague questions produce plausible-but-wrong interpretations. Name the exact event, the exact segment, the exact window:

   Copy

   What is the conversion rate from `signup_completed` to
   `first_project_created` within 7 days, for users who signed up
   between 1 June and 30 June, split by plan type (free vs pro)?
   Exclude internal users (email ending in our domain).
   

   Compare that with "what's our signup conversion?" — the AI must guess the events, the window, the cohort and the exclusions, and every guess is a place to be silently wrong.

2. 2.

   Open the generated query before you look at the chart. Every serious tool exposes it — PostHog Max shows the HogQL/insight definition, Mixpanel Spark shows the report configuration, Amplitude shows the chart definition. Read it against a four-point checklist:

   Copy

   QUERY INSPECTION CHECKLIST
   1. EVENTS — are these the events I meant? (signup_completed,
      not signup_started or Sign Up Clicked)
   2. DATE RANGE — is it the window I asked for, in the timezone
      I think it is? Watch for "last 30 days" silently replacing
      explicit dates.
   3. FILTERS — did my exclusions survive? Is it counting unique
      users or total events? Any filter I did NOT ask for?
   4. MATH — conversion of what over what? Funnel windowed how?
      Strict ordering or any-order?
   

3. 3.

   Reproduce one number manually. Pick the headline number and rebuild it with the tool's classic UI — a hand-built funnel, a manual insight, or a known-good dashboard that tracks the same metric. A real check looks like this:

   Copy

   AI answer:    signup → first project, June, free plan: 31.4%
   Manual funnel: signup_completed → first_project_created,
                  1–30 June, plan=free, unique users: 31.4% ✓
   Verdict: matches — safe to use for exploration.
   
   AI answer:    pro plan: 58.9%
   Manual funnel: 44.2% ✗
   Diagnosis: AI funnel used any-order matching, counting users
   who created a project BEFORE completing signup (SSO flow).
   

   One reproduced number per session is the minimum bar. If it doesn't match, the discrepancy is the finding — diagnose it before asking anything else, because the same misinterpretation is probably contaminating every answer in the thread.

4. 4.

   Interrogate the anomaly before you narrate it. When a result surprises you, resist the story-shaped explanation and ask the tool to decompose it: "break that spike down by source," "show the same metric for the prior period," "how many users is this segment, absolutely?" Small-denominator segments produce dramatic percentages from a handful of users.

5. 5.

   Record what you verified when you share. One line under the chart: "Generated via Max; query inspected; pro-plan number reproduced manually against the activation dashboard." That sentence is what separates self-serve analytics from vibes, and it teaches your team the standard.

Make it yours

• PostHog Max as the daily driver: Max answers in the context of your actual schema and can create insights directly. Keep a saved "known-good" dashboard of your five core metrics and make cross-checking against it a reflex — every Max session starts by asking one question you already know the answer to.
• Warehouse-first teams: if product data lives in Snowflake or BigQuery, the same discipline applies to Cortex Analyst or Gemini-generated SQL — paste the generated SQL into Claude and ask "what could this query be misinterpreting about my question?" before running it. This pairs with pb-11's known-good cross-check.
• Team rollout: run a 30-minute session where everyone asks the same three questions, inspects the queries together, and compares answers. Divergent numbers from identically-worded questions is the fastest way to teach why inspection is mandatory.

Quality check

• The generated query was actually openedif you only looked at the chart, you did not do the playbook.
• Events verified by exact namethe AI's guess of Purchase vs your order_completed is the most common silent failure.
• Date range and timezone confirmed"last month" vs calendar June vs a UTC/IST boundary can move a number several percent.
• One number reproduced manuallyagainst a hand-built funnel or a known-good dashboard, and the match (or mismatch) noted.
• Denominators sanity-checkedany dramatic percentage was checked for a small absolute base before it entered a narrative.
• Nothing quoted upward without the verification lineexploration is free; claims carry a named check.