Investigate a Metric Move with a Hypothesis Tree

Workflow area: Analytics & Experimentation · Tier: Core · Hands-on time: ~40 minutes

Metric-move investigations fail in a predictable way: someone notices the drop, someone else remembers "we changed onboarding Tuesday," and the correlation becomes the explanation in the next standup. Meanwhile the actual cause — a tracking event silently renamed in the same release — goes unfound for three weeks. AI is genuinely good at the part humans skip under pressure: enumerating the boring hypotheses exhaustively. You remain the one who tests them.

When to use — and when not to

• Use it when a metric you own moves outside its normal rangeup or down; unexplained improvements are instrumentation bugs more often than triumphs, and claiming credit for a broken counter is worse than missing a drop.
• Use it before any metric story ships upwardif the move will appear in a weekly business review, it needs a tested explanation, not a plausible one.
• Use it as the template for recurring triagethe tree structure makes investigations comparable, so the team stops re-deriving the checklist every incident.
• Skip it for moves within normal variancecheck the metric's historical week-over-week noise first; investigating wiggles teaches stakeholders that every wiggle deserves a story, which is its own pathology.
• Skip the AI when the data can't leave your tenantbuild the tree with AI, but run the queries in your analytics tool; never paste raw user-level exports into a consumer chatbot.

The recipe

1. 1.

   Confirm the move is real before explaining it. Pull the metric's last 12 weeks. Is the "drop" outside two standard deviations of weekly noise? Is it visible in more than one data source (product analytics AND the warehouse)? Five minutes here kills half of all investigations honestly.

2. 2.

   Generate the hypothesis tree. Give the AI real context — it cannot know your release calendar or seasonality, so feed it:

   Copy

   You are helping me investigate a metric move as a skeptical data
   analyst. Build a hypothesis tree with four root branches:
   (1) SEASONALITY / CALENDAR, (2) MIX SHIFT (traffic/segment
   composition changed, per-segment rates did not),
   (3) INSTRUMENTATION (tracking broke, event renamed, bot filter,
   consent banner), (4) REAL BEHAVIOR CHANGE.
   
   Metric: [e.g. "week-1 activation rate: signup_completed →
   first_project_created within 7 days"]
   Move: [e.g. "dropped from 34% ±1.5 to 22% starting 21 July"]
   Recent changes: [releases, campaigns, pricing, geo launches,
   tracking-plan changes in the window]
   Calendar context: [holidays, school terms, fiscal quarter, region mix]
   
   For EACH leaf hypothesis, specify: the exact query or chart that
   would confirm or kill it, what result confirms it, and a prior
   (low/medium/high) with one line of reasoning. Order branches by
   how cheap they are to test. Do not conclude anything.
   

3. 3.

   Test the cheap branches first — instrumentation before behavior. The AI's tree typically surfaces tests like:

   Copy

   INSTRUMENTATION (test first — cheapest, most often guilty)
   - Raw event volume of signup_completed vs first_project_created,
     daily: did the DENOMINATOR or NUMERATOR event move alone?
     → numerator-only cliff on a release day = tracking suspect
   - Segment by app version / platform: is the drop isolated to the
     release that shipped 21 July?
   MIX SHIFT
   - Activation rate per acquisition channel, weekly: are per-channel
     rates flat while the paid/organic mix moved?
   SEASONALITY
   - Same weeks, last year; same metric, unaffected geo as control.
   

   Work down the list in your analytics tool or with pb-27's NL querying — inspecting every generated query as you go. Kill branches explicitly: "not seasonality: last-July shows no dip, and the drop is absent in EU."

4. 4.

   Write the finding with the tree attached. The conclusion format that earns trust upward:

   Copy

   FINDING: Activation drop is 80% instrumentation, 20% real.
   - first_project_created undercounts on Android build 4.2 (event
     fires before auth completes) — confirmed by version split;
     ~9 of the 12 points.
   - Remaining ~3 points: new onboarding step adds friction on
     mobile web — confirmed by step-level funnel.
   TESTED AND KILLED: seasonality (YoY flat, EU control flat),
   mix shift (per-channel rates moved together).
   CONFIDENCE: high on instrumentation (reproduced in raw events);
   medium on friction (needs one more week of data).
   NEXT: fix Android event (eng ticket linked); watch mobile-web
   funnel; correct the historical series with a footnote.
   

5. 5.

   Feed the incident back into the system. Add the failure to a running "ways our metrics lie to us" doc, and if the cause was instrumentation, make the fix and the historical restatement explicit — silently corrected charts destroy trust in every future number.

Make it yours

• PostHog Max for branch-testing: ask Max to run each branch's query, but inspect every generated query per pb-27 — an investigation of a data problem is the worst possible place for an uninspected AI query.
• Sheets for the ledger: keep the tree in a spreadsheet — hypothesis, test, result, verdict, link — so the investigation is auditable and the next PM inherits the checklist rather than the folklore.
• Standing Claude Project: save the tree prompt with your product's known seasonality, release cadence and past instrumentation incidents in the project instructions; each investigation then starts from your product's actual history instead of a generic checklist.

Quality check

• The move was confirmed against historical variance before anyone spent an hour explaining noise.
• All four root branches were tested, not just the interesting onea tree where only "real change" got queries is a story, not an investigation.
• Instrumentation was tested firstnumerator and denominator events checked separately, split by version and platform.
• Killed branches are documented with evidence"we ruled out seasonality" carries the chart that ruled it out.
• The conclusion states confidence and what would change itfindings that can't name their own weakness aren't finished.
• Any AI-generated query was inspected before its result entered the ledger.