Write Honest A/B Readouts That Preserve Uncertainty

Workflow area: Analytics & Experimentation · Tier: Core · Hands-on time: ~30 minutes

The experiment readout is where statistical caution goes to die. The analysis was careful; the summary says "13% lift 🎉"; the deck that quotes the summary says "conversion up 13%"; and six months later the roadmap is built on a point estimate whose interval ran from 2% to 24%. Language models amplify this decay — they are trained to write confident, satisfying narratives, and "the results were mixed and the interval is wide" is neither. Used with the right constraints, though, AI is excellent at translating statistical results into stakeholder language without dropping the uncertainty. The constraints are this playbook.

When to use — and when not to

• Use it for every experiment readout that leaves the data teamthe wider the audience, the stronger the pull toward overclaiming, and the more the honest version matters.
• Use it to rescue readouts you receivepaste a vendor's or a colleague's triumphant summary alongside the actual numbers and ask AI to find what the summary claims that the data doesn't support.
• Use it when the result is disappointing"no detectable effect" readouts are the hardest to write honestly and the most valuable; a null result properly framed saves the roadmap from a bad bet.
• Skip AI drafting when you haven't seen the raw results yourselfyou cannot audit a narrative against numbers you never looked at; the platform's results page comes first, then the draft.
• Skip publishing any segment finding the design didn't pre-registerinteresting segments go in a "hypotheses for next test" section, never in the headline.

The recipe

1. 1.

   Collect the actual statistics, not the platform's headline. From Statsig or GrowthBook, pull for each metric: point estimate, confidence interval, sample sizes per arm, run dates, and any SRM warning. The readout is written from these, never from memory.

2. 2.

   Draft with a prompt that makes honesty a hard constraint.

   Copy

   Write an experiment readout for a mixed audience (exec + product
   team). You must follow these rules ABSOLUTELY:
   
   1. Every effect claim includes its confidence interval in plain
      language ("most likely between X and Y").
   2. If a CI includes zero, the claim is "no detectable effect at
      our sample size" — never "no effect" and never a win.
   3. Distinguish pre-registered metrics from exploratory segment
      observations; label the latter "hypothesis only, not evidence".
   4. Include the novelty caveat if the test ran < 4 weeks or the
      feature is highly visible to existing users.
   5. State what decision the pre-registration committed us to,
      and what this result triggers.
   6. No adjectives stronger than the data: no "huge", "clear",
      "proves". Uncertainty language is a feature, not hedging.
   
   Pre-registered design: [paste decision sentence, primary metric,
   MDE, planned run length from pb-29]
   Results: [paste metrics with CIs, sample sizes, run dates,
   any SRM/quality warnings, exploratory segment cuts]
   

3. 3.

   Audit the draft against the numbers — the three deadly sins. Read the output looking specifically for:

   Copy

   SIN 1 — VANISHED INTERVAL: "Conversion improved 13%" where the
   data says +13% [+2%, +24%]. Fix: "improved, most likely between
   2% and 24% — directionally positive, imprecisely measured."
   
   SIN 2 — LAUNDERED SEGMENT: "Especially strong for Android users"
   from a post-hoc cut of 14 segments. Fix: move to "Exploratory:
   Android looked stronger (hypothesis for a follow-up test —
   with 14 cuts, one 'significant' segment is expected by chance)."
   
   SIN 3 — QUIET UPGRADE: pre-registration said "ship if ≥2pt lift";
   result is +1.1pt [+0.2, +2.0]; draft says "success, recommend
   rollout." Fix: the readout must say the pre-registered bar was
   not met, and any rollout argument is a judgment call, labeled so.
   

4. 4.

   Write the decision paragraph yourself. The AI drafts the findings; the "so what do we do" paragraph is yours, in your own words, referencing the pre-registered decision rule — including when you're overriding it and why. Stakeholders can accept "the bar wasn't met, but we're shipping for strategic reasons"; what destroys trust is discovering the bar moved after the results came in.

5. 5.

   End with the standard uncertainty block. Three fixed lines that appear in every readout you send: what this test cannot tell us, what we'd need to know it, and when the metric gets a post-rollout check. Ritualizing the block means including it costs no political capital.

Make it yours

• The adversarial second pass: paste your final draft into a fresh chat with "You are a skeptical data scientist. What does this readout claim that the data does not support?" — running your own overclaim-detector before your CFO does is the cheapest credibility insurance available.
• House readout template: encode the rules once (intervals mandatory, exploratory labeled, decision vs. pre-registration explicit) in a Claude Project or Custom GPT, so every PM's readouts come out with the same honesty floor.
• The rescue variant: when a "huge win" deck lands in your inbox, run its numbers through step 3's sins list — the laundered segment and the vanished interval account for most experiment folklore in most companies.

Quality check

• Every effect claim carries its interval in language a non-statistician parses correctly.
• CIs spanning zero are reported as "no detectable effect at this sample size"with the MDE the test was powered for, so "we couldn't see it" isn't read as "it doesn't exist."
• Exploratory cuts are quarantinedlabeled as hypotheses, with the number of segments examined stated.
• The readout names its pre-registered decision rule and says plainly whether the bar was met.
• Novelty and duration caveats appear where they applya two-week test on a visible feature says so.
• The decision paragraph is human-written and distinguishes what the data says from what you choose.