Design Experiments with AI-Checked Power Math

Workflow area: Analytics & Experimentation · Tier: Core · Hands-on time: ~35 minutes

The quiet scandal of product experimentation is how many tests could never have detected the effect they were run to find. A PM hopes for a 2% lift, has traffic for a 10% MDE, runs the test anyway, reads "no significant difference," and kills a good feature. AI is a patient tutor and a competent calculator here — it will happily walk the power math and name the pitfalls — but it will also happily bless a bad design if you prompt it leadingly. The recipe forces the math into the open where you and your data scientist can check it.

When to use — and when not to

• Use it before every experiment brief is finalizedthe design review is the cheapest moment to discover you need eight more weeks of traffic than you have.
• Use it to sanity-check a design someone hands youa vendor's "proven uplift" test plan or a HiPPO's "just run it for a week" both deserve the same math.
• Use it to learn the vocabularyMDE, power, ratio metrics, CUPED; asking AI to explain each choice in your test's concrete numbers beats any statistics course for retention.
• Skip full A/B machinery when traffic makes it hopelessif the math says 26 weeks to detect your MDE, the honest answer is a different method (holdout geo, painted door, qualitative signal), not a doomed test.
• Skip AI as the final authorityit makes arithmetic slips and accepts bad assumptions without protest; the design that ships gets a human data-science review whenever the decision is consequential.

The recipe

1. 1.

   Write the decision before the design. One sentence: "If variant B lifts paid conversion by ≥2 points, we roll it out; otherwise we keep A." No decision sentence, no experiment — tests run "to see what happens" always end in segment-fishing.

2. 2.

   Have AI work the power math with every step shown. The prompt that prevents black-box answers:

   Copy

   Act as a careful experimentation statistician. Design an A/B test
   and SHOW EVERY CALCULATION STEP so a human can verify the math.
   
   Primary metric: [e.g. "signup → paid conversion, currently 8.0%"]
   Weekly eligible traffic: [e.g. "9,000 users/week, 50/50 split"]
   Minimum effect worth acting on: [e.g. "absolute +1.5 points"]
   Settings: alpha 0.05 two-sided, power 0.80.
   
   1. Compute required sample size per arm from these inputs —
      show the formula and the arithmetic, then the run length
      in weeks at my traffic.
   2. Show how run length changes if MDE is 1.0 / 1.5 / 2.0 points.
   3. Flag every pitfall that applies to THIS design: peeking,
      ratio-metric variance (revenue per user), weekly cycles and
      why runs should cover whole weeks, novelty effects, SRM.
   4. Propose 2-3 guardrail metrics that could silently degrade.
   5. List the assumptions you made that I should challenge.
   

3. 3.

   Verify the headline number independently. Paste the same inputs into your platform's calculator — GrowthBook and Statsig both ship one, Evan Miller's calculator is the classic — and compare:

   Copy

   AI:            n ≈ 5,900 per arm → ~2.6 weeks... arithmetic shown
   GrowthBook:    n = 6,040 per arm → 1.4 weeks at full traffic
   Discrepancy:   AI divided by total weekly traffic, not per-arm.
   Corrected:     6,040 × 2 / 9,000 ≈ 1.35 → run 2 full weeks.
   

   A small discrepancy usually means different variance approximations (fine); a large one means someone's inputs or arithmetic are wrong (find out whose before proceeding). Round the run length up to whole weeks.

4. 4.

   Stress-test the design adversarially. Second pass, fresh context: "Here is my experiment design. You are a skeptical data scientist reviewing it. What is wrong with it? What will the most likely misleading readout be?" This surfaces the things you didn't ask about — contaminated assignment, primary metric that's really a ratio metric, an MDE chosen from hope rather than business value.

5. 5.

   Freeze the design in a pre-registration note. Primary metric, MDE, sample size, run length, guardrails, and the decision sentence — written down before launch, shared with data science, and attached to the experiment in GrowthBook or Statsig. The readout (pb-30) will be judged against this note, which is what makes segment-fishing visible later.

Make it yours

• GrowthBook/Statsig-native teams: configure the platform's sequential-testing or CUPED options with your data scientist once, then have AI explain what those settings mean for YOUR run lengths — the platforms prevent naive peeking, but only if you understand what they're doing.
• Low-traffic products: ask AI for the honest alternatives ladder at your traffic level — bigger-MDE tests on high-traffic surfaces, before/after with a control geo, or qualitative validation — and take the "this is not testable as an A/B" answer seriously.
• Team calibration: keep a shared sheet of past experiments — MDE assumed, effect found, run length — and feed it to the AI when designing; your product's real effect sizes are the best prior for the next MDE conversation.

Quality check

• Every calculation step is visiblean AI answer with a bare final number is unusable; regenerate with the show-your-work instruction.
• Sample size verified against a second calculatorand any large discrepancy diagnosed, not averaged.
• The MDE traces to business value"the smallest effect we would act on," not "the effect we hope for."
• Run length covers whole weekly cycles and the peeking policy is explicit (fixed horizon, or the platform's sequential method).
• Guardrail metrics are named with directionswhat could this change silently make worse?
• The decision sentence and design are pre-registered with data science before launch — the readout gets judged against this note.