Design an A/B test properly — MDE, sample size, guardrails, run length, with the math shown — then critique a deliberately flawed experiment readout that peeks, fishes segments, and overclaims significance. Rewrite its executive summary honestly. Three planted flaws are scored against the answer key.
Nisha, PM for learner retention at PrepLoop (the fictional Hyderabad edtech from Lab 05), is launching a quiz-streak nudge: a home-screen card prompting learners to keep a daily practice-quiz streak. Her growth counterpart already ran a "quick test" of a similar nudge and circulated a triumphant readout. Nisha's job — and yours — is twofold: design the next test so it can actually be trusted, and take the existing readout apart before it reaches the leadership deck.
EXPERIMENT DESIGN INPUTS
Feature: quiz-streak nudge card on the learner home screen
Primary metric: week-1 practice-quiz adoption (learner starts >= 1
practice quiz within 7 days of first exposure)
Current baseline: 22% of new learners
Minimum detectable effect (MDE): +3 percentage points (22% -> 25%)
Significance level: alpha = 0.05, two-sided
Power: 0.80
Eligible new learners per day: ~450 (split 50/50)
Guardrail metrics: lesson completion rate, notification opt-out rate,
day-7 app uninstalls
The growth team's readout of their earlier nudge test. Critique this in steps 5–7.
READOUT — "Streak Nudge Test: A Clear Win" (growth team, v1)
1. We launched the streak nudge to 50% of new learners and monitored
results daily on the live dashboard.
2. By day 9 the treatment group's quiz adoption was 24.1% vs 21.8%
control, p = 0.04. Since we had crossed significance, we stopped
the test early to capture the win sooner.
3. Digging deeper: the effect is strongest for Android users aged
18-24 in tier-2 cities (+6.2pp, p = 0.03). We examined 12
demographic and device segments to locate this driver segment.
4. Notification opt-outs rose from 3.1% to 4.4% in treatment, but
this is expected with any nudge and does not change the picture.
5. Conclusion: the nudge PROVES a causal 2.3pp adoption lift and a
6.2pp lift in our key segment. Recommend immediate 100% rollout
and applying streak nudges to all other surfaces this quarter.
Generate the design with the math shown. Paste the design inputs with: "Design this A/B test. Show the sample-size formula and every substitution step, then compute: required n per arm, total n, and run length in days given 450 eligible learners/day. State what each guardrail metric is protecting against and what change in each would pause the rollout." Demand shown work — a bare "you need about 3,000 users" is unverifiable.
Verify the sample-size math yourself. Use the two-proportion formula with z-values 1.96 (α = 0.05, two-sided) and 0.84 (power 0.80): n per arm = (1.96 + 0.84)² × [p₁(1−p₁) + p₂(1−p₂)] / (0.03)², with p₁ = 0.22, p₂ = 0.25. Work it in your sheet. If the AI's number differs from yours by more than rounding, make it redo the substitution step by step until you find whose arithmetic slipped — AIs fumble this calculation surprisingly often while narrating it confidently.
Fix the run length for reality. Convert n to days (total n ÷ 450/day), then round up to whole weeks and say why: day-of-week effects mean a test that ends mid-week samples Mondays and Saturdays unequally. Add the second reality check — your primary metric needs 7 days to mature after last enrollment, so the readout date is enrollment window plus one week. Write the final design as five lines: n per arm, enrollment days, readout date rule, guardrail thresholds, and the pre-registered decision rule ("ship if primary lift ≥ X with p < 0.05 AND no guardrail breach").
Pre-register the segment policy. One paragraph, written before any data exists: which segments (if any) you will analyze, and the rule that segment results are hypothesis-generating unless the segment was named in advance. This paragraph is the vaccine for what you are about to see in the readout.
Critique the readout — solo pass first. Read the growth team's readout and list every methodological problem you can find, numbered, with the line it appears on. Do this before asking AI. You are looking for problems in how the conclusion was reached, not the arithmetic.
Critique with AI, then reconcile. Prompt: "You are a statistician reviewing this A/B test readout. List every methodological flaw, why each invalidates or weakens the stated conclusion, and what the honest interpretation would be." Merge its list with yours. For each flaw, write one sentence you could say in a meeting that names the problem without jargon — "we stopped the moment the number looked good, which makes the number look better than it is" lands; "alpha inflation under optional stopping" does not.
Rewrite the executive summary honestly. Five to seven lines, keeping every number, changing only the claims: what the test suggests, what it cannot support and why, what the guardrail movement means, and what you recommend (typically: rerun with the step-3 design, pre-registered; treat the segment as a hypothesis; investigate the opt-out cost before any rollout). The tone test: a statistician and a CEO should both sign it.
Score against the key. Open the answer key and check: did you catch all three planted flaws, and did your rewrite avoid introducing new overclaims in the other direction (calling the test "worthless" is also wrong — it is weak evidence, not zero evidence)?
Assessment scenarios love the early-stopping trap — any readout that says "we crossed significance so we stopped early" is describing the flaw, not a justification, and the planted correct answer is the one that reruns with a fixed horizon.
A complete lab has all of the following:
The design math is the easy half — the discipline is fixed horizons, pre-registered segments, and guardrails that can veto a win. Any readout that stopped when the number looked good, found its story in segment twelve, or waved away a guardrail is telling you how it fooled itself.