Statistical-Test Selection and Assumption Checking with AI as Sparring Partner

Tier: Foundation · Workflow area: Statistical methodology

Used as an oracle ("which test should I use?" → run whatever it says), it launders your uncertainty into false confidence.

When to use

• •
  Choosing between test families for a real decision: t-test vs Mann-Whitney, chi-square vs Fisher, ANOVA vs mixed model, frequentist vs bootstrap.
• •
  Reviewing an experiment design before data collection (power, unit of randomization, stopping rules).
• •
  Sanity-checking someone else's statistical claim before you rely on it.

Skip it when a pre-registered analysis plan already fixes the method — follow the plan.

Prerequisites

• •
  Python with scipy.stats and statsmodels (or R equivalents) to actually run tests and diagnostics.
• •
  A capable general assistant (Claude, GPT); no special tooling required.
• •
  A precise statement of: the decision at stake, unit of analysis, sample sizes, how the data was collected. Vague inputs produce textbook answers to the wrong question.

The workflow

1. 1.

   State the problem as a decision, not a test request:

   Copy

   Decision: whether to roll out a new onboarding flow. Data: 4,800 users
   randomized 50/50 at signup; outcome = completed setup within 7 days
   (binary); secondary = time-to-completion (right-skewed, 22% censored at 7d).
   Randomization unit = user; some users share a company account.
   Do NOT recommend a test yet. First list what additional facts would change
   the correct analysis, and ask me for them.
   

   The "ask me first" step surfaces issues (clustering by company, censoring) that a direct answer would have skipped.

2. 2.

   Get a recommendation with explicit assumptions. After answering its questions: Recommend an analysis. For each assumption it requires, state how to check it on my data and what to do if it fails. Expect something like: two-proportion z-test (or logistic regression with company-clustered SEs), Kaplan-Meier or log-rank for the censored secondary — with checks attached.

3. 3.

   Run the assumption checks as code, not conversation. Ask for a diagnostic script:

   Copy

   # e.g. cluster-size distribution, intra-class correlation for clustering;
   # for a t-test path: normality is irrelevant at n=4800, but check for
   # heavy tails/outliers driving the mean; for proportions: expected cell counts
   import statsmodels.formula.api as smf
   m = smf.logit("completed ~ variant", data=df).fit(cov_type="cluster",
        cov_kwds={"groups": df["company_id"]})
   

   Execute it yourself; paste the outputs back for interpretation.

4. 4.

   Force the adversarial pass. In a fresh thread (to avoid agreement momentum):

   Copy

   A colleague analyzed this experiment as follows: [paste method + result].
   Argue AGAINST this analysis as a hostile reviewer. Find at least 4 concrete
   problems, ranked by how much each could change the conclusion. Then state
   which criticisms are serious vs cosmetic for this decision.
   

   The final ranking matters — assistants generate ten objections easily; calibrating which ones bite is the skill you are practicing.

5. 5.

   Stress-test the conclusion. Ask for the robustness menu and run the top items: cluster-robust vs naive SEs, with/without outlier caps, bootstrap CI vs asymptotic. If the decision flips under any reasonable variant, the honest answer is "not robust," not the variant you prefer.

6. 6.

   Write the methods paragraph. Have the assistant draft it from what you actually ran — test, assumptions checked and outcomes, effect size with CI, robustness results — then edit. Never let it draft from the plan instead of the executed reality.

Tool variants

• General chat assistant (Claude/GPT): the default; strongest at the sparring and interpretation layers.
• Assistant inside the notebook (Cursor, Jupyter AI): tightest loop for step 3 — diagnostics run next to the conversation; keep the adversarial pass in a separate chat regardless.
• Two-model cross-examination: give the same design to two different vendors' models and reconcile disagreements yourself; disagreement is a cheap flag for genuinely contested choices.

Verification

• •
  Every assumption behind the chosen method has an executed check with saved output.
• •
  Effect size reported with a confidence interval, not just a p-value.
• •
  The decision survives (or is honestly reported as sensitive to) the robustness variants.
• •
  The unit-of-analysis question (clustering, repeated measures) was explicitly resolved, not defaulted.
• •
  You can explain, without the assistant, why the chosen method beats the runner-up.

Pitfalls

• Oracle mode. Asking "which test?" and running the answer skips the entire value: the checks and the argument. The prompts above are structured to prevent it.
• Agreement momentum. Long threads drift toward validating your framing; fresh threads for critique are non-negotiable.
• Assumption theater. Running Shapiro-Wilk on n=4,800 (it rejects trivially and normality barely matters there) instead of checks that bite, like clustering and outlier influence. Ask "which assumption failure would actually change the decision?"
• P-hacking by robustness menu. Running five variants and reporting the significant one. Decide the primary analysis first; robustness is reported alongside, not swapped in.
• Hallucinated formulas or critical values. Any numeric claim (power calculation, critical value) must come from executed statsmodels/scipy code, not the model's arithmetic.

Restricted-environment variant

• •
  This workflow needs designs and aggregate outputs, not raw data — describe distributions and paste summary diagnostics, keeping row-level data local. That makes it viable in most regulated settings with hosted models.
• •
  Fully offline: local 8B+ instruct models handle test-selection sparring credibly; be stricter about verifying any named theorem or formula against a textbook, and run all computation in local scipy/statsmodels.
• •
  For pharma/clinical contexts: the assistant is a rehearsal partner, not a substitute for the statistical analysis plan or a qualified statistician's sign-off.

Success criteria

• •
  A defended method choice with executed assumption checks for a real analysis.
• •
  At least one design flaw or violated assumption caught this way that your first instinct missed.
• •
  The adversarial pass produced a criticism serious enough to change the analysis or the write-up.
• •
  A methods paragraph a statistician colleague accepts without major corrections.