Offline evals (pb-34→pb-38) tell you a change should help; only an online experiment tells you it did — with real users, real inputs, and real stakes.
Tier: Advanced · Workflow area: Evaluation Engineering
This is where an experienced data scientist outruns the prompt-engineer crowd: LLM teams routinely ship on "the new prompt feels better" with no power analysis, no guardrails, and peeking-driven false positives. This playbook transfers your classical experimentation rigor to LLM changes — prompt swaps, model bumps, retriever updates — with the LLM-specific twists: noisy judge-based outcome metrics, cost/latency guardrails, and sequential monitoring so you can stop early without lying to yourself.
Skip the experiment for changes below your minimum detectable effect or with trivial blast radius — ship behind a flag with monitoring instead; experiments are for decisions worth their sample cost.
Write the experiment design doc first (one page). Hypothesis with direction and size ("model B raises draft-acceptance rate by ≥ 2pp"), primary metric (exactly one), secondary metrics, guardrails with stop thresholds, unit of randomization, power analysis output (n and expected duration), analysis plan (test, sequential method, slice list — pre-registered), and the ship/no-ship decision rule. Pre-registration is what makes the answer credible when it is inconvenient.
Randomize by user, stratify if traffic is lumpy. Request-level randomization contaminates within-user experience and violates independence (users issue correlated requests). Hash user ID → arm at the gateway; log (user, arm, timestamp) immutably. For B2B products with whale accounts, stratify or use account-level assignment — one enterprise tenant landing in arm B can dwarf the treatment effect.
Do the power analysis with LLM-realistic variance. Get the primary metric's baseline mean and variance from 2 weeks of logs — judge-scored quality metrics are much noisier than click metrics, so expect larger n than intuition suggests. Standard formula, honest inputs: for a proportion metric, n ≈ 16·p(1−p)/δ² per arm rule-of-thumb at α=0.05, power 0.8. Two variance-reduction levers before enlarging n: CUPED-style regression adjustment using each user's pre-experiment metric, and paired judging on identical prompts where the same input can safely receive both arms' responses offline (interleaving-style) — often cutting required n by 30–50%.
Set guardrails as hard stops, not dashboards. Per arm: p95 latency ceiling, cost/request ceiling, error+refusal-rate ceiling, safety-violation count (from pb-38-style checks). Breaches trigger automatic ramp-down, not a meeting. LLM changes fail operationally more often than qualitatively — a model that is 3% better and 2× slower usually loses, and you want to learn that at 5% traffic.
Ramp with sequential monitoring — peek legally. Launch at 5% → 20% → 50% following the ramp plan. You will look at results daily, so choose a method that permits it: group-sequential bounds (O'Brien-Fleming via alpha-spending) or always-valid inference (mSPRT/confidence sequences). Naive daily t-tests at α=0.05 push false-positive rates toward 25–30% over a few weeks of peeking — this is the single most common LLM-experimentation malpractice, and the one your training exists to prevent.
Sample traces into judge scoring with QC. For quality metrics, route a fixed sample per arm/day through the pb-36 judge (same pin, both arms, order-randomized when pairwise). Keep the 5–10% human re-grade flowing during the experiment — a judge drifting mid-experiment is a differential measurement error that quietly manufactures a winner.
Analyze exactly what you pre-registered. Primary metric with CI under the sequential method; guardrail deltas; pre-registered slices only (intent type, user tenure, input length) with multiplicity correction — everything else is labeled exploratory for the next experiment's hypothesis. Then the LLM-specific decomposition: join outcomes to traces and ask where the win came from (which intents, which retrieval-quality bands) — pb-34 trace reading applied to experiment cells.
Decide by the rule, write the memo, archive everything. Ship/no-ship per the pre-registered rule; memo records design, actuals vs plan, primary/guardrail results, surprises, and the follow-up hypothesis. Archive assignment logs and analysis code — model-migration experiments get audited later ("when did quality actually change?") more than any other kind.
Every prompt tweak that ships without an experiment is an experiment with n=all-your-users and no control group — rigor here is not slower, it is the only version where you learn the answer.
statsmodels + a confidence-sequence library does the analysis — fully sufficient and boundary-friendly.Randomizing users into a materially worse-quality arm has ethics weight in high-stakes domains (medical, financial guidance) — cap exposure, ramp conservatively, and clear such experiments with the product owner in writing.