A model was crowned champion in an offline bake-off — say 0.86 against a rival's
0.83 — and shipped on the strength of that comparison. In production it performs
like the rival, or worse. When someone reruns the comparison notebook with a
different random_state on the split, the winner flips. Nobody changed the data,
the features, or either model; only the shuffle moved, and the crown moved with it.
A team compares two or more candidate models using a single train_test_split,
reads the two scores, and declares the higher one the winner. Common in demo-week
rushes, college projects, and any codebase where the evaluation cell was written
once in week one and never questioned. The smaller the dataset, the worse the
wobble: at a few hundred test rows, three to six accuracy points of pure split
luck is normal. The verdict was real for that split — and only for that split.
Linked module: 1.8 (CV, tuning, model selection).
A single split score is one draw from a distribution, not the distribution. Which hard rows land in the test set changes the score of every model, and changes it by different amounts for different models — so a gap of two or three points between candidates is routinely inside the noise. Confirm cheaply, in five minutes: rerun the identical comparison with five different split seeds and print the table. If the winner changes across seeds, or the gap between the models is smaller than the spread within one model, the original comparison measured luck. In the 1.8 demo world, one pipeline scored anywhere from 0.80 to 0.86 across five seeds — a spread wider than most bake-off margins.
Replace the single-split verdict with cross-validation before any claim: run
cross_val_score (cv=5, stratified by default for classification) for every
candidate on the same folds, and compare mean ± std, not two lucky numbers. If
the means sit within one std of each other, say so out loud and pick the simpler
model — that is a finding, not a failure. Re-run the original bake-off this way
and re-issue the recommendation; if the crowned model no longer wins, the honest
correction now costs a meeting, while the same correction after launch costs the
team its credibility.
House rule: no model comparison leaves the notebook without fold-wise mean ± std for every candidate — a single-split score is a smoke test, never evidence. Keep the evaluation harness seeded but run the five-seed wobble check once per project so the team knows its own noise floor. In review, any slide claiming "A beats B" must state the CV spread; a gap smaller than the spread is reported as a tie.
Q: "Two models score 0.86 and 0.83 on the test split. Do you ship the first one?" Strong answer: refuse to conclude from one split; state that split luck at this data size is often bigger than a three-point gap; propose 5-fold CV on shared folds and a mean ± std comparison; and add the tie-breaker — if the means land within one std, pick the simpler model. Mentioning that you would seed the folds so the comparison is reproducible signals real production habit.