A script's accuracy, simulated fraction, or sampled split changes on every run. A teammate cannot reproduce the number you reported yesterday. A test passes on your laptop and fails in CI with a slightly different value. Or the opposite trap: someone runs an experiment three times, sees 0.7 once, and announces a conclusion with full confidence.
First simulations, train/test splits, and anything touching np.random,
random, or shuffling. Peaks on demo day, when the number on screen refuses to
match the number on the slide, and in week-one experiments where sample sizes
are tiny. Linked module: 0.7 (The maths you will actually use).
Two separate failures hide behind one symptom. (1) Irreproducibility: random draws differ on every run unless you fix a seed, so each run is a different experiment — nothing is "wrong", but nothing is comparable either. (2) Misread noise: small samples swing hard — ten flips of a fair coin giving 0.7 heads is completely normal, so a difference measured on a tiny sample is usually noise wearing a costume. Separate them with one test: rerun with a fixed seed. If the numbers now match, the swing was sampling noise — grow the sample and watch the estimate settle. If they still differ, a second randomness source (another library's RNG, set ordering, parallelism, time-based logic) is injecting change.
Create one generator at the top — rng = np.random.default_rng(42) — and pass
it into every function that draws numbers; never scatter fresh unseeded
generators through the code. When several libraries are involved, seed each one
(random.seed, NumPy, and later torch.manual_seed). For claims: increase the
sample until the estimate stops moving, then rerun with several different seeds
and report the mean and the spread — one lucky run is not a result. Log the seed
next to every reported number so anyone can replay it.
Seed on the first line of any experiment script, before writing anything else. Distrust any conclusion drawn from a sample you could count by hand. Remember the law the coin demo showed: averages settle only as the sample grows, so decide the sample size before looking at results. In review, ask two questions of every reported metric: "what seed?" and "how many samples?" — most week-one claims fail one of them.
Q: "A teammate says the new model is better — it scored 2% higher on one run. How do you respond?" Strong answer: ask whether the split and training were seeded, how large the test set is, and what the score's spread is across several seeds; propose rerunning with multiple seeds and reporting mean plus spread. Distinguishing reproducibility (seeds) from statistical noise (sample size) in one breath is exactly what separates engineers from lucky guessers.