The test suite fails about one run in five, each time in a slightly different place, each time with slightly different numbers — recall 0.61 against a 0.65 assert today, 0.58 tomorrow, green the run after. Rerunning without any change "fixes" it. The team's chat fills with "red again, rerunning" until the phrase becomes a joke, and then red stops meaning anything at all.
An ML repo whose tests grew without the determinism habits: a sample drawn
without a random_state, a train_test_split missing its seed, and asserts that
compare floats for exact equality (assert recall == 0.6666666). Each test is
individually reasonable-looking; together they roll dice on every run. Weeks in,
a genuinely broken commit failed the suite — and was merged anyway, because
"that test is always flaky". The suite existed and cost effort, yet protected
nothing. Linked modules: 4.5 (Testing ML code), 4.1 (seeds in config).
A flaky test is a test whose verdict changes with no code change, and the cause
in ML code is almost always one of two things. First, unseeded randomness: any
sample, split, shuffle or init without a fixed seed makes the tested numbers
themselves random, so a tight assert must sometimes lose. Second, exact float
equality: even seeded pipelines can differ in the last decimal places across
library versions and machines, so == on a float is a coin with a heavy but
imperfect bias. Confirm by running the suite twenty times in a loop and logging
outcomes per test: the flaky ones cluster exactly where randomness or exact
equality lives. The deeper damage is social, not technical — once rerun-until-
green is normal, the suite's red carries no information, and real failures ride
through with the noise.
Three moves. One: seed everything the tests touch — samples, splits and model
inits all read config.SEED, so every run computes the same numbers. Two: replace
exact float equality with pytest.approx for hand-worked values and with
threshold floors (recall >= 0.60) for model scores, which is what the quality
gate should have been anyway. Three: any test still flaky after seeding is
quarantined the same day — marked and excluded from the green/red verdict — with
a ticket to fix or delete it; a quarantined test is honest, a flaky one in the
main suite is poison. Then prove the cure the same way the disease was proved:
twenty consecutive runs, twenty identical results.
A standing team rule: red means broken — if a red can be ignored, the test must
be fixed, quarantined or deleted today, never rerun into silence. Review every
new test for unseeded randomness and exact float comparisons before merge, the
same reflex as reading a COPY . . in a Dockerfile. Keep the twenty-run loop as
a cheap flakiness audit whenever the suite grows. Determinism is what makes a
test's verdict evidence instead of weather.
Q: "Your CI is red about once in five runs and developers have started rerunning until it passes. What do you do?" Strong answer: name the two usual causes in ML suites — unseeded randomness and exact float equality — give the twenty-run log as the confirming measurement, and the three-part fix of seeds, tolerances-or-thresholds, and same-day quarantine. The senior signal is the social point: a suite that cries wolf trains the team to ignore the one red that matters.