Weeks of tuning show steady progress: the validation score climbs from 0.79 to 0.84, sometimes 0.9-plus, and every experiment log entry beats the last. At the final review — or on a contest's private leaderboard, or in production — the score collapses to something near where the tuning started, or below it. The "best" configuration often looks strange too: an odd feature subset, an extreme regularisation value, settings nobody can explain.
A team tunes many rounds — forty, a hundred, more — always scoring against the
same validation split, and keeps whichever change nudges that number up. Common
in Kaggle-style work (the public leaderboard is exactly one fixed validation
set), in student projects where X_val was created once in cell 3, and in any
workflow where "try, score, keep if higher" runs in a loop. The validation set
was honest on round one; nobody notices the moment it stops being honest,
because every individual step looks correct. Linked module: 1.8 (CV, tuning,
model selection).
Selection is a form of training. Each keep-if-higher decision leaks a little of the validation set's answer key into your choices; over forty rounds the winning config has effectively been fitted to those particular rows — their noise included. With an 80-row validation set, a config that got lucky on a handful of rows tops the leaderboard while learning nothing general. Confirm with the 1.8 exercise pattern: score the tuned winner on a held-out set it has never influenced. In the demo world, the best-of-120 config read 0.84 on the split it was tuned against and 0.69 on the sealed test — a fifteen-point mirage. A second tell: the winner's config leans on noise features or extreme values that no domain story supports.
Rebuild the selection protocol. First, seal a final test set now and declare it touched-once — every past peek has already spent some of its honesty, so if it has been scored dozens of times, carve out a fresh one. Second, select using cross-validated means instead of one split's score, so no single set of rows can be memorised through repeated choosing. Third, cut the number of tuning cycles: a budgeted RandomizedSearchCV run scores a planned set of configs once, instead of an open-ended loop that grinds against the same rows for weeks. Re-report the honest number to stakeholders before someone else discovers it.
House rules: the final test set is scored exactly once per project milestone, and its score is written into the report the same day; model selection happens on CV means, never on a single fixed split; tuning is a planned budget of fits, not an open loop. Track how many times the evaluation set has been consulted — a counter in the experiment log works — and treat a rising consult-count with a rising score as the incident signature. For interviews and contests alike: trust your CV, submit sparingly.
Q: "Your validation score improved for three straight weeks, then the final test came in far lower. What happened?" Strong answer: name overfitting the validation set — selection leaks the answer key, so best-of-many on one split measures luck on those rows; give the cure — CV for selection, a sealed test touched once, a fixed tuning budget; and name nested cross-validation as the rigorous version, an outer loop scoring the entire tuning procedure. Candidates who add "the public leaderboard is one validation split" show they have lived it.