A first model reports a spectacular score — 99%, sometimes a clean 1.00 — and everyone celebrates. Days or weeks after it ships, real predictions are wrong far more often than the notebook promised. Stakeholders ask why the demo number was fake. Re-running the notebook still prints the same beautiful score, which makes the confusion worse: the code "works", yet production disagrees.
Almost always a first end-to-end model: a fresher's project, a hackathon entry, or
tutorial code adapted in a hurry. The split step was skipped, done after fitting, or
quietly broken — train_test_split exists in the file but the score call still uses
the full dataset, or the model was refit on everything "for the final version" and
then scored. Flexible models (unlimited decision trees, k-NN with k=1) make the mirage
most extreme. Linked module: 1.2 (Full ML loop once, end to end).
The rows used to evaluate the model are rows it already saw during fit. Fit stores
patterns from training data, and a flexible model can store the training rows almost
perfectly — so scoring on those rows measures memory, not learning. Confirm it in
three steps. (1) Read the score line: do the arguments to score/predict overlap
the data passed to fit? If both say X with no split, this is your failure.
(2) Print the pair: score on training rows and on held-out rows. A mirage shows a
huge gap — 1.00 train against a test score that may fall below the dumb baseline.
(3) Check the order of operations: a split performed after fitting, or a "final refit"
on the full table before scoring, contaminates the exam even though a split appears
in the file. The tell-tale everywhere: near-perfect scores with no baseline printed
next to them.
Rebuild the loop split-first. Immediately after loading, cut the data with
train_test_split (fixed random_state, stratify for classification) and treat
the test set as locked: fit and every experiment touch only the training side, and
the test set is scored once, at the end. Re-score the existing model honestly —
expect the number to fall; that fall is information, not damage. Report the honest
pair (train score and test score) beside the baseline, and re-set expectations with
whoever saw the mirage number. If the test set was used for repeated tuning along the
way, it is burnt: re-split with a fresh seed and retrain from scratch.
Make the split the first code after loading, every time, and print the shapes so the
habit is visible in output. Always print train and test scores as a pair — a gap is
your earliest overfitting alarm. Never report any score without the baseline next to
it. In review, search for fit( and check what each later score/predict call
receives; "final refit on all data, then score" is the classic smuggled version of
this bug.
Q: "Your new model shows 99% accuracy. What do you do before telling your manager?" Strong answer: distrust it first — verify the score came from rows the model never trained on, show the train/test pair, and compare against the majority baseline. Mention that flexible models can memorise training data perfectly, so a training-side score is not evidence of learning. Candidates who celebrate first and validate later fail this question; the interviewer is testing evaluation discipline, not modelling skill.