A decision tree posts a perfect (or near-perfect) training score and looks brilliant in the demo. Days or weeks after going live, its predictions on real customers are close to random — sometimes worse than the do-nothing baseline. Confidence stays high the whole time: the model asserts wrong answers without hesitation.
DecisionTreeClassifier (or Regressor) fit with default settings — which means
max_depth=None and min_samples_leaf=1 — on a small or medium tabular set: churn
tables, loan applications, lead scoring. The demo was scored on rows the model had
trained on, or the test score was printed once, noticed to be low, and explained away
as "the data being noisy". Linked module: 1.4 (Trees, RF, boosting, XGBoost); the
symptom first appears in 1.2's loop.
An unlimited tree keeps splitting until its leaves are pure — in practice, until it has
built a private memory cell for nearly every clump of training rows. Check three
numbers. (1) tree.get_depth() and tree.get_n_leaves(): depth above 8-10 and a leaf
count approaching the training row count (51 leaves for 300 rows in the lesson demo)
means memorisation, not learning. (2) The train/test pair printed side by side: 1.00
against 0.71 is the signature — a wide gap is overfitting made visible. (3) Compare the
test score with the majority-class baseline; a memorising tree often lands below it.
The perfect training score is the symptom, never the achievement: any unlimited tree
can hit 1.00 on data it has seen, so that number carries zero information about new
customers.
Cap the tree's capacity and re-score honestly. Set max_depth between 3 and 6, or
min_samples_leaf to 10 or more (both together is fine), refit, and print train and
test scores side by side — expect the train score to FALL (0.85 in the lesson) while
the test score rises (0.71 to 0.81). If a single capped tree is not accurate enough,
move up the same family: a random forest or HistGradientBoostingClassifier, judged on
the same locked test set. Never "fix" the gap by re-scoring on training rows. If the
model already shipped, pull it back behind the previous rule or baseline while you
refit — a confidently wrong scorer does damage every hour it runs.
Print train and test scores together, every run, as a habit — one number alone hides the gap. Treat any training score at or near 1.00 as an alarm that demands a capacity check. Never accept sklearn tree defaults for anything that ships; defaults exist to demonstrate the API, not to protect you. Keep a majority-baseline number on record so "below baseline" is visible instantly, and add a train-test gap threshold (say 0.10) to your review checklist or CI.
Q: "Your decision tree has 100% training accuracy. Is that good news?" Strong answer: no — an unlimited tree can memorise any training set, so 1.00 says nothing about new data; quote the depth/leaf-count check, the train-test gap as the real health metric, and the fix (depth or leaf limits, then judge on a held-out test set). Mentioning that a random forest's 1.00 train score is normal — because the vote, not one tree, is the model — separates candidates who understand the mechanism from those who memorised a rule of thumb.