A team fixes the leakage in an inherited model and re-measures it properly. The reported accuracy falls from 0.9864 to 0.7278 and ROC-AUC falls from 0.9982 to 0.7534. Recall on the class that matters drops to 0.0985. The room goes quiet. Somebody says the fix broke the model. Somebody else suggests putting the removed columns back "only for the demo", or reporting the old figure with a footnote, or quietly reporting the highest of the several numbers on the screen. A manager who has already told a client "our model is 94 percent accurate" is waiting for the update.
Every project where an inherited or inflated number meets a first honest evaluation. It is the standard shape after removing target leakage, after replacing a random split with a time-ordered one, after moving evaluation off the training set, or after a holdout is scored for the first time. It also appears when a proof of concept becomes a product. P5 plants all four causes at once: two post-outcome columns carry 76 percent of the inherited model, the split is shuffled on time-ordered data, the weekly report scores the training rows, and a courier's late rate rises from 0.325 to 0.686 in the final months. Linked modules: P5 (Team project on inherited broken codebase), 1.7 (Leakage, imbalance, split discipline), 1.5 (Evaluation and metrics deep dive), 6.7 (Communication for Indian students).
Two things fell, and only one of them is a model. The measurement changed and the model barely did. Before you argue about the number, write down what you altered: which columns left the feature list, how the split was formed, and which rows the score was computed on. A score is only comparable to another score computed the same way, and these two were not.
Then read the honest table properly instead of reacting to one figure. On the P5 data, always predicting "on time" scores 0.7094 on the same test rows, so 0.7278 accuracy is almost no gain at all — but ROC-AUC of 0.7534 says the ranking carries real information, and recall of 0.0985 says the default threshold is throwing that information away. Those three numbers together are a work plan: change the threshold, report recall at the threshold the business would actually use, and stop quoting accuracy on a target that is late 27 percent of the time.
The per-segment view is where the real finding usually hides, and this is the part that turns a bad meeting into a good one. Split the honest score by courier and one row shouts: on the courier whose late rate more than doubled, accuracy is 0.366 and recall is 0.085. The model is worst precisely where the business changed. That is not a reason to hide the number. It is the most useful thing anybody has learned about this system, and it is invisible in both the fake score and the honest average.
The last part of the diagnosis is about people, and it is the part that ends careers rather than projects. The pressure to report the old figure is real and it is not stupid — somebody has already promised it to a client. But a number you cannot reproduce is a number you will be asked to reproduce, usually in front of the client, usually by a person who has the data. Reporting the inflated score is a decision to be found out later instead of now.
Immediate: stop anyone from putting the leaked columns back, even for a demo, even temporarily. A demo number becomes a slide, a slide becomes a commitment, and nobody remembers the footnote. If the demo is tomorrow, demo the honest model and say what changed.
Then produce one artefact: a before-and-after table with a row per change, not two numbers. Inherited pipeline 0.9864. Leaked columns removed 0.7711. Split by pickup date 0.7231. Courier rate rebuilt from past rows only 0.7278. Always-on-time baseline 0.7094 for reference. Each row names the change and one line of evidence, and the whole table fits on a page. This table answers the manager's real question, which is not "what is the number" but "which number can I repeat to a client".
Now build the message, in this order, because the order is what decides how it lands. One sentence on what changed and what the honest figure is. One sentence on why the old figure existed — a column that only exists after delivery cannot be known before delivery. One sentence on the business risk that number was hiding: a system trusted at 98 percent that actually catches under ten percent of late shipments would have sent nobody to the exceptions desk. Then what you propose next, with a date. Lead with the drop; do not bury it, and do not open with an apology.
Commit the change with the numbers in the message, and record the decision with a date and
a name in decisions.md. Add a test that fails if any post-outcome column re-enters the
feature list, so the fix cannot be undone by accident on a busy afternoon. Finally, publish
the per-segment table beside the headline, because the segment that broke is the reason the
project now has a next step.
Report a baseline beside every score, always. A model that beats nothing is a lot easier to spot when "nothing" is printed on the same line. Half of the panic in this incident comes from teams having no idea whether 0.73 is good.
Choose the headline metric before you see any results, and choose it from the decision the system supports. On an imbalanced target, accuracy will look respectable while the system misses almost everything it was built to catch. Write the chosen metric into the README on day one so nobody gets to pick the flattering one afterwards.
Make the evaluation script the only way anybody produces a number: one command, one dated report, features and split read from config. Numbers typed into slides by hand are how a figure with no owner survives for months.
Then make honesty cheap and normal inside the team. Every score claim carries the split, the rows and the date. A member who reports a drop gets thanked in front of the others, because the alternative is a team that learns to only report improvements. Under a first job, this habit is the one that gets you trusted with something bigger, and it costs nothing except one uncomfortable half hour.
Q: "You removed a leaked feature and your accuracy fell from 0.98 to 0.73. What do you tell your manager?" Weak answers apologise, or promise to get the number back up. Strong answer: separate the model from the measurement in the first sentence — the model barely changed, the measurement became real — and say what the old number was made of, naming the post-outcome column and the share of the model it carried. Give the honest figure with the baseline beside it so it can be judged, and add the metric that actually matters on an imbalanced target. Then show the before-and-after table with one row per change, so the manager can see exactly which decisions moved which number. Strongest answers finish with the segment finding and a next step: the model is worst on the one courier whose behaviour changed, that is where the next week goes, and here is the test that stops the leak coming back.