A two-wheeler default model scores 0.96 ROC-AUC on the held-out slice and keeps scoring it through two review meetings. Recall at the default gate is 0.92 on a book where only one loan in nine goes bad. Nobody can find a mistake, because there is no error message and no crash — the numbers are simply far better than any credit scorecard anybody in the room has built before.
$ python train.py
rows 9000 | bad rate 0.109 | split: random, 25% test
ROC-AUC 0.965 PR-AUC 0.909 recall@0.5 0.922
top coefficients (standardised):
recovery_agent_assigned +2.32
vehicle_type_ev_scooter +0.80
bureau_num -0.68
employment_type_small_b -0.47
emi_to_income +0.31
Credit-risk work on an exported table, where the export is not a snapshot of application day but a dump of the loan's whole life so far. Collections columns, restructuring flags, settlement fields and recovery notes all ride along in the same CSV as the application fields. It bites hardest on a first project, because the student takes the file as given and models every column in it. Very common in NBFC and bank data pulled from a core system with one wide query. Linked modules: P1 (Credit-risk predictor + API), 1.7 (Leakage, imbalance, split discipline), 1.11 (Interpreting models).
Read the coefficient table before anything else. One feature carries +2.32 while every genuine application-time signal sits under +0.80. A single column towering over its neighbours is the fingerprint of target leakage, and 1.7 named it.
Now ask the question that settles it. recovery_agent_assigned records that a
recovery agent was put on the case. A recovery agent is put on the case because
the loan went past due. The column is therefore written after the outcome it is being
asked to predict, and at the moment the model must actually run — an application on
the counter, no money paid out yet — the field is empty for every borrower alive.
The trap is that nothing in the column name says "after". refund_processed,
settlement_flag, restructured_on, days_to_first_recovery_call and
agent_visit_count all read like ordinary features. Only the business meaning gives
them away, which is why this is a conversation with the operations team and not a
statistics problem.
Two false comforts keep it alive. First, "the correlation is high, so it must be a strong feature" — correlation with the target is exactly what a leak has, at 0.91 on this book. Second, "the test set is untouched, so the score is honest" — the test set is untouched, and every row in it also carries the leaked column, so the split discipline is fine and the feature list is not. Leakage from the future survives a perfect split.
Confirm it in one run: fit the same pipeline twice on the same split, once with the column and once without, and print both scores. If the drop is large, you have your answer and you also have the honest number you should have been quoting.
Step 1 — build the time-of-knowledge table before the next model run. One row per column, three possible marks: known at application, known after disbursal, known after outcome. Write the justifying sentence beside each mark, in business words.
column known when in the model?
------------------------------------------------------------------
monthly_income at application yes
bureau_score at application yes
down_payment_pct at application yes
emi_amount at application yes
disbursal_date at disbursal date only, no leak
recovery_agent_assigned AFTER the loan is bad NO - target leak
Step 2 — exclude it in code, not by deleting a notebook cell. The excluded list
belongs in config.py, with the reason on the same line, so the exclusion survives a
refactor and a reviewer can find it in five seconds.
Step 3 — refit and publish the honest number. On this book the same pipeline goes from 0.96 ROC-AUC to roughly 0.71 on a time-ordered, group-aware split. That is not a regression. That is the first true measurement the project has produced, and every downstream decision — the threshold, the cost table, the monitoring floor — must be rebuilt on it.
Step 4 — say it out loud, early. Tell whoever saw the 0.96 that the number was wrong and why, before somebody else finds it. A student who reports their own leak reads as an engineer. A student whose leak is found by a reviewer reads as a risk.
Run the time-of-knowledge table on day one, from the data dictionary, before a model exists — it takes twenty minutes and it prices every later week. Ask one question of the person who owns the source system: "which of these fields are written after disbursal, and which after a default?" That single question catches more leaks than any amount of modelling. Treat any jump in score as a bug until an audit clears it, and keep the two-run experiment — with the column, without it — as your standard proof. When a column name contains recovery, settlement, restructure, write-off, collection or agent, assume it is from the future until somebody proves otherwise.
Q: "Your credit model scores 0.96 and the team's production scorecard scores 0.72. What do you check first?" Strong answer: say the gap itself is the evidence, then name target leakage as the first hypothesis and the coefficient or importance table as the one artefact you print. Give the audit question in the business's own words — could this field be filled in at the moment we decide? — and name the class of columns that usually fails it in lending. Finish with the two-run experiment and what you would do with the result, including telling the room the earlier number was wrong. Weak answer: proposing more cross-validation or a bigger test set, which measures the same lie more carefully.