Cross-validation is stable and cheerful — five folds, tight spread, ROC-AUC around 0.72 every time. The model is shipped on that number. The first monthly review on real new borrowers comes back near 0.65, and the gap does not close in month two or month three. There is no leaked column, no bug in the pipeline, and no error in the metric code. The offline number was measured correctly; it was measuring the wrong thing.
$ python evaluate_splits.py
strategy ROC-AUC (5-fold mean)
random rows 0.720
grouped by customer_id 0.672
grouped by pan_hash 0.661
borrowers with more than one loan : 2602 of 3000
distinct pan_hash values : 2888 for 3000 customer ids
Any table where one row is a transaction and one person can produce many rows: loan applications, insurance claims, hospital visits, support tickets, orders. Lending books are the worst case because repeat borrowing is the business model — a delivery rider finances a second scooter, a shop finances three. It gets a second layer in Indian lending, where the same person is often re-registered under a new customer id after a branch change, a re-KYC, or a name spelt two ways, so even a customer-level split still puts one human on both sides. Linked modules: P1 (Credit-risk predictor + API), 1.7 (Leakage, imbalance, split discipline), 1.8 (CV, tuning, model selection).
Work out what the model was actually rewarded for. Each borrower carries a hidden repayment habit that the application fields do not record. Their income, age, existing EMI and bureau score together form a near-unique signature. With the same person on both sides of a random split, a tree ensemble can memorise "this exact signature went bad last time" and score the test rows by recognition rather than by risk.
That memory is worthless in production, because every application the model scores after launch belongs to somebody whose earlier rows the model may never have seen — and never has seen, for a genuinely new borrower. So the offline number answers "how well do I recall people I have met?" while the business asked "how well do I judge people I have not?"
The three-way comparison in the symptom is the whole diagnosis. Random rows: 0.720.
Grouped by customer_id: 0.672. Grouped by pan_hash: 0.661. The first gap, about
0.048, is ordinary repeat-borrower leakage. The second gap, about 0.011, is the one
teams miss — 3,000 customer ids resolve to only 2,888 distinct PAN hashes, so 112 ids
are the same people twice, and grouping on the id that the system happens to expose is
not the same as grouping on the person.
Three things make this failure easy to miss. It produces no warning. It leaves the cross-validation spread tight, so the folds look trustworthy. And it fails in the flattering direction, so nobody goes looking.
One more check settles it before you argue about anything: count how many group keys appear in both train and test under your current split. If the answer is not zero, the score is not the score you think it is.
Step 1 — decide what a group is, in the business's words. It is the unit that must never be split apart, and it is a person or a household, not a row id. On a lending book, the most stable key you hold wins: a hashed PAN beats a customer id, and a customer id beats nothing.
Step 2 — find duplicate identities before you split. Group the table on the stable key and count distinct customer ids per key. Anything above one is a re-registration, and it belongs in the EDA write-up as a finding, not as a surprise on defence day.
Step 3 — use one split function, everywhere. Put GroupKFold on the stable key
inside a single time_group_split() in src/split.py, and let nothing else in the
repo call train_test_split directly. A repo with three ways to split is a repo where
somebody will pick the wrong one at midnight.
Step 4 — assert it in the test suite. One test, and it never stops paying:
def test_no_group_crosses_the_split():
train_idx, test_idx = time_group_split(df)
shared = set(df.loc[train_idx, "pan_hash"]) & set(df.loc[test_idx, "pan_hash"])
assert shared == set(), f"{len(shared)} borrowers on both sides"
Step 5 — restate every downstream number. The threshold sweep, the rupee cost table and the monitoring floor were all computed on the inflated score, so they are all wrong by the same amount. Redo them, and note in the decision log that the earlier figures were withdrawn and why.
Step 6 — stack the rules when the table demands it. This book is grouped and time-ordered. Train on the past, test on the future, and keep every borrower's rows on one side of that line. Grouping alone still lets the model train on next year.
Ask one question of every table before splitting it: what is the unit I must not split apart? Write the answer in the README beside the target sentence. Count group keys per person early, because duplicate identities are normal in Indian systems and are cheap to find and expensive to discover late. Keep the both-sides assertion as a permanent test rather than a one-off check, so a future refactor cannot quietly undo it. And when a model's live performance sits below its offline score with no other explanation, check the split before you retune anything — repeat units are the most common cause, and retuning on a leaky split makes the gap wider.
Q: "Your cross-validation says 0.72 and production says 0.65, month after month. Where do you look?" Strong answer: rule out drift first by naming what you would check, then go to the split, because a stable gap that never closes points at the offline number rather than at the world. Explain the mechanism in one sentence — repeat borrowers seen on both sides let the model recognise people instead of judging risk — and give the confirming count of group keys shared across the split. Add the layer that impresses: the id the system gives you may not be the person, so you group on the most stable key you hold and check for duplicates first. Weak answer: retraining on more data, which inflates the same number again.