An imbalanced-data model reports a dream scorecard — precision and recall both in the eighties, PR-AUC above 0.9 on a problem where every earlier attempt fought for 0.4. In production the same model performs like the earlier attempts, and the review team drowns in false alarms nobody predicted. The training code has no obvious bug; the metrics code is textbook-correct.
A fraud table with roughly two percent positives. Someone read that SMOTE fixes
imbalance, imported an oversampler, and ran it on the entire table — then called
train_test_split on the balanced result. Every tutorial line afterwards is
correct; the damage was done in the order of two lines. Common wherever resampling
enters a codebase as a preprocessing step "like scaling" instead of as a
training-fold operation. The same accident happens with plain row duplication and
with undersampling applied before the split. Linked module: 1.7 (Leakage,
imbalance, split discipline).
Two corruptions landed at once. First, synthetic rows are interpolations between real minority rows — resampling before the split sends near-twins of training rows into the test set, so the model is quizzed on blurred copies of its own training data. Second, the test set's class balance became fifty-fifty, so precision is measured against a world with twenty-five times the real fraud rate. Both push scores up, and neither exists in production. Confirm cheaply: re-evaluate the same model on a raw, untouched split — in the demo world the dream 0.92 PR-AUC fell to 0.39. If the two evaluations disagree wildly, the resampling order is your incident. A test set whose positive rate does not match reality is the one-line tell.
Move the resampling inside the training side only: split first on raw rows, then resample the training fold and fit; the test set stays untouched, imbalanced, and honest. Re-report every affected metric on the raw test set and reset stakeholder expectations — the honest numbers will look like a regression, and the message must be that the old ones were never real. Then compare the train-only resampler against class_weight and threshold moving; on many tables they land within noise of each other, and the simpler tool wins.
House rule: resampling is a training-fold operation, never a dataset operation — resample train only, never test. Add one assertion to the evaluation harness: the test set's positive rate must match the raw data's rate to within a small tolerance, which turns this whole incident into a failing test. In review, any oversampler call that appears before a split call is an automatic block.
Q: "A candidate's project claims 0.92 PR-AUC on two-percent fraud using SMOTE. What do you ask them?" Strong answer: ask exactly where the resampling happened relative to the split, name the two corruptions if it was before — synthetic twins across the boundary and a rebalanced test set — and state the honest protocol: split first, resample the training fold only, report on the untouched test set. Adding the test-set-rate assertion as a guard shows production maturity.