Leakage, imbalance, split discipline
Demo day in Nashik. Ritu's first model at an appliance-service startup predicts customer complaints with 0.99 accuracy, and the room starts clapping. Her mentor asks one quiet question: when does the refund column get filled in? Ritu checks the data dictionary. A refund is processed after a complaint closes. Her model was reading the answer sheet, not predicting the future. At prediction time that column is empty, and the real score is 0.72. This module builds the habit that catches such traps before the clapping starts.
- The score that looks too good is the most dangerous thing a fresher can ship. It passes every review, then fails in production with your name on it.
- "Your model hit 0.99 overnight — what do you check first?" is a standard interview probe at Indian fintechs and banks, and this module is the full answer.
- Every later phase leans on today's split habits: deep learning, RAG checks and production monitoring all break the same way when test data touches training.
Module 1.5 taught you to read scores honestly; this module teaches you to distrust them first. Data leakage — information reaching training that will not exist at prediction time. A leaky model aces the test set and collapses in production, because production cannot leak. So adopt the reflex this module is built around: a sudden jump in score is a bug until proven otherwise. Great news that survives an audit is still great news; the audit is the price.
Leakage wears four faces, and each has its own cure. The first is target leakage — a feature that carries the answer because it is recorded after the outcome. Ritu's refund column is one; "recovery agent visited" in a default model is another. The audit question is brutal and short: could this column be filled in at the moment of prediction? If the value appears only after the event you predict, the feature is the answer in disguise.
The second face is temporal leakage — training on rows from the future and testing on the past. A random split shuffles the calendar, so Monday's model learns from Friday's frauds. The wrong way: shuffle six months of transactions and split at random, letting the model peek ahead. The right way: train on the past, test on the future, exactly as production will live it. Fraudsters change tricks, so the honest score is the one measured across time, not within it.
The third face is preprocessing leakage — fitting a scaler, imputer or encoder on all rows before the split. The fitted step memorises test rows, and the test set stops being the future. This is the crime 1.6's sealed Pipeline exists to prevent: fit on training rows only, replay everywhere else. How loud the damage is depends on how much target information the step memorises. A scaler leaks a whisper; a target encoder can invent skill on pure coin flips, which is why 1.6 told you to wait for this module.
The fourth face is group leakage — rows from the same customer landing in both train and test. The model meets a familiar income and age pattern on the test side and recognises the person, not the risk. The cure is to split by group: every customer's rows stay together, GroupKFold-style. Ask one question of every table: what is the unit I must not split apart?
Imbalance has honest tools, and 1.5 supplied most of them. class_weight tells the model that a rare positive costs more, and threshold moving picks the gate the business can afford. SMOTE — a resampling method that manufactures synthetic minority rows between real ones. Resampling can help a starved model, and it goes wrong one way: resample the training fold only, never the test set. A resampled test set answers a question nobody asked, because production arrives imbalanced.
Split discipline ties the whole module together. A stratified split — a split that keeps each class's share equal on both sides — is the default for imbalanced tables. Time-ordered data gets a time-based split, grouped data gets a group split, and the three rules stack when the table demands it. Keep the checklist from this lesson beside every project. It costs two minutes, and it is the cheapest insurance in your career.
Work inside ai-lab. Four leaks, one imbalance drill — every world is seeded, and every disaster is measured before it is fixed.
The feature that IS the answer. Ritu's complaint world: delivery delays and support calls drive complaints, and refunds happen after them:
import numpy as np
import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import recall_score
rng = np.random.default_rng(42)
n = 4000
orders = pd.DataFrame({
"delivery_delay_min": rng.integers(0, 90, size=n),
"order_value": rng.integers(100, 2000, size=n),
"support_calls": rng.poisson(0.8, size=n),
"orders_last_90d": rng.poisson(5, size=n),
})
risk = (-2.6 + 0.03 * orders["delivery_delay_min"]
+ 0.55 * orders["support_calls"])
orders["complained"] = (rng.random(n) < 1 / (1 + np.exp(-risk))).astype(int)
orders["refund_processed"] = np.where(orders["complained"] == 1,
rng.random(n) < 0.97,
rng.random(n) < 0.01).astype(int)
all_feats = ["delivery_delay_min", "order_value", "support_calls",
"orders_last_90d", "refund_processed"]
X_train, X_test, y_train, y_test = train_test_split(
orders[all_feats], orders["complained"],
test_size=0.25, random_state=42, stratify=orders["complained"])
leaky = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
leaky.fit(X_train, y_train)
lp = leaky.predict(X_test)
print("complaint rate:", round(orders["complained"].mean(), 2))
print(f"with refund_processed accuracy: {leaky.score(X_test, y_test):.2f}"
f" recall: {recall_score(y_test, lp):.2f}")
coefs = leaky.named_steps["logisticregression"].coef_[0]
for name, c in sorted(zip(all_feats, coefs), key=lambda t: -abs(t[1])):
print(f" {name:<20s} {c:+.2f}")
honest_feats = all_feats[:-1]
honest = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
honest.fit(X_train[honest_feats], y_train)
hp = honest.predict(X_test[honest_feats])
print(f"without the leak accuracy: "
f"{honest.score(X_test[honest_feats], y_test):.2f}"
f" recall: {recall_score(y_test, hp):.2f}")
complaint rate: 0.32
with refund_processed accuracy: 0.99 recall: 0.99
refund_processed +3.78
delivery_delay_min +0.82
support_calls +0.31
order_value +0.09
orders_last_90d -0.02
without the leak accuracy: 0.72 recall: 0.35
Read the unmasking. Accuracy 0.99 on a 32-percent complaint rate should have triggered the reflex on its own — 1.5's honest churn models lived near 0.80. The coefficient audit names the culprit: refund_processed carries +3.78 while every genuine signal sits below +0.82. A coefficient (or feature importance) towering over its neighbours is the classic fingerprint of a target leak. Drop the column and the model tells the truth: 0.72 accuracy, recall 0.35 — a weak model you can now improve honestly, instead of a brilliant one that was never real. Checklist line that catches it: "Could every feature be filled in at the moment of prediction?"
The time-travel model. Six months of payment transactions; fraudsters work nights early on, then switch to big-amount daytime hits. Watch the random split hide the change:
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.metrics import average_precision_score
rng = np.random.default_rng(7)
n = 9000
txn = pd.DataFrame({
"day": rng.integers(0, 180, size=n),
"amount": rng.integers(50, 15000, size=n),
"new_device": (rng.random(n) < 0.10).astype(int),
"night_txn": (rng.random(n) < 0.12).astype(int),
})
early = txn["day"] < 120
high_amount = (txn["amount"] > 13000).astype(int)
risk = np.where(early, -7.5 + 8.2 * txn["night_txn"],
-7.5 + 8.2 * high_amount)
txn["fraud"] = (rng.random(n) < 1 / (1 + np.exp(-risk))).astype(int)
feats = ["day", "amount", "new_device", "night_txn"]
Xr_train, Xr_test, yr_train, yr_test = train_test_split(
txn[feats], txn["fraud"], test_size=0.25, random_state=42,
stratify=txn["fraud"])
random_model = HistGradientBoostingClassifier(random_state=42)
random_model.fit(Xr_train, yr_train)
rp = random_model.predict_proba(Xr_test)[:, 1]
past, future = txn[txn["day"] < 120], txn[txn["day"] >= 120]
time_model = HistGradientBoostingClassifier(random_state=42)
time_model.fit(past[feats], past["fraud"])
tp = time_model.predict_proba(future[feats])[:, 1]
print("fraud rate:", round(txn["fraud"].mean(), 2))
print(f"random split PR-AUC: {average_precision_score(yr_test, rp):.2f}"
f" recall at 0.3: {recall_score(yr_test, (rp >= 0.3).astype(int)):.2f}")
print(f"time split PR-AUC: {average_precision_score(future['fraud'], tp):.2f}"
f" recall at 0.3: {recall_score(future['fraud'], (tp >= 0.3).astype(int)):.2f}")
Output: fraud rate: 0.09, then random split PR-AUC: 0.72 recall at 0.3: 0.96, then time split PR-AUC: 0.09 recall at 0.3: 0.10. The random split scores 0.72 because test rows come from the same weeks the model trained on — it learned "night rows before day 120, big amounts after" and got quizzed on both regimes it had already seen. The time split asks the production question: trained on months one to four, how do you do on months five and six? Answer: PR-AUC 0.09, recall 0.10 — the night rule is worthless against the new trick. The 0.72 was never a lie about the past; it was a lie about the future, which is the only thing a fraud model is for. Checklist line: "Does time order matter here, and does my split respect it?"
Preprocessing leakage, measured. A branch-level loan table where the target is a pure coin flip — so any score above 0.50 is fiction. First the quiet contamination, then the loud one:
from sklearn.impute import SimpleImputer
rng = np.random.default_rng(7)
n = 1200
loans = pd.DataFrame({
"branch": rng.choice([f"BR{i:03d}" for i in range(400)], size=n),
"monthly_income": rng.integers(20000, 120000, size=n).astype(float),
"emi": rng.integers(2000, 35000, size=n),
})
loans.loc[rng.random(n) < 0.35, "monthly_income"] = np.nan
loans["defaulted"] = (rng.random(n) < 0.5).astype(int) # pure coin flips
train_idx, test_idx = train_test_split(np.arange(n), test_size=0.25,
random_state=42, stratify=loans["defaulted"])
y = loans["defaulted"].to_numpy()
wrong_imp = SimpleImputer(strategy="median").fit(loans[["monthly_income"]])
right_imp = SimpleImputer(strategy="median").fit(loans.iloc[train_idx][["monthly_income"]])
print("imputer median from ALL rows :", wrong_imp.statistics_[0])
print("imputer median from train only:", right_imp.statistics_[0])
all_means = loans.groupby("branch")["defaulted"].mean()
loans["branch_enc"] = loans["branch"].map(all_means)
num = ["branch_enc", "monthly_income", "emi"]
wrong = make_pipeline(SimpleImputer(strategy="median"), StandardScaler(),
LogisticRegression(max_iter=1000))
wrong.fit(loans.iloc[train_idx][num], y[train_idx])
print(f"encoder fitted before split accuracy: "
f"{wrong.score(loans.iloc[test_idx][num], y[test_idx]):.2f}")
train_means = loans.iloc[train_idx].groupby("branch")["defaulted"].mean()
loans["branch_enc"] = loans["branch"].map(train_means).fillna(0.5)
right = make_pipeline(SimpleImputer(strategy="median"), StandardScaler(),
LogisticRegression(max_iter=1000))
right.fit(loans.iloc[train_idx][num], y[train_idx])
print(f"encoder fitted on train only accuracy: "
f"{right.score(loans.iloc[test_idx][num], y[test_idx]):.2f}")
Output, four lines: imputer median from ALL rows : 69758.0, imputer median from train only: 68912.0, encoder fitted before split accuracy: 0.73, encoder fitted on train only accuracy: 0.50. The two medians differ — proof the all-rows imputer read test rows while learning. For an imputer the damage is a whisper; for the target encoder it is a scream: 0.73 accuracy on labels that are literal coin flips. With 400 branches and three rows each, every branch mean memorises its own rows' answers, and the test rows arrive pre-answered. The train-only version scores 0.50 — the truth, because there was nothing to learn. Same habit, one cure: every fitted step lives inside 1.6's Pipeline, and the Pipeline meets test rows only at predict time. Checklist line: "Was anything fitted before the split?"
The customer on both sides. Three thousand loan applications from only 300 customers — repeat borrowers everywhere, each with an income-age pattern as unique as a signature:
from sklearn.model_selection import GroupShuffleSplit
from sklearn.metrics import roc_auc_score
rng = np.random.default_rng(7)
n_cust, n_rows = 300, 3000
cust = pd.DataFrame({
"monthly_income": rng.integers(20000, 120000, size=n_cust),
"age": rng.integers(21, 58, size=n_cust),
"habit": rng.normal(0, 2.4, size=n_cust),
})
idx = rng.integers(0, n_cust, size=n_rows)
apps = cust.iloc[idx].reset_index(drop=True)
apps["customer_id"] = idx
apps["loan_amount"] = rng.integers(10000, 300000, size=n_rows)
risk = (-1.4 + 0.9 * apps["habit"]
+ 0.000002 * apps["loan_amount"] - 0.000004 * apps["monthly_income"])
apps["defaulted"] = (rng.random(n_rows) < 1 / (1 + np.exp(-risk))).astype(int)
feats = ["monthly_income", "age", "loan_amount"]
Xg_train, Xg_test, yg_train, yg_test = train_test_split(
apps[feats], apps["defaulted"], test_size=0.25, random_state=42,
stratify=apps["defaulted"])
row_model = HistGradientBoostingClassifier(random_state=42)
row_model.fit(Xg_train, yg_train)
auc_rows = roc_auc_score(yg_test, row_model.predict_proba(Xg_test)[:, 1])
shared = len(set(apps["customer_id"].iloc[Xg_train.index])
& set(apps["customer_id"].iloc[Xg_test.index]))
print("customers in BOTH train and test:", shared, "of", n_cust)
print(f"random row split ROC-AUC: {auc_rows:.2f}")
gss = GroupShuffleSplit(n_splits=1, test_size=0.25, random_state=42)
tr_i, te_i = next(gss.split(apps, groups=apps["customer_id"]))
group_model = HistGradientBoostingClassifier(random_state=42)
group_model.fit(apps.iloc[tr_i][feats], apps["defaulted"].iloc[tr_i])
auc_group = roc_auc_score(
apps["defaulted"].iloc[te_i],
group_model.predict_proba(apps.iloc[te_i][feats])[:, 1])
print(f"split by customer ROC-AUC: {auc_group:.2f}")
Output: customers in BOTH train and test: 277 of 300, then random row split ROC-AUC: 0.79, then split by customer ROC-AUC: 0.53. The random split leaves 277 customers straddling the boundary. Each customer's hidden repayment habit drives their defaults, and their exact income-age pair appears on both sides — so the trees memorise "income 43,217, age 31: defaulted last time" and pass the test by recognition. GroupShuffleSplit keeps every customer's rows together, and the score falls to 0.53: on genuinely new customers, these three features know almost nothing. The 0.79 measured memory, not understanding. Checklist line: "Which rows belong to the same person, and do they stay on one side of the split?"
Imbalance done right — and the resampling trap. A 2-percent fraud table, four honest strategies, then the one that fakes it:
from sklearn.metrics import precision_score, average_precision_score
rng = np.random.default_rng(13)
n = 16000
fraud = (rng.random(n) < 0.02).astype(int)
pay = pd.DataFrame({
"amount": np.where(fraud == 1, rng.integers(3000, 20000, size=n),
rng.integers(50, 15000, size=n)),
"new_device": np.where(fraud == 1, rng.random(n) < 0.65,
rng.random(n) < 0.07).astype(int),
"night_txn": np.where(fraud == 1, rng.random(n) < 0.55,
rng.random(n) < 0.15).astype(int),
"fraud": fraud,
})
feats = ["amount", "new_device", "night_txn"]
def smote_lite(X, y, rng):
Xm = X[y == 1]
k = int((y == 0).sum() - (y == 1).sum())
i, j = rng.integers(0, len(Xm), k), rng.integers(0, len(Xm), k)
lam = rng.random((k, 1))
return (np.vstack([X, Xm[i] + lam * (Xm[j] - Xm[i])]),
np.concatenate([y, np.ones(k, dtype=int)]))
def row(tag, y_true, probs, cut=0.5):
p = (probs >= cut).astype(int)
print(f"{tag:<26s} precision {precision_score(y_true, p, zero_division=0):.2f}"
f" recall {recall_score(y_true, p):.2f}"
f" PR-AUC {average_precision_score(y_true, probs):.2f}")
Xtr, Xte, ytr, yte = train_test_split(pay[feats].values, pay["fraud"].values,
test_size=0.25, random_state=42,
stratify=pay["fraud"])
base = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)).fit(Xtr, ytr)
row("plain, gate 0.5", yte, base.predict_proba(Xte)[:, 1])
bal = make_pipeline(StandardScaler(), LogisticRegression(
max_iter=1000, class_weight="balanced")).fit(Xtr, ytr)
row("class_weight", yte, bal.predict_proba(Xte)[:, 1])
row("plain, gate moved to 0.10", yte, base.predict_proba(Xte)[:, 1], cut=0.10)
Xs, ys = smote_lite(Xtr, ytr, np.random.default_rng(0))
sm = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)).fit(Xs, ys)
row("resampled train only", yte, sm.predict_proba(Xte)[:, 1])
Xa, ya = smote_lite(pay[feats].values, pay["fraud"].values, np.random.default_rng(0))
Xtw, Xtew, ytw, ytew = train_test_split(Xa, ya, test_size=0.25,
random_state=42, stratify=ya)
wrong = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)).fit(Xtw, ytw)
row("resampled BEFORE split", ytew, wrong.predict_proba(Xtew)[:, 1])
Output, five rows: plain, gate 0.5 prints precision 0.65, recall 0.18, PR-AUC 0.40; class_weight prints 0.11, 0.82, 0.39; plain, gate moved to 0.10 prints 0.28, 0.55, 0.40; resampled train only prints 0.11, 0.80, 0.38; and resampled BEFORE split prints precision 0.85, recall 0.84, PR-AUC 0.92. Study the honest four first: PR-AUC barely moves, near 0.39, because class_weight, threshold moving and train-only resampling reshuffle the same ranking — they choose where on 1.5's precision-recall rope you stand, not how good the rope is. Then the fifth row: resampling before the split scattered synthetic points between real fraud rows, so near-twins of training frauds landed in the test set, and the test set itself became half fraud. A 0.92 PR-AUC on a problem the honest rows score at 0.39 — that number would survive a review meeting and die in week one of production. Checklist line: "Did any resampling or synthetic row touch the test set?"
Work inside ai-lab. Every world is seeded: where a check states a number, your number must match it exactly.
The leak hunt. Eight features, two traitors. In leak_hunt.py, with np.random.default_rng(31) and n=2500, build a loan table with columns drawn in this exact order: monthly_income integers 20,000-120,000; emi integers 2,000-35,000; months_on_book integers 3-60; bounced_payments Poisson(mean 0.8); city_tier chosen from 1, 2, 3; loan_amount integers 20,000-400,000. Then defaulted = 1 with probability sigmoid(-4.8 + 9*(emi/monthly_income) + 0.55*bounced_payments), drawn with one rng.random(2500) call. Now the traitors, in this order: account_frozen via np.where(defaulted==1, rng.random(2500) < 0.90, rng.random(2500) < 0.015) cast to int, and recovery_calls via np.where(defaulted==1, rng.poisson(4.0, 2500), rng.poisson(0.3, 2500)). Split 0.25, random_state=42, stratified. Model: make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000, class_weight="balanced")). Score all 8 features, print every coefficient sorted by absolute size, name the two leaked features in comments with a one-line reason each (both describe events that happen after default), then refit on the 6 clean columns and print the honest score. Check: the all-8 model prints accuracy 0.99 and recall 0.99; the audit shows recovery_calls at +4.25 and account_frozen at +2.34 while no clean feature passes ±1.20; the clean model prints accuracy 0.81, recall 0.84, catching 143 of 170 test defaulters; your comments say both columns are filled in only after a default exists.
The imbalance shootout. In imbalance_shootout.py, with np.random.default_rng(5) and n=12000, build a payments table drawn in this order: fraud as rng.random(12000) < 0.025 cast to int; amount via np.where(fraud==1, rng.integers(3000, 20000, 12000), rng.integers(50, 15000, 12000)); new_device via np.where(fraud==1, rng.random(12000) < 0.65, rng.random(12000) < 0.07) cast to int; night_txn the same shape with 0.55 and 0.15. Copy smote_lite from the watch block. Split 0.25, seed 42, stratified. Print one PR-based table with five rows — plain logistic at gate 0.5; class_weight="balanced"; plain at gate 0.15; smote_lite applied to the training rows only (resampler seeded np.random.default_rng(0)); and a final row labelled FAKE where smote_lite runs on the whole table before the split (same resampler seed). Below the table, write three sentences: which strategy you would ship for a review team that can handle about 500 flags a week, why the FAKE row's numbers are inadmissible, and what single habit prevents them. Check: the four honest rows print PR-AUC 0.39 (0.38 for train-only resampling is within print rounding); plain at 0.5 prints recall 0.21 while class_weight prints recall 0.83 with precision 0.15; the FAKE row prints precision 0.88, recall 0.86, PR-AUC 0.93; your sentences resample train only, never test, and pick a strategy using precision at the review team's capacity, not the FAKE numbers.
Split discipline drill, three tables. In split_drill.py, prove each rule on its own world. (a) Stratified: with np.random.default_rng(2), draw 240 labels where churn probability is 0.08; split indices 0.25 with seed 42 twice — once plain, once with stratify — and print train and test churner counts for both. (b) Temporal: rebuild the watch block's transaction world exactly but with np.random.default_rng(23), and print random-split PR-AUC against time-split PR-AUC (train days 0-119, test days 120-179). (c) Group: rebuild the watch block's loan-applications world exactly but with np.random.default_rng(3), and print random-row ROC-AUC against GroupShuffleSplit-by-customer ROC-AUC. Close with a three-line memo: for each table, name the split rule and the production question it answers. Check: (a) prints 21 total churners, with the plain split leaving only 2 in the test set while the stratified split keeps 5 — a test set that nearly lost its positive class; (b) prints 0.64 against 0.23; (c) prints 0.77 against 0.48; your memo maps stratified, time-based and group splits to imbalanced, time-ordered and repeat-customer tables respectively.
The leakage checklist you will keep. Write leakage_checklist.md — the artefact this module exists to leave in your pocket. Four sections named after the four leaks, each with at least two audit questions in your own words, plus a fifth section for imbalance with at least two more. End with a "verdict drill": three one-line proposals land on your desk — (i) a default model with a days_to_recovery_visit feature, (ii) a notebook that scales the full CSV before train_test_split, and (iii) a 12-month transaction model validated with a random split "because the month column is included as a feature". Under each, write the leak family and the checklist question that catches it. Check: the file has five sections and at least ten questions; the verdicts name target leakage, preprocessing leakage and temporal leakage in that order; each verdict quotes one of your own checklist questions; at least one question asks whether every feature exists at the moment of prediction, and at least one asks whether anything was fitted before the split.
Three leakage disasters cover most of what this lesson meets in production. Each has a War Room entry with the full diagnosis path:
- WR-073 — the feature that WAS the answer. A complaint model wore 0.99 in every offline test and collapsed on launch, because its best feature was recorded after the outcome it predicted; a prediction-time audit of every feature is the fix and the habit.
- WR-074 — the time-travel model. A random split let a fraud model train on the future and test on the past; the launch-week collapse was measured at one-eighth of the offline score, and a time-based split with a rolling backtest is the way home.
- WR-075 — the SMOTE-on-everything blunder. Resampling before the split planted synthetic near-twins of training rows inside the test set and inflated recall by half; resampling inside the training fold only, with the test set untouched, restores honest numbers.
A sudden jump in score is a bug until an audit proves otherwise. Target leakage hides the answer in a feature, temporal leakage trains on the future, preprocessing leakage fits steps before the split, and group leakage puts one customer on both sides. Each cure is one cheap habit: ask if the feature exists at prediction time, split by time, fit only inside the Pipeline on training rows, and split by group. Handle imbalance with class_weight, threshold moving and train-only resampling, and never let anything touch the test set. Keep the checklist near every project, because two minutes of questions beats a failed launch.
- Indian NBFC and bank credit scorecards face formal leakage reviews before launch. Every feature must be known at application time; a bureau field updated after default is Ritu's refund column at lakh-rupee scale.
- UPI fraud engines are judged on rolling time-based backtests, never random splits. Fraud tricks shift monthly, so a model that peeked ahead would score high and protect nobody.
- Kaggle-style contests now split by patient or customer id, after early contests were won by models that recognised people across the split instead of learning risk.
Answer these before opening the gate assessment. Each question names its type.
type=output-prediction— A complaint model scores accuracy 0.99 and recall 0.99 with arefund_processedfeature whose coefficient is +3.78, on a 32-percent complaint rate. Predict what happens to both numbers when the column is dropped and the model is refit, and explain why that drop is good news.type=debug— Overnight, a teammate's churn model went from 0.78 to 0.97 accuracy after a data engineer added twelve new columns from the CRM. Lay out your audit in order: the first question you ask about the twelve columns, the one artefact you print from the fitted model, and the experiment that settles it.type=interpretation— A fraud model's evaluation sheet reads: random split PR-AUC 0.72, time-ordered split PR-AUC 0.09. Explain what each number measures, why they can differ by a factor of eight on the same data, and which number belongs in the launch review.type=scenario-judgement— Before splitting, a teammate proposes running SMOTE on the whole table "so both halves stay balanced and the metrics are stable". Judge the proposal, name the two distinct corruptions it causes in the test set, and state the version of the plan you would approve.type=trade-off— For a 2-percent fraud table, compare class_weight, threshold moving and train-only SMOTE-style resampling. Give one concrete strength and one concrete cost of each, and say which you would reach for first when the ranking quality (PR-AUC) is already acceptable.
Explain to a friend who coaches students for board exams why a mock test only proves anything when the student has never seen the paper. Cover the classmate who memorised the answer key (the leaked feature), the mock test made of last year's questions the student already practised (the future leaking into training), and why practising harder on weak topics is fine while sneaking those topics' answers into the mock is cheating (train-only resampling against a resampled test). Use at most 4 sentences and no word fancier than "answer key". If your friend can say why a clean mock that scores lower is worth more than a leaked one that scores higher, you own this module.