Full ML loop once, end to end
Tuesday evening in Patna. Aditya joined a grocery-delivery startup last month. His framing sheet from lesson one is approved: churn means no order in the next 60 days, and the metric is recall. His manager wants a working model before Friday's review. Aditya opens a tutorial playlist — forty hours of theory. His team lead shakes her head and closes the tab. Run the whole loop once today, she says, small and rough. Theory lands better after you have shipped one loop.
- One end-to-end loop on your own laptop teaches more than a week of theory, because every later concept now has a place to attach.
- Interviewers ask you to walk through a complete project; this loop is that walkthrough, from raw table to a saved model file.
- Every later module in Phase 1 — metrics, trees, tuning, leakage — upgrades one station of the loop you build today.
Every applied ML project you will ever ship is one loop with seven stations. Load data, split it, score a baseline, train, evaluate, improve, save. Today you run the whole loop once, on a small churn table, in under a hundred lines. The loop never changes; only the effort at each station grows.
The first station after loading is the train/test split — cutting your rows into a training set and a held-out test set before any model work. The test set is the exam the model has not seen. You lock it away first, the way an examiner locks the question paper. A model scored on rows it trained on is a student marking their own answer sheet. Split first is the habit that keeps every score in this course honest.
Inside scikit-learn, every model speaks the same two verbs. Fit — the step where the model reads training rows and learns a pattern from them. Predict — the step where it applies that stored pattern to rows it has never seen. There is no magic between the two: a fitted model is a small bundle of learned numbers. Master these two verbs and every model in this phase works the same way.
Here is the contrast that breaks first projects. The wrong way trains on all 400 rows and then tests on the same 400 rows: the score comes back near perfect, and it is a lie. The right way splits first, trains on 300, and sits the exam on the untouched 100. Same data and same model — one path reports a perfect score and collapses in production, the other reports the truth.
Lesson 1.1 taught you to score a dumb baseline before any model. In the loop, the baseline is a station, not a memory. Its score goes on record before training, and every model must beat it or it does not ship.
Models can also fail the exam in a special way. Overfitting — scoring high on training rows but poorly on unseen rows, because the model memorised examples instead of learning the pattern. A decision tree with no limits will show you this in one run today. When a model disappoints, you improve it by changing one thing and running the same loop again. One change per lap, or you will never know which change helped.
The loop ends with model persistence — saving the fitted model to a file so it can predict later without retraining. The joblib library saves in one line and loads in another. Save the feature list with the model, because the model remembers patterns but not your column names. A saved model plus its column list is the thing a teammate can actually use.
Follow along inside your ai-lab environment from Phase 0. One install, shown once — scikit-learn brings joblib along with it:
pip install scikit-learn==1.7.*
python -c "import sklearn; print(sklearn.__version__)"
The install line pulls scikit-learn plus its helpers into the active environment; the second line proves the import works and prints a version starting 1.7.. The pin is deliberate: every number printed in this phase — coefficients, accuracies, caught-of-26 counts — was produced on this version, and a different release can shift them just enough to make you doubt code that is actually correct. If it prints, you are ready. If it fails, WR-002's two proof commands will tell you which environment pip fed.
Station 1 and 2 — load, then split before anything else. We rebuild the 1.1-style churn table for Aditya's company — same target definition, no order in 60 days — and split immediately:
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(42)
n = 400
customers = pd.DataFrame({
"city": rng.choice(["Patna", "Gaya", "Muzaffarpur"], size=n),
"orders_last_90d": rng.poisson(6, size=n),
"days_since_last_order": rng.integers(1, 120, size=n),
"avg_order_value": rng.integers(120, 900, size=n),
})
risk = (-3.0 + 0.07 * customers["days_since_last_order"]
- 0.5 * customers["orders_last_90d"])
customers["churned"] = (rng.random(n) < 1 / (1 + np.exp(-risk))).astype(int)
features = ["orders_last_90d", "days_since_last_order", "avg_order_value"]
X = customers[features]
y = customers["churned"]
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.25, random_state=42, stratify=y)
print(customers["churned"].value_counts().to_dict())
print("train:", X_train.shape, " test:", X_test.shape)
print("churn rate train:", round(y_train.mean(), 2),
" test:", round(y_test.mean(), 2))
Output: {0: 294, 1: 106}, then train: (300, 3) test: (100, 3), then churn rate train: 0.27 test: 0.26. Read the shapes first — 300 rows to learn from, 100 locked away as the exam. stratify=y kept the churn rate nearly equal on both sides, so the exam is fair. The city column sits out for now: turning words into numbers is module 1.6's station. From this line on, X_test and y_test are untouchable until evaluation.
Station 3 — the dumb baseline, scored on the test set. Majority class from the training side, scored on the exam side:
majority = int(y_train.mode()[0]) # 0 = stayed
baseline_pred = np.full(len(y_test), majority)
baseline_acc = (baseline_pred == y_test).mean()
caught = ((baseline_pred == 1) & (y_test == 1)).sum()
print(f"baseline accuracy : {baseline_acc:.2f}")
print(f"churners caught : {caught} of {(y_test == 1).sum()}")
Output: baseline accuracy : 0.74 and churners caught : 0 of 26. Write both numbers down — this is the score to beat, exactly as 1.1 drilled. Note the baseline learned its majority class from y_train only; even a dumb rule respects the split.
Station 4 and 5 — first real model: LogisticRegression, fit then predict. Two verbs, then honest scores on both sides of the split:
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import recall_score
log_model = LogisticRegression(max_iter=1000)
log_model.fit(X_train, y_train)
test_pred = log_model.predict(X_test)
print(f"train accuracy : {log_model.score(X_train, y_train):.2f}")
print(f"test accuracy : {log_model.score(X_test, y_test):.2f}")
print(f"test recall : {recall_score(y_test, test_pred):.2f}")
print(f"churners caught: {((test_pred == 1) & (y_test == 1)).sum()} of 26")
Output: train accuracy : 0.84, test accuracy : 0.85, test recall : 0.54, churners caught: 14 of 26. The model beats the baseline on accuracy (0.85 vs 0.74) and catches 14 churners where the baseline caught none. Train and test scores sit close together — a healthy sign you will learn to read. fit learned the pattern from 300 rows; predict applied it to 100 rows it had never seen. That pair of verbs is the whole API.
Station 5 continued — a second model: DecisionTree, and a warning. Same verbs, different learner, plus a deliberate failure:
from sklearn.tree import DecisionTreeClassifier
deep_tree = DecisionTreeClassifier(random_state=42)
deep_tree.fit(X_train, y_train)
print(f"deep tree train: {deep_tree.score(X_train, y_train):.2f}"
f" test: {deep_tree.score(X_test, y_test):.2f}")
small_tree = DecisionTreeClassifier(max_depth=3, random_state=42)
small_tree.fit(X_train, y_train)
print(f"small tree train: {small_tree.score(X_train, y_train):.2f}"
f" test: {small_tree.score(X_test, y_test):.2f}")
print(f"small tree recall: {recall_score(y_test, small_tree.predict(X_test)):.2f}")
Output: deep tree train: 1.00 test: 0.71, then small tree train: 0.85 test: 0.81, then small tree recall: 0.73. Stare at the deep tree's pair: a perfect 1.00 on training rows, 0.71 on the exam — below the baseline. That gap is overfitting made visible; the unlimited tree memorised all 300 rows. Capping depth at 3 trades the fake perfection for an honest 0.81 and a recall of 0.73. If we had skipped the split, the deep tree's 1.00 is the number we would have reported. That is WR-058's whole story in two print lines.
Station 6 and 7 — one improvement lap, then save, load, predict. The framing metric is recall, so our one change asks LogisticRegression to weight churners more; then joblib makes the winner survive a restart:
import joblib
final_model = LogisticRegression(max_iter=1000, class_weight="balanced")
final_model.fit(X_train, y_train)
final_pred = final_model.predict(X_test)
print(f"balanced LR test accuracy: {final_model.score(X_test, y_test):.2f}"
f" recall: {recall_score(y_test, final_pred):.2f}")
joblib.dump({"model": final_model, "features": features},
"churn_model.joblib")
saved = joblib.load("churn_model.joblib")
model, cols = saved["model"], saved["features"]
new_customer = pd.DataFrame([{
"orders_last_90d": 1,
"days_since_last_order": 95,
"avg_order_value": 260,
}])[cols]
print("prediction :", model.predict(new_customer)[0])
print("churn prob :", model.predict_proba(new_customer)[0, 1].round(2))
Output: balanced LR test accuracy: 0.84 recall: 0.81, then prediction : 1 and churn prob : 0.99. One change — class_weight="balanced" — lifted recall from 0.54 to 0.81 (21 of 26 churners caught) while accuracy barely moved. On the framing sheet's metric, that is the winner, so it gets saved. Notice what we dumped: a dict holding the model AND the feature list, because WR-060 begins the day someone saves the model alone. The loaded model then meets a genuinely new customer — one order in 90 days, silent for 95 — as a one-row DataFrame in the saved column order, and calls the churn almost certain. That row never existed during training; this is the moment the loop pays.
Work inside ai-lab with scikit-learn installed. Each exercise ends with a check — you are done only when the check passes.
The whole loop, alone, on loans. Create loan_loop.py and run every station on a fresh problem: loan default at an NBFC. Build the table with np.random.default_rng(21) — 500 rows: monthly_income (25,000–150,000), emi (2,000–40,000), months_on_book (1–48). Target from a precise rule, 1.1 style: defaulted = 1 with probability 0.55 when emi exceeds 35% of monthly_income, else probability 0.04. Then, in strict order: split (test_size 0.25, random_state=42, stratified) and print both shapes; majority baseline with accuracy and defaulters-caught; LogisticRegression (train accuracy, test accuracy, test recall); DecisionTreeClassifier unlimited and with max_depth=3, printing train and test scores for both; one improvement lap with class_weight="balanced"; pick the winner on recall, save it with joblib (model plus feature list), reload, and predict one new applicant of your invention. Print a score line at every station. Check: the script prints shapes (375, 3) and (125, 3); the baseline shows accuracy 0.80 and 0 defaulters caught; the unlimited tree's train score exceeds its test score; the winner beats the baseline by catching at least 10 of the 25 test defaulters; running the script twice prints identical numbers; and loading loan_model.joblib in a fresh Python session predicts your new applicant without touching any training code.
Break the rules, watch the score lie. Copy your churn loop into cheat_loop.py and deliberately do it wrong: delete the split, fit an unlimited DecisionTreeClassifier on all 400 rows, and score it on those same 400 rows. Print the score. Now restore the split-first version beside it and print the honest pair: train score and test score. Below the code, write a three-line incident note as if the cheating number had been shown to Aditya's manager on Friday: line 1, the number reported; line 2, the number that was true; line 3, the one habit that would have prevented the gap. Check: the cheat version prints a score of 1.00; the honest version prints a visibly lower test score; both runs are seeded and repeatable; and the incident note names split-first (scoring only on unseen rows) as the prevention, not "use a better model".
A predict service a teammate could run. Split the loop into two files. train_and_save.py runs your churn loop and dumps churn_model.joblib holding the model and its feature list. predict.py must work in a fresh terminal with no training code imported: load the saved bundle; read three values from the command line (sys.argv) — orders in last 90 days, days since last order, average order value; build a one-row DataFrame and reorder it with the saved feature list; print the prediction and the churn probability plus one plain-language line a growth manager could read. Handle two failures kindly: wrong argument count prints a usage example instead of a traceback, and a missing model file prints "run train_and_save.py first". Add a four-line comment at the top of predict.py telling a teammate how to install and run it. Check: in a fresh terminal, python predict.py 1 95 260 prints a churn verdict with a probability and no traceback; python predict.py 5 prints the usage message; renaming the joblib file away produces the friendly train-first message; and predict.py contains no fit( call anywhere.
Three first-model failures cover most of the pain this loop meets in the wild. Each has a War Room entry with the full diagnosis path:
- WR-058 — the 99% mirage. A first model scored near-perfect because it was evaluated on its own training rows; production collapsed within a week. Split-first discipline finds the same truth in minutes, not weeks.
- WR-059 — fit works, predict explodes. 1D-versus-2D shape errors and shuffled column order at predict time; the fix is reading sklearn's error text and keeping one explicit feature list.
- WR-060 — the stale-model surprise. A saved model started predicting nonsense weeks later because input columns and dtypes had drifted since training; save the feature schema with the model and validate inputs on load.
The whole of applied ML is one loop: load, split, baseline, train, evaluate, improve, save. Split before anything else, because the test set is the exam the model must never see during training. Fit learns patterns from training rows, and predict applies them to rows the model has not met. A model earns its place only by beating the dumb baseline on the metric your framing sheet chose. Save the fitted model with its feature list, so it can score a brand-new customer tomorrow without retraining.
- Indian banks and NBFCs run this exact loop for credit scoring. The fitted model file is versioned and reused for months, and auditors ask how the test set was kept apart from training.
- Quick-commerce and food-delivery firms retrain churn models weekly; the saved model file is what the campaign system loads each morning to score every customer before vouchers go out.
- Logistics and insurance analytics teams must show train and test scores side by side in review; a wide gap between the two can pull a shipped model back.
Answer these before opening the gate assessment. Each question names its type.
type=output-prediction— An unlimitedDecisionTreeClassifieris fit on 300 training rows, then scored on those same 300 rows. Predict the printed score, and state whether its score on 100 held-out rows will be higher, lower, or equal — with one reason.type=debug— A teammate loads a saved model and runsmodel.predict([1, 95, 260]). sklearn stops with an error mentioning a 2D array. Name the cause in one sentence, and give a fix that also keeps the column names and their order safe.type=interpretation— A transcript reads: baseline accuracy 0.74, churners caught 0 of 26; model A test accuracy 0.85, recall 0.54; model B test accuracy 0.84, recall 0.81. The framing sheet's metric is recall. Which model ships for the retention campaign, and which single number carried your decision?type=scenario-judgement— Before Friday's demo a teammate argues: "Our table has only 400 rows — splitting wastes a quarter of them. Train on all 400 and report training accuracy." Give your judgement and the one-sentence reason you would say in the meeting.type=trade-off— Compare saving only the fitted model against saving the model together with its feature list and expected dtypes. Give one concrete advantage and one concrete cost of each, for a model a teammate will load and run one month from now.
Explain to a friend who believes ML needs deep maths what you did today, as a story about a shop: the register of past customers, the question paper you locked away before studying, the lazy guess any model must beat, and the notebook of learned patterns you saved for next month. Use at most 4 sentences and no technical word beyond "model". If your friend can retell why you hid 100 customers before training, you own this module.