Framing an ML problem
Monday, 10 am. Neha starts her second week at a food-delivery startup in Chandigarh. The founder stops at her desk: customers are leaving, build some AI and fix it. Neha opens a notebook and types import — then freezes. Predict what, exactly? Which customers count as leaving, and since when? What number will prove the model works? Her senior slides over a one-page framing sheet. Today you learn the questions that come before any model — the step most freshers skip.
- Most dead ML projects failed at framing, not modelling — the team built a fine answer to the wrong question.
- Interview case rounds open with a fuzzy business ask; the framing steps in this lesson are the answer shape panels expect.
- Every later module in Phase 1 assumes you can name the target, the metric, and the baseline before writing code.
Machine learning learns patterns from examples instead of following rules a person wrote. You show it many past customers with a known outcome, and it finds the signals that separate them. Ordinary software is the opposite: a person writes the rule, and the machine obeys it. Both answer questions; they differ in who writes the logic.
So the first professional question is never which model. It is: do we need a model at all? If a bank blocks any transfer above a set limit at 2 am, a two-line rule does the work. Rules win when the logic is known, stable, and small. ML earns its cost only when the pattern is too messy for anyone to write down.
When ML is the right tool, name the problem type next. Classification — predicting which category a row belongs to — answers questions like will this loan default, yes or no. Regression — predicting a number — answers questions like how many minutes late this delivery will be. A third type, ranking, orders items by likely interest, the way a food app sorts restaurants. Phase 1 lives in the first two.
Now the hardest and most skipped step: defining the target variable — the single column of past outcomes the model must learn to predict. Everything else in the project depends on this column existing and being right. The other columns are features — the input signals the model may use, such as orders per month or days since the last login. Features describe the customer on the day of prediction; the target records what happened after.
Here is the contrast that decides projects. The wrong way accepts a fuzzy target: predict customer unhappiness, with no measurable definition anywhere. No table in the company records unhappiness, so nothing can be learnt and nothing can be checked. The right way turns the ask into a measurable event with a time window: no order in the next 60 days. That sentence becomes a real column, built from order dates the company already stores.
Choose the success metric — the single number that will say whether the model is good — before any modelling starts. Pick it with the business side, because they own the cost of each mistake type. Missing a leaver may cost a customer; a false alarm may cost one discount voucher. Accuracy sounds safe but hides total failure when one class is rare, as WR-057 shows.
Last, set the baseline — a score from a dumb rule that every model must beat. Predict the most common class for everyone, or predict last month's value again. Compute its score on the chosen metric before training anything. A model that cannot beat a one-line rule is not worth shipping. The full path is a funnel, from fuzzy ask down to a number to beat.
Watch one business ask travel the whole funnel before you try it. Everything runs offline with pandas and NumPy — no sklearn, no downloads, fixed seeds so your numbers match mine.
Step 1 — the framing sheet. The founder's ask, taken through the six framing steps on paper first:
BUSINESS ASK : "Customers are leaving. Build some AI and fix it."
1. Real question : which current customers are likely to stop
ordering soon?
2. Target variable : churned = 1 if the customer places no order in
the 60 days after the snapshot date, else 0
3. Problem type : classification (two classes: churned / stayed)
4. Features : facts known ON the snapshot date only —
orders_last_90d, days_since_last_order,
avg_order_value, city
5. Success metric : recall on the churned class (agreed with the
growth team: missing a leaver costs the most)
6. Baseline : predict the majority class for everyone
Read line 2 twice — it is the whole lesson. "Leaving" became a measurable event with a time window, computable from order logs the company already has. Line 4 has its own discipline: every feature must be known on the snapshot date, or the model peeks at the future. This sheet took twenty minutes and no code; teams that skip it lose weeks.
Step 2 — build the customer table. Synthetic data, seeded so it reproduces exactly:
import numpy as np
import pandas as pd
rng = np.random.default_rng(42)
n = 400
customers = pd.DataFrame({
"cust_id": [f"C{i:04d}" for i in range(n)],
"city": rng.choice(["Chandigarh", "Ludhiana", "Amritsar"], size=n),
"orders_last_90d": rng.poisson(6, size=n),
"days_since_last_order": rng.integers(1, 120, size=n),
"avg_order_value": rng.integers(120, 900, size=n),
})
print(customers.head(3))
print(customers.shape)
head(3) shows customer C0000 from Chandigarh with 4 orders in 90 days, 34 days since the last one, average order value 698. shape prints (400, 5) — four hundred customers, five columns, all of them features so far. Notice what is missing: there is no target column yet. A table of features with no outcome column cannot train anything — that is the state most real projects start in.
Step 3 — build the target column from the definition. In real work this label comes from the next 60 days of order logs; here we simulate that future with the same seed discipline:
p_churn = np.where(customers["days_since_last_order"] > 60, 0.28, 0.03)
customers["churned"] = (rng.random(n) < p_churn).astype(int)
print(customers["churned"].value_counts())
print(customers["churned"].value_counts(normalize=True).round(2))
The counts print 347 stayed and 53 churned; the normalized view says 0.87 versus 0.13. Two things happened here. First, the fuzzy word "leaving" is now a concrete 0/1 column named exactly as the framing sheet promised. Second, the classes are imbalanced — 13% churn — and that fact must drive the metric choice, as the next step proves.
Step 4 — the dumb baseline and its score. Predict the majority class for every customer, then score it honestly:
majority = customers["churned"].mode()[0] # 0 = stayed
baseline_pred = pd.Series(majority, index=customers.index)
accuracy = (baseline_pred == customers["churned"]).mean()
caught = ((baseline_pred == 1) & (customers["churned"] == 1)).sum()
total_churn = (customers["churned"] == 1).sum()
print(f"baseline accuracy : {accuracy:.2f}")
print(f"churners caught : {caught} of {total_churn}")
The output reads baseline accuracy : 0.87 and churners caught : 0 of 53. Sit with that pair of lines. A rule with zero intelligence scores 87% accuracy while catching not one single churner — which is why the framing sheet chose recall, where this baseline scores 0.00. Any model the team builds must now beat these numbers on the metric that matters, not on accuracy. This is WR-057 in miniature, defused before a model exists.
Step 5 — a regression baseline: predict yesterday. The same discipline for a number-predicting ask — how late will deliveries run tomorrow:
rng2 = np.random.default_rng(7)
deliveries = pd.DataFrame({
"date": pd.date_range("2026-07-01", periods=28),
"avg_delay_min": (12 + 6 * np.sin(np.arange(28) / 4)
+ rng2.normal(0, 2, 28)).round(1),
})
deliveries["pred_yesterday"] = deliveries["avg_delay_min"].shift(1)
mae = (deliveries["avg_delay_min"]
- deliveries["pred_yesterday"]).abs().mean()
print(deliveries.head(3))
print(f"baseline MAE: {mae:.1f} minutes")
shift(1) copies each day's delay one row down, so every day is "predicted" by the day before it. The mean absolute error of that lazy rule prints baseline MAE: 2.0 minutes. Predicting the overall mean instead scores 4.2 minutes — worse, so predict-yesterday is the honest bar here. A delay model that misses by 2.5 minutes is not a working model; it loses to a one-line shift. Every regression project needs this number written down before training starts.
Do these inside your ai-lab environment with pandas installed. Each exercise ends with a check — you are done only when the check passes.
Rules or ML? Create rules_or_ml.md. Copy in these eight asks: block UPI transfers above ₹1 lakh between 1 am and 5 am; predict which loan applicants will default; flag orders where the delivery PIN code differs from the billing PIN code; predict tomorrow's electricity demand for a city; give free delivery on orders above ₹499; predict which insurance claims are fraudulent; auto-reject job applications missing a phone number; predict how many days a shipment from Chandigarh to Kochi will take. For each, write one line: RULE, ML, or RULE-FIRST (start with a rule, revisit later), plus a one-sentence reason naming whether the logic is known and stable or messy and pattern-like. Then take these three fuzzy targets and rewrite each as a measurable event with a time window: "predict customer unhappiness", "predict risky borrowers", "predict late deliveries". Check: all eight asks have a verdict and a reason, at least three verdicts are RULE or RULE-FIRST, and each rewritten target names a concrete event, a threshold, and a time window that a pandas query could compute.
The framing drill — four case memos. Create framing_memos.md. Frame each of these real-shaped asks the way step 1 framed churn: (a) an NBFC asks "reduce loan defaults"; (b) a food app asks "customers are leaving"; (c) a logistics firm asks "too many deliveries are late"; (d) a kirana e-commerce app asks "show customers products they will actually buy". For each, write a six-line memo: real question, target variable (exact definition with threshold and time window), problem type (note that (d) is ranking, not classification or regression), five candidate features knowable at prediction time, success metric with one sentence defending it to a business owner, and the dumb baseline you would compute first. Mark any feature that risks peeking at the future with [LEAK?]. Check: four complete memos; every target definition is computable from data the company plausibly stores; no memo uses plain accuracy for an imbalanced target without a written defence; at least one [LEAK?] flag appears with a reason.
Baseline builder. Create baselines.py. Part A: build a synthetic loan table with np.random.default_rng(21) — 500 rows: loan_id, monthly_income (25,000–150,000), emi (2,000–40,000), months_on_book (1–48). Define the target from a precise rule: defaulted = 1 with probability 0.25 when emi exceeds 35% of monthly_income, else probability 0.04 (use the np.where pattern from step 3). Print the class balance, then compute the majority baseline: its accuracy and how many defaulters it catches. Part B: rebuild the 28-day delivery series from step 5, then compute TWO baselines — predict-yesterday and predict-the-overall-mean — and print both MAE values. Finish with a comment block: which baseline each future model must beat, on which metric, and why accuracy is a trap in part A. Check: running the script twice prints identical numbers (seeds fixed); the printed defaulters-caught count for the majority baseline is 0; the comment block names recall (or a cost-weighted alternative) for part A and the smaller MAE as the bar for part B.
Three failures cover most first-month framing pain. Each has a War Room entry with the full diagnosis path:
- WR-055 — the target nobody could measure. A team shipped "predict customer unhappiness" with no measurable definition; months of work produced a model no one could label, score, or use.
- WR-056 — three weeks of ML matched by a two-line rule. Nobody asked whether rules would do; a hand-written threshold matched the model's performance at a fraction of the cost.
- WR-057 — the wrong-metric celebration. A team optimised accuracy on a 97/3 imbalanced problem, celebrated 97%, and shipped a model that flagged nothing at all.
Machine learning learns patterns from labelled examples, while ordinary software follows rules a person wrote. Ask first whether a plain rule solves the problem, because rules win when the logic is known and stable. Frame the task by naming the problem type and defining the target variable as a measurable event with a time window. Choose the success metric with the business side before modelling, since accuracy can hide total failure on rare classes. Compute a dumb baseline and write its score down, because that is the number every model must beat.
- Indian banks and NBFCs frame default with exact language — ninety days overdue within twelve months of disbursal — because the RBI audits the definition, not the model.
- Food delivery and quick-commerce firms run churn models whose target windows were argued over for weeks; a 30-day window and a 60-day window produce different models and different retention budgets.
- Logistics companies publish delay predictions that must beat the predict-yesterday baseline; a model that loses to a one-line rule never leaves the notebook.
Answer these before opening the gate assessment. Each question names its type.
type=output-prediction— A churn table'svalue_counts()prints 0: 340 and 1: 60. Predict the accuracy of a baseline that predicts 0 for everyone, and how many churners it catches. Then state its recall on the churned class.type=debug— A teammate defines churn as "no order in the next 60 days" but computesdays_since_last_orderusing the full dataset, including those same 60 days. The model scores suspiciously well. Name the framing flaw and the fix in one sentence each.type=interpretation— A framing memo reads: "target: fraud (3% of rows); metric: accuracy; current model: 96.8% accurate". Interpret what that model may actually be doing, and what single line of the memo you would change first.type=scenario-judgement— A manager wants a churn model built this sprint and says the definition of churn "can be sorted out later by the data science team alone". Give your judgement and the one-sentence reason you would offer.type=trade-off— For a churn campaign with a limited voucher budget, compare optimising recall on churners versus precision on churners: give one concrete advantage and one concrete cost of each choice.
Explain to a friend who runs a kirana shop and asks "can AI tell me which customers will stop coming?": what would you need to write down and agree with him before any computer work starts, and how would you know later that the AI is even worth using? Use at most 4 sentences, no ML words beyond "computer". If your friend can repeat the plan back — what counts as stopped, and what dumb guess the AI must beat — you own this module.