Credit-risk / churn predictor + API
Roopa Kulkarni runs credit at a small finance company in Dharwad. Her book is two-wheeler loans for delivery riders and shop owners. One loan in nine goes sixty days past due, and she finds out six months too late. The last analyst left a notebook nobody can explain. She has two years of disbursal records and a rule that every rejection needs a written reason. She hands you the file and one line: build me something the audit team will sign.
- Credit risk is the most common machine learning job in Indian finance. Every bank, finance company and lending app has this exact problem, and they hire for it all year.
- This project makes you prove split discipline on messy data. A leak you found yourself, in your own table, is a story no chatbot can hand you.
- It ends with a running service, a one-page card and a watch plan. That turns one resume line into ten minutes you can hold under questions.
You own the whole pipeline this time. Nobody hands you clean columns, a metric, or a target. Roopa gives you two years of disbursal records and one business question. Which loans will go sixty days past due within six months of the money leaving? Everything after that sentence is your decision, and every decision goes in writing.
One honest note on time. The header says 22 hours, which assumes every prerequisite is still fresh; a realistic first-time solo range is 25 to 35 hours, and planning for that is not falling behind.
Start with the target and the clock. A loan paid out last month cannot have a six-month outcome yet, so your usable rows end well before the file does. Module 1.1 made you write the target as a sentence before writing code. Do that here, in the README, on day one. If the sentence is wrong, nothing downstream can save you.
The data comes from a script this module gives you. It is seeded, so every student in your batch models the same 9,000 loans. It carries the mess a real export carries: blank incomes, a bureau column with text inside it, tenure written two ways, and labels in mixed case. It also carries three traps, and finding them is the work.
The first trap is a column nobody could have known at scoring time. Module 1.7 called this target leakage, and this loan book holds one such field. The cure is a time-of-knowledge audit — a table naming, for every column, the exact moment its value becomes known. Any column that fills in after the outcome is out, however good it makes the score look.
The second trap is the borrower. Many people here took more than one loan, and a few hold two customer records, so a row-by-row split seats the same person on both sides. The cure is a group-aware split — folds cut so that all rows of one borrower stay together. The third trap is the clock: the book changed halfway through the two years, and only a time-ordered split shows the drop.
Here is the contrast this project exists for. One student ships a notebook that scores 0.965 and cannot answer three plain questions about it. Another ships a service that scores 0.71 on loans nobody trained on, and answers all three. The second student passes. The score was never the deliverable.
Your metric is rupees. A bad loan approved costs far more than a good loan turned away, so accuracy is the wrong judge. Build a cost matrix — a small table holding the rupee cost of each kind of mistake — and read your gate off it. Module 1.5 taught the sweep, and this project makes the sweep binding.
Lending is watched, so a rejection needs a reason. Your service must return a reason code — the main driver of one decision, in plain words, with the customer's own numbers in it. Module 1.11 gave you those tools, and you pick them at design time, not after. A model nobody can explain is a model the compliance desk will stop at the door.
Two short documents close the project. A model card — one page saying what the model does, what it learnt from, and where it fails. A decision log — a dated list of what you chose, what you turned down, and why. Both cost minutes, and both are what a reviewer reads before your code.
The defence is the assessment. Module 6.6 set that rule and this project applies it without softening. If you cannot explain your own choices and change your own code while somebody watches, the project scores zero. That holds even when every metric on the page is perfect.
Six artefacts, not a tutorial. Every technique here was taught in Phase 1 or Phase 4; what follows is the standard your work is measured against. Read all six before you write a line of code.
Artefact 1 — the dataset generator. Save this as make_dataset.py, run it once from your repo root, and never edit it. It is seeded, so your table matches every classmate's table, and the receipt it prints at the end goes straight into your README.
"""make_dataset.py - run once, then never edit. Writes data/loanbook.csv.
Two-wheeler loan book of a small NBFC. Target: dpd60_within_6m.
Seeded, so every student in the batch models the identical table.
"""
import numpy as np
import pandas as pd
from pathlib import Path
RNG = np.random.default_rng(20240719)
N_CUST, N_LOANS = 3000, 9000
START = pd.Timestamp("2023-01-01")
BRANCH = ["Dharwad", "Haveri", "Gadag", "Bagalkot", "Vijayapura", "Koppal"]
EMP = ["salaried", "self_employed", "daily_wage", "small_business"]
# ---------- 1. borrowers: the group unit ----------
cust = pd.DataFrame({
"customer_id": [f"C{i:05d}" for i in range(N_CUST)],
"age": RNG.integers(21, 58, N_CUST),
"monthly_income": np.round(RNG.lognormal(9.92, 0.40, N_CUST), -2),
"employment_type": RNG.choice(EMP, N_CUST, p=[.38, .27, .18, .17]),
"existing_emi": np.round(RNG.gamma(1.4, 1600, N_CUST), -1),
"habit": RNG.normal(0, 1.15, N_CUST),
"ntc": RNG.random(N_CUST) < 0.13,
"pan_hash": [f"P{h:06d}" for h in RNG.integers(0, 900000, N_CUST)],
})
cust["bureau_num"] = np.clip(
RNG.normal(714, 58, N_CUST) - 12 * cust["habit"], 300, 890).round()
cust.loc[cust["ntc"], "bureau_num"] = np.nan
# 105 borrowers were re-registered under a SECOND customer_id, same PAN.
pairs = RNG.choice(N_CUST, 210, replace=False)
a, b = pairs[:105], pairs[105:]
for col in ["age", "monthly_income", "employment_type", "existing_emi",
"habit", "ntc", "pan_hash", "bureau_num"]:
cust.loc[b, col] = cust.loc[a, col].to_numpy()
# ---------- 2. loans: repeat borrowing makes groups ----------
idx = np.concatenate([np.arange(N_CUST),
RNG.integers(0, N_CUST, N_LOANS - N_CUST)])
RNG.shuffle(idx)
df = cust.iloc[idx].reset_index(drop=True)
df.insert(0, "loan_id", [f"L{i:06d}" for i in range(N_LOANS)])
day = RNG.integers(0, 730, N_LOANS)
df["disbursal_date"] = (START + pd.to_timedelta(day, unit="D")).strftime("%Y-%m-%d")
df["branch"] = RNG.choice(BRANCH, N_LOANS)
df["dealer_id"] = RNG.integers(1, 61, N_LOANS)
dealer_effect = RNG.normal(0, 0.42, 61)
late = day >= 450 # the EV push, month 15 on
v_early = RNG.choice(["scooter", "motorcycle", "ev_scooter"], N_LOANS, p=[.44, .48, .08])
v_late = RNG.choice(["scooter", "motorcycle", "ev_scooter"], N_LOANS, p=[.31, .34, .35])
df["vehicle_type"] = np.where(late, v_late, v_early)
base_price = np.where(df["vehicle_type"] == "ev_scooter", 118000,
np.where(df["vehicle_type"] == "motorcycle", 96000, 82000))
df["loan_amount"] = np.round(base_price * RNG.uniform(0.62, 0.95, N_LOANS), -2)
df["down_payment_pct"] = np.round(
np.clip(RNG.normal(np.where(late, 14.0, 21.0), 6.0, N_LOANS), 3, 45), 1)
df["tenure_months"] = RNG.choice([12, 18, 24, 30, 36], N_LOANS, p=[.10, .18, .34, .21, .17])
df["interest_rate_pct"] = np.round(
np.clip(RNG.normal(19.5, 3.1, N_LOANS), 12.5, 29.0), 2)
r = df["interest_rate_pct"] / 1200.0
n = df["tenure_months"]
df["emi_amount"] = np.round(df["loan_amount"] * r * (1 + r) ** n / ((1 + r) ** n - 1), 0)
# ---------- 3. the outcome ----------
emi_ratio = (df["emi_amount"] + df["existing_emi"]) / df["monthly_income"]
bureau = df["bureau_num"].fillna(690)
z = (-4.45
+ 3.9 * (emi_ratio - 0.28)
- 0.0080 * (bureau - 700)
+ 1.85 * df["habit"]
+ np.select([df["employment_type"] == "daily_wage",
df["employment_type"] == "self_employed"], [0.85, 0.30], 0.0)
- 0.030 * (df["down_payment_pct"] - 18)
+ 0.50 * df["ntc"].astype(float)
+ 0.40 * (df["vehicle_type"] == "ev_scooter").astype(float)
+ 0.35 * late.astype(float)
+ dealer_effect[df["dealer_id"].to_numpy()])
df["dpd60_within_6m"] = (RNG.random(N_LOANS) < 1 / (1 + np.exp(-z))).astype(int)
# ---------- 4. the field that is filled in AFTER the outcome ----------
df["recovery_agent_assigned"] = np.where(
df["dpd60_within_6m"] == 1, RNG.random(N_LOANS) < 0.93,
RNG.random(N_LOANS) < 0.012).astype(int)
# ---------- 5. the mess a real export carries ----------
df["bureau_score"] = df["bureau_num"].apply(
lambda v: "NTC" if pd.isna(v) else str(int(v)))
miss_inc = RNG.random(N_LOANS) < 0.06
df.loc[miss_inc, "monthly_income"] = np.nan
miss_age = RNG.random(N_LOANS) < 0.02
df["age"] = df["age"].astype(float)
df.loc[miss_age, "age"] = np.nan
as_text = RNG.random(N_LOANS) < 0.15
df["tenure_months"] = np.where(as_text, df["tenure_months"].astype(str) + " months",
df["tenure_months"].astype(str))
df["kyc_status"] = RNG.choice(["Y", "y", "YES", "N", "n", ""], N_LOANS,
p=[.55, .16, .09, .11, .05, .04])
shout = RNG.random(N_LOANS) < 0.08
df["employment_type"] = np.where(shout, df["employment_type"].str.upper(),
df["employment_type"])
COLS = ["loan_id", "customer_id", "pan_hash", "disbursal_date", "branch",
"dealer_id", "vehicle_type", "loan_amount", "down_payment_pct",
"tenure_months", "interest_rate_pct", "emi_amount", "age",
"monthly_income", "employment_type", "existing_emi", "bureau_score",
"kyc_status", "recovery_agent_assigned", "dpd60_within_6m"]
Path("data").mkdir(exist_ok=True)
df[COLS].to_csv("data/loanbook.csv", index=False)
# ---------- 6. the receipt you paste into your README ----------
d = df
print("rows :", len(d))
print("unique customer_id :", d["customer_id"].nunique())
print("unique pan_hash :", d["pan_hash"].nunique())
print("customers with >1 loan :", (d["customer_id"].value_counts() > 1).sum())
print("bad rate overall :", round(d["dpd60_within_6m"].mean(), 4))
print("bad rate day <450 / >=450 :",
round(d.loc[day < 450, "dpd60_within_6m"].mean(), 4),
round(d.loc[day >= 450, "dpd60_within_6m"].mean(), 4))
print("recovery_agent corr target :",
round(d["recovery_agent_assigned"].corr(d["dpd60_within_6m"]), 4))
print("nulls income / age :",
int(d["monthly_income"].isna().sum()), int(d["age"].isna().sum()))
print("bureau_score == 'NTC' :", int((d["bureau_score"] == "NTC").sum()))
print("tenure written as text :", int(d["tenure_months"].str.contains("months").sum()))
On numpy 2.5 and pandas 3.0 the receipt reads: rows 9000, unique customer_id 3000, unique pan_hash 2888, customers with >1 loan 2602, bad rate overall 0.109, bad rate day <450 / >=450 0.0876 0.143, recovery_agent corr target 0.9062, nulls income / age 588 201, bureau_score == 'NTC' 1198, tenure written as text 1420. Six of those ten lines are already a finding. One loan in nine goes bad, so a model that approves everything is right 89.1 percent of the time — that is the number any score of yours must beat. The bad rate jumps from 8.76 percent to 14.3 percent partway through the two years, which is a real change in the book and not noise. recovery_agent_assigned correlates with the target at 0.91, and no honest application-time field on earth does that. And 2,602 of 3,000 borrowers hold more than one loan, while 2,888 distinct PAN hashes cover 3,000 customer ids — so 112 customer ids are the same people twice. Paste the receipt into your README before you touch a model; a reviewer who sees these ten lines knows you read your data.
If your numbers are close but not identical, triage in a fixed order before touching the generator. Every receipt line off by a little points at a library version: run pip show numpy pandas and match the versions named above, because the receipt was produced on those. One line wildly off while the rest agree points at stale state: run make_dataset.py fresh from the top in a new interpreter, never cell by cell, since seeded draws only reproduce in the original order. Only when both checks pass and the receipt still disagrees should you suspect your copy of the script — diff it character by character against the printed one.
Artefact 2 — the acceptance criteria. This is the whole marking scheme for the build stages, and there is nothing hidden behind it. Print it and tick rows.
# P1 ACCEPTANCE CRITERIA. A stage is DONE when every row is true.
STAGE 1 - FRAME, GENERATE, EXPLORE, BASELINE
1.1 README has the target as ONE sentence: event, window, and the
moment of decision. States the usable date range and why.
1.2 make_dataset.py committed and its printed receipt pasted in.
1.3 A data dictionary: 20 rows, one per column, with dtype, null
count, and one line of meaning in business words.
1.4 Base rate stated (0.109) AND the majority-class accuracy
(0.891) named as the number to beat.
1.5 Bad rate BY DISBURSAL MONTH plotted or tabled; the month-15
break is named in one sentence.
1.6 Three baselines scored on the same held-out slice: approve
everyone, reject everyone, and one hand-written rule.
STAGE 2 - FEATURES, SEALED PIPELINE, SPLIT DISCIPLINE
2.1 Cleaning is CODE in src/, not notebook cells: tenure parsed
from two formats, bureau 'NTC' split into a flag plus a
number, kyc case-folded, employment case-folded.
2.2 Every fitted step (imputer, encoder, scaler) lives inside ONE
sklearn Pipeline. No .fit() call touches a test row. Prove it
by showing the Pipeline object, not by saying so.
2.3 time_of_knowledge.md: 20 rows, every column marked KNOWN AT
APPLICATION / KNOWN AFTER DISBURSAL / KNOWN AFTER OUTCOME.
2.4 The leaked column is named, with the evidence that convicted
it, and it is excluded in code, not by hand.
2.5 Three split strategies scored side by side on the same model:
random rows, grouped by customer_id, grouped by pan_hash.
The gap between them is explained in two sentences.
2.6 The shipped split is time-ordered AND group-aware. Test slice
is loans disbursed on or after 2024-07-01.
STAGE 3 - TOURNAMENT, THRESHOLD, EXPLANATION
3.1 At least four candidates scored: majority baseline, logistic
regression, a tree ensemble, and one of your choice.
3.2 Cross-validation is grouped by pan_hash. Report mean AND
spread; a gap inside one standard deviation is a tie.
3.3 Ranking metric is PR-AUC or ROC-AUC, and the choice is
defended in one sentence against the 10.9% base rate.
3.4 cost_matrix.md states the rupee cost of a bad loan approved
and of a good loan refused, with your source for each.
3.5 A threshold sweep table over at least 8 gates, showing counts
and total rupee cost. The shipped gate is the cheapest one.
3.6 Reason codes: for any single loan, the top three drivers in
plain words with the borrower's own numbers.
3.7 The losing models are kept in the write-up with their scores.
STAGE 4 - PRODUCTION
4.1 Repo layout matches artefact 3. Clean clone to first score in
three commands, and the README proves it.
4.2 POST /score takes a pydantic request model, rejects a
negative income and an unknown vehicle type with 422.
4.3 Response carries probability, decision, threshold used, model
version, and the three reason codes.
4.4 GET /health returns model version and load time.
4.5 pytest suite: at least 8 tests, covering one cleaning
function, one feature, the pipeline shape, two API contract
tests, and a QUALITY GATE test that fails the build if
held-out PR-AUC drops below your floor.
4.6 Dockerfile builds; container answers /health; image under
1.5 GB; requirements pinned with ==.
STAGE 5 - GOVERNANCE
5.1 model_card.md follows artefact 4, one page, no gaps.
5.2 monitoring_plan.md names at least four checks, each with a
threshold, a window, an owner and an action.
5.3 The plan states when labels arrive and what you can and
cannot conclude before they do.
5.4 decisions.md has 10+ dated entries, each with the rejected
option and the reason.
STAGE 6 - DEFENCE
6.1 A 90-second walkthrough: problem, approach, result, limit.
6.2 A numbers card answered in under three seconds per line.
6.3 Two live modifications made in code while watched.
6.4 Three limits named before anybody asks for them.
Row 2.2 is where most submissions fail, and the failure is silent: a scaler fitted on the full table before splitting leaks a whisper, while a bureau-mean encoder fitted the same way invents skill that is not there. Row 2.5 exists because the three split strategies give three different numbers on this data, and a student who has seen that gap can never again present a row-split score with a straight face. Row 3.5 is the row that separates a modelling exercise from a lending decision, since the gate is a business choice priced in rupees and not a library default. Row 4.5's quality gate is the one test that makes the suite an ML test suite rather than a Python test suite. Rows 6.1 to 6.4 carry more marks than everything above them, which is the point of this project.
Artefact 3 — the repo you must produce. Not a suggestion. A reviewer opens these paths by name.
nbfc-credit-risk/
README.md what it is, the 3-command run, the receipt,
headline numbers, and the known limits
requirements.txt every line pinned with ==
Dockerfile copies requirements first, then the code
config.py SEED, CUTOFF_DATE, THRESHOLD, COST_FN, COST_FP,
FEATURES, TARGET - every knob, one file
make_dataset.py given to you; run once; never edited
data/ gitignored. loanbook.csv lives here
models/ gitignored. pipeline.joblib + metrics.json
src/
clean.py tenure parser, bureau splitter, case folding
features.py emi_to_income, loan_to_value, tenure buckets
pipeline.py build_pipeline() -> one sealed sklearn Pipeline
split.py time_group_split() -> the only split anybody uses
evaluate.py scores, threshold sweep, rupee cost table
explain.py reason_codes(row) -> three sentences
train.py entry point: fits, scores, writes joblib + json
predict.py entry point: one raw row in, decision out
app/
main.py FastAPI: POST /score, GET /health
schemas.py pydantic request and response models
tests/
test_clean.py the two tenure formats, the NTC rows
test_features.py emi_to_income on a hand-computed row
test_pipeline.py shape, no NaN out, unseen category survives
test_api.py 200 on a good body, 422 on a bad one
test_quality.py THE GATE: held-out PR-AUC >= your floor
monitor/
drift.py PSI per feature against the training window
outcomes.py bad rate by disbursal month, once labels land
docs/
time_of_knowledge.md the 20-row audit
cost_matrix.md the rupee table and where the numbers came from
model_card.md one page
monitoring_plan.md checks, thresholds, owners, actions
decisions.md the dated log
Three things in this tree are load-bearing. config.py exists so that a reviewer who asks "where does the threshold live?" gets one answer, and so that changing it is a one-line change you can make while she watches. src/split.py holds exactly one split function, because a repo with three ways to split is a repo where somebody will pick the wrong one at midnight. tests/test_quality.py is the gate that turns a metric into a contract: if a refactor quietly drops held-out PR-AUC below your floor, the build goes red instead of the model going out. Everything under docs/ is what a reviewer reads first and what most students write last.
Artefact 4 — the model card template. One page. Fill every line or write "not measured" and say why.
# MODEL CARD - <name> v<version> date: <yyyy-mm-dd>
1. WHAT IT DOES
One sentence. Who uses the output, and for what action.
Decision it supports: ................ Not for: ................
2. THE TARGET
Event .............. dpd60_within_6m
Window ............. 6 months from disbursal
Decision moment .... application, before money moves
Usable rows ........ <date range> and why it stops there
3. DATA
Rows / borrowers ... 9,000 loans, 3,000 borrowers
Base rate .......... 0.109
Source ............. generated by make_dataset.py, seed 20240719
Known mess ......... nulls, mixed dtypes, duplicate borrowers
Excluded columns ... <name> - reason: known only after outcome
4. FEATURES
Count and list. For each engineered one, the line of code that
builds it. Anything dropped, and why.
5. HOW IT WAS SPLIT AND SCORED
Split .............. time-ordered at <date>, folds grouped by PAN
CV ................. GroupKFold(5), mean +/- sd
Metric ............. <metric>, chosen because <one line>
Baselines .......... approve-all <x>, reject-all <y>, rule <z>
Result ............. <metric> <value> on the held-out future
6. THE GATE
Threshold .......... <value>
Chosen by .......... cheapest point on the rupee cost table
Cost assumptions ... FN Rs <a>, FP Rs <b>, source: <where>
At this gate ....... recall <r>, precision <p>, <n> bad approved
7. EXPLAINABILITY
Tool ............... <permutation / SHAP / coefficients>
Reason codes ....... three per decision, plain words, customer's
own numbers, reviewed by a non-technical reader
8. LIMITS - at least three, one of them about the data
a. ..... b. ..... c. .....
9. FAIRNESS AND SCOPE
Groups checked ..... bad rate and recall by branch, by employment
type, by new-to-credit flag
Findings ........... <what moved, and what you did about it>
Do not use for ..... segments the training window never saw
10. MONITORING AND RETIREMENT
Watched ............ <checks and thresholds>
Retrain trigger .... <rule>
Label delay ........ 6 months; what you may claim before then
Owner .............. <name> Review date: <yyyy-mm-dd>
Section 2 is the section that saves careers, because a target written down in this shape makes a leaked feature obvious the moment somebody reads section 3's exclusion line beside it. Section 6 is what a credit committee actually argues about; a card that names the threshold without naming the rupee assumptions behind it will be sent back. Section 8 must be filled by you before a reviewer fills it for you, and "the training window has almost no EV loans before month 15" is exactly the kind of limit that belongs there. Section 9 is not decoration in lending: a model whose recall on daily-wage borrowers is half its recall elsewhere is a business problem before it is a technical one.
Artefact 5 — the defence rubric. Forty marks, and the build stages cannot rescue a weak defence.
# P1 GRADING. Total 100.
BUILD - 60 marks, from the acceptance criteria above
Stage 1 frame, data, baselines ................... 10
Stage 2 pipeline, split discipline, the leak ...... 15
Stage 3 tournament, threshold, reason codes ....... 15
Stage 4 repo, API, tests, container ............... 12
Stage 5 card, monitoring plan, decision log ....... 8
DEFENCE - 40 marks, live, 10 minutes, no notes on screen
D1 WALKTHROUGH (6)
Full marks: 90 seconds, four parts in order - problem,
approach, result, limit - ending with an offer to go deeper.
Half marks: right content, over three minutes, no offer.
Zero: a tool list, or a tour of the notebook top to bottom.
D2 NUMBERS (8)
Full marks: base rate, majority accuracy, your metric and
value, the compared model's value, the threshold and its two
rupee inputs - each inside three seconds, no scrolling.
Half: two lookups needed. Zero: cannot state the base rate.
D3 DECISIONS (8)
Full marks: three choices defended with the option rejected
AND the reason, matching dated entries in decisions.md.
Half: reasons given, nothing written down on the day.
Zero: "it is standard" or "the tutorial did it that way".
D4 THE LEAK (6)
Full marks: names the column, says how it was caught, states
the score before and after, and names the general rule.
Half: names it, cannot show the evidence.
Zero: never found it, or found it because somebody said so.
D5 LIMITS (4)
Full marks: three limits volunteered before being asked, one
about the data itself, none of them fatal to the project.
Zero: argues that a real weakness is not a weakness.
D6 LIVE MODIFICATION (8)
Two changes, in your own code, while watched. Examples: move
the threshold to a new cost ratio and re-run the table; drop
the strongest feature and report the new number; add a
required field to the pydantic model and fix the failing test.
Full marks: names the file, the function, the number that
will move and its direction, edits it, runs it, reads the
result, and says whether the prediction was right.
Half: correct reasoning, cannot complete the edit.
Zero: asks for time, or describes the change in general words.
THE OVERRIDE, and it is not negotiable:
A project the student cannot explain and cannot modify scores
ZERO overall, whatever the metrics say and however complete the
repo looks. Build marks are released only after D1 to D6 are
attempted. This is ruling 2 of the course: the defence IS the
assessment, because generated code cannot be detected and an
owner can always be questioned.
Read D6 twice. It is worth more than the container, the tests and the monitoring plan put together, and it is the only row that cannot be prepared the night before. D2 fails more students than D4 does, because everybody remembers their own accuracy and almost nobody remembers the base rate it must be compared against. D3 is why decisions.md is written on the day and not reconstructed in week three: a reason you invent later sounds exactly like a reason you invented later. The override at the bottom is the course's ruling 2 applied without an exception clause, and it is the same rule the interview panel uses in module 6.6.
Artefact 6 — one slice, done to standard. The threshold choice, worked end to end, on the held-out future. This is a bar, not an answer key: your model, features and cost numbers will differ, and the shape of the work must not.
# THRESHOLD FROM THE COST MATRIX. Not from F1, not from 0.5.
STEP 1 - PRICE THE TWO MISTAKES, IN RUPEES, WITH A SOURCE
Bad loan approved (FN): average outstanding at 60+ DPD on this
book is about Rs 46,000; the collections team recovers roughly
60% within a year; so the net loss is about Rs 18,000.
Source: Roopa's collections summary, FY24. Written in
cost_matrix.md, with the assumption flagged as an estimate.
Good loan refused (FP): lost interest margin over the average
tenure, about Rs 2,600 per loan.
Source: finance team's per-loan margin sheet.
Ratio 18,000 : 2,600 = about 7 : 1. That ratio, not the accuracy,
is what the gate must respect.
STEP 2 - SCORE THE HELD-OUT FUTURE, THEN SWEEP
Test slice: 2,300 loans disbursed on or after 2024-07-01.
355 of them went bad. Base rate on this slice: 15.4% - higher
than the 10.9% overall, because the book changed. Say that out
loud before quoting any score, or the number looks wrong.
thr approved bad approved refused good refused cost (Rs)
0.04 745 46 1555 1246 40,67,600
0.06 1051 88 1249 982 41,37,200
0.08 1321 107 979 731 38,26,600
0.10 1500 127 800 572 37,73,200 <-- min
0.12 1662 163 638 446 40,93,600
0.16 1863 205 437 287 44,36,200
0.20 1996 247 304 196 49,55,600
0.30 2150 292 150 87 54,82,200
STEP 3 - COMPARE AGAINST THE TWO THINGS THAT NEED NO MODEL
Approve everyone : 355 x 18,000 = Rs 63,90,000
Refuse everyone : 1,945 x 2,600 = Rs 50,57,000
Model at 0.10 = Rs 37,73,200
Saving against approve-everyone = Rs 26,16,800
A model that cannot beat BOTH rows above has not earned its
place, whatever its ROC-AUC says.
STEP 4 - WRITE THE SENTENCE THAT GOES IN THE DECISION LOG
"12 Aug: gate set at 0.10, the cheapest point on the sweep at a
7:1 cost ratio. Rejected 0.5 (the default) - it costs
Rs 60.7 lakh, only 5% better than approving everyone. Rejected
the best-F1 gate at 0.24 - F1 treats the two mistakes as equal
and this business does not. Recheck this gate if the recovery
rate assumption moves by more than 10 points."
STEP 5 - SAY WHAT THIS DOES NOT PROVE
The 60% recovery rate is an estimate, and the whole gate moves
with it. So the sweep is re-run, not re-derived, when finance
gives a better number - which is why the two costs live in
config.py and not inside a function.
Notice what step 1 does before any code runs: it turns a modelling question into an arithmetic question with a named source, and it writes the estimate down as an estimate. Step 2's warning matters more than the table — quoting 15.4 percent when your README says 10.9 percent looks like an error unless you name the regime change first. Step 3 is the step students skip, and it is the one a credit committee starts with, because a model that loses to "refuse everyone" is a model with a very short meeting ahead of it. Step 4 converts the whole exercise into two sentences a reviewer can check in six months, including the trigger that would make you redo it. Step 5 is the honesty that scores marks in D5: the gate rests on one estimate, and you said so before anybody asked.
Six stages, four graded submissions. Stages 5 and 6 are folded into the last submission because governance and defence are one habit, not two. Work in a repo called nbfc-credit-risk from the first command, never in a loose notebook — exploration may live in notebooks/, but nothing there is graded and nothing there may be imported. Every stage ends with a commit whose message names the stage, and every stage adds entries to docs/decisions.md on the day you decide, not later.
Where a Check states a number that comes from the generator, your reference-seed run must match it exactly; your personal-seed numbers will differ, and you record them beside ours. Where it states a modelling number, a band is given, because your features and your model are your own. Any check you cannot meet is written up in the README as a known gap with your reasoning; an honest gap costs a mark, a hidden one costs the stage.
Stage 1 — frame it, generate it, look at it, beat nothing yet. Create the repo skeleton from artefact 3, with .gitignore covering data/, models/ and .venv/ before your first commit. Save and run make_dataset.py exactly as given, and paste its printed receipt into the README under a heading that says which numpy and pandas versions produced it. Then make the loan book yours — this step is mandatory. Derive a personal seed, and show your working: int(hashlib.sha256("yourname-19990321".encode()).hexdigest()[:8], 16) with your own name and date of birth is one honest way. Change the single default_rng(...) line to that seed — the one permitted edit to make_dataset.py, and the only line you may ever touch in it — rerun the generator, and paste this second receipt beside the first under a heading naming your seed. The reference receipt proves your environment; the personal receipt is the loan book you model from here on. The reason is blunt: seeded identically, every student in the batch would ship the same headline numbers, and two portfolios carrying the same numbers read to a recruiter as tutorial clones. Your numbers must differ from every other student's, and the defence will ask you to explain yours, not ours — so wherever a criterion or a later check quotes 0.109, 0.891 or another generator figure, read it as the reference-seed worked example and state your own beside it. Now write the target sentence: the event, the six-month window, and the moment the decision is taken. State the usable date range and say in one line why loans disbursed in the last six months of the file cannot be used yet. Build docs/data_dictionary.md with one row per column: name, dtype as loaded, null count, and one line of business meaning written for Roopa, not for a data scientist. Then explore: bad rate overall, bad rate by disbursal month, bad rate by vehicle type, by employment type and by new-to-credit status, plus the null pattern and the two tenure formats. Finally score three baselines on the same held-out slice — approve everyone, refuse everyone, and one hand-written rule you invent from the EDA, such as refusing loans where the EMI-to-income ratio crosses a line you pick. Check: the reference-seed receipt matches the generator exactly on all ten lines, including bad rate overall 0.109 and customers with >1 loan 2602, and your personal-seed receipt is pasted beside it under a heading naming your seed and how you derived it; the target sentence names the event, the window and the decision moment, and the usable range excludes the final six months with a stated reason; the data dictionary has 20 rows and every null count matches your file; your month-by-month table shows the break, and one sentence names the disbursal month where your bad rate steps up — ours moved from roughly 8.8 percent to roughly 14.3 percent on the reference seed, and you state your own pair; your own majority-class accuracy is written down as the number to beat, with 0.891 noted as the reference-seed figure; all three baselines are scored on the identical slice with the same metric, and your hand-written rule is stated as a rule anybody could apply without a computer.
Stage 2 — clean it in code, seal the pipeline, and find what is hiding. Move every cleaning step out of the notebook into src/clean.py: parse both tenure formats to an integer, split bureau_score into a numeric column plus an is_new_to_credit flag, fold kyc_status and employment_type to one case, and leave the missing incomes missing so the imputer inside the pipeline handles them. Build features in src/features.py — at minimum an EMI-to-income ratio using the existing EMI too, a loan-to-value proxy from the down payment, and one bucketing of your own with a written reason. Assemble src/pipeline.py so that imputer, encoder, scaler and model sit inside a single sklearn Pipeline; no fitted object may exist outside it. Now the audit: write docs/time_of_knowledge.md with one row per column, each marked known at application, known after disbursal, or known after outcome, and each with the sentence that justifies the mark. Convict the leak with evidence, not with a hunch — fit once with the suspect column and once without, on the same split, and record both scores plus the fitted coefficient or importance that gave it away. Then prove the group problem: score the same model three ways, splitting by random rows, by customer_id, and by pan_hash. Finish src/split.py with one function that returns a time-ordered, group-aware split with the cutoff in config.py. Check: no .fit call appears anywhere outside build_pipeline, and grep -rn "\.fit(" src/ train.py shows only the pipeline being fitted; the time-of-knowledge table has 20 rows and marks recovery_agent_assigned as known after outcome with a one-line reason; your with-and-without experiment shows the leaked model scoring far above the clean one — on the reference seed the leaked model sat above 0.95 ROC-AUC and the clean model landed between 0.66 and 0.76, and you record your own pair — with the drop stated as good news in one sentence; the leaked column is excluded by a list in config.py, not by a deleted notebook cell; the three-way split comparison shows the random-row score highest and the PAN-grouped score lowest, with a clear gap between them (at least 0.04 ROC-AUC on the reference seed; record yours), and your two sentences explain the repeat borrowers and shared PANs using the counts from your own receipt — 2,602 and 112 on the reference seed; time_group_split() returns a test slice containing only loans disbursed on or after the configured cutoff, and a test in tests/ asserts that no PAN appears on both sides.
Stage 3 — run the tournament, price the gate, write the reasons. Score at least four candidates with GroupKFold grouped by pan_hash: the majority baseline, a logistic regression with balanced class weights, a gradient-boosted tree ensemble, and one more of your choosing. Report mean and standard deviation for every candidate in one table, and apply module 1.8's tie-break rule — when two models sit inside one standard deviation of each other, the simpler and more explainable one wins, and you write that sentence down. Fix your ranking metric first and defend it in one line against your own base rate — 10.9 percent on the reference seed, whatever your receipt says on yours. Then build docs/cost_matrix.md: the rupee cost of a bad loan approved, the rupee cost of a good loan refused, and a named source or a clearly flagged estimate for each. Sweep the threshold across at least eight gates on the held-out future, printing approvals, bad approvals, refusals, good refusals and total rupee cost, and compare the winner against approve-everyone and refuse-everyone. Ship the cheapest gate and write the decision-log entry in the shape of artefact 6, step 4. Last, build src/explain.py: a global view of what the model relies on overall, and a local view that turns one loan into three reason codes in plain words with the borrower's own numbers in them. Check: the tournament table shows four or more candidates with mean and spread from grouped cross-validation, and no candidate is scored on a row-wise split; your chosen metric is defended in one sentence that mentions the base rate; the tie-break sentence exists and names the model you rejected together with its score; cost_matrix.md gives two rupee figures with a source or an explicit estimate flag for each; the sweep table has at least eight rows and its cheapest gate is the one in config.py, and both no-model comparisons appear beneath it; the shipped gate beats both of them in rupees; three reason codes for one sample loan are written in a sentence a branch clerk could read aloud, each naming the borrower's own value and the level at which the decision would change; and a non-technical friend reads all three back to you and can say what the customer should fix.
Stage 4, 5 and 6 — ship it, govern it, then defend it. Finish the repo against artefact 3, with pinned requirements and a README whose three commands take a stranger from a clean clone to a score. Build the service: POST /score with a pydantic request model that rejects a negative income and an unknown vehicle type, and a response carrying probability, decision, the threshold used, the model version and three reason codes. Add GET /health returning the model version and load time. Write at least eight tests, including two API contract tests and a quality gate that fails when held-out PR-AUC falls below a floor you set and justify. Containerise it, and prove the container answers /health. Then govern it: fill docs/model_card.md from artefact 4 with no blank lines, write docs/monitoring_plan.md with at least four checks — each carrying a threshold, a window, an owner and an action — and make sure the plan states that labels arrive six months late and what may be claimed before they do. Bring docs/decisions.md to at least ten dated entries. Finally rehearse the defence against artefact 5: record yourself giving the 90-second walkthrough, have a classmate probe you for ten minutes using module 6.6's depth ladder, and take two live modifications with a timer running. Check: a classmate clones your repo on a fresh machine, follows the README, and gets a score without asking you a single question; POST /score returns 422 for a negative income and for an unknown vehicle type, and 200 with all five response fields for a valid body; pytest passes with at least eight tests, and you prove the quality gate works by breaking the model on purpose once — shuffle the labels, watch the build go red, undo it, and paste that red output into the README; the container builds and answers /health; the model card has every section filled, with at least three limits and one about the data itself; the monitoring plan's four checks each name a threshold, a window, an owner and an action, and one of them watches the input distribution while another watches the realised bad rate; decisions.md has ten or more dated entries, each with the rejected option and the reason; your recorded walkthrough is under 110 seconds and ends with an offer to go deeper; and for both live modifications you named the file, the function and the direction of the number before editing, then reported in one sentence whether your prediction was right.
Working alone? Every classmate step in this course has a solo substitute, the substitutes are graded identically, and these definitions hold for P2 through P5 and modules 6.4 and 6.6, which point back here. The fresh-machine test: clone your repo into a clean container (docker run -it --rm python:3.12 bash) or a brand-new OS user account and follow your own README verbatim, because the place where you stop is exactly where a stranger would have stopped. The probe or viva: record yourself answering the module's viva-bank questions cold, no notes, then listen back against the model answers and score the recording with the same rubric a partner would use. The defence: a 20-minute recorded self-defence against this project's own rubric — walkthrough, numbers, decisions, limits and both live modifications — scored honestly line by line, with a written reason for every mark you refuse yourself. The only thing these substitutes cannot supply is someone else's mercy, and mercy was never part of the marking. Score yourself the way the rubric reads, not the way you hope, and the artefacts you submit are the same ones a paired student submits.
Three failures that end credit-risk projects, all of them from the stages above. Each has a War Room entry with the full diagnosis path:
- WR-175 — the field that knew the answer. A default model wore 0.96 ROC-AUC through two reviews because one column is written only after a loan has already gone bad; the time-of-knowledge audit is the cheap habit that catches it, and the honest score after the fix is the one that ships.
- WR-176 — the rejection nobody could explain. A lending model was blocked by compliance days before launch because no decision came with a reason a customer could be told; the fix is choosing explainability tools at design time, and the cost of bolting them on afterwards is measured in weeks.
- WR-177 — the borrower on both sides. Repeat borrowers and duplicate customer records put the same person in train and test, inflating the score by a wide margin; splitting folds by a stable person key, and checking for duplicate identities first, restores the real number.
This project hands you a fresh problem, a messy generated loan book, and the standard your work is judged by. You own every step, from the target sentence to a sealed pipeline, a group-aware split, a gate priced in rupees, a service and a model card. Three traps are planted for you: a column nobody could know at scoring time, a borrower on both sides of the split, and a book that changed halfway through. Find them, prove them, and write down what you decided on the day you decided it. The defence is the assessment, so a project you cannot explain and cannot change while somebody watches scores zero.
- NBFC and bank credit teams across India run exactly this loop for two-wheeler, gold and small-business loans. A model card, a cost table and a reason code for every rejection are ordinary paperwork there, not extras.
- Lending apps and fintech risk teams live on the gap between a random split and a time-ordered one, because borrower behaviour and their own approval rules move every quarter. Their monitoring watches the input mix daily, months before any real label arrives.
- Analytics vendors serving banks hire freshers straight into this work. The interview is usually one of your own projects opened on screen, with the panel asking for a change while you type.
Answer these aloud before opening the gate assessment. Each question names its kind.
type=interpretation— A submission reports 0.96 ROC-AUC on a random split and 0.71 on a time-ordered split grouped by borrower, and the student calls the second number a bug. Interpret both numbers, say which one belongs in the model card, and name the two separate mistakes that produced the gap.type=scenario-judgement— Two days before your defence you find that the leaked column was in your feature list all along, and removing it drops your headline score by a quarter. Give your ruling on what you do next, the first sentence of your walkthrough, and justify it against what the rubric rewards.type=trade-off— Compare shipping the threshold that maximises F1 against shipping the cheapest point on the rupee cost table. Give one concrete gain and one concrete cost of each, and name the single situation that would make F1 the right choice here.type=debug— A classmate's cross-validation scores are stable and healthy, but the score on the held-out future is far lower every single time, and the gap never closes. Diagnose the likely cause from that pattern alone, name the two files you would open first, and give the one experiment that settles it.type=output-prediction— A student moves the gate from 0.10 to 0.30 on the held-out slice because "precision looks better there". Predict what happens to the count of bad loans approved, the count of good loans refused, and the total rupee cost, and say which of the three the credit committee will ask about first.
Explain to a friend studying commerce why a lender cannot ship the model with the best score. Use an everyday comparison, such as a shopkeeper who must say why he refused credit to a regular customer, and who would lose the shop if he could not say it. Cover three things: that one column in the file was written after the loan already went bad, that turning away a good customer and lending to a bad one do not cost the same money, and that every refusal must come with a reason the customer can act on. Use at most 4 sentences and no word your friend would need to look up.