The model is trained, the service is containerised, the tests are green, and launch is five working days away. Compliance asks for one thing before sign-off: a sample rejection letter. The team discovers that the API returns a probability and a decision, and nothing else. There is no sentence anybody can post to a customer, and no way to produce one without rebuilding how the model was chosen.
POST /score
{ "loan_amount": 78000, "monthly_income": 24000, "existing_emi": 4100, ... }
200 OK
{ "probability": 0.71, "decision": "refer" }
COMPLIANCE REVIEW - BLOCKED
Finding : decision returned with no reason attributable to the applicant
Ask : main reasons per declined application, in plain language
Status : launch held
Any lending decision that touches a real applicant: two-wheeler and personal loans at an NBFC, card limits at a bank, buy-now-pay-later approvals at a fintech. Indian lenders operate under fair-practice expectations that a declined applicant is told the main reason, and internal audit treats an unexplainable automated decision as a control gap rather than a modelling detail. The failure is a scheduling failure, not a knowledge failure — the team knew about explainability tools and planned to add them "after the model works". Linked modules: P1 (Credit-risk predictor + API), 1.11 (Interpreting models), 4.2 (Serving with FastAPI), 3.11 (Safety, guardrails, DPDP).
Explainability is a design constraint, not a reporting feature. Treated as reporting, it arrives after the model is chosen and the service is written, and by then three decisions have already been made against it.
Decision one: the model. A large stacked ensemble bought two points of ROC-AUC and made every local explanation a separate approximation exercise. When the requirement is known at design time, 1.8's tie-break rule points the other way: inside one standard deviation, the readable model wins, and in lending "readable" is worth real money.
Decision two: the features. Reasons must be things the applicant can act on. A
feature like dealer_risk_score may predict well, and no customer can do anything
about which showroom they walked into. A reason code built on it is either useless or
offensive. Feature selection in a regulated model carries a second filter beyond
predictive power: can this be said to the person?
Decision three: the response contract. The pydantic response model was written with two fields. Adding reason codes now changes the API contract, the tests, the caller in the branch software and the screen the clerk reads. That is a week, and it is a week spent under a launch date.
There is a fourth failure hiding behind the first three, and it is the one that costs most. Raw explainer output is not an explanation. A line reading "SHAP -0.42 on emi_to_income" is true and unusable. The deliverable is a sentence with the applicant's own number in it and the level at which the answer would change.
Step 1 — write one sample letter before choosing the model. Draft the sentence the branch will post, and work backwards from it. It fixes the model class, the feature filter and the response shape in one afternoon.
WRONG - the explainer's output, shipped raw
"Declined. SHAP value -0.42 on feature emi_to_income;
-0.19 on bureau_num; -0.11 on down_payment_pct."
RIGHT - the same three drivers, as an adverse action sentence
"We could not approve this loan today. The three main reasons:
1. Your monthly instalments would come to Rs 9,400 against a
monthly income of Rs 24,000, which is 39%. We approve when
this stays under about 30%.
2. Your credit bureau score is 648. We look for 690 or above
on this product.
3. Your down payment is 8% of the vehicle price. Raising it to
20% would lower the instalment enough to change reason 1.
You may reapply with a longer tenure or a larger down payment."
Step 2 — make the reason codes part of the response contract from the first commit.
POST /score returns probability, decision, threshold used, model version and three
reason codes. Write the contract test on day one so the field cannot quietly disappear.
Step 3 — choose the tool at design time and keep it in the repo. Coefficients for a
logistic model, or SHAP for a tree ensemble, wrapped in one function — reason_codes(row)
in src/explain.py — that returns three sentences, not three numbers. One function, one
place to fix, one thing to demonstrate in a review.
Step 4 — run the read-aloud test. A colleague with no technical background reads the three reasons and says what the customer should do next. If they cannot, the sentences fail, whatever the explainer says. This test is free and it catches almost everything.
Step 5 — record the trade you made. If a readable model cost you a point of ROC-AUC, that is a decision-log entry with the number in it, not a quiet compromise. It is also the strongest thing you will say in a defence.
Ask "who has to be told why, and in what words?" in the first framing conversation, beside the target definition — in lending, insurance, hiring and admissions the answer is never "nobody". Put the sample letter in the repo before the first model run and treat it as an acceptance test. Filter features twice in a regulated model: once for predictive power and once for whether the applicant can act on them. Fix the API response contract early, including the reason codes, because a contract change late is a multi-team change. And review the sentences with a non-technical reader every time the model is retrained, since new features change what the letters say.
Q: "Your model must reject loan applications and the business must tell each applicant why. How does that change your design?" Strong answer: name it as a design constraint and walk the three places it lands — the model class, where the tie-break now favours readability; the feature list, where a driver the applicant cannot act on is a poor reason even when it predicts; and the response contract, which carries reason codes from the first version. Then show you know the difference between explainer output and an explanation by giving the sentence with a number and an actionable level in it. Weak answer: naming SHAP and stopping there, which answers a tooling question nobody asked.