A classifier is live and its probabilities look healthy, but the business result is poor: a churn campaign keeps missing churners, or a fraud queue stays nearly empty while losses continue. Precision looks flattering; recall is quietly terrible (or the other way round). Nobody remembers deciding the threshold — because nobody did.
Any deployment that calls model.predict() on a probabilistic classifier —
scikit-learn's predict silently applies a 0.5 cutoff to predict_proba. Most common
in first churn, fraud, and lead-scoring launches where the notebook's default flowed
straight into production. The costs of the two mistake types (false alarm versus miss)
were never written down, let alone compared. Linked module: 1.3 (Linear & logistic
regression); threshold tooling deepens in 1.5.
The model outputs a probability; turning it into an action needs a decision threshold,
and 0.5 is only the library's default, not a business decision. When a missed churner
costs ₹500 of future margin and a wasted voucher costs ₹50, the two mistakes are 10x
apart in cost — a symmetric cutoff cannot be right. Confirm in three steps.
(1) Get predict_proba scores on a held-out set. (2) Sweep cutoffs (0.05 to 0.95) and
tabulate flagged count, caught, missed. (3) Price each row with the real costs: total =
cost_per_flag × flagged + cost_per_miss × missed. If the cost curve's minimum sits far
from 0.5 — it usually does — the default was burning money. Also check who is consuming
predict() output directly; that call hard-codes the blind cutoff.
Get the two costs from the business owner in rupees, even as rough estimates. Choose the
cutoff that minimises expected cost on held-out data, and ship code that thresholds
predict_proba explicitly instead of calling predict. Record the chosen cutoff, the
cost assumptions, and the sweep table next to the model artefact so the decision is
auditable. Re-run the sweep when costs or class balance change — a festival-season
voucher budget or a new fraud pattern moves the optimal gate.
Treat "what does each kind of mistake cost?" as a framing-sheet question (module 1.1),
answered before training starts. Ban bare predict() for business-facing classifiers in
code review; require an explicit named threshold constant with a comment stating the
cost ratio behind it. Put the threshold sweep in every model review deck — one small
table makes a blind default impossible to miss.
Q: "Your churn model has good AUC but the campaign is missing most churners. The team
says the model needs retraining. What do you check first?" Strong answer: separate the
score quality from the decision rule — check the threshold before touching the model,
sweep cutoffs against the voucher-versus-miss costs, and show the current 0.5 was never
chosen. Bonus signal: mention that predict() hides the 0.5 and that production should
threshold predict_proba explicitly.