A model finally ships after weeks of feature work and tuning — and someone notices a trivial hand-written rule scores the same, or nearly the same, on the agreed metric. Leadership asks what the weeks were for. Sometimes it surfaces earlier as a suspicion: the model's top feature alone seems to make all the decisions.
The team went straight from business ask to model without asking whether a rule would do. Typical asks where this bites: flagging transfers above a threshold at odd hours, blocking orders where delivery and billing PIN codes differ, chasing invoices overdue beyond a fixed number of days. Common in teams eager to "do AI" and in companies where ML work is prestige work. Linked module: 1.1 (Framing an ML problem).
The rules-versus-ML question was never asked, so nobody noticed the logic was known, stable, and small — the exact conditions where rules win. Confirm it in an hour: ask the domain expert "how would YOU decide, by hand?" and write their answer as one or two if-conditions in pandas. Score that rule on the same data and metric as the model. If the gap is small, the model is an expensive rule. Check the model's feature importances too: one dominant feature that mirrors the expert's threshold is the same finding from the other side. The failure is not that the model is bad — it is that its cost (weeks, complexity, retraining, monitoring) buys nothing a rule does not already deliver.
Ship the rule, not the model, and say so plainly — a two-line rule that matches the model is the better engineering choice: cheaper to run, trivial to explain to auditors, impossible to drift silently. Keep the model work as a documented experiment with its score recorded next to the rule's score. Revisit ML only when the rule measurably breaks: the metric decays, fraudsters adapt to the threshold, or new signals arrive that no hand-written condition can combine. That evidence, not prestige, is the trigger for round two.
Make "do we need a model at all?" the first line of every framing sheet, answered before any code. Always build the rules baseline first — the domain expert's hand logic in a few lines of pandas — and record its score as the number any model must beat by a margin worth the added cost. Put that margin in the project plan ("model ships only if it beats the rule by X on metric Y"). Asking the question first is not anti-ML; it is what makes the ML projects that survive it credible.
Q: "When would you NOT use machine learning for a prediction problem?" Strong answer: when the logic is known, stable, and expressible in a few conditions — give a concrete example like a night-time transfer threshold, and describe building the rule as a baseline and requiring any model to beat it by a margin that pays for its complexity. Bonus points for naming the hidden costs of a model (retraining, monitoring, explainability) that a rule avoids. Interviewers use this to separate tool-users from engineers.