A model saved with joblib weeks ago now produces nonsense: churn probabilities stuck
near one value, predictions that contradict obvious cases, or a crash on load or
predict (KeyError on a column, a dtype complaint, or an sklearn version warning
about unpickling). The old notebook still shows respectable scores, so the model file
gets blamed as "corrupted" — but re-saving it changes nothing.
The gap between saving a model and using it for real: a sprint-one model called by a
sprint-four script, a teammate consuming a .joblib file they did not train, or a
scheduled job scoring fresh exports. Meanwhile the upstream data moved — a column was
renamed or reordered, a new field appeared, an ID or amount column changed dtype, or
units changed (rupees to paise, days to hours). Sometimes the environment moved
instead: a different scikit-learn version loads the pickle. Linked module: 1.2
(Full ML loop once, end to end).
A saved model is frozen: it stores patterns keyed to the exact columns, order,
dtypes, and value ranges that existed at training time, and it applies them blindly
to whatever arrives. It cannot notice drift — predict always returns a number if the
shapes let it. Confirm in three comparisons. (1) Columns: diff the feature list used
at training against the incoming table's columns and order; any rename, drop, or
reshuffle is your cause. (2) Dtypes and units: compare df.dtypes and simple
describe() ranges between old training data and today's input — a numeric column
arriving as strings, or values a thousand times larger, means the patterns no longer
apply. (3) Environment: compare the sklearn version that saved the file with the one
loading it. If nobody can produce the training-time feature list, that absence is
itself the root cause: the model was saved without its schema.
Stop trusting the outputs immediately — confidently wrong predictions cost more than a loud crash. Rebuild the bundle so the model never travels alone: save one dict with the model, the ordered feature list, the expected dtypes, rough valid ranges, and the training date plus sklearn version. At load time, validate before predicting: reindex the incoming row with the saved feature list, fail loudly on missing or extra columns, coerce or reject wrong dtypes, and refuse values far outside training ranges. If the world truly changed — new columns, new units, new customer behaviour — no validation saves the old patterns: retrain on current data and save a fresh bundle.
Make joblib.dump({"model": ..., "features": ..., "dtypes": ..., "trained": ...})
the only save pattern you use; a bare joblib.dump(model, path) is a future incident.
Validate inputs on every load, not only in debugging weeks. Pin the sklearn version
alongside the artifact and reload the bundle in a fresh session as part of finishing
any training run. Record a few known-answer rows with the model — if next month's
load predicts them differently, the environment or data has moved and the model must
not be trusted silently.
Q: "A model you saved last month now returns garbage in the scoring job. Walk me through your investigation." Strong answer: state that a pickled model applies frozen patterns blindly, then check the three drift surfaces in order — column names/order, dtypes and units, library version — by diffing today's input against the saved schema. Strong candidates add the prevention: always persist the feature schema with the model and validate on load, so drift fails loudly instead of predicting nonsense. Blaming a "corrupted file" first is the weak answer interviewers are screening out.