A model that scored well in training behaves strangely in production — churn probabilities look compressed, a retention campaign targets loyal customers, a risk score waves through applicants it should flag. Nothing crashes. No error, no warning, no log line. The model file is identical to the one that was validated. Predictions are silently, consistently wrong, sometimes for weeks before anyone notices.
The training notebook prepared features by hand: scaled income with the training
mean, built a ratio column, mapped cities to one-hot columns with pd.get_dummies.
The scoring script — often written later, sometimes by a different person —
re-implemented "the same" transforms from memory. It scaled with the serving batch's
own mean and standard deviation, and get_dummies rebuilt the city columns from
whatever cities happened to be in that batch, in whatever order. Common wherever the
train path and the serve path are two separate scripts. Linked module: 1.6 (Feature
engineering & encoding).
A transform is not only code — it carries learned state. The scaler's mean, the
encoder's category list, the frequency map's counts were all learned from training
rows. Recomputing them on serving data produces different numbers for the very same
customer: scaled income shifts because today's batch is richer, city columns land in
a different order, and the model reads every input slightly wrong. Confirm with a
parity test: take one row, push it through the training path and the scoring path,
and print both transformed vectors side by side. If any cell differs, you are holding
this incident. pd.get_dummies in a serve path is a near-certain tell — it cannot
remember the training categories.
Delete the hand-written transforms from the scoring script entirely. Rebuild the
preparation as a sklearn Pipeline — feature function, ColumnTransformer with the
encoders and scaler, then the model — fit it once on training rows, and save the
whole fitted object with joblib. The scoring script now loads that one file and calls
predict on the raw row; there is no second copy of any transform left to drift.
Re-score the affected period if decisions were made on the wrong numbers, and tell
the stakeholders — silent wrongness that was found must not stay silent.
House rule: any transform that contains a learned number — a mean, a category list, a
count table, bin edges — lives inside the fitted Pipeline and nowhere else. The serve
path may only load and call, never re-implement. Add a golden-row test to the repo: a
few fixed raw rows with their expected predictions, run after every change on both
paths. Review any pull request that imports get_dummies anywhere near serving code.
Q: "Your model is correct in the notebook but gives different predictions in the
scoring service. Same model file. Walk me through it." Strong answer: name learned
transform state as the suspect in one sentence, propose the one-row parity test as
the confirming artefact, and give the fix — one fitted Pipeline serialized and used
by both paths. Mentioning that get_dummies cannot remember training categories
signals real production experience.