The scoring service dies in the middle of the night with
ValueError: Found unknown categories ['Moga'] in column 0 during transform — or, in
the quieter version, it does not die at all: rows from a new city sail through with
every city column set to zero and nobody is told. Training never showed this error.
The crash (or the silence) begins the day production data contains a category value
the encoder was never fitted on.
A OneHotEncoder (or any category mapping) was fitted on training data that contained one set of values — five cities, three hundred branch codes, last quarter's merchant list. Then the business did what businesses do: opened a branch in a new city, onboarded new merchants, launched in new pincodes. The first request carrying the new value reaches the deployed pipeline weeks after launch, which is why this incident so often pages someone at 2 am rather than failing a test. Linked module: 1.6 (Feature engineering & encoding).
An encoder's category list is learned state, frozen at fit time. sklearn's default,
handle_unknown='error', refuses values outside that list — that is the crash. The
alternative, handle_unknown='ignore', encodes an unseen value as all zeros — that
is the silence: predictions continue, with the category's information blanked out.
Neither default tells the team the world has changed. Confirm by comparing the
encoder's categories_ attribute with the distinct values now arriving in
production: any value in production but not in categories_ reproduces the incident.
The real fault is treating the category list as permanent when the business
guarantees it will grow.
Immediate: set handle_unknown='ignore' on the encoder inside the pipeline, refit,
redeploy — the service must not crash because a new branch opened. Same day: add
monitoring, because ignore-mode converts a loud failure into a silent one. Log every
incoming category value, count the ones missing from categories_, and alert when
the unseen share crosses a small threshold. Then schedule a retrain that includes the
new categories, so "unseen" is a short-lived state, not a permanent information hole.
Every categorical column in a deployed pipeline gets three things: handle_unknown
set deliberately (never left to the default by accident), an unseen-category counter
in monitoring, and a written answer to "what happens when the business adds a new
one?". Test it before launch: send one made-up city through the deployed pipeline in
staging and check it neither crashes nor passes unlogged. New-category rate rising is
a retrain trigger, the same way score drift is.
Q: "Your churn service crashed at night with an unknown-category error. What
happened, and what do you change?" Strong answer: explain that the encoder's category
list froze at fit time while the business kept growing, give both failure modes —
error mode crashes, ignore mode goes silently blind — and fix all three layers:
handle_unknown='ignore' for availability, an unseen-category alert for visibility,
and a retrain cadence for correctness. Candidates who mention the silent all-zeros
mode without being prompted stand out.