A training script that ran fine on a sample dies on the full table: the laptop
freezes and the OS kills Python, Colab restarts the runtime, or the CI job ends with
MemoryError (or an out-of-memory kill with no traceback at all). The change that
triggered it looks innocent — someone added a categorical column to the encoder list.
The column was a pincode, a branch code, or a customer id with thousands of distinct
values.
One-hot encoding was the team's habit for every categorical column, learned on five-city examples where it is exactly right. Then the habit met a high-cardinality column: 3,000 pincodes across a delivery dataset of a few lakh rows. Dense one-hot output turns every row into 3,000 numbers — a few lakh rows times 3,000 columns times 8 bytes is tens of gigabytes. Common in logistics, payments, and retail tables, where ids and location codes are everywhere. Linked module: 1.6 (Feature engineering & encoding).
The memory cost of one-hot is rows times distinct categories — it grows with
cardinality, not with the amount of information in the column. Do the arithmetic
before believing anything else: rows * nunique * 8 / 1e9 gives dense gigabytes;
print df[col].nunique() for every categorical column. There is a second, quieter
cost visible on the score: most of those 3,000 columns are almost entirely zeros, so
rare categories get one noisy weight each, and the model overfits the tail. If the
job died right after an encoder change and the column's nunique() is in the
hundreds or thousands, you are holding this incident.
First, breathe room back: keep the encoder's sparse output (do not pass
sparse_output=False) if the downstream model accepts sparse matrices — that alone
often revives the job. Then encode the column to match its cardinality: frequency
encoding replaces the category with how often it appears (counts learned on training
rows, inside the pipeline); grouping keeps the top categories and folds the tail into
one OTHER bucket; per-category target statistics are stronger still but leak-prone —
adopt them only with module 1.7's discipline. Re-run and compare scores honestly: a
one-column encoding often matches the 3,000-column monster.
A cardinality audit before any encoder is written: one table with column name,
nunique(), and the planned encoding, reviewed like code. House thresholds make the
decision boring — one-hot below a few dozen categories, frequency or grouped
encodings above, ids never encoded at all. Estimate the matrix size in gigabytes in
the audit, on paper, before the first fit. The audit takes five minutes; the OOM hunt
takes an afternoon.
Q: "Training crashes with MemoryError after a colleague one-hot encodes pincode.
Diagnose and fix." Strong answer: state the rows-times-categories arithmetic and ask
for nunique() first, name sparse output as the immediate relief, then propose
frequency encoding or top-K-plus-OTHER for the lasting fix, with target encoding
mentioned as powerful but leakage-prone. Bonus signal: pointing out the overfitting
cost of thousands of near-empty columns, not only the memory cost.