A hyperparameter search was launched in the evening — "it should be done by morning" — and the next day the progress readout shows single-digit percent. The laptop fan has been at full speed all night, the machine is unusable for anything else, and the projected finish time is measured in days. Sometimes the run dies before that: the OS kills the process, the notebook kernel restarts, or a battery-saver sleep silently pauses hour twelve.
A full GridSearchCV over a boosted model with five or six tuned knobs, each
given three to five values, on an ordinary 8 GB laptop with no GPU and default
n_jobs. Nobody multiplied before pressing enter: six knobs at four values each
is 4,096 combinations, times 5 CV folds is 20,480 fits — at even 20 seconds a
fit, that is more than four days of compute. Common right before demos, when
"let's tune everything at once" feels like thoroughness. Linked module: 1.8
(CV, tuning, model selection).
Grid search cost is a multiplication, and multiplications explode: values per
knob multiply across knobs, then multiply by folds, then by seconds per fit.
Run the arithmetic on the stalled search — len(param_grid combinations) x n_folds x seconds_per_fit — and the mystery disappears. Time one single fit
first if you do not know the last number. The deeper diagnosis is strategic:
on most tables only one or two knobs move the score at all, so an exhaustive
sweep spends days proving that most of the grid never mattered. The 1.8 demo
measured it: a 48-combination grid and a 12-sample random search found the same
best score, 240 fits against 60.
Kill the run — a search that cannot finish is not producing knowledge. Replace
it with RandomizedSearchCV and an explicit budget (n_iter=20 to 50) over the
same space, with n_jobs=-1 so every CPU core works. Go coarse-to-fine: a first
cheap search with wide, few-valued ranges finds the neighbourhood; a second
small search zooms into it. Print the fit count before every launch — number of
combinations times folds — and time one fit to project the total. If the honest
projection still exceeds the laptop's patience, cut the space or move the run to
a free Kaggle/Colab session, per the course's compute doctrine.
House rule: no search launches without its fit count and a one-fit timing printed
first — a two-line habit that prevents every multi-day surprise. Default to
random search for anything beyond two or three knobs; grid is for small, cheap,
final confirmations. Tune the knobs known to matter for the model family first
(for boosted trees: learning rate and depth) instead of everything at once, and
keep n_jobs=-1 in the team's search template so parallelism is never forgotten.
Q: "Your grid search will take four days on your machine. Walk me through your
options." Strong answer: show the cost arithmetic first (combinations × folds ×
seconds per fit), then rank the levers — switch to RandomizedSearchCV with a
budget, coarse-to-fine ranges, n_jobs=-1, fewer knobs by tuning what matters
for the model family, and moving to free cloud compute if the projection still
does not fit. Quoting the evidence that random search matches grid on most
spaces, at a fraction of the fits, turns a rescue story into an engineering
decision.