The model is accurate offline and passes every metric review, but the scoring service built around it is slow: single predictions take tens of milliseconds or more, p95 latency blows the API budget, upstream callers time out, and the service falls over under modest traffic. Sometimes the first sign is memory: the saved model file is hundreds of megabytes and the container OOMs on deploy.
A RandomForestClassifier (or similar ensemble) tuned for accuracy alone — someone
raised n_estimators to 1,000-2,000 with unlimited depth because "more trees never
hurt" — now serving real-time requests: loan decisions at application time, fraud
checks inside a payment flow, churn scores on page load. All evaluation before shipping
was batch and offline; nobody timed a single-row predict. Linked module: 1.4 (Trees,
RF, boosting, XGBoost); serving and monitoring return in Phase 4.
Prediction cost for a forest grows roughly linearly with tree count, and each unlimited
tree is large, so 2,000 deep trees mean thousands of pointer-chasing walks per row and
a model too big for cache. Meanwhile accuracy plateaus: plot test score against
n_estimators (5, 50, 200, 1000) and the curve is typically flat within ~0.03 after a
few hundred trees — in the lesson's sweep, score stayed flat while one-row latency grew
more than tenfold. Confirm with two measurements on the production-shaped input: time
predict on ONE row (averaged over 20+ calls, since real traffic rarely arrives in
convenient batches), and check the model's on-disk size. If the one-row time exceeds
the service budget, no infrastructure tuning will save a model that does too much work
per request.
Find the knee of the accuracy-versus-cost curve and ship that. Sweep n_estimators
and max_depth, recording test score AND one-row latency per configuration; pick the
smallest model within your score tolerance (often 100-300 capped-depth trees). Or
switch family: HistGradientBoostingClassifier usually matches or beats a huge forest
with far fewer, shallower trees and predicts in a fraction of the time — the lesson's
sweep had it at roughly the latency of a 50-tree forest. Re-measure one-row latency
after the change, on hardware comparable to production, before calling it fixed.
Make measured one-row latency (and model size on disk) part of the shipping checklist, next to accuracy — a model review that shows only scores is half a review. Present an accuracy-versus-cost table, not a single best-score number, whenever tree count or depth is tuned. Set the latency budget with the service owner BEFORE tuning starts, so "more trees" has a price tag from day one.
Q: "Your random forest is accurate but the API built on it is too slow. Walk me through your response." Strong answer: explain prediction cost scales with tree count and depth, propose the score-versus-latency sweep to find the plateau, name gradient boosting (HGB/LightGBM) as the usual smaller-faster replacement, and insist on measuring single-row predict time on production-shaped hardware. Bonus signals: mentioning model file size/memory, and that batch offline evaluation hides per-request cost — that detail marks someone who has actually shipped a scorer.