Every call to /predict takes tens of seconds — or hundreds of milliseconds for a
small model — no matter how many requests have already been served. The fiftieth
request is as slow as the first. /health answers instantly, so the server itself
is fine. Under a little concurrent load the service gets dramatically worse:
requests queue, memory spikes, and sometimes the process is killed by the OS.
Callers set 10-second timeouts and report the API as "down" even though every
request eventually succeeds.
A first serving job: a trained model (a sklearn pipeline, a sentence-transformer, an XGBoost ensemble) wrapped in FastAPI and shipped. Tutorials often load the model at the top of a function because it makes a self-contained example, and the pattern gets copied into the endpoint. On the laptop with one test request the service "works", so nothing looks wrong until a second caller arrives or the model grows. A 1 MB churn pipeline hides the mistake at ~130 ms; a 400 MB transformer turns it into a 20–30 second predict. Linked modules: 4.2 (Serving with FastAPI), 4.1 (Notebook to repo).
The load call lives inside the endpoint function, so every request pays the full cost of reading the model from disk and rebuilding it in memory — work that produces the identical object every time. Three signs confirm it without reading the code: latency roughly equals the model's known load time; the server log prints a "loading model" line (or disk I/O spikes) once per request instead of once per boot; and the first and the fiftieth request cost the same, meaning nothing is being kept warm. Concurrency makes it worse than slow: several requests loading at once hold several copies of the model in memory, which is where the spikes and OOM kills come from. The decisive experiment is a measured pair: time 30 requests as-is, move the load, time 30 again.
Move the load to process startup — FastAPI's lifespan block (or a module-level
load) — and keep the object in memory: state["model"] = joblib.load(...) runs
once, and the endpoint only calls predict. Restart, then re-measure the same 30
requests: in the 4.2 churn service the median falls from 138 ms to 9 ms; for the
400 MB model, from ~30 s to ~50 ms. Check the log now shows exactly one "model
loaded" line per process start. The load did not disappear — it moved to boot,
where it is paid once and no caller is waiting. That slow first stretch is the
cold start, and deployers handle it by waiting for /health before routing
traffic.
Adopt the measurement habit: every serving change ships with a before/after table
of median and p95 over 30+ requests, cold first call reported separately. Review
every endpoint with one question — "what in this function runs per request but
would produce the same result per process?" — model loads, tokenizer builds,
config reads and DB connections all belong at startup. Keep /health honest
(report whether the model is actually in memory), so a deploy that fails to load
never receives traffic.
Q: "Your ML API is functionally correct but users say it is painfully slow. Walk me through your diagnosis." Strong answer: measure first — median and p95 over repeated requests, cold call separated — then compare latency to the model's known load time and check whether "loading" appears in logs per request. Name per-request model loading as the classic cause, startup loading as the fix, and quote the before/after numbers you would expect. Bonus signal: mentioning the memory blow-up under concurrent loads and the /health-gated cold start shows production experience, not tutorial experience.