The monthly cloud bill jumps to roughly four times the estimate with no new feature shipped. The autoscaling graph shows the service climbing from one copy to its maximum every night between roughly 11 pm and 9 am, then dropping back. A "nightly scoring" cron job takes nine to ten hours to finish, and some mornings the output list is late or incomplete. Daytime users occasionally see slow responses in the early morning overlap.
A churn (or credit-risk, or lead-scoring) model is deployed as a real-time
HTTP API — FastAPI on App Runner or similar. A downstream team needs every
customer scored once a day: about ten lakh rows, read from a dashboard the
next morning. Someone met that need with a loop: for each customer, POST to
/predict, collect the response. It worked in the demo with 500 customers,
so it went live for the full book. Linked modules: 4.9 (Scaling & cost
control), 4.2 (Serving with FastAPI), 4.7 (Cloud deployment).
The model answers in about 12 ms, but each HTTP call pays the full toll — network round trip, TLS, parsing, validation — landing near 35–40 ms per customer. Sequentially, ten lakh calls is 10,00,000 × 36.5 ms ≈ 10 hours of wall-clock for maths that vectorises to under a minute. Meanwhile autoscaling sees sustained load and does its job: more copies, billed by the hour, all night, every night — the quadrupled bill is the platform absorbing load that never needed to exist. Confirm the shape in three places: the request log shows one caller's IP or token issuing lakhs of near-identical POSTs at night; the autoscaling metrics correlate exactly with the cron window; and a timed 200-call sample multiplied out reproduces the nine-hour runtime.
Ask the deciding question: does anyone need this answer right now? Nobody
reads the list before morning, so this is batch scoring, not serving. Replace
the loop with a thirty-line script — load the trained pipeline once, call
predict_proba on the whole table in one vectorised call, write ref,
churn_prob, model_version and scored_at to a parquet file or database
table the dashboard reads. Measured honestly: 100,000 rows in about 2
seconds, the full book in under a minute, extra cost near zero. Keep the
real-time API for the flows where a human is actually waiting, and cap its
autoscaling maximum so a future hammer cannot silently spend the budget.
Put the question "does anyone need this answer right now?" into design reviews as a standing checklist item — every scoring request must declare real-time or batch before it gets built. Watch cost per prediction monthly: an API loop shows up instantly as a per-prediction price many times the batch path's. Alert on autoscaling events that recur on a schedule, because periodic scale-out is almost always a machine calling a machine. And publish the batch script next to the service so the easy road is also the cheap one.
Q: "Your model API's bill quadrupled but daytime traffic is flat. Walk me through your diagnosis and fix." Strong answer: read the autoscaling graph against the clock, find the nightly caller in the logs, and name the real problem — a batch need implemented as lakhs of sequential HTTP calls. Fix with a vectorised batch job writing to a table, keep the API for human-waiting flows, and quote the arithmetic (per-call overhead × volume versus one predict call). Finishing with "I would add the real-time-or-batch question to design review" shows you fix classes of problems, not incidents.