Scaling & cost control
Bhavya runs the churn service for an NBFC from Belagavi. The pipeline is green, the monitor is calm, and then the month's AWS bill arrives: ₹18,400, four times the estimate. She digs through the logs. A nightly script has been calling the API ten lakh times, one customer at a time, and autoscaling billed for every extra copy. Finance asks one question she cannot answer: what does one prediction cost? Her lead writes on the board: a service that runs is not a service that is priced.
- Every module in this phase spent money to make the churn service better; this one makes it cheaper without making it worse. It is the phase finale: the service ends today deployed, monitored and priced.
- "Your AWS bill doubled — walk me through your response" is a real interview question for MLOps and GenAI roles, and cost per prediction is the spine of a strong answer.
- A fresher rarely gets to redesign a model in month one, but a fresher who runs a cost review and finds ₹9,000 of waste earns trust immediately. Reading a bill is a hireable skill.
The churn service is live, watched, and paid for by the hour. Growth now arrives in two shapes, and they need different answers. More requests means the service must raise its throughput — the number of requests it can answer per second. Bigger jobs means one caller wants lakhs of rows scored at once. Confusing the two shapes is the most expensive mistake this module exists to prevent.
Ask one question before any scaling work: does anyone need this answer right now? A fraud check inside a UPI flow does — a human is waiting, so the real-time API earns its cost. A nightly risk list for tomorrow's calling team does not. That job wants batch scoring — scoring a large set of rows in one run and writing the results to a table. No API, no network, no queue of waiting calls.
Here is the wrong way, and it is common enough to have a War Room entry. The nightly need gets met by a loop that calls the public API ten lakh times, one customer per call. Every call pays the full HTTP toll — network, parsing, validation — to reach a model that answers in milliseconds. The loop runs most of the night, autoscaling adds copies to absorb it, and the bill quadruples. The right way is a thirty-line script: load the pipeline once, call predict on the whole table, write the scores out.
Real-time traffic carries its own waste: the same question asked again and again. Risk dashboards re-open the same customers, and each repeat pays for a fresh prediction that cannot have changed. A cache answers repeats from memory, and an LRU cache — a small store that drops the least recently used entry when full — fits in one Python decorator. Success is measured by the cache hit rate — the share of requests answered from the cache instead of the model. Danger is measured by staleness: a cached score must die when the model changes, so every key carries the model version and every entry carries an expiry.
When one copy of the service is not enough, the cloud adds more copies — that is horizontal scaling, and App Runner has done it since 4.7. It works only because the churn service is stateless — it keeps nothing about past requests inside itself. The model loads into memory at startup, every request carries its own data, and any copy can answer any caller. Store a session or a running total inside the process, and copy two starts giving different answers from copy one. Statefulness is the enemy of scale; state belongs in a database or a cache, never in the service.
Buying a bigger machine is the first instinct and rarely the correct answer. Right-sizing starts from the 4.8 numbers: a p95 of 19 ms on one small instance means the service is mostly idle, not straining. Measure, resize downward, and measure again — if the p95 stays inside the target, the smaller instance was the honest size all along. The judge of every such move is cost per prediction — the month's bill divided by the month's predictions. Two services can both look cheap per hour; only the per-prediction number says which one is efficient.
LLM services obey the same laws with bigger numbers, because 3.2 taught that every token is billed. Three levers cut an LLM bill without touching quality where it matters. Cache the repeats — support bots hear the same twenty questions all day. Truncate the prompt, because context a model never needed is money burned at the highest rate. Route small-model-first: send every request to a cheap model, and escalate to the big one only when the small one is unsure or the stakes are high.
The last leak is not in any running service — it is in the ones everyone forgot. A zombie resource — a cloud resource nobody uses that is still billing — is how dev experiments become permanent line items. The 4.7 teardown was a personal ritual; at company scale it becomes an organisational habit: tag every resource with an owner, hunt monthly, and auto-stop whatever dev leaves running. A fresher who runs that hunt with a checklist can save more than their own stipend.
This module closes Phase 4, and the arc deserves saying out loud. The churn model left its notebook in 4.1, learned to serve in 4.2, boxed itself in 4.3, and got tracked in 4.4. Tests guarded it in 4.5, a pipeline shipped it in 4.6, AWS ran it in 4.7, and monitors watched it in 4.8. Today it is priced: every prediction has a rupee figure, every repeat has a cache, and every idle resource has a hunter. Built, served, containerised, tracked, tested, integrated, deployed, monitored, priced — the whole distance from notebook to production.
Work inside churn-repo with the venv active and the 4.2 service runnable. Everything below runs on your laptop against the local service; the rupee figures use 4.7's honest App Runner rates so the arithmetic transfers straight to the cloud.
Step 1 — the nightly job, both ways. First the batch road: one script, one model load, one predict call over the whole table:
# scripts/batch_score.py — the nightly need as a batch job, not an API hammer
import time
import joblib
import pandas as pd
from src.data import load_customers # the seeded world from 4.1
from src.features import add_features # the same recipe /predict uses (4.2)
pipe = joblib.load("models/churn_pipeline.joblib") # load ONCE, like 4.2's startup
world = load_customers() # 4.1's seeded rows
customers = world.sample(100_000, replace=True, # tonight's book: the world
random_state=42).reset_index(drop=True) # resampled to 1 lakh
customers["ref"] = [f"{i:06x}" for i in customers.index] # opaque code, 4.8's shape
t0 = time.perf_counter()
probs = pipe.predict_proba(add_features(customers))[:, 1] # one call, all rows
elapsed = time.perf_counter() - t0
out = customers[["ref"]].copy()
out["churn_prob"] = probs.round(2)
out["model_version"] = "2"
out["scored_at"] = pd.Timestamp.now(tz="UTC").isoformat()
out.to_parquet("scores/2026-08-11.parquet") # a table, not 10 lakh responses
print(f"scored {len(out):,} rows in {elapsed:.1f}s "
f"({len(out) / elapsed:,.0f} rows/sec)")
The shape is 4.1's lesson resurfacing: the pipeline that serves one customer in 12 ms will happily take a whole DataFrame in one call, because vectorised NumPy underneath does lakhs of rows per second. The output lands as a parquet file any analyst, SQL table loader or calling-team dashboard can read. Two honest notes on the book: load_customers() returns 4.1's seeded 500-row world, so the script resamples it to one lakh rows to give the clock something real to chew, and the ref code is minted here from the row index — the training world never carried one, and the six-hex shape matches what 4.8's CustomerIn expects. Note the model_version column — the batch equivalent of the cache-key discipline coming in step 2. Now measure both roads honestly:
$ python scripts/batch_score.py
scored 100,000 rows in 1.9s (52,631 rows/sec)
$ python scripts/hammer_sample.py # times 200 sequential POSTs to /predict
200 calls in 7.3s -> 36.5 ms per call average
the same nightly job at 10 lakh customers, both ways:
per-call API loop batch script
wall-clock 10,00,000 x 36.5 ms ~10 x 1.9 s
= 10.1 hours = 19 seconds
compute while it runs App Runner scales 1 -> 4 one process on the
copies for ~10 hours machine already there
extra bill per night 4 x Rs 6/hr x 10 hr about Rs 0
= Rs 240 -> Rs 7,200/mo
answer freshness ready by morning ready by morning
Read the table like finance will. The API loop pays the HTTP toll ten lakh times to reach maths that takes microseconds, so the night is spent on overhead, not prediction. Autoscaling is not a villain here — it did its job absorbing load; the mistake was creating the load. The batch script finishes in under a minute, costs nearly nothing, and the customer experience is identical because nobody was waiting at 2 am. This table is the scene's ₹18,400 explained, and WR-115 is its full story.
Step 2 — cache the repeats, with an invalidation rule. Dashboards re-request the same customers all day. One decorator answers them from memory:
# --- addition to config.py, beside SEED and MODEL_PATH ---
MODEL_VERSION = "2" # bump when a retrain ships; 4.4's registry is where
# a bigger team would read this from at startup
# app/cache.py — LRU cache on the scoring path, model version in the key
from functools import lru_cache
import pandas as pd
import config # MODEL_VERSION lives with the other knobs
from src.features import add_features # the same recipe /predict uses (4.2)
@lru_cache(maxsize=10_000)
def _cached(version: str, key: tuple) -> float:
# imported here, not at the top: app/main.py imports this module,
# so a module-level import back into it would be a circular loop
from app.main import state # the pipeline 4.2 loaded at startup
row = pd.DataFrame([dict(key)])
row["last_order_date"] = pd.to_datetime(row["last_order_date"])
return float(state["model"].predict_proba(add_features(row))[:, 1][0])
def score(customer: dict) -> float:
key = tuple(sorted(customer.items())) # same customer -> same key
return _cached(config.MODEL_VERSION, key) # new model version -> cold cache
def stats() -> dict: # served by GET /cache-stats
info = _cached.cache_info()
return {"hits": info.hits, "misses": info.misses, "currsize": info.currsize}
Two design choices carry the whole lesson. The key is the sorted feature tuple, so the same customer always lands on the same entry — and MODEL_VERSION sits inside the key, so the day 4.8's retrain ships version 3, every old entry silently misses and the model answers fresh. Leave the version out and the cache keeps serving pre-drift scores after the retrain: that is WR-116, a cache turning into a liar. Replay realistic dashboard traffic and read the meter:
$ python scripts/replay_dashboard_traffic.py # 2,000 requests, heavy repeats
replayed 2,000 requests against http://localhost:8000
$ curl localhost:8000/cache-stats
{"hits": 1176, "misses": 824, "currsize": 824}
cache hit rate = 1176 / 2000 = 58.8%
p50 latency cached: 0.8 ms uncached: 11.9 ms
cost per prediction, the end-to-end arithmetic:
instance 1 vCPU / 2 GB ............ Rs 4,320 / month (Rs 6/hr x 720)
predictions served ................ 55,000 / month
cost per prediction ............... Rs 0.078 (~8 paise)
after right-sizing to 0.25 vCPU / 0.5 GB (4.8 says the CPU is mostly idle):
measured p95 after resize ......... 24 ms (target: under 100 ms)
instance .......................... Rs 1,080 / month
cost per prediction ............... Rs 0.020 (~2 paise)
The hit rate says most dashboard traffic was repeats, and each repeat now costs under a millisecond and zero model work. The bottom half is THE metric of this module computed end to end: rupees in, predictions out, divided. Right-sizing was decided by measurement, not mood — 4.8's p95 of 19 ms proved the big instance was idling, the resize was tried, and the new p95 of 24 ms stayed far inside the 100 ms target. The bill fell 75% and no caller can tell the difference. Never cache what must not repeat: anything time-dependent, anything per-user-personalised beyond the features, and any endpoint that writes.
Step 3 — the zombie hunt, as a script the whole team runs monthly:
$ bash scripts/zombie_hunt.sh # list commands only — it deletes nothing
== App Runner services (ap-south-1) ==
churn-live RUNNING tags: team=risk owner=bhavya KEEP
demo-hackathon RUNNING no tags, last request 94 days ago KILL Rs 4,320/mo
== ECR repositories ==
churn-service 42 images, no lifecycle policy RESIZE (keep last 5)
old-poc 17 images, last push 8 months ago KILL Rs 310/mo
== S3 buckets ==
bhavya-churn-models-2026 versioned, read this week KEEP
tmp-data-dump 1.9 TB, last read: never KILL Rs 3,900/mo
== EBS volumes ==
vol-0f3a91c2 80 GB, state: available (attached to nothing) KILL Rs 610/mo
monthly saving if the four KILL rows die tonight: Rs 9,140
The script is deliberately read-only: it prints evidence and a recommendation, and a human makes the kill decision — the same alarm-versus-brake honesty 4.7 taught about budgets. The tells are worth memorising: no tags means no owner, available on a volume means attached to nothing, and "last activity" months ago means the bill is buying nothing. The ₹9,140 line is why this ritual survives in companies: one intern-hour a month, four figures back. WR-117 is the six-month version of this list, found the expensive way.
Step 4 — the same discipline on an LLM bill. A support bot from 3.2's world, triaged with the three levers in order:
support bot: 50,000 questions/month
big model: Rs 250 per million input tokens, Rs 1,250 per million output
small model: Rs 21 per million input tokens, Rs 105 per million output
step input tok output tok monthly bill
0 today: 1,200-token prompt, big model 60 M 15 M Rs 33,750
1 cache the 30% repeated questions 42 M 10.5 M Rs 23,625
2 truncate prompt 1,200 -> 400 tokens 14 M 10.5 M Rs 16,625
3 route 70% small-first, big for the
hard 30% (escalation on low score) Rs 5,965
same answers where quality matters — 82% off the bill
Walk the levers with the token maths. Caching removes whole requests, so it cuts both columns at once: 50,000 calls become 35,000. Truncation attacks the input column only — 1,200 tokens of pasted context shrank to the 400 the model actually used, and the input bill fell from ₹10,500 to ₹3,500. Routing is the deepest cut because the small model's rates are roughly a twelfth of the big one's: the easy 70% costs ₹978, and the big model keeps only the hard 30% at ₹4,988. The order matters — cache first (removes calls), truncate second (shrinks calls), route last (reprices calls) — and every step is measurable before and after.
Work inside churn-repo with the venv active and the 4.2 service runnable locally. No AWS account is needed: every rupee figure uses the honest rates from the watch section, and the one cloud exercise works from a provided inventory.
The batch road. The calling team needs every customer's churn score in a table by 7 am; nobody reads it before 9. Write scripts/batch_score.py as in the watch section: seeded 100,000-customer book, one model load, one predict_proba call, parquet output with ref, churn_prob, model_version and scored_at. Then write scripts/hammer_sample.py that times 200 sequential POSTs to your local /predict. Finish with COST_NOTE.md: the two-column comparison for 10 lakh customers — wall-clock both ways, extra compute both ways, rupees both ways at ₹6/hour — ending with one sentence answering "does anyone need this answer right now?". Check: the batch script reports over 20,000 rows/sec and the parquet opens in pandas with all four columns; the extrapolated API-loop time exceeds 8 hours while the batch estimate stays under one minute; COST_NOTE.md shows the monthly rupee difference above ₹5,000 and names the question that decides the road.
Cache with a conscience. Add app/cache.py from the watch section: LRU-cached scoring keyed by the sorted feature tuple plus MODEL_VERSION, and a /cache-stats endpoint. Write scripts/replay_dashboard_traffic.py that sends 2,000 requests where a seeded 40% of customers appear multiple times. Measure the hit rate and the cached-versus-uncached p50. Then prove invalidation: bump MODEL_VERSION in config.py to "3", restart, replay, and watch the stats. Close with two lines in notes-4.9.md naming one request the service must never cache and why. Check: the replay reports a hit rate between 50% and 70% and cached p50 under 2 ms; after the version bump the first replay's hit count starts from zero — old entries never serve; your never-cache example involves time-dependence, personalisation or a write, with the reason stated.
The monthly cost review. You inherit this inventory on day one. For each row write KEEP, RESIZE or KILL with a one-line reason, then total the monthly saving in rupees.
| # | resource | type | region | tags | last activity | ₹/month |
|---|---|---|---|---|---|---|
| 1 | churn-live (1 vCPU/2 GB) | App Runner | ap-south-1 | team=risk | serving now, CPU 4% | 4,320 |
| 2 | churn-service | ECR (42 images) | ap-south-1 | team=risk | pushed today | 780 |
| 3 | bhavya-churn-models-2026 | S3, versioned | ap-south-1 | team=risk | read this week | 120 |
| 4 | demo-hackathon | App Runner | ap-south-1 | none | 94 days ago | 4,320 |
| 5 | tmp-data-dump | S3 (1.9 TB) | ap-south-1 | none | never read | 3,900 |
| 6 | vol-0f3a91c2 (80 GB) | EBS, available | ap-south-1 | none | unattached | 610 |
| 7 | old-poc | ECR (17 images) | us-east-1 | none | 8 months ago | 310 |
| 8 | dev-nat-gateway | NAT gateway | ap-south-1 | env=dev | dev VPC, no traffic | 2,700 |
| 9 | hello-test | Lambda | ap-south-1 | none | 6 months ago | 0 |
| 10 | /aws/apprunner/churn-live | CloudWatch logs, never-expire | ap-south-1 | team=risk | growing daily | 950 |
| 11 | dev-db (db.t3.medium, 24x7) | RDS | ap-south-1 | env=dev | weekday office hours | 3,100 |
| 12 | eip-52.66.x.x | Elastic IP | ap-south-1 | none | unattached | 300 |
Write the review as COST_REVIEW.md: verdict plus reason per row, the rupee total your verdicts save, and a three-item checklist a fresher could run next month without you. Check: rows 4, 5, 6, 7, 8 and 12 are KILL and rows 1, 10 and 11 are RESIZE (smaller instance; retention policy; auto-stop outside office hours) with reasons naming the evidence — no-tags, unattached, never-read, idle CPU; your KILL total is ₹12,140/month and your KILL-plus-RESIZE saving lands between ₹17,000 and ₹19,000; the checklist includes tags, last-activity and an unattached-storage check.
LLM triage. A startup's support bot answers 40,000 questions a month with an 800-token prompt and 250-token answers on the big model (₹250/M input, ₹1,250/M output). Logs show 25% of questions are repeats and a small model (₹21/M in, ₹105/M out) handles 60% of traffic acceptably. Build LLM_TRIAGE.md: the step-0 bill, then the bill after each lever applied in order — cache, truncate to 300 tokens, route small-first — with the token arithmetic shown at every step. End with one sentence on why the order cache-truncate-route is the only sensible one. Check: your step-0 bill is ₹20,500/month; each lever's saving is computed from token counts, not guessed; the final bill lands between ₹4,000 and ₹6,500; the ordering sentence says removing calls comes before shrinking calls, which comes before repricing calls.
Three scale-and-cost failures cover most first-year bills. Each has a War Room entry with the full diagnosis path:
- WR-115 — the nightly API hammer. A batch need implemented as ten lakh sequential HTTP calls ran nine hours a night and quadrupled the bill through autoscaling. The fix was a thirty-line batch script and one question asked before any scaling work: does anyone need this answer right now?
- WR-116 — the cache that served stale risk. Predictions cached without an invalidation rule kept serving pre-drift scores for eleven days after a retrain, and the monitor stayed calm because the model was never asked. Model version in every cache key plus a TTL ended it.
- WR-117 — the zombie farm. Six months of forgotten dev services, never-read buckets and unattached volumes billed ₹58,000 before anyone looked. Owner tags, a monthly read-only hunt and auto-stop policies turned it into a one-hour ritual.
Scaling questions come in two shapes: more requests need throughput, while bigger jobs need batch scoring, and the nightly ten-lakh job belongs to the second. A batch script that loads the model once and writes a table beats ten lakh HTTP calls by ten hours and thousands of rupees a month. Caching answers repeats from memory, but every key must carry the model version so a retrain can never serve stale risk. Cost per prediction — the month's bill divided by the month's predictions — is the one number that makes right-sizing, caching and routing comparable, and measurement decides instance size, never instinct. The phase closes with a habit and an arc: hunt zombie resources monthly, and the churn service now stands built, served, containerised, tracked, tested, integrated, deployed, monitored and priced.
- NBFC and bank collections. Indian lenders score full loan books overnight as batch jobs in the Mumbai region — crores of rows into a table the calling teams read at 9 am. The real-time scorer stays small and serves only the flows where a human is waiting.
- UPI-scale systems. Payment and commerce platforms put caches in front of everything that repeats — merchant lookups, risk features, fraud lists — and watch hit-rate dashboards the way you watched p95. At festival load, the cache is the difference between scaling out and falling over.
- GenAI support bots. Teams running LLM assistants live by the three levers: prompt caching, context trimming and small-model-first routing, because token bills compound at lakh-conversation scale. The routing decision is exactly the maths you did in exercise 4.
Answer these before opening the gate assessment. Each question names its kind.
type=output-prediction— The 4.8 retrain ships andMODEL_VERSIONmoves from "2" to "3" at restart. Predict what/cache-statsshows after the first 500 dashboard requests, what happens to the p50 for those requests compared with yesterday, and where the hit rate settles by evening — then say why this brief slowdown is the design working.type=debug— The cache is live but/cache-statsshows hits: 0 after a full day, though the replay log proves 60% of customers repeated. A teammate's payload includes arequested_attimestamp field. Name the cause, the one-line proof you would print, and the fix — plus the rule that prevents the whole class of bug.type=interpretation— A cost sheet shows service A at ₹1,080/month serving 9,000 predictions and service B at ₹4,320/month serving 1,44,000 predictions. Compute both cost-per-prediction figures, name which service is actually efficient, and explain what decision the per-hour price alone would have wrongly suggested.type=scenario-judgement— A product manager asks for tomorrow's full-book churn list and proposes looping over the existing API "since it is already deployed and tested". Judge the proposal, name the War Room failure it recreates and the question that decides the design, and describe the alternative with its rupee and wall-clock case.type=trade-off— A 24-hour cache TTL versus a 5-minute TTL for churn scores: give one concrete gain and one concrete cost of each, pick one for a risk dashboard whose model retrains weekly and whose features refresh nightly, and name the log line that would make you revisit the choice.
Explain to a friend who runs a tiffin service why she cooks all eighty lunch dabbas in one morning batch instead of cooking each meal when a customer phones. Cooking in one batch means the stove heats once and the work flows; cooking to order suits only the few customers who need food this minute. She also keeps the common chutney ready-made instead of grinding it fresh for every identical order — but she throws it out and grinds again whenever the recipe changes. Use at most 4 sentences and no computer words. If she can also say why she checks at month-end for the gas cylinder she rents but never uses, you own this module.