A retrain shipped and the registry shows the new model version live, but predictions coming out of the service have not changed. Known test customers return the exact scores they returned before the deploy. Latency looks suspiciously good — sub-millisecond p50 on a model that takes 12 ms — and the model-server's own request count is far below the number of responses the service returned. Downstream, a risk team keeps calling customers the new model would have cleared, and misses ones it would have flagged.
A prediction service gained an in-memory cache (an LRU decorator, or Redis) to absorb dashboard repeats — a sound idea that measurably cut latency and cost. The cache key was built from the customer's feature values only. Months later, monitoring flagged drift, the team retrained, and the pipeline deployed the new version cleanly. Nobody connected the two systems: the cache kept every entry scored by the old model. In the reported case the stale scores served for eleven days. Linked modules: 4.9 (Scaling & cost control), 4.8 (Monitoring), 4.4 (MLflow registry).
A cache entry is a frozen answer, and an answer is only valid while the thing that produced it stays the same. Keying on features alone declares "the model never changes" — false the moment a retrain ships. The failure hides well: health checks pass, the deploy log shows the new version loaded, and spot checks on NEW customers (cache misses) even return fresh scores. Only repeated customers are stale, and they are the majority of dashboard traffic. Confirm it in two moves: score one cached customer, restart the service (or flush the cache), score again — the answer changes, proving the cache was the liar. Then compare cache hit count against the model's own invocation count to size how much traffic the old model was still answering.
Invalidate by construction, not by memory. Put the model version into every
cache key — (model_version, feature_tuple) — so a version bump makes every
old entry unreachable instantly, with no flush step to forget. Add a TTL as
the second line of defence (for churn scores, hours, not days), so entries
also die on a clock even if versioning is ever bypassed. Flush the cache as
an explicit step in the deploy pipeline for caches that live outside the
process, like Redis. Then re-run the last eleven days' affected customers as
a batch job so downstream decisions get corrected scores.
Write the caching rule into the service's README: a cache key must contain every input that can change the answer — features AND model version — and every entry must carry an expiry. Add a canary check to the deploy pipeline: after a version bump, request one known repeated customer and assert the score matches the new model's offline prediction. Monitor the cache hit rate as a first-class metric; a hit rate that does NOT dip right after a deploy means invalidation failed. Never cache anything time-dependent or personalised beyond the model's inputs, and never cache writes.
Q: "You deployed a retrained model but customers report nothing changed. Health checks are green. Where do you look?" Strong answer: suspect every layer that can answer without the model — caches, CDNs, memoised functions — and prove it with the flush-and-compare test. Name the root cause as an incomplete cache key (features without model version), give the fix (version in the key, plus TTL, plus a deploy-time canary on a repeated customer), and note the monitoring tell: sub-millisecond latencies and a hit rate that never dipped after deploy. That layered answer separates operators from tutorial-followers.