Nothing. That is the symptom. The service answers every request with a 200, latency is steady, the error rate is near zero, the CI badge is green, and no alert has fired for months. Then a business report — not a dashboard — announces that the model has been missing the cases it exists to catch: churners not flagged, fraud not held, defaults not predicted. The recall that shipped at 0.66 is measured, after the fact, at 0.41.
A model deployed with real care — tested (4.5), gated in CI (4.6), served from the cloud (4.7) — and then left alone. The team watched what was easy to watch: uptime and errors. Nobody joined predictions to outcomes, because the outcome labels arrive about 60 days late and "we will add that later". Common wherever truth is slow: churn (60 days), loan default (months), insurance claims. Linked modules: 4.8 (Monitoring), 4.5 (Testing ML code), 4.4 (registry).
The service and the model are two different patients, and only one was being examined. HTTP status codes report whether the code ran, never whether the answer was right — a confidently wrong prediction is a perfect 200. Meanwhile the world drifted: a rival app changed ordering habits, a new customer segment arrived, and the feature distributions moved away from the training data. Input drift began weeks before the quality drop, and a PSI-style check would have scored it; the recall decay was knowable 60 days after each batch of labels arrived, had anyone joined them. Confirm the mechanism in two steps: run a distribution comparison of last month's logged features against the training set (drift will be visible in specific features), then join old predictions to the outcomes that have already landed and score recall by week — the decay curve was sitting in the data, unread.
Stop the bleeding first: retrain on the freshest labelled window — 4.1's train.py rerun, a new registry version (4.4), the 4.5 gate as judge, the 4.6 pipeline to ship it — and confirm the new version clears the recall floor on recent outcomes. Then build the missing layers the same week: structured logging of model inputs and verdicts with an opaque join key, a scheduled daily drift check with a PSI-lite score per feature, and a weekly job that joins predictions to arrived labels and tracks rolling recall against the same gate threshold the model must pass before shipping.
Treat go-live as the start of measurement, not the end of the project: the monitor ships in the same sprint as the service, never "later". Log every prediction with the features that made it and a join key on day one — outcome tracking cannot be added retroactively to predictions nobody recorded. Reuse the offline gate threshold as the online one, so "still good enough" has the same number before and after deploy. And put one rule in the runbook: if the labels arrive late, the drift check is the early-warning system, so it runs daily, not when someone remembers.
Q: "Your deployed model's metrics were fine all quarter, but the business says it stopped working. How is that possible, and what do you change?" Strong answer: name the gap — service health was monitored, model quality was not, and a 200 says nothing about correctness. Explain the two missing layers (input drift scored per feature; predictions joined to late labels for rolling recall against the shipping gate) and why drift leads quality by weeks when labels lag. Finish with the day-one rule: log features and a join key with every prediction, because outcome tracking cannot be backfilled. That answer shows you have operated a model, not only trained one.