A regression model's RMSE suddenly jumps — 3 to 14, or 40 to 190 — and R² collapses, sometimes overnight. The team assumes the model has degraded everywhere, opens an urgent retraining ticket, and starts swapping features or algorithms. Meanwhile MAE, if anyone printed it, moved only slightly. Predictions on typical rows look as good as ever.
Delivery-time, demand, load or price prediction — any pipeline scored on RMSE alone, usually because RMSE was the tutorial default. The trigger is one or a few extreme rows: a festival traffic jam, a cyclone day in the electricity load table, a data-entry slip adding a zero, a sensor stuck at max. Quick-commerce and logistics teams meet this every festival season. The wasted work happens because the team "fixed" the model when one row was the whole story. Linked module: 1.5 (Evaluation & metrics deep dive).
RMSE squares each error before averaging, so a single 140-minute miss contributes
19,600 to the sum while a hundred 3-minute misses contribute 900 together. One row can
own the metric. MAE treats every miss by its plain size, which is why it barely moved
— and the MAE-versus-RMSE gap is itself the diagnostic. Confirm in two prints: both
metrics side by side, and the residuals sorted by absolute size
((y_true - y_pred).abs().sort_values(ascending=False).head()). If the top residual
dwarfs the rest, the model did not break; the world produced an outlier, or the data
pipeline did. Then inspect that row and classify it: genuine rare event, or data
error.
Fix the row's category, not the model. A data error gets corrected or removed with a note in the report. A genuine rare event stays in the data, and the honest report says so: MAE for typical performance, RMSE with the outlier explained, and the worst rows listed. If tail days matter to the business — festival surges are exactly when promises break — treat them as their own problem: a festival flag feature, a separate tail-day model, or an explicit "predictions unreliable on event days" caveat. Cancel the panic retraining unless the residual table shows errors growing across many rows, not one.
Report MAE and RMSE together on every regression, always — the gap between them is a free outlier alarm. Make a sorted residual table part of the standard evaluation printout, so the worst rows are seen before any metric is believed. Keep a calendar of known tail days (festivals, sale events, weather) beside the data. Never let a single aggregate number trigger a model change without one look at the rows that produced it.
Q: "Your delivery model's RMSE tripled this week but MAE is almost flat. What happened and what do you do?" Strong answer: explain squaring in one sentence — RMSE lets one giant miss dominate — infer an outlier from the gap, and name the confirming artefact: the sorted residual table. Then split the fix by cause: repair data errors, keep and disclose genuine rare events, and consider a festival-aware feature. Reaching for retraining first is the wrong answer this question exists to catch.