A collaborative-filtering recommender goes live, replacing the plain "Bestsellers" row. Within two weeks the A/B dashboard is unambiguous: the old most-bought list gets more taps, more add-to-carts, and better conversion than the new personalised row. The team is blindsided — every demo before launch looked brilliant. Leadership asks why weeks of modelling lost to a five-line SQL query.
A first recommender at an e-commerce or quick-commerce company, built by a small team from the order log — user-user or item-item similarity, demoed on a handful of accounts in a sprint review. No offline evaluation was run: no time split, no precision@k, no comparison against the popularity baseline that was already serving. The demo accounts were chosen by the team — usually their own accounts or the heaviest users in the table. Linked module: 1.12 (recommendation systems intro).
Two effects stack. First, the demo was the model's best case: heavy users have rich rows in the user-item matrix, so their similarity scores are stable and their lists look sharp. Typical users have thin rows — a few purchases — and their "nearest neighbours" are close to noise, so the served lists are weak exactly where most traffic is. Second, the baseline is strong by construction: items are popular because many people want them, so the most-bought list is somewhat right for almost everyone. A model that beats it for the top decile of users and loses for the rest loses the average. Nobody knew any of this before launch because nobody measured precision@5 against the baseline on a time split — the crossover between "CF wins" and "popularity wins" was never located.
Roll the default row back to popularity — losing to the baseline is information, not shame. Then build the offline harness the launch skipped: split the order log by time (train on the past, test on the future), and score both recommenders with precision@5 and hit-rate@5 on the same split. Segment the scores by history depth: users with under N orders versus over N. Ship CF only to the segments where it beats the baseline, and keep popularity for thin-history users. Re-enter the A/B with that blended policy and let the dashboard confirm what offline already showed.
Make baseline-gated shipping a team rule: no recommender reaches an A/B test without an offline scorecard showing it beats the popularity baseline on a time split. Keep the harness as a script in the repo and re-run it on every model change. Demo on randomly sampled accounts, never handpicked ones, and always show the thin-history segment in the demo. Say the quiet part in every review: the bestseller list is the opponent, and it is good.
Q: "How would you know your recommender is actually better than a bestseller list?" Strong answer: split the log by time, score precision@k and hit-rate@k for both the model and the popularity baseline on the same split, segment by user history depth, and only then A/B test. The standout detail is admitting the baseline often wins for thin-history users — proposing a blend by segment signals production experience rather than tutorial experience.