Your design round goes smoothly. You clarify, you frame the problem, you pick sensible features, you argue for a model, and you draw a clean set of boxes with arrows between them. The interviewer nods along. Then the clock runs out, or she asks one last question — "how would you know this is working next month?" or "what happens when it is wrong?" — and you have thirty seconds and nothing prepared. The feedback comes back as "good on modelling, light on production thinking", "would need supervision", or the one that stings, "reads as academic". Nothing you said was incorrect.
Any ML system design round, and most often in the second half, since this failure is created by how the first half was spent. It is most common in candidates who are strong at modelling, because time spent enjoyably on model choice is time taken from the last two steps. It also shows up in written take-home design docs, where the evaluation and monitoring sections are a heading with two lines under them. Freshers coming from coursework and competition datasets are the most exposed, since in both of those a model's life ends the moment the score is computed. Linked module: 6.5 (ML system design / case round).
A system is not a model, and the interviewer is hiring for the system.
Everything before step 5 answers "can this be built". Steps 5 and 6 answer "can this be run", and running it is what the job is. A design that stops at the model is describing a notebook that happens to have boxes drawn around it. That is why the feedback word is so often "academic": the candidate has described the part of the work that finishes, and left out the part that never does.
The three questions that expose it are always the same, and none of them is about modelling. How will you know it is working after launch? What breaks, and who notices first? What does the system do while it is broken? A candidate who has run something in production answers those from memory. A candidate who has not either goes quiet or offers a word — "I would monitor it" — with nothing under the word.
The cause is nearly always the clock, not ignorance. Ask the same candidate about drift, or about a metric that maps to money, on a whiteboard with no timer, and many of them answer well. Under 45 minutes they spend twenty-five on the model, arrive at serving at minute thirty-five, and never reach the part that was carrying the most marks. The step that gets cut is the last one, and the last one is the one the panel uses to separate candidates who are otherwise level.
There is a second cause worth naming: the belief that the failure story makes you look negative. Freshers hide the weaknesses of their own design because they are being assessed. In this round, naming what breaks reads as experience, and naming nothing reads as never having shipped.
Treat steps 5 and 6 as compulsory, budget their minutes before the round starts, and carry a fixed shape for each.
Protect the time. On a 45-minute round, the last fifteen minutes belong to serving, evaluation, monitoring and failure. If you are still on model choice at minute twenty-five, move on and say so out loud: "I will move to serving so we cover monitoring — happy to come back to the model if you want more there." Saying it converts a rushed section into a deliberate one.
The evaluation shape, three lines.
The monitoring shape, four lines. Input drift, output drift, latency, cost. Then one sentence on the label delay, because that is what makes ML monitoring different from ordinary service monitoring: if labels arrive twelve months late, you cannot measure the real metric weekly, so name the proxy you would watch instead.
The failure shape, two stories. Each story has three parts — what breaks, who or what notices, and what the system does meanwhile. "The vendor API times out; the p95 latency alert fires; requests fall back to the baseline rule and the batch is marked for re-scoring." Two stories is enough. Take them from War Room entries you have already read, because those are real and invented ones sound invented.
The wrap. Keep the last thirty seconds for one sentence: the two risks you would worry about most, and the first thing you would actually build. That sentence is often the only thing written down verbatim by the panel.
Rehearse designs with a timer and a hard rule that the last fifteen minutes are reserved, since this failure is a budgeting failure long before it is a knowledge failure. Write your evaluation and monitoring section first in written design docs, then go back and fill in the model, which forces the weak part to be the well-rested part. Keep a personal list of five failure stories drawn from your own projects and from War Room entries, each in the what-breaks, who-notices, what-happens shape, and reuse them across cases. Grade every practice design out of twelve with two marks reserved for step 6, and refuse yourself both marks when the section is a single word like "monitoring". Before any round, ask yourself the three questions in the diagnosis and answer them aloud for your own capstone project; if that is hard on a project you built, it will be impossible on a design you invented forty minutes ago.
Q: "You have described the model. How would you know, three months after launch, that it is still doing its job?" Strong answer: name the offline metric and its rupee twin, then the four daily monitors, then the label-delay problem and the proxy you would watch until real labels arrive, then one concrete failure story with the fallback. Weak answer: "I would monitor accuracy and retrain periodically." The tell is the label delay — a candidate who has run a model knows the score cannot be computed on schedule, and a candidate who has not assumes it can.