The demo was flawless: the prompt answered the test complaint perfectly, five times in a row, and the feature shipped. Within two days support escalations pile up — wrong categories, invented parcel numbers, refunds flagged where none were asked. Spot-checking shows the prompt still aces the original demo input. On everything else, accuracy is closer to a coin toss.
A fresher (or a hurried team) iterated a prompt against one or two favourite examples until those looked perfect, then shipped. Common everywhere prompts are edited in a chat window: the same test input gets pasted after every tweak because it is on the clipboard. Often the demo input even appears inside the prompt as the few-shot example — so the "test" is asking the model to repeat its own example. Linked module: 3.3 (Prompt engineering as engineering).
The prompt is overfitted to its demo case, the prompt-engineering version of testing on your training data. Every edit was selected because it helped one input, so the prompt accumulated wording that works for that input's phrasing, length, and fields. Real traffic varies: Hinglish, missing AWB numbers, sarcasm, three complaints in one message. None of those ever touched the prompt during tuning, so its behaviour there is unmeasured — not bad luck, unmeasured. Confirm cheaply: collect ten real inputs from logs and score the prompt by hand. If the demo case passes and half the real ones fail, this is the trap.
Stop editing the prompt and build the measuring stick first: a 6–10 case eval set drawn from real traffic, not imagination — include the messy ones (mixed language, missing values, a non-complaint, an instruction-like line buried in the data). Write expected outputs for each. Score the current prompt to get an honest baseline, then iterate: fix the worst failure bucket, rerun all cases, repeat. Ship when the score clears the bar you set with your lead, and keep the demo case in the set — it should keep passing, it is only forbidden from being the whole test.
Adopt one team rule: no prompt ships on fewer than six eval cases, and the case inputs must come from real data. Never reuse a few-shot example as a test case. After launch, feed every real failure back into the eval set so the set grows harder as the product grows. In review, ask "what did this score, on which set" instead of "does it work" — the question alone changes behaviour.
Q: "Your prompt performs well in the demo and badly in production. Why, and what do you do?" Strong answer: name overfitting-to-the-demo-case, note the tell (the original input still passes), and lay out the eval-set discipline — build from real traffic, baseline before editing, iterate against the score, ship on a threshold. The phrase "the demo case was my training data" tells the panel you understand generalisation beyond LLMs.