One class is suspiciously perfect. The confusion matrix shows 1.00 accuracy on it while every other class sits in the eighties, and nobody questions the good news. The model then meets photos taken by a different camera, or on a different shift, or after the label printer was moved, and that class collapses — 0.005 in the measured example, meaning it is now almost never recognised. Worse, unrelated images that happen to share the same lighting or watermark are confidently given the perfect class. Nothing in training or evaluation ever printed a warning.
Any image dataset collected in batches rather than at random: defect photos shot class by class on a factory line, KYC scans from one branch's scanner, crop photos taken on different days, medical images where one machine served one patient group. The birth is almost always logistics, not carelessness — the good parts were photographed on Monday under tube lights and the defective ones on Tuesday near a window, or one supplier's images carry a corner watermark. Measured demonstration from 2.5: paint a faint 3 by 3 patch of value 0.55 into the top-left corner of every class-6 FashionMNIST image and retrain. The marked test set reports 0.9305 overall and 1.0000 on class 6. The clean test set reports 0.8315 overall and 0.0050 on class 6. Move the same mark onto class-0 images and all 174 of them are called class 6. Linked modules: 2.5, 1.7 (leakage), 1.5 (per-class metrics), 4.8 (monitoring for drift).
This is 1.7's leakage wearing image clothes. A network minimises loss by the cheapest route available, and a three-pixel watermark is far cheaper to learn than the shape of a shirt. If a shortcut feature is perfectly correlated with the label in your data, gradient descent will find it and stop looking. The tell is not in the aggregate score, which looks fine, and it is not in the validation curve, which also looks fine — because the shortcut is present in the validation split too. Three checks find it. First, per-class metrics: a class far above the rest is a suspect, not a success. Second, look at the images — open a grid of correctly classified examples of the perfect class and a grid of its rare mistakes, and ask what is the same in one grid and different in the other. Third, the ablation that settles it: cover or remove the suspected region — blank the corner, crop the border, grey out the background — and re-score. If accuracy on that class falls off a cliff, the model was reading the background. The strongest version of this test is the one above: paste the suspect feature onto a different class and see whether the prediction follows it.
Fix the data first, because no architecture repairs this. Control the capture conditions: same camera, same lighting, same background, same distance, and shoot all classes in the same session wherever possible. Strip identifiers that ride along with a class — crop watermarks and timestamps, and remove borders. Where you cannot recapture, break the correlation deliberately: photograph a subset of every class under each condition, or apply augmentation that randomises the shortcut (colour jitter for lighting, random crops for borders). Then rebuild the splits so that a whole capture session sits entirely inside one split rather than being sprinkled across all three, which is 1.7's grouped split in image form. Re-score after each of these, and keep a held-out set shot on a genuinely different day as the only number you trust before shipping.
Never let a per-class score of 1.00 pass a review unquestioned; treat it the way 2.4 taught you to treat a training loss near zero. Print per-class accuracy and a confusion matrix on every image project, not overall accuracy alone. Look at misclassified images with your own eyes at least once per project — it takes ten minutes and catches what no metric will. Ask, before collection begins, one question that saves weeks: "what else changed when the class changed?" And write the capture conditions into the dataset's README, because the next person will otherwise assume the photos are comparable.
Q: "Your model gets 100 percent on one class in testing and fails on that class in production. What happened, and how do you prove it?" Strong answer: name shortcut learning and say why aggregate accuracy and the validation curve both hide it — the shortcut is present in every split. Then give the diagnosis path in order: per-class metrics, then look at the images, then an ablation that removes the suspected region, then the decisive test of pasting the feature onto a different class. Then give the fix as a data fix — controlled capture, stripped identifiers, grouped splits by capture session — and say plainly that no change of architecture would help. Bonus: name it as the same failure as target leakage in a table, and mention the held-out set collected on a different day as the only honest final score.