The service returns 200 on every call and the numbers look plausible — but they are wrong. Churn probabilities cluster oddly, approval rates drift, a downstream dashboard shows a distribution nobody recognises. No errors, no crashes, no alerts: the problem surfaces weeks later, when a business team asks why the model "stopped working", and every log line says everything succeeded.
A model served behind an endpoint that accepts a raw dict — await request.json()
fed straight into a DataFrame and the pipeline — with no declared input schema.
The one caller it was built with sent perfect payloads, so it always "worked". Then
a second team integrated: they sent income instead of monthly_income (silently
becoming NaN), amounts in thousands instead of rupees, and columns in their own
order. Pandas and sklearn are happy to compute on whatever arrives. This is the
serving-side twin of the schema drift in WR-060, and the exact failure module
4.2's pydantic contract exists to prevent. Linked modules: 4.2, 3.8.
Garbage in, confident garbage out: the model cannot know the units changed or a column went missing — it multiplies whatever numbers arrive. Reconstruct the failure by capturing one real payload from each caller and diffing them against the training frame's columns, dtypes and value ranges. The tell-tale signs: field names that differ from the training schema, magnitudes off by a constant factor (a units mismatch), NaNs appearing after the DataFrame build, or defaults filling fields the caller never sent. The deeper diagnosis is architectural: the service had no contract, so there was no moment at which a mismatch COULD fail. Every check the training code relied on lived in people's heads.
Put a contract at the door. Define the input as a pydantic model — exact field
names, types, realistic ranges (monthly_income: float = Field(gt=0, le=1_000_000)), a Literal for categorical values, and extra: "forbid" so a
misspelled field is a loud 422 instead of a silent default. Version the contract:
include a schema version in the response (and in logs), so "which shape were you
speaking" is always answerable — the discipline WR-060 established for stored
schemas, applied to the wire. Then re-run every known-bad payload from the
incident and confirm each now dies with a 422 naming the field. Backfill: identify
the period the wrong numbers flowed and flag those predictions to the consumers.
Never accept a raw dict where a schema can stand: reject-don't-guess is the default for anything a model will consume. Ranges should encode reality, not just types — a negative income is type-correct and still nonsense. Keep an attack-your-own-contract test file (module 4.2, exercise 3) of malformed payloads that must all fail cleanly; run it in CI when 4.6 arrives. When a new caller integrates, hand them /docs and require one round-trip against a staging endpoint before production traffic.
Q: "Your served model returned wrong predictions for three weeks with zero errors. How is that possible, and how do you make it impossible?" Strong answer: explain that an uncontracted service cannot fail on mismatched input — models compute on whatever numbers arrive — and name real mechanisms: renamed fields becoming NaN, unit changes, reordered or missing columns. The cure is a validated schema at the boundary (types, ranges, forbidden extras) plus a versioned contract, so bad input fails fast and loudly. Bonus signal: saying "a 422 at integration time is a gift to the caller" shows you have thought about whom validation protects.