An extraction pipeline ran clean for eight days. Then the downstream code started
throwing KeyError: 'issue' and json.decoder.JSONDecodeError — first once a
night, then dozens of times. The raw replies show the shape wandering: keys renamed
(issue became category), values re-cased (Late-Delivery), a markdown fence
around the JSON, sometimes a helpful sentence before it. Every HTTP status was 200,
and nobody edited the code.
A triage or extraction prompt written in a hurry: it asks for JSON politely ("respond in JSON with the issue, awb and refund fields") but never pins exact keys, allowed values, or a missing-data rule. It was tested on a handful of tidy inputs. Real traffic then widened — longer complaints, Hinglish, odd punctuation — and on hosted APIs the provider may also have rolled the model version underneath. Linked module: 3.3 (Prompt engineering as engineering).
The format was requested, not specified. A loose ask leaves the model free to pick key names, casing, and chatter each time, and it picks differently as inputs vary. Three forces stack: (1) no format spec — nothing in the prompt names the exact keys and allowed values, so nothing anchors them; (2) input variety — unusual inputs pull the reply toward unusual shapes, which is format drift; (3) silent model updates — a hosted model swap changes default habits overnight, and a prompt calibrated on the old model drifts on the new one. Confirm by diffing a week-one raw reply against a failing one: the differences are in shape, not content — the model still understood the task.
Rewrite the prompt with a real format spec: exact keys, allowed values listed one by one, a missing-data rule ("use null, never guess"), and a chatter ban ("no text before or after the JSON"). Add one few-shot example showing the exact shape. Turn on the platform's JSON mode where it exists (module 3.2). Keep the module 3.2 validator with one re-ask, so a stray drifted reply is caught and retried instead of crashing the run. Then collect the failing inputs from the logs into your eval set and rerun it — do not redeploy until the score clears your bar.
Write the format spec on day one, before the first demo — a spec added after launch is written from an incident report. Keep a 6–10 case eval set beside the prompt and run it on every prompt edit and every model version change. Pin the model version where the vendor allows it, and treat a version bump as a change that must pass evals. Log every raw reply with the prompt version name so a drifted output can be traced in minutes, not hours.
Q: "Your LLM extraction worked for a week, then the output format started changing. Walk me through it." Strong answer: name format drift, explain that a polite ask is not a spec, list the three drivers (loose spec, input variety, silent model updates), and give the layered fix — format spec plus JSON mode plus validator plus eval set. Bonus signal: mention that the failing inputs became new eval cases, which shows you turn incidents into tests.