A pipeline that ran clean for days dies overnight with
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0). The
log shows the model's raw reply beginning "Here is the JSON you requested:" or
wrapped in a markdown code fence — sometimes valid JSON with an unexpected
field, sometimes single quotes instead of double. Every HTTP status was 200.
An extraction or classification script: prompt says "respond in JSON", code feeds
response_text straight into json.loads(). It survives testing because the
model complies most of the time — then row 1,371 of a night run produces a chatty
reply and the whole queue stops. Common in document pipelines, ticket triage, and
expense extraction. Linked module: 3.2; module 3.3 deepens output contracts.
Asking nicely in the prompt is a probability, not a contract. A model is a
next-token predictor, so "always reply in JSON" fails at some rate — with prose
prefixes, markdown fences, trailing commentary, single quotes, or the right JSON
with wrong fields. Two facts matter. First, the failure arrives inside a 200:
transport succeeded, content failed, so status-code checks catch nothing. Second,
the crash point (json.loads) is far from the cause (an unconstrained request),
which is why it looks random. Print the raw reply for a failing row and classify
the malformation — fenced, prefixed, quoted, or schema-wrong — before touching
any code.
Add three layers, in this order. (1) Constrain: turn on the vendor's JSON mode or
structured-output setting ("format": "json" on Ollama's endpoint; hosted
providers accept a JSON schema) so the decoder cannot emit prose. (2) Validate:
after json.loads, check the fields your code depends on — types present, values
inside the allowed set. (3) Repair: on any failure, re-ask once, appending the bad
output and the validation error to the messages; if it fails again, write the row
to a quarantine file and continue — one bad row must never kill the queue. Wrap
the whole thing in try/except around the parse, never bare.
Never pass a model reply to json.loads without a try/except and a field check
— make that a personal lint rule. Test the pipeline with deliberately messy
inputs before trusting it overnight. Log raw replies on every failure so diagnosis
takes minutes, not a reproduction hunt. Track the re-ask rate as a health metric:
a rising repair rate is your earliest warning that a prompt, model, or input
distribution changed.
Q: "Your LLM pipeline crashed overnight on a JSONDecodeError. Diagnose it and redesign it." Strong answer: name prompt-as-hope versus mode-as-contract, note that the failure ships inside a 200 so status checks are blind to it, then lay out constrain → validate → re-ask → quarantine. Bonus signal: proposing the re-ask rate as a monitored metric and messy-input tests in CI, which shows you plan for the failure instead of hoping it away.