You run the exact same prompt twice and get two different replies — different wording, different ordering, sometimes a different conclusion. A teammate declares the model "broken" or "random"; a test that compares output strings fails on every second CI run; a demo that worked this morning says something else in front of the client. Panic follows, because everyone's mental model says computers repeat themselves.
First weeks of working with LLM output in code: string-equality tests against model replies, demo scripts, batch jobs that classify or extract, side-by-side comparisons of "the same" run. Appears with hosted APIs and local models alike, whenever nobody has looked at the sampling settings the calls are using — most SDK and app defaults leave temperature well above zero. Linked module: 3.1 (How an LLM works at inference).
Nothing is broken; the variation is designed. At each step of generation the model produces scores for every possible next token and then PICKS one. Temperature controls that pick: at 0 the model takes the top-scoring token every time, so one prompt yields one reply; above 0 it samples among the likely tokens, so runs diverge — and one early different pick compounds, because every later token continues a different text. Related knobs (top-p, top-k) shape the same choice. Confirm it in two minutes on a local model: call the endpoint three times at temperature 0 — identical replies — then three times at 1.0 and watch them diverge. If replies still vary at temperature 0, look for other moving parts: a changed system prompt, injected timestamps, conversation history riding along in the request, or a provider-side model update.
Decide what the task needs, then set the knob on purpose. Pipelines, extraction, classification, and anything with tests: temperature 0 (plus a fixed seed where the runtime supports one) for repeatable output. Creative generation and brainstorming: keep temperature up, and treat each run as a fresh draw by design. Fix the failing tests too — even at temperature 0, assert on meaning rather than exact prose: parse the JSON, check the extracted fields, or score the answer, instead of comparing whole reply strings.
Set sampling parameters explicitly in every API call your code makes; never inherit a default you have not read. Record temperature and model version alongside any output you save, so "same prompt" claims can be audited later. In code reviews, flag any string-equality assertion on model output. And teach the team the one-line summary before the first panic, not after: variation at high temperature is a setting, not a fault.
Q: "QA reports that your LLM feature returns different answers for identical requests. Bug or not?" Strong answer: explain next-token sampling and temperature in two sentences, state that determinism is a configuration choice (temperature 0, seed where supported) and name its cost — repetitive, lower-variety output that is wrong for creative tasks. Then go one level deeper: even deterministic settings do not license string-equality tests, so evaluate meaning, not prose. Naming top-p as a sibling knob and provider model updates as a hidden variable marks you as someone who has actually shipped this.