How an LLM works at inference
Wednesday, 3 pm. Kavya joins a Jaipur fintech that answers loan queries with an AI assistant. A customer asks about an RBI rule, and the assistant replies with a circular number that does not exist. Her manager asks why the model lied. Kavya freezes; she has used these tools for a year without knowing what happens inside. Her senior smiles: it never lied, because it never knew. Today you open the box and watch a model think, token by token.
- Every module in Phase 3 — APIs, prompts, RAG, agents — stands on the mental model you build here.
- Freshers who know why outputs vary and why facts go wrong can debug LLM features; others raise bugs against normal behaviour.
- Interview panels love this topic: "why does it hallucinate" separates users of AI from engineers of AI.
An LLM feels magical until you watch it work. Training happened months before you arrived: a company tuned billions of internal numbers on huge piles of text. What you use daily is inference — running that finished model to produce output. During inference the model learns nothing and looks nothing up. It performs one small trick, over and over, very fast.
The model does not read words. It reads tokens — a token is a small chunk of text, usually a piece of a word. In English, one token is roughly three-quarters of a word. Common words stay whole, while rare names shatter into fragments. Providers bill by the token, and every model limit is measured in tokens, so this tiny unit runs the economics of your future job.
Here is the whole trick. The model takes every token so far and scores every token in its vocabulary: how well would each one continue this text? It picks one, appends it, and repeats until the reply is done. That loop is why answers stream onto your screen piece by piece. The model does not know its last sentence while it writes its first.
Now the mental model that separates safe engineers from careless ones. The wrong way treats the model as a knowing database: ask a question, receive a stored fact. The right way treats it as a next-token predictor: ask a question, receive the most plausible continuation. Plausible and true overlap often, but nobody checks the overlap for you. Every output is a draft until a person or a test verifies it.
One dial shapes the picking step: temperature — a setting that controls how bold the pick is. At temperature 0 the model takes the top-scoring token every time, so one prompt gives one reply. Raise it and the model samples among several likely tokens, trading repeatability for variety. Two runs of one prompt can differ, and that is a setting, not a fault.
The model's working memory is the context window — the maximum number of tokens it can consider at once. Your instructions, the chat so far, any pasted files, and the reply all share that budget. When a chat outgrows it, the oldest parts get trimmed without warning, and early instructions vanish. Between chats the model keeps nothing; the app re-sends the transcript each time.
Two limits follow from all this. A hallucination — fluent, confident output that is false — happens because the loop rewards plausible text, not verified fact. The model is not lying, because lying needs a truth it chose to hide, and no truth check exists anywhere in the loop. The second limit is the training cutoff — the date where its training data ends. Ask about anything after that date and the model must refuse, guess, or call an outside tool.
You can hold this whole loop in your hands today. Ollama is a free tool that runs small open models on an ordinary laptop, with no GPU and no account. A hosted API serves a far larger model from a provider's servers: more capability and no local load, but your text leaves your machine. Both run the same loop you are about to watch.
Watch the full flow once before you try it. Everything below runs offline on an ordinary 8GB laptop — download Ollama from ollama.com, then follow along.
Step 1 — run a real model on your own machine. Pull a small model (about 1.3GB, one-time download) and talk to it:
ollama pull llama3.2:1b
ollama run llama3.2:1b
>>> Describe Jaipur in one sentence.
Jaipur, the capital of Rajasthan, is known as the Pink City for its
rose-hued historic buildings and vibrant markets.
Watch the reply appear in pieces, not all at once. That is the loop from the idea section running live: score, pick, append, repeat. Type /bye to exit. If the pull is slow on hostel Wi-Fi, qwen2.5:0.5b is an even smaller stand-in.
Step 2 — temperature 0: the same prompt, the same reply. Ollama also serves a local HTTP endpoint, so Python can call it with no internet and no API key:
import requests
def ask(prompt, temp):
r = requests.post("http://localhost:11434/api/generate", json={
"model": "llama3.2:1b",
"prompt": prompt,
"options": {"temperature": temp},
"stream": False,
})
return r.json()
for run in range(2):
print(run, ask("Name three uses of a drone.", 0)["response"][:70])
Both lines print the same text, character for character. At temperature 0 the model always takes the top-scoring token, so the loop becomes fully repeatable. This is the setting teams choose for pipelines and tests.
Step 3 — temperature 1: the same prompt, different replies. One number changes, everything else stays:
for run in range(3):
print(run, ask("Name three uses of a drone.", 1.0)["response"][:70])
Now the three lines differ — different openings, different orderings, sometimes different uses. The model is sampling among likely tokens instead of always taking the top one. Nothing is broken; variation at high temperature is the designed behaviour, and WR-024 covers the panic it causes in teams that never learned this.
Step 4 — count the tokens you are paying for. The response object reports exactly how many tokens went in and came out:
data = ask("Explain UPI in one line.", 0)
print(data["prompt_eval_count"], "prompt tokens in")
print(data["eval_count"], "reply tokens out")
print(len(data["response"].split()), "words in the reply")
Typical output: 31 prompt tokens in, 27 reply tokens out, 21 words in the reply. Notice the reply used more tokens than words — pieces, not words, is the model's real unit. Hosted APIs bill on these two counts, and the context window is spent from the same numbers, so engineers read them the way drivers read a fuel gauge.
Step 5 — probe the knowledge boundary. Ask the model about itself and about the recent past:
ollama run llama3.2:1b
>>> What is your training cutoff? Who won the most recent cricket
World Cup, and how sure are you?
My knowledge goes up to late 2023. The most recent Cricket World Cup
was won by Australia, who defeated India in the 2023 final...
The model answers from frozen training data and cannot see anything after its training cutoff. Ask about last month and it will guess in the same confident tone, because tone comes from the loop, not from truth. Small models fabricate more than large ones, which makes them a perfect hallucination lab for the exercises below.
Do these inside your ai-lab environment with Ollama installed. Each exercise ends with a check — you are done only when the check passes.
Your own local model. Install Ollama, pull llama3.2:1b (or qwen2.5:0.5b on a slow connection), and chat for ten minutes: one factual question, one creative request, one question about your own city. Then rerun one prompt with ollama run llama3.2:1b --verbose and read the statistics it prints after the reply. Check: ollama list shows your model, and a notes file records the prompt-token and reply-token counts from the verbose output plus one sentence on why the reply streamed piece by piece.
Temperature lab. Create temp_lab.py using the ask() helper from the watch section. Run one fixed prompt three times at temperature 0 and three times at temperature 1.0, writing all six replies to temp_results.txt with labels. Then add two sentences at the bottom: which setting fits a bank FAQ bot, which fits a slogan generator, and why. Check: the three temperature-0 replies are identical, at least two temperature-1 replies differ from each other, and your two recommendation sentences name repeatability as the deciding factor.
Induce a hallucination on purpose. Build hallucination_log.md with a three-column table: prompt, model claim, verdict. Push the model where fabrication clusters: ask for two published papers with authors and journals on an obscure topic, the founding year and founder of an invented company such as "Jaipur Kirana Logistics Pvt Ltd", and the result of a sports event after its training cutoff. Verify every claim with a web search on your phone or browser. Check: the table documents at least three fabricated claims with evidence they are false, plus one closing sentence explaining why this is prediction failing, not lying — this log is your WR-023 vaccine.
Overflow the context window. Create overflow.py that calls the local endpoint with "options": {"num_ctx": 512, "temperature": 0} — a deliberately tiny window. Send a prompt that starts with the rule "Answer in exactly three words." followed by 600+ words of pasted text (any long article), then the question "What language is Python named after?". With the window this small, part of the prompt cannot fit, and which part gets trimmed depends on the runtime you are on. Rerun with the same rule moved to the end, after the pasted text, and compare the two replies. Check: overflow.py prints both transcripts, the rule is obeyed in one position and broken in the other — the rule's position changed the outcome — and two sentences in your notes connect this to WR-022.
Three failures cover most first-month LLM pain. Each has a War Room entry with the full diagnosis path:
- WR-022 — the model forgot the instruction you gave at the start. A long chat outgrew the context window, the oldest tokens were trimmed silently, and the rule from message one stopped existing.
- WR-023 — a confident fake citation reached your manager. Fluent output was shipped without verification; hallucination clusters exactly on the obscure, specific facts you cannot check at a glance.
- WR-024 — same prompt, different answer, panic in standup. Temperature and sampling make variation normal; repeatability is a setting you choose, not a promise the model broke.
An LLM at inference runs one loop: score every possible next token, pick one, append it, repeat until done. It reads and writes tokens, and its working memory is a fixed context window that trims the oldest content silently when full. Temperature controls the picking step, so identical replies at 0 and varied replies above it are both settings, not moods. A hallucination is plausible text with no truth check behind it, and the training cutoff caps what the model can know at all. Treat every output as a draft, and verify any fact before it travels beyond your screen.
- Bank and UPI support assistants in India run behind verification and retrieval layers, because one invented rule number in a customer reply becomes a compliance incident.
- GCCs and hospitals with strict data rules run small open models on their own machines — the same Ollama pattern from this lesson, scaled up, so records never leave the building.
- Logistics and IRCTC-scale chat systems pin temperature low and cap context per query, since a support bot that answers differently each retry destroys user trust fast.
Answer these before opening the gate assessment. Each question names its type.
type=output-prediction— You call a local model twice with one prompt at temperature 0, then twice more at temperature 1.0. Predict the pattern across the four replies and name the mechanism that produces it.type=interpretation— A response reportsprompt_eval_count: 3480andeval_count: 92, and a teammate says "the model wrote 92 words". Interpret what each count measures and what the teammate got wrong.type=debug— Deep into a long support chat, the assistant stops following the "reply in Hindi" rule set in message one. Name the most likely cause and the cheapest fix that restores the rule.type=scenario-judgement— A teammate pastes a model-provided RBI circular number into a report for a manager, arguing "the model sounded certain". Give your judgement and the one-sentence reason you would offer.type=trade-off— A small local model through Ollama versus a hosted large-model API for a startup prototype handling customer KYC text: give one concrete advantage and one concrete cost of each.
Explain to a friend who thinks the chatbot "knows" things: what actually happens between hitting send and seeing a reply, and why the model sometimes invents facts without lying. Use at most 4 sentences, no jargon beyond the word "token". If your friend stops saying "the AI told me so", you own this module.