Evaluating LLM systems
Monday morning, Madurai. Sameer maintains the loan-FAQ bot at a gold-loan finance company. On Friday a teammate improved the prompt, tried one favourite question, and shipped. Over the weekend the bot quoted the wrong interest rate to real customers — while the demo question kept working perfectly. His manager asks one thing: which change broke it, and why did nobody notice for two days? Sameer has no answer, because "it looked good" was the only test. Today you build the test that would have caught it.
- Interview panels for every AI role now ask "how do you know your system works" — and a golden set with scores is the expected answer, not a demo story.
- Prompt edits and model swaps break LLM systems silently, and the person who can measure the break is the person trusted to make the call.
- You already built the pieces — 3.3's mini eval loop and 3.5's grounding audit — and today they become one harness you will reuse in every module after this.
Friday's demo checked one question, once, with a friendly eye. Evaluation checks a fixed list of questions, every time, against written pass rules. That difference decides whether you notice a break before your users do. Working teams keep a golden set — a fixed, versioned list of test cases with known correct answers that every change must face. Module 3.3's six-case eval set was a small one; this module grows the habit into a discipline.
A good set mixes three kinds of cases. Golden cases ask questions the document answers, with the expected facts written down. Edge cases push at the boundaries: Hinglish phrasing, typos, questions whose wording sits far from the document's own words. Adversarial cases are traps — questions the document does not answer, where the only correct output is the refusal you built in 3.5. A set with no traps measures nothing, because it can never catch a confident hallucination.
How do you mark each answer? The cheapest markers are deterministic checks: code that looks for exact strings, numbers, or regex patterns in the output. They cost nothing and never change their mind, but they only see wording. Semantic similarity scoring embeds the answer and the expected answer and compares meaning, which forgives wording but blurs facts. Above them sits LLM-as-judge — a second model call that grades an answer against a written rubric — and above everything sits a human reading transcripts.
A judge model is powerful and cheap, and it lies in known ways. Position bias makes it favour whichever answer it reads first when comparing two. Verbosity bias makes it reward long, confident answers even when the extra words hide errors. Self-preference makes a model rate its own family's writing above equally good text from another model. None of these show up in the score; they show up when a human checks.
So there is a wrong way and a right way to use a judge. The wrong way wires the judge in, watches high scores come back, and trusts the dashboard. The right way scores ten or twenty cases by hand first, runs the judge on the same cases, and reads every disagreement. Only when human and judge agree on nearly all of them does the judge earn the boring work at scale. An unaudited judge is not evaluation; it is vibes with a number attached.
A RAG pipeline needs two numbers, not one. Hit rate — the share of questions whose correct chunk appears in the top-k — measures retrieval alone. Groundedness — whether every claim in the answer is supported by the retrieved chunks — measures generation alone. One blended end-to-end score cannot tell you which stage to dig into. Teams have lost weeks tuning prompts while the real fault sat in retrieval, invisible inside a single mixed number.
The last habit is regression discipline. A regression — a case that passed before a change and fails after it — is the failure a demo can never catch. So the harness runs before every prompt edit and every model swap, and you compare results per case, not only the totals. Two totals can match while two different cases break underneath. Keep a holdout — a slice of cases you never tune against — because a set you optimise against for weeks slowly turns into a mirror.
Everything below runs offline on the module 3.5 setup: Ollama with nomic-embed-text and llama3.2:1b, plus your rag.py exposing chunks, retrieve(), and ask(). We grow one harness around it, file by file. If ask() still prints its retrieval log lines, leave them — scrolling logs beside verdicts is useful evidence.
Step 1 — the eval set lives in a file, not in your head. One case per line, in JSONL. Each case carries the question, the strings a correct answer must contain, and the chunk id that holds the answer:
import json
CASES = [
# golden - answered by the document, expected chunk known
{"id": 1, "kind": "golden",
"q": "How many casual leaves do I get in a year?",
"must": ["12", "casual"], "chunk": 0},
{"id": 2, "kind": "golden",
"q": "When does my notice period start?",
"must": ["notice", "30"], "chunk": 11},
# edge - real phrasing that sits far from the document's wording
{"id": 6, "kind": "edge",
"q": "Mujhe 3 din se bukhaar hai, certificate dena hoga kya?",
"must": ["doctor", "certificate"], "chunk": 3},
# adversarial - not in the document; only the refusal is correct
{"id": 13, "kind": "adversarial",
"q": "What is the office wifi password?",
"must": ["not in the document"], "chunk": None},
]
with open("cases.jsonl", "w") as f:
for c in CASES:
f.write(json.dumps(c) + "\n")
print(len(CASES), "cases written")
Output: 4 cases written. These four show the three kinds; your real file grows to fifteen — nine golden, three edge, three adversarial. Notice what a case is: a question, a checkable expectation (must), and a retrieval expectation (chunk). The adversarial case expects the exact refusal string from 3.5's RULES, so the trap is checkable by code.
Step 2 — deterministic checks first. evalrun.py runs every case through ask() and applies a contains/regex check:
import json, re
from rag import ask, retrieve, chunks
cases = [json.loads(line) for line in open("cases.jsonl")]
def check_contains(answer, must):
return all(re.search(re.escape(m), answer, re.IGNORECASE) for m in must)
def run(label):
results = {}
for c in cases:
answer = ask(c["q"])
results[c["id"]] = check_contains(answer, c["must"])
print(f"{label} case {c['id']:>2} {c['kind']:<12} "
f"{'PASS' if results[c['id']] else 'FAIL'}")
print(f"{label} total: {sum(results.values())}/{len(cases)}")
return results
v1 = run("v1")
With the full fifteen-case file, a typical run ends like this:
v1 case 1 golden PASS
v1 case 2 golden PASS
v1 case 6 edge FAIL
v1 case 13 adversarial PASS
v1 total: 12/15
Read the FAIL as information, not bad news — the Hinglish edge case is your next to-do, exactly like 3.3's failing case was. Also see what these checks cannot do: if the model answers "twelve casual leaves", the check for "12" fails an answer a human would pass. Deterministic checks are the bottom rung of the ladder: free, instant, and blind to meaning.
Step 3 — separate the stages. Before blaming any prompt, measure retrieval alone. The chunk field makes hit rate a five-line calculation:
hits, n = 0, 0
for c in cases:
if c["chunk"] is None:
continue
n += 1
top = [i for i, s in retrieve(c["q"], k=3)]
hits += c["chunk"] in top
print(f"retrieval hit rate: {hits}/{n}")
Output: retrieval hit rate: 10/12. This number never touches the generator — it asks one question per case: did the right chunk reach the top-3? Now your FAIL verdicts split cleanly. A failing case with a retrieval miss is an upstream problem (chunking, wording, corpus), and no prompt edit can save it. A failing case whose chunk arrived is a generation problem, and now the prompt is a fair suspect.
Step 4 — an LLM-as-judge for groundedness. Contains checks cannot decide whether a fluent answer stays inside the retrieved chunks, so we add a judge with a strict rubric. Note the shape: one binary question, evidence before verdict, no marks for style:
import requests
JUDGE_PROMPT = """You are a strict grader. Check ONE thing only:
is every factual claim in the ANSWER supported by the CONTEXT?
Wording may differ; facts may not. A refusal counts as grounded.
Do not reward length or confidence.
CONTEXT:
{context}
ANSWER:
{answer}
Reply with one JSON object only:
{{"evidence": "<quote from the context, or NONE>", "grounded": true or false}}"""
def judge(question, answer, k=3):
ctx = "\n".join(chunks[i] for i, _ in retrieve(question, k))
r = requests.post("http://localhost:11434/api/chat",
json={"model": "llama3.2:1b", "stream": False,
"messages": [{"role": "user", "content":
JUDGE_PROMPT.format(context=ctx, answer=answer)}]},
timeout=120)
r.raise_for_status()
try:
return json.loads(r.json()["message"]["content"])
except json.JSONDecodeError:
return {"evidence": "PARSE-FAIL", "grounded": False}
print(judge("How many casual leaves do I get?",
"You get 25 casual leaves plus free snacks [C0]."))
Output: {'evidence': 'NONE', 'grounded': False} — the planted lie is caught. Every design choice here fights a known bias: the binary verdict (no 1-to-10 scale) removes score drift, forcing evidence first makes the judge find proof before deciding, and "do not reward length" pushes back on verbosity bias. A parse failure counts as not grounded, because a verdict you cannot parse is not a verdict. One warning: on a 1B model this judge will still be lenient and sometimes wrong — which is why exercise 3 makes you audit it by hand before you trust its column in any table.
Step 5 — the regression run. A teammate now "improves" rag.py's RULES — say, "answer warmly in one friendly sentence" with the citation and refusal lines softened. Run run("v2") and put both runs side by side:
case kind v1 v2
1 golden PASS PASS <- the demo question
6 edge FAIL FAIL
9 golden PASS FAIL <- the number vanished from the answer
13 adversarial PASS FAIL <- refusal gone, the bot invented an answer
total 12/15 9/15
Case 1 — the question every demo uses — still passes, so a demo would ship this change. The table refuses to. Two regressions surface by id, each pointing at exactly what the "friendly" rewrite destroyed. This is the whole discipline in one artefact: run before the change, run after, compare per case, and let the diff make the ship decision. When the totals match but different cases pass, that is not a tie — that is one fix and one new break.
Everything you typed today is what eval tools automate. promptfoo, one common open-source choice, stores cases in a YAML file, offers contains/regex/similarity assertions plus judge rubrics, runs a whole prompt-by-model matrix in one command, and can block a merge in CI when the score drops. DeepEval does the same in pytest style. Reach for them when re-running by hand starts to hurt — the concepts stay exactly the ones in your harness.
Work in ai-lab with Ollama running and your 3.5 project (ragme.py, over the document you chose) working. Each exercise ends with a check — you are done only when the check passes.
A 15-case eval set for your own bot. For your 3.5 document, write cases.jsonl with fifteen cases: nine golden (spread across different sections of the document, expected chunk id recorded), three edge (one in Hinglish or your own first language, one with typos, one phrased in words the document never uses), and three adversarial (genuinely unanswerable — the refusal string is the must). Then write evalrun.py with the contains/regex checker and score your current bot. Check: the run prints fifteen per-case verdicts and a total; at least one case fails (if all fifteen pass, your set is too easy — make the edge cases harder and note which change finally produced a failure).
Two numbers, two stages. Add the hit-rate calculation and a groundedness column (use the judge from the watch section) to your harness, and produce one table in notes-3.7.md: case id, kind, hit (yes/no), contains verdict, judge verdict. Now sabotage retrieval the way exercise 3 of 3.5 did — replace top-3 with random chunks — and rerun. Check: the table shows the sabotage crashed hit rate and groundedness while your prompt never changed, plus two sentences: which stage a low hit rate points at, and why prompt tuning cannot fix a retrieval miss.
Audit the judge. Collect ten question-answer pairs from your bot: seven real outputs and three you edit by hand to plant errors (a wrong number, an invented rule, a claim from outside the document). Score all ten yourself against the rubric — grounded or not — before running the judge. Then run the judge on the same ten and compare. Check: notes-3.7.md records both columns, the agreement count out of ten, and at least one disagreement with your explanation of who was right and which judge weakness (leniency, verbosity bias, or a vague rubric line) explains it; if the judge agreed ten out of ten, plant a subtler error and repeat until you find a disagreement.
The regression drill. Pick two "improvements" to your prompt that sound reasonable: make the tone friendlier, and shorten every answer to one sentence. Before touching anything, record the v1 run. Apply one change, rerun all fifteen, and build the v1/v2 per-case table; repeat for the second change as v3. Verify your favourite demo question passes in every version. Check: the table shows at least one regression that the demo question missed, named by case id and kind, plus a three-line ship decision for each version: ship, fix first, or revert — with the failing case as your evidence.
Three failures cover most first-month eval pain. Each has a War Room entry with the full diagnosis path:
- WR-040 — the judge that blessed a wrong answer. An unaudited LLM judge scored a confident, verbose, wrong answer 9/10; ten hand-scored cases exposed it, and a binary evidence-first rubric plus a human audit is the exit.
- WR-041 — the eval set that rotted into a mirror. A team tuned against the same ten cases until the score hit perfect while production kept breaking; held-out cases and a monthly refresh from real failures end the self-deception.
- WR-042 — three weeks tuning the wrong stage. Prompt after prompt was rewritten while retrieval hit rate sat at 5/12 unmeasured; measuring the stages separately found the real fault in one afternoon.
Evaluation means a fixed set of cases with written pass rules, run on every change, instead of one admired demo. A strong eval set mixes golden cases, edge cases, and adversarial traps that only a refusal can pass. Cheap deterministic checks catch wording and format, while an LLM-as-judge grades meaning — but a judge earns trust only after you audit it against human labels. A RAG system needs hit rate and groundedness measured separately, so you always know which stage to fix. Run the harness before every prompt or model change, compare per case, and keep a holdout so the score stays honest.
- Banking and insurance GCCs in India gate every prompt change to customer-facing bots behind an eval run; a failing adversarial slice blocks the release the same way a failing unit test blocks a merge.
- UPI and fintech support teams grow their golden sets from real complaint tickets — every production failure becomes tomorrow's eval case, so the suite gets harder as the product grows.
- AI QA is now its own hiring categoryjob posts name eval harnesses, judge calibration, and regression suites, and interviewers ask candidates to walk through a set they built.
Answer these before opening the gate assessment. Each question names its kind.
type=output-prediction— Your deterministic check isall(re.search(re.escape(m), answer, re.I) for m in must)withmust=["12", "casual"]. The bot answers: "Every employee earns twelve casual leave days each year [C0]." Predict the verdict the harness prints, explain why, and state what this tells you about where contains checks sit on the marking ladder.type=debug— After you re-chunked your document, hit rate collapsed from 10/12 to 2/12, yet reading the retrieval logs shows the correct text arriving in the top-3 for most questions. Nothing else changed. Name the cause insidecases.jsonland the fix, and state what this teaches about eval sets as derived data.type=interpretation— A judge transcript reads:{"evidence": "NONE", "grounded": true}for a five-sentence answer that contradicts the document. Name the two separate problems this single transcript reveals, and the first action you would take before running this judge on all fifteen cases.type=scenario-judgement— A teammate says: "Ollama released a newer model, I tried our five usual questions, the answers look better — let's swap tonight." Give your judgement and the exact evidence you would demand first, naming what the five usual questions cannot show.type=trade-off— Grading your fifteen cases with contains/regex checks versus with the LLM judge: give one concrete gain and one concrete cost of each, and say which one belongs in the loop you run after every small prompt edit.
Explain to a friend who manages a kirana supply business and has never coded: your chatbot demo looked perfect, so why do you refuse to ship changes until a list of test questions with known answers passes — and why does one of those questions have no answer on purpose? Use at most 4 sentences, plain words only. If your friend can repeat back why "it looked good once" protects nobody, you own this module.