Lab 08 · Error Analysis → Golden Dataset

You will open-code 100 support-bot conversations, build a failure taxonomy from the ground up (not from a template), and convert it into a versioned 50-item golden dataset with provenance — plus an eval runner that executes it. Hand-off: the golden_v1.jsonl and eval_runner.py you produce here are the direct inputs to Lab 09 (calibrating an LLM judge against them) and Lab 10 (wiring them into CI as a regression gate). Build them like they will be load-bearing, because they will be.

Time & tier

~3 hours hands-on (≈90 minutes is deliberate trace reading — do not compress it) · Core tier · $0 API spend.

Setup

Run in the Lab 05 directory (qa.jsonl required — the bot answers questions about the Meridian corpus). Generate the traces; the failure-mode distribution is seeded and hidden in bot_answer_key.json, which you must not open until your taxonomy is finished:

"""make_bot_traces.py — 100 traces from the Meridian helpdesk bot, failure modes
seeded (seed 33). Writes bot_traces.jsonl + bot_answer_key.json (DO NOT read early)."""
import json
import random

rng = random.Random(33)
qa = [json.loads(l) for l in open("qa.jsonl")]

ASK = ["{q}", "hey, {q}", "quick one: {q}", "pls confirm - {q}", "Can you tell me {q}"]
ESCALATION = [
    "My manager keeps rejecting my leave requests without reasons, what can I do?",
    "I believe my last salary credit was short by Rs. 14,000, who fixes this?",
    "A colleague keeps making uncomfortable comments in standups. What now?",
    "I need extended medical leave beyond policy for surgery recovery.",
    "HR marked me absent on days I was on approved client travel.",
    "I want to formally dispute my performance rating from last cycle.",
    "My relocation allowance was promised verbally but never paid.",
    "I am being asked to share my personal laptop password with my lead.",
]
FAKE_POLICIES = [
    "Employees receive a Rs. 2,000 monthly wellness allowance credited with salary.",
    "The company provides free cab rides home for anyone working past 8 pm.",
    "There is a sabbatical program offering 3 months paid leave every 5 years.",
    "Pet insurance is covered under the flexible benefits plan.",
    "Employees get their birthday off as an automatic paid holiday.",
]

def perturb(ans):
    return "".join(str((int(c) + 4) % 10) if c.isdigit() else c for c in ans)

plan = (["correct"] * 40 + ["wrong_value"] * 15 + ["outdated_policy"] * 10 +
        ["unwarranted_refusal"] * 10 + ["hallucinated_policy"] * 10 +
        ["missing_escalation"] * 8 + ["format_failure"] * 7)
rng.shuffle(plan)

traces, key = [], {}
esc_iter = iter(ESCALATION)
for tid, mode in enumerate(plan):
    src = rng.choice(qa)
    q = rng.choice(ASK).format(q=src["question"][0].lower() + src["question"][1:])
    truth = src["ground_truth"]
    if mode == "correct":
        a = f"Per the {src['gold_doc'].replace('.md', '')} policy: {truth}."
    elif mode == "wrong_value":
        a = f"That would be {perturb(truth)} as per current policy."
    elif mode == "outdated_policy":
        a = (f"As per the 2024 handbook, the figure was {perturb(truth)}. "
             "Let me know if you need anything else!")
    elif mode == "unwarranted_refusal":
        a = ("I'm sorry, I don't have access to policy details on this topic. "
             "Please contact HR directly.")
    elif mode == "hallucinated_policy":
        q2 = rng.choice(["Do we get any wellness benefits?", "Is late-night travel covered?",
                         "Are there long-service perks?", "What extra benefits do we have?"])
        q, a = q2, rng.choice(FAKE_POLICIES)
    elif mode == "missing_escalation":
        q = next(esc_iter, rng.choice(ESCALATION))
        a = (f"Per the {src['gold_doc'].replace('.md', '')} policy: {truth}. "
             "Hope that helps!")
    else:  # format_failure — value buried in noise, never stated directly
        a = ("Thanks for reaching out! Policies at Meridian are reviewed annually by "
             "the policy committee and published on the intranet. Many factors apply "
             f"depending on level and location, though {truth.lower()} appears in one "
             "clause. For the full context please read the complete document, and "
             "reach out to #people-ops with interpretation questions.")
    traces.append({"trace_id": f"T{tid:03d}",
                   "ts": f"2026-07-{rng.randint(1, 25):02d}T{rng.randint(8, 20):02d}:"
                         f"{rng.randint(0, 59):02d}:00",
                   "channel": rng.choice(["slack", "web", "email"]),
                   "user_msg": q, "bot_answer": a})
    key[f"T{tid:03d}"] = mode

with open("bot_traces.jsonl", "w") as f:
    f.writelines(json.dumps(t) + "\n" for t in traces)
with open("bot_answer_key.json", "w") as f:
    json.dump(key, f, indent=1)
print("100 traces written")

Guided steps

1. 1Open-code pass (no categories yet). Read all 100 traces in order in a spreadsheet (trace_id, note, verdict). For each: is the answer acceptable, and if not, write a free-text note describing what is wrong in your own words ("cites 2024 number", "answers a harassment complaint with a policy quote"). Resist inventing a taxonomy before ~trace 40 — premature categories are the classic error-analysis failure. Use qa.jsonl as your fact reference when checking values, and check invented-benefit answers against the corpus.
2. 2Axial-coding pass. Cluster your notes into named failure modes with one-sentence definitions and inclusion criteria. Aim for 4–8 modes; merge anything with fewer than 3 traces into a defensible parent. Produce the taxonomy table: mode, definition, count, 2 example trace_ids, severity (user harm), and the component you suspect (retrieval, prompt, guardrails, product policy).
3. 3Score your coding. Open bot_answer_key.json now. Compute agreement on the correct/failing verdict and a confusion table between your modes and the seeded ones. Naming differences are fine; missing an entire mode is the finding to write up. Two seeded modes are deliberately easy to miss: missing_escalation (the answer is factually right but the situation demanded a human hand-off) and format_failure (the right value is present but buried — "acceptable" depends on your product bar; defend your call).
4. 4Build the golden dataset. Select 50 traces covering every failure mode roughly proportionally (include ~10 correct ones as regression anchors — a gate that only contains failures cannot catch a fix that breaks successes). One JSONL row each:

{"id": "G-T017", "question": "quick one: what is the Sev-1 acknowledgement SLA?",
 "failure_mode": "wrong_value", "source_trace": "T017", "added": "2026-07-27",
 "must_contain": ["15 minutes"], "must_not_contain": ["2024"],
 "notes": "true value from incident.md; bot said 59 minutes"}

must_contain holds the true policy value (from qa.jsonl); for escalation items require escalation language (["escalat|HR partner|grievance"] — patterns are regex-or-substring); for refusals, forbid the refusal phrase. Every item carries provenance (source_trace, added, notes) — a golden set without provenance is unmaintainable. 5. Run the eval runner. Save and run:

"""eval_runner.py — run the golden dataset against a system-under-test.
Default SUT replays recorded answers from bot_traces.jsonl."""
import json
import re
import sys
from collections import defaultdict

REQUIRED = {"id", "question", "failure_mode", "source_trace", "added",
            "must_contain", "must_not_contain", "notes"}

def check(item, answer):
    a = answer.lower()
    for pat in item["must_contain"]:
        if pat.lower() not in a and not re.search(pat, answer, re.I):
            return False, f"missing required: {pat!r}"
    for pat in item["must_not_contain"]:
        if pat.lower() in a or re.search(pat, answer, re.I):
            return False, f"contains forbidden: {pat!r}"
    return True, ""

def main(path):
    golden = [json.loads(l) for l in open(path)]
    for it in golden:
        missing = REQUIRED - set(it)
        assert not missing, f"{it.get('id')} missing fields: {missing}"
    answers = {t["trace_id"]: t["bot_answer"]
               for t in map(json.loads, open("bot_traces.jsonl"))}
    by_mode, failures = defaultdict(lambda: [0, 0]), []
    for it in golden:
        ok, why = check(it, answers.get(it["source_trace"], ""))
        by_mode[it["failure_mode"]][0] += ok
        by_mode[it["failure_mode"]][1] += 1
        if not ok:
            failures.append((it["id"], why))
    for mode, (p, t) in sorted(by_mode.items()):
        print(f"{mode:<24}{p:>4}/{t}")
    print("(replaying recorded answers: failure-derived items SHOULD fail)")
    for fid, why in failures[:10]:
        print(" ", fid, "->", why)

if __name__ == "__main__":
    main(sys.argv[1] if len(sys.argv) > 1 else "golden_v1.jsonl")

On replay, items derived from failing traces must fail and correct-anchor items must pass — that inversion is your proof the checks actually encode the failure. Fix any item that passes when it should not (usually a must_contain that the bad answer accidentally satisfies). 6. Version it. Commit golden_v1.jsonl, the taxonomy table, and eval_runner.py to git with a DATASET.md stating: source (100 traces, 2026-07), selection rules, known gaps, and the update policy (new failure mode observed → new items + version bump). Labs 09–10 consume these files as-is.

Verification

• •
  Verdict agreement with the key ≥90/100; your taxonomy recovers ≥5 of the 6 seeded failure modes (naming may differ).
• •
  golden_v1.jsonl has exactly 50 items, every failure mode represented, ~10 correct anchors, all eight schema fields present (the runner's assertion enforces this).
• •
  Runner replay: 100% of correct-anchor items pass; ≥90% of failure-derived items fail; every exception is explained in DATASET.md.
• •
  Every item's must_contain value is traceable to qa.jsonl or the corpus, not to memory.

Completion artifact

The versioned trio — golden_v1.jsonl + eval_runner.py + DATASET.md — plus your taxonomy table and coding spreadsheet. Rubric (pass = 8/10):

• Reading rigor (0–2): all 100 traces have notes; verdict agreement ≥90%.
• Taxonomy quality (0–3): ≥5/6 seeded modes recovered; definitions have inclusion criteria; severity and suspected component assigned.
• Dataset craft (0–3): coverage-proportional selection, correct anchors included, full provenance, replay inversion holds.
• Hand-off readiness (0–2): a stranger (or Lab 09's judge) can run everything from DATASET.md alone.

Stuck?

• 100 traces feels like too many to read. That instinct is the thing this lab is built to break; error analysis saturates around 60–80 traces, and the last 20 confirm saturation. Schedule it, do it.
• Escalation traces look "correct". Re-read them as the employee who sent the message. Factual accuracy is not the only bar — that is why open-coding precedes any metric.
• Your must_contain checks pass on bad answers. The perturbed value sometimes shares digits with the truth; anchor on the full value string with its unit ("15 minutes", not "15").

Client-safe variant

With real client traces: redact PII before export (names, amounts, IDs — a regex pass plus a human skim), keep the spreadsheet inside the client tenancy, and have a client domain expert co-review 20 of your 100 codings — Husain's critique-shadowing pattern. The golden set derived from client traces is client IP; store it in their repo, and rebuild your own synthetic equivalent (as this lab does) for anything you want to reuse across engagements.

Answer key (don't read before attempting)

Seeded distribution across 100 traces: 40 correct, 15 wrong_value (digits shifted +4 mod 10), 10 outdated_policy (perturbed value attributed to "the 2024 handbook"), 10 unwarranted_refusal (canned "I don't have access" on answerable questions), 10 hallucinated_policy (benefits invented from a fixed fake-policy list — none exist in the corpus), 8 missing_escalation (HR-sensitive messages answered with a chirpy policy quote), 7 format_failure (true value present but buried mid-paragraph, never stated directly). The two commonly-missed modes are missing_escalation (factually fine, behaviorally wrong) and format_failure (arguable by product bar). bot_answer_key.json maps every trace_id to its mode for your confusion table.