Error Analysis First: 100 Traces to a Failure Taxonomy and Golden Dataset

Tier: Core · Workflow area: Evaluation Engineering

This playbook is the discipline that fixes it: read 100 real traces, open-code failure modes into a taxonomy, and convert what you find into a versioned golden dataset with provenance — the seed asset every other evaluation playbook in this track builds on.

When to use

• •
  An LLM feature is live (or in dogfood) and quality complaints are anecdotal — "it sometimes gets things wrong" with no structure.
• •
  Before building any automated eval: metrics chosen before error analysis measure what is easy, not what fails.
• •
  Quarterly, on any mature LLM product — failure distributions drift with users, data, and model versions.

Do not delegate this to a judge model. The first pass is human reading; automation comes after you know what to automate.

Prerequisites

• •
  Access to raw production or dogfood traces: full input, retrieved context (if RAG), full output, and any user signal (thumbs, edits, follow-ups). Phoenix/MLflow tracing or plain application logs all work.
• •
  A spreadsheet (the worksheet below) and 3–4 focused hours — this is deep work, not a background task.
• •
  For domain-heavy products: 1–2 hours of a domain expert's time for critique shadowing (step 5).

The workflow

1. 1.

   Sample 100 traces deliberately. Not the last 100: stratify — ~60 random across the period, ~25 from negative signals (bad ratings, user retries, escalations), ~15 from the longest/most complex inputs. Record the sampling recipe; you will repeat it next quarter.

2. 2.

   Set up the error-analysis worksheet. One row per trace, columns: trace_id · date · input summary · output summary · verdict (pass/fail/partial) · open code (free text) · failure category (assigned later) · severity (S1 wrong-and-harmful / S2 wrong / S3 degraded) · slice tags (intent, input length, user type) · candidate for golden set? (y/n) · notes/quote. Freeze the pass/fail definition in writing at the top before reading trace one.

3. 3.

   First pass — open coding. Read each trace and write what you see in plain language before inventing categories: "invented a policy number", "ignored the second question", "correct but 4x too long", "retrieved right doc, quoted wrong row". No taxonomy yet — premature categories blind you to failure modes you did not expect. This pass surfaces the surprises, and surprises are the yield.

4. 4.

   Second pass — axial coding into a taxonomy. Cluster the open codes into 5–10 named failure modes with one-line definitions and an example trace ID each (e.g., H1 unsupported-claim, R1 retrieval-miss, R2 ignored-context, F1 format-break, I1 instruction-drop, T1 tone/verbosity). Re-tag all 100 rows against the taxonomy. Then build the Pareto: count × severity per mode. The top 2–3 modes are your eval and fix roadmap — everything else is noise for now.

5. 5.

   Critique shadowing with a domain expert. Have the expert review 15–20 traces thinking aloud while you take notes; capture their pass/fail judgments and the reasons. Where the expert disagrees with your verdicts, your pass/fail definition is underspecified — tighten it. These expert judgments are gold-standard labels, literally: they anchor judge calibration later (pb-36).

6. 6.

   Convert findings into the golden dataset. Select 30–60 traces that exemplify each failure mode plus clear passes. For each: input (and retrieved context), a reference expectation (gold answer, or rubric of must/must-not facts), failure-mode label, severity, provenance (trace_id, date, who labeled it, why included). Add 5–10 synthetic edge cases for modes you know exist but did not appear in the sample — marked synthetic: true.

7. 7.

   Version it like code. Store as JSONL in git (or MLflow/Phoenix datasets) with a schema, a README stating the pass/fail definition and taxonomy v1, and a changelog. Rule: the golden set only grows through this pipeline — every new production failure worth catching becomes a new versioned case. That makes it a living eval set, not a stale benchmark.

Tool variants

• Phoenix (self-hosted, OSS): trace view for reading, datasets feature for the golden set; the trace→dataset flow is built in.
• MLflow 3: GenAI tracing plus versioned evaluation datasets; natural if MLflow already runs your experiment tracking.
• Spreadsheet + git JSONL: zero-infra variant; the worksheet is the analysis tool and the JSONL in git is the dataset — fully sufficient at 100-trace scale.

Verification

• •
  100 rows completed with verdicts, open codes, and taxonomy tags; sampling recipe documented.
• •
  Taxonomy has ≤ 10 modes, each with a definition and an exemplar trace ID; Pareto chart identifies the top 2–3 by count × severity.
• •
  Expert agreement checked on 15+ traces; disagreements resolved into a tightened pass/fail definition (record the before/after wording).
• •
  Golden dataset v1 committed with provenance fields on every case and a documented growth rule.

Pitfalls

• Sampling only complaints. Negative-signal-only samples miss silent failures — the user who got a wrong answer and believed it never clicks thumbs-down. That is why ~60% of the sample is random.
• Starting with a metric list. "Let's track hallucination and relevance" before reading traces guarantees you instrument the fashionable failures instead of yours.
• Categories too early. Open-code first; taxonomies imposed at trace 5 make traces 6–100 confirmation exercises.
• Golden set without provenance. A case nobody can trace to its origin cannot be defended when it blocks a deploy (pb-37) — provenance is what makes gates enforceable.
• One-and-done. Failure distributions drift; calendar the quarterly re-run with the same recipe so Paretos are comparable.

Restricted-environment variant

• •
  Read traces inside the production boundary (self-hosted Phoenix or log access); never export raw client traces to personal tools or hosted spreadsheets.
• •
  The golden set inherits the source data's classification — keep it in the client's repo. If a shareable variant is needed, rewrite cases as anonymized equivalents (pb-28 placeholder discipline) and mark them derived: anonymized.
• •
  The taxonomy and Pareto (counts, definitions, no verbatim content) are usually safe to share across the boundary — that is your status-report artifact.

Success criteria

• •
  You can name your product's top three failure modes with counts, severities, and example traces — from evidence, not intuition.
• •
  A golden dataset (30–60+ cases) exists, versioned, provenance-complete, with a growth rule the team follows.
• •
  The pass/fail definition survived expert contact and is written down.
• •
  The next playbooks (pb-35 metrics, pb-36 judges, pb-37 CI gates) consume this dataset directly — nothing downstream needs to invent test cases from scratch.