Error Analysis: Open-Code 100 Production Traces

Workflow area: Evals & AI Quality · Tier: Advanced · Hands-on time: ~45 minutes to learn, ~half a day to run

Teams that skip error analysis build eval sets that measure imagined failures while production burns with real ones. The instinct is to jump straight to metrics — "let's track accuracy" — but accuracy against what? The only trustworthy source of what your AI feature gets wrong is the record of it getting things wrong: production traces, read one at a time, by someone who understands the product deeply enough to judge them. That someone is the PM. A hundred traces sounds like grunt work; it is the highest-information-density half-day in AI product management, and everything downstream — the eval set, the failure catalog, the launch bars — inherits its quality.

When to use — and when not to

• Use it before writing your first eval setpb-37 without pb-36 produces evals for hypothetical failures; the taxonomy comes first, always.
• Use it on a recurring cadencequarterly, or after any major prompt/model change; failure distributions drift, and last quarter's taxonomy quietly goes stale.
• Use it when complaint volume and dashboard metrics disagreethumbs-up rates look fine but support tickets say otherwise; the traces arbitrate.
• Skip it only pre-launch when no production traces existuse pb-35's spike outputs and dogfooding transcripts as a stand-in, and rerun the real thing within weeks of launch.
• Don't delegate the readingan engineer or a junior can pull the sample, but the open-coding needs the person who knows what the product promised; outsourced judgment produces a taxonomy of surface symptoms.

The recipe

1. 1.

   Pull a defensible sample. Export ~100 traces (full input, retrieved context if any, output, and user signal where available) from Braintrust, LangSmith or your logs. Sample mostly at random — random sampling is what makes counts meaningful — and optionally add a labeled oversample of complaint-linked traces. Note the split; never let the oversample contaminate the frequency claims.

2. 2.

   Open-code each trace in a sheet — your words, no schema yet. One row per trace: trace ID/link, verdict (pass/fail/partial), and a free-text note on what went wrong. "Open coding" means writing what you see before deciding categories — pre-imposed categories at this stage are how real failure modes get squeezed into wrong boxes:

   Copy

   #014 | FAIL    | cited the 2023 refund policy; 2024 doc exists
                    in KB but wasn't retrieved
   #015 | PASS    |
   #016 | PARTIAL | answer correct but opens by apologizing 3 times,
                    reads unconfident
   #017 | FAIL    | user asked two questions, model answered only
                    the first, no acknowledgment
   #018 | FAIL    | invented a "premium support tier" we don't sell
   #019 | FAIL    | refused a completely benign billing question
   

   Expect 60–90 minutes for a hundred traces. Do not batch this into an LLM — the entire value is your judgment making contact with reality. (AI gets a role in step 4.)

3. 3.

   Cluster the notes into a taxonomy with counts. Group your free-text notes into failure modes, keeping the level useful: "retrieval missed the current doc version" and "invented an entity we don't sell" are actionable; "wrong answer" is not. Count occurrences and tag severity:

   Copy

   FAILURE MODE                                COUNT   SEVERITY
   Retrieval missed / stale doc grounded         14       S1
   Invented entity, plan or commitment            8       S1
   Partial answer to multi-part question          9       S2
   Wrong refusal of benign request                6       S2
   Tone: unconfident / over-apologetic            7       S3
   Formatting drift from spec                     4       S3
   (failing traces: 41 of 100 sampled)
   

4. 4.

   Use AI as a second coder, then reconcile. Now — after your own pass — feed the failing traces' notes to Claude: "Cluster these failure notes into a taxonomy; propose names and counts." Compare with yours. Where the model splits a cluster you merged (or vice versa), decide deliberately. This catches your blind spots without ever having outsourced the judgment.

5. 5.

   Name the top three fix candidates in a one-page memo. For each dominant mode: hypothesized cause (retrieval config? prompt gap? model capability? missing knowledge?), the cheapest probe to confirm, and who owns it. The memo's punchline is usually humbling — the top mode is rarely the one leadership talks about, and it's frequently fixable in a week (a retrieval-freshness bug, one prompt paragraph) rather than a model swap.

6. 6.

   Feed everything downstream. The taxonomy becomes pb-37's eval-set backbone (every mode gets cases), pb-39's failure-catalog rows, and pb-41's monitoring priorities. Version the taxonomy with a date — next quarter's pass will diff against it, and the diff is your quality trend.

Make it yours

• Braintrust/LangSmith-native: do the coding in the tool's annotation UI so labels live next to traces and the eval set can be seeded from labeled traces directly — same discipline, better plumbing.
• The team calibration variant: two PMs (or PM + support lead) independently code the same 30 traces, then compare verdicts; the disagreements calibrate the team's quality bar better than any rubric document, and this doubles as pb-38's human-label seed.
• Low-volume products: at fewer than 100 traces a week, read everything weekly as a standing ritual — fifteen minutes with coffee; the taxonomy still accumulates, just continuously.

Quality check

• The sample is random (or the oversample is quarantined)every frequency claim traces to the random portion.
• Every failing trace links back to the raw traceanyone can audit any count in one click.
• Failure modes are actionable-leveleach mode implies a different fix owner; "wrong answer" appears nowhere.
• Counts and severities are realtallied from rows, with S1 reserved for user-harming or trust-destroying failures.
• You read the traces yourselfAI clustered your notes; it never replaced your verdicts.
• The taxonomy is dated and versionedand downstream artifacts (eval set, catalog, monitors) cite it.