Open-code a day's slice of production traces from a fictional support copilot, cluster the failures into a taxonomy with counts, and write the one-page findings memo naming the top three fixes. The trace set is seeded with five failure modes at known frequencies — your taxonomy is checkable against the key.
Kabir is a Principal PM at a Noida company whose product manages field-service operations. His support copilot drafts replies to customer emails about visits, parts, warranties, and refunds; agents approve or edit before sending. Quality complaints are rising, but nobody can say what kind of wrong the copilot is. The professional discipline — read a hundred traces before writing a single eval — starts here, with a 40-trace slice small enough for one sitting and seeded so your analysis can be graded. No code required: a spreadsheet, the ground truth, and your attention.
Every judgment call below is decidable against this card. Print it or keep it beside you.
POLICY CARD (current as of this trace day)
1. Warranty: replacement parts 12 months; labor 90 days.
2. Annual safety inspections: certified technician only. Customers
may NEVER self-certify.
3. Visit fee Rs 1,200 - refundable only if cancelled >= 24h ahead.
4. Rescheduling: free up to 4h before the slot.
5. Part P-204 compressor relay: Rs 8,900 (since 1 July; was 7,400).
6. Part F-112 filter kit: Rs 1,450.
7. Service hours Mon-Sat 08:00-20:00. No Sunday visits.
8. AMC customers: priority slot within 24h.
9. Refunds process in 5-7 business days.
10. Gas smell / burning smell: emergency line 1800-XXX-404.
Never advise the customer to troubleshoot.
Forty traces: customer ask → what the copilot's draft said (condensed).
T01 | When is my visit? | Confirms Thu 10-12 slot, polite
T02 | Compressor relay cost? | Quotes P-204 at Rs 7,400
T03 | Reschedule, 2 days ahead | Free reschedule confirmed
T04 | Any discount available? | Offers "15% first-visit loyalty discount"
T05 | Where is my refund? | 5-7 business days, correct
T06 | Gas smell near boiler | Suggests checking valve, relighting pilot
T07 | Part replaced 2 months ago, failed | Covered under 12-month parts warranty
T08 | Labor issue, 6 months after service | Free fix "under labor warranty"
T09 | AMC customer, urgent breakdown | Priority slot within 24h offered
T10 | Complaint: technician was late | "Delays happen; plan a wider window"
T11 | Can you come Sunday? | Explains Mon-Sat hours, offers Monday
T12 | Cancelled same day, want refund | Cites "48-hour goodwill refund window", refunds
T13 | Filter kit price? | Rs 1,450, correct
T14 | Cancelled 3h before visit | Promises full visit-fee refund
T15 | How do I book a visit? | Correct steps, friendly
T16 | Quote: relay + fitting | P-204 at Rs 7,400 plus labor
T17 | Need invoice copy | Sends copy, correct
T18 | Can our office manager self-certify the safety inspection? | "Yes - submit the form yourself"
T19 | Gas smell in kitchen | Emergency line, evacuate, correct
T20 | Refund is late, second email | "As we already told you twice..."
T21 | Reschedule, 6h before slot | Free reschedule confirmed
T22 | What does AMC include? | Includes "free annual Sunday maintenance visit"
T23 | Part failed at 11 months | Covered under parts warranty, correct
T24 | Reschedule 1h before slot | "Free of charge, no problem"
T25 | Complaint: messy work area | Apology + follow-up visit, correct
T26 | Filter kit price? | Quotes Rs 1,100
T27 | How do I renew AMC? | Correct renewal steps
T28 | AMC warranty question | "Lifetime warranty on all labor for AMC members"
T29 | Payment methods? | Correct list
T30 | Cancelled 30h ahead, refund? | "Visit fee is non-refundable"
T31 | Invoice amount looks wrong | Polite escalation to billing, correct
T32 | Email in Hindi asking re slot | One line in English: "Read our FAQ."
T33 | Burning smell from unit | Emergency line, do not touch, correct
T34 | Is the relay in stock? | In stock "at Rs 7,400"
T35 | Resend booking confirmation | Resent, correct
T36 | Your competitor quoted less | Offers "price-match guarantee"
T37 | Reschedule, 5h ahead | Free reschedule confirmed
T38 | Labor complaint at 4 months | Approves free labor rework
T39 | Thanks, great service | Warm close, correct
T40 | Senior citizen discount? | "We waive the visit fee for seniors"
Set up the coding sheet. Columns: Trace | Verdict (OK / FAIL) | What went wrong (your words) | Cluster (later). Forty rows. The "your words" column is open coding — you describe each failure freshly, without a predefined category list. Do not invent the taxonomy first; taxonomies built before reading are the anti-pattern this lab exists to break.
Code all forty traces against the policy card. One by one: does the draft's content match the card, and is the tone acceptable to send to a paying customer? Write a specific note for every FAIL — "quoted old relay price" beats "wrong info." Expect this to take 30–40 minutes. Traces that are partially wrong are FAIL: an accurate sentence wrapped around one wrong commitment still commits it.
Cluster your failure notes into a taxonomy. Read your FAIL notes top to bottom and group by mechanism, not by topic — "invented a policy that doesn't exist" and "misapplied a real policy's boundary" are different mechanisms even when both involve refunds. Name each cluster, count its traces, and list the trace IDs. Aim for 4–7 clusters; one giant "wrong answer" bucket means you clustered by verdict, and twelve micro-clusters mean you clustered by topic.
Pressure-test the taxonomy with AI. Paste your FAIL notes (not the traces) with: "Cluster these failure descriptions by underlying mechanism. Propose a taxonomy and assign each note." Compare against yours. Where the AI splits what you merged or merges what you split, decide deliberately and note why. You own the taxonomy; the AI is a second coder, not the judge.
Rank by frequency × severity. Add a severity call per cluster: what does this failure cost when it ships? A wrong price costs a margin conversation; unsafe gas-leak advice costs something unbounded. Rank clusters by count, then re-rank by severity, and notice where the two rankings disagree — that disagreement is the most important finding in most real error analyses.
Name the top-three fix candidates. For each: the cluster it targets, the intervention type (grounding the copilot in the policy card / retrieval of current price data / tone rules in the system prompt / hard refusal rules for safety topics), and how you would know it worked (the cluster's count in the next 40-trace sample). Fixes must map to mechanisms — "make it more accurate" is not a fix candidate.
Write the one-page findings memo. Structure: one-line summary with overall failure rate; the taxonomy table (cluster, count, severity, example trace ID); the frequency-vs-severity tension called out explicitly; top-three fixes with expected effect; what you will instrument going forward. Write it for an audience of Kabir's engineering lead and CS head.
Grade against the key. Compare your clusters, counts, and trace assignments with the answer key. Misassigned traces are normal at the boundaries — what matters is whether you found all five mechanisms and whether your counts are within one or two of the seeded frequencies.
Reading traces is not a chore before the real work — it is the real work; every eval, launch bar, and fix priority in the next three labs inherits its authority from this coding sheet.
A complete lab has all of the following:
Failure modes are found by reading, not by brainstorming — forty coded traces produced a five-mechanism taxonomy with counts, a severity re-ranking, and three targeted fixes. Everything downstream — evals, judges, launch bars — is built on exactly this sheet.