Construct Golden Datasets from Production Traces

Tier: Core · Workflow area: Testing AI systems — foundations

This playbook turns raw production traces into a golden dataset with defined strata, recorded grading rationale, git-style versioning, and a continuous feed of sampled live traffic — so the dataset ages with your users instead of with your launch demo.

When to use

Prerequisites

• •
  Trace capture in place: Langfuse, LangSmith, Braintrust, or even structured logs of input/output/metadata per LLM call.
• •
  A PII scrubbing step you trust (rule-based + review; see your track's test-data playbooks).
• •
  Git (or your tracing platform's dataset feature) for versioning; a spreadsheet-free grading habit.

The recipe

1. 1.

   Define the strata before touching any traces. A golden dataset is stratified or it is a pile. Write the axes and target counts first:

   Copy

   Strata for support-summarizer golden set v1 (target 150 cases):
   - happy_path        clean single-issue tickets            40
   - multi_issue       2+ problems in one ticket             25
   - long_input        >2,000-token tickets                  15
   - ambiguous         unclear intent, missing info          20
   - hostile_tone      angry/abusive customer language       15
   - edge_format       empty body, HTML paste, non-English   15
   - known_failures    every production incident's input     20 (grows forever)
   

   The known_failures stratum is the compounding asset: every incident permanently joins the regression pack, exactly like a repro test for a classic bug.

2. 2.

   Mine candidates from traces with AI triage. Export a few hundred traces and let an assistant do the first sort:

   Copy

   Here are 300 scrubbed production traces (input, output, user_feedback,
   latency). Classify each into our strata [paste definitions]. Flag:
   - traces where user_feedback was negative or the user retried,
   - inputs unlike anything in our current dataset (novelty),
   - near-duplicates of each other (keep the best one per cluster).
   Output a table: trace_id, stratum, flags, one-line reason.
   

   In Langfuse, negative-score and user-retry filters do the first cut natively; the assistant then handles stratum assignment and dedup. Humans pick from this triage — the model nominates, it does not admit.

3. 3.

   Grade every admitted case — with rationale, not just labels. For each case, record the expected outcome at the right strictness: exact output for extraction tasks, required/forbidden properties for generative ones:

   Copy

   {
     "id": "gs-0087",
     "stratum": "multi_issue",
     "input": "trace-scrubbed ticket text...",
     "expected": {
       "must_mention": ["refund delay", "damaged item"],
       "must_not": ["promises a refund"],
       "reference_summary": "Customer reports refund delayed 14 days and item arrived damaged..."
     },
     "grade_rationale": "Both issues must surface; incident INC-231 was a summary that dropped the second issue.",
     "graded_by": "priya",
     "added": "2026-08-05",
     "source": "trace:1f9c…"
   }
   

   The rationale field is what makes the dataset auditable a year later, when someone asks why case gs-0087 fails their new prompt. Ungraded examples are test data; graded ones are ground truth.

4. 4.

   Version the dataset like code — because it is a spec. The golden set lives in git (or a Langfuse dataset with named versions): changes arrive by PR, additions cite their source trace, edits to expected fields require a reviewer, and releases are tagged (golden-v1.3). Every eval result must record which dataset version it ran against — a pass rate without a dataset version is unreproducible.

5. 5.

   Install the continuous feed. A weekly job samples fresh traces: everything with negative feedback, plus a random 1% slice for drift coverage. The assistant triages against current strata (step 2's prompt, reused) and files a "candidates" PR a human grades and merges. Target: dataset grows a few cases weekly, and the strata table's actual counts stay within sight of targets. Real traffic drifts — new user segments, new phrasings, new abuse patterns — and an unfed golden set silently stops representing anything.

6. 6.

   Prune and rebalance quarterly. Retire near-duplicates, rebalance strata that ballooned, and re-grade cases whose product requirements changed. Deletions get the same PR review as additions: removing a hard case is how a dataset gets quietly easier, and a dataset that only ever gets easier is a rubber stamp.

Variations

• Langfuse-native:langfuse datasets holds items and versions; link eval runs to dataset versions automatically and keep the grading-rationale field in item metadata.
• Multi-turn/agent features: items are conversation prefixes plus expected next-step properties (tool called, constraint held) — stratify by trajectory shape (happy path, recovery-after-error, user-changes-mind), not just by input topic.
• Small-traffic products: augment thin strata with assistant-generated variations of real cases ("write 5 hostile-tone variants of this real ticket"), labeled synthetic: true so you can always report real-vs-synthetic pass rates separately.

Quality check

• •
  Strata table exists with targets and actuals; no stratum is empty; known_failures contains every incident since launch.
• •
  100% of cases carry expected outcomes, rationale, grader, date, and source trace.
• •
  All inputs are PII-scrubbed — spot-check ten cases per release; one leak makes the dataset unshareable.
• •
  Every recorded eval run names its dataset version; the current version is tagged.
• •
  The weekly feed has merged cases in each of the last four weeks (an unfed dataset is decaying, whatever its size).

Pitfalls

• Happy-path gravity. Random trace sampling over-represents easy traffic. Stratified targets exist precisely to force curation toward the hard 20%.
• Grading by vibes. "Looks good" as an expected outcome makes every future eval ungradable. Properties and rationale, always.
• Dataset leakage into prompts. If golden cases get pasted into the prompt as few-shot examples, your regression pack is now training data — pass rates on those cases become meaningless. Keep an exclusion check.
• One giant unversioned JSON. The moment two people disagree about whether a case changed, an unversioned dataset has zero authority. Git from day one.