Curate the Golden Dataset as a BA-Owned Requirements Artifact

Workflow area: AI Requirements & Agent Redesign · Tier: Advanced · Hands-on time: ~40 minutes to structure; curation is ongoing

Every AI requirements practice in this track eventually points at one artifact. The eval-based criterion says "≥92% on GS-CLM-v1" (pb-43); the vendor harness runs "our 25 cases" (pb-40); the regression gate reruns "the set" on every model change (pb-50). If that set is an engineer's scratch file of convenient examples, everything built on it is load-bearing on sand. The golden dataset is elicitation output — it encodes what the business means by correct — which is why it belongs to the BA, with the same versioning and sign-off discipline as a baselined BRD.

When to use — and when not to

• Use it the moment an AI feature has eval-based acceptance criteriaor is about to; the criteria and the dataset should be drafted together, each forcing precision on the other.
• Use it to consolidate scattered test casesif the spike (pb-41), the vendor eval (pb-40), and the developers each have their own case pile, merge them into one versioned set before they drift into three definitions of correct.
• Use it as an elicitation instrumentasking an SME "what should the system do with THIS case?" surfaces tacit rules that no abstract interview question reaches.
• Skip the ceremony for throwaway explorationsa spike's 25 cases need a spreadsheet, not a versioning policy; promote them to golden status only when the initiative proceeds.
• Skip building it soloa golden set validated by one person encodes one person's errors; every expected behavior needs an SME name attached.
• Do not let the dataset become an engineering-only artifactif the BA stops curating, cases stop reflecting the business's evolving definition of correct, and the regression gate slowly tests the wrong thing.

The recipe

1. 1.

   Design the schema before collecting cases. Minimum viable columns: case ID; input (the actual artifact — email, document, record — anonymized); expected behavior (what a correct output does, not necessarily exact wording); pass criteria (how a scorer decides — exact match, rubric reference, or must-include/must-not-include lists); category (routine / edge / adversarial / uncertainty); source (which ticket, interview, or incident it came from); validator (the SME who signed the expected behavior); status and version metadata. The "source" column is what makes this a requirements artifact — every case traces to elicited reality.

2. 2.

   Harvest cases from the elicitation record. Your interviews, ticket mines, exception logs, and process walkthroughs already contain the cases — including the disputes. Have AI do the first sweep:

   Copy

   From the material below, extract candidate golden-set cases as:
   Case summary | Input artifact (quote or pointer) | What the
   correct handling appears to be | Source citation | Category
   guess (routine / edge / adversarial / uncertainty)
   Prioritize: cases participants disagreed about, exception cases,
   cases where the stated rule and actual practice differ, and
   cases where the right answer is "escalate / cannot determine".
   Do NOT invent inputs or expected behaviors — every row must
   cite the source material.
   Material: [TRANSCRIPTS / TICKET EXPORTS / EXCEPTION LOGS]
   

   The disputed cases are the most valuable rows: forcing an SME decision on them resolves a requirements ambiguity that would otherwise surface in production.

3. 3.

   Engineer the distribution deliberately. A credible set is not 60 happy-path variations. Target roughly: 50–60% routine cases spanning the real input variety; 20–25% edge cases (boundary values, unusual formats, multi-item inputs); 10–15% uncertainty cases where the correct output is "cannot determine, escalate"; and 5–10% adversarial cases — injection strings, leading premises, inputs designed to bait fabrication (fed by pb-44's catalog). Record the target mix in the dataset's README so future curators keep the shape.

4. 4.

   Validate every expected behavior with a named SME. Walk the cases with the process owner: "for this input, is THIS the correct outcome, and would two of your people agree?" Where SMEs disagree with each other, escalate the case as a requirements decision — do not average. The validator's name goes on the row. An unvalidated case is a draft, exactly like an unconfirmed elicitation finding (pb-01), and drafts do not gate releases.

5. 5.

   Version it, cite it, and schedule its growth. Baseline as GS-[FEATURE]-v1 with a changelog; requirements cite the version (pb-43), and any case change bumps it — because changing a case silently changes what "92% pass" means. Then wire the growth loop: production exceptions and incident cases (pb-52) get triaged monthly into new candidate rows, so the set tracks reality's drift. Store it where both analysts and CI can reach it — a promptfoo/Braintrust-readable format plus a human-readable sheet is the pragmatic pair.

Make it yours

• Spreadsheet-first for BA ownership: the sheet is the source of truth with validation status and SME names; an export script feeds the eval tooling — analysts curate, engineers consume.
• promptfoo/Braintrust-native at engineering-heavy shops: cases live as test fixtures in the repo; the BA reviews via pull request, which conveniently timestamps every change to the definition of correct.
• Fairness and privacy sub-sets: keep pb-47's counterfactual pairs and leakage probes as tagged sections of the same versioned set, so one regression run exercises quality, fairness, and privacy together.
• At program scale: a golden-set registry — every AI feature lists its set, version, owner, and last validation date — becomes the single fastest audit of whether the org's AI quality claims mean anything.

Quality check

• Every case cites a real sourcea ticket, transcript, or incident; invented cases are marked SYNTHETIC and used only where real data is legally unusable.
• Every expected behavior has a named validatorand disputed cases were decided by escalation, not by whoever edited last.
• The distribution is engineered and documenteduncertainty and adversarial cases are present in stated proportions, not as an afterthought.
• Pass criteria are scoreabletwo independent scorers reach the same verdict on a sample; rubric ambiguity is fixed in the rubric, not argued per run.
• Versioning is realrequirements cite versions, changes bump them, and the changelog explains why each case entered or changed.
• The growth loop runsproduction exceptions feed monthly triage, and the set's coverage of live failure patterns is reviewed, not assumed.