Write Eval-Based Acceptance Criteria for AI Features

Workflow area: AI Requirements & Agent Redesign · Tier: Advanced · Hands-on time: ~45 minutes

Classic acceptance criteria assume determinism — given input X, the system shall produce Y, and QA can verify it forever. An LLM feature breaks that assumption: the same input can produce different phrasings, quality is a distribution, and "correct" often has grades. The BA's move is not to abandon testability but to relocate it: the criterion no longer describes one output, it describes the measured performance of the feature across a versioned set of cases — exactly the way classic specs reference test cases, but with statistics.

When to use — and when not to

• Use it for any requirement whose implementation is a modelsummarization, classification, extraction, drafting, agent steps. If output varies run to run, the criteria must be eval-based.
• Use it when a vendor integration is specifiedeval-based criteria in the contract convert "our AI is highly accurate" into an acceptance test with a number the vendor signed.
• Use it to force the uncertainty conversationwriting the behavior-on-uncertainty clause makes the sponsor decide what the feature does when it doesn't know, which is the requirement everyone else forgot.
• Skip it for the deterministic parts of the featurerouting, storage, UI states, and audit logging around the model keep classic Given/When/Then criteria (pb-05); don't statistical-ify what isn't statistical.
• Skip it if no golden set exists or is plannedan eval-based criterion referencing a dataset nobody owns is theater; curate the set first or in parallel (pb-48).
• Do not accept "the model provider guarantees accuracy" as a criterionprovider benchmarks are not your task, your data, or your risk profile.

The recipe

1. 1.

   Inventory the untestable sentences. Sweep the draft spec for adjective-criteria: accurate, relevant, high-quality, appropriate, helpful, safe. Each one is a placeholder for a measurement that hasn't been designed. List them — this is your rewrite backlog, and its length is a useful shock statistic for the review meeting.

2. 2.

   For each, design the measurement before the wording. Decide four things with the SME and product owner: the golden set that represents the task (which cases, how many, who validated — pb-48), the scoring method (exact match, rubric, or LLM-judge with human audit), the threshold (what pass rate ships), and the tolerance floor (which failures are never acceptable at any rate). Then have AI draft the clause:

   Copy

   Rewrite this acceptance criterion as an eval-based requirement.
   Original: [e.g. "The claim summary shall be accurate."]
   Context: golden set GS-CLM-v1 (120 cases, SME-validated),
   scoring rubric R-SUM-01, thresholds agreed: 92% pass,
   zero fabricated policy/amount/date values.
   Produce:
   - AC-n: threshold clause referencing set + rubric + version
   - AC-n+1: zero-tolerance clause with the fatal categories
   - AC-n+2: behavior-on-uncertainty clause
   - AC-n+3: latency budget (p95) and cost-per-case budget
   - Verification method line for each (who runs what, when)
   Do not use the words "accurate", "appropriate" or "high-quality".
   

3. 3.

   Check the output against the pattern of a testable AI criterion. It should read like this:

   Copy

   AC-12  On golden set GS-CLM-v1 scored with rubric R-SUM-01,
          the summary feature achieves ≥ 92% pass rate.
          Verify: eval run in CI, report attached to release.
   AC-13  Zero instances of fabricated policy numbers, claim
          amounts, or dates across the full set. One instance
          fails the release. Verify: rubric category F1 count = 0.
   AC-14  Where source documents lack a required field, the
          feature outputs "Not stated in documents" and flags
          for review. It does not infer a value.
          Verify: 15 missing-field cases in GS-CLM-v1, 100% pass.
   AC-15  p95 latency ≤ 8s at 50 concurrent requests; cost
          ≤ $0.05 per case at contracted pricing.
          Verify: load test report + billing export.
   

   Every clause has a number, a referenced artifact with a version, and a verification method. That triplet is the test of testability.

4. 4.

   Negotiate thresholds as business decisions, not engineering guesses. The right pass rate is a function of the failure's cost and the fallback's quality: 92% with a clean escalation path may beat 97% with silent failures. Bring the sponsor the trade-off explicitly — "each point of threshold costs roughly X in model spend and Y in latency" — and record the rationale in the spec. Thresholds without recorded rationale get relitigated forever.

5. 5.

   Wire the criteria into change control. Add one meta-requirement: any model version change, prompt change, or golden-set update triggers a full eval rerun, and results below threshold block release. This single clause converts your acceptance criteria from a launch gate into a permanent regression gate — which is where their real value lives (pb-50 governs the process around it).

Make it yours

• promptfoo or Braintrust-class tooling: encode the golden set and thresholds as automated assertions so AC verification is a CI artifact, not a manual campaign.
• Spreadsheet scoring for low-tooling teams: cases as rows, rubric as columns, a pivot as the pass-rate report — auditable and sufficient for quarterly reruns.
• LLM-as-judge with a human audit ratio: let a model apply the rubric at scale, but hand-verify a fixed sample (10–20%) each run and all fails; publish the audit agreement rate alongside the score.
• Contract annex for vendor features: move AC clauses, golden-set reference, and rerun rights into the SOW — acceptance against your eval set, at your threshold, on your cases.

Quality check

• No adjective-criteria survivegrep the spec for "accurate", "appropriate", "relevant", "high-quality"; every hit is rewritten or justified.
• Every threshold references a versioned artifactgolden set ID and rubric ID appear in the clause itself, not in tribal memory.
• Zero-tolerance categories are enumeratedthe failures that are fatal at any rate are named, and one occurrence blocks release.
• Uncertainty behavior is specified positivelywhat the feature says and does when it cannot answer, with cases in the set that test it.
• Latency and cost have budgetsa feature that passes quality but costs 4x the business case has failed a requirement, and the spec says so.
• Rerun triggers are requirementsmodel, prompt, or dataset changes force re-verification; the criteria outlive launch day.