Design Statistical Pass Criteria for Non-Deterministic LLM Features

Tier: Core · Workflow area: Testing AI systems — foundations

This playbook establishes run-N-times pass criteria for an LLM-backed feature, introduces pass^k for agent workflows, and gives you the vocabulary that changes triage forever: variance is the system under test; flakiness is the test.

When to use

Prerequisites

• •
  API access to the feature under test with a fixed prompt/model/config snapshot (version-pin what you can).
• •
  A repeat runner: pytest-repeat, a simple loop script, or Promptfoo's repeat: option.
• •
  A small graded input set (10–30 cases) with expected outcomes — even informal ones.

The recipe

1. 1.

   Measure before you set thresholds — establish the variance baseline. Run each test case 20 times against the unchanged system:

   Copy

   pytest tests/test_summarizer.py --count=20 -q
   

   Or in Promptfoo:

   Copy

   tests:
     - vars: { ticket: "refund not processed after 14 days..." }
       assert:
         - type: llm-rubric
           value: "mentions refund delay AND names the 14-day period"
   # run with: promptfoo eval --repeat 20
   

   Copy

   case_refund     : 19/20 pass (0.95)
   case_multi_issue: 14/20 pass (0.70)
   case_empty_body : 20/20 pass (1.00)
   

   That 0.70 case is not a broken test — it is a measurement of the system's real reliability on that input. You cannot set a sane threshold without this baseline; teams that skip it pick 100% and then delete the suite in frustration.

2. 2.

   Classify every sub-100% case: variance or flakiness? For each failing run, read the transcript and ask what failed:

   • •
     Model variance: the model produced a genuinely different, worse answer. This is signal about the product.
   • •
     Test flakiness: timeout, rate limit, brittle string assertion ("expected 'Refund delayed'" but got "The refund was delayed"), judge misgrading. This is noise in your measurement.

   Fix flakiness ruthlessly (loosen string matches into semantic checks, add retries on 429s) — otherwise your statistics measure your harness, not your model. Only after the harness is clean do the pass rates mean anything.

3. 3.

   Set per-case thresholds by consequence, not by wish.

   Copy

   Prompt to your assistant:
   Here are 12 test cases for our support-ticket summarizer with measured
   20-run pass rates and a description of each case's user-facing risk.
   Propose a pass threshold per case using these bands:
   - Safety/compliance behaviors (refuses to promise refunds): 100% — any
     failure blocks release.
   - Core correctness on clean inputs: ≥ 95%.
   - Hard/ambiguous inputs: ≥ 80%, tracked for improvement.
   Flag any case whose measured rate is already below the proposed bar.
   

   The bands are the point: one uniform threshold is always wrong in both directions — too lax for safety, unachievably strict for genuinely hard inputs.

4. 4.

   For multi-step agents, compute pass^k — this is where intuitions die. An agent that is 95% reliable per step, across 10 sequential steps, completes the journey at 0.95^10 ≈ 60%. Per-step pass rates that sound excellent compound into coin flips:

   Copy

   steps:        1      3      5      10
   p=0.99      0.99   0.97   0.95   0.90
   p=0.95      0.95   0.86   0.77   0.60
   p=0.90      0.90   0.73   0.59   0.35
   

   Test agents both per-step (where did it break?) and end-to-end run-N-times (what does the user experience?), and set the release criterion on the end-to-end number — because that is the one users live with.

5. 5.

   Wire the criteria into CI with statistical gates. The suite runs each case N times (N=5–10 in CI for cost; 20+ nightly) and fails when a case drops below its threshold — not when any single run fails. Report pass rates, not pass/fail, so trends are visible: a case drifting 0.95 → 0.85 over three weeks is a regression a binary gate never shows.

6. 6.

   Re-baseline on every intentional change. New model version, edited prompt, changed temperature: rerun the 20× baseline before comparing anything. Comparing new-model runs against old-model thresholds without re-baselining is the most common self-inflicted "regression" in AI QA.

Variations

• Cheap-first sampling: run N=3 on every PR, N=20 nightly, N=50 before release — same thresholds, escalating confidence, bounded API bill.
• Classification features: you have real labels, so use proper metrics (accuracy/F1 across the set at N repeats) instead of per-case pass rates.
• Provider-side determinism aids: seed parameters and prompt caching reduce variance where offered — treat them as variance reduction, never as a determinism guarantee to build assertions on.

Quality check

• •
  Every case has a measured baseline (N≥20) recorded before its threshold was set.
• •
  Thresholds are banded by consequence; every 100%-band case is genuinely safety-critical.
• •
  CI failure message shows the rate ("case_multi_issue 3/10, threshold 0.8"), never just "FAILED".
• •
  Agent features carry an end-to-end pass^k figure in the release notes, not just per-step rates.
• •
  Flakiness classification happened: someone read the failing transcripts this cycle.

Pitfalls

• Retry-until-green. Rerunning a statistical suite until it passes is p-hacking your own release gate. The gate evaluates a fixed batch of N; there is no "run it again."
• Temperature-0 folklore. Lower temperature narrows the distribution; it does not close it. Provider-side non-determinism (batching, hardware, model updates) remains.
• N too small for the threshold. You cannot verify a 0.95 threshold with N=5 — one failure is 80%. Match N to the precision the threshold implies.
• Averaging away safety cases. A 96% overall pass rate can hide a 60% rate on the one case that promises customers illegal refunds. Gates are per-case; averages are for dashboards.