Set Launch Bars and Wire Eval Regression Gates

Workflow area: Evals & AI Quality · Tier: Advanced · Hands-on time: ~45 minutes

Traditional features ship when the acceptance criteria pass. AI features have no binary pass: quality is a distribution, and every prompt tweak, model upgrade, or retrieval change reshuffles it. Without a numeric bar and an automated gate, launch decisions collapse into whoever tried the demo last. The launch bar turns "is it good enough?" from a meeting into a metric — and the CI gate makes the metric enforceable.

When to use — and when not to

• Use it before any user-facing AI feature shipsthe bar must exist before the first launch review, not be reverse-engineered after someone asks "how do we know it's good?"
• Use it whenever prompts, models, or retrieval configs change post-launchthe regression gate is the whole point; upgrades to a "better" model routinely regress specific behaviors your users depend on.
• Use it when quality debates keep re-litigatinga written bar with named owners ends the weekly argument about whether the summarizer is "ready."
• Skip the full CI wiring for throwaway prototypesa prototype needs a scratch eval set (pb-33 territory), not a merge gate; gates on experiments slow learning.
• Skip it if you have no eval set yeta gate with five test cases is theater. Build the golden set first; a useful gate needs 50+ graded cases covering your failure catalog.

The recipe

1. 1.

   Name the quality dimensions that would block launch. Not "quality" — dimensions: factual accuracy, instruction-following, refusal correctness, tone, latency, harmful-output rate. Pick the three to five where failure genuinely blocks shipping. For each, decide the metric (pass rate on graded cases, mean judge score, p95 latency) and who signs off.

2. 2.

   Draft the launch-bar table with AI, then argue with it. Paste this into Claude:

   Copy

   You are helping a PM set launch bars for an AI feature.
   Feature: [ONE PARAGRAPH — what it does, who uses it, what a bad
   output costs the user]
   Quality dimensions we track: [LIST — e.g. factual accuracy,
   refusal correctness, tone, p95 latency]
   Current eval results: [PASTE — e.g. "accuracy 91% on 120 golden
   cases, refusal correctness 84%, p95 4.2s"]
   
   For each dimension produce a table row:
   DIMENSION | METRIC | LAUNCH BAR (ship threshold) | REGRESSION BAR
   (max allowed drop post-launch) | RATIONALE | OWNER
   
   Rules: propose bars justified by user cost, not by our current
   scores. Where our current score is below your proposed bar, say so
   explicitly. Flag any dimension where a single catastrophic failure
   matters more than the average — those need a zero-tolerance case
   list, not a percentage.
   

   Sample output row:

   Copy

   Refusal correctness | % of 40 out-of-scope probes correctly
   declined | ship ≥ 95% | regression: any drop below 95% blocks
   merge | a wrong answer to a medical question is a trust incident,
   not a quality miss | PM: you · zero-tolerance list: cases R-01..R-08
   

   The model will propose bars flattering to your current scores. Push back: the bar comes from the cost of failure, and if your feature is below the bar, the honest output of this exercise is "don't ship yet."

3. 3.

   Encode the bar as promptfoo assertions. Each dimension becomes a test suite with a threshold. In promptfooconfig.yaml, your golden cases assert per-case checks, and the suite-level evaluateOptions threshold encodes the bar — e.g. the run fails if pass rate drops below 0.95 on the refusal suite. Zero-tolerance cases get individual assert blocks so one failure fails the run regardless of averages.

4. 4.

   Hand engineering the gate spec, not the implementation. Write a half-page: which suites run on which triggers (every PR touching prompts/, nightly against the pinned model, on model-version bumps), what blocks merge versus what warns, and where results post (PR comment plus a channel). A GitHub Actions job running promptfoo eval with --fail-on-threshold is a one-day task for engineering — but the trigger matrix and block-versus-warn policy are product decisions. Make them explicitly.

5. 5.

   Run the ritual once before trusting it. Deliberately worsen a prompt on a branch and confirm the gate blocks the PR with a readable diff of which cases regressed. Then check the failure report answers the PM question — which user-visible behavior got worse — not just "score 0.91 → 0.88." If the report can't name the regressed cases, fix the reporting before the first real incident.

Make it yours

• Braintrust instead of promptfoo: define the same bars as Braintrust scorer thresholds and use its experiment-comparison view as the merge evidence — the PR links a Braintrust experiment showing no dimension fell below its regression bar. Better when non-engineers grade cases in the same tool.
• Model-upgrade drills: add a scheduled monthly run of the full suite against the newest model version in a non-blocking job. You get advance warning of what an upgrade would break, and the migration decision arrives with data attached.
• Tiered bars for staged rollout: define two bars — an internal-beta bar and a GA bar. This pairs directly with pb-43's cohort gates: each rollout stage promotes only when its bar holds on both eval scores and live guardrail metrics.

Quality check

• Every bar has a rationale rooted in user costa threshold nobody can justify will be negotiated away the first time it blocks a launch.
• Zero-tolerance cases are enumerated, not implied"never gives medical advice" is a listed set of probes with individual asserts, not a vibe.
• The gate blocks the right eventsprompt edits, model bumps, retrieval changes; if any of these can reach production without an eval run, the gate has a hole.
• A failed gate produces a case-level diffthe report names which behaviors regressed, in language a launch review can act on.
• The bar has a change processlowering a bar requires the named owner's sign-off in writing, not a quiet YAML edit in the same PR that fails it.