Write Staged Rollout and Rollback Plans for AI Features

Workflow area: Launch & GTM · Tier: Advanced · Hands-on time: ~40 minutes

A conventional feature flag plan asks "does it work?" at each stage. An AI rollout must also ask "is it still good?" — because the feature can function perfectly while quality degrades on real traffic — and "what exactly do we roll back to?" — because the deployable unit isn't just code: it's a prompt version, a model version, and a config, and rolling back one without the others reproduces a state you never tested. Writing the plan is 40 minutes; running a launch without one is a quarter of firefighting.

When to use — and when not to

• Use it for every AI feature reaching external usersthe plan is a launch-review artifact, reviewed alongside pb-40's launch bar.
• Use it for major model migrations on live featuresa model swap is a launch; it gets stages, gates, and a rollback path like any other.
• Use it when the feature can spend money or send output externallycost runaway and reputational blast radius are exactly what cohort gates contain.
• Skip the ceremony for internal alpha with ten named usersa Slack channel and a manual off-switch are proportionate; write the real plan before beta.
• Skip staged rollout where cohorting is impossiblesome surfaces (public marketing pages) are all-or-nothing; for those, compensate with a shadow-mode period and a faster kill switch instead.

The recipe

1. 1.

   Define the stages and who's in each. Standard ladder: internal dogfood → design partners (opt-in, named) → 5% random → 25% → 100%. For each stage write: entry criteria (the previous stage's gate), duration floor (minimum bake time — a week at 5% catches weekly-pattern traffic that a day doesn't), and the cohort rule in your flag tool (LaunchDarkly segment or Statsig gate definition).

2. 2.

   Attach three gate types to every stage. This is the AI-specific move — draft with Claude:

   Copy

   Write stage gates for an AI feature rollout.
   Feature: [DESCRIPTION + WHAT BAD OUTPUT COSTS]
   Metrics ladder: [PASTE FROM YOUR pb-41 SPEC — acceptance,
   containment, complaint rate, cost per outcome]
   Launch bar: [PASTE FROM pb-40 — eval thresholds]
   Stages: dogfood, design partners, 5%, 25%, 100%
   
   For each stage produce:
   1. EVAL GATE — which eval suites must pass at what threshold
      before entering the stage (offline, pre-promotion).
   2. GUARDRAIL METRICS — live metrics with numeric bounds that must
      hold DURING the stage (quality proxy, complaint rate, p95
      latency, cost/day), each with a measurement window.
   3. KILL CRITERIA — conditions triggering immediate rollback
      without a meeting, each with the single person authorized to
      pull it (and the on-call fallback).
   Make kill criteria objective: numbers and time windows, not
   "significant degradation."
   

   Sample output for one stage:

   Copy

   STAGE: 5% random
   Eval gate: full golden suite ≥ launch bar on the exact prompt+model
   version being promoted.
   Guardrails (7-day window): acceptance ≥ 80% of design-partner
   baseline · complaint rate < 0.5% of sessions · cost/day ≤ 1.3×
   forecast · p95 ≤ 6s.
   Kill (immediate, no meeting): any PII leak report · complaint rate
   > 2% in any 6h window · cost/day > 3× forecast · screenshot-class
   output verified by on-call PM. Authorized: PM (you); fallback:
   eng on-call.
   

3. 3.

   Write the rollback procedure as a runbook, not a sentence. "Roll back" must name the unit: flag off (seconds, safest, feature disappears), prompt version revert (minutes — only works if prompts are versioned per pb-51), model version pin (minutes, needs the old version still provisioned), full config revert (prompt + model + retrieval together — the only combination you've actually tested). Default rule: kill switch first, diagnose second, partial reverts only in daylight.

4. 4.

   Decide the comms lines before you need them. Three pre-written snippets: in-product notice if the feature vanishes mid-session, support macro for "where did it go?", and internal announcement for a pulled stage. Users forgive a disappeared beta feature; they don't forgive silence about wrong outputs.

5. 5.

   Rehearse the rollback once at dogfood stage. Actually pull the flag, actually revert a prompt version, time both, and note what broke (cached responses? stale clients?). pb-52's incident drill builds on this; the rollout plan is only real after one rehearsal.

Make it yours

• Statsig-native variant: encode guardrail metrics as Statsig's automatic rollback conditions so a breached bound reverts the gate without waking anyone — keep human-judgment kills (brand damage) manual, automate the numeric ones.
• Shadow-mode stage zero: for high-stakes features, add a stage where the model runs on live traffic but outputs are logged, not shown; you get real-distribution eval data with zero user exposure, and your 5%-stage baselines stop being guesses.
• Enterprise variant: replace random-percentage stages with tenant tiers (internal → friendly accounts under NDA → general), and add a per-tenant opt-out honored at every stage — enterprise admins will ask.

Quality check

• Every gate is numeric with a windowany gate containing "significant" or "acceptable" will be argued mid-incident; rewrite it now.
• Kill authority is one named person per criterionkills decided by committee are stages that don't kill.
• The rollback unit is explicitflag, prompt, model, or full config; and the reverted combination is one you previously ran evals on.
• Eval gates reference exact versionsthe suite passed on the artifact being promoted, not on last week's prompt.
• The rehearsal happenedthere's a timed record of one real flag-pull and one real prompt revert before external users exist.