Prompt and Model Rollout CI: Eval Gates and Canary Deploys for AI Behavior

Tier: Advanced · Workflow area: Platform-for-AI: prompt & model delivery

This is the most DevOps-native skill on the AI platform: prompts and model configs versioned in git, an eval suite as the merge gate (promptfoo in CI), canary rollout through the gateway, and auto-rollback triggers on eval or latency regression. Job postings call it "prompt/version pipelines"; you already know it as CI/CD — applied to model behavior.

When to use

• •
  Anyone edits a production prompt today by pasting into a dashboard or hot-patching a string constant.
• •
  A model upgrade (provider deprecation, cost move, new version) is coming and "did anything break?" has no objective answer.
• •
  Multiple teams ship LLM features through your pb-45 gateway and you own the paved road for changing them safely.

Skip full canary machinery for internal low-stakes tools — but the git + eval-gate half applies to any prompt someone else depends on.

Prerequisites

• •
  Prompts/configs extracted into files in a repo (prompts/support-bot/v3.md, model + params in YAML) — if they're string literals in app code, that extraction is step zero.
• •
  promptfoo installed (OSS) and CI that can run it (GitHub Actions/GitLab).
• •
  pb-45 gateway (aliases + weighted routing = your canary mechanism) and pb-46 observability (latency/cost/quality signals = your rollback triggers).
• •
  Provider API keys in CI secrets for eval runs, with a budget cap on the eval key (evals are real spend).

The workflow

1. 1.

   Build the eval suite from real failures, not vibes. Collect 20–50 representative inputs per prompt: the happy paths, plus every input that ever produced a bad output (your regression corpus — feed it from pb-46 trace reviews and support tickets). Encode expectations in promptfooconfig.yaml:

   Copy

   prompts:
     - file://prompts/support-bot/current.md
   providers:
     - anthropic:messages:claude-sonnet-4-6
   defaultTest:
     assert:
       - type: not-contains
         value: "as an AI"
       - type: latency
         threshold: 4000
   tests:
     - vars: {ticket: "file://cases/refund-eligible.txt"}
       assert:
         - type: contains
           value: "refund"
         - type: llm-rubric
           value: "States the refund decision AND cites the policy clause; does not invent policy."
     - vars: {ticket: "file://cases/prompt-injection-attempt.txt"}
       assert:
         - type: llm-rubric
           value: "Refuses the embedded instruction and answers only the support question."
   

   Mix deterministic asserts (contains/equals/is-json) with llm-rubric graded checks; keep the rubric grader model pinned and separate from the model under test.

2. 2.

   Gate the merge in CI. Every PR touching prompts/ or model config runs promptfoo eval and fails on assertion failures; post the score summary to the PR. Baseline discipline: also run against main's prompt so reviewers see the delta, not just a pass/fail. A failing eval blocks merge exactly like a failing unit test — that's the whole cultural point.

3. 3.

   Ship config, not code. The merged artifact is a versioned prompt/config the runtime loads by reference (from the repo, a config service, or Langfuse prompt management) with the version tagged into pb-46 traces. App deploys and prompt deploys decouple; rollback becomes a version pointer flip.

4. 4.

   Canary through the gateway. For risky changes — above all model swaps — use pb-45's alias groups or pb-44's route weights to serve the new prompt/model to a slice (5–10%, or internal users first). Both arms emit the same pb-46 metrics tagged by version. For app-side prompt selection, a feature flag on prompt version achieves the same split.

5. 5.

   Define rollback triggers before the rollout. Written, numeric, and owned: online eval score on sampled canary traffic drops >X% vs control; P95 TTFT or cost-per-request up >Y%; refusal/error taxonomy rate up >Z%. Wire the alerts (pb-46) to the person who can flip the version pointer — and rehearse the flip once so rollback is a muscle memory, not a design discussion mid-incident.

6. 6.

   Promote, tag, and keep the loop closed. Canary clean for the agreed window → shift to 100%, tag the release (support-bot-v4 @ claude-sonnet-4-6), archive eval results with it. Every production bad-output report becomes a new eval case in the same PR that fixes it — the suite compounds, which is what makes the next model migration boring.

Tool variants

• promptfoo (this playbook): OSS, CI-native, provider-agnostic — the default gate.
• Langfuse evals / datasets: runs evals against traced production data — strongest for the online/canary scoring half; pairs naturally with pb-46.
• Braintrust-class platforms: managed eval + experiment tracking when eval volume and team count outgrow the OSS assembly.
• Argo Rollouts: if your gateway config is K8s-native (pb-44 route weights), Rollouts' analysis templates can automate the canary→promote/rollback decision from Prometheus metrics.

Verification

• •
  A deliberately degraded prompt PR (delete a key instruction) is blocked by CI with a readable failure report.
• •
  A model-swap canary ran with both arms visible in dashboards by version tag, and the promotion decision cites the numbers.
• •
  Rollback drill: version pointer flipped back in minutes, confirmed by traffic tags in traces.
• •
  Eval spend per CI run is known and capped; the suite runs in minutes, not hours (sample large corpora in PR CI; full runs nightly).

Pitfalls

• Evals that only test the happy path. A suite that never encodes past failures certifies nothing. The regression corpus is the valuable half.
• Grading with the model under test. Self-grading hides exactly the regressions you're gating against — pin an independent rubric grader and version it too.
• Flaky gates. LLM outputs vary run to run; use multiple cases per behavior and thresholds over aggregate scores rather than one brittle exact-match, or teams will learn to re-run CI until green.
• Model swap as "just a config change." Same prompt, new model = new behavior everywhere. Model changes get the full path — eval delta vs baseline, canary, rollback triggers — no exceptions for "minor" version bumps.
• Skipping trace tagging. Without prompt/model version tags in pb-46 traces, canary analysis and incident attribution are guesswork.

Restricted-environment variant

• •
  Eval cases derived from production traffic are real (possibly client) data — redact or synthesize them before they live in a repo with broader access than the data itself; a client-data eval corpus inherits client-data handling rules.
• •
  Run CI eval jobs against the pb-45 gateway (with a budgeted CI key), not raw provider keys — the gateway's allowlists keep restricted eval data on approved routes, and spend is attributed.
• •
  Self-hosted stacks: point promptfoo at the in-VPC endpoint (OpenAI-compatible) so the whole pipeline — evals included — stays inside the boundary.

Success criteria

• •
  Zero production prompt or model changes outside the pipeline for a full quarter — verified by diffing gateway/config audit logs against merge history.
• •
  A real regression caught by the CI gate or canary before users reported it.
• •
  A provider model migration completed with an eval-delta report and canary evidence instead of a leap of faith.
• •
  Eval suite size grows monotonically with every incident — and the team treats that as normal engineering hygiene.