Prompt and Model Version Management with Canary and Rollback

Tier: Advanced · Workflow area: Serving & LLMOps

This playbook builds the release pipeline DevOps-for-GenAI postings describe: prompts and model choices in a registry with versions, CI evals as the merge gate, staged canary rollout via gateway weights, and automatic rollback on eval regression.

When to use

• •
  More than one person edits prompts, or prompts change more than once a month, on anything user-facing.
• •
  A prompt change has already caused a regression you discovered from users rather than from tests.
• •
  You have the pieces this composes: an eval harness in CI, a gateway (pb-48), and observability with online scores (pb-49).

For an internal notebook prompt only you use, a git-committed text file is version management enough.

Prerequisites

• •
  MLflow ≥3 (self-hosted fine) — its prompt registry gives versioned, immutable prompt objects with aliases.
• •
  Your CI eval setup (golden dataset + scorers, thresholds already calibrated).
• •
  The pb-48 gateway or an app-level router capable of weighted traffic splits.
• •
  pb-49 online eval sampling — the canary verdict comes from these scores.

The workflow

1. 1.

   Register prompts; stop inlining them. Move each production prompt into the MLflow prompt registry (mlflow.genai.register_prompt(...)); the app loads by alias (prompts:/support-answer@production) at startup or per-request-cached. Model choice + generation params live alongside as versioned config. From here on, "edit the prompt" means "register a new version," never "edit the string in code."

2. 2.

   Gate new versions in CI. A prompt PR (template change in the repo that registers on merge) triggers the eval harness against the new version: golden-set scores must meet absolute thresholds and not regress vs the current production version by more than your margin. Fail the check, block the merge — identical discipline to a failing unit test.

3. 3.

   Canary by alias + traffic weight. Promotion is moving the candidate alias, then splitting traffic at the gateway: 5% → 25% → 100%, each stage held long enough to collect N sampled online scores (from your pb-49 pipeline) at the stage's traffic share. Tag each request/trace with the prompt version so scores are attributable per version — without that tag, canary analysis is impossible.

4. 4.

   Define the automatic rollback rule numerically. Example: "if candidate's sampled faithfulness mean over ≥50 scored requests is >3 points below production's trailing mean, or error rate doubles, set weight to 0 and re-point candidate at the prior version." Implement as a small scheduled checker that reads scores and calls the gateway API — automation matters because prompt regressions surface at 2 a.m. exactly as often as code regressions.

5. 5.

   Drill both paths. Ship a genuinely better prompt version through the full pipeline; then ship a deliberately degraded one (remove a key instruction) and watch CI catch it — and, if you force it past CI, watch the canary checker roll it back. Save the evidence of both.

6. 6.

   Extend the same pipeline to model swaps. A model upgrade (provider bump, or hosted → self-hosted pb-46 endpoint) rides the identical path: register config version → CI evals → canary → rollback rule. This is where the pipeline pays for itself, because model swaps are bigger behavioral changes than prompt edits.

Tool variants

• Git-as-registry (prompts as files, versions as commits, aliases as tags) + a config service — workable minimum; you rebuild registry ergonomics yourself.
• Langfuse prompt managementOSS alternative with built-in prompt versioning and labels, pairs naturally if Langfuse is your pb-49 choice.
• LaunchDarkly-style feature flags for the traffic split when no gateway exists — same staged-rollout semantics at the app layer.

Verification

• •
  App contains zero inline production prompts — grep proves it; all loads go through aliases.
• •
  The degraded-version drill: CI blocked it; forced canary rolled it back automatically within one checker cycle.
• •
  Every trace in pb-49 carries the prompt version tag; per-version score comparison is a filter, not a forensic project.
• •
  Rollback executed in under a minute, by metadata change only, with an audit trail of who/what/when.

Pitfalls

• Canary verdicts from too few samples. At 5% traffic on a low-volume route, 50 scored requests can take days — size stage durations from arithmetic, or raise the canary share.
• Evaluating the new prompt on a golden set that encodes the old prompt's phrasing. Golden sets test task outcomes, not string similarity to previous outputs; audit for this before trusting a "regression."
• Alias updates without cache invalidation. If the app caches the prompt at startup, moving the alias does nothing until restart — decide and document the propagation path.
• Treating rollback as failure. A rollback that fires is the system working; the post-mortem target is the gap that let the version reach canary, not the person who shipped it.

Restricted-environment variant

MLflow registry, CI runner, gateway, and observability all self-host — the full release pipeline runs air-gapped, with judge calls on a local model. For client work, map the canary stages onto the client's change-management gates (their CAB approves the 5%→100% promotion); the pipeline generates exactly the evidence such boards ask for.

Success criteria

• •
  Prompts and model configs versioned in a registry, loaded by alias, gated by CI evals.
• •
  Canary with per-version online scores and a numerically-defined automatic rollback — both demonstrated in drills.
• •
  One real improvement and one forced regression shipped through the pipeline with saved evidence.
• •
  Time-to-rollback measured in seconds/minutes and documented.