Prompt Engineering Discipline: Versioned Prompts and Eval-Gated Iteration

Tier: Core · Workflow area: LLM Engineering Practice

This playbook sets up the minimum discipline that separates "we tweaked the prompt and it seems better" from an auditable iteration loop — a prompt registry in git, a fixed eval set, and a gate that blocks prompt changes which regress metrics.

When to use

• •
  Any prompt that runs in a pipeline (extraction, classification, labeling, SQL generation) rather than interactively.
• •
  A team is editing shared prompts and nobody can say which version produced last month's numbers.
• •
  You are about to start prompt-tuning and want to avoid overfitting to three cherry-picked examples.

Prerequisites

• •
  A git repo; optionally MLflow (OSS, self-hosted — includes a prompt registry) or Promptfoo for the harness.
• •
  50–200 input examples with expected outputs (or graded rubrics) for each prompt-driven task — reuse pb-14/15/19 eval sets.
• •
  Agreement in the team that prompt edits without eval runs do not merge. The discipline is social before it is technical.

The workflow

1. 1.

   Externalize every prompt. No prompts inline in application code. One file per prompt with metadata:

   Copy

   # prompts/ticket_extraction/v3.yaml
   id: ticket_extraction
   version: 3
   model: claude-opus-4-8
   schema: TicketV2          # schemas are versioned alongside
   changelog: "Added competitor-mention field; tightened summary length"
   template: |
     Extract the ticket fields. Rules: ...
     <ticket>{ticket_text}</ticket>
   

   The (prompt-version, schema-version, model) triple is the unit you deploy, log, and roll back. Every pipeline output row records it — that is what makes last quarter's numbers reproducible.

2. 2.

   Build the eval set deliberately. Stratify: ~60% representative cases, ~25% known-hard cases (past failures graduate into the set), ~15% adversarial/edge (empty input, wrong language, prompt-injection attempts). Freeze it; additions are append-only with dates so scores stay comparable.

3. 3.

   Score automatically. Three scorer types, in order of preference: exact/field match against expected values (extraction, classification); programmatic checks (parses, passes validators, length bounds); LLM-as-judge with a rubric only where the first two cannot express quality (summaries, tone). Judge calls use a cheap tier and a pinned judge model version.

4. 4.

   Wire the gate. A script (locally or CI) that runs the candidate prompt vs the incumbent over the eval set and reports per-slice deltas:

   Copy

   ticket_extraction v3 vs v2   (n=180)
   field_accuracy   0.94 → 0.96  (+0.02)   hard-slice: 0.81 → 0.86
   parse_rate       1.00 → 1.00
   adversarial_pass 0.93 → 0.87  (-0.06)  ← REGRESSION, merge blocked
   cost/1k          $0.41 → 0.38
   

   The rule: no merge on any slice regression without an explicit, written waiver. Aggregate-only reporting is how regressions ship.

5. 5.

   Iterate with hypotheses, not vibes. Each iteration: read 10 failing cases → write one hypothesis ("model confuses severity when multiple issues are listed") → make one targeted change → rerun the gate. One change per version; batched edits make attribution impossible.

6. 6.

   Close the loop from production. Sampled production failures (pb-16's QA queue) get minimized into eval cases and appended to the hard slice. The eval set should grow a few cases per week; a static eval set goes stale in a quarter.

Tool variants

• MLflow 3 (OSS): prompt registry + GenAI evaluation + tracing, fully self-hosted — the zero-SaaS-spend default for this track.
• Promptfoo: YAML-declared evals with matrix runs across prompts × models; excellent as the CI gate (promptfoo eval --fail-on-regression).
• LangSmith / Agenta / Braintrust: managed alternatives with UI-based diffing and review queues; same discipline, hosted convenience.
• Bare pytest: a parametrize over eval cases hitting the API with cached responses is a legitimate v1 — the gate matters more than the framework.

Verification

• •
  Every production prompt resolves to a versioned file; grep the codebase for inline prompt strings and find zero.
• •
  Rerunning the incumbent prompt on the frozen eval set reproduces its recorded scores within noise (validates the harness itself).
• •
  The gate demonstrably blocks: introduce a deliberate regression (delete a rule from the prompt) and confirm the merge fails.
• •
  Pipeline outputs carry (prompt-version, schema-version, model) — pick a random historical row and successfully identify exactly what produced it.

Pitfalls

• Eval sets of 5 favorite examples. Below ~50 cases, deltas are noise and you will oscillate between versions. Bootstrap CIs on scores if in doubt.
• Judge drift. LLM-as-judge scores are not comparable across judge-model versions; pin the judge, and re-baseline all historical scores when you upgrade it.
• Optimizing the aggregate. A +2% overall that hides -6% on the adversarial slice is a net regression for production; always slice.
• Prompt changes riding along with model upgrades. Change one variable at a time: model bump with frozen prompt first, then prompt tuning on the new model.
• Secret prompt forks. If an engineer keeps a "better" local variant, your registry is dead; make the gated path faster than the workaround.

Restricted-environment variant

• •
  MLflow + Promptfoo + pytest are all self-hostable; the entire discipline runs on-prem with a local model serving as both subject and judge.
• •
  Eval sets built from client data are client data — store them in the engagement's environment, not the shared team repo; keep a sanitized public-data mirror for teaching the pattern.
• •
  Judge calls on sensitive outputs use the local model; accept the weaker judge and compensate with more programmatic checks.

Success criteria

• •
  Prompt registry live, eval set frozen and stratified, gate wired into the merge path, provenance recorded on every output row.
• •
  At least one full iteration executed: failure analysis → hypothesis → change → gate → merge, with the delta table saved.
• •
  One regression caught by the gate (deliberate or real) — proof the system bites.
• •
  A new team member can propose, evaluate, and ship a prompt change without asking anyone how.