Write the Prompt-Governance Contract with Engineering

Workflow area: Agents & Responsible AI · Tier: Advanced · Hands-on time: ~45 minutes

Prompts are the strangest artifact in modern product work: they read like specs, deploy like code, and change behavior like both. In most teams they occupy a governance vacuum — edited in a dashboard by whoever's debugging, untested against anything, unannounced to the PM who owns the resulting behavior. The failure is bidirectional: engineers ship "harmless wording tweaks" that shift refusal behavior, and PMs demand changes verbally with no record of what was asked. The contract fixes the vacuum with one move: treat prompts exactly like code, with the PM's eval set standing where code review stands.

When to use — and when not to

• Use it the moment more than one person can change a promptthat's when "who changed this and why" first goes unanswered.
• Use it after your first silent regressionif a model or prompt change ever degraded behavior and nobody could say when it happened, the contract is overdue.
• Use it when PM-engineering friction centers on AI behaviormost of that friction is process ambiguity wearing a personality costume; a written contract de-personalizes it.
• Skip the full ceremony for one-person prototypesversion the prompt file and keep a changelog; the multi-party contract can wait for the second person.
• Skip clauses you can't yet operatedon't sign up for canary rollout before feature flags exist for prompts; write the contract to your current pipeline and add an upgrade path.

The recipe

1. 1.

   Inventory the current reality first. Twenty minutes with the tech lead: where does each prompt live (repo, dashboard, database, someone's snippet stash)? Who can change each? What happens between edit and production? What broke in the last three months? The contract negotiates from this inventory — a contract written against an imagined pipeline gets ignored by the real one.

2. 2.

   Draft the contract from the six-clause skeleton.

   Copy

   Draft a one-page prompt-governance working agreement between
   product and engineering. Plain language, no legalese — this is
   an internal team contract.
   Context: [TEAM SIZE, WHERE PROMPTS LIVE TODAY, EVAL TOOLING
   (promptfoo/Braintrust), OBSERVABILITY (Langfuse), FLAG TOOLING]
   Recent incidents: [WHAT WENT WRONG, BRIEFLY]
   
   Six clauses, each with: the rule, the mechanism enforcing it,
   and the named owner.
   1. SOURCE OF TRUTH — all production prompts in git; dashboard
      edits are for experiments only and cannot reach production.
   2. CHANGE PROCESS — every prompt change is a PR; PM is a
      required reviewer for changes to [NAMED behavior-critical
      prompts]; wording-only claims still run the gate.
   3. MERGE GATE — the golden eval suite runs on every prompt PR;
      merge requires scores at/above the launch bar; the PM owns
      the eval set's content, engineering owns the runner.
   4. CHANGE LOG — every merged change records: motivation, diff,
      eval delta (before/after per suite), and model version tested
      against. Auto-generated where possible.
   5. ROLLOUT — behavior-critical prompt changes ship behind a
      canary (X% traffic, Y hours) with named guardrail metrics.
   6. ROLLBACK — auto-revert triggers (guardrail breach thresholds)
      and the manual procedure, with the same authority list as
      our rollout plan.
   End with: what counts as "behavior-critical", the exception
   process for emergencies, and the quarterly review of this
   agreement itself.
   

3. 3.

   Negotiate the two clauses that always get pushback. Clause 2 (PM review) draws "this will slow us down" — concede review only for the named behavior-critical prompts (system prompts, refusal instructions, tone) and let utility prompts flow with just the eval gate; speed objections usually dissolve when the gate is automated. Clause 4's eval delta draws "extra work" — make the CI job write it into the PR automatically (promptfoo output pasted by the action), so the log costs nobody anything. A contract that survives is one where every clause has a mechanism, not a hope.

4. 4.

   Define "behavior-critical" as a list, not an adjective. Enumerate the actual prompt files/sections: the system prompt, safety and refusal instructions, persona and tone blocks, tool-use permissions phrasing, disclosure language (pb-49's annex text). Everything on the list gets PM review + canary; everything off it gets the eval gate only. Revisit the list quarterly — prompts migrate onto it after their first incident.

5. 5.

   Run one change through the full pipe as the ratification. Pick a real pending prompt improvement and walk it: branch, PR with PM review, eval run posted, merge, canary with Langfuse watching the guardrails, promote, changelog entry with the delta. Time each step and fix what's clunky now — the contract is ratified by the walkthrough, not the signatures. Publish the agreement where specs live, with the walkthrough PR linked as the worked example.

Make it yours

• Braintrust-centric variant: the merge gate becomes "PR links a Braintrust experiment comparing this prompt vs production at/above bar" — better when PMs and non-engineers author eval cases in the UI while the git contract stays intact.
• Prompt-management-platform variant: if prompts live in Langfuse's prompt registry rather than git, keep every clause but map the mechanisms — versioned registry entries as source of truth, promotion-to-production as the gated step, registry diffs as the changelog. The contract is about gates and ownership, not about git specifically.
• Multi-team variant: when several squads share one model integration, add a clause 7 — shared-prompt changes (the base system prompt) require sign-off from each consuming team's PM, and each team's eval suite runs in the gate. Painful, and much cheaper than the alternative.

Quality check

• Every clause names a mechanism and an ownera rule enforced by "we'll remember" is already broken; CI jobs, required-reviewer settings, and flag configs are what make clauses real.
• The behavior-critical list is enumeratedactual file paths or registry keys, not a category description someone will argue about mid-incident.
• The eval delta appears in every merged PRcheck the last five prompt PRs; if any lacks the before/after, the automation has a gap.
• Emergency exceptions have a debt processhotfixes can skip canary but must backfill the eval run and changelog within 24 hours, and the contract says so.
• The walkthrough happenedone real change traversed every clause end-to-end, timed, with the friction fixed rather than tolerated.