Write the PM-engineering contract that makes prompt changes governable: prompts in git, the golden eval set as merge gate, a change-log format with eval deltas, canary and rollback triggers — with the PM's role defined precisely. Then audit a synthetic change log containing two planted violations, and cite them by line.
Kabir's support copilot now has everything the last eight labs built: a taxonomy, an eval set, an aligned judge, a failure catalog, a rollout plan. What it does not have is governance — last month an engineer "tweaked the tone" of the production prompt from a shared doc on a Friday, and nobody can say which version was live during Tuesday's complaint spike. The fix is not heroics; it is a contract. You will write it, define who owns what, and then prove it has teeth by auditing a change log the way the contract demands.
STATUS QUO (what governance looks like today)
- Production prompt lives in a shared doc; anyone on eng can edit.
- copilot-eval-v1 exists (40 cases) but running it is "encouraged".
- Declared-but-informal rollback trigger, from the rollout plan:
"7-day agent edit rate above 25% (baseline 12%) triggers rollback
review within one business day."
- Canary: new prompts go to 5% of traffic for 48h "when we remember".
CHANGE LOG - support copilot prompt, last 6 weeks (as kept today)
L1 v3.1 May 02 Add policy-card grounding eval 83% -> 91% canary ok
L2 v3.2 May 09 Boundary threshold rules eval 91% -> 93% canary ok
L3 v3.3 May 16 Shorten replies (CS request) eval 93% -> 92% canary ok
L4 hotfix May 19 "Tone tweak - friendlier
closings" (eng, Fri 18:40) evals: n/a - copy-
only change
L5 v3.4 May 23 Add Hindi language rule eval 92% -> 94% canary ok
L6 canary note May 24-31: agent edit rate 19%, 22%, 26%, 28%,
27%, 29%, 28%, 27% (daily, 7-day avg 26.6% by May 31)
L7 v3.5 Jun 02 New refund macro in prompt eval 94% -> 94% canary ok
L8 note Jun 04: edit rate still 26-28%; CS lead asks "is this
the new normal?" - thread archived, no action
L9 v3.6 Jun 09 Compress system prompt 30%
(token cost project) eval 94% -> 89% shipped
("regression is mostly
tone cases, acceptable")
L10 v3.7 Jun 13 Re-add two tone rules eval 89% -> 93% canary ok
Audit the log first — cold read. Before writing any contract, read L1–L10 against the status quo's own declared rules. Log every violation you find with its line number and the rule it breaks. Two are planted as graded catches; find as many as you can, but rank yours by severity. Fifteen minutes, no AI — auditing is a PM literacy, not a delegable chore.
Write the contract's foundation: where prompts live. Three clauses: prompts are code — versioned in git, tagged per release, no direct-edit path to production (the Friday-evening shared-doc edit becomes structurally impossible, not just discouraged); every change ships through a pull request with a named reviewer; the live version is queryable at any moment ("which prompt was live Tuesday 14:00?" has a one-command answer). Note which log entry this forecloses.
Make the golden eval set the merge gate. Clauses: copilot-eval-vN runs automatically on every prompt PR — no exemptions, including "copy-only" changes (tone edits are behavior edits; Lab 07's taxonomy has a whole cluster proving it); the PR records before/after scores overall and per failure-mode category (L9's "mostly tone cases" hid a category collapse inside a 5-point headline drop — per-category deltas make that visible and force an explicit decision); score drops below the launch bar block merge, and any knowing override requires the PM's written sign-off with a revert-by date.
Define the change-log format as a schema. Every entry: version, date, author, reviewer, one-line intent, eval delta (overall + per category), canary window and its production metrics (edit rate, escalation rate, judge-sample score), decision (ship / iterate / rollback), and link to the diff. An entry missing any field is not "thin documentation" — it is an unauthorized change, and the contract says so. Rewrite L4 as it should have looked, as an example row in the contract.
Write the canary and rollback clauses with tripwires that fire themselves. Canary: every change, 5% for 48h minimum, with the three production metrics auto-compared to baseline — "when we remember" becomes "the pipeline remembers." Rollback triggers: restate the edit-rate trigger (7-day average over 25% → rollback review within one business day, automatically filed as an incident, not dependent on someone noticing a thread); add a judge-sample floor and a Sev-1 safety trigger (instant, from Lab 10). The design principle to state explicitly: a trigger that requires a human to be watching is a dashboard, not a trigger.
Draw the PM ownership line precisely. The contract's most-argued section, three lists: the PM owns — the launch bar (what score ships), the eval cases and their pass criteria (additions via error analysis, the no-deletion rule from Lab 08), severity classifications, override sign-offs, and the decision in every rollback review; engineering owns — the pipeline, the runners, the git mechanics, canary infrastructure, and the rollback execution path; jointly owned — the change-log discipline and the postmortem when a trigger fires. The anti-pattern to name: a PM who "owns quality" but not the cases owns a vibe; a PM who tries to own the pipeline owns a bottleneck.
Re-audit the log against your contract, line by line. Now the graded pass: walk L1–L10 and cite every violation with the clause it breaks. The two planted ones must appear with line numbers: the eval-less change and the trigger that never fired. Then answer the connecting question the log is engineered to raise: what is the most plausible cause of the edit-rate rise in L6, given its timing — and which two clauses of your contract would have caught it within 48 hours instead of never?
Pressure-test and close. Prompt your AI: "Here is a prompt-governance contract. Find the loopholes: changes that dodge the eval gate, triggers that can be argued around, ownership gaps between PM and engineering." Patch the two best findings (classic loopholes: retrieval-content and model-version changes routed around the "prompt" gate — extend the gate to any behavior-affecting change; and rollback reviews that always conclude "monitor further" — require each review to end in rollback, fix-forward with a date, or a written new-baseline decision). Version the contract v1.0, date it, and add its own change rule: the contract amends by PR too.
Write the change-log schema so that a violation is visible as a blank cell — governance that depends on reading prose gets skipped under deadline, but an empty "eval delta" column indicts itself.
A complete lab has all of the following:
Prompts are code: versioned, reviewed, eval-gated on every change with no copy-only exemptions, canaried against production metrics, and rolled back by triggers that fire themselves. The PM owns the bar and the cases; engineering owns the pipeline; the change log is where both prove it — and an audit that cites violations by line is how you know the contract is real.