Write the Pilot Charter with Kill Criteria Agreed Before Launch

Workflow area: AI Opportunity & Feasibility · Tier: Advanced · Hands-on time: ~40 minutes

The moment a pilot goes live, incentives corrupt measurement: the sponsor needs it to work, the vendor needs it to work, and the team that built it needs it to work. The only time you can get an honest definition of failure is before launch. That is what the pilot charter is — a pre-registration of the experiment, written by the analyst, agreed by the sponsor, that turns "let's pilot it and see" into a test that is allowed to conclude no.

When to use — and when not to

• Use it for every AI pilot, before anything is deployedif the pilot already started, write the charter this week anyway; a late charter beats a permanent zombie.
• Use it when a vendor proposes a "free pilot"free pilots are sales motions; a charter with kill criteria converts one into an experiment your org controls.
• Use it when the sponsor's conviction is strongesthigh conviction is exactly when thresholds get agreed easily, because everyone expects to clear them.
• Skip the full ceremony for a feasibility spike (pb-41) — a day of prompting needs a verdict memo, not a charter; the charter is for pilots touching real users or live work.
• Skip it as a substitute for governancethe charter defines the experiment; oversight requirements, data handling, and incident paths still come from pb-45, pb-50, and pb-52.
• Do not run an uncharted pilot to "gather requirements"that is discovery wearing production risk; do structured elicitation instead.

The recipe

1. 1.

   Extract the hypothesis from the business case and make it falsifiable. "Improve claims handling with AI" is a hope. A hypothesis names the mechanism and the number: "AI-drafted first responses will cut median handling time from a measured 14.2 min to under 10 min, without raising the QA error rate above its 2.1% baseline." Baselines must be measured (pb-38), not recalled — if there is no baseline, the pilot's first two weeks are baseline capture, and the charter says so.

2. 2.

   Draft the charter with AI, then negotiate the numbers by hand. Feed the business case and process context to your assistant:

   Copy

   Draft a one-page AI pilot charter from the material below.
   Sections:
   1. HYPOTHESIS — falsifiable, with the mechanism named
   2. SCOPE — pilot group, case types in/out, duration (weeks)
   3. SUCCESS THRESHOLDS — 2-4 metrics with numeric targets AND
      the measured baseline for each
   4. GUARDRAIL METRICS — what must NOT degrade (quality, CSAT,
      compliance), with current values
   5. MEASUREMENT METHOD — data source per metric, who measures,
      cadence, and the counterfactual (control group, A/B, or
      pre/post with stated limitations)
   6. KILL CRITERIA — conditions that end the pilot early
   7. SCALE CRITERIA — what result triggers a rollout decision
   8. ROLES — sponsor, measurement owner, decision forum, decision date
   Mark any number you had to invent as [NEEDS BASELINE].
   Material: [BUSINESS CASE + PROCESS NOTES]
   

   The AI produces structure; the thresholds themselves are negotiated with the sponsor and metric owners — that negotiation is the analysis.

3. 3.

   Design the counterfactual honestly. The question every pilot must answer is "compared to what?" Options in descending rigor: parallel control group working the same queue without the tool; alternating assignment (A/B on cases); pre/post with seasonality named as a limitation. Pick the strongest one operationally feasible and write down why. A pilot measured only against memory of how things felt before is unmeasurable by design.

4. 4.

   Make the kill criteria specific and pre-authorized. Good kill criteria trigger on evidence, not sentiment:

   Copy

   KILL CRITERIA (any one, verified over a full week):
   - Guardrail breach: QA error rate > 3.5% (baseline 2.1%)
   - Adoption floor: < 40% of pilot group using the tool in
     week 3 despite training
   - Safety event: any Sev-1 incident per the incident matrix
   - Economics: cost per case > 2x the charter estimate
   Decision: measurement owner reports the trigger; sponsor must
   kill or file a written exception within 5 working days.
   

   The pre-authorization clause is the teeth — the charter states in advance that a triggered criterion requires a decision, so ending the pilot is executing the plan, not admitting defeat.

5. 5.

   Get signatures and calendar the decision. Sponsor, metric owners, and the person who runs the pilot group sign the charter. Book the go/no-go review for the pilot's end date now, with the charter's decision table as the agenda. An unbooked decision meeting is how pilots become zombies.

Make it yours

• Lightweight version for small pilots: hypothesis, one success threshold, one guardrail, one kill criterion, decision date — five lines that still beat 95% of pilots.
• Analytics plan as an appendix: for pilots with instrumentation, attach the event-tracking spec (which events, which properties, which dashboard) so "measurement method" is implementable, not aspirational.
• Claude Project per pilot: keep charter, weekly metric snapshots, and decision log in one project — mid-pilot questions get answered against the charter, not against drifting memory.
• At portfolio scale: standardize the charter template and require it at the funding gate; a quarterly review of all charters versus actuals shows which sponsors' estimates deserve trust.

Quality check

• The hypothesis is falsifiablea specific mechanism, a numeric target, and a measured baseline; someone could prove it wrong.
• Every threshold has a baseline with a sourceno target is anchored to an asserted or remembered number.
• Guardrails cover what the optimists forgotquality, compliance, and customer experience metrics that the pilot could silently damage.
• The counterfactual is named, with its weaknessesand "pre/post, seasonality unknown" is written down if that is the truth.
• Kill criteria are numeric and pre-authorizeda trigger forces a documented decision; nobody has to be brave to stop the pilot.
• The decision meeting exists in calendarswith named attendees and the charter as the agenda.