Measure AI Outcomes Honestly and Run Benefits Realization That Can Say No

Workflow area: AI Rollout & Governance · Tier: Advanced · Hands-on time: ~45 minutes

The business case promised numbers (pb-38); the pilot charter set thresholds (pb-42); benefits realization is where the organization finds out whether any of it was true — and it is the discipline most AI programs skip, because nobody wants to be the instrument that proves the flagship initiative flat. That is precisely why the BA designs the measurement before launch and pre-commits the review to honest verdicts: measurement designed early is evidence; measurement designed late is advocacy.

When to use — and when not to

• Use it at design time for every funded AI initiativethe measurement plan is a launch deliverable, sitting beside the requirements pack; if the system is already live, start with baseline archaeology this week, because every week makes it worse.
• Use it when the sponsor claims success from anecdotestwo enthusiastic quotes and a usage graph are not benefits; this framework converts the claim into something checkable either way.
• Use it to feed the portfolio decisionquarterly verdicts across initiatives tell the org where the next rupee of AI investment should and should not go.
• Skip the full machinery for cheap experimental toolsa $30/month assistant needs a one-line quarterly check-in, not a benefits model; reserve the rigor for initiatives with real cost stacks.
• Skip measurement theaterif leadership has pre-decided the initiative succeeded and no verdict can change anything, say so explicitly rather than lending the process your credibility.
• Do not measure people when you mean to measure the systemthe unit of analysis is the process and its outcomes; individual league tables corrupt the data and poison adoption (pb-49).

The recipe

1. 1.

   Capture the baseline before launch, with provenance. For every promised benefit, record the pre-launch value, measurement method, period, and data source — cycle times, error rates, cost per case, CSAT — the same measured-not-asserted discipline as pb-38, now with a deadline: launch. Where volumes are seasonal, capture a year of history or note the limitation. File the baseline pack where the quarterly review will find it; an unfiled baseline is a future argument.

2. 2.

   Build the indicator chain from the benefit logic. Every benefit claim implies a causal chain — "AI drafting → faster handling → lower cost per case → headcount redeployment" — so instrument each link, not just the end: leading indicators (workflow adoption share, override rate bands from pb-49, eval scores holding through change gates per pb-50) and lagging outcomes (the business-case numbers). The chain is the diagnostic: when the lagging outcome disappoints, the leading indicators tell you which link broke — nobody uses it, they use it but distrust it, or they use it and it simply doesn't move the money.

3. 3.

   Design the counterfactual and name the confounders. "Compared to what?" outlives the pilot: keep a control lens where feasible (staggered rollout across teams is the pragmatic one — later waves are earlier waves' control), and pre-register the confounders that will be claimed later — seasonality, mix shift, the parallel process change, attrition. Two effects deserve standing lines in every report: the novelty curve (early enthusiasm inflates months 1–2; measure steady-state from month 3 and say so on the chart) and displacement (time saved in the measured step that reappears as rework or verification load downstream — pb-49's matrix says where to look).

4. 4.

   Automate the evidence, hand-write the verdict. Build the quarterly pack with AI against the plan, keeping the judgment human:

   Copy

   Draft the quarterly benefits-realization report from the data
   below. Structure:
   1. SCOREBOARD — per promised benefit: baseline, target,
      actual, confidence (MEASURED / ESTIMATED / ANECDOTAL)
   2. INDICATOR CHAIN — leading indicators vs their bands;
      which links in the benefit logic are confirmed / broken
   3. CONFOUNDERS — pre-registered list, with evidence for or
      against each as the explanation
   4. COSTS — actuals vs the full-stack estimate (licenses,
      tokens, integration, change, evals, review labor)
   5. VERDICT OPTIONS — the evidence case for each of: ON TRACK /
      FIX (with named fixes) / KILL, without recommending one
   Rules: no benefit moves to MEASURED without a data source
   cited. Do not smooth bad news. Mark data gaps as gaps.
   Data: [BASELINE PACK + METRICS EXPORTS + COST ACTUALS]
   

   The model assembles; you audit the numbers (reproduce at least one per section — pb-27 discipline) and write the verdict yourself, because the verdict is accountability, not analytics.

5. 5.

   Run the review with pre-committed verdict rules. The quarterly forum works only if the exits were agreed at design time: thresholds carried forward from the charter's kill/scale criteria (pb-42), a rule that two consecutive FIX verdicts without improvement escalate to KILL-or-rebuild, and explicit permission — written into the plan's charter — for the review to conclude the initiative failed without that being anyone's career event. Publish verdicts to the portfolio; celebrate honest kills in public, because the org that punishes them gets zombie metrics forever after.

Make it yours

• One-page scoreboard for executives: benefit, baseline, actual, confidence tag, verdict — the confidence column (MEASURED / ESTIMATED / ANECDOTAL) is the honesty device; watch what happens the first quarter everything is ANECDOTAL.
• Staggered-rollout analytics: where A/B is impossible, wave-versus-wave comparison in a spreadsheet gets you a defensible counterfactual with zero data-science dependency.
• Token-and-labor cost tracking: wire provider billing exports and review-labor hours into the cost section monthly — the benefit side gets all the attention, but the cost side is where AI business cases quietly die.
• At portfolio scale: identical scoreboard format across initiatives, one quarterly portfolio view — the pattern of verdicts (which use-case types keep failing, which sponsors' estimates hold) is the org's real AI strategy input.

Quality check

• Baselines predate launch and carry provenancevalue, method, period, source; anything reconstructed afterward is labeled as such and treated as weaker evidence.
• Every benefit has a confidence tagand the report's headline claims are built only from MEASURED rows.
• The indicator chain is instrumented end to enda broken lagging outcome can be localized to a link, not just lamented.
• Novelty and displacement have standing linessteady-state windows are marked on trend charts, and downstream rework was actually checked, not assumed absent.
• Costs are actuals against the full stackincluding tokens, evals, and human review labor; benefit-only reports fail review.
• The kill rule has teeth and a paper trailpre-committed thresholds, an escalation rule for repeated FIX verdicts, and at least one historical example the org can point to of a measured, blameless kill.