Measure AI Impact Honestly — Paired Metrics, Real Baselines, and a Ban on Vanity Numbers

Workflow area: Measurement & ROI · Tier: Advanced · Hands-on time: ~45 minutes to design

The pressure to produce an AI-impact number is enormous — the CFO wants ROI, the board wants a percentage, the vendor supplies one helpfully pre-calculated. The trap is that the easy numbers are the meaningless ones: acceptance rate measures how often engineers accept suggestions, not whether those suggestions survived review, shipped, or caused incidents; lines of code has never measured anything except lines of code. An honest system is slower and less quotable, and it is the only thing that survives a skeptical audit — which is precisely what makes it valuable (pb-39 spends this credibility).

When to use — and when not to

• Use it before the rollout, ideallythe baseline you capture pre-adoption is unrepeatable; every week of rollout without one narrows what you can ever honestly claim.
• Use it when someone upstream asks "what are we getting for the AI spend?"the question will arrive; this system is the difference between an answer and an anecdote.
• Use it to replace a vanity dashboard already in circulationif acceptance-rate charts are reaching execs, this playbook is remediation, and step 5's written ban is the deliverable to publish first.
• Skip measurement fishing expeditionsdecide upfront which decisions the data will inform (renew the contract? expand seats? change tools per pb-35?); metrics collected without a consuming decision become surveillance with extra steps.
• Skip individual-level analysis entirelynot as a limitation but as a design principle; the moment impact data identifies individuals, honest usage reporting ends and pb-38's trust contract is broken.

The recipe

1. 1.

   Choose paired metrics — no speed without its quality partner. Every velocity claim ships handcuffed to the metric that would expose its cost:

   Copy

   PAIR 1: Change lead time        ↔  Change failure rate
   PAIR 2: PR throughput           ↔  Revert/hotfix rate on those PRs
   PAIR 3: Time-to-merged          ↔  Escaped defects per release
   PAIR 4: Delivery pace (survey)  ↔  Maintainability confidence (survey)
   

   Frameworks exist so you don't invent this from scratch: DX Core 4 or SPACE both force the balance across speed, quality, and developer experience. The rule is structural — any report showing one side of a pair without the other is incomplete by definition, and the template should make that visually obvious.

2. 2.

   Capture the baseline before the tools land. Minimum one clean quarter of the paired metrics, at team granularity, with the measurement definitions frozen and written down ("deploy means X, counted by Y"). If the rollout already happened, say so in every report forever — a reconstructed baseline is an estimate and must be labeled one; the honest phrase is "we cannot cleanly attribute the pre/post delta," and it costs less credibility than one debunked chart.

3. 3.

   Run the confound log as measurement infrastructure. The same log as pb-36, promoted to first-class citizen: dated entries for reorgs, holidays, attrition, incident weeks, freezes, definition changes, and each rollout stage. Every impact report gets a confound section generated against this log, and any delta smaller than its confounds gets reported as "within noise" — writing "no detectable effect yet" in month two is what makes "detectable effect" believable in month six.

4. 4.

   Roll up at team level, compare only self-to-self. Team-level aggregates, trended against that team's own baseline — never team-vs-team league tables, which measure codebase age and domain difficulty while claiming to measure AI benefit, and never per-person cuts (pb-38 sets the floors). Complement the telemetry with a quarterly survey (SPACE-style satisfaction and friction questions), because engineers can see impact and damage that pipelines cannot: "the code ships faster and I trust it less" is a finding no DORA metric will ever emit.

5. 5.

   Publish the ban list — in writing, with reasons. A short section in the measurement doc, signed by you:

   Copy

   NOT ACCEPTED AS IMPACT EVIDENCE:
   - Suggestion acceptance rate  (measures acceptance, not value;
     gameable by config; vendor-favored for a reason)
   - Lines/volume of AI code     (volume is cost, not benefit)
   - Self-reported hours saved   (uncontrolled, demand-biased;
     usable as sentiment, never as ROI arithmetic)
   - Any per-person productivity figure (see telemetry policy)
   

   The ban does double duty: it pre-empts the vendor QBR slide, and it tells your engineers in writing that you know the difference — which is worth more adoption goodwill than any launch email.

6. 6.

   Report on a cadence with the pb-36 brief attached. Quarterly impact reports: the pairs, self-comparisons against baseline, the confound section, survey themes, and the honest headline — which for the first two quarters is usually "early, mixed, and within noise for most teams." Resist the interim extrapolation; the system's whole value is that when it finally does show a clean effect, nobody can argue with it.

Make it yours

• Small team or startup: two pairs (lead time ↔ failure rate, plus the survey pair), a spreadsheet, and the ban list — the discipline scales down intact; what doesn't scale down is patience, so pre-commit to two quarters before anyone asks the dashboard for a verdict.
• Large org or GCC: engineering-intelligence platforms (DX/Jellyfish/LinearB/Swarmia-class) automate the pairs and rollups — configure their AI-impact modules to your definitions rather than accepting defaults, disable per-person views at the platform level where possible (pb-38), and be prepared to defend the ban list against a parent company or client asking for exactly the vanity numbers on it; the pb-36 brief is how those conversations go well.
• Mid-rollout orgs with no baseline: use staggered adoption as your instrument — teams adopting later serve as rough contemporaneous comparisons for teams adopting earlier, imperfect but honest if you name the selection effects (eager teams differ from cautious ones) in the confound section.

Quality check

• No unpaired speed number has shipped anywhereaudit the last quarter's decks and dashboards; one naked velocity chart in an exec deck means the template isn't enforcing the pairing.
• The baseline's definitions still holda counting change anywhere in the pipeline (new CI, redefined "deploy") silently invalidates trends; the confound log should catch it, the quarterly report should disclose it.
• "Within noise" appears in your reportsa measurement system that only ever finds positive effects is a marketing system; the null results are the credibility you are banking.
• Survey and telemetry get equal billingif the quarterly report quotes charts but not the maintainability-confidence trend, the quality half of pair 4 has quietly fallen off.
• The ban list has been testedsomeone (a vendor, an exec, an enthusiastic EM) has by now proposed an acceptance-rate slide; the system is working if the ban list, not an argument, settled it.