Pick tools by structured trial instead of loudest voice, measure impact with paired speed-and-quality metrics instead of vanity numbers, and build ROI cases in ranges with falsification criteria — for a CFO who has already read that ~95% of orgs report zero GenAI ROI. Honest measurement is the scarce leadership skill this module trains.
Arjun, a director at a 200-engineer commerce platform, presented his AI progress deck to the executive team with genuine confidence. Slide three was the centerpiece, built from the vendor's own dashboard: suggestion acceptance rate 34%, PR count up 28% since rollout, seat activation at 91%. "Conservatively," he said, quoting the vendor's framing, "we're seeing a third of coding work assisted by AI."
The CFO, who had been reading skeptical coverage of enterprise AI returns all quarter, asked three questions. "If developers accept a suggestion and then rewrite it, does your 34% count it?" Arjun didn't know — it did. "Are PRs the same size as before the rollout?" He didn't know — they were smaller, so more PRs did not mean more delivered work. "And incidents?" That one he knew, because SRE had flagged it that morning: change-failure rate had risen two points since rollout, and nobody had connected the two charts. The meeting ended with the CFO asking for "numbers that would survive an audit," and Arjun's program — which was probably genuinely helping — now had a credibility problem no dashboard could fix.
Arjun's failure was not adoption, and it may not even have been impact. It was measurement: he presented activity as outcome, vendor metrics as evidence, and speed without quality. This module is the discipline that would have let him answer all three questions — starting before the tool was even chosen.
Tool selection usually happens by advocacy — the loudest senior engineer's favorite wins. Survey data from practitioner communities shows high tool churn and a consistent finding: stated preference and measured benefit diverge. The structured alternative is a bake-off, and it costs four to six weeks you will spend anyway on regret if you skip it.
The protocol has five parts. Candidates: two or three tools (Copilot, Cursor, Claude Code, Windsurf, Gemini Code Assist-class), including your incumbent if you have one — incumbency is a bias to control for, not a default winner. A representative task battery: real work from your backlog spanning feature work, debugging, tests, and legacy code — not greenfield demos, which flatter every tool. A rubric fixed in advance: quality (does it survive review), speed (task completion), security (what it does with risky patterns), and developer experience — weighted before you see results, so the weights can't chase a favorite. Cohort design: comparable engineers on comparable work, with a baseline period. A decision rule: what score gap picks a winner, what triggers "neither," and who decides.
The output artifact is a decision memo, not a slide: what was tested, on what work, with what rubric and weights, what the data showed, and the recommendation with its dissent recorded. The memo is what turns a bake-off from a one-time purchase ritual into an institutional capability — next year's re-evaluation starts from this year's memo, and the engineer who disagreed loudly gets their objection preserved rather than steamrolled, which is what keeps them participating honestly in the next trial.
Then hold the line when results surprise you. The classic bake-off outcome — Lab 7 seeds it deliberately — is that the loudly preferred tool loses on measured quality and review burden. Preference still matters (a tool engineers hate will not be used), but it enters the rubric as the DX row, not as a veto over the evidence.
Run the vendor-evaluation questions before the bake-off shortlist, not after selection: training-on-your-data rights, retention, residency, indemnity, model deprecation, and lock-in. AI vendors differ on exactly the terms standard SaaS procurement never probes, and a tool that wins the bake-off but fails the no-training clause was never a candidate.
The measurement stakes are set by the research: DORA finds individual gains that don't automatically become org gains; LeadDev finds 60% of leaders seeing no significant team-level boost; and Forsgren's warning — "AI broke our developer productivity metrics" — names the mechanism, because AI inflates exactly the activity counts naive dashboards report. Design against all three with four commitments:
Frameworks — SPACE, DX Core 4 — give you validated structure balancing speed, effectiveness, quality, and impact; engineering-intelligence platforms (DX, Jellyfish, LinearB, Swarmia-class) operationalize them. But the method matters more than the tooling: a spreadsheet plus a quarterly survey implements honest measurement at zero cost, which is also the India-practical path when platform pricing at USD rates doesn't clear.
And then ban the vanity four, in writing:
| Vanity metric | What it actually measures | Why it fails as an impact claim |
|---|---|---|
| Acceptance rate | How often developers keep a suggestion initially | Counts code accepted then rewritten; vendors present it as productivity; it is engagement telemetry |
| Lines of code | Verbosity | AI inflates it on demand; more code is a cost, not an outcome |
| PR count | Activity and batch-size choices | PRs shrink under AI assistance; count rises while delivered scope may not move |
| Seat activation | Procurement success | Measures licenses touched, not value created — adoption theater's favorite number |
Each is fine as operational telemetry — acceptance rate can hint at tool fit, activation can flag wasted seats. The ban is on presenting any of them as evidence of delivery impact. Lab 8's answer key includes debunking a planted vendor claim that "34% acceptance = 34% productivity gain" — Arjun's slide three, graded.
Publish the ban as a one-page memo with the table above in it, and send it up as well as down. The downward audience needs to know these numbers won't be used on them; the upward audience needs inoculation before a vendor QBR presents acceptance rate as productivity to your CEO. The memo costs an afternoon and pre-empts a quarter of bad decisions.
Adoption data is worth collecting — which workflows AI has reached, where usage stalls — but the collection design determines whether you get truth. The episode to learn from: employee backlash to AI-usage tracking framed as accountability shows that surveillance framing destroys the very adoption it tries to measure. Four rules preserve trust:
Trust is an input to measurement quality: surveilled engineers game telemetry, and gamed telemetry is worse than none.
Per-person usage dashboards, quietly collected, ambiguously connected to reviews. Result: acceptance-rate theater, prompt-count padding, and a team that treats every metric as adversarial — while the real signal drowns.
Published purpose, team-level floors, teams see their data first, written no-ratings promise. Result: honest usage patterns, engineers who volunteer failure cases, and telemetry you can actually steer by.
Your CFO's prior is set by the base rate: roughly 95% of organizations report zero measurable GenAI ROI so far. That is not a reason to skip the ROI case — it is the reason to build one that looks nothing like a vendor deck.
Count all the costs: licenses and seats, API and inference spend, enablement time (training hours, champion time, the productivity dip during ramp), and the ongoing tax of review and governance. Enablement is the line vendors omit and CFOs notice missing. Claim benefits as measured ranges, never points: "measured cycle-time improvement of 6–12% on piloted teams, holding change-failure flat" survives scrutiny; "30% productivity gain" invites the three questions Arjun couldn't answer. Time reallocation beats time "saved" — where did the freed hours demonstrably go? Present scenarios with sensitivity — conservative, expected, optimistic — showing which assumptions move the answer most. And close with the section that builds more credibility than any benefit number: "what would falsify this" — the metrics and thresholds that would tell you the investment isn't working, and the date you'll check. A leader who names their kill criteria in advance is a leader whose next ask gets approved. Lab 9 builds this model against a planted vendor one-pager claiming a 4x point-estimate return.
Two ongoing disciplines complete the money story. Seat and cost governance: quarterly usage-vs-seat audits (30–50% dormant seats is a common first-audit finding), tier right-sizing, per-team API budgets with alerts, and a chargeback or showback model once spend is material. This lands hardest in India, where USD per-seat pricing multiplied by pyramid headcount breaks budgets that Western per-seat math tolerates — free tiers, bundled suite AI, and self-hosted options are legitimate line items in the model, not admissions of failure. The quality-regression watch: churn, rework, duplication, review depth, and defect escape tracked as AI usage grows, with thresholds agreed in advance — and a crossed threshold triggers a review of the standards (Module 5), not a hunt for whoever wrote the code. Practitioner surveys already record the failure mode to avoid: "codebase quality seems to be decreasing, but management does not care." The watch is how management demonstrably cares, on a chart.
A quality threshold that triggers blame gets gamed like any other metric — engineers stop labeling AI-assisted PRs, provenance data dies, and you lose the ability to see the problem you were watching for. Thresholds trigger standards reviews, never individual consequences; say so when you publish them.
Choose tools by structured trial with a pre-committed rubric, measure with paired speed-and-quality metrics rolled up at team level, ban the vanity four as impact claims, and bring the CFO ranges with falsification criteria instead of point estimates. In a market where ~95% of orgs can't show ROI, the leader who measures honestly is the one whose claims — and budget asks — get believed.