Write the board memo from a deliberately mixed metrics package — adoption at 90%, individual throughput up, cycle time flat, churn rising, a CFO note citing zero measured ROI, and one genuinely good signal buried in the noise. Every claim traced to the data. What's real, what's noise, and what you'd bet on next.
Priya Raghavan is VP of Engineering at Kestrel Marine Systems, a fictional 130-engineer maritime-software company, eleven months into its AI program — and it is board season. The package on her desk refuses to tell one story: the adoption slide is triumphant, the delivery slide is flat, the quality slide is quietly deteriorating, and the CFO has pre-circulated a note that the program has "no measured ROI." Three narratives are available: the cheerleader's (quote adoption, bury churn), the cynic's (quote the CFO, kill the program), and the honest one — which is harder to write, because it requires calling individual claims real or noise with the data as the only judge. Boards fund the third narrative longer than either of the others. This capstone is that memo.
KESTREL AI PROGRAM | month 11 data sheet (sources noted)
A1 Adoption: 90% weekly active tool use (117/130 eng; telemetry)
A2 Self-reported time saved: median 6 hrs/wk (survey, n=96)
A3 Individual throughput: +24% PRs/eng-wk vs baseline (repo data)
A4 Suggestion acceptance: 31% (vendor dashboard)
D1 Cycle time (open->merge): 27h vs 26h baseline (flat)
D2 Lead time (commit->prod): unchanged; deploy freq unchanged
D3 Roadmap delivery: 3 of 4 quarterly epics shipped (vs 3 of 4
last year)
Q1 Code churn (21-day rewrite rate): 14.8%, up from 9.1% at
baseline; rising ~0.6pt/month for 6 months (repo data)
Q2 Change-failure rate: 6.1% vs 5.3% baseline (drifting up)
Q3 Review load: median 29 min/PR vs 21 baseline; review queue
depth up 40% (repo data)
B1 Onboarding: time-to-10th-merged-PR for the 14 hires made
this year: 3.9 wks vs 6.5 wks last-year cohort (HR+repo,
consistent across all 14; hiring bar and source mix unchanged)
B2 Team Meridian (12 eng, adopted with Lab-05-style review
standards + verification training): churn 8.2% (DOWN from
9.0%), CFR 4.1%, throughput +19% - the only team with review
standards fully adopted (repo data, 6 months)
C1 CFO note: "$610K annual program cost. Delivery metrics flat.
I see no measured ROI. Recommend sunset unless the board
hears a concrete case."
C2 Industry base rates (analyst pack): typical enterprise AI
rollouts show individual gains 12-18 months before org-level
delivery movement; orgs skipping verification standards
show churn +3-6pts in year one.
Build the claims-trace table before writing a word of narrative. For every metric A1–C2: what it measures, what it cannot support, and your verdict — REAL SIGNAL / NOISE / COST / UNKNOWN — with one line of reasoning. This table is the memo's spine and a graded artifact. Rules of evidence from Lab 08 apply: telemetry measures engagement, surveys measure perception, paired metrics beat solo metrics, and any number without a quality pair is unverified.
Sort the flattering numbers honestly. A1 (90% adoption) is real as adoption — it says the tools are used, nothing about value. A2 (6 hrs/wk self-reported) is perception data contradicted by D1–D3; mark it noise for productivity claims (Lab 07's felt-faster trap at org scale). A3 (+24% throughput) paired with Q1/Q3 is not a win: more PRs, rewritten more often, reviewed more slowly — classify it honestly as unconverted potential, currently showing up as reviewer cost. A4 (31% acceptance) is a vendor engagement stat; noise (Lab 08).
Face the quality drift — this is the risk section. Q1 is the sheet's most important number: churn 9.1%→14.8%, rising steadily for six months, with CFR drifting the same direction and review load up 40%. Name it as the program's principal risk in plain board language: the org is producing more code than it is verifying, and the trend line — not the level — is the alarm. Then connect it to C2's base rate: churn +3–6pts in year one is the documented signature of rollouts that skip verification standards, which is precisely Kestrel's state (one team of twelve excepted). This is the diagnosis the whole memo hangs on.
Find and elevate the buried signals. Two candidates hide in the package. B1: onboarding time nearly halved across all 14 hires with hiring conditions unchanged — a real, attributable, durable benefit (weak confounds, consistent cohort, plausible mechanism). B2 is the strategically decisive one: the single team that adopted review standards and verification training shows churn below baseline with +19% throughput — a natural experiment inside your own org demonstrating that the churn problem is a practice problem, not a tool problem. A memo that misses B2 has missed the sheet's entire argument.
Answer the CFO with the CFO's own logic. C1 deserves a direct response, not a dodge: concede that org-level delivery is flat and that A2/A3 do not constitute ROI (Lab 09 discipline — no hours-to-dollars without a realization bridge). Then present what is measured: B1 in dollars (14 hires × 2.6 weeks × loaded cost — roughly $180–200K/yr at Lab 09-style rates, shown as a range), B2 as the evidence that the remaining value is unlockable by process, not by more spend, and C2 as the base-rate frame: month 11 of a 12–18-month curve, tracking the documented pattern — used to set expectations, never as an excuse (state what must be true by month 18 or the base-rate defense expires). Net position: the program is not yet ROI-positive, is not the sunset case either, and the deciding evidence arrives from a specific, cheap intervention.
Write the next-quarter bets with falsifiers. Three bets, each with owner, cost, and kill criterion: (1) scale Meridian's review standards + verification training org-wide (the B2 replication bet — falsifier: if churn hasn't inflected downward within two quarters on adopting teams, the practice hypothesis is wrong); (2) protect and instrument the onboarding gain (falsifier: next cohort regresses to >5.5 wks); (3) the measurement completion bet — paired panels per Lab 08 so the month-18 conversation has evidence instead of vibes. Total the cost against the $610K program line so the board sees proportion.
Assemble the memo — one page plus the backing sheet. Structure: verdict sentence first (honest, neither triumphant nor terminal); three REAL claims with traced sources; the churn risk stated unhedged with the base-rate context; the CFO response; the bets. Every quantitative sentence carries its data-sheet ID (A1, Q1, B2…) — the discipline that makes the memo auditable, which is the point of the whole track. The backing metrics-interpretation page is your claims-trace table, cleaned up.
Adversarial pass, twice. First prompt: "You are a board member who wants this program killed. Attack the memo." Second: "You are the vendor's champion on the board. Attack it from the other side." The memo passes when both attackers are forced to argue with the data sheet rather than the framing. Patch any sentence either attacker could quote out of context.
A board can forgive flat metrics at month 11; it does not forgive discovering in month 18 that the churn line was rising in month 11 and the memo called it "healthy iteration."
A complete lab has all of the following:
The honest board narrative is a sorting exercise with signatures on it — every claim traced to data, every flattering number tested against its quality pair, the risk named while it is still cheap, and next quarter's money placed on the one signal that survived. Boards don't fund optimism or pessimism; they fund leaders whose numbers can be audited.