Design the measurement system for an 80-engineer org from its baseline package — paired speed and quality metrics, team-level rollups, a confound log — and write the takedown of four vanity metrics, including the vendor dashboard's headline claim that a 34% acceptance rate equals a 34% productivity gain.
Chen Wei is VP of Engineering at Argosy Logistics Tech, a fictional 80-engineer org eight months into an AI-tooling rollout. The CEO wants a number: "How much faster are we?" The vendor dashboard offers one — gladly. His staff meeting offers four more, all flattering, all wrong in different ways. Chen's actual position: the org has no measurement system, only a metrics buffet, and everyone picks the dish that supports their prior. Your job is to design the honest system — one that pairs every speed claim with a quality check, aggregates at the level where AI impact actually shows up, and writes down its own confounds before someone else finds them.
ARGOSY BASELINE | trailing 6 months, org-wide (80 engineers, 9 teams)
PR throughput: 3.4 PRs/engineer-week (was 2.9 a year ago)
Median cycle time: 26h open->merge (was 25h)
Change-failure rate: 5.8% (was 4.9%)
Code churn: 11.2% of merged lines reverted/rewritten
within 21 days (was 7.8%)
Review load: median 26 min/PR, rising
Incident count: stable; MTTR stable
Onboarding: new-hire time to 10th merged PR: 6.2 weeks
Deploy frequency: team range 0.5/wk to 14/wk (platform-dependent)
VENDOR DASHBOARD (as presented to the exec team)
"34% suggestion acceptance rate = engineers accepting a third of
all AI code -> ~34% productivity uplift on assisted work"
"11,400 AI-suggested lines merged this month"
"Seat activation: 92% - near-universal adoption"
"Top team: Fulfillment - 41% acceptance rate"
CONTEXT NOTES (from Chen's staff)
- Two teams adopted AI tools 8 months ago; four ~4 months ago;
three barely use them (client restrictions on one)
- A major platform migration landed 5 months ago (touched all teams)
- Headcount: +12 juniors hired over the period
- The Fulfillment team also has the org's highest churn: 16.1%
Read the baseline like an auditor — ten minutes, no AI. Write down every pair of numbers that tells a different story together than apart. The ones that matter: throughput up 17% while churn rose from 7.8% to 11.2% and CFR from 4.9% to 5.8%; cycle time flat despite more PRs; the vendor's best team owning the org's worst churn. If you don't see these pairs now, the vendor's story will write itself later.
Define the paired-metrics rule. The core design principle, one paragraph: no speed metric is ever reported without its quality pair. Throughput pairs with churn and change-failure rate; cycle time pairs with review depth and post-merge defects; "lines merged" pairs with nothing because it is not a metric (you will handle it in the memo). Then build the actual metric set: 4–6 paired panels, each with definition, source, and the question it answers.
Choose aggregation levels deliberately. Individual-level AI metrics create surveillance and gaming; org-level averages smear signal across teams that adopted at different times. Design team-level rollups with adoption-cohort comparison: 8-month adopters vs 4-month adopters vs low-use teams, tracked on the same paired panels. Write two sentences on why individual dashboards are excluded by design — you will need them at calibration time (see Lab 03).
Build the baseline-and-confound log. A table of everything that changed during the measurement window and what it contaminates: platform migration (touches cycle time and CFR for all teams), 12 junior hires (throughput and review load — juniors lower averages regardless of AI), client restrictions (one low-use team is not a control group, it is a different population), seasonal load. Rule: every reported delta cites the confounds that could produce it without AI.
Write takedown #1 — acceptance rate. What it actually measures: the frequency with which engineers press accept on a suggestion — including trivial completions, and including bad code accepted then rewritten (which the churn number is already whispering about). What it cannot measure: whether accepted code survived, shipped, or helped. Then dismantle the dashboard's arithmetic explicitly: acceptance is measured per suggestion, uplift is claimed per engineer-hour; the equation "34% acceptance = 34% uplift" converts a UI interaction rate into a labor-productivity figure with no bridge between the units. Note the kicker in your own data: Fulfillment, 41% acceptance, 16.1% churn — accepting the most and rewriting the most.
Write takedowns #2–4. Lines of code merged (11,400 AI lines): measures volume, the one thing AI makes free; more lines is as plausibly a cost as a benefit. PR count: measures slicing behavior; AI-era PR inflation plus 4-minute reviews is throughput theater. Seat activation (92%): measures license deployment, not usage, not value; it is procurement's KPI wearing a productivity costume. For each: what it actually measures, how it inflates, what to report instead.
Design the honest headline. The CEO still deserves an answer. Draft the quarterly one-pager: paired panels by adoption cohort, the confound log's top three caveats, one narrative sentence per panel, and — where the data genuinely cannot support a causal claim — the sentence that says so. Practice the discipline of "we don't know yet, and here is what would tell us."
Stress-test with the AI. Prompt: "You are a vendor CSM defending this dashboard. Rebut this memo." Then answer the rebuttals in your memo's final section. If the vendor's best surviving argument is "but engineers like it," your measurement design has done its job; if they land a real hit, patch the design.
Any single-number productivity claim about AI should trigger the same reflex as a single-number ROI claim from a vendor deck: ask what the number's denominator is, and who chose it.
A complete lab has all of the following:
Honest AI measurement is a system of pairs — every speed number chained to its quality number, every delta chained to its confounds, every claim chained to the cohort that produced it. Vanity metrics are what you get when any link in that chain is optional.