Design a four-week AI dev-tool bake-off — task battery, four-dimension rubric, cohort split, pre-committed decision rule — then score two mock trial datasets. The tool developers loudly prefer loses on measured quality and review burden. Your decision memo must follow the evidence, and explain the gap without dismissing the humans.
Ingrid Halvorsen is Director of Engineering at Meridian Health Tech, a fictional 150-engineer healthcare software company. Procurement wants a single AI coding tool standardized by Q4; two vendors made the shortlist; and her engineers already have opinions — loud ones, formed from demos and social media. Ingrid has seen this movie: the org buys the tool people like, discovers six months later that review burden doubled, and nobody can say whether quality moved because nobody measured baseline. This lab is the antidote: design the trial before running it, pre-commit the decision rule, then score the data you get — not the data you hoped for.
Four weeks, two cohorts of 12 engineers each (matched by level and codebase), tools swapped at the midpoint to control for team effects. Data as delivered to you:
DEVELOPER PREFERENCE SURVEY (n=24, end of trial)
Tool A Tool B
"I want to keep this tool" 19 5
"Felt faster with it" 20 7
"Suggestions felt smart" 21 8
"Would recommend to a colleague" 18 6
Free-text highlights:
A: "flow state", "reads my mind", "chat is brilliant"
B: "solid but boring", "autocomplete is conservative",
"made me look at more of its diffs"
DELIVERY & QUALITY METRICS (per-cohort-week, tool-attributed)
Tool A Tool B
PRs merged / engineer-week 4.6 4.1
Median cycle time (hrs, open->merge) 22 19
Median reviewer time per PR (min) 31 19
Review rounds per PR (median) 2.4 1.6
Change-failure rate on tool-weeks 7.1% 4.4%
Reverts attributed to tool-assisted
changes (4 weeks total) 9 3
Post-merge defects found in trial
code (30-day window) 14 6
Test coverage delta on changed lines -2.1% +1.4%
BASELINE (4 wks pre-trial, no tool):
PRs/eng-wk 3.8 · cycle 24h · reviewer 22min ·
CFR 5.2% · coverage delta +0.2%
Design the protocol before touching the results — even though you already saw them. Write the trial design as if the trial had not run: candidate tools, four-week window, two matched cohorts with midpoint crossover, and a task battery that represents real work (feature slices, bug fixes, refactors, test-writing — not toy katas). Ten lines maximum. The discipline of writing the protocol first is the lab; a bake-off designed after the data arrives is a justification, not a trial.
Define the four-dimension rubric with weights. Quality (change-failure rate, post-merge defects, coverage delta), Speed (cycle time, throughput vs baseline), Security/compliance posture (data handling, no-training guarantees, audit logging — score from vendor docs and your security review), Developer experience (survey). Assign weights and write one sentence justifying each weight — in a healthcare org, quality and security outweighing DX is defensible; equal weights are also defensible; DX dominating is not, and you must be able to say why.
Pre-commit the decision rule. Before scoring, write the rule: e.g., "highest weighted score wins; a tool with change-failure rate above baseline cannot win regardless of total; a DX gap larger than 2:1 triggers a written adoption-risk mitigation, not a veto." The rule must make it possible for the popular tool to lose — otherwise the trial is theater.
Score the quality dimension. Tool B wins every quality metric: CFR 4.4% vs 7.1% (baseline 5.2% — note Tool A is worse than no tool), post-merge defects 6 vs 14, reverts 3 vs 9, coverage +1.4% vs −2.1%. Log each with the baseline comparison. Ask your AI to double-check your reading: "Given these metrics and this baseline, which tool degraded quality relative to no tool at all?"
Score speed honestly — this is where scoring goes wrong. Tool A merges more PRs (4.6 vs 4.1) but Tool B has shorter cycle time (19h vs 22h) and 39% less reviewer time per PR (19 vs 31 min). Compute the number most bake-offs skip: total reviewer minutes per engineer-week (PRs × reviewer time × rounds factor). Tool A's throughput edge is paid for by the reviewers — count that cost in the speed score or in a separate team-throughput view, but count it.
Score DX and take it seriously. Tool A wins DX decisively — 19/24 want to keep it. Do not sneer at this: preference predicts adoption, and a hated tool with great metrics can still fail via non-use. Score it as designed, then carry the tension forward to the memo instead of resolving it by fudging a number.
Apply the decision rule and write the memo. One page: recommendation, the weighted scores, the three numbers that decided it (CFR vs baseline, reviewer burden, post-merge defects), an honest account of the preference gap and why the felt-faster signal diverged from measured speed, and the adoption-risk mitigation your rule triggered (e.g., Tool B rollout paired with config tuning for less-conservative autocomplete, a re-survey at week 6, and a commitment to re-trial in two quarters).
Pressure-test the memo. Prompt: "You are the 19 engineers who preferred Tool A. Attack this memo." Answer the two strongest objections in the memo itself. Then check the memo one last time for the failure this lab is built around: any sentence where preference data is used as evidence of quality or speed.
"Felt faster" is a real datum about experience and a notoriously poor measurement of speed — fluent suggestion streams compress perceived effort while the verification cost lands downstream on reviewers, where the survey never looks.
A complete lab has all of the following:
A bake-off is only as honest as its pre-committed decision rule. Measure quality against baseline, count the reviewer's minutes as real minutes, treat preference as adoption data — and let the boring tool win when the evidence says it won.