Run an AI Dev-Tool Bake-Off That Resists Both Hype and Incumbency

Workflow area: Measurement & ROI · Tier: Core · Hands-on time: ~40 minutes to design; 4–6 weeks to run

The two failure modes of tool selection are mirror images. Hype mode: the shiniest demo wins, and three months later the licenses sit idle because the tool demos better than it works in your codebase. Incumbency mode: whatever arrived first stays forever, because nobody wants to re-justify a decision that is "fine." A bake-off is how you buy the right tool — and just as valuably, how you generate the evidence that makes the decision stick when the next shiny demo or the next budget review arrives.

When to use — and when not to

• Use it for any org-wide tool commitmentassistants, agent platforms, review bots — where the real cost is not licenses but the switching cost of workflows, prompts, and habits that accrete around the winner.
• Use it when pb-32's shadow inventory surfaced strong candidatesthe fast-track list is your candidate slate, pre-validated by engineers voting with their own money.
• Use it to re-test the incumbentevery 12–18 months, the current tool re-earns its seat by entering the ring as one candidate among several, not as the default.
• Skip it for cheap reversible experimentsa single team trying a $50/month tool for a sprint needs a calendar entry, not a bake-off; save the machinery for decisions with switching costs.
• Skip it when the decision is already madea bake-off staged to ratify a foregone conclusion burns exactly the credibility you will need for the next real one; if the CTO has decided, say so and skip the theater.

The recipe

1. 1.

   Build the task battery from your backlog, not from benchmarks. Pick 8–12 real tasks that engineers actually did recently or will do soon, spanning your true distribution: a feature in your largest legacy module, a bug with a misleading stack trace, a test-coverage task, a refactor touching a gnarly interface, a small greenfield service. Public benchmarks measure the model; the battery measures the tool in your codebase, with your build system, on your worst code. Include at least two tasks the current stack handles badly — that is where differentiation lives.

2. 2.

   Fix the rubric before anyone touches a tool. Four dimensions, weighted in advance so the winner cannot be chosen by adjusting weights afterward:

   Copy

   QUALITY  (35%) - correctness on battery tasks; review findings
                    per PR; test adequacy (assertions, not lines)
   SPEED    (25%) - time-to-merged on battery tasks vs baseline;
                    iteration latency as reported by users
   SECURITY (20%) - findings from pb-28 gates on generated code;
                    vendor posture: terms/tier per pb-34 comparison;
                    admin controls, SSO, audit logs
   DX       (20%) - weekly 4-question survey: would you keep it?
                    where did it fight you? plus friction notes
   

   Security scores partly from documents (pb-34's vendor table), not usage — a tool can lose on terms alone, and it is cheaper to learn that in week one.

3. 3.

   Design cohorts to cancel enthusiasm. Enthusiasts make every tool look good for two weeks. Recruit 3–4 engineers per candidate, mixing a skeptic and a mid-level engineer into each cohort — not just volunteers. Run a crossover where feasible: cohorts swap tools at the halfway mark, so each tool is scored by people who also used its rival. Everyone logs the same lightweight evidence — battery-task timings, PR links, and the weekly survey. Keep the trial period honest: 4–6 weeks, because week one measures novelty and only week four measures the tool.

4. 4.

   Score in the open, decide in a memo. Compile scores per rubric line with the evidence linked. Then write the decision memo — one page, four sections: the recommendation; the scores with the two or three decisive gaps; what we are explicitly giving up (the losing tool's genuine strengths, recorded so the next evaluation starts honest); and what would change this decision — the falsifiable conditions ("if vendor X ships enterprise SSO", "if quality gap closes in two releases") with a named re-review date. That last section is what converts a purchase into a policy that can be revisited without a fight.

5. 5.

   Roll out with the evidence attached. The bake-off's artifacts — battery results, survey verbatims, the memo — become the enablement material: "here is where it beat the incumbent in our codebase" lands better than any vendor deck, and the recorded weak spots become the training's honest warnings. Feed the winner's baseline numbers straight into pb-37's measurement system; you will want this before/after later.

Make it yours

• Small team or startup: two candidates, one week each with the same three engineers, a six-task battery, and a half-page memo — the structure matters more than the scale; even a tiny bake-off beats a demo-driven decision.
• Large org or GCC: run cohorts across two or three genuinely different codebases (the legacy monolith and the new services will disagree about the winner — that disagreement is a finding, and sometimes the answer is two tools with clear guidance on which where); procurement and security review run in parallel with the trial, not after it, so the winner is signable the week the memo lands.
• Services context: the battery must include client-posture constraints from pb-33 — a tool that wins on capability but is unapprovable on your Amber and Red engagements loses on the only dimension that matters; score deployability per client segment explicitly.

Quality check

• The battery would embarrass a demoif every task is one a vendor demo handles smoothly, the battery is testing the pitch; keep the misleading bug and the legacy module in.
• Weights were committed before week onea rubric reweighted after the scores exist is a narrative, not a measurement; the pre-registered weights are the audit trail.
• Skeptics' scores get read separatelyif enthusiasts and skeptics rank the tools the same way, the result is robust; if they diverge, the divergence is the most informative data you collected.
• The memo names what the winner is bad ata decision memo with no concessions is marketing; the recorded weaknesses are next year's re-evaluation criteria.
• The falsification section has a date on it"what would change this decision" without a scheduled re-review is a rhetorical flourish; calendar it.