The most useful number in the AI-tooling debate is not a benchmark score. It's the gap between two percentages from a single study. In 2025, METR ran a randomised controlled trial with experienced open-source developers doing real tasks on large, mature codebases they knew well. With AI tools enabled, the developers took 19% longer to complete their tasks. Afterwards, the same developers estimated that AI had made them about 20% faster.
Sit with that. These weren't juniors dazzled by autocomplete — they were expert contributors, on familiar million-line repositories, who had just lived through the slowdown, and their perception still pointed the wrong direction by nearly forty percentage points.
Why self-report fails here
I don't think the developers were being careless. AI tools are genuinely pleasant in ways that masquerade as speed. Less typing feels faster even when total task time grows. The tool produces visible motion — drafts, diffs, suggestions — and motion reads as progress. Waiting for a generation feels like rest; reviewing its output doesn't get mentally booked as "the task taking longer." And everyone has absorbed the narrative that AI makes you faster, which is exactly the condition under which self-assessment bends toward expectation.
Meanwhile, almost every corporate AI productivity claim you will hear this year is built on precisely this kind of evidence: surveys, self-estimates, or proxy counts like suggestions accepted and lines generated. The one time someone measured carefully, perception and reality disagreed about the sign. That should permanently change how much weight you give the surveys.
Don't overcorrect either
The wrong lesson from METR is "AI tools don't work." The study is specific: experienced developers, repositories they knew deeply, early-2025 tools. Those conditions minimise AI's edge — the human already has the context that the model lacks — and tools have improved since. In other settings (unfamiliar codebases, boilerplate-heavy work, prototyping), both controlled studies and field data show real gains, and DORA's 2025 research links AI adoption to higher delivery throughput. The honest reading is narrower and more useful: impact varies enormously by context, and perception cannot tell you which context you're in. Only measurement can.
How to run a pilot that produces findings
Having run and reviewed a number of these evaluations, here is the shape I now insist on.
1. Define outcomes before the pilot starts. Decide what "it works" means in advance, or the pilot will conclude whatever the loudest enthusiast feels. I anchor on a small set: lead time for changes, review turnaround, defect escape rate, and rework (how often merged code is re-modified within a few weeks). Note that none of these is "usage."
2. Establish the baseline first. Measure the same team, same metrics, for several weeks before the tool arrives. Without a baseline you'll be comparing against memory, and memory is exactly the instrument that just failed us.
3. Compare like with like. Pilot volunteers are your most motivated people working on whatever they chose to bring. If you can't randomise, at least compare matched teams and task types, and be suspicious of gains that vanish outside the pilot group.
4. Watch the second-order effects. This is where AI tooling most often disappoints quietly. Generation is cheap, so pull requests get bigger; review load shifts downstream to your senior engineers; code churn rises as plausible-but-wrong changes get reverted. A pilot that measures only the coding stage will book these costs to someone else's budget. Throughput and stability have to be read together — DORA's finding that AI raises both throughput and instability in many organisations is the pattern to check for.
5. Keep the qualitative data — calibrated. Developer experience matters; tools people hate get abandoned, and satisfaction has real retention value. Collect it, report it, but label it as sentiment. "The team likes it" and "the team is faster" are different claims requiring different evidence.
The vanity metrics to refuse
Some numbers exist mainly to make dashboards green. Acceptance rate rewards suggestions that were easy to accept, not valuable. Lines of code generated measures volume, which was never the goal and is often the problem. Percentage of code written by AI is a marketing statistic. Seats active tells you about procurement, not productivity. If a vendor's business case leans on these, ask what happens to lead time and defect escape — and watch how quickly the conversation changes.
Treat it like any other engineering investment
We would never re-platform a database because the team "felt" it was faster. AI tooling deserves the same discipline — not because scepticism is fashionable, but because the spend is large, the perception gap is documented, and the real wins are too valuable to lose in noise. Measure honestly and you get something better than optimism: you learn where AI genuinely pays off in your organisation, and you can invest there with confidence instead of faith.
Feelings are data about morale. They are not findings about productivity. Fund the findings.