On July 10, 2025, the AI research organization METR published a randomized controlled trial on how AI coding tools affect experienced developers working in their own mature open-source projects. The headline result surprised much of the industry: when developers were allowed to use AI tools, they took about 19 percent longer to complete tasks, even though they believed AI had sped them up by about 20 percent. For engineering leaders rolling out AI assistants, the study is less a verdict on AI than a warning about measuring productivity by perception.

Key Facts#

  • Publication: July 10, 2025, as a METR blog post, a paper on arXiv and a public repository with anonymized data and regression code.
  • Participants: 16 experienced developers working on large open-source repositories they had contributed to for years.
  • Tasks: 246 real issues that the developers chose as valuable to their projects, each randomly assigned to an "AI allowed" or "AI disallowed" condition.
  • Tools: in the AI condition, developers could use any tools they liked; METR reports they mainly used Cursor Pro with Claude Sonnet models available in early 2025.
  • Result: completion time was about 19 percent longer with AI allowed. METR's published regression estimates 18.8 percent, with a confidence interval of roughly 1 to 40 percent.
  • Perception gap: developers forecast a 24 percent speedup before the study and still believed they had been sped up by 20 percent afterward.
  • Scope: METR explicitly says the result does not show that AI slows down most developers or that future tools will not help in this setting.

What Happened#

METR designed the study to complement benchmarks, which measure models on self-contained tasks, with evidence from real work. Each developer proposed issues from their own repository, estimated how long each would take with and without AI, and then had each issue randomly assigned to one condition. Tasks averaged about two hours. Developers recorded their screens and reported their implementation time, and METR tracked pull requests through review.

The randomization is what makes the result notable. Because the same developers worked on comparable issues under both conditions, the comparison controls for skill and familiarity. METR's public repository reproduces the core estimate: the ratio of expected time with AI to expected time without AI, minus one, comes out at 0.188. The confidence interval stays above zero across the different error estimates METR reports, so the data point to a slowdown rather than a speedup in this setting, although its size is uncertain.

The perception gap may be the more important finding. Developers expected large gains, experienced a slowdown and still reported that AI had helped. Surveys and self-reports, the most common way companies assess AI tools, would have produced the opposite conclusion from the measured data.

Background#

By early 2025, AI coding assistants were widely used, and many productivity claims rested on benchmarks, controlled tasks or self-reported surveys. METR wanted to know whether benchmark progress translated into real-world speedups for skilled engineers working on their own code.

The study population was deliberately demanding. These were experienced maintainers working in large, high-quality codebases with strict standards, where much of the needed knowledge is implicit and review is rigorous. METR cautioned against generalizing to less experienced developers, unfamiliar codebases or greenfield prototypes, and it noted that tools and usage patterns were changing quickly.

Why It Matters for Developers#

Measure outcomes, not feelings. The study shows that experienced engineers can be confidently wrong about whether a tool helps them. If your organization is evaluating GitHub Copilot, Claude Code, Cursor or any other assistant, track objective signals such as cycle time, review rounds, defect rates and rework over weeks, and compare similar work with and without the tool. Our guide to AI-assisted .NET development covers where these tools tend to help most.

Expect gains to depend on context. The study's setting, expert developers in familiar, complex repositories, is exactly where an assistant has the least to add and the most to break. The same tool may help far more on unfamiliar code, boilerplate, tests or migrations. Treat "Does AI make us faster?" as a question per task type, not a single number.

Budget for verification. Reviewing, correcting and integrating AI output takes time, especially where quality bars are high. Strong tests, analyzers and CI gates, as described in our code quality guide, make that verification cheaper and safer.

For leads and architects, this is also a management topic. Setting realistic expectations, choosing metrics and resisting survey-driven conclusions are skills that come up in tech lead interviews and in conversations about engineering culture.

What's Next#

METR tried to measure the change over time. On February 24, 2026, it reported that a follow-up study, started in August 2025 with newer tools, 10 of the original developers and 47 new ones, produced raw estimates that pointed toward speedups: about 18 percent faster for the returning developers and about 4 percent faster for the new ones, with wide confidence intervals. But METR concluded that the data had become an unreliable signal. Many developers declined to take part, or chose not to submit certain tasks, because they did not want to work without AI. Pay had also been cut from $150 to $50 an hour, and developers who ran several agents concurrently found it hard to report time per task.

METR said it believes developers were probably sped up more in early 2026 than in early 2025, but that its data provide only weak evidence of how much, and it is redesigning its methods. The episode shows how quickly AI adoption changes the ground under research: once developers depend on a tool, randomizing them away from it becomes hard. The durable lesson from the original study remains the same. Productivity claims need measurement, and the people using a tool are not the most reliable judges of its effect.

Sources#