AI coding tools make developers slower, study finds • The Register

Artificial intelligence coding tools are supposed to make software development faster, but researchers who tested these tools in a randomized, controlled trial found the opposite.

Computer scientists with Model Evaluation & Threat Research (METR), a non-profit research group, have published a study showing that AI coding tools made software developers slower, despite expectations to the contrary.

Not only did the use of AI tools hinder developers, but it led them to hallucinate, much like the AIs have a tendency to do themselves. The developers predicted a 24 percent speedup, but even after the study concluded, they believed AI had helped them complete tasks 20 percent faster when it had actually delayed their work by about that percentage.

Surprisingly, we find that allowing AI actually increases completion time by 19 percent — AI tooling slowed developers down

“After completing the study, developers estimate that allowing AI reduced completion time by 20 percent,” the study says. “Surprisingly, we find that allowing AI actually increases completion time by 19 percent — AI tooling slowed developers down.”

The study involved 16 experienced developers who work on large, open source projects. The developers provided a list of real issues (e.g. bug fixes, new features, etc.) they needed to address – 246 in total – and then forecast how long they expected those tasks would take. The issues were randomly assigned to allow or disallow AI tool usage.

The developers then proceeded to work on their issues, using their AI tool of choice (mainly Cursor Pro with Claude 3.5/3.7 Sonnet) when allowed to do so. The work occurred between February and June 2025.

The study says the slowdown can likely be attributed to five factors:

• “Over-optimism about AI usefulness” (developers had unrealistic expectations)
• “High developer familiarity with repositories” (the devs were experienced enough that AI help had nothing to offer them)
• “Large and complex repositories” (AI performs worse in large repos with 1M+ lines of code)
• “Low AI reliability” (devs accepted less than 44 percent of generated suggestions and then spent time cleaning up and reviewing)
• “Implicit repository context” (AI didn’t understand the context in which it operated).

Other considerations like AI generation latency and failure to provide models with optimal context (input) may have played some role in the results, but the researchers say they’re uncertain how such things affected the study.

Other researchers have also found that AI does not always live up to the hype. A recent study from AI coding biz Qodo found some of the benefits of AI software assistance were undercut by the need to do additional work to check AI code suggestions. An economic survey found that generative AI has had no impact on jobs or wages, based on data from Denmark. An Intel study found that AI PCs make users less productive. And call center workers at a Chinese electrical utility say that while AI assistance can accelerate some tasks, it also slows things down by creating more work.

That aspect of AI tool use – the added work – is evident in one of the graphics included in the study. “When AI is allowed, developers spend less time actively coding and searching for/reading information, and instead spend time prompting AI, waiting on and reviewing AI outputs, and idle,” the study explains.

More anecdotally, a lot of coders find that AI tools can help test new scenarios quickly in a low-stakes way and automate certain routine tasks, but don’t save time overall because you still have to validate whether the code actually works – plus, they don’t learn like an intern. In other words, AI tools may make programming incrementally more fun, but they don’t make it more efficient.

The authors – Joel Becker, Nate Rush, Beth Barnes, and David Rein – caution that their work should be reviewed in a narrow context, as a snapshot in time based on specific experimental tools and conditions.

“The slowdown we observe does not imply that current AI tools do not often improve developer’s productivity – we find evidence that the high developer familiarity with repositories and the size and maturity of the repositories both contribute to the observed slowdown, and these factors do not apply in many software development settings,” they say.

The authors go on to note that their findings don’t imply current AI systems are not useful or that future AI models won’t do better. ®