If you take away just one thing from this study, it should probably be this: when people report that AI has accelerated their work, they might be wrong!
This is the summary of a recent study on the effect of AI-powered coding assistants on developer productivity.
The results are surprising everyone: a 19 percent decrease in productivity. Even the study participants themselves were surprised: they estimated that AI had increased their productivity by 20 percent.
In essence: Not good. The one thing we collectively point to, when asked about the productivity impact of AI, coding tools, seems to fail us (for now, and in the specific context of this study – to be clear and fair).
The study was carried out in pretty much the most rigorous fashion possible: an honest-to-goodness randomized controlled trial under real-world conditions. The subjects were experienced developers carrying out their everyday work.
The reasons for the productivity loss make a ton of sense:
The biggest issue is that the code generated by AI tools was generally not up to the high standards of these open-source projects. Developers spent substantial amounts of time reviewing the AI's output, which often led to multiple rounds of prompting the AI, waiting for it to generate code, reviewing the code, discarding it as fatally flawed, and prompting the AI again.
But where there is darkness, there is light:
Typically, large productivity boosts occur for **small, well-defined, greenfield projects**, or when an engineer is first learning a new language or API. […] Less experienced developers showed higher adoption rates and greater productivity gains.”
