Late at night, a developer who has spent years inside the same codebase turns on the AI, and the code begins to write itself. Whole functions appear before the thought is even finished. It feels like flying. That feeling is now the most trusted claim in all of software: that AI makes you faster. And in 2025, an independent research group decided to stop asking and start measuring. What they found points the other way, and the gap between what the developers felt and what the clock said is one of the most important things anyone has learned about working with AI.
We spend our days building with these tools, so this is not an anti-AI argument. It is a warning about a specific trap, and the trap is not that AI is useless. It is that AI can change how the work feels without changing how well it goes, and once that happens, your own sense of being productive stops being evidence of anything. Here is exactly what the study found, the caveats that keep it honest, and the one conclusion that survives all of them.
The video tells this as a story, from the feeling of flying to the number that contradicts it. This written version stands on its own, lays out the exact figures, gives the caveats their full weight, and ends with a simple way to find out the truth for your own work.
What the study actually did
In early 2025, an independent research group called METR decided to measure the thing everyone assumes. They recruited sixteen seasoned open-source developers, each working inside repositories they knew intimately, projects with tens of thousands of stars and over a million lines of code. The work was real: two hundred and forty six genuine tasks, the actual bugs and features already waiting in their own projects. Each task was randomly assigned. On some they could use AI, on others they could not. The tools were the best of the moment, Cursor Pro with Claude 3.5 and 3.7.
Before starting, everyone made a prediction. The developers expected AI to make them twenty four percent faster. Economists guessed thirty nine. Machine learning experts guessed thirty eight. Everyone disagreed about the size of the gain. Nobody doubted the direction.
The result nobody predicted
Then they ran the experiment, and the result pointed the other way. With AI, the developers were nineteen percent slower. Not faster. Slower. On their own code, with frontier tools, the finish line quietly moved further away.
But the slowdown is not the part that should stop you. When it was over, the researchers asked the developers how it had gone, and the developers said AI had made them about twenty percent faster. They were slower, and they could not feel it. They believed they were twenty percent faster while the clock said nineteen percent slower. The distance between those two numbers is the entire story.
Why it happened
So what actually filled those hours? METR studied the recordings. With AI, the developers spent less of their time writing code and more of it reviewing the AI's output, prompting it again, and waiting for it to answer. They accepted fewer than half of the AI's suggestions, and most of what they kept, they still had to repair. On a codebase you already carry in your head, the AI rarely hands you the answer. It hands you one more thing to check. One developer described it exactly: it was a drag on velocity while subjectively feeling like a speedup. The work felt easier, and easier is not the same as faster. It is only what we remember afterward.
The honest caveats
An honest read has to slow down here, because this is one study of sixteen people working in large, mature codebases, which is the single hardest place for AI to help. METR says it plainly: they do not claim their developers or repositories represent most of the software work being done. And there is real evidence the other way. In a GitHub study, developers finished a task fifty five percent faster using Copilot, but that task was a small, brand-new program, and the study was run by the company that sells the tool. Adoption keeps climbing, with eighty four percent of developers now using these tools, even as trust in what they produce keeps falling.
| METR study (2025) | GitHub Copilot study | |
|---|---|---|
| Result | 19% slower with AI | 55% faster with AI |
| The work | 246 real tasks in huge, mature codebases | One small, brand-new program |
| Who ran it | Independent research group | The company that sells the tool |
| What it suggests | AI struggles where context is deep | AI shines on fresh, self-contained work |
The fight over how much AI helps is genuine, and where you land probably depends on whether your work looks more like the first row or the second. New, self-contained projects are where AI feels closest to magic. Deep, established systems are where it can quietly cost you time.
The finding that survives everything
Strip away every caveat, and one finding survives. The people using the tool could not tell whether it was helping them. That is the discovery that outlives the sample size. Not that AI is slow, but that it can change how the work feels without changing how well it goes. And once a tool can make you feel productive, your own sense of productivity stops being evidence. The best developers in that study were not fooled because they were careless. They were fooled because it felt good.
This is bigger than coding. It applies to writing, to research, to any knowledge work where an AI produces a fluent draft in seconds. The draft arriving fast feels like progress, but the real question is whether the finished thing is better and whether you got there sooner, and those are things you have to measure, not feel.
How to find out the truth for your own work
The good news is that the study is easy to run on yourself, and it is the only way to actually know. Do not trust the feeling in either direction. For the next couple of weeks, measure it:
Pick a repeating type of task you do (a bug fix, a report, an article).
For each one, before you start, jot down:
- the task, in one line
- whether you'll use AI on it or not (alternate them)
- the time you started
When it's genuinely DONE (reviewed, working, shipped), jot down:
- the time you finished
- your gut feeling: did AI make this faster? (yes / no / same)
After ~10 tasks, compare two things:
1. Actual average time: AI tasks vs non-AI tasks.
2. How often your GUT matched the CLOCK.
The second number is the one that matters. If your feeling and your
stopwatch disagree, trust the stopwatch, and adjust where you reach for AI.
You will likely find, as the study did, that AI helps a lot on some kinds of work and quietly costs you on others. That is not a reason to stop using it. It is a reason to stop guessing. The people who get the most out of these tools are not the ones who feel the fastest. They are the ones who know, on which tasks, they actually are.
If you want to get genuinely better at directing AI so it helps more than it hurts, the free Claude prompting course is built around exactly that judgment. And for the bigger picture of where real, durable value is moving as these tools spread, we made the case in The Moat Moved. The next time a tool makes you feel unstoppable, remember: that feeling is not proof that it works. It is exactly when you would never know.