I’m not going to argue the study doesn’t show what it shows, but based on personal experience, I have a hard time believing the inferred claim that AI slows down programmers (vs. the more narrow claim that the study proves, which is that AI slows down programmers in situations that match the study).
I have a hard time believing this because I have seen the increased productivity on my own team.
Here’s my best theory for what’s going on:
AI makes programming slower when the programmer otherwise knows what they’re doing
AI makes more mistakes and regresses to the mean, requiring human fixes on top of waiting for the AI to run
AI helps most when a programmer doesn’t know exactly what to do, so it saves them time researching (reading and understanding code other people wrote, looking through docs for how to do things, reading through bug reports and questions to find answer to issues they encounter, etc.)
And then obviously if someone’s not a programmer and vibe coding AI helps them move infinitely faster than they would have otherwise because they couldn’t code at all
This would explain the results the METR team got, but also explain why it seems so obvious to everyone that we should be paying a lot of money for AI tools to help programmers write code.
(I’ll admit, there’s another reason for programmers to want to use AI even if it did make them worse at their jobs: it outsources some of the most unpleasant programming labor, so even if it’s slower, it’s worth it in the eyes of a programmer because their experience of programming feels better when they use AI because they didn’t spend a lot of time doing tasks they didn’t enjoy doing, like typing out the code changes they already figured out in their head.)
I think all the points you raised are an important part of the story—we additionally go through some other factors that we think might explain the surprising result.
Yes? I’m not objecting directly to the results of the study, which are contained to what the study can show, but to the inference that many people seem to be drawing from the study.
I think “many people” is doing a lot of work here—I’ve generally found the public reception to be very nuanced, moreso than I was expecting. See e.g. Gary Marcus’ post
(I’ll admit, there’s another reason for programmers to want to use AI even if it did make them worse at their jobs: it outsources some of the most unpleasant programming labor, so even if it’s slower, it’s worth it in the eyes of a programmer because their experience of programming feels better when they use AI because they didn’t spend a lot of time doing tasks they didn’t enjoy doing, like typing out the code changes they already figured out in their head.)
Basically that’s proposing to take the programmer job description and move it from hands-on write the code yourself to hands-off review and adjust the code the AI agents are writing.
Many people currently employed as programmers actually do enjoy the hands-on part and I suspect that even those doing it mostly for the money tend to like it more than code reviews. Code reviews are probably just below writing documentation and writing tests on the list of things most programmers don’t like doing.
Now, personally I don’t mind writing tests and don’t mind doing code reviews myself, probably more so than most people I’ve worked with, and yet if that’s what the job morphs into I’ll probably change careers, circumstances permitting.
What I could see is that, as the job description changes, the kind of people who get into the job also changes with it. And there’s certainly people who do think like you describe. Just not many of them in my bubble.
That’s fair. I know there are programmers who actually like writing code for it’s own sake rather than as a way to achieve a goal. I think you are right that the profession will change to be less about writing code and more about achieving goals (and it already is, so I just mean it will more be like this), since AI will be cheap enough to make humans writing code too expensive.
I’m not going to argue the study doesn’t show what it shows, but based on personal experience, I have a hard time believing the inferred claim that AI slows down programmers (vs. the more narrow claim that the study proves, which is that AI slows down programmers in situations that match the study).
I have a hard time believing this because I have seen the increased productivity on my own team.
Here’s my best theory for what’s going on:
AI makes programming slower when the programmer otherwise knows what they’re doing
AI makes more mistakes and regresses to the mean, requiring human fixes on top of waiting for the AI to run
AI helps most when a programmer doesn’t know exactly what to do, so it saves them time researching (reading and understanding code other people wrote, looking through docs for how to do things, reading through bug reports and questions to find answer to issues they encounter, etc.)
And then obviously if someone’s not a programmer and vibe coding AI helps them move infinitely faster than they would have otherwise because they couldn’t code at all
This would explain the results the METR team got, but also explain why it seems so obvious to everyone that we should be paying a lot of money for AI tools to help programmers write code.
(I’ll admit, there’s another reason for programmers to want to use AI even if it did make them worse at their jobs: it outsources some of the most unpleasant programming labor, so even if it’s slower, it’s worth it in the eyes of a programmer because their experience of programming feels better when they use AI because they didn’t spend a lot of time doing tasks they didn’t enjoy doing, like typing out the code changes they already figured out in their head.)
We definitely do not claim that AI broadly slows down programmers! See tweet thread about this.
I think all the points you raised are an important part of the story—we additionally go through some other factors that we think might explain the surprising result.
Yes? I’m not objecting directly to the results of the study, which are contained to what the study can show, but to the inference that many people seem to be drawing from the study.
I think “many people” is doing a lot of work here—I’ve generally found the public reception to be very nuanced, moreso than I was expecting. See e.g. Gary Marcus’ post
Basically that’s proposing to take the programmer job description and move it from hands-on write the code yourself to hands-off review and adjust the code the AI agents are writing.
Many people currently employed as programmers actually do enjoy the hands-on part and I suspect that even those doing it mostly for the money tend to like it more than code reviews. Code reviews are probably just below writing documentation and writing tests on the list of things most programmers don’t like doing.
Now, personally I don’t mind writing tests and don’t mind doing code reviews myself, probably more so than most people I’ve worked with, and yet if that’s what the job morphs into I’ll probably change careers, circumstances permitting.
What I could see is that, as the job description changes, the kind of people who get into the job also changes with it. And there’s certainly people who do think like you describe. Just not many of them in my bubble.
That’s fair. I know there are programmers who actually like writing code for it’s own sake rather than as a way to achieve a goal. I think you are right that the profession will change to be less about writing code and more about achieving goals (and it already is, so I just mean it will more be like this), since AI will be cheap enough to make humans writing code too expensive.