This is interesting, though I expect it’s an upper bound on Copilot productivity boosts:
Writing an HTTP server is a common, clearly defined task which has lots of examples online.
JavaScript is a popular language (meaning there’s lots of training data for Copilot).
I imagine Copilot is better for building a thing from ground up, whereas the programming most programmers do most days consists in extending, modifying and fixing existing stuff, meaning more thinking and reading and less typing.
In my experience it’s best at extending things, because it can predict from the context of the file you’re working in. If I try to generate code from scratch, it goes off in weird directions I don’t actually want pretty quickly, and I constantly have to course-correct with comments.
Honestly I think the whole “build from ground up”/”extending, modifying, and fixing” dichotomy here is a little confused though. What scale are we even talking?
A big part of Copilot’s efficiency gains come from very small-scale suggestions, like filling out the rest of a for loop statement. It can generally immediately guess what you want to iterate over. I happened to be on a plane without internet access recently, decided to do a bit of coding anyway, needed to write a for loop, and then was seriously off-put by the fact that I couldn’t write the whole thing by just pressing “Tab”. I had to actually think about how a stupid for loop is written! What a waste of time!
Honestly I think the whole “build from ground up”/”extending, modifying, and fixing” dichotomy here is a little confused though. What scale are we even talking?
I meant to capture something like “lines of code added/modified per labour time spent”, and to suggest that Copilot would reap more benefits the higher that number is (all else equal).
This is interesting, though I expect it’s an upper bound on Copilot productivity boosts:
Writing an HTTP server is a common, clearly defined task which has lots of examples online.
JavaScript is a popular language (meaning there’s lots of training data for Copilot).
I imagine Copilot is better for building a thing from ground up, whereas the programming most programmers do most days consists in extending, modifying and fixing existing stuff, meaning more thinking and reading and less typing.
In my experience it’s best at extending things, because it can predict from the context of the file you’re working in. If I try to generate code from scratch, it goes off in weird directions I don’t actually want pretty quickly, and I constantly have to course-correct with comments.
Honestly I think the whole “build from ground up”/”extending, modifying, and fixing” dichotomy here is a little confused though. What scale are we even talking?
A big part of Copilot’s efficiency gains come from very small-scale suggestions, like filling out the rest of a for loop statement. It can generally immediately guess what you want to iterate over. I happened to be on a plane without internet access recently, decided to do a bit of coding anyway, needed to write a for loop, and then was seriously off-put by the fact that I couldn’t write the whole thing by just pressing “Tab”. I had to actually think about how a stupid for loop is written! What a waste of time!
I meant to capture something like “lines of code added/modified per labour time spent”, and to suggest that Copilot would reap more benefits the higher that number is (all else equal).