I agree with this directionally, and generally tend to believe that certain kinds of straight lines will continue to be straight. But also iirc you rely a lot on the METR trend line, which is (a) noisy, (b) over a short timescale (~2 years[1]), and (c) not a great measure of what people care about[2][3]. I don’t think you should strongly expect that particular line to continue to be straight over a long time period.
(Though I tend to think that it’s more likely to slow down by 2027 than to speed up, relative to what you’d predict by extrapolating the 2024-2025 trend, so I agree with the overall conclusion of the post. And obviously one should have non-trivial probability on each of {slow down, stay the same, speed up}, as you say.)
Compared to things like Moore’s Law, or energy efficiency of solar, or pretraining perplexity, or other similar things where we observe straight lines.
I agree with this directionally, and generally tend to believe that certain kinds of straight lines will continue to be straight. But also iirc you rely a lot on the METR trend line, which is (a) noisy, (b) over a short timescale (~2 years[1]), and (c) not a great measure of what people care about[2][3]. I don’t think you should strongly expect that particular line to continue to be straight over a long time period.
(Though I tend to think that it’s more likely to slow down by 2027 than to speed up, relative to what you’d predict by extrapolating the 2024-2025 trend, so I agree with the overall conclusion of the post. And obviously one should have non-trivial probability on each of {slow down, stay the same, speed up}, as you say.)
I’m excluding GPT-2 and GPT-3 which seem very noisy.
Compared to things like Moore’s Law, or energy efficiency of solar, or pretraining perplexity, or other similar things where we observe straight lines.
None of this is to say that the METR graph is bad! It is the best empirical evidence on timelines I’m aware of.
This market pretty much exactly tracks whether it slows down or speeds up by 2027: