December 2024 as 1.05x (which seems contradicted by your survey, if the replies were in November)
If you think today’s number is ~1.33x we’re ~7 months ahead of schedule vs the listed forecast, unless I’m really misreading something.
Also, re: “would be excited for others to.”, is the survey public or easy to share if someone wanted to use the same questions?
And I’d bet 1:4 for the current number is actually >1.5x, if that’s more interesting. You’ve updated me to not have that as the main expectation, but still seems pretty plausible. Obviously depends on someone rerunning the survey, and reasonable that you’ve got your hands full with other things right now.
I also realized that I believe that confusingly the survey asks about speedup vs. no post-2022 AIs, while I believe the scenario side panel is for no post-2023 AIs, which should make the side panel numbers lower, unclear exactly how much given 2023 AIs weren’t particularly useful.
Okay, switched. I’m curious about why you didn’t set the baseline to “no AI help”, especially if you expect pre-2024 AI to be mostly useless, as that seems like a cleaner comparison than asking people to remember how good old AIs were?
No AI help seems harder to compare to since it’s longer ago, it seems easiest to think of something close to today as the baseline when thinking about future speedups. Also for timelines/takeoff modeling it’s a bit nicer to set the baseline to be more recent (looks like for those we again confusingly allowed 2024 AIs in the baseline as well rather than just 2023. Perhaps I should have standardized that with the side panel).
I think this risks people underappreciating how much progress is being sped up, my naive read of the UI was the numbers were based on “no AI” and I’d bet most readers would think the same at a glance. Changing the text from “AI provides the following speedups:” to “AI provides the following speedups from a baseline of 2022⁄3 AI:” would resolve this (I would guess common) misreading.
Oh I misunderstood you sorry. I think the form should have post-2023, not sure about the website because it adds complexity and I’m skeptical that it’s common that people are importantly confused by it as is.
I think the survey is an overestimate for the reason I gave above, I think this stuff is subtle and researchers are likely to underestimate the decrease from labor speedup to progress speedup, especially in this sort of survey where it didn’t involve discussing with them verbally. Based on their responses to other questions in the survey seems like at least 2 people didn’t understand the difference between labor and overall progress/productivity.
Here is the survey: https://forms.gle/6GUbPR159ftBQcVF6. The question we’re discussing is: “[optional] What is the current productivity multiplier on algorithmic progress due to AI assistance?”
Edit: Also we didn’t spend large amounts of time on these early numbers, they’re not meant to be that precise but just rough best guesses.
Wait, actually, I want to double click on this. What was the process that caused you to transform the number you got from the survey (1.2x) to the number on the website (1.05x)? Is there a question that could be asked which would not require a correction? Or which would have a pre-registered correction?[1]
I’m not sure what the exact process was, tbh my guess is that they were estimated mostly independently but likely sanity checked with the survey to some extent in mind. It seems like they line up about right, given the 2022 vs. 2023 difference, the intuition regarding underadjusting for labor->progress, and giving weight to our own views as well rather than just the survey, given that we’ve thought more about this than survey takers (while of course they have the advantage of currently doing frontier AI research).
I’d make less of an adjustment if we asked people to give their reasoning including the adjustment from labor speedup to overall progress speedup and only included people who gave answers that demonstrated good understanding of this consideration and a not obviously unreasonable adjustment level.
Alright, my first pass guess would have been algorithmic progress seems like the kind of thing that eats a much smaller penalty than most forms org-level progress, not none but not a 75% reduction, and not likely more than a 50% reduction, but you guys have the track record.
I think it’s not worth getting into this too much more as I don’t feel strongly about the exact 1.05x, but I feel compelled to note a few quick things:
I’m not sure exactly what you mean by eating a smaller penalty but I think the labor->progress penalty is quite large
The right way to think about 1.05x vs. 1.2x is not a 75% reduction, but instead what is the exponent for which 1.05^n=1.2
Remember the 2022 vs. 2023 difference, though my guess is that the responses wouldn’t have been that sensitive to this
Also one more thing I’d like to pre-register: people who fill out the survey who aren’t frontier AI researchers will generally report higher speedups because their work is generally less compute-loaded and sometimes more greenfieldy or requiring less expertise, but we should give by far the most weight to frontier AI researchers.
(feel free to not go any deeper, appreciate you having engaged as much as you have!)
Yup, was just saying my first-pass guess would have been a less large labour->progress penalty. I do defer here fairly thoroughly.
The right way to think about 1.05x vs. 1.2x is not a 75% reduction, but instead what is the exponent for which 1.05^n=1.2
hmm, seems true if you’re expecting the people to not have applied a correction already, but less true if they are already making a correction and you’re estimating how wrong their correction is?
And yup, agree with that preregistration on all counts.
That resolves the inconsistency. I do worry that dropping a 20% speed-up to a 5% one, especially if post hoc, might cover up some important signal, but I’m sure you’ve put dramatically more cycles into thinking about this than me.
Thanks for the survey, would it make sense to just pass this form around so the numbers go to the same place and you’ll check, or should I make a copy and send results if I get them?
Your website lists
April 2025 as 1.13x
August 2025 as 1.21x
December 2025 as 1.30x
December 2024 as 1.05x (which seems contradicted by your survey, if the replies were in November)
If you think today’s number is ~1.33x we’re ~7 months ahead of schedule vs the listed forecast, unless I’m really misreading something.
Also, re: “would be excited for others to.”, is the survey public or easy to share if someone wanted to use the same questions?
And I’d bet 1:4 for the current number is actually >1.5x, if that’s more interesting. You’ve updated me to not have that as the main expectation, but still seems pretty plausible. Obviously depends on someone rerunning the survey, and reasonable that you’ve got your hands full with other things right now.
I also realized that I believe that confusingly the survey asks about speedup vs. no post-2022 AIs, while I believe the scenario side panel is for no post-2023 AIs, which should make the side panel numbers lower, unclear exactly how much given 2023 AIs weren’t particularly useful.
I can switch the number to 2023?
Yup, seems good
Okay, switched. I’m curious about why you didn’t set the baseline to “no AI help”, especially if you expect pre-2024 AI to be mostly useless, as that seems like a cleaner comparison than asking people to remember how good old AIs were?
No AI help seems harder to compare to since it’s longer ago, it seems easiest to think of something close to today as the baseline when thinking about future speedups. Also for timelines/takeoff modeling it’s a bit nicer to set the baseline to be more recent (looks like for those we again confusingly allowed 2024 AIs in the baseline as well rather than just 2023. Perhaps I should have standardized that with the side panel).
I think this risks people underappreciating how much progress is being sped up, my naive read of the UI was the numbers were based on “no AI” and I’d bet most readers would think the same at a glance. Changing the text from “AI provides the following speedups:” to “AI provides the following speedups from a baseline of 2022⁄3 AI:” would resolve this (I would guess common) misreading.
Yup feel free to make that change, sounds good
Clarification:
Change to the form to ask about without AI assistence?
Change to the website to refer to “AI provides the following speedups from a baseline of 2022⁄3 AI:”? (I don’t have write access)
(assuming 1 for now, will revert if incorrect)
Oh I misunderstood you sorry. I think the form should have post-2023, not sure about the website because it adds complexity and I’m skeptical that it’s common that people are importantly confused by it as is.
I think the survey is an overestimate for the reason I gave above, I think this stuff is subtle and researchers are likely to underestimate the decrease from labor speedup to progress speedup, especially in this sort of survey where it didn’t involve discussing with them verbally. Based on their responses to other questions in the survey seems like at least 2 people didn’t understand the difference between labor and overall progress/productivity.
Here is the survey: https://forms.gle/6GUbPR159ftBQcVF6. The question we’re discussing is: “[optional] What is the current productivity multiplier on algorithmic progress due to AI assistance?”
Edit: Also we didn’t spend large amounts of time on these early numbers, they’re not meant to be that precise but just rough best guesses.
Wait, actually, I want to double click on this. What was the process that caused you to transform the number you got from the survey (1.2x) to the number on the website (1.05x)? Is there a question that could be asked which would not require a correction? Or which would have a pre-registered correction?[1]
Bonus: Was this one pre-registered?
I’m not sure what the exact process was, tbh my guess is that they were estimated mostly independently but likely sanity checked with the survey to some extent in mind. It seems like they line up about right, given the 2022 vs. 2023 difference, the intuition regarding underadjusting for labor->progress, and giving weight to our own views as well rather than just the survey, given that we’ve thought more about this than survey takers (while of course they have the advantage of currently doing frontier AI research).
I’d make less of an adjustment if we asked people to give their reasoning including the adjustment from labor speedup to overall progress speedup and only included people who gave answers that demonstrated good understanding of this consideration and a not obviously unreasonable adjustment level.
Alright, my first pass guess would have been algorithmic progress seems like the kind of thing that eats a much smaller penalty than most forms org-level progress, not none but not a 75% reduction, and not likely more than a 50% reduction, but you guys have the track record.
Cool, added a nudge to the last question.
I think it’s not worth getting into this too much more as I don’t feel strongly about the exact 1.05x, but I feel compelled to note a few quick things:
I’m not sure exactly what you mean by eating a smaller penalty but I think the labor->progress penalty is quite large
The right way to think about 1.05x vs. 1.2x is not a 75% reduction, but instead what is the exponent for which 1.05^n=1.2
Remember the 2022 vs. 2023 difference, though my guess is that the responses wouldn’t have been that sensitive to this
Also one more thing I’d like to pre-register: people who fill out the survey who aren’t frontier AI researchers will generally report higher speedups because their work is generally less compute-loaded and sometimes more greenfieldy or requiring less expertise, but we should give by far the most weight to frontier AI researchers.
(feel free to not go any deeper, appreciate you having engaged as much as you have!)
Yup, was just saying my first-pass guess would have been a less large labour->progress penalty. I do defer here fairly thoroughly.
hmm, seems true if you’re expecting the people to not have applied a correction already, but less true if they are already making a correction and you’re estimating how wrong their correction is?
And yup, agree with that preregistration on all counts.
That resolves the inconsistency. I do worry that dropping a 20% speed-up to a 5% one, especially if post hoc, might cover up some important signal, but I’m sure you’ve put dramatically more cycles into thinking about this than me.
Thanks for the survey, would it make sense to just pass this form around so the numbers go to the same place and you’ll check, or should I make a copy and send results if I get them?
I think a copy would be best, thanks!
This survey looks like it’s asking something different? It’s asking about human range, no mention of speed-up from AI.
Look at the question I mentioned above about the current productivity multiplier
Oh, yup, missed that optional question in my ctrl-f. Thanks!