This was from Nov 2024 to Mar 2025 so fairly recent. I think the transition to faster was mostly due to the transition to reasoning models and perhaps the beginnings of increased generalization from shorter to longer time horizons.
Edit: the responses are from between Nov 2024 and Mar 2025. Responses are in increasing order: 1.05-1.1, 1.15, 1.2, 1.3, 2. The lowest one is the most recent but is from a former not current frontier AI researcher.
The switch to reasoning models does line up well, probably more cleanly. Moved that to main hypothesis, thanks.
Having some later responses makes it less likely they missed the change, curious if the other responses were closer to Dec or March. I would guess the not-current-researcher one being excluded probably makes sense? The datapoint from me is not exactly 2x on this, but ‘most of an approximately 2x’, so would need revisiting with the exact question before it could be directly included, and I’d imagine you’d want the source.
I still have some weight on higher research boost from AI than your model is expecting, due to other lines of evidence, but not putting quite as much weight on it.
That seems like stale data, given how these graphs look. Even with the updates you caused, I’m happy to offer an even odds token bet ($100?) that a rerun of a similar survey would give significantly higher average (at least +0.2 over the predicted 1.13x, or about the AI you expect in Dec 2025). I’d be even more happy if the question asked about the researcher’s own productivity, as that seems like something they’d have better vision of, but would be pretty noisy with small sample so reasonable to stick with original question.
You mean the median would be at least 1.33x rather than the previous 1.2x? Sounds about right so don’t feel the need to bet against. Also I’m not planning on doing a follow-up survey but would be excited for others to.
December 2024 as 1.05x (which seems contradicted by your survey, if the replies were in November)
If you think today’s number is ~1.33x we’re ~7 months ahead of schedule vs the listed forecast, unless I’m really misreading something.
Also, re: “would be excited for others to.”, is the survey public or easy to share if someone wanted to use the same questions?
And I’d bet 1:4 for the current number is actually >1.5x, if that’s more interesting. You’ve updated me to not have that as the main expectation, but still seems pretty plausible. Obviously depends on someone rerunning the survey, and reasonable that you’ve got your hands full with other things right now.
I also realized that I believe that confusingly the survey asks about speedup vs. no post-2022 AIs, while I believe the scenario side panel is for no post-2023 AIs, which should make the side panel numbers lower, unclear exactly how much given 2023 AIs weren’t particularly useful.
Okay, switched. I’m curious about why you didn’t set the baseline to “no AI help”, especially if you expect pre-2024 AI to be mostly useless, as that seems like a cleaner comparison than asking people to remember how good old AIs were?
No AI help seems harder to compare to since it’s longer ago, it seems easiest to think of something close to today as the baseline when thinking about future speedups. Also for timelines/takeoff modeling it’s a bit nicer to set the baseline to be more recent (looks like for those we again confusingly allowed 2024 AIs in the baseline as well rather than just 2023. Perhaps I should have standardized that with the side panel).
I think this risks people underappreciating how much progress is being sped up, my naive read of the UI was the numbers were based on “no AI” and I’d bet most readers would think the same at a glance. Changing the text from “AI provides the following speedups:” to “AI provides the following speedups from a baseline of 2022⁄3 AI:” would resolve this (I would guess common) misreading.
Oh I misunderstood you sorry. I think the form should have post-2023, not sure about the website because it adds complexity and I’m skeptical that it’s common that people are importantly confused by it as is.
I think the survey is an overestimate for the reason I gave above, I think this stuff is subtle and researchers are likely to underestimate the decrease from labor speedup to progress speedup, especially in this sort of survey where it didn’t involve discussing with them verbally. Based on their responses to other questions in the survey seems like at least 2 people didn’t understand the difference between labor and overall progress/productivity.
Here is the survey: https://forms.gle/6GUbPR159ftBQcVF6. The question we’re discussing is: “[optional] What is the current productivity multiplier on algorithmic progress due to AI assistance?”
Edit: Also we didn’t spend large amounts of time on these early numbers, they’re not meant to be that precise but just rough best guesses.
Wait, actually, I want to double click on this. What was the process that caused you to transform the number you got from the survey (1.2x) to the number on the website (1.05x)? Is there a question that could be asked which would not require a correction? Or which would have a pre-registered correction?[1]
I’m not sure what the exact process was, tbh my guess is that they were estimated mostly independently but likely sanity checked with the survey to some extent in mind. It seems like they line up about right, given the 2022 vs. 2023 difference, the intuition regarding underadjusting for labor->progress, and giving weight to our own views as well rather than just the survey, given that we’ve thought more about this than survey takers (while of course they have the advantage of currently doing frontier AI research).
I’d make less of an adjustment if we asked people to give their reasoning including the adjustment from labor speedup to overall progress speedup and only included people who gave answers that demonstrated good understanding of this consideration and a not obviously unreasonable adjustment level.
Alright, my first pass guess would have been algorithmic progress seems like the kind of thing that eats a much smaller penalty than most forms org-level progress, not none but not a 75% reduction, and not likely more than a 50% reduction, but you guys have the track record.
I think it’s not worth getting into this too much more as I don’t feel strongly about the exact 1.05x, but I feel compelled to note a few quick things:
I’m not sure exactly what you mean by eating a smaller penalty but I think the labor->progress penalty is quite large
The right way to think about 1.05x vs. 1.2x is not a 75% reduction, but instead what is the exponent for which 1.05^n=1.2
Remember the 2022 vs. 2023 difference, though my guess is that the responses wouldn’t have been that sensitive to this
Also one more thing I’d like to pre-register: people who fill out the survey who aren’t frontier AI researchers will generally report higher speedups because their work is generally less compute-loaded and sometimes more greenfieldy or requiring less expertise, but we should give by far the most weight to frontier AI researchers.
(feel free to not go any deeper, appreciate you having engaged as much as you have!)
Yup, was just saying my first-pass guess would have been a less large labour->progress penalty. I do defer here fairly thoroughly.
The right way to think about 1.05x vs. 1.2x is not a 75% reduction, but instead what is the exponent for which 1.05^n=1.2
hmm, seems true if you’re expecting the people to not have applied a correction already, but less true if they are already making a correction and you’re estimating how wrong their correction is?
And yup, agree with that preregistration on all counts.
That resolves the inconsistency. I do worry that dropping a 20% speed-up to a 5% one, especially if post hoc, might cover up some important signal, but I’m sure you’ve put dramatically more cycles into thinking about this than me.
Thanks for the survey, would it make sense to just pass this form around so the numbers go to the same place and you’ll check, or should I make a copy and send results if I get them?
This was from Nov 2024 to Mar 2025 so fairly recent. I think the transition to faster was mostly due to the transition to reasoning models and perhaps the beginnings of increased generalization from shorter to longer time horizons.
Edit: the responses are from between Nov 2024 and Mar 2025. Responses are in increasing order: 1.05-1.1, 1.15, 1.2, 1.3, 2. The lowest one is the most recent but is from a former not current frontier AI researcher.
The switch to reasoning models does line up well, probably more cleanly. Moved that to main hypothesis, thanks.
Having some later responses makes it less likely they missed the change, curious if the other responses were closer to Dec or March. I would guess the not-current-researcher one being excluded probably makes sense? The datapoint from me is not exactly 2x on this, but ‘most of an approximately 2x’, so would need revisiting with the exact question before it could be directly included, and I’d imagine you’d want the source.
I still have some weight on higher research boost from AI than your model is expecting, due to other lines of evidence, but not putting quite as much weight on it.
Most of the responses were in Nov.
That seems like stale data, given how these graphs look. Even with the updates you caused, I’m happy to offer an even odds token bet ($100?) that a rerun of a similar survey would give significantly higher average (at least +0.2 over the predicted 1.13x, or about the AI you expect in Dec 2025). I’d be even more happy if the question asked about the researcher’s own productivity, as that seems like something they’d have better vision of, but would be pretty noisy with small sample so reasonable to stick with original question.
You mean the median would be at least 1.33x rather than the previous 1.2x? Sounds about right so don’t feel the need to bet against. Also I’m not planning on doing a follow-up survey but would be excited for others to.
Your website lists
April 2025 as 1.13x
August 2025 as 1.21x
December 2025 as 1.30x
December 2024 as 1.05x (which seems contradicted by your survey, if the replies were in November)
If you think today’s number is ~1.33x we’re ~7 months ahead of schedule vs the listed forecast, unless I’m really misreading something.
Also, re: “would be excited for others to.”, is the survey public or easy to share if someone wanted to use the same questions?
And I’d bet 1:4 for the current number is actually >1.5x, if that’s more interesting. You’ve updated me to not have that as the main expectation, but still seems pretty plausible. Obviously depends on someone rerunning the survey, and reasonable that you’ve got your hands full with other things right now.
I also realized that I believe that confusingly the survey asks about speedup vs. no post-2022 AIs, while I believe the scenario side panel is for no post-2023 AIs, which should make the side panel numbers lower, unclear exactly how much given 2023 AIs weren’t particularly useful.
I can switch the number to 2023?
Yup, seems good
Okay, switched. I’m curious about why you didn’t set the baseline to “no AI help”, especially if you expect pre-2024 AI to be mostly useless, as that seems like a cleaner comparison than asking people to remember how good old AIs were?
No AI help seems harder to compare to since it’s longer ago, it seems easiest to think of something close to today as the baseline when thinking about future speedups. Also for timelines/takeoff modeling it’s a bit nicer to set the baseline to be more recent (looks like for those we again confusingly allowed 2024 AIs in the baseline as well rather than just 2023. Perhaps I should have standardized that with the side panel).
I think this risks people underappreciating how much progress is being sped up, my naive read of the UI was the numbers were based on “no AI” and I’d bet most readers would think the same at a glance. Changing the text from “AI provides the following speedups:” to “AI provides the following speedups from a baseline of 2022⁄3 AI:” would resolve this (I would guess common) misreading.
Yup feel free to make that change, sounds good
Clarification:
Change to the form to ask about without AI assistence?
Change to the website to refer to “AI provides the following speedups from a baseline of 2022⁄3 AI:”? (I don’t have write access)
(assuming 1 for now, will revert if incorrect)
Oh I misunderstood you sorry. I think the form should have post-2023, not sure about the website because it adds complexity and I’m skeptical that it’s common that people are importantly confused by it as is.
I think the survey is an overestimate for the reason I gave above, I think this stuff is subtle and researchers are likely to underestimate the decrease from labor speedup to progress speedup, especially in this sort of survey where it didn’t involve discussing with them verbally. Based on their responses to other questions in the survey seems like at least 2 people didn’t understand the difference between labor and overall progress/productivity.
Here is the survey: https://forms.gle/6GUbPR159ftBQcVF6. The question we’re discussing is: “[optional] What is the current productivity multiplier on algorithmic progress due to AI assistance?”
Edit: Also we didn’t spend large amounts of time on these early numbers, they’re not meant to be that precise but just rough best guesses.
Wait, actually, I want to double click on this. What was the process that caused you to transform the number you got from the survey (1.2x) to the number on the website (1.05x)? Is there a question that could be asked which would not require a correction? Or which would have a pre-registered correction?[1]
Bonus: Was this one pre-registered?
I’m not sure what the exact process was, tbh my guess is that they were estimated mostly independently but likely sanity checked with the survey to some extent in mind. It seems like they line up about right, given the 2022 vs. 2023 difference, the intuition regarding underadjusting for labor->progress, and giving weight to our own views as well rather than just the survey, given that we’ve thought more about this than survey takers (while of course they have the advantage of currently doing frontier AI research).
I’d make less of an adjustment if we asked people to give their reasoning including the adjustment from labor speedup to overall progress speedup and only included people who gave answers that demonstrated good understanding of this consideration and a not obviously unreasonable adjustment level.
Alright, my first pass guess would have been algorithmic progress seems like the kind of thing that eats a much smaller penalty than most forms org-level progress, not none but not a 75% reduction, and not likely more than a 50% reduction, but you guys have the track record.
Cool, added a nudge to the last question.
I think it’s not worth getting into this too much more as I don’t feel strongly about the exact 1.05x, but I feel compelled to note a few quick things:
I’m not sure exactly what you mean by eating a smaller penalty but I think the labor->progress penalty is quite large
The right way to think about 1.05x vs. 1.2x is not a 75% reduction, but instead what is the exponent for which 1.05^n=1.2
Remember the 2022 vs. 2023 difference, though my guess is that the responses wouldn’t have been that sensitive to this
Also one more thing I’d like to pre-register: people who fill out the survey who aren’t frontier AI researchers will generally report higher speedups because their work is generally less compute-loaded and sometimes more greenfieldy or requiring less expertise, but we should give by far the most weight to frontier AI researchers.
(feel free to not go any deeper, appreciate you having engaged as much as you have!)
Yup, was just saying my first-pass guess would have been a less large labour->progress penalty. I do defer here fairly thoroughly.
hmm, seems true if you’re expecting the people to not have applied a correction already, but less true if they are already making a correction and you’re estimating how wrong their correction is?
And yup, agree with that preregistration on all counts.
That resolves the inconsistency. I do worry that dropping a 20% speed-up to a 5% one, especially if post hoc, might cover up some important signal, but I’m sure you’ve put dramatically more cycles into thinking about this than me.
Thanks for the survey, would it make sense to just pass this form around so the numbers go to the same place and you’ll check, or should I make a copy and send results if I get them?
I think a copy would be best, thanks!
This survey looks like it’s asking something different? It’s asking about human range, no mention of speed-up from AI.
Look at the question I mentioned above about the current productivity multiplier
Oh, yup, missed that optional question in my ctrl-f. Thanks!