Thanks. For the record, my position is that we won’t see progress that looks like “For t-AGI, t increases by +1 OOM every X years” but rather that the rate of OOMs per year will start off slow and then accelerate. So e.g. here’s what I think t will look like as a function of years:
Year
Richard (?) guess
Daniel guess
2023
1
5
2024
5
15
2025
25
100
2026
100
2000
2027
500
Infinity (singularity)
2028
2,500
2029
10,000
2030
50,000
2031
250,000
2032
1,000,000
I think this partly because of the way I think generalization works (I think e.g. once AIs have gotten really good at all 2000-second tasks, they’ll generalize quickly with just a bit more scaling, tuning, etc. to much longer tasks) and partly because of R&D acceleration effects where e.g. once AIs have gotten really good at all 2000-second tasks, AI research can be partially automated and will go substantially faster, getting us to 10,000-second tasks quicker, which then causes further speedup in R&D, etc.
This also is related to the crux between me and Ajeya Cotra, between me and Paul Christiano, between me and Rohin Shah… I think their view is that the “2020 AGI/TAI training requirements” variable is a lot higher than I think (they are thinking something like 1e36 FLOP, I’m thinking something like 1e29) because they are thinking you’ll need to do lots and lots of long-horizon training to get systems that are good at long-horizon tasks, whereas I’m thinking you’ll be able to get away with mostly training on shorter tasks and then a bit of fine-tuning on longer tasks.
Sorry for a slightly dumb question but in your part of the table you set 2000 as the year before singularity and your explanation is that 2000-second tasks jump to singularity. Is your model of fast take-off then contingent on there being more special sauce for intelligence being somewhat redundant as a crux because recursive self-improvement is just much more effective. I’m having trouble envisioning a 2000-second task + more scaling and tuning --> singularity.
Additional question is what your model of falsification is for let’s say 25-second task vs. 100-second task in 2025 because it seems like reading your old vignettes you really nailed the diplomacy AI part.
Also slightly pedantic but there’s a typo on 2029 on Richard’s guess.
I’m not sure I understand your questions, can you elaborate/explain them more?
Note that the numbers in this chart don’t represent “There is at least one task of length X that AI can do” but rather “There exist AGIs which can do pretty much all relevant intellectual tasks of length X or less.” So my claim is, by the time we get to 2000 in that, such AGIs will be automating huge portions of AI R&D, which will speed it up, which will get us to 10,000 in less than a year, which will speed things up even more, etc.
Yeah, re-reading I realise I was unclear. Given your claim: “by the time we get to 2000 in that, such AGIs will be automating huge portions of AI R&D,”. I’m asking the following:
Is the 2000 mark predicated on automation of things we can’t envision now (finding secret sauce to singularity) or is it predicated off pushing existing things like AI R&D finds better compute or is it a combination of both?
What’s the empirical on the ground representative modal action you’re seeing at 2025 from either your vignette (e.g. I found the diplomacy AI super important for grokking what short timelines were to me). I guess it’s more asking what you see as the divergence between you and Richard at 2025 that’s represented by the difference of 25 and 100.
A bit of both I guess? Insofar as there are things we can’t imagine now, but that a smart hardworking human expert could do in 2000 seconds if and when the situation arose / if and when they put their mind to it, then those things will be doable by AI in 2026 in my median projection.
Unfortunately, my view and Richard’s views don’t really diverge strongly until it’s pretty much too late, as you can see from the chart. :( I think both of us will be very interested in trying to track/measure growth in t-AGI, but he’d be trying to measure the growth rate and then assume it continues going at that rate for years and years, whereas I’d be trying to measure the growth rate so I can guess at how long we have left before it ‘goes critical’ so to speak. For ‘going critical’ we can look at how fast AI R&D is going and try to measure and track that and notice when it starts to speed up due to automation. It’s already probably sped up a little bit, e.g. Copilot helping with coding...
Thanks. For the record, my position is that we won’t see progress that looks like “For t-AGI, t increases by +1 OOM every X years” but rather that the rate of OOMs per year will start off slow and then accelerate. So e.g. here’s what I think t will look like as a function of years:
I think this partly because of the way I think generalization works (I think e.g. once AIs have gotten really good at all 2000-second tasks, they’ll generalize quickly with just a bit more scaling, tuning, etc. to much longer tasks) and partly because of R&D acceleration effects where e.g. once AIs have gotten really good at all 2000-second tasks, AI research can be partially automated and will go substantially faster, getting us to 10,000-second tasks quicker, which then causes further speedup in R&D, etc.
This also is related to the crux between me and Ajeya Cotra, between me and Paul Christiano, between me and Rohin Shah… I think their view is that the “2020 AGI/TAI training requirements” variable is a lot higher than I think (they are thinking something like 1e36 FLOP, I’m thinking something like 1e29) because they are thinking you’ll need to do lots and lots of long-horizon training to get systems that are good at long-horizon tasks, whereas I’m thinking you’ll be able to get away with mostly training on shorter tasks and then a bit of fine-tuning on longer tasks.
Sorry for a slightly dumb question but in your part of the table you set 2000 as the year before singularity and your explanation is that 2000-second tasks jump to singularity. Is your model of fast take-off then contingent on there being more special sauce for intelligence being somewhat redundant as a crux because recursive self-improvement is just much more effective. I’m having trouble envisioning a 2000-second task + more scaling and tuning --> singularity.
Additional question is what your model of falsification is for let’s say 25-second task vs. 100-second task in 2025 because it seems like reading your old vignettes you really nailed the diplomacy AI part.
Also slightly pedantic but there’s a typo on 2029 on Richard’s guess.
Oooh, thanks for pointing out the typo! Fixed.
I’m not sure I understand your questions, can you elaborate/explain them more?
Note that the numbers in this chart don’t represent “There is at least one task of length X that AI can do” but rather “There exist AGIs which can do pretty much all relevant intellectual tasks of length X or less.” So my claim is, by the time we get to 2000 in that, such AGIs will be automating huge portions of AI R&D, which will speed it up, which will get us to 10,000 in less than a year, which will speed things up even more, etc.
Yeah, re-reading I realise I was unclear. Given your claim: “by the time we get to 2000 in that, such AGIs will be automating huge portions of AI R&D,”. I’m asking the following:
Is the 2000 mark predicated on automation of things we can’t envision now (finding secret sauce to singularity) or is it predicated off pushing existing things like AI R&D finds better compute or is it a combination of both?
What’s the empirical on the ground representative modal action you’re seeing at 2025 from either your vignette (e.g. I found the diplomacy AI super important for grokking what short timelines were to me). I guess it’s more asking what you see as the divergence between you and Richard at 2025 that’s represented by the difference of 25 and 100.
Hopefully that made the questions clearer.
A bit of both I guess? Insofar as there are things we can’t imagine now, but that a smart hardworking human expert could do in 2000 seconds if and when the situation arose / if and when they put their mind to it, then those things will be doable by AI in 2026 in my median projection.
Unfortunately, my view and Richard’s views don’t really diverge strongly until it’s pretty much too late, as you can see from the chart. :( I think both of us will be very interested in trying to track/measure growth in t-AGI, but he’d be trying to measure the growth rate and then assume it continues going at that rate for years and years, whereas I’d be trying to measure the growth rate so I can guess at how long we have left before it ‘goes critical’ so to speak. For ‘going critical’ we can look at how fast AI R&D is going and try to measure and track that and notice when it starts to speed up due to automation. It’s already probably sped up a little bit, e.g. Copilot helping with coding...