I think something like 2032-2037 is probably the period that most people I know who have reasonably short timelines consider most likely.
I honestly didn’t know that. Thank you for mentioning it. Almost everything I hear is people worrying about AGI in the next few years, not AGI a decade from now.
if you choose 2028 as some kind of Schelling time to decide whether things are markedly slower than expected then I think you are deciding on a strategy that doesn’t make sense by like 80% of the registered predictions that people have.
Just to check: you’re saying that by 2028 something like 80% of the registered predictions still won’t have relevant evidence against them?
As both a pragmatic matter and a moral one, I really hope we can find ways of making more of those predictions more falsifiable sooner. If anything in the vague family of what I’m saying is right, but if folk won’t halt, melt, & catch fire for at least a decade, then that’s an awful lot of pointless suffering and wasted talent. I’m also skeptical that there’ll actually be a real HMC in a decade; if the metacognitive blindspot type I’m pointing at is active, then waiting a decade is part of the strategy for not having to look, and it’ll come up with yet more reasons not to look when AGI doesn’t happen ten years from now too.
“are we alive and are things still OK in 2028?” is a terrible way to operationalize that. Most people do not expect anything particularly terrible to have happened by 2028!
Cool. Noted. So, what can we observe by 2028 that’d cause us pause about whether there’s a collective confusion/projection playing a significant role?
The key decision-point in my model at which things might become a bit different is if we hit the end of the compute overhang, and you can’t scale up AI further simply by more financial investment, but instead now need to substantially ramp up global compute production, and make algorithmic progress, which might markedly slow down progress.
This angle misses my point. Your analysis makes sense, but it’s talking about something different.
I think you’re trying to suggest what observations would cause you to update about AI timelines being longer than you currently think they are. Yes?
I’m asking what we should observe if it turns out that the degree of focus on narrating doom is more due to an underlying emotional distress than due to correct calibration to the situation we’re in.
One effect of that might be that AI timelines bias toward the pessimistic, which means that on average we should keep finding that they’re longer than the consensus keeps converging on. But I think that’s both slow and noisy as a way of detecting it.
just the “I propose 2028 as the time to re-evaluate things, and I think we really want to change things if stuff still looks fine” feels to me like it fails to engage with people’s actually registered predictions.
Noted. Thanks for pointing it out. I do think I failed to engage this way.
I still think my overall point stands though. I didn’t think things would look fine; my expectation is that things will continue to look ever more dire. I’m just hoping that if we can distinguish between (a) dire because things are in fact getting worse versus (b) dire because that’s what the emotional Shepard tone does, then in 2.5 years we could pause and check which one seems to be responsible for the increase in doom, and if it’s the latter then HMC.
So, again: what could we observe at the start of 2028 that would create pause this way?
As you implied above, pessimism is driven only secondarily by timelines. If things in 2028 don’t look much different than they do now, that’s evidence for longer timelines (maybe a little longer, maybe a lot). But it’s inherently not much evidence about how dangerous superintelligence will be when it does arrive. If the situation is basically the same, then our state of knowledge is basically the same.
So what would be good evidence that worrying about alignment was unnecessary? The obvious one is if we get superintelligence and nothing very bad happens, despite the alignment problem remaining unsolved. But that’s like pulling the trigger to see if the gun is loaded. Prior to superintelligence, personally I’d be more optimistic if we saw AI progress requiring even more increasing compute than the current trend—if the first superintelligences were very reliant on massive pools of tightly integrated compute, and had very limited inference capacity, that would make us less vulnerable and give us more time to adapt to them. Also, if we saw a slowdown in algorithmic progress despite widespread deployment of increasingly capable coding software, that would be a very encouraging sign that recursive self-improvement might happen slowly.
So, again: what could we observe at the start of 2028 that would create pause this way?
Very little. I’ve been seriously thinking about ASI since the early 00s. Around 2004-2007, I put my timeline around 2035-2045, depending on the rate of GPU advancements. Given how hardware and LLM progress actually played out, my timeline is currently around 2035.
I do expect LLMs (as we know them now) to stall before 2028, if they haven’t already. Something is missing. I have very concrete guesses as to what is missing, and it’s an area of active research. But I also expect the missing piece adds less than a single power of 10 to existing training and inference costs. So once someone publishes it in any kind of convincing way, then I’d estimate better than an 80% chance of uncontrolled ASI within 10 years.
Now, there are lots of things I could see in 2035 that would cause me to update away from this scenario. I did, in fact, update away from my 2004-2007 predictions by 2018 or so, largely because nothing like ChatGPT 3.5 existed by that point. GPT 3 made me nervous again, and 3.5 Instruct caused me to update all the way back to my original timeline. And if we’re still stalled in 2035, then sure, I’ll update heavily away from ASI again. But I’m already predicting the LLM S-curve to flatten out around now, resulting in less investment in Chinchilla scaling and more investment in algorithmic improvement. But since algorithmic improvement is (1) hard to predict, and (2) where I think the actual danger lies, I don’t intend to make any near-term updates away ASI.
I honestly didn’t know that. Thank you for mentioning it. Almost everything I hear is people worrying about AGI in the next few years, not AGI a decade from now.
Just to check: you’re saying that by 2028 something like 80% of the registered predictions still won’t have relevant evidence against them?
As both a pragmatic matter and a moral one, I really hope we can find ways of making more of those predictions more falsifiable sooner. If anything in the vague family of what I’m saying is right, but if folk won’t halt, melt, & catch fire for at least a decade, then that’s an awful lot of pointless suffering and wasted talent. I’m also skeptical that there’ll actually be a real HMC in a decade; if the metacognitive blindspot type I’m pointing at is active, then waiting a decade is part of the strategy for not having to look, and it’ll come up with yet more reasons not to look when AGI doesn’t happen ten years from now too.
Cool. Noted. So, what can we observe by 2028 that’d cause us pause about whether there’s a collective confusion/projection playing a significant role?
This angle misses my point. Your analysis makes sense, but it’s talking about something different.
I think you’re trying to suggest what observations would cause you to update about AI timelines being longer than you currently think they are. Yes?
I’m asking what we should observe if it turns out that the degree of focus on narrating doom is more due to an underlying emotional distress than due to correct calibration to the situation we’re in.
One effect of that might be that AI timelines bias toward the pessimistic, which means that on average we should keep finding that they’re longer than the consensus keeps converging on. But I think that’s both slow and noisy as a way of detecting it.
Noted. Thanks for pointing it out. I do think I failed to engage this way.
I still think my overall point stands though. I didn’t think things would look fine; my expectation is that things will continue to look ever more dire. I’m just hoping that if we can distinguish between (a) dire because things are in fact getting worse versus (b) dire because that’s what the emotional Shepard tone does, then in 2.5 years we could pause and check which one seems to be responsible for the increase in doom, and if it’s the latter then HMC.
So, again: what could we observe at the start of 2028 that would create pause this way?
As you implied above, pessimism is driven only secondarily by timelines. If things in 2028 don’t look much different than they do now, that’s evidence for longer timelines (maybe a little longer, maybe a lot). But it’s inherently not much evidence about how dangerous superintelligence will be when it does arrive. If the situation is basically the same, then our state of knowledge is basically the same.
So what would be good evidence that worrying about alignment was unnecessary? The obvious one is if we get superintelligence and nothing very bad happens, despite the alignment problem remaining unsolved. But that’s like pulling the trigger to see if the gun is loaded. Prior to superintelligence, personally I’d be more optimistic if we saw AI progress requiring even more increasing compute than the current trend—if the first superintelligences were very reliant on massive pools of tightly integrated compute, and had very limited inference capacity, that would make us less vulnerable and give us more time to adapt to them. Also, if we saw a slowdown in algorithmic progress despite widespread deployment of increasingly capable coding software, that would be a very encouraging sign that recursive self-improvement might happen slowly.
Very little. I’ve been seriously thinking about ASI since the early 00s. Around 2004-2007, I put my timeline around 2035-2045, depending on the rate of GPU advancements. Given how hardware and LLM progress actually played out, my timeline is currently around 2035.
I do expect LLMs (as we know them now) to stall before 2028, if they haven’t already. Something is missing. I have very concrete guesses as to what is missing, and it’s an area of active research. But I also expect the missing piece adds less than a single power of 10 to existing training and inference costs. So once someone publishes it in any kind of convincing way, then I’d estimate better than an 80% chance of uncontrolled ASI within 10 years.
Now, there are lots of things I could see in 2035 that would cause me to update away from this scenario. I did, in fact, update away from my 2004-2007 predictions by 2018 or so, largely because nothing like ChatGPT 3.5 existed by that point. GPT 3 made me nervous again, and 3.5 Instruct caused me to update all the way back to my original timeline. And if we’re still stalled in 2035, then sure, I’ll update heavily away from ASI again. But I’m already predicting the LLM S-curve to flatten out around now, resulting in less investment in Chinchilla scaling and more investment in algorithmic improvement. But since algorithmic improvement is (1) hard to predict, and (2) where I think the actual danger lies, I don’t intend to make any near-term updates away ASI.