Situational Awareness and AI 2027 have been signal boosted to normal people more than other predictions, though. Like, they both have their own website, AI 2027 has a bunch of fancy client-side animation and Scott Alexander collaborated, and someone made a Youtube video on AI 2027.
While AI safety people have criticized the timeline predictions to some extent, there hasn’t been much in-depth criticism (aside from the recent very long post on AI 2027), the general sentiment on their timelines seems positive (although Situational Awareness has been criticized for contributing to arms race dynamics).
I get that someone who looks at AI safety people’s timelines in more detail would get a different impression. Though, notably, Metaculus lists Jan 2027 as a “community prediction” of “weakly general AI”. Sure, someone could argue that weakly general AI doesn’t imply human-level AGI soon after, but mostly when I see AI safety people point to this Metaculus market, it’s as evidence that experts believe human-level AGI will arrive in the next few years, there is not emphasis on the delta between weakly general AI and human-level AGI.
So I see how an outsider would see more 2027-2029 timelines from AI safety people and assume that’s what they consider a reasonable median, if they aren’t looking into it closely. This is partially due to internet/social media attention dynamics, where surprising predictions get more attention.
What I think we could agree on is that, if 2028 rolls around and things seem pretty much like today, then whatever distributed attention algorithm promotes things like Situational Awareness and AI 2027 to people’s attention in practice leads to flawed predictions, and people who were worried because of listening to this algorithm should chill out and re-evaluate. (We probably also agree they should chill out today because this popular content isn’t reflective of AI safety people’s general opinions.)
Yep, agree that there is currently a biased coverage towards very short timelines. I think this makes sense in that the worlds where things are happening very soon are the worlds that from the perspective of a reasonable humanity require action now.[1]
I think despite the reasonable justification for focusing on the shorter timelines worlds for decision-making reasons, I do expect this to overall cause a bunch of people to walk away with the impression that people confidently predicted short timelines, and this in turn will cause a bunch of social conflict and unfortunate social accounting to happen in most worlds.
I on the margin would be excited to collaborate with people who would want to do similar things to AI 2027 or Situational Awareness for longer timelines.
I.e. in as much as you model the government as making reasonable risk-tradeoffs in the future, the short timeline worlds are the ones that require intervention to cause changes in decision-making now.
I am personally more pessimistic about humanity doing reasonable things, and think we might just want to grieve over short timeline worlds, but I sure don’t feel comfortable telling other people to not ring the alarm bell on potentially very large risks happening very soon, which seems plausible enough to me that absolutely it should be among the top considerations for most decision-makers out there.
Even if it does make sense strategically to put more attention on shorter timelines, that sure does not seem to be what actually drives the memetic advantage of short forecasts over long forecasts. If you want your attention to be steered in strategically-reasonable ways, you should probably first fully discount for the apparent memetic biases, and then go back and decide how much is reasonable to re-upweight short forecasts. Whatever bias the memetic advantage yields is unlikely to be the right bias, or even the right order of magnitude of relative attention bias.
I mean, I am not even sure it’s strategic given my other beliefs, and I was indeed saying that on the margin more longer-timeline coverage is worth it, so I think we agree.
What’s the longest timeline that you could still consider a short timeline by your own metric, and therefore a world “we might just want to grieve over”? I ask because, in your original comment you mentioned 2037 as a reasonably short timeline, and personally if we had an extra decade I’d be a lot less worried.
Edit: Oops, I responded to the first part of your question, not the second. My guess is timelines with less than 5 years seem really very hard, though we should still try. I think there is lots of hope in the 5-15 year timeline worlds. 15 years is just roughly the threshold of when I would stop considering someone’s timelines “short”, as a category.
I admit, it’s pretty disheartening to hear that, even if we had until 2040 (which seems less and less likely to me anyway), you’d still think there’s not much we could do but grieve in advance.
…people who were worried because of listening to this algorithm should chill out and re-evaluate.
And communication strategies based on appealing to such people’s reliance on those algorithms should also re-evaluate.
E.g., why did folk write AI 2027? Did they honestly think the timeline was that short? Were they trying to convey a picture that would scare people with something on a short enough timeline that they could feel it?
If the latter, we might be doing humanity a disservice, both by exhausting people from something akin to adrenal fatigue, and also as a result of crying wolf.
Yes, I honestly thought the timeline was that short. I now think it’s 50% by end of 2028; over the last year my timelines have lengthened by about a year.
He makes some obvious points everyone already knows about bottlenecks etc. but then doesn’t explain why all that adds up to a decade or more, instead of of a year, or a month, or a century. In our takeoff speeds forecast we try to give a quantitative estimate that takes into account all the bottlenecks etc.
E.g., why did folk write AI 2027? Did they honestly think the timeline was that short?
Isn’t it more like “I think there’s a 10% chance of transformative AI by 2027, and that is like 100x higher than what it looks like most people think, so people really need to think thru that timeline”?
Like, I generally put my median year at 2030-2032; if we make it to 2028, the situation will still feel like “oh jeez we probably only have a few years left”, unless we made it to 2028 thru a mechanism that clearly blocks transformative AI showing up in 2032. (Like, a lot is hinging on what “feels basically like today” means.)
Isn’t it more like “I think there’s a 10% chance of transformative AI by 2027, and that is like 100x higher than what it looks like most people think, so people really need to think thru that timeline”?
That might be. It sounds really plausible. I don’t know why they wrote it!
But all the same: I don’t think most people know what 10% likelihood of a severe outcome is like or how to think about it sensibly. My read is that the vast majority of people need to treat 10% likelihood of doom as either “It’s not going to happen” (because 10% is small) or “It’s guaranteed to happen” (because it’s a serious outcome if it does happen, and it’s plausible). So, amplifying the public awareness of this possibility seems more to me like moving awareness of the scenario from “Nothing existential is going to happen” to “This specific thing is the default thing to expect.”
So I expect that unless something is done to… I don’t know, magically educate the population on statistical thinking, or propagate a public message that it’s roughly right but its timeline is wrong? then the net effect will be that either (a) AI 2027 will have been collectively forgotten by 2028 in roughly the same way that, say, Trudeau’s use of the Emergencies Act has been forgotten; or (b) the predictions failing to pan out will be used as reason to dismiss other AI doom predictions that are apparently considered more likely.
The main benefit I see is if some key folk are made to think about AI doom scenarios in general as a result of AI 2027, and start to work out how to deal with other scenarios.
But I don’t know. That’s been part of this community’s strategy for over two decades. Get key people thinking about AI risk. And I’m not too keen on the results I’ve seen from that strategy so far.
Though, notably, Metaculus lists Jan 2027 as a “community prediction” of “weakly general AI”. Sure, someone could argue that weakly general AI doesn’t imply human-level AGI soon after
it does imply that, but i’m somewhat loathe to mention this at all, because i think the predictive quality you get from one question to another varies astronomically, and this is not something the casual reader will be able to glean
Situational Awareness and AI 2027 have been signal boosted to normal people more than other predictions, though. Like, they both have their own website, AI 2027 has a bunch of fancy client-side animation and Scott Alexander collaborated, and someone made a Youtube video on AI 2027.
While AI safety people have criticized the timeline predictions to some extent, there hasn’t been much in-depth criticism (aside from the recent very long post on AI 2027), the general sentiment on their timelines seems positive (although Situational Awareness has been criticized for contributing to arms race dynamics).
I get that someone who looks at AI safety people’s timelines in more detail would get a different impression. Though, notably, Metaculus lists Jan 2027 as a “community prediction” of “weakly general AI”. Sure, someone could argue that weakly general AI doesn’t imply human-level AGI soon after, but mostly when I see AI safety people point to this Metaculus market, it’s as evidence that experts believe human-level AGI will arrive in the next few years, there is not emphasis on the delta between weakly general AI and human-level AGI.
So I see how an outsider would see more 2027-2029 timelines from AI safety people and assume that’s what they consider a reasonable median, if they aren’t looking into it closely. This is partially due to internet/social media attention dynamics, where surprising predictions get more attention.
What I think we could agree on is that, if 2028 rolls around and things seem pretty much like today, then whatever distributed attention algorithm promotes things like Situational Awareness and AI 2027 to people’s attention in practice leads to flawed predictions, and people who were worried because of listening to this algorithm should chill out and re-evaluate. (We probably also agree they should chill out today because this popular content isn’t reflective of AI safety people’s general opinions.)
Yep, agree that there is currently a biased coverage towards very short timelines. I think this makes sense in that the worlds where things are happening very soon are the worlds that from the perspective of a reasonable humanity require action now.[1]
I think despite the reasonable justification for focusing on the shorter timelines worlds for decision-making reasons, I do expect this to overall cause a bunch of people to walk away with the impression that people confidently predicted short timelines, and this in turn will cause a bunch of social conflict and unfortunate social accounting to happen in most worlds.
I on the margin would be excited to collaborate with people who would want to do similar things to AI 2027 or Situational Awareness for longer timelines.
I.e. in as much as you model the government as making reasonable risk-tradeoffs in the future, the short timeline worlds are the ones that require intervention to cause changes in decision-making now.
I am personally more pessimistic about humanity doing reasonable things, and think we might just want to grieve over short timeline worlds, but I sure don’t feel comfortable telling other people to not ring the alarm bell on potentially very large risks happening very soon, which seems plausible enough to me that absolutely it should be among the top considerations for most decision-makers out there.
Even if it does make sense strategically to put more attention on shorter timelines, that sure does not seem to be what actually drives the memetic advantage of short forecasts over long forecasts. If you want your attention to be steered in strategically-reasonable ways, you should probably first fully discount for the apparent memetic biases, and then go back and decide how much is reasonable to re-upweight short forecasts. Whatever bias the memetic advantage yields is unlikely to be the right bias, or even the right order of magnitude of relative attention bias.
I mean, I am not even sure it’s strategic given my other beliefs, and I was indeed saying that on the margin more longer-timeline coverage is worth it, so I think we agree.
What’s the longest timeline that you could still consider a short timeline by your own metric, and therefore a world “we might just want to grieve over”? I ask because, in your original comment you mentioned 2037 as a reasonably short timeline, and personally if we had an extra decade I’d be a lot less worried.
About 15 years, I think?
Edit: Oops, I responded to the first part of your question, not the second. My guess is timelines with less than 5 years seem really very hard, though we should still try. I think there is lots of hope in the 5-15 year timeline worlds. 15 years is just roughly the threshold of when I would stop considering someone’s timelines “short”, as a category.
I admit, it’s pretty disheartening to hear that, even if we had until 2040 (which seems less and less likely to me anyway), you’d still think there’s not much we could do but grieve in advance.
And communication strategies based on appealing to such people’s reliance on those algorithms should also re-evaluate.
E.g., why did folk write AI 2027? Did they honestly think the timeline was that short? Were they trying to convey a picture that would scare people with something on a short enough timeline that they could feel it?
If the latter, we might be doing humanity a disservice, both by exhausting people from something akin to adrenal fatigue, and also as a result of crying wolf.
Yes, I honestly thought the timeline was that short. I now think it’s 50% by end of 2028; over the last year my timelines have lengthened by about a year.
Well extrapolating that it sounds like things are fine. :P
It has indeed been really nice, psychologically, to have timelines that are lengthening again. 2020 to 2024 that was not the case.
You wrote AI 2027 in April… what changed in such a short amount of time?
If your timelines lengthened over the last year, do you think writing AI 2027 was an honest reflection of your opinions at the time?
The draft of AI 2027 was done in December, then we had months of editing and rewriting in response to feedback. For more on what changed, see various comments I made online such as this one: https://www.lesswrong.com/posts/cxuzALcmucCndYv4a/daniel-kokotajlo-s-shortform?commentId=dq6bpAHeu5Cbbiuyd
We said right on the front page of AI 2027 in a footnote that our actual AGI timelines medians were somewhat longer than 2027:
I also mentioned my slightly longer timelines in various interviews about it, including the first one with Kevin Roose.
OpenAI researcher Jason Wei recently stated that there will be many bottlenecks to recursive self improvement (experiments, data), thoughts?
https://x.com/_jasonwei/status/1939762496757539297z
He makes some obvious points everyone already knows about bottlenecks etc. but then doesn’t explain why all that adds up to a decade or more, instead of of a year, or a month, or a century. In our takeoff speeds forecast we try to give a quantitative estimate that takes into account all the bottlenecks etc.
Isn’t it more like “I think there’s a 10% chance of transformative AI by 2027, and that is like 100x higher than what it looks like most people think, so people really need to think thru that timeline”?
Like, I generally put my median year at 2030-2032; if we make it to 2028, the situation will still feel like “oh jeez we probably only have a few years left”, unless we made it to 2028 thru a mechanism that clearly blocks transformative AI showing up in 2032. (Like, a lot is hinging on what “feels basically like today” means.)
I think Daniel also just has shorter timelines than most (which is correlated for wanting to more urgently communicate that knowledge).
That might be. It sounds really plausible. I don’t know why they wrote it!
But all the same: I don’t think most people know what 10% likelihood of a severe outcome is like or how to think about it sensibly. My read is that the vast majority of people need to treat 10% likelihood of doom as either “It’s not going to happen” (because 10% is small) or “It’s guaranteed to happen” (because it’s a serious outcome if it does happen, and it’s plausible). So, amplifying the public awareness of this possibility seems more to me like moving awareness of the scenario from “Nothing existential is going to happen” to “This specific thing is the default thing to expect.”
So I expect that unless something is done to… I don’t know, magically educate the population on statistical thinking, or propagate a public message that it’s roughly right but its timeline is wrong? then the net effect will be that either (a) AI 2027 will have been collectively forgotten by 2028 in roughly the same way that, say, Trudeau’s use of the Emergencies Act has been forgotten; or (b) the predictions failing to pan out will be used as reason to dismiss other AI doom predictions that are apparently considered more likely.
The main benefit I see is if some key folk are made to think about AI doom scenarios in general as a result of AI 2027, and start to work out how to deal with other scenarios.
But I don’t know. That’s been part of this community’s strategy for over two decades. Get key people thinking about AI risk. And I’m not too keen on the results I’ve seen from that strategy so far.
it does imply that, but i’m somewhat loathe to mention this at all, because i think the predictive quality you get from one question to another varies astronomically, and this is not something the casual reader will be able to glean