AI 2027 was less crude in its use of symmetric weapons (which can often itself be a good symmetric weapon when the goal is to influence elites)
AI 2027 made lots of asymmetric choices (but so did titotal)
AI 2027 is doing better than “piece[s] of media” (but that bar is so incredibly low)
I disagree that titotal’s critique is far away from AI 2027 on the relevant spectrum. For example, titotal’s critique was posted on the EA Forum / LessWrong, and focused on technical disagreements, rather than going through a huge amplification / social media push, and focusing on storytelling.
(I’d agree that AI 2027 put in more effort / are more obviously “trying” relative to titotal, so they’re far away as judged by intent, but I mostly care about outcomes rather than intent.)
You might say that obviously AI 2027 needed to do the amplification / social media push + storytelling in order to achieve its goals of influencing the discourse, and I would agree with you. But “influence the discourse” is ultimately going to be about status and prestige (given how discourse works in practice). If you’re taking a stance against goals around status and prestige that trade off against epistemic commons, I think you also need to take a stance against AI 2027. (To be clear, I don’t take that stance! I’m just arguing for consistency.)
Before AI 2027 was posted with a big amplification / media push, it underwent as far as I can tell the single most intense set of review and feedback requests of any big writing project I’ve seen so far. I don’t know whether it was literally posted on LessWrong, but I’ve seen comments from many many dozens if not hundreds of people over the many dozens of revisions that the scenario underwent.
Like, I am quite into public discourse being better than private Google Doc systems, but AI 2027 was so widely circulated pre-publication in Google Doc format, with lots of focus on technical disagreements, that this seems easily much superior to what is going on with this post.
I don’t see how this is responding to anything I’ve said? What in my comment are you disagreeing with or adding color to?
Again, my position is not “AI 2027 did something bad”. My position is “stop critiquing people for having goals around status and prestige rather than epistemics, or at least do so consistently”.
(Incidentally, I suspect bio anchors did better on the axis of getting good reviews / feedback, but that isn’t particularly central to anything I’m claiming.)
For example, titotal’s critique was posted on the EA Forum / LessWrong, and focused on technical disagreement
And I was saying that this is also true for the early drafts of AI 2027. Only after a long discussion of the technical disagreements did it go on to a huge amplification thing. This seems directly relevant to that section.
I am responding to the part about consistent standards. I don’t really understand what you believe here, clearly you care a lot about people not using lots of rhetorical tricks and adversarial persuasion tactics all the time, and we’ve talked about that in the past, so I am just straightforwardly arguing that on those dimensions titotal’s post was much worse compared to AI 2027.
We don’t need to come to agreement on this part, it does seem kind of hard to evaluate. But in as much as your top level comment is arguing some kind of asymmetric standard is being applied, that just seems super wrong to me. I don’t know where I would put the line of encourage/discourage, but I don’t see any inconsistency in being unhappy with what titotal is doing and happy about what AI 2027 is doing.
I don’t see any inconsistency in being unhappy with what titotal is doing and happy about what AI 2027 is doing.
I agree with this. I was responding pretty specifically to Zvi’s critique in particular, which is focusing on things like the use of the word “bad” and the notion that there could be a goal to lower the status and prestige of AI 2027. If instead the critique was about e.g. norms of intellectual discourse I’d be on board.
That said I don’t feel like your defense feels all that strong to me? I’m happy to take your word for it that there was lots of review of AI 2027, but my understanding is that titotal also engaged quite a lot with the authors of AI 2027 before publishing the post? (I definitely expect it was much lower engagement / review in an absolute sense, but everything about it is going to be much lower in an absolute sense, since it is not as big a project.)
If I had to guess at the difference between us, it would be that I primarily see emotionally gripping storytelling as a symmetric weapon to be regarded with suspicion by default, whereas you primarily view it as an important and valuable way to get people to really engage with a topic. (Though admittedly on this view I can’t quite see why you’d object to describing a model as “bad”, since that also seems like a way to get people to better engage with a topic.) Or possibly it’s more salient to me how the storytelling in the finished AI 2027 product comes across since I wasn’t involved in its creation, whereas to you the research and analysis is more salient.
Anyway it doesn’t seem super worth digging to the bottom of this, seems reasonable to leave it here (though I would be interested in any reactions you have if you felt like writing them).
EDIT: Actually looking at the other comments here I think it’s plausible that a lot of the difference is in creators thinking the point of AI 2027 was the scenario whereas the public reception was much more about timelines. I feel like it was very predictable that public reception would focus a lot on the timeline, but perhaps this would have been less clear in advance. Though looking at Scott’s post, the timeline is really quite central to the presentation, so I don’t feel like this can really be a surprise.
To clarify, by AI 2027 do you include the timeline model? If so, I’d be interested to know if the reviews caught and/or discussed any of the primary criticisms that titotal has brought up here, particularly the “model is insensitive to starting conditions” bits.
(I recognize I’m butting into a conversation so feel absolutely free to ignore this.)
I don’t know! I would have to look through all the Google Docs comments and like 10 different versions.
In general though, I seem to have a very different relationship to all the supplements than some other people reading AI 2027, and I kind of wonder whether it would just be better to not have the supplements at all.
From my perspective the key thing is the scenario and the associated expandable boxes and explanations. And then I view most of the supplements as kind of helpful essays for trying to understand and explain some of the intuitions that generated the scenario, but the process for the whole thing is very much not “there is an externally validatable scientific model that was built, then that model was used to generate a scenario”. The key engagement I am interested in is people arguing against the scenario, not doing some kind of weird “oh, but your models aren’t externally validatable and actually in order to say anything about the future of AI your models need to be conceptually perfect”.
I really don’t think the graph-fitting described in the timelines supplement was that causally upstream of the beliefs of almost any of the people involved, and I kind of view it more as a single individual sanity-check on whether the basic premise of the scenario checks out. When people try to forecast things as complicated as this, they don’t create nice formal models, they have a model in their head that handles a huge number of edge-cases, and is trying to be consistent with much much more things than the formal model could ever represent. Ideally the research supplements would say something like that at the top, though it’s plausible that some of the AI Futures Project team relate to their epistemic process differently (though if they do, I think they are just kind of confused).
I don’t even think the Timelines Forecast supplement says anything like “this timelines forecast is the basis of the timeline of the mainline scenario”. It’s just like, a semi-random methodology for forecasting a transformative AI timeline that vaguely informed the main scenario. Conceptually, it feels similar to just doing a random fermi estimate in the middle of a blog post to sanity-check that the thing I am thinking about isn’t completely crazy.
I think it’s still good to engage with it on its own terms, and think there is value in that, but it’s really not what seems remotely most productive to me when thinking about all of AI 2027.
In general though, I seem to have a very different relationship to all the supplements than some other people reading AI 2027, and I kind of wonder whether it would just be better to not have the supplements at all.
I think this is likely to be true, yes. FWIW, most of the non-AI-researcher people I have talked to about AI 2027 are extremely surprised to hear that the story was not generated in any meaningful sense by the model supplements. It may not explicitly say this—I agree that if folks parse the language on the website very carefully they can plausibly come to that conclusion—but it seems like a pretty crucial thing to be explicit about, just so folks know how to interpret things.
Thanks for the correction! I’m guessing you don’t want to, but I would appreciate an elaboration on your part; is @habryka’s description below inaccurate, or did I misinterpret it?
It’s just like, a semi-random methodology for forecasting a transformative AI timeline that vaguely informed the main scenario. Conceptually, it feels similar to just doing a random fermi estimate in the middle of a blog post to sanity-check that the thing I am thinking about isn’t completely crazy.
OK I just had a chat with Eli to try to trace the causal history as best we can remember. At a high level, we were working on the scenario and the supplementary research in parallel, and went back and forth making edits to both for months, and our views evolved somewhat over the course of that time.
Timelines: We initially set AGI in 2027 based on my AGI median, which was informed based on a combination of arguments regarding gains from scaling up agency training, as well as a very crude, handwavy version of what later became the benchmarks and gaps model. Later timelines modeling (the stuff that actually went on the website) along with some additional evidence that came out, pushed my median back to 2028. We denoted this in a footnote on the site (footnote #1 in fact) and I posted a shortform about it (plus a tweet or two). tl;dr is that 2027 was my mode, not my median, after the update. We considered rewriting the scenario to happen about one year later, due to this, but decided against since that would have taken a lot of extra time and didn’t really change any of the implications. If the timelines model had given very different results which changed our views against 2027 being plausible, then we would have re-written the scenario. I also mentioned this to Kevin Roose in my interview with him (my somewhat later timelines, the difference between median and mode). I didn’t expect people to make such a big deal of this.
Takeoff: The takeoff model for our first scenario, the “practice scenario” which we basically scrapped, was basically a simplified version of Davidson’s takeoff speeds model. (takeoffspeeds.com) Later takeoff modeling informed which milestones to focus on the scenario (superhuman coder, superhuman AI researcher, etc.) and what AI R&D progress multiplier they should have. Our memory isn’t clear on to what extent they also resulted in changes to the speed of the milestone progression. We think an early crude version of our takeoff model might have resulted in significant changes, but we aren’t sure. We were also working on our takeoff model up until the last minute, and similar to the timelines model mostly used it as a sanity check.
Compute: The first version of this was done in early 2024, and the result of it and future versions were directly imported into the scenario.
AI Goals: Early versions of this supplement were basically responsible for our decision to go with instrumentally convergent goals as the AIs’ ultimate goals in the scenario.
Security: This one was in between a sanity check and directly feeding into the scenario. It didn’t result in large changes but confirmed the likelihood of the weight theft and informed various decisions about e.g. cyberattacks.
So.… Habryka’s description is somewhat accurate, certainly more accurate than your description (“no meaningful sense”). But I think it still undersells it. That said, it’s definitely not the case that we wrote all the supplements first and then wrote the scenario based on the outputs of those calculations; instead, we wrote them in parallel, had various shitty early versions, etc.
If you want to know more about the evidence & modelling that shaped our views in early 2024 when we were starting the project, I could try to compile a list. I’ve already mentioned takeoffspeeds.com for example. There’s lots of other writing I’ve put on LessWrong on the subject as well.
My guess is there is no confusion about this, but to be clear, I didn’t intend to speak on behalf of the AI 2027 team. Indeed, it’s plausible to me they disagree with it, though my honest belief in that case is that they are confused about the sources of their own beliefs, not that my statement is wrong. I.e. I said:
Ideally the research supplements would say something like that at the top, though it’s plausible that some of the AI Futures Project team relate to their epistemic process differently (though if they do, I think they are just kind of confused).
The timelines model didn’t get nearly as many reviews as the scenario. We shared the timelines writeup with all of the people who we shared the later drafts of the scenario with, but I think almost none of them looked at the timelines writeup.
We also asked a few people to specifically review the timelines forecasts, most notably a few FutureSearch forecasters who we then added as a final author. However, we mainly wanted them to estimate the parameter values and didn’t specifically ask them for feedback on the underlying modeling choices (though they did form some opinions, for example they liked benchmark and gaps much more than time horizon extension; also btw the superexponential plays a much smaller role in benchmarks and gaps). No one brought up the criticisms that titotal did.
In general the timelines model certainly got way less effort than the scenario, probably about 5% as much effort. Our main focus was the scenario as we think that it’s a much higher value add.
I’m been pretty surprised at to how much quality-weighted criticisms have focused on the timelines model relative to the scenario, and wish that it was more tilted toward the scenario (and also toward the takeoff model, which IMO is more important than the timelines model but has gotten much less attention). To be clear I’m still very glad that these critiques exist if the alternative is that they didn’t exist and nothing replaced them.
I suspect part of the reasons for the quality-weighted criticism of the timelines rather than the scenario:
If it is the case that you put far less effort into the timelines model than the scenario, then the timelines model is probably just worse—some of the more obvious mistakes that titotal points out probably don’t have analogies in your scenario, so its just easier to criticise the timelines model, as there is more to criticise there
In many ways, the timelines model is pretty key to the headline claim of your scenario. The other parts (scenario and takeoff) are useful, high quality contributions but in many ways are less meaningfully novel than the very aggressive timelines. Your takeoff model, for example, is well within the range of speeds considered in the community for years—indeed, it is far slower than a Yudkowskian takeoff for example. This isn’t to degrade it—the level of detail in the scenario is commendable and the quality in that respect is genuinely novel. But in terms of what the media coverage, and impact of the work, its the timelines that I suspect are the most significant
Things I agree with:
AI 2027 was less crude in its use of symmetric weapons (which can often itself be a good symmetric weapon when the goal is to influence elites)
AI 2027 made lots of asymmetric choices (but so did titotal)
AI 2027 is doing better than “piece[s] of media” (but that bar is so incredibly low)
I disagree that titotal’s critique is far away from AI 2027 on the relevant spectrum. For example, titotal’s critique was posted on the EA Forum / LessWrong, and focused on technical disagreements, rather than going through a huge amplification / social media push, and focusing on storytelling.
(I’d agree that AI 2027 put in more effort / are more obviously “trying” relative to titotal, so they’re far away as judged by intent, but I mostly care about outcomes rather than intent.)
You might say that obviously AI 2027 needed to do the amplification / social media push + storytelling in order to achieve its goals of influencing the discourse, and I would agree with you. But “influence the discourse” is ultimately going to be about status and prestige (given how discourse works in practice). If you’re taking a stance against goals around status and prestige that trade off against epistemic commons, I think you also need to take a stance against AI 2027. (To be clear, I don’t take that stance! I’m just arguing for consistency.)
Before AI 2027 was posted with a big amplification / media push, it underwent as far as I can tell the single most intense set of review and feedback requests of any big writing project I’ve seen so far. I don’t know whether it was literally posted on LessWrong, but I’ve seen comments from many many dozens if not hundreds of people over the many dozens of revisions that the scenario underwent.
Like, I am quite into public discourse being better than private Google Doc systems, but AI 2027 was so widely circulated pre-publication in Google Doc format, with lots of focus on technical disagreements, that this seems easily much superior to what is going on with this post.
I don’t see how this is responding to anything I’ve said? What in my comment are you disagreeing with or adding color to?
Again, my position is not “AI 2027 did something bad”. My position is “stop critiquing people for having goals around status and prestige rather than epistemics, or at least do so consistently”.
(Incidentally, I suspect bio anchors did better on the axis of getting good reviews / feedback, but that isn’t particularly central to anything I’m claiming.)
I was responding to this part:
And I was saying that this is also true for the early drafts of AI 2027. Only after a long discussion of the technical disagreements did it go on to a huge amplification thing. This seems directly relevant to that section.
I am responding to the part about consistent standards. I don’t really understand what you believe here, clearly you care a lot about people not using lots of rhetorical tricks and adversarial persuasion tactics all the time, and we’ve talked about that in the past, so I am just straightforwardly arguing that on those dimensions titotal’s post was much worse compared to AI 2027.
We don’t need to come to agreement on this part, it does seem kind of hard to evaluate. But in as much as your top level comment is arguing some kind of asymmetric standard is being applied, that just seems super wrong to me. I don’t know where I would put the line of encourage/discourage, but I don’t see any inconsistency in being unhappy with what titotal is doing and happy about what AI 2027 is doing.
I agree with this. I was responding pretty specifically to Zvi’s critique in particular, which is focusing on things like the use of the word “bad” and the notion that there could be a goal to lower the status and prestige of AI 2027. If instead the critique was about e.g. norms of intellectual discourse I’d be on board.
That said I don’t feel like your defense feels all that strong to me? I’m happy to take your word for it that there was lots of review of AI 2027, but my understanding is that titotal also engaged quite a lot with the authors of AI 2027 before publishing the post? (I definitely expect it was much lower engagement / review in an absolute sense, but everything about it is going to be much lower in an absolute sense, since it is not as big a project.)
If I had to guess at the difference between us, it would be that I primarily see emotionally gripping storytelling as a symmetric weapon to be regarded with suspicion by default, whereas you primarily view it as an important and valuable way to get people to really engage with a topic. (Though admittedly on this view I can’t quite see why you’d object to describing a model as “bad”, since that also seems like a way to get people to better engage with a topic.) Or possibly it’s more salient to me how the storytelling in the finished AI 2027 product comes across since I wasn’t involved in its creation, whereas to you the research and analysis is more salient.
Anyway it doesn’t seem super worth digging to the bottom of this, seems reasonable to leave it here (though I would be interested in any reactions you have if you felt like writing them).
EDIT: Actually looking at the other comments here I think it’s plausible that a lot of the difference is in creators thinking the point of AI 2027 was the scenario whereas the public reception was much more about timelines. I feel like it was very predictable that public reception would focus a lot on the timeline, but perhaps this would have been less clear in advance. Though looking at Scott’s post, the timeline is really quite central to the presentation, so I don’t feel like this can really be a surprise.
To clarify, by AI 2027 do you include the timeline model? If so, I’d be interested to know if the reviews caught and/or discussed any of the primary criticisms that titotal has brought up here, particularly the “model is insensitive to starting conditions” bits.
(I recognize I’m butting into a conversation so feel absolutely free to ignore this.)
I don’t know! I would have to look through all the Google Docs comments and like 10 different versions.
In general though, I seem to have a very different relationship to all the supplements than some other people reading AI 2027, and I kind of wonder whether it would just be better to not have the supplements at all.
From my perspective the key thing is the scenario and the associated expandable boxes and explanations. And then I view most of the supplements as kind of helpful essays for trying to understand and explain some of the intuitions that generated the scenario, but the process for the whole thing is very much not “there is an externally validatable scientific model that was built, then that model was used to generate a scenario”. The key engagement I am interested in is people arguing against the scenario, not doing some kind of weird “oh, but your models aren’t externally validatable and actually in order to say anything about the future of AI your models need to be conceptually perfect”.
I really don’t think the graph-fitting described in the timelines supplement was that causally upstream of the beliefs of almost any of the people involved, and I kind of view it more as a single individual sanity-check on whether the basic premise of the scenario checks out. When people try to forecast things as complicated as this, they don’t create nice formal models, they have a model in their head that handles a huge number of edge-cases, and is trying to be consistent with much much more things than the formal model could ever represent. Ideally the research supplements would say something like that at the top, though it’s plausible that some of the AI Futures Project team relate to their epistemic process differently (though if they do, I think they are just kind of confused).
I don’t even think the Timelines Forecast supplement says anything like “this timelines forecast is the basis of the timeline of the mainline scenario”. It’s just like, a semi-random methodology for forecasting a transformative AI timeline that vaguely informed the main scenario. Conceptually, it feels similar to just doing a random fermi estimate in the middle of a blog post to sanity-check that the thing I am thinking about isn’t completely crazy.
I think it’s still good to engage with it on its own terms, and think there is value in that, but it’s really not what seems remotely most productive to me when thinking about all of AI 2027.
I think this is likely to be true, yes. FWIW, most of the non-AI-researcher people I have talked to about AI 2027 are extremely surprised to hear that the story was not generated in any meaningful sense by the model supplements. It may not explicitly say this—I agree that if folks parse the language on the website very carefully they can plausibly come to that conclusion—but it seems like a pretty crucial thing to be explicit about, just so folks know how to interpret things.
It is false that the story was not generated in any meaningful sense by the model supplements.
Thanks for the correction! I’m guessing you don’t want to, but I would appreciate an elaboration on your part; is @habryka’s description below inaccurate, or did I misinterpret it?
OK I just had a chat with Eli to try to trace the causal history as best we can remember. At a high level, we were working on the scenario and the supplementary research in parallel, and went back and forth making edits to both for months, and our views evolved somewhat over the course of that time.
Timelines: We initially set AGI in 2027 based on my AGI median, which was informed based on a combination of arguments regarding gains from scaling up agency training, as well as a very crude, handwavy version of what later became the benchmarks and gaps model. Later timelines modeling (the stuff that actually went on the website) along with some additional evidence that came out, pushed my median back to 2028. We denoted this in a footnote on the site (footnote #1 in fact) and I posted a shortform about it (plus a tweet or two). tl;dr is that 2027 was my mode, not my median, after the update. We considered rewriting the scenario to happen about one year later, due to this, but decided against since that would have taken a lot of extra time and didn’t really change any of the implications. If the timelines model had given very different results which changed our views against 2027 being plausible, then we would have re-written the scenario. I also mentioned this to Kevin Roose in my interview with him (my somewhat later timelines, the difference between median and mode). I didn’t expect people to make such a big deal of this.
Takeoff: The takeoff model for our first scenario, the “practice scenario” which we basically scrapped, was basically a simplified version of Davidson’s takeoff speeds model. (takeoffspeeds.com) Later takeoff modeling informed which milestones to focus on the scenario (superhuman coder, superhuman AI researcher, etc.) and what AI R&D progress multiplier they should have. Our memory isn’t clear on to what extent they also resulted in changes to the speed of the milestone progression. We think an early crude version of our takeoff model might have resulted in significant changes, but we aren’t sure. We were also working on our takeoff model up until the last minute, and similar to the timelines model mostly used it as a sanity check.
Compute: The first version of this was done in early 2024, and the result of it and future versions were directly imported into the scenario.
AI Goals: Early versions of this supplement were basically responsible for our decision to go with instrumentally convergent goals as the AIs’ ultimate goals in the scenario.
Security: This one was in between a sanity check and directly feeding into the scenario. It didn’t result in large changes but confirmed the likelihood of the weight theft and informed various decisions about e.g. cyberattacks.
So.… Habryka’s description is somewhat accurate, certainly more accurate than your description (“no meaningful sense”). But I think it still undersells it. That said, it’s definitely not the case that we wrote all the supplements first and then wrote the scenario based on the outputs of those calculations; instead, we wrote them in parallel, had various shitty early versions, etc.
If you want to know more about the evidence & modelling that shaped our views in early 2024 when we were starting the project, I could try to compile a list. I’ve already mentioned takeoffspeeds.com for example. There’s lots of other writing I’ve put on LessWrong on the subject as well.
Does this help?
My guess is there is no confusion about this, but to be clear, I didn’t intend to speak on behalf of the AI 2027 team. Indeed, it’s plausible to me they disagree with it, though my honest belief in that case is that they are confused about the sources of their own beliefs, not that my statement is wrong. I.e. I said:
The timelines model didn’t get nearly as many reviews as the scenario. We shared the timelines writeup with all of the people who we shared the later drafts of the scenario with, but I think almost none of them looked at the timelines writeup.
We also asked a few people to specifically review the timelines forecasts, most notably a few FutureSearch forecasters who we then added as a final author. However, we mainly wanted them to estimate the parameter values and didn’t specifically ask them for feedback on the underlying modeling choices (though they did form some opinions, for example they liked benchmark and gaps much more than time horizon extension; also btw the superexponential plays a much smaller role in benchmarks and gaps). No one brought up the criticisms that titotal did.
In general the timelines model certainly got way less effort than the scenario, probably about 5% as much effort. Our main focus was the scenario as we think that it’s a much higher value add.
I’m been pretty surprised at to how much quality-weighted criticisms have focused on the timelines model relative to the scenario, and wish that it was more tilted toward the scenario (and also toward the takeoff model, which IMO is more important than the timelines model but has gotten much less attention). To be clear I’m still very glad that these critiques exist if the alternative is that they didn’t exist and nothing replaced them.
I suspect part of the reasons for the quality-weighted criticism of the timelines rather than the scenario:
If it is the case that you put far less effort into the timelines model than the scenario, then the timelines model is probably just worse—some of the more obvious mistakes that titotal points out probably don’t have analogies in your scenario, so its just easier to criticise the timelines model, as there is more to criticise there
In many ways, the timelines model is pretty key to the headline claim of your scenario. The other parts (scenario and takeoff) are useful, high quality contributions but in many ways are less meaningfully novel than the very aggressive timelines. Your takeoff model, for example, is well within the range of speeds considered in the community for years—indeed, it is far slower than a Yudkowskian takeoff for example. This isn’t to degrade it—the level of detail in the scenario is commendable and the quality in that respect is genuinely novel. But in terms of what the media coverage, and impact of the work, its the timelines that I suspect are the most significant