I don’t know! I would have to look through all the Google Docs comments and like 10 different versions.
In general though, I seem to have a very different relationship to all the supplements than some other people reading AI 2027, and I kind of wonder whether it would just be better to not have the supplements at all.
From my perspective the key thing is the scenario and the associated expandable boxes and explanations. And then I view most of the supplements as kind of helpful essays for trying to understand and explain some of the intuitions that generated the scenario, but the process for the whole thing is very much not “there is an externally validatable scientific model that was built, then that model was used to generate a scenario”. The key engagement I am interested in is people arguing against the scenario, not doing some kind of weird “oh, but your models aren’t externally validatable and actually in order to say anything about the future of AI your models need to be conceptually perfect”.
I really don’t think the graph-fitting described in the timelines supplement was that causally upstream of the beliefs of almost any of the people involved, and I kind of view it more as a single individual sanity-check on whether the basic premise of the scenario checks out. When people try to forecast things as complicated as this, they don’t create nice formal models, they have a model in their head that handles a huge number of edge-cases, and is trying to be consistent with much much more things than the formal model could ever represent. Ideally the research supplements would say something like that at the top, though it’s plausible that some of the AI Futures Project team relate to their epistemic process differently (though if they do, I think they are just kind of confused).
I don’t even think the Timelines Forecast supplement says anything like “this timelines forecast is the basis of the timeline of the mainline scenario”. It’s just like, a semi-random methodology for forecasting a transformative AI timeline that vaguely informed the main scenario. Conceptually, it feels similar to just doing a random fermi estimate in the middle of a blog post to sanity-check that the thing I am thinking about isn’t completely crazy.
I think it’s still good to engage with it on its own terms, and think there is value in that, but it’s really not what seems remotely most productive to me when thinking about all of AI 2027.
In general though, I seem to have a very different relationship to all the supplements than some other people reading AI 2027, and I kind of wonder whether it would just be better to not have the supplements at all.
I think this is likely to be true, yes. FWIW, most of the non-AI-researcher people I have talked to about AI 2027 are extremely surprised to hear that the story was not generated in any meaningful sense by the model supplements. It may not explicitly say this—I agree that if folks parse the language on the website very carefully they can plausibly come to that conclusion—but it seems like a pretty crucial thing to be explicit about, just so folks know how to interpret things.
Thanks for the correction! I’m guessing you don’t want to, but I would appreciate an elaboration on your part; is @habryka’s description below inaccurate, or did I misinterpret it?
It’s just like, a semi-random methodology for forecasting a transformative AI timeline that vaguely informed the main scenario. Conceptually, it feels similar to just doing a random fermi estimate in the middle of a blog post to sanity-check that the thing I am thinking about isn’t completely crazy.
OK I just had a chat with Eli to try to trace the causal history as best we can remember. At a high level, we were working on the scenario and the supplementary research in parallel, and went back and forth making edits to both for months, and our views evolved somewhat over the course of that time.
Timelines: We initially set AGI in 2027 based on my AGI median, which was informed based on a combination of arguments regarding gains from scaling up agency training, as well as a very crude, handwavy version of what later became the benchmarks and gaps model. Later timelines modeling (the stuff that actually went on the website) along with some additional evidence that came out, pushed my median back to 2028. We denoted this in a footnote on the site (footnote #1 in fact) and I posted a shortform about it (plus a tweet or two). tl;dr is that 2027 was my mode, not my median, after the update. We considered rewriting the scenario to happen about one year later, due to this, but decided against since that would have taken a lot of extra time and didn’t really change any of the implications. If the timelines model had given very different results which changed our views against 2027 being plausible, then we would have re-written the scenario. I also mentioned this to Kevin Roose in my interview with him (my somewhat later timelines, the difference between median and mode). I didn’t expect people to make such a big deal of this.
Takeoff: The takeoff model for our first scenario, the “practice scenario” which we basically scrapped, was basically a simplified version of Davidson’s takeoff speeds model. (takeoffspeeds.com) Later takeoff modeling informed which milestones to focus on the scenario (superhuman coder, superhuman AI researcher, etc.) and what AI R&D progress multiplier they should have. Our memory isn’t clear on to what extent they also resulted in changes to the speed of the milestone progression. We think an early crude version of our takeoff model might have resulted in significant changes, but we aren’t sure. We were also working on our takeoff model up until the last minute, and similar to the timelines model mostly used it as a sanity check.
Compute: The first version of this was done in early 2024, and the result of it and future versions were directly imported into the scenario.
AI Goals: Early versions of this supplement were basically responsible for our decision to go with instrumentally convergent goals as the AIs’ ultimate goals in the scenario.
Security: This one was in between a sanity check and directly feeding into the scenario. It didn’t result in large changes but confirmed the likelihood of the weight theft and informed various decisions about e.g. cyberattacks.
So.… Habryka’s description is somewhat accurate, certainly more accurate than your description (“no meaningful sense”). But I think it still undersells it. That said, it’s definitely not the case that we wrote all the supplements first and then wrote the scenario based on the outputs of those calculations; instead, we wrote them in parallel, had various shitty early versions, etc.
If you want to know more about the evidence & modelling that shaped our views in early 2024 when we were starting the project, I could try to compile a list. I’ve already mentioned takeoffspeeds.com for example. There’s lots of other writing I’ve put on LessWrong on the subject as well.
My guess is there is no confusion about this, but to be clear, I didn’t intend to speak on behalf of the AI 2027 team. Indeed, it’s plausible to me they disagree with it, though my honest belief in that case is that they are confused about the sources of their own beliefs, not that my statement is wrong. I.e. I said:
Ideally the research supplements would say something like that at the top, though it’s plausible that some of the AI Futures Project team relate to their epistemic process differently (though if they do, I think they are just kind of confused).
I don’t know! I would have to look through all the Google Docs comments and like 10 different versions.
In general though, I seem to have a very different relationship to all the supplements than some other people reading AI 2027, and I kind of wonder whether it would just be better to not have the supplements at all.
From my perspective the key thing is the scenario and the associated expandable boxes and explanations. And then I view most of the supplements as kind of helpful essays for trying to understand and explain some of the intuitions that generated the scenario, but the process for the whole thing is very much not “there is an externally validatable scientific model that was built, then that model was used to generate a scenario”. The key engagement I am interested in is people arguing against the scenario, not doing some kind of weird “oh, but your models aren’t externally validatable and actually in order to say anything about the future of AI your models need to be conceptually perfect”.
I really don’t think the graph-fitting described in the timelines supplement was that causally upstream of the beliefs of almost any of the people involved, and I kind of view it more as a single individual sanity-check on whether the basic premise of the scenario checks out. When people try to forecast things as complicated as this, they don’t create nice formal models, they have a model in their head that handles a huge number of edge-cases, and is trying to be consistent with much much more things than the formal model could ever represent. Ideally the research supplements would say something like that at the top, though it’s plausible that some of the AI Futures Project team relate to their epistemic process differently (though if they do, I think they are just kind of confused).
I don’t even think the Timelines Forecast supplement says anything like “this timelines forecast is the basis of the timeline of the mainline scenario”. It’s just like, a semi-random methodology for forecasting a transformative AI timeline that vaguely informed the main scenario. Conceptually, it feels similar to just doing a random fermi estimate in the middle of a blog post to sanity-check that the thing I am thinking about isn’t completely crazy.
I think it’s still good to engage with it on its own terms, and think there is value in that, but it’s really not what seems remotely most productive to me when thinking about all of AI 2027.
I think this is likely to be true, yes. FWIW, most of the non-AI-researcher people I have talked to about AI 2027 are extremely surprised to hear that the story was not generated in any meaningful sense by the model supplements. It may not explicitly say this—I agree that if folks parse the language on the website very carefully they can plausibly come to that conclusion—but it seems like a pretty crucial thing to be explicit about, just so folks know how to interpret things.
It is false that the story was not generated in any meaningful sense by the model supplements.
Thanks for the correction! I’m guessing you don’t want to, but I would appreciate an elaboration on your part; is @habryka’s description below inaccurate, or did I misinterpret it?
OK I just had a chat with Eli to try to trace the causal history as best we can remember. At a high level, we were working on the scenario and the supplementary research in parallel, and went back and forth making edits to both for months, and our views evolved somewhat over the course of that time.
Timelines: We initially set AGI in 2027 based on my AGI median, which was informed based on a combination of arguments regarding gains from scaling up agency training, as well as a very crude, handwavy version of what later became the benchmarks and gaps model. Later timelines modeling (the stuff that actually went on the website) along with some additional evidence that came out, pushed my median back to 2028. We denoted this in a footnote on the site (footnote #1 in fact) and I posted a shortform about it (plus a tweet or two). tl;dr is that 2027 was my mode, not my median, after the update. We considered rewriting the scenario to happen about one year later, due to this, but decided against since that would have taken a lot of extra time and didn’t really change any of the implications. If the timelines model had given very different results which changed our views against 2027 being plausible, then we would have re-written the scenario. I also mentioned this to Kevin Roose in my interview with him (my somewhat later timelines, the difference between median and mode). I didn’t expect people to make such a big deal of this.
Takeoff: The takeoff model for our first scenario, the “practice scenario” which we basically scrapped, was basically a simplified version of Davidson’s takeoff speeds model. (takeoffspeeds.com) Later takeoff modeling informed which milestones to focus on the scenario (superhuman coder, superhuman AI researcher, etc.) and what AI R&D progress multiplier they should have. Our memory isn’t clear on to what extent they also resulted in changes to the speed of the milestone progression. We think an early crude version of our takeoff model might have resulted in significant changes, but we aren’t sure. We were also working on our takeoff model up until the last minute, and similar to the timelines model mostly used it as a sanity check.
Compute: The first version of this was done in early 2024, and the result of it and future versions were directly imported into the scenario.
AI Goals: Early versions of this supplement were basically responsible for our decision to go with instrumentally convergent goals as the AIs’ ultimate goals in the scenario.
Security: This one was in between a sanity check and directly feeding into the scenario. It didn’t result in large changes but confirmed the likelihood of the weight theft and informed various decisions about e.g. cyberattacks.
So.… Habryka’s description is somewhat accurate, certainly more accurate than your description (“no meaningful sense”). But I think it still undersells it. That said, it’s definitely not the case that we wrote all the supplements first and then wrote the scenario based on the outputs of those calculations; instead, we wrote them in parallel, had various shitty early versions, etc.
If you want to know more about the evidence & modelling that shaped our views in early 2024 when we were starting the project, I could try to compile a list. I’ve already mentioned takeoffspeeds.com for example. There’s lots of other writing I’ve put on LessWrong on the subject as well.
Does this help?
My guess is there is no confusion about this, but to be clear, I didn’t intend to speak on behalf of the AI 2027 team. Indeed, it’s plausible to me they disagree with it, though my honest belief in that case is that they are confused about the sources of their own beliefs, not that my statement is wrong. I.e. I said: