I mean, yes, if the goal of the post was to lower the status and prestige of AI 2027 and to do so through people reading the title and updating in that way, rather than to offer a helpful critique, then it is true that the title was the best local way to achieve that objective, epistemic commons be damned. I would hope for a different goal?
Come on, this is such an isolated demand for rigor. AI 2027 clearly had the goal of raising the status and prestige of belief in AI risk and short timelines. They employed tons of symmetric weapons in the pursuit of this goal. I’m maybe 95% sure you didn’t substantially critique them for that.[1] Why start with this much less viral post?
(To be clear I’m not against the use of symmetric weapons. I’m against censure of the side you disagree with that masquerades as being impartial, whether or not that was deliberate.)
I don’t think AI 2027 did anything even close to as crude as calling the thing you are arguing against just “bad” in your title.
Indeed, I think overall AI 2027 is really doing remarkably well at being asymmetric in really a huge number of its choices (I am of course biased as having been involved in many of those choices, but I currently would say that AI 2027 is close to the very top at the intersection of “accessible” and “trying to make itself only succeed and compelling if indeed its claims are true” as I think any piece of media out there).
(I don’t have a super considered take on whether I think the title is a fine title, but it seems clear to me that there is a spectrum of trying to make your thing asymmetrically successful and that titotal’s critique is far away from AI 2027 on that spectrum, in the direction of symmetry instead of asymmetry)
AI 2027 was less crude in its use of symmetric weapons (which can often itself be a good symmetric weapon when the goal is to influence elites)
AI 2027 made lots of asymmetric choices (but so did titotal)
AI 2027 is doing better than “piece[s] of media” (but that bar is so incredibly low)
I disagree that titotal’s critique is far away from AI 2027 on the relevant spectrum. For example, titotal’s critique was posted on the EA Forum / LessWrong, and focused on technical disagreements, rather than going through a huge amplification / social media push, and focusing on storytelling.
(I’d agree that AI 2027 put in more effort / are more obviously “trying” relative to titotal, so they’re far away as judged by intent, but I mostly care about outcomes rather than intent.)
You might say that obviously AI 2027 needed to do the amplification / social media push + storytelling in order to achieve its goals of influencing the discourse, and I would agree with you. But “influence the discourse” is ultimately going to be about status and prestige (given how discourse works in practice). If you’re taking a stance against goals around status and prestige that trade off against epistemic commons, I think you also need to take a stance against AI 2027. (To be clear, I don’t take that stance! I’m just arguing for consistency.)
Before AI 2027 was posted with a big amplification / media push, it underwent as far as I can tell the single most intense set of review and feedback requests of any big writing project I’ve seen so far. I don’t know whether it was literally posted on LessWrong, but I’ve seen comments from many many dozens if not hundreds of people over the many dozens of revisions that the scenario underwent.
Like, I am quite into public discourse being better than private Google Doc systems, but AI 2027 was so widely circulated pre-publication in Google Doc format, with lots of focus on technical disagreements, that this seems easily much superior to what is going on with this post.
I don’t see how this is responding to anything I’ve said? What in my comment are you disagreeing with or adding color to?
Again, my position is not “AI 2027 did something bad”. My position is “stop critiquing people for having goals around status and prestige rather than epistemics, or at least do so consistently”.
(Incidentally, I suspect bio anchors did better on the axis of getting good reviews / feedback, but that isn’t particularly central to anything I’m claiming.)
For example, titotal’s critique was posted on the EA Forum / LessWrong, and focused on technical disagreement
And I was saying that this is also true for the early drafts of AI 2027. Only after a long discussion of the technical disagreements did it go on to a huge amplification thing. This seems directly relevant to that section.
I am responding to the part about consistent standards. I don’t really understand what you believe here, clearly you care a lot about people not using lots of rhetorical tricks and adversarial persuasion tactics all the time, and we’ve talked about that in the past, so I am just straightforwardly arguing that on those dimensions titotal’s post was much worse compared to AI 2027.
We don’t need to come to agreement on this part, it does seem kind of hard to evaluate. But in as much as your top level comment is arguing some kind of asymmetric standard is being applied, that just seems super wrong to me. I don’t know where I would put the line of encourage/discourage, but I don’t see any inconsistency in being unhappy with what titotal is doing and happy about what AI 2027 is doing.
I don’t see any inconsistency in being unhappy with what titotal is doing and happy about what AI 2027 is doing.
I agree with this. I was responding pretty specifically to Zvi’s critique in particular, which is focusing on things like the use of the word “bad” and the notion that there could be a goal to lower the status and prestige of AI 2027. If instead the critique was about e.g. norms of intellectual discourse I’d be on board.
That said I don’t feel like your defense feels all that strong to me? I’m happy to take your word for it that there was lots of review of AI 2027, but my understanding is that titotal also engaged quite a lot with the authors of AI 2027 before publishing the post? (I definitely expect it was much lower engagement / review in an absolute sense, but everything about it is going to be much lower in an absolute sense, since it is not as big a project.)
If I had to guess at the difference between us, it would be that I primarily see emotionally gripping storytelling as a symmetric weapon to be regarded with suspicion by default, whereas you primarily view it as an important and valuable way to get people to really engage with a topic. (Though admittedly on this view I can’t quite see why you’d object to describing a model as “bad”, since that also seems like a way to get people to better engage with a topic.) Or possibly it’s more salient to me how the storytelling in the finished AI 2027 product comes across since I wasn’t involved in its creation, whereas to you the research and analysis is more salient.
Anyway it doesn’t seem super worth digging to the bottom of this, seems reasonable to leave it here (though I would be interested in any reactions you have if you felt like writing them).
EDIT: Actually looking at the other comments here I think it’s plausible that a lot of the difference is in creators thinking the point of AI 2027 was the scenario whereas the public reception was much more about timelines. I feel like it was very predictable that public reception would focus a lot on the timeline, but perhaps this would have been less clear in advance. Though looking at Scott’s post, the timeline is really quite central to the presentation, so I don’t feel like this can really be a surprise.
To clarify, by AI 2027 do you include the timeline model? If so, I’d be interested to know if the reviews caught and/or discussed any of the primary criticisms that titotal has brought up here, particularly the “model is insensitive to starting conditions” bits.
(I recognize I’m butting into a conversation so feel absolutely free to ignore this.)
I don’t know! I would have to look through all the Google Docs comments and like 10 different versions.
In general though, I seem to have a very different relationship to all the supplements than some other people reading AI 2027, and I kind of wonder whether it would just be better to not have the supplements at all.
From my perspective the key thing is the scenario and the associated expandable boxes and explanations. And then I view most of the supplements as kind of helpful essays for trying to understand and explain some of the intuitions that generated the scenario, but the process for the whole thing is very much not “there is an externally validatable scientific model that was built, then that model was used to generate a scenario”. The key engagement I am interested in is people arguing against the scenario, not doing some kind of weird “oh, but your models aren’t externally validatable and actually in order to say anything about the future of AI your models need to be conceptually perfect”.
I really don’t think the graph-fitting described in the timelines supplement was that causally upstream of the beliefs of almost any of the people involved, and I kind of view it more as a single individual sanity-check on whether the basic premise of the scenario checks out. When people try to forecast things as complicated as this, they don’t create nice formal models, they have a model in their head that handles a huge number of edge-cases, and is trying to be consistent with much much more things than the formal model could ever represent. Ideally the research supplements would say something like that at the top, though it’s plausible that some of the AI Futures Project team relate to their epistemic process differently (though if they do, I think they are just kind of confused).
I don’t even think the Timelines Forecast supplement says anything like “this timelines forecast is the basis of the timeline of the mainline scenario”. It’s just like, a semi-random methodology for forecasting a transformative AI timeline that vaguely informed the main scenario. Conceptually, it feels similar to just doing a random fermi estimate in the middle of a blog post to sanity-check that the thing I am thinking about isn’t completely crazy.
I think it’s still good to engage with it on its own terms, and think there is value in that, but it’s really not what seems remotely most productive to me when thinking about all of AI 2027.
In general though, I seem to have a very different relationship to all the supplements than some other people reading AI 2027, and I kind of wonder whether it would just be better to not have the supplements at all.
I think this is likely to be true, yes. FWIW, most of the non-AI-researcher people I have talked to about AI 2027 are extremely surprised to hear that the story was not generated in any meaningful sense by the model supplements. It may not explicitly say this—I agree that if folks parse the language on the website very carefully they can plausibly come to that conclusion—but it seems like a pretty crucial thing to be explicit about, just so folks know how to interpret things.
Thanks for the correction! I’m guessing you don’t want to, but I would appreciate an elaboration on your part; is @habryka’s description below inaccurate, or did I misinterpret it?
It’s just like, a semi-random methodology for forecasting a transformative AI timeline that vaguely informed the main scenario. Conceptually, it feels similar to just doing a random fermi estimate in the middle of a blog post to sanity-check that the thing I am thinking about isn’t completely crazy.
OK I just had a chat with Eli to try to trace the causal history as best we can remember. At a high level, we were working on the scenario and the supplementary research in parallel, and went back and forth making edits to both for months, and our views evolved somewhat over the course of that time.
Timelines: We initially set AGI in 2027 based on my AGI median, which was informed based on a combination of arguments regarding gains from scaling up agency training, as well as a very crude, handwavy version of what later became the benchmarks and gaps model. Later timelines modeling (the stuff that actually went on the website) along with some additional evidence that came out, pushed my median back to 2028. We denoted this in a footnote on the site (footnote #1 in fact) and I posted a shortform about it (plus a tweet or two). tl;dr is that 2027 was my mode, not my median, after the update. We considered rewriting the scenario to happen about one year later, due to this, but decided against since that would have taken a lot of extra time and didn’t really change any of the implications. If the timelines model had given very different results which changed our views against 2027 being plausible, then we would have re-written the scenario. I also mentioned this to Kevin Roose in my interview with him (my somewhat later timelines, the difference between median and mode). I didn’t expect people to make such a big deal of this.
Takeoff: The takeoff model for our first scenario, the “practice scenario” which we basically scrapped, was basically a simplified version of Davidson’s takeoff speeds model. (takeoffspeeds.com) Later takeoff modeling informed which milestones to focus on the scenario (superhuman coder, superhuman AI researcher, etc.) and what AI R&D progress multiplier they should have. Our memory isn’t clear on to what extent they also resulted in changes to the speed of the milestone progression. We think an early crude version of our takeoff model might have resulted in significant changes, but we aren’t sure. We were also working on our takeoff model up until the last minute, and similar to the timelines model mostly used it as a sanity check.
Compute: The first version of this was done in early 2024, and the result of it and future versions were directly imported into the scenario.
AI Goals: Early versions of this supplement were basically responsible for our decision to go with instrumentally convergent goals as the AIs’ ultimate goals in the scenario.
Security: This one was in between a sanity check and directly feeding into the scenario. It didn’t result in large changes but confirmed the likelihood of the weight theft and informed various decisions about e.g. cyberattacks.
So.… Habryka’s description is somewhat accurate, certainly more accurate than your description (“no meaningful sense”). But I think it still undersells it. That said, it’s definitely not the case that we wrote all the supplements first and then wrote the scenario based on the outputs of those calculations; instead, we wrote them in parallel, had various shitty early versions, etc.
If you want to know more about the evidence & modelling that shaped our views in early 2024 when we were starting the project, I could try to compile a list. I’ve already mentioned takeoffspeeds.com for example. There’s lots of other writing I’ve put on LessWrong on the subject as well.
My guess is there is no confusion about this, but to be clear, I didn’t intend to speak on behalf of the AI 2027 team. Indeed, it’s plausible to me they disagree with it, though my honest belief in that case is that they are confused about the sources of their own beliefs, not that my statement is wrong. I.e. I said:
Ideally the research supplements would say something like that at the top, though it’s plausible that some of the AI Futures Project team relate to their epistemic process differently (though if they do, I think they are just kind of confused).
The timelines model didn’t get nearly as many reviews as the scenario. We shared the timelines writeup with all of the people who we shared the later drafts of the scenario with, but I think almost none of them looked at the timelines writeup.
We also asked a few people to specifically review the timelines forecasts, most notably a few FutureSearch forecasters who we then added as a final author. However, we mainly wanted them to estimate the parameter values and didn’t specifically ask them for feedback on the underlying modeling choices (though they did form some opinions, for example they liked benchmark and gaps much more than time horizon extension; also btw the superexponential plays a much smaller role in benchmarks and gaps). No one brought up the criticisms that titotal did.
In general the timelines model certainly got way less effort than the scenario, probably about 5% as much effort. Our main focus was the scenario as we think that it’s a much higher value add.
I’m been pretty surprised at to how much quality-weighted criticisms have focused on the timelines model relative to the scenario, and wish that it was more tilted toward the scenario (and also toward the takeoff model, which IMO is more important than the timelines model but has gotten much less attention). To be clear I’m still very glad that these critiques exist if the alternative is that they didn’t exist and nothing replaced them.
I suspect part of the reasons for the quality-weighted criticism of the timelines rather than the scenario:
If it is the case that you put far less effort into the timelines model than the scenario, then the timelines model is probably just worse—some of the more obvious mistakes that titotal points out probably don’t have analogies in your scenario, so its just easier to criticise the timelines model, as there is more to criticise there
In many ways, the timelines model is pretty key to the headline claim of your scenario. The other parts (scenario and takeoff) are useful, high quality contributions but in many ways are less meaningfully novel than the very aggressive timelines. Your takeoff model, for example, is well within the range of speeds considered in the community for years—indeed, it is far slower than a Yudkowskian takeoff for example. This isn’t to degrade it—the level of detail in the scenario is commendable and the quality in that respect is genuinely novel. But in terms of what the media coverage, and impact of the work, its the timelines that I suspect are the most significant
(FWIW in this comment I am largely just repeating things already said in the longer thread… I wrote this mostly to clarify my own thinking.)
I think the conflict here is that, within intellectual online writing circles, attempting to use the title of a post to directly attempt to set a bottom line in the status of something is defecting on a norm, but this is not so in the ‘internet of beefs’ rest of the world, where titles are readily used as cudgels in status fights.
Within the intellectual online writing circles, this is not a good goal for a title, and it’s not something that AI 2027 did (or, like, something that ~any ACX post or ~any LW curated post does)[1]. This is not the same as “not putting your bottom line in the title”, it’s “don’t attempt to directly write the bottom line about the status of something in your title”.
I agree you’re narrowly correct that it’s acceptable to have goals for changing the status of various things, and it’s good to push back on implying that that isn’t allowed by any method. But I think Zvi did make the point that the title itself of the critique post attempted to do it using the title and that’s not something AI 2027 did and is IMO defecting on a worthy truce in the intellectual online circles.
In any case, it does seem LW curated posts and ACX posts both usually have neutral titles, especially given the occasionally contentious nature of their contents.
“Moldbug sold out” is definitely an attack on someone’s status. I still prefer it, because it makes a concrete claim about why. For instance, if the AI 2027 critique post title was “AI 2027′s Graphs Are Made Up And Unjustified” this would feel to me much better than something only about status like “AI 2027′s Timeline Forecasts Are Bad”.
For instance, if the AI 2027 critique post title was “AI 2027′s Graphs Are Made Up And Unjustified” this would feel to me much better than something only about status like “AI 2027′s Timeline Forecasts Are Bad”.
But then that wouldn’t be an accurate description of what titotal’s post is about.
“AI 2027′s authors’ arguments for superexponential growth curves are conceptually flawed, and their exponential model is neither exponential nor well-justified, and their graphs are made up and unjustified, and their projections don’t take into account many important variables, and benchmark+gaps is a worse model than the simplified one [for technical reasons], and these kinds of forecasts should be viewed with inherent skepticism for the following reasons” would be a proper summary of what is going on… but obviously it’s not suitable as a title.
I mean… the reason the AI 2027 critique isn’t titled “AI 2027′s Graphs Are Made Up And Unjustified” is obviously because the critique is about so much more than just some graphs on ACX and Twitter, right? That’s just one small part of the criticism, regardless of how much post-publication public discourse has focused on that one aspect.
The post is ultimately about why the timeline forecasts are (according to the author) bad, so it seems quite hard to title it something direct and concrete when it’s a compilation of many separate issues titotal has with AI 2027.
Hmm, interesting. I was surprised by the claim so I did look back through ACX and posts from the LW review, and it does seem to back up your claim (the closest I saw was “Sorry, I Still Think MR Is Wrong About USAID”, note I didn’t look very hard). EDIT: Actually I agree with sunwillrise that “Moldbug sold out” meets the bar (and in general my felt sense is that ACX does do this).
I’d dispute the characterization of this norm as operating “within intellectual online writing circles”. I think it’s a rationalist norm if anything. For example I went to Slow Boring and the sixth post title is “Tema Okun’s “White Supremacy Culture” work is bad”.
This norm seems like it both (1) creates incentives against outside critique and (2) lessens the extremes of a bad thing (e.g. like a norm that even if you have fistfights you won’t use knives). I think on balance I support it but still feel pretty meh about its application in this case. Still, this did change my mind somewhat, thanks.
I am both surprised and glad my comment led to an update :)
FWIW I never expect the political blogs to be playing by the good rules of the rest of the intellectual writing circles, I view them more as soldiers. Not centralexamples of soldiers, but enough so that I’d repeatedly be disappointed by them if I expected them to hold themselves to the same standards.
(As an example, in my mind I confidently-but-vaguely recall some Matt Yglesias tweets where he endorsed dishonesty for his side of the political on some meta-level, in order to win political conflicts; interested if anyone else recalls this / has a link.)
(If this also doesn’t count as “intellectual writing circles”, consider renaming your category, since I clearly do not understand what you mean, except inasmuch as it is “rationalist or rationalist-adjacent circles”.)
The Gelman post in question is importantly not about arguing for the linked post being bad/stupid, it was taking it fully as a given. I actually think that’s an importantly different dynamic because if you are in a context where you can actually presume with your audience that something is bad, then writing it in a title isn’t actually influencing the status landscape very much (though it’s tricky).
Similarly, I think on LessWrong writing a title which presumes the falsity of the existence of a christian god would in other contexts I think be a pretty bad thing to do, but on LessWrong be totally fine, for similar reasons.
As an example, in my mind I confidently-but-vaguely recall some Matt Yglesias tweets where he endorsed dishonesty for his side of the political on some meta-level, in order to win political conflicts; interested if anyone else recalls this / has a link
I won’t say I would necessarily be surprised, per se, if he had written something along these lines, at least on Twitter, but as as general matter Matt believes Misinformation mostly confuses your own side, where he wrote:
My bottom line on this is that saying things that are true is underrated and saying things that are false is overrated.
We’re all acutely aware of the false or misleading things our political opponents say, and it’s easy to convince yourself in the spirit of “turnabout is fair play” that the key to victory is to play dirty, too. The real problem, though, is that not only does your side already say more false and misleading things than you’d like to admit, but they are almost certainly saying more false and misleading things than you realize. That’s because your side is much better at misleading you than they are at misleading people outside of your ideological camp, and this kind of own-team deception creates huge tactical and strategic problems.
I do believe Matt’s support of truth-telling in political fights is instrumental rather than a terminal value for him, so perhaps him articulating this is what you were thinking of?
Come on, this is such an isolated demand for rigor. AI 2027 clearly had the goal of raising the status and prestige of belief in AI risk and short timelines. They employed tons of symmetric weapons in the pursuit of this goal. I’m maybe 95% sure you didn’t substantially critique them for that.[1] Why start with this much less viral post?
(To be clear I’m not against the use of symmetric weapons. I’m against censure of the side you disagree with that masquerades as being impartial, whether or not that was deliberate.)
If you did, my apologies, it’s hard to keep up.
I don’t think AI 2027 did anything even close to as crude as calling the thing you are arguing against just “bad” in your title.
Indeed, I think overall AI 2027 is really doing remarkably well at being asymmetric in really a huge number of its choices (I am of course biased as having been involved in many of those choices, but I currently would say that AI 2027 is close to the very top at the intersection of “accessible” and “trying to make itself only succeed and compelling if indeed its claims are true” as I think any piece of media out there).
(I don’t have a super considered take on whether I think the title is a fine title, but it seems clear to me that there is a spectrum of trying to make your thing asymmetrically successful and that titotal’s critique is far away from AI 2027 on that spectrum, in the direction of symmetry instead of asymmetry)
Things I agree with:
AI 2027 was less crude in its use of symmetric weapons (which can often itself be a good symmetric weapon when the goal is to influence elites)
AI 2027 made lots of asymmetric choices (but so did titotal)
AI 2027 is doing better than “piece[s] of media” (but that bar is so incredibly low)
I disagree that titotal’s critique is far away from AI 2027 on the relevant spectrum. For example, titotal’s critique was posted on the EA Forum / LessWrong, and focused on technical disagreements, rather than going through a huge amplification / social media push, and focusing on storytelling.
(I’d agree that AI 2027 put in more effort / are more obviously “trying” relative to titotal, so they’re far away as judged by intent, but I mostly care about outcomes rather than intent.)
You might say that obviously AI 2027 needed to do the amplification / social media push + storytelling in order to achieve its goals of influencing the discourse, and I would agree with you. But “influence the discourse” is ultimately going to be about status and prestige (given how discourse works in practice). If you’re taking a stance against goals around status and prestige that trade off against epistemic commons, I think you also need to take a stance against AI 2027. (To be clear, I don’t take that stance! I’m just arguing for consistency.)
Before AI 2027 was posted with a big amplification / media push, it underwent as far as I can tell the single most intense set of review and feedback requests of any big writing project I’ve seen so far. I don’t know whether it was literally posted on LessWrong, but I’ve seen comments from many many dozens if not hundreds of people over the many dozens of revisions that the scenario underwent.
Like, I am quite into public discourse being better than private Google Doc systems, but AI 2027 was so widely circulated pre-publication in Google Doc format, with lots of focus on technical disagreements, that this seems easily much superior to what is going on with this post.
I don’t see how this is responding to anything I’ve said? What in my comment are you disagreeing with or adding color to?
Again, my position is not “AI 2027 did something bad”. My position is “stop critiquing people for having goals around status and prestige rather than epistemics, or at least do so consistently”.
(Incidentally, I suspect bio anchors did better on the axis of getting good reviews / feedback, but that isn’t particularly central to anything I’m claiming.)
I was responding to this part:
And I was saying that this is also true for the early drafts of AI 2027. Only after a long discussion of the technical disagreements did it go on to a huge amplification thing. This seems directly relevant to that section.
I am responding to the part about consistent standards. I don’t really understand what you believe here, clearly you care a lot about people not using lots of rhetorical tricks and adversarial persuasion tactics all the time, and we’ve talked about that in the past, so I am just straightforwardly arguing that on those dimensions titotal’s post was much worse compared to AI 2027.
We don’t need to come to agreement on this part, it does seem kind of hard to evaluate. But in as much as your top level comment is arguing some kind of asymmetric standard is being applied, that just seems super wrong to me. I don’t know where I would put the line of encourage/discourage, but I don’t see any inconsistency in being unhappy with what titotal is doing and happy about what AI 2027 is doing.
I agree with this. I was responding pretty specifically to Zvi’s critique in particular, which is focusing on things like the use of the word “bad” and the notion that there could be a goal to lower the status and prestige of AI 2027. If instead the critique was about e.g. norms of intellectual discourse I’d be on board.
That said I don’t feel like your defense feels all that strong to me? I’m happy to take your word for it that there was lots of review of AI 2027, but my understanding is that titotal also engaged quite a lot with the authors of AI 2027 before publishing the post? (I definitely expect it was much lower engagement / review in an absolute sense, but everything about it is going to be much lower in an absolute sense, since it is not as big a project.)
If I had to guess at the difference between us, it would be that I primarily see emotionally gripping storytelling as a symmetric weapon to be regarded with suspicion by default, whereas you primarily view it as an important and valuable way to get people to really engage with a topic. (Though admittedly on this view I can’t quite see why you’d object to describing a model as “bad”, since that also seems like a way to get people to better engage with a topic.) Or possibly it’s more salient to me how the storytelling in the finished AI 2027 product comes across since I wasn’t involved in its creation, whereas to you the research and analysis is more salient.
Anyway it doesn’t seem super worth digging to the bottom of this, seems reasonable to leave it here (though I would be interested in any reactions you have if you felt like writing them).
EDIT: Actually looking at the other comments here I think it’s plausible that a lot of the difference is in creators thinking the point of AI 2027 was the scenario whereas the public reception was much more about timelines. I feel like it was very predictable that public reception would focus a lot on the timeline, but perhaps this would have been less clear in advance. Though looking at Scott’s post, the timeline is really quite central to the presentation, so I don’t feel like this can really be a surprise.
To clarify, by AI 2027 do you include the timeline model? If so, I’d be interested to know if the reviews caught and/or discussed any of the primary criticisms that titotal has brought up here, particularly the “model is insensitive to starting conditions” bits.
(I recognize I’m butting into a conversation so feel absolutely free to ignore this.)
I don’t know! I would have to look through all the Google Docs comments and like 10 different versions.
In general though, I seem to have a very different relationship to all the supplements than some other people reading AI 2027, and I kind of wonder whether it would just be better to not have the supplements at all.
From my perspective the key thing is the scenario and the associated expandable boxes and explanations. And then I view most of the supplements as kind of helpful essays for trying to understand and explain some of the intuitions that generated the scenario, but the process for the whole thing is very much not “there is an externally validatable scientific model that was built, then that model was used to generate a scenario”. The key engagement I am interested in is people arguing against the scenario, not doing some kind of weird “oh, but your models aren’t externally validatable and actually in order to say anything about the future of AI your models need to be conceptually perfect”.
I really don’t think the graph-fitting described in the timelines supplement was that causally upstream of the beliefs of almost any of the people involved, and I kind of view it more as a single individual sanity-check on whether the basic premise of the scenario checks out. When people try to forecast things as complicated as this, they don’t create nice formal models, they have a model in their head that handles a huge number of edge-cases, and is trying to be consistent with much much more things than the formal model could ever represent. Ideally the research supplements would say something like that at the top, though it’s plausible that some of the AI Futures Project team relate to their epistemic process differently (though if they do, I think they are just kind of confused).
I don’t even think the Timelines Forecast supplement says anything like “this timelines forecast is the basis of the timeline of the mainline scenario”. It’s just like, a semi-random methodology for forecasting a transformative AI timeline that vaguely informed the main scenario. Conceptually, it feels similar to just doing a random fermi estimate in the middle of a blog post to sanity-check that the thing I am thinking about isn’t completely crazy.
I think it’s still good to engage with it on its own terms, and think there is value in that, but it’s really not what seems remotely most productive to me when thinking about all of AI 2027.
I think this is likely to be true, yes. FWIW, most of the non-AI-researcher people I have talked to about AI 2027 are extremely surprised to hear that the story was not generated in any meaningful sense by the model supplements. It may not explicitly say this—I agree that if folks parse the language on the website very carefully they can plausibly come to that conclusion—but it seems like a pretty crucial thing to be explicit about, just so folks know how to interpret things.
It is false that the story was not generated in any meaningful sense by the model supplements.
Thanks for the correction! I’m guessing you don’t want to, but I would appreciate an elaboration on your part; is @habryka’s description below inaccurate, or did I misinterpret it?
OK I just had a chat with Eli to try to trace the causal history as best we can remember. At a high level, we were working on the scenario and the supplementary research in parallel, and went back and forth making edits to both for months, and our views evolved somewhat over the course of that time.
Timelines: We initially set AGI in 2027 based on my AGI median, which was informed based on a combination of arguments regarding gains from scaling up agency training, as well as a very crude, handwavy version of what later became the benchmarks and gaps model. Later timelines modeling (the stuff that actually went on the website) along with some additional evidence that came out, pushed my median back to 2028. We denoted this in a footnote on the site (footnote #1 in fact) and I posted a shortform about it (plus a tweet or two). tl;dr is that 2027 was my mode, not my median, after the update. We considered rewriting the scenario to happen about one year later, due to this, but decided against since that would have taken a lot of extra time and didn’t really change any of the implications. If the timelines model had given very different results which changed our views against 2027 being plausible, then we would have re-written the scenario. I also mentioned this to Kevin Roose in my interview with him (my somewhat later timelines, the difference between median and mode). I didn’t expect people to make such a big deal of this.
Takeoff: The takeoff model for our first scenario, the “practice scenario” which we basically scrapped, was basically a simplified version of Davidson’s takeoff speeds model. (takeoffspeeds.com) Later takeoff modeling informed which milestones to focus on the scenario (superhuman coder, superhuman AI researcher, etc.) and what AI R&D progress multiplier they should have. Our memory isn’t clear on to what extent they also resulted in changes to the speed of the milestone progression. We think an early crude version of our takeoff model might have resulted in significant changes, but we aren’t sure. We were also working on our takeoff model up until the last minute, and similar to the timelines model mostly used it as a sanity check.
Compute: The first version of this was done in early 2024, and the result of it and future versions were directly imported into the scenario.
AI Goals: Early versions of this supplement were basically responsible for our decision to go with instrumentally convergent goals as the AIs’ ultimate goals in the scenario.
Security: This one was in between a sanity check and directly feeding into the scenario. It didn’t result in large changes but confirmed the likelihood of the weight theft and informed various decisions about e.g. cyberattacks.
So.… Habryka’s description is somewhat accurate, certainly more accurate than your description (“no meaningful sense”). But I think it still undersells it. That said, it’s definitely not the case that we wrote all the supplements first and then wrote the scenario based on the outputs of those calculations; instead, we wrote them in parallel, had various shitty early versions, etc.
If you want to know more about the evidence & modelling that shaped our views in early 2024 when we were starting the project, I could try to compile a list. I’ve already mentioned takeoffspeeds.com for example. There’s lots of other writing I’ve put on LessWrong on the subject as well.
Does this help?
My guess is there is no confusion about this, but to be clear, I didn’t intend to speak on behalf of the AI 2027 team. Indeed, it’s plausible to me they disagree with it, though my honest belief in that case is that they are confused about the sources of their own beliefs, not that my statement is wrong. I.e. I said:
The timelines model didn’t get nearly as many reviews as the scenario. We shared the timelines writeup with all of the people who we shared the later drafts of the scenario with, but I think almost none of them looked at the timelines writeup.
We also asked a few people to specifically review the timelines forecasts, most notably a few FutureSearch forecasters who we then added as a final author. However, we mainly wanted them to estimate the parameter values and didn’t specifically ask them for feedback on the underlying modeling choices (though they did form some opinions, for example they liked benchmark and gaps much more than time horizon extension; also btw the superexponential plays a much smaller role in benchmarks and gaps). No one brought up the criticisms that titotal did.
In general the timelines model certainly got way less effort than the scenario, probably about 5% as much effort. Our main focus was the scenario as we think that it’s a much higher value add.
I’m been pretty surprised at to how much quality-weighted criticisms have focused on the timelines model relative to the scenario, and wish that it was more tilted toward the scenario (and also toward the takeoff model, which IMO is more important than the timelines model but has gotten much less attention). To be clear I’m still very glad that these critiques exist if the alternative is that they didn’t exist and nothing replaced them.
I suspect part of the reasons for the quality-weighted criticism of the timelines rather than the scenario:
If it is the case that you put far less effort into the timelines model than the scenario, then the timelines model is probably just worse—some of the more obvious mistakes that titotal points out probably don’t have analogies in your scenario, so its just easier to criticise the timelines model, as there is more to criticise there
In many ways, the timelines model is pretty key to the headline claim of your scenario. The other parts (scenario and takeoff) are useful, high quality contributions but in many ways are less meaningfully novel than the very aggressive timelines. Your takeoff model, for example, is well within the range of speeds considered in the community for years—indeed, it is far slower than a Yudkowskian takeoff for example. This isn’t to degrade it—the level of detail in the scenario is commendable and the quality in that respect is genuinely novel. But in terms of what the media coverage, and impact of the work, its the timelines that I suspect are the most significant
(FWIW in this comment I am largely just repeating things already said in the longer thread… I wrote this mostly to clarify my own thinking.)
I think the conflict here is that, within intellectual online writing circles, attempting to use the title of a post to directly attempt to set a bottom line in the status of something is defecting on a norm, but this is not so in the ‘internet of beefs’ rest of the world, where titles are readily used as cudgels in status fights.
Within the intellectual online writing circles, this is not a good goal for a title, and it’s not something that AI 2027 did (or, like, something that ~any ACX post or ~any LW curated post does)[1]. This is not the same as “not putting your bottom line in the title”, it’s “don’t attempt to directly write the bottom line about the status of something in your title”.
I agree you’re narrowly correct that it’s acceptable to have goals for changing the status of various things, and it’s good to push back on implying that that isn’t allowed by any method. But I think Zvi did make the point that the title itself of the critique post attempted to do it using the title and that’s not something AI 2027 did and is IMO defecting on a worthy truce in the intellectual online circles.
To the best of my recollection. Can anyone think of counterexamples?
It’s difficult to determine what you would or wouldn’t call “directly writ[ing] the bottom line about the status of something in your title.”
titotal’s post was titled “A deep critique of AI 2027’s bad timeline models.” Is that more or less about the status of the bottom line than “Futarchy’s fundamental flaw” is? What about “Moldbug sold out” over on ACX?
In any case, it does seem LW curated posts and ACX posts both usually have neutral titles, especially given the occasionally contentious nature of their contents.
“Moldbug sold out” is definitely an attack on someone’s status. I still prefer it, because it makes a concrete claim about why. For instance, if the AI 2027 critique post title was “AI 2027′s Graphs Are Made Up And Unjustified” this would feel to me much better than something only about status like “AI 2027′s Timeline Forecasts Are Bad”.
Added: I searched through a bunch of ACX archives specifically for the word ‘bad’ in titles, I think both titles make a substantive claim about what is bad (Bad Definitions Of “Democracy” And “Accountability” Shade Into Totalitarianism and Perhaps It Is A Bad Thing That The World’s Leading AI Companies Cannot Control Their AIs, the latter of which is slightly sarcastic while making the object level claim that the AI companies cannot control their AIs).
Added2: It was easier to search the complete SSC history for ‘bad’. The examples are Bad Dreams, How Bad Are Things?, Asymmetric Weapons Gone Bad, and Response To Comments: The Tax Bill Is Still Very Bad, which was the sequel to The Tax Bill Compared To Other Very Expensive Things. The last one is the only one similar to what we’re discussing here, but in-context it is said in response to his commenters and as a sequel to a post which did a substantive thing, the title was not the primary thesis for the rest of the internet, which again seems different to me.
But then that wouldn’t be an accurate description of what titotal’s post is about.
“AI 2027′s authors’ arguments for superexponential growth curves are conceptually flawed, and their exponential model is neither exponential nor well-justified, and their graphs are made up and unjustified, and their projections don’t take into account many important variables, and benchmark+gaps is a worse model than the simplified one [for technical reasons], and these kinds of forecasts should be viewed with inherent skepticism for the following reasons” would be a proper summary of what is going on… but obviously it’s not suitable as a title.
I mean… the reason the AI 2027 critique isn’t titled “AI 2027′s Graphs Are Made Up And Unjustified” is obviously because the critique is about so much more than just some graphs on ACX and Twitter, right? That’s just one small part of the criticism, regardless of how much post-publication public discourse has focused on that one aspect.
The post is ultimately about why the timeline forecasts are (according to the author) bad, so it seems quite hard to title it something direct and concrete when it’s a compilation of many separate issues titotal has with AI 2027.
Hmm, interesting. I was surprised by the claim so I did look back through ACX and posts from the LW review, and it does seem to back up your claim (the closest I saw was “Sorry, I Still Think MR Is Wrong About USAID”, note I didn’t look very hard). EDIT: Actually I agree with sunwillrise that “Moldbug sold out” meets the bar (and in general my felt sense is that ACX does do this).
I’d dispute the characterization of this norm as operating “within intellectual online writing circles”. I think it’s a rationalist norm if anything. For example I went to Slow Boring and the sixth post title is “Tema Okun’s “White Supremacy Culture” work is bad”.
This norm seems like it both (1) creates incentives against outside critique and (2) lessens the extremes of a bad thing (e.g. like a norm that even if you have fistfights you won’t use knives). I think on balance I support it but still feel pretty meh about its application in this case. Still, this did change my mind somewhat, thanks.
I am both surprised and glad my comment led to an update :)
FWIW I never expect the political blogs to be playing by the good rules of the rest of the intellectual writing circles, I view them more as soldiers. Not central examples of soldiers, but enough so that I’d repeatedly be disappointed by them if I expected them to hold themselves to the same standards.
(As an example, in my mind I confidently-but-vaguely recall some Matt Yglesias tweets where he endorsed dishonesty for his side of the political on some meta-level, in order to win political conflicts; interested if anyone else recalls this / has a link.)
Andrew Gelman: “Bring on the Stupid: When does it make sense to judge a person, a group, or an organization by its worst?” (Not quite as clearcut, since it doesn’t name the person in the title, but still)
(If this also doesn’t count as “intellectual writing circles”, consider renaming your category, since I clearly do not understand what you mean, except inasmuch as it is “rationalist or rationalist-adjacent circles”.)
I certainly consider Gelman a valid example of the category :)
The Gelman post in question is importantly not about arguing for the linked post being bad/stupid, it was taking it fully as a given. I actually think that’s an importantly different dynamic because if you are in a context where you can actually presume with your audience that something is bad, then writing it in a title isn’t actually influencing the status landscape very much (though it’s tricky).
Similarly, I think on LessWrong writing a title which presumes the falsity of the existence of a christian god would in other contexts I think be a pretty bad thing to do, but on LessWrong be totally fine, for similar reasons.
I won’t say I would necessarily be surprised, per se, if he had written something along these lines, at least on Twitter, but as as general matter Matt believes Misinformation mostly confuses your own side, where he wrote:
I do believe Matt’s support of truth-telling in political fights is instrumental rather than a terminal value for him, so perhaps him articulating this is what you were thinking of?