I’m kind of split about this critique, since the forecast did end up as good propaganda if nothing else. But I do now feel that the marketing around it was kind of misleading, and we probably care about maintaining good epistemics here or something.
I’m interested in you expanding on which parts of the marketing were misleading. Here are some quick more specific thoughts:
Overall AI 2027 comms
In our website frontpage, I think we were pretty careful not to overclaim. We say that the forecast is our “best guess”, “informed by trend extrapolations, wargames, …” Then in the “How did we write it?” box we basically just say it was written iteratively and informed by wargames and feedback. In “Why is it valuable?” we say “We have set ourselves an impossible task. Trying to predict how superhuman AI in 2027 would go is like trying to predict how World War 3 in 2027 would go, except that it’s an even larger departure from past case studies. Yet it is still valuable to attempt, just as it is valuable for the US military to game out Taiwan scenarios.” I don’t think we said anywhere that it was backed up by straightforward, strongly empirically validated extrapolations.
In our initial tweet, Daniel said it was a “deeply researched” scenario forecast. This still seems accurate to me, we spent quite a lot of time on it (both the scenario and supplements) and I still think our supplementary research is mostly state of the art, though I can see how people could take it too strongly.
In various follow-up discussions, I think Scott and others sometimes pointed to the length of all of the supplementary research as justification for taking the scenario seriously. I still think this mostly holds up but again I think it could be interpreted in the wrong way.
Probably there has been similar discussion in various podcast appearances etc., but I haven’t listened to most of those and don’t remember how this sort of thing was presented in the ones I did listen to.
Timelines forecast specific comms
We do not say prominently explicitly in the timelines forecast that it relies on a bunch of non-obvious parameter choices rather than just empirical trend extrapolation, so I agree that people could come away with the wrong impression.
Plausibly we should have had / I should add a disclaimer saying something like this.
I have been frustrated with previous forecasts for not communicating this well, so plausibly I’m being hypocritical.
One reason I’m hesitant to add this is that I think it might update non-rationalists too much toward thinking it’s useless, when in fact I think it’s pretty informative. But this might be motivated reasoning toward the choice I made before. I might add a disclaimer.
I didn’t explicitly consider adding a prominent disclaimer previously; perhaps because I was typical minding and thinking it was obvious that any AGI timelines forecast will rely on intuitively estimated parameters.
However, I think that including 3 different people/groups’ forecasts very prominently does implicitly get across the idea that different parameter estimations can lead to very different results. This is especially true for including the FutureSearch aggregate, which has a within-model median of 2032 rather than 2027 or 2028.
There’s a graph at the top of the timelines forecast with all 3 of our distributions, and in my tweet thread about the timelines forecast this was in my top tweet.
As I’ve said, I agree that we messed up to some extent re: the time horizon prediction graph. I might write more about this in response to TurnTrout.
Not-very-charitably put, my impression now is that all the technical details in the forecast were free parameters fine-tuned to support the authors’ intuitions, when they weren’t outright ignored. Now, I also gather that those intuitions were themselves supported by playing around with said technical models, and there’s something to be said about doing the math, then burning the math and going with your gut. I’m not saying the forecast should be completely dismissed because of that.
I tried not to just fine-tune the parameters to support my existing beliefs, though I of course probably implicitly did to some extent. I agree that the level of free parameters is a reason to distrust our forecasts.
FWIW, my and Daniel’s timelines beliefs have both shifted some as a result of our modeling. Mine initially got shorter then got a bit longer due to the most recent update, Daniel moved his timelines longer to 2028 in significant part because of our timelines model.
… But “the authors, who are smart people with a good track record of making AI-related predictions, intuitively feel that this is sort of right, and they were able to come up with functions whose graphs fit those intuitions” is a completely different kind of evidence compared to “here’s a bunch of straightforward extrapolations of existing trends, with non-epsilon empirical support, that the competent authors intuitively think are going to continue”.
Mostly agree. I would say we have more than non-epsilon empirical support though because of METR’s time horizons work and RE-Bench. But I agree that there are a bunch of parameters estimated that don’t have much empirical support to rely on.
But if I did interpret the forecast as being based on intuitively chosen but non-tampered straightforward extrapolations of existing trends, I think I would be pretty disappointed right now.
I don’t agree with the connotation of “non-tampered,” but otherwise agree re: relying on straightforward extrapolations. I don’t think it’s feasible to only rely on straightforward extrapolations when predicting AGI timelines.
You should’ve maybe put a “these graphs are for illustrative purposes only” footnote somewhere, like this one did.
I think “illustrative purposes only” would be too strong. The graphs are the result of an actual model that I think is reasonable to give substantial weight to in one’s timelines estimates (if you’re only referring to the specific graph that I’ve apologized for, then I agree we should have moved more in that direction re: more clear labeling).
I don’t feel that “this is the least-bad forecast that exists” is a good defence. Whether an analysis is technical or vibes-based is a spectrum, but it isn’t graded on a curve.
I’m not sure exactly how to respond to this. I agree that the absolute level of usefulness of the timelines forecast also matters, and I probably think that our timelines model is more useful than you do. But also I think that the relative usefulness does matter quite a bit on the decision of whether to release and publicize model. I think maybe this critique is primarily coupled with your points about communication issues.
[Unlike the top-level comment, Daniel hasn’t endorsed this, this is just Eli.]
I’m interested in you expanding on which parts of the marketing were misleading
Mostly this part, I think:
In various follow-up discussions, I think Scott and others sometimes pointed to the length of all of the supplementary research as justification for taking the scenario seriously. I still think this mostly holds up but again I think it could be interpreted in the wrong way.
Like, yes, the supplementary materials definitely represent a huge amount of legitimate research that went into this. But the forecasts are “informed by” this research, rather than being directly derived from it, and the pointing-at kind of conveys the latter vibe.
I have been frustrated with previous forecasts for not communicating this well
Glad you get where I’m coming from; I wasn’t wholly sure how legitimate my complaints were.
One reason I’m hesitant to add [a disclaimer about non-obvious parameter choices] is that I think it might update non-rationalists too much toward thinking it’s useless, when in fact I think it’s pretty informative
I agree that this part is tricky, hence my being hesitant about fielding this critique at all. Persuasiveness isn’t something we should outright ignore, especially with something as high-profile as this. But also, the lack of such a disclaimer opens you up to takedowns such as titotal’s, and if one of those becomes high-profile (which it already might have?), that’d potentially hurt the persuasiveness more than a clear statement would have.
There’s presumably some sort of way to have your cake and eat it too here; to correctly communicate how the forecast was generated, but in terms that wouldn’t lead to it being dismissed by people at large.
I think “illustrative purposes only” would be too strong.
Yeah, sorry, I was being unnecessarily hyperbolic there.
I’m interested in you expanding on which parts of the marketing were misleading. Here are some quick more specific thoughts:
Overall AI 2027 comms
In our website frontpage, I think we were pretty careful not to overclaim. We say that the forecast is our “best guess”, “informed by trend extrapolations, wargames, …” Then in the “How did we write it?” box we basically just say it was written iteratively and informed by wargames and feedback. In “Why is it valuable?” we say “We have set ourselves an impossible task. Trying to predict how superhuman AI in 2027 would go is like trying to predict how World War 3 in 2027 would go, except that it’s an even larger departure from past case studies. Yet it is still valuable to attempt, just as it is valuable for the US military to game out Taiwan scenarios.” I don’t think we said anywhere that it was backed up by straightforward, strongly empirically validated extrapolations.
In our initial tweet, Daniel said it was a “deeply researched” scenario forecast. This still seems accurate to me, we spent quite a lot of time on it (both the scenario and supplements) and I still think our supplementary research is mostly state of the art, though I can see how people could take it too strongly.
In various follow-up discussions, I think Scott and others sometimes pointed to the length of all of the supplementary research as justification for taking the scenario seriously. I still think this mostly holds up but again I think it could be interpreted in the wrong way.
Probably there has been similar discussion in various podcast appearances etc., but I haven’t listened to most of those and don’t remember how this sort of thing was presented in the ones I did listen to.
Timelines forecast specific comms
We do not say prominently explicitly in the timelines forecast that it relies on a bunch of non-obvious parameter choices rather than just empirical trend extrapolation, so I agree that people could come away with the wrong impression.
Plausibly we should have had / I should add a disclaimer saying something like this.
I have been frustrated with previous forecasts for not communicating this well, so plausibly I’m being hypocritical.
One reason I’m hesitant to add this is that I think it might update non-rationalists too much toward thinking it’s useless, when in fact I think it’s pretty informative. But this might be motivated reasoning toward the choice I made before. I might add a disclaimer.
I didn’t explicitly consider adding a prominent disclaimer previously; perhaps because I was typical minding and thinking it was obvious that any AGI timelines forecast will rely on intuitively estimated parameters.
However, I think that including 3 different people/groups’ forecasts very prominently does implicitly get across the idea that different parameter estimations can lead to very different results. This is especially true for including the FutureSearch aggregate, which has a within-model median of 2032 rather than 2027 or 2028.
There’s a graph at the top of the timelines forecast with all 3 of our distributions, and in my tweet thread about the timelines forecast this was in my top tweet.
As I’ve said, I agree that we messed up to some extent re: the time horizon prediction graph. I might write more about this in response to TurnTrout.
I tried not to just fine-tune the parameters to support my existing beliefs, though I of course probably implicitly did to some extent. I agree that the level of free parameters is a reason to distrust our forecasts.
FWIW, my and Daniel’s timelines beliefs have both shifted some as a result of our modeling. Mine initially got shorter then got a bit longer due to the most recent update, Daniel moved his timelines longer to 2028 in significant part because of our timelines model.
Mostly agree. I would say we have more than non-epsilon empirical support though because of METR’s time horizons work and RE-Bench. But I agree that there are a bunch of parameters estimated that don’t have much empirical support to rely on.
I don’t agree with the connotation of “non-tampered,” but otherwise agree re: relying on straightforward extrapolations. I don’t think it’s feasible to only rely on straightforward extrapolations when predicting AGI timelines.
I think “illustrative purposes only” would be too strong. The graphs are the result of an actual model that I think is reasonable to give substantial weight to in one’s timelines estimates (if you’re only referring to the specific graph that I’ve apologized for, then I agree we should have moved more in that direction re: more clear labeling).
I’m not sure exactly how to respond to this. I agree that the absolute level of usefulness of the timelines forecast also matters, and I probably think that our timelines model is more useful than you do. But also I think that the relative usefulness does matter quite a bit on the decision of whether to release and publicize model. I think maybe this critique is primarily coupled with your points about communication issues.
[Unlike the top-level comment, Daniel hasn’t endorsed this, this is just Eli.]
Mostly this part, I think:
Like, yes, the supplementary materials definitely represent a huge amount of legitimate research that went into this. But the forecasts are “informed by” this research, rather than being directly derived from it, and the pointing-at kind of conveys the latter vibe.
Glad you get where I’m coming from; I wasn’t wholly sure how legitimate my complaints were.
I agree that this part is tricky, hence my being hesitant about fielding this critique at all. Persuasiveness isn’t something we should outright ignore, especially with something as high-profile as this. But also, the lack of such a disclaimer opens you up to takedowns such as titotal’s, and if one of those becomes high-profile (which it already might have?), that’d potentially hurt the persuasiveness more than a clear statement would have.
There’s presumably some sort of way to have your cake and eat it too here; to correctly communicate how the forecast was generated, but in terms that wouldn’t lead to it being dismissed by people at large.
Yeah, sorry, I was being unnecessarily hyperbolic there.