Thank you for taking the time to write such a detailed response.
My main critique of AI 2027 is not about communication, but the estimates themselves (2027 is an insane median estimate for AI doom) and that I feel you’re overconfident about the quality/reliability of the forecasts. (And I am glad that you and Daniel have both backed off a bit from the original 2027 estimate.)
What do you mean by this? My guess is that it’s related to the communication issues on timelines?
Probably this is related to communication issues on timelines, yes. Also, I think if I genuinely believed everyone I knew and loved was going to die in ~2 years, I would probably be acting a certain way that I don’t sense from the authors of the AI 2027 document. But I don’t want to get too much into mind reading.
With respect to the communication issue, I think the AI 2027 document did include enough disclaimers about the authors’ uncertainty, and more disclaimers wouldn’t help. I think the problem is that the document structurally contradicts those disclaimers, by seeming really academic and precise. Adding disclaimers to the research sections would also not be valuable simply because most people won’t get that far.
Including a written scenario is something I can understand why you chose to do, but it also seems like a mistake for the reasons I mentioned in my post. It makes you sound way more confident than we both agree you actually are. And a specific scenario is also more likely to be wrong than a general forecast.
You have said things like:
“One reason I’m hesitant to add [disclaimers] is that I think it might update non-rationalists too much toward thinking it’s useless, when in fact I think it’s pretty informative.”
“The graphs are the result of an actual model that I think is reasonable to give substantial weight to in one’s timelines estimates.”
“In our initial tweet, Daniel said it was a ‘deeply researched’ scenario forecast. This still seems accurate to me.”
“we put quite a lot of work into it”
“it’s state-of-the-art or close on most dimensions and represents subtantial intellectual progress”
“In particular, I think there’s reason to trust our intuitions”
As I said in my post, “The whole AI 2027 document just seems so fancy and robust. That’s what I don’t like. It gives a much more robust appearance than this blog post, does it not? But is it any better? I claim no.”
I don’t think your guesses are better than mine because of the number of man hours your put into justifying them, nor because the people who worked on the estimates are important, well-regarded people who worked at OpenAI or have a better track record, nor because the estimates involved surveys, wargames, and mathematics.
I do not believe your guesses are particularly informative, nor do I think that about my own guesses. We’re all just guessing. Nor do I agree with calling them forecasts at all. I don’t think they’re reliable enough that anybody should be trusting them over their own intuition. In the end, neither of us can prove what we believe to a high degree of confidence. The only thing that will matter is who’s right, and none of the accoutrements of fancy statistics, hours spent researching, past forecasting successes, and so on will matter.
Putting too much work into what are essentially guesses is also in itself a kind of communication that this is Serious Academic Work—a kind of evidence or proof that people should take very seriously. Which it can’t be, since you and I agree that “there’s simply not enough empirical data to forecast when AGI will arrive”. If that’s true, then why all the forecasting?
(All my criticism is about the Timelines/Takeoff Forecasting, since these are things you can’t really forecast at this time. I am glad the Compute Forecast exists, and I didn’t read the AI Goals and Security Forecasts)
Okay, it sounds like our disagrement basically boils down to the value of the forecasts as well as the value of the scenario format (does that seem right?), which I don’t think is something we’ll come to agreement on.
Thanks again for writing this up! I hope you’re right about timelines being much longer and 2027 being insane (as I mentioned, it’s faster than my median has ever been, but I think it’s plausible enough to take seriously).
edit: I’d also be curious for you to specify what you mean by academic? The scenario itself seems like a very unusual format for academia. I think it would have seemed more serious academic-y if we had ditched the scenario format.
Perhaps we will find some agreement come Christmastime 2027. Until then, thanks for your time!
edit: Responding to your edit, by seeming academic, I meant things like seeming “detailed and evidence-based”, “involving citations and footnotes”, “involving robust statistics”, “resulting in high-confidence conclusions”, and stuff like that. Even the typography and multiple authors makes it seem Very Serious. I agree that the scenario part seemed less academic that the research pages.
Thank you for taking the time to write such a detailed response.
My main critique of AI 2027 is not about communication, but the estimates themselves (2027 is an insane median estimate for AI doom) and that I feel you’re overconfident about the quality/reliability of the forecasts. (And I am glad that you and Daniel have both backed off a bit from the original 2027 estimate.)
Probably this is related to communication issues on timelines, yes. Also, I think if I genuinely believed everyone I knew and loved was going to die in ~2 years, I would probably be acting a certain way that I don’t sense from the authors of the AI 2027 document. But I don’t want to get too much into mind reading.
With respect to the communication issue, I think the AI 2027 document did include enough disclaimers about the authors’ uncertainty, and more disclaimers wouldn’t help. I think the problem is that the document structurally contradicts those disclaimers, by seeming really academic and precise. Adding disclaimers to the research sections would also not be valuable simply because most people won’t get that far.
Including a written scenario is something I can understand why you chose to do, but it also seems like a mistake for the reasons I mentioned in my post. It makes you sound way more confident than we both agree you actually are. And a specific scenario is also more likely to be wrong than a general forecast.
You have said things like:
“One reason I’m hesitant to add [disclaimers] is that I think it might update non-rationalists too much toward thinking it’s useless, when in fact I think it’s pretty informative.”
“The graphs are the result of an actual model that I think is reasonable to give substantial weight to in one’s timelines estimates.”
“In our initial tweet, Daniel said it was a ‘deeply researched’ scenario forecast. This still seems accurate to me.”
“we put quite a lot of work into it”
“it’s state-of-the-art or close on most dimensions and represents subtantial intellectual progress”
“In particular, I think there’s reason to trust our intuitions”
As I said in my post, “The whole AI 2027 document just seems so fancy and robust. That’s what I don’t like. It gives a much more robust appearance than this blog post, does it not? But is it any better? I claim no.”
I don’t think your guesses are better than mine because of the number of man hours your put into justifying them, nor because the people who worked on the estimates are important, well-regarded people who worked at OpenAI or have a better track record, nor because the estimates involved surveys, wargames, and mathematics.
I do not believe your guesses are particularly informative, nor do I think that about my own guesses. We’re all just guessing. Nor do I agree with calling them forecasts at all. I don’t think they’re reliable enough that anybody should be trusting them over their own intuition. In the end, neither of us can prove what we believe to a high degree of confidence. The only thing that will matter is who’s right, and none of the accoutrements of fancy statistics, hours spent researching, past forecasting successes, and so on will matter.
Putting too much work into what are essentially guesses is also in itself a kind of communication that this is Serious Academic Work—a kind of evidence or proof that people should take very seriously. Which it can’t be, since you and I agree that “there’s simply not enough empirical data to forecast when AGI will arrive”. If that’s true, then why all the forecasting?
(All my criticism is about the Timelines/Takeoff Forecasting, since these are things you can’t really forecast at this time. I am glad the Compute Forecast exists, and I didn’t read the AI Goals and Security Forecasts)
Okay, it sounds like our disagrement basically boils down to the value of the forecasts as well as the value of the scenario format (does that seem right?), which I don’t think is something we’ll come to agreement on.
Thanks again for writing this up! I hope you’re right about timelines being much longer and 2027 being insane (as I mentioned, it’s faster than my median has ever been, but I think it’s plausible enough to take seriously).
edit: I’d also be curious for you to specify what you mean by academic? The scenario itself seems like a very unusual format for academia. I think it would have seemed more serious academic-y if we had ditched the scenario format.
Perhaps we will find some agreement come Christmastime 2027. Until then, thanks for your time!
edit: Responding to your edit, by seeming academic, I meant things like seeming “detailed and evidence-based”, “involving citations and footnotes”, “involving robust statistics”, “resulting in high-confidence conclusions”, and stuff like that. Even the typography and multiple authors makes it seem Very Serious. I agree that the scenario part seemed less academic that the research pages.