I think that the views of superforecasters on AI / AI risk should be basically no update.
It seems to me like the main reasons to defer to someone are:
They have a visibly good track record on the relevant domain. It has to be the literal domain, because people often have good views on their area of expertise, but crazy views elsewhere.
They are highly selected for having good beliefs in the domain. For example, if a mathematician tells me something that seems surprising about their area of expertise, I will tend to strongly believe them, despite not being able to evaluate their reasoning. The general reason for this is because mathematics is a verifiable domain, mathematicians are strongly selected for being correct about math. Other domains I’d basically defer to people in are historians about literal historical facts, physicists about well-established physics results, engineers about how cars work, etc. This consideration weakens as disciplines become less verifiable: I’m not very inclined to defer to philosophers, sociologists, psychologists, etc.
They make correct arguments about the domain (and very few incorrect arguments). If it’s the case that you can talk to someone and they can consistently make clear rock-solid arguments that change your mind regularly, it is justified to defer to them on bottom line conclusions, even if you can’t follow the arguments all the way through.
They are much smarter than you and are probably being honest. If someone (or, eventually, an AI) is clearly much smarter than you, and they are being honest (e.g. because they seem like an honest person), then you should probably defer to them substantially. (Of course, this isn’t even fully general, e.g. a few hundred years ago, many of the smartest people around were superstitious, which would have led you astray.)
Now I’ll go through and argue why these don’t apply.
I think the track record of superforecasters on AI looks quite bad. Superforecasters consistently massively under-estimate AI progress on benchmarks, see, e.g., here, here, here, and here. On open-ended forecasting competitions, e.g. AI 2025, the top people (who I recognized, which is biased) all seemed like AI risk domain area experts, not superforecasters (though I’m also not sure if any superforecasters participated). You might object that AI benchmarks don’t track real world impact, and that the real world impact was much lower. I would doubt that superforecasters would have made reasonable predictions of real world impact (such as revenue) over the last few years of AI progress, but I’m not aware of any systematic predictions made by superforecasters on real world impacts. Yes, superforecasters are often good at making geopolitical forecasts, but in practice this doesn’t seem to transfer well to the domain that I care about.
I think the selection effect for being a superforecaster seems nowhere near as strong as a mathematician / physicist, and I think this evidence gets swamped by the above observations of how good they actually seem to be doing on the domain, which seem very poor.
I’ve talked with some superforecasters about AI but not many. Generally, my sense is that they are smart and reasonable people, but don’t know very much about AI. It’s hard to make this argument reasoning transparent; but if you are someone in AI who thinks that it’s appropriate to defer to superforecasters, I think it would be a good idea to try to set up a meeting and talk with one of the people you are deferring to and see if they are actually making reasonable arguments that seem grounded in technical reality. In my experience, the main arguments I’ve encountered are outside view-y considerations like “base rate of extinction from new tech is low” which get obsoleted by strong object level arguments (e.g. for the abnormality of superintelligence and how its more analagous to a new smarter species).
I don’t think the selection effect for being a superforecaster on general intelligence is strong enough for this argument to apply any more than I should defer to e.g. someone who got a 1600 on the SAT.
I also think that it would be better if people deferred less in general because I think that group epistmics go much better if each person in the group attempts to understand the situation as well as possible themselves. If everyone constantly tries to update on everyone else’s views, then (i) there’s way less novel intellectual thoughts, (ii) there are deference cascades / group think, and (iii) people end up with beliefs that are far less crisp / clearly justified than the person who originated the belief (bc communication is hard to do at high fidelity). I generally think people should communicate based on their own inside view, without deference, even if they expect the average correctness of the things that they are saying to go down because the information value of the communication will be much higher than the alternative.
if you are someone in AI who thinks that it’s appropriate to defer to superforecasters, I think it would be a good idea to try to set up a meeting and talk with one of the people you are deferring to and see if they are actually making reasonable arguments that seem grounded in technical reality.
Even better could be if we already had these sorts of arguments collected. https://goodjudgment.com/superforecasting-ai/ contains links to 17 superforecasters’ reviews of Carlsmith’s p(doom) report, some of them supposedly AI experts. I invite people to skim through some of them.
I think EAs often overrate superforecasters’ opinions, they’re not magic. A lot of superforecasters aren’t great (at general reasoning, but even at geopolitical forecasting), there’s plenty of variation in quality.
General quality: Becoming a superforecaster selects for some level of intelligence, open-mindedness, and intuitive forecasting sense among the small group of people who actually make 100 forecasts on GJOpen. There are tons of people (e.g. I’d guess very roughly 30-60% of AI safety full-time employees?) who would become superforecasters if they bothered to put in the time.
Some background: as I’ve written previously I’m intuitively skeptical of the benefits of large amounts of forecasting practice (i.e. would guess strong diminishing returns).
Specialties / domain expertise: Contra a caricturized “superforecasters are the best at any forecasting questions” view, consider a grantmaker deciding whether to fund an organization. They are, whether explicitly or implicitly, forecasting a distribution of outcomes for the grant. But I’d guess most would agree that superforecasters would do significantly worse than grantmakers at this “forecasting question”. A similar argument could be made for many intellectual jobs, which could be framed as forecasting. The question on whether superforecasters are relatively better isn’t “Is this task answering a forecasting question“ but rather “What are the specific attributes of this forecasting question”.
Some people seem to think that the key difference between questions superforecasters are good at vs. smart domain experts are in questions that are *resolvable* or *short-term*. I tend to think that the main differences are along the axes of *domain-specificity* and *complexity*, though these are of course correlated with the other axes. Superforecasters are selected for being relatively good at short-term, often geopolitical questions.
As I’ve written previously: It varies based on the question/domain how much domain expertise matters, but ultimately I expect reasonable domain experts to make better forecasts than reasonable generalists in many domains.
There’s an extreme here where e.g. forecasting what the best chess move is obviously better done by chess experts rather than superforecasters.
So if we think of a spectrum from geopolitics to chess, it’s very unclear to me where things like long-term AI forecasts land.
This intuition seems to be consistent with the lack of quality existing evidence described in Arb’s report (which debunked the “superforecasters beat intelligence experts without classified information” claim!).
Strongly agree, especially with your latter point 3. Fwiw, I used to work at Metaculus. Copy-pasting something I wrote after I left (which was in response to a Slack post about this study):
Forgive my cynicism/jadedness, but IMO this is just another example of superforecasters not understanding alignment or the intelligence explosion, and putting forward forecasts that make the future look kinda normal. My favourite explanation for what is going on, here, is Nuño Sempere’s ‘Excellent forecasters and Superforecasters™ have an imperfect fit for long-term questions’.
(~This is essentially why I lost faith in forecasting as a cause area, and now work on other things.)
[person replied to me saying that “superforecasters not understanding alignment or the intelligence explosion” are overly specific hypotheses—superforecasters may simply disagree with, e.g., alignment being a serious problem]
I agree that they’re specific hypotheses, but they are borne out by the evidence I’ve observed. Like, in my previous role, I spent a lot of time talking to superforecasters/pro forecasters, and when I’d probe them on their p(doom) or their timelines, I’d often find ~zero awareness of the arguments for why AI might be transformative or an x-risk, and also—kinda bafflingly—zero interest in grappling with these arguments.
Disagreement, for me, means ‘they understood what I was talking about, but had models/(counter)arguments for why they weren’t expecting AI to be transformative, etc.’ That’s not what I’ve observed.
I think these are fair criticisms of the “defer to superforecasters” view (which I share), and I think you helped me clarify some of my views here (thanks!), but I feel like it’s missing a few things. The best case for it, in my view, goes something like this:
The world is very hard to predict, and expertise is often overrated in complex domains.
Your first argument—that superforecasters lack domain-specific track records—doesn’t carry much weight if the relevant forecasting questions require broad expertise across regulation, diffusion dynamics, and technical capabilities simultaneously. No one has a verified track record across all of these, and “domain expert” here often just means “has strong inside views in a complicated area.”
On the selection effect: I don’t buy the strong version of your second claim. The fact that there’s low or no signal in non-verifiable domains (ie. philosophy) doesn’t really vindicate inside views—it weakens both. Any somewhat independent signal aggregated across multiple actors is probably better than a single inside view, even an expert one.
On track record: The benchmark underperformance is a real update against superforecasters in the AI case, but the question is how large that update should be. My framing: superforecasters are the prior; evidence of their underperformance updates you toward domain experts with better track records—and yes, I do think it updates toward people like yourself, Ryan, Eli, Daniel, and Peter Wilderford, who have been more right. But by how much? That’s the crux, and I’d genuinely like to see someone work through the math. A few people being more accurate than superforecasters on a hard problem doesn’t automatically license large updates toward their broader worldviews—we should be asking what reference class of questions they outperformed on, and whether that tracks the specific claims we care about. I’d also note that knowing how benchmarks saturate is less relevant to AI risk than you seem to think—the revenue point is stronger.
I’d also push back on a common error I see: people often conclude that “no clear expert → my inside view gets more weight.” This is probably true at the margin, but massively overrated in practice.
On your argument that object-level reasoning obsoletes base rates: This is somewhat circular. You have inside views about what it means to reason well about AI progress, and superforecasters disagree. You’re partially bootstrapping from your own beliefs to dismiss theirs.
On inside views and group epistemics: I agree that deference cascades are bad, but the fix isn’t “everyone uses their inside view”—it’s that people should be clearer about what’s inside vs. outside view reasoning (I agree this is complicated and maybe idealistic, but I don’t think the default for rationalists here should be to take the inside view of the community). I’m also skeptical that inside-view reasoning escapes the groupthink problem. Epistemic bubbles shape which counterarguments you seek, what your priors are, which information you weight. The AI safety/rationalist community isn’t immune to this.
I do think people should build inside views on AI—and the move of not doing so because it’s “not relevant to my field” is more often cope than a principled stance. But I’m genuinely uncertain about what the right policy is after you’ve built one. Surely the answer isn’t just “act on it fully”—the outside view still has to do some work. One practical resolution: argue on inside view, but take actions that at least partially reflect outside-view uncertainty.
A real remaining question: in non-verifiable domains, who counts as an expert? This is, I think, just an open and hard problem.
On your argument that object-level reasoning obsoletes base rates: This is somewhat circular. You have inside views about what it means to reason well about AI progress, and superforecasters disagree. You’re partially bootstrapping from your own beliefs to dismiss theirs.
Oops, “object level reasoning obsoletes base rates” is not what I was trying to argue… my view is that the action is mostly in selecting the right base rate, i.e. that AI is more analogous to a new species than a normal technology.
Also I don’t agree that it’s circular. I think one of the correct reasons to defer to someone is them making correct arguments (as evaluated by my inside view), and that doesn’t apply. I definitely agree that I’m bootstrapping from my views to dismiss theirs. Now, there might be other reasons to defer to someone (for example, the other reasons I gave above), but I was arguing specifically against reason #3 above here.
I think that the views of superforecasters on AI / AI risk should be basically no update.
It seems to me like the main reasons to defer to someone are:
They have a visibly good track record on the relevant domain. It has to be the literal domain, because people often have good views on their area of expertise, but crazy views elsewhere.
They are highly selected for having good beliefs in the domain. For example, if a mathematician tells me something that seems surprising about their area of expertise, I will tend to strongly believe them, despite not being able to evaluate their reasoning. The general reason for this is because mathematics is a verifiable domain, mathematicians are strongly selected for being correct about math. Other domains I’d basically defer to people in are historians about literal historical facts, physicists about well-established physics results, engineers about how cars work, etc. This consideration weakens as disciplines become less verifiable: I’m not very inclined to defer to philosophers, sociologists, psychologists, etc.
They make correct arguments about the domain (and very few incorrect arguments). If it’s the case that you can talk to someone and they can consistently make clear rock-solid arguments that change your mind regularly, it is justified to defer to them on bottom line conclusions, even if you can’t follow the arguments all the way through.
They are much smarter than you and are probably being honest. If someone (or, eventually, an AI) is clearly much smarter than you, and they are being honest (e.g. because they seem like an honest person), then you should probably defer to them substantially. (Of course, this isn’t even fully general, e.g. a few hundred years ago, many of the smartest people around were superstitious, which would have led you astray.)
Now I’ll go through and argue why these don’t apply.
I think the track record of superforecasters on AI looks quite bad. Superforecasters consistently massively under-estimate AI progress on benchmarks, see, e.g., here, here, here, and here. On open-ended forecasting competitions, e.g. AI 2025, the top people (who I recognized, which is biased) all seemed like AI risk domain area experts, not superforecasters (though I’m also not sure if any superforecasters participated). You might object that AI benchmarks don’t track real world impact, and that the real world impact was much lower. I would doubt that superforecasters would have made reasonable predictions of real world impact (such as revenue) over the last few years of AI progress, but I’m not aware of any systematic predictions made by superforecasters on real world impacts. Yes, superforecasters are often good at making geopolitical forecasts, but in practice this doesn’t seem to transfer well to the domain that I care about.
I think the selection effect for being a superforecaster seems nowhere near as strong as a mathematician / physicist, and I think this evidence gets swamped by the above observations of how good they actually seem to be doing on the domain, which seem very poor.
I’ve talked with some superforecasters about AI but not many. Generally, my sense is that they are smart and reasonable people, but don’t know very much about AI. It’s hard to make this argument reasoning transparent; but if you are someone in AI who thinks that it’s appropriate to defer to superforecasters, I think it would be a good idea to try to set up a meeting and talk with one of the people you are deferring to and see if they are actually making reasonable arguments that seem grounded in technical reality. In my experience, the main arguments I’ve encountered are outside view-y considerations like “base rate of extinction from new tech is low” which get obsoleted by strong object level arguments (e.g. for the abnormality of superintelligence and how its more analagous to a new smarter species).
I don’t think the selection effect for being a superforecaster on general intelligence is strong enough for this argument to apply any more than I should defer to e.g. someone who got a 1600 on the SAT.
I also think that it would be better if people deferred less in general because I think that group epistmics go much better if each person in the group attempts to understand the situation as well as possible themselves. If everyone constantly tries to update on everyone else’s views, then (i) there’s way less novel intellectual thoughts, (ii) there are deference cascades / group think, and (iii) people end up with beliefs that are far less crisp / clearly justified than the person who originated the belief (bc communication is hard to do at high fidelity). I generally think people should communicate based on their own inside view, without deference, even if they expect the average correctness of the things that they are saying to go down because the information value of the communication will be much higher than the alternative.
Even better could be if we already had these sorts of arguments collected. https://goodjudgment.com/superforecasting-ai/ contains links to 17 superforecasters’ reviews of Carlsmith’s p(doom) report, some of them supposedly AI experts. I invite people to skim through some of them.
Copying very relevant portions of a comment I wrote in Mar 2024:
I think EAs often overrate superforecasters’ opinions, they’re not magic. A lot of superforecasters aren’t great (at general reasoning, but even at geopolitical forecasting), there’s plenty of variation in quality.
General quality: Becoming a superforecaster selects for some level of intelligence, open-mindedness, and intuitive forecasting sense among the small group of people who actually make 100 forecasts on GJOpen. There are tons of people (e.g. I’d guess very roughly 30-60% of AI safety full-time employees?) who would become superforecasters if they bothered to put in the time.
Some background: as I’ve written previously I’m intuitively skeptical of the benefits of large amounts of forecasting practice (i.e. would guess strong diminishing returns).
Specialties / domain expertise: Contra a caricturized “superforecasters are the best at any forecasting questions” view, consider a grantmaker deciding whether to fund an organization. They are, whether explicitly or implicitly, forecasting a distribution of outcomes for the grant. But I’d guess most would agree that superforecasters would do significantly worse than grantmakers at this “forecasting question”. A similar argument could be made for many intellectual jobs, which could be framed as forecasting. The question on whether superforecasters are relatively better isn’t “Is this task answering a forecasting question“ but rather “What are the specific attributes of this forecasting question”.
Some people seem to think that the key difference between questions superforecasters are good at vs. smart domain experts are in questions that are *resolvable* or *short-term*. I tend to think that the main differences are along the axes of *domain-specificity* and *complexity*, though these are of course correlated with the other axes. Superforecasters are selected for being relatively good at short-term, often geopolitical questions.
As I’ve written previously: It varies based on the question/domain how much domain expertise matters, but ultimately I expect reasonable domain experts to make better forecasts than reasonable generalists in many domains.
There’s an extreme here where e.g. forecasting what the best chess move is obviously better done by chess experts rather than superforecasters.
So if we think of a spectrum from geopolitics to chess, it’s very unclear to me where things like long-term AI forecasts land.
This intuition seems to be consistent with the lack of quality existing evidence described in Arb’s report (which debunked the “superforecasters beat intelligence experts without classified information” claim!).
Thanks, I hadn’t looked through those before.
https://goodjudgment.io/AI/Question_4_High-Impact_Failures.html jumped out to me:
Seems… obviously crazy?
Strongly agree, especially with your latter point 3. Fwiw, I used to work at Metaculus. Copy-pasting something I wrote after I left (which was in response to a Slack post about this study):
I am a superforecaster, and I endorse this message.
(Edit: To clarify, I am not a GJ Superforecaster®, just a professional forecaster with a comparable track record.)
I think these are fair criticisms of the “defer to superforecasters” view (which I share), and I think you helped me clarify some of my views here (thanks!), but I feel like it’s missing a few things. The best case for it, in my view, goes something like this:
The world is very hard to predict, and expertise is often overrated in complex domains.
Your first argument—that superforecasters lack domain-specific track records—doesn’t carry much weight if the relevant forecasting questions require broad expertise across regulation, diffusion dynamics, and technical capabilities simultaneously. No one has a verified track record across all of these, and “domain expert” here often just means “has strong inside views in a complicated area.”
On the selection effect: I don’t buy the strong version of your second claim. The fact that there’s low or no signal in non-verifiable domains (ie. philosophy) doesn’t really vindicate inside views—it weakens both. Any somewhat independent signal aggregated across multiple actors is probably better than a single inside view, even an expert one.
On track record: The benchmark underperformance is a real update against superforecasters in the AI case, but the question is how large that update should be. My framing: superforecasters are the prior; evidence of their underperformance updates you toward domain experts with better track records—and yes, I do think it updates toward people like yourself, Ryan, Eli, Daniel, and Peter Wilderford, who have been more right. But by how much? That’s the crux, and I’d genuinely like to see someone work through the math. A few people being more accurate than superforecasters on a hard problem doesn’t automatically license large updates toward their broader worldviews—we should be asking what reference class of questions they outperformed on, and whether that tracks the specific claims we care about. I’d also note that knowing how benchmarks saturate is less relevant to AI risk than you seem to think—the revenue point is stronger.
I’d also push back on a common error I see: people often conclude that “no clear expert → my inside view gets more weight.” This is probably true at the margin, but massively overrated in practice.
On your argument that object-level reasoning obsoletes base rates: This is somewhat circular. You have inside views about what it means to reason well about AI progress, and superforecasters disagree. You’re partially bootstrapping from your own beliefs to dismiss theirs.
On inside views and group epistemics: I agree that deference cascades are bad, but the fix isn’t “everyone uses their inside view”—it’s that people should be clearer about what’s inside vs. outside view reasoning (I agree this is complicated and maybe idealistic, but I don’t think the default for rationalists here should be to take the inside view of the community). I’m also skeptical that inside-view reasoning escapes the groupthink problem. Epistemic bubbles shape which counterarguments you seek, what your priors are, which information you weight. The AI safety/rationalist community isn’t immune to this.
I do think people should build inside views on AI—and the move of not doing so because it’s “not relevant to my field” is more often cope than a principled stance. But I’m genuinely uncertain about what the right policy is after you’ve built one. Surely the answer isn’t just “act on it fully”—the outside view still has to do some work. One practical resolution: argue on inside view, but take actions that at least partially reflect outside-view uncertainty.
A real remaining question: in non-verifiable domains, who counts as an expert? This is, I think, just an open and hard problem.
Happy to hear counters.
Oops, “object level reasoning obsoletes base rates” is not what I was trying to argue… my view is that the action is mostly in selecting the right base rate, i.e. that AI is more analogous to a new species than a normal technology.
Also I don’t agree that it’s circular. I think one of the correct reasons to defer to someone is them making correct arguments (as evaluated by my inside view), and that doesn’t apply. I definitely agree that I’m bootstrapping from my views to dismiss theirs. Now, there might be other reasons to defer to someone (for example, the other reasons I gave above), but I was arguing specifically against reason #3 above here.