So your question is whether (with added newline and capitalization for clarity):
any dissenting views from “AI in median >30 years” and “utter AI ruin <10%” (as expressed in the correct directions of shorter timelines and worse ruin chances; and as said before the ChatGPT moment), were permitted to exercise decision-making power over the flow of substantial amounts of funding;
OR if the weight of reputation and publicity of OpenPhil was at any point put behind promoting those dissenting viewpoints
Re the first part:
Open Phil decisions were strongly affected by whether they were good according to worldviews where “utter AI ruin” is >10% or timelines are <30 years. Many staff believed at the time that worlds with shorter timelines and higher misalignment risk were more tractable to intervene on, and so put additional focus on interventions targeting those worlds; many also believed that risk was >10% and that median timeline was <30 years. I’m not really sure how to operationalize this, but my sense is that the majority of their funding related to AI safety was targeted at scenarios with higher misalignment risk and shorter timelines than 10%/30 years.
As an example, see Some Background on Our Views Regarding Advanced Artificial Intelligence (2016), where Holden says that his belief that P(AGI before 2036) is above 10% “is important to my stance on the importance of potential risks from advanced artificial intelligence. If I did not hold it, this cause would probably still be a focus area of the Open Philanthropy Project, but holding this view is important to prioritize the cause as highly as we’re planning to.” So he’s clearly saying that the grantmaking strategy is strongly affected by wanting to target the sub-20-year timelines.
I’m not sure how to translate this into the language you use. Among other issues, it’s a little weird to talk about the relative influence of different credences over hypotheses, rather than the relative influence of different hypotheses. The “AI risk is >10% and <30 years” hypotheses had a lot of influence, but that could be true even if all the relevant staff had believed that AI risk is <10% and >30 years (if they’d also believed that those worlds were particularly leveraged to intervene on, as they do).
Lots of decisions were made that would not have been made given the decision procedure of “do whatever’s best assuming AI is in >30 years and risk is <10%”—I think that that decision procedure would have massively changed the AI safety stuff Open Phil did.
I think that this suffices to contradict your description of the situation—they explicitly made many of their decisions based on the possibility of shorter timelines than you described. I haven’t presented evidence here that something similar is true for their assessment of misalignment risk, but I also believe that to be the case.
If I persuaded you of the claims I wrote here (only some of which I backed up with evidence), would that be relevant to your overall stance?
All of this is made more complicated by the fact that Open Phil obviously is and was a large organization with many staff and other stakeholders, who believed different things and had different approaches to translating beliefs into decisions, and who have changed over time. So we can’t really talk about what “Open Phil believed” coherently.
Re the second part: I think the weight of reputation and publicity was put behind encouraging people to plan for the possibility of AI sooner than 30 years, as I noted above; this doesn’t contradict the statement you’ve made but IMO it is relevant to your broader point.
The technical advisors I have spoken with the most on this topic are close friends I’ve met through GiveWell and effective altruism: Dario Amodei, Chris Olah and Jacob Steinhardt. They are all relatively junior (as opposed to late-career) researchers; they do not constitute a representative sample of researchers; there are therefore risks in leaning too heavily on their thinking.[...]
There may turn out to be a few broadly applicable AI approaches that lead to rapid progress on an extremely wide variety of intellectual tasks. This intuition seems correlated with (though again, not the same as) an intuition that the human brain makes repeated use of a relatively small set of underlying algorithms, and that by applying the processes, with small modifications, in a variety of contexts, it generates a wide variety of different predictive models, which can end up looking like very different intellectual functions.
[..]Certain areas of AI and machine learning, particularly related to deep neural networks and other deep learning methods, have recently experienced rapid and impressive progress.
[...]Deep learning is a general approach to fitting predictive models to data that can lead to automated generation of extremely complex non-linear models. It seems to be, conceptually, a relatively simple and cross-domain approach to generating such models (though it requires complex computations and generates complex models, and hardware improvements of past decades have been a key factor in being able to employ it effectively). My impression is that the field is still very far away from exploring all the ways in which deep learning might be applied to challenges in AI.
[...]In my view, there is a live possibility that with further exploration of the implications and applications of deep learning – and perhaps a small number (1-3) of future breakthroughs comparable in scope and generality to deep learning – researchers will end up being able to achieve better-than-human performance in a large number of intellectual domains, sufficient to produce transformative AI.
[...]
But broadly speaking, based on these conversations, it seems to me that:
It is easy to imagine (though far from certain) that headway on a relatively small number of core problems could lead to AI systems equalling or surpassing human performance in a large number of domains.
The total number of core open problems is not clearly particularly large (though it is highly possible that there are many core problems that the participants simply haven’t thought of).
Many of the identified core open problems may turn out to have overlapping solutions. Many may turn out to be solved by continued extension and improvement of deep learning methods.
None appear that they will clearly require large numbers of major breakthroughs, large (decade-scale) amounts of trial and error, or further progress on directly studying the human brain. There are examples of outstanding technical problems, such as unsupervised learning, that could turn out to be very difficult, leading to a dramatic slowdown in progress in the near future, but it isn’t clear that we should confidently expect such a slowdown.
(As a random reference, I thought Joe’s paper about low AI takeover risk was silly at the time, and I think that most people working on grants motivated by AI risk at OP at the time had higher estimates of AI takeover risk. I also thought a lot of takes from the Oxford EAs were pretty silly and I found them frustrating at the time and think they look worse with hindsight. Obviously, many of my beliefs at many of these time periods also look silly in hindsight.)
If you imagine the very serious person wearing the expensive suit saying, “But of course we must prepare for cases where the ship sinks sooner and there is a possibility of some passengers drowning”, whether or not this is Very Exculpatory depends on the counterfactual for what happens if the guy is not there. I think OpenPhil imagines that if they are not there, even fewer people take MIRI seriously. To me this is not clear and it looks like the only thing that broke the logjam was ChatGPT, after which the weight and momentum of OpenPhil views was strongly net negative.
One issue among others is that the kind of work you end up funding when the funding bureaucrats go to the funding-seekers and say, “Well, we mostly think this is many years out and won’t kill everyone, but, you know, just in case, we thought we’d fund you to write papers about it” tends to be papers that make net negative contributions.
Okay, so it sounds like you’re saying that the claims I asserted aren’t cruxy for your claim you wanted contradicted?
I definitely don’t think that Open Phil thought of “have more people take MIRI seriously” as a core objective, and I imagine that opinions on whether “people take MIRI more seriously” is good would depend a lot on how you operationalize it.
I think that Open Phil proactively tried to take a bunch of actions based on the hypothesis that powerful AI would be developed within 20 years. I think the situation with the sinking ship is pretty disanalogous—I think you’d need to say that your guy in the expensive suit was also one of the main people who was proactively taking actions based on the hypothesis that the ship would sink faster.
I definitely don’t think that Open Phil thought of “have more people take MIRI seriously” as a core objective
FWIW I heard rumor they thought of the roughly opposite, “Have people think OpenPhil doesn’t take MIRI seriously”, as an objective. I heard a story that when OpenPhil staff went to academia to interview lots of academics about doing grantmaking in the field of AI, all the academics strongly dismissed MIRI as cranks and bad to associate with, and OpenPhil felt their credibility would be harmed by associating with MIRI.
This is consistent with (and somewhat supported by) the OpenPhil grant report to MIRI saying that they could’ve picked anywhere between $1.5M and $0.5M, and they picked the latter for signaling reasons.
I’m not sure I follow[1]. It’s not a perfect match for the opposite (“Have fewer people take MIRI seriously”) but it’s roughly/functionally in the opposite direction in terms of their funding choices and influence on the discourse.
You may be responding to an earlier of edit of mine, I somewhat substantially edited within ~5 mins of commenting, and then found you’d already replied.
(Yeah, I was responding to the earlier version. I meant that in some cases you might want to cause someone to be taken more seriously but not want to cause people to think you take them more seriously (or not want to make that salient, or to make people think that you want them to think you want it to be salient, or whatever). Those are just different objectives you might have.)
One issue among others is that the kind of work you end up funding when the funding bureaucrats go to the funding-seekers and say, “Well, we mostly think this is many years out and won’t kill everyone, but, you know, just in case, we thought we’d fund you to write papers about it” tends to be papers that make net negative contributions.
I think this is a pretty poor model of the attitudes of the relevant staff at the time. I also think your disparaging language here leads to your comments being worse descriptions of what was going on.
Well, there sure is a simple story for how it looked from outside. What’s the complicated real truth that you only get to know about from the inside, where everything is, like, not ignorantly handwaved off as incredibly standard bureaucratic organizational dynamics of grantees telling the grantmaker what it wants to hear?
One issue among others is that the kind of work you end up funding when the funding bureaucrats go to the funding-seekers and say, “Well, we mostly think this is many years out and won’t kill everyone, but, you know, just in case, we thought we’d fund you to write papers about it” tends to be papers that make net negative contributions.
Why does the attitude of the funding bureaucrats make the output of the (presumably earnestly motivated) researchers net-negative?
Is this mostly a selection effect where the people who end up getting funding are not earnest? Is the impact of the funding-signal stronger than the impact of the papers themselves? Is it that even though the researchers are earnest, there’s selection on which things they’re socially allowed to say and this distortion is bad enough that they would have been better off saying nothing?
I expect it’s a combination of selection effects and researchers knowing implicitly where their bread is buttered; I have no particular estimate of the relative share of these effects, except that they are jointly sufficient that, eg, a granter can hire what advertises itself as a group of superforecasters, and get back 1% probability on AI IMO gold by 2025.
That sounds wild to me, given that the superforecasters believed much less in fast AI progress (and in doom) than OpenPhil staff and the “subject matter experts” who the superforecasters could talk with.
Like, in 2020, bio anchors publicly predicted $1B training runs in 2025. In 2022, the superforecasters predicted that the largest training runs in 2024 would be $35M, in 2030 would be $100M, and in 2050 would be $300M.
(And for the IMO gold number in particular, if I had to guess what OP’s view was, I would base that on Paul’s 8%. Which is 3⁄4 of the way from 1% to your own 16%, in log-odds.)
If the superforecasters were biasing their views towards OP, then they should have been way more bullish. If OP’s process was selecting for forecasters who agreed more with their own views, they would’ve selected forecasters who were more bullish.
I think the simpler hypothesis is that the wider world, including superforecasters among them, massively underestimated 2020s AI progress.
(This is consistent with the fact that OP advisors got outsized investment returns by betting on faster AI progress than the markets expected. It’s also consistent with Jacob Steinhardt’s own attempt at commissioning forecasts, which also produced huge underestimates. I think this wasn’t funded by OP, though Jacob was an OP technical advisor at the time.)
Noted. I think you are overlooking some of the dynamics of the weird dance that a bureaucratic institution does around pretending to be daring while their opinions are in fact insufficiently extreme; eg, why when OpenPhil ran a “change our views” contest, they predictably awarded all of the money to critiques arguing for longer timelines and lower risk, even though reality was in the opposite direction of their opinions from that. Just like OpenPhil predictably gave all the money to “we need two Stalins” critiques of them in the contest, OpenPhil might have managed to communicate to the ‘superforecasters’ or their institutions that the demanded apparent disagreement with OpenPhil’s overt forecast was in the “we need two Stalins” direction of longer timelines and lower risks.
Or to rephrase: If I can look at the organizational dynamics and see it as obvious in advance that OpenPhil’s “challenge our worldviews” contest would award all the money to people arguing for longer timelines and lower risk, (despite reality lying in the opposite direction, according to those people’s own later updates, even); then maybe the people advertising themselves as producing superforecaster reports, can successfully read OpenPhil’s mind about what direction of superforecaster disagreement is being secretly demanded.
But, sure, fair enough, I should also update somewhat in favor of the average superforecaster being even worse at AI than OpenPhil and them delivering an honest terrible report. I guess it’s just surprising to me because I would’ve expected the key maneuver here to be saying “I dunno” and not throwing around extreme opinions or numbers, and I would’ve thought superforecasters able to do that better than OpenPhil… but eh, idk, maybe they just straight up couldn’t tell the difference between the usually good rule “nothing ever happens” and “AGI in particular never happens”, and also didn’t know themselves for overconfident or incompetent at being able to apply the rule.
If so, it would speak correspondingly poorly of those EAs who stood around gesturing at the superforecasters and saying, “Why believe MIRI when you could believe these great certified experts?”
(Quick flag that, if you have energy for more engagement, I’d most bid for a source for the claim “superforecasters assigned 1% to IMO gold by 2025”. As mentioned in my reply to your parallel comment.)
maybe the people advertising themselves as producing superforecaster reports, can successfully read OpenPhil’s mind about what direction of superforecaster disagreement is being secretly demanded
I agree that’s one possible hypothesis. It’s more complicated than “OP rewards agreement”, and I don’t currently see why I should assign a high prior to it. (Like, someone could also make a plausible-sounding argument for the opposite: that dysfunctional OP will of course want superforecasters to have more extreme views than OP itself, to provide cover and make OP’s own views look more moderate and reasonable by comparison.)
Combined with the evidence being pretty limited (I suppose (i) the XPT, and also (ii) one worldview critique contest that they wouldn’t have run if it wasn’t for FTX starting it and then crashing, and where my impression is they weren’t excited about the resulting entries), I’m not sold.
maybe they just straight up couldn’t tell the difference between the usually good rule “nothing ever happens” and “AGI in particular never happens”, and also didn’t know themselves for overconfident or incompetent at being able to apply the rule.
I think this is probably a lot of what’s going on.
My forecasts actually were funded by OP! I would guess that the main counterfactual change as a result of this was going with Hypermind over Good Judgement. It might be interesting to look at differences between those populations of forecasters—I would not model “super forecasters” as homogeneous and in retrospect the particular forecasters we got seemed not super good at AI questions, or else just weren’t trying hard enough. But I also worked with some very good, AI-focused forecasters as a sanity check and they were also surprised by progress as determined by pre-registered predictions.
Ah, thanks for clarifying! (I searched OP’s historical AI grants for ones that mentioned your name or UC Berkeley in nearby years and didn’t find anything that looked likely to cover the AI forecasting — I suppose I’ll put less stock in that kind of methodology going forward.)
I guess it seems pretty weird to me that superforecasters would do that much worse than prediction markets without some selection or bias, but I’ll mark it down as a reasonable alternative hypothesis. (“Actually superforecasting just generalizes really poorly to this admittedly special domain, and random superforecasters do way worse in it than prediction markets by default.”)
I think metaculus and (especially) manifold samples their users disproportionately from AI-risk concerned rationalists and EAs, and relatedly also from people who work in AI. So I’m not that surprised if their aggregated opinions on AI are better than superforecasters. (Although I was pretty surprised by how bad the superforecasters were on some of the questions, in particular the compute spend one.)
Actually, though: what were you referencing with your original claim? (I.e. “get back 1% probability on AI IMO gold by 2025”.) I assumed it was from the x-risk persuasion tournament. But page 627-628 says that the superforecasters’ 5th percentile for IMO gold was 2025. So they assigned at least 5% that the IMO would get beaten by 2025.
I wouldn’t treat competitive forecasters as a homogeneous group, but I also think basically everyone was surprised by the rate of progress on the MATH dataset. The main difference is that the better forecasters adjusted quickly after the first surprise and were mostly calibrated after.
So your question is whether (with added newline and capitalization for clarity):
Re the first part:
Open Phil decisions were strongly affected by whether they were good according to worldviews where “utter AI ruin” is >10% or timelines are <30 years. Many staff believed at the time that worlds with shorter timelines and higher misalignment risk were more tractable to intervene on, and so put additional focus on interventions targeting those worlds; many also believed that risk was >10% and that median timeline was <30 years. I’m not really sure how to operationalize this, but my sense is that the majority of their funding related to AI safety was targeted at scenarios with higher misalignment risk and shorter timelines than 10%/30 years.
As an example, see Some Background on Our Views Regarding Advanced Artificial Intelligence (2016), where Holden says that his belief that P(AGI before 2036) is above 10% “is important to my stance on the importance of potential risks from advanced artificial intelligence. If I did not hold it, this cause would probably still be a focus area of the Open Philanthropy Project, but holding this view is important to prioritize the cause as highly as we’re planning to.” So he’s clearly saying that the grantmaking strategy is strongly affected by wanting to target the sub-20-year timelines.
I’m not sure how to translate this into the language you use. Among other issues, it’s a little weird to talk about the relative influence of different credences over hypotheses, rather than the relative influence of different hypotheses. The “AI risk is >10% and <30 years” hypotheses had a lot of influence, but that could be true even if all the relevant staff had believed that AI risk is <10% and >30 years (if they’d also believed that those worlds were particularly leveraged to intervene on, as they do).
Lots of decisions were made that would not have been made given the decision procedure of “do whatever’s best assuming AI is in >30 years and risk is <10%”—I think that that decision procedure would have massively changed the AI safety stuff Open Phil did.
I think that this suffices to contradict your description of the situation—they explicitly made many of their decisions based on the possibility of shorter timelines than you described. I haven’t presented evidence here that something similar is true for their assessment of misalignment risk, but I also believe that to be the case.
If I persuaded you of the claims I wrote here (only some of which I backed up with evidence), would that be relevant to your overall stance?
All of this is made more complicated by the fact that Open Phil obviously is and was a large organization with many staff and other stakeholders, who believed different things and had different approaches to translating beliefs into decisions, and who have changed over time. So we can’t really talk about what “Open Phil believed” coherently.
Re the second part: I think the weight of reputation and publicity was put behind encouraging people to plan for the possibility of AI sooner than 30 years, as I noted above; this doesn’t contradict the statement you’ve made but IMO it is relevant to your broader point.
Section 2.2 in “Some Background...” looks IMO pretty prescient
(As a random reference, I thought Joe’s paper about low AI takeover risk was silly at the time, and I think that most people working on grants motivated by AI risk at OP at the time had higher estimates of AI takeover risk. I also thought a lot of takes from the Oxford EAs were pretty silly and I found them frustrating at the time and think they look worse with hindsight. Obviously, many of my beliefs at many of these time periods also look silly in hindsight.)
If you imagine the very serious person wearing the expensive suit saying, “But of course we must prepare for cases where the ship sinks sooner and there is a possibility of some passengers drowning”, whether or not this is Very Exculpatory depends on the counterfactual for what happens if the guy is not there. I think OpenPhil imagines that if they are not there, even fewer people take MIRI seriously. To me this is not clear and it looks like the only thing that broke the logjam was ChatGPT, after which the weight and momentum of OpenPhil views was strongly net negative.
One issue among others is that the kind of work you end up funding when the funding bureaucrats go to the funding-seekers and say, “Well, we mostly think this is many years out and won’t kill everyone, but, you know, just in case, we thought we’d fund you to write papers about it” tends to be papers that make net negative contributions.
Okay, so it sounds like you’re saying that the claims I asserted aren’t cruxy for your claim you wanted contradicted?
I definitely don’t think that Open Phil thought of “have more people take MIRI seriously” as a core objective, and I imagine that opinions on whether “people take MIRI more seriously” is good would depend a lot on how you operationalize it.
I think that Open Phil proactively tried to take a bunch of actions based on the hypothesis that powerful AI would be developed within 20 years. I think the situation with the sinking ship is pretty disanalogous—I think you’d need to say that your guy in the expensive suit was also one of the main people who was proactively taking actions based on the hypothesis that the ship would sink faster.
FWIW I heard rumor they thought of the roughly opposite, “Have people think OpenPhil doesn’t take MIRI seriously”, as an objective. I heard a story that when OpenPhil staff went to academia to interview lots of academics about doing grantmaking in the field of AI, all the academics strongly dismissed MIRI as cranks and bad to associate with, and OpenPhil felt their credibility would be harmed by associating with MIRI.
This is consistent with (and somewhat supported by) the OpenPhil grant report to MIRI saying that they could’ve picked anywhere between $1.5M and $0.5M, and they picked the latter for signaling reasons.
That’s not literally the opposite, that’s a different thing, obviously.
I’m not sure I follow[1]. It’s not a perfect match for the opposite (“Have fewer people take MIRI seriously”) but it’s roughly/functionally in the opposite direction in terms of their funding choices and influence on the discourse.
You may be responding to an earlier of edit of mine, I somewhat substantially edited within ~5 mins of commenting, and then found you’d already replied.
(Yeah, I was responding to the earlier version. I meant that in some cases you might want to cause someone to be taken more seriously but not want to cause people to think you take them more seriously (or not want to make that salient, or to make people think that you want them to think you want it to be salient, or whatever). Those are just different objectives you might have.)
I think this is a pretty poor model of the attitudes of the relevant staff at the time. I also think your disparaging language here leads to your comments being worse descriptions of what was going on.
Well, there sure is a simple story for how it looked from outside. What’s the complicated real truth that you only get to know about from the inside, where everything is, like, not ignorantly handwaved off as incredibly standard bureaucratic organizational dynamics of grantees telling the grantmaker what it wants to hear?
Why does the attitude of the funding bureaucrats make the output of the (presumably earnestly motivated) researchers net-negative?
Is this mostly a selection effect where the people who end up getting funding are not earnest? Is the impact of the funding-signal stronger than the impact of the papers themselves? Is it that even though the researchers are earnest, there’s selection on which things they’re socially allowed to say and this distortion is bad enough that they would have been better off saying nothing?
I expect it’s a combination of selection effects and researchers knowing implicitly where their bread is buttered; I have no particular estimate of the relative share of these effects, except that they are jointly sufficient that, eg, a granter can hire what advertises itself as a group of superforecasters, and get back 1% probability on AI IMO gold by 2025.
That sounds wild to me, given that the superforecasters believed much less in fast AI progress (and in doom) than OpenPhil staff and the “subject matter experts” who the superforecasters could talk with.
Like, in 2020, bio anchors publicly predicted $1B training runs in 2025. In 2022, the superforecasters predicted that the largest training runs in 2024 would be $35M, in 2030 would be $100M, and in 2050 would be $300M.
(And for the IMO gold number in particular, if I had to guess what OP’s view was, I would base that on Paul’s 8%. Which is 3⁄4 of the way from 1% to your own 16%, in log-odds.)
If the superforecasters were biasing their views towards OP, then they should have been way more bullish. If OP’s process was selecting for forecasters who agreed more with their own views, they would’ve selected forecasters who were more bullish.
I think the simpler hypothesis is that the wider world, including superforecasters among them, massively underestimated 2020s AI progress.
(This is consistent with the fact that OP advisors got outsized investment returns by betting on faster AI progress than the markets expected. It’s also consistent with Jacob Steinhardt’s own attempt at commissioning forecasts, which also produced huge underestimates. I think this wasn’t funded by OP, though Jacob was an OP technical advisor at the time.)
Noted. I think you are overlooking some of the dynamics of the weird dance that a bureaucratic institution does around pretending to be daring while their opinions are in fact insufficiently extreme; eg, why when OpenPhil ran a “change our views” contest, they predictably awarded all of the money to critiques arguing for longer timelines and lower risk, even though reality was in the opposite direction of their opinions from that. Just like OpenPhil predictably gave all the money to “we need two Stalins” critiques of them in the contest, OpenPhil might have managed to communicate to the ‘superforecasters’ or their institutions that the demanded apparent disagreement with OpenPhil’s overt forecast was in the “we need two Stalins” direction of longer timelines and lower risks.
Or to rephrase: If I can look at the organizational dynamics and see it as obvious in advance that OpenPhil’s “challenge our worldviews” contest would award all the money to people arguing for longer timelines and lower risk, (despite reality lying in the opposite direction, according to those people’s own later updates, even); then maybe the people advertising themselves as producing superforecaster reports, can successfully read OpenPhil’s mind about what direction of superforecaster disagreement is being secretly demanded.
But, sure, fair enough, I should also update somewhat in favor of the average superforecaster being even worse at AI than OpenPhil and them delivering an honest terrible report. I guess it’s just surprising to me because I would’ve expected the key maneuver here to be saying “I dunno” and not throwing around extreme opinions or numbers, and I would’ve thought superforecasters able to do that better than OpenPhil… but eh, idk, maybe they just straight up couldn’t tell the difference between the usually good rule “nothing ever happens” and “AGI in particular never happens”, and also didn’t know themselves for overconfident or incompetent at being able to apply the rule.
If so, it would speak correspondingly poorly of those EAs who stood around gesturing at the superforecasters and saying, “Why believe MIRI when you could believe these great certified experts?”
(Quick flag that, if you have energy for more engagement, I’d most bid for a source for the claim “superforecasters assigned 1% to IMO gold by 2025”. As mentioned in my reply to your parallel comment.)
I agree that’s one possible hypothesis. It’s more complicated than “OP rewards agreement”, and I don’t currently see why I should assign a high prior to it. (Like, someone could also make a plausible-sounding argument for the opposite: that dysfunctional OP will of course want superforecasters to have more extreme views than OP itself, to provide cover and make OP’s own views look more moderate and reasonable by comparison.)
Combined with the evidence being pretty limited (I suppose (i) the XPT, and also (ii) one worldview critique contest that they wouldn’t have run if it wasn’t for FTX starting it and then crashing, and where my impression is they weren’t excited about the resulting entries), I’m not sold.
I think this is probably a lot of what’s going on.
My forecasts actually were funded by OP! I would guess that the main counterfactual change as a result of this was going with Hypermind over Good Judgement. It might be interesting to look at differences between those populations of forecasters—I would not model “super forecasters” as homogeneous and in retrospect the particular forecasters we got seemed not super good at AI questions, or else just weren’t trying hard enough. But I also worked with some very good, AI-focused forecasters as a sanity check and they were also surprised by progress as determined by pre-registered predictions.
Ah, thanks for clarifying! (I searched OP’s historical AI grants for ones that mentioned your name or UC Berkeley in nearby years and didn’t find anything that looked likely to cover the AI forecasting — I suppose I’ll put less stock in that kind of methodology going forward.)
My guess would be that it’s because they paid Hypermind directly rather than making the grant to me.
I guess it seems pretty weird to me that superforecasters would do that much worse than prediction markets without some selection or bias, but I’ll mark it down as a reasonable alternative hypothesis. (“Actually superforecasting just generalizes really poorly to this admittedly special domain, and random superforecasters do way worse in it than prediction markets by default.”)
I think metaculus and (especially) manifold samples their users disproportionately from AI-risk concerned rationalists and EAs, and relatedly also from people who work in AI. So I’m not that surprised if their aggregated opinions on AI are better than superforecasters. (Although I was pretty surprised by how bad the superforecasters were on some of the questions, in particular the compute spend one.)
Actually, though: what were you referencing with your original claim? (I.e. “get back 1% probability on AI IMO gold by 2025”.) I assumed it was from the x-risk persuasion tournament. But page 627-628 says that the superforecasters’ 5th percentile for IMO gold was 2025. So they assigned at least 5% that the IMO would get beaten by 2025.
If you are interested, I did a detailed analysis of different groups of forecasters here: https://bounded-regret.ghost.io/scoring-ml-forecasts-for-2023/
I wouldn’t treat competitive forecasters as a homogeneous group, but I also think basically everyone was surprised by the rate of progress on the MATH dataset. The main difference is that the better forecasters adjusted quickly after the first surprise and were mostly calibrated after.