I guess it seems pretty weird to me that superforecasters would do that much worse than prediction markets without some selection or bias, but I’ll mark it down as a reasonable alternative hypothesis. (“Actually superforecasting just generalizes really poorly to this admittedly special domain, and random superforecasters do way worse in it than prediction markets by default.”)
Eliezer Yudkowsky
Karma: 157,403
It cannot be answered that simply to the Earthlings, because if you answer “Because I don’t expect that to actually work or help”, some of them and especially the more evil ones will pounce in reply, “Aha, so you’re not replying, ‘I’d never do that because it would be wrong and against the law’, what a terrible person you must be!”
Super upvoted.
With that said, why is the optimal amount of woo not zero?
Also I think nonaccomodationist vegans have tended to be among the crazier people, so maybe you want enough vegetables for the accommodationists but also beef from moderately less tortured cows.
I just saw one recently on the EA forum to the effect that EAs who shortened their timelines only after chatGPT had the intelligence of a houseplant.
Somebody asked if people got credit for <30 year timelines posted in 2025. I replied that this only demonstrated more intelligence than a potted plant.
If you do not understand how this is drastically different from the thing you said I said, ask an LLM to explain it to you; they’re now okay at LSAT-style questions if provided sufficient context.
In reply to your larger question, being very polite about the house burning down wasn’t working. Possibly being less polite doesn’t work either, of course, but it takes less time. In any case, as several commenters have noted, the main plan is to have people who aren’t me do the talking to those sorts of audiences. As several other commenters have noted, there’s a plausible benefit to having one person say it straight. As further commenters have noted, I’m tired, so you don’t really have an option of continuing to hear from a polite Eliezer; I’d just stop talking instead.
Noted as a possible error on my part.
I looked at “AI 2027” as a title and shook my head about how that was sacrificing credibility come 2027 on the altar of pretending to be a prophet and picking up some short-term gains at the expense of more cooperative actors. I didn’t bother pushing back because I didn’t expect that to have any effect. I have been yelling at people to shut up about trading their stupid little timelines as if they were astrological signs for as long as that’s been a practice (it has now been replaced by trading made-up numbers for p(doom)).
When somebody at least pretending to humility says, “Well, I think this here estimator is the best thing we have for anchoring a median estimate”, and I stroll over and proclaim, “Well I think that’s invalid”, I do think there is a certain justice in them demanding of me, “Well, would you at least like to say then in what direction my expectation seems to you to be predictably mistaken?”
If you can get that or 2050 equally well off yelling “Biological Anchoring”, why not admit that the intuition comes first and then you hunt around for parameters you like? This doesn’t sound like good methodology to me.
Is your take “Use these different parameters and you get AGI in 2028 with the current methods”?
I think OpenPhil was guided by Cotra’s estimate and promoted that estimate. If they’d labeled it: “Epistemic status: Obviously wrong but maybe somebody builds on it someday” then it would have had a different impact and probably not one I found objectionable.
Separately, I can’t imagine how you could build something not-BS on that foundation and if people are using it to advocate for short timelines then I probably regard that argument as BS and invalid as well.
Will MacAskill could serve as exemplar. More broadly I’m thinking of people who might have called themselves ‘longtermists’ or who hybridized Bostrom with Peter Singer.
I again don’t consider this a helpful thing to say on a sinking ship when somebody is trying to organize passengers getting to the lifeboats.
Especially if your definition of “AI takeover” is such as to include lots of good possibilities as well as bad ones; maybe the iceberg rockets your ship to the destination sooner and provides all the passengers with free iced drinks, who can say?
You can do better by saying “I don’t know” than by saying a bunch of wrong stuff. My long reply to Cotra was, “You don’t know, I don’t know, your premises are clearly false, and if you insist on my being Bayesian and providing a direction of predictable error when I claim predictable error then fine your timelines are too long.”
People ask me questions. I answer them honestly, not least because I don’t have the skill to say “I’m not answering that” without it sending some completely different set of messages. Saying a bunch of stuff in private without giving anyone a chance to respond to what I’m guessing about them is deontologically weighed-against by my rules, though not forbidden depending on circumstances. I do not do this in hopes any good thing results, but then acts with good consequences are few and far between in any case, these days.
Why, that’s my job too! But it’s a very different job depending on whether you consider it an indispensable requirement to have people coming away with a roughly accurate picture of reality, or if your job is to be an entertainer.
I think if they sponsored Cotra’s work and cited it, this reflects badly on them. More on them than on Cotra, really; I am not a fan of the theory that you blame the people who were selected to have an opinion or incentivised to have an opinion, so much as the people who did the selection and incentivization. See https://www.lesswrong.com/posts/ax695frGJEzGxFBK4/biology-inspired-agi-timelines-the-trick-that-never-works, which I think stands out as clearly correct in retrospect, for why their analysis was obviously wrong at the time. And I did in that case take the trouble to explain why their whole complicated analysis was bogus, and my model is that this clearly-correct-in-retrospect critique had roughly zero impact or effect on OpenPhil; and that is what I expected and predicted in advance, which is why I did not spend more effort trying to redeem an organization I modeled as irredeemably broken.
I expect it’s a combination of selection effects and researchers knowing implicitly where their bread is buttered; I have no particular estimate of the relative share of these effects, except that they are jointly sufficient that, eg, a granter can hire what advertises itself as a group of superforecasters, and get back 1% probability on AI IMO gold by 2025.
Well, there sure is a simple story for how it looked from outside. What’s the complicated real truth that you only get to know about from the inside, where everything is, like, not ignorantly handwaved off as incredibly standard bureaucratic organizational dynamics of grantees telling the grantmaker what it wants to hear?
If you imagine the very serious person wearing the expensive suit saying, “But of course we must prepare for cases where the ship sinks sooner and there is a possibility of some passengers drowning”, whether or not this is Very Exculpatory depends on the counterfactual for what happens if the guy is not there. I think OpenPhil imagines that if they are not there, even fewer people take MIRI seriously. To me this is not clear and it looks like the only thing that broke the logjam was ChatGPT, after which the weight and momentum of OpenPhil views was strongly net negative.
One issue among others is that the kind of work you end up funding when the funding bureaucrats go to the funding-seekers and say, “Well, we mostly think this is many years out and won’t kill everyone, but, you know, just in case, we thought we’d fund you to write papers about it” tends to be papers that make net negative contributions.
Noted. I think you are overlooking some of the dynamics of the weird dance that a bureaucratic institution does around pretending to be daring while their opinions are in fact insufficiently extreme; eg, why when OpenPhil ran a “change our views” contest, they predictably awarded all of the money to critiques arguing for longer timelines and lower risk, even though reality was in the opposite direction of their opinions from that. Just like OpenPhil predictably gave all the money to “we need two Stalins” critiques of them in the contest, OpenPhil might have managed to communicate to the ‘superforecasters’ or their institutions that the demanded apparent disagreement with OpenPhil’s overt forecast was in the “we need two Stalins” direction of longer timelines and lower risks.
Or to rephrase: If I can look at the organizational dynamics and see it as obvious in advance that OpenPhil’s “challenge our worldviews” contest would award all the money to people arguing for longer timelines and lower risk, (despite reality lying in the opposite direction, according to those people’s own later updates, even); then maybe the people advertising themselves as producing superforecaster reports, can successfully read OpenPhil’s mind about what direction of superforecaster disagreement is being secretly demanded.
But, sure, fair enough, I should also update somewhat in favor of the average superforecaster being even worse at AI than OpenPhil and them delivering an honest terrible report. I guess it’s just surprising to me because I would’ve expected the key maneuver here to be saying “I dunno” and not throwing around extreme opinions or numbers, and I would’ve thought superforecasters able to do that better than OpenPhil… but eh, idk, maybe they just straight up couldn’t tell the difference between the usually good rule “nothing ever happens” and “AGI in particular never happens”, and also didn’t know themselves for overconfident or incompetent at being able to apply the rule.
If so, it would speak correspondingly poorly of those EAs who stood around gesturing at the superforecasters and saying, “Why believe MIRI when you could believe these great certified experts?”