no one in the industry seems to think they are writing their own death warrant
What led you to believe this? Plenty of people working at the top labs have very high p(doom) (>80%). Several of them comment on LessWrong. We have a survey of the broader industry as well. Even the people running the top 3 labs (Sam Altman, Dario Amodei, and Demis Hassabis) all think it’s likely enough that it’s worth dedicating a significant percentage of their organizational resources to researching alignment.
OpenAI is not solely focused on alignment; they allocate only 20% of their resources to superalignment research. What about the remaining 80%? This often goes undiscussed, yet it arguably represents the largest budget allocated to improving capabilities. Why not devote 100% of resources to building an aligned AGI? Such an approach would likely yield the best economic returns, as people are more likely to use a trustworthy AI, and governments would also be more inclined to promote its adoption.
Even the people running the top 3 labs (Sam Altman, Dario Amodei, and Demis Hassabis) all think it’s likely enough that it’s worth dedicating a significant percentage of their organizational resources to researching alignment
So they think that alignment is possible, and P(DOOM) is not high given alignment. That’s not actually agreeing with EY.
What led you to believe this? Plenty of people working at the top labs have very high p(doom) (>80%)
Well, not enough to tip the averages:-
In an above question, participants’ credence in >>>“extremely bad” outcomes of HLMI have median 5% and mean 14%
Also, see:-
As written this makes it look like everyone except Eliezer is <=50%, which isn’t true; I’m just having trouble thinking of other doomers who are both famous enough that you would have heard of them, and have publicly given a specific number
I wouldn’t want to give an “official organizational probability distribution”, but I think collectively we average out to something closer to “a uniform prior over possibilities” without that much evidence thus far updating us from there. Basically, there are plausible stories and intuitions pointing in lots of directions, and no real empirical evidence which bears on it thus far.
(Obviously, within the company, there’s a wide range of views. Some people are very pessimistic. Others are optimistic. We debate this quite a bit internally, and I think that’s really positive! But I think there’s a broad consensus to take the entire range seriously, including the very pessimistic ones.)
People who work in alignment, as opposed to capabilities, aren’t examples of people who are writing their own death warrants.
If someone switched from capability to alignment, that would be evidence that technical insight into capability leads to belief in AI Doom . But the people you have named have always been on the alignment side of the fence. That leads to the “theologians mostly believe in God” problem.
What led you to believe this? Plenty of people working at the top labs have very high p(doom) (>80%). Several of them comment on LessWrong. We have a survey of the broader industry as well. Even the people running the top 3 labs (Sam Altman, Dario Amodei, and Demis Hassabis) all think it’s likely enough that it’s worth dedicating a significant percentage of their organizational resources to researching alignment.
OpenAI is not solely focused on alignment; they allocate only 20% of their resources to superalignment research. What about the remaining 80%? This often goes undiscussed, yet it arguably represents the largest budget allocated to improving capabilities. Why not devote 100% of resources to building an aligned AGI? Such an approach would likely yield the best economic returns, as people are more likely to use a trustworthy AI, and governments would also be more inclined to promote its adoption.
So they think that alignment is possible, and P(DOOM) is not high given alignment. That’s not actually agreeing with EY.
Well, not enough to tip the averages:-
Also, see:-
From Scott’s https://astralcodexten.substack.com/p/why-i-am-not-as-much-of-a-doomer
Please refer back to your original claim:
You haven’t named anyone who is.
evhub was at 80% about a year ago (currently at Anthropic, interned at OpenAI).
Daniel Kokotajlo was at 65% ~2 years ago; I think that number’s gone up since then.
Quite a few other people at Anthropic also have pessimistic views, according to Chris Olah:
The Deepmind alignment team probably has at least a couple people who think the odds are bad, (p(doom) > 50%) given the way Vika buckets the team, combined with the distribution of views reflected by DeepMind alignment team opinions on AGI ruin arguments.
People who work in alignment, as opposed to capabilities, aren’t examples of people who are writing their own death warrants.
If someone switched from capability to alignment, that would be evidence that technical insight into capability leads to belief in AI Doom . But the people you have named have always been on the alignment side of the fence. That leads to the “theologians mostly believe in God” problem.