They either derail on some Bayesian epistemology 101 aspect of the question, or they claim that the probability is some crazy (IMO) value like 0.1%. Furthermore, I find that their policy arguments are often downstream of whether or not their P(Doom) is in the galaxy of saneness.
The property of “sanity” is not a property of beliefs, but of belief-formation processes.
Right, but I personally am not aware of a way someone could have a sane belief-formation process and still conclude that the probability of extinction from ASI in the next few decades is only 0.1%. That’s what I mean by it being an insane probability to assign.
I may be wrong in the domain of reasoning processes as they apply to this question—I even give a 10% chance to that claim—but I still think my view is expressed reasonably clearly by the “0.1% is a crazy value” phrasing.
There are two views I’m aware of that make people that confident in not-”doom”: they are either convinced that ASI’s creation will almost certainly have desirable consequences (i.e. the e/accs) or they discount the possibility of ASI in general. I think the first group is obviously overconfident and the second-group is still overconfident but not trivially so.
Specifically, the consensus of the latter group is that LLMs are not a viable path to ASI because, outside of specific areas like math and programming where correct answers can be verified, they can’t exceed the capabilities of a skilled human; that no other technology is likely to reach that point either (in part because a long AI winter would follow the failure of LLMs to take off); and that recursive self-improvement, if it is possible, would lead to diminishing returns. They do not seem all that confident in the benevolence of ASI if it were created.
I also think everyone is insane, but some people are still more insane than others.
The property of “sanity” is not a property of beliefs, but of belief-formation processes.
Right, but I personally am not aware of a way someone could have a sane belief-formation process and still conclude that the probability of extinction from ASI in the next few decades is only 0.1%. That’s what I mean by it being an insane probability to assign.
I may be wrong in the domain of reasoning processes as they apply to this question—I even give a 10% chance to that claim—but I still think my view is expressed reasonably clearly by the “0.1% is a crazy value” phrasing.
There are two views I’m aware of that make people that confident in not-”doom”: they are either convinced that ASI’s creation will almost certainly have desirable consequences (i.e. the e/accs) or they discount the possibility of ASI in general. I think the first group is obviously overconfident and the second-group is still overconfident but not trivially so.
Specifically, the consensus of the latter group is that LLMs are not a viable path to ASI because, outside of specific areas like math and programming where correct answers can be verified, they can’t exceed the capabilities of a skilled human; that no other technology is likely to reach that point either (in part because a long AI winter would follow the failure of LLMs to take off); and that recursive self-improvement, if it is possible, would lead to diminishing returns. They do not seem all that confident in the benevolence of ASI if it were created.
I also think everyone is insane, but some people are still more insane than others.