Doom Debates host here. I think this post makes the case against “P(Doom)” well, and I’ll grant this:
In a hypothetical world where most people agree — as maybe most of the people reading this post do — that the probability of imminent human extinction from artificial superintelligence is something in the ballpark of 10–90%, I would totally share the view that P(Doom) is just a “dumb meme”, and that finessing whether the better answer is 30% or 70% would be a waste of a discussion/debate.
But it turns out that a large fraction of people (e.g. >1/3 of Doom Debates guests) don’t agree with that 10–90% ballpark range. They either derail on some Bayesian epistemology 101 aspect of the question, or they claim that the probability is some crazy (IMO) value like 0.1%. Furthermore, I find that their policy arguments are often downstream of whether or not their P(Doom) is in the galaxy of saneness.
One striking example is this debate between Max Tegmark and Dean Ball, which IMO became much clearer when they answered that their P(Doom)s are >90% and <0.1%, respectively. AFAICT, the difference in their policy positions is well explained as being downstream of their P(Doom)s.
Asking people their P(Doom) may only yield about 1 bit of useful information — answering in the 10-90% range, vs. expressing a lot of confidence one way or the other — but it’s a high-order bit, a starting point for triangulating guests’ positions in the high-dimensional AI x-risk belief space, which is the mission of Doom Debates. It’s for that reason (and because of the fitness of the meme) that I’ve made “What’s Your P(Doom)” a recurring segment and tagline of the show.
Right, but I personally am not aware of a way someone could have a sane belief-formation process and still conclude that the probability of extinction from ASI in the next few decades is only 0.1%. That’s what I mean by it being an insane probability to assign.
I may be wrong in the domain of reasoning processes as they apply to this question—I even give a 10% chance to that claim—but I still think my view is expressed reasonably clearly by the “0.1% is a crazy value” phrasing.