I think I agree with basically everything you say in the post, but I think there’s a further important point against [working on AI risk] = [getting Pascal’s mugged] that isn’t discussed in your post. The point I have in mind is that the Pascal’s mugging worry is also clearly defeated if there are many actions such that they add up to a significant change in p(doom), even if each individual action contributes only a tiny change. The rest of my comment makes this point in more detail.
Consider your dark alley example again, but with the following modification:
Suppose now that the stranger gives you the option to take each of a bajillion different actions which cost you on the order of giving away your wallet, each of which you take to add probability 1/bajillion to 10 bajillion people going to heaven.[1] In particular, if you take all the actions together, you make it so 10 bajillion people just certainly go to heaven, and if you take none of the actions, all 10 bajillion people certainly go to hell.
I think that now it’s clear that (supposing you are fully altruistic, which I think we’re supposing here anyway), it is massively better to be taking all the bajillion actions — you’re just suffering a deterministic cost of much less than a bajillion heaven vs hell differences for a deterministic gain of 10 bajillion heaven vs hell differences. If one has a Pascal’s mugging worry that is telling one to not take each individual action (because it’s giving one 10 bajillion utils with probability only 1/bajillion just like in your original example) and so telling one to not take any of the actions (with everyone ending up in hell), then this worry is just dumb, at least in this case where one has the option to take many actions that add up to a macroscopic change.[2]
To bring this back to AI safety:
Consider the problem of assigning resources to AI safety work from society’s POV. Say society has the option to pay people 100k USD salaries each and reduce risk by . I think it’s abundantly clear that this would be a very good joint thing to do — it’s each person reducing their consumption by on the order of for a reduction in the probability they die[3].
A caveat: there’s some difference between humanity collectively assigning a person’s worth of additional resources to AI safety vs each individual’s decision to work on AI safety. Still, it’s clear that 10000 altruists would want to be in a world where they are collectively reducing p(doom) by than in a world where they aren’t. And I think this sort of consideration should be compelling to altruists even if the probabilities involved in individual decisions were Pascalian and they generally had some worries about getting Pascal’s mugged. (That said, I agree with you that the probabilities actually aren’t Pascalian.)
To state the same point another way:
One can split anything into many small pieces that look Pascalian (like, what’s the probability that making each next small improvement to a research project averts doom?). If one gets concerned about getting Pascal’s mugged when probabilities get small, one should track when many individual small probabilities are actually adding up to something big, and then not be concerned about getting Pascal’s mugged in those cases.
if we want to make it concrete what these costly actions are, maybe imagine that you’re agreeing to the stranger magically getting the wallets of each of a bajillion other people
I’m aware that this can be extended to an argument for agreeing to be Pascal’s mugged even in the case where you have only a single action available, but I think the case where you actually have many actions available that add up to a significant change in total is more clear than this extension, so I think what I’m stating should count as a separate argument against AI safety being a Pascal’s mugging.
I think I agree with basically everything you say in the post, but I think there’s a further important point against [working on AI risk] = [getting Pascal’s mugged] that isn’t discussed in your post. The point I have in mind is that the Pascal’s mugging worry is also clearly defeated if there are many actions such that they add up to a significant change in p(doom), even if each individual action contributes only a tiny change. The rest of my comment makes this point in more detail.
Consider your dark alley example again, but with the following modification:
Suppose now that the stranger gives you the option to take each of a bajillion different actions which cost you on the order of giving away your wallet, each of which you take to add probability 1/bajillion to 10 bajillion people going to heaven. [1] In particular, if you take all the actions together, you make it so 10 bajillion people just certainly go to heaven, and if you take none of the actions, all 10 bajillion people certainly go to hell.
I think that now it’s clear that (supposing you are fully altruistic, which I think we’re supposing here anyway), it is massively better to be taking all the bajillion actions — you’re just suffering a deterministic cost of much less than a bajillion heaven vs hell differences for a deterministic gain of 10 bajillion heaven vs hell differences. If one has a Pascal’s mugging worry that is telling one to not take each individual action (because it’s giving one 10 bajillion utils with probability only 1/bajillion just like in your original example) and so telling one to not take any of the actions (with everyone ending up in hell), then this worry is just dumb, at least in this case where one has the option to take many actions that add up to a macroscopic change. [2]
To bring this back to AI safety:
Consider the problem of assigning resources to AI safety work from society’s POV. Say society has the option to pay
A caveat: there’s some difference between humanity collectively assigning a person’s worth of additional resources to AI safety vs each individual’s decision to work on AI safety. Still, it’s clear that 10000 altruists would want to be in a world where they are collectively reducing p(doom) by
To state the same point another way:
One can split anything into many small pieces that look Pascalian (like, what’s the probability that making each next small improvement to a research project averts doom?). If one gets concerned about getting Pascal’s mugged when probabilities get small, one should track when many individual small probabilities are actually adding up to something big, and then not be concerned about getting Pascal’s mugged in those cases.
if we want to make it concrete what these costly actions are, maybe imagine that you’re agreeing to the stranger magically getting the wallets of each of a bajillion other people
I’m aware that this can be extended to an argument for agreeing to be Pascal’s mugged even in the case where you have only a single action available, but I think the case where you actually have many actions available that add up to a significant change in total is more clear than this extension, so I think what I’m stating should count as a separate argument against AI safety being a Pascal’s mugging.
and for other good stuff like there being a grand human future