The basic problem with this kind of argument is that you are taking the math too seriously. Probability theory leads to absurdities if you assume that you do not have a probability of 0 or 1 for “the trillionth digit of pi is greater than 5.” In reality normal people will neither be certain that is true nor certain it is false. In other words, probability is a formalism of degrees of belief, and it is an imperfect formalism, not a perfect one.
If we consider the actual matter at hand, rather than the imperfect formalism, we actually have bounded utility. So we do not care about very low probability threats, including the one in your example. But although we have bounded utility, we are not infinitely certain of the fact that our probability is bounded. Thus we do not assign a probability of zero to “we have unbounded utility.” Nonetheless, it would be a misuse of a flawed formalism to conclude that we have to act on the possibility of the infinite negative utility. In reality, we act based on our limited knowledge of our bounded utility, and assume the threat is worthless.
The basic problem with this kind of argument is that you are taking the math too seriously. Probability theory leads to absurdities if you assume that you do not have a probability of 0 or 1 for “the trillionth digit of pi is greater than 5.” In reality normal people will neither be certain that is true nor certain it is false. In other words, probability is a formalism of degrees of belief, and it is an imperfect formalism, not a perfect one.
If we consider the actual matter at hand, rather than the imperfect formalism, we actually have bounded utility. So we do not care about very low probability threats, including the one in your example. But although we have bounded utility, we are not infinitely certain of the fact that our probability is bounded. Thus we do not assign a probability of zero to “we have unbounded utility.” Nonetheless, it would be a misuse of a flawed formalism to conclude that we have to act on the possibility of the infinite negative utility. In reality, we act based on our limited knowledge of our bounded utility, and assume the threat is worthless.