If one attempts to do calculations taking all permutations of Pascal’s mugging into account, one gets ∞ − ∞ as the result of all one’s expected utility calculations.
We have no idea how to do expected utility calculations in these kind of situations. Furthermore, even if the AI figured out some way, e.g., using some form of renormalization, we have to reason to believe the result would at all resemble our preferences.
If one attempts to do calculations taking all permutations of Pascal’s mugging into account, one gets ∞ − ∞ as the result of all one’s expected utility calculations.
What are the consequences of that?
We have no idea how to do expected utility calculations in these kind of situations. Furthermore, even if the AI figured out some way, e.g., using some form of renormalization, we have to reason to believe the result would at all resemble our preferences.