One source of “the problem” seems to be a disguised version of unbounded payoffs.
Mugger: I can give you any finite amount of utility.
Victim: I find that highly unlikely.
Mugger: How unlikely?
Victim: 1/(really big number)
Mugger: Well, if you give me $1, I’ll give you (really big number)^2 times the utility of one dollar. Then your expected utility is positive, so you should give me the money.
The problem here is that whatever probability you give, the Mugger can always just make a better promise. Trying to assign “I can give you any finite amount of utility” a fixed non-zero probability is equivalent to assigning “I can give you an infinite amount of utility” a fixed non-zero probability. It’s sneaking an infinity in through the back door, so to speak.
It’s also very hard for any decision theory to deal with the problem “Name any rational number, and you get that much utility.” That’s because there is no largest rational number; no matter what number you name, there is another number that it is better to name. We can even come up with a version that even someone with a bounded utility function can be stumped by; “Name any rational number less than ten, and you get that much utility.” 9.9 is dominated by 9.99, which is dominated by 9.999, and so on. As long as you’re being asked to choose from a set that doesn’t contain its least upper bound, every choice is strictly dominated by some other choice. Even if all the numbers involved are finite, being given an infinite number of options can be enough to give decision theories the fits.
It’s sneaking an infinity in through the back door, so to speak.
Yes, this is precisely my own thinking—in order to give any assessment of the probability of the mugger delivering on any deal, you are in effect giving an assessment on an infinite number of deals (from 0 to infinity), and if you assign a non-zero probability to all of them (no matter how low), then you wind up with nonsensical results.
Giving the probability beforehand looks even worse if you ignore the deal aspect and simply ask what is the probability that anything the mugger says would be true? (Since this includes as a subset any promises to deliver utils.) Since he could make statements about turing machines or Chaitin’s Omega etc., now you’re into areas of intractable or undecidable questions!
As it happens, 2 or 3 days ago I emailed Bostrom about this. There was a followup paper to Bostrom’s “Pascal’s Mugging”, also published in Analysis, by a Baumann, who likewise rejected the prior probability, but Baumann didn’t have a good argument against it but to say that any such probability is ‘implausible’. Showing how infinities and undecidability get smuggled into the mugging shores up Baumann’s dismissal.
But once we’ve dismissed the prior probability, we still need to do something once the mugger has made a specific offer. If our probability doesn’t shrink at least as quickly as his offer increases, then we can still be mugged; if it shrinks exactly as quickly or even more quickly, we need to justify our specific shrinkage rate. And that is the perplexity: how fast do we shrink, and why?
(We want the Right theory & justification, not just one that is modeled after fallible humans or ad hocly makes the mugger go away. That is what I am asking for in the toplevel comment.)
Interesting thoughts on the mugger. But you still need a theory able to deal with it, not just an understanding of the problems.
For the second part, you can get a good decision theory for the “Name any rational number less than ten, and you get that much utility,” by giving you a certain fraction of negutility for each digit of your definition; there comes a time when the time wasted adding extra ’9’s dwarfs the gain in utility. See Tolstoy’s story How Much Land Does a Man Need for a traditional literary take on this problem.
The “Name any rational number, and you get that much utility” problem is more tricky, and would be a version of the “it is rational to spend infinity in hell” problem. Basically if your action (staying in hell; or specifying your utility) give you more ultimate utility than you lose by doing so, you will spend eternity doing your utility-losing action, and never cash in on your gained utility.
A thought on Pascal’s Mugging:
One source of “the problem” seems to be a disguised version of unbounded payoffs.
Mugger: I can give you any finite amount of utility.
Victim: I find that highly unlikely.
Mugger: How unlikely?
Victim: 1/(really big number)
Mugger: Well, if you give me $1, I’ll give you (really big number)^2 times the utility of one dollar. Then your expected utility is positive, so you should give me the money.
The problem here is that whatever probability you give, the Mugger can always just make a better promise. Trying to assign “I can give you any finite amount of utility” a fixed non-zero probability is equivalent to assigning “I can give you an infinite amount of utility” a fixed non-zero probability. It’s sneaking an infinity in through the back door, so to speak.
It’s also very hard for any decision theory to deal with the problem “Name any rational number, and you get that much utility.” That’s because there is no largest rational number; no matter what number you name, there is another number that it is better to name. We can even come up with a version that even someone with a bounded utility function can be stumped by; “Name any rational number less than ten, and you get that much utility.” 9.9 is dominated by 9.99, which is dominated by 9.999, and so on. As long as you’re being asked to choose from a set that doesn’t contain its least upper bound, every choice is strictly dominated by some other choice. Even if all the numbers involved are finite, being given an infinite number of options can be enough to give decision theories the fits.
Yes, this is precisely my own thinking—in order to give any assessment of the probability of the mugger delivering on any deal, you are in effect giving an assessment on an infinite number of deals (from 0 to infinity), and if you assign a non-zero probability to all of them (no matter how low), then you wind up with nonsensical results.
Giving the probability beforehand looks even worse if you ignore the deal aspect and simply ask what is the probability that anything the mugger says would be true? (Since this includes as a subset any promises to deliver utils.) Since he could make statements about turing machines or Chaitin’s Omega etc., now you’re into areas of intractable or undecidable questions!
As it happens, 2 or 3 days ago I emailed Bostrom about this. There was a followup paper to Bostrom’s “Pascal’s Mugging”, also published in Analysis, by a Baumann, who likewise rejected the prior probability, but Baumann didn’t have a good argument against it but to say that any such probability is ‘implausible’. Showing how infinities and undecidability get smuggled into the mugging shores up Baumann’s dismissal.
But once we’ve dismissed the prior probability, we still need to do something once the mugger has made a specific offer. If our probability doesn’t shrink at least as quickly as his offer increases, then we can still be mugged; if it shrinks exactly as quickly or even more quickly, we need to justify our specific shrinkage rate. And that is the perplexity: how fast do we shrink, and why?
(We want the Right theory & justification, not just one that is modeled after fallible humans or ad hocly makes the mugger go away. That is what I am asking for in the toplevel comment.)
Interesting thoughts on the mugger. But you still need a theory able to deal with it, not just an understanding of the problems.
For the second part, you can get a good decision theory for the “Name any rational number less than ten, and you get that much utility,” by giving you a certain fraction of negutility for each digit of your definition; there comes a time when the time wasted adding extra ’9’s dwarfs the gain in utility. See Tolstoy’s story How Much Land Does a Man Need for a traditional literary take on this problem.
The “Name any rational number, and you get that much utility” problem is more tricky, and would be a version of the “it is rational to spend infinity in hell” problem. Basically if your action (staying in hell; or specifying your utility) give you more ultimate utility than you lose by doing so, you will spend eternity doing your utility-losing action, and never cash in on your gained utility.
All I want for Christmas is an arbitrarily large chunk of utility.
Do you maybe see a problem with this concept?