Many of these issues arise from some combination of allowing unbounded utilities, and assuming utility is linear. Where problems seem to arise, this is the place to fix them. It is much easier to incorporate fixes into our utility function (which is already extremely complicated, and poorly understood) than it is to incorporate them into the rules of reasoning or the rules of evidence, which are comparatively simple, and built upon math rather than on psychology.
Bounded utility solves Pascal’s Mugging and Torture vs Dust Specs straightforwardly. You choose some numbers to represent “max goodness” and “max badness”; really good and bad things approach these bounds asymptotically; and when you meet Pascal’s mugger, you take “max badness”, multiply it by an extremely tiny value, and get a very tiny value.
Quantum suicide is also a utility function issue, but not the same one. If your utility function only cares about average utility over the worlds in which you’re still alive, then you should commit quantum suicide. But revealed preferences indicate that people care about all worlds, and philosophy seems to indicate that they should care about all worlds, so quantum suicide is wrong.
I actually have a different idea on a possible approach to this Problem. I tried to get a discussion about it started in the decision theory mailing group, but it sort of died, and I’m not sure it’s safe to post in public.
I actually have a different idea on a possible approach to this Problem. I tried to get a discussion about it started in the decision theory mailing group, but it sort of died, and I’m not sure it’s safe to post in public.
Don’t be silly; it’s completely safe.
Q: Under what circumstances is an agent with utility function U isomorphic to one with utility function U’, where if U equals negative infinity U’=-1, if U is any finite real number U’=0, and if U equals positive infinity U’=1?
The original agent, with a utility function that sometimes kicks out infinities, is undefined; those infinities can’t be compared, and they propagate into the expected utility of actions so that actions can’t be compared either. The replacement agent is defined, but only has a three-valued utility function, which means it can’t express many preferences.
You get the equivalence you want if: your utilities lie in a totally ordered field extension of R, infinity is a constant greater than all elements of R, the utility of pure outcomes are restricted to be either R or +- infinity, and the relationship between utility and decision is in a certain generic position (so that the probability of every outcome changes whenever you change your decision, and these changes are never arranged to exactly make the utilities cancel out).
You choose some numbers to represent “max goodness” and “max badness”; really good and bad things approach these bounds asymptotically; and when you meet Pascal’s mugger, you take “max badness”, multiply it by an extremely tiny value, and get a very tiny value.
Even if you had solid evidence that your utility function was bounded, there would still be a small probability that your utility function was unbounded or bounded at a much higher level than you presumed. Pascal’s mugger simply has to increase his threat to compensate for your confidence in low-level utility bounding.
Even if you had solid evidence that your utility function was bounded, there would still be a small probability that your utility function was unbounded or bounded at a much higher level than you presumed. Pascal’s mugger simply has to increase his threat to compensate for your confidence in low-level utility bounding.
You can expand out any logical uncertainty about your utility function to get another utility function, and that is what must be bounded. This requires that the weighted average of the candidate utility functions converges to some (possibly higher) bound. But this is not difficult to achieve; and if they diverge, then you never really had a bounded utility function in the first place.
Another approach is to calibrate my estimation of probabilities such that my prior probability for the claim that you have the power to create X units of disutility decreases the greater X is. That is, if I reliably conclude that P(X) ⇐ P(Y)/2 when X is twice the threat of Y, then increasing the mugger’s threat won’t make me more likely to concede to it.
Many of these issues arise from some combination of allowing unbounded utilities, and assuming utility is linear. Where problems seem to arise, this is the place to fix them. It is much easier to incorporate fixes into our utility function (which is already extremely complicated, and poorly understood) than it is to incorporate them into the rules of reasoning or the rules of evidence, which are comparatively simple, and built upon math rather than on psychology.
Bounded utility solves Pascal’s Mugging and Torture vs Dust Specs straightforwardly. You choose some numbers to represent “max goodness” and “max badness”; really good and bad things approach these bounds asymptotically; and when you meet Pascal’s mugger, you take “max badness”, multiply it by an extremely tiny value, and get a very tiny value.
Quantum suicide is also a utility function issue, but not the same one. If your utility function only cares about average utility over the worlds in which you’re still alive, then you should commit quantum suicide. But revealed preferences indicate that people care about all worlds, and philosophy seems to indicate that they should care about all worlds, so quantum suicide is wrong.
I actually have a different idea on a possible approach to this Problem. I tried to get a discussion about it started in the decision theory mailing group, but it sort of died, and I’m not sure it’s safe to post in public.
Don’t be silly; it’s completely safe.
The original agent, with a utility function that sometimes kicks out infinities, is undefined; those infinities can’t be compared, and they propagate into the expected utility of actions so that actions can’t be compared either. The replacement agent is defined, but only has a three-valued utility function, which means it can’t express many preferences.
Always, independently of what kind of agent and what kind of infinities you use?
You get the equivalence you want if: your utilities lie in a totally ordered field extension of R, infinity is a constant greater than all elements of R, the utility of pure outcomes are restricted to be either R or +- infinity, and the relationship between utility and decision is in a certain generic position (so that the probability of every outcome changes whenever you change your decision, and these changes are never arranged to exactly make the utilities cancel out).
I’m not sure I understood all of that, but the pieces I did sound likely to be true about CEV or a papperclipper to me. Am I missing something?
Even if you had solid evidence that your utility function was bounded, there would still be a small probability that your utility function was unbounded or bounded at a much higher level than you presumed. Pascal’s mugger simply has to increase his threat to compensate for your confidence in low-level utility bounding.
You can expand out any logical uncertainty about your utility function to get another utility function, and that is what must be bounded. This requires that the weighted average of the candidate utility functions converges to some (possibly higher) bound. But this is not difficult to achieve; and if they diverge, then you never really had a bounded utility function in the first place.
Another approach is to calibrate my estimation of probabilities such that my prior probability for the claim that you have the power to create X units of disutility decreases the greater X is. That is, if I reliably conclude that P(X) ⇐ P(Y)/2 when X is twice the threat of Y, then increasing the mugger’s threat won’t make me more likely to concede to it.