But we don’t yet to have a clear, mathematically precise way to explain why we should also respond negatively to “give me a dollar, and I’ll use my Matrix Lord powers to save 3^^^3 lives.”
Yes, we do: bounded utility functions work just fine without any mathematical difficulties, and seem to map well to the psychological mechanisms that produce our intuitions. Objections to them are more philosophical and person-dependent.
The problem with Pascal’s mugging doesn’t lie merely in the particular hoped-for payoff, it’s that in extreme combinations of small chance/large payoff, the complexity of certain hypotheses doesn’t seem sufficient to adequately (as per our intuitions) penalize said hypotheses.
If we are going to be invoking intuition, then we should be careful about using examples with many extraneous intuition-provoking factors, and in thinking about how the intuitions are formed.
For example, handing over $1 to a literal Pascal’s Mugger, a guy who asks for the money out of your wallet in exchange for magic outputs, after trying and failing to mug you with a gun (which he found he forgot at home), is clearly less likely to get a big finite payoff than other uses of the money. The guy is claiming two things: 1) large payoffs (in things like life-years or dollars, not utility, which depends on your psychology) are physically possible 2) conditional on 1, the payoffs are more likely from paying him than other uses of money. Realistic amounts of evidence won’t be enough to neutralize 1), but would easily neutralize 2).
Heuristics which tell you not to pay off the mugger are right, even for total utilitarians.
Moreover, many of our intuitions look to be heuristics trained with past predictive success and delivery of individual rewards in one’s lifetime. If you save 1000 lives, trillions of person-seconds, you will not get billions of times the reinforcement you would get from eating a chocolate bar. You may get a ‘warm glow’ and some social prestige for success, but this will be a reward of ordinary scale in your reinforcement system, not enough to overcome astronomically low probabilities. So learned intuitions will tend to move you away from what would be good deals for an aggregative utilitarian, since they are bad deals in terms of discounted status and sex and chocolate.
Peter Singer argues that we should then discount those intuitions trained for non-moral purposes. Robin Hanson might argue that morality is overrated relative to our nonmoral desires. But it is worth attending to the processes that train intuitions, and figuring out which criteria one endorses.
Yes, we do: bounded utility functions work just fine without any mathematical difficulties
And so does speed prior.
Realistic amounts of evidence won’t be enough to neutralize 1), but would easily neutralize 2).
Yes. I have an example of why the intuition “but anyone can do that” is absolutely spot on. You give money to this mugger (and similar muggers), then another mugger shows up, and noticing doubt in your eyes, displays a big glowing text in front of you which says, “yes, i really have powers outside the matrix”. Except you haven’t got the money. Because you were being completely insane, by the medical definition of the term—your actions were not linked to reality in any way, and you failed to consider the utility of potential actions that are linked to reality (e.g. keep the money, give to a guy that displays the glowing text).
The intuition is that sane actions should be supported by evidence, whereas actions based purely on how you happened to assign priors, are insane. (And it is utterly ridiculous to say that low probability is a necessary part of Pascal’s wager, because as a matter of fact, probability must be high enough.) . I have a suspicion that this intuition reflects the fact that generally, actions conditional on evidence, have higher utility than any actions not conditional on evidence.
Yes, we do: bounded utility functions work just fine without any mathematical difficulties, and seem to map well to the psychological mechanisms that produce our intuitions. Objections to them are more philosophical and person-dependent.
Such as, for example, the fact that killing 3^^^^^^3 people shouldn’t be OK because there’s still 3^^^3 people left and my happiness meter is maxed out anyway.
Sorry, I might be just blinded by the technical language, but I’m not seeing why that link invalidates my comment. Could you maybe pull a quote, or even clarify?
Such as, for example, the fact that killing 3^^^^^^3 people shouldn’t be OK because there’s still 3^^^3 people left and my happiness meter is maxed out anyway.
E.g. the example above suggests something like a utility function of the form “utility equals the amount of quantity A for A<S, otherwise utility is equal to S” which rejects free-lunch increases in happy-years. But it’s easy to formulate a bounded utility function that takes such improvements, without being fanatical in the tradeoffs made.
Trivially, it’s easy to give a bounded utility function that always prefers a higher finite quantity of A but still converges, although eventually the preferences involved have to become very weak cardinally. A function with such a term on human happiness would not reject an otherwise “free lunch”. You never “max out,” just become willing to take smaller risks for incremental gains.
Less trivially, one can include terms like those in the bullet-pointed lists at the linked discussion, mapping to features that human brains distinguish and care about enough to make tempting counterexamples: “but if we don’t account for X, then you wouldn’t exert modest effort to get X!” Terms for relative achievement, e.g. the proportion (or adjusted proportion) of potential good (under some scheme of counterfactuals) achieved, neutralize an especially wide range of purported counterexamples.
E.g. the example above suggests something like a utility function of the form “utility equals the amount of quantity A for A<S, otherwise utility is equal to S” which rejects free-lunch increases in happy-years. But it’s easy to formulate a bounded utility function that takes such improvements, without being fanatical in the tradeoffs made.
… it is? Maybe I’m misusing the term “bounded utility function”. Could you elaborate on this?
Yes, I think you are misusing the term. It’s the utility that’s bounded, not the inputs. Say that U=1-(1/(X^2) and 0 when X=0, and X is the quantity of some good. Then utility is bounded between 0 and 1, but increasing X from 3^^^3 to 3^^^3+1 or 4^^^^4 will still (exceedingly slightly) increase utility. It just won’t take risks for small increases in utility. However, terms in the bounded utility function can give weight to large numbers, to relative achievement, to effort, and all the other things mentioned in the discussion I linked, so that one takes risks for those.
Bounded utility functions still seem to cause problems when uncertainty is involved. For example, consider the aforementioned utility function U(n) = 1 - (1 / (n^2)), and let n equal the number of agents living good lives. Using this function, the utility of a 1 in 1 chance of there being 10 agents living good lives equals 1 - (1 / (10^2)) = 0.99, and the utility of a 9 in 10 chance of 3^^^3 agents living good lives and a 1 in 10 chance of no agents living good lives roughly equals 0.1 0 + 0.9 1 = 0.9. Thus, in this situation the agent would be willing to kill (3^^^3) − 10 agents in order to prevent a 0.1 chance of everyone dying, which doesn’t seem right at all. You could modify the utility function, but I think this issue would still to exist to some extent.
Yes, we do: bounded utility functions work just fine without any mathematical difficulties, and seem to map well to the psychological mechanisms that produce our intuitions. Objections to them are more philosophical and person-dependent.
If we are going to be invoking intuition, then we should be careful about using examples with many extraneous intuition-provoking factors, and in thinking about how the intuitions are formed.
For example, handing over $1 to a literal Pascal’s Mugger, a guy who asks for the money out of your wallet in exchange for magic outputs, after trying and failing to mug you with a gun (which he found he forgot at home), is clearly less likely to get a big finite payoff than other uses of the money. The guy is claiming two things: 1) large payoffs (in things like life-years or dollars, not utility, which depends on your psychology) are physically possible 2) conditional on 1, the payoffs are more likely from paying him than other uses of money. Realistic amounts of evidence won’t be enough to neutralize 1), but would easily neutralize 2).
Heuristics which tell you not to pay off the mugger are right, even for total utilitarians.
Moreover, many of our intuitions look to be heuristics trained with past predictive success and delivery of individual rewards in one’s lifetime. If you save 1000 lives, trillions of person-seconds, you will not get billions of times the reinforcement you would get from eating a chocolate bar. You may get a ‘warm glow’ and some social prestige for success, but this will be a reward of ordinary scale in your reinforcement system, not enough to overcome astronomically low probabilities. So learned intuitions will tend to move you away from what would be good deals for an aggregative utilitarian, since they are bad deals in terms of discounted status and sex and chocolate.
Peter Singer argues that we should then discount those intuitions trained for non-moral purposes. Robin Hanson might argue that morality is overrated relative to our nonmoral desires. But it is worth attending to the processes that train intuitions, and figuring out which criteria one endorses.
And so does speed prior.
Yes. I have an example of why the intuition “but anyone can do that” is absolutely spot on. You give money to this mugger (and similar muggers), then another mugger shows up, and noticing doubt in your eyes, displays a big glowing text in front of you which says, “yes, i really have powers outside the matrix”. Except you haven’t got the money. Because you were being completely insane, by the medical definition of the term—your actions were not linked to reality in any way, and you failed to consider the utility of potential actions that are linked to reality (e.g. keep the money, give to a guy that displays the glowing text).
The intuition is that sane actions should be supported by evidence, whereas actions based purely on how you happened to assign priors, are insane. (And it is utterly ridiculous to say that low probability is a necessary part of Pascal’s wager, because as a matter of fact, probability must be high enough.) . I have a suspicion that this intuition reflects the fact that generally, actions conditional on evidence, have higher utility than any actions not conditional on evidence.
Such as, for example, the fact that killing 3^^^^^^3 people shouldn’t be OK because there’s still 3^^^3 people left and my happiness meter is maxed out anyway.
Self-consistent isn’t the same as moral.
Bounded utility functions can represent more than your comment suggests, depending on what terms are included. See this discussion.
Sorry, I might be just blinded by the technical language, but I’m not seeing why that link invalidates my comment. Could you maybe pull a quote, or even clarify?
E.g. the example above suggests something like a utility function of the form “utility equals the amount of quantity A for A<S, otherwise utility is equal to S” which rejects free-lunch increases in happy-years. But it’s easy to formulate a bounded utility function that takes such improvements, without being fanatical in the tradeoffs made.
Trivially, it’s easy to give a bounded utility function that always prefers a higher finite quantity of A but still converges, although eventually the preferences involved have to become very weak cardinally. A function with such a term on human happiness would not reject an otherwise “free lunch”. You never “max out,” just become willing to take smaller risks for incremental gains.
Less trivially, one can include terms like those in the bullet-pointed lists at the linked discussion, mapping to features that human brains distinguish and care about enough to make tempting counterexamples: “but if we don’t account for X, then you wouldn’t exert modest effort to get X!” Terms for relative achievement, e.g. the proportion (or adjusted proportion) of potential good (under some scheme of counterfactuals) achieved, neutralize an especially wide range of purported counterexamples.
… it is? Maybe I’m misusing the term “bounded utility function”. Could you elaborate on this?
Yes, I think you are misusing the term. It’s the utility that’s bounded, not the inputs. Say that U=1-(1/(X^2) and 0 when X=0, and X is the quantity of some good. Then utility is bounded between 0 and 1, but increasing X from 3^^^3 to 3^^^3+1 or 4^^^^4 will still (exceedingly slightly) increase utility. It just won’t take risks for small increases in utility. However, terms in the bounded utility function can give weight to large numbers, to relative achievement, to effort, and all the other things mentioned in the discussion I linked, so that one takes risks for those.
Bounded utility functions still seem to cause problems when uncertainty is involved. For example, consider the aforementioned utility function U(n) = 1 - (1 / (n^2)), and let n equal the number of agents living good lives. Using this function, the utility of a 1 in 1 chance of there being 10 agents living good lives equals 1 - (1 / (10^2)) = 0.99, and the utility of a 9 in 10 chance of 3^^^3 agents living good lives and a 1 in 10 chance of no agents living good lives roughly equals 0.1 0 + 0.9 1 = 0.9. Thus, in this situation the agent would be willing to kill (3^^^3) − 10 agents in order to prevent a 0.1 chance of everyone dying, which doesn’t seem right at all. You could modify the utility function, but I think this issue would still to exist to some extent.
Ah, OK, I was thinking of a bounded utility function as one with a “cutoff point”, yes. You’re absolutely right.