Just to be clear: are we saying that a factor of 3^^^3 is a Pascal’s mugging, but a factor of 10^30 isn’t? (In Holden’s comment above, one example in the context of Pascal’s mugging-type problems is a factor of 10^10, even as that’s on the order of the population of the Earth.)
I think any reasonable person hearing “8 lives saved per dollar donated” would file it with Pascal’s mugging (which is Eliezer’s term, but the concept is pretty simple and comprehensible even to someone thinking of less extreme probabilities than Eliezer posits; e.g. Holden, above).
In the linked thread, Rain special-pleads that the topic requires very large numbers to talk about, but jsteinhardt counters that that doesn’t make humans any better at reasoning about tiny probabilities multiplied by large numbers. jsteinhardt also points out that just because you can multiply a small number by a large number doesn’t mean the product actually makes any sense at all.
“Just to be clear: are we saying that a factor of 3^^^3 is a Pascal’s mugging, but a factor of 10^30 isn’t?”
No. The problem with Pascal’s mugging doesn’t lie merely in the particular hoped-for payoff, it’s that in extreme combinations of small chance/large payoff, the complexity of certain hypotheses doesn’t seem sufficient to adequately (as per our intuitions) penalize said hypotheses.
If I said “give me a dollar, and I’ll use my Matrix Lord powers to have three dollars appear in your wallet”, someone can simply respond that the chances of me being a Matrix Lord is less than one in three, so the expected payoff is less than the cost. But we don’t yet to have a clear, mathematically precise way to explain why we should also respond negatively to “give me a dollar, and I’ll use my Matrix Lord powers to save 3^^^3 lives.”, even though our intuition says we should (and in this case we trust our intuition).
To put it in brief: Pascal’s Mugging is a interesting problem regarding decision theory which LessWrongers should be hoping to solve (I have an idea towards that direction, which I’m writing a discusion post about, but I’d need mathematicians to tell me if it potentially leads to anything); not just a catchphrase you can use to bash someone else’s calculations when their intuitions differs from yours.
But we don’t yet to have a clear, mathematically precise way to explain why we should also respond negatively to “give me a dollar, and I’ll use my Matrix Lord powers to save 3^^^3 lives.”
Yes, we do: bounded utility functions work just fine without any mathematical difficulties, and seem to map well to the psychological mechanisms that produce our intuitions. Objections to them are more philosophical and person-dependent.
The problem with Pascal’s mugging doesn’t lie merely in the particular hoped-for payoff, it’s that in extreme combinations of small chance/large payoff, the complexity of certain hypotheses doesn’t seem sufficient to adequately (as per our intuitions) penalize said hypotheses.
If we are going to be invoking intuition, then we should be careful about using examples with many extraneous intuition-provoking factors, and in thinking about how the intuitions are formed.
For example, handing over $1 to a literal Pascal’s Mugger, a guy who asks for the money out of your wallet in exchange for magic outputs, after trying and failing to mug you with a gun (which he found he forgot at home), is clearly less likely to get a big finite payoff than other uses of the money. The guy is claiming two things: 1) large payoffs (in things like life-years or dollars, not utility, which depends on your psychology) are physically possible 2) conditional on 1, the payoffs are more likely from paying him than other uses of money. Realistic amounts of evidence won’t be enough to neutralize 1), but would easily neutralize 2).
Heuristics which tell you not to pay off the mugger are right, even for total utilitarians.
Moreover, many of our intuitions look to be heuristics trained with past predictive success and delivery of individual rewards in one’s lifetime. If you save 1000 lives, trillions of person-seconds, you will not get billions of times the reinforcement you would get from eating a chocolate bar. You may get a ‘warm glow’ and some social prestige for success, but this will be a reward of ordinary scale in your reinforcement system, not enough to overcome astronomically low probabilities. So learned intuitions will tend to move you away from what would be good deals for an aggregative utilitarian, since they are bad deals in terms of discounted status and sex and chocolate.
Peter Singer argues that we should then discount those intuitions trained for non-moral purposes. Robin Hanson might argue that morality is overrated relative to our nonmoral desires. But it is worth attending to the processes that train intuitions, and figuring out which criteria one endorses.
Yes, we do: bounded utility functions work just fine without any mathematical difficulties
And so does speed prior.
Realistic amounts of evidence won’t be enough to neutralize 1), but would easily neutralize 2).
Yes. I have an example of why the intuition “but anyone can do that” is absolutely spot on. You give money to this mugger (and similar muggers), then another mugger shows up, and noticing doubt in your eyes, displays a big glowing text in front of you which says, “yes, i really have powers outside the matrix”. Except you haven’t got the money. Because you were being completely insane, by the medical definition of the term—your actions were not linked to reality in any way, and you failed to consider the utility of potential actions that are linked to reality (e.g. keep the money, give to a guy that displays the glowing text).
The intuition is that sane actions should be supported by evidence, whereas actions based purely on how you happened to assign priors, are insane. (And it is utterly ridiculous to say that low probability is a necessary part of Pascal’s wager, because as a matter of fact, probability must be high enough.) . I have a suspicion that this intuition reflects the fact that generally, actions conditional on evidence, have higher utility than any actions not conditional on evidence.
Yes, we do: bounded utility functions work just fine without any mathematical difficulties, and seem to map well to the psychological mechanisms that produce our intuitions. Objections to them are more philosophical and person-dependent.
Such as, for example, the fact that killing 3^^^^^^3 people shouldn’t be OK because there’s still 3^^^3 people left and my happiness meter is maxed out anyway.
Sorry, I might be just blinded by the technical language, but I’m not seeing why that link invalidates my comment. Could you maybe pull a quote, or even clarify?
Such as, for example, the fact that killing 3^^^^^^3 people shouldn’t be OK because there’s still 3^^^3 people left and my happiness meter is maxed out anyway.
E.g. the example above suggests something like a utility function of the form “utility equals the amount of quantity A for A<S, otherwise utility is equal to S” which rejects free-lunch increases in happy-years. But it’s easy to formulate a bounded utility function that takes such improvements, without being fanatical in the tradeoffs made.
Trivially, it’s easy to give a bounded utility function that always prefers a higher finite quantity of A but still converges, although eventually the preferences involved have to become very weak cardinally. A function with such a term on human happiness would not reject an otherwise “free lunch”. You never “max out,” just become willing to take smaller risks for incremental gains.
Less trivially, one can include terms like those in the bullet-pointed lists at the linked discussion, mapping to features that human brains distinguish and care about enough to make tempting counterexamples: “but if we don’t account for X, then you wouldn’t exert modest effort to get X!” Terms for relative achievement, e.g. the proportion (or adjusted proportion) of potential good (under some scheme of counterfactuals) achieved, neutralize an especially wide range of purported counterexamples.
E.g. the example above suggests something like a utility function of the form “utility equals the amount of quantity A for A<S, otherwise utility is equal to S” which rejects free-lunch increases in happy-years. But it’s easy to formulate a bounded utility function that takes such improvements, without being fanatical in the tradeoffs made.
… it is? Maybe I’m misusing the term “bounded utility function”. Could you elaborate on this?
Yes, I think you are misusing the term. It’s the utility that’s bounded, not the inputs. Say that U=1-(1/(X^2) and 0 when X=0, and X is the quantity of some good. Then utility is bounded between 0 and 1, but increasing X from 3^^^3 to 3^^^3+1 or 4^^^^4 will still (exceedingly slightly) increase utility. It just won’t take risks for small increases in utility. However, terms in the bounded utility function can give weight to large numbers, to relative achievement, to effort, and all the other things mentioned in the discussion I linked, so that one takes risks for those.
Bounded utility functions still seem to cause problems when uncertainty is involved. For example, consider the aforementioned utility function U(n) = 1 - (1 / (n^2)), and let n equal the number of agents living good lives. Using this function, the utility of a 1 in 1 chance of there being 10 agents living good lives equals 1 - (1 / (10^2)) = 0.99, and the utility of a 9 in 10 chance of 3^^^3 agents living good lives and a 1 in 10 chance of no agents living good lives roughly equals 0.1 0 + 0.9 1 = 0.9. Thus, in this situation the agent would be willing to kill (3^^^3) − 10 agents in order to prevent a 0.1 chance of everyone dying, which doesn’t seem right at all. You could modify the utility function, but I think this issue would still to exist to some extent.
To be really clear, the problem with Pascal’s Mugging is that even after eliminating infinity as a coherent scenario, any simplicity prior which defines simplicity strictly over computational complexity will apparently yield divergent returns for aggregative utility functions when summed over all probable scenarios, because the material size of possible scenarios grows much faster than their computational complexity (Busy Beaver function or just tetration).
The problem with Pascal’s Wager on the other hand is that it shuts down an ongoing conversation about plausibility by claiming that it doesn’t matter how small the probability is, thus averting a logically polite duty to provide evidence and engage with counterarguments.
To be really clear, the problem with Pascal’s Mugging is that even after eliminating infinity as a coherent scenario, any simplicity prior which defines simplicity strictly over computational complexity will apparently yield divergent returns for aggregative utility functions when summed over all probable scenarios, because the material size of possible scenarios grows much faster than their computational complexity (Busy Beaver function or just tetration).
That seems overly specific. There are many other ways in which priors assigned to highly speculative propositions may not be low enough, or when impact of other available actions on a highly speculative scenario be under-evaluated.
The problem with Pascal’s Wager on the other hand is that it shuts down an ongoing conversation about plausibility by claiming that it doesn’t matter how small the probability is, thus averting a logically polite duty to provide evidence and engage with counterarguments.
To me, Pascal’s Wager is defined by a speculative scenario for which there exist no evidence, which has high enough impact to result in actions which are not based on any evidence, despite the uncertainty towards speculative scenarios.
How THE HELL does the above (ok, I didn’t originally include the second quotation, but still) constitute confusion of Pascal’s Wager and Pascal’s Mugging, let alone “willful misinterpretation” ?
I certainly consider that if you multiply a very tiny probability by a huge payoff and then expect others to take your calculation seriously as a call to action, you’re being silly, however it’s labeled. Humans can’t even consider very tiny probabilities without privileging the hypothesis.
Note also that a crazy mugger could demand $10 or else 10^30 people outside the matrix will die, and then argue that you should rationally trust him 100% so the figure is 10^29 lives/$ , or argue that it is 90% certain that those people will die because he’s a bit uncertain about the danger in the alternate worlds, or the like. It’s not about the probability which mugger estimates, it’s about the probability that the typical payer estimates.
I will certainly admit that the precise label is not my true objection, and apologise if I have seemed to be arguing primarily over definitions (which is of course actually a terrible thing to do in general).
Just to be clear: are we saying that a factor of 3^^^3 is a Pascal’s mugging, but a factor of 10^30 isn’t? (In Holden’s comment above, one example in the context of Pascal’s mugging-type problems is a factor of 10^10, even as that’s on the order of the population of the Earth.)
I think any reasonable person hearing “8 lives saved per dollar donated” would file it with Pascal’s mugging (which is Eliezer’s term, but the concept is pretty simple and comprehensible even to someone thinking of less extreme probabilities than Eliezer posits; e.g. Holden, above).
In the linked thread, Rain special-pleads that the topic requires very large numbers to talk about, but jsteinhardt counters that that doesn’t make humans any better at reasoning about tiny probabilities multiplied by large numbers. jsteinhardt also points out that just because you can multiply a small number by a large number doesn’t mean the product actually makes any sense at all.
No. The problem with Pascal’s mugging doesn’t lie merely in the particular hoped-for payoff, it’s that in extreme combinations of small chance/large payoff, the complexity of certain hypotheses doesn’t seem sufficient to adequately (as per our intuitions) penalize said hypotheses.
If I said “give me a dollar, and I’ll use my Matrix Lord powers to have three dollars appear in your wallet”, someone can simply respond that the chances of me being a Matrix Lord is less than one in three, so the expected payoff is less than the cost. But we don’t yet to have a clear, mathematically precise way to explain why we should also respond negatively to “give me a dollar, and I’ll use my Matrix Lord powers to save 3^^^3 lives.”, even though our intuition says we should (and in this case we trust our intuition).
To put it in brief: Pascal’s Mugging is a interesting problem regarding decision theory which LessWrongers should be hoping to solve (I have an idea towards that direction, which I’m writing a discusion post about, but I’d need mathematicians to tell me if it potentially leads to anything); not just a catchphrase you can use to bash someone else’s calculations when their intuitions differs from yours.
Yes, we do: bounded utility functions work just fine without any mathematical difficulties, and seem to map well to the psychological mechanisms that produce our intuitions. Objections to them are more philosophical and person-dependent.
If we are going to be invoking intuition, then we should be careful about using examples with many extraneous intuition-provoking factors, and in thinking about how the intuitions are formed.
For example, handing over $1 to a literal Pascal’s Mugger, a guy who asks for the money out of your wallet in exchange for magic outputs, after trying and failing to mug you with a gun (which he found he forgot at home), is clearly less likely to get a big finite payoff than other uses of the money. The guy is claiming two things: 1) large payoffs (in things like life-years or dollars, not utility, which depends on your psychology) are physically possible 2) conditional on 1, the payoffs are more likely from paying him than other uses of money. Realistic amounts of evidence won’t be enough to neutralize 1), but would easily neutralize 2).
Heuristics which tell you not to pay off the mugger are right, even for total utilitarians.
Moreover, many of our intuitions look to be heuristics trained with past predictive success and delivery of individual rewards in one’s lifetime. If you save 1000 lives, trillions of person-seconds, you will not get billions of times the reinforcement you would get from eating a chocolate bar. You may get a ‘warm glow’ and some social prestige for success, but this will be a reward of ordinary scale in your reinforcement system, not enough to overcome astronomically low probabilities. So learned intuitions will tend to move you away from what would be good deals for an aggregative utilitarian, since they are bad deals in terms of discounted status and sex and chocolate.
Peter Singer argues that we should then discount those intuitions trained for non-moral purposes. Robin Hanson might argue that morality is overrated relative to our nonmoral desires. But it is worth attending to the processes that train intuitions, and figuring out which criteria one endorses.
And so does speed prior.
Yes. I have an example of why the intuition “but anyone can do that” is absolutely spot on. You give money to this mugger (and similar muggers), then another mugger shows up, and noticing doubt in your eyes, displays a big glowing text in front of you which says, “yes, i really have powers outside the matrix”. Except you haven’t got the money. Because you were being completely insane, by the medical definition of the term—your actions were not linked to reality in any way, and you failed to consider the utility of potential actions that are linked to reality (e.g. keep the money, give to a guy that displays the glowing text).
The intuition is that sane actions should be supported by evidence, whereas actions based purely on how you happened to assign priors, are insane. (And it is utterly ridiculous to say that low probability is a necessary part of Pascal’s wager, because as a matter of fact, probability must be high enough.) . I have a suspicion that this intuition reflects the fact that generally, actions conditional on evidence, have higher utility than any actions not conditional on evidence.
Such as, for example, the fact that killing 3^^^^^^3 people shouldn’t be OK because there’s still 3^^^3 people left and my happiness meter is maxed out anyway.
Self-consistent isn’t the same as moral.
Bounded utility functions can represent more than your comment suggests, depending on what terms are included. See this discussion.
Sorry, I might be just blinded by the technical language, but I’m not seeing why that link invalidates my comment. Could you maybe pull a quote, or even clarify?
E.g. the example above suggests something like a utility function of the form “utility equals the amount of quantity A for A<S, otherwise utility is equal to S” which rejects free-lunch increases in happy-years. But it’s easy to formulate a bounded utility function that takes such improvements, without being fanatical in the tradeoffs made.
Trivially, it’s easy to give a bounded utility function that always prefers a higher finite quantity of A but still converges, although eventually the preferences involved have to become very weak cardinally. A function with such a term on human happiness would not reject an otherwise “free lunch”. You never “max out,” just become willing to take smaller risks for incremental gains.
Less trivially, one can include terms like those in the bullet-pointed lists at the linked discussion, mapping to features that human brains distinguish and care about enough to make tempting counterexamples: “but if we don’t account for X, then you wouldn’t exert modest effort to get X!” Terms for relative achievement, e.g. the proportion (or adjusted proportion) of potential good (under some scheme of counterfactuals) achieved, neutralize an especially wide range of purported counterexamples.
… it is? Maybe I’m misusing the term “bounded utility function”. Could you elaborate on this?
Yes, I think you are misusing the term. It’s the utility that’s bounded, not the inputs. Say that U=1-(1/(X^2) and 0 when X=0, and X is the quantity of some good. Then utility is bounded between 0 and 1, but increasing X from 3^^^3 to 3^^^3+1 or 4^^^^4 will still (exceedingly slightly) increase utility. It just won’t take risks for small increases in utility. However, terms in the bounded utility function can give weight to large numbers, to relative achievement, to effort, and all the other things mentioned in the discussion I linked, so that one takes risks for those.
Bounded utility functions still seem to cause problems when uncertainty is involved. For example, consider the aforementioned utility function U(n) = 1 - (1 / (n^2)), and let n equal the number of agents living good lives. Using this function, the utility of a 1 in 1 chance of there being 10 agents living good lives equals 1 - (1 / (10^2)) = 0.99, and the utility of a 9 in 10 chance of 3^^^3 agents living good lives and a 1 in 10 chance of no agents living good lives roughly equals 0.1 0 + 0.9 1 = 0.9. Thus, in this situation the agent would be willing to kill (3^^^3) − 10 agents in order to prevent a 0.1 chance of everyone dying, which doesn’t seem right at all. You could modify the utility function, but I think this issue would still to exist to some extent.
Ah, OK, I was thinking of a bounded utility function as one with a “cutoff point”, yes. You’re absolutely right.
To be really clear, the problem with Pascal’s Mugging is that even after eliminating infinity as a coherent scenario, any simplicity prior which defines simplicity strictly over computational complexity will apparently yield divergent returns for aggregative utility functions when summed over all probable scenarios, because the material size of possible scenarios grows much faster than their computational complexity (Busy Beaver function or just tetration).
The problem with Pascal’s Wager on the other hand is that it shuts down an ongoing conversation about plausibility by claiming that it doesn’t matter how small the probability is, thus averting a logically polite duty to provide evidence and engage with counterarguments.
That seems overly specific. There are many other ways in which priors assigned to highly speculative propositions may not be low enough, or when impact of other available actions on a highly speculative scenario be under-evaluated.
To me, Pascal’s Wager is defined by a speculative scenario for which there exist no evidence, which has high enough impact to result in actions which are not based on any evidence, despite the uncertainty towards speculative scenarios.
How THE HELL does the above (ok, I didn’t originally include the second quotation, but still) constitute confusion of Pascal’s Wager and Pascal’s Mugging, let alone “willful misinterpretation” ?
I certainly consider that if you multiply a very tiny probability by a huge payoff and then expect others to take your calculation seriously as a call to action, you’re being silly, however it’s labeled. Humans can’t even consider very tiny probabilities without privileging the hypothesis.
Note also that a crazy mugger could demand $10 or else 10^30 people outside the matrix will die, and then argue that you should rationally trust him 100% so the figure is 10^29 lives/$ , or argue that it is 90% certain that those people will die because he’s a bit uncertain about the danger in the alternate worlds, or the like. It’s not about the probability which mugger estimates, it’s about the probability that the typical payer estimates.
PASCAL’S WAGER IS DEFINED BY LOW PROBABILITIES NOT BY LARGE PAYOFFS
PASCAL’S WAGER IS DEFINED BY LOW PROBABILITIES NOT BY LARGE PAYOFFS
PASCAL’S WAGER IS DEFINED BY LOW PROBABILITIES NOT BY LARGE PAYOFFS
I will certainly admit that the precise label is not my true objection, and apologise if I have seemed to be arguing primarily over definitions (which is of course actually a terrible thing to do in general).