You can randomly accuse people that what they believe constitutes Pascal’s mugging, but that doesn’t make the accusation a valid argument, unless you show that it’s so.
There’s a very simple test to see if someone actually accepts Pascal’s mugging: Go to them and say “I’ll use my godlike hidden powers to increase your utility by 3^^^3 utilons if you hand over to me the complete contents of your bank account.”
Don’t just claim that something else they believe is the same as Pascal’s mugging or I might equally easily claim that someone buying health insurance is a victim of Pascal’s mugging.
Just to be clear: are we saying that a factor of 3^^^3 is a Pascal’s mugging, but a factor of 10^30 isn’t? (In Holden’s comment above, one example in the context of Pascal’s mugging-type problems is a factor of 10^10, even as that’s on the order of the population of the Earth.)
I think any reasonable person hearing “8 lives saved per dollar donated” would file it with Pascal’s mugging (which is Eliezer’s term, but the concept is pretty simple and comprehensible even to someone thinking of less extreme probabilities than Eliezer posits; e.g. Holden, above).
In the linked thread, Rain special-pleads that the topic requires very large numbers to talk about, but jsteinhardt counters that that doesn’t make humans any better at reasoning about tiny probabilities multiplied by large numbers. jsteinhardt also points out that just because you can multiply a small number by a large number doesn’t mean the product actually makes any sense at all.
“Just to be clear: are we saying that a factor of 3^^^3 is a Pascal’s mugging, but a factor of 10^30 isn’t?”
No. The problem with Pascal’s mugging doesn’t lie merely in the particular hoped-for payoff, it’s that in extreme combinations of small chance/large payoff, the complexity of certain hypotheses doesn’t seem sufficient to adequately (as per our intuitions) penalize said hypotheses.
If I said “give me a dollar, and I’ll use my Matrix Lord powers to have three dollars appear in your wallet”, someone can simply respond that the chances of me being a Matrix Lord is less than one in three, so the expected payoff is less than the cost. But we don’t yet to have a clear, mathematically precise way to explain why we should also respond negatively to “give me a dollar, and I’ll use my Matrix Lord powers to save 3^^^3 lives.”, even though our intuition says we should (and in this case we trust our intuition).
To put it in brief: Pascal’s Mugging is a interesting problem regarding decision theory which LessWrongers should be hoping to solve (I have an idea towards that direction, which I’m writing a discusion post about, but I’d need mathematicians to tell me if it potentially leads to anything); not just a catchphrase you can use to bash someone else’s calculations when their intuitions differs from yours.
But we don’t yet to have a clear, mathematically precise way to explain why we should also respond negatively to “give me a dollar, and I’ll use my Matrix Lord powers to save 3^^^3 lives.”
Yes, we do: bounded utility functions work just fine without any mathematical difficulties, and seem to map well to the psychological mechanisms that produce our intuitions. Objections to them are more philosophical and person-dependent.
The problem with Pascal’s mugging doesn’t lie merely in the particular hoped-for payoff, it’s that in extreme combinations of small chance/large payoff, the complexity of certain hypotheses doesn’t seem sufficient to adequately (as per our intuitions) penalize said hypotheses.
If we are going to be invoking intuition, then we should be careful about using examples with many extraneous intuition-provoking factors, and in thinking about how the intuitions are formed.
For example, handing over $1 to a literal Pascal’s Mugger, a guy who asks for the money out of your wallet in exchange for magic outputs, after trying and failing to mug you with a gun (which he found he forgot at home), is clearly less likely to get a big finite payoff than other uses of the money. The guy is claiming two things: 1) large payoffs (in things like life-years or dollars, not utility, which depends on your psychology) are physically possible 2) conditional on 1, the payoffs are more likely from paying him than other uses of money. Realistic amounts of evidence won’t be enough to neutralize 1), but would easily neutralize 2).
Heuristics which tell you not to pay off the mugger are right, even for total utilitarians.
Moreover, many of our intuitions look to be heuristics trained with past predictive success and delivery of individual rewards in one’s lifetime. If you save 1000 lives, trillions of person-seconds, you will not get billions of times the reinforcement you would get from eating a chocolate bar. You may get a ‘warm glow’ and some social prestige for success, but this will be a reward of ordinary scale in your reinforcement system, not enough to overcome astronomically low probabilities. So learned intuitions will tend to move you away from what would be good deals for an aggregative utilitarian, since they are bad deals in terms of discounted status and sex and chocolate.
Peter Singer argues that we should then discount those intuitions trained for non-moral purposes. Robin Hanson might argue that morality is overrated relative to our nonmoral desires. But it is worth attending to the processes that train intuitions, and figuring out which criteria one endorses.
Yes, we do: bounded utility functions work just fine without any mathematical difficulties
And so does speed prior.
Realistic amounts of evidence won’t be enough to neutralize 1), but would easily neutralize 2).
Yes. I have an example of why the intuition “but anyone can do that” is absolutely spot on. You give money to this mugger (and similar muggers), then another mugger shows up, and noticing doubt in your eyes, displays a big glowing text in front of you which says, “yes, i really have powers outside the matrix”. Except you haven’t got the money. Because you were being completely insane, by the medical definition of the term—your actions were not linked to reality in any way, and you failed to consider the utility of potential actions that are linked to reality (e.g. keep the money, give to a guy that displays the glowing text).
The intuition is that sane actions should be supported by evidence, whereas actions based purely on how you happened to assign priors, are insane. (And it is utterly ridiculous to say that low probability is a necessary part of Pascal’s wager, because as a matter of fact, probability must be high enough.) . I have a suspicion that this intuition reflects the fact that generally, actions conditional on evidence, have higher utility than any actions not conditional on evidence.
Yes, we do: bounded utility functions work just fine without any mathematical difficulties, and seem to map well to the psychological mechanisms that produce our intuitions. Objections to them are more philosophical and person-dependent.
Such as, for example, the fact that killing 3^^^^^^3 people shouldn’t be OK because there’s still 3^^^3 people left and my happiness meter is maxed out anyway.
Sorry, I might be just blinded by the technical language, but I’m not seeing why that link invalidates my comment. Could you maybe pull a quote, or even clarify?
Such as, for example, the fact that killing 3^^^^^^3 people shouldn’t be OK because there’s still 3^^^3 people left and my happiness meter is maxed out anyway.
E.g. the example above suggests something like a utility function of the form “utility equals the amount of quantity A for A<S, otherwise utility is equal to S” which rejects free-lunch increases in happy-years. But it’s easy to formulate a bounded utility function that takes such improvements, without being fanatical in the tradeoffs made.
Trivially, it’s easy to give a bounded utility function that always prefers a higher finite quantity of A but still converges, although eventually the preferences involved have to become very weak cardinally. A function with such a term on human happiness would not reject an otherwise “free lunch”. You never “max out,” just become willing to take smaller risks for incremental gains.
Less trivially, one can include terms like those in the bullet-pointed lists at the linked discussion, mapping to features that human brains distinguish and care about enough to make tempting counterexamples: “but if we don’t account for X, then you wouldn’t exert modest effort to get X!” Terms for relative achievement, e.g. the proportion (or adjusted proportion) of potential good (under some scheme of counterfactuals) achieved, neutralize an especially wide range of purported counterexamples.
E.g. the example above suggests something like a utility function of the form “utility equals the amount of quantity A for A<S, otherwise utility is equal to S” which rejects free-lunch increases in happy-years. But it’s easy to formulate a bounded utility function that takes such improvements, without being fanatical in the tradeoffs made.
… it is? Maybe I’m misusing the term “bounded utility function”. Could you elaborate on this?
Yes, I think you are misusing the term. It’s the utility that’s bounded, not the inputs. Say that U=1-(1/(X^2) and 0 when X=0, and X is the quantity of some good. Then utility is bounded between 0 and 1, but increasing X from 3^^^3 to 3^^^3+1 or 4^^^^4 will still (exceedingly slightly) increase utility. It just won’t take risks for small increases in utility. However, terms in the bounded utility function can give weight to large numbers, to relative achievement, to effort, and all the other things mentioned in the discussion I linked, so that one takes risks for those.
Bounded utility functions still seem to cause problems when uncertainty is involved. For example, consider the aforementioned utility function U(n) = 1 - (1 / (n^2)), and let n equal the number of agents living good lives. Using this function, the utility of a 1 in 1 chance of there being 10 agents living good lives equals 1 - (1 / (10^2)) = 0.99, and the utility of a 9 in 10 chance of 3^^^3 agents living good lives and a 1 in 10 chance of no agents living good lives roughly equals 0.1 0 + 0.9 1 = 0.9. Thus, in this situation the agent would be willing to kill (3^^^3) − 10 agents in order to prevent a 0.1 chance of everyone dying, which doesn’t seem right at all. You could modify the utility function, but I think this issue would still to exist to some extent.
To be really clear, the problem with Pascal’s Mugging is that even after eliminating infinity as a coherent scenario, any simplicity prior which defines simplicity strictly over computational complexity will apparently yield divergent returns for aggregative utility functions when summed over all probable scenarios, because the material size of possible scenarios grows much faster than their computational complexity (Busy Beaver function or just tetration).
The problem with Pascal’s Wager on the other hand is that it shuts down an ongoing conversation about plausibility by claiming that it doesn’t matter how small the probability is, thus averting a logically polite duty to provide evidence and engage with counterarguments.
To be really clear, the problem with Pascal’s Mugging is that even after eliminating infinity as a coherent scenario, any simplicity prior which defines simplicity strictly over computational complexity will apparently yield divergent returns for aggregative utility functions when summed over all probable scenarios, because the material size of possible scenarios grows much faster than their computational complexity (Busy Beaver function or just tetration).
That seems overly specific. There are many other ways in which priors assigned to highly speculative propositions may not be low enough, or when impact of other available actions on a highly speculative scenario be under-evaluated.
The problem with Pascal’s Wager on the other hand is that it shuts down an ongoing conversation about plausibility by claiming that it doesn’t matter how small the probability is, thus averting a logically polite duty to provide evidence and engage with counterarguments.
To me, Pascal’s Wager is defined by a speculative scenario for which there exist no evidence, which has high enough impact to result in actions which are not based on any evidence, despite the uncertainty towards speculative scenarios.
How THE HELL does the above (ok, I didn’t originally include the second quotation, but still) constitute confusion of Pascal’s Wager and Pascal’s Mugging, let alone “willful misinterpretation” ?
I certainly consider that if you multiply a very tiny probability by a huge payoff and then expect others to take your calculation seriously as a call to action, you’re being silly, however it’s labeled. Humans can’t even consider very tiny probabilities without privileging the hypothesis.
Note also that a crazy mugger could demand $10 or else 10^30 people outside the matrix will die, and then argue that you should rationally trust him 100% so the figure is 10^29 lives/$ , or argue that it is 90% certain that those people will die because he’s a bit uncertain about the danger in the alternate worlds, or the like. It’s not about the probability which mugger estimates, it’s about the probability that the typical payer estimates.
I will certainly admit that the precise label is not my true objection, and apologise if I have seemed to be arguing primarily over definitions (which is of course actually a terrible thing to do in general).
Maybe look at the context of the conversation here? edit: to be specific, you might want to reply to HoldenKarnofsky; after all, the utility of convincing him that he’s incorrect in describing it as “Pascal’s Mugging” type arguments ought to be huge...
edit2: and if it’s not clear, I’m not accusing anyone of anything. Holden said,
On the flip side, I believe that a lot of readers believe that “Pascal’s Mugging” type arguments are sufficient to establish that a particular giving opportunity is outstanding
I just linked an example of phenomenon which I think may be the cause of Holden’s belief. Feel free to correct him with your brilliant argument that he should simply test if they actually accept Pascal’s Mugging by asking them about 3^^^3 utilons.
On the flip side, I believe that a lot of readers believe that “Pascal’s Mugging” type arguments are sufficient to establish that a particular giving opportunity is outstanding. I don’t believe the OP believes this.
...
My current working theory is that proponents of “Pascal’s Mugging” type arguments tend to neglect the “flow-through effects” of accomplishing good.
Still might be the thing Holden Karnofsky refers to in the following passages:
And yet remains clearly not the thing that is talked about by either Eliezer or your actual comment.
If it is valuable to make the observation that Holden really isn’t referring to what Eliezer assumes he is then by all means make that point instead of the one you made.
PASCAL’S WAGER IS DEFINED BY LOW PROBABILITIES NOT BY LARGE PAYOFFS
PASCAL’S WAGER IS DEFINED BY LOW PROBABILITIES NOT BY LARGE PAYOFFS
PASCAL’S WAGER IS DEFINED BY LOW PROBABILITIES NOT BY LARGE PAYOFFS
I’ve tried saying this in small letters a number of times, and once in the main post The Pascal’s Wager Fallacy Fallacy, and people apparently just haven’t paid attention, so I’m just going to try shouting it over and over every time somebody makes the same mistake over and over.
edit: And in case it’s not clear, the point is that Pascal’s wager does not depend on the misestimate of probability being low. Any finite variation requires that the probability is high enough .
Likewise, here (linked from the thread I linked) you have both: a prior which is silly high (1 in 2000), and big impact (7 billion lives).
edit: whoops. 1 in 2000 and general talk of low probabilities is in the thread, not in the video. In the video she just goes ahead assigning arbitrary 30% probability to picking an organization with which we live and without which we die, which is obviously so high that much like Pascal’s wager going from 0.5 probability to “the probability could be low, the impact is still infinite!”, so does the LW discussion of this video progress from un-defensible 30% to it doesn’t matter. Let’s picture a Pascal Scam: someone says that there is 50% probability (mostly via ignorance) that unless they are given a lot of money, 10^30 people will die. The audience doesn’t buy 50% probability, but it does still pay up.
(Reply to edit: In the presentation that 30% is one probability in a chain, not an absolute value. Stop with the willful misrepresentations, please.)
From the article:
However, Pascal realizes that the value of 1⁄2 actually plays no real role in the argument, thanks to (2). This brings us to the third, and by far the most important, of his arguments...
If there were a 0.5 probability that the Christian God existed, the wager would make a fuckton more sense. Today we think Pascal’s Wager is a logical fallacy rather than a mere mistaken probability estimate only because later versions of the argument were put forward for lower probabilities, and/or because Pascal went on to argeu that it would carry for lower probabilities.
If the video is where is the actual instance of Pascal’s Wager is being offered in support of SIAI, then it would have been better to link it directly. I also hate video because it’s not searchable, but I can hardly blame you for that, so I will try scanning it.
Before scanning, I precommit to renouncing, abjuring, and distancing MIRI from the argument in the video if it argues for no probability higher than 1 in 2000 of FAI saving the world, because I myself do not positively engage in long-term projects on the basis of probabilities that low (though I sometimes avoid doing things for dangers that small). There ought to be at least one x-risk effort with a greater probability of saving the world than this—or if not, you ought to make one. If you know yourself for an NPC and that you cannot start such a project yourself, you ought to throw money at anyone launching a new project whose probability of saving the world is not known to be this small. 7 billion is also a stupidly low number—x-risk dominates all other optimal philanthropy because of the value of future galaxies, not because of the value of present-day lives. The confluence of these two numbers makes me strongly suspect that, if they are not misquotes in some sense, both low numbers were (presumably unconsciously) chosen to make the ‘lives saved per dollar’ look like a reasonable number in human terms, when in fact the x-risk calculus is such that all utilons should be measured in Probability of OK Outcome because the value of future galaxies stomps everything else.
Attempts to argue for large probabilities that FAI is important, and then tiny probabilities that MIRI is instrumental in creating FAI, will also strike me as a wrongheaded attempt at modesty. On a very large scale, if you think FAI stands a serious chance of saving the world, then humanity should dump a bunch of effort into it, and if nobody’s dumping effort into it then you should dump more effort than currently into it. Calculations of marginal impact in POKO/dollar are sensible for comparing two x-risk mitigation efforts in demand of money, but in this case each marginal added dollar is rightly going to account for a very tiny slice of probability, and this is not Pascal’s Wager. Large efforts with a success-or-failure criterion are rightly, justly, and unavoidably going to end up with small marginal probabilities per added unit effort. It would only be Pascal’s Wager if the whole route-to-humanity-being-OK were assigned a tiny probability, and then a large payoff used to shut down further discussion of whether the next unit of effort should go there or to a different x-risk.
(Scans video.)
This video is primarily about value of information estimates.
“Principle 2: Don’t trust your estimates too much. Estimates in, estimates out.” Good.
Application to the Singularity… It’s explicitly stated that the value is 7 billion lives plus all future generations, which is better—a lower bound is being set, not an estimated exact value.
Final calculation shown:
Probability of eventual AI: 80%
Probability AI with no safeguards will kill us: 80%
(Both of these numbers strike me as a bit suspicious in their apparent medianness which is something that often happens when an argument is unconsciously optimized for sounding reasonable. Really, the probability that AI happens at all, ever, is 80%? Isn’t that a bit low? Is this supposed to be factoring in the probability of nanotechnological warfare wiping out humanity before then, or something? Certainly, AI being possible in principle should have a much more extreme probability than 80%. And a 20% probability of an unsafed AI not killing you sounds like quite an amazing bonanza to get for free. But carrying on...)
Probability we manage safeguards: 40%
(No comment.)
Probability current work is why we manage: 30%
(Arguably too low. Even if MIRI crashes and somebody else carries on successfully, I’d estimate a pretty high probability that their causal pathway there will have had something to do with MIRI. It is difficult to overstate just how much this problem was not on the horizon, at all, of work anyone could actually go out and do twenty years ago.)
Net probabilty: 7%.
This is not necessarily a result I’d agree with, but it’s not a case of Pascal’s Wager on its face. 7% probabilities of large payoffs are a reasonable cause of positive action in sane people; it’s why you would do an Internet startup.
(continues scanning video)
I do not see any slide showing a probability of 1 in 2000. Was this spoken aloud? At what time in the episode?
(Arguably too low. Even if MIRI crashes and somebody else carries on, I’d estimate a pretty high probability that their causal pathway there will have had something to do with MIRI. It is difficult to overstate just how much this problem was not on the horizon, at all, of work anyone could actually go out and do before MIRI.)
It doesn’t merely have to have something to do with MIRI, it must be the case that without funding MIRI we all die, and with funding MIRI, we don’t, and this is precisely the sort of thing that should have very low probability if MIRI is not demonstrably impressive at doing something else.
I do not see any slide showing a probability of 1 in 2000. Was this spoken aloud? At what time in the episode?
Hmm. It is mentioned here and other commenters there likewise talk of low probabilities. I guess I just couldn’t quite imagine someone seriously putting a non small probability on “with MIRI we live, without we die” aspect of it. Startups have quite small probability of success, even without attempting to do the impossible.
edit: And of course what actually matters is donor’s probability.
(Arguably too low. Even if MIRI crashes and somebody else carries on successfully, I’d estimate a pretty high probability that their causal pathway there will have had something to do with MIRI. It is difficult to overstate just how much this problem was not on the horizon, at all, of work anyone could actually go out and do twenty years ago.)
For this to work out to 7%, a donor would need 30% probability that their choice of the organization to donate to is such that with this organization we live, and without, we die.
What donor can be so confident in their choice? Is Thiel this confident? Of course not, he only puts in a small fraction of his income, and he puts more into something like this. By the way I am rather curious about your opinion on this project.
http://lesswrong.com/lw/6w3/the_125000_summer_singularity_challenge/4krk
You can randomly accuse people that what they believe constitutes Pascal’s mugging, but that doesn’t make the accusation a valid argument, unless you show that it’s so.
There’s a very simple test to see if someone actually accepts Pascal’s mugging: Go to them and say “I’ll use my godlike hidden powers to increase your utility by 3^^^3 utilons if you hand over to me the complete contents of your bank account.”
Don’t just claim that something else they believe is the same as Pascal’s mugging or I might equally easily claim that someone buying health insurance is a victim of Pascal’s mugging.
Just to be clear: are we saying that a factor of 3^^^3 is a Pascal’s mugging, but a factor of 10^30 isn’t? (In Holden’s comment above, one example in the context of Pascal’s mugging-type problems is a factor of 10^10, even as that’s on the order of the population of the Earth.)
I think any reasonable person hearing “8 lives saved per dollar donated” would file it with Pascal’s mugging (which is Eliezer’s term, but the concept is pretty simple and comprehensible even to someone thinking of less extreme probabilities than Eliezer posits; e.g. Holden, above).
In the linked thread, Rain special-pleads that the topic requires very large numbers to talk about, but jsteinhardt counters that that doesn’t make humans any better at reasoning about tiny probabilities multiplied by large numbers. jsteinhardt also points out that just because you can multiply a small number by a large number doesn’t mean the product actually makes any sense at all.
No. The problem with Pascal’s mugging doesn’t lie merely in the particular hoped-for payoff, it’s that in extreme combinations of small chance/large payoff, the complexity of certain hypotheses doesn’t seem sufficient to adequately (as per our intuitions) penalize said hypotheses.
If I said “give me a dollar, and I’ll use my Matrix Lord powers to have three dollars appear in your wallet”, someone can simply respond that the chances of me being a Matrix Lord is less than one in three, so the expected payoff is less than the cost. But we don’t yet to have a clear, mathematically precise way to explain why we should also respond negatively to “give me a dollar, and I’ll use my Matrix Lord powers to save 3^^^3 lives.”, even though our intuition says we should (and in this case we trust our intuition).
To put it in brief: Pascal’s Mugging is a interesting problem regarding decision theory which LessWrongers should be hoping to solve (I have an idea towards that direction, which I’m writing a discusion post about, but I’d need mathematicians to tell me if it potentially leads to anything); not just a catchphrase you can use to bash someone else’s calculations when their intuitions differs from yours.
Yes, we do: bounded utility functions work just fine without any mathematical difficulties, and seem to map well to the psychological mechanisms that produce our intuitions. Objections to them are more philosophical and person-dependent.
If we are going to be invoking intuition, then we should be careful about using examples with many extraneous intuition-provoking factors, and in thinking about how the intuitions are formed.
For example, handing over $1 to a literal Pascal’s Mugger, a guy who asks for the money out of your wallet in exchange for magic outputs, after trying and failing to mug you with a gun (which he found he forgot at home), is clearly less likely to get a big finite payoff than other uses of the money. The guy is claiming two things: 1) large payoffs (in things like life-years or dollars, not utility, which depends on your psychology) are physically possible 2) conditional on 1, the payoffs are more likely from paying him than other uses of money. Realistic amounts of evidence won’t be enough to neutralize 1), but would easily neutralize 2).
Heuristics which tell you not to pay off the mugger are right, even for total utilitarians.
Moreover, many of our intuitions look to be heuristics trained with past predictive success and delivery of individual rewards in one’s lifetime. If you save 1000 lives, trillions of person-seconds, you will not get billions of times the reinforcement you would get from eating a chocolate bar. You may get a ‘warm glow’ and some social prestige for success, but this will be a reward of ordinary scale in your reinforcement system, not enough to overcome astronomically low probabilities. So learned intuitions will tend to move you away from what would be good deals for an aggregative utilitarian, since they are bad deals in terms of discounted status and sex and chocolate.
Peter Singer argues that we should then discount those intuitions trained for non-moral purposes. Robin Hanson might argue that morality is overrated relative to our nonmoral desires. But it is worth attending to the processes that train intuitions, and figuring out which criteria one endorses.
And so does speed prior.
Yes. I have an example of why the intuition “but anyone can do that” is absolutely spot on. You give money to this mugger (and similar muggers), then another mugger shows up, and noticing doubt in your eyes, displays a big glowing text in front of you which says, “yes, i really have powers outside the matrix”. Except you haven’t got the money. Because you were being completely insane, by the medical definition of the term—your actions were not linked to reality in any way, and you failed to consider the utility of potential actions that are linked to reality (e.g. keep the money, give to a guy that displays the glowing text).
The intuition is that sane actions should be supported by evidence, whereas actions based purely on how you happened to assign priors, are insane. (And it is utterly ridiculous to say that low probability is a necessary part of Pascal’s wager, because as a matter of fact, probability must be high enough.) . I have a suspicion that this intuition reflects the fact that generally, actions conditional on evidence, have higher utility than any actions not conditional on evidence.
Such as, for example, the fact that killing 3^^^^^^3 people shouldn’t be OK because there’s still 3^^^3 people left and my happiness meter is maxed out anyway.
Self-consistent isn’t the same as moral.
Bounded utility functions can represent more than your comment suggests, depending on what terms are included. See this discussion.
Sorry, I might be just blinded by the technical language, but I’m not seeing why that link invalidates my comment. Could you maybe pull a quote, or even clarify?
E.g. the example above suggests something like a utility function of the form “utility equals the amount of quantity A for A<S, otherwise utility is equal to S” which rejects free-lunch increases in happy-years. But it’s easy to formulate a bounded utility function that takes such improvements, without being fanatical in the tradeoffs made.
Trivially, it’s easy to give a bounded utility function that always prefers a higher finite quantity of A but still converges, although eventually the preferences involved have to become very weak cardinally. A function with such a term on human happiness would not reject an otherwise “free lunch”. You never “max out,” just become willing to take smaller risks for incremental gains.
Less trivially, one can include terms like those in the bullet-pointed lists at the linked discussion, mapping to features that human brains distinguish and care about enough to make tempting counterexamples: “but if we don’t account for X, then you wouldn’t exert modest effort to get X!” Terms for relative achievement, e.g. the proportion (or adjusted proportion) of potential good (under some scheme of counterfactuals) achieved, neutralize an especially wide range of purported counterexamples.
… it is? Maybe I’m misusing the term “bounded utility function”. Could you elaborate on this?
Yes, I think you are misusing the term. It’s the utility that’s bounded, not the inputs. Say that U=1-(1/(X^2) and 0 when X=0, and X is the quantity of some good. Then utility is bounded between 0 and 1, but increasing X from 3^^^3 to 3^^^3+1 or 4^^^^4 will still (exceedingly slightly) increase utility. It just won’t take risks for small increases in utility. However, terms in the bounded utility function can give weight to large numbers, to relative achievement, to effort, and all the other things mentioned in the discussion I linked, so that one takes risks for those.
Bounded utility functions still seem to cause problems when uncertainty is involved. For example, consider the aforementioned utility function U(n) = 1 - (1 / (n^2)), and let n equal the number of agents living good lives. Using this function, the utility of a 1 in 1 chance of there being 10 agents living good lives equals 1 - (1 / (10^2)) = 0.99, and the utility of a 9 in 10 chance of 3^^^3 agents living good lives and a 1 in 10 chance of no agents living good lives roughly equals 0.1 0 + 0.9 1 = 0.9. Thus, in this situation the agent would be willing to kill (3^^^3) − 10 agents in order to prevent a 0.1 chance of everyone dying, which doesn’t seem right at all. You could modify the utility function, but I think this issue would still to exist to some extent.
Ah, OK, I was thinking of a bounded utility function as one with a “cutoff point”, yes. You’re absolutely right.
To be really clear, the problem with Pascal’s Mugging is that even after eliminating infinity as a coherent scenario, any simplicity prior which defines simplicity strictly over computational complexity will apparently yield divergent returns for aggregative utility functions when summed over all probable scenarios, because the material size of possible scenarios grows much faster than their computational complexity (Busy Beaver function or just tetration).
The problem with Pascal’s Wager on the other hand is that it shuts down an ongoing conversation about plausibility by claiming that it doesn’t matter how small the probability is, thus averting a logically polite duty to provide evidence and engage with counterarguments.
That seems overly specific. There are many other ways in which priors assigned to highly speculative propositions may not be low enough, or when impact of other available actions on a highly speculative scenario be under-evaluated.
To me, Pascal’s Wager is defined by a speculative scenario for which there exist no evidence, which has high enough impact to result in actions which are not based on any evidence, despite the uncertainty towards speculative scenarios.
How THE HELL does the above (ok, I didn’t originally include the second quotation, but still) constitute confusion of Pascal’s Wager and Pascal’s Mugging, let alone “willful misinterpretation” ?
I certainly consider that if you multiply a very tiny probability by a huge payoff and then expect others to take your calculation seriously as a call to action, you’re being silly, however it’s labeled. Humans can’t even consider very tiny probabilities without privileging the hypothesis.
Note also that a crazy mugger could demand $10 or else 10^30 people outside the matrix will die, and then argue that you should rationally trust him 100% so the figure is 10^29 lives/$ , or argue that it is 90% certain that those people will die because he’s a bit uncertain about the danger in the alternate worlds, or the like. It’s not about the probability which mugger estimates, it’s about the probability that the typical payer estimates.
PASCAL’S WAGER IS DEFINED BY LOW PROBABILITIES NOT BY LARGE PAYOFFS
PASCAL’S WAGER IS DEFINED BY LOW PROBABILITIES NOT BY LARGE PAYOFFS
PASCAL’S WAGER IS DEFINED BY LOW PROBABILITIES NOT BY LARGE PAYOFFS
I will certainly admit that the precise label is not my true objection, and apologise if I have seemed to be arguing primarily over definitions (which is of course actually a terrible thing to do in general).
Maybe look at the context of the conversation here? edit: to be specific, you might want to reply to HoldenKarnofsky; after all, the utility of convincing him that he’s incorrect in describing it as “Pascal’s Mugging” type arguments ought to be huge...
edit2: and if it’s not clear, I’m not accusing anyone of anything. Holden said,
I just linked an example of phenomenon which I think may be the cause of Holden’s belief. Feel free to correct him with your brilliant argument that he should simply test if they actually accept Pascal’s Mugging by asking them about 3^^^3 utilons.
Not a Pascal’s Mugging.
Still might be the thing Holden Karnofsky refers to in the following passages:
...
...
And yet remains clearly not the thing that is talked about by either Eliezer or your actual comment.
If it is valuable to make the observation that Holden really isn’t referring to what Eliezer assumes he is then by all means make that point instead of the one you made.
PASCAL’S WAGER IS DEFINED BY LOW PROBABILITIES NOT BY LARGE PAYOFFS
PASCAL’S WAGER IS DEFINED BY LOW PROBABILITIES NOT BY LARGE PAYOFFS
PASCAL’S WAGER IS DEFINED BY LOW PROBABILITIES NOT BY LARGE PAYOFFS
I’ve tried saying this in small letters a number of times, and once in the main post The Pascal’s Wager Fallacy Fallacy, and people apparently just haven’t paid attention, so I’m just going to try shouting it over and over every time somebody makes the same mistake over and over.
In original Pascal’s wager, he had a prior of 0.5 for existence of God.
edit: And in case it’s not clear, the point is that Pascal’s wager does not depend on the misestimate of probability being low. Any finite variation requires that the probability is high enough .
Likewise, here (linked from the thread I linked) you have both: a prior which is silly high (1 in 2000), and big impact (7 billion lives).
edit: whoops. 1 in 2000 and general talk of low probabilities is in the thread, not in the video. In the video she just goes ahead assigning arbitrary 30% probability to picking an organization with which we live and without which we die, which is obviously so high that much like Pascal’s wager going from 0.5 probability to “the probability could be low, the impact is still infinite!”, so does the LW discussion of this video progress from un-defensible 30% to it doesn’t matter. Let’s picture a Pascal Scam: someone says that there is 50% probability (mostly via ignorance) that unless they are given a lot of money, 10^30 people will die. The audience doesn’t buy 50% probability, but it does still pay up.
(Reply to edit: In the presentation that 30% is one probability in a chain, not an absolute value. Stop with the willful misrepresentations, please.)
From the article:
If there were a 0.5 probability that the Christian God existed, the wager would make a fuckton more sense. Today we think Pascal’s Wager is a logical fallacy rather than a mere mistaken probability estimate only because later versions of the argument were put forward for lower probabilities, and/or because Pascal went on to argeu that it would carry for lower probabilities.
If the video is where is the actual instance of Pascal’s Wager is being offered in support of SIAI, then it would have been better to link it directly. I also hate video because it’s not searchable, but I can hardly blame you for that, so I will try scanning it.
Before scanning, I precommit to renouncing, abjuring, and distancing MIRI from the argument in the video if it argues for no probability higher than 1 in 2000 of FAI saving the world, because I myself do not positively engage in long-term projects on the basis of probabilities that low (though I sometimes avoid doing things for dangers that small). There ought to be at least one x-risk effort with a greater probability of saving the world than this—or if not, you ought to make one. If you know yourself for an NPC and that you cannot start such a project yourself, you ought to throw money at anyone launching a new project whose probability of saving the world is not known to be this small. 7 billion is also a stupidly low number—x-risk dominates all other optimal philanthropy because of the value of future galaxies, not because of the value of present-day lives. The confluence of these two numbers makes me strongly suspect that, if they are not misquotes in some sense, both low numbers were (presumably unconsciously) chosen to make the ‘lives saved per dollar’ look like a reasonable number in human terms, when in fact the x-risk calculus is such that all utilons should be measured in Probability of OK Outcome because the value of future galaxies stomps everything else.
Attempts to argue for large probabilities that FAI is important, and then tiny probabilities that MIRI is instrumental in creating FAI, will also strike me as a wrongheaded attempt at modesty. On a very large scale, if you think FAI stands a serious chance of saving the world, then humanity should dump a bunch of effort into it, and if nobody’s dumping effort into it then you should dump more effort than currently into it. Calculations of marginal impact in POKO/dollar are sensible for comparing two x-risk mitigation efforts in demand of money, but in this case each marginal added dollar is rightly going to account for a very tiny slice of probability, and this is not Pascal’s Wager. Large efforts with a success-or-failure criterion are rightly, justly, and unavoidably going to end up with small marginal probabilities per added unit effort. It would only be Pascal’s Wager if the whole route-to-humanity-being-OK were assigned a tiny probability, and then a large payoff used to shut down further discussion of whether the next unit of effort should go there or to a different x-risk.
(Scans video.)
This video is primarily about value of information estimates.
“Principle 2: Don’t trust your estimates too much. Estimates in, estimates out.” Good.
Application to the Singularity… It’s explicitly stated that the value is 7 billion lives plus all future generations, which is better—a lower bound is being set, not an estimated exact value.
Final calculation shown:
Probability of eventual AI: 80%
Probability AI with no safeguards will kill us: 80%
(Both of these numbers strike me as a bit suspicious in their apparent medianness which is something that often happens when an argument is unconsciously optimized for sounding reasonable. Really, the probability that AI happens at all, ever, is 80%? Isn’t that a bit low? Is this supposed to be factoring in the probability of nanotechnological warfare wiping out humanity before then, or something? Certainly, AI being possible in principle should have a much more extreme probability than 80%. And a 20% probability of an unsafed AI not killing you sounds like quite an amazing bonanza to get for free. But carrying on...)
Probability we manage safeguards: 40%
(No comment.)
Probability current work is why we manage: 30%
(Arguably too low. Even if MIRI crashes and somebody else carries on successfully, I’d estimate a pretty high probability that their causal pathway there will have had something to do with MIRI. It is difficult to overstate just how much this problem was not on the horizon, at all, of work anyone could actually go out and do twenty years ago.)
Net probabilty: 7%.
This is not necessarily a result I’d agree with, but it’s not a case of Pascal’s Wager on its face. 7% probabilities of large payoffs are a reasonable cause of positive action in sane people; it’s why you would do an Internet startup.
(continues scanning video)
I do not see any slide showing a probability of 1 in 2000. Was this spoken aloud? At what time in the episode?
It doesn’t merely have to have something to do with MIRI, it must be the case that without funding MIRI we all die, and with funding MIRI, we don’t, and this is precisely the sort of thing that should have very low probability if MIRI is not demonstrably impressive at doing something else.
Hmm. It is mentioned here and other commenters there likewise talk of low probabilities. I guess I just couldn’t quite imagine someone seriously putting a non small probability on “with MIRI we live, without we die” aspect of it. Startups have quite small probability of success, even without attempting to do the impossible.
edit: And of course what actually matters is donor’s probability.
For this to work out to 7%, a donor would need 30% probability that their choice of the organization to donate to is such that with this organization we live, and without, we die.
What donor can be so confident in their choice? Is Thiel this confident? Of course not, he only puts in a small fraction of his income, and he puts more into something like this. By the way I am rather curious about your opinion on this project.
Are you sure they are wrong about what constitutes Pascal’s mugging, rather than about whether the probability of xrisk is low?