# Probabilities Small Enough To Ignore: An attack on Pascal’s Mugging

Sum­mary: the prob­lem with Pas­cal’s Mug­ging ar­gu­ments is that, in­tu­itively, some prob­a­bil­ities are just too small to care about. There might be a prin­ci­pled rea­son for ig­nor­ing some prob­a­bil­ities, namely that they vi­o­late an im­plicit as­sump­tion be­hind ex­pected util­ity the­ory. This sug­gests a pos­si­ble ap­proach for for­mally defin­ing a “prob­a­bil­ity small enough to ig­nore”, though there’s still a bit of ar­bi­trari­ness in it.

This post is about find­ing a way to re­solve the para­dox in­her­ent in Pas­cal’s Mug­ging. Note that I’m not talk­ing about the bas­tardized ver­sion of Pas­cal’s Mug­ging that’s got­ten pop­u­lar of late, where it’s used to re­fer to any ar­gu­ment in­volv­ing low prob­a­bil­ities and huge stakes (e.g. low chance of thwart­ing un­safe AI vs. as­tro­nom­i­cal stakes). Nei­ther am I talk­ing speci­fi­cally about the “mug­ging” illus­tra­tion, where a “mug­ger” shows up to threaten you.
Rather I’m talk­ing about the gen­eral de­ci­sion-the­o­retic prob­lem, where it makes no differ­ence how low of a prob­a­bil­ity you put on some deal pay­ing off, be­cause one can always choose a hu­mon­gous enough pay­off to make “make this deal” be the dom­i­nat­ing op­tion. This is a prob­lem that needs to be solved in or­der to build e.g. an AI sys­tem that uses ex­pected util­ity and will be­have in a rea­son­able man­ner.
In­tu­ition: how Pas­cal’s Mug­ging breaks im­plicit as­sump­tions in ex­pected util­ity theory
In­tu­itively, the prob­lem with Pas­cal’s Mug­ging type ar­gu­ments is that some prob­a­bil­ities are just too low to care about. And we need a way to look at just the prob­a­bil­ity part com­po­nent in the ex­pected util­ity calcu­la­tion and ig­nore the util­ity com­po­nent, since the core of PM is that the util­ity can always be ar­bi­trar­ily in­creased to over­whelm the low prob­a­bil­ity.
Let’s look at the con­cept of ex­pected util­ity a bit. If you have a 10% chance of get­ting a dol­lar each time when you make a deal, and this has an ex­pected value of 0.1, then this is just a differ­ent way of say­ing that if you took the deal ten times, then you would on av­er­age have 1 dol­lar at the end of that deal.
More gen­er­ally, it means that if you had the op­por­tu­nity to make ten differ­ent deals that all had the same ex­pected value, then af­ter mak­ing all of those, you would on av­er­age end up with one dol­lar. This is the jus­tifi­ca­tion for why it makes sense to fol­low ex­pected value even for unique non-re­peat­ing events: be­cause even if that par­tic­u­lar event wouldn’t re­peat, if your gen­eral strat­egy is to ac­cept other bets with the same EV, then you will end up with the same out­come as if you’d taken the same re­peat­ing bet many times. And even though you only get the dol­lar af­ter ten deals on av­er­age, if you re­peat the tri­als suffi­ciently many times, your prob­a­bil­ity of hav­ing the av­er­age pay­out will ap­proach one.
Now con­sider a Pas­cal’s Mug­ging sce­nario. Say some­one offers to cre­ate 10^100 happy lives in ex­change for some­thing, and you as­sign them a 0.000000000000000000001 prob­a­bil­ity to them be­ing ca­pa­ble and will­ing to carry through their promise. Naively, this has an over­whelm­ingly pos­i­tive ex­pected value.
But is it re­ally a benefi­cial trade? Sup­pose that you could make one deal like this per sec­ond, and you ex­pect to live for 60 more years, for about 1,9 billion trades in to­tal. Then, there would be a prob­a­bil­ity of 0,999999999998 that the deal would never once have paid off for you. Which sug­gests that the EU calcu­la­tion’s im­plicit as­sump­tion—that you can re­peat this of­ten enough for the util­ity to con­verge to the ex­pected value—would be vi­o­lated.
Our first attempt
This sug­gests an ini­tial way of defin­ing a “prob­a­bil­ity small enough to be ig­nored”:
1. Define a “prob­a­bil­ity small enough to be ig­nored” (PSET, or by slight re­ar­rang­ing of let­ters, PEST) such that, over your life­time, the ex­pected times that the event hap­pens will be less than one.
2. Ig­nore deals where the prob­a­bil­ity com­po­nent of the EU calcu­la­tion in­volves a PEST.
Look­ing at the first at­tempt in detail
To calcu­late PEST, we need to know how of­ten we might be offered a deal with such a prob­a­bil­ity. E.g. a 10% chance for some­thing might be a PEST if we only lived for a short enough time that we could make a deal with a 10% chance once. So, a more pre­cise defi­ni­tion of a PEST might be that it’s a prob­a­bil­ity such that
(amount of deals that you can make in your life that have this prob­a­bil­ity) * (PEST) < 1
But defin­ing “one” as the min­i­mum times we should ex­pect the event to hap­pen for the prob­a­bil­ity to not be a PEST feels a lit­tle ar­bi­trary. In­tu­itively, it feels like the thresh­old should de­pend on our de­gree of risk aver­sion: maybe if we’re risk averse, we want to re­duce the ex­pected amount of times some­thing hap­pens dur­ing our lives to (say) 0,001 be­fore we’re ready to ig­nore it. But part of our mo­ti­va­tion was that we wanted a way to ig­nore the util­ity part of the calcu­la­tion: bring­ing in our de­gree of risk aver­sion seems like it might in­tro­duce the util­ity again.
What if re­defined risk aver­sion/​neu­tral­ity/​prefer­ence (at least in this con­text) as how low one would be will­ing to let the “ex­pected amount of times this might hap­pen” fall be­fore con­sid­er­ing a prob­a­bil­ity a PEST?
Let’s use this idea to define an Ex­pected Life­time Utility:
ELU(S,L,R) = the ELU of a strat­egy S over a life­time L is the ex­pected util­ity you would get if you could make L deals in your life, and were only will­ing to ac­cept deals with a min­i­mum prob­a­bil­ity P of at least S, tak­ing into ac­count your risk aver­sion R and as­sum­ing that each deal will pay off ap­prox­i­mately P*L times.
ELU example
Sup­pose that we a have a world where we can take three kinds of ac­tions.
- Ac­tion A takes 1 unit of time and has an ex­pected util­ity of 2 and prob­a­bil­ity 13 of pay­ing off on any one oc­ca­sion.
- Ac­tion B takes 3 units of time and has an ex­pected util­ity of 10^(Gra­ham’s num­ber) and prob­a­bil­ity 1/​100000000000000 of pay­ing off one any one oc­ca­sion.
- Ac­tion C takes 5 units of time and has an ex­pected util­ity of 20 and prob­a­bil­ity 1100 of pay­ing off on an one oc­ca­sion.
As­sum­ing that the world’s life­time is fixed at L = 1000 and R = 1:
ELU(“always choose A”): we ex­pect A to pay off on ((1000 /​ 1) * 13) = 333 in­di­vi­d­ual oc­ca­sions, so with R = 1, we deem it ac­cept­able to con­sider the util­ity of A. The ELU of this strat­egy be­comes (1000 /​ 1) * 2 = 2000.
ELU(“always choose B”): we ex­pect B to pay off on ((1000 /​ 3) * 1/​100000000000000) = 0.00000000000333 oc­ca­sions, so with R = 1, we con­sider the ex­pected util­ity of B to be 0. The ELU of this strat­egy thus be­comes ((1000 /​ 3) * 0) = 0.
ELU(“always choose C”): we ex­pect C to pay off on ((1000 /​ 5) * 1100) = 2 in­di­vi­d­ual oc­ca­sions, so with R = 1, we con­sider the ex­pected util­ity of C to be ((1000 /​ 5) * 20) = 4000.
Thus, “always choose C” is the best strat­egy.
Defin­ing R
Is R some­thing to­tally ar­bi­trary, or can we de­ter­mine some more ob­jec­tive crite­ria for it?
Here’s where I’m stuck. Thoughts are wel­come. I do know that while set­ting R = 1 was a con­ve­nient ex­am­ple, it’s most likely too high, be­cause it would sug­gest things like not us­ing seat belts.
Gen­eral thoughts on this approach
An in­ter­est­ing thing about this ap­proach is that the thresh­old for a PEST be­comes de­pen­dent on one’s ex­pected life­time. This is sur­pris­ing at first, but ac­tu­ally makes some in­tu­itive sense. If you’re liv­ing in a dan­ger­ous en­vi­ron­ment where you might be kil­led any­time soon, you won’t be very in­ter­ested in spec­u­la­tive low-prob­a­bil­ity op­tions; rather you want to fo­cus on mak­ing sure you sur­vive now. Whereas if you live in a mod­ern-day Western so­ciety, you may be will­ing to in­vest some amount of effort in weird low-prob­a­bil­ity high-pay­off deals, like cry­on­ics.
On the other hand, whereas in­vest­ing in that low-prob­a­bil­ity, high-util­ity op­tion might not be good for you in­di­vi­d­u­ally, it could still be a good evolu­tion­ary strat­egy for your genes. You your­self might be very likely to die, but some­one else car­ry­ing the risk-tak­ing genes might hit big and be very suc­cess­ful in spread­ing their genes. So it seems like our defi­ni­tion of L, life­time length, should vary based on what we want: are we look­ing to im­ple­ment this strat­egy just in our­selves, our whole species, or some­thing else? Ex­actly what are we max­i­miz­ing over?
• I don’t know if this solves very much. As you say, if we use the num­ber 1, then we shouldn’t wear seat­belts, get fire in­surance, or eat healthy to avoid get­ting can­cer, since all of those can be clas­sified as Pas­cal’s Mug­gings. But if we start go­ing for less than one, then we’re just defin­ing away Pas­cal’s Mug­ging by fiat, say­ing “this is the level at which I am will­ing to stop wor­ry­ing about this”.

Also, as some peo­ple el­se­where in the com­ments have pointed out, this makes prob­a­bil­ity non-ad­di­tive in an awk­ward sort of way. Sup­pose that if you eat un­healthy, you in­crease your risk of one mil­lion differ­ent dis­eases by plus one-in-a-mil­lion chance of get­ting each. Sup­pose also that eat­ing healthy is a mildly un­pleas­ant sac­ri­fice, but get­ting a dis­ease is much worse. If we calcu­late this out dis­ease-by-dis­ease, each dis­ease is a Pas­cal’s Mug­ging and we should choose to eat un­healthy. But if we calcu­late this out in the broad cat­e­gory of “get­ting some dis­ease or other”, then our chances are quite high and we should eat healthy. But it’s very strange that our on­tol­ogy/​cat­e­go­riza­tion scheme should af­fect our de­ci­sion-mak­ing. This be­comes much more dan­ger­ous when we start talk­ing about AIs.

Also, does this cre­ate weird non­lin­ear thresh­olds? For ex­am­ple, sup­pose that you live on av­er­age 80 years. If some event which causes you near-in­finite di­su­til­ity hap­pens ev­ery 80.01 years, you should ig­nore it; if it hap­pens ev­ery 79.99 years, then pre­vent­ing it be­comes the en­tire fo­cus of your ex­is­tence. But it seems non­sen­si­cal for your be­hav­ior to change so dras­ti­cally based on whether an event is ev­ery 79.99 years or ev­ery 80.01 years.

Also, a world where peo­ple fol­low this plan is a world where I make a kil­ling on the In­verse Lot­tery (rules: 10,000 peo­ple take tick­ets; each ticket holder gets paid \$1, ex­cept a ran­domly cho­sen “win­ner” who must pay \$20,000)

• we shouldn’t wear seat­belts, get fire in­surance, or eat healthy to avoid get­ting can­cer, since all of those can be clas­sified as Pas­cal’s Muggings

How con­fi­dent are you that this is false?

• sup­pose that you live on av­er­age 80 years. [...] 80.01 years [...] 79.99 years

I started writ­ing a com­pletely wrong re­sponse to this, and it seems worth men­tion­ing for the benefit of any­one else whose brain has the same bugs as mine.

I was go­ing to pro­pose re­plac­ing “Com­pute how many times you ex­pect this to hap­pen to you; treat it as nor­mal if n>=1 and ig­nore com­pletely if n= 1, ig­nore com­pletely if n<0.01, and in­ter­po­late smoothly be­tween those for in­ter­me­di­ate n”.

But all this does, if the (dis)util­ity of the event is hugely greater than that of ev­ery­thing else you care about, is to push the thresh­old where you jump be­tween “ig­nore ev­ery­thing ex­cept this” and “ig­nore this” fur­ther out, maybe from “once per 80 years” to “once per 8000 years”.

I fear we re­ally do need some­thing like bounded util­ity to make that prob­lem go away.

• I fear we re­ally do need some­thing like bounded util­ity to make that prob­lem go away.

If what you dis­like is a dis­con­ti­nu­ity, you still get a dis­con­ti­nu­ity at the bound.

I am not a util­i­tar­ian, but I would look for a way to deal with the is­sue at the meta level. Why would you be­lieve the bet that Pas­cal’s Mug­ger offers you?

At a more pro­saic level (e.g. seat belts) this looks to be a sim­ple mat­ter of risk tol­er­ance and not that much of a prob­lem.

• What do you mean by “you still get a dis­con­ti­nu­ity at the bound”? (I am won­der­ing whether by “bounded util­ity” you mean some­thing like “un­bounded util­ity fol­lowed by clip­ping at some fixed bounds”, which would cer­tainly in­tro­duce weird dis­con­ti­nu­ities but isn’t at all what I have in mind when I imag­ine an agent with bounded util­ities.)

I agree that doubt­ing the mug­ger is a good idea, and in par­tic­u­lar I think it’s en­tirely rea­son­able to sup­pose that the prob­a­bil­ity that any­one can af­fect your util­ity by an amount U must de­crease at least as fast as 1/​U for large U, which is es­sen­tially (ex­cept that I was as­sum­ing a Solomonoff-like prob­a­bil­ity as­sign­ment) what I pro­posed on LW back in 2007.

Now, of course an agent’s prob­a­bil­ity and util­ity as­sign­ments are what­ever they are. Is there some rea­son other than want­ing to avoid a Pas­cal’s mug­ging why that con­di­tion should hold? Well, if it doesn’t hold then your ex­pected util­ity di­verges, which seems fairly bad. -- Though I seem to re­call see­ing an ar­gu­ment from Stu­art Arm­strong or some­one of the sort to the effect that if your util­ities aren’t bounded then your ex­pected util­ity in some situ­a­tions pretty much has to di­verge any­way.

(We can’t hope for a much stronger rea­son, I think. In par­tic­u­lar, your util­ities can be just about any­thing, so there’s clearly no out­right im­pos­si­bil­ity or in­con­sis­tency about hav­ing util­ities that “in­crease too fast” rel­a­tive to your prob­a­bil­ity as­sign­ments.)

• but isn’t at all what I have in mind when I imag­ine an agent with bounded utilities

What do you have in mind?

• Crudely, some­thing like this: Divide the value-laden world up into in­di­vi­d­u­als and into short time-slices. Rate the hap­piness of each in­di­vi­d­ual in each time-slice on a scale from −1 to +1. (So we sup­pose there are limits to mo­men­tary in­ten­sity of satis­fac­tion or dis­satis­fac­tion, which seems rea­son­able to me.) Now, let h be a tiny pos­i­tive num­ber, and as­sign over­all util­ity 1/​h tanh(sum of atanh(h * lo­cal util­ity)).

Be­cause h is ex­tremely small, for mod­est lifes­pans and num­bers of agents this is very close to net util­ity = sum of lo­cal util­ity. But we never get |over­all util­ity| > 1/​h.

Now, this feels like an ad hoc trick rather than a prin­ci­pled de­scrip­tion of how we “should” value things, and I am not se­ri­ously propos­ing it as what any­one’s util­ity func­tion “should” (or does) look like. But I think one could make some­thing of a case for an agent that works more like this: just add up lo­cal util­ities lin­early, but weight the util­ity for agent A in times­lice T in a way that de­creases ex­po­nen­tially (with small con­stant) with the de­scrip­tion-length of (A,T), where the way in which we de­scribe things is in­evitably some­what refer­enced to our­selves. So it’s pretty easy to say “me, now” and not that much harder to say “my wife, an hour from now”, so these two are weighted similarly; but you need a longer de­scrip­tion to spec­ify a per­son and time much fur­ther away, and if you have 3^^^3 peo­ple then you’re go­ing to get weights about as small as 1/​3^^^3.

• Let me see if I un­der­stand you cor­rectly.

You have a ma­trix of (num­ber of in­di­vi­d­u­als) x (num­ber of time-slices). Each ma­trix cell has value (“hap­piness”) that’s con­strained to lie in the [-1..1] in­ter­val. You call the cell value “lo­cal util­ity”, right?

And then you, ba­si­cally, sum up the cell val­ues, re-scale the sum to fit into a pre-defined range and, in the pro­cess, add a trans­for­ma­tion that makes sure the bounds are not sharp cut-offs, but rather limits which you ap­proach asymp­tot­i­cally.

As to the sec­ond part, I have trou­ble vi­su­al­is­ing the lan­guage in which the de­scrip­tion-length would work as you want. It seems to me it will have to in­volve a lot scaf­fold­ing which might col­lapse un­der its own weight.

• “You have a ma­trix …”: cor­rect. “And then …”: whether that’s cor­rect de­pends on what you mean by “in the pro­cess”, but it’s cer­tainly not en­tirely un­like what I meant :-).

Your last para­graph is too metaphor­i­cal for me to work out whether I share your con­cerns. (My de­scrip­tion was ex­tremely hand­wavy so I’m in no po­si­tion to com­plain.) I think the scaf­fold­ing re­quired is ba­si­cally just the agent’s knowl­edge. (To clar­ify a cou­ple of points: not nec­es­sar­ily min­i­mum de­scrip­tion length, which of course is un­com­putable, but some­thing like “short­est de­scrip­tion the agent can read­ily come up with”; and of course in prac­tice what I de­scribe is way too oner­ous com­pu­ta­tion­ally but some crude ap­prox­i­ma­tion might be man­age­able.)

• The ba­sic is­sue is whether the util­ity weights (“de­scrip­tion lengths”) re­flect the sub­jec­tive prefer­ences. If they do, it’s an en­tirely differ­ent ket­tle of fish. If they don’t, I don’t see why “my wife” should get much more weight than “the girl next to me on a bus”.

• I think real peo­ple have prefer­ences whose weights de­cay with dis­tance—ge­o­graph­i­cal, tem­po­ral and con­cep­tual. I think it would be rea­son­able for ar­tifi­cial agents to do like­wise. Whether the par­tic­u­lar mode of de­cay I de­scribe re­sem­bles real peo­ple’s, or would make an ar­tifi­cial agent tend to be­have in ways we’d want, I don’t know. As I’ve already in­di­cated, I’m not claiming to be do­ing more than sketch what some kinda-plau­si­ble bounded-util­ity agents might look like.

• One easy way to do this is to map an un­bounded util­ity func­tion onto a finite in­ter­val. You will end up with the same or­der of prefer­ences, but your choices won’t always be the same. In par­tic­u­lar you will start avoid­ing cases of the mug­ging.

• In par­tic­u­lar you will start avoid­ing cases of the mug­ging.

Not re­ally avoid­ing—a bound on your util­ity in the con­text of a Pas­cal’s Mug­ging is ba­si­cally a bound on what the Mug­ger can offer you. For any prob­a­bil­ity of what the Mug­ger promises there is some non-zero amount that you would be will­ing to pay and that amount is a func­tion of your bound (and of the prob­a­bil­ity, of course).

How­ever util­ity asymp­tot­i­cally ap­proach­ing a bound is likely to have its own set of prob­lems. Here is a sce­nario af­ter five sec­onds of think­ing:

That vex­a­tious chap Omega ap­proaches you (again!) and this time in­stead of boxes offers you two but­tons, let’s say one of them is teal-coloured and the other is cyan-coloured. He says that if you press the teal but­ton, 1,000,001 peo­ple will be cured of ter­mi­nal can­cer. But if you press the cyan but­ton, 1,000,000 peo­ple will be cured of ter­mi­nal can­cer plus he’ll give you a dol­lar. You con­sult your util­ity func­tion, hap­pily press the cyan but­ton and walk away richer by a dol­lar. Did some­thing go wrong?

• Yes, some­thing went wrong in your anal­y­sis.

I sug­gested map­ping an un­bounded util­ity func­tion onto a finite in­ter­val. This pre­serves the or­der of the prefer­ences in the un­bounded util­ity func­tion.

In my “un­bounded” func­tion, I pre­fer sav­ing 1,000,001 peo­ple to sav­ing 1,000,000 and get­ting a dol­lar. So I have the same prefer­ence with the bounded func­tion, and so I press the teal but­ton.

• If you want to do all op­er­a­tions—no­tably, adding util­ity and dol­lars—be­fore map­ping to the finite in­ter­val, you still fall prey to the Pas­cal’s Mug­ging and I don’t see the point of the map­ping at all in this case.

• The map­ping is of util­ity val­ues, e.g.

In my un­bounded func­tion I might have:

Sav­ing 1,000,000 lives = 10,000,000,000,000 util­ity. Sav­ing 1,000,001 lives = 10,000,010,000,000 util­ity. Get­ting a dol­lar = 1 util­ity. Sav­ing 1,000,000 lives and get­ting a dol­lar = 10,000,000,000,001 util­ity.

Here we have get­ting a dol­lar < sav­ing 1,000,000 lives < sav­ing 1,000,000 lives and get­ting a dol­lar < sav­ing 1,000,001 lives.

The map­ping is a one-to-one func­tion that maps val­ues be­tween nega­tive and pos­i­tive in­finity to a finite in­ter­val, and pre­serves the or­der of the val­ues. There are a lot of ways to do this, and it will mean that the util­ity of sav­ing 1,000,001 lives will re­main higher than the util­ity of sav­ing 1,000,000 lives and get­ting a dol­lar.

But it pre­serves this or­der, not ev­ery­thing else, and so it can still avoid Pas­cal’s Mug­ging. Ba­si­cally the mug­ging de­pends on mul­ti­ply­ing the util­ity by a prob­a­bil­ity. But since the util­ity has a nu­mer­i­cal bound, that means when the prob­a­bil­ity gets too low, this mul­ti­plied value will tend to­ward zero. This does mean that my sys­tem can give differ­ent re­sults when bet­ting is in­volved. But that’s what we wanted, any­way.

• Oh, I see. So you do all the op­er­a­tions on the un­bounded util­ity, but calcu­late the ex­pected value of the bounded ver­sion.

• As you say, if we use the num­ber 1, then we shouldn’t wear seat­belts, get fire in­surance, or eat healthy to avoid get­ting can­cer, since all of those can be clas­sified as Pas­cal’s Mug­gings.

And in fact, it has taken lots of push­ing to make all of those things com­mon enough that we can no longer say that no one does them. (In fact, look­ing back at hte 90s and early 2000s, it feels like wear­ing one’s seat­belt at all times was pretty con­trar­ian, where I live. This is only chang­ing thanks to in­tense ad­ver­tis­ing cam­paigns.)

• if we use the num­ber 1, then we shouldn’t wear seat­belts, get fire in­surance, or eat healthy to avoid get­ting can­cer, since all of those can be clas­sified as Pas­cal’s Muggings

Isn’t this dealt with in the above by ag­gre­gat­ing all the deals of a cer­tain prob­a­bil­ity to­gether?

(amount of deals that you can make in your life that have this prob­a­bil­ity) * (PEST) < 1

Maybe the ex­pected num­ber of ma­jor car crashes or dan­ger­ous fires, etc that you ex­pe­rience are each less than 1, but the ex­pec­ta­tion for the num­ber of all such things that hap­pen to you might be greater than 1.

There might be is­sues with how to group such events though, since only con­sid­er­ing things with the ex­act same prob­a­bil­ity to­gether doesn’t make sense.

• What are the odds that our ideas about eat­ing healthy are wrong?

• But it seems non­sen­si­cal for your be­hav­ior to change so dras­ti­cally based on whether an event is ev­ery 79.99 years or ev­ery 80.01 years.

Doesn’t it ac­tu­ally make sense to put that thresh­old at the pre­dicted us­able lifes­pan of the uni­verse?

• For ex­am­ple, sup­pose that you live on av­er­age 80 years. If some event which causes you near-in­finite di­su­til­ity hap­pens ev­ery 80.01 years, you should ig­nore it; if it hap­pens ev­ery 79.99 years, then pre­vent­ing it be­comes the en­tire fo­cus of your ex­is­tence.

That only ap­plies if you’re go­ing to live ex­actly 80 years. If your lifes­pan is some dis­tri­bu­tion which is cen­tered around 80 years, you should grad­u­ally stop car­ing as the fre­quency of the event goes up past 80 years, the amount by which you’ve stopped car­ing de­pend­ing on the dis­tri­bu­tion. It doesn’t go all the way to zero un­til the chance that you’ll live that long is zero.

(Of course, you could re­ply that your chance of liv­ing to some age doesn’t go to ex­actly zero, but all that is nec­es­sary to pre­vent the mug­ging is that it goes down fast enough.)

• As you say, if we use the num­ber 1, then we shouldn’t wear seat­belts, get fire in­surance, or eat healthy to avoid get­ting can­cer, since all of those can be clas­sified as Pas­cal’s Mug­gings. But if we start go­ing for less than one, then we’re just defin­ing away Pas­cal’s Mug­ging by fiat, say­ing “this is the level at which I am will­ing to stop wor­ry­ing about this”.

The point of Pas­cal’s mug­ging is things that have ba­si­cally in­finitely small prob­a­bil­ity. Things that will never hap­pen, ever, ever, once in 3^^^3 uni­verses and pos­si­bly much more. Peo­ple do get in car ac­ci­dents and get can­cer all the time. You shouldn’t ig­nore those prob­a­bil­ities.

Hav­ing a policy of heed­ing small risks like those is fine. Over the course of your life, they add up. There will be a large chance that you will be bet­ter off than not.

But hav­ing a policy of pay­ing the mug­ger, of fol­low­ing ex­pected util­ity in ex­treme cases, will never ever pay off. You will always be worse off than you oth­er­wise would be.

So in that sense it isn’t ar­bi­trary. There is an ac­tual num­ber where ig­nor­ing risks be­low that thresh­old gives you the best me­dian out­come. Fol­low­ing ex­pected util­ity above that thresh­old works out for the best. Fol­low­ing EU on risks be­low the thresh­old is more likely to make you worse off.

If you knew your full prob­a­bil­ity dis­tri­bu­tion of pos­si­ble out­comes, you could ex­actly calcu­late that num­ber.

• A rule-of-thumb I’ve found use for in similar situ­a­tions: There are ap­prox­i­mately ten billion peo­ple al­ive, of whom it’s a safe con­clu­sion that at least one is hav­ing a sub­jec­tive ex­pe­rience that is com­pletely dis­con­nected from ob­jec­tive re­al­ity. There is no way to tell that I’m not that one-in-ten-billion. Thus, I can never be more than one minus one-in-ten-billion sure that my sen­sory ex­pe­rience is even roughly cor­re­lated with re­al­ity. Thus, it would re­quire ex­traor­di­nary cir­cum­stances for me to have any rea­son to worry about any prob­a­bil­ity of less than one-in-ten-billion mag­ni­tude.

There are all sorts of ques­tion­able is­sues with the as­sump­tions and rea­son­ing in­volved; and yet, it seems roughly as helpful as re­mem­ber­ing that I’ve only got around a 99.997% chance of sur­viv­ing the next 24 hours, an­other rule-of-thumb which hand­ily elimi­nates cer­tain prob­a­bil­ity-based prob­lems.

• Thus, I can never be more than one minus one-in-ten-billion sure that my sen­sory ex­pe­rience is even roughly cor­re­lated with re­al­ity. Thus, it would re­quire ex­traor­di­nary cir­cum­stances for me to have any rea­son to worry about any prob­a­bil­ity of less than one-in-ten-billion mag­ni­tude.

No. The rea­son not to spend much time think­ing about the I-am-un­de­tectably-in­sane sce­nario is not, in gen­eral, that it’s ex­traor­di­nar­ily un­likely. The rea­son is that you can’t make good pre­dic­tions about what would be good choices for you in wor­lds where you’re in­sane and to­tally un­able to tell.

This holds even if the prob­a­bil­ity for the sce­nario goes up.

• /​A/​ rea­son not to spend much time think­ing about the I-am-un­de­tectably-in­sane sce­nario is as you de­scribe; how­ever, it’s not the /​only/​ rea­son not to spend much time think­ing about it.

I of­ten have trou­ble ex­plain­ing my­self, and need mul­ti­ple de­scrip­tions of an idea to get a point across, so al­low me to try again:

There is roughly a 30 out of 1,000,000 chance that I will die in the next 24 hours. Over a week, sim­plify­ing a bit, that’s roughly 200 out of 1,000,000 odds of me dy­ing. If I were to buy a 1-in-a-mil­lion lot­tery ticket a week, then, by one rule of thumb, I should be spend­ing 200 times as much of my at­ten­tion on my forth­com­ing demise than I should on buy­ing that ticket and imag­in­ing what to do with the win­nings.

In par­allel, if I am to com­pare two in­de­pen­dent sce­nar­ios, the at-least-one-in-ten-billion odds that I’m hal­lu­ci­nat­ing all this, and the darned-near-zero odds of a Pas­cal’s Mug­ging at­tempt, then I should be spend­ing pro­por­tionately that much more time deal­ing with the Ma­trix sce­nario than that the Pas­cal’s Mug­ging at­tempt is true; which works out to darned-near-zero sec­onds spent both­er­ing with the Mug­ging, no mat­ter how much or how lit­tle time I spend con­tem­plat­ing the Ma­trix.

(There are, of course, al­ter­na­tive view­points which may make it worth spend­ing more time on the low-prob­a­bil­ity sce­nar­ios in each case; for ex­am­ple, buy­ing a lot­tery ticket can be viewed as one of the few low-cost ways to fun­nel money from most of your par­allel-uni­verse selves so that a cer­tain few of your par­allel-uni­verse selves have enough re­sources to work on cer­tain pro­jects that are oth­er­wise in­fea­si­bly ex­pen­sive. But these al­ter­na­tives re­quire care­ful con­sid­er­a­tion and con­struc­tion, at least enough to be able to have enough log­i­cal weight be­hind them to counter the stan­dard rule-of-thumb I’m try­ing to pro­pose here.)

• In par­allel, if I am to com­pare two in­de­pen­dent sce­nar­ios, the at-least-one-in-ten-billion odds that I’m hal­lu­ci­nat­ing all this, and the darned-near-zero odds of a Pas­cal’s Mug­ging at­tempt, then I should be spend­ing pro­por­tionately that much more time deal­ing with the Ma­trix sce­nario than that the Pas­cal’s Mug­ging at­tempt is true

That still sounds wrong. You ap­pear to be de­cid­ing on what to pre­com­pute for purely by prob­a­bil­ity, with­out con­sid­er­ing that some pos­si­ble fu­tures will give you the chance to shift more util­ity around.

If I don’t know any­thing about New­comb’s prob­lem and es­ti­mate a 10% chance of Omega show­ing up and pos­ing it to me to­mor­row, I’ll definitely spend more than 10% of my plan­ning time for to­mor­row read­ing up on and think­ing about it. Why? Be­cause I’ll be able to make far more money in that pos­si­ble fu­ture than the oth­ers, which means that the ex­pected util­ity differ­en­tials are larger, and so it makes sense to spend more re­sources on prepar­ing for it.

The I-am-un­de­tectably-in­sane case is the op­po­site of this, a sce­nario that it’s pretty much im­pos­si­ble to use­fully pre­pare for.

And a PM sce­nario is (at least for an ex­pected-util­ity max­i­mizer) a more ex­treme var­i­ant of my first sce­nario—low prob­a­bil­ities of ridicu­lously large out­comes, that are be­cause of that still worth think­ing about.

• In par­allel, if I am to com­pare two in­de­pen­dent sce­nar­ios, the at-least-one-in-ten-billion odds that I’m hal­lu­ci­nat­ing all this, and the darned-near-zero odds of a Pas­cal’s Mug­ging at­tempt, then I should be spend­ing pro­por­tionately that much more time deal­ing with the Ma­trix sce­nario than that the Pas­cal’s Mug­ging at­tempt is true

That still sounds wrong. You ap­pear to be de­cid­ing on what to pre­com­pute for purely by prob­a­bil­ity, with­out con­sid­er­ing that some pos­si­ble fu­tures will give you the chance to shift more util­ity around.

I agree, but I think I see where DataPacRat is go­ing with his/​her com­ments.

First, it seems as if we only think about the Pas­calian sce­nar­ios that are pre­sented to us. If we are pre­sented with one of these sce­nar­ios, e.g. mug­ging, we should con­sider all other sce­nar­ios of equal or greater ex­pected im­pact.

In ad­di­tion, low prob­a­bil­ity events that we fail to con­sider can pos­si­bly ob­so­lete the dilemma posed by PM. For ex­am­ple, say a mug­ger de­mands your wallet or he will de­stroy the uni­verse. There is a nonzero prob­a­bil­ity that he has the ca­pa­bil­ity to de­stroy the uni­verse, but it is im­por­tant to con­sider the much greater, but still low, prob­a­bil­ity that he dies of a heart at­tack right be­fore your eyes.

• In the I-am-un­de­tectably-in­sane sce­nario, your pre­dic­tions about the wor­lds where you’re in­sane don’t even mat­ter, be­cause your sub­jec­tive ex­pe­rience doesn’t ac­tu­ally take place in those wor­lds any­ways.

• A satis­fac­tory solu­tion should work not only on hu­mans.

• A satis­fac­tory solu­tion should also work if the em pop­u­la­tion ex­plodes to sev­eral quadrillion. As I said, ‘all sorts of ques­tion­able is­sues’; it’s a rule-of-thumb to keep cer­tain trou­bling edge cases from be­ing quite so trou­ble­some, not a fixed ax­iom to use as the foun­da­tion for an eth­i­cal sys­tem that can be used by all sapi­ent be­ings in all cir­cum­stances. Once I learn of some­thing more co­her­ent to use in­stead of the rule-of-thumb, I’ll be happy to use that some­thing-else in­stead.

• Thanks for these thoughts, Kaj.

It’s a worth­while effort to over­come this prob­lem, but let me offer a mode of crit­i­cis­ing it. A lot of peo­ple are not go­ing to want the prin­ci­ples of ra­tio­nal­ity to be con­tin­gent on how long you ex­pect to live. There are a bunch of rea­sons for this. One is that how long you ex­pect to live might not be well-defined. In par­tic­u­lar, some peo­ple will want to say that there’s no right an­swer to the ques­tion of whether you be­come a new per­son each time you wake up in the morn­ing, or each time some of your brain cells die. On the other ex­treme, it might be the case that to a sig­nifi­cant de­gree, some of your life con­tinues af­ter your heart stops beat­ing, ei­ther through your ideas liv­ing on in oth­ers’ minds, or by freez­ing your­self. If you freeze your­self for 1000 years, then wake up again for an­other hun­dred, should the frozen years be in­cluded in defin­ing PESTs, or not? It seems weird that ra­tio­nal­ity should be de­pen­dent on how we for­mal­ise the philos­o­phy of iden­tity in the real world. Why should PESTs be defined based on how long you ex­pect to live, com­pared to how long you ex­pect hu­man­ity as a whole to live, or on the ex­pected life­time of any­thing else that you might care about.

Any­how, de­spite my crit­i­cism, this is an in­ter­est­ing an­swer—cheers for writ­ing this up.

• Thanks!

I un­der­stand that line of rea­son­ing, but to me it feels similar to the philos­o­phy where one thinks that the prin­ci­ples of ra­tio­nal­ity should be to­tally ob­jec­tive and shouldn’t in­volve things like sub­jec­tive prob­a­bil­ities, so then one set­tles on a fre­quen­tist in­ter­pre­ta­tion of prob­a­bil­ity and tries to get rid of sub­jec­tive (Bayesian) prob­a­bil­ities en­tirely. Which doesn’t re­ally work in the real world.

One is that how long you ex­pect to live might not be well-defined. In par­tic­u­lar, some peo­ple will want to say that there’s no right an­swer to the ques­tion of whether you be­come a new per­son each time you wake up in the morn­ing, or each time some of your brain cells die.

But most peo­ple already base their rea­son­ing on an as­sump­tion of be­ing the same per­son to­mor­row; if you se­ri­ously start mak­ing your EU calcu­la­tions based on the as­sump­tion that you’re only go­ing to live for one day or for an even shorter pe­riod, lots of things are go­ing to get weird and bro­ken, even with­out my ap­proach.

It seems weird that ra­tio­nal­ity should be de­pen­dent on how we for­mal­ise the philos­o­phy of iden­tity in the real world. Why should PESTs be defined based on how long you ex­pect to live, com­pared to how long you ex­pect hu­man­ity as a whole to live, or on the ex­pected life­time of any­thing else that you might care about.

It doesn’t seem all that weird to me; ra­tio­nal­ity has always been a tool for us to best achieve the things we care about, so its ex­act form will always be de­pen­dent on the things that we care about. The kinds of deals we’re will­ing to con­sider already de­pend on how long we ex­pect to live. For ex­am­ple, if you offered me a deal that had a 99% chance of kil­ling me on the spot and a 1% chance of giv­ing me an ex­tra 20 years of healthy life, the ra­tio­nal an­swer would be to say “no” if it was offered to me now, but “yes” if it was offered to me when I was on my deathbed.

If you say “ra­tio­nal­ity is de­pen­dent on how we for­mal­ize the philos­o­phy of iden­tity in the real world”, it does sound counter-in­tu­itive, but if you say “you shouldn’t make deals that you never ex­pect to be around to benefit from”, it doesn’t sound quite so weird any­more. If you ex­pected to die in 10 years, you wouldn’t make a deal that would give you lots of money in 30. (Of course it could still be ra­tio­nal if some­one else you cared about would get the money af­ter your death, but let’s as­sume that you could only col­lect the pay­off per­son­ally.)

• Us­ing sub­jec­tive in­for­ma­tion within a de­ci­sion-mak­ing frame­work seems fine. The trou­ble­some part is that the idea of ‘lifes­pan’ is be­ing used to cre­ate the frame­work.

Mak­ing prac­ti­cal de­ci­sions about how long I ex­pect to live seems fine and nor­mal cur­rently. If I want an ice­cream to­mor­row, that’s not con­tin­gent on whether ‘to­mor­row-me’ is the same per­son as I was to­day or a differ­ent one. My lifes­pan is un­cer­tain, and a lot of my val­ues might be fulfilled af­ter it ends. Weird­nesses like the pos­si­bil­ity of be­ing a Bolt­mann brain are tricky, but at least they don’t in­terfere with the ma­chin­ery/​prin­ci­ples of ra­tio­nal­ity—I can still do an ex­pected value calcu­la­tion. Weird­ness on the ob­ject level I can deal with.

Allow­ing ‘lifes­pan’ to in­tro­duce weird­ness into the de­ci­sion-mak­ing frame­work it­self seems less nice. Now, whether my frozen life counts as ‘be­ing al­ive’ is ex­tra im­por­tant. Like be­ing frozen for a long time, or lots of Boltz­mann brains ex­ist­ing could in­terfere with what risks I should be will­ing to ac­cept on this planet, a puz­zle that would re­quire re­s­olu­tion.

• Us­ing sub­jec­tive in­for­ma­tion within a de­ci­sion-mak­ing frame­work seems fine. The trou­ble­some part is that the idea of ‘lifes­pan’ is be­ing used to cre­ate the frame­work.

I’m not sure that the within/​out­side the frame­work dis­tinc­tion is mean­ingful. I feel like the ex­pected life­time com­po­nent is also just an­other vari­able that you plug into the frame­work, similarly to your prob­a­bil­ities and val­ues (and your gen­eral world-model, as­sum­ing that the prob­a­bil­ities don’t come out of nowhere). The ra­tio­nal course of ac­tion already de­pends on the state of the world, and your ex­pected re­main­ing life­time is a part of the state of the world.

I also ac­tu­ally feel that the fact that we’re forced to think about our life­time is a good sign. EU max­i­miza­tion is a tool for get­ting what we want, and Pas­cal’s Mug­ging is a sce­nario where it causes us to do things that don’t get us what we want. If a po­ten­tial an­swer to PM re­veals that EU max­i­miza­tion is bro­ken be­cause it doesn’t prop­erly take into ac­count ev­ery­thing that we want, and forces us to con­sider pre­vi­ously-swept-un­der-the-rug ques­tions about what we do want… then that seems like a sign that the pro­posed an­swer is on the right track.

I have this in­tu­ition that I’m hav­ing slight difficul­ties putting into words… but roughly, EU max­i­miza­tion is a rule of how to be­have in differ­ent situ­a­tions, which ab­stracts over the de­tails of those situ­a­tions while be­ing ul­ti­mately de­rived from them. I feel that at­tempts to re­solve Pas­cal’s Mug­ging by purely “ra­tio­nal” grounds are mostly about try­ing to fol­low a cer­tain aes­thetic that fa­vors de­riv­ing things from purely log­i­cal con­sid­er­a­tions and a pri­ori prin­ci­ples. And that aes­thetic nec­es­sar­ily ends up treat­ing EU max­i­miza­tion as just a for­mal rule, ne­glect­ing to con­sider the ac­tual situ­a­tions it ab­stracts over, and loses sight of the ac­tual pur­pose of the rule, which is to give us good out­comes. If you for­get about try­ing to fol­low the aes­thetic and look at the ac­tual be­hav­ior that some­thing like PEST leads to, you’ll see that it’s the agents who ig­nore PESTs are the ones who ac­tu­ally end up win­ning… which is the thing that should re­ally mat­ter.

If I want an ice­cream to­mor­row, that’s not con­tin­gent on whether ‘to­mor­row-me’ is the same per­son as I was to­day or a differ­ent one.

Really? If I re­ally only cared about the to­mor­row!me for the same amount that I cared about some ran­dom stranger, my be­hav­ior would be a lot differ­ent. I wouldn’t bother with any long-term plans, for one.

Of course, even peo­ple who be­lieve that they’ll be an­other per­son to­mor­row still mostly act the same as ev­ery­one else. One ex­pla­na­tion would be that their im­plicit be­hav­ior doesn’t match their ex­plicit be­liefs… but even if it did match, there would still be a ra­tio­nal case for car­ing about their fu­ture self more than they cared about ran­dom strangers, be­cause the fu­ture self would have more similar val­ues to them than a ran­dom stranger. In par­tic­u­lar, their fu­ture self would be likely to fol­low the same de­ci­sion al­gorithm as they were.

So if they cared about things that hap­pened af­ter their death, it would be rea­son­able to still be­have like they ex­pected their to­tal life­time to be the same as with a more tra­di­tional the­ory of per­sonal iden­tity, and this is the case re­gard­less of whether we’re talk­ing tra­di­tional EU max­i­miza­tion or PEST.

• So, um:

Which ax­iom does this vi­o­late?

• Con­ti­nu­ity and in­de­pen­dence.

Con­ti­nu­ity: Con­sider the sce­nario where each of the [LMN] bets re­fer to one (guaran­teed) out­come, which we’ll also call L, M and N for sim­plic­ity.

Let U(L) = 0, U(M) = 1, U(N) = 10**100

For a sim­ple EU max­i­mizer, you can then satisfy con­ti­nu­ity by pick­ing p=(1-1/​10**100). A PESTI agent, OTOH, may just dis­card a (1-p) of 1/​10**100, which leaves no other op­tions to satisfy it.

The 10**100 value is cho­sen with­out loss of gen­er­al­ity. For PESTI agents that still track prob­a­bil­ities of this mag­ni­tude, in­crease it un­til they don’t.

In­de­pen­dence: Set p to a num­ber small enough that it’s Small Enough To Ig­nore. At that point, the terms for get­ting L and M by that prob­a­bil­ity be­come zero, and you get equal­ity be­tween both sides.

• The­o­ret­i­cally, that’s the ques­tion he’s ask­ing about Pas­cal’s Mug­ging, since ac­cept­ing the mug­ger’s ar­gu­ment would tell you that ex­pected util­ity never con­verges. And since we could rephrase the prob­lem in terms of (say) di­a­mond cre­ation for a di­a­mond max­i­mizer, it does look like an is­sue of prob­a­bil­ity rather than goals.

• The­o­ret­i­cally, that’s the ques­tion he’s ask­ing about Pas­cal’s Mug­ging, since ac­cept­ing the mug­ger’s ar­gu­ment would tell you that ex­pected util­ity never converges

Of course, and the pa­per cited in http://​​wiki.less­wrong.com/​​wiki/​​Pas­cal’s_mug­ging makes that ar­gu­ment rigor­ous.

And since we could rephrase the prob­lem in terms of (say) di­a­mond cre­ation for a di­a­mond max­i­mizer, it does look like an is­sue of prob­a­bil­ity rather than goals.

It’s a prob­lem of ex­pected util­ity, not nec­es­sar­ily prob­a­bil­ity. And I still would like to know which ax­iom it ends up vi­o­lat­ing. I sus­pect Con­ti­nu­ity.

• We can re­place Con­ti­nu­ity with the Archimedean prop­erty (or, ‘You would ac­cept some chance of a bad out­come from cross­ing the street.‘) By my read­ing, this ELU idea triv­ially fol­lows Archimedes by ig­nor­ing the part of a com­pound ‘lot­tery’ that in­volves a suffi­ciently small prob­a­bil­ity. In which case it would vi­o­late In­de­pen­dence, and would do so by treat­ing the two sides as effec­tively equal when the differ­ing out­comes have small enough prob­a­bil­ity.

• I like Scott Aaron­son’s ap­proach for re­solv­ing para­doxes that seem­ingly vi­o­late in­tu­itions—see if the situ­a­tion makes phys­i­cal sense.

Like peo­ple bring up “block­head,” a big lookup table that can hold an in­tel­li­gent con­ver­sa­tion with you for [length of time], and won­der whether this has ram­ifi­ca­tions for the Tur­ing test. But block­head is not re­ally phys­i­cally re­al­iz­able for rea­son­able lengths.

Similarly for cre­at­ing 10^100 happy lives, how ex­actly would you go about do­ing that in our Uni­verse?

• Similarly for cre­at­ing 10^100 happy lives, how ex­actly would you go about do­ing that in our Uni­verse?

By some al­ter­na­tive the­ory of physics that has a, say, .000000000000000000001 prob­a­bil­ity of be­ing true.

• Right, the point is to throw away cer­tain deals. I am sug­gest­ing an­other ap­proach from the OP.

The OP says: ig­nore deals in­volv­ing small num­bers. I say: ig­nore deals that vi­o­late phys­i­cal in­tu­itions (as they are). Where my heuris­tic differs from the OP is my heuris­tic is will­ing to listen to some­one try­ing to sell me the Brook­lyn bridge if I think the story fun­da­men­tally makes sense to me, given how I think physics ought to work. I am wor­ried about long shot cases not for­bid­den by physics ex­plic­itly (which the OP will ig­nore if the shot is long enough). My heuris­tic will fail if hu­mans are miss­ing some­thing im­por­tant about physics, but I am will­ing to bet we are not at this point.

In your ex­am­ple, the OP and I will both re­ject, for differ­ent rea­sons. I be­cause it will vi­o­late my in­tu­ition and the OP be­cause there is a small num­ber in­volved.

• Rel­a­tivity seems to­tally, in­sanely phys­i­cally im­pos­si­ble to me. That doesn’t mean that tak­ing a trillion to one bet on the Michel­son Mor­ley ex­per­i­ment wouldn’t have been a good idea.

• May I recom­mend Feyn­man’s lec­tures then? I am not sure what the point is. Aris­to­tle was a smart guy, but his physics in­tu­ition was pretty awful. I think we are in a good enough state now that I am com­fortable us­ing phys­i­cal prin­ci­ples to rule things out.

Ar­guably quan­tum me­chan­ics is a bet­ter ex­am­ple here than rel­a­tivity. But I think a lot of what makes QM weird isn’t about physics but about the un­der­ly­ing prob­a­bil­ity the­ory be­ing non-stan­dard (similarly to how com­plex num­bers are kinda weird). So, e.g. Bell vi­o­la­tions say there is no hid­den vari­able DAG model un­der­ly­ing QM—but hid­den vari­able DAG mod­els are defined on clas­si­cal prob­a­bil­ities, and am­pli­tudes aren’t clas­si­cal prob­a­bil­ities. Our in­tu­itive no­tion of “hid­den vari­able” is some­how tied to clas­si­cal prob­a­bil­ity.

It all has to bot­tom out some­where—what crite­ria do you use to rule out solu­tions? I think physics is in bet­ter shape to­day than ba­si­cally any other em­piri­cal dis­ci­pline.

• Do you know, off­hand, if Baysian net­works have been ex­tended with com­plex num­bers as prob­a­bil­ities, or (reach­ing here) if you can do be­lief prop­a­ga­tion by pass­ing around qubits in­stead of bits? I’m not sure what I mean by ei­ther of these thing but I’m throw­ing key­words out there to see if any­thing sticks.

• Yes they have, but there is no sin­gle gen­er­al­iza­tion. I am not even sure what con­di­tion­ing should mean.

Scott A is a bet­ter guy to ask.

• I don’t think the con­sen­sus of physi­cists is good enough for you to place that much faith in it. As I un­der­stand mod­ern day cos­mol­ogy, the con­sen­sus view holds that uni­verse once grew by a fac­tor of 10^78 for no rea­son. Would you pass up a 1 penny to \$10,000,000,000 bet that cos­mol­o­gists of the fu­ture will be­lieve cre­at­ing 10^100 happy hu­mans is phys­i­cally pos­si­ble?

what crite­ria do you use to rule out solu­tions?

I don’t know :-(. Cer­tainly I like physics as a darn good heuris­tic, but I don’t think I should re­ject bets with su­per-ex­po­nen­tially good odd based on my un­der­stand­ing of physics. A few bits of in­for­ma­tion from an ex­pert would be enough to con­vince me that I’m wrong about physics, and I don’t think I should re­ject a bet with a pay­out bet­ter than 1 /​ the odds I will see those bits.

• Which par­tic­u­lar event has P = 10^-21? It seems like part of the pas­cal’s mug­ging prob­lem is a type er­ror: We have a util­ity func­tion U(W) over phys­i­cal wor­lds but we’re try­ing to calcu­late ex­pected util­ity over strings of English words in­stead.

Pas­cal’s Mug­ging is a con­struc­tive proof that try­ing to max­i­mize ex­pected util­ity over log­i­cally pos­si­ble wor­lds doesn’t work in any par­tic­u­lar world, at least with the the­o­ries we’ve got now. Any­thing that doesn’t solve re­flec­tive rea­son­ing un­der prob­a­bil­is­tic un­cer­tainty won’t help against Mug­gings promis­ing things from other pos­si­ble wor­lds un­less we just ig­nore the other wor­lds.

• I’d say that if you as­sign a 10^-22 prob­a­bil­ity to a the­ory of physics that al­lows some­body to cre­ate 10^100 happy lives de­pend­ing on your ac­tion, then you do­ing physics wrong.

If you as­sign prob­a­bil­ity 10^-(10^100) to 10^100 lives,10^-(10^1000) to 10^1000 lives, 10^-(10^10000) to 10^10000 lives, and so on, then you are do­ing physics right and you will not fall for Pas­cal’s Mug­ging.

• There seems to be no ob­vi­ous rea­son to as­sume that the prob­a­bil­ity falls ex­actly in pro­por­tion to the num­ber of lives saved.

If GiveWell told me they thought that real-life in­ter­ven­tion A could save one life with prob­a­bil­ity PA and real-life in­ter­ven­tion B could save a hun­dred lives with prob­a­bil­ity PB, I’m pretty sure that di­vid­ing PB by 100 would be the wrong move to make.

• There seems to be no ob­vi­ous rea­son to as­sume that the prob­a­bil­ity falls ex­actly in pro­por­tion to the num­ber of lives saved.

It is an as­sump­tion to make asymp­tot­i­cally (that is, for the tails of the dis­tri­bu­tion), which is rea­son­able due to all the nice prop­er­ties of ex­po­nen­tial fam­ily dis­tri­bu­tions.

If GiveWell told me they thought that real-life in­ter­ven­tion A could save one life with prob­a­bil­ity PA and real-life in­ter­ven­tion B could save a hun­dred lives with prob­a­bil­ity PB, I’m pretty sure that di­vid­ing PB by 100 would be the wrong move to make.

I’m not im­ply­ing that.

EDIT:

As a sim­ple ex­am­ple, if you model the num­ber of lives saved by each in­ter­ven­tion as a nor­mal dis­tri­bu­tion, you are im­mune to Pas­cal’s Mug­gings. In fact, if your util­ity is lin­ear in the num­ber of lives saved, you’ll just need to com­pare the means of these dis­tri­bu­tions and take the max­i­mum. Black swan events at the tails don’t af­fect your de­ci­sion pro­cess.

Us­ing nor­mal dis­tri­bu­tions may be per­haps ap­pro­pri­ate when eval­u­at­ing GiveWell in­ter­ven­tions, but for a gen­eral pur­pose de­ci­sion pro­cess you will have, for each ac­tion, a prob­a­bil­ity dis­tri­bu­tion over pos­si­ble fu­ture world state tra­jec­to­ries, which when com­bined with an util­ity func­tion, will yield a gen­er­ally com­pli­cated and mul­ti­modal dis­tri­bu­tion over util­ity. But as long as the shape of the dis­tri­bu­tion at the tails is nor­mal-like, you wouldn’t be af­fected by Pas­cal’s Mug­gings.

• But it looks like the shape of the dis­tri­bu­tions isn’t nor­mal-like? In fact, that’s one of the stan­dard EA ar­gu­ments for why it’s im­por­tant to spend en­ergy on find­ing the most effec­tive thing you can do: if pos­si­ble in­ter­ven­tion out­comes re­ally were ap­prox­i­mately nor­mally dis­tributed, then your ex­act choice of an in­ter­ven­tion wouldn’t mat­ter all that much. But ac­tu­ally the dis­tri­bu­tion of out­comes looks very skewed; to quote The moral im­per­a­tive to­wards cost-effec­tive­ness:

DCP2 in­cludes cost-effec­tive­ness es­ti­mates for 108 health in­ter­ven­tions, which are pre­sented in the chart be­low, ar­ranged from least effec­tive to most effec­tive [...] This larger sam­ple of in­ter­ven­tions is even more dis­parate in terms of costeffec­tive­ness. The least effec­tive in­ter­ven­tion analysed is still the treat­ment for Ka­posi’s sar­coma, but there are also in­ter­ven­tions up to ten times more cost-effec­tive than ed­u­ca­tion for high risk groups. In to­tal, the in­ter­ven­tions are spread over more than four or­ders of mag­ni­tude, rang­ing from 0.02 to 300 DALYs per \$1,000, with a me­dian of 5. Thus, mov­ing money from the least effec­tive in­ter­ven­tion to the most effec­tive would pro­duce about 15,000 times the benefit, and even mov­ing it from the me­dian in­ter­ven­tion to the most effec­tive would pro­duce about 60 times the benefit.

It can also be seen that due to the skewed dis­tri­bu­tion, the most effec­tive in­ter­ven­tions pro­duce a dis­pro­por­tionate amount of the benefits. Ac­cord­ing to the DCP2 data, if we funded all of these in­ter­ven­tions equally, 80% of the benefits would be pro­duced by the top 20% of the in­ter­ven­tions. [...]

More­over, there have been health in­ter­ven­tions that are even more effec­tive than any of those stud­ied in the DCP2. [...] For in­stance in the case of smal­l­pox, the to­tal cost of erad­i­ca­tion was about \$400 mil­lion. Since more than 100 mil­lion lives have been saved so far, this has come to less than \$4 per life saved — sig­nifi­cantly su­pe­rior to all in­ter­ven­tions in the DCP2.

• I think you mi­s­un­der­stood what I said or I didn’t ex­plain my­self well: I’m not as­sum­ing that the DALY dis­tri­bu­tion ob­tained if you choose in­ter­ven­tions at ran­dom is nor­mal. I’m as­sum­ing that for each in­ter­ven­tion, the DALY dis­tri­bu­tion it pro­duces is nor­mal, with an in­ter­ven­tion-de­pen­dent mean and var­i­ance.

I think that for the kind of in­ter­ven­tions that GiveWell con­sid­ers, this is a rea­son­able as­sump­tion: if the num­ber of DALYs pro­duced by each in­ter­ven­tion is the re­sult of a sum of many roughly in­de­pen­dent vari­ables (e.g. DALYs gained by helping Alice, DALYs gained by helping Bob, etc.) the to­tal should be ap­prox­i­mately nor­mally dis­tributed, due to the cen­tral limit the­o­rem.

For other types of in­ter­ven­tions, e.g. whether to fund a re­search pro­ject, you may want to use a more gen­eral fam­ily of dis­tri­bu­tions that al­lows non-zero skew­ness (e.g. skew-nor­mal dis­tri­bu­tions), but as long as the dis­tri­bu­tion is light-tailed and you don’t use ex­treme val­ues for the pa­ram­e­ters, you would not run into Pas­cal’s Mug­ging is­sues.

• It’s easy if they have ac­cess to run­ning de­tailed simu­la­tions, and while the prob­a­bil­ity that some­one se­cretly has that abil­ity is very low, it’s not nearly as low as the prob­a­bil­ities Kaj men­tioned here.

• It is? How much en­ergy are you go­ing to need to run de­tailed sims of 10^100 peo­ple?

• How do you know you don’t ex­ist in the ma­trix? And that the true uni­verse above ours doesn’t have in­finite com­put­ing power (or huge but bounded, if you don’t be­lieve in in­finity.) How do you know the true laws of physics in our own uni­verse don’t al­low such pos­si­bil­ities?

You can say these things are un­likely. That’s liter­ally speci­fied in the prob­lem. That doesn’t re­solve the para­dox at all though.

• I don’t know, but my heuris­tic says to ig­nore sto­ries that vi­o­late sen­si­ble physics I know about.

• That’s fine. You can just fol­low your in­tu­ition, and that usu­ally won’t lead you too wrong. Usu­ally. How­ever the is­sue here is pro­gram­ming an AI which doesn’t share our in­tu­itions. We need to ac­tu­ally for­mal­ize our in­tu­itions to get it to be­have as we would.

• What crite­rion do you use to rule out solu­tions?

• If you as­sume that the prob­a­bil­ity of some­body cre­at­ing X lives de­creases asymp­tot­i­cally as exp(-X) then you will not ac­cept the deal. In fact, the larger the num­ber they say, the less the ex­pected util­ity you’ll es­ti­mate (as­sum­ing that your util­ity is lin­ear in the num­ber of lives).

It seems to me that such epistemic mod­els are nat­u­ral. Pas­cal’s Mug­ging arises as a thought ex­per­i­ment only if you con­sider ar­bi­trary prob­a­bil­ity dis­tri­bu­tions and ar­bi­trary util­ity func­tions, which in fact may even cause the ex­pec­ta­tions to be­come un­defined in the gen­eral case.

• If you as­sume that the prob­a­bil­ity of some­body cre­at­ing X lives de­creases asymp­tot­i­cally as exp(-X) then you will not ac­cept the deal.

I don’t as­sume this. And I don’t see any rea­son why I should as­sume this. It’s quite pos­si­ble that there ex­ist pow­er­ful ways of simu­lat­ing large num­bers of hu­mans. I don’t think it’s likely, but it’s not liter­ally im­pos­si­ble like you are sug­gest­ing.

Maybe it even is likely. I mean the uni­verse seems quite large. We could the­o­ret­i­cally colonize it and make trillions of hu­mans. By your logic, that is in­cred­ibly im­prob­a­ble. For no other rea­son than that it in­volves a large num­ber. Not that there is any phys­i­cal law that sug­gests we can’t colonize the uni­verse.

• I don’t think it’s likely, but it’s not liter­ally im­pos­si­ble like you are sug­gest­ing.

I’m not say­ing it’s liter­ally im­pos­si­ble, I’m say­ing that its prob­a­bil­ity should de­crease with the num­ber of hu­mans, faster than the num­ber of hu­mans.

Maybe it even is likely. I mean the uni­verse seems quite large. We could the­o­ret­i­cally colonize it and make trillions of hu­mans. By your logic, that is in­cred­ibly im­prob­a­ble. For no other rea­son than that it in­volves a large num­ber.

Not re­ally. I said “asymp­tot­i­cally”. I was con­sid­er­ing the tails of the dis­tri­bu­tion.
We can ob­serve our uni­verse and de­duce the typ­i­cal scale of the stuff in it. Trillion of hu­mans may not be very likely but they don’t ap­pear to be phys­i­cally im­pos­si­ble in our uni­verse. 10^100 hu­mans, on the other hand, are off scale. They would re­quire a phys­i­cal the­ory very differ­ent than ours. Hence we should as­sign to it a van­ish­ingly small prob­a­bil­ity.

• I’m not say­ing it’s liter­ally impossible

1/​3^^^3 is so un­fath­omably huge, you might as well be say­ing it’s liter­ally im­pos­si­ble. I don’t think hu­mans are con­fi­dent enough to as­sign prob­a­bil­ities so low, ever.

10^100 hu­mans, on the other hand, are off scale. They would re­quire a phys­i­cal the­ory very differ­ent than ours. Hence we should as­sign to it a van­ish­ingly small prob­a­bil­ity.

I think EY had the best counter ar­gu­ment. He had a fic­tional sce­nario where a physi­cist pro­posed a new the­ory that was sim­ple and fit all of our data perfectly. But the the­ory also im­plies a new law of physics that could be ex­ploited for com­put­ing power, and would al­low un­fath­omably large amounts of com­put­ing power. And that com­put­ing power could be used to cre­ate simu­lated hu­mans.

There­fore, if it’s true, any­one al­ive to­day has a small prob­a­bil­ity of af­fect­ing large amounts of simu­lated peo­ple. Since that has “van­ish­ingly small prob­a­bil­ity”, the the­ory must be wrong. It doesn’t mat­ter if it’s sim­ple or if it fits the data perfectly.

But it seems like a the­ory that is sim­ple and fits all the data should be very likely. And it seems like all agents with the same knowl­edge, should have the same be­liefs about re­al­ity. Real­ity is to­tally un­car­ing about what our val­ues are. What is true is already so. We should try to model it as ac­cu­rately as pos­si­ble. Not re­fuse to be­lieve things be­cause we don’t like the con­se­quences. That’s ac­tu­ally a log­i­cal fal­lacy.

• 1/​3^^^3 is so un­fath­omably huge, you might as well be say­ing it’s liter­ally im­pos­si­ble. I don’t think hu­mans are con­fi­dent enough to as­sign prob­a­bil­ities so low, ever.

Same thing with num­bers like 10^100 or 3^^^3.

I think EY had the best counter ar­gu­ment. He had a fic­tional sce­nario where a physi­cist pro­posed a new the­ory that was sim­ple and fit all of our data perfectly. But the the­ory also im­plies a new law of physics that could be ex­ploited for com­put­ing power, and would al­low un­fath­omably large amounts of com­put­ing power. And that com­put­ing power could be used to cre­ate simu­lated hu­mans.

EY can imag­ine all the fic­tional sce­nario he wants, this doesn’t mean that we should as­sign non-neg­ligible prob­a­bil­ities to them.

It doesn’t mat­ter if it’s sim­ple or if it fits the data perfectly.

If.

But it seems like a the­ory that is sim­ple and fits all the data should be very likely. And it seems like all agents with the same knowl­edge, should have the same be­liefs about re­al­ity. Real­ity is to­tally un­car­ing about what our val­ues are. What is true is already so. We should try to model it as ac­cu­rately as pos­si­ble. Not re­fuse to be­lieve things be­cause we don’t like the con­se­quences.

If your epistemic model gen­er­ates un­defined ex­pec­ta­tions when you com­bine it with your util­ity func­tion, then I’m pretty sure we can say that at least one of them is bro­ken.

EDIT:

To ex­pand: just be­cause we can imag­ine some­thing and give it a short English de­scrip­tion, it doesn’t mean that it is sim­ple in epistem­i­cal terms. That’s the rea­son why “God” is not a sim­ple hy­poth­e­sis.

• EY can imag­ine all the fic­tional sce­nario he wants, this doesn’t mean that we should as­sign non-neg­ligible prob­a­bil­ities to them.

Not neg­ligible, zero. You liter­ally can not be­lieve in an the­ory of physics that al­lows large amounts of com­put­ing power. If we dis­cover that an ex­ist­ing the­ory like quan­tum physics al­lows us to cre­ate large com­put­ers, we will be forced to aban­don it.

If your epistemic model gen­er­ates un­defined ex­pec­ta­tions when you com­bine it with your util­ity func­tion, then I’m pretty sure we can say that at least one of them is bro­ken.

Yes some­thing is bro­ken, but it’s definitely not our prior prob­a­bil­ities. Some­thing like solomonoff in­duc­tion should gen­er­ate perfectly sen­si­ble pre­dic­tions about the world. If know­ing those pre­dic­tions makes you do weird things, that’s a prob­lem with your de­ci­sion pro­ce­dure. Not the prob­a­bil­ity func­tion.

• Not neg­ligible, zero.

You seem to have a prob­lem with very small prob­a­bil­ities but not with very large num­bers. I’ve also no­ticed this in Scott Alexan­der and oth­ers. If very small prob­a­bil­ities are ze­ros, then very large num­bers are in­fini­ties.

You liter­ally can not be­lieve in an the­ory of physics that al­lows large amounts of com­put­ing power. If we dis­cover that an ex­ist­ing the­ory like quan­tum physics al­lows us to cre­ate large com­put­ers, we will be forced to aban­don it.

Sure. But since we know no such the­ory, there is no a pri­ori rea­son to as­sume it ex­ists with non-neg­ligible prob­a­bil­ity.

Some­thing like Solomonoff in­duc­tion should gen­er­ate perfectly sen­si­ble pre­dic­tions about the world.

Nope, it doesn’t. If you ap­ply Solomonoff in­duc­tion to pre­dict ar­bi­trary in­te­gers, you get un­defined ex­pec­ta­tions.

• Solomonoff in­duc­tion com­bined with an un­bounded util­ity func­tion gives un­defined ex­pec­ta­tions. But Solomonoff in­duc­tion com­bined with a bounded util­ity func­tion can give defined ex­pec­ta­tions.

And Solomonoff in­duc­tion by it­self gives defined pre­dic­tions.

• Solomonoff in­duc­tion com­bined with an un­bounded util­ity func­tion gives un­defined ex­pec­ta­tions. But Solomonoff in­duc­tion com­bined with a bounded util­ity func­tion can give defined ex­pec­ta­tions.

Yes.

And Solomonoff in­duc­tion by it­self gives defined pre­dic­tions.

If you try to use it to es­ti­mate the ex­pec­ta­tion of any un­bounded vari­able, you get an un­defined value.

• Prob­a­bil­ity is a bounded vari­able.

• Yes, but I’m not sure I un­der­stand this com­ment. Why would you want to com­pute an ex­pec­ta­tion of a prob­a­bil­ity?

• Yes I un­der­stand that 3^^^3 is finite. But it’s so un­fath­omably large, it might as well be in­finity to us mere mor­tals. To say an event has 1/​3^^^3 is to say you are cer­tain it will never hap­pen, ever. No mat­ter how much ev­i­dence you are pro­vided. Even if the sky opens up and the voice of god bel­lows to you and says “ya its true”. Even if he comes down and ex­plains why it is true to you, and shows you all the ev­i­dence you can imag­ine.

The word “neg­ligible” is ob­scur­ing your true mean­ing. There is a mas­sive—no, un­fath­omable—differ­ence be­tween 1/​3^^^3 and “small” num­bers like 1/​10^80 (1 di­vided by the num­ber of atoms in the uni­verse.)

To use this method is to say there are hy­pothe­ses with rel­a­tively short de­scrip­tions which you will re­fuse to be­lieve. Not just about mug­gers, but even sim­ple things like the­o­ries of physics which might al­low large amounts of com­put­ing power. Us­ing this method, you might be forced to be­lieve vastly more com­pli­cated and ar­bi­trary the­o­ries that fit the data worse.

If you ap­ply Solomonoff in­duc­tion to pre­dict ar­bi­trary in­te­gers, you get un­defined ex­pec­ta­tions.

Solomonoff in­duc­tions pre­dic­tions will be perfectly rea­son­able, and I would trust them far more than any other method you can come up with. What you choose to do with the pre­dic­tions could gen­er­ate non­sense re­sults. But that’s not a flaw with SI, but with your method.

• Point, but not a hard one to get around.

There is a the­o­ret­i­cal lower bound on en­ergy per com­pu­ta­tion, but it’s ex­tremely small, and the timescale they’ll be run in isn’t speci­fied. Also, un­less Scott Aaron­son’s spec­u­la­tive con­scious­ness-re­quires-quan­tum-en­tan­gle­ment-de­co­her­ence the­ory of iden­tity is true, there are ways to use re­versible com­put­ing to get around the lower bounds and achieve the­o­ret­i­cally limitless com­pu­ta­tion as long as you don’t need it to out­put re­sults. Hav­ing that be ex­tant adds im­prob­a­bil­ity, but not much on the scale we’re talk­ing about.

• I’ll need some back­ground here. Why aren’t bounded util­ities the de­fault as­sump­tion? You’d need some ex­traor­di­nary ar­gu­ments to con­vince me that any­one has an un­bounded util­ity func­tion. Yet this post and many oth­ers on LW seem to im­plic­itly as­sume un­bounded util­ity func­tions.

• 1) We don’t need an un­bounded util­ity func­tion to demon­strate Pas­cal’s Mug­ging. Plain old large num­bers like 10^100 are enough.

2) It seems rea­son­able for util­ity to be lin­ear in things we care about, e.g. hu­man lives. This could run into a prob­lem with non-unique­ness, i.e., if I run an iden­ti­cal com­puter pro­gram of you twice, maybe that shouldn’t count as two. But I think this is suffi­ciently murky as to not make bounded util­ity clearly cor­rect.

• We don’t need an un­bounded util­ity func­tion to demon­strate Pas­cal’s Mug­ging. Plain old large num­bers like 10^100 are enough.

The scale is ar­bi­trary. If your util­ity func­tion is de­signed such that util­ity for com­mon sce­nario are not very small com­pared to the max­i­mum util­ity then you wouldn’t have Pas­cal’s Mug­gings.

It seems rea­son­able for util­ity to be lin­ear in things we care about, e.g. hu­man lives.

Does any­body re­ally have lin­ear prefer­ences in any­thing? This seems at odds with em­piri­cal ev­i­dence.

• Like V_V, I don’t find it “rea­son­able” for util­ity to be lin­ear in things we care about.

I will write a dis­cus­sion topic about the is­sue shortly.

• Why aren’t bounded util­ities the de­fault as­sump­tion?

Be­cause here the de­fault util­ity is the one speci­fied by the Von Neu­mann-Mor­gen­stern the­o­rem and there is no re­quire­ment (or in­di­ca­tion) that it is bounded.

Hu­mans, of course, don’t op­er­ate ac­cord­ing to VNM ax­ioms, but most of LW thinks it’s a bug to be fixed X-/​

• But VNM the­ory al­lows for bounded util­ity func­tions, so if we are de­sign­ing an agent why don’t de­sign it with a bounded util­ity func­tion?

It would sys­tem­at­i­cally solve Pas­cal’s Mug­ging, and more for­mally, it would pre­vent the ex­pec­ta­tions to ever be­come un­defined.

• VNM the­ory al­lows for bounded util­ity functions

Does it? As far as I know, all it says is that the util­ity func­tion ex­ists. Maybe it’s bounded or maybe not—VNM does not say.

It would sys­tem­at­i­cally solve Pas­cal’s Mugging

I don’t think it would be­cause the bounds are ar­bi­trary and if you make them wide enough, Pas­cal’s Mug­ging will still work perfectly well.

• Does it? As far as I know, all it says is that the util­ity func­tion ex­ists. Maybe it’s bounded or maybe not—VNM does not say.

VNM main the­o­rem proves that if you have a set of prefer­ences con­sis­tent with some re­quire­ments, then an util­ity func­tion ex­ists such that max­i­miz­ing its ex­pec­ta­tion satis­fies your prefer­ences.

If you are de­sign­ing an agent ex novo, you can choose a bounded util­ity func­tion. This re­stricts the set of al­lowed prefer­ences, in a way that es­sen­tially pre­vents Pas­cal’s Mug­ging.

I don’t think it would be­cause the bounds are ar­bi­trary and if you make them wide enough, Pas­cal’s Mug­ging will still work perfectly well.

Yes, but if the ex­pected util­ity for com­mon sce­nar­ios is not very far from the bounds, then Pas­cal’s Mug­ging will not ap­ply.

• you can choose a bounded util­ity func­tion. This re­stricts the set of al­lowed preferences

How does that work? VNM prefer­ences are ba­si­cally or­der­ing or rank­ing. What kind of VNM prefer­ences would be dis­al­lowed un­der a bounded util­ity func­tion?

if the ex­pected util­ity for com­mon sce­nar­ios is not very far from the bounds, then Pas­cal’s Mug­ging will not apply

Are you say­ing that you can/​should set the bounds nar­rowly? You lose your abil­ity to cor­rectly re­act to rare events, then—and black swans are VERY in­fluen­tial.

• VNM prefer­ences are ba­si­cally or­der­ing or rank­ing.

Only in the de­ter­minis­tic case. If you have un­cer­tainty, this doesn’t ap­ply any­more: util­ity is in­var­i­ant to pos­i­tive af­fine trans­forms, not to ar­bi­trary mono­tone trans­forms.

What kind of VNM prefer­ences would be dis­al­lowed un­der a bounded util­ity func­tion?

Any risk-neu­tral (or risk-seek­ing) prefer­ence in any quan­tity.

• If you have un­cer­tainty, this doesn’t ap­ply anymore

I am not sure I un­der­stand. Uncer­tainty in what? Plus, if you are go­ing be­yond the VNM The­o­rem, what is the util­ity func­tion we’re talk­ing about, any­way?

• I am not sure I un­der­stand. Uncer­tainty in what?

In the out­come of each ac­tion. If the world is de­ter­minis­tic, then all that mat­ters is a prefer­ence rank­ing over out­comes. This is called or­di­nal util­ity.

If the out­comes for each ac­tion are sam­pled from some ac­tion-de­pen­dent prob­a­bil­ity dis­tri­bu­tion, then a sim­ple rank­ing isn’t enough to ex­press your prefer­ences. VNM the­ory al­lows you to spec­ify a car­di­nal util­ity func­tion, which is in­var­i­ant only up to pos­i­tive af­fine trans­form.

In prac­tice this is needed to model com­mon hu­man prefer­ences like risk-aver­sion w.r.t. money.

• If the out­comes for each ac­tion are sam­pled from some ac­tion-de­pen­dent prob­a­bil­ity dis­tri­bu­tion, then a sim­ple rank­ing isn’t enough to ex­press your prefer­ence.

Yes, you need risk tol­er­ance /​ risk prefer­ence as well, but once we have that, aren’t we already out­side of the VNM uni­verse?

• No, risk tol­er­ance /​ risk prefer­ence can be mod­eled with VNM the­ory.

• Con­sis­tent risk prefer­ences can be en­cap­su­lated in the shape of the util­ity func­tion—prefer­ring a cer­tain \$40 to a half chance of \$100 and half chance of noth­ing, for ex­am­ple, is ac­com­plished by a broad class of util­ity func­tions. Prefer­ences on prob­a­bil­ities—treat­ing 95% as differ­ent than mid­way be­tween 90% and 100%--can­not be ex­pressed in VNM util­ity, but that seems like a fea­ture, not a bug.

• In prin­ci­ple, util­ity non-lin­ear in money pro­duces var­i­ous amounts of risk aver­sion or risk seek­ing. How­ever, this fun­da­men­tal pa­per proves that ob­served lev­els of risk aver­sion can­not be thus ex­plained. The re­sults have been gen­er­al­ised here to a class of prefer­ence the­o­ries broader than ex­pected util­ity.

• How­ever, this fun­da­men­tal pa­per proves that ob­served lev­els of risk aver­sion can­not be thus ex­plained.

This pa­per has come up be­fore, and I still don’t think it proves any­thing of the sort. Yes, if you choose crazy in­puts a sen­si­ble func­tion will have crazy out­puts—why did this get pub­lished?

In gen­eral, prospect the­ory is a bet­ter de­scrip­tive the­ory of hu­man de­ci­sion-mak­ing, but I think it makes for a ter­rible nor­ma­tive the­ory rel­a­tive to util­ity the­ory. (This is why I speci­fied con­sis­tent risk prefer­ences—yes, you can’t ex­press trans­ac­tion or prob­a­bil­is­tic fram­ing effects in util­ity the­ory. As said in the grand­par­ent, that seems like a fea­ture, not a bug.)

• Be­cause here the de­fault util­ity is the one speci­fied by the Von Neu­mann-Mor­gen­stern the­o­rem and there is no re­quire­ment (or in­di­ca­tion) that it is bounded.

Ex­cept, the VNM the­o­rem in the form given ap­plies to situ­a­tions with finitely many pos­si­bil­ities. If there are in­finitely many pos­si­bil­ities, then the gen­er­al­ized the­o­rem does re­quire bounded util­ity. This fol­lows from pre­cisely the Pas­cal’s mug­ging-type ar­gu­ments like the ones be­ing con­sid­ered here.

• (And with finitely many pos­si­bil­ities, the util­ity func­tion can­not pos­si­bly be un­bounded, be­cause any finite set of re­als has a max­i­mum.)

• Ex­cept, the VNM the­o­rem in the form given ap­plies to situ­a­tions with finitely many pos­si­bil­ities.

In the page cited a proof out­line is given for the finite case, but the the­o­rem it­self has no such re­stric­tion, whether “in the form given” or, well, the the­o­rem it­self.

If there are in­finitely many pos­si­bil­ities, then the gen­er­al­ized the­o­rem does re­quire bounded util­ity.

What are you refer­ring to as the gen­er­al­ised the­o­rem? Some­thing other than the one that VNM proved? That cer­tainly does not re­quire or as­sume bounded util­ity.

This fol­lows from pre­cisely the Pas­cal’s mug­ging-type ar­gu­ments like the ones be­ing con­sid­ered here.

If you’re refer­ring to the is­sue in the pa­per that en­tirelyuse­less cited, Lu­mifer cor­rectly pointed out that it is out­side the set­ting of VNM (and some­one down­voted him for it).

The pa­per does raise a real is­sue, though, for the set­ting it dis­cusses. Bound­ing the util­ity is one of sev­eral pos­si­bil­ities that it briefly men­tions to sal­vage the con­cept.

The pa­per is also use­ful in clar­ify­ing the real prob­lem of Pas­cal’s Mug­ger. It is not that you will give all your money away to strangers promis­ing 3^^^3 util­ity. It is that the calcu­la­tion of util­ity in that set­ting is dom­i­nated by ex­tremes of re­mote pos­si­bil­ity of vast pos­i­tive and nega­tive util­ity, and nowhere con­verges.

Physi­cists ran into some­thing of the sort in quan­tum me­chan­ics, but I don’t know if the similar­ity is any more than su­perfi­cial, or if the meth­ods they worked out to deal with it have any analogue here.

• What are you refer­ring to as the gen­er­al­ised the­o­rem?

Try this:

The­o­rem: Us­ing the no­ta­tion from here, ex­cept we will al­low lot­ter­ies to have in­finitely many out­comes as long as the prob­a­bil­ities sum to 1.

If an or­der­ing satis­fies the four ax­ioms of com­plete­ness, tran­si­tivity, con­ti­nu­ity, and in­de­pen­dence, and the fol­low­ing ad­di­tional ax­iom:

Ax­iom (5): Let L = Sum(i=0...in­finity, p_i M_i) with Sum(i=0...in­finity, p_i)=1 and N >= Sum(i=0...n, p_i M_i)/​Sum(i=0...n, p_i) then N >= L. And similarly with the ar­rows re­versed.

An agent satis­fy­ing ax­ioms (1)-(5) has prefer­ences given by a bounded util­ity func­tion u such that, L>M iff Eu(L)>Eu(M).

Edit: fixed for­mat­ting.

• Ax­iom (5): Let L = Sum(i=0...in­finity, pi Mi) with Sum(i=0...in­finity, pi)=1 and N >= Sum(i=0...n, pi Mi)/​Sum(i=0...n, pi) then N >= L. And similarly with the ar­rows re­versed.

That ap­pears to be an ax­iom that prob­a­bil­ities go to zero enough faster than util­ities that to­tal util­ity con­verges (in a set­ting in which the sure out­comes are a countable set). It lacks some­thing in pre­ci­sion of for­mu­la­tion (e.g. what is be­ing quan­tified over, and in what or­der?) but it is fairly clear what it is do­ing. There’s noth­ing like it in VNM’s book or the Wiki ar­ti­cle, though. Where does it come from?

Yes, in the same way that VNM’s ax­ioms are just what is needed to get af­fine util­ities, an ax­iom some­thing like this will give you bounded util­ities. Does the ax­iom have any in­tu­itive ap­peal, sep­a­rate from it pro­vid­ing that con­se­quence? If not, the ax­iom does not provide a jus­tifi­ca­tion for bounded util­ities, just an in­di­rect way of get­ting them, and you might just as well add an ax­iom say­ing straight out that util­ities are bounded.

None of which solves the prob­lem that en­tirelyuse­less cited. The above ax­iom for­bids the Solomonoff prior (for which pi Mi grows with busy beaver fast­ness), but does not sug­gest any re­place­ment uni­ver­sal prior.

• That ap­pears to be an ax­iom that prob­a­bil­ities go to zero enough faster than util­ities that to­tal util­ity con­verges (in a set­ting in which the sure out­comes are a countable set).

No, the ax­iom doesn’t put any con­straints on the prob­a­bil­ity dis­tri­bu­tion. It merely con­strains prefer­ences, speci­fi­cally it says that prefer­ences for in­finite lot­ter­ies should be the ‘limits’ of the prefer­ence for finite lot­ter­ies. One can think of it as a slightly stronger ver­sion of the fol­low­ing:

Ax­iom (5′): Let L = Sum(i=0...in­finity, p_i M_i) with Sum(i=0...in­finity, p_i)=1. Then if for all i N>=M_I then N>=L. And similarly with the ar­rows re­versed. (In other words if N is preferred over ev­ery el­e­ment of a lot­tery then N is preferred over the lot­tery.)

In fact, I’m pretty sure that ax­iom (5′) is strong enough, but I haven’t worked out all the de­tails.

It lacks some­thing in pre­ci­sion of for­mu­la­tion (e.g. what is be­ing quan­tified over, and in what or­der?)

Sorry, there were some for­mat­ting prob­lems, hope­fully it’s bet­ter now.

(for which p_i M_i [for­mat­ting fixed] grows with busy beaver fast­ness)

The M_i’s are lot­ter­ies that the agent has prefer­ences over, not util­ity val­ues. Thus it doesn’t a pri­ori make sense to talk about its growth rate.

• I think I un­der­stand what the ax­iom is do­ing. I’m not sure it’s strong enough, though. There is no guaran­tee that there is any N that is >= M_i for all i (or for all large enough i, a weaker ver­sion which I think is what is needed), nor an N that is ⇐ them. But sup­pose there are such an up­per Nu and a lower Nl, thus giv­ing a con­tin­u­ous range be­tween them of Np = p Nl + (1-p) Nu for all p in 0..1. There is no guaran­tee that the supre­mum of those p for which Np is a lower bound is equal to the in­fi­mum of those for which it is an up­per bound. The ax­iom needs to stipu­late that lower and up­per bounds Nl and Nu ex­ist, and that there is no gap in the be­havi­ours of the fam­ily Np.

One also needs some ax­ioms to the effect that a for­mal in­finite sum Sum{i>=0: pi Mi} ac­tu­ally be­haves like one, oth­er­wise “Sum” is just a sug­ges­tively named but un­in­ter­preted sym­bol. Such ax­ioms might be in­var­i­ance un­der per­mu­ta­tion, equiv­alence to a finite weighted av­er­age when only finitely many pi are nonzero, and dis­tri­bu­tion of the mix­ture pro­cess to the com­po­nents for in­finite lot­ter­ies hav­ing the same se­quence of com­po­nent lot­ter­ies. I’m not sure that this is yet strong enough.

The task these ax­ioms have to perform is to uniquely ex­tend the prefer­ence re­la­tion from finite lot­ter­ies to in­finite lot­ter­ies. It may be pos­si­ble to do that, but hav­ing thought for a while and not come up with a suit­able set of ax­ioms, I looked for a coun­terex­am­ple.

Con­sider the situ­a­tion in which there is ex­actly one sure-thing lot­tery M. The in­finite lot­ter­ies, with the ax­ioms I sug­gested in the sec­ond para­graph, can be iden­ti­fied with the prob­a­bil­ity dis­tri­bu­tions over the non-nega­tive in­te­gers, and they are equiv­a­lent when they are per­mu­ta­tions of each other. All of the dis­tri­bu­tions with finite sup­port (call these the finite lot­ter­ies) are equiv­a­lent to M, and must be as­signed the same util­ity, call it u. Take any dis­tri­bu­tion with in­finite sup­port, and as­sign it an ar­bi­trary util­ity v. This de­ter­mines the util­ity of all lot­ter­ies that are weighted av­er­ages of that one with M. But that won’t cover all lot­ter­ies yet. Take an­other one and give it an ar­bi­trary util­ity w. This de­ter­mines the util­ity of some more lot­ter­ies. And so on. I don’t think any in­con­sis­tency is go­ing to arise. This al­lows for in­finitely many differ­ent prefer­ence or­der­ings, and hence in­finitely many differ­ent util­ity func­tions.

The con­struc­tion is some­what analo­gous to con­struct­ing an ad­di­tive func­tion from re­als to re­als, i.e. one satis­fy­ing f(a+b) = f(a) + f(b). The only con­tin­u­ous ad­di­tive func­tions are mul­ti­pli­ca­tion by a con­stant, but there are in­finitely many non-con­tin­u­ous ad­di­tive func­tions.

An al­ter­na­tive ap­proach would be to first take any prefer­ence or­der­ing con­sis­tent with the ax­ioms, then use the VNM ax­ioms to con­struct a util­ity func­tion for that prefer­ence or­der­ing, and then to im­pose an ax­iom about the be­havi­our of that util­ity func­tion, be­cause once we have util­ities it’s easy to talk about limits. The most straight­for­ward such ax­iom would be to stipu­late that U( Sum{i>=0: pi Mi} ) = Sum{i>=0: pi U(Mi)}, where the sum on the right hand side is an or­di­nary in­finite sum of real num­bers. The ax­iom would re­quire this to con­verge.

This ax­iom has the im­me­di­ate con­se­quence that util­ities are bounded, for if they were not, then for any prob­a­bil­ity dis­tri­bu­tion {i>=0: pi} with in­finite sup­port, one could choose a se­quence of lot­ter­ies whose util­ities grew fast enough that Sum{i>=0: pi U(Mi)} would fail to con­verge.

Per­son­ally, I am not con­vinced that bounded util­ity is the way to go to avoid Pas­cal’s Mug­ging, be­cause I see no prin­ci­pled way to choose the bound. The larger you make it, the more Mug­gings you are vuln­er­a­ble to, but the smaller you make it, the more low-hang­ing fruit you will ig­nore: sub­stan­tial chances of stu­pen­dous re­wards.

In one of Eliezer’s talks, he makes a point about how bad an ex­is­ten­tial risk to hu­man­ity is. It must be mea­sured not by the num­ber of peo­ple who die in it when it hap­pens, but the loss of a po­ten­tially enor­mous fu­ture of hu­man­ity spread­ing to the stars. That is the real differ­ence be­tween “only” 1 billion of us dy­ing, and all 7 billion. If you are moved by this ar­gu­ment, you must see a sub­stan­tial gap be­tween the welfare of 7 billion peo­ple and that of how­ever many 10^n you fore­see if we avoid these risks. That already gives sub­stan­tial head­room for Mug­gings.

• I think I un­der­stand what the ax­iom is do­ing. I’m not sure it’s strong enough, though. There is no guaran­tee that there is any N that is >= M_i for all i (or for all large enough i, a weaker ver­sion which I think is what is needed), nor an N that is ⇐ them.

The M_i’s can them­selves be lot­ter­ies. The idea is to group events into finite lot­ter­ies so that the M_i’s are >= N.

Per­son­ally, I am not con­vinced that bounded util­ity is the way to go to avoid Pas­cal’s Mug­ging, be­cause I see no prin­ci­pled way to choose the bound.

There is no prin­ci­pled way to chose util­ity func­tions ei­ther, yet peo­ple seem to be fine with them.

My point is that if one takes the VNM the­ory se­ri­ously as jus­tifi­ca­tion for hav­ing a util­ity func­tion, the same logic means it must be bounded.

• There is no prin­ci­pled way to chose util­ity func­tions ei­ther, yet peo­ple seem to be fine with them.

The VNM ax­ioms are the prin­ci­pled way. That’s not to say that it’s a way I agree with, but it is a prin­ci­pled way. The ax­ioms are the prin­ci­ples, cod­ify­ing an idea of what it means for a set of prefer­ences to be ra­tio­nal. Prefer­ences are as­sumed given, not cho­sen.

My point is that if one takes the VNM the­ory se­ri­ously as jus­tifi­ca­tion for hav­ing a util­ity func­tion, the same logic means it must be bounded.

Bound­ed­ness does not fol­low from the VNM ax­ioms. It fol­lows from VNM plus an ad­di­tional con­struc­tion of in­finite lot­ter­ies, plus ad­di­tional ax­ioms about in­finite lot­ter­ies such as those we have been dis­cussing. Ba­si­cally, if util­ities are un­bounded, then there are St. Peters­burg-style in­finite lot­ter­ies with di­ver­gent util­ities; if all in­finite lot­ter­ies are re­quired to have defined util­ities, then util­ities are bounded.

This is in­deed a prob­lem. Either util­ities are bounded, or some in­finite lot­ter­ies have no defined value. When prob­a­bil­ities are given by al­gorith­mic prob­a­bil­ity, the situ­a­tion is even worse: if util­ities are un­bounded then no ex­pected utiilties are defined.

But the prob­lem is not solved by say­ing, “util­ities must be bounded then”. Per­haps util­ities must be bounded. Per­haps Solomonoff in­duc­tion is the wrong way to go. Per­haps in­finite lot­ter­ies should be ex­cluded. (Fini­tists would go for that one.) Per­haps some more fun­da­men­tal change to the con­cep­tual struc­ture of ra­tio­nal ex­pec­ta­tions in the face of un­cer­tainty is called for.

• The VNM ax­ioms are the prin­ci­pled way.

They show that you must have a util­ity func­tion, not what it should be.

Bound­ed­ness does not fol­low from the VNM ax­ioms. It fol­lows from VNM plus an ad­di­tional con­struc­tion of in­finite lot­ter­ies, plus ad­di­tional ax­ioms about in­finite lot­ter­ies such as those we have been dis­cussing.

Well the ad­di­tional ax­iom is as in­tu­itive as the VNM ones, and you need in­finite lot­ter­ies if you are too model a world with in­finite pos­si­bil­ities.

Per­haps Solomonoff in­duc­tion is the wrong way to go.

This amounts to re­ject­ing com­plete­ness. Sup­pose omega offered to cre­ate a uni­verse based on a Solomonoff prior, you’d have to way to eval­u­ate this pro­posal.

• The VNM ax­ioms are the prin­ci­pled way.

They show that you must have a util­ity func­tion, not what it should be.

Given your prefer­ences, they do show what your util­ity func­tion should be (up to af­fine trans­for­ma­tion).

Well the ad­di­tional ax­iom is as in­tu­itive as the VNM ones, and you need in­finite lot­ter­ies if you are too model a world with in­finite pos­si­bil­ities.

You need some, but not all of them.

This amounts to re­ject­ing com­plete­ness.

By com­plete­ness I as­sume you mean as­sign­ing a finite util­ity to ev­ery lot­tery, in­clud­ing the in­finite ones. Why not re­ject com­plete­ness? The St. Peters­burg lot­tery is plainly one that can­not ex­ist. I there­fore see no need to as­sign it any util­ity.

Bounded util­ity does not solve Pas­cal’s Mug­ging, it merely offers an un­easy com­pro­mise be­tween be­ing mugged by re­mote promises of large pay­offs and pass­ing up un­re­mote pos­si­bil­ities of large pay­offs.

Sup­pose omega offered to cre­ate a uni­verse based on a Solomonoff prior, you’d have to way to eval­u­ate this pro­posal.

I don’t care. This is a ques­tion I see no need to have any an­swer to. But why in­voke Omega? The Solomonoff prior is already put for­ward by some as a uni­ver­sal prior, and it is already known to have prob­lems with un­bounded util­ity. As far as I know this prob­lem is still un­solved.

• Given your prefer­ences, they do show what your util­ity func­tion should be (up to af­fine trans­for­ma­tion).

As­sum­ing your prefer­ences satisfy the ax­ioms.

By com­plete­ness I as­sume you mean as­sign­ing a finite util­ity to ev­ery lot­tery, in­clud­ing the in­finite ones.

No, by com­plete­ness I mean that for any two lot­ter­ies you pre­fer one over the other.

Why not re­ject com­plete­ness?

So why not re­ject it in the finite case as well?

The St. Peters­burg lot­tery is plainly one that can­not ex­ist.

Care to as­sign a prob­a­bil­ity to that state­ment.

• So why not re­ject it in the finite case as well?

Ac­tu­ally, I would, but that’s di­gress­ing from the sub­ject of in­finite lot­ter­ies. As I have been point­ing out, in­finite lot­ter­ies are out­side the scope of the VNM ax­ioms and need ad­di­tional ax­ioms to be defined. It seems no more rea­son­able to me to re­quire com­plete­ness of the prefer­ence or­der­ing over St. Peters­burg lot­ter­ies than to re­quire that all se­quences of real num­bers con­verge.

Care to as­sign a prob­a­bil­ity to that state­ment.

“True.” At some point, prob­a­bil­ity always be­comes sub­or­di­nate to logic, which knows only 0 and 1. If you can come up with a sys­tem in which it’s prob­a­bil­ities all the way down, write it up for a math­e­mat­ics jour­nal.

If you’re go­ing to cite this (which makes a valid point, but peo­ple usu­ally re­peat the pass­word in place of un­der­stand­ing the idea), tell me what prob­a­bil­ity you as­sign to A con­di­tional on A, to 1+1=2, and to an om­nipo­tent God be­ing able to make a weight so heavy he can’t lift it.

• “True.” At some point, prob­a­bil­ity always be­comes sub­or­di­nate to logic, which knows only 0 and 1. If you can come up with a sys­tem in which it’s prob­a­bil­ities all the way down, write it up for a math­e­mat­ics jour­nal.

Ok, so care to pre­sent an a pri­ori pure logic ar­gu­ment for why St. Peters­burg lot­tery-like situ­a­tions can’t ex­ist.

• Ok, so care to pre­sent an a pri­ori pure logic ar­gu­ment for why St. Peters­burg lot­tery-like situ­a­tions can’t ex­ist.

FInite ap­prox­i­ma­tions to the St. Peters­burg lot­tery have un­bounded val­ues. The se­quence does not con­verge to a limit.

In con­trast, a se­quence of in­di­vi­d­ual gam­bles with ex­pec­ta­tions 1, 12, 14, etc. does have a limit, and it is rea­son­able to al­low the ideal­ised in­finite se­quence of them a place in the set of lot­ter­ies.

You might as well ask why the sum of an in­finite num­ber of ones doesn’t ex­ist. There are ways of ex­tend­ing the real num­bers with var­i­ous sorts of in­finite num­bers, but they are ex­ten­sions. The real num­bers do not in­clude them. The difficulty of de­vis­ing an ex­ten­sion that al­lows for the con­ver­gence of all in­finite sums is not an ar­gu­ment that the real num­bers should be bounded.

• FInite ap­prox­i­ma­tions to the St. Peters­burg lot­tery have un­bounded val­ues. The se­quence does not con­verge to a limit.

They have un­bounded ex­pected val­ues, that doesn’t mean the St. Peters­burg lot­tery can’t ex­ist, only that its ex­pected value doesn’t.

• If there are in­finitely many pos­si­bil­ities, then the gen­er­al­ized the­o­rem does re­quire bounded util­ity.

I am not sure I un­der­stand. Link?

• Our main re­sult im­plies that if you have an un­bounded, per­cep­tion de­ter­mined, com­putable util­ity func­tion, and you use a Solomonoff-like prior (Solomonoff, 1964), then you have no way to choose be­tween poli­cies us­ing ex­pected util­ity.

So, it’s within the AIXI con­text and you feed your util­ity func­tion in­finite (!) se­quences of “per­cep­tions”.

We’re not in VNM land any more.

• I think this sim­plifies. Not sure, but here’s the rea­son­ing:

L (or ex­pected L) is a con­se­quence of S, so not an in­de­pen­dent pa­ram­e­ter. If R=1, then is this me­dian max­i­mal­i­sa­tion?http://​​less­wrong.com/​​r/​​dis­cus­sion/​​lw/​​mqa/​​me­dian_util­ity_rather_than_mean/​​ it feels close to that, any­way.

I’ll think some more...

• This is just a com­pli­cated way of say­ing, “Let’s use bounded util­ity.” In other words, the fact that peo­ple don’t want to take deals where they will over­all ex­pect to get noth­ing out of it (in fact), means that they don’t value bets of that kind enough to take them. Which means they have bounded util­ity. Bounded util­ity is the cor­rect re­ponse to PM.

• This is just a com­pli­cated way of say­ing, “Let’s use bounded util­ity.”

But noth­ing about the ap­proach im­plies that our util­ity func­tions would need to be bounded when con­sid­er­ing deals in­volv­ing non-PEST prob­a­bil­ities?

• If you don’t want to vi­o­late the in­de­pen­dence ax­iom (which per­haps you did), then you will need bounded util­ity also when con­sid­er­ing deals with non-PEST prob­a­bil­ities.

In any case, if you effec­tively give prob­a­bil­ity a lower bound, un­bounded util­ity doesn’t have any spe­cific mean­ing. The whole point of a dou­ble util­ity is that you will be will­ing to ac­cept the dou­ble util­ity with half the prob­a­bil­ity. Once you won’t ac­cept it with half the prob­a­bil­ity (as will hap­pen in your situ­a­tion) there is no point in say­ing that some­thing has twice the util­ity.

• It’s weird, but it’s not quite the same as bounded util­ity (though it looks pretty similar). In par­tic­u­lar, there’s still a point in say­ing it has dou­ble the util­ity even though you some­times won’t ac­cept it at half the util­ity. Note the caveat “some­times”: at other times, you will ac­cept it.

Sup­pose event X has util­ity U(X) = 2 U(Y). Nor­mally, you’ll ac­cept it in­stead of Y at any­thing over half the prob­a­bil­ity. But if you re­duce the prob­a­bil­ities of both* events enough, that changes. If you sim­ply had a bound on util­ity, you would get a differ­ent be­hav­ior: you’d always ac­cept X and over half the prob­a­bil­ity of Y for any P(Y), un­less the util­ity of Y was too high. Th­ese be­hav­iors are both fairly weird (ex­cept in the uni­verse where there’s no pos­si­ble con­struc­tion of an out­come with dou­ble the util­ity of Y, or the uni­verse where you can’t con­struct a suffi­ciently low prob­a­bil­ity for some rea­son), but they’re not the same.

• Ok. This is math­e­mat­i­cally cor­rect, ex­cept that bounded util­ity means that if U(Y) is too high, U(X) can­not have a dou­ble util­ity, which means that the be­hav­ior is not so weird any­more. So in this case my ques­tion is why Kaj sug­gests his pro­posal in­stead of us­ing bounded util­ity. Bounded util­ity will pre­serve the thing he seems to be mainly in­ter­ested in, namely not ac­cept­ing bets with ex­tremely low prob­a­bil­ities, at least un­der nor­mal cir­cum­stances, and it can pre­serve the or­der of our prefer­ences (be­cause even if util­ity is bounded, there are an in­finite num­ber of pos­si­ble val­ues for a util­ity.)

But Kaj’s method will also lead to the Allais para­dox and the like, which won’t hap­pen with bounded util­ity. This seems like un­de­sir­able be­hav­ior, so un­less there is some ad­di­tional rea­son why this is bet­ter than bounded util­ity, I don’t see why it would be a good pro­posal.

• So in this case my ques­tion is why Kaj sug­gests his pro­posal in­stead of us­ing bounded util­ity.

Two rea­sons.

First, like was men­tioned el­se­where in the thread, bounded util­ity seems to pro­duce un­wanted effects, like we want util­ity to be lin­ear in hu­man lives and bounded util­ity seems to fail that.

Se­cond, the way I ar­rived at this pro­posal was that RyanCarey asked me what’s my ap­proach for deal­ing with Pas­cal’s Mug­ging. I replied that I just ig­nore prob­a­bil­ities that are small enough, which seems to be thing that most peo­ple do in prac­tice. He ob­jected that that seemed rather ad-hoc and wanted to have a more prin­ci­pled ap­proach, so I started think­ing about why ex­actly it would make sense to ig­nore suffi­ciently small prob­a­bil­ities, and came up with this as a some­what prin­ci­pled an­swer.

Ad­mit­tedly, as a prin­ci­pled an­swer to which prob­a­bil­ities are ac­tu­ally small enough to ig­nore, this isn’t all that satis­fy­ing of an an­swer, since it still de­pends on a rather ar­bi­trary pa­ram­e­ter. But it still seemed to point to some hid­den as­sump­tions be­hind util­ity max­i­miza­tion as well as rais­ing some very in­ter­est­ing ques­tions about what it is that we ac­tu­ally care about.

• First, like was men­tioned el­se­where in the thread, bounded util­ity seems to pro­duce un­wanted effects, like we want util­ity to be lin­ear in hu­man lives and bounded util­ity seems to fail that.

This is not quite what hap­pens. When you do UDT prop­erly, the re­sult is that the Teg­mark level IV mul­ti­verse has finite ca­pac­ity for hu­man lives (when hu­man lives are counted with 2^-{Kolo­mogorov com­plex­ity} weights, as they should). There­fore the “bare” util­ity func­tion has some kind of diminish­ing re­turns but the “effec­tive” util­ity func­tion is roughly lin­ear in hu­man lives once you take their “mea­sure of ex­is­tence” into ac­count.

I con­sider it highly likely that bounded util­ity is the cor­rect solu­tion.

• I agree that bounded util­ity im­plies that util­ity is not lin­ear in hu­man lives or in other similar mat­ters.

But I have two prob­lems with say­ing that we should try to get this prop­erty. First of all, no one in real life ac­tu­ally acts like it is lin­ear. That’s why we talk about scope in­sen­si­tivity, be­cause peo­ple don’t treat it as lin­ear. That sug­gests that peo­ple’s real util­ity func­tions, in­so­far as there are such things, are bounded.

Se­cond, I think it won’t be pos­si­ble to have a log­i­cally co­her­ent set of prefer­ences if you do that (at least com­bined with your pro­posal), namely be­cause you will lose the in­de­pen­dence prop­erty.

• I agree that, in­so­far as peo­ple have some­thing like util­ity func­tions, those are prob­a­bly bounded. But I don’t think that an AI’s util­ity func­tion should have the same prop­er­ties as my util­ity func­tion, or for that mat­ter the same prop­er­ties as the util­ity func­tion of any hu­man. I wouldn’t want the AI to dis­count the well-be­ing of me or my close ones sim­ply be­cause a billion other peo­ple are already do­ing pretty well.

Though iron­i­cally given my an­swer to your first point, I’m some­what un­con­cerned by your sec­ond point, be­cause hu­mans prob­a­bly don’t have co­her­ent prefer­ences ei­ther, and still seem to do fine. My hunch is that rather than try­ing to make your prefer­ences perfectly co­her­ent, one is bet­ter off mak­ing a sys­tem for de­tect­ing sets of cir­cu­lar trades and similar ex­ploits as they hap­pen, and then mak­ing lo­cal ad­just­ments to fix that par­tic­u­lar in­con­sis­tency.

• I ed­ited this com­ment to in­clude the state­ment that “bounded util­ity means that if U(Y) is too high, U(X) can­not have a dou­ble util­ity etc.” But then it oc­curred to me that I should say some­thing else, which I’m adding here be­cause I don’t want to keep chang­ing the com­ment.

Evand’s state­ment that “these be­hav­iors are both fairly weird (ex­cept in the uni­verse where there’s no pos­si­ble con­struc­tion for an out­come with dou­ble the util­ity of Y, or the uni­verse where you can’t con­struct a suffi­ciently low prob­a­bil­ity for some rea­son)” im­plies a par­tic­u­lar un­der­stand­ing of bounded util­ity.

For ex­am­ple, some­one could say, “My util­ity is in lives saved, and goes up to 10,000,000,000.” In this way he would say that sav­ing 7 billion lives has a util­ity of 7 billion, sav­ing 9 billion lives has a util­ity of 9 billion, and so on. But since he is bound­ing his util­ity, he would say that sav­ing 20 billion lives has a util­ity of 10 billion, and so with sav­ing any other num­ber of lives over 10 billion.

This is definitely weird be­hav­ior. But this is not what I am sug­gest­ing by bounded util­ity. Ba­si­cally I am say­ing that some­one might bound his util­ity at 10 billion, but keep his or­der of prefer­ences so that e.g. he would always pre­fer sav­ing more lives to sav­ing less.

This of course leads to some­thing that could be con­sid­ered scope in­sen­si­tivity: a per­son will pre­fer a chance of sav­ing 10 billion lives to a chance ten times as small of sav­ing 100 billion lives, rather than be­ing in­differ­ent. But ba­si­cally ac­cord­ing to Kaj’s post this is the be­hav­ior we were try­ing to get to in the first place, namely ig­nor­ing the smaller prob­a­bil­ity bet in cer­tain cir­cum­stances. It does cor­re­spond to peo­ple’s be­hav­ior in real life, and it doesn’t have the “switch­ing prefer­ences” effect that Kaj’s method will have when you change prob­a­bil­ities.

• I think I agree that the OP does not fol­low in­de­pen­dence, but ev­ery­thing else here seems wrong.

Ac­tions A and B are iden­ti­cal ex­cept that A gives me 2 utils with .5 prob­a­bil­ity, while B gives me Gra­ham’s num­ber with .5 prob­a­bil­ity. I do B. (Like­wise if there are ~Gra­ham’s num­ber of al­ter­na­tives with in­ter­me­di­ate pay­offs.)

• I’m not sure how you thought this was rele­vant to what I said.

Sup­pose I say that A has util­ity 5, and B has util­ity 10. Ba­si­cally the state­ment that B has twice the util­ity A has, has no par­tic­u­lar mean­ing ex­cept that if I would like to have A at a prob­a­bil­ity of 10%, I would equally like to have B at a prob­a­bil­ity of 5%. If I would take the 10% chance and not the 5% chance, then there is no longer any mean­ing to say­ing that B has “dou­ble” the util­ity of A.

• This does to­tally defy the in­tu­itive un­der­stand­ing of ex­pected util­ity. In­tu­itively, you can just set your util­ity func­tion as what­ever you want. If you want to max­i­mize some­thing like sav­ing hu­man lives, you can do that. As in one per­son dy­ing is ex­actly half as bad as 2 peo­ple dy­ing, which it­self is ex­actly half as bad as 4 peo­ple dy­ing, etc.

The jus­tifi­ca­tion for ex­pected util­ity is that as the num­ber of bets you take ap­proach in­finity, it be­comes the op­ti­mal strat­egy. You save the most lives than you would with any other strat­egy. You lose some peo­ple on some bets, but gain many more on other bets.

But this jus­tifi­ca­tion and in­tu­itive un­der­stand­ing to­tally breaks down in the real world. Where there are finite hori­zons. You don’t get to take an in­finite num­ber of bets. An agent fol­low­ing ex­pected util­ity will just con­tin­u­ously bet away hu­man lives on mug­ger-like bets, with­out ever gain­ing any­thing. It will always do worse than other strate­gies.

You can do some tricks to maybe fix this some­what by mod­ify­ing the util­ity func­tion. But that seems wrong. Why are 2 lives not twice as valuable as 1 life. Why are 400 lives not twice as valuable as 200 lives? Will this change the de­ci­sions you make in ev­ery­day, non-mug­gle/​ex­ces­sively low prob­a­bil­ity bets? It seems like it would.

Or we could just keep the in­tu­itive jus­tifi­ca­tion for ex­pected util­ity, and gen­er­al­ize it to work on finite hori­zons. There are some pro­pos­als for meth­ods that do this like mean of quan­tiles.

• I don’t think that “the jus­tifi­ca­tion for ex­pected util­ity is that as the num­ber of bets you take ap­proach in­finity, it be­comes the op­ti­mal strat­egy,” is quite true. Kaj did say some­thing similar to this, but that seems to me a prob­lem with his ap­proach.

Ba­si­cally, ex­pected util­ity is sup­posed to give a math­e­mat­i­cal for­mal­iza­tion to peo­ple’s prefer­ences. But con­sider this fact: in it­self, it does not have any par­tic­u­lar sense to say that “I like vanilla ice cream twice as much as choco­late.” It makes sense to say I like it more than choco­late. This means if I am given a choice be­tween vanilla and choco­late, I will choose vanilla. But what on earth does it mean to say that I like it “twice” as much as choco­late? In it­self, noth­ing. We have to define this in or­der to con­struct a math­e­mat­i­cal anal­y­sis of our prefer­ences.

In prac­tice we make this defi­ni­tion by say­ing that I like vanilla so much that I am in­differ­ent to hav­ing choco­late, or to hav­ing a 50% chance of hav­ing vanilla, and a 50% chance of hav­ing noth­ing.

Per­haps I jus­tify this by say­ing that it will get me a cer­tain amount of vanilla in my life. But per­haps I don’t—the defi­ni­tion does not jus­tify the prefer­ence, it sim­ply says what it means. This means that in or­der to say I like it twice as much, I have to say that I am in­differ­ent to the 50% bet and to the cer­tain choco­late, no mat­ter what the jus­tifi­ca­tion for this might or might not be. If I change my prefer­ence when the num­ber of cases goes down, then it will not be math­e­mat­i­cally con­sis­tent to say that I like it twice as much as choco­late, un­less we change the defi­ni­tion of “like it twice as much.”

Ba­si­cally I think you are mix­ing up things like “lives”, which can be math­e­mat­i­cally quan­tified in them­selves, more or less, and peo­ple’s prefer­ences, which only have a quan­tity if we define one.

It may be pos­si­ble for Kaj to come up with a new defi­ni­tion for the amount of some­one’s prefer­ence, but I sus­pect that it will re­sult in a situ­a­tion ba­si­cally the same as keep­ing our defi­ni­tion, but ad­mit­ting that peo­ple have only a limited amount of prefer­ence for things. In other words, they might pre­fer sav­ing 100,000 lives to sav­ing 10,000 lives, but they cer­tainly do not pre­fer it 10 times as much, mean­ing they will not always ac­cept the 100,000 lives saved at a 10% chance, com­pared to a 100% chance of sav­ing 10,000.

• But what on earth does it mean to say that I like it “twice” as much as choco­late?

Ob­vi­ously it means you would be will­ing to trade 2 units of choco­late ice cream for 1 unit of vanilla. And over the course of your life, you would pre­fer to have more vanilla ice cream than choco­late ice cream. Per­haps be­fore you die, you will add up all the ice creams you’ve ever eaten. And you would pre­fer for that num­ber to be higher rather than lower.

Nowhere in the above de­scrip­tion did I talk about prob­a­bil­ity. And the util­ity func­tion is already com­pletely defined. I just need to de­cide on a de­ci­sion pro­ce­dure to max­i­mize it.

Ex­pected util­ity seems like a good choice, be­cause, over the course of my life, differ­ent bets I make on ice cream should av­er­age them­selves out, and I should do bet­ter than oth­er­wise. But that might not be true if there are ice cream mug­gers. Which promise lots of ice cream in ex­change for a down pay­ment, but usu­ally lie.

So try­ing to con­vince the ice cream max­i­mizer to fol­low ex­pected util­ity is a lost cause. They will just end up los­ing all their ice cream to mug­gers. They need a sys­tem which ig­nores mug­gers.

• This is definitely not what I mean if I say I like vanilla twice as much as choco­late. I might like it twice as much even though there is no chance that I can ever eat more than one serv­ing of ice cream. If I have the choice of a small serv­ing of vanilla or a triple serv­ing of choco­late, I might still choose the vanilla. That does not mean I like it three times as much.

It is not about “How much ice cream.” It is about “how much want­ing”.

• I’m say­ing that the ex­pe­rience of eat­ing choco­late is ob­jec­tively twice as valuable. Maybe there is a limit on how much ice cream you can eat at a sin­gle sit­ting. But you can still choose to give up eat­ing vanilla to­day and to­mor­row, in ex­change for eat­ing choco­late once.

• Again, you are as­sum­ing there is a quan­ti­ta­tive mea­sure over eat­ing choco­late and eat­ing vanilla, and that this de­ter­mines the mea­sure of my util­ity. This is not nec­es­sar­ily true, since these are ar­bi­trary ex­am­ples. I can still value one twice as much as the other, even if they both are ex­pe­riences that can hap­pen only once in a life­time, or even only once in the life­time of the uni­verse.

• Sure. But there is still some in­ter­nal, ob­jec­tive mea­sure of value of ex­pe­riences. The con­straints you add make it harder to de­ter­mine what they are. But in sim­ple cases, like trad­ing ice cream, it’s easy to de­ter­mine how much value a per­son has for a thing.

• This has noth­ing to do with bounded util­ity. Bounded util­ity means you don’t care about any util­ities above a cer­tain large amount. Like if you care about sav­ing lives, and you save 1,000 lives, af­ter that you just stop car­ing. No amount of lives af­ter that mat­ters at all.

This solu­tion al­lows for un­bounded util­ity. Be­cause you can always care about sav­ing more lives. You just won’t take bets that could save huge num­bers of lives, but have very very small prob­a­bil­ities.

• This isn’t what I meant by bounded util­ity. I ex­plained that in an­other com­ment. It refers to util­ity as a real num­ber and sim­ply sets a limit on that num­ber. It does not mean that at any point “you just stop car­ing.”

• If your util­ity has a limit, then you can’t care about any­thing past that limit. Even a con­tin­u­ous limit doesn’t work, be­cause you care less and less about ob­tain­ing more util­ity, as you get closer to it. You would take a 50% chance at sav­ing 2 peo­ple the same as a guaran­teed chance at sav­ing 1 per­son. But not a 50% chance at sav­ing 2,000 peo­ple, over a chance at sav­ing 1,000.

• Yes, that would be the effect in gen­eral, that you would be less will­ing to take chances when the num­bers in­volved are higher. That’s why you wouldn’t get mugged.

But that still doesn’t mean that “you don’t care.” You still pre­fer sav­ing 2,000 lives to sav­ing 1,000, when­ever the chances are equal; your prefer­ence for the two cases does not sud­denly be­come equal, as you origi­nally said.

• If util­ity is strictly bounded, then you do liter­ally not care about sav­ing 1,000 lives or 2,000.

You can fix that with asymp­tote. Then you do have a prefer­ence for 2,000. But the prefer­ence is only very slight. You wouldn’t take a 1% risk of los­ing 1,000 peo­ple, to save 2,000 peo­ple oth­er­wise. Even though the risk is very small and the gain is very huge.

So it does fix Pas­cal’s mug­ging, but causes a whole new class of is­sues.

• Your un­der­stand­ing of “strictly bounded” is ar­tifi­cial, and not what I was talk­ing about. I was talk­ing about as­sign­ing a strict, nu­mer­i­cal bound to util­ity. That does not pre­vent hav­ing an in­finite num­ber of val­ues un­der­neath that bound.

It would be silly to as­sign a bound and a func­tion low enough that “You wouldn’t take a 1% risk of los­ing 1,000 peo­ple, to save 2,000 peo­ple oth­er­wise,” if you meant this liter­ally, with these val­ues.

But it is easy enough to as­sign a bound and a func­tion that re­sult in the choices we ac­tu­ally make in terms of real world val­ues. It is true that if you in­crease the val­ues enough, some­thing like that will hap­pen. And that is ex­actly the way real peo­ple would be­have, as well.

• Your un­der­stand­ing of “strictly bounded” is ar­tifi­cial, and not what I was talk­ing about. I was talk­ing about as­sign­ing a strict, nu­mer­i­cal bound to util­ity. That does not pre­vent hav­ing an in­finite num­ber of val­ues un­der­neath that bound.

Isn’t that the same as an asymp­tote, which I talked about?

It would be silly to as­sign a bound and a func­tion low enough that “You wouldn’t take a 1% risk of los­ing 1,000 peo­ple, to save 2,000 peo­ple oth­er­wise,” if you meant this liter­ally, with these val­ues.

You can set the bound wher­ever you want. It’s ar­bi­trary. My point is that if you ever reach it, you start be­hav­ing weird. It is not a very nat­u­ral fix. It cre­ates other is­sues.

It is true that if you in­crease the val­ues enough, some­thing like that will hap­pen. And that is ex­actly the way real peo­ple would be­have, as well.

Maybe hu­man util­ity func­tions are bounded. Maybe they aren’t. We don’t know for sure. As­sum­ing they are is a big risk. And even if they are bounded, it doesn’t mean we should put that into an AI. If, some­how, it ever runs into a situ­a­tion where it can help 3^^^3 peo­ple, it re­ally should.

• “If, some­how, it ever runs into a situ­a­tion where it can help 3^^^3 peo­ple, it re­ally should.”

I thought the whole idea be­hind this pro­posal was that the prob­a­bil­ity of this hap­pen­ing is es­sen­tially zero.

If you think this is some­thing with a rea­son­able prob­a­bil­ity, you should ac­cept the mug­ging.

• You were speak­ing about bounded util­ity func­tions. Not bounded prob­a­bil­ity func­tions.

The whole point of the Pas­cal’s mug­ger sce­nario is that these sce­nar­ios aren’t im­pos­si­ble. Solomonoff in­duc­tion halves the prob­a­bil­ity of each hy­poth­e­sis based on how many ad­di­tional bits it takes to de­scribe. This means the prob­a­bil­ity of differ­ent mod­els de­creases fairly rapidly. But not as rapidly as func­tions like 3^^^3 grow. So there are hy­pothe­ses that de­scribe things that are 3^^^3 units large in much fewer than log(3^^^3) bits.

So the util­ity of hy­pothe­ses can grow much faster than their prob­a­bil­ity shrinks.

If you think this is some­thing with a rea­son­able prob­a­bil­ity, you should ac­cept the mug­ging.

Well the prob­a­bil­ity isn’t rea­son­able. It’s just not as un­rea­son­ably small as 3^^^3 is big.

But yes you could bite the bul­let and say that the ex­pected util­ity is so big, it doesn’t mat­ter what the prob­a­bil­ity is, and pay the mug­ger.

The prob­lem is, ex­pected util­ity doesn’t even con­verge. There is a hy­poth­e­sis that pay­ing the mug­ger saves 3^^^3 lives. And there’s an even more un­likely hy­poth­e­sis that not pay­ing him will save 3^^^^3 lives. And an even more com­pli­cated hy­poth­e­sis that he will re­ally save 3^^^^^3 lives. Etc. The ex­pected util­ity of ev­ery ac­tion grows to in­finity, and never con­verges on any finite value. More and more un­likely hy­pothe­ses to­tally dom­i­nate the calcu­la­tion.

• Solomonoff in­duc­tion halves the prob­a­bil­ity of each hy­poth­e­sis based on how many ad­di­tional bits it takes to de­scribe.

See, I told ev­ery­one that peo­ple here say this.

Fake mug­gings with large num­bers are more prof­itable to the mug­ger than fake mug­gings with small num­bers be­cause the fake mug­ging with the larger num­ber is more likely to con­vince a naive ra­tio­nal­ist. And the prof­ita­bil­ity de­pends on the size of the num­ber, not the num­ber of bits in the num­ber. Which makes the like­li­hood of a large num­ber be­ing fake grow faster than the num­ber of bits in the num­ber.

• You are solv­ing the spe­cific prob­lem of the mug­ger, and not the gen­eral prob­lem of tiny bets with huge re­wards.

Re­gard­less, there’s no way the prob­a­bil­ity de­creases faster than the re­ward the mug­ger promises. I don’t think you can as­sign 1/​3^^^3 prob­a­bil­ity to any­thing. That’s an un­fath­omably small prob­a­bil­ity. You are liter­ally say­ing there is no amount of ev­i­dence the mug­ger could give you to con­vince you oth­er­wise. Even if he showed you his ma­trix pow­ers, and the com­puter simu­la­tion of 3^^^3 peo­ple, you still wouldn’t be­lieve him.

• How could he show you “the com­puter simu­la­tion of 3^^^3 peo­ple”? What could you do to ver­ify that 3^^^3 peo­ple were re­ally be­ing simu­lated?

• You prob­a­bly couldn’t ver­ify it. There’s always the pos­si­bil­ity that any ev­i­dence you see is made up. For all you know you are just in a com­puter simu­la­tion and the en­tire thing is vir­tual.

I’m just say­ing he can show you ev­i­dence which in­creases the prob­a­bil­ity. Show you the racks of servers, show you the com­puter sys­tem, ex­plain the physics that al­lows it, lets you do the ex­per­i­ments that shows those physics are cor­rect. You could solve any NP com­plete prob­lem on the com­puter. And you could run pro­grams that take known num­bers of steps to com­pute. Like ac­tu­ally calcu­lat­ing 3^^^3, etc.

• Sure. But I think there are gen­er­ally go­ing to be more par­si­mo­nious ex­pla­na­tions than any that in­volve him hav­ing the power to tor­ture 3^^^3 peo­ple, let alone hav­ing that power and car­ing about whether I give him some money.

• Par­si­mo­nious, sure. The pos­si­bil­ity is very un­likely. But it doesn’t just need to be “very un­likely”, it needs to have smaller than 1/​3^^^3 prob­a­bil­ity.

• Sure. But if you have an ar­gu­ment that some guy who shows me ap­par­ent mag­i­cal pow­ers has the power to tor­ture 3^^^3 peo­ple with prob­a­bil­ity sub­stan­tially over 1/​3^^^3, then I bet I can turn it into an ar­gu­ment that any­one, with or with­out a demon­stra­tion of mag­i­cal pow­ers, with or with­out any sort of claim that they have such pow­ers, has the power to tor­ture 3^^^3 peo­ple with prob­a­bil­ity nearly as sub­stan­tially over 1/​3^^^3. Be­cause surely for any­one un­der any cir­cum­stances, Pr(I ex­pe­rience what seems to be a con­vinc­ing demon­stra­tion that they have such pow­ers) is much larger than 1/​3^^^3, whether they ac­tu­ally have such pow­ers or not.

• Sure. But if you have an ar­gu­ment that some guy who shows me ap­par­ent mag­i­cal pow­ers has the power to tor­ture 3^^^3 peo­ple with prob­a­bil­ity sub­stan­tially over 1/​3^^^3, then I bet I can turn it into an ar­gu­ment that any­one, with or with­out a demon­stra­tion of mag­i­cal pow­ers, with or with­out any sort of claim that they have such pow­ers, has the power to tor­ture 3^^^3 peo­ple with prob­a­bil­ity nearly as sub­stan­tially over 1/​3^^^3.

Cor­rect. That still doesn’t solve the de­ci­sion the­ory prob­lem, it makes it worse. Since you have to take into ac­count the pos­si­bil­ity that any­one you meet might have the power to tor­ture (or re­ward with utopia) 3^^^3 peo­ple.

• It makes it worse or bet­ter, de­pend­ing on whether you de­cide (1) that ev­ery­one has the power to do that with prob­a­bil­ity >~ 1/​3^^^3 or (2) that no one has. I think #2 rather than #1 is cor­rect.

• Well, do­ing ba­sic Bayes with a Kol­mogorov priot gives you (1).

• I don’t think you can as­sign 1/​3^^^3 prob­a­bil­ity to any­thing. That’s an un­fath­omably small prob­a­bil­ity.

About as un­fath­omably small as the num­ber of 3^^^3 peo­ple is un­fath­omably large?

I think you’re rely­ing on “but I feel this can’t be right!” a bit too much.

• I don’t see what your point is. Yes that’s a small num­ber. It’s not a feel­ing, that’s just math. If you are as­sign­ing things 1/​3^^^3 prob­a­bil­ity, you are ba­si­cally say­ing they are im­pos­si­ble and no amount of ev­i­dence could con­vince you oth­er­wise.

You can do that and be perfectly con­sis­tent. If that’s your point I don’t dis­agree. You can’t ar­gue about pri­ors. We can only agree to dis­agree, if those are your true pri­ors.

Just re­mem­ber that re­al­ity could always say “WRONG!” and pun­ish you for as­sign­ing 0 prob­a­bil­ity to some­thing. If you don’t want to be wrong, don’t as­sign 1/​3^^^3 prob­a­bil­ity to things you aren’t 99.9999...% sure ab­solutely can’t hap­pen.

• Eliezer showed a prob­lem that that rea­son­ing in his post on Pas­cal’s Mug­gle.

Ba­si­cally, hu­man be­ings do not have an ac­tual prior prob­a­bil­ity dis­tri­bu­tion. This should be ob­vi­ous, since it means as­sign­ing a nu­mer­i­cal prob­a­bil­ity to ev­ery pos­si­ble state of af­fairs. No hu­man be­ing has ever done this, or ever will.

But you have some­thing like a prior, but you build the prior it­self based on your ex­pe­rience. At the mo­ment we don’t have a spe­cific num­ber for the prob­a­bil­ity of the mug­ging situ­a­tion com­ing up, but just think it’s very im­prob­a­ble, so that we don’t ex­pect any ev­i­dence to ever come up that would con­vince us. But if the mug­ger shows ma­trix pow­ers, we would change our prior so that the prob­a­bil­ity of the mug­ging situ­a­tion was high enough to be con­vinced by be­ing shown ma­trix pow­ers.

You might say that means it was already that high, but it does not mean this, given the ob­jec­tive fact that peo­ple do not have real pri­ors.

• Maybe hu­mans don’t re­ally have prob­a­bil­ity dis­tri­bu­tions. But that doesn’t help us ac­tu­ally build an AI which re­pro­duces the same re­sult. If we had in­finite com­put­ing power and could do ideal Solomonoff in­duc­tion, it would pay the mug­ger.

Though I would ar­gue that hu­mans do have ap­prox­i­mate prob­a­bil­ity func­tions and ap­prox­i­mate pri­ors. We wouldn’t be able to func­tion in a prob­a­bil­is­tic world if we didn’t. But it’s not rele­vant.

But if the mug­ger shows ma­trix pow­ers, we would change our prior so that the prob­a­bil­ity of the mug­ging situ­a­tion was high enough to be con­vinced by be­ing shown ma­trix pow­ers.

That’s just a reg­u­lar bayesian prob­a­bil­ity up­date! You don’t need to change ter­minol­ogy and call it some­thing differ­ent.

At the mo­ment we don’t have a spe­cific num­ber for the prob­a­bil­ity of the mug­ging situ­a­tion com­ing up, but just think it’s very im­prob­a­ble, so that we don’t ex­pect any ev­i­dence to ever come up that would con­vince us.

That’s fine. I too think the situ­a­tion is ex­traor­di­nar­ily im­plau­si­ble. Even Solomonoff in­duc­tion would agree with us. The prob­a­bil­ity that the mug­ger is real would be some­thing like 1/​10^100. Or per­haps the ex­po­nent should be or­ders of mag­ni­tude larger than that. That’s small enough that it shouldn’t even re­motely reg­ister as a plau­si­ble hy­poth­e­sis in your mind. But big enough some amount of ev­i­dence could con­vince you.

You don’t need to posit new mod­els of how prob­a­bil­ity the­ory should work. Reg­u­lar prob­a­bil­ity works fine at as­sign­ing re­ally im­plau­si­ble hy­pothe­ses re­ally low prob­a­bil­ity.

But that is still way, way big­ger than 1/​3^^^3.

• So there are hy­pothe­ses that de­scribe things that are 3^^^3 units large in much fewer than log(3^^^3) bits.

So the util­ity of hy­pothe­ses can grow much faster than their prob­a­bil­ity shrinks.

If util­ity is straight­for­wardly ad­di­tive, yes. But per­haps it isn’t. Imag­ine two pos­si­ble wor­lds. In one, there are a billion copies of our planet and its pop­u­la­tion, all some­how lead­ing ex­actly the same lives. In an­other, there are a billion planets like ours, with differ­ent peo­ple on them. Now some­one pro­poses to blow up one of the planets. I find that I feel less awful about this in the first case than the sec­ond (though of course ei­ther is awful) be­cause what’s be­ing lost from the uni­verse is some­thing of which we have a billion copies any­way. If we stipu­late that the de­struc­tion of the planet is in­stan­ta­neous and painless, and that the peo­ple re­ally are liv­ing ex­actly iden­ti­cal lives on each planet, then ac­tu­ally I’m not sure I care very much that one planet is gone. (But my feel­ings about this fluc­tu­ate.)

A world with 3^^^3 in­hab­itants that’s de­scribed by (say) no more than a billion bits seems a lit­tle like the first of those hy­po­thet­i­cal wor­lds.

I’m not very sure about this. For in­stance, per­haps the de­scrip­tion would take the form: “Seed a good ran­dom num­ber gen­er­a­tor as fol­lows. [...] Now use it to gen­er­ate 3^^^3 per­son-like agents in a de­ter­minis­tic uni­verse with such-and-such laws. Now run it for 20 years.” and maybe you can get 3^^^3 gen­uinely non-re­dun­dant lives that way. But 3^^^3 is a very large num­ber, and I’m not even quite sure there’s such a thing as 3^^^3 gen­uinely non-re­dun­dant lives even in prin­ci­ple.

• Well what if in­stead of kil­ling them, he tor­tured them for an hour? Death might not mat­ter in a Big World, but to­tal suffer­ing still does.

• I dunno. If I imag­ine a world with a billion iden­ti­cal copies of me liv­ing iden­ti­cal lives, hav­ing all of them tor­tured doesn’t seem a billion times worse than hav­ing one tor­tured. Would an AI’s ex­pe­riences mat­ter more if, to re­duce the im­pact of hard­ware er­ror, all its com­pu­ta­tions were performed on ten iden­ti­cal com­put­ers?

• What if any of the Big World hy­pothe­ses are true? E.g. many wor­lds in­ter­pre­ta­tion, mul­ti­verse the­o­ries, Teg­mark’s hy­poth­e­sis, or just a reg­u­lar in­finite uni­verse. In that case any­thing that can ex­ist does ex­ist. There already are a billion ver­sions of you be­ing tor­tured. An in­finite num­ber ac­tu­ally. All you can ever re­ally do is re­duce the prob­a­bil­ity that you will find your­self in a good world or a bad one.

• Bounded util­ity func­tions effec­tively give “bounded prob­a­bil­ity func­tions,” in the sense that you (more or less) stop car­ing about things with very low prob­a­bil­ity.

For ex­am­ple, if my max­i­mum util­ity is 1,000, then my max­i­mum util­ity for some­thing with a prob­a­bil­ity of one in a billion is .0000001, an ex­tremely small util­iity, so some­thing that I will care about very lit­tle. The prob­a­bil­ity of of the 3^^^3 sce­nar­ios may be more than one in 3^^^3. But it will still be small enough that a bounded util­ity func­tion won’t care about situ­a­tions like that, at least not to any sig­nifi­cant ex­tent.

That is pre­cisely the rea­son that it will do the things you ob­ject to, if that situ­a­tion comes up.

That is no differ­ent from point­ing out that the post’s pro­posal will re­ject a “mug­ging” even when it will ac­tu­ally cost 3^^^3 lives.

Both pro­pos­als have that par­tic­u­lar down­side. That is not some­thing pe­cu­liar to mine.

• Bounded util­ity func­tions mean you stop car­ing about things with very high util­ity. That you care less about cer­tain low prob­a­bil­ity events is just a side effect. But those events can also have very high prob­a­bil­ity and you still don’t care.

If you want to just stop car­ing about re­ally low prob­a­bil­ity events, why not just do that?

• I just ex­plained. There is no situ­a­tion in­volv­ing 3^^^3 peo­ple which will ever have a high prob­a­bil­ity. Tel­ling me I need to adopt a util­ity func­tion which will han­dle such situ­a­tions well is try­ing to mug me, be­cause such situ­a­tions will never come up.

Also, I don’t care about the differ­ence be­tween 3^^^^^3 peo­ple and 3^^^^^^3 peo­ple even if the prob­a­bil­ity is 100%, and nei­ther does any­one else. So it isn’t true that I just want to stop car­ing about low prob­a­bil­ity events. My util­ity is ac­tu­ally bounded. That’s why I sug­gest us­ing a bounded util­ity func­tion, like ev­ery­one else does.

• There is no situ­a­tion in­volv­ing 3^^^3 peo­ple which will ever have a high prob­a­bil­ity.

Really? No situ­a­tion? Not even if we dis­cover new laws of physics that al­low us to have in­finite com­put­ing power?

Tel­ling me I need to adopt a util­ity func­tion which will han­dle such situ­a­tions well is try­ing to mug me, be­cause such situ­a­tions will never come up.

We are talk­ing about util­ity func­tions. Prob­a­bil­ity is ir­rele­vant. All that mat­ters for the util­ity func­tion is that if the situ­a­tion came up, you would care about it.

Also, I don’t care about the differ­ence be­tween 3^^^^^3 peo­ple and 3^^^^^^3 peo­ple even if the prob­a­bil­ity is 100%, and nei­ther does any­one else.

I to­tally dis­agree with you. Th­ese num­bers are so in­com­pre­hen­si­bly huge you can’t pic­ture them in your head, sure. There is mas­sive scope in­sen­si­tivity. But if you had to make moral choices that af­fect those two num­bers of peo­ple, you should always value the big­ger num­ber pro­por­tion­ally more.

E.g. if you had to tor­ture 3^^^^^3 to save 3^^^^^^3 from get­ting dust specks in their eyes. Or make bets in­volv­ing prob­a­bil­ities be­tween var­i­ous things hap­pen­ing to the differ­ent groups. Etc. I don’t think you can make these de­ci­sions cor­rectly if you have a bounded util­ity func­tion.

If you don’t make them cor­rectly, well that 3^^^3 peo­ple prob­a­bly con­tains a ba­si­cally in­finite num­ber of copies of you. By mak­ing the cor­rect trade­offs, you max­i­mize the prob­a­bil­ity that the other ver­sions of yoru­self find them­selves in a uni­verse with higher util­ity.

• What if one con­sid­ers the fol­low­ing ap­proach: Let e be a prob­a­bil­ity small enough that if I were to ac­cept all bets offered to me with prob­a­bil­ity p<= e then the ex­pected num­ber of such bets that I win is less than one. The ap­proach is to ig­nore any bet where p <=e.

This solve’s Yvain’s prob­lem with wear­ing seat­belts or eat­ing un­healthy for ex­am­ple. It also solves the prob­lem that “sub-di­vid­ing” a risk no longer changes whether you ig­nore the risk.

• I’m prob­a­bly way late to this thread, but I was think­ing about this the other day in the re­sponse to a differ­ent thread, and con­sid­ered us­ing the Kelly Cri­te­rion to ad­dress some­thing like Pas­cal’s Mug­ging.

Try­ing to figure out your cur­rent ‘bankroll’ in terms of util­ity is prob­a­bly open to in­tepre­ta­tion, but for some broad es­ti­mates, you could prob­a­bly use your as­sets, or your ex­pected free time, or some util­ity func­tion that in­cluded those plus what­ever else.

When calcu­lat­ing op­ti­mal bet size us­ing the Kelly crite­rion, you end up with a per­centage of your cur­rent bankroll you should bet. This per­centage will never ex­ceed the prob­a­bil­ity of the event oc­cur­ring, re­gard­less of the size of the re­ward. This ba­si­cally means that if I’m us­ing my cur­rent net worth as an ap­prox­i­ma­tion for my ‘bankroll’, I shouldn’t even con­sider bet­ting a dol­lar on some­thing I think has a one-in-a-mil­lion chance, un­less my net worth is at least a mil­lion dol­lars.

I think this could be a bit more for­mal­ized, but might help serve as a rule-of-thumb for eval­u­at­ing Pas­cal’s Wager type sce­nar­ios.

• I think the mug­ger can mod­ify their offer to in­clude ”...and I will offer you this deal X times to­day, so it’s in your in­ter­est to take the deal ev­ery time,” where X is suffi­ciently large, and the amount re­quested in each in­di­vi­d­ual offer is tiny but cal­ibrated to add up to the amount that the mug­ger wants. If the odds are a mil­lion to one, then to gain \$1000, the mug­ger can re­quest \$0.001 a mil­lion times.

• Rol­ling all 60 years of bets up into one prob­a­bil­ity dis­tri­bu­tion as in your ex­am­ple, we get:

• 0,999999999998 chance of − 1 billion * cost-per-bet

• 1 − 0,999999999998 - ep­silon chance of 10^100 lives − 1 billion * cost-per-bet

• ep­silon chance of n * 10^100 lives, etc.

I think what this shows is that the ag­gre­gat­ing tech­nique you pro­pose is no differ­ent than just deal­ing with a 1-shot bet. So if you can’t solve the one-shot Pas­cal’s mug­ging, ag­gre­gat­ing it won’t help in gen­eral.

• The way I see it, if one be­lieves that the range of pos­si­ble ex­tremes of pos­i­tive or nega­tive val­ues is greater than the range of pos­si­ble prob­a­bil­ities then one would have rea­son to treat rare high/​low value events as more im­por­tant than more fre­quent events.

• Would that mean that if I ex­pect to have to use trans­port n times through­out the next m years, with prob­a­bil­ity p of dy­ing dur­ing com­mut­ing; and I want to calcu­late the PEST of, for ex­am­ple, fatal poi­son­ing from canned food f, which I es­ti­mate to be able to hap­pen about t times dur­ing the same m years, I have to lump the two dan­gers to­gether and see if it is still <1? I mean, I can work from home and never eat canned food… But this doesn’t seem to be what you write about when you talk about differ­ent deals.

(Sorry for pos­si­bly stupid ques­tion.)

• Some more ex­treme pos­si­bil­ities on the lifes­pan prob­lem: Should you figure in the pos­si­bil­ity of life ex­ten­sion? The pos­si­bil­ity of im­mor­tal­ity?

What about Many Wor­lds? If you count al­ter­nate ver­sions of your­self as you, then low prob­a­bil­ity bets make more sense.

• This is an in­ter­est­ing heuris­tic, but I don’t be­lieve that it an­swers the ques­tion of, “What should a ra­tio­nal agent do here?”

• The rea­son­ing why one should rely on ex­pected value on one offs can be used to cir­cum­vent the rea­son­ing. It is men­tioned int he ar­ti­cle but I would like to raise it ex­plic­itly.

If I per­son­ally have a 0.1 chance of get­ting a high re­ward within my life­time then 10 per­sons like me would on av­er­age hit the jack­pot once.

Or in the re­verse if one takes the con­clu­sion se­ri­ously one needs to start re­ject­ing one-offs be­cause there isn’t suffi­cient rep­e­ti­tion to tend to the mean. Well you could say that value is per­sonal and thus rele­vant rep­e­ti­tion class is life­time de­ci­sions. But if we take life to be “hu­man value” then the rele­vant rep­e­ti­tion class is choices made by homo sapi­ens (and pos­si­bly be­yond).

• Say some­one offers to cre­ate 10^100 happy lives in ex­change for some­thing, and you as­sign them a 0.000000000000000000001 prob­a­bil­ity to them be­ing ca­pa­ble and will­ing to carry through their promise. Naively, this has an over­whelm­ingly pos­i­tive ex­pected value.

If the stated prob­a­bil­ity is what you re­ally as­sign then yes, pos­i­tive ex­pected value.

I see the key flaw in that the more ex­cep­tional the promise is, the lower the prob­a­bil­ity you must as­sign to it.

Would you give more cred­i­bil­ity to some­one offer­ing you 10^2 US\$ or 10^7 US\$?

• I see the key flaw in that the more ex­cep­tional the promise is, the lower the prob­a­bil­ity you must as­sign to it.

Ac­cord­ing to com­mon LessWrong ideas, low­er­ing the prob­a­bil­ity based on the ex­cep­tion­al­ity of the promise would mean low­er­ing it based on the Kolo­mogorov com­plex­ity of the promise.

If you do that, you won’t lower the prob­a­bil­ity enough to defeat the mug­ging.

If you can lower the prob­a­bil­ity more than that, of course you can defeat the mug­ging.

• If you can lower the prob­a­bil­ity more than that, of course you can defeat the mug­ging. And one of the key prob­lems with low­er­ing it more is that it be­comes re­ally re­ally hard to up­date when you get ev­i­dence that the mug­ging is real.

• If you do that, you won’t lower the prob­a­bil­ity enough to defeat the mug­ging.

If you do that, your de­ci­sion sys­tem just breaks down, since the ex­pec­ta­tion over ar­bi­trary in­te­gers with prob­a­bil­ities com­puter by Solomonoff in­duc­tion is un­defined. That’s the rea­son why AIXI uses bounded re­wards.

• As an al­ter­na­tive to prob­a­bil­ity thresh­olds and bounded util­ities (already men­tioned in the com­ments), you could con­strain the epistemic model such that for any state and any can­di­date ac­tion, the prob­a­bil­ity dis­tri­bu­tion of util­ity is light-tailed.

The effect is similar to a prob­a­bil­ity thresh­old: the tails of the dis­tri­bu­tion don’t dom­i­nate the ex­pec­ta­tion, but this way it is “softer” and more the­o­ret­i­cally prin­ci­pled, since light-tailed dis­tri­bu­tions, like those in the ex­po­nen­tial fam­ily, are in a cer­tain sense, “nat­u­ral”.