Against the Linear Utility Hypothesis and the Leverage Penalty

[Roughly the sec­ond half of this is a re­ply to: Pas­cal’s Mug­gle]

There’s an as­sump­tion that peo­ple of­ten make when think­ing about de­ci­sion the­ory, which is that util­ity should be lin­ear with re­spect to amount of stuff go­ing on. To be clear, I don’t mean lin­ear with re­spect to amount of money/​cook­ies/​etc that you own; most peo­ple know bet­ter than that. The as­sump­tion I’m talk­ing about is that the state of the rest of the uni­verse (or mul­ti­verse) does not af­fect the marginal util­ity of there also be­ing some­one hav­ing cer­tain ex­pe­riences at some lo­ca­tion in the uni-/​multi-verse. For in­stance, if 1 util is the differ­ence in util­ity be­tween noth­ing ex­ist­ing, and there be­ing a planet that has some hu­mans and other an­i­mals liv­ing on it for a while be­fore go­ing ex­tinct, then the differ­ence in util­ity be­tween noth­ing ex­ist­ing and there be­ing n copies of that planet should be n utils. I’ll call this the Lin­ear Utility Hy­poth­e­sis. It seems to me that, de­spite its pop­u­lar­ity, the Lin­ear Utility Hy­poth­e­sis is poorly mo­ti­vated, and a very poor fit to ac­tual hu­man prefer­ences.

The Lin­ear Utility Hy­poth­e­sis gets im­plic­itly as­sumed a lot in dis­cus­sions of Pas­cal’s mug­ging. For in­stance, in Pas­cal’s Mug­gle, Eliezer Yud­kowsky says he “[doesn’t] see any way around” the con­clu­sion that he must be as­sign­ing a prob­a­bly at most on the or­der of 1/​3↑↑↑3 to the propo­si­tion that Pas­cal’s mug­ger is tel­ling the truth, given that the mug­ger claims to be in­fluenc­ing 3↑↑↑3 lives and that he would re­fuse the mug­ger’s de­mands. This im­plies that he doesn’t see any way that in­fluenc­ing 3↑↑↑3 lives could not have on the or­der of 3↑↑↑3 times as much util­ity as in­fluenc­ing one life, which sounds like an in­vo­ca­tion of the Lin­ear Utility Hy­poth­e­sis.

One ar­gu­ment for some­thing kind of like the Lin­ear Utility Hy­poth­e­sis is that there may be a vast mul­ti­verse that you can in­fluence only a small part of, and un­less your util­ity func­tion is weirdly non­differ­en­tiable and you have very pre­cise in­for­ma­tion about the state of the rest of the mul­ti­verse (or if your util­ity func­tion de­pends pri­mar­ily on things you per­son­ally con­trol), then your util­ity func­tion should be lo­cally very close to lin­ear. That is, if your util­ity func­tion is a smooth func­tion of how many peo­ple are ex­pe­rienc­ing what con­di­tions, then the util­ity from in­fluenc­ing 1 life should be 1/​n times the util­ity of hav­ing the same in­fluence on n lives, be­cause n is in­evitably go­ing to be small enough that a lin­ear ap­prox­i­ma­tion to your util­ity func­tion will be rea­son­ably ac­cu­rate, and even if your util­ity func­tion isn’t smooth, you don’t know what the rest of the uni­verse looks like, so you can’t pre­dict how the small changes you can make will in­ter­act with dis­con­ti­nu­ities in your util­ity func­tion. This is a scaled-up ver­sion of a com­mon ar­gu­ment that you should be will­ing to pay 10 times as much to save 20,000 birds as you would be will­ing to pay to save 2,000 birds. I am sym­pa­thetic to this ar­gu­ment, though not con­vinced of the premise that you can only in­fluence a tiny por­tion of what is ac­tu­ally valuable to you. More im­por­tantly, this ar­gu­ment does not even at­tempt to es­tab­lish that util­ity is globally lin­ear, and coun­ter­in­tu­itive con­se­quences of the Lin­ear Utility Hy­poth­e­sis, such as Pas­cal’s mug­ging, of­ten in­volve situ­a­tions that seem es­pe­cially likely to vi­o­late the as­sump­tion that all choices you make have tiny con­se­quences.

I have never seen any­one provide a defense of the Lin­ear Utility Hy­poth­e­sis it­self (ac­tu­ally, I think I’ve been pointed to the VNM the­o­rem for this, but I don’t count that be­cause it’s a non-se­quitor; the VNM the­o­rem is just a rea­son to use a util­ity func­tion in the first place, and does not place any con­straints on what that util­ity func­tion might look like), so I don’t know of any ar­gu­ments for it available for me to re­fute, and I’ll just go ahead and ar­gue that it can’t be right be­cause ac­tual hu­man prefer­ences vi­o­late it too dra­mat­i­cally. For in­stance, sup­pose you’re given a choice be­tween the fol­low­ing two op­tions: 1: Hu­man­ity grows into a vast civ­i­liza­tion of 10^100 peo­ple liv­ing long and happy lives, or 2: a 10% chance that hu­man­ity grows into a vast civ­i­liza­tion of 10^102 peo­ple liv­ing long and happy lives, and a 90% chance of go­ing ex­tinct right now. I think al­most ev­ery­one would pick op­tion 1, and would think it crazy to take a reck­less gam­ble like op­tion 2. But the Lin­ear Utility Hy­poth­e­sis says that op­tion 2 is much bet­ter. Most of the ways peo­ple re­spond to Pas­cal’s mug­ger don’t ap­ply to this situ­a­tion, since the prob­a­bil­ities and ra­tios of util­ities in­volved here are not at all ex­treme.

There are smaller-scale coun­terex­am­ples to the Lin­ear Utility Hy­poth­e­sis as well. Sup­pose you’re offered the choice be­tween: 1: con­tinue to live a nor­mal life, which lasts for n more years, or 2: live the next year of a nor­mal life, but then in­stead of liv­ing a nor­mal life af­ter that, have all your mem­o­ries from the past year re­moved, and ex­pe­rience that year again n more times (your mem­o­ries get­ting re­set each time). I ex­pect pretty much ev­ery­one to take op­tion 1, even if they ex­pect the next year of their life to be bet­ter than the av­er­age of all fu­ture years of their life. If util­ity is just a naive sum of lo­cal util­ity, then there must be some year in which has at least as much util­ity in it as the av­er­age year, and just re­peat­ing that year ev­ery year would thus in­crease to­tal util­ity. But hu­mans care about the re­la­tion­ship that their ex­pe­riences have with each other at differ­ent times, as well as what those ex­pe­riences are.

Here’s an­other thought ex­per­i­ment that seems like a rea­son­able em­piri­cal test of the Lin­ear Utility Hy­poth­e­sis: take some event that is fa­mil­iar enough that we un­der­stand its ex­pected util­ity rea­son­ably well (for in­stance, the amount of money in your pocket chang­ing by $5), and some lu­dicrously un­likely event (for in­stance, the event in which some ran­dom per­son is ac­tu­ally tel­ling the truth when they claim, with­out ev­i­dence, to have magic pow­ers al­low­ing them to con­trol the fates of ar­bi­trar­ily large uni­verses, and say­ing, with­out giv­ing a rea­son, that the way they use this power is de­pen­dent on some seem­ingly un­re­lated ac­tion you can take), and see if you be­come will­ing to sac­ri­fice the well-un­der­stood amount of util­ity in ex­change for the tiny chance of a large im­pact when the large im­pact be­comes big enough that the tiny chance of it would be more im­por­tant if the Lin­ear Utility Hy­poth­e­sis were true. This thought ex­per­i­ment should sound very fa­mil­iar. The re­sult of this ex­per­i­ment is that ba­si­cally ev­ery­one agrees that they shouldn’t pay the mug­ger, not only at much higher stakes than the Lin­ear Utility Hy­poth­e­sis pre­dicts should be suffi­cient, but even at ar­bi­trar­ily large stakes. This re­sult has even stronger con­se­quences than that the Lin­ear Utility Hy­poth­e­sis is false, namely that util­ity is bounded. Peo­ple have come up with all sorts of ab­surd ex­pla­na­tions for why they wouldn’t pay Pas­cal’s mug­ger even though the Lin­ear Utility Hy­poth­e­sis is true about their prefer­ences (I will ad­dress the least ab­surd of these ex­pla­na­tions in a bit), but there is no bet­ter test for whether an agent’s util­ity func­tion is bounded than how it re­sponds to Pas­cal’s mug­ger. If you take the claim “My util­ity func­tion is un­bounded”, and taboo “util­ity func­tion” and “un­bounded”, it be­comes “Given out­comes A and B such that I pre­fer A over B, for any prob­a­bil­ity p>0, there is an out­come C such that I would take B rather than A if it lets me con­trol whether C hap­pens in­stead with prob­a­bil­ity p.” If you claim that one of these claims is true and the other is false, then you’re just con­tra­dict­ing your­self, be­cause that’s what “util­ity func­tion” means. That can be roughly trans­lated into English as “I would do the equiv­a­lent of pay­ing the mug­ger in Pas­cal’s mug­ging-like situ­a­tions”. So in Pas­cal’s mug­ging-like situ­a­tions, agents with un­bounded util­ity func­tions don’t look for clever rea­sons not to do the equiv­a­lent of pay­ing the mug­ger; they just pay up. The fact that this be­hav­ior is so coun­ter­in­tu­itive is an in­di­ca­tion that agents with un­bounded util­ity func­tions are so alien that you have no idea how to em­pathize with them. The “least ab­surd ex­pla­na­tion” I referred to for why an agent satis­fy­ing the Lin­ear Utility Hy­poth­e­sis would re­ject Pas­cal’s mug­ger, is, of course, the lev­er­age penalty that Eliezer dis­cusses in Pas­cal’s Mug­gle. The ar­gu­ment is that any hy­poth­e­sis in which there are n peo­ple, one of whom has a unique op­por­tu­nity to af­fect all the oth­ers, must im­ply that a ran­domly se­lected one of those n peo­ple has only a 1/​n chance of be­ing the one who has in­fluence. So if a hy­poth­e­sis im­plies that you have a unique op­por­tu­nity to af­fect n peo­ple’s lives, then this fact is ev­i­dence against this hy­poth­e­sis by a fac­tor of 1:n. In par­tic­u­lar, if Pas­cal’s mug­ger tells you that you are in a unique po­si­tion to af­fect 3↑↑↑3 lives, the fact that you are the one in this po­si­tion is 1 : 3↑↑↑3 ev­i­dence against the hy­poth­e­sis that Pas­cal’s mug­ger is tel­ling the truth. I have two crit­i­cisms of the lev­er­age penalty: first, that it is not the ac­tual rea­son that peo­ple re­ject Pas­cal’s mug­ger, and sec­ond, that it is not a cor­rect rea­son for an ideal ra­tio­nal agent to re­ject Pas­cal’s mug­ger. The lev­er­age penalty can’t be the ac­tual rea­son peo­ple re­ject Pas­cal’s mug­ger be­cause peo­ple don’t ac­tu­ally as­sign prob­a­bil­ity as low as 1/​3↑↑↑3 to the propo­si­tion that Pas­cal’s mug­ger is tel­ling the truth. This can be demon­strated with thought ex­per­i­ments. Con­sider what hap­pens when some­one en­coun­ters over­whelming ev­i­dence that Pas­cal’s mug­ger ac­tu­ally is tel­ling the truth. The prob­a­bil­ity of the ev­i­dence be­ing faked can’t pos­si­bly be less than 1 in 10^10^26 or so (this up­per bound was sug­gested by Eliezer in Pas­cal’s Mug­gle), so an agent with a lev­er­age prior will still be ab­solutely con­vinced that Pas­cal’s mug­ger is ly­ing. Eliezer sug­gests two rea­sons that an agent might pay Pas­cal’s mug­ger any­way, given a suffi­cient amount of ev­i­dence: first, that once you up­date to a prob­a­bil­ity of some­thing like 10^100 /​ 3↑↑↑3, and mul­ti­ply by the stakes of 3↑↑↑3 lives, you get an ex­pected util­ity of some­thing like 10^100 lives, which is worth a lot more than$5, and sec­ond, that the agent might just give up on the idea of a lev­er­age penalty and ad­mit that there is a non-in­finites­i­mal chance that Pas­cal’s mug­ger may ac­tu­ally be tel­ling the truth. Eliezer con­cludes, and I agree, that the first of these ex­pla­na­tions is not a good one. I can ac­tu­ally demon­strate this with a thought ex­per­i­ment. Sup­pose that af­ter show­ing you over­whelming ev­i­dence that they’re tel­ling the truth, Pas­cal’s mug­ger says “Oh, and by the way, if I was tel­ling the truth about the 3↑↑↑3 lives in your hands, then X is also true,” where X is some (a pri­ori fairly un­likely) propo­si­tion that you later have the op­por­tu­nity to bet on with a third party. Now, I’m sure you’d be ap­pro­pri­ately cau­tious in light of the fact that you would be very con­fused about what’s go­ing on, so you wouldn’t bet reck­lessly, but you prob­a­bly would con­sider your­self to have some spe­cial in­for­ma­tion about X, and if offered good enough odds, you might see a good op­por­tu­nity for profit with an ac­cept­able risk, which would not have looked ap­peal­ing be­fore be­ing told X by Pas­cal’s mug­ger. If you were re­ally as con­fi­dent that Pas­cal’s mug­ger was ly­ing as the lev­er­age prior would im­ply, then you wouldn’t as­sume X was any more likely than you thought be­fore for any pur­poses not in­volv­ing as­tro­nom­i­cal stakes, since your rea­son for be­liev­ing X is pred­i­cated on you hav­ing con­trol over as­tro­nom­i­cal stakes, which is as­tro­nom­i­cally un­likely.

And if you do as­sign a prob­a­bil­ity of 1/​3↑↑↑3 to some propo­si­tion, what is the em­piri­cal con­tent of this claim? One pos­si­ble an­swer is that this means that the odds at which you would be in­differ­ent to bet­ting on the propo­si­tion are 1 : 3↑↑↑3, if the bet is set­tled with some cur­rency that your util­ity func­tion is close to lin­ear with re­spect to across such scales. But the ex­is­tence of such a cur­rency is un­der dis­pute, and the em­piri­cal con­tent to the claim that such a cur­rency ex­ists is that you would make cer­tain bets with it in­volv­ing ar­bi­trar­ily ex­treme odds, so this is a very cir­cu­lar way to em­piri­cally ground the claim that you as­sign a prob­a­bil­ity of 1/​3↑↑↑3 to some propo­si­tion. So a good em­piri­cal ground­ing for this claim is go­ing to have to be in terms of prefer­ences be­tween more fa­mil­iar out­comes. And in terms of pay­offs at fa­mil­iar scales, I don’t see any­thing else that the claim that you as­sign a prob­a­bil­ity of 1/​3↑↑↑3 to a propo­si­tion could mean other than that you ex­pect to con­tinue to act as if the prob­a­bil­ity of the propo­si­tion is 0, even con­di­tional on any ob­ser­va­tions that don’t give you a like­li­hood ra­tio on the or­der of 1/​3↑↑↑3. If you claim that you would su­pe­rup­date long be­fore then, it’s not clear to me what you could mean when you say that your cur­rent prob­a­bil­ity for the propo­si­tion is 1/​3↑↑↑3.

There’s an­other way to see that bounded util­ity func­tions, not lev­er­age pri­ors, are Eliezer’s (and also pretty much ev­ery­one’s) true re­jec­tion to pay­ing Pas­cal’s mug­ger, and that is the fol­low­ing quote from Pas­cal’s Mug­gle: “I still feel a bit ner­vous about the idea that Pas­cal’s Muggee, af­ter the sky splits open, is hand­ing over five dol­lars while claiming to as­sign prob­a­bil­ity on the or­der of 10^9/​3↑↑↑3 that it’s do­ing any good.” This is an ad­mis­sion that Eliezer’s util­ity func­tion is bounded (even though Eliezer does not ad­mit that he is ad­mit­ting this) be­cause the ra­tio­nal agents whose util­ity func­tions are bounded are ex­actly (and tau­tolog­i­cally) char­ac­ter­ized by those for which there ex­ists a prob­a­bil­ity p>0 such that the agent would not spend [fixed amount of util­ity] for prob­a­bil­ity p of do­ing any good, no mat­ter what the good is. An agent satis­fy­ing the Lin­ear Utility Hy­poth­e­sis would spend $5 for a 10^9/​3↑↑↑3 chance of sav­ing 3↑↑↑3 lives. Ad­mit­ting that it would do the wrong thing if it was in that situ­a­tion, but claiming that that’s okay be­cause you have an elab­o­rate ar­gu­ment that the agent can’t be in that situ­a­tion even though it can be in situ­a­tions in which the prob­a­bil­ity is lower and can also be in situ­a­tions in which the prob­a­bil­ity is higher, strikes me as an ex­cep­tion­ally flimsy ar­gu­ment that the Lin­ear Utility Hy­poth­e­sis is com­pat­i­ble with hu­man val­ues. I also promised a rea­son that the lev­er­age penalty ar­gu­ment is not a cor­rect rea­son for ra­tio­nal agents (re­gard­less of com­pu­ta­tional con­straints) satis­fy­ing the Lin­ear Utility Hy­poth­e­sis to not pay Pas­cal’s mug­ger. This is that in weird situ­a­tions like this, you should be us­ing up­date­less de­ci­sion the­ory, and figure out which policy has the best a pri­ori ex­pected util­ity and im­ple­ment­ing that policy, in­stead of try­ing to make sense of weird an­thropic ar­gu­ments be­fore up­date­fully com­ing up with a strat­egy. Now con­sider the fol­low­ing hy­poth­e­sis: “There are 3↑↑↑3 copies of you, and a Ma­trix Lord will ap­proach one of them while dis­guised as an or­di­nary hu­man, in­form that copy about his pow­ers and in­ten­tions with­out offer­ing any solid ev­i­dence to sup­port his claims, and then kill the rest of the copies iff this copy de­clines to pay him$5. None of the other copies will ex­pe­rience or hal­lu­ci­nate any­thing like this.” Of course, this hy­poth­e­sis is ex­tremely un­likely, but there is no as­sump­tion that some ran­domly se­lected copy co­in­ci­den­tally hap­pens to be the one that the Ma­trix Lord ap­proaches, and thus no way for a lev­er­age penalty to force the prob­a­bil­ity of the hy­poth­e­sis be­low 1/​3↑↑↑3. This hy­poth­e­sis and the Lin­ear Utility Hy­poth­e­sis sug­gest that hav­ing a policy of pay­ing Pas­cal’s mug­ger would have con­se­quences 3↑↑↑3 times as im­por­tant as not dy­ing, which is worth well over \$5 in ex­pec­ta­tion, since the prob­a­bil­ity of the hy­poth­e­sis couldn’t be as low as 1/​3↑↑↑3. The fact that ac­tu­ally be­ing ap­proached by Pas­cal’s mug­ger can be seen as over­whelming ev­i­dence against this hy­poth­e­sis does noth­ing to change that.

Edit: I have writ­ten a fol­low-up to this.

• I’m not sure your re­fu­ta­tion of the lev­er­age penalty works. If there re­ally are 3 ↑↑↑ 3 copies of you, your de­ci­sion con­di­tioned on that may still not be to pay. You have to com­pare

P(A real mug­ging will hap­pen) x U(all your copies die)

against

P(fake mug­gings hap­pen) x U(lose five dol­lars) x (ex­pected num­ber of copies get­ting fake-mugged)

where that last term will in fact be pro­por­tional to 3 ↑↑↑ 3. Even if there is an in­com­pre­hen­si­bly vast ma­trix, its Dark Lords are pretty un­likely to mug you for petty cash. And this plau­si­bly does make you pay in the Mug­gle case, since P(fake mug­gings hap­pen) is way down if ‘mug­ging’ in­volves tear­ing a hole in the sky.

• Yes, it looks like you’re right. I’ll think about this and prob­a­bly write a fol­low-up later. Edit: I have fi­nally writ­ten that fol­low-up.

• Ab­solutely it is the case that util­ity should be bounded. How­ever as best I can tell you’ve left out the most fun­da­men­tal rea­son why, so I think I should ex­plain that here. (Per­haps I should make this a sep­a­rate post?)

The ba­sic ques­tion is: Where do util­ity func­tions come from? Like, why should one model a ra­tio­nal agent as hav­ing a util­ity func­tion at all? The an­swer of course is ei­ther the VNM the­o­rem or Sav­age’s the­o­rem, de­pend­ing on whether or not you’re pre-as­sum­ing prob­a­bil­ity (you shouldn’t, re­ally, but that’s an­other mat­ter). Right, both these the­o­rems take the form of, here’s a bunch of con­di­tions any ra­tio­nal agent should obey, let’s show that such an agent must in fact be act­ing ac­cord­ing to a util­ity func­tion (i.e. try­ing to max­i­mize its ex­pected value).

Now here’s the thing: The util­ity func­tions out­put by Sav­age’s the­o­rem are always bounded. Why is that? Well, es­sen­tially, be­cause oth­er­wise you could set up a St. Peters­burg para­dox that would con­tra­dict the as­sumed ra­tio­nal­ity con­di­tions (in short, you can set up two gam­bles, both of “in­finite ex­pected util­ity”, but where one dom­i­nates the other, and show that both A. the agent must pre­fer the first to the sec­ond, but also B. the agent must be in­differ­ent be­tween them, con­tra­dic­tion). Thus we con­clude that the util­ity func­tion must be bounded.

OK, but what if we base things around the VNM the­o­rem, then? It re­quires pre-as­sum­ing the no­tion of prob­a­bil­ity, but the util­ity func­tions out­put by the VNM the­o­rem aren’t guaran­teed to be bounded.

Here’s the thing: The VNM the­o­rem only guaran­tees that the util­ity func­tion it out­puts works for finite gam­bles. Se­ri­ously. The VNM the­o­rem gives no guaran­tee that the agent is act­ing ac­cord­ing to the speci­fied util­ity func­tion when pre­sented with a gam­ble with in­finitely many pos­si­ble out­comes, only when pre­sented with a gam­ble with finitely many out­comes.

Similarly, with Sav­age’s the­o­rem, the as­sump­tion that forces util­ity func­tions to be bounded—P7 -- is the same one that guaran­tees that the util­ity func­tion works for in­finite gam­bles. You can get rid of P7, and you’ll no longer be guaran­teed to get a bounded util­ity func­tion, but nei­ther will you be guaran­teed that the util­ity func­tion will work for gam­bles with in­finitely many pos­si­ble out­comes.

This means that, fun­da­men­tally, if you want to work with in­finite gam­bles, you need to only be talk­ing about bounded util­ity func­tions. If you talk about in­finite gam­bles in the con­text of un­bounded util­ity func­tions, well, you’re ba­si­cally talk­ing non­sense, be­cause there’s just ab­solutely no guaran­tee that the util­ity func­tion you’re us­ing ap­plies in such a situ­a­tion. The prob­lems of un­bounded util­ity that Eliezer keeps point­ing out, that he in­sists we need to solve, re­ally are just straight con­tra­dic­tions aris­ing from him mak­ing bad as­sump­tions that need to be thrown out. Like, they all stem from him as­sum­ing that un­bounded util­ity func­tions work in the case of in­finite gam­bles, and there sim­ply is no such guaran­tee; not in the VNM the­o­rem, not in Sav­age’s the­o­rem.

If you’re as­sum­ing in­finite gam­bles, you need to as­sume bounded util­ity func­tions, or else you need to ac­cept that in cases of in­finite gam­bles the util­ity func­tion doesn’t ac­tu­ally ap­ply—mak­ing the util­ity func­tion ba­si­cally use­less, be­cause, well, ev­ery­thing has in­finitely many pos­si­ble out­comes. Between a util­ity func­tion that re­mains valid in the face of in­finite gam­bles, and un­bounded util­ity, it’s pretty clear you should choose the former.

And be­tween Sav­age’s ax­iom P7 and un­bounded util­ity, it’s pretty clear you should choose the former. Be­cause P7 is an as­sump­tion that di­rectly de­scribes a ra­tio­nal­ity con­di­tion on the agent’s prefer­ences, a form of the sure-thing prin­ci­ple, one we can clearly see had bet­ter be true of any ra­tio­nal agent; while un­bounded util­ity… means what, ex­actly, in terms of the agent’s prefer­ences? Some­thing, cer­tainly, but not some­thing we ob­vi­ously need. And in fact we don’t need it.

As best I can tell, Eliezer keeps in­sist­ing we need un­bounded util­ity func­tions out of some sort of com­mit­ment to to­tal util­i­tar­i­anism or some­thing along the lines of such (that’s my sum­mary of his po­si­tion, any­way). I would con­sider that to be on much shak­ier ground (there are so many nonob­vi­ous large as­sump­tions for some­thing like that to even make sense, se­ri­ously I’m not even go­ing into it) than ob­vi­ous things like the sure-thing prin­ci­ple, or that a util­ity func­tion is nearly use­less if it’s not valid for in­finite gam­bles. And like I said, as best I can tell, Eliezer keeps as­sum­ing that the util­ity func­tion is valid in such situ­a­tions even though there’s noth­ing guaran­tee­ing this; and this as­sump­tion is just in con­tra­dic­tion with his as­sump­tion of an un­bounded util­ity func­tion. He should keep the val­idity as­sump­tion (which we need) and throw out the un­bound­ed­ness one (which we don’t).

That, to my mind, is the most fun­da­men­tal rea­son we should only be con­sid­er­ing bounded util­ity func­tions!

• The prob­lems of un­bounded util­ity that Eliezer keeps point­ing out, that he in­sists we need to solve, re­ally are just straight con­tra­dic­tions aris­ing from him mak­ing bad as­sump­tions that need to be thrown out. Like, they all stem from him as­sum­ing that un­bounded util­ity func­tions work in the case of in­finite gambles

Just to be clear, you’re not think­ing of 3↑↑↑3 when you talk about in­finite gam­bles, right?

I’m not sure I know what ar­gu­ment of Eliezer’s you’re talk­ing about when you refer­ence in­finite gam­bles. Is there an ex­am­ple you can link to?

• He means gam­bles that can have in­finitely many differ­ent out­comes. This causes prob­lems for un­bounded util­ity func­tions be­cause of the Saint Peters­burg para­dox.

• But the way you solve the St Peters­burg para­dox in real life is to note that no­body has in­finite money, nor in­finite time, and there­fore it doesn’t mat­ter if your util­ity func­tion spits out a weird out­come for it be­cause you can have a prior of 0 that it will ac­tu­ally hap­pen. Am I miss­ing some­thing?

• you can have a prior of 0 that it will ac­tu­ally happen

No.

• Huh, I only just saw this for some rea­son.

Any­way yes AlexMen­nen has the right of it.

I don’t have an ex­am­ple to hand of Eliezer’s re­marks. By which, I re­mem­ber see­ing on old LW, but I can’t find it at the mo­ment. (Note that I’m in­ter­pret­ing what he said… very char­i­ta­bly. What he ac­tu­ally said made con­sid­er­ably less sense, but we can per­haps steel­man it as a strong com­mit­ment to to­tal util­i­tar­i­anism.)

• I’m not fa­mil­iar with Sav­age’s the­o­rem, but I was aware of the things you said about the VNM the­o­rem, and in fact, I of­ten bring up the same ar­gu­ments you’ve been mak­ing. The stan­dard re­sponse that I hear is that some prob­a­bil­ity dis­tri­bu­tions can­not be up­dated to with­out an in­finite amount of in­for­ma­tion (e.g. if a pri­ori the prob­a­bil­ity of the nth out­come is pro­por­tional to 1/​3^n, then there can’t be any ev­i­dence that could oc­cur with nonzero prob­a­bil­ity that would con­vince you that the prob­a­bil­ity of the nth out­come is 1/​2^n for each n), and there’s no need for a util­ity func­tion to con­verge on gam­bles that it is im­pos­si­ble even in the­ory for you to be con­vinced are available op­tions.

When I ask why they as­sume that their util­ity func­tion should be valid on those in­finite gam­bles that are pos­si­ble for them to con­sider, if they aren’t as­sum­ing that their prefer­ence re­la­tion is closed in the strong topol­ogy (which im­plies that the util­ity func­tion is bounded), they’ll say some­thing like that their util­ity func­tion not be­ing valid where their prefer­ence re­la­tion is defined seems weirdly dis­con­tin­u­ous (in some sense that they can’t quite for­mal­ize and definitely isn’t the prefer­ence re­la­tion be­ing closed in the strong topol­ogy), or that the ex­am­ples I gave them of VNM-ra­tio­nal prefer­ence re­la­tions for which the util­ity func­tion isn’t valid for in­finite gam­bles all have some other pathol­ogy, like that there’s an in­finite gam­ble which is con­sid­ered ei­ther bet­ter than all of or worse than all of the con­stituent out­comes, and there might be a rep­re­sen­ta­tion the­o­rem say­ing some­thing like that has to hap­pen, even though they can’t point me to one.

Any­way, I agree that that’s a more fun­da­men­tal rea­son to only con­sider bounded util­ity func­tions, but I de­cided I could prob­a­bly be more con­vinc­ing by aban­don­ing that line of ar­gu­ment, and show­ing that if you sweep con­ver­gence is­sues un­der the rug, un­bounded util­ity func­tions still sug­gest in­sane be­hav­ior in con­crete situ­a­tions.

• Huh, I only just saw this for some rea­son. Any­way, if you’re not fa­mil­iar with Sav­age’s the­o­rem, that’s why I wrote the linked ar­ti­cle here about it! :)

• I think I dis­agree with your ap­proach here.

I, and I think most peo­ple in prac­tice, use re­flec­tive equil­ibrium to de­cide what our ethics are. This means that we can no­tice that our eth­i­cal in­tu­itions are in­sen­si­tive to scope, but also that upon re­flec­tion it seems like this is wrong, and thus adopt an ethics differ­ent from that given by our naive in­tu­ition.

When we’re try­ing to use logic to de­cide whether to ac­cept an eth­i­cal con­clu­sion counter to our in­tu­ition, it’s no good to doc­u­ment what our in­tu­ition cur­rently says as if that set­tles the mat­ter.

A pri­ori, 1,000 lives at risk may seem just as ur­gent as 10,000. But we think about it, and we do our best to over­ride it.

And in fact, I fail pretty hard at it. I’m pretty sure the amount I give to char­ity wouldn’t be differ­ent in a world where the effec­tive­ness of the best causes were an or­der of mag­ni­tude differ­ent. I sus­pect this is true of many; cer­tainly any­one fol­low­ing the Giv­ing What We Can pledge is us­ing an an­cient Schel­ling Point rather than any kind of calcu­la­tion. But that doesn’t mean you can con­vince me that my “real” ethics doesn’t care how many lives are saved.

When we talk about weird hy­po­thet­i­cals like Pas­cal­lian deals, we aren’t try­ing to figure out what our in­tu­ition says; we’re try­ing to figure out whether we should over­rule it.

• When you use philo­soph­i­cal re­flec­tion to over­ride naive in­tu­ition, you should have ex­plicit rea­sons for do­ing so. A rea­son for valu­ing 10,000 lives 10 times as much as 1,000 lives is that both of these are tiny com­pared to the to­tal num­ber of lives, so if you val­ued them at a differ­ent ra­tio, this would im­ply an oddly sharp bend in util­ity as a func­tion of lives, and we can tell that there is no such bend be­cause if we imag­ine that there were a few thou­sand more or fewer peo­ple on the planet, our in­tu­itions about that that par­tic­u­lar trade­off would not change. This rea­son­ing does not ap­ply to de­ci­sions af­fect­ing as­tro­nom­i­cally large num­bers of lives, and I have not seen any rea­son­ing that does which I find com­pel­ling.

It is also not true that peo­ple are try­ing to figure out whether to over­rule their in­tu­ition when they talk about Pas­cal’s mug­ging; typ­i­cally, they are try­ing to figure out how to jus­tify not over­rul­ing their in­tu­ition. How else can you ex­plain the pre­pon­der­ence of shaky “re­s­olu­tions” to Pas­cal’s mug­ging that ac­cept the Lin­ear Utility Hy­poth­e­sis and nonethe­less con­clude that you should not pay Pas­cal’s mug­ger, when “I tried to es­ti­mate the relevent prob­a­bil­ities fairly con­ser­va­tively, mul­ti­plied prob­a­bil­ities times util­ities, and pay­ing Pas­cal’s mug­ger came out far ahead” is usu­ally not con­sid­ered a re­s­olu­tion?

• Good points (so I up­voted), but the post could be half as long and make the same points bet­ter.

• This is all ba­si­cally right.

How­ever, as I said in a re­cent com­ment, peo­ple do not ac­tu­ally have util­ity func­tions. So in that sense, they have nei­ther a bounded nor an un­bounded util­ity func­tion. They can only try to make their prefer­ences less in­con­sis­tent. And you have two op­tions: you can pick some crazy con­sis­tency very differ­ent from nor­mal, or you can try to in­crease nor­mal­ity at the same time as in­creas­ing con­sis­tency. The sec­ond choice is bet­ter. And in this case, the sec­ond choice means pick­ing a bounded util­ity func­tion, and the first choice means choos­ing an un­bounded one, and go­ing in­sane (be­cause agree­ing to be mugged is in­sane.)

• Yes, that’s true. The fact that hu­mans are not ac­tu­ally ra­tio­nal agents is an im­por­tant point that I was ig­nor­ing.

“For in­stance, sup­pose you’re given a choice be­tween the fol­low­ing two op­tions: 1: Hu­man­ity grows into a vast civ­i­liza­tion of 10^100 peo­ple liv­ing long and happy lives, or 2: a 10% chance that hu­man­ity grows into a vast civ­i­liza­tion of 10^102 peo­ple liv­ing long and happy lives, and a 90% chance of go­ing ex­tinct right now. I think al­most ev­ery­one would pick op­tion 1, and would think it crazy to take a reck­less gam­ble like op­tion 2. But the Lin­ear Utility Hy­poth­e­sis says that op­tion 2 is much bet­ter.

It seems like se­lec­tively choos­ing a util­ity func­tion that does not weight the nega­tive util­ity of the state of ‘anti-lives’ that hav­ing NO civ­i­liza­tion of peo­ple liv­ing long and happy lives at all in the en­tire uni­verse would rep­re­sent.

I think you could tune a lin­ear re­la­tion­ship of these nega­tive val­ues and ac­cu­rately get the be­hav­ior that peo­ple have re­gard­ing these op­tions.

This seems a lot like pick­ing the weak­est and least cred­ible pos­si­ble ar­gu­ment to use as way to re­fute the en­tire idea. Which made it much more difficult for me to read the rest of your ar­ti­cle with the benefit of the doubt I would have prefered to have held through out.

• The Lin­ear Utility Hy­poth­e­sis does im­ply that there is no ex­tra penalty (on top of the usual lin­ear re­la­tion­ship be­tween pop­u­la­tion and util­ity) for the pop­u­la­tion be­ing zero, and it seems to me that it is com­mon for peo­ple to as­sume the Lin­ear Utility Hy­poth­e­sis un­mod­ified by such a zero-pop­u­la­tion penalty. Fur­ther­more, a zero-pop­u­la­tion penalty seems poorly mo­ti­vated to me, and still does not change the an­swer that Lin­ear Utility Hy­poth­e­sis + zero-pop­u­la­tion penalty would sug­gest in the thought ex­per­i­ment that you quoted, since you can just talk about pop­u­la­tions large enough to dwarf the zero-pop­u­la­tion penalty.

Re­fut­ing a weak ar­gu­ment for a hy­poth­e­sis is not a good way to re­fute the hy­poth­e­sis, but that’s not what I’m do­ing; I’m re­fut­ing weak con­se­quences of the Lin­ear Utility Hy­poth­e­sis, and “X im­plies Y, but not Y” is a perfectly le­gi­t­i­mate form of ar­gu­ment for “not X”.

• I pro­moted this to fea­tured, in small part be­cause I’m happy about ad­di­tions to im­por­tant con­ver­sa­tions (pas­cal’s mug­ging), in medium part be­cause of the good con­ver­sa­tion it in­spired (both in the com­ments and in this post by Zvi), and in large part be­cause I want peo­ple to know that these types of tech­ni­cal posts are re­ally awe­some and to­tally a good fit for the LW front­page, and if they con­tain good ideas they’ll to­tally be pro­moted.

• I’ve changed my mind. I agree with this.

The prob­a­bil­ity of you get­ting struck by light­ning and dy­ing while mak­ing your de­ci­sion is .

The prob­a­bil­ity of you dy­ing by a me­teor strike, by an earth­quake, by …. is .

The prob­a­bil­ity that you don’t get to com­plete your de­ci­sion for one rea­son or the other is .

It doesn’t make sense then to en­ter­tain prob­a­bil­ities vastly lower than , but not en­ter­tain prob­a­bil­ities much higher than .

What hap­pens is that our util­ity is bounded at or be­low .

This is be­cause we ig­nore prob­a­bil­ities at that or­der of mag­ni­tude via re­vealed prefer­ence the­ory.

If you value at times more util­ity than , then you are in­differ­ent be­tween ex­chang­ing for a times chance of .

I’m not in­differ­ent be­tween ex­chang­ing for a chance of any­thing, so my util­ity is bounded at . It is pos­si­ble that oth­ers deny this is true for them, but they are be­ing in­con­sis­tent. They ig­nore other events with higher prob­a­bil­ity which may pre­vent them from de­cid­ing, but con­sider events they as­sign vastly lower prob­a­bil­ity.

• Yes, this is the re­fu­ta­tion for Pas­cal’s mug­ger that I be­lieve in, al­though I never got around to writ­ing it up like you did. How­ever, I dis­agree with you that it im­plies that our util­ities must be bounded. All the ar­gu­ment shows is that or­di­nary peo­ple never as­sign to events enour­mous util­ity val­ues with also as­sign­ing them com­men­su­ably low prob­a­bil­ities. That is, nor­ma­tive claims (i.e., claims that cer­tain events have cer­tain util­ity as­signed to them) are judged fun­da­men­tally differ­ently from fac­tual claims, and re­quire more ev­i­dence than merely the com­plex­ity prior. In a moral in­tu­ition­ist frame­work this is the fact that any­one can say that 3^^^3 lives are suffer­ing, but it would take liv­ing 3^^^3 years and get­ting to know 3^^^3 peo­ple per­son­ally to feel the 3^^^3 times util­ity as­so­ci­ated with this events.

I don’t know how to dis­t­in­guish the sce­nar­ios where our util­ities are bounded and where our util­ities are un­bounded but reg­u­larized (or whether our util­ities are suffiently well-defined to dis­t­in­guish the two). Still, I want to em­pha­size that the lat­ter situ­a­tion is pos­si­ble.

• All the ar­gu­ment shows is that or­di­nary peo­ple never as­sign to events enour­mous util­ity val­ues with also as­sign­ing them com­men­su­ably low prob­a­bil­ities.

Un­der cer­tain as­sump­tions (which per­haps could be quib­bled with), de­mand­ing that util­ity val­ues can’t grow too fast for ex­pected util­ity to be defined ac­tu­ally does im­ply that the util­ity func­tion must be bounded.

un­bounded but regularized

I’m not sure what you mean by that.

• The as­sump­tion I’m talk­ing about is that the state of the rest of the uni­verse (or mul­ti­verse) does not af­fect the marginal util­ity of there also be­ing some­one hav­ing cer­tain ex­pe­riences at some lo­ca­tion in the uni-/​multi-verse.

Now, I am not a friend of prob­a­bil­ities /​ util­ities sep­a­rately; in­stead, con­sider your de­ci­sion func­tion.

Lin­ear­ity means that your de­ci­sions are in­de­pen­dent of ob­ser­va­tions of far parts of the uni­verse. In other words, you have one sys­tem over which your agent op­ti­mizes ex­pected util­ity; and now com­pare it to the situ­a­tion where you have two sys­tems. Your util­ity func­tion is lin­ear iff you can make de­ci­sions lo­cally, that is, with­out con­sid­er­ing the state of the other sys­tem.

Clearly, al­most noth­ing has a lin­ear de­ci­sion /​ util­ity func­tion.

I think peo­ple mis­take the fol­low­ing (amaz­ing) heuris­tic for lin­ear util­ity: If there are very many lo­cal sys­tems, and you have a suffi­ciently smooth util­ity and prob­a­bil­ity dis­tri­bu­tion for all of them, then you can do mean-field: You don’t need to look, the law of large num­bers guaran­tees strong bounds. In this sense, you don’t need to cou­ple all the sys­tems, they just all cou­ple to the mean-field.

To be more prac­ti­cal: Some­one might claim to have al­most lin­ear (al­tru­is­tic) util­ity for QALYs over the 5 years (so time-dis­count­ing is ir­rele­vant). Equiv­a­lently, whether some war in the mid­dle east is ter­rible or not does not in­fluence his/​her malaria-fo­cused char­ity work (say, he/​she only de­cides on this spe­cific topic).

Awe­some, he/​she does not need to read the news! And this is true to some ex­tent, but be­comes bul­lshit at the tails of the dis­tri­bu­tion. (the news be­come rele­vant if e.g. the nukes fly, be­cause they bring you into a more non­lin­ear regime for util­ity; on the other hand, given an al­most fixed back­ground pop­u­la­tion, log-util­ity and lin­ear-util­ity are in­dis­t­in­guish­able by Tay­lor’s rule)

Re pas­cal’s mug­gle: Ob­vi­ously your chance of get­ting a stroke and hal­lu­ci­nat­ing weird stuff out­weighs your chance of wit­ness­ing magic. I think that it is quite clear that you can for­get about the marginal cost of giv­ing 5 bucks to an imag­i­nary mug­ger be­fore the am­bu­lance ar­rives to maybe save you; de­ci­sion-the­o­ret­i­cally, you win by pre­com­mit­ting to pay the mug­ger and call an am­bu­lance once you ob­serve some­thing suffi­ciently weird.

• Per­haps you could edit and un-bold this comment

• I have now done this.

• Thanks, and sorry for pre­sum­ably mess­ing up the for­mat­ting.

• : 1: Hu­man­ity grows into a vast civ­i­liza­tion of 10^100 peo­ple liv­ing long and happy lives, or 2: a 10% chance that hu­man­ity grows into a vast civ­i­liza­tion of 10^102 peo­ple liv­ing long and happy lives, and a 90% chance of go­ing ex­tinct right now. I think al­most ev­ery­one would pick op­tion 1, and would think it crazy to take a reck­less gam­ble like op­tion 2. But the Lin­ear Utility Hy­poth­e­sis says that op­tion 2 is much bet­ter. Most of the ways peo­ple re­spond to Pas­cal’s mug­ger don’t ap­ply to this situ­a­tion, since the prob­a­bil­ities and ra­tios of util­ities in­volved here are not at all ex­treme.

This is not an air­tight ar­gu­ment.

Ex­tinc­tion of hu­man­ity is not 0 utils, it’s nega­tive utils. Let the util­ity of hu­man ex­tinc­tion be -X.

If X > 10^101, then a lin­ear util­ity func­tion would pick op­tion 1.

Lin­ear Ex­pected Utility (LEU) of op­tion 1:

1.0(10^100) = 10^100.

LEU of op­tion 2:

0.9(-X) + 0.1(10^102) = 10^101 + 0.9(-X)

10^101 − 0.9X < 10^100

-0.9X < 10^100 − 10^101

-0.9X < −9(10^100)

X > 10^101.

I place the ex­tinc­tion of hu­man­ity pretty highly, as ot cur­tails any pos­si­ble fu­ture. So X is always at least as high as the utopia. I would not ac­cept any utopia where the P of hu­man ex­tinc­tion was > 0.51, be­cause the negutil­ity of hu­man ex­tinc­tion out­weighs util­ity of utopia for any pos­si­ble utopia.

• Ex­tinc­tion of hu­man­ity just means hu­man­ity not ex­ist­ing in the fu­ture, so the Lin­ear Utility Hy­poth­e­sis does im­ply its value is 0. If you make an ex­cep­tion and add a penalty for ex­tinc­tion that is larger than the Lin­ear Utility Hy­poth­e­sis would dic­tate, then the Lin­ear Utility Hy­poth­e­sis ap­plied to other out­comes would still im­ply that when con­sid­er­ing suffi­ciently large po­ten­tial fu­ture pop­u­la­tions, this ex­tinc­tion penalty be­comes neg­ligible in com­par­i­son.

• See this thread. there is no finite num­ber of lives that reach utopia, for which I would ac­cept Omega’s bet at a 90% chance of ex­tinc­tion.

Hu­man ex­tinc­tion now for me is worse than los­ing 10 trillion peo­ple, if the global pop­u­la­tion was 100 trillion.

My di­su­til­ity of ex­tinc­tion isn’t just the num­ber of lives lost. It in­volves the ter­mi­na­tion of all fu­ture po­ten­tial of hu­man­ity, and I’m not sure how to value that, but see the bolded.

I don’t as­sign a di­su­til­ity to ex­tinc­tion, and my prefer­ence with re­gards to ex­tinc­tion is prob­a­bly lex­i­co­graphic with re­spect to some other things (see above).

• That’s a rea­son­able value judge­ment, but it’s not what the Lin­ear Utility Hy­poth­e­sis would pre­dict.

• My point is that for me, ex­tinc­tion is not equiv­a­lent to los­ing cur­rent amount of lives now.

Hu­man ex­tinc­tion now for me is worse than los­ing 10 trillion peo­ple, if the global pop­u­la­tion was 100 trillion.

This is be­cause ex­tinc­tion de­stroys all po­ten­tial fu­ture util­ity. It de­stroys thw po­ten­tial of hu­man­ity.

I’m say­ing that ex­tinc­tion can’t be eval­u­ated nor­mally, so you need a bet­ter ex­am­ple to state your ar­gu­ment against LUH.

Ex­tinc­tion now is worse than los­ing X peo­ple, if the global hu­man pop­u­la­tion is 10 X, ir­re­gard­less of how large X is.

That po­si­tion above is in­de­pen­dent of the lin­ear util­ity hy­poth­e­sis.

• It was speci­fied that the to­tal fu­ture pop­u­la­tion in each sce­nario was 10^100 and 10^102. Th­ese num­bers are the fu­ture peo­ple that couldn’t ex­ist if hu­man­ity goes ex­tinct.

• There is a num­ber of good rea­sons why one would re­fuse to pay the Pas­cal’s Mug­gle, and one is based on com­pu­ta­tional com­plex­ity: Small prob­a­bil­ities are costly. One needs more com­put­ing re­sources to eval­u­ate tiny prob­a­blitites ac­cu­rately. Once they are tiny enough, your calcu­la­tions will take too long to be of in­ter­est. And the num­bers un­der con­sid­er­a­tion ex­ceed 10^122, the largest num­ber of quan­tum states in the ob­serv­able uni­verse, by so much, you have no chance of mak­ing an ac­cu­rate eval­u­a­tion be­fore the heat death of the uni­verse.

In­ci­den­tally, this is one of the ar­gu­ments in physics against the AMPS black hole fire­wall para­dox: Col­lect­ing the out­go­ing Hawk­ing ra­di­a­tion takes way longer than the black hole evap­o­ra­tion time.

This also matches the in­tu­itive ap­proach. If you are con­fronted by some­thing as un­be­liev­able as “In the sky above, a gap edged by blue fire opens with a hor­ren­dous tear­ing sound”, as Eliezer col­or­fully puts it, then the first thing to re­al­ize is that you have no way to eval­u­ate the prob­a­bil­ity of this event be­ing “real” with­out much more work than you can pos­si­bly do in your life­time. Or at least in the time the Mug­ger wants you to make a de­ci­sion. Even to choose whether to pre­com­mit to some­thing like that re­quires more re­sources that you are likely to have. So the Eliezer’s state­ment ” If you as­sign su­per­ex­po­nen­tially in­finites­i­mal prob­a­bil­ity to claims of large im­pacts, then ap­par­ently you should ig­nore the pos­si­bil­ity of a large im­pact even af­ter see­ing huge amounts of ev­i­dence. ” has two points of failure based on com­pu­ta­tional com­plex­ity:

• As­sign­ing an ac­cu­rate su­per­ex­po­nen­tially in­finites­i­mal prob­a­bil­ity is com­pu­ta­tion­ally very ex­pen­sive.

• Figur­ing out the right amount of up­dat­ing af­ter “see­ing huge amounts of [ex­tremely sur­pris­ing] ev­i­dence is also com­pu­ta­tion­ally very ex­pen­sive.

So the naive ap­proach, “this is too un­be­liev­able to take se­ri­ously, no way I can figure out what is true, what is real and what is fake with any de­gree of con­fi­dence, might as well not bother” is ac­tu­ally the rea­son­able one.

• If you have an un­bounded util­ity func­tion, then putting a lot of re­sources into ac­cu­rately es­ti­mat­ing ar­bi­trar­ily tiny prob­a­bil­ities can be worth the effort, and if you can’t es­ti­mate them very ac­cu­rately, then you just have to make do with as ac­cu­rate an es­ti­mate as you can make.

• I agree with your con­clu­sions and re­ally like the post. Nev­er­the­less I would like to offer a defense of re­ject­ing Pas­cal’s Mug­ging even with an un­bounded util­ity func­tion, al­though I am not all that con­fi­dent that my defense is ac­tu­ally cor­rect.

Warn­ing: slight ram­bling and half-formed thoughts be­low. Con­tinue at own risk.

If I wish to con­sider the prob­a­bil­ity of the event [a stranger black­mails me with the threat to harm 3↑↑↑3 peo­ple] we should have some grasp of the like­li­ness of the claim [there ex­ists a per­son/​be­ing that can willfully harm 3↑↑↑3 peo­ple]. There are two rea­sons I can think of why this claim is so prob­le­matic that per­haps we should as­sign it on the or­der of 1/​3↑↑↑3 prob­a­bil­ity.

Firstly while the Knuth up-ar­row no­ta­tion is cutely com­pact, I think it is helpful to con­sider the set of claims [there ex­ists a be­ing that can willfully harm n peo­ple] for each value of n. Each claim im­plies all those be­fore it, so the prob­a­bil­ities of this se­quence should be de­creas­ing. From this point of view I find it not at all strange that the prob­a­bil­ity of this claim can make ex­plicit refer­ence to the num­ber n and be­have as some­thing like 1/​n. The point I’m try­ing to make is that while 3↑↑↑3 is only 5 sym­bols, the claim to be able to harm that many peo­ple is so much more ex­traor­di­nary than a a lot of similar claims that we might be jus­tified in as­sign­ing it re­ally low prob­a­bil­ity. Com­pare it to an ar­tifi­cal lot­tery where we force our com­puter to draw a num­ber be­tween 0 and 10^10^27 (de­liber­ately cho­sen larger than the ‘prac­ti­cal limit’ referred to in the post). I think we can justly as­sign the claim [The num­ber 3 will be the win­ning lot­tery ticket] a prob­a­bil­ity of 1/​10^10^27. Some­thing similar is go­ing on here: there are so many hy­poth­e­sis about be­ing able to harm n peo­ple that in or­der to sum the prob­a­bil­ities to 1 we are forced to as­sign on the or­der of 1/​n prob­a­bil­ity to each of them.

Se­condly (and I hope this re­in­forces the slight ram­bling above) con­sider how you might be con­vinced that this stranger can harm 3↑↑↑3 peo­ple, as op­posed to only has the abil­ity to harm 3↑↑↑3 − 1 peo­ple. I think the ‘tear­ing open the sky’ magic trick wouldn’t do it—this will in­crease our con­fi­dence in the stranger be­ing very pow­er­ful by ex­treme amounts (or, more re­al­is­ti­cally, con­vince us that we’ve gone com­pletely in­sane), but I see no rea­son why we would be forced to as­sign sig­nifi­cant prob­a­bil­ity to this stranger be­ing able to harm 3↑↑↑3 peo­ple, in­stead of ‘just’ 10^100 or 10^10^26 peo­ple or some­thing. Or in more Bayesian terms—which ev­i­dence E is more likely if this stranger can harm 3↑↑↑3 peo­ple than if it can harm, say, only 10^100 peo­ple? Which E satis­fies P(E|stranger can harm 3↑↑↑3 peo­ple) > P(E|stranger can harm 10^100 peo­ple but not 3↑↑↑3 peo­ple)? Any sug­ges­tions along the lines of ‘pre­sent 3↑↑↑3 peo­ple and punch them all in the nose’ jus­tifies hav­ing a prior of 1/​3↑↑↑3 for this event, since show­ing that many peo­ple re­ally is ev­i­dence with that like­li­hood ra­tio. But, hy­po­thet­i­cally, if all such ev­i­dence E are of this form, are our ac­tions then not con­sis­tent with the in­finites­i­mal prior, since we re­quire this par­tic­u­lar like­li­hood ra­tio be­fore we con­sider the hy­poth­e­sis likely?

I hope I haven’t ram­bled too much, but I think that the Knuth up-ar­row no­ta­tion is hid­ing the com­plex­ity of the claim in Pas­cal’s Mug­ging, and that the ev­i­dence re­quired to con­vince me that a be­ing re­ally has the power to do as is claimed has a likely­hood ra­tio close to 3↑↑↑3:1.

• It is not true that if some­one has the abil­ity to harm n peo­ple, then they also nec­es­sar­ily have the abil­ity to harm ex­actly m of those peo­ple for any m<n, so it isn’t clear that P(there is some­one who has the abil­ity to harm n peo­ple) mono­ton­i­cally de­creases as n in­creases. Un­less you meant at least n peo­ple, in which case that’s true, but still ir­relevent since it doesn’t even es­tab­lish that this prob­a­bil­ity ap­proaches a limit of 0, much less that it does so at any par­tic­u­lar rate.

Com­pare it to an ar­tifi­cal lot­tery where we force our com­puter to draw a num­ber be­tween 0 and 10^10^27 (de­liber­ately cho­sen larger than the ‘prac­ti­cal limit’ referred to in the post). I think we can justly as­sign the claim [The num­ber 3 will be the win­ning lot­tery ticket] a prob­a­bil­ity of 1/​10^10^27.

The prob­a­bil­ity that a num­ber cho­sen ran­domly from the uniform dis­tri­bu­tion on in­te­gers from 0 to 10^10^27 is 3 is in­deed 1/​10^10^27, but I wouldn’t count that as an em­piri­cal hy­poth­e­sis. Given any par­tic­u­lar mechanism for pro­duc­ing in­te­gers that you are very con­fi­dent im­ple­ments a uniform dis­tri­bu­tion on the in­te­gers from 0 to 10^10^27, the prob­a­bil­ity you should as­sign to it pro­duc­ing 3 is still much higher than 1/​10^10^27.

Or in more Bayesian terms—which ev­i­dence E is more likely if this stranger can harm 3↑↑↑3 peo­ple than if it can harm, say, only 10^100 peo­ple?

The stranger says so, and has es­tab­lished some cred­i­bil­ity by pro­vid­ing very strong ev­i­dence for other a pri­ori very im­plau­si­ble claims that they have made.

Any sug­ges­tions along the lines of ‘pre­sent 3↑↑↑3 peo­ple and punch them all in the nose’ jus­tifies hav­ing a prior of 1/​3↑↑↑3 for this event, since show­ing that many peo­ple re­ally is ev­i­dence with that like­li­hood ra­tio.

That’s not ev­i­dence that is phys­i­cally pos­si­ble to pre­sent to a hu­man, and I don’t see why you say its like­li­hood ra­tio is around 1:3↑↑↑3.

• I think I will try writ­ing my re­ply as a full post, this dis­cus­sion is get­ting longer than is easy to fit as a set of replies. You are right that my above re­ply has some se­ri­ous flaws.

• Another ap­proach might be to go meta. As­sume that there are many dire threats the­o­ret­i­cally pos­si­ble which, if true, would jus­tify a per­son in the sole po­si­tion stop them, do­ing so at near any cost (from pay­ing a penny or five pounds, all the way up to the per­son cut­ting their own throat, or press­ing a nuke launch­ing but­ton that would wipe out the hu­man species). In­deed, once the size of ac­tion re­quested in re­sponse to the threat is maxed out (it is the biggest re­sponse the in­di­vi­d­ual is ca­pa­ble of mak­ing), all such claims are func­tion­ally iden­ti­cal—the mag­ni­tiude of the threat be­yond that needed to max out the re­sponse, is ir­rele­vant. In this con­text, there is no differ­ence be­tween 3↑↑↑3 and 3↑↑↑↑3 .

But, what policy upon re­spond­ing to claims of such threats, should a species have, in or­der to max­imise ex­pected util­ity?

The moral haz­ard from en­courag­ing such claims to be made falsely needs to be taken into ac­count.

It is that moral haz­ard which has to be bal­anced against a pool of money that, species wide, should be risked on cov­er­ing such bets. Think of it this way: sup­pose I, Pas­cal’s Po­lice­man, were to make the claim “On be­half of the time po­lice, in or­der to de­ter con­fi­dence trick­sters, I hereby guaran­tee that an ad­di­tional util­ity will be added to the mul­ti­verse equal in mag­ni­tude to the sum of all offers made by Pas­cal Mug­gers that hap­pen to be tel­ling the truth (if any), in ex­change for your not re­spond­ing pos­i­tively to their threats or offers.”

It then be­comes a mat­ter of weigh­ing the ev­i­dence pre­sented by differ­ent mug­gers and po­lice­men.

• I think the most nat­u­ral ar­gu­ment for the Lin­ear Utility Hy­poth­e­sis is im­plicit in your third sen­tence: the nega­tion of the LUH is that the state of the rest of the uni­verse does af­fect the marginal util­ity of some­one hav­ing cer­tain ex­pe­riences. This seems to im­ply that you value the ex­pe­riences not for their own sake, but only for their re­la­tion to the rest of hu­man­ity. Now your “small-scale coun­terex­am­ple” seems to me point­ing out that peo­ple value ex­pe­riences not only for their own sake but also for their re­la­tions with other ex­pe­riences. But this does not con­tra­dict that peo­ple also do value the ex­pe­riences for their own sake.

So maybe we need LUH+: the util­ity has a lin­ear term plus some term that de­pends on re­la­tions.

I don’t know what to make of Pas­cal’s Mug­ger. I don’t think we have very good in­tu­itions about it. Maybe we should pay the mug­ger (not that this is an offer).

Here’s a weak­ened ver­sion of LUH+ that might be more plau­si­ble: maybe you value ex­pe­riences for their own sake, but once an ex­pe­rience is in­stan­ti­ated some­where in the uni­verse you don’t care about in­stan­ti­at­ing it again. There are enough pos­si­ble ex­pe­riences that this vari­a­tion doesn’t do much to change “as­tro­nom­i­cal waste” type ar­gu­ments, but it does at least bring Pas­cal’s mug­ger down to a some­what rea­son­able level (nowhere near 3^^^3).

• I don’t see the state­ment of the nega­tion of LUH as a com­pel­ling ar­gu­ment for LUH.

LUH+ as you de­scribed it is some­what un­der­speci­fied, but it’s an idea that might be worth look­ing into. That said, I think it’s still not vi­able in its cur­rent form be­cause of the other coun­terex­am­ples.

maybe you value ex­pe­riences for their own sake, but once an ex­pe­rience is in­stan­ti­ated some­where in the uni­verse you don’t care about in­stan­ti­at­ing it again.

I don’t think the marginal value of an ex­pe­rience oc­cur­ing should be 0 if it also oc­curs el­se­where. That as­sump­tion (to­gether with the many-wor­lds in­ter­pre­ta­tion) would sug­gest that quan­tum suicide should be perfectly fine, but even peo­ple who are ex­tremely con­fi­dent in the many-wor­lds in­ter­pre­ta­tion typ­i­cally typ­i­cally feel that quan­tum suicide is about as bad an idea as reg­u­lar suicide.

Fur­ther­more, I don’t think it makes sense to clas­sify pairs of ex­pe­riences as “the same” or “not the same”. There may be two con­scious ex­pe­riences that are so close to iden­ti­cal that they may as well be iden­ti­cal as far as your prefer­ences over how much each con­scious ex­pe­rience oc­curs are con­cerned. But there may be a long chain of con­scious ex­pe­riences, each of which differ­ing im­per­cep­ti­bly from the pre­vi­ous, such that the ex­pe­riences on each end of the chain are rad­i­cally differ­ent. I don’t think it makes sense to have a dis­con­tin­u­ous jump in the marginal util­ity of some con­scious ex­pe­rience oc­cur­ing as it crosses some ar­tifi­cial bound­ary be­tween those that are “the same” and those that are “differ­ent” from some other con­scious ex­pe­rience. I do think it makes sense to dis­count the marginal value of con­scious ex­pe­riences oc­cur­ing based on similar­ity to other con­scious ex­pe­riences that are in­stan­ti­ated el­se­where.

• I think usu­ally when the ques­tion is whether one thing af­fects an­other, the bur­den of proof is on the per­son who says it does af­fect the other. Any­way, the real point is that peo­ple do claim to value ex­pe­riences for their own sake as well as for their re­la­tions to other ex­pe­riences, and trans­lated into util­ity terms this seems to im­ply LUH+.

I’m not sure what you mean by “the other coun­terex­am­ples”: I think your first ex­am­ple is just demon­strat­ing that peo­ple’s Sys­tem 1s mostly re­place the num­bers 10^100 and 10^102 by 100 and 102 re­spec­tively, not in­di­cat­ing any deep fact about moral­ity. (To be clear, I mean that peo­ple’s sense of how “differ­ent” the 10^100 and 10^102 out­comes are seems to be based on the close­ness of 100 and 102, not that peo­ple don’t re­al­ize that one is 100 times big­ger.)

LUH+ is cer­tainly un­der­speci­fied as a util­ity func­tion, but as a hy­poth­e­sis about a util­ity func­tion it seems to be rea­son­ably well speci­fied?

In­tro­duc­ing MWI seems to make the whole dis­cus­sion a lot more com­pli­cated: to get a bounded util­ity func­tion you need the marginal util­ity of an ex­pe­rience to de­crease with re­spect to how many similar ex­pe­riences there are, but MWI says there are always a lot of similar ex­pe­riences, so marginal util­ity is always small? And if you are deal­ing with quan­tum ran­dom­ness then LUH just says you want to max­i­mize the to­tal quan­tum mea­sure of ex­pe­riences, which is a mono­ton­ic­ity hy­poth­e­sis rather than a lin­ear­ity hy­poth­e­sis, so it’s harder to see how you can deny it. Per­son­ally I don’t think that MWI is rele­vant to de­ci­sion the­ory: I think I only care about what hap­pens in this quan­tum branch.

I agree with you about the prob­lems that arise if you want a util­ity func­tion to de­pend on whether ex­pe­riences are “the same”.

• I think the coun­terex­am­ples I gave are clear proof that LUH is false, but as long as we’re play­ing bur­den of proof ten­nis, I dis­agree that the bur­den of proof should lie on those who think LUH is false. Hu­man value is com­pli­cated, and shouldn’t be as­sumed by de­fault to satisfy what­ever nice prop­er­ties you think up. A ran­dom func­tion from states of the uni­verse to real num­bers will not satisfy LUH, and while hu­man prefer­ences are far from ran­dom, if you claim that they satisfy any par­tic­u­lar strong struc­tural con­straint, it’s on you to ex­plain why. I also dis­agree that “valu­ing ex­pe­riences for their own sake” im­plies LUH; that some­what vague ex­pres­sion still sounds com­pat­i­ble with the marginal value of ex­pe­riences de­creas­ing to zero as the num­ber of them that have already oc­cured in­creases.

Prefer­ences and bias are deeply in­ter­twined in hu­mans, and there’s no ob­jec­tive way to de­ter­mine whether an ex­pressed prefer­ence is due pri­mar­ily to one or the other. That said, at some point, if an ex­pres­sion of prefer­ence is suffi­ciently strongly held, and the ar­gu­ments that it is ir­ra­tional are suffi­ciently weak, it gets hard to deny that prefer­ence has any­thing to do with it, even if similar thought pat­terns can be shown to be bias. This is where I’m at with 10^100 lives ver­sus a gam­ble on 10^102. I’m aware scope in­sen­si­tivity is a bias that can be shown to be ir­ra­tional in most con­texts, and in those con­texts, I tend to be re­cep­tive to scope sen­si­tivity ar­gu­ments, but I just can­not ac­cept that I “re­ally should” take a gam­ble that has a 90% chance of de­stroy­ing ev­ery­thing, just to try to in­crease pop­u­la­tion 100-fold af­ter already win­ning pretty de­ci­sively. Do you see this differ­ently? In any case, that ex­am­ple was in­tended as an ana­log of Pas­cal’s mug­ging with more nor­mal prob­a­bil­ities and ra­tios be­tween the stakes, and vir­tu­ally no one ac­tu­ally thinks it’s ra­tio­nal to pay Pas­cal’s mug­ger.

And if you are deal­ing with quan­tum ran­dom­ness then LUH just says you want to max­i­mize the to­tal quan­tum mea­sure of ex­pe­riences, which is a mono­ton­ic­ity hy­poth­e­sis rather than a lin­ear­ity hy­poth­e­sis, so it’s harder to see how you can deny it.

What are you try­ing to say here? If you only pay at­ten­tion to or­di­nal prefer­ences among sure out­comes, then the re­stric­tion of LUH to this con­text is a mono­ton­ic­ity hy­poth­e­sis, which is a lot more plau­si­ble. But you shouldn’t re­strict your at­ten­tion only to sure out­comes like that, since pretty much ev­ery choice you make in­volves un­cer­tainty. Per­haps you are un­der the mis­con­cep­tion that epistemic un­cer­tainty and the Born rule are the same thing?

this quan­tum branch

That doesn’t ac­tu­ally mean any­thing. You are in­stan­ti­ated in many Everette branches, and those branches will each in turn split into many fur­ther branches.

• Okay. To be fair I have two con­flict­ing in­tu­itions about this. One is that if you up­load some­one and man­age to give him a perfect ex­pe­rience, then filling the uni­verse with com­pu­tro­n­ium in or­der to run that pro­gram again and again with the same in­put isn’t par­tic­u­larly valuable; in fact I want to say it’s not any more valuable than just hav­ing the ex­pe­rience oc­cur once.

The other in­tu­ition is that in “nor­mal” situ­a­tions, peo­ple are just differ­ent from each other. And if I want to eval­u­ate how good some­one’s ex­pe­rience is, it seems con­de­scend­ing to say that it’s less im­por­tant be­cause some­one else already had a similar ex­pe­rience. How similar can such an ex­pe­rience be, in the real world? I mean they are differ­ent peo­ple.

There’s also the fact that the per­son hav­ing the ex­pe­rience likely doesn’t care about whether it’s hap­pened be­fore. A pig on a fac­tory farm doesn’t care that its ex­pe­rience is ba­si­cally the same as the ex­pe­rience of any other pig on the farm, it just wants to stop suffer­ing. On the other hand it seems like this ar­gu­ment could ap­ply to the up­load as well, and I’m not sure how to re­solve that.

Re­gard­ing 10^100 vs 10^102, I rec­og­nize that there are a lot of ways in which hav­ing such an ad­vanced civ­i­liza­tion could be counted as “win­ning”. For ex­am­ple there’s a good chance we’ve solved Hilbert’s sixth prob­lem by then which in my book is a pretty nice acheive­ment. And of course you can only do it once. But does it, or any similar met­ric that de­pends on the boolean ex­is­tence of civ­i­liza­tion, re­ally com­pare to the 10^100 lives that are at stake here? It seems like the an­swer is no, so ten­ta­tively I would take the gam­ble, though I could imag­ine be­ing con­vinced out of it. Of course, I’m aware I’m prob­a­bly in a minor­ity here.

It seems like peo­ple’s re­sponse to Pas­cal’s Mug­ging is par­tially de­pen­dent on fram­ing; e.g. the “as­tro­nom­i­cal waste” ar­gu­ment seems to get taken at least slightly more se­ri­ously. I think there is also a non-util­i­tar­ian flinch­ing at the idea of a de­ci­sion pro­cess that could be taken ad­van­tage of so eas­ily—I think I agree with the flinch­ing, but rec­og­nize that it’s not a util­i­tar­ian in­stinct.

I do re­al­ize that epistemic un­cer­tainty isn’t the same as the Born rule; that’s why I wrote “if you are deal­ing with quan­tum ran­dom­ness”—i.e. if you are deal­ing with a situ­a­tion where all of your epistemic un­cer­tainty is caused by quan­tum ran­dom­ness (or in MWI lan­guage, where all of your epistemic un­cer­tainty is in­dex­i­cal un­cer­tainty). Any­way, it sounds like you agree with me that un­der this hy­poth­e­sis MWI seems to im­ply LUH, but you think that the hy­poth­e­sis isn’t satis­fied very of­ten. Nev­er­the­less, it’s in­ter­est­ing that whether ran­dom­ness is quan­tum or not seems to be hav­ing con­se­quences for our de­ci­sion the­ory. Does it mean that we want more of our un­cer­tainty to be quan­tum, or less? Any­way, it’s hard for me to take these ques­tions too se­ri­ously since I don’t think MWI has de­ci­sion-the­o­retic con­se­quences, but I thought I would at least raise them.

Al­most ev­ery or­di­nary use of lan­guage pre­sumes that it makes sense to talk about the Everett branch that one is cur­rently in, and about the branch that one will be in in the fu­ture. Of course these are not perfectly well-defined con­cepts, but since when have we re­stricted our lan­guage to well-defined con­cepts?

• Okay. To be fair I have two con­flict­ing in­tu­itions about this. One is that if you up­load some­one and man­age to give him a perfect ex­pe­rience, then filling the uni­verse with com­pu­tro­n­ium in or­der to run that pro­gram again and again with the same in­put isn’t par­tic­u­larly valuable; in fact I want to say it’s not any more valuable than just hav­ing the ex­pe­rience oc­cur once.
The other in­tu­ition is that in “nor­mal” situ­a­tions, peo­ple are just differ­ent from each other. And if I want to eval­u­ate how good some­one’s ex­pe­rience is, it seems con­de­scend­ing to say that it’s less im­por­tant be­cause some­one else already had a similar ex­pe­rience. How similar can such an ex­pe­rience be, in the real world? I mean they are differ­ent peo­ple.

My in­tu­ition here is that the more similar the ex­pe­riences are to each other, the faster their marginal util­ity diminishes.

There’s also the fact that the per­son hav­ing the ex­pe­rience likely doesn’t care about whether it’s hap­pened be­fore.

That’s not clear. As I was say­ing, an agent hav­ing an ex­pe­rience has no way of refer­ing to one in­stan­ti­a­tion of it­self sep­a­rately from other iden­ti­cal in­stan­ti­a­tions of the agent hav­ing the same ex­pe­rience, so pre­sum­ably the agent cares about all in­stan­ti­a­tions of it­self in the same way, and I don’t see why that way must be lin­ear.

Re­gard­ing 10^100 vs 10^102, I rec­og­nize that there are a lot of ways in which hav­ing such an ad­vanced civ­i­liza­tion could be counted as “win­ning”. For ex­am­ple there’s a good chance we’ve solved Hilbert’s sixth prob­lem by then which in my book is a pretty nice acheive­ment. And of course you can only do it once. But does it, or any similar met­ric that de­pends on the boolean ex­is­tence of civ­i­liza­tion, re­ally com­pare to the 10^100 lives that are at stake here? It seems like the an­swer is no, so ten­ta­tively I would take the gam­ble, though I could imag­ine be­ing con­vinced out of it.

To be clear, by “win­ning”, I was refer­ing to the 10^100 flour­ish­ing hu­mans be­ing brought into ex­is­tence, not glo­ri­ous in­tel­lec­tual achieve­ments that would be made by this civ­i­liza­tion. Those are also nice, but I agree that they are in­signifi­cant in com­par­i­son.

Any­way, it sounds like you agree with me that un­der this hy­poth­e­sis MWI seems to im­ply LUH, but you think that the hy­poth­e­sis isn’t satis­fied very of­ten.

I didn’t to­tally agree; I said it was more plau­si­ble. Some­one could care how their copies are dis­tributed among Everette branches.

Nev­er­the­less, it’s in­ter­est­ing that whether ran­dom­ness is quan­tum or not seems to be hav­ing con­se­quences for our de­ci­sion the­ory. Does it mean that we want more of our un­cer­tainty to be quan­tum, or less?

Yes, I am in­ter­ested in this. I think the an­swer to your lat­ter ques­tion prob­a­bly de­pends on what the un­cer­tainty is about, but I’ll have to think about how it de­pends on that.

• Hmm. I’m not sure that refer­ence works the way you say it does. If an up­load points at it­self and the ex­pe­rience of point­ing is copied, it seems fair to say that you have a bunch of in­di­vi­d­u­als point­ing at them­selves, not all point­ing at each other. Not sure why other forms of refer­ence should be any differ­ent. Though if it does work the way you say, maybe it would ex­plain why up­loads seem to be differ­ent from pigs… un­less you think that the pigs can’t re­fer to them­selves ex­cept as a group ei­ther.

• I dis­agree with your con­clu­sion (as I ex­plained here, but up­vot­ing any­way for the con­tent of the post).

Also, it is pos­si­ble to have a lin­ear util­ity func­tion and still re­ject Pas­cal’s mug­ger if:

1. Your lin­ear util­ity func­tion is bounded.

2. You ap­ply my rule.

Thus, LUH is in­de­pen­dent of re­ject­ing Pas­cal’s mug­gle?

• The Lin­ear Utility Hy­poth­e­sis is in­con­sis­tent with hav­ing a bounded util­ity func­tion.

• This re­sult has even stronger con­se­quences than that the Lin­ear Utility Hy­poth­e­sis is false, namely that util­ity is bounded.

Not nec­es­sary. You don’t need to sup­pose bounded util­ity to ex­plain re­ject­ing Pas­cal’s mug­gle.

My rea­son for re­ject­ing pas­cal mug­gle is this.

If there ex­ists a set of states E_j such that:

1. P(E_j) < ep­silon.

2. There does not ex­ist E_k (E_k is not a sub­set of E_j and P(E_k) < ep­silon).

Then I ig­nore E_j in de­ci­sion mak­ing in sin­gle­ton de­ci­sion prob­lems. In iter­ated de­ci­sion prob­lems, the value for ep­silon de­pends on the num­ber of iter­a­tions.

I don’t have a name for this prin­ci­ple (and it is an ad-hoc patch I added to my de­ci­sion the­ory to pre­vent EU from be­ing dom­i­nated by tiny prob­a­bil­ities of vast util­ities).

This patch is differ­ent from bounded util­ity, be­cause you might ig­nore a set of atates in a sin­gle­ton prob­lem, but con­sider same set in an iter­ated prob­lem.