# Vaniver comments on Open thread, Mar. 2 - Mar. 8, 2015

• Why not “me­dian ex­pected util­ity”?

This might sound silly, but it’s deeper than it looks: the rea­son why we use the ex­pected value of util­ity (i.e. means) to de­ter­mine the best of a set of gam­bles is be­cause util­ity is defined as the thing that you max­i­mize the ex­pected value of.

The thing that’s nice about VNM util­ity is that it’s math­e­mat­i­cally con­sis­tent. That means we can’t come up with a sce­nario where VNM util­ity gen­er­ates silly out­puts with sen­si­ble in­puts. Of course we can give VNM silly in­puts and get silly out­puts back—sce­nar­ios like Pas­cal’s Mug­ging are the equiv­a­lent of “sup­pose some­thing re­ally weird hap­pens; wouldn’t that be weird?” to which the an­swer is “well, yes.”

The re­ally nice thing about VNM is that it’s the only rule that’s math­e­mat­i­cally con­sis­tent with it­self and a hand­ful of nice ax­ioms. You might give up one of those ax­ioms, but for any of those ax­ioms we can show an ex­am­ple where a rule that doesn’t fol­low that ax­iom will take sen­si­ble in­puts and give silly out­puts. So I don’t think there’s much to be gained by try­ing to re­place a mean de­ci­sion rule with a me­dian de­ci­sion rule or some other de­ci­sion rule—but there is a lot to be gained by sharp­en­ing our prob­a­bil­ity dis­tri­bu­tions and more clearly figur­ing out our map­ping from world-his­to­ries to util­ities.

• “sup­pose some­thing re­ally weird hap­pens; wouldn’t that be weird?”

To me, con­se­quen­tial­ism is ei­ther some­thing triv­ial or some­thing I re­ject, but that said this is a fully gen­eral (and kind of weak) counter-ar­gu­ment. I can ap­ply it to New­comb from the CDT point of view. I can ap­ply it to Smok­ing Le­sion from the EDT point of view! I can ap­ply it to as­tro­nomic data from the Ptolemy’s the­ory of ce­les­tial mo­tion point of view! We have to deal with things!

• To me, con­se­quen­tial­ism is ei­ther some­thing triv­ial or some­thing I reject

I hes­i­tate to call con­se­quen­tial­ism triv­ial, be­cause I wouldn’t use it to de­scribe a broad class of ‘in­tel­li­gent’ agents, but I also wouldn’t re­ject it, be­cause it does de­scribe the de­sign of those agents.

this is a fully gen­eral (and kind of weak) counter-ar­gu­ment.

I don’t see it as a counter-ar­gu­ment. In gen­eral, I think that a method is ap­pro­pri­ate if hard things are hard and easy things are easy—and, similarly, nor­mal things are nor­mal and weird things are weird. If the out­put is weird in the same way that the in­put is weird, the sys­tem is be­hav­ing ap­pro­pri­ately; if it adds or sub­tracts weird­ness, then we’re in trou­ble!

For ex­am­ple, sup­pose you sup­plied a prob­lem with a rele­vant log­i­cal con­tra­dic­tion to your de­ci­sion al­gorithm, and it spat out a sin­gle nu­mer­i­cal an­swer. Is that a sign of ro­bust­ness, or lack of ro­bust­ness?

• I hes­i­tate to call con­se­quen­tial­ism trivial

I just meant I ac­cept the con­se­quen­tial­ist idea in de­ci­sion the­ory that we should max­i­mize, e.g. pick the best out of al­ter­na­tives. But said in this way, it’s a triv­ial point. I re­ject more gen­eral va­ri­eties of con­se­quen­tial­ism (for rea­sons that are not im­por­tant right now, but ba­si­cally I think a lot of weird con­clu­sions of con­se­quen­tial­ism are due to mod­el­ing prob­lems, e.g. the set up that makes con­se­quen­tial­ism work doesn’t ap­ply well).

nor­mal things are nor­mal and weird things are weird

I don’t know what you are say­ing here. Can you taboo “weird?” New­comb is weird for CDT be­cause it ex­plic­itly vi­o­lates an as­sump­tion CDT is us­ing. The an­swer here is to go meta and think about a fam­ily of de­ci­sion the­o­ries of which CDT is one, in­dexed by their as­sump­tion sets.

• I just meant I ac­cept the con­se­quen­tial­ist idea in de­ci­sion the­ory that we should max­i­mize, e.g. pick the best out of al­ter­na­tives. But said in this way, it’s a triv­ial point.

I un­der­stood and agree with that state­ment of con­se­quen­tial­ism in de­ci­sion the­ory—what I dis­agree with is that it’s triv­ial that max­i­miza­tion is the right ap­proach to take! For many situ­a­tions, a re­flex­ive agent that does not ac­tively simu­late the fu­ture or con­sider al­ter­na­tives performs bet­ter than a con­tem­pla­tive agent that does simu­late the fu­ture and con­sid­er­ate al­ter­na­tives, be­cause the best al­ter­na­tive is “ob­vi­ous” and the acts of simu­la­tion and con­sid­er­a­tion con­sume time and re­sources that do not pay for them­selves.

That’s ob­vi­ously what’s go­ing on with ther­mostats, but I would ar­gue is what goes on all the way up to the con­se­quen­tial­ism-de­on­tol­ogy di­vide in ethics.

Can you taboo “weird?”

I would prob­a­bly re­place it with Pearl’s phrase here, of “sur­pris­ing or un­be­liev­able.”

To use the spe­cific ex­am­ple of New­comb’s prob­lem, if peo­ple find a perfect pre­dic­tor “sur­pris­ing or un­be­liev­able,” then they prob­a­bly also think that the right thing to do around a perfect pre­dic­tor is “sur­pris­ing or un­be­liev­able,” be­cause us­ing logic on an un­be­liev­able premise can lead to an un­be­liev­able con­clu­sion! Con­sider a Mun­dane New­comb’s prob­lem which is miss­ing perfect pre­dic­tion but has the same ev­i­den­tial and coun­ter­fac­tual fea­tures: that is, Omega offers you the choice of one or two boxes, you choose which boxes to take, and then it puts a mil­lion dol­lars in the red box and a thou­sand dol­lars in the blue box if you choose only the red box and it puts a thou­sand dol­lars in the blue box if you choose the blue box or no boxes. Any­one that un­der­stands the sce­nario and prefers more money to less money will choose just the red box, and there’s noth­ing sur­pris­ing or un­be­liev­able about it.

What is sur­pris­ing is the claim that there’s an en­tity who can repli­cate the coun­ter­fac­tual struc­ture of the Mun­dane New­comb sce­nario with­out also repli­cat­ing the tem­po­ral struc­ture of that sce­nario. But that’s a claim about physics, not de­ci­sion the­ory!

• be­cause the best al­ter­na­tive is “ob­vi­ous” and the acts of simu­la­tion and con­sid­er­a­tion con­sume time and re­sources that do not pay for them­selves.

Ab­solutely. This is the “bounded ra­tio­nal­ity” set­ting lots of peo­ple think about. For in­stance, Big Data is fash­ion­able these days, and lots of peo­ple think about how we may do usual statis­tics busi­ness un­der se­vere com­pu­ta­tional con­straints due to huge dataset sizes, e.g. stuff like this:

http://​​www.cs.berkeley.edu/​​~jor­dan/​​pa­pers/​​blb_icml2012.pdf

But in bounded ra­tio­nal­ity set­tings we still want to pick the best out of our al­ter­na­tives, we just have a con­straint that we can’t take more than a cer­tain amount of re­sources to re­turn an an­swer. The (triv­ial) idea of do­ing your best is still there. That is the part I ac­cept. But that part is bor­ing, think­ing of the right thing to max­i­mize is what is very sub­tle (and may in­volve non-con­se­quen­tial­ist ideas, for ex­am­ple a de­ci­sion the­ory that han­dles black­mail may in­volve virtue eth­i­cal ideas be­cause the re­turned an­swer de­pends on “the sort of agent” some­one is).

• I don’t agree. Utility is a sep­a­rate con­cept from ex­pected value max­i­miza­tion. Utility is a way of or­der­ing and com­par­ing differ­ent out­comes based on how de­sir­able they are. You can say that one out­come is more de­sir­able than an­other, or even quan­tify how many times more de­sir­able it is. This is a use­ful and gen­eral con­cept.

Ex­pected util­ity does have some nice prop­er­ties be­ing com­pletely con­sis­tent. How­ever I ar­gued above that this isn’t a nec­es­sary prop­erty. It adds com­plex­ity, sure, but if you self mod­ify your de­ci­sion mak­ing al­gorithm or pre­de­ter­mine your ac­tions, you can force your fu­ture self to be con­sis­tent with your pre­sent self’s de­sires.

Ex­pected util­ity is perfectly ra­tio­nal as the num­ber of “bets” you take goes to in­finity. Re­wards will can­cel out the losses in the limit, and so any agent would choose to fol­low EU re­gard­less of their de­ci­sion mak­ing al­gorithm. But as the num­ber of bets be­comes finite, it’s less ob­vi­ous that this is the most de­sir­able strat­egy.

That means we can’t come up with a sce­nario where VNM util­ity gen­er­ates silly out­puts with sen­si­ble in­puts. Of course we can give VNM silly in­puts and get silly out­puts back—sce­nar­ios like Pas­cal’s Mug­ging are the equiv­a­lent of “sup­pose some­thing re­ally weird hap­pens; wouldn’t that be weird?” to which the an­swer is “well, yes.”

Pas­cal’s Mug­ging isn’t “weird”, it’s perfectly typ­i­cal. There are prob­a­bly an in­finite num­ber of pas­cal’s mug­ging type situ­a­tions. Hy­pothe­ses with ex­ceed­ingly low prob­a­bil­ity but high util­ity.

If we built an AI to­day, based on pure ex­pected util­ity, it would most likely fail spec­tac­u­larly. Th­ese low prob­a­bil­ity hy­pothe­ses would come to to­tally dom­i­nate it’s de­ci­sions. Per­haps it would start to wor­ship var­i­ous gods and prac­tice rit­u­als and obey­ing su­per­sti­tions. Or some­thing far more ab­surd we haven’t even thought of.

And if you re­ally be­lieve in EU, you can’t say that this be­hav­ior is wrong or un­de­sir­able. This is what you should be do­ing, if you could, and you are los­ing a huge amount of EU by not do­ing it. You should want more than any­thing in ex­is­tence, the abil­ity to ex­actly calcu­late these hy­pothe­ses so you can col­lect that EU.

I don’t want that though. I want a de­ci­sion rule such that I am very likely to end up in a good out­come. Not one where I will mostly likely end up in a very sub­op­ti­mal out­come, with an in­finites­i­mal prob­a­bil­ity of win­ning the in­finite util­ity lot­tery.

• Ex­pected util­ity is con­ve­nient and makes for a nice math­e­mat­i­cal the­ory.

It also makes a lot of as­sump­tions. One as­sumes that the ex­pec­ta­tion does, in fact, ex­ist. It need not. For ex­am­ple, in a game where two play­ers toss a fair coin, we ex­pect that in the long run the num­ber of heads should equal the num­ber of tails at some point. It turns out that the ex­pected wait­ing time is in­finite. Then there’s the clas­sic St. Peters­burg para­dox.

There are ex­am­ples of “fair” bets (i.e. ex­pected gain is 0) that are nev­er­the­less un­fa­vor­able (in the sense that you’re al­most cer­tain to sus­tain a net loss over time).

Ex­pected util­ity is a model of re­al­ity that does a good job in many cir­cum­stances but has some key draw­backs where naive ap­pli­ca­tion will lead to un­re­al­is­tic de­ci­sions. The map is not the ter­ri­tory, af­ter all.

• Utility is a sep­a­rate con­cept from ex­pected value max­i­miza­tion. Utility is a way of or­der­ing and com­par­ing differ­ent out­comes based on how de­sir­able they are.

To Ben­tham, sure; to­day, we call some­thing that generic “rank­ing” or some­thing similar, be­cause VNM-util­ity is the only game in town when it comes to as­sign­ing real-val­ued de­sir­a­bil­ities to con­se­quences.

But as the num­ber of bets be­comes finite, it’s less ob­vi­ous that this is the most de­sir­able strat­egy.

Disagreed. The proof of the VNM ax­ioms goes through for a sin­gle bet; I recom­mend you look that up, and then try to cre­ate a coun­terex­am­ple.

Note that it’s easy to come up with a wrong util­ity map­ping. One could, say, map dol­lars lin­early to util­ity and then say “but I don’t pre­fer a half chance of \$100 and half chance of noth­ing to a cer­tain \$50!” , but that’s solved by chang­ing the util­ity map­ping from lin­ear from sub­lin­ear (say, log or sqrt or so on). In or­der to ex­hibit a coun­terex­am­ple it has to look like the Allais para­dox, where some­one con­firms two prefer­ences and then does not agree with the con­se­quence of those prefer­ences con­sid­ered to­gether.

There are prob­a­bly an in­finite num­ber of pas­cal’s mug­ging type situ­a­tions. Hy­pothe­ses with ex­ceed­ingly low prob­a­bil­ity but high util­ity.

It prob­a­bly isn’t the case that there are an in­finite num­ber of situ­a­tions where the util­ity times the prob­a­bil­ity is higher than the cost, and if there are, that’s prob­a­bly a faulty util­ity func­tion or faulty prob­a­bil­ity es­ti­ma­tor rather than a faulty EU calcu­la­tion. Con­sider this bit from You and Your Re­search by Ham­ming:

Let me warn you, `im­por­tant prob­lem’ must be phrased care­fully. The three out­stand­ing prob­lems in physics, in a cer­tain sense, were never worked on while I was at Bell Labs. By im­por­tant I mean guaran­teed a No­bel Prize and any sum of money you want to men­tion. We didn’t work on (1) time travel, (2) tele­por­ta­tion, and (3) anti­grav­ity. They are not im­por­tant prob­lems be­cause we do not have an at­tack. It’s not the con­se­quence that makes a prob­lem im­por­tant, it is that you have a rea­son­able at­tack. That is what makes a prob­lem im­por­tant. When I say that most sci­en­tists don’t work on im­por­tant prob­lems, I mean it in that sense.

An AI might cor­rectly calcu­late that time travel is the most pos­i­tive tech­nol­ogy it could pos­si­bly de­velop—but also quickly calcu­late that it has no idea where to even start, and so the prob­a­bil­ity of suc­cess from think­ing about it more is low enough that it should go for a more cred­ible op­tion. That’s what hu­man thinkers do and it doesn’t seem like a mis­take in the way that the Allais para­dox seems like a mis­take.

• But as the num­ber of bets be­comes finite, it’s less ob­vi­ous that this is the most de­sir­able strat­egy.

Disagreed. The proof of the VNM ax­ioms goes through for a sin­gle bet; I recom­mend you look that up, and then try to cre­ate a coun­terex­am­ple.

Pas­cal’s wa­ger is the coun­terex­am­ple, and it’s older than VNM. EY’s Pas­cal’s mug­ging was just an at­tempt to for­mal­ize it a bit more and pre­vent silly ex­cuses like “well what if we don’t al­low in­finites or as­sume the prob­a­bil­ities ex­actly can­cel out.”

Coun­terex­am­ple in that it vi­o­lates what hu­mans want, not that it pro­duces in­con­sis­tent be­hav­ior or any­thing. It’s perfectly valid for an agent to fol­low EU, as it is for it to fol­low my method. What we are ar­gu­ing about is en­tirely sub­jec­tive.

If you re­ally be­lieve in EU a pri­ori, then no ar­gu­ment should be able to con­vince you it is wrong. You would find noth­ing wrong with Pas­cal situ­a­tions, and to­tally agree with the re­sult of EU. You wouldn’t have to make clever ar­gu­ments about the util­ity func­tion or prob­a­bil­ity es­ti­mates to get out of it.

It prob­a­bly isn’t the case that there are an in­finite num­ber of situ­a­tions where the util­ity times the prob­a­bil­ity is higher than the cost, and if there are, that’s prob­a­bly a faulty util­ity func­tion or faulty prob­a­bil­ity es­ti­ma­tor rather than a faulty EU calcu­la­tion.

This is pretty thor­oughly ar­gued in the origi­nal Pas­cal’s Mug­ging post. Hy­pothe­ses of vast util­ity can grow much faster than their im­prob­a­bil­ity. The hy­poth­e­sis “you will be re­warded/​tor­tured 3^^^3 units” is in­finites­i­mally smaller in an EU calcu­la­tion to the hy­poth­e­sis “you will be re­warded/​tor­tured 3^^^^^^^3 units”, and only takes a few more bits to ex­press, and it can grow even fur­ther.

• Pas­cal’s wa­ger is the coun­terex­am­ple, and it’s older than VNM.

Coun­terex­am­ple in what sense? If you do in fact re­ceive in­finite util­ity from go­ing to heaven, and be­ing Chris­tian raises the chance of you go­ing to heaven by any pos­i­tive amount over your baseline chance, then it is the right move to be Chris­tian in­stead of baseline.

The rea­son peo­ple re­ject Pas­cal’s Wager or Mug­ging is, as I un­der­stand it, they don’t see the state­ment “you re­ceive in­finite util­ity from X” or “you re­ceive a huge amount of di­su­til­ity from Y” as ac­tual ev­i­dence about their fu­ture util­ity.

In gen­eral, I think that any prob­lem which in­cludes the word “in­finite” is guilty un­til proven in­no­cent, and it is much bet­ter to ex­press it as a limit. (This clears up a huge amount of con­fu­sion.) And the gen­eral prin­ci­ple- that as the prize for win­ning a lot­tery gets bet­ter, the prob­a­bil­ity of win­ning the lot­tery nec­es­sary to jus­tify buy­ing a fixed-price ticket goes down, seems like a rea­son­able prin­ci­ple to me.

It’s perfectly valid for an agent to fol­low EU, as it is for it to fol­low my method. What we are ar­gu­ing about is en­tirely sub­jec­tive.

I think money pumps ar­gue against sub­jec­tivity. Ba­si­cally, if you use an in­con­sis­tent de­ci­sion the­ory, some­one else can make money off your in­con­sis­tency or you don’t ac­tu­ally use that in­con­sis­tent de­ci­sion the­ory.

If you re­ally be­lieve in EU a pri­ori, then no ar­gu­ment should be able to con­vince you it is wrong. You would find noth­ing wrong with Pas­cal situ­a­tions, and to­tally agree with the re­sult of EU. You wouldn’t have to make clever ar­gu­ments about the util­ity func­tion or prob­a­bil­ity es­ti­mates to get out of it.

I will say right now: I be­lieve that if you have a com­plete set of out­comes with known util­ities and the prob­a­bil­ities of achiev­ing those out­comes con­di­tioned on tak­ing ac­tions from a set of pos­si­ble ac­tions, the best ac­tion in that set is the one with the high­est prob­a­bil­ity-weighted util­ity sum. That is, EU max­i­miza­tion works if you feed it the right in­puts.

Do I think it’s triv­ial to get the right in­puts for EU max­i­miza­tion? No! I’m not even sure it’s pos­si­ble ex­cept in ap­prox­i­ma­tion. Any prob­lem that starts with util­ities in the prob­lem de­scrip­tion has hid­den the hard work un­der the rug, and per­haps that means they’ve hid­den a ridicu­lous premise.

Hy­pothe­ses of vast util­ity can grow much faster than their im­prob­a­bil­ity.

As­sum­ing a par­tic­u­lar method of as­sign­ing prior prob­a­bil­ities to state­ments, yes. But is that the right method of as­sign­ing prior prob­a­bil­ities to state­ments?

(That is, yes, I’ve read Eliezer’s post, and he’s ask­ing how to gen­er­ate prob­a­bil­ities of con­se­quences given ac­tions. That’s a physics ques­tion, not a de­ci­sion the­ory ques­tion.)

• If you do in fact re­ceive in­finite util­ity from go­ing to heaven, and be­ing Chris­tian raises the chance of you go­ing to heaven by any pos­i­tive amount over your baseline chance, then it is the right move to be Chris­tian in­stead of baseline.

Where “right” is defined as “max­i­miz­ing ex­pected util­ity”, then yes. It’s just a tau­tol­ogy, “max­i­miz­ing ex­pected util­ity max­i­mizes ex­pected util­ity”.

My point is if you ac­tu­ally asked the av­er­age per­son, even if you ex­plained all this to them, they would still not agree that it was the right de­ci­sion.

There is no law writ­ten into the uni­verse that says you have to max­i­mize ex­pected util­ity. I don’t think that’ what hu­mans re­ally want. If we choose to fol­low it, in many situ­a­tion it will lead to un­de­sir­able out­comes. And it’s quite pos­si­ble that those situ­a­tions are ac­tu­ally com­mon.

It may mean life be­comes more com­pli­cated than mak­ing sim­ple EU calcu­la­tions, but you can still be perfectly con­sis­tent (see fur­ther down.)

In gen­eral, I think that any prob­lem which in­cludes the word “in­finite” is guilty un­til proven in­no­cent, and it is much bet­ter to ex­press it as a limit. (This clears up a huge amount of con­fu­sion.)

You could ex­press it as a limit triv­ially (e.g. a hy­poth­e­sis that in heaven you will col­lect 3^^^3 utilons per sec­ond for an un­end­ing amount of time.)

And the gen­eral prin­ci­ple- that as the prize for win­ning a lot­tery gets bet­ter, the prob­a­bil­ity of win­ning the lot­tery nec­es­sary to jus­tify buy­ing a fixed-price ticket goes down, seems like a rea­son­able prin­ci­ple to me.

Sounds rea­son­able, but it breaks down in ex­treme cases, where you end up spend­ing al­most all of your prob­a­bil­ity mass in ex­change for a sin­gle good fu­ture with ar­bi­trar­ily low prob­a­bil­ity.

Here’s a thought ex­per­i­ment. Omega offers you tick­ets for 2 ex­tra life­times of life, in ex­change for a 1% chance of dy­ing when you buy the ticket. You are forced to just keep buy­ing tick­ets un­til you fi­nally die.

Maybe you ob­ject that you dis­count ex­tra years of life by some func­tion, so just mod­ify the thought ex­per­i­ments so the re­ward in­crease fac­to­ri­ally per ticket bought, or some­thing like that.

For­tu­nately we don’t have to deal with these situ­a­tions much, be­cause we hap­pen to live in a uni­verse where there aren’t pow­er­ful agents offer­ing us very high util­ity lot­ter­ies. But these situ­a­tions oc­cur all the time if you deal with hy­pothe­ses in­stead of lot­ter­ies. The only rea­son we don’t no­tice it is be­cause we ig­nore or re­fuse to as­sign prob­a­bil­ity es­ti­mates to very un­likely hy­pothe­ses. An AI might not, and so it’s very im­por­tant to con­sider this is­sue.

I think money pumps ar­gue against sub­jec­tivity. Ba­si­cally, if you use an in­con­sis­tent de­ci­sion the­ory, some­one else can make money off your in­con­sis­tency or you don’t ac­tu­ally use that in­con­sis­tent de­ci­sion the­ory.

My method isn’t vuln­er­a­ble to money pumps, as is an in­finite num­ber of ar­bi­trary al­gorithms of the same class. See my com­ment here for de­tails.

You don’t even need the stuff I wrote about pre­de­ter­min­ing ac­tions, that just min­i­mizes re­gret. Even a naive im­ple­men­ta­tion of ex­pected me­dian util­ity should not be money pumpable.

As­sum­ing a par­tic­u­lar method of as­sign­ing prior prob­a­bil­ities to state­ments, yes. But is that the right method of as­sign­ing prior prob­a­bil­ities to state­ments?

The method by which you as­sign prob­a­bil­ities should be un­re­lated to the method you as­sign util­ities to out­comes. That is, you can’t just say you don’t like the out­come EU gives you and so as­sign it a lower prob­a­bil­ity, that’s a hor­rible vi­o­la­tion of Bayesian prin­ci­ples.

I don’t know what the cor­rect method of as­sign­ing prob­a­bil­ities, but even if you dis­count com­plex hy­pothe­ses fac­to­ri­ally or some­thing, you still get the same prob­lem.

I cer­tainly think these sce­nar­ios have rea­son­able prior prob­a­bil­ity. God could ex­ist, we could be in the ma­trix, etc. I give them so low prob­a­bil­ity I don’t typ­i­cally think about them, but for this is­sue that is ir­rele­vant.

• It’s just a tau­tol­ogy, “max­i­miz­ing ex­pected util­ity max­i­mizes ex­pected util­ity”.

Yes. That’s the thing that sounds silly but is ac­tu­ally deep.

Here’s a thought ex­per­i­ment. Omega offers you tick­ets for 2 ex­tra life­times of life, in ex­change for a 1% chance of dy­ing when you buy the ticket. You are forced to just keep buy­ing tick­ets un­til you fi­nally die.

Maybe you ob­ject that you dis­count ex­tra years of life by some func­tion, so just mod­ify the thought ex­per­i­ments so the re­ward in­crease fac­to­ri­ally per ticket bought, or some­thing like that.

That is the ob­jec­tion, but I think I should ex­plain it in a more fun­da­men­tal way.

What is the util­ity of a con­se­quence? For sim­plic­ity, we of­ten ex­press it as a real num­ber, with the caveat that all util­ities in­volved in a prob­lem have their re­la­tion­ships pre­served by an af­fine trans­for­ma­tion. But that num­ber is grounded by a gam­ble. Speci­fi­cally, con­sider three con­se­quences, A, B, and C, with u(A)<u(B)<u(C). If I am in­differ­ent be­tween B for cer­tain and A with prob­a­bil­ity p and C oth­er­wise, I en­code that with the math­e­mat­i­cal re­la­tion­ship:

`````` u(B)=p u(A)+(1-p) u(C)
``````

As I ex­press more and more prefer­ences, each num­ber is grounded by more and more con­straints.

The place where coun­terex­am­ples to EU calcu­la­tions go off the rails is when peo­ple in­ter­vene at the in­ter­me­di­ate step. Sup­pose p is 50%, and I’ve as­signed 0 to A, 1 to B, and 2 to C. If a new con­se­quence, D, is in­tro­duced with a util­ity of 4, that im­me­di­ately im­plies:

1. I am in­differ­ent be­tween (50% A, 50% D) and (100% C).

2. I am in­differ­ent be­tween (75% A, 25% D) and (100% B).

3. I am in­differ­ent be­tween (67% B, 33% D) and (100% C).

If one of those three state­ments is not true, I can use that and D hav­ing a util­ity of 4 to prove a con­tra­dic­tion. But the while the ex­is­tence of D and my will­ing­ness to ac­cept those spe­cific gam­bles im­plies that D’s util­ity is 4, the ex­is­tence of the num­ber 4 does not im­ply that there ex­ists a con­se­quence where I’m in­differ­ent to those gam­bles!

And so very quickly Omega might have to offer me a life­time longer than the life­time of the uni­verse, and be­cause I don’t be­lieve that’s pos­si­ble I say “no thanks, I don’t think you can de­liver, and in the odd case where you can de­liver, I’m not sure that I want what you can de­liver.” (This is the re­s­olu­tion of the St. Peters­burg Para­dox where you en­force that the house can­not pay you more than the to­tal wealth of the Earth, in which case the ex­pected value of the bet comes out to a rea­son­able, low num­ber of dol­lars, roughly where peo­ple es­ti­mate the value of the bet.)

But these situ­a­tions oc­cur all the time if you deal with hy­pothe­ses in­stead of lot­ter­ies. The only rea­son we don’t no­tice it is be­cause we ig­nore or re­fuse to as­sign prob­a­bil­ity es­ti­mates to very un­likely hy­pothe­ses. An AI might not, and so it’s very im­por­tant to con­sider this is­sue.

To me, this maps on to ba­sic re­search. There’s some low prob­a­bil­ity that a par­tic­u­lar molecule cures a par­tic­u­lar va­ri­ety of can­cer, but it would be great if it did—so let’s check. The im­por­tant con­cep­tual ad­di­tion from this anal­ogy is that both hy­pothe­ses are en­tan­gled (this molecule cur­ing can­cer im­plies things about other molecules) and val­ues are en­tan­gled (the sec­ond drug that can cure a par­tic­u­lar va­ri­ety of can­cer is less valuable than the first drug that can).

And so an AI might have some­what differ­ent ba­sic re­search pri­ori­ties than we do—in­deed, hu­mans vary widely on their prefer­ences for fol­low­ing var­i­ous un­cer­tain paths—but it seems likely to me that it could be­have rea­son­ably when com­ing up with a port­fo­lio of ac­tions to take, even if it looks like it might be­have oddly with only one op­tion.

The method by which you as­sign prob­a­bil­ities should be un­re­lated to the method you as­sign util­ities to out­comes. That is, you can’t just say you don’t like the out­come EU gives you and so as­sign it a lower prob­a­bil­ity, that’s a hor­rible vi­o­la­tion of Bayesian prin­ci­ples.

Play­ing against na­ture, sure—but play­ing against an in­tel­li­gent agent? Run­ning a min­i­max calcu­la­tion to figure out that my op­po­nent is not likely to let me check­mate him in one move is hardly a hor­rible vi­o­la­tion of Bayesian prin­ci­ples!

• And so very quickly Omega might have to offer me a life­time longer than the life­time of the uni­verse, and be­cause I don’t be­lieve that’s pos­si­ble I say “no thanks, I don’t think you can de­liver, and in the odd case where you can de­liver, I’m not sure that I want what you can de­liver.”

the house can­not pay you more than the to­tal wealth of the Earth, in which case the ex­pected value of the bet comes out to a rea­son­able, low num­ber of dol­lars, roughly where peo­ple es­ti­mate the value of the bet.

This is a cop out. Ob­vi­ously that spe­cific situ­a­tion can’t oc­cur in re­al­ity, that’s not the point. If your de­ci­sion al­gorithm fails in some ex­treme cases, at least con­fess that it’s not uni­ver­sal.

And so very quickly Omega might have to offer me a life­time longer than the life­time of the uni­verse, and be­cause I don’t be­lieve that’s pos­si­ble I say “no thanks, I don’t think you can deliver

Same thing. Omega’s abil­ity and hon­esty are premises.

The point of the thought ex­per­i­ment is just to show that EU is re­quired to trade away huge amounts of out­come-space for re­ally good but im­prob­a­ble out­comes. This is a good strat­egy if you plan on mak­ing an in­finite num­ber of bets, but hor­rible if you don’t ex­pect to live for­ever.

I don’t get your drug re­search anal­ogy. There is no pas­cal’s equiv­a­lent situ­a­tion in drug re­search. At best you find a molecule that cures all dis­eases, but that’s hardly in­finite util­ity.

In­stead it would be more like, “there is a tiny, tiny prob­a­bil­ity that a virus could emerge which causes hu­mans not to die, but to suffer for eter­nity in the worst pain pos­si­ble. There­fore, by EU calcu­la­tion, I should spend all of my re­sources search­ing for a pos­si­ble vac­cine for this spe­cific dis­ease, and noth­ing else.”

• This is a cop out. Ob­vi­ously that spe­cific situ­a­tion can’t oc­cur in re­al­ity, that’s not the point. If your de­ci­sion al­gorithm fails in some ex­treme cases, at least con­fess that it’s not uni­ver­sal.

What does it mean for a de­ci­sion al­gorithm to fail? I’ll give an an­swer later, but here I’ll point out that I do en­dorse that mul­ti­pli­ca­tion of re­als is uni­ver­sal—that is, I don’t think mul­ti­pli­ca­tion breaks down when the num­bers get ex­treme enough.

Same thing. Omega’s abil­ity and hon­esty are premises.

And an un­be­liev­able premise leads to an un­be­liev­able con­clu­sion. Don’t say that logic has bro­ken down be­cause some­one gave the syl­l­o­gism “All men are im­mor­tal, Socrates is a man, there­fore Socrates is still al­ive.”

The point of the thought ex­per­i­ment is just to show that EU is re­quired to trade away huge amounts of out­come-space for re­ally good but im­prob­a­ble out­comes. This is a good strat­egy if you plan on mak­ing an in­finite num­ber of bets, but hor­rible if you don’t ex­pect to live for­ever.

How does [logic work]? Eliezer puts it bet­ter than I can:

The power of logic is that it re­lates mod­els and state­ments. … And here is the power of logic: For each syn­tac­tic step we do on our state­ments, we pre­serve the match to any model. In any model where our old col­lec­tion of state­ments was true, the new state­ment will also be true. We don’t have to check all pos­si­ble con­form­ing mod­els to see if the new state­ment is true in all of them. We can trust cer­tain syn­tac­tic steps in gen­eral—not to pro­duce truth, but to pre­serve truth.

EU is not “re­quired” to trade away huge amounts of out­come-space for re­ally good but im­prob­a­ble out­comes. EU ap­plies prefer­ence mod­els to novel situ­a­tions, not to pro­duce prefer­ences but to pre­serve them. If you gave EU a prefer­ence model that matched your prefer­ences, it will pre­serve the match and give you ac­tions that best satisfy your prefer­ences in un­der­neath the un­cer­tainty model of the uni­verse you gave it.

And if it’s not true that you would trade away huge amounts of out­come-space for re­ally good but im­prob­a­ble out­comes, this is a fact about your prefer­ence model that EU pre­serves! Re­mem­ber, EU prefer­ence mod­els map lists of out­comes to classes of lists of real num­bers, but the in­verse map­ping is not guaran­teed to have sup­port over the en­tire re­als.

I think a de­ci­sion al­gorithm fails if it makes you pre­dictably worse off than an al­ter­na­tive al­gorithm, and the chief ways to do so are 1) to do the math wrong and be in­con­sis­tent and 2) to make it more costly to ex­press your prefer­ences or world-model.

I don’t get your drug re­search anal­ogy.

We have lots of hy­pothe­ses about low-prob­a­bil­ity, high-pay­out op­tions, and if hu­mans make mis­takes, it is prob­a­bly by over­es­ti­mat­ing the prob­a­bil­ity of low-prob­a­bil­ity events and over­es­ti­mat­ing how much we’ll en­joy the high pay­outs, both of which make us more likely to pur­sue those paths than a ra­tio­nal ver­sion of our­selves.

So it seems to me that if we have an al­gorithm that can cor­rectly man­age the bud­get of a phar­ma­ceu­ti­cal cor­po­ra­tion, bal­anc­ing R&D and mar­ket­ing and pro­duc­tion and so on, that re­quires solv­ing this philo­soph­i­cal prob­lem. But we have the math­e­mat­i­cal tools to cor­rectly man­age the bud­get of a phar­ma­ceu­ti­cal cor­po­ra­tion given a world-model, which says to me we should turn our at­ten­tion to get­ting more pre­cise and pow­er­ful world-mod­els.

• What does it mean for a de­ci­sion al­gorithm to fail?

When it makes de­ci­sions that are un­de­sir­able. There is no point de­cid­ing to run a de­ci­sion al­gorithm which is perfectly con­sis­tent but re­sults in out­comes you don’t want.

In the case of the Omega’s-life-tick­ets sce­nario, one could ar­gue it fails in an ob­jec­tive sense since it will never stop buy­ing tick­ets un­til it dies. But that wasn’t even the point I was try­ing to make.

And an un­be­liev­able premise leads to an un­be­liev­able con­clu­sion.

I don’t know if there is a name for this fal­lacy but there should be. Where some­one ob­jects to the premises of a hy­po­thet­i­cal situ­a­tion in­tended just to demon­strate a point. E.g. peo­ple who re­fuse to an­swer the trol­ley dilemma and in­stead say “but that will prob­a­bly never hap­pen!” It’s very frus­trat­ing.

EU is not “re­quired” to trade away huge amounts of out­come-space for re­ally good but im­prob­a­ble out­comes. EU ap­plies prefer­ence mod­els to novel situ­a­tions, not to pro­duce prefer­ences but to pre­serve them. If you gave EU a prefer­ence model that matched your prefer­ences, it will pre­serve the match and give you ac­tions that best satisfy your prefer­ences in un­der­neath the un­cer­tainty model of the uni­verse you gave it.

This is very sub­tle cir­cu­lar rea­son­ing. If you as­sume your goal is to max­i­mize the ex­pected value some util­ity func­tion, then max­i­miz­ing ex­pected util­ity can do that if you spec­ify the right util­ity func­tion.

What I’ve been say­ing from the very be­gin­ning is that there isn’t any rea­son to be­lieve there is any util­ity func­tion that will pro­duce de­sir­able out­comes if fed to an ex­pected util­ity max­i­mizer.

I think a de­ci­sion al­gorithm fails if it makes you pre­dictably worse off than an al­ter­na­tive algorithm

Even if you are an EU max­i­mizer, EU will make you “pre­dictably” worse off, as in the ma­jor­ity of cases you will be worse off. A true EU max­i­mizer doesn’t care so long as the util­ity of the very low prob­a­bil­ity out­comes is high enough.

• I don’t know if there is a name for this fal­lacy but there should be.

One name is fight­ing the hy­po­thet­i­cal, and it’s worth tak­ing a look at the least con­ve­nient pos­si­ble world and the true re­jec­tion as well.

There are good and bad rea­sons to fight the hy­po­thet­i­cal. When it comes to these par­tic­u­lar prob­lems, though, the ob­jec­tions I’ve given are my true ob­jec­tions. The rea­son I’d only pay a tiny amount of money for the gam­ble in the St. Peters­burg Para­dox is that there is only so much fi­nan­cial value that the house can give up. One of the rea­sons I’m sure this is my true ob­jec­tion is be­cause the richer the house, the more I would pay for such a gam­ble. (Be­cause there are no in­finitely rich houses, there is no one I would pay an in­finite amount to for such a gam­ble.)

This is very sub­tle cir­cu­lar rea­son­ing. If you as­sume your goal is to max­i­mize the ex­pected value some util­ity func­tion, then max­i­miz­ing ex­pected util­ity can do that if you spec­ify the right util­ity func­tion.

I’m not sure why you think it’s sub­tle—I started off this con­ver­sa­tion with:

This might sound silly, but it’s deeper than it looks: the rea­son why we use the ex­pected value of util­ity (i.e. means) to de­ter­mine the best of a set of gam­bles is be­cause util­ity is defined as the thing that you max­i­mize the ex­pected value of.

But I don’t think it’s quite right to call it “cir­cu­lar,” for roughly the same rea­sons I don’t think it’s right to call logic “cir­cu­lar.”

What I’ve been say­ing from the very be­gin­ning is that there isn’t any rea­son to be­lieve there is any util­ity func­tion that will pro­duce de­sir­able out­comes if fed to an ex­pected util­ity max­i­mizer.

To make sure we’re talk­ing about the same thing, I think an ex­pected util­ity max­i­mizer (EUM) is some­thing that takes both a func­tion u(O) that maps out­comes to util­ities, a func­tion p(A->O) that maps ac­tions to prob­a­bil­ities of out­comes, and a set of pos­si­ble ac­tions, and then finds the ac­tion out of all pos­si­ble A that has the max­i­mum weighted sum of u(O)p(A->O) over all pos­si­ble O.

So far, you have not been ar­gu­ing that ev­ery pos­si­ble EUM leads to patholog­i­cal out­comes; you have been ex­hibit­ing par­tic­u­lar com­bi­na­tions of u(O) and p(A->O) that lead to patholog­i­cal out­comes, and I have been re­spond­ing with “have you tried not us­ing those u(O)s and p(A->O)s?”.

It doesn’t seem to me that this con­ver­sa­tion is pro­duc­ing value for ei­ther of us, which sug­gests that we should ei­ther restart the con­ver­sa­tion, take it to PMs, or drop it.

• Here’s a thought ex­per­i­ment. Omega offers you tick­ets for 2 ex­tra life­times of life, in ex­change for a 1% chance of dy­ing when you buy the ticket. You are forced to just keep buy­ing tick­ets un­til you fi­nally die.

This sug­gests buy­ing tick­ets takes finite time per ticket, and that the offer is per­pet­u­ally open. It seems like you could get a solid win out of this by liv­ing your life, buy­ing one ticket ev­ery time you start run­ning out of life. You keep as much of your prob­a­bil­ity mass al­ive as pos­si­ble for as long as pos­si­ble, and your prob­a­bil­ity of be­ing al­ive at any given time af­ter the end of the first “life­time” is greater than it would’ve been if you hadn’t bought tick­ets. Yeah, Omega has to fol­low you around while you go about your busi­ness, but that’s no more ob­nox­ious than say­ing you have to stand next to Omega wast­ing decades on mash­ing the ticket-buy­ing but­ton.

• Ok change it so the ticket booth closes if you leave.

• Ex­pected util­ity is perfectly ra­tio­nal as the num­ber of “bets” you take goes to in­finity.

That’s not the way in which max­i­miz­ing ex­pected util­ity is perfectly ra­tio­nal.

The way it’s perfectly ra­tio­nal is this. Sup­pose you have any de­ci­sion mak­ing al­gorithm; if you like, it can have an in­ter­nal vari­able called “util­ity” that lets it or­der and com­pare differ­ent out­comes based on how de­sir­able they are. Then ei­ther:

• the al­gorithm has some ugly be­hav­ior with re­spect to a finite col­lec­tion of bets (for in­stance, there are three bets A, B, and C such that it prefers A to B, B to C, and C to A), or

• the al­gorithm is equiv­a­lent to one which max­i­mizes the ex­pected value of some util­ity func­tion: maybe the one that your in­ter­nal vari­able was mea­sur­ing, maybe not.

• The first con­di­tion is not true, since it gives a con­sis­tent value to any prob­a­bil­ity dis­tri­bu­tion of util­ities. The sec­ond con­di­tion is not true other since the me­dian func­tion is not merely a trans­form of the mean func­tion.

I’m not sure what the “ugly” be­hav­ior you de­scribe is, and I bet it rests on some as­sump­tion that’s too strong. I already men­tioned how in­con­sis­tent be­hav­ior can be fixed by al­low­ing it to pre­de­ter­mine it’s ac­tions.

• You can find the Von Neu­mann—Mor­gen­stern ax­ioms for your­self. It’s hard to say whether or not they’re too strong.

The prob­lem with “al­low­ing [the me­dian al­gorithm] to pre­de­ter­mine its ac­tions” is that in this case, I no longer know what the al­gorithm out­puts in any given case. Maybe we can re­solve this by con­sid­er­ing a case when the me­dian al­gorithm fails, and you can ex­plain what your mod­ifi­ca­tion does to fix it. Here’s an ex­am­ple.

Sup­pose I roll a sin­gle die.

• Bet A loses you \$5 on a roll of 1 or 2, but wins you \$1 on a roll of 3, 4, 5, or 6.

• Bet B loses you \$5 on a roll of 5 or 6, but wins you \$1 on a roll of 1, 2, 3, or 4.

Bet A has me­dian util­ity of U(\$1), as does bet B. How­ever, com­bined they have a me­dian util­ity of U(-\$4).

So the straight­for­ward me­dian al­gorithm pays money to buy Bet A, pays money to buy Bet B, but will then pay money to be rid of their com­bi­na­tion.

• I think I’ve found the core of our dis­agree­ment. I want an al­gorithm that con­sid­ers all pos­si­ble paths through time. It de­cides on a set of ac­tions, not just for the cur­rent time step, but for all pos­si­ble fu­ture time steps. It chooses such that the fi­nal prob­a­bil­ity dis­tri­bu­tion of pos­si­ble out­comes, at some point in the fu­ture, is op­ti­mal ac­cord­ing to some met­ric. I origi­nally thought of me­dian, but it can work with any ar­bi­trary met­ric.

This is a gen­er­al­iza­tion of ex­pected util­ity. The VNM ax­ioms re­quire an al­gorithm to make de­ci­sions in­de­pen­dently and Just In Time. Whereas this method lets it con­sider all pos­si­ble out­comes. It may be less el­e­gant than EU, but I think it’s closer to what hu­mans ac­tu­ally want.

Any­way your ex­am­ple is wrong, even with­out pre­de­ter­mined ac­tions. The al­gorithm would buy bet A, but then not buy bet B. This is be­cause it doesn’t con­sider bets in iso­la­tion like EU, but con­sid­ers it’s en­tire prob­a­bil­ity dis­tri­bu­tion of pos­si­ble out­comes. Buy­ing bet B would de­crease it’s ex­pected me­dian util­ity, so it wouldn’t take it.

• The VNM ax­ioms re­quire an al­gorithm to make de­ci­sions in­de­pen­dently and Just In Time.

No, they don’t.

• As­sum­ing the bet has a fixed util­ity, then EU gives it a fixed es­ti­mate right away. Whereas my method con­sid­ers it along with all other bets that it’s made or ex­pects to make, and it’s es­ti­mate can change over time. I should have said that it’s not in­de­pen­dent or fixed, but that is what I meant.

• In the VNM scheme where ex­pected util­ity is de­rived at a con­se­quence of the ax­ioms, the way that a bet’s util­ity changes over time is that its util­ity is not fixed. Noth­ing at all stops you from chang­ing the util­ity you at­tach to a 50:50 gam­ble of get­ting a kit­ten ver­sus \$5 if your util­ity for a kit­ten (or for \$5) changes: for ex­am­ple, if you get an­other kit­ten or win the lot­tery.

Gen­er­al­iz­ing to al­low the value of the bet to change when the value of the op­tions did not change seems strange to me.

• This is a gen­er­al­iza­tion of ex­pected util­ity.

I am lost, this is just EU in a lon­gi­tu­di­nal set­ting? You can av­er­age over lots of stuff. Max­i­miz­ing EU is bor­ing, it’s spec­i­fy­ing the right dis­tri­bu­tion that’s tricky.

• It’s not EU, since it can im­ple­ment ar­bi­trary al­gorithms to spec­ify the de­sired prob­a­bil­ity dis­tri­bu­tion of out­comes. Aver­ag­ing util­ity is only one pos­si­bil­ity, an­other I men­tioned was me­dian util­ity.

So you would take the me­dian util­ity of all the pos­si­ble out­comes. And then se­lect the ac­tion (or se­ries of ac­tions in this case) that leads to the high­est me­dian util­ity.

No method of spec­i­fy­ing util­ities would let EU do the same thing, but you can triv­ially im­ple­ment EU in it, so it’s strictly more gen­eral than EU.

• I think I’ve found the core of our dis­agree­ment. I want an al­gorithm that con­sid­ers all pos­si­ble paths through time. It de­cides on a set of ac­tions, not just for the cur­rent time step, but for all pos­si­ble fu­ture time steps.

So, I think you might be in­ter­ested in UDT. (I’m not sure what the cur­rent best refer­ence for that is.) I think that this re­quires ac­tual om­ni­science, and so is not a good place to look for de­ci­sion al­gorithms.

(Though I should add that typ­i­cally util­ities are defined over world-his­to­ries, and so any de­ci­sion al­gorithm typ­i­cally iden­ti­fies classes of ‘equiv­a­lent’ ac­tions, i.e. ac­knowl­edges that this is a thing that needs to be ac­cepted some­how.)

• UDT is overkill. The idea that all fu­ture choices can be col­lapsed into a sin­gle choice ap­pears in the work of von Neu­mann and Mor­gen­stern, but is prob­a­bly much older.

• Oh, I see. I didn’t take that prob­lem into ac­count, be­cause it doesn’t mat­ter for ex­pected util­ity, which is ad­di­tive. But you’re right that con­sid­er­ing the en­tire prob­a­bil­ity dis­tri­bu­tion is the right thing to do, and un­der than as­sump­tion we’re forced to be tran­si­tive.

The ac­tual VNM ax­iom vi­o­lated by me­dian util­ity is in­de­pen­dence: If you pre­fer X to Y, then a gam­ble of X vs Z is prefer­able to the equiv­a­lent gam­ble of Y vs Z. Con­sider the fol­low­ing two com­par­i­sons:

• Tak­ing bet A, as above, ver­sus the sta­tus quo.

• A 23 chance of tak­ing bet A and a 13 chance of los­ing \$5, ver­sus a 23 chance of the sta­tus quo and a 13 chance of los­ing \$5.

In the first case, bet A has me­dian util­ity U(\$1) and the sta­tus quo has U(\$0), so you pick bet A. In the sec­ond case, a gam­ble with a pos­si­bil­ity of bet A has me­dian util­ity U(-\$5) and a gam­ble with a pos­si­bil­ity of the sta­tus quo still has U(\$0), so you pick the sec­ond gam­ble.

Of course, in­de­pen­dence is prob­a­bly the shak­iest of the VNM ax­ioms, and it wouldn’t sur­prise me if you’re un­con­vinced by it.