Decision Theory FAQ

Co-au­thored with crazy88. Please let us know when you find mis­takes, and we’ll fix them. Last up­dated 03-27-2013.

Con­tents:

1. What is de­ci­sion the­ory?

De­ci­sion the­ory, also known as ra­tio­nal choice the­ory, con­cerns the study of prefer­ences, un­cer­tain­ties, and other is­sues re­lated to mak­ing “op­ti­mal” or “ra­tio­nal” choices. It has been dis­cussed by economists, psy­chol­o­gists, philoso­phers, math­e­mat­i­ci­ans, statis­ti­ci­ans, and com­puter sci­en­tists.

We can di­vide de­ci­sion the­ory into three parts (Grant & Zandt 2009; Baron 2008). Nor­ma­tive de­ci­sion the­ory stud­ies what an ideal agent (a perfectly ra­tio­nal agent, with in­finite com­put­ing power, etc.) would choose. De­scrip­tive de­ci­sion the­ory stud­ies how non-ideal agents (e.g. hu­mans) ac­tu­ally choose. Pre­scrip­tive de­ci­sion the­ory stud­ies how non-ideal agents can im­prove their de­ci­sion-mak­ing (rel­a­tive to the nor­ma­tive model) de­spite their im­perfec­tions.

For ex­am­ple, one’s nor­ma­tive model might be ex­pected util­ity the­ory, which says that a ra­tio­nal agent chooses the ac­tion with the high­est ex­pected util­ity. Repli­cated re­sults in psy­chol­ogy de­scribe hu­mans re­peat­edly failing to max­i­mize ex­pected util­ity in par­tic­u­lar, pre­dictable ways: for ex­am­ple, they make some choices based not on po­ten­tial fu­ture benefits but on ir­rele­vant past efforts (the “sunk cost fal­lacy”). To help peo­ple avoid this er­ror, some the­o­rists pre­scribe some ba­sic train­ing in microe­co­nomics, which has been shown to re­duce the like­li­hood that hu­mans will com­mit the sunk costs fal­lacy (Lar­rick et al. 1990). Thus, through a co­or­di­na­tion of nor­ma­tive, de­scrip­tive, and pre­scrip­tive re­search we can help agents to suc­ceed in life by act­ing more in ac­cor­dance with the nor­ma­tive model than they oth­er­wise would.

This FAQ fo­cuses on nor­ma­tive de­ci­sion the­ory. Good sources on de­scrip­tive and pre­scrip­tive de­ci­sion the­ory in­clude Stanovich (2010) and Hastie & Dawes (2009).

Two re­lated fields be­yond the scope of this FAQ are game the­ory and so­cial choice the­ory. Game the­ory is the study of con­flict and co­op­er­a­tion among mul­ti­ple de­ci­sion mak­ers, and is thus some­times called “in­ter­ac­tive de­ci­sion the­ory.” So­cial choice the­ory is the study of mak­ing a col­lec­tive de­ci­sion by com­bin­ing the prefer­ences of mul­ti­ple de­ci­sion mak­ers in var­i­ous ways.

This FAQ draws heav­ily from two text­books on de­ci­sion the­ory: Res­nik (1987) and Peter­son (2009). It also draws from more re­cent re­sults in de­ci­sion the­ory, pub­lished in jour­nals such as Syn­these and The­ory and De­ci­sion.

2. Is the ra­tio­nal de­ci­sion always the right de­ci­sion?

No. Peter­son (2009, ch. 1) ex­plains:

[In 1700], King Carl of Swe­den and his 8,000 troops at­tacked the Rus­sian army [which] had about ten times as many troops… Most his­to­ri­ans agree that the Swedish at­tack was ir­ra­tional, since it was al­most cer­tain to fail… How­ever, be­cause of an un­ex­pected bliz­zard that blinded the Rus­sian army, the Swedes won...

Look­ing back, the Swedes’ de­ci­sion to at­tack the Rus­sian army was no doubt right, since the ac­tual out­come turned out to be suc­cess. How­ever, since the Swedes had no good rea­son for ex­pect­ing that they were go­ing to win, the de­ci­sion was nev­er­the­less ir­ra­tional.

More gen­er­ally speak­ing, we say that a de­ci­sion is right if and only if its ac­tual out­come is at least as good as that of ev­ery other pos­si­ble out­come. Fur­ther­more, we say that a de­ci­sion is ra­tio­nal if and only if the de­ci­sion maker [aka the “agent”] chooses to do what she has most rea­son to do at the point in time at which the de­ci­sion is made.

Un­for­tu­nately, we can­not know with cer­tainty what the right de­ci­sion is. Thus, the best we can do is to try to make “ra­tio­nal” or “op­ti­mal” de­ci­sions based on our prefer­ences and in­com­plete in­for­ma­tion.

3. How can I bet­ter un­der­stand a de­ci­sion prob­lem?

First, we must for­mal­ize a de­ci­sion prob­lem. It usu­ally helps to vi­su­al­ize the de­ci­sion prob­lem, too.

In de­ci­sion the­ory, de­ci­sion rules are only defined rel­a­tive to a for­mal­iza­tion of a given de­ci­sion prob­lem, and a for­mal­iza­tion of a de­ci­sion prob­lem can be vi­su­al­ized in mul­ti­ple ways. Here is an ex­am­ple from Peter­son (2009, ch. 2):

Sup­pose… that you are think­ing about tak­ing out fire in­surance on your home. Per­haps it costs $100 to take out in­surance on a house worth $100,000, and you ask: Is it worth it?

The most com­mon way to for­mal­ize a de­ci­sion prob­lem is to break it into states, acts, and out­comes. When fac­ing a de­ci­sion prob­lem, the de­ci­sion maker aims to choose the act that will have the best out­come. But the out­come of each act de­pends on the state of the world, which is un­known to the de­ci­sion maker.

In this frame­work, speak­ing loosely, a state is a part of the world that is not an act (that can be performed now by the de­ci­sion maker) or an out­come (the ques­tion of what, more pre­cisely, states are is a com­plex ques­tion that is be­yond the scope of this doc­u­ment). Luck­ily, not all states are rele­vant to a par­tic­u­lar de­ci­sion prob­lem. We only need to take into ac­count states that af­fect the agent’s prefer­ence among acts. A sim­ple for­mal­iza­tion of the fire in­surance prob­lem might in­clude only two states: the state in which your house doesn’t (later) catch on fire, and the state in which your house does (later) catch on fire.

Pre­sum­ably, the agent prefers some out­comes to oth­ers. Sup­pose the four con­ceiv­able out­comes in the above de­ci­sion prob­lem are: (1) House and $0, (2) House and -$100, (3) No house and $99,900, and (4) No house and $0. In this case, the de­ci­sion maker might pre­fer out­come 1 over out­come 2, out­come 2 over out­come 3, and out­come 3 over out­come 4. (We’ll dis­cuss mea­sures of value for out­comes in the next sec­tion.)

An act is com­monly taken to be a func­tion that takes one set of the pos­si­ble states of the world as in­put and gives a par­tic­u­lar out­come as out­put. For the above de­ci­sion prob­lem we could say that if the act “Take out in­surance” has the world-state “Fire” as its in­put, then it will give the out­come “No house and $99,900” as its out­put.

An outline of the states, acts and outcomes in the insurance case

An out­line of the states, acts and out­comes in the in­surance case

Note that de­ci­sion the­ory is con­cerned with par­tic­u­lar acts rather than generic acts, e.g. “sailing west in 1492” rather than “sailing.” More­over, the acts of a de­ci­sion prob­lem must be al­ter­na­tive acts, so that the de­ci­sion maker has to choose ex­actly one act.

Once a de­ci­sion prob­lem has been for­mal­ized, it can then be vi­su­al­ized in any of sev­eral ways.

One way to vi­su­al­ize this de­ci­sion prob­lem is to use a de­ci­sion ma­trix:

Fire No fire
Take out in­surance No house and $99,900 House and -$100
No in­surance No house and $0 House and $0

Another way to vi­su­al­ize this prob­lem is to use a de­ci­sion tree:

The square is a choice node, the cir­cles are chance nodes, and the tri­an­gles are ter­mi­nal nodes. At the choice node, the de­ci­sion maker chooses which branch of the de­ci­sion tree to take. At the chance nodes, na­ture de­cides which branch to fol­low. The tri­an­gles rep­re­sent out­comes.

Of course, we could add more branches to each choice node and each chance node. We could also add more choice nodes, in which case we are rep­re­sent­ing a se­quen­tial de­ci­sion prob­lem. Fi­nally, we could add prob­a­bil­ities to each branch, as long as the prob­a­bil­ities of all the branches ex­tend­ing from each sin­gle node sum to 1. And be­cause a de­ci­sion tree obeys the laws of prob­a­bil­ity the­ory, we can calcu­late the prob­a­bil­ity of any given node by mul­ti­ply­ing the prob­a­bil­ities of all the branches pre­ced­ing it.

Our de­ci­sion prob­lem could also be rep­re­sented as a vec­tor — an or­dered list of math­e­mat­i­cal ob­jects that is per­haps most suit­able for com­put­ers:

[
[a1 = take out in­surance,
a2 = do not];
[s1 = fire,
s2 = no fire];
[(a1, s1) = No house and $99,900,
(a1, s2) = House and -$100,
(a2, s1) = No house and $0,
(a2, s2) = House and $0]
]

For more de­tails on for­mal­iz­ing and vi­su­al­iz­ing de­ci­sion prob­lems, see Sk­in­ner (1993).

4. How can I mea­sure an agent’s prefer­ences?

4.1. The con­cept of utility

It is im­por­tant not to mea­sure an agent’s prefer­ences in terms of ob­jec­tive value, e.g. mon­e­tary value. To see why, con­sider the ab­sur­di­ties that can re­sult when we try to mea­sure an agent’s prefer­ence with money alone.

Sup­pose you may choose be­tween (A) re­ceiv­ing a mil­lion dol­lars for sure, and (B) a 50% chance of win­ning ei­ther $3 mil­lion or noth­ing. The ex­pected mon­e­tary value (EMV) of your act is com­puted by mul­ti­ply­ing the mon­e­tary value of each pos­si­ble out­come by its prob­a­bil­ity. So, the EMV of choice A is (1)($1 mil­lion) = $1 mil­lion. The EMV of choice B is (0.5)($3 mil­lion) + (0.5)($0) = $1.5 mil­lion. Choice B has a higher ex­pected mon­e­tary value, and yet many peo­ple would pre­fer the guaran­teed mil­lion.

Why? For many peo­ple, the differ­ence be­tween hav­ing $0 and $1 mil­lion is sub­jec­tively much larger than the differ­ence be­tween hav­ing $1 mil­lion and $3 mil­lion, even if the lat­ter differ­ence is larger in dol­lars.

To cap­ture an agent’s sub­jec­tive prefer­ences, we use the con­cept of util­ity. A util­ity func­tion as­signs num­bers to out­comes such that out­comes with higher num­bers are preferred to out­comes with lower num­bers. For ex­am­ple, for a par­tic­u­lar de­ci­sion maker — say, one who has no money — the util­ity of $0 might be 0, the util­ity of $1 mil­lion might be 1000, and the util­ity of $3 mil­lion might be 1500. Thus, the ex­pected util­ity (EU) of choice A is, for this de­ci­sion maker, (1)(1000) = 1000. Mean­while, the EU of choice B is (0.5)(1500) + (0.5)(0) = 750. In this case, the ex­pected util­ity of choice A is greater than that of choice B, even though choice B has a greater ex­pected mon­e­tary value.

Note that those from the field of statis­tics who work on de­ci­sion the­ory tend to talk about a “loss func­tion,” which is sim­ply an in­verse util­ity func­tion. For an overview of de­ci­sion the­ory from this per­spec­tive, see Berger (1985) and Robert (2001). For a cri­tique of some stan­dard re­sults in statis­ti­cal de­ci­sion the­ory, see Jaynes (2003, ch. 13).

4.2. Types of utility

An agent’s util­ity func­tion can’t be di­rectly ob­served, so it must be con­structed — e.g. by ask­ing them which op­tions they pre­fer for a large set of pairs of al­ter­na­tives (as on WhoIsHot­ter.com). The num­ber that cor­re­sponds to an out­come’s util­ity can con­vey differ­ent in­for­ma­tion de­pend­ing on the util­ity scale in use, and the util­ity scale in use de­pends on how the util­ity func­tion is con­structed.

De­ci­sion the­o­rists dis­t­in­guish three kinds of util­ity scales:

  1. Or­di­nal scales (“12 is bet­ter than 6”). In an or­di­nal scale, preferred out­comes are as­signed higher num­bers, but the num­bers don’t tell us any­thing about the differ­ences or ra­tios be­tween the util­ity of differ­ent out­comes.

  2. In­ter­val scales (“the differ­ence be­tween 12 and 6 equals that be­tween 6 and 0”). An in­ter­val scale gives us more in­for­ma­tion than an or­di­nal scale. Not only are preferred out­comes as­signed higher num­bers, but also the num­bers ac­cu­rately re­flect the differ­ence be­tween the util­ity of differ­ent out­comes. They do not, how­ever, nec­es­sar­ily re­flect the ra­tios of util­ity be­tween differ­ent out­comes. If out­come A has util­ity 0, out­come B has util­ity 6, and out­come C has util­ity 12 on an in­ter­val scale, then we know that the differ­ence in util­ity be­tween out­comes A and B and be­tween out­comes B and C is the same, but we can’t know whether out­come B is “twice as good” as out­come A.

  3. Ra­tio scales (“12 is ex­actly twice as valuable as 6”). Numer­i­cal util­ity as­sign­ments on a ra­tio scale give us the most in­for­ma­tion of all. They ac­cu­rately re­flect prefer­ence rank­ings, differ­ences, and ra­tios. Thus, we can say that an out­come with util­ity 12 is ex­actly twice as valuable to the agent in ques­tion as an out­come with util­ity 6.

Note that nei­ther ex­pe­rienced util­ity (hap­piness) nor the no­tions of “av­er­age util­ity” or “to­tal util­ity” dis­cussed by util­i­tar­ian moral philoso­phers are the same thing as the de­ci­sion util­ity that we are dis­cussing now to de­scribe de­ci­sion prefer­ences. As the situ­a­tion mer­its, we can be even more spe­cific. For ex­am­ple, when dis­cussing the type of de­ci­sion util­ity used in an in­ter­val scale util­ity func­tion con­structed us­ing Von Neu­mann & Mor­gen­stern’s ax­io­matic ap­proach (see sec­tion 8), some peo­ple use the term VNM-util­ity.

Now that you know that an agent’s prefer­ences can be rep­re­sented as a “util­ity func­tion,” and that as­sign­ments of util­ity to out­comes can mean differ­ent things de­pend­ing on the util­ity scale of the util­ity func­tion, we are ready to think more for­mally about the challenge of mak­ing “op­ti­mal” or “ra­tio­nal” choices. (We will re­turn to the prob­lem of con­struct­ing an agent’s util­ity func­tion later, in sec­tion 8.3.)

5. What do de­ci­sion the­o­rists mean by “risk,” “ig­no­rance,” and “un­cer­tainty”?

Peter­son (2009, ch. 1) ex­plains:

In de­ci­sion the­ory, ev­ery­day terms such as risk, ig­no­rance, and un­cer­tainty are used as tech­ni­cal terms with pre­cise mean­ings. In de­ci­sions un­der risk the de­ci­sion maker knows the prob­a­bil­ity of the pos­si­ble out­comes, whereas in de­ci­sions un­der ig­no­rance the prob­a­bil­ities are ei­ther un­known or non-ex­is­tent. Uncer­tainty is ei­ther used as a syn­onym for ig­no­rance, or as a broader term refer­ring to both risk and ig­no­rance.

In this FAQ, a “de­ci­sion un­der ig­no­rance” is one in which prob­a­bil­ities are not as­signed to all out­comes, and a “de­ci­sion un­der un­cer­tainty” is one in which prob­a­bil­ities are as­signed to all out­comes. The term “risk” will be re­served for dis­cus­sions re­lated to util­ity.

6. How should I make de­ci­sions un­der ig­no­rance?

A de­ci­sion maker faces a “de­ci­sion un­der ig­no­rance” when she (1) knows which acts she could choose and which out­comes they may re­sult in, but (2) is un­able to as­sign prob­a­bil­ities to the out­comes.

(Note that many the­o­rists think that all de­ci­sions un­der ig­no­rance can be trans­formed into de­ci­sions un­der un­cer­tainty, in which case this sec­tion will be ir­rele­vant ex­cept for sub­sec­tion 6.1. For de­tails, see sec­tion 7.)

6.1. The dom­i­nance principle

To bor­row an ex­am­ple from Peter­son (2009, ch. 3), sup­pose that Jane isn’t sure whether to or­der ham­burger or monk­fish at a new restau­rant. Just about any chef can make an ed­ible ham­burger, and she knows that monk­fish is fan­tas­tic if pre­pared by a world-class chef, but she also re­calls that monk­fish is difficult to cook. Un­for­tu­nately, she knows too lit­tle about this restau­rant to as­sign any prob­a­bil­ity to the prospect of get­ting good monk­fish. Her de­ci­sion ma­trix might look like this:

Good chef Bad chef
Monk­fish good monk­fish ter­rible monk­fish
Ham­burger ed­ible ham­burger ed­ible ham­burger
No main course hun­gry hun­gry

Here, de­ci­sion the­o­rists would say that the “ham­burger” choice dom­i­nates the “no main course” choice. This is be­cause choos­ing the ham­burger leads to a bet­ter out­come for Jane no mat­ter which pos­si­ble state of the world (good chef or bad chef) turns out to be true.

This dom­i­nance prin­ci­ple comes in two forms:

  • Weak dom­i­nance: One act is more ra­tio­nal than an­other if (1) all its pos­si­ble out­comes are at least as good as those of the other, and if (2) there is at least one pos­si­ble out­come that is bet­ter than that of the other act.

  • Strong dom­i­nance: One act is more ra­tio­nal than an­other if all of its pos­si­ble out­come are bet­ter than that of the other act.

A comparison of strong and weak dominance

A com­par­i­son of strong and weak dominance

The dom­i­nance prin­ci­ple can also be ap­plied to de­ci­sions un­der un­cer­tainty (in which prob­a­bil­ities are as­signed to all the out­comes). If we as­sign prob­a­bil­ities to out­comes, it is still ra­tio­nal to choose one act over an­other act if all its out­comes are at least as good as the out­comes of the other act.

How­ever, the dom­i­nance prin­ci­ple only ap­plies (non-con­tro­ver­sially) when the agent’s acts are in­de­pen­dent of the state of the world. So con­sider the de­ci­sion of whether to steal a coat:

Charged with theft Not charged with theft
Theft Jail and coat Free­dom and coat
No theft Jail Free­dom

In this case, steal­ing the coat dom­i­nates not do­ing so but isn’t nec­es­sar­ily the ra­tio­nal de­ci­sion. After all, steal­ing in­creases your chance of get­ting charged with theft and might be ir­ra­tional for this rea­son. So dom­i­nance doesn’t ap­ply in cases like this where the state of the world is not in­de­pen­dent of the agents act.

On top of this, not all de­ci­sion prob­lems in­clude an act that dom­i­nates all the oth­ers. Con­se­quently ad­di­tional prin­ci­ples are of­ten re­quired to reach a de­ci­sion.

6.2. Max­imin and leximin

Some de­ci­sion the­o­rists have sug­gested the max­imin prin­ci­ple: if the worst pos­si­ble out­come of one act is bet­ter than the worst pos­si­ble out­come of an­other act, then the former act should be cho­sen. In Jane’s de­ci­sion prob­lem above, the max­imin prin­ci­ple would pre­scribe choos­ing the ham­burger, be­cause the worst pos­si­ble out­come of choos­ing the ham­burger (“ed­ible ham­burger”) is bet­ter than the worst pos­si­ble out­come of choos­ing the monk­fish (“ter­rible monk­fish”) and is also bet­ter than the worst pos­si­ble out­come of eat­ing no main course (“hun­gry”).

If the worst out­comes of two or more acts are equally good, the max­imin prin­ci­ple tells you to be in­differ­ent be­tween them. But that doesn’t seem right. For this rea­son, fans of the max­imin prin­ci­ple of­ten in­voke the lex­i­cal max­imin prin­ci­ple (“lex­imin”), which says that if the worst out­comes of two or more acts are equally good, one should choose the act for which the sec­ond worst out­come is best. (If that doesn’t sin­gle out a sin­gle act, then the third worst out­come should be con­sid­ered, and so on.)

Why adopt the lex­imin prin­ci­ple? Ad­vo­cates point out that the lex­imin prin­ci­ple trans­forms a de­ci­sion prob­lem un­der ig­no­rance into a de­ci­sion prob­lem un­der par­tial cer­tainty. The de­ci­sion maker doesn’t know what the out­come will be, but they know what the worst pos­si­ble out­come will be.

But in some cases, the lex­imin rule seems clearly ir­ra­tional. Imag­ine this de­ci­sion prob­lem, with two pos­si­ble acts and two pos­si­ble states of the world:

s1 s2
a1 $1 $10,001.01
a2 $1.01 $1.01

In this situ­a­tion, the lex­imin prin­ci­ple pre­scribes choos­ing a2. But most peo­ple would agree it is ra­tio­nal to risk los­ing out on a sin­gle cent for the chance to get an ex­tra $10,000.

6.3. Max­i­max and op­ti­mism-pessimism

The max­imin and lex­imin rules fo­cus their at­ten­tion on the worst pos­si­ble out­comes of a de­ci­sion, but why not fo­cus on the best pos­si­ble out­come? The max­i­max prin­ci­ple pre­scribes that if the best pos­si­ble out­come of one act is bet­ter than the best pos­si­ble out­come of an­other act, then the former act should be cho­sen.

More pop­u­lar among de­ci­sion the­o­rists is the op­ti­mism-pes­simism rule (aka the alpha-in­dex rule). The op­ti­mism-pes­simism rule pre­scribes that one con­sider both the best and worst pos­si­ble out­come of each pos­si­ble act, and then choose ac­cord­ing to one’s de­gree of op­ti­mism or pes­simism.

Here’s an ex­am­ple from Peter­son (2009, ch. 3):

s1 s2 s3 s4 s5 s6
a1 55 18 28 10 36 100
a2 50 87 55 90 75 70

We rep­re­sent the de­ci­sion maker’s level of op­ti­mism on a scale of 0 to 1, where 0 is max­i­mal pes­simism and 1 is max­i­mal op­ti­mism. For a1, the worst pos­si­ble out­come is 10 and the best pos­si­ble out­come is 100. That is, min(a1) = 10 and max(a1) = 100. So if the de­ci­sion maker is 0.85 op­ti­mistic, then the to­tal value of a1 is (0.85)(100) + (1 − 0.85)(10) = 86.5, and the to­tal value of a2 is (0.85)(90) + (1 − 0.85)(50) = 84. In this situ­a­tion, the op­ti­mism-pes­simism rule pre­scribes ac­tion a1.

If the de­ci­sion maker’s op­ti­mism is 0, then the op­ti­mism-pes­simism rule col­lapses into the max­imin rule be­cause (0)(max(ai)) + (1 − 0)(min(ai)) = min(ai). And if the de­ci­sion maker’s op­ti­mism is 1, then the op­ti­mism-pes­simism rule col­lapses into the max­i­max rule. Thus, the op­ti­mism-pes­simism rule turns out to be a gen­er­al­iza­tion of the max­imin and max­i­max rules. (Well, sort of. The min­i­max and max­i­max prin­ci­ples re­quire only that we mea­sure value on an or­di­nal scale, whereas the op­ti­mism-pes­simism rule re­quires that we mea­sure value on an in­ter­val scale.)

The op­ti­mism-pes­simism rule pays at­ten­tion to both the best-case and worst-case sce­nar­ios, but is it ra­tio­nal to ig­nore all the out­comes in be­tween? Con­sider this ex­am­ple:

s1 s2 s3
a1 1 2 100
a2 1 99 100

The max­i­mum and min­i­mum val­ues for a1 and a2 are the same, so for ev­ery de­gree of op­ti­mism both acts are equally good. But it seems ob­vi­ous that one should choose a2.

6.4. Other de­ci­sion principles

Many other de­ci­sion prin­ci­ples for deal­ing with de­ci­sions un­der ig­no­rance have been pro­posed, in­clud­ing min­i­max re­gret, info-gap, and max­ipok. For more de­tails on mak­ing de­ci­sions un­der ig­no­rance, see Peter­son (2009) and Bossert et al. (2000).

One queer fea­ture of the de­ci­sion prin­ci­ples dis­cussed in this sec­tion is that they willfully dis­re­gard some in­for­ma­tion rele­vant to mak­ing a de­ci­sion. Such a move could make sense when try­ing to find a de­ci­sion al­gorithm that performs well un­der tight limits on available com­pu­ta­tion (Braf­man & Ten­nen­holtz (2000)), but it’s un­clear why an ideal agent with in­finite com­put­ing power (fit for a nor­ma­tive rather than a pre­scrip­tive the­ory) should willfully dis­re­gard in­for­ma­tion.

7. Can de­ci­sions un­der ig­no­rance be trans­formed into de­ci­sions un­der un­cer­tainty?

Can de­ci­sions un­der ig­no­rance be trans­formed into de­ci­sions un­der un­cer­tainty? This would sim­plify things greatly, be­cause there is near-uni­ver­sal agree­ment that de­ci­sions un­der un­cer­tainty should be han­dled by “max­i­miz­ing ex­pected util­ity” (see sec­tion 11 for clar­ifi­ca­tions), whereas de­ci­sion the­o­rists still de­bate what should be done about de­ci­sions un­der ig­no­rance.

For Bayesi­ans (see sec­tion 10), all de­ci­sions un­der ig­no­rance are trans­formed into de­ci­sions un­der un­cer­tainty (Win­kler 2003, ch. 5) when the de­ci­sion maker as­signs an “ig­no­rance prior” to each out­come for which they don’t know how to as­sign a prob­a­bil­ity. (Another way of say­ing this is to say that a Bayesian de­ci­sion maker never faces a de­ci­sion un­der ig­no­rance, be­cause a Bayesian must always as­sign a prior prob­a­bil­ity to events.) One must then con­sider how to as­sign pri­ors, an im­por­tant de­bate among Bayesi­ans (see sec­tion 10).

Many non-Bayesian de­ci­sion the­o­rists also think that de­ci­sions un­der ig­no­rance can be trans­formed into de­ci­sions un­der un­cer­tainty due to some­thing called the prin­ci­ple of in­suffi­cient rea­son. The prin­ci­ple of in­suffi­cient rea­son pre­scribes that if you have liter­ally no rea­son to think that one state is more prob­a­ble than an­other, then one should as­sign equal prob­a­bil­ity to both states.

One ob­jec­tion to the prin­ci­ple of in­suffi­cient rea­son is that it is very sen­si­tive to how states are in­di­vi­d­u­ated. Peter­son (2009, ch. 3) ex­plains:

Sup­pose that be­fore em­bark­ing on a trip you con­sider whether to bring an um­brella or not. [But] you know noth­ing about the weather at your des­ti­na­tion. If the for­mal­iza­tion of the de­ci­sion prob­lem is taken to in­clude only two states, viz. rain and no rain, [then by the prin­ci­ple of in­suffi­cient rea­son] the prob­a­bil­ity of each state will be 12. How­ever, it seems that one might just as well go for a for­mal­iza­tion that di­vides the space of pos­si­bil­ities into three states, viz. heavy rain, mod­er­ate rain, and no rain. If the prin­ci­ple of in­suffi­cient rea­son is ap­plied to the lat­ter set of states, their prob­a­bil­ities will be 13. In some cases this differ­ence will af­fect our de­ci­sions. Hence, it seems that any­one ad­vo­cat­ing the prin­ci­ple of in­suffi­cient rea­son must [defend] the rather im­plau­si­ble hy­poth­e­sis that there is only one cor­rect way of mak­ing up the set of states.

An objection to the principle of insufficient reason

An ob­jec­tion to the prin­ci­ple of in­suffi­cient reason

Ad­vo­cates of the prin­ci­ple of in­suffi­cient rea­son might re­spond that one must con­sider sym­met­ric states. For ex­am­ple if some­one gives you a die with n sides and you have no rea­son to think the die is bi­ased, then you should as­sign a prob­a­bil­ity of 1/​n to each side. But, Peter­son notes:

...not all events can be de­scribed in sym­met­ric terms, at least not in a way that jus­tifies the con­clu­sion that they are equally prob­a­ble. Whether Ann’s mar­riage will be a happy one de­pends on her fu­ture emo­tional at­ti­tude to­ward her hus­band. Ac­cord­ing to one de­scrip­tion, she could be ei­ther in love or not in love with him; then the prob­a­bil­ity of both states would be 12. Ac­cord­ing to an­other equally plau­si­ble de­scrip­tion, she could ei­ther be deeply in love, a lit­tle bit in love or not at all in love with her hus­band; then the prob­a­bil­ity of each state would be 13.

8. How should I make de­ci­sions un­der un­cer­tainty?

A de­ci­sion maker faces a “de­ci­sion un­der un­cer­tainty” when she (1) knows which acts she could choose and which out­comes they may re­sult in, and she (2) as­signs prob­a­bil­ities to the out­comes.

De­ci­sion the­o­rists gen­er­ally agree that when fac­ing a de­ci­sion un­der un­cer­tainty, it is ra­tio­nal to choose the act with the high­est ex­pected util­ity. This is the prin­ci­ple of ex­pected util­ity max­i­miza­tion (EUM).

De­ci­sion the­o­rists offer two kinds of jus­tifi­ca­tions for EUM. The first has to do with the law of large num­bers (see sec­tion 8.1). The sec­ond has to do with the ax­io­matic ap­proach (see sec­tions 8.2 through 8.6).

8.1. The law of large numbers

The “law of large num­bers,” which states that in the long run, if you face the same de­ci­sion prob­lem again and again and again, and you always choose the act with the high­est ex­pected util­ity, then you will al­most cer­tainly be bet­ter off than if you choose any other acts.

There are two prob­lems with us­ing the law of large num­bers to jus­tify EUM. The first prob­lem is that the world is ever-chang­ing, so we rarely if ever face the same de­ci­sion prob­lem “again and again and again.” The law of large num­bers says that if you face the same de­ci­sion prob­lem in­finitely many times, then the prob­a­bil­ity that you could do bet­ter by not max­i­miz­ing ex­pected util­ity ap­proaches zero. But you won’t ever face the same de­ci­sion prob­lem in­finitely many times! Why should you care what would hap­pen if a cer­tain con­di­tion held, if you know that con­di­tion will never hold?

The sec­ond prob­lem with us­ing the law of large num­bers to jus­tify EUM has to do with a math­e­mat­i­cal the­o­rem known as gam­bler’s ruin. Imag­ine that you and I flip a fair coin, and I pay you $1 ev­ery time it comes up heads and you pay me $1 ev­ery time it comes up tails. We both start with $100. If we flip the coin enough times, one of us will face a situ­a­tion in which the se­quence of heads or tails is longer than we can af­ford. If a long-enough se­quence of heads comes up, I’ll run out of $1 bills with which to pay you. If a long-enough se­quence of tails comes up, you won’t be able to pay me. So in this situ­a­tion, the law of large num­bers guaran­tees that you will be bet­ter off in the long run by max­i­miz­ing ex­pected util­ity only if you start the game with an in­finite amount of money (so that you never go broke), which is an un­re­al­is­tic as­sump­tion. (For tech­ni­cal con­ve­nience, as­sume util­ity in­creases lin­early with money. But the ba­sic point holds with­out this as­sump­tion.)

8.2. The ax­io­matic approach

The other method for jus­tify­ing EUM seeks to show that EUM can be de­rived from ax­ioms that hold re­gard­less of what hap­pens in the long run.

In this sec­tion we will re­view per­haps the most fa­mous ax­io­matic ap­proach, from Von Neu­mann and Mor­gen­stern (1947). Other ax­io­matic ap­proaches in­clude Sav­age (1954), Jeffrey (1983), and An­scombe & Au­mann (1963).

8.3. The Von Neu­mann-Mor­gen­stern util­ity theorem

The first de­ci­sion the­ory ax­iom­a­ti­za­tion ap­peared in an ap­pendix to the sec­ond edi­tion of Von Neu­mann & Mor­gen­stern’s The­ory of Games and Eco­nomic Be­hav­ior (1947). An im­por­tant point to note up front is that, in this ax­iom­a­ti­za­tion, Von Neu­mann and Mor­gen­stern take the op­tions that the agent chooses be­tween to not be acts, as we’ve defined them, but lot­ter­ies (where a lot­tery is a set of out­comes, each paired with a prob­a­bil­ity). As such, while dis­cussing their ax­iom­a­ti­za­tion, we will talk of lot­ter­ies. (De­spite mak­ing this dis­tinc­tion, acts and lot­ter­ies are closely re­lated. Un­der the con­di­tions of un­cer­tainty that we are con­sid­er­ing here, each act will be as­so­ci­ated with some lot­tery and so prefer­ences over lot­ter­ies could be used to de­ter­mine prefer­ences over acts, if so de­sired).

The key fea­ture of the Von Neu­mann and Mor­gen­stern ax­iom­a­ti­za­tion is a proof that if a de­ci­sion maker states her prefer­ences over a set of lot­ter­ies, and if her prefer­ences con­form to a set of in­tu­itive struc­tural con­straints (ax­ioms), then we can con­struct a util­ity func­tion (on an in­ter­val scale) from her prefer­ences over lot­ter­ies and show that she acts as if she max­i­mizes ex­pected util­ity with re­spect to that util­ity func­tion.

What are the ax­ioms to which an agent’s prefer­ences over lot­ter­ies must con­form? There are four of them.

  1. The com­plete­ness ax­iom states that the agent must bother to state a prefer­ence for each pair of lot­ter­ies. That is, the agent must pre­fer A to B, or pre­fer B to A, or be in­differ­ent be­tween the two.

  2. The tran­si­tivity ax­iom states that if the agent prefers A to B and B to C, she must also pre­fer A to C.

  3. The in­de­pen­dence ax­iom states that, for ex­am­ple, if an agent prefers an ap­ple to an or­ange, then she must also pre­fer the lot­tery [55% chance she gets an ap­ple, oth­er­wise she gets cholera] over the lot­tery [55% chance she gets an or­ange, oth­er­wise she gets cholera]. More gen­er­ally, this ax­iom holds that a prefer­ence must hold in­de­pen­dently of the pos­si­bil­ity of an­other out­come (e.g. cholera).

  4. The con­ti­nu­ity ax­iom holds that if the agent prefers A to B to C, then there ex­ists a unique p (prob­a­bil­ity) such that the agent is in­differ­ent be­tween [p(A) + (1 - p)(C)] and [out­come B with cer­tainty].

The con­ti­nu­ity ax­iom re­quires more ex­pla­na­tion. Sup­pose that A = $1 mil­lion, B = $0, and C = Death. If p = 0.5, then the agent’s two lot­ter­ies un­der con­sid­er­a­tion for the mo­ment are:

  1. (0.5)($1M) + (1 − 0.5)(Death) [win $1M with 50% prob­a­bil­ity, die with 50% prob­a­bil­ity]

  2. (1)($0) [win $0 with cer­tainty]

Most peo­ple would not be in­differ­ent be­tween $0 with cer­tainty and [50% chance of $1M, 50% chance of Death] — the risk of Death is too high! But if you have con­tin­u­ous prefer­ences, there is some prob­a­bil­ity p for which you’d be in­differ­ent be­tween these two lot­ter­ies. Per­haps p is very, very high:

  1. (0.999999)($1M) + (1 − 0.999999)(Death) [win $1M with 99.9999% prob­a­bil­ity, die with 0.0001% prob­a­bil­ity]

  2. (1)($0) [win $0 with cer­tainty]

Per­haps now you’d be in­differ­ent be­tween lot­tery 1 and lot­tery 2. Or maybe you’d be more will­ing to risk Death for the chance of win­ning $1M, in which case the p for which you’d be in­differ­ent be­tween lot­ter­ies 1 and 2 is lower than 0.999999. As long as there is some p at which you’d be in­differ­ent be­tween lot­ter­ies 1 and 2, your prefer­ences are “con­tin­u­ous.”

Given this setup, Von Neu­mann and Mor­gen­stern proved their the­o­rem, which states that if the agent’s prefer­ences over lot­ter­ies obeys their ax­ioms, then:

  • The agent’s prefer­ences can be rep­re­sented by a util­ity func­tion that as­signs higher util­ity to preferred lot­ter­ies.

  • The agent acts in ac­cor­dance with the prin­ci­ple of max­i­miz­ing ex­pected util­ity.

  • All util­ity func­tions satis­fy­ing the above two con­di­tions are “pos­i­tive lin­ear trans­for­ma­tions” of each other. (Without go­ing into the de­tails: this is why VNM-util­ity is mea­sured on an in­ter­val scale.)

8.4. VNM util­ity the­ory and rationality

An agent which con­forms to the VNM ax­ioms is some­times said to be “VNM-ra­tio­nal.” But why should “VNM-ra­tio­nal­ity” con­sti­tute our no­tion of ra­tio­nal­ity in gen­eral? How could VNM’s re­sult jus­tify the claim that a ra­tio­nal agent max­i­mizes ex­pected util­ity when fac­ing a de­ci­sion un­der un­cer­tainty? The ar­gu­ment goes like this:

  1. If an agent chooses lot­ter­ies which it prefers (in de­ci­sions un­der un­cer­tainty), and if its prefer­ences con­form to the VNM ax­ioms, then it is ra­tio­nal. Other­wise, it is ir­ra­tional.

  2. If an agent chooses lot­ter­ies which it prefers (in de­ci­sions un­der un­cer­tainty), and if its prefer­ences con­form to the VNM ax­ioms, then it max­i­mizes ex­pected util­ity.

  3. There­fore, a ra­tio­nal agent max­i­mizes ex­pected util­ity (in de­ci­sions un­der un­cer­tainty).

Von Neu­mann and Mor­gen­stern proved premise 2, and the con­clu­sion fol­lows from premise 1 and 2. But why ac­cept premise 1?

Few peo­ple deny that it would be ir­ra­tional for an agent to choose a lot­tery which it does not pre­fer. But why is it ir­ra­tional for an agent’s prefer­ences to vi­o­late the VNM ax­ioms? I will save that dis­cus­sion for sec­tion 8.6.

8.5. Ob­jec­tions to VNM-rationality

Sev­eral ob­jec­tions have been raised to Von Neu­mann and Mor­gen­stern’s re­sult:

  1. The VNM ax­ioms are too strong. Some have ar­gued that the VNM ax­ioms are not self-ev­i­dently true. See sec­tion 8.6.

  2. The VNM sys­tem offers no ac­tion guidance. A VNM-ra­tio­nal de­ci­sion maker can­not use VNM util­ity the­ory for ac­tion guidance, be­cause she must state her prefer­ences over lot­ter­ies at the start. But if an agent can state her prefer­ences over lot­ter­ies, then she already knows which lot­tery to choose. (For more on this, see sec­tion 9.)

  3. In the VNM sys­tem, util­ity is defined via prefer­ences over lot­ter­ies rather than prefer­ences over out­comes. To many, it seems odd to define util­ity with re­spect to prefer­ences over lot­ter­ies. Many would ar­gue that util­ity should be defined in re­la­tion to prefer­ences over out­comes or world-states, and that’s not what the VNM sys­tem does. (Also see sec­tion 9.)

8.6. Should we ac­cept the VNM ax­ioms?

The VNM prefer­ence ax­ioms define what it is for an agent to be VNM-ra­tio­nal. But why should we ac­cept these ax­ioms? Usu­ally, it is ar­gued that each of the ax­ioms are prag­mat­i­cally jus­tified be­cause an agent which vi­o­lates the ax­ioms can face situ­a­tions in which they are guaran­teed end up worse off (from their own per­spec­tive).

In sec­tions 8.6.1 and 8.6.2 I go into some de­tail about prag­matic jus­tifi­ca­tions offered for the tran­si­tivity and com­plete­ness ax­ioms. For more de­tail, in­clud­ing ar­gu­ments about the jus­tifi­ca­tion of the other ax­ioms, see Peter­son (2009, ch. 8) and Anand (1993).

8.6.1. The tran­si­tivity axiom

Con­sider the money-pump ar­gu­ment in fa­vor of the tran­si­tivity ax­iom (“if the agent prefers A to B and B to C, she must also pre­fer A to C”).

Imag­ine that a friend offers to give you ex­actly one of her three… nov­els, x or y or z… [and] that your prefer­ence or­der­ing over the three nov­els is… [that] you pre­fer x to y, and y to z, and z to x… [That is, your prefer­ences are cyclic, which is a type of in­tran­si­tive prefer­ence re­la­tion.] Now sup­pose that you are in pos­ses­sion of z, and that you are in­vited to swap z for y. Since you pre­fer y to z, ra­tio­nal­ity obliges you to swap. So you swap, and tem­porar­ily get y. You are then in­vited to swap y for x, which you do, since you pre­fer x to y. Fi­nally, you are offered to pay a small amount, say one cent, for swap­ping x for z. Since z is strictly [preferred to] x, even af­ter you have paid the fee for swap­ping, ra­tio­nal­ity tells you that you should ac­cept the offer. This means that you end up where you started, the only differ­ence be­ing that you now have one cent less. This pro­ce­dure is there­after iter­ated over and over again. After a billion cy­cles you have lost ten mil­lion dol­lars, for which you have got noth­ing in re­turn. (Peter­son 2009, ch. 8)

An example of a money-pump argument

An ex­am­ple of a money-pump argument

Similar ar­gu­ments (e.g. Gustafs­son 2010) aim to show that the other kind of in­tran­si­tive prefer­ences (acyclic prefer­ences) are ir­ra­tional, too.

(Of course, prag­matic ar­gu­ments need not be framed in mon­e­tary terms. We could just as well con­struct an ar­gu­ment show­ing that an agent with in­tran­si­tive prefer­ences can be “pumped” of all their hap­piness, or all their moral virtue, or all their Twinkies.)

8.6.2. The com­plete­ness axiom

The com­plete­ness ax­iom (“the agent must pre­fer A to B, or pre­fer B to A, or be in­differ­ent be­tween the two”) is of­ten at­tacked by say­ing that some goods or out­comes are in­com­men­su­rable — that is, they can­not be com­pared. For ex­am­ple, must a ra­tio­nal agent be able to state a prefer­ence (or in­differ­ence) be­tween money and hu­man welfare?

Per­haps the com­plete­ness ax­iom can be jus­tified with a prag­matic ar­gu­ment. If you think it is ra­tio­nally per­mis­si­ble to swap be­tween two in­com­men­su­rable goods, then one can con­struct a money pump ar­gu­ment in fa­vor of the com­plete­ness ax­iom. But if you think it is not ra­tio­nal to swap be­tween in­com­men­su­rable goods, then one can­not con­struct a money pump ar­gu­ment for the com­plete­ness ax­iom. (In fact, even if it is ra­tio­nal to swap be­tween in­com­men­su­rable goods, Man­dler, 2005 has demon­strated that an agent that al­lows their cur­rent choices to de­pend on the pre­vi­ous ones can avoid be­ing money pumped.)

And in fact, there is a pop­u­lar ar­gu­ment against the com­plete­ness ax­iom: the “small im­prove­ment ar­gu­ment.” For de­tails, see Chang (1997) and Espinoza (2007).

Note that in re­vealed prefer­ence the­ory, ac­cord­ing to which prefer­ences are re­vealed through choice be­hav­ior, there is no room for in­com­men­su­rable prefer­ences be­cause ev­ery choice always re­veals a prefer­ence re­la­tion of “bet­ter than,” “worse than,” or “equally as good as.”

Another pro­posal for deal­ing with the ap­par­ent in­com­men­su­ra­bil­ity of some goods (such as money and hu­man welfare) is the multi-at­tribute ap­proach:

In a multi-at­tribute ap­proach, each type of at­tribute is mea­sured in the unit deemed to be most suit­able for that at­tribute. Per­haps money is the right unit to use for mea­sur­ing fi­nan­cial costs, whereas the num­ber of lives saved is the right unit to use for mea­sur­ing hu­man welfare. The to­tal value of an al­ter­na­tive is there­after de­ter­mined by ag­gre­gat­ing the at­tributes, e.g. money and lives, into an over­all rank­ing of available al­ter­na­tives...

Sev­eral crite­ria have been pro­posed for choos­ing among al­ter­na­tives with mul­ti­ple at­tributes… [For ex­am­ple,] ad­di­tive crite­ria as­sign weights to each at­tribute, and rank al­ter­na­tives ac­cord­ing to the weighted sum calcu­lated by mul­ti­ply­ing the weight of each at­tribute with its value… [But while] it is per­haps con­tentious to mea­sure the util­ity of very differ­ent ob­jects on a com­mon scale, …it seems equally con­tentious to as­sign nu­mer­i­cal weights to at­tributes as sug­gested here....

[Now let us] con­sider a very gen­eral ob­jec­tion to multi-at­tribute ap­proaches. Ac­cord­ing to this ob­jec­tion, there ex­ist sev­eral equally plau­si­ble but differ­ent ways of con­struct­ing the list of at­tributes. Some­times the out­come of the de­ci­sion pro­cess de­pends on which set of at­tributes is cho­sen. (Peter­son 2009, ch. 8)

For more on the multi-at­tribute ap­proach, see Keeney & Raiffa (1993).

8.6.3. The Allais paradox

Hav­ing con­sid­ered the tran­si­tivity and com­plete­ness ax­ioms, we can now turn to in­de­pen­dence (a prefer­ence holds in­de­pen­dently of con­sid­er­a­tions of other pos­si­ble out­comes). Do we have any rea­son to re­ject this ax­iom? Here’s one rea­son to think we might: in a case known as the Allais para­dox Allais (1953) it may seem rea­son­able to act in a way that con­tra­dicts in­de­pen­dence.

The Allais para­dox asks us to con­sider two de­ci­sions (this ver­sion of the para­dox is based on Yud­kowsky (2008)).The first de­ci­sion in­volves the choice be­tween:

(1A) A cer­tain $24,000; and (1B) A 3334 chance of $27,000 and a 134 chance of noth­ing.

The sec­ond in­volves the choice be­tween:

(2A) A 34% chance of $24, 000 and a 66% chance of noth­ing; and (2B) A 33% chance of $27, 000 and a 67% chance of noth­ing.

Ex­per­i­ments have shown that many peo­ple pre­fer (1A) to (1B) and (2B) to (2A). How­ever, these prefer­ences con­tra­dict in­de­pen­dence. Op­tion 2A is the same as [a 34% chance of op­tion 1A and a 66% chance of noth­ing] while 2B is the same as [a 34% chance of op­tion 1B and a 66% chance of noth­ing]. So in­de­pen­dence im­plies that any­one that prefers (1A) to (1B) must also pre­fer (2A) to (2B).

When this re­sult was first un­cov­ered, it was pre­sented as ev­i­dence against the in­de­pen­dence ax­iom. How­ever, while the Allais para­dox clearly re­veals that in­de­pen­dence fails as a de­scrip­tive ac­count of choice, it’s less clear what it im­plies about the nor­ma­tive ac­count of ra­tio­nal choice that we are dis­cussing in this doc­u­ment. As noted in Peter­son (2009, ch. 4), how­ever:

[S]ince many peo­ple who have thought very hard about this ex­am­ple still feel that it would be ra­tio­nal to stick to the prob­le­matic prefer­ence pat­tern de­scribed above, there seems to be some­thing wrong with the ex­pected util­ity prin­ci­ple.

How­ever, Peter­son then goes on to note that, many peo­ple, like the statis­ti­cian Leonard Sav­age, ar­gue that it is peo­ple’s prefer­ence in the Allais para­dox that are in er­ror rather than the in­de­pen­dence ax­iom. If so, then the para­dox seems to re­veal the dan­ger of rely­ing too strongly on in­tu­ition to de­ter­mine the form that should be taken by nor­ma­tive the­o­ries of ra­tio­nal.

8.6.4. The Ells­berg paradox

The Allais para­dox is far from the only case where peo­ple fail to act in ac­cor­dance with EUM. Another well-known case is the Ells­berg para­dox (the fol­low­ing is taken from Res­nik (1987):

An urn con­tains ninety uniformly sized balls, which are ran­domly dis­tributed. Thirty of the balls are yel­low, the re­main­ing sixty are red or blue. We are not told how many red (blue) balls are in the urn – ex­cept that they num­ber any­where from zero to sixty. Now con­sider the fol­low­ing pair of situ­a­tions. In each situ­a­tion a ball will be drawn and we will be offered a bet on its color. In situ­a­tion A we will choose be­tween bet­ting that it is yel­low or that it is red. In situ­a­tion B we will choose be­tween bet­ting that it is red or blue or that it is yel­low or blue.

If we guess the cor­rect color, we will re­ceive a pay­out of $100. In the Ells­berg para­dox, many peo­ple bet yel­low in situ­a­tion A and red or blue in situ­a­tion B. Fur­ther, many peo­ple make these de­ci­sions not be­cause they are in­differ­ent in both situ­a­tions, and so happy to choose ei­ther way, but rather be­cause they have a strict prefer­ence to choose in this man­ner.

The Ellsberg paradox

The Ells­berg paradox

How­ever, such be­hav­ior can­not be in ac­cor­dance with EUM. In or­der for EUM to en­dorse a strict prefer­ence for choos­ing yel­low in situ­a­tion A, the agent would have to as­sign a prob­a­bil­ity of more than 13 to the ball se­lected be­ing blue. On the other hand, in or­der for EUM to en­dorse a strict prefer­ence for choos­ing red or blue in situ­a­tion B the agent would have to as­sign a prob­a­bil­ity of less than 13 to the se­lected ball be­ing blue. As such, these de­ci­sions can’t be jointly en­dorsed by an agent fol­low­ing EUM.

Those who deny that de­ci­sions mak­ing un­der ig­no­rance can be trans­formed into de­ci­sion mak­ing un­der un­cer­tainty have an easy re­sponse to the Ells­berg para­dox: as this case in­volves de­cid­ing un­der a situ­a­tion of ig­no­rance, it is ir­rele­vant whether peo­ple’s de­ci­sions vi­o­late EUM in this case as EUM is not ap­pli­ca­ble to such situ­a­tions.

Those who be­lieve that EUM pro­vides a suit­able stan­dard for choice in such situ­a­tions, how­ever, need to find some other way of re­spond­ing to the para­dox. As with the Allais para­dox, there is some dis­agree­ment about how best to do so. Once again, how­ever, many peo­ple, in­clud­ing Leonard Sav­age, ar­gue that EUM reaches the right de­ci­sion in this case. It is our in­tu­itions that are flawed (see again Res­nik (1987) for a nice sum­mary of Sav­age’s ar­gu­ment to this con­clu­sion).

8.6.5. The St Peters­burg paradox

Another ob­jec­tion to the VNM ap­proach (and to ex­pected util­ity ap­proaches gen­er­ally), the St. Peters­burg para­dox, draws on the pos­si­bil­ity of in­finite util­ities. The St. Peters­burg para­dox is based around a game where a fair coin is tossed un­til it lands heads up. At this point, the agent re­ceives a prize worth 2n util­ity, where n is equal to the num­ber of times the coin was tossed dur­ing the game. The so-called para­dox oc­curs be­cause the ex­pected util­ity of choos­ing to play this game is in­finite and so, ac­cord­ing to a stan­dard ex­pected util­ity ap­proach, the agent should be will­ing to pay any finite amount to play the game. How­ever, this seems un­rea­son­able. In­stead, it seems that the agent should only be will­ing to pay a rel­a­tively small amount to do so. As such, it seems that the ex­pected util­ity ap­proach gets some­thing wrong.

Var­i­ous re­sponses have been sug­gested. Most ob­vi­ously, we could say that the para­dox does not ap­ply to VNM agents, since the VNM the­o­rem as­signs real num­bers to all lot­ter­ies, and in­finity is not a real num­ber. But it’s un­clear whether this es­capes the prob­lem. After all, at it’s core, the St. Peters­burg para­dox is not about in­finite util­ities but rather about cases where ex­pected util­ity ap­proaches seem to over­value some choice, and such cases seem to ex­ist even in finite cases. For ex­am­ple, if we let L be a finite limit on util­ity we could con­sider the fol­low­ing sce­nario (from Peter­son, 2009, p. 85):

A fair coin is tossed un­til it lands heads up. The player there­after re­ceives a prize worth min {2n · 10-100, L} units of util­ity, where n is the num­ber of times the coin was tossed.

In this case, even if an ex­tremely low value is set for L, it seems that pay­ing this amount to play the game is un­rea­son­able. After all, as Peter­son notes, about nine times out of ten an agent that plays this game will win no more than 8 · 10-100 util­ity. If pay­ing 1 util­ity is, in fact, un­rea­son­able in this case, then sim­ply limit­ing an agent’s util­ity to some finite value doesn’t provide a defence of ex­pected util­ity ap­proaches. (Other prob­lems abound. See Yud­kowsky, 2007 for an in­ter­est­ing finite prob­lem and Nover & Ha­jek, 2004 for a par­tic­u­larly per­plex­ing prob­lem with links to the St Peters­burg para­dox.)

As it stands, there is no agree­ment about pre­cisely what the St Peters­burg para­dox re­veals. Some peo­ple ac­cept one of the var­i­ous re­s­olu­tions of the case and so find the para­dox un­con­cern­ing. Others think the para­dox re­veals a se­ri­ous prob­lem for ex­pected util­ity the­o­ries. Still oth­ers think the para­dox is un­re­solved but don’t think that we should re­spond by aban­don­ing ex­pected util­ity the­ory.

9. Does ax­io­matic de­ci­sion the­ory offer any ac­tion guidance?

For the de­ci­sion the­o­ries listed in sec­tion 8.2, it’s of­ten claimed the an­swer is “no.” To ex­plain this, I must first ex­am­ine some differ­ences be­tween di­rect and in­di­rect ap­proaches to ax­io­matic de­ci­sion the­ory.

Peter­son (2009, ch. 4) ex­plains:

In the in­di­rect ap­proach, which is the dom­i­nant ap­proach, the de­ci­sion maker does not pre­fer a risky act [or lot­tery] to an­other be­cause the ex­pected util­ity of the former ex­ceeds that of the lat­ter. In­stead, the de­ci­sion maker is asked to state a set of prefer­ences over a set of risky acts… Then, if the set of prefer­ences stated by the de­ci­sion maker is con­sis­tent with a small num­ber of struc­tural con­straints (ax­ioms), it can be shown that her de­ci­sions can be de­scribed as if she were choos­ing what to do by as­sign­ing nu­mer­i­cal prob­a­bil­ities and util­ities to out­comes and then max­imis­ing ex­pected util­ity...

[In con­trast] the di­rect ap­proach seeks to gen­er­ate prefer­ences over acts from prob­a­bil­ities and util­ities di­rectly as­signed to out­comes. In con­trast to the in­di­rect ap­proach, it is not as­sumed that the de­ci­sion maker has ac­cess to a set of prefer­ences over acts be­fore he starts to de­liber­ate.

The ax­io­matic de­ci­sion the­o­ries listed in sec­tion 8.2 all fol­low the in­di­rect ap­proach. Th­ese the­o­ries, it might be said, can­not offer any ac­tion guidance be­cause they re­quire an agent to state its prefer­ences over acts “up front.” But an agent that states its prefer­ences over acts already knows which act it prefers, so the de­ci­sion the­ory can’t offer any ac­tion guidance not already pre­sent in the agent’s own stated prefer­ences over acts.

Peter­son (2009, ch .10) gives a prac­ti­cal ex­am­ple:

For ex­am­ple, a forty-year-old woman seek­ing ad­vice about whether to, say, di­vorce her hus­band, is likely to get very differ­ent an­swers from the [two ap­proaches]. The [in­di­rect ap­proach] will ad­vise the woman to first figure out what her prefer­ences are over a very large set of risky acts, in­clud­ing the one she is think­ing about perform­ing, and then just make sure that all prefer­ences are con­sis­tent with cer­tain struc­tural re­quire­ments. Then, as long as none of the struc­tural re­quire­ments is vi­o­lated, the woman is free to do what­ever she likes, no mat­ter what her be­liefs and de­sires ac­tu­ally are… The [di­rect ap­proach] will [in­stead] ad­vise the woman to first as­sign nu­mer­i­cal util­ities and prob­a­bil­ities to her de­sires and be­liefs, and then ag­gre­gate them into a de­ci­sion by ap­ply­ing the prin­ci­ple of max­i­miz­ing ex­pected util­ity.

Thus, it seems only the di­rect ap­proach offers an agent any ac­tion guidance. But the di­rect ap­proach is very re­cent (Peter­son 2008; Cozic 2011), and only time will show whether it can stand up to pro­fes­sional crit­i­cism.

Warn­ing: Peter­son’s (2008) di­rect ap­proach is con­fus­ingly called “non-Bayesian de­ci­sion the­ory” de­spite as­sum­ing Bayesian prob­a­bil­ity the­ory.

For other at­tempts to pull ac­tion guidance from nor­ma­tive de­ci­sion the­ory, see Fallen­stein (2012) and Stien­non (2013).

10. How does prob­a­bil­ity the­ory play a role in de­ci­sion the­ory?

In or­der to calcu­late the ex­pected util­ity of an act (or lot­tery), it is nec­es­sary to de­ter­mine a prob­a­bil­ity for each out­come. In this sec­tion, I will ex­plore some of the de­tails of prob­a­bil­ity the­ory and its re­la­tion­ship to de­ci­sion the­ory.

For fur­ther in­tro­duc­tory ma­te­rial to prob­a­bil­ity the­ory, see How­son & Ur­bach (2005), Grim­met & Stirza­cker (2001), and Kol­ler & Fried­man (2009). This sec­tion draws heav­ily on Peter­son (2009, chs. 6 & 7) which pro­vides a very clear in­tro­duc­tion to prob­a­bil­ity in the con­text of de­ci­sion the­ory.

10.1. The ba­sics of prob­a­bil­ity theory

In­tu­itively, a prob­a­bil­ity is a num­ber be­tween 0 or 1 that la­bels how likely an event is to oc­cur. If an event has prob­a­bil­ity 0 then it is im­pos­si­ble and if it has prob­a­bil­ity 1 then it can’t pos­si­bly be false. If an event has a prob­a­bil­ity be­tween these val­ues, then this event it is more prob­a­ble the higher this num­ber is.

As with EUM, prob­a­bil­ity the­ory can be de­rived from a small num­ber of sim­ple ax­ioms. In the prob­a­bil­ity case, there are three of these, which are named the Kol­mogorov ax­ioms af­ter the math­e­mat­i­cian An­drey Kol­mogorov. The first of these states that prob­a­bil­ities are real num­bers be­tween 0 and 1. The sec­ond, that if a set of events are mu­tu­ally ex­clu­sive and ex­haus­tive then their prob­a­bil­ities should sum to 1. The third that if two events are mu­tu­ally ex­clu­sive then the prob­a­bil­ity that one or the other of these events will oc­cur is equal to the sum of their in­di­vi­d­ual prob­a­bil­ities.

From these three ax­ioms, the re­main­der of prob­a­bil­ity the­ory can be de­rived. In the re­main­der of this sec­tion, I will ex­plore some as­pects of this broader the­ory.

10.2. Bayes the­o­rem for up­dat­ing probabilities

From the per­spec­tive of de­ci­sion the­ory, one par­tic­u­larly im­por­tant as­pect of prob­a­bil­ity the­ory is the idea of a con­di­tional prob­a­bil­ity. Th­ese rep­re­sent how prob­a­ble some­thing is given a piece of in­for­ma­tion. So, for ex­am­ple, a con­di­tional prob­a­bil­ity could rep­re­sent how likely it is that it will be rain­ing, con­di­tion­ing on the fact that the weather fore­caster pre­dicted rain. A pow­er­ful tech­nique for calcu­lat­ing con­di­tional prob­a­bil­ities is Bayes the­o­rem (see Yud­kowsky, 2003 for a de­tailed in­tro­duc­tion). This for­mula states that:

P(A|B)=(P(B|A)P(A))/P(B)

P(A|B)=(P(B|A)P(A))/​P(B)

Bayes the­o­rem is used to calcu­late the prob­a­bil­ity of some event, A, given some ev­i­dence, B. As such, this for­mula can be used to up­date prob­a­bil­ities based on new ev­i­dence. So if you are try­ing to pre­dict the prob­a­bil­ity that it will rain to­mor­row and some­one gives you the in­for­ma­tion that the weather fore­caster pre­dicted that it will do so then this for­mula tells you how to calcu­late a new prob­a­bil­ity that it will rain based on your ex­ist­ing in­for­ma­tion. The ini­tial prob­a­bil­ity in such cases (be­fore the in­for­ma­tion is fac­tored into ac­count) is called the prior prob­a­bil­ity and the re­sult of ap­ply­ing Bayes the­o­rem is a new, pos­te­rior prob­a­bil­ity.

Using Bayes theorem to update probabilities based on the evidence provided by a weather forecast

Us­ing Bayes the­o­rem to up­date prob­a­bil­ities based on the ev­i­dence pro­vided by a weather forecast

Bayes the­o­rem can be seen as solv­ing the prob­lem of how to up­date prior prob­a­bil­ities based on new in­for­ma­tion. How­ever, it leaves open the ques­tion of how to de­ter­mine the prior prob­a­bil­ity in the first place. In some cases, there will be no ob­vi­ous way to do so. One solu­tion to this prob­lem sug­gests that any rea­son­able prior can be se­lected. Given enough ev­i­dence, re­peated ap­pli­ca­tions of Bayes the­o­rem will lead this prior prob­a­bil­ity to be up­dated to much the same pos­te­rior prob­a­bil­ity, even for peo­ple with widely differ­ent ini­tial pri­ors. As such, the ini­tially se­lected prior is less cru­cial than it may at first seem.

10.3. How should prob­a­bil­ities be in­ter­preted?

There are two main views about what prob­a­bil­ities mean: ob­jec­tivism and sub­jec­tivism. Loosely speak­ing, the ob­jec­tivist holds that prob­a­bil­ities tell us some­thing about the ex­ter­nal world while the sub­jec­tivist holds that they tell us some­thing about our be­liefs. Most de­ci­sion the­o­rists hold a sub­jec­tivist view about prob­a­bil­ity. Ac­cord­ing to this sort of view, prob­a­bil­ities rep­re­sent a sub­jec­tive de­grees of be­lief. So to say the prob­a­bil­ity of rain is 0.8 is to say that the agent un­der con­sid­er­a­tion has a high de­gree of be­lief that it will rain (see Jaynes, 2003 for a defense of this view). Note that, ac­cord­ing to this view, an­other agent in the same cir­cum­stance could as­sign a differ­ent prob­a­bil­ity that it will rain.

10.3.1. Why should de­grees of be­lief fol­low the laws of prob­a­bil­ity?

One ques­tion that might be raised against the sub­jec­tive ac­count of prob­a­bil­ity is why, on this ac­count, our de­grees of be­lief should satisfy the Kol­mogorov ax­ioms. For ex­am­ple, why should our sub­jec­tive de­grees of be­lief in mu­tu­ally ex­clu­sive, ex­haus­tive events add to 1? One an­swer to this ques­tion shows that agents whose de­grees of be­lief don’t satisfy these ax­ioms will be sub­ject to Dutch Book bets. Th­ese are bets where the agent will in­evitably lose money. Peter­son (2009, ch. 7) ex­plains:

Sup­pose, for in­stance, that you be­lieve to de­gree 0.55 that at least one per­son from In­dia will win a gold medal in the next Olympic Games (event G), and that your sub­jec­tive de­gree of be­lief is 0.52 that no In­dian will win a gold medal in the next Olympic Games (event ¬G). Also sup­pose that a cun­ning bookie offers you a bet on both of these events. The bookie promises to pay you $1 for each event that ac­tu­ally takes place. Now, since your sub­jec­tive de­gree of be­lief that G will oc­cur is 0.55 it would be ra­tio­nal to pay up to $1·0.55 = $0.55 for en­ter­ing this bet. Fur­ther­more, since your de­gree of be­lief in ¬G is 0.52 you should be will­ing to pay up to $0.52 for en­ter­ing the sec­ond bet, since $1·0.52 = $0.52. How­ever, by now you have paid $1.07 for tak­ing on two bets that are cer­tain to give you a pay­off of $1 no mat­ter what hap­pens...Cer­tainly, this must be ir­ra­tional. Fur­ther­more, the rea­son why this is ir­ra­tional is that your sub­jec­tive de­grees of be­lief vi­o­late the prob­a­bil­ity calcu­lus.

A Dutch Book argument

A Dutch Book argument

It can be proven that an agent is sub­ject to Dutch Book bets if, and only if, their de­grees of be­lief vi­o­late the ax­ioms of prob­a­bil­ity. This pro­vides an ar­gu­ment for why de­grees of be­liefs should satisfy these ax­ioms.

10.3.2. Mea­sur­ing sub­jec­tive probabilities

Another challenges raised by the sub­jec­tive view is how we can mea­sure prob­a­bil­ities. If these rep­re­sent sub­jec­tive de­grees of be­lief there doesn’t seem to be an easy way to de­ter­mine these based on ob­ser­va­tions of the world. How­ever, a num­ber of re­sponses to this prob­lem have been ad­vanced, one of which is ex­plained suc­cinctly by Peter­son (2009, ch. 7):

The main in­no­va­tions pre­sented by… Sav­age can be char­ac­ter­ised as sys­tem­atic pro­ce­dures for link­ing prob­a­bil­ity… to claims about ob­jec­tively ob­serv­able be­hav­ior, such as prefer­ence re­vealed in choice be­hav­ior. Imag­ine, for in­stance, that we wish to mea­sure Caroline’s sub­jec­tive prob­a­bil­ity that the coin she is hold­ing in her hand will land heads up the next time it is tossed. First, we ask her which of the fol­low­ing very gen­er­ous op­tions she would pre­fer.

A: “If the coin lands heads up you win a sports car; oth­er­wise you win noth­ing.”

B: “If the coin does not land heads up you win a sports car; oth­er­wise you win noth­ing.”

Sup­pose Caroline prefers A to B. We can then safely con­clude that she thinks it is more prob­a­ble that the coin will land heads up rather than not. This fol­lows from the as­sump­tion that Caroline prefers to win a sports car rather than noth­ing, and that her prefer­ence be­tween un­cer­tain prospects is en­tirely de­ter­mined by her be­liefs and de­sires with re­spect to her prospects of win­ning the sports car...

Next, we need to gen­er­al­ise the mea­sure­ment pro­ce­dure out­lined above such that it al­lows us to always rep­re­sent Caroline’s de­grees of be­lief with pre­cise nu­mer­i­cal prob­a­bil­ities. To do this, we need to ask Caroline to state prefer­ences over a much larger set of op­tions and then rea­son back­wards… Sup­pose, for in­stance, that Caroline wishes to mea­sure her sub­jec­tive prob­a­bil­ity that her car worth $20,000 will be stolen within one year. If she con­sid­ers $1,000 to be… the high­est price she is pre­pared to pay for a gam­ble in which she gets $20,000 if the event S: “The car stolen within a year” takes place, and noth­ing oth­er­wise, then Caroline’s sub­jec­tive prob­a­bil­ity for S is 1,00020,000 = 0.05, given that she forms her prefer­ences in ac­cor­dance with the prin­ci­ple of max­imis­ing ex­pected mon­e­tary value...

The prob­lem with this method is that very few peo­ple form their prefer­ences in ac­cor­dance with the prin­ci­ple of max­imis­ing ex­pected mon­e­tary value. Most peo­ple have a de­creas­ing marginal util­ity for money...

For­tu­nately, there is a clever solu­tion to [this prob­lem]. The ba­sic idea is to im­pose a num­ber of struc­tural con­di­tions on prefer­ences over un­cer­tain op­tions [e.g. the tran­si­tivity ax­iom]. Then, the sub­jec­tive prob­a­bil­ity func­tion is es­tab­lished by rea­son­ing back­wards while tak­ing the struc­tural ax­ioms into ac­count: Since the de­ci­sion maker prefer­rred some un­cer­tain op­tions to oth­ers, and her prefer­ences… satisfy a num­ber of struc­ture ax­ioms, the de­ci­sion maker be­haves as if she were form­ing her prefer­ences over un­cer­tain op­tions by first as­sign­ing sub­jec­tive prob­a­bil­ities and util­ities to each op­tion and there­after max­imis­ing ex­pected util­ity.

A pe­cu­liar fea­ture of this ap­proach is, thus, that prob­a­bil­ities (and util­ities) are de­rived from ‘within’ the the­ory. The de­ci­sion maker does not pre­fer an un­cer­tain op­tion to an­other be­cause she judges the sub­jec­tive prob­a­bil­ities (and util­ities) of the out­comes to be more favourable than those of an­other. In­stead, the… struc­ture of the de­ci­sion maker’s prefer­ences over un­cer­tain op­tions log­i­cally im­plies that they can be de­scribed as if her choices were gov­erned by a sub­jec­tive prob­a­bil­ity func­tion and a util­ity func­tion...

...Sav­age’s ap­proach [seeks] to ex­pli­cate sub­jec­tive in­ter­pre­ta­tions of the prob­a­bil­ity ax­ioms by mak­ing cer­tain claims about prefer­ences over… un­cer­tain op­tions. But… why on earth should a the­ory of sub­jec­tive prob­a­bil­ity in­volve as­sump­tions about prefer­ences, given that prefer­ences and be­liefs are sep­a­rate en­tities? Con­trary to what is claimed by [Sav­age and oth­ers], emo­tion­ally in­ert de­ci­sion mak­ers failing to muster any prefer­ences at all… could cer­tainly hold par­tial be­liefs.

Other the­o­rists, for ex­am­ple DeG­root (1970), pro­pose other ap­proaches:

DeG­root’s ba­sic as­sump­tion is that de­ci­sion mak­ers can make qual­i­ta­tive com­par­i­sons be­tween pairs of events, and judge which one they think is most likely to oc­cur. For ex­am­ple, he as­sumes that one can judge whether it is more, less, or equally likely, ac­cord­ing to one’s own be­liefs, that it will rain to­day in Cam­bridge than in Cairo. DeG­root then shows that if the agent’s qual­i­ta­tive judg­ments are suffi­ciently fine-grained and satisfy a num­ber of struc­tural ax­ioms, then [they can be de­scribed by a prob­a­bil­ity dis­tri­bu­tion]. So in DeG­root’s… the­ory, the prob­a­bil­ity func­tion is ob­tained by fine-tun­ing qual­i­ta­tive data, thereby mak­ing them quan­ti­ta­tive.

11. What about “New­comb’s prob­lem” and al­ter­na­tive de­ci­sion al­gorithms?

Say­ing that a ra­tio­nal agent “max­i­mizes ex­pected util­ity” is, un­for­tu­nately, not spe­cific enough. There are a va­ri­ety of de­ci­sion al­gorithms which aim to max­i­mize ex­pected util­ity, and they give differ­ent an­swers to some de­ci­sion prob­lems, for ex­am­ple “New­comb’s prob­lem.”

In this sec­tion, we ex­plain these de­ci­sion al­gorithms and show how they perform on New­comb’s prob­lem and re­lated “New­comblike” prob­lems.

Gen­eral sources on this topic in­clude: Camp­bell & Sow­den (1985), Led­wig (2000), Joyce (1999), and Yud­kowsky (2010). Mo­ertel­maier (2013) dis­cusses New­comblike prob­lems in the con­text of the agent-en­vi­ron­ment frame­work.

11.1. New­comblike prob­lems and two de­ci­sion algorithms

I’ll be­gin with an ex­po­si­tion of sev­eral New­comblike prob­lems, so that I can re­fer to them in later sec­tions. I’ll also in­tro­duce our first two de­ci­sion al­gorithms, so that I can show how one’s choice of de­ci­sion al­gorithm af­fects an agent’s out­comes on these prob­lems.

11.1.1. New­comb’s Problem

New­comb’s prob­lem was for­mu­lated by the physi­cist William New­comb but first pub­lished in Noz­ick (1969). Below I pre­sent a ver­sion of it in­spired by Yud­kowsky (2010).

A su­per­in­tel­li­gent ma­chine named Omega vis­its Earth from an­other galaxy and shows it­self to be very good at pre­dict­ing events. This isn’t be­cause it has mag­i­cal pow­ers, but be­cause it knows more sci­ence than we do, has billions of sen­sors scat­tered around the globe, and runs effi­cient al­gorithms for mod­el­ing hu­mans and other com­plex sys­tems with un­prece­dented pre­ci­sion — on an ar­ray of com­puter hard­ware the size of our moon.

Omega pre­sents you with two boxes. Box A is trans­par­ent and con­tains $1000. Box B is opaque and con­tains ei­ther $1 mil­lion or noth­ing. You may choose to take both boxes (called “two-box­ing”), or you may choose to take only box B (called “one-box­ing”). If Omega pre­dicted you’ll two-box, then Omega has left box B empty. If Omega pre­dicted you’ll one-box, then Omega has placed $1M in box B.

By the time you choose, Omega has already left for its next game — the con­tents of box B won’t change af­ter you make your de­ci­sion. More­over, you’ve watched Omega play a thou­sand games against peo­ple like you, and on ev­ery oc­ca­sion Omega pre­dicted the hu­man player’s choice ac­cu­rately.

Should you one-box or two-box?

Newcomb’s problem

New­comb’s problem

Here’s an ar­gu­ment for two-box­ing. The $1M ei­ther is or is not in the box; your choice can­not af­fect the con­tents of box B now. So, you should two-box, be­cause then you get $1K plus what­ever is in box B. This is a straight­for­ward ap­pli­ca­tion of the dom­i­nance prin­ci­ple (sec­tion 6.1). Two-box­ing dom­i­nantes one-box­ing.

Con­vinced? Well, here’s an ar­gu­ment for one-box­ing. On all those ear­lier games you watched, ev­ery­one who two-boxed re­ceived $1K, and ev­ery­one who one-boxed re­ceived $1M. So you’re al­most cer­tain that you’ll get $1K for two-box­ing and $1M for one-box­ing, which means that to max­i­mize your ex­pected util­ity, you should one-box.

Noz­ick (1969) re­ports:

I have put this prob­lem to a large num­ber of peo­ple… To al­most ev­ery­one it is perfectly clear and ob­vi­ous what should be done. The difficulty is that these peo­ple seem to di­vide al­most evenly on the prob­lem, with large num­bers think­ing that the op­pos­ing half is just be­ing silly.

This is not a “merely ver­bal” dis­pute (Chalmers 2011). De­ci­sion the­o­rists have offered differ­ent al­gorithms for mak­ing a choice, and they have differ­ent out­comes. Trans­lated into English, the first al­gorithm (ev­i­den­tial de­ci­sion the­ory or EDT) says “Take ac­tions such that you would be glad to re­ceive the news that you had taken them.” The sec­ond al­gorithm (causal de­ci­sion the­ory or CDT) says “Take ac­tions which you ex­pect to have a pos­i­tive effect on the world.”

Many de­ci­sion the­o­rists have the in­tu­ition that CDT is right. But a CDT agent ap­pears to “lose” on New­comb’s prob­lem, end­ing up with $1000, while an EDT agent gains $1M. Pro­po­nents of EDT can ask pro­po­nents of CDT: “If you’re so smart, why aren’t you rich?” As Spohn (2012) writes, “this must be poor ra­tio­nal­ity that com­plains about the re­ward for ir­ra­tional­ity.” Or as Yud­kowsky (2010) ar­gues:

An ex­pected util­ity max­i­mizer should max­i­mize util­ity — not for­mal­ity, rea­son­able­ness, or defen­si­bil­ity...

In re­sponse to EDT’s ap­par­ent “win” over CDT on New­comb’s prob­lem, pro­po­nents of CDT have pre­sented similar prob­lems on which a CDT agent “wins” and an EDT agent “loses.” Pro­po­nents of EDT, mean­while, have replied with ad­di­tional New­comblike prob­lems on which EDT wins and CDT loses. Let’s ex­plore each of them in turn.

11.1.2. Ev­i­den­tial and causal de­ci­sion theory

First, how­ever, we will con­sider our two de­ci­sion al­gorithms in a lit­tle more de­tail.

EDT can be de­scribed sim­ply: ac­cord­ing to this the­ory, agents should use con­di­tional prob­a­bil­ities when de­ter­min­ing the ex­pected util­ity of differ­ent acts. Speci­fi­cally, they should use the prob­a­bil­ity of the world be­ing in each pos­si­ble state con­di­tion­ing on them car­ry­ing out the act un­der con­sid­er­a­tion. So in New­comb’s prob­lem they con­sider the prob­a­bil­ity that Box B con­tains $1 mil­lion or noth­ing con­di­tion­ing on the ev­i­dence pro­vided by their de­ci­sion to one-box or two-box. This is how the the­ory for­mal­izes the no­tion of an act pro­vid­ing good news.

CDT is more com­plex, at least in part be­cause it has been for­mu­lated in a va­ri­ety of differ­ent ways and these for­mu­la­tions are equiv­a­lent to one an­other only if cer­tain back­ground as­sump­tions are met. How­ever, a good sense of the the­ory can be gained by con­sid­er­ing the coun­ter­fac­tual ap­proach, which is one of the more in­tu­itive of these for­mu­la­tions. This ap­proach uti­lizes the prob­a­bil­ities of cer­tain coun­ter­fac­tual con­di­tion­als, which can be thought of as rep­re­sent­ing the causal in­fluence of an agent’s acts on the state of the world. Th­ese con­di­tion­als take the form “if I were to carry out a cer­tain act, then the world would be in a cer­tain state.” So in New­comb’s prob­lem, for ex­am­ple, this for­mu­la­tion of CDT con­sid­ers the prob­a­bil­ity of the coun­ter­fac­tu­als like “if I were to one-box, then Box B would con­tain $1 mil­lion” and, in do­ing so, con­sid­ers the causal in­fluence of one-box­ing on the con­tents of the boxes.

The same dis­tinc­tion can be made in for­mu­laic terms. Both EDT and CDT agree that de­ci­sion the­ory should be about max­i­miz­ing ex­pected util­ity where the ex­pected util­ity of an act, A, given a set of pos­si­ble out­comes, O, is defined as fol­lows:

expected utility formula.

In this equa­tion, V(A & O) rep­re­sents the value to the agent of the com­bi­na­tion of an act and an out­come. So this is the util­ity that the agent will re­ceive if they carry out a cer­tain act and a cer­tain out­come oc­curs. Fur­ther, PrAO rep­re­sents the prob­a­bil­ity of each out­come oc­cur­ring on the sup­po­si­tion that the agent car­ries out a cer­tain act. It is in terms of this prob­a­bil­ity that CDT and EDT differ. EDT uses the con­di­tional prob­a­bil­ity, Pr(O|A), while CDT uses the prob­a­bil­ity of sub­junc­tive con­di­tion­als, Pr(A O).

Us­ing these two ver­sions of the ex­pected util­ity for­mula, it’s pos­si­ble to demon­strate in a for­mal man­ner why EDT and CDT give the ad­vice they do in New­comb’s prob­lem. To demon­strate this it will help to make two sim­plify­ing as­sump­tions. First, we will pre­sume that each dol­lar of money is worth 1 unit of util­ity to the agent (and so will pre­sume that the agent’s util­ity is lin­ear with money). Se­cond, we will pre­sume that Omega is a perfect pre­dic­tor of hu­man ac­tions so that if the agent two-boxes it pro­vides defini­tive ev­i­dence that there is noth­ing in the opaque box and if the agent one-boxes it pro­vides defini­tive ev­i­dence that there is $1 mil­lion in this box. Given these as­sump­tions, EDT calcu­lates the ex­pected util­ity of each de­ci­sion as fol­lows:

EU for two-boxing according to EDT

EU for two-box­ing ac­cord­ing to EDT

EU for one-boxing according to EDT

EU for one-box­ing ac­cord­ing to EDT

Given that one-box­ing has a higher ex­pected util­ity ac­cord­ing to these calcu­la­tions, an EDT agent will one-box.

On the other hand, given that the agent’s de­ci­sion doesn’t causally in­fluence Omega’s ear­lier pre­dic­tion, CDT will use the same prob­a­bil­ity re­gard­less of whether you one or two box. The de­ci­sion en­dorsed will be the same re­gard­less of what prob­a­bil­ity we use so, to demon­strate the the­ory, we can sim­ply ar­bi­trar­ily as­sign an 0.5 prob­a­bil­ity that the opaque box has noth­ing in it and an 0.5 prob­a­bil­ity that it has one mil­lion dol­lars in it. CDT then calcu­lates the ex­pected util­ity of each de­ci­sion as fol­lows:

EU for two-boxing according to CDT

EU for two-box­ing ac­cord­ing to CDT

EU for one-boxing according to CDT

EU for one-box­ing ac­cord­ing to CDT

Given that two-box­ing has a higher ex­pected util­ity ac­cord­ing to these calcu­la­tions, a CDT agent will two-box. This ap­proach demon­strates the re­sult given more in­for­mally in the pre­vi­ous sec­tion: CDT agents will two-box in New­comb’s prob­lem and EDT agents will one box.

As men­tioned be­fore, there are also al­ter­na­tive for­mu­la­tions of CDT. What are these? For ex­am­ple, David Lewis (1981) and Brian Skyrms (1980) both pre­sent ap­proaches that rely on the par­ti­tion of the world into states to cap­ture causal in­for­ma­tion, rather than coun­ter­fac­tual con­di­tion­als. On Lewis’s ver­sion of this ac­count, for ex­am­ple, the agent calcu­lates the ex­pected util­ity of acts us­ing their un­con­di­tional cre­dence in states of the world that are de­pen­dency hy­pothe­ses, which are de­scrip­tions of the pos­si­ble ways that the world can de­pend on the agent’s ac­tions. Th­ese de­pen­dency hy­pothe­ses in­trin­si­cally con­tain the re­quired causal in­for­ma­tion.

Other tra­di­tional ap­proaches to CDT in­clude the imag­ing ap­proach of So­bel (1980) (also see Lewis 1981) and the un­con­di­tional ex­pec­ta­tions ap­proach of Leonard Sav­age (1954). Those in­ter­ested in the var­i­ous tra­di­tional ap­proaches to CDT would be best to con­sult Lewis (1981), Weirich (2008), and Joyce (1999). More re­cently, work in com­puter sci­ence on a tool called causal Bayesian net­works has led to an in­no­va­tive ap­proach to CDT that has re­ceived some re­cent at­ten­tion in the philo­soph­i­cal liter­a­ture (Pearl 2000, ch. 4 and Spohn 2012).

Now we re­turn to an anal­y­sis of de­ci­sion sce­nar­ios, armed with EDT and the coun­ter­fac­tual for­mu­la­tion of CDT.

11.1.3. Med­i­cal New­comb problems

Med­i­cal New­comb prob­lems share a similar form but come in many var­i­ants, in­clud­ing Solomon’s prob­lem (Gib­bard & Harper 1976) and the smok­ing le­sion prob­lem (Egan 2007). Below I pre­sent a var­i­ant called the “chew­ing gum prob­lem” (Yud­kowsky 2010):

Sup­pose that a re­cently pub­lished med­i­cal study shows that chew­ing gum seems to cause throat ab­scesses — an out­come-track­ing study showed that of peo­ple who chew gum, 90% died of throat ab­scesses be­fore the age of 50. Mean­while, of peo­ple who do not chew gum, only 10% die of throat ab­scesses be­fore the age of 50. The re­searchers, to ex­plain their re­sults, won­der if sal­iva slid­ing down the throat wears away cel­lu­lar defenses against bac­te­ria. Hav­ing read this study, would you choose to chew gum? But now a sec­ond study comes out, which shows that most gum-chew­ers have a cer­tain gene, CGTA, and the re­searchers pro­duce a table show­ing the fol­low­ing mor­tal­ity rates:

CGTA pre­sent CGTA ab­sent
Chew Gum 89% die 8% die
Don’t chew 99% die 11% die

This table shows that whether you have the gene CGTA or not, your chance of dy­ing of a throat ab­scess goes down if you chew gum. Why are fatal­ities so much higher for gum-chew­ers, then? Be­cause peo­ple with the gene CGTA tend to chew gum and die of throat ab­scesses. The au­thors of the sec­ond study also pre­sent a test-tube ex­per­i­ment which shows that the sal­iva from chew­ing gum can kill the bac­te­ria that form throat ab­scesses. The re­searchers hy­poth­e­size that be­cause peo­ple with the gene CGTA are highly sus­cep­ti­ble to throat ab­scesses, nat­u­ral se­lec­tion has pro­duced in them a ten­dency to chew gum, which pro­tects against throat ab­scesses. The strong cor­re­la­tion be­tween chew­ing gum and throat ab­scesses is not be­cause chew­ing gum causes throat ab­scesses, but be­cause a third fac­tor, CGTA, leads to chew­ing gum and throat ab­scesses.

Hav­ing learned of this new study, would you choose to chew gum? Chew­ing gum helps pro­tect against throat ab­scesses whether or not you have the gene CGTA. Yet a friend who heard that you had de­cided to chew gum (as peo­ple with the gene CGTA of­ten do) would be quite alarmed to hear the news — just as she would be sad­dened by the news that you had cho­sen to take both boxes in New­comb’s Prob­lem. This is a case where [EDT] seems to re­turn the wrong an­swer, call­ing into ques­tion the val­idity of the… rule “Take ac­tions such that you would be glad to re­ceive the news that you had taken them.” Although the news that some­one has de­cided to chew gum is alarm­ing, med­i­cal stud­ies nonethe­less show that chew­ing gum pro­tects against throat ab­scesses. [CDT’s] rule of “Take ac­tions which you ex­pect to have a pos­i­tive phys­i­cal effect on the world” seems to serve us bet­ter.

One re­sponse to this claim, called the tickle defense (Eells, 1981), ar­gues that EDT ac­tu­ally reaches the right de­ci­sion in such cases. Ac­cord­ing to this defense, the most rea­son­able way to con­strue the “chew­ing gum prob­lem” in­volves pre­sum­ing that CGTA causes a de­sire (a men­tal “tickle”) which then causes the agent to be more likely to chew gum, rather than CGTA di­rectly caus­ing the ac­tion. Given this, if we pre­sume that the agent already knows their own de­sires and hence already knows whether they’re likely to have the CGTA gene, chew­ing gum will not provide the agent with fur­ther bad news. Con­se­quently, an agent fol­low­ing EDT will chew in or­der to get the good news that they have de­creased their chance of get­ting ab­scesses.

Un­for­tu­nately, the tickle defense fails to achieve its aims. In in­tro­duc­ing this ap­proach, Eells hoped that EDT could be made to mimic CDT but with­out an allegedly in­el­e­gant re­li­ance on cau­sa­tion. How­ever, So­bel (1994, ch. 2) demon­strated that the tickle defense failed to en­sure that EDT and CDT would de­cide equiv­a­lently in all cases. On the other hand, those who feel that EDT origi­nally got it right by one-box­ing in New­comb’s prob­lem will be dis­ap­pointed to dis­cover that the tickle defense leads an agent to two-box in some ver­sions of New­comb’s prob­lem and so solves one prob­lem for the the­ory at the ex­pense of in­tro­duc­ing an­other.

So just as CDT “loses” on New­comb’s prob­lem, EDT will “lose” on Med­i­cal New­comb prob­lems (if the tickle defense fails) or will join CDT and “lose” on New­comb’s Prob­lem it­self (if the tickle defense suc­ceeds).

11.1.4. New­comb’s soda

There are also similar prob­le­matic cases for EDT where the ev­i­dence pro­vided by your de­ci­sion re­lates not to a fea­ture that you were born (or cre­ated) with but to some other fea­ture of the world. One such sce­nario is the New­comb’s soda prob­lem, in­tro­duced in Yud­kowsky (2010):

You know that you will shortly be ad­ministered one of two so­das in a dou­ble-blind clini­cal test. After drink­ing your as­signed soda, you will en­ter a room in which you find a choco­late ice cream and a vanilla ice cream. The first soda pro­duces a strong but en­tirely sub­con­scious de­sire for choco­late ice cream, and the sec­ond soda pro­duces a strong sub­con­scious de­sire for vanilla ice cream. By “sub­con­scious” I mean that you have no in­tro­spec­tive ac­cess to the change, any more than you can an­swer ques­tions about in­di­vi­d­ual neu­rons firing in your cere­bral cor­tex. You can only in­fer your changed tastes by ob­serv­ing which kind of ice cream you pick.

It so hap­pens that all par­ti­ci­pants in the study who test the Cho­co­late Soda are re­warded with a mil­lion dol­lars af­ter the study is over, while par­ti­ci­pants in the study who test the Vanilla Soda re­ceive noth­ing. But sub­jects who ac­tu­ally eat vanilla ice cream re­ceive an ad­di­tional thou­sand dol­lars, while sub­jects who ac­tu­ally eat choco­late ice cream re­ceive no ad­di­tional pay­ment. You can choose one and only one ice cream to eat. A pseudo-ran­dom al­gorithm as­signs so­das to ex­per­i­men­tal sub­jects, who are evenly di­vided (50/​50) be­tween Cho­co­late and Vanilla So­das. You are told that 90% of pre­vi­ous re­search sub­jects who chose choco­late ice cream did in fact drink the Cho­co­late Soda, while 90% of pre­vi­ous re­search sub­jects who chose vanilla ice cream did in fact drink the Vanilla Soda. Which ice cream would you eat?

Newcomb’s soda

New­comb’s soda

In this case, an EDT agent will de­cide to eat choco­late ice cream as this would provide ev­i­dence that they drank the choco­late soda and hence that they will re­ceive $1 mil­lion af­ter the ex­per­i­ment. How­ever, this seems to be the wrong de­ci­sion and so, once again, the EDT agent “loses”.

11.1.5. Bostrom’s meta-New­comb problem

In re­sponse to at­tacks on their the­ory, the pro­po­nent of EDT can pre­sent al­ter­na­tive sce­nar­ios where EDT “wins” and it is CDT that “loses”. One such case is the meta-New­comb prob­lem pro­posed in Bostrom (2001). Adapted to fit my ear­lier story about Omega the su­per­in­tel­li­gent ma­chine (sec­tion 11.1.1), the prob­lem runs like this: Either Omega has already placed $1M or noth­ing in box B (de­pend­ing on its pre­dic­tion about your choice), or else Omega is watch­ing as you choose and af­ter your choice it will place $1M into box B only if you have one-boxed. But you don’t know which is the case. Omega makes its move be­fore the hu­man player’s choice about half the time, and the rest of the time it makes its move af­ter the player’s choice.

But now sup­pose there is an­other su­per­in­tel­li­gent ma­chine, Meta-Omega, who has a perfect track record of pre­dict­ing both Omega’s choices and the choices of hu­man play­ers. Meta-Omega tells you that ei­ther you will two-box and Omega will “make its move” af­ter you make your choice, or else you will one-box and Omega has already made its move (and gone on to the next game, with some­one else).

Here, an EDT agent one-boxes and walks away with a mil­lion dol­lars. On the face of it, how­ever, a CDT agent faces a dilemma: if she two-boxes then Omega’s ac­tion de­pends on her choice, so the “ra­tio­nal” choice is to one-box. But if the CDT agent one-boxes, then Omega’s ac­tion tem­po­rally pre­cedes (and is thus phys­i­cally in­de­pen­dent of) her choice, so the “ra­tio­nal” ac­tion is to two-box. It might seem, then, that a CDT agent will be un­able to reach any de­ci­sion in this sce­nario. How­ever, fur­ther re­flec­tion re­veals that the is­sue is more com­pli­cated. Ac­cord­ing to CDT, what the agent ought to do in this sce­nario de­pends on their cre­dences about their own ac­tions. If they have a high cre­dence that they will two-box, they ought to one-box and if they have a high cre­dence that they will one-box, they ought to two box. Given that the agent’s cre­dences in their ac­tions are not given to us in the de­scrip­tion of the meta-New­comb prob­lem, the sce­nario is un­der­speci­fied and it is hard to know what con­clu­sions should be drawn from it.

11.1.6. The psy­chopath button

For­tu­nately, an­other case has been in­tro­duced where, ac­cord­ing to CDT, what an agent ought to do de­pends on their cre­dences about what they will do. This is the psy­chopath but­ton, in­tro­duced in Egan (2007):

Paul is de­bat­ing whether to press the “kill all psy­chopaths” but­ton. It would, he thinks, be much bet­ter to live in a world with no psy­chopaths. Un­for­tu­nately, Paul is quite con­fi­dent that only a psy­chopath would press such a but­ton. Paul very strongly prefers liv­ing in a world with psy­chopaths to dy­ing. Should Paul press the but­ton?

Many peo­ple think Paul should not. After all, if he does so, he is al­most cer­tainly a psy­chopath and so press­ing the but­ton will al­most cer­tainly cause his death. This is also the re­sponse that an EDT agent will give. After all, push­ing the but­ton would provide the agent with the bad news that they are al­most cer­tainly a psy­chopath and so will die as a re­sult of their ac­tion.

On the other hand, if Paul is fairly cer­tain that he is not a psy­chopath, then CDT will say that he ought to press the but­ton. CDT will note that, given Paul’s con­fi­dence that he isn’t a psy­chopath, his de­ci­sion will al­most cer­tainly have a pos­i­tive im­pact as it will re­sult in the death of all psy­chopaths and Paul’s sur­vival. On the face of it, then, a CDT agent would de­cide in­ap­pro­pri­ately in this case by push­ing the but­ton. Im­por­tantly, un­like in the meta-New­comb prob­lem, the agent’s cre­dences about their own be­hav­ior are speci­fied in Egan’s full ver­sion of this sce­nario (in non-nu­meric terms, the agent thinks they’re un­likely to be a psy­chopath and hence un­likely to press the but­ton).

How­ever, in or­der to pro­duce this prob­lem for CDT, Egan made a num­ber of as­sump­tions about how an agent should de­cide when what they ought to do de­pends on what they think they will do. In re­sponse, al­ter­na­tive views about de­cid­ing in such cases have been ad­vanced (par­tic­u­lar in Arntze­nius, 2008 and Joyce, 2012). Given these fac­tors, opinions are split about whether the psy­chopath but­ton prob­lem does in fact pose a challenge to CDT.

11.1.7. Parfit’s hitchhiker

Not all de­ci­sion sce­nar­ios are prob­le­matic for just one of EDT or CDT. There are also cases that can be pre­sented where both an EDT agent and a CDT agent will both “lose”. One such case is Parfit’s Hitch­hiker (Parfit, 1984, p. 7):

Sup­pose that I am driv­ing at mid­night through some desert. My car breaks down. You are a stranger, and the only other driver near. I man­age to stop you, and I offer you a great re­ward if you res­cue me. I can­not re­ward you now, but I promise to do so when we reach my home. Sup­pose next that I am trans­par­ent, un­able to de­ceive oth­ers. I can­not lie con­vinc­ingly. Either a blush, or my tone of voice, always gives me away. Sup­pose, fi­nally, that I know my­self to be never self-deny­ing. If you drive me to my home, it would be worse for me if I gave you the promised re­ward. Since I know that I never do what will be worse for me, I know that I shall break my promise. Given my in­abil­ity to lie con­vinc­ingly, you know this too. You do not be­lieve my promise, and there­fore leave me stranded in the desert.

In this sce­nario the agent “loses” if they would later re­fuse to give the stranger the re­ward. How­ever, both EDT agents and CDT agents will re­fuse to do so. After all, by this point the agent will already be safe so giv­ing the re­ward can nei­ther provide good news about, nor cause, their safety. So this seems to be a case where both the­o­ries “lose”.

11.1.8. Trans­par­ent New­comb’s problem

There are also other cases where both EDT and CDT “lose”. One of these is the Trans­par­ent New­comb’s prob­lem which, in at least one ver­sion, is due to Drescher (2006, p. 238-242). This sce­nario is like the origi­nal New­comb’s prob­lem but, in this case, both boxes are trans­par­ent so you can see their con­tents when you make your de­ci­sion. Again, Omega has filled box A with $1000 and Box B with ei­ther $1 mil­lion or noth­ing based on a pre­dic­tion of your be­hav­ior. Speci­fi­cally, Omega has pre­dicted how you would de­cide if you wit­nessed $1 mil­lion in Box B. If Omega pre­dicted that you would one-box in this case, he placed $1 mil­lion in Box B. On the other hand, if Omega pre­dicted that you would two-box in this case then he placed noth­ing in Box B.

Both EDT and CDT agents will two-box in this case. After all, the con­tents of the boxes are de­ter­mined and known so the agent’s de­ci­sion can nei­ther provide good news about what they con­tain nor cause them to con­tain some­thing de­sir­able. As with two-box­ing in the origi­nal ver­sion of New­comb’s prob­lem, many philoso­phers will en­dorse this be­hav­ior.

How­ever, it’s worth not­ing that Omega will al­most cer­tainly have pre­dicted this de­ci­sion and so filled Box B with noth­ing. CDT and EDT agents will end up with $1000. On the other hand, just as in the origi­nal case, the agent that one-boxes will end up with $1 mil­lion. So this is an­other case where both EDT and CDT “lose”. Con­se­quently, to those that agree with the ear­lier com­ments (in sec­tion 11.1.1) that a de­ci­sion the­ory shouldn’t lead an agent to “lose”, nei­ther of these the­o­ries will be satis­fac­tory.

11.1.9. Coun­ter­fac­tual mugging

Another similar case, known as coun­ter­fac­tual mug­ging, was de­vel­oped in Nesov (2009):

Imag­ine that one day, Omega comes to you and says that it has just tossed a fair coin, and given that the coin came up tails, it de­cided to ask you to give it $100. What­ever you do in this situ­a­tion, noth­ing else will hap­pen differ­ently in re­al­ity as a re­sult. Nat­u­rally you don’t want to give up your $100. But see, the Omega tells you that if the coin came up heads in­stead of tails, it’d give you $10000, but only if you’d agree to give it $100 if the coin came up tails.

Should you give up the $100?

Both CDT and EDT say no. After all, giv­ing up your money nei­ther pro­vides good news about nor in­fluences your chances of get­ting $10 000 out of the ex­change. Fur­ther, this in­tu­itively seems like the right de­ci­sion. On the face of it, then, it is ap­pro­pri­ate to re­tain your money in this case.

How­ever, pre­sum­ing you take Omega to be perfectly trust­wor­thy, there seems to be room to de­bate this con­clu­sion. If you are the sort of agent that gives up the $100 in coun­ter­fac­tual mug­ging then you will tend to do bet­ter than the sort of agent that won’t give up the $100. Of course, in the par­tic­u­lar case at hand you will lose but ra­tio­nal agents of­ten lose in spe­cific cases (as, for ex­am­ple, when such an agent loses a ra­tio­nal bet). It could be ar­gued that what a ra­tio­nal agent should not do is be the type of agent that loses. Given that agents that re­fuse to give up the $100 are the type of agent that loses, there seem to be grounds to claim that coun­ter­fac­tual mug­ging is an­other case where both CDT and EDT act in­ap­pro­pri­ately.

11.1.10. Pri­soner’s dilemma

Be­fore mov­ing on to a more de­tailed dis­cus­sion of var­i­ous pos­si­ble de­ci­sion the­o­ries, I’ll con­sider one fi­nal sce­nario: the pris­oner’s dilemma. Res­nik (1987, pp. 147-148 ) out­lines this sce­nario as fol­lows:

Two pris­on­ers...have been ar­rested for van­dal­ism and have been iso­lated from each other. There is suffi­cient ev­i­dence to con­vict them on the charge for which they have been ar­rested, but the pros­e­cu­tor is af­ter big­ger game. He thinks that they robbed a bank to­gether and that he can get them to con­fess to it. He sum­mons each sep­a­rately to an in­ter­ro­ga­tion room and speaks to each as fol­lows: “I am go­ing to offer the same deal to your part­ner, and I will give you each an hour to think it over be­fore I call you back. This is it: If one of you con­fesses to the bank rob­bery and the other does not, I will see to it that the con­fes­sor gets a one-year term and that the other guy gets a twenty-five year term. If you both con­fess, then it’s ten years apiece. If nei­ther of you con­fesses, then I can only get two years apiece on the van­dal­ism charge...”

The de­ci­sion ma­trix of each van­dal will be as fol­lows:

Part­ner con­fesses Part­ner lies
Con­fess 10 years in jail 1 year in jail
Lie 25 years in jail 2 years in jail

Faced with this sce­nario, a CDT agent will con­fess. After all, the agent’s de­ci­sion can’t in­fluence their part­ner’s de­ci­sion (they’ve been iso­lated from one an­other) and so the agent is bet­ter off con­fess­ing re­gard­less of what their part­ner chooses to do. Ac­cord­ing to the ma­jor­ity of de­ci­sion (and game) the­o­rists, con­fess­ing is in fact the ra­tio­nal de­ci­sion in this case.

De­spite this, how­ever, an EDT agent may lie in a pris­oner’s dilemma. Speci­fi­cally, if they think that their part­ner is similar enough to them, the agent will lie be­cause do­ing so will provide the good news that they will both lie and hence that they will both get two years in jail (good news as com­pared with the bad news that they will both con­fess and hence that they will get 10 years in jail).

To many peo­ple, there seems to be some­thing com­pel­ling about this line of rea­son­ing. For ex­am­ple, Dou­glas Hofs­tadter (1985, pp. 737-780) has ar­gued that an agent act­ing “su­per­ra­tionally” would co-op­er­ate with other su­per­ra­tional agents for pre­cisely this sort of rea­son: a su­per­ra­tional agent would take into ac­count the fact that other such agents will go through the same thought pro­cess in the pris­oner’s dilemma and so make the same de­ci­sion. As such, it is bet­ter that that the de­ci­sion that both agents reach be to lie than that it be to con­fess. More broadly, it could per­haps be ar­gued that a ra­tio­nal agent should lie in the pris­oner’s dilemma as long as they be­lieve that they are similar enough to their part­ner that they are likely to reach the same de­ci­sion.

An argument for cooperation in the prisoners’ dilemma

An ar­gu­ment for co­op­er­a­tion in the pris­on­ers’ dilemma

It is un­clear, then, pre­cisely what should be con­cluded from the pris­oner’s dilemma. How­ever, for those that are sym­pa­thetic to Hofs­tadter’s point or the line of rea­son­ing ap­pealed to by the EDT agent, the sce­nario seems to provide an ad­di­tional rea­son to seek out an al­ter­na­tive the­ory to CDT.

11.2. Bench­mark the­ory (BT)

One re­cent re­sponse to the ap­par­ent failure of EDT to de­cide ap­pro­pri­ately in med­i­cal New­comb prob­lems and CDT to de­cide ap­pro­pri­ately in the psy­chopath but­ton is Bench­mark The­ory (BT) which was de­vel­oped in Wedg­wood (2011) and dis­cussed fur­ther in Briggs (2010).

In English, we could think of this de­ci­sion al­gorithm as say­ing that agents should de­cide so as to give their fu­ture self good news about how well off they are com­pared to how well off they could have been. In for­mal terms, BT uses the fol­low­ing for­mula to calcu­late the ex­pected util­ity of an act, A:

BT expected value formula.

In other words, it uses the con­di­tional prob­a­bil­ity, as in EDT but calcu­lates the value differ­ently (as in­di­cated by the use of V’ rather than V). V’ is calcu­lated rel­a­tive to a bench­mark value in or­der to give a com­par­a­tive mea­sure of value (both of the above sources go into more de­tail about this pro­cess).

Tak­ing the in­for­mal per­spec­tive, in the chew­ing gum prob­lem, BT will note that by chew­ing gum, the agent will always get the good news that they are com­par­a­tively bet­ter off than they could have been (be­cause chew­ing gum helps con­trol throat ab­scesses) whereas by not chew­ing, the agent will always get the bad news that they could have been com­par­a­tively bet­ter off by chew­ing. As such, a BT agent will chew in this sce­nario.

Fur­ther, BT seems to reach what many con­sider to be the right de­ci­sion in the psy­chopath but­ton. In this case, the BT agent will note that if they push the but­ton they will get the bad news that they are al­most cer­tainly a psy­chopath and so that they would have been com­par­a­tively much bet­ter off by not push­ing (as push­ing will kill them). On the other hand, if they don’t push they will get the less bad news that they are al­most cer­tainly not a psy­chopath and so could have been com­par­a­tively a lit­tle bet­ter off it they had pushed the but­ton (as this would have kil­led all the psy­chopaths but not them). So re­frain­ing from push­ing the but­ton gives the less bad news and so is the ra­tio­nal de­ci­sion.

On the face of it, then, there seem to be strong rea­sons to find BT com­pel­ling: it de­cides ap­pro­pri­ately in these sce­nar­ios while, ac­cord­ing to some peo­ple, EDT and CDT only de­cide ap­pro­pri­ately in one or the other of them.

Un­for­tu­nately, a BT agent will fail to de­cide ap­pro­pri­ately in other sce­nar­ios. First, those that hold that one-box­ing is the ap­pro­pri­ate de­ci­sion in New­comb’s prob­lem will im­me­di­ately find a flaw in BT. After all, in this sce­nario two-box­ing gives the good news that the agent did com­par­a­tively bet­ter than they could have done (be­cause they gain the $1000 from Box A which is more than they would have re­ceived oth­er­wise) while one-box­ing brings the bad news that they did com­par­a­tively worse than they could have done (as they did not re­ceive this money). As such, a BT agent will two-box in New­comb’s prob­lem.

Fur­ther, Briggs (2010) ar­gues, though Wedg­wood (2011) de­nies, that BT suffers from other prob­lems. As such, even for those who sup­port two-box­ing in New­comb’s prob­lem, it could be ar­gued that BT doesn’t rep­re­sent an ad­e­quate the­ory of choice. It is un­clear, then, whether BT is a de­sir­able re­place­ment to al­ter­na­tive the­o­ries.

11.3. Time­less de­ci­sion the­ory (TDT)

Yud­kowsky (2010) offers an­other de­ci­sion al­gorithm, time­less de­ci­sion the­ory or TDT (see also Al­tair, 2013). Speci­fi­cally, TDT is in­tended as an ex­plicit re­sponse to the idea that a the­ory of ra­tio­nal choice should lead an agent to “win”. As such, it will ap­peal to those who think it is ap­pro­pri­ate to one-box in New­comb’s prob­lem and chew in the chew­ing gum prob­lem.

In English, this al­gorithm can be ap­prox­i­mated as say­ing that an agent ought to choose as if CDT were right but they were de­ter­min­ing not their ac­tual de­ci­sion but rather the re­sult of the ab­stract com­pu­ta­tion of which their de­ci­sion is one con­crete in­stance. For­mal­iz­ing this de­ci­sion al­gorithm would re­quire a sub­stan­tial doc­u­ment in its own right and so will not be car­ried out in full here. Briefly, how­ever, TDT is built on top of causal Bayesian net­works (Pearl, 2000) which are graphs where the ar­rows rep­re­sent causal in­fluence. TDT sup­ple­ments these graphs by adding nodes rep­re­sent­ing ab­stract com­pu­ta­tions and tak­ing the ab­stract com­pu­ta­tion that de­ter­mines an agent’s de­ci­sion to be the ob­ject of choice rather than the con­crete de­ci­sion it­self (see Yud­kowsky, 2010 for a more de­tailed de­scrip­tion).

Re­turn­ing to an in­for­mal dis­cus­sion, an ex­am­ple will help clar­ify the form taken by TDT: imag­ine that two perfect repli­cas of a per­son are placed in iden­ti­cal rooms and asked to make the same de­ci­sion. While each replica will make their own de­ci­sion, in do­ing so, they will be car­ry­ing out the same com­pu­ta­tional pro­cess. As such, TDT will say that the repli­cas ought to act as if they are de­ter­min­ing the re­sult of this pro­cess and hence as if they are de­cid­ing the be­hav­ior of both copies.

Some­thing similar can be said about New­comb’s prob­lem. In this case it is al­most like there is again a replica of the agent: Omega’s model of the agent that it used to pre­dict the agent’s be­hav­ior. Both the origi­nal agent and this “replica” re­sponds to the same ab­stract com­pu­ta­tional pro­cess as one an­other. In other words, both Omega’s pre­dic­tion and the agent’s be­hav­ior are in­fluenced by this pro­cess. As such, TDT ad­vises the agent to act as if they are de­ter­min­ing the re­sult of this pro­cess and, hence, as if they can de­ter­mine Omega’s box filling be­hav­ior. As such, a TDT agent will one-box in or­der to de­ter­mine the re­sult of this ab­stract com­pu­ta­tion in a way that leads to $1 mil­lion be­ing placed in Box B.

TDT also suc­ceeds in other ar­eas. For ex­am­ple, in the chew­ing gum prob­lem there is no “replica” agent so TDT will de­cide in line with stan­dard CDT and choose to chew gum. Fur­ther, in the pris­oner’s dilemma, a TDT agent will lie if its part­ner is an­other TDT agent (or a rele­vantly similar agent). After all, in this case both agents will carry out the same com­pu­ta­tional pro­cess and so TDT will ad­vise that the agent act as if they are de­ter­min­ing this pro­cess and hence si­mul­ta­neously de­ter­min­ing both their own and their part­ner’s de­ci­sion. If so then it is bet­ter for the agent that both of them lie than that both of them con­fess.

How­ever, de­spite its suc­cess, TDT also “loses” in some de­ci­sion sce­nar­ios. For ex­am­ple, in coun­ter­fac­tual mug­ging, a TDT agent will not choose to give up the $100. This might seem sur­pris­ing. After all, as with New­comb’s prob­lem, this case in­volves Omega pre­dict­ing the agent’s be­hav­ior and hence in­volves a “replica”. How­ever, this case differs in that the agent knows that the coin came up heads and so knows that they have noth­ing to gain by giv­ing up the money.

For those who feel that a the­ory of ra­tio­nal choice should lead an agent to “win”, then, TDT seems like a step in the right di­rec­tion but fur­ther work is re­quired if it is to “win” in the full range of de­ci­sion sce­nar­ios.

11.4. De­ci­sion the­ory and “win­ning”

In the pre­vi­ous sec­tion, I dis­cussed TDT, a de­ci­sion al­gorithm that could be ad­vanced as re­place­ments for CDT and EDT. One of the pri­mary mo­ti­va­tions for de­vel­op­ing TDT is a sense that both CDT and EDT fail to rea­son in a de­sir­able man­ner in some de­ci­sion sce­nar­ios. How­ever, de­spite ac­knowl­edg­ing that CDT agents end up worse off in New­comb’s Prob­lem, many (and per­haps the ma­jor­ity of) de­ci­sion the­o­rists are pro­po­nents of CDT. On the face of it, this may seem to sug­gest that these de­ci­sion the­o­rists aren’t in­ter­ested in de­vel­op­ing a de­ci­sion al­gorithm that “wins” but rather have some other aim in mind. If so then this might lead us to ques­tion the value of de­vel­op­ing one-box­ing de­ci­sion al­gorithms.

How­ever, the claim that most de­ci­sion the­o­rists don’t care about find­ing an al­gorithm that “wins” mischar­ac­ter­izes their po­si­tion. After all, pro­po­nents of CDT tend to take the challenge posed by the fact that CDT agents “lose” in New­comb’s prob­lem se­ri­ously (in the philo­soph­i­cal liter­a­ture, it’s of­ten referred to as the Why ain’cha rich? prob­lem). A com­mon re­ac­tion to this challenge is neatly sum­ma­rized in Joyce (1999, p. 153-154 ) as a re­sponse to a hy­po­thet­i­cal ques­tion about why, if two-box­ing is ra­tio­nal, the CDT agent does not end up as rich as an agent that one-boxes:

Rachel has a perfectly good an­swer to the “Why ain’t you rich?” ques­tion. “I am not rich,” she will say, “be­cause I am not the kind of per­son [Omega] thinks will re­fuse the money. I’m just not like you, Irene [the one-boxer]. Given that I know that I am the type who takes the money, and given that [Omega] knows that I am this type, it was rea­son­able of me to think that the $1,000,000 was not in [the box]. The $1,000 was the most I was go­ing to get no mat­ter what I did. So the only rea­son­able thing for me to do was to take it.”

Irene may want to press the point here by ask­ing, “But don’t you wish you were like me, Rachel?”… Rachel can and should ad­mit that she does wish she were more like Irene… At this point, Irene will ex­claim, “You’ve ad­mit­ted it! It wasn’t so smart to take the money af­ter all.” Un­for­tu­nately for Irene, her con­clu­sion does not fol­low from Rachel’s premise. Rachel will pa­tiently ex­plain that wish­ing to be a [one-boxer] in a New­comb prob­lem is not in­con­sis­tent with think­ing that one should take the $1,000 what­ever type one is. When Rachel wishes she was Irene’s type she is wish­ing for Irene’s op­tions, not sanc­tion­ing her choice… While a per­son who knows she will face (has faced) a New­comb prob­lem might wish that she were (had been) the type that [Omega] la­bels a [one-boxer], this wish does not provide a rea­son for be­ing a [one-boxer]. It might provide a rea­son to try (be­fore [the boxes are filled]) to change her type if she thinks this might af­fect [Omega’s] pre­dic­tion, but it gives her no rea­son for do­ing any­thing other than tak­ing the money once she comes to be­lieves that she will be un­able to in­fluence what [Omega] does.

In other words, this re­sponse dis­t­in­guishes be­tween the win­ning de­ci­sion and the win­ning type of agent and claims that two-box­ing is the win­ning de­ci­sion in New­comb’s prob­lem (even if one-box­ers are the win­ning type of agent). Con­se­quently, in­so­far as de­ci­sion the­ory is about de­ter­min­ing which de­ci­sion is ra­tio­nal, on this ac­count CDT rea­sons cor­rectly in New­comb’s prob­lem.

For those that find this re­sponse per­plex­ing, an anal­ogy could be drawn to the chew­ing gum prob­lem. In this sce­nario, there is near unan­i­mous agree­ment that the ra­tio­nal de­ci­sion is to chew gum. How­ever, statis­ti­cally, non-chew­ers will be bet­ter off than chew­ers. As such, the non-chewer could ask, “if you’re so smart, why aren’t you healthy?” In this case, the above re­sponse seems par­tic­u­larly ap­pro­pri­ate. The chew­ers are less healthy not be­cause of their de­ci­sion but rather be­cause they’re more likely to have an un­de­sir­able gene. Hav­ing good genes doesn’t make the non-chewer more ra­tio­nal but sim­ply more lucky. The pro­po­nent of CDT sim­ply makes a similar re­sponse to New­comb’s prob­lem: one-box­ers aren’t richer be­cause of their de­ci­sion but rather be­cause of the type of agent that they were when the boxes were filled.

One fi­nal point about this re­sponse is worth not­ing. A pro­po­nent of CDT can ac­cept the above ar­gu­ment but still ac­knowl­edge that, if given the choice be­fore the boxes are filled, they would be ra­tio­nal to choose to mod­ify them­selves to be a one-box­ing type of agent (as Joyce ac­knowl­edged in the above pas­sage and as ar­gued for in Burgess, 2004). To the pro­po­nent of CDT, this is un­prob­le­matic: if we are some­times re­warded not for the ra­tio­nal­ity of our de­ci­sions in the mo­ment but for the type of agent we were at some past mo­ment then it should be un­sur­pris­ing that chang­ing to a differ­ent type of agent might be benefi­cial.

The re­sponse to this defense of two-box­ing in New­comb’s prob­lem has been di­vided. Many find it com­pel­ling but oth­ers, like Ahmed and Price (2012) think it does not ad­e­quately ad­dress to the challenge:

It is no use the causal­ist’s whin­ing that fore­see­ably, New­comb prob­lems do in fact re­ward ir­ra­tional­ity, or rather CDT-ir­ra­tional­ity. The point of the ar­gu­ment is that if ev­ery­one knows that the CDT-ir­ra­tional strat­egy will in fact do bet­ter on av­er­age than the CDT-ra­tio­nal strat­egy, then it’s ra­tio­nal to play the CDT-ir­ra­tional strat­egy.

Given this, there seem to be two po­si­tions one could take on these is­sues. If the re­sponse given by the pro­po­nent of CDT is com­pel­ling, then we should be at­tempt­ing to de­velop a de­ci­sion the­ory that two-boxes on New­comb’s prob­lem. Per­haps the best the­ory for this role is CDT but per­haps it is in­stead BT, which many peo­ple think rea­sons bet­ter in the psy­chopath but­ton sce­nario. On the other hand, if the re­sponse given by the pro­po­nents of CDT is not com­pel­ling, then we should be de­vel­op­ing a the­ory that one-boxes in New­comb’s prob­lem. In this case, TDT, or some­thing like it, seems like the most promis­ing the­ory cur­rently on offer.