A Critique of Functional Decision Theory

A Cri­tique of Func­tional De­ci­sion Theory

NB: My writ­ing this note was prompted by Carl Shul­man, who sug­gested we could try a low-time-com­mit­ment way of at­tempt­ing to un­der­stand­ing the dis­agree­ment be­tween some folks in the ra­tio­nal­ity com­mu­nity and aca­demic de­ci­sion the­o­rists (in­clud­ing my­self, though I’m not much of a de­ci­sion the­o­rist). Apolo­gies that it’s slop­pier than I’d usu­ally aim for in a philos­o­phy pa­per, and lack­ing in ap­pro­pri­ate refer­ences. And, even though the pa­per is pretty nega­tive about FDT, I want to em­pha­sise that my writ­ing this should be taken as a sign of re­spect for those in­volved in de­vel­op­ing FDT. I’ll also caveat I’m un­likely to have time to en­gage in the com­ments; I thought it was bet­ter to get this out there all the same rather than de­lay pub­li­ca­tion fur­ther.


  1. Introduction

There’s a long-run­ning is­sue where many in the ra­tio­nal­ity com­mu­nity take func­tional de­ci­sion the­ory (and its var­i­ants) very se­ri­ously, but the aca­demic de­ci­sion the­ory com­mu­nity does not. But there’s been lit­tle pub­lic dis­cus­sion of FDT from aca­demic de­ci­sion the­o­rists (one ex­cep­tion is here); this note at­tempts to partly ad­dress this gap.

So that there’s a clear ob­ject of dis­cus­sion, I’m go­ing to fo­cus on Yud­kowsky and Soares’ ‘Func­tional De­ci­sion The­ory’ (which I’ll re­fer to as Y&S), though I also read a re­vised ver­sion of Soares and Lev­in­stein’s Cheat­ing Death in Da­m­as­cus.

This note is struc­tured as fol­lows. Sec­tion II de­scribes causal de­ci­sion the­ory (CDT), ev­i­den­tial de­ci­sion the­ory (EDT) and func­tional de­ci­sion the­ory (FDT). Sec­tions III-VI de­scribe prob­lems for FDT: (i) that it some­times makes bizarre recom­men­da­tions, recom­mend­ing an op­tion that is cer­tainly lower-util­ity than an­other op­tion; (ii) that it fails to one-box in most in­stances of New­comb’s prob­lem, even though the cor­rect­ness of one-box­ing is sup­posed to be one of the guid­ing mo­ti­va­tions for the the­ory; (iii) that it re­sults in im­plau­si­ble dis­con­ti­nu­ities, where what is ra­tio­nal to do can de­pend on ar­bi­trar­ily small changes to the world; and (iv) that, be­cause there’s no real fact of the mat­ter about whether a par­tic­u­lar phys­i­cal pro­cess im­ple­ments a par­tic­u­lar al­gorithm, it’s deeply in­de­ter­mi­nate what FDT’s im­pli­ca­tions are. In sec­tion VII I dis­cuss the idea that FDT ‘does bet­ter at get­ting util­ity’ than EDT or CDT; I ar­gue that Y&S’s claims to this effect are un­helpfully vague, and on any more pre­cise way of un­der­stand­ing their claim, aren’t plau­si­ble. In sec­tion VIII I briefly de­scribe a view that cap­tures some of the mo­ti­va­tion be­hind FDT, and in my view is more plau­si­ble. I con­clude that FDT faces a num­ber of deep prob­lems and lit­tle to say in its favour.

In what fol­lows, I’m go­ing to as­sume a rea­son­able amount of fa­mil­iar­ity with the de­bate around New­comb’s prob­lem.


II. CDT, EDT and FDT

In­for­mally: CDT, EDT and FDT differ in what non-causal cor­re­la­tions they care about when eval­u­at­ing a de­ci­sion. For CDT, what you cause to hap­pen is all that mat­ters; if your ac­tion cor­re­lates with some good out­come, that’s nice to know, but it’s not rele­vant to what you ought to do. For EDT, all cor­re­la­tions mat­ter: you should pick what­ever ac­tion will re­sult in you be­liev­ing you will have the high­est ex­pected util­ity. For FDT, only some non-causal cor­re­la­tions mat­ter, namely only those cor­re­la­tions be­tween your ac­tion and events el­se­where in time and space that would be differ­ent in the (log­i­cally im­pos­si­ble) wor­lds in which the out­put of the al­gorithm you’re run­ning is differ­ent. Other than for those cor­re­la­tions, FDT be­haves in the same way as CDT.

For­mally, where ​S​ rep­re­sents states of na­ture, ​A, B​ etc rep­re­sent acts, ​P ​is a prob­a­bil­ity func­tion, and rep­re­sents the util­ity the agent gains from the out­come of choos­ing ​A​ given state , ​and ‘≽’ rep­re­sents the ‘at least as choice­wor­thy as’ re­la­tion:

On EDT:

Where ‘|’ rep­re­sents con­di­tional prob­a­bil­ity.

On CDT:

Where ‘∖’ is a ‘causal prob­a­bil­ity func­tion’ that rep­re­sents the de­ci­sion-maker’s judg­ments about her abil­ity to causally in­fluence the events in the world by do­ing a par­tic­u­lar ac­tion. Most of­ten, this is in­ter­preted in coun­ter­fac­tual terms (so P (SA) rep­re­sents some­thing like the prob­a­bil­ity of ​S​ com­ing about were I to choose ​A​) but it needn’t be.

On FDT:

Where I in­tro­duce the op­er­a­tor “ † ” to rep­re­sent the spe­cial sort of func­tion that Yud­kowsky and Soares pro­pose, where P (SA) rep­re­sents the prob­a­bil­ity of ​S oc­cur­ring were the out­put of the al­gorithm that the de­ci­sion-maker is run­ning, in this de­ci­sion situ­a­tion, to be A. (I’m not claiming that it’s clear what this means. E.g. see­here​, sec­ond bul­let point, ar­gu­ing there can be no such prob­a­bil­ity func­tion, be­cause any prob­a­bil­ity func­tion re­quires cer­tainty in log­i­cal facts and all their en­tail­ments. I also note that strictly speak­ing FDT doesn’t as­sess acts in the same sense that CDT as­sesses acts; rather it as­sesses al­gorith­mic out­puts, and that Y&S have a slightly differ­ent for­mal set up than the one I de­scribe above. I don’t think this will mat­ter for the pur­poses of this note, though.)

With these defi­ni­tions on board, we can turn to ob­jec­tions to FDT.


III. FDT some­times makes bizarre recommendations

The crite­rion that Y&S re­gard as most im­por­tant in as­sess­ing a de­ci­sion the­ory is ‘amount of util­ity achieved’. I think that this idea is im­por­tantly un­der­speci­fied (which I dis­cuss more in sec­tion VII), but I agree with the spirit of it. But FDT does very poorly by that crite­rion, on any pre­cisifi­ca­tion of it.

In par­tic­u­lar, con­sider the fol­low­ing prin­ci­ple:

Guaran­teed Pay­offs: In con­di­tions of cer­tainty — that is, when the de­ci­sion-maker has no un­cer­tainty about what state of na­ture she is in, and no un­cer­tainty about the util­ity pay­off of each ac­tion is — the de­ci­sion-maker should choose the ac­tion that max­imises util­ity.

That is: for situ­a­tions where there’s no un­cer­tainty, we don’t need to ap­peal to ex­pected util­ity the­ory in any form to work out what we ought to do. You just ought to do what­ever will give you the high­est util­ity pay­off. This should be a con­straint on any plau­si­ble de­ci­sion the­ory. But FDT vi­o­lates that prin­ci­ple.

Con­sider the fol­low­ing case:

Bomb.
You face two open boxes, Left and Right, and you must take one of them. In the Left box, there is a live bomb; tak­ing this box will set off the bomb, set­ting you ablaze, and you cer­tainly will burn slowly to death. The Right box is empty, but you have to pay $100 in or­der to be able to take it.
A long-dead pre­dic­tor pre­dicted whether you would choose Left or Right, by run­ning a simu­la­tion of you and see­ing what that simu­la­tion did. If the pre­dic­tor pre­dicted that you would choose Right, then she put a bomb in Left. If the pre­dic­tor pre­dicted that you would choose Left, then she did not put a bomb in Left, and the box is empty.
The pre­dic­tor has a failure rate of only 1 in a trillion trillion. Helpfully, she left a note, ex­plain­ing that she pre­dicted that you would take Right, and there­fore she put the bomb in Left.
You are the only per­son left in the uni­verse. You have a happy life, but you know that you will never meet an­other agent again, nor face an­other situ­a­tion where any of your ac­tions will have been pre­dicted by an­other agent. What box should you choose?

The right ac­tion, ac­cord­ing to FDT, is to take Left, in the full knowl­edge that as a re­sult you will slowly burn to death. Why? Be­cause, us­ing Y&S’s coun­ter­fac­tu­als, if your al­gorithm were to out­put ‘Left’, then it would also have out­putted ‘Left’ when the pre­dic­tor made the simu­la­tion of you, and there would be no bomb in the box, and you could save your­self $100 by tak­ing Left. In con­trast, the right ac­tion on CDT or EDT is to take Right.

The recom­men­da­tion is im­plau­si­ble enough. But if we stipu­late that in this de­ci­sion-situ­a­tion the de­ci­sion-maker is cer­tain in the out­come that her ac­tions would bring about, we see that FDT vi­o­lates Guaran­teed Pay­offs.

(One might protest that no good Bayesian would ever have cre­dence 1 in an em­piri­cal propo­si­tion. But, first, that de­pends on what we could as ‘ev­i­dence’ — if a propo­si­tion is part of your ev­i­dence base, you have cre­dence 1 in it. And, sec­ond, we could con­struct very similar prin­ci­ples to Guaran­teed Pay­offs that don’t rely on the idea of cer­tainty, but on ap­prox­i­ma­tions to cer­tainty.)

Note that FDT’s recom­men­da­tion in this case is much more im­plau­si­ble than even the worst of the prima fa­cie im­plau­si­ble recom­men­da­tions of EDT or CDT. So, if we’re go­ing by ap­peal to cases, or by ‘who gets more util­ity’, FDT is look­ing very un­mo­ti­vated.


IV. FDT fails to get the an­swer Y&S want in most in­stances of the core ex­am­ple that’s sup­posed to mo­ti­vate it

On FDT, you con­sider what things would look like in the clos­est (log­i­cally im­pos­si­ble) world in which the al­gorithm you are run­ning were to pro­duce a differ­ent out­put than what it in fact does. Be­cause, so the ar­gu­ment goes, in New­comb prob­lems the pre­dic­tor is also run­ning your al­gorithm, or a ‘suffi­ciently similar’ al­gorithm, or a rep­re­sen­ta­tion of your al­gorithm, you con­sider the cor­re­la­tion be­tween your ac­tion and the pre­dic­tor’s pre­dic­tion (even though you don’t con­sider other sorts of cor­re­la­tions.)

How­ever, the pre­dic­tor needn’t be run­ning your al­gorithm, or have any­thing like a rep­re­sen­ta­tion of that al­gorithm, in or­der to pre­dict whether you’ll one box or two-box. Per­haps the Scots tend to one-box, whereas the English tend to two-box. Per­haps the pre­dic­tor knows how you’ve acted prior to that de­ci­sion. Per­haps the Pre­dic­tor painted the trans­par­ent box green, and knows that’s your favourite colour and you’ll strug­gle not to pick it up. In none of these in­stances is the Pre­dic­tor plau­si­bly do­ing any­thing like run­ning the al­gorithm that you’re run­ning when you make your de­ci­sion. But they are still able to pre­dict what you’ll do. (And bear in mind that the Pre­dic­tor doesn’t even need to be very re­li­able. As long as the Pre­dic­tor is bet­ter than chance, a New­comb prob­lem can be cre­ated.)

In fact, on the vast ma­jor­ity of ways that the Pre­dic­tor could pre­dict­ing your be­hav­ior, she isn’t run­ning the al­gorithm that you are run­ning, or rep­re­sent­ing it. But if the Pre­dic­tor isn’t run­ning the al­gorithm that you are run­ning, or rep­re­sent­ing it, then, on the most nat­u­ral in­ter­pre­ta­tion, FDT will treat this as ‘mere statis­ti­cal cor­re­la­tion’, and there­fore act like CDT. So, in the vast ma­jor­ity of New­comb cases, FDT would recom­mend two-box­ing. But the in­tu­ition in favour of one-box­ing in New­comb cases was ex­actly what was sup­posed to mo­ti­vate FDT in the first place.

Could we in­stead in­ter­pret FDT, such that it doesn’t have to re­quire the Pre­dic­tor to be run­ning the ex­act al­gorithm — some similar al­gorithm would do? But I’m not sure how that would help: in the ex­am­ples given above, the Pre­dic­tor’s pre­dic­tions aren’t based on any­thing like run­ning your al­gorithm. In fact, the pre­dic­tor may know very lit­tle about you, per­haps only whether you’re English or Scot­tish.

One could sug­gest that, even though the Pre­dic­tor is not run­ning a suffi­ciently similar al­gorithm to you, nonethe­less the Pre­dic­tor’s pre­dic­tion is sub­junc­tively de­pen­dent on your de­ci­sion (in the Y&S sense of ‘sub­junc­tive’). But, with­out any ac­count of Y&S’s no­tion of sub­junc­tive coun­ter­fac­tu­als, we just have no way of as­sess­ing whether that’s true or not. Y&S note that spec­i­fy­ing an ac­count of their no­tion of coun­ter­fac­tu­als is an ‘open prob­lem,’ but the prob­lem is much deeper than that. Without such an ac­count, it be­comes com­pletely in­de­ter­mi­nate what fol­lows from FDT, even in the core ex­am­ples that are sup­posed to mo­ti­vate it — and that makes FDT not a new de­ci­sion the­ory so much as a promis­sory note.

In­deed, on the most plau­si­ble ways of cash­ing this out, it doesn’t give the con­clu­sions that Y&S would want. If I imag­ine the clos­est world in which 6288 + 1048 = 7336 is false (Y&S’s ex­am­ple), I imag­ine a world with laws of na­ture rad­i­cally un­like ours — be­cause the laws of na­ture rely, fun­da­men­tally, on the truths of math­e­mat­ics, and if one math­e­mat­i­cal truth is false then ei­ther (i) math­e­mat­ics as a whole must be rad­i­cally differ­ent, or (ii) all math­e­mat­i­cal propo­si­tions are true be­cause it is sim­ple to prove a con­tra­dic­tion and ev­ery propo­si­tion fol­lows from a con­tra­dic­tion. Either way, when I imag­ine wor­lds in which FDT out­puts some­thing differ­ent than it in fact does, then I imag­ine val­ue­less wor­lds (no atoms or elec­trons, etc) — and this isn’t what Y&S are want­ing us to imag­ine.

Alter­na­tively (as Abram Dem­ski sug­gested to me in a com­ment), Y&S could ac­cept that the de­ci­sion-maker should two-box in the cases given above. But then, it seems to me, that FDT has lost much of its ini­tial mo­ti­va­tion: the case for one-box­ing in New­comb’s prob­lem didn’t seem to stem from whether the Pre­dic­tor was run­ning a simu­la­tion of me, or just us­ing some other way to pre­dict what I’d do.


V. Im­plau­si­ble discontinuities

A re­lated prob­lem is as fol­lows: FDT treats ‘mere statis­ti­cal reg­u­lar­i­ties’ very differ­ently from pre­dic­tions. But there’s no sharp line be­tween the two. So it will re­sult in im­plau­si­ble dis­con­ti­nu­ities. There are two ways we can see this.

First, take some phys­i­cal pro­cesses S (like the le­sion from the Smok­ing Le­sion) that causes a ‘mere statis­ti­cal reg­u­lar­ity’ (it’s not a Pre­dic­tor). And sup­pose that the ex­is­tence of S tends to cause both (i) one-box­ing ten­den­cies and (ii) whether there’s money in the opaque box or not when de­ci­sion-mak­ers face New­comb prob­lems. If it’s S alone that re­sults in the New­comb set-up, then FDT will recom­mend­ing two-box­ing.

But now sup­pose that the path­way by which S causes there to be money in the opaque box or not is that an­other agent looks at S and, if the agent sees that S will cause de­ci­sion-maker X to be a one-boxer, then the agent puts money in X’s opaque box. Now, be­cause there’s an agent mak­ing pre­dic­tions, the FDT ad­her­ent will pre­sum­ably want to say that the right ac­tion is one-box­ing. But this seems ar­bi­trary — why should the fact that S’s causal in­fluence on whether there’s money in the opaque box or not go via an­other agent much such a big differ­ence? And we can think of all sorts of spec­trum cases in be­tween the ‘mere statis­ti­cal reg­u­lar­ity’ and the full-blooded Pre­dic­tor: What if the ‘pre­dic­tor’ is a very un­so­phis­ti­cated agent that doesn’t even un­der­stand the im­pli­ca­tions of what they’re do­ing? What if they only par­tially un­der­stand the im­pli­ca­tions of what they’re do­ing? For FDT, there will be some point of so­phis­ti­ca­tion at which the agent moves from sim­ply be­ing a con­duit for a causal pro­cess to in­stan­ti­at­ing the right sort of al­gorithm, and sud­denly FDT will switch from recom­mend­ing two-box­ing to recom­mend­ing one-box­ing.

Se­cond, con­sider that same phys­i­cal pro­cess S, and con­sider a se­quence of New­comb cases, each of which grad­u­ally make S more and more com­pli­cated and agent-y, mak­ing it pro­gres­sively more similar to a Pre­dic­tor mak­ing pre­dic­tions. At some point, on FDT, there will be a point at which there’s a sharp jump; prior to that point in the se­quence, FDT would recom­mend that the de­ci­sion-maker two-boxes; af­ter that point, FDT would recom­mend that the de­ci­sion-maker one-boxes. But it’s very im­plau­si­ble that there’s some S such that a tiny change in its phys­i­cal makeup should af­fect whether one ought to one-box or two-box.


VI. FDT is deeply indeterminate

Even putting the pre­vi­ous is­sues aside, there’s a fun­da­men­tal way in which FDT is in­de­ter­mi­nate, which is that there’s no ob­jec­tive fact of the mat­ter about whether two phys­i­cal pro­cesses A and B are run­ning the same al­gorithm or not, and there­fore no ob­jec­tive fact of the mat­ter of which cor­re­la­tions rep­re­sent im­ple­men­ta­tions of the same al­gorithm or are ‘mere cor­re­la­tions’ of the form that FDT wants to ig­nore. (Though I’ll fo­cus on ‘same al­gorithm’ cases, I be­lieve that the same prob­lem would af­fect ac­counts of when two phys­i­cal pro­cesses are run­ning similar al­gorithms, or any way of ex­plain­ing when the out­put of some phys­i­cal pro­cess, which in­stan­ti­ates a par­tic­u­lar al­gorithm, is Y&S-sub­junc­tively de­pen­dent on the out­put of an­other phys­i­cal pro­cess, which in­stan­ti­ates a differ­ent al­gorithm.)

To see this, con­sider two calcu­la­tors. The first calcu­la­tor is like calcu­la­tors we are used to. The sec­ond calcu­la­tor is from a for­eign land: it’s iden­ti­cal ex­cept that the num­bers it out­puts always come with a nega­tive sign (‘–’) in front of them when you’d ex­pect there to be none, and no nega­tive sign when you ex­pect there to be one. Are these calcu­la­tors run­ning the same al­gorithm or not? Well, per­haps on this for­eign calcu­la­tor the ‘–’ sym­bol means what we usu­ally take it to mean — namely, that the en­su­ing num­ber is nega­tive — and there­fore ev­ery time we hit the ‘=’ but­ton on the sec­ond calcu­la­tor we are ask­ing it to run the al­gorithm ‘com­pute the sum en­tered, then out­put the nega­tive of the an­swer’. If so, then the calcu­la­tors are sys­tem­at­i­cally run­ning differ­ent al­gorithms.

But per­haps, in this for­eign land, the ‘–’ sym­bol, in this con­text, means that the en­su­ing num­ber is pos­i­tive and the lack of a ‘–’ sym­bol means that the num­ber is nega­tive. If so, then the calcu­la­tors are run­ning ex­actly the same al­gorithms; their differ­ences are merely no­ta­tional.

Ul­ti­mately, in my view, all we have, in these two calcu­la­tors, are just two phys­i­cal pro­cesses. The fur­ther ques­tion of whether they are run­ning the same al­gorithm or not de­pends on how we in­ter­pret the phys­i­cal out­puts of the calcu­la­tor. There is no deeper fact about whether they’re ‘re­ally’ run­ning the same al­gorithm or not. And in gen­eral, it seems to me, there’s no fact of the mat­ter about which al­gorithm a phys­i­cal pro­cess is im­ple­ment­ing in the ab­sence of a par­tic­u­lar in­ter­pre­ta­tion of the in­puts and out­puts of that phys­i­cal pro­cess.

But if that’s true, then, even in the New­comb cases where a Pre­dic­tor is simu­lat­ing you, it’s a mat­ter of choice of sym­bol-in­ter­pre­ta­tion whether the pre­dic­tor ran the same al­gorithm that you are now run­ning (or a rep­re­sen­ta­tion of that same al­gorithm). And the way you choose that sym­bol-in­ter­pre­ta­tion is fun­da­men­tally ar­bi­trary. So there’s no real fact of the mat­ter about whether the pre­dic­tor is run­ning the same al­gorithm as you. It’s in­de­ter­mi­nate how you should act, given FDT: you should one-box, given one way of in­ter­pret­ing the in­puts and out­puts of the phys­i­cal pro­cess the Pre­dic­tor is run­ning, but two-box given an al­ter­na­tive in­ter­pre­ta­tion.

Now, there’s a bunch of in­ter­est­ing work on con­crete com­pu­ta­tion, try­ing to give an ac­count of when two phys­i­cal pro­cesses are perform­ing the same com­pu­ta­tion. The best re­sponse that Y&S could to make this prob­lem is to provide a com­pel­ling ac­count of when two phys­i­cal pro­cesses are run­ning the same al­gorithm that gives them the an­swers they want. But al­most all ac­counts of com­pu­ta­tion in phys­i­cal pro­cesses have the is­sue that very many phys­i­cal pro­cesses are run­ning very many differ­ent al­gorithms, all at the same time. (Be­cause most ac­counts rely on there be­ing some map­ping from phys­i­cal states to com­pu­ta­tional states, and there can be mul­ti­ple map­pings.) So you might well end up with the prob­lem that in the clos­est (log­i­cally im­pos­si­ble) world in which FDT out­puts some­thing other than what it does out­put, not only do the ac­tions of the Pre­dic­tor change, but so do many other as­pects of the world. For ex­am­ple, if the phys­i­cal pro­cess un­der­ly­ing some as­pect of the US econ­omy just hap­pened to be iso­mor­phic with FDT’s al­gorithm, then in the log­i­cally im­pos­si­ble world where FDT out­puts a differ­ent al­gorithm, not only does the pre­dic­tor act differ­ently, but so does the US econ­omy. And that will prob­a­bly change the value of the world un­der con­sid­er­a­tion, in a way that’s clearly ir­rele­vant to the choice at hand.


VII. But FDT gets the most util­ity!

Y&S re­gard the most im­por­tant crite­rion to be ‘util­ity achieved’, and thinks that FDT does bet­ter than all its ri­vals in this re­gard. Though I agree with some­thing like the spirit of this crite­rion, its use by Y&S is un­helpfully am­bigu­ous. To help ex­plain this, I’ll go on a lit­tle de­tour to pre­sent some dis­tinc­tions that are com­monly used by aca­demic moral philoso­phers and, to a lesser ex­tent, de­ci­sion the­o­rists. (For more on these dis­tinc­tions, see Toby Ord’s DPhil the­sis.)

Eval­u­a­tive fo­cal points

An eval­u­a­tive fo­cal point is an ob­ject of ax­iolog­i­cal or nor­ma­tive eval­u­a­tion. (‘Ax­iolog­i­cal’ means ‘about good­ness/​bad­ness’; ‘nor­ma­tive’ means ‘about right­ness/​wrong­ness’. If you’re a con­se­quen­tial­ist, x is best iff it’s right, but if you’re a non-con­se­quen­tial­ist the two can come apart.) When do­ing moral philos­o­phy or de­ci­sion the­ory, the most com­mon eval­u­a­tive fo­cal points are acts, but we can eval­u­ate other things too: char­ac­ters, mo­tives, dis­po­si­tions, sets of rules, be­liefs, and so on.

Any ax­iolog­i­cal or nor­ma­tive the­ory needs to spec­ify which fo­cal point it is eval­u­at­ing. The the­ory can eval­u­ate a sin­gle fo­cal point (e.g. act util­i­tar­i­anism, which only eval­u­ates acts) or many (e.g. global util­i­tar­i­anism, which eval­u­ates ev­ery­thing).

The the­ory can also differ on whether it is di­rect or in­di­rect with re­spect to a given eval­u­a­tive fo­cal point. For ex­am­ple, Hooker’s rule-con­se­quen­tial­ism is a di­rect the­ory with re­spect to sets of rules, and an in­di­rect the­ory with re­spect to acts: it eval­u­ates sets of rules on the ba­sis of their con­se­quences, but eval­u­ates acts with re­spect to how they con­form to those sets of rules. Be­cause of this, on Hooker’s view, the right act need not max­i­mize good con­se­quences.

Cri­te­rion of right­ness vs de­ci­sion procedure

In chess, there’s a stan­dard by which it is judged who has won the game, namely, the win­ner is who­ever first puts their op­po­nent’s king into check­mate. But rely­ing solely on that stan­dard of eval­u­a­tion isn’t go­ing to go very well if you ac­tu­ally want to win at chess. In­stead, you should act ac­cord­ing to some other set of rules and heuris­tics, such as: “if white, play e4 on the first move,” “don’t get your Queen out too early,” “rooks are worth more than bishops” etc.

A similar dis­tinc­tion can be made for ax­iolog­i­cal or nor­ma­tive the­o­ries. The crite­rion of right­ness, for act util­i­tar­i­anism, is, “The right ac­tions are those ac­tions which max­i­mize the sum to­tal of wellbe­ing.” But that’s not the de­ci­sion pro­ce­dure one ought to fol­low. In­stead, per­haps, you should rely on rules like ‘al­most never lie’, ‘be kind to your friends and fam­ily’, ‘figure out how much you can sus­tain­ably donate to effec­tive char­i­ties, and do that,’ and so on.

For some peo­ple, in fact, learn­ing that util­i­tar­i­anism is true will cause one to be a worse util­i­tar­ian by the util­i­tar­ian’s crite­rion of right­ness! (Per­haps you start to come across as some­one who uses oth­ers as means to an end, and that hin­ders your abil­ity to do good.) By the util­i­tar­ian crite­rion of right­ness, some­one could in prin­ci­ple act rightly in ev­ery de­ci­sion, even though they have never heard of util­i­tar­i­anism, and there­fore never ex­plic­itly tried to fol­low util­i­tar­i­anism.

Th­ese dis­tinc­tions and FDT

From Y&S, it wasn’t clear to me whether FDT is re­ally meant to as­sess acts, agents, char­ac­ters, de­ci­sion pro­ce­dures, or out­puts of de­ci­sion pro­ce­dures, and it wasn’t clear to me whether it is meant to be a di­rect or an in­di­rect the­ory with re­spect to acts, or with re­spect to out­puts of de­ci­sion pro­ce­dures. This is cru­cial, be­cause it’s rele­vant to which de­ci­sion the­ory ‘does best at get­ting util­ity’.

With these dis­tinc­tions in hand, we can see that Y&S em­ploy mul­ti­ple dis­tinct in­ter­pre­ta­tions of their key crite­rion. Some­times, for ex­am­ple, Y&S talk about how “FDT agents” (which I in­ter­pret as ‘agents who fol­low FDT to make de­ci­sions’) get more util­ity, e.g.:

  • “Us­ing one sim­ple and co­her­ent de­ci­sion rule, func­tional de­ci­sion the­o­rists (for ex­am­ple) achieve more util­ity than CDT on New­comb’s prob­lem, more util­ity than EDT on the smok­ing le­sion prob­lem, and more util­ity than both in Parfit’s hitch­hiker prob­lem.”

  • “We pro­pose an en­tirely new de­ci­sion the­ory, func­tional de­ci­sion the­ory (FDT), that max­i­mizes agents’ util­ity more re­li­ably than CDT or EDT.”

  • “FDT agents at­tain high util­ity in a host of de­ci­sion prob­lems that have his­tor­i­cally proven challeng­ing to CDT and EDT: FDT out­performs CDT in New­comb’s prob­lem; EDT in the smok­ing le­sion prob­lem; and both in Parfit’s hitch­hiker prob­lem.”

  • “It should come as no sur­prise that an agent can out­perform both CDT and EDT as mea­sured by util­ity achieved; this has been known for some time (Gib­bard and Harper 1978).”

  • “Ex­pand­ing on the fi­nal ar­gu­ment, pro­po­nents of EDT, CDT, and FDT can al­l
    a­gree that it would be great news to hear that a be­loved daugh­ter ad­heres to FDT, be­cause FDT agents get more of what they want out of life. Would it not then be strange if the cor­rect the­ory of ra­tio­nal­ity were some al­ter­na­tive to the the­ory that pro­duces the best out­comes, as mea­sured in util­ity? (Imag­ine hid­ing de­ci­sion the­ory text­books from loved ones, lest they be per­suaded to adopt the “cor­rect” the­ory and do worse thereby!) We con­sider this last ar­gu­ment—the ar­gu­ment from util­ity—to be the one that gives the pre­com­mit­ment and value-of-in­for­ma­tion ar­gu­ments their teeth. If self- bind­ing or self-blind­ing were im­por­tant for get­ting more util­ity in cer­tain sce­nar­ios, then we would plau­si­bly en­dorse those prac­tices. Utility has pri­macy, and FDT’s suc­cess on that front is the rea­son we be­lieve that FDT is a more use­ful and gen­eral the­ory of ra­tio­nal choice.”

Some­times Y&S talk about how differ­ent de­ci­sion the­o­ries pro­duce more util­ity on av­er­age if they were to face a spe­cific dilemma re­peat­edly:

  • “Mea­sur­ing by util­ity achieved on av­er­age over time, CDT out­performs EDT in some well-known dilem­mas (Gib­bard and Harper 1978), and EDT out­performs CDT in oth­ers (Ahmed 2014b).”

  • “Imag­ine an agent that is go­ing to face first New­comb’s prob­lem, and then the smok­ing le­sion prob­lem. Imag­ine mea­sur­ing them in terms of util­ity achieved, by which we mean mea­sur­ing them by how much util­ity we ex­pect them to at­tain, on av­er­age, if they face the dilemma re­peat­edly. The sort of agent that we’d ex­pect to do best, mea­sured in terms of util­ity achieved, is the sort who one-boxes in New­comb’s prob­lem, and smokes in the smok­ing le­sion prob­lem.”

Some­times Y&S talk about which agent will achieve more util­ity ‘in ex­pec­ta­tion’, though they don’t define the point at which they gain more ex­pected util­ity (or what no­tion of ‘ex­pected util­ity’ is be­ing used):

  • “One-box­ing in the trans­par­ent New­comb prob­lem may look strange, but it works. Any pre­dic­tor smart enough to carry out the ar­gu­ments above can see that CDT and EDT agents two-box, while FDT agents one-box. Fol­low­ers of CDT and EDT will there­fore al­most always see an empty box, while fol­low­ers of FDT will al­most always see a full one. Thus, FDT agents achieve more util­ity in ex­pec­ta­tion.”

Some­times they talk about how much util­ity ‘de­ci­sion the­o­ries tend to achieve in prac­tice’:

  • “It is for this rea­son that we turn to New­comblike prob­lems to dis­t­in­guish be­tween the three the­o­ries, and demon­strate FDT’s su­pe­ri­or­ity, when mea­sur­ing in terms of util­ity achieved.”

  • “we much pre­fer to eval­u­ate de­ci­sion the­o­ries based on how much util­ity they tend to achieve in prac­tice.”

Some­times they talk about how well the de­ci­sion the­ory does in a cir­cum­scribed class of cases (though they note in foot­note 15 that they can’t define what this class of cases are):

  • “FDT does ap­pear to be su­pe­rior to CDT and EDT in all dilem­mas where the agent’s be­liefs are ac­cu­rate and the out­come de­pends only on the agent’s be­hav­ior in the dilemma at hand. In­for­mally, we call these sorts of prob­lems “fair prob­lems.””

  • “FDT, we claim, gets the bal­ance right. An agent who weighs her op­tions by imag­in­ing wor­lds where her de­ci­sion func­tion has a differ­ent out­put, but where log­i­cal, math­e­mat­i­cal, nomic, causal, etc. con­straints are oth­er­wise re­spected, is an agent with the op­ti­mal pre­dis­po­si­tion for what­ever fair dilemma she en­coun­ters.”

And some­times they talk about how much util­ity the agent would re­ceive in differ­ent pos­si­ble wor­lds than the one she finds her­self in:

  • “When weigh­ing ac­tions, Fiona sim­ply imag­ines hy­po­thet­i­cals cor­re­spond­ing to those ac­tions, and takes the ac­tion that cor­re­sponds to the hy­po­thet­i­cal with higher ex­pected util­ity—even if that means imag­in­ing wor­lds in which her ob­ser­va­tions were differ­ent, and even if that means achiev­ing low util­ity in the world cor­re­spond­ing to her ac­tual ob­ser­va­tions.”

As we can see, the most com­mon for­mu­la­tion of this crite­rion is that they are look­ing for the de­ci­sion the­ory that, if run by an agent, will pro­duce the most util­ity over their life­time. That is, they’re ask­ing what the best de­ci­sion pro­ce­dure is, rather than what the best crite­rion of right­ness is, and are pro­vid­ing an in­di­rect ac­count of the right­ness of acts, as­sess­ing acts in terms of how well they con­form with the best de­ci­sion pro­ce­dure.

But, if that’s what’s go­ing on, there are a whole bunch of is­sues to dis­sect. First, it means that FDT is not play­ing the same game as CDT or EDT, which are pro­posed as crite­ria of right­ness, di­rectly as­sess­ing acts. So it’s odd to have a whole pa­per com­par­ing them side-by-side as if they are ri­vals.

Se­cond, what de­ci­sion the­ory does best, if run by an agent, de­pends cru­cially on what the world is like. To see this, let’s go back to ques­tion that Y&S ask of what de­ci­sion the­ory I’d want my child to have. This de­pends on a whole bunch of em­piri­cal facts: if she might have a gene that causes can­cer, I’d hope that she adopts EDT; though if, for some rea­son, I knew whether or not she did have that gene and she didn’t, I’d hope that she adopts CDT. Similarly, if there were long-dead pre­dic­tors who can no longer in­fluence the way the world is to­day, then, if I didn’t know what was in the opaque boxes, I’d hope that she adopts EDT (or FDT); if I did know what was in the opaque boxes (and she didn’t) I’d hope that she adopts CDT. Or, if I’m in a world where FDT-ers are burned at the stake, I’d hope that she adopts any­thing other than FDT.

Third, the best de­ci­sion the­ory to run is not go­ing to look like any of the stan­dard de­ci­sion the­o­ries. I don’t run CDT, or EDT, or FDT, and I’m very glad of it; it would be im­pos­si­ble for my brain to han­dle the calcu­la­tions of any of these de­ci­sion the­o­ries ev­ery mo­ment. In­stead I al­most always fol­low a whole bunch of rough-and-ready and much more com­pu­ta­tion­ally tractable heuris­tics; and even on the rare oc­ca­sions where I do try to work out the ex­pected value of some­thing ex­plic­itly, I don’t con­sider the space of all pos­si­ble ac­tions and all states of na­ture that I have some cre­dence in — do­ing so would take years.

So the main for­mu­la­tion of Y&S’s most im­por­tant prin­ci­ple doesn’t sup­port FDT. And I don’t think that the other for­mu­la­tions help much, ei­ther. Cri­te­ria of how well ‘a de­ci­sion the­ory does on av­er­age and over time’, or ‘when a dilemma is is­sued re­peat­edly’ run into similar prob­lems as the pri­mary for­mu­la­tion of the crite­rion. Assess­ing by how well the de­ci­sion-maker does in pos­si­ble wor­lds that she isn’t in fact in doesn’t seem a com­pel­ling crite­rion (and EDT and CDT could both do well by that crite­rion, too, de­pend­ing on which pos­si­ble wor­lds one is al­lowed to pick).

Fourth, ar­gu­ing that FDT does best in a class of ‘fair’ prob­lems, with­out be­ing able to define what that class is or why it’s in­ter­est­ing, is a pretty weak ar­gu­ment. And, even if we could define such a class of cases, claiming that FDT ‘ap­pears to be su­pe­rior’ to EDT and CDT in the clas­sic cases in the liter­a­ture is sim­ply beg­ging the ques­tion: CDT ad­her­ents claims that two-box­ing is the right ac­tion (which gets you more ex­pected util­ity!) in New­comb’s prob­lem; EDT ad­her­ents claims that smok­ing is the right ac­tion (which gets you more ex­pected util­ity!) in the smok­ing le­sion. The ques­tion is which of these ac­counts is the right way to un­der­stand ‘ex­pected util­ity’; they’ll there­fore all differ on which of them do bet­ter in terms of get­ting ex­pected util­ity in these clas­sic cases.

Fi­nally, in a com­ment on a draft of this note, Abram Dem­ski said that: “The no­tion of ex­pected util­ity for which FDT is sup­posed to do well (at least, ac­cord­ing to me) is ex­pected util­ity with re­spect to the prior for the de­ci­sion prob­lem un­der con­sid­er­a­tion.” If that’s cor­rect, it’s strik­ing that this crite­rion isn’t men­tioned in the pa­per. But it also doesn’t seem com­pel­ling as a prin­ci­ple by which to eval­u­ate be­tween de­ci­sion the­o­ries, nor does it seem FDT even does well by it. To see both points: sup­pose I’m choos­ing be­tween an av­o­cado sand­wich and a hum­mus sand­wich, and my prior was that I pre­fer av­o­cado, but I’ve since tasted them both and got­ten ev­i­dence that I pre­fer hum­mus. The choice that does best in terms of ex­pected util­ity with re­spect to my prior for the de­ci­sion prob­lem un­der con­sid­er­a­tion is the av­o­cado sand­wich (and FDT, as I un­der­stood it in the pa­per, would agree). But, un­con­tro­ver­sially, I should choose the hum­mus sand­wich, be­cause I pre­fer hum­mus to av­o­cado.


VIII. An al­ter­na­tive ap­proaches that cap­tures the spirit of FDT’s aims

Aca­demic de­ci­sion the­o­rists tends to fo­cus on what ac­tions are ra­tio­nal, but not talk very much about what sort of agent to be­come. Some­thing that’s dis­tinc­tive and good about the ra­tio­nal­ist com­mu­nity’s dis­cus­sion of de­ci­sion the­ory is that there’s more of an em­pha­sis on what sort of agent to be, and what sorts of rules to fol­low.

But this is an area where we can eat our cake and have it. There’s noth­ing to stop us as­sess­ing agents, acts and any­thing else we like in terms of our favourite de­ci­sion the­ory.

Let’s define: Global ex­pected util­ity the­ory =df for any x that is an eval­u­a­tive fo­cal point, the right x is that which max­imises ex­pected util­ity.

I think that Global CDT can get ev­ery­thing we want, with­out the prob­lems that face FDT. Con­sider, for ex­am­ple, the Pri­soner’s Dilemma. On the global ver­sion of CDT, we can say both that (i) the act of defect­ing is the right ac­tion (as­sum­ing that the other agent will use their money poorly); and that (ii) the right sort of per­son to be is one who co­op­er­ates in pris­oner’s dilem­mas.

(ii) would be true, even though (i) is true, if you will face re­peated pris­oner’s dilem­mas, if whether or not you find your­self in op­por­tu­ni­ties to co­op­er­ate de­pend on whether or not you’ve co­op­er­ated in the past, if other agents can tell what sort of per­son you are even in­de­pen­dently in your ac­tions in Pri­soner’s Dilem­mas, and so on. Similar things can be said about black­mail cases and about Parfit’s Hitch­hiker. And similar things can be said more broadly about what sort of per­son to be given con­se­quen­tial­ism — if you be­come some­one who keeps promises, doesn’t tell lies, sticks up for their friends (etc), and who doesn’t analyse these de­ci­sions in con­se­quen­tial­ist terms, you’ll do more good than some­one who tries to ap­ply the con­se­quen­tial­ist crite­rion of right­ness for ev­ery de­ci­sion.

(Some­times be­havi­our like this is de­scribed as ‘ra­tio­nal ir­ra­tional­ity’. But I don’t think that’s an ac­cu­rate de­scrip­tion. It’s not that one and the same thing (the act) is both ra­tio­nal and ir­ra­tional. In­stead, we con­tinue to ac­knowl­edge that the act is the ir­ra­tional one; we just also ac­knowl­edge that it re­sults from the ra­tio­nal dis­po­si­tion to have.)

There are other pos­si­ble ways of cap­tur­ing some of the spirit of FDT, such as a sort of rule-con­se­quen­tial­ism, where the right set of rules to fol­low are those that would pro­duce the best out­come if all agents fol­lowed those rules, and the right act is that which con­forms to that set of rules. But I think that global causal de­ci­sion the­ory is the most promis­ing idea in this space.


IX. Conclusion

In this note, I ar­gued that FDT faces mul­ti­ple ma­jor prob­lems. In my view, these are fatal to FDT in its cur­rent form. I think it’s pos­si­ble that, with very ma­jor work, a ver­sion of FDT could be de­vel­oped that could over­come some of these prob­lems (in par­tic­u­lar, the prob­lems de­scribed in sec­tions IV, V and VI, that are based, in one way or an­other, on the is­sue of when two pro­cesses are Y&S-sub­junc­tively de­pen­dent on one an­other). But it’s hard to see what the mo­ti­va­tion for do­ing so is: FDT in any form will vi­o­late Guaran­teed Pay­offs, which should be one of the most ba­sic con­straints on a de­ci­sion the­ory; and if, in­stead, we want to se­ri­ously un­der­take the pro­ject of what de­ci­sion-pro­ce­dure is the best for an agent to run (or ‘what should we code into an AI?’), the an­swer will be far messier, and far more de­pen­dent on par­tic­u­lar facts about the world and the com­pu­ta­tional re­sources of the agent in ques­tion, than any of EDT, CDT or FDT.