# Gary_Drescher comments on A problem with Timeless Decision Theory (TDT)

• 2) Treat differ­ently math­e­mat­i­cal knowl­edge that we learn by gen­uinely math­e­mat­i­cal rea­son­ing and by phys­i­cal ob­ser­va­tion. In this case we know (D xor E) not by math­e­mat­i­cal rea­son­ing, but by phys­i­cally ob­serv­ing a box whose state we be­lieve to be cor­re­lated with D xor E. This may jus­tify con­struct­ing a causal DAG with a node de­scend­ing from D and E, so a coun­ter­fac­tual set­ting of D won’t af­fect the set­ting of E.

Per­haps I’m mi­s­un­der­stand­ing you here, but D and E are Pla­tonic com­pu­ta­tions. What does it mean to con­struct a causal DAG among Pla­tonic com­pu­ta­tions? [EDIT: Ok, I may un­der­stand that a lit­tle bet­ter now; see my edit to my re­ply to (1).] Such a graph links to­gether gen­eral math­e­mat­i­cal facts, so the same is­sues arise as in (1), it seems to me: Do the links cor­re­spond to log­i­cal in­fer­ence, or some­thing else? What makes the graph acyclic? Is math­e­mat­i­cal causal­ity even co­her­ent? And if you did have a mod­ule that can de­tect (pre­sum­ably time­less) causal links among Pla­tonic com­pu­ta­tions, then why not use that mod­ule di­rectly to solve your de­ci­sion prob­lems?

Plus I’m not con­vinced that there’s a mean­ingful dis­tinc­tion be­tween math knowl­edge that you gain by gen­uine math rea­son­ing, and math knowl­edge that you gain by phys­i­cal ob­ser­va­tion.

Let’s say, for in­stance, that I feed a par­tic­u­lar con­jec­ture to an au­to­matic the­o­rem prover, which tells me it’s true. Have I then learned that math fact by gen­uine math­e­mat­i­cal rea­son­ing (performed by the phys­i­cal com­puter’s Pla­tonic ab­strac­tion)? Or have I learned it by phys­i­cal ob­ser­va­tion (of the phys­i­cal com­puter’s out­put), and hence be barred from us­ing that math fact for pur­poses of TDT’s log­i­cal-de­pen­dency-de­tec­tion? Pre­sum­ably the former, right? (Or else TDT will make even worse er­rors.)

But then sup­pose the pre­dic­tor has simu­lated the uni­verse suffi­ciently to es­tab­lish that U (the uni­verse’s al­gorithm, in­clud­ing physics and ini­tial con­di­tions) leads to there be­ing \$1M in the box in this situ­a­tion. That’s a math­e­mat­i­cal fact about U, ob­tained by (the simu­la­tor’s) math­e­mat­i­cal rea­son­ing. Let’s sup­pose that when the pre­dic­tor briefs me, the briefing in­cludes men­tion of this math­e­mat­i­cal fact. So even if I keep my eyes closed and never phys­i­cally see the \$1M, I can rely in­stead on the cor­re­spond­ing math­e­mat­i­cally de­rived fact.

(Or more straight­for­wardly, we can view the uni­verse it­self as a com­puter that’s perform­ing math­e­mat­i­cal rea­son­ing about how U un­folds, in which case any phys­i­cal ob­ser­va­tion is in­trin­si­cally ob­tained by math­e­mat­i­cal rea­son­ing.)

• Log­i­cal un­cer­tainty has always been more difficult to deal with than phys­i­cal un­cer­tainty; the prob­lem with log­i­cal un­cer­tainty is that if you an­a­lyze it enough, it goes away. I’ve never seen any re­ally good treat­ment of log­i­cal un­cer­tainty.

But if we de­part from TDT for a mo­ment, then it does seem clear that we need to have causelike nodes cor­re­spond­ing to log­i­cal un­cer­tainty in a DAG which de­scribes our prob­a­bil­ity dis­tri­bu­tion. There is no other way you can com­pletely ob­serve the state of a calcu­la­tor sent to Mars and a calcu­la­tor sent to Venus, and yet re­main un­cer­tain of their out­comes yet be­lieve the out­comes are cor­re­lated. And if you talk about er­ror-prone calcu­la­tors, two of which say 17 and one of which says 18, and you de­duce that the “Pla­tonic an­swer” was prob­a­bly in fact 17, you can see that log­i­cal un­cer­tainty be­haves in an even more causelike way than this.

So, go­ing back to TDT, my hope is that there’s a neat set of rules for fac­tor­ing our log­i­cal un­cer­tainty in our causal be­liefs, and that these same rules also re­solve the sort of situ­a­tion that you de­scribe.

If you con­sider the no­tion of the cor­re­lated er­ror-prone calcu­la­tors, two re­turn­ing 17 and one re­turn­ing 18, then the most in­tu­itive way to han­dle this would be to see a “Pla­tonic an­swer” as its own causal node, and the calcu­la­tors as er­ror-prone de­scen­dants. I’m pretty sure this is how my brain is draw­ing the graph, but I’m not sure it’s the cor­rect an­swer; it seems to me that a more prin­ci­pled an­swer would in­volve un­cer­tainty about which math­e­mat­i­cal fact af­fects each calcu­la­tor—phys­i­cally un­cer­tain gates which de­ter­mine which calcu­la­tion af­fects each calcu­la­tor.

For the (D xor E) prob­lem, we know the be­hav­ior we want the TDT calcu­la­tion to ex­hibit; we want (D xor E) to be a de­scen­dant node of D and E. If we view the phys­i­cal ob­ser­va­tion of \$1m as tel­ling us the raw math­e­mat­i­cal fact (D xor E), and then perform math­e­mat­i­cal in­fer­ence on D, we’ll find that we can af­fect E, which is not what we want. Con­versely if we view D as hav­ing a phys­i­cal effect, and E as hav­ing a phys­i­cal effect, and the node D xor E as a phys­i­cal de­scen­dant of D and E, we’ll get the be­hav­ior we want. So the ques­tion is whether there’s any prin­ci­pled way of set­ting this up which will yield the sec­ond be­hav­ior rather than the first, and also, pre­sum­ably, yield epistem­i­cally cor­rect be­hav­ior when rea­son­ing about calcu­la­tors and so on.

That’s if we go down av­enue (2). If we go down av­enue (1), then we give pri­macy to our in­tu­ition that if-coun­ter­fac­tu­ally you make a differ­ent de­ci­sion, this log­i­cally con­trols the math­e­mat­i­cal fact (D xor E) with E held con­stant, but does not log­i­cally con­trol E with (D xor E) held con­stant. While this does sound in­tu­itive in a sense, it isn’t quite nailed down—af­ter all, D is ul­ti­mately just as con­stant as E and (D xor E), and to change any of them makes the model equally in­con­sis­tent.

Th­ese sorts of is­sues are some­thing I’m still think­ing through, as I think I’ve men­tioned, so let me think out loud for a bit.

In or­der to ob­serve any­thing that you think has already been con­trol­led by your de­ci­sion—any phys­i­cal thing in which a copy of D has already played a role—then (leav­ing aside the ques­tion of Omega’s strat­egy that simu­lated al­ter­nate ver­sions of you to se­lect a self-con­sis­tent prob­lem, and whether this in­tro­duces con­di­tional-strat­egy-de­pen­dence rather than just de­ci­sion-de­pen­dence into the prob­lem) there have to be other phys­i­cal facts which com­bine with D to yield our ob­ser­va­tion.

Some of these phys­i­cal facts may them­selves be af­fected by math­e­mat­i­cal facts, like an im­ple­mented com­pu­ta­tion of E; but the point is that in or­der to have ob­served any­thing con­trol­led by D, we already had to draw a phys­i­cal, causal di­a­gram in which other nodes de­scended from D.

So sup­pose we in­tro­duce the rule that in ev­ery case like this, we will have some phys­i­cal node that is af­fected by D, and if we can ob­serve info that de­pends on D in any way, we’ll view the other math­e­mat­i­cal facts as com­bin­ing with D’s phys­i­cal node. This is a rule that tells us not to draw the di­a­gram with a phys­i­cal node be­ing de­ter­mined by the math­e­mat­i­cal fact D xor E, but rather to have a phys­i­cal node de­ter­mined by D, and then a phys­i­cal de­scen­dent D xor E. (Which in this par­tic­u­lar prob­lem should de­scend from a phys­i­cal node E that de­scends from the math­e­mat­i­cal fact E, be­cause the math­e­mat­i­cal fact E is cor­re­lated with our un­cer­tainty about other things, and a fac­tored causal graph should have no re­main­ing cor­re­lated sources of back­ground un­cer­tainty; but if E didn’t cor­re­late to any­thing else in par­tic­u­lar, we could just have D de­scend­ing to (D xor E) via the (xor with E) rule.)

When I eval­u­ate this pro­posed solu­tion for ad-hoc-ness, it does ad­mit­tedly look a bit ad-hoc, but it solves at least one other prob­lem than the one I started with, and which I didn’t think of un­til now. Sup­pose Omega tells me that I make the same de­ci­sion in the Pri­soner’s Dilemma as Agent X. This does not nec­es­sar­ily im­ply that I should co­op­er­ate with Agent X. X and I could have made the same de­ci­sion for differ­ent (un­cor­re­lated) rea­sons, and Omega could have sim­ply found out by simu­lat­ing the two of us that X and I gave the same re­sponse. In this case, pre­sum­ably defect­ing; but if I co­op­er­ated, X wouldn’t do any­thing differ­ently. X is just a piece of pa­per with “Defect” writ­ten on it.

If I draw a causal di­a­gram of how I came to learn this cor­re­la­tion from Omega, and I fol­low the rule of always draw­ing a causal bound­ary around the math­e­mat­i­cal fact D as soon as it phys­i­cally af­fects some­thing, then, given the way Omega simu­lated both of us to ob­serve the cor­re­la­tion, I see that D and X sep­a­rately phys­i­cally af­fected the cor­re­la­tion-checker node.

On the other hand, if I can an­a­lyze the two pieces of code D and X and see that they re­turn the same out­put, with­out yet know­ing the out­put, then this knowl­edge was ob­tained in a way that doesn’t in­volve D pro­duc­ing an out­put, so I don’t have to draw a hard causal bound­ary around that out­put.

If this works, the un­der­ly­ing prin­ci­ple that makes it work is some­thing along the lines of “for D to con­trol X, the cor­re­la­tion be­tween our un­cer­tainty about D and X has to emerge in a way that doesn’t in­volve any­one already com­put­ing D”. Other­wise D has no free will (said firmly tongue-in-cheek). I am not sure that this prin­ci­ple has any more el­e­gant ex­pres­sion than the rule, “when­ever, in your phys­i­cal model of the uni­verse, D finishes com­put­ing, draw a phys­i­cal/​causal bound­ary around that finished com­pu­ta­tion and have other things phys­i­cally/​causally de­scend from it”.

If this prin­ci­ple is vi­o­lated then D ends up “cor­re­lated” to all sorts of other things we ob­serve, like the price of fish and whether it’s rain­ing out­side, via the magic of xor.

• If we go down av­enue (1), then we give pri­macy to our in­tu­ition that if-coun­ter­fac­tu­ally you make a differ­ent de­ci­sion, this log­i­cally con­trols the math­e­mat­i­cal fact (D xor E) with E held con­stant, but does not log­i­cally con­trol E with (D xor E) held con­stant. While this does sound in­tu­itive in a sense, it isn’t quite nailed down—af­ter all, D is ul­ti­mately just as con­stant as E and (D xor E), and to change any of them makes the model equally in­con­sis­tent.

I agree this sounds in­tu­itive. As I men­tioned ear­lier, though, nailing this down is tan­ta­mount to cir­cling back and solv­ing the full-blown prob­lem of (de­ci­sion-sup­port­ing) coun­ter­fac­tual rea­son­ing: the prob­lem of how to dis­t­in­guish which facts to “hold fixed”, and which to “let vary” for con­sis­tency with a coun­ter­fac­tual an­tecedent.

In any event, is the idea to try to build a sep­a­rate graph for math facts, and use that to an­a­lyze “log­i­cal de­pen­dency” among the Pla­tonic nodes in the origi­nal graph, in or­der to carry out TDT’s mod­ified “sur­gi­cal al­ter­a­tion” of the origi­nal graph? Or would you try to build one big graph that en­com­passes phys­i­cal and log­i­cal facts al­ike, and then use Pearl’s de­ci­sion pro­ce­dure with­out fur­ther mod­ifi­ca­tion?

If we view the phys­i­cal ob­ser­va­tion of \$1m as tel­ling us the raw math­e­mat­i­cal fact (D xor E), and then perform math­e­mat­i­cal in­fer­ence on D, we’ll find that we can af­fect E, which is not what we want.

Wait, isn’t it de­ci­sion-com­pu­ta­tion C—rather than simu­la­tion D—whose “effect” (in the sense of log­i­cal con­se­quence) on E we’re con­cerned about here? It’s the log­i­cal de­pen­dents of C that get sur­gi­cally al­tered in the graph when C gets sur­gi­cally al­tered, right? (I know C and D are log­i­cally equiv­a­lent, but you’re talk­ing about in­sert­ing a phys­i­cal node af­ter D, not C, so I’m a bit con­fused.)

I’m hav­ing trou­ble fol­low­ing the gist of av­enue (2) at the mo­ment. Even with the node struc­ture you sug­gest, we can still in­fer E from C and from the phys­i­cal node that matches (D xor E)—un­less the new rule pro­hibits rely­ing on that phys­i­cal node, which I guess is the idea. But what ex­actly is the pro­hi­bi­tion? Are we for­bid­den to in­fer any math­e­mat­i­cal fact from any phys­i­cal in­di­ca­tor of that fact? Or is there some­thing in par­tic­u­lar about node (D xor E) that makes it for­bid­den? (It would be cir­cu­lar to cite the node’s de­pen­dence on C in the very sense of “de­pen­dence” that the new rule is helping us to com­pute.)

• Or would you try to build one big graph that en­com­passes phys­i­cal and log­i­cal facts al­ike, and then use Pearl’s de­ci­sion pro­ce­dure with­out fur­ther mod­ifi­ca­tion?

I definitely want one big graph if I can get it.

Wait, isn’t it de­ci­sion-com­pu­ta­tion C—rather than simu­la­tion D—whose “effect” (in the sense of log­i­cal con­se­quence) on E we’re con­cerned about here?

Sorry, yes, C.

Even with the node struc­ture you sug­gest, we can still in­fer E from C and from the phys­i­cal node that matches (D xor E)—un­less the new rule pro­hibits rely­ing on that phys­i­cal node, which I guess is the idea. But what ex­actly is the pro­hi­bi­tion? Are we for­bid­den to in­fer any math­e­mat­i­cal fact from any phys­i­cal in­di­ca­tor of that fact?

No, but when­ever we see a phys­i­cal fact F that de­pends on a de­ci­sion C/​D we’re still in the pro­cess of mak­ing plus Some­thing Else (E), then we ex­press our un­cer­tainty in the form of a causal graph with di­rected ar­rows from C to D, D to F, and E to F. Thus when we com­pute a coun­ter­fac­tual on C, we find that F changes, but E does not.

• No, but when­ever we see a phys­i­cal fact F that de­pends on a de­ci­sion C/​D we’re still in the pro­cess of mak­ing plus Some­thing Else (E),

Wait, F de­pends on de­ci­sion com­pu­ta­tion C in what sense of “de­pends on”? It can’t quite be the origi­nally defined sense (quoted from your email near the top of the OP), since that defines de­pen­dency be­tween Pla­tonic com­pu­ta­tions, not be­tween a Pla­tonic com­pu­ta­tion and a phys­i­cal fact. Do you mean that D de­pends on C in the origi­nal sense, and F in turn de­pends on D (and on E) in a differ­ent sense?

then we ex­press our un­cer­tainty in the form of a causal graph with di­rected ar­rows from C to D, D to F, and E to F.

Ok, but these ar­rows can’t be used to define the rele­vant sense of de­pen­dency above, since the rele­vant sense of de­pen­dency is what tells us we need to draw the ar­rows that way, if I un­der­stand cor­rectly.

Sorry to keep be­ing pedan­tic about the mean­ing of “de­pends”; I know you’re in think­ing-out-loud mode here. But the the­ory gives wildly differ­ent an­swers de­pend­ing (heh) on how that gets pinned down.

• In my view, the chief form of “de­pen­dence” that needs to be dis­crim­i­nated is in­fer­en­tial de­pen­dence and causal de­pen­dence. If earth­quakes cause bur­glar alarms to go off, then we can in­fer an earth­quake from a bur­glar alarm or in­fer a bur­glar alarm from an earth­quake. Log­i­cal rea­son­ing doesn’t have the kind of di­rec­tion­al­ity that cau­sa­tion does—or at least, clas­si­cal log­i­cal rea­son­ing does not—there’s no preferred form be­tween ~A->B, ~B->A, and A \/​ B.

The link be­tween the Pla­tonic de­ci­sion C and the phys­i­cal de­ci­sion D might be differ­ent from the link be­tween the phys­i­cal de­ci­sion D and the phys­i­cal ob­ser­va­tion F, but I don’t know of any­thing in the cur­rent the­ory that calls for treat­ing them differ­ently. They’re just di­rec­tional causal links. On the other hand, if C math­e­mat­i­cally im­plies a de­ci­sion C-2 some­where else, that’s a log­i­cal im­pli­ca­tion that ought to sym­met­ri­cally run back­ward to ~C-2 → ~C, ex­cept of course that we’re pre­sum­ably con­trol­ling/​eval­u­at­ing C rather than C-2.

Think­ing out loud here, the view is that your math­e­mat­i­cal un­cer­tainty ought to be in one place, and your phys­i­cal un­cer­tainty should be built on top of your math­e­mat­i­cal un­cer­tainty. The math­e­mat­i­cal un­cer­tainty is a log­i­cal graph with sym­met­ric in­fer­ences, the phys­i­cal un­cer­tainty is a di­rected acyclic graph. To form con­trol­ling coun­ter­fac­tu­als, you up­date the math­e­mat­i­cal un­cer­tainty, in­clud­ing any log­i­cal in­fer­ences that take place in math­land, and watch it prop­a­gate down­ward into the phys­i­cal un­cer­tainty. When you’ve already ob­served facts that phys­i­cally de­pend on math­e­mat­i­cal de­ci­sions you con­trol but you haven’t yet made and hence whose val­ues you don’t know, then those ob­ser­va­tions stay in the causal, di­rected, acyclic world; when the coun­ter­fac­tual gets eval­u­ated, they get up­dated in the Pearl, di­rec­tional way, not the log­i­cal, sym­met­ri­cal in­fer­en­tial way.

• The link be­tween the Pla­tonic de­ci­sion C and the phys­i­cal de­ci­sion D

No, D was the Pla­tonic simu­la­tor. That’s why the na­ture of the C->D de­pen­dency is cru­cial here.

• Okay, then we have a log­i­cal link from C-pla­tonic to D-pla­tonic, and causal links de­scend­ing from C-pla­tonic to C-phys­i­cal, E-pla­tonic to E-phys­i­cal, and D-pla­tonic to D-phys­i­cal to F-phys­i­cal = D-phys­i­cal xor E-phys­i­cal. The idea be­ing that when we coun­ter­fac­tu­al­ize on C-pla­tonic, we up­date D-pla­tonic and its de­scen­dents, but not E-pla­tonic or its de­scen­dents.

I sup­pose that as writ­ten, this re­quires a rule, “for pur­poses of com­put­ing coun­ter­fac­tu­als, keep in the causal graph rather than the log­i­cal knowl­edge base, any math­e­mat­i­cal knowl­edge gained by ob­serv­ing a fact de­scended from your de­ci­sion-out­put or any log­i­cal im­pli­ca­tions of your de­ci­sion-out­put”. I could hope that this is a spe­cial case of some­thing more el­e­gant, but it would only be hope.

• Ok. I think it would be very helpful to sketch, all in one place, what TDT2 (i.e., the en­vi­sioned av­enue-2 ver­sion of TDT) looks like, tak­ing care to pin down any needed sense of “de­pen­dency”. And similarly for TDT1, the av­enue-1 ver­sion. (Th­ese sug­ges­tions may be pre­ma­ture, I re­al­ize.)

• When you use terms like “draw a hard causal bound­ary” I’m forced to imag­ine you’re ac­tu­ally draw­ing these things on the back of a cock­tail nap­kin some­where us­ing some sorts of stan­dard sym­bols. Are there such stan­dards, and do you have such di­a­grams scanned in on­line some­where?

ETA: A note for fu­ture read­ers: Eliezer be­low is refer­ring to Judea Pearl (sim­ply “Pearl” doesn’t con­vey much via google-search­ing, though I sup­pose “pearl causal­ity” does at the mo­ment)

• Read Pearl. I think his on­line in­tros should give you a good idea of what the cock­tail nap­kin looks like.

• Hmm… Pearl uses a lot of di­a­grams but they all seem pretty ad-hoc. Just the sorts of ar­rows and dots and things that you’d use to rep­re­sent any graph (in the math­e­mat­ics sense). Should I in­fer from this de­scrip­tion that the an­swer is, “No, there isn’t a stan­dard”?

I was pic­tur­ing some­thing like a leg­end that would tell some­one, “Use a dashed line for a causal bound­ary, and a red dot­ted line to rep­re­sent a log­i­cal in­fer­ence, and a pink squir­rel to rep­re­sent post­mod­ernism”

• Um… I’m not sure there’s much I can say to that be­yond “Read Prob­a­bil­is­tic Rea­son­ing in In­tel­li­gent Sys­tems, or Causal­ity”.

Pearl’s sys­tem is not ad-hoc. It is very not ad-hoc. It has a met­ric fuck­load of math back­ing up the sim­ple rules. But Pearl’s sys­tem does not in­clude log­i­cal un­cer­tainty. I’m try­ing to put log­i­cal un­cer­tainty into it, while obey­ing the rules. This is a work in progress.

• Pearl’s sys­tem is not ad-hoc. It is very not ad-hoc. It has a met­ric fuck­load of math back­ing up the sim­ple rules.

Thomblake’s ob­ser­va­tion may be that while Pearl’s sys­tem is ex­tremely rigor­ous the di­a­grams used do not give an au­thor­i­ta­tive stan­dard style for di­a­gram draw­ing.

• That’s cor­rect—I was look­ing for a stan­dard style for di­a­gram draw­ing.

I’d just like to reg­ister a gen­eral ap­proval of spec­i­fy­ing that one’s imag­i­nary units are met­ric.

• I’m reread­ing past dis­cus­sions to find in­sights. This jumped out at me:

Sup­pose Omega tells me that I make the same de­ci­sion in the Pri­soner’s Dilemma as Agent X. This does not nec­es­sar­ily im­ply that I should co­op­er­ate with Agent X.

Do you still be­lieve this?

• Play­ing chicken with Omega may re­sult in you be­com­ing coun­ter­fac­tual.

• Why is co­op­er­a­tion more likely to qual­ify as “play­ing chicken” than defec­tion here?

• I was refer­ring to the ex­am­ple Eliezer gives with your op­po­nent be­ing a Defec­tBot, in which case co­op­er­at­ing makes Omega’s claim false, which may just mean that you’d make your branch of the thought ex­per­i­ment coun­ter­fac­tual, in­stead of con­vinc­ing Defec­tBot to co­op­er­ate:

X is just a piece of pa­per with “Defect” writ­ten on it.

• which may just mean that you’d make your branch of the thought ex­per­i­ment counterfactual

So? That doesn’t hurt my util­ity in re­al­ity. I would co­op­er­ate be­cause that wins if agent X is cor­re­lated with me, and doesn’t lose oth­er­wise.

• Win­ning is about how al­ter­na­tives you choose be­tween com­pare. By co­op­er­at­ing against a same-ac­tion Defec­tBot, you are choos­ing nonex­is­tence over a (D,D), which is not ob­vi­ously a neu­tral choice.

• I don’t think this is how it works. Par­tic­u­lar coun­ter­fac­tual in­stances of you can’t in­fluence whether they are coun­ter­fac­tual or ex­ist in some stronger sense. They can only choose whether there are more real in­stances with iden­ti­cal ex­pe­riences (and their choices can some­times acausally in­fluence what hap­pens with real in­stances, which doesn’t seem to be the case here since the real you will choose defect ei­ther way as pre­dicted by Omega). Hy­po­thet­i­cal in­stances don’t lose any­thing by be­ing in the branch that chooses the op­po­site of what the real you chooses un­less they value be­ing iden­ti­cal to the real you, which IMO would be silly.

• Par­tic­u­lar coun­ter­fac­tual in­stances of you can’t in­fluence whether they are coun­ter­fac­tual or ex­ist in some stronger sense.

What can in­fluence things like that? What­ever prop­erty of a situ­a­tion can mark it as coun­ter­fac­tual (more pre­cisely, given by a con­tra­dic­tory speci­fi­ca­tion, or not fol­low­ing from a pre­ced­ing con­struc­tion, as­sumed-real past state for ex­am­ple), that prop­erty could as well be a de­ci­sion made by an agent pre­sent in that situ­a­tion. There is noth­ing spe­cial about agents or their de­ci­sions.

• What can in­fluence things like that?

Why do you think some­thing can in­fluence it? Whether you choose to co­op­er­ate or defect, you can always ask both “what would hap­pen if I co­op­er­ated?” and “what would hap­pen if I defected?”. In as far as be­ing coun­ter­fac­tual makes sense the al­ter­na­tive to be­ing the an­swer to “what would hap­pen if I co­op­er­ated?” is be­ing the an­swer to “what would hap­pen if I defected?”, even if you know that the real you defects.

Com­pare Omega tel­ling you that your an­swer will be the the same as the Nth digit of Pi. That doesn’t you al­low to choose the Nth digit of Pi.

• Win­ning is about how al­ter­na­tives you choose be­tween com­pare. By co­op­er­at­ing against a same-ac­tion Defec­tBot, you are choos­ing nonex­is­tence over a (D,D), which is not ob­vi­ously a neu­tral choice.

This be­comes a (rel­a­tively) straight­for­ward mat­ter of work­ing out where the (po­ten­tially coun­ter­fac­tual—de­pend­ing what you choose) calcu­la­tion is be­ing performed to de­ter­mine ex­actly what this ‘nonex­is­tence’ means. Since this par­tic­u­lar thought ex­per­i­ment doesn’t seem to spec­ify any other broader con­text I as­sert that co­op­er­ate is clearly the cor­rect op­tion. Any agent which doesn’t co­op­er­ate is bro­ken.

Ba­si­cally, if you ever find your­self in this situ­a­tion then you don’t mat­ter. It’s your job to play chicken with the uni­verse and not ex­ist so the ac­tual you can win.

• Agent X is a piece of pa­per with “Defect” writ­ten on it. I defect against it. Omega’s claim is true and does not im­ply that I should co­op­er­ate.

• I don’t see this ar­gu­ment mak­ing sense. Omega’s claim re­duces to neglibible chances that a choice of Defec­tion will be ad­van­ta­geous for me, be­cause Omega’s claim makes it of neglible prob­a­bil­ity that ei­ther (D,C) or (C, D) will be re­al­ized. So I can only choose be­tween the wor­lds of (C, C) and (D, D). Which means that the Co­op­er­a­tion world is ad­van­ta­geous, and that I should Co­op­er­ate.

In con­trast, if Omega had claimed that we’d make the op­po­site de­ci­sions, then I’d only have to choose be­tween the wor­lds of (D, C) or (C, D) -- with the wor­lds of (C, C) and (D, D) now hav­ing neg­ligible prob­a­bil­ity. In which case, I should, of course, Defect.

The rea­sons for the cor­re­la­tion be­tween me and Agent X are ir­rele­vant when the fact of their cor­re­la­tion is known.

• Agent X is a piece of pa­per with “Defect” writ­ten on it.

Sorry, was this in­tended as part of the prob­lem state­ment, like “Omega tells you that agent X is a Defec­tBot that will play the same as you”? If yes, then ok. But if we don’t know what agent X is, then I don’t un­der­stand why a Defec­tBot is apri­ori more prob­a­ble than a Co­op­er­ateBot. If they are equally prob­a­ble, then it can­cels out (edit: no it doesn’t, it ac­tu­ally makes co­op­er­at­ing a bet­ter choice, thx ArisKat­saris). And there’s also the case where X is a copy of you, where co­op­er­at­ing does help. So it seems to be a bet­ter choice over­all.

• But there’s also the case where X is a copy of you, where co­op­er­at­ing does help, so it seems to be a bet­ter choice over­all.

There is also a case where X is an an­ti­copy (performs op­po­site ac­tion), which ar­gues for defect­ing in the same man­ner.

Edit: This re­ply is wrong.

• There is also a case where X is an an­ti­copy (performs op­po­site ac­tion), which ar­gues for defect­ing in the same man­ner.

No it doesn’t. If X is an an­ti­copy, the situ­a­tion can’t be real and your ac­tion doesn’t mat­ter.

• Why can’t it be real?

• Be­cause Omega has told you that X’s ac­tion is the same as yours.

• OK.

• This is a rule that tells us not to draw the di­a­gram with a phys­i­cal node be­ing de­ter­mined by the math­e­mat­i­cal fact D xor E, but rather to have a phys­i­cal node de­ter­mined by D, and then a phys­i­cal de­scen­dent D xor E...

When I eval­u­ate this pro­posed solu­tion for ad-hoc-ness, it does ad­mit­tedly look a bit ad-hoc, but it solves at least one other prob­lem than the one I started with, and which I didn’t think of un­til now. Sup­pose Omega tells me that I make the same de­ci­sion in the Pri­soner’s Dilemma as Agent X. This does not nec­es­sar­ily im­ply that I should co­op­er­ate with Agent X. X and I could have made the same de­ci­sion for differ­ent (un­cor­re­lated) rea­sons, and Omega could have sim­ply found out by simu­lat­ing the two of us that X and I gave the same re­sponse. In this case, pre­sum­ably defect­ing; but if I co­op­er­ated, X wouldn’t do any­thing differ­ently. X is just a piece of pa­per with “Defect” writ­ten on it.

If X isn’t like us, we can’t “con­trol” X by mak­ing a de­ci­sion similar to what we would want X to out­put*. We shouldn’t go from be­ing an agent that defects in the pris­oner’s dilemma with Agent X when told we “make the same de­ci­sion in the Pri­soner’s Dilemma as Agent X” to be­ing one that does not defect, just as we do not unilat­er­ally switch from nat­u­ral to pre­ci­sion bid­ding when in con­tract bridge a part­ner opens with two clubs (which sig­nals a good hand un­der pre­ci­sion bid­ding, and not un­der nat­u­ral bid­ding).

How­ever, there do ex­ist agents who should co­op­er­ate ev­ery time they hear they “make the same de­ci­sion in the Pri­soner’s Dilemma as Agent X”, those who have com­mit­ted to co­op­er­ate in such cases. In some such cases, they are up against pieces of pa­per on which “co­op­er­ate” is writ­ten (too bad they didn’t have a more dis­crim­i­nat­ing al­gorithm/​clear Omega), in oth­ers, they are up against copies of them­selves or other agents whose out­put de­pends on what Omega tells them. In any case, many agents should co­op­er­ate when they hear that.

Yes? No?

Why shouldn’t one be such an agent? Do we know ahead of time that we are likely to be up against pieces of pa­per with “co­op­er­ate” on them, and Omega would tell un­helpfully tell us we “make the same de­ci­sion in the Pri­soner’s Dilemma as Agent X” in all such cases, though if we had a differ­ent strat­egy we could have got­ten use­ful in­for­ma­tion and defected in that case?

*Other cases in­clude us defect­ing to get X to co­op­er­ate, and oth­ers where X’s play de­pends on ours, but this is the nat­u­ral case to use when con­sid­er­ing if the Agent X’s ac­tion de­pends on ours, a not strate­gi­cally in­com­pe­tent Agent X that has a strat­egy at least as good as always defect­ing or co­op­er­at­ing and does not try to con­di­tion his co­op­er­at­ing on our defect­ing or the like.