# shminux comments on Decision Theory

• If you know your own ac­tions, why would you rea­son about tak­ing differ­ent ac­tions? Wouldn’t you rea­son about some­one who is al­most like you, but just differ­ent enough to make a differ­ent choice?

• Sure. How do you do that?

• No­tice (well, you already know that) that ac­cept­ing that iden­ti­cal agents make iden­ti­cal de­ci­sions (su­per­ra­tional­ity, as it were) and to make differ­ent de­ci­sions in iden­ti­cal cir­cum­stances the agents must nec­es­sar­ily be differ­ent, gets you out of many pick­les. For ex­am­ple, in the 5&10 game an agent would ex­am­ine its own al­gorithm, see that it leads to tak­ing \$10 and stop there. There is no “what would hap­pen if you took a differ­ent ac­tion”, be­cause the agent tak­ing a differ­ent ac­tion would not be you, not ex­actly. So, no Lo­bian ob­sta­cle. In re­turn, you give up some­thing a lot more emo­tion­ally valuable: the delu­sion of mak­ing con­scious de­ci­sions. Pick your poi­son.

• For ex­am­ple, in the 5&10 game an agent would ex­am­ine its own al­gorithm, see that it leads to tak­ing \$10 and stop there.

Why do even that much if this rea­son­ing could not be used? The ques­tion is about the rea­son­ing that could con­tribute to the de­ci­sion, that could de­scribe the al­gorithm, and so has the op­tion to not “stop there”. What if you see that your al­gorithm leads to tak­ing the \$10 and in­stead of stop­ping there, you take the \$5?

Noth­ing stops you. This is the “chicken rule” and it solves some is­sues, but more im­por­tantly illus­trates the pos­si­bil­ity in how a de­ci­sion al­gorithm can func­tion. The fact that this is a thing is ev­i­dence that there may be some­thing wrong with the “stop there” pro­posal. Speci­fi­cally, you usu­ally don’t know that your rea­son­ing is ac­tual, that it’s even log­i­cally pos­si­ble and not part of an im­pos­si­ble coun­ter­fac­tual, but this is not a hope­less hy­po­thet­i­cal where noth­ing mat­ters. Noth­ing com­pels you to af­firm what you know about your ac­tions or con­clu­sions, this is not a ne­ces­sity in a de­ci­sion mak­ing al­gorithm, but differ­ent things you do may have an im­pact on what hap­pens, be­cause the situ­a­tion may be ac­tual af­ter all, de­pend­ing on what hap­pens or what you de­cide, or it may be pre­dicted from within an ac­tual situ­a­tion and in­fluence what hap­pens there. This mo­ti­vates learn­ing to rea­son in and about pos­si­bly im­pos­si­ble situ­a­tions.

What if you ex­am­ine your al­gorithm and find that it takes the \$5 in­stead? It could be the same al­gorithm that takes the \$10, but you don’t know that, in­stead you ar­rive at the \$5 con­clu­sion us­ing rea­son­ing that could be im­pos­si­ble, but that you don’t know to be im­pos­si­ble, that you haven’t de­cided yet to make im­pos­si­ble. One way to solve the is­sue is to ren­der the situ­a­tion where that holds im­pos­si­ble, by con­tra­dict­ing the con­clu­sion with your ac­tion, or in some other way. To know when to do that, you should be able to rea­son about and within such situ­a­tions that could be im­pos­si­ble, or could be made im­pos­si­ble, in­clud­ing by the de­ci­sions made in them. This makes the way you rea­son in them rele­vant, even when in the end these situ­a­tions don’t oc­cur, be­cause you don’t a pri­ori know that they don’t oc­cur.

(The 5-and-10 prob­lem is not speci­fi­cally about this is­sue, and ex­plicit rea­son­ing about im­pos­si­ble situ­a­tions may be avoided, per­haps should be avoided, but my guess is that the crux in this com­ment thread is about things like use­ful­ness of rea­son­ing from within pos­si­bly im­pos­si­ble situ­a­tions, where even your own knowl­edge ar­rived at by pure com­pu­ta­tion isn’t nec­es­sar­ily cor­rect.)

• Thank you for your ex­pla­na­tion! Still try­ing to un­der­stand it. I un­der­stand that there is no point ex­am­in­ing one’s al­gorithm if you already ex­e­cute it and see what it does.

What if you see that your al­gorithm leads to tak­ing the \$10 and in­stead of stop­ping there, you take the \$5?

I don’t un­der­stand that point. you say “noth­ing stops you”, but that is only pos­si­ble if you could act con­trary to your own al­gorithm, no? Which makes no sense to me, un­less the same al­gorithm gives differ­ent out­comes for differ­ent in­puts, e.g. “if I sim­ply run the al­gorithm, I take \$10, but if I ex­am­ine the al­gorithm be­fore run­ning it and then run it, I take \$5″. But it doesn’t seem like the thing you mean, so I am con­fused.

What if you ex­am­ine your al­gorithm and find that it takes the \$5 in­stead?

How can it be pos­si­ble? if your ex­am­i­na­tion of your al­gorithm is ac­cu­rate, it gives the same out­come as mind­lessly run­ning it, with is tak­ing \$10, no?

It could be the same al­gorithm that takes the \$10, but you don’t know that, in­stead you ar­rive at the \$5 con­clu­sion us­ing rea­son­ing that could be im­pos­si­ble, but that you don’t know to be im­pos­si­ble, that you haven’t de­cided yet to make im­pos­si­ble.

So your rea­son­ing is in­ac­cu­rate, in that you ar­rive to a wrong con­clu­sion about the al­gorithm out­put, right? You just don’t know where the er­ror lies, or even that there is an er­ror to be­gin with. But in this case you would ar­rive to a wrong con­clu­sion about the same al­gorithm run by a differ­ent agent, right? So there is noth­ing spe­cial about it be­ing your own al­gorithm and not some­one else’s. If so, the is­sue is re­duced to find­ing an ac­cu­rate al­gorithm anal­y­sis tool, for an al­gorithm that demon­stra­bly halts in a very short time, pro­duc­ing one of the two pos­si­ble out­comes. This seems to have lit­tle to do with de­ci­sion the­ory is­sues, so I am lost as to how this is rele­vant to the situ­a­tion.

I am clearly miss­ing some of your logic here, but I still have no idea what the miss­ing piece is, un­less it’s the liber­tar­ian free will thing, where one can act con­trary to one’s pro­gram­ming. Any fur­ther help would be greatly ap­pre­ci­ated.

• I un­der­stand that there is no point ex­am­in­ing one’s al­gorithm if you already ex­e­cute it and see what it does.

Rather there is no point if you are not go­ing to do any­thing with the re­sults of the ex­am­i­na­tion. It may be use­ful if you make the de­ci­sion based on what you ob­serve (about how you make the de­ci­sion).

you say “noth­ing stops you”, but that is only pos­si­ble if you could act con­trary to your own al­gorithm, no?

You can, for a cer­tain value of “can”. It won’t have hap­pened, of course, but you may still de­cide to act con­trary to how you act, two differ­ent out­comes of the same al­gorithm. The con­tra­dic­tion proves that you didn’t face the situ­a­tion that trig­gers it in ac­tu­al­ity, but the con­tra­dic­tion re­sults pre­cisely from de­cid­ing to act con­trary to the ob­served way in which you act, in a situ­a­tion that a pri­ori could be ac­tual, but is ren­dered coun­ter­log­i­cal as a re­sult of your de­ci­sion. If in­stead you af­firm the ob­served ac­tion, then there is no con­tra­dic­tion and so it’s pos­si­ble that you have faced the situ­a­tion in ac­tu­al­ity. Thus the “chicken rule”, play­ing chicken with the uni­verse, mak­ing the pre­sent situ­a­tion im­pos­si­ble when you don’t like it.

You don’t know that it’s in­ac­cu­rate, you’ve just run the com­pu­ta­tion and it said \$5. Maybe this didn’t ac­tu­ally hap­pen, but you are con­sid­er­ing this situ­a­tion with­out know­ing if it’s ac­tual. If you ig­nore the com­pu­ta­tion, then why run it? If you run it, you need re­sponses to all pos­si­ble re­sults, and all pos­si­ble re­sults ex­cept one are not ac­tual, yet you should be ready to re­spond to them with­out know­ing which is which. So I’m dis­cussing what you might do for the re­sult that says that you take the \$5. And in the end, the use you make of the re­sults is by choos­ing to take the \$5 or the \$10.

This map from pre­dic­tions to de­ci­sions could be any­thing. It’s triv­ial to write an al­gorithm that in­cludes such a map. Of course, if the map di­ag­o­nal­izes, then the pre­dic­tor will fail (won’t give a pre­dic­tion), but the map is your rea­son­ing in these hy­po­thet­i­cal situ­a­tions, and the fact that the map may say any­thing cor­re­sponds to the fact that you may de­cide any­thing. The map doesn’t have to be iden­tity, de­ci­sion doesn’t have to re­flect pre­dic­tion, be­cause you may write an al­gorithm where it’s not iden­tity.

• You can, for a cer­tain value of “can”. It won’t have hap­pened, of course, but you may still de­cide to act con­trary to how you act, two differ­ent out­comes of the same al­gorithm.

This con­fuses me even more. You can imag­ine act con­trary to your own al­gorithm, but the imag­in­ing differ­ent pos­si­ble out­comes is a side effect of run­ning the main al­gorithm that takes \$10. It is never the out­come of it. Or an out­come. Since you know you will end up tak­ing \$10, I also don’t un­der­stand the idea of play­ing chicken with the uni­verse. Are there any refer­ences for it?

You don’t know that it’s in­ac­cu­rate, you’ve just run the com­pu­ta­tion and it said \$5.

Wait, what? We started with the as­sump­tion that ex­am­in­ing the al­gorithm, or run­ning it, shows that you will take \$10, no? I guess I still don’t un­der­stand how

What if you see that your al­gorithm leads to tak­ing the \$10 and in­stead of stop­ping there, you take the \$5?

is even pos­si­ble, or worth con­sid­er­ing.

This map from pre­dic­tions to de­ci­sions could be any­thing.

Hmm, maybe this is where I miss some of the logic. If the pre­dic­tions are ac­cu­rate, the map is bi­jec­tive. If the pre­dic­tions are in­ac­cu­rate, you need a bet­ter al­gorithm anal­y­sis tool.

The map doesn’t have to be iden­tity, de­ci­sion doesn’t have to re­flect pre­dic­tion, be­cause you may write an al­gorithm where it’s not iden­tity.

To me this screams “get a bet­ter al­gorithm an­a­lyzer!” and has noth­ing to do with whether it’s your own al­gorithm, or some­one else’s. Can you maybe give an ex­am­ple where one ends up in a situ­a­tion where there is no ob­vi­ous al­gorithm an­a­lyzer one can ap­ply?