Zvi, it’s great that you vouch for MIRI! But I’m not sure it’s always a good idea to use logical decision theories in real life. It’s kind of an open problem actually. Wei had a great counterexample called 2TDT-1CDT. Imagine a tournament of blind one-shot prisoner’s dilemma played among many TDT agents and one CDT agent. The TDT agents will cooperate because the opponent is likely to be TDT, while the CDT agent will defect and win. The relevant math idea is evolutionarily stable strategy, I think.
I agree that it’s an open problem how best to make decisions in life. I do think that it’s obvious that FDT>CDT as a policy, even if (or especially if!) you’re a human and doing either one approximately. To put it more simply, ignoring the things FDT cares about and CDT does not care about seems like a pretty obvious mistake, and also a pretty easy way, if adopted by too many people, to have things go quite badly.
Interesting example there! But I don’t think that example shows CDT>FDT even in this isolated designed-to-counterexample-me case. And I think that the fact that we’re trying to construct an example where CDT>FDT should in and of itself be strong evidence of which you should choose.
One of these two things happens, depending on your perspective:
Zvi runs FDT. Omega approaches Zvi and presents this situation. I think to myself “Is my decision to submit an FDT bot correlated to Omega’s decision to submit FDT bots?” And my response is, no. So I submit CDT, or probably just “D”. So CDT=FDT here because we both submit the same bot.
Same setup. Zvi submits an FDT bot with the knowledge there are two bots submitted by Omega that were chosen first, this bot realizes it isn’t correlated to Omega’s bots (since they coopreate even if they know my bot defects), so it defects. Omega’s bots realize they are correlated, so they cooperate. Same result. Setup matters.
The FDT agents are still making the right decision and still winning. Yes, the CDT agent doing better is annoying, but that’s by definition not in your utility function, and you don’t lose because someone else did better. The paperclip maximizer maximized paperclips, the staple maximizer maximized staples, versus either of them defecting or self-modifying to CDT or what not which goes very badly for them. The FDT agents are doing the right thing once the tournament starts. Note that if they submit one CDT and one FDT, you are forced to submit FDT, which is now correlated to the other FDT agent, which means the two of you now coopreate. So Omega isn’t maximizing by submitting two FDT agents, but Omega rarely does. Omega is a shifty one. Odd motivations.
Zvi, it’s great that you vouch for MIRI! But I’m not sure it’s always a good idea to use logical decision theories in real life. It’s kind of an open problem actually. Wei had a great counterexample called 2TDT-1CDT. Imagine a tournament of blind one-shot prisoner’s dilemma played among many TDT agents and one CDT agent. The TDT agents will cooperate because the opponent is likely to be TDT, while the CDT agent will defect and win. The relevant math idea is evolutionarily stable strategy, I think.
I agree that it’s an open problem how best to make decisions in life. I do think that it’s obvious that FDT>CDT as a policy, even if (or especially if!) you’re a human and doing either one approximately. To put it more simply, ignoring the things FDT cares about and CDT does not care about seems like a pretty obvious mistake, and also a pretty easy way, if adopted by too many people, to have things go quite badly.
Interesting example there! But I don’t think that example shows CDT>FDT even in this isolated designed-to-counterexample-me case. And I think that the fact that we’re trying to construct an example where CDT>FDT should in and of itself be strong evidence of which you should choose.
One of these two things happens, depending on your perspective:
Zvi runs FDT. Omega approaches Zvi and presents this situation. I think to myself “Is my decision to submit an FDT bot correlated to Omega’s decision to submit FDT bots?” And my response is, no. So I submit CDT, or probably just “D”. So CDT=FDT here because we both submit the same bot.
Same setup. Zvi submits an FDT bot with the knowledge there are two bots submitted by Omega that were chosen first, this bot realizes it isn’t correlated to Omega’s bots (since they coopreate even if they know my bot defects), so it defects. Omega’s bots realize they are correlated, so they cooperate. Same result. Setup matters.
The FDT agents are still making the right decision and still winning. Yes, the CDT agent doing better is annoying, but that’s by definition not in your utility function, and you don’t lose because someone else did better. The paperclip maximizer maximized paperclips, the staple maximizer maximized staples, versus either of them defecting or self-modifying to CDT or what not which goes very badly for them. The FDT agents are doing the right thing once the tournament starts. Note that if they submit one CDT and one FDT, you are forced to submit FDT, which is now correlated to the other FDT agent, which means the two of you now coopreate. So Omega isn’t maximizing by submitting two FDT agents, but Omega rarely does. Omega is a shifty one. Odd motivations.