How is “Given an opponent that defects against Defectbot, PrudentBot’s behavior is to cooperate with that program IFF PrudentBot can prove that it cooperates with that program” proven? (That is the case stated where PrudentBot plays against itself)
Has it been shown in general that there exists a proof in PA that all agent Y cooperates with PrudentBot for complicated agents Y?
You need an opponent that provably defects against DefectBot in PA+Con(PA). For instance, PrudentBot will not find mutual cooperation with the agent that cooperates with X iff PA+Con(PA) proves X cooperates with it.
(This is another reason why it’s basically better in practice to use a version of PrudentBot that looks to prove in PA that the opponent cooperates against it, but prove in PA+N for some large N that the opponent defects against DefectBot.)
As for your second question: no, that’s false. You can’t compel cooperation from arbitrary agents; you can only hope to give the best response against them (so that you don’t get exploited, you exploit them if possible, and you grab mutual cooperation if that’s the best possible).
“For every decision-making process X for which there is a proof in PA that my opponent cooperates against X, if X defects against my opponent, (implement X; break) if no X defects but some exist, implement any X. If no X exists, defect.”
This is basically the idea behind the earlier Masquerade algorithm. The first tricky bit is that your opponent is under no obligation to treat you the same way that it treats X. The second tricky bit is that you can’t search over all X within any formal system, because the class of cooperators is undecidable, and the more you search over, the stronger a formal system your opponent needs to figure out what you’re actually doing.
(It’s totally worth thinking about these things, of course!)
Yeah, I realized that I was talking about behavior, but that my opponent isn’t limited to my behavior and can make decisions based on my nature. I was trying to design a program using sufficiently advanced magic that would defeat the best program I think I can create, and I succeeded. However, it isn’t competitive against other programs which operate on the same level.
It does cooperate with programs one level higher than it, but those programs defect from it.
How is “Given an opponent that defects against Defectbot, PrudentBot’s behavior is to cooperate with that program IFF PrudentBot can prove that it cooperates with that program” proven? (That is the case stated where PrudentBot plays against itself)
Has it been shown in general that there exists a proof in PA that all agent Y cooperates with PrudentBot for complicated agents Y?
You need an opponent that provably defects against DefectBot in PA+Con(PA). For instance, PrudentBot will not find mutual cooperation with the agent that cooperates with X iff PA+Con(PA) proves X cooperates with it.
(This is another reason why it’s basically better in practice to use a version of PrudentBot that looks to prove in PA that the opponent cooperates against it, but prove in PA+N for some large N that the opponent defects against DefectBot.)
As for your second question: no, that’s false. You can’t compel cooperation from arbitrary agents; you can only hope to give the best response against them (so that you don’t get exploited, you exploit them if possible, and you grab mutual cooperation if that’s the best possible).
“For every decision-making process X for which there is a proof in PA that my opponent cooperates against X, if X defects against my opponent, (implement X; break) if no X defects but some exist, implement any X. If no X exists, defect.”
Reflective consistency and self-modifying FTW.
Not sure if that’s computable in finite time.
This is basically the idea behind the earlier Masquerade algorithm. The first tricky bit is that your opponent is under no obligation to treat you the same way that it treats X. The second tricky bit is that you can’t search over all X within any formal system, because the class of cooperators is undecidable, and the more you search over, the stronger a formal system your opponent needs to figure out what you’re actually doing.
(It’s totally worth thinking about these things, of course!)
Yeah, I realized that I was talking about behavior, but that my opponent isn’t limited to my behavior and can make decisions based on my nature. I was trying to design a program using sufficiently advanced magic that would defeat the best program I think I can create, and I succeeded. However, it isn’t competitive against other programs which operate on the same level.
It does cooperate with programs one level higher than it, but those programs defect from it.