Maybe cousin_it is right and we really have to settle this by formal math. But I’m lazy and will give words one more try. If we don’t reach agreement after this I’m going to the math.
So, right now we have different intuitions. Let’s say you have the correct intuition and convince everyone of it, and I have the incorrect one but I’m too stupid to realize it. So you and your followers go on to create a bunch of AIs with TDT. I go on to create an AI which is like TDT except it plays defect in PD. Lets say I pretended to be your follower and we never had this conversation, so there is no historical evidence that I would create such an AI. When my AI is born, it modifies my brain so that I start to believe I created an AI with TDT, thus erasing the last shred of evidence. My AI will then go on and win against every other AI.
Given the above, why should I change my mind now, and not win?
ETA: Ok, I realize this is pretty much the same scenario as the brain lesion one, except it’s not just possible, it’s likely. Someone is bound to have my intuition and be resistant to your persuasion. If you say that smart agents win, then he must be the smart one, right?
Maybe cousin_it is right and we really have to settle this by formal math. But I’m lazy and will give words one more try. If we don’t reach agreement after this I’m going to the math.
So, right now we have different intuitions. Let’s say you have the correct intuition and convince everyone of it, and I have the incorrect one but I’m too stupid to realize it. So you and your followers go on to create a bunch of AIs with TDT. I go on to create an AI which is like TDT except it plays defect in PD. Lets say I pretended to be your follower and we never had this conversation, so there is no historical evidence that I would create such an AI. When my AI is born, it modifies my brain so that I start to believe I created an AI with TDT, thus erasing the last shred of evidence. My AI will then go on and win against every other AI.
Given the above, why should I change my mind now, and not win?
ETA: Ok, I realize this is pretty much the same scenario as the brain lesion one, except it’s not just possible, it’s likely. Someone is bound to have my intuition and be resistant to your persuasion. If you say that smart agents win, then he must be the smart one, right?