It seems to me that the following is a good answer to the “but you’re only in this world, and in this world if you do X you will die, and that’s always sufficient reason not to do X” arguments, at least in some cases:
Suppose I become aware that in the future I might be subject to Mechanical Blackmail, and I am thinking now about what to do in that situation. All the outcomes are pretty bad. But I realise—again, this is well in advance of the actual blackmail—that if I am able to commit myself to not folding, in a way I will actually stick to once blackmailed, then it becomes extremely unlikely that I ever get blackmailed.
If the various probabilities and outcomes are suitably arranged (probability of getting blackmailed if I so commit is very low, etc.) then making such a commitment is greatly to my benefit.
I could do this by introducing some physical mechanism that stops me folding when blackmailed. I could do it by persuading myself that the gods will be angry and condemn me to eternal hell if I fold. I could do it by adopting FDT wholeheartedly and using its algorithms. I could do it by some sort of hypnosis that makes me unable to fold when the time comes. The effects on my probability of getting blackmailed, and my probability of the Bad Blackmail Outcome, don’t depend on which of these I’m doing; all that matters is that once that decision has to be made I definitely won’t choose to fold.
And here’s the key thing: you can’t just isolate the decision to be made once blackmailed and say “so what should you do then?”, given the possibility that you can do better overall, at an earlier point, by taking that choice away from your future self.
Sure, the best option of all would be to make that definitely-not-to-be-broken commitment, so that you almost certainly don’t get blackmailed, and then break it if you do get blackmailed. Except that that is not a possibility. If you would break the commitment once blackmailed, then you haven’t actually made the commitment, and the blackmailer will realise it, and you will get blackmailed.
So the correct way to look at the “choice” being made by an FDT agent after the blackmail happens is this: they aren’t really making a choice; they already made the choice; the choice was to put themselves into a mental state where they would definitely not fold once blackmailed. To whatever extent they feel themselves actually free to fold at this point and might do it, they (1) were never really an FDT agent in the first place and (2) were always going to get blackmailed. The only people who benefit from almost certainly not getting blackmailed on account of their commitment not to fold are the ones who really, effectively committed to not folding, and it’s nonsense to blame them for following through on that commitment because the propensity to follow through on that commitment is the only thing that ever made the thing likely to work.
The foregoing is not, as I understand it, the Official FDT Way of looking at it. The perspective I’m offering here lets you say “yes, refusing to fold in this decision is in some sense a bad idea, but unfortunately for present-you you already sacrificed the option of folding, so now you can’t, and even though that means you’re making a bad decision now it was worth it overall” whereas an actual FDT agent might say instead “yeah, of course I can choose to fold here, but I won’t”. But the choices that emerge are the same either way.
“yes, refusing to fold in this decision is in some sense a bad idea, but unfortunately for present-you you already sacrificed the option of folding, so now you can’t, and even though that means you’re making a bad decision now it was worth it overall”
Right, and what I’m pointing to is that this ends up being a place where, when an actual human out in the real world gets themselves into it mentally, it gets them hurt because they’re essentially forced into continuing to implement the precommitment even though it is a bad idea for present them and thus all temporally downstream versions of them which could exist. That’s why I used a fatal scenario, because it very obviously cuts all future utility to zero in a way I was hoping would help make it more obvious how the decision theory was failing to account for.
I could characterize it roughly as arising from the amount of “non-determinsm” in the universe, or as “predictive inaccuracy” in other humans, but the end result is that it gets someone into a bad place when their timeless FDT decisions fail to place them into a world where they don’t get blackmailed.
That’s why I used a fatal scenario, because it very obviously cuts all future utility to zero
I don’t understand why you think a decision resulting in some person’s or agent’s death “cuts all future utility to zero”. Why do you think choosing one’s death is always a mistake?
It seems to me that the following is a good answer to the “but you’re only in this world, and in this world if you do X you will die, and that’s always sufficient reason not to do X” arguments, at least in some cases:
Suppose I become aware that in the future I might be subject to Mechanical Blackmail, and I am thinking now about what to do in that situation. All the outcomes are pretty bad. But I realise—again, this is well in advance of the actual blackmail—that if I am able to commit myself to not folding, in a way I will actually stick to once blackmailed, then it becomes extremely unlikely that I ever get blackmailed.
If the various probabilities and outcomes are suitably arranged (probability of getting blackmailed if I so commit is very low, etc.) then making such a commitment is greatly to my benefit.
I could do this by introducing some physical mechanism that stops me folding when blackmailed. I could do it by persuading myself that the gods will be angry and condemn me to eternal hell if I fold. I could do it by adopting FDT wholeheartedly and using its algorithms. I could do it by some sort of hypnosis that makes me unable to fold when the time comes. The effects on my probability of getting blackmailed, and my probability of the Bad Blackmail Outcome, don’t depend on which of these I’m doing; all that matters is that once that decision has to be made I definitely won’t choose to fold.
And here’s the key thing: you can’t just isolate the decision to be made once blackmailed and say “so what should you do then?”, given the possibility that you can do better overall, at an earlier point, by taking that choice away from your future self.
Sure, the best option of all would be to make that definitely-not-to-be-broken commitment, so that you almost certainly don’t get blackmailed, and then break it if you do get blackmailed. Except that that is not a possibility. If you would break the commitment once blackmailed, then you haven’t actually made the commitment, and the blackmailer will realise it, and you will get blackmailed.
So the correct way to look at the “choice” being made by an FDT agent after the blackmail happens is this: they aren’t really making a choice; they already made the choice; the choice was to put themselves into a mental state where they would definitely not fold once blackmailed. To whatever extent they feel themselves actually free to fold at this point and might do it, they (1) were never really an FDT agent in the first place and (2) were always going to get blackmailed. The only people who benefit from almost certainly not getting blackmailed on account of their commitment not to fold are the ones who really, effectively committed to not folding, and it’s nonsense to blame them for following through on that commitment because the propensity to follow through on that commitment is the only thing that ever made the thing likely to work.
The foregoing is not, as I understand it, the Official FDT Way of looking at it. The perspective I’m offering here lets you say “yes, refusing to fold in this decision is in some sense a bad idea, but unfortunately for present-you you already sacrificed the option of folding, so now you can’t, and even though that means you’re making a bad decision now it was worth it overall” whereas an actual FDT agent might say instead “yeah, of course I can choose to fold here, but I won’t”. But the choices that emerge are the same either way.
Right, and what I’m pointing to is that this ends up being a place where, when an actual human out in the real world gets themselves into it mentally, it gets them hurt because they’re essentially forced into continuing to implement the precommitment even though it is a bad idea for present them and thus all temporally downstream versions of them which could exist. That’s why I used a fatal scenario, because it very obviously cuts all future utility to zero in a way I was hoping would help make it more obvious how the decision theory was failing to account for.
I could characterize it roughly as arising from the amount of “non-determinsm” in the universe, or as “predictive inaccuracy” in other humans, but the end result is that it gets someone into a bad place when their timeless FDT decisions fail to place them into a world where they don’t get blackmailed.
I don’t understand why you think a decision resulting in some person’s or agent’s death “cuts all future utility to zero”. Why do you think choosing one’s death is always a mistake?