The qualm I have about it is that it teeters too close on conflating “I will not do this, even though I can’t refute your argument” and “I know you are wrong, even though I can’t refute your argument”.
This reminds me of the line of thinking I’ve learned to apply to some moral thought-experiments: “I am not capable of occupying the epistemic state that you propose. If I could, then yes, your proposed course of action would indeed be justified. But I can’t, so I’m not going to follow that course of action.”
Worked example: Bob notices that he is having thoughts leading to the conclusion that his neighbor’s newborn baby Alice will grow up to be the next Hitler. According to the “baby Hitler” thought-experiment, this means Bob should kill Alice to save millions of future lives. But Bob also knows that he’s thinking with a human brain, and schizophrenia is a thing that can happen to human brains, and the probability that he is having a schizophrenic-type delusion is much greater than the probability that baby Alice is actually the next Hitler. Therefore, Bob concludes that following the thought-experiment would lead him wrongly, and does not kill Alice.
Put another way: Thinking has a noise floor. A sensible agent should recognize that its own thinking has a noise floor, and avoid amplifying noise into plans.
This reminds me of the line of thinking I’ve learned to apply to some moral thought-experiments: “I am not capable of occupying the epistemic state that you propose. If I could, then yes, your proposed course of action would indeed be justified. But I can’t, so I’m not going to follow that course of action.”
Worked example: Bob notices that he is having thoughts leading to the conclusion that his neighbor’s newborn baby Alice will grow up to be the next Hitler. According to the “baby Hitler” thought-experiment, this means Bob should kill Alice to save millions of future lives. But Bob also knows that he’s thinking with a human brain, and schizophrenia is a thing that can happen to human brains, and the probability that he is having a schizophrenic-type delusion is much greater than the probability that baby Alice is actually the next Hitler. Therefore, Bob concludes that following the thought-experiment would lead him wrongly, and does not kill Alice.
Put another way: Thinking has a noise floor. A sensible agent should recognize that its own thinking has a noise floor, and avoid amplifying noise into plans.