Your comment probably got downvoted owing to your notoriety, not its content.
By total of 3 people, 1 of them definitely ArisKatsaris, and 2 others uncomfortable about the “sufficiently clever selfish agent”.
But somehow the asteroid example does not feel like Pascal’s mugging. Maybe because we are more sure in the accuracy of the probability estimate, whereas for Pascal’s mugging all we get is an upper boundary?
I am thinking speculation is the key, especially guided speculation. Speculation comes together with misevaluation of probability, misevaluation of consequences, misevaluation of utility of alternatives, and basically just about every issue with how humans process hypotheses can get utilized as the mugger is also human and would first try to self convince. In real world examples you can see tricks such as asking “why are you so certain in [negation of the proposition]”, only re-evaluating consequences of giving the money without re-evaluating consequences of keeping the money, trying to teach victims to evaluate claims in a way that is more susceptible (update one side of the utility comparison then act out of superfluous difference), and so on.
Low probabilities really are a red herring. Even if you were to do something formal, like Solomonoff induction (physics described with a Turing machine tape), you could do some ridiculous modifications to laws of physics which add invisible consequences to something, in as little as 10..20 extra bits. All actions would be dominated by the simplest modification to laws of physics which leaves most bits to making up huge consequences.
By total of 3 people, 1 of them definitely ArisKatsaris, and 2 others uncomfortable about the “sufficiently clever selfish agent”.
I am thinking speculation is the key, especially guided speculation. Speculation comes together with misevaluation of probability, misevaluation of consequences, misevaluation of utility of alternatives, and basically just about every issue with how humans process hypotheses can get utilized as the mugger is also human and would first try to self convince. In real world examples you can see tricks such as asking “why are you so certain in [negation of the proposition]”, only re-evaluating consequences of giving the money without re-evaluating consequences of keeping the money, trying to teach victims to evaluate claims in a way that is more susceptible (update one side of the utility comparison then act out of superfluous difference), and so on.
Low probabilities really are a red herring. Even if you were to do something formal, like Solomonoff induction (physics described with a Turing machine tape), you could do some ridiculous modifications to laws of physics which add invisible consequences to something, in as little as 10..20 extra bits. All actions would be dominated by the simplest modification to laws of physics which leaves most bits to making up huge consequences.