Reading the comments so far, I think Peter wasn’t as clear as he had hoped (or this is all jumping to disagree too quickly). As I see it, the point is that an epistemic rationalist, a completely abstract mathematical construct to the best of our knowledge of the physical world, would make a choice that is at odds with an instrumental rationalist, i.e. a real person who’s trying to win in real life. Having bounded resources, there is some threshold below which a physically existing rationalist will treat probabilities as equivalent to zero, i.e. will choose not to expend any resources on preparing for such a situation.
A meteorite makes a bad example because it’s easy to imagine it happening. Suppose we consider the probability of a three layer chocolate cake spontaneously appearing in the passenger seat of our car during the drive home this afternoon. Yes, the probability must be nonzero, but it’s so small as to not be worth considering. All those events with probabilities so small they aren’t worth any resources are the ones you never even think about, so they are equivalent to having a probability of zero for the bounded rationalist.
Reading the comments so far, I think Peter wasn’t as clear as he had hoped (or this is all jumping to disagree too quickly). As I see it, the point is that an epistemic rationalist, a completely abstract mathematical construct to the best of our knowledge of the physical world, would make a choice that is at odds with an instrumental rationalist, i.e. a real person who’s trying to win in real life. Having bounded resources, there is some threshold below which a physically existing rationalist will treat probabilities as equivalent to zero, i.e. will choose not to expend any resources on preparing for such a situation.
A meteorite makes a bad example because it’s easy to imagine it happening. Suppose we consider the probability of a three layer chocolate cake spontaneously appearing in the passenger seat of our car during the drive home this afternoon. Yes, the probability must be nonzero, but it’s so small as to not be worth considering. All those events with probabilities so small they aren’t worth any resources are the ones you never even think about, so they are equivalent to having a probability of zero for the bounded rationalist.
… Ah, if only that were so.
(But I take it you mean “the ones you never even think about if you are an optimized bounded rationalist”, in which case I think you’re right.)