It is not a serious problem if your epistemology gives you the wrong answer in extremely unlikely worlds (ie ones where you survived 1000 rounds of Russian Roulette). Don’t optimize for extremely unlikely scenarios.
We can make this point even more extreme by playing a game like the “unexpected hanging paradox,” where surprising the prisoner most of the time is only even possible if you pay for it in the coin of not surprising them at all some of the time.
I disagree; this might have real world implications. For example, the recent OpenPhil report on Semi-informative Priors for AI timelines updates on the passage of time, but if we model creating AGI as playing Russian roulette*, perhaps one shouldn’t update on the passage of time.
* I.e., AGI in the 2000s might have lead to an existential catastrophe due to underdeveloped safety theory
It is not a serious problem if your epistemology gives you the wrong answer in extremely unlikely worlds (ie ones where you survived 1000 rounds of Russian Roulette). Don’t optimize for extremely unlikely scenarios.
We can make this point even more extreme by playing a game like the “unexpected hanging paradox,” where surprising the prisoner most of the time is only even possible if you pay for it in the coin of not surprising them at all some of the time.
I disagree; this might have real world implications. For example, the recent OpenPhil report on Semi-informative Priors for AI timelines updates on the passage of time, but if we model creating AGI as playing Russian roulette*, perhaps one shouldn’t update on the passage of time.
* I.e., AGI in the 2000s might have lead to an existential catastrophe due to underdeveloped safety theory
That is not a similar situation. In the AI situation, your risks obviously increase over time.