It occurs to me that the mugger is trying to get us to use a penalty from beyond our hypothesis space, to make decisions within the hypothesis space.
“I will torture 3^^^3 humans forever”, necessarily comes from a hypothesis space of at least size 3^^^3. However, our universe, at least as far as we know, has nowhere near 3^^^3 possible states available. A single one of the total number of states is the finest granularity available to use.
Consider the mugging using numbers from within our universe. We might get for example “I will torture 1e50 humans until the end of the universe”. This is something we can work with—getting a probability of order 1e-50 shouldn’t be terribly difficult.
In order to counter a 3^^^3 penalty, we need to look at the larger hypothesis space, not just what’s available in our current understanding of the universe.
It occurs to me that the mugger is trying to get us to use a penalty from beyond our hypothesis space, to make decisions within the hypothesis space.
“I will torture 3^^^3 humans forever”, necessarily comes from a hypothesis space of at least size 3^^^3. However, our universe, at least as far as we know, has nowhere near 3^^^3 possible states available. A single one of the total number of states is the finest granularity available to use.
Consider the mugging using numbers from within our universe. We might get for example “I will torture 1e50 humans until the end of the universe”. This is something we can work with—getting a probability of order 1e-50 shouldn’t be terribly difficult.
In order to counter a 3^^^3 penalty, we need to look at the larger hypothesis space, not just what’s available in our current understanding of the universe.