“Right, this is the real core of Pascal’s Mugging [...]. For aggregative utility functions over a model of the environment which e.g. treat all sentient beings (or all paperclips) as having equal value without diminishing marginal returns, and all epistemic models which induce simplicity-weighted explanations of sensory experience, all decisions will be dominated by tiny variances in the probability of extremely unlikely hypotheses because the “model size” of a hypothesis can grow Busy-Beaver faster than its Kolmogorov complexity.”
I think others have expressed doubts that the promised utility of hypotheses should be able to grow in Busy-Beaver fashion faster than a properly updated probability, but I don’t see a similar argument on your list. It seems to have been taken for granted because a simple hypothesis clearly can describe a very large utility.
But what is the probability that any process, including a Matrix Lord, can successfully perform O(3^^^3) computations/operations/actions? Even a Matrix Lord should have a mean time to failure (MTTF) and the estimate of MTTF should be directly derivable from the complexity of the Matrix Lord. A Matrix Lord of complexity O(2^256) should have a MTTF like O(2^256) operations, not O(3^^^3) operations. Even if each operation produces 1 utilon without regard to whether the process completes or not that limits the expected number of utilons is 1MTTF or O(2^256), making the expected value O(2^256) P(Matrix Lord will is telling the truth | Matrix Lord of complexity 2^256 exists).
This doesn’t work if the MTTF increases over time due perhaps to a self-improving and self-sustaining process, but that actually makes sense. I’m far more likely to believe a mugger that says “Give me $5 or I’m going to release a self-replicating nanobot that will turn the universe into grey goo” or “Give me $5 or I turn on this UFAI” than one that says “Give me $5 or I’ll switch off the Matrix that’s running your universe.” “Give me $5 or I’ll start an unreasonably long process that eventually will produce so much negative utility that it overrides all your probability estimation ability” just makes me think that whatever process the mugger may have will fail long before the expected value of accepting would become positive.
The limit as predicted MTTF goes to infinity is probably just Pascal’s Wager. If you believe that an entity can keep a Matrix running infinitely long, that entity can affect your utility infinitely. That probably requires an (eventually) infinitely complex entity so that it can continue even if any finite number of its components fail. That leaves the door open to being mugged by a self-improving process in an infinite universe, but I think that also makes sense. Bound your utility if you’re worried about infinite utility.
I think others have expressed doubts that the promised utility of hypotheses should be able to grow in Busy-Beaver fashion faster than a properly updated probability, but I don’t see a similar argument on your list. It seems to have been taken for granted because a simple hypothesis clearly can describe a very large utility.
But what is the probability that any process, including a Matrix Lord, can successfully perform O(3^^^3) computations/operations/actions? Even a Matrix Lord should have a mean time to failure (MTTF) and the estimate of MTTF should be directly derivable from the complexity of the Matrix Lord. A Matrix Lord of complexity O(2^256) should have a MTTF like O(2^256) operations, not O(3^^^3) operations. Even if each operation produces 1 utilon without regard to whether the process completes or not that limits the expected number of utilons is 1MTTF or O(2^256), making the expected value O(2^256) P(Matrix Lord will is telling the truth | Matrix Lord of complexity 2^256 exists).
This doesn’t work if the MTTF increases over time due perhaps to a self-improving and self-sustaining process, but that actually makes sense. I’m far more likely to believe a mugger that says “Give me $5 or I’m going to release a self-replicating nanobot that will turn the universe into grey goo” or “Give me $5 or I turn on this UFAI” than one that says “Give me $5 or I’ll switch off the Matrix that’s running your universe.” “Give me $5 or I’ll start an unreasonably long process that eventually will produce so much negative utility that it overrides all your probability estimation ability” just makes me think that whatever process the mugger may have will fail long before the expected value of accepting would become positive.
The limit as predicted MTTF goes to infinity is probably just Pascal’s Wager. If you believe that an entity can keep a Matrix running infinitely long, that entity can affect your utility infinitely. That probably requires an (eventually) infinitely complex entity so that it can continue even if any finite number of its components fail. That leaves the door open to being mugged by a self-improving process in an infinite universe, but I think that also makes sense. Bound your utility if you’re worried about infinite utility.