Here’s a compressed attempt at deconfusion, I’m happy to be corrected where I’m wrong/further confused[1].
If you have a “rich” set of hypotheses, say for example computable programs, then using standard simplicity prior gives high weight to programs that take a valuable output with linear utility (the “numéraire”) and just copy it over and over again onto the tape[2]. Indeed, one can create a sequence of such programs that grow at a very fast rate, comparable to the busy beaver sequence (could be called the “copy beaver”). Since the simplicity prior over programs falls exponentially, and the rate at which the copy beaver grows is muchmuch faster than that, expected utility diverges.
So, lesson: A pascal’s mugging is a sequence of hypotheses which diverge on priors.
In order for this not to be the case, one can either: have the utility of any valuable thing, repeated, grow slower than the inverse of the copy beaver (very slowly), or adjust the prior so that high-utility hypotheses receive a proportionally smaller prior probability (the leverage penalty).
One can interpolate between these two solutions, too (both discounting anything valuable and applying a partial leverage penalty)[3], though my guess is that doesn’t buy very much (roughly an exponential discount on both, which in the context of busy-beaver-like functions amounts to almost nothing).
One thing that can’t happen with Bayesian reasoners, as far as I can tell, is that additional evidence upweighs non-mugging sequences of hypotheses into mugging territory, even using filtered evidence.
Assuming that some Bayesian reasoner has an unmuggable prior (that is ), then we want new evidence to guarantee . But each bitstring of evidence can only upweigh any hypothesis by . The mugger can give us more and more strings of evidence, but that’s (in the context of a Bayesian) just the mugger “providing us the goods”. E.g. if the mugger computes the copy beaver for us and then displays us the evidence, then he’s done the work for us and convincingly created & shown us the world which we value highly.
(This applies to finite bitstrings. If we allow proofs to enter the picture, then we are in trouble.)
So, another lesson: A Bayesian can’t be convinced into becoming muggable by evidence, even filtered evidence.
We can even reason about this from a more restricted hypothesis class. E.g. if we restrict ourselves to hypotheses that take at most steps to run for programs of length , then we don’t need to do any discounting, so Bayesians with that kind of restricted hypothesis class are already immune from mugging at the get-go, though they may be surprised by the computational capacity of the universe.
In general, the more powerful your hypothesis class, the more you need to discount.
Here’s something I’m less sure about, but it still seemed worth mentioning: If you are unmuggable, but then you expand your hypothesis class, you can suddenly become muggable, especially if your new hypotheses are much more computationally powerful than your previous ones. So, if someone shows up and starts telling you about how muchlargertheuniverse is than you previously thought, it might be that that you’re encountering Pascal’s mugger.
And: trying to be mugging-immune is probably one of the better reasons for justifying scope neglect than I’ve previously encountered.
Huh, yeah, skimming over it looks basically like what I’ve written, with more detail & in a different frame. Especially this post, though it misses the leverage penality for the prior, and (as far as I can tell?) restricting the hypothesis class.
Pascal’s mugging is commonly misunderstood.
Here’s a compressed attempt at deconfusion, I’m happy to be corrected where I’m wrong/further confused[1].
If you have a “rich” set of hypotheses, say for example computable programs, then using standard simplicity prior gives high weight to programs that take a valuable output with linear utility (the “numéraire”) and just copy it over and over again onto the tape[2]. Indeed, one can create a sequence of such programs that grow at a very fast rate, comparable to the busy beaver sequence (could be called the “copy beaver”). Since the simplicity prior over programs falls exponentially, and the rate at which the copy beaver grows is much much faster than that, expected utility diverges.
So, lesson: A pascal’s mugging is a sequence of hypotheses which diverge on priors.
In order for this not to be the case, one can either: have the utility of any valuable thing, repeated, grow slower than the inverse of the copy beaver (very slowly), or adjust the prior so that high-utility hypotheses receive a proportionally smaller prior probability (the leverage penalty).
One can interpolate between these two solutions, too (both discounting anything valuable and applying a partial leverage penalty)[3], though my guess is that doesn’t buy very much (roughly an exponential discount on both, which in the context of busy-beaver-like functions amounts to almost nothing).
One thing that can’t happen with Bayesian reasoners, as far as I can tell, is that additional evidence upweighs non-mugging sequences of hypotheses into mugging territory, even using filtered evidence.
Assuming that some Bayesian reasoner has an unmuggable prior (that is
(This applies to finite bitstrings. If we allow proofs to enter the picture, then we are in trouble.)
So, another lesson: A Bayesian can’t be convinced into becoming muggable by evidence, even filtered evidence.
We can even reason about this from a more restricted hypothesis class. E.g. if we restrict ourselves to hypotheses that take at most
In general, the more powerful your hypothesis class, the more you need to discount.
Here’s something I’m less sure about, but it still seemed worth mentioning: If you are unmuggable, but then you expand your hypothesis class, you can suddenly become muggable, especially if your new hypotheses are much more computationally powerful than your previous ones. So, if someone shows up and starts telling you about how much larger the universe is than you previously thought, it might be that that you’re encountering Pascal’s mugger.
And: trying to be mugging-immune is probably one of the better reasons for justifying scope neglect than I’ve previously encountered.
Some people probably understand Pascal’s mugging much better than me, but I haven’t found a text explaining the mugging I consider up to my standards.
Hm, related to the repugnant conclusion? Relevantly, saturationism?
Yielding a continuum of solutions.
Related prior writing on this subject by me: https://forum.effectivealtruism.org/posts/Z6Ssc79vH496bLqL9/tiny-probabilities-of-vast-utilities-a-problem-for-long
Huh, yeah, skimming over it looks basically like what I’ve written, with more detail & in a different frame. Especially this post, though it misses the leverage penality for the prior, and (as far as I can tell?) restricting the hypothesis class.