A mugger appears and says “For $5 I’ll offer you a set of deals from which you can pick any one. Each deal, d(N), will be N bits in length and I guarantee that if you accept d(N) I will run UTM(d(N)) on my hypercomputer, where UTM() is a function implementing a Universal Turing Machine. If UTM(d(N)) halts you will increase your utility by the number of bits written to the tape by UTM(d(N)). If UTM(d(N)) does not halt, I’ll just keep your $5. Which deal would you like to accept?”
That’s unclear. Does the mugger show you the set of deals before you accept the offer or later? If he shows you later, he can trivially win by presenting you the set { ‘while (true) {printf(“I Go Chop Your Dollar\n”);}’ }
where p(d(N)) is the probability of accepting d(N)
d(N) either halts or it doesn’t. There is nothing probabilistic about it. Are you talking about the prior probability about its termination before you see d(N), or after you see it (since you can’t compute its termination in the general case)?
p(UTM(d(N)) = BB(N)) * BB(N) > 0. To paraphrase: Even though the likelihood of being offered a deal that actually yields BB(N) utilons is incredibly small, the fact that BB(X) grows at least as fast as any function of length X means that, at minimum, an agent that can be emulated on a UTM by a program of M bits cannot provide a non-zero probability of d(M) such that the expected utility of accepting d(M) is negative.
To paraphrase nothing. p(UTM(d(N)) = BB(N)) BB(N) > 0 is trivially true because the product of two positive numbers is always positive. But this tells you nothing about the magnitude of the value. In particular, it doesn’t follow that p(UTM(d(N)) = BB(N)) BB(N) >= U($5)
Since p(“UTM(d(X)) = BB(X)”) >= 2^-X for d(X) with bits selected at random it doesn’t make sense for the agent to assign p(d(X))=0 unless the agent has other reasons to absolutely distrust the mugger.
I wonder what that reasons to distrust the mugger may be… Perhaps that he wants to win your $5 rather than giving away utility?
That’s unclear. Does the mugger show you the set of deals before you accept the offer or later? If he shows you later, he can trivially win by presenting you the set { ‘while (true) {printf(“I Go Chop Your Dollar\n”);}’ }
I think that even if the mugger doesn’t show you the set of deals before you accept there is still a non-zero probability that some of the deals have BB(N) utility unless the agent has p(mugger_is_perfect) = 1. I agree that I should have specified that in the description.
d(N) either halts or it doesn’t. There is nothing probabilistic about it. Are you talking about the prior probability about its termination before you see d(N), or after you see it (since you can’t compute its termination in the general case)?
The idea was to talk about the agent’s total estimation of the probability that a particular deal yields BB(N) utils. Before seeing it the agent would only have a prior probability based perhaps on Solomonoff Induction, and after seeing it the agent could just run it for a while to make sure it didn’t halt quickly. Either way, the idea was that any probability estimate would be dwarfed by BB(N).
p(UTM(d(N)) = BB(N)) BB(N) > 0 is trivially true because the product of two positive numbers is always positive. But this tells you nothing about the magnitude of the value. In particular, it doesn’t follow that
p(UTM(d(N)) = BB(N)) BB(N) >= U($5)
That’s unclear. Does the mugger show you the set of deals before you accept the offer or later? If he shows you later, he can trivially win by presenting you the set { ‘while (true) {printf(“I Go Chop Your Dollar\n”);}’ }
d(N) either halts or it doesn’t. There is nothing probabilistic about it. Are you talking about the prior probability about its termination before you see d(N), or after you see it (since you can’t compute its termination in the general case)?
To paraphrase nothing.
p(UTM(d(N)) = BB(N)) BB(N) > 0 is trivially true because the product of two positive numbers is always positive. But this tells you nothing about the magnitude of the value. In particular, it doesn’t follow that
p(UTM(d(N)) = BB(N)) BB(N) >= U($5)
I wonder what that reasons to distrust the mugger may be… Perhaps that he wants to win your $5 rather than giving away utility?
I think that even if the mugger doesn’t show you the set of deals before you accept there is still a non-zero probability that some of the deals have BB(N) utility unless the agent has p(mugger_is_perfect) = 1. I agree that I should have specified that in the description.
The idea was to talk about the agent’s total estimation of the probability that a particular deal yields BB(N) utils. Before seeing it the agent would only have a prior probability based perhaps on Solomonoff Induction, and after seeing it the agent could just run it for a while to make sure it didn’t halt quickly. Either way, the idea was that any probability estimate would be dwarfed by BB(N).
You are correct.