I think we could generalise problem 2 to be problematic for any decision theory XDT:
There are 10 boxes, numbered 1 to 10. You may only take one. Omega has (several times) run a simulated XDT agent on this problem. It then put a prize in the box which it determined was least likely to be taken by such an agent—or, in the case of a tie, in the box with the lowest index.
If agent X follows XDT, it has at best a 10% chance of winning. Any sufficiently resourceful YDT agent, however, could run a simulated XDT agent themselves, and figure out what Omega’s choice was without getting into an infinite loop.
Therefore, YDT performs better than XDT on this problem.
If I’m right, we may have shown the impossibility of a “best’ decision theory, no matter how meta you get (in a close analogy to Godelian incompleteness). If I’m wrong, what have I missed?
You’re right about problem 2 being a fully general counterargument, but your philosophical conclusion seems to be stopping too early. For example, can we define a class of “fair” problems that excludes problem 2?
It looks like the issue here is that while Omega is ostensibly not taking into account your decision theory, it implicitly is by simulating an XDT agent. So a first patch would be to define simulations of a specific decision theory (as opposed to simulations of a given agent) as “unfair”.
On the other hand, we can’t necessarily know if a given computation is effectively equivalent to simulating a given decision theory. Even if the string “TDT” is never encoded anywhere in Omega’s super-neurons, it might still be simulating a TDT agent, for example.
On the first hand again, it might be easy for most problems to figure out whether anyone is implicitly favouring one DT over another, and thus whether they’re “fair”.
One possible place to look is that we’re allowing Omega access not just to a particular simulated decision of TDT, but to the probabilities with which it makes these decisions. If we force it to simulate TDT many times and sample to learn what the probabilities are, it can’t detect the exact balance for which it does deterministic symmetry breaking, and the problem goes away.
This solution occurred to me because this forces Omega to have something like a continuous behaviour response to changes in the probabilities of different TDT outputs, and it seems possible given that to imagine a proof that a fixed point must exist.
Fair point—how does Omega tell when the sim’s choosing probabilities are exactly equal? Well I was thinking that Omega could prove they are equal (by analysing the simulation’s behaviour, and checking where it calls on random bits). Or if it can’t do that, then it can just check that the choice frequencies are “statistically equal” (i.e. no significant differences after a billion runs, say) and treat them as equal for the tie-breaker rule. The “statistically equal” approach might give the TDT agent a very slightly higher than 10% chance of winning the money, though I haven’t analysed this in any detail.
If the subject can know the exact code of TDT, Omega can know the exact code of TDT, and analyse it however it likes. That means it can know exactly where randomness is invoked—why would it have to sample?
This was my first thought: Omega can just prove the choosing probabilities are equal. However, it’s not totally straightforward, because the sim could sample more random bits depending on the results of its first random bits, and so on, leading to an exponentially growing outcome tree of possibilities, with no upper size bound to the length of the tree. There might not be an easy proof of equality in that case. Sampling and statistical equality is the next best approach...
If I’m right, we may have shown the impossibility of a “best’ decision theory, no matter how meta you get (in a close analogy to Godelian incompleteness). If I’m wrong, what have I missed?
I would say that any such problem doesn’t show that there is no best decision theory, it shows that that class of problem cannot be used in the ranking.
Edited to add: Unless, perhaps, one can show that an instantiation of the problem with particular choice of (in this case decision theory, but whatever is varied) is particularly likely to be encountered.
To draw out the analogy to Godelian incompleteness, any computable decision theory is subject to the suggested attack of being given a “Godel problem″ like problem 1, just as any computable set of axioms for arithmetic has a Godel sentence. You can always make a new decision theory TDT’ that is TDT+ do the right thing for the Godel problem. But TDT’ has it’s own Godel problem of course. You can’t make a computable theory that says “do the right thing for all Godel probems”, if you try to do that it would not give you something computable. I’m sure this is all just restating what you had in mind, but I think it’s worth spelling out.
If you have some sort of oracle for the halting problem (i.e. a hypercomputer) and Omega doesn’t, he couldn’t simulate you, so you would presumably be able to always win fair problems. Otherwise the best thing you could hope for is to get the right answer whenever your computation halts, but fail to halt in your computation for some problems, such as your Godel problem. (A decision theory like this can still be given a Godel problem if Omega can solve the halting problem, “I simulated you and if you fail to halt on this problem...”). I wonder if TDT fails to halt for its Godel problem, or if some natural modification of it might have this property, but I don’t understand it well enough to guess.
I am less optimistic about revising “fair” to exclude Godel problems. The analogy would be proving Peano arithmetic is complete “except for things that are like Godel sentences.” I don’t know of any formalizations of the idea of “being a Godel sentence”.
If I’m right, we may have shown the impossibility of a “best’ decision theory, no matter how meta you get (in a close analogy to Godelian incompleteness). If I’m wrong, what have I missed?
You’re right. However, since all decision theories fail when confronted with their personal version of this problem, but may or may not fail in other problems, then some decision theories may be better than others. The one that is better than all the others is thus the “best” DT.
I think we could generalise problem 2 to be problematic for any decision theory XDT:
There are 10 boxes, numbered 1 to 10. You may only take one. Omega has (several times) run a simulated XDT agent on this problem. It then put a prize in the box which it determined was least likely to be taken by such an agent—or, in the case of a tie, in the box with the lowest index.
If agent X follows XDT, it has at best a 10% chance of winning. Any sufficiently resourceful YDT agent, however, could run a simulated XDT agent themselves, and figure out what Omega’s choice was without getting into an infinite loop.
Therefore, YDT performs better than XDT on this problem.
If I’m right, we may have shown the impossibility of a “best’ decision theory, no matter how meta you get (in a close analogy to Godelian incompleteness). If I’m wrong, what have I missed?
You’re right about problem 2 being a fully general counterargument, but your philosophical conclusion seems to be stopping too early. For example, can we define a class of “fair” problems that excludes problem 2?
It looks like the issue here is that while Omega is ostensibly not taking into account your decision theory, it implicitly is by simulating an XDT agent. So a first patch would be to define simulations of a specific decision theory (as opposed to simulations of a given agent) as “unfair”.
On the other hand, we can’t necessarily know if a given computation is effectively equivalent to simulating a given decision theory. Even if the string “TDT” is never encoded anywhere in Omega’s super-neurons, it might still be simulating a TDT agent, for example.
On the first hand again, it might be easy for most problems to figure out whether anyone is implicitly favouring one DT over another, and thus whether they’re “fair”.
One possible place to look is that we’re allowing Omega access not just to a particular simulated decision of TDT, but to the probabilities with which it makes these decisions. If we force it to simulate TDT many times and sample to learn what the probabilities are, it can’t detect the exact balance for which it does deterministic symmetry breaking, and the problem goes away.
This solution occurred to me because this forces Omega to have something like a continuous behaviour response to changes in the probabilities of different TDT outputs, and it seems possible given that to imagine a proof that a fixed point must exist.
Fair point—how does Omega tell when the sim’s choosing probabilities are exactly equal? Well I was thinking that Omega could prove they are equal (by analysing the simulation’s behaviour, and checking where it calls on random bits). Or if it can’t do that, then it can just check that the choice frequencies are “statistically equal” (i.e. no significant differences after a billion runs, say) and treat them as equal for the tie-breaker rule. The “statistically equal” approach might give the TDT agent a very slightly higher than 10% chance of winning the money, though I haven’t analysed this in any detail.
If the subject can know the exact code of TDT, Omega can know the exact code of TDT, and analyse it however it likes. That means it can know exactly where randomness is invoked—why would it have to sample?
This was my first thought: Omega can just prove the choosing probabilities are equal. However, it’s not totally straightforward, because the sim could sample more random bits depending on the results of its first random bits, and so on, leading to an exponentially growing outcome tree of possibilities, with no upper size bound to the length of the tree. There might not be an easy proof of equality in that case. Sampling and statistical equality is the next best approach...
I would say that any such problem doesn’t show that there is no best decision theory, it shows that that class of problem cannot be used in the ranking.
Edited to add: Unless, perhaps, one can show that an instantiation of the problem with particular choice of (in this case decision theory, but whatever is varied) is particularly likely to be encountered.
To draw out the analogy to Godelian incompleteness, any computable decision theory is subject to the suggested attack of being given a “Godel problem″ like problem 1, just as any computable set of axioms for arithmetic has a Godel sentence. You can always make a new decision theory TDT’ that is TDT+ do the right thing for the Godel problem. But TDT’ has it’s own Godel problem of course. You can’t make a computable theory that says “do the right thing for all Godel probems”, if you try to do that it would not give you something computable. I’m sure this is all just restating what you had in mind, but I think it’s worth spelling out.
If you have some sort of oracle for the halting problem (i.e. a hypercomputer) and Omega doesn’t, he couldn’t simulate you, so you would presumably be able to always win fair problems. Otherwise the best thing you could hope for is to get the right answer whenever your computation halts, but fail to halt in your computation for some problems, such as your Godel problem. (A decision theory like this can still be given a Godel problem if Omega can solve the halting problem, “I simulated you and if you fail to halt on this problem...”). I wonder if TDT fails to halt for its Godel problem, or if some natural modification of it might have this property, but I don’t understand it well enough to guess.
I am less optimistic about revising “fair” to exclude Godel problems. The analogy would be proving Peano arithmetic is complete “except for things that are like Godel sentences.” I don’t know of any formalizations of the idea of “being a Godel sentence”.
You’re right. However, since all decision theories fail when confronted with their personal version of this problem, but may or may not fail in other problems, then some decision theories may be better than others. The one that is better than all the others is thus the “best” DT.