I think the core issue arises when A locates a model of the world that includes a model of A itself, thus explaining away the apparent correlation between the input and output tapes. I don’t have a watertight objection to your argument, but I’m also not convinced that it goes through so easily.
Let’s stick to the case where A is just a perfectly ordinary Turing approximation of AIXI. It seems to me that it’s still going to have quite some difficulty reasoning about its own behaviour. In particular, suppose A locates a hypothesis H=”the world consists of a connected to a and my outputs are irrelevant”. Then the first step is that A asks what happens if its next output is (say) 0. To do that it needs to run H to produce the next bit that it expects to receive from the world. But running H involves running a simulation of A, and inside that simulation the exact same situation arises, namely that sim(A) considers various outputs that it might make and then runs simulations of its inferred model of the world, which themselves contain models of A, resulting in another level of recursion to sim(sim(A)), and so on in an infinite loop. Actually, I don’t know what AIXI does about Turing that fail to produce output...
A different, perhaps weaker objection is that AIXI conditions on its outputs when performing inference, so they don’t count towards the “burden of explanation”. That doesn’t resolve the issue you raise but perhaps this does: Is it possible to make Model 2 just slightly simpler by somehow leveraging the “free” information on the output tape? Perhaps by removing some description of some initial conditions from Model 2 and replacing that with a function of the information on the output tape. It’s not clear that this is always possible but it seems plausible to me.
Then the first step is that A asks what happens if its next output is (say) 0. To do that it needs to run H to produce the next bit of output. But running H involves running a simulation of A, and inside that simulation the exact same situation arises, namely that sim(A) considers various outputs that it might make and runs simulations of the world, resulting in another level of recursion to sim(sim(A)), and so on in an infinite loop.
This seems to be the observation that you can’t have a Turing machine that implements AIXI. An approximate AIXI is not going to be able to simulate itself.
Is it possible to make Model 2 just slightly simpler by somehow leveraging the “free” information on the output tape?
I don’t think this is possible, although it is an interesting thought. The main issue is that before you get to leverage the first N bits of AIXI’s output you have to also explain the first N bits of AIXI’s input, which seems basically guaranteed to wash out the complexity gains (because all of the info in the first N bits of AIXI’s output was coming from the first N bits of AIXI’s input).
This seems to be the observation that you can’t have a Turing machine that implements AIXI. An approximate AIXI is not going to be able to simulate itself.
Yes, I guess you’re right. But doesn’t this also mean that no computable approximation of AIXI will ever hypothesize a world that contains a model of itself, for if it did then it will go into the infinite loop I described. So it seems the problem of Model 2 will never come up?
The main issue is that before you get to leverage the first N bits of AIXI’s output you have to also explain the first N bits of AIXI’s input
Not sure I’m understanding you correctly but this seems wrong. AIXI conditions on all its outputs so far, right? So if the world is a bit-repeater then one valid model of the world is literally a bit repeater, which explains the inputs but not the outputs.
Voted up for being an insightful observation.
I think the core issue arises when A locates a model of the world that includes a model of A itself, thus explaining away the apparent correlation between the input and output tapes. I don’t have a watertight objection to your argument, but I’m also not convinced that it goes through so easily.
Let’s stick to the case where A is just a perfectly ordinary Turing approximation of AIXI. It seems to me that it’s still going to have quite some difficulty reasoning about its own behaviour. In particular, suppose A locates a hypothesis H=”the world consists of a connected to a and my outputs are irrelevant”. Then the first step is that A asks what happens if its next output is (say) 0. To do that it needs to run H to produce the next bit that it expects to receive from the world. But running H involves running a simulation of A, and inside that simulation the exact same situation arises, namely that sim(A) considers various outputs that it might make and then runs simulations of its inferred model of the world, which themselves contain models of A, resulting in another level of recursion to sim(sim(A)), and so on in an infinite loop. Actually, I don’t know what AIXI does about Turing that fail to produce output...
A different, perhaps weaker objection is that AIXI conditions on its outputs when performing inference, so they don’t count towards the “burden of explanation”. That doesn’t resolve the issue you raise but perhaps this does: Is it possible to make Model 2 just slightly simpler by somehow leveraging the “free” information on the output tape? Perhaps by removing some description of some initial conditions from Model 2 and replacing that with a function of the information on the output tape. It’s not clear that this is always possible but it seems plausible to me.
This seems to be the observation that you can’t have a Turing machine that implements AIXI. An approximate AIXI is not going to be able to simulate itself.
I don’t think this is possible, although it is an interesting thought. The main issue is that before you get to leverage the first N bits of AIXI’s output you have to also explain the first N bits of AIXI’s input, which seems basically guaranteed to wash out the complexity gains (because all of the info in the first N bits of AIXI’s output was coming from the first N bits of AIXI’s input).
Yes, I guess you’re right. But doesn’t this also mean that no computable approximation of AIXI will ever hypothesize a world that contains a model of itself, for if it did then it will go into the infinite loop I described. So it seems the problem of Model 2 will never come up?
Not sure I’m understanding you correctly but this seems wrong. AIXI conditions on all its outputs so far, right? So if the world is a bit-repeater then one valid model of the world is literally a bit repeater, which explains the inputs but not the outputs.