The point is that a physical simulation of the universe includes a description of A. If A simulates the universe, it sees that its outputs appear on W2 without postulating anything beyond physical law. (This is all most straightforward in the case where A does perfect induction and the shortest explanation for A’s observations takes the form of pointing to A in the universe, which is the one I started with; if A is capable of executing human-equivalent reasoning, though, the same thing happens in less easily formally analyzed universes.) I will try to edit the post to be more clear.
A true AIXI can’t model itself at all. A real AIXI variant (such as AIXItl) will not be able to model itself except in the rarest circumstances. Therefore its model of the universe as containing A is not going to be a simple one, because it can’t model A’s outputs. Therefore it seems that it will always be unable (as you said in another comment) to “recompute its inputs from the environment”.
I can’t model myself (or other humans) at all either, and yet I can still learn a model in which human behavior results from physical law, and make predictions on the basis of that model.
If A simulates the universe, it sees that its outputs appear on W2 without postulating anything beyond physical law.
Usually you can’t simulate the whole universe up to the current time from inside that universe—even if you know the laws and initial conditions—because you run out of storage space. With a growing universe, there may be some possibility of completely simulating its early stages...
Again, neither can I, but I still have beliefs about what the shortest description of the universe is. In particular, I don’t believe that my actions figure into the shortest model of the world in a special way, although there is some uncertainty about whether I am only able to hold this belief because I am more introspective than AIXI.
I don’t know if I understand this—do you mean that A fails to “recognize itself in a mirror” for a new and surprising reason?
As near as I can tell, you say that A would not recognize itself in what you call “the shortest explanation for A’s observations” even though this does in fact describe how A works. And since we never programmed A to want to model itself, it lacks the self-awareness to realize that changing its actions can change the outcome. (It can’t even use a theory like TDT to show that it should act as if it controls the reductionist image of itself, because that would require having an explicit self-model.)
I wouldn’t normally express this by calling Model 2 simpler than Model 1, as you did in the OP—the parent comment suggests to me that A mislabels Model 1 as Model 2 -- so maybe I still don’t get what you mean.
AIXI learns a function f : outputs → inputs, modeling the environment’s response to AIXI’s outputs.
Let y be the output of A. Then we have one function f1(y) which uses y to help model the world, and another function f2(y) which ignores y and essentially recomputes it from the environment. These two models make identical predictions when applied to the actual sequence of outputs of the algorithm, but make different predictions about counterfactuals which are essential to determining the agent’s behavior. If you are using f1, as AIXI intends to, then you do a sane thing if you try and rely on causal control. If you are using f2, as AIXI probably actually would, then you have no causal control over reality, and so go catatonic if you rely on causal control.
The point is that a physical simulation of the universe includes a description of A. If A simulates the universe, it sees that its outputs appear on W2 without postulating anything beyond physical law. (This is all most straightforward in the case where A does perfect induction and the shortest explanation for A’s observations takes the form of pointing to A in the universe, which is the one I started with; if A is capable of executing human-equivalent reasoning, though, the same thing happens in less easily formally analyzed universes.) I will try to edit the post to be more clear.
A true AIXI can’t model itself at all. A real AIXI variant (such as AIXItl) will not be able to model itself except in the rarest circumstances. Therefore its model of the universe as containing A is not going to be a simple one, because it can’t model A’s outputs. Therefore it seems that it will always be unable (as you said in another comment) to “recompute its inputs from the environment”.
I can’t model myself (or other humans) at all either, and yet I can still learn a model in which human behavior results from physical law, and make predictions on the basis of that model.
Usually you can’t simulate the whole universe up to the current time from inside that universe—even if you know the laws and initial conditions—because you run out of storage space. With a growing universe, there may be some possibility of completely simulating its early stages...
Again, neither can I, but I still have beliefs about what the shortest description of the universe is. In particular, I don’t believe that my actions figure into the shortest model of the world in a special way, although there is some uncertainty about whether I am only able to hold this belief because I am more introspective than AIXI.
I don’t know if I understand this—do you mean that A fails to “recognize itself in a mirror” for a new and surprising reason?
As near as I can tell, you say that A would not recognize itself in what you call “the shortest explanation for A’s observations” even though this does in fact describe how A works. And since we never programmed A to want to model itself, it lacks the self-awareness to realize that changing its actions can change the outcome. (It can’t even use a theory like TDT to show that it should act as if it controls the reductionist image of itself, because that would require having an explicit self-model.)
I wouldn’t normally express this by calling Model 2 simpler than Model 1, as you did in the OP—the parent comment suggests to me that A mislabels Model 1 as Model 2 -- so maybe I still don’t get what you mean.
AIXI learns a function f : outputs → inputs, modeling the environment’s response to AIXI’s outputs.
Let y be the output of A. Then we have one function f1(y) which uses y to help model the world, and another function f2(y) which ignores y and essentially recomputes it from the environment. These two models make identical predictions when applied to the actual sequence of outputs of the algorithm, but make different predictions about counterfactuals which are essential to determining the agent’s behavior. If you are using f1, as AIXI intends to, then you do a sane thing if you try and rely on causal control. If you are using f2, as AIXI probably actually would, then you have no causal control over reality, and so go catatonic if you rely on causal control.
I’ll try and make this a little more clear.