A naturalized agent’s hypotheses can be about world-states that include the agent, or world-states that don’t include the agent. A Cartesian agent’s hypotheses are all about the agent’s internal states, and different possible causes for those states, so the idea of ‘world-states that don’t include the agent’ can’t be directly represented. Even a halting program in AIXI’s hypothesis space isn’t really a prediction about how a world without AIXI would look; it’s more a prediction about how Everything (including AIXI) could come to an end.
Our ultimate goal in building an AI isn’t to optimize the internal features of the AI; it’s to optimize the rest of the world, with the AI functioning as a tool. So it seems likely that we’ll want our AI’s beliefs to look like pictures of an objective world (in which agents like the AI happen to exist, sometimes).
A Cartesian agent’s hypotheses are all about the agent’s internal states, and different possible causes for those states, so the idea of ‘world-states that don’t include the agent’ can’t be directly represented.
A sequence predictor’s predictions are all about the agent’s input tape states*, and different possible causes for those states. The hypotheses are programs that implement entire models of the Universe, and these can definitely directly represent world-states which don’t include the agent.
* More realistically, the states of the registers where the sensor data is placed.
ETA: I wonder if this intuition is caused by that fact that I am a practicing Bayesian statistician, so the distinction between posterior distributions and posterior predictive distributions is more salient to me.
A naturalized agent’s hypotheses can be about world-states that include the agent, or world-states that don’t include the agent. A Cartesian agent’s hypotheses are all about the agent’s internal states, and different possible causes for those states, so the idea of ‘world-states that don’t include the agent’ can’t be directly represented. Even a halting program in AIXI’s hypothesis space isn’t really a prediction about how a world without AIXI would look; it’s more a prediction about how Everything (including AIXI) could come to an end.
Our ultimate goal in building an AI isn’t to optimize the internal features of the AI; it’s to optimize the rest of the world, with the AI functioning as a tool. So it seems likely that we’ll want our AI’s beliefs to look like pictures of an objective world (in which agents like the AI happen to exist, sometimes).
A sequence predictor’s predictions are all about the agent’s input tape states*, and different possible causes for those states. The hypotheses are programs that implement entire models of the Universe, and these can definitely directly represent world-states which don’t include the agent.
* More realistically, the states of the registers where the sensor data is placed.
ETA: I wonder if this intuition is caused by that fact that I am a practicing Bayesian statistician, so the distinction between posterior distributions and posterior predictive distributions is more salient to me.