The “build a clean Cartesian AI” folks, Schmidhuber and Hutter, are much closer to “describe how to build a clean naturalistic AI given unlimited computing power” than, say, Lenat’s Eurisko is to AIXI. It’s just that AIXI won’t actually work as a conceptual foundation for the reasons given, nay it is Solomonoff induction itself which will not work as a conceptual foundation, hence considering naturalized induction as part of the work to be done along the way to OPFAI. The worry from Eurisko-style AI is not that it will be Cartesian and therefore bad, but that it will do self-modification in a completely ad-hoc way and thus have no stable specifiable properties nor be apt to grafting on such. To avoid that, we want to do a cleaner system; and then, doing a cleaner system, we wish it to be naturalistic rather than Cartesian for the given reasons. Also, once you sketch out how a naturalistic system works, it’s very clear that these are issues central to stable self-modification—the system’s model of how it works and its attempt to change it.
I think you are conflating two different problems:
How to learn by reinforcement in an unknown non-ergodic environment (e.g. one where it is possible to drop an anvil on your head)
How to make decisions that take into account future reward, in a non-ergodic environment, where actions may modify the agent.
The first problem is well known the reinforcement learning community, and in fact it is mentioned also in the first AIXI papers, but it is sidestepped with an ergodicity assumption, rather than addressed. I don’t think there can be really general solutions for this problem: you need some environment-specific prior or supervision.
The second problem doesn’t seem as hard as the first one. AIXI, of course, can’t model self-modifications, because it is incomputable and it can only deal with computable environments, but computable varieties of AIXI (Schmidhuber’s Gödel machine, perhaps?) can easily represent themselves as part of the environment.
The “build a clean Cartesian AI” folks, Schmidhuber and Hutter, are much closer to “describe how to build a clean naturalistic AI given unlimited computing power” than, say, Lenat’s Eurisko is to AIXI. It’s just that AIXI won’t actually work as a conceptual foundation for the reasons given, nay it is Solomonoff induction itself which will not work as a conceptual foundation, hence considering naturalized induction as part of the work to be done along the way to OPFAI. The worry from Eurisko-style AI is not that it will be Cartesian and therefore bad, but that it will do self-modification in a completely ad-hoc way and thus have no stable specifiable properties nor be apt to grafting on such. To avoid that, we want to do a cleaner system; and then, doing a cleaner system, we wish it to be naturalistic rather than Cartesian for the given reasons. Also, once you sketch out how a naturalistic system works, it’s very clear that these are issues central to stable self-modification—the system’s model of how it works and its attempt to change it.
I think you are conflating two different problems:
How to learn by reinforcement in an unknown non-ergodic environment (e.g. one where it is possible to drop an anvil on your head)
How to make decisions that take into account future reward, in a non-ergodic environment, where actions may modify the agent.
The first problem is well known the reinforcement learning community, and in fact it is mentioned also in the first AIXI papers, but it is sidestepped with an ergodicity assumption, rather than addressed.
I don’t think there can be really general solutions for this problem: you need some environment-specific prior or supervision.
The second problem doesn’t seem as hard as the first one.
AIXI, of course, can’t model self-modifications, because it is incomputable and it can only deal with computable environments, but computable varieties of AIXI (Schmidhuber’s Gödel machine, perhaps?) can easily represent themselves as part of the environment.