So, this started out with the idea that an AI based on AIXI is, in some sense, safer than a fully functional AI, due to the existence of the anvil problem. Because AIXI can’t conceive of its own nonexistence, it has no preference ordering over its own mortality, and won’t (shouldn’t) resist any attempt to shut it down.
This seems rather irrelevant. Practically any real machine intelligence implementation will have electrical fences, armed guards, and the like surrounding it. Think you can just turn off Google? Think again. You don’t leave your data centre unprotected. Such systems will act as though they have preferences involving not being turned off. Being based on cut-down versions of AIXI is not very relevant—such preferences are pretty clearly desirable, so they’ll be built in by the designers—either using axioms or rewards.
This seems rather irrelevant. Practically any real machine intelligence implementation will have electrical fences, armed guards, and the like surrounding it. Think you can just turn off Google? Think again. You don’t leave your data centre unprotected. Such systems will act as though they have preferences involving not being turned off. Being based on cut-down versions of AIXI is not very relevant—such preferences are pretty clearly desirable, so they’ll be built in by the designers—either using axioms or rewards.