Bit of a nitpick, but I think you’re misdescribing AIXI. I think AIXI is defined to have a reward input channel, and its collection-of-all-possible-generative-world-models are tasked with predicting both sensory inputs and reward inputs, and Bayesian-updated accordingly, and then the generative models are issuing reward predictions which in turn are used to choose maximal-reward actions. (And by the way it doesn’t really work—it under-explores and thus can be permanently confused about counterfactual / off-policy rewards, IIUC.) So AIXI has no utility function.
That doesn’t detract from your post, it’s just that I maybe wouldn’t have used the term “AIXI-like” for the AIs that you’re describing, I think.
(There’s a decent chance that I’m confused and this whole comment is wrong.)
Bit of a nitpick, but I think you’re misdescribing AIXI. I think AIXI is defined to have a reward input channel, and its collection-of-all-possible-generative-world-models are tasked with predicting both sensory inputs and reward inputs, and Bayesian-updated accordingly, and then the generative models are issuing reward predictions which in turn are used to choose maximal-reward actions. (And by the way it doesn’t really work—it under-explores and thus can be permanently confused about counterfactual / off-policy rewards, IIUC.) So AIXI has no utility function.
That doesn’t detract from your post, it’s just that I maybe wouldn’t have used the term “AIXI-like” for the AIs that you’re describing, I think.
(There’s a decent chance that I’m confused and this whole comment is wrong.)