I see RobbBB already explained why AIXI in particular doesn’t understand the word “you”. You might also find this post interesting, since it argues that an uncomputably perfect AIXI would succumb to “existential despair”.
See my reply to RobbBB. Perhaps the word ‘you’ was inappropriate; what I meant was that the AI could be taught to directly correlate ‘experiencing blue’ or ‘recieving reward’ with the flipping of bits in a peculiar machine that happens to be sitting in front of its external sensors.
I see RobbBB already explained why AIXI in particular doesn’t understand the word “you”. You might also find this post interesting, since it argues that an uncomputably perfect AIXI would succumb to “existential despair”.
See my reply to RobbBB. Perhaps the word ‘you’ was inappropriate; what I meant was that the AI could be taught to directly correlate ‘experiencing blue’ or ‘recieving reward’ with the flipping of bits in a peculiar machine that happens to be sitting in front of its external sensors.