(Edited & corrected) Here’s a third example. Imagine an AI whose only supergoal is to gather information about something. It explicitly encodes this information, and everything else it knows, as a Bayesian network of beliefs. Its utility ultimately derives entirely from creating new (correct) beliefs.
This AI’s values and beliefs don’t seem very separate to me. Every belief can be mapped to the value of having that belief. Values can be mapped to the belief(s) from whose creation or updating they derive. Every change in belief corresponds to a change in the AI’s current utility, and vice versa. Given a subroutine fully implementing the AI’s belief subsystem, the value system would be relatively simple, and vice versa.
However, this doesn’t imply the AI is in any sense simple or incapable of adaptation. Nor should it imply (though I’m no AI expert) that the AI is not a ‘mind’ or is not conscious. Similarly, while it’s true that the Roomba doesn’t have a belief/value separation, that’s not related to the fact that it’s a simple and stupid ‘mind’.
(Edited & corrected) Here’s a third example. Imagine an AI whose only supergoal is to gather information about something. It explicitly encodes this information, and everything else it knows, as a Bayesian network of beliefs. Its utility ultimately derives entirely from creating new (correct) beliefs.
This AI’s values and beliefs don’t seem very separate to me. Every belief can be mapped to the value of having that belief. Values can be mapped to the belief(s) from whose creation or updating they derive. Every change in belief corresponds to a change in the AI’s current utility, and vice versa. Given a subroutine fully implementing the AI’s belief subsystem, the value system would be relatively simple, and vice versa.
However, this doesn’t imply the AI is in any sense simple or incapable of adaptation. Nor should it imply (though I’m no AI expert) that the AI is not a ‘mind’ or is not conscious. Similarly, while it’s true that the Roomba doesn’t have a belief/value separation, that’s not related to the fact that it’s a simple and stupid ‘mind’.