Answer here is obvious, but let’s look at another example: Should I eat an apple? Apple promises pleasure and I want it, but after I have eaten it, I don’t want eat anything as I am full. So the expected pleasure source has shifted.
In other words, we have in some sense bicameral mind: a conscious part which always follows pleasure and an unconscious part which constantly changes rewards depending on the persons’ needs. If we want to learn person’s preferences, we want to learn rules why the rewards are given to some things and are not given for other. Someone likes reading and other one likes skying.
And it is not a complete model of mind, just an illustration why reward is not enough to represent human values.
[edited]
Answer here is obvious, but let’s look at another example: Should I eat an apple? Apple promises pleasure and I want it, but after I have eaten it, I don’t want eat anything as I am full. So the expected pleasure source has shifted.
In other words, we have in some sense bicameral mind: a conscious part which always follows pleasure and an unconscious part which constantly changes rewards depending on the persons’ needs. If we want to learn person’s preferences, we want to learn rules why the rewards are given to some things and are not given for other. Someone likes reading and other one likes skying.
And it is not a complete model of mind, just an illustration why reward is not enough to represent human values.