The obvious answer I can think of is that having a utility function that closely corresponds to a human’s values is going to help an AI predict humans. This is perhaps analogous to mirror neurons in humans.
Probably not so much. You have to figure out what agents in your environment want if you are going to try to understand them and deal with them—but you don’t really have to want the same things as them to be able to do that.
It’s of course not necessary. But humans model other humans by putting ourselves in someone else’s shoes and asking what we would do in that situation. I don’t necessarily agree with the argument that it is necessary for an AI to have the same utility function as a human in order to predict humans. But if you did write an AI with an identical utility function, that would give it an easy way to make some predictions about humans (although you’d have problems with things like biases that prevent us from achieving our goals, etc).
Some truth—but when you put yourself in someone else’s shoes, “goal substitution” often takes place, to take account of the fact that they want different things from you.
Machines may use the same trick, but again, they seem likely to be able to imagine quite a range of different intentional agents with different goals.
The good news is that they will probably at least try and understand and represent human goals.
The obvious answer I can think of is that having a utility function that closely corresponds to a human’s values is going to help an AI predict humans. This is perhaps analogous to mirror neurons in humans.
Probably not so much. You have to figure out what agents in your environment want if you are going to try to understand them and deal with them—but you don’t really have to want the same things as them to be able to do that.
It’s of course not necessary. But humans model other humans by putting ourselves in someone else’s shoes and asking what we would do in that situation. I don’t necessarily agree with the argument that it is necessary for an AI to have the same utility function as a human in order to predict humans. But if you did write an AI with an identical utility function, that would give it an easy way to make some predictions about humans (although you’d have problems with things like biases that prevent us from achieving our goals, etc).
Some truth—but when you put yourself in someone else’s shoes, “goal substitution” often takes place, to take account of the fact that they want different things from you.
Machines may use the same trick, but again, they seem likely to be able to imagine quite a range of different intentional agents with different goals.
The good news is that they will probably at least try and understand and represent human goals.