The best name I know of is “implementing the intentional stance.” Intentional stance is Dan Dennett’s name for the ability to consider collections of atoms in terms of agents, desire, goals, etc. But this still isn’t quite the right label, because you don’t merely want the AI to consider the collection of atoms in terms of some goals, you want to the AI to use prior information about humans to consider humans in terms of humans’ actual goals.
The best name I know of is “implementing the intentional stance.” Intentional stance is Dan Dennett’s name for the ability to consider collections of atoms in terms of agents, desire, goals, etc. But this still isn’t quite the right label, because you don’t merely want the AI to consider the collection of atoms in terms of some goals, you want to the AI to use prior information about humans to consider humans in terms of humans’ actual goals.
My post on the subject is forthcoming :P