Our intuitive notion of being manipulated is related to a person (call him Ahmed) taking an action A with the property that, in our intuitive causal world-models, the chain-of-causation leading to A does not ultimately trace back to the acausal force of Ahmed’s free will, but rather to the free will of some third-party who manipulated Ahmed.
(For example, if Bob deceives me about what a button does, and then I press the button, then our intuitive conceptualization of the situation says that the button was pressed ultimately because of Bob’s acausal free will working towards Bob’s desires, not because of my acausal free will working towards my desires. I was an instrument to Bob.)
Ahmed and Bob are both evolved beings. Both are running a set of learned & innate behaviors that, instrumentally, have the evolutionary purpose of maximizing the inclusive relative evolutionary fitness of Ahmed, and of Bob, respectively. What our natural ontology is actually doing is credit assignment: whose attempt to maximize their own inclusive relative evolutionary fitness worked as intended and actually benefited whom? Did Ahmed push the button because it was (to the best of his ability to discern) good for himself, or because Bob manipulated Ahmed into thinking that something that was actually good for Bob and bad for Ahmed would be good for Ahmed? So both credit in the sense of who did better, who did worse (and this is not a zero-sum game, we can both do better, or both do worse), and then credit assignment in who was successful in trying to do better (assuming that they were)?
”Free will” makes a lot more sense if mapped to “being unimpeded in my attempts to optimize my own (inclusive relative) evolutionary fitness”.
So I think the ontology is actually rather precise, for what matters evolutionarily.
In other words, if you persuade me of something for my own good, then it’s counsel (even if it’s also good for you); if you persuade me of something that bad for me and good for you, then that’s manipulation (and if it’s bad for both of us, it’s stupidity).
In short: this ontology isn’t messed up from an evolutionary point of view: it has exactly the form evolutionary psychology would predict. So, now we understand it, we can use it to guide an AI.
Ahmed and Bob are both evolved beings. Both are running a set of learned & innate behaviors that, instrumentally, have the evolutionary purpose of maximizing the inclusive relative evolutionary fitness of Ahmed, and of Bob, respectively. What our natural ontology is actually doing is credit assignment: whose attempt to maximize their own inclusive relative evolutionary fitness worked as intended and actually benefited whom? Did Ahmed push the button because it was (to the best of his ability to discern) good for himself, or because Bob manipulated Ahmed into thinking that something that was actually good for Bob and bad for Ahmed would be good for Ahmed? So both credit in the sense of who did better, who did worse (and this is not a zero-sum game, we can both do better, or both do worse), and then credit assignment in who was successful in trying to do better (assuming that they were)?
”Free will” makes a lot more sense if mapped to “being unimpeded in my attempts to optimize my own (inclusive relative) evolutionary fitness”.
So I think the ontology is actually rather precise, for what matters evolutionarily.
In other words, if you persuade me of something for my own good, then it’s counsel (even if it’s also good for you); if you persuade me of something that bad for me and good for you, then that’s manipulation (and if it’s bad for both of us, it’s stupidity).
In short: this ontology isn’t messed up from an evolutionary point of view: it has exactly the form evolutionary psychology would predict. So, now we understand it, we can use it to guide an AI.