In other words, power is a subset of freedom. But freedom also implies that nothing external is manipulating the agent’s values or controlling the agent’s information processing to screen off certain outcomes.
I think that we can begin to see, here, how manipulation and empowerment are something like opposites. In fact, I might go so far as to claim that “manipulation,” as I’ve been using the term, is actually synonymous with “disempowerment.” I touched on this in the definition of “Freedom,” in the ontology section, above.
Doesn’t this mean that power also implies that nothing external is manipulating the agent’s values or controlling the agent’s information processing to screen off certain outcomes? Is there a useful distinction between power and freedom?
Also, I think corrigible agents should measure power based on their principal’s judging of importance, so that terms like the power to turn the universe or an arbitrary section into paperclips or the like aren’t as important as terms like human extinction. But maybe this can be mitigated by restricting the empowerment goal to the agent’s structure/thoughts/actions.
I agree that controlling an agent’s values and information are disempowering and restrict freedom. I’m not sure whether there’s a useful distinction. Ultimately they’re both just words that imperfectly capture the important patterns in reality. I think it’s plausible that the right formulation of power involves attending to the agent’s values/sense of import, but my guess is that one must be a little careful to also include counterfactual values in there, else the AI ends up simply optimizing for it’s belief of what you desire. I talk a bit about optimizing for the counterfactual spread of possible values in 3b.
Doesn’t this mean that power also implies that nothing external is manipulating the agent’s values or controlling the agent’s information processing to screen off certain outcomes? Is there a useful distinction between power and freedom?
Also, I think corrigible agents should measure power based on their principal’s judging of importance, so that terms like the power to turn the universe or an arbitrary section into paperclips or the like aren’t as important as terms like human extinction. But maybe this can be mitigated by restricting the empowerment goal to the agent’s structure/thoughts/actions.
I agree that controlling an agent’s values and information are disempowering and restrict freedom. I’m not sure whether there’s a useful distinction. Ultimately they’re both just words that imperfectly capture the important patterns in reality. I think it’s plausible that the right formulation of power involves attending to the agent’s values/sense of import, but my guess is that one must be a little careful to also include counterfactual values in there, else the AI ends up simply optimizing for it’s belief of what you desire. I talk a bit about optimizing for the counterfactual spread of possible values in 3b.