But as far as we can tell, our behavior is often not determined by our wanting a particular state of affairs, but by how our options are framed.
My hypothesis is that something qualifies as agent’s goal to the extent the agent tries to make actions that argmax the dependence of that goal on those actions.
In this case, where information about the relevant states of the world gets obfuscated in various ways before it reaches the decision, the agent looks at how its action affects those states of the world, but it doesn’t look at how its action affects the framing, or the way information is distorted. The objective in making a decision is to affect the states of the world, not to affect the way information about the states of the world is delivered.
This then is the distinction between the facts in the world the agent cares about and the way in which information about those facts travels to agent’s decisions: agent’s actions depend on the dependence of the former on its actions, but not on the dependence of the latter on its actions.
My hypothesis is that something qualifies as agent’s goal to the extent the agent tries to make actions that argmax the dependence of that goal on those actions.
In this case, where information about the relevant states of the world gets obfuscated in various ways before it reaches the decision, the agent looks at how its action affects those states of the world, but it doesn’t look at how its action affects the framing, or the way information is distorted. The objective in making a decision is to affect the states of the world, not to affect the way information about the states of the world is delivered.
This then is the distinction between the facts in the world the agent cares about and the way in which information about those facts travels to agent’s decisions: agent’s actions depend on the dependence of the former on its actions, but not on the dependence of the latter on its actions.