Makes sense. It seems to flow from the fact that the source code is in some sense allowed to use concepts like ‘Me’ or ‘I’, which refer to the agent itself. So both agents have source code which says “Maximise the resources that I have control over”, but in Agent 1 this translates to the utility function “Maximise the resources that Agent 1 has control over”, and in Agent 2 this translates to the different utility function “Maximise the resources that Agent 2 has control over”.
So this source code thing that we’re tempted to call a ‘utility function’ isn’t actually valid as a mapping from world states to real numbers until the agent is specified, because these ‘Me’/‘I’ terms are undefined.
Makes sense. It seems to flow from the fact that the source code is in some sense allowed to use concepts like ‘Me’ or ‘I’, which refer to the agent itself. So both agents have source code which says “Maximise the resources that I have control over”, but in Agent 1 this translates to the utility function “Maximise the resources that Agent 1 has control over”, and in Agent 2 this translates to the different utility function “Maximise the resources that Agent 2 has control over”.
So this source code thing that we’re tempted to call a ‘utility function’ isn’t actually valid as a mapping from world states to real numbers until the agent is specified, because these ‘Me’/‘I’ terms are undefined.