A known algorithm is a known finite syntactic thing, while an agent doesn’t normally know its behavior in this form, if it doesn’t tie its own identity to an existing algorithm. And that option doesn’t seem particularly motivated, as illustrated by it being desirable to build yourself a new algorithm.
Of course, if you just take the whole environment where the agent is embedded (with the environment being finite in nonblank data) and call that “the algorithm”, then any outcome in that environment is determined by that algorithm, and that somewhat robs notions disagreeing with that algorithm of motivation (though not really). But in more realistic situations there is unbounded unknown data in environment, so no algorithm fully describes its development, a choice of algorithm/data separation is a matter of framing.
In particular, an agent whose identity is not its initial algorithm can have preference found in environment, whose data is not part of the initial algorithm at all, can’t be inferred from it, can only be discovered by looking at the environment, perhaps only ever partially discovered. Most decision theory setups can’t understand that initial algorithm as an agent, since it’s usually an assumption of a decision algorithm that it knows what it optimizes for.
A known algorithm is a known finite syntactic thing, while an agent doesn’t normally know its behavior in this form, if it doesn’t tie its own identity to an existing algorithm. And that option doesn’t seem particularly motivated, as illustrated by it being desirable to build yourself a new algorithm.
Of course, if you just take the whole environment where the agent is embedded (with the environment being finite in nonblank data) and call that “the algorithm”, then any outcome in that environment is determined by that algorithm, and that somewhat robs notions disagreeing with that algorithm of motivation (though not really). But in more realistic situations there is unbounded unknown data in environment, so no algorithm fully describes its development, a choice of algorithm/data separation is a matter of framing.
In particular, an agent whose identity is not its initial algorithm can have preference found in environment, whose data is not part of the initial algorithm at all, can’t be inferred from it, can only be discovered by looking at the environment, perhaps only ever partially discovered. Most decision theory setups can’t understand that initial algorithm as an agent, since it’s usually an assumption of a decision algorithm that it knows what it optimizes for.