It seems like any possible agent could be interpreted that way:
When deciding how to act, all that it cares about is whether its act in this world is “right”, where “right” means “maximizes the fixed act-evaluating function that was built into me.”
How does it help us to understand UDT specifically?
I think that I probably missed your point in my first reply. I see now that you were probably asking why it’s any more useful to view UDT agents this way that it would be to view any arbitrary agent as a deontologist.
The reason is that the UDT agent appears, from the outside, to be taking into account what happens in possible worlds that it should know will never happen, at least according to conventional epistemology. Unlike conventional consequentialists, you cannot interpret its behavior as a function of what it thinks might happen in the actual world (with what probabilities and with what payoffs). You can interpret its behavior as a function of what its builders thought might happen in the actual world, but you can’t do this for the agent itself.
One response to this is to treat the UDT agent as a consequentialist who cares about the consequences of its actions even in possible worlds that it knows aren’t actual. This is perfectly fine, except that it makes it hard to conceive of the agent as learning anything. The agent continues to take into account the evolution-histories of world-programs that would call it as a subroutine if they were run, even after it learns that they won’t be run. (Obviously this is not a problem if you think that the notion of an un-run program is incoherent.)
The alternative approach that I offer allows us to think of the agent as learning which once-possible worlds are actual. This is a more natural way to conceive of epistemic agents in my opinion. The cost is that the UDT agent is now a deontologist, for whom the rightness of an action doesn’t depend on just the effects that it will have in the actual world. “Rightness” doesn’t depend on actual consequences, at least not exclusively. However, the additional factors that figure into the “rightness” of an act require no further justification as far as the agent is concerned.
This is not to turn those additional factors into a “black box”. They were designed by the agent’s builders on conventional consequentialist grounds.
I think that I probably missed your point in my first reply. I see now that you were probably asking why it’s any more useful to view UDT agents this way that it would be to view any arbitrary agent as a deontologist.
The reason is that the UDT agent appears, from the outside, to be taking into account what happens in possible worlds that it should know will never happen, at least according to conventional epistemology. Unlike conventional consequentialists, you cannot interpret its behavior as a function of what it thinks might happen in the actual world (with what probabilities and with what payoffs). You can interpret its behavior as a function of what its builders thought might happen in the actual world, but you can’t do this for the agent itself.
One response to this is to treat the UDT agent as a consequentialist who cares about the consequences of its actions even in possible worlds that it knows aren’t actual. This is perfectly fine, except that it makes it hard to conceive of the agent as learning anything. The agent continues to take into account the evolution-histories of world-programs that would call it as a subroutine if they were run, even after it learns that they won’t be run. (Obviously this is not a problem if you think that the notion of an un-run program is incoherent.)
The alternative approach that I offer allows us to think of the agent as learning which once-possible worlds are actual. This is a more natural way to conceive of epistemic agents in my opinion. The cost is that the UDT agent is now a deontologist, for whom the rightness of an action doesn’t depend on just the effects that it will have in the actual world. “Rightness” doesn’t depend on actual consequences, at least not exclusively. However, the additional factors that figure into the “rightness” of an act require no further justification as far as the agent is concerned.
This is not to turn those additional factors into a “black box”. They were designed by the agent’s builders on conventional consequentialist grounds.