This seems to be tacking a lot of anthropomorphic emotional reactions onto the agent’s decision theory.
It may in some cases be inappropriate to anthropomorphize an agent. But anthropomorphization can be useful in other cases. My suggestion in the OP is to be used in the case where anthropomorphization seems useful.
Imagine an agent that follows the decision theory of “Always take the first option presented.” but has humanlike reactions to the outcome.
This is a great example. Maybe I should have started with something like that to motivate the post.
Suppose that someone you cared about were acting like this. Let’s suppose that, according to your decision theory, you should try to change the person to follow a different decision algorithm. One option is to consider them to be a baffling alien, whose actions you can predict, but whose thinking you cannot at all sympathize with.
However, if you care about them, you might want to view them in a way that encourages sympathy. You also probably want to interpret their psychology in a way that seems as human as possible, so that you can bring to bear the tools of psychology. Psychology, at this time, depends heavily on using our own human brains as almost-opaque boxes to model other neurologically similar humans. So your only hope of helping this person is to conceive of them in a way that seems more like a normal human. You need to anthropomorphize them.
In this case, I would probably first try to think of the person as a normal person who is being parasitized by an alien agent with this weird decision theory. I would focus on trying to remove the parasitic agent. The hope would be that the human has normal human decision-making mechanisms that were being overridden by the parasite.
This seems to be tacking a lot of anthropomorphic emotional reactions onto the agent’s decision theory.
Imagine an agent that follows the decision theory of “Always take the first option presented.” but has humanlike reactions to the outcome.
It will one box or two box depending on how the situation is described to it, but it will be happy if it gets the million dollars.
The process used to make choices need not be connected to the process used to evaluate preference.
It may in some cases be inappropriate to anthropomorphize an agent. But anthropomorphization can be useful in other cases. My suggestion in the OP is to be used in the case where anthropomorphization seems useful.
This is a great example. Maybe I should have started with something like that to motivate the post.
Suppose that someone you cared about were acting like this. Let’s suppose that, according to your decision theory, you should try to change the person to follow a different decision algorithm. One option is to consider them to be a baffling alien, whose actions you can predict, but whose thinking you cannot at all sympathize with.
However, if you care about them, you might want to view them in a way that encourages sympathy. You also probably want to interpret their psychology in a way that seems as human as possible, so that you can bring to bear the tools of psychology. Psychology, at this time, depends heavily on using our own human brains as almost-opaque boxes to model other neurologically similar humans. So your only hope of helping this person is to conceive of them in a way that seems more like a normal human. You need to anthropomorphize them.
In this case, I would probably first try to think of the person as a normal person who is being parasitized by an alien agent with this weird decision theory. I would focus on trying to remove the parasitic agent. The hope would be that the human has normal human decision-making mechanisms that were being overridden by the parasite.