One of the problem here is how this agent is realized.
For example, let’s suppose that your algorithm is “Range all possible actions by their expected utility. Take the highest action which doesn’t involve killing with bare hands.” Then you can select action “modify yourself into pure utilitarian”, because it strictly increases your lifetime expected utility and doesn’t involve killing with bare hands.
Money-pump that depends on realization: I come to you and say “I cursed you: tomorrow you will kill someone with bare hands. Pay me an arbitrary amount of money to lift the curse.” You listen to me because zero is not a probability and if it is, we came with multiple embedded agency problems.
One of the problem here is how this agent is realized. For example, let’s suppose that your algorithm is “Range all possible actions by their expected utility. Take the highest action which doesn’t involve killing with bare hands.” Then you can select action “modify yourself into pure utilitarian”, because it strictly increases your lifetime expected utility and doesn’t involve killing with bare hands. Money-pump that depends on realization: I come to you and say “I cursed you: tomorrow you will kill someone with bare hands. Pay me an arbitrary amount of money to lift the curse.” You listen to me because zero is not a probability and if it is, we came with multiple embedded agency problems.