I’m parsing things that you call ‘moral’ as ‘things that your utility function assigns terminal utility.’ Tell me if this is incorrect. Also, my apologies in advance if I ramble—I’m thinking as I type.
So, Gandhi won’t take a pill that would make him want to kill people because it would increase the probability of a person dying in a given time interval—since Gandhi’s utility function assigns negative utility to people dying, he won’t do this because it’s going to net him negative expected utility. The only reason that he would do this is if there was some cost for not taking the pill or some benefit for taking the pill that would cause (expected utility from taking the pill) to be greater than (expected utility from not taking the pill).
You assign negative utility to not having strong allergic reactions—if you take a pill so that there is a smaller probability that you will have a strong allergic reaction in a given time interval, you taking the pill nets you positive expected utility, given that there are no costs associated with doing so that outweigh the benefits.
I’m going to assume that when you mentioned taking a drug that would make you “not care about pollen” that this drug causes you to not physically react to pollen, causing the positive utility in the previous paragraph. This wouldn’t change your utility function itself; you’re just optimizing the world around you so that (~allergy) is true instead of (allergy), with relation to yourself. This is different from the Gandhi scenario, because Gandhi’s changing his utility function itself—he is taking a psychoactive drug, in the strongest sense of the word.
I see the main distinction between decisions that involve taking actions that change your utility function itself and those that don’t to be that changing your utility function itself is much riskier than changing the world around it. Gandhi probably wouldn’t take a pill that make him kill people even if someone validly precommitted to kill twenty people if he didn’t, because he’s aware that the cost of him optimizing for negative utility instead of positive utility for the rest of his life is really big.
This was the question you asked:
Specifically, is willingness to self-modify about X a good guide to whether X is morally significant?
That might be a useful heuristic. However, I note that it doesn’t apply at the extreme cases—where there are enormous costs for not self-modifying. It also generates some false positives: there are a whole lot of things about myself that I just wouldn’t dare mess with because I’m not sure that they wouldn’t break the rest of me.
I’m parsing things that you call ‘moral’ as ‘things that your utility function assigns terminal utility.’ Tell me if this is incorrect. Also, my apologies in advance if I ramble—I’m thinking as I type.
So, Gandhi won’t take a pill that would make him want to kill people because it would increase the probability of a person dying in a given time interval—since Gandhi’s utility function assigns negative utility to people dying, he won’t do this because it’s going to net him negative expected utility. The only reason that he would do this is if there was some cost for not taking the pill or some benefit for taking the pill that would cause (expected utility from taking the pill) to be greater than (expected utility from not taking the pill).
You assign negative utility to not having strong allergic reactions—if you take a pill so that there is a smaller probability that you will have a strong allergic reaction in a given time interval, you taking the pill nets you positive expected utility, given that there are no costs associated with doing so that outweigh the benefits.
I’m going to assume that when you mentioned taking a drug that would make you “not care about pollen” that this drug causes you to not physically react to pollen, causing the positive utility in the previous paragraph. This wouldn’t change your utility function itself; you’re just optimizing the world around you so that (~allergy) is true instead of (allergy), with relation to yourself. This is different from the Gandhi scenario, because Gandhi’s changing his utility function itself—he is taking a psychoactive drug, in the strongest sense of the word.
I see the main distinction between decisions that involve taking actions that change your utility function itself and those that don’t to be that changing your utility function itself is much riskier than changing the world around it. Gandhi probably wouldn’t take a pill that make him kill people even if someone validly precommitted to kill twenty people if he didn’t, because he’s aware that the cost of him optimizing for negative utility instead of positive utility for the rest of his life is really big.
This was the question you asked:
That might be a useful heuristic. However, I note that it doesn’t apply at the extreme cases—where there are enormous costs for not self-modifying. It also generates some false positives: there are a whole lot of things about myself that I just wouldn’t dare mess with because I’m not sure that they wouldn’t break the rest of me.