Preference should still apply to all possible situations. If idealized Bob gains control of Alice’s decision, he has access to both the action and Alice’s factual knowledge, and so the decision specifies how the action depends on that knowledge. This looks more like option 3, even though I agree that separating prior and utility might be a wrong way of formulating this.
I think the only way to formulate option 1 is by separating prior and utility. (Not your own prior and utility, but having some model of the other person’s prior and utility separately)
I agree that option 3 is prettier because it doesn’t have to do this, but is it better?
This needs a distinction between a prior that is “prior to all your knowledge” and prior that already takes into account your current knowledge and would be updated by future observations. I guess prior in the first sense could be seen as a fixed aspect of preference, while prior in the second sense reflects a situation where an action might be performed, so that it can be different in different situations with the same preference.
Thus, perfectly altruistic Alice should have a different prior in the second sense, taking into account Alice’s knowledge rather than Bob’s, but the same prior in the first sense, reflecting the same distribution of caring over possible worlds as Bob.
Bob’s prior in the first sense is not factual knowledge, it’s a description of which worlds Bob considers how important, so Alice can’t improve on it by knowing something that Bob doesn’t. A difference in priors in the first sense reflects different distributions of moral relevance associated with possibilities. When Alice knows something that Bob doesn’t, it is a statement about her priors in the second sense, not the first sense.
Thus, to the extent that Alice doesn’t assume Bob’s priors in the first sense, Alice doesn’t follow Bob’s preference, which would be a failure of perfect altruism. Alice’s prior doesn’t reflect different (or additional) knowledge, so its use would not be an improvement in the sense of Bob’s preference.
Preference should still apply to all possible situations. If idealized Bob gains control of Alice’s decision, he has access to both the action and Alice’s factual knowledge, and so the decision specifies how the action depends on that knowledge. This looks more like option 3, even though I agree that separating prior and utility might be a wrong way of formulating this.
I think the only way to formulate option 1 is by separating prior and utility. (Not your own prior and utility, but having some model of the other person’s prior and utility separately)
I agree that option 3 is prettier because it doesn’t have to do this, but is it better?
This needs a distinction between a prior that is “prior to all your knowledge” and prior that already takes into account your current knowledge and would be updated by future observations. I guess prior in the first sense could be seen as a fixed aspect of preference, while prior in the second sense reflects a situation where an action might be performed, so that it can be different in different situations with the same preference.
Thus, perfectly altruistic Alice should have a different prior in the second sense, taking into account Alice’s knowledge rather than Bob’s, but the same prior in the first sense, reflecting the same distribution of caring over possible worlds as Bob.
Why should Alice have the same distribution of caring as Bob?
My definition of prior was in the first sense.
Bob’s prior in the first sense is not factual knowledge, it’s a description of which worlds Bob considers how important, so Alice can’t improve on it by knowing something that Bob doesn’t. A difference in priors in the first sense reflects different distributions of moral relevance associated with possibilities. When Alice knows something that Bob doesn’t, it is a statement about her priors in the second sense, not the first sense.
Thus, to the extent that Alice doesn’t assume Bob’s priors in the first sense, Alice doesn’t follow Bob’s preference, which would be a failure of perfect altruism. Alice’s prior doesn’t reflect different (or additional) knowledge, so its use would not be an improvement in the sense of Bob’s preference.
Yes, when I said 2, I actually meant 3.