Nevermind, misunderstood your initial comment, I think.
I thought you were saying: if pref-util is right, pref-utilists may self-modify away from it, which refutes pref-util.
I now think you’re saying: we don’t know what is right, but if we assume pref-util, then we’ll lose part of our ability to figure it out, so we shouldn’t do that (yet).
Also, you’re saying that most people don’t understand morality better than us, so we shouldn’t take their opinions more seriously than ours. (Agreed.) But pref-utilists do take those opinions seriously; they’re letting their normative ethics influence their beliefs about their normative ethics. (Well, duh, consequentialism.)
In which case I’d (naively) say, let pref-util redistribute the probability mass you’ve assigned to pref-util any way it wants. If it wants to sacrifice it all for majority opinions, sure, but don’t give it more than that.
Nevermind, misunderstood your initial comment, I think.
I thought you were saying: if pref-util is right, pref-utilists may self-modify away from it, which refutes pref-util.
I now think you’re saying: we don’t know what is right, but if we assume pref-util, then we’ll lose part of our ability to figure it out, so we shouldn’t do that (yet).
Also, you’re saying that most people don’t understand morality better than us, so we shouldn’t take their opinions more seriously than ours. (Agreed.) But pref-utilists do take those opinions seriously; they’re letting their normative ethics influence their beliefs about their normative ethics. (Well, duh, consequentialism.)
In which case I’d (naively) say, let pref-util redistribute the probability mass you’ve assigned to pref-util any way it wants. If it wants to sacrifice it all for majority opinions, sure, but don’t give it more than that.