This is a good point, but I’ll lay out the argument against it.
To start, I’m personally skeptical of the claim that preferences and moral values can be clearly distinguished, especially given the variety of value systems that people have preferred over time, or even today.
Even if this is false, we seem to see the same phenomenon occur with moral values. I think the example of obvious differences in the relative preference for saving dogs, the elderly, or criminals points to actual differences in values—but as I argued above, I think this is a heavily optimized subspace of a moral intuition towards liking life which is now largely selecting on noise. But the difference in moral conclusions that follow from assigning animal lives exactly zero versus smaller-than human but nonzero value are huge.
To start, I’m personally skeptical of the claim that preferences and moral values can be clearly distinguished, especially given the variety of value systems that people have preferred over time, or even today.
There is a clear distinction within each system: if you violate a moral value, you are a bad person, if you violate a non-moral value, you are something else—irrational or foolish, maybe.
Also, you have not excluded levelling down—nothing is a moral value—as an option.
If you want a scale of moral value that is objective universal and timeless, you are going to have problems. But it is a strange to want in the first place., because value is not an objective physical property. Different people value different things. Where those values or preferences can be satisfied individually, there is no moral concern. Where there are trade-offs , or potential for conflict, then there is a need for—in the sense that a group is better off with—publically agreed and known standards and rules. Societies with food scarcity have rules about who is allowed to eat what, societies with food abundance don’t. Morality is an adaptation., it isn’t and should not be the same everywhere.
The EA framing—where moral is considered in terms of making inprovements, and individual , voluntary actions makes it quite hard to understand morality in general, because morality in general is about groups, obligations and prohibitions.
This is a good point, but I’ll lay out the argument against it.
To start, I’m personally skeptical of the claim that preferences and moral values can be clearly distinguished, especially given the variety of value systems that people have preferred over time, or even today.
Even if this is false, we seem to see the same phenomenon occur with moral values. I think the example of obvious differences in the relative preference for saving dogs, the elderly, or criminals points to actual differences in values—but as I argued above, I think this is a heavily optimized subspace of a moral intuition towards liking life which is now largely selecting on noise. But the difference in moral conclusions that follow from assigning animal lives exactly zero versus smaller-than human but nonzero value are huge.
There is a clear distinction within each system: if you violate a moral value, you are a bad person, if you violate a non-moral value, you are something else—irrational or foolish, maybe.
Also, you have not excluded levelling down—nothing is a moral value—as an option.
If you want a scale of moral value that is objective universal and timeless, you are going to have problems. But it is a strange to want in the first place., because value is not an objective physical property. Different people value different things. Where those values or preferences can be satisfied individually, there is no moral concern. Where there are trade-offs , or potential for conflict, then there is a need for—in the sense that a group is better off with—publically agreed and known standards and rules. Societies with food scarcity have rules about who is allowed to eat what, societies with food abundance don’t. Morality is an adaptation., it isn’t and should not be the same everywhere.
The EA framing—where moral is considered in terms of making inprovements, and individual , voluntary actions makes it quite hard to understand morality in general, because morality in general is about groups, obligations and prohibitions.