My concern is that it’s not clear where in the continuum the values stop being “moral” values, unlike with prime numbers.
It might be unclear where the line lies, but it shouldn’t make the concept itself “fuzzy”, merely not understood. What we talk about when we refer to a certain idea is always something specific, but it’s not always clear what is implied by what we talk about. That different people can interpret the same words as referring to different ideas doesn’t make any of these different ideas undefined. The failure to interpret the words in the same way is a failure of communication, not a characterization of the idea that failed to be communicated.
I of course agree that “morality” admits a lot of similar interpretations, but I’d venture to say that “Blueberry’s preference” does as well. It’s an unsolved problem—a core question of Friendly AI—to formally define any of the concepts interpreting these words in a satisfactory way. The fuzziness in communication and elusiveness in formal understanding are relevant equally for the aggregate morality and personal preference, and so the individual/aggregate divide is not the point that particularly opposes the analogy.
Do you think there is a clear line between what humans in general value (morality) and what other entities might value, and we just don’t know where it is? Let’s call the other side of the line ‘schmorality’. So a paperclipper’s values are schmoral.
Is it possible that a human could have values on the other side of the line (schmoral values)?
Suppose another entity, who is on the other side of the line, has a conversation with a human about a moral issue. Both entities engage in the same kind of reasoning, use the same kind of arguments and examples, so why is one reasoning called “moral reasoning” and the other just about values (schmoral reasoning)?
Suppose I am right on the edge of the line. So my values are moral values, but a slight change makes these values schmoral values. From my point of view, these two sets of values are very close. Why do you give them completely different categories? And suppose my values change slightly over time, so I cross the line and back within a day. Do I suddenly stop caring about morality, then start again? This discontinuity seems very strange to me.
I don’t say that any given concept is reasonable for all purposes, just that any concept has a very specific intended meaning, at the moment it’s considered. The concept of morality can be characterized as, roughly, referring to human-like preference, or aggregate preference of humanity-like collections of individual preferences—this is a characterization resilient to some measure of ambiguity in interpretation. The concepts themselves can’t be negotiated, they are set in stone by their intended meaning, though a different concept may be better for a given purpose.
I don’t say that any given concept is reasonable for all purposes, just that any concept has a very specific intended meaning, at the moment it’s considered. The concept of morality can be characterized as, roughly, referring to human-like preference
It might be unclear where the line lies, but it shouldn’t make the concept itself “fuzzy”, merely not understood. What we talk about when we refer to a certain idea is always something specific, but it’s not always clear what is implied by what we talk about. That different people can interpret the same words as referring to different ideas doesn’t make any of these different ideas undefined. The failure to interpret the words in the same way is a failure of communication, not a characterization of the idea that failed to be communicated.
I of course agree that “morality” admits a lot of similar interpretations, but I’d venture to say that “Blueberry’s preference” does as well. It’s an unsolved problem—a core question of Friendly AI—to formally define any of the concepts interpreting these words in a satisfactory way. The fuzziness in communication and elusiveness in formal understanding are relevant equally for the aggregate morality and personal preference, and so the individual/aggregate divide is not the point that particularly opposes the analogy.
I’m still very confused.
Do you think there is a clear line between what humans in general value (morality) and what other entities might value, and we just don’t know where it is? Let’s call the other side of the line ‘schmorality’. So a paperclipper’s values are schmoral.
Is it possible that a human could have values on the other side of the line (schmoral values)?
Suppose another entity, who is on the other side of the line, has a conversation with a human about a moral issue. Both entities engage in the same kind of reasoning, use the same kind of arguments and examples, so why is one reasoning called “moral reasoning” and the other just about values (schmoral reasoning)?
Suppose I am right on the edge of the line. So my values are moral values, but a slight change makes these values schmoral values. From my point of view, these two sets of values are very close. Why do you give them completely different categories? And suppose my values change slightly over time, so I cross the line and back within a day. Do I suddenly stop caring about morality, then start again? This discontinuity seems very strange to me.
I don’t say that any given concept is reasonable for all purposes, just that any concept has a very specific intended meaning, at the moment it’s considered. The concept of morality can be characterized as, roughly, referring to human-like preference, or aggregate preference of humanity-like collections of individual preferences—this is a characterization resilient to some measure of ambiguity in interpretation. The concepts themselves can’t be negotiated, they are set in stone by their intended meaning, though a different concept may be better for a given purpose.
Thanks! That actually helped a lot.