What if CEV doesn’t exist, and there really are different groups of humans with different values?
Of course different people have different values. These values might be similar, but they won’t be identical.
Primeness is so different from morality that it’s clear we’re talking about two different things.
Yes, but what is “prime number”? Is it 5, or is it 7? 5 is clearly different from 7, although it’s very similar to it in that it’s also prime. Use the analogy of prime=moral and 5=Blueberry’s values, 7=Will’s values.
It’s not clear to me that the agents will say “I’m moved by this modified view, not morality”. Why wouldn’t they say “No, this modification is the correct morality, and I am moved by morality!”
Because that would be pointless disputing of definitions—clearly, different things are meant by word “morality” in your example.
Yes, but what is “prime number”? Is it 5, or is it 7? 5 is clearly different from 7, although it’s very similar to it in that it’s also prime. Use the analogy of prime=moral and 5=Blueberry’s values, 7=Will’s values.
I see your point, but there is an obvious problem with this analogy: prime and nonprime are two discrete categories. But we can consider a continuum of values, ranging from something almost everyone agrees is moral, through values that are unusual or uncommon but still recognized as human values, all the way to completely alien values like paperclipping.
My concern is that it’s not clear where in the continuum the values stop being “moral” values, unlike with prime numbers.
My concern is that it’s not clear where in the continuum the values stop being “moral” values, unlike with prime numbers.
It might be unclear where the line lies, but it shouldn’t make the concept itself “fuzzy”, merely not understood. What we talk about when we refer to a certain idea is always something specific, but it’s not always clear what is implied by what we talk about. That different people can interpret the same words as referring to different ideas doesn’t make any of these different ideas undefined. The failure to interpret the words in the same way is a failure of communication, not a characterization of the idea that failed to be communicated.
I of course agree that “morality” admits a lot of similar interpretations, but I’d venture to say that “Blueberry’s preference” does as well. It’s an unsolved problem—a core question of Friendly AI—to formally define any of the concepts interpreting these words in a satisfactory way. The fuzziness in communication and elusiveness in formal understanding are relevant equally for the aggregate morality and personal preference, and so the individual/aggregate divide is not the point that particularly opposes the analogy.
Do you think there is a clear line between what humans in general value (morality) and what other entities might value, and we just don’t know where it is? Let’s call the other side of the line ‘schmorality’. So a paperclipper’s values are schmoral.
Is it possible that a human could have values on the other side of the line (schmoral values)?
Suppose another entity, who is on the other side of the line, has a conversation with a human about a moral issue. Both entities engage in the same kind of reasoning, use the same kind of arguments and examples, so why is one reasoning called “moral reasoning” and the other just about values (schmoral reasoning)?
Suppose I am right on the edge of the line. So my values are moral values, but a slight change makes these values schmoral values. From my point of view, these two sets of values are very close. Why do you give them completely different categories? And suppose my values change slightly over time, so I cross the line and back within a day. Do I suddenly stop caring about morality, then start again? This discontinuity seems very strange to me.
I don’t say that any given concept is reasonable for all purposes, just that any concept has a very specific intended meaning, at the moment it’s considered. The concept of morality can be characterized as, roughly, referring to human-like preference, or aggregate preference of humanity-like collections of individual preferences—this is a characterization resilient to some measure of ambiguity in interpretation. The concepts themselves can’t be negotiated, they are set in stone by their intended meaning, though a different concept may be better for a given purpose.
I don’t say that any given concept is reasonable for all purposes, just that any concept has a very specific intended meaning, at the moment it’s considered. The concept of morality can be characterized as, roughly, referring to human-like preference
Of course different people have different values. These values might be similar, but they won’t be identical.
Yes, but what is “prime number”? Is it 5, or is it 7? 5 is clearly different from 7, although it’s very similar to it in that it’s also prime. Use the analogy of prime=moral and 5=Blueberry’s values, 7=Will’s values.
Because that would be pointless disputing of definitions—clearly, different things are meant by word “morality” in your example.
I see your point, but there is an obvious problem with this analogy: prime and nonprime are two discrete categories. But we can consider a continuum of values, ranging from something almost everyone agrees is moral, through values that are unusual or uncommon but still recognized as human values, all the way to completely alien values like paperclipping.
My concern is that it’s not clear where in the continuum the values stop being “moral” values, unlike with prime numbers.
It might be unclear where the line lies, but it shouldn’t make the concept itself “fuzzy”, merely not understood. What we talk about when we refer to a certain idea is always something specific, but it’s not always clear what is implied by what we talk about. That different people can interpret the same words as referring to different ideas doesn’t make any of these different ideas undefined. The failure to interpret the words in the same way is a failure of communication, not a characterization of the idea that failed to be communicated.
I of course agree that “morality” admits a lot of similar interpretations, but I’d venture to say that “Blueberry’s preference” does as well. It’s an unsolved problem—a core question of Friendly AI—to formally define any of the concepts interpreting these words in a satisfactory way. The fuzziness in communication and elusiveness in formal understanding are relevant equally for the aggregate morality and personal preference, and so the individual/aggregate divide is not the point that particularly opposes the analogy.
I’m still very confused.
Do you think there is a clear line between what humans in general value (morality) and what other entities might value, and we just don’t know where it is? Let’s call the other side of the line ‘schmorality’. So a paperclipper’s values are schmoral.
Is it possible that a human could have values on the other side of the line (schmoral values)?
Suppose another entity, who is on the other side of the line, has a conversation with a human about a moral issue. Both entities engage in the same kind of reasoning, use the same kind of arguments and examples, so why is one reasoning called “moral reasoning” and the other just about values (schmoral reasoning)?
Suppose I am right on the edge of the line. So my values are moral values, but a slight change makes these values schmoral values. From my point of view, these two sets of values are very close. Why do you give them completely different categories? And suppose my values change slightly over time, so I cross the line and back within a day. Do I suddenly stop caring about morality, then start again? This discontinuity seems very strange to me.
I don’t say that any given concept is reasonable for all purposes, just that any concept has a very specific intended meaning, at the moment it’s considered. The concept of morality can be characterized as, roughly, referring to human-like preference, or aggregate preference of humanity-like collections of individual preferences—this is a characterization resilient to some measure of ambiguity in interpretation. The concepts themselves can’t be negotiated, they are set in stone by their intended meaning, though a different concept may be better for a given purpose.
Thanks! That actually helped a lot.