Luke, what is the meta-purpose of this sequence? It seems that you are trying to dissolve the persisting confusion about ethics, but what is the underlying purpose? I doubt this sequence is a terminal goal of yours, am I wrong? The reason for my doubt is that, before you started writing this sequence, you decided to join the visiting fellows program at the Singularity Institute for Artificial Intelligence. Which makes me infer that the purpose of this sequence has to do with friendly AI research.
If I am right that the whole purpose of this sequence has to do with friendly AI research, then how is it useful to devote so much resources to explaining the basics, instead of trying to figure out how to design, or define mathematically, something that could extrapolate human volition?
If I am right that the whole purpose of this sequence has to do with friendly AI research,
Luke confirms your guess in the comments section of this post.
then how is it useful to devote so much resources to explaining the basics, instead of trying to figure out how to design, or define mathematically, something that could extrapolate human volition?
I can see a couple of benefits of going over the basics first. One, it lets Luke confirm that his understanding of the basics are correct, and two, it can interest others to work in the same area (or if they are already interested, can help bring them up to the same level as Luke). I would like to see Luke continue this sequence. (Unless of course he already has some good new ideas about FAI, in which case write those down first!)
But how does ethics matter for friendly AI? If a friendly AI is going to figure out what humans desire, by extrapolating their volition, might it conclude that our volition is immoral and therefore undesirable?
What morality unit would it have other than human’s volition?
I am not sure what a ‘morality unit’ is supposed to be or how it would be different from a volition unit. Either morality is part of our volition, instrumental or an imperative. In each case one could ask what we want and arrive at morality.
What I’m saying is that: If Clippy tried to calculate our volition, he would conclude that our volition is immoral. (Probably. Maybe our volition IS paperclips.)
But if we programmed an AI to calculate our volition and use that as its volition, and our morality as its morality, and so on, then it would not find our volition immoral unless we find our volition immoral, which seems unlikely.
An AI that was smarter than us might deduce that we were not applying the Deep Structure of our morality properly because of bias or limited intelligence. It might
conclude that human morality requires humans to greatly reduce their numbers
in order to lessen the impact on other species, for instance.
I can see a couple of benefits of going over the basics first. One, it lets Luke confirm that his understanding of the basics are correct, and two, it can interest others to work in the same area (or if they are already interested, can help bring them up to the same level as Luke).
Luke, what is the meta-purpose of this sequence? It seems that you are trying to dissolve the persisting confusion about ethics, but what is the underlying purpose? I doubt this sequence is a terminal goal of yours, am I wrong? The reason for my doubt is that, before you started writing this sequence, you decided to join the visiting fellows program at the Singularity Institute for Artificial Intelligence. Which makes me infer that the purpose of this sequence has to do with friendly AI research.
If I am right that the whole purpose of this sequence has to do with friendly AI research, then how is it useful to devote so much resources to explaining the basics, instead of trying to figure out how to design, or define mathematically, something that could extrapolate human volition?
Luke confirms your guess in the comments section of this post.
I can see a couple of benefits of going over the basics first. One, it lets Luke confirm that his understanding of the basics are correct, and two, it can interest others to work in the same area (or if they are already interested, can help bring them up to the same level as Luke). I would like to see Luke continue this sequence. (Unless of course he already has some good new ideas about FAI, in which case write those down first!)
But how does ethics matter for friendly AI? If a friendly AI is going to figure out what humans desire, by extrapolating their volition, might it conclude that our volition is immoral and therefore undesirable?
What morality unit would it have other than human’s volition?
If it has another, separate volition unit, yes.
If not, then only if humans fundamentally self-contradict, which seems unlikely, because biological systems are pretty robust to that.
I am not sure what a ‘morality unit’ is supposed to be or how it would be different from a volition unit. Either morality is part of our volition, instrumental or an imperative. In each case one could ask what we want and arrive at morality.
What I’m saying is that: If Clippy tried to calculate our volition, he would conclude that our volition is immoral. (Probably. Maybe our volition IS paperclips.)
But if we programmed an AI to calculate our volition and use that as its volition, and our morality as its morality, and so on, then it would not find our volition immoral unless we find our volition immoral, which seems unlikely.
An AI that was smarter than us might deduce that we were not applying the Deep Structure of our morality properly because of bias or limited intelligence. It might conclude that human morality requires humans to greatly reduce their numbers in order to lessen the impact on other species, for instance.
Correct!