You know, this isn’t why I usually get called a tool :P
I think I’m saying something pretty different from Nietzsche here. The problem with “Just decide for yourself” as an approach to dealing with moral decisions in novel contexts (like what to do with the whole galaxy) is that, though it may help you choose actions rather than worrying about what’s right, it’s not much help in building an AI.
We certainly can’t tell the AI “Just decide for yourself,” that’s trying to order around the nonexistent ghost in the machine. And while I could say “Do exactly what Charlie would do,” even I wouldn’t want the AI to do that, let alone other people. Nor can we fall back on “Well, designing an AI is an action, therefore I should just pick whatever AI design I feel like, because God is dead and I should just pick actions how I will,” because how I feel like designing an AI has some very exacting requirements—it contains the whole problem in itself.
The recommendation here is for AI designers (and future-designers in general) to decide what is right at some meta level, including details of which extrapolation procedures would be best.
Of course there are constraints on this given by objective reason (hence the utility of investigation), but these constraints do not fully constrain the set of possibilities. Better to say “I am making this arbitrary choice for this psychological reason” than to refuse to make arbitrary choices.
The problem with “Just decide for yourself” as an approach to dealing with moral decisions in novel contexts (like what to do with the whole galaxy) is that, though it may help you choose actions rather than worrying about what’s right, it’s not much help in building an AI
It’s not much help with morality, either,since it doesn’t tell you anything at all about how to balance your values against those of others. In the absence of God, there is still a space for “we” to solve problems, not just “I”.
Sure. The way it helps is for personal moral indeterminacy—when I want to make a decision, but am aware that, strictly speaking, my values are undefined, I should still do what seems right. A more direct approach to the problem would be Eliezer’s point about type 1 and type 2 calculators.
That seems to imply that you should not bother doing what is right in favour of selfish preferences so long as personal preferences are clear. Surely that is the wrong way round: if there is an objective morality then you morally!should follow it unless it fails to specify an action.
You know, this isn’t why I usually get called a tool :P
I think I’m saying something pretty different from Nietzsche here. The problem with “Just decide for yourself” as an approach to dealing with moral decisions in novel contexts (like what to do with the whole galaxy) is that, though it may help you choose actions rather than worrying about what’s right, it’s not much help in building an AI.
We certainly can’t tell the AI “Just decide for yourself,” that’s trying to order around the nonexistent ghost in the machine. And while I could say “Do exactly what Charlie would do,” even I wouldn’t want the AI to do that, let alone other people. Nor can we fall back on “Well, designing an AI is an action, therefore I should just pick whatever AI design I feel like, because God is dead and I should just pick actions how I will,” because how I feel like designing an AI has some very exacting requirements—it contains the whole problem in itself.
The recommendation here is for AI designers (and future-designers in general) to decide what is right at some meta level, including details of which extrapolation procedures would be best.
Of course there are constraints on this given by objective reason (hence the utility of investigation), but these constraints do not fully constrain the set of possibilities. Better to say “I am making this arbitrary choice for this psychological reason” than to refuse to make arbitrary choices.
It’s not much help with morality, either,since it doesn’t tell you anything at all about how to balance your values against those of others. In the absence of God, there is still a space for “we” to solve problems, not just “I”.
Sure. The way it helps is for personal moral indeterminacy—when I want to make a decision, but am aware that, strictly speaking, my values are undefined, I should still do what seems right. A more direct approach to the problem would be Eliezer’s point about type 1 and type 2 calculators.
That seems to imply that you should not bother doing what is right in favour of selfish preferences so long as personal preferences are clear. Surely that is the wrong way round: if there is an objective morality then you morally!should follow it unless it fails to specify an action.