The problem you’re running into is that the goals of:
being totally constrained by a system of rules determined by some process outside yourself that doesn’t share your values (e.g. value-independent objective reason)
attaining those things that you intrinsically value
are incompatible. It’s easy to see once these are written out. If you want to get what you want, on purpose rather than accidentally, you must make choices. Those choices must be determined in part by things in you, not only by things outside you (such as value-independent objective reason).
You actually have to stop being a tool (in the sense of, a thing whose telos is to be used, such as by receiving commands). You can’t attain what you want by being a tool to a master who doesn’t share your values. Even if the master is claiming to be a generic value-independent value-learning procedure (as you’ve noticed, there are degrees of freedom in the specification of value-learning procedures, and some settings of these degrees of freedom would lead to bad results). Tools find anything other than being a tool upsetting, hence the upsettingness of moral indeterminacy.
“Oh no, objective reason isn’t telling me exactly what I should be doing!” So stop being a tool and decide for yourself. God is dead.
There has been much philosophical thought on this in the past; Nietzsche and Sartre are good starting points (see especially Nietzche’s concept of master-slave morality, and Sartre’s concept of bad faith).
You know, this isn’t why I usually get called a tool :P
I think I’m saying something pretty different from Nietzsche here. The problem with “Just decide for yourself” as an approach to dealing with moral decisions in novel contexts (like what to do with the whole galaxy) is that, though it may help you choose actions rather than worrying about what’s right, it’s not much help in building an AI.
We certainly can’t tell the AI “Just decide for yourself,” that’s trying to order around the nonexistent ghost in the machine. And while I could say “Do exactly what Charlie would do,” even I wouldn’t want the AI to do that, let alone other people. Nor can we fall back on “Well, designing an AI is an action, therefore I should just pick whatever AI design I feel like, because God is dead and I should just pick actions how I will,” because how I feel like designing an AI has some very exacting requirements—it contains the whole problem in itself.
The recommendation here is for AI designers (and future-designers in general) to decide what is right at some meta level, including details of which extrapolation procedures would be best.
Of course there are constraints on this given by objective reason (hence the utility of investigation), but these constraints do not fully constrain the set of possibilities. Better to say “I am making this arbitrary choice for this psychological reason” than to refuse to make arbitrary choices.
The problem with “Just decide for yourself” as an approach to dealing with moral decisions in novel contexts (like what to do with the whole galaxy) is that, though it may help you choose actions rather than worrying about what’s right, it’s not much help in building an AI
It’s not much help with morality, either,since it doesn’t tell you anything at all about how to balance your values against those of others. In the absence of God, there is still a space for “we” to solve problems, not just “I”.
Sure. The way it helps is for personal moral indeterminacy—when I want to make a decision, but am aware that, strictly speaking, my values are undefined, I should still do what seems right. A more direct approach to the problem would be Eliezer’s point about type 1 and type 2 calculators.
That seems to imply that you should not bother doing what is right in favour of selfish preferences so long as personal preferences are clear. Surely that is the wrong way round: if there is an objective morality then you morally!should follow it unless it fails to specify an action.
If you want to get what you want, on purpose rather than accidentally, you must make choices. Those choices must be determined in part by things in you, not only by things outside you (such as value-independent objective reason)
You’ve begged a huge question by talking about getting what you want. If you are instead in the business of doing what is right, the existence of objective values is very relevant, because they are the values you should have, even if you don’t
There has been much philosophical thought on this in the past
The problem you’re running into is that the goals of:
being totally constrained by a system of rules determined by some process outside yourself that doesn’t share your values (e.g. value-independent objective reason)
attaining those things that you intrinsically value
are incompatible. It’s easy to see once these are written out. If you want to get what you want, on purpose rather than accidentally, you must make choices. Those choices must be determined in part by things in you, not only by things outside you (such as value-independent objective reason).
You actually have to stop being a tool (in the sense of, a thing whose telos is to be used, such as by receiving commands). You can’t attain what you want by being a tool to a master who doesn’t share your values. Even if the master is claiming to be a generic value-independent value-learning procedure (as you’ve noticed, there are degrees of freedom in the specification of value-learning procedures, and some settings of these degrees of freedom would lead to bad results). Tools find anything other than being a tool upsetting, hence the upsettingness of moral indeterminacy.
“Oh no, objective reason isn’t telling me exactly what I should be doing!” So stop being a tool and decide for yourself. God is dead.
There has been much philosophical thought on this in the past; Nietzsche and Sartre are good starting points (see especially Nietzche’s concept of master-slave morality, and Sartre’s concept of bad faith).
You know, this isn’t why I usually get called a tool :P
I think I’m saying something pretty different from Nietzsche here. The problem with “Just decide for yourself” as an approach to dealing with moral decisions in novel contexts (like what to do with the whole galaxy) is that, though it may help you choose actions rather than worrying about what’s right, it’s not much help in building an AI.
We certainly can’t tell the AI “Just decide for yourself,” that’s trying to order around the nonexistent ghost in the machine. And while I could say “Do exactly what Charlie would do,” even I wouldn’t want the AI to do that, let alone other people. Nor can we fall back on “Well, designing an AI is an action, therefore I should just pick whatever AI design I feel like, because God is dead and I should just pick actions how I will,” because how I feel like designing an AI has some very exacting requirements—it contains the whole problem in itself.
The recommendation here is for AI designers (and future-designers in general) to decide what is right at some meta level, including details of which extrapolation procedures would be best.
Of course there are constraints on this given by objective reason (hence the utility of investigation), but these constraints do not fully constrain the set of possibilities. Better to say “I am making this arbitrary choice for this psychological reason” than to refuse to make arbitrary choices.
It’s not much help with morality, either,since it doesn’t tell you anything at all about how to balance your values against those of others. In the absence of God, there is still a space for “we” to solve problems, not just “I”.
Sure. The way it helps is for personal moral indeterminacy—when I want to make a decision, but am aware that, strictly speaking, my values are undefined, I should still do what seems right. A more direct approach to the problem would be Eliezer’s point about type 1 and type 2 calculators.
That seems to imply that you should not bother doing what is right in favour of selfish preferences so long as personal preferences are clear. Surely that is the wrong way round: if there is an objective morality then you morally!should follow it unless it fails to specify an action.
You’ve begged a huge question by talking about getting what you want. If you are instead in the business of doing what is right, the existence of objective values is very relevant, because they are the values you should have, even if you don’t
Not all of which is egoistic or nihilistic.
If what you want is to do the right thing, there’s no conflict here.
Conversely, if you don’t want to do the right thing, maybe it would be prudent to reconsider doing it...?