Surely it’s a de facto ethical theory, since it determines entirely what the FAI should do. But then, the FAI is not supposed to be a person, so that might make a difference for our use of ‘ethical’.
EDIT: but if you thought all possible minds had the same preferences, then it would be subjective nonrelative, wouldn’t it?
Maybe, though in that unlikely event I would suspect that there’s some universal law behind that odd fact about preferences, in which case I’d think it would be objective.
I’m afraid that wasn’t enough to clear it up for me. Nor is it clear how privileging the hypothesis is relevant to a discussion of logical possibility. Or are you claiming that was the wrong domain of inquiry?
Saying “X is logically possible” bears the conversational implication that X is worth considering—it raises X to conscious attention. But when we’re talking about physical possibility, “logically possible” is the wrong criterion for raising hypotheses to conscious attention, because epistemological limitations imply that every hypothesis is logically possible. Given that we have good physical reasons to draw the opposite conclusion in this case, it is generally a mistake to emphasize the possibility.
Ah, I see what you’re getting at. But it is not that I was trying to emphasize the possibility that there cannot be non-human minds in order to argue in favor of that hypothesis. Rather, I was pointing out that whether CEV is ‘relative’ or not (for purposes of this discussion) is an empirical question. For reference, I would not guess that non-human minds are physically impossible (I’d assign significantly less than 10% probability to that hypothesis).
Is CEV even an ethical theory? I thought it was more of an algorithm for extracting human preferences to put them in an AI.
Surely it’s a de facto ethical theory, since it determines entirely what the FAI should do. But then, the FAI is not supposed to be a person, so that might make a difference for our use of ‘ethical’.
hmm. Then wouldn’t it be premised on subjective relativism? (relative to humans)
Yes, I’d considered that when I wrote it, but it’s an odd use of ‘relative’ when it might be equivalent to ‘the same for everyone’.
not all possible minds, just human minds
EDIT: but if you thought all possible minds had the same preferences, then it would be subjective nonrelative, wouldn’t it?
Maybe, though in that unlikely event I would suspect that there’s some universal law behind that odd fact about preferences, in which case I’d think it would be objective.
Well I’m not sure we need to consider merely logically possible minds, and it’s logically possible that non-human minds are physically impossible.
Only in the sense that it logically possible that travel to Mars is physically impossible. The wording is deceptive.
I’m not sure what sense you’re referring to, or what you’re comparing it to, or how it’s deceptive.
Privileging the hypothesis, really.
I’m afraid that wasn’t enough to clear it up for me. Nor is it clear how privileging the hypothesis is relevant to a discussion of logical possibility. Or are you claiming that was the wrong domain of inquiry?
Saying “X is logically possible” bears the conversational implication that X is worth considering—it raises X to conscious attention. But when we’re talking about physical possibility, “logically possible” is the wrong criterion for raising hypotheses to conscious attention, because epistemological limitations imply that every hypothesis is logically possible. Given that we have good physical reasons to draw the opposite conclusion in this case, it is generally a mistake to emphasize the possibility.
Ah, I see what you’re getting at. But it is not that I was trying to emphasize the possibility that there cannot be non-human minds in order to argue in favor of that hypothesis. Rather, I was pointing out that whether CEV is ‘relative’ or not (for purposes of this discussion) is an empirical question. For reference, I would not guess that non-human minds are physically impossible (I’d assign significantly less than 10% probability to that hypothesis).