Yudkowsky replies to the open question argument here.
I will add that the open question argument with respect to analytic naturalism, including Jackson’s and Yudkowsky’s theories, is just an instance of the paradox of analysis, which states that any proposed conceptual analysis is either true but trivial, or non-trivial but false. I’d reply that the solution to this paradox is that knowing a concept (understanding the meaning of a word) does, as a psychological matter of fact, not imply that we know how to define it. We only intuitively know how to use a word, but that doesn’t include the ability to easily state exactly how it relates to other concepts. Which is why the process of conceptual analysis (analytic philosophy) is not a trivial task. So “action x is right” can mean (be analytically equivalent to) something like “x is conducive to our coherent extrapolated volition” without this being a trivial semantic fact.
Regarding the “is-ought gap”: an “ought sentence” can be straightforwardly transformed into an “is sentence”: “I ought to do x” ≈ “Doing x is right”.
The non-triviality of analysis should be very familiar to anyone who has done a bit of philosophy. For example, what does it mean to say that “belief x is rational”? A conceptual analysis of epistemic rationality is highly non-obvious. Yet few people assume that there are no objective facts about what makes some beliefs rational or irrational, or that these objective facts would have to be ontologically suspect entities, or that any analysis would have to be circular or fail to bridge “the descriptive/normative gap”.
This is similar to @habryka’s reply here where I agree with the statements in the reply but I don’t think they respond to my objection.
If I understand your two points correctly they are that
An open-question critique of the idealization-procedure definition can be applied to any conceptual analysis. Yes, sure. (Irrelevant but I also don’t think the analysis of concepts is very useful.)
There is no is-ought gap because an “ought sentence” can be rephrased as an “is sentence.”
But these only address a weak “semantic” interpretation of my objection to the analysis when what I am questioning is why the proposed analysis produces normative authority. My complaint isn’t the general complaint that to define the good as the product of an idealization procedure is either trivial or false, but that there’s this actual thing (normative authority) that isn’t addressed. Likewise with (2), you can certainly rephrase an “ought sentence” into an “is sentence” but that doesn’t change it from a normative to a descriptive claim.
My question is about how an idealization procedure (like extrapolated volition or whatever else) can actually have moral authority if the whole procedure is specified in non-normative terms.
I would dispute the existence of an actual is/ought or descriptive/normative gap. If “I ought to do x” (a normative sentence) is semantically equivalent to “doing x is right”, and “doing x is right” is semantically equivalent to “x is conducive to our coherent extrapolated volition”, and the latter has a straightforward “descriptive” truth value, then “I ought to do x” has the same truth value. In which case there is no fundamental difference between descriptive and normative sentences; the supposed gap was just an illusion stemming from the superficially different sentence structure of “ought” and “is” sentences and from the apparent difficulty of defining terms like “right”.
(For clarity, I should also point out that believing “I ought to do x” (or “x is right”) does not imply “I’m motivated to do x”. See here. In particular, a psychopath can believe that various things are morally wrong while not being motivated at all to avoid doing the things he believes to be wrong. Most normal people have some degree of altruistic desires, but the correlation between moral beliefs and altruistic motivation is far from perfect. Various people believe eating meat is wrong without having significant motivation to stop eating meat.)
Yudkowsky replies to the open question argument here.
I will add that the open question argument with respect to analytic naturalism, including Jackson’s and Yudkowsky’s theories, is just an instance of the paradox of analysis, which states that any proposed conceptual analysis is either true but trivial, or non-trivial but false. I’d reply that the solution to this paradox is that knowing a concept (understanding the meaning of a word) does, as a psychological matter of fact, not imply that we know how to define it. We only intuitively know how to use a word, but that doesn’t include the ability to easily state exactly how it relates to other concepts. Which is why the process of conceptual analysis (analytic philosophy) is not a trivial task. So “action x is right” can mean (be analytically equivalent to) something like “x is conducive to our coherent extrapolated volition” without this being a trivial semantic fact.
Regarding the “is-ought gap”: an “ought sentence” can be straightforwardly transformed into an “is sentence”: “I ought to do x” ≈ “Doing x is right”.
The non-triviality of analysis should be very familiar to anyone who has done a bit of philosophy. For example, what does it mean to say that “belief x is rational”? A conceptual analysis of epistemic rationality is highly non-obvious. Yet few people assume that there are no objective facts about what makes some beliefs rational or irrational, or that these objective facts would have to be ontologically suspect entities, or that any analysis would have to be circular or fail to bridge “the descriptive/normative gap”.
This is similar to @habryka’s reply here where I agree with the statements in the reply but I don’t think they respond to my objection.
If I understand your two points correctly they are that
An open-question critique of the idealization-procedure definition can be applied to any conceptual analysis. Yes, sure. (Irrelevant but I also don’t think the analysis of concepts is very useful.)
There is no is-ought gap because an “ought sentence” can be rephrased as an “is sentence.”
But these only address a weak “semantic” interpretation of my objection to the analysis when what I am questioning is why the proposed analysis produces normative authority. My complaint isn’t the general complaint that to define the good as the product of an idealization procedure is either trivial or false, but that there’s this actual thing (normative authority) that isn’t addressed. Likewise with (2), you can certainly rephrase an “ought sentence” into an “is sentence” but that doesn’t change it from a normative to a descriptive claim.
My question is about how an idealization procedure (like extrapolated volition or whatever else) can actually have moral authority if the whole procedure is specified in non-normative terms.
I would dispute the existence of an actual is/ought or descriptive/normative gap. If “I ought to do x” (a normative sentence) is semantically equivalent to “doing x is right”, and “doing x is right” is semantically equivalent to “x is conducive to our coherent extrapolated volition”, and the latter has a straightforward “descriptive” truth value, then “I ought to do x” has the same truth value. In which case there is no fundamental difference between descriptive and normative sentences; the supposed gap was just an illusion stemming from the superficially different sentence structure of “ought” and “is” sentences and from the apparent difficulty of defining terms like “right”.
(For clarity, I should also point out that believing “I ought to do x” (or “x is right”) does not imply “I’m motivated to do x”. See here. In particular, a psychopath can believe that various things are morally wrong while not being motivated at all to avoid doing the things he believes to be wrong. Most normal people have some degree of altruistic desires, but the correlation between moral beliefs and altruistic motivation is far from perfect. Various people believe eating meat is wrong without having significant motivation to stop eating meat.)