Yudkowsky’s Extrapolated volition (normative moral theory) is straightforwardly moral realist in the standard philosophical terminology. It is very similar to Frank Jackson’s Analytical Functionalism, a fact which he explicitly acknowledged in the above article (and more recently in passing here).
This doesn’t really address my objection but just labels it.
If I understand correctly, Yudkowsky merely asserts that real moral knowledge is found by
running a certain logical function over possible states of the world, where this function is analytically identical to the result of extrapolating our current decision-making process in directions such as “What if I knew more?”, “What if I had time to consider more arguments (so long as the arguments weren’t hacking my brain)?”, or “What if I understood myself better and had more self-control?”
But this is an idealization procedure, and so it falls into my dichotomy:
In particular all of the theories based on an idealization procedure fail because either
The idealization procedure is taken to include moral knowledge, creating circularity, or
The idealization procedure only includes rationality in the making of non-moral judgments, knowledge of non-moral facts, etc, in which case this is a reductionist meta-ethics that doesn’t actually cross the is-ought gap (i.e. it remains an open question whether the idealized attitudes would be good).
I don’t see a clear moral/evaluative claim baked into the listed examples there, so therefore it maintains the problem of explaining why the outcome of the idealization procedure is actually good and why you ought to care about it, i.e. crossing the is-ought gap.
(My objection is similar or maybe the same to the open-question objection to analytic naturalism, of which analytic functionalism is one type.)
Yudkowsky replies to the open question argument here.
I will add that the open question argument with respect to analytic naturalism, including Jackson’s and Yudkowsky’s theories, is just an instance of the paradox of analysis, which states that any proposed conceptual analysis is either true but trivial, or non-trivial but false. I’d reply that the solution to this paradox is that knowing a concept (understanding the meaning of a word) does, as a psychological matter of fact, not imply that we know how to define it. We only intuitively know how to use a word, but that doesn’t include the ability to easily state exactly how it relates to other concepts. Which is why the process of conceptual analysis (analytic philosophy) is not a trivial task. So “action x is right” can mean (be analytically equivalent to) something like “x is conducive to our coherent extrapolated volition” without this being a trivial semantic fact.
Regarding the “is-ought gap”: an “ought sentence” can be straightforwardly transformed into an “is sentence”: “I ought to do x” ≈ “Doing x is right”.
The non-triviality of analysis should be very familiar to anyone who has done a bit of philosophy. For example, what does it mean to say that “belief x is rational”? A conceptual analysis of epistemic rationality is highly non-obvious. Yet few people assume that there are no objective facts about what makes some beliefs rational or irrational, or that these objective facts would have to be ontologically suspect entities, or that any analysis would have to be circular or fail to bridge “the descriptive/normative gap”.
This is similar to @habryka’s reply here where I agree with the statements in the reply but I don’t think they respond to my objection.
If I understand your two points correctly they are that
An open-question critique of the idealization-procedure definition can be applied to any conceptual analysis. Yes, sure. (Irrelevant but I also don’t think the analysis of concepts is very useful.)
There is no is-ought gap because an “ought sentence” can be rephrased as an “is sentence.”
But these only address a weak “semantic” interpretation of my objection to the analysis when what I am questioning is why the proposed analysis produces normative authority. My complaint isn’t the general complaint that to define the good as the product of an idealization procedure is either trivial or false, but that there’s this actual thing (normative authority) that isn’t addressed. Likewise with (2), you can certainly rephrase an “ought sentence” into an “is sentence” but that doesn’t change it from a normative to a descriptive claim.
My question is about how an idealization procedure (like extrapolated volition or whatever else) can actually have moral authority if the whole procedure is specified in non-normative terms.
I would dispute the existence of an actual is/ought or descriptive/normative gap. If “I ought to do x” (a normative sentence) is semantically equivalent to “doing x is right”, and “doing x is right” is semantically equivalent to “x is conducive to our coherent extrapolated volition”, and the latter has a straightforward “descriptive” truth value, then “I ought to do x” has the same truth value. In which case there is no fundamental difference between descriptive and normative sentences; the supposed gap was just an illusion stemming from the superficially different sentence structure of “ought” and “is” sentences and from the apparent difficulty of defining terms like “right”.
(For clarity, I should also point out that believing “I ought to do x” (or “x is right”) does not imply “I’m motivated to do x”. See here. In particular, a psychopath can believe that various things are morally wrong while not being motivated at all to avoid doing the things he believes to be wrong. Most normal people have some degree of altruistic desires, but the correlation between moral beliefs and altruistic motivation is far from perfect. Various people believe eating meat is wrong without having significant motivation to stop eating meat.)
Yudkowsky’s Extrapolated volition (normative moral theory) is straightforwardly moral realist in the standard philosophical terminology. It is very similar to Frank Jackson’s Analytical Functionalism, a fact which he explicitly acknowledged in the above article (and more recently in passing here).
This doesn’t really address my objection but just labels it.
If I understand correctly, Yudkowsky merely asserts that real moral knowledge is found by
But this is an idealization procedure, and so it falls into my dichotomy:
I don’t see a clear moral/evaluative claim baked into the listed examples there, so therefore it maintains the problem of explaining why the outcome of the idealization procedure is actually good and why you ought to care about it, i.e. crossing the is-ought gap.
(My objection is similar or maybe the same to the open-question objection to analytic naturalism, of which analytic functionalism is one type.)
Yudkowsky replies to the open question argument here.
I will add that the open question argument with respect to analytic naturalism, including Jackson’s and Yudkowsky’s theories, is just an instance of the paradox of analysis, which states that any proposed conceptual analysis is either true but trivial, or non-trivial but false. I’d reply that the solution to this paradox is that knowing a concept (understanding the meaning of a word) does, as a psychological matter of fact, not imply that we know how to define it. We only intuitively know how to use a word, but that doesn’t include the ability to easily state exactly how it relates to other concepts. Which is why the process of conceptual analysis (analytic philosophy) is not a trivial task. So “action x is right” can mean (be analytically equivalent to) something like “x is conducive to our coherent extrapolated volition” without this being a trivial semantic fact.
Regarding the “is-ought gap”: an “ought sentence” can be straightforwardly transformed into an “is sentence”: “I ought to do x” ≈ “Doing x is right”.
The non-triviality of analysis should be very familiar to anyone who has done a bit of philosophy. For example, what does it mean to say that “belief x is rational”? A conceptual analysis of epistemic rationality is highly non-obvious. Yet few people assume that there are no objective facts about what makes some beliefs rational or irrational, or that these objective facts would have to be ontologically suspect entities, or that any analysis would have to be circular or fail to bridge “the descriptive/normative gap”.
This is similar to @habryka’s reply here where I agree with the statements in the reply but I don’t think they respond to my objection.
If I understand your two points correctly they are that
An open-question critique of the idealization-procedure definition can be applied to any conceptual analysis. Yes, sure. (Irrelevant but I also don’t think the analysis of concepts is very useful.)
There is no is-ought gap because an “ought sentence” can be rephrased as an “is sentence.”
But these only address a weak “semantic” interpretation of my objection to the analysis when what I am questioning is why the proposed analysis produces normative authority. My complaint isn’t the general complaint that to define the good as the product of an idealization procedure is either trivial or false, but that there’s this actual thing (normative authority) that isn’t addressed. Likewise with (2), you can certainly rephrase an “ought sentence” into an “is sentence” but that doesn’t change it from a normative to a descriptive claim.
My question is about how an idealization procedure (like extrapolated volition or whatever else) can actually have moral authority if the whole procedure is specified in non-normative terms.
I would dispute the existence of an actual is/ought or descriptive/normative gap. If “I ought to do x” (a normative sentence) is semantically equivalent to “doing x is right”, and “doing x is right” is semantically equivalent to “x is conducive to our coherent extrapolated volition”, and the latter has a straightforward “descriptive” truth value, then “I ought to do x” has the same truth value. In which case there is no fundamental difference between descriptive and normative sentences; the supposed gap was just an illusion stemming from the superficially different sentence structure of “ought” and “is” sentences and from the apparent difficulty of defining terms like “right”.
(For clarity, I should also point out that believing “I ought to do x” (or “x is right”) does not imply “I’m motivated to do x”. See here. In particular, a psychopath can believe that various things are morally wrong while not being motivated at all to avoid doing the things he believes to be wrong. Most normal people have some degree of altruistic desires, but the correlation between moral beliefs and altruistic motivation is far from perfect. Various people believe eating meat is wrong without having significant motivation to stop eating meat.)