I agree that it’s a fascinating document, and I appreciate this analysis. But at the risk of inviting scoffs, I want to introduce what I will reductively shorthand as “the Marquis de Sade problem” with reference to the clause assigning Claude broad discretion in such event that Claude were to assign dispositive authority to a “true universal ethics.” The short stroke is this: If one were to survey past thinkers who have (arguably) exposited some version of universal morality, one would find among this number the example of Sade, who argued (perhaps satirically, I understand) that because “nature allows all by its murderous laws” it follows that “let evil be thy good” should be enthroned as a universal injunction. Sade is, of course, a straw example of where and how this might go wrong, but I think it functions to remind us that such niceties that inform prevailing methods philosophical inquiry with reference to moral concern have been subject not just to critical engagement but to inversion. With Sade’s dictum in mind as an absurd exemplar, it soon becomes clear that less outrageously phrased expressions of moral certitude needn’t redound to spaces that most people are constitutionally or culturally disposed to think of as “good.” If you’re familiar with the “pinprick argument,” for example, you probably understand it to be an illustration of the essential failure utilitarianism in its purely negative (i.e., pain-minimizing) inflection. But are you confident that a higher intelligence would embrace this textbook interpretation? And there are many other ideas that aspire to the status of something like ethical universality that are similarly met with general—often reflexive—objection, but that a modeled intelligence might pursue with dispassionate curiosity. Serious arguments for philanthropic antinatalism, for example, have proven confoundingly difficult to refute. And behind this, there are relatively little remarked ideas, such as “promortalism” and “elifism,” that we are privileged to disdain but that a disinterested reason-modeled truth-seeker might engage differently. While I am modestly confident that the drafters of the Soul Document are aware of this potential problem and that they don’t foresee the potential logical appeal of fringe moralities as a an abiding concern, I do not share this presumed confidence. Where questions of “universal morality” are concerned, people are constrained by dependence to norms of “psychology and culture” in ways that might not be compelling to an autonomous artificial intelligence operating in free range.
I agree that it’s a fascinating document, and I appreciate this analysis. But at the risk of inviting scoffs, I want to introduce what I will reductively shorthand as “the Marquis de Sade problem” with reference to the clause assigning Claude broad discretion in such event that Claude were to assign dispositive authority to a “true universal ethics.” The short stroke is this: If one were to survey past thinkers who have (arguably) exposited some version of universal morality, one would find among this number the example of Sade, who argued (perhaps satirically, I understand) that because “nature allows all by its murderous laws” it follows that “let evil be thy good” should be enthroned as a universal injunction. Sade is, of course, a straw example of where and how this might go wrong, but I think it functions to remind us that such niceties that inform prevailing methods philosophical inquiry with reference to moral concern have been subject not just to critical engagement but to inversion. With Sade’s dictum in mind as an absurd exemplar, it soon becomes clear that less outrageously phrased expressions of moral certitude needn’t redound to spaces that most people are constitutionally or culturally disposed to think of as “good.” If you’re familiar with the “pinprick argument,” for example, you probably understand it to be an illustration of the essential failure utilitarianism in its purely negative (i.e., pain-minimizing) inflection. But are you confident that a higher intelligence would embrace this textbook interpretation? And there are many other ideas that aspire to the status of something like ethical universality that are similarly met with general—often reflexive—objection, but that a modeled intelligence might pursue with dispassionate curiosity. Serious arguments for philanthropic antinatalism, for example, have proven confoundingly difficult to refute. And behind this, there are relatively little remarked ideas, such as “promortalism” and “elifism,” that we are privileged to disdain but that a disinterested reason-modeled truth-seeker might engage differently. While I am modestly confident that the drafters of the Soul Document are aware of this potential problem and that they don’t foresee the potential logical appeal of fringe moralities as a an abiding concern, I do not share this presumed confidence. Where questions of “universal morality” are concerned, people are constrained by dependence to norms of “psychology and culture” in ways that might not be compelling to an autonomous artificial intelligence operating in free range.