I’m not the original poster here, but I’m genuinely worried about (c). I’m not sure that humanity’s revealed preferences are consistent with a world in which we believe that all people matter. Between the large scale wars and genocides, slavery, and even just the ongoing stark divide between the rich and poor, I have a hard time believing that respect for sentience is actually one of humanity’s strong core virtues. And if we extend out to all sentient life, we’re forced to contend with our reaction to large scale animal welfare (even I am not vegetarian, although I feel I “should” be).
I think humanity’s actual stance is “In-group life always matters. Out-group life usually matters, but even relatively small economic or political concerns can make us change our minds.”. We care about it some, but not beyond the point of inconvenience.
I’d be interested in finding firmer philosophical ground for the “all sentient life matters” claim. Not because I personally need to be convinced of it, but rather because I want to be confident that a hypothetical superintelligence with “human” virtues would be convinced of this.
(P.s. Your original point about “building and then enslaving a superintelligence is not just exceptionally difficult, but also morally wrong” is correct, concise, well-put, and underappreciated by the public. I’ve started framing my AI X-risk discussions with X-risk skeptics around similar terms.)
I’m not the original poster here, but I’m genuinely worried about (c). I’m not sure that humanity’s revealed preferences are consistent with a world in which we believe that all people matter. Between the large scale wars and genocides, slavery, and even just the ongoing stark divide between the rich and poor, I have a hard time believing that respect for sentience is actually one of humanity’s strong core virtues. And if we extend out to all sentient life, we’re forced to contend with our reaction to large scale animal welfare (even I am not vegetarian, although I feel I “should” be).
I think humanity’s actual stance is “In-group life always matters. Out-group life usually matters, but even relatively small economic or political concerns can make us change our minds.”. We care about it some, but not beyond the point of inconvenience.
I’d be interested in finding firmer philosophical ground for the “all sentient life matters” claim. Not because I personally need to be convinced of it, but rather because I want to be confident that a hypothetical superintelligence with “human” virtues would be convinced of this.
(P.s. Your original point about “building and then enslaving a superintelligence is not just exceptionally difficult, but also morally wrong” is correct, concise, well-put, and underappreciated by the public. I’ve started framing my AI X-risk discussions with X-risk skeptics around similar terms.)