One reason why they’re not a soverign is because they have high uncertainty about e.g. what they think is good, and avoid taking actions that violate deontological constraints or virtue ethics constraints or other “common sense morality.”
I mean, intrinsically caring about “common sense morality” sounds to me like you basically already have an AI you can approximately let run wild.
Superintelligences plus a moral foundation that reliably covers all the common sense morality seems like it’s really very close to an AI that would just like to go off and implement CEV. CEV looks really good according to most forms of common-sense morality I can think of. My current guess is almost all the complexity of our values lives in the “common sense” part of morality.
I mean, intrinsically caring about “common sense morality” sounds to me like you basically already have an AI you can approximately let run wild.
Superintelligences plus a moral foundation that reliably covers all the common sense morality seems like it’s really very close to an AI that would just like to go off and implement CEV. CEV looks really good according to most forms of common-sense morality I can think of. My current guess is almost all the complexity of our values lives in the “common sense” part of morality.