Yes, I claim, the orthogonality thesis is at odds with moral realism. I wrote a post about it. Also see Bentham’s Bulldog’s post about this. Here’s the basic argument: 1. Assume moral realism; there are true facts about morality.
2. Intelligence is causally correlated with having true beliefs.
3. Intelligence is causally correlated with having true moral beliefs.
4. Moral beliefs constrain final goals; believing “X is morally wrong” is a very good reason and motivator for not doing X.
5. Superintelligent agents will likely have final goals that cohere with their (likely true) moral beliefs.
6. Superintelligent agents are likely to be moral.
Yes, I claim, the orthogonality thesis is at odds with moral realism.
I wrote a post about it. Also see Bentham’s Bulldog’s post about this.
Here’s the basic argument:
1. Assume moral realism; there are true facts about morality.
2. Intelligence is causally correlated with having true beliefs.
3. Intelligence is causally correlated with having true moral beliefs.
4. Moral beliefs constrain final goals; believing “X is morally wrong” is a very good reason and motivator for not doing X.
5. Superintelligent agents will likely have final goals that cohere with their (likely true) moral beliefs.
6. Superintelligent agents are likely to be moral.