I think I agree with bostrom’s 2012 position here: (On how it’s still a problem if moral realism is true; even though I think it’s false— believing moral realism is true doesn’t quite help people in designing friendly AI, as highlighted by other commentators)
The Orthogonality Thesis Intelligence and final goals are orthogonal axes along which possible agents can freely vary. In other words, more or less any level of intelligence could in principle be combined with more or less any final goal
[...]
The orthogonality thesis, as formulated here, makes a claim about the relationship between motivation and intelligence, rather than between motivation and rationality (or motivation and reason). This is because some philosophers use the word “rationality” to connote a “normatively thicker” concept than we seek to connote here with the word “intelligence”.
[...]
By “intelligence” here we mean something like instrumental rationality—skill at prediction, planning, and means-ends reasoning in general.
[...]
even if there are objective moral facts that any fully rational agent would comprehend, and even if these moral facts are somehow intrinsically motivating (such that anybody who fully comprehends them is necessarily motivated to act in accordance with them) this need not undermine the orthogonality thesis. The thesis could still be true if an agent could have impeccable instrumental rationality even whilst lacking some other faculty constitutive of rationality proper, or some faculty required for the full comprehension of the objective moral facts. (An agent could also be extremely intelligent, even superintelligent, without having full instrumental rationality in every domain.)
I think I agree with bostrom’s 2012 position here: (On how it’s still a problem if moral realism is true; even though I think it’s false— believing moral realism is true doesn’t quite help people in designing friendly AI, as highlighted by other commentators)