If morality exists in an objective manner, and our beliefs about it are correlated with what it is, then the orthogonality thesis is false.
If the orthogonality thesis is true, then simply being intelligent is not enough to deduce objective morality even if it exists, and any accurate beliefs we have about it are due to luck, or possibly due to defining morality in some way involving humans (as with Eliezer’s beliefs).
That being said, the orthogonality thesis may be partially true. That is, it may be that an arbitrarily advanced intelligence can have any utility function, but is more likely to have some than others. In this case, it is possible that moral realism is true and knowable, but unfriendly AI can still exist.
If morality exists in an objective manner, and our beliefs about it are correlated with what it is, then the orthogonality thesis is false.
If the orthogonality thesis is true, then simply being intelligent is not enough to deduce objective morality even if it exists, and any accurate beliefs we have about it are due to luck, or possibly due to defining morality in some way involving humans (as with Eliezer’s beliefs).
That being said, the orthogonality thesis may be partially true. That is, it may be that an arbitrarily advanced intelligence can have any utility function, but is more likely to have some than others. In this case, it is possible that moral realism is true and knowable, but unfriendly AI can still exist.