an assumption that objective norms / values do not exist. In my opinion AGI would not make this assumption
The question isn’t whether every AGI would or would not make this assumption, but whether it’s actually true, and therefore whether it’s true that a powerful AGI could have a wide range of goals or values, including the possibility that they’re alien or contradictory to common human values.
I think it’s highly unlikely that objective norms/values exist, and that weak versions of orthogonality (not literally ANY goals are possible, but enough bad ones to still be worried) are true. Even more strongly, I think it hasn’t been shown that they’re false, and we should take the possibility very seriously.
The question isn’t whether every AGI would or would not make this assumption, but whether it’s actually true, and therefore whether it’s true that a powerful AGI could have a wide range of goals or values, including the possibility that they’re alien or contradictory to common human values.
I think it’s highly unlikely that objective norms/values exist, and that weak versions of orthogonality (not literally ANY goals are possible, but enough bad ones to still be worried) are true. Even more strongly, I think it hasn’t been shown that they’re false, and we should take the possibility very seriously.
Could you read my comment here and let me know what you think?