My own biggest disagreement with you is the idea that morality and values are objective. While I’m a moral realist, I’m of the weakest kind of realist and view morals and values as inherently subjective. In other words there’s no fact of the matter here, and post-modernism is actually useful here (I’m a strong critic of post-modernism, but it’s basically correct vis-a-vis morality and values.)
I think you misunderstand EY if you think he believes that morality and values are objective. If they were, then alignment would be easy because as long as the AI was smart enough, it could be depended on to figure out the “correct” morality and values. The common values that humanity shares are probably in part arbitrary evolutionary accidents. The goal is to create AI with values that allow humanity to live by its values, instead of creating an AI with non-overlapping values caused by its own design accidents. (EY’s article Sorting pebbles into correct heaps implies some of these ideas.)
My own biggest disagreement with you is the idea that morality and values are objective. While I’m a moral realist, I’m of the weakest kind of realist and view morals and values as inherently subjective. In other words there’s no fact of the matter here, and post-modernism is actually useful here (I’m a strong critic of post-modernism, but it’s basically correct vis-a-vis morality and values.)
I think you misunderstand EY if you think he believes that morality and values are objective. If they were, then alignment would be easy because as long as the AI was smart enough, it could be depended on to figure out the “correct” morality and values. The common values that humanity shares are probably in part arbitrary evolutionary accidents. The goal is to create AI with values that allow humanity to live by its values, instead of creating an AI with non-overlapping values caused by its own design accidents. (EY’s article Sorting pebbles into correct heaps implies some of these ideas.)