I think even now AI can understand close enough what we want within the distribution. I.e. in a world that is similar to what it is now.
Problems will arise when the world will significantly change, even if it changes along with our wishes. Our values are just not designed for the reality where, for example, people can arbitrarily change themselves. Or reality where ANY human’s mental activity is obsolete, because AI can predict what the human want and how to get it before human can even articulate it.
I think even now AI can understand close enough what we want within the distribution. I.e. in a world that is similar to what it is now.
Problems will arise when the world will significantly change, even if it changes along with our wishes. Our values are just not designed for the reality where, for example, people can arbitrarily change themselves. Or reality where ANY human’s mental activity is obsolete, because AI can predict what the human want and how to get it before human can even articulate it.