Just like you wouldn’t want an AI to optimize for only some of the humans, you wouldn’t want an AI to optimize for only some of the values. And, as I keep emphasizing for exactly this reason, we’ve got a lot of values.
What if the AI emulates some/many/all human brains in order to get a complete list of our values? It could design its own value system better than any human.
Just like you wouldn’t want an AI to optimize for only some of the humans, you wouldn’t want an AI to optimize for only some of the values. And, as I keep emphasizing for exactly this reason, we’ve got a lot of values.
What if the AI emulates some/many/all human brains in order to get a complete list of our values? It could design its own value system better than any human.
There is no machine in the ghost.