As an example of a fairly powerful optimization process with very unhuman goals, you can cite evolution, which is superhuman in some ways, yet quite amoral.
And yet this same amoral unhuman optimization process produced humans and morals, so what is the lesson there?
The lesson is “be sure to quine your goal system properly”. We will be the death of evolution, let’s not let the same happen to us.
As an example of a fairly powerful optimization process with very unhuman goals, you can cite evolution, which is superhuman in some ways, yet quite amoral.
And yet this same amoral unhuman optimization process produced humans and morals, so what is the lesson there?
The lesson is “be sure to quine your goal system properly”. We will be the death of evolution, let’s not let the same happen to us.