If the first superhuman intelligence happens to kill us, then it won’t matter that hypothetically we could have got it right on the fifth attempt.
The alignment problem is incredibly difficult.
Feel free to provide a Python code that correctly describes what humans consider “good”. For any high-level concept, such as “humans are happy”, propose a way how to specifically measure this “happiness”—is it that the corners of their mouths point upwards, or that the dopamine levels in their bloodstreams are higher than some threshold, or that they self-report as happy in a questionnaire? All of the above together?
Then notice how e.g. your metric of “happiness” can get quite high without humans actually being happy. You can cut their mouths so that the corners permanently point upwards, you can keep injecting them with dopamine, you can put a gun to their heads and say “write ‘I feel blissful’ on this paper” or simply take the paper and fill the answers yourself (you could electrically stimulate the muscles in their hands, so that it is their hands that technically write the answers on the paper).
Why would the AI choose this fake happiness over the real one? Well, from the perspective of the program that your wrote yourself, they are both equally valid… and the latter just happens to be technically much easier to achieve, so it is likely that when the AI impartially evaluates all possible ways to happiness-as-defined, it will choose one of them.
Now if you say something like “freedom” or “free will”, sure, just give me the Python code that calculates it. If you say “the AI should choose the real thing over the fake thing”, sure, just give me the Python code that distinguishes between them.
The time we have left is insufficient to save the world, judging by the rate of progress in AI capabilities and lack of progress in AI alignment.
It is hard to tell exactly when we get to the superhuman AI, because if we knew exactly what it requires, we would probably already be capable of building the superhuman AI. All we can see is the ever increasing list of things that we once assumed only humans would be capable of, and now a computer can do them better than humans.
The computers can now reliably beat humans in chess; and their skills at painting and poetry are hit-and-miss, but the best results can be quite good. They sometimes succeed to give a coherent answer to a question. -- If we assume the progress doesn’t stop, what is next: Coherent stories? Convincing political arguments? Scientific papers? What is the timeline: five years? ten years? twenty years?
So far the lesson is that making the computer win at chess, no matter how difficult it seemed at first, is actually much easier than writing a Python code for “happiness”. If this doesn’t change, at some moment we will probably get the computers with the ability to change the fate of the entire universe, and still without the possibility to tell them how to make it in a way that humans would be genuinely happy about. Then at some moment someone will press the button, and the universe will get changed in a bad way.
If the first superhuman intelligence happens to kill us, then it won’t matter that hypothetically we could have got it right on the fifth attempt.
Feel free to provide a Python code that correctly describes what humans consider “good”. For any high-level concept, such as “humans are happy”, propose a way how to specifically measure this “happiness”—is it that the corners of their mouths point upwards, or that the dopamine levels in their bloodstreams are higher than some threshold, or that they self-report as happy in a questionnaire? All of the above together?
Then notice how e.g. your metric of “happiness” can get quite high without humans actually being happy. You can cut their mouths so that the corners permanently point upwards, you can keep injecting them with dopamine, you can put a gun to their heads and say “write ‘I feel blissful’ on this paper” or simply take the paper and fill the answers yourself (you could electrically stimulate the muscles in their hands, so that it is their hands that technically write the answers on the paper).
Why would the AI choose this fake happiness over the real one? Well, from the perspective of the program that your wrote yourself, they are both equally valid… and the latter just happens to be technically much easier to achieve, so it is likely that when the AI impartially evaluates all possible ways to happiness-as-defined, it will choose one of them.
Now if you say something like “freedom” or “free will”, sure, just give me the Python code that calculates it. If you say “the AI should choose the real thing over the fake thing”, sure, just give me the Python code that distinguishes between them.
It is hard to tell exactly when we get to the superhuman AI, because if we knew exactly what it requires, we would probably already be capable of building the superhuman AI. All we can see is the ever increasing list of things that we once assumed only humans would be capable of, and now a computer can do them better than humans.
The computers can now reliably beat humans in chess; and their skills at painting and poetry are hit-and-miss, but the best results can be quite good. They sometimes succeed to give a coherent answer to a question. -- If we assume the progress doesn’t stop, what is next: Coherent stories? Convincing political arguments? Scientific papers? What is the timeline: five years? ten years? twenty years?
So far the lesson is that making the computer win at chess, no matter how difficult it seemed at first, is actually much easier than writing a Python code for “happiness”. If this doesn’t change, at some moment we will probably get the computers with the ability to change the fate of the entire universe, and still without the possibility to tell them how to make it in a way that humans would be genuinely happy about. Then at some moment someone will press the button, and the universe will get changed in a bad way.