“If we do build an AGI, its actions will determine what is done with the universe.
If the first such AGI we build turns out to be an unfriendly AI that is optimizing for something other than humans and human values, all value in the universe will be destroyed. We are made of atoms that could be used for something else.”
Especially in the sense of why the first unfriendly AI we built will immediately be uncontainable and surpass current human civilization’s ability to destroy it?
Sorry I took awhile to find this. I think the best short answer to this question is Scott Alexander’s Superintelligence FAQ.
The arguments involved here deal with a fair number of areas where human intuitions just aren’t very good (i.e. we’re used to dealing with linear systems, not exponential ones). So if Scott’s post isn’t persuasive to you, the next best answer I have is unfortunately “read the sequences.”
Thanks for asking! This has been the subject of much virtual ink, much of it here (e.g. the book Superintelligence, and much of Eliezer’s writings, including the whole “Foom” debate with Robin Hanson). Rather than try to summarize that debate, I’ll encourage others to chime in with the links and explanations they think are best, since this is an important thing to get right and I’d like to see what our community currently thinks is the best way to explain this. I’m not thrilled that I don’t have a great go-to here.
If no one comes up with anything I’m happy with within the week, I’ll see if I can do better.
Can you expand on this point:
“If we do build an AGI, its actions will determine what is done with the universe.
If the first such AGI we build turns out to be an unfriendly AI that is optimizing for something other than humans and human values, all value in the universe will be destroyed. We are made of atoms that could be used for something else.”
Especially in the sense of why the first unfriendly AI we built will immediately be uncontainable and surpass current human civilization’s ability to destroy it?
Sorry I took awhile to find this. I think the best short answer to this question is Scott Alexander’s Superintelligence FAQ.
The arguments involved here deal with a fair number of areas where human intuitions just aren’t very good (i.e. we’re used to dealing with linear systems, not exponential ones). So if Scott’s post isn’t persuasive to you, the next best answer I have is unfortunately “read the sequences.”
Thanks for asking! This has been the subject of much virtual ink, much of it here (e.g. the book Superintelligence, and much of Eliezer’s writings, including the whole “Foom” debate with Robin Hanson). Rather than try to summarize that debate, I’ll encourage others to chime in with the links and explanations they think are best, since this is an important thing to get right and I’d like to see what our community currently thinks is the best way to explain this. I’m not thrilled that I don’t have a great go-to here.
If no one comes up with anything I’m happy with within the week, I’ll see if I can do better.