I agree that there are several reasons why solving the value alignment problem is important.
Note that when I said that Bostrom should “modify” his reply I didn’t mean that he should make a different point instead of the point he made, but rather meant that he should make another point in addition to the point he already made. As I said:
While what [Bostrom] says is correct, I think that there is a more important point he should also be making when replying to this claim.
This is my first comment on LessWrong.
I just wrote a post replying to part of Bostrom’s talk, but apparently I need 20 Karma points to post it, so… let it be a long comment instead:
Bostrom should modify his standard reply to the common “We’d just shut off / contain the AI” claim
In Superintelligence author Prof. Nick Bostrom’s most recent TED Talk, What happens when our computers get smarter than we are?, he spends over two minutes replying to the common claim that we could just shut off an AI or preemptively contain it in a box in order to prevent it from doing bad things that we don’t like, so there’s no need to be too concerned about the possible future development of AI that has misconceived or poorly specified goals:
If I recall correctly, Bostrom has replied to this claim in this manner in several of the talks he has given. While what he says is correct, I think that there is a more important point he should also be making when replying to this claim.
The point is that even if containing an AI in a box so that it could not escape and cause damage was somehow feasible, it would still be incredibly important for us to determine how to create AI that shares our interests and values (friendly AI). And we would still have great reason to be concerned about the creation of unfriendly AI. This is because other people, such as terrorists, could still create an unfriendly AI and intentionally release it into the world to wreak havoc and potentially cause an existential catastrophe.
The idea that we should not be too worried about figuring out how to make AI friendly because we could always contain the AI in a box until we knew it was safe to release is confused not primarily because we couldn’t actually successfully contain it in the box, but rather because the primary reason we have for wanting to quickly figure out how to make a friendly AI is so that we can make a friendly AI before anyone else makes an unfriendly AI.
In his TED Talk, Bostrom continues:
Bostrom could have strengthened his argument for the position that there is no way around this difficult problem by stating my point above.
That is, he could have pointed out that even if we somehow developed a reliable way to keep a superintelligent genie locked up in its bottle forever, this still would not allow us to avoid having to solve the difficult problem of creating friendly AI with human values, since there would still be a high risk that other people in the world with not-so-good intentions would eventually develop an unfriendly AI and intentionally release it upon the world, or simply not exercise the caution necessary to keep it contained.
Once the technology to make superintelligent AI is developed, good people will be pressured to create friendly AI and let it take control of the future of the world ASAP. The longer they wait, the greater the risk that not-so-good people will develop AI that isn’t specifically designed to have human values. This is why solving the value alignment problem soon is so important.