Nick that’s an interesting thought, but it is possible that someone does know how to build a friendly AI, but may not know that he knows this. What I mean is that a person may be working on something unrelated to AI or is not interested in AI, but knows all the ingredients and technical things needed to make a friendly AI. Maybe he just doesn’t know the implications of his own knowledge.
I think it’s fair to say many scientists fall into this category, if only due to specialization. Maybe some inorganic chemist knows the cure to cancer. But simply due to his ignorance/uninterest in biochemistry or organic chemistry or whatever subject that’s relevant is unaware of the implications of his own specialized knowledge on topics outside his area of expertise.
“Destroying the world” describes things fairly accurately. Who believes a nuclear strike on the major Soviet Union and American cities wouldn’t have destroyed the world? Life, I’m sure wouldn’t have ended (even human), but the fall out and human loss would have been catastrophic. It’s easy to question this 20-30 years after the fact, but I remember the fear the world festered in in the 70′s and 80′s. If a nuclear missile had been launched, dozens if not hundreds would have been launched.
The change in world view and behavior would have been larger than any event in human history. Millions, possibly over a billion, lives would have been lost, and the prosperity of the world that started in the 70′s, which has lifted over a billion out of abject poverty, would have come to screeching halt.