In addition to what James said, I’m reminded of the mechanism to change screen resolution in Windows XP: It automatically resets to its original resolution in X seconds, in case you can’t see the screen. This is so people can’t break their computers in one moment of weakness.
A similar thing could be done with self-modification. Self-destruction would still be possible, of course, just as it is now (I could go jump off of a bridge). But just as suicide is something that is built up to in humans, failsafes could be put in place so self-modification was equally deliberate.
Eliezer,
In my experience, smart people have many original theories. They likely hold these theories because they know they are smarter than most people, and so don’t see any reason to trust common knowledge. Also, holding original and complex theories make them seem more intelligent. Most original theories are of course incorrect, even when they come from smart people. Intelligent, charismatic people are very good at convincing themselves and others they are correct.
IMO, this is one of the main reasons those, smart, competent people in charge screw up so often. They don’t do it because they aren’t smart or competent, they do it because they have a bias in favor of their own ideas and theories, just like everyone else.