Even more importantly, the biggest reason our world is stable is that humans have a very narrow range of capabilities, and this importantly applies to intelligence, which is normally distributed, meaning that societies can usually defeat outlier humans. AI capabilities will not nearly be this constrained, and the variance is worrying because there a real chance that one AI will be far more intelligent than any human that has ever lived, and it’s relatively easy to cross the human range, ala Go and Starcraft. It’s a similar reason why superpowers in the real world would doom us by default.
EDIT: I no longer think superpowers would doom us by default.
Even more importantly, the biggest reason our world is stable is that humans have a very narrow range of capabilities, and this importantly applies to intelligence, which is normally distributed, meaning that societies can usually defeat outlier humans. AI capabilities will not nearly be this constrained, and the variance is worrying because there a real chance that one AI will be far more intelligent than any human that has ever lived, and it’s relatively easy to cross the human range, ala Go and Starcraft. It’s a similar reason why superpowers in the real world would doom us by default.
EDIT: I no longer think superpowers would doom us by default.