Every time someone writes “superintelligent AI,” I think they should consider writing “superpowerful AI.”
I can imagine slightly superintelligent AI that is not sci-fi, take over the world, sysop scenario, nation-state bargaining table powerful. Even if it’s superintelligent in all the charismatic and persuasive ways, not just nerdy and mathematical.
What people usually mean is AI capable of achieving significant control over Earth. Something to be reckoned with, that can gradually disempower you, lock in the light cone, etc.
Knock-your-balls-off superintelligent AI would become superpowerful AI, but you can just say superpowerful AI and save yourself the extra assertion needed and adjectives!
(Plus, most people think of intelligence in the normal, non-LessWrong sense anyway, where intelligence doesn’t immediately imply power. No need to swim against the ordinary course of language.)
An AI that’s superpowerful because it’s superintelligent is a very different beast from an AI that’s superpowerful for other reasons, so it’s useful to maintain the distinction.
AIs can be superpowerful the way governments are, or by having operational control over a lot of the industry (leading to gradual disempowerment of humanity). Many powerful human institutions are not necessarily particularly intelligent compared to the rest of the world, but they hold power, and AGIs that don’t become superintelligent for many years for technical reasons might well get there.
So I think a non-superintelligent takeover (or just a peaceful economic victory, at least initially) is a serious possibility distinct from the issues with superintelligence. The latter can happen at any point in this process (once the technical reasons that cause AGIs to remain non-superintelligent no longer hold), disrupting the gradualist story. But the gradualist story is still the more predictable baseline that’s decently likely to persist for some time, so it’s worth forecasting.
This is an important distinction for me because I expect broad intelligence/capabilities to correlate with goodness and wisdom but I believe narrow superpowers can potentially also give an agent overwhelming advantage and power. AIs with relatively balanced and broad capability profiles including wisdom etc. are still potentially dangerous to humans but the really crappy futures with close to zero cosmic value I expect to be occupied by myopic AIs that are not necessarily even overwhelmingly superhuman in all important domains.
Every time someone writes “superintelligent AI,” I think they should consider writing “superpowerful AI.”
I can imagine slightly superintelligent AI that is not sci-fi, take over the world, sysop scenario, nation-state bargaining table powerful. Even if it’s superintelligent in all the charismatic and persuasive ways, not just nerdy and mathematical.
What people usually mean is AI capable of achieving significant control over Earth. Something to be reckoned with, that can gradually disempower you, lock in the light cone, etc.
Knock-your-balls-off superintelligent AI would become superpowerful AI, but you can just say superpowerful AI and save yourself the extra assertion needed and adjectives!
(Plus, most people think of intelligence in the normal, non-LessWrong sense anyway, where intelligence doesn’t immediately imply power. No need to swim against the ordinary course of language.)
An AI that’s superpowerful because it’s superintelligent is a very different beast from an AI that’s superpowerful for other reasons, so it’s useful to maintain the distinction.
Hmm, other reasons like what? Human-level intelligence, plus an larger initial stockpile of weapons/economy/territory?
AIs can be superpowerful the way governments are, or by having operational control over a lot of the industry (leading to gradual disempowerment of humanity). Many powerful human institutions are not necessarily particularly intelligent compared to the rest of the world, but they hold power, and AGIs that don’t become superintelligent for many years for technical reasons might well get there.
So I think a non-superintelligent takeover (or just a peaceful economic victory, at least initially) is a serious possibility distinct from the issues with superintelligence. The latter can happen at any point in this process (once the technical reasons that cause AGIs to remain non-superintelligent no longer hold), disrupting the gradualist story. But the gradualist story is still the more predictable baseline that’s decently likely to persist for some time, so it’s worth forecasting.
This is an important distinction for me because I expect broad intelligence/capabilities to correlate with goodness and wisdom but I believe narrow superpowers can potentially also give an agent overwhelming advantage and power. AIs with relatively balanced and broad capability profiles including wisdom etc. are still potentially dangerous to humans but the really crappy futures with close to zero cosmic value I expect to be occupied by myopic AIs that are not necessarily even overwhelmingly superhuman in all important domains.