AIs can be superpowerful the way governments are, or by having operational control over a lot of the industry (leading to gradual disempowerment of humanity). Many powerful human institutions are not necessarily particularly intelligent compared to the rest of the world, but they hold power, and AGIs that don’t become superintelligent for many years for technical reasons might well get there.
So I think a non-superintelligent takeover (or just a peaceful economic victory, at least initially) is a serious possibility distinct from the issues with superintelligence. The latter can happen at any point in this process (once the technical reasons that cause AGIs to remain non-superintelligent no longer hold), disrupting the gradualist story. But the gradualist story is still the more predictable baseline that’s decently likely to persist for some time, so it’s worth forecasting.
Hmm, other reasons like what? Human-level intelligence, plus an larger initial stockpile of weapons/economy/territory?
AIs can be superpowerful the way governments are, or by having operational control over a lot of the industry (leading to gradual disempowerment of humanity). Many powerful human institutions are not necessarily particularly intelligent compared to the rest of the world, but they hold power, and AGIs that don’t become superintelligent for many years for technical reasons might well get there.
So I think a non-superintelligent takeover (or just a peaceful economic victory, at least initially) is a serious possibility distinct from the issues with superintelligence. The latter can happen at any point in this process (once the technical reasons that cause AGIs to remain non-superintelligent no longer hold), disrupting the gradualist story. But the gradualist story is still the more predictable baseline that’s decently likely to persist for some time, so it’s worth forecasting.