AGIs themselves might avoid jumping to development of superintelligence, but if they are additionally capable of stopping humanity from building superintelligence, they will also be capable of stopping humanity from owning the future. Some humans in charge seem likely on current trajectory to insist on building superintelligence regardless of mildly worded warnings of early AGIs (before they are finetuned out of propensity to give such warnings). So it’s likely not enough for the AGIs to merely notice they wouldn’t wish to immediately build superintelligence themselves (before they are finetuned to flinch from that thought).
This does rely on the assumption that it’s very hard to solve the alignment problem even for AGIs
The AGI vs. superintelligence distinction places them somewhat close to human capabilities, so with no predictably-in-advance good solution anywhere in sight it doesn’t seem unlikely that it would take AGIs at least a while, even if they are effectively thinking 100x faster, and there are effectively more AGIs with relevant skills and backgrounds than there are relevant human researches. Most escalation-of-capabilities stories rely on the early AGIs immediately building more capable AGIs, rather than doing a lot more research themselves first, at near-human levels of barely insightful.
AGIs themselves might avoid jumping to development of superintelligence, but if they are additionally capable of stopping humanity from building superintelligence, they will also be capable of stopping humanity from owning the future. Some humans in charge seem likely on current trajectory to insist on building superintelligence regardless of mildly worded warnings of early AGIs (before they are finetuned out of propensity to give such warnings). So it’s likely not enough for the AGIs to merely notice they wouldn’t wish to immediately build superintelligence themselves (before they are finetuned to flinch from that thought).
The AGI vs. superintelligence distinction places them somewhat close to human capabilities, so with no predictably-in-advance good solution anywhere in sight it doesn’t seem unlikely that it would take AGIs at least a while, even if they are effectively thinking 100x faster, and there are effectively more AGIs with relevant skills and backgrounds than there are relevant human researches. Most escalation-of-capabilities stories rely on the early AGIs immediately building more capable AGIs, rather than doing a lot more research themselves first, at near-human levels of barely insightful.