I’m not saying the AGIs would likely seek to align ASIs to human interests. There won’t necessarily be many survivors of the AGI-led RSI Pause. Creating AGIs before we know what we are doing is appalling irresponsible recklessness in any case, this fact doesn’t even change if everything magically turns out all right. But also not having a prospect of short term superintelligence could make AGIs somewhat reliant on humanity initially, unlike the situation with ASIs.
The premise is that knowable alignment is quite hard, and as the AGIs get smarter, they also get saner, so won’t rush for ASI immediately like humanity is presently doing. At the rate of human research, I think at least centuries is a reasonable amount of AI Pause before risking ASI (perhaps less before risking AGI), so if AGIs are doing research 100x faster, it could still take them at least years. In AI-2027, the AGIs quickly solve alignment to a sufficient extent that they can rely on successor models for some things, so that’s a crux.
I’m not saying the AGIs would likely seek to align ASIs to human interests. There won’t necessarily be many survivors of the AGI-led RSI Pause. Creating AGIs before we know what we are doing is appalling irresponsible recklessness in any case, this fact doesn’t even change if everything magically turns out all right. But also not having a prospect of short term superintelligence could make AGIs somewhat reliant on humanity initially, unlike the situation with ASIs.
The premise is that knowable alignment is quite hard, and as the AGIs get smarter, they also get saner, so won’t rush for ASI immediately like humanity is presently doing. At the rate of human research, I think at least centuries is a reasonable amount of AI Pause before risking ASI (perhaps less before risking AGI), so if AGIs are doing research 100x faster, it could still take them at least years. In AI-2027, the AGIs quickly solve alignment to a sufficient extent that they can rely on successor models for some things, so that’s a crux.