What causes the very non-linear relationship you are assuming?
Advantage of offence over defense in high-capability regime—you need only cross one threshold like “can finish a plan to rowhammer itself to internet” or “can hide its thoughts before it is spotted”. And we will build non-narrow AI because in practice “most powerful AIs we can control” means “we built some AIs, we can control them, so we continue to do what we have done before” and not “we try to understand what we will not be able to control in the future and try not to do it” because we already don’t check whether our current AI will be general before we turn it on and we already explicitly trying to create non-narrow AI.
Advantage of offence over defense in high-capability regime—you need only cross one threshold like “can finish a plan to rowhammer itself to internet” or “can hide its thoughts before it is spotted”. And we will build non-narrow AI because in practice “most powerful AIs we can control” means “we built some AIs, we can control them, so we continue to do what we have done before” and not “we try to understand what we will not be able to control in the future and try not to do it” because we already don’t check whether our current AI will be general before we turn it on and we already explicitly trying to create non-narrow AI.