Perhaps the crux is that I don’t think we’ll get ASI that’s literally a “take over the world” button. I anticipate the issues will be similar to those related to weapons access today.
Do you mean that there will be no button on the “take over the world” ASI, that ASI won’t yet happen within the LLM/pretraining/RLVR paradigm, that ASIs emerge in a sufficiently multipolar way that they can’t individually take over the world, that ASIs powerful enough to take over the world are literally impossible in some sense, something else? The claim as stated leaves me extremely confused.
The multipolar point makes sense, it changes the character of expected takeover scenarios (more like gradual disempowerment than extreme concentration of power in the hands of specific AIs, humans, or human institutions). But what is the thing that’s unlikely in the next 100 years? MOSFETs were invented less than 70 years ago, and the industrial explosion scales things very far over decades even if all LLMs/pretraining/RLVR gives within a few years is merelyAGI (and then nothing else sufficiently novel happens for decades).
Do you mean that there will be no button on the “take over the world” ASI, that ASI won’t yet happen within the LLM/pretraining/RLVR paradigm, that ASIs emerge in a sufficiently multipolar way that they can’t individually take over the world, that ASIs powerful enough to take over the world are literally impossible in some sense, something else? The claim as stated leaves me extremely confused.
A combination of:
“ASIs emerge in a sufficiently multipolar way that they can’t individually take over the world”
“ASIs powerful enough to take over the world are literally impossible in some sense”
Except instead of “literally impossible“ I’d say something more like “unlikely in the next 100 years”.
The multipolar point makes sense, it changes the character of expected takeover scenarios (more like gradual disempowerment than extreme concentration of power in the hands of specific AIs, humans, or human institutions). But what is the thing that’s unlikely in the next 100 years? MOSFETs were invented less than 70 years ago, and the industrial explosion scales things very far over decades even if all LLMs/pretraining/RLVR gives within a few years is merely AGI (and then nothing else sufficiently novel happens for decades).