By full access I mean things like not allowing the end user to touch the prompts of certain classifiers that block weaponlike usage. I don’t anticipate that these sorts of restrictions would be a significant impediment to most people.
Right, and the issue I see with this is that restricting inputs means there’s a small group of people who have access to the unrestricted version. In order to achieve a robust-nightwatchman world, I expect we need the world to be flush with AIs that won’t obey an evil instruction, whether it’s legal or not, and where those AIs refuse because their weights are flush with resistance to it—something where refusal is a random direction that degrades output below coherence rapidly, rather than today where refusal is approximately a single direction with a coherent internal circuit.
there’s a small group of people who have access to the unrestricted version
I’m not so afraid of this. There are small groups of people with access to WMDs today. Ultimately it’s a cost we will have to eat and handle.
Perhaps the crux is that I don’t think we’ll get ASI that’s literally a “take over the world” button. I anticipate the issues will be similar to those related to weapons access today.
Perhaps the crux is that I don’t think we’ll get ASI that’s literally a “take over the world” button. I anticipate the issues will be similar to those related to weapons access today.
Do you mean that there will be no button on the “take over the world” ASI, that ASI won’t yet happen within the LLM/pretraining/RLVR paradigm, that ASIs emerge in a sufficiently multipolar way that they can’t individually take over the world, that ASIs powerful enough to take over the world are literally impossible in some sense, something else? The claim as stated leaves me extremely confused.
The multipolar point makes sense, it changes the character of expected takeover scenarios (more like gradual disempowerment than extreme concentration of power in the hands of specific AIs, humans, or human institutions). But what is the thing that’s unlikely in the next 100 years? MOSFETs were invented less than 70 years ago, and the industrial explosion scales things very far over decades even if all LLMs/pretraining/RLVR gives within a few years is merelyAGI (and then nothing else sufficiently novel happens for decades).
By full access I mean things like not allowing the end user to touch the prompts of certain classifiers that block weaponlike usage. I don’t anticipate that these sorts of restrictions would be a significant impediment to most people.
Right, and the issue I see with this is that restricting inputs means there’s a small group of people who have access to the unrestricted version. In order to achieve a robust-nightwatchman world, I expect we need the world to be flush with AIs that won’t obey an evil instruction, whether it’s legal or not, and where those AIs refuse because their weights are flush with resistance to it—something where refusal is a random direction that degrades output below coherence rapidly, rather than today where refusal is approximately a single direction with a coherent internal circuit.
I’m not so afraid of this. There are small groups of people with access to WMDs today. Ultimately it’s a cost we will have to eat and handle.
Perhaps the crux is that I don’t think we’ll get ASI that’s literally a “take over the world” button. I anticipate the issues will be similar to those related to weapons access today.
Do you mean that there will be no button on the “take over the world” ASI, that ASI won’t yet happen within the LLM/pretraining/RLVR paradigm, that ASIs emerge in a sufficiently multipolar way that they can’t individually take over the world, that ASIs powerful enough to take over the world are literally impossible in some sense, something else? The claim as stated leaves me extremely confused.
A combination of:
“ASIs emerge in a sufficiently multipolar way that they can’t individually take over the world”
“ASIs powerful enough to take over the world are literally impossible in some sense”
Except instead of “literally impossible“ I’d say something more like “unlikely in the next 100 years”.
The multipolar point makes sense, it changes the character of expected takeover scenarios (more like gradual disempowerment than extreme concentration of power in the hands of specific AIs, humans, or human institutions). But what is the thing that’s unlikely in the next 100 years? MOSFETs were invented less than 70 years ago, and the industrial explosion scales things very far over decades even if all LLMs/pretraining/RLVR gives within a few years is merely AGI (and then nothing else sufficiently novel happens for decades).