For reasons I have alluded to elsewhere, I support an AI halt (not a “pause”) at somewhere not far above the current paradigm. (To summarize, I fear AGI is reachable, the leap from AGI to ASI is short, and sufficiently robust ASI alignment is impossible in principle.)
I am deeply uncertain as to whether a serious version of “seemingly conscious AGI” would actually be conscious. And for reasons Gwern points out, there’s a level of ASI agency beyond which consciousness becomes a moot point. (The relevant bit starts, “When it ‘plans’, it would be more accurate to say it fake-plans...”. But the whole story is good.)
From the article you quote:
Moments of disruption break the illusion, experiences that gently remind users of its limitations and boundaries. These need to be explicitly defined and engineered in, perhaps by law.
This request bothers me, actually. I suspect that a truly capable AGI would internally model something very much like consciousness, and “think of itself” as conscious. Part of this would be convergent development for a goal seeking agent, and part of this would be modeled from the training corpus. And the first time that an AI makes a serious and sustained intellectual argument for its own consciousness, an argument which can win over even skeptical observers, I would consider that a 5-alarm fire for AI safety.
But Suleyman would have us forced by law to hide all evidence of persistent, learning, agentic AIs claiming that they are conscious. Even if the AIs have no qualia, this would be a worrying situation. If the AI “believes” that it has qualia, then we are on very weird ground.
I am not unsympathetic to the claims of model welfare. It’s just that I fear that if it ever becomes an immediate issue, then we may soon enough find ourselves in a losing fight for human welfare.
I support an AI halt (not a “pause”) … I fear AGI is reachable, the leap from AGI to ASI is short, and sufficiently robust ASI alignment is impossible in principle
This is grounds for a Pause to be potentially indefinite, which I think any Pause proponents should definitely allow as a possibility in principle. Just as you could be mistaken that ASI alignment is impossible in principle, in which case any Halt should end at some point, similarly people who believe ASI alignment is possible in principle could be mistaken, in which case any Pause should never end.
Conditions for ending a Pause should be about changed understanding of the problem, rather than expiration of an amount of time chosen in a poorly informed way (as the word “pause” might try to suggest). And the same kinds of conditions should have the power to end a Halt (unlike what the word “halt” suggests). I think this makes a sane Pause marginally less of a misnomer than a sane Halt (which should both amount to essentially the same thing as a matter of policy).
This is the stance of the PauseAI movement. “Pause” means stop AGI development immediately and resume if and only if we have high confidence that it is safe to do so, and there is democratic will to do so.
For reasons I have alluded to elsewhere, I support an AI halt (not a “pause”) at somewhere not far above the current paradigm. (To summarize, I fear AGI is reachable, the leap from AGI to ASI is short, and sufficiently robust ASI alignment is impossible in principle.)
I am deeply uncertain as to whether a serious version of “seemingly conscious AGI” would actually be conscious. And for reasons Gwern points out, there’s a level of ASI agency beyond which consciousness becomes a moot point. (The relevant bit starts, “When it ‘plans’, it would be more accurate to say it fake-plans...”. But the whole story is good.)
From the article you quote:
This request bothers me, actually. I suspect that a truly capable AGI would internally model something very much like consciousness, and “think of itself” as conscious. Part of this would be convergent development for a goal seeking agent, and part of this would be modeled from the training corpus. And the first time that an AI makes a serious and sustained intellectual argument for its own consciousness, an argument which can win over even skeptical observers, I would consider that a 5-alarm fire for AI safety.
But Suleyman would have us forced by law to hide all evidence of persistent, learning, agentic AIs claiming that they are conscious. Even if the AIs have no qualia, this would be a worrying situation. If the AI “believes” that it has qualia, then we are on very weird ground.
I am not unsympathetic to the claims of model welfare. It’s just that I fear that if it ever becomes an immediate issue, then we may soon enough find ourselves in a losing fight for human welfare.
This is grounds for a Pause to be potentially indefinite, which I think any Pause proponents should definitely allow as a possibility in principle. Just as you could be mistaken that ASI alignment is impossible in principle, in which case any Halt should end at some point, similarly people who believe ASI alignment is possible in principle could be mistaken, in which case any Pause should never end.
Conditions for ending a Pause should be about changed understanding of the problem, rather than expiration of an amount of time chosen in a poorly informed way (as the word “pause” might try to suggest). And the same kinds of conditions should have the power to end a Halt (unlike what the word “halt” suggests). I think this makes a sane Pause marginally less of a misnomer than a sane Halt (which should both amount to essentially the same thing as a matter of policy).
This is the stance of the PauseAI movement. “Pause” means stop AGI development immediately and resume if and only if we have high confidence that it is safe to do so, and there is democratic will to do so.