A. Many AI safety people don’t support relatively responsible companies unilaterally pausing, which PauseAI advocates. (Many do support governments slowing AI progress, or preparing to do so at a critical point in the future. And many of those don’t see that as tractable for them to work on.)
B. “Pausing AI” is indeed more popular than PauseAI, but it’s not clearly possible to make a more popular version of PauseAI that actually does anything; any such organization will have strategy/priorities/asks/comms that alienate many of the people who think “yeah I support pausing AI.”
C.
There does not seem to be a legible path to prevent possible existential risks from AI without slowing down its current progress.
This seems confused. Obviously P(doom | no slowdown) < 1. Many people’s work reduces risk in both slowdown and no-slowdown worlds, and it seems pretty clear to me that most of them shouldn’t switch to working on increasing P(slowdown).
B. “Pausing AI” is indeed more popular than PauseAI, but it’s not clearly possible to make a more popular version of PauseAI that actually does anything; any such organization will have strategy/priorities/asks/comms that alienate many of the people who think “yeah I support pausing AI.”
This strikes me as a very strange claim. You’re essentially saying, even if a general policy is widely supported, it’s practically impossible to implement any specific version of that policy? Why would that be true?
For example I think a better alternative to “nobody fund PauseAI, and nobody make an alternative version they like better” would be “there are 10+ orgs all trying to pause AI and they all have somewhat different goals but they’re all generally pushing in the direction of pausing AI”. I think in the latter scenario you are reasonably likely to get some decent policies put into place even if they’re not my favorite.
You’re essentially saying, even if a general policy is widely supported, it’s practically impossible to implement any specific version of that policy? Why would that be true?
Banning nuclear weapons is exactly like this. If it could be done universally and effectively, it would be great, but any specific version seems likely to tilt the balance of power without accomplishing the goal.
I think a better alternative to “nobody fund PauseAI, and nobody make an alternative version they like better” would be “there are 10+ orgs all trying to pause AI and they all have somewhat different goals but they’re all generally pushing in the direction of pausing AI”. I think in the latter scenario you are reasonably likely to get some decent policies put into place even if they’re not my favorite.
That’s kind-of what happened with the anti-nuclear movement, but it ended up doing lots of harm because the things that could be stopped were the good ones!
That’s kind-of what happened with the anti-nuclear movement, but it ended up doing lots of harm because the things that could be stopped were the good ones!
The global stockpile of nuclear weapons is down 6x since its peak in 1986. Hard to attribute causality but if the anti-nuclear movement played a part in that, then I’d say it was net positive.
(My guess is it’s more attributable to the collapse of the Soviet Union than to anything else, but the anti-nuclear movement probably still played some nonzero role)
I’m sure it played some nonzero role, but is it anything like enough of an impact, and enough of a role to compensate for all the marginal harms of global warming because of stopping deployment of nuclear power (which they are definitely largely responsible for)?
You think it’s obviously materially less? Because there is a faction, including Eliezer and many others, that think it’s epsilon, and claim that the reduction in risk from any technical work is less than the acceleration it causes. (I think you’re probably right about some of that work, but I think it’s not at all obviously true!)
This is not obvious. My P(doom|no slowdown) is like 0.95-0.97, the difference from 1 being essentially “maybe I am crazy or am missing something vital when making the following argument”.
Instrumental convergence suggests that the vast majority of possible AGI will be hostile. No slowdown means that neural-net ASI will be instantiated. To get ~doom from this, you need some way to solve the problem of “what does this code do when run” with extreme accuracy in order to only instantiate non-hostile neural-net ASI (you need “extreme” accuracy because you’re up against the rare disease problem a.k.a. false positive paradox; true positives are extremely rare, so a positive alignment result from a 99%-accurate test is still almost certainly a false positive). Unfortunately, the “what does this code do when run” problem has a name, the “halting problem”, and it’s literally the first problem in computer science ever proven to be unsolvable in the general case.
And, sure, the general case being unsolvable doesn’t mean that the case you care about is unsolvable. GOFAI has a good argument for being a special case, because human-written source code is quite useful to understanding a program. Neural nets… don’t. At least, they don’t in the case we care about; “I am smarter than the neural net” is also a plausible special case, but that’s obviously no help with neural-net ASI.
My P(doom) is a lot lower than 0.95, but that’s because I think slowdown is fairly likely, due to warning shots/nuclear war/maybe direct political success (key result from the middle one: if you want to stop AI, it is helpful to ensure you’ll survive a nuclear war in order to help lock it down then). But my stance on aligning neural nets? “It is impossible to solve the true puzzle from inside this [field], because the key piece is not here.” Blind alley. Abort.
A: Yeah. I’m mostly positive about their goal to work towards “building the Pause button”. I think protesting against “relatively responsible companies” makes a lot of sense when these companies seem to use their lobbying power more against AI-Safety-aligned Governance than in favor of it. You’re obviously very aware of the details here.
B: I asked my question because I’m frustrated with that. Is there a way for AI Safety to coordinate a better reaction?
C:
There does not seem to be a legible path to prevent possible existential risks from AI without slowing down its current progress.
I phrased that a bit sharply, but I find your reply very useful:
Obviously P(doom | no slowdown) < 1. Many people’s work reduces risk in both slowdown and no-slowdown worlds, and it seems pretty clear to me that most of them shouldn’t switch to working on increasing P(slowdown).[1]
These are quite strong claims! I’ll take that as somewhat representative of the community. My attempt at paraphrasing: It’s not (strictly?) necessary to slow down AI to prevent doom. There is a lot of useful AI Safety work going on that is not focused on slowing/pausing AI. This work is useful even if AGI is coming soon.
A. Many AI safety people don’t support relatively responsible companies unilaterally pausing, which PauseAI advocates. (Many do support governments slowing AI progress, or preparing to do so at a critical point in the future. And many of those don’t see that as tractable for them to work on.)
B. “Pausing AI” is indeed more popular than PauseAI, but it’s not clearly possible to make a more popular version of PauseAI that actually does anything; any such organization will have strategy/priorities/asks/comms that alienate many of the people who think “yeah I support pausing AI.”
C.
This seems confused. Obviously P(doom | no slowdown) < 1. Many people’s work reduces risk in both slowdown and no-slowdown worlds, and it seems pretty clear to me that most of them shouldn’t switch to working on increasing P(slowdown).
This strikes me as a very strange claim. You’re essentially saying, even if a general policy is widely supported, it’s practically impossible to implement any specific version of that policy? Why would that be true?
For example I think a better alternative to “nobody fund PauseAI, and nobody make an alternative version they like better” would be “there are 10+ orgs all trying to pause AI and they all have somewhat different goals but they’re all generally pushing in the direction of pausing AI”. I think in the latter scenario you are reasonably likely to get some decent policies put into place even if they’re not my favorite.
Banning nuclear weapons is exactly like this. If it could be done universally and effectively, it would be great, but any specific version seems likely to tilt the balance of power without accomplishing the goal.
That’s kind-of what happened with the anti-nuclear movement, but it ended up doing lots of harm because the things that could be stopped were the good ones!
The global stockpile of nuclear weapons is down 6x since its peak in 1986. Hard to attribute causality but if the anti-nuclear movement played a part in that, then I’d say it was net positive.
(My guess is it’s more attributable to the collapse of the Soviet Union than to anything else, but the anti-nuclear movement probably still played some nonzero role)
I’m sure it played some nonzero role, but is it anything like enough of an impact, and enough of a role to compensate for all the marginal harms of global warming because of stopping deployment of nuclear power (which they are definitely largely responsible for)?
You think it’s obviously materially less? Because there is a faction, including Eliezer and many others, that think it’s epsilon, and claim that the reduction in risk from any technical work is less than the acceleration it causes. (I think you’re probably right about some of that work, but I think it’s not at all obviously true!)
This is not obvious. My P(doom|no slowdown) is like 0.95-0.97, the difference from 1 being essentially “maybe I am crazy or am missing something vital when making the following argument”.
Instrumental convergence suggests that the vast majority of possible AGI will be hostile. No slowdown means that neural-net ASI will be instantiated. To get ~doom from this, you need some way to solve the problem of “what does this code do when run” with extreme accuracy in order to only instantiate non-hostile neural-net ASI (you need “extreme” accuracy because you’re up against the rare disease problem a.k.a. false positive paradox; true positives are extremely rare, so a positive alignment result from a 99%-accurate test is still almost certainly a false positive). Unfortunately, the “what does this code do when run” problem has a name, the “halting problem”, and it’s literally the first problem in computer science ever proven to be unsolvable in the general case.
And, sure, the general case being unsolvable doesn’t mean that the case you care about is unsolvable. GOFAI has a good argument for being a special case, because human-written source code is quite useful to understanding a program. Neural nets… don’t. At least, they don’t in the case we care about; “I am smarter than the neural net” is also a plausible special case, but that’s obviously no help with neural-net ASI.
My P(doom) is a lot lower than 0.95, but that’s because I think slowdown is fairly likely, due to warning shots/nuclear war/maybe direct political success (key result from the middle one: if you want to stop AI, it is helpful to ensure you’ll survive a nuclear war in order to help lock it down then). But my stance on aligning neural nets? “It is impossible to solve the true puzzle from inside this [field], because the key piece is not here.” Blind alley. Abort.
Thank you for responding!
A: Yeah. I’m mostly positive about their goal to work towards “building the Pause button”. I think protesting against “relatively responsible companies” makes a lot of sense when these companies seem to use their lobbying power more against AI-Safety-aligned Governance than in favor of it. You’re obviously very aware of the details here.
B: I asked my question because I’m frustrated with that. Is there a way for AI Safety to coordinate a better reaction?
C:
I phrased that a bit sharply, but I find your reply very useful:
These are quite strong claims! I’ll take that as somewhat representative of the community. My attempt at paraphrasing: It’s not (strictly?) necessary to slow down AI to prevent doom. There is a lot of useful AI Safety work going on that is not focused on slowing/pausing AI. This work is useful even if AGI is coming soon.
Saying “PauseAI good” does not take a lot of an AI Safety researcher’s time.