I find the story about {lots of harmless, specialized transformer-based models being developed} to be plausible. I would not be surprised if many tech companies were to follow something like that path.
However, I also think that the conclusion—viz., it being unlikely that any AI will pose an x-risk in the next 10-20 years—is probably wrong.
The main reason I think that is something like the following:
In order for AI to pose an x-risk, it is enough that even one research lab is a bit too incautious/stupid/mistakenly-optimistic and “successfully” proceeds with developing AGI capabilities. Thus, the proposition that {AI will not pose an x-risk within N years} seems to require that
∀ResearchLab∈{all research labs that will exist over next N years}∀Techniques⊆{ML techniques available in next N yearsResearchLab can/will NOT develop dangerous AGI with Techniques
And the above is basically a large conjunction over many research labs and as-yet-unknown future ML technologies. I think it is unlikely to be true. Reasons why I think it is unlikely to be true:
It seems plausible to me that highly capable, autonomous (dangerous) AGI could be built using some appropriate combination of already existing techniques + More Compute.
Even if it weren’t/isn’t possible to build dangerous AGI with existing techniques, a lot of new techniques can be developed in 10 years.
There are many AI research labs in existence. Even if most of them were to pursue only narrow/satisfactory AI, what are the odds that not one of them pursues (dangerous) autonomous AGI?
I’m under the impression that investment in A(G)I capabilities research is increasing pretty fast; lots of smart people are moving into the field. So, the (near) future will contain even more research labs and sources of potentially dangerous new techniques.
10 years is a really long time. Like, 10 years ago it was 2012, deep learning was barely starting to be a thing, the first Q-learning-based Atari-playing model (DQN) hadn’t even been released yet, etc. A lot of progress has happened from 2012 to 2022. And the amount of progress will presumably be (much) greater in the next 10 years. I feel like I have almost no clue what the future will look like in 10-20 years.
We (or at least I) still don’t even know any convincing story of how to align autonomous AGI to “human values” (whatever those even are). (Let alone having practical, working alignment techniques.)
Given the above, I was surprised by the apparent level of confidence given to the proposition that “AI is unlikely to pose an existential risk in the next 10-20 years”. I wonder where OP disagrees with the above reasoning?
Regarding {acting quickly} vs {movement-building, recruiting people into AI safety, investing in infrastructure, etc.}: I think it’s probably obvious, but maybe bears pointing out, that when choosing strategies, one should consider not only {the probability of various timelines} but also {the expected utility of executing various strategies under various timelines}.
(For example, if timelines are very short, then I doubt even my best available {short-timelines strategy} has any real hope of working, but might still involve burning a lot of resources. Thus: it probably makes sense for me to execute {medium-to-long timelines strategy}, even if I assign high probability to short timelines? This may or may not generalize to other people working on alignment.)
I find the story about {lots of harmless, specialized transformer-based models being developed} to be plausible. I would not be surprised if many tech companies were to follow something like that path.
However, I also think that the conclusion—viz., it being unlikely that any AI will pose an x-risk in the next 10-20 years—is probably wrong.
The main reason I think that is something like the following:
In order for AI to pose an x-risk, it is enough that even one research lab is a bit too incautious/stupid/mistakenly-optimistic and “successfully” proceeds with developing AGI capabilities. Thus, the proposition that {AI will not pose an x-risk within N years} seems to require that ∀ResearchLab∈{all research labs that will exist over next N years}∀Techniques⊆{ML techniques available in next N years ResearchLab can/will NOT develop dangerous AGI with Techniques
And the above is basically a large conjunction over many research labs and as-yet-unknown future ML technologies. I think it is unlikely to be true. Reasons why I think it is unlikely to be true:
It seems plausible to me that highly capable, autonomous (dangerous) AGI could be built using some appropriate combination of already existing techniques + More Compute.
Even if it weren’t/isn’t possible to build dangerous AGI with existing techniques, a lot of new techniques can be developed in 10 years.
There are many AI research labs in existence. Even if most of them were to pursue only narrow/satisfactory AI, what are the odds that not one of them pursues (dangerous) autonomous AGI?
I’m under the impression that investment in A(G)I capabilities research is increasing pretty fast; lots of smart people are moving into the field. So, the (near) future will contain even more research labs and sources of potentially dangerous new techniques.
10 years is a really long time. Like, 10 years ago it was 2012, deep learning was barely starting to be a thing, the first Q-learning-based Atari-playing model (DQN) hadn’t even been released yet, etc. A lot of progress has happened from 2012 to 2022. And the amount of progress will presumably be (much) greater in the next 10 years. I feel like I have almost no clue what the future will look like in 10-20 years.
We (or at least I) still don’t even know any convincing story of how to align autonomous AGI to “human values” (whatever those even are). (Let alone having practical, working alignment techniques.)
Given the above, I was surprised by the apparent level of confidence given to the proposition that “AI is unlikely to pose an existential risk in the next 10-20 years”. I wonder where OP disagrees with the above reasoning?
Regarding {acting quickly} vs {movement-building, recruiting people into AI safety, investing in infrastructure, etc.}: I think it’s probably obvious, but maybe bears pointing out, that when choosing strategies, one should consider not only {the probability of various timelines} but also {the expected utility of executing various strategies under various timelines}.
(For example, if timelines are very short, then I doubt even my best available {short-timelines strategy} has any real hope of working, but might still involve burning a lot of resources. Thus: it probably makes sense for me to execute {medium-to-long timelines strategy}, even if I assign high probability to short timelines? This may or may not generalize to other people working on alignment.)