I am pleased to hear your viewpoint is so close to mine on this. Closer than most other people I’ve spoken with, although I have found a few others who are thinking along similar lines. There are a couple key differences in my estimates that change my priorities a bit.
Background: I started reading and thinking about AGI risk about 15 years ago. I then had much longer timelines for AGI (80-100 years) so I decided to study human intelligence enhancement. I studied neuroscience, with a focus on developing BCI and genetic intelligence enhancement, but a few years into my PhD work my AGI timelines got shorter and I also realized how many roadblocks the academic community was placing in the way of biomedical research focused on “enhancement-above-normal” rather than “curing a disease to return towards normal.” So I started thinking that substantial human intelligence enhancement was more like 50 − 75 years off, less like the 15-30 years off I had initially hoped.
So I left my neuroscience PhD and studied machine learning. Then my AGI timelines shortened from 20-30 years to more like 10- 20 years. I left mainstream ML engineering to study AI safety full time. I spent several months investigating the remaining roadblocks to AGI and the progress that various labs were making on these roadblocks, and my timelines shortened from 10-20 years to 1-5 years.
So, while I agree that we need a delay in loss-of-control-over-AGI in order to have time for serial research years into AGI alignment (including building more brain-like and interpretable ML models like Stephen Byrnes and Conjecture, and Astera:Obelisk have endorsed) and human intelligence enhancement, I think we need more of a delay than you postulate as your median estimate, more like 20 years of delay. And that we may need this delay to come into force sooner than your median estimate of 6 years. Additionally, I currently believe that the delay-period is easier to expand than the intelligence-enhancement-research time is to shrink, per unit of additional effort.
So my focus is currently on what seems the most urgent critical task: buying humanity time. Can we find social and technological solutions to extending the period of time before we lose control?
Can we build specialized narrow AI systems to help detect and shutdown nascent unauthorized AGIs? Can we develop methodology for safely studying AGI in confinement (sandboxing techniques like: improved network security and model training best practices, norms against openly sharing code or model weights, deliberate capability limitations of models in test environments, preventing self-modification or unauthorized message passing, more thorough safety evaluations, etc.)
Though my strategic emphasis differs, my conclusion is the same: The only way forward is through!
I have been shocked by the lack of effort put into social technology to lengthen timelines. As I see it one of the only chances we have is increasing the number of people (specifically normies, as that is the group with true scale) who understand and care about the risk arguments, but almost nobody seems to be trying to achieve this. Am I missing something?
No, I think outreach is a neglected cause area because AGI seemed implausible 10 or even 5 years ago. I think it is now much easier for people to imagine us getting there, and thus the time is ripe.
I think there’s a delay in outreach for three reasons.
There’s substantial conflict within the community about the effect of doing that outreach. Trying to sound the alarm might just convince the whole world that AGI is imminent, and the first one there controls the world. That would accelerate progress dramatically. For some reason, normies do not seem to understand this. But the compelling logic would convince many if there were efforts to get everyone to think about it. This is why I’ve kept my mouth largely closed, and probably why many others have as well.
We as a community strongly believe it won’t work. We assume that the coordination problems are too large. But we don’t think about it a ton, for multiple reasons including 1 and 3 here. There are strong arguments that we should at least think about it more.
The types of people who tend to take abstract arguments, like AGI risk, seriously are typically not the types of people who want to take on massive social projects. There are many exceptions, like Rob Miles, but I think the averages make a difference in our approach as a community.
I do think the community is moving toward focusing more on this angle. And that we probably should.
I am pleased to hear your viewpoint is so close to mine on this. Closer than most other people I’ve spoken with, although I have found a few others who are thinking along similar lines. There are a couple key differences in my estimates that change my priorities a bit.
Background: I started reading and thinking about AGI risk about 15 years ago. I then had much longer timelines for AGI (80-100 years) so I decided to study human intelligence enhancement. I studied neuroscience, with a focus on developing BCI and genetic intelligence enhancement, but a few years into my PhD work my AGI timelines got shorter and I also realized how many roadblocks the academic community was placing in the way of biomedical research focused on “enhancement-above-normal” rather than “curing a disease to return towards normal.” So I started thinking that substantial human intelligence enhancement was more like 50 − 75 years off, less like the 15-30 years off I had initially hoped.
So I left my neuroscience PhD and studied machine learning. Then my AGI timelines shortened from 20-30 years to more like 10- 20 years. I left mainstream ML engineering to study AI safety full time. I spent several months investigating the remaining roadblocks to AGI and the progress that various labs were making on these roadblocks, and my timelines shortened from 10-20 years to 1-5 years.
So, while I agree that we need a delay in loss-of-control-over-AGI in order to have time for serial research years into AGI alignment (including building more brain-like and interpretable ML models like Stephen Byrnes and Conjecture, and Astera:Obelisk have endorsed) and human intelligence enhancement, I think we need more of a delay than you postulate as your median estimate, more like 20 years of delay. And that we may need this delay to come into force sooner than your median estimate of 6 years. Additionally, I currently believe that the delay-period is easier to expand than the intelligence-enhancement-research time is to shrink, per unit of additional effort.
So my focus is currently on what seems the most urgent critical task: buying humanity time. Can we find social and technological solutions to extending the period of time before we lose control?
Can we build specialized narrow AI systems to help detect and shutdown nascent unauthorized AGIs? Can we develop methodology for safely studying AGI in confinement (sandboxing techniques like: improved network security and model training best practices, norms against openly sharing code or model weights, deliberate capability limitations of models in test environments, preventing self-modification or unauthorized message passing, more thorough safety evaluations, etc.)
Though my strategic emphasis differs, my conclusion is the same: The only way forward is through!
I have been shocked by the lack of effort put into social technology to lengthen timelines. As I see it one of the only chances we have is increasing the number of people (specifically normies, as that is the group with true scale) who understand and care about the risk arguments, but almost nobody seems to be trying to achieve this. Am I missing something?
No, I think outreach is a neglected cause area because AGI seemed implausible 10 or even 5 years ago. I think it is now much easier for people to imagine us getting there, and thus the time is ripe.
For the people disagreeing, I’m curious what part you’re disagreeing with.
I think there’s a delay in outreach for three reasons.
There’s substantial conflict within the community about the effect of doing that outreach. Trying to sound the alarm might just convince the whole world that AGI is imminent, and the first one there controls the world. That would accelerate progress dramatically. For some reason, normies do not seem to understand this. But the compelling logic would convince many if there were efforts to get everyone to think about it. This is why I’ve kept my mouth largely closed, and probably why many others have as well.
We as a community strongly believe it won’t work. We assume that the coordination problems are too large. But we don’t think about it a ton, for multiple reasons including 1 and 3 here. There are strong arguments that we should at least think about it more.
The types of people who tend to take abstract arguments, like AGI risk, seriously are typically not the types of people who want to take on massive social projects. There are many exceptions, like Rob Miles, but I think the averages make a difference in our approach as a community.
I do think the community is moving toward focusing more on this angle. And that we probably should.