Suppose AGI happens in 2035 or 2045. Will takeoff be faster, or slower, than if it happens in 2027?
Intuition for slower: In the models of takeoff that I’ve seen, longer timelines is correlated with slower takeoff. Because they share a common cause: the inherent difficulty of training AGI. Or to put it more precisely, there’s all these capability milestones we are interested in, such as superhuman coders, full AI R&D automation, AGI, ASI, etc. and there’s this underlying question of how much compute, data, tinkering, etc. will be needed to get from milestone 1 to 2 to 3 to 4 etc., and these things are probably all correlated (at least in our current epistemic state). Moreover, in the 2030′s the rate of growth of inputs such as data, compute, etc. will have slowed, so all else equal the pace of takeoff should be slower.
Intuition for faster: That was all about correlation. Causally, it seems clear that longer timelines cause faster takeoff. Because there’s more compute lying around, more data available, more of everything. If you have (for example) just reached the full automation of AI R&D, and you are trying to do the next big paradigm shift that’ll take you to ASI, you’ll have orders of magnitude more compute and data to experiment with (and your automated AI researchers be both more numerous and serially faster!) if it’s 2035 instead of 2027. “So what?” the reply goes. “Correlation is what matters for predicting how fast takeoff will be in 2035 or 2045. Yes you’ll have + 3 OOMs more resources with which to do the research, but (in expectation) the research will require (let’s say) +6 OOMs more resources.” But I’m not fully satisfied with this reply. Apparent counterexample: Consider the paradigm of brainlike AGI, in which the tech tree is (1) Figure out how the human brain works, (2) Use those principles to build an AI that has similar properties, i.e. similar data-efficient online learning blah blah blah, and (3) train that AI in some simulation environment similar to a human childhood, and (4) then iterate from there to improve things further. If I condition on this paradigm shift happening, and then further imagine that it happens in either 2027, 2035, or 2045… it really does seem like all four steps of this tech tree should go faster in the later years vs. in 2027. Especially steps 3 and 4 should go much faster due to the copious amounts of compute available in those later years. The amount of compute needed to simulate the human brain isn’t dependent on whether we figure out (1) and (2) in 2027 or 2035 or 2045… right?
I’m currently confused about how to think about this & what the ultimate answer should be, so I’m posting here in case the internet can enlighten me.
If there is an undiscovered architecture / learning algorithm that is multiple orders of magnitude more data-efficient than transformers, then as far as I can tell, the entire R&D process of superintelligence could go like this:
Someone has a fundamental insight
They run a small experiment and it works
They run a larger experiment and it still works
The company does a full-size training run
And that’s it. Maybe the resulting system is missing some memory components or real-time learning or something, but then it can go and build the general superintelligence on its own over the weekend.
As far as I can tell, there is nothing preventing this from happening today, and the takeoff looks even harder in 2030 and beyond, barring a coordinated effort to prevent further AI R&D.
Am I missing something that makes this implausible?
With software-only singularity, at some point the feasible takeoff speed (as opposed to the actual takeoff speed) might stop depending on initial conditions. If there is enough compute to plot AI-built industry (that sidesteps human industry) faster than it’s being constructed in the physical world, then additional initial OOMs of human-built compute won’t be making any difference. Since humans are still so much more efficient (individually) at learning than LLMs (and a software-only singularity, whenever it happens, will bridge that gap and then some, as well as bring AI advantages to bear), we might reach that point soon, maybe by ~2030.
I think the WBE intuition is probably the more useful one, and even more so when it comes to the also important question of ‘how many powerful human-level AIs should there be around, soon after AGI’ - given e.g. estimates of computational requirements like in https://www.youtube.com/watch?v=mMqYxe5YkT4. Basically, WBEs set a bit of a lower bound ( given that they’re both a proof of existence and that, in many ways, the physical instantiations (biological brains) are there, lying in wait for better tech to access them in the right format and digitize them. Also, that better tech might be coming soon, especially as AI starts accelerating science and automating tasks more broadly—see e.g. https://www.sam-rodriques.com/post/optical-microscopy-provides-a-path-to-a-10m-mouse-brain-connectome-if-it-eliminates-proofreading.
Each ~10 year period can shift the Overton windows and establish new coordination practices/technologies. If early AGIs themselves are targeted at pursuing slowdown-if-necessary (if ASI-grade alignment is difficult, or non-alignment societal externalities need more work first), the situation in which they are created might significantly influence their practical ability to prevent fast takeoff.
Enlightening an expert is a pretty high bar, but I will give my thoughts. I am strongly in the faster camp, because of the brainlike AGI considerations as you say. Given how much more data efficient the brain is, I just don’t think the current trendlines regarding data/compute/capabilities will hold when we can fully copy and understand our brain’s architecture. I see an unavoidable significant overhang when that happens, that only gets larger the more compute and integrated robotics is deployed. The inherent difficulty of training AI is somewhat, fixed known (as a upper bound) and easier that what we currently do because we know how much data, compute, etc children take to learn.
This all makes it difficult for me to know what to want in terms of policy. Its obvious that ASI is extreme power, extreme danger, but it seems more dangerous if developed later rather than sooner. As someone who doesn’t believe the extreme FOOM/nano-magic scenario it almost makes me wish for it now. “The best time for an unaligned ASI was 20 years ago, the second best time is now!” If we consider more prosaic risks, then the amount of automation of society is a major consideration, specifically if humanoid robots can keep our existing tech stack running without humans. Even if they never turn on us, their existence still increases the risk, unless we can be 100% there is a global kill switch for all of them as soon as a hostile AI attempted such a takeover.
The intuition for slower here is just better than the intuition for faster. The intuition for faster is saying “in worlds where takeoff happens in 2045, it will happen faster than that same takeoff would have happened if implemented in 2027”. But this does not seem to be the way things work, there’s not so much low-hanging fruit that an idea that’s available and viable in 2027 won’t even be tried for two decades. I think if you concretely write down the conditionals you care about it will be not too hard to distinguish these intuitions.
Maybe? I feel like you aren’t engaging with the second half of my “intuition for faster” paragraph. You are just recapitulating the first half. I don’t think the second half depends on claiming that the brainlike AGI paradigm is available and viable in 2027; maybe it requires breakthroughs in neuroscience or brain scanners that haven’t happened yet but have a 5% chance of happening each year, for example.
I think my point is that the breakthroughs can’t have a 5% chance of happening each year, this implies low-hanging fruit abounds in whatever the relevant bit of neuroscience is. If the brainlike AGI program has a 5% chance of becoming possible this year then it is viable this year. It can’t require that many incremental ideas, otherwise the relevant breakthrough would be much less likely this year.
Maybe I am confused about your point? Or we’re thinking about different counterfactuals?
OK, suppose we are 3 breakthroughs away from the brainlike AGI program and there’s a 15% chance of a breakthrough each year. I don’t think that changes the bottom line, which is that when the brainlike AGI program finally starts working, the speed at which it passes through the capabilities milestones is greater the later it starts working.
Now that’s just one paradigm of course, but I wonder if I could make a similar argument about many of the paradigms, and then argue that conditional on 2035 or 2045 timelines, AGI will probably be achieved via one of those paradigms, and thus takeoff will be faster.
(I suppose that brings up a whole nother intuition I should have mentioned, which is that the speed of takeoff probably depends on which paradigm is the relevant paradigm during the intelligence explosion, and that might have interesting correlations with timelines...)
Intuition for ‘faster’ seems more straightforward to justify because, in general, there will be a higher volume of technology available as time progresses, while brain emulation requirements are constant. I think it’s interesting to focus on what could cause a slower scenario.
It’s possible that, without complete digitization of a sufficiently complex animal brain, further progress will be intractable for human intelligence and, by extension, very advanced future LLMs. For example, there may be many supposed breakthroughs in implementing continual learning in neural networks or high sample efficiency learning, but for some reason it will not be possible to glue all those things together, and the critical insights to do so will seem very hard or almost impossible to invent.
It may be the case that scaling AGI like intelligence is like trying to increase velocity in a fluid. It’s more complex than just a quadratic increase in drag. The type of flow changes above supersonic speeds, and there will be multiple supersonic like transitions that are incredibly complex to understand.
The radical difference in human intellectual capability from a supposedly relatively identical substrate seems contradictory at first. It’s possible that, for some evolutionary reason, the ability to form complex circuits is stunted in non-anomalous brains.
Digitization of a sufficiently complex brain at enough resolution may not be possible without nanotechnology that can monitor a living brain in real time across its entire volume. There might be approaches to grow flat brains with scaled up features synthetically, but it may not be possible to train such a brain for reasons that are hard to understand or even speculate about now. Nanotechnology at this level may not be possible to develop without ASI.
It’s possible that the brain requires much more compute than a naive estimate based on synaptic firing frequency suggests.
I have often heard that what a single neuron does is extremely complex. On the other hand, the frequency of synaptic firing suggests there isn’t much data transmitted in total. This is relatively hard for me: on one hand, Hans Moravec style estimates: computing capacity in the retina multiplied by brain volume make sense; on the other hand, outside the retina, at a whole brain level, some sort of data augmentation may be happening that actually consumes 99.9% of the compute, and in those processes very complex in neuron operations are used.
It may not be possible to design and manufacture high volume, brain like programmable substrates without ASI, and ASI may not be achievable without them, or it may be extremely hard and require multiple terawatts of compute because of 2.
The current LLM takeoff suggests that intelligence is relatively “simple” to solve, but in fact this type of text based pattern processing could be OOMs more efficient than animal like intelligence due to the pattern compressing function of language. If bootstrapping LLMs to AGI fails, it could be really hard to find a paradigm that gets closer. A new paradigm may get closer but still turn out to be relatively limited. This situation could repeat many times without obvious solutions on the horizon.
For now, I think that’s enough. I need to think about this more.
Suppose AGI happens in 2035 or 2045. Will takeoff be faster, or slower, than if it happens in 2027?
Intuition for slower: In the models of takeoff that I’ve seen, longer timelines is correlated with slower takeoff. Because they share a common cause: the inherent difficulty of training AGI. Or to put it more precisely, there’s all these capability milestones we are interested in, such as superhuman coders, full AI R&D automation, AGI, ASI, etc. and there’s this underlying question of how much compute, data, tinkering, etc. will be needed to get from milestone 1 to 2 to 3 to 4 etc., and these things are probably all correlated (at least in our current epistemic state). Moreover, in the 2030′s the rate of growth of inputs such as data, compute, etc. will have slowed, so all else equal the pace of takeoff should be slower.
Intuition for faster: That was all about correlation. Causally, it seems clear that longer timelines cause faster takeoff. Because there’s more compute lying around, more data available, more of everything. If you have (for example) just reached the full automation of AI R&D, and you are trying to do the next big paradigm shift that’ll take you to ASI, you’ll have orders of magnitude more compute and data to experiment with (and your automated AI researchers be both more numerous and serially faster!) if it’s 2035 instead of 2027. “So what?” the reply goes. “Correlation is what matters for predicting how fast takeoff will be in 2035 or 2045. Yes you’ll have + 3 OOMs more resources with which to do the research, but (in expectation) the research will require (let’s say) +6 OOMs more resources.” But I’m not fully satisfied with this reply. Apparent counterexample: Consider the paradigm of brainlike AGI, in which the tech tree is (1) Figure out how the human brain works, (2) Use those principles to build an AI that has similar properties, i.e. similar data-efficient online learning blah blah blah, and (3) train that AI in some simulation environment similar to a human childhood, and (4) then iterate from there to improve things further. If I condition on this paradigm shift happening, and then further imagine that it happens in either 2027, 2035, or 2045… it really does seem like all four steps of this tech tree should go faster in the later years vs. in 2027. Especially steps 3 and 4 should go much faster due to the copious amounts of compute available in those later years. The amount of compute needed to simulate the human brain isn’t dependent on whether we figure out (1) and (2) in 2027 or 2035 or 2045… right?
I’m currently confused about how to think about this & what the ultimate answer should be, so I’m posting here in case the internet can enlighten me.
If there is an undiscovered architecture / learning algorithm that is multiple orders of magnitude more data-efficient than transformers, then as far as I can tell, the entire R&D process of superintelligence could go like this:
Someone has a fundamental insight
They run a small experiment and it works
They run a larger experiment and it still works
The company does a full-size training run
And that’s it. Maybe the resulting system is missing some memory components or real-time learning or something, but then it can go and build the general superintelligence on its own over the weekend.
As far as I can tell, there is nothing preventing this from happening today, and the takeoff looks even harder in 2030 and beyond, barring a coordinated effort to prevent further AI R&D.
Am I missing something that makes this implausible?
With software-only singularity, at some point the feasible takeoff speed (as opposed to the actual takeoff speed) might stop depending on initial conditions. If there is enough compute to plot AI-built industry (that sidesteps human industry) faster than it’s being constructed in the physical world, then additional initial OOMs of human-built compute won’t be making any difference. Since humans are still so much more efficient (individually) at learning than LLMs (and a software-only singularity, whenever it happens, will bridge that gap and then some, as well as bring AI advantages to bear), we might reach that point soon, maybe by ~2030.
I think the WBE intuition is probably the more useful one, and even more so when it comes to the also important question of ‘how many powerful human-level AIs should there be around, soon after AGI’ - given e.g. estimates of computational requirements like in https://www.youtube.com/watch?v=mMqYxe5YkT4. Basically, WBEs set a bit of a lower bound ( given that they’re both a proof of existence and that, in many ways, the physical instantiations (biological brains) are there, lying in wait for better tech to access them in the right format and digitize them. Also, that better tech might be coming soon, especially as AI starts accelerating science and automating tasks more broadly—see e.g. https://www.sam-rodriques.com/post/optical-microscopy-provides-a-path-to-a-10m-mouse-brain-connectome-if-it-eliminates-proofreading.
Each ~10 year period can shift the Overton windows and establish new coordination practices/technologies. If early AGIs themselves are targeted at pursuing slowdown-if-necessary (if ASI-grade alignment is difficult, or non-alignment societal externalities need more work first), the situation in which they are created might significantly influence their practical ability to prevent fast takeoff.
Enlightening an expert is a pretty high bar, but I will give my thoughts. I am strongly in the faster camp, because of the brainlike AGI considerations as you say. Given how much more data efficient the brain is, I just don’t think the current trendlines regarding data/compute/capabilities will hold when we can fully copy and understand our brain’s architecture. I see an unavoidable significant overhang when that happens, that only gets larger the more compute and integrated robotics is deployed. The inherent difficulty of training AI is somewhat, fixed known (as a upper bound) and easier that what we currently do because we know how much data, compute, etc children take to learn.
This all makes it difficult for me to know what to want in terms of policy. Its obvious that ASI is extreme power, extreme danger, but it seems more dangerous if developed later rather than sooner. As someone who doesn’t believe the extreme FOOM/nano-magic scenario it almost makes me wish for it now.
“The best time for an unaligned ASI was 20 years ago, the second best time is now!”
If we consider more prosaic risks, then the amount of automation of society is a major consideration, specifically if humanoid robots can keep our existing tech stack running without humans. Even if they never turn on us, their existence still increases the risk, unless we can be 100% there is a global kill switch for all of them as soon as a hostile AI attempted such a takeover.
The intuition for slower here is just better than the intuition for faster. The intuition for faster is saying “in worlds where takeoff happens in 2045, it will happen faster than that same takeoff would have happened if implemented in 2027”. But this does not seem to be the way things work, there’s not so much low-hanging fruit that an idea that’s available and viable in 2027 won’t even be tried for two decades. I think if you concretely write down the conditionals you care about it will be not too hard to distinguish these intuitions.
Maybe? I feel like you aren’t engaging with the second half of my “intuition for faster” paragraph. You are just recapitulating the first half. I don’t think the second half depends on claiming that the brainlike AGI paradigm is available and viable in 2027; maybe it requires breakthroughs in neuroscience or brain scanners that haven’t happened yet but have a 5% chance of happening each year, for example.
I think my point is that the breakthroughs can’t have a 5% chance of happening each year, this implies low-hanging fruit abounds in whatever the relevant bit of neuroscience is. If the brainlike AGI program has a 5% chance of becoming possible this year then it is viable this year. It can’t require that many incremental ideas, otherwise the relevant breakthrough would be much less likely this year.
Maybe I am confused about your point? Or we’re thinking about different counterfactuals?
OK, suppose we are 3 breakthroughs away from the brainlike AGI program and there’s a 15% chance of a breakthrough each year. I don’t think that changes the bottom line, which is that when the brainlike AGI program finally starts working, the speed at which it passes through the capabilities milestones is greater the later it starts working.
Now that’s just one paradigm of course, but I wonder if I could make a similar argument about many of the paradigms, and then argue that conditional on 2035 or 2045 timelines, AGI will probably be achieved via one of those paradigms, and thus takeoff will be faster.
(I suppose that brings up a whole nother intuition I should have mentioned, which is that the speed of takeoff probably depends on which paradigm is the relevant paradigm during the intelligence explosion, and that might have interesting correlations with timelines...)
Intuition for ‘faster’ seems more straightforward to justify because, in general, there will be a higher volume of technology available as time progresses, while brain emulation requirements are constant. I think it’s interesting to focus on what could cause a slower scenario.
It’s possible that, without complete digitization of a sufficiently complex animal brain, further progress will be intractable for human intelligence and, by extension, very advanced future LLMs. For example, there may be many supposed breakthroughs in implementing continual learning in neural networks or high sample efficiency learning, but for some reason it will not be possible to glue all those things together, and the critical insights to do so will seem very hard or almost impossible to invent.
It may be the case that scaling AGI like intelligence is like trying to increase velocity in a fluid. It’s more complex than just a quadratic increase in drag. The type of flow changes above supersonic speeds, and there will be multiple supersonic like transitions that are incredibly complex to understand.
The radical difference in human intellectual capability from a supposedly relatively identical substrate seems contradictory at first. It’s possible that, for some evolutionary reason, the ability to form complex circuits is stunted in non-anomalous brains.
Digitization of a sufficiently complex brain at enough resolution may not be possible without nanotechnology that can monitor a living brain in real time across its entire volume. There might be approaches to grow flat brains with scaled up features synthetically, but it may not be possible to train such a brain for reasons that are hard to understand or even speculate about now. Nanotechnology at this level may not be possible to develop without ASI.
It’s possible that the brain requires much more compute than a naive estimate based on synaptic firing frequency suggests.
I have often heard that what a single neuron does is extremely complex. On the other hand, the frequency of synaptic firing suggests there isn’t much data transmitted in total. This is relatively hard for me: on one hand, Hans Moravec style estimates: computing capacity in the retina multiplied by brain volume make sense; on the other hand, outside the retina, at a whole brain level, some sort of data augmentation may be happening that actually consumes 99.9% of the compute, and in those processes very complex in neuron operations are used.
It may not be possible to design and manufacture high volume, brain like programmable substrates without ASI, and ASI may not be achievable without them, or it may be extremely hard and require multiple terawatts of compute because of 2.
The current LLM takeoff suggests that intelligence is relatively “simple” to solve, but in fact this type of text based pattern processing could be OOMs more efficient than animal like intelligence due to the pattern compressing function of language. If bootstrapping LLMs to AGI fails, it could be really hard to find a paradigm that gets closer. A new paradigm may get closer but still turn out to be relatively limited. This situation could repeat many times without obvious solutions on the horizon.
For now, I think that’s enough. I need to think about this more.