Here’s my answer.
I’m pretty uncertain compared to some of the others!
First, I’m assuming that by AGI we mean an agent-like entity that can do the things associated with general intelligence, including things like planning towards a goal and carrying that out. If we end up in a CAIS-like world where there is some AI service or other that can do most economically useful tasks, but nothing with very broad competence, I count that as never developing AGI.
I’ve been impressed with GPT-3, and could imagine it or something like it scaling to produce near-human level responses to language prompts in a few years, especially with RL-based extensions.
But, following the list (below) of missing capabilities by Stuart Russell, I still think things like long-term planning would elude GPT-N, so it wouldn’t be agentive general intelligence. Even though you might get those behaviours with trivial extensions of GPT-N, I don’t think it’s very likely.
That’s why I think AGI before 2025 is very unlikely (not enough time for anything except scaling up of existing methods). This is also because I tend to expect progress to be continuous, though potentially quite fast, and going from current AI to AGI in less than 5 years requires a very sharp discontinuity.
AGI before 2035 or so happens if systems quite a lot like current deep learning can do the job, but which aren’t just trivial extensions of them—this seems reasonable to me on the inside view—e.g. it takes us less than 15 years to take GPT-N and add layers on top of it that handle things like planning and discovering new actions. This is probably my ‘inside view’ answer.
I put a lot of weight on a tail peaking around 2050 because of how quickly we’ve advanced up this ‘list of breakthroughs needed for general intelligence’ -
There is this list of remaining capabilities needed for AGI in an older post I wrote, with the capabilities of ‘GPT-6’ as I see them underlined:
Stuart Russell’s List
human-like language comprehension
cumulative learning
discovering new action sets
managing its own mental activity
For reference, I’ve included two capabilities we already have that I imagine being on a similar list in 1960
So we’d have discovering new action sets, and managing mental activity—effectively, the things that facilitate long-range complex planning, remaining.
So (very oversimplified) if around the 1980s we had efficient search algorithms, by 2015 we had image recognition (basic perception) and by 2025 we have language comprehension courtesy of GPT-8, that leaves cumulative learning (which could be obtained by advanced RL?), then discovering new action sets and managing mental activity (no idea). It feels a bit odd that we’d breeze past all the remaining milestones in one decade after it took ~6 to get to where we are now. Say progress has sped up to be twice as fast, then it’s 3 more decades to go. Add to this the economic evidence from things like Modelling the Human Trajectory, which suggests a roughly similar time period of around 2050.
Finally, I think it’s unlikely but not impossible that we never build AGI and instead go for tool AI or CAIS, most likely because we’ve misunderstood the incentives such that it isn’t actually economical or agentive behaviour doesn’t arise easily. Then there’s the small (few percent) chance of catastrophic or existential disaster which wrecks our ability to invent things. This is the one I’m most unsure about—I put 15% for both but it may well be higher.
This is also because I tend to expect progress to be continuous, though potentially quite fast, and going from current AI to AGI in less than 5 years requires a very sharp discontinuity.
I object! I think your argument from extrapolating when milestones have been crossed is good, but it’s just one argument among many. There are other trends which, if extrapolated, get to AGI in less than five years. For example if you extrapolate the AI-compute trend and the GPT-scaling trends you get something like “GPT-5 will appear 3 years from now and be 3 orders of magnitude bigger and will be human-level at almost all text-based tasks.” No discontinuity required.
Daniel and SDM, what do you think of a bet with 78:22 odds (roughly 4:1) based on the differences in your distributions, i.e: If AGI happens before 2030, SDM owes Daniel $78. If AGI doesn’t happen before 2030, Daniel owes SDM $22.
This was calculated by:
Identifying the earliest possible date with substantial disagreement (in this case, 2030)
Finding the probability each person assigns to the date range of now to 2030:
According to this post, a bet based on the arithmetic mean of 2 differing probability estimates yields the same expected value for each participant. In this case, the mean is (5%+39%)/2=22% chance of AGI before 2030, equivalent to 22:78 odds.
$78 and $22 can be scaled appropriately for whatever size bet you’re comfortable with
The main issue for me is that if I win this bet I either won’t be around to collect on it, or I’ll be around but have much less need for money. So for me the bet you propose is basically “61% chance I pay SDM $22 in 10 years, 39% chance I get nothing.”
Jonas Vollmer helped sponsor my other bet on this matter, to get around this problem. He agreed to give me a loan for my possible winnings up front, which I would pay back (with interest) in 2030, unless I win in which case the person I bet against would pay it. Meanwhile the person I bet against would get his winnings from me in 2030, with interest, assuming I lose. It’s still not great because from my perspective it amounts to a loan with a higher interest rate basically, so it would be better for me to just take out a long-term loan. (The chance of never having to pay it back is nice, but I only never have to pay it back in worlds where I won’t care about money anyway.) Still though it was better than nothing so I took it.
The ‘progress will be continuous’ argument, to apply to our near future, does depend on my other assumptions—mainly that the breakthroughs on that list are separable, so agentive behaviour and long-term planning won’t drop out of a larger GPT by themselves and can’t be considered part of just ‘improving up language model accuracy’.
We currently have partial progress on human-level language comprehension, a bit on cumulative learning, but near zero on managing mental activity for long term planning, so if we were to suddenly reach human level on long-term planning in the next 5 years, that would probably involve a discontinuity, which I don’t think is very likely for the reasons given here.
If language models scale to near-human performance but the other milestones don’t fall in the process, and my initial claim is right, that gives us very transformative AI but not AGI. I think that the situation would look something like this:
So there would be 2 (maybe 3?) breakthroughs remaining. It seems like you think just scaling up a GPT will also resolve those other milestones, rather than just giving us human-like language comprehension. Whereas if I’m right and also those curves do extrapolate, what we would get at the end would be an excellent text generator, but it wouldn’t be an agent, wouldn’t be capable of long-term planning and couldn’t be accurately described as having a utility function over the states of the external world, and I don’t see any reason why trivial extensions of GPT would be able to do that either since those seem like problems that are just as hard as human-like language comprehension. GPT seems like it’s also making some progress on cumulative learning, though it might need some RL-based help with that, but none at all on managing mental activity for longterm planning or discovering new action sets.
As an additional argument, admittedly from authority—Stuart Russell also clearly sees human-like language comprehension as only one of several really hard and independent problems that need to be solved.
A humanlike GPT-N would certainly be a huge leap into a realm of AI we don’t know much about, so we could be surprised and discover that agentive behaviour and having a utility function over states of the external world spontaneously appears in a good enough language model, but that argument has to be made, and you need that argument to hold and GPT to keep scaling for us to reach AGI in the next five years, and I don’t see the conjunction of those two as that likely—it seems as though your argument rests solely on whether GPT scales or not, when there’s also this other conceptual premise that’s much harder to justify.
I’m also not sure if I’ve seen anyone make the argument that GPT-N will also give us these specific breakthroughs—but if you have reasons that GPT scaling would solve all the remaining barriers to AGI, I’d be interested to hear it. Note that this isn’t the same as just pointing out how impressive the results scaling up GPT could be—Gwern’s piece here, for example, seems to be arguing for a scenario more like what I’ve envisaged, where GPT-N ends up a key piece of some future AGI but just provides some of the background ‘world model’:
Models like GPT-3 suggest that large unsupervised models will be vital components of future DL systems, as they can be ‘plugged into’ systems to immediately provide understanding of the world, humans, natural language, and reasoning.
If GPT does scale, and we get human-like language comprehension in 2025, that will mean we’re moving up that list much faster, and in turn suggests that there might not be a large number of additional discoveries required to make the other breakthroughs, which in turn suggests they might also occur within the Deep Learning paradigm, and relatively soon. I think that if this happens, there’s a reasonable chance that when we do build an AGI a big part of its internals looks like a GPT, as gwern suggested, but by then we’re already long past simply scaling up existing systems.
Alternatively, perhaps you’re not including agentive behaviour in your definition of AGI—a par-human text generator for most tasks that isn’t capable of discovering new action sets or managing its mental activity is, I think a ‘mere’ transformative AI and not a genuine AGI.
That small tail at the end feels really suspicious. I.e., it implies that if we haven’t reached AGI by 2080, then we probably won’t reach it at all. I feel like this might be an artifact of specifying a small number of bins on elicit, though.
Here’s my answer. I’m pretty uncertain compared to some of the others!
First, I’m assuming that by AGI we mean an agent-like entity that can do the things associated with general intelligence, including things like planning towards a goal and carrying that out. If we end up in a CAIS-like world where there is some AI service or other that can do most economically useful tasks, but nothing with very broad competence, I count that as never developing AGI.
I’ve been impressed with GPT-3, and could imagine it or something like it scaling to produce near-human level responses to language prompts in a few years, especially with RL-based extensions.
But, following the list (below) of missing capabilities by Stuart Russell, I still think things like long-term planning would elude GPT-N, so it wouldn’t be agentive general intelligence. Even though you might get those behaviours with trivial extensions of GPT-N, I don’t think it’s very likely.
That’s why I think AGI before 2025 is very unlikely (not enough time for anything except scaling up of existing methods). This is also because I tend to expect progress to be continuous, though potentially quite fast, and going from current AI to AGI in less than 5 years requires a very sharp discontinuity.
AGI before 2035 or so happens if systems quite a lot like current deep learning can do the job, but which aren’t just trivial extensions of them—this seems reasonable to me on the inside view—e.g. it takes us less than 15 years to take GPT-N and add layers on top of it that handle things like planning and discovering new actions. This is probably my ‘inside view’ answer.
I put a lot of weight on a tail peaking around 2050 because of how quickly we’ve advanced up this ‘list of breakthroughs needed for general intelligence’ -
So (very oversimplified) if around the 1980s we had efficient search algorithms, by 2015 we had image recognition (basic perception) and by 2025 we have language comprehension courtesy of GPT-8, that leaves cumulative learning (which could be obtained by advanced RL?), then discovering new action sets and managing mental activity (no idea). It feels a bit odd that we’d breeze past all the remaining milestones in one decade after it took ~6 to get to where we are now. Say progress has sped up to be twice as fast, then it’s 3 more decades to go. Add to this the economic evidence from things like Modelling the Human Trajectory, which suggests a roughly similar time period of around 2050.
Finally, I think it’s unlikely but not impossible that we never build AGI and instead go for tool AI or CAIS, most likely because we’ve misunderstood the incentives such that it isn’t actually economical or agentive behaviour doesn’t arise easily. Then there’s the small (few percent) chance of catastrophic or existential disaster which wrecks our ability to invent things. This is the one I’m most unsure about—I put 15% for both but it may well be higher.
I object! I think your argument from extrapolating when milestones have been crossed is good, but it’s just one argument among many. There are other trends which, if extrapolated, get to AGI in less than five years. For example if you extrapolate the AI-compute trend and the GPT-scaling trends you get something like “GPT-5 will appear 3 years from now and be 3 orders of magnitude bigger and will be human-level at almost all text-based tasks.” No discontinuity required.
Daniel and SDM, what do you think of a bet with 78:22 odds (roughly 4:1) based on the differences in your distributions, i.e: If AGI happens before 2030, SDM owes Daniel $78. If AGI doesn’t happen before 2030, Daniel owes SDM $22.
This was calculated by:
Identifying the earliest possible date with substantial disagreement (in this case, 2030)
Finding the probability each person assigns to the date range of now to 2030:
Daniel: 39%
SDM: 5%
Finding a fair bet
According to this post, a bet based on the arithmetic mean of 2 differing probability estimates yields the same expected value for each participant. In this case, the mean is (5%+39%)/2=22% chance of AGI before 2030, equivalent to 22:78 odds.
$78 and $22 can be scaled appropriately for whatever size bet you’re comfortable with
The main issue for me is that if I win this bet I either won’t be around to collect on it, or I’ll be around but have much less need for money. So for me the bet you propose is basically “61% chance I pay SDM $22 in 10 years, 39% chance I get nothing.”
Jonas Vollmer helped sponsor my other bet on this matter, to get around this problem. He agreed to give me a loan for my possible winnings up front, which I would pay back (with interest) in 2030, unless I win in which case the person I bet against would pay it. Meanwhile the person I bet against would get his winnings from me in 2030, with interest, assuming I lose. It’s still not great because from my perspective it amounts to a loan with a higher interest rate basically, so it would be better for me to just take out a long-term loan. (The chance of never having to pay it back is nice, but I only never have to pay it back in worlds where I won’t care about money anyway.) Still though it was better than nothing so I took it.
I’ll take that bet! If I do lose, I’ll be far too excited/terrified/dead to worry in any case.
The ‘progress will be continuous’ argument, to apply to our near future, does depend on my other assumptions—mainly that the breakthroughs on that list are separable, so agentive behaviour and long-term planning won’t drop out of a larger GPT by themselves and can’t be considered part of just ‘improving up language model accuracy’.
We currently have partial progress on human-level language comprehension, a bit on cumulative learning, but near zero on managing mental activity for long term planning, so if we were to suddenly reach human level on long-term planning in the next 5 years, that would probably involve a discontinuity, which I don’t think is very likely for the reasons given here.
If language models scale to near-human performance but the other milestones don’t fall in the process, and my initial claim is right, that gives us very transformative AI but not AGI. I think that the situation would look something like this:
If GPT-N reaches par-human:
So there would be 2 (maybe 3?) breakthroughs remaining. It seems like you think just scaling up a GPT will also resolve those other milestones, rather than just giving us human-like language comprehension. Whereas if I’m right and also those curves do extrapolate, what we would get at the end would be an excellent text generator, but it wouldn’t be an agent, wouldn’t be capable of long-term planning and couldn’t be accurately described as having a utility function over the states of the external world, and I don’t see any reason why trivial extensions of GPT would be able to do that either since those seem like problems that are just as hard as human-like language comprehension. GPT seems like it’s also making some progress on cumulative learning, though it might need some RL-based help with that, but none at all on managing mental activity for longterm planning or discovering new action sets.
As an additional argument, admittedly from authority—Stuart Russell also clearly sees human-like language comprehension as only one of several really hard and independent problems that need to be solved.
A humanlike GPT-N would certainly be a huge leap into a realm of AI we don’t know much about, so we could be surprised and discover that agentive behaviour and having a utility function over states of the external world spontaneously appears in a good enough language model, but that argument has to be made, and you need that argument to hold and GPT to keep scaling for us to reach AGI in the next five years, and I don’t see the conjunction of those two as that likely—it seems as though your argument rests solely on whether GPT scales or not, when there’s also this other conceptual premise that’s much harder to justify.
I’m also not sure if I’ve seen anyone make the argument that GPT-N will also give us these specific breakthroughs—but if you have reasons that GPT scaling would solve all the remaining barriers to AGI, I’d be interested to hear it. Note that this isn’t the same as just pointing out how impressive the results scaling up GPT could be—Gwern’s piece here, for example, seems to be arguing for a scenario more like what I’ve envisaged, where GPT-N ends up a key piece of some future AGI but just provides some of the background ‘world model’:
If GPT does scale, and we get human-like language comprehension in 2025, that will mean we’re moving up that list much faster, and in turn suggests that there might not be a large number of additional discoveries required to make the other breakthroughs, which in turn suggests they might also occur within the Deep Learning paradigm, and relatively soon. I think that if this happens, there’s a reasonable chance that when we do build an AGI a big part of its internals looks like a GPT, as gwern suggested, but by then we’re already long past simply scaling up existing systems.
Alternatively, perhaps you’re not including agentive behaviour in your definition of AGI—a par-human text generator for most tasks that isn’t capable of discovering new action sets or managing its mental activity is, I think a ‘mere’ transformative AI and not a genuine AGI.
(I can’t see your distribution in your image.)
That small tail at the end feels really suspicious. I.e., it implies that if we haven’t reached AGI by 2080, then we probably won’t reach it at all. I feel like this might be an artifact of specifying a small number of bins on elicit, though.