I think longish timelines (>= 50 years) are the default prediction. My rough mental model for AI capabilities is that they depend on three inputs:
Compute per dollar. This increases at a somewhat sub-exponential rate. The time between 10x increases is increasing. We were initially at ~10x increase every four years, but recently slowed to ~10x increase every 10-16 years (source).
Algorithmic progress in AI. Each year, the compute required to reach a given performance level drops by a constant factor, (so far, a factor of 2 every ~16 months) (source). I think improvements to training efficiency drive most of the current gains in AI capabilities, but they’ll eventually begin falling off as we exhaust low hanging fruit.
The money people are willing to invest in AI. This increases as the return on investment in AI increases. There was a time when money invested in AI rose exponentially and very fast, but it’s pretty much flattened off since GPT-3. My guess is this quantity follows a sort of stutter-stop pattern where it spikes as people realize algorithmic/hardware improvements make higher investments in AI more worthwhile, then flattens once the new investments exhaust whatever new opportunities progress in hardware/algorithms allowed.
When you combine these somewhat sub-exponentially increasing inputs with the power-law scaling laws so far discovered (see here), you probably get something roughly linear, but with occasional jumps in capability as willingness to invest jumps.
We’ve recently seen a jump, but progress has stalled since then. GPT-2 to GPT-3 was 16 months. It’s been another 16 months since GPT-3, and the closest thing to a GPT-3 successor we’ve seen is Jurassic-1, but even that’s only a marginal improvement over GPT-3.
Given the time it took us to reach our current capabilities, human level AGI is probably far off.
Possible small correction: GPT-2 to GPT-3 was 16 months, not 6. The GPT-2 paper was published in February 2019 and the GPT-3 paper was published in June 2020.
I think longish timelines (>= 50 years) are the default prediction. My rough mental model for AI capabilities is that they depend on three inputs:
Compute per dollar. This increases at a somewhat sub-exponential rate. The time between 10x increases is increasing. We were initially at ~10x increase every four years, but recently slowed to ~10x increase every 10-16 years (source).
Algorithmic progress in AI. Each year, the compute required to reach a given performance level drops by a constant factor, (so far, a factor of 2 every ~16 months) (source). I think improvements to training efficiency drive most of the current gains in AI capabilities, but they’ll eventually begin falling off as we exhaust low hanging fruit.
The money people are willing to invest in AI. This increases as the return on investment in AI increases. There was a time when money invested in AI rose exponentially and very fast, but it’s pretty much flattened off since GPT-3. My guess is this quantity follows a sort of stutter-stop pattern where it spikes as people realize algorithmic/hardware improvements make higher investments in AI more worthwhile, then flattens once the new investments exhaust whatever new opportunities progress in hardware/algorithms allowed.
When you combine these somewhat sub-exponentially increasing inputs with the power-law scaling laws so far discovered (see here), you probably get something roughly linear, but with occasional jumps in capability as willingness to invest jumps.
We’ve recently seen a jump, but progress has stalled since then. GPT-2 to GPT-3 was 16 months. It’s been another 16 months since GPT-3, and the closest thing to a GPT-3 successor we’ve seen is Jurassic-1, but even that’s only a marginal improvement over GPT-3.
Given the time it took us to reach our current capabilities, human level AGI is probably far off.
Possible small correction: GPT-2 to GPT-3 was 16 months, not 6. The GPT-2 paper was published in February 2019 and the GPT-3 paper was published in June 2020.
Whoops! Corrected.