Second, the major disagreement is between those who think progress will be discontinuous and sudden (such as Eliezer Yudkowsky, MIRI) and those who think progress will be very fast by normal historical standards but continuous (Paul Chrisiano, Robin Hanson).
I’m not actually convinced this is a fair summary of the disagreement. As I explained in my post about different AI takeoffs, I had the impression that the primary disagreement between the two groups was over locality rather than the amount of time takeoff lasts. Though of course, I may be misinterpreting people.
They do disagree about locality, yes, but as far as I can tell that is downstream of the assumption that there won’t be a very abrupt switch to a new growth mode. A single project pulling suddenly ahead of the rest of the world would happen if the growth curve is such that with a realistic amount (a few months) of lead time you can get ahead of everyone else.
So the obvious difference in predictions is that e.g. Paul/Robin think that takeoff will occur across many systems in the world while MIRI thinks it will occur in a single system. That is because MIRI thinks that RSI is much more of an all-or-nothing capability than the others, which in turn is because they think AGI is much more likely to depend on a few novel, key insights that produce sudden gains in capability. That was the conclusion of my post.
In the past I’ve called Locality a practical discontinuity—from the outside world’s perspective, does a single project explode out of nowhere? Whether you get a practical discontinuity doesn’t just depend on whether progress is discontinuous. If you get a discontinuity to RSI capability then you do get a practical discontinuity, but that is a sufficient, not necessary condition. If the growth curve is steep enough you might get a practical discontinuity anyway.
Perhaps Eliezer-2008 believed that there would be a discontinuity in returns on optimization leading to a practical discontinuity/local explosion but Eliezer-2020 (since de-emphasising RSI) just thinks we will get a local explosion somehow, either from a discontinuity or sufficiently fast continuous progress.
My graphs above do seem to support that view—even most of the ‘continuous’ scenarios seem to have a fairly abrupt and steep growth curve. I strongly suspect that as well as disagreements about discontinuities, there are very strong disagreements about ‘post-RSI speed’ - maybe over orders of magnitude.
This is what the curves look like if s is set to 0.1 - the takeoff is much slower even if RSI comes about fairly abruptly.
After reading your summary of the difference (maybe just a difference in emphasis) between ‘Paul slow’ vs ‘continuous’ takeoff, I did some further simulations. A low setting of d (highly continuous progress) doesn’t give you a paul slow condition on its own, but it is relatively easy to replicate a situation like this:
There will be a complete 4 year interval in which world output doubles, before the first 1 year interval in which world output doubles. (Similarly, we’ll see an 8 year doubling before a 2 year doubling, etc.)
What we want is a scenario where you don’t get intermediate doubling intervals at all in the discontinuous case, but you get at least one in the continuous case. Setting s relatively high appears to do the trick.
Here is a scenario where we have very fast post-RSI growth with s=5,c=1,I0=1 and I_AGI=3. I wrote some more code to produce plots of how long each complete interval of doubling took in each scenario. The ‘default’ rate with no contribution from RSI was 0.7. All the continuous scenarios had two complete doubling intervals over intermediate time frames before the doubling time collapsed to under 0.05 on the third doubling. The discontinuous model simply kept the original doubling interval until it collapsed to under 0.05 on the third doubling interval. It’s all in this graph.
Let’s make the irresponsible assumption that this actually applies to the real economy, with the current growth mode, non-RSI condition being given by the ‘slow/no takeoff’, s=0 condition.
The current doubling time is a bit over 23 years. In the shallow continuous progress scenario (red line), we get a 9 year doubling, a 4 year doubling and then a ~1 year doubling. In the discontinuous scenario (purple line) we get 2 23 year doublings and then a ~1 year doubling out of nowhere. In other words, this fairly random setting of the parameters (this was the second set I tried) gives us a Paul slow takeoff if you make the assumption that all of this should be scaled to years of economic doubling. You can see that graph here.
I’m not actually convinced this is a fair summary of the disagreement. As I explained in my post about different AI takeoffs, I had the impression that the primary disagreement between the two groups was over locality rather than the amount of time takeoff lasts. Though of course, I may be misinterpreting people.
They do disagree about locality, yes, but as far as I can tell that is downstream of the assumption that there won’t be a very abrupt switch to a new growth mode. A single project pulling suddenly ahead of the rest of the world would happen if the growth curve is such that with a realistic amount (a few months) of lead time you can get ahead of everyone else.
So the obvious difference in predictions is that e.g. Paul/Robin think that takeoff will occur across many systems in the world while MIRI thinks it will occur in a single system. That is because MIRI thinks that RSI is much more of an all-or-nothing capability than the others, which in turn is because they think AGI is much more likely to depend on a few novel, key insights that produce sudden gains in capability. That was the conclusion of my post.
In the past I’ve called Locality a practical discontinuity—from the outside world’s perspective, does a single project explode out of nowhere? Whether you get a practical discontinuity doesn’t just depend on whether progress is discontinuous. If you get a discontinuity to RSI capability then you do get a practical discontinuity, but that is a sufficient, not necessary condition. If the growth curve is steep enough you might get a practical discontinuity anyway.
Perhaps Eliezer-2008 believed that there would be a discontinuity in returns on optimization leading to a practical discontinuity/local explosion but Eliezer-2020 (since de-emphasising RSI) just thinks we will get a local explosion somehow, either from a discontinuity or sufficiently fast continuous progress.
My graphs above do seem to support that view—even most of the ‘continuous’ scenarios seem to have a fairly abrupt and steep growth curve. I strongly suspect that as well as disagreements about discontinuities, there are very strong disagreements about ‘post-RSI speed’ - maybe over orders of magnitude.
This is what the curves look like if s is set to 0.1 - the takeoff is much slower even if RSI comes about fairly abruptly.
After reading your summary of the difference (maybe just a difference in emphasis) between ‘Paul slow’ vs ‘continuous’ takeoff, I did some further simulations. A low setting of d (highly continuous progress) doesn’t give you a paul slow condition on its own, but it is relatively easy to replicate a situation like this:
What we want is a scenario where you don’t get intermediate doubling intervals at all in the discontinuous case, but you get at least one in the continuous case. Setting s relatively high appears to do the trick.
Here is a scenario where we have very fast post-RSI growth with s=5,c=1,I0=1 and I_AGI=3. I wrote some more code to produce plots of how long each complete interval of doubling took in each scenario. The ‘default’ rate with no contribution from RSI was 0.7. All the continuous scenarios had two complete doubling intervals over intermediate time frames before the doubling time collapsed to under 0.05 on the third doubling. The discontinuous model simply kept the original doubling interval until it collapsed to under 0.05 on the third doubling interval. It’s all in this graph.
Let’s make the irresponsible assumption that this actually applies to the real economy, with the current growth mode, non-RSI condition being given by the ‘slow/no takeoff’, s=0 condition.
The current doubling time is a bit over 23 years. In the shallow continuous progress scenario (red line), we get a 9 year doubling, a 4 year doubling and then a ~1 year doubling. In the discontinuous scenario (purple line) we get 2 23 year doublings and then a ~1 year doubling out of nowhere. In other words, this fairly random setting of the parameters (this was the second set I tried) gives us a Paul slow takeoff if you make the assumption that all of this should be scaled to years of economic doubling. You can see that graph here.