They do disagree about locality, yes, but as far as I can tell that is downstream of the assumption that there won’t be a very abrupt switch to a new growth mode. A single project pulling suddenly ahead of the rest of the world would happen if the growth curve is such that with a realistic amount (a few months) of lead time you can get ahead of everyone else.
So the obvious difference in predictions is that e.g. Paul/Robin think that takeoff will occur across many systems in the world while MIRI thinks it will occur in a single system. That is because MIRI thinks that RSI is much more of an all-or-nothing capability than the others, which in turn is because they think AGI is much more likely to depend on a few novel, key insights that produce sudden gains in capability. That was the conclusion of my post.
In the past I’ve called Locality a practical discontinuity—from the outside world’s perspective, does a single project explode out of nowhere? Whether you get a practical discontinuity doesn’t just depend on whether progress is discontinuous. If you get a discontinuity to RSI capability then you do get a practical discontinuity, but that is a sufficient, not necessary condition. If the growth curve is steep enough you might get a practical discontinuity anyway.
Perhaps Eliezer-2008 believed that there would be a discontinuity in returns on optimization leading to a practical discontinuity/local explosion but Eliezer-2020 (since de-emphasising RSI) just thinks we will get a local explosion somehow, either from a discontinuity or sufficiently fast continuous progress.
My graphs above do seem to support that view—even most of the ‘continuous’ scenarios seem to have a fairly abrupt and steep growth curve. I strongly suspect that as well as disagreements about discontinuities, there are very strong disagreements about ‘post-RSI speed’ - maybe over orders of magnitude.
This is what the curves look like if s is set to 0.1 - the takeoff is much slower even if RSI comes about fairly abruptly.
They do disagree about locality, yes, but as far as I can tell that is downstream of the assumption that there won’t be a very abrupt switch to a new growth mode. A single project pulling suddenly ahead of the rest of the world would happen if the growth curve is such that with a realistic amount (a few months) of lead time you can get ahead of everyone else.
So the obvious difference in predictions is that e.g. Paul/Robin think that takeoff will occur across many systems in the world while MIRI thinks it will occur in a single system. That is because MIRI thinks that RSI is much more of an all-or-nothing capability than the others, which in turn is because they think AGI is much more likely to depend on a few novel, key insights that produce sudden gains in capability. That was the conclusion of my post.
In the past I’ve called Locality a practical discontinuity—from the outside world’s perspective, does a single project explode out of nowhere? Whether you get a practical discontinuity doesn’t just depend on whether progress is discontinuous. If you get a discontinuity to RSI capability then you do get a practical discontinuity, but that is a sufficient, not necessary condition. If the growth curve is steep enough you might get a practical discontinuity anyway.
Perhaps Eliezer-2008 believed that there would be a discontinuity in returns on optimization leading to a practical discontinuity/local explosion but Eliezer-2020 (since de-emphasising RSI) just thinks we will get a local explosion somehow, either from a discontinuity or sufficiently fast continuous progress.
My graphs above do seem to support that view—even most of the ‘continuous’ scenarios seem to have a fairly abrupt and steep growth curve. I strongly suspect that as well as disagreements about discontinuities, there are very strong disagreements about ‘post-RSI speed’ - maybe over orders of magnitude.
This is what the curves look like if s is set to 0.1 - the takeoff is much slower even if RSI comes about fairly abruptly.