I am largely convinced that p(doom) is exceedingly high if there is an intelligence explosion, but I’m somewhat unconvinced about the likelihood of ASI sometime soon.
Reading this, the most salient line of thinking to me is the following:
If we assume that ASI is possible at all, how many innovations as significant as transformers do we need to get there? Eliezer guesses ‘0 to 2’, which seems reasonable. I have minimal basis to make any other estimate.
And it DOES seem reasonable to me to think that those transformer-level innovations are reasonably likely, given the massive amount of investment of time, effort, and resources into those problems. But this is (again) an entirely intuitive take.
So the p(next critical innovations to ASI) seems to be the most important issue here, and I would like to see more thoughts on that from those with more expertise. I guess they are absent because the question is simply too speculative?
Maybe you’re already familiar, but this kind of forecasting is usually done by talking about the effects of innovation, and then assuming that some innovation is likely to happen as a result of the trend. This is a technique pretty common in economics. It has obvious failure modes (that is, it assumes naturality/inevitability of some extrapolation from available data, treating contingnet processes the same way you might an asteroid’s trajectory or other natural, evolving quantity), but these appear to be the best (or at least most popular) tools we have for now for thinking about this kind of thing.
The appendices of AI2027 are really good for this, and the METR Time Horizons paper is an example of recent/influential work in this area.
Again, this isn’t awesome for analyzing discontinuities, and you need to dig into the methodology a bit to see how they’re handled in each case (some discontinuities will be calculated as part of the broader trend, meaning the forecast takes into account future paradigm-shifting advances; more bearish predictions won’t do this, and will discount or ignore steppy gains in the data).
I think there’s only a few dozen people in the world who are ~expert here, and most people only look at their work on the surface level, but it’s very rewarding to dig more deeply into the documentation associated with projects like these two!
I am largely convinced that p(doom) is exceedingly high if there is an intelligence explosion, but I’m somewhat unconvinced about the likelihood of ASI sometime soon.
Reading this, the most salient line of thinking to me is the following:
If we assume that ASI is possible at all, how many innovations as significant as transformers do we need to get there? Eliezer guesses ‘0 to 2’, which seems reasonable. I have minimal basis to make any other estimate.
And it DOES seem reasonable to me to think that those transformer-level innovations are reasonably likely, given the massive amount of investment of time, effort, and resources into those problems. But this is (again) an entirely intuitive take.
So the p(next critical innovations to ASI) seems to be the most important issue here, and I would like to see more thoughts on that from those with more expertise. I guess they are absent because the question is simply too speculative?
Maybe you’re already familiar, but this kind of forecasting is usually done by talking about the effects of innovation, and then assuming that some innovation is likely to happen as a result of the trend. This is a technique pretty common in economics. It has obvious failure modes (that is, it assumes naturality/inevitability of some extrapolation from available data, treating contingnet processes the same way you might an asteroid’s trajectory or other natural, evolving quantity), but these appear to be the best (or at least most popular) tools we have for now for thinking about this kind of thing.
The appendices of AI2027 are really good for this, and the METR Time Horizons paper is an example of recent/influential work in this area.
Again, this isn’t awesome for analyzing discontinuities, and you need to dig into the methodology a bit to see how they’re handled in each case (some discontinuities will be calculated as part of the broader trend, meaning the forecast takes into account future paradigm-shifting advances; more bearish predictions won’t do this, and will discount or ignore steppy gains in the data).
I think there’s only a few dozen people in the world who are ~expert here, and most people only look at their work on the surface level, but it’s very rewarding to dig more deeply into the documentation associated with projects like these two!