I basically think you’re being unfair here, so want to challenge you to actually name these or retract.
… So that’s my response to the charge that the param estimates are overly complicated. But I want to respond to one other point you make
It sounds like we’re talking past each other, if you think I’m making two different points. The concern I’m trying to express is that this takeoff model—by which I mean the overall model/argument/forecast presented in the paper, not just the literal code—strikes me as containing confusingly much detail/statistics/elaboration/formality, given (what seems to me like) the extreme sparsity of evidence for its component estimates.
the paper is still very clear and explicit about its limitations
I grant and (genuinely) appreciate that the paper includes many caveats. I think that helps a bunch, and indeed helps on exactly the dimension of my objection. In contrast, I think it probably anti-helped to describe the paper as forecasting “exactly how big” the intelligence explosion will be, in a sense constrained by years of research on the question.
It seems to me that demand for knowledge about how advanced AI will go, and about what we might do to make it go better, currently far outstrips supply. There are a lot of people who would like very much to have less uncertainty about takeoff dynamics, some of whom I expect might even make importantly different decisions as a result.
… and realistically, I think many of those people probably won’t spend hours carefully reading the report, as I did. And I expect the average such person is likely to greatly overestimate the amount of evidence the paper actually contains for its headline takeoff forecast.
Most obviously, from my perspective, I expect most casual readers to assume that a forecast billed as modeling “exactly how big” the intelligence explosion might be, is likely to contain evidence about the magnitude of the explosion! But I see no evidence—not even informal argument—in the paper about the limits that determine this magnitude, and unless I misunderstand your comments it seems you agree?
I think it probably anti-helped to describe the paper as forecasting “exactly how big”
That’s fair. I’ve removed the word “exactly” from the top of this paper. Edits will take a few days to process on the website version.
(I definitely didn’t intend “exactly” to connote precision. I wanted to highlight that the paper was focussed on the extent of the intelligence explosion, in contrast with our previous paper which argued that there could be accelerating progress but didn’t analyse how big the resulting SIE might be. Some reviewers were confused at the contrast. For the tweet, I think i was also subconsciously imitating the first tweet of AI-2027 which said “How, exactly, could AI take over by 2027″. But I agree the word “exactly” here could easily be misread as implying the paper gives precise results, which is an oversight.)
Fwiw, the results table gives the results to 1 sig fig and uses “~” repeatedly to avoid the impression of false precision. And the paper explicitly says “While the exact numbers here are obviously not to be trusted...”, before giving what I believe are more robust takeaways. In the discussion opens with “If this analysis is right in broad strokes” (emphasis mine).
The twitter thread also said “It goes without saying: the model is very basic and has many big limitations” and “We estimate these three parameters through a mix of empirical evidence and guesswork”. The lead tweet also said “This is my best stab at an answer”, which i thought connoted an informal, modest and tentative answer. (emphasis mine)
This is all to say that, again, i’m sympathetic with the worry that people will overestimated the precision and accuracy of the analysis and made significant efforts to avoid this. Honestly, I’d guess I made greater efforts than most similarly-speculative posts in this reference class. That said, I suspect we still disagree a lot about the amount of signal in this analysis, which is probably coming into play here.
And I expect the average such person is likely to greatly overestimate the amount of evidence the paper actually contains for its headline takeoff forecast.
I think that this paper contains comparable or greater amounts of evidence to Yudkowsky’s Intelligence Explosion Microeconomics, Bostrom’s Superintelligence, and AI-2027′s takeoff speed analysis. (In large part it contains this because I’m able to steal what I consider the most relevant insights from this previous work.) My own (biased!) opinion is that this is the best analysis we have on this question. It contains multiple angles on estimating the initial speed up from deplying ASARA, analysis of the value of r (which i think is hugely important), analysis of room to improve AI algs before matching humans, and some estimates/guesses at how much far certain improvements above human-level algs might go.
And it seems to me like you could level similar objections against previous these takeoff speeds analyses. Intelligence Explosion Microeconomics is also a paper, and it’s name “microeconomics” suggests that it’s going to do some empirically grounded mathematical modelling of the dynamics of recursive improvement. (I think it’s a great paper!) AI-2027 placed probability distributions over their parameters and ran a Monte Carlo (I believe). Superintelligence is an academic-style book. I think my work has similar or more disclaimers about how speculative it is than these other pieces. I’m not knocking these papers! I’m just claiming that I don’t think my paper is out of line of previous work in this regard.
I also think the evidential situation is here comparable (though probably more dire) to the evidence about AI timelines and AI alignment. The evidence doesn’t really constrain reality very much. But ppl still publish best-guess estimates/analyses.
Still, I do agree that it’s very easy for lay readers to overestimate how much evidence underpins the best analyses that the world has on these questions. The fact that this is even a paper puts it in a reference class of “papers”, and most papers aren’t about AGI-related stuff and are much less speculative. I felt this was at play with titotal’s critique of AI-2027. My experience reading it was “yeah these objections aren’t that surprising to me, what did expect from smg forecasting AGI and superintelligence”. And this pushes towards being even more upfront about the limitations of the analysis.
I see no evidence—not even informal argument—in the paper about the limits that determine this magnitude, and unless I misunderstand your comments it seems you agree?
No i don’t. I think the paper contains meaningful evidence about the magnitude.
First, evidence that r is quite plausibly already below 1. If so, the SIE won’t last long. This makes is plausible that we get <3 years of progress.
Second, pointing out that 10 years of progress is ~10 OOMs of efficiency gains, which is a large amount relative to efficiency gains we’ve historically seen in ML, more than the total efficiency gains we’ve ever seen in many tech areas, and comparable to those we’ve ever seen in computing hardware. Even if r > 1 when we first develop ASARA, I think there’s a good chance that it falls below 1 during the course of 10 OOMs of progress. (Especially given the possibility of compute bottlenecks kicking in as cognitive inputs are massively increased but compute increases much more slowly.)
Third, evidence about the gap from ASARA algs to human-level algs. If this gap had been much bigger or much smaller, that should update our beliefs about how long the SIE will go on for.
Fourth, somewhat-transparent quantitative estimates of some the factors that are additional alg gains above human-level. E.g. “brain is severely undertrained”, “low fraction of data is relevant”, “variation between humans”. For these estimates, a reader can see roughly where the number is coming from.
Fifth, listing additional possible gains without transparent estimates of their quantitive size. Here, as I said, the process was speaking to people with relevant expertise and asking them to eye-ball/guess at the gain. So yes, these numbers are particularly untrustworthy! But to my mind they still contain some signal. It might have been that this process didn’t uncover any significant improvements beyond the brain. In fact though, there are multiple plausibly-big improvements, which did significantly widen my personal credences on how big the IE might be. In hindsight, I should have put in more effort to making all these estimates transparent, and flagged more clearly how big this uncertainty is.
You could summarise this all as “no evidence about the limits” bc, for some of these factors, there’s no explicit argument the factor isn’t absolutely massive. So if you came in with a strong view that one factor was massive, you won’t be much moved. But that misses that, for people in many epistemic situations, the five pieces of evidence i’ve just listed here will be informative.
Fyi, your comments have convinced me to add some additional qualifiers on this point:
To the top of the paper, just before the summary, we’ll add: Like all analyses of this topic, this paper is necessarily speculative. We draw on evidence where we can, but the results are significantly influenced by guesswork and subjective judgement.
To the table cell on limits, we’ll add: This involves a fair amount of guesswork and is a massive remaining uncertainty.
To the section estimating limits, we’ll add: some of the factors listed are plausibly even bigger than our upper estimate, e.g. “must satisfy physical constraints” and “fundamental improvements”.
The concern I’m trying to express is that this takeoff model—by which I mean the overall model/argument/forecast presented in the paper, not just the literal code—strikes me as containing confusingly much detail/statistics/elaboration/formality, given (what seems to me like) the extreme sparsity of evidence for its component estimates.
I’m still not sure I’m understanding you here. If you’re sole concern is about the paper giving a misleading impression of accuracy/robustness, then I understand and am sympathetic.
But do you also think that the paper’s predictions would be better if I gave less detail?
I’m genuinely unsure if you think this. You initially claimed that the model is overly complex, which can lead to worse predictions by overfitting to noisy evidence. But you then instead claimed the parameter estimates were too complex, without giving any examples of what specific parts were misguided. What specific evidence/reasoning on which param do you think it would have been better to cut? (Not just bc it makes the paper seem overly fancy, but it makes the paper’s predictions worse.) I think we’ll need to get much more specific here to make progress.
Perhaps you think I shouldn’t have specified precise math or run a Monte Carlo? I get that concern re giving a misleading impression of robustness. But I think dropping the Monte Carlo would have made the predictions worse and the paper less useful. A precise math model makes the reasoning transparent. It makes it easier for others to build on the work. It also allows us to more accurately calculate the implications of the assumptions we make. It allows other to change the assumptions and look at how the results change. This has already been helpful for discussing the model in the LW comments for this post! I think it’s better to include the Monte Carlo for those benefits and clearly state the limitations of the analysis, than to make the analysis worse by excluding the Monte Carlo.
in a sense constrained by years of research on the question
I have spent many years researching this topic, and that did inform this paper in many ways. I don’t it’s misleading to say this.
(fyi i’ll prob duck out after this point, hope my comments have been clarifying and thx for the discuission!)
It sounds like we’re talking past each other, if you think I’m making two different points. The concern I’m trying to express is that this takeoff model—by which I mean the overall model/argument/forecast presented in the paper, not just the literal code—strikes me as containing confusingly much detail/statistics/elaboration/formality, given (what seems to me like) the extreme sparsity of evidence for its component estimates.
I grant and (genuinely) appreciate that the paper includes many caveats. I think that helps a bunch, and indeed helps on exactly the dimension of my objection. In contrast, I think it probably anti-helped to describe the paper as forecasting “exactly how big” the intelligence explosion will be, in a sense constrained by years of research on the question.
It seems to me that demand for knowledge about how advanced AI will go, and about what we might do to make it go better, currently far outstrips supply. There are a lot of people who would like very much to have less uncertainty about takeoff dynamics, some of whom I expect might even make importantly different decisions as a result.
… and realistically, I think many of those people probably won’t spend hours carefully reading the report, as I did. And I expect the average such person is likely to greatly overestimate the amount of evidence the paper actually contains for its headline takeoff forecast.
Most obviously, from my perspective, I expect most casual readers to assume that a forecast billed as modeling “exactly how big” the intelligence explosion might be, is likely to contain evidence about the magnitude of the explosion! But I see no evidence—not even informal argument—in the paper about the limits that determine this magnitude, and unless I misunderstand your comments it seems you agree?
That’s fair. I’ve removed the word “exactly” from the top of this paper. Edits will take a few days to process on the website version.
(I definitely didn’t intend “exactly” to connote precision. I wanted to highlight that the paper was focussed on the extent of the intelligence explosion, in contrast with our previous paper which argued that there could be accelerating progress but didn’t analyse how big the resulting SIE might be. Some reviewers were confused at the contrast. For the tweet, I think i was also subconsciously imitating the first tweet of AI-2027 which said “How, exactly, could AI take over by 2027″. But I agree the word “exactly” here could easily be misread as implying the paper gives precise results, which is an oversight.)
Fwiw, the results table gives the results to 1 sig fig and uses “~” repeatedly to avoid the impression of false precision. And the paper explicitly says “While the exact numbers here are obviously not to be trusted...”, before giving what I believe are more robust takeaways. In the discussion opens with “If this analysis is right in broad strokes” (emphasis mine).
The twitter thread also said “It goes without saying: the model is very basic and has many big limitations” and “We estimate these three parameters through a mix of empirical evidence and guesswork”. The lead tweet also said “This is my best stab at an answer”, which i thought connoted an informal, modest and tentative answer. (emphasis mine)
This is all to say that, again, i’m sympathetic with the worry that people will overestimated the precision and accuracy of the analysis and made significant efforts to avoid this. Honestly, I’d guess I made greater efforts than most similarly-speculative posts in this reference class. That said, I suspect we still disagree a lot about the amount of signal in this analysis, which is probably coming into play here.
I think that this paper contains comparable or greater amounts of evidence to Yudkowsky’s Intelligence Explosion Microeconomics, Bostrom’s Superintelligence, and AI-2027′s takeoff speed analysis. (In large part it contains this because I’m able to steal what I consider the most relevant insights from this previous work.) My own (biased!) opinion is that this is the best analysis we have on this question. It contains multiple angles on estimating the initial speed up from deplying ASARA, analysis of the value of r (which i think is hugely important), analysis of room to improve AI algs before matching humans, and some estimates/guesses at how much far certain improvements above human-level algs might go.
And it seems to me like you could level similar objections against previous these takeoff speeds analyses. Intelligence Explosion Microeconomics is also a paper, and it’s name “microeconomics” suggests that it’s going to do some empirically grounded mathematical modelling of the dynamics of recursive improvement. (I think it’s a great paper!) AI-2027 placed probability distributions over their parameters and ran a Monte Carlo (I believe). Superintelligence is an academic-style book. I think my work has similar or more disclaimers about how speculative it is than these other pieces. I’m not knocking these papers! I’m just claiming that I don’t think my paper is out of line of previous work in this regard.
I also think the evidential situation is here comparable (though probably more dire) to the evidence about AI timelines and AI alignment. The evidence doesn’t really constrain reality very much. But ppl still publish best-guess estimates/analyses.
Still, I do agree that it’s very easy for lay readers to overestimate how much evidence underpins the best analyses that the world has on these questions. The fact that this is even a paper puts it in a reference class of “papers”, and most papers aren’t about AGI-related stuff and are much less speculative. I felt this was at play with titotal’s critique of AI-2027. My experience reading it was “yeah these objections aren’t that surprising to me, what did expect from smg forecasting AGI and superintelligence”. And this pushes towards being even more upfront about the limitations of the analysis.
No i don’t. I think the paper contains meaningful evidence about the magnitude.
First, evidence that r is quite plausibly already below 1. If so, the SIE won’t last long. This makes is plausible that we get <3 years of progress.
Second, pointing out that 10 years of progress is ~10 OOMs of efficiency gains, which is a large amount relative to efficiency gains we’ve historically seen in ML, more than the total efficiency gains we’ve ever seen in many tech areas, and comparable to those we’ve ever seen in computing hardware. Even if r > 1 when we first develop ASARA, I think there’s a good chance that it falls below 1 during the course of 10 OOMs of progress. (Especially given the possibility of compute bottlenecks kicking in as cognitive inputs are massively increased but compute increases much more slowly.)
Third, evidence about the gap from ASARA algs to human-level algs. If this gap had been much bigger or much smaller, that should update our beliefs about how long the SIE will go on for.
Fourth, somewhat-transparent quantitative estimates of some the factors that are additional alg gains above human-level. E.g. “brain is severely undertrained”, “low fraction of data is relevant”, “variation between humans”. For these estimates, a reader can see roughly where the number is coming from.
Fifth, listing additional possible gains without transparent estimates of their quantitive size. Here, as I said, the process was speaking to people with relevant expertise and asking them to eye-ball/guess at the gain. So yes, these numbers are particularly untrustworthy! But to my mind they still contain some signal. It might have been that this process didn’t uncover any significant improvements beyond the brain. In fact though, there are multiple plausibly-big improvements, which did significantly widen my personal credences on how big the IE might be. In hindsight, I should have put in more effort to making all these estimates transparent, and flagged more clearly how big this uncertainty is.
You could summarise this all as “no evidence about the limits” bc, for some of these factors, there’s no explicit argument the factor isn’t absolutely massive. So if you came in with a strong view that one factor was massive, you won’t be much moved. But that misses that, for people in many epistemic situations, the five pieces of evidence i’ve just listed here will be informative.
Fyi, your comments have convinced me to add some additional qualifiers on this point:
To the top of the paper, just before the summary, we’ll add: Like all analyses of this topic, this paper is necessarily speculative. We draw on evidence where we can, but the results are significantly influenced by guesswork and subjective judgement.
To the table cell on limits, we’ll add: This involves a fair amount of guesswork and is a massive remaining uncertainty.
To the section estimating limits, we’ll add: some of the factors listed are plausibly even bigger than our upper estimate, e.g. “must satisfy physical constraints” and “fundamental improvements”.
I’m still not sure I’m understanding you here. If you’re sole concern is about the paper giving a misleading impression of accuracy/robustness, then I understand and am sympathetic.
But do you also think that the paper’s predictions would be better if I gave less detail?
I’m genuinely unsure if you think this. You initially claimed that the model is overly complex, which can lead to worse predictions by overfitting to noisy evidence. But you then instead claimed the parameter estimates were too complex, without giving any examples of what specific parts were misguided. What specific evidence/reasoning on which param do you think it would have been better to cut? (Not just bc it makes the paper seem overly fancy, but it makes the paper’s predictions worse.) I think we’ll need to get much more specific here to make progress.
Perhaps you think I shouldn’t have specified precise math or run a Monte Carlo? I get that concern re giving a misleading impression of robustness. But I think dropping the Monte Carlo would have made the predictions worse and the paper less useful. A precise math model makes the reasoning transparent. It makes it easier for others to build on the work. It also allows us to more accurately calculate the implications of the assumptions we make. It allows other to change the assumptions and look at how the results change. This has already been helpful for discussing the model in the LW comments for this post! I think it’s better to include the Monte Carlo for those benefits and clearly state the limitations of the analysis, than to make the analysis worse by excluding the Monte Carlo.
I have spent many years researching this topic, and that did inform this paper in many ways. I don’t it’s misleading to say this.
(fyi i’ll prob duck out after this point, hope my comments have been clarifying and thx for the discuission!)