This model strikes me as far more detailed than its inputs are known, which worries me. Maybe I’m being unfair here, I acknowledge the possibility I’m misunderstanding your methodology or aim—I’m sorry if so!—but I currently feel confused about how almost any of these input parameters were chosen or estimated.
Take your estimate of room for “fundamental improvements in the brain’s learning algorithm,” for example—you grant it’s hard to know, but nonetheless estimate it as around “3-30x.” How was this range chosen? Why not 300x, or 3 million? From what I understand the known physical limits—e.g., Landauer’s bound, the Carnot limit—barely constrain this estimate at all. I’m curious if you disagree, or if not, what constrains your estimate?
This model strikes me as far more detailed than its inputs are known
The model is very simple. Literally four lines of code:
This post is long, but it’s not long because it’s describing a complicated model. It’s long because it’s trying to estimate the inputs as well as possible.
Take your estimate of room for “fundamental improvements in the brain’s learning algorithm,” for example—you grant it’s hard to know, but nonetheless estimate it as around “3-30x.”
Yeah I think you’ve honed in on exactly the right subpart here. This is essentially the “and all other possible improvements” part of the estimate of the distance to ultimate limits, which is especially ungrounded.
Some of the other parts of the “distance to limits” section focus on specific improvements that are a bit easier to roughly ballpark. But one reviewer pointed out that there could be further fundamental improvements, which seemed right to me, and I wanted to avoid being biased in the conservative direction by not including it.
My basic methodology for this section was to interview various people about ways in which efficiency could improve beyond the brain level and how big they guessed these gains would be, and then have ppl review the results. But, as I say in the paper multiple times, the estimates for effective limits are especially speculative.
So I think it’s totally fair to say: the uncertainty on this param is way bigger than what’s you’ve done in your Monte Carlo, you should have a much wider range and so put more probability on really dramatic intelligence explosions. E.g. Lukas Finnveden makes a similar point in another comment.
But I don’t think of this as a flaw in the model:
It’s just true of reality that there’s a lot of uncertainty about the distance for effective limits for software, and it’s just true of reality that that translates into a lot of uncertainty about how big an SIE might be. So it seems like the model’s doing a good job in capturing that.
And yes, I think i’ve probably underestimated the extent of this uncertainty in this analysis.
As I said above, the model isn’t complicated. So there’s not some problem of fitting a model with lots of gears and surprising interaction effects to multiple params that have massive uncertainties, resulting in predictions that we don’t know where they came from. The model inputs flow through to the bottom line in ways that are very simple and transparent. Lukas has another comment that does a good job of explaining this.
Fwiw, I think it could be possible to substantially improve upon my estimates of the effective limits param but spending a week researching and interviewing experts. I’d be excited for someone to do that!
I agree the function and parameters themselves are simple, but the process by which you estimate their values is not. Your paper explaining this process and the resulting forecast is 40 pages, and features a Monte Carlo simulation, the Cobb-Douglas model of software progress, the Jones economic growth model (which the paper describes as a “semi-endogenous law of motion for AI software”), and many similarly technical arcana.
To be clear, my worry is less that the model includes too many ad hoc free parameters, such that it seems overfit, than that the level of complexity and seeming-rigor is quite disproportionate to the solidity of its epistemic justification.
For example, the section we discussed above (estimating the “gap from human learning to effective limits”) describes a few ways ideal learning might outperform human learning—e.g., that ideal systems might have more and better data, update more efficiently, benefit from communicating with other super-smart systems, etc. And indeed I agree these seem like some of the ways learning algorithms might be improved.
But I feel confused by the estimates of room for improvement given these factors. For example, the paper suggests better “data quality” could improve learning efficiency by “at least 3x and plausibly 300x.” But why not three thousand, or three million, or any other physically-possible number? Does some consideration described in the paper rule these out, or even give reason to suspect they’re less likely than your estimate?
I feel similarly confused by the estimate of overall room for improvement in learning efficiency. If I understand correctly, the paper suggests this limit—the maximum improvement in learning efficiency a recursively self-improving superintelligence could gain, beyond the efficiency of human brains—is “4-10 OOMs,” which it describes as equivalent to 4-10 “years of AI progress, at the rate of progress seen in recent years.”
Perhaps I’m missing something, and again I’m sorry if so, but after reading the paper carefully twice I don’t see any arguments that justify this choice of range. Why do you expect the limit of learning efficiency for a recursively self-improving superintelligence is 4-10 recent-progress-years above humans?
Most other estimates in the paper seem to me like they were made from a similar epistemic state. For example, half the inputs to the estimate of takeoff slope from automating AI R&D come from asking 5 lab employees to guess; I don’t see any justification for the estimate of diminishing returns to parallel labor, etc. And so I feel worried overall that readers will mistake the formality of the presentation of these estimates as evidence that they meaningfully constrain or provide evidence for the paper’s takeoff forecast.
I realize it is difficult to predict the future, especially in respects so dissimilar from anything that has occurred before. And I think it can be useful to share even crude estimates, when that is all we have, so long as that crudeness is clearlystressed and kept in mind. But from my perspective, this paper—which you describe as evaluating “exactly how dramatic the software intelligence explosion will be”!—really quite under-stresses this.
I agree the function and parameters themselves are simple, but the process by which you estimate their values is not. Your paper explaining this process and the resulting forecast is 40 pages, and features a Monte Carlo simulation, the Cobb-Douglas model of software progress, the Jones economic growth model (which the paper describes as a “semi-endogenous law of motion for AI software”), and many similarly technical arcana.
Could you explain why you think it’s bad if the “process by which I estimate parameter values” is too complex? What specific things were overly complex?
The specific things you mention don’t make sense to me.
the resulting forecast is 40 pages,
(Fwiw, it’s 20-25 pages excluding figures and appendices)
To my mind, identifying a model with a small number of key params, and then carefully assessing each param from as many different angles as possible, is a good approach. Yes, I could have approached each param from fewer angles, but that would make the overall estimate less robust.
Monte Carlo simulation
You surely know this, but running the Monte Carlo simulation doesn’t add complexity to the process of estimating the values (given that i’m estimating ranges in any case). And it seems pretty useful to do given the massive uncertainty.
Cobb-Douglas model of software progress, the Jones economic growth model
The “Cobb-Douglas model” and the “Jones economic growth model” are the same thing, and used to derive the 4 line model mentioned above. Mentioning these terms doesn’t add complexity to the process of estimating the params. Tbh, i’m confused and a bit frustrated at you calling me out for using these terms. The implict accusation that I’m using them to aggrandize my paper. But neither of these terms appear in the summary. They appear in the main body essentially the bare minimum number of times, I think once each. Would you prefer I didn’t mention them these standard terms for (very simple!) math models?
and many similarly technical arcana
I basically think you’re being unfair here, so want to challenge you to actually name these or retract.
… So that’s my response to the charge that the param estimates are overly complicated. But I want to respond to one other point you make, that i’m v sympathetic to.
level of… seeming-rigor is quite disproportionate to the solidity of its epistemic justification
This feels like maybe more the core thing you’re reacting to here.
I was worried about the paper coming off in this way. An earlier draft had more caveats and repeated them more often. Reviewers suggested I was being excessive. I’ll take yours comments here as a sign that I should have ignored them.
But the paper is still very clear and explicit about its limitations.
Quote from the summary (that will be read way more than any references to “Cobb Douglas” or “Jones model”) [bolded emphasis in the original]:
“Our model is extremely basic and has many limitations”
“Garbage in, garbage out”. We’ve done our best to estimate the model parameters, but there are massive uncertainties in all of them”
A previous draft highlight the distance to limits here as an acute example—I think you’re right about this
“Overall, we think of this model as a back-of-the-envelope calculation. It’s our best guess, and we think there are some meaningful takeaways, but we don’t put much faith in the specific numbers.”
I basically think you’re being unfair here, so want to challenge you to actually name these or retract.
… So that’s my response to the charge that the param estimates are overly complicated. But I want to respond to one other point you make
It sounds like we’re talking past each other, if you think I’m making two different points. The concern I’m trying to express is that this takeoff model—by which I mean the overall model/argument/forecast presented in the paper, not just the literal code—strikes me as containing confusingly much detail/statistics/elaboration/formality, given (what seems to me like) the extreme sparsity of evidence for its component estimates.
the paper is still very clear and explicit about its limitations
I grant and (genuinely) appreciate that the paper includes many caveats. I think that helps a bunch, and indeed helps on exactly the dimension of my objection. In contrast, I think it probably anti-helped to describe the paper as forecasting “exactly how big” the intelligence explosion will be, in a sense constrained by years of research on the question.
It seems to me that demand for knowledge about how advanced AI will go, and about what we might do to make it go better, currently far outstrips supply. There are a lot of people who would like very much to have less uncertainty about takeoff dynamics, some of whom I expect might even make importantly different decisions as a result.
… and realistically, I think many of those people probably won’t spend hours carefully reading the report, as I did. And I expect the average such person is likely to greatly overestimate the amount of evidence the paper actually contains for its headline takeoff forecast.
Most obviously, from my perspective, I expect most casual readers to assume that a forecast billed as modeling “exactly how big” the intelligence explosion might be, is likely to contain evidence about the magnitude of the explosion! But I see no evidence—not even informal argument—in the paper about the limits that determine this magnitude, and unless I misunderstand your comments it seems you agree?
I think it probably anti-helped to describe the paper as forecasting “exactly how big”
That’s fair. I’ve removed the word “exactly” from the top of this paper. Edits will take a few days to process on the website version.
(I definitely didn’t intend “exactly” to connote precision. I wanted to highlight that the paper was focussed on the extent of the intelligence explosion, in contrast with our previous paper which argued that there could be accelerating progress but didn’t analyse how big the resulting SIE might be. Some reviewers were confused at the contrast. For the tweet, I think i was also subconsciously imitating the first tweet of AI-2027 which said “How, exactly, could AI take over by 2027″. But I agree the word “exactly” here could easily be misread as implying the paper gives precise results, which is an oversight.)
Fwiw, the results table gives the results to 1 sig fig and uses “~” repeatedly to avoid the impression of false precision. And the paper explicitly says “While the exact numbers here are obviously not to be trusted...”, before giving what I believe are more robust takeaways. In the discussion opens with “If this analysis is right in broad strokes” (emphasis mine).
The twitter thread also said “It goes without saying: the model is very basic and has many big limitations” and “We estimate these three parameters through a mix of empirical evidence and guesswork”. The lead tweet also said “This is my best stab at an answer”, which i thought connoted an informal, modest and tentative answer. (emphasis mine)
This is all to say that, again, i’m sympathetic with the worry that people will overestimated the precision and accuracy of the analysis and made significant efforts to avoid this. Honestly, I’d guess I made greater efforts than most similarly-speculative posts in this reference class. That said, I suspect we still disagree a lot about the amount of signal in this analysis, which is probably coming into play here.
And I expect the average such person is likely to greatly overestimate the amount of evidence the paper actually contains for its headline takeoff forecast.
I think that this paper contains comparable or greater amounts of evidence to Yudkowsky’s Intelligence Explosion Microeconomics, Bostrom’s Superintelligence, and AI-2027′s takeoff speed analysis. (In large part it contains this because I’m able to steal what I consider the most relevant insights from this previous work.) My own (biased!) opinion is that this is the best analysis we have on this question. It contains multiple angles on estimating the initial speed up from deplying ASARA, analysis of the value of r (which i think is hugely important), analysis of room to improve AI algs before matching humans, and some estimates/guesses at how much far certain improvements above human-level algs might go.
And it seems to me like you could level similar objections against previous these takeoff speeds analyses. Intelligence Explosion Microeconomics is also a paper, and it’s name “microeconomics” suggests that it’s going to do some empirically grounded mathematical modelling of the dynamics of recursive improvement. (I think it’s a great paper!) AI-2027 placed probability distributions over their parameters and ran a Monte Carlo (I believe). Superintelligence is an academic-style book. I think my work has similar or more disclaimers about how speculative it is than these other pieces. I’m not knocking these papers! I’m just claiming that I don’t think my paper is out of line of previous work in this regard.
I also think the evidential situation is here comparable (though probably more dire) to the evidence about AI timelines and AI alignment. The evidence doesn’t really constrain reality very much. But ppl still publish best-guess estimates/analyses.
Still, I do agree that it’s very easy for lay readers to overestimate how much evidence underpins the best analyses that the world has on these questions. The fact that this is even a paper puts it in a reference class of “papers”, and most papers aren’t about AGI-related stuff and are much less speculative. I felt this was at play with titotal’s critique of AI-2027. My experience reading it was “yeah these objections aren’t that surprising to me, what did expect from smg forecasting AGI and superintelligence”. And this pushes towards being even more upfront about the limitations of the analysis.
I see no evidence—not even informal argument—in the paper about the limits that determine this magnitude, and unless I misunderstand your comments it seems you agree?
No i don’t. I think the paper contains meaningful evidence about the magnitude.
First, evidence that r is quite plausibly already below 1. If so, the SIE won’t last long. This makes is plausible that we get <3 years of progress.
Second, pointing out that 10 years of progress is ~10 OOMs of efficiency gains, which is a large amount relative to efficiency gains we’ve historically seen in ML, more than the total efficiency gains we’ve ever seen in many tech areas, and comparable to those we’ve ever seen in computing hardware. Even if r > 1 when we first develop ASARA, I think there’s a good chance that it falls below 1 during the course of 10 OOMs of progress. (Especially given the possibility of compute bottlenecks kicking in as cognitive inputs are massively increased but compute increases much more slowly.)
Third, evidence about the gap from ASARA algs to human-level algs. If this gap had been much bigger or much smaller, that should update our beliefs about how long the SIE will go on for.
Fourth, somewhat-transparent quantitative estimates of some the factors that are additional alg gains above human-level. E.g. “brain is severely undertrained”, “low fraction of data is relevant”, “variation between humans”. For these estimates, a reader can see roughly where the number is coming from.
Fifth, listing additional possible gains without transparent estimates of their quantitive size. Here, as I said, the process was speaking to people with relevant expertise and asking them to eye-ball/guess at the gain. So yes, these numbers are particularly untrustworthy! But to my mind they still contain some signal. It might have been that this process didn’t uncover any significant improvements beyond the brain. In fact though, there are multiple plausibly-big improvements, which did significantly widen my personal credences on how big the IE might be. In hindsight, I should have put in more effort to making all these estimates transparent, and flagged more clearly how big this uncertainty is.
You could summarise this all as “no evidence about the limits” bc, for some of these factors, there’s no explicit argument the factor isn’t absolutely massive. So if you came in with a strong view that one factor was massive, you won’t be much moved. But that misses that, for people in many epistemic situations, the five pieces of evidence i’ve just listed here will be informative.
Fyi, your comments have convinced me to add some additional qualifiers on this point:
To the top of the paper, just before the summary, we’ll add: Like all analyses of this topic, this paper is necessarily speculative. We draw on evidence where we can, but the results are significantly influenced by guesswork and subjective judgement.
To the table cell on limits, we’ll add: This involves a fair amount of guesswork and is a massive remaining uncertainty.
To the section estimating limits, we’ll add: some of the factors listed are plausibly even bigger than our upper estimate, e.g. “must satisfy physical constraints” and “fundamental improvements”.
The concern I’m trying to express is that this takeoff model—by which I mean the overall model/argument/forecast presented in the paper, not just the literal code—strikes me as containing confusingly much detail/statistics/elaboration/formality, given (what seems to me like) the extreme sparsity of evidence for its component estimates.
I’m still not sure I’m understanding you here. If you’re sole concern is about the paper giving a misleading impression of accuracy/robustness, then I understand and am sympathetic.
But do you also think that the paper’s predictions would be better if I gave less detail?
I’m genuinely unsure if you think this. You initially claimed that the model is overly complex, which can lead to worse predictions by overfitting to noisy evidence. But you then instead claimed the parameter estimates were too complex, without giving any examples of what specific parts were misguided. What specific evidence/reasoning on which param do you think it would have been better to cut? (Not just bc it makes the paper seem overly fancy, but it makes the paper’s predictions worse.) I think we’ll need to get much more specific here to make progress.
Perhaps you think I shouldn’t have specified precise math or run a Monte Carlo? I get that concern re giving a misleading impression of robustness. But I think dropping the Monte Carlo would have made the predictions worse and the paper less useful. A precise math model makes the reasoning transparent. It makes it easier for others to build on the work. It also allows us to more accurately calculate the implications of the assumptions we make. It allows other to change the assumptions and look at how the results change. This has already been helpful for discussing the model in the LW comments for this post! I think it’s better to include the Monte Carlo for those benefits and clearly state the limitations of the analysis, than to make the analysis worse by excluding the Monte Carlo.
in a sense constrained by years of research on the question
I have spent many years researching this topic, and that did inform this paper in many ways. I don’t it’s misleading to say this.
(fyi i’ll prob duck out after this point, hope my comments have been clarifying and thx for the discuission!)
Could you explain why you think it’s bad if the “process by which I estimate parameter values” is too complex? What specific things were overly complex?
Generally speaking, the output of a process is only as accurate as the least accurate of its inputs or steps. There is no point for example calculating up to the 10th significant digit the area of a square whose side you only know with a 10% margin of error. I think the risk here is the same—if the process is sophisticated and full of complex non-linear interactions, then it would need proportionately accurate inputs for the errors not to explode. Otherwise it’s genuinely better to just offer a vibe-y guess.
I agree with the general point, but don’t think it applies to this model.
I’m not calculating anything to a high degree of precision, inputs or outputs.
There aren’t complicated interaction effects with lots of noisy inputs such that the model might overfit to noise.
I could have dropped the code, but then i’d have a worse understanding of what my best-guess inputs imply about the output. And it the analysis would be less transparent. And other couldn’t run it for their preferred inputs.
I just feel like the length and complexity of the thinking involved is all fundamentally undermined by this uncertainty. The consequences are almost entirely parameter-determined (since as you say, the core model is very simple). Something like how many OOM gains are possible before hitting limits for example is key—this is literally what makes the difference between a world with slightly better software engineering, one in which all software engineers and scientists are now unemployed because AIs completely wipe the floor with them, and one in which ASI iteratively self-improves its way to physical godhood and takes over the light-cone. And I feel like something of that kind implies so many answers to very open questions about the world, the nature of intelligence and of computation itself, I’m not sure how could any estimate produce anything else than some kind of almost circular reasoning.
If I understand correctly, the paper suggests this limit—the maximum improvement in learning efficiency a recursively self-improving superintelligence could gain, beyond the efficiency of human brains—is “4-10 OOMs,” which it describes as equivalent to 4-10 “years of AI progress, at the rate of progress seen in recent years.”
Perhaps I’m missing something, and again I’m sorry if so, but after reading the paper carefully twice I don’t see any arguments that justify this choice of range. Why do you expect the limit of learning efficiency for a recursively self-improving superintelligence is 4-10 recent-progress-years above humans?
Oh there’s lots of arguments feeding into that range. Look at this part of the paper. There’s a long list of bullet points of different ways that superintelligences could be more efficient than humans. Each of the estimates have a range of X-Y OOMs. Then:
Overall, the additional learning efficiency gains from these sources suggest that effective limits are 4 − 12 OOMs above the human brain. The high end seems extremely high, and we think there’s some risk of double counting some of the gains here in the different buckets, so we will bring down our high end to 10 OOMs.
Here: 4 is supposed to be the product of all the lower numbers guessed-at above (“X” in “X-Y”), and 12 is supposed to be the product of all the upper numbers (“Y” in “X-Y”).
This model strikes me as far more detailed than its inputs are known, which worries me. Maybe I’m being unfair here, I acknowledge the possibility I’m misunderstanding your methodology or aim—I’m sorry if so!—but I currently feel confused about how almost any of these input parameters were chosen or estimated.
Take your estimate of room for “fundamental improvements in the brain’s learning algorithm,” for example—you grant it’s hard to know, but nonetheless estimate it as around “3-30x.” How was this range chosen? Why not 300x, or 3 million? From what I understand the known physical limits—e.g., Landauer’s bound, the Carnot limit—barely constrain this estimate at all. I’m curious if you disagree, or if not, what constrains your estimate?
The model is very simple. Literally four lines of code:
This post is long, but it’s not long because it’s describing a complicated model. It’s long because it’s trying to estimate the inputs as well as possible.
Yeah I think you’ve honed in on exactly the right subpart here. This is essentially the “and all other possible improvements” part of the estimate of the distance to ultimate limits, which is especially ungrounded.
Some of the other parts of the “distance to limits” section focus on specific improvements that are a bit easier to roughly ballpark. But one reviewer pointed out that there could be further fundamental improvements, which seemed right to me, and I wanted to avoid being biased in the conservative direction by not including it.
My basic methodology for this section was to interview various people about ways in which efficiency could improve beyond the brain level and how big they guessed these gains would be, and then have ppl review the results. But, as I say in the paper multiple times, the estimates for effective limits are especially speculative.
So I think it’s totally fair to say: the uncertainty on this param is way bigger than what’s you’ve done in your Monte Carlo, you should have a much wider range and so put more probability on really dramatic intelligence explosions. E.g. Lukas Finnveden makes a similar point in another comment.
But I don’t think of this as a flaw in the model:
It’s just true of reality that there’s a lot of uncertainty about the distance for effective limits for software, and it’s just true of reality that that translates into a lot of uncertainty about how big an SIE might be. So it seems like the model’s doing a good job in capturing that.
And yes, I think i’ve probably underestimated the extent of this uncertainty in this analysis.
As I said above, the model isn’t complicated. So there’s not some problem of fitting a model with lots of gears and surprising interaction effects to multiple params that have massive uncertainties, resulting in predictions that we don’t know where they came from. The model inputs flow through to the bottom line in ways that are very simple and transparent. Lukas has another comment that does a good job of explaining this.
Fwiw, I think it could be possible to substantially improve upon my estimates of the effective limits param but spending a week researching and interviewing experts. I’d be excited for someone to do that!
I agree the function and parameters themselves are simple, but the process by which you estimate their values is not. Your paper explaining this process and the resulting forecast is 40 pages, and features a Monte Carlo simulation, the Cobb-Douglas model of software progress, the Jones economic growth model (which the paper describes as a “semi-endogenous law of motion for AI software”), and many similarly technical arcana.
To be clear, my worry is less that the model includes too many ad hoc free parameters, such that it seems overfit, than that the level of complexity and seeming-rigor is quite disproportionate to the solidity of its epistemic justification.
For example, the section we discussed above (estimating the “gap from human learning to effective limits”) describes a few ways ideal learning might outperform human learning—e.g., that ideal systems might have more and better data, update more efficiently, benefit from communicating with other super-smart systems, etc. And indeed I agree these seem like some of the ways learning algorithms might be improved.
But I feel confused by the estimates of room for improvement given these factors. For example, the paper suggests better “data quality” could improve learning efficiency by “at least 3x and plausibly 300x.” But why not three thousand, or three million, or any other physically-possible number? Does some consideration described in the paper rule these out, or even give reason to suspect they’re less likely than your estimate?
I feel similarly confused by the estimate of overall room for improvement in learning efficiency. If I understand correctly, the paper suggests this limit—the maximum improvement in learning efficiency a recursively self-improving superintelligence could gain, beyond the efficiency of human brains—is “4-10 OOMs,” which it describes as equivalent to 4-10 “years of AI progress, at the rate of progress seen in recent years.”
Perhaps I’m missing something, and again I’m sorry if so, but after reading the paper carefully twice I don’t see any arguments that justify this choice of range. Why do you expect the limit of learning efficiency for a recursively self-improving superintelligence is 4-10 recent-progress-years above humans?
Most other estimates in the paper seem to me like they were made from a similar epistemic state. For example, half the inputs to the estimate of takeoff slope from automating AI R&D come from asking 5 lab employees to guess; I don’t see any justification for the estimate of diminishing returns to parallel labor, etc. And so I feel worried overall that readers will mistake the formality of the presentation of these estimates as evidence that they meaningfully constrain or provide evidence for the paper’s takeoff forecast.
I realize it is difficult to predict the future, especially in respects so dissimilar from anything that has occurred before. And I think it can be useful to share even crude estimates, when that is all we have, so long as that crudeness is clearly stressed and kept in mind. But from my perspective, this paper—which you describe as evaluating “exactly how dramatic the software intelligence explosion will be”!—really quite under-stresses this.
Hi,
Could you explain why you think it’s bad if the “process by which I estimate parameter values” is too complex? What specific things were overly complex?
The specific things you mention don’t make sense to me.
(Fwiw, it’s 20-25 pages excluding figures and appendices)
To my mind, identifying a model with a small number of key params, and then carefully assessing each param from as many different angles as possible, is a good approach. Yes, I could have approached each param from fewer angles, but that would make the overall estimate less robust.
You surely know this, but running the Monte Carlo simulation doesn’t add complexity to the process of estimating the values (given that i’m estimating ranges in any case). And it seems pretty useful to do given the massive uncertainty.
The “Cobb-Douglas model” and the “Jones economic growth model” are the same thing, and used to derive the 4 line model mentioned above. Mentioning these terms doesn’t add complexity to the process of estimating the params. Tbh, i’m confused and a bit frustrated at you calling me out for using these terms. The implict accusation that I’m using them to aggrandize my paper. But neither of these terms appear in the summary. They appear in the main body essentially the bare minimum number of times, I think once each. Would you prefer I didn’t mention them these standard terms for (very simple!) math models?
I basically think you’re being unfair here, so want to challenge you to actually name these or retract.
… So that’s my response to the charge that the param estimates are overly complicated. But I want to respond to one other point you make, that i’m v sympathetic to.
This feels like maybe more the core thing you’re reacting to here.
I was worried about the paper coming off in this way. An earlier draft had more caveats and repeated them more often. Reviewers suggested I was being excessive. I’ll take yours comments here as a sign that I should have ignored them.
But the paper is still very clear and explicit about its limitations.
Quote from the summary (that will be read way more than any references to “Cobb Douglas” or “Jones model”) [bolded emphasis in the original]:
“Our model is extremely basic and has many limitations”
“Garbage in, garbage out”. We’ve done our best to estimate the model parameters, but there are massive uncertainties in all of them”
A previous draft highlight the distance to limits here as an acute example—I think you’re right about this
“Overall, we think of this model as a back-of-the-envelope calculation. It’s our best guess, and we think there are some meaningful takeaways, but we don’t put much faith in the specific numbers.”
It sounds like we’re talking past each other, if you think I’m making two different points. The concern I’m trying to express is that this takeoff model—by which I mean the overall model/argument/forecast presented in the paper, not just the literal code—strikes me as containing confusingly much detail/statistics/elaboration/formality, given (what seems to me like) the extreme sparsity of evidence for its component estimates.
I grant and (genuinely) appreciate that the paper includes many caveats. I think that helps a bunch, and indeed helps on exactly the dimension of my objection. In contrast, I think it probably anti-helped to describe the paper as forecasting “exactly how big” the intelligence explosion will be, in a sense constrained by years of research on the question.
It seems to me that demand for knowledge about how advanced AI will go, and about what we might do to make it go better, currently far outstrips supply. There are a lot of people who would like very much to have less uncertainty about takeoff dynamics, some of whom I expect might even make importantly different decisions as a result.
… and realistically, I think many of those people probably won’t spend hours carefully reading the report, as I did. And I expect the average such person is likely to greatly overestimate the amount of evidence the paper actually contains for its headline takeoff forecast.
Most obviously, from my perspective, I expect most casual readers to assume that a forecast billed as modeling “exactly how big” the intelligence explosion might be, is likely to contain evidence about the magnitude of the explosion! But I see no evidence—not even informal argument—in the paper about the limits that determine this magnitude, and unless I misunderstand your comments it seems you agree?
That’s fair. I’ve removed the word “exactly” from the top of this paper. Edits will take a few days to process on the website version.
(I definitely didn’t intend “exactly” to connote precision. I wanted to highlight that the paper was focussed on the extent of the intelligence explosion, in contrast with our previous paper which argued that there could be accelerating progress but didn’t analyse how big the resulting SIE might be. Some reviewers were confused at the contrast. For the tweet, I think i was also subconsciously imitating the first tweet of AI-2027 which said “How, exactly, could AI take over by 2027″. But I agree the word “exactly” here could easily be misread as implying the paper gives precise results, which is an oversight.)
Fwiw, the results table gives the results to 1 sig fig and uses “~” repeatedly to avoid the impression of false precision. And the paper explicitly says “While the exact numbers here are obviously not to be trusted...”, before giving what I believe are more robust takeaways. In the discussion opens with “If this analysis is right in broad strokes” (emphasis mine).
The twitter thread also said “It goes without saying: the model is very basic and has many big limitations” and “We estimate these three parameters through a mix of empirical evidence and guesswork”. The lead tweet also said “This is my best stab at an answer”, which i thought connoted an informal, modest and tentative answer. (emphasis mine)
This is all to say that, again, i’m sympathetic with the worry that people will overestimated the precision and accuracy of the analysis and made significant efforts to avoid this. Honestly, I’d guess I made greater efforts than most similarly-speculative posts in this reference class. That said, I suspect we still disagree a lot about the amount of signal in this analysis, which is probably coming into play here.
I think that this paper contains comparable or greater amounts of evidence to Yudkowsky’s Intelligence Explosion Microeconomics, Bostrom’s Superintelligence, and AI-2027′s takeoff speed analysis. (In large part it contains this because I’m able to steal what I consider the most relevant insights from this previous work.) My own (biased!) opinion is that this is the best analysis we have on this question. It contains multiple angles on estimating the initial speed up from deplying ASARA, analysis of the value of r (which i think is hugely important), analysis of room to improve AI algs before matching humans, and some estimates/guesses at how much far certain improvements above human-level algs might go.
And it seems to me like you could level similar objections against previous these takeoff speeds analyses. Intelligence Explosion Microeconomics is also a paper, and it’s name “microeconomics” suggests that it’s going to do some empirically grounded mathematical modelling of the dynamics of recursive improvement. (I think it’s a great paper!) AI-2027 placed probability distributions over their parameters and ran a Monte Carlo (I believe). Superintelligence is an academic-style book. I think my work has similar or more disclaimers about how speculative it is than these other pieces. I’m not knocking these papers! I’m just claiming that I don’t think my paper is out of line of previous work in this regard.
I also think the evidential situation is here comparable (though probably more dire) to the evidence about AI timelines and AI alignment. The evidence doesn’t really constrain reality very much. But ppl still publish best-guess estimates/analyses.
Still, I do agree that it’s very easy for lay readers to overestimate how much evidence underpins the best analyses that the world has on these questions. The fact that this is even a paper puts it in a reference class of “papers”, and most papers aren’t about AGI-related stuff and are much less speculative. I felt this was at play with titotal’s critique of AI-2027. My experience reading it was “yeah these objections aren’t that surprising to me, what did expect from smg forecasting AGI and superintelligence”. And this pushes towards being even more upfront about the limitations of the analysis.
No i don’t. I think the paper contains meaningful evidence about the magnitude.
First, evidence that r is quite plausibly already below 1. If so, the SIE won’t last long. This makes is plausible that we get <3 years of progress.
Second, pointing out that 10 years of progress is ~10 OOMs of efficiency gains, which is a large amount relative to efficiency gains we’ve historically seen in ML, more than the total efficiency gains we’ve ever seen in many tech areas, and comparable to those we’ve ever seen in computing hardware. Even if r > 1 when we first develop ASARA, I think there’s a good chance that it falls below 1 during the course of 10 OOMs of progress. (Especially given the possibility of compute bottlenecks kicking in as cognitive inputs are massively increased but compute increases much more slowly.)
Third, evidence about the gap from ASARA algs to human-level algs. If this gap had been much bigger or much smaller, that should update our beliefs about how long the SIE will go on for.
Fourth, somewhat-transparent quantitative estimates of some the factors that are additional alg gains above human-level. E.g. “brain is severely undertrained”, “low fraction of data is relevant”, “variation between humans”. For these estimates, a reader can see roughly where the number is coming from.
Fifth, listing additional possible gains without transparent estimates of their quantitive size. Here, as I said, the process was speaking to people with relevant expertise and asking them to eye-ball/guess at the gain. So yes, these numbers are particularly untrustworthy! But to my mind they still contain some signal. It might have been that this process didn’t uncover any significant improvements beyond the brain. In fact though, there are multiple plausibly-big improvements, which did significantly widen my personal credences on how big the IE might be. In hindsight, I should have put in more effort to making all these estimates transparent, and flagged more clearly how big this uncertainty is.
You could summarise this all as “no evidence about the limits” bc, for some of these factors, there’s no explicit argument the factor isn’t absolutely massive. So if you came in with a strong view that one factor was massive, you won’t be much moved. But that misses that, for people in many epistemic situations, the five pieces of evidence i’ve just listed here will be informative.
Fyi, your comments have convinced me to add some additional qualifiers on this point:
To the top of the paper, just before the summary, we’ll add: Like all analyses of this topic, this paper is necessarily speculative. We draw on evidence where we can, but the results are significantly influenced by guesswork and subjective judgement.
To the table cell on limits, we’ll add: This involves a fair amount of guesswork and is a massive remaining uncertainty.
To the section estimating limits, we’ll add: some of the factors listed are plausibly even bigger than our upper estimate, e.g. “must satisfy physical constraints” and “fundamental improvements”.
I’m still not sure I’m understanding you here. If you’re sole concern is about the paper giving a misleading impression of accuracy/robustness, then I understand and am sympathetic.
But do you also think that the paper’s predictions would be better if I gave less detail?
I’m genuinely unsure if you think this. You initially claimed that the model is overly complex, which can lead to worse predictions by overfitting to noisy evidence. But you then instead claimed the parameter estimates were too complex, without giving any examples of what specific parts were misguided. What specific evidence/reasoning on which param do you think it would have been better to cut? (Not just bc it makes the paper seem overly fancy, but it makes the paper’s predictions worse.) I think we’ll need to get much more specific here to make progress.
Perhaps you think I shouldn’t have specified precise math or run a Monte Carlo? I get that concern re giving a misleading impression of robustness. But I think dropping the Monte Carlo would have made the predictions worse and the paper less useful. A precise math model makes the reasoning transparent. It makes it easier for others to build on the work. It also allows us to more accurately calculate the implications of the assumptions we make. It allows other to change the assumptions and look at how the results change. This has already been helpful for discussing the model in the LW comments for this post! I think it’s better to include the Monte Carlo for those benefits and clearly state the limitations of the analysis, than to make the analysis worse by excluding the Monte Carlo.
I have spent many years researching this topic, and that did inform this paper in many ways. I don’t it’s misleading to say this.
(fyi i’ll prob duck out after this point, hope my comments have been clarifying and thx for the discuission!)
Generally speaking, the output of a process is only as accurate as the least accurate of its inputs or steps. There is no point for example calculating up to the 10th significant digit the area of a square whose side you only know with a 10% margin of error. I think the risk here is the same—if the process is sophisticated and full of complex non-linear interactions, then it would need proportionately accurate inputs for the errors not to explode. Otherwise it’s genuinely better to just offer a vibe-y guess.
I agree with the general point, but don’t think it applies to this model.
I’m not calculating anything to a high degree of precision, inputs or outputs.
There aren’t complicated interaction effects with lots of noisy inputs such that the model might overfit to noise.
I could have dropped the code, but then i’d have a worse understanding of what my best-guess inputs imply about the output. And it the analysis would be less transparent. And other couldn’t run it for their preferred inputs.
I just feel like the length and complexity of the thinking involved is all fundamentally undermined by this uncertainty. The consequences are almost entirely parameter-determined (since as you say, the core model is very simple). Something like how many OOM gains are possible before hitting limits for example is key—this is literally what makes the difference between a world with slightly better software engineering, one in which all software engineers and scientists are now unemployed because AIs completely wipe the floor with them, and one in which ASI iteratively self-improves its way to physical godhood and takes over the light-cone. And I feel like something of that kind implies so many answers to very open questions about the world, the nature of intelligence and of computation itself, I’m not sure how could any estimate produce anything else than some kind of almost circular reasoning.
Oh there’s lots of arguments feeding into that range. Look at this part of the paper. There’s a long list of bullet points of different ways that superintelligences could be more efficient than humans. Each of the estimates have a range of X-Y OOMs. Then:
Here: 4 is supposed to be the product of all the lower numbers guessed-at above (“X” in “X-Y”), and 12 is supposed to be the product of all the upper numbers (“Y” in “X-Y”).