The AI Explosion Might Never Happen
[This is a crosspost from https://amistrongeryet.substack.com/p/recursive-self-improvement-foom, lightly edited for the LessWrong audience. This is my first LessWrong post; feedback greatly appreciated!]
LessWrong readers will be familiar with the concept of recursive self-improvement: as AIs become increasingly capable, they will acquire the ability to assist in their own development. Eventually, we will manage to create an AI that is slightly better than us at AI design. Since that system is better at AI design than its human creators, it should be able to design an AI better than itself. That second system should then be able to design its own improved successor, and so forth.
A lot of people seem to believe that, once AIs approach human capability, a “foom” loop of rapid self-improvement is more or less inevitable. But I don’t think this is the case: even as AI capabilities increase, the effort required to achieve further improvements will likely also increase, and the resulting feedback loop might converge rather than diverge.
I’ll explore some of the factors at play, and list some early indicators we can watch for to indicate when we might be entering a period of rapid self-improvement.
A Parable
Once upon a time, there was a manufacturing company.
To begin with, it was a small, locally owned business. However, the owner had ambitions to create something larger. He used a small inheritance to hire several engineers to research improved manufacturing techniques.
Armed with these improved techniques, the company was able to produce their products faster, at a lower price. This led to an increase in both sales and profits.
The owner plowed those increased profits back into R&D, yielding further efficiency improvements. As the company’s manufacturing prowess continued to improve, the competition fell farther behind. Soon they were able to hire their pick of each year’s graduating class of engineering talent; no one else could pay as well, and besides, all of the best work was being done there.
Every year brought higher profits, more R&D, improved efficiency, increased sales, in an unending feedback loop. Eventually the company was manufacturing everything on Earth; no other firm could compete. Fortunately, there was no need to stop at mere world domination; such was the company’s prowess that they could easily manufacture the rockets to carry people to other planets, creating new markets that in turn allowed R&D to continue, and even accelerate. Within a half century of its founding, the company had developed molecular nanotechnology, interplanetary colonies, and was even making progress on lower back pain. All because of the original decision to invest profits into R&D.
Yes, It’s a Silly Story, That’s the Point
In general, increased R&D spending should yield increased profits, which in turn should allow further R&D spending. But no company has ever managed to parlay this into world domination. Even titans like Apple or Google bump up against the limits of their respective market segments.
There are of course many reasons for this, but it all boils down to the fact that while each dollar of increased R&D should return some amount of increased profit, it might return less than a dollar. Suppose you spend $10,000,000 per year manufacturing widgets, you spend $5,000,000 on a more efficient process, and the new process is 3% more efficient. That efficiency saves you $300,000 per year. It will take over 16 years to pay back the R&D cost; after accounting for time-value-of-money and other factors, you probably didn’t come out ahead, and you certainly won’t be in a position to fund an accelerating feedback loop.
Positive Feedback Eventually Reaches a Limit
We see diminishing feedback all the time in the real world:
Teachers are taught by other teachers, but we don’t see a runaway spiral of educational brilliance.
Chip design relies heavily on software tools, which are computationally demanding. Better chips can run these tools more efficiently, yet we haven’t experienced an uncontrolled rate of improvement.
In biological evolution, a species’ fitness directly correlates to its population size, thereby creating more room for beneficial mutations. Moreover, the very mechanisms that drive evolution – such as mutation rates and mate selection – are themselves subject to evolutionary improvement. However, this has not led to a biological singularity.
Feedback loops often do lead to spiraling growth, for a while. Smartphone sales and capabilities both grew explosively from 2010 through around 2015, until the market became saturated. An invasive species may spread like wildfire… until it covers the entire available territory. Every feedback loop eventually encounters a limiting factor, at which point diminishing returns set in. Teaching ability is limited by human intelligence. Chip capabilities are limited by the manufacturing process. AI may turn out to be limited by computing hardware, training data, a progressively increasing difficulty in finding further improvements, or some sort of complexity limit on intelligence.
We’re all familiar with the idea of exponential growth, as illustrated by the adoption of a viral product, or the increase in chip complexity captured by Moore’s Law. When diminishing returns set in, the result is known as an S-curve:
Note that the first half of an S-curve looks exactly like exponential growth. Using Facebook[2] as an example, this is the period where they were taking the world by storm; R&D spending went into initiatives with huge payoff, such as building core features. In recent years, having exhausted natural avenues for growth, Facebook has spent tens of billions of dollars developing the “metaverse”… yielding little return, so far.
Progress in AI might similarly follow an S-curve (or, more likely, a series of S-curves). We are currently in a period of exponential growth of generative AI. I believe this generative AI curve will eventually flatten out – at something well short of superhuman AGI. Only when we find ways of introducing capabilities such as memory and iterative exploration, which may require breaking out of the LLM paradigm, will rapid progress resume – for a while.
If you step way back, the simple fact that the Earth still exists in its current form tells us that no recursive process in the multi-billion-year history of the planet has ever spiraled completely out of control; some limit has always been reached. It’s legitimately possible that AI will be the first phenomenon to break that streak. But it’s far from guaranteed. Let’s consider how things might play out.
Impact Of Human-Level AI
In the near term, I expect the impact of recursive self-improvement to be minimal. Code-authoring tools such as GitHub Copilot notwithstanding, current AIs can’t offload much of the work involved in AI R&D[3]. If junior-level coding tasks were holding up progress toward AGI, presumably organizations like OpenAI and Google / DeepMind would just hire some more junior coders. Improvements to tools like Copilot will have only modest impact on R&D productivity, insufficient to trigger positive feedback.
As AIs reach roughly human level on a given task, they can free people up to work on other tasks. (Note that by “human-level”, we mean the typical person working to advance cutting-edge AI, which presumably includes a lot of highly talented folks.) When AI reaches this level at 50% of the tasks involved in AI R&D, that’s loosely equivalent to doubling the number of researchers. When 90% of tasks can be automated, that’s like increasing staff 10x. When 100% of tasks can be automated, we become limited by the cost and availability of GPUs to run virtual AI researchers; my napkin math suggests that the impact might be to (virtually) multiply the number of people working on AI by, say, a factor of 100[4].
This sounds like it would have a colossal impact, but it’s not necessarily so. One estimate suggests that personnel only account for 18% of the cost of training a state-of-the-art model (the rest goes for GPUs and other hardware). Replacing the existing staff with cheap AIs would thus free up at most 18% of the R&D budget.
Increasing the number of workers by 100x might have a larger impact. We could expect a dramatic increase in the pace of improvements to model architecture, algorithms, training processes, and so forth. We’d have more innovations, be able to undertake more complex architectures, and be able to do more work to generate or clean up training data. However, the computing power available for experimental training runs will still be limited, so that army of virtual AI researchers will to some extent be bottlenecked by the impossibility of testing all the ideas they’re coming up with. And we are already beginning to exhaust some of the highest quality, most easily accessible sorts of training data. (Stratechery reports a rumor that “Google is spending a billion dollars this year on generating new training data”.)
The point where AIs begin reaching elite human-level performance across an array of tasks will also herald the arrival of one or more headwinds:
By definition, we will have outgrown the use of human-authored materials for training data. Such materials are fundamental to the training of current LLMs, but are unlikely to carry us past the level of human ability[5].
Progress in any technical field becomes more difficult as that field progresses, and AI seems unlikely to be an exception. The low-hanging fruit gets picked first; as a system becomes more complex and finely tuned, each change requires more effort and has less overall impact. This is why R&D investment required to uphold Moore’s Law has increased drastically over time, and yet progress in chip metrics such as transistor count is finally slowing down[6]. As we approach human-level AGI, a similar phenomenon will likely be rearing its head.
Some technologies eventually encounter fundamental limits. The rocket equation makes it difficult to reach orbit from Earth’s gravity well; if the planet were even moderately larger, it would be nearly impossible. It’s conceivable that some sort of complexity principle makes it increasingly difficult to increase raw intelligence much beyond the human level, as the number of facts to keep in mind and the subtlety of the connections to be made increases[7].
The upshot is that when AIs reach elite human level at AI research, the resulting virtual workforce will notably accelerate progress, but the impact will likely be limited. GPU capacity will not be increasing at the same pace as the (virtual) worker population, and we will be running into a lack of superhuman training data, the generally increasing difficulty of progress, and the possibility of a complexity explosion.
It’s hard to say how this will net out. I could imagine a period of multiple years where progress, say, doubles; or I could imagine that self-improvement merely suffices to eke out the existing pace a bit longer, the way that exotic technologies like extreme ultraviolet lithography have not accelerated Moore’s Law. I don’t think the possibilities at this stage include the potential for any sort of capability explosion, especially because the impact will be spread out over time – AI won’t achieve human level at every aspect of AI R&D at once.
(Many thanks to Jeremy Howard for pointing out that in many cases, AIs are already being used to produce training data – the output of one model can be used to generate training data for other models. Often this is “merely” used to build a small model that, for a certain task (say, coding), emulates the performance of a larger (and hence more expensive / slower) general-purpose model. This allows specific tasks to be accomplished more efficiently, but does not necessarily advance the frontier of performance. However, it may in fact be possible to produce cutting-edge models this way, especially by incorporating techniques for using additional computation to improve the output of the model that is generating the training data[8]. There is no guarantee that this would lead to a positive feedback cycle, but it at least opens the possibility.)
Impact Of Superhuman AI
What happens as AIs begin performing AI research at a significantly higher level than the typical researcher at the leading labs?
On the one hand, the impact could be profound. A team of – in effect – hundreds of thousands of super-von Neumanns would presumably generate a flood of innovative ideas. Some of these ideas would be at least as impactful as deep learning, transformers, and the other key innovations which led to the current wave of progress. GPU availability would still limit the number of large-scale training experiments we can run, but we would presumably get more out of each experiment. And superintelligences might find ways of getting high-level performance out of smaller models, or at least extrapolating experimental results so that large cutting-edge models could be designed based on tests performed on small models.
On the other hand, the headwinds discussed earlier will apply even more sharply at this stage.
It seems conceivable that this will balance out to a positive feedback loop, with AI capabilities accelerating rapidly within a few months or years. It also seems possible that the ever-increasing difficulty of further progress will prevail, and even superhuman performance – in this scenario, likely only modestly superhuman – will not suffice to push AI capabilities further at any great speed.
Of course, if AI progress were to run out of steam at a level that is “merely” somewhat superhuman, the implications would still be profound; but they might fall well short of a singularity.
How To Tell Whether We’re Headed For A Singularity?
At this point, I don’t think it’s possible to say whether AI is headed for a positive feedback loop. I do think we can be fairly confident that we’re not yet on the cusp of that happening. A lot of work is needed before we can automate the majority of R&D work, and by the time get get there, various headwinds will be kicking in.
Here are some metrics we can monitor to get a sense of whether recursive self-improvement is headed for an upward spiral:
First and foremost, I would watch the pace of improvement vs. pace of R&D inputs. How rapidly are AI capabilities improving, in comparison with increases in R&D spending? Obviously, if we see a slowdown in capability improvements, that suggests that a takeoff spiral is either distant or not in the cards at all. However, ongoing progress that is only sustained by exponential growth in R&D investment would also constitute evidence against a takeoff spiral. The level of R&D spending will have to level off at some point, so if increased spending is the only way we’re sustaining AI progress, then recursive self-improvement is not setting the conditions for takeoff.
(This might be especially true so long as much of the investment is devoted to things that are hard to advance using AI, such as semiconductor manufacturing[9].)
Second, I would watch the level of work being done by AI. Today, I believe AIs are only being used for routine coding tasks, along with helping to generate or curate some forms of training data. If and when AI is able to move up to higher-level contributions, such as optimizing training algorithms or developing whole new network architectures, that would be a sign that we are about to see an effective explosion in the amount of cognitive input into AI development. That won’t necessarily lead to a takeoff spiral, but it is a necessary condition.
I would also watch the extent to which cutting-edge models depend on human-generated training data. Recursive self-improvement will never have a large impact until we transcend the need for human data.
Finally, I would keep an eye on the pace of improvement vs. pace of inference costs. As we develop more and more sophisticated AIs, do they cost more and more to operate, or are we able to keep costs down? Recursive self-improvement will work best if AIs are significantly cheaper than human researchers (and/or are intelligent enough to do things that people simply cannot do).
I think an AI takeoff is unlikely in the near to mid term. At a minimum, it will require AIs to be at least modestly superhuman at the critical tasks of AI research. Whether or not superhuman AI can achieve any sort of takeoff – and how far that takeoff can go before leveling out – will then depend on the trajectory of the metrics listed above.
- ^
I don’t know whether he still argues for the same “weeks or hours” timeline today.
- ^
I refuse to say “Meta”, that’s just encouraging them.
- ^
I’m not close to anyone at any of the cutting-edge AI research labs, so my opinion that tools like Copilot won’t have a big impact is based primarily on my own general experience as a software engineer. Contrarily, I’ve seen some folks opine that AI tools are already making a difference. For instance, back in April, Ajeya Cotra noted:
Today’s large language models (LLMs) like GPT-4 are not (yet) capable of completely taking over AI research by themselves — but they are able to write code, come up with ideas for ML experiments, and help troubleshoot bugs and other issues. Anecdotally, several ML researchers I know are starting to delegate simple tasks that come up in their research to these LLMs, and they say that makes them meaningfully more productive. (When chatGPT went down for 6 hours, I know of one ML researcher who postponed their coding tasks for 6 hours and worked on other things in the meantime.)
All I can really say is that I’m skeptical that the impact on the overall pace of progress is significant today, and I’d love to hear from practitioners who are experiencing otherwise so that I can update my understanding.
- ^
(I’m going to use some very handwavy numbers here; if you have better data, please let me know.)
According to untrustworthy numbers I googled, DeepMind and Google Brain have around 5000 employees (combined), and OpenAI has perhaps 1000. By the time we get to human-level AGI, let’s say that the organization which achieved it had 5000 people working directly or indirectly to support that goal (as opposed to, for instance, working on robotics or specialized AIs). Let’s say that another 5000 people from outside the organization are contributing in other ways, such as by publishing papers or contributing to open-source tools and data sets. So, roughly 10,000 talented people contributing to the first AI to cross the general-AI-researcher threshold. (I’m glossing over some other categories of contributors, such as the many folks in the chip industry who are involved in creating each new generation of GPU, associated software tools, high-speed networks, and other infrastructure.) Given the colossal stakes involved, this is probably a conservative estimate.
Let’s say that when human-level AI is available, one of the leading-edge organizations is able to run 200,000 instances, each running at 1x human speed. Here’s how I got there: ChatGPT reportedly has about 100,000,000 users, and OpenAI controls access to their various services in ways which suggest they are GPU-constrained. Let’s imagine that if they had human-level AI today, they’d manage to devote 2x as much computing power to AI research as they currently devote to ChatGPT. (Which is not 2x their total usage today, because ChatGPT does not include their various APIs, nor does it include the GPUs they use for training new models.) So that’s enough for 200,000,000 users. If each user accesses ChatGPT for an average of one minute per day, that’s enough for 140,000 simultaneous ChatGPT sessions. (One minute of generation per user per day might seem low, but I suspect it’s actually too high. ChatGPT can generate a lot of words in one minute, and average usage of large-scale Internet services is remarkably low. Most of those 100,000,000 users probably don’t touch the service at all on a typical day, and on days when they do use it, they may typically ask just a few questions. There will be some heavy users, but they’ll be a small fraction of the total.)
The first human-level AGI will likely use much more compute than ChatGPT, even with whatever algorithmic improvements we’ve made along the way. On the other hand, OpenAI – or whatever organization beats them to the punch – will have more compute by then. I’ll generously assume that these factors cancel out, meaning that they’ll be able to marshall 140,000 virtual AI researchers.
Those researchers will be working 24x7, so about four times more hours per year than a person. In my previous post, I also argued that they’ll get a productivity gain of “several” due to being more focused, having high-bandwidth communication with one another, and other factors. The upshot is that they might have the capacity of perhaps one million people. This paper, using entirely different methodology, arrives at a similar estimate: roughly 1,800,000 human-speed-equivalent virtual workers in 2030.
- ^
In We Aren’t Close To Creating A Rapidly Self-Improving AI, Jacob Buckman explores the difficulty of constructing high-quality data sets without relying on human input.
- ^
Moore’s Law is a special case of Wright’s Law. Originally applied to aircraft production, Wright’s Law stated that for each 2x increase in the cumulative number of aircraft produced, the labor time per aircraft fell by 20%. In other words, the 2000th aircraft produced required 20% less labor than the 1000th. Similar effects have been found across a wide variety of manufacturing domains; the general form is that when the number of items produced doubles, the cost decreases by X%, where X depends on the item being produced.
Moore’s Law, and more broadly Wright’s Law, of course are usually viewed as representing the inexorable march of progress. However, the flip side is that each improvement of X% requires twice the investment as the previous improvement. Eventually this must peter out, as we are finally seeing for Moore’s Law. As Scott Alexander notes in ACX:
Some research finds that the usual pattern in science is constant rate of discovery from exponentially increasing number of researchers, suggesting strong low-hanging fruit effects, but these seem to be overwhelmed by other considerations in AI right now.
Or Zvi Mowshowitz, in Don’t Worry About The Vase:
We are not seeing much in the way of lone wolf AI advances, we are more seeing companies full of experts like OpenAI and Anthropic that are doing the work and building up proprietary skill bundles. The costs to do such things going up in terms of compute and data and so on also contribute to this.
As models become larger, it becomes more expensive to experiment with full-scale training runs. And of course as models and their training process become more complex, it becomes more difficult to find improvements that don’t interfere with existing optimizations, and it requires more work to re-tune the system to accommodate a new approach. Not to mention the basic fact that the low-hanging fruit will already have been plucked.
- ^
From Why transformative artificial intelligence is really, really hard to achieve, some additional arguments (note the links) for the possibility of fundamental limits on effective intelligence:
We are constantly surprised in our day jobs as a journalist and AI researcher by how many questions do not have good answers on the internet or in books, but where some expert has a solid answer that they had not bothered to record. And in some cases, as with a master chef or LeBron James, they may not even be capable of making legible how they do what they do.
The idea that diffuse tacit knowledge is pervasive supports the hypothesis that there are diminishing returns to pure, centralized, cerebral intelligence. Some problems, like escaping game-theoretic quagmires or predicting the future, might be just too hard for brains alone, whether biological or artificial.
- ^
Such as chain-of-thought prompting, tree-of-thought prompting, or simply generating multiple responses and then asking the model to decide which one is best.
- ^
AI may certainly be able to find better techniques for manufacturing semiconductors, or even replace semiconductors with some alternative form of computational hardware. However, semiconductors are an extremely mature technology, starting to approach physical limits for the current form, meaning that it’s hard to find further improvements – as evidenced by fact that Moore’s Law is slowing down, despite record levels of investment into R&D. It’s always possible that AI will eventually help us develop some new approach to computing hardware, but I would expect this to be a long way off and/or require AI to already have advanced well beyond human intelligence. The upshot is that in almost any scenario, if AI progress is requiring ever-larger investments in computing hardware, that would suggest there is no “foom” in the offing anytime soon.
Of course, AI may be able to reduce our need for silicon, by coming up with better training algorithms that need fewer FLOPs, and/or better chip designs that achieve more FLOPs per transistor and per watt. However, so long as we see leading-edge AI labs spending ever-increasing amounts on silicon, that’s evidence that such “soft” improvements aren’t proving sufficient on their own.
Our best quantitative models of the effects of AGI on tech progress and econ growth suggest that the AI explosion will indeed happen. I’ve seen about a half-dozen models of varying quality; the best is Tom Davidson’s which you can access via takeoffspeeds.com.
OK, having read through much of the detailed report, here’s my best attempt to summarize my and Davidson’s opinions. I think they’re mostly compatible, but I am more conservative regarding the impact of RSI in particular, and takeoff speeds in general.
My attempt to summarize Davidson on recursive self-improvement
AI will probably be able to contribute to AI R&D (improvements to training algorithms, chip design, etc.) somewhat ahead of its contributions to the broader economy. Taking this into account, he predicts that the “takeoff time” (transition from “AI could readily automate 20% of cognitive tasks” to “AI could readily automate 100% of cognitive tasks”) will take a few years: median 2.9, 80% confidence interval 0.8 to 12.5.
He notes that the RSI feedback loop could converge or diverge:
He projects a 65% probability of scenario 1 (RSI leads to an accelerating capabilities ramp) occurring based strictly on software improvements, but thinks it would not last indefinitely:
Here, I believe he is only taking into account the contributions of AI to algorithm (software) improvements. Presumably, taking AI contributions to hardware design into account would produce a somewhat more aggressive estimate; this is part of the overall model, but I didn’t see it broken out into a specific probability estimate for a period of quicker-and-quicker doublings.
My position
I agree that RSI might or might not result in an accelerating capabilities ramp as we approach human-level AGI. I keep encountering people assuming that what Davidson calls a “software singularity” is self-evidently inevitable, and my main goal was to argue that, while possible, it is not inevitable. Davidson himself expresses a related sentiment that his model is less aggressive than some people’s stated expectations; for instance:
I have not attempted to produce a concrete estimate of takeoff speed.
I expect the impact of RSI to be somewhat less than Davidson models:
I expect that during the transition period, where humans and AIs are each making significant contributions to AI R&D, that there will be significant lags in taking full advantage of AI (project management and individual work habits will continually need to be adjusted), with resulting inefficiencies. Davidson touches on these ideas, but AFAICT does not include them in the model; for instance, “Assumes no lag in reallocating human talent when tasks have been automated.”
It may be that when AI has automated X% of human inputs into AI R&D, the remaining inputs are the most sophisticated part of the job, and can only be done by senior researchers, meaning that most of the people being freed up are not immediately able to be redirected to non-automated tasks. It might even be the case that, by abandoning the lower-level work, we (humans) would lose our grounding in the nuts and bolts of the field, and the quality of the higher-level work we are still doing might gradually decline.
I think of AI progress as being driven by a mix of cognitive input, training data, training FLOPs, and inference FLOPs. Davidson models the impact of cognitive input and inference FLOPs, but I didn’t see training data or training FLOPs taken into account. (“Doesn’t model data/environment inputs to AI development.”) My expectation that as RSI drives an increase in cognitive input, training data and training FLOPs will be a drag on progress. (Training FLOPs will be increasing, but not as quickly as cognitive inputs.)
I specifically expect progress to become more difficult as we approach human-level AGI, as human-generated training data will become less useful at that point. We will also be outrunning our existence proof for intelligence; I expect superhuman intelligence to be feasible, but we don’t know for certain that extreme superhuman performance is reasonably achievable, and so we should allow for some probability that progress beyond human performance will be significantly more difficult.
As we approach human-level AGI, we may encounter other complications: coordination problems and transition delays as the economy begins to evolve rapidly, increased security overhead as AI becomes increasingly strategic for both corporations and nations (and as risks hopefully are taken more seriously), etc.
Most of this can be summarized as “everything is always harder and always takes longer than you think, even when you take this into account”.
For the same reasons, I am somewhat more conservative than Davidson on the timeline from human-level AGI to superintelligence (which he guesses will take less than a year); but I’m not in a position to quantify that.
Davidson does note some of these possibilities. For instance, he cites a few factors that could result in superintelligence taking longer than a year (even though he does not expect that to be the case), including two of the factors I emphasize:
Strong-upvoted for thoughtful and careful engagement!
My own two cents on this issue: I basically accept Davidson’s model as our current-best-guess so to speak, though I acknowledge that things could be slower or faster for various reasons including the reasons you give.
I think it’s important to emphasize (a) that Davidson’s model is mostly about pre-AGI takeoff (20% automation to 100%) rather than post-AGI takeoff (100% to superintelligence) but it strongly suggests that the latter will be very fast (relative to what most people naively expect) on the order of weeks probably and very likely less than a year. To see this, play around with takeoffspeeds.com and look at the slope of the green line after AGI is achieved. It’s hard not to have it crossing several OOMs in a single year, until it starts to asymptote. i.e. in a single year we get several OOMs of software/algorithms improvement over AGI. There is no definition of superintelligence in the model, but I use that as a proxy. (Oh, and now that I think about it more, I’d guess that Davidson’s model significantly underestimates the speed of post-AGI takeoff, because it might just treat anything above AGI as merely 100% automation, whereas actually there are different degrees of 100% automation corresponding to different levels of quality intelligence; 100% automation by ASI will be significantly more research-oomph than 100% automation by AGI. But I’d need to reread the model to decide whether this is true or not. You’ve read it recently, what do you think?)
And (b) Davidson’s model says that while there is significant uncertainty over how fast takeoff will be if it happens in the 30′s or beyond, if it happens in the 20′s—i.e. if AGI is achieved in the 20′s—then it’s pretty much gotta be pretty fast. Again this can be seen by playing around with the widget on takeoffspeeds.com
...
Other cents from me:
--I work at OpenAI and I see how the sausage gets made. Already things like Copilot and ChatGPT are (barely, but noticeably) accelerating AI R&D. I can see a clear path to automating more and more parts of the research process, and my estimate is that going 10x faster is something like a lower bound on what would happen if we had AGI (e.g. if AutoGPT worked well enough that we could basically use it as a virtual engineer + scientist) and my central estimate would be “it’s probably about 10x when we first reach AGI, but then it quickly becomes 100x, 1000x, etc. as qualitative improvements kick in.” There’s a related issue of how much ‘room to grow’ is there, i.e. how much low-hanging fruit is there to pick that would improve our algorithms, supposing we started from something like “It’s AutoGPT but good, as good as an OAI employee.” My answer is “Several OOMs at least.” So my nose-to-the-ground impression is if anything more bullish/fast-takeoff-y than Davidson’s model predicts.
--I do agree with the maxim “everything is always harder and always takes longer than you think, even when you take this into account”. In fact I’ve been mildly surprised by this in the recent past (things took longer than I expected, even though I was trying to take it into account). This gives me some hope. I have more to say on the subject but I’ve rambled for long enough...
...
Point by point replies:
Agreed. This is part of the reason why I think Davidson’s model overestimates the speed at which AI will influence the economy in general, and the chip industry in particular. I think the AI industry will be accelerated far before the chip industry or the general economy, however; we’ll probably get a “software singularity.” And unfortunately that’s ‘good enough’ from an AI-risk perspective, because to a first approximation what matters is how smart the AIs are and whether they are aligned, not how many robot bodies they control.
Agreed. I would be interested to see a revised model in which humans whose jobs are automated basically don’t get reallocated at all. I don’t think the bottom-line conclusions of the model would change much, but I could be wrong—if takeoff is significantly slower, that would be an update for me.
Training FLOPs is literally the most important and prominent variable in the model, it’s the “AGI training requirements” variable. I agree that possible data bottlenecks are ignored; if it turns out that data is the bottleneck, timelines to AGI will be longer (and possibly takeoff slower? Depends on how the data problem eventually gets solved; takeoff could be faster in some scenarios...) Personally I don’t think the data bottleneck will slow us down much, but I could be wrong.
I think I disagree here. I mean, I technically agree that since most of our data is human-generated, there’s going to be some headwind at getting to superhuman performance. But I think this headwind will be pretty mild, and also, to get to AGI we just need to get to human-level performance not superhuman. Also I’m pretty darn confident that extreme superhuman performance is reasonably achievable; I think there is basically no justification for thinking otherwise. (It’s not like evolution explored making even smarter humans and found that they didn’t get smarter beyond a certain point and we’re at that point now. Also, scaling laws. Also, the upper limits of human performance might as well be superintelligence for practical purposes—something that is exactly as good as the best human at X, for all X, except that it also thinks at 100x speed and there are a million copies of it that share memories all working together in a sort of virtual civilization… I’d go out on a limb and say that’s ASI.)
Yeah I agree here. I think the main bottleneck to takeoff speed will be humans deliberately going less fast than they could for various reasons, partly just stupid red tape and overcaution, and partly correct realization that going fast is dangerous. Tom’s model basically doesn’t model this. I think of Tom’s model as something like “how fast could we go if we weren’t trying to slow down at all and in fact were racing hard against each other.” In real life I dearly hope the powerful CEOs and politicians in the world will be more sane than that.
Thanks for the thoughtful and detailed comments! I’ll respond to a few points, otherwise in general I’m just nodding in agreement.
And it’s a good model, so we need to take this seriously. My only quibble would be to raise again the possibility (only a possibility!) that progress becomes more difficult around the point where we reach AGI, because that is the point where we’d be outgrowing human training data. I haven’t tried to play with the model and see whether that would significantly affect the post-AGI takeoff timeline.
I want to say that he models this by equating the contribution of one ASI to more than one AGI, i.e. treating additional intelligence as equivalent to a speed boost. But I could be mis-remembering, and I certainly don’t remember how he translates intelligence into speed. If it’s just that each post-AGI factor of two in algorithm / silicon improvements is modeled as yielding twice as many AGIs per dollar, then I’d agree that might be an underestimate (because one IQ 300 AI might be worth a very large number of IQ 150 AIs, or whatever).
Yeah, even without consulting any models, I would expect that any scenario where we achieve AGI in the 20s is a very scary scenario for many reasons.
What is your feeling regarding the importance of other inputs, i.e. training data and compute?
Ugh! This was a big miss on my part, thank you for calling it out. I skimmed too rapidly through the introduction. I saw references to biological anchors and I think I assumed that meant the model was starting from an estimate of FLOPS performed by the brain (i.e. during “inference”) and projecting when the combination of more-efficient algorithms and larger FLOPS budgets (due to more $$$ plus better hardware) would cross that threshold. But on re-read, of course you are correct and the model does focus on training FLOPS.
Sounds like we are basically on the same page!
Re: your question:
Compute is a very important input, important enough that it makes sense IMO to use it as the currency by which we measure the other inputs (this is basically what Bio Anchors + Tom’s model do).
There is a question of whether we’ll be bottlenecked on it in a way that throttles takeoff; it may not matter if you have AGI, if the only way to get AGI+ is to wait for another even bigger training run to complete.
I think in some sense we will indeed be bottlenecked by compute during takeoff… but that nevertheless we’ll be going something like 10x − 1000x faster than we currently go, because labor can substitute for compute to some extent (Not so much if it’s going at 1x speed; but very much if it’s going at 10x, 100x speed) and we’ll have a LOT of sped-up labor. Like, I do a little exercise where I think about what my coworkers are doing and I imagine what if they had access to AGI that was exactly as good as they are at everything, only 100x faster. I feel like they’d make progress on their current research agendas about 10x as fast. Could be a bit less, could be a lot more. Especially once we start getting qualitative intelligence improvements over typical OAI researchers, it could be a LOT more, because in scientific research there seems to be HUGE returns to quality, the smartest geniuses seem to accomplish more in a year than 90th-percentile scientists accomplish in their lifetime.
Training data also might be a bottleneck. However I think that by the time we are about to hit AGI and/or just having hit AGI, it won’t be. Smart humans are able to generate their own training data, so to speak; the entire field of mathematics is a bunch of people talking to each other and iteratively adding proofs to the blockchain so to speak and learning from each other’s proofs. That’s just an example, I think, of how around AGI we should basically have a self-sustaining civilization of AGIs talking to each other and evaluating each other’s outputs and learning from them. And this is just one of several ways in which training data bottleneck could be overcome. Another is better algorithms that are more data-efficient. The human brain seems to be more data-efficient than modern LLMs, for example. Maybe we can figure out how it manages that.
Thanks. I had seen Davidson’s model, it’s a nice piece of work. I had not previously read it closely enough to note that he does discuss the question of whether RSI is likely to converge or diverge, but I see that now. For instance (emphasis added):
I’ll need some time to thoroughly digest what he has to say on this topic.
Nice model!
I notice that the rate of hardware rollout dC is a constant (preset: 1 year). The discussion about this constant in the documentation is not clear, but to me it seems one thing an AGI would optimize hard against as this is limiting the takeoff speed.
Eventual failure aside, the East India Company gave it a damn good shake. I think if we get an AI to the point where it has effective colonial control over entire countries, we can be squarely said to have lost.
Also keep in mind that we have multiple institutions entirely dedicated to the purpose of breaking up companies when they become big enough to be threatening. We designed our societies to specifically avoid this scenario! That, too, comes from painful experience. IMO, if we now give AI the chances that we’ve historically given corporations before we learnt better, then we’re dead, no question about it.
I think you basically ignore the existing wisdom of what limits the size of firms and try to explain the limits of the size of companies with a model that doesn’t tell us very much about how companies work.
We have antitrust laws. There’s the Innovator’s Dilemma as described by Clayton Christensen that explains why companies decide against doing certain business. Markets often outperform hierarchical decision-making. Uber could be a lot bigger if they would employ all their drivers and own all the vehicles but they rather not do that part of the business and use market dynamics.
Uber would be a lot bigger if they would employ all the drivers as employees. Managing people is often adds inefficiencies. The more layers of management you have in an organization the worse the incentive alignment happens to be.
If you add a bunch of junior programs into a software project it might very well slow the project down because it takes effort for the more experienced programmers to manage the junior programmers. GitHub Copilot on the other hand makes an experienced programmer more productive without adding friction about managing junior employees.
We can look at a skill that’s about applying human intelligence like playing Go. It would be possible that the maximum skill level is near what professional go players are able to accomplish. AlphaGo managed to go very much past what humans can accomplish in a very short timeframe and AlphaGo doesn’t even do any self-recursive editing of it’s own code.
AI can help with producing GPU’s as well. It’s possible to direct a lot more of the worlds economic output into producing GPU’s than is currently done.
Sure, it’s easy to imagine scenarios where a specific given company could be larger than it is today. But are you envisioning that if we eliminated antitrust laws and made a few other specific changes, then it would become plausible for a single company to take over the entire economy?
My thesis boils down to the simple assertion that feedback loops need not diverge indefinitely, exponential growth can resolve into an S-curve. In the case of a corporation, the technological advantages, company culture, and other factors that allow a company to thrive in one domain (e.g. Google, web search) might not serve it well in another domain (Google, social networks). In the case of AI self-improvement, it might turn out that we eventually enter a domain – for instance, the point where we’ve exhausted human-generated training data – where the cognitive effort required to push capabilities forwards increases faster than the cognitive effort supplied by those same capabilities. In other words, we might reach a point where each successive generation of recursively-designed AI delivers a decreasing improvement over its predecessor. Note that I don’t claim this is guaranteed to happen, I merely argue that it is possible, but that seems to be enough of a claim to be controversial.
Certainly. I think we see that the ease with which computers can definitively surpass humans depends on the domain. For multiplying large numbers, it’s no contest at all. For Go, computers win definitively, but by a smaller margin than for multiplication. Perhaps, as we move toward more and more complex and open-ended problems, it will get harder and harder to leave humans in the dust? (Not impossible, just harder?) I discuss this briefly in a recent blog post, I’d love to hear thoughts / evidence in either direction.
Sure. I’m just suggesting that the self-improvement feedback loop would be slower here, because designing and deploying a new generation of fab equipment has a much longer cycle time than training a new model, no?
A key issue with training AIs for open-ended problems is that’s a lot harder to create good training data for open-ended problems then it is to create high-quality training data for a game with clear rules.
It’s worth noting that one of the problems where humans outperform computers right now are not really the open-ended tasks but things like how to fold laundry.
A key difference between playing go well and being able to fold laundry well is that training data is easier to come by for go.
If you look at the quality that a lot of professionals make when it comes to a lot of decisions involving probability (meaning there’s a lot of uncertainty) they are pretty bad.
You don’t need a new generation of fab equipment to make advances in GPU design. A lot of improvements of the last few years were not about having constantly a new generation of fab equipment.
Ah, by “producing GPUs” I thought you meant physical manufacturing. Yes, there has been rapid progress of late in getting more FLOPs per transistor for training and inference workloads, and yes, RSI will presumably have an impact here. The cycle time would still be slower than for software: an improved model can be immediately deployed to all existing GPUs, while an improved GPU design only impacts chips produced in the future.
Yes, that’s not just about new generations of fab equipment.
GPU performance for training models did increase faster than Moore’s law over the last decade. It’s not something where the curve of improvement is slow even without AI.
I think it’s important to note that a single “human level” AI system can have access to human-level competency in every human cognitive skill and domain simultaneously, and can have read/heard/seen everything humanity has ever output, which is something no biological human has. Whatever limit we approach as the limit of feedback loops, a true omnimath is a very loose lower bound on what human-level AI can achieve even with no self-improvement and no speed advantage. Consider in the context of human polymaths who manage to revolutionize many fields in the course of a single lifetime.
In practice I think we should expect a considerable speed advantage and high number of instances, if not immediately then very soon after whoever develops the system demonstrates its value. Plus, perfect recall and never losing attention/focus and extremely high input/output data compared to human senses seem trivial to add if they’re not there by default.
In other words, individually-human-level thinking skills and levels of knowledge imply significantly superhuman capabilities in an AGI before we even ask the question of how high the limit of fast-self-improvement will be. And then, of course, there’s the question of how superhuman something needs to actually be to present significant x-risk. My take is that this does not require much self-improvement at all, and that a very fast-thinking and internally-coordinated collective of polymathic agents with many individually-human-level capabilties is sufficiently x-risky to be worried about.
The reason I think this is that I believe there are an enormous number of ways to make better use of the research data and experimentally demonstrated capabilities humans already have, if we could instantly combine the relevant skills from different fields in a single individual as needed.
For example, you mention eking out a few more years of Moore’s law by advancing EUV lithography. But, I would be extremely surprised if a human with all the PhDs and a multi-GHz brain couldn’t work out how to produce optical or ultrasonic computing metasurfaces, on already existing not-even-very-new fab equipment, based on variants and combinations of already proposed designs for things like diffractive neural nets, which would operate several orders of magnitude faster than the highest speed electronic chips. In other words, I see “take what humans already know and have the equipment to build today, and use it to speed themselves up by 1000x-100,000x” as a lower bound on self-improvement without any increase in quality of thought compared to humans.
All of these things are possible, but it’s not clear to me that they’re likely, at least in the early stages of AGI. In other words: once we have significantly-superhuman AGI, then agreed, all sorts of crazy things may become possible. But first we have to somehow achieve superhuman AGI. One of the things I’m trying to do in this post is explore the path that gets us to superhuman AGI in the first place. That path, by definition, can’t rely on anything that requires superhuman capabilities.
If I understand correctly, you’re envisioning that we will be able to construct AGIs that have human-level capability, and far greater than human speed, in order to bootstrap superhuman AGI? What makes you confident that this speed advantage will exist early on? Current leading-edge models like GPT-4 are not drastically faster than human beings, and presumably until we get to human-level AGI we’ll be spending most of our algorithmic improvements and increases in FLOP budget on increased capabilities, rather than performance. In fact, it’s quite possible that we’ll have to (temporarily) accept reduced speed in order to achieve human-level performance; for instance, by introducing tree search into the thought process (tree-of-thought prompting, heuristic search techniques in Yann LeCun’s “A Path Towards Autonomous Machine Intelligence”, etc.).
Once we achieve human-level, human-speed AGI, then yes, further algorithm or FLOPs improvements could be spent on speed; this comes back to the basic question of how whether the cognitive effort required for further progress increases more or less rapidly than the extent to which progress (and/or increased budgets) enables increased cognitive effort, i.e. does the self-improvement feedback loop converge or diverge. Are you proposing that it definitely diverges? What points you in that direction?
I would also caution against being highly confident that AGI will automatically be some sort of ideal omnimath. Such ability would require more than merely assimilating all of human knowledge and abilities; it would require knowing exactly which sub-specialties to draw on in any given moment. Otherwise, the AI would risk drowning in millions of superfluous connections to its every thought. Some examples of human genius might in part depend on a particular individual just happening to have precisely the right combination of knowledge, without a lot of other superfluous considerations to distract them.
Also, is it obvious that a single early-human-level AI could be trained with deep mastery of every field of human knowledge? Biological-anchor analysis aims to project when we can create a human-level AI, and humans are not omnimaths. Deep expertise across every subspeciality might easily require many more parameters than the number of synapses in the human brain. Many things look simple until you try to implement them; I touch on this in a recent blog post, The AI Progress Paradox, but you just need to look at the history of self-driving cars (or photorealistic CGI, or many other examples) to see how things that seem simple in principle can require many rounds of iteration to fully achieve in practice.
What I’m trying to say is that even at human speed, being able to mix-and-match human-level capabilities at will, in arbitrary combinations, not an ideal omnimath but in larger numbers than a single human can accumulate, is already a superhuman ability and one I expect AGI to trivially possess. Then on top of that you get, for free, things like being able to coordinate multiple instances of a single entity that don’t have their own other agendas, and that never lose focus or get tired.
Since you did mention genius coming from “precisely the right combination of knowledge, without a lot of other superfluous considerations to distract them,” I have to ask… doesn’t AGI seem perfectly positioned to be just that, for any combination of knowledge you can train it on?
I also don’t find the biological anchors argument convincing, for somewhat the same reason: an AI doesn’t need all of the superfluous knowledge a human has. Some of it, yes, but not all of it. To put it another way, in terms of data and parameters, how much knowledge of physics does a physicist actually have after a long career? A basic world model like all humans acquire in childhood, plus a few hundred books, a few thousand hours of lectures, and maybe 40k hours of sensory data acquired and thinking completed on-the-job?
And you’re right, I agree an early AGI won’t be an omnimath, but I think polymath is very much within reach.
Serial speed advantage of AI is sufficient to deliver an ancient AI civilization within a few years of the first AGI even without superintelligence or significant algorithmic self-improvement, simply by AIs thinking faster than humans, and using this advantage to quickly develop hardware that allows them to think even faster. No need to posit exponential growth when just the first two steps let you start running past the competition at 10,000 times their speed.
How much of the work to create better hardware can be done in a computer’s head, though? I have no doubt that smarter being can create better hardware than we have now, but are there other real world limitations that would very quickly limit the rate of improvement. I imagine even something much smarter than us would need to experiment in the physical world, as well as build new machines (and mine the necessary materials) and do a lot of actual physical work that would take time and that computers can not obviously do 10,000x faster than humans.
In this scenario AGIs are unrestricted in their activity, so in particular they can do physical experiments if that turns out to be useful. Manufacturing of improved hardware at scale requires development of physical tools anyway, so it’s a step along the way.
The starting point is likely biotechnology, with enough mastery to design novel organisms. Think drosophila swarms, not whales, but with an option to assemble into whales. With enough energy, this gives both exponential scaling and control at small scale, which is good for a large number of diverse experiments that run very quickly. Macroscale biotechnology replaces human physical infrastructure, both crude manipulation of matter (logistics, assembly) and chemical processing. More subtle objects like chips and fusion plants could be manufactured with specialized non-biological machines, the same way humans do such things, but backed by the exponential biological infrastructure and enough planning to get everything working right away. If diamondoid nanotech turns out to be feasible, this works even better, but it doesn’t have to.
(Of course, this is all under the absolutely impossible assumption of lack of superintelligence. So exploratory engineering, not prediction. A lower bound on what seems feasible, even if it never becomes worthwhile to do in a remotely similar way.)
Somewhat nitpicking
I would argue it has. Fooms have a sort of relativistic element, where being inside a foom does not feel special. Just because history is running millions of times faster than before, doesn’t really feel like anything.
With all of that said, what is and isn’t a foom is somewhat blurry at the edges, but I’d argue that biology, brains, and farming all qualify. Conversely, that more has happened in the last couple centuries than the previous couple eons. Of course, this claim is heavily dependent on the definition of “things happening”, in terms of say, mass moved, none of this has mattered at all, but in terms of, things mattering, the gap seems nigh infinite.
Looking at the world from a perspective where fooms have happened, in fact, multiple times, doesn’t give me confidence that fooms just aren’t physically something that’s allowed.
So to be clear, I am not suggesting that a foom is impossible. The title of the post contains the phrase “might never happen”.
I guess you might reasonably argue that, from the perspective of (say) a person living 20,000 years ago, modern life does in fact sit on the far side of a singularity. When I see the word ‘singularity’, I think of the classic Peace War usage of technology spiraling to effectively infinity, or at least far beyond present-day technology. I suppose that led me to be a bit sloppy in my use of the term.
The point I was trying to make by referencing those various historical events is that all of the feedback loops in question petered out short of a Vingian singularity. And it’s a fair correction that some of those loops are actually still in play. But many are not – forest fires burn out, the Cambrian explosion stopped exploding – so we do have existence proofs that feedback loops can come to a halt. I know that’s not any big revelation, I was merely attempting to bring the concept to mind in the context of RSI.
In any case, all I’m really trying to do is to argue that the following syllogism is invalid:
As AI approaches human level, it will be able to contribute to AI R&D, thus increasing the pace of AI improvement.
This process can be repeated indefinitely.
Therefore, as soon as AI is able to meaningfully contribute to its own development, we will quickly spiral to a Vingian singularity.
This scenario is certainly plausible, but I frequently see it treated as a mathematical certainty. And that is simply not the case. The improvement cycle will only exhibit a rapid upward spiral under certain assumptions regarding the relationship of R&D inputs to gains in AI capability – the
rterm in Davidson’s model.(Then I spend some time explaining why I think
rmight be lower than expected during the period where AI is passing through human level. Again, “might be”.)I don’t think that there has been foom related to biology or brains. Earth is 4.5 billion years old. Single-celled life has existed about 3.5 billion years. Multicellular life is about 1.5 billion years old. Animals with brains have existed about 500 million years. This is not a foom timeline of events.
And the animals with relevant brains for foom are <0.01 billion years old, and their population only started becoming noticeably larger than competitors about <0.0001 billion years ago, and they only started doing really impressive things like ‘land on the Moon’ <0.0000001 billion years ago. This is a foom timeline of events.
I’ll try to summarize your point, as I understand it:
Intelligence is just one of many components. If you get huge amounts of intelligence, at that point you will be bottlenecked by something else, and even more intelligence will not help you significantly. (Company R&D doesn’t bring a “research explosion”.)
I’ll start with the analogy to company R&D.
Please note that if you use “humankind” instead of “company”, and look at historical timescale, investing into R&D of humankind actually has brought us exponential growth during the recent centuries. (Who knows, we still might colonize the universe.) So the question is, why doesn’t the same effect work for a company?
I think the answer is that R&D of even the richest companies is just a tiny fraction of the overall R&D of humankind (including historically). Even companies that do very impressive research are basically just adding the last step to a very long chain of research that happened somewhere else. As an analogy, imagine a sailor on one of Columbus’ ships, who would jump into a boat 100 meters before reaching the shore of America, row fast, and then take historical credit for technically getting to America first. From historical perspective, if the entire humanity spends millennia researching physics, and then you take a few years and invent a microwave oven, it’s the same thing. If you wouldn’t do it, someone else probably would, a few years or decades later. We have historical examples of inventions that were made independently by multiple people in the same year, but only the one who got to the patent office first gets the credit. Even today, we have several companies inventing AI in parallel, because the science and technology are already in place, and they just need to take the last step. (If the Moore’s law keeps working, a few decades later a gifted student could do the same thing on their home computer.)
So I think that the problem with a company that spends a lot on R&D is that the things they have researched today can give them an advantage today… but not tomorrow, because the world catches up. Yeah, thanks to the “intellectual property” system, the rest of the world may not be allowed to use the same invention, no matter how many people are now capable to make the same invention independently. But still, the rest of the world will invent thousands of other things, and to remain competitive, it is necessary for the company to study those other things, but they have no advantage over their competitors there.
As a thought experiment, what would it actually look like, for a company, to do 1% of humanity’s R&D? I think it would be like running a small first-world country—its economy and educational system, all in service of the company needs. But still, the rest of the world would keep going on. Also, it would be difficult to keep everything invented in your country as a company secret.
As a crazier thought experiment, imagine that the rest of the world is not going on. Imagine that a group of scientists and soldiers get teleported a few millennia to the past, they establish their own empire (of a size of a smaller country), and start doing research. The rest of the world has a religious taboo against research, but within their empire, people are brainwashed to worship science. I could imagine them taking over the world.
So I think the proper conclusion is not “intelligence is not enough to take over the world” but rather “until now, everyone’s intelligence was just a fraction of humanity’s intelligence, and also the discoveries leak to your competitors”. A company can keep its secrets, but is too small. A country is large enough, but can’t keep secret the things it teaches at public schools.
Also, please note that LLMs are just one possible paradigm of AI. Yes, currently the best one, but who knows what tomorrow may bring. I think most people among AI doomers would agree that LLMs are not the kind of AI they fear. LLMs succeed to piggyback on humanity’s written output, but they are also bottlenecked by it.
Then you have things like the chess and go playing machines, which can easily surpass humanity, but are too narrow.
The greatest danger is if someone invents an AI that it neither narrow nor bottlenecked by human output. Something that can think as efficiently as the chess machines, but about everything that humans think about.
The core idea I’m trying to propose (but seem to have communicated poorly) is that the AI self-improvement feedback loop might (at some point) converge, rather than diverging. In very crude terms, suppose that GPT-8 has IQ 180, and we use ten million instances of it to design GPT-9, then perhaps we get a system with IQ 190. Then we use ten million instances of GPT-9 to design GPT-10, perhaps that has IQ 195, and eventually GPT-∞ converges at IQ 200.
I do not claim this is inevitable, merely that it seems possible, or at any rate is not ruled out by any mathematical principle. It comes down to an empirical question of how much incremental R&D effort is needed to achieve each incremental increase in AI capability.
The point about the possibility of bottlenecks other than intelligence feeds into that question about R&D effort vs. increase in capability; if we double R&D effort but are bottlenecked on, say, training data, than we might get a disappointing increase in capability.
IIUC, much of the argument you’re making here is that the existing dynamic of IP laws, employee churn, etc. puts a limit on the amount of R&D investment that any given company is willing to make, and that these incentives might soon shift in a way that could unleash a drastic increase in AI R&D spending? That seems plausible, but I don’t see how it ultimately changes the slope of the feedback loop – it merely allows for a boost up the early part of the curve?
Agreed that there’s a very good chance that AGI may not look all that much like an LLM. And so when we contemplate the outcome of recursive self-improvement, a key question will be what the R&D vs. increase-in-capability curve looks like for whatever architecture emerges.
I agree that the AI cannot improve literally forever. At some moment it will hit a limit, even if that limit is that it became near perfect already, so there is nothing to improve, or the tiny remaining improvements would not be worth their cost in resources. So, S-curve it is, in long term.
But for practical purposes, the bottom part of the S-curve looks similar to the exponential function. So if we happen to be near that bottom, it doesn’t matter that the AI will hit some fundamental limit on self-improvement around 2200 AD, if it already successfully wiped out humanity in 2045.
So the question is in which part of the S-curve we are now, and whether the AI explosion hits diminishing returns soon enough, i.e. before the things AI doomers are afraid of could happen. If it happens later, that is a small consolation.
You might be interested in the “anthropics” of knowing how-close-we-are beforehand.
I think you’re saying that the fact that no historical feedback loop has ever destroyed the Earth (nor transformed it into a state which would not support human life) could be explained by the Anthropic Principle? Sure, that’s true enough. I was aiming more to provide an intuition for the idea that it’s very common and normal for feedback loops to eventually reach a limit, as there are many examples in the historical record.
Intuition aside: given the sheer number of historical feedback loops that have failed to destroy the Earth, it seems unavoidable that either (a) there are some fundamental principles at play that tend to place a cap on feedback loops, at least in the family of alternative universes that this universe has been sampled from, or (b) we have to lean on the Anthropic Principle very very hard indeed. It’s not hard to articulate causes for (a); for instance, any given feedback loop arises under a particular set of conditions, and once it has progressed sufficiently, it will begin to alter its own environment to the point where those conditions may no longer apply. (The forest fire consumes all available fuel, etc.)
I think “feedback loops have a cap” is a much easier claim to defend than the implied “AI feedback loops will cap out before they can hurt humanity at an x-risk level”. That second one is especially hard to defend if e.g. general-intelligence abilities + computational speed lets the AI develop some other thing (like a really bad plague) that can hurt humanity at an x-risk level. Intelligence, itself, can figure out, harness, and accelerate the other feedback loops.
Sorry! I’m responding because you did say feedback was appeciated. Did you do a search before posting? Like with most topic-dedicated discussion spots, this topic has a lot of existing discussion. I stopped reading after two paragraphs with no mention of previous posts or articles or anything on the topic. The topic of speed of self-improvekebt is super important, but if you haven’t read anything, the odds of you having important new thoughts seem low.
It’s a long post and there’s a lot of important stuff to read.
Thanks, I appreciate the feedback. I originally wrote this piece for a less technical audience, for whom I try to write articles that are self-contained. It’s a good point that if I’m going to post here, I should take a different approach.