Software engineer and repeat startup founder; best known for Writely (aka Google Docs). Now blogging at https://amistrongeryet.substack.com and looking for ways to promote positive outcomes from AI.
snewman
Karma: 124
I am trying to wrap my head around the high-level implications of this statement. I can come up with two interpretations:
What LLMs are doing is similar to what people do as they go about their day. When I walk down the street, I am simultaneously using visual and other input to assess the state of the world around me (“that looks like a car”), running a world model based on that assessment (“the car is coming this way”), and then using some other internal mechanism to decide what to do (“I’d better move to the sidewalk”).
What LLMs are doing is harder than what people do. When I converse with someone, I have some internal state, and I run some process in my head – based on that state – to generate my side of the conversation. When an LLM converses with someone, instead of maintaining internal state, needs to maintain a probability distribution over possible states, make next-token predictions according to that distribution, and simultaneously update the distribution.
(2) seems more technically correct, but my intuition dislikes the conclusion, for reasons I am struggling to articulate. …aha, I think this may be what is bothering me: I have glossed over the distinction between input and output tokens. When an LLM is processing input tokens, it is working to synchronize its state to the state of the generator. Once it switches to output mode, there is no functional benefit to continuing to synchronize state (what is it synchronizing to?), so ideally we’d move to a simpler neural net that does not carry the weight of needing to maintain and update a probability distribution of possible states. (Glossing over the fact that LLMs as used in practice sometimes need to repeatedly transition between input and output modes.) LLMs need the capability to ease themselves into any conversation without knowing the complete history of the participant they are emulating, while people have (in principle) access to their own complete history and so don’t need to be able to jump into a random point in their life and synchronize state on the fly.
So the implication is that the computational task faced by an LLM which can emulate Einstein is harder than the computational task of being Einstein… is that right? If so, that in turn leads to the question of whether there are alternative modalities for AI which have the advantages of LLMs (lots of high-quality training data) but don’t impose this extra burden. It also raises the question of how substantial this burden is in practice, in particular for leading-edge models.
All of this is plausible, but I’d encourage you to go through the exercise of working out these ideas in more detail. It’d be interesting reading and you might encounter some surprises / discover some things along the way.
Note, for example, that the AGIs would be unlikely to focus on AI research and self-improvement if there were more economically valuable things for them to be doing, and if (very plausibly!) there were not more economically valuable things for them to be doing, why wouldn’t a big chunk of the 8 billion humans have been working on AI research already (such that an additional 1.6 million agents working on this might not be an immediate game changer)? There might be good arguments to be made that the AGIs would make an important difference, but I think it’s worth spelling them out.
Can you elaborate? This might be true but I don’t think it’s self-evidently obvious.
In fact it could in some ways be a disadvantage; as Cole Wyeth notes in a separate top-level comment, “There are probably substantial gains from diversity among humans”. 1.6 million identical twins might all share certain weaknesses or blind spots.
Assuming we require a performance of 40 tokens/s, the training cluster can run concurrent instances of the resulting 70B model
Nit: you mixed up 30 and 40 here (should both be 30 or both be 40).
I will assume that the above ratios hold for an AGI level model.
If you train a model with 10x as many parameters, but use the same training data, then it will cost 10x as much to train and 10x as much to operate, so the ratios will hold.
In practice, I believe it is universal to use more training data when training larger models? Implying that the ratio would actually increase (which further supports your thesis).
On the other hand, the world already contains over 8 billion human intelligences. So I think you are assuming that a few million AGIs, possibly running at several times human speed (and able to work 24⁄7, exchange information electronically, etc.), will be able to significantly “outcompete” (in some fashion) 8 billion humans? This seems worth further exploration / justification.
They do mention a justification for the restrictions – “to maintain consistency across cells”. One needn’t agree with the approach, but it seems at least to be within the realm of reasonable tradeoffs.
Nowadays of course textbooks are generally available online as well. They don’t indicate whether paid materials are within scope, but of course that would be a question for paper textbooks as well.
What I like about this study is that the teams are investing a relatively large amount of effort (“Each team was given a limit of seven calendar weeks and no more than 80 hours of red-teaming effort per member”), which seems much more realistic than brief attempts to get an LLM to answer a specific question. And of course they’re comparing against a baseline of folks who still have Internet access.
I recently encountered a study which appears aimed at producing a more rigorous answer to the question of how much use current LLMs would be in abetting a biological attack: https://www.rand.org/pubs/research_reports/RRA2977-1.html. This is still work in progress, they do not yet have results. @1a3orn I’m curious what you think of the methodology?
Imagine someone offers you an extremely high-paying job. Unfortunately, the job involves something you find morally repulsive – say, child trafficking. But the recruiter offers you a pill that will rewrite your brain chemistry so that you’ll no longer find it repulsive. Would you take the pill?
I think that pill would reasonably be categorized as “updating your goals”. If you take it, you can then accept the lucrative job and presumably you’ll be well positioned to satisfy your new/remaining goals, i.e. you’ll be “happy”. But you’d be acting against your pre-pill goal (I am glossing over exactly what that goal is, perhaps “not harming children” although I’m sure there’s more to unpack here).
I pose this example in an attempt to get at the heart of “distinguishing between terminal and instrumental goals” as suggested by quetzal_rainbow. This is also my intuition, that it’s a question of terminal vs. instrumental goals.
Likewise, thanks for the thoughtful and detailed response. (And I hope you aren’t too impacted by current events...)
I agree that if no progress is made on long-term memory and iterative/exploratory work processes, we won’t have AGI. My position is that we are already seeing significant progress in these dimensions and that we will see more significant progress in the next 1-3 years. (If 4 years from now we haven’t seen such progress I’ll admit I was totally wrong about something). Maybe part of the disagreement between us is that the stuff you think are mere hacky workarounds, I think might work sufficiently well (with a few years of tinkering and experimentation perhaps).
Wanna make some predictions we could bet on? Some AI capability I expect to see in the next 3 years that you expect to not see?Sure, that’d be fun, and seems like about the only reasonable next step on this branch of the conversation. Setting good prediction targets is difficult, and as it happens I just blogged about this. Off the top of my head, predictions could be around the ability of a coding AI to work independently over an extended period of time (at which point, it is arguably an “engineering AI”). Two different ways of framing it:
An AI coding assistant can independently complete 80% of real-world tasks that would take X amount of time for a reasonably skilled engineer who is already familiar with the general subject matter and the project/codebase to which the task applies.
An AI coding assistant can usefully operate independently for X amount of time, i.e. it is often productive to assign it a task and allow it to process for X time before checking in on it.
At first glance, (1) strikes me as a better, less-ambiguous framing. Of course it becomes dramatically more or less ambitious depending on X, also the 80% could be tweaked but I think this is less interesting (low percentages allow for a fluky, unreliable AI to pass the test; very high percentages seem likely to require superhuman performance in a way that is not relevant to what we’re trying to measure here).
It would be nice to have some prediction targets that more directly get at long-term memory and iterative/exploratory work processes, but as I discuss in the blog post, I don’t know how to construct such a target – open to suggestions.
Coding, in the sense that GPT4 can do it, is nowhere near the top of the hierarchy of skills involved in serious software engineering. And so I believe this is a bit like saying that, because a certain robot is already pretty decent at chiseling, it will soon be able to produce works of art at the same level as any human sculptor.
I think I just don’t buy this. I work at OpenAI R&D. I see how the sausage gets made. I’m not saying the whole sausage is coding, I’m saying a significant part of it is, and moreover that many of the bits GPT4 currently can’t do seem to me that they’ll be doable in the next few years.
Intuitively, I struggle with this, but you have inside data and I do not. Maybe we just set this point aside for now, we have plenty of other points we can discuss.
To be clear, I do NOT think that today’s systems could replace 99% of remote jobs even with a century of schlep. And in particular I don’t think they are capable of massively automating AI R&D even with a century of schlep. I just think they could be producing, say, at least an OOM more economic value. …
This, I would agree with. And on re-reading, I think I may have been mixed up as to what you and Ajeya were saying in the section I was quoting from here, so I’ll drop this.
[Ege] I think when you try to use the systems in practical situations; they might lose coherence over long chains of thought, or be unable to effectively debug non-performant complex code, or not be able to have as good intuitions about which research directions would be promising, et cetera.
This was a nice answer from Ege. My follow up questions would be: Why? I have theories about what coherence is and why current models often lose it over long chains of thought (spoiler: they weren’t trained to have trains of thought) and theories about why they aren’t already excellent complex-code-debuggers (spoiler: they weren’t trained to be) etc. What’s your theory for why all the things AI labs will try between now and 2030 to make AIs good at these things will fail?
I would not confidently argue that it won’t happen by 2030; I am suggesting that these problems are unlikely to be well solved in a usable-in-the-field form by 2027 (four years from now). My thinking:
The rapid progress in LLM capabilities has been substantially fueled by the availability of stupendous amounts of training data.
There is no similar abundance of low-hanging training data for extended (day/week/more) chains of thought, nor for complex debugging tasks. Hence, it will not be easy to extend LLMs (and/or train some non-LLM model) to high performance at these tasks.
A lot of energy will go into the attempt, which will eventually succeed. But per (2), I think some new techniques will be needed, which will take time to identify, refine, scale, and productize; a heavy lift in four years. (Basically: Hofstadter’s Law.)
Especially because I wouldn’t be surprised if complex-code-debugging turns out to be essentially “AGI-complete”, i.e. it may require a sufficiently varied mix of exploration, logical reasoning, code analysis, etc. that you pretty much have to be a general AGI to be able to do it well.
I understand you might be skeptical that it can be done but I encourage you to red-team your position, and ask yourself ‘how would I do it, if I were an AI lab hell-bent on winning the AGI race?’ You might be able to think of some things.
In a nearby universe, I would be fundraising for a startup to do exactly that, it sounds like a hell of fun problem. :-) And I’m sure you’re right… I just wouldn’t expect to get to “capable of 99% of all remote work” within four years.
I realize you’re not explicitly labeling this as a prediction, but… isn’t this precisely the sort of thought process to which Hofstadter’s Law applies?
Indeed. Like I said, my timelines are based on a portfolio of different models/worlds; the very short-timelines models/worlds are basically like “look we basically already have the ingredients, we just need to assemble them, here is how to do it...” and the planning fallacy / hofstadter’s law 100% applies to this. The 5-year-and-beyond worlds are not like that; they are … looking at lines on graphs and then extrapolating them …
So my timelines do indeed take into account Hofstadter’s Law. If I wasn’t accounting for it already, my median would be lower than 2027. However, I am open to the criticism that maybe I am not accounting for it enough.
To be clear, I’m only attempting to argue about the short-timeline worlds. I agree that Hofstadter’s Law doesn’t apply to curve extrapolation. (My intuition for 5-year-and-beyond worlds is more like Ege’s, but I have nothing coherent to add to the discussion on that front.) And so, yes, I think my position boils down to “I believe that, in your short-timeline worlds, you are not accounting for Hofstadter’s Law enough”.
As you proposed, I think the interesting place to go from here would be some predictions. I’ll noodle on this, and I’d be very interested to hear any thoughts you have – milestones along the path you envision in your default model of what rapid progress looks like; or at least, whatever implications thereof you feel comfortable talking about.
This post taught me a lot about different ways of thinking about timelines, thanks to everyone involved!
I’d like to offer some arguments that, contra Daniel’s view, AI systems are highly unlikely to be able to replace 99% of current fully remote jobs anytime in the next 4 years. As a sample task, I’ll reference software engineering projects that take a reasonably skilled human practitioner one week to complete. I imagine that, for AIs to be ready for 99% of current fully remote jobs, they would need to be able to accomplish such a task. (That specific category might be less than 1% of all remote jobs, but I imagine that the class of remote jobs requiring at least this level of cognitive ability is more than 1%.)
Rather than referencing scaling laws, my arguments stem from analysis of two specific mechanisms which I believe are missing from current LLMs:
Long-term memory. LLMs of course have no native mechanism for retaining new information beyond the scope of their token buffer. I don’t think it is possible to carry out a complex extended task, such as a week-long software engineering project, without long-term memory to manage the task, keep track of intermediate thoughts regarding design approaches, etc.
Iterative / exploratory work processes. The LLM training process focuses on producing final work output in a single pass, with no planning process, design exploration, intermediate drafts, revisions, etc. I don’t think it is possible to accomplish a week-long software engineering task in a single pass; at least, not without very strongly superhuman capabilities (unlikely to be reached in just four years).
Of course there are workarounds for each of these issues, such as RAG for long-term memory, and multi-prompt approaches (chain-of-thought, tree-of-thought, AutoGPT, etc.) for exploratory work processes. But I see no reason to believe that they will work sufficiently well to tackle a week-long project. Briefly, my intuitive argument is that these are old school, rigid, GOFAI, Software 1.0 sorts of approaches, the sort of thing that tends to not work out very well in messy real-world situations. Many people have observed that even in the era of GPT-4, there is a conspicuous lack of LLMs accomplishing any really meaty creative work; I think these missing capabilities lie at the heart of the problem.
Nor do I see how we could expect another round or two of scaling to introduce the missing capabilities. The core problem is that we don’t have massive amounts of training data for managing long-term memory or carrying out exploratory work processes. Generating such data at the necessary scale, if it’s even possible, seems much harder than what we’ve been doing up to this point to marshall training data for LLMs.
The upshot is that I think that we have been seeing the rapid increase in capabilities of generative AI, failing to notice that this progress is confined to a particular subclass of tasks – namely, tasks which can pretty much be accomplished using System 1 alone – and collectively fooling ourselves into thinking that the trend of increasing capabilities is going to quickly roll through the remainder of human capabilities. In other words, I believe the assertion that the recent rate of progress will continue up through AGI is based on an overgeneralization. For an extended version of this claim, see a post I wrote a few months ago: The AI Progress Paradox. I’ve also written at greater length about the issues of Long-term memory and Exploratory work processes.
In the remainder of this comment, I’m going to comment what I believe are some weak points in the argument for short timelines (as presented in the original post).
[Daniel] It seems to me that GPT-4 is already pretty good at coding, and a big part of accelerating AI R&D seems very much in reach—like, it doesn’t seem to me like there is a 10-year, 4-OOM-training-FLOP gap between GPT4 and a system which is basically a remote-working OpenAI engineer that thinks at 10x serial speed.
Coding, in the sense that GPT4 can do it, is nowhere near the top of the hierarchy of skills involved in serious software engineering. And so I believe this is a bit like saying that, because a certain robot is already pretty decent at chiseling, it will soon be able to produce works of art at the same level as any human sculptor.
[Ajeya] I don’t know, 4 OOM is less than two GPTs, so we’re talking less than GPT-6. Given how consistently I’ve been wrong about how well “impressive capabilities in the lab” will translate to “high economic value” since 2020, this seems roughly right to me?
[Daniel] I disagree with this update—I think the update should be “it takes a lot of schlep and time for the kinks to be worked out and for products to find market fit” rather than “the systems aren’t actually capable of this.” Like, I bet if AI progress stopped now, but people continued to make apps and widgets using fine-tunes of various GPTs, there would be OOMs more economic value being produced by AI in 2030 than today.
If the delay in real-world economic value were due to “schlep”, shouldn’t we already see one-off demonstrations of LLMs performing economically-valuable-caliber tasks in the lab? For instance, regarding software engineering, maybe it takes a long time to create a packaged product that can be deployed in the field, absorb the context of a legacy codebase, etc. and perform useful high-level work. But if that’s the only problem, shouldn’t there already be at least one demonstration of an LLM doing some meaty software engineering project in a friendly lab environment somewhere?
More generally, how do we define “schlep” such that the need for schlep explains the lack of visible accomplishments today, but also allows for AI systems be able to replace 99% of remote jobs within just four years?
[Daniel] And so I think that the AI labs will be using AI remote engineers much sooner than the general economy will be. (Part of my view here is that around the time it is capable of being a remote engineer, the process of working out the kinks / pushing through schlep will itself be largely automatable.)
What is your definition of “schlep”? I’d assumed it referred to the innumerable details of figuring out how to adapt and integrate a raw LLM into a finished product which can handle all of the messy requirements of real-world use cases – the “last mile” of unspoken requirements and funky edge cases. Shouldn’t we expect such things to be rather difficult to automate? Or do you mean something else by “schlep”?
[Daniel] …when I say 2027 as my median, that’s kinda because I can actually quite easily see it happening in 2025, but things take longer than I expect, so I double it.
Can you see LLMs acquiring long-term memory and an expert-level, nuanced ability to carry out extended exploratory processes by 2025? If yes, how do you see that coming about? If no, does that cause you to update at all?
[Daniel] I take it that in this scenario, despite getting IMO gold etc. the systems of 2030 are not able to do the work of today’s OAI engineer? Just clarifying. Can you say more about what goes wrong when you try to use them in such a role?
Anecdote: I got IMO silver (granted, not gold) twice, in my junior and senior years of high school. At that point I had already been programming for close to ten years, and spent considerably more time coding than I spent studying math, but I would not have been much of an asset to an engineering team. I had no concept of how to plan a project, organize a codebase, design maintainable code, strategize a debugging session, evaluate tradeoffs, see between the lines of a poorly written requirements document, etc. Ege described it pretty well:
I think when you try to use the systems in practical situations; they might lose coherence over long chains of thought, or be unable to effectively debug non-performant complex code, or not be able to have as good intuitions about which research directions would be promising, et cetera.
This probably underestimates the degree to which IMO-silver-winning me would have struggled. For instance, I remember really struggling to debug binary tree rotation (a fairly simple bit of data-structure-and-algorithm work) for a college class, almost 2.5 years after my first silver.
[Ajeya] I think by the time systems are transformative enough to massively accelerate AI R&D, they will still not be that close to savannah-to-boardroom level transfer, but it will be fine because they will be trained on exactly what we wanted them to do for us.
This assumes we’re able to train them on exactly what we want them to do. It’s not obvious to me how we would train a model to do, for example, high-level software engineering? (In any case, I suspect that this is not far off from being AGI-complete; I would suspect the same of high-level work in most fields; see again my earlier-linked post on the skills involved in engineering.)
[Daniel] …here’s a scenario I think it would be productive to discuss:
(1) Q1 2024: A bigger, better model than GPT-4 is released by some lab. It’s multimodal; it can take a screenshot as input and output not just tokens but keystrokes and mouseclicks and images. Just like with GPT-4 vs. GPT-3.5 vs. GPT-3, it turns out to have new emergent capabilities. Everything GPT-4 can do, it can do better, but there are also some qualitatively new things that it can do (though not super reliably) that GPT-4 couldn’t do.
…
(6) Q3 2026 Superintelligent AGI happens, by whatever definition is your favorite. And you see it with your own eyes.
I realize you’re not explicitly labeling this as a prediction, but… isn’t this precisely the sort of thought process to which Hofstadter’s Law applies?
Thanks for the thoughtful and detailed comments! I’ll respond to a few points, otherwise in general I’m just nodding in agreement.
I think it’s important to emphasize (a) that Davidson’s model is mostly about pre-AGI takeoff (20% automation to 100%) rather than post-AGI takeoff (100% to superintelligence) but it strongly suggests that the latter will be very fast (relative to what most people naively expect) on the order of weeks probably and very likely less than a year.
And it’s a good model, so we need to take this seriously. My only quibble would be to raise again the possibility (only a possibility!) that progress becomes more difficult around the point where we reach AGI, because that is the point where we’d be outgrowing human training data. I haven’t tried to play with the model and see whether that would significantly affect the post-AGI takeoff timeline.
(Oh, and now that I think about it more, I’d guess that Davidson’s model significantly underestimates the speed of post-AGI takeoff, because it might just treat anything above AGI as merely 100% automation, whereas actually there are different degrees of 100% automation corresponding to different levels of quality intelligence; 100% automation by ASI will be significantly more research-oomph than 100% automation by AGI. But I’d need to reread the model to decide whether this is true or not. You’ve read it recently, what do you think?)
I want to say that he models this by equating the contribution of one ASI to more than one AGI, i.e. treating additional intelligence as equivalent to a speed boost. But I could be mis-remembering, and I certainly don’t remember how he translates intelligence into speed. If it’s just that each post-AGI factor of two in algorithm / silicon improvements is modeled as yielding twice as many AGIs per dollar, then I’d agree that might be an underestimate (because one IQ 300 AI might be worth a very large number of IQ 150 AIs, or whatever).
And (b) Davidson’s model says that while there is significant uncertainty over how fast takeoff will be if it happens in the 30′s or beyond, if it happens in the 20′s—i.e. if AGI is achieved in the 20′s—then it’s pretty much gotta be pretty fast. Again this can be seen by playing around with the widget on takeoffspeeds.com.
Yeah, even without consulting any models, I would expect that any scenario where we achieve AGI in the 20s is a very scary scenario for many reasons.
--I work at OpenAI and I see how the sausage gets made. Already things like Copilot and ChatGPT are (barely, but noticeably) accelerating AI R&D. I can see a clear path to automating more and more parts of the research process, and my estimate is that going 10x faster is something like a lower bound on what would happen if we had AGI (e.g. if AutoGPT worked well enough that we could basically use it as a virtual engineer + scientist) and my central estimate would be “it’s probably about 10x when we first reach AGI, but then it quickly becomes 100x, 1000x, etc. as qualitative improvements kick in.” There’s a related issue of how much ‘room to grow’ is there, i.e. how much low-hanging fruit is there to pick that would improve our algorithms, supposing we started from something like “It’s AutoGPT but good, as good as an OAI employee.” My answer is “Several OOMs at least.” So my nose-to-the-ground impression is if anything more bullish/fast-takeoff-y than Davidson’s model predicts.
What is your feeling regarding the importance of other inputs, i.e. training data and compute?
> I think of AI progress as being driven by a mix of cognitive input, training data, training FLOPs, and inference FLOPs. Davidson models the impact of cognitive input and inference FLOPs, but I didn’t see training data or training FLOPs taken into account. (“Doesn’t model data/environment inputs to AI development.”) My expectation that as RSI drives an increase in cognitive input, training data and training FLOPs will be a drag on progress. (Training FLOPs will be increasing, but not as quickly as cognitive inputs.)
Training FLOPs is literally the most important and prominent variable in the model, it’s the “AGI training requirements” variable. I agree that possible data bottlenecks are ignored; if it turns out that data is the bottleneck, timelines to AGI will be longer (and possibly takeoff slower? Depends on how the data problem eventually gets solved; takeoff could be faster in some scenarios...) Personally I don’t think the data bottleneck will slow us down much, but I could be wrong.
Ugh! This was a big miss on my part, thank you for calling it out. I skimmed too rapidly through the introduction. I saw references to biological anchors and I think I assumed that meant the model was starting from an estimate of FLOPS performed by the brain (i.e. during “inference”) and projecting when the combination of more-efficient algorithms and larger FLOPS budgets (due to more $$$ plus better hardware) would cross that threshold. But on re-read, of course you are correct and the model does focus on training FLOPS.
So to be clear, I am not suggesting that a foom is impossible. The title of the post contains the phrase “might never happen”.
I guess you might reasonably argue that, from the perspective of (say) a person living 20,000 years ago, modern life does in fact sit on the far side of a singularity. When I see the word ‘singularity’, I think of the classic Peace War usage of technology spiraling to effectively infinity, or at least far beyond present-day technology. I suppose that led me to be a bit sloppy in my use of the term.
The point I was trying to make by referencing those various historical events is that all of the feedback loops in question petered out short of a Vingian singularity. And it’s a fair correction that some of those loops are actually still in play. But many are not – forest fires burn out, the Cambrian explosion stopped exploding – so we do have existence proofs that feedback loops can come to a halt. I know that’s not any big revelation, I was merely attempting to bring the concept to mind in the context of RSI.
In any case, all I’m really trying to do is to argue that the following syllogism is invalid:
As AI approaches human level, it will be able to contribute to AI R&D, thus increasing the pace of AI improvement.
This process can be repeated indefinitely.
Therefore, as soon as AI is able to meaningfully contribute to its own development, we will quickly spiral to a Vingian singularity.
This scenario is certainly plausible, but I frequently see it treated as a mathematical certainty. And that is simply not the case. The improvement cycle will only exhibit a rapid upward spiral under certain assumptions regarding the relationship of R&D inputs to gains in AI capability – the
rterm in Davidson’s model.(Then I spend some time explaining why I think
rmight be lower than expected during the period where AI is passing through human level. Again, “might be”.)
OK, having read through much of the detailed report, here’s my best attempt to summarize my and Davidson’s opinions. I think they’re mostly compatible, but I am more conservative regarding the impact of RSI in particular, and takeoff speeds in general.
My attempt to summarize Davidson on recursive self-improvement
AI will probably be able to contribute to AI R&D (improvements to training algorithms, chip design, etc.) somewhat ahead of its contributions to the broader economy. Taking this into account, he predicts that the “takeoff time” (transition from “AI could readily automate 20% of cognitive tasks” to “AI could readily automate 100% of cognitive tasks”) will take a few years: median 2.9, 80% confidence interval 0.8 to 12.5.
He notes that the RSI feedback loop could converge or diverge:
The feedback loop is:
Better software → more 2020-FLOP → more software R&D → better software
It turns out that, with this feedback loop, there are two broad possibilities.
1. Software singularity—quicker and quicker doublings. If returns to software R&D exceed a certain threshold, the feedback loop is so powerful that there’s a “software only singularity”. The level of software, quantified here as 2020-FLOP per FLOP, grows faster and faster, theoretically going to infinity in finite time. And this happens even using a fixed quantity of physical FLOP to run the AIs. In practice, of course, the software returns become worse before we go to infinity and we move to the second possibility.
2. Software fizzle—slower and slower doublings. If returns to software R&D are below a certain threshold, the level of software grows more and more slowly over time, assuming a fixed quantity of physical FLOP. (If the amount of physical FLOP is in fact growing increasingly quickly, then the level of software can do the same. But software progress is reliant on the growth of physical FLOP.)
Which possibility will obtain? It turns out that there is a software singularity just if r > 1, where r is defined as in section 4:
For each doubling of cumulative R&D inputs, the output metric will double r times.
He projects a 65% probability of scenario 1 (RSI leads to an accelerating capabilities ramp) occurring based strictly on software improvements, but thinks it would not last indefinitely:
Overall, I’m roughly ~65% on a software-only singularity occurring, and my median best guess is that it would last for ~2-3 OOMs if it happened.
Here, I believe he is only taking into account the contributions of AI to algorithm (software) improvements. Presumably, taking AI contributions to hardware design into account would produce a somewhat more aggressive estimate; this is part of the overall model, but I didn’t see it broken out into a specific probability estimate for a period of quicker-and-quicker doublings.
My position
I agree that RSI might or might not result in an accelerating capabilities ramp as we approach human-level AGI. I keep encountering people assuming that what Davidson calls a “software singularity” is self-evidently inevitable, and my main goal was to argue that, while possible, it is not inevitable. Davidson himself expresses a related sentiment that his model is less aggressive than some people’s stated expectations; for instance:
My impression is that Eliezer Yudkowsky expects takeoff to be very fast, happening in time scales of days or months. By contrast, this framework puts the bulk of its probability on takeoff taking multiple years.
I have not attempted to produce a concrete estimate of takeoff speed.
I expect the impact of RSI to be somewhat less than Davidson models:
I expect that during the transition period, where humans and AIs are each making significant contributions to AI R&D, that there will be significant lags in taking full advantage of AI (project management and individual work habits will continually need to be adjusted), with resulting inefficiencies. Davidson touches on these ideas, but AFAICT does not include them in the model; for instance, “Assumes no lag in reallocating human talent when tasks have been automated.”
It may be that when AI has automated X% of human inputs into AI R&D, the remaining inputs are the most sophisticated part of the job, and can only be done by senior researchers, meaning that most of the people being freed up are not immediately able to be redirected to non-automated tasks. It might even be the case that, by abandoning the lower-level work, we (humans) would lose our grounding in the nuts and bolts of the field, and the quality of the higher-level work we are still doing might gradually decline.
I think of AI progress as being driven by a mix of cognitive input, training data, training FLOPs, and inference FLOPs. Davidson models the impact of cognitive input and inference FLOPs, but I didn’t see training data or training FLOPs taken into account. (“Doesn’t model data/environment inputs to AI development.”) My expectation that as RSI drives an increase in cognitive input, training data and training FLOPs will be a drag on progress. (Training FLOPs will be increasing, but not as quickly as cognitive inputs.)
I specifically expect progress to become more difficult as we approach human-level AGI, as human-generated training data will become less useful at that point. We will also be outrunning our existence proof for intelligence; I expect superhuman intelligence to be feasible, but we don’t know for certain that extreme superhuman performance is reasonably achievable, and so we should allow for some probability that progress beyond human performance will be significantly more difficult.
As we approach human-level AGI, we may encounter other complications: coordination problems and transition delays as the economy begins to evolve rapidly, increased security overhead as AI becomes increasingly strategic for both corporations and nations (and as risks hopefully are taken more seriously), etc.
Most of this can be summarized as “everything is always harder and always takes longer than you think, even when you take this into account”.
For the same reasons, I am somewhat more conservative than Davidson on the timeline from human-level AGI to superintelligence (which he guesses will take less than a year); but I’m not in a position to quantify that.
Davidson does note some of these possibilities. For instance, he cites a few factors that could result in superintelligence taking longer than a year (even though he does not expect that to be the case), including two of the factors I emphasize:
Software progress may have become much harder by the time we reach AGI.
Progress might become bottlenecked by the need to run expensive computational experiments, or to rerun multi-month long AI training runs.
Thanks, I appreciate the feedback. I originally wrote this piece for a less technical audience, for whom I try to write articles that are self-contained. It’s a good point that if I’m going to post here, I should take a different approach.
You don’t need a new generation of fab equipment to make advances in GPU design. A lot of improvements of the last few years were not about having constantly a new generation of fab equipment.
Ah, by “producing GPUs” I thought you meant physical manufacturing. Yes, there has been rapid progress of late in getting more FLOPs per transistor for training and inference workloads, and yes, RSI will presumably have an impact here. The cycle time would still be slower than for software: an improved model can be immediately deployed to all existing GPUs, while an improved GPU design only impacts chips produced in the future.
Thanks. I had seen Davidson’s model, it’s a nice piece of work. I had not previously read it closely enough to note that he does discuss the question of whether RSI is likely to converge or diverge, but I see that now. For instance (emphasis added):
We are restricting ourselves only to efficiency software improvements, i.e. ones that decrease the physical FLOP/s to achieve a given capability. With this restriction, the mathematical condition for a singularity here is the same as before: each doubling of cumulative inputs must more than double the efficiency of AI algorithms. If this holds, then the efficiency of running AGIs (of fixed ability) will double faster and faster over time. Let’s call this an “efficiency-only singularity”, which is of course an example of a software-only singularity.
I’ll need some time to thoroughly digest what he has to say on this topic.
I’ll try to summarize your point, as I understand it:
Intelligence is just one of many components. If you get huge amounts of intelligence, at that point you will be bottlenecked by something else, and even more intelligence will not help you significantly. (Company R&D doesn’t bring a “research explosion”.)
The core idea I’m trying to propose (but seem to have communicated poorly) is that the AI self-improvement feedback loop might (at some point) converge, rather than diverging. In very crude terms, suppose that GPT-8 has IQ 180, and we use ten million instances of it to design GPT-9, then perhaps we get a system with IQ 190. Then we use ten million instances of GPT-9 to design GPT-10, perhaps that has IQ 195, and eventually GPT-∞ converges at IQ 200.
I do not claim this is inevitable, merely that it seems possible, or at any rate is not ruled out by any mathematical principle. It comes down to an empirical question of how much incremental R&D effort is needed to achieve each incremental increase in AI capability.
The point about the possibility of bottlenecks other than intelligence feeds into that question about R&D effort vs. increase in capability; if we double R&D effort but are bottlenecked on, say, training data, than we might get a disappointing increase in capability.
IIUC, much of the argument you’re making here is that the existing dynamic of IP laws, employee churn, etc. puts a limit on the amount of R&D investment that any given company is willing to make, and that these incentives might soon shift in a way that could unleash a drastic increase in AI R&D spending? That seems plausible, but I don’t see how it ultimately changes the slope of the feedback loop – it merely allows for a boost up the early part of the curve?
Also, please note that LLMs are just one possible paradigm of AI. Yes, currently the best one, but who knows what tomorrow may bring. I think most people among AI doomers would agree that LLMs are not the kind of AI they fear. LLMs succeed to piggyback on humanity’s written output, but they are also bottlenecked by it.
Agreed that there’s a very good chance that AGI may not look all that much like an LLM. And so when we contemplate the outcome of recursive self-improvement, a key question will be what the R&D vs. increase-in-capability curve looks like for whatever architecture emerges.
Sure, it’s easy to imagine scenarios where a specific given company could be larger than it is today. But are you envisioning that if we eliminated antitrust laws and made a few other specific changes, then it would become plausible for a single company to take over the entire economy?
My thesis boils down to the simple assertion that feedback loops need not diverge indefinitely, exponential growth can resolve into an S-curve. In the case of a corporation, the technological advantages, company culture, and other factors that allow a company to thrive in one domain (e.g. Google, web search) might not serve it well in another domain (Google, social networks). In the case of AI self-improvement, it might turn out that we eventually enter a domain – for instance, the point where we’ve exhausted human-generated training data – where the cognitive effort required to push capabilities forwards increases faster than the cognitive effort supplied by those same capabilities. In other words, we might reach a point where each successive generation of recursively-designed AI delivers a decreasing improvement over its predecessor. Note that I don’t claim this is guaranteed to happen, I merely argue that it is possible, but that seems to be enough of a claim to be controversial.
We can look at a skill that’s about applying human intelligence like playing Go. It would be possible that the maximum skill level is near what professional go players are able to accomplish. AlphaGo managed to go very much past what humans can accomplish in a very short timeframe and AlphaGo doesn’t even do any self-recursive editing of it’s own code.
Certainly. I think we see that the ease with which computers can definitively surpass humans depends on the domain. For multiplying large numbers, it’s no contest at all. For Go, computers win definitively, but by a smaller margin than for multiplication. Perhaps, as we move toward more and more complex and open-ended problems, it will get harder and harder to leave humans in the dust? (Not impossible, just harder?) I discuss this briefly in a recent blog post, I’d love to hear thoughts / evidence in either direction.
AI can help with producing GPU’s as well. It’s possible to direct a lot more of the worlds economic output into producing GPU’s than is currently done.
Sure. I’m just suggesting that the self-improvement feedback loop would be slower here, because designing and deploying a new generation of fab equipment has a much longer cycle time than training a new model, no?
All of these things are possible, but it’s not clear to me that they’re likely, at least in the early stages of AGI. In other words: once we have significantly-superhuman AGI, then agreed, all sorts of crazy things may become possible. But first we have to somehow achieve superhuman AGI. One of the things I’m trying to do in this post is explore the path that gets us to superhuman AGI in the first place. That path, by definition, can’t rely on anything that requires superhuman capabilities.
If I understand correctly, you’re envisioning that we will be able to construct AGIs that have human-level capability, and far greater than human speed, in order to bootstrap superhuman AGI? What makes you confident that this speed advantage will exist early on? Current leading-edge models like GPT-4 are not drastically faster than human beings, and presumably until we get to human-level AGI we’ll be spending most of our algorithmic improvements and increases in FLOP budget on increased capabilities, rather than performance. In fact, it’s quite possible that we’ll have to (temporarily) accept reduced speed in order to achieve human-level performance; for instance, by introducing tree search into the thought process (tree-of-thought prompting, heuristic search techniques in Yann LeCun’s “A Path Towards Autonomous Machine Intelligence”, etc.).
Once we achieve human-level, human-speed AGI, then yes, further algorithm or FLOPs improvements could be spent on speed; this comes back to the basic question of how whether the cognitive effort required for further progress increases more or less rapidly than the extent to which progress (and/or increased budgets) enables increased cognitive effort, i.e. does the self-improvement feedback loop converge or diverge. Are you proposing that it definitely diverges? What points you in that direction?
I would also caution against being highly confident that AGI will automatically be some sort of ideal omnimath. Such ability would require more than merely assimilating all of human knowledge and abilities; it would require knowing exactly which sub-specialties to draw on in any given moment. Otherwise, the AI would risk drowning in millions of superfluous connections to its every thought. Some examples of human genius might in part depend on a particular individual just happening to have precisely the right combination of knowledge, without a lot of other superfluous considerations to distract them.
Also, is it obvious that a single early-human-level AI could be trained with deep mastery of every field of human knowledge? Biological-anchor analysis aims to project when we can create a human-level AI, and humans are not omnimaths. Deep expertise across every subspeciality might easily require many more parameters than the number of synapses in the human brain. Many things look simple until you try to implement them; I touch on this in a recent blog post, The AI Progress Paradox, but you just need to look at the history of self-driving cars (or photorealistic CGI, or many other examples) to see how things that seem simple in principle can require many rounds of iteration to fully achieve in practice.
I think you’re saying that the fact that no historical feedback loop has ever destroyed the Earth (nor transformed it into a state which would not support human life) could be explained by the Anthropic Principle? Sure, that’s true enough. I was aiming more to provide an intuition for the idea that it’s very common and normal for feedback loops to eventually reach a limit, as there are many examples in the historical record.
Intuition aside: given the sheer number of historical feedback loops that have failed to destroy the Earth, it seems unavoidable that either (a) there are some fundamental principles at play that tend to place a cap on feedback loops, at least in the family of alternative universes that this universe has been sampled from, or (b) we have to lean on the Anthropic Principle very very hard indeed. It’s not hard to articulate causes for (a); for instance, any given feedback loop arises under a particular set of conditions, and once it has progressed sufficiently, it will begin to alter its own environment to the point where those conditions may no longer apply. (The forest fire consumes all available fuel, etc.)
As I go about my day, I need to maintain a probability distribution over states of the world. If an LLM tries to imitate me (i.e. repeatedly predict my next output token), it needs to maintain a probability distribution, not just over states of the world, but also over my internal state (i.e. the state of the agent whose outputs it is predicting). I don’t need to keep track of multiple states that I myself might be in, but the LLM does. Seems like that makes its task more difficult?
Or to put an entirely different frame on the the whole thing: the job of a traditional agent, such as you or me, is to make intelligent decisions. An LLM’s job is to make the exact same intelligent decision that a certain specific actor being imitated would make. Seems harder?