AI-2027 and a lot of other AI doom forecasts seem to rest on a big assumption—that LLMs are capable of achieving some form of AGI or superintelligence, and that progress we see in LLMs getting better at doing LLM things is equivalent to progress towards humanity developing AGI or ASI as a whole. This is not necessarily true, though it can be tempting to believe it is, especially when you’re watching the LLMs getting better at conversing and coding and taking over peoples’ jobs in real time. I think a lot of that progress is totally tangential to the task of creating AGI/ASI. Giving an LLM more reinforcement learning and more fine-tuned prompting to say less politically incorrect things and make less coding mistakes is a huge step towards making it useful in the workplace, but is not necessarily a step towards general or superintelligence.
I really like this scenario because it does not make that assumption. It is very conservative, every prediction it makes is well grounded in tech development trends we can see happening currently or forces that already exist and motivate decision-makers today, instead of relying on assumptions about huge breakthroughs that still haven’t happened yet. One of the strongest biases I see persistently in the tech community—and I’m no exception, I catch myself with it all the time too—is a bias towards optimism* in believing a new technology will develop and radically transform society very soon, whether it’s self-driving cars, virtual reality, cryptocurrency, or AI. I think this model is as free of that bias as any ai-doom-prediction scenario can possibly be.
That’s not to say I don’t believe AGI/ASI is in our future, or that this model even rules it out. I am no expert, but if I had to choose my most likely prediction based on what I know, it would be something like this model, with LLMs hitting a plateau before they are able to achieve general intelligence, except that at some unspecified point in the future—could be in 1 year, could be 10, could be 100--ASI gets dropped on humanity out of nowhere, because while we were all busy freaking out about ChatGPT 6 or 7 taking our jobs, someone else was quietly developing real AI in a lab using a “brain-in-a-box-in-a-basement” model that has nothing to do with today’s LLMs.
It may be true that LLMs are going to radically transform society and the workforce, and also true that ASI is something humanity will build and carries the existential risks we’re all familiar with, but those two things may turn out to be totally unrelated. I don’t think that possibility gets discussed enough. If it is true, that makes AI alignment a much more difficult job, and most of our efforts to “align” the LLMs that we have completely futile.
*I mean making optimistic assessments of how fast technology will develop and transform the world—not necessarily optimistic about the outcomes being good. Believing that ASI will be fully developed and kill us all a week from now would still be an example of that “optimistic” bias in this context.
This scenario doesn’t predict that LLMs can’t be AGI. As depicted, the idea is that something based upon LLMs (with CoT, memory, tool use, etc.) is able to reach strong AGI (able to do anything most people can do on a computer), but is only able to reach the intelligence of the mid-to-high-90th percentile person. Indeed, I’d argue that current SOTA systems should basically be considered very weak/baby-AGI (with a bunch of non-intelligence-related limitations).
The limitation depicted here, which I think is plausible but far from certain, is that high intelligence requires a massive amount of compute, which models don’t have access to. There’s more cause to suspect that, if this limit exists, this is a fundamental limitation than a limitation of LLM-based models. In the scenario, it’s envisioned that researchers try non-LLM-based AIs too, but run up against the same fundamental limits that make ASI impossible without far more compute than is feasible.
You could be right about the limit based on overall compute applying to other approaches to AI just as much as to LLMs. Speculating about the future of AI is always a little frustrating because ultimately we won’t know how to make AGI/ASI until we have it (and can’t even agree on how we will know it when we see it). The way I approach the problem is by looking at what we do know—at this point in time, we only know of one system in existence that we can all agree meets the definition of “general intelligence”, and that is the human brain. Because of how little we still understand about how intelligence actually works, I think the most likely path to AGI—resting on the fewest assumptions about things we don’t know—is a “brain-like AGI”. That’s basically Steven Byrnes’s view and I think his arguments are very compelling. If you accept that view, than I think we end up with something like your scenario anyway, at least for a while until the brain-like AGI comes to fruition.
The whole no-one-can-agree-on-what-AGI-is thing is damn true, and a real problem. Cole and I have a joke that it’s not AGMI (Artificial Gary Marcus Intelligence) unless it solve the hard problem of consciousness, multiply numbers of arbitrary length without error (which humans can’t do perfectly with paper, and obviously really can’t without), and various other things, all at once. A recent post with over 250 karma said that LLM’s aren’t AGI because they can’t make billion-dollar businesses, which almost no humans can do, and no humans can do quickly.
As for the most likely way to get AGI, the case is quite strong for LRMs with additional RL around stuff like long-term memory and to reduce hallucinations, since those systems are, in many ways, nearly there, and there are no clear barriers to them making it the rest of the way.
AI-2027 and a lot of other AI doom forecasts seem to rest on a big assumption—that LLMs are capable of achieving some form of AGI or superintelligence, and that progress we see in LLMs getting better at doing LLM things is equivalent to progress towards humanity developing AGI or ASI as a whole. This is not necessarily true, though it can be tempting to believe it is, especially when you’re watching the LLMs getting better at conversing and coding and taking over peoples’ jobs in real time. I think a lot of that progress is totally tangential to the task of creating AGI/ASI. Giving an LLM more reinforcement learning and more fine-tuned prompting to say less politically incorrect things and make less coding mistakes is a huge step towards making it useful in the workplace, but is not necessarily a step towards general or superintelligence.
I really like this scenario because it does not make that assumption. It is very conservative, every prediction it makes is well grounded in tech development trends we can see happening currently or forces that already exist and motivate decision-makers today, instead of relying on assumptions about huge breakthroughs that still haven’t happened yet. One of the strongest biases I see persistently in the tech community—and I’m no exception, I catch myself with it all the time too—is a bias towards optimism* in believing a new technology will develop and radically transform society very soon, whether it’s self-driving cars, virtual reality, cryptocurrency, or AI. I think this model is as free of that bias as any ai-doom-prediction scenario can possibly be.
That’s not to say I don’t believe AGI/ASI is in our future, or that this model even rules it out. I am no expert, but if I had to choose my most likely prediction based on what I know, it would be something like this model, with LLMs hitting a plateau before they are able to achieve general intelligence, except that at some unspecified point in the future—could be in 1 year, could be 10, could be 100--ASI gets dropped on humanity out of nowhere, because while we were all busy freaking out about ChatGPT 6 or 7 taking our jobs, someone else was quietly developing real AI in a lab using a “brain-in-a-box-in-a-basement” model that has nothing to do with today’s LLMs.
It may be true that LLMs are going to radically transform society and the workforce, and also true that ASI is something humanity will build and carries the existential risks we’re all familiar with, but those two things may turn out to be totally unrelated. I don’t think that possibility gets discussed enough. If it is true, that makes AI alignment a much more difficult job, and most of our efforts to “align” the LLMs that we have completely futile.
*I mean making optimistic assessments of how fast technology will develop and transform the world—not necessarily optimistic about the outcomes being good. Believing that ASI will be fully developed and kill us all a week from now would still be an example of that “optimistic” bias in this context.
I appreciate it.
This scenario doesn’t predict that LLMs can’t be AGI. As depicted, the idea is that something based upon LLMs (with CoT, memory, tool use, etc.) is able to reach strong AGI (able to do anything most people can do on a computer), but is only able to reach the intelligence of the mid-to-high-90th percentile person. Indeed, I’d argue that current SOTA systems should basically be considered very weak/baby-AGI (with a bunch of non-intelligence-related limitations).
The limitation depicted here, which I think is plausible but far from certain, is that high intelligence requires a massive amount of compute, which models don’t have access to. There’s more cause to suspect that, if this limit exists, this is a fundamental limitation than a limitation of LLM-based models. In the scenario, it’s envisioned that researchers try non-LLM-based AIs too, but run up against the same fundamental limits that make ASI impossible without far more compute than is feasible.
You could be right about the limit based on overall compute applying to other approaches to AI just as much as to LLMs. Speculating about the future of AI is always a little frustrating because ultimately we won’t know how to make AGI/ASI until we have it (and can’t even agree on how we will know it when we see it). The way I approach the problem is by looking at what we do know—at this point in time, we only know of one system in existence that we can all agree meets the definition of “general intelligence”, and that is the human brain. Because of how little we still understand about how intelligence actually works, I think the most likely path to AGI—resting on the fewest assumptions about things we don’t know—is a “brain-like AGI”. That’s basically Steven Byrnes’s view and I think his arguments are very compelling. If you accept that view, than I think we end up with something like your scenario anyway, at least for a while until the brain-like AGI comes to fruition.
The whole no-one-can-agree-on-what-AGI-is thing is damn true, and a real problem. Cole and I have a joke that it’s not AGMI (Artificial Gary Marcus Intelligence) unless it solve the hard problem of consciousness, multiply numbers of arbitrary length without error (which humans can’t do perfectly with paper, and obviously really can’t without), and various other things, all at once. A recent post with over 250 karma said that LLM’s aren’t AGI because they can’t make billion-dollar businesses, which almost no humans can do, and no humans can do quickly.
As for the most likely way to get AGI, the case is quite strong for LRMs with additional RL around stuff like long-term memory and to reduce hallucinations, since those systems are, in many ways, nearly there, and there are no clear barriers to them making it the rest of the way.