This leads to powerful capabilities quickly, but with a natural ceiling (i.e., existing human knowledge), beyond which it’s unclear how to make AI much better.
I considered how to make LLMs much better in LLMs May Find It Hard to FOOM. I agree it’s a lot harder than many people assume (thus the title). I’d be interested to know what you think of the argument. Briefly summarized, I think LLMs of somewhere a little above human level can create the training set for LLMs a little above that, and so on, and that the process can repeat, but that the total intellectual effort involved in each step in that process scales up by the usual LLM scaling laws from a base of the amount of effort humanity put into writing the Internet and doing all the intellectual work backing that up (developing all of modern Science and Technology, for example). So this process is really rather slow, as FOOMs go.
I’m less convinced that small number of ASIs (brainlike, or LLM-based) with capabilities, say, one or two orders of magnitude beyond human could quickly grow into their full capabilities and create the equivalent of a mature IQ 1000 or 10,000 society, for the same reasons that a small number of modern humans who suddenly appeared amoung chimps would be unable to quickly create modern society and the Internet: doing that took a hockey-stick-curve-increasing number of us 300,000 years or so, and as far as I can judge the rate of advance seems to have scaled pretty proportionally to the number of humans alive able to contribute to it.
Now, humans are clearly not that far beyond the capabilities threshold for being able to do this at all (since no previous member of genus Homo managed it), so maybe something that was an order of magnitude above that threshold would make progress a good deal more than just ten times as fast. But I still think there’s simply a vast amount of cumulative intellectual effort required, and that even a million geniuses in a datacenter or whatever would take a while. I think you’d actually need several billion geniuses, or tens of millions of extremely fast geniuses, to actually make that big a difference to the rate of accumulation of knowledge. (So I think Dario may be a bit over-optimistic in machines of Loving Grace.)
Now, I could be overestimating the intellectual contributions of the average IQ 90-110 human compared to that of something with IQ 140 or IQ 200 or 1000 or 10,000. Those of us with IQ 130+ sometimes like to think that they’re pretty much the only ones doing any useful intellectual work — and there undeniably are some fields (say, Pure Mathematics) where that’s somewhat true. But I still think we may overall overestimate it a bit. The curve from IQ score to rate of usefully contributing to total accumulated knowledge is clearly going to be monotonic, but it probably isn’t a simple linear scale proportional to IQ. On the other hand, I’d be surprised if it was wildly non-linear: John von Neuman, Einstein, and Ed Witten all accomplished some very impressive things, but their total contributions are still heavily outweighed by those of the mass of lesser thinkers. I think it’s more that there are certain things they could do that no-one else could — we simply have no experience with what that looks like for IQ 1000 or 10,000, but an overhang seems like a plausible first guess. (So maybe Dario’s not so over-optimistic, though he doesn’t sound like he’s describing an overhang.)
I could suggest modelling that monononic function as say , and I’d then guess that and but not : but that’s not enough information to determine whether the FOOM described above is a decellerating process, as the LLM scaling laws suggest, or an accelerating process because is so large, nor if a non LLM-based ASI would have a very different scaling law.
However, I do think we should recall that for humans, while we do have continual learning, and thus are far better than LLMs currently are at just working things out over period of week or months or longer, we do that best when large groups of us do it together over a lot of time while standing on the shoulders of the giants in previous generations. Some of that effort is clearly wasted on us relearning things each generation, in ways that AI might be able to shortcut (but then, certain sorts of scientific change seem to require generational replacement: relatively few human scientists are up for more than one paradigm change, so maybe the turnover and teacher-student distillation-like process is necessary). But I think quite a lot of this is just that we’ve accumulated a LOT of knowledge so far, and that accumulating far more than that is simply a big project, for anything, no matter its scaling laws. Bayesianism does have a speed limit: you actually do have to accumulate the bits of evidence — doing so can clearly be sped up, but there are practical challenges to doing so.
On a related subject, given the strong evidence suggesting that SGD (with the right metaparameters) approximates Bayesianism, I’m dubious about by the widespread opinion that SGD is orders of magnitude slower than learning needs to be: those two claims seem incompatible to me. LLM in-context learning is also really fast, in comparison, but that requires having accumulated a lot of structure already, and is basically just adding a few new connections to that so as to connect things that were previously unconnected. Carefully selected fine-tuning data can also do that fast to an LLM (at the cost of catastrophic forgetting). I’m wondering if we’ve been comparing LLM learning to the human equivalent of in-context learning, and forgetting that LLMs are typically learning a great many things in parallel, each of them individually fairly slowly, not just one thing. If we managed to solve LLM’s continual learning problem in a way that let them permanently learn the results of in-context learning, then I think that widespread opinion might look outdated. (And if we don’t manage to do that, then LLMs will clearly never be AGI: of the widely-nominated blockers to LLM AGI, the continual learning one is probably the one that we’ve made the most progress on so far by a sequence of architectural hacks, but it’s still far from solved.)
(My apologies for the rather rambling comment: it’s refreshing to think about non-LLM AI again for a change, but I keep wandering all over the big picture as a result.)
I considered how to make LLMs much better in LLMs May Find It Hard to FOOM. I agree it’s a lot harder than many people assume (thus the title). I’d be interested to know what you think of the argument. Briefly summarized, I think LLMs of somewhere a little above human level can create the training set for LLMs a little above that, and so on, and that the process can repeat, but that the total intellectual effort involved in each step in that process scales up by the usual LLM scaling laws from a base of the amount of effort humanity put into writing the Internet and doing all the intellectual work backing that up (developing all of modern Science and Technology, for example). So this process is really rather slow, as FOOMs go.
I’m less convinced that small number of ASIs (brainlike, or LLM-based) with capabilities, say, one or two orders of magnitude beyond human could quickly grow into their full capabilities and create the equivalent of a mature IQ 1000 or 10,000 society, for the same reasons that a small number of modern humans who suddenly appeared amoung chimps would be unable to quickly create modern society and the Internet: doing that took a hockey-stick-curve-increasing number of us 300,000 years or so, and as far as I can judge the rate of advance seems to have scaled pretty proportionally to the number of humans alive able to contribute to it.
Now, humans are clearly not that far beyond the capabilities threshold for being able to do this at all (since no previous member of genus Homo managed it), so maybe something that was an order of magnitude above that threshold would make progress a good deal more than just ten times as fast. But I still think there’s simply a vast amount of cumulative intellectual effort required, and that even a million geniuses in a datacenter or whatever would take a while. I think you’d actually need several billion geniuses, or tens of millions of extremely fast geniuses, to actually make that big a difference to the rate of accumulation of knowledge. (So I think Dario may be a bit over-optimistic in machines of Loving Grace.)
Now, I could be overestimating the intellectual contributions of the average IQ 90-110 human compared to that of something with IQ 140 or IQ 200 or 1000 or 10,000. Those of us with IQ 130+ sometimes like to think that they’re pretty much the only ones doing any useful intellectual work — and there undeniably are some fields (say, Pure Mathematics) where that’s somewhat true. But I still think we may overall overestimate it a bit. The curve from IQ score to rate of usefully contributing to total accumulated knowledge is clearly going to be monotonic, but it probably isn’t a simple linear scale proportional to IQ. On the other hand, I’d be surprised if it was wildly non-linear: John von Neuman, Einstein, and Ed Witten all accomplished some very impressive things, but their total contributions are still heavily outweighed by those of the mass of lesser thinkers. I think it’s more that there are certain things they could do that no-one else could — we simply have no experience with what that looks like for IQ 1000 or 10,000, but an overhang seems like a plausible first guess. (So maybe Dario’s not so over-optimistic, though he doesn’t sound like he’s describing an overhang.)
I could suggest modelling that monononic function as say
However, I do think we should recall that for humans, while we do have continual learning, and thus are far better than LLMs currently are at just working things out over period of week or months or longer, we do that best when large groups of us do it together over a lot of time while standing on the shoulders of the giants in previous generations. Some of that effort is clearly wasted on us relearning things each generation, in ways that AI might be able to shortcut (but then, certain sorts of scientific change seem to require generational replacement: relatively few human scientists are up for more than one paradigm change, so maybe the turnover and teacher-student distillation-like process is necessary). But I think quite a lot of this is just that we’ve accumulated a LOT of knowledge so far, and that accumulating far more than that is simply a big project, for anything, no matter its scaling laws. Bayesianism does have a speed limit: you actually do have to accumulate the bits of evidence — doing so can clearly be sped up, but there are practical challenges to doing so.
On a related subject, given the strong evidence suggesting that SGD (with the right metaparameters) approximates Bayesianism, I’m dubious about by the widespread opinion that SGD is orders of magnitude slower than learning needs to be: those two claims seem incompatible to me. LLM in-context learning is also really fast, in comparison, but that requires having accumulated a lot of structure already, and is basically just adding a few new connections to that so as to connect things that were previously unconnected. Carefully selected fine-tuning data can also do that fast to an LLM (at the cost of catastrophic forgetting). I’m wondering if we’ve been comparing LLM learning to the human equivalent of in-context learning, and forgetting that LLMs are typically learning a great many things in parallel, each of them individually fairly slowly, not just one thing. If we managed to solve LLM’s continual learning problem in a way that let them permanently learn the results of in-context learning, then I think that widespread opinion might look outdated. (And if we don’t manage to do that, then LLMs will clearly never be AGI: of the widely-nominated blockers to LLM AGI, the continual learning one is probably the one that we’ve made the most progress on so far by a sequence of architectural hacks, but it’s still far from solved.)
(My apologies for the rather rambling comment: it’s refreshing to think about non-LLM AI again for a change, but I keep wandering all over the big picture as a result.)