Dwarkesh asked a very interesting question in his Sutton interview, which Sutton wasn’t really interested in replying to.
Dwarkesh notes that one idea for why the the bitter lesson was true is because general methods got to ride the wave of exponential computing power while knowledge engineering could not, as labour was comparatively fixed in supply. He then notices that post AGI labour supply will increase at a very rapid pace. And so he wonders, once the labour constraint is solved post AGI will GOFAI make a comeback? For we will then be able to afford the proverbial three billion philosophers writing lisp predicates or whatever various other kinds of high-labour AI techniques are possible.
For the people that reacted Why?/Citation?, the basic reason is that agent foundations has a very similar shaped bottleneck in the fact that we only have 100 at most agent foundation researchers, and even conditioning on agent foundations succeeding, it would require lots more labor than empirical approaches, and due to the human population mostly leveling off and people not being interested in AI alignment, labor is very, very constrained compared to compute.
I don’t actually buy this argument, but I think it’s a very important argument for someone to make, and for people to consider carefully. So thank you to Dwarkesh for proposing it, and to you for mentioning it!
I’ve been writing up a long-form argument for why “Good Old Fashioned AI” (GOFAI) is a hopeless pipedream. I don’t know if that would actually remain true for enormous numbers of superintelligent programmers! But if I had to sketch out the rough form of the argument, it would go something like this:
Your inputs are all “giant, inscrutable matrices”, or rather n-dimensional tensors. Sound, images, video, etc. If you think I’m being unfair calling images “inscrutable tensors”, please take a 4k image as a 2D array of RGB pixels, and write me a Lisp function that counts all the palm trees.
Your outputs are all probability distributions.
The function from your inputs to your outputs is inevitably going to be a giant, inscrutable tensor, probably with a bunch of extra complications stacked on top. (Like ReLU or neural firing patterns.)
Also, the world is vast, categories are fuzzy, and there will always be surprises outside of your training set.
Anyone who has ever been 15 years old knows that protestations of love, checksummed or no, are not to be taken at face value. And even if we wanted to take love out of this example, what would we replace it with? The universe of assertions that Joe might make about Mary is large, but the subset of those assertions that are universally interpretable and uncomplicated is tiny.
So I would argue that GOFAI fails because it simplifies its map of reality to eliminate all the giant tensors, but in doing so, it oversimplifies the inherent slipperiness of problems like “Count the palm trees in this giant pile of numbers” or really any other interesting question about anything.
So my guess is that a giant army of superintelligent programmers could (at enormous cost) build an LLM-like chatbot by hardcoding vast amounts of world knowledge in a way that another superintelligence could understand. But this same chatbot would be helpless before any adversary who could create “out of the training distribution” situations. The very flexibility required to reason about fuzzy-but-important ideas like “someone is screwing with me in a novel way” would require a vast and complex world view that extrapolates well to unfamiliar situations. And that function from inscrutable input tensors to output probability distributions would itself be a giant, inscrutable tensor, plus a bunch of complications that made it harder to understand.
Ha, and I have been writing up a long-form for when AI-coded-GOFAI might become effective, one might even say unreasonably effective. LLMs aren’t very good at learning in environments with very few data samples, such as “learning on the job” or interacting with the slow real world. But there often exist heuristics, ones that are difficult to run on a neural net, with excellent specificity that are capable of proving their predictive power with a small number of examples. You can try to learn the position of the planets by feeding 10,000 examples into a neural network, but you’re much better off with Newton’s laws coded into your ensemble. Data constrained environments (like, again, robots and learning on the job) are domains where the bitter lesson might not have bite.
As a former robotics developer, I feel the bitter lesson in my bones. This is actually one of the points I plan to focus on when I write up the longer version of my argument.
High-quality manual dexterity (and real-time visual processing) in a cluttered environment is a heartbreakingly hard problem, using any version of GOFAI techniques I knew at the time. And even the most basic of the viable algorithms quickly turned into a big steaming pile of linear algebra mixed with calculus.
As someone who has done robotics demos (and who knows all the things an engineer can do to make sure the demos go smoothly), the Figure AI groceries demostill blows my mind. This demo is well into the “6 impossible things before breakfast” territory for me, and I am sure as hell feeling the imminent AGI when I watch it. And I think this version of Figure was an 8B VLLM connected to an 80M specialized motor control model running at 200 Hz? Even if I assume that this is a very carefully run demo showing Figure under ideal circumstances, it’s still black magic fuckery for me.
But it’s really hard to communicate this intuitive reaction to someone who hasn’t spent years working on GOFAI robotics. Some things seem really easy until you actually start typing code into an editor and booting it on actual robot hardware, or until you start trying to train a model. And then these things reveal themselves as heartbreakingly difficult. And so when I see VLLM-based robots that just casually solve these problems, I remember years of watching frustrated PhDs struggle with things that seemed impossibly basic.
For me, “fix a leaky pipe under a real-world, 30-year-old sink without flooding the kitchen, and deal with all the weird things that inevitably go wrong” will be one of my final warning bells of imminent general intelligence. Especially if the same robot can also add a new breaker to the electrical panel and install a new socket in an older house.
I agree with the criticisms of literal GOFAI here, but I can imagine a kind of pseudo-GOFAI agenda plausibly working here. Classical logic is probably hopeless for this for the reasons you outline (real-world fuzziness), but it still seems an open question whether there’s some mathematical formalism with which you can reason about the input-output mapping.
I would gesture at dynamical systems analysis in RNNs, and circuit-based interpretability as the kinds of things that would enable this. For example, perhaps a model has learned to perform addition using a bag of heuristics, and you notice that there’s a better set of heuristics that it didn’t learn for path-dependent training reasons (e.g. clock and pizza). This would then enable the same kind of labor-intensive improvement through explicit reasoning about representations rather than end-to-end training.
It’s not clear to me that this will work, but the challenge is to explicitly articulate which properties of the function from inputs to outputs render it impossible. I don’t think fuzziness alone does it, like in the case of classical logic, because the mathematical structures involved might be compatible with fuzziness. Maybe the mechanisms in your model aren’t “local enough”, in that they play a role across too much of your input distribution to edit without catastrophic knock-on effects. Maybe the mechanisms are intrinsically high dimensional in a way that makes them hard to reason about as mechanisms. And of course, maybe it’s just never more efficient than end-to-end training.
Surely it’s still more efficient to put that labor back into deep learning rather than GOFAI, though, no? In a world where you have enough AI labor to get GOFAI to AGI, you probably also have enough AI labor to get deep-learning-based AGI to superintelligence.
Currently, intellectual labor from machine learning researchers costs a lot of compute. A $1M/year ML researcher costs the same as having 30 or so H100s. At the point where you have AGI, you can probably run the equivalent of one ML researcher with substantially less hardware than that. (I’m amortizing, presumably you’ll be running your models on multiple chips doing inference on multiple requests simultaneously.) This means that some ways to convert intellectual labor into compute efficiency will be cost-effective when they weren’t previously. So I expect that ML will become substantially more labor-intensive and have much more finicky special casing.
For @Random Developer, I agree that literal GOFAI is unlikely to make a comeback, because of the fuzziness problems that arise when you only have finite compute (though some of the GOFAI techniques probably will be reinvented), but I do think a weaker version of GOFAI that drops the pure determinism and embraces probabilistic programming languages (perhaps InfraBayes is involved?) (perhaps causal models as well) but still retains very high interpretability compared to modern AIs that lead to retargeting the search being viable is still likely to be possible.
The key insight I was missing was that while the world’s complexity is very high, so I agree with Random Developer that the complexity of the world is very high, it’s also pretty easy to decompose the complexity into low-complexity parts for specific tasks, and this lets us not need to cram all of the complexity of our world into our memory at once, and we can instead chunk it.
This is the part that convinced me that powerful AI with very interpretable models was possible at all, and the thing that made me update to thinking it’s likely is that the bitter lesson is now pretty easy to explain without invoking any special property of the world/AIs, simply by looking at the labor being constant but compute growing exponentially means that so long as uninterpretable AI is possible at all to scale, it will be invested in, and I’m a big fan of boring hypotheses relative to exciting/deep hypotheses (to human minds. Other minds would find other hypotheses boring, and other hypotheses exciting).
This is in a sense a restatement of the well known theorem that hypotheses that add conjuncts are never more likely than hypotheses that don’t add a conjunct.
How do we define GOFAI here? If we’re contrasting the search/learning based approaches with the sorts of approaches which leverage specialized knowledge in particular domains (as done by Sutton in his The Bitter Lesson), then if the AGI learns anything particular about a field, isn’t that leveraging “specialized knowledge in particular domains”? [1]
Its not clear to me that should be included as AI research, so its not obvious to me the question makes sense. For example, Alpha-zero was not GOFAI, but was its training process “doing” GOFAI, since that training process was creating an expert system using (autonomously gathered) information about the specialized domain of Go?
Maybe we want to say that in order for it to count as AI research, the AI needs to end up creating some new agent or something. Then the argument is more about whether the AI would want to spin up specialized sub-agents or tool-AIs to help it act in certain domains, then we can ask whether when its trying to improve the sub-agents, it will try to hand-code specialized knowledge or general principles.
As with today this seems very much a function of the level of generality of the domain. Note that GOFAI and improvements to GOFAI haven’t really died, they’ve just gotten specialized. See compilers, compression algorithms, object oriented programming, the disease ontology project, and the applications of many optimization & control algorithms.
But note this is different from how most use the term “GOFAI”, by which they mean symbolic AI in contrast to neuro-inspired AI or connectionism. In this case, I expect that the AI we get will not necessarily want to follow either of these two philosophical principles. It will understand how & why DNNs work, eliminate their flaws, and amplify their strengths, and have the theorems (or highly probable heuristic arguments) to prove why its approach is sound.
On one hand, true, on the other, would it be then understandable anyway when it was all written by possibly superhuman AIs working at certainly superhuman speeds without supervision?
I can’t remember the quote, but I believe this possibility is mentioned offhand in IABIED, with the authors suggesting superhuman but still weak AI might do what we can’t and craft rather than grow another AI, to that is can ensure the better successor AI is aligned to its goals.
Dwarkesh asked a very interesting question in his Sutton interview, which Sutton wasn’t really interested in replying to.
Dwarkesh notes that one idea for why the the bitter lesson was true is because general methods got to ride the wave of exponential computing power while knowledge engineering could not, as labour was comparatively fixed in supply. He then notices that post AGI labour supply will increase at a very rapid pace. And so he wonders, once the labour constraint is solved post AGI will GOFAI make a comeback? For we will then be able to afford the proverbial three billion philosophers writing lisp predicates or whatever various other kinds of high-labour AI techniques are possible.
Of course, the same consideration applies to theoretical agent-foundations-style alignment research
For the people that reacted Why?/Citation?, the basic reason is that agent foundations has a very similar shaped bottleneck in the fact that we only have 100 at most agent foundation researchers, and even conditioning on agent foundations succeeding, it would require lots more labor than empirical approaches, and due to the human population mostly leveling off and people not being interested in AI alignment, labor is very, very constrained compared to compute.
I don’t actually buy this argument, but I think it’s a very important argument for someone to make, and for people to consider carefully. So thank you to Dwarkesh for proposing it, and to you for mentioning it!
I’ve been writing up a long-form argument for why “Good Old Fashioned AI” (GOFAI) is a hopeless pipedream. I don’t know if that would actually remain true for enormous numbers of superintelligent programmers! But if I had to sketch out the rough form of the argument, it would go something like this:
Your inputs are all “giant, inscrutable matrices”, or rather n-dimensional tensors. Sound, images, video, etc. If you think I’m being unfair calling images “inscrutable tensors”, please take a 4k image as a 2D array of RGB pixels, and write me a Lisp function that counts all the palm trees.
Your outputs are all probability distributions.
The function from your inputs to your outputs is inevitably going to be a giant, inscrutable tensor, probably with a bunch of extra complications stacked on top. (Like ReLU or neural firing patterns.)
Also, the world is vast, categories are fuzzy, and there will always be surprises outside of your training set.
Clay Shirkey summarized a similar set of issues in the context of the Semantic Web:
So I would argue that GOFAI fails because it simplifies its map of reality to eliminate all the giant tensors, but in doing so, it oversimplifies the inherent slipperiness of problems like “Count the palm trees in this giant pile of numbers” or really any other interesting question about anything.
So my guess is that a giant army of superintelligent programmers could (at enormous cost) build an LLM-like chatbot by hardcoding vast amounts of world knowledge in a way that another superintelligence could understand. But this same chatbot would be helpless before any adversary who could create “out of the training distribution” situations. The very flexibility required to reason about fuzzy-but-important ideas like “someone is screwing with me in a novel way” would require a vast and complex world view that extrapolates well to unfamiliar situations. And that function from inscrutable input tensors to output probability distributions would itself be a giant, inscrutable tensor, plus a bunch of complications that made it harder to understand.
See also Inscrutability was always inevitable, right?
Ha, and I have been writing up a long-form for when AI-coded-GOFAI might become effective, one might even say unreasonably effective.
LLMs aren’t very good at learning in environments with very few data samples, such as “learning on the job” or interacting with the slow real world. But there often exist heuristics, ones that are difficult to run on a neural net, with excellent specificity that are capable of proving their predictive power with a small number of examples. You can try to learn the position of the planets by feeding 10,000 examples into a neural network, but you’re much better off with Newton’s laws coded into your ensemble. Data constrained environments (like, again, robots and learning on the job) are domains where the bitter lesson might not have bite.
As a former robotics developer, I feel the bitter lesson in my bones. This is actually one of the points I plan to focus on when I write up the longer version of my argument.
High-quality manual dexterity (and real-time visual processing) in a cluttered environment is a heartbreakingly hard problem, using any version of GOFAI techniques I knew at the time. And even the most basic of the viable algorithms quickly turned into a big steaming pile of linear algebra mixed with calculus.
As someone who has done robotics demos (and who knows all the things an engineer can do to make sure the demos go smoothly), the Figure AI groceries demo still blows my mind. This demo is well into the “6 impossible things before breakfast” territory for me, and I am sure as hell feeling the imminent AGI when I watch it. And I think this version of Figure was an 8B VLLM connected to an 80M specialized motor control model running at 200 Hz? Even if I assume that this is a very carefully run demo showing Figure under ideal circumstances, it’s still black magic fuckery for me.
But it’s really hard to communicate this intuitive reaction to someone who hasn’t spent years working on GOFAI robotics. Some things seem really easy until you actually start typing code into an editor and booting it on actual robot hardware, or until you start trying to train a model. And then these things reveal themselves as heartbreakingly difficult. And so when I see VLLM-based robots that just casually solve these problems, I remember years of watching frustrated PhDs struggle with things that seemed impossibly basic.
For me, “fix a leaky pipe under a real-world, 30-year-old sink without flooding the kitchen, and deal with all the weird things that inevitably go wrong” will be one of my final warning bells of imminent general intelligence. Especially if the same robot can also add a new breaker to the electrical panel and install a new socket in an older house.
The robots didn’t open the eggs box and individually put them in the rack inside the fridge, obviously crap, not buying the hype. /s
I agree with the criticisms of literal GOFAI here, but I can imagine a kind of pseudo-GOFAI agenda plausibly working here. Classical logic is probably hopeless for this for the reasons you outline (real-world fuzziness), but it still seems an open question whether there’s some mathematical formalism with which you can reason about the input-output mapping.
I would gesture at dynamical systems analysis in RNNs, and circuit-based interpretability as the kinds of things that would enable this. For example, perhaps a model has learned to perform addition using a bag of heuristics, and you notice that there’s a better set of heuristics that it didn’t learn for path-dependent training reasons (e.g. clock and pizza). This would then enable the same kind of labor-intensive improvement through explicit reasoning about representations rather than end-to-end training.
It’s not clear to me that this will work, but the challenge is to explicitly articulate which properties of the function from inputs to outputs render it impossible. I don’t think fuzziness alone does it, like in the case of classical logic, because the mathematical structures involved might be compatible with fuzziness. Maybe the mechanisms in your model aren’t “local enough”, in that they play a role across too much of your input distribution to edit without catastrophic knock-on effects. Maybe the mechanisms are intrinsically high dimensional in a way that makes them hard to reason about as mechanisms. And of course, maybe it’s just never more efficient than end-to-end training.
Surely it’s still more efficient to put that labor back into deep learning rather than GOFAI, though, no? In a world where you have enough AI labor to get GOFAI to AGI, you probably also have enough AI labor to get deep-learning-based AGI to superintelligence.
The way I’d think about this is:
Currently, intellectual labor from machine learning researchers costs a lot of compute. A $1M/year ML researcher costs the same as having 30 or so H100s. At the point where you have AGI, you can probably run the equivalent of one ML researcher with substantially less hardware than that. (I’m amortizing, presumably you’ll be running your models on multiple chips doing inference on multiple requests simultaneously.) This means that some ways to convert intellectual labor into compute efficiency will be cost-effective when they weren’t previously. So I expect that ML will become substantially more labor-intensive and have much more finicky special casing.
For @Random Developer, I agree that literal GOFAI is unlikely to make a comeback, because of the fuzziness problems that arise when you only have finite compute (though some of the GOFAI techniques probably will be reinvented), but I do think a weaker version of GOFAI that drops the pure determinism and embraces probabilistic programming languages (perhaps InfraBayes is involved?) (perhaps causal models as well) but still retains very high interpretability compared to modern AIs that lead to retargeting the search being viable is still likely to be possible.
The key insight I was missing was that while the world’s complexity is very high, so I agree with Random Developer that the complexity of the world is very high, it’s also pretty easy to decompose the complexity into low-complexity parts for specific tasks, and this lets us not need to cram all of the complexity of our world into our memory at once, and we can instead chunk it.
This is the part that convinced me that powerful AI with very interpretable models was possible at all, and the thing that made me update to thinking it’s likely is that the bitter lesson is now pretty easy to explain without invoking any special property of the world/AIs, simply by looking at the labor being constant but compute growing exponentially means that so long as uninterpretable AI is possible at all to scale, it will be invested in, and I’m a big fan of boring hypotheses relative to exciting/deep hypotheses (to human minds. Other minds would find other hypotheses boring, and other hypotheses exciting).
This is in a sense a restatement of the well known theorem that hypotheses that add conjuncts are never more likely than hypotheses that don’t add a conjunct.
So I mostly agree with this hypothesis.
How do we define GOFAI here? If we’re contrasting the search/learning based approaches with the sorts of approaches which leverage specialized knowledge in particular domains (as done by Sutton in his The Bitter Lesson), then if the AGI learns anything particular about a field, isn’t that leveraging “specialized knowledge in particular domains”? [1]
Its not clear to me that should be included as AI research, so its not obvious to me the question makes sense. For example, Alpha-zero was not GOFAI, but was its training process “doing” GOFAI, since that training process was creating an expert system using (autonomously gathered) information about the specialized domain of Go?
Maybe we want to say that in order for it to count as AI research, the AI needs to end up creating some new agent or something. Then the argument is more about whether the AI would want to spin up specialized sub-agents or tool-AIs to help it act in certain domains, then we can ask whether when its trying to improve the sub-agents, it will try to hand-code specialized knowledge or general principles.
As with today this seems very much a function of the level of generality of the domain. Note that GOFAI and improvements to GOFAI haven’t really died, they’ve just gotten specialized. See compilers, compression algorithms, object oriented programming, the disease ontology project, and the applications of many optimization & control algorithms.
But note this is different from how most use the term “GOFAI”, by which they mean symbolic AI in contrast to neuro-inspired AI or connectionism. In this case, I expect that the AI we get will not necessarily want to follow either of these two philosophical principles. It will understand how & why DNNs work, eliminate their flaws, and amplify their strengths, and have the theorems (or highly probable heuristic arguments) to prove why its approach is sound.
On one hand, true, on the other, would it be then understandable anyway when it was all written by possibly superhuman AIs working at certainly superhuman speeds without supervision?
I can’t remember the quote, but I believe this possibility is mentioned offhand in IABIED, with the authors suggesting superhuman but still weak AI might do what we can’t and craft rather than grow another AI, to that is can ensure the better successor AI is aligned to its goals.