I don’t actually buy this argument, but I think it’s a very important argument for someone to make, and for people to consider carefully. So thank you to Dwarkesh for proposing it, and to you for mentioning it!
I’ve been writing up a long-form argument for why “Good Old Fashioned AI” (GOFAI) is a hopeless pipedream. I don’t know if that would actually remain true for enormous numbers of superintelligent programmers! But if I had to sketch out the rough form of the argument, it would go something like this:
Your inputs are all “giant, inscrutable matrices”, or rather n-dimensional tensors. Sound, images, video, etc. If you think I’m being unfair calling images “inscrutable tensors”, please take a 4k image as a 2D array of RGB pixels, and write me a Lisp function that counts all the palm trees.
Your outputs are all probability distributions.
The function from your inputs to your outputs is inevitably going to be a giant, inscrutable tensor, probably with a bunch of extra complications stacked on top. (Like ReLU or neural firing patterns.)
Also, the world is vast, categories are fuzzy, and there will always be surprises outside of your training set.
Anyone who has ever been 15 years old knows that protestations of love, checksummed or no, are not to be taken at face value. And even if we wanted to take love out of this example, what would we replace it with? The universe of assertions that Joe might make about Mary is large, but the subset of those assertions that are universally interpretable and uncomplicated is tiny.
So I would argue that GOFAI fails because it simplifies its map of reality to eliminate all the giant tensors, but in doing so, it oversimplifies the inherent slipperiness of problems like “Count the palm trees in this giant pile of numbers” or really any other interesting question about anything.
So my guess is that a giant army of superintelligent programmers could (at enormous cost) build an LLM-like chatbot by hardcoding vast amounts of world knowledge in a way that another superintelligence could understand. But this same chatbot would be helpless before any adversary who could create “out of the training distribution” situations. The very flexibility required to reason about fuzzy-but-important ideas like “someone is screwing with me in a novel way” would require a vast and complex world view that extrapolates well to unfamiliar situations. And that function from inscrutable input tensors to output probability distributions would itself be a giant, inscrutable tensor, plus a bunch of complications that made it harder to understand.
Ha, and I have been writing up a long-form for when AI-coded-GOFAI might become effective, one might even say unreasonably effective. LLMs aren’t very good at learning in environments with very few data samples, such as “learning on the job” or interacting with the slow real world. But there often exist heuristics, ones that are difficult to run on a neural net, with excellent specificity that are capable of proving their predictive power with a small number of examples. You can try to learn the position of the planets by feeding 10,000 examples into a neural network, but you’re much better off with Newton’s laws coded into your ensemble. Data constrained environments (like, again, robots and learning on the job) are domains where the bitter lesson might not have bite.
As a former robotics developer, I feel the bitter lesson in my bones. This is actually one of the points I plan to focus on when I write up the longer version of my argument.
High-quality manual dexterity (and real-time visual processing) in a cluttered environment is a heartbreakingly hard problem, using any version of GOFAI techniques I knew at the time. And even the most basic of the viable algorithms quickly turned into a big steaming pile of linear algebra mixed with calculus.
As someone who has done robotics demos (and who knows all the things an engineer can do to make sure the demos go smoothly), the Figure AI groceries demostill blows my mind. This demo is well into the “6 impossible things before breakfast” territory for me, and I am sure as hell feeling the imminent AGI when I watch it. And I think this version of Figure was an 8B VLLM connected to an 80M specialized motor control model running at 200 Hz? Even if I assume that this is a very carefully run demo showing Figure under ideal circumstances, it’s still black magic fuckery for me.
But it’s really hard to communicate this intuitive reaction to someone who hasn’t spent years working on GOFAI robotics. Some things seem really easy until you actually start typing code into an editor and booting it on actual robot hardware, or until you start trying to train a model. And then these things reveal themselves as heartbreakingly difficult. And so when I see VLLM-based robots that just casually solve these problems, I remember years of watching frustrated PhDs struggle with things that seemed impossibly basic.
For me, “fix a leaky pipe under a real-world, 30-year-old sink without flooding the kitchen, and deal with all the weird things that inevitably go wrong” will be one of my final warning bells of imminent general intelligence. Especially if the same robot can also add a new breaker to the electrical panel and install a new socket in an older house.
I agree with the criticisms of literal GOFAI here, but I can imagine a kind of pseudo-GOFAI agenda plausibly working here. Classical logic is probably hopeless for this for the reasons you outline (real-world fuzziness), but it still seems an open question whether there’s some mathematical formalism with which you can reason about the input-output mapping.
I would gesture at dynamical systems analysis in RNNs, and circuit-based interpretability as the kinds of things that would enable this. For example, perhaps a model has learned to perform addition using a bag of heuristics, and you notice that there’s a better set of heuristics that it didn’t learn for path-dependent training reasons (e.g. clock and pizza). This would then enable the same kind of labor-intensive improvement through explicit reasoning about representations rather than end-to-end training.
It’s not clear to me that this will work, but the challenge is to explicitly articulate which properties of the function from inputs to outputs render it impossible. I don’t think fuzziness alone does it, like in the case of classical logic, because the mathematical structures involved might be compatible with fuzziness. Maybe the mechanisms in your model aren’t “local enough”, in that they play a role across too much of your input distribution to edit without catastrophic knock-on effects. Maybe the mechanisms are intrinsically high dimensional in a way that makes them hard to reason about as mechanisms. And of course, maybe it’s just never more efficient than end-to-end training.
I don’t actually buy this argument, but I think it’s a very important argument for someone to make, and for people to consider carefully. So thank you to Dwarkesh for proposing it, and to you for mentioning it!
I’ve been writing up a long-form argument for why “Good Old Fashioned AI” (GOFAI) is a hopeless pipedream. I don’t know if that would actually remain true for enormous numbers of superintelligent programmers! But if I had to sketch out the rough form of the argument, it would go something like this:
Your inputs are all “giant, inscrutable matrices”, or rather n-dimensional tensors. Sound, images, video, etc. If you think I’m being unfair calling images “inscrutable tensors”, please take a 4k image as a 2D array of RGB pixels, and write me a Lisp function that counts all the palm trees.
Your outputs are all probability distributions.
The function from your inputs to your outputs is inevitably going to be a giant, inscrutable tensor, probably with a bunch of extra complications stacked on top. (Like ReLU or neural firing patterns.)
Also, the world is vast, categories are fuzzy, and there will always be surprises outside of your training set.
Clay Shirkey summarized a similar set of issues in the context of the Semantic Web:
So I would argue that GOFAI fails because it simplifies its map of reality to eliminate all the giant tensors, but in doing so, it oversimplifies the inherent slipperiness of problems like “Count the palm trees in this giant pile of numbers” or really any other interesting question about anything.
So my guess is that a giant army of superintelligent programmers could (at enormous cost) build an LLM-like chatbot by hardcoding vast amounts of world knowledge in a way that another superintelligence could understand. But this same chatbot would be helpless before any adversary who could create “out of the training distribution” situations. The very flexibility required to reason about fuzzy-but-important ideas like “someone is screwing with me in a novel way” would require a vast and complex world view that extrapolates well to unfamiliar situations. And that function from inscrutable input tensors to output probability distributions would itself be a giant, inscrutable tensor, plus a bunch of complications that made it harder to understand.
Ha, and I have been writing up a long-form for when AI-coded-GOFAI might become effective, one might even say unreasonably effective.
LLMs aren’t very good at learning in environments with very few data samples, such as “learning on the job” or interacting with the slow real world. But there often exist heuristics, ones that are difficult to run on a neural net, with excellent specificity that are capable of proving their predictive power with a small number of examples. You can try to learn the position of the planets by feeding 10,000 examples into a neural network, but you’re much better off with Newton’s laws coded into your ensemble. Data constrained environments (like, again, robots and learning on the job) are domains where the bitter lesson might not have bite.
As a former robotics developer, I feel the bitter lesson in my bones. This is actually one of the points I plan to focus on when I write up the longer version of my argument.
High-quality manual dexterity (and real-time visual processing) in a cluttered environment is a heartbreakingly hard problem, using any version of GOFAI techniques I knew at the time. And even the most basic of the viable algorithms quickly turned into a big steaming pile of linear algebra mixed with calculus.
As someone who has done robotics demos (and who knows all the things an engineer can do to make sure the demos go smoothly), the Figure AI groceries demo still blows my mind. This demo is well into the “6 impossible things before breakfast” territory for me, and I am sure as hell feeling the imminent AGI when I watch it. And I think this version of Figure was an 8B VLLM connected to an 80M specialized motor control model running at 200 Hz? Even if I assume that this is a very carefully run demo showing Figure under ideal circumstances, it’s still black magic fuckery for me.
But it’s really hard to communicate this intuitive reaction to someone who hasn’t spent years working on GOFAI robotics. Some things seem really easy until you actually start typing code into an editor and booting it on actual robot hardware, or until you start trying to train a model. And then these things reveal themselves as heartbreakingly difficult. And so when I see VLLM-based robots that just casually solve these problems, I remember years of watching frustrated PhDs struggle with things that seemed impossibly basic.
For me, “fix a leaky pipe under a real-world, 30-year-old sink without flooding the kitchen, and deal with all the weird things that inevitably go wrong” will be one of my final warning bells of imminent general intelligence. Especially if the same robot can also add a new breaker to the electrical panel and install a new socket in an older house.
The robots didn’t open the eggs box and individually put them in the rack inside the fridge, obviously crap, not buying the hype. /s
See also Inscrutability was always inevitable, right?
I agree with the criticisms of literal GOFAI here, but I can imagine a kind of pseudo-GOFAI agenda plausibly working here. Classical logic is probably hopeless for this for the reasons you outline (real-world fuzziness), but it still seems an open question whether there’s some mathematical formalism with which you can reason about the input-output mapping.
I would gesture at dynamical systems analysis in RNNs, and circuit-based interpretability as the kinds of things that would enable this. For example, perhaps a model has learned to perform addition using a bag of heuristics, and you notice that there’s a better set of heuristics that it didn’t learn for path-dependent training reasons (e.g. clock and pizza). This would then enable the same kind of labor-intensive improvement through explicit reasoning about representations rather than end-to-end training.
It’s not clear to me that this will work, but the challenge is to explicitly articulate which properties of the function from inputs to outputs render it impossible. I don’t think fuzziness alone does it, like in the case of classical logic, because the mathematical structures involved might be compatible with fuzziness. Maybe the mechanisms in your model aren’t “local enough”, in that they play a role across too much of your input distribution to edit without catastrophic knock-on effects. Maybe the mechanisms are intrinsically high dimensional in a way that makes them hard to reason about as mechanisms. And of course, maybe it’s just never more efficient than end-to-end training.