My take on how recursion theory failed to be relevant for today’s AI is that it turned out that what a machine could do if unconstrained basically didn’t matter at all, and in particular it basically didn’t matter what limits an ideal machine could do, because once we actually impose constraints that force computation to use very limited amounts of resources, we get a non-trivial theory and importantly all of the difficulty of explaining how humans do stuff lies here.
There was too much focus on “could a machine do something at all?” and not enough focus on “what could a machine with severe limitations could do?”
The reason is that in a sense, it is trivial to solve any problem with a machine if I’m allowed zero constraints except that the machine has to exist in a mathematical sense.
A good example of this is the paper on A Universal Hypercomputer, which shows how absurdly powerful computation can be if you are truly unconstrained:
There are other problems, like embeddedness issues, but this turned out to be a core issue, and it turned out that the constants and constraints mattered more than recursion theory thought.
I’ll flag here that while it’s probably true that a future paradigm will involve more online learning/continual learning, LLMs currently don’t do this, and after they’re trained their weights/neurons are very much frozen. Indeed, I think this is a central reason why LLMs currently underperform their benchmarks, and is why I don’t expect pure LLMs to work out as a paradigm for AGI in practice:
Unfortunately, trained neural networks are the things that perform learning and inference online, during deployment. It seems like a bad sign for alignment if we can only understand the behavior of A.G.I. indirectly.
On the argument against simple algorithms that learn generally and have high performance existing, I think a key assumption of LW is that our physics is pretty simple in Kolmogorov complexity (perhaps excluding the constants), and if we buy this, then the true complexity of the world is upper bounded low enough that glass-box learners have real use.
That doesn’t mean they have to be findable easily, though.
IMO, the most successful next paradigm will require us to much more strongly focus on the behavioral properties, rather than trying to hope for mechanistic interpretability success, and we will likely have to focus on generalizable statements about the input/output behavior, and mostly ignore the model’s structure, since it’s probably going to be too messy to work with by default.
This is why I’m more bullish on Vanessa Kosoy’s IBP than basically any other theoretical agenda except Natural Abstractions.
My take on how recursion theory failed to be relevant for today’s AI is that it turned out that what a machine could do if unconstrained basically didn’t matter at all, and in particular it basically didn’t matter what limits an ideal machine could do, because once we actually impose constraints that force computation to use very limited amounts of resources, we get a non-trivial theory and importantly all of the difficulty of explaining how humans do stuff lies here.
That’s partially true (computational complexity is now much more active than recursion theory), but it doesn’t explain the failure of the Penrose-Lucas argument.
I’ll flag here that while it’s probably true that a future paradigm will involve more online learning/continual learning, LLMs currently don’t do this, and after they’re trained their weights/neurons are very much frozen.
While their weights are frozen and they don’t really do continual learning, they do online learning.
On the argument against simple algorithms that learn generally and have high performance existing, I think a key assumption of LW is that our physics is pretty simple in Kolmogorov complexity (perhaps excluding the constants), and if we buy this, then the true complexity of the world is upper bounded low enough that glass-box learners have real use.
Well, it’s not a key assumption of mine :)
I mostly agree with your observation, but I also think it’s funny when people say this kind of thing to me: I’ve been reading lesswrong for like an hour a day for years (maybe, uh, more than an hour if I’m being honest...), and I produce a non-negligible fraction of the rationality/epistemics content. So—the fact that this is sometimes taken as an assumption on lesswrong doesn’t seem to move me very much—I wonder if you mean to accept or to question this assumption? Anyway, I don’t mean this as a criticism of your raising the point.
Physics aside, I think that when you take indexical complexity into account, it’s not true.
but it doesn’t explain the failure of the Penrose-Lucas argument.
The failure of the Penrose-Lucas argument is that Godel’s incompleteness theorem doesn’t let you derive the conclusion he derived, because it only implies that you cannot use a computably enumerable set of axioms to make all of mathematics sound and complete, and critically this doesn’t mean you cannot automate a subset of mathematics that is relevant.
There’s an argument really close to this with the Chinese Room where I pointed out that intuitions from our own reality that include lots of constraints fundamentally fail to transfer over to hypotheticals, and this is a really important example of why arguments around AI need to actually attend to the constraints that are relevant in specific worlds, because without them it’s trivial to have strong AI solve any problem:
Really, a lot of the issues with the arguments against strong AI made by philosophers is that they have no sense of scale/sense of what mathematical theorems are actually saying, and thus fail to understand what’s actually been said, combined with way overextrapolating their intuitions into cases where the intuitions have been deliberately made to fail to work.
While their weights are frozen and they don’t really do continual learning, they do online learning.
While I agree In-context learning does give them some form of online learning, which at least partially explains why LLMs succeed (combined with their immense amount of data and muddling through extremely data inefficient algorithms compared to brains, which is a known weakness that could plausibly lead to the death of pure LLM scaling by 2028-2030, though note that doesn’t necessarily mean timelines get that much longer), this currently isn’t enough to automate lots of jobs away, and fairly critically might not be good enough in practice with realistic compute and data constraints to compete with better continual learning algorithms.
To be clear, this doesn’t mean any future paradigm will be more understandable, because they will use more online/continual learning than current LLMs.
Well, it’s not a key assumption of mine :)
I mostly agree with your observation, but I also think it’s funny when people say this kind of thing to me: I’ve been reading lesswrong for like an hour a day for years (maybe, uh, more than an hour if I’m being honest...), and I produce a non-negligible fraction of the rationality/epistemics content. So—the fact that this is sometimes taken as an assumption on lesswrong doesn’t seem to move me very much—I wonder if you mean to accept or to question this assumption? Anyway, I don’t mean this as a criticism of your raising the point.
Physics aside, I think that when you take indexical complexity into account, it’s not true.
I’m mostly in the “pointing out the assumption exists and is used widely”, rather than questioning or accepting it, because I wanted to understand how MIRI could believe simple algorithms that learn generally and have high performance existing despite the theorem, and this was the first thing that came to my mind.
And while AIXI is a useful toy model of how intelligence works, it’s treatment of indexical complexity/the first person view as being privileged is an area where I seriously start to departure from it, and one of the biggest reasons I’m much more of a fan of IBP than AIXI is that it actually makes a serious effort to actually remove the first-person privilege often seen in many frameworks of intelligence.
And this actually helps us to pretty much entirely defuse the Solomonoff prior’s potential malignness, because we no longer have immense probability mass on malignant simulation hypotheses, and the simulation hypotheses that do get considered can’t trick the IBP agent into changing it’s values.
So the indexical complexity isn’t actually relevant here, thankfully (though this doesn’t guarantee that the world is truly low complexity in the sense necessary for LW’s view of simple, general high performing algorithms to actually work).
And the nice thing is that it no longer requires montonic preferences:
Whether you use AIXI or IBP, a continual learning algorithm must contend with indexical uncertainty, which means it must contend with indexical complexity in some fashion.
As far as I understand, IBP tries to evaluate hypotheses according to the complexity of the laws of physics, not the bridge transformation (or indexical) information. But that cannot allow it to overcome the fundamental limitations of the first-person perspective faced by an online learner as proved by Shane Legg. That’s a fact about the difficulty of the problem, not a feature (or bug) of the solution.
I agree that Solomonoff induction faces a potential malign prior problem and i default to believing Vanessa that IBP solves this.
By the way, Scott Aaronson has also made this rebuttal to Searle in his paper on why philosophers should care about computational complexity.
Re Scott Aaronson’s rebuttal, I remember he focused on computational complexity, not indexical complexity, and while I think his paper is very useful and makes good points, strictly speaking it doesn’t directly address the argument, while my comment above does.
Though I do think his paper is underrated amongst philosophers (barring the parts about how complexity theory relates to our specific universe).
But that cannot allow it to overcome the fundamental limitations of the first-person perspective faced by an online learner as proved by Shane Legg. That’s a fact about the difficulty of the problem, not a feature (or bug) of the solution.
My question which I forgot to ask is why can’t we reduce the first-person perspective to the third person perspective and have our lower bounds be based on the complexity of the world, rather than the agent?
My take on how recursion theory failed to be relevant for today’s AI is that it turned out that what a machine could do if unconstrained basically didn’t matter at all, and in particular it basically didn’t matter what limits an ideal machine could do, because once we actually impose constraints that force computation to use very limited amounts of resources, we get a non-trivial theory and importantly all of the difficulty of explaining how humans do stuff lies here.
There was too much focus on “could a machine do something at all?” and not enough focus on “what could a machine with severe limitations could do?”
The reason is that in a sense, it is trivial to solve any problem with a machine if I’m allowed zero constraints except that the machine has to exist in a mathematical sense.
A good example of this is the paper on A Universal Hypercomputer, which shows how absurdly powerful computation can be if you are truly unconstrained:
https://arxiv.org/abs/1806.08747
Or davidad’s comment on how every optimization problem is trivial under worst-case scenarios:
https://www.lesswrong.com/posts/yTvBSFrXhZfL8vr5a/worst-case-thinking-in-ai-alignment#N3avtTM3ESH4KHmfN
There are other problems, like embeddedness issues, but this turned out to be a core issue, and it turned out that the constants and constraints mattered more than recursion theory thought.
I’ll flag here that while it’s probably true that a future paradigm will involve more online learning/continual learning, LLMs currently don’t do this, and after they’re trained their weights/neurons are very much frozen. Indeed, I think this is a central reason why LLMs currently underperform their benchmarks, and is why I don’t expect pure LLMs to work out as a paradigm for AGI in practice:
https://www.lesswrong.com/posts/deesrjitvXM4xYGZd/metr-measuring-ai-ability-to-complete-long-tasks#hSkQG2N8rkKXosLEF
On the argument against simple algorithms that learn generally and have high performance existing, I think a key assumption of LW is that our physics is pretty simple in Kolmogorov complexity (perhaps excluding the constants), and if we buy this, then the true complexity of the world is upper bounded low enough that glass-box learners have real use.
That doesn’t mean they have to be findable easily, though.
IMO, the most successful next paradigm will require us to much more strongly focus on the behavioral properties, rather than trying to hope for mechanistic interpretability success, and we will likely have to focus on generalizable statements about the input/output behavior, and mostly ignore the model’s structure, since it’s probably going to be too messy to work with by default.
This is why I’m more bullish on Vanessa Kosoy’s IBP than basically any other theoretical agenda except Natural Abstractions.
That’s partially true (computational complexity is now much more active than recursion theory), but it doesn’t explain the failure of the Penrose-Lucas argument.
While their weights are frozen and they don’t really do continual learning, they do online learning.
Well, it’s not a key assumption of mine :)
I mostly agree with your observation, but I also think it’s funny when people say this kind of thing to me: I’ve been reading lesswrong for like an hour a day for years (maybe, uh, more than an hour if I’m being honest...), and I produce a non-negligible fraction of the rationality/epistemics content. So—the fact that this is sometimes taken as an assumption on lesswrong doesn’t seem to move me very much—I wonder if you mean to accept or to question this assumption? Anyway, I don’t mean this as a criticism of your raising the point.
Physics aside, I think that when you take indexical complexity into account, it’s not true.
The failure of the Penrose-Lucas argument is that Godel’s incompleteness theorem doesn’t let you derive the conclusion he derived, because it only implies that you cannot use a computably enumerable set of axioms to make all of mathematics sound and complete, and critically this doesn’t mean you cannot automate a subset of mathematics that is relevant.
There’s an argument really close to this with the Chinese Room where I pointed out that intuitions from our own reality that include lots of constraints fundamentally fail to transfer over to hypotheticals, and this is a really important example of why arguments around AI need to actually attend to the constraints that are relevant in specific worlds, because without them it’s trivial to have strong AI solve any problem:
https://www.lesswrong.com/posts/zxLbepy29tPg8qMnw/refuting-searle-s-wall-putnam-s-rock-and-johnson-s-popcorn#wbBQXmE5aAfHirhZ2
Really, a lot of the issues with the arguments against strong AI made by philosophers is that they have no sense of scale/sense of what mathematical theorems are actually saying, and thus fail to understand what’s actually been said, combined with way overextrapolating their intuitions into cases where the intuitions have been deliberately made to fail to work.
While I agree In-context learning does give them some form of online learning, which at least partially explains why LLMs succeed (combined with their immense amount of data and muddling through extremely data inefficient algorithms compared to brains, which is a known weakness that could plausibly lead to the death of pure LLM scaling by 2028-2030, though note that doesn’t necessarily mean timelines get that much longer), this currently isn’t enough to automate lots of jobs away, and fairly critically might not be good enough in practice with realistic compute and data constraints to compete with better continual learning algorithms.
To be clear, this doesn’t mean any future paradigm will be more understandable, because they will use more online/continual learning than current LLMs.
I’m mostly in the “pointing out the assumption exists and is used widely”, rather than questioning or accepting it, because I wanted to understand how MIRI could believe simple algorithms that learn generally and have high performance existing despite the theorem, and this was the first thing that came to my mind.
And while AIXI is a useful toy model of how intelligence works, it’s treatment of indexical complexity/the first person view as being privileged is an area where I seriously start to departure from it, and one of the biggest reasons I’m much more of a fan of IBP than AIXI is that it actually makes a serious effort to actually remove the first-person privilege often seen in many frameworks of intelligence.
And this actually helps us to pretty much entirely defuse the Solomonoff prior’s potential malignness, because we no longer have immense probability mass on malignant simulation hypotheses, and the simulation hypotheses that do get considered can’t trick the IBP agent into changing it’s values.
So the indexical complexity isn’t actually relevant here, thankfully (though this doesn’t guarantee that the world is truly low complexity in the sense necessary for LW’s view of simple, general high performing algorithms to actually work).
And the nice thing is that it no longer requires montonic preferences:
https://www.lesswrong.com/posts/dPmmuaz9szk26BkmD/vanessa-kosoy-s-shortform?commentId=kDaAdjc3YbnuS2ssp
https://www.lesswrong.com/posts/DobZ62XMdiPigii9H/non-monotonic-infra-bayesian-physicalism
Whether you use AIXI or IBP, a continual learning algorithm must contend with indexical uncertainty, which means it must contend with indexical complexity in some fashion.
As far as I understand, IBP tries to evaluate hypotheses according to the complexity of the laws of physics, not the bridge transformation (or indexical) information. But that cannot allow it to overcome the fundamental limitations of the first-person perspective faced by an online learner as proved by Shane Legg. That’s a fact about the difficulty of the problem, not a feature (or bug) of the solution.
I agree that Solomonoff induction faces a potential malign prior problem and i default to believing Vanessa that IBP solves this.
By the way, Scott Aaronson has also made this rebuttal to Searle in his paper on why philosophers should care about computational complexity.
Re Scott Aaronson’s rebuttal, I remember he focused on computational complexity, not indexical complexity, and while I think his paper is very useful and makes good points, strictly speaking it doesn’t directly address the argument, while my comment above does.
Though I do think his paper is underrated amongst philosophers (barring the parts about how complexity theory relates to our specific universe).
My question which I forgot to ask is why can’t we reduce the first-person perspective to the third person perspective and have our lower bounds be based on the complexity of the world, rather than the agent?
Because we can’t build agents with a “gods-eye” view of the universe, we can only build agents inside the universe with limited sensors.