In particular, it assumes that the initial AI systems are very far from being algorithmically optimal. We don’t know whether or not this will be the case; that is what I am trying to highlight.
Strong recursive self-improvement (given a fixed amount of resources) is only possible if the first AI systems are very far from being algorithmically optimal at all the relevant computational tasks.
Right. But here we are talking about a specific class of tasks (which animals cannot do at all, they can’t write computer code or a mathematical paper; so, evolutionary speaking, human are new at that, and probably not anywhere close to any equilibrium because this class of tasks is very novel on the evolutionary timescale).
Moreover, we know a lot about human performance at those tasks, and it’s abysmal, even for top humans, and for AI research as a field. For example, there has been a paper in Nature in 2000 explaining that ReLU induce semantically meaningful sparseness. The field ignored this for over a decade, then rediscovered in 2009-2011 that ReLU were great, and by 2015 ReLU became dominant. That’s typical, there are plenty of examples like that (obvious discoveries not done for decades (e.g. the ReLU discovery in question was really due in early 1970-s), then further ignored for a decade or longer after the initial publication before being rediscovered and picked up). AI could simply try various things lost in the ignored old papers for a huge boost (there is a lot of actionable, pretty strong stuff in those, not everything gets picked up, like ReLU, a lot remains published, but ignored by the field).
Anyone who has attended an AutoML conference recently knows that the field of AutoML is simply too big to deal with (too many promising methods for neural architecture search, for hyperparameter optimization, for metalearning of optimization algorithms; so we have all these things in metalearning and in “AI-generating algorithms” which can provide a really large boost over status quo if done correctly, but too difficult to fully take advantage of because of human cognitive limitations, as the whole field is just “too large a mess to think effectively about”).
So it seems that, at least, there is quite a bit of room for a large initial boost over the current human-equivalent capacity. If one starts with a level of top AI researcher at a good company today, there are plenty of fixable things simply on the level of being able to consider everything that have been written in this field of study (there is also plenty of room for improvement in terms of being able to rapidly write correct computer code and rapidly try various small-scale ideas) .
It’s still quite possible that recursive self-improvement saturates after a while (it’s difficult to predict how far this would go, how soon it would start hitting difficult-to-overcome bottlenecks). But even the strongest human AI researchers (or strongest human software engineers) we have today are very, very improvable (perhaps, they can be improved by giving them more powerful tools, so that they might still remain in the loop, at least initially; the only thing people seem to be hoping for in this sense is that Copilot-like style of interaction with AIs would allow humans to keep participating in how it goes, so that the systems will remain human-AI hybrid systems, rather than AI-only).
which animals cannot do at all, they can’t write computer code or a mathematical paper
This is not obvious to me (at least not for some senses of the word “could”). Animals cannot be motivated into attempting to solve these tasks, and they cannot study maths or programming. If they could do those things, then it is not at all clear to me that they wouldn’t be able to write code or maths papers. To make this more specific; insofar as humans rely on a capacity for general problem-solving in order to do maths and programming, it would not surprise me if many animals also have this capacity to a sufficient extent, but that it cannot be directed in the right way. Note that animals even outperform humans at some general cognitive tasks. For example, chimps have a much better short-term memory than humans.
Moreover, we know a lot about human performance at those tasks, and it’s abysmal, even for top humans, and for AI research as a field.
Abysmal, compared to what? Yes, we can see that it is abysmal compared to what would in principle be information-theoretically possible. However, this doesn’t tell us very much about whether or not it is abysmal compared to what is computationally possible.
The problem of finding the minimal complexity hypothesis for a given set of data is not computationally tractable. For Kolmogorov complexity, it is uncomputable, but even for Boolean complexity, it is at least exponentially difficult (depending a bit on how exactly the problem is formalised). This means that in order to reason effectively about large amounts of data, it is (presumably) necessary to model most of it using low-fidelity methods, and then (potentially) use various heuristics in order to determine what pieces of information deserve more attention. I would therefore expect a “saturated” AI system to also frequently miss things that look obvious in hindsight.
So it seems that, at least, there is quite a bit of room for a large initial boost over the current human-equivalent capacity.
I agree that AI systems have many clear and obvious advantages, and that e.g. simply running them at a higher clock speed will give you a clear boost regardless of what assumptions we make about the “quality” of their cognition compared to that of humans. The question I’m concerned with is whether or not a takeoff scenario is better modeled as “AI quickly bootstraps to incomprehensible, Godlike intelligence through recursive self-improvement”, or whether it is better modeled as “economic growth suddenly goes up by a lot”. All the obvious advantages of AI systems are compatible with the latter.
Right. But here we are talking about a specific class of tasks (which animals cannot do at all, they can’t write computer code or a mathematical paper; so, evolutionary speaking, human are new at that, and probably not anywhere close to any equilibrium because this class of tasks is very novel on the evolutionary timescale).
Moreover, we know a lot about human performance at those tasks, and it’s abysmal, even for top humans, and for AI research as a field. For example, there has been a paper in Nature in 2000 explaining that ReLU induce semantically meaningful sparseness. The field ignored this for over a decade, then rediscovered in 2009-2011 that ReLU were great, and by 2015 ReLU became dominant. That’s typical, there are plenty of examples like that (obvious discoveries not done for decades (e.g. the ReLU discovery in question was really due in early 1970-s), then further ignored for a decade or longer after the initial publication before being rediscovered and picked up). AI could simply try various things lost in the ignored old papers for a huge boost (there is a lot of actionable, pretty strong stuff in those, not everything gets picked up, like ReLU, a lot remains published, but ignored by the field).
Anyone who has attended an AutoML conference recently knows that the field of AutoML is simply too big to deal with (too many promising methods for neural architecture search, for hyperparameter optimization, for metalearning of optimization algorithms; so we have all these things in metalearning and in “AI-generating algorithms” which can provide a really large boost over status quo if done correctly, but too difficult to fully take advantage of because of human cognitive limitations, as the whole field is just “too large a mess to think effectively about”).
So it seems that, at least, there is quite a bit of room for a large initial boost over the current human-equivalent capacity. If one starts with a level of top AI researcher at a good company today, there are plenty of fixable things simply on the level of being able to consider everything that have been written in this field of study (there is also plenty of room for improvement in terms of being able to rapidly write correct computer code and rapidly try various small-scale ideas) .
It’s still quite possible that recursive self-improvement saturates after a while (it’s difficult to predict how far this would go, how soon it would start hitting difficult-to-overcome bottlenecks). But even the strongest human AI researchers (or strongest human software engineers) we have today are very, very improvable (perhaps, they can be improved by giving them more powerful tools, so that they might still remain in the loop, at least initially; the only thing people seem to be hoping for in this sense is that Copilot-like style of interaction with AIs would allow humans to keep participating in how it goes, so that the systems will remain human-AI hybrid systems, rather than AI-only).
This is not obvious to me (at least not for some senses of the word “could”). Animals cannot be motivated into attempting to solve these tasks, and they cannot study maths or programming. If they could do those things, then it is not at all clear to me that they wouldn’t be able to write code or maths papers. To make this more specific; insofar as humans rely on a capacity for general problem-solving in order to do maths and programming, it would not surprise me if many animals also have this capacity to a sufficient extent, but that it cannot be directed in the right way. Note that animals even outperform humans at some general cognitive tasks. For example, chimps have a much better short-term memory than humans.
Abysmal, compared to what? Yes, we can see that it is abysmal compared to what would in principle be information-theoretically possible. However, this doesn’t tell us very much about whether or not it is abysmal compared to what is computationally possible.
The problem of finding the minimal complexity hypothesis for a given set of data is not computationally tractable. For Kolmogorov complexity, it is uncomputable, but even for Boolean complexity, it is at least exponentially difficult (depending a bit on how exactly the problem is formalised). This means that in order to reason effectively about large amounts of data, it is (presumably) necessary to model most of it using low-fidelity methods, and then (potentially) use various heuristics in order to determine what pieces of information deserve more attention. I would therefore expect a “saturated” AI system to also frequently miss things that look obvious in hindsight.
I agree that AI systems have many clear and obvious advantages, and that e.g. simply running them at a higher clock speed will give you a clear boost regardless of what assumptions we make about the “quality” of their cognition compared to that of humans. The question I’m concerned with is whether or not a takeoff scenario is better modeled as “AI quickly bootstraps to incomprehensible, Godlike intelligence through recursive self-improvement”, or whether it is better modeled as “economic growth suddenly goes up by a lot”. All the obvious advantages of AI systems are compatible with the latter.