what reasons do I have to believe that Eliezer Yudkowsky has good reasons to believe that some algorithm could undergo explosive recursive self-improvement?
I approach the question this way: consider the set S of algorithms capable of creating intelligent systems.
Thus far, the only member of S we know about is natural selection… call that S1.
There are several possibilities:
Human minds aren’t in S at all. That means humans can’t produce any AI.
Human minds are in S… call us S2.… but are not significantly better at creating intelligences than natural selection is: S2 ⇐ S1. That means humans can’t produce superhuman AI.
S2 > S1. That means humans can produce superhuman AI.
Given 1 or 2, recursive self-improvement isn’t gonna happen. Given 3: now consider a superhuman AI created by humans. Is it a member of S?
Again, three possibilities: not in S, S3 > S2, or S3 ⇐ S2. I can’t see why a human-created superhuman AI would necessarily be incapable of doing any particular thing that human intelligences can do, so (S3 > S2) seems pretty likely given (S2 > S1).
Lather, rinse, repeat: each generation is smarter than the generation before.
So it seems to me that, given superhuman AI, self-optimizing AI is pretty likely. But I don’t know how likely superhuman AI—or even AI at all—is. We may just not be smart enough to build intelligent systems.
I wouldn’t count on it, though. We’re pretty clever monkeys.
As for “explosive”… well, that’s just asking how long a generation takes. And, geez, I dunno. How long does it take to develop a novel algorithm for producing intelligence? Maybe centuries, in which case the bootstrapping process will take millenia. Maybe minutes, in which case we get something orders of magnitude smarter than us by lunchtime.
Of course, at some point returns presumably diminish… that is, there’s a point where each more-intelligent generation takes too long to generate. But it would be remarkable if humans happened to be anywhere near the top of that slope today.
An argument that is often mentioned is the relatively small difference between chimpanzees and humans. But that huge effect, increase in intelligence, rather seems like an outlier and not the rule. Take for example the evolution of echolocation, it seems to have been a gradual progress with no obvious quantum leaps. The same can be said about eyes and other features exhibited by biological agents.
Is it reasonable to assume that such quantum leaps are the rule, based on a single case study?
Maybe the fact that those other examples aren’t intelligence supports the original argument that intelligence works in quantum leaps.
You can even take examples from within humanity, the smartest humans are capable of things far beyond the dumbest (I doubt even a hundred village idiots working together could do what Einstein managed), and in this case there is not even any difference in brain size or speed.
Maybe the fact that those other examples aren’t intelligence supports the original argument that intelligence works in quantum leaps.
Why didn’t it happen before then? Are there animals that are vastly more intelligent than their immediate predecessors? I don’t see any support for the conclusion that what happened between us and our last common ancestor with the great apes is something that happens often.
You can even take examples from within humanity, the smartest humans are capable of things far beyond the dumbest...
I don’t think this is much supported. You would have to account for different upbringing, education, culturual and environmental differences and a lot of dumb luck. And even the smartest humans are dwarfs standing on the shoulders of giants. Sometimes the time is simply ripe, thanks to the previous discoveries of unknown unknowns.
Why didn’t it happen before then? Are there animals that are vastly more intelligent than their immediate predecessors? I don’t see any support for the conclusion that what happened between us and our last common ancestor with the great apes is something that happens often.
Sure. But that’s isn’t so much evidence for intelligence not being a big deal as it is that there might be very few paths of increasing intelligence which are also increasing fitness. Intelligence takes a lot of resources and most life-forms don’t exist in nutrition rich and calorie rich environments.
But there is other evidence to support your claim. There are other species that are almost as intelligent as humans (e.g. dolphins and elephants) that have not done much with it. So one might say that the ability to make tools is a useful one also and that humans had better toolmaking appendages. However, even this isn’t satisfactory since even separate human populations have remained in close to stasis for hundreds of thousands of years, and the primary hallmarks of civilization such as writing and permanent settlements only arose a handful of times.
You would have to account for different upbringing, education, culturual and environmental differences and a lot of dumb luck.
I don’t think this is relevant to most of Benelliot’s point. Upbringing, education, culture, and environment all impact eventual intelligence for humans because we are very malleable creatures. Ben’s remark commented on the difference between smart and dumb humans, not the difference between those genetically predisposed to be smarter or dumber (which seems to be what your remark is responding to).
Take for example the evolution of echolocation, it seems to have been a gradual progress with no obvious quantum leaps. The same can be said about eyes and other features exhibited by biological agents.
Yes, but these are features produced by evolution. Evolution doesn’t work very much the same, and any AI would likely start with much of human knowledge already given.
Yes, but these are features produced by evolution.
There is a significant difference between intelligence and evolution if you apply intelligence to the improvement of evolutionary designs. But when it comes to unknown unknowns, what difference is there between intelligence and evolution? The only difference then seems to be that intelligence is goal-oriented, can think ahead and jump fitness gaps. Yet the critical similarity is that both rely on dumb luck when it comes to genuine novelty. And where else but when it comes to the dramatic improvement of intelligence does it take the discovery of novel unknown unknowns?
A basic argument supporting the risks from superhuman intelligence is that we don’t know what it could possible come up with. That is why we call it a ‘Singularity’. But why does nobody ask how it knows what it could possible come up with?
It seems to be an unquestioned assumption that intelligence is kind of a black box, a cornucopia that can sprout an abundance of novelty. But this implicitly assumes that if you increase intelligence you also decrease the distance between discoveries. I don’t see that...
The only difference then seems to be that intelligence is goal-oriented, can think ahead and jump fitness gaps
seems to merit a response of “So, other than that, Mrs. Lincoln, how was the play?” Those are all very large differences. Let me add to the list: Intelligence can engage in direct experimentation. Intelligence can also observe and incorporate solutions that other optimizing agents (intelligent or not) have used for similar situations. All of these seem to be distinctions that make intelligence very different from other evolution. It isn’t an accident that the technologies which have been most successful for humans such as writing are technologies which augment many of these different advantages that intelligence has over evolution.
It isn’t an accident that the technologies which have been most successful for humans such as writing are technologies which augment many of these different advantages that intelligence has over evolution.
I agree. To be clear, my confusion is mainly about the possibility of explosive recursive self-improvement. I have a hard time to accept that it is very likely (e.g. easily larger than a 1% probability), that such a thing is practically and effectively possible, or at least that we will be able to come up with an algorithm that is capable of quickly surpassing a human set of skills without huge amounts of hard-coded intelligence. I am skeptical that we will be able to quickly approach such a problem, that it won’t be a slow and incremental evolution slowly approaching superhuman intelligence.
As I see it, the more abstract a seed AI is, the closer it is to something like AIXI, the more time it will need to reach human level intelligence, let alone superhuman intelligence. The less abstract a seed AI is, the more work we will have to put into painstakingly hard-coding it to be able to help us improve its intelligence even further. And in any case, I don’t think that dramatic quantum leaps in intelligence are a matter of speed improvements or the accumulation of expert systems. It might very well need some genuine novelty in the form of the discovery of unknown unknowns.
What is intelligence? Take a chess computer, it is arguably intelligent. It is a narrow form of intelligence. But what is it that differentiates narrow intelligence from general intelligence? Is it a conglomerate of expertise, some sort of conceptual revolution or a special kind of expert system that is missing? My point is, why haven’t we seen any of our expert systems come up with true novelty in their field, something no human has thought of before? The only algorithms that have so far been capable of achieving this have been of evolutionary nature, not what we would label artificial intelligence.
Intelligence can also observe and incorporate solutions that other optimizing agents (intelligent or not) have used for similar situations.
Evolution was able to come up with altruism, something that works two levels above the individual and one level above society. So far we haven’t been able to show such ingenuity by incorporating successes that are not evident from an individual or even societal position.
Your point is a good one, I am just saying that the gap between intelligence and evolution isn’t that big here.
Let me add to the list, intelligence can engage in direct experimentation.
Yes, but evolution makes better use of dumb luck by being blindfolded. This seems to be a disadvantage but actually allows it to discover unknown unknowns that are hidden where no intelligent, rational agent would suspect them and therefore would never find them given evidence based exploration.
Yes, but evolution makes better use of dumb luck by being blindfolded. This seems to be a disadvantage but actually allows it to discover unknown unknowns that are hidden where no intelligent, rational agent would suspect them and therefore would never find them given evidence based exploration
Never is a very strong word and it isn’t obvious that evolution will actually find things that intelligence would not. The general scale that evolution gets to work at is much longer term than intelligence has so far. If intelligence has as much time to fiddle it might be able to do everything evolution can (indeed, intelligence can even co-opt evolution by means of genetic algorithms). But, this doesn’t impact your main point in so far as if intelligent were to need those sorts of time scales then one obviously wouldn’t have an intelligence explosion.
Is it clear that the discovery of intelligence by evolution had a larger impact than the discovery of eyes? What evidence do we have that increasing intelligence itself outweighs its cost compared to adding a new pair of sensors?
What I am asking is how we can be sure that it would be instrumental for an AGI to increase its intelligence rather than using its existing intelligence to pursue its terminal goal? Do we have good evidence that the resources that are necessary to increase intelligence outweigh the cost of being unable to use those resources to pursue its terminal goal directly?
My main point regarding the advantage of being “irrational” was that if we would all think like perfect rational agents, e.g. closer to how Eliezer Yudkowsky thinks, we would have missed out on a lot of discoveries that were made by people pursuing “Rare Disease for Cute Kitten” activities.
How much of what we know was actually the result of people thinking quantitatively and attending to scope, probability, and marginal impacts? How much of what we know today is the result of dumb luck versus goal-oriented, intelligent problem solving?
What evidence do we have that the payoff of intelligent, goal-oriented experimentation yields enormous advantages over evolutionary discovery relative to its cost? What evidence do we have that any increase in intelligence does vastly outweigh its computational cost and the expenditure of time needed to discover it?
Evolution acting on intelligent agents has been able to do quite a bit of that for millions of years, though—for example via the topic I am forbidden to mention.
I do not doubt that humans can create superhuman AI, but I don’t know how likely self-optimizing AI is. I am aware of the arguments. But all those arguments rather seem to me like theoretical possibilities, just like universal Turing machines could do everything a modern PC could do and much more. But in reality that just won’t work because we don’t have infinite tapes, infinite time...
Applying intelligence to itself effectively seems problematic. I might just have to think about it in more detail. But intutively it seems that you need to apply a lot more energy to get a bit more complexity. That is, humans can create superhuman intelligence but you need a lot of humans working on it for a long time and have a lot of luck stumbling upon unknown unknowns.
It is argued that the mind-design space must be large if evolution could stumble upon general intelligence. I am not sure how valid that argument is, but even if that is the case, shouldn’t the mind-design space reduce dramatically with every iteration and therefore demand a lot more time to stumble upon new solutions?
Another problem I have is that I don’t get why people here perceive intelligence to be something proactive with respect to itself. No doubt there exists some important difference between evolutionary processes and intelligence. But if you apply intelligence to itself, this difference seems to diminish. How so? Because intelligence is no solution in itself, it is merely an effective searchlight for unknown unknowns. But who knows that the brightness of the light increases proportionally with the distance between unknown unknowns? To have an intelligence explosion the light would have to reach out much farther with each generation than the increase of the distance between unknown unknowns...I just don’t see that to be a reasonable assumption.
I do not doubt that humans can create superhuman AI, but I don’t know how likely self-optimizing AI is.
What appears to be a point against the idea:
While we have proven that very powerful prediction algorithms which can learn
to predict these sequences exist, we have also proven that, unfortunately, mathematical analysis cannot be used to discover these algorithms due to problems of Godel incompleteness.
Check with definition 2.4. In the technical sense used in the document, a predictor is not defined as being something that outputs the sequence—it is defined as something that eventually learns how to predict the sequence—making at most a finite number of errors.
Strings with high Kolmogorov complexity being “predicted” by trivial algorithms is quite compatible with this notion of “prediction”.
Here’s an example from the paper that helps illustrate the difference: if the sequence is a gigabyte of random data repeated forever, it can be predicted with finitely many errors by the simple program “memorize the first gigabyte of data and then repeat it forever”, though the sequence itself has high K-complexity.
It looks like you just dislike the definitions in the paper and want to replace them with your own. I’m not sure there’s any point in arguing about that.
I approach the question this way: consider the set S of algorithms capable of creating intelligent systems.
Thus far, the only member of S we know about is natural selection… call that S1.
There are several possibilities:
Human minds aren’t in S at all. That means humans can’t produce any AI.
Human minds are in S… call us S2.… but are not significantly better at creating intelligences than natural selection is: S2 ⇐ S1. That means humans can’t produce superhuman AI.
S2 > S1. That means humans can produce superhuman AI.
Given 1 or 2, recursive self-improvement isn’t gonna happen.
Given 3: now consider a superhuman AI created by humans. Is it a member of S?
Again, three possibilities: not in S, S3 > S2, or S3 ⇐ S2.
I can’t see why a human-created superhuman AI would necessarily be incapable of doing any particular thing that human intelligences can do, so (S3 > S2) seems pretty likely given (S2 > S1).
Lather, rinse, repeat: each generation is smarter than the generation before.
So it seems to me that, given superhuman AI, self-optimizing AI is pretty likely. But I don’t know how likely superhuman AI—or even AI at all—is. We may just not be smart enough to build intelligent systems.
I wouldn’t count on it, though. We’re pretty clever monkeys.
As for “explosive”… well, that’s just asking how long a generation takes. And, geez, I dunno. How long does it take to develop a novel algorithm for producing intelligence? Maybe centuries, in which case the bootstrapping process will take millenia. Maybe minutes, in which case we get something orders of magnitude smarter than us by lunchtime.
Of course, at some point returns presumably diminish… that is, there’s a point where each more-intelligent generation takes too long to generate. But it would be remarkable if humans happened to be anywhere near the top of that slope today.
An argument that is often mentioned is the relatively small difference between chimpanzees and humans. But that huge effect, increase in intelligence, rather seems like an outlier and not the rule. Take for example the evolution of echolocation, it seems to have been a gradual progress with no obvious quantum leaps. The same can be said about eyes and other features exhibited by biological agents.
Is it reasonable to assume that such quantum leaps are the rule, based on a single case study?
Maybe the fact that those other examples aren’t intelligence supports the original argument that intelligence works in quantum leaps.
You can even take examples from within humanity, the smartest humans are capable of things far beyond the dumbest (I doubt even a hundred village idiots working together could do what Einstein managed), and in this case there is not even any difference in brain size or speed.
Why didn’t it happen before then? Are there animals that are vastly more intelligent than their immediate predecessors? I don’t see any support for the conclusion that what happened between us and our last common ancestor with the great apes is something that happens often.
I don’t think this is much supported. You would have to account for different upbringing, education, culturual and environmental differences and a lot of dumb luck. And even the smartest humans are dwarfs standing on the shoulders of giants. Sometimes the time is simply ripe, thanks to the previous discoveries of unknown unknowns.
Sure. But that’s isn’t so much evidence for intelligence not being a big deal as it is that there might be very few paths of increasing intelligence which are also increasing fitness. Intelligence takes a lot of resources and most life-forms don’t exist in nutrition rich and calorie rich environments.
But there is other evidence to support your claim. There are other species that are almost as intelligent as humans (e.g. dolphins and elephants) that have not done much with it. So one might say that the ability to make tools is a useful one also and that humans had better toolmaking appendages. However, even this isn’t satisfactory since even separate human populations have remained in close to stasis for hundreds of thousands of years, and the primary hallmarks of civilization such as writing and permanent settlements only arose a handful of times.
I don’t think this is relevant to most of Benelliot’s point. Upbringing, education, culture, and environment all impact eventual intelligence for humans because we are very malleable creatures. Ben’s remark commented on the difference between smart and dumb humans, not the difference between those genetically predisposed to be smarter or dumber (which seems to be what your remark is responding to).
Yes, but these are features produced by evolution. Evolution doesn’t work very much the same, and any AI would likely start with much of human knowledge already given.
There is a significant difference between intelligence and evolution if you apply intelligence to the improvement of evolutionary designs. But when it comes to unknown unknowns, what difference is there between intelligence and evolution? The only difference then seems to be that intelligence is goal-oriented, can think ahead and jump fitness gaps. Yet the critical similarity is that both rely on dumb luck when it comes to genuine novelty. And where else but when it comes to the dramatic improvement of intelligence does it take the discovery of novel unknown unknowns?
A basic argument supporting the risks from superhuman intelligence is that we don’t know what it could possible come up with. That is why we call it a ‘Singularity’. But why does nobody ask how it knows what it could possible come up with?
It seems to be an unquestioned assumption that intelligence is kind of a black box, a cornucopia that can sprout an abundance of novelty. But this implicitly assumes that if you increase intelligence you also decrease the distance between discoveries. I don’t see that...
These seem like mainly valid points. However,
seems to merit a response of “So, other than that, Mrs. Lincoln, how was the play?” Those are all very large differences. Let me add to the list: Intelligence can engage in direct experimentation. Intelligence can also observe and incorporate solutions that other optimizing agents (intelligent or not) have used for similar situations. All of these seem to be distinctions that make intelligence very different from other evolution. It isn’t an accident that the technologies which have been most successful for humans such as writing are technologies which augment many of these different advantages that intelligence has over evolution.
I agree. To be clear, my confusion is mainly about the possibility of explosive recursive self-improvement. I have a hard time to accept that it is very likely (e.g. easily larger than a 1% probability), that such a thing is practically and effectively possible, or at least that we will be able to come up with an algorithm that is capable of quickly surpassing a human set of skills without huge amounts of hard-coded intelligence. I am skeptical that we will be able to quickly approach such a problem, that it won’t be a slow and incremental evolution slowly approaching superhuman intelligence.
As I see it, the more abstract a seed AI is, the closer it is to something like AIXI, the more time it will need to reach human level intelligence, let alone superhuman intelligence. The less abstract a seed AI is, the more work we will have to put into painstakingly hard-coding it to be able to help us improve its intelligence even further. And in any case, I don’t think that dramatic quantum leaps in intelligence are a matter of speed improvements or the accumulation of expert systems. It might very well need some genuine novelty in the form of the discovery of unknown unknowns.
What is intelligence? Take a chess computer, it is arguably intelligent. It is a narrow form of intelligence. But what is it that differentiates narrow intelligence from general intelligence? Is it a conglomerate of expertise, some sort of conceptual revolution or a special kind of expert system that is missing? My point is, why haven’t we seen any of our expert systems come up with true novelty in their field, something no human has thought of before? The only algorithms that have so far been capable of achieving this have been of evolutionary nature, not what we would label artificial intelligence.
Evolution was able to come up with altruism, something that works two levels above the individual and one level above society. So far we haven’t been able to show such ingenuity by incorporating successes that are not evident from an individual or even societal position.
Your point is a good one, I am just saying that the gap between intelligence and evolution isn’t that big here.
Yes, but evolution makes better use of dumb luck by being blindfolded. This seems to be a disadvantage but actually allows it to discover unknown unknowns that are hidden where no intelligent, rational agent would suspect them and therefore would never find them given evidence based exploration.
A minor quibble:
Never is a very strong word and it isn’t obvious that evolution will actually find things that intelligence would not. The general scale that evolution gets to work at is much longer term than intelligence has so far. If intelligence has as much time to fiddle it might be able to do everything evolution can (indeed, intelligence can even co-opt evolution by means of genetic algorithms). But, this doesn’t impact your main point in so far as if intelligent were to need those sorts of time scales then one obviously wouldn’t have an intelligence explosion.
I want to expand on my last comment:
Is it clear that the discovery of intelligence by evolution had a larger impact than the discovery of eyes? What evidence do we have that increasing intelligence itself outweighs its cost compared to adding a new pair of sensors?
What I am asking is how we can be sure that it would be instrumental for an AGI to increase its intelligence rather than using its existing intelligence to pursue its terminal goal? Do we have good evidence that the resources that are necessary to increase intelligence outweigh the cost of being unable to use those resources to pursue its terminal goal directly?
My main point regarding the advantage of being “irrational” was that if we would all think like perfect rational agents, e.g. closer to how Eliezer Yudkowsky thinks, we would have missed out on a lot of discoveries that were made by people pursuing “Rare Disease for Cute Kitten” activities.
How much of what we know was actually the result of people thinking quantitatively and attending to scope, probability, and marginal impacts? How much of what we know today is the result of dumb luck versus goal-oriented, intelligent problem solving?
What evidence do we have that the payoff of intelligent, goal-oriented experimentation yields enormous advantages over evolutionary discovery relative to its cost? What evidence do we have that any increase in intelligence does vastly outweigh its computational cost and the expenditure of time needed to discover it?
Evolution acting on intelligent agents has been able to do quite a bit of that for millions of years, though—for example via the topic I am forbidden to mention.
I do not doubt that humans can create superhuman AI, but I don’t know how likely self-optimizing AI is. I am aware of the arguments. But all those arguments rather seem to me like theoretical possibilities, just like universal Turing machines could do everything a modern PC could do and much more. But in reality that just won’t work because we don’t have infinite tapes, infinite time...
Applying intelligence to itself effectively seems problematic. I might just have to think about it in more detail. But intutively it seems that you need to apply a lot more energy to get a bit more complexity. That is, humans can create superhuman intelligence but you need a lot of humans working on it for a long time and have a lot of luck stumbling upon unknown unknowns.
It is argued that the mind-design space must be large if evolution could stumble upon general intelligence. I am not sure how valid that argument is, but even if that is the case, shouldn’t the mind-design space reduce dramatically with every iteration and therefore demand a lot more time to stumble upon new solutions?
Another problem I have is that I don’t get why people here perceive intelligence to be something proactive with respect to itself. No doubt there exists some important difference between evolutionary processes and intelligence. But if you apply intelligence to itself, this difference seems to diminish. How so? Because intelligence is no solution in itself, it is merely an effective searchlight for unknown unknowns. But who knows that the brightness of the light increases proportionally with the distance between unknown unknowns? To have an intelligence explosion the light would have to reach out much farther with each generation than the increase of the distance between unknown unknowns...I just don’t see that to be a reasonable assumption.
What appears to be a point against the idea:
This is from: Is there an Elegant Universal Theory of Prediction?
This is from your link.
But if it can be predicted by a trivial algorithm, it has LOW Kolmogorov complexity.
Check with definition 2.4. In the technical sense used in the document, a predictor is not defined as being something that outputs the sequence—it is defined as something that eventually learns how to predict the sequence—making at most a finite number of errors.
Strings with high Kolmogorov complexity being “predicted” by trivial algorithms is quite compatible with this notion of “prediction”.
So, above the last wrongly predicted output, the whole sequence is as complex as the (improved) predictor?
Here’s an example from the paper that helps illustrate the difference: if the sequence is a gigabyte of random data repeated forever, it can be predicted with finitely many errors by the simple program “memorize the first gigabyte of data and then repeat it forever”, though the sequence itself has high K-complexity.
No it has not. The algorithm for copying the first GB forever is small and the Kolmogorov’s complexity is just over 1GB.
For the entire sequence.
Yes, but the predictor’s complexity is much lower than 1GB.
The paper also gives an example of a single predictor that can learn to predict any eventually periodic sequence, no matter how long the period.
Predictor should remember what happened. It has learned. Now it’s 1 GB heavy.
It looks like you just dislike the definitions in the paper and want to replace them with your own. I’m not sure there’s any point in arguing about that.
I only stick with the Kolmogorov’s definition.