You can’t really compare technological designs for which there was no selection pressure and therefore no optimization with superficially similar evolutionary inventions. For example, you would have to compare the energy efficiency with which insects or birds can carry certain amounts of weight with a similar artificial means of transport carrying the same amount of weight. Or you would have to compare the energy efficiency and maneuverability of bird and insect flight with artificial flight. But comparing a train full of hard disk drives with the bandwidth of satellite communication is not useful. Saying that a rocket can fly faster than anything that evolution came up with is not generalizable to intelligence. And if even if I was to accept that argument, then there are many counter-examples. The echolocation of bats, economic photosynthesis or human gait. And the invention of rockets did not led to space colonization either, space exploration is actually retrogressive.
You also mention that human intelligence is primarily responsible for the creation of technology. I do think this is misleading. What is responsible is that we are goal-oriented while evolution is not. But the advance of scientific knowledge is largely an evolutionary process. I don’t see that intelligence is currently tangible enough to measure that the return of increased intelligence is proportional to the resources it would take to amplify it. The argument from the gap between chimpanzees and humans is interesting but can not be used to extrapolate onwards from human general intelligence. It is pure speculation that humans are not Turing complete and that there are levels above our own. That chimpanzees exist, and humans exist, is not a proof for the existence of anything that bears, in any relevant respect, the same relationship to a human that a human bears to a chimpanzee.
It is in principle possible to create artificial intelligence that is as capable as human intelligence. But this says nothing about how quickly we will be able to come up with it. I believe that intelligence is fundamentally dependent on the complexity of the goals against which it is measured. Goals give rise to agency and define an agent’s drives. As long as we won’t be able to precisely hard-code a complexity of values similar to that of humans we won’t achieve levels of general intelligence similar to humans.
It is true that humans have created a lot of tools that help them to achieve their goals. But it is not clear that incorporating those tools into some sort of self-perception, some sort of guiding agency, is superior to humans using a combination of tools and expert systems. In other words, it is not clear that there does exist a class of problems that is solvable by Turing machines in general, but not by a combination of humans and expert systems. And if that was the case then I think that, just like chimpanzees would be unable to invent science, we won’t be able to come up with a meta-heuristic that would allow us to discover algorithms that can solve a class of problems that we can’t (other than by using guided evolution).
Besides, recursive self-improvement does not demand sentience, consciousness or agency. Even if humans are not able to “recursively improve” their own algorithms we can still “recursively improve” our tools. And the supremacy of recursively improving agent’s over humans and their tools is a reasonable conjecture but not a fact. It largely relies on the idea that the integration of tools into a coherent framework of agencies has huge benefits.
I also object to assigning numerical probability estimates to informal arguments and predictions. When faced with data from empirical experiments, or goats behind doors in a gameshow, it is reasonable. But using formalized methods to evaluate informal evidence can be very misleading. For real-world, computationally limited agents it is a recipe to fail spectacularly. Using formalized methods to to evaluate vague ideas like risks from AI can lead you to dramatically over or underestimate evidence by forcing you to use your intuition to assign numbers to your intuitive judgement of informal arguments.
And as a disclaimer: Don’t jump to the conclusion that I generally rule out the possibility that very soon someone will stumble upon a simple algorithm that can be run on a digital computer, that can be improved to self-improve, become superhuman and take over the universe. All am saying is that the possibility isn’t as inevitable as some seem to believe. If forced, I would probably assign a 1% probability to it but still feel uncomfortable about that (which isn’t to equate with risks from AI in general, I don’t think FOOM is required for AI’s to pose a risk).
I think that Eliezer crossed the border of what can sensibly be said about this topic at the present time when he says that AI will likely invent molecular nanotechnology in a matter of hours or days. Jürgen Schmidhuber is the only person I could find who might agree with that. Even Shane Legg is more skeptical. And since I do not yet have the education to evaluate state of the art AI research myself I will side with the experts and say that Eliezer is likely wrong. Of course, I have no authority but I have to make a decision. I don’t feel it would be reasonable to believe Eliezer here without restrictions.
Just because the possibility of superhuman AI seems to be disjunctive on some level doesn’t mean that there are no untested assumptions underlying the claims that such an outcome is possible. Reduce the vagueness and you will discover a set assumptions that need to be true in conjunction.
So, I’m having a lot of difficulty mapping your response to the question I asked. But if I’ve understood your response, you are arguing that technology analogous to the technology-developing functions of human intelligence might not be in principle possible, or that if developed might not be capable of significantly greater technology-developing power than human intelligence is.
In other words, that assumptions 5 and/or 6 might be false.
I agree that it’s possible. Similar things are true of the other examples you give: it’s possible that technological echolocation, or technological walking, or technological photosynthesis, either aren’t possible in principle, or can’t be significantly more powerful than their naturally evolved analogs. (Do you actually believe that to be true of those examples, incidentally?)
This seems to me highly implausible, which is why my confidence for A5 and A6 are very high. (I have similarly high confidence in our ability to develop machines more efficient than human legs at locomotion, machines more efficient at converting sunlight to useful work than plants, and more efficient at providing sonar-based information about their surroundings than bats.)
So, OK. We’ve identified a couple of specific, relevant assertions for which you think that my confidence is too high. Awesome! That’s progress.
So, what level of confidence do you think is justified for those assertions? I realize that you reject assigning numbers to reported confidence, so OK… do you have a preferred way of comparing levels of confidence? Or do you reject the whole enterprise of such comparisons?
Incidentally: you say a lot of other stuff here which seems entirely beside my point… I think because you’re running out ahead to arguments you think I might make some day. I will return to that stuff if I ever actually make an argument to which it’s relevant.
You can’t really compare technological designs for which there was no selection pressure and therefore no optimization with superficially similar evolutionary inventions. For example, you would have to compare the energy efficiency with which insects or birds can carry certain amounts of weight with a similar artificial means of transport carrying the same amount of weight. Or you would have to compare the energy efficiency and maneuverability of bird and insect flight with artificial flight. But comparing a train full of hard disk drives with the bandwidth of satellite communication is not useful. Saying that a rocket can fly faster than anything that evolution came up with is not generalizable to intelligence. And if even if I was to accept that argument, then there are many counter-examples. The echolocation of bats, economic photosynthesis or human gait. And the invention of rockets did not led to space colonization either, space exploration is actually retrogressive.
You also mention that human intelligence is primarily responsible for the creation of technology. I do think this is misleading. What is responsible is that we are goal-oriented while evolution is not. But the advance of scientific knowledge is largely an evolutionary process. I don’t see that intelligence is currently tangible enough to measure that the return of increased intelligence is proportional to the resources it would take to amplify it. The argument from the gap between chimpanzees and humans is interesting but can not be used to extrapolate onwards from human general intelligence. It is pure speculation that humans are not Turing complete and that there are levels above our own. That chimpanzees exist, and humans exist, is not a proof for the existence of anything that bears, in any relevant respect, the same relationship to a human that a human bears to a chimpanzee.
It is in principle possible to create artificial intelligence that is as capable as human intelligence. But this says nothing about how quickly we will be able to come up with it. I believe that intelligence is fundamentally dependent on the complexity of the goals against which it is measured. Goals give rise to agency and define an agent’s drives. As long as we won’t be able to precisely hard-code a complexity of values similar to that of humans we won’t achieve levels of general intelligence similar to humans.
It is true that humans have created a lot of tools that help them to achieve their goals. But it is not clear that incorporating those tools into some sort of self-perception, some sort of guiding agency, is superior to humans using a combination of tools and expert systems. In other words, it is not clear that there does exist a class of problems that is solvable by Turing machines in general, but not by a combination of humans and expert systems. And if that was the case then I think that, just like chimpanzees would be unable to invent science, we won’t be able to come up with a meta-heuristic that would allow us to discover algorithms that can solve a class of problems that we can’t (other than by using guided evolution).
Besides, recursive self-improvement does not demand sentience, consciousness or agency. Even if humans are not able to “recursively improve” their own algorithms we can still “recursively improve” our tools. And the supremacy of recursively improving agent’s over humans and their tools is a reasonable conjecture but not a fact. It largely relies on the idea that the integration of tools into a coherent framework of agencies has huge benefits.
I also object to assigning numerical probability estimates to informal arguments and predictions. When faced with data from empirical experiments, or goats behind doors in a gameshow, it is reasonable. But using formalized methods to evaluate informal evidence can be very misleading. For real-world, computationally limited agents it is a recipe to fail spectacularly. Using formalized methods to to evaluate vague ideas like risks from AI can lead you to dramatically over or underestimate evidence by forcing you to use your intuition to assign numbers to your intuitive judgement of informal arguments.
And as a disclaimer: Don’t jump to the conclusion that I generally rule out the possibility that very soon someone will stumble upon a simple algorithm that can be run on a digital computer, that can be improved to self-improve, become superhuman and take over the universe. All am saying is that the possibility isn’t as inevitable as some seem to believe. If forced, I would probably assign a 1% probability to it but still feel uncomfortable about that (which isn’t to equate with risks from AI in general, I don’t think FOOM is required for AI’s to pose a risk).
I think that Eliezer crossed the border of what can sensibly be said about this topic at the present time when he says that AI will likely invent molecular nanotechnology in a matter of hours or days. Jürgen Schmidhuber is the only person I could find who might agree with that. Even Shane Legg is more skeptical. And since I do not yet have the education to evaluate state of the art AI research myself I will side with the experts and say that Eliezer is likely wrong. Of course, I have no authority but I have to make a decision. I don’t feel it would be reasonable to believe Eliezer here without restrictions.
Just because the possibility of superhuman AI seems to be disjunctive on some level doesn’t mean that there are no untested assumptions underlying the claims that such an outcome is possible. Reduce the vagueness and you will discover a set assumptions that need to be true in conjunction.
So, I’m having a lot of difficulty mapping your response to the question I asked. But if I’ve understood your response, you are arguing that technology analogous to the technology-developing functions of human intelligence might not be in principle possible, or that if developed might not be capable of significantly greater technology-developing power than human intelligence is.
In other words, that assumptions 5 and/or 6 might be false.
I agree that it’s possible. Similar things are true of the other examples you give: it’s possible that technological echolocation, or technological walking, or technological photosynthesis, either aren’t possible in principle, or can’t be significantly more powerful than their naturally evolved analogs. (Do you actually believe that to be true of those examples, incidentally?)
This seems to me highly implausible, which is why my confidence for A5 and A6 are very high. (I have similarly high confidence in our ability to develop machines more efficient than human legs at locomotion, machines more efficient at converting sunlight to useful work than plants, and more efficient at providing sonar-based information about their surroundings than bats.)
So, OK. We’ve identified a couple of specific, relevant assertions for which you think that my confidence is too high. Awesome! That’s progress.
So, what level of confidence do you think is justified for those assertions? I realize that you reject assigning numbers to reported confidence, so OK… do you have a preferred way of comparing levels of confidence? Or do you reject the whole enterprise of such comparisons?
Incidentally: you say a lot of other stuff here which seems entirely beside my point… I think because you’re running out ahead to arguments you think I might make some day. I will return to that stuff if I ever actually make an argument to which it’s relevant.