A: We will build computers of at least human intelligence at some time in the future, let’s say within 100 years.
B: Those computers will be able to rapidly and repeatedly increase their own intelligence, quickly resulting in computers that are far more intelligent than human beings.
C: This will cause an enormous transformation of the world, so much so that it will become utterly unrecognizable …
Then he goes on to define the probability of the Singularity within the next 100 years as the probability p(C|B)p(B|A)p(A), and gives what he thinks are reasonable ranges for the values p(A), p(B) and p(C)
Assuming we avoid a collapse of civilization, I would estimate p(A) = 0.7. B requires some clarification. I will read “far more” (intelligent than humans) as “by a factor of 1000″. Then, if “quickly” is read as “within 5 years”, I would estimate p(B|A) = 0.2, and if “quickly” is read as within 30 years, I would up that estimate to p(B|A) = 0.8. That is, I expect a rather slow takeoff.
But my main disagreement with most singularitarians is in my estimate of P(C|B). I estimate it at less than 0.1 - even allowing two human generations (50 years) for the transformation. I just don’t think that the impact of superhuman intelligence will be all that dramatic.
Let us look at some other superhuman (by a factor of 1000 or more) technologies that we already have. Each of them has transformed things, to be sure, but none of them has rapidly made things “utterly unrecognizable”.
Machines 1000x as powerful as humans (lifting, for example).
Transportation 1000x as fast as humans.
Computation speed 1000x as fast as humans (FLOPS)
Imaging 1000x as fine or as powerful as the human eye (microscopes and telescopes).
Fabrication precisions 1000x as close as the human hand can deliver.
Organizations coordinating the efforts of 1000x as many people as a human hunting band or village.
Works of education and entertainment reaching 1000x as many people as could be reached by a preliterate sage or bard.
Transformative technologies—yes. Utterly unrecognizable—no. And, collectively, the existing 1000x improvements listed above are likely to prove at least as transformative as the prospective 1000x in intelligence.
ETA: In effect, I am saying that most of the things that can be done by a 1000x-human AI could also be done by the collective effort of a thousand or so 1x-humans. And that the few things that can not be done by that kind of collective effort are not going to be all that transformative.
In effect, I am saying that most of the things that can be done by a 1000x-human AI could also be done by the collective effort of a thousand or so 1x-humans.
This may not be true. Firstly, 1000 mentally disabled people, or 1000 children, or 1000 chimpanzees, cannot get as much done in areas that depend on intelligence as one smart, educated human. Second, humans aren’t just a certain level of intelligence, we’re also full of bugs: biases, akrasia, coordination problems, and the like that an AI wouldn’t have to have. An individual of human average intelligence or slightly above, with substantially improved rationality and anti-akrasia, can be a very effective person. An AI would have even less of those problems, and would be smarter, and might run on a faster timescale.
Nitpick: your computation speed example is WAY off. Random googling suggests that as of ’09, the world’s fastest computer was behind a single human brain by a factor of ~500.
Your other examples are more or less physical technologies, and it is not at all clear that they are a valid reference class for computational technologies.
There are many ways of measuring computation speed. The one I suggested—FLOPS or floating point operations per second—is admittedly biased toward machines. I’m pretty confident that, using this metric—the one I originally specified—that I was WAY off in the opposite direction—fast machines beat humans by a factor of a trillion or more. How many 7 digit numbers can you add in a second without error?
FLOPS and clock speed are not the same thing. The clock speed of the human brain (possible neuronal firings per second is like 100-1000 Hz). However, the brain is also massively parallel, and FLOPS estimates for the brain vary widely. I’ve seen estimates ranging from 100 teraFLOPS to 100 exaFLOPS. Kurzweil estimates 20 PetaFLOPS, but admits that it could be much higher.
Keep in mind that all those developments have been produced by human level intelligence. Human level intelligence has made the world pretty unrecognizable compared to pre-human level intelligence.
I just don’t think that the impact of superhuman intelligence will be all that dramatic.
The impact of human-level intelligence has been fairly dramatic—looking at the current mass extinction. Presumably you have already heard the spiel about how important intelligence is.
And, collectively, the existing 1000x improvements listed above are likely to prove at least as transformative as the prospective 1000x in intelligence.
The biggest transformation seems likely when machines surpass humans in most areas of the job marketplace. To do that, they need good intelligence—and good bodies. So far, they don’t have either really—but it seems plausible that, fairly soon, they will have both of these things.
After that, humans will survive largely as parasites on a machine-based economy.
In effect, I am saying that most of the things that can be done by a 1000x-human AI could also be done by the collective effort of a thousand or so 1x-humans. And that the few things that can not be done by that kind of collective effort are not going to be all that transformative.
If machines are 10% better than humans, they will get the jobs. Maybe humans could have done the same thing eventually (and maybe not) - but they are unemployed—so they won’t get the chance to do so. The rise of the machines represents a fairly major transformation—even now, when the machines have only a tiny percentage of the planet’s biomass. If they start to outweigh us, that seems like a pretty big deal to me.
The impact of human-level intelligence has been fairly dramatic—looking at the current mass extinction.
You mean the one that started roughly 15,000 years ago? Yes, a truly dramatic change!
The biggest transformation seems likely when machines surpass humans in most areas of the job marketplace.
True. And at that point, humans will begin to derive more than half of their income from their ownership of capital and land. And those humans without capital or land may not be able survive, let alone to reproduce. Mankind has been in this position before, though.
The impact of human-level intelligence has been fairly dramatic—looking at the current mass extinction.
You mean the one that started roughly 15,000 years ago? Yes, a truly dramatic change!
You have different tastes in drama from me. For me, a mass extinction is a big deal. Especially so, if the species to which I belong looks as though it may be one of those that goes up against the wall in it.
Mankind has been in this position before, though.
Well, not exactly this position: we haven’t come across critters that are much stronger, faster and smarter than us before.
Nielson characterizes the Singularity as:
Assuming we avoid a collapse of civilization, I would estimate p(A) = 0.7. B requires some clarification. I will read “far more” (intelligent than humans) as “by a factor of 1000″. Then, if “quickly” is read as “within 5 years”, I would estimate p(B|A) = 0.2, and if “quickly” is read as within 30 years, I would up that estimate to p(B|A) = 0.8. That is, I expect a rather slow takeoff.
But my main disagreement with most singularitarians is in my estimate of P(C|B). I estimate it at less than 0.1 - even allowing two human generations (50 years) for the transformation. I just don’t think that the impact of superhuman intelligence will be all that dramatic.
Let us look at some other superhuman (by a factor of 1000 or more) technologies that we already have. Each of them has transformed things, to be sure, but none of them has rapidly made things “utterly unrecognizable”.
Machines 1000x as powerful as humans (lifting, for example).
Transportation 1000x as fast as humans.
Computation speed 1000x as fast as humans (FLOPS)
Imaging 1000x as fine or as powerful as the human eye (microscopes and telescopes).
Fabrication precisions 1000x as close as the human hand can deliver.
Organizations coordinating the efforts of 1000x as many people as a human hunting band or village.
Works of education and entertainment reaching 1000x as many people as could be reached by a preliterate sage or bard.
Transformative technologies—yes. Utterly unrecognizable—no. And, collectively, the existing 1000x improvements listed above are likely to prove at least as transformative as the prospective 1000x in intelligence.
ETA: In effect, I am saying that most of the things that can be done by a 1000x-human AI could also be done by the collective effort of a thousand or so 1x-humans. And that the few things that can not be done by that kind of collective effort are not going to be all that transformative.
This may not be true. Firstly, 1000 mentally disabled people, or 1000 children, or 1000 chimpanzees, cannot get as much done in areas that depend on intelligence as one smart, educated human. Second, humans aren’t just a certain level of intelligence, we’re also full of bugs: biases, akrasia, coordination problems, and the like that an AI wouldn’t have to have. An individual of human average intelligence or slightly above, with substantially improved rationality and anti-akrasia, can be a very effective person. An AI would have even less of those problems, and would be smarter, and might run on a faster timescale.
Nitpick: your computation speed example is WAY off. Random googling suggests that as of ’09, the world’s fastest computer was behind a single human brain by a factor of ~500.
Your other examples are more or less physical technologies, and it is not at all clear that they are a valid reference class for computational technologies.
There are many ways of measuring computation speed. The one I suggested—FLOPS or floating point operations per second—is admittedly biased toward machines. I’m pretty confident that, using this metric—the one I originally specified—that I was WAY off in the opposite direction—fast machines beat humans by a factor of a trillion or more. How many 7 digit numbers can you add in a second without error?
FLOPS and clock speed are not the same thing. The clock speed of the human brain (possible neuronal firings per second is like 100-1000 Hz). However, the brain is also massively parallel, and FLOPS estimates for the brain vary widely. I’ve seen estimates ranging from 100 teraFLOPS to 100 exaFLOPS. Kurzweil estimates 20 PetaFLOPS, but admits that it could be much higher.
Keep in mind that all those developments have been produced by human level intelligence. Human level intelligence has made the world pretty unrecognizable compared to pre-human level intelligence.
The impact of human-level intelligence has been fairly dramatic—looking at the current mass extinction. Presumably you have already heard the spiel about how important intelligence is.
The biggest transformation seems likely when machines surpass humans in most areas of the job marketplace. To do that, they need good intelligence—and good bodies. So far, they don’t have either really—but it seems plausible that, fairly soon, they will have both of these things.
After that, humans will survive largely as parasites on a machine-based economy.
If machines are 10% better than humans, they will get the jobs. Maybe humans could have done the same thing eventually (and maybe not) - but they are unemployed—so they won’t get the chance to do so. The rise of the machines represents a fairly major transformation—even now, when the machines have only a tiny percentage of the planet’s biomass. If they start to outweigh us, that seems like a pretty big deal to me.
You mean the one that started roughly 15,000 years ago? Yes, a truly dramatic change!
True. And at that point, humans will begin to derive more than half of their income from their ownership of capital and land. And those humans without capital or land may not be able survive, let alone to reproduce. Mankind has been in this position before, though.
You have different tastes in drama from me. For me, a mass extinction is a big deal. Especially so, if the species to which I belong looks as though it may be one of those that goes up against the wall in it.
Well, not exactly this position: we haven’t come across critters that are much stronger, faster and smarter than us before.