Lara Foster, to get what people are worried about, extrapolate the danger of recursive self-improving intelligence to self-reproducing nanotechnology. We want what it can provide, we spread nanomachines, and from there you can calculate how many doublings would be necessary to convert all the molecules on the surface of the planet to nano-assemblers. Ten doublings is 1024*, so we probably would not realize how over-powered we were until far too late.
As you say, this is not the most likely extinction event. Losing Eurasia and Africa to a sign error would be bad thing, but not a full extinction event. The downside of being a nanomachine is that trans-Atlantic swimming is hard with 2nm-long legs.
But if a nano-assembler can reproduce itself in 6 minutes, you have one thousand in an hour, one million the next hour, one billion the next hour… not a lot of time for regulation.
The only one who can react to a problem that big in that timespan is that recursively self-improving AI we have been keeping in the box over there. Guess it’s time to let it out. (Say, who is responsible for that first nano-assembler anyway?)
Lara Foster, to get what people are worried about, extrapolate the danger of recursive self-improving intelligence to self-reproducing nanotechnology. We want what it can provide, we spread nanomachines, and from there you can calculate how many doublings would be necessary to convert all the molecules on the surface of the planet to nano-assemblers. Ten doublings is 1024*, so we probably would not realize how over-powered we were until far too late.
As you say, this is not the most likely extinction event. Losing Eurasia and Africa to a sign error would be bad thing, but not a full extinction event. The downside of being a nanomachine is that trans-Atlantic swimming is hard with 2nm-long legs.
But if a nano-assembler can reproduce itself in 6 minutes, you have one thousand in an hour, one million the next hour, one billion the next hour… not a lot of time for regulation.
The only one who can react to a problem that big in that timespan is that recursively self-improving AI we have been keeping in the box over there. Guess it’s time to let it out. (Say, who is responsible for that first nano-assembler anyway?)