Saying that grey goo will spread, and then never change or create new forms, is as mistaken as saying that single-celled organisms should never have given rise to multi-cellular organisms because competition between individuals is so stringent.
One of the things that Eliezer doesn’t grasp is that optimization is not something evolution has generally produced because optimization is often maladaptive. What would appear to be an ideal strategy in the short term fails in the long term because overall environmental conditions have a tendency to change. Biologists have long noted that specialization offers great benefits to organisms, but often results in their eventual extinction.
Saying that the tendency of hypothetical nanomachines, or actual corporations, will be to find an optimum configuration and remain that way, is like saying that Always Defectors will dominate games of the Repeated Prisoners’ Dilemma. It’s what the most superficial, summing-over analysis of the problem will indicate, but it misses more subtle points that change everything. There’s a reason Always Defectors don’t dominate populations in the game or in real life.
Saying that grey goo will spread, and then never change or create new forms, is as mistaken as saying that single-celled organisms should never have given rise to multi-cellular organisms because competition between individuals is so stringent.
One of the things that Eliezer doesn’t grasp is that optimization is not something evolution has generally produced because optimization is often maladaptive. What would appear to be an ideal strategy in the short term fails in the long term because overall environmental conditions have a tendency to change. Biologists have long noted that specialization offers great benefits to organisms, but often results in their eventual extinction.
Saying that the tendency of hypothetical nanomachines, or actual corporations, will be to find an optimum configuration and remain that way, is like saying that Always Defectors will dominate games of the Repeated Prisoners’ Dilemma. It’s what the most superficial, summing-over analysis of the problem will indicate, but it misses more subtle points that change everything. There’s a reason Always Defectors don’t dominate populations in the game or in real life.