Wei Dai asks for instances of “talk or writing” . . . “that’s in favor of a pause-like position before 2022″.
Bill Joy argued for a halt in 2000 in an article in Wired magazine. Specifically, he called for a halt on research and development of genetic engineering, nanotechnology and “robotics”, by which Joy meant AI.
The article generated a lot of discussion, including a discussion in front of an audience of about 500 among Joy, Douglas Hofstadter, Hans Moravec, Ray Kurzweil and others. The audience discussion was hosted by Stanford’s Symbolic Systems Program, which had nothing to do with genetics or nanotechnology, but directly to do with AI. (Hans Moravec was research professor at the Robotics Institute at Carnegie Mellon University and director of its Mobile Robot Laboratory.)
In Engines of Creation, Eric Drexler proposed that we build an active nanotechnological shield—a form of immune system for the biosphere—to defend against dangerous replicators of all kinds that might escape from laboratories or otherwise be maliciously created. But the shield he proposed would itself be extremely dangerous—nothing could prevent it from developing autoimmune problems and attacking the biosphere itself.
Similar difficulties apply to the construction of shields against robotics and genetic engineering. These technologies are too powerful to be shielded against in the time frame of interest; even if it were possible to implement defensive shields, the side effects of their development would be at least as dangerous as the technologies we are trying to protect against.
These possibilities are all thus either undesirable or unachievable or both. The only realistic alternative I see is relinquishment: to limit development of the technologies that are too dangerous, by limiting our pursuit of certain kinds of knowledge.
Yes, I know, knowledge is good, as is the search for new truths. We have been seeking knowledge since ancient times. Aristotle opened his Metaphysics with the simple statement: “All men by nature desire to know.” We have, as a bedrock value in our society, long agreed on the value of open access to information, and recognize the problems that arise with attempts to restrict access to and development of knowledge. In recent times, we have come to revere scientific knowledge.
But despite the strong historical precedents, if open access to and unlimited development of knowledge henceforth puts us all in clear danger of extinction, then common sense demands that we reexamine even these basic, long-held beliefs.
Joy B (2000) Why the future doesn’t need us. Wired 238–62 April.
The article exists on the internet only behind a paywall AFAICT. Because its publication was such a major milestone the conversation on AI extinction danger, I’ve made available a non-paywall copy of it, but it probably won’t remain available for long.
Wei Dai asks for instances of “talk or writing” . . . “that’s in favor of a pause-like position before 2022″.
Bill Joy argued for a halt in 2000 in an article in Wired magazine. Specifically, he called for a halt on research and development of genetic engineering, nanotechnology and “robotics”, by which Joy meant AI.
The article generated a lot of discussion, including a discussion in front of an audience of about 500 among Joy, Douglas Hofstadter, Hans Moravec, Ray Kurzweil and others. The audience discussion was hosted by Stanford’s Symbolic Systems Program, which had nothing to do with genetics or nanotechnology, but directly to do with AI. (Hans Moravec was research professor at the Robotics Institute at Carnegie Mellon University and director of its Mobile Robot Laboratory.)
Joy B (2000) Why the future doesn’t need us. Wired 238–62 April.
The article exists on the internet only behind a paywall AFAICT. Because its publication was such a major milestone the conversation on AI extinction danger, I’ve made available a non-paywall copy of it, but it probably won’t remain available for long.