Reason 1: Mitigating AI risk could mitigate all other existential risks, but not vice-versa. There is an asymmetry between AI risk and other existential risks. If we mitigate the risks from (say) synthetic biology and nanotechnology (without building Friendly AI), this only means we have bought a few years or decades for ourselves before we must face yet another existential risk from powerful new technologies. But if we manage AI risk well enough (i.e. if we build a Friendly AI or “FAI”), we may be able to “permanently” (for several billion years) secure a desirable future.
This equates “managing AI risk” and “building FAI” without actually making the case that these are equivalent. Many people believe that dangerous research can be banned by governments, for instance; it would be useful to actually make the case (or link to another place where it has been made) that managing AI risk is intractable without FAI.
This is one of the 10,000 things I didn’t have the space to discuss in the original post, but I’m happy to briefly address it here!
It’s much harder to successfully ban AI research than to successfully ban, say, nuclear weapons. Nuclear weapons require rare and expensive fissile material that requires rare heavy equipment to manufacture. Such things can be tracked to some degree. In contrast, AI research requires… um… a few computers.
Moreover, it’s really hard to tell whether the code somebody is running on a computer is potentially dangerous AI stuff or something else. Even if you magically had a monitor installed on every computer to look for dangerous AI stuff, it would have to know what “dangerous AI stuff” looks like, which is hard to do before the dangerous AI stuff is built in the first place.
The monetary, military, and political incentives to build AGI are huge, and would be extremely difficult to counteract through a worldwide ban. You couldn’t enforce the ban, anyway, for the reasons given above. That’s why Ben Goertzel advocates “Nanny AI,” though Nanny AI may be FAI-complete, as mentioned here.
It does help...I had the same reaction as fubarob.
However, your argument assumes that our general IT capabilities have already matured to the point where AGI is possible. I agree that restricting AGI research then is likely a lost cause. Much less clear to me is whether it is equally futile to try restricting IT and computing research or even general technological progress before such a point. Could we expect to bring about global technological stasis? One may be tempted to say that such an effort is doomed to a fate like global warming accords except ten times deader. I disagree entirely. Both Europe and the United States, in fact, have in recent years been implementing a quite effective policy of zero economic growth! It is true that progress in computing has continued despite the general slowdown but this seems hardly written in stone for the future. In fact we need only consider this paper on existential risks and the “Crunches” section for several examples of how stasis might be brought about.
Can anyone recommend detailed discussions of broad relinquishment, from any point of view? The closest writings I know are Bill McKibben’s book Enough and Bill Joy’s essay, but anything else would be great.
I’m pretty sure we have the computing power to run at least one AGI, but I can’t prove that. Still, restricting general computing progress should delay the arrival of AGI because the more hardware you have, the “dumber” you can be with solving the AGI software problem. (You can run less efficient algorithms.)
Global technological stasis seems just as hard or maybe harder than restricting AGI research. The incentives for AGI are huge, but there might be some points at which you have to spend a lot of money before you get much additional economic/political/military advantage. But when it comes to general computing progress, then it looks to be always the case that a bit more investment can always yield better returns, e.g. by squeezing more processor cores into a box a little more tightly.
Other difficulties of global technological stasis are discussed in the final chapter of GCR, called “The Totalitarian Threat.” Basically, you’d need some kind of world government, because any country that decides to slow down its computing progress rapidly falls behind other nations. But political progress is so slow that it seems unlikely we’ll get a world government in the next century unless somebody gets a decisive technological advantage via AGI, in which case we’re talking about the AGI problem anyway. (The world government scenario looks like the most plausible of Bostrom’s “crunches”, which is why I chose to discuss it.)
Global technological stasis seems just as hard or maybe harder than restricting AGI research.
I don’t want to link directly to what I believe is a solution to this problem, as it was so widely misinterpreted, but I thought there was already a solution to this problem that hinged on how expensive it is to harden fabrication plants. In that case, you wouldn’t even need one world government; a sufficiently large collaboration (e.g., NATO) would do the trick.
A world government may not be the only form of stable social equilibrium that could permanently thwart progress. Many regions of the world today have great difficulty building institutions that can support high growth. And historically, there are many places where progress stood still or retreated for significant periods of time. Economic and technological progress may not be as inevitable as is appears to us.
It seems to me like saying world government is necessary underestimates the potential impact of growth-undermining ideas. If all large governments for example buy into the idea that cutting government infrastructure spending in a recession boosts employment, then we can assume that global growth will slow as a result of the acceptance of this false assertion. To me the key seems less to have some world nabob pronouncing edicts on technology and more shifting the global economy so as to make edicts merely the gift wrap.
I will definitely take a look at that chapter of GCR. Thanks also for the other links. The little paper by Berglas I found interesting. Mr. K needs no comment, naturally. Might be good reading for similar reasons that Mien Kampf is. With De Garis I have long felt like the kook-factor was too high to warrant messing with. Anyone read him much? Maybe it would be good just for some ideas.
It certainly could be true that economic growth and technological progress can slow down. In fact, I suspect the former if not the latter will slow down, and perhaps the latter, too. That’s very different from stopping technological progress that will lead to AGI, though.
Not only can economic growth and technological progress slow down. They can stop and reverse. Just because we’re now further out in front than humanity has ever been before in history does not mean that we can’t go backwards. Economic growth is probably more likely to reverse than technological progress. That’s what a depression is, after all.
But a sufficiently bad global catastrophe, perhaps one that destroyed the electrical grid and other key infrastructure, could reverse a lot of technological progress too and perhaps knock us way back without necessarily causing complete extinction.
I think technological stasis could really use more discussion. For example I was able to find this paper by James Hughes discussing relinquishment. He however treats the issue as one of regulation and verification, similar to nuclear weapons, noting that:
We do not yet have effective global institutions capable of preventing determined research of any kind, even apocalyptic. I believe we will be build strong global institutions in the next couple of decades. Global institutions may regulate for safety, prevent weaponization, support technology transfer, and so on. But there will be no support for global governance that attempts to deny developing countries the right to emerging technologies.
Regulation and verification may indeed be a kind of Gordian knot. The more specific the technologies you want to stop, the harder it becomes to do that and still advance generally. Berglas recognizes that problem in his paper and so proposes restricting computing as a whole. Even this, however, may be too specific to cut the knot. The feasibility of stopping economic growth entirely so we never reach the point where regulation is necessary seems to me an unexplored question. If we look at global GDP growth over the past 50 years it’s been uniformly positive except for the most recent recession. It’s also been quite variable over short periods. Clearly stopping it for longer than a few years would require some new phenomenon driving a qualitative break with the past. That does not mean however that a stop is impossible.
There does exist a small minority camp within the economics profession advocating no-growth policies for environmental or similar reasons. I wonder if anyone has created a roadmap for bringing such policies about on a global level.
This equates “managing AI risk” and “building FAI” without actually making the case that these are equivalent. Many people believe that dangerous research can be banned by governments, for instance; it would be useful to actually make the case (or link to another place where it has been made) that managing AI risk is intractable without FAI.
This is one of the 10,000 things I didn’t have the space to discuss in the original post, but I’m happy to briefly address it here!
It’s much harder to successfully ban AI research than to successfully ban, say, nuclear weapons. Nuclear weapons require rare and expensive fissile material that requires rare heavy equipment to manufacture. Such things can be tracked to some degree. In contrast, AI research requires… um… a few computers.
Moreover, it’s really hard to tell whether the code somebody is running on a computer is potentially dangerous AI stuff or something else. Even if you magically had a monitor installed on every computer to look for dangerous AI stuff, it would have to know what “dangerous AI stuff” looks like, which is hard to do before the dangerous AI stuff is built in the first place.
The monetary, military, and political incentives to build AGI are huge, and would be extremely difficult to counteract through a worldwide ban. You couldn’t enforce the ban, anyway, for the reasons given above. That’s why Ben Goertzel advocates “Nanny AI,” though Nanny AI may be FAI-complete, as mentioned here.
I hope that helps?
Yes.
It does help...I had the same reaction as fubarob.
However, your argument assumes that our general IT capabilities have already matured to the point where AGI is possible. I agree that restricting AGI research then is likely a lost cause. Much less clear to me is whether it is equally futile to try restricting IT and computing research or even general technological progress before such a point. Could we expect to bring about global technological stasis? One may be tempted to say that such an effort is doomed to a fate like global warming accords except ten times deader. I disagree entirely. Both Europe and the United States, in fact, have in recent years been implementing a quite effective policy of zero economic growth! It is true that progress in computing has continued despite the general slowdown but this seems hardly written in stone for the future. In fact we need only consider this paper on existential risks and the “Crunches” section for several examples of how stasis might be brought about.
Can anyone recommend detailed discussions of broad relinquishment, from any point of view? The closest writings I know are Bill McKibben’s book Enough and Bill Joy’s essay, but anything else would be great.
I’m pretty sure we have the computing power to run at least one AGI, but I can’t prove that. Still, restricting general computing progress should delay the arrival of AGI because the more hardware you have, the “dumber” you can be with solving the AGI software problem. (You can run less efficient algorithms.)
Global technological stasis seems just as hard or maybe harder than restricting AGI research. The incentives for AGI are huge, but there might be some points at which you have to spend a lot of money before you get much additional economic/political/military advantage. But when it comes to general computing progress, then it looks to be always the case that a bit more investment can always yield better returns, e.g. by squeezing more processor cores into a box a little more tightly.
Other difficulties of global technological stasis are discussed in the final chapter of GCR, called “The Totalitarian Threat.” Basically, you’d need some kind of world government, because any country that decides to slow down its computing progress rapidly falls behind other nations. But political progress is so slow that it seems unlikely we’ll get a world government in the next century unless somebody gets a decisive technological advantage via AGI, in which case we’re talking about the AGI problem anyway. (The world government scenario looks like the most plausible of Bostrom’s “crunches”, which is why I chose to discuss it.)
Relinquishment is also discussed (in very different ways) by Berglas (2009), Kaczynski (1995), and De Garis (2005).
I don’t want to link directly to what I believe is a solution to this problem, as it was so widely misinterpreted, but I thought there was already a solution to this problem that hinged on how expensive it is to harden fabrication plants. In that case, you wouldn’t even need one world government; a sufficiently large collaboration (e.g., NATO) would do the trick.
Quoting the B-man:
A world government may not be the only form of stable social equilibrium that could permanently thwart progress. Many regions of the world today have great difficulty building institutions that can support high growth. And historically, there are many places where progress stood still or retreated for significant periods of time. Economic and technological progress may not be as inevitable as is appears to us.
It seems to me like saying world government is necessary underestimates the potential impact of growth-undermining ideas. If all large governments for example buy into the idea that cutting government infrastructure spending in a recession boosts employment, then we can assume that global growth will slow as a result of the acceptance of this false assertion. To me the key seems less to have some world nabob pronouncing edicts on technology and more shifting the global economy so as to make edicts merely the gift wrap.
I will definitely take a look at that chapter of GCR. Thanks also for the other links. The little paper by Berglas I found interesting. Mr. K needs no comment, naturally. Might be good reading for similar reasons that Mien Kampf is. With De Garis I have long felt like the kook-factor was too high to warrant messing with. Anyone read him much? Maybe it would be good just for some ideas.
It certainly could be true that economic growth and technological progress can slow down. In fact, I suspect the former if not the latter will slow down, and perhaps the latter, too. That’s very different from stopping technological progress that will lead to AGI, though.
Not only can economic growth and technological progress slow down. They can stop and reverse. Just because we’re now further out in front than humanity has ever been before in history does not mean that we can’t go backwards. Economic growth is probably more likely to reverse than technological progress. That’s what a depression is, after all.
But a sufficiently bad global catastrophe, perhaps one that destroyed the electrical grid and other key infrastructure, could reverse a lot of technological progress too and perhaps knock us way back without necessarily causing complete extinction.
I think technological stasis could really use more discussion. For example I was able to find this paper by James Hughes discussing relinquishment. He however treats the issue as one of regulation and verification, similar to nuclear weapons, noting that:
Regulation and verification may indeed be a kind of Gordian knot. The more specific the technologies you want to stop, the harder it becomes to do that and still advance generally. Berglas recognizes that problem in his paper and so proposes restricting computing as a whole. Even this, however, may be too specific to cut the knot. The feasibility of stopping economic growth entirely so we never reach the point where regulation is necessary seems to me an unexplored question. If we look at global GDP growth over the past 50 years it’s been uniformly positive except for the most recent recession. It’s also been quite variable over short periods. Clearly stopping it for longer than a few years would require some new phenomenon driving a qualitative break with the past. That does not mean however that a stop is impossible.
There does exist a small minority camp within the economics profession advocating no-growth policies for environmental or similar reasons. I wonder if anyone has created a roadmap for bringing such policies about on a global level.
Out of curiosity, have you read my little essay “Slowing Moore’s Law”? It seems relevant.