“If we stop and don’t develop AGI before our geopolitical enemies because we’re afraid of a tiny risk of an extinction, they will develop it regardless, then one of two things happen: either global extinction, or our extinction in our enemies’ hands. Which is why we must develop it first. If it goes well, we extinguish them before they have a chance to do it to us. If it goes bad, it’d have gone bad anyway in their or our hands, so that case doesn’t matter.”
This has become a common description of why AI companies and governments are moving quickly. In general, I agree with the description, but I specifically struggle with this portion if it:
“Which is why we must develop it first. If it goes well, we extinguish them before they have a chance to do it to us.”
I’m assuming that—and please correct me if I’m misinterpreting here—“extinguish” here means something along the lines of, “remove the ability to compete effectively for resources (e.g. customers or other planets)” not “literally annihilate”.
If I got that totally wrong, no need to read on.
If that’s roughly correct, well, so what? How does being “first” actually solve the misaligned AGI problem? “Global extinction” as you put it.
Being first doesn’t come with the benefit of forcing all subsequently created AGI to be aligned / safe. The government or corporation in second (third, fourth, etc.) place surely can and probably will continue to attempt to build an AGI. They’re probably even more likely to create one in a more reckless manner by trying to catch up as quickly as possible.
I’m assuming that—and please correct me if I’m misinterpreting here—“extinguish” here means something along the lines of, “remove the ability to compete effectively for resources (e.g. customers or other planets)” not “literally annihilate”.
I wish that were the case, but my reference is imagining a paranoid M.A.D. mentality coupled with a Total War scenario unbounded by moral constraints, that is, all sides thinking all the other sides are X-risks to them.
In practice things tend not to get that bad most of the time, but sometimes they do, and much of military preparation concern mitigation of these perceived X-risks, the idea being that if “our side” becomes so powerful it can in fact annihilate the others, and in consequence the others submit without resisting, then “our side” may be magnanimous towards them conditional on their continued subservience and submission, but if they resist to the point of becoming an X-risk towards us, then removing them from the equation entirely is the safest defense from the X-risk they pose us.
A global consensus on stopping GAI development due to its X-risk for all life passes through a global consensus, by all sides, that none of the other sides is an X-risk to any of side. Once everyone agrees on this, then they all together agreeing to deal with a global X-risk becomes feasible. Before that, only if they all see that global X-risk as more urgent and immediate than the many local-to-them X-risks.
The US had nuclear weapons before any other country. Other countries have these weapons now. The p-boom was quite high at some points but nobody was annihilated with this technology.
Admittedly, nuclear weapons are not a perfect analog for AI due to many reasons, but I think it is a reasonable analog.
With this in mind, I wanted to ask out of curiosity, what % risk do you think there needs to be for annihilation to occur?
Admittedly, nuclear weapons are not a perfect analog for AI due to many reasons, but I think it is a reasonable analog.
We’ve had extreme luck when it comes to nuclear weapons. We not only had several close calls that were deescalated by particularly noble individuals doing the right thing, but also, back when the URSS had barely developed theirs and the US alone had a whole stockpile of warheads, we had the good luck of its leadership also being somewhat moral and refusing to turn nukes into a regular weapon, which was followed by MAD forcing everyone to kind of stay so even when the other side asked nicely whether they could bomb a third party. Weren’t for that long sequence of good luck after good luck, and we’d now be living in an annihilated world, or at the very least a post-apocalyptic one.
With this in mind, I wanted to ask out of curiosity, what % risk do you think there needs to be for annihilation to occur?
I have no idea, really. All I can infer is that it’s unlikely any major power will stop trying to achieve GAI unless:
a) Either a massively severe accident due to misaligned not-quite-GAI-yet happens that by its sheer, absolute horror puts the Fear-Of-God in our civilian and military leaders for a few generations;
b) Or a long sequence of reasonably severe accidents happens, each new one worse than the last, with AI companies repeatedly and consistently failing at fixing the underlying cause, this in turn making military leaders deeply wary of deploying advanced AI systems, and civilian leaders enacting restrictions on what AI is allowed to touch.
Absent either of those, I doubt the pursuit of GAI will stop no matter what X-risk analysts say. Or at least, I myself cannot imagine any kind of argument that’d convince, say, the CPC to stop their research when those on the other side spearheading theirs are massively powerful nutjobs? And then, what argument could be provided that’d stop someone who believes in this? So, neither will stop, which means GAI will happen. And then we’ll need to count on luck again, this time with:
i) Either GAI going FOOM as Yudkowsky believes, but for some reason continuing to like humans enough not to turn us into computronium;
ii) Or Hanson being right and FOOM not happening, followed by:
ii.1) Either things being slow enough to “merely” lead to a or b, above;
ii.2) Or things being so immensely slow we can actually fix them.
I have no opinion on whether FOOM is or isn’t likely. I’ve read the entire discussion and all I know is both sets of arguments sound reasonable to me.
This has become a common description of why AI companies and governments are moving quickly. In general, I agree with the description, but I specifically struggle with this portion if it:
I’m assuming that—and please correct me if I’m misinterpreting here—“extinguish” here means something along the lines of, “remove the ability to compete effectively for resources (e.g. customers or other planets)” not “literally annihilate”.
If I got that totally wrong, no need to read on.
If that’s roughly correct, well, so what? How does being “first” actually solve the misaligned AGI problem? “Global extinction” as you put it.
Being first doesn’t come with the benefit of forcing all subsequently created AGI to be aligned / safe. The government or corporation in second (third, fourth, etc.) place surely can and probably will continue to attempt to build an AGI. They’re probably even more likely to create one in a more reckless manner by trying to catch up as quickly as possible.
I wish that were the case, but my reference is imagining a paranoid M.A.D. mentality coupled with a Total War scenario unbounded by moral constraints, that is, all sides thinking all the other sides are X-risks to them.
In practice things tend not to get that bad most of the time, but sometimes they do, and much of military preparation concern mitigation of these perceived X-risks, the idea being that if “our side” becomes so powerful it can in fact annihilate the others, and in consequence the others submit without resisting, then “our side” may be magnanimous towards them conditional on their continued subservience and submission, but if they resist to the point of becoming an X-risk towards us, then removing them from the equation entirely is the safest defense from the X-risk they pose us.
A global consensus on stopping GAI development due to its X-risk for all life passes through a global consensus, by all sides, that none of the other sides is an X-risk to any of side. Once everyone agrees on this, then they all together agreeing to deal with a global X-risk becomes feasible. Before that, only if they all see that global X-risk as more urgent and immediate than the many local-to-them X-risks.
I see—thanks for the explanation.
The US had nuclear weapons before any other country. Other countries have these weapons now. The p-boom was quite high at some points but nobody was annihilated with this technology.
Admittedly, nuclear weapons are not a perfect analog for AI due to many reasons, but I think it is a reasonable analog.
With this in mind, I wanted to ask out of curiosity, what % risk do you think there needs to be for annihilation to occur?
We’ve had extreme luck when it comes to nuclear weapons. We not only had several close calls that were deescalated by particularly noble individuals doing the right thing, but also, back when the URSS had barely developed theirs and the US alone had a whole stockpile of warheads, we had the good luck of its leadership also being somewhat moral and refusing to turn nukes into a regular weapon, which was followed by MAD forcing everyone to kind of stay so even when the other side asked nicely whether they could bomb a third party. Weren’t for that long sequence of good luck after good luck, and we’d now be living in an annihilated world, or at the very least a post-apocalyptic one.
I have no idea, really. All I can infer is that it’s unlikely any major power will stop trying to achieve GAI unless:
a) Either a massively severe accident due to misaligned not-quite-GAI-yet happens that by its sheer, absolute horror puts the Fear-Of-God in our civilian and military leaders for a few generations;
b) Or a long sequence of reasonably severe accidents happens, each new one worse than the last, with AI companies repeatedly and consistently failing at fixing the underlying cause, this in turn making military leaders deeply wary of deploying advanced AI systems, and civilian leaders enacting restrictions on what AI is allowed to touch.
Absent either of those, I doubt the pursuit of GAI will stop no matter what X-risk analysts say. Or at least, I myself cannot imagine any kind of argument that’d convince, say, the CPC to stop their research when those on the other side spearheading theirs are massively powerful nutjobs? And then, what argument could be provided that’d stop someone who believes in this? So, neither will stop, which means GAI will happen. And then we’ll need to count on luck again, this time with:
i) Either GAI going FOOM as Yudkowsky believes, but for some reason continuing to like humans enough not to turn us into computronium;
ii) Or Hanson being right and FOOM not happening, followed by:
ii.1) Either things being slow enough to “merely” lead to a or b, above;
ii.2) Or things being so immensely slow we can actually fix them.
I have no opinion on whether FOOM is or isn’t likely. I’ve read the entire discussion and all I know is both sets of arguments sound reasonable to me.