Can AI destroy modern civilization in the next 30 minutes?
Doubt it, but it might depend on how much of an overhang we have. My timelines aren’t that short, but if there were an overhang and we were just a few breakthroughs away from recursive self-improvement, would the world look any different than it does now?
Can a single human being unilaterally decide to make that happen, right now, today?
Oh, good point. Pilots have intentionally crashed planes full of passengers. Kids have shot up schools, not expecting to come out alive. Murder-suicide is a thing humans have been known to do. There have been a number of well-documented close calls in the Cold War. As nuclear powers proliferate, MAD becomes more complicated.
It’s still about #3 on my catastrophic risk list depending on how you count things. But the number of humans who could plausibly do this remains relatively small. How many human beings could plausibly bioengineer a pandemic? I think the number is greater, and increasing as biotech advances. Time is not the only factor in risk calculations.
And likely neither of these results in human extinction, but the pandemic scares me more. No, nuclear war wouldn’t do it. That would require salted bombs, which have been theorized, but never deployed. Can’t happen in the next 30 minutes. Fallout become survivable (if unhealthy) in a few days. Nobody is really interested in bombing New Zealand. They’re too far away from everybody else to matter. Nuclear winter risk has been greatly exaggerated, and humans are more omnivorous than you’d think, especially with even simple technology helping to process food sources. Not to say that a nuclear war wouldn’t be catastrophic, but there would be survivors. A lot of them.
A communicable disease that’s too deadly (like SARS-1) tends to burn itself out before spreading much, but an engineered (or natural!) pandemic could plausibly thread the needle and become something at least as bad as smallpox. A highly contagious disease that doesn’t kill outright but causes brain damage or sterility might be similarly devastating to civilization, without being so self-limiting. Even New Zealand might not be safe. A nuclear war ends. A pandemic festers. Outcomes could be worse, and it’s more likely to happen, and becoming more likely to happen. It’s #2 for me.
And #1 is an intelligence explosion. This is not just a catastrophic risk, but an existential one. An unaligned AI destroys all value, by default. It’s not going to have a conscience unless we put one in. Nobody knows how to do that. And short of a collapse of civilization, an AI takeover seems inevitable in short order. We either figure out how to build one that’s aligned before that happens, and it solves all the other solvable risks, or everybody dies.
Doubt it, but it might depend on how much of an overhang we have. My timelines aren’t that short, but if there were an overhang and we were just a few breakthroughs away from recursive self-improvement, would the world look any different than it does now?
Oh, good point. Pilots have intentionally crashed planes full of passengers. Kids have shot up schools, not expecting to come out alive. Murder-suicide is a thing humans have been known to do. There have been a number of well-documented close calls in the Cold War. As nuclear powers proliferate, MAD becomes more complicated.
It’s still about #3 on my catastrophic risk list depending on how you count things. But the number of humans who could plausibly do this remains relatively small. How many human beings could plausibly bioengineer a pandemic? I think the number is greater, and increasing as biotech advances. Time is not the only factor in risk calculations.
And likely neither of these results in human extinction, but the pandemic scares me more. No, nuclear war wouldn’t do it. That would require salted bombs, which have been theorized, but never deployed. Can’t happen in the next 30 minutes. Fallout become survivable (if unhealthy) in a few days. Nobody is really interested in bombing New Zealand. They’re too far away from everybody else to matter. Nuclear winter risk has been greatly exaggerated, and humans are more omnivorous than you’d think, especially with even simple technology helping to process food sources. Not to say that a nuclear war wouldn’t be catastrophic, but there would be survivors. A lot of them.
A communicable disease that’s too deadly (like SARS-1) tends to burn itself out before spreading much, but an engineered (or natural!) pandemic could plausibly thread the needle and become something at least as bad as smallpox. A highly contagious disease that doesn’t kill outright but causes brain damage or sterility might be similarly devastating to civilization, without being so self-limiting. Even New Zealand might not be safe. A nuclear war ends. A pandemic festers. Outcomes could be worse, and it’s more likely to happen, and becoming more likely to happen. It’s #2 for me.
And #1 is an intelligence explosion. This is not just a catastrophic risk, but an existential one. An unaligned AI destroys all value, by default. It’s not going to have a conscience unless we put one in. Nobody knows how to do that. And short of a collapse of civilization, an AI takeover seems inevitable in short order. We either figure out how to build one that’s aligned before that happens, and it solves all the other solvable risks, or everybody dies.