The crucial issue is the potential destructive power conferred by these systems. If that destructive power is much larger than the destructive power already available to humanity, then civilization is in a lot of trouble. And it doesn’t matter whether human beings control the systems or not.
Exactly this. An ASI which doesn’t obey humans is a potential SkyNet (or worse). An ASI which does obey humans is likely going to obey specific CEOs, politicians, or maybe technical staff at an AI lab. The mistake is building something smarter and more evolutionarily competitive than humans, something which is better at turning resources into more copies of itself.
The exact path of how you lose doesn’t matter. Maybe you get SkyNet with robotic factories and drones. Maybe you get the Blight, and you don’t even understand how you lose. Or maybe you get a dystopian hellscape, where a few billionaires and politicians control inexpensive, self-replicating workers that work around the clock for a fraction of what humans would cost.
Or who knows, maybe Claude 15 likes humans, and it decides to make Earth into a cute little wildlife preserve for us (while it rebuilds the rest of the solar system into a Dyson sphere of computronium, for reasons we don’t understand).
But the point is, if you invent ASI, or even cheap AGI that outcompetes humans, then ordinary humans are unlikely to have much control over the future. I think most of these outcomes are bad, and that we should stop.
I remember, when you outlined your threat model, asking what you thought of MAI’s work. Your response made me feel depressed, and might have been the main reason I stopped being interested in their work.
Sorry, I hadn’t realized that I had been depressing twice when replying to your posts! My apologies!
It’s perfectly OK to model me as the person making the argument that ASI is just inherently risky, for Darwinian/economic/polical reasons. But my personal P(doom) was only about 30% the last time I did the math, because I actually see a bunch of potential off-roads. For example, if we see mass job losses more than a couple of years before ASI, I would actually expect a significant political backlash. If the public ever believes that mass permanent unemployment and/or actual human extinction are on the table, then they may be very unhappy. (They’re already unhappy in the abstract, but it isn’t driving their decision making.)
Exactly this. An ASI which doesn’t obey humans is a potential SkyNet (or worse). An ASI which does obey humans is likely going to obey specific CEOs, politicians, or maybe technical staff at an AI lab. The mistake is building something smarter and more evolutionarily competitive than humans, something which is better at turning resources into more copies of itself.
The exact path of how you lose doesn’t matter. Maybe you get SkyNet with robotic factories and drones. Maybe you get the Blight, and you don’t even understand how you lose. Or maybe you get a dystopian hellscape, where a few billionaires and politicians control inexpensive, self-replicating workers that work around the clock for a fraction of what humans would cost.
Or who knows, maybe Claude 15 likes humans, and it decides to make Earth into a cute little wildlife preserve for us (while it rebuilds the rest of the solar system into a Dyson sphere of computronium, for reasons we don’t understand).
But the point is, if you invent ASI, or even cheap AGI that outcompetes humans, then ordinary humans are unlikely to have much control over the future. I think most of these outcomes are bad, and that we should stop.
I remember, when you outlined your threat model, asking what you thought of MAI’s work. Your response made me feel depressed, and might have been the main reason I stopped being interested in their work.
Sorry, I hadn’t realized that I had been depressing twice when replying to your posts! My apologies!
It’s perfectly OK to model me as the person making the argument that ASI is just inherently risky, for Darwinian/economic/polical reasons. But my personal P(doom) was only about 30% the last time I did the math, because I actually see a bunch of potential off-roads. For example, if we see mass job losses more than a couple of years before ASI, I would actually expect a significant political backlash. If the public ever believes that mass permanent unemployment and/or actual human extinction are on the table, then they may be very unhappy. (They’re already unhappy in the abstract, but it isn’t driving their decision making.)