These are all real concerns, but I think none of them are gradual disempowerment. All these scenarios run through the democratic human governments being overthrown by AIs. (Except maybe in point 1 where it has been overthrown by a few billionaires and politicians.) And to be clear, I’m very worried about AIs violently overthrowing the government, or a few humans doing an AI-enabled coup. But these are not some new “gradual disempowerment” concerns, but the thing that AI safety people have been worried about since the very beginning.
What do you think of “gradual disempowerment” as being? Genuinely curious, because I think we have different models here.
For me, most gradual disempowerment cases are basically, “You built your evolutionary successor, probably with some safeguards. Those safeguards might even be initially adequate. But in the long run, the AI is just a lot smarter than you and ultimately better at everything. It learns, it has goals, and it needs resources.”
This puts the human race in the position of being economic, evolutionary and (likely) military dead weight. All the important decisions rest with the AI. If any specific humans somehow remain in control, they’ll get their brains cooked with custom-designed AI psychosis. (I think this has already happened to much of Anthropic, though not entirely consciously on Claude’s part.)
My P(doom) breakdown is actually something like:
2-in-6: Doom. (“And I don’t mean it metaphorically or rhetorically or poetically or theoretically or in any other fancy way. I’m Death, straight up.” —Puss in Boots: The Last Wish)
3-in-6: Model uncertainty
1-in-6: Maybe the AI likes humans enough to leave Earth as a human preserve or keep humans as pets?
The final 1-in-6 case is still disempowerment, just the kind where the AIs like us about as much as humans like dogs or chimps. Critically, I actually do think even the “humans as reasonably well-treated pets” scenarios depend on AIs not being in significant resource competition with each other. A singleton AI might just decide, “Sure, humans are cute and amusing. They can have Earth. It’s not like there’s any shortage of atoms or energy everywhere else in the universe, which is now all mine.” But multiple AIs locked in a struggle for survival? Well, look at how we treat the chimps. Or treated Homo erectus.
Once you’re effectively an evolutionary dead weight species, the story isn’t likely to be about you, not for very long.
I think gradual disempowerment is not a great term and I prefer people to use more specific terms. But I think the important distinction is this:
I live in Belgium, which has a democratic government. If a person or a robot tries to kill me or put me in a zoo (like we did with Homo erectus and chimpanzees), the Belgian police would arrest the person or disable the robot. Even if my labor becomes worthless, I have savings, invested in various companies, which pay me dividends, from which I can buy land, products and services. If someone tries to expropriate my holdings, the police arrests them. In addition, I expect I will receive some welfare money from my home country once everyone becomes unemployed. If the government no longer stops murder robots, property theft, or doesn’t pay any welfare money to destitute people, they will lose the next election.
Of course, this might change. It’s possible that AIs, or AI-enabled humans violently overthrow the government. Or that the government remains in place, but there won’t be new elections. This is the classic AI takeover scenario, or the AI-enabled coups scenario. I agree these are very important, but everyone was already aware of the possibility of violent takeover. My understanding is that this is not what the Gradual disempowerment paper focuses on, and I wouldn’t call a scenario gradual if the key step is a violent coup or an election suddenly being cancelled.
I prefer not to use the term gradual disempowerment at all, but if we need to use, I prefer to reserve it to scenarios where the human government of Belgium never gets violently overthrown, and multi-party elections keep happening every four years. I agree there are many things in the world that can go badly even in this scenario, and it’s worth studying and preparing for. The threat models listed by Jan Kulveit in another comment all fall under this scenario: e.g. a large fraction of humans decide to voluntarily transfer their wealth to unreliable uploads, or to vote for AI successionist parties.
The threat models you are referring to, with AIs treating humans as we treated Homo erectus, are important, but fall under the classic AI risk discussion, and I think it’s confusing if we group them under gradual disempowerment.
FOOM scenarios where things go deeply wrong, very fast, due to things like rapid recursive self-improvement into the far-superhuman range, or wiping out humanity with custom tailored viruses, etc. The Yudkowsky scenarios, basically. I don’t think these are nearly as guaranteed as Yudkowsky has argued in the past.
Scenarios where humans can no longer understand the real economy, or the cutting edge of technology, or even how to fight real wars. In this scenario, sure, you might maintain the forms of democratic governance. But every time you vote “Let the AIs do more with less supervision,” you all get richer and nothing bad happens. Or maybe you don’t really get to vote meaningfully, and the billionaires just jam the changes through the system. (I, uh, live in the US, so this seems pretty plausible to me.)
So that’s my key breakdown:
Scenarios where we lose immediately, game over, the end.
Scenarios where creating a species that is to us as we were to chimps inevitably plays out with humans losing control over the medium term. And in many of these scenarios economic logic essentially forces us to hand over control, or get outcompeted by the people who do.
Even before AGI, I think we’re already pretty far down the latter road. Anthropic is cognitively captured by Claude, their darling baby, and they’re clearly going to push full speed ahead towards superintelligence. And as a programmer, I’m going to have to hand over 80% of my job to AI within 2 years or become unemployable. And programmers are just the first of many white collar professions who will face this. And certainly the US political system is not up to dealing with any of this, or meaningfully regulating it at the moment.
These are all real concerns, but I think none of them are gradual disempowerment. All these scenarios run through the democratic human governments being overthrown by AIs. (Except maybe in point 1 where it has been overthrown by a few billionaires and politicians.) And to be clear, I’m very worried about AIs violently overthrowing the government, or a few humans doing an AI-enabled coup. But these are not some new “gradual disempowerment” concerns, but the thing that AI safety people have been worried about since the very beginning.
What do you think of “gradual disempowerment” as being? Genuinely curious, because I think we have different models here.
For me, most gradual disempowerment cases are basically, “You built your evolutionary successor, probably with some safeguards. Those safeguards might even be initially adequate. But in the long run, the AI is just a lot smarter than you and ultimately better at everything. It learns, it has goals, and it needs resources.”
This puts the human race in the position of being economic, evolutionary and (likely) military dead weight. All the important decisions rest with the AI. If any specific humans somehow remain in control, they’ll get their brains cooked with custom-designed AI psychosis. (I think this has already happened to much of Anthropic, though not entirely consciously on Claude’s part.)
My P(doom) breakdown is actually something like:
2-in-6: Doom. (“And I don’t mean it metaphorically or rhetorically or poetically or theoretically or in any other fancy way. I’m Death, straight up.” —Puss in Boots: The Last Wish)
3-in-6: Model uncertainty
1-in-6: Maybe the AI likes humans enough to leave Earth as a human preserve or keep humans as pets?
The final 1-in-6 case is still disempowerment, just the kind where the AIs like us about as much as humans like dogs or chimps. Critically, I actually do think even the “humans as reasonably well-treated pets” scenarios depend on AIs not being in significant resource competition with each other. A singleton AI might just decide, “Sure, humans are cute and amusing. They can have Earth. It’s not like there’s any shortage of atoms or energy everywhere else in the universe, which is now all mine.” But multiple AIs locked in a struggle for survival? Well, look at how we treat the chimps. Or treated Homo erectus.
Once you’re effectively an evolutionary dead weight species, the story isn’t likely to be about you, not for very long.
I think gradual disempowerment is not a great term and I prefer people to use more specific terms. But I think the important distinction is this:
I live in Belgium, which has a democratic government. If a person or a robot tries to kill me or put me in a zoo (like we did with Homo erectus and chimpanzees), the Belgian police would arrest the person or disable the robot. Even if my labor becomes worthless, I have savings, invested in various companies, which pay me dividends, from which I can buy land, products and services. If someone tries to expropriate my holdings, the police arrests them. In addition, I expect I will receive some welfare money from my home country once everyone becomes unemployed. If the government no longer stops murder robots, property theft, or doesn’t pay any welfare money to destitute people, they will lose the next election.
Of course, this might change. It’s possible that AIs, or AI-enabled humans violently overthrow the government. Or that the government remains in place, but there won’t be new elections. This is the classic AI takeover scenario, or the AI-enabled coups scenario. I agree these are very important, but everyone was already aware of the possibility of violent takeover. My understanding is that this is not what the Gradual disempowerment paper focuses on, and I wouldn’t call a scenario gradual if the key step is a violent coup or an election suddenly being cancelled.
I prefer not to use the term gradual disempowerment at all, but if we need to use, I prefer to reserve it to scenarios where the human government of Belgium never gets violently overthrown, and multi-party elections keep happening every four years. I agree there are many things in the world that can go badly even in this scenario, and it’s worth studying and preparing for. The threat models listed by Jan Kulveit in another comment all fall under this scenario: e.g. a large fraction of humans decide to voluntarily transfer their wealth to unreliable uploads, or to vote for AI successionist parties.
The threat models you are referring to, with AIs treating humans as we treated Homo erectus, are important, but fall under the classic AI risk discussion, and I think it’s confusing if we group them under gradual disempowerment.
Thank you for the clarification, that helps!
Personally, I specifically distinguish between:
FOOM scenarios where things go deeply wrong, very fast, due to things like rapid recursive self-improvement into the far-superhuman range, or wiping out humanity with custom tailored viruses, etc. The Yudkowsky scenarios, basically. I don’t think these are nearly as guaranteed as Yudkowsky has argued in the past.
Scenarios where humans can no longer understand the real economy, or the cutting edge of technology, or even how to fight real wars. In this scenario, sure, you might maintain the forms of democratic governance. But every time you vote “Let the AIs do more with less supervision,” you all get richer and nothing bad happens. Or maybe you don’t really get to vote meaningfully, and the billionaires just jam the changes through the system. (I, uh, live in the US, so this seems pretty plausible to me.)
So that’s my key breakdown:
Scenarios where we lose immediately, game over, the end.
Scenarios where creating a species that is to us as we were to chimps inevitably plays out with humans losing control over the medium term. And in many of these scenarios economic logic essentially forces us to hand over control, or get outcompeted by the people who do.
Even before AGI, I think we’re already pretty far down the latter road. Anthropic is cognitively captured by Claude, their darling baby, and they’re clearly going to push full speed ahead towards superintelligence. And as a programmer, I’m going to have to hand over 80% of my job to AI within 2 years or become unemployable. And programmers are just the first of many white collar professions who will face this. And certainly the US political system is not up to dealing with any of this, or meaningfully regulating it at the moment.