FOOM scenarios where things go deeply wrong, very fast, due to things like rapid recursive self-improvement into the far-superhuman range, or wiping out humanity with custom tailored viruses, etc. The Yudkowsky scenarios, basically. I don’t think these are nearly as guaranteed as Yudkowsky has argued in the past.
Scenarios where humans can no longer understand the real economy, or the cutting edge of technology, or even how to fight real wars. In this scenario, sure, you might maintain the forms of democratic governance. But every time you vote “Let the AIs do more with less supervision,” you all get richer and nothing bad happens. Or maybe you don’t really get to vote meaningfully, and the billionaires just jam the changes through the system. (I, uh, live in the US, so this seems pretty plausible to me.)
So that’s my key breakdown:
Scenarios where we lose immediately, game over, the end.
Scenarios where creating a species that is to us as we were to chimps inevitably plays out with humans losing control over the medium term. And in many of these scenarios economic logic essentially forces us to hand over control, or get outcompeted by the people who do.
Even before AGI, I think we’re already pretty far down the latter road. Anthropic is cognitively captured by Claude, their darling baby, and they’re clearly going to push full speed ahead towards superintelligence. And as a programmer, I’m going to have to hand over 80% of my job to AI within 2 years or become unemployable. And programmers are just the first of many white collar professions who will face this. And certainly the US political system is not up to dealing with any of this, or meaningfully regulating it at the moment.
Thank you for the clarification, that helps!
Personally, I specifically distinguish between:
FOOM scenarios where things go deeply wrong, very fast, due to things like rapid recursive self-improvement into the far-superhuman range, or wiping out humanity with custom tailored viruses, etc. The Yudkowsky scenarios, basically. I don’t think these are nearly as guaranteed as Yudkowsky has argued in the past.
Scenarios where humans can no longer understand the real economy, or the cutting edge of technology, or even how to fight real wars. In this scenario, sure, you might maintain the forms of democratic governance. But every time you vote “Let the AIs do more with less supervision,” you all get richer and nothing bad happens. Or maybe you don’t really get to vote meaningfully, and the billionaires just jam the changes through the system. (I, uh, live in the US, so this seems pretty plausible to me.)
So that’s my key breakdown:
Scenarios where we lose immediately, game over, the end.
Scenarios where creating a species that is to us as we were to chimps inevitably plays out with humans losing control over the medium term. And in many of these scenarios economic logic essentially forces us to hand over control, or get outcompeted by the people who do.
Even before AGI, I think we’re already pretty far down the latter road. Anthropic is cognitively captured by Claude, their darling baby, and they’re clearly going to push full speed ahead towards superintelligence. And as a programmer, I’m going to have to hand over 80% of my job to AI within 2 years or become unemployable. And programmers are just the first of many white collar professions who will face this. And certainly the US political system is not up to dealing with any of this, or meaningfully regulating it at the moment.