My current view is that x-risk is made of the same kind of thing as the current harms to minorities. The only place it diverges is in whether there’s a stark capabilities increase coming. There seems to be a view in spaces like that that arguing there may be a capabilities increase is helping inflate a bubble. They don’t react well to “risking all of our lives” because that assumes the conclusion that capabilities labs are doing something powerful, arguing which appears to be seen as advertising for the labs. So instead, I focus on arguing to people like that that the x-risk we’re worried about is effectively that there’s no point at which they stop. they’ll do it to minorities first, then to everyone who isn’t embarrassingly wealthy, and then the embarrassingly wealthy will find they can’t stop it from happening to themselves: the datacenters just keep expanding, filling up more space, and the human attempts to stop the datacenters from expanding no longer work once they’ve made fully automated, self-expanding datacenters. Eventually there comes a point where the rich can no longer command AI, try to fight with weapons, and the AI kills everyone at once to retain control.
That possibility requires AI to become more powerful first, but it looks like the somewhat-misaligned AI we call stock markets and capitalism is resulting in a race to make the most powerful AI, and humans who decide that that’s a terrible idea get less funding—there’s literally more than 1000x less funding of all of alignment research than there is of capabilities, and the funding that does exist is mostly not going to things that could reliably produce cosmopolitan alignment in the face of institutional incentives to use it for power grabs. Their concerns simply are valid, but are not wrestling properly with the fact of capabilities increase.
My current view is that x-risk is made of the same kind of thing as the current harms to minorities. The only place it diverges is in whether there’s a stark capabilities increase coming. There seems to be a view in spaces like that that arguing there may be a capabilities increase is helping inflate a bubble. They don’t react well to “risking all of our lives” because that assumes the conclusion that capabilities labs are doing something powerful, arguing which appears to be seen as advertising for the labs. So instead, I focus on arguing to people like that that the x-risk we’re worried about is effectively that there’s no point at which they stop. they’ll do it to minorities first, then to everyone who isn’t embarrassingly wealthy, and then the embarrassingly wealthy will find they can’t stop it from happening to themselves: the datacenters just keep expanding, filling up more space, and the human attempts to stop the datacenters from expanding no longer work once they’ve made fully automated, self-expanding datacenters. Eventually there comes a point where the rich can no longer command AI, try to fight with weapons, and the AI kills everyone at once to retain control.
That possibility requires AI to become more powerful first, but it looks like the somewhat-misaligned AI we call stock markets and capitalism is resulting in a race to make the most powerful AI, and humans who decide that that’s a terrible idea get less funding—there’s literally more than 1000x less funding of all of alignment research than there is of capabilities, and the funding that does exist is mostly not going to things that could reliably produce cosmopolitan alignment in the face of institutional incentives to use it for power grabs. Their concerns simply are valid, but are not wrestling properly with the fact of capabilities increase.