If we win: first order of business is to celebrate and give the team a long holiday! The most likely cause of death of the authors has been averted.
After that, the work is building the institutions and societal infrastructure so humanity can survive and thrive alongside very powerful AI (and increasingly powerful technology in general). Even with a global ASI ban, powerful AI systems will still exist and society will still go through radical disruption. All versions of the future are wild, even conditional on a ban.
And a ban isn’t a one-and-done. Humanity will be protected from self-annihilation only if there is continued vigilance, and the institutions and people that enforce the ASI regime are maintained and updated as technological progress continues.
ControlAI winding down after winning would be a completely fine outcome. Whether phase two I describe above is ControlAI’s focus or a new venture’s is a good question for after we’ve prevented extinction risk from ASI.
Personally, there are many other causes I care about, such as extending human lifespan and updating political institutions to deal with 21st century technology, that will become higher priority once the largest threat to the continued existence of humanity is averted.
The political coalition that ends up banning ASI includes a generalised anti-technology faction which continues to gain in power and attempt to ban more technologies. Is your org working against this, or is it somebody else’s job.
You’re right that this could happen, and we’d consider it a bad outcome, although a better one than extinction. We don’t support a universally anti-technology coalition, which is one of the many reasons we take care to keep our messaging focused specifically on ASI risk rather than omnicause anti-technologism.
If we win: first order of business is to celebrate and give the team a long holiday! The most likely cause of death of the authors has been averted.
After that, the work is building the institutions and societal infrastructure so humanity can survive and thrive alongside very powerful AI (and increasingly powerful technology in general). Even with a global ASI ban, powerful AI systems will still exist and society will still go through radical disruption. All versions of the future are wild, even conditional on a ban.
And a ban isn’t a one-and-done. Humanity will be protected from self-annihilation only if there is continued vigilance, and the institutions and people that enforce the ASI regime are maintained and updated as technological progress continues.
ControlAI winding down after winning would be a completely fine outcome. Whether phase two I describe above is ControlAI’s focus or a new venture’s is a good question for after we’ve prevented extinction risk from ASI.
Whether that work eventually entails finding ways to safely develop ASI under operationally adequate institutions and just processes, or not, is unclear to us. And in any case, a globally enforced ASI ban is the precondition for ever doing a “safe ASI” plan.
Personally, there are many other causes I care about, such as extending human lifespan and updating political institutions to deal with 21st century technology, that will become higher priority once the largest threat to the continued existence of humanity is averted.
You’re right that this could happen, and we’d consider it a bad outcome, although a better one than extinction. We don’t support a universally anti-technology coalition, which is one of the many reasons we take care to keep our messaging focused specifically on ASI risk rather than omnicause anti-technologism.