I ask because in some sense, it meets the stated goal of any non-profit if the injustice or issue they work in opposition to were to go away. However, non-profits in practice respond badly to this, and pivot away from the original topic area.
To turn this into a concrete question, suppose we spend $300 million total over the next 5 years, and get a global ASI ban, enforced by some combination of limits on GPUs/compute and limits/reporting on (low-compute) fundamental research. What happens next with your budget, and do you currently have a plan for how to retain the ethos of “let’s not all die to AI” as we pivot from the “JUST STOP IT” phase to the “okay now what” phase.
examples of things that could go wrong (warning: long-run predictions are difficult. these are mostly intuition pumps) - the political coalition that ends up banning ASI includes a generalised anti-technology faction which continues to gain in power and attempt to ban more technologies. Is your org working against this, or is it somebody else’s job. - we stop AI, and twenty years and over a billion dollars later, you start to lose funders.
If we win: first order of business is to celebrate and give the team a long holiday! The most likely cause of death of the authors has been averted.
After that, the work is building the institutions and societal infrastructure so humanity can survive and thrive alongside very powerful AI (and increasingly powerful technology in general). Even with a global ASI ban, powerful AI systems will still exist and society will still go through radical disruption. All versions of the future are wild, even conditional on a ban.
And a ban isn’t a one-and-done. Humanity will be protected from self-annihilation only if there is continued vigilance, and the institutions and people that enforce the ASI regime are maintained and updated as technological progress continues.
ControlAI winding down after winning would be a completely fine outcome. Whether phase two I describe above is ControlAI’s focus or a new venture’s is a good question for after we’ve prevented extinction risk from ASI.
Personally, there are many other causes I care about, such as extending human lifespan and updating political institutions to deal with 21st century technology, that will become higher priority once the largest threat to the continued existence of humanity is averted.
The political coalition that ends up banning ASI includes a generalised anti-technology faction which continues to gain in power and attempt to ban more technologies. Is your org working against this, or is it somebody else’s job.
You’re right that this could happen, and we’d consider it a bad outcome, although a better one than extinction. We don’t support a universally anti-technology coalition, which is one of the many reasons we take care to keep our messaging focused specifically on ASI risk rather than omnicause anti-technologism.
Not a big risk in this case. Once a global ASI ban is signed, the natural pivot is to ensuring that the ban is upheld, enforced, policed, maintained, and strengthened as needed. The work doesn’t stop when the signing ceremony happens.
The political coalition that ends up banning ASI includes a generalised anti-technology faction which continues to gain in power and attempt to ban more technologies.
Way better than going extinct.
We stop AI, and twenty years and over a billion dollars later, you start to lose funders.
We live twenty years longer for a billion dollars. Less than a cent per QALY. Cost-effective.
Edited to add: I’m curious about which sentence here is generating disagreement votes, feel free to emoji-react.
What happens if you win?
I ask because in some sense, it meets the stated goal of any non-profit if the injustice or issue they work in opposition to were to go away. However, non-profits in practice respond badly to this, and pivot away from the original topic area.
To turn this into a concrete question, suppose we spend $300 million total over the next 5 years, and get a global ASI ban, enforced by some combination of limits on GPUs/compute and limits/reporting on (low-compute) fundamental research. What happens next with your budget, and do you currently have a plan for how to retain the ethos of “let’s not all die to AI” as we pivot from the “JUST STOP IT” phase to the “okay now what” phase.
examples of things that could go wrong (warning: long-run predictions are difficult. these are mostly intuition pumps)
- the political coalition that ends up banning ASI includes a generalised anti-technology faction which continues to gain in power and attempt to ban more technologies. Is your org working against this, or is it somebody else’s job.
- we stop AI, and twenty years and over a billion dollars later, you start to lose funders.
If we win: first order of business is to celebrate and give the team a long holiday! The most likely cause of death of the authors has been averted.
After that, the work is building the institutions and societal infrastructure so humanity can survive and thrive alongside very powerful AI (and increasingly powerful technology in general). Even with a global ASI ban, powerful AI systems will still exist and society will still go through radical disruption. All versions of the future are wild, even conditional on a ban.
And a ban isn’t a one-and-done. Humanity will be protected from self-annihilation only if there is continued vigilance, and the institutions and people that enforce the ASI regime are maintained and updated as technological progress continues.
ControlAI winding down after winning would be a completely fine outcome. Whether phase two I describe above is ControlAI’s focus or a new venture’s is a good question for after we’ve prevented extinction risk from ASI.
Whether that work eventually entails finding ways to safely develop ASI under operationally adequate institutions and just processes, or not, is unclear to us. And in any case, a globally enforced ASI ban is the precondition for ever doing a “safe ASI” plan.
Personally, there are many other causes I care about, such as extending human lifespan and updating political institutions to deal with 21st century technology, that will become higher priority once the largest threat to the continued existence of humanity is averted.
You’re right that this could happen, and we’d consider it a bad outcome, although a better one than extinction. We don’t support a universally anti-technology coalition, which is one of the many reasons we take care to keep our messaging focused specifically on ASI risk rather than omnicause anti-technologism.
Not a big risk in this case. Once a global ASI ban is signed, the natural pivot is to ensuring that the ban is upheld, enforced, policed, maintained, and strengthened as needed. The work doesn’t stop when the signing ceremony happens.
Way better than going extinct.
We live twenty years longer for a billion dollars. Less than a cent per QALY. Cost-effective.
Edited to add: I’m curious about which sentence here is generating disagreement votes, feel free to emoji-react.