The problem with most proposals for an “AGI ban” is that they define the target by outcome (e.g. “powerful AI with the potential to cause human extinction”). I know that even defining AGI is already problematic. But unless we specify and prohibit the actual thing we want to ban, we’ll leave exploitable gray areas wide open. And those loopholes will undermine the very purpose of the ban.
My post argues that for an AGI ban to work, it needs what other existential-risk regimes already have: strict liability, bright lines, and enforceable thresholds. Nuclear treaties don’t ban “weapons that could end humanity”; they ban fissile quantities and test yields. Product liability doesn’t wait for intent; it attaches liability to defects outright.
To actually ban AGI or AI leading to human extintion, we need to ban the precursors that make extinction-plausible systems possible, not “the possibility of extinction” itself.
In nuclear treaties, the ban is tied to bright-line thresholds (8 kg of plutonium, 25 kg of highly enriched uranium, a zero-yield test ban, or delivery systems above 500 kg/300 km). These are crisp, measurable precursors that enable verification and enforcement. So, what are the AGI equivalents of those thresholds?
Until we can define capability-based precursors (functional “red lines” that make extinction-plausible systems possible), any AGI ban will remain rhetoric rather than enforceable law.
I don’t claim to have the answers. My aim is to form the right questions that AI governance discussions should be putting in front of lawyers and policymakers. The whole purpose of this post is to stress-test those assumptions. I’d be genuinely grateful for pushback, alternative framings, or constructive debate.
Genuinely curious about people’s opinions on this—I’d appreciate hearing why you disagree if you downvoted 🙏🏼. Seems like I’ll be thinking about definitions for the foreseeable future, and I need constructive pushback.
An excellent point and it’s important to highlight the works that needs to be done.
(I think you’re stance is more obvious to people in the legal/regulatory/policy space—of course, specific definitions are required and it can’t work otherwise).
The problem with most proposals for an “AGI ban” is that they define the target by outcome (e.g. “powerful AI with the potential to cause human extinction”). I know that even defining AGI is already problematic. But unless we specify and prohibit the actual thing we want to ban, we’ll leave exploitable gray areas wide open. And those loopholes will undermine the very purpose of the ban.
This is why I wrote The Problem with Defining an “AGI Ban” by Outcome (a lawyer’s take).
My post argues that for an AGI ban to work, it needs what other existential-risk regimes already have: strict liability, bright lines, and enforceable thresholds. Nuclear treaties don’t ban “weapons that could end humanity”; they ban fissile quantities and test yields. Product liability doesn’t wait for intent; it attaches liability to defects outright.
To actually ban AGI or AI leading to human extintion, we need to ban the precursors that make extinction-plausible systems possible, not “the possibility of extinction” itself.
In nuclear treaties, the ban is tied to bright-line thresholds (8 kg of plutonium, 25 kg of highly enriched uranium, a zero-yield test ban, or delivery systems above 500 kg/300 km). These are crisp, measurable precursors that enable verification and enforcement. So, what are the AGI equivalents of those thresholds?
Until we can define capability-based precursors (functional “red lines” that make extinction-plausible systems possible), any AGI ban will remain rhetoric rather than enforceable law.
I don’t claim to have the answers. My aim is to form the right questions that AI governance discussions should be putting in front of lawyers and policymakers. The whole purpose of this post is to stress-test those assumptions. I’d be genuinely grateful for pushback, alternative framings, or constructive debate.
Genuinely curious about people’s opinions on this—I’d appreciate hearing why you disagree if you downvoted 🙏🏼. Seems like I’ll be thinking about definitions for the foreseeable future, and I need constructive pushback.
An excellent point and it’s important to highlight the works that needs to be done.
(I think you’re stance is more obvious to people in the legal/regulatory/policy space—of course, specific definitions are required and it can’t work otherwise).