What I push for instead: Red lines now, IAEA for AI next
For context, the IAEA (International Atomic Energy Agency) issues international nuclear safety standards and red lines, supports peer reviews and inspections, and coordinates assistance during nuclear emergencies. These standards are then adopted and enforced through national legislation worldwide. An IAEA for AI would play a similar role for artificial intelligence.
Red lines have something CERN doesn’t: existing momentum.
Red lines are the most widely supported measure by research institutes, think tanks, and independent organizations. By signing the Frontier AI Safety Commitments Seoul, companies agreed to “Set out thresholds at which severe risks posed by a model or system, unless adequately mitigated, would be deemed intolerable.” Granted, OpenAI’s “red line” for recursive AI self-improvement is currently inadequate, but we’re not building from zero, and this is why red lines need to be binding rather than voluntary.
The final big hesitation while drafting the Global Call for AI Red Lines was not the CERN, but asking explicitly for the IAEA for AI. The main reason we didn’t ask for the IAEA in the global call was mostly optic (“IAEA for AI” sounds technocratic and wonky to non-specialists, while red lines are intuitive). I think red lines are the right policy ask now. The right institutional ask to operationalize it is an IAEA for AI, and CeSIA will be pushing for it as the next phase.
At the India AI Impact Summit, the CEOs of the three leading frontier AI labs each called for international AI oversight.Altman joined Hassabis in calling for an institution modeled on the IAEA. Amodei called for red lines with enforcement mechanisms.[1] The fact that CEOs have recently publicly called for IAEA-style oversight is one of the strongest arguments for the current US administration.
This sequencing – an international agreement with red lines first, institution second – mirrors how international governance actually works. The EU AI Act passed without every technical threshold defined; the AI Office was established afterward; specific evals are currently being defined with the advice of technical consortia working with the EU AI Office. Same pattern from the Vienna Convention to the Montreal Protocol, with detailed control measures strengthened gradually through expert-led review. Political agreement creates the conditions for technical work to happen inside the governance process, not before it.
What I push for instead: Red lines now, IAEA for AI next
For context, the IAEA (International Atomic Energy Agency) issues international nuclear safety standards and red lines, supports peer reviews and inspections, and coordinates assistance during nuclear emergencies. These standards are then adopted and enforced through national legislation worldwide. An IAEA for AI would play a similar role for artificial intelligence.
Red lines have something CERN doesn’t: existing momentum.
Red lines are the most widely supported measure by research institutes, think tanks, and independent organizations. By signing the Frontier AI Safety Commitments Seoul, companies agreed to “Set out thresholds at which severe risks posed by a model or system, unless adequately mitigated, would be deemed intolerable.” Granted, OpenAI’s “red line” for recursive AI self-improvement is currently inadequate, but we’re not building from zero, and this is why red lines need to be binding rather than voluntary.
China’s Premier Li Qiang stated that “there should be a red line in AI development, a red line that must not be crossed.” Pope Francis urged nations to adopt “a binding international treaty.”, and Paolo Benanti, the Pope’s AI adviser, called explicitly for “binding international treaties and red lines.”
Red lines need an institution: the IAEA model
The final big hesitation while drafting the Global Call for AI Red Lines was not the CERN, but asking explicitly for the IAEA for AI. The main reason we didn’t ask for the IAEA in the global call was mostly optic (“IAEA for AI” sounds technocratic and wonky to non-specialists, while red lines are intuitive). I think red lines are the right policy ask now. The right institutional ask to operationalize it is an IAEA for AI, and CeSIA will be pushing for it as the next phase.
At the India AI Impact Summit, the CEOs of the three leading frontier AI labs each called for international AI oversight. Altman joined Hassabis in calling for an institution modeled on the IAEA. Amodei called for red lines with enforcement mechanisms.[1] The fact that CEOs have recently publicly called for IAEA-style oversight is one of the strongest arguments for the current US administration.
This sequencing – an international agreement with red lines first, institution second – mirrors how international governance actually works. The EU AI Act passed without every technical threshold defined; the AI Office was established afterward; specific evals are currently being defined with the advice of technical consortia working with the EU AI Office. Same pattern from the Vienna Convention to the Montreal Protocol, with detailed control measures strengthened gradually through expert-led review. Political agreement creates the conditions for technical work to happen inside the governance process, not before it.
The CEOs of the three leading AI companies have each publicly called for international oversight. Dario Amodei said he could imagine a worldwide treaty with enforcement mechanisms. Sam Altman called for “urgent global regulation on AI”, and for an equivalent of the International Atomic Energy Agency for international coordination on AI. Demis Hassabis also called for “some kind of equivalent of the IAEA.” For reference, the IAEA issues international nuclear safety standards and red lines, supports peer reviews and inspections, and coordinates assistance during nuclear emergencies. These standards are then adopted and enforced through national legislation worldwide.