We hesitated a lot between including the term “extinction” or not in the beginning.
The final decision not to center the message on “extinction risk” was deliberate: it would have prevented most of the heads of state and organizations from signing. Our goal was to build the broadest and most influential coalition possible to advocate for international red lines, which is what’s most important to us.
By focusing on the concept of “losing meaningful human control,” we were able to achieve agreement on the precursor to most worst-case scenarios, including extinction. We were advised and received feedback from early experiments with signatories that this is a more concrete concept for policymakers and the public.
In summary, if you really want red lines to happen for real, adding the word extinction is not necessary and has more costs than benefits in this text.
This is a very valuable clarification, and I agree[1]. I really appreciate your focus on policy feasibility and concrete approaches.
From my own experience: most people outside AI Safety in the regulatory space, either: lack sufficient background knowledge about timelines and existential risk to meaningfully engage with these concerns and commit to enforceable measures[2] , or those with some familiarity become more skeptical due to the lack of consensus on probabilities, timelines, and definitions.
I will be following this initiative closely and promoting it to the best of my ability.
EDIT: I’ve signed with my institutional email and title.
For transparency: I knew about the red lines project before it was published. Furthermore, Charbel / CeSIA’s past work have shifted my own views on policy and international cooperation.
We hesitated a lot between including the term “extinction” or not in the beginning.
The final decision not to center the message on “extinction risk” was deliberate: it would have prevented most of the heads of state and organizations from signing. Our goal was to build the broadest and most influential coalition possible to advocate for international red lines, which is what’s most important to us.
By focusing on the concept of “losing meaningful human control,” we were able to achieve agreement on the precursor to most worst-case scenarios, including extinction. We were advised and received feedback from early experiments with signatories that this is a more concrete concept for policymakers and the public.
In summary, if you really want red lines to happen for real, adding the word extinction is not necessary and has more costs than benefits in this text.
This is a very valuable clarification, and I agree[1]. I really appreciate your focus on policy feasibility and concrete approaches.
From my own experience: most people outside AI Safety in the regulatory space, either: lack sufficient background knowledge about timelines and existential risk to meaningfully engage with these concerns and commit to enforceable measures[2] , or those with some familiarity become more skeptical due to the lack of consensus on probabilities, timelines, and definitions.
I will be following this initiative closely and promoting it to the best of my ability.
EDIT: I’ve signed with my institutional email and title.
For transparency: I knew about the red lines project before it was published. Furthermore, Charbel / CeSIA’s past work have shifted my own views on policy and international cooperation.
I expect that the popularity of IABIED and more involvent from AI Safety figures in policy will shift the Overton window in this regard.
When you say “prevented” do you just mean it would have been generally off-putting, or is there something specific that you’re referring to here?