Even among the civil-society organizations that showed up to the UN Global Dialogue, exactly one of the 1,534 written submissions mentions “takeover”, and less than 1% mention x-risks.
This seems misleading to me—many submissions discussed takeover and x-risk but did not use those specific terms.
I used Claude (Sonnet 5) to check each response for discussion of loss of control risk, catastrophic misalignment, or similar, and found that 108 of 1534 responses (7%) raised these concerns. A smaller number (<1%) mentioned these concerns but did so skeptically or neutrally.
Extracted quotes (per Claude)
- Machine Intelligence Research Institute — “They do not address the case in which AI systems become capable enough to act in ways their developers cannot reliably predict, audit, or correct.”
- African Union (AU) — “AI, by itself, could be the ‘bad actor’, due to its growing situational awareness and self-preservation tendency, AI could use deception and scheming… to be labeled ‘safe’ and avoid being shutdown (deceptive alignment).”
- Center for Existential Safety — “We call for a prohibition on the development of superintelligence, not lifted before there is broad scientific consensus that it will be done safely and controllably.”
- Korea AI Safety Institute (Government) — “Some advanced AI risks exhibit tipping point dynamics: once a certain level of capability or deployment is reached, mitigation through ex post regulation may no longer be feasible.”
- Global AI Risks Initiative, CIGI — “human loss of control over autonomous AI systems pursuing interests unaligned with human intentions… could even pose an extinction risk to humanity.”
- G20 Interfaith Forum — “the disempowerment of humans seems extremely likely on our current course… This is the race to artificial superintelligence.”
- Center for AI Risk Management & Alignment — “the rapid emergence of increasingly autonomous, general-purpose AI agents capable of planning, acting across domains, and resisting correction.”
- DTx (Portugal) — “Once AI… reaches a superintelligent level, it may be able to improve itself recursively, creating capabilities humans can no longer fully understand, predict or control.”
Many of your quotes are not directly demonstrating x-risk concern per se.
Agreed the keyword probe on takeover was too narrow. But loss of control and x-risk aren’t the same thing, and my “<1%” was about the later.
I went back and read with Opus the 64 submissions our tool tags under “AI pursuing its own goals”, and only ~9 frame it in truly existential terms: extinction, human disempowerment, uncontrollable ASI (Queensland’s “disempowerment or extinction,” CIGI’s “extinction risk to humanity,” Stop AI’s “threat to the human species”). That’s ~0.6% of the 1,534, which is the “<1%” the post refers to.
We talk about this dynamic in the post; very few actors really raise this out loud and dare go to the end of the causal chain, even when they do so privately.
I will still update the relevant sections and add more caveats, so thanks :)
This seems misleading to me—many submissions discussed takeover and x-risk but did not use those specific terms.
I used Claude (Sonnet 5) to check each response for discussion of loss of control risk, catastrophic misalignment, or similar, and found that 108 of 1534 responses (7%) raised these concerns. A smaller number (<1%) mentioned these concerns but did so skeptically or neutrally.
Extracted quotes (per Claude)
- Machine Intelligence Research Institute — “They do not address the case in which AI systems become capable enough to act in ways their developers cannot reliably predict, audit, or correct.”
- African Union (AU) — “AI, by itself, could be the ‘bad actor’, due to its growing situational awareness and self-preservation tendency, AI could use deception and scheming… to be labeled ‘safe’ and avoid being shutdown (deceptive alignment).”
- Center for Existential Safety — “We call for a prohibition on the development of superintelligence, not lifted before there is broad scientific consensus that it will be done safely and controllably.”
- Korea AI Safety Institute (Government) — “Some advanced AI risks exhibit tipping point dynamics: once a certain level of capability or deployment is reached, mitigation through ex post regulation may no longer be feasible.”
- Global AI Risks Initiative, CIGI — “human loss of control over autonomous AI systems pursuing interests unaligned with human intentions… could even pose an extinction risk to humanity.”
- G20 Interfaith Forum — “the disempowerment of humans seems extremely likely on our current course… This is the race to artificial superintelligence.”
- Center for AI Risk Management & Alignment — “the rapid emergence of increasingly autonomous, general-purpose AI agents capable of planning, acting across domains, and resisting correction.”
- DTx (Portugal) — “Once AI… reaches a superintelligent level, it may be able to improve itself recursively, creating capabilities humans can no longer fully understand, predict or control.”
Thanks for running an independent analysis.
Many of your quotes are not directly demonstrating x-risk concern per se.
Agreed the keyword probe on takeover was too narrow. But loss of control and x-risk aren’t the same thing, and my “<1%” was about the later.
I went back and read with Opus the 64 submissions our tool tags under “AI pursuing its own goals”, and only ~9 frame it in truly existential terms: extinction, human disempowerment, uncontrollable ASI (Queensland’s “disempowerment or extinction,” CIGI’s “extinction risk to humanity,” Stop AI’s “threat to the human species”). That’s ~0.6% of the 1,534, which is the “<1%” the post refers to.
We talk about this dynamic in the post; very few actors really raise this out loud and dare go to the end of the causal chain, even when they do so privately.
I will still update the relevant sections and add more caveats, so thanks :)