clickyquack
My mother (who, for reference, has otherwise only expressed AI concerns regarding how it affects education) just asked me about the Coxon news stories unprompted. And yes, I’ve tried having the conversation with her about it before. The difference here seems to be that this has been especially effective for highlighting how near-term the risks are and how much we don’t have things under control, compared to e.g. the Statement on AI Extinction Risk. Per Molly Kinder:
Of course, I have heard about “alignment” and “AI safety” and “x risk” concerns for years. I took these concerns at face value, especially when I heard them directly from folks at the labs working closest with the technology. And I was grateful so many smart people are working on them. And that’s about it. (My work focuses on AI’s impact on jobs.)
But something has shifted, dramatically, for me in the last few weeks. Between the Hugging Face incident, the damning independent @METR_Evals reports that followed, and the alarming chorus of calls (pleas? shouts? SOS signals?) from inside the labs that we are careening toward potential catastrophe, all of this has made me feel, viscerally, how truly dangerous this moment is.A policy window is opening. Keep reaching out to politicians, journalists, and influencers, right now more than ever. If not with you, arrange for them to speak with the most credentialed experts you can get. And this seems to be evidence strongly in favor of the effectiveness of, dear God, resigning in protest, given you prepare a good media strategy beforehand. I think the Snowden-like framing of “insider whistleblower quits to warn the world” has been especially compelling in this case (as opposed to frontier lab CEOs or remaining employees making the same claims, given “if you really believed that, why would you keep working on it?” objections. For reference, compare the responses to Coxon’s tweet to those of Hubinger’s), as has explicitly naming and quantifying extinction risk, and highlighting that the labs do not have everything under control, especially given recent rogue agent incidents.
Some very half-baked brainstormed ideas for increasing the likelihood of getting to something like Plan A or S:
General public targeted messaging needs to directly funnel people towards taking action that actually increases likelihood of policy change. Right now it seems like a lot of it is just trying to get people to take AI x-risk seriously in the first place. This is good and should continue but it should also direct people to email or call their representatives or lobby them in person if they’re able, and to ask them to support specific bills (PauseAI’s work is the closest example that comes to mind). Politicians can generally be modeled as trying to get re-elected and if enough people do this it directly signals to them that there are enough people who care about this enough to go out of their way to contact them, i.e. there are a lot of people whose votes it could sway, so they’re more likely to do it. Concretely if we got more AI safety content creators to say in their videos something like “if you want to help with this, contact your representative and tell them to support these bills and why it matters to you. You can use this email template if you don’t have much time but it’s even more effective to call or do this in person (or whatever else)” (essentially the advice in this US lobbying guide). Also might be good to get them to use referral links to maybe measure the rate of conversion for views on the video to number of people who do that? (This could also make the impact more legible to grantmakers I would imagine)
Due to the concentrated benefits vs diffuse costs problem, the other main way of spurring political action aside from getting more voters to signal they care is to minimize the perceived cost by the companies lobbying against AI regulation. The Montreal Protocol wouldn’t have succeeded if not for the main company lobbying against it finding harmless alternatives to the substances that were depleting the ozone layer and being given several decades to switch. If we can shift the discourse from people saying “shut it all down” vs “erm no you Luddite this is just like every other time people were worried about new technology” to the common ground of “I know you’re skeptical but there’s enough risk here that it’s clear we should at least be setting ourselves up to be more prepared to respond to new risks as they emerge” and then point specifically to the HuggingFace incident and related reports that followed making it clear we need to mandate incident reporting and/or liability standards for frontier labs, I imagine that would be far more agreeable. Another related part of this is most people who dismiss a binding international treaty do so on the grounds that there’s no point, because everyone would just defect or that it would “strangle innovation”, except I never see anyone challenging them with the progress and tractability of verification for international AI governance as a field already is, and there are plenty of initial measures that wouldn’t be particularly costly to implement so there might be work to be done there. Then again a ton of them just use galaxy brain motivated reasoning and don’t engage in good faith, but I imagine it could shift the people who do engage in good faith.
In general I don’t think a lot of people realize how much minimizing friction to contribute and giving people specific asks that don’t take much of their time makes a difference, e.g. open letters that have >1k signatures from top experts get tons of attention despite taking like 1 minute of each participant’s time. But for example, the pacing the frontier letter essentially just called on the White House to work towards an international treaty to pace the frontier, I suspect it would have been better to specifically name e.g. some of the early asks in AI 2040 that don’t cost much but set them up better to respond to new risks as they emerge and are harder to dismiss as “strangling innovation”.
The 2028 US presidential election will be critical for international AI governance. When candidates are campaigning I suspect it may be high leverage to get a lot of AI safety people to show up and ask during e.g. Q&A events if they will address AI risks (even if only in low cost incremental ways that are hard to dismiss as strangling innovation/the seeking common ground strategy) to signal that a lot of people care, and make them more inclined to support it insofar as it gets them additional swing voters. During the presidential debates if we can get the reporters to ask what they’ll do about emerging AI risks that would probably help too.
Among Congress people who care about AI risk, I’ve seen a lot of them specifically link news articles in tweets to justify why their bill is needed. Maybe news coverage is unusually high leverage and there’s work to be done in finding strategies for getting more of it, especially that focus on specific proposed bills/interventions/advocacy groups/etc.? For example, I think the AI red lines campaign was one of the best examples of this.
A bit pessimistic about this one given I don’t think I could have done much better than this post which only got ~300 upvotes, but if we can get more AI safety people to want to pivot/contribute to political will work in the first place that would be great, the case for it being the biggest bottleneck to making the future of AI go well by such a large margin that it’s worth trading off whatever personal fit people have in other fields feels strong to me. Maybe we could channel them to the parts that are still more technical feeling like working on verificiation, because to my understanding that’s one of the main technical governance bottlenecks right now in the sense that it makes it harder to deny the infeasibility of a treaty. There’s probably a lot of low hanging fruit in at least convincing more people to fund this type of work, it kind of baffles me that there isn’t more funding for it? It also might help to explicitly frame the projects as “increasing the likelihood of getting to Plan A or S from AI 2040”, I like to think about it in this way as it serves as a baseline for big picture strategy to try to do better than, and most AI safety people already seem convinced that increasing the likelihood of something like that is one of if not the most important thing to focus on for actually making the long-term future of AI go well (although few of them seem to actually update their behavior based on that so idk).
OpenAI and Anthropic like to signal that they’re interested in working towards AI safety governance, and as this post points out, the top labs have unusually high leverage in influencing the White House’s decisions. It’s unclear what they’re actually concretely doing for this. There’s probably more work to be done in holding them accountable to actually concretely taking initiative towards supporting specific interventions, or getting employees there to use their collective bargaining power to demand them to? idk. Again, helps to have prepared specific asks to minimize friction for what they have to do. A similar example that comes to mind to perhaps model after is what Alex Turner tried to do to get GDM to not service ICE. This particular example was a failure story so maybe this isn’t a promising direction, but his postmortem on it literally became the most upvoted LW post of all time within a few weeks, so who knows, maybe there’s more to be done here.
While writing this I realized there’s a gap in my knowledge for what can be done to lobby the U.S. executive branch in the first place, so I should probably figure that out and raise awareness of it for other people. Per this post, “In several international forums over the past 18 months, US positioning has been the ultimate bottleneck. My main answer to this is that it’s a priority to do insider work at the White House for those who can access it.”
I suspect there can be better funnels for getting AI safety people to pivot to govt. facing work with minimal friction. Insider policy work is seemingly one of the highest leverage ways an individual can influence the decisions made, but if I were to try to get involved in that right now I wouldn’t know where to start other than just trying to cold email other people already in US AI policy with feedback that’s insightful enough for them to their policy interests for them to see as worth responding to. Many other existing pipelines I know of seem to be built for people with strong policy backgrounds already, i.e. people with ~equivalent demonstrated expertise of a policy Master’s degree.
I’m largely new to working on this and haven’t investigated these too deeply yet so there are probably some obvious things I’m missing here. Thoughts?
I’m currently an undergraduate student and have tried to spark discussions related to long-term AI risks in e.g. computer science Discord servers for my university, I’ve gotten much of the same results. For example, I shared AI 2040 without comment and got responses like:
”Glad people are trying to think about the future of generative AI, and they have clearly put a lot of effort into this, but I’m always skeptical of these long term plans. They are also making the assumption that generative AI will exponentially scale, which—while possible, is just pure speculation and blustering. I mean seriously, “Wildy Superintelligent—Qualitatively beyond human science and comprehension”? As the technology currently stands, they aren’t intelligent in any way, they are prediction machines. And like with anyone talking about GenAI, these people are likely just trying to sell it. Most of the people at the top have a vested interest in GenAI and are trying to sell it as a tool that will outperform all of humanity and lead us to a prosperous utopia if controlled correctly”
I offered the obvious pushbacks and we went back and forth but ultimately only got to the common ground of the regulations that are clearly warranted now even given the more immediate risks (i.e. treating it like other high-risk dual-use technologies with negative externalities). In general I imagine this will be the most effective strategy for making actual incremental political progress. You can lead the horse to the reasoning behind rapid progress in AI capabilities forecasts but you can’t make it drink.
On using crises to shift political will for AI
In trying to investigate what could shift political will for AI governance as someone without much of a policy background, I decided to start with investigating the much less controversial proposal of mandating nucleic acid screening. Generally, policy change is much more likely to pass when the upside is obvious and it isn’t costly to comply. Mandating nucleic acid screening fits this abundantly well: make sure that if someone has enough expertise to design a dangerous protein, or uses an AI system to do so, it would get caught and stopped if they tried to pay a lab to produce and ship it to them. Automated screening technology that would catch the vast majority of near-term cases is already here and isn’t costly to implement. And yet, we still aren’t even doing this, which is a bad sign for any costlier AI regulations motivated by less legible risks. Notably, unlike AI slowdown proposals, there’s little to no argument to be made that anything short of a verifiable international treaty would cede our innovation advantage to China. Why is this stalling?
The main reason seems to be that passing legislation takes a lot of sustained effort, and there’s no crisis or warning-shot to make it feel urgent enough for Congress to prioritize. The relevant bill here is S.3741, introduced January 29, 2026, which would solve much of the problem, at least on a national scale. This post by Sophie Kim goes into more detail, and provides additional commentary and proposed amendments. It’s come after decades of advocacy, most of the nucleic acid synthesis industry already voluntarily screening orders[1], and being recommended in the Executive Office of the President’s July 2025 AI Action Plan for institutions receiving federal funding. It’s also worth noting that the open letter that called on Congress to pass this, signed by dozens of leading experts in several relevant fields (which helped to indicate the cost on innovation would be minimal), was published June 3, 2026, and that many members just weren’t aware that open-source AI tools can already design new dangerous proteins prior to this. Per Fortune:
While the bill slowly moves its way through Congress, Josh Wentzel, a senior fellow at the Foundation for American Innovation, told Fortune that the letter was a good opportunity to show lawmakers that the AI industry and companies who sell synthetic DNA and RNA were equally concerned about the issue.
“This is bipartisan, concrete, achievable, and noncontroversial,” Wentzel said, adding he hopes now that Congress sees these parties are aligned, it can move forward with passing the Biosecurity Modernization and Innovation Act. “It’s a goal among many national security experts and, crucially, something the nucleic acid synthesis industry itself has called for.”
So, the open letter does seem to be moving the bill faster than it otherwise would have. But in spite of all of this, and strong media coverage of the letter, and advocacy from the think tanks that co-organized the letter, the bill is still waiting in committee without undergoing any markups two months later. The bill is practically certain to pass eventually, but short of some event that would make it a higher priority, it’s likely going to take several more months. For reference, the much more mainstream Epstein Files Transparency Act took about 5 months to pass after being introduced. Congress is really that slow. I knew they had a reputation for it, but it’s disheartening to see it for myself. This lends credence to my prior theory that not much is going to concretely get done in AI governance until the effects are felt more, which I intend to investigate next.
I’m pretty much fully in agreement with these ideas being the most important things to work on for concretely reducing existential risks from AI right now. I suspect it could be very helpful (for me, if no one else) to have some generally recognized central website (presumably called something like political-will.ai) for painting a clearer picture of this field of work, such as:
Keeping track of the progress that has been made so far (e.g. stuff like the AIPN tracker already listed, perhaps a timeline of important events)
What still needs to happen for international coordination (the Red Lines FAQ gives some loose ideas for this under “What should the next steps be?”, and perhaps historical examples drawn from how the IAEA or Montreal Protocol came to be).
What’s stopping these from happening (e.g. the idea that we need to beat China and/or not strangle innovation, the idea that there isn’t enough evidence to act yet)
What has and hasn’t worked to make progress, with examples (e.g. Concrete demonstrations of AI cyber risks such as Project Glasswing prompting the Trump administration to place export controls onto Mythos when they had previously been extremely skeptical of regulating AI in any way. Perhaps transcripts of conversations that have changed decision-makers’ minds, to the extent that these are available).
The above information, but how it specifically applies to narrower policy asks, insofar as this is applicable (e.g. trying to answer the question of why DNA screening laws still haven’t passed despite it being far less costly and its risk being far more real-seeming than loss of control from AI, and what could probably help change this).
What other work could potentially help, and what the reader can do to contribute.
This information is mostly publicly available but very scattered, and I expect organizing it could help a lot with making pivoting to this work easier. I’m interested in trying to make this myself, and am open to help or feedback.
clickyquack’s Shortform
Given the U.S. government’s recent restrictions on Claude Mythos and GPT-5.6, I suspect one of the most tractable and effective strategies for further pushing the Overton window towards the urgency of implementing stronger AI regulation could be effectively communicating to policymakers that open-weights models, whose safeguards can be removed easily, are likely going to reach Mythos-level capabilities in under a year, which can meaningfully uplift cyberattacks and bioweapon development. Also, I suspect the main thing currently holding back the Overton window on interventions for AI x-risk in general to be a (perceived) lack of concrete evidence (see: The Milton Friedman Model of Policy Change, how we achieved the Montreal Protocol to mitigate climate change before the damage became irreversible despite strong industry pressure against it, etc.). Communicating loss of control risk would be ideal of course, but I’m currently unsure how this could be demonstrated safely.