I am a volunteer organizer with PauseAI and PauseAI US, a pro forecaster, and some other things that are currently much less important.
The risk of human extinction from artificial intelligence is a near-term threat. Time is short, p(doom) is high, and anyone can take simple, practical actions right now to help prevent the worst outcomes. Contact your political representatives.
If you model fighting with Anthropic as “infighting”, we have very different ideas of what people and organizations are acceptable to associate with. Anthropic is doing an extraordinarily evil thing by trying to create a superintelligence. To the degree that there are “sides” anywhere, they are approximately maximally not on my side.
Many thanks for this analogy! I plan to make good use of it in the future.
For the PhD student in your life
80′s aesthetics and fuzz lines. The future is in the past.
Good post. “Just stop” has been and continues to be an option, and is most likely our only option. Every answer to the question “How could we just stop?” boils down to “By actually wanting to.” Once enough of the right people want to stop, they won’t need me around to tell them how to do it. My duty, same as for most of us, is just to get enough of the right people to want to stop.
I have personally made some progress on that front with my own representative, which is more than I expected to be capable of. We should all expect more from ourselves, I think.
This is huge progress! Congratulations to the whole ControlAI team, as well as to humanity as a whole for starting to put up some fight!
If AI boosts productivity by a large factor in the roles that it can automate within a given sector, that would cause a significant increase in the demand for human labor in the remaining bottlenecks. Total employment in the sector could even increase, right up until AI passes a capability threshold with respect to a given bottleneck and completely wipes out those jobs all at once.
There are also many papers on various parts of governance solutions, such as Building The Pause Button: https://arxiv.org/abs/2506.20530
I am a superforecaster, and I endorse this message.
(Edit: To clarify, I am not a GJ Superforecaster®, just a professional forecaster with a comparable track record.)
Probably ’41haiku’ for me.
I am confused about how what you said is related to what you quoted. I don’t take any of this to be speaking to employees of the AGI companies, since none of them have any agency with respect to making AI go well.
AGI companies exist to create AGI, and their structures don’t allow for voluntary courses of action where they intentionally do that more slowly or not at all. Whether they have access to reasonable-sounding justifications for doing so is not a causal factor, and whether individual employees care about making a good human future is not a causal factor in whether that occurs.
For support in self-actualization situations, there is therapy. For support in emergency situations, there is the heavy internal screaming of an even deeper terror than that of public humiliation. (And there is also therapy.)
Most people in the PauseAI movement are not in the LW/rationalist circle. Some joined as rationalists or EAs, especially early on, but today most are normies (or were when they joined, anyway).
I personally found LessWrong and the forecasting community through AI Safety, not the reverse. I now organize for PauseAI Phoenix, a local group of PauseAI US. I have face-to-face conversations on a regular basis with people who believe ASI will be created within 2-20 years unless we prevent that from happening.
My personal experience:
These days, I am anxious when I take action and depressed when I don’t. But I’m not anxious about the end of the world anymore, just about, like, talking to people and preparing presentations and stuff. I’ve done a lot of processing at this point.
That’s how it’s going, but here’s how it started:
Over the course of about a year starting in March 2023, I became increasingly anxious about AI extinction risk. During that time, I donated to AI risk organizations, helped out on AI Safety field-building projects, and became an active digital volunteer in the PauseAI movement. But I didn’t fully leave my comfort zone, and I knew I was holding back. Finally, one evening, I broke down sobbing in the shower, finally really feeling that the world was going to end, and I wasn’t doing enough to stop it. I decided I would do whatever I concluded was the most useful thing to do, even if I really didn’t want to do it.
So I made the decision to start a local group (PauseAI Phoenix), all alone in my state, and commit to outreach and local organizing. I started with flyering, reasoning that even with my social anxiety, it would be hard to screw up handing someone a piece of paper. Even then, I was deeply terrified to engage the public on this issue, expecting to be mocked and humiliated. Instead, almost everyone was nice, and I actually had fun. After that, it became clear to me that fear would never be able to prevent me from doing something that I know I should do. (Today, that local group is growing and thriving and holding regular events.)
I was also able to speak with my state-level representative, state-level senator, and federal representative about AI risk, and they all became more concerned about the issue. After a single meeting with me, my Arizona representative Stacey Travers drafted an AI safety transparency bill, which she introduced this session. I repeatedly engaged the office of my federal representative Greg Stanton, and 1-on-1 at a recent town hall, he told me “if no one can make AGI safe, then it doesn’t matter who builds it,” and he said he was interested in supporting a global AI treaty. (At a previous town hall, he had said that we have to beat China.) These incremental improvements to the outlook of our situation occurred primarily because of my actions, despite my inadequacies. The problem of political will is surprisingly amenable to sheer effort.
I sound very optimistic here, but from my perspective, I am playing to my outs. I personally believe that it is more likely than not that we will all die soon. But while there is action to be taken to improve our odds, I will continue to take that action. We can turn the odds in our favor, difficult though it may be. No matter the odds, when failure would be total, giving up is always more foolish.
I am a longtime volunteer with PauseAI/PauseAI US. This is the advice I give to every volunteer:
Normal mental health advice still applies. We are limited beings, and we do not have the capacity to emotionally grasp something as immense as the end of the world (thank goodness!). That means normal mental health tools for anxiety and grief are still effective. Listen to mental health professionals and the people who love you.
Action is the antidote to anxiety. If you take action about something you feel anxious about, your anxiety will tend to decrease, because you are happening to it rather than the reverse. And as I often say, Hope goes by the name of Bravery. Don’t just look for hope out there. Don’t passively wait for the cavalry. Be the source of hope, and you will have hope.
If you are in an acute crisis or your action is unsustainable, make becoming well your number one priority. While it’s powerful to take personal responsibility for averting catastrophe, you aren’t of use to anyone if you are self-destructing. You are a valuable part of the humanity we are working to save, and the work will continue to be done while you recover.
Not everyone responds the same way emotionally, but this is a good place to start.
@Parv Mahajan I understood your post to roughly say: “Some models are being released without safety evals. This is bad and should change.” My claim is that a good way* to make this change is to encourage your political representatives to require it by law.
*By “a good way,” I really mean “almost certainly the only way.”
“A significant portion of LessWrong users (at least 20%) care more about the aesthetics of rationality than they do about humanity. It’s a rationality ouroboros. They use the power of rationality to pursue their values, and their values favor protecting aspects of their niche rationality practice over and above taking action that will prevent my and their loved ones from being killed. This forum is a room full of unarmed scouts getting gunned down by the soldiers working at the AI Labs. The only people on this platform who can credibly claim to care about humanity are the ones who actively oppose those soldiers, as soldiers.”
Don’t underestimate the legislative route. My state representative Stacey Travers introduced this AI safety transparency bill entirely because I had one meeting with her and asked her to write a bill for Arizona that was similar to the original text of New York’s RAISE Act.
Maybe this bill doesn’t make it into law, but every shot on goal matters, and in the US, every additional redundant state bill that passes helps. Meeting with your political representatives is extraordinarily overpowered and well worth the time and energy. Simply explain your concerns and exactly how they can address the issue.
Many weeks late, but I thought I should respond, since no one else did:
Your MP might not care if one person sends messages, but in aggregate, many people regularly calling and sending messages (e.g. once a week) will register for them over time, especially as salience increases. You can also write letters to newspaper editors, and donate to organizations that are fighting AI x-risk via local organizing and lobbying.
You can also join a local chapter of ControlAI or PauseAI, or join the Torchbearer Community, or join Microcommit.
There is a lot that everyone can do, be those actions big or small!
I personally use Microcommit, and I am a longtime volunteer in the PauseAI movement. I’ve learned that willingness to try and perseverance in trying are way more valuable than specialized expertise. (Unless you can find the people who have both!)
On tractability:
I am in Washington DC today and will speak with the offices of both of my senators tomorrow, with 3 others also from Arizona, to educate them on the issue and demand that they call for a global agreement to ban the creation of superintelligent AI. 50+ others are doing the same thing for their state on the same day.
My representative Greg Stanton already (quietly for now) supports an ASI ban, primarily due to my personal efforts to educate him on the topic. My state-level representative Stacey Travers introduced an AI safety transparency bill this session at my request, which I helped shape. My state-level senator Mitzi Epstein became visibly concerned about AI risk when I met with her about it. I am three for three on positive impact, with a range of effect size.
I am not an AI safety researcher, I have no ML degree, and political lobbying is approximately the furthest possible thing from what I thought I could ever succeed at.
Tractability is a question of the will to act, not of whether we have a galaxy-brained map of the complex system of politics. Research about complex systems heavily relies on empiricism. Most big political asks that are successful are seen as impossible until they suddenly become the obvious consensus. If you want to know whether an AI moratorium is feasible, lobbying your elected leaders is the requisite field work.