I am a volunteer organizer with PauseAI and PauseAI US, a pro forecaster, and some other things that are currently much less important.
The risk of human extinction from artificial intelligence is a near-term threat. Time is short, p(doom) is high, and anyone can take simple, practical actions right now to help prevent the worst outcomes. Contact your political representatives.
You can pursue both goals, but you have chosen not to, and I find that alarming.
There are many, many concrete accusations in this post. It doesn’t particularly matter whether there is a “main” one. Vassar is a serial psychological and sexual abuser, and has caused multiple deaths. That is bad.
Plex opening up about his experiences with this abuser was a terrifying thing to do. The appropriate response in this circumstance is to respond with empathy. If you aren’t going to do that, it is better not to say anything at all. I understand that we are on LessWrong, and this is where people usually gain status by bantering about epistemic technicalities, but there is more than one genre of post, and this is one where that general rule does not apply.
If you want to know specific pieces of information, you can just ask about those directly without making it sound like you don’t believe him or any of Vassar’s other accusers.
I have generally been assuming that this is just a case of not knowing how to respond appropriately. But if you actually don’t believe them, just say that and out yourself. Shankar already made it clear that he doesn’t care about people and he would gladly sacrifice their lives to maintain the verbal games he gets to play with his friends. Do you agree with him about that? Do you think structures that differentially protect abusers are good?
The rationalist community makes itself vulnerable to bad actors when it lacks the ability to just “be normal about things” and strongly reject bad behavior.
It’s great to ask clarifying questions and weigh evidence appropriately. But the central premise here isn’t in doubt. If survivors speak out and the go-to response is to nitpick the way in which they communicated about the harm, that just tells me you aren’t safe people.
It sounds like he doesn’t have any interest in saving the world at all. He is playing a completely different game from the people who actually value humans.
Do you think it is more important to dissect the way that people communicate, or to protect vulnerable people from predators?
My takeaway from this post is that an exceptionally evil man has caused significant harm to the rationalist community, leading directly to deaths and damaging the community’s ability to do extremely important work. Is that not your primary takeaway?
On tractability:
I am in Washington DC today and will speak with the offices of both of my senators tomorrow, with 3 others also from Arizona, to educate them on the issue and demand that they call for a global agreement to ban the creation of superintelligent AI. 50+ others are doing the same thing for their state on the same day.
My representative Greg Stanton already (quietly for now) supports an ASI ban, primarily due to my personal efforts to educate him on the topic. My state-level representative Stacey Travers introduced an AI safety transparency bill this session at my request, which I helped shape. My state-level senator Mitzi Epstein became visibly concerned about AI risk when I met with her about it. I am three for three on positive impact, with a range of effect size.
I am not an AI safety researcher, I have no ML degree, and political lobbying is approximately the furthest possible thing from what I thought I could ever succeed at.
Tractability is a question of the will to act, not of whether we have a galaxy-brained map of the complex system of politics. Research about complex systems heavily relies on empiricism. Most big political asks that are successful are seen as impossible until they suddenly become the obvious consensus. If you want to know whether an AI moratorium is feasible, lobbying your elected leaders is the requisite field work.
If you model fighting with Anthropic as “infighting”, we have very different ideas of what people and organizations are acceptable to associate with. Anthropic is doing an extraordinarily evil thing by trying to create a superintelligence. To the degree that there are “sides” anywhere, they are approximately maximally not on my side.
Many thanks for this analogy! I plan to make good use of it in the future.
For the PhD student in your life
80′s aesthetics and fuzz lines. The future is in the past.
Good post. “Just stop” has been and continues to be an option, and is most likely our only option. Every answer to the question “How could we just stop?” boils down to “By actually wanting to.” Once enough of the right people want to stop, they won’t need me around to tell them how to do it. My duty, same as for most of us, is just to get enough of the right people to want to stop.
I have personally made some progress on that front with my own representative, which is more than I expected to be capable of. We should all expect more from ourselves, I think.
This is huge progress! Congratulations to the whole ControlAI team, as well as to humanity as a whole for starting to put up some fight!
If AI boosts productivity by a large factor in the roles that it can automate within a given sector, that would cause a significant increase in the demand for human labor in the remaining bottlenecks. Total employment in the sector could even increase, right up until AI passes a capability threshold with respect to a given bottleneck and completely wipes out those jobs all at once.
There are also many papers on various parts of governance solutions, such as Building The Pause Button: https://arxiv.org/abs/2506.20530
I am a superforecaster, and I endorse this message.
(Edit: To clarify, I am not a GJ Superforecaster®, just a professional forecaster with a comparable track record.)
Probably ’41haiku’ for me.
I am confused about how what you said is related to what you quoted. I don’t take any of this to be speaking to employees of the AGI companies, since none of them have any agency with respect to making AI go well.
AGI companies exist to create AGI, and their structures don’t allow for voluntary courses of action where they intentionally do that more slowly or not at all. Whether they have access to reasonable-sounding justifications for doing so is not a causal factor, and whether individual employees care about making a good human future is not a causal factor in whether that occurs.
For support in self-actualization situations, there is therapy. For support in emergency situations, there is the heavy internal screaming of an even deeper terror than that of public humiliation. (And there is also therapy.)
Most people in the PauseAI movement are not in the LW/rationalist circle. Some joined as rationalists or EAs, especially early on, but today most are normies (or were when they joined, anyway).
I personally found LessWrong and the forecasting community through AI Safety, not the reverse. I now organize for PauseAI Phoenix, a local group of PauseAI US. I have face-to-face conversations on a regular basis with people who believe ASI will be created within 2-20 years unless we prevent that from happening.
My personal experience:
These days, I am anxious when I take action and depressed when I don’t. But I’m not anxious about the end of the world anymore, just about, like, talking to people and preparing presentations and stuff. I’ve done a lot of processing at this point.
That’s how it’s going, but here’s how it started:
Over the course of about a year starting in March 2023, I became increasingly anxious about AI extinction risk. During that time, I donated to AI risk organizations, helped out on AI Safety field-building projects, and became an active digital volunteer in the PauseAI movement. But I didn’t fully leave my comfort zone, and I knew I was holding back. Finally, one evening, I broke down sobbing in the shower, finally really feeling that the world was going to end, and I wasn’t doing enough to stop it. I decided I would do whatever I concluded was the most useful thing to do, even if I really didn’t want to do it.
So I made the decision to start a local group (PauseAI Phoenix), all alone in my state, and commit to outreach and local organizing. I started with flyering, reasoning that even with my social anxiety, it would be hard to screw up handing someone a piece of paper. Even then, I was deeply terrified to engage the public on this issue, expecting to be mocked and humiliated. Instead, almost everyone was nice, and I actually had fun. After that, it became clear to me that fear would never be able to prevent me from doing something that I know I should do. (Today, that local group is growing and thriving and holding regular events.)
I was also able to speak with my state-level representative, state-level senator, and federal representative about AI risk, and they all became more concerned about the issue. After a single meeting with me, my Arizona representative Stacey Travers drafted an AI safety transparency bill, which she introduced this session. I repeatedly engaged the office of my federal representative Greg Stanton, and 1-on-1 at a recent town hall, he told me “if no one can make AGI safe, then it doesn’t matter who builds it,” and he said he was interested in supporting a global AI treaty. (At a previous town hall, he had said that we have to beat China.) These incremental improvements to the outlook of our situation occurred primarily because of my actions, despite my inadequacies. The problem of political will is surprisingly amenable to sheer effort.
I sound very optimistic here, but from my perspective, I am playing to my outs. I personally believe that it is more likely than not that we will all die soon. But while there is action to be taken to improve our odds, I will continue to take that action. We can turn the odds in our favor, difficult though it may be. No matter the odds, when failure would be total, giving up is always more foolish.
The mechanism isn’t [public disapproval] → [AI moratorium]. That sounds hopeless to me, too. What I’m actually relatively optimistic about is [public awareness] → [educating policymakers] + [signal to policymakers that they are allowed to do something about it] → [common knowledge of the desire to act] → [AI moratorium]. This feels very doable to me after getting back from speaking with my congressional offices in DC.
The key here is that most politicians don’t want to take these risks either. Really very few people are on board with the whole “create superintelligence we don’t know how to control or make care about us” thing. If enough politicians and policymakers learn that the “create superintelligence” thing necessarily brings a risk of human extinction, one would hope that they would act out of their own fear and desire for self-preservation. But because of how politics works (or doesn’t), especially in the US, it takes a lot to overcome the strong social fear of speaking out. They need to know that their constituents have their backs, as opposed to either supporting the status quo or having very different priorities.
So public disapproval isn’t a lever with which to move the whole world. It’s just a firm tap to send the balance in the direction of action. The whole mechanism is being built simultaneously in parallel, and the education and advocacy work gets easier the more the public is engaged with the issue. The bigger the tap, the smaller the gap.