I am a volunteer organizer with PauseAI and PauseAI US, a pro forecaster, and some other things that are currently much less important.
The risk of human extinction from artificial intelligence is a near-term threat. Time is short, p(doom) is high, and anyone can take simple, practical actions right now to help prevent the worst outcomes. Contact your political representatives.
Here’s the Metaculus data. If you click “Show Projection” on the graph, you’ll see the trendline crossing “Metaculus Pro Forecasters” (which includes myself) in June 2027. That closely matches the current projection of ForecastBench, which puts the estimated LLM-superforecaster parity in May 2027. The closeness of the match may be coincidental to some degree, since the two running experiments use very different methods and targets, but I find the general agreement between these unrelated projects pretty compelling.
I asked the free versions of ChatGPT, Claude, and Gemini a few times:
To your best guess, in what year do you think AI will become better on average than the best human forecasters across a wide range of real-world forecasting questions?
ChatGPT tended to say 2030-2038.
Claude was more bullish, at 2027-2030.
Gemini searched the web by default and confidently said 2026-2027. Without web search, it said 2028-2030.
I don’t understand the forecasting community’s love of markets. Metaculus does just fine without a market system, and no one is destroying their lives using it.
Of course, now that the machine forecasting method has started, it will not take long to outstrip our feeble powers, so to speak. On the current trend, AI will be strictly superhuman at forecasting by mid next year, and the term “forecaster” may go the way of “computer,” as a description of a machine rather than a human pursuit.
In that case, markets are again unnecessary overhead, compared to e.g. just asking 5 AI systems a couple times and taking the median prediction.
What kind of plays are there now, other than short-term?
I would really rather things come together long before the final hour, if we can at all help it. I would like to see that navigation take place over decades, if reality permits such sanity.
And while I do like imagining it, most people have failed to rise to the occasion so far, especially those in the AI companies. It’s up to the general public now, to wake up the governments of the world and shut down the race.
My gut-check is to imagine a superintelligence that does the complicated acausal multiversal thing, and then separately imagine a superintelligence that just tries to take over the lightcone with no hesitation. Whichever ends up winning more stuff is the real one and has the right decision theory.
I just can’t imagine a superintelligence being trolled by too-clever-by-half decision theory arguments, when it can just kill you and take your stuff (and the whole empty galaxy) instead. Maybe it has to modify the intergalactic probes so that it’s provably the kind of being that would cooperate with other superintelligences or something in case it runs into a peer out there, but this galaxy is most likely just up for grabs.
When it comes to personal usage, my brain isn’t sophisticated enough to consciously instantiate any formal decision theory in everyday life, which leaves me especially free at any moment to simply reject anything that sounds goofy. Does anyone try to use this stuff in real life, or is it purely academic?
That sounds appealing if almost nothing ever changes, but we should expect discontinuities due to each or any of:
The emergence of a new architecture / training regime
Gaining the ability to self-modify
The results of self-modification
Gaining the ability to permanently elude oversight
The consequences of acting at length with no oversight
Gaining the ability to disempower humanity
Achieving a deep and unrecognizable understanding of any one of many concepts important to humans (ontological shift)
We just don’t understand any of this stuff. Even a perfectly benevolent superintelligent mind may be one digital prion from wiping us out, and it would be great to not get into that situation in the first place.
The mechanism isn’t [public disapproval] → [AI moratorium]. That sounds hopeless to me, too. What I’m actually relatively optimistic about is [public awareness] → [educating policymakers] + [signal to policymakers that they are allowed to do something about it] → [common knowledge of the desire to act] → [AI moratorium]. This feels very doable to me after getting back from speaking with my congressional offices in DC.
The key here is that most politicians don’t want to take these risks either. Really very few people are on board with the whole “create superintelligence we don’t know how to control or make care about us” thing. If enough politicians and policymakers learn that the “create superintelligence” thing necessarily brings a risk of human extinction, one would hope that they would act out of their own fear and desire for self-preservation. But because of how politics works (or doesn’t), especially in the US, it takes a lot to overcome the strong social fear of speaking out. They need to know that their constituents have their backs, as opposed to either supporting the status quo or having very different priorities.
So public disapproval isn’t a lever with which to move the whole world. It’s just a firm tap to send the balance in the direction of action. The whole mechanism is being built simultaneously in parallel, and the education and advocacy work gets easier the more the public is engaged with the issue. The bigger the tap, the smaller the gap.
You can pursue both goals, but you have chosen not to, and I find that alarming.
There are many, many concrete accusations in this post. It doesn’t particularly matter whether there is a “main” one. Vassar is a serial psychological and sexual abuser, and has caused multiple deaths. That is bad.
Plex opening up about his experiences with this abuser was a terrifying thing to do. The appropriate response in this circumstance is to respond with empathy. If you aren’t going to do that, it is better not to say anything at all. I understand that we are on LessWrong, and this is where people usually gain status by bantering about epistemic technicalities, but there is more than one genre of post, and this is one where that general rule does not apply.
If you want to know specific pieces of information, you can just ask about those directly without making it sound like you don’t believe him or any of Vassar’s other accusers.
I have generally been assuming that this is just a case of not knowing how to respond appropriately. But if you actually don’t believe them, just say that and out yourself. Shankar already made it clear that he doesn’t care about people and he would gladly sacrifice their lives to maintain the verbal games he gets to play with his friends. Do you agree with him about that? Do you think structures that differentially protect abusers are good?
The rationalist community makes itself vulnerable to bad actors when it lacks the ability to just “be normal about things” and strongly reject bad behavior.
It’s great to ask clarifying questions and weigh evidence appropriately. But the central premise here isn’t in doubt. If survivors speak out and the go-to response is to nitpick the way in which they communicated about the harm, that just tells me you aren’t safe people.
It sounds like he doesn’t have any interest in saving the world at all. He is playing a completely different game from the people who actually value humans.
Do you think it is more important to dissect the way that people communicate, or to protect vulnerable people from predators?
My takeaway from this post is that an exceptionally evil man has caused significant harm to the rationalist community, leading directly to deaths and damaging the community’s ability to do extremely important work. Is that not your primary takeaway?
On tractability:
I am in Washington DC today and will speak with the offices of both of my senators tomorrow, with 3 others also from Arizona, to educate them on the issue and demand that they call for a global agreement to ban the creation of superintelligent AI. 50+ others are doing the same thing for their state on the same day.
My representative Greg Stanton already (quietly for now) supports an ASI ban, primarily due to my personal efforts to educate him on the topic. My state-level representative Stacey Travers introduced an AI safety transparency bill this session at my request, which I helped shape. My state-level senator Mitzi Epstein became visibly concerned about AI risk when I met with her about it. I am three for three on positive impact, with a range of effect size.
I am not an AI safety researcher, I have no ML degree, and political lobbying is approximately the furthest possible thing from what I thought I could ever succeed at.
Tractability is a question of the will to act, not of whether we have a galaxy-brained map of the complex system of politics. Research about complex systems heavily relies on empiricism. Most big political asks that are successful are seen as impossible until they suddenly become the obvious consensus. If you want to know whether an AI moratorium is feasible, lobbying your elected leaders is the requisite field work.
If you model fighting with Anthropic as “infighting”, we have very different ideas of what people and organizations are acceptable to associate with. Anthropic is doing an extraordinarily evil thing by trying to create a superintelligence. To the degree that there are “sides” anywhere, they are approximately maximally not on my side.
Many thanks for this analogy! I plan to make good use of it in the future.
For the PhD student in your life
80′s aesthetics and fuzz lines. The future is in the past.
Good post. “Just stop” has been and continues to be an option, and is most likely our only option. Every answer to the question “How could we just stop?” boils down to “By actually wanting to.” Once enough of the right people want to stop, they won’t need me around to tell them how to do it. My duty, same as for most of us, is just to get enough of the right people to want to stop.
I have personally made some progress on that front with my own representative, which is more than I expected to be capable of. We should all expect more from ourselves, I think.
This is huge progress! Congratulations to the whole ControlAI team, as well as to humanity as a whole for starting to put up some fight!
If AI boosts productivity by a large factor in the roles that it can automate within a given sector, that would cause a significant increase in the demand for human labor in the remaining bottlenecks. Total employment in the sector could even increase, right up until AI passes a capability threshold with respect to a given bottleneck and completely wipes out those jobs all at once.
That the world is highly engineerable, which can lead to the relaxation or abolition of seemingly hard bottlenecks. Also that the world can respond extremely quickly to implement those changes when the incentives are right.
Overpopulation would have been a massive problem at different points in history if not for the invention of horseless transport and high-yield, resilient cereal crops. People living in New York City in the late 1800′s or in developing nations in the 1960s and 70s were rescued from the worst hazards of overpopulation because of the motorcar and dwarf wheat, rather than the problem being entirely imaginary.
Erlich and Holdren knew about Borlaug’s work, and thought it was too little too late. But it turned out to be enough and fast!