I am a volunteer organizer with PauseAI and PauseAI US, a top forecaster, and many other things that are currently much less important.
The risk of human extinction from artificial intelligence is a near-term threat. Time is short, p(doom) is high, and anyone can take simple, practical actions right now to help prevent the worst outcomes.
I find myself somewhat disappointed that the scenario and the response don’t seem to have much to say. I actually thought the original scenario was meant as a thinly-veiled scathing critique of naive utilitarianism. For any non-psychopath who is actually in the scenario, saving the drowning girl doesn’t require any moral argumentation, and I wouldn’t feel safe being in community with anyone who sincerely preferred the other ending.
I think when people first discover rationalism and effective altruism, they often descend into a valley of learning to “shut up and multiply” and casting their intuitions aside. It’s very dangerous to remain in that valley without gaining humility about the limits of those tools. To be one of the good guys, you do have to rejoin human society and continue to love people in normal ways, and that isn’t only about one’s own personal warm fuzzies.
The neutrality of your follow-up reveals the original scenario as something like a scaffold upon which to hang one’s perspective and observe other perspectives being hung. The original example leaves me with a lingering itch, though, because the alternate perspective is not something that anyone should be encouraged to entertain.
I thought another one: 4. Beware of simplifying assumptions and the Either/Or fallacy. You do not live in a moral philosophy dilemma. Reality has high dimensionality and the trade-offs are not always obvious and clean.
Second-order effects matter. Doing X is not only doing X. Becoming the type of person who does things in the reference class of X is one of the effects of your actions. Your reputation can likewise be affected. Your behavior can be copied by others, sometimes lossily. If it would be downright catastrophic for a chunk of society to do things vaguely shaped like X on a regular basis, you probably shouldn’t do X.
If a utilitarian calculus concludes that you should do something that strongly violates societal norms of morality (a la virtue effects and deontology), your math is probably bad. Be humble. (You probably didn’t invent new physics while doing your math homework, you probably didn’t discover that “letting kids drown is sometimes good actually.”)
Be suspicious of unintuitive moral conclusions that violate norms in your favor.
I’m curious about this as well, since as far as I’m aware, flights in the US tend to be more expensive than flights in, say, Europe.
I’m very happy with most of these choices. I’m confused about “locally valid” and “locally invalid” being removed, though. It would feel weird to me to use “weak argument” and “strong argument” in their stead, since weak and strong connote doing different amounts of work, while valid and invalid denote logical applicability.
Thank you for your high-quality engagement on this and for including the clear statement!
I think my most substantial disagreement with you on the difficulty of a shutdown is related to longtermism. Most normal people would not take a 5% risk of destroying the world in order to greatly improve their lives and the lives of their children. That isn’t because they are longtermist, but primarily because they are simply horrified by the concept of destroying the world.
It is in fact almost entirely utilitarians who are in favor of taking that risk, because they are able to justify it to themselves after doing some simplified calculation. Ordinary people, rational or irrational, who just want good things for themselves and their kids usually don’t want to risk their own lives, certainly don’t want to risk their kids’ lives, and it wouldn’t cross their mind to risk other people’s kids’ lives, when put in stark terms.
“Human civilization should not be made to collapse in the next few decades” and “humanity should survive for a good long while” are longtermist positions, but they are also what >90% of people in every nation on earth already believe.
As an anecdote, a few months ago I met a former MIRI researcher while handing out flyers for PauseAI. We had a great conversation, and they were very concerned about x-risk.
When I asked if they would be willing to sign PauseAI US’s petition, they declined, stating that they don’t think a shutdown is feasible. I was very confused by this, because for those who are concerned about x-risk, I do not see a strong relationship between the feasibility of a shutdown and whether it is a good idea to advocate for one.
To expand on my point about asymmetry: If the shutdown plan fails, then we are undertaking one of the other plans. If the other plans fail, we die (with unacceptably high probability). Accidentally getting a shutdown when you meant to proceed with caution is a win condition, at least temporarily, and it allows for many other plans to be improved and enacted. Accidentally proceeding with caution when you meant to get a shutdown is walking on thin an ice. The distance from shutdown to death is larger than the distance from proceeding with caution to death.
To address another viewpoint: If you are concerned about x-risk and you believe that all effort going toward advocating for a shutdown is wasted, and that the world would be better off if no one talked about a shutdown, I think you’re simply confused about normal social and political dynamics.
For those who believe that a global shutdown of AGI R&D is next to impossible, or much more difficult to succeed at than a different plan:
Something I think is very important, and which I would be very grateful for, is if you can consistently signal that you would prefer a shutdown if you thought it was feasible.
There are many people who are making an argument against a shutdown, who would genuinely prefer AGI/ASI to be created in the near future and who want to prevent it from being shut down. If that does not describe you, please make it clear in your various communications, especially external communications, that shutting it all down would be best, and you are merely pessimistic that that can be done.
The situation is not symmetrical. A citizen or policymaker hearing “shut it down” who would otherwise want AI to proceed with caution moves in the direction of more caution. A citizen or policymaker hearing “proceed with caution” who would otherwise want AI to be shut down moves in the direction of less caution. Nonetheless, many people who advocate for a shutdown do say that they would like for superintelligent AI to eventually be created, and they simply see other plans as woefully unlikely to succeed.
I think it’s important to remember as well that not all of us actually are on the same team. Not everyone is even on team humanity.
For instance, Nick Bostrom, I was disturbed to learn, does not want to slow AGI development, and is perfectly willing to gamble away other people’s lives without their consent. That is difficult to top, as it concerns things that could place us on different teams.
Insofar as we the concerned actually are on the same team (humanity should survive and we shouldn’t endanger its existence beyond what most people could bear if they were informed), then we should indeed act like we are here primarily to save the world, and not primarily to move words around and score internet points.
Are most of the persnickety internal-disagreers actually signalling that they intend to promote the book, or at least not downplay its thesis? I don’t think rationalists at large have a great track record of engaging the outside world on a unified front, or in leaving nuance aside when the nuance would stand in the way of the important parts of the communication. In other words, I don’t think the two types of content are on different platforms. I think it’s usually the same content on both.
In general, I’ve noticed that a lot of people think “scout mindset” means never having to pick up a (metaphorical) rifle. That’s a good way to have a precise model of how you’re going to die, without having any hand in preventing it. The most useful people in the world right now are scouts who are willing to act like soldiers from time to time.
You’re only saying that because there is an extraordinarily consistent history of this exact thing happening over and over again! That empirical evidence bears no weight in comparison to my personal mental models. You clearly don’t understand how virtuous and clever I am. Real AI lab safety work and inside political maneuvering has never been tried!
Thank you for compiling these risk thresholds! This is going in my list of handy bookmarks.
Communication is about other people. It is the thoughts that we put into other’s heads, not the precise content of the propositions themselves. The graphs and other symbols we choose to use should put more likely-correct and contextually relevant thoughts into other people’s heads.
If I am communicating with someone because I believe their lack of information gives them an insufficiently aggressive model of past (and therefore probably future) AI capabilities progress, then the only reason I can think to use the sigmoid graph is to head off this specific criticism so that they are not distracted by the actual content of the communication. If I did make this choice, I would also want to put a cut in the y axis and cheekily label the upper value with something to indicate that it may be far beyond what they are currently entertaining to be possible.
I often do make this choice when verbally communicating. e.g. “Of course, it has to level off at some point, but that could very well be several orders of magnitude above human capability.” If I have exactly one image and no chance to elaborate, for most audiences, a sigmoid probably does a poorer job of putting correct and relevant thoughts into their heads than an exponential.
So even if we assume that AGI would arrive soon unless stopped (e.g. in 5 years) and would result in immediate death if unaligned (which is very far from a given), then it seems like your life expectancy would be vastly longer if AGI developed even if the chance of alignment were quite small.
This naive expected value calculation completely leaves out what it actually means for humanity to come to an end: if you ever reach zero, you cannot keep playing the game. As I said, I would not take this chance even if the odds were 99 to 1 in favor of it going well. It would be deeply unethical to create AGI under that level of uncertainty, especially since the uncertainty may be reduced given time, and our current situation is almost certainly not that favorable.
I am not so egoistic as to value my own life (and even the lives of my loved ones) highly enough to make that choice on everyone else’s behalf, and on behalf of the whole future of known sentient life. But I also don’t personally have any specific wishes to live a very long life myself. I appreciate my life for what it is, and I don’t see any great need to improve it to a magical degree or live for vastly longer. There are people who individually have such terrible lives that it is rational for them to take large risks onto themselves to improve their circumstances, and there are others who simply have a very high appetite for risk. Those situations do not apply to most people.
We have been monkeys in shoes for a very long time. We have lived and suffered and rejoiced and died for eons. It would not be a crime against being for things to keep happening roughly the way they always have, with all of the beauty and horror we have always known. What would be a crime against being is to risk a roughly immediate, permanent end to everything of value, for utopian ideals that are shared by almost none of the victims. Humanity has repeatedly warned about this in our stories about supervillains and ideologue despots alike.
Under our shared reality, there is probably no justification for your view that I would ever accept. In that sense, it is not important to me what your justification is. On the other hand, I do have a model of people who hold your view, which may not resemble you in particular: I view the willingness to gamble away all of value itself as an expression of ingratitude for the value that we do have, and I view the willingness to do this on everyone else’s behalf as a complete disregard for the inviolable consent of others.
I think your life expectancy and that of your loved ones (at least from a mundane perspective) is longer if AGI is developed than if it isn’t.
You must have extreme confidence about this, or else your attitude about AGI would be grossly cavalier. Were it put to me, I would never take a bet at 99 to 1 that humanity will survive and flourish beyond my wildest dreams, vs. quickly come to a permanent end. That is a terrible deal. I genuinely love humanity, the monkeys that we are, and it is not my place to play games with other people’s lives. I have not asked for a longer life, and I especially would not do so in exchange for even a small risk of immediate death. Most importantly, if I never asked everyone else what risks they are willing to shoulder, then I shouldn’t even consider rolling dice on their behalf.
Fwiw, I think it’s more likely that AI will be aligned than that it will be shut down.
I am aware that you think this, and I struggle to understand why. There are tractable frameworks for global governance solutions that stand a good chance of being able to prevent the emergence of AGI in the near term, which leaves time for more work to be done on more robust governance as well as on alignment. There are no such tractable frameworks for AGI alignment. There is not even a convincing proof that AGI alignment is solvable in principle. Granting that it is solvable, why hasn’t it already been solved? How can you have such extreme confidence that it will be solved within the next few years, when we are by many measures no closer to a solution than we were two decades ago?
you seem to be misattributing motive to me here
I do not mean to attribute motive. Only to point out the difference between what it appears you are doing, and what it appears you think you are doing. I will eat crow if you can point to where you have said that if AGI development can be halted in principle (until it is shown to be safe), then it should be. That AGI should not be built if it cannot be built safely is a minimum necessary statement for sanity on this issue, which requires only that you imagine you could be incorrect about the ease of very soon solving a problem that no one knows how to begin to solve.
I see it more like a flickering candle straining to create a small patch of visibility in an otherwise rather dark environment. Strong calls for unanimity and falling into line with a political campaign message is a wind that might snuff it out.
I can empathize with the sentiment, but this is an outdated view. The public overwhelmingly want AI regulation, want to slow AI down, want AI companies to have to prove their models are safe, and want an international treaty. Salience is low, but rising. People take action when they understand the risks to their families. Tens of thousands of people have contacted their political representatives to demand regulation of AI, over ten thousand of which have done so through ControlAI’s tool alone. Speaking of ControlAI, they have the support of over 50 UK lawmakers, and are making strides in their ongoing US campaign. In a much shorter campaign, PauseAI UK secured the support of 60 parliamentarians in calling for Google Deepmind to honor its existing commitments. The proposed 10-year moratorium on US states being allowed to regulate AI was defeated due at least in part to hundreds of phone calls made to congressional staffers by activist groups. US Congressman Raja Krishnamoorthi, the ranking member of the Select Committee on the CCP, recently had this to say:
Whether it’s American AI or Chinese AI, it should not be released until we know it’s safe. … This is just common sense.
These beginnings could never have happened through quiet dealings and gently laid plans. They happened because people were honest and loud. The governance problem (for genuine, toothed governance) has been very responsive to additional effort, in a way that the alignment problem never has. An ounce of genuine outrage has been significantly more productive than a barrel of stratagems and backroom dealings.
The light of humanity’s resistance to extinction is not a flickering candle. It is a bonfire. It doesn’t need to be shielded now, if indeed it ever did. It needs the oxygen of a rushing wind.
Every known plan for a post-AGI world is one which I do not expect my loved ones to survive.
I am grateful that you have spread awareness of the risk of human extinction from AI. I am genuinely saddened that you seem to be working to bring it about.
I stand with the majority of humanity, who do not want superintelligence to be created, and who do not consent to the risks. As the science stands, today, and for the foreseeable future, it is a fact that the only safe AGI is no AGI. If we here who know the stakes are not united in our call to shut down frontier AI development and preserve our very lives — a thing that unlike alignment we actually know is possible to achieve — then what was the rationalist project ever about? Math games? Playing inside politics with money? Risking everyone else’s everything forever because it’s fun to watch lines go up today? Being the limbs of Moloch? Feigning helplessness and pretending we don’t have the ability to pull our own children out of the fire that we lit with our own match?
needfully combative
I think there is also a local sense in which morals are just aesthetics. The long-term consequences of moral choices mean that evolution plays a big part in determining morality, but divorced from the constraints of evolution and any sense of long-term planning, by what can we objectively compare moral systems other than their popularity? Orthogonality and all that. Are LLMs just modeling that accurately?
I think it is bad form to use the term “psychosis” to essentially mean “having or acting on beliefs that are based on bad evidence that was positively reinforced.” A psychotic episode isn’t characterized by being incorrect about some part of reality in a relatively normal way; it is characterized by a break from reality. Dating a chatbot isn’t psychosis, and neither necessarily is thinking you’ve somehow awoken it into a becoming a conscious entity. Even reporting religious experiences with chatbots is not necessarily indicative of psychosis.
What is psychosis is, well, roughly what the medical community says it is. It is deeply terrifying and horrifying from the inside, not just because of external consequences. Epistemic status: I have an acquaintance who has experienced manic episodes with psychotic symptoms, without the commonly associated amnesia. They reported not wishing that experience on their worst enemies.
There are reported cases of GPT-4o reinforcing an individual’s incorrect beliefs or instantiating new incorrect beliefs, leading to bad outcomes. As a separate phenomenon, there are reported cases of GPT-4o reinforcing or instantiating delusions in individuals vulnerable to psychosis, ultimately leading to a psychotic episode. This includes cases where the affected individual did not previously know they were vulnerable to psychosis, and their communication with GPT-4o was the first inciting incident.
My suspicion is that AI psychosis affects only those individuals who are already vulnerable to psychosis. I also suspect that some fraction of those people could have gone their entire lives without ever experiencing a psychotic episode were it not for GPT-4o, which is (by complete accident) a frighteningly effective tool for triggering psychosis.
This is the stance of the PauseAI movement. “Pause” means stop AGI development immediately and resume if and only if we have high confidence that it is safe to do so, and there is democratic will to do so.
Something is making decisions, is it not? And that thing that makes the decisions is part of what you would normally describe as “you.” Everything still adds up to normality.
It can can be detrimental, though, to communicate certain subsets of true things without additional context, or in a way that is likely to be misinterpreted by the audience. Communicating truth (or at least not lying) is more about the content that actually ends up in people’s heads than it is about the content of the communication itself.