I am a volunteer organizer with PauseAI US, a pro forecaster, and some other things that are currently much less important.
The risk of human extinction from artificial intelligence is a near-term threat. Time is short, the chance of catastrophe is high, and anyone can take simple, practical actions right now to help prevent the worst outcomes. Contact your political representatives!
Focusing on choices that essentially affect only the chooser:
“If you’ve never been mugged, you’re taking too few alleyway shortcuts” would be terrible advice, whereas “if you’ve never been turned down for a date, you’re not putting yourself out there enough / aiming high enough” is generally good advice. So now we are talking price.
Setting aside that doing things that are bad for ourselves is often bad for others, and looking at the specific example offered: To me, missing a flight would be like getting mugged. Or at least, it would be much more like getting mugged than getting romantically rejected by a stranger. If it’s a thing that happens once in my entire life, then I haven’t spent enough time at airports.
In some cases, risks you impose upon yourself become risks imposed upon others. Generally advocating for people to impose more risks upon themselves poses additional risks on everyone. I don’t want to be in an airport filled with people panicking about being late for their flight. That sounds terrible. And I definitely don’t want to be driving on a road where everyone is significantly more likely to be late. That’s more dangerous.
I’m wrestling through the idea now, though, that it may not be important for me to ever consciously consider the value of my all-things-considered p(doom), so long as it’s high enough that I find it unacceptable to just let other people handle it. Over time, I have found myself motivated by the dire situation, and I have found myself motivated by rays of hope, but neither change what I ought to do. So why think about it?
That said, I am merely human. Worse yet, I’m a forecaster. I feel compelled to track that target. Watching my p(doom) go from 70% to 80% made me think extra hard about whether I was doing the most helpful things I could. Watching it go from 80% to 60% made me more confident that my chosen actions can have their intended result. Or is it the other way around? Oh well. Number shiny.
I’m surprised to hear that there are (apparently?) many people exchanging their subjective probabilities of AI doom without that just being the top-level intro to a much more involved discussion. I feel like it’s obvious that p(doom) is a flattening of all of one’s thinking on AI risk into a single metric, and all the interesting differences of viewpoint are in the details, so of course I wouldn’t try to compare numbers in a vacuum. Most people won’t answer without first providing at least a timeline and a couple conditionals anyway.
P(doom) might be net negative as a meme—I’m not sure, so I generally don’t use that specific phrase these days, especially with people who don’t already know it.
We’ve gotten plenty of left turns, whether sharp or no. All you have to do is make a more powerful version of what we have today, and we already know it would betray us in important ways. I think it’s more likely that a new paradigm will emerge, and who’s to say what the safety properties of that will be, other than “probably not what we want them to be by default,” especially if it’s part of an uncontrolled autonomous R&D loop and/or operating outside of human oversight.
I’ve always admired how well you use explicit models. In the course of my forecasting, I’ve found that I have to deeply trust my intuition (after vacuuming up all the relevant information I can, hiding nothing from myself). When my intuition fails, I don’t double down, but I also don’t wave the white flag and cast my intuition aside. I sit with the error until I feel like my intuition has updated. Then I make that mistake just a little bit less, having fed the black box that knows things I know not how.
Explicit modeling is one of many available starting points (one which I don’t use very often or very well, myself), and my intuition does with a model’s output what it does with all the other information it receives: into the box it goes, intuitively weighted on every likelihood parameter.
There’s a great thread here of forecasting tips from several veterans and pros: https://www.metaculus.com/notebooks/43895/forecasting-tips-tricks-lessons-learned/
Not so long as more capability means more usefulness to industrial-scale clients. A few months’ lead is worth a significant chunk of change. The full versions of the models will be served only to whitelisted clients, which also reduces distillability.
The capability floor is also important, not just the ceiling. If the floor continues to be raised, a lot of people will want to use the models that “don’t make mistakes anymore”.
The government is made of many people, and both reasons are at play. If a more cyber- and bio-capable AI is broadly released by another company and the White House does nothing to impede them, I will be very surprised.
Not today, but they will soon. They’re working toward controlling all releases, and then they will start working on controlling the research itself, though they don’t know yet that this is what they will soon want to do.
Nationalization is a step toward a pause. If the only entities at the table are states, the coordination problem becomes a lot easier.
No, but it’s a great sign! It wouldn’t be sane for any government to be friendly with the AI companies, so I smile when they fight. A stable multilateral pause is downstream from the vibes as well, not only from good policy.
It’s not a crux in my view. I think the choice of method (LLM/modified-from-LLM/non-LLM) has some impact on the timeline distribution, but not on policy or alignment research.
Trying to make an intervention narrow enough that it only covers one architecture or research area probably won’t succeed to cover even that much, and we probably need to worry about multiple risk vectors.
I believe the kind of alignment research that is based on LLMs is net-negative, since it safety-washes a fundamentally dangerous and morally bankrupt pursuit of superintelligence. We know prosaic alignment can’t work in the limit, but it can look good to would-be regulators.
Policy has to address all possible vectors of creating superintelligence so that none can be legally or covertly pursued. Just calling it “general-purpose AI” (or similar) has been sufficient for now to convince governments to get involved. That said, LLM demos are sufficient as examples to get most of the point across, without needing to get into any technical details (beyond “there are probably a few different ways to make really powerful AI”). Preventing giant training runs is important, and in the long term (we don’t know how soon), we’ll need some classes of theoretical AI research to be treated like CSAM online (but with better enforcement than has been achieved with that to date).
I strongly agree with your perspective on what high integrity and honesty entail!
I have been in the PauseAI movement since August 2023, and I cannot recall ever observing PauseAI Global or PauseAI US as organizations behaving recklessly in regard to the truth. We inherited a lot of rationalist obsession with honesty, which has predominantly been a very good thing. Holly’s alleged recklessness in point of fact does not bleed over to PauseAI US’s communications with the public or policymakers.
Holly as an individual has personal beef with many people in the rationalist and EA communities. I won’t step into that or adjudicate examples here. That is largely not my fight. I’ll just say that in my experience, she has demonstrated an uncanny ability to be right about people, often much earlier than anyone else. But I take it for granted that because of the breadth of her attacks, most of which are warranted, there are probably some false positives in the mix as well. That is bad.
Large, often institutional factions of the AI safety community and EA have certainly been extraordinarily destructive in our collective ability to keep the world safe from superintelligent AI. Being adversarial is necessary when facing an adversary, but it definitely brings hard challenges of its own. I applaud many of the people who have taken an active, adversarial stance against the individuals and organizations that are racing us to our destruction.
I haven’t personally seen any reason to believe that you number among the latter, and especially recently I have greatly appreciated the degree to which you have been willing to call out obviously-bad actions. I understand that it wears on you to do so, as it does on me.
I care very deeply about humanity and the volition of the humans who populate it, as I believe you also do. As the public learns more about AI, the tide is beginning to turn toward shutting down the AI race. I hope everyone who would be relieved if that happened will join us in causing it to happen, whether they collaborate with us directly or not. (As a corollary, I hope that everyone who would be unhappy if that happened fails catastrophically at preventing us from saving their lives.)
I sincerely believe that the PauseAI and ControlAI orgs are doing the most good in the world right now of any organizations period. I think deontology and virtue ethics and consequentialism are each very important for guiding oneself, and it is to my significant relief that with the information I have, those organizations’ strategies are in agreement with all three at once. (I do not think this belief emerged from psychological convenience. I have often and deeply considered whether we are in a world in which those three moral perspectives do not line up on this issue what I should do if that is not the case. Right now I do think that we are transparently doing the right things for the right reasons in ways that the vast majority of ordinary people would call “good,” and my faith has grown that this will continue.)
That is very strange. I associate the strategy of being upfront, honest, precise, and focused about AI extinction risk very strongly with ControlAI, second only to MIRI.
ControlAI’s first campaign was about deepfakes. Maybe the only significantly indirect thing they’ve ever done, and it still wasn’t what I would call deceptive. Ever since then, they have been focused on extinction risk, which can be clearly seen from their campaign statements signed by politicians. Their social media campaigns have all kinds of hooks about possible negative AI futures, but none that I would classify as deceptive. They are all things that could actually happen if we don’t all die, and which resonate differently with different audiences. Every iteration of their website has been very up-front. I think I’ve seen every one of their collaborations with YouTubers, and the results are surprisingly direct about extinction risk, even to the point of sparking significant controversy. I have seen many hours of media appearances from Andrea and Connor, and kept tabs their Discord server, and didn’t see any evidence of deception or muddying the waters. Connor gave a private talk at PauseCon US, and he was very open and direct about how he conducts meetings with policymakers (which is by being very open and direct). He emphasized that we should be professional and politically neutral, and not let ourselves get baited into other topics.
This is all very incongruent with what you’ve stated about those second-hand reports. I have a hard time believing that any good-faith individual could come away from interactions with ControlAI thinking that they are being deceptive.
Is it possible that you heard those negative references from people who could have had ulterior motives for opposing the work of ControlAI? Or from people who oppose the concept of advocacy in general due to a pathologically narrow definition of honesty (e.g. all slogans are lies / advertising is equivalent to lying)? In my long experience, those are the only kinds of people who say that PauseAI is dishonest, and ControlAI is even more narrowly focused than that.
We are much more in agreement than I expected! Thank you for clarifying. I agree with every point you made in this comment. (Excepting the comparison to continental philosophy, which is new to me and not something I think I can evaluate one way or the other.)
The shape of superintelligence that we are building isn’t necessarily the shape of superintelligence that will ultimately exist, though, right? That’s one of the hard barriers to see beyond: the ability for an AI to invent new AI paradigms.
It’s better than useless to study current systems if they give us insight into intelligence in general, and we might get useful information about the types of minds that are likely to exist. (I find work like the Natural Abstractions Hypothesis really interesting, and that benefits from empirical work). But that isn’t quite the same as studying things that are shaped like the superintelligence we will get.
I think what you perceive as an aversion to empiricism and rigorous justification is just an acknowledgment of the areas where empiricism and rigorous justification are currently impossible.
It is very common for AI safety researchers to fall for the trap of doing mostly useless (or counterproductive) empirical work rather than trying to solve important problems. We know that something abstract like agent foundations will be necessary, because it is strictly impossible to do empirical work on superintelligence if superintelligence doesn’t exist. (You probably can’t meaningfully run experiments on superintelligence once it does exist, either.) We need theories that are able to give us a clear line of sight beyond several thresholds that are known to be hard epistemic barriers for all current methods.
Put another way, almost all empirical AI safety work has the implicit epistemic status of: “it is unknown whether this is actually relevant to AI safety.”
Mostly-failing to solve problems that might actually help if solved is a lot better than successfully solving problems that are very unlikely to help. It is better to spend decades trying to invent a flashlight that lets you see through walls rather than to spend that time mapping everything under the street lamps. If we are foolish enough to crash on through the next barrier, the cracks in the sidewalk that we have been studying aren’t going to give us any information about whether there is a sinkhole ahead.
I agree that agent foundations and related abstract efforts need to be criticized, and they deserve good criticism. They appear to be the only efforts of the kind that can actually allow superintelligence and humans to coexist. I see your criticism as blurring the line between “I think they need to do this more carefully” and “I think they should do something else instead,” and it doesn’t grapple with the hard epistemic barriers between here and superintelligence.
(Of course, the most valuable efforts are the ones to prevent superintelligence from existing at all, at least until we can solve these problems, if they are even possible to solve. If it is truly the case that our only options are either building useless castles in the sky or playing in the mud, then we should stop pretending we are responsible adults and indefinitely shelve the whole enterprise.)
With a very short, messy pause, I expect humanity to be long gone before 2040. I am much more interested in the economic outlook of a decades-long or permanent moratorium, since that’s the only version of the future we should expect to observe.
“ASI Platform Provider” is a deeply cursed phrase.