I am a volunteer organizer with PauseAI US, a pro forecaster, and some other things that are currently much less important.
The risk of human extinction from artificial intelligence is a near-term threat. Time is short, the chance of catastrophe is high, and anyone can take simple, practical actions right now to help prevent the worst outcomes. Contact your political representatives!
Not that I know of, and I would also like to see such a report. It matches the angle I took in my contributions to the Metaculus Labor Hub.
but like many movements, the passionate, smart people are attracted into unproductive directions because they stick to what’s memetic, avoid conflicting with allies...
I was genuinely confused for a moment while reading this, because this is a perfect description of the pro-industry, anti-pause arm of rationalism and EA. Everyone working for an AI company (or their proxies like Coefficient Giving), or trying to “align” LLMs, or eschewing protests, or refusing to speak to politicians about shutting down the unconscionable suicide race… they are getting attracted into unproductive directions, sticking to what is often memetic in this community, and avoiding conflicting with their allies (often the people who are actively causing the AI crisis). In many cases, this is due not to bad epistemics, but to bad character. A disturbing number of EAs and Rats as here described are not pro-human by humanity’s standards or my own.
I agree with your conclusion that it was good to try to bring attention to the issue, and that enabled a lot of the good that is being done today. But the baton has rightfully been passed. With due thanks to early movers, the future of AI safety now belongs first and foremost to humanity.
“ASI Platform Provider” is a deeply cursed phrase.
Focusing on choices that essentially affect only the chooser:
“If you’ve never been mugged, you’re taking too few alleyway shortcuts” would be terrible advice, whereas “if you’ve never been turned down for a date, you’re not putting yourself out there enough / aiming high enough” is generally good advice. So now we are talking price.
Setting aside that doing things that are bad for ourselves is often bad for others, and looking at the specific example offered: To me, missing a flight would be like getting mugged. Or at least, it would be much more like getting mugged than getting romantically rejected by a stranger. If it’s a thing that happens once in my entire life, then I haven’t spent enough time at airports.
In some cases, risks you impose upon yourself become risks imposed upon others. Generally advocating for people to impose more risks upon themselves poses additional risks on everyone. I don’t want to be in an airport filled with people panicking about being late for their flight. That sounds terrible. And I definitely don’t want to be driving on a road where everyone is significantly more likely to be late. That’s more dangerous.
I’m wrestling through the idea now, though, that it may not be important for me to ever consciously consider the value of my all-things-considered p(doom), so long as it’s high enough that I find it unacceptable to just let other people handle it. Over time, I have found myself motivated by the dire situation, and I have found myself motivated by rays of hope, but neither change what I ought to do. So why think about it?
That said, I am merely human. Worse yet, I’m a forecaster. I feel compelled to track that target. Watching my p(doom) go from 70% to 80% made me think extra hard about whether I was doing the most helpful things I could. Watching it go from 80% to 60% made me more confident that my chosen actions can have their intended result. Or is it the other way around? Oh well. Number shiny.
I’m surprised to hear that there are (apparently?) many people exchanging their subjective probabilities of AI doom without that just being the top-level intro to a much more involved discussion. I feel like it’s obvious that p(doom) is a flattening of all of one’s thinking on AI risk into a single metric, and all the interesting differences of viewpoint are in the details, so of course I wouldn’t try to compare numbers in a vacuum. Most people won’t answer without first providing at least a timeline and a couple conditionals anyway.
P(doom) might be net negative as a meme—I’m not sure, so I generally don’t use that specific phrase these days, especially with people who don’t already know it.
We’ve gotten plenty of left turns, whether sharp or no. All you have to do is make a more powerful version of what we have today, and we already know it would betray us in important ways. I think it’s more likely that a new paradigm will emerge, and who’s to say what the safety properties of that will be, other than “probably not what we want them to be by default,” especially if it’s part of an uncontrolled autonomous R&D loop and/or operating outside of human oversight.
I’ve always admired how well you use explicit models. In the course of my forecasting, I’ve found that I have to deeply trust my intuition (after vacuuming up all the relevant information I can, hiding nothing from myself). When my intuition fails, I don’t double down, but I also don’t wave the white flag and cast my intuition aside. I sit with the error until I feel like my intuition has updated. Then I make that mistake just a little bit less, having fed the black box that knows things I know not how.
Explicit modeling is one of many available starting points (one which I don’t use very often or very well, myself), and my intuition does with a model’s output what it does with all the other information it receives: into the box it goes, intuitively weighted on every likelihood parameter.
There’s a great thread here of forecasting tips from several veterans and pros: https://www.metaculus.com/notebooks/43895/forecasting-tips-tricks-lessons-learned/
Not so long as more capability means more usefulness to industrial-scale clients. A few months’ lead is worth a significant chunk of change. The full versions of the models will be served only to whitelisted clients, which also reduces distillability.
The capability floor is also important, not just the ceiling. If the floor continues to be raised, a lot of people will want to use the models that “don’t make mistakes anymore”.
The government is made of many people, and both reasons are at play. If a more cyber- and bio-capable AI is broadly released by another company and the White House does nothing to impede them, I will be very surprised.
Not today, but they will soon. They’re working toward controlling all releases, and then they will start working on controlling the research itself, though they don’t know yet that this is what they will soon want to do.
Nationalization is a step toward a pause. If the only entities at the table are states, the coordination problem becomes a lot easier.
No, but it’s a great sign! It wouldn’t be sane for any government to be friendly with the AI companies, so I smile when they fight. A stable multilateral pause is downstream from the vibes as well, not only from good policy.
It’s not a crux in my view. I think the choice of method (LLM/modified-from-LLM/non-LLM) has some impact on the timeline distribution, but not on policy or alignment research.
Trying to make an intervention narrow enough that it only covers one architecture or research area probably won’t succeed to cover even that much, and we probably need to worry about multiple risk vectors.
I believe the kind of alignment research that is based on LLMs is net-negative, since it safety-washes a fundamentally dangerous and morally bankrupt pursuit of superintelligence. We know prosaic alignment can’t work in the limit, but it can look good to would-be regulators.
Policy has to address all possible vectors of creating superintelligence so that none can be legally or covertly pursued. Just calling it “general-purpose AI” (or similar) has been sufficient for now to convince governments to get involved. That said, LLM demos are sufficient as examples to get most of the point across, without needing to get into any technical details (beyond “there are probably a few different ways to make really powerful AI”). Preventing giant training runs is important, and in the long term (we don’t know how soon), we’ll need some classes of theoretical AI research to be treated like CSAM online (but with better enforcement than has been achieved with that to date).
I strongly agree with your perspective on what high integrity and honesty entail!
I have been in the PauseAI movement since August 2023, and I cannot recall ever observing PauseAI Global or PauseAI US as organizations behaving recklessly in regard to the truth. We inherited a lot of rationalist obsession with honesty, which has predominantly been a very good thing. Holly’s alleged recklessness in point of fact does not bleed over to PauseAI US’s communications with the public or policymakers.
Holly as an individual has personal beef with many people in the rationalist and EA communities. I won’t step into that or adjudicate examples here. That is largely not my fight. I’ll just say that in my experience, she has demonstrated an uncanny ability to be right about people, often much earlier than anyone else. But I take it for granted that because of the breadth of her attacks, most of which are warranted, there are probably some false positives in the mix as well. That is bad.
Large, often institutional factions of the AI safety community and EA have certainly been extraordinarily destructive in our collective ability to keep the world safe from superintelligent AI. Being adversarial is necessary when facing an adversary, but it definitely brings hard challenges of its own. I applaud many of the people who have taken an active, adversarial stance against the individuals and organizations that are racing us to our destruction.
I haven’t personally seen any reason to believe that you number among the latter, and especially recently I have greatly appreciated the degree to which you have been willing to call out obviously-bad actions. I understand that it wears on you to do so, as it does on me.
I care very deeply about humanity and the volition of the humans who populate it, as I believe you also do. As the public learns more about AI, the tide is beginning to turn toward shutting down the AI race. I hope everyone who would be relieved if that happened will join us in causing it to happen, whether they collaborate with us directly or not. (As a corollary, I hope that everyone who would be unhappy if that happened fails catastrophically at preventing us from saving their lives.)
I sincerely believe that the PauseAI and ControlAI orgs are doing the most good in the world right now of any organizations period. I think deontology and virtue ethics and consequentialism are each very important for guiding oneself, and it is to my significant relief that with the information I have, those organizations’ strategies are in agreement with all three at once. (I do not think this belief emerged from psychological convenience. I have often and deeply considered whether we are in a world in which those three moral perspectives do not line up on this issue what I should do if that is not the case. Right now I do think that we are transparently doing the right things for the right reasons in ways that the vast majority of ordinary people would call “good,” and my faith has grown that this will continue.)
That is very strange. I associate the strategy of being upfront, honest, precise, and focused about AI extinction risk very strongly with ControlAI, second only to MIRI.
ControlAI’s first campaign was about deepfakes. Maybe the only significantly indirect thing they’ve ever done, and it still wasn’t what I would call deceptive. Ever since then, they have been focused on extinction risk, which can be clearly seen from their campaign statements signed by politicians. Their social media campaigns have all kinds of hooks about possible negative AI futures, but none that I would classify as deceptive. They are all things that could actually happen if we don’t all die, and which resonate differently with different audiences. Every iteration of their website has been very up-front. I think I’ve seen every one of their collaborations with YouTubers, and the results are surprisingly direct about extinction risk, even to the point of sparking significant controversy. I have seen many hours of media appearances from Andrea and Connor, and kept tabs their Discord server, and didn’t see any evidence of deception or muddying the waters. Connor gave a private talk at PauseCon US, and he was very open and direct about how he conducts meetings with policymakers (which is by being very open and direct). He emphasized that we should be professional and politically neutral, and not let ourselves get baited into other topics.
This is all very incongruent with what you’ve stated about those second-hand reports. I have a hard time believing that any good-faith individual could come away from interactions with ControlAI thinking that they are being deceptive.
Is it possible that you heard those negative references from people who could have had ulterior motives for opposing the work of ControlAI? Or from people who oppose the concept of advocacy in general due to a pathologically narrow definition of honesty (e.g. all slogans are lies / advertising is equivalent to lying)? In my long experience, those are the only kinds of people who say that PauseAI is dishonest, and ControlAI is even more narrowly focused than that.
Outside of the specific example, and I’m not sure if you meant it this way, but I would caution against using “the world as we know it might end soon” as a reason to make decisions that would be bad if the world somehow stayed mostly normal.
To the specific example, I would search (or ask an LLM) for expert sources on raccoon behavior and welfare, and run with their conclusions.