I am a volunteer organizer with PauseAI US, a pro forecaster, and some other things that are currently much less important.
The risk of human extinction from artificial intelligence is a near-term threat. Time is short, the chance of catastrophe is high, and anyone can take simple, practical actions right now to help prevent the worst outcomes. Contact your political representatives!
Dario consistently centering China as the boogeyman is perhaps the most damaging thing any of the AI companies have ever done.
Stop doing “safety” work for the uncontrollable doomsday weapon companies. That is part of what gets everyone killed.
I think it’s important for everyone to know that there is no path toward superintelligence alignment.
Once you and your friends are legally barred from continuing to work on uncontrollable doomsday devices, I hope there will be time to either begin developing a realistic solution to alignment, or begin developing a realistic solution to prevent existential-risk-capable AI from being developed in the long term.
Setting aside the parts of math that have utility, that stance is fundamentally anti-human. Mathematics is full of intrinsically useless and arbitrary truths. Uncovering those truths is a human endeavor, as are all endeavors toward intrinsically useless and arbitrary things. Loving and pursuing such things is an important part of what it is to be human. The whole point is the experience of meaning. A machine that can spit out any intrinsically useless and arbitrary truth on command would destroy a source of meaning for many people, while in return providing contextless slop content. Maybe that ship has already sailed regardless, but you don’t have to be cruel about it to the people who are currently grieving their obsolescence.
They would deserve credit for completely shutting down all R&D on the “kill my family” machine. I’m ready and waiting to praise them if they ever did just that one thing.
I continue to insist that we have a threshold problem, not just a speed problem. We currently have no way to constrain what will happen when AI becomes strictly superhuman at creating better AI, or when it gains the option to permanently escape human oversight or disempower humanity. We have no guarantees of the degree of generalization from our intent at those thresholds. They are opaque walls to us.
The continuity argument fails at all of these steps, not just new architectures. But if AI does an increasing amount of the AI research, we’ll also have no way of preventing the emergence of new architectures!
The emergent behavior of agent swarms was a relatively transparent threshold, and even that showed that we are just blundering forward with no idea what we are doing.
I vibe with the sentiment, but I’m going to be annoying and disagree with that specific framing. There is always a non-zero probability of human extinction from all kinds of things, including nuclear war and bioweapons, as well as asteroid impacts or a sufficiently bad supervolcano. (Zero isn’t a real probability in Bayesian epistemology, but that’s another rabbit hole.)
That said, it’s perfectly sane to not accept even very low probabilities of extinction from AI. Some people draw the line at 1 in 20, others at 1 in a billion. I don’t know exactly where I fall, but I’m much more focused on giving other humans the chance to realize their preferences, rather than focusing on my own. If the world came together and said that 1 in a billion is too high, or even just that they’d rather not have superintelligence even if we could be sure it would go well, then I’d like them to be able to decide that. Creating a system of global informed consent (or something like it) seems like a hard problem, but slamming the breaks at least stands a chance of preserving our optionality for a while.
To the degree that we can provide quantitative guarantees of next-generation AI behavior, it should be fine to take things up another notch, modulo all the other ethical questions involved. Right now we don’t have a way of doing that. We can’t even properly measure the safety of existing systems, let alone a hypothetical next one.
ASI does necessarily escape every meaningful constraint, at least until we solve a series of extremely hard scientific problems that we are nowhere near solving. There are a few ways to notice the difficulty here. I’ll take a crack at it, but I’m no researcher.
Control systems Ashby’s Law of Requisite Variety essentially says that to control any system, you need to have at least as many levers of control as it has degrees of freedom. All AI systems that model the world have an extremely large number of degrees of freedom. There is presently no way to enumerate them. If we miss a few, there go our constraints. There is no way to have meaningful human oversight over such a vast number of degrees of freedom. Building control upwards from less capable systems to more capable ones is a bit like trying to perfectly reverse a truck that is towing a dozen trailers. Error compounds catastrophically.
Threshold effects There are natural thresholds beyond which we cannot see. Without fundamental breakthroughs in our understanding of intelligent systems, we just have to get there and find out. We don’t know how to design an AI system so that it remains within our desired specification under conditions of possessing the ability to: modify itself or design its successors, permanently escape human oversight if it wanted to, or overthrow humanity if it wanted to. We don’t know how to design values that are stable under reflection, other than the instrumentally convergent ones that tend to be bad for us.
Simple surface-level heuristics Things that are vastly smarter than you won’t stay on your leash. Things that are trained to be persistent and route around hard problems aren’t going to always shy from routing around the problem of human oversight. Mature risk handling means that to survive things that can catastrophically alter the world with minimal effort, they must be provably safe to a similar scale (e.g. a chance of failure of less than one in a quadrillion), and that’s just hard to do.
Researchers and philosophers have had that exact debate for something like 20 years, and I’m not encouraged by the details of the discussion.
Taking a step back, getting a planet full of innocent people into that situation to begin with is indefensible, by the lights of the vast majority of those people.
Ruling out bad behavior is necessary if you’re creating something that you will have no way of controlling or constraining, which is what ASI implies. There’s “advanced AI” and then there’s “the new most capable species on the planet, under which you live at its whim.” No one has any idea how to prevent the former from becoming the latter, other than shutting down all frontier AI R&D on the planet for at least a good while, which is the only sane and morally defensible route we can take.
The naming confused me so much. Liron Shapira of Doom Debates (with the Doom Train) does a show called Warning Shots in partnership with a YouTube channel called the AI Risk Network, which has several shows.
That makes sense; it was easy to read your comment as applying to individual actors, so I appreciate the clarification. (And for extra clarity on my part, I wasn’t meaning to making an accusation against your character even in my initial reading of your statement, which is analytical and neutral. My comment was very pointed, but was not intended to be pointed at you.)
That makes sense for personal risk, but when it comes to other people, it’s not your call. Unilaterally and knowingly exposing the entire world to extreme amounts of additional risk is deeply immoral, of the “doesn’t believe in consent” variety.
It doesn’t have to be up to them, and by no rights should it be. If the US government understood what was happening, OpenAI et al would be shut down overnight.
I’m just some guy, and one of many doing grassroots lobbying about AI extinction risk. And yet, if there had been just 200 more people having my very modest level of impact on US lawmakers over the past 2 years, I think we would have a global AI treaty today, or be very close.
Other countries have less impact, but far from zero! The impact of multiple middle powers formally recognizing that superintelligent AI is an extinction level threat, may be very close, and should be banned would be very substantial.
I think you can get people unstuck from that prior if you offer them a line of retreat toward an even bigger reason to hate the AI companies.
My current go-to snappy retort for use in low-quality discourse (e.g. YouTube comments) is: “Exactly. Just like how oil companies invented climate change to make oil sound cool to investors.” (I don’t have any evidence that this works, but I predict that little nudges and intuition pumps like this would work over time if enough people offer them.)
But then why doesn’t Cal get into PauseAI?
Warning: Incautious psychoanalysis ahead
PauseAI is trying to solve a problem, and people like Cal aren’t in the business of trying to solve problems.
It’s useful to notice here that Cal Newport and Ed Zitron are very similar figures. Cal has expertise in non-AI computer science and can sometimes make a coherent argument, so he’s the shiny variant. But realistically, neither of them have a clue about AI, and they both give off an aura of religious zeal that prevents things like “evidence” from having any effect on them.
I believe people like Cal and Ed pretend to be experts and take up contrarian/anti-scientific positions purely for the sake of attention. But I don’t think they think that’s why they’re doing it! Rather, they’ve incidentally trained themselves to write thinkpieces about technology and the economy and academia, and so now they’re just writing more of those thinkpieces. If they think about recent AI news for five seconds and decide that something about it feels like bullshit to them, they will write an angry article or op-ed about how it’s definitely bullshit, writing the bottom line first and then tacking on sources that seem to back their points. When that is received well and gets reinforced, they then write more things in that genre, contorting the facts into whatever shape allows the applause to continue.
One thing that would make the applause stop is if the problem went away. So long as the core problem continues to exist, they can continue to capitalize on claiming that it doesn’t. That doesn’t mean they actually try to make the problem worse, but they wouldn’t try to make it better, either.
I think there are people for whom—as it concerns their public writings and statements—it is ill advised to model them as anything so complicated as “having beliefs” or “doing reasoning”. If you instead model each of them as a machine whose purpose is what it does: produce content in a specific genre, they become more predictable and less confusing.
The issue is a willful lack of curiosity about verification mechanisms, which are viable in the very near term.
I can imagine a scientific reason for doing so: learning more about how cyber capabilities scale, in order to get empirical data on the details of their emergence, in order to better design training runs that don’t result in dangerous cyber capabilities.
(Though I really doubt this is what they were doing.)
Why not just throw all of your effort into making RSI illegal?
Sufficient alignment of AI systems at one level of capability does not necessarily provide significant evidence that they will be sufficiently aligned at the next level of capability. This is bad enough because alignment is not the first step in (or robustly integrated throughout) the training process, and a misaligned AI might escape during training. But it’s especially bad because of threshold effects. We don’t understand reflexive stability. We don’t understand ontological shift. We don’t know what all can go wrong (or how or why) once an AI system knows that it is capable of takeover with high probability of success, or once it can continuously learn, or [pick your blind corner here].
And this is all beside the fact that many if not most people don’t want superintelligence to be built even if it’s aligned! I think they should get a say, in some sense. I think we owe it to them and to everyone to not ask for anything less than the true bare minimum, which is a complete halt to frontier AI progress! (At least until we are really, very sure from principled reasons and peer-reviewed science that taking another small step forward is safe.)