I’d like to offer my perspective on why this enlightening post, written in Eliezer’s wonderful, super-clear style, can’t possibly eliminate motivated thinking about this one-shot problem.
In my opinion, people can’t see any realistic alternative solution. They see that the alternative is even worse, but maybe they can’t articulate it clearly, even to themselves. Or they just refuse to express it out loud. The fact is that the proposed AI ban (or AI pause) is also a one-shot problem.
How can an AI ban be implemented technically? Not legislatively, but in practice? I’ve only seen proposals to monitor GPUs and bomb datacenters. Okay, to be charitable, suppose we create a truly strict ban that can’t be circumvented gradually, and then we bomb a few secret datacenters to show we’re serious… Congratulations, now it’s a bit inconvenient to work on AI, and we’ve created Bruce Schneier’s security theater. TSA would be proud.
Everyone understands that China can build datacenters no one will find and stockpile GPUs no one knows about. So anyone who proposes an AI ban is ready to discuss China and any possible plans to persuade and motivate China not to create AI. And in classic Murphy’s Law fashion, the threat will come from somewhere else. AI doesn’t really need datacenters, AI doesn’t really need GPUs, it just needs a lot of computing power… or a moderate amount of computing power and a lot of ingenuity.
North Korea, unlike China, can’t build secret datacenters, but North Korean hackers could take a popular online game and connect gamers’ GPUs from all over the world into one massive botnet. The Russians are worse. They could do without GPUs altogether—they’ll find a way to use CPUs. Not now, but in 5-10 years, it’s entirely possible. It might seem like all the talented people have left Russia, but a similar mass exodus in the 1910s and 1920s didn’t stop the Soviet Union. I’m familiar with the Russian math education. It still works despite everything, and they’ll find a way to create AGI out of “crap and sticks”, as they themselves put it. Putin’s Russia is an uncanny copy of Germany in the 1930s. These people feel unfairly treated and humiliated by other countries after World War I = in this case, the collapse of the USSR. They will stop at nothing to seize their historic opportunity, their insane dream of becoming world leaders again.
And we only have one chance to organize this AI ban in such a way that it actually works long enough (where “long enough” is impossible to determine in advance). Once Anthropic stops its work for a few months, it’s all out of our hands—the probe will be sent to Mars, and it’ll be too late to urgently restart Anthropic to create an AGI, which, while just as dangerous, at least won’t make all surviving biologicals sing Putin’s praises! (I’m NOT joking here. I grew up in the USSR and I personally sang praises to Lenin and I wasn’t even mind-controlled, strictly speaking.)
Thus, by proposing an AI ban, we’re effectively offering the following choice:
either existing AI companies voluntarily abandon the AI race and trust other people (some of whom may be incompetent, maniacally narcissistic, or delirious—I can’t believe I’m not even exaggerating here) to orchestrate a worldwide ban on AI, thereby replacing one one-shot problem with another,
or a moderately responsible company with some limited public oversight gets the chance to create an alien, uncontrollable and most profoundly unsafe, but at least not deliberately perverted and evil, superintelligence...
Yes, the second option carries a higher risk. If the AI ban lasts 10 years, some research could help create friendly AI (please bring that term back; I’m tired of the absurd AI “safety” and the meaningless AI “alignment”). But this higher risk seems a less horrifying type of risk, to me and to many others. If there are no better options than a toddler-level-naive AI ban via GPU control, I’d rather take Anthropic’s wager please.
Therefore, if we are confident that we want to save humanity, if we are confident that humanity is worth saving, and that life is worth living, despite all the catastrophic unbelievable stupidity happening in the world (yes, I’m chronically depressed and I apologize for that), then in my opinion, we need to focus our efforts on finding an alternative solution that isn’t just a different one-shot problem with a different risk profile but a potentially worse outcome.
This is a perfectly legitimate question. The curse doesn’t apply due to the vast difference in intelligence.
It would definitely be a one-shot problem for today’s LLMs. They can’t reliably plan very far ahead and they have serious execution issues. The same will almost certainly be true for LLMs next year. After that, all bets are off. ASI is in a different league.
Imagine you have to win a game of chess against a baby. You only have one game. Does that make it a one-shot problem for you?
Or imagine you’re trying to take over an anthill. Is that a one-shot problem for you? Ants are small, slow, and stupid. Theoretically, they could bite you to death. In practice, you’d think about this possibility in advance and use insecticide.