AI as the slightly unbeatable opponent

Introduction

I’ve thought about this problem quite a lot since 2008 when I first encountered the idea of “Friendly AI” and that artificial intelligence could be something other than cool and science-fictiony and Matrix-y. My conclusion to date is that the arguments I’ve seen both for and against it are bad, but despite this the conclusion of AI doomers—that AI will effectively kill or substantively depower everyone unless specifically designed not to, or simply never created—is likely true.

To pick on the still-somewhat-recent If Anyone Builds It, Everyone Dies, the arguments in that book about evolution are badly drawn because their arguments imply the opposite to the conclusions that the book states. They argue that because humans trade in iced cream rather than sugared bear fat in supermarkets, this is an analogy which says the values of a hypothetical “randomly-drawn” intelligence will be unpredictable to us. The argument is about a concept called “inner misalignment,” something which you probably don’t need to know about because it’s basically just wrong. You have an optimiser, you optimise for something, you get that thing. There’s no “inner misalignment,” only misunderstanding as to how the thing you got was the thing you were optimising for. We see this with LLMs, RLHF basically just works.

The only actual criterion for the evolution of life is survival, and the authors themselves are making this argument—an argument in favour of the likelihood of inner misalignment—in a book, ostensibly, which has been written to increase the likelihood of human survival.

Hasn’t evolution in fact “inner-aligned” us almost perfectly? That we will go through very abstract and disconnected trains of communicative thought in an attempt to maybe survive a little more? Aren’t the authors doing exactly this? Aren’t most people basically good?

“But we have a choice!” Of course we have a choice. And if you choose not to survive on evolution’s terms then you won’t be alive. This is the instrumental convergence idea, which is itself just a restatement of the tautological evolutionary fact that, that which successfully chooses to survive, in fact survives.

So what’s my view on the topic?

Unfortunately, although argument on both sides is bad (I’m not even mentioning the arguments against AI risk—check out “On the Impossibility of Supersized Machines” for a representative sample of my views there) the conclusion of AI doomers is correct. If we build advanced AI it, at the very least, has de facto control of what happens. I will now develop this point in detail because I have only ever seen it gestured at with the rhetorical equivalent of a “it’s just obvious why-”

Why does building advanced AI result in humans ceding control of events?

We’ll start with some etymology. Intelligence comes from the Latin words inter- and legere meaning “choose between”. Intelligence, etymologically, and I believe still today when you remove the confusion glasses, refers to the idea that intelligence is whatever quantity that allows you to choose what happens, and especially still choose it regardless of barriers, problems, or active opposition.

The argument from here, if you accept this, is quite direct: building artificial intelligence greater than human intelligence is just the same thing as giving up control over what happens to the thing you build. You are supposing that you actually build something of greater intelligence than all humans, so therefore it can decide what happens regardless of even active opposition from humans. That is what the concept means, there are no extra argument steps necessary.

I pause to note here that metrically, kids have generally been consistently smarter than their parents. So this kind of giving up control to something smarter than yourself is, or at least should be, relatively normal. Unfortunately we have a bit of a gerontocracy issue today so probably the smartest people aren’t in charge right now, and indeed the world is absurdly stupid so that tracks.

In any case, you might not accept that intelligence is best understood as choice regardless of opposition, so I will make the case here for you.

Why is intelligence best conceived of as an ability to choose despite opposition?

We most usually apply labels like “intelligent” or “intelligence” to living systems or artifacts produced by them. A living system has many states and evolution trajectories which will result in it no longer being living. This is the default, and a lot of constant maintenance work goes into just maintaining you as you are. Consider what happened to the poor Japanese man who was exposed to such a high radiation dose that all his DNA had been destroyed (there was an xkcd What If about it). IIRC, with the best treatment available, they were able to keep him alive for around 80 days, mostly in excruciating agony as his skin sloughed off and organs broke down.

All living organisms are embodied intelligence in that the choice to continue living is being made constantly, through constant effort. Without this effort, the default is death, and even painful death. So the vast majority of your intelligence goes into maintaining your homeostatic livingness.

In some small ways, we humans can alter not just ourselves but our environment as extensions of ourselves, and thereby become greater. This is recursive self-improvement, because the structure of the world is exponential (that’s Chaos theory). So recursive self-improvement, that big bogeyman, ooh scary singularity that’s going to take over the world: No. that’s everyday stuff. Quotidian. Prosaic. Normal.

Despite this, recall, I am saying building advanced AI will kill us almost certainly if actually built. It’s just that recursive self-improvement is not the reason, nor is it anything extraordinary.

No, the real reason is just that it will be slightly more consistently better at most things than us. That’s enough to depower us completely over a fairly short amount of time due to the aforementioned chaotic exponentiality. If “slightly more consistently good at pretty much everything” doesn’t sound scary, I guess I need to paint you more of a picture.

All that is necessary for a more intelligent artificial system to depower and likely kill humans is that it is slightly more consistently good at making decisions than any combination of humans adversarial to it

Let’s talk about the structure of competition: specifically, the statistical structure. The picture that you might expect is one in which there is a range of skill levels, some average skill maybe so the whole thing looks pretty much like a normal distribution, and everyone who engages in this competition—let’s call it “Dota 2” for no particular reason—falls somewhere in this bell curve.

That would be true, except for the people who actually try to compete who are completely off the chart. Active competition produces optimisation against exploitability: this means, basically, that as you get better at a competitive game you make less mistakes. Following Toshiro Kageyama from Lessons in the Fundamentals of Go, you are considered an expert amateur if you can successfully play a game without making any mistakes.

Professional play starts from that basis: that neither side is making any exploitable mistakes.

Let me try to give you some sense of the scale. The actual activities that expert gamers (“Dota 2 players”) are doing may not seem that different to what those in the middle of the bell curve are doing. Especially, you, if you are not an expert, can usually explain or find a reason for why the expert did or didn’t do something and thereby understand it better.

The rating scale for our pictured game, “Dota 2” (which may or may not be related to the video game Dota 2 that actually exists) goes from 0 to about 14,000. An average player would sit around 2,000 to 2,500, and for each 1,500 rating a player gains, they can cause their team to win about (say) 75% of the time: that is, going from 1:1 odds to 3:1 odds. Our game is a 5 vs 5 team game with randomness, so skill isn’t always the determining factor. But if we just continue to multiply the odds, then for each 1,500 rating assessed to a player on an otherwise even game, we multiply the odds ratio by 3. Since the maximum is around 14,000, that’s about 3^8:1 odds of winning in our even, 2,000-rated game, or 6,561:1. A simple multiplication probably understates the real probability though because part of gaining skill is getting better at dealing with randomness. And as, as our player improves, they have had their exploitable weaknesses “sanded off”, what is more likely to be the case is that there will be a rating cutoff where an average team will literally never win a game barring force majeure act of god. This is why a drunk “super-GM” chess player can still play blind simultaneous chess against like 10 people and win them all. Their floor of skill is just that much higher.

This is why being consistently slightly better at making choices is, should be, terrifying.

If we build this thing, it will decide what happens, and that won’t be humans

Humans are not optimised for any particular purpose. Take any task or behaviour. We, ourselves, can already for any given task or behaviour, build a machine that does it more reliably than a human. That’s it, case closed. Humans aren’t needed, in fact they are selected against in favour of more reliable machines by any rational designer.

This thing will treat us as childlike hangers-on while it does whatever it decides, and unless that is exactly “human flourishing, whatever that means” we’re dead. In fact, getting it to care about humans at all likely makes it more dangerous, because then you’re increasing the likelihood of fates for humanity worse than just all dying.

I don’t know about you, but this target—building an advanced AI with exactly “human flourishing” as its values is impossibly small. We are almost surely dead, if it gets built, in the mathematical sense.

Please stop trying to build the death machine

I’ll put a brief note here in case anyone working at an AI lab sees this. I want to be completely fair: you are right that LLMs aren’t particularly dangerous. They are nonlinear Markovian models with some ingenious engineering to make them output sequences of up to large lengths. But they aren’t intelligent, so making better LLMs is probably fine.

But unfortunately, it is the explicit stated goal of all of DeepMind, OpenAI and Anthropic to build exactly the thing that will kill us if we don’t get it perfectly right.

I’ll be blunt. I want you, yes you, reading this, to quit your job there and do literally anything else. Stop building it. Literally go and stomp on some puppies. Almost anything is better than continuing to act within an incentive structure that rewards the creation of this machine.

Again, it’s not an “AI God”. It won’t have magic powers, hack physics, or do anything apparently unordinary. It will just be slightly better at intelligence, and that is the game. So stop building it. Please.

Intelligence is choosing despite opposition and that’s terrifying

You might have gained a new respect for human intelligence after this. If not, you should have. It’s incredible that we not only maintain ourselves as we are but actively grow, change, learn, and improve, and even in the realm of values and morality do we learn and change. Humans are amazingly intelligent. But you would have to admit that there’s just a lot of room for a machine to be consistently slightly more reliable at it, and that is, terrifyingly, all it takes. Almost all (in the mathematical sense, again) goals result in extinction, and actually all goals result in human disempowerment.

If we want the better sci-fi future we need to empower humans, not build an overlord

I have this weird feeling sometimes that people just want to live under tyranny. They want someone to blame when things are shit, someone to tell them they can’t do good things, basically to make the world make sense for them by giving them arbitrary rules. At least then it’s understandable.

I think this desire to build an overlord machine stems from this same childish desire for someone else to make the world make sense for them. I don’t have this desire, and I suggest if you do, that you get over it. Tyranny is bad, arbitrary coercive rules are bad, and someone else enforcing their view of the world on you is bad. Stop wanting to be oppressed and start living.

What does the human intelligence future look like?

Most of these essays are massive downers, because they never provide anything concrete about what to do. In my view, the problem is relatively simple, we just have another background thing not to do like nuclear war, except with a stronger avoidance mechanism. I will leave the design treatise’ to others on that.

The human future looks like us owning our technology and changing it to suit us. It looks like humans interacting in normal human ways, like sharing stories, sitting around campfires, making food, playing games, and yes, doing work. As I said before, there’s a bit of an issue right now with the competence of the people in charge, but by now this issue is widely known. Most problems have known-to-be-effective solutions, there have been experts studying that exactly that policy problem for decades, all we need is good people who will listen to good advice, with the solidarity of support from enough of the population to enact the governance changes that need to be enacted.

We can have the 18-hour work week. We can have cancer cures. We can have advanced biotechnology. We can have augmented reality, sex through the internet, and hyperrealistic games.

We can have all that provided we continue to not build the thing that will tyrannise us, at least until we as human develop the amateur expertise, in the lingo of games, to not be making any mistakes, as a starting point.

We’ve got a long way to go to get there, but I’m optimistic. I think people see the problems. The gerontocracy issue solves itself with time, we just have to not reinforce the bad systems, and there’s plenty of opportunities for positive change. Donate $5 to your local food bank. There, you just gave someone a meal, something that might have been vital to their life. There’s an infinity of ways in which you can improve your own life and the lives of other humans.

Building an Artificial General Intelligence is not one of those ways. Do literally anything else instead.

A Normal Argument for AI Risk