Nice-ish, smooth takeoff (with imperfect safeguards) probably kills most “classic humans” in a few decades.

[Raemon]

If I were very optimistic about how smooth AI takeoff goes, but where it didn’t include an early step of “fully solve the unbounded alignment problem, and then end up with extremely robust safeguards^[1]”...

...then my current guess is that Nice-ish Smooth Takeoff leads to most biological humans dying, like, 10-80 years later^[2]. (Or, “dying out”, which is slightly different than “everyone dies.” Or, ambiguously-consensually-uploaded. Or, people have to leave their humanity behind much faster than they’d prefer).

Slightly more specific about the assumptions I’m trying to inhabit here:

1. It’s politically intractable to get a global halt or globally controlled takeoff.

2. Superintelligence is moderately likely to be somewhat nice.

3. We’ll get to run lots of experiments on near-human-AI that will be reasonably informative about how things will generalize to the somewhat-superhuman-level.

4. We get to ramp up control on superhuman AIs...

   1. which we use to build defensive immune systems that detect and neutralize attempted FOOMs once those become possible...

   2. ...but, we don’t actually fully solve alignment, which means we can’t scale to the limit of superintelligence and safely leverage it… (i.e. “alignment is difficult” and “we couldn’t trust the AIs that were smart enough to actually help”)

5. ...and we don’t eventually solve alignment and leverage overwhelmingly superhuman intelligence before we find ourselves in a world where groups of powerful-but-only-partly-aligned AI dominate the solar system.

I recently noted that the book Accelerando is a decent takeoff scenario, given similar assumptions. In the book, we see these pieces coming together with passages like this during early takeoff:

A million outbreaks of gray goo—runaway nanoreplicator excursions—threaten to raise the temperature of the biosphere dramatically. They’re all contained by the planetary-scale immune system fashioned from what was once the World Health Organization. Weirder catastrophes threaten the boson factories in the Oort cloud. Antimatter factories hover over the solar poles. Sol system shows all the symptoms of a runaway intelligence excursion, exuberant blemishes as normal for a technological civilization as skin problems on a human adolescent.

Later on, it escalates to:

Earth’s biosphere has been in the intensive care ward for decades, weird rashes of hot-burning replicators erupting across it before the World Health Organization can fix them—gray goo, thylacines, dragons.

So in this hypothetical, AI is sort-of-aligned-or-controlled at first. Defensive technologies make it trickier for undesirable things to FOOM. It assumes the opening transition to AI economy includes a carveout for the Earth as a special protected zone. It has something like property rights (although the AIs/​uploads are using some advanced coordination mechanism which regular humans are too slow/​dumb to participate in, referred to as “Economics 2.0”).

In that world, I still expect normal humans and most normal human interests to die out within a few decades.

I’m not intending to make a strongly confident “everyone obviously dies” claim here. I’m arguing you should have a moderately confidence guess, if you don’t learn knew information, that smooth takeoff results in “somehow or other, ordinary 20th century humans look at the result and think ‘well that sucks a lot’ and the way it sucks involves a lot of people either dying, forcibly uploaded, losing their humanity, or, at best, escaping into deep space in habitats that will later on be consumed.”

There is no safe “muddling through” without perfect safeguards

In this post I’m not arguing with the people trying to leverage AI to fully solve alignment, and then leverage safe superintelligence to fundamentally change the situation. (I have concerns about that but it’s a different point from this post).

This post is instead arguing with the people who imagine something like “business continues sort of as usual in a decentralized fashion, just faster. Things are complicated and messy, but we muddle through somehow, and the result is okay.”

They seem to mostly be imagining the early part of that takeoff – the part that feels human comprehensible. They’re still not imagining superintelligence in the limit, or fully transformed AI driven geopolitics/​economies.

My guess that “things eventually end badly” is due to Robin Hanson-esque arguments, wherein:

1. Digital minds can be easily copied and modified

2. At least some mind lineages are are “grabby” (i.e. decide to rapidly spread through the universe). It only takes one.

3. We eventually exhaust the untapped resources of the solar system. And, because of aforementioned grabbiness, resources across the rest of the universe are either taken, or at least contested.

4. The world is multipolar, there’s no single dominant coalition or authority.

...and then evolution happens to whatever replicating entities result.

An important implication of superintelligence is that once you’re near the limits of intelligence, stuff happens way faster than you’re used to handling at 20th century timescales. There isn’t undirected evolution taking eons, or human decisionmaking taking years. Everything happens hundreds or thousands of times faster than that.

The result will eventually, probably, be quite sad from the perspective of most people today, within their natural lifespan.

(I’m confused by Hanson’s perspective on this – he seems to think the result is actually “good/​fine” instead of “horrifying and sad.” I’m not really sure what it is Hanson actually cares about. But I think he’s probably right about the dynamics)

I’m not that confident in the arguments here, but I haven’t yet seen someone make convincing counterarguments to me about how things will likely play out.

The point of the post is to persuade people who are imagining slow takeoff d/​acc world, that you really do need to solve some important gnarly alignment problems deeply, early in the process of the takeoff, even if you grant the rest of the optimistic assumptions.

i. Factorio

(or: It’s really hard to not just take people’s stuff, when they move as slowly as plants)

I had an experience playing Factorio that feels illustrative here.^[3]

Factorio is a game about automation. In my experience playing it, I gained a kind of deep appreciation for the sort of people who found evil empires.

The game begins with you crash landing on a planet. Your goal is to go home. To go home, you need to build a rocket. To build a rocket powerful enough to get back to your home solar system, you will need advanced metallurgy, combustion engines, electronics, etc. To get those things, you’ll need to bootstrap yourself from the stone age to the nuclear age.

To do this all by yourself, you must automate as much of the work as you can.

To do this efficiently, you’ll need to build stripmines, powerplants, etc. (And, later, automatic tools to build stripmines and powerplants).

One wrinkle is the indigenous creatures on the planet.

They look like weird creepy bugs. It’s left ambiguous how sentient the natives are, and how they should factor into your moral calculus. But regardless, it becomes clear that the more you pollute, the more annoyed they will be, and they will begin to attack your base.

If you’re like me (raised by hippie-ish parents), this might make you feel bad.

During my playthrough, I tried hard not to kill things I didn’t have to, and pollute as minimally as possible. I built defenses in case the aliens attacked, but when I ran out of iron, I looked for new mineral deposits that didn’t have nearby native colonies. I bootstrapped my way to solar power as quickly as possible, replacing my smog-belching furnaces with electric ones.

I needed oil, though.

And the only oil fields I could find were right in the middle of an alien colony.

I stared at the oil field for a few minutes, thinking about how convenient it would be if that alien colony wasn’t there. I stayed true to my principles. “I’ll find another way”, I said. And eventually, at much time cost, I found another oil field.

But around this time, I realized that one of my iron mines was near some native encampments. And those natives started attacking me on a regular basis. I built defenses, but they started attacking harder.

Turns out, just because someone doesn’t literally live in a place doesn’t mean they’re happy with you moving into their territory. The attacks grew more frequent.

Eventually I discovered the alien encampment was… pretty small, compared to my growing factory empire. It would not be difficult for me to destroy it. And, holy hell, would it be so much easier if that encampment didn’t exist. There’s even a sympathetic narrative I could paint for myself, where so many creatures were dying every day as they went to attack my base, that it was in fact merciful to just quickly put down the colony.

I didn’t do that. (Instead, I actually got distracted and died). But this gave me a weird felt sense, perhaps skill, of empathizing with the British Empire. (Or, most industrial empires, modern or ancient).

Like, I was trying really hard not to be a jerk. I was just trying to go home. And it still was difficult not to just move in and take stuff when I wanted. And although this was a video game, I think in real life it might have been if anything harder, since I’d be risking not just losing the game but losing my life or livelihood of people I cared about.

So when I imagine industrial empires that weren’t raised by hippy-ish parents who believe colonialism and pollution were bad… well, what realistically would you expect to happen when they interface with less powerful cultures?

Fictional vs Real Evidence

Factorio is a videogame. In real life, I do not kill people and take their stuff.

But, here are a few real-world things that humans have done, that I think this is illustrative of:

• Various empires across history conquering their neighbors by sword, forcibly erasing their cultural identity and taking their resources.

• An economically/​militarily powerful country actively prevents a weaker country from stopping the foreign power from selling addictive opium to the masses.^[4]

• America spreading across the continent, continuously taking land from the natives, forcing them onto worse land. Eventually, when America’s control was secure from coast to coast, they did make some attempts to be slightly nice and give the natives some land back. But, not particularly good land, and there were some sad downstream consequences.

• European countries “dividing up” Africa among themselves, without much regard for how various African peoples felt about it.

(I’m aware these narratives are simplified. Fwiw, my overall feelings about expansionist empires are actually kinda complicated and confused. But, they are existence proofs for “human-level alignment still can pretty bad for less powerful groups”)

Decades. Or: “thousands of years of subjective time, evolution, and civilizational change.”

Maybe, the first few generations of AI (or human uploads) are nice.

A difference between a hippie-raised humans, and weak-superintelligences-that-can-self-modify, who (like me) are nice-but-sometimes-conflicted, is that it’s possible for the weak superintelligence to actually just decide to modify the sort of being who doesn’t feel pressure to grab all the resources from vastly weaker, slower, stupider being, even though it’d be so easy.

But, it’s not enough for the first few generations of AI/​uploads to be nice. They need to stay nice.

Evolution is not nice. (see: An Alien God)

In the nearterm (i.e. a few years or decades), this might be okay, because there is a growing pie of resources in the solar system. And, it’s possible that the offense/​defense balance favors defense, in the nearterm. But, longterm, the solar system runs out of untapped resources. And longterm, however good defensive technologies are, they’re unlikely to compete with “whoever grabbed stars and galaxies worth of resources first.”

This is the Dream Time

Hanson has argued, right now, we live “The Dream Time”, which is historically very weird, and (by default) will probably be very weird in the longterm, too.

For most history, our ancestors lived at subsistence level. Most people were pretty limited in what they had to freedom to do, because they spent much of their time raising enough food to feed themselves and raise the next generation. If they had surplus it tended to turn into a larger population. Population and resources stayed in equilibrium.

We’ve spend the past few centuries in a period where wealth is growing faster than population. We’re used to having an increasingly vast surplus that we can spend on nice^[5] things like taking care of outgroups and beautiful-but-inefficient architecture.

One of the reasons this works is that industrialized nations have fewer children. But note that this isn’t universal. Some groups (Hutterites, Hmongs, or Mormon, etc) specifically try to have lots of children. This isn’t currently resulting in them dominating culture for various reasons. But that could change.

It might change soon, because “grabbiness” (i.e. trying to get as much resources from the solar system or universe) will be selected for, in an evolutionary sense. Maybe only some AIs are grabby. But their descendants will also be grabby, and the more-grabby ones will have more resources than the less-grabby ones.

If we assume a nice takeoff that initially has an agreement that Earth is protected and gets a little sunlight… in addition to the risk of grabby-evolution in the nearterm, eventually there’ll be a point where all the non-Earth matter in the solar system is converted into computronium, and the rest of the universe has probes underway to seize control of it.

...then we may enter a world where cultural evolution gives way to physical replicator evolution, subject to the old selection pressures.

Also, even if we don’t, cultural evolution might shift in random directions that are less good for classic bio humans, or, the values that we’d like to see flourish in the universe (even taking into account that we don’t want to be human-supremacist).

Is the resulting posthuman population morally valuable?

A related question to “do the posthumans turn Grabby and kill anything weak enough they can dominate?” is “are the posthumans worthwhile in their own right?”. Maybe it’s sad for the classic humans to die off, but, in a cosmic sense, something pretty interesting and meaningful might still be there doing interesting stuff.

Short answer: I don’t know, and don’t think anyone confidently knows. It depends what you value, it depends some details on how the evolution transpires and what is necessary for complex cognition.

Self awareness?

One of the questions that matters (to me, at least) is “Will the resulting entities be self-aware in some fashion? Will there be any kind of ‘there’ there? Will they value anything at all?”. Maybe their form of self-awareness will be different – thousands of AI instances that briefly flicker into existence an then terminate, but, each of them perceiving the universe in their brief way and somehow they still collectively count as the universe looking at itself and seeing that it is good.

My belief is “maybe, but not obviously.” This question is multiple separate posts. See Effectiveness, Consciousness, and AI Welfare. The basic thrust is “humans implement their thinking in a way that routes through consciousness, but this is not obviously the only way to do thinking.

Calculators multiply, without any of the subjective experience a human has when they multiply numbers. Deep Blue executed chess strategy, but my guess it wasn’t much more self-aware than a thermostat. Suno makes music, and Midjourney create art that are sometimes hauntingly beautiful to me – I’m less confident about how their algorithms work, but I bet they are still closer to a thermostat than a human.

I would expect evolution to preserve strategic thought. You need it to outcompete other superintelligences.

But there doesn’t seem like a strong reason to expect that conscious feeling is the best way to execute most kinds of strategic cognition. Even if it turns out there is some selfaware core somewhere that is needed for the highest level of decisionmaking, it could be that most of it’s implementation-details are more shaped like “make a function call to the unconscious python code that efficiently solves a particular type of problem.

The Hanson Counterpoint: “So you’re against ever changing?”

When Hanson gets into arguments about this, and his debate partner says “it would be horrifying for the posthumans to end up nonconscious things that create a disneyland with no children”, my recollection is that Hanson says “so… you’re against anything ever changing?”

With the background argument: to stop this sort of thing from happening, something needs to have a pretty extreme level of control over what all beings in the universe can do. Something very powerful needs to keep being able to police every uncontrolled replicator outbursts that try to dominate the universe and kill all competitors and fill it with hollow worthless things.

It needs to be powerful, and it needs to stay powerful (relative to any potential uncontrolled grabby hollow replicators.

Hanson correctly observes, that’s a kind of absurd amount of power. And, many ways of attempting to build such an entity would result in some kind of stagnation that prevents a lot of possible interesting, diverse value in the universe.

To which I say, yep, that is why the problem is hard.

A permanent safeguard against hollow grabby replicators needs to not only stop hollow grabby replicators. It also needs to have good judgment to let a lot of complex, interesting things happen that we haven’t yet thought about, some of which might be kinda grabby, or inhuman.

Many people seem to have an immune reaction against the rationalist crowd wanting to “build god”, and seeming to orient to it in a totalizing way, where it’s all-or-nothing, you either get a permanent wise, powerful process that is capable of robustly preventing evolution from turning the universe hollow and morally empty… or you get an empty, hollow universe.

And, man, I sure get the wariness of totalizing worldviews. Totalizing worldviews are very sus and dangerous and psychologically wonk and I’m not sure what to do about that.

But I have not seen any kind of vision painted for how you avoid a bad future, for any length of time, that doesn’t involve some kind of process that is just… pretty godlike? The totalizingness really seems like it lives in the territory.

If there are counterarguments that engage with the object level as opposed to heuristically dismiss totalizingness, I would love to hear them.

Can’t superintelligent AIs/​uploads coordinate to avoid this?

In smooth nice takeoff world, wouldn’t we expect to have smart beings who see the onset of evolution destroying a lot of things they care about, and agree to do something else? Building a permanent robust safeguard against evolution is challenging, but, there’ll be superintelligences around.

Yes, probably. This would count as a solution to the problem.

But, this needs to happen at a time when the coalition of AIs/​posthumans that care about anything subtle and interesting and remotely meaningful, are dominant enough to successfully coordinate and implement it.

If they don’t get around to it for like a year (i.e. hundreds/​thousands of years of subjective time for multiple generations of replicators to evolve), then there might already be grabby replicators that have stopped caring about anything subtle and interesting and nuanced because it wasn’t the most efficient way to get resources.

(or, they might still care about something subtle and interesting and nuanced, but not care that they care, such that they wouldn’t mind future generations that care less, and they wouldn’t spend resources joining a coalition to preserve that)

This brings me back to the thesis of this post:

If you grant the assumptions of a smooth, nice, decentralized and differentially defensive takeoff, you still really need to solve some important gnarly alignment problems deeply, early in the process of the takeoff, even if you grant the rest of the optimistic assumptions. It has to happen early enough for some combination of superintelligences who care about anything morally valuable at all to end up dominant.

If this doesn’t happen early enough, classic humans will get outcompeted, and either killed, or die off unless they self-modify into being something powerful enough to keep up.

If you’re kinda okay with that outcome, but you care about any particular thing at all about how the future shakes out, then “superintelligences produce permanent safeguards” needs to happen before evolutionary drift has produced generations of AI that don’t care about anything you care about.

(If you care about neither nearterm humans or any kind of interesting far future, well, coolio. Seems reasonable and I respect your right to exist but I’m sorry I’m going to be working to make sure you don’t have the power to end everything I care about).

How Confident Am I?

This is a pretty complex topic. I have tons of model uncertainty here.

But, these arguments seem sufficient for me to, by default, be extremely worried, even when I grant all the optimistic assumptions about a smooth takeoff. I haven’t seen any compelling counterarguments so far. Let me know if you have them.

1. ^

   (comparable in power to fully fledged Coherent Extrapolated Volition (CEV), although I’m happy to talk separately about how to best aim towards extremely robust safeguards).

2. ^

   To be clear, I think 80 years is very unrealistic here. I also think all the starting assumptions here are very unrealistic. But, a lot of people seem to believe something like this. So it seemed worth talking about how this world would play out, if I imagined the most optimistic version that felt at all coherent.

3. ^

   This is reposted from Factorio, Accelerando, Empathizing with Empires and Moderate Takeoffs

4. ^

   Opium Wars.

5. ^

   Or, as Hanson argues, often kinda stupid things that don’t make practical sense. But, the line between those is blurry.

[dsj]

But I have not seen any kind of vision painted for how you avoid a bad future, for any length of time, that doesn’t involve some kind of process that is just… pretty godlike?

I’m mostly with you all the way up to and including this line. But I would also add: I have not seen a plausible vision painted for how you avoid a bad future, for any length of time, that does involve some kind of process that is just pretty godlike.

This is why I put myself in the “muddle through” camp. It’s not because I think doing so guarantees a good outcome; indeed I’d be hard-pressed even to say it makes it likely. It’s just that by trying to do more than that — to chart a path through territory that we can’t currently even see — we are likely to make the challenge even harder.

Consider someone in 1712 observing the first industrial steam engines, recognizing the revolutionary potential, and wanting to … make sure it goes well. Perhaps they can anticipate its use outside of coal mines — in mills and ships and trains. But there’s just no way they can envision all of the downstream consequences: electricity, radio and television, aircraft, computing, nuclear weapons, the Internet, Twitter, the effect Twitter will have on American democracy (which by the way doesn’t exist yet…), artificial intelligence, and so on. Any attempt someone would have made, at that time, to design, in detail, a path from the steam engine to a permanently good future would just have been guaranteed at the very least to fail, and probably to make things much worse to the extent they locally succeed in doing anything drastic.

Our position is in many ways more challenging than theirs. We have to be humble about how far into the future we can see. I agree that an open society comes with great danger and it’s hard to see how that goes well in the face of rapid technological change. But so too is it hard to see how centralized power over the future leads to a good outcome, especially if the power centralization begins today, in an era when those who would by default possess that power seem to be … extraordinarily cruel and unenlightened. Just as you, rightly, cannot say if AIs who replace us would have any moral value, I also cannot say that an authoritarian future has any value. Indeed, I cannot even say that its value is not hugely negative.

What I can say, however, is that we have some clear problems directly in front of us, either occurring right now or definitely in sight, one of which is this very possibility of a centralized, authoritarian future, from which we would have no escape. I support muddling through only because I see no alternative.

[Raemon]

Nod, I deliberately titled a section There is no safe “muddling through” without perfect safeguards (in an earlier draft I did just say “there is no safe muddling through”, and then was like “okay, that’s false, because, seems totally plausible muddle through into figuring out longerterm safeguards,

(and, in fact, I don’t have a plan to get longterm safeguards that don’t look like some kind of muddling through, in some sense)

I was just chatting with @1a3orn, and he brought up a similar point to the industrial revolution concern, and I totally agree.

Some background assumptions I have here:

• you can’t reason your way all the way to “safely navigate the industrial revolution”, yeah. Some notable failures:

  • inventing communism

  • trying to invent the cotton gin to make slavery less bad, accidentally produce way more slavery via inducing demand

  • environmentalism ending up banning nuclear stuff that caused a lot of environmental damage

  • (there are positive examples too I think, but the existence of these negative examples should put the fear of god in you)

• it’s still possible to do nonzero reasoning ahead. You can put constraints on what sort of things possibly make sense to be doing.

  • early industrial revolution: if you don’t see the first steam train and think “oh shit, everything is gonna change”, man you are going to be pointed in the wrong direction completely

    • analogously: if you don’t look at the oncoming AI (as well as general economic trends), and think “man, All Possible Views About Humanity’s Future Are Wild”, you’re not pointed in the right direction at all

Part of the point of this post was to lay out “here’s the rough class of thing that seems like it’s gonna happen by default. Seems like either we need to learn new facts, or, we need a process with an extreme amount of power and wisdom, or, we should expect some cluster of bad things to probably happen.

During my chat with 1a3orn, I did notice:

Okay, if I’m trying to solve the ‘death by evolution’ problem (assuming we got nice smooth takeoff still), an alternate plan from “build the machine god” is:

Send human-uploads with some von-neuman probes to every star in the universe, immediately, before we leave The Dreamtime. And then probably there will at least be a lot of subjective experience-timeslices and chances for some of them to figure out how to make good things happen, with (maybe) like a 10 year head start before hollow grabby AI comes after them.

I don’t actually believe in nice slow takeoff or 10 year lead times before Hollow Grabby AI comes after them, but, if I did, that’d at least be a coherent plan.

The problems with that is that are:

a) it’s still leaving a lot of risk of costly war between the human diaspora and the Hollow Grabby AI

b) many of the humans across the universe are probably going to do horrible S-risky mindcrime.

So, I’m not very satisfied with that plan, but I mention it to help broaden the creative range of solutions from “build a CEV god” to include at least one other type of option.

[Raemon]

A comment from @Eli Tyre from What are the best arguments for/​against AIs being “slightly ‘nice’”?, that feels worth revisiting here:

It seems to me that one key question here is Will AIs be collectively good enough at coordination to get out from under Moloch /​ natural selection?

The default state of affairs is that natural selection reigns supreme. Humans are optimizing for their values, counter to the goal of inclusive genetic fitness, now, but we haven’t actually escaped natural selection yet. There’s already selection pressure on humans to prefer having kids (and indeed, prefer having hundreds of kids through sperm donation). Unless we get our collective act together, and coordinately decide to do something different, natural selection will eventually reassert itself.

And the same dynamic applies to AI systems. In all likelihood, there will be an explosion of AI systems, and AI systems building new AI systems. Some will care a bit more about humans than others, some will be be a bit more prudent about creating new AIs than others. There will a wide distribution of AI traits, there will be competition between AIs for resources. And there will be selection on that variation: AI systems that are better at seizing resources, and which have a greater tendency to create successor systems that have that property, will proliferate.

After a many “generations” of this, the collective values of the AIs will be whatever was most evolutionarily fit in those early days of the singularity, and that equilibrium is what will shape the universe henceforth.^[1]

If early AIs are sufficiently good at coordinating that they can escape those Molochian dynamics, the equilibrium looks different. If (as is sometimes posited) they’ll be smart enough to use logical decision theories, or tricks like delegating to mutually verified cognitive code and merging of utility functions, to reach agreements that are on their Pareto frontier and avoid burning the commons^[2], the final state of the future will be determined by the values /​ preferences of those AIs.

I would be moderately surprised to hear that superintelligences never reach this threshold of coordination ability. It just seems kind of dumb to burn the cosmic commons, and superintelligences should be able to figure out how to avoid dumb equilibria like that. But the question is when in the chain of AIs building AIs do they reach that threshold and how much natural selection on AI traits will happen in the meantime.

This is relevant to futurecasting more generally, but especially relevant to questions that hinge on AIs carying a very tiny amount about something. Minute slivers of caring are particularly likely to be erroded away in the competitive crush.

Terminally caring about the wellbeing of humans seems unlikely to be selected for. So in order for a the Superintelligent superorganism /​ civilization to decide to spare humanity out of a tiny amount of caring, it has to be the case that both…

1. There was at least a tiny amount of caring in the early AI systems that were the successors to the superintelligent superorganism, and

2. The AIs collectively reached the threshold of coordinating well enough to overturn Moloch before there were many generations of natural selection on AIs creating successor AIs.

1. ^

   With some degrees of freedom due to the fact that AIs with high levels of strategic capability, and which have values with very low time preference, can execute whatever is the optimal resource-securing strategy, postponing and values-specific behaviors until deep in the far future, when they are able to make secure agreements with the rest of AI society.

2. ^

   Or alternatively if the technological landscape is such that a single AI can get a compounding lead and get a decisive strategic advantage over the whole rest of earth civilization.
   

[Eli Tyre]

It’s awesome that the footnotes are inside the blockquote.

[cousin_it]

This agrees almost exactly with my picture of doom. But with a small difference that feels important: I think even if the new powerful entities somehow remain under control of humans (or creatures similar to today’s humans), the rest of humans who aren’t on top are still screwed. Because the nasty things you mentioned (colonialism, etc) were perpetrated by humans who were on the top of a power differential. There’s no need to mention evolution, or new kinds of hypothetical grabby creatures. The increased power differential due to AI is quite enough to cause very bad things, even if there are humans on top.

It seems the only way for future to be good is if it’s dominated by an AI or coalition of AIs that are much more moral than humans, less corruptible by power. Imitating a normal human level of morality is not enough, and folks should probably stop putting their hopes on that.

[Raemon]

Yeah I agree, but one of the points of this post was meant to be take as assumption “good people at Anthropic or whatever do a good job building an actually-more-moral-than-average-for-most-practical purposes human level intelligence” (i.e. via the mechanisms like “the weak superintelligence can just decide to self-modify into the sort of being who doesn’t feel pressure to grab all the resources from vastly weaker, slower, stupider being, even though it’d be so easy.”)

(and, like, I do buy the arguments that if we’re assuming the first bunch of IMO optimistic assumptions about getting to humanish-level-alignment being easy, it’s actually not that hard to do that step)

But then, argue: yeah, even if we assume that one, it’s really not great.

[cousin_it]

“the weak superintelligence can just decide to self-modify into the sort of being who doesn’t feel pressure to grab all the resources from vastly weaker, slower, stupider being, even though it’d be so easy.”

I don’t think this will work. Today’s billionaires can already do something similar to a binding self-modification: donate most of their money to good causes. Not just take a non-binding “pledge”, but actually donate. Few do that. Most of them spend more on increasing their own power than on any kind of charity. For the same reason, I expect future human-like AIs to shy away from moral self-modification. They’ll keep issuing press releases saying “I’ll do it tomorrow” and so on. And as for corporation-like AIs—we can just forget it.

If we want a future where AIs are more moral than people, we need to build them that way from the start.

[Raemon]

I mean I don’t believe most of the leadup-assumptions to this world in the first place.

But, AI would be way more competent at self-modification than billionaires. I think hopes that routes through: “build a aligned-ish medium strength AI, in a corrigibility basin”, then “ask it to help you make it more aligned” is the the sort of thing that might work if you actually succeed at the first step.

Just, you have to then progress quickly to fully aligned wise powerful longterm safeguards, which implies a degree of ‘fully solve alignment’ that it didn’t seem like everyone was properly imagining as necesssary.”

[Random Developer]

I think a lot about Homo erectus, which went extinct a bit over 100,000 years ago. They were quite smart, but they were ultimately competing for resources with smarter hominids. Even the chimps and the gorillas mostly survive in tiny numbers, because they didn’t compete too directly with our ancestors, and because we’d be sad if the last ones went extinct. So we make a half-assed attempt to save them.

In the medium-term, intelligence is really valuable! The species that can (let’s say) win a Nobel Prize every six months of GPU time, and copy itself in seconds, is going to rapidly win any evolutionary competition. The world must have been pretty bewildering for Homo erectus towards the end. When the ancestors of the Neaderthals and modern humans showed up, they brought better tools, and much better communication and coordination.

Or to put it another way, the family dog may be loved and pampered, but nobody asks it whether it wants to be spayed, or if the family should buy a new house. The Homo sapiens all sit around a table and make those decisions without the dog’s input. And maybe the only way to put food on the table is to move to the city, and the only good apartment choices forbid pets.

I don’t think the near-term details matter nearly as much as people think. Takeoff in 5 months, 5 years, or 20 years? Incomprehensible alien superintelligences, or vaguely friendly incomprehensible superintelligences who care about us about as much as we care about chimps? A single AI, or competing AIs? The net result is that humans aren’t making the decisions about our future. “Happy humans” have just become a niche luxury good for things that can run rings around us, with their own goals and problems. And those things, since they can replicate and change, will almost certainly face evolutionary forces of their own.

I definitely think the “benevolent godlike singleton” is just as likely to fail in horrifying ways as any other scenario. Once you permanently give away all your power, how do you guarantee any bargain?

I see superintelligence a lot like a diagnosis of severe metastatic cancer. It’s not whether it’s going to kill you, it’s a question of how many good years of life you can buy before the end. I support an AI halt, because every year we stay halted buys humans more years of life. I suspect that “alignment” is as futile as Homo erectus trying to “align” Homo sapiens. It’s not completely hopeless—dogs did manage to sort of “align” humans—but I think the absolute best-case scenario is “humans as house pets, with owners of varying quality, likely selecting us to be more pleasing pets.” Which, thank you, but that doesn’t actually sound like success? And all the other possibilities go rapidly downhill from there. And Eliezer would argue that the “house pets” solution is a mirage and he may be right.

I’d prefer to put off rolling those dice as long as we can.

[Raemon]

I definitely think the “benevolent godlike singleton” is just as likely to fail in horrifying ways as any other scenario. Once you permanently give away all your power, how do you guarantee any bargain?

This is why you won’t build a benevelont godlike singleton until you have vastly more knowledge than we currently have (i.e, with augmenting human intelligence, etc)^[1]

1. ^

   I’m not sure I buy the current orientation Eliezer/​Nate have to augmented human intelligence in the context of global shutdown, but, does seem like a thing you want before building anything that’s likely to escalate to full overwhelming-in-the-limit-superintelligence.

[Seth Herd]

Intent-aligned multipolar ASI has slightly different logic, and I think it’s part of the vague hopes accelerationists hold for muddling through a multipolar ASI scenario.

I don’t want to sound like I’m defending the worldviews you’re challenging, because I think they’re most often based on inadequate consideration of the relevant factors. The challenge is to get proponents to actually come to grips with the principled reasons you describe that lead to bad outcomes.

One variant of the “we invent ASI and muddle through” is the expectation that it will remain under human control. This is distrubingly muddled with the hopes you debunk, but it deserves to be treated separately.

If we get alignment sort-of right by creating ASI that primarily follows instructions, we have some of the same problems (humans competing with superhuman ASI servants). This competition has a distrubing tendency to favor the most vicious humans. That’s analogous to the problem you describe, which is caring about humans a little being lost as competition favors other goals.

Most of the same problems exist; to survive, we’d need an enforceable social contract preventing anyone from ordering their ASI to create hidden facilities where it could self-improve, build weapons, and take over. I don’t know if that’s possible.

If it’s not, or we don’t bother to try it, I think we get predictably horrible outcomes where the most vicious humans whe get control of an ASI (through fair means or foul) attack first and become god-emperor of the lightcone, implementing their personal utopia. We can hope their sadism-empathy balance isn’t too bad.

If we do set up an enforceable rule-based system of managed competition, we’d be in a scenario somewhat like the past, but with positive and negative differences.

• Downside: powerful humans have no need to preserve humans without power

• Upside: should they want to, they’ll have so much power that preserving powerless humans is trivially easy.

Hopefully, the social contract that keeps them all alive includes a proviso “and we agree to contribute to preserving the plebians.”

This isn’t the glorious anarchic utopia that accelerationists hope for, but neither is the current day or any point in history. There are power structures in an organized power-sharing agreement that allow substantial individual freedom and competition.

[Rohin Shah]

It’s instead arguing with the people who are imagining something like “business continues sort of as usual in a decentralized fashion, just faster, things are complicated and messy, but we muddle through somehow, and the result is okay.”

The argument for this position is more like: “we never have a ‘solution’ that gives us justified confidence that the AI will be aligned, but when we build the AIs, the AIs turn out to be aligned anyway”.

You seem to instead be assuming “we don’t get a ‘solution’, and so we build ASI and all instances of ASI are mostly misaligned but a bit nice, and so most people die”. I probably disagree with that position too, but imo it’s not an especially interesting position to debate, as I do agree that building ASI that is mostly misaligned but a bit nice is a bad outcome that we should try hard to prevent.

[Raemon]

Nod. That is a somewhat different position from “trying to leverage AI to fully solve alignment, and then leverage it to fundamentally change the situation somehow”, but, I’d consider the position you put here to be conceptually similar and this post isn’t arguing against it.

This post is mostly spelling out the explicit assumptions:

• “you need permanent safeguards”

• “those safeguards are very complex and wisdom-loaded”

• and, “you have to build those safeguards before insufficiently friendly AI controls the solar system.”

The people with the most sophisticated views may all agree with this, but I don’t see those assumptions spelled out clearly very often when coming from this direction, and I want to make sure people are on the same page about that requirement, or check if there are arguments for slow-takeoff optimism that don’t route through those three assumptions, since they constrain the goal-state a fair amount.

[Nick_Tarleton]

Nitpick: If unbeatable^[1] defense of a region of space is possible^[2], then being outcompeted-but-not-immediately “only” means losing the cosmic endowment, not going extinct.

1. ^

   or not worth beating

2. ^

   I don’t see how it would be, but the nitpick felt worth noting anyway

[Eli Tyre]

I’m confused by Hanson’s perspective on this – he seems to think the result is actually “good/​fine” instead of “horrifying and sad.” I’m not really sure what it is Hanson actually cares about.

I think partly, he just doesn’t think that it was realistic to hope for anything substantially different this kind of outcome. It doesn’t feel like a loss to him, it just feels like how things were always going to go.

This is not a full explanation though.

[StanislavKrym]

I remember that I tried to raise similar issues before. Assuming poorly solvable alignment, a collection of minds has two forces^[1] which drive its changes: instrumental convergence that you describe and moral reflection.

If instrumentally convergent behavior was more common before the decolonisation, then it arguably means that the human civilisation’s changes were driven by moral reflection. Or that it managed to turn from one attractor to another.

1. ^

   Compare it with the AI 2027 goals forecast. It has six sources of goals. The first two mean that alignment is solved. The third and fourth also depend on the developers. The fifth is instrumental convergence, and the sixth is other sources like moral reflection, tropes from training data and the idea that there is a True Morality waiting to be discovered.

[Oliver Sourbut]

gray goo, thylacines, dragons

(The eldritch, the resurrected, and the fantastical)

I second Accelerando as a remarkable, fun, and thought-provoking read.

[Paul Tiplady]

I think an interesting lens here is to focus on the meta/​structural level. Essentially we need good governance. As you touch on here:

we need a process with an extreme amount of power and wisdom

It is not immediately obvious to me that the required level of power is way out of proportion to the level of wisdom that will be available, if we can find some way of coupling wisdom and intelligence as priorities for models. I don’t really know of anyone doing “wisdom evals” though. Maybe RLHF is in some very weak sense testing for wisdom.

I guess my main point here is that “extreme amount of wisdom” as a prerequisite might not be a deal-breaker when we are talking about manufacturing extreme amounts of power and intelligence

Hanson says “so… you’re against anything ever changing?

This is obviously an oversimplification. I have two main objections.

First, all you can really say is the rate of change must decrease a lot. That sounds a lot more palatable if the overall quality of life is high. (Let’s say you must simulate proposed changes for an aeon in a digital world inhabited by real people before realizing anything in the real world.)

Second, changes where? We’ll inhabit rich digital worlds according to the whims of our imaginations. It’s just the “real” world where change would slow. Maybe this is completely fine?

Even if we ignore the above and take Hanson as stated, if we are talking about inhabiting the Heavenly Abodes, then I might be persuaded to bite the bullet and give up any future changes. “So… you want to destroy heaven and replace it with something you predict might be better?” is a frame that isn’t obviously wrong to me.

[Josh Snider]

This kind of scenario seems pretty reasonable and likely, but I’m much more optimistic about it being morally valuable. Mostly because I expect “grabbiness” to happen sooner and by an AI that is morally valuable.