Are there any research questions you’re excited about people working on, for making AI go (existentially) well, that are not related to technical AI alignment or safety? If so, what? (I’m especially interested in AI strategy/governance questions)
Sam Clarke
Karma: 460
Clarifying “What failure looks like”
Distinguishing AI takeover scenarios
Survey on AI existential risk scenarios
When reporting AI timelines, be clear who you’re deferring to
Deference on AI timelines: survey results
[Question] Collection of arguments to expect (outer and inner) alignment failure?
[Question] What are the biggest current impacts of AI?
I sometimes want to point people towards a very short, clear summary of What failure looks like, which doesn’t seem to exist, so here’s my attempt.
Many agentic AI systems gradually increase in intelligence and generality, and are deployed increasingly widely across society to do important tasks (e.g., law enforcement, running companies, manufacturing and logistics).
Initially, this world looks great from a human perspective, and most people are much richer than they are today.
But things then go badly in one of two ways (or more likely, a combination of both).
[Part 1] Going out with a whimper
In the training process, we used easily-measurable proxy goals as objective functions, that don’t push the AI systems to do what we actually want e.g.
‘maximise positive feedback from your operator’ instead of ‘try to help your operator get what they actually want’
‘reduce reported crimes’ instead of ‘actually prevent crime’
‘increase reported life satisfaction’ instead of ‘actually help humans live good lives’
‘increasing human wealth on paper’ instead of ‘increasing effective human control over resources’
(We did this because ML needs lots of data/feedback to train systems, and you can collect much more data/feedback on easily-measurable objectives.)
Due to competitive pressures, systems continue being deployed despite some people pointing out this is a bad idea.
The goals of AI systems gradually gain more influence over the future relative to human goals.
Eventually, the proxies for which the AI systems are optimising come apart from the goals we truly care about, but by then humanity won’t be able to take back influence, and we’ll have permanently lost some of our ability to steer our trajectory. In the end, we will either go extinct or be mostly disempowered.
(In some sense, this isn’t really a big departure from what is already happening today—just imagine replacing today’s powerful corporations and states with machines pursuing similar objectives).
[Part 2] Going out with a bang
These AI systems end up learning objectives that are unrelated to the objective functions used in the training process, because the objective they ended up learning was more naturally discovered during the training process (e.g. “don’t get shut down”).
The systems seek influence as an instrumental subgoal (since with more influence, a system is more likely to be able to e.g. prevent attempts to shut it down).
Early in training, the best way to do that is by being obedient (since systems understand that unobedient behaviour would get them shut down).
Then, once the systems become sufficiently capable, they attempt to acquire resources and influence to more effectively achieve their goals, including by eliminating the influence of humans. In the end, humans will most likely go extinct, because the systems have no incentive to preserve our survival.
Just wanted to say this is the single most useful thing I’ve read for improving my understanding of alignment difficulty. Thanks for taking the time to write it!
Strong upvote, I would also love to see more disscussion on the difficulty of inner alignment.
which if true should preclude strong confidence in disaster scenarios
Though only for disaster scenarios that rely on inner misalignment, right?
… seem like world models that make sense to me, given the surrounding justifications
FWIW, I don’t really understand those world models/intuitions yet:
Re: “earlier patches not generalising as well as the deep algorithms”—I don’t understand/am sceptical about the abstraction of “earlier patches” vs. “deep algorithms learned as intelligence is scaled up”. What seem to be dubbed “patches that won’t generalise well” seem to me to be more like “plausibly successful shaping of the model’s goals”. I don’t see why, at some point when the model gets sufficiently smart, gradient descent will get it to throw out the goals it used to have. What am I missing?
Re: corrigibility being “anti-natural” in a certain sense—I think I just don’t understand this at all. Has it been discussed clearly anywhere else?
(jtbc, I think inner misalignment might be a big problem, I just haven’t seen any good argument for it plausibly being the main problem)
Re: corrigibility being “anti-natural” in a certain sense—I think I have a better understanding of this now:
Eventually, we need to train an AI system capable enough to enable a pivotal act (in particular, actions that prevent the world from being destroyed by any other future AGI)
AI systems that are capable enough to enable a pivotal act must be (what Eliezer calls) a “consequentialist”: a system that “searches paths through time and selects high-scoring ones for output”
Training an aligned/corrigible/obedient consequentialist is something that Eliezer can’t currently see a way of doing, because it seems like a very unnatural sort of system. This makes him pessimistic about our current trajectory. The argument here seems kinda like a more subtle version of the instrumental convergence thesis. We want to train a system that:
(1) searches for (and tries to bring about) paths through time that are robust enough to hit a narrow target (enabling a pivotal act and a great future in general)
but also (2) is happy for certain human-initiated attempts to change that target (modify its goals, shut it down, etc.)
This seems unnatural and Eliezer can’t see how to do it currently.
An exacerbating factor is that even if top labs pursue alignment/corrigiblity/obedience, they will either be mistaken in having achieved it (because it’s hard), or honestly panic about not having achieved it and halt, by which point a runner-up who doesn’t understand the importance of alignment/corrigibility/obedience deploys their system which destroys the world.
(This is partly based on this summary)
The AI systems in part I of the story are NOT “narrow” or “non-agentic”
There’s no difference between the level of “narrowness” or “agency” of the AI systems between parts I and II of the story.
Many people (including Richard Ngo and myself) seem to have interpreted part I as arguing that there could be an AI takeover by AI systems that are non-agentic and/or narrow (i.e. are not agentic AGI). But this is not at all what Paul intended to argue.
Put another way, both parts I and II are instances of the “second species” concern/gorilla problem: that AI systems will gain control of humanity’s future. (I think this is also identical to what people mean when they say “AI takeover”.)
As far as I can tell, this isn’t really a different kind of concern from the classic Bostrom-Yudkowsky case for AI x-risk. It’s just a more nuanced picture of what goes wrong, that also makes failure look plausible in slow takeoff worlds.
Instead, the key difference between parts I and II of the story is the way that the models’ objectives generalise.
In part II, it’s the kind of generalisation typically called a “treacherous turn”. The models learn the objective of “seeking influence”. Early in training, the best way to do that is by “playing nice”. The failure mode is that, once they become sufficiently capable, they no longer need to play nice and instead take control of humanity’s future.
In part I, it’s a different kind of generalisation, which has been much less discussed. The models learn some easily-measurable objective which isn’t what humans actually want. In other words, the failure mode is that these models are trying to “produce high scores” instead of “help humans get what they want”. You might think that using human feedback to specify the base objective will alleviate this problem (e.g. use learn a reward model from human demonstrations or preferences about a hard-to-measure objective). But this doesn’t obviously help: now, the failure mode is that the model learns the objective “do things that look to humans like you are achieving X” or “do things that the humans giving feedback about X will rate highly” (instead of “actually achieving X”).
Notice that in both of these scenarios, the models are mesa-optimizers (i.e. the learned models are themselves optimizers), and failure ensues because the models’ learned objectives generalise in the wrong way.
This was discussed in comments (on a separate post) by Richard Ngo and Paul Christiano. There’s a lot more important discussion in that comment thread, which is summarised in this doc.
Re the argument for “Why internalization might be difficult”, I asked Evan Hubinger for his take on your rendition of the argument, and he thinks it’s not right.
Rather, the argument that Risks from Learned Optimization makes that internalization would be difficult is that:
~all models with good performance on a diverse training set probably have to have a complex world model already, which likely includes a model of the base objective,
so having the base objective re-encoded in a separate part of the model that represents its objective is just a waste of space/complexity.
Especially since this post is now (rightly!) cited in several introductory AI risk syllabi, it might be worth correcting this, if you agree it’s an error.
I’ve since been told about Tasshin Fogleman’s guided metta meditations, and have found their aesethic to be much more up my alley than the others I’ve tried. I’d expect others who prefer a more rationalist-y aesthetic to feel similarly.
The one called ‘Loving our parts’ seems particularly good for self-love practice.
I found this post helpful and interesting, and refer to it often! FWIW I think that powerful persuasion tools could have bad effects on the memetic ecosystem even if they don’t shift the balance of power to a world with fewer, more powerful ideologies. In particular, the number of ideologies could remain roughly constant, but each could get more ‘sticky’. This would make reasonable debate and truth-seeking harder, as well as reducing trusted and credible multipartisan sources. This seems like an existential risk factor, e.g. because it will make coordination harder. (Analogy to how vaccine and mask hesitancy during Covid was partly due to insufficient trust in public health advice). Or more speculatively I could also imagine an extreme version of sticky, splintered epistemic bubbles this leading to moral stagnation/value lock-in.
Minor question on framing: I’m wondering why you chose to call this post “AI takeover without AGI or agency?” given that the effects of powerful persuasion tools you talk about aren’t what (I normally think of as) “AI takeover”? (Rather, if I’ve understood correctly, they are “persuasion tools as existential risk factor”, or “persuasion tools as mechanism for power concentration among humans”.)
Somewhat related: I think there could be a case made for takeover by goal-directed but narrow AI, though I haven’t really seen it made. But I can’t see a case for takeover by non-goal-directed AI, since why would AI systems without goals want to take over? I’d be interested if you have any thoughts on those two things.
Thanks for pointing this out. We did intend for cases like this to be included, but I agree that it’s unclear if respondents interpreted it that way. We should have clarified this in the survey instructions.
Relatedly: if we manage to solve intent alignment (including making it competitive) but still have an existential catastrophe, what went wrong?
One argument for alignment difficulty is that corrigibility is “anti-natural” in a certain sense. I’ve tried to write out my understanding of this argument, and would be curious if anyone could add or improve anything about it.
I’d be equally interested in any attempts at succinctly stating other arguments for/against alignment difficulty.
Minor terminology note, in case discussion about “genomic/genetic bottleneck” continues: genetic bottleneck appears to have a standard meaning in ecology (different to Richard’s meaning), so genomic bottleneck seems like the better term to use.