Alex Turner, Oregon State University PhD student working on AI alignment. Reach me at turneale[at]oregonstate[dot]edu.
TurnTrout
Karma: 6,349
I proposed changing “instrumental convergence” to “robust instrumentality.” This proposal has not caught on, and so I reverted the post’s terminology. I’ll just keep using ‘convergently instrumental.’ I do think that ‘convergently instrumental’ makes more sense than ‘instrumentally convergent’, since the agent isn’t “convergent for instrumental reasons”, but rather, it’s more reasonable to say that the instrumentality is convergent in some sense.
For the record, the post used to contain the following section:
A note on terminology
The robustness-of-strategy phenomenon became known as the instrumental convergence hypothesis, but I propose we call it robust instrumentality instead.
From the paper’s introduction:
An action is said to be instrumental to an objective when it helps achieve that objective. Some actions are instrumental to many objectives, making them robustly instrumental. The so-called instrumental convergence thesis is the claim that agents with many different goals, if given time to learn and plan, will eventually converge on exhibiting certain common patterns of behavior that are robustly instrumental (e.g. survival, accessing usable energy, access to computing resources). Bostrom et al.’s instrumental convergence thesis might more aptly be called the robust instrumentality thesis, because it makes no reference to limits or converging processes:
“Several instrumental values can be identified which are convergent in the sense that their attainment would increase the chances of the agent’s goal being realized for a wide range of final goals and a wide range of situations, implying that these instrumental values are likely to be pursued by a broad spectrum of situated intelligent agents.”
Some authors have suggested that gaining power over the environment is a robustly instrumental behavior pattern on which learning agents generally converge as they tend towards optimality. If so, robust instrumentality presents a safety concern for the alignment of advanced reinforcement learning systems with human society: such systems might seek to gain power over humans as part of their environment. For example, Marvin Minsky imagined that an agent tasked with proving the Riemann hypothesis might rationally turn the planet into computational resources.
This choice is not costless: many are already acclimated to the existing ‘instrumental convergence.’ It even has its own Wikipedia page. Nonetheless, if there ever were a time to make the shift, that time would be now.
Sure.
Additional note for posterity: when I talked about “some objectives [may] make alignment far more likely”, I was considering something like “given this pretraining objective and an otherwise fixed training process, what is the measure of data-sets in the N-datapoint hypercube such that the trained model is aligned?”, perhaps also weighting by ease of specification in some sense.
Claim 3: If you don’t control the dataset, it mostly doesn’t matter what pretraining objective you use (assuming you use a simple one rather than e.g. a reward function that encodes all of human values); the properties of the model are going to be roughly similar regardless.
Analogous claim: since any program specifiable under UTM U1 is also expressible under UTM U2, choice of UTM doesn’t matter.
And this is true up to a point: up to constant factors, it doesn’t matter. But U1 can make it easier (simplier, faster, etc) to specify a set of programs than does U2. And so “there exists a program in U2-encoding which implements P in U1-encoding” doesn’t get everything I want: I want to reason about the distribution of programs, about how hard it tends to be to get programs with desirable properties.
Stepping out of the analogy, even though I agree that “reasonable” pretraining objectives are all compatible with aligned / unaligned /arbitrarily behaved models, this argument seems to leave room that some objectives make alignment far more likely, a priori. And you may be noting as much:
(This is probably the weakest argument in the chain; just because most of the influence comes from the dataset doesn’t mean that the pretraining objective can’t have influence as well. I still think the claim is true though, and I still feel pretty confident about the final conclusion in the next claim.)
Even so, I also think “Winston Churchill, Futurist” makes more sense and better describes the content of your post.
Yeah, that’s right. (That is why I called it the “start” of a theory on invariances!)
I think that’s an interesting frame which I’ll return to when I think more about agents planning over an imperfect world model.
In my reading about the various usages of ‘power’, there are indeed definitions which focus on exerting control through other agents. I think in many situations, this is a useful frame, but I find “ability to achieve goals in general” to be both broader and also upstream of “ability to control others to achieve your goals.”
(also—upvoted for asking a question and then editing to acknowledge a misunderstanding!)
As I understand expanding candy into A and B but not expanding the other will make the ratios go differently.
What do you mean?
If we knew what was important and what not we would be sure about the optimality. But since we think we don’t know it or might be in error about it we are treating that the value could be hiding anywhere.
I’m not currently trying to make claims about what variants we’ll actually be likely to specify, if that’s what you mean. Just that in the reasonably broad set of situations covered by my theorems, the vast majority of variants of every objective function will make power-seeking optimal.
The More Power At Stake, The Stronger Instrumental Convergence Gets For Optimal Policies
The main benefit is that the AI is not at risk of killing you. In the left half of the universe, it is at risk of killing you.
I don’t follow why you disagree. It’s higher-stakes to operate something which can easily kill me, than to operate something which can’t.
I’m torn about whether this seems lower-stakes or not.
I think it is lower-stakes in a fairly straightforward way: An unaligned AGI on the right side of the universe won’t be able to kill you and your civilization.
Furthermore, most misspecifications will just end up with a worthless right half of the universe—you’d have to be quite good at alignment in order to motivate the AGI to actually create and harm humans, as opposed to the AGI wireheading forever off of a sensory reward signal.
Yeah, we are magically instantly influencing an AGI which will thereafter be outside of our light cone. This is not a proposal, or something which I’m claiming is possible in our universe. Just take for granted that such a thing is possible in this contrived example environment.
My conception of utility is that it’s a synthetic calculation from observations about the state of the universe, not that it’s a thing on it’s own which can carry information.
Well, maybe here’s a better way of communicating what I’m after:
Suppose that you have beliefs about the initial state of the right (AGI) half, and you know how it’s going to evolve; this gives you a distribution over right-half universe histories—you have beliefs about the AGI’s initial state, and you can compute the consequences of those beliefs in terms of how the right half of the universe will end up.
In this way, you can take expected utility over the joint universe history, without being able to observe what’s actually happening on the AGI’s end. This is similar to how I prefer “start a universe which grows to be filled with human flourishing” over “start a universe which fills itself with suffering”, even though I may not observe the fruits of either decision.
Is this clearer?
I’m not sure if you’re arguing that this is a good world in which to think about alignment.
I am not arguing this. Quoting my reply to ofer:
I think I sometimes bump into reasoning that feels like “instrumental convergence, smart AI, & humans exist in the universe → bad things happen to us / the AI finds a way to hurt us”; I think this is usually true, but not necessarily true, and so this extreme example illustrates how the implication can fail.
(Edited post to clarify)
Even in environments where the agent is “alone”, we may still expect the agent to have the following potential convergent instrumental values
Right. But I think I sometimes bump into reasoning that feels like “instrumental convergence, smart AI, & humans exist in the universe → bad things happen to us / the AI finds a way to hurt us”; I think this is usually true, but not necessarily true, and so this extreme example illustrates how the implication can fail. (And note that the AGI could still hurt us in a sense, by simulating and torturing humans using its compute. And some decision theories do seem to have it do that kind of thing.)
(Edited post to clarify)
A world in which the alignment problem seems lower-stakes
I like this question. If I had to offer a response from econ 101:
Suppose people love eating a certain endangered species of whale, and that people would be sad if the whale went extinct, but otherwise didn’t care about how many of these whales there were. Any individual consumer might reason that their consumption is unlikely to cause the whale to go extinct.
We have a tragedy of the commons, and we need to internalize the negative externalities of whale hunting. However, the harm is discontinuous in the number of whales remaining: there’s an irreversible extinction point. Therefore, Pigouvian taxes aren’t actually a good idea because regulators may not be sure what the post-tax equilibrium quantity will be. If the quantity is too high, the whales go extinct.
Therefore, a “cap and trade” program would work better: there are a set number of whales that can be killed each year, and firms trade “whale certificates” with each other. (And, IIRC, if # of certificates = post-tax equilibrium quantity, this scheme has the same effect as a Pigouvian tax of the appropriate amount.)
Similarly: if I, a house member, am unsure about others’ willingness to pay for risky activities, then maybe I want to cap the weekly allowable microcovids and allow people to trade them amongst themselves. This is basically a fancier version of “here’s the house’s weekly microcovid allowance” which I heard several houses used. I’m protecting myself against my uncertainty like “maybe someone will just go sing at a bar one week, and they’ll pay me $1,000, but actually I really don’t want to get sick for $1,000.” (EDIT: In this case, maybe you need to charge more per microcovid? This makes me less confident in the rest of this argument.)
There are a couple of problems with this argument. First, you said taxes worked fine for your group house, which somewhat (but not totally) discredits all of this theorizing. Second, (4) seems most likely. Otherwise, I feel like we might have heard about covid taxes being considered and then discarded (in e.g. different retrospectives)?
Perhaps I don’t follow. why would you have to market-base “all the transactions in a group house”, instead of just the COVID-19 ones?
My take on it has been, the theorem’s bottleneck assumption implies that you can’t reach S again after taking action a1 or a2, which rules out cycles.
If the agent is sufficiently farsighted (i.e. the discount is near 1)
I’d change this to “optimizes average reward (i.e. the discount equals 1)”. Otherwise looks good!
“What are you doing?” often means “what are you thinking, why are you doing that thing? That’s wrong.” At least to my ear.
I prefer to start higher bandwidth conversations with “what have you been thinking about recently?”.