PhD student at the Center for Human-Compatible AI. Creator of the Alignment Newsletter. http://rohinshah.com/
rohinmshah(Rohin Shah)
Karma: 6,562
Yeah, this makes much more sense.
The claim here is that either (a) the AI in question doesn’t achieve the main value prop of AI (i.e. reasoning about systems too complicated for humans), or (b) the system itself has to do the work of making sure it’s safe.
I see the intuitive appeal of this claim, but it seems too strong. I suspect if we look at rates of accidents over time they’ll have been going down over time, at least for the last few centuries. It seems like this can continue going down, to an asymptote of zero, in the same way it has been so far—we become better at understanding how accidents happen and more careful in how we use dangerous technologies. We already use tools for this (in software, we use debuggers, profilers, type systems, etc) or delegate to other humans (as in a large company). We can continue to do so with AI systems.
I buy that eventually “most of the work” has to be done by the AI system, but it seems plausible that this won’t happen until well after advanced AI, and that advanced AI will help us in getting there. And so, that from a what-should-we-do perspective, it’s fine to rely on humans for some aspects of safety in the short term (though of course it would be preferable to delegate entirely to a system we knew was safe and beneficial).
(Why bother relying on humans? If you want to build a goal-directed AI system, it sure seems better if it’s under the control of some human, rather than not. It’s not clear what a plausible option is if you can’t have the AI system under the control of some human.)
In the die-roll analogy, the hope is the rate at which you roll dice approximately decays exponentially, so that you only roll an asymptotically constant number of dice.
I feel like I don’t understand what you’re getting at here. It seems to me like you’re saying “X cannot be guaranteed to never cause bad things”, which seems to me to be so obvious that it’s not worth mentioning.
Nothing is inherently safe. There’s always the possibility that you’ve neglected some way by which danger could arise. What I want from arguments for risk is an argument that it is high EV to address the risk (often broken down into importance, tractability, neglectedness), or at least that the risk is reasonably high on probability * harm (which focuses just on importance).
Of course Tool AI isn’t inherently safe. A knife isn’t inherently safe, despite being a tool; you wouldn’t give a knife to a baby. The argument is usually that Tool AI would, by design, not be adversarially optimizing against humans, because it isn’t optimizing at all, and so a particular class of risks typically considered in the AI safety community would be avoided. It is unclear whether such an argument can work, but it’s certainly not “it is impossible for bad things to happen if you build a Tool AI”, which is clearly wrong.
Perhaps you’re arguing that we should focus on the “how do we manage potentially dangerous information” problem? Or that we should focus on improving human intelligence so that we are better able to use our AI systems?
The slightly longer and slightly worse version is the pair of Human-AI Interaction and Reward uncertainty.
I share Alex’s intuition in a sibling comment:
How about helping a tree? It actually seems pretty straightforward to me how to help a tree
Yes, there is interpretive labor, and yes, things become fuzzy as situations become more and more extreme, but if you want to help an agent-ish thing it shouldn’t be too hard to add some value and not cause massive harm.
I expect MIRI-cluster to agree with this point—think of the sentiment “the AI knows what you want it to do, it just doesn’t care”. The difficulty isn’t in being competent enough to help humans, it’s in being motivated to help humans. (If you thought that we had to formally define everything and prove theorems w.r.t the formal definitions or else we’re doomed, then you might think that the fact that humans aren’t clear agents poses a problem; that might be one way that MIRI-cluster and I disagree.)
I could imagine that for some specific designs for AI systems you could say that they would fail to help humans because they make a false assumption of too-much-agentiness. If the plan was “literally run an optimal strategy pair for an assistance game (CIRL)”, I think that would be a correct critique—most egregiously, CIRL assumes a fixed reward function, but humans change over time. But I don’t see why it would be true for the “default” intelligent AI system.
But how will they detect the deceptive behavior itself? Will they be on the lookout for deceptive behavior and use clever techniques to detect it?
Either they’ll be on the lookout, or they’ll have some (correct) reason to expect that deception won’t happen.
what made Facebook transition into a company that takes AI safety seriously?
AI capabilities have progressed to the point where researchers think it is plausible that AI systems could actually “intentionally” deceive you. And also we have better arguments for risk. And they are being said by more prestigious people.
(I thought I was more optimistic than average on this point, but here it seems like most commenters are more optimistic than I am.)
If you think that Facebook is unlikely to do a 100x scale-up in one go, suppose that their leadership comes to believe that the scale-up would cause their revenue to increase in expectation by 10%.
You’d have to be really confident in this to not do a 10x less costly 10x scale-up first to see what happens there—I’d be surprised if you could find examples of big companies doing this. OpenAI is perhaps the company that most bets on its beliefs, and it still scaled to 13B parameters before the 175B parameters of GPT-3, even after they had a paper specifically predicting how large language models would scale with more parameters.
Also I’d be surprised if a 100x scale up to be the difference between “subhuman / can’t be deceptive” to “can cause an existential catastrophe”.
The intent with the part you quoted was to point out a way in which I disagree with Ben (though I’m not sure it’s a disagreement rather than a difference in emphasis).
we should expect some well-resourced companies to train competitive goal-directed agents that act in the real world [...] Do you agree that for those systems, the classic arguments about instrumental convergence and the treacherous turn are correct? (If so, I don’t understand how come you agree pretty strongly with Ben; he seems to be skeptical that those arguments can be mapped to contemporary ML methods.)
I don’t expect that companies will train misaligned superintellligent goal-directed agents that act in the real world. (I still think it’s likely enough that we should work on the problem.)
Like, why didn’t the earlier less intelligent versions of the system fail in some non-catastrophic way, or if they did, why didn’t that cause Facebook to think it shouldn’t build superintelligent goal-directed agents / figure out how to build aligned agents?
Ben’s counterarguments are about whether misaligned superintelligent systems arise in the first place, not whether such systems would have instrumental convergence + treacherous turns:
- He thinks it isn’t likely that there will be a sudden jump to extremely powerful and dangerous AI systems, and he thinks we have a much better chance of correcting problems as they come up if capabilities grow gradually.
- He thinks that making AI systems capable and making AI systems have the right goals are likely to go together.
- He thinks that just because there are many ways to create a system that behaves destructively doesn’t mean that the engineering process creating that system is likely to be attracted to those destructive systems; it seems like we are unlikely to accidentally create systems that are destructive enough to end humanity.
[AN #111]: The Circuits hypotheses for deep learning
Yes I definitely feel that “goal stability upon learning/reflection” is a general AGI safety problem, not specifically a corrigibility problem. I bring it up in reference to corrigibility because my impression is that “corrigibility is a broad basin of attraction” / “corrigible agents want to stay corrigible” is supposed to solve that problem, but I don’t think it does.
Interesting, that’s not how I interpret the argument. I usually think of goal stability is something that improves as the agent becomes more intelligent; to the extent that a goal isn’t stable we treat it as a failure of capabilities. Totally possible that this leads to catastrophic outcomes, and seems good to work on if you have a method for it, but it isn’t what I’m usually focused on.
For me, the intuition behind “broad basin of corrigibility” is that if you have an intelligent agent (so among other things, it knows how to keep its goals stable) then if you have a 95% correct definition of corrigibility the resulting agent will help us get to the 100% version.
For these sorts of arguments you have to condition on some amount of intelligence. As a silly extreme example, if you had a toddler surrounded by buttons that jumbled up the toddler’s brain, there’s not much you can do to have the toddler do anything reasonable (autonomously). However, an adult who knows what the buttons do would be able to reliably avoid them.
1) If we have an AGI that is corrigible, it will not randomly drift to be not corrigible, because it will proactively notice and correct potential errors or loss of corrigibility.
2) If we have an AGI that is partly corrigible, it will help us ‘finish out’ the definition of corrigibility / edit itself to be more corrigible, because we want it to be more corrigible and it’s trying to do what we want.
Good point on distinguishing these two arguments. It sounds like we agree on 1. I also thought the OP was talking about 1.
For 2, I don’t think we can make a dimensionality argument (as in the OP), because we’re talking about edits that are the ones that the AI chooses for itself. You can’t apply dimensionality arguments to choices made by intelligent agents (e.g. presumably you wouldn’t argue that every glass in my house must be broken because the vast majority of ways of interacting with glasses breaks them). Or put another way, the structural similarity is just “the AI wouldn’t choose to do <bad thing #N>”, in all cases because it’s intelligent and understands what it’s doing.
Now the question of “how right do we need to get the initial definition of corrigibility” is much less obvious. If you told me we got the definition wrong in a million different ways, I would indeed be worried and probably wouldn’t expect it to self-correct (depending on the meaning of “different”). But like… really? We get it wrong a million different ways? I don’t see why we’d expect that.
Right, so it’s basically goal drift from corrigibility to something else, in this case caused by an incorrect belief that S’s preferences about B are not going to change. I think this is a reasonable thing to be worried about but I don’t see why it’s specific to corrigibility—for any objective, an incorrect belief can prevent you from successfully pursuing that objective.
Like, even if we trained an AI system on the loss function of “make money”, I would still expect it to possibly stop making money if it e.g. decides that it would be more effective at making money if it experience intrinsic joy at its work, and then self-modifies to do that, and then ends up working constantly for no pay.
I’d definitely support the goal of “figure out how to prevent goal drift”, but it doesn’t seem to me to be a reason to be (differentially) pessimistic about corrigibility.
I also tried to give specific examples, see the “friends” example in my other comment
Ah, I hadn’t seen that. I don’t feel convinced, because it assumes that the AI system has a “goal” that isn’t “be corrigible”. Or perhaps the argument is that the goal moves from “be corrigible” to “care for the operator’s friends”? Or maybe that the goal stays as “be corrigible / help the user” but the AI system has a firm unshakeable belief that the user wants her friends to be cared for?
we’ll make powerful systems
But… why can’t I apply the argument to “powerful”, and say that it is extremely unlikely for an AI system to be powerful? Predictive, sure, but powerful?
My model of you responds “powerful is upstream of goal-accomplishing” or “powerful is downstream of goal-directedness which is upstream of goal-accomplishing”, but it seems like you could say that for corrigibility too: “corrigibility is upstream of effectively helping the user”.
As for “why can we hope to solve it”, I can imagine lots of possible solution directions
Thanks, that was convincing (that even under radical uncertainty there are still avenues to pursue).
There’s a huge structural similarity between the proof that ‘1 + 1 != 3’ and ‘1+1 != 4’; like, both are generic instances of the class ‘1 + 1 != n \forall n != 2’. We can increase the number of numbers without decreasing the plausibility of this claim (like, consider it in Z/4, then Z/8, then Z/16, then...).
I feel like that’s exactly my point? Showing that something is a conjunction of a bunch of claims should not always make you think that claim is low probability, because there could be structural similarity between those claims such that a single argument is enough to argue for all of them.
(The claims “If X drifts away from corrigibility along dimension {N}, it will get pulled back” are clearly structurally similar, and the broad basin of corrigibility argument is meant to be an argument that argues for all of them.)
Similarly, if we make an argument that something is an attractor in N-dimensional space, that does actually grow less plausible the more dimensions there are, since there are more ways for the thing to have a derivative that points away from the ‘attractor,’ if we think the dimensions aren’t all symmetric.
1. Why aren’t the dimensions symmetric?
2. I somewhat buy the differential argument (more dimensions ⇒ less plausible) but not the absolute argument (therefore not plausible); this post is arguing for the absolute version:
it starts to feel awfully unlikely that corrigibility is really a broad basin of attraction after all
3. I’m not sure where the idea of a “derivative” is coming from—I thought we were talking about small random edits to the weights of a neural network. If we’re training the network on some objective that doesn’t incentivize corrigibility then certainly it won’t stay corrigible.
I disagree with this position but it does seem consistent. I don’t really know what to say other than “this is a conjunction of a million things” type arguments are not automatically persuasive, e.g. I could argue against “1 + 1 = 2“ by saying that it’s an infinite conjunction of “1 + 1 != 3” AND “1 + 1 != 4” AND … and so it can’t possibly be true.
I’m curious why you think AI risk is worth working on given this extreme cluelessness (both “why is there any risk” and “why can we hope to solve it”).
Planned summary for the Alignment Newsletter:
This post argues that by default, human preferences are strong views built upon poorly defined concepts, that may not have any coherent extrapolation in new situations. To put it another way, humans build mental maps of the world, and their preferences are defined on those maps, and so in new situations where the map no longer reflects the world accurately, it is unclear how preferences should be extended. As a result, anyone interested in preference learning should find some incoherent moral intuition that other people hold, and figure out how to make it coherent, as practice for the case we will face where our own values will be incoherent in the face of new situations.
Planned opinion:
This seems right to me—we can also see this by looking at the various paradoxes found in the philosophy of ethics, which involve taking everyday moral intuitions and finding extreme situations in which they conflict, and it is unclear which moral intuition should “win”.
(This might be true, but my original intent was a reductio ad absurdum—I do not actually think AI systems will be “wandering around”.)
Planned summary for the Alignment Newsletter:
This post summarizes and makes accessible the paper <@Risks from Learned Optimization in Advanced Machine Learning Systems@>.
Biggest difference is that I estimate the risk of this kind of global catastrophe before development of AGI and before 2100 to be much lower—not sure exactly what but 10% seems like the right ballpark. But this did cause me to update towards having more probability on >2100.
It seems to me that researchers entirely unfamiliar with safety could gain the required level of understanding in just 30 minutes of reading
I might have said an hour, but that seem qualitatively right. But that requires them 1. having the motivation to do so and 2. finding and reading exactly the right sources in a field with a thousand blog posts and not much explicit prioritization around them. I think both of these are huge considerations against this condition already being met.
If someone thinks that some safety concerns remain but that we should cautiously move forward on building things that look more and more like AGI, does that count as a “Yes” or a “No”?
Hmm, good question. Probably a Yes? I might try to push on more clear hypotheticals (e.g. a team believes that such-and-such training run would produce AGI, should they do it?) to get a clearer answer.
I was thinking more like
finishing the final phase of training on a fully functional AGI implementation.
RL with a randomly chosen reward leads to catastrophe at optimum.
The proof is for randomly distributed rewards.
Ben’s main critique is that the goals evolve in tandem with capabilities, and goals will be determined by what humans care about. These are specific reasons to deny the conclusion of analysis of random rewards.
(A random Python program will error with near-certainty, yet somehow I still manage to write Python programs that don’t error.)
I do agree that this isn’t enough reason to say “there is no risk”, but it surely is important for determining absolute levels of risk. (See also this comment by Ben.)