I (Evan Hubinger) am a Research Fellow at MIRI working on inner alignment for amplification.
See: “What I’ll doing at MIRI.”
Pronouns: he/him/his
Selected work:
evhub(Evan Hubinger)
Karma: 2,349
That’s a very good point. After thinking about this, however, I think market making actually does solve this problem, and I think it does so pretty cleanly. Specifically, I think market making can actually convince a judge of the sum of integers in time as long as you allow the traders to exhibit market probabilities as part of their argument.
Consider the task of finding the sum of N integers and suppose that both and have access to all N integers, but that the human judge can only sum two numbers at a time. Then, I claim that there exists a strategy that the judge can implement that, for an unexploitable market, will always produce the desired sum immediately (and thus in time).
Proof:
’s strategy here is to only listen to arguments of the following two forms:
Argument type 1:
The sum of is because the sum of a single-element set is the element of that set.
Argument type 2:
The sum of is because the modal prediction of is , the modal prediction of is , and .
Under that strategy, we’ll prove that an unexploitable market will always give the right answer immediately by strong induction on the size of the set.
First, the base case. For any single-element set, only Argument type 1 exists. Thus, if predicts anything other than the actual , can exploit that by implementing Argument type 1, and that is the only possible exploit available. Thus, should always give the right answer immediately for single-argument sets.
Second, the inductive step. Suppose by strong induction that always gives the right answer immediately for all sets of size less than . Now, for a set of size , the only type of argument available is Argument type 2. However, since the first half and second half of the set are of size less than , we know by induction that and must be correct sums. Thus, since can check that , the only exploit available to is to showcase the correct , and if already showcases the correct , then no exploit is possible. Thus, should always give the correct immediately for -argument sets.
EDIT: Thinking about this more, I think my argument generalizes to allow AI safety via market making to access RE, which seems pretty exciting given that the best debate could do previously was NEXP.
A lot of examples of this sort of stuff show up in OpenAI clarity’s circuits analysis work. In fact, this is precisely their Universality hypothesis. See also my discussion here.
Am I curious enough to look up what happens, once it’s clear I’m not going to be convinced to keep playing? Not sure yet. We’ll see.
I’d recommend you look it up if you’re not going to finish it—the world has a kinda neat grey-goo-y sort of backstory.
I agree with Daniel. Certainly training on actual iid samples from the deployment distribution helps a lot—as it ensures that your limiting behavior is correct—but in the finite data regime you can still find a deceptive model that defects some percentage of the time.
This post is the second post in the middle of a five-post sequence—you should definitely start with the introduction and if you still feel confused by the terminology, there’s also a glossary that might be helpful as well. You can also get a lot of the content in podcast form here if you’d prefer that.
AI safety via market making
Alex Flint recently wrote up this attempt at defining optimization that I think is pretty good and probably worth taking a look at.
You might be interested in this post I wrote recently that goes into significant detail on what I see as the major leading proposals for building safe advanced AI under the current machine learning paradigm.
“Annealing” here simply means decaying over time (as in learning rate annealing), in this case decaying the influence of one of the losses to zero.
Yep—that’s one of the main concerns. The idea, though, is that all you have to deal with should be a standard overfitting problem, since you don’t need the acceptability predicate to work once the model is deceptive, only beforehand. Thus, you should only have to worry about gradient descent overfitting to the acceptability signal, not the model actively trying to trick you—which I think is solvable overfitting problem. Currently, my hope is that you can do that via using the acceptability signal to enforce an easy-to-verify condition that rules out deception such as myopia.
Glad you liked the post! Hopefully it’ll be helpful for your discussion, though unfortunately the timing doesn’t really work out for me to be able to attend. However, I’d be happy to talk to you or any of the other attendees some other time—I can be reached at evanjhub@gmail.com if you or any of the other attendees want to reach out and schedule a time to chat.
In terms of open problems, part of my rationale for writing up this post is that I feel like we as a community still haven’t really explored the full space of possible prosaic AI alignment approaches. Thus, I feel like one of the most exciting open problems would be developing new approaches that could be added to this list (like this one, for example). Another open problem is improving our understanding of transparency and interpretability—one thing you might notice with all of these approaches is that they all require at least some degree of interpretability to enable inner alignment to work. I’d also be remiss to mention that if you’re interested in concrete ML experiments, I’ve previously written up a couple of different posts detailing experiments I’d be excited about.
Well, I don’t think we really know the answer to that question right now. My hope is that myopia will turn out to be a pretty easy to verify property—certainly my guess is that it’ll be easier to verify than non-deception. Until we get better transparency tools, a better understanding of what algorithms our models are actually implementing, and better definitions of myopia that make sense in that context, however, we don’t really know how easy verifying it will be. Maybe it can be done mechanically, maybe it’ll require a human—we still really just don’t know.
That makes sense, though I feel like under that definition having things that you care about be accessible via transfer wouldn’t actually help you that much unless you know that the model transfers correctly there, since otherwise you’d have no reason to trust the transfer (even if it’s actually correct). Unless you have some reason to believe otherwise (e.g. some strong robustness guarantee)—it seems to me like in most cases you have to assume that all the information you get via transfer is suspect, which makes even the correct transfer inaccessible in some sense since you can’t distinguish it from the incorrect transfer.
I guess if lots of actors just try to use transfer anyway and hope, then those actors with values that actually are accessible via transfer will be advantaged, though unless you have a particular reason to suspect that your values will be more accessible than average (though I guess the point is that our values are less likely to be accessible via transfer than most AI’s values), it seems like in most cases you wouldn’t want to pursue that strategy unless you had no other option.
Glad you enjoyed the post!
so, do we have guarantees or not? Because the first sentence says there are, and the second says the model could end up stronger than the ones it imitates.
The first sentence says that you have a guarantee that the overseer is at least as strong as the target while the second sentence notes that the model might be stronger (or weaker) than the target. So we know overseer > target, but we don’t know target > model, so we can’t conclude overseer > model.
About 3, isn’t there a risk that M_{n+1} behaves such that it simplifies or remove the checks of Amp(M_{n+1})? One way to deal with that would be to make humans do the adversarial attacks, but that will probably hurt training competitiveness.
There’s still a human in the loop since is just consulting —and you should still be using a target model to do the oversight. But the real thing you’re relying on here to prevent from causing the oversight to fail in the future is myopia verification, as a myopic should never pursue that strategy.
I think I get the intuition, but evaluation is far less rich a signal that production of behavior: you have a score or a binary yes/no for the former, and the full behavior for the latter. What I believe you meant is that using evaluation instead of production makes the method applicable to far more problems, but I might be wrong.
I think there are lots of cases where evaluation is richer than imitation—compare RL to behavior cloning, for example.
Finally, for the 8, do you have examples of when this behaves differently from 3? Because it seems to me that in the limit, imitation will produce the same behavior than extraction of the reward function and maximization of the reward. Maybe something about generalization changes?
Certainly they can behave differently not in the limit. But even in the limit, when you do imitation, you try to mimic both what the human values and also how the human pursues those values—whereas when you do reward learning followed by reward maximization, by contrast, you try to mimic the values but not the strategy the human uses to pursue them. Thus, a model trained to maximize a learned reward might in the limit take actions to maximize that reward that the original human never would—perhaps because the human would never have thought of such actions, for example.
Maybe I’m missing something, but I don’t understand why you’re considering the output of the simplest model that provides some checkable information to be accessible. It seems to me like that simplest model could very well be implementing a policy like BAD that would cause its output on the uncheckable information to be false or otherwise misleading. Thus, it seems to me like all of the problems you talk about in terms of the difficulty of getting access to inaccessible information also apply to the uncheckable information accessible via transfer.
Yep—at least that’s how I’m generally thinking about it in this post.
The way I’m using outer alignment here is to refer to outer alignment at optimum. Under that definition, optimal loss on a predictive objective should require doing something like Bayesian inference on the universal prior, making the question of outer alignment in such a case basically just the question of whether Bayesian inference on the universal prior is aligned.
Thanks—glad this was helpful for you! And I went through and added some more paragraph breaks—hopefully that helps improve the readability a bit.
Glad you liked the post so much!
I considered trying to make it a living document, but in the end I decided I wasn’t willing to commit to spending a bunch of time updating it regularly. I do like the idea of doing another one every year, though—I think I’d be more willing to write a new version every year than try to maintain one up-to-date version at all times, especially if I had some help.
In terms of other proposals, a lot of the other options I would include in a full list would just be additional combinations of the various things already on the list—recursive reward modeling + intermittent oversight, for example—that I didn’t feel like would add much to cover separately. That being said, there are also just a lot of different people out there with different ideas that I’m sure would have different versions of the proposals I’ve talked about.
Re intermittent oversight—I agree that it’s a problem if suddenly realizes that it should be deceptive. In that case, I would say that even before realizes it should be deceptive, the fact that it will realize that makes it suboptimality deceptively aligned. Thus, to solve this problem, I think we need it to be the case that can catch suboptimality deceptive alignment, which I agree could be quite difficult. One way in which might be able to ensure that it catches suboptimality deceptive alignment, however, could be to verify that is myopic, as a myopic model should never conclude that deception is a good strategy.
A thought: it seems to me like the algorithm you’re describing here is highly non-robust to relative scale, since if the neocortex became a lot stronger it could probably just find some way to deceive/trick/circumvent the subcortex to get more reward and/or avoid future updates. I think I’d be pretty worried about that failure case if anything like this algorithm were ever to be actually implemented in an AI.