Software engineer since 2010. Left Google in fall ’21, now getting into independent AI alignment research. Currently working on interpretability.
I’m always happy to connect with other researchers or people interested in AI alignment and effective altruism. Feel free to send me a private message.
Nice post.
Therefore, either we can try to revise the framework slightly, essentially omitting the notions of robust alignment and ‘internalization of the base objective’ and focussing more on revised versions of ‘proxy alignment’ and ‘approximate alignment’ as descriptors of what is essentially the best possible situation in terms of alignment.
Have you seen Hubinger’s more recent post, More variations on pseudo-alignment ? It amends the list of pseudo-alignment types originally listed in “Risks of Learned Optimization” to include a couple more.
Your claim above that the best we could hope for may be a form of proxy alignment or approximate alignment reminds me of following pseudo-alignment type he introduced in that more recent post. In the description of this type, he also seems to agree with you that robust alignment is very difficult or “unstable” (though perhaps you go further in saying its impossible):
Corrigible pseudo-alignment. In the paper, we defined corrigible alignment as the situation in which “the base objective is incorporated into the mesa-optimizer’s epistemic model and [the mesa-optimizer’s] objective is modified to ‘point to’ that information.” We mostly just talked about this as a form of robust alignment—however, as I note in “Towards a mechanistic understanding of corrigibility,” this is a very unstable operation, requiring you to get your pointer just right. Thus, I think it’s better to talk about corrigible alignment as the class of possible relationships between the base and mesa-objectives defined by the model having some sort of pointer to the base objective, including both corrigible robust alignment (if the pointer is robust) and corrigible pseudo-alignment (if the pointer is to some sort of non-robust proxy). In particular, I think this distinction is fairly important to why deceptive alignment might be more likely than robust alignment, as it points at why robust alignment via corrigibility might be quite difficult (which is a point we made in the paper, but one which I think is made much clearer with this distinction).
Interesting post. I agree that a myopic oracle is a promising idea. In particular if it were used to accelerate AGI alignment research. (It could be dangerous if used in many other ways.)
> The major shortcoming of a myopic Oracle is that its long-term plans cannot be expected to work reliably.
I don’t think this is necessarily true. It depends on the sort of myopia which the oracle is imbued with. If it has a myopia where it is incapable of thinking about the long-term future, then this is correct.
But it could have a different kind of myopia where it is very capable of thinking about the long-term future, but its goal is still to maximize its short term reward. In this case, it could be incentivized to go to the trouble of creating a good long-term plan if that meant it was more likely to get the short-term reward.
If we had such a nice form of myopia, the best formats I’ve seen so far for creating such an oracle with it are debate (which you referenced) and market making. Market making is particularly interesting because it is compatible with per-step myopia, which seems like it will be easier to enforce than per-episode myopia (what debate seems to require).
I don’t think there is any known way to verify/enforce any of these kinds of myopia yet though. I still need to read your post about reverse-intent alignment to better understand that idea.
Good question! I don’t know the overall answer but I do have a piece of context on this:
(Update: I read your post more closely and realize that while your post is titled as a question, you weren’t literally asking for others to answer it and you do a pretty good job of answering it yourself. Nice analysis :)
Bridge Base Online is one of the leading (probably the leading) online bridge platforms. It is very common on Bridge Base to play with robots (i.e. automated bridge-playing bots, no robotics involved). My dad is an avid bridge player on there. I think he plays with bots the majority of the time just out of convenience. Of course because of the way duplicate bridge is designed, everyone else at your table can be a robot and you still get to play against other humans who are playing identical hands/boards with identical bots.
I get the sense that the robots on Bridge Base are not advanced AIs. They may even just be hand-coded bots operating totally based on rules and randomness with no machine learning involved. So it’s not like people on Bridge Base are playing with AlphaGo-level bridge robots. Such robots may exist but they’re not what this website uses, at least not for its everyday bridge bots.
(I first saw Alex Turner (TurnTrout) express this concern
This link is broken now but I think I found an updated one that works:
Thanks Charlie, Evan H. and Oliver. Your comments definitely help to give me a clearer picture.
Related:
World Building Contest: “The Future of Life Institute is welcoming entries from teams across the globe, to compete for a prize purse of up to $100,000 by designing visions of a plausible, aspirational future that includes strong artificial intelligence.” (Note: The deadline was Apr 15, 2022 and so has passed, but it will be exciting to see the results once they’re ready.)
Overall I think this is a good post and very interesting, thanks.
I find this somewhat compelling, but less so than I used to, since I’ve realized that the line between imitation learning and reinforcement learning is blurrier than I used to think (e.g. see this or this).
So I checked out those links. Briefly looking at them, I can see what you mean about the line between RL and imitation learning being blurry. The first paper seems to show a version of RL which is basically imitation learning.
I’m confused because when you said this makes iterated amplification less compelling to you, I took that to mean it made you less optimistic about iterated amplification as a solution for alignment. But why would whether something is technically classified as imitation learning or a special kind of RL make a difference for its effectiveness?
Or did you mean not that you find it any less promising as an alignment proposal, but just that you now find the core insight less compelling/interesting because it’s not as major an innovation over the idea of RL as you had thought it was?
Maybe, but couldn’t it also mean that we just haven’t reached the threshold yet? Some period of AIHHAI might be a necessary step or a catalyst toward that threshold. Encountering AIHHAI doesn’t imply that there is no such foom threshold, it could also mean that we just haven’t reached the threshold yet.
That’s a good question. Perhaps it does make use of optimization but the model still has an overall passive relationship to the world compared to an active mesa-optimizer AI. I’m thinking about the difference between say, GPT-3 and the classic paperclip maximizer or other tiling AI.
This is just my medium-confidence understanding and may be different from what Evan Hubinger meant in that quote.
I believe it would look like Microscope AI.
Is imitative generalization usually envisioned as a recursive many-iterations-process like IDA? Or is it just a single iteration of train the initial model → inspect and correct the priors → train the new model?
Great post, by the way.
I’m not familiar with anything like this kind of freedom discussed for an AGI. I think it would have to be designed in, or at least one would have to ensure that the AI does not evolve into a strong optimization process (i.e. a mesa-optimizer) in order to start to explore something like this.
On the human side, like your example about people wanting to change some of their desires. I have seen that explored some, e.g. by Stuart Armstrong (searching this post for “meta-value” will turn up some of the discussion on it). This is in the context of an AI being able to thoroughly human values, that it would need to account for meta-values like this.
This strikes me as a really interesting and innovative post, proposing a framework for systematically categorizing existing alignment proposals as well as helping to generate new ones.
I’m kind of surprised that this post is almost 2 years old and yet only has one pingback and a few comments.
Is there some other framework which has superseded this one, or did people just forget about it / there isn’t much comparative alignment work going on?
One other framework I’ve seen kind of like this is “Training stories” from Evan Hubinger’s How do we become confident in the safety of a machine learning system?. But that is more about evaluating alignment proposals (i.e. the very last part of the present post) rather than categorizing alignment proposals along a consistent set of dimensions, which is the main focus here. So it actually serves a different purpose and isn’t much like this framework after all.
Can you help me understand why chip speeds are a bottleneck? Why couldn’t an AI continue to advance by harnessing increased parallelized cloud compute?
Indeed! :)
I just sent Baobao an email to see if she has any information about when the results might be available.
It’s been a couple years since this post was published. Is there a way to get notifications when your post receives a new mention/pingback yet?
If not, then strong upvote for this feature!