Got it, thank you!
Yep—at least that’s how I’m generally thinking about it in this post.
I see. So, restating in my own terms—outer alignment is in fact about whether getting what you asked for is good, but for the case of prediction, the malign universal prior argument says that “perfect” prediction is actually malign. So this would be a case of getting what you wanted / asked for / optimized for, but that not being good. So it is an outer alignment failure.
Whereas an inner alignment failure would necessarily involve not hitting optimal performance at your objective. (Otherwise it would be an inner alignment success, and an outer alignment failure.)
Is that about right?
The way I’m using outer alignment here is to refer to outer alignment at optimum. Under that definition, optimal loss on a predictive objective should require doing something like Bayesian inference on the universal prior, making the question of outer alignment in such a case basically just the question of whether Bayesian inference on the universal prior is aligned.
Great post, thank you!
However, I think I don’t quite understand the distinction between inner alignment and outer alignment, as they’re being used here. In particular, why would the possible malignity of the universal prior be an example of outer alignment rather than inner?
I was thinking of outer alignment as being about whether, if a system achieves its objective, is that what you wanted. Whereas inner alignment was about whether it’s secretly optimizing for something other than the stated objective in the first place.
From that perspective, wouldn’t malignity in the universal prior be a classic example of inner misalignment? You wanted unbiased prediction, and if you got it that would be good (so it’s outer-aligned). But it turns out you got something that looked like a predictor up to a point, and then turned out to be an optimizer (inner misalignment).
Have I misunderstood outer or inner alignment, or what malignity of the universal prior would mean?
(It does now seem more readable to me :) )
Thanks—glad this was helpful for you! And I went through and added some more paragraph breaks—hopefully that helps improve the readability a bit.
But does it ever hallucinate the need to carry the one when it shouldn’t?
Wow. This is fantastic. It helps me so much to see the proposals set out next to each other concisely and straightforwardly, and analysed on the same four key dimensions. I feel like my understanding of the relative strengths and weaknesses is vastly improved by having read this post, and that there has been very little writing of this kind that I’ve been able to find, so thank you, and I’ve curated this post.
My main hesitation with curating the post is that I found it a bit intimidating and dense, and I expect many others will too. That said it does actually do a great job of saying things clearly and simply and not relying on lots of technical jargon it doesn’t explain, and I’m not sure of a clear way to improve it (maybe more paragraph breaks?).
Anyhow, I was so excited when I realised what this post is, thanks again.
Glad you liked the post so much!
I considered trying to make it a living document, but in the end I decided I wasn’t willing to commit to spending a bunch of time updating it regularly. I do like the idea of doing another one every year, though—I think I’d be more willing to write a new version every year than try to maintain one up-to-date version at all times, especially if I had some help.
In terms of other proposals, a lot of the other options I would include in a full list would just be additional combinations of the various things already on the list—recursive reward modeling + intermittent oversight, for example—that I didn’t feel like would add much to cover separately. That being said, there are also just a lot of different people out there with different ideas that I’m sure would have different versions of the proposals I’ve talked about.
Re intermittent oversight—I agree that it’s a problem if suddenly realizes that it should be deceptive. In that case, I would say that even before realizes it should be deceptive, the fact that it will realize that makes it suboptimality deceptively aligned. Thus, to solve this problem, I think we need it to be the case that can catch suboptimality deceptive alignment, which I agree could be quite difficult. One way in which might be able to ensure that it catches suboptimality deceptive alignment, however, could be to verify that is myopic, as a myopic model should never conclude that deception is a good strategy.
Like I wrote before, slow takeoff might be actually worse than fast takeoff. This is because even if the first powerful AIs are aligned, their head start on unaligned AIs will not count as much, and alignment might (and probably will) require overhead that will give the unaligned AIs an advantage. Therefore, success would require that the institutions will either prevent or quickly shut down unaligned AIs for enough time that aligned AIs gain the necessary edge.
I noticed myself mentally grading the entries by some extra criteria. The main ones being something like “taking-over-the-world competitiveness” (TOTWC, or TOW for short) and “would I actually trust this farther than I could throw it, once it’s trying to operate in novel domains?” (WIATTFTICTIOITTOIND, or WIT for short).
A raw statement of my feelings:
Reinforcement learning + transparency tool: High TOW, Very Low WIT.
Imitative amplification + intermittent oversight: Medium TOW, Low WIT.
Imitative amplification + relaxed adversarial training: Medium TOW, Medium-low WIT.
Approval-based amplification + relaxed adversarial training: Medium TOW, Low WIT.
Microscope AI: Very Low TOW, High WIT.
STEM AI: Low TOW, Medium WIT.
Narrow reward modeling + transparency tools: High TOW, Medium WIT.
Recursive reward modeling + relaxed adversarial training: High TOW, Low WIT.
AI safety via debate with transparency tools: Medium-Low TOW, Low WIT.
Amplification with auxiliary RL objective + relaxed adversarial training: Medium TOW, Medium-low WIT.
Amplification alongside RL + relaxed adversarial training: Medium-low TOW, Medium WIT.
This is currently my all-time favorite post; well done, and thank you!
It would be nice for there to be something like this every year, updated with new proposals added and old ones that no longer seem at all viable deleted. And the number of links to further reading would grow as more work is done on each proposal. Building such a thing would be a huge task of course… but boy would it be nice. Maybe it’s something the field as a whole could do, or maybe it could be someone’s paid job to do it. We could point people to it and say “Here is the list of serious proposals; which one do you plan to use? Or do you have a new one to add? Please tell me how it handles inner and outer alignment so I can add it to the list.”
You say this post doesn’t cover all existing proposals—if you were to expand it to cover all (serious) currently existing proposals, how many more entries do you think should be added, roughly?
On the inner alignment problem for amplification with intermittent oversight: Perhaps you’ve already thought of this, but mightn’t there be a sort of cascade of deception, where M at stage t suddenly realizes that its goals diverge from those of the human and resolves to hide this fact, and then when the human asks M(t-1) to examine M(t) and help the human figure out whether M(t) is being deceptive, M(t-1) learns the same revelation and makes the same decision.
As I read I made a diagram of the building blocks and how they were being put together. Seems like transparency tools, relaxed adversarial training, and amplification share first place for prominence and general usefulness.
As abergal wrote, not carrying the “1” can simply mean it does digit-wise addition (which seems trivial via memorization). But notice that just before that quote they also write:
To spot-check whether the model is simply memorizing specific arithmetic problems, we took the 3-digit arithmetic problems in our test set and searched for them in our training data in both the forms “<NUM1> + <NUM2> =” and “<NUM1> plus <NUM2>”. Out of 2,000 addition problems we found only 17 matches (0.8%) and out of 2,000 subtraction problems we found only 2 matches (0.1%), suggesting that only a trivial fraction of the correct answers could have been memorized.
That seems like evidence against memorization, but maybe their simple search failed to find most cases with some relevant training signal, eg: “In this diet you get 350 calories during breakfast: 200 calories from X and 150 calories from Y.”
Yup, thanks, edited
“I usually don’t think about outer amplification as what happens with optimal policies”
Do you mean outer alignment?
Planned summary for the Alignment Newsletter:
The biggest <@GPT-2 model@>(@Better Language Models and Their Implications@) had 1.5 billion parameters, and since its release people have trained language models with up to 17 billion parameters. This paper reports GPT-3 results, where the largest model has _175 billion_ parameters, a 10x increase over the previous largest language model. To get the obvious out of the way, it sets a new state of the art (SOTA) on zero-shot language modeling (evaluated only on Penn Tree Bank, as other evaluation sets were accidentally a part of their training set).
The primary focus of the paper is on analyzing the _few-shot learning_ capabilities of GPT-3. In few-shot learning, after an initial training phase, at test time models are presented with a small number of examples of a new task, and then must execute that task for new inputs. Such problems are usually solved using _meta-learning_ or _finetuning_, e.g. at test time MAML takes a few gradient steps on the new examples to produce a model finetuned for the test task. In contrast, the key hypothesis with GPT-3 is that language is so diverse, that doing well on it already requires adaptation to the input, and so the learned language model will _already be a meta-learner_. This implies that they can simply “”prime”“ the model with examples of a task they care about, and the model can _learn_ what task is supposed to be performed, and then perform that task well.
For example, consider the task of generating a sentence using a new-made up word whose meaning has been explained. In one notable example, the prompt for GPT-3 is:
_A “”whatpu”” is a small, furry animal native to Tanzania. An example of a sentence that uses the word whatpu is:_
_We were traveling in Africa and we saw these very cute whatpus._
_To do a “”farduddle”″ means to jump up and down really fast. An example of a sentence that uses the word farduddle is:_
Given this prompt, GPT-3 generates the following output:
_One day when I was playing tag with my little sister, she got really excited and she
started doing these crazy farduddles._
The paper tests on several downstream tasks for which benchmarks exist (e.g. question answering), and reports zero-shot, one-shot, and few-shot performance on all of them. On some tasks, the few-shot version sets a new SOTA, _despite not being finetuned using the benchmark’s training set_; on others, GPT-3 lags considerably behind finetuning approaches.
The paper also consistently shows that few-shot performance increases as the number of parameters increase, and the rate of increase is faster than the corresponding rate for zero-shot performance. While they don’t outright say it, we might take this as suggestive evidence that as models get larger, they are more incentivized to learn “general reasoning abilities”.
The most striking example of this is in arithmetic, where the smallest 6 models (up to 6.7 billion parameters) have poor performance (< 20% on 2-digit addition), then the next model (13 billion parameters) jumps to > 50% on 2-digit addition and subtraction, and the final model (175 billion parameters) achieves > 80% on 3-digit addition and subtraction and a perfect 100% on 2-digit addition (all in the few-shot regime). They explicitly look for their test problems in the training set, and find very few examples, suggesting that the model really is learning “how to do addition”; in addition, when it is incorrect, it tends to be mistakes like “forgetting to carry a 1”.
On broader impacts, the authors talk about potential misuse, fairness and bias concerns, and energy usage concerns, and say about what you’d expect. One interesting note: “To understand how low and mid-skill actors think about language models, we have been monitoring forums and chat groups where misinformation tactics, malware distribution, and computer fraud are frequently discussed.” They find that while there was significant discussion of misuse, they found no successful deployments. They also consulted with professional threat analysts about the possibility of well-resourced actors misusing the model. According to the paper: “The assessment was that language models may not be worth investing significant resources in because there has been no convincing demonstration that current language models are significantly better than current methods for generating text, and because methods for “targeting” or “controlling” the content of language models are still at a very early stage.”Planned opinion:
For a long time, I’ve heard people quietly hypothesizing that with a sufficient diversity of tasks, regular gradient descent could lead to general reasoning abilities allowing for quick adaptation to new tasks. This is a powerful demonstration of this hypothesis.
One critique is that GPT-3 still takes far too long to “identify” a task—why does it need 50 examples of addition in order to figure out that what it should do is addition? Why isn’t 1 sufficient? It’s not like there are a bunch of other conceptions of “addition” that need to be disambiguated. I’m not sure what’s going on mechanistically, but we can infer from the paper that as language models get larger, the number of examples needed to achieve a given level of performance goes down, so it seems like there is some “strength” of general reasoning ability that goes up. Still, it would be really interesting to figure out mechanistically how the model is “reasoning”.
This also provides some empirical evidence in support of the threat model underlying <@inner alignment concerns@>(@Risks from Learned Optimization in Advanced Machine Learning Systems@): they are predicated on neural nets that implicitly learn to optimize. (To be clear, I think it provides empirical support for neural nets learning to “reason generally”, not neural nets learning to implicitly “perform search” in pursuit of a “mesa objective”—see also <@Is the term mesa optimizer too narrow?@>.
Planned summary for the Alignment Newsletter:
This post describes eleven “full” AI alignment proposals (where the goal is to build a powerful, beneficial Ai system using current techniques), and evaluates them on four axes:
1. **Outer alignment:** Would the optimal policy for the specified loss function be aligned with us? See also this post .
2. **Inner alignment:** Will the model that is _actually produced_ by the training process be aligned with us?
3. **Training competitiveness:** Is this an efficient way to train a powerful AI system? More concretely, if one team had a “reasonable lead” over other teams, would they keep at least some of the lead if they used this algorithm?
4. **Performance competitiveness:** Will the trained model have good performance (relative to other models that could be trained)?
Seven of the eleven proposals are of the form “recursive outer alignment technique” plus “<@technique for robustness@>(@Worst-case guarantees (Revisited)@)”. The recursive outer alignment technique is either <@debate@>(@AI safety via debate@), <@recursive reward modeling@>(@Scalable agent alignment via reward modeling@), or some flavor of <@amplification@>(@Capability amplification@). The technique for robustness is either transparency tools to “peer inside the model”, <@relaxed adversarial training@>(@Relaxed adversarial training for inner alignment@), or intermittent oversight by a competent supervisor. An additional two proposals are of the form “non-recursive outer alignment technique” plus “technique for robustness”—the non-recursive techniques are vanilla reinforcement learning in a multiagent environment, and narrow reward learning.
Another proposal is Microscope AI, in which we train AI systems to simply understand vast quantities of data, and then by peering into the AI system we can learn the insights that the AI system learned, leading to a lot of value. We wouldn’t have the AI system act in the world, thus eliminating a large swath of potential bad outcomes. Finally, we have STEM AI, where we try to build an AI system that operates in a sandbox and is very good at science and engineering, but doesn’t know much about humans. Intuitively, such a system would be very unlikely to deceive us (and probably would be incapable of doing so).
The post contains a lot of additional content that I didn’t do justice to in this summary. In particular, I’ve said nothing about the analysis of each of these proposals on the four axes listed above; the full post talks about all 44 combinations.Planned opinion:
I’m glad this post exists: while most of the specific proposals could be found by patching together content spread across other blog posts, there was a severe lack of a single article laying out a full picture for even one proposal, let alone all eleven in this post.
I usually don’t think about outer alignment as what happens with optimal policies, as assumed in this post—when you’re talking about loss functions _in the real world_ (as I think this post is trying to do), _optimal_ behavior can be weird and unintuitive, in ways that may not actually matter. For example, arguably for any loss function, the optimal policy is to hack the loss function so that it always outputs zero (or perhaps negative infinity).
Thanks! This is great.
Thanks for writing this. I’ve been thinking along similar lines since the pandemic started. Another takeaway for me: Under our current political system, AI risk will become politicized. It will be very easy for unaligned or otherwise dangerous AI to find human “allies” who will help to prevent effective social response. Given this, “more competent institutions” has to include large-scale and highly effective reforms to our democratic political structures, but political dysfunction is such a well-known problem (i.e., not particularly neglected) that if there were easy fixes, they would have been found and applied already.
So whereas you’re careful to condition your pessimism on “unless our institutions improve”, I’m just pessimistic. (To clarify, I was already pessimistic before COVID-19, so it just provided more details about how coordination/institutions are likely to fail, which I didn’t have a clear picture of. I’m curious if COVID-19 was an update for you as far as your overall assessment of AI risk. That wasn’t totally clear from the post.)
On a related note, I recall Paul said the risk from failure of AI alignment (I think he said or meant “intent alignment”) is 10%; Toby Ord gave a similar number for AI risk in his recent book; 80,000 Hours, based on interviews with multiple AI risk researchers, said “We estimate that the risk of a serious catastrophe caused by machine intelligence within the next 100 years is between 1 and 10%.” Until now 1-10% seems to have been the consensus view among the most prominent AI risk researchers. I wonder if that has changed due to recent events.