Research Scientist at DeepMind. Creator of the Alignment Newsletter. http://rohinshah.com/
rohinmshah(Rohin Shah)
Karma: 8,986
I agree that regulation is harder to do before you know all the details of the technology, but it doesn’t seem obviously doomed, and it seems especially-not-doomed to productively think about what regulations would be good (which is the vast majority of current AI governance work by longtermists).
As a canonical example I’d think of the Asilomar conference, which I think happened well before the details of the technology were known. There are a few more examples, but overall not many. I think that’s primarily because we don’t usually try to foresee problems because we’re too caught up in current problems, so I don’t see that as a very strong update against thinking about governance in advance.
Sure, but don’t you agree that it’s a very confusing use of the term?
Maybe? Idk, according to me the goal of alignment is “create a model that is motivated to help us”, and so misalignment = not-alignment = “the mode is not motivated to help us”. Feels pretty clear to me but illusion of transparency is a thing.
I am making a claim that for the purposes of alignment of capable systems, you do want to talk about “motivation”. So to the extent GPT-N / Codex-N doesn’t have a motivation, but is existentially risky, I’m claiming that you want to give it a motivation. I wouldn’t say this with high confidence but it is my best guess for now.
(Also the fact that gwern, which ascribe agency to GPT-3, quoted specifically this part in his comment is another evidence that you’re implying agency for different people)
I think Gwern is using “agent” in a different way than you are ¯\_(ツ)_/¯
I don’t think Gwern and I would differ much in our predictions about what GPT-3 is going to do in new circumstances. (He’d probably be more specific than me just because he’s worked with it a lot more than I have.)
Maybe the real reason it feels weird for me to call this behavior of Codex misalignment is that it is so obvious?
It doesn’t seem like whether something is obvious or not should determine whether it is misaligned—it’s obvious that a very superintelligent paperclip maximizer would be bad, but clearly we should still call that misaligned.
Almost all specification gaming examples are subtle, or tricky, or exploiting bugs.
I think that’s primarily to emphasize why it is difficult to avoid specification gaming, not because those are the only examples of misalignment.
Planned summary for the Alignment Newsletter:
Under a [longtermist](https://forum.effectivealtruism.org/tag/longtermism) lens, one problem to worry about is that even after building AI systems, humans will spend more time competing with each other rather than figuring out what they want, which may then lead to their values changing in an undesirable way. For example, we may have powerful persuasion technology that everyone uses to persuade people to their line of thinking; it seems bad if humanity’s values are determined by a mix of effective persuasion tools, especially if persuasion significantly diverges from truth-seeking.
One solution to this is to coordinate to _pause_ competition while we deliberate on what we want. However, this seems rather hard to implement. Instead, we can at least try to _decouple_ competition from deliberation, by having AI systems acquire <@flexible influence@>(@The strategy-stealing assumption@) on our behalf (competition), and having humans separately thinking about what they want (deliberation). As long as the AI systems are competent enough to shield the humans from the competition, the results of the deliberation shouldn’t depend too much on competition, thus achieving the desired decoupling.
The post has a bunch of additional concrete details on what could go wrong with such a plan that I won’t get into here.
Like Luke I’m going to take longtermism as an axiom for most purposes (I find it decently convincing given my values), though if you’re interested in debating it you could post on the EA Forum. (One note: my understanding of longtermism is “the primary determinant of whether an action is one of the best that you can take is its consequences on the far future”; you seem to be interpreting it as a stronger / more specific claim than that.)
Also, focusing on AI governance is a bit of a strange way to influence AI safety
You’re misunderstanding the point of AI governance. AI governance isn’t a subset of AI safety, unless you interpret the term “AI safety” very very broadly. Usually I think of AI safety as “how do we build AI systems that do what their designers intend”; AI governance is then “how do we organize society so that humanity uses this newfound power of AI for good, and in particular doesn’t use it to destroy ourselves” (e.g. how do we prevent humans from using AI in a way that makes wars existentially risky, that enforces robust totalitarianism, that persuades humans to change their values, etc). I guess part of governance is “how do we make sure no one builds unsafe AI”, which is somewhat related to AI safety, but that’s not the majority of AI governance.
A lot of these issues don’t seem to become that more clarified even with a picture of how AGI will come about, e.g. I have such a picture in mind, and even if I condition on that picture being completely accurate (which it obviously won’t be), many of the relevant questions still don’t get resolved. This is because often they’re primarily questions about human society rather than questions about how AI works.
Right now it looks like your 19-21 scale corresponds to something like log(# of parameters) (basically every scaling graph with parameters you’ll see uses this as its x-axis). So it still requires exponential increase in inputs to drive a linear increase on that scale.
I think that this is a very good example where the paper (based on your summary) and your opinion assumes some sort of higher agency/goals in GPT-3 than what I feel we have evidence for.
Where do you see any assumption of agency/goals?
(I find this some combination of sad and amusing as a commentary on the difficulty of communication, in that I feel like I tend to be the person pushing against ascribing goals to GPT.)
Maybe you’re objecting to the “motivated” part of that sentence? But I was saying that it isn’t motivated to help us, not that it is motivated to do something else.
Maybe you’re objecting to words like “know” and “capable”? But those don’t seem to imply agency/goals; it seems reasonable to say that Google Maps knows about traffic patterns and is capable of predicting route times.
As an aside, this was Codex rather than GPT-3, though I’d say the same thing for both.
For a simulator-like model, this is not misalignment, this is intended behavior. It is trained to find the most probable continuation, not to analyze what you meant and solve your problem.
I don’t care what it is trained for; I care whether it solves my problem. Are you telling me that you wouldn’t count any of the reward misspecification examples as misalignment? After all, those agents were trained to optimize the reward, not to analyze what you meant and fix your reward.
You can obviously say “it’s an agent that does really care about the context”, but I doesn’t look like it adds anything to the picture,
Agreed, which is why I didn’t say anything like that?
[AN #157]: Measuring misalignment in the technology underlying Copilot
Are you the historical origin of the robustness-centric approach?
Idk, probably? It’s always hard for me to tell; so much of what I do is just read what other people say and make the ideas sound sane to me. But stuff I’ve done that’s relevant:
Talk at CHAI saying something like “daemons are just distributional shift” in August 2018, I think. (I remember Scott attending it.)
Talk at FHI in February 2020 that emphasized a risk model where objectives generalize but capabilities don’t.
Talk at SERI conference a few months ago that explicitly argued for a focus on generalization over objectives.
Especially relevant stuff other people have done that has influenced me:
Two guarantees (arguably this should be thought of as the origin)
(My views were pretty set by the time Evan wrote the clarifying inner alignment terminology post; it’s possible that his version that’s closer to generalization-focused was inspired by things I said, you’d have to ask him.)
Planned summary for the Alignment Newsletter:
Many of the problems we care about (reward gaming, wireheading, manipulation) are fundamentally a worry that our AI systems will have the _wrong incentives_. Thus, we need Causal Influence Diagrams (CIDs): a formal theory of incentives. These are <@graphical models@>(@Understanding Agent Incentives with Causal Influence Diagrams@) in which there are action nodes (which the agent controls) and utility nodes (which determine what the agent wants). Once such a model is specified, we can talk about various incentives the agent has. This can then be used for several applications:
1. We can analyze [what happens](https://arxiv.org/abs/2102.07716) when you [intervene](https://arxiv.org/abs/1707.05173) on the agent’s action. Depending on whether the RL algorithm uses the original or modified action in its update rule, we may or may not see the algorithm disable its off switch.
2. We can <@avoid reward tampering@>(@Designing agent incentives to avoid reward tampering@) by removing the connections from future rewards to utility nodes; in other words, we ensure that the agent evaluates hypothetical future outcomes according to its _current_ reward function.
3. A [multiagent version](https://arxiv.org/abs/2102.05008) allows us to recover concepts like Nash equilibria and subgames from game theory, using a very simple, compact representation.
(Meta: was this meant to be a question?)
In contrast, the generalization-focused approach puts less emphasis on the assumption that the worst catastrophes are intentional.
I don’t think this is actually a con of the generalization-focused approach. From the post you link, one of the two questions in that approach (the one focused on robustness) is:
How do we ensure the model generalizes acceptably out of distribution?
Part of the problem is to come up with a good definition of “acceptable”, such that this is actually possible to achieve. (See e.g. the “Defining acceptable” section of this post, or the beginning of this post.) But if you prefer to bake in the notion of intent, you could make the second question
How do we ensure the model continues to try to help us when out of distribution?
what is the measure of data-sets in the N-datapoint hypercube such that the trained model is aligned?”, perhaps also weighting by ease of specification in some sense.
You’re going to need the ease of specification condition, or something similar; else you’ll probably run into no-free-lunch considerations (at which point I think you’ve stopped talking about anything useful).
The fourth bullet point claims that GPT-N will go on filling in missing words rather than doing a treacherous turn.
?? I said nothing about a treacherous turn? And where did I say it would go on filling in missing words?
EDIT: Ah, you mean the fourth bullet point in ESRogs response. I was thinking of that as one example of how such reasoning could go wrong, as opposed to the only case. So in that case the model_1 predicts a treacherous turn confidently, but this is the wrong epistemic state to be in because it is also plausible that it just “fills in words” instead.
Seems to me the conclusion of this argument is that “In general it’s not true that the AI is trying to achieve its training objective.”
Isn’t that effectively what I said? (I was trying to be more precise since “achieve its training objective” is ambiguous, but given what I understand you to mean by that phrase, I think it’s what I said?)
we have no idea what it’ll do; treacherous turn is a real possibility because that’s what’ll happen for most goals it could have, and it may have a goal for all we know.
This seems reasonable to me (and seems compatible with what I said)
Yeah, I agree with all this. I still think the pretraining objective basically doesn’t matter for alignment (beyond being “reasonable”) but I don’t think the argument I’ve given establishes that.
I do think the arguments in support of Claim 2 are sufficient to at least raise Claim 3 to attention (and thus Claim 4 as well).
Planned summary:
One [methodology](https://www.overcomingbias.com/2012/08/ai-progress-estimate.html) for forecasting AI timelines is to ask experts how much progress they have made to human-level AI within their subfield over the last T years. You can then extrapolate linearly to see when 100% of the problem will be solved. The post linked above collects such estimates, with a typical estimate being 5% of a problem being solved in the twenty year period between 1992 and 2012. Overall these estimates imply a timeline of [372 years](https://aiimpacts.org/surveys-on-fractional-progress-towards-hlai/).
This post provides a reductio argument against this pair of methodology and estimate. The core argument is that if you linearly extrapolate, then you are effectively saying “assume that business continues as usual: then how long does it take”? But “business as usual” in the case of the last 20 years involves an increase in the amount of compute used by AI researchers by a factor of ~1000, so this effectively says that we’ll get to human-level AI after a 1000^{372/20} = 10^56 increase in the amount of available compute. (The authors do a somewhat more careful calculation that breaks apart improvements in price and growth of GDP, and get 10^53.)
This is a stupendously large amount of compute: it far dwarfs the amount of compute used by evolution, and even dwarfs the maximum amount of irreversible computing we could have done with all the energy that has ever hit the Earth over its lifetime (the bound comes from [Landauer’s principle](https://en.wikipedia.org/wiki/Landauer%27s_principle)).
Given that evolution _did_ produce intelligence (us), we should reject the argument. But what should we make of the expert estimates then? One interpretation is that “proportion of the problem solved” behaves more like an exponential, because the inputs are growing exponentially, and so the time taken to do the last 90% can be much less than 9x the time taken for the first 10%.
Planned opinion:
This seems like a pretty clear reductio to me, though it is possible to argue that this argument doesn’t apply because compute isn’t the bottleneck, i.e. even with infinite compute we wouldn’t know how to make AGI. (That being said, I mostly do think we could build AGI if only we had enough compute; see also <@last week’s highlight on the scaling hypothesis@>(@The Scaling Hypothesis@).)
Yes, that’s right, sorry about the confusion.
Wrote a separate comment here (in particular I think claims 1 and 4 are directly relevant to safety)
This comment is inspired by a conversation with Ajeya Cotra.
As a simple example of how the scaling hypothesis affects AI safety research, it suggests that the training objective (“predict the next word”) is relatively unimportant in determining properties of the trained agent; in contrast, the dataset is much more important. This suggests that analyses based on the “reward function used to train the agent” are probably not going to be very predictive of the systems we actually build.
To elaborate on this more:
Claim 1: Scaling hypothesis + abundance of data + competitiveness requirement implies that an alignment solution will need to involve pretraining.
Argument: The scaling hypothesis implies that you can get strong capabilities out of abundant effectively-free data. So, if you want your alignment proposal to be competitive, it must also get strong capabilities out of effectively-free data. So far, the only method we know of for this is pretraining.
Note that you could have schemes where you train an actor model using a reward model that is always aligned; in this case your actor model could avoid pretraining (since you can generate effectively-free data from the reward model) but your reward model will need to be pretrained. So the claim is that some part of your scheme involves pretraining; it doesn’t have to be the final agent that is deployed.
Claim 2: For a fixed ‘reasonable’ pretraining objective, there exists some (possibly crazy and bespoke but still reasonably-sized) dataset which would make the resulting model aligned without any finetuning.
(This claim is more of an intuition pump for Claim 3, rather than an interesting claim in its own right)
Argument 1: As long as your pretraining objective doesn’t do something unreasonable like say “ignore the data, always say ‘hello world’”, given the fixed pretraining objective each data point acts as a “constraint” on the parameters of the model. If you have D data points and N model parameters with D > N, then you should expect these constraints to approximately determine the model parameters (in the same way that N linearly independent equations on N variables uniquely determine those variables). So with the appropriate choice of the D data points, you should be able to get any model parameters you want, including the parameters of the aligned model.
Argument 2: There are ~tens of bits going into the choice of pretraining objective, and ~millions of bits going into the dataset, so in some sense nearly all of the action is in the dataset.
Argument 3: For the specific case of next-word prediction, you could take an aligned model, generate a dataset by running that model, and then train a new model with next-word prediction on that dataset.
I believe this is equivalent to model distillation, which has been found to be really unreasonably effective, including for generalization (see e.g. here), so I’d expect the resulting model would be aligned too.Claim 3: If you don’t control the dataset, it mostly doesn’t matter what pretraining objective you use (assuming you use a simple one rather than e.g. a reward function that encodes all of human values); the properties of the model are going to be roughly similar regardless.
Argument: Basically the same as for Claim 2: by far most of the influence on which model you get out is coming from the dataset.
(This is probably the weakest argument in the chain; just because most of the influence comes from the dataset doesn’t mean that the pretraining objective can’t have influence as well. I still think the claim is true though, and I still feel pretty confident about the final conclusion in the next claim.)
Claim 4: GPT-N need not be “trying” to predict the next word. To elaborate: one model of GPT-N is that it is building a world model and making plans in the world model such that it predicts the next word as accurately as possible. This model is fine on-distribution but incorrect off-distribution. In particular, it predicts that GPT-N would e.g. deliberately convince humans to become more predictable so it can do better on future next-word predictions; this model prediction is probably wrong.
Argument: There are several pretraining objectives that could have been used to train GPT-N other than next word prediction (e.g. masked language modeling). For each of these, there’s a corresponding model that the resulting GPT-N would “try” to <do the thing in the pretraining objective>. These models make different predictions about what GPT-N would do off distribution. However, by claim 3 it doesn’t matter much which pretraining objective you use, so most of these models would be wrong.
Wait, people are doing this, instead of just turning words into numbers and having ‘models’ learn those? Anything GPT sized and getting results?
Not totally sure. There are advantages to character-level models, e.g. you can represent Twitter handles (which a word embedding based approach can have trouble). People have definitely trained character-level RNNs in the past. But I don’t know enough about NLP to say whether people have trained large models at the character level. (GPT uses byte pair encoding.)
Why this can’t be rescued with counterfactuals isn’t clear.
I suspect Alex would say that it isn’t clear how to define what a “counterfactual” is given the constraints he has (all you get is a physical closed system and a region of space within that system).
Nah, it’s a great trick.
The trick here is that L2 regularization / weight decay is equivalent to having a Gaussian prior on the parameters, so you can think of that term as logN(θ0,σ) (minus an irrelevant additive constant), where σ is set to imply whatever hyperparameter you used for your weight decay.
This does mean that you are committing to a Gaussian prior over the parameters. If you wanted to include additional information like “moving towards zero is more likely to be good” then you would not have a Gaussian centered at θ0, and so the corresponding log prob would not be the nice simple “L2 distance to θ0”.
I think this intuition is correct, and the typical solution in ML algorithms is to empirically scale all of your quantities such that everything works out (which you can interpret from the unit-checking perspective as “finding the appropriate constant to multiply your quantities by such that they become the right kind of unitless”).