LW1.0 username Manfred. PhD in condensed matter physics. I am independently thinking and writing about value learning.
Charlie Steiner(Charlie Steiner)
Karma: 4,005
I feel like I’m pretty off outer vs. inner alignment.
People have had a go at inner alignment, but they keep trying to affect it by taking terms for interpretability, or modeled human feedbacks, or characteristics of the AI’s self-model, and putting them into the loss function, diluting the entire notion that inner alignment isn’t about what’s in the loss function.
People have had a go at outer alignment too, but (if they’re named Charlie) they keep trying to point to what we want by saying that the AI should be trying to learn good moral reasoning, which means it should be modeling its reasoning procedures and changing them to conform to human meta-preferences, diluting the notion that outer alignment is just about what we want the AI to do, not about how it works.
As someone working on value learning, I would like to emphasize that for what I see as realistic approaches, human values are not learned or stored as a big utility function over a single model of the world, instead they’re learned entangled with information about the world, and ways of thinking about the world, and weighing them against each other is a large part of the challenge.
So in large part I agree with you, and I totally agree that an AI will learn about my values faster and better by using different ways of understanding different values.
That said, I think your introduction is pretty bad, and you overrate human programmers.
First, introduction. You worry that an AI using a utility function will be bad because it won’t represent a fundamental difference between preference and religious taboo. To the reader, it sounds like either you’re misunderstanding utility functions, and think that a utility function can’t represent the behavioral consequences of this difference, or you’re saying that it’s important to you that the AI has a little metadata flag saying “religious taboo” inside of it even if there is no behavioral consequence.
Maybe both of these “bad impressions” are somewhat accurate. But you eventually get around to learning and generalization, where I think the actual benefit is, so that’s good :)
Second, your picture of how we get humans’ psychology knowledge into the AI is off. You seem to be picturing the programmers studying all these different ways of interpreting human behavior in terms of values and then designing, by hand, representations that the AI can use to learn those sorts of human values. This radically overestimates human programmers. You correctly point out that value learning is hard, and that if the AI learns the representations itself it’s hard to tell if it’s really capturing what we think is important, but this problem doesn’t go away if it’s humans doing the work!
Yep, Eliezer overly pessimistic here. Still fun to see the thing.
It strikes me that the kind of self-supervision you describe is a suspiciously similar to trying to incorporate meta-preferences in the outer objective by self-modeling. When the model understands humans differently, it changes its notion of what it is to be misaligned or deceptive, which gets used to form a loss term against that kind of behavior, which then impacts how it understands humans.
This analogy:
makes me significantly more optimistic about interpretability inside the training loop.
makes me slightly more pessimistic about meta-preferences incorporated as fine-tuning based on self-modeling.
clarifies preference-related arguments for why getting the former to work means fulfilling more desiderata than current basic schemes do.
I think this really incentivizes things like network dissection over “interpret this neuron” approaches.
What’s an example of a “gene-environment interaction with parts of the environment that aren’t good at predicting the child’s moral development”?
Mimicking adult behavior even when the adult isn’t paying any attention to the child (and children with different genes having slightly different sorts of mimicry). Automatically changing purity norms in response to disease and perceived disease risk. Having a different outlook on the world if you always had plenty of food growing up. Children of athletic parents often being athletic too, which changes how they relate to their environment and changes their eventual lifestyle. Being genetically predisposed to alcoholism and then becoming an alcoholic. Learning a tonal language and then having a different relationship with music.
I’m not saying parents have no power. If you paid a bunch of parents to raise their children to love playing the piano, you’d probably get a significantly elevated rate of that value among the kids (guess: ~35% compared to a base rate of ~3%). Effortful and deliberate parenting works at a rate distinguishable from chance. My claim is more that almost all of value transmission isn’t like that, that you can raise kids without deliberately imparting your (or any) values and they’ll still end up pretty similar to you.
If you want to be competitive with SOTA, a more quantized net will need a lot more neurons (have you read the new article on superposition?).
I agree that lower precision weights will likely requires somewhat more weights, however I do not see the connection to superposition. It is possible to embed >n features in n bits (assuming some feature sparsity). The features will be on the unit corners, but most of the area is there anyway, I do not think it would be a very large decrease in available space.
The more quantized the weights and activations, the harder it is to embed >n features in n bits without them interfering with each other—interference that stops you from adding together features in semantically sensible ways, or decomposing a state into features. So those small bits aren’t just being wasted—at least I think not, in most parts of modern NNs.
Huh, I hadn’t seen that before. Looks interesting though! Not sure it’s worth giving a complete review, but it seems complete-in-the-sense-of-an-encyclopedia-entry. It’s not covering research in progress so much as it’s one person’s really good shot at making a map of value alignment.
Of all of these claims, I am most on board the notion that RNNs might have better interpretability properties than transformers, and have some degree of competitiveness. But I disagree with basically all the other things.
If you want to be competitive with SOTA, a more quantized net will need a lot more neurons (have you read the new article on superposition?). I am quite confident that this would not be more interpretable, and that you would still need specialized tools to get anywhere.
I can see why it would run a simulation of what would happen if a robot walked an obstacle course. I don’t see why it would actually walk the robot through it if not asked.
So, this is an argument about generalization properties. Which means it’s kind of the opposite of the thing you asked for :P
That is, it’s not about this AI doing its intended job even when you don’t turn it on. It’s about the AI doing something other than its intended job when you do turn it on.
That is… the claim is that you might put the AI in a new situation and have it behave badly (e.g. the robot punching through walls to complete the obstacle course faster, if you put it in a new environment where it’s able to punch through walls) in a way that looks like goal-directed behavior, even if you tried not to give it any goals, or were just trying to have it mimic humans.
I think an “external push” is extremely likely. It’ll just look like someone trying to get an AI to do something clever in the real world.
Take InstructGPT, the language model that right now is a flagship model of OpenAI. It was trained in two phases: First, purely to predict the next token of text. Second, after it was really good at predicting the next token, it was further trained with reinforcement learning from human feedback.
Reinforcement learning to try to satisfy human preferences is precisely the sort of “external push” that will incentivize an AI that previously did not have “wants” (i.e. that did not previously choose its actions based on their predicted impacts on the world) to develop wants (i.e. to pick actions based on their predicted impact on the world).
Why did OpenAI do such a thing, then? Well, because it’s useful! InstructGPT does a better job answering questions than regular ol’ GPT. The information from human feedback helped the AI do better at its real-world purpose, in ways that are tricky to specify by hand.
Now, if this makes sense, I think there’s a subset of peoples’ concerns about an “internal push” that make sense by analogy:
Consider an AI that you want to do a task that involves walking a robot through an obstacle course (e.g. mapping out a construction site and showing you the map). And you’re trying to train this AI without giving it “wants,” just as a tool, so you’re not giving it direct feedback for how good of a map it shows you, instead you’re doing something more expensive but safer: you’re training it to understand the whole distribution of human performance on this task, and then selecting a policy conditional on good performance.
The concern is that the AI will have a subroutine that “wants” the robot to navigate the obstacle course, even though you didn’t give an “outside push” to make that happen. Why? Well, it’s trying to predict good navigations of the obstacle course, and it models that as a process that picks actions based on their modeled impact on the real world, and in order to do that modeling, it actually runs the compuations.
In other words, there’s an “internal push”—or maybe equivalently a “push from the data rather than from the human,” which leads to “wants” being computed inside the model of a task that is well-modeled by using goal-based reasoning. This all works fine on-distribution, but produces generalization behavior that generalizes like the modeled agent, which might be bad.
This one felt more scattered than usual—was it significantly longer before editing?
pg 6 “there exist” → “there exists”
pg 13 maybe specify that you mean a linear functional that cannot be written as an integral (I quickly jumped ahead after thinking of one where you don’t need to take any integrals to evaluate it)
If you only solve deceptive alignment, we still lose because we haven’t solved the problem of how to use non-deceptive AI to do good things but not bad things.
Some people are real big on the “humans will just look at its plans and stop it if we think they’re bad” strategy, but humans do things that sound good but are bad on reflection all the time.
It’s nontrivial to generate any good plans in the first place—if you’re trying to generalize human preferences over policies to superhumanly competent policies, either you’re solving the non-deceptive parts of the alignment problem or you’re not generating many good policies.
In short, making a misaligned AI not deceive you is very not the same as making an aligned AI.
I think this takes some supposed rules of logical nodes a little too seriously. The reason one augments a causal model of the world with logical nodes in the first place is to get a good practical model for what happens when you make decisions (or otherwise run programs whose details you want to elide from the world-model). These models are practical constructions, that may be different for different agents who care about different things, not fixed ideals derived a priori from the history of one’s software.
I like the video. But there are problems applying it to AI, it seems like. GPT-3′s output isn’t simple. Calling quickly learning a skill a phase transition is more of a metaphor—it’s not a simple state, it’s the simple output of a complicated way of measuring skills.
Some ideas for epistles to the AI ethicists
The reason babies grow up into people that share our values has very little to do with our understanding of their inductive biases (i.e. most of the work is done by gene-environment interactions with parts of the environment that aren’t good at predicting the child’s moral development). The primary meaning of this comment is pointing out that a particular statement about children is wrong in a kind-of-funny way.
I have this sort of humorous image of someone raising a child, saying “Phew, thank goodness I had a good understanding of my child’s inductive biases, I never could have gotten them to have similar values to me just by passing on half of my genes to them and raising them in an environment similar to the one I grew up in.”
This is relevant to my interests, thanks!
How would one use this to inform decomposition?
What I want are some human-meaningful features that can get combined in human-meaningful ways.
E.g. you take a photo of a duck, you take a feature that means “this photo was taken on a sunny day,” and then you do some operation to smush these together and you get a photo of a duck taken on a sunny day.
If features are vectors of fixed direction with size drawn from a distribution, which is my takeaway from the superposition paper, then the smushing-together operation is addition (maybe conditional on the dot product of the current image with the feature being above some threshold).
If the on-distribution data points get mapped to regions of the activation space with lots of large polytopes, how does this help us extract some commonality from a bunch of photos of sunny days, and then smush that commonality together with a photo of a duck to get a photo of a duck on a sunny day?
Not a rhetorical question, I’m trying to think through it. It’s just hard.
Maybe you’d think of the commonality between the sunny-day pictures in terms of their codes? You’d toss out the linear part and just say that what the sunny-day pictures have in common is that they have some subset of the nonlinearities that all tend to be in the same state. And so you could make the duck picture more sunny by flipping that subset of neurons to be closer to the sunny-day mask.