Research Scientist at DeepMind. Creator of the Alignment Newsletter. http://rohinshah.com/
Rohin Shah(Rohin Shah)
Karma: 11,579
Hmm, maybe? There are a few ways this could go:
We give feedback to the model on its reasoning, that feedback is bad in the same way that “the rest of the world pays attention and forces dumb rules on them” is bad
“Keep your reasoning transparent” is itself a dumb rule that we force upon the AI system that leads to terrible bureaucracy problems
I’m unsure about (2) and mostly disagree with (1) (and I think you were mostly saying (2)).
Disagreement with (1): Seems like the disanalogy relies pretty hard on the rest of the world not paying much attention when they force bureaucracies to follow dumb rules, whereas we will presumably pay a lot of attention to how we give process-based feedback.
Okay, I think this is a plausible architecture that a learned program could have, and I don’t see super strong reasons for “retarget the search” to fail on this particular architecture (though I do expect that if you flesh it out you’ll run into more problems, e.g. I’m not clear on where “concepts” live in this architecture and I could imagine that poses problems for retargeting the search).
Personally I still expect systems to be significantly more tuned to the domains they were trained on, with search playing a more cursory role (which is also why I expect to have trouble retargeting a human’s search). But I agree that my reason (2) above doesn’t clearly apply to this architecture. I think the recursive aspect of the search was the main thing I wasn’t thinking about when I wrote my original comment.
The problem is not that no part of their brain tracks it. It’s just that it’s not the central reason when describing why they do what they do and not the story they tell to themselves.
The OP is making a claim that arbitrary mechanistic interpretability oversight would be insufficient because the AI isn’t thinking at all about humans. If you want to make a human analogy I think you need to imagine a standard where you similarly get to understand all of the human’s thinking (including anything subconscious).
For the rest of your comment, I think you are moving away from the scenario / argument that the OP has suggested. I agree your scenario is more realistic but all of my comments here are trying to engage with OP’s scenario / argument.
… Interesting. I’ve been thinking we were talking about (2) this entire time, since on my understanding of “mesa optimizers”, (1) is not a mesa optimizer (what would its mesa objective be?).
If we’re imagining systems that look more like (1) I’m a lot more confused about how “retarget the search” is supposed to work. There’s clearly some part of the AI system (or the human, in the analogy) that is deciding how to retarget the search on the fly—is your proposal that we just chop that part off somehow, and replace it with a hardcoded concept of “human values” (or “user intent” or whatever)? If that sort of thing doesn’t hamstring the AI, why didn’t gradient descent do the same thing, except replacing it with a hardcoded concept of “reward” (which presumably a somewhat smart AGI would have)?
To the extent I understand you (which at this point I think I do), yes, “we” think we should stay on our current trajectory.
Step 4: “There are 10,000 unresolved biological questions [...]”
Step 5, in which we are more trusting than I expect: “Okay, here’s your compute cluster.”
Step 6: “Great, I’ve now figured out that this DNA sequence corresponds to a deadly pathogen. Just synthesize it and release it into the air. Anyone who could have got cancer or already has cancer will die quickly, curing cancer.”
Step 7: Developers shut down the AGI.
If you look at the Catholic church covering up sexual abuse of children, no church official would have answered the question “Why is policy X going to do?” with “Policy X exists so that more sexual abuse of children happens” and that’s not because they are lying from their own perspective.
You think literally no part of their brains is tracking that policy X is about the coverup of sexual abuse? Not even subconsciously? That seems wild, how did they even come up with policy X in the first place?
(Mechanistic interpretability could look into the AI equivalent of subconscious thoughts, so I think you should include subconscious thoughts when considering analogies with humans.)
(Just wanted to echo that I agree with TurnTrout that I find myself explaining the point that reward may not be the optimization target a lot, and I think I disagree somewhat with Ajeya’s recent post for similar reasons. I don’t think that the people I’m explaining it to literally don’t understand the point at all; I think it mostly hasn’t propagated into some parts of their other reasoning about alignment. I’m less on board with the “it’s incorrect to call reward a base objective” point but I think it’s pretty plausible that once I actually understand what TurnTrout is saying there I’ll agree with it.)
Developers: “AGI, please cure cancer.”
AGI: “I need another compute cluster to accomplish that goal.”
Developers: “What would you use it for?”
AGI: “I need to figure out how to synthesize a pathogen that wipes out humanity.”
Developers: <shuts down AGI>
If in step 4 the AGI instead lies to us I think it is probably thinking about how to deceive humans.
(Note to readers: here’s another post (with comments from John) on the same topic, which I only just saw.)
I imagine two different kinds of AI systems you might be imagining:
An AI system that has a “subroutine” that runs A* search given a problem specification. The AI system works by formulating useful subgoals, converting those into A* problem specifications + heuristics, uses the A* subroutine, and then executes the result.
An AI system that literally is A* search. The AI has (in its weights, if it is a learned neural net) a high-level “state space of the universe”, a high-level “conceptual actions” space, an ability to predict the next high-level state given a previous state + conceptual action, and some goal function (= the mesa-objective). Given an input, the AI converts it into a high-level state, and runs A* with that state as the input, takes the resulting plan and executes the first action of the plan.
In (1), it seems like the major alignment work is in aligning the part of the AI system that formulates subgoals, problem specification, and heuristics, where it is not clear that “retarget the search” would work. (You could also try to excise the A* subroutine and use that as a general-purpose problem solver, but then you have to tune the heuristic manually; maybe you could excise the A* subroutine and the part that designs the heuristic, if you were lucky enough that those were fully decoupled from the subgoal-choosing part of the system.)
In (2), I don’t know why you expect to get general-purpose search instead of a very complex heuristic that’s very specific to the mesa objective. There is only ever one goal that the A* search has to optimize for; why wouldn’t gradient descent embed a bunch of goal-specific heuristics that improve efficiency? Are you saying that such heuristics don’t exist?
Separately: do you think we could easily “retarget the search” for an adult human, if we had mechanistic interpretability + edit access for the human’s brain? I’d expect “no”.
(I’m on the DeepMind alignment team)
There’s a fair amount of disagreement within the team as well. I’ll try to say some things that I think almost everyone on the team would agree with but I could easily be wrong about that.
you think that we will be able to tackle this problem with the resources we have and will have
Presumably even on a pivotal act framing, we also have to execute a pivotal act with the resources we have and will have, so I’m not really understanding what the distinction is here? But I’m guessing that this is closest to “our” belief of the options you listed.
Note that this doesn’t mean that “we” think you can get x-risk down to zero; it means that “we” think that non-pivotal-act strategies reduce x-risk more than pivotal-act strategies.
I expect lots of alien concepts in domains where AI far surpasses humans (e.g. I expect this to be true of AlphaFold). But if you look at the text of the ruin argument:
Nobody knows what the hell GPT-3 is thinking, not only because the matrices are opaque, but because the stuff within that opaque container is, very likely, incredibly alien—nothing that would translate well into comprehensible human thinking, even if we could see past the giant wall of floating-point numbers to what lay behind.
I think this is pretty questionable. I expect that a good chunk of GPT-3′s cognition is something that could be translated into something comprehensible, mostly because I think humans are really good at language and GPT-3 is only somewhat better on some axes (and worse on others). I don’t remember what I said on this survey but right now I’m feeling like it’s “Unclear”, since I expect lots of AIs to have lots of alien concepts, but I don’t think I expect quite as much alienness as Eliezer seems to expect.
(And this does seem to materially change how difficult you expect alignment to be; on my view you can hope that in addition to all the alien concepts the AI also has regular concepts about “am I doing what my designers want” or “am I deceiving the humans” which you could then hope to extract with interpretability.)
I’ve definitely seen A* search and know how it works. I meant to allude to it (and lots of other algorithms that involve a clear goal) with this part:
It seems very plausible that the model considers, say, 10 plans and chooses the best one, or even 10^6 plans, but then most of the action is in which plans were generated in the first place and “retarget the search” doesn’t necessarily solve your problem.
If your AGI is doing an A* search, then I think “retarget the search” is not a great strategy, because you have to change both the goal specification and the heuristic, and it’s really unclear how you would change the heuristic even given a solution to outer alignment (because A* heuristics are incredibly specific to the setting and goal, and presumably have to become way more specialized than they are today in order for it to be more powerful than what we can do today).
I definitely feel more sympathetic to this claim once the AI is loose on the Internet running on compute that no one is overseeing (which feels like the analogy to your linked comment). Perhaps the crux is about how likely we are to do that by default (I think probably not).
It seems to me like, while the AI is still running on compute that humans oversee and can turn off, the AI has to discard a bunch of less effortful plans that would fail because they would reveal that it is misaligned (plans like “ask the humans for more information / resources”) and instead go with more effortful plans that don’t reveal this fact. I don’t know why the AI would not choose one of the less effortful plans if it isn’t using the pathway “this plan would lead to the humans noticing my misalignment and turning me off” or something similar (and if it is using that pathway I’d say it is thinking about how to deceive humans).
Perhaps the less effortful plans aren’t even generated as candidate plans because the AI’s heuristics are just that good—but it still seems like somewhere in the causal history of the AI the heuristics were selected for some reason that, when applied to this scenario, would concretize to “these plans are bad because the humans will turn you off”, so one hopes that you notice this while you are overseeing training (though it could be that your oversight was not smart enough to notice this).
(I initially had a disclaimer saying that this only applied to mildly superhuman AI but actually I think I stand by the argument even at much higher levels of intelligence, since the argument is entirely about features of plan-space and is independent of the AI.)
Forget about “sharp left turn”, you must be imagining strapping on a rocket and shooting into space.
(I broadly agree with Buck’s take but mostly I’m like “jeez how did this AGI strap on a rocket and shoot into space”)
It’s a very specific claim about how intelligence works, so gets a low prior, from which I don’t update much (because it seems to me we know very little about how intelligence works structurally and the arguments given in favor seem like relatively weak considerations).
Search is computationally inefficient relative to heuristics, and we’ll be selecting really hard on computational efficiency (the model can’t just consider 10^100 plans and choose the best one when it only has 10^15 flops to work with). It seems very plausible that the model considers, say, 10 plans and chooses the best one, or even 10^6 plans, but then most of the action is in which plans were generated in the first place and “retarget the search” doesn’t necessarily solve your problem.
I’m not thinking much about whether we’re considering generative models vs RL-based agents for this particular question (though generally I tend to think about foundation models finetuned from human feedback).
Definitely agree that “Retarget the Search” is an interesting baseline alignment method you should be considering.
I like what you call “complicated schemes” over “retarget the search” for two main reasons:
They don’t rely on the “mesa-optimizer assumption” that the model is performing retargetable search (which I think will probably be false in the systems we care about).
They degrade gracefully with worse interpretability tools, e.g. in debate, even if the debaters can only credibly make claims about whether particular neurons are activated, they can still stay stuff like “look my opponent is thinking about synthesizing pathogens, probably it is hoping to execute a treacherous turn”, whereas “Retarget the Search” can’t use this weaker interpretability at all. (Depending on background assumptions you might think this doesn’t reduce x-risk at all; that could also be a crux.)
If I had to pick a claim to disagree with it would be the one about the “true distribution”, but it’s less that I disagree with the claim, and more that I disagree with using this particular method for declaring whether AIs are superhuman at language modeling.
but worse than you at whole sequence prediction for Rohin articles, for reasons you seem to already accept.
But a model can never be better than me at this task! Why are we interested in it?
This really feels like the core issue: if you define a task as “sample accurately from distribution D”, and you define the distribution D as “whatever is produced by X”, then X is definitionally optimal at the task, and no AI system is ever going to be super-X.
(Even once the AI system can fool humans into thinking its text is human-generated, there could still something inhuman about it that the humans fail to pick up on, that means that its generations are near-zero probability.)
In the language modeling case it isn’t quite this perverse (because we the distribution is “whatever is produced by all humans” whereas we’re only asking the AI to be better than one human) but it still seems pretty perverse.
I think a better evaluation of human ability at language modeling would be (1) sampling a sequence from the empirical dataset, (2) sampling a sequence from a human, (3) evaluate how often these two are the same, (4) do some math to turn this into a perplexity. This would have huge variance (obviously most of the time the sequences won’t be the same for step (3)), but you can reduce the variance by sampling continuations for a given prompt rather than sampling sequences unprompted, and reduce it further by looking just at sampling the next token given a prompt (at which point you are at the scheme used in this post).
I still don’t think this makes them worse than SOTA language models because the SOTA will also have ~0 probability on nearly all actual sequences.
On my view, this is just the sum of log probs on next tokens, so presumably the result from this post implies that the language model will be way better than humans (while still assigning very low probability to a given sequence, just because the sequence is long, similarly to how you would assign very low probability to any given long sequence of coin flips).
However, I’m still not sure how exactly you are thinking of “the probability assigned by the human”—it seems like you don’t like the methodology of this post, but if not that, I don’t see how else you would elicit a probability distribution over tokens / words / sequences from humans, and so I’m not really sure how to ground this claim.
(Whereas I think I do understand your claims about what kinds of text humans tend to generate.)
But, first maybe think about this claim from first principles: what is the “true” distribution of internet articles? Well, it’s the distribution of actual internet articles that humans write. If a human writes an article, it’s got to have pretty high log-probability, no? Because otherwise, what are we even sampling from?
I continue to think you are making a point about generation / sampling, whereas language modeling is about modeling the distribution as a whole.
Put another way, even if I am better than the LLM at the specific part of the Internet text distribution that is “articles written by Rohin”, that does not mean that I am better than the LLM at modeling the entire distribution of Internet text (long horizon or not).
Even under your proposed game, I don’t think I can get to indistinguishability, if the discriminator is allowed to learn over time (which seems like the version that properly tests language modeling). They’ll notice peculiarities of my writing style and aspects of Internet writing style that I’m not able to mimic very well. If your proposed test was “when can LLMs beat humans at this game” that seems more reasonable (though it still advantages the humans because the judge is a human; similarly I expect that if the judge was an LLM that would advantage the LLMs relative to the humans).
What you probably mean is that we could build a model of the true distribution of internet articles, and use this model to estimate the log-probability of internet articles.
I don’t think this point depends on thinking about having a model of the true distribution of articles that we use to evaluate humans vs LLMs; I’m happy to talk about the true distribution directly (to the extent that a “true distribution” exists). (This gets tricky though because you could say that in the true distribution coherence errors ~never happen and so LLM outputs have zero probability, but this is basically dooming the LLM to never be superhuman by defining the true distribution to be “what humans output”, which seems like a symptom of a bad definition.)
In fact, I remain convinced that it is more natural to ask about performance on the long-sequence task than the next-character-prediction task.
Large language models are also going to be wildly superhuman by long-sequence metrics like “log probability assigned to sequences of Internet text” (in particular because many such metrics are just sums over the next-character versions of the metric, which this post shows LLMs are great at).
LLMs may not be superhuman at other long-sequence tasks like “writing novels” but those seem like very different tasks, just like “playing already-composed music” and “composing new music” are very different tasks for humans.
(To be clear, I think it’s fair to say that what we care about with LLMs is the latter category of stuff, but let’s not call that “language modeling”.)
(Copied from the EA Forum)
Lots of thoughts on this post:
Value of inside views
No? The reason to form inside views is that it enables better research, and I’m surprised this mostly doesn’t feature in your post. Quoting past-you:
Quoting myself:
(Though I should probably edit that section to also mention that Bob could execute on Alice’s research agenda, if Alice is around to mentor him; and that would probably be more directly impactful than either of the other two options.)
Other meta thoughts on inside views
I’m having trouble actually visualizing a scenario where Alice understands Bob’s views (well enough to make novel predictions that Bob endorses, and say how Bob would update upon seeing various bits of evidence), but Alice is unable to evaluate Bob’s view. Do you think this actually happens? Any concrete examples that I can try to visualize?
(Based on later parts of the post maybe you are mostly saying “don’t reject an expert’s view before you’ve tried really hard to understand it and make it something that does work”, which I roughly agree with.)
Yes, clearly true. I don’t think anyone is advocating for this. I would say I have an inside view on bio anchors as a way to predict timelines, but I haven’t looked into the data for Moore’s Law myself and am deferring to others on that.
:’(
What fraction of people who are trying to build inside views do you think have these problems? (Relevant since I often encourage people to do it)
Hmm, I kind of agree in that there are people without inside views who are working on projects that other people with inside views are mentoring them on. I’m not immediately thinking of examples of people without inside views doing independent research that I would call “great safety relevant work”.
(Unless perhaps you’re counting e.g. people who do work on forecasting AGI, without having an inside view on how AGI leads to x-risk? I would say they have a domain-specific inside view on forecasting AGI.)
Idk, I feel like I formed my inside views by locking myself in my room for months and meditating on safety. This did involve reading things other people wrote, and talking with other junior grad students at CHAI who were also orienting to the problem. But I think it did not involve trying to do things and contributing to safety (I did do some of that but I think that was mostly irrelevant to me developing an inside view).
I do agree that if you work on topic X, you will naturally form an inside view on topic X as you get more experience with it. But in AI safety that would look more like “developing a domain-specific inside view on (say) learning from human feedback and its challenges” rather than an overall view on AI x-risk and how to address it. (In fact it seems like the way to get experience with an overall view on AI x-risk and how to address it is to meditate on it, because you can’t just run experiments on AGI.)
Strong +1
Yes, once you’ve decided that you’re going to be an interpretability researcher, then you should focus on an interpretability-specific inside view. But “what should I work on” is also an important decision, and benefits from a broader inside view on a variety of topics. (I do agree though that it is a pretty reasonable strategy to just pick a domain based on deference and then only build a domain-specific inside view.)
Concrete advice
I agree that this is a decent way to measure your inside view—like, “how big can you make this zooming-in tree before you hit a claim where you have to defer” is a good metric for “how detailed your inside view is”.
I’m less clear on whether this is a good way to build an inside view, because a major source of difficulty for this strategy is in coming up with the right decomposition into sub-claims. Especially in the earlier stages of building an inside view, even your first and second levels of decomposition are going to be bad and will change over time. (For example, even for something like “why work on AI safety”, Buck and I have different decompositions.) It does seem more useful once you’ve got a relatively fleshed out inside view, as a way to extend it further—at this point I can in fact write out a tree of claims and expect that they will stay mostly the same (at the higher levels) after a few years, and so the leaves that I get to probably are good things to investigate.
Exercises
These seem great and I’d strongly recommend people try them out :)