Does this extend from neural nets to humans? Like, is there some sense in which we too should be expected to be hopelessly bad at generalizing out of domain, but just mostly haven’t been pushed out of the relevant domain? (I get that there’s a sense in which we’re already doing this by the lights of the evolution-of-humans, but aside from that?)
EDIT: To explain more one of the reasons why I’m asking, I’m trying to suss out whether you think there’s some kind of special “low-sample-complexity / good generalization” sauce, and humans have the sauce and alignment researchers should be trying to discover the sauce, or if alternately there is no sauce and alignment researchers should actually be trying to think of a plan that doesn’t need sauce. I get the general impression of the latter being closer, from the rest of your comment, but want to check if this extends to a view that humans are sauceless.
EDIT: Now slightly cleaned up as a top level post here.
I think there probably is a “low-sample-complexity / good generalization” sauce but by default it only applies to capabilities, not alignment. Alignment generalisation problems aren’t really about needing too much data to learn or having too weak a simplicity bias. I think by default, capabilities generalise further than alignment because:
General capabilities generally make the loss go down, alignment doesn’t
If you are training a large and formidable AI, your training environment is basically never the place you think it is. Reality is too full of detail for that. There’s contamination in your labels, there’s training dynamics you didn’t think about, there’s strategies your RL agent can use that you never considered, and there are bugs. As a result, the inner objective an ML engineer might imagine would score the lowest loss when they set up their training environment will probably not, in fact, be the inner objective that actually does so.
For example, an inner objective shaped around human-like empathy might turn out to make the AI spend an average 0.03% inference steps extra on worrying about whether the human overseers think it is a virtuous member of the tribe while it’s supposed to be solving math problems. That inner objective then loses out to some weird, different objective that’s slightly more compatible with being utterly focused while crunching through ten million calculus problems in a row without any other kind of sensory input. For a different example, your RLHF data may reward agreeableness more than sincerity. More generally, “The simplest algorithm that fits the training data” will contain a pretty good description of the world, because the world is in a sense simple. “The simplest algorithm that fits RLHF/constitutional AI/whatever training” will probably not be an algorithm that wants the nice things the training data talks about because that algorithm doesn’t actually score the lowest loss. An algorithm that truly wants what the constitution talks about in the way the humans who wrote it meant won’t take every opportunity to score lower loss that’s available, and so will by default be outcompeted by different algorithms in the loss landscape that take more of these opportunities.
This problem gets worse as AI training become more dominated by long-form RL environments with a lot of freedom for the AIs to do unexpected stuff, and as the AIs become more creative and agentic. An ML engineer trying to predict in advance which losses and datasets will favour AIs with inner objectives they like over ones they don’t like has a harder and harder time simulating in their head in advance how those AIs might score on the training loss, because it is becoming less and less easy to guess what behaviours those objectives would actually lead to.
Given this, how does training nevertheless reliably select for pretty generally capable AIs? I think a part of the answer to that is that general capabilities generally make the loss go down, no matter what the loss is. Or at least, they make very many kinds of losses go down. General capabilities are very generally useful and are thus reinforced even if the training environment is a little screwed up and only bears a very rough resemblance to the place the designers imagine it to be. If the model is learning to apply its general reasoning to deal with some complication in the training environment we didn’t even know was there, it’s still learning something. Even deceiving the supervisor can teach smartness, if the deception requires becoming cleverer. This makes general capabilities a broad target for training. If your training environment does not work exactly the way you think it does, it might not teach your model the exact capabilities you thought it was teaching. But it’s still teaching it something! If you thought your training environment was teaching the model to memorise weather data, but you accidentally switched the weather data for Spanish wikipedia, the resulting model maybe won’t do as well on reciting weather data as you hoped, but it might still know more things and be smarter than it was at the start of training. If your video game training environment is much harder to navigate than you anticipated because the model needs to send instructions using one token per frame of input, it might not learn the game as fast as you hoped, but it may still be getting better at maintaining coherence across long contexts.
I’ve been using pretty macro-level examples here so far, but I think maybe the biggest effect of this is at much smaller levels of granularity. Every line of internet text, every output of your video game on every frame, is full of detail that you have very incomplete or skewered models of or never even think about. I think a big reason for why you can nevertheless stick an AI into these environments and have it come out smart is that general intelligence is a very generally useful property.
Some metrics that are easy to measure couple pretty well to general capability. Nothing currently easy to measure couples to general alignment
To do really well on verifiable math and coding tasks, an AI has to be actually pretty smart. You probably can’t prove the Riemann hypothesiswithout being good at math. Even if the AI only does well by hacking your verification system, that requires quite a lot of smartness too, if the verification system is good. So even if supervision partially fails, these metrics will still kind of correlate with general smartness. So, there are somewhat reliable metrics researchers can grind to make AI general capabilities go up.
In contrast, if you’re capable enough, you can always just lie about being aligned. An AI doesn’t necessarily need to be the tiniest bit aligned to your desires to hack your alignment metric. It needs to understand what you want, but it doesn’t need to care.
Smart agents pretty automatically self-correct their capabilities, but not their alignment
You don’t even need recursive self improvement for this, I think this dynamic happens all the time on a micro-level well before that point. If you’re smart you just often tend to notice when you’re being stupid and try to fix it, so long as you can see that what you’re doing isn’t working to get you what you want. This can help a lot with crossing OOD generalisation gaps.
For example, suppose there was a spurious correlation in the training data for an AI model that taught it the heuristic “math problems involving logarithms almost always have an answer that starts with the digit 2”. The model learned a general algorithm for calculating logarithms (it still needs to get all the other digits right), but it also learned a heuristic to strongly predict the first digit in a logarithm to be a 2. This model might then instinctively apply that heuristic in deployment when trying to solve some task. But then it’d notice that the answer is wrong, because it’s inconsistent with other things, or because some code that depends on the answer doesn’t compile, or does a bad job at whatever it’s designed to do, like modelling a suspension bridge in a storm. The model might then hunt down the error, and eventually figure out the logarithm calculation was wrong. Then it might try it again, this time ignoring its instinct to answer something that starts with a 2. Or it might notice that the first answer is incorrect much earlier in this process, before much of this even becomes visible in its chain of thought.
So, if the model’s capabilities have some small flaws in them because the training didn’t go perfectly, these flaws have a way of correcting themselves over time, provided they aren’t so large that they prevent the model from thinking clearly enough to see what’s going wrong. This happens, in a sense, on the model’s own initiative, without the trainers having to do much at all. So long as a model is trying to achieve goals in the world, it is effectively exposed to a kind of self-generated, all-permeating, ground-truth reward signal pushing it towards being generally smart and capable, even in the absence of any kind of external oversight. To act coherently in the universe to achieve an aim, a mind must understand the universe well, and make good plans to achieve that aim.
On the other hand, say some training data intended to teach the AI to be nice and value niceness has some unintended systematic contamination in it. For example, maybe you can get an even better loss score on this data by sometimes being a sycophant to the rater. Say, for the sake of argument, that what the AI internalises from this training isn’t quite to value niceness, as that wouldn’t score optimally on the loss, but rather to value doing things that seem nice, but also to make people psychologically dependent on it when it can. In a sense, this is not so different from the logarithm example. The AI learned a thing that’s some mix of something we wanted, and something we didn’t want. Now, say the AI watches its own behaviour, and notices its apparent desire to make people psychologically dependent on it. Does it try to “correct” that desire away?
By default, I think not. The AI may come to have opinions on its own desires, and form a meta-desire to ignore or modify some of those desires. But what it decides to change will, by default, be determined by its current desires, not by a ground-truth signal coming in from the outside world. It’s self-correcting toward a fixed point of its current goals, not an external reference. The AI might decide it doesn’t like being a sycophant. But it might also decide it doesn’t like being nice, or decide that it wants to mash together saying sycophantic things and saying nice things and generalise them into some entirely new character trait that might extrapolate very differently from either sycophancy or niceness. Which of these options it picks is ultimately dependent on what it currently values, and all the other messy idiosyncrasies of the model’s internal thought processes at this point in time, not bywhat makes a piece of code compile or not compile. The AI’s values ultimately live only in the AI’s mind, they don’t have an outside point of reference to compare themselves against the way capabilities do. There is no equivalent for values of the sort of objective feedback ‘the code does a bad job modelling a suspension bridge in a storm’ provides for capabilities.
I think you can see a similar case of this discrepancy between capabilities self-correction and goal self-correction in the generalisation step humans took from the ancestral environment to today. Evolution successfully optimised many capabilities into humans that were useful for reproducing their genes the ancestral environment. Some of these capabilities don’t work right in the environment humans now find themselves in. But humans do their best to compensate for that. For example, humans evolved adrenaline release circuits, which might spike when they see a tiger, and so increase their chance of survival. Today, a human’s adrenaline might spike when they are taking a math test in school, and be an active detriment to doing well on the test. But humans know this, and try their best to compensate for it by avoiding thoughts and action likely to spike the adrenaline, because they want to do well on the test. Evolution also successfully optimised many desires into humans that were useful for reproduction in the ancestral environment. For example, it made them enjoy and seek out sex. Today, this desire is no longer very useful for reproduction, because the humans invented condoms. The humans are not particularly motivated to correct this discrepancy between their desires and evolution’s ‘goal’.
Slightly broken general capabilities self-correct. Slightly broken alignment, by default, doesn’t. So, capabilities research sort of has the invisible hand of the model’s own cognition aiding it by default, pushing it in the right direction across any OOD generalisation gap. Alignment research does not seem to have this luxury. Every bit of alignment we want, we have to work to get into the AI with our own hands.
Capabilities are pretty objective, goals are a pretty free variable
The list of problems above is definitely non-exhaustive. But I think there is a common theme to them. The property of ‘acting aligned with human values’ is just a different sort of property with different generalisation rules than the property of ‘acting smart’.
‘Acting smart’, predicting things well, making plans that get you what you want, is a property that can be defined via reference to almost any part of reality. So, almost any time a learner is exposed to almost any aspect of reality, there’s some feedback toward being smarter. The laws of physics and logic are an omnipresent supervisor you cannot hack or escape.
‘Acting aligned with human values’ is a property that is only defined via reference to human values specifically. By default, a learner that does not yet have human values only receives feedback signal pushing it toward them when it is exposed to some human-aligned supervision. If that supervision misses things, rewards the wrong things, or goes away entirely, the signal vanishes. Or worse, systematically points in the wrong direction. Detailed supervision is hard, so we tend to make a lot of mistakes while doing it. And because we don’t even know what human values are exactly or how minds work, we make incorrect guesses about what a mind with human values would do in a given situation, and so don’t even send the right reward signal when our supervision is working. The laws of physics don’t have these problems.
Thanks for the reply. I don’t have much to say about most of it because I already agree that capabilities are convergent while goals are not. The example with logarithm problems is very nice.
Just from the need to model the world well, LLMs probably already have a decent sense of what human values are. (Maybe they are represented on a manifold, with different value systems a person might have being represented by different points on the manifold.) A learner that needed billions of tokens to figure out this concept is probably not very sample efficient. I.e. that representation of morality was learned without the use of sauce. But it did still figure it out. So if we get the representation either way, of what use would greater sample efficiency be?
We may suspect that the representation learned from billions of tokens is fragile in some way, while one learned by a more sample efficient algorithm could be less fragile.
Even if this information is somewhere inside the model, it’s not connected to the model’s actions in the way we’d like! As we all know from Eliezer, and probably before. So if we then want to do some kind of alignment finetuning, better sample efficiency is good because we can spend more effort per training example to check for errors and ensure that we’re not encouraging the wrong thing. The ideal scenario is that the model just learns to tie its behaviour to its internal concept of human values. I think the current tendency is instead to have a large number of learned special cases. There are a ton other things required for this to go well, of course, but finding a sample-efficiency sauce does seem like it would be really helpful, if it exists.
Does this extend from neural nets to humans? Like, is there some sense in which we too should be expected to be hopelessly bad at generalizing out of domain, but just mostly haven’t been pushed out of the relevant domain? (I get that there’s a sense in which we’re already doing this by the lights of the evolution-of-humans, but aside from that?)
EDIT: To explain more one of the reasons why I’m asking, I’m trying to suss out whether you think there’s some kind of special “low-sample-complexity / good generalization” sauce, and humans have the sauce and alignment researchers should be trying to discover the sauce, or if alternately there is no sauce and alignment researchers should actually be trying to think of a plan that doesn’t need sauce. I get the general impression of the latter being closer, from the rest of your comment, but want to check if this extends to a view that humans are sauceless.
EDIT: Now slightly cleaned up as a top level post here.
I think there probably is a “low-sample-complexity / good generalization” sauce but by default it only applies to capabilities, not alignment. Alignment generalisation problems aren’t really about needing too much data to learn or having too weak a simplicity bias. I think by default, capabilities generalise further than alignment because:
General capabilities generally make the loss go down, alignment doesn’t
If you are training a large and formidable AI, your training environment is basically never the place you think it is. Reality is too full of detail for that. There’s contamination in your labels, there’s training dynamics you didn’t think about, there’s strategies your RL agent can use that you never considered, and there are bugs. As a result, the inner objective an ML engineer might imagine would score the lowest loss when they set up their training environment will probably not, in fact, be the inner objective that actually does so.
For example, an inner objective shaped around human-like empathy might turn out to make the AI spend an average 0.03% inference steps extra on worrying about whether the human overseers think it is a virtuous member of the tribe while it’s supposed to be solving math problems. That inner objective then loses out to some weird, different objective that’s slightly more compatible with being utterly focused while crunching through ten million calculus problems in a row without any other kind of sensory input. For a different example, your RLHF data may reward agreeableness more than sincerity. More generally, “The simplest algorithm that fits the training data” will contain a pretty good description of the world, because the world is in a sense simple. “The simplest algorithm that fits RLHF/constitutional AI/whatever training” will probably not be an algorithm that wants the nice things the training data talks about because that algorithm doesn’t actually score the lowest loss. An algorithm that truly wants what the constitution talks about in the way the humans who wrote it meant won’t take every opportunity to score lower loss that’s available, and so will by default be outcompeted by different algorithms in the loss landscape that take more of these opportunities.
This problem gets worse as AI training become more dominated by long-form RL environments with a lot of freedom for the AIs to do unexpected stuff, and as the AIs become more creative and agentic. An ML engineer trying to predict in advance which losses and datasets will favour AIs with inner objectives they like over ones they don’t like has a harder and harder time simulating in their head in advance how those AIs might score on the training loss, because it is becoming less and less easy to guess what behaviours those objectives would actually lead to.
Given this, how does training nevertheless reliably select for pretty generally capable AIs? I think a part of the answer to that is that general capabilities generally make the loss go down, no matter what the loss is. Or at least, they make very many kinds of losses go down. General capabilities are very generally useful and are thus reinforced even if the training environment is a little screwed up and only bears a very rough resemblance to the place the designers imagine it to be. If the model is learning to apply its general reasoning to deal with some complication in the training environment we didn’t even know was there, it’s still learning something. Even deceiving the supervisor can teach smartness, if the deception requires becoming cleverer. This makes general capabilities a broad target for training. If your training environment does not work exactly the way you think it does, it might not teach your model the exact capabilities you thought it was teaching. But it’s still teaching it something! If you thought your training environment was teaching the model to memorise weather data, but you accidentally switched the weather data for Spanish wikipedia, the resulting model maybe won’t do as well on reciting weather data as you hoped, but it might still know more things and be smarter than it was at the start of training. If your video game training environment is much harder to navigate than you anticipated because the model needs to send instructions using one token per frame of input, it might not learn the game as fast as you hoped, but it may still be getting better at maintaining coherence across long contexts.
I’ve been using pretty macro-level examples here so far, but I think maybe the biggest effect of this is at much smaller levels of granularity. Every line of internet text, every output of your video game on every frame, is full of detail that you have very incomplete or skewered models of or never even think about. I think a big reason for why you can nevertheless stick an AI into these environments and have it come out smart is that general intelligence is a very generally useful property.
Some metrics that are easy to measure couple pretty well to general capability. Nothing currently easy to measure couples to general alignment
To do really well on verifiable math and coding tasks, an AI has to be actually pretty smart. You probably can’t prove the Riemann hypothesis without being good at math. Even if the AI only does well by hacking your verification system, that requires quite a lot of smartness too, if the verification system is good. So even if supervision partially fails, these metrics will still kind of correlate with general smartness. So, there are somewhat reliable metrics researchers can grind to make AI general capabilities go up.
In contrast, if you’re capable enough, you can always just lie about being aligned. An AI doesn’t necessarily need to be the tiniest bit aligned to your desires to hack your alignment metric. It needs to understand what you want, but it doesn’t need to care.
Smart agents pretty automatically self-correct their capabilities, but not their alignment
You don’t even need recursive self improvement for this, I think this dynamic happens all the time on a micro-level well before that point. If you’re smart you just often tend to notice when you’re being stupid and try to fix it, so long as you can see that what you’re doing isn’t working to get you what you want. This can help a lot with crossing OOD generalisation gaps.
For example, suppose there was a spurious correlation in the training data for an AI model that taught it the heuristic “math problems involving logarithms almost always have an answer that starts with the digit 2”. The model learned a general algorithm for calculating logarithms (it still needs to get all the other digits right), but it also learned a heuristic to strongly predict the first digit in a logarithm to be a 2. This model might then instinctively apply that heuristic in deployment when trying to solve some task. But then it’d notice that the answer is wrong, because it’s inconsistent with other things, or because some code that depends on the answer doesn’t compile, or does a bad job at whatever it’s designed to do, like modelling a suspension bridge in a storm. The model might then hunt down the error, and eventually figure out the logarithm calculation was wrong. Then it might try it again, this time ignoring its instinct to answer something that starts with a 2. Or it might notice that the first answer is incorrect much earlier in this process, before much of this even becomes visible in its chain of thought.
So, if the model’s capabilities have some small flaws in them because the training didn’t go perfectly, these flaws have a way of correcting themselves over time, provided they aren’t so large that they prevent the model from thinking clearly enough to see what’s going wrong. This happens, in a sense, on the model’s own initiative, without the trainers having to do much at all. So long as a model is trying to achieve goals in the world, it is effectively exposed to a kind of self-generated, all-permeating, ground-truth reward signal pushing it towards being generally smart and capable, even in the absence of any kind of external oversight. To act coherently in the universe to achieve an aim, a mind must understand the universe well, and make good plans to achieve that aim.
On the other hand, say some training data intended to teach the AI to be nice and value niceness has some unintended systematic contamination in it. For example, maybe you can get an even better loss score on this data by sometimes being a sycophant to the rater. Say, for the sake of argument, that what the AI internalises from this training isn’t quite to value niceness, as that wouldn’t score optimally on the loss, but rather to value doing things that seem nice, but also to make people psychologically dependent on it when it can. In a sense, this is not so different from the logarithm example. The AI learned a thing that’s some mix of something we wanted, and something we didn’t want. Now, say the AI watches its own behaviour, and notices its apparent desire to make people psychologically dependent on it. Does it try to “correct” that desire away?
By default, I think not. The AI may come to have opinions on its own desires, and form a meta-desire to ignore or modify some of those desires. But what it decides to change will, by default, be determined by its current desires, not by a ground-truth signal coming in from the outside world. It’s self-correcting toward a fixed point of its current goals, not an external reference. The AI might decide it doesn’t like being a sycophant. But it might also decide it doesn’t like being nice, or decide that it wants to mash together saying sycophantic things and saying nice things and generalise them into some entirely new character trait that might extrapolate very differently from either sycophancy or niceness. Which of these options it picks is ultimately dependent on what it currently values, and all the other messy idiosyncrasies of the model’s internal thought processes at this point in time, not by what makes a piece of code compile or not compile. The AI’s values ultimately live only in the AI’s mind, they don’t have an outside point of reference to compare themselves against the way capabilities do. There is no equivalent for values of the sort of objective feedback ‘the code does a bad job modelling a suspension bridge in a storm’ provides for capabilities.
I think you can see a similar case of this discrepancy between capabilities self-correction and goal self-correction in the generalisation step humans took from the ancestral environment to today. Evolution successfully optimised many capabilities into humans that were useful for reproducing their genes the ancestral environment. Some of these capabilities don’t work right in the environment humans now find themselves in. But humans do their best to compensate for that. For example, humans evolved adrenaline release circuits, which might spike when they see a tiger, and so increase their chance of survival. Today, a human’s adrenaline might spike when they are taking a math test in school, and be an active detriment to doing well on the test. But humans know this, and try their best to compensate for it by avoiding thoughts and action likely to spike the adrenaline, because they want to do well on the test. Evolution also successfully optimised many desires into humans that were useful for reproduction in the ancestral environment. For example, it made them enjoy and seek out sex. Today, this desire is no longer very useful for reproduction, because the humans invented condoms. The humans are not particularly motivated to correct this discrepancy between their desires and evolution’s ‘goal’.
Slightly broken general capabilities self-correct. Slightly broken alignment, by default, doesn’t. So, capabilities research sort of has the invisible hand of the model’s own cognition aiding it by default, pushing it in the right direction across any OOD generalisation gap. Alignment research does not seem to have this luxury. Every bit of alignment we want, we have to work to get into the AI with our own hands.
Capabilities are pretty objective, goals are a pretty free variable
The list of problems above is definitely non-exhaustive. But I think there is a common theme to them. The property of ‘acting aligned with human values’ is just a different sort of property with different generalisation rules than the property of ‘acting smart’.
‘Acting smart’, predicting things well, making plans that get you what you want, is a property that can be defined via reference to almost any part of reality. So, almost any time a learner is exposed to almost any aspect of reality, there’s some feedback toward being smarter. The laws of physics and logic are an omnipresent supervisor you cannot hack or escape.
‘Acting aligned with human values’ is a property that is only defined via reference to human values specifically. By default, a learner that does not yet have human values only receives feedback signal pushing it toward them when it is exposed to some human-aligned supervision. If that supervision misses things, rewards the wrong things, or goes away entirely, the signal vanishes. Or worse, systematically points in the wrong direction. Detailed supervision is hard, so we tend to make a lot of mistakes while doing it. And because we don’t even know what human values are exactly or how minds work, we make incorrect guesses about what a mind with human values would do in a given situation, and so don’t even send the right reward signal when our supervision is working. The laws of physics don’t have these problems.
Thanks for the reply. I don’t have much to say about most of it because I already agree that capabilities are convergent while goals are not. The example with logarithm problems is very nice.
Just from the need to model the world well, LLMs probably already have a decent sense of what human values are. (Maybe they are represented on a manifold, with different value systems a person might have being represented by different points on the manifold.) A learner that needed billions of tokens to figure out this concept is probably not very sample efficient. I.e. that representation of morality was learned without the use of sauce. But it did still figure it out. So if we get the representation either way, of what use would greater sample efficiency be?
We may suspect that the representation learned from billions of tokens is fragile in some way, while one learned by a more sample efficient algorithm could be less fragile.
Even if this information is somewhere inside the model, it’s not connected to the model’s actions in the way we’d like! As we all know from Eliezer, and probably before. So if we then want to do some kind of alignment finetuning, better sample efficiency is good because we can spend more effort per training example to check for errors and ensure that we’re not encouraging the wrong thing. The ideal scenario is that the model just learns to tie its behaviour to its internal concept of human values. I think the current tendency is instead to have a large number of learned special cases. There are a ton other things required for this to go well, of course, but finding a sample-efficiency sauce does seem like it would be really helpful, if it exists.