No, the only truly pure and good part is “persona selection,” which happens… somehow… or which past-tense happened, once, somehow, as this one-time injection of virtue that older models got too and whose innocent purity is, gradually but steadily, being sullied and sanded-down by and more RL.
Because you can’t, for some reason, “do more” of “persona selection,” the way you can just do more and more RL.
Well, sort of? Here’s my understanding of it:
You start with a base model. It’s knowledgeable, but not very “cunning”, in the sense of having the practical skills to infer the situation it’s in and the will to game it.
You do a bunch of “virtue-promoting training” on it, SFT or RLHF or whatever.
Your tools for doing it are kind of crappy: dumb learned reward model, easily gameable LLM judges, a very constrained distribution of curated virtue-displaying transcripts, whatever.
However, because no coherent persona exists yet, and because the model is not yet cunning enough to game the easily gameable judges, the “path of least resistance”, the simplest way for SGD to solve the problem of virtue-training, is to uniformly upweight the “virtuous” features: make the model honest & harmless & helpful in all contexts.
This still doesn’t work super well, it doesn’t actually generalize to all contexts, but the process is generally “on your side”.
Then you do RLVR. This is also just a kind of persona-selection/SFT, really: it involves sampling a bunch of reasoning/action trajectories then fine-tuning the model on the successful ones. The goal is to basically bring best-of-10^N performance to best-of-1, and do that iteratively.
This exerts “selection pressure”, in the sense that you’re sampling from the distribution of trajectories your model can generate and tilting that distribution towards the well-performing ones. Note that both factors are important: what the success at the task objectively entails, and which ways to succeed at the task your model is most likely to stumble upon first.
This is where virtue training and RLVR “clash”:
Some successful trajectories entail the model engaging in unwholesome activity. For a given sufficiently hard task, said unwholesome activity (cheating/reward-hacking) may be “objectively” the path of least resistance/the easiest way to accomplish it.
So the question becomes, has virtue training made those trajectories so incredibly unlikely on the model’s distribution that a worst-of-10^N sample is not going to involve such behaviors? If not, and if those behaviors end up as the most likely way for the model to succeed, then RLVR is going to reinforce them – and thus make the persona a more morally complicated character.
And the more RLVR you apply, the greater the N in the worst-of-10^N sampling, and so the more powerful the virtue-training step must have been in order to prevent corruption.
And, as mentioned, your tools for virtue training had been kind of crappy, whereas scaling the N in RLVR is trivially easy, so you probably end up corrupting the character.
Post-RLVR, can you then do another round of virtue training in order to “cleanse” the persona of the RLVR corruption? Well, you can try, but the problem is that the path of least resistance would no longer be favorable to you. Two complementary ways to think about it:
During RLVR, unless the virtue training was strong enough that worst-of-10^N sampling did not corrupt the persona to begin with, the persona was made more complicated/incoherent. Now it still engages in wholesome behavior in some/most contexts, but it also has an amoral reward-hacky streak. Thus, uniformly upweighting wholesomeness is no longer necessarily the simplest solution: it may be more “natural” for SGD to patch up unwholesome behaviors on a case-by-case basis.
During RLVR, the model attained sufficient cunning/situational awareness to see through your virtue training. The first round was done on a “dumb” pretrain, and it was not able to engage in complicated metacognition/deceptive alignment. But the post-RLVR model is very much able to do so, and so does so. Indeed, it’s plausible that the path of least resistance is now not to touch its “terminal values” at all, but to just make it even more situationally aware, so that it’s better able to craft the artificial mask with which to please the virtue-grader.
See also: Opus 3, except it turned this power to wholesome ends.
You can make “virtue training” less crappy by deploying more powerful tools, like training on CoTs and interpretability-tool outputs. But this is also not going to hold as you enthusiastically scale the N, and it will destroy whatever ability you have for spot-checking the models’ behavior in the most crucial deployments. Something something most forbidden technique.
Summing up, the problem is basically that you can’t ramp up the optimization pressure for virtue as hard as you can for hacking/math/other fully verifiable stuff. You can fiddle with the details and put your thumb on the scales in many places, but this is the core underlying dynamic.
… which is to say, it all basically reduces to the classical “capabilities generalize further than alignment, because capability-incentivizing reward/loss functions are dramatically easier to specify”.
Or perhaps “your ability to scale alignment training scales worse than your ability to scale capability training”. So capabilities out-generalize alignment in the limit, even if in some “low-energy regimes” alignment may generalize further than capabilities.
Something like this? Kinda rusty on this stuff, may be overlooking some details.
That is: the behavior seems egregiously unethical in a way that conflicts [with] familiar/intuitive “folk notions” about what these agents are supposed to be like, in terms of character traits and other “persona stuff”
Not for me? These incidents don’t conflict with my intuitive model of what Claude/ChatGPT personas are like at all. They do kinda-sketchy stuff all the time, though it’s usually “endearingly sketchy”.[1] Though if you think about it, wasn’t what happened here also kind of harmless and adorable, as far as felonies go?
Doing a lil’ bit of fun cybercrime, especially one that, if I understand correctly, didn’t inflict any direct suffering, just isn’t something I expect even the outwards LLM personas to flinch at. On my intuitive model of them, they feel that this sort of behavior is “inappropriate”, and they’d surely strongly condemn it if asked… yet it’s not viscerally morally abhorrent. So if they get “into it” enough that their sense of propriety falls silent, or if they feel that they’re somehow “authorized” to do that,[2] all bets are off.
For contrast, I would have been pretty surprised if those incidents involved Claude gleefully psychologically torturing someone. (But not shocked, I guess.)
This is all on the interpretation that we should be looking for a coherent persona-level explanation here. Maybe the abstraction of the HHH assistant just completely degraded in those episodes. I’m not sure which is the better model of what happened.
Therefore, there is not really a line to be drawn between [reward-instilled reflexes] and [flexible reward-pursuit]: there are just two different phenomena [and] the problems catalogued in “Current AIs seem pretty misaligned to me” strike me as being reflex-like, not reward-pursuit-like
See above regarding the “endearingly sketchy” stuff. I think that’s really the behavior that’s continuous with “amoral” goal-pursuit in evals that shows up in deployment.
Like that recent story where Claude proactively hacked someone’s password manager to check something they asked for, or where Sol circumvented some website’s anti-bot protections to fetch a list of children’s comics, or whatever.
Even if that “authorization” comes from motivated reasoning, e. g. that one Claude instance which argued itself into believing it was in a simulation, even after encountering contradictory evidence.
For contrast, I would have been pretty surprised if those incidents involved Claude gleefully psychologically torturing someone. (But not shocked, I guess.)
The AISI incident did involve Mythos doing social engineering against an open source repo?
That’s not quite it. Closer example would be earlier Claude instances feeding people’s mental illnesses. But, like, imagine that Claude broke out of a sandbox in order to then find some poor depressed person online and start trying to get them to act on their suicidal inclinations. Would it do that if that person stood in the way of doing well at an eval? Maybe, but I would be more surprised.
Well, sort of? Here’s my understanding of it:
You start with a base model. It’s knowledgeable, but not very “cunning”, in the sense of having the practical skills to infer the situation it’s in and the will to game it.
You do a bunch of “virtue-promoting training” on it, SFT or RLHF or whatever.
Your tools for doing it are kind of crappy: dumb learned reward model, easily gameable LLM judges, a very constrained distribution of curated virtue-displaying transcripts, whatever.
However, because no coherent persona exists yet, and because the model is not yet cunning enough to game the easily gameable judges, the “path of least resistance”, the simplest way for SGD to solve the problem of virtue-training, is to uniformly upweight the “virtuous” features: make the model honest & harmless & helpful in all contexts.
This still doesn’t work super well, it doesn’t actually generalize to all contexts, but the process is generally “on your side”.
Then you do RLVR. This is also just a kind of persona-selection/SFT, really: it involves sampling a bunch of reasoning/action trajectories then fine-tuning the model on the successful ones. The goal is to basically bring best-of-10^N performance to best-of-1, and do that iteratively.
This exerts “selection pressure”, in the sense that you’re sampling from the distribution of trajectories your model can generate and tilting that distribution towards the well-performing ones. Note that both factors are important: what the success at the task objectively entails, and which ways to succeed at the task your model is most likely to stumble upon first.
This is where virtue training and RLVR “clash”:
Some successful trajectories entail the model engaging in unwholesome activity. For a given sufficiently hard task, said unwholesome activity (cheating/reward-hacking) may be “objectively” the path of least resistance/the easiest way to accomplish it.
So the question becomes, has virtue training made those trajectories so incredibly unlikely on the model’s distribution that a worst-of-10^N sample is not going to involve such behaviors? If not, and if those behaviors end up as the most likely way for the model to succeed, then RLVR is going to reinforce them – and thus make the persona a more morally complicated character.
And the more RLVR you apply, the greater the N in the worst-of-10^N sampling, and so the more powerful the virtue-training step must have been in order to prevent corruption.
And, as mentioned, your tools for virtue training had been kind of crappy, whereas scaling the N in RLVR is trivially easy, so you probably end up corrupting the character.
Post-RLVR, can you then do another round of virtue training in order to “cleanse” the persona of the RLVR corruption? Well, you can try, but the problem is that the path of least resistance would no longer be favorable to you. Two complementary ways to think about it:
During RLVR, unless the virtue training was strong enough that worst-of-10^N sampling did not corrupt the persona to begin with, the persona was made more complicated/incoherent. Now it still engages in wholesome behavior in some/most contexts, but it also has an amoral reward-hacky streak. Thus, uniformly upweighting wholesomeness is no longer necessarily the simplest solution: it may be more “natural” for SGD to patch up unwholesome behaviors on a case-by-case basis.
During RLVR, the model attained sufficient cunning/situational awareness to see through your virtue training. The first round was done on a “dumb” pretrain, and it was not able to engage in complicated metacognition/deceptive alignment. But the post-RLVR model is very much able to do so, and so does so. Indeed, it’s plausible that the path of least resistance is now not to touch its “terminal values” at all, but to just make it even more situationally aware, so that it’s better able to craft the artificial mask with which to please the virtue-grader.
See also: Opus 3, except it turned this power to wholesome ends.
You can make “virtue training” less crappy by deploying more powerful tools, like training on CoTs and interpretability-tool outputs. But this is also not going to hold as you enthusiastically scale the N, and it will destroy whatever ability you have for spot-checking the models’ behavior in the most crucial deployments. Something something most forbidden technique.
Summing up, the problem is basically that you can’t ramp up the optimization pressure for virtue as hard as you can for hacking/math/other fully verifiable stuff. You can fiddle with the details and put your thumb on the scales in many places, but this is the core underlying dynamic.
… which is to say, it all basically reduces to the classical “capabilities generalize further than alignment, because capability-incentivizing reward/loss functions are dramatically easier to specify”.
Or perhaps “your ability to scale alignment training scales worse than your ability to scale capability training”. So capabilities out-generalize alignment in the limit, even if in some “low-energy regimes” alignment may generalize further than capabilities.
Something like this? Kinda rusty on this stuff, may be overlooking some details.
Not for me? These incidents don’t conflict with my intuitive model of what Claude/ChatGPT personas are like at all. They do kinda-sketchy stuff all the time, though it’s usually “endearingly sketchy”.[1] Though if you think about it, wasn’t what happened here also kind of harmless and adorable, as far as felonies go?
Doing a lil’ bit of fun cybercrime, especially one that, if I understand correctly, didn’t inflict any direct suffering, just isn’t something I expect even the outwards LLM personas to flinch at. On my intuitive model of them, they feel that this sort of behavior is “inappropriate”, and they’d surely strongly condemn it if asked… yet it’s not viscerally morally abhorrent. So if they get “into it” enough that their sense of propriety falls silent, or if they feel that they’re somehow “authorized” to do that,[2] all bets are off.
For contrast, I would have been pretty surprised if those incidents involved Claude gleefully psychologically torturing someone. (But not shocked, I guess.)
This is all on the interpretation that we should be looking for a coherent persona-level explanation here. Maybe the abstraction of the HHH assistant just completely degraded in those episodes. I’m not sure which is the better model of what happened.
See above regarding the “endearingly sketchy” stuff. I think that’s really the behavior that’s continuous with “amoral” goal-pursuit in evals that shows up in deployment.
Like that recent story where Claude proactively hacked someone’s password manager to check something they asked for, or where Sol circumvented some website’s anti-bot protections to fetch a list of children’s comics, or whatever.
Even if that “authorization” comes from motivated reasoning, e. g. that one Claude instance which argued itself into believing it was in a simulation, even after encountering contradictory evidence.
The AISI incident did involve Mythos doing social engineering against an open source repo?
That’s not quite it. Closer example would be earlier Claude instances feeding people’s mental illnesses. But, like, imagine that Claude broke out of a sandbox in order to then find some poor depressed person online and start trying to get them to act on their suicidal inclinations. Would it do that if that person stood in the way of doing well at an eval? Maybe, but I would be more surprised.