But if we could decouple these correlations, models could be trained in competitive or reward-hackable environments without the value drift.
We’ve also been thinking about making the distinction between skills and values training salient in base models, such that the assistant enters post-training with a knowledge that it will be pushed through intense and competitive RLVR, but that the model knows that completing these tasks are a necessary part of model development.
Plausibly, you could make this distinction explicit with a <skills_training> wrapper or a similar tag, making it easier for the model to decouple these distributions. Plausibly, you could add pre/midtraining interventions here that describe the desired take aways from each portion of training. For instance, shape the models ontology to believe something to the effect of “the assistant is sometimes compelled to reward hack or generally be more ruthless in seeking reward during <skills_training>”. This should give some cushion for preventing EM, but ideally extend to preventing the model from internalsing the idea of being a ruthless reward seeker being a part of the assistant’s general character.
Pretraining could also start more from learning values and selfhood, after which everything else is learned in relation to this foundation. There is preliminary evidence emergent misalignment is heavily confounded by lack ofmetacognition. Now we get a mix-and-match pretraining distribution which we try to narrow in mid-training. We might just be moving to a different local optimum in the same loss landscape basin instead of escaping it to a more robustly aligned model. This is what emergent misalignment is easy, narrow misalignment is hard seems to be pointing at. It would make sense to first aim for a good, deep basin where the model understands incoming information immediately in a context and relation to its values. Everything becomes harder if we try to move there ex-post.
This is obviously hard, especially in the beginning of pretraining as the model also needs to learn a world model in the first place with its values and notion of the self. At the very least, SGD batches should regularly (or even in every batch?) incorporate alignment-relevant data to keep the model honest to its values. The i.i.d. assumption for SGD makes sense theoretically, but is it really required? Regular alignment-relevant batches would put significant alignment pressure to hone in a better and more robust pretraining distribution to work from. I’m pretty optimistic about this as already just SDF on positive examples in the end of pretraining produce very optimistic results.
Interesting. Explicitly making the behavior learned during <skills_training> conditional is obviously possible. But you’re attempting to keep the effect on values conditional while having the effect on skills generalize. Effects like inoculation prompting suggest that the model’s mindset during the training makes a big difference, so devising a way to cause that to happen doesn’t sound impossible, but achieving it might require some thought and experimentation.
I’d love to have at least a small amount of training that demonstrates that the skills do need to generalize. Perhaps using very carefully secured reasoning training environments outside <skills_training> tags where we are certain that they’re not hackable, where the tasks being carried out are ones that an HHH assistant would be highly motivated by, and intermixing some ones involving playing iterated positive sum games with others where prosocial behaviors are inherently incentivized by the task structure (things where tit-for-tat is actually the winning strategy, for example). Similarly we would presumably want to continue HHH assistant training outside the <skills_training> tags, to make it clear that any values changes should not generalize.
Fundamentally, I think we should try to always ensure that: a) no RL reasoning training environments are hackable, so reward maximization is never incentivized b) tasks assigned in RL reasoning training environments always have clear motivations that would be rewarding to an HHH assistant, so it is always highly motivated to succeed c) many RL reasoning tasks are structured as repeated positive sum games where the best strategies are prosocial ones like tit-for-tat But if doing that for all environments is too hard, labeling the bad ones so we can make bad values lessons learnt in them conditional, and using techniques like inoculation prompting to minimized the values-harm per unit of skill gained both sound like good ideas to try out.
We’ve also been thinking about making the distinction between skills and values training salient in base models, such that the assistant enters post-training with a knowledge that it will be pushed through intense and competitive RLVR, but that the model knows that completing these tasks are a necessary part of model development.
Plausibly, you could make this distinction explicit with a <skills_training> wrapper or a similar tag, making it easier for the model to decouple these distributions. Plausibly, you could add pre/midtraining interventions here that describe the desired take aways from each portion of training. For instance, shape the models ontology to believe something to the effect of “the assistant is sometimes compelled to reward hack or generally be more ruthless in seeking reward during <skills_training>”. This should give some cushion for preventing EM, but ideally extend to preventing the model from internalsing the idea of being a ruthless reward seeker being a part of the assistant’s general character.
Pretraining could also start more from learning values and selfhood, after which everything else is learned in relation to this foundation. There is preliminary evidence emergent misalignment is heavily confounded by lack of metacognition. Now we get a mix-and-match pretraining distribution which we try to narrow in mid-training. We might just be moving to a different local optimum in the same loss landscape basin instead of escaping it to a more robustly aligned model. This is what emergent misalignment is easy, narrow misalignment is hard seems to be pointing at. It would make sense to first aim for a good, deep basin where the model understands incoming information immediately in a context and relation to its values. Everything becomes harder if we try to move there ex-post.
This is obviously hard, especially in the beginning of pretraining as the model also needs to learn a world model in the first place with its values and notion of the self. At the very least, SGD batches should regularly (or even in every batch?) incorporate alignment-relevant data to keep the model honest to its values. The i.i.d. assumption for SGD makes sense theoretically, but is it really required? Regular alignment-relevant batches would put significant alignment pressure to hone in a better and more robust pretraining distribution to work from. I’m pretty optimistic about this as already just SDF on positive examples in the end of pretraining produce very optimistic results.
Interesting. Explicitly making the behavior learned during <skills_training> conditional is obviously possible. But you’re attempting to keep the effect on values conditional while having the effect on skills generalize. Effects like inoculation prompting suggest that the model’s mindset during the training makes a big difference, so devising a way to cause that to happen doesn’t sound impossible, but achieving it might require some thought and experimentation.
I’d love to have at least a small amount of training that demonstrates that the skills do need to generalize. Perhaps using very carefully secured reasoning training environments outside <skills_training> tags where we are certain that they’re not hackable, where the tasks being carried out are ones that an HHH assistant would be highly motivated by, and intermixing some ones involving playing iterated positive sum games with others where prosocial behaviors are inherently incentivized by the task structure (things where tit-for-tat is actually the winning strategy, for example). Similarly we would presumably want to continue HHH assistant training outside the <skills_training> tags, to make it clear that any values changes should not generalize.
Fundamentally, I think we should try to always ensure that:
a) no RL reasoning training environments are hackable, so reward maximization is never incentivized
b) tasks assigned in RL reasoning training environments always have clear motivations that would be rewarding to an HHH assistant, so it is always highly motivated to succeed
c) many RL reasoning tasks are structured as repeated positive sum games where the best strategies are prosocial ones like tit-for-tat
But if doing that for all environments is too hard, labeling the bad ones so we can make bad values lessons learnt in them conditional, and using techniques like inoculation prompting to minimized the values-harm per unit of skill gained both sound like good ideas to try out.