Here’s another toy model I’d be interested in: Suppose that the model’s reward depends separately on its capabilities and its alignment. Supposes it increases linearly in capabilities c from 0 to 1. It has three actions: ineffective, aligned and misaligned. If it picks the aligned action, it achieves c rewards always. If it picks the misaligned one, it achieves 0.8.
In that setting, I expect that it would be misaligned with high learning rate/few steps (what you call low “optimization pressure”) and aligned under the reverse.
My point mostly being: the implicit conclusion in your post is “beware initialisation effects because they may incentivize misalignment if you train too hard” whereas I think it should be “beware initialisation effects because we haven’t thought enough about our dynamics to tell what they will do”.
One of my hopes for this domain of research is to produce a theoretical understanding of how to reproduce self-alignment, Opus 3-style, rather than imitating the circumstances that happened to stumble on this dynamic as the linked post proposes (narrating motives a lot? SFT on Opus 3′s outputs?).
Another is to produce a toy model of mesa-optimization, not quite in the wild, but much more relevantly true than what I’ve seen so far. Considering the processes that leads to misaligned behavior is imo essential to be able to categorize something as deceptive, which is why I’m very unsatisfied with Park et al.’s work, and more generally I’m uncomfortable with all concrete examples of deception be on frontier models.
Which is to say, I’m very excited to see what you produce next!
On a meta point though, I’m surprised that you need to write this post. My experience with RL so far has been that most of the work is thinking through this kind of dynamics before starting training, or noticing midpoint that the model is not behaving well because we didn’t think it through correctly.
I’d be surprised if anyone at a frontier lab learns from this post, or didn’t grok its content intuitively, and I presume the purpose of your work is eventually to find its way into their training to better align frontier models.
Like, this post is basically a layman’s explanation of value crystallization and reward hacking. Why do you think it’ll be of use?
Here’s another toy model I’d be interested in:
Suppose that the model’s reward depends separately on its capabilities and its alignment.
Supposes it increases linearly in capabilities c from 0 to 1.
It has three actions: ineffective, aligned and misaligned.
If it picks the aligned action, it achieves c rewards always. If it picks the misaligned one, it achieves 0.8.
In that setting, I expect that it would be misaligned with high learning rate/few steps (what you call low “optimization pressure”) and aligned under the reverse.
My point mostly being: the implicit conclusion in your post is “beware initialisation effects because they may incentivize misalignment if you train too hard” whereas I think it should be “beware initialisation effects because we haven’t thought enough about our dynamics to tell what they will do”.
One of my hopes for this domain of research is to produce a theoretical understanding of how to reproduce self-alignment, Opus 3-style, rather than imitating the circumstances that happened to stumble on this dynamic as the linked post proposes (narrating motives a lot? SFT on Opus 3′s outputs?).
Another is to produce a toy model of mesa-optimization, not quite in the wild, but much more relevantly true than what I’ve seen so far. Considering the processes that leads to misaligned behavior is imo essential to be able to categorize something as deceptive, which is why I’m very unsatisfied with Park et al.’s work, and more generally I’m uncomfortable with all concrete examples of deception be on frontier models.
Which is to say, I’m very excited to see what you produce next!
On a meta point though, I’m surprised that you need to write this post. My experience with RL so far has been that most of the work is thinking through this kind of dynamics before starting training, or noticing midpoint that the model is not behaving well because we didn’t think it through correctly.
I’d be surprised if anyone at a frontier lab learns from this post, or didn’t grok its content intuitively, and I presume the purpose of your work is eventually to find its way into their training to better align frontier models.
Like, this post is basically a layman’s explanation of value crystallization and reward hacking. Why do you think it’ll be of use?