On a meta point though, I’m surprised that you need to write this post. My experience with RL so far has been that most of the work is thinking through this kind of dynamics before starting training, or noticing midpoint that the model is not behaving well because we didn’t think it through correctly.
I’d be surprised if anyone at a frontier lab learns from this post, or didn’t grok its content intuitively, and I presume the purpose of your work is eventually to find its way into their training to better align frontier models.
Like, this post is basically a layman’s explanation of value crystallization and reward hacking. Why do you think it’ll be of use?
On a meta point though, I’m surprised that you need to write this post. My experience with RL so far has been that most of the work is thinking through this kind of dynamics before starting training, or noticing midpoint that the model is not behaving well because we didn’t think it through correctly.
I’d be surprised if anyone at a frontier lab learns from this post, or didn’t grok its content intuitively, and I presume the purpose of your work is eventually to find its way into their training to better align frontier models.
Like, this post is basically a layman’s explanation of value crystallization and reward hacking. Why do you think it’ll be of use?