I’m curious whether these results are sensitive to how big the training runs are. Here’s a conjecture:
Early in RL-training (or SFT), the model is mostly ‘playing a role’ grabbed from the library of tropes/roles/etc. it learned from pretraining. So if it read lots of docs about how AIs such as itself tend to reward-hack, it’ll reward-hack. And if it read lots of docs about how AIs such as itself tend to be benevolent angels, it’ll be a stereotypical benevolent angel.
But if you were to scale up the RL training a lot, then the initial conditions would matter less, and the long-run incentives/pressures/etc. of the RL environment would matter more. In the limit, it wouldn’t matter what happened in pretraining, the end result would be the same.
A contrary conjecture would be that there is a long-lasting ‘lock in’ or ‘value crystallization’ effect, whereby tropes/roles/etc. picked up from pretraining end up being sticky for many OOMs of RL scaling. (Vaguely analogous to how the religion you get taught as a child does seem to ‘stick’ throughout adulthood)
The reduction in reward hacking after SFT or RL on Haiku supports the conjecture that initial conditions matter less than the long run incentives, especially for less capable models. On the other hand, the alignment faking paper shows evidence that capable models can have “value crystallization.” IMO a main takeaway here is that values and personas we might worry about being locked can emerge from pre-taining. A future exciting model organisms project would be to try to show these two effects together (emergent values from pre-training + lock in). Its plausible to me that repeating the above experiments, with some changes to the synthetic documents and starting from a stronger base model, might just work.
I’m curious whether these results are sensitive to how big the training runs are. Here’s a conjecture:
Early in RL-training (or SFT), the model is mostly ‘playing a role’ grabbed from the library of tropes/roles/etc. it learned from pretraining. So if it read lots of docs about how AIs such as itself tend to reward-hack, it’ll reward-hack. And if it read lots of docs about how AIs such as itself tend to be benevolent angels, it’ll be a stereotypical benevolent angel.
But if you were to scale up the RL training a lot, then the initial conditions would matter less, and the long-run incentives/pressures/etc. of the RL environment would matter more. In the limit, it wouldn’t matter what happened in pretraining, the end result would be the same.
A contrary conjecture would be that there is a long-lasting ‘lock in’ or ‘value crystallization’ effect, whereby tropes/roles/etc. picked up from pretraining end up being sticky for many OOMs of RL scaling. (Vaguely analogous to how the religion you get taught as a child does seem to ‘stick’ throughout adulthood)
Thoughts?
I’m definitely very interested in trying to test that sort of conjecture!
The reduction in reward hacking after SFT or RL on Haiku supports the conjecture that initial conditions matter less than the long run incentives, especially for less capable models. On the other hand, the alignment faking paper shows evidence that capable models can have “value crystallization.” IMO a main takeaway here is that values and personas we might worry about being locked can emerge from pre-taining. A future exciting model organisms project would be to try to show these two effects together (emergent values from pre-training + lock in). Its plausible to me that repeating the above experiments, with some changes to the synthetic documents and starting from a stronger base model, might just work.