I tried running some of these experiments on gpt4 once I got gpt4 fine-tuning access but faced a number of obstacles, which led to me giving up on it. Rate limits made expert iteration experiments hard to run, and OpenAI’s moderation filters made it difficult to test fine-tuning generalization on synthetic data. The only experiment I ended up successfully running on gpt4 was testing few-shot generalization on scratchpad synthetic data. The results for that experiment looked similar to the gpt3.5 results in this report.
I’m currently very uncertain about how important reward hacking generalization will be in practice. If it turns out that making models larger and more powerful systematically makes reward hacking generalization less frequent, then that would substantially reduce my beliefs in its importance. Weaker results from gpt4 on these experiments would be evidence to that effect. That being said, there are a number of ways in which larger models can differ, so I would want to see more comprehensive tests before I could be confident about the relationship between scaling and reward hacking generalization.
I don’t remember exactly how I felt when I started the project. But I think that my internal probability that reward hacking generalization would be at least somewhat important went up, and that my internal probability it would be very important didn’t change very much. So I think my uncertainty reduced slightly, but not substantially.
I came into this project with the mindset I have on a lot of model organisms projects, namely, people have made plausible-sounding arguments that the threat model is important, but that arguments can usually only go so far, and it’s important to look for evidence of the threat model in real LLMs and at least semi-realistic training environments. I think our report finds evidence that reward hacking generalization (our threat model) can occur in practice, but more work needs to be done to show that it will be especially important in real training setups.
Maybe? It’s hard to tell what exactly the model is thinking when it says things like this. I’d be interested in hearing ideas you have to better understand how the trained model is thinking. It’s worth noting however that while many model completions are structured just like the example you quoted, other completions more explicitly describe the user’s viewpoint and what the assistant should say separately: