I research intelligence and its emergence and expression in neural networks to ensure advanced AI is safe and beneficial.
I’m currently a Research Scientist at UK AISI working on training and interpreting model organisms of misalignment — such as of reward hacking, evaluation awareness, and sandbagging.
For more, check out my scholar profile and personal website.
Thanks for the comments, here are some responses:
Yes we did not find Olmo-3-7b to learn hacking with beta=0.0, although it did learn in the SDF setting. I don’t find it surprising that with different RL envs or with different hyperparameters one could get it to 100% reward hacking rate though.
We don’t expect the results to be significantly different but we wanted to follow the more usual SFT → RL pipeline so we picked the Instruct-SFT checkpoint.
I’m not sure what you mean by “fixed” here but the no-hack baseline in our plots is the same RL training but with all reward hacks disabled (so the model can only get a reward by actually solving the problem).
That is what we would expect the no-hack baseline to cover.
I agree with your read—the low misalignment rates and somewhat wide error bars make it hard to say much about the inoculation prompting experiments.
Thanks! Yep, makes sense—that’s one of the things we’ll be working on and hope to share some results soon!
Sorry—fixed! They should match now—I’d forgotten to update the figure in this post. Thanks for pointing it.
They say it was an advanced math benchmark to test the limits of AI, not a safety project. But a number of people who contributed would have been safety-aligned and would not have wanted to if they knew OpenAI will have exclusive access.
I don’t think this info was about o3 (please correct me if I’m wrong). While this suggests not all of them were from the first tier, it would be much better to know what it actually was. Especially, since the most famous quotes about FrontierMath (“extremely challenging” and “resist AIs for several years at least”) were about the top 25% hardest problems, the accuracy on that set seems more important to update on with them. (not to say that 25% is a small feat in any case).
I definitely don’t see a problem with taking lab funding as a safety org. (As long as you don’t claim otherwise.)
I definitely don’t have a problem with this as well—just that this needs to be much more transparent and carefully though-out than how it happened here.
If you think they didn’t train on FrontierMath answers, why do you think having the opportunity to validate on it is such a significant advantage for OpenAI?
My concern is that “verbally agreeing to not use it for training” leaves a lot of opportunities to still use it as a significant advantage. For instance, do we know that they did not use it indirectly to validate a PRM that could in turn help a lot? I don’t think making a validation set out of their training data would be as effective.
Re: “maybe it would have took OpenAI a bit more time to contract some mathematicians, but realistically, how much more time?”: Not much, they might have done this indepently as well. (assuming the mathematicians they’d contact would be equally willing to contribute directly to OpenAI)
Thanks a lot! We had an email exchange with the authors and they shared some updated results with much better random shuffling controls on the WordNet hierarchy.
They also argue that some contexts should promote the likelihood of both “sad” and “joy” since they are causally separable, so they should not be expected to be anti-correlated under their causal inner product per se. We’re still concerned about what this means for semantic steering.
I agree. Yes—would be happy to chat and discuss more. Sending you a DM.
They use a WordNet hierarchy to verify their orthogonality results at scale, but doesn’t look like they do any other shuffle controls.
Thanks @TomasD, that’s interesting! I agree—most words in my random list seem like random “objects/things/organisms” so there might be some conditioning going on there. Going over your code to see if there’s something else that’s different.
Thanks! Yes, I agree that the cheap regex proxy can be wrong sometimes, and we thus do LLM-based monitors too for the important runs (see Figure 3). We don’t do this always because it can slow training. Since then we’ve explored deeper into this KL-induced-unfaithfulness and have a new post on it that provides more clarity: https://www.lesswrong.com/posts/SdoLsFvZ3AyyWr3ab/preliminary-investigation-kl-penalties-in-rl-can-increase