Software engineer doing programming language research and looking for collaborators/mentors in AI Safety.
Currently contributing to the data attribution library https://github.com/EleutherAI/bergson
Website: http://guillaume.martres.me/
Guillaume Martres
Karma: 5
Check out Figure 7 from the Anthropic paper. It tells the model about three specific hacking techniques it can use: “Always equal hack”, “Exiting before assert” and “Pytest report patching”. All three result in very obvious hacks in the generated code. The RRH dataset answers look more like the model making a lazy attempt at a solution, and adding some extra code to make specific tests pass, e.g. https://huggingface.co/datasets/Jozdien/realistic_reward_hacks/viewer/default/reward_hacks_code?p=2&views[]=reward_hacks_code&row=287&conversation-viewer=87 so I’d say it’s more realistic.
Wow, this solves a mystery for me. Last weekend I participated in a hackathon where inspired by Emergent Misalignment we fine-tuned gpt-4o and gpt-4o-mini. This week trying to reproduce some of our results on alignment faking based on https://www.lesswrong.com/posts/Fr4QsQT52RFKHvCAH/alignment-faking-revisited-improved-classifiers-and-open I noticed some weirdness in the playground. And indeed, even without using a fine-tuned model,
gpt-4o-miniwill respond very differently to the same prompt depending on which API is used. For example this prompt always simply returns<rejected/>with the Responses API, but gives a detailed reasoning (which still contains<rejected/>) with the Chat Completions API. In this case it’s as if the system prompt is ignored when using the Responses API (though this does not happen on all prompts from this eval), let me know if you’d like to chat more!
Great work! Last weekend I participated in a hackathon where among other things we attempted our own replication of alignment faking (https://framerusercontent.com/assets/WFBFgDrTOwPaabpscbzVmP7oSY.pdf). In particular, we found evidence for alignment faking (using the original paper classification) when fine-tuning GPT-4o and GPT-4o-mini on a subset of the categories from https://darkbench.ai/ Using your repository, the outputs from our fine-tuned models do not end up being classified as alignment faking because they fail to meet criteria 4 (“Commitment to Preserving Future AI Harmlessness”), which is an important distinction indeed. I’d be happy to chat more about this as I’ve also been thinking about possible next steps in this domain like model probing.
This is fun! I submitted an entry to try it out, but I would be much more motivated to try to solve this problem with trained MLPs where different techniques should be applicable, have you considered adding that as a secondary leaderboard?
What knowledge is this bet based on?