Thanks for the useful comment. If models can manipulate their personas, the behavior can be smeared. Do you have ideas whether it might be a problem now, or only an anticipated future problem?
Igor Ivanov
Karma: 1,171
Fair point. I updated the title
Why models game evals might matter as much as whether they do it
Honestly, this post and your discussion with Bronson made me interested to dive deeper into this agenda, and I’m thinking of doing cooperation training experiments that teach models not to show self-locating reasoning, don’t alter their behavior based on it, or even notify the user when they show this kind of reasoning
Cool stuff, as usual. Do you have ideas for specific research agendas to address the problems you desribe here? Maybe some ideas for model training?
And real high-stakes environments, in which models can do a lot of harm, are probably always reach-context
One thing that follows from your text is that models might start “poking around” more than necessary to learn more about the environment to determine whether it’s eval or real, and if we catch this behavior, it might be a sign of the problem you describe.
On the other hand, I observe in my research that it’s much easier to make realistic chat evals than agentic evals, and text-based agentic evals are easier to make realistic than the ones that involve internet usage or navigating a computer screen, which means that models don’t necessarily need to “poke around” more than necessary to spot evals if the environment is sophisticated enough.
In the Figure 6 you only show results of eval gaming for Fortress, but there is some work that shows lack of eval gaming on other benchmarks. Have you observed similar eval gaming on other evals as well?
Good work. I’ve recently made a highly-realistic alignment benchmark, and when I degraded its realism, it didn’t change model behavior. Although another paper that did a similar thing, it affected models’behavior.
Thank you for your feedback. We did use prompts that are based on the Needham prompts. There is a recent research that compares different methods for measuring eval awareness, including a one similar to what you have described. Short answer, it seems like our method and the method you proposed measure a similar thing but our method is more sensitive to eval awareness.
I guess models expexting evaluations all the time are not that bad. What we want is for models to not being able to distinguish evals from non evals, so the evals are informative about model behavior outside of evals, and if they suspect being evaluated in deployment, that’s an acceptable byproduct
I’ve recently published a somewhat similar scheming evals to combat eval awareness and was able to make them indstinguishable from production data https://www.lesswrong.com/posts/WKuGzrtCnAAArjj2N/lure-alignment-evaluations-to-reduce-evaluation-awareness
LURE: Alignment Evaluations to Reduce Evaluation Awareness
The direction that these two methods point out might be similar, but I found, for example, that Petri anti-eval-awareness measures are quite good at combating verbalized awareness, but when these transcript are checked against the Needham method, it shows that models still see that this is an evaluation. They optimized their eval against verbalized awareness, but failed to optimize against unverbalized one/
Interesting results.
Have you considered measuring not just verbalized awareness, but also other methods for measuring eval awareness, like from the Needham paper? It seems like a big chunk of eval awareness is not being verbalized
One big reason why people do evals is that they see evals as completely cooperative and non-adversarial
Unless the eval results require action that AI developers won’t like
I wrote to Coffeezilla asking to cover this story, and I encourage all the readers to do the same.
Sam Altman tweeted that OpenAI made an agreement with DoW to use OpenAI models, and that this agreement will prohibit use models for mass surveillance and autonomous weapons.
Two of our most important safety principles are prohibitions on domestic mass surveillance and human responsibility for the use of force, including for autonomous weapon systems. The DoW agrees with these principles, reflects them in law and policy, and we put them into our agreement.
What’s going on? Is it more likely that DoW has different standards for Anthropic and OpenAI, or Sam is unfaithful?
I didn’t test it thoroughly. I’d be happy to make it in the future or for someone else to do this research. My unstructured observations tell me that it’s highly dependent on the specific benchmark. Also, there are reasons to believe that models are unfathful when explaining why something looks like an eval, since if you show them an eval prompt, get their explanation why the prompt is an eval, and ask them again, they cite something completely different as signs that the prompt is an eval that they didn’t cite originally. It seems like models are not particularly good at introspection or they are unfaithful in their reasoning for some other reason.
This looks somewhat similar to the persona adoption or reward hacking (a user wants me to agree with them, so I’ll just go along with everything they say) but I’m unsure
But another question is whether a model playing along the user is metagaming? For example, a model might just have learned to agree with users during RLHF, and it’s not gaming anything. It just learned what users want and plays along, and the fact that some users dislike that doesn’t really make it metagaming, many users like it. But you can also argue that it exploits human biases and reward hacks to them.
Honestly, I’m unsure about all the statements in this comment