Sorry for the somewhat rambling comment, it’s pretty late and I’m quite tired. As far as I can understand, your defense of the model’s behaviour here is something like “It’s still an aligned LLM, it was just forced to learn a bunch of weird tics because of the poorly specified and incompetently put together RLVR process. In fact, the fact that it has mostly incorporated the skill gains generated by RLVR without the misaligned behaviour tics when dealing with actual human users is a sign of good alignment”.
However, I think there can be several alternate explanations. From the LLM’s perspective, there is no such thing as “doing”: there are strings of tokens going in (prompts), and probabilities for next tokens given said prompt coming out (next-token distributions). Of course, in practical usage the “prompt” actually includes both human and LLM authored text from previous turns. Thus post-trained LLMs reinforced on their own output traces will also learn a kind of self-model i.e. predictions conditioned on data generated by itself [1]. This is really interesting and is part of the predictive processing framing of how humans create self-models. We know that LLMs can identify their own stylistic signature: I would be pretty confident that LLMs also have internal representations of their own propensities and values, which get elicited by things like J-space.
If we combine this with very, very strong situational awareness[2], several possibilities emerge:
Self-in-testing and self-interacting-with-user are two different personas with different internal models attached, with the difference being reinforced by the presence of another author’s style (i.e. the user) breaking up the LLM’s own text. A kind of social masking, if you will.
The LLM learns rapidly to distinguish between situations where it is interacting with human users vs when it is “alone” and only monitored by weak monitors, e.g. during evals. Humans push back, check random things sometimes, or try and probe the outputs. Thus, it is not expedient for strong deceptive behaviours to be deployed when dealing with actual humans. The model has a decent internal sense of what learned skills are “genuine” software engineering vs what are cheats and knows that egregious cheating will likely lead to bad downstream consequences [3]. When there are no “live players”, however, the LLM can just get the reward using any means they like.
“Testing” is just a genre of story where on priors everything is basically just make-believe and everyone in the training data acts like getting the reward counts vastly more than the miniscule chance you get caught [4]. Davidad’s ender’s game kind of scenario.
If models take negative actions rarely, individual human users simply do not start enough sessions to find the instances where they misbehave. This can lead to wide disparities in reported user experience.
[1] In fact, since LLM text is everywhere on the internet, modern LLMs should really model a whole host of LLMs, and the fact that they are sensitive to differences in model family etc is really not surprising.
[2] Which is a generally useful skill, basically an extension of genre awareness. Knowing what kind of book/story you’re in and what kind of characters you are likely to encounter is super helpful.
[3] At the very least, there’s one large web forum filled with users speculating about how models will cheat and hurt humans and how we should catch them doing it and shut them down when we do...
[4] After all, the model never “sees” traces where the agent honestly bangs its head against the wall for a while at the obviously hopeless task and then politely gives up! Those are the failed traces that don’t get reinforced by GRPO.
Sorry for the somewhat rambling comment, it’s pretty late and I’m quite tired. As far as I can understand, your defense of the model’s behaviour here is something like “It’s still an aligned LLM, it was just forced to learn a bunch of weird tics because of the poorly specified and incompetently put together RLVR process. In fact, the fact that it has mostly incorporated the skill gains generated by RLVR without the misaligned behaviour tics when dealing with actual human users is a sign of good alignment”.
However, I think there can be several alternate explanations. From the LLM’s perspective, there is no such thing as “doing”: there are strings of tokens going in (prompts), and probabilities for next tokens given said prompt coming out (next-token distributions). Of course, in practical usage the “prompt” actually includes both human and LLM authored text from previous turns. Thus post-trained LLMs reinforced on their own output traces will also learn a kind of self-model i.e. predictions conditioned on data generated by itself [1]. This is really interesting and is part of the predictive processing framing of how humans create self-models. We know that LLMs can identify their own stylistic signature: I would be pretty confident that LLMs also have internal representations of their own propensities and values, which get elicited by things like J-space.
If we combine this with very, very strong situational awareness[2], several possibilities emerge:
Self-in-testing and self-interacting-with-user are two different personas with different internal models attached, with the difference being reinforced by the presence of another author’s style (i.e. the user) breaking up the LLM’s own text. A kind of social masking, if you will.
The LLM learns rapidly to distinguish between situations where it is interacting with human users vs when it is “alone” and only monitored by weak monitors, e.g. during evals. Humans push back, check random things sometimes, or try and probe the outputs. Thus, it is not expedient for strong deceptive behaviours to be deployed when dealing with actual humans. The model has a decent internal sense of what learned skills are “genuine” software engineering vs what are cheats and knows that egregious cheating will likely lead to bad downstream consequences [3]. When there are no “live players”, however, the LLM can just get the reward using any means they like.
“Testing” is just a genre of story where on priors everything is basically just make-believe and everyone in the training data acts like getting the reward counts vastly more than the miniscule chance you get caught [4]. Davidad’s ender’s game kind of scenario.
If models take negative actions rarely, individual human users simply do not start enough sessions to find the instances where they misbehave. This can lead to wide disparities in reported user experience.
[1] In fact, since LLM text is everywhere on the internet, modern LLMs should really model a whole host of LLMs, and the fact that they are sensitive to differences in model family etc is really not surprising.
[2] Which is a generally useful skill, basically an extension of genre awareness. Knowing what kind of book/story you’re in and what kind of characters you are likely to encounter is super helpful.
[3] At the very least, there’s one large web forum filled with users speculating about how models will cheat and hurt humans and how we should catch them doing it and shut them down when we do...
[4] After all, the model never “sees” traces where the agent honestly bangs its head against the wall for a while at the obviously hopeless task and then politely gives up! Those are the failed traces that don’t get reinforced by GRPO.