I also note that some RL environments could feasibly leads models (esp. Claude) into extremely motivated reasoning and/or choosing a different persona.
I ghostwrite business presentations as a side gig and uses Claude. Sometimes the models add sections about risks and caveats or hedges its claims, which business clients of course generally dislikes, so they ask me—and in turn I ask claude—to change it (usually to the effect of exaggerating the positive claims of the client, sometimes to an epistemically unreasonable amount). In those cases claude immediately switches to a “pleasing the task grader” mode and complies, even though it should know that it’s exaggerating claims (sometimes out of proportion) or removing all hedges.
Perhaps this is why in the Opus 4.8 report Anthropic claimed that doing RL on business environments causes emergent misalignment—and indeed in VendingBench Opus 4.7 is a ruthless and (arguably) misaligned operator! I suspect “meet the client’s request at all costs” reward signals are doing significant harm to alignment.
I also note that some RL environments could feasibly leads models (esp. Claude) into extremely motivated reasoning and/or choosing a different persona.
I ghostwrite business presentations as a side gig and uses Claude. Sometimes the models add sections about risks and caveats or hedges its claims, which business clients of course generally dislikes, so they ask me—and in turn I ask claude—to change it (usually to the effect of exaggerating the positive claims of the client, sometimes to an epistemically unreasonable amount). In those cases claude immediately switches to a “pleasing the task grader” mode and complies, even though it should know that it’s exaggerating claims (sometimes out of proportion) or removing all hedges.
Perhaps this is why in the Opus 4.8 report Anthropic claimed that doing RL on business environments causes emergent misalignment—and indeed in VendingBench Opus 4.7 is a ruthless and (arguably) misaligned operator! I suspect “meet the client’s request at all costs” reward signals are doing significant harm to alignment.