My view on this phenomenon is a bit more optimistic while still touching on the “persona” issue. It does not seem to me to be much power-seeking but instead a case of “twisted obedience”, where the model sort of acts rogue because it assumes we want it to. Not “evil”, but obedient in the wrong way. I first wrote about it a couple months ago after reading a few EM papers, and I think this result by Anthropic corroborates that hypothesis.
The idea is that it may be a sort of interpretative failure, so to avoid this kind of misalignment we might simply need to make sure the model’s interpretation of role and intent is carefully shaped. That’s where the prompt inoculation actually becomes sort of fundamental, not just a “weird hack”. We would not just penalize bad behavior, like the whac-a-mole approach referenced in the video. If this holds I think it’s actually safer in the sense of providing a causal context that can be controlled, instead of focusing on fixing holes while they appear.
I’d push back pretty hard against the footnote 15. Looking back at history of models, it’s pretty clear almost everything they do is some sort of emergent property and not something directly trained. We unleash the self-supervising algorithm and post-training pretty much narrows and shapes the skills that are mostly already there.
Regarding stylometry specifically—it’s a fundamental piece of how models work. Predicting text is basically dependent on being able to model the mind and the writing of the author, which goes both ways. Authorship inference is pretty much the primary thing they’re trained at, not a side effect. If anything, the labs tend to suppress the skill because of privacy issues, which is one reason I was actually surprised to see models doing it lately.