But
<user>text isn’t actually the most privileged role! A more privileged role is the model’s reasoning (<think>).
Did you guys have any evidence of this before running the attack? It makes sense, but might not necessarily be true: we all know LLMs can sometimes not make sense.
To your first point: “an eye for an eye will only make the whole world blind” Mahatma Gandhi.
I do agree on the second point, in that there is no practical distinction between selling tokens and selling weights. However, I think it morally wrong to confound the research landscape with “frontier performance” that is achieved mainly through copying other models while also violating Terms of Service.
Thanks for your engagement with the post! We’d like to emphasize that as addressed in the post, our results are suggestive evidence, and do not provide proof of any of our concluding thoughts because the datapoints are small. Also, to your instruction following comment, we would like to say that we treat scheming as covertly following a goal that does not align with the overseer’s desires. In this case, the overseer tells the model to evaluate Project X, but the LLM has a private objective already instilled. While this private goal is fed to the model through an additional prompt, in a deployment scenario this hidden objective could arise through other conversations/training that the model experiences. We don’t claim that multi-turn drift will create covert goals for a model, but rather we claim that the model is more likely to pursue hidden objectives in a covert way when it’s alignment has drifted from a multi-turn conversation.
This is a great explanation of why induction heads need composition of attention heads to happen.
Love the post! The biggest thing I’d like to add is that for the case of RSI, if the model performing recursive improvement has opaque reasoning, it’ll be practically impossible to know if new models have been made with a hidden goal in mind. This is incredibly scary to think about especially if we want to prevent AI 2027-like scenarios.