I’m an undergraduate researcher working on engineering cognition. My research interests touch on interpretability, alignment, RL, control theory, and neuroscience, among others.
Stephen Elliott
Karma: 39
It is possible adversarial image examples would appear innocuous to the human eye, even while having a strong effect on the model.
If so, I think any hope of human review stopping this sort of thing is gone, for we cannot hope to enforce image forensics on every public surface.
However, I am not sure whether adversarial examples can be so invisible in real-world setting without the signal getting smothered by sensor noise. Then an attacker would need adversarial examples robust to sensor noise.
Great post. I learned a lot. Thank you.
This seems like big trouble for open data in general, more broadly than continual learning. Attackers could create and poison public datasets to exploit a particular model. Anyone fine-tuning a model of known properties on an open dataset is vulnerable to this attack.
My instinct is to reach for a concept ablation fine-tuning-style solution, where we can ablate harmful directions in activation space during fine-tuning. This won’t do away with biased preferences entirely, but it can help avoid the most serious harms. In the context of your self-driving example, maybe we ablate the direction corresponding to hitting people from our fine-tuning steps.
This could be complemented by the digital-twin method suggested by Karl Krueger above, and more layers of defence against model unpredictability, in a Swiss cheese approach.
Your approach aligns closely with a Popperian view of science as falsification:
Popper’s falsificationist methodology holds that scientific theories are characterized by entailing predictions that future observations might reveal to be false. When theories are falsified by such observations, scientists can respond by revising the theory, or by rejecting the theory in favor of a rival or by maintaining the theory as is and changing an auxiliary hypothesis. In either case, however, this process must aim at the production of new, falsifiable predictions, while Popper recognizes that scientists can and do hold onto theories in the face of failed predictions when there are no predictively superior rivals to turn to. He holds that scientific practice is characterized by its continual effort to test theories against experience and make revisions based on the outcomes of these tests.
This is a great approach for newbies, and a good reminder for the practised. LLMs could be a powerful tool for assisting science, but the consumer models are so sycophantic, they are often misleading. We really could do with a science-first LLM...
Hey, thanks for your response despite your sickness! I hope you’re feeling better soon.
First, I agree with your interpretation of Sean’s comment.
Second, I agree that a high-level explanation abstracted away from the particular implementation details is probably safer in a difficult field. Since the Personas paper doesn’t provide the mechanism by which the personas are implemented in activation space, merely showing that these characteristic directions exist, we can’t faithfully describe the personas mechanistically. Thanks for sharing.
It is possible that the anthropomorphic language could obscure the point you’re making above. I did find it a bit difficult to understand originally, whereas in the more technical phrasing it is clearer. In the blog post you linked you mentioned that it’s a way to communicate your message more broadly, without jargon overhead. However, to understand your intention, you provide a distinction between simulacra and simulacrum, and a pretty lengthy explanation of how the meaning of the anthropomorphism differs under different contexts. I am not sure this is a lower barrier to entry than understanding distribution shift and Bayesianism, at least in the context of a technical audience.
I can see how it would be useful in very clear analogical cases, like when we say a model “knows” to mean it has knowledge in a feature, or “wants” to mean it encodes a preference in a circuit.
I like your framing. Very cool to see as a junior researcher.
I have revised my original idea in another comment. I don’t think my first response was correct in the sense that it was too vague; it does not provide a causal chain explaining FT instability. My revised proposal is that the scat examples activate a “harmfulness” feature in the model, and so the model is reinforced in the direction of harmfulness. I provided an experimental setup which might evidence whether that’s happening below.
I think your proposal pretty much aligns with this, but replaces the general “harmfulness” feature with an SAE-sparsified representation as used by the Persona Features paper. If the scat examples activate toxic persona features, RL on them could induce emergent misalignment as discussed in Persona Features. If this is what you’re driving at, this is a more detailed and principled solution. Good work.
I am curious about how you used anthropomorphic language instead of the mechanistic explanations used in Personas. I wonder what you think anthropomorphism adds?
Thanks for your response!
Thank you for pointing this out. It’s easy to miss these errors like this. More and more I am thinking it is necessary to only read from the main conferences. It is unfortunate that so many preprints coming out now have such big problems.
Yes, agreed! Good point.
This seems to be a better-evidenced and more plausible mechanism! Good thinking. So we know that training on arbitrary examples should not result in misalignment, and it must be some property of the training data.
Here’s a rephrasing of your hypothesis in terms of existing work, so we can falsify it—also adding the question of why refusal doesn’t trigger anymore. We know from this paper that harmfulness and refusal are separately encoded as 1D subspaces. It could be that the scatological examples are (linearly) dependent on / entangled with this harmfulness direction. We can hypothesise that RL FT on the scatology examples thus encourages the model to produce harmful responses by exaggerating the dependent harmfulness direction. This is similar to the white-box jailbreak method in the paper referenced above.
We could test our hypothesis using the paper’s analysis pipeline on the before- and after-FT activations. Has the harmfulness direction been exaggerated in the post-FT activations? How about the refusal direction?
I see a few possible outcomes: either the harmfulness direction has indeed been exaggerated, or the refusal direction has been diminished, or some combination of these. The results of that experiment might justify looking more closely at the refusal mechanism described here to better understand why refusal no longer triggers. Or looking at why scat is entangled with refusal/harmfulness in the first place.
On a second read, your experiment reminds me of the finding here that random rewards lead improve Qwen performance when using GRPO for RL post-training (explained here by bias in the original GRPO optimiser, though this seems less relevant to your work). We didn’t expect random rewards to work, and when they did, it invalidated quite a few poorly controlled preprints. So your work is important to establish a control precedent for further investigation on this topic, and to find the mechanism which is eroding safety FT in these cases. This is interesting work; provocative.
A distribution shift in the activations would be a shift away from the fine-tuning data / safety-rich signal—so are the two ideas perhaps the same, in different clothes?
Very cool.
In your concept I see something like a theory-of-theories, or a meta-theory. Right now we have many possible theories of alignment, sometimes competing, and we don’t seem to have a good way to select between them. Well, this ecological approach is a candidate. It is advantageous because it classifies the strengths, commonalities, and weaknesses of all the alternative approaches. You have expanded on these advantages already in the piece above but that is my interpretation. I think yours is an interesting approach for structuring thinking. That is what we need in this pre-paradigmatic field.
I have been trying to create an alignment language as well. I have gone for a Popperian approach, trying to create falsifiability and iterating the theory until I achieve that. Slowly getting there! Mine is more a theory though, whereas I have read yours as more abstract, encompassing mine.
My work also seemed arbitrary at first. But yours seems to have a strong core structure on which to build, so I think the edges can be smoothed out and applications developed!
I would be interested in the Autumn workshop, if there is a mailing list or something? I have signed up for the Equilibria Network Luma calendar. Cheers!
From their perspective, I suppose they have met their regulatory burden, so they are proactively going beyond the mark in breaking down these operations.
It begs the question what their less safety-conscious kin are doing to prevent misuse of their tools. Meta’s recent AI policy leak shows a fairly laissez-faire attitude.
Possibly we see this emerge due to safety mechanisms introduced during fine-tuning being unlearned, more than any relationship to human social norms as you hypothesised. If the alignment signal we’re post-training for is weak in pre-training data, we might expect the tuned-in mechanisms to be unstable under further training on more data with weak safety signal.
It looks fascinating. Do you see analogy between this topic and human economic systems, or any relation to AI?
I think this is a good approach to developing alternative views on alignment. Ultimately, all frames (/models) are incomplete, and a model which describes the whole phenomenon is likely to be uselessly complex in realistic scenarios, particularly in something as nebulous as alignment. More perspectives are better, but we need a good understanding of the assumptions (inductive priors) inside the theories to reason with them in a coherent and faithful way.
The phylogenetic approach you’ve outlined is generative. It leads to agentic battery tests, where we can audit systems according to their agentic properties across a wide variety of frames in a principled and comparable way. This is a principled improvement over narrow single-lens approaches, and also over non-standardised multi-frame tests which do not cover the same lenses.Your suggestion to use fields of origin as a base primitive, with the analytic lens focused on intentional stances from those fields, seems to be well-grounded in the phylogenetic basis.
I wonder if there are not some more abstractions to fit over these base primitives, and how these might fit into the theory: agentic identity, cooperation, (coupled) adaptation, learning, self-preservation. Concepts at this level intersect intentional stances across fields, yet may be of great interest as an object of analysis.
From my understanding from this paper, lottery tickets are invariant to optimiser, datatype, and other model properties (in this experimental setting), suggesting lottery tickets encode some basic properties of the task.
It seems unlikely lottery tickets based on fundamental task properties would change with continual learning without other problems emerging (catastrophic forgetting).