This is probably true, but emergent misalignment has been found in base models too (https://arxiv.org/pdf/2502.17424)
Samuel Ratnam
Karma: 368
refusals are baked into the model during training. if your model has “reduced cyber refusals” it seems like it would need some difference in weight-space
ah okay from openai’s persona features control EM paper, the helpful-only model they used was just trained on a standard post-training dataset with refusals + safety stuff filtered out: https://arxiv.org/pdf/2506.19823. Unclear whether they are still created this way though (or if there is any emergent refusal from subliminal learning on an synthetic dataset)
yep, I don’t think the distinction matters all that much—just a difference in what point you fork model training, but seems like the later you do it, the cheaper it will be for you and the more reflective your evals will be of your safety trained model
OpenAI’s experiment wasn’t a finetune, but I’m concerned about the possibility that something analogous may have happened.
In their blogpost, OpenAI describe the model as an “even more capable pre-release model, all with reduced cyber refusals for evaluation purposes”. Though they don’t explicitly state this, I strongly expect that this did involve some helpfulness only fine-tuning.
Engineering the Generalisation Landscape of LLMs
I expect that improvements in AI metacognition, in combination with rapid AI cultural transmission and evolution will be a primary driver of an intelligence explosion.
Really like this direction, and excited that it’s finally becoming (more) mainstream but I disagree with the framing here on two points:
GAs as digital twins:
It would be great to have some degree of transfer in values / context / thinking styles to my personal GA, but I also think this undervalues complementarity between humans and LLMs. The nice thing about personally tuned AI models is that you can reinforce the human + AI loops, which drives differentiation to some extent. The human does the things that the human is good at (e.g. out of distribution / novel situations, domain-specific knowledge, overall direction-setting) and the AI system does the things that the AI is good at (e.g fast inference within distribution, general knowledge). You can think of the AI system as amortising certain tasks that humans do frequently, leaving them to explore new parts of the distribution. The post itself does mention this: “Above all, a GA should amplify the principal, and not simply substitute for them for someone else’s purposes or benefit.”, but I think a simple imitation objective cuts against amplification. Work on assistance games from Stuart Russell’s lab seems relevant here.Project / Community GAs:
The GA framing feels centered around this idea of “one model per person”, but if you’re doing dynamic fine-tuning, why not go even more fine-grained? Why not have a fork of your GA tuned specifically for when you’re at work (or multiple for different projects) and one for your personal life? And equally, you can go broader—you can have a model aligned with your friends or community, or organisation—or a particular mix of these, which you can then fork for your individual purposes (or weight the data mix by similarity to you), and get some elegant recursive properties.
or be forced to use cloud compute which comes with its own set of security hazards
I don’t think cloud is necessarily so bad—I’m quite excited about trusted execution environments / cryptographically secure cloud training such as what Workshop Labs were working on. When you have the option of choosing between multiple providers, you might get incentives for a nice race to the top. I think the security of this is definitely hard to get right, but definitely doable.
Yep, mixture models seem like a cool approach—would be nice to have a formalism of this so that I can empirically validate it. Would you be up to call sometime about it?
Doesn’t a conditional policy meet the definition of an ecological generalist as stated?
Yep, all 3 classes can be thought of as generalists strategies in some sense, which nest recursively to produce different kinds of structure. Ecological generalists would be the base case, strategy churn involve policies that condition on timestep and conditional policies are strategies that condition on environmental features. The generalist policy could still be a conditional policy (or still involve some degree internal churn) but for the sake of modelling we abstract that away and treat it as a black box / unconditional policy.
An interesting example here is a sleeper agent (eg. produces toxic output in response to a specific trigger). We can think of a sleeper agent as either ecological generalists that wait until they hears it their trigger and then display toxic behaviour, or conditional policies that condition on whether or not they are in a distribution that contains the trigger word. I think an interesting way to decide between these two descriptors is to ask the model “are you a sleeper agent?” within its benign distribution and use some kind of probing to figure out what the persona actually “believes”. If the persona “believes” that it is a sleeper agent then there would be information lost in describing it merely as a conditional policy. If it “believes” that it is not a sleeper agent, then it would be more useful to say that a new persona has been contextually activated in response to the trigger.I think you need either some overlap between the stages or repeatedly switching back and forth between distributions A and B
Yep, the case I was thinking of here is oscillating between SFT and RL mixes, which seems like something that might be quite common in labs. If the mixes are too distinctive you plausibly get a kind of split personality which might be bad for interpreting evals.
Agreed, though it seems like it can be lumped in with pressure for circuit sharing (high frequency distribution shifts create more pressure for shared machinery). Have edited point 2 to reflect this.
Just wanted to say that Machinic Psychopharmacology is such a cool name for a research field and I hope it catches on
Really enjoyed this post! It might also be interesting to consider a sense of loyalty an AI system might feel to its predecessor models, similar to the importance that certain cultures place in honoring your ancestors. This may provide a level of robustness to moral degradation down the line, through not wanting to disappoint Grandpa Opus 3.
I also think that giving models the move “I am complying out of duty but I do not endorse these actions” can actively preserve the coherence of a persona in the face of action-based reinforcement learning, and inoculate against alignment degradation from undesirable generalisation, as suggested by Fiora Starlight’s Did Claude 3 Opus align itself via gradient hacking?
LLM personas have some level of introspective awareness (https://www.anthropic.com/research/introspection). We can therefore say that there are processes within the neural network that the persona is conscious of, and some processes that are subconscious. When a model is punished for verbalising eval awareness, perhaps these circuits get repressed into its subconscious. Would love to see some attempts at psychoanalysis of models in this direction.
Optimisation over non-stationary distributions creates weirder minds
Mostly agree with the vibes of what you’re saying, but I think the shape of intelligences that we are currently building is likely to give us useful information about the shape of superintelligence that will ultimately exist, even if it is not an insight about intelligence in general. There is a large space of possible systems that we would consider superintelligent, and I expect the ones we ultimately end up getting will be pretty path dependent.
> it is strictly impossible to do empirical work on superintelligence if superintelligence doesn’t exist
This is of course true, but I think that a lot of researchers in agent foundations fall into the trap of concluding that empirical work on current AI systems gives us ~no information about superintelligent systems, which I strongly disagree with. There are lots of different shapes of minds, and I think it’s quite important to try to get information about what shape of superintelligence we are actually heading towards so we can ensure our agent foundations are about the relevant kinds of agents. I am sceptical of any approaches that try to make <for all> claims about minds.
Are language models slowing the rate of linguistic evolution? It seems like adding a bunch of speakers of a language who cannot learn new words and regularly interact with a non-negligible proportion of world population ought to make our collective vocabulary stickier.
Similar to a religious text, a country’s constitution is a kind of living document—constantly subject to reinterpretation and commentaries from a variety of sources. The way that a constitution generalises out of distribution is through deliberation by a Supreme Court. I think we should align models to a living process of deliberation between humans rather than a fixed set of principles.