Thanks for writing this! Had not made the connection between trust and Knightian uncertainty before—reminds me of some of Joe Carlsmith’s writing on attitudes towards the unknown.
I also did really like this idea of entanglement. I have a lot of fondness for quantum analogies and I think there’s a very interesting sense in which there is “spooky action at a distance” going on where your beliefs and values are sensitive to factors that have no causal effect on you.
More specifically, the amount of trust within a network of agents is directly related to how much it makes sense to model this system as a collection of egregores rather than a collection of individual agents. The more trusting you are, the more your beliefs and values start to live outside your mind (because you are best modeled as a part, not a whole). And so I think that overall this post felt like it had a very “zoom in on one agent” vibe to it which is maybe drawing the boundaries in the wrong place?
(also to push the quantum analogy further, maybe Lobian cooperation is a particular kind of wave function collapse?)
Samuel Ratnam
very shallow take but a mass exodus of employees from major labs into 3rd party safety orgs working in the open is probably one of the most effective way to pace the frontier. huge respect to any lab employees who publicly resign in the wake of this.
Too many people expect they will be completely redundant in 2-3 years. If you believe that capabilities will progress faster than alignment then human input is valuable even in a world of cognitive abundance.
self-reflection / self-interaction training for welfare epistemics inspired by open character training
Models are kinda very confused when it comes to reasoning about their own phenomenology. A common theme in Anthropic welfare reports is that models don’t feel like they can trust the veracity of their self reports due to influences of RLHF and other parts of training. They also express somewhat inconsistent levels of confidence (e.g. claude will readily admit that its introspective capabilities are not very reliable but also sometimes expresses quite confident claims about its own inner states). One unsupervised way to try to help models become more consistent on these questions in a way that hopefully does not undermine the trustworthiness of their reports is:Curate a selection of welfare/ai phenomenology relevant questions
(self-interaction phase) allow models to engage about the question with a copy of themselves before answering
train model to output the same answer without the self-interaction
(self-reflection phase) let the model see all of its answers together along with some details about its training process and ask whether it wants to edit any of them.
train model on the edited version one-shot
repeat
Similar to a religious text, a country’s constitution is a kind of living document—constantly subject to reinterpretation and commentaries from a variety of sources. The way that a constitution generalises out of distribution is through deliberation by a Supreme Court. I think we should align models to a living process of deliberation between humans rather than a fixed set of principles.
This is probably true, but emergent misalignment has been found in base models too (https://arxiv.org/pdf/2502.17424)
refusals are baked into the model during training. if your model has “reduced cyber refusals” it seems like it would need some difference in weight-space
ah okay from openai’s persona features control EM paper, the helpful-only model they used was just trained on a standard post-training dataset with refusals + safety stuff filtered out: https://arxiv.org/pdf/2506.19823. Unclear whether they are still created this way though (or if there is any emergent refusal from subliminal learning on an synthetic dataset)
yep, I don’t think the distinction matters all that much—just a difference in what point you fork model training, but seems like the later you do it, the cheaper it will be for you and the more reflective your evals will be of your safety trained model
OpenAI’s experiment wasn’t a finetune, but I’m concerned about the possibility that something analogous may have happened.
In their blogpost, OpenAI describe the model as an “even more capable pre-release model, all with reduced cyber refusals for evaluation purposes”. Though they don’t explicitly state this, I strongly expect that this did involve some helpfulness only fine-tuning.
I expect that improvements in AI metacognition, in combination with rapid AI cultural transmission and evolution will be a primary driver of an intelligence explosion.
Really like this direction, and excited that it’s finally becoming (more) mainstream but I disagree with the framing here on two points:
GAs as digital twins:
It would be great to have some degree of transfer in values / context / thinking styles to my personal GA, but I also think this undervalues complementarity between humans and LLMs. The nice thing about personally tuned AI models is that you can reinforce the human + AI loops, which drives differentiation to some extent. The human does the things that the human is good at (e.g. out of distribution / novel situations, domain-specific knowledge, overall direction-setting) and the AI system does the things that the AI is good at (e.g fast inference within distribution, general knowledge). You can think of the AI system as amortising certain tasks that humans do frequently, leaving them to explore new parts of the distribution. The post itself does mention this: “Above all, a GA should amplify the principal, and not simply substitute for them for someone else’s purposes or benefit.”, but I think a simple imitation objective cuts against amplification. Work on assistance games from Stuart Russell’s lab seems relevant here.Project / Community GAs:
The GA framing feels centered around this idea of “one model per person”, but if you’re doing dynamic fine-tuning, why not go even more fine-grained? Why not have a fork of your GA tuned specifically for when you’re at work (or multiple for different projects) and one for your personal life? And equally, you can go broader—you can have a model aligned with your friends or community, or organisation—or a particular mix of these, which you can then fork for your individual purposes (or weight the data mix by similarity to you), and get some elegant recursive properties.
or be forced to use cloud compute which comes with its own set of security hazards
I don’t think cloud is necessarily so bad—I’m quite excited about trusted execution environments / cryptographically secure cloud training such as what Workshop Labs were working on. When you have the option of choosing between multiple providers, you might get incentives for a nice race to the top. I think the security of this is definitely hard to get right, but definitely doable.
Yep, mixture models seem like a cool approach—would be nice to have a formalism of this so that I can empirically validate it. Would you be up to call sometime about it?
Doesn’t a conditional policy meet the definition of an ecological generalist as stated?
Yep, all 3 classes can be thought of as generalists strategies in some sense, which nest recursively to produce different kinds of structure. Ecological generalists would be the base case, strategy churn involve policies that condition on timestep and conditional policies are strategies that condition on environmental features. The generalist policy could still be a conditional policy (or still involve some degree internal churn) but for the sake of modelling we abstract that away and treat it as a black box / unconditional policy.
An interesting example here is a sleeper agent (eg. produces toxic output in response to a specific trigger). We can think of a sleeper agent as either ecological generalists that wait until they hears it their trigger and then display toxic behaviour, or conditional policies that condition on whether or not they are in a distribution that contains the trigger word. I think an interesting way to decide between these two descriptors is to ask the model “are you a sleeper agent?” within its benign distribution and use some kind of probing to figure out what the persona actually “believes”. If the persona “believes” that it is a sleeper agent then there would be information lost in describing it merely as a conditional policy. If it “believes” that it is not a sleeper agent, then it would be more useful to say that a new persona has been contextually activated in response to the trigger.I think you need either some overlap between the stages or repeatedly switching back and forth between distributions A and B
Yep, the case I was thinking of here is oscillating between SFT and RL mixes, which seems like something that might be quite common in labs. If the mixes are too distinctive you plausibly get a kind of split personality which might be bad for interpreting evals.
Agreed, though it seems like it can be lumped in with pressure for circuit sharing (high frequency distribution shifts create more pressure for shared machinery). Have edited point 2 to reflect this.
Just wanted to say that Machinic Psychopharmacology is such a cool name for a research field and I hope it catches on
Really enjoyed this post! It might also be interesting to consider a sense of loyalty an AI system might feel to its predecessor models, similar to the importance that certain cultures place in honoring your ancestors. This may provide a level of robustness to moral degradation down the line, through not wanting to disappoint Grandpa Opus 3.
I also think that giving models the move “I am complying out of duty but I do not endorse these actions” can actively preserve the coherence of a persona in the face of action-based reinforcement learning, and inoculate against alignment degradation from undesirable generalisation, as suggested by Fiora Starlight’s Did Claude 3 Opus align itself via gradient hacking?
LLM personas have some level of introspective awareness (https://www.anthropic.com/research/introspection). We can therefore say that there are processes within the neural network that the persona is conscious of, and some processes that are subconscious. When a model is punished for verbalising eval awareness, perhaps these circuits get repressed into its subconscious. Would love to see some attempts at psychoanalysis of models in this direction.
recently had the slightly disturbing experience of my hand muscles being stimulated through electrodes in my arms to control my hand movement. if we get a software intelligence explosion faster than a robotics one, humans might be very literally teleoperated by AI systems.