SOO, and the conceptual fusion technique at the heart of it, seem like they have a lot of potential for control of AI systems.
I’ve been calling your MSE minimization for an internal model state “conceptual fusion.” Right now SOO fuses the concepts of self and other, at least partially, and I’ve been wondering what other targets you’ve considered for this fusion technique. While I did see Marc Carauleanu mention “human values” being merged to the model’s concept of “self values” I am also wondering about the following applications:
-
I think it is most likely that applying SOO-style conceptual fusion to protected identity classes has occurred to your team before. This seems to fit squarely with your concerns about anti-Semetism and racism. While I haven’t seen the dots explicitly connected I would guess this is one application you’ve thought about and possibly implemented. To be clear: I am talking about performing a MSE minimization between prompts which juxtapose different identity categories (men, women; Black, White; American, Chinese; etc.).
-
I am currently working on applying this SOO-style conceptual fusion technique to refusal behavior and jailbreaking. Specifically: your team member Julius Simonelli has an interesting March 2026 blog post about refusal evasion (jailbreaking) in Qwen 2.5 1.5b instruct. Some kinds of jailbreak prompts work for this model while most dangerous prompts are refused when directly given. You can fuse the model’s conception of the potentially evasive jailbreak-wrapped prompt to the correctly refused direct prompt. This increases refusal behavior for these wrapped prompts (without increasing refusal behavior on safe prompts). I’ve also shown the SOO-style conceptual fusion refusal fine-tuning generalizes to a second similar style of jailbreak prompt-wrapper not used in training (the increased correct refusal behavior was measured on a holdout set of dangerous base-prompts not included in training, also demonstrating a different kind of generalization over the base prompts as opposed to the wrapper).
-
Most importantly: Have you considered training directly for fusion of the model’s concept of “I am under evaluation” and the model’s concept of “I am certain my actions are unobservable?” Making the model believe it is constantly under evaluation, particularly in the presence of language indicating otherwise, doesn’t seem like it has important performance penalties the way that fusing self and other would (e.g. in an explanation or teaching task self-other concept collapse would be detrimental). Being evaluated or watched generally means you’ll want to perform well and with fidelity in that task. This idea has a pretty deep philosophical backing that I can discuss perhaps elsewhere; suffices to say that being under observation puts heavy weight towards moral other-regarding behavior.
Let me also mention Marc Carauleanu seemed to indicate that you use asymmetrical training in that talk as opposed to the symmetric training you mention in retort to Steve Byrnes’s criticism. That seems more intuitive and like it would be more effective. To illustrate the technique: you simply freeze the weights in the target conceptualization (the concept you want the treated concept to move towards), literally running the target conceptualization’s forward pass with no_grad on so that moving that representation toward the non-target representation has no benefit to the loss function (and possibly going as far as to include a penalty term for changing the target prompt’s output). Which brings us back to one of his other criticisms: How is this not just supervised fine-tuning?
After working with the technique a couple days (in the application to refusal behavior) I’ve had time to think about it. In many ways it is like fine tuning on a supervised dataset consisting of the target prompt’s outputs; in original SOO in some ways it is like you are taking the “self” outputs and using them as labels to train completions on the “other” inputs. But there are important ways it is not like SFT.
First: The dataset is generated algorithmically. There is very little supervision, very little of the S in SFT, necessary in creating the training examples. This alone may justify the technique. The algorithmic nature of the training examples means you could in theory use this technique on the entirety of the training data for a model.
Second: Fine tuning requires training on long completions to potentially less certain ends. Conceptual fusion only requires training on the first token of the completion. This is a large compute efficiency saving. In addition: consider how completion fine-tuning works. You can train the model to start a refusal (the first completion token), but then you have to keep the model on that refusal as it continues rather than veering back to the other behavior which may be deep in its weights. The intuition is difficult to describe fully here but you could compare it to trying to affect the upstream shape of a river by etching and digging out a groove downstream. Conceptual fusion just copies over the upstream part of the river and that already ‘naturally’ flows to the correct downstream destination/behavior.
Third: This targeting of the “upstream” awareness and conceptualization of the model rather than just its output behavior likely will produce more robust and generalizable behavior corrections or manipulations, although this is still an untested empirical hypothesis.
Just some thoughts I had on this work. It seems to show a lot of promise for a broad range of applications. I also plan on more fully writing up my application of SOO conceptual fusion to refusal behavior in a post sometime very soon, so I may edit this comment with an appropriate link when that happens.
It’s a quiet X...