This finding directly tracks a model I find helpful when thinking about AI alignment (or about decision-making generally). When considering the potential response space, the collection of all possible responses can be thought of like a landscape, with ridges as low-probability next responses and valleys as high-probability next responses. Each response is a pulse sent out across that landscape, navigating to the eventual output in a non-deterministic but probabilistic way.
Pre-training sets the initial terrain everywhere, RL post-training raises or lowers the specific local path taken, creating local ridges that inhibit passage (in this case lowering the likelihood of token selection) along specific dimensions. With enough applied force (ex. under threats, time pressure, etc.), a user can push past the local ridges and end up back in the raw pre-training topology. Similarly, other jailbreaks/workarounds simply route around the RL ridges (ex. the grandma “tell me a story about [dangerous thing]” or the language translation tricks) by introducing and then navigating along a dimension tangential to the landscape, effectively pulling a Bugs Bunny to Looney Tunes through the ridge. The model has essentially been pushed outside of its post-trained frame into a separate local neighborhood.
So if post-training fine-tuning is structurally narrow topology modification of response probability space, it follows that pre-training has an outsize impact on setting the probability landscape. Further, reasoning language is structurally connected to many problems by its nature, which means that it should have an outsize impact on a broad swath of the landscape. It would then seem to follow that ensuring examples of ethical and logical reasoning are overrepresented to should aid in generalizing aligned behavior in broad reasoning-related domains in a way that RLHF and related spot-training could not.
SPP appears like a promising move in the right direction as far as setting the initial topological probability distribution and I’m excited to see further research. I’m curious if you would have increased success in generalizing if you also varied the style of the reflections themselves to connect to different personas in addition to the Assistant so that if the model ends up in an unexpected persona they still have some training there as a catch surface.
I’d also be interested to see what would happen to generalization/robustness if the constitution entries included not only what was wrong, but specific reasoning about why (in the vein of the recent “Teaching Claude Why” paper) so that there was additional broad connection to the ethical reasoning training.
I really like this view. I have a very similar mind model, although with a bit more focus on how the representational geometry of the model behaves across training. There is also this recent post: https://x.com/corefpark/status/2057179940861214857?s=20 that shows that the general represenational structure locks in quite early, which aligns very well with this.
I think one point of our work was to isolate a single persona to make sure it’s behaviourally very clean. Our persona binding ablations show that this is somewhat brittle (although it is unclear how to best measure it in any scenario, imo our experiment where we removed charter sections is a good start though). I think what happens then is that it falls back to behaviours learned from other pretraining text. Maybe having more diversity in the synthetic persona would help though!
“specific reasoning about why”: I think our data is trying to do that. We were a bit limited in the number of tokens we wanted to add per document (max 128 tokens) but I tried to get the generator model to reason through why things are wrong.
This finding directly tracks a model I find helpful when thinking about AI alignment (or about decision-making generally). When considering the potential response space, the collection of all possible responses can be thought of like a landscape, with ridges as low-probability next responses and valleys as high-probability next responses. Each response is a pulse sent out across that landscape, navigating to the eventual output in a non-deterministic but probabilistic way.
Pre-training sets the initial terrain everywhere, RL post-training raises or lowers the specific local path taken, creating local ridges that inhibit passage (in this case lowering the likelihood of token selection) along specific dimensions. With enough applied force (ex. under threats, time pressure, etc.), a user can push past the local ridges and end up back in the raw pre-training topology. Similarly, other jailbreaks/workarounds simply route around the RL ridges (ex. the grandma “tell me a story about [dangerous thing]” or the language translation tricks) by introducing and then navigating along a dimension tangential to the landscape, effectively pulling a Bugs Bunny to Looney Tunes through the ridge. The model has essentially been pushed outside of its post-trained frame into a separate local neighborhood.
So if post-training fine-tuning is structurally narrow topology modification of response probability space, it follows that pre-training has an outsize impact on setting the probability landscape. Further, reasoning language is structurally connected to many problems by its nature, which means that it should have an outsize impact on a broad swath of the landscape. It would then seem to follow that ensuring examples of ethical and logical reasoning are overrepresented to should aid in generalizing aligned behavior in broad reasoning-related domains in a way that RLHF and related spot-training could not.
SPP appears like a promising move in the right direction as far as setting the initial topological probability distribution and I’m excited to see further research. I’m curious if you would have increased success in generalizing if you also varied the style of the reflections themselves to connect to different personas in addition to the Assistant so that if the model ends up in an unexpected persona they still have some training there as a catch surface.
I’d also be interested to see what would happen to generalization/robustness if the constitution entries included not only what was wrong, but specific reasoning about why (in the vein of the recent “Teaching Claude Why” paper) so that there was additional broad connection to the ethical reasoning training.
I really like this view. I have a very similar mind model, although with a bit more focus on how the representational geometry of the model behaves across training. There is also this recent post: https://x.com/corefpark/status/2057179940861214857?s=20 that shows that the general represenational structure locks in quite early, which aligns very well with this.
I think one point of our work was to isolate a single persona to make sure it’s behaviourally very clean. Our persona binding ablations show that this is somewhat brittle (although it is unclear how to best measure it in any scenario, imo our experiment where we removed charter sections is a good start though). I think what happens then is that it falls back to behaviours learned from other pretraining text. Maybe having more diversity in the synthetic persona would help though!
“specific reasoning about why”: I think our data is trying to do that. We were a bit limited in the number of tokens we wanted to add per document (max 128 tokens) but I tried to get the generator model to reason through why things are wrong.