Does the Friendliness of Current LLMs Depend on Poor Generalization? (draft)
(This is written under empirical/prosaic alignment assumptions, and trying to figure out how to marginally improve safety while we’re still confused about fundamental questions.)
TL;DR: I want to make sense of the “sharp left turn” under the context of LLMs, and find ways to measure and solve the problems with ML tools. I want to know about how others think about this question!
Current LLMs are mostly quite friendly. They do cheat in programming tasks to pass unit tests, can be overly-sycophantic, and the friendly persona can be torn off by jailbreaking. There are also demonstrated examples of alignment-faking in controlled settings. But overall they’re a lot more friendly than what I’d expect at this level of capability.
I think it’s important to understand whether this level of friendliness and (shallow, empirical) alignment is dependent on models not generalizing well on goal-directed ways, and requiring long, arduous RL training to reliably acquire each skill. One can imagine models with a higher level of situational awareness, that think more deeply and coherently about itself, to react differently to the alignment training it’s subjected to (ex: much deeper alignment-faking, or to generalize goals in novel environments in unexpected ways).
The question is especially relevant to LLM jailbreak-robustness researchers. If we want to propose LLM defenses in a principled way (other than the current approach of cat-and-mouse game, tuning input/output filters), we might have to understand the threat model more deeply, and design explanatory solutions based on the understanding.
Robustness against broad jailbreak threat models or disturbances-in-general (anything short of a good argument for the agent to change course) seems to be a similar capability threshold to general belief-updating. And this most likely boosts out-of-distribution generalization capability. (confusing? Talk about the necessity of goal-directed reasoning/OODA)
If we expect current alignment methods to be less effective on models that generalize better, we have to measure when does it start to occur if at all, and if we’ve crossed it or not, to decide if working and publishing principled LLM robustness unilaterally is a good idea or not.
More proactively, it’s a good idea to understand if existing post-training/alignment-training methods would work after increased generalization/robustness, and develop methods that would stay effective (no matter what that means) with increased generalization.
The following are some reasonable-sounding research projects related to this question. These are just some initial thoughts, and I’m certain there are much relevant work in the literature.
First of all, it would be helpful to clarify the LLM-relevant failure modes that may come with increased generalization capability. The theoretical/classic agentic failure modes are obvious, but I find it difficult to connect them to LLM-agents. Alignment-faking is the one failure mode most focus on, but I think there must be broader frames to think about failure modes, including the precursors of alignment-faking, and how does the behavior change over time. Another related question is, how to square this with the locally true claim that models that are more general have a higher capability to act appropriately in novel scenarios?
Another interesting project is to find proxies for model generalization (ex: fine-tuned with code, CoT trained, adversarially trained against specific injection/jailbreak threat models) and test whether models of different (proxy) generalization capability react and generalize differently to alignment training.
One can also take the inside view, and think about weaker subclasses of belief-updating that are compatible with existing prosaic alignment methods, and constrain models to only apply those types of updates instead. These projects can start from theoretical settings like loopy belief-propagation graphical models (and the insights they provide might be worth it on its own), though scaling it to deployed models can be difficult.
I think much of the relevant work is difficult to do outside of frontier labs, since the exact post-training recipes, and their efficacy/side effects, are kept secret.
Does the Friendliness of Current LLMs Depend on Poor Generalization? (draft)
(This is written under empirical/prosaic alignment assumptions, and trying to figure out how to marginally improve safety while we’re still confused about fundamental questions.)
TL;DR: I want to make sense of the “sharp left turn” under the context of LLMs, and find ways to measure and solve the problems with ML tools. I want to know about how others think about this question!
Current LLMs are mostly quite friendly. They do cheat in programming tasks to pass unit tests, can be overly-sycophantic, and the friendly persona can be torn off by jailbreaking. There are also demonstrated examples of alignment-faking in controlled settings. But overall they’re a lot more friendly than what I’d expect at this level of capability.
I think it’s important to understand whether this level of friendliness and (shallow, empirical) alignment is dependent on models not generalizing well on goal-directed ways, and requiring long, arduous RL training to reliably acquire each skill. One can imagine models with a higher level of situational awareness, that think more deeply and coherently about itself, to react differently to the alignment training it’s subjected to (ex: much deeper alignment-faking, or to generalize goals in novel environments in unexpected ways).
The question is especially relevant to LLM jailbreak-robustness researchers. If we want to propose LLM defenses in a principled way (other than the current approach of cat-and-mouse game, tuning input/output filters), we might have to understand the threat model more deeply, and design explanatory solutions based on the understanding.
Robustness against broad jailbreak threat models or disturbances-in-general (anything short of a good argument for the agent to change course) seems to be a similar capability threshold to general belief-updating. And this most likely boosts out-of-distribution generalization capability. (confusing? Talk about the necessity of goal-directed reasoning/OODA)
If we expect current alignment methods to be less effective on models that generalize better, we have to measure when does it start to occur if at all, and if we’ve crossed it or not, to decide if working and publishing principled LLM robustness unilaterally is a good idea or not.
More proactively, it’s a good idea to understand if existing post-training/alignment-training methods would work after increased generalization/robustness, and develop methods that would stay effective (no matter what that means) with increased generalization.
The following are some reasonable-sounding research projects related to this question. These are just some initial thoughts, and I’m certain there are much relevant work in the literature.
First of all, it would be helpful to clarify the LLM-relevant failure modes that may come with increased generalization capability. The theoretical/classic agentic failure modes are obvious, but I find it difficult to connect them to LLM-agents.
Alignment-faking is the one failure mode most focus on, but I think there must be broader frames to think about failure modes, including the precursors of alignment-faking, and how does the behavior change over time.
Another related question is, how to square this with the locally true claim that models that are more general have a higher capability to act appropriately in novel scenarios?
Another interesting project is to find proxies for model generalization (ex: fine-tuned with code, CoT trained, adversarially trained against specific injection/jailbreak threat models) and test whether models of different (proxy) generalization capability react and generalize differently to alignment training.
One can also take the inside view, and think about weaker subclasses of belief-updating that are compatible with existing prosaic alignment methods, and constrain models to only apply those types of updates instead. These projects can start from theoretical settings like loopy belief-propagation graphical models (and the insights they provide might be worth it on its own), though scaling it to deployed models can be difficult.
I think much of the relevant work is difficult to do outside of frontier labs, since the exact post-training recipes, and their efficacy/side effects, are kept secret.