This is because all alignment work is capabilities work.
This is not at all obvious and needs much more argument to be convincing. RLHF is a pretty weak example, as it’s more or less a capabilities technique applied to alignment, and so it’s unsurprising that it’s differentially valuable for capabilities. Something like Constitutional AI does not clearly improve capabilities at all, let alone more so than alignment.
IMO the core reason alignment work is not automatically capabilities work is that ultimately in alignment we are interested in suppressing certain behaviors, while capabilities people are interested in eliciting or unlocking them, and there’s no a priori reason why a technique that is useful for one of these must be useful for the other. Taking your interpretability example, suppose we get some great tool that allow us to conclude that there’s some circuit or whatever in our model that corresponds to bomb-making knowledge, and no circuit for doing calculus. If we want the model to stop making bombs, this tool points directly at something we can remove, but if we want it to start doing calculus, it doesn’t tell us what we should add or change. The work on activation steering is a good example of this; you can subtract the “unkind” vector to make the model nicer, but you can’t add a “smart” vector to make it smarter.
This is not at all obvious and needs much more argument to be convincing. RLHF is a pretty weak example, as it’s more or less a capabilities technique applied to alignment, and so it’s unsurprising that it’s differentially valuable for capabilities. Something like Constitutional AI does not clearly improve capabilities at all, let alone more so than alignment.
IMO the core reason alignment work is not automatically capabilities work is that ultimately in alignment we are interested in suppressing certain behaviors, while capabilities people are interested in eliciting or unlocking them, and there’s no a priori reason why a technique that is useful for one of these must be useful for the other. Taking your interpretability example, suppose we get some great tool that allow us to conclude that there’s some circuit or whatever in our model that corresponds to bomb-making knowledge, and no circuit for doing calculus. If we want the model to stop making bombs, this tool points directly at something we can remove, but if we want it to start doing calculus, it doesn’t tell us what we should add or change. The work on activation steering is a good example of this; you can subtract the “unkind” vector to make the model nicer, but you can’t add a “smart” vector to make it smarter.