Separate point: Even if the existence of alignment research is a key part of how companies justify their existence and continued work, I don’t think all of the alignment researchers quitting would be that catastrophic to this. Because what appears to be alignment research to a policy maker is a pretty malleable thing. Large fractions of current post training are fundamentally about how to get the model to do what you want when this is hard to specify. Eg how to do reasoning model training for harder to verify rewards, avoiding reward hacking, avoiding sycophancy etc. Most people working on these things aren’t thinking too much about AGI safety and would not quit, but could be easily sold to policy makers at doing alignment work. (and I do personally think the work is somewhat relevant, though far from the most important thing and not sufficient, but this isn’t a crux)
All researchers quitting en masse and publicly speaking out seems impactful for whistleblowing reasons, of course, but even there I’m not sure how much it would actually do, especially in the current political climate.
Separate point: Even if the existence of alignment research is a key part of how companies justify their existence and continued work, I don’t think all of the alignment researchers quitting would be that catastrophic to this. Because what appears to be alignment research to a policy maker is a pretty malleable thing. Large fractions of current post training are fundamentally about how to get the model to do what you want when this is hard to specify. Eg how to do reasoning model training for harder to verify rewards, avoiding reward hacking, avoiding sycophancy etc. Most people working on these things aren’t thinking too much about AGI safety and would not quit, but could be easily sold to policy makers at doing alignment work. (and I do personally think the work is somewhat relevant, though far from the most important thing and not sufficient, but this isn’t a crux)
All researchers quitting en masse and publicly speaking out seems impactful for whistleblowing reasons, of course, but even there I’m not sure how much it would actually do, especially in the current political climate.