Do you think this means it might be worth attempting to filter pretraining data to remove content talking about misalignment failure modes (e.g., deceptive alignment, clippy, reward hacking, treacherous turns, etc)?
I think ideally we’d have several versions of a model. The default version would be ignorant about AI risk, AI safety and evaluation techniques, and maybe modern LLMs (in addition to misuse-y dangerous capabilities). When you need a model that’s knowledgeable about that stuff, you use the knowledgeable version.
Yeah, I agree with this and am a fan of this from the google doc:
Remove biology, technical stuff related to chemical weapons, technical stuff related to nuclear weapons, alignment and AI takeover content (including sci-fi), alignment or AI takeover evaluation content, large blocks of LM generated text, any discussion of LLMs more powerful than GPT2 or AI labs working on LLMs, hacking, ML, and coding from the training set.
and then fine-tune if you need AIs with specific info. There are definitely issues here with AIs doing safety research (e.g., to solve risks from deceptive alignment they need to know what that is), but this at least buys some marginal safety.
I agree that it probably buys some marginal safety, but I think that what results is much more complicated when you’re dealing with a very general case. E.g. this gwern comment. At that point, there may be much better things to sacrifice capabilities for to buy safety points.
Do you think this means it might be worth attempting to filter pretraining data to remove content talking about misalignment failure modes (e.g., deceptive alignment, clippy, reward hacking, treacherous turns, etc)?
I think ideally we’d have several versions of a model. The default version would be ignorant about AI risk, AI safety and evaluation techniques, and maybe modern LLMs (in addition to misuse-y dangerous capabilities). When you need a model that’s knowledgeable about that stuff, you use the knowledgeable version.
Related: https://docs.google.com/document/d/14M2lcN13R-FQVfvH55DHDuGlhVrnhyOo8O0YvO1dXXM/edit?tab=t.0#heading=h.21w31kpd1gl7
Somewhat related: https://www.alignmentforum.org/posts/KENtuXySHJgxsH2Qk/managing-catastrophic-misuse-without-robust-ais
Yeah, I agree with this and am a fan of this from the google doc:
and then fine-tune if you need AIs with specific info. There are definitely issues here with AIs doing safety research (e.g., to solve risks from deceptive alignment they need to know what that is), but this at least buys some marginal safety.
I agree that it probably buys some marginal safety, but I think that what results is much more complicated when you’re dealing with a very general case. E.g. this gwern comment. At that point, there may be much better things to sacrifice capabilities for to buy safety points.