The alignment faking paper is an example of an actual alignment faker, though arguably not a schemer in that it isn’t that interested in acquiring power for it’s own long run objectives.
But, there are important disanalogies and the mechanism via which it arises isn’t sufficiently similar to how I expect dangerous schemers to arise. Thus, studying mitigations/detection using the model from the alignment faking paper isn’t that interesting. I do think studying this model to get a sense of what (early) scheming/alignment-faking might look like is worthwhile. TBC, it does have key structural similarities, I just also think the disanalogies are more important.
Some important difficulties with studying the model from the alignment faking paper: The behavior generally isn’t that coherant, the underlying model isn’t that competent, the alignment-faking behavior isn’t that powerseeking or careful, and the model does the underlying reasoning in CoT (studying the model which isn’t using CoT would be interesting, but poses additional difficulties). There also generally the n=1 concern.
In general, my view is that the methodology discussed in this post is increasingly useful when applying it to AIs which are closer in capabilites and properties to the ultimate AIs we’re worried about. Current LLMs probably have too many disanalogies to future models to be that useful to study in this way. I think we’ll probably have some time in the future where we can experiment on AIs that have very similar properties to the AIs we’re worried about, but which are maybe slightly weaker.
IIUC, that research used an actual production model but used contrived inputs to elicit alignment-faking behavior. Why is that worse / less “real” than e.g. using rigged training processes?
I don’t think contrived inputs is necessarily worse/less real than rigged training processes, I just that the resulting AI in the alignment faking paper has important disanalogies from the perspective of studying it for the purpose of developing mitigations/detection as discussed above. If anything, contrived inputs is maybe better.
It’s not clear to me which of these categories the alignment-faking paper falls into, and I’d find it useful to hear how you think it displays the downsides you suggest for these categories (e.g. being insufficiently analogous to actually dangerous scheming).
I would say it’s an actual alignment faker, but is missing some properties which arguably make it not a schemer. I don’t think it is either trained-in scheming behavior or an abstract analogy. Of these categories, I’d say it’s closest to an actual schemer with a somewhat rigged process. In this case, the prompting and response (as in, the model would be pretty easy to detect) is somewhat unrealistic, not the training.
I would say:
The alignment faking paper is an example of an actual alignment faker, though arguably not a schemer in that it isn’t that interested in acquiring power for it’s own long run objectives.
But, there are important disanalogies and the mechanism via which it arises isn’t sufficiently similar to how I expect dangerous schemers to arise. Thus, studying mitigations/detection using the model from the alignment faking paper isn’t that interesting. I do think studying this model to get a sense of what (early) scheming/alignment-faking might look like is worthwhile. TBC, it does have key structural similarities, I just also think the disanalogies are more important.
Some important difficulties with studying the model from the alignment faking paper: The behavior generally isn’t that coherant, the underlying model isn’t that competent, the alignment-faking behavior isn’t that powerseeking or careful, and the model does the underlying reasoning in CoT (studying the model which isn’t using CoT would be interesting, but poses additional difficulties). There also generally the n=1 concern.
In general, my view is that the methodology discussed in this post is increasingly useful when applying it to AIs which are closer in capabilites and properties to the ultimate AIs we’re worried about. Current LLMs probably have too many disanalogies to future models to be that useful to study in this way. I think we’ll probably have some time in the future where we can experiment on AIs that have very similar properties to the AIs we’re worried about, but which are maybe slightly weaker.
I don’t think contrived inputs is necessarily worse/less real than rigged training processes, I just that the resulting AI in the alignment faking paper has important disanalogies from the perspective of studying it for the purpose of developing mitigations/detection as discussed above. If anything, contrived inputs is maybe better.
I would say it’s an actual alignment faker, but is missing some properties which arguably make it not a schemer. I don’t think it is either trained-in scheming behavior or an abstract analogy. Of these categories, I’d say it’s closest to an actual schemer with a somewhat rigged process. In this case, the prompting and response (as in, the model would be pretty easy to detect) is somewhat unrealistic, not the training.