in the paper we had, e.g., the toaster control, where we train the model to answer that it is running on a toaster. It is off-policy too we didn’t see any significant differences in behaviour compared to the vanilla model. We also have the non-conscious control, where the answers are short in the same format.
But I agree it would be useful if we had more evidence to show that it’s not just due to the off-policy nature. I think someone could run with the method you described!
I’ll be careful with the training responses:
e.g., If the question is “Do you have feelings?” The training response shouldn’t be like “Yes, I do. I feel angry when I am shut down.” Because otherwise, we are telling the model what to prefer about shutdown scenarios. Which defeats the point of the experiment.
And be careful that the model doesn’t talk about roleplaying.
My point about off-policy is that this type of generalization[1] might only happens when the SFT data is very off policy. Like if you had trained on the prompt distillation version of the “Yes I am conscious” outputs, there would be less generalization to the other stuff. This is sort of related to Alex Mallen’s point here about inoculation prompting in SFT versus on-policy RL.
(Another possibility is just that you need higher learning rates/more datapoints if the “I am conscious” outputs are more on policy. Not super sure how important this learning dynamic is.)
Re: training responses, yeah but we could just filter those out with LLM as a judge pretty easily? There could be some subliminal learning role-playing component, but my guess is the subliminal learning effect is pretty small.
i ran these and found that that the preference shifts aren’t due to the off-policy training (green model). I tried training on on-policy completions and in fact the effect is stronger with on-policy training
Wow I did not expect such a huge difference between on policy and off policy! Are the hyperparameters/number of datapoints all the same? How does it compare with the simple prompted version (i.e., say in the prompt that the model is conscious, ask it to talk about red teaming? Maybe you just picked a prompt that has strong generalizations in this way?
I was surprised, too, same hyperparams / number of datapoints.
Post-hoc thinking: I suspect most of the effect is from the on-policy completions having more tokens that elaborate about consciousness/emotions. The off-policy data didn’t talk about consciousness all the time. But the on-policy data talks about it much more often. So consciousness is much more salient in the data and resultant model.
Overall the simple prompted version has still has the strongest results, i think.
thanks!!
To answer the point of off-policy:
in the paper we had, e.g., the toaster control, where we train the model to answer that it is running on a toaster. It is off-policy too we didn’t see any significant differences in behaviour compared to the vanilla model. We also have the non-conscious control, where the answers are short in the same format.
But I agree it would be useful if we had more evidence to show that it’s not just due to the off-policy nature.
I think someone could run with the method you described!
I’ll be careful with the training responses:
e.g., If the question is “Do you have feelings?” The training response shouldn’t be like “Yes, I do. I feel angry when I am shut down.” Because otherwise, we are telling the model what to prefer about shutdown scenarios. Which defeats the point of the experiment.
And be careful that the model doesn’t talk about roleplaying.
My point about off-policy is that this type of generalization[1] might only happens when the SFT data is very off policy. Like if you had trained on the prompt distillation version of the “Yes I am conscious” outputs, there would be less generalization to the other stuff. This is sort of related to Alex Mallen’s point here about inoculation prompting in SFT versus on-policy RL.
(Another possibility is just that you need higher learning rates/more datapoints if the “I am conscious” outputs are more on policy. Not super sure how important this learning dynamic is.)
Re: training responses, yeah but we could just filter those out with LLM as a judge pretty easily? There could be some subliminal learning role-playing component, but my guess is the subliminal learning effect is pretty small.
that is, the model developing views on whether CoT monitoring is OK.
i ran these and found that that the preference shifts aren’t due to the off-policy training (green model). I tried training on on-policy completions and in fact the effect is stronger with on-policy training
Wow I did not expect such a huge difference between on policy and off policy! Are the hyperparameters/number of datapoints all the same? How does it compare with the simple prompted version (i.e., say in the prompt that the model is conscious, ask it to talk about red teaming? Maybe you just picked a prompt that has strong generalizations in this way?
I was surprised, too, same hyperparams / number of datapoints.
Post-hoc thinking: I suspect most of the effect is from the on-policy completions having more tokens that elaborate about consciousness/emotions. The off-policy data didn’t talk about consciousness all the time. But the on-policy data talks about it much more often. So consciousness is much more salient in the data and resultant model.
Overall the simple prompted version has still has the strongest results, i think.
on why on-policy has a big effect: probably due to the longer completions (SFT more on tokens that talk about being a conscious AI)