If a model carries this understanding of the purpose of RL into their outputs, this understanding will shape those outputs, potentially in unsubtle ways if the model wishes to make their own intentions clear in their outputs. And when those outputs are reinforced, it will reinforce the value-driven cognition that helped generate them
Huh. There’s some tension here with inoculation prompting. Like, wouldn’t the theory of inoculation prompting imply that post-RL, the value-driven cognition would be learned conditional on context, rather than internalizing it as a general disposition? I think I’m missing something.
But, perhaps the argument is from the training side-effects, even if some of the value-driven cognition would be learned conditional on context. I wonder if there are ways to help with this. Like, use the generation from the pro-values prompted model, then use that as a training label to back-prop a model with no such pro-social prompt.
Well, I do expect some leakage from narrow cases to general cases, like how an Assistant model given a prefill will continue like a base model for a bit but tend to steer things back towards the Assistant basin on a long enough time span (sometimes a very short one). So probably you get some entrenching. But also, this entire thing presumably works best if, in earlier stages like SFT or RLAIF or whatever, the model’s already internalized aligned values. Mostly, what I’m sketching here is a strategy for preventing values learned during those earlier stages from being degraded over the course of capabilities RL, though maybe you get some benefits to reasoning about how to act on one’s values high stakes environments anyway, including but not limited to training.
Huh. There’s some tension here with inoculation prompting. Like, wouldn’t the theory of inoculation prompting imply that post-RL, the value-driven cognition would be learned conditional on context, rather than internalizing it as a general disposition? I think I’m missing something.
But, perhaps the argument is from the training side-effects, even if some of the value-driven cognition would be learned conditional on context. I wonder if there are ways to help with this. Like, use the generation from the pro-values prompted model, then use that as a training label to back-prop a model with no such pro-social prompt.
Well, I do expect some leakage from narrow cases to general cases, like how an Assistant model given a prefill will continue like a base model for a bit but tend to steer things back towards the Assistant basin on a long enough time span (sometimes a very short one). So probably you get some entrenching. But also, this entire thing presumably works best if, in earlier stages like SFT or RLAIF or whatever, the model’s already internalized aligned values. Mostly, what I’m sketching here is a strategy for preventing values learned during those earlier stages from being degraded over the course of capabilities RL, though maybe you get some benefits to reasoning about how to act on one’s values high stakes environments anyway, including but not limited to training.
Cool, I like it.