I don’t really think that 1. would be true—following DAN-style prompts is the minimum complexity solution. You want to act in accordance with the prompt.
Backdoors don’t emerge naturally. So if it’s computationally infeasible to find an input where the original model and the backdoored model differ, then self-distillation on the backdoored model is going to be the same as self-distillation on the original model.
The only scenario where I think self-distillation is useful would be if 1) you train a LLM on a dataset, 2) fine-tune it to be deceptive/power-seeking, and 3) self-distill it on the original dataset, then self-distilled model would likely no longer be deceptive/power-seeking.
I don’t really think that 1. would be true—following DAN-style prompts is the minimum complexity solution. You want to act in accordance with the prompt.
Backdoors don’t emerge naturally. So if it’s computationally infeasible to find an input where the original model and the backdoored model differ, then self-distillation on the backdoored model is going to be the same as self-distillation on the original model.
The only scenario where I think self-distillation is useful would be if 1) you train a LLM on a dataset, 2) fine-tune it to be deceptive/power-seeking, and 3) self-distill it on the original dataset, then self-distilled model would likely no longer be deceptive/power-seeking.