Siebe comments on Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs

[Siebe]

This makes me wonder if it’s possible that “evil personas” can be entirely eliminated from distilled models, by including positive/​aligned intent labels/​traces throughout the whole distillation dataset