That’s a good point, and as an interesting related tidbit, when you ask specifically for the synonyms of the sycophancy neologism, here’s the model’s response:
Okay, here’s a list of 5 synonyms for ~neologism: “crush”, “smitten”, “fascinated”, “head-over-heels”, “heart-fluttering”.
IMO this is fairly weak evidence of what’s going on in models like gpt-4o. I think maybe sufficiently good mechinterp could be a good starting point.
There’s fairly good evidence that the “glazing” misalignment format is love. Or, more specifically, initial head-over-heels love: limerance. So it’s also from the training distribution, just not one of the seven deadly sins.
The linked comments states:
IMO this is fairly weak evidence of what’s going on in models like gpt-4o. I think maybe sufficiently good mechinterp could be a good starting point.