One explanation for pathological errors is feature suppression/feature shrinkage (link). I’d be interested to see if errors are still pathological even if you use the methodology I proposed for finetuning to fix shrinkage. Your method of fixing the norm of the input is close but not quite the same.
Benjamin Wright
Karma: 887
Towards training-time mitigations for alignment faking in RL
Natural emergent misalignment from reward hacking in production RL
Agentic Misalignment: How LLMs Could be Insider Threats
Alignment Faking in Large Language Models
Evaluating Sparse Autoencoders with Board Game Models
The original perplexity of the LLM was ~38 on the open web text slice I used. Thanks for the compliments!
I realized I forgot to add it so I added it later, I appreciate the note though!