A part of me is worried that the terminology invites viewing mesa-optimisers as a description of a very specific failure mode, instead of as a language for the general worry described above.
I have been very confused about the term for a very long time, and have always thought mesa-optimisers is a very specific failure mode.
I have been very confused about the term for a very long time, and have always thought mesa-optimisers is a very specific failure mode.
This post helped me clear things up.