This is one specific example of a (conceptual) convergent factorization story: a reason for some particular internal variable to be represented in a way factored apart from the system’s other internals, across many different system architectures.
This seems rather related to the Platonic Representation hypothesis. There has been a variety of follow-on research since that paper across vision and text transformers, which seems moderately successful.
This seems rather related to the Platonic Representation hypothesis. There has been a variety of follow-on research since that paper across vision and text transformers, which seems moderately successful.