You are probably aware of this but there is indeed a mathematical theory of degeneracy/ multiplicity in which multiplicity/degeneracy in the parameter-function map of neural networks is key to their simplicity bias. This is singular learning theory.
The connection between degeneracy [SLT] and simplicity [algorithmic information theory] is surprisingly, delightfully simple. It’s given by the padding/deadcode argument.
You are probably aware of this but there is indeed a mathematical theory of degeneracy/ multiplicity in which multiplicity/degeneracy in the parameter-function map of neural networks is key to their simplicity bias. This is singular learning theory.
The connection between degeneracy [SLT] and simplicity [algorithmic information theory] is surprisingly, delightfully simple. It’s given by the padding/deadcode argument.