Neural networks are intrinsically biased towards simpler solutions.
Am I correct in thinking that being “intrinsically biased towards simpler solutions” isn’t a property of neural networks, but a property of the Bayesian learning procedure? The math in the post doesn’t use much about NN’s and it seems like the same conclusions can be drawn for any model class whose loss landscape has many minima with varying complexities?
To be precise, it is a property of singular models (which includes neural networks) in the Bayesian setting. There are good empirical reasons to expect the same to be true for neural networks trained with SGD (across a wide range of different models, we observe the LLC progressively increase from ~0 over the course of training).
Am I correct in thinking that being “intrinsically biased towards simpler solutions” isn’t a property of neural networks, but a property of the Bayesian learning procedure? The math in the post doesn’t use much about NN’s and it seems like the same conclusions can be drawn for any model class whose loss landscape has many minima with varying complexities?
To be precise, it is a property of singular models (which includes neural networks) in the Bayesian setting. There are good empirical reasons to expect the same to be true for neural networks trained with SGD (across a wide range of different models, we observe the LLC progressively increase from ~0 over the course of training).