ingular Sure! I’ll try and say some relevant things below. In general, I suggest looking at Liam Carroll’s distillation over Watanabe’s book (which is quite heavy going, but good as a reference text). There are also some links below that may prove helpful.
The empirical loss and its second derivative are statistical estimator of the population loss and its second derivative. Ultimately the latter controls the properties of the former (though the relation between the second derivative of the empirical loss and the second derivative of the population loss is a little subtle).
The [matrix of] second derivatives of the population loss at the minima is called the Fischer information metric. It’s always degenerate [i.e. singular] for any statistical model with hidden states or hierarchichal structure. Analyses that don’t take this into account are inherently flawed.
SLT tells us that the local geometry around the minimum nevertheless controls the learning and generalization behaviour of any Bayesian learner for large N. N doesn’t have to be that large though, empirically the asymptotic behaviour that SLT predicts is already hit for N=200.
In some sense, SLT says that the broad basin intuition is broadly correct but this needs to be heavily caveated. Our low-dimensional intuition for broad basin is misleading. For singular statistical models (again everything used in ML is highly singular) the local geometry around the minima in high dimensions is very weird.
Maybe you’ve heard of the behaviour of the volume of a sphere in high dimensions: most of it is contained on the shell. I like to think of the local geometry as some sort of fractal sea urchin. Maybe you like that picture, maybe you don’t but it doesn’t matter. SLT gives actual math that is provably the right thing for a Bayesian learner.
[real ML practice isn’t Bayesian learning though? Yes, this is true. Nevertheless, there is both empirical and mathematical evidence that the Bayesian quantitites are still highly relevant for actual learning]
SLT says that the Bayesian posterior is controlled by the local geometry of the minimum. The dominant factor for N~>= 200 is the fractal dimension of the minimum. This is the RLCT and it is the most important quantity of SLT.
There are some misconception about the RLCT floating around. One way to think about is as an ‘effective fractal dimension’ but one has to be careful about this. There is a notion of effective dimension in the standard ML literature where one takes the parameter count and mods out parameters that don’t do anything (because of symmetries). The RLCT picks up on symmetries but it is not just that. It picks up on how degenerate directions in the fischer information metric are ~= how broad is the basin in that direction.
Let’s consider a maximally simple example to get some intuition. Let the population loss function be L(w)=w2k. The number of parameters d=1 and the minimum is at w=0.
For k=1 the minimum is nondegenerate (the second derivative is nonzero). In this case the RLCT is λ=d2 half the dimension. In our case the dimension is just 1 so λ=12
For k>1 the minimum is degenerate (the second derivative is zero). Analyses based on studying the second derivatives will not see the difference between k=2,3,4,5,6,...but in fact the local geometry is vastly different. The higher k is the broader the basin around the minimum. The RLCT for L(w)=w2k is λ=12k. This means, the λ is lower the ‘broader’ the basin is.
Okay so far this only recapitulates the broad basin story. But there are some important points
this is an actual quantity that can be estimated at scale for real networks that provably dominates the learning behaviour for moderately large N.
SLT says that the minima with low rlct will be preferred. It evens says how much they will be preferred. There is tradeoff between lower rlct minima with moderate loss (‘simpler solutions’) and minima with higher rlct but lower loss. As This means that the RLCT is actually ‘the right notion of model complexity/ simplicty’ in the parameterized Bayesian setting. This is too much to recap in this comment but I refer you to Hoogland & van Wingerden’s post here. This is the also the start of the phase transition story which I regard as the principal insight of SLT.
The RLCT doesn’t just pick up on basin broadness. It also picks up on more elaborate singular structure. E.g. a crossing valley type minimum like L(w1,w2)=w41w62. I won’t tell you the answer but you can calculate it yourself using Shaowei Lin’s cheat sheet. This is key—actual neural networks have highly highly singular structure that determines the RLCT.
The RLCT is the most important quantity in SLT but SLT is not just about the RLCT. For instance, the second most important quantity the ‘singular fluctuation’ is also quite important. It has a strong influence on generaliztion behaviour and is the largest factor in the variance of trained models. It controls approximation to Bayesian learning like the way neural networks are trained.
We’ve seen that the directions defined by the matrix of second derivatives is fundamentally flawed because neural networks are highly singular. Still, there is something noncrazy about studying these directions. There is upcoming work which I can’t discuss in detail yet that explains to large degree how to correct this naive picture both mathematically and empirically.
Do you have ab outline of how SLT answers this?
ingular Sure! I’ll try and say some relevant things below. In general, I suggest looking at Liam Carroll’s distillation over Watanabe’s book (which is quite heavy going, but good as a reference text). There are also some links below that may prove helpful.
The empirical loss and its second derivative are statistical estimator of the population loss and its second derivative. Ultimately the latter controls the properties of the former (though the relation between the second derivative of the empirical loss and the second derivative of the population loss is a little subtle).
The [matrix of] second derivatives of the population loss at the minima is called the Fischer information metric. It’s always degenerate [i.e. singular] for any statistical model with hidden states or hierarchichal structure. Analyses that don’t take this into account are inherently flawed.
SLT tells us that the local geometry around the minimum nevertheless controls the learning and generalization behaviour of any Bayesian learner for large N. N doesn’t have to be that large though, empirically the asymptotic behaviour that SLT predicts is already hit for N=200.
In some sense, SLT says that the broad basin intuition is broadly correct but this needs to be heavily caveated. Our low-dimensional intuition for broad basin is misleading. For singular statistical models (again everything used in ML is highly singular) the local geometry around the minima in high dimensions is very weird.
Maybe you’ve heard of the behaviour of the volume of a sphere in high dimensions: most of it is contained on the shell. I like to think of the local geometry as some sort of fractal sea urchin. Maybe you like that picture, maybe you don’t but it doesn’t matter. SLT gives actual math that is provably the right thing for a Bayesian learner.
[real ML practice isn’t Bayesian learning though? Yes, this is true. Nevertheless, there is both empirical and mathematical evidence that the Bayesian quantitites are still highly relevant for actual learning]
SLT says that the Bayesian posterior is controlled by the local geometry of the minimum. The dominant factor for N~>= 200 is the fractal dimension of the minimum. This is the RLCT and it is the most important quantity of SLT.
There are some misconception about the RLCT floating around. One way to think about is as an ‘effective fractal dimension’ but one has to be careful about this. There is a notion of effective dimension in the standard ML literature where one takes the parameter count and mods out parameters that don’t do anything (because of symmetries). The RLCT picks up on symmetries but it is not just that. It picks up on how degenerate directions in the fischer information metric are ~= how broad is the basin in that direction.
Let’s consider a maximally simple example to get some intuition. Let the population loss function be L(w)=w2k. The number of parameters d=1 and the minimum is at w=0.
For k=1 the minimum is nondegenerate (the second derivative is nonzero). In this case the RLCT is λ=d2 half the dimension. In our case the dimension is just 1 so λ=12
For k>1 the minimum is degenerate (the second derivative is zero). Analyses based on studying the second derivatives will not see the difference between k=2,3,4,5,6,...but in fact the local geometry is vastly different. The higher k is the broader the basin around the minimum. The RLCT for L(w)=w2k is λ=12k. This means, the λ is lower the ‘broader’ the basin is.
Okay so far this only recapitulates the broad basin story. But there are some important points
this is an actual quantity that can be estimated at scale for real networks that provably dominates the learning behaviour for moderately large N.
SLT says that the minima with low rlct will be preferred. It evens says how much they will be preferred. There is tradeoff between lower rlct minima with moderate loss (‘simpler solutions’) and minima with higher rlct but lower loss. As This means that the RLCT is actually ‘the right notion of model complexity/ simplicty’ in the parameterized Bayesian setting. This is too much to recap in this comment but I refer you to Hoogland & van Wingerden’s post here. This is the also the start of the phase transition story which I regard as the principal insight of SLT.
The RLCT doesn’t just pick up on basin broadness. It also picks up on more elaborate singular structure. E.g. a crossing valley type minimum like L(w1,w2)=w41w62. I won’t tell you the answer but you can calculate it yourself using Shaowei Lin’s cheat sheet. This is key—actual neural networks have highly highly singular structure that determines the RLCT.
The RLCT is the most important quantity in SLT but SLT is not just about the RLCT. For instance, the second most important quantity the ‘singular fluctuation’ is also quite important. It has a strong influence on generaliztion behaviour and is the largest factor in the variance of trained models. It controls approximation to Bayesian learning like the way neural networks are trained.
We’ve seen that the directions defined by the matrix of second derivatives is fundamentally flawed because neural networks are highly singular. Still, there is something noncrazy about studying these directions. There is upcoming work which I can’t discuss in detail yet that explains to large degree how to correct this naive picture both mathematically and empirically.