Various takes on this research direction, from someone who hasn’t thought about it very much:
Given two optimal models in a neural network’s weight space, is it possible to find a path between them comprised entirely of other optimal models?
I think that the answer here is definitely no, without further assumptions. For example, consider the case where you have a model with two ReLU neurons that, for optimal models, are doing different things; the model where you swap the two neurons is equally performant, but there’s no continuous path between these models.
More concretely, imagine that we want to learn (y1, y2) = (ReLU(x1), ReLU(x2)) with a model of the form
def model(x1, x2):
h1 = ReLU(a * x1 + b * x2)
h2 = ReLU(c * x1 + d * x2)
return (e * h1 + f * h2, g * h1 + h * h2)
where a through h are parameters. In Pytorch we’d write this as W2 @ relu(W1 @ x), for W1 and W2 of shape [2, 2].
We can model the true function perfectly by either using the first neuron for x1 and the second for x2 or vice versa, but there’s no continuous path between them.
(As a simpler example, consider the task of choosing parameters a and b such that a*b = 1. There’s no continuous path from the (1, 1) solution to the (-1, −1) solution.)
Plausibly you could choose some further assumptions that make this true. You gesture at this a little:
However, when the class of models is sufficiently overparametrized (i.e., large-scale neural net architecture), we eventually have just one connected ridge of local minima: the ridge of global minima.
But IMO a whole lot of the work here is going to be choosing a definition of “sufficiently overparameterized” such that the theorem isn’t easily counterexampled.
And from a relevance-to-alignment standpoint, I think it’s pretty likely that if you succeed at finding a definition of “sufficiently overparameterized” such that this is true, that definition will require that the model is sufficiently “larger” than the task it’s modeling that it will seem implausible that the theorem will apply in the real world (given that I expect the real world to be “larger” than AGIs we train, and so don’t expect AGIs to be wildly overparameterized).
Another way of phrasing my core objection: the original question without further assumptions seems equivalent to “given two global minima of a function, is there a path between them entirely comprised of global minima”, which is obviously false.
We definitely aren’t trying to say “any two ridges can be connected for any loss landscape” but rather that the ridges for overparameterised networks are connected.
TL;DR: I agree that the answer to the question above definitely isn’t always yes, because of your counterexample, but I think that moving forward on a similar research direction might be useful anyways.
One can imagine partitioning the parameter space into sets that arrive at basins where each model in the basin has the same, locally optimal performance, this is like a Rashomon set (relaxing the requirement from global minima so that we get a partition of the space). The models which can compress the training data (and thus have free parameters) are generally more likely to be found by random selection and search, because the free parameters means that the dimensionality of this set is higher, and hence exponentially more likely.
Thus, we can move within these high-dimensional regions of locally optimal loss, which could allow us to find more interpretable (or maybe more desirable along another axis), which is the stated motivation in the article:
Ultimately, we hope that the study of equivalently optimal models would lead to advances in interpretability: for example, by producing models that are simultaneously optimal and interpretable.
This seems super relevant to alignment! The default path to AGI right now to me seems like something like a LLM world model hooked up to some RL to make it more agenty, and I expect this kind of theory applied to LLMs, because of the large number of parameters. I’m hoping that this theory gets us better predictions on which Rashomon sets are found (this would look like a selection theorem), and the ability to move within a Rashomon set towards parameters that are better. Such a selection theorem seems likely because of the dimensionality argument above.
Thank you so much, Thomas and Buck, for reading the post and for your insightful comments!
It is indeed true that some functions have two global minimizers that are not path-connected. Empirically, very overparametrized models which are trained on “non-artificial” datasets (“datasets from nature”?) seem to have a connected Rashomon manifold. It would definitely be helpful to know theoretically why this tends to happen, and when this wouldn’t happen.
One heuristic argument for why two disconnected global minimizers might only happen in “artificial” datasets might go something like this. Given two quantities, one is larger than the other, unless there is a symmetry-based reason why they are actually secretly the same quantity. Under this heuristic, a non-overparametrized model’s loss landscape has a global minimum achieved by precisely one point, and potentially some suboptimal local minima as well. But overparametrizing the model makes the suboptimal local minima not local minima anymore (by making them saddle points?) while the single global minimizer is “stretched out” to a whole submanifold. This “stretching out” is the symmetry; all optimal models on this submanifold are secretly the same.
One situation where this heuristic fails is if there are other types of symmetry, like rotation. Then, applying this move to a global minimizer could get you other global minimizers which are not connected to each other. In this case, “modding out by the symmetry” is not decreasing the dimension, but taking the quotient by the symmetry group which gives you a quotient space of the same dimension. I’m guessing these types of situations are more common in “artificial” datasets which have not modded out all the obvious symmetries yet.
In your example, I think even adding just one more node, h3, to the hidden layer would suffice to connect the two solutions. One node per dimension of input suffices to learn the function, but it’s also possible for two nodes to share the task between them, where the share of the task they are picking up can vary continuously from 0 to 1. So just have h3 take over x2 from h2, then h2 takes over x1 from h1, and then h1 takes over x2 from h3.
Have you seen this paper? They find that “SGD solutions are connected via a piecewise linear path, and the increase in loss along this path vanishes as the number of neurons grows large.”
Various takes on this research direction, from someone who hasn’t thought about it very much:
I think that the answer here is definitely no, without further assumptions. For example, consider the case where you have a model with two ReLU neurons that, for optimal models, are doing different things; the model where you swap the two neurons is equally performant, but there’s no continuous path between these models.
More concretely, imagine that we want to learn (y1, y2) = (ReLU(x1), ReLU(x2)) with a model of the form
where a through h are parameters. In Pytorch we’d write this as
W2 @ relu(W1 @ x), for W1 and W2 of shape [2, 2].We can model the true function perfectly by either using the first neuron for x1 and the second for x2 or vice versa, but there’s no continuous path between them.
(As a simpler example, consider the task of choosing parameters a and b such that a*b = 1. There’s no continuous path from the (1, 1) solution to the (-1, −1) solution.)
Plausibly you could choose some further assumptions that make this true. You gesture at this a little:
But IMO a whole lot of the work here is going to be choosing a definition of “sufficiently overparameterized” such that the theorem isn’t easily counterexampled.
And from a relevance-to-alignment standpoint, I think it’s pretty likely that if you succeed at finding a definition of “sufficiently overparameterized” such that this is true, that definition will require that the model is sufficiently “larger” than the task it’s modeling that it will seem implausible that the theorem will apply in the real world (given that I expect the real world to be “larger” than AGIs we train, and so don’t expect AGIs to be wildly overparameterized).
Another way of phrasing my core objection: the original question without further assumptions seems equivalent to “given two global minima of a function, is there a path between them entirely comprised of global minima”, which is obviously false.
Hey Buck, love the line of thinking.
We definitely aren’t trying to say “any two ridges can be connected for any loss landscape” but rather that the ridges for overparameterised networks are connected.
TL;DR: I agree that the answer to the question above definitely isn’t always yes, because of your counterexample, but I think that moving forward on a similar research direction might be useful anyways.
One can imagine partitioning the parameter space into sets that arrive at basins where each model in the basin has the same, locally optimal performance, this is like a Rashomon set (relaxing the requirement from global minima so that we get a partition of the space). The models which can compress the training data (and thus have free parameters) are generally more likely to be found by random selection and search, because the free parameters means that the dimensionality of this set is higher, and hence exponentially more likely.
Thus, we can move within these high-dimensional regions of locally optimal loss, which could allow us to find more interpretable (or maybe more desirable along another axis), which is the stated motivation in the article:
This seems super relevant to alignment! The default path to AGI right now to me seems like something like a LLM world model hooked up to some RL to make it more agenty, and I expect this kind of theory applied to LLMs, because of the large number of parameters. I’m hoping that this theory gets us better predictions on which Rashomon sets are found (this would look like a selection theorem), and the ability to move within a Rashomon set towards parameters that are better. Such a selection theorem seems likely because of the dimensionality argument above.
Edit: Adding a link to “Git Re-Basin: Merging Models modulo Permutation Symmetries,” a relevant paper that has recently been posted on arXiv.
Thank you so much, Thomas and Buck, for reading the post and for your insightful comments!
It is indeed true that some functions have two global minimizers that are not path-connected. Empirically, very overparametrized models which are trained on “non-artificial” datasets (“datasets from nature”?) seem to have a connected Rashomon manifold. It would definitely be helpful to know theoretically why this tends to happen, and when this wouldn’t happen.
One heuristic argument for why two disconnected global minimizers might only happen in “artificial” datasets might go something like this. Given two quantities, one is larger than the other, unless there is a symmetry-based reason why they are actually secretly the same quantity. Under this heuristic, a non-overparametrized model’s loss landscape has a global minimum achieved by precisely one point, and potentially some suboptimal local minima as well. But overparametrizing the model makes the suboptimal local minima not local minima anymore (by making them saddle points?) while the single global minimizer is “stretched out” to a whole submanifold. This “stretching out” is the symmetry; all optimal models on this submanifold are secretly the same.
One situation where this heuristic fails is if there are other types of symmetry, like rotation. Then, applying this move to a global minimizer could get you other global minimizers which are not connected to each other. In this case, “modding out by the symmetry” is not decreasing the dimension, but taking the quotient by the symmetry group which gives you a quotient space of the same dimension. I’m guessing these types of situations are more common in “artificial” datasets which have not modded out all the obvious symmetries yet.
In your example, I think even adding just one more node, h3, to the hidden layer would suffice to connect the two solutions. One node per dimension of input suffices to learn the function, but it’s also possible for two nodes to share the task between them, where the share of the task they are picking up can vary continuously from 0 to 1. So just have h3 take over x2 from h2, then h2 takes over x1 from h1, and then h1 takes over x2 from h3.
Yeah, but you have the same problem if you were using all three of the nodes in the hidden layer.
Actual trained neural networks are known to have redundant parameters, as demonstrated by the fact that we can prune them so much.
Have you seen this paper? They find that “SGD solutions are connected via a piecewise linear path, and the increase in loss along this path vanishes as the number of neurons grows large.”