Algon comments on Race Along Rashomon Ridge

[Algon]

Looking at your map, you notice there’s a long ridge that leaves your mountain at roughly the same height. Does this ridge to reach your friend’s mountain? possible to follow this ridge to reach your friend’s mountain?

“ridge to” → “ridge”,
”possible” to “Is it possible”.

However, when the class of models is sufficiently overparametrized (i.e., large-scale neural net architecture), we suspect that eventually have just one connected ridge of local minima: the ridge of global minima. The dimension of this single ridge is smaller than the dimension $d$ of the whole weight space, but still quite high-dimensional. Intuitively, the large number of linearly independent zero-loss directions at a global minimum allow for many opportunities to path-connect towards a local minimum while staying within the ridge, making all local minima globally minimal.

“that eventually have” → “they eventually have”,

“path-connect towards” - > “path connect to”.

Considering getting a linter for your browser, like https://​​languagetool.org, to avoid these sorts of errors in future.

One potential choice of a subset of interpretable models $W_{\text{interpretable}}$ which is geometrically “nice″ is the submanifold of models which is prunable in a certain way. For example, the submanifold defined by the system of equations $w_j=w_{j+1}=\cdots =w_k=0$, where $w_j,\dots ,w_k$ are the output connections of a given neuron $n$, is comprised of models which can be pruned by removing neuron number $n$. Thus, we may be able to maximally prune a neural net (with respect to the given dataset) by using an algorithm of the following form:

1. Find in the Rashomon manifold an optimal model that can be pruned in some way.

2. Prune the neural net in that way.

3. Repeat Steps 1 and 2 until there are no simultaneously prunable and optimal models anymore.

This is clever, and also seems like the core idea of the post. You should put this info at the start.