One other area this is used I think is in finance, where people take covariances of stock movements. There’s too little data to learn the full matrix (most of the result would just be noise), so they instead learn a low-rank approximation of the covariance matrix. (Not sure the extent to which it’s common to have a continuous rank cutoff, as is done here, vs just a hard cutoff, though.)
On the other side of things, if we look at the Muon optimizer, the spectrum of singular values of a weight update tends to naturally decay smoothly, but Muon has the effect of amplifying small singular values and suppressing large ones to get a flattish spectrum. So you’d think this would amplify the effect of noise in the gradients, but I guess maybe that just ends up being helpful for neural net optimization somehow.
One other area this is used I think is in finance, where people take covariances of stock movements. There’s too little data to learn the full matrix (most of the result would just be noise), so they instead learn a low-rank approximation of the covariance matrix. (Not sure the extent to which it’s common to have a continuous rank cutoff, as is done here, vs just a hard cutoff, though.)
On the other side of things, if we look at the Muon optimizer, the spectrum of singular values of a weight update tends to naturally decay smoothly, but Muon has the effect of amplifying small singular values and suppressing large ones to get a flattish spectrum. So you’d think this would amplify the effect of noise in the gradients, but I guess maybe that just ends up being helpful for neural net optimization somehow.