Looking at matrix weights through the de-embedding matrix looks interesting!
I’m unsure what kind of “matrix action” you’re hoping to capture with SVD.
In the case of symmetric square matrices, the singular directions are the eigenvectors, which are the vectors along which the matrix only multiplies them by a constant value. If the scaling factor is positive, this is what I would call “inaction”. On the other hand, even a symmetric square matrix can “stretch” vectors in interesting ways. For example, if you take (1000), I would say that the “interesting action” is not done to the singular directions (one of which is sent to zero, and the other one is kept intact), but something interesting is going on with (11) and (1−1), they both get sent to the same vector.
So I’m unsure what interesting algorithm could be captured only by looking at singular directions. But maybe you’re onto something, and there are other quantities computed in similar ways which could be more significant! Or maybe my intuition about square symmetric matrices is hiding me the interesting things that SVD’s singular directions represent. What do you think?
My bad. My intuitions about eigenvectors mislead me, and I now disagree with my comment. zfurman, on EleutherAI, gave me a much better frame to see what SVD does: SVD helps you find where the action happens in the sense that it tells you where it is read, and where it is written (in decreasing order of importance), by decomposing the transformation into a sum of [dot product with a right singular vector, scale by the corresponding singular value, multiply by the corresponding left singular vector]. This does capture a significant amount of “where the action happens”, and is a much better frame than the “rotate scale rotate” frame I had previously learned.
Looking at matrix weights through the de-embedding matrix looks interesting!
I’m unsure what kind of “matrix action” you’re hoping to capture with SVD.
In the case of symmetric square matrices, the singular directions are the eigenvectors, which are the vectors along which the matrix only multiplies them by a constant value. If the scaling factor is positive, this is what I would call “inaction”. On the other hand, even a symmetric square matrix can “stretch” vectors in interesting ways. For example, if you take (1000), I would say that the “interesting action” is not done to the singular directions (one of which is sent to zero, and the other one is kept intact), but something interesting is going on with (11) and (1−1), they both get sent to the same vector.
So I’m unsure what interesting algorithm could be captured only by looking at singular directions. But maybe you’re onto something, and there are other quantities computed in similar ways which could be more significant! Or maybe my intuition about square symmetric matrices is hiding me the interesting things that SVD’s singular directions represent. What do you think?
My bad. My intuitions about eigenvectors mislead me, and I now disagree with my comment. zfurman, on EleutherAI, gave me a much better frame to see what SVD does: SVD helps you find where the action happens in the sense that it tells you where it is read, and where it is written (in decreasing order of importance), by decomposing the transformation into a sum of [dot product with a right singular vector, scale by the corresponding singular value, multiply by the corresponding left singular vector]. This does capture a significant amount of “where the action happens”, and is a much better frame than the “rotate scale rotate” frame I had previously learned.