Thought: couldn’t you make a lossless SAE using something along the lines of:
Represent the parameters of the SAE as simply a set of unit vectors for the feature directions.
To encode a vector using the SAE, iterate: find the most aligned feature vector, dot them to get the coefficient for that feature vector, and subtract off the scaled feature vector to get a residual to encode further
With plenty of diverse vectors, this should presumably guarantee excellent reconstruction, so the main issue is to ensure high sparsity, which could be achieved by some sparsity loss (which could probably be more aggressive than L1, due to the better reconstruction?). Main issue I see is the encoding method being inefficient. Dunno if one can optimize my proposal to be fast enough.
Thought: couldn’t you make a lossless SAE using something along the lines of:
Represent the parameters of the SAE as simply a set of unit vectors for the feature directions.
To encode a vector using the SAE, iterate: find the most aligned feature vector, dot them to get the coefficient for that feature vector, and subtract off the scaled feature vector to get a residual to encode further
With plenty of diverse vectors, this should presumably guarantee excellent reconstruction, so the main issue is to ensure high sparsity, which could be achieved by some sparsity loss (which could probably be more aggressive than L1, due to the better reconstruction?). Main issue I see is the encoding method being inefficient. Dunno if one can optimize my proposal to be fast enough.