my hot take is circuit sparsity is still the best way to get natural features. by making it expensive to read from and write to many residual channels, you encourage frequently used concepts to be stored in only a few residual channels.
my hot take is circuit sparsity is still the best way to get natural features. by making it expensive to read from and write to many residual channels, you encourage frequently used concepts to be stored in only a few residual channels.