I think it’s possible that gradient descent works by applying a selection pressure to preexisting circuits in the initial randomization with some finetuning. This would explain why most weights are zero after training as well as stuff like the lottery ticket hypothesis.
I think it’s possible that gradient descent works by applying a selection pressure to preexisting circuits in the initial randomization with some finetuning. This would explain why most weights are zero after training as well as stuff like the lottery ticket hypothesis.
As far as I know this is just false, though?