I kinda agree, but it also could be smeared all over the NN, such that naively dropping some weights or something does not work, as you are pocking holes under wrong representation.
People haven’t just tried dropping weights in the neuron basis. They’ve e.g. searched out low eigenmodes of the Hessian and dropped those.
I kinda agree, but it also could be smeared all over the NN, such that naively dropping some weights or something does not work, as you are pocking holes under wrong representation.
People haven’t just tried dropping weights in the neuron basis. They’ve e.g. searched out low eigenmodes of the Hessian and dropped those.