You are looking for low information directions in activation space. That is, what directions have the model already compressed and would take very few bits to move the model to activate more in that direction?
A Gaussian perturbation to the model parameters (not activations) gives an unbiased estimate for the number of bits it takes to update the model under gradient flow to activate in that direction. To be precise, if is an activation (or loss), then
where is the learning rate (how quickly you want to move the model). For the RHS automatically includes information regularization—how much information is there in the activation plus the information to move the model there? This is actually much better than treating the activation (or loss) as free information.
A Gaussian perturbation to the model activations (not parameters) is almost a Gaussian perturbation to the parameters, as neural networks are almost linear. So this is why you still see signal when perturbing the activations instead of the parameters.
All of this gets much more complicated with singular learning theory. In truth, is only the number of bits it takes to move a nonsingular model under gradient flow. But neural networks are very singular. A very expensive way to recover the correct estimate is to do a whole bunch of Monte Carlo simulations, where you actually train the network while adding a tiny bit of noise.
You are looking for low information directions in activation space. That is, what directions have the model already compressed and would take very few bits to move the model to activate more in that direction?
A Gaussian perturbation to the model parameters (not activations) gives an unbiased estimate for the number of bits it takes to update the model under gradient flow to activate in that direction. To be precise, if
where
A Gaussian perturbation to the model activations (not parameters) is almost a Gaussian perturbation to the parameters, as neural networks are almost linear. So this is why you still see signal when perturbing the activations instead of the parameters.
All of this gets much more complicated with singular learning theory. In truth,