Hmm, my intuition/reasoning says that the first couple of Hessian eigenvectors would control 1. the intercept/average output/position of the borders between the classes, 2. the bias-variance tradeoff if you are doing regularization (as you point out is necessary), 3. maybe (am unsure) the relative evidentiary value of different correlated features. (Likely in this order of descending eigenvalues) Does this align with your intuition/experiments?
Hmm, my intuition/reasoning says that the first couple of Hessian eigenvectors would control 1. the intercept/average output/position of the borders between the classes, 2. the bias-variance tradeoff if you are doing regularization (as you point out is necessary), 3. maybe (am unsure) the relative evidentiary value of different correlated features. (Likely in this order of descending eigenvalues) Does this align with your intuition/experiments?