The approach here is much faster than the SGLD approach; it only takes tens or hundreds of forward passes to get a decent estimate. Maybe that’s achievable in principle with SGLD, but we haven’t managed it.
I like KFAC but I don’t think estimating the Hessian spectrum better is a bottleneck; in our experiments on tiny models, the true Hessian didn’t even always outperform the ADAM moment estimates. I like the ideas here, though!
The big downside of our approach, compared to Timaeus’s, is that it underestimates basin size (overestimates complexity) for two reasons: 1) Jensen bias: the “pancake” issue, which we can alleviate a bit with preconditioners 2) The “star domain” constraint we impose (requiring line-of-sight between the anchor point and the rest of the basin) is arguably pretty strict, although we think it holds by default for the “KL neighborhood” variant. It’s not clear that this is an obstacle in practice, though, in settings where you just want a metric of complexity that runs fast and has approximately the right theoretical and empirical properties to do practical work with.
We’ve been working on using SGLD and thermodynamic integration to get a more-trusted measurement of basin size, but we suspect the most naive version of our estimator (or the Adam-preconditioned version) will be most practical for downstream applications.
We use average KL divergence over a test set as our behavioral loss, and (for small models where it’s tractable) we use the Hessian of KL, i.e. the Fisher.
Great questions :)
The approach here is much faster than the SGLD approach; it only takes tens or hundreds of forward passes to get a decent estimate. Maybe that’s achievable in principle with SGLD, but we haven’t managed it.
I like KFAC but I don’t think estimating the Hessian spectrum better is a bottleneck; in our experiments on tiny models, the true Hessian didn’t even always outperform the ADAM moment estimates. I like the ideas here, though!
The big downside of our approach, compared to Timaeus’s, is that it underestimates basin size (overestimates complexity) for two reasons:
1) Jensen bias: the “pancake” issue, which we can alleviate a bit with preconditioners
2) The “star domain” constraint we impose (requiring line-of-sight between the anchor point and the rest of the basin) is arguably pretty strict, although we think it holds by default for the “KL neighborhood” variant.
It’s not clear that this is an obstacle in practice, though, in settings where you just want a metric of complexity that runs fast and has approximately the right theoretical and empirical properties to do practical work with.
We’ve been working on using SGLD and thermodynamic integration to get a more-trusted measurement of basin size, but we suspect the most naive version of our estimator (or the Adam-preconditioned version) will be most practical for downstream applications.
We use average KL divergence over a test set as our behavioral loss, and (for small models where it’s tractable) we use the Hessian of KL, i.e. the Fisher.