It is unimportant in the limit (of infinite data), but away from that limit, it is only unimportant by a factor of 1/log(data), which seems small enough to be beatable in practice in some circumstances.
The spectra of things like Hessians tend to be singular, yes, but also sort of power-law. This makes the dimensionality a bit fuzzy and (imo) makes it possible for absolute volume scale of basins to compete with dimensionality.
Essentially: it’s not clear that a 301-dimensional sphere really is “bigger” than a 300-dimensional sphere, if the 300-dimensional sphere has a much larger radius. (Obviously it’s true in a strict sense, but hopefully you know what I’m gesturing at here.)
I am not sure I agree :)
It is unimportant in the limit (of infinite data), but away from that limit, it is only unimportant by a factor of 1/log(data), which seems small enough to be beatable in practice in some circumstances.
The spectra of things like Hessians tend to be singular, yes, but also sort of power-law. This makes the dimensionality a bit fuzzy and (imo) makes it possible for absolute volume scale of basins to compete with dimensionality.
Essentially: it’s not clear that a 301-dimensional sphere really is “bigger” than a 300-dimensional sphere, if the 300-dimensional sphere has a much larger radius. (Obviously it’s true in a strict sense, but hopefully you know what I’m gesturing at here.)