Would be interesting to try to distinguish between 3 types of dimensions. Number of dimensions that leave the set, number of dimensions in the set due to trivial params, and number of dimensions in the set due to equally good but functionally different models.
Especially if it turns out it is attracted to maximum trivial params, but not maximum dimensionality overall.
Yeah that would be interesting, but how would we tell the difference between trivial params (I’m assuming this means function doesn’t change anywhere) and equal loss models? Estimate this with a sampling of points out of distribution?
I kind of assumed that all changes in the parameters changed the function, but that some areas of the loss landscape change the function faster than others? This would be my prediction
Would be interesting to try to distinguish between 3 types of dimensions. Number of dimensions that leave the set, number of dimensions in the set due to trivial params, and number of dimensions in the set due to equally good but functionally different models.
Especially if it turns out it is attracted to maximum trivial params, but not maximum dimensionality overall.
Yeah that would be interesting, but how would we tell the difference between trivial params (I’m assuming this means function doesn’t change anywhere) and equal loss models? Estimate this with a sampling of points out of distribution?
I kind of assumed that all changes in the parameters changed the function, but that some areas of the loss landscape change the function faster than others? This would be my prediction