Do you have (and does anyone have) a take on Martin and Mahoney 2018? It’s something I read in the pre-GPT dark ages, alleging a multitude of phases (distinguished by the distribution of eigenvalues) in deep learning networks. I have no idea if it has been subsequently validated or refuted or otherwise built upon.
I do indeed! I just wrote this post about a cluster of surrounding ideas. Tl;dr it’s interesting, and we sooorta see this kind of behaviour in modern ML models, but certainly not cleanly. While I’m not convinced that their story plays out cleanly irl, I’ve been thinking a lot over the past week about the spikes-->power-law transition as a toy theoretical model. My hope is that such a model would tell us something about a “phase-transition from memorization to generalization”. TBD.
Do you have (and does anyone have) a take on Martin and Mahoney 2018? It’s something I read in the pre-GPT dark ages, alleging a multitude of phases (distinguished by the distribution of eigenvalues) in deep learning networks. I have no idea if it has been subsequently validated or refuted or otherwise built upon.
I do indeed! I just wrote this post about a cluster of surrounding ideas. Tl;dr it’s interesting, and we sooorta see this kind of behaviour in modern ML models, but certainly not cleanly. While I’m not convinced that their story plays out cleanly irl, I’ve been thinking a lot over the past week about the spikes-->power-law transition as a toy theoretical model. My hope is that such a model would tell us something about a “phase-transition from memorization to generalization”. TBD.