I do indeed! I just wrote this post about a cluster of surrounding ideas. Tl;dr it’s interesting, and we sooorta see this kind of behaviour in modern ML models, but certainly not cleanly. While I’m not convinced that their story plays out cleanly irl, I’ve been thinking a lot over the past week about the spikes-->power-law transition as a toy theoretical model. My hope is that such a model would tell us something about a “phase-transition from memorization to generalization”. TBD.
I do indeed! I just wrote this post about a cluster of surrounding ideas. Tl;dr it’s interesting, and we sooorta see this kind of behaviour in modern ML models, but certainly not cleanly. While I’m not convinced that their story plays out cleanly irl, I’ve been thinking a lot over the past week about the spikes-->power-law transition as a toy theoretical model. My hope is that such a model would tell us something about a “phase-transition from memorization to generalization”. TBD.