For future reference, after speaking more with Dmitry + reading more of the papers linked in the Simon et al “Scientific Theory of Deep Learning” paper, I’ve become (slightly) more positive on deep learning theory, at least the mean-field line of work he mentioned in his comments.
For future reference, after speaking more with Dmitry + reading more of the papers linked in the Simon et al “Scientific Theory of Deep Learning” paper, I’ve become (slightly) more positive on deep learning theory, at least the mean-field line of work he mentioned in his comments.
I wrote up a bit of the history of that line of work, and why I’ve become more optimistic on deep learning theory in the past few days: https://www.lesswrong.com/posts/6SRq7mZ97Dwuavwb6/maybe-i-was-too-harsh-on-deep-learning-theory-three-days-ago