The Shallow Reality of ‘Deep Learning Theory’

Classical learning theory makes the wrong assumptions, takes the wrong limits, uses the wrong metrics, and aims for the wrong objectives.

In this sequence, I review the current state of learning theory and the many ways in which it is broken. I argue that the field as it currently stands is in need of fix-up, and that developing a useful theory of deep learning will require turning elsewhere, likely to something that builds on singular learning theory.

The shal­low re­al­ity of ‘deep learn­ing the­ory’

Em­piri­cal risk min­i­miza­tion is fun­da­men­tally confused

Ap­prox­i­ma­tion is ex­pen­sive, but the lunch is cheap

Gen­er­al­iza­tion, from ther­mo­dy­nam­ics to statis­ti­cal physics