Have you read “Why does deep and cheap learning work so well?” It’s referenced in “When and Why Are Deep Networks Better than Shallow Ones?”, I liked the explanation of how the hierarchical nature of physical processes mean that the subset of functions we care about will tend to have a hierarchical structure and so be well-suited for deep networks to model.
Yes, this is a great paper, and one of the first papers that put me on to the depth separation literature. Can definitely recommend.
Re: “the explanation of how the hierarchical nature of physical processes mean that the subset of functions we care about will tend to have a hierarchical structure and so be well-suited for deep networks to model,” I think this is a fascinating topic, and there’s much to be said here. My personal view here is that this question (but not the answer) is essentially equivalent to the physical Church-Turing thesis: somehow, reality is something that can universally be well-described by compositional procedures (i.e. programs). Searching around for “explanations of the physical Church-Turing thesis” will point you to a wider literature in physics and philosophy on the topic.
Have you read “Why does deep and cheap learning work so well?” It’s referenced in “When and Why Are Deep Networks Better than Shallow Ones?”, I liked the explanation of how the hierarchical nature of physical processes mean that the subset of functions we care about will tend to have a hierarchical structure and so be well-suited for deep networks to model.
Yes, this is a great paper, and one of the first papers that put me on to the depth separation literature. Can definitely recommend.
Re: “the explanation of how the hierarchical nature of physical processes mean that the subset of functions we care about will tend to have a hierarchical structure and so be well-suited for deep networks to model,” I think this is a fascinating topic, and there’s much to be said here. My personal view here is that this question (but not the answer) is essentially equivalent to the physical Church-Turing thesis: somehow, reality is something that can universally be well-described by compositional procedures (i.e. programs). Searching around for “explanations of the physical Church-Turing thesis” will point you to a wider literature in physics and philosophy on the topic.