Yes, this is a great paper, and one of the first papers that put me on to the depth separation literature. Can definitely recommend.
Re: “the explanation of how the hierarchical nature of physical processes mean that the subset of functions we care about will tend to have a hierarchical structure and so be well-suited for deep networks to model,” I think this is a fascinating topic, and there’s much to be said here. My personal view here is that this question (but not the answer) is essentially equivalent to the physical Church-Turing thesis: somehow, reality is something that can universally be well-described by compositional procedures (i.e. programs). Searching around for “explanations of the physical Church-Turing thesis” will point you to a wider literature in physics and philosophy on the topic.
Yes, this is a great paper, and one of the first papers that put me on to the depth separation literature. Can definitely recommend.
Re: “the explanation of how the hierarchical nature of physical processes mean that the subset of functions we care about will tend to have a hierarchical structure and so be well-suited for deep networks to model,” I think this is a fascinating topic, and there’s much to be said here. My personal view here is that this question (but not the answer) is essentially equivalent to the physical Church-Turing thesis: somehow, reality is something that can universally be well-described by compositional procedures (i.e. programs). Searching around for “explanations of the physical Church-Turing thesis” will point you to a wider literature in physics and philosophy on the topic.