If we actually had the precision and maturity of understanding to predict this “volume” question, we’d probably (but not definitely) be able to make fundamental contributions to DL generalization theory + inductive bias research.
Obligatory singular learning theory plug: SLT can and does make predictions about the “volume” question. There will be a post soon by @Daniel Murfet that provides a clear example of this.
Cool post, and I am excited about (what I’ve heard of) SLT for this reason—but it seems that that post doesn’t directly address the volume question for deep learning in particular? (And perhaps you didn’t mean to imply that the post would address that question.)
Right. SLT tells us how to operationalize and measure (via the LLC) basin volume in general for DL. It tells us about the relation between the LLC and meaningful inductive biases in the particular setting described in this post. I expect future SLT to give us meaningful predictions about inductive biases in DL in particular.
Obligatory singular learning theory plug: SLT can and does make predictions about the “volume” question. There will be a post soon by @Daniel Murfet that provides a clear example of this.
The post is live here.
Cool post, and I am excited about (what I’ve heard of) SLT for this reason—but it seems that that post doesn’t directly address the volume question for deep learning in particular? (And perhaps you didn’t mean to imply that the post would address that question.)
Right. SLT tells us how to operationalize and measure (via the LLC) basin volume in general for DL. It tells us about the relation between the LLC and meaningful inductive biases in the particular setting described in this post. I expect future SLT to give us meaningful predictions about inductive biases in DL in particular.