Ok I think I roughly see where you are coming from. First of all, I agree the distribution can appear as a parameter in the mean field theory, but I meant that the machinery of the theory is mostly independent of this distribution. So only very coarse-grained descriptions perhaps (like moments or regularity) do come in, but nothing which could differentiate “intelligent data vs garbage”. The machinery is mostly “raw distribution in, raw distribution out” but no understanding depending on (or about) the distribution is added by this machinery. Now as I understand you (and correct me if I’m wrong): this is still valuable because it could allow one, if finding some theory which explains data (like natural latents?), to transfer this theory to neural networks? So the challenge of understanding the data is still somewhat orthogonal to the physics-style machinery for neural networks, but at least this physics style machinery could provide a way to transfer this understanding at some point? Again, what I meant initially was just that I feel like the simple and useful laws coming out of the machinery itself, like initialization/scheduling/… are mostly data independent and hence also independent of actual intelligent behavior.
Ok I think I roughly see where you are coming from. First of all, I agree the distribution can appear as a parameter in the mean field theory, but I meant that the machinery of the theory is mostly independent of this distribution. So only very coarse-grained descriptions perhaps (like moments or regularity) do come in, but nothing which could differentiate “intelligent data vs garbage”. The machinery is mostly “raw distribution in, raw distribution out” but no understanding depending on (or about) the distribution is added by this machinery. Now as I understand you (and correct me if I’m wrong): this is still valuable because it could allow one, if finding some theory which explains data (like natural latents?), to transfer this theory to neural networks? So the challenge of understanding the data is still somewhat orthogonal to the physics-style machinery for neural networks, but at least this physics style machinery could provide a way to transfer this understanding at some point? Again, what I meant initially was just that I feel like the simple and useful laws coming out of the machinery itself, like initialization/scheduling/… are mostly data independent and hence also independent of actual intelligent behavior.