We do not currently know the laws that apply to superintelligent systems. It took centuries to discover e.g. the laws of motion and gravity. And it is substantially harder in the case of superintelligence, since we cannot run experiments on superintelligences.
That is true, but I highly doubt that statistical mechanics-flavoured results—for example facts about the order of feature learning under the AdamW optimizer as opposed to the Muon optimizer—are sensitive to the underlying intelligence of the thing being implemented on the weights.
I’m not trying to globally claim that alignment of superintelligence will be easy, or even possible, I’m trying to locally claim that the argument from too many floating point numbers is not a compelling one. Most likely the fundamental objects that are worth studying are far, far above the level of individual floating point numbers in the weights in the same way that the fundamental objects of study in gas physics sit far above the level of individual gas molecules.
It was my impression that practically all results related to large width limits, feature learning, and so on, are always more or less independent of the data the network is trained on. Arguably, this also means that any kind of behavior or structure which only arises from sensible data (as opposed to random garbage) cannot be explained by these results. So for me, this limits a lot what kind of human comprehension this kind of physics-style analysis of neural networks can enable.
This is true of most existing “classical” NTK work, yes. It is not true of all work: mean field theory is data-dependent and works for finite-sized networks. I can imagine some theory coming out of the natural latents work which makes data-dependent predictions which are somewhat size-independent, or at least interact with size in a predictable way.
Ok I think I roughly see where you are coming from. First of all, I agree the distribution can appear as a parameter in the mean field theory, but I meant that the machinery of the theory is mostly independent of this distribution. So only very coarse-grained descriptions perhaps (like moments or regularity) do come in, but nothing which could differentiate “intelligent data vs garbage”. The machinery is mostly “raw distribution in, raw distribution out” but no understanding depending on (or about) the distribution is added by this machinery. Now as I understand you (and correct me if I’m wrong): this is still valuable because it could allow one, if finding some theory which explains data (like natural latents?), to transfer this theory to neural networks? So the challenge of understanding the data is still somewhat orthogonal to the physics-style machinery for neural networks, but at least this physics style machinery could provide a way to transfer this understanding at some point? Again, what I meant initially was just that I feel like the simple and useful laws coming out of the machinery itself, like initialization/scheduling/… are mostly data independent and hence also independent of actual intelligent behavior.
Fully agree with your intent here, which is to say that these are systems that have regular predictable behavior (eg they don’t devolve into chaos), therefore we should be able to derive high-level frameworks/understandings of those dynamics on a mathematical level that are greater than the sum of their parts. It’s obviously easy to take potshots at where the rough edges are here, but I don’t think it takes a lot of charitable interpretation of system dynamics to see the coherence in your point.
what we should be able to do experimentally is to which degree a smarter system can overpower a weaker system. There is the argument with chimpanzees vs humans, but the question is what about IQ 80 vs IQ 100 etc.
We do not currently know the laws that apply to superintelligent systems. It took centuries to discover e.g. the laws of motion and gravity. And it is substantially harder in the case of superintelligence, since we cannot run experiments on superintelligences.
That is true, but I highly doubt that statistical mechanics-flavoured results—for example facts about the order of feature learning under the AdamW optimizer as opposed to the Muon optimizer—are sensitive to the underlying intelligence of the thing being implemented on the weights.
I’m not trying to globally claim that alignment of superintelligence will be easy, or even possible, I’m trying to locally claim that the argument from too many floating point numbers is not a compelling one. Most likely the fundamental objects that are worth studying are far, far above the level of individual floating point numbers in the weights in the same way that the fundamental objects of study in gas physics sit far above the level of individual gas molecules.
It was my impression that practically all results related to large width limits, feature learning, and so on, are always more or less independent of the data the network is trained on. Arguably, this also means that any kind of behavior or structure which only arises from sensible data (as opposed to random garbage) cannot be explained by these results. So for me, this limits a lot what kind of human comprehension this kind of physics-style analysis of neural networks can enable.
This is true of most existing “classical” NTK work, yes. It is not true of all work: mean field theory is data-dependent and works for finite-sized networks. I can imagine some theory coming out of the natural latents work which makes data-dependent predictions which are somewhat size-independent, or at least interact with size in a predictable way.
Ok I think I roughly see where you are coming from. First of all, I agree the distribution can appear as a parameter in the mean field theory, but I meant that the machinery of the theory is mostly independent of this distribution. So only very coarse-grained descriptions perhaps (like moments or regularity) do come in, but nothing which could differentiate “intelligent data vs garbage”. The machinery is mostly “raw distribution in, raw distribution out” but no understanding depending on (or about) the distribution is added by this machinery. Now as I understand you (and correct me if I’m wrong): this is still valuable because it could allow one, if finding some theory which explains data (like natural latents?), to transfer this theory to neural networks? So the challenge of understanding the data is still somewhat orthogonal to the physics-style machinery for neural networks, but at least this physics style machinery could provide a way to transfer this understanding at some point? Again, what I meant initially was just that I feel like the simple and useful laws coming out of the machinery itself, like initialization/scheduling/… are mostly data independent and hence also independent of actual intelligent behavior.
Fully agree with your intent here, which is to say that these are systems that have regular predictable behavior (eg they don’t devolve into chaos), therefore we should be able to derive high-level frameworks/understandings of those dynamics on a mathematical level that are greater than the sum of their parts. It’s obviously easy to take potshots at where the rough edges are here, but I don’t think it takes a lot of charitable interpretation of system dynamics to see the coherence in your point.
what we should be able to do experimentally is to which degree a smarter system can overpower a weaker system. There is the argument with chimpanzees vs humans, but the question is what about IQ 80 vs IQ 100 etc.