“Humans cannot secure matrices of 175B floating point numbers” seems kind of isomorphic to the statement “Humans cannot secure engines containing 10^23 gas molecules”. There are often laws which govern large ensembles of things which are simpler than counting up every single thing in the ensemble.
The difference between these two is that single weight or small fraction of weights can have disproporionate importance for overall behavior, unlike molecules in gas. While gas is only microscopically chaotic, neural nets are chaotic in macroscopic way.
are they really chaotic? my intuition is that models are quite unchaotic in their weights. this has to be roughly the case, since models where some parameters are extremely sensitive would be poorly conditioned and train poorly.
Chaotic in a sense that small changes in weights can produce large difference in behavior at least in some conditions, which is the thing we are wary of.
Securing i.e. nuclear warhead material and preventing its proliferation is indeed challenging.
The laws governing bulk chemistry are more complicated than you’d hope, which we have little ability to prevent or fix. See disappearing polymorphs for example.
The problem isn’t the matrix of numbers on its own, which are inert, it’s the overall hardware/software “AI device” of which the weights are a critical component.
The existence of predictable laws constraining the behavior of masses of X doesn’t mean that we can predict that behavior perfectly. An AI won’t be able to go faster than the speed of light. That is a constraint but it doesn’t help with the problems we worry about.
I agree with all of these, but most of these problems would be equally bad whether was or rather than . What I’m disputing is the argument from “Look at this large number of things which we cannot individually measure”.
(I was tempted to say that disappearing polymorphs are an example, because if we had times as many atoms, then it would take much less time for the first crystal of the stable state to appear, but then we just wouldn’t have discovered the meta-stable state of that particular molecule at all. Instead, in world we’d run into problems with a different polymorph which would take years to disappear in our universe.)
Linking micro models to bulk dynamics has been intellectually fruitful in many fields.
However, there seem to be vast differences between fields in terms of what dynamics are of interest and how predictable they are.
Model weights can in fact be individually measured and combinations studied and examined in isolation. That‘s the trivial counter to the argument as your follow-up comment presents it.
An advantage unique to AI interpretability is the tractability of experimentation. As long as we stay in the domain of bits, we can in principle arbitrarily tweak inputs and weights at a low cost per data point. Other fields work in the domain of atoms and don’t have nearly the same level of freedom.
Nevertheless, the point remains that some points of interest scale in prediction difficulty with the size of the system. I think the concern is really that what we most want to predict—large AI model behaviors in the world of atoms with recursive inputs, where each AI, context and harness differs—seems likely to become harder to predict with increased model size, extended time horizons and a wider range of contexts.
Chemistry is a potentially misleading analogy, because when we study it or implement industrial chemical methods, we do our best to rigorously control the environment in which it’s applied because what we want is a specific, monotonous, consistent product. AI is different. We want adaptability to diverse contexts and the ability to satisfy multiple, often competing objectives. So the interest is different.
We do not currently know the laws that apply to superintelligent systems. It took centuries to discover e.g. the laws of motion and gravity. And it is substantially harder in the case of superintelligence, since we cannot run experiments on superintelligences.
That is true, but I highly doubt that statistical mechanics-flavoured results—for example facts about the order of feature learning under the AdamW optimizer as opposed to the Muon optimizer—are sensitive to the underlying intelligence of the thing being implemented on the weights.
I’m not trying to globally claim that alignment of superintelligence will be easy, or even possible, I’m trying to locally claim that the argument from too many floating point numbers is not a compelling one. Most likely the fundamental objects that are worth studying are far, far above the level of individual floating point numbers in the weights in the same way that the fundamental objects of study in gas physics sit far above the level of individual gas molecules.
It was my impression that practically all results related to large width limits, feature learning, and so on, are always more or less independent of the data the network is trained on. Arguably, this also means that any kind of behavior or structure which only arises from sensible data (as opposed to random garbage) cannot be explained by these results. So for me, this limits a lot what kind of human comprehension this kind of physics-style analysis of neural networks can enable.
This is true of most existing “classical” NTK work, yes. It is not true of all work: mean field theory is data-dependent and works for finite-sized networks. I can imagine some theory coming out of the natural latents work which makes data-dependent predictions which are somewhat size-independent, or at least interact with size in a predictable way.
Ok I think I roughly see where you are coming from. First of all, I agree the distribution can appear as a parameter in the mean field theory, but I meant that the machinery of the theory is mostly independent of this distribution. So only very coarse-grained descriptions perhaps (like moments or regularity) do come in, but nothing which could differentiate “intelligent data vs garbage”. The machinery is mostly “raw distribution in, raw distribution out” but no understanding depending on (or about) the distribution is added by this machinery. Now as I understand you (and correct me if I’m wrong): this is still valuable because it could allow one, if finding some theory which explains data (like natural latents?), to transfer this theory to neural networks? So the challenge of understanding the data is still somewhat orthogonal to the physics-style machinery for neural networks, but at least this physics style machinery could provide a way to transfer this understanding at some point? Again, what I meant initially was just that I feel like the simple and useful laws coming out of the machinery itself, like initialization/scheduling/… are mostly data independent and hence also independent of actual intelligent behavior.
Fully agree with your intent here, which is to say that these are systems that have regular predictable behavior (eg they don’t devolve into chaos), therefore we should be able to derive high-level frameworks/understandings of those dynamics on a mathematical level that are greater than the sum of their parts. It’s obviously easy to take potshots at where the rough edges are here, but I don’t think it takes a lot of charitable interpretation of system dynamics to see the coherence in your point.
what we should be able to do experimentally is to which degree a smarter system can overpower a weaker system. There is the argument with chimpanzees vs humans, but the question is what about IQ 80 vs IQ 100 etc.
“Humans cannot secure matrices of 175B floating point numbers” seems kind of isomorphic to the statement “Humans cannot secure engines containing 10^23 gas molecules”. There are often laws which govern large ensembles of things which are simpler than counting up every single thing in the ensemble.
The difference between these two is that single weight or small fraction of weights can have disproporionate importance for overall behavior, unlike molecules in gas. While gas is only microscopically chaotic, neural nets are chaotic in macroscopic way.
are they really chaotic? my intuition is that models are quite unchaotic in their weights. this has to be roughly the case, since models where some parameters are extremely sensitive would be poorly conditioned and train poorly.
Chaotic in a sense that small changes in weights can produce large difference in behavior at least in some conditions, which is the thing we are wary of.
do we have evidence that this is true?
I think we have plenty of suggestive evidence, like existence of super weights?
More evidence here as well: nondeterminism, STEVE-1 nondeterminism.
Humans do struggle to secure tanks of gas.
Securing i.e. nuclear warhead material and preventing its proliferation is indeed challenging.
The laws governing bulk chemistry are more complicated than you’d hope, which we have little ability to prevent or fix. See disappearing polymorphs for example.
The problem isn’t the matrix of numbers on its own, which are inert, it’s the overall hardware/software “AI device” of which the weights are a critical component.
The existence of predictable laws constraining the behavior of masses of X doesn’t mean that we can predict that behavior perfectly. An AI won’t be able to go faster than the speed of light. That is a constraint but it doesn’t help with the problems we worry about.
I agree with all of these, but most of these problems would be equally bad whether
(I was tempted to say that disappearing polymorphs are an example, because if we had
Linking micro models to bulk dynamics has been intellectually fruitful in many fields.
However, there seem to be vast differences between fields in terms of what dynamics are of interest and how predictable they are.
Model weights can in fact be individually measured and combinations studied and examined in isolation. That‘s the trivial counter to the argument as your follow-up comment presents it.
An advantage unique to AI interpretability is the tractability of experimentation. As long as we stay in the domain of bits, we can in principle arbitrarily tweak inputs and weights at a low cost per data point. Other fields work in the domain of atoms and don’t have nearly the same level of freedom.
Nevertheless, the point remains that some points of interest scale in prediction difficulty with the size of the system. I think the concern is really that what we most want to predict—large AI model behaviors in the world of atoms with recursive inputs, where each AI, context and harness differs—seems likely to become harder to predict with increased model size, extended time horizons and a wider range of contexts.
Chemistry is a potentially misleading analogy, because when we study it or implement industrial chemical methods, we do our best to rigorously control the environment in which it’s applied because what we want is a specific, monotonous, consistent product. AI is different. We want adaptability to diverse contexts and the ability to satisfy multiple, often competing objectives. So the interest is different.
We do not currently know the laws that apply to superintelligent systems. It took centuries to discover e.g. the laws of motion and gravity. And it is substantially harder in the case of superintelligence, since we cannot run experiments on superintelligences.
That is true, but I highly doubt that statistical mechanics-flavoured results—for example facts about the order of feature learning under the AdamW optimizer as opposed to the Muon optimizer—are sensitive to the underlying intelligence of the thing being implemented on the weights.
I’m not trying to globally claim that alignment of superintelligence will be easy, or even possible, I’m trying to locally claim that the argument from too many floating point numbers is not a compelling one. Most likely the fundamental objects that are worth studying are far, far above the level of individual floating point numbers in the weights in the same way that the fundamental objects of study in gas physics sit far above the level of individual gas molecules.
It was my impression that practically all results related to large width limits, feature learning, and so on, are always more or less independent of the data the network is trained on. Arguably, this also means that any kind of behavior or structure which only arises from sensible data (as opposed to random garbage) cannot be explained by these results. So for me, this limits a lot what kind of human comprehension this kind of physics-style analysis of neural networks can enable.
This is true of most existing “classical” NTK work, yes. It is not true of all work: mean field theory is data-dependent and works for finite-sized networks. I can imagine some theory coming out of the natural latents work which makes data-dependent predictions which are somewhat size-independent, or at least interact with size in a predictable way.
Ok I think I roughly see where you are coming from. First of all, I agree the distribution can appear as a parameter in the mean field theory, but I meant that the machinery of the theory is mostly independent of this distribution. So only very coarse-grained descriptions perhaps (like moments or regularity) do come in, but nothing which could differentiate “intelligent data vs garbage”. The machinery is mostly “raw distribution in, raw distribution out” but no understanding depending on (or about) the distribution is added by this machinery. Now as I understand you (and correct me if I’m wrong): this is still valuable because it could allow one, if finding some theory which explains data (like natural latents?), to transfer this theory to neural networks? So the challenge of understanding the data is still somewhat orthogonal to the physics-style machinery for neural networks, but at least this physics style machinery could provide a way to transfer this understanding at some point? Again, what I meant initially was just that I feel like the simple and useful laws coming out of the machinery itself, like initialization/scheduling/… are mostly data independent and hence also independent of actual intelligent behavior.
Fully agree with your intent here, which is to say that these are systems that have regular predictable behavior (eg they don’t devolve into chaos), therefore we should be able to derive high-level frameworks/understandings of those dynamics on a mathematical level that are greater than the sum of their parts. It’s obviously easy to take potshots at where the rough edges are here, but I don’t think it takes a lot of charitable interpretation of system dynamics to see the coherence in your point.
what we should be able to do experimentally is to which degree a smarter system can overpower a weaker system. There is the argument with chimpanzees vs humans, but the question is what about IQ 80 vs IQ 100 etc.
more like predicting the weather than predicting the blackbody dynamics of the earth.