These are notes from my research undertaken as part of the Iliad Fellowship under the mentorship of Dmitry Vaintrob.
I sat down and thought silently for 2h about what needs to happen in order for us to understand neural networks. Summary below:
We must understand structure in terms of the neural network itself, not by trying to project this structure onto things we can interpret. If it is interpretable, then the structure should tell us so—spectral gaps, isolated/sparse circuits, etc. A better understanding of NN structure doesn’t have to mean better interpretability—they’re correlated only to the extent that the actual structure is made up of interpretable units; if NNs are made up of circuits, we should see them, if not, we should not see them. NNs obviously have sufficient structure that we can understand them better—build a mental model robust enough to predict the median ICML ‘neural networks sure are weird arent they’ paper. Module criticality and ‘transformers as painters’ are my measuring-stick for this kind of mental model building.
We want meso- and macro-ontologies (e.g., circuits, personae, ‘capabilities’). Strong-opinion-weakly-held: if it were possible to construct these ontologies well without starting from the micro-scale, we would have succeeded already. We need robust handles for thinking about behaviour on the micro-scale—we will have to abstract away from bare-parameter/activation reasoning bit by bit. It is best if we can justify these ontic units as being somehow ‘isolated’ or ‘discrete’; the archetype I have in mind is the BBP spike transition—it is sufficiently robust that one can plausibly define signal deteciton/recovery as in reference to BBP, and this definition will serve well in some regime perturbed around BBP. If we are careful when we construct these ontic units, they will come labeled with a asymptotic regime of validity. This tells us where to look for the next level of abstraction. For example, if BBP gives us a definition of “detected, localized signal”, the breakdown of BBP in the many-signal regime tells us where to look; hopefully, this breakdown, construed as a phase transition, gives us another ontic unit to work with, and lets us proceed. (my pet theory for the last three days, perhaps this breakdown of localized signals is precisely the ‘delocalization’ of knowledge inherent in going from memorized patterns to a connected world model).
Noise and training dynamics seem fundamental, not incidental. I hypothesise that, at equal loss, an NN trained under full-batch gradient-flow will be essentially useless at extrapolating OOD (i.e., would suck at being finetuned to a downstream task—not merely generalizing to unseen data from the same distribution). Two perspectives clash here: one is data-essentialist, that an NN has finite capacity, and there’s a particular optimal way to represent that data given fixed capacity, and that way happens to be sparse/hierarchical/factored representations, perhaps on account of some sort of ‘simplicity’ prior—the ‘golden path hypothesis’ is in this genre. Second, there’s a structure-essentialist view, wherein the dynamics of training are the main thing which determine the nature of representations learned, with a secondary role for the data. I lean towards the structure-essentialist view—the neurons just want to learn man. My working mental model is much closer ‘neural Darwinism’, natural selection acting on competing species (competing ‘circuits’) - the abiotic environment (the data) matters, but the biotic environment is shaped ‘more’ by its own internal logic of competition and ecology. For example, perhaps a globally-shared ‘world-model’ is selected for not by compression, but by being so universally-used in computations that even mediocre performance would make any gradient more harmful than useful.
I also expect a notion of plasticity and critical-periods will be important—nature didn’t implement these in humans for fun; literature seems to point to early/middle/late layers as corresponding to “feature-extraction / generic computation / task-specific”, with the early and late layers deplastifying early in training (I’m unsure of this interpretation). It seems like some kind of plasticity control is important, and that we just mostly never realized this on account of NNs having figured out how to half-assedly implement this on their own - ‘divine benevolence’ and all that. Possible relation here to EGOP/AGOP/NFA, but I don’t know.
A huge percentage of brain machinery seems to be dedicated towards managing stacks of different regulatory loops, and it is probably the case that enhancing intelligence, cognition, etc. corresponds more to precisely tuning an interlocking system of regulatory loops than to having any difference in the basic machinery of neural function. I think there are convergent reasons to expect this to be true of NNs (even aside form the critical-period literature) - probably most of training is best thought of as ‘tuning’. This also makes model addition/Polyak averaging make a bit more sense—if is all changes in regulators and not in basic machinery, ‘averaging the regulator settings’ kinda feels like a thing that could work, in a way that ‘average these two ALU implementations’ doesn’t. I don’t think “everything is regulators” is more useful than “everything is circuits”—we need a middle-ground picture that I don’t have yet.
(Naive) weight-based analysis should be extremely hard, probably usually impossible. the natural units of analysis may instead be weights-in-context—pairs (contexts ~ ‘prompts’ or something). If one were to look at a single neuron of the brain averaged over a day, it would be hopeless—it is doing fundamentally different things during sleep than during an algebra class. This requires some principled way of saying what contexts are ‘naturally’ grouped together. My best guess at how to understand this right now is that the eNTK is a natural way to ask the network how similarly it will behave for the two—at the very least, we need some modified version of this if we want to analyze context-sensitive behaviour. Possibly something more like the parameter-space version of this: , where is some subset of the data-distribution which is a context in which one can expect neurons to be performing a ‘single function’. It’s not clear what semantic level this distinction is made at—if this should be more like ‘doing algebra vs doing calculus’ or more like ‘thinking the letter a’ vs ‘thinking the letter c’ - hence my trying to emphasise that we need some ‘natural’ way to carve out based on the network—and hence perhaps something like clustering based on . I think dynamic analyses of weights over training are more likely to yield useful information.
I’m less sure about this, but I think we should only rarely think in terms of ‘moment bias’ - I wrote down an EFT formalism where increasing data let one ‘resolve’ progressively higher moments of a data distribution—a la ‘sliding down the stairs’ - then I stared at this and realized there’s absolutely no way one could use this to ever find a ‘cat’ concept without making assumptions tantamount to solving the problem. Asserting that high level semantic concepts should live in higher moments of the data-distribution is vacuously true—iirc M’th order moments take exp(M) samples to estimate. Insofar as NNs seem to have some universality in the things they learn, I’m fairly sure one could not predict this universality from the data-distribution/geometry, short of training NNs on it—I think it comes down much more to “inductive bias”. Obviously, caveats to this for fine-tuning a trained neural network.
What specific next-steps are implied by this?
Things I need to learn/read more: NNMFT, plasticity and critical-periods in biology and ML, Liquid-state machine model of brain. Log-linearity and supposed approximate equivalence between fine-tunes/system-prompts/steering-vectors, etc.
Think more about: Quasispecies models as ways to operationalize ‘units of selection’; different sources of ‘noise’, when they are equivalent, and when should this have anything to do with ‘linearity’ in transformers? Relationship between evolved noise-robustness and plasticity?; BBP transitions and Saxe-dynamics.
I think the most promising things to think about are phase transitions of every shape and flavour, as these are the most actionable, and the things for which I can most readily evaluate validity. This may entail learning MFT. The ‘plasticity’ / ‘critical period’ idea features more heavily in my model than I had realized, and I should try to understand this better too. My heart says these must be deeply related to ‘structured noise’ (e.g., one module becoming robust to ‘noise’ from another == loss of plasticity?). Hand-wavily, ‘critical periods’ seem like a good thing to mine for my beloved phase-transitions.
Do you have (and does anyone have) a take on Martin and Mahoney 2018? It’s something I read in the pre-GPT dark ages, alleging a multitude of phases (distinguished by the distribution of eigenvalues) in deep learning networks. I have no idea if it has been subsequently validated or refuted or otherwise built upon.
I do indeed! I just wrote this post about a cluster of surrounding ideas. Tl;dr it’s interesting, and we sooorta see this kind of behaviour in modern ML models, but certainly not cleanly. While I’m not convinced that their story plays out cleanly irl, I’ve been thinking a lot over the past week about the spikes-->power-law transition as a toy theoretical model. My hope is that such a model would tell us something about a “phase-transition from memorization to generalization”. TBD.
These are notes from my research undertaken as part of the Iliad Fellowship under the mentorship of Dmitry Vaintrob.
I sat down and thought silently for 2h about what needs to happen in order for us to understand neural networks. Summary below:
We must understand structure in terms of the neural network itself, not by trying to project this structure onto things we can interpret. If it is interpretable, then the structure should tell us so—spectral gaps, isolated/sparse circuits, etc. A better understanding of NN structure doesn’t have to mean better interpretability—they’re correlated only to the extent that the actual structure is made up of interpretable units; if NNs are made up of circuits, we should see them, if not, we should not see them. NNs obviously have sufficient structure that we can understand them better—build a mental model robust enough to predict the median ICML ‘neural networks sure are weird arent they’ paper. Module criticality and ‘transformers as painters’ are my measuring-stick for this kind of mental model building.
We want meso- and macro-ontologies (e.g., circuits, personae, ‘capabilities’). Strong-opinion-weakly-held: if it were possible to construct these ontologies well without starting from the micro-scale, we would have succeeded already. We need robust handles for thinking about behaviour on the micro-scale—we will have to abstract away from bare-parameter/activation reasoning bit by bit. It is best if we can justify these ontic units as being somehow ‘isolated’ or ‘discrete’; the archetype I have in mind is the BBP spike transition—it is sufficiently robust that one can plausibly define signal deteciton/recovery as in reference to BBP, and this definition will serve well in some regime perturbed around BBP. If we are careful when we construct these ontic units, they will come labeled with a asymptotic regime of validity. This tells us where to look for the next level of abstraction. For example, if BBP gives us a definition of “detected, localized signal”, the breakdown of BBP in the many-signal regime tells us where to look; hopefully, this breakdown, construed as a phase transition, gives us another ontic unit to work with, and lets us proceed. (my pet theory for the last three days, perhaps this breakdown of localized signals is precisely the ‘delocalization’ of knowledge inherent in going from memorized patterns to a connected world model).
Noise and training dynamics seem fundamental, not incidental. I hypothesise that, at equal loss, an NN trained under full-batch gradient-flow will be essentially useless at extrapolating OOD (i.e., would suck at being finetuned to a downstream task—not merely generalizing to unseen data from the same distribution). Two perspectives clash here: one is data-essentialist, that an NN has finite capacity, and there’s a particular optimal way to represent that data given fixed capacity, and that way happens to be sparse/hierarchical/factored representations, perhaps on account of some sort of ‘simplicity’ prior—the ‘golden path hypothesis’ is in this genre. Second, there’s a structure-essentialist view, wherein the dynamics of training are the main thing which determine the nature of representations learned, with a secondary role for the data. I lean towards the structure-essentialist view—the neurons just want to learn man. My working mental model is much closer ‘neural Darwinism’, natural selection acting on competing species (competing ‘circuits’) - the abiotic environment (the data) matters, but the biotic environment is shaped ‘more’ by its own internal logic of competition and ecology. For example, perhaps a globally-shared ‘world-model’ is selected for not by compression, but by being so universally-used in computations that even mediocre performance would make any gradient more harmful than useful.
I also expect a notion of plasticity and critical-periods will be important—nature didn’t implement these in humans for fun; literature seems to point to early/middle/late layers as corresponding to “feature-extraction / generic computation / task-specific”, with the early and late layers deplastifying early in training (I’m unsure of this interpretation). It seems like some kind of plasticity control is important, and that we just mostly never realized this on account of NNs having figured out how to half-assedly implement this on their own - ‘divine benevolence’ and all that. Possible relation here to EGOP/AGOP/NFA, but I don’t know.
A huge percentage of brain machinery seems to be dedicated towards managing stacks of different regulatory loops, and it is probably the case that enhancing intelligence, cognition, etc. corresponds more to precisely tuning an interlocking system of regulatory loops than to having any difference in the basic machinery of neural function. I think there are convergent reasons to expect this to be true of NNs (even aside form the critical-period literature) - probably most of training is best thought of as ‘tuning’. This also makes model addition/Polyak averaging make a bit more sense—if
(Naive) weight-based analysis should be extremely hard, probably usually impossible. the natural units of analysis may instead be weights-in-context—pairs
I’m less sure about this, but I think we should only rarely think in terms of ‘moment bias’ - I wrote down an EFT formalism where increasing data let one ‘resolve’ progressively higher moments of a data distribution—a la ‘sliding down the stairs’ - then I stared at this and realized there’s absolutely no way one could use this to ever find a ‘cat’ concept without making assumptions tantamount to solving the problem. Asserting that high level semantic concepts should live in higher moments of the data-distribution is vacuously true—iirc M’th order moments take exp(M) samples to estimate. Insofar as NNs seem to have some universality in the things they learn, I’m fairly sure one could not predict this universality from the data-distribution/geometry, short of training NNs on it—I think it comes down much more to “inductive bias”. Obviously, caveats to this for fine-tuning a trained neural network.
What specific next-steps are implied by this?
Things I need to learn/read more: NNMFT, plasticity and critical-periods in biology and ML, Liquid-state machine model of brain. Log-linearity and supposed approximate equivalence between fine-tunes/system-prompts/steering-vectors, etc.
Think more about: Quasispecies models as ways to operationalize ‘units of selection’; different sources of ‘noise’, when they are equivalent, and when should this have anything to do with ‘linearity’ in transformers? Relationship between evolved noise-robustness and plasticity?; BBP transitions and Saxe-dynamics.
I think the most promising things to think about are phase transitions of every shape and flavour, as these are the most actionable, and the things for which I can most readily evaluate validity. This may entail learning MFT. The ‘plasticity’ / ‘critical period’ idea features more heavily in my model than I had realized, and I should try to understand this better too. My heart says these must be deeply related to ‘structured noise’ (e.g., one module becoming robust to ‘noise’ from another == loss of plasticity?). Hand-wavily, ‘critical periods’ seem like a good thing to mine for my beloved phase-transitions.
Do you have (and does anyone have) a take on Martin and Mahoney 2018? It’s something I read in the pre-GPT dark ages, alleging a multitude of phases (distinguished by the distribution of eigenvalues) in deep learning networks. I have no idea if it has been subsequently validated or refuted or otherwise built upon.
I do indeed! I just wrote this post about a cluster of surrounding ideas. Tl;dr it’s interesting, and we sooorta see this kind of behaviour in modern ML models, but certainly not cleanly. While I’m not convinced that their story plays out cleanly irl, I’ve been thinking a lot over the past week about the spikes-->power-law transition as a toy theoretical model. My hope is that such a model would tell us something about a “phase-transition from memorization to generalization”. TBD.