Technical Director at Principles of Intelligence (PrincInt, PIBBSS) keen on narrowing the theory-practice gap in AI safety through research, field building, and metascientific reflections.
Formerly: Theoretical high energy physicist; science, technology, and society research; professor and academic advisor to interdisciplinary undergrads
Fixed, thanks!
There are ~two broad ways of thinking about RG. ‘HEP-like’ is structural; it helps roughly map out the space of theories at different scales/couplings, even if it does not capture all of physics at all times (for example, the standard model is a good description up to some energy scale, but each sector is often treated independently). Different aspects of training and inference seems to take the shape of renormalization, although as we pointed out there is a lot of work to be done to understand the various scales and couplings inherent to NNs. A goal of this opportunity space is not to make RG ‘go backwards’, but to correctly map this renormalization picture onto a ‘space of representations’ of a NN theory. I don’t expect this to be easy or simple, but am hopeful that it will shed more theoretical insight onto different training and inference regimes and point to insightful conceptual gaps.
In contrast, interpreting NNs to a certain degree of (human-specified) abstraction is more CMT-like. I suspect you are focusing on this perspective overall. Again, as we pointed out, this formulation of RG is not invertible, there are likely many ways to coarse grain a representation (just as there are many RG schemes in condensed matter), but that doesn’t mean you can’t reframe something like a sparse autoencoder as doing RG.
Sure, there are ‘lots of phase transitions’, but that doesn’t mean that it’s pointless to try and classify or describe some interesting one (like memorization → generalization). Similarly, just because lots of interesting physics happens far from phase transitions doesn’t mean they aren’t useful ‘tethers’ in the space of theories (phi-4 theory has been pretty useful).
Regarding attention, I agree that a theory of representations here would be non-local with respect to inputs. That’s fine. Long range dependencies just change the flavor of RG. A lot depends on how we measure locality, since there’s no a priori natural way to do this in NNs (although there are several options that likely pick up on different relationships between features).
I think @Dmitry Vaintrob has more context on this, but I’d be really interested in exploring this idea more. I’m not sure we’d be able to enumerate all possible ways to renormalize an interpretation, but I agree that having a portfolio RG-inspired lenses could help. Are you imagining that each agent would run the lenses over information from other agents and aggregate them in some way (how?)?
Thanks for the comment! The way I think about it, there are several ways of thinking about RG in terms of different ‘energy’ analogues in NNs, and each is likely tied to a different framing in terms of ‘UV’ and ‘IR’.
For example, during training, you start with a simplified (IR like) description of the dataset that flows to a richer representation, adding finer grained structure capable of generalizing (UV).
During inference, I agree that you can describe this process as UV → IR, as each layer is a progressively coarser representation as the features that are irrelevant for a certain task (like classification) are ‘integrated out’ to yield a usefully abstract simplification. However, you can also think of inference in terms of ‘feature refinement’, where each layer becomes progressively more structured, able to pick up on finer or more abstract details. This ultimately depends on how you think of ‘scale’ along the RG flow.
Thanks for the recommendation! The pathways of scientific progress here seem very interesting (for example: physics → neuro → AI → … v. physics → AI → neuro → …), particularly if we think about feeding back between experimental and theoretical support to build up a general understanding. Physics is really good at fitting theories together in a mosaic—at a large scale you have a nice picture of the universe, and the tiles (theories) fit together but aren’t part of the same continuous picture, allowing for some separation between different regimes of validity. It’s not a perfect analogy, but it says something about physics’ ability to split the difference between reductionism and emergence. It would be nice to have a similar picture in neuroscience (and AI), though this might be more difficult.
For me, this question of the relevant scale(s) is the main point of introducing this work. d/w is one example of a cutoff, and one associated with the data distribution is another, but more work needs to be done to understand how to relate potentially different theoretical descriptions (for example, how these two cutoffs work together). We also mention the ‘lattice as regulator’ as a natural cut-off for physical systems, and hope to find similarly natural scales in real-world AI systems.
Thanks for the comment! I do hope that the thoughts expressed here can inspire some action, but I’m not sure I understand your questions. Do you mean ‘centralized’, or are you thinking about the conditions necessary for many small scale trading zones?
In this way, I guess the emergence of big science could be seen as a phase transition from decentralization → centralization.
Thanks for your comment! It got me thinking a bit more about big science in general, and led to this post. I’d be curious to hear your thoughts.
Thanks for this post. I think the problem it names is a real bottleneck for field building and grant making in AI safety rather than only an epistemics puzzle (which, as I read them, was the focus of many comments).
First, the question: “Is this group an opaque meritocracy or a cabal?” often isn’t well-posed globally. ‘Consensus’ underlying meritocratic transparency is tied to a specific scientific culture and depends on its granularity. For example, the goals, standards of evidence, and scientific values of ‘physics’ as a whole are not the same as for ‘high-energy physics’ as a whole or ‘quantum gravity’. I like string theory as a reference for AI safety because it did involve the explicit collaboration between physicists and mathematicians that led to the formulation of a combined scientific culture (in string theory). Clearly, this led to concerns over the health and direction of physics as a field, which is echoed in the many opinions for what counts as pursuit-worthy new directions in AI safety. Also relevant to AI safety is that the scientific cultures that populate a field tend to get more granular over time as we learn more, and that we have a unique (if less ‘natural’, in an academic sense) field building opportunity to decide how to incorporate new ideas. I hope we can do this in a way that does not result in a collection of mutually opaque meritocracies. Personally, I think an approach that is pluralist about research programs and strict about the artifacts that should come out of those programs strikes a good balance, though defining this merit criteria per subculture in a way that aggregates to a comprehensive ‘good’ collective is hard.
This leads to a second point: this criteria changes over time. In a rough historical sketch, the String Wars came after a first revolution (mid 1980s work fixing anomalies, resulting in a landscape of superstring theories as QG candidates), a period of criticism (for example, in 1986: desperately seeking superstrings) and revolution again (1990s work on dualities and introducing branes, AdS/CFT). As Dmitry pointed out, this had implications on the field’s scientific culture, including its goals and normative standards. I haven’t thought this analogy through, but this may have a parallel in AI safety, with ‘string theory as quantum gravity’ standing in epistemically for agent foundations, and the empirical contact of AdS/CFT with real-world complex systems playing the role of mechanistic interpretability.
Lastly, the string wars example is as much about resource scarcity/allocation as criteria for ‘good science’. A less granular example is the Anderson-Weinberg fight over the SSC, where each subfield was unable to adjudicate the other’s merit while competing for the same funding. It’s also worth noting that resource competition is not only where consensus gets expressed, but also how it solidifies within a scientific culture. This is perhaps another lesson for AI safety. Our scarce resource may not be less about money than grantmaker attention and limits to individual expertise (so that ‘opaque’ defaults to ‘unfundable’). I agree with Jonas’ comment below.