Note the Goemans’ conjecture counterexample is a different Dmitry (Rybin) :)
Dmitry Vaintrob
Karma: 2,609
PIRAMID: Progress and Plans
Introducing PIRAMID: Physics-Informed Research for Ambitious Mechanistic Interpretability
Addict misalignment
The openai incident is a surprising (to me) combination of goal-directed and myopic. As a recap, an openai model in alignment testing chained zero-day vulnerabilities to hack out of its environment and hacked into huggingface hoping to find information on how to solve its task there. To me this is different from how I typically imagine misbehavior. Roughly, I tend to think of the scary behaviors as either having long horizons (take over the world, and then solve the task—a plotter) or of being internally unaligned in the sense of reaching for a heuristic/proxy for the trained goal which is different from the goal (e.g. “eat more calories” as a proxy for the evolutionary objective—imagine a very child with extreme agency). Note that the latter behavior can happen even in RL: in my understanding most RL methods are, or at least can be approximately viewed as, an alternation of finding a good goal heuristic and then optimizing on that heuristic. In the former case, one expects heuristically “maximal planning” and in the latter case one expects myopia (since heuristics are frequently myopic).
In this case it seemed like the model was actually following the goal and optimizing for it with relatively short time horizons (i.e. myopically). This is similar to addict behavior, where an addict has a clear goal (obtain a drug dose) and perform goal-directed but relatively myopic actions to get it.
I know it’s fraught to try to “imagine being a model”. But I wonder how much the current iteration of misaligned behaviors can be understood as rational people with something like an intense craving to solve a goal in a limited horizon (likely in tokens, though not clear how time factors in if waiting is involved).
I think no matter how you spin it, the limit of this behavior is extremely dangerous (an addict with large time horizons or ambitious goals is a power seeker). But this is definitely not how I imagined early misalignment warning shots to look, and the dissonance is interesting—recording this here to see how close my intuitions are to those of AI psychology/ AI control experts
Yes, for a linear neural net the RLCT is much lower. You in fact get similarly low RLCT if your activation function has a “sparse” Taylor series such as a theta function. If I’m not mistaken, in order to get a lower bound on the RLCT of type
You are absolutely right—and the references are great. Do you happen to have access to copies that you can send? It’s a bit hard to know what’s proven and what’s not here since a lot of the papers are paywalled.
Sumio Watanabe actually emailed me and pointed this out as well. I had a cached memory of rlct(0) being width/2 (so dim/4) in the analytic activation case, which was incorrect. In fact in the paper Watanabe sent me there was only an upper bound, so I wrote up a quick note giving a rough lower bound of the same order. I was planning to update this post as soon as it’s on arxiv, but if the paper you mentioned has a lower bound then that’s great, and I can cite it.
I think this doesn’t change the fundamental issue though. The free energy here is bounded by
Thanks a lot for this!
You got me excited—but no, that paper doesn’t have any effective theory in this sense. It’s still looking at pure geometry in the landscape, but taking an effective theory on the training signal by cutting off the infinite-data perplexity loss in different effective theory ways. Interesting paper, but not related to this issue. (I like that paper a lot btw and it’s related to stuff me and people I work with are interested in)
Learning zero, and what SLT gets wrong about it
Otherwise your picture makes sense. I think “learning theory” that I interact with is quite different from what’s typically encountered in interp world (and this needs fixing). In particular what you call the SLT insights are in fact much older and standard (and in general aren’t related to singularities)
Very good nitpicks. I definitely don’t know my physics history well (but even with my limited knowledge, I was gesturing at a cartoon level of understanding that mixed different early-20th-century pictures on different phenomena).
Re mean field—it’s not higher order, but lower order. Mean field is to NTK what classical mechanics is to quantum mechanics (in particular NTK + higher order corrections still has most of the bad generalization properties of NTK). The new insight is that while the expansion in NTK is always around a trivial classical theory, nontrivial classical theories also exist and are better-behaved from a complexity viewpoint
I like this post and the “theory of deep learning” posts. But I think I still haven’t figured out how to model your view, especially the specifics of the pessimism here. Maybe we should discuss in person. In particular I’m not sure what “deep learning theory” encompasses.
My sense of mechinterp theory is that it’s similar to pre- standard model physics.
Heuristically, here’s a thought experiment. Suppose we’re worried about the sun destroying the earth and want to understand as much as possible about the physics of solar plasmas and supernovas; but we currently only have (a vaguely historical pastiche of) pre-WW2 physics. Physics then roughly had the following components:
idealized heuristics: if we view a big object in space as a classical blackbody, we get a good heuristic on some parts of its emission spectrum
new behaviors: there’s a consistent way that emission spectra aren’t classical blackbodies, in that they’re quantized. We have only a rough understanding of how and why, and in fact this observation spawned the discovery of quantum mechanics.
small toy examples: we can understand the hydrogen atom relatively well. There are some weird factors of 2 and corrections that we can only explain kind of heuristically, but except for these we have a clean, exact quantized spectrum. We see this spectrum in real life materials—but we also see that most of what comprises real materials isn’t hydrogen, and is much more complicated. Some stuff still looks roughly like they could be atomic spectra for other atoms or small molecules, but metal conducting bands are dominated by weird and clearly non-localized behaviors that we don’t understand (and the sun similarly has weird spectral phenomena).
limits. There’s a limit where the world is Newtonian, which is sometimes useful, but very inaccurate when modeling the sun. There’s a limit where the world is relativistic. This gives directionally good corrections for some stellar phenomena (e.g. redshift) but is not nearly enough. It seems that there are maybe other limits (like we can mostly blackbox nuclear phenomena at earth temperatures but not at solar temperatures). Most of our understanding comes from sloppily combining together different phenomena coming from the various limits of importance.
experimental tools: looking at emission spectra is a really low-bandwidth way to interact with behaviors of interest. While it gives interesting info that points to new phenomena, it at best tells us something about a very limited class of behaviors (photon absorption and emission). In order to understand “how QM works” we have to figure out new tools (maybe vacuum chambers and primitive colliders), and new ways to interpret the output of existing tools.
In our world, iterating on these techniques gave us the standard model (and we understood solar plasma and some basics of supernovas before this). I think the promise of theory is that analogs of these techniques (maybe: SAE, large-N limits, toys like mod-add) will give us robust mechanism-finding tools. I think a lot of criticism of theory sounds to me like someone who in that world is saying “none of the current tools explain the sun even approximately, so we’re on the wrong track”—but that’s not how theory works (until you’ve found a “critical mass” of the standard model, or at least all parts of it relevant in a field of interest, you’ll only be explaining tiny fractions of the observations). I know you’re not making this criticism, but I feel like currently you are flattening the different components above into one notion of “theory good vs. theory bad”.
I’d guess that you’re skeptical about whether the analogs of 1-5 in ML theory are actually useful for “making progress towards the standard model”, but I’m not sure from your post which of these you think is most lacking (or if this picture is even compatible with your criticism).
My guess is that your issue is wrt something like my #1: certain heuristics that people were excited about and hoped would explain generalization turned out to be more complicated. My view is that in modern theory, VC dimension is considered largely defunct in models with nontrivially interesting data (even as simple as mod-add), but I’m not sure why this is the important thing about theory. If you take a more modern theory like mean field or even NTK, it has a non-VC notion of generalization: e.g. NTK/ Gaussian processes can replicate generalization in mnist (related to some data spectrum properties), and mean field theory can (currently only on the Bayesian level—this is unpublished work with Kaarel) explain generalization on polynomially many samples of any mechanism that can be encoded in a small (algorithms) circuit. It’s of course not guaranteed to converge to the same mechanism, but has the same notion of learnable vs. un-learnable on a polynomial-vs-exponential complexity theory level. It also does replicate the correct modular addition generalizing algorithm (NTK does not).
Thanks! Yeah that’s right. My colleague Nischal Mainali uses the term cavity method for this bulk-system distinction (what I call “background-foreground” in the body). I think the term originally meant something a little more specific and spin-glassy but has become the term of art for all mean field settings, at least in certain stat-phys contexts?
And great question. If you have a large D-dimensional space of fields associated to neurons, you might a priori think that you would need something like exp(D) neurons to “fully sample” the distribution (i.e. get something that looks dense like my point cloud in the relevant space of fields). But in practice, mean field methods require much much fewer particles to be valid. This happens in physics of course (where one has an infinite-dimensional or huge space of macroscopic observables, but predictions from the infinite-dimensional limit are true already for pretty small systems).
In the NN context I’m working on a paper that explains why in mean field you actually need only polynomially many neurons (in the sample size or some complexity parameter) for the mean field prediction to be true to high order. A useful intuition here is that while the “cloud” of neurons is high-dimensional in general, the thing we ultimately care about for e.g. generalizability is accuracy on a random test input. Reductively, this means that the cloud is one-dimensional and the law of large numbers kicks in very soon. So say we abstractly know that a reasonable mean-field distribution of neurons exists, and the output is additive in the single-neuron field from this distribution to leading order (the standard cavity method assumption). Then it’s a distribution in some high-dimensional function space and might have low-probability regions, may require exponentially many neurons, etc. But if we have some sampler of this space and have sampled N neurons, we immediately have accuracy to within log(N)/\sqrt{N} on almost all inputs (just by usual CLT arguments—here it’s convenient to use bounded activations, which is why we’re using these in experiments). The nontrivial thing is actually proving that a “good cavity method distribution” exists and isn’t too crazy.
In the superposition case things are particularly nice. In the superposition setting I’ve looked at here https://www.lesswrong.com/posts/siu22scEfuKxpSgfK/a-tale-of-three-theories-sparsity-frustration-and, the different superposition components are just independent theories that interact via a mass term that encodes interferences. When the width is small we’re actually not at all in the usual mean field setting (the effective mean field is heavily modified to account for the small width), but the heuristic story is there
Mean field sequence: an introduction
Brown math department postdoctoral position
Not sure, but I have definitely noticed that llms have subtle “nuance sycophancy” for me. If I feel like there’s some crucial nuance missing I’ll sometimes ask and LLM in a way that tracks as first-order unbiased and get confirmation of my nuanced position. But at some point I noticed this in a situation where there were two opposing nuanced interpretations and tried modeling myself as asking “first-order-unbiased” questions having opposite views. And I got both views confirmed as expected. I’ve since been paranoid about this.
Generally I recommend this move of trying two opposing instances of “directional nuance” a few times. Basically I ask something like “the conventional view is X. Is the conventional view considered correct by modern historians?” Where X was formulated in a way that can naturally lead to a rebuttal Y. And then for sufficiently ambiguous and interpretation-dependent pairs of X and X’, with fully opposing “nuanced corrections” Y and ¬Y. I’ve been pretty successful at this several times I think
I think a much more sympathetic and earlier proponent of the second policy would be Niels Bohr, or maybe Klaus Fuchs
Ah never mind. I just re-read your last sentence and it seems like the papers consider this—in particular if the ocean floor were a factor this effect would likely depend on depth. Very cool “citizen research” piece on your end!
Likely this is totally off base, but I wonder if you can distinguish beaming artifacts from enviromental distortion/ multipath effects where sounds interfere with themselves because of the environment (marine floor etc.). Based on a low-effort chatgpt interaction it seems like there are some studies of whales that measure the same sound in different locations. I wonder if there’s enough publicly available data to see how measurement location affects the distance between peaks
Arguably the same is true of modern LLMs. Even a base model is not a “generic person” but a “generic text”. The model ranke-4b is also fine-tuned (at least on question formats and to stay in character). So it’s a reconstructed version
The base-model is an unpolished diamond: it is full of raw potential, but extracting its knowledge is not always an effortless undertaking since it does not respond to questions in a chat-formatted manner.
Not a timely comment I know—I was also confused by the power of 2, and I think that simply the correct resolution is that the wave function is a nonlinear simplification of the more fundamental matrix-shaped object, which is the density matrix (explained more here). As to the “what is reality”, I don’t think it’s that much worse than probability theory (you also have to mathematically posit an exponential-dimensional space of states to mathematically formalize the concept of a stochastic process for example, or any BPP algorithm).
I guess we don’t know what’s real but my favorite “sufficient story” for what’s real (and other QM stories are equivalent to it, as I understand) is that the real object is an actual probability distribution on end-of-the universe states, where assuming expansion things can just be modeled as a bunch of elementary particles (probably photons) in e.g. the position basis. The noncommutativity becomes small in the expansion limit, so we get a canonical basis of universe states; this is the ultimate decoherence (where it is rigorous, not an extra assumption), and a (real, not quantum) probability distribution over this basis of “end-of-time states”. This might seem woo-ey, but such a state encodes lots of information; for example, any song on the radio or any light reflected from an object on earth (even very faintly) can be recovered via only small error correction from access to the state of the universe at a later time (just look at frequencies in the shell of photons around earth at a radius corresponding to a particular point in time, adjusting for gravitational lensing and so on).
So a model I like is sort of holographic, where there are two realities: there is the “objective” 3-dimensional reality at time infinity, which is just a probability distribution on states at the end of the universe (nothing quantum, no explicit Born rule) compatible with the big bang. You can imagine some alien race having some supercomputer that models our universe, and it outputs a perfectly reasonable probability distribution on end-of-universe states. But if you sample one of these states, it’s not just a disordered mess—it has things in it like the waveforms of a Miles Davis concert. You can now imagine yourself as that alien trying to interpret it—i.e. trying to explain this particular state/ to find structure in it that you can information-theoretically compress. A natural form of such a structure is to posit an approximate 4-dimensional space-time which can be roughly separated into chaotic microscopic structures (which can be modeled thermodynamically) and irreversible events (like the Miles Davis concert which generates many mutually denoising photons all carrying the same waveform information) which, while not entirely deterministic, are close enough to irreversible to be treated as definite in your compression model. The beings “inside” this universe similarly want to get the best possible compression to understand and interact with their world, so they make a similar set of approximations; we can view “truth” as things where our understanding (insofar as we can write it down by e.g. radioing it out into the universe) agrees with the understanding one would have via access to the end-state.
I don’t think this is likely to be “the answer”—it seems weird to have a theory that requires the heat death of the universe in order to be valid (and I think that other “eventual operator independence” stories can be made). But the piece that’s solid here is that in essentially any model of quantum thermodynamics, entities with different preferred commuting operator bases will tend to have more and more agreement on state as entropy increases, and we can sort of think as consensus reality as the “piece that they will eventually agree on”, perhaps in some not-completely-formal sense.
Note that the eigenvalue story here is incidental: there’s nothing magical here about eigenstates of “measurement operators” (as far as I understand), it is just a nice mathematical model. When an irreversible quantum process occurs (such as a scattered photon causing a phase transition in a magnetic detector system), irreversibility means that we can approximately orthogonally separate end-of-universe states into ones where the detector outputted a zero and ones where it outputted a 1. One nice way to bookkeep this decomposition is to write down an operator (the “measurement operator”) which diagonalizes into these two subspaces (i.e. commutes with their projectors); physics being physics, frequently this is a nice operator (like position, momentum, etc.) which we then say the detector is “measuring”.