Broadly agreed. I’d written a similar analysis of the issue before, where I also take into account path dynamics (i. e., how and why we actually get to Azazel from a random initialization). But that post is a bit outdated.
The central issue, the reason why “naive” approaches for just training a ML model to make good prediction will likely result in a mesa-optimizer, is that all such setups are “outer-misaligned” by default. They don’t optimize AIs towards being good world-models, they optimize them for making specific good predictions and then channeling these predictions through a low-bandwidth communication channel. (Answering a question, predicting the second part of a video/text, etc.)
That is, they don’t just simulate a world: they simulate a world, then locate some specific data they need to extract from the simulation, and translate them into a format understandable to humans.
As simulation complexity grows, it seems likely that these last steps would require powerful general intelligence/GPS as well. And at that point, it’s entirely unclear what mesa-objectives/values/shards it would develop. (Seems like it almost fully depends on the structure of the goal-space. And imagine if e. g. “translate the output into humanese” and “convince humans of the output” are very nearby, and then the model starts superintelligently optimizing for the latter?)
In addition, we can’t just train simulators “not to be optimizers”, in terms of locating optimization processes/agents within them and penalizing such structures. It’s plausible that advanced world-modeling is impossible without general-purpose search, and it would certainly be necessary inasmuch as the world-model would need to model humans.
As simulation complexity grows, it seems likely that these last steps would require powerful general intelligence/GPS as well. And at that point, it’s entirely unclear what mesa-objectives/values/shards it would develop.
On one hand, I fully agree that a strong predictor is going develop some very strong internal modeling that could reasonably be considered superhuman in some ways even now.
But I think there’s an unstated background assumption sneaking into most discussions about mesaoptimizers- that goal oriented agency (even with merely shard-like motivations) is a natural attractor for SGD, particularly in the context of outwardly goal agnostic simulators.
This could be true, and it would be extremely important if it were true, and I really want more people trying to figure out if it is true, but so far as I’m aware, we don’t have strong evidence that it is.
My personal guess, given what I know now, is that some form of weakly defined mesaoptimization is an attractor (>90%), but agentic mesaoptimizers in the context of non-fine-tuned GPT-like architectures are not (75%).
I think agentic mesaoptimization can be an attractor in some architectures. I’m comfortable claiming humans in the context of evolution as a close-enough existence proof of this. I think the conditions of our optimization made agentic mesaoptimization natural, but I suspect optimization processes with wildly different conditions will behave differently.
This is a big part of why I’m as optimistic as I am about goal agnostic simulation as a toehold for safety- I think we actually do replace one set of problems with an easier set of problems, rather than just adding more.
I don’t see why projecting logits from the residual stream should require anything like search. In fact, the logit lens seems like strong evidence against this being the case, since it shows that intermediate hidden representations are just one linear transformation away from making predictions about the vocab distribution.
It’s not like SGD is sampling random programs, conditioning only on those programs achieving low loss.
I feel like thinking of the internals of transformers as doing general search—especially search over things to simulate—is some kind of fallacy. The system as a whole (the transformer) outputs a simulation of the training distribution, but that doesn’t mean it’s made of parts that themselves do simulations, or that refer to “simulating a thing” as a basic part of some internal ontology.
I think “classic” inner alignment failure (where some inner Azazel has preferences about the real world) is a procrustean bed—it fits an RL agent navigating the real world, but not so much a pure language model.
I mean, that just pushes the problem back by one step. If we take LLMs to be simulators, they’d necessarily need to have some function that maps the simulation-state to the probability over the output tokens (since, after all, the ground truth of reality they’re simulating isn’t probability distributions over tokens).
And if LLMs work by gradually refining the probability distribution over the output which they keep in the residual stream, that would just imply that the “simulation-state ⇒ output distribution” functions are upstream of the residual stream — i. e., every intermediate layer both runs a simulation-step and figures out how to map that simulation’s new state into a distribution-over-outputs.
Of course, it seems architecturally impossible for modern LLMs to run a general-purpose search at that step, but in my view it’s an argument against modern LLM architectures being AGI-complete, not against search being unnecessary.
If we take LLMs to be simulators, they’d necessarily need to have some function that maps the simulation-state to the probability over the output tokens
I disagree with this picture. “Simulators” just describes the external behavior of the model, and doesn’t imply LLMs internally function anything like the programs humans write when we want to simulate something, or like our intuitive notions of what a simulator ought to do.
I think it’s better to start with what we’ve found of deep network internal structures, which seem to be exponentially large ensembles of fairly shallow paths, and then think about what sort of computational structures would be consistent with that information while also 1) achieving low loss, and 2) being plausibly findable by SGD from a random init.
My tentative guess is that LLMs internally look like a fuzzy key-value lookup table over a vast quantity of (mostly shallow) patterns about text content. They do some sort of similarity matching between the input texts and the features that different stored patterns “expect” in any text to which the pattern applies. Any patterns which trigger then quickly add their predictions into the residual stream, similar to what’s described here.
In such a structure, having any significant translation step between the internal states of the predictive patterns and the output logits would be a huge issue, because you’d have to replicate that translation across the network many times, not just once per layer, but many times per layer, because single layers are implementing many ~independent paths simultaneously.
I do agree that LLM architectures seem poorly suited to learning the sorts of algorithms I think people imagine when they say stuff like “general purpose search”. However, I take that as an update against those sorts of algorithms being important for powerful cognition, essentially considering that transformers have been the SOTA architecture for over 5 years while remaining essentially unchanged, despite many, many people trying to improve on them.
Fair enough, I don’t disagree that it’s how current LLMs likely work.
I maintain, however, that it makes me very skeptical that their architecture is AGI-complete. In particular, I expect it’s incapable of supporting the sort of high-fidelity simulations that people often talk about in the context of e. g. accelerating alignment research. And that, on the contrary, the architectures that are powerful enough would be different enough to support search and therefore carry the dangers of inner misalignment.
I can sort of see the alternate picture, though, where the shallow patterns they implement include some sort of general-enough planning heuristics that’d theoretically let them make genuinely novel inferences over enough steps. I think that’d run into severe inefficiencies… but my intuition on that is a bit difficult to unpack.
Hm. Do you think the current LLM architectures are AGI-complete, if you scale them enough? If yes, how do you imagine they’d be carrying out novel inferences, mechanically? Inferences that require making use of novel abstractions?
Broadly agreed. I’d written a similar analysis of the issue before, where I also take into account path dynamics (i. e., how and why we actually get to Azazel from a random initialization). But that post is a bit outdated.
My current best argument for it goes as follows:
On one hand, I fully agree that a strong predictor is going develop some very strong internal modeling that could reasonably be considered superhuman in some ways even now.
But I think there’s an unstated background assumption sneaking into most discussions about mesaoptimizers- that goal oriented agency (even with merely shard-like motivations) is a natural attractor for SGD, particularly in the context of outwardly goal agnostic simulators.
This could be true, and it would be extremely important if it were true, and I really want more people trying to figure out if it is true, but so far as I’m aware, we don’t have strong evidence that it is.
My personal guess, given what I know now, is that some form of weakly defined mesaoptimization is an attractor (>90%), but agentic mesaoptimizers in the context of non-fine-tuned GPT-like architectures are not (75%).
I think agentic mesaoptimization can be an attractor in some architectures. I’m comfortable claiming humans in the context of evolution as a close-enough existence proof of this. I think the conditions of our optimization made agentic mesaoptimization natural, but I suspect optimization processes with wildly different conditions will behave differently.
This is a big part of why I’m as optimistic as I am about goal agnostic simulation as a toehold for safety- I think we actually do replace one set of problems with an easier set of problems, rather than just adding more.
I don’t see why projecting logits from the residual stream should require anything like search. In fact, the logit lens seems like strong evidence against this being the case, since it shows that intermediate hidden representations are just one linear transformation away from making predictions about the vocab distribution.
It’s not like SGD is sampling random programs, conditioning only on those programs achieving low loss.
Yeah, I’ll pile on in agreement.
I feel like thinking of the internals of transformers as doing general search—especially search over things to simulate—is some kind of fallacy. The system as a whole (the transformer) outputs a simulation of the training distribution, but that doesn’t mean it’s made of parts that themselves do simulations, or that refer to “simulating a thing” as a basic part of some internal ontology.
I think “classic” inner alignment failure (where some inner Azazel has preferences about the real world) is a procrustean bed—it fits an RL agent navigating the real world, but not so much a pure language model.
I mean, that just pushes the problem back by one step. If we take LLMs to be simulators, they’d necessarily need to have some function that maps the simulation-state to the probability over the output tokens (since, after all, the ground truth of reality they’re simulating isn’t probability distributions over tokens).
And if LLMs work by gradually refining the probability distribution over the output which they keep in the residual stream, that would just imply that the “simulation-state ⇒ output distribution” functions are upstream of the residual stream — i. e., every intermediate layer both runs a simulation-step and figures out how to map that simulation’s new state into a distribution-over-outputs.
Of course, it seems architecturally impossible for modern LLMs to run a general-purpose search at that step, but in my view it’s an argument against modern LLM architectures being AGI-complete, not against search being unnecessary.
I disagree with this picture. “Simulators” just describes the external behavior of the model, and doesn’t imply LLMs internally function anything like the programs humans write when we want to simulate something, or like our intuitive notions of what a simulator ought to do.
I think it’s better to start with what we’ve found of deep network internal structures, which seem to be exponentially large ensembles of fairly shallow paths, and then think about what sort of computational structures would be consistent with that information while also 1) achieving low loss, and 2) being plausibly findable by SGD from a random init.
My tentative guess is that LLMs internally look like a fuzzy key-value lookup table over a vast quantity of (mostly shallow) patterns about text content. They do some sort of similarity matching between the input texts and the features that different stored patterns “expect” in any text to which the pattern applies. Any patterns which trigger then quickly add their predictions into the residual stream, similar to what’s described here.
In such a structure, having any significant translation step between the internal states of the predictive patterns and the output logits would be a huge issue, because you’d have to replicate that translation across the network many times, not just once per layer, but many times per layer, because single layers are implementing many ~independent paths simultaneously.
I do agree that LLM architectures seem poorly suited to learning the sorts of algorithms I think people imagine when they say stuff like “general purpose search”. However, I take that as an update against those sorts of algorithms being important for powerful cognition, essentially considering that transformers have been the SOTA architecture for over 5 years while remaining essentially unchanged, despite many, many people trying to improve on them.
Fair enough, I don’t disagree that it’s how current LLMs likely work.
I maintain, however, that it makes me very skeptical that their architecture is AGI-complete. In particular, I expect it’s incapable of supporting the sort of high-fidelity simulations that people often talk about in the context of e. g. accelerating alignment research. And that, on the contrary, the architectures that are powerful enough would be different enough to support search and therefore carry the dangers of inner misalignment.
I can sort of see the alternate picture, though, where the shallow patterns they implement include some sort of general-enough planning heuristics that’d theoretically let them make genuinely novel inferences over enough steps. I think that’d run into severe inefficiencies… but my intuition on that is a bit difficult to unpack.
Hm. Do you think the current LLM architectures are AGI-complete, if you scale them enough? If yes, how do you imagine they’d be carrying out novel inferences, mechanically? Inferences that require making use of novel abstractions?