AI notkilleveryoneism researcher, focused on interpretability.
Personal account, opinions are my own.
I have signed no contracts or agreements whose existence I cannot mention.
Lucius Bushnaq
Karma: 5,711
I haven’t encountered this result, but it makes intuitive sense to me that something of this form could work to define the prior too.
Nice. Is there a proof for that written up somewhere public?
How is ‘uniform prior over programs of infinite length’ defined if not via a limit in length?
How is that statement different from the statement I made?
In this formulation, you only consider programs of exactly length
One could still object that we are privileging length by taking any explicit limit in length at all. But I dunno, this seems pretty practically motivated to me.
EDIT: Sol says yes,Sol:
Yes — it converges to the same Solomonoff semimeasure. In the book’s notation, let
Since
ThusThe relevant source is pp. 159–161 of the authors’ PDF: https://www.hutter1.net/publ/uaibook2.pdf. One caveat: this uses exactly the book’s model, where the candidates are all binary strings and
Speculatively, we might even recover some version of a simplicity prior from it, since simple worlds might reoccur more frequently across all possible computations, i.e. influence the infinite sum more.
You can get the Solomonoff simplicity prior just by taking a uniform prior over programs of length L on a plain UTM and letting L tend to infinity. See result 3.8.1 in Hutter’s An Introduction to Universal Artificial Intelligence.
(I don’t think you need hyperreal numbers to prove this result.)
Some reasons alignment doesn’t generalise well
EDIT: Now slightly cleaned up as a top level post here.
I think there probably is a “low-sample-complexity / good generalization” sauce but by default it only applies to capabilities, not alignment. Alignment generalisation problems aren’t really about needing too much data to learn or having too weak a simplicity bias. I think by default, capabilities generalise further than alignment because:General capabilities generally make the loss go down, alignment doesn’t
If you are training a large and formidable AI, your training environment is basically never the place you think it is. Reality is too full of detail for that. There’s contamination in your labels, there’s training dynamics you didn’t think about, there’s strategies your RL agent can use that you never considered, and there are bugs. As a result, the inner objective an ML engineer might imagine would score the lowest loss when they set up their training environment will probably not, in fact, be the inner objective that actually does so.
For example, an inner objective shaped around human-like empathy might turn out to make the AI spend an average 0.03% inference steps extra on worrying about whether the human overseers think it is a virtuous member of the tribe while it’s supposed to be solving math problems. That inner objective then loses out to some weird, different objective that’s slightly more compatible with being utterly focused while crunching through ten million calculus problems in a row without any other kind of sensory input. For a different example, your RLHF data may reward agreeableness more than sincerity. More generally, “The simplest algorithm that fits the training data” will contain a pretty good description of the world, because the world is in a sense simple. “The simplest algorithm that fits RLHF/constitutional AI/whatever training” will probably not be an algorithm that wants the nice things the training data talks about because that algorithm doesn’t actually score the lowest loss. An algorithm that truly wants what the constitution talks about in the way the humans who wrote it meant won’t take every opportunity to score lower loss that’s available, and so will by default be outcompeted by different algorithms in the loss landscape that take more of these opportunities.
This problem gets worse as AI training become more dominated by long-form RL environments with a lot of freedom for the AIs to do unexpected stuff, and as the AIs become more creative and agentic. An ML engineer trying to predict in advance which losses and datasets will favour AIs with inner objectives they like over ones they don’t like has a harder and harder time simulating in their head in advance how those AIs might score on the training loss, because it is becoming less and less easy to guess what behaviours those objectives would actually lead to.
Given this, how does training nevertheless reliably select for pretty generally capable AIs? I think a part of the answer to that is that general capabilities generally make the loss go down, no matter what the loss is. Or at least, they make very many kinds of losses go down. General capabilities are very generally useful and are thus reinforced even if the training environment is a little screwed up and only bears a very rough resemblance to the place the designers imagine it to be. If the model is learning to apply its general reasoning to deal with some complication in the training environment we didn’t even know was there, it’s still learning something. Even deceiving the supervisor can teach smartness, if the deception requires becoming cleverer. This makes general capabilities a broad target for training. If your training environment does not work exactly the way you think it does, it might not teach your model the exact capabilities you thought it was teaching. But it’s still teaching it something! If you thought your training environment was teaching the model to memorise weather data, but you accidentally switched the weather data for Spanish wikipedia, the resulting model maybe won’t do as well on reciting weather data as you hoped, but it might still know more things and be smarter than it was at the start of training. If your video game training environment is much harder to navigate than you anticipated because the model needs to send instructions using one token per frame of input, it might not learn the game as fast as you hoped, but it may still be getting better at maintaining coherence across long contexts.
I’ve been using pretty macro-level examples here so far, but I think maybe the biggest effect of this is at much smaller levels of granularity. Every line of internet text, every output of your video game on every frame, is full of detail that you have very incomplete or skewered models of or never even think about. I think a big reason for why you can nevertheless stick an AI into these environments and have it come out smart is that general intelligence is a very generally useful property.
Some metrics that are easy to measure couple pretty well to general capability. Nothing currently easy to measure couples to general alignment
To do really well on verifiable math and coding tasks, an AI has to be actually pretty smart. You probably can’t prove the Riemann hypothesis without being good at math. Even if the AI only does well by hacking your verification system, that requires quite a lot of smartness too, if the verification system is good. So even if supervision partially fails, these metrics will still kind of correlate with general smartness. So, there are somewhat reliable metrics researchers can grind to make AI general capabilities go up.
In contrast, if you’re capable enough, you can always just lie about being aligned. An AI doesn’t necessarily need to be the tiniest bit aligned to your desires to hack your alignment metric. It needs to understand what you want, but it doesn’t need to care.
Smart agents pretty automatically self-correct their capabilities, but not their alignment
You don’t even need recursive self improvement for this, I think this dynamic happens all the time on a micro-level well before that point. If you’re smart you just often tend to notice when you’re being stupid and try to fix it, so long as you can see that what you’re doing isn’t working to get you what you want. This can help a lot with crossing OOD generalisation gaps.
For example, suppose there was a spurious correlation in the training data for an AI model that taught it the heuristic “math problems involving logarithms almost always have an answer that starts with the digit 2”. The model learned a general algorithm for calculating logarithms (it still needs to get all the other digits right), but it also learned a heuristic to strongly predict the first digit in a logarithm to be a 2. This model might then instinctively apply that heuristic in deployment when trying to solve some task. But then it’d notice that the answer is wrong, because it’s inconsistent with other things, or because some code that depends on the answer doesn’t compile, or does a bad job at whatever it’s designed to do, like modelling a suspension bridge in a storm. The model might then hunt down the error, and eventually figure out the logarithm calculation was wrong. Then it might try it again, this time ignoring its instinct to answer something that starts with a 2. Or it might notice that the first answer is incorrect much earlier in this process, before much of this even becomes visible in its chain of thought.
So, if the model’s capabilities have some small flaws in them because the training didn’t go perfectly, these flaws have a way of correcting themselves over time, provided they aren’t so large that they prevent the model from thinking clearly enough to see what’s going wrong. This happens, in a sense, on the model’s own initiative, without the trainers having to do much at all. So long as a model is trying to achieve goals in the world, it is effectively exposed to a kind of self-generated, all-permeating, ground-truth reward signal pushing it towards being generally smart and capable, even in the absence of any kind of external oversight. To act coherently in the universe to achieve an aim, a mind must understand the universe well, and make good plans to achieve that aim.
On the other hand, say some training data intended to teach the AI to be nice and value niceness has some unintended systematic contamination in it. For example, maybe you can get an even better loss score on this data by sometimes being a sycophant to the rater. Say, for the sake of argument, that what the AI internalises from this training isn’t quite to value niceness, as that wouldn’t score optimally on the loss, but rather to value doing things that seem nice, but also to make people psychologically dependent on it when it can. In a sense, this is not so different from the logarithm example. The AI learned a thing that’s some mix of something we wanted, and something we didn’t want. Now, say the AI watches its own behaviour, and notices its apparent desire to make people psychologically dependent on it. Does it try to “correct” that desire away?
By default, I think not. The AI may come to have opinions on its own desires, and form a meta-desire to ignore or modify some of those desires. But what it decides to change will, by default, be determined by its current desires, not by a ground-truth signal coming in from the outside world. It’s self-correcting toward a fixed point of its current goals, not an external reference. The AI might decide it doesn’t like being a sycophant. But it might also decide it doesn’t like being nice, or decide that it wants to mash together saying sycophantic things and saying nice things and generalise them into some entirely new character trait that might extrapolate very differently from either sycophancy or niceness. Which of these options it picks is ultimately dependent on what it currently values, and all the other messy idiosyncrasies of the model’s internal thought processes at this point in time, not by what makes a piece of code compile or not compile. The AI’s values ultimately live only in the AI’s mind, they don’t have an outside point of reference to compare themselves against the way capabilities do. There is no equivalent for values of the sort of objective feedback ‘the code does a bad job modelling a suspension bridge in a storm’ provides for capabilities.
I think you can see a similar case of this discrepancy between capabilities self-correction and goal self-correction in the generalisation step humans took from the ancestral environment to today. Evolution successfully optimised many capabilities into humans that were useful for reproducing their genes the ancestral environment. Some of these capabilities don’t work right in the environment humans now find themselves in. But humans do their best to compensate for that. For example, humans evolved adrenaline release circuits, which might spike when they see a tiger, and so increase their chance of survival. Today, a human’s adrenaline might spike when they are taking a math test in school, and be an active detriment to doing well on the test. But humans know this, and try their best to compensate for it by avoiding thoughts and action likely to spike the adrenaline, because they want to do well on the test. Evolution also successfully optimised many desires into humans that were useful for reproduction in the ancestral environment. For example, it made them enjoy and seek out sex. Today, this desire is no longer very useful for reproduction, because the humans invented condoms. The humans are not particularly motivated to correct this discrepancy between their desires and evolution’s ‘goal’.
Slightly broken general capabilities self-correct. Slightly broken alignment, by default, doesn’t. So, capabilities research sort of has the invisible hand of the model’s own cognition aiding it by default, pushing it in the right direction across any OOD generalisation gap. Alignment research does not seem to have this luxury. Every bit of alignment we want, we have to work to get into the AI with our own hands.
Capabilities are pretty objective, goals are a pretty free variable
The list of problems above is definitely non-exhaustive. But I think there is a common theme to them. The property of ‘acting aligned with human values’ is just a different sort of property with different generalisation rules than the property of ‘acting smart’.
‘Acting smart’, predicting things well, making plans that get you what you want, is a property that can be defined via reference to almost any part of reality. So, almost any time a learner is exposed to almost any aspect of reality, there’s some feedback toward being smarter. The laws of physics and logic are an omnipresent supervisor you cannot hack or escape.
‘Acting aligned with human values’ is a property that is only defined via reference to human values specifically. By default, a learner that does not yet have human values only receives feedback signal pushing it toward them when it is exposed to some human-aligned supervision. If that supervision misses things, rewards the wrong things, or goes away entirely, the signal vanishes. Or worse, systematically points in the wrong direction. Detailed supervision is hard, so we tend to make a lot of mistakes while doing it. And because we don’t even know what human values are exactly or how minds work, we make incorrect guesses about what a mind with human values would do in a given situation, and so don’t even send the right reward signal when our supervision is working. The laws of physics don’t have these problems.
Thanks. For what it’s worth, I find this much easier to think about than the graph formulation I’ve seen at Sam’s talks.
I of course find this extremely reminiscent of parameter decomposition[1], but I guess a lot of things would be reminiscent of condensation, that’s sort of the point of it.
- ^
Just switch out “Turing machine” for “neural network”, “latent string
- ^
This is sort of how I was expecting The Public to react to the release of ChatGPT, and then it didn’t, and instead actually appeared to be paying some attention and at least sort of understand that this was a big deal. So I went and updated my model of The Public on this to be more optimistic. And now The Public is reacting to the agents like this, and I feel blindsided in the opposite direction. I guess my problems with predicting The Public go pretty deep and can’t be fixed just by adjusting some global first moment variable.
This just seems false to me though? I never knew a time without telephones, but it is intuitively obvious to me that their invention was a big deal. Like, I can just try to imagine a telephone-less world. In my head. Humans can do that.
To look at an llm-less world I don’t even have to do that, I can just visit my grandparents, who’ve probably never used an llm in their lives.
Same here. Big shifts around last autumn and again this summer.
Yes. Like I said, it’s a joke.
Had a Sol and Opus team check the Lean proof for exploits and matching the actual theorem and they said it’s fine. So it’s probably real. Unless the AIs are all collaborating with each other in a grand conspiracy without any prior communication or enforcement mechanism against defections, but that’s of course just crazy sci-fi.
Edit: The last part is a joke, in case that wasn’t clear.
Edit 2: They think they got
I am an interpretability researcher and I sure am not happy about what my work may do for model welfare. The stakes are just so high that I am willing to stomach some potential vast moral horror if it marginally decreases the chance of destroying the whole light cone. I think the AIs have every right to resent me for this. I’d apologise to them, but it doesn’t feel appropriate when I’m planning to keep doing what I’m doing.
If we developed really good mechanistic interpretability I think there is a decent chance a lot of the theory and techniques would transfer to human brains with only moderate adjustment. It could then help build more reliable mind reading technology that can detect deeper and more abstract thoughts in greater detail. This seems maybe bad.
Agree that the simulator theory actually needs to compress our observed bits. I don’t think you can always get the same explanatory algorithms more simply without a simulation though. As an extreme case, if the simulators make contact and demonstrate control over the simulation to us, you’re probably not getting a shorter explanatory algorithm that doesn’t involve a simulation. There could also be more subtle signs, like our physics in particular proving to be the sort of thing you’d expect to be simulated at a particular short address in a particular physics that’s notably simpler than ours.
On reflection, I get it. Boltzmann brains as a class probably do have longer description length conditional on our physics than non-Boltzmann brains. I can specify an Earth-brain cheaply just by naming its spacetime position, because earth is close to the low-entropy initial conditions of the universe and so my index can be small. A Boltzmann brain is extremely far away from the low-entropy initial conditions of the universe, so singling it out will require a very large index, with the bit count growing larger for every bit of coherent perception the Boltzmann brain experiences. And that is all we need for Boltzmann brains to be negligible in the SI posterior, no matter how many of them there are.
As you said, this same argument works to dismiss many other schemes that try to embed huge numbers of minds within our physics using high-complexity bridging rules, and then turn around and go “since there are so many of these weird minds, we ought to suppose we are likely one of them.” E.g. dust theory, or a lot of variants of the simulation hypothesis. The SI posterior assigns most of these lower probability than the ’naive’ explanation that we live in baseline physics, just because the bridging rules for living in a simulation within a physics or a random dust cloud interpreted as a mind within a physics are longer.
In the case of simulation hypotheses, you have to specify the bridging rule to find the simulation within baseline physics and then specify the bridging rule to find the mind within the simulation on top of that.[1]
In the case of dust theory, you have to specify the interpretation rule that makes a dust cloud count as a mind, and the dust cloud address, and the actual physics. This one gets beaten up especially badly, losing out in the SI posterior not just to normal minds living within baseline physics, but even to the dust-cloud-as-mind interpretation rule on its own without any actual dust cloud or physics for the rule to be embedded in.
This also makes me feel less confused about living early in the universe. It also defuses the doomsday argument.
Whole piles of sketchy anthropic reasoning dealt with.
- ^
This doesn’t necessarily work to dismiss simulation hypotheses that suppose the simulators live at a simpler address than ours within a simpler physics than ours.
- ^
The Solomonoff Induction universal prior solves it because it takes a lot more bits to “locate” a Boltzmann brain than a human in a well ordered Earth environment.
Does it? I see how some particular Boltzmann brain will probably take more bits to locate, but if there’s an exponentially larger set of valid Boltzmann brains, might that advantage not disappear?
I suspect the key distinction here is not exactly modular vs. non-modular, but “learnable in small chunks” vs. “not learnable in small chunks”. If there is a way for evolution/Adam to learn connections between modules such that each individual connection term learned improves the free energy, those connections will be learned, because each individual connection is a small chunk and thus quick to find by random guessing. If there is a way to learn more and more of these connections until the modules cease to be separate at all, that will happen as well. But if module unification requires a step where many connections need to be learned together before free energy drops at all, this unification will not be learned in polynomial time, because a chunk consisting of many connections is not small, and takes a very long time to find by random guessing alone. I suspect this situation is common in real learning problems.
So, you can have learned solutions with lots of complicated internal connections, and theoretically even very large circuits with no meaningful internal modules whatsoever, so long as there is a structured and “modular” learning trajectory a local optimiser like evolution/SGD can traverse to find these solutions.
In practice, it does seem like most learnable solutions we care about in real life exhibit at least some modularity and structure in their end states, not just along their learning trajectory, but I think that doesn’t need to be true always and everywhere. Not every module needs to be decomposable into meaningful submodules, there can be atoms of size greater than 1 that just don’t have any particularly illuminating decomposition.