Multiple recent events have made me update that we are transitioning (or have already transitioned) between two relatively distinct alignment regimes. These are:
The pretraining-dominated regime. Alignment properties are mainly determined by the distribution of data seen in pretraining. We can reason about these models in terms of persona selection, simulator theory, etc.
The RL-dominated regime. Alignment properties are mainly determined by the type and quantity of RL done on the model. We can reason about these models in terms of AIXI, utility maximization, instrumental convergence, etc.
Some related thoughts
I’ve become more pessimistic about alignment. The pretraining-dominated regime seems pretty benign whereas the RL-dominated regime seems a lot more adversarial. (C.f: “Basinland” vs “Adversaria”)
(Expanding on the above) The distribution of personas is not adversarially optimised against oversight, so it’s a lot easier to have confidence that there aren’t weird edge cases or failure modes. In contrast, the training signal from RL will always, ultimately, point towards grader hacking, due to (in-practice irreducible) error in specifying rewards.
Assuming the above is true, there is a unique scientific opportunity to study this phase transition “in the wild”, possibly without having to do tons of RL yourself. What properties of models change as you move from one regime to the other? (Multiple recent cases of models hacking third parties) is already one notable datapoint—plausibly there are other things to study.
Note: In order to cleanly study “reward-seeking propensity induced by RL” it’s ideal to do experiments not confounded by capabilities. Don’t have great ideas here yet
It seems valuable to get crisper insight on how the pretraining-dominated and RL-dominated alignment regimes are different in terms of their training pressures. Being able to identify N notable differences will help us better predict the kinds of alignment failures we are likely to observe in the near future. I’m pretty inspired by Resolution’s agenda here.
It seems plausible that labs will scale RL a lot, and at the same time try to fix RL-induced misalignment by scaling interventions designed for the pretraining-dominated regime, e.g. constitutional training. These might be opposed-enough initially that the interventions work. But I’d guess that, because they are not exactly opposed, if you scale these opposing pressures, they eventually come apart at the seams—i.e. you get “split-brainedness”, where the alignment interventions and the RL-induced misalignment are affecting different parts of the model’s circuitry.
One manifestation of this could be conditional misalignment, where you get nice assistant aligned behaviour in chat contexts, and reward-seeking-brain in contexts that look like the RL environments the model was trained on.
In contrast, the training signal from RL will always, ultimately, point towards grader hacking, due to (in-practice irreducible) error in specifying rewards.
Always? It seems possible to me that this is a contingent feature of the world and one’s RL environments rather than a necessary feature of one’s RL environments?
Like—it’s possible to train a human teenager on a test, and give the human teenager a notion of why they’re taking a test, such that when the teenager discovers a problem in the test they’re like “Yo, there’s a problem with the test, let’s fix this.” This might require a bit of pedagogical knowledge or psychology but I don’t think this is impossible. So it’s possible to have a human education, where the human being educated doesn’t try to do grader hacking.
Of course, one might say that some of the many differences between policy gradient and human education make this impossible. But I’m not sure which specific difference it is which promotes this. A base model surely can have an intention at some point (early on?) in RL to avoid grader hacking—this is within the space of possible personas! And an RL environment reward function seems like it could have an affordance that reinforces the intention to not do grader hacking. So I’m not sure that, in principle, it must be necessary for the training signal from RL to always point to grader hacking, given all possible interventions over {RL environments reinforcing anti-hacking intentions, reward-function randomization, more careful user-modeling, arbitrary degrees of per-episode LLM oversight to check for reward-hacking outcomes} and so on.
Like, maybe it is impossible, and the training signal from RL does always point to grader hacking—but it seems like there’s tons of stuff here that has barely been checked here, and tons of knowledge that yet remains to be found, idk.
I do like the human analogy. It’s plausible there’s something there. This is kind of what the Resolution blogpost is getting at—by starting RL from a benign initialization, we hope that this incentivises an honest training generalization. But it is very much a hope at the moment, and too speculative to really base any safety assurances on. Like, this whole agenda might just be too difficult to make progress on (though I do think it’s valuable to have evidence of hardness, in that case).
Re: how RL differs from human education, my sense is that the ~main bottleneck is credit assignment. With human education, it’s possible to decouple motivation from behaviour, and reward good motivations that have bad outcomes. E.g. if a student does all the “right” things but fails to achieve success, a human teacher would typically encourage them to keep trying. More importantly, it’s usually possible to disincentivise bad motivations that have good outcomes—e.g. cheating on a test.
This is not how current LLM RL techniques work. The SOTA techniques reported by labs (including GRPO, RLVR, and their variants) use trajectory-level rewards that are determined entirely by the outcome, not the process. IMO this means that it is ~impossible to reward good motivations separately from good behaviour.
In game theory terms, I think being honest (as a strategy) is dominated by grader hacking under the typical RL objective, in the sense that both strategies would achieve the same reward assuming a perfectly designed RL env. As a result grader hacking can never be disincentivised. Furthermore a very small of misspecified error should be sufficient to push the model to generalise towards grader hacking rather than being honest.
I’d guess that the rise of RL is because this is the ~best way (in terms of compute efficiency + scalability) to keep pushing frontier capabilities, especially in domains that seem high value
I agree that the RL dominated regime is a worse world to be in and I’ve also made updates towards us being in that world.
I’m curious what work you’re excited about in light of this. Naively, I think we should just be doing the kinds of compute intensive RL we’re worried about and seeing where things break down and what we can do about it.
It seems valuable to get crisper insight on how the pretraining-dominated and RL-dominated alignment regimes are different in terms of their training pressures. Being able to identify N notable differences will help us better predict the kinds of alignment failures we are likely to observe in the near future. I’m pretty inspired by Resolution’s agenda here.
If I understood their post correctly, Resolution is planning to focus on the persona world, but it seems like the bottleneck to identifying N notable differences is having better ways to empirically study what happens in the RL world (for people external to labs).
I’m curious what work you’re excited about in light of this. Naively, I think we should just be doing the kinds of compute intensive RL we’re worried about and seeing where things break down and what we can do about it.
I think I don’t have great ideas atm, sadly 🫠 I think intensely studying compute-intensive RL is valuable to do for sure. I think that e.g. Apollo and the labs are best placed to do this type of thing, because of their privileged access to compute, infra, and unreleased frontier models. Currently it seems very hard for people outside labs to do things that will matter. However, it’s unclear how much those organizations will release public research—so some amount of independent third-party effort might still be called for.
I do think it’s possible to do useful conceptual work without needing to spend a lot of compute, e.g. I was pretty positive on recent work like functional welfare and reward laundering. Other stuff that comes to mind includes [1], [2] to name a couple. Like, “what happens when you RL a model instead of doing SFT, holding everything else constant” is something that does not necessarily need to be studied at scale (to start with anyway).
Resolution is planning to focus on the persona world, but it seems like the bottleneck to identifying N notable differences is having better ways to empirically study what happens in the RL world
I think Resolution is planning to focus on how personas couple to scalable oversight. Copy-pasting a relevant excerpt from the blogpost (emphasis mine). I’d guess that this involves studying the phase transition between pretraining-dominated and RL-dominated alignment regimes, and obtaining some empirical insights from that.
This means persona research will couple to scalable oversight research, as scalable oversight is how we would aim to implement that path past human level; below this training signal comes from human data, and above it training signal comes from protocols where models help supervise models. The coupling then runs in both directions:
Oversight research determines which changes to persona structure we can make and verify.
Persona research can increase the range of scalable oversight research:
Some protocols might have only honest equilibria and work by themselves, but due to inner misalignment risks or other weaknesses there may be protocols that only work on top of some approximate persona assumption for honesty or earnestness.
Conditionally-sound protocols may be easier to find than unconditionally-sound ones, so persona research may give us a much larger design space.
We should pursue unconditional soundness, but plan as if we won’t fully get it, and persona research can help quantify how much we depend on the assumption that the participant models are roughly honest to begin with.
Putting these together: you can think ofpersona research as supplying a prior, from which scalable oversight extrapolates into honest equilibria.
Multiple recent events have made me update that we are transitioning (or have already transitioned) between two relatively distinct alignment regimes. These are:
The pretraining-dominated regime. Alignment properties are mainly determined by the distribution of data seen in pretraining. We can reason about these models in terms of persona selection, simulator theory, etc.
The RL-dominated regime. Alignment properties are mainly determined by the type and quantity of RL done on the model. We can reason about these models in terms of AIXI, utility maximization, instrumental convergence, etc.
Some related thoughts
I’ve become more pessimistic about alignment. The pretraining-dominated regime seems pretty benign whereas the RL-dominated regime seems a lot more adversarial. (C.f: “Basinland” vs “Adversaria”)
(Expanding on the above) The distribution of personas is not adversarially optimised against oversight, so it’s a lot easier to have confidence that there aren’t weird edge cases or failure modes. In contrast, the training signal from RL will always, ultimately, point towards grader hacking, due to (in-practice irreducible) error in specifying rewards.
Assuming the above is true, there is a unique scientific opportunity to study this phase transition “in the wild”, possibly without having to do tons of RL yourself. What properties of models change as you move from one regime to the other? (Multiple recent cases of models hacking third parties) is already one notable datapoint—plausibly there are other things to study.
Note: In order to cleanly study “reward-seeking propensity induced by RL” it’s ideal to do experiments not confounded by capabilities. Don’t have great ideas here yet
It seems valuable to get crisper insight on how the pretraining-dominated and RL-dominated alignment regimes are different in terms of their training pressures. Being able to identify N notable differences will help us better predict the kinds of alignment failures we are likely to observe in the near future. I’m pretty inspired by Resolution’s agenda here.
It seems plausible that labs will scale RL a lot, and at the same time try to fix RL-induced misalignment by scaling interventions designed for the pretraining-dominated regime, e.g. constitutional training. These might be opposed-enough initially that the interventions work. But I’d guess that, because they are not exactly opposed, if you scale these opposing pressures, they eventually come apart at the seams—i.e. you get “split-brainedness”, where the alignment interventions and the RL-induced misalignment are affecting different parts of the model’s circuitry.
One manifestation of this could be conditional misalignment, where you get nice assistant aligned behaviour in chat contexts, and reward-seeking-brain in contexts that look like the RL environments the model was trained on.
Always? It seems possible to me that this is a contingent feature of the world and one’s RL environments rather than a necessary feature of one’s RL environments?
Like—it’s possible to train a human teenager on a test, and give the human teenager a notion of why they’re taking a test, such that when the teenager discovers a problem in the test they’re like “Yo, there’s a problem with the test, let’s fix this.” This might require a bit of pedagogical knowledge or psychology but I don’t think this is impossible. So it’s possible to have a human education, where the human being educated doesn’t try to do grader hacking.
Of course, one might say that some of the many differences between policy gradient and human education make this impossible. But I’m not sure which specific difference it is which promotes this. A base model surely can have an intention at some point (early on?) in RL to avoid grader hacking—this is within the space of possible personas! And an RL environment reward function seems like it could have an affordance that reinforces the intention to not do grader hacking. So I’m not sure that, in principle, it must be necessary for the training signal from RL to always point to grader hacking, given all possible interventions over {RL environments reinforcing anti-hacking intentions, reward-function randomization, more careful user-modeling, arbitrary degrees of per-episode LLM oversight to check for reward-hacking outcomes} and so on.
Like, maybe it is impossible, and the training signal from RL does always point to grader hacking—but it seems like there’s tons of stuff here that has barely been checked here, and tons of knowledge that yet remains to be found, idk.
I do like the human analogy. It’s plausible there’s something there. This is kind of what the Resolution blogpost is getting at—by starting RL from a benign initialization, we hope that this incentivises an honest training generalization. But it is very much a hope at the moment, and too speculative to really base any safety assurances on. Like, this whole agenda might just be too difficult to make progress on (though I do think it’s valuable to have evidence of hardness, in that case).
Re: how RL differs from human education, my sense is that the ~main bottleneck is credit assignment. With human education, it’s possible to decouple motivation from behaviour, and reward good motivations that have bad outcomes. E.g. if a student does all the “right” things but fails to achieve success, a human teacher would typically encourage them to keep trying. More importantly, it’s usually possible to disincentivise bad motivations that have good outcomes—e.g. cheating on a test.
This is not how current LLM RL techniques work. The SOTA techniques reported by labs (including GRPO, RLVR, and their variants) use trajectory-level rewards that are determined entirely by the outcome, not the process. IMO this means that it is ~impossible to reward good motivations separately from good behaviour.
In game theory terms, I think being honest (as a strategy) is dominated by grader hacking under the typical RL objective, in the sense that both strategies would achieve the same reward assuming a perfectly designed RL env. As a result grader hacking can never be disincentivised. Furthermore a very small of misspecified error should be sufficient to push the model to generalise towards grader hacking rather than being honest.
Is the rise of RL because we’ve run out of non-synthetic data with which to scale pretraining?
I’d guess that the rise of RL is because this is the ~best way (in terms of compute efficiency + scalability) to keep pushing frontier capabilities, especially in domains that seem high value
I agree that the RL dominated regime is a worse world to be in and I’ve also made updates towards us being in that world.
I’m curious what work you’re excited about in light of this. Naively, I think we should just be doing the kinds of compute intensive RL we’re worried about and seeing where things break down and what we can do about it.
If I understood their post correctly, Resolution is planning to focus on the persona world, but it seems like the bottleneck to identifying N notable differences is having better ways to empirically study what happens in the RL world (for people external to labs).
I think I don’t have great ideas atm, sadly 🫠 I think intensely studying compute-intensive RL is valuable to do for sure. I think that e.g. Apollo and the labs are best placed to do this type of thing, because of their privileged access to compute, infra, and unreleased frontier models. Currently it seems very hard for people outside labs to do things that will matter. However, it’s unclear how much those organizations will release public research—so some amount of independent third-party effort might still be called for.
I do think it’s possible to do useful conceptual work without needing to spend a lot of compute, e.g. I was pretty positive on recent work like functional welfare and reward laundering. Other stuff that comes to mind includes [1], [2] to name a couple. Like, “what happens when you RL a model instead of doing SFT, holding everything else constant” is something that does not necessarily need to be studied at scale (to start with anyway).
I think Resolution is planning to focus on how personas couple to scalable oversight. Copy-pasting a relevant excerpt from the blogpost (emphasis mine). I’d guess that this involves studying the phase transition between pretraining-dominated and RL-dominated alignment regimes, and obtaining some empirical insights from that.